2024-03-06T07:10:16.440 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:10:16.440 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --verbose --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/orig.config.yaml 2024-03-06T07:10:16.483 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:10:16.577 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583057' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOEIuWLTm+12VJ3bov/rzO8dy4841aqWY9ismk0tI5pfA7XJBpRwqrixXaDo/t+4RBvJ731qpeKn7LEUALvZCbM= smithi158.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyZ5EEyC61NKKkHzOLiGLA0ph4BnHiCZO+77TglQp5ChIFcRXcKjC/M0t5kp1Ge+qFN7ZktM/qNok9s+xD5a04= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:10:16.578 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:10:16.578 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:10:16.579 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:10:16.583 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:10:16.607 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:10:16.607 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:10:16.608 INFO:teuthology.packaging:ref: None 2024-03-06T07:10:16.608 INFO:teuthology.packaging:tag: None 2024-03-06T07:10:16.608 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:10:16.608 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:10:16.608 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:10:16.830 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:10:16.831 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:10:16.837 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:10:16.837 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:10:16.875 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:10:16.891 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:10:16.900 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:10:16.924 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:39.409421', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOEIuWLTm+12VJ3bov/rzO8dy4841aqWY9ismk0tI5pfA7XJBpRwqrixXaDo/t+4RBvJ731qpeKn7LEUALvZCbM='} 2024-03-06T07:10:16.942 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:39.407981', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyZ5EEyC61NKKkHzOLiGLA0ph4BnHiCZO+77TglQp5ChIFcRXcKjC/M0t5kp1Ge+qFN7ZktM/qNok9s+xD5a04='} 2024-03-06T07:10:16.942 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:10:16.947 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:10:16.947 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:10:16.947 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:10:17.028 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff6f0924f70>, signals=[15]) 2024-03-06T07:10:17.029 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:10:17.056 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:10:17.056 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:10:17.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:17.151 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:10:17.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:17.239 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:10:17.244 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-03-06T07:10:17.261 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-03-06T07:10:17.262 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-03-06T07:10:17.318 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-03-06T07:10:17.318 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-03-06T07:10:17.318 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-03-06T07:10:17.318 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:17.318 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-03-06T07:10:17.318 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:17.318 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:17.318 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:17.319 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:17.319 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:17.319 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:17.319 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:17.319 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:17.319 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:17.319 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-03-06T07:10:17.344 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2024-03-06T07:10:17.361 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2024-03-06T07:10:17.361 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="CentOS Stream" 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="9" 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:ID="centos" 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="9" 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:17.418 INFO:teuthology.orchestra.run.smithi158.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:17.419 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:17.419 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:17.419 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:17.419 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:17.419 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:17.419 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2024-03-06T07:10:17.443 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:10:17.451 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:10:17.467 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:10:17.467 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:17.470 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:17.485 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:10:17.491 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:10:17.492 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:17.527 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:17.557 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:10:17.608 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:10:17.622 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-06T07:10:17.623 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-06T07:10:17.623 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:17.623 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:17.638 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2024-03-06T07:10:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:17.639 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-03-06T07:10:17.694 INFO:teuthology.orchestra.run.smithi158.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:17.694 INFO:teuthology.task.kernel:Running kernel on smithi158: 5.14.0-361.el9.x86_64 2024-03-06T07:10:17.695 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum install -y kernel 2024-03-06T07:10:17.696 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:17.696 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.14.0-361.el9.x86_64 2024-03-06T07:10:17.696 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-03-06T07:10:19.859 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-06T07:10:20.040 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-06T07:10:21.593 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-06T07:10:21.690 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-06T07:10:22.639 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-06T07:10:30.615 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:06 2024-03-06T07:10:31.057 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 904 kB/s | 8.0 MB 00:09 2024-03-06T07:10:34.675 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 19 MB 00:02 2024-03-06T07:10:36.332 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.1 MB 00:02 2024-03-06T07:10:37.899 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-06T07:10:39.186 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-06T07:10:39.818 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.1 MB 00:01 2024-03-06T07:10:41.260 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 15 kB 00:00 2024-03-06T07:10:42.303 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-06T07:10:44.101 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-06T07:10:46.332 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:46.333 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:46.430 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-03-06T07:10:46.439 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-03-06T07:10:46.439 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:46.439 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-03-06T07:10:46.439 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-03-06T07:10:46.439 INFO:teuthology.orchestra.run.smithi158.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:46.439 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-03-06T07:10:46.440 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:46.440 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:46.440 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:46.440 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:10:46.440 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-03-06T07:10:46.440 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-03-06T07:10:46.440 INFO:teuthology.orchestra.run.smithi158.stdout:Install 4 Packages 2024-03-06T07:10:46.440 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:10:46.441 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 100 M 2024-03-06T07:10:46.441 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 123 M 2024-03-06T07:10:46.441 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-03-06T07:10:47.211 INFO:teuthology.orchestra.run.smithi158.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 8.6 MB/s | 6.0 MB 00:00 2024-03-06T07:10:47.261 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-06T07:10:47.752 INFO:teuthology.orchestra.run.smithi158.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 17 MB/s | 21 MB 00:01 2024-03-06T07:10:48.586 INFO:teuthology.orchestra.run.smithi158.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 19 MB/s | 39 MB 00:02 2024-03-06T07:10:49.502 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:49.502 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:49.607 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-03-06T07:10:49.618 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-06T07:10:49.618 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:49.618 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-06T07:10:49.618 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-03-06T07:10:49.618 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:49.618 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-03-06T07:10:49.618 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:49.618 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:49.619 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:49.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:10:49.619 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-03-06T07:10:49.619 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-06T07:10:49.619 INFO:teuthology.orchestra.run.smithi077.stdout:Install 4 Packages 2024-03-06T07:10:49.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:10:49.620 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 100 M 2024-03-06T07:10:49.620 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 123 M 2024-03-06T07:10:49.620 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-03-06T07:10:51.221 INFO:teuthology.orchestra.run.smithi077.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.1 MB/s | 6.0 MB 00:01 2024-03-06T07:10:53.105 INFO:teuthology.orchestra.run.smithi077.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 6.2 MB/s | 21 MB 00:03 2024-03-06T07:10:54.388 INFO:teuthology.orchestra.run.smithi077.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 8.5 MB/s | 39 MB 00:04 2024-03-06T07:10:55.522 INFO:teuthology.orchestra.run.smithi077.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 7.8 MB/s | 33 MB 00:04 2024-03-06T07:10:55.522 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:10:55.522 INFO:teuthology.orchestra.run.smithi077.stdout:Total 17 MB/s | 100 MB 00:05 2024-03-06T07:10:56.406 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-03-06T07:10:56.776 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-03-06T07:10:56.776 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-03-06T07:10:58.114 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-03-06T07:10:58.114 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-03-06T07:10:59.677 INFO:teuthology.orchestra.run.smithi158.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 2.7 MB/s | 33 MB 00:12 2024-03-06T07:10:59.677 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:10:59.677 INFO:teuthology.orchestra.run.smithi158.stdout:Total 7.5 MB/s | 100 MB 00:13 2024-03-06T07:11:00.565 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-03-06T07:11:00.975 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-03-06T07:11:00.976 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-03-06T07:11:01.032 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-03-06T07:11:01.696 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:01.824 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:02.665 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-03-06T07:11:02.666 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-03-06T07:11:04.635 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:04.810 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:05.407 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-03-06T07:11:06.061 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:06.196 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:09.008 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:09.241 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:09.519 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:09.645 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:13.978 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:14.202 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:14.424 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:18.940 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:43.711 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:43.711 INFO:teuthology.orchestra.run.smithi077.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:11:43.712 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:11:43.796 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:44.277 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:44.277 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:44.277 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:44.277 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:44.647 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:44.647 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:11:44.647 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-03-06T07:11:44.647 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:44.648 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:44.648 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:11:44.648 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:44.648 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:11:44.648 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-03-06T07:11:45.121 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:45.265 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:45.265 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:45.265 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:11:45.265 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:11:45.265 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:11:45.266 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-06T07:11:45.266 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:11:45.266 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:45.385 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:45.386 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:45.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2024-03-06T07:11:45.424 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2024-03-06T07:11:45.424 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2024-03-06T07:11:45.424 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.06 2024-03-06T07:11:45.424 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 68.el9 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Group : Unspecified 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 8264254 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : builder@centos.org 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2024-03-06T07:11:45.425 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:11:45.426 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:11:45.426 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:11:45.426 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:11:45.426 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2024-03-06T07:11:45.426 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:11:45.426 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:11:45.427 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:11:45.427 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:11:45.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:11:46.068 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2024-03-06T07:11:47.455 INFO:teuthology.orchestra.run.smithi077.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:11:47.491 INFO:teuthology.orchestra.run.smithi077.stderr:done 2024-03-06T07:11:47.493 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-03-06T07:11:47.508 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.rT8dYrWc8o 2024-03-06T07:11:47.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rT8dYrWc8o 2024-03-06T07:11:47.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.rT8dYrWc8o 2024-03-06T07:11:47.721 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.rT8dYrWc8o is 6KB 2024-03-06T07:11:47.770 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.rT8dYrWc8o 2024-03-06T07:11:47.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:11:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:11:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:11:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:11:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:11:47.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:11:47.995 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2024-03-06T07:11:48.053 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:48.053 INFO:teuthology.orchestra.run.smithi158.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:11:48.053 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:11:48.139 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:48.634 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:48.634 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:48.634 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:48.634 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:49.112 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:49.113 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:11:49.113 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-03-06T07:11:49.113 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:49.113 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:49.113 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:11:49.113 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:49.114 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:11:49.114 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-03-06T07:11:49.679 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:49.824 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:49.825 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:49.825 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:11:49.825 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:11:49.825 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:11:49.825 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-06T07:11:49.825 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi158.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:11:49.825 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:49.945 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:49.945 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:49.946 DEBUG:teuthology.orchestra.run.smithi158:> sudo rpm -qi grub2-tools 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Name : grub2-tools 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Epoch : 1 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Version : 2.06 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Release : 68.el9 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Architecture: x86_64 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Group : Unspecified 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Size : 8264254 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:License : GPLv3+ 2024-03-06T07:11:49.991 INFO:teuthology.orchestra.run.smithi158.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:Packager : builder@centos.org 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:Vendor : CentOS 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:Summary : Support tools for GRUB. 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:Description : 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:hardware devices. 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:11:49.992 INFO:teuthology.orchestra.run.smithi158.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:11:49.993 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:11:49.993 INFO:teuthology.task.kernel:Updating grub on smithi158 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:11:49.994 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:11:50.550 INFO:teuthology.orchestra.run.smithi158.stderr:Generating grub configuration file ... 2024-03-06T07:11:51.974 INFO:teuthology.orchestra.run.smithi158.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:11:52.001 INFO:teuthology.orchestra.run.smithi158.stderr:done 2024-03-06T07:11:52.003 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2024-03-06T07:11:52.018 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.Ul0nP9ZGCw 2024-03-06T07:11:52.019 DEBUG:teuthology.orchestra.run.smithi158:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ul0nP9ZGCw 2024-03-06T07:11:52.088 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.Ul0nP9ZGCw 2024-03-06T07:11:52.221 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.Ul0nP9ZGCw is 6KB 2024-03-06T07:11:52.270 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.Ul0nP9ZGCw 2024-03-06T07:11:52.284 DEBUG:teuthology.orchestra.run.smithi158:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:11:52.372 INFO:teuthology.orchestra.run.smithi158.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:11:52.372 INFO:teuthology.orchestra.run.smithi158.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:11:52.372 INFO:teuthology.orchestra.run.smithi158.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:11:52.373 INFO:teuthology.orchestra.run.smithi158.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:11:52.373 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:11:52.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo shutdown -r now 2024-03-06T07:12:18.020 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-06T07:12:18.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-06T07:12:18.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:22.488 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-06T07:12:22.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-03-06T07:12:22.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:57.153 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-06T07:13:06.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-03-06T07:13:06.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:09.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-06T07:13:18.032 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:13:21.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-03-06T07:13:21.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:27.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-06T07:13:27.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:27.361 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-03-06T07:13:27.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-06T07:13:27.822 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:13:27.823 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-03-06T07:13:27.837 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:13:27.837 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:13:27.837 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:13:27.838 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:13:28.838 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-06T07:13:28.839 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:13:28.960 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-03-06T07:13:28.995 DEBUG:teuthology.parallel:result is None 2024-03-06T07:13:39.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-06T07:13:54.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-03-06T07:13:54.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:54.947 DEBUG:teuthology.orchestra.run.smithi158:> true 2024-03-06T07:13:55.345 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-03-06T07:13:55.346 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:13:55.346 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2024-03-06T07:13:55.360 INFO:teuthology.orchestra.run.smithi158.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:13:55.361 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:13:55.361 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:13:55.361 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:13:56.362 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-06T07:13:56.362 DEBUG:teuthology.orchestra.run.smithi158:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:13:56.438 INFO:teuthology.orchestra.run.smithi158.stdout:ttyS1 2024-03-06T07:13:56.475 DEBUG:teuthology.parallel:result is None 2024-03-06T07:13:56.475 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:13:56.483 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:13:56.483 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:13:56.487 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:13:56.502 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:13:56.508 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:13:56.559 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:13:56.559 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:13:56.562 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:13:56.627 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:13:56.634 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:13:56.634 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:13:56.637 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:13:56.672 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:56.679 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:56.687 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:56.692 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:56.694 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:13:56.701 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:13:56.701 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:13:56.731 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:13:56.759 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:13:56.769 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:13:56.769 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:13:56.802 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:13:56.819 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:56.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:56.960 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:57.044 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:57.088 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:13:57.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:13:57.157 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:57.199 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:57.282 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:57.323 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:57.362 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:13:57.362 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:13:57.426 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-03-06T07:13:57.429 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2024-03-06T07:13:57.480 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:13:57.517 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:13:57.987 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:13:57.995 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:13:57.996 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:13:58.038 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:13:58.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-03-06T07:13:58.154 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-03-06T07:13:58.156 DEBUG:teuthology.orchestra.run.smithi158:> sudo service auditd rotate 2024-03-06T07:13:58.258 INFO:teuthology.orchestra.run.smithi158.stdout:Rotating logs: 2024-03-06T07:13:58.260 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:13:58.261 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-03-06T07:13:58.288 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-03-06T07:13:58.289 DEBUG:teuthology.orchestra.run.smithi158:> /usr/sbin/getenforce 2024-03-06T07:13:58.315 INFO:teuthology.orchestra.run.smithi158.stdout:Permissive 2024-03-06T07:13:58.315 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi158.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:13:58.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:13:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:13:58.360 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:13:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:13:58.389 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:13:58.389 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:13:58.449 DEBUG:teuthology.orchestra.run.smithi158:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:13:58.483 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:13:58.494 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:13:58.511 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:13:58.513 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi158.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:18:24.908 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi158.front.sepia.ceph.com')] 2024-03-06T07:18:24.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-06T07:18:24.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:25.006 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-03-06T07:18:25.095 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-06T07:18:25.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-03-06T07:18:25.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:25.186 DEBUG:teuthology.orchestra.run.smithi158:> true 2024-03-06T07:18:25.259 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-03-06T07:18:25.259 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:18:25.270 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:18:25.270 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:18:25.271 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:18:25.273 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:18:25.273 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:18:25.303 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:18:25.320 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:18:25.331 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:18:25.347 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:18:25.349 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-03-06T07:18:25.362 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-03-06T07:18:25.377 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: ntpd: command not found 2024-03-06T07:18:25.378 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:18:25.391 INFO:teuthology.orchestra.run.smithi158.stdout:506 Cannot talk to daemon 2024-03-06T07:18:25.394 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:18:25.407 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:18:25.424 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:18:25.446 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:18:25.449 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:18:25.449 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-03-06T07:18:25.449 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:25.449 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:25.449 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:25.449 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:25.481 INFO:teuthology.orchestra.run.smithi158.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:18:25.484 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:18:25.484 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2024-03-06T07:18:25.484 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:25.484 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:25.484 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:25.485 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:25.485 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:18:25.494 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:18:25.494 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:18:25.494 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:18:25.494 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:18:25.499 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:18:25.499 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:18:25.500 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:18:25.500 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:18:25.502 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:18:25.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:18:25.645 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:18:25.645 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:18:25.651 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:18:25.651 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:18:25.974 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:18:25.974 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:18:25.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:18:26.002 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:18:26.003 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:18:26.042 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:18:26.043 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:18:26.043 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:18:26.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:18:26.072 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:18:26.072 DEBUG:teuthology.orchestra.run.smithi158:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:18:26.142 DEBUG:teuthology.orchestra.run.smithi158:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:18:26.151 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:18:26.218 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-03-06T07:18:26.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-03-06T07:18:26.223 DEBUG:teuthology.orchestra.run.smithi158:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:18:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:check_obsoletes = 1 2024-03-06T07:18:26.291 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum clean all 2024-03-06T07:18:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:74 files removed 2024-03-06T07:18:26.499 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:18:26.541 INFO:teuthology.orchestra.run.smithi158.stdout:74 files removed 2024-03-06T07:18:26.568 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:18:27.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 81 kB/s | 83 kB 00:01 2024-03-06T07:18:28.057 INFO:teuthology.orchestra.run.smithi158.stdout:ceph packages for x86_64 71 kB/s | 83 kB 00:01 2024-03-06T07:18:28.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-06T07:18:28.899 INFO:teuthology.orchestra.run.smithi158.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-06T07:18:29.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-06T07:18:29.666 INFO:teuthology.orchestra.run.smithi158.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-06T07:18:29.968 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-06T07:18:30.261 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:18:31.452 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-06T07:18:31.644 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-03-06T07:18:32.277 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-06T07:18:32.668 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-06T07:18:34.221 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-06T07:18:38.072 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-06T07:18:39.119 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 19 MB 00:05 2024-03-06T07:18:40.441 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 15 kB 00:00 2024-03-06T07:18:43.383 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 21 MB 00:02 2024-03-06T07:18:48.275 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-06T07:18:50.754 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:50.754 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:50.769 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:18:50.770 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:18:50.771 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:18:50.772 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:50.774 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:50.777 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:18:50.778 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-03-06T07:18:50.895 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-06T07:18:50.895 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-03-06T07:18:50.895 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-06T07:18:50.895 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-03-06T07:18:50.896 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:18:50.896 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:18:50.896 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:18:50.896 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:18:50.896 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:18:50.896 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:18:50.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:18:50.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:18:50.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:18:50.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:18:50.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:18:50.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:18:50.898 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:18:50.898 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:18:50.898 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:18:50.898 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:18:50.898 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:18:50.898 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:18:50.899 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:18:50.899 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:18:50.899 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:18:50.899 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:18:50.899 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:18:50.899 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:18:50.900 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:18:50.900 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:50.900 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:18:50.900 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:18:50.900 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:18:50.900 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-03-06T07:18:50.901 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:50.902 INFO:teuthology.orchestra.run.smithi077.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:18:50.903 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:18:50.904 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:18:50.904 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:50.904 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:50.904 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:18:50.904 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:18:50.904 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:18:50.904 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:50.905 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-03-06T07:18:50.905 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:18:50.905 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:18:50.905 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:18:50.905 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:18:50.905 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:18:50.905 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:18:50.906 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:18:50.907 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:18:50.908 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:18:50.909 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:18:50.910 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:18:50.911 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:18:50.912 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:18:50.913 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout:Install 150 Packages 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 26 Packages 2024-03-06T07:18:50.914 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:18:50.915 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 224 M 2024-03-06T07:18:50.915 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-03-06T07:18:51.302 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - CRB 721 kB/s | 6.1 MB 00:08 2024-03-06T07:18:51.809 INFO:teuthology.orchestra.run.smithi077.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-06T07:18:52.917 INFO:teuthology.orchestra.run.smithi077.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 802 kB/s | 888 kB 00:01 2024-03-06T07:18:53.019 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-03-06T07:18:53.143 INFO:teuthology.orchestra.run.smithi077.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 644 kB/s | 145 kB 00:00 2024-03-06T07:18:54.660 INFO:teuthology.orchestra.run.smithi077.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.5 MB/s | 2.3 MB 00:01 2024-03-06T07:18:54.788 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-06T07:18:55.701 INFO:teuthology.orchestra.run.smithi077.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 1.4 MB/s | 1.5 MB 00:01 2024-03-06T07:18:57.510 INFO:teuthology.orchestra.run.smithi077.stdout:(6/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 2.6 MB/s | 4.7 MB 00:01 2024-03-06T07:18:58.560 INFO:teuthology.orchestra.run.smithi077.stdout:(7/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 762 kB/s | 5.3 MB 00:07 2024-03-06T07:18:59.738 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-06T07:19:02.333 INFO:teuthology.orchestra.run.smithi158.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:19:02.333 INFO:teuthology.orchestra.run.smithi158.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:19:02.346 INFO:teuthology.orchestra.run.smithi158.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:19:02.346 INFO:teuthology.orchestra.run.smithi158.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:19:02.347 INFO:teuthology.orchestra.run.smithi158.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:19:02.348 INFO:teuthology.orchestra.run.smithi158.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:19:02.350 INFO:teuthology.orchestra.run.smithi158.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:19:02.353 INFO:teuthology.orchestra.run.smithi158.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:19:02.353 INFO:teuthology.orchestra.run.smithi158.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:19:02.436 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-03-06T07:19:02.456 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-03-06T07:19:02.457 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-03-06T07:19:02.457 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-03-06T07:19:02.457 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-03-06T07:19:02.457 INFO:teuthology.orchestra.run.smithi158.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:19:02.457 INFO:teuthology.orchestra.run.smithi158.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:19:02.458 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:19:02.458 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:19:02.458 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:19:02.458 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:19:02.458 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:19:02.459 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:19:02.459 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:19:02.459 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:19:02.459 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:19:02.459 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:19:02.460 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:19:02.460 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:19:02.460 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:19:02.460 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:19:02.460 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:19:02.461 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:19:02.461 INFO:teuthology.orchestra.run.smithi158.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:19:02.461 INFO:teuthology.orchestra.run.smithi158.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:19:02.461 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:19:02.461 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:19:02.462 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:19:02.462 INFO:teuthology.orchestra.run.smithi158.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:19:02.462 INFO:teuthology.orchestra.run.smithi158.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:19:02.462 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:19:02.462 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:19:02.462 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:19:02.463 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:19:02.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:19:02.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:19:02.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:19:02.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:19:02.463 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:19:02.464 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:19:02.464 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:19:02.464 INFO:teuthology.orchestra.run.smithi158.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:19:02.464 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrading: 2024-03-06T07:19:02.464 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:19:02.464 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:19:02.465 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:19:02.465 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:19:02.465 INFO:teuthology.orchestra.run.smithi158.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:19:02.465 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:19:02.465 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:19:02.465 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:19:02.466 INFO:teuthology.orchestra.run.smithi158.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:19:02.466 INFO:teuthology.orchestra.run.smithi158.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:19:02.466 INFO:teuthology.orchestra.run.smithi158.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:19:02.466 INFO:teuthology.orchestra.run.smithi158.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:19:02.466 INFO:teuthology.orchestra.run.smithi158.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:19:02.466 INFO:teuthology.orchestra.run.smithi158.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:19:02.467 INFO:teuthology.orchestra.run.smithi158.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:19:02.467 INFO:teuthology.orchestra.run.smithi158.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:19:02.467 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:19:02.467 INFO:teuthology.orchestra.run.smithi158.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:19:02.467 INFO:teuthology.orchestra.run.smithi158.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:19:02.467 INFO:teuthology.orchestra.run.smithi158.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:19:02.467 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:19:02.468 INFO:teuthology.orchestra.run.smithi158.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:19:02.468 INFO:teuthology.orchestra.run.smithi158.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:19:02.468 INFO:teuthology.orchestra.run.smithi158.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:19:02.468 INFO:teuthology.orchestra.run.smithi158.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:19:02.468 INFO:teuthology.orchestra.run.smithi158.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:19:02.468 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-03-06T07:19:02.468 INFO:teuthology.orchestra.run.smithi158.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:19:02.469 INFO:teuthology.orchestra.run.smithi158.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:19:02.469 INFO:teuthology.orchestra.run.smithi158.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:19:02.469 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:19:02.469 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:19:02.469 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:19:02.469 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:19:02.469 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:19:02.470 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:19:02.470 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:19:02.470 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:19:02.470 INFO:teuthology.orchestra.run.smithi158.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:19:02.470 INFO:teuthology.orchestra.run.smithi158.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:19:02.470 INFO:teuthology.orchestra.run.smithi158.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:19:02.471 INFO:teuthology.orchestra.run.smithi158.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:19:02.471 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:19:02.471 INFO:teuthology.orchestra.run.smithi158.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:19:02.471 INFO:teuthology.orchestra.run.smithi158.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:19:02.472 INFO:teuthology.orchestra.run.smithi158.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:19:02.472 INFO:teuthology.orchestra.run.smithi158.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:19:02.472 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:19:02.472 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:19:02.472 INFO:teuthology.orchestra.run.smithi158.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:19:02.472 INFO:teuthology.orchestra.run.smithi158.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:19:02.473 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:19:02.473 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:19:02.473 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:19:02.473 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:19:02.473 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:19:02.473 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:19:02.474 INFO:teuthology.orchestra.run.smithi158.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:19:02.474 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:19:02.474 INFO:teuthology.orchestra.run.smithi158.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:19:02.474 INFO:teuthology.orchestra.run.smithi158.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:19:02.474 INFO:teuthology.orchestra.run.smithi158.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:19:02.474 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:19:02.474 INFO:teuthology.orchestra.run.smithi158.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:19:02.475 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:19:02.475 INFO:teuthology.orchestra.run.smithi158.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:19:02.475 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:19:02.475 INFO:teuthology.orchestra.run.smithi158.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:19:02.475 INFO:teuthology.orchestra.run.smithi158.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:19:02.475 INFO:teuthology.orchestra.run.smithi158.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:19:02.475 INFO:teuthology.orchestra.run.smithi158.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:19:02.476 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:19:02.476 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:19:02.476 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:19:02.476 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:19:02.476 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:19:02.476 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:19:02.477 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:19:02.477 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:19:02.477 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:19:02.477 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:19:02.477 INFO:teuthology.orchestra.run.smithi158.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:19:02.477 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:19:02.477 INFO:teuthology.orchestra.run.smithi158.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:19:02.478 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:19:02.478 INFO:teuthology.orchestra.run.smithi158.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:19:02.478 INFO:teuthology.orchestra.run.smithi158.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:19:02.478 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:19:02.478 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:19:02.478 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:19:02.479 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:19:02.479 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:19:02.479 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:19:02.479 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:19:02.479 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:19:02.479 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:19:02.479 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:19:02.479 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:19:02.480 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:19:02.481 INFO:teuthology.orchestra.run.smithi158.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:19:02.482 INFO:teuthology.orchestra.run.smithi158.stdout:Installing weak dependencies: 2024-03-06T07:19:02.483 INFO:teuthology.orchestra.run.smithi158.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:19:02.483 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:19:02.483 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-03-06T07:19:02.483 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-03-06T07:19:02.483 INFO:teuthology.orchestra.run.smithi158.stdout:Install 150 Packages 2024-03-06T07:19:02.483 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrade 26 Packages 2024-03-06T07:19:02.483 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:19:02.483 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 224 M 2024-03-06T07:19:02.484 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-03-06T07:19:03.339 INFO:teuthology.orchestra.run.smithi158.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-06T07:19:04.739 INFO:teuthology.orchestra.run.smithi158.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 635 kB/s | 888 kB 00:01 2024-03-06T07:19:05.055 INFO:teuthology.orchestra.run.smithi158.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 461 kB/s | 145 kB 00:00 2024-03-06T07:19:07.984 INFO:teuthology.orchestra.run.smithi077.stdout:(8/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 1.6 MB/s | 17 MB 00:10 2024-03-06T07:19:08.098 INFO:teuthology.orchestra.run.smithi158.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 777 kB/s | 2.3 MB 00:03 2024-03-06T07:19:08.126 INFO:teuthology.orchestra.run.smithi077.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 178 kB/s | 25 kB 00:00 2024-03-06T07:19:09.673 INFO:teuthology.orchestra.run.smithi158.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 971 kB/s | 1.5 MB 00:01 2024-03-06T07:19:10.026 INFO:teuthology.orchestra.run.smithi077.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 873 kB/s | 9.8 MB 00:11 2024-03-06T07:19:10.326 INFO:teuthology.orchestra.run.smithi077.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 735 kB/s | 220 kB 00:00 2024-03-06T07:19:10.435 INFO:teuthology.orchestra.run.smithi077.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 250 kB/s | 27 kB 00:00 2024-03-06T07:19:10.552 INFO:teuthology.orchestra.run.smithi077.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 268 kB/s | 31 kB 00:00 2024-03-06T07:19:11.402 INFO:teuthology.orchestra.run.smithi077.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 879 kB/s | 747 kB 00:00 2024-03-06T07:19:11.619 INFO:teuthology.orchestra.run.smithi077.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 754 kB/s | 163 kB 00:00 2024-03-06T07:19:11.706 INFO:teuthology.orchestra.run.smithi158.stdout:(6/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 626 kB/s | 5.3 MB 00:08 2024-03-06T07:19:11.819 INFO:teuthology.orchestra.run.smithi077.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 633 kB/s | 127 kB 00:00 2024-03-06T07:19:12.528 INFO:teuthology.orchestra.run.smithi077.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 708 kB/s | 501 kB 00:00 2024-03-06T07:19:14.873 INFO:teuthology.orchestra.run.smithi158.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 923 kB/s | 4.7 MB 00:05 2024-03-06T07:19:15.403 INFO:teuthology.orchestra.run.smithi077.stdout:(18/176): ceph-common-19.0.0-1739.g1bc250c7.el9 870 kB/s | 20 MB 00:24 2024-03-06T07:19:15.603 INFO:teuthology.orchestra.run.smithi077.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 226 kB/s | 45 kB 00:00 2024-03-06T07:19:15.812 INFO:teuthology.orchestra.run.smithi077.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 631 kB/s | 131 kB 00:00 2024-03-06T07:19:16.104 INFO:teuthology.orchestra.run.smithi077.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 554 kB/s | 162 kB 00:00 2024-03-06T07:19:16.537 INFO:teuthology.orchestra.run.smithi077.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 740 kB/s | 321 kB 00:00 2024-03-06T07:19:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 701 kB/s | 298 kB 00:00 2024-03-06T07:19:17.163 INFO:teuthology.orchestra.run.smithi077.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 496 kB/s | 99 kB 00:00 2024-03-06T07:19:17.355 INFO:teuthology.orchestra.run.smithi077.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 445 kB/s | 85 kB 00:00 2024-03-06T07:19:17.722 INFO:teuthology.orchestra.run.smithi077.stdout:(26/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.0 MB/s | 5.1 MB 00:05 2024-03-06T07:19:17.930 INFO:teuthology.orchestra.run.smithi077.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 819 kB/s | 171 kB 00:00 2024-03-06T07:19:18.097 INFO:teuthology.orchestra.run.smithi077.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 188 kB/s | 31 kB 00:00 2024-03-06T07:19:18.289 INFO:teuthology.orchestra.run.smithi077.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 721 kB/s | 138 kB 00:00 2024-03-06T07:19:19.764 INFO:teuthology.orchestra.run.smithi077.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.2 MB/s | 1.8 MB 00:01 2024-03-06T07:19:20.940 INFO:teuthology.orchestra.run.smithi077.stdout:(31/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 883 kB/s | 3.1 MB 00:03 2024-03-06T07:19:21.323 INFO:teuthology.orchestra.run.smithi077.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 648 kB/s | 248 kB 00:00 2024-03-06T07:19:21.916 INFO:teuthology.orchestra.run.smithi077.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 83 kB/s | 49 kB 00:00 2024-03-06T07:19:22.040 INFO:teuthology.orchestra.run.smithi077.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 123 kB/s | 15 kB 00:00 2024-03-06T07:19:22.624 INFO:teuthology.orchestra.run.smithi077.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 452 kB/s | 264 kB 00:00 2024-03-06T07:19:23.916 INFO:teuthology.orchestra.run.smithi077.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 579 kB/s | 748 kB 00:01 2024-03-06T07:19:24.025 INFO:teuthology.orchestra.run.smithi077.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 169 kB/s | 18 kB 00:00 2024-03-06T07:19:24.158 INFO:teuthology.orchestra.run.smithi077.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-06T07:19:24.217 INFO:teuthology.orchestra.run.smithi077.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-06T07:19:24.250 INFO:teuthology.orchestra.run.smithi077.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-06T07:19:24.263 INFO:teuthology.orchestra.run.smithi158.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 1.0 MB/s | 9.8 MB 00:09 2024-03-06T07:19:24.284 INFO:teuthology.orchestra.run.smithi077.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 615 kB/s | 20 kB 00:00 2024-03-06T07:19:24.326 INFO:teuthology.orchestra.run.smithi077.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-06T07:19:24.356 INFO:teuthology.orchestra.run.smithi158.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 275 kB/s | 25 kB 00:00 2024-03-06T07:19:24.359 INFO:teuthology.orchestra.run.smithi077.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 955 kB/s | 32 kB 00:00 2024-03-06T07:19:24.468 INFO:teuthology.orchestra.run.smithi077.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-06T07:19:24.493 INFO:teuthology.orchestra.run.smithi077.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 695 kB/s | 17 kB 00:00 2024-03-06T07:19:24.518 INFO:teuthology.orchestra.run.smithi077.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 403 kB/s | 10 kB 00:00 2024-03-06T07:19:24.552 INFO:teuthology.orchestra.run.smithi077.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.6 MB/s | 159 kB 00:00 2024-03-06T07:19:24.577 INFO:teuthology.orchestra.run.smithi077.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-06T07:19:24.628 INFO:teuthology.orchestra.run.smithi077.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-06T07:19:24.670 INFO:teuthology.orchestra.run.smithi077.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-06T07:19:24.703 INFO:teuthology.orchestra.run.smithi077.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 730 kB/s | 24 kB 00:00 2024-03-06T07:19:24.729 INFO:teuthology.orchestra.run.smithi077.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-06T07:19:24.771 INFO:teuthology.orchestra.run.smithi077.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-06T07:19:24.829 INFO:teuthology.orchestra.run.smithi077.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-06T07:19:24.863 INFO:teuthology.orchestra.run.smithi077.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-06T07:19:24.988 INFO:teuthology.orchestra.run.smithi077.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-06T07:19:25.080 INFO:teuthology.orchestra.run.smithi077.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 33 MB/s | 3.0 MB 00:00 2024-03-06T07:19:25.139 INFO:teuthology.orchestra.run.smithi077.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 26 MB/s | 1.5 MB 00:00 2024-03-06T07:19:25.189 INFO:teuthology.orchestra.run.smithi077.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-06T07:19:25.223 INFO:teuthology.orchestra.run.smithi077.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-06T07:19:25.373 INFO:teuthology.orchestra.run.smithi077.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-06T07:19:25.432 INFO:teuthology.orchestra.run.smithi077.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-06T07:19:25.465 INFO:teuthology.orchestra.run.smithi077.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-06T07:19:25.616 INFO:teuthology.orchestra.run.smithi077.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-06T07:19:25.658 INFO:teuthology.orchestra.run.smithi077.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 5.3 MB/s | 227 kB 00:00 2024-03-06T07:19:25.691 INFO:teuthology.orchestra.run.smithi077.stdout:(66/176): python3-jsonpatch-1.21-16.el9.noarch. 732 kB/s | 24 kB 00:00 2024-03-06T07:19:25.725 INFO:teuthology.orchestra.run.smithi077.stdout:(67/176): python3-jsonpointer-2.0-4.el9.noarch. 539 kB/s | 18 kB 00:00 2024-03-06T07:19:25.758 INFO:teuthology.orchestra.run.smithi077.stdout:(68/176): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-06T07:19:25.792 INFO:teuthology.orchestra.run.smithi077.stdout:(69/176): python3-markupsafe-1.1.1-12.el9.x86_6 946 kB/s | 32 kB 00:00 2024-03-06T07:19:25.917 INFO:teuthology.orchestra.run.smithi077.stdout:(70/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-06T07:19:25.959 INFO:teuthology.orchestra.run.smithi077.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-06T07:19:25.992 INFO:teuthology.orchestra.run.smithi077.stdout:(72/176): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-06T07:19:26.018 INFO:teuthology.orchestra.run.smithi077.stdout:(73/176): python3-packaging-20.9-5.el9.noarch.r 2.7 MB/s | 69 kB 00:00 2024-03-06T07:19:26.051 INFO:teuthology.orchestra.run.smithi077.stdout:(74/176): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:26.085 INFO:teuthology.orchestra.run.smithi077.stdout:(75/176): python3-protobuf-3.14.0-13.el9.noarch 7.0 MB/s | 238 kB 00:00 2024-03-06T07:19:26.110 INFO:teuthology.orchestra.run.smithi077.stdout:(76/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 5.1 MB/s | 132 kB 00:00 2024-03-06T07:19:26.152 INFO:teuthology.orchestra.run.smithi077.stdout:(77/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-06T07:19:26.186 INFO:teuthology.orchestra.run.smithi077.stdout:(78/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-06T07:19:26.219 INFO:teuthology.orchestra.run.smithi077.stdout:(79/176): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-06T07:19:26.545 INFO:teuthology.orchestra.run.smithi077.stdout:(80/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-06T07:19:26.578 INFO:teuthology.orchestra.run.smithi077.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-06T07:19:26.612 INFO:teuthology.orchestra.run.smithi077.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:26.646 INFO:teuthology.orchestra.run.smithi077.stdout:(83/176): qatlib-23.11.0-1.el9.x86_64.rpm 6.4 MB/s | 215 kB 00:00 2024-03-06T07:19:26.679 INFO:teuthology.orchestra.run.smithi077.stdout:(84/176): qatlib-service-23.11.0-1.el9.x86_64.r 1.0 MB/s | 34 kB 00:00 2024-03-06T07:19:26.714 INFO:teuthology.orchestra.run.smithi077.stdout:(85/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-06T07:19:26.755 INFO:teuthology.orchestra.run.smithi077.stdout:(86/176): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-06T07:19:26.780 INFO:teuthology.orchestra.run.smithi077.stdout:(87/176): uuid-1.6.2-55.el9.x86_64.rpm 2.2 MB/s | 56 kB 00:00 2024-03-06T07:19:26.805 INFO:teuthology.orchestra.run.smithi077.stdout:(88/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 3.4 MB/s | 87 kB 00:00 2024-03-06T07:19:26.831 INFO:teuthology.orchestra.run.smithi077.stdout:(89/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-06T07:19:26.856 INFO:teuthology.orchestra.run.smithi077.stdout:(90/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-06T07:19:26.890 INFO:teuthology.orchestra.run.smithi077.stdout:(91/176): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-06T07:19:26.923 INFO:teuthology.orchestra.run.smithi077.stdout:(92/176): jq-1.6-16.el9.x86_64.rpm 5.4 MB/s | 186 kB 00:00 2024-03-06T07:19:26.974 INFO:teuthology.orchestra.run.smithi077.stdout:(93/176): libgfortran-11.4.1-3.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-03-06T07:19:27.007 INFO:teuthology.orchestra.run.smithi077.stdout:(94/176): libquadmath-11.4.1-3.el9.x86_64.rpm 5.7 MB/s | 194 kB 00:00 2024-03-06T07:19:27.056 INFO:teuthology.orchestra.run.smithi158.stdout:(10/176): ceph-common-19.0.0-1739.g1bc250c7.el9 868 kB/s | 20 MB 00:24 2024-03-06T07:19:27.091 INFO:teuthology.orchestra.run.smithi077.stdout:(95/176): ceph-mgr-diskprediction-local-19.0.0- 1.0 MB/s | 7.4 MB 00:07 2024-03-06T07:19:27.116 INFO:teuthology.orchestra.run.smithi077.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.9 MB/s | 217 kB 00:00 2024-03-06T07:19:27.141 INFO:teuthology.orchestra.run.smithi077.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-06T07:19:27.183 INFO:teuthology.orchestra.run.smithi077.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-06T07:19:27.208 INFO:teuthology.orchestra.run.smithi077.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-06T07:19:27.234 INFO:teuthology.orchestra.run.smithi077.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-03-06T07:19:27.259 INFO:teuthology.orchestra.run.smithi077.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-06T07:19:27.284 INFO:teuthology.orchestra.run.smithi077.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-06T07:19:27.368 INFO:teuthology.orchestra.run.smithi077.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-06T07:19:27.431 INFO:teuthology.orchestra.run.smithi158.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 588 kB/s | 220 kB 00:00 2024-03-06T07:19:27.435 INFO:teuthology.orchestra.run.smithi077.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.4 MB/s | 245 kB 00:00 2024-03-06T07:19:27.535 INFO:teuthology.orchestra.run.smithi077.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 5.0 MB/s | 863 kB 00:00 2024-03-06T07:19:27.574 INFO:teuthology.orchestra.run.smithi158.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 191 kB/s | 27 kB 00:00 2024-03-06T07:19:27.657 INFO:teuthology.orchestra.run.smithi158.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 376 kB/s | 31 kB 00:00 2024-03-06T07:19:28.127 INFO:teuthology.orchestra.run.smithi077.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 522 kB/s | 309 kB 00:00 2024-03-06T07:19:28.244 INFO:teuthology.orchestra.run.smithi077.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-03-06T07:19:28.541 INFO:teuthology.orchestra.run.smithi158.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 846 kB/s | 747 kB 00:00 2024-03-06T07:19:28.586 INFO:teuthology.orchestra.run.smithi077.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 476 kB/s | 548 kB 00:01 2024-03-06T07:19:28.716 INFO:teuthology.orchestra.run.smithi077.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 200 kB/s | 26 kB 00:00 2024-03-06T07:19:28.791 INFO:teuthology.orchestra.run.smithi158.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 654 kB/s | 163 kB 00:00 2024-03-06T07:19:28.828 INFO:teuthology.orchestra.run.smithi077.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 435 kB/s | 49 kB 00:00 2024-03-06T07:19:28.937 INFO:teuthology.orchestra.run.smithi077.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2024-03-06T07:19:28.959 INFO:teuthology.orchestra.run.smithi158.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 760 kB/s | 127 kB 00:00 2024-03-06T07:19:29.204 INFO:teuthology.orchestra.run.smithi077.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 566 kB/s | 151 kB 00:00 2024-03-06T07:19:29.567 INFO:teuthology.orchestra.run.smithi158.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 823 kB/s | 501 kB 00:00 2024-03-06T07:19:29.921 INFO:teuthology.orchestra.run.smithi077.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 838 kB 00:00 2024-03-06T07:19:29.987 INFO:teuthology.orchestra.run.smithi077.stdout:(114/176): libarrow-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 4.4 MB 00:01 2024-03-06T07:19:30.104 INFO:teuthology.orchestra.run.smithi077.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 372 kB/s | 43 kB 00:00 2024-03-06T07:19:30.213 INFO:teuthology.orchestra.run.smithi077.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-03-06T07:19:30.347 INFO:teuthology.orchestra.run.smithi077.stdout:(117/176): python3-certifi-2021.10.8-2.el9.noar 112 kB/s | 15 kB 00:00 2024-03-06T07:19:30.372 INFO:teuthology.orchestra.run.smithi077.stdout:(118/176): python3-asyncssh-2.13.2-4.el9.noarch 1.2 MB/s | 548 kB 00:00 2024-03-06T07:19:30.522 INFO:teuthology.orchestra.run.smithi077.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 981 kB/s | 172 kB 00:00 2024-03-06T07:19:30.648 INFO:teuthology.orchestra.run.smithi077.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 1.3 MB/s | 358 kB 00:00 2024-03-06T07:19:30.673 INFO:teuthology.orchestra.run.smithi077.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-03-06T07:19:30.740 INFO:teuthology.orchestra.run.smithi077.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 384 kB/s | 35 kB 00:00 2024-03-06T07:19:30.799 INFO:teuthology.orchestra.run.smithi077.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-06T07:19:30.915 INFO:teuthology.orchestra.run.smithi077.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-06T07:19:31.007 INFO:teuthology.orchestra.run.smithi077.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2024-03-06T07:19:31.099 INFO:teuthology.orchestra.run.smithi077.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-03-06T07:19:31.208 INFO:teuthology.orchestra.run.smithi077.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-03-06T07:19:31.308 INFO:teuthology.orchestra.run.smithi077.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-06T07:19:31.408 INFO:teuthology.orchestra.run.smithi158.stdout:(18/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 874 kB/s | 17 MB 00:19 2024-03-06T07:19:31.425 INFO:teuthology.orchestra.run.smithi077.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 169 kB/s | 20 kB 00:00 2024-03-06T07:19:31.525 INFO:teuthology.orchestra.run.smithi158.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 388 kB/s | 45 kB 00:00 2024-03-06T07:19:31.650 INFO:teuthology.orchestra.run.smithi077.stdout:(130/176): python3-grpcio-1.46.7-10.el9.x86_64. 2.2 MB/s | 2.0 MB 00:00 2024-03-06T07:19:31.717 INFO:teuthology.orchestra.run.smithi158.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 685 kB/s | 131 kB 00:00 2024-03-06T07:19:31.828 INFO:teuthology.orchestra.run.smithi077.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 2.5 MB/s | 1.0 MB 00:00 2024-03-06T07:19:31.868 INFO:teuthology.orchestra.run.smithi077.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 213 kB/s | 46 kB 00:00 2024-03-06T07:19:31.909 INFO:teuthology.orchestra.run.smithi158.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 843 kB/s | 162 kB 00:00 2024-03-06T07:19:31.926 INFO:teuthology.orchestra.run.smithi077.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 808 kB/s | 79 kB 00:00 2024-03-06T07:19:31.985 INFO:teuthology.orchestra.run.smithi077.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 494 kB/s | 58 kB 00:00 2024-03-06T07:19:32.093 INFO:teuthology.orchestra.run.smithi077.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-06T07:19:32.119 INFO:teuthology.orchestra.run.smithi077.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2024-03-06T07:19:32.211 INFO:teuthology.orchestra.run.smithi077.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 771 kB/s | 90 kB 00:00 2024-03-06T07:19:32.236 INFO:teuthology.orchestra.run.smithi077.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-03-06T07:19:32.251 INFO:teuthology.orchestra.run.smithi158.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 939 kB/s | 321 kB 00:00 2024-03-06T07:19:32.319 INFO:teuthology.orchestra.run.smithi077.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-03-06T07:19:32.361 INFO:teuthology.orchestra.run.smithi077.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-06T07:19:32.428 INFO:teuthology.orchestra.run.smithi077.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-06T07:19:32.462 INFO:teuthology.orchestra.run.smithi077.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-03-06T07:19:32.537 INFO:teuthology.orchestra.run.smithi077.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 470 kB/s | 51 kB 00:00 2024-03-06T07:19:32.543 INFO:teuthology.orchestra.run.smithi158.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.0 MB/s | 298 kB 00:00 2024-03-06T07:19:32.579 INFO:teuthology.orchestra.run.smithi077.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-06T07:19:32.638 INFO:teuthology.orchestra.run.smithi077.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 896 kB/s | 90 kB 00:00 2024-03-06T07:19:32.738 INFO:teuthology.orchestra.run.smithi077.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-03-06T07:19:32.780 INFO:teuthology.orchestra.run.smithi077.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 2.1 MB/s | 427 kB 00:00 2024-03-06T07:19:32.839 INFO:teuthology.orchestra.run.smithi077.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-06T07:19:32.843 INFO:teuthology.orchestra.run.smithi158.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 331 kB/s | 99 kB 00:00 2024-03-06T07:19:33.077 INFO:teuthology.orchestra.run.smithi158.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 366 kB/s | 85 kB 00:00 2024-03-06T07:19:33.289 INFO:teuthology.orchestra.run.smithi077.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-03-06T07:19:34.444 INFO:teuthology.orchestra.run.smithi158.stdout:(26/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.0 MB/s | 5.1 MB 00:04 2024-03-06T07:19:34.794 INFO:teuthology.orchestra.run.smithi158.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 488 kB/s | 171 kB 00:00 2024-03-06T07:19:34.894 INFO:teuthology.orchestra.run.smithi158.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 315 kB/s | 31 kB 00:00 2024-03-06T07:19:35.211 INFO:teuthology.orchestra.run.smithi158.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 436 kB/s | 138 kB 00:00 2024-03-06T07:19:35.978 INFO:teuthology.orchestra.run.smithi158.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 1.1 MB/s | 3.1 MB 00:02 2024-03-06T07:19:36.264 INFO:teuthology.orchestra.run.smithi077.stdout:(150/176): librados2-19.0.0-1739.g1bc250c7.el9. 1.0 MB/s | 3.4 MB 00:03 2024-03-06T07:19:36.297 INFO:teuthology.orchestra.run.smithi077.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:19:36.381 INFO:teuthology.orchestra.run.smithi077.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 27 MB/s | 2.3 MB 00:00 2024-03-06T07:19:36.489 INFO:teuthology.orchestra.run.smithi077.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-03-06T07:19:36.523 INFO:teuthology.orchestra.run.smithi077.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:36.573 INFO:teuthology.orchestra.run.smithi077.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-06T07:19:36.607 INFO:teuthology.orchestra.run.smithi077.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:36.641 INFO:teuthology.orchestra.run.smithi077.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:19:36.674 INFO:teuthology.orchestra.run.smithi077.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 5.6 MB/s | 191 kB 00:00 2024-03-06T07:19:36.708 INFO:teuthology.orchestra.run.smithi077.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 7.5 MB/s | 254 kB 00:00 2024-03-06T07:19:36.758 INFO:teuthology.orchestra.run.smithi077.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-03-06T07:19:36.808 INFO:teuthology.orchestra.run.smithi077.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.5 MB/s | 433 kB 00:00 2024-03-06T07:19:36.834 INFO:teuthology.orchestra.run.smithi077.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 887 kB/s | 22 kB 00:00 2024-03-06T07:19:36.875 INFO:teuthology.orchestra.run.smithi077.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 107 kB 00:00 2024-03-06T07:19:36.918 INFO:teuthology.orchestra.run.smithi077.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 154 kB 00:00 2024-03-06T07:19:36.951 INFO:teuthology.orchestra.run.smithi077.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 4.0 MB/s | 135 kB 00:00 2024-03-06T07:19:36.976 INFO:teuthology.orchestra.run.smithi077.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 2.5 MB/s | 63 kB 00:00 2024-03-06T07:19:37.002 INFO:teuthology.orchestra.run.smithi077.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 28 kB 00:00 2024-03-06T07:19:37.069 INFO:teuthology.orchestra.run.smithi077.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-03-06T07:19:37.186 INFO:teuthology.orchestra.run.smithi077.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 18 MB/s | 2.2 MB 00:00 2024-03-06T07:19:37.319 INFO:teuthology.orchestra.run.smithi077.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 16 MB/s | 2.2 MB 00:00 2024-03-06T07:19:37.370 INFO:teuthology.orchestra.run.smithi077.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 8.6 MB/s | 438 kB 00:00 2024-03-06T07:19:37.403 INFO:teuthology.orchestra.run.smithi077.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 904 kB/s | 30 kB 00:00 2024-03-06T07:19:37.470 INFO:teuthology.orchestra.run.smithi077.stdout:(173/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 768 kB/s | 3.1 MB 00:04 2024-03-06T07:19:37.495 INFO:teuthology.orchestra.run.smithi077.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 3.4 MB/s | 321 kB 00:00 2024-03-06T07:19:37.554 INFO:teuthology.orchestra.run.smithi077.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-06T07:19:37.578 INFO:teuthology.orchestra.run.smithi158.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 770 kB/s | 1.8 MB 00:02 2024-03-06T07:19:37.853 INFO:teuthology.orchestra.run.smithi158.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 905 kB/s | 248 kB 00:00 2024-03-06T07:19:37.920 INFO:teuthology.orchestra.run.smithi158.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 741 kB/s | 49 kB 00:00 2024-03-06T07:19:37.996 INFO:teuthology.orchestra.run.smithi158.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 204 kB/s | 15 kB 00:00 2024-03-06T07:19:38.263 INFO:teuthology.orchestra.run.smithi158.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 988 kB/s | 264 kB 00:00 2024-03-06T07:19:38.913 INFO:teuthology.orchestra.run.smithi158.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.1 MB/s | 748 kB 00:00 2024-03-06T07:19:38.980 INFO:teuthology.orchestra.run.smithi158.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 275 kB/s | 18 kB 00:00 2024-03-06T07:19:39.214 INFO:teuthology.orchestra.run.smithi158.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 662 kB 00:00 2024-03-06T07:19:39.281 INFO:teuthology.orchestra.run.smithi158.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 13 MB/s | 920 kB 00:00 2024-03-06T07:19:39.315 INFO:teuthology.orchestra.run.smithi158.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-06T07:19:39.348 INFO:teuthology.orchestra.run.smithi158.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 616 kB/s | 20 kB 00:00 2024-03-06T07:19:39.390 INFO:teuthology.orchestra.run.smithi158.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-06T07:19:39.441 INFO:teuthology.orchestra.run.smithi158.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 633 kB/s | 32 kB 00:00 2024-03-06T07:19:39.624 INFO:teuthology.orchestra.run.smithi158.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-06T07:19:39.683 INFO:teuthology.orchestra.run.smithi158.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 294 kB/s | 17 kB 00:00 2024-03-06T07:19:39.741 INFO:teuthology.orchestra.run.smithi158.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 173 kB/s | 10 kB 00:00 2024-03-06T07:19:39.784 INFO:teuthology.orchestra.run.smithi158.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-06T07:19:39.817 INFO:teuthology.orchestra.run.smithi158.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:19:39.859 INFO:teuthology.orchestra.run.smithi158.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 16 MB/s | 661 kB 00:00 2024-03-06T07:19:39.926 INFO:teuthology.orchestra.run.smithi158.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-06T07:19:39.960 INFO:teuthology.orchestra.run.smithi158.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 733 kB/s | 24 kB 00:00 2024-03-06T07:19:40.002 INFO:teuthology.orchestra.run.smithi158.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-06T07:19:40.044 INFO:teuthology.orchestra.run.smithi158.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-06T07:19:40.078 INFO:teuthology.orchestra.run.smithi158.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-03-06T07:19:40.111 INFO:teuthology.orchestra.run.smithi158.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-06T07:19:40.303 INFO:teuthology.orchestra.run.smithi158.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-03-06T07:19:40.395 INFO:teuthology.orchestra.run.smithi158.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 33 MB/s | 3.0 MB 00:00 2024-03-06T07:19:40.470 INFO:teuthology.orchestra.run.smithi158.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-06T07:19:40.521 INFO:teuthology.orchestra.run.smithi158.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-06T07:19:40.555 INFO:teuthology.orchestra.run.smithi158.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-06T07:19:40.722 INFO:teuthology.orchestra.run.smithi158.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-06T07:19:40.772 INFO:teuthology.orchestra.run.smithi158.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-03-06T07:19:40.806 INFO:teuthology.orchestra.run.smithi158.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-06T07:19:40.956 INFO:teuthology.orchestra.run.smithi158.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 38 MB/s | 5.8 MB 00:00 2024-03-06T07:19:40.998 INFO:teuthology.orchestra.run.smithi158.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 5.3 MB/s | 227 kB 00:00 2024-03-06T07:19:41.032 INFO:teuthology.orchestra.run.smithi158.stdout:(66/176): python3-jsonpatch-1.21-16.el9.noarch. 737 kB/s | 24 kB 00:00 2024-03-06T07:19:41.065 INFO:teuthology.orchestra.run.smithi158.stdout:(67/176): python3-jsonpointer-2.0-4.el9.noarch. 541 kB/s | 18 kB 00:00 2024-03-06T07:19:41.091 INFO:teuthology.orchestra.run.smithi158.stdout:(68/176): python3-mako-1.1.4-6.el9.noarch.rpm 5.6 MB/s | 144 kB 00:00 2024-03-06T07:19:41.124 INFO:teuthology.orchestra.run.smithi158.stdout:(69/176): python3-markupsafe-1.1.1-12.el9.x86_6 948 kB/s | 32 kB 00:00 2024-03-06T07:19:41.316 INFO:teuthology.orchestra.run.smithi158.stdout:(70/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-06T07:19:41.417 INFO:teuthology.orchestra.run.smithi158.stdout:(71/176): ceph-mgr-diskprediction-local-19.0.0- 1.4 MB/s | 7.4 MB 00:05 2024-03-06T07:19:41.442 INFO:teuthology.orchestra.run.smithi158.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-06T07:19:41.467 INFO:teuthology.orchestra.run.smithi158.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-06T07:19:41.493 INFO:teuthology.orchestra.run.smithi158.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-06T07:19:41.518 INFO:teuthology.orchestra.run.smithi158.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 818 kB/s | 41 kB 00:00 2024-03-06T07:19:41.543 INFO:teuthology.orchestra.run.smithi158.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-06T07:19:41.560 INFO:teuthology.orchestra.run.smithi158.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-06T07:19:41.594 INFO:teuthology.orchestra.run.smithi158.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:19:41.619 INFO:teuthology.orchestra.run.smithi158.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 806 kB/s | 47 kB 00:00 2024-03-06T07:19:41.644 INFO:teuthology.orchestra.run.smithi158.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-06T07:19:41.928 INFO:teuthology.orchestra.run.smithi158.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 136 kB/s | 39 kB 00:00 2024-03-06T07:19:42.237 INFO:teuthology.orchestra.run.smithi158.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-06T07:19:42.262 INFO:teuthology.orchestra.run.smithi158.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 123 kB/s | 41 kB 00:00 2024-03-06T07:19:42.296 INFO:teuthology.orchestra.run.smithi158.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 215 kB 00:00 2024-03-06T07:19:42.321 INFO:teuthology.orchestra.run.smithi158.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 577 kB/s | 34 kB 00:00 2024-03-06T07:19:42.347 INFO:teuthology.orchestra.run.smithi158.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 867 kB/s | 43 kB 00:00 2024-03-06T07:19:42.372 INFO:teuthology.orchestra.run.smithi158.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-06T07:19:42.397 INFO:teuthology.orchestra.run.smithi158.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-06T07:19:42.422 INFO:teuthology.orchestra.run.smithi158.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-06T07:19:42.448 INFO:teuthology.orchestra.run.smithi158.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-06T07:19:42.473 INFO:teuthology.orchestra.run.smithi158.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-06T07:19:42.498 INFO:teuthology.orchestra.run.smithi158.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-06T07:19:42.524 INFO:teuthology.orchestra.run.smithi158.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-06T07:19:42.549 INFO:teuthology.orchestra.run.smithi158.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-03-06T07:19:42.574 INFO:teuthology.orchestra.run.smithi158.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 194 kB 00:00 2024-03-06T07:19:42.600 INFO:teuthology.orchestra.run.smithi158.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-06T07:19:42.625 INFO:teuthology.orchestra.run.smithi158.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-06T07:19:42.667 INFO:teuthology.orchestra.run.smithi158.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-06T07:19:42.692 INFO:teuthology.orchestra.run.smithi158.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-06T07:19:42.718 INFO:teuthology.orchestra.run.smithi158.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi158.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-06T07:19:42.768 INFO:teuthology.orchestra.run.smithi158.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-06T07:19:42.852 INFO:teuthology.orchestra.run.smithi158.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-06T07:19:42.953 INFO:teuthology.orchestra.run.smithi158.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.1 MB/s | 245 kB 00:00 2024-03-06T07:19:43.144 INFO:teuthology.orchestra.run.smithi158.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.9 MB/s | 863 kB 00:00 2024-03-06T07:19:44.186 INFO:teuthology.orchestra.run.smithi158.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 296 kB/s | 309 kB 00:01 2024-03-06T07:19:44.303 INFO:teuthology.orchestra.run.smithi158.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-06T07:19:44.437 INFO:teuthology.orchestra.run.smithi158.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 369 kB/s | 548 kB 00:01 2024-03-06T07:19:44.554 INFO:teuthology.orchestra.run.smithi158.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-03-06T07:19:44.671 INFO:teuthology.orchestra.run.smithi158.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-03-06T07:19:44.871 INFO:teuthology.orchestra.run.smithi158.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 337 kB/s | 67 kB 00:00 2024-03-06T07:19:45.171 INFO:teuthology.orchestra.run.smithi158.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 504 kB/s | 151 kB 00:00 2024-03-06T07:19:46.688 INFO:teuthology.orchestra.run.smithi158.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 553 kB/s | 838 kB 00:01 2024-03-06T07:19:47.630 INFO:teuthology.orchestra.run.smithi158.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 582 kB/s | 548 kB 00:00 2024-03-06T07:19:47.780 INFO:teuthology.orchestra.run.smithi158.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 290 kB/s | 43 kB 00:00 2024-03-06T07:19:47.939 INFO:teuthology.orchestra.run.smithi158.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 203 kB/s | 32 kB 00:00 2024-03-06T07:19:48.048 INFO:teuthology.orchestra.run.smithi158.stdout:(117/176): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-06T07:19:48.356 INFO:teuthology.orchestra.run.smithi158.stdout:(118/176): python3-cheroot-8.6.0-4.el9.noarch.r 557 kB/s | 172 kB 00:00 2024-03-06T07:19:48.932 INFO:teuthology.orchestra.run.smithi158.stdout:(119/176): python3-cherrypy-18.6.1-2.el9.noarch 624 kB/s | 358 kB 00:00 2024-03-06T07:19:49.415 INFO:teuthology.orchestra.run.smithi158.stdout:(120/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 627 kB/s | 303 kB 00:00 2024-03-06T07:19:49.524 INFO:teuthology.orchestra.run.smithi158.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-03-06T07:19:50.166 INFO:teuthology.orchestra.run.smithi158.stdout:(122/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 1.8 MB/s | 47 MB 00:25 2024-03-06T07:19:50.191 INFO:teuthology.orchestra.run.smithi158.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 304 kB/s | 203 kB 00:00 2024-03-06T07:19:51.058 INFO:teuthology.orchestra.run.smithi158.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 167 kB/s | 144 kB 00:00 2024-03-06T07:19:51.208 INFO:teuthology.orchestra.run.smithi158.stdout:(125/176): libarrow-9.0.0-10.el9.x86_64.rpm 655 kB/s | 4.4 MB 00:06 2024-03-06T07:19:51.225 INFO:teuthology.orchestra.run.smithi158.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 64 kB/s | 11 kB 00:00 2024-03-06T07:19:51.342 INFO:teuthology.orchestra.run.smithi158.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 132 kB/s | 18 kB 00:00 2024-03-06T07:19:51.361 INFO:teuthology.orchestra.run.smithi158.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 173 kB/s | 23 kB 00:00 2024-03-06T07:19:51.451 INFO:teuthology.orchestra.run.smithi158.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-06T07:19:51.468 INFO:teuthology.orchestra.run.smithi158.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-06T07:19:51.576 INFO:teuthology.orchestra.run.smithi158.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 428 kB/s | 46 kB 00:00 2024-03-06T07:19:51.694 INFO:teuthology.orchestra.run.smithi158.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 675 kB/s | 79 kB 00:00 2024-03-06T07:19:51.802 INFO:teuthology.orchestra.run.smithi158.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 532 kB/s | 58 kB 00:00 2024-03-06T07:19:52.203 INFO:teuthology.orchestra.run.smithi158.stdout:(134/176): python3-pecan-1.4.2-3.el9.noarch.rpm 681 kB/s | 272 kB 00:00 2024-03-06T07:19:52.319 INFO:teuthology.orchestra.run.smithi158.stdout:(135/176): python3-portend-3.1.0-2.el9.noarch.r 141 kB/s | 16 kB 00:00 2024-03-06T07:19:52.529 INFO:teuthology.orchestra.run.smithi158.stdout:(136/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 431 kB/s | 90 kB 00:00 2024-03-06T07:19:52.662 INFO:teuthology.orchestra.run.smithi158.stdout:(137/176): python3-pyperclip-1.8.2-2.el9.noarch 212 kB/s | 28 kB 00:00 2024-03-06T07:19:52.787 INFO:teuthology.orchestra.run.smithi158.stdout:(138/176): python3-repoze-lru-0.7-16.el9.noarch 246 kB/s | 31 kB 00:00 2024-03-06T07:19:52.803 INFO:teuthology.orchestra.run.smithi077.stdout:(176/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 1.1 MB/s | 47 MB 00:44 2024-03-06T07:19:52.808 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:52.808 INFO:teuthology.orchestra.run.smithi077.stdout:Total 3.6 MB/s | 224 MB 01:01 2024-03-06T07:19:52.855 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-03-06T07:19:53.088 INFO:teuthology.orchestra.run.smithi158.stdout:(139/176): python3-routes-2.5.1-5.el9.noarch.rp 628 kB/s | 188 kB 00:00 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-03-06T07:19:53.213 INFO:teuthology.orchestra.run.smithi158.stdout:(140/176): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-06T07:19:53.338 INFO:teuthology.orchestra.run.smithi158.stdout:(141/176): python3-tempora-5.0.0-2.el9.noarch.r 286 kB/s | 36 kB 00:00 2024-03-06T07:19:53.455 INFO:teuthology.orchestra.run.smithi158.stdout:(142/176): python3-typing-extensions-4.4.0-2.el 437 kB/s | 51 kB 00:00 2024-03-06T07:19:53.764 INFO:teuthology.orchestra.run.smithi158.stdout:(143/176): python3-kubernetes-26.1.0-2.el9.noar 452 kB/s | 1.0 MB 00:02 2024-03-06T07:19:53.789 INFO:teuthology.orchestra.run.smithi158.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 688 kB/s | 230 kB 00:00 2024-03-06T07:19:53.964 INFO:teuthology.orchestra.run.smithi158.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 447 kB/s | 90 kB 00:00 2024-03-06T07:19:54.090 INFO:teuthology.orchestra.run.smithi158.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2024-03-06T07:19:54.282 INFO:teuthology.orchestra.run.smithi158.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 868 kB/s | 427 kB 00:00 2024-03-06T07:19:54.532 INFO:teuthology.orchestra.run.smithi158.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 433 kB/s | 192 kB 00:00 2024-03-06T07:19:54.615 INFO:teuthology.orchestra.run.smithi158.stdout:(149/176): python3-grpcio-1.46.7-10.el9.x86_64. 469 kB/s | 2.0 MB 00:04 2024-03-06T07:19:55.024 INFO:teuthology.orchestra.run.smithi158.stdout:(150/176): librados2-19.0.0-1739.g1bc250c7.el9. 6.9 MB/s | 3.4 MB 00:00 2024-03-06T07:19:55.058 INFO:teuthology.orchestra.run.smithi158.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:19:55.166 INFO:teuthology.orchestra.run.smithi158.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-06T07:19:55.199 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-03-06T07:19:55.199 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-03-06T07:19:55.250 INFO:teuthology.orchestra.run.smithi158.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-06T07:19:55.284 INFO:teuthology.orchestra.run.smithi158.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:55.326 INFO:teuthology.orchestra.run.smithi158.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:19:55.359 INFO:teuthology.orchestra.run.smithi158.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:55.393 INFO:teuthology.orchestra.run.smithi158.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:19:55.435 INFO:teuthology.orchestra.run.smithi158.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:19:55.476 INFO:teuthology.orchestra.run.smithi158.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:19:55.535 INFO:teuthology.orchestra.run.smithi158.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:55.577 INFO:teuthology.orchestra.run.smithi158.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-03-06T07:19:55.611 INFO:teuthology.orchestra.run.smithi158.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 660 kB/s | 22 kB 00:00 2024-03-06T07:19:55.653 INFO:teuthology.orchestra.run.smithi158.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 107 kB 00:00 2024-03-06T07:19:55.687 INFO:teuthology.orchestra.run.smithi158.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 4.5 MB/s | 154 kB 00:00 2024-03-06T07:19:55.720 INFO:teuthology.orchestra.run.smithi158.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.9 MB/s | 135 kB 00:00 2024-03-06T07:19:55.746 INFO:teuthology.orchestra.run.smithi158.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 2.5 MB/s | 63 kB 00:00 2024-03-06T07:19:55.788 INFO:teuthology.orchestra.run.smithi158.stdout:(167/176): thrift-0.15.0-2.el9.x86_64.rpm 1.1 MB/s | 1.6 MB 00:01 2024-03-06T07:19:55.813 INFO:teuthology.orchestra.run.smithi158.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 412 kB/s | 28 kB 00:00 2024-03-06T07:19:55.847 INFO:teuthology.orchestra.run.smithi158.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2024-03-06T07:19:55.905 INFO:teuthology.orchestra.run.smithi158.stdout:(170/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-03-06T07:19:55.956 INFO:teuthology.orchestra.run.smithi158.stdout:(171/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-06T07:19:55.997 INFO:teuthology.orchestra.run.smithi158.stdout:(172/176): util-linux-2.37.4-18.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-06T07:19:56.023 INFO:teuthology.orchestra.run.smithi158.stdout:(173/176): util-linux-core-2.37.4-18.el9.x86_64 3.8 MB/s | 438 kB 00:00 2024-03-06T07:19:56.048 INFO:teuthology.orchestra.run.smithi158.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 332 kB/s | 30 kB 00:00 2024-03-06T07:19:56.073 INFO:teuthology.orchestra.run.smithi158.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-06T07:19:56.107 INFO:teuthology.orchestra.run.smithi158.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-06T07:19:56.111 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:56.111 INFO:teuthology.orchestra.run.smithi158.stdout:Total 4.2 MB/s | 224 MB 00:53 2024-03-06T07:19:56.153 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-03-06T07:19:56.462 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-03-06T07:19:56.462 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-03-06T07:19:58.517 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-03-06T07:19:58.517 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-03-06T07:20:02.126 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-03-06T07:20:02.241 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:20:02.276 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:20:02.305 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:02.384 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:20:02.423 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:20:02.477 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:20:02.643 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:20:02.668 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:20:02.712 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:20:03.040 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:20:03.103 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:03.225 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:03.253 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:03.344 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:03.377 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:03.540 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:20:03.604 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:20:03.633 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:03.735 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:03.758 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:03.807 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:03.854 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:20:03.877 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:03.892 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:04.068 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:04.107 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:04.139 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:20:04.340 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:20:04.448 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:04.501 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:04.596 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:04.643 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:04.845 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:04.948 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:20:05.042 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:20:05.073 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:20:05.141 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:20:05.195 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:20:05.226 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:20:05.261 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:20:05.302 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:20:05.364 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:20:05.401 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:20:05.444 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:20:05.445 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-03-06T07:20:05.477 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:20:05.505 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:20:05.544 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:20:05.573 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:20:05.579 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:20:05.609 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:20:05.626 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:05.653 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:20:05.654 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:20:05.682 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:20:05.705 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:20:05.750 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:20:05.796 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:20:05.797 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:20:05.833 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:20:05.858 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:20:05.869 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:20:05.914 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:20:06.002 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:20:06.057 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:20:06.250 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:20:06.310 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:06.433 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:06.575 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:20:06.636 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:06.660 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:20:06.727 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:06.743 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:20:06.760 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:06.797 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:06.805 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:06.844 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:20:06.881 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:20:06.914 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:20:06.916 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:20:06.971 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:20:06.999 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:07.022 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:20:07.102 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:07.103 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:20:07.125 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:07.190 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:20:07.221 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:20:07.244 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:07.271 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:20:07.388 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:20:07.461 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:07.491 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:20:07.523 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:20:07.556 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:20:07.643 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:20:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:20:07.693 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:20:07.820 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:07.839 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:07.873 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:07.891 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:20:07.929 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:08.000 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:08.000 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:08.032 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:08.060 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:08.069 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:08.116 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:08.152 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:08.195 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:08.237 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:08.271 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:20:08.361 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:08.403 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:20:08.409 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:20:08.439 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:20:08.480 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:20:08.502 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:20:08.531 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:20:08.574 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:20:08.601 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:20:08.614 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:20:08.654 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:20:08.655 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:20:08.680 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:20:08.684 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:20:08.705 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:20:08.752 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:20:08.782 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:20:08.816 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:20:08.851 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:20:08.904 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:20:08.927 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:20:08.954 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:20:09.012 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:20:09.023 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:20:09.051 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:20:09.054 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:20:09.086 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:20:09.102 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:20:09.115 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:20:09.158 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:20:09.186 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:20:09.238 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:20:09.311 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:20:09.343 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:20:09.391 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:20:09.448 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:20:09.498 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:20:09.518 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:20:09.565 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:20:09.597 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:20:09.629 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:20:09.665 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:20:09.695 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:20:09.723 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:20:09.758 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:20:09.797 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:20:09.829 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:20:09.859 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:20:09.934 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:20:09.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:20:10.002 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:20:10.024 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:20:10.043 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:20:10.078 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:20:10.102 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:20:10.167 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:20:10.193 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:20:10.256 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:10.264 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:10.273 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:20:10.303 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:20:10.318 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:20:10.341 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:20:10.348 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:20:10.373 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:20:10.395 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:20:10.484 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:20:10.492 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:20:10.549 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:20:10.558 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:20:10.832 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:20:11.021 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:20:11.101 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:20:11.145 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:20:11.264 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:20:11.283 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:11.305 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:20:11.339 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:20:11.340 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:11.386 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:20:11.396 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:11.466 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:20:11.469 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:11.498 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:20:11.507 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:11.535 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:11.582 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:11.618 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:11.661 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:11.674 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:20:11.704 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:11.704 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:11.788 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:11.838 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:11.884 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:20:11.906 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:20:11.940 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:20:12.034 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:20:12.110 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:20:12.281 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:20:12.405 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:20:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:12.559 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:20:12.644 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:20:12.772 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:20:12.888 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:20:12.929 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:20:12.978 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:20:13.034 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:20:13.219 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:20:13.268 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:20:13.405 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:20:13.454 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:20:13.490 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:20:13.514 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:20:13.548 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:20:13.579 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:20:13.607 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:20:13.642 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:20:13.681 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:20:13.713 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:20:13.743 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:20:13.755 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:13.819 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:20:13.832 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:13.854 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:20:13.858 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:13.886 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:20:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:20:13.963 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:20:14.052 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:20:14.142 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:20:14.171 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:14.185 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:20:14.214 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:14.216 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:20:14.255 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:20:14.304 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:14.327 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:14.355 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:20:14.371 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:14.410 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:20:14.824 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:14.850 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:14.913 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:14.943 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:15.151 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:20:15.192 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:20:15.226 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:20:15.274 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:20:15.363 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:20:15.402 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:20:15.581 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:20:15.625 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:15.703 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:16.353 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:16.481 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:17.173 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:17.215 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:17.722 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:17.753 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:17.785 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:17.804 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:17.830 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:17.875 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:17.876 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:17.876 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:17.876 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:18.141 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:18.169 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:18.284 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:18.316 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:18.353 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:18.806 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:18.830 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:18.902 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:18.932 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:21.132 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:21.187 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:21.725 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:21.765 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:21.858 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:21.858 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:21.859 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:21.859 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:21.892 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:29.654 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-03-06T07:20:29.654 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-03-06T07:20:29.654 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-03-06T07:20:29.654 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-03-06T07:20:29.654 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-03-06T07:20:29.654 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-03-06T07:20:29.655 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-03-06T07:20:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:29.830 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:29.830 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:31.327 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:31.352 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:31.352 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:31.352 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:31.353 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:31.353 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:31.353 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:31.578 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:31.599 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:31.599 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:31.600 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:31.600 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:31.600 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:31.600 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:32.036 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:32.058 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:32.058 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:32.058 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:32.059 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:32.059 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:32.059 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:32.147 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:32.572 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:32.595 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:32.626 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:32.687 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:32.710 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:32.937 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:32.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:33.932 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:33.961 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:34.062 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /sys 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /proc 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /mnt 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /var/tmp 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /home 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /root 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /tmp 2024-03-06T07:20:34.168 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:34.231 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:34.255 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:34.277 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:34.277 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:34.277 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:34.277 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:34.277 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:34.277 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:34.322 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:34.332 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:34.332 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:34.344 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:34.344 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:34.344 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:34.344 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:34.374 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:35.218 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:35.249 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:35.271 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:35.271 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:35.271 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:35.271 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:35.272 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:35.272 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:35.316 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:35.338 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:35.338 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:35.338 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:35.338 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:35.392 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:35.413 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:35.414 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:35.414 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:35.414 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:35.414 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:35.414 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:35.734 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:35.756 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:35.756 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:35.756 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:35.756 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:35.756 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:35.757 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:35.841 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:35.866 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:35.866 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:35.866 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:35.866 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:35.866 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:35.866 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:36.084 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:36.110 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:36.110 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:36.110 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:36.110 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:36.110 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:36.110 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:36.552 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:36.574 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:36.574 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:36.574 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:36.574 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:36.574 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:36.575 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:36.673 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:37.115 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:37.154 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:37.186 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:37.246 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:37.268 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:37.498 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:37.528 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:38.492 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:38.520 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:38.621 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:38.782 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:38.822 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:38.845 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:38.845 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:38.845 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:38.845 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:38.845 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:38.845 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:38.890 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:38.900 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:39.804 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:39.833 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:39.855 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:39.855 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:39.856 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:39.856 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:39.856 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:39.856 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:39.900 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:39.922 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:39.922 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:39.923 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:39.923 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:39.968 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:39.989 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:39.990 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:39.990 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:39.990 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:39.990 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:39.990 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:40.164 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:40.204 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:40.235 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:40.262 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:40.310 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:40.332 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:40.332 INFO:teuthology.orchestra.run.smithi158.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:40.332 INFO:teuthology.orchestra.run.smithi158.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:40.332 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:40.333 INFO:teuthology.orchestra.run.smithi158.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:40.333 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:40.455 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:40.578 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:40.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:40.684 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:40.735 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:40.818 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:40.862 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:40.917 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:40.945 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:40.974 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:41.052 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:41.107 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:41.139 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:41.765 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:41.804 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:41.823 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:42.630 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:42.707 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:42.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:42.833 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:42.869 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:42.894 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:42.919 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:42.946 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:42.946 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:43.014 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:43.041 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:43.080 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:43.080 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:43.166 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:43.192 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:43.225 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:43.250 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:43.280 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:43.310 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:43.351 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:43.422 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:43.456 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:43.481 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:43.507 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:43.532 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:43.538 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:43.538 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:43.801 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:43.833 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:43.860 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:43.885 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:43.910 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:43.976 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:44.740 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:44.789 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:44.819 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:44.847 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:45.036 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:45.146 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:45.186 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:45.244 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:45.303 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:45.383 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:45.430 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:45.482 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:45.514 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:45.543 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:45.628 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:45.690 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:45.725 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:46.368 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:46.415 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:46.434 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:47.268 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:47.343 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:47.385 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:47.476 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:47.505 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:47.530 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:47.556 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:47.582 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:47.582 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:47.675 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:47.703 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:47.733 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:47.733 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:47.802 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:47.828 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:47.870 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:47.903 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:47.940 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:47.971 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:48.005 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:48.052 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:48.093 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:48.118 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:48.143 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:48.168 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:48.195 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:48.200 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:48.200 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:48.380 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:48.380 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:48.471 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:48.503 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:48.530 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:48.555 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:48.589 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:48.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:48.635 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:48.636 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:48.636 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:48.636 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:48.636 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:48.655 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:48.655 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:48.655 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:48.655 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:48.655 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:48.656 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:48.657 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:48.658 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:48.659 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:48.660 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:48.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:48.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:48.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:48.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:48.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:48.667 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:48.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:48.670 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:48.670 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:48.670 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:48.670 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:48.670 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:48.670 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:48.670 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:48.670 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:51.646 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:51.646 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:51.646 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-03-06T07:20:51.646 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:51.646 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.647 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:51.648 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:51.650 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:51.651 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:51.652 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:51.653 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:51.654 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:51.655 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:51.656 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:51.657 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:51.658 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:51.659 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:51.659 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:51.659 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:51.659 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:51.659 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:20:51.659 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-03-06T07:20:52.270 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:53.668 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:53.669 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:53.669 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:53.669 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:53.669 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:53.669 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:53.684 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:53.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:53.686 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:53.687 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:53.688 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:53.689 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:53.690 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:53.691 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:53.692 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:53.693 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:53.694 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:53.695 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:53.696 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:53.697 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:53.697 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:56.733 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:56.733 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:56.733 INFO:teuthology.orchestra.run.smithi158.stdout:Upgraded: 2024-03-06T07:20:56.733 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.734 INFO:teuthology.orchestra.run.smithi158.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-03-06T07:20:56.735 INFO:teuthology.orchestra.run.smithi158.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.736 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.737 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.738 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:56.739 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:56.740 INFO:teuthology.orchestra.run.smithi158.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:56.745 INFO:teuthology.orchestra.run.smithi158.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:56.746 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:56.747 INFO:teuthology.orchestra.run.smithi158.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:56.747 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:20:56.747 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-03-06T07:20:57.429 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:57.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:57.574 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:57.600 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:57.601 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:57.601 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:57.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:57.745 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:57.771 INFO:teuthology.orchestra.run.smithi158.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:57.771 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:57.771 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:57.774 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:20:57.774 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:20:57.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:57.808 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:20:57.809 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:57.846 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:20:57.846 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:20:57.846 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:57.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:57.947 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:20:57.947 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:57.978 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:58.048 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:20:58.048 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:20:58.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:58.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:58.145 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:20:58.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:58.175 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:58.244 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:20:58.245 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:20:58.245 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:58.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:58.340 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:20:58.340 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:58.368 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:58.436 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:20:58.527 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:20:58.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:58.529 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:58.556 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:20:58.557 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:58.596 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:58.623 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:20:58.624 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:58.663 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:58.697 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:20:58.697 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:20:58.697 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:20:58.697 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:20:58.697 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:58.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:58.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:58.887 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:20:58.887 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:58.903 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:59.020 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:59.052 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:20:59.053 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:20:59.053 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583057', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOEIuWLTm+12VJ3bov/rzO8dy4841aqWY9ismk0tI5pfA7XJBpRwqrixXaDo/t+4RBvJ731qpeKn7LEUALvZCbM=', 'smithi158.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyZ5EEyC61NKKkHzOLiGLA0ph4BnHiCZO+77TglQp5ChIFcRXcKjC/M0t5kp1Ge+qFN7ZktM/qNok9s+xD5a04='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:20:59.053 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:59.056 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:59.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:59.111 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:59.170 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:20:59.171 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:59.191 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:59.191 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:59.246 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:59.246 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:59.246 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-06T07:20:59.246 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:59.246 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:59.247 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-06 07:20:44.302927061 +0000 2024-03-06T07:20:59.247 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-06 07:17:53.646937675 +0000 2024-03-06T07:20:59.247 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-06 07:17:53.646937675 +0000 2024-03-06T07:20:59.247 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-06 07:17:53.646937675 +0000 2024-03-06T07:20:59.247 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:59.312 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-06T07:20:59.312 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-06T07:20:59.312 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000195418 s, 2.6 MB/s 2024-03-06T07:20:59.313 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:59.369 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:59.424 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:59.424 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:59.424 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-06T07:20:59.424 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:59.425 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:59.425 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-06 07:20:44.303927039 +0000 2024-03-06T07:20:59.425 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-06 07:17:54.002929745 +0000 2024-03-06T07:20:59.425 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-06 07:17:54.002929745 +0000 2024-03-06T07:20:59.425 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-06 07:17:54.002929745 +0000 2024-03-06T07:20:59.425 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:59.490 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-06T07:20:59.490 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-06T07:20:59.490 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000191058 s, 2.7 MB/s 2024-03-06T07:20:59.491 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:59.546 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:59.600 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:59.600 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:59.600 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-06T07:20:59.600 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:59.601 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:59.601 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-06 07:20:44.303927039 +0000 2024-03-06T07:20:59.601 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-06 07:17:54.358921816 +0000 2024-03-06T07:20:59.601 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-06 07:17:54.358921816 +0000 2024-03-06T07:20:59.601 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-06 07:17:54.358921816 +0000 2024-03-06T07:20:59.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:59.666 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-06T07:20:59.666 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-06T07:20:59.666 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000195345 s, 2.6 MB/s 2024-03-06T07:20:59.667 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:59.723 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-06 07:20:44.303927039 +0000 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-06 07:17:54.691914400 +0000 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-06 07:17:54.691914400 +0000 2024-03-06T07:20:59.777 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-06 07:17:54.691914400 +0000 2024-03-06T07:20:59.778 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-06T07:20:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-06T07:20:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000192572 s, 2.7 MB/s 2024-03-06T07:20:59.844 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:59.899 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:59.899 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:20:59.899 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:59.916 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:59.916 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-03-06 07:20:49.005655309 +0000 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-03-06 07:18:02.455120312 +0000 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-03-06 07:18:02.455120312 +0000 2024-03-06T07:20:59.973 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-03-06 07:18:02.455120312 +0000 2024-03-06T07:20:59.974 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:21:00.040 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-03-06T07:21:00.041 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-03-06T07:21:00.041 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000106271 s, 4.8 MB/s 2024-03-06T07:21:00.042 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:21:00.099 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_2 2024-03-06T07:21:00.154 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:21:00.155 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:00.155 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-06T07:21:00.155 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:00.155 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:00.155 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-03-06 07:20:49.006655298 +0000 2024-03-06T07:21:00.155 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-03-06 07:18:02.810116243 +0000 2024-03-06T07:21:00.155 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-03-06 07:18:02.810116243 +0000 2024-03-06T07:21:00.155 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-03-06 07:18:02.810116243 +0000 2024-03-06T07:21:00.155 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:21:00.220 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-03-06T07:21:00.220 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-03-06T07:21:00.220 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 8.4471e-05 s, 6.1 MB/s 2024-03-06T07:21:00.221 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:21:00.278 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_3 2024-03-06T07:21:00.332 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:21:00.332 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:00.332 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-06T07:21:00.333 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:00.333 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:00.333 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-03-06 07:20:49.006655298 +0000 2024-03-06T07:21:00.333 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-03-06 07:18:03.152112323 +0000 2024-03-06T07:21:00.333 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-03-06 07:18:03.152112323 +0000 2024-03-06T07:21:00.333 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-03-06 07:18:03.152112323 +0000 2024-03-06T07:21:00.333 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:21:00.397 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-03-06T07:21:00.398 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-03-06T07:21:00.398 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 9.5393e-05 s, 5.4 MB/s 2024-03-06T07:21:00.399 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:21:00.455 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_4 2024-03-06T07:21:00.511 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:21:00.511 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:21:00.511 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-06T07:21:00.511 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:21:00.511 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:21:00.512 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-03-06 07:20:49.006655298 +0000 2024-03-06T07:21:00.512 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-03-06 07:18:03.486108494 +0000 2024-03-06T07:21:00.512 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-03-06 07:18:03.486108494 +0000 2024-03-06T07:21:00.512 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-03-06 07:18:03.486108494 +0000 2024-03-06T07:21:00.513 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:21:00.577 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-03-06T07:21:00.577 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-03-06T07:21:00.577 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000100853 s, 5.1 MB/s 2024-03-06T07:21:00.578 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:21:00.634 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:00.634 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:21:00.634 INFO:tasks.ceph:Generating config... 2024-03-06T07:21:00.635 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:21:00.635 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:21:00.635 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:21:00.635 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:21:00.635 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:21:00.636 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:21:00.636 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:21:00.637 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:21:00.637 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:21:00.637 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:21:00.637 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:21:00.637 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:21:00.638 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:21:00.638 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:21:00.638 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:21:00.638 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:21:00.638 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:21:00.638 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:21:00.638 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:21:00.638 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:00.697 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:21:00.700 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:21:00.747 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:21:00.774 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.77'), ('mon.b', '172.21.15.158'), ('mon.c', '[v2:172.21.15.158:3301,v1:172.21.15.158:6790]')] 2024-03-06T07:21:00.774 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:21:00.775 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:00.775 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:21:00.829 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --add b 172.21.15.158 --addv c '[v2:172.21.15.158:3301,v1:172.21.15.158:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = quincy 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:fsid eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2024-03-06T07:21:00.912214+0000 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-03-06T07:21:00.912214+0000 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 17 (quincy) 2024-03-06T07:21:00.913 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2024-03-06T07:21:00.914 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-06T07:21:00.914 INFO:teuthology.orchestra.run.smithi077.stdout:1: [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] mon.b 2024-03-06T07:21:00.914 INFO:teuthology.orchestra.run.smithi077.stdout:2: [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] mon.c 2024-03-06T07:21:00.914 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:21:00.915 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:21:00.929 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID eee1dc2e-9e65-49b7-aabb-947580759696... 2024-03-06T07:21:00.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:21:00.972 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:21:01.019 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-06T07:21:01.020 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-03-06T07:21:01.020 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:21:01.020 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-03-06T07:21:01.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.020 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-03-06T07:21:01.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.020 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-03-06T07:21:01.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-03-06T07:21:01.021 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-03-06T07:21:01.022 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.023 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:21:01.024 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-03-06T07:21:01.025 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-03-06T07:21:01.026 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.027 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-03-06T07:21:01.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-03-06T07:21:01.029 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:21:01.030 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2024-03-06T07:21:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2024-03-06T07:21:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2024-03-06T07:21:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug trim objects = true 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2024-03-06T07:21:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = bitmap 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2024-03-06T07:21:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.037 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:21:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:21:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: mon op complaint time = 120 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:21:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: client mount timeout = 600 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = "" 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.b] 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.c] 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout:[mds] 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds = 20 2024-03-06T07:21:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds balancer = 20 2024-03-06T07:21:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:21:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug frag = True 2024-03-06T07:21:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug scatterstat = True 2024-03-06T07:21:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: mds op complaint time = 180 2024-03-06T07:21:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: mds verify scatter = True 2024-03-06T07:21:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:21:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:21:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:21:01.044 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:21:01.044 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:21:01.140 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:21:01.140 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:01.140 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:21:01.155 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:01.155 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:21:01.209 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:21:01.209 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:01.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:21:01.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:21:01.289 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:01.289 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:01.344 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:21:01.345 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:01.345 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:21:01.345 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:21:01.385 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:01.385 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:01.441 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:21:01.441 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:21:01.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:21:01.497 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:21:01.499 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:21:09.215 INFO:teuthology.orchestra.run.smithi158.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:21:09.218 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:21:09.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:21:09.284 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:21:09.286 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:21:09.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:21:09.411 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:21:09.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:21:09.442 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:21:09.498 INFO:teuthology.orchestra.run.smithi158.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:21:09.501 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:21:09.528 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:21:09.631 INFO:teuthology.orchestra.run.smithi158.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:21:09.633 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:21:09.660 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:21:09.660 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:21:09.702 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:21:09.718 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:21:09.800 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:21:09.815 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:21:09.860 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:21:09.875 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:21:09.958 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:21:09.974 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:21:09.974 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:21:09.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:10.001 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:10.001 INFO:tasks.ceph:role: osd.0 2024-03-06T07:21:10.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:21:10.001 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:21:10.075 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:10.075 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:10.075 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:10.075 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:10.076 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:10.076 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:10.076 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:10.076 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:10.076 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:10.076 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:10.589 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-03-06T07:21:10.591 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-03-06T07:21:10.591 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:10.704 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:10.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:10.810 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:10.810 INFO:tasks.ceph:role: osd.1 2024-03-06T07:21:10.810 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:21:10.810 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:10.886 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:10.887 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:10.887 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:11.347 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-03-06T07:21:11.349 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-03-06T07:21:11.349 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:11.444 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:11.478 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:11.548 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:11.548 INFO:tasks.ceph:role: osd.2 2024-03-06T07:21:11.548 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:21:11.548 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:21:11.624 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:11.624 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:11.624 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:11.624 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:11.624 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:11.625 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:11.625 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:11.625 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:11.625 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:11.625 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:12.110 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-03-06T07:21:12.112 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-03-06T07:21:12.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:12.260 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:12.294 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:12.363 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:12.363 INFO:tasks.ceph:role: osd.3 2024-03-06T07:21:12.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:21:12.364 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:21:12.439 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:12.439 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:12.439 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:12.439 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:12.440 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:12.955 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-03-06T07:21:12.957 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-03-06T07:21:12.957 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:13.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:13.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:13.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:13.250+0000 7fa6b8611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:21:13.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:13.251+0000 7fa6b8611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:21:13.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:13.251+0000 7fa6b8611640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:13.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:13.251+0000 7fa6b8611640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:13.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:13.251+0000 7fa6b8611640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:21:13.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:13.252+0000 7fa6b8611640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:21:16.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:21:16.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:16.226+0000 7f2e4c618640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:21:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:16.226+0000 7f2e4c618640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:21:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:16.226+0000 7f2e4c618640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:16.226+0000 7f2e4c618640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:16.226+0000 7f2e4c618640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:21:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:16.226+0000 7f2e4c618640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:21:19.074 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:19.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:19.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:19.192+0000 7f0352cf9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:19.192+0000 7f0352cf9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:19.192+0000 7f0352cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:19.192+0000 7f0352cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:19.192+0000 7f0352cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:19.193+0000 7f0352cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:21:22.052 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:22.083 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:22.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:22.172+0000 7f9534611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:21:22.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:22.172+0000 7f9534611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:21:22.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:22.172+0000 7f9534611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:22.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:22.172+0000 7f9534611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:22.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:22.172+0000 7f9534611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:22.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:22.173+0000 7f9534611640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:21:25.032 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:25.063 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:25.095 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:25.096 INFO:tasks.ceph:role: osd.4 2024-03-06T07:21:25.096 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:21:25.096 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:21:25.166 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:25.166 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:25.166 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:25.166 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:25.166 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:25.166 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:25.166 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:25.166 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:25.167 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:25.167 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:25.199 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2024-03-06T07:21:25.200 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2024-03-06T07:21:25.201 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:25.273 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:25.346 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:25.412 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:25.413 INFO:tasks.ceph:role: osd.5 2024-03-06T07:21:25.413 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:21:25.413 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:21:25.478 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:25.478 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:25.478 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:25.478 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:25.478 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:25.479 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:25.479 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:25.479 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:25.479 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:25.480 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:25.511 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2024-03-06T07:21:25.512 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2024-03-06T07:21:25.512 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:25.589 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:25.660 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:25.728 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:25.728 INFO:tasks.ceph:role: osd.6 2024-03-06T07:21:25.728 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:21:25.728 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:21:25.794 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:25.794 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:25.794 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:25.794 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:25.794 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:25.794 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:25.795 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:25.795 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:25.795 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:25.795 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:25.826 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2024-03-06T07:21:25.828 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2024-03-06T07:21:25.828 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:25.899 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:25.973 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:26.040 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:26.041 INFO:tasks.ceph:role: osd.7 2024-03-06T07:21:26.041 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:21:26.041 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:21:26.107 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:26.107 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:26.107 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:26.107 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:26.107 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:26.108 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:26.108 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:26.108 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:26.108 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:26.108 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:26.140 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2024-03-06T07:21:26.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2024-03-06T07:21:26.142 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:26.212 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:26.283 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:26.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:26.400+0000 7f3856f93640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:21:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:26.400+0000 7f3856f93640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:21:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:26.401+0000 7f3856f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:26.401+0000 7f3856f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:26.401+0000 7f3856f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:26.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:26.401+0000 7f3856f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:21:29.054 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:29.057 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:29.243 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:29.274 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:29.364+0000 7f2cb3338640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:21:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:29.364+0000 7f2cb3338640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:21:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:29.364+0000 7f2cb3338640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:29.364+0000 7f2cb3338640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:29.364+0000 7f2cb3338640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:29.366 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:29.364+0000 7f2cb3338640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:21:32.216 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:32.248 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:32.338+0000 7f644600c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:21:32.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:32.338+0000 7f644600c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:32.338+0000 7f644600c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:32.338+0000 7f644600c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:32.338+0000 7f644600c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:32.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:32.339+0000 7f644600c640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:21:35.189 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:35.218 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:35.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:35.306+0000 7f37eeca4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:21:35.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:35.306+0000 7f37eeca4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:21:35.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:35.306+0000 7f37eeca4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:35.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:35.306+0000 7f37eeca4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:35.309 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:35.306+0000 7f37eeca4640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:35.309 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:35.306+0000 7f37eeca4640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:21:38.164 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:38.196 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:21:38.196 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:38.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:21:38.228 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:38.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:21:38.299 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:38.299 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:21:38.370 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:38.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:21:38.439 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:38.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:21:38.508 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:38.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:21:38.578 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:38.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:21:38.645 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:38.646 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:21:38.677 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:38.677 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:21:38.748 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:38.749 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:21:38.817 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:38.817 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:21:38.886 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:38.886 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:21:38.954 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:38.955 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:21:39.021 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:39.021 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:21:39.088 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:39.089 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:21:39.106 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:39.106 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:21:39.163 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:39.163 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:21:39.178 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:39.178 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:21:39.233 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:21:39.233 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:39.235 DEBUG:teuthology.orchestra.run.smithi158:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:39.277 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.y] 2024-03-06T07:21:39.277 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ== 2024-03-06T07:21:39.277 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-03-06T07:21:39.277 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w== 2024-03-06T07:21:39.277 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w== 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g== 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ== 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g== 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.3] 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ== 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g== 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg== 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-03-06T07:21:39.278 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ== 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.4] 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA== 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.5] 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A== 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.6] 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA== 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.7] 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA== 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout:[client.2] 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ== 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout:[client.1] 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w== 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout:[client.3] 2024-03-06T07:21:39.279 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg== 2024-03-06T07:21:39.298 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr.y] 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ== 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w== 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w== 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.0] 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g== 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.1] 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ== 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.2] 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g== 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.3] 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ== 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr.x] 2024-03-06T07:21:39.299 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g== 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg== 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ== 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.4] 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA== 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.5] 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A== 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.6] 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA== 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.7] 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA== 2024-03-06T07:21:39.300 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2024-03-06T07:21:39.301 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:21:39.301 INFO:teuthology.orchestra.run.smithi158.stdout:[client.2] 2024-03-06T07:21:39.301 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ== 2024-03-06T07:21:39.301 INFO:teuthology.orchestra.run.smithi158.stdout:[client.1] 2024-03-06T07:21:39.301 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w== 2024-03-06T07:21:39.301 INFO:teuthology.orchestra.run.smithi158.stdout:[client.3] 2024-03-06T07:21:39.301 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg== 2024-03-06T07:21:39.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:39.323 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:39.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:39.398 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:39.454 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:39.456 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:39.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.513 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.570 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.622 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.625 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.678 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.680 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:39.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:39.789 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:39.791 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:39.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:39.850 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:39.904 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.907 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:39.961 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:40.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:40.018 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:40.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:40.077 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:40.129 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:40.132 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:40.188 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:40.190 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:40.246 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:40.249 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:40.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:40.308 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:40.363 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:21:40.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:40.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:40.488 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:40.516 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:40.543 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:40.641 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:40.668 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:40.735 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:40.833 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:40.860 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:40.862 DEBUG:teuthology.orchestra.run.smithi158:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:40.916 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:21:40.917 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:21:40.917 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:21:40.919 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:21:40.920 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:21:40.920 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:21:40.959 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:21:40.959 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:21:40.959 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:21:40.961 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:21:40.961 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:21:40.961 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:21:40.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:21:40.963 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:21:40.963 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:21:40.964 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:21:40.965 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:21:40.966 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:21:40.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:21:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.064+0000 7f03fd8af640 1 Processor -- start 2024-03-06T07:21:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.064+0000 7f03fd8af640 1 -- start start 2024-03-06T07:21:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03fd8af640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 0x7f03f80cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03fd8af640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f03f806b6c0 0x7f03f80c7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03fd8af640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f80cbf30 con 0x7f03f8160890 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03fd8af640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f806e980 con 0x7f03f815c130 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03fd8af640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f8160400 con 0x7f03f806b6c0 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03f6ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 0x7f03f80cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03f6ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 0x7f03f80cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35032/0 (socket says 172.21.15.77:35032) 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03f6ffd640 1 -- 172.21.15.77:0/3836526957 learned_addr learned my addr 172.21.15.77:0/3836526957 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03f6ffd640 1 -- 172.21.15.77:0/3836526957 >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 msgr2=0x7f03f80cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03f6ffd640 1 -- 172.21.15.77:0/3836526957 >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 msgr2=0x7f03f80cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03f6ffd640 1 --2- 172.21.15.77:0/3836526957 >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 0x7f03f80cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03f6ffd640 1 --2- 172.21.15.77:0/3836526957 >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 0x7f03f80cb670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:41.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.065+0000 7f03f77fe640 1 --2- 172.21.15.77:0/3836526957 >> v2:172.21.15.158:3300/0 conn(0x7f03f806b6c0 0x7f03f80c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.066+0000 7f03f77fe640 1 -- 172.21.15.77:0/3836526957 >> v2:172.21.15.158:3300/0 conn(0x7f03f806b6c0 msgr2=0x7f03f80c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:21:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.066+0000 7f03f77fe640 1 -- 172.21.15.77:0/3836526957 >> v2:172.21.15.158:3300/0 conn(0x7f03f806b6c0 msgr2=0x7f03f80c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.066+0000 7f03f77fe640 1 --2- 172.21.15.77:0/3836526957 >> v2:172.21.15.158:3300/0 conn(0x7f03f806b6c0 0x7f03f80c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:41.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.066+0000 7f03f77fe640 1 --2- 172.21.15.77:0/3836526957 >> v2:172.21.15.158:3300/0 conn(0x7f03f806b6c0 0x7f03f80c7490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727741970 0 0) 0x7f03e0001110 con 0x7f03f8160890 2024-03-06T07:21:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e4003520 con 0x7f03f8160890 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 <== mon.1 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3653833943 0 0) 0x7f03e4003520 con 0x7f03f8160890 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 >> v2:172.21.15.158:3300/0 conn(0x7f03f806b6c0 msgr2=0x7f03f80c7490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 --2- 172.21.15.77:0/3836526957 >> v2:172.21.15.158:3300/0 conn(0x7f03f806b6c0 0x7f03f80c7490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 msgr2=0x7f03f80cb670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 --2- 172.21.15.77:0/3836526957 >> v2:172.21.15.77:3300/0 conn(0x7f03f815c130 0x7f03f80cb670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f815c510 con 0x7f03f8160890 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 <== mon.1 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f03e0001840 con 0x7f03f8160890 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 >> v1:172.21.15.158:6789/0 conn(0x7f03f8160890 legacy=0x7f03f806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03e40039b0 0x7f03e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e4004800 0x7f03e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e40051c0 0x7f03e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f806e980 con 0x7f03e4004800 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8160400 con 0x7f03e40051c0 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e400c0c0 con 0x7f03e40039b0 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03effff640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e4004800 0x7f03e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f6ffd640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03e40039b0 0x7f03e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.256+0000 7f03f77fe640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e40051c0 0x7f03e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.257+0000 7f03effff640 1 -- 172.21.15.77:0/3836526957 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03e40039b0 msgr2=0x7f03e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.257+0000 7f03effff640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03e40039b0 0x7f03e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.257+0000 7f03effff640 1 -- 172.21.15.77:0/3836526957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e40051c0 msgr2=0x7f03e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.257+0000 7f03effff640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e40051c0 0x7f03e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.257+0000 7f03f77fe640 1 -- 172.21.15.77:0/3836526957 reap_dead start 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.257+0000 7f03effff640 1 -- 172.21.15.77:0/3836526957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e4003520 con 0x7f03e4004800 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.258+0000 7f03effff640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e4004800 0x7f03e4004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03e0007fd0 tx=0x7f03e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d1e3f636567492 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.258+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0007e30 con 0x7f03e4004800 2024-03-06T07:21:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.258+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e002cbd0 con 0x7f03e4004800 2024-03-06T07:21:41.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.258+0000 7f03f67fc640 1 -- 172.21.15.77:0/3836526957 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0007030 con 0x7f03e4004800 2024-03-06T07:21:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.259+0000 7f03fd8af640 1 -- 172.21.15.77:0/3836526957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e4004800 msgr2=0x7f03e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.259+0000 7f03fd8af640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e4004800 0x7f03e4004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03e0007fd0 tx=0x7f03e4005830 comp rx=0 tx=0).stop 2024-03-06T07:21:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.259+0000 7f03fd8af640 1 -- 172.21.15.77:0/3836526957 shutdown_connections 2024-03-06T07:21:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.259+0000 7f03fd8af640 1 --2- 172.21.15.77:0/3836526957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e4004800 0x7f03e4004c80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.259+0000 7f03fd8af640 1 -- 172.21.15.77:0/3836526957 >> 172.21.15.77:0/3836526957 conn(0x7f03f80cd420 msgr2=0x7f03f80cd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.259+0000 7f03fd8af640 1 -- 172.21.15.77:0/3836526957 shutdown_connections 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.259+0000 7f03fd8af640 1 -- 172.21.15.77:0/3836526957 wait complete. 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.261+0000 7f03fd8af640 1 Processor -- start 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.261+0000 7f03fd8af640 1 -- start start 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03fd8af640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e4004800 0x7f03f80ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03fd8af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03f814f110 0x7f03f814f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03fd8af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03f80ceb90 0x7f03f80cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03fd8af640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f815fdb0 con 0x7f03f814f110 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03fd8af640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f80ce1d0 con 0x7f03e4004800 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03fd8af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8160090 con 0x7f03f80ceb90 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03f814f110 0x7f03f814f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03f814f110 0x7f03f814f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35054/0 (socket says 172.21.15.77:35054) 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03effff640 1 -- 172.21.15.77:0/2501127250 learned_addr learned my addr 172.21.15.77:0/2501127250 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03f6ffd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e4004800 0x7f03f80ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03f77fe640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03f80ceb90 0x7f03f80cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03effff640 1 -- 172.21.15.77:0/2501127250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03f80ceb90 msgr2=0x7f03f80cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03effff640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03f80ceb90 0x7f03f80cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03effff640 1 -- 172.21.15.77:0/2501127250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e4004800 msgr2=0x7f03f80ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03effff640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e4004800 0x7f03f80ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03effff640 1 -- 172.21.15.77:0/2501127250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f80cf900 con 0x7f03f814f110 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.262+0000 7f03f6ffd640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e4004800 0x7f03f80ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.263+0000 7f03effff640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03f814f110 0x7f03f814f570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03e000c3b0 tx=0x7f03f8160370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.263+0000 7f03f4ff9640 1 -- 172.21.15.77:0/2501127250 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0042070 con 0x7f03f814f110 2024-03-06T07:21:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.263+0000 7f03f4ff9640 1 -- 172.21.15.77:0/2501127250 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e0007020 con 0x7f03f814f110 2024-03-06T07:21:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.263+0000 7f03f4ff9640 1 -- 172.21.15.77:0/2501127250 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0007410 con 0x7f03f814f110 2024-03-06T07:21:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.263+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f80cfbe0 con 0x7f03f814f110 2024-03-06T07:21:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.263+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f80d01a0 con 0x7f03f814f110 2024-03-06T07:21:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.263+0000 7f03f4ff9640 1 -- 172.21.15.77:0/2501127250 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f03e0006b60 con 0x7f03f814f110 2024-03-06T07:21:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.263+0000 7f03f4ff9640 1 -- 172.21.15.77:0/2501127250 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f03e0047790 con 0x7f03f814f110 2024-03-06T07:21:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.264+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f806b4e0 con 0x7f03f814f110 2024-03-06T07:21:41.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.268+0000 7f03f4ff9640 1 -- 172.21.15.77:0/2501127250 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f03f806b4e0 con 0x7f03f814f110 2024-03-06T07:21:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.333+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f03f806b2e0 con 0x7f03f814f110 2024-03-06T07:21:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.334+0000 7f03f4ff9640 1 -- 172.21.15.77:0/2501127250 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f03f806b2e0 con 0x7f03f814f110 2024-03-06T07:21:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-03-06T07:21:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03f814f110 msgr2=0x7f03f814f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03f814f110 0x7f03f814f570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03e000c3b0 tx=0x7f03f8160370 comp rx=0 tx=0).stop 2024-03-06T07:21:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 shutdown_connections 2024-03-06T07:21:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03f814f110 0x7f03f814f570 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03f80ceb90 0x7f03f80cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 --2- 172.21.15.77:0/2501127250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e4004800 0x7f03f80ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 >> 172.21.15.77:0/2501127250 conn(0x7f03f80cd420 msgr2=0x7f03f80b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 shutdown_connections 2024-03-06T07:21:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:41.335+0000 7f03fd8af640 1 -- 172.21.15.77:0/2501127250 wait complete. 2024-03-06T07:21:41.348 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:21:41.349 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:21:41.349 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:41.349 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:21:41.375 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:41.376 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:21:41.446 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:41.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:21:41.513 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:21:41.513 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:21:41.579 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:41.580 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:21:41.607 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:41.607 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:21:41.674 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:41.674 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:21:41.728 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:41.726+0000 7fc777a13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:41.739 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:41.737+0000 7efdbe00d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:41.741 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:21:41.741 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:21:41.784 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:41.782+0000 7fc777a13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:41.791 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:41.789+0000 7efdbe00d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:41.808 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd new 2d18ca08-fdf5-44b6-9cc6-a264001ea4fa 0 2024-03-06T07:21:41.950 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.948+0000 7f7ae5116640 1 Processor -- start 2024-03-06T07:21:41.950 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.948+0000 7f7ae5116640 1 -- start start 2024-03-06T07:21:41.951 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ae5116640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ae006b600 0x7f7ae00b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.951 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ae5116640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ae015ecb0 0x7f7ae0157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.951 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ae5116640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae00b3a00 con 0x7f7ae006b220 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ae5116640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00b3ce0 con 0x7f7ae015ecb0 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ae5116640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae015e4b0 con 0x7f7ae006b600 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ade59c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ae006b600 0x7f7ae00b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ade59c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ae006b600 0x7f7ae00b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34364/0 (socket says 172.21.15.158:34364) 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ade59c640 1 -- 172.21.15.158:0/3841873835 learned_addr learned my addr 172.21.15.158:0/3841873835 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7adf59e640 1 --2- 172.21.15.158:0/3841873835 >> v2:172.21.15.77:3300/0 conn(0x7f7ae015ecb0 0x7f7ae0157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ade59c640 1 -- 172.21.15.158:0/3841873835 >> v2:172.21.15.77:3300/0 conn(0x7f7ae015ecb0 msgr2=0x7f7ae0157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ade59c640 1 --2- 172.21.15.158:0/3841873835 >> v2:172.21.15.77:3300/0 conn(0x7f7ae015ecb0 0x7f7ae0157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ade59c640 1 -- 172.21.15.158:0/3841873835 >> v1:172.21.15.158:6789/0 conn(0x7f7ae006b220 legacy=0x7f7ae0072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.949+0000 7f7ade59c640 1 -- 172.21.15.158:0/3841873835 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae015e790 con 0x7f7ae006b600 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.950+0000 7f7ade59c640 1 --2- 172.21.15.158:0/3841873835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 0x7f7ae00b58f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ae006fb00 tx=0x7f7ad002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a14a3016160fdafe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.950+0000 7f7addd9b640 1 -- 172.21.15.158:0/3841873835 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0035020 con 0x7f7ae006b600 2024-03-06T07:21:41.952 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.950+0000 7f7addd9b640 1 -- 172.21.15.158:0/3841873835 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad0031870 con 0x7f7ae006b600 2024-03-06T07:21:41.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.950+0000 7f7addd9b640 1 -- 172.21.15.158:0/3841873835 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0031c40 con 0x7f7ae006b600 2024-03-06T07:21:41.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.950+0000 7f7ae5116640 1 -- 172.21.15.158:0/3841873835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 msgr2=0x7f7ae00b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.950+0000 7f7ae5116640 1 --2- 172.21.15.158:0/3841873835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 0x7f7ae00b58f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ae006fb00 tx=0x7f7ad002b910 comp rx=0 tx=0).stop 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.950+0000 7f7ae5116640 1 -- 172.21.15.158:0/3841873835 shutdown_connections 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.950+0000 7f7ae5116640 1 --2- 172.21.15.158:0/3841873835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 0x7f7ae00b58f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.951+0000 7f7ae5116640 1 --2- 172.21.15.158:0/3841873835 >> v2:172.21.15.77:3300/0 conn(0x7f7ae015ecb0 0x7f7ae0157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.951+0000 7f7ae5116640 1 -- 172.21.15.158:0/3841873835 >> 172.21.15.158:0/3841873835 conn(0x7f7ae006c0c0 msgr2=0x7f7ae006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.951+0000 7f7ae5116640 1 -- 172.21.15.158:0/3841873835 shutdown_connections 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.951+0000 7f7ae5116640 1 -- 172.21.15.158:0/3841873835 wait complete. 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.952+0000 7f7ae5116640 1 Processor -- start 2024-03-06T07:21:41.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.952+0000 7f7ae5116640 1 -- start start 2024-03-06T07:21:41.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ae5116640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae006b220 0x7f7ae014d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ae5116640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 0x7f7ae01e8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ae5116640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ae015ecb0 0x7f7ae00b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ae5116640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae015f2c0 con 0x7f7ae006b220 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ae5116640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae015f140 con 0x7f7ae006b600 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ae5116640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae0158390 con 0x7f7ae015ecb0 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ae015ecb0 0x7f7ae00b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ade59c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 0x7f7ae01e8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ae015ecb0 0x7f7ae00b29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.158:46744/0 (socket says 172.21.15.158:46744) 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 -- 172.21.15.158:0/4043712452 learned_addr learned my addr 172.21.15.158:0/4043712452 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7aded9d640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae006b220 0x7f7ae014d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 -- 172.21.15.158:0/4043712452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 msgr2=0x7f7ae01e8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 0x7f7ae01e8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 -- 172.21.15.158:0/4043712452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae006b220 msgr2=0x7f7ae014d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae006b220 0x7f7ae014d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 -- 172.21.15.158:0/4043712452 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae014f480 con 0x7f7ae015ecb0 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ade59c640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 0x7f7ae01e8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:41.957 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7adf59e640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ae015ecb0 0x7f7ae00b29b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800cc60 tx=0x7f7ad800a5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ac77fe640 1 -- 172.21.15.158:0/4043712452 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8003990 con 0x7f7ae015ecb0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.953+0000 7f7ac77fe640 1 -- 172.21.15.158:0/4043712452 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad8003b30 con 0x7f7ae015ecb0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.954+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae014f760 con 0x7f7ae015ecb0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.954+0000 7f7ac77fe640 1 -- 172.21.15.158:0/4043712452 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8004020 con 0x7f7ae015ecb0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.954+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae014fd20 con 0x7f7ae015ecb0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.954+0000 7f7ac77fe640 1 -- 172.21.15.158:0/4043712452 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7ad8004550 con 0x7f7ae015ecb0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.954+0000 7f7ac77fe640 1 -- 172.21.15.158:0/4043712452 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7ad8018de0 con 0x7f7ae015ecb0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.954+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae00733b0 con 0x7f7ae015ecb0 2024-03-06T07:21:41.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:41.957+0000 7f7ac77fe640 1 -- 172.21.15.158:0/4043712452 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7ad8014030 con 0x7f7ae015ecb0 2024-03-06T07:21:42.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.029+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2d18ca08-fdf5-44b6-9cc6-a264001ea4fa", "id": 0} v 0) v1 -- 0x7f7ae00722e0 con 0x7f7ae015ecb0 2024-03-06T07:21:42.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.032+0000 7f7ac77fe640 1 -- 172.21.15.158:0/4043712452 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2d18ca08-fdf5-44b6-9cc6-a264001ea4fa", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7ad8018400 con 0x7f7ae015ecb0 2024-03-06T07:21:42.033 INFO:teuthology.orchestra.run.smithi158.stdout:0 2024-03-06T07:21:42.034 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.032+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ae015ecb0 msgr2=0x7f7ae00b29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.034 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.032+0000 7f7ae5116640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ae015ecb0 0x7f7ae00b29b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800cc60 tx=0x7f7ad800a5c0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.033+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 shutdown_connections 2024-03-06T07:21:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.033+0000 7f7ae5116640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ae015ecb0 0x7f7ae00b29b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.033+0000 7f7ae5116640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ae006b600 0x7f7ae01e8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.033+0000 7f7ae5116640 1 --2- 172.21.15.158:0/4043712452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae006b220 0x7f7ae014d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.033+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 >> 172.21.15.158:0/4043712452 conn(0x7f7ae006c0c0 msgr2=0x7f7ae0156b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.033+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 shutdown_connections 2024-03-06T07:21:42.035 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.033+0000 7f7ae5116640 1 -- 172.21.15.158:0/4043712452 wait complete. 2024-03-06T07:21:42.045 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd new 22f22df8-eb96-4e3b-a851-2e51f91abc45 1 2024-03-06T07:21:42.146 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.144+0000 7f35d3840640 1 Processor -- start 2024-03-06T07:21:42.147 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.144+0000 7f35d3840640 1 -- start start 2024-03-06T07:21:42.150 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.145+0000 7f35d3840640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35cc15f2a0 0x7f35cc0b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.150 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.145+0000 7f35d3840640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f35cc072d00 0x7f35cc073170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.150 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.145+0000 7f35d3840640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc06ba60 con 0x7f35cc0d0a40 2024-03-06T07:21:42.150 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.145+0000 7f35d3840640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35cc072590 con 0x7f35cc15f2a0 2024-03-06T07:21:42.150 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.145+0000 7f35d3840640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35cc06bd00 con 0x7f35cc072d00 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.145+0000 7f35d0ddb640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f35cc0d0a40 0x7f35cc0b09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52426/0 (socket says 172.21.15.158:52426) 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.145+0000 7f35d0ddb640 1 -- 172.21.15.158:0/3632335876 learned_addr learned my addr 172.21.15.158:0/3632335876 (peer_addr_for_me v1:172.21.15.158:0/0) 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.145+0000 7f35d1ddd640 1 --2- 172.21.15.158:0/3632335876 >> v2:172.21.15.158:3300/0 conn(0x7f35cc072d00 0x7f35cc073170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.146+0000 7f35d15dc640 1 --2- 172.21.15.158:0/3632335876 >> v2:172.21.15.77:3300/0 conn(0x7f35cc15f2a0 0x7f35cc0b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.146+0000 7f35d1ddd640 1 -- 172.21.15.158:0/3632335876 >> v2:172.21.15.77:3300/0 conn(0x7f35cc15f2a0 msgr2=0x7f35cc0b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.146+0000 7f35d1ddd640 1 --2- 172.21.15.158:0/3632335876 >> v2:172.21.15.77:3300/0 conn(0x7f35cc15f2a0 0x7f35cc0b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.146+0000 7f35d1ddd640 1 -- 172.21.15.158:0/3632335876 >> v1:172.21.15.158:6789/0 conn(0x7f35cc0d0a40 legacy=0x7f35cc0b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.146+0000 7f35c3fff640 1 -- 172.21.15.158:0/3632335876 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102314850 0 0) 0x7f35cc06ba60 con 0x7f35cc0d0a40 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.146+0000 7f35d1ddd640 1 -- 172.21.15.158:0/3632335876 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc06bfe0 con 0x7f35cc072d00 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.147+0000 7f35d1ddd640 1 --2- 172.21.15.158:0/3632335876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc072d00 0x7f35cc073170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f35cc072710 tx=0x7f35c402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f0125a48c99b8bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.147+0000 7f35c3fff640 1 -- 172.21.15.158:0/3632335876 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4039070 con 0x7f35cc072d00 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.147+0000 7f35c3fff640 1 -- 172.21.15.158:0/3632335876 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c402cc60 con 0x7f35cc072d00 2024-03-06T07:21:42.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.147+0000 7f35c3fff640 1 -- 172.21.15.158:0/3632335876 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4034660 con 0x7f35cc072d00 2024-03-06T07:21:42.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.148+0000 7f35d3840640 1 -- 172.21.15.158:0/3632335876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc072d00 msgr2=0x7f35cc073170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.148+0000 7f35d3840640 1 --2- 172.21.15.158:0/3632335876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc072d00 0x7f35cc073170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f35cc072710 tx=0x7f35c402beb0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.148+0000 7f35d3840640 1 -- 172.21.15.158:0/3632335876 shutdown_connections 2024-03-06T07:21:42.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.148+0000 7f35d3840640 1 --2- 172.21.15.158:0/3632335876 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc072d00 0x7f35cc073170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.148+0000 7f35d3840640 1 --2- 172.21.15.158:0/3632335876 >> v2:172.21.15.77:3300/0 conn(0x7f35cc15f2a0 0x7f35cc0b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.148+0000 7f35d3840640 1 -- 172.21.15.158:0/3632335876 >> 172.21.15.158:0/3632335876 conn(0x7f35cc0b4fe0 msgr2=0x7f35cc0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.148+0000 7f35d3840640 1 -- 172.21.15.158:0/3632335876 shutdown_connections 2024-03-06T07:21:42.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.148+0000 7f35d3840640 1 -- 172.21.15.158:0/3632335876 wait complete. 2024-03-06T07:21:42.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.149+0000 7f35d3840640 1 Processor -- start 2024-03-06T07:21:42.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.149+0000 7f35d3840640 1 -- start start 2024-03-06T07:21:42.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d3840640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc072d00 0x7f35cc149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d3840640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc0d0a40 0x7f35cc1596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d3840640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc15f2a0 0x7f35cc0ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d3840640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc15f800 con 0x7f35cc072d00 2024-03-06T07:21:42.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d3840640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc15f680 con 0x7f35cc15f2a0 2024-03-06T07:21:42.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d3840640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0b10f0 con 0x7f35cc0d0a40 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d1ddd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc15f2a0 0x7f35cc0ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d0ddb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc0d0a40 0x7f35cc1596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d1ddd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc15f2a0 0x7f35cc0ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34406/0 (socket says 172.21.15.158:34406) 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d1ddd640 1 -- 172.21.15.158:0/38919069 learned_addr learned my addr 172.21.15.158:0/38919069 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d15dc640 1 --2- 172.21.15.158:0/38919069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc072d00 0x7f35cc149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d1ddd640 1 -- 172.21.15.158:0/38919069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc0d0a40 msgr2=0x7f35cc1596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d1ddd640 1 --2- 172.21.15.158:0/38919069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc0d0a40 0x7f35cc1596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d1ddd640 1 -- 172.21.15.158:0/38919069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc072d00 msgr2=0x7f35cc149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d1ddd640 1 --2- 172.21.15.158:0/38919069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc072d00 0x7f35cc149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.150+0000 7f35d1ddd640 1 -- 172.21.15.158:0/38919069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc153c30 con 0x7f35cc15f2a0 2024-03-06T07:21:42.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.151+0000 7f35d1ddd640 1 --2- 172.21.15.158:0/38919069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc15f2a0 0x7f35cc0ce5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f35c4009e50 tx=0x7f35c4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.151+0000 7f35c1ffb640 1 -- 172.21.15.158:0/38919069 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4039050 con 0x7f35cc15f2a0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.151+0000 7f35c1ffb640 1 -- 172.21.15.158:0/38919069 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c4045070 con 0x7f35cc15f2a0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.151+0000 7f35c1ffb640 1 -- 172.21.15.158:0/38919069 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4009450 con 0x7f35cc15f2a0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.151+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc153f10 con 0x7f35cc15f2a0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.151+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35cc149ad0 con 0x7f35cc15f2a0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.151+0000 7f35c1ffb640 1 -- 172.21.15.158:0/38919069 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f35c403e850 con 0x7f35cc15f2a0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.151+0000 7f35c1ffb640 1 -- 172.21.15.158:0/38919069 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f35c4040450 con 0x7f35cc15f2a0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.152+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc09e4e0 con 0x7f35cc15f2a0 2024-03-06T07:21:42.157 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.154+0000 7f35c1ffb640 1 -- 172.21.15.158:0/38919069 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f35c403ecd0 con 0x7f35cc15f2a0 2024-03-06T07:21:42.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.227+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd new", "uuid": "22f22df8-eb96-4e3b-a851-2e51f91abc45", "id": 1} v 0) v1 -- 0x7f35cc068ff0 con 0x7f35cc15f2a0 2024-03-06T07:21:42.231 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.230+0000 7f35c1ffb640 1 -- 172.21.15.158:0/38919069 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "22f22df8-eb96-4e3b-a851-2e51f91abc45", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f35c404a460 con 0x7f35cc15f2a0 2024-03-06T07:21:42.232 INFO:teuthology.orchestra.run.smithi158.stdout:1 2024-03-06T07:21:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc15f2a0 msgr2=0x7f35cc0ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 --2- 172.21.15.158:0/38919069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc15f2a0 0x7f35cc0ce5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f35c4009e50 tx=0x7f35c4004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 shutdown_connections 2024-03-06T07:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 --2- 172.21.15.158:0/38919069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc15f2a0 0x7f35cc0ce5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 --2- 172.21.15.158:0/38919069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc0d0a40 0x7f35cc1596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 --2- 172.21.15.158:0/38919069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc072d00 0x7f35cc149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 >> 172.21.15.158:0/38919069 conn(0x7f35cc0b4fe0 msgr2=0x7f35cc15e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 shutdown_connections 2024-03-06T07:21:42.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.231+0000 7f35d3840640 1 -- 172.21.15.158:0/38919069 wait complete. 2024-03-06T07:21:42.243 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd new 069ab0e2-cfb9-4f92-a8af-e473269ccb97 2 2024-03-06T07:21:42.315 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.313+0000 7efdbe00d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:42.334 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.332+0000 7fc777a13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:42.345 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.343+0000 7fe6e35f6640 1 Processor -- start 2024-03-06T07:21:42.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.343+0000 7fe6e35f6640 1 -- start start 2024-03-06T07:21:42.353 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.344+0000 7fe6e35f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.353 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.344+0000 7fe6e35f6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6dc06cd20 con 0x7fe6dc1620e0 2024-03-06T07:21:42.353 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.344+0000 7fe6e35f6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6dc06d000 con 0x7fe6dc06c860 2024-03-06T07:21:42.353 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.344+0000 7fe6e35f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc06d2e0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.353 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.344+0000 7fe6e1392640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.353 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.344+0000 7fe6e0b91640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fe6dc06c860 0x7fe6dc1619d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52430/0 (socket says 172.21.15.158:52430) 2024-03-06T07:21:42.353 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.344+0000 7fe6e0b91640 1 -- 172.21.15.158:0/4162409653 learned_addr learned my addr 172.21.15.158:0/4162409653 (peer_addr_for_me v1:172.21.15.158:0/0) 2024-03-06T07:21:42.353 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.345+0000 7fe6e1392640 1 -- 172.21.15.158:0/4162409653 >> v1:172.21.15.158:6789/0 conn(0x7fe6dc06c860 legacy=0x7fe6dc1619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.345+0000 7fe6d3fff640 1 -- 172.21.15.158:0/4162409653 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199929819 0 0) 0x7fe6dc06d000 con 0x7fe6dc06c860 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.345+0000 7fe6e1392640 1 -- 172.21.15.158:0/4162409653 >> v1:172.21.15.77:6789/0 conn(0x7fe6dc1620e0 legacy=0x7fe6dc15f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.345+0000 7fe6e1392640 1 -- 172.21.15.158:0/4162409653 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc06d5c0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.345+0000 7fe6e1392640 1 --2- 172.21.15.158:0/4162409653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0c77b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0097d0 tx=0x7fe6cc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=303a25efb2393a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6d3fff640 1 -- 172.21.15.158:0/4162409653 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc035020 con 0x7fe6dc1614f0 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6d3fff640 1 -- 172.21.15.158:0/4162409653 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc031800 con 0x7fe6dc1614f0 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6d3fff640 1 -- 172.21.15.158:0/4162409653 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc031bd0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4162409653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 msgr2=0x7fe6dc0c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6e35f6640 1 --2- 172.21.15.158:0/4162409653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0c77b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0097d0 tx=0x7fe6cc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4162409653 shutdown_connections 2024-03-06T07:21:42.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6e35f6640 1 --2- 172.21.15.158:0/4162409653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0c77b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4162409653 >> 172.21.15.158:0/4162409653 conn(0x7fe6dc0d1b40 msgr2=0x7fe6dc0d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4162409653 shutdown_connections 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.346+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4162409653 wait complete. 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.347+0000 7fe6e35f6640 1 Processor -- start 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.347+0000 7fe6e35f6640 1 -- start start 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e35f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6dc06c860 0x7fe6dc1f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e35f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e35f6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc1620e0 0x7fe6dc0cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e35f6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0c7cf0 con 0x7fe6dc06c860 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e35f6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0d28f0 con 0x7fe6dc1620e0 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e35f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc1f3e50 con 0x7fe6dc1614f0 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e0b91640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e1b93640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc1620e0 0x7fe6dc0cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e0b91640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0cf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.158:46776/0 (socket says 172.21.15.158:46776) 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e1b93640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc1620e0 0x7fe6dc0cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34416/0 (socket says 172.21.15.158:34416) 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e0b91640 1 -- 172.21.15.158:0/4070593025 learned_addr learned my addr 172.21.15.158:0/4070593025 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.348+0000 7fe6e1392640 1 --2- 172.21.15.158:0/4070593025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6dc06c860 0x7fe6dc1f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6e0b91640 1 -- 172.21.15.158:0/4070593025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc1620e0 msgr2=0x7fe6dc0cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6e0b91640 1 --2- 172.21.15.158:0/4070593025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc1620e0 0x7fe6dc0cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6e0b91640 1 -- 172.21.15.158:0/4070593025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6dc06c860 msgr2=0x7fe6dc1f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6e0b91640 1 --2- 172.21.15.158:0/4070593025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6dc06c860 0x7fe6dc1f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6e0b91640 1 -- 172.21.15.158:0/4070593025 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc14baa0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6e0b91640 1 --2- 172.21.15.158:0/4070593025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0cf8e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800c3f0 tx=0x7fe6d800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6d1ffb640 1 -- 172.21.15.158:0/4070593025 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80074e0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6d1ffb640 1 -- 172.21.15.158:0/4070593025 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d80047e0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6dc14bd80 con 0x7fe6dc1614f0 2024-03-06T07:21:42.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6d1ffb640 1 -- 172.21.15.158:0/4070593025 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8004c10 con 0x7fe6dc1614f0 2024-03-06T07:21:42.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6dc14c340 con 0x7fe6dc1614f0 2024-03-06T07:21:42.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6d1ffb640 1 -- 172.21.15.158:0/4070593025 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe6d8011070 con 0x7fe6dc1614f0 2024-03-06T07:21:42.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.349+0000 7fe6d1ffb640 1 -- 172.21.15.158:0/4070593025 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe6d80137e0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.350+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6dc09fdf0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.352+0000 7fe6d1ffb640 1 -- 172.21.15.158:0/4070593025 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe6d8004db0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.369 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.367+0000 7efdbe00d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:42.391 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.389+0000 7fc777a13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:42.413 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.411+0000 7efdbe00d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:42.427 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.425+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd new", "uuid": "069ab0e2-cfb9-4f92-a8af-e473269ccb97", "id": 2} v 0) v1 -- 0x7fe6dc068ff0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.430+0000 7fe6d1ffb640 1 -- 172.21.15.158:0/4070593025 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "069ab0e2-cfb9-4f92-a8af-e473269ccb97", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe6d80134d0 con 0x7fe6dc1614f0 2024-03-06T07:21:42.432 INFO:teuthology.orchestra.run.smithi158.stdout:2 2024-03-06T07:21:42.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 msgr2=0x7fe6dc0cf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 --2- 172.21.15.158:0/4070593025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0cf8e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800c3f0 tx=0x7fe6d800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 shutdown_connections 2024-03-06T07:21:42.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 --2- 172.21.15.158:0/4070593025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6dc1620e0 0x7fe6dc0cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 --2- 172.21.15.158:0/4070593025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6dc1614f0 0x7fe6dc0cf8e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 --2- 172.21.15.158:0/4070593025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6dc06c860 0x7fe6dc1f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 >> 172.21.15.158:0/4070593025 conn(0x7fe6dc0d1b40 msgr2=0x7fe6dc0c6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 shutdown_connections 2024-03-06T07:21:42.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.431+0000 7fe6e35f6640 1 -- 172.21.15.158:0/4070593025 wait complete. 2024-03-06T07:21:42.438 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.436+0000 7fc777a13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:42.443 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd new 11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20 3 2024-03-06T07:21:42.466 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.464+0000 7efdbe00d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:42.495 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.493+0000 7fc777a13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:42.508 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.506+0000 7efdbe00d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:42.539 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.538+0000 7fc777a13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.543+0000 7f871a363640 1 Processor -- start 2024-03-06T07:21:42.544 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.543+0000 7f871a363640 1 -- start start 2024-03-06T07:21:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.543+0000 7f871a363640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8714071a70 0x7f8714071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.543+0000 7f871a363640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87140b42e0 0x7f871415c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.543+0000 7f871a363640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8714072390 con 0x7f871415ad90 2024-03-06T07:21:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.543+0000 7f871a363640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871415c020 con 0x7f8714071a70 2024-03-06T07:21:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.543+0000 7f871a363640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87140b4720 con 0x7f87140b42e0 2024-03-06T07:21:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.544+0000 7f8718900640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87140b42e0 0x7f871415c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.544+0000 7f8718900640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87140b42e0 0x7f871415c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.158:46788/0 (socket says 172.21.15.158:46788) 2024-03-06T07:21:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.544+0000 7f8718900640 1 -- 172.21.15.158:0/2985768062 learned_addr learned my addr 172.21.15.158:0/2985768062 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.544+0000 7f8713fff640 1 --2- 172.21.15.158:0/2985768062 >> v2:172.21.15.77:3300/0 conn(0x7f8714071a70 0x7f8714071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.544+0000 7f8718900640 1 -- 172.21.15.158:0/2985768062 >> v2:172.21.15.77:3300/0 conn(0x7f8714071a70 msgr2=0x7f8714071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.544+0000 7f8718900640 1 --2- 172.21.15.158:0/2985768062 >> v2:172.21.15.77:3300/0 conn(0x7f8714071a70 0x7f8714071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.544+0000 7f8718900640 1 -- 172.21.15.158:0/2985768062 >> v1:172.21.15.77:6789/0 conn(0x7f871415ad90 legacy=0x7f87140b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.544+0000 7f8718900640 1 -- 172.21.15.158:0/2985768062 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871415b260 con 0x7f87140b42e0 2024-03-06T07:21:42.548 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.545+0000 7f8718900640 1 --2- 172.21.15.158:0/2985768062 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87140b42e0 0x7f871415c210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f870c00d150 tx=0x7f870c02e0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46ccafbb3518cc46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.548 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.545+0000 7f8712ffd640 1 -- 172.21.15.158:0/2985768062 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c02ea80 con 0x7f87140b42e0 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.545+0000 7f8712ffd640 1 -- 172.21.15.158:0/2985768062 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c02ec20 con 0x7f87140b42e0 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.545+0000 7f8712ffd640 1 -- 172.21.15.158:0/2985768062 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c009600 con 0x7f87140b42e0 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.546+0000 7f871a363640 1 -- 172.21.15.158:0/2985768062 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87140b42e0 msgr2=0x7f871415c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.546+0000 7f871a363640 1 --2- 172.21.15.158:0/2985768062 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87140b42e0 0x7f871415c210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f870c00d150 tx=0x7f870c02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.546+0000 7f871a363640 1 -- 172.21.15.158:0/2985768062 shutdown_connections 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.546+0000 7f871a363640 1 --2- 172.21.15.158:0/2985768062 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87140b42e0 0x7f871415c210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.546+0000 7f871a363640 1 --2- 172.21.15.158:0/2985768062 >> v2:172.21.15.77:3300/0 conn(0x7f8714071a70 0x7f8714071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.546+0000 7f871a363640 1 -- 172.21.15.158:0/2985768062 >> 172.21.15.158:0/2985768062 conn(0x7f87140b2660 msgr2=0x7f87140b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.546+0000 7f871a363640 1 -- 172.21.15.158:0/2985768062 shutdown_connections 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.546+0000 7f871a363640 1 -- 172.21.15.158:0/2985768062 wait complete. 2024-03-06T07:21:42.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.547+0000 7f871a363640 1 Processor -- start 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.547+0000 7f871a363640 1 -- start start 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f871a363640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8714071a70 0x7f8714157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f871a363640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87140b42e0 0x7f8714151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f871a363640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f871415ad90 0x7f8714146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f871a363640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871415bb00 con 0x7f87140b42e0 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f871a363640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871415b820 con 0x7f8714071a70 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f871a363640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871415c750 con 0x7f871415ad90 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8713fff640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8714071a70 0x7f8714157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8718900640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f871415ad90 0x7f8714146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8713fff640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8714071a70 0x7f8714157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34422/0 (socket says 172.21.15.158:34422) 2024-03-06T07:21:42.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8713fff640 1 -- 172.21.15.158:0/4195147992 learned_addr learned my addr 172.21.15.158:0/4195147992 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f87137fe640 1 --2- 172.21.15.158:0/4195147992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87140b42e0 0x7f8714151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8713fff640 1 -- 172.21.15.158:0/4195147992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f871415ad90 msgr2=0x7f8714146b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8713fff640 1 --2- 172.21.15.158:0/4195147992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f871415ad90 0x7f8714146b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8713fff640 1 -- 172.21.15.158:0/4195147992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87140b42e0 msgr2=0x7f8714151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8713fff640 1 --2- 172.21.15.158:0/4195147992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87140b42e0 0x7f8714151150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.548+0000 7f8713fff640 1 -- 172.21.15.158:0/4195147992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87141568d0 con 0x7f8714071a70 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.549+0000 7f8713fff640 1 --2- 172.21.15.158:0/4195147992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8714071a70 0x7f8714157970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f870400e9b0 tx=0x7f870400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.549+0000 7f8710ff9640 1 -- 172.21.15.158:0/4195147992 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704017070 con 0x7f8714071a70 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.549+0000 7f8710ff9640 1 -- 172.21.15.158:0/4195147992 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870400c9a0 con 0x7f8714071a70 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.549+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8714156bb0 con 0x7f8714071a70 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.549+0000 7f8710ff9640 1 -- 172.21.15.158:0/4195147992 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870400cd90 con 0x7f8714071a70 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.549+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8714157170 con 0x7f8714071a70 2024-03-06T07:21:42.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.549+0000 7f8710ff9640 1 -- 172.21.15.158:0/4195147992 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f870401a070 con 0x7f8714071a70 2024-03-06T07:21:42.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.549+0000 7f8710ff9640 1 -- 172.21.15.158:0/4195147992 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f870401c9d0 con 0x7f8714071a70 2024-03-06T07:21:42.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.550+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8714073390 con 0x7f8714071a70 2024-03-06T07:21:42.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.552+0000 7f8710ff9640 1 -- 172.21.15.158:0/4195147992 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f870401cd30 con 0x7f8714071a70 2024-03-06T07:21:42.621 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.620+0000 7efdbe00d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:42.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.624+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd new", "uuid": "11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20", "id": 3} v 0) v1 -- 0x7f87140cfcf0 con 0x7f8714071a70 2024-03-06T07:21:42.632 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.630+0000 7f8710ff9640 1 -- 172.21.15.158:0/4195147992 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f870401c610 con 0x7f8714071a70 2024-03-06T07:21:42.632 INFO:teuthology.orchestra.run.smithi158.stdout:3 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8714071a70 msgr2=0x7f8714157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 --2- 172.21.15.158:0/4195147992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8714071a70 0x7f8714157970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f870400e9b0 tx=0x7f870400ee70 comp rx=0 tx=0).stop 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 shutdown_connections 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 --2- 172.21.15.158:0/4195147992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87140b42e0 0x7f8714151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 --2- 172.21.15.158:0/4195147992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f871415ad90 0x7f8714146b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 --2- 172.21.15.158:0/4195147992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8714071a70 0x7f8714157970 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 >> 172.21.15.158:0/4195147992 conn(0x7f87140b2660 msgr2=0x7f87140b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 shutdown_connections 2024-03-06T07:21:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.631+0000 7f871a363640 1 -- 172.21.15.158:0/4195147992 wait complete. 2024-03-06T07:21:42.643 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd new b25fa93b-6204-41c7-8853-cd24c0c8f832 4 2024-03-06T07:21:42.663 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.661+0000 7fc777a13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:42.691 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.689+0000 7efdbe00d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:42.736 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.734+0000 7efdbe00d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:42.736 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.734+0000 7fc777a13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:42.744 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.742+0000 7fb6fb75c640 1 Processor -- start 2024-03-06T07:21:42.744 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6fb75c640 1 -- start start 2024-03-06T07:21:42.746 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6fb75c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f415e680 0x7fb6f40d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.746 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6fb75c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6f406f2c0 0x7fb6f40d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.746 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6fb75c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f415eb10 con 0x7fb6f4144560 2024-03-06T07:21:42.746 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6fb75c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f4144940 con 0x7fb6f406f2c0 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6fb75c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f415edf0 con 0x7fb6f415e680 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6f9cf9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6f406f2c0 0x7fb6f40d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6f8cf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f415e680 0x7fb6f40d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6f9cf9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6f406f2c0 0x7fb6f40d0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34430/0 (socket says 172.21.15.158:34430) 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.743+0000 7fb6f8cf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f415e680 0x7fb6f40d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.158:46804/0 (socket says 172.21.15.158:46804) 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.744+0000 7fb6f9cf9640 1 -- 172.21.15.158:0/1598629730 learned_addr learned my addr 172.21.15.158:0/1598629730 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.744+0000 7fb6f8cf7640 1 -- 172.21.15.158:0/1598629730 >> v2:172.21.15.158:3300/0 conn(0x7fb6f406f2c0 msgr2=0x7fb6f40d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.744+0000 7fb6f8cf7640 1 --2- 172.21.15.158:0/1598629730 >> v2:172.21.15.158:3300/0 conn(0x7fb6f406f2c0 0x7fb6f40d0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.744+0000 7fb6f8cf7640 1 -- 172.21.15.158:0/1598629730 >> v1:172.21.15.158:6789/0 conn(0x7fb6f4144560 legacy=0x7fb6f40cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.744+0000 7fb6f8cf7640 1 -- 172.21.15.158:0/1598629730 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f415f0d0 con 0x7fb6f415e680 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.744+0000 7fb6f9cf9640 1 --2- 172.21.15.158:0/1598629730 >> v2:172.21.15.158:3300/0 conn(0x7fb6f406f2c0 0x7fb6f40d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6f8cf7640 1 --2- 172.21.15.158:0/1598629730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f415e680 0x7fb6f40d0840 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400dbc0 tx=0x7fb6e402fcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddb83d06de7facf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6ebfff640 1 -- 172.21.15.158:0/1598629730 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4030860 con 0x7fb6f415e680 2024-03-06T07:21:42.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6ebfff640 1 -- 172.21.15.158:0/1598629730 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e4030a00 con 0x7fb6f415e680 2024-03-06T07:21:42.748 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6ebfff640 1 -- 172.21.15.158:0/1598629730 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4030dd0 con 0x7fb6f415e680 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6fb75c640 1 -- 172.21.15.158:0/1598629730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f415e680 msgr2=0x7fb6f40d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6fb75c640 1 --2- 172.21.15.158:0/1598629730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f415e680 0x7fb6f40d0840 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400dbc0 tx=0x7fb6e402fcc0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6fb75c640 1 -- 172.21.15.158:0/1598629730 shutdown_connections 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6fb75c640 1 --2- 172.21.15.158:0/1598629730 >> v2:172.21.15.158:3300/0 conn(0x7fb6f406f2c0 0x7fb6f40d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6fb75c640 1 --2- 172.21.15.158:0/1598629730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f415e680 0x7fb6f40d0840 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6fb75c640 1 -- 172.21.15.158:0/1598629730 >> 172.21.15.158:0/1598629730 conn(0x7fb6f406c0c0 msgr2=0x7fb6f406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6fb75c640 1 -- 172.21.15.158:0/1598629730 shutdown_connections 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.745+0000 7fb6fb75c640 1 -- 172.21.15.158:0/1598629730 wait complete. 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6fb75c640 1 Processor -- start 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6fb75c640 1 -- start start 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6fb75c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f406f2c0 0x7fb6f40b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6fb75c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f4144560 0x7fb6f41f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6fb75c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6f415c840 0x7fb6f415d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6fb75c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f406e280 con 0x7fb6f4144560 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6fb75c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f406e100 con 0x7fb6f415c840 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6fb75c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f406e400 con 0x7fb6f406f2c0 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6f94f8640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f406f2c0 0x7fb6f40b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6f9cf9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6f415c840 0x7fb6f415d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6f94f8640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f406f2c0 0x7fb6f40b17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.158:46814/0 (socket says 172.21.15.158:46814) 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.747+0000 7fb6f94f8640 1 -- 172.21.15.158:0/3545829985 learned_addr learned my addr 172.21.15.158:0/3545829985 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f8cf7640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f4144560 0x7fb6f41f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f94f8640 1 -- 172.21.15.158:0/3545829985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6f415c840 msgr2=0x7fb6f415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f94f8640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6f415c840 0x7fb6f415d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f94f8640 1 -- 172.21.15.158:0/3545829985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f4144560 msgr2=0x7fb6f41f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f94f8640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f4144560 0x7fb6f41f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.752 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f94f8640 1 -- 172.21.15.158:0/3545829985 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f40b1ed0 con 0x7fb6f406f2c0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f9cf9640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6f415c840 0x7fb6f415d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f8cf7640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f4144560 0x7fb6f41f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6f94f8640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f406f2c0 0x7fb6f40b17f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c400 tx=0x7fb6f000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6e9ffb640 1 -- 172.21.15.158:0/3545829985 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f00052d0 con 0x7fb6f406f2c0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6e9ffb640 1 -- 172.21.15.158:0/3545829985 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f0005490 con 0x7fb6f406f2c0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6e9ffb640 1 -- 172.21.15.158:0/3545829985 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0007260 con 0x7fb6f406f2c0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f40b21b0 con 0x7fb6f406f2c0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6e9ffb640 1 -- 172.21.15.158:0/3545829985 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb6f0007400 con 0x7fb6f406f2c0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.748+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f40b2770 con 0x7fb6f406f2c0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.749+0000 7fb6e9ffb640 1 -- 172.21.15.158:0/3545829985 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fb6f0003640 con 0x7fb6f406f2c0 2024-03-06T07:21:42.753 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.749+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f40733b0 con 0x7fb6f406f2c0 2024-03-06T07:21:42.754 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.751+0000 7fb6e9ffb640 1 -- 172.21.15.158:0/3545829985 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb6f0021020 con 0x7fb6f406f2c0 2024-03-06T07:21:42.785 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.783+0000 7fc777a13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:42.790 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.786+0000 7efdbe00d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:42.825 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.824+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b25fa93b-6204-41c7-8853-cd24c0c8f832", "id": 4} v 0) v1 -- 0x7fb6f4068ff0 con 0x7fb6f406f2c0 2024-03-06T07:21:42.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.829+0000 7fb6e9ffb640 1 -- 172.21.15.158:0/3545829985 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b25fa93b-6204-41c7-8853-cd24c0c8f832", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb6f0003aa0 con 0x7fb6f406f2c0 2024-03-06T07:21:42.831 INFO:teuthology.orchestra.run.smithi158.stdout:4 2024-03-06T07:21:42.832 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.830+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f406f2c0 msgr2=0x7fb6f40b17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.832 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.830+0000 7fb6fb75c640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f406f2c0 0x7fb6f40b17f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb6f000c400 tx=0x7fb6f000c800 comp rx=0 tx=0).stop 2024-03-06T07:21:42.832 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.830+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 shutdown_connections 2024-03-06T07:21:42.833 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.830+0000 7fb6fb75c640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f4144560 0x7fb6f41f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.833 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.830+0000 7fb6fb75c640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6f415c840 0x7fb6f415d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.833 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.830+0000 7fb6fb75c640 1 --2- 172.21.15.158:0/3545829985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6f406f2c0 0x7fb6f40b17f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.833 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.830+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 >> 172.21.15.158:0/3545829985 conn(0x7fb6f406c0c0 msgr2=0x7fb6f40cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.833 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.830+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 shutdown_connections 2024-03-06T07:21:42.833 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.831+0000 7fb6fb75c640 1 -- 172.21.15.158:0/3545829985 wait complete. 2024-03-06T07:21:42.834 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:42.833+0000 7efdbe00d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:42.840 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.839+0000 7fc777a13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:42.842 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd new 9e451073-42e6-4695-a7e1-d5dc0b6108d2 5 2024-03-06T07:21:42.889 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:42.887+0000 7fc777a13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:42.985 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.983+0000 7f34a9a4e640 1 Processor -- start 2024-03-06T07:21:42.985 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.983+0000 7f34a9a4e640 1 -- start start 2024-03-06T07:21:42.986 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.984+0000 7f34a9a4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f34a4143bf0 0x7f34a4143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.984+0000 7f34a9a4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.984+0000 7f34a9a4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a406e7b0 con 0x7f34a40b5b50 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.984+0000 7f34a9a4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a40b4450 con 0x7f34a4143bf0 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.984+0000 7f34a9a4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a406ea50 con 0x7f34a406edf0 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.984+0000 7f34a2ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f34a4143bf0 0x7f34a4143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.985+0000 7f34a37fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.985+0000 7f34a2ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f34a4143bf0 0x7f34a4143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34448/0 (socket says 172.21.15.158:34448) 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.985+0000 7f34a2ffd640 1 -- 172.21.15.158:0/3943301741 learned_addr learned my addr 172.21.15.158:0/3943301741 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.985+0000 7f34a37fe640 1 -- 172.21.15.158:0/3943301741 >> v2:172.21.15.158:3300/0 conn(0x7f34a4143bf0 msgr2=0x7f34a4143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.985+0000 7f34a37fe640 1 --2- 172.21.15.158:0/3943301741 >> v2:172.21.15.158:3300/0 conn(0x7f34a4143bf0 0x7f34a4143fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.985+0000 7f34a37fe640 1 -- 172.21.15.158:0/3943301741 >> v1:172.21.15.158:6789/0 conn(0x7f34a40b5b50 legacy=0x7f34a40b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.985+0000 7f34a37fe640 1 -- 172.21.15.158:0/3943301741 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a40b46a0 con 0x7f34a406edf0 2024-03-06T07:21:42.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.985+0000 7f34a2ffd640 1 --2- 172.21.15.158:0/3943301741 >> v2:172.21.15.158:3300/0 conn(0x7f34a4143bf0 0x7f34a4143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.986+0000 7f34a37fe640 1 --2- 172.21.15.158:0/3943301741 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406f270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f34a40b45d0 tx=0x7f349c02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e79a103ec8de7a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.986+0000 7f34a27fc640 1 -- 172.21.15.158:0/3943301741 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c039070 con 0x7f34a406edf0 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.986+0000 7f34a27fc640 1 -- 172.21.15.158:0/3943301741 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c02cbb0 con 0x7f34a406edf0 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.986+0000 7f34a27fc640 1 -- 172.21.15.158:0/3943301741 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c034440 con 0x7f34a406edf0 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.987+0000 7f34a9a4e640 1 -- 172.21.15.158:0/3943301741 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 msgr2=0x7f34a406f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.987+0000 7f34a9a4e640 1 --2- 172.21.15.158:0/3943301741 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406f270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f34a40b45d0 tx=0x7f349c02be00 comp rx=0 tx=0).stop 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.987+0000 7f34a9a4e640 1 -- 172.21.15.158:0/3943301741 shutdown_connections 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.987+0000 7f34a9a4e640 1 --2- 172.21.15.158:0/3943301741 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406f270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.987+0000 7f34a9a4e640 1 --2- 172.21.15.158:0/3943301741 >> v2:172.21.15.158:3300/0 conn(0x7f34a4143bf0 0x7f34a4143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.987+0000 7f34a9a4e640 1 -- 172.21.15.158:0/3943301741 >> 172.21.15.158:0/3943301741 conn(0x7f34a40d0a30 msgr2=0x7f34a40d0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.989 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.987+0000 7f34a9a4e640 1 -- 172.21.15.158:0/3943301741 shutdown_connections 2024-03-06T07:21:42.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.987+0000 7f34a9a4e640 1 -- 172.21.15.158:0/3943301741 wait complete. 2024-03-06T07:21:42.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.989+0000 7f34a9a4e640 1 Processor -- start 2024-03-06T07:21:42.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.989+0000 7f34a9a4e640 1 -- start start 2024-03-06T07:21:42.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a9a4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a9a4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34a40b5b50 0x7f34a406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a9a4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34a4143bf0 0x7f34a41523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a9a4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4144910 con 0x7f34a4143bf0 2024-03-06T07:21:42.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a9a4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4144790 con 0x7f34a40b5b50 2024-03-06T07:21:42.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a9a4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40cde20 con 0x7f34a406edf0 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.158:46824/0 (socket says 172.21.15.158:46824) 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 -- 172.21.15.158:0/544869219 learned_addr learned my addr 172.21.15.158:0/544869219 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f349bfff640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34a40b5b50 0x7f34a406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a37fe640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34a4143bf0 0x7f34a41523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 -- 172.21.15.158:0/544869219 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34a40b5b50 msgr2=0x7f34a406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34a40b5b50 0x7f34a406b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 -- 172.21.15.158:0/544869219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34a4143bf0 msgr2=0x7f34a41523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34a4143bf0 0x7f34a41523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 -- 172.21.15.158:0/544869219 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a41531a0 con 0x7f34a406edf0 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a37fe640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34a4143bf0 0x7f34a41523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.990+0000 7f34a2ffd640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406afb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f349400e9e0 tx=0x7f349400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.991+0000 7f34a0ff9640 1 -- 172.21.15.158:0/544869219 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3494017070 con 0x7f34a406edf0 2024-03-06T07:21:42.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.991+0000 7f34a0ff9640 1 -- 172.21.15.158:0/544869219 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349400c9d0 con 0x7f34a406edf0 2024-03-06T07:21:42.995 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.991+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a4153480 con 0x7f34a406edf0 2024-03-06T07:21:42.995 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.991+0000 7f34a0ff9640 1 -- 172.21.15.158:0/544869219 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349400cd80 con 0x7f34a406edf0 2024-03-06T07:21:42.995 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.991+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a4153a40 con 0x7f34a406edf0 2024-03-06T07:21:42.995 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.991+0000 7f34a0ff9640 1 -- 172.21.15.158:0/544869219 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f349401a070 con 0x7f34a406edf0 2024-03-06T07:21:42.995 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.991+0000 7f34a0ff9640 1 -- 172.21.15.158:0/544869219 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f349401c9d0 con 0x7f34a406edf0 2024-03-06T07:21:42.995 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.992+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a40734b0 con 0x7f34a406edf0 2024-03-06T07:21:42.995 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:42.993+0000 7f34a0ff9640 1 -- 172.21.15.158:0/544869219 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f349401c450 con 0x7f34a406edf0 2024-03-06T07:21:43.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.066+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9e451073-42e6-4695-a7e1-d5dc0b6108d2", "id": 5} v 0) v1 -- 0x7f34a40d1340 con 0x7f34a406edf0 2024-03-06T07:21:43.072 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.071+0000 7f34a0ff9640 1 -- 172.21.15.158:0/544869219 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e451073-42e6-4695-a7e1-d5dc0b6108d2", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f349401c700 con 0x7f34a406edf0 2024-03-06T07:21:43.073 INFO:teuthology.orchestra.run.smithi158.stdout:5 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 msgr2=0x7f34a406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406afb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f349400e9e0 tx=0x7f349400eea0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 shutdown_connections 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34a4143bf0 0x7f34a41523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34a40b5b50 0x7f34a406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 --2- 172.21.15.158:0/544869219 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34a406edf0 0x7f34a406afb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 >> 172.21.15.158:0/544869219 conn(0x7f34a40d0a30 msgr2=0x7f34a4142ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 shutdown_connections 2024-03-06T07:21:43.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.072+0000 7f34a9a4e640 1 -- 172.21.15.158:0/544869219 wait complete. 2024-03-06T07:21:43.084 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd new e460d8c9-90e8-400a-8934-b5654984cebc 6 2024-03-06T07:21:43.208 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:43.206+0000 7efdbe00d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:43.226 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.224+0000 7f332c3f4640 1 Processor -- start 2024-03-06T07:21:43.226 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.224+0000 7f332c3f4640 1 -- start start 2024-03-06T07:21:43.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332c3f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f332406afb0 0x7f332406b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332c3f4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33240ceda0 0x7f33240cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332c3f4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332406b8d0 con 0x7f33240cd250 2024-03-06T07:21:43.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332c3f4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33240725a0 con 0x7f332406afb0 2024-03-06T07:21:43.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332c3f4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3324072db0 con 0x7f33240ceda0 2024-03-06T07:21:43.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332a991640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33240ceda0 0x7f33240cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332a991640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33240ceda0 0x7f33240cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34460/0 (socket says 172.21.15.158:34460) 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332a991640 1 -- 172.21.15.158:0/3071409354 learned_addr learned my addr 172.21.15.158:0/3071409354 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.225+0000 7f332a190640 1 --2- 172.21.15.158:0/3071409354 >> v2:172.21.15.77:3300/0 conn(0x7f332406afb0 0x7f332406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.226+0000 7f332a991640 1 -- 172.21.15.158:0/3071409354 >> v2:172.21.15.77:3300/0 conn(0x7f332406afb0 msgr2=0x7f332406b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.226+0000 7f332a991640 1 --2- 172.21.15.158:0/3071409354 >> v2:172.21.15.77:3300/0 conn(0x7f332406afb0 0x7f332406b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.226+0000 7f332a991640 1 -- 172.21.15.158:0/3071409354 >> v1:172.21.15.158:6789/0 conn(0x7f33240cd250 legacy=0x7f33240d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.226+0000 7f332a991640 1 -- 172.21.15.158:0/3071409354 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33240727f0 con 0x7f33240ceda0 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.226+0000 7f332a991640 1 --2- 172.21.15.158:0/3071409354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33240ceda0 0x7f33240cf230 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f33240718e0 tx=0x7f331c02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccdf4461f88b4cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332918e640 1 -- 172.21.15.158:0/3071409354 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c02ca80 con 0x7f33240ceda0 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332918e640 1 -- 172.21.15.158:0/3071409354 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c02cc20 con 0x7f33240ceda0 2024-03-06T07:21:43.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332918e640 1 -- 172.21.15.158:0/3071409354 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c0343f0 con 0x7f33240ceda0 2024-03-06T07:21:43.232 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332c3f4640 1 -- 172.21.15.158:0/3071409354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33240ceda0 msgr2=0x7f33240cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332c3f4640 1 --2- 172.21.15.158:0/3071409354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33240ceda0 0x7f33240cf230 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f33240718e0 tx=0x7f331c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332c3f4640 1 -- 172.21.15.158:0/3071409354 shutdown_connections 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332c3f4640 1 --2- 172.21.15.158:0/3071409354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33240ceda0 0x7f33240cf230 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332c3f4640 1 --2- 172.21.15.158:0/3071409354 >> v2:172.21.15.77:3300/0 conn(0x7f332406afb0 0x7f332406b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332c3f4640 1 -- 172.21.15.158:0/3071409354 >> 172.21.15.158:0/3071409354 conn(0x7f332406c0c0 msgr2=0x7f332406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332c3f4640 1 -- 172.21.15.158:0/3071409354 shutdown_connections 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.227+0000 7f332c3f4640 1 -- 172.21.15.158:0/3071409354 wait complete. 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.228+0000 7f332c3f4640 1 Processor -- start 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.228+0000 7f332c3f4640 1 -- start start 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332c3f4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f332406afb0 0x7f3324157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332c3f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33240cd250 0x7f332415d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332c3f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33240ceda0 0x7f332415d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332c3f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33240d0990 con 0x7f33240ceda0 2024-03-06T07:21:43.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332c3f4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33240d06b0 con 0x7f332406afb0 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332c3f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3324145880 con 0x7f33240cd250 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332a190640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f332406afb0 0x7f3324157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332a190640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f332406afb0 0x7f3324157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34464/0 (socket says 172.21.15.158:34464) 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332a190640 1 -- 172.21.15.158:0/23189007 learned_addr learned my addr 172.21.15.158:0/23189007 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332998f640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33240cd250 0x7f332415d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.229+0000 7f332a991640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33240ceda0 0x7f332415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f332a190640 1 -- 172.21.15.158:0/23189007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33240cd250 msgr2=0x7f332415d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f332a190640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33240cd250 0x7f332415d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f332a190640 1 -- 172.21.15.158:0/23189007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33240ceda0 msgr2=0x7f332415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f332a190640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33240ceda0 0x7f332415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f332a190640 1 -- 172.21.15.158:0/23189007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3324143be0 con 0x7f332406afb0 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f332a190640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f332406afb0 0x7f3324157570 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f331400e8a0 tx=0x7f331400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f331affd640 1 -- 172.21.15.158:0/23189007 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314017070 con 0x7f332406afb0 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f331affd640 1 -- 172.21.15.158:0/23189007 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331400c940 con 0x7f332406afb0 2024-03-06T07:21:43.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3324143ec0 con 0x7f332406afb0 2024-03-06T07:21:43.235 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f331affd640 1 -- 172.21.15.158:0/23189007 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331400cd30 con 0x7f332406afb0 2024-03-06T07:21:43.235 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3324144480 con 0x7f332406afb0 2024-03-06T07:21:43.235 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f331affd640 1 -- 172.21.15.158:0/23189007 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f331401a070 con 0x7f332406afb0 2024-03-06T07:21:43.235 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.230+0000 7f331affd640 1 -- 172.21.15.158:0/23189007 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f331401ca40 con 0x7f332406afb0 2024-03-06T07:21:43.235 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.231+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33240733b0 con 0x7f332406afb0 2024-03-06T07:21:43.235 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.233+0000 7f331affd640 1 -- 172.21.15.158:0/23189007 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f331401c450 con 0x7f332406afb0 2024-03-06T07:21:43.292 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:43.290+0000 7fc777a13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:43.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.307+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e460d8c9-90e8-400a-8934-b5654984cebc", "id": 6} v 0) v1 -- 0x7f3324068ff0 con 0x7f332406afb0 2024-03-06T07:21:43.314 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.312+0000 7f331affd640 1 -- 172.21.15.158:0/23189007 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e460d8c9-90e8-400a-8934-b5654984cebc", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f331401c630 con 0x7f332406afb0 2024-03-06T07:21:43.314 INFO:teuthology.orchestra.run.smithi158.stdout:6 2024-03-06T07:21:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f332406afb0 msgr2=0x7f3324157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f332406afb0 0x7f3324157570 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f331400e8a0 tx=0x7f331400ed60 comp rx=0 tx=0).stop 2024-03-06T07:21:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 shutdown_connections 2024-03-06T07:21:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33240ceda0 0x7f332415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33240cd250 0x7f332415d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 --2- 172.21.15.158:0/23189007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f332406afb0 0x7f3324157570 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 >> 172.21.15.158:0/23189007 conn(0x7f332406c0c0 msgr2=0x7f33240b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 shutdown_connections 2024-03-06T07:21:43.316 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.313+0000 7f332c3f4640 1 -- 172.21.15.158:0/23189007 wait complete. 2024-03-06T07:21:43.325 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster ceph osd new 4784eeb3-95ef-461e-b695-6294c46b6e32 7 2024-03-06T07:21:43.344 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:43.342+0000 7efdbe00d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:43.425 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.423+0000 7f7e00433640 1 Processor -- start 2024-03-06T07:21:43.425 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.423+0000 7f7e00433640 1 -- start start 2024-03-06T07:21:43.427 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.424+0000 7f7e00433640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7df806fd10 0x7f7df80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.427 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.424+0000 7f7e00433640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df80cd4b0 con 0x7f7df80d0af0 2024-03-06T07:21:43.427 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.424+0000 7f7e00433640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df80cd790 con 0x7f7df815c050 2024-03-06T07:21:43.427 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.424+0000 7f7e00433640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df80cda70 con 0x7f7df806fd10 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.424+0000 7f7dfe1cf640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7df815c050 0x7f7df806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.158:52478/0 (socket says 172.21.15.158:52478) 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.424+0000 7f7dfe1cf640 1 -- 172.21.15.158:0/899239440 learned_addr learned my addr 172.21.15.158:0/899239440 (peer_addr_for_me v1:172.21.15.158:0/0) 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.425+0000 7f7dfd9ce640 1 --2- 172.21.15.158:0/899239440 >> v2:172.21.15.77:3300/0 conn(0x7f7df806fd10 0x7f7df80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.425+0000 7f7dfd1cd640 1 -- 172.21.15.158:0/899239440 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778238899 0 0) 0x7f7df80cd790 con 0x7f7df815c050 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.425+0000 7f7dfd1cd640 1 -- 172.21.15.158:0/899239440 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de4003520 con 0x7f7df815c050 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.425+0000 7f7dfd1cd640 1 -- 172.21.15.158:0/899239440 <== mon.1 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 623885121 0 0) 0x7f7de4003520 con 0x7f7df815c050 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.425+0000 7f7dfd9ce640 1 -- 172.21.15.158:0/899239440 >> v1:172.21.15.158:6789/0 conn(0x7f7df815c050 legacy=0x7f7df806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.425+0000 7f7dfd9ce640 1 -- 172.21.15.158:0/899239440 >> v1:172.21.15.77:6789/0 conn(0x7f7df80d0af0 legacy=0x7f7df80ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.425+0000 7f7dfd9ce640 1 -- 172.21.15.158:0/899239440 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df80cdd50 con 0x7f7df806fd10 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.426+0000 7f7dfd9ce640 1 --2- 172.21.15.158:0/899239440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df806fd10 0x7f7df80cc760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009720 tx=0x7f7df402b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52700374a626f855 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.426+0000 7f7dfd1cd640 1 -- 172.21.15.158:0/899239440 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4037070 con 0x7f7df806fd10 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.426+0000 7f7dfd1cd640 1 -- 172.21.15.158:0/899239440 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df4030510 con 0x7f7df806fd10 2024-03-06T07:21:43.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.426+0000 7f7dfd1cd640 1 -- 172.21.15.158:0/899239440 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8001670 con 0x7f7df806fd10 2024-03-06T07:21:43.429 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.426+0000 7f7e00433640 1 -- 172.21.15.158:0/899239440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df806fd10 msgr2=0x7f7df80cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.429 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.426+0000 7f7e00433640 1 --2- 172.21.15.158:0/899239440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df806fd10 0x7f7df80cc760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009720 tx=0x7f7df402b570 comp rx=0 tx=0).stop 2024-03-06T07:21:43.430 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.427+0000 7f7e00433640 1 -- 172.21.15.158:0/899239440 shutdown_connections 2024-03-06T07:21:43.430 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.427+0000 7f7e00433640 1 --2- 172.21.15.158:0/899239440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df806fd10 0x7f7df80cc760 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.430 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.427+0000 7f7e00433640 1 -- 172.21.15.158:0/899239440 >> 172.21.15.158:0/899239440 conn(0x7f7df80b3800 msgr2=0x7f7df80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.430 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.427+0000 7f7e00433640 1 -- 172.21.15.158:0/899239440 shutdown_connections 2024-03-06T07:21:43.430 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.427+0000 7f7e00433640 1 -- 172.21.15.158:0/899239440 wait complete. 2024-03-06T07:21:43.430 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7e00433640 1 Processor -- start 2024-03-06T07:21:43.430 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7e00433640 1 -- start start 2024-03-06T07:21:43.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7e00433640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7df806fd10 0x7f7df814c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7e00433640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df80d0af0 0x7f7df8159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7e00433640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df815c050 0x7f7df8149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7e00433640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df8166e50 con 0x7f7df815c050 2024-03-06T07:21:43.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7e00433640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df8166cd0 con 0x7f7df80d0af0 2024-03-06T07:21:43.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7e00433640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df8166fd0 con 0x7f7df806fd10 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7dfe1cf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7df806fd10 0x7f7df814c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.428+0000 7f7dfd9ce640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df80d0af0 0x7f7df8159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfd9ce640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df80d0af0 0x7f7df8159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:34470/0 (socket says 172.21.15.158:34470) 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfd9ce640 1 -- 172.21.15.158:0/2701640350 learned_addr learned my addr 172.21.15.158:0/2701640350 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfe9d0640 1 --2- 172.21.15.158:0/2701640350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df815c050 0x7f7df8149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfd9ce640 1 -- 172.21.15.158:0/2701640350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7df806fd10 msgr2=0x7f7df814c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfd9ce640 1 --2- 172.21.15.158:0/2701640350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7df806fd10 0x7f7df814c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfd9ce640 1 -- 172.21.15.158:0/2701640350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df815c050 msgr2=0x7f7df8149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfd9ce640 1 --2- 172.21.15.158:0/2701640350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df815c050 0x7f7df8149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfd9ce640 1 -- 172.21.15.158:0/2701640350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df80b68e0 con 0x7f7df80d0af0 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7dfd9ce640 1 --2- 172.21.15.158:0/2701640350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df80d0af0 0x7f7df8159e10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009720 tx=0x7f7df40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7deeffd640 1 -- 172.21.15.158:0/2701640350 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4042070 con 0x7f7df80d0af0 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7deeffd640 1 -- 172.21.15.158:0/2701640350 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df4004220 con 0x7f7df80d0af0 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df80b6320 con 0x7f7df80d0af0 2024-03-06T07:21:43.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7deeffd640 1 -- 172.21.15.158:0/2701640350 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4037040 con 0x7f7df80d0af0 2024-03-06T07:21:43.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df814a150 con 0x7f7df80d0af0 2024-03-06T07:21:43.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7deeffd640 1 -- 172.21.15.158:0/2701640350 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7df403a040 con 0x7f7df80d0af0 2024-03-06T07:21:43.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.429+0000 7f7deeffd640 1 -- 172.21.15.158:0/2701640350 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7df4003c70 con 0x7f7df80d0af0 2024-03-06T07:21:43.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.430+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df809fe60 con 0x7f7df80d0af0 2024-03-06T07:21:43.434 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:43.432+0000 7fc777a13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:43.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.432+0000 7f7deeffd640 1 -- 172.21.15.158:0/2701640350 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7df403e670 con 0x7f7df80d0af0 2024-03-06T07:21:43.457 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:43.455+0000 7efdbe00d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:43.503 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:43.501+0000 7efdbe00d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:43.506 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.505+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4784eeb3-95ef-461e-b695-6294c46b6e32", "id": 7} v 0) v1 -- 0x7f7df8070540 con 0x7f7df80d0af0 2024-03-06T07:21:43.512 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.510+0000 7f7deeffd640 1 -- 172.21.15.158:0/2701640350 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4784eeb3-95ef-461e-b695-6294c46b6e32", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7df403e850 con 0x7f7df80d0af0 2024-03-06T07:21:43.512 INFO:teuthology.orchestra.run.smithi158.stdout:7 2024-03-06T07:21:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df80d0af0 msgr2=0x7f7df8159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 --2- 172.21.15.158:0/2701640350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df80d0af0 0x7f7df8159e10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009720 tx=0x7f7df40046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 shutdown_connections 2024-03-06T07:21:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 --2- 172.21.15.158:0/2701640350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df815c050 0x7f7df8149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 --2- 172.21.15.158:0/2701640350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7df80d0af0 0x7f7df8159e10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 --2- 172.21.15.158:0/2701640350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7df806fd10 0x7f7df814c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.513 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 >> 172.21.15.158:0/2701640350 conn(0x7f7df80b3800 msgr2=0x7f7df80b3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 shutdown_connections 2024-03-06T07:21:43.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:21:43.511+0000 7f7e00433640 1 -- 172.21.15.158:0/2701640350 wait complete. 2024-03-06T07:21:43.523 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:21:43.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:21:43.526 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:21:43.526 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:21:43.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:21:43.528 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:21:43.528 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:21:43.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:21:43.530 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:21:43.530 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:21:43.530 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:21:43.533 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:21:43.533 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:21:43.533 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:21:43.535 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:21:43.535 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:21:43.535 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:21:43.537 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:21:43.537 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:21:43.537 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:21:43.539 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:21:43.539 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:21:43.540 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:21:43.542 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:21:43.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:43.557 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:43.556+0000 7fc777a13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:43.610 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:43.608+0000 7fc777a13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:43.655 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:43.653+0000 7efdbe00d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:43.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.669+0000 7f7c52cb5640 1 Processor -- start 2024-03-06T07:21:43.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c52cb5640 1 -- start start 2024-03-06T07:21:43.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c52cb5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c4c15dfe0 0x7f7c4c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c52cb5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7c4c0cf830 0x7f7c4c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c52cb5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4c0b3430 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c52cb5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4c0b31e0 con 0x7f7c4c15dfe0 2024-03-06T07:21:43.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c52cb5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4c0b3d60 con 0x7f7c4c0cf830 2024-03-06T07:21:43.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c50a51640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c4c15dfe0 0x7f7c4c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c50a51640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c4c15dfe0 0x7f7c4c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35096/0 (socket says 172.21.15.77:35096) 2024-03-06T07:21:43.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.670+0000 7f7c50a51640 1 -- 172.21.15.77:0/2108368124 learned_addr learned my addr 172.21.15.77:0/2108368124 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:43.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.671+0000 7f7c4bfff640 1 --2- 172.21.15.77:0/2108368124 >> v2:172.21.15.158:3300/0 conn(0x7f7c4c0cf830 0x7f7c4c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.671+0000 7f7c50a51640 1 -- 172.21.15.77:0/2108368124 >> v2:172.21.15.158:3300/0 conn(0x7f7c4c0cf830 msgr2=0x7f7c4c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.671+0000 7f7c50a51640 1 --2- 172.21.15.77:0/2108368124 >> v2:172.21.15.158:3300/0 conn(0x7f7c4c0cf830 0x7f7c4c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.671+0000 7f7c50a51640 1 -- 172.21.15.77:0/2108368124 >> v1:172.21.15.158:6789/0 conn(0x7f7c4c0d27f0 legacy=0x7f7c4c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.671+0000 7f7c50a51640 1 -- 172.21.15.77:0/2108368124 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c0b3710 con 0x7f7c4c15dfe0 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.671+0000 7f7c50a51640 1 --2- 172.21.15.77:0/2108368124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c15dfe0 0x7f7c4c0cf2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009870 tx=0x7f7c4002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5e2371fd5b5fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c4b7fe640 1 -- 172.21.15.77:0/2108368124 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40034070 con 0x7f7c4c15dfe0 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c4b7fe640 1 -- 172.21.15.77:0/2108368124 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c40030860 con 0x7f7c4c15dfe0 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c4b7fe640 1 -- 172.21.15.77:0/2108368124 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40030c30 con 0x7f7c4c15dfe0 2024-03-06T07:21:43.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2108368124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c15dfe0 msgr2=0x7f7c4c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c52cb5640 1 --2- 172.21.15.77:0/2108368124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c15dfe0 0x7f7c4c0cf2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009870 tx=0x7f7c4002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2108368124 shutdown_connections 2024-03-06T07:21:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c52cb5640 1 --2- 172.21.15.77:0/2108368124 >> v2:172.21.15.158:3300/0 conn(0x7f7c4c0cf830 0x7f7c4c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c52cb5640 1 --2- 172.21.15.77:0/2108368124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c15dfe0 0x7f7c4c0cf2f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2108368124 >> 172.21.15.77:0/2108368124 conn(0x7f7c4c0d3cd0 msgr2=0x7f7c4c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2108368124 shutdown_connections 2024-03-06T07:21:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.672+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2108368124 wait complete. 2024-03-06T07:21:43.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.674+0000 7f7c52cb5640 1 Processor -- start 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.674+0000 7f7c52cb5640 1 -- start start 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.674+0000 7f7c52cb5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c4c0cf830 0x7f7c4c1f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.674+0000 7f7c52cb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c0d27f0 0x7f7c4c1f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.674+0000 7f7c52cb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c4c15dfe0 0x7f7c4c1f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c52cb5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0d1df0 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c52cb5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0d1c70 con 0x7f7c4c0cf830 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c52cb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c15e3c0 con 0x7f7c4c15dfe0 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c0d27f0 0x7f7c4c1f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c0d27f0 0x7f7c4c1f4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35102/0 (socket says 172.21.15.77:35102) 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 -- 172.21.15.77:0/2329385288 learned_addr learned my addr 172.21.15.77:0/2329385288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c50a51640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c4c0cf830 0x7f7c4c1f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c51252640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c4c15dfe0 0x7f7c4c1f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 -- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c4c15dfe0 msgr2=0x7f7c4c1f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c4c15dfe0 0x7f7c4c1f71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 -- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c4c0cf830 msgr2=0x7f7c4c1f6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c4c0cf830 0x7f7c4c1f6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 -- 172.21.15.77:0/2329385288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c0cc3d0 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c50a51640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c4c0cf830 0x7f7c4c1f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c4bfff640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c0d27f0 0x7f7c4c1f4da0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800d5c0 tx=0x7f7c3800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c497fa640 1 -- 172.21.15.77:0/2329385288 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c380125f0 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c497fa640 1 -- 172.21.15.77:0/2329385288 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c38019070 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4c0cc6b0 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c497fa640 1 -- 172.21.15.77:0/2329385288 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c380129d0 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.675+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4c0ccc70 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.676+0000 7f7c497fa640 1 -- 172.21.15.77:0/2329385288 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7c38012b70 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.676+0000 7f7c497fa640 1 -- 172.21.15.77:0/2329385288 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7c3801eb30 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.676+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4c0b5200 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.679+0000 7f7c497fa640 1 -- 172.21.15.77:0/2329385288 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7c4c0b5200 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.727 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:43.725+0000 7fc777a13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:43.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.738+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c4c0cce70 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.738+0000 7f7c497fa640 1 -- 172.21.15.77:0/2329385288 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f7c4c0cce70 con 0x7f7c4c0d27f0 2024-03-06T07:21:43.740 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:43.741 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:43.506795+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:43.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c0d27f0 msgr2=0x7f7c4c1f4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c0d27f0 0x7f7c4c1f4da0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800d5c0 tx=0x7f7c3800da80 comp rx=0 tx=0).stop 2024-03-06T07:21:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 shutdown_connections 2024-03-06T07:21:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c4c0d27f0 0x7f7c4c1f4da0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c4c15dfe0 0x7f7c4c1f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 --2- 172.21.15.77:0/2329385288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c4c0cf830 0x7f7c4c1f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 >> 172.21.15.77:0/2329385288 conn(0x7f7c4c0d3cd0 msgr2=0x7f7c4c06c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 shutdown_connections 2024-03-06T07:21:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:43.739+0000 7f7c52cb5640 1 -- 172.21.15.77:0/2329385288 wait complete. 2024-03-06T07:21:43.752 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:21:43.752 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:21:43.753 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:43.751+0000 7efdbe00d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:43.822 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:43.820+0000 7fc777a13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:43.908 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:43.906+0000 7efdbe00d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:43.951 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:43.949+0000 7efdbe00d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:43.977 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:43.975+0000 7fc777a13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:44.019 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:44.017+0000 7fc777a13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:44.103 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:44.101+0000 7efdbe00d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:44.130 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-03-06T07:21:44.128+0000 7f74427bb640 -1 Falling back to public interface 2024-03-06T07:21:44.144 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-03-06T07:21:44.142+0000 7faf4a32a640 -1 Falling back to public interface 2024-03-06T07:21:44.147 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-03-06T07:21:44.145+0000 7f38db610640 -1 Falling back to public interface 2024-03-06T07:21:44.148 INFO:tasks.ceph.osd.4.smithi158.stderr:2024-03-06T07:21:44.146+0000 7fc58b819640 -1 Falling back to public interface 2024-03-06T07:21:44.148 INFO:tasks.ceph.osd.6.smithi158.stderr:2024-03-06T07:21:44.146+0000 7f742cdff640 -1 Falling back to public interface 2024-03-06T07:21:44.148 INFO:tasks.ceph.osd.4.smithi158.stderr:2024-03-06T07:21:44.146+0000 7fc58a225640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-03-06T07:21:44.158 INFO:tasks.ceph.osd.7.smithi158.stderr:2024-03-06T07:21:44.156+0000 7f59f3c25640 -1 Falling back to public interface 2024-03-06T07:21:44.159 INFO:tasks.ceph.osd.5.smithi158.stderr:2024-03-06T07:21:44.157+0000 7fe9d0171640 -1 Falling back to public interface 2024-03-06T07:21:44.164 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-03-06T07:21:44.163+0000 7f73d5a19640 -1 Falling back to public interface 2024-03-06T07:21:44.174 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:44.172+0000 7fc777a13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:44.558 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:44.649 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.645+0000 7f511d439640 1 Processor -- start 2024-03-06T07:21:44.649 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.646+0000 7f511d439640 1 -- start start 2024-03-06T07:21:44.649 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.646+0000 7f511d439640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f511815e2b0 0x7f51180d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.649 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.646+0000 7f511d439640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51180d29d0 0x7f51180b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.649 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.646+0000 7f511d439640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180d4a50 con 0x7f51180b2c50 2024-03-06T07:21:44.649 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.646+0000 7f511d439640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51180d4800 con 0x7f511815e2b0 2024-03-06T07:21:44.649 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.646+0000 7f511d439640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51180cf300 con 0x7f51180d29d0 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f5116ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f511815e2b0 0x7f51180d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f51177fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51180d29d0 0x7f51180b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f5116ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f511815e2b0 0x7f51180d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:36396/0 (socket says 172.21.15.77:36396) 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f5116ffd640 1 -- 172.21.15.77:0/169665288 learned_addr learned my addr 172.21.15.77:0/169665288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f51177fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51180d29d0 0x7f51180b0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:41890/0 (socket says 172.21.15.77:41890) 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f5116ffd640 1 -- 172.21.15.77:0/169665288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51180d29d0 msgr2=0x7f51180b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f5116ffd640 1 --2- 172.21.15.77:0/169665288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51180d29d0 0x7f51180b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f5116ffd640 1 -- 172.21.15.77:0/169665288 >> v1:172.21.15.158:6789/0 conn(0x7f51180b2c50 legacy=0x7f51180d22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f5116ffd640 1 -- 172.21.15.77:0/169665288 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51180cf550 con 0x7f511815e2b0 2024-03-06T07:21:44.650 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.647+0000 7f51177fe640 1 --2- 172.21.15.77:0/169665288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51180d29d0 0x7f51180b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.648+0000 7f5116ffd640 1 --2- 172.21.15.77:0/169665288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f511815e2b0 0x7f51180d1ac0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5108009870 tx=0x7f510802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=174888045891f856 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.648+0000 7f5115ffb640 1 -- 172.21.15.77:0/169665288 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108035020 con 0x7f511815e2b0 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.648+0000 7f5115ffb640 1 -- 172.21.15.77:0/169665288 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108031640 con 0x7f511815e2b0 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.648+0000 7f5115ffb640 1 -- 172.21.15.77:0/169665288 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510802d040 con 0x7f511815e2b0 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.649+0000 7f511d439640 1 -- 172.21.15.77:0/169665288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f511815e2b0 msgr2=0x7f51180d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.649+0000 7f511d439640 1 --2- 172.21.15.77:0/169665288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f511815e2b0 0x7f51180d1ac0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5108009870 tx=0x7f510802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.649+0000 7f511d439640 1 -- 172.21.15.77:0/169665288 shutdown_connections 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.649+0000 7f511d439640 1 --2- 172.21.15.77:0/169665288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51180d29d0 0x7f51180b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.652 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.649+0000 7f511d439640 1 --2- 172.21.15.77:0/169665288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f511815e2b0 0x7f51180d1ac0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.653 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.649+0000 7f511d439640 1 -- 172.21.15.77:0/169665288 >> 172.21.15.77:0/169665288 conn(0x7f51180d3ce0 msgr2=0x7f51180d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.653 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.649+0000 7f511d439640 1 -- 172.21.15.77:0/169665288 shutdown_connections 2024-03-06T07:21:44.653 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.649+0000 7f511d439640 1 -- 172.21.15.77:0/169665288 wait complete. 2024-03-06T07:21:44.653 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.650+0000 7f511d439640 1 Processor -- start 2024-03-06T07:21:44.653 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.650+0000 7f511d439640 1 -- start start 2024-03-06T07:21:44.656 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f511d439640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51180b2c50 0x7f51180b8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.656 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f511d439640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51180d29d0 0x7f511814f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.656 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f511d439640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f511815e2b0 0x7f51180cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.656 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f511d439640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51180b10c0 con 0x7f51180b2c50 2024-03-06T07:21:44.656 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f511d439640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51180cfb10 con 0x7f51180d29d0 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f511d439640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51180b1240 con 0x7f511815e2b0 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51180b2c50 0x7f51180b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51180b2c50 0x7f51180b8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35110/0 (socket says 172.21.15.77:35110) 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 -- 172.21.15.77:0/1678653436 learned_addr learned my addr 172.21.15.77:0/1678653436 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f51167fc640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51180d29d0 0x7f511814f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f51177fe640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f511815e2b0 0x7f51180cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 -- 172.21.15.77:0/1678653436 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f511815e2b0 msgr2=0x7f51180cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f511815e2b0 0x7f51180cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 -- 172.21.15.77:0/1678653436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51180d29d0 msgr2=0x7f511814f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51180d29d0 0x7f511814f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 -- 172.21.15.77:0/1678653436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511814f650 con 0x7f51180b2c50 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f5116ffd640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51180b2c50 0x7f51180b8020 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5108000c00 tx=0x7f51080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.657 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.651+0000 7f50ff7fe640 1 -- 172.21.15.77:0/1678653436 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108040070 con 0x7f51180b2c50 2024-03-06T07:21:44.658 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.652+0000 7f50ff7fe640 1 -- 172.21.15.77:0/1678653436 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51080399f0 con 0x7f51180b2c50 2024-03-06T07:21:44.658 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.652+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51181f4110 con 0x7f51180b2c50 2024-03-06T07:21:44.658 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.652+0000 7f50ff7fe640 1 -- 172.21.15.77:0/1678653436 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108035040 con 0x7f51180b2c50 2024-03-06T07:21:44.658 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.652+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51181f46d0 con 0x7f51180b2c50 2024-03-06T07:21:44.658 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.652+0000 7f50ff7fe640 1 -- 172.21.15.77:0/1678653436 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f510804b060 con 0x7f51180b2c50 2024-03-06T07:21:44.658 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.652+0000 7f50ff7fe640 1 -- 172.21.15.77:0/1678653436 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f5108039e30 con 0x7f51180b2c50 2024-03-06T07:21:44.658 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.653+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511804f330 con 0x7f51180b2c50 2024-03-06T07:21:44.658 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.656+0000 7f50ff7fe640 1 -- 172.21.15.77:0/1678653436 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5108063dd0 con 0x7f51180b2c50 2024-03-06T07:21:44.720 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.718+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51180d20a0 con 0x7f51180b2c50 2024-03-06T07:21:44.721 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.719+0000 7f50ff7fe640 1 -- 172.21.15.77:0/1678653436 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f5108052070 con 0x7f51180b2c50 2024-03-06T07:21:44.721 INFO:teuthology.misc.health.smithi077.stdout: 2024-03-06T07:21:44.722 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":10,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:43.506795+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51180b2c50 msgr2=0x7f51180b8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51180b2c50 0x7f51180b8020 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5108000c00 tx=0x7f51080046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 shutdown_connections 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f511815e2b0 0x7f51180cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51180d29d0 0x7f511814f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 --2- 172.21.15.77:0/1678653436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51180b2c50 0x7f51180b8020 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 >> 172.21.15.77:0/1678653436 conn(0x7f51180d3ce0 msgr2=0x7f51180b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 shutdown_connections 2024-03-06T07:21:44.723 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:44.720+0000 7f511d439640 1 -- 172.21.15.77:0/1678653436 wait complete. 2024-03-06T07:21:44.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:44.922 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:44.920+0000 7efdbe00d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:45.003 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:45.001+0000 7fc777a13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:45.196 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:45.195+0000 7efdbe00d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:45.249 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:45.247+0000 7efdbe00d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:45.280 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:45.277+0000 7fc777a13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:45.330 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:45.329+0000 7fc777a13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:45.364 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:45.362+0000 7efdbe00d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:45.452 INFO:tasks.ceph.mgr.x.smithi158.stderr:2024-03-06T07:21:45.450+0000 7fc777a13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:46.390 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:46.388+0000 7efd5e89d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:21:46.390 INFO:tasks.ceph.mgr.y.smithi077.stderr:2024-03-06T07:21:46.388+0000 7efd5e89d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:21:47.599 INFO:tasks.ceph.osd.6.smithi158.stderr:2024-03-06T07:21:47.597+0000 7f742cdff640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:21:47.600 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-03-06T07:21:47.598+0000 7f74427bb640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:21:47.646 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-03-06T07:21:47.644+0000 7faf4a32a640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:21:47.664 INFO:tasks.ceph.osd.5.smithi158.stderr:2024-03-06T07:21:47.662+0000 7fe9d0171640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:21:47.668 INFO:tasks.ceph.osd.7.smithi158.stderr:2024-03-06T07:21:47.667+0000 7f59f3c25640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:21:47.669 INFO:tasks.ceph.osd.4.smithi158.stderr:2024-03-06T07:21:47.667+0000 7fc58b819640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:21:47.674 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-03-06T07:21:47.673+0000 7f38db610640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:21:47.680 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-03-06T07:21:47.679+0000 7f73d5a19640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:21:50.762 INFO:tasks.ceph.osd.5.smithi158.stderr:2024-03-06T07:21:50.759+0000 7fe9cd11c640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:21:50.773 INFO:tasks.ceph.osd.5.smithi158.stderr:2024-03-06T07:21:50.771+0000 7fe9c3f16640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:50.804 INFO:tasks.ceph.osd.7.smithi158.stderr:2024-03-06T07:21:50.801+0000 7f59f0584640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:21:50.810 INFO:tasks.ceph.osd.4.smithi158.stderr:2024-03-06T07:21:50.808+0000 7fc5869e9640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:21:50.813 INFO:tasks.ceph.osd.7.smithi158.stderr:2024-03-06T07:21:50.811+0000 7f59e737e640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:50.820 INFO:tasks.ceph.osd.4.smithi158.stderr:2024-03-06T07:21:50.818+0000 7fc57f01b640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:50.851 INFO:tasks.ceph.osd.6.smithi158.stderr:2024-03-06T07:21:50.849+0000 7f7428572640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:21:50.869 INFO:tasks.ceph.osd.6.smithi158.stderr:2024-03-06T07:21:50.867+0000 7f7420ba4640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:51.412 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-03-06T07:21:51.410+0000 7f743df2e640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:21:51.425 INFO:tasks.ceph.osd.1.smithi077.stderr:2024-03-06T07:21:51.424+0000 7f7436560640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:51.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-03-06T07:21:51.447+0000 7faf45a9d640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:21:51.456 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-03-06T07:21:51.455+0000 7faf3e0cf640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:51.473 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-03-06T07:21:51.471+0000 7f38d7ee9640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:21:51.486 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-03-06T07:21:51.485+0000 7f73d0c5a640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:21:51.488 INFO:tasks.ceph.osd.3.smithi077.stderr:2024-03-06T07:21:51.487+0000 7f38cece3640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:51.510 INFO:tasks.ceph.osd.2.smithi077.stderr:2024-03-06T07:21:51.508+0000 7f73c928c640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:51.537 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:51.626 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.624+0000 7eff4f822640 1 Processor -- start 2024-03-06T07:21:51.627 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.625+0000 7eff4f822640 1 -- start start 2024-03-06T07:21:51.630 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.625+0000 7eff4f822640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff4806ca10 0x7eff4806cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.625+0000 7eff4f822640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4806d330 con 0x7eff480d25c0 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.625+0000 7eff4f822640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4806c440 con 0x7eff480d0d80 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.625+0000 7eff4f822640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff480b1130 con 0x7eff4806ca10 2024-03-06T07:21:51.631 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4cdbd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7eff480d25c0 0x7eff480ccd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54444/0 (socket says 172.21.15.77:54444) 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4cdbd640 1 -- 172.21.15.77:0/1048805124 learned_addr learned my addr 172.21.15.77:0/1048805124 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4d5be640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff4806ca10 0x7eff4806cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414809501 0 0) 0x7eff4806d330 con 0x7eff480d25c0 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff34003520 con 0x7eff480d25c0 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1535599082 0 0) 0x7eff34003520 con 0x7eff480d25c0 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff4806ca10 msgr2=0x7eff4806cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff4806ca10 0x7eff4806cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 >> v1:172.21.15.158:6789/0 conn(0x7eff480d0d80 legacy=0x7eff480cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.633 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff480cc290 con 0x7eff480d25c0 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7eff38001670 con 0x7eff480d25c0 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 >> v1:172.21.15.77:6789/0 conn(0x7eff480d25c0 legacy=0x7eff480ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff340039e0 0x7eff34003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff34004800 0x7eff34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff4806ca10 0x7eff340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3400ba70 con 0x7eff34004800 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3400bbf0 con 0x7eff340039e0 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3400bdf0 con 0x7eff4806ca10 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4cdbd640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff34004800 0x7eff34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4d5be640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff340039e0 0x7eff34003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4ddbf640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff4806ca10 0x7eff340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4cdbd640 1 -- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff4806ca10 msgr2=0x7eff340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4cdbd640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff4806ca10 0x7eff340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.634 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4cdbd640 1 -- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff340039e0 msgr2=0x7eff34003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4cdbd640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff340039e0 0x7eff34003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.626+0000 7eff4cdbd640 1 -- 172.21.15.77:0/1048805124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34003520 con 0x7eff34004800 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.627+0000 7eff4cdbd640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff34004800 0x7eff34004c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff38001eb0 tx=0x7eff34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495845725f0efc6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.627+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff38007d30 con 0x7eff34004800 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.627+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff38030070 con 0x7eff34004800 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.627+0000 7eff3ffff640 1 -- 172.21.15.77:0/1048805124 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff38005d30 con 0x7eff34004800 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.628+0000 7eff4f822640 1 -- 172.21.15.77:0/1048805124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff34004800 msgr2=0x7eff34004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.628+0000 7eff4f822640 1 --2- 172.21.15.77:0/1048805124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff34004800 0x7eff34004c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff38001eb0 tx=0x7eff34005830 comp rx=0 tx=0).stop 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.628+0000 7eff4ddbf640 1 -- 172.21.15.77:0/1048805124 reap_dead start 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.628+0000 7eff4f822640 1 -- 172.21.15.77:0/1048805124 shutdown_connections 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.628+0000 7eff4f822640 1 -- 172.21.15.77:0/1048805124 >> 172.21.15.77:0/1048805124 conn(0x7eff480d1aa0 msgr2=0x7eff480d1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.635 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.628+0000 7eff4f822640 1 -- 172.21.15.77:0/1048805124 shutdown_connections 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.628+0000 7eff4f822640 1 -- 172.21.15.77:0/1048805124 wait complete. 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.629+0000 7eff4f822640 1 Processor -- start 2024-03-06T07:21:51.636 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.629+0000 7eff4f822640 1 -- start start 2024-03-06T07:21:51.638 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4f822640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff34004800 0x7eff48150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.638 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4f822640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff480affd0 0x7eff480b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.638 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4f822640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff480b2740 0x7eff480b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.638 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4f822640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff48068070 con 0x7eff480affd0 2024-03-06T07:21:51.638 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4f822640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff480cc850 con 0x7eff34004800 2024-03-06T07:21:51.638 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4f822640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4815e320 con 0x7eff480b2740 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff480affd0 0x7eff480b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff480affd0 0x7eff480b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46630/0 (socket says 172.21.15.77:46630) 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 -- 172.21.15.77:0/131742255 learned_addr learned my addr 172.21.15.77:0/131742255 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4d5be640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff34004800 0x7eff48150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4ddbf640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff480b2740 0x7eff480b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 -- 172.21.15.77:0/131742255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff480b2740 msgr2=0x7eff480b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff480b2740 0x7eff480b2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 -- 172.21.15.77:0/131742255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff34004800 msgr2=0x7eff48150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff34004800 0x7eff48150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 -- 172.21.15.77:0/131742255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff481f7020 con 0x7eff480affd0 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4cdbd640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff480affd0 0x7eff480b0450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c2d0 tx=0x7eff38001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff3dffb640 1 -- 172.21.15.77:0/131742255 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff38045070 con 0x7eff480affd0 2024-03-06T07:21:51.639 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff3dffb640 1 -- 172.21.15.77:0/131742255 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff38004bf0 con 0x7eff480affd0 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff481f7300 con 0x7eff480affd0 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.630+0000 7eff3dffb640 1 -- 172.21.15.77:0/131742255 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff38007d10 con 0x7eff480affd0 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.631+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff481f78c0 con 0x7eff480affd0 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.632+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4804f330 con 0x7eff480affd0 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.632+0000 7eff3dffb640 1 -- 172.21.15.77:0/131742255 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7eff38007070 con 0x7eff480affd0 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.632+0000 7eff3dffb640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff2c06b470 0x7eff2c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.632+0000 7eff3dffb640 1 -- 172.21.15.77:0/131742255 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7eff380aab30 con 0x7eff480affd0 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.632+0000 7eff4d5be640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff2c06b470 0x7eff2c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.640 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.633+0000 7eff4d5be640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff2c06b470 0x7eff2c06d930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7eff40014510 tx=0x7eff4000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.641 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.637+0000 7eff3dffb640 1 -- 172.21.15.77:0/131742255 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff38079a80 con 0x7eff480affd0 2024-03-06T07:21:51.814 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.812+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff480cd1a0 con 0x7eff480affd0 2024-03-06T07:21:51.814 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.813+0000 7eff3dffb640 1 -- 172.21.15.77:0/131742255 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10897 (secure 0 0 0) 0x7eff3807db20 con 0x7eff480affd0 2024-03-06T07:21:51.815 INFO:teuthology.misc.health.smithi077.stdout: 2024-03-06T07:21:51.815 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":13,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:51.514235+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.814+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff2c06b470 msgr2=0x7eff2c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.814+0000 7eff4f822640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff2c06b470 0x7eff2c06d930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7eff40014510 tx=0x7eff4000b040 comp rx=0 tx=0).stop 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.814+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff480affd0 msgr2=0x7eff480b0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.814+0000 7eff4f822640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff480affd0 0x7eff480b0450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c2d0 tx=0x7eff38001a40 comp rx=0 tx=0).stop 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.815+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 shutdown_connections 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.815+0000 7eff4f822640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff2c06b470 0x7eff2c06d930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.815+0000 7eff4f822640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff480affd0 0x7eff480b0450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.815+0000 7eff4f822640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff480b2740 0x7eff480b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.815+0000 7eff4f822640 1 --2- 172.21.15.77:0/131742255 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff34004800 0x7eff48150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.815+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 >> 172.21.15.77:0/131742255 conn(0x7eff480d1aa0 msgr2=0x7eff4815cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.817 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.815+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 shutdown_connections 2024-03-06T07:21:51.818 INFO:teuthology.misc.health.smithi077.stderr:2024-03-06T07:21:51.815+0000 7eff4f822640 1 -- 172.21.15.77:0/131742255 wait complete. 2024-03-06T07:21:51.826 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:21:51.826 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:21:51.826 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:21:51.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:21:51.829 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:21:51.829 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:21:51.829 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:21:51.831 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:21:51.831 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:21:51.832 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:21:51.834 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:21:51.835 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:21:51.835 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:21:51.837 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:21:51.837 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:21:51.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:51.906 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:21:51.906 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:21:51.920 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:21:51.921 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.941+0000 7f372abf6640 1 Processor -- start 2024-03-06T07:21:51.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.941+0000 7f372abf6640 1 -- start start 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f372abf6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37240d2710 0x7f37240b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f372abf6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37240b3430 con 0x7f37240d22b0 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f372abf6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37240b3710 con 0x7f372415dfe0 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f372abf6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37240b31e0 con 0x7f37240d2710 2024-03-06T07:21:51.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f3729193640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37240d2710 0x7f37240b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f3723fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f37240d22b0 0x7f37240b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54450/0 (socket says 172.21.15.77:54450) 2024-03-06T07:21:51.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f3723fff640 1 -- 172.21.15.77:0/3406594240 learned_addr learned my addr 172.21.15.77:0/3406594240 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:21:51.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f3729193640 1 -- 172.21.15.77:0/3406594240 >> v1:172.21.15.158:6789/0 conn(0x7f372415dfe0 legacy=0x7f37240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f3729193640 1 -- 172.21.15.77:0/3406594240 >> v1:172.21.15.77:6789/0 conn(0x7f37240d22b0 legacy=0x7f37240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f37237fe640 1 -- 172.21.15.77:0/3406594240 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430448639 0 0) 0x7f37240b3430 con 0x7f37240d22b0 2024-03-06T07:21:51.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.942+0000 7f3729193640 1 -- 172.21.15.77:0/3406594240 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37240b39f0 con 0x7f37240d2710 2024-03-06T07:21:51.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.943+0000 7f3729193640 1 --2- 172.21.15.77:0/3406594240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b4900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f371c009dc0 tx=0x7f371c02bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66021e3f9218edbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.943+0000 7f37237fe640 1 -- 172.21.15.77:0/3406594240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c039070 con 0x7f37240d2710 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.943+0000 7f37237fe640 1 -- 172.21.15.77:0/3406594240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c02ccd0 con 0x7f37240d2710 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.943+0000 7f37237fe640 1 -- 172.21.15.77:0/3406594240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c034660 con 0x7f37240d2710 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.943+0000 7f372abf6640 1 -- 172.21.15.77:0/3406594240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 msgr2=0x7f37240b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.944+0000 7f372abf6640 1 --2- 172.21.15.77:0/3406594240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b4900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f371c009dc0 tx=0x7f371c02bf20 comp rx=0 tx=0).stop 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.944+0000 7f372abf6640 1 -- 172.21.15.77:0/3406594240 shutdown_connections 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.944+0000 7f372abf6640 1 --2- 172.21.15.77:0/3406594240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b4900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.944+0000 7f372abf6640 1 -- 172.21.15.77:0/3406594240 >> 172.21.15.77:0/3406594240 conn(0x7f37240d3cd0 msgr2=0x7f37240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.944+0000 7f372abf6640 1 -- 172.21.15.77:0/3406594240 shutdown_connections 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.944+0000 7f372abf6640 1 -- 172.21.15.77:0/3406594240 wait complete. 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.945+0000 7f372abf6640 1 Processor -- start 2024-03-06T07:21:51.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.945+0000 7f372abf6640 1 -- start start 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f372abf6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37240d22b0 0x7f37240b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f372abf6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f372abf6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f372415dfe0 0x7f37240d0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f372abf6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37240cfa00 con 0x7f37240d2710 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f372abf6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37240b3d60 con 0x7f37240d22b0 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f372abf6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37240d1c70 con 0x7f372415dfe0 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46690/0 (socket says 172.21.15.77:46690) 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 -- 172.21.15.77:0/3847718845 learned_addr learned my addr 172.21.15.77:0/3847718845 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3728992640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37240d22b0 0x7f37240b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 -- 172.21.15.77:0/3847718845 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f372415dfe0 msgr2=0x7f37240d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f372415dfe0 0x7f37240d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 -- 172.21.15.77:0/3847718845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37240d22b0 msgr2=0x7f37240b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37240d22b0 0x7f37240b19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 -- 172.21.15.77:0/3847718845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724160820 con 0x7f37240d2710 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f3723fff640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b9410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f371000c4a0 tx=0x7f371000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f37217fa640 1 -- 172.21.15.77:0/3847718845 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710018070 con 0x7f37240d2710 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f37217fa640 1 -- 172.21.15.77:0/3847718845 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37100074e0 con 0x7f37240d2710 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3724160b00 con 0x7f37240d2710 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f37217fa640 1 -- 172.21.15.77:0/3847718845 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710004a20 con 0x7f37240d2710 2024-03-06T07:21:51.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.946+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37241610c0 con 0x7f37240d2710 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.947+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372415e470 con 0x7f37240d2710 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.948+0000 7f37217fa640 1 -- 172.21.15.77:0/3847718845 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f371001b070 con 0x7f37240d2710 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.948+0000 7f37217fa640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f370806b440 0x7f370806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.948+0000 7f3728992640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f370806b440 0x7f370806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.948+0000 7f37217fa640 1 -- 172.21.15.77:0/3847718845 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f3710088f80 con 0x7f37240d2710 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.948+0000 7f3728992640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f370806b440 0x7f370806d900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3718000f60 tx=0x7f3718009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:51.952+0000 7f37217fa640 1 -- 172.21.15.77:0/3847718845 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3710058320 con 0x7f37240d2710 2024-03-06T07:21:52.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.123+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37240747a0 con 0x7f37240d2710 2024-03-06T07:21:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.124+0000 7f37217fa640 1 -- 172.21.15.77:0/3847718845 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10897 (secure 0 0 0) 0x7f3710057c20 con 0x7f37240d2710 2024-03-06T07:21:52.126 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:52.126 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:51.514235+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f370806b440 msgr2=0x7f370806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f370806b440 0x7f370806d900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3718000f60 tx=0x7f3718009040 comp rx=0 tx=0).stop 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 msgr2=0x7f37240b9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b9410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f371000c4a0 tx=0x7f371000c960 comp rx=0 tx=0).stop 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 shutdown_connections 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f370806b440 0x7f370806d900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37240d2710 0x7f37240b9410 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f372415dfe0 0x7f37240d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 --2- 172.21.15.77:0/3847718845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37240d22b0 0x7f37240b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 >> 172.21.15.77:0/3847718845 conn(0x7f37240d3cd0 msgr2=0x7f372406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 shutdown_connections 2024-03-06T07:21:52.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.126+0000 7f372abf6640 1 -- 172.21.15.77:0/3847718845 wait complete. 2024-03-06T07:21:52.138 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-06T07:21:52.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:52.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.236+0000 7fe663d93640 1 Processor -- start 2024-03-06T07:21:52.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.237+0000 7fe663d93640 1 -- start start 2024-03-06T07:21:52.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.237+0000 7fe663d93640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe65c1515e0 0x7fe65c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.237+0000 7fe663d93640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe65c15b430 0x7fe65c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.237+0000 7fe663d93640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 0x7fe65c0cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.237+0000 7fe663d93640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65c15b8e0 con 0x7fe65c1515e0 2024-03-06T07:21:52.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.237+0000 7fe663d93640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65c070a70 con 0x7fe65c15b430 2024-03-06T07:21:52.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe663d93640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c070d50 con 0x7fe65c0cd7a0 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe661b2f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe65c1515e0 0x7fe65c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe661b2f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe65c1515e0 0x7fe65c0cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46694/0 (socket says 172.21.15.77:46694) 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe661b2f640 1 -- 172.21.15.77:0/3555758366 learned_addr learned my addr 172.21.15.77:0/3555758366 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe66132e640 1 --2- 172.21.15.77:0/3555758366 >> v2:172.21.15.158:3300/0 conn(0x7fe65c15b430 0x7fe65c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe661b2f640 1 -- 172.21.15.77:0/3555758366 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 msgr2=0x7fe65c0cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe662330640 1 --2- 172.21.15.77:0/3555758366 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 0x7fe65c0cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe661b2f640 1 --2- 172.21.15.77:0/3555758366 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 0x7fe65c0cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe661b2f640 1 -- 172.21.15.77:0/3555758366 >> v2:172.21.15.158:3300/0 conn(0x7fe65c15b430 msgr2=0x7fe65c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe661b2f640 1 --2- 172.21.15.77:0/3555758366 >> v2:172.21.15.158:3300/0 conn(0x7fe65c15b430 0x7fe65c071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe661b2f640 1 -- 172.21.15.77:0/3555758366 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c071030 con 0x7fe65c1515e0 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.238+0000 7fe66132e640 1 --2- 172.21.15.77:0/3555758366 >> v2:172.21.15.158:3300/0 conn(0x7fe65c15b430 0x7fe65c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe661b2f640 1 --2- 172.21.15.77:0/3555758366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c0cce40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0cc110 tx=0x7fe65402cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812ca46d2f0d9e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe660b2d640 1 -- 172.21.15.77:0/3555758366 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65403a070 con 0x7fe65c1515e0 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe660b2d640 1 -- 172.21.15.77:0/3555758366 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65402dba0 con 0x7fe65c1515e0 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe660b2d640 1 -- 172.21.15.77:0/3555758366 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65400b040 con 0x7fe65c1515e0 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 -- 172.21.15.77:0/3555758366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 msgr2=0x7fe65c0cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 --2- 172.21.15.77:0/3555758366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c0cce40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0cc110 tx=0x7fe65402cdf0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 -- 172.21.15.77:0/3555758366 shutdown_connections 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 --2- 172.21.15.77:0/3555758366 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 0x7fe65c0cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 --2- 172.21.15.77:0/3555758366 >> v2:172.21.15.158:3300/0 conn(0x7fe65c15b430 0x7fe65c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 --2- 172.21.15.77:0/3555758366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c0cce40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 -- 172.21.15.77:0/3555758366 >> 172.21.15.77:0/3555758366 conn(0x7fe65c0cf8a0 msgr2=0x7fe65c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 -- 172.21.15.77:0/3555758366 shutdown_connections 2024-03-06T07:21:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.239+0000 7fe663d93640 1 -- 172.21.15.77:0/3555758366 wait complete. 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.240+0000 7fe663d93640 1 Processor -- start 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.240+0000 7fe663d93640 1 -- start start 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe663d93640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 0x7fe65c148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe663d93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe663d93640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe65c15b430 0x7fe65c06cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe663d93640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c071d70 con 0x7fe65c1515e0 2024-03-06T07:21:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe663d93640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c0ce0f0 con 0x7fe65c15b430 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe663d93640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c071ef0 con 0x7fe65c0cd7a0 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe66132e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe66132e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46706/0 (socket says 172.21.15.77:46706) 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe66132e640 1 -- 172.21.15.77:0/1025329685 learned_addr learned my addr 172.21.15.77:0/1025329685 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe662330640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe65c15b430 0x7fe65c06cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe661b2f640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 0x7fe65c148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe66132e640 1 -- 172.21.15.77:0/1025329685 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 msgr2=0x7fe65c148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe66132e640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 0x7fe65c148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe66132e640 1 -- 172.21.15.77:0/1025329685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe65c15b430 msgr2=0x7fe65c06cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe66132e640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe65c15b430 0x7fe65c06cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.241+0000 7fe66132e640 1 -- 172.21.15.77:0/1025329685 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c158170 con 0x7fe65c1515e0 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.242+0000 7fe66132e640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c06c800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe65000c820 tx=0x7fe65000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.242+0000 7fe65a7fc640 1 -- 172.21.15.77:0/1025329685 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe650015070 con 0x7fe65c1515e0 2024-03-06T07:21:52.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.242+0000 7fe65a7fc640 1 -- 172.21.15.77:0/1025329685 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe650007a30 con 0x7fe65c1515e0 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.242+0000 7fe65a7fc640 1 -- 172.21.15.77:0/1025329685 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe650007de0 con 0x7fe65c1515e0 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.242+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c158450 con 0x7fe65c1515e0 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.242+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c158a10 con 0x7fe65c1515e0 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.243+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c068840 con 0x7fe65c1515e0 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.243+0000 7fe65a7fc640 1 -- 172.21.15.77:0/1025329685 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fe650018070 con 0x7fe65c1515e0 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.243+0000 7fe65a7fc640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe64806b440 0x7fe64806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.243+0000 7fe661b2f640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe64806b440 0x7fe64806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.243+0000 7fe65a7fc640 1 -- 172.21.15.77:0/1025329685 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fe650089ab0 con 0x7fe65c1515e0 2024-03-06T07:21:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.244+0000 7fe661b2f640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe64806b440 0x7fe64806d900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe654037000 tx=0x7fe654031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.248+0000 7fe65a7fc640 1 -- 172.21.15.77:0/1025329685 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe650058f70 con 0x7fe65c1515e0 2024-03-06T07:21:52.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.458+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe65c153a10 con 0x7fe65c1515e0 2024-03-06T07:21:52.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.458+0000 7fe65a7fc640 1 -- 172.21.15.77:0/1025329685 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fe650058870 con 0x7fe65c1515e0 2024-03-06T07:21:52.460 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:52.460 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:21:52.461 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 2 2024-03-06T07:21:52.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe64806b440 msgr2=0x7fe64806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe64806b440 0x7fe64806d900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe654037000 tx=0x7fe654031000 comp rx=0 tx=0).stop 2024-03-06T07:21:52.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 msgr2=0x7fe65c06c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c06c800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe65000c820 tx=0x7fe65000cce0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 shutdown_connections 2024-03-06T07:21:52.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe64806b440 0x7fe64806d900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65c1515e0 0x7fe65c06c800 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe65c15b430 0x7fe65c06cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 --2- 172.21.15.77:0/1025329685 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe65c0cd7a0 0x7fe65c148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 >> 172.21.15.77:0/1025329685 conn(0x7fe65c0cf8a0 msgr2=0x7fe65c14c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 shutdown_connections 2024-03-06T07:21:52.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:52.460+0000 7fe663d93640 1 -- 172.21.15.77:0/1025329685 wait complete. 2024-03-06T07:21:54.473 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:54.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.582+0000 7fea34c40640 1 Processor -- start 2024-03-06T07:21:54.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.582+0000 7fea34c40640 1 -- start start 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.583+0000 7fea34c40640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 0x7fea300d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.583+0000 7fea34c40640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fea300cd8c0 0x7fea300d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.583+0000 7fea34c40640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea30161980 con 0x7fea300cfff0 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.583+0000 7fea34c40640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea30161c60 con 0x7fea300cd8c0 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.583+0000 7fea34c40640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea30161f40 con 0x7fea3015bb20 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.583+0000 7fea2dd9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fea300cfff0 0x7fea300ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54466/0 (socket says 172.21.15.77:54466) 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2dd9b640 1 -- 172.21.15.77:0/3545204643 learned_addr learned my addr 172.21.15.77:0/3545204643 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2e59c640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 0x7fea300d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2ed9d640 1 --2- 172.21.15.77:0/3545204643 >> v2:172.21.15.158:3300/0 conn(0x7fea300cd8c0 0x7fea300d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336848686 0 0) 0x7fea30161980 con 0x7fea300cfff0 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea18003520 con 0x7fea300cfff0 2024-03-06T07:21:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3311216447 0 0) 0x7fea18003520 con 0x7fea300cfff0 2024-03-06T07:21:54.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 msgr2=0x7fea300d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 0x7fea300d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 >> v2:172.21.15.158:3300/0 conn(0x7fea300cd8c0 msgr2=0x7fea300d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 --2- 172.21.15.77:0/3545204643 >> v2:172.21.15.158:3300/0 conn(0x7fea300cd8c0 0x7fea300d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30162220 con 0x7fea300cfff0 2024-03-06T07:21:54.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fea24001670 con 0x7fea300cfff0 2024-03-06T07:21:54.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2e59c640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 0x7fea300d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 >> v1:172.21.15.77:6789/0 conn(0x7fea300cfff0 legacy=0x7fea300ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea180039b0 0x7fea18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea18004800 0x7fea18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 0x7fea180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30161c60 con 0x7fea180039b0 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30161f40 con 0x7fea18004800 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1800b9c0 con 0x7fea3015bb20 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2e59c640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea180039b0 0x7fea18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2dd9b640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea18004800 0x7fea18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.584+0000 7fea2ed9d640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 0x7fea180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2e59c640 1 -- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 msgr2=0x7fea180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2e59c640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea3015bb20 0x7fea180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2e59c640 1 -- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea18004800 msgr2=0x7fea18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2e59c640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea18004800 0x7fea18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2e59c640 1 -- 172.21.15.77:0/3545204643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18003520 con 0x7fea180039b0 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2e59c640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea180039b0 0x7fea18003dd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fea2800e350 tx=0x7fea18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b3350fe1f7524c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28039070 con 0x7fea180039b0 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2802cd70 con 0x7fea180039b0 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.585+0000 7fea2d59a640 1 -- 172.21.15.77:0/3545204643 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28009de0 con 0x7fea180039b0 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.586+0000 7fea34c40640 1 -- 172.21.15.77:0/3545204643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea180039b0 msgr2=0x7fea18003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.586+0000 7fea34c40640 1 --2- 172.21.15.77:0/3545204643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea180039b0 0x7fea18003dd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fea2800e350 tx=0x7fea18005830 comp rx=0 tx=0).stop 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.586+0000 7fea2ed9d640 1 -- 172.21.15.77:0/3545204643 reap_dead start 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.586+0000 7fea34c40640 1 -- 172.21.15.77:0/3545204643 shutdown_connections 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.586+0000 7fea34c40640 1 -- 172.21.15.77:0/3545204643 >> 172.21.15.77:0/3545204643 conn(0x7fea3009a9f0 msgr2=0x7fea3009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.586+0000 7fea34c40640 1 -- 172.21.15.77:0/3545204643 shutdown_connections 2024-03-06T07:21:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.586+0000 7fea34c40640 1 -- 172.21.15.77:0/3545204643 wait complete. 2024-03-06T07:21:54.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.587+0000 7fea34c40640 1 Processor -- start 2024-03-06T07:21:54.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.587+0000 7fea34c40640 1 -- start start 2024-03-06T07:21:54.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea34c40640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea180039b0 0x7fea30154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea34c40640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea30154da0 0x7fea3015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea34c40640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea300cc8b0 0x7fea300ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea34c40640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea300d1b00 con 0x7fea300cc8b0 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea34c40640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea300d1980 con 0x7fea180039b0 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea34c40640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea300d1df0 con 0x7fea30154da0 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea300cc8b0 0x7fea300ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea300cc8b0 0x7fea300ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46712/0 (socket says 172.21.15.77:46712) 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2ed9d640 1 -- 172.21.15.77:0/498442757 learned_addr learned my addr 172.21.15.77:0/498442757 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2dd9b640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea30154da0 0x7fea3015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2e59c640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea180039b0 0x7fea30154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2ed9d640 1 -- 172.21.15.77:0/498442757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea30154da0 msgr2=0x7fea3015a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2ed9d640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea30154da0 0x7fea3015a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2ed9d640 1 -- 172.21.15.77:0/498442757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea180039b0 msgr2=0x7fea30154080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.588+0000 7fea2ed9d640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea180039b0 0x7fea30154080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.589+0000 7fea2ed9d640 1 -- 172.21.15.77:0/498442757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3015e5e0 con 0x7fea300cc8b0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.589+0000 7fea2ed9d640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea300cc8b0 0x7fea300ccd30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fea301618c0 tx=0x7fea20011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.589+0000 7fea1effd640 1 -- 172.21.15.77:0/498442757 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea20019020 con 0x7fea300cc8b0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.589+0000 7fea1effd640 1 -- 172.21.15.77:0/498442757 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea20004240 con 0x7fea300cc8b0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.589+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea300c5670 con 0x7fea300cc8b0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.589+0000 7fea1effd640 1 -- 172.21.15.77:0/498442757 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2000f740 con 0x7fea300cc8b0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.589+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea300c5910 con 0x7fea300cc8b0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.590+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3004f330 con 0x7fea300cc8b0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.590+0000 7fea1effd640 1 -- 172.21.15.77:0/498442757 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fea2001c020 con 0x7fea300cc8b0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.591+0000 7fea1effd640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea1006b440 0x7fea1006d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.591+0000 7fea2e59c640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea1006b440 0x7fea1006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.591+0000 7fea1effd640 1 -- 172.21.15.77:0/498442757 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4846+0+0 (secure 0 0 0) 0x7fea20089f60 con 0x7fea300cc8b0 2024-03-06T07:21:54.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.591+0000 7fea2e59c640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea1006b440 0x7fea1006d900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fea28036000 tx=0x7fea2802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.595+0000 7fea1effd640 1 -- 172.21.15.77:0/498442757 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea20023020 con 0x7fea300cc8b0 2024-03-06T07:21:54.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.806+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea300b1820 con 0x7fea300cc8b0 2024-03-06T07:21:54.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.806+0000 7fea1effd640 1 -- 172.21.15.77:0/498442757 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7fea3015e5e0 con 0x7fea300cc8b0 2024-03-06T07:21:54.809 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:54.809 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:21:54.809 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 3 2024-03-06T07:21:54.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea1006b440 msgr2=0x7fea1006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea1006b440 0x7fea1006d900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fea28036000 tx=0x7fea2802e000 comp rx=0 tx=0).stop 2024-03-06T07:21:54.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea300cc8b0 msgr2=0x7fea300ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea300cc8b0 0x7fea300ccd30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fea301618c0 tx=0x7fea20011cf0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 shutdown_connections 2024-03-06T07:21:54.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea1006b440 0x7fea1006d900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea300cc8b0 0x7fea300ccd30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea30154da0 0x7fea3015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 --2- 172.21.15.77:0/498442757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea180039b0 0x7fea30154080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 >> 172.21.15.77:0/498442757 conn(0x7fea3009a9f0 msgr2=0x7fea3009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 shutdown_connections 2024-03-06T07:21:54.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.809+0000 7fea34c40640 1 -- 172.21.15.77:0/498442757 wait complete. 2024-03-06T07:21:54.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:54.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.920+0000 7fbb05efb640 1 Processor -- start 2024-03-06T07:21:54.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.920+0000 7fbb05efb640 1 -- start start 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbb05efb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb0015bdd0 0x7fbb000c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbb05efb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015fa10 0x7fbb000ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbb05efb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbb000d16e0 0x7fbb000d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbb05efb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb000cebe0 con 0x7fbb0015bdd0 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbb05efb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb000ceec0 con 0x7fbb000d16e0 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbb05efb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000cf1a0 con 0x7fbb0015fa10 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbaff7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb0015bdd0 0x7fbb000c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbaff7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb0015bdd0 0x7fbb000c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46714/0 (socket says 172.21.15.77:46714) 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbaff7fe640 1 -- 172.21.15.77:0/347128183 learned_addr learned my addr 172.21.15.77:0/347128183 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:54.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbafffff640 1 --2- 172.21.15.77:0/347128183 >> v2:172.21.15.158:3300/0 conn(0x7fbb000d16e0 0x7fbb000d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbafeffd640 1 --2- 172.21.15.77:0/347128183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015fa10 0x7fbb000ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbaff7fe640 1 -- 172.21.15.77:0/347128183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015fa10 msgr2=0x7fbb000ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbaff7fe640 1 --2- 172.21.15.77:0/347128183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015fa10 0x7fbb000ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbaff7fe640 1 -- 172.21.15.77:0/347128183 >> v2:172.21.15.158:3300/0 conn(0x7fbb000d16e0 msgr2=0x7fbb000d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbaff7fe640 1 --2- 172.21.15.77:0/347128183 >> v2:172.21.15.158:3300/0 conn(0x7fbb000d16e0 0x7fbb000d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.921+0000 7fbaff7fe640 1 -- 172.21.15.77:0/347128183 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb000cc8e0 con 0x7fbb0015bdd0 2024-03-06T07:21:54.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.922+0000 7fbafeffd640 1 --2- 172.21.15.77:0/347128183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015fa10 0x7fbb000ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:54.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.922+0000 7fbaff7fe640 1 --2- 172.21.15.77:0/347128183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015bdd0 0x7fbb000c5610 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb000729e0 tx=0x7fbaf002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68e14f2d9ccdda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.923+0000 7fbafe7fc640 1 -- 172.21.15.77:0/347128183 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf002e070 con 0x7fbb0015bdd0 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.923+0000 7fbafe7fc640 1 -- 172.21.15.77:0/347128183 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf0037070 con 0x7fbb0015bdd0 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.923+0000 7fbafe7fc640 1 -- 172.21.15.77:0/347128183 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0031a00 con 0x7fbb0015bdd0 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 -- 172.21.15.77:0/347128183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015bdd0 msgr2=0x7fbb000c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 --2- 172.21.15.77:0/347128183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015bdd0 0x7fbb000c5610 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb000729e0 tx=0x7fbaf002b880 comp rx=0 tx=0).stop 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 -- 172.21.15.77:0/347128183 shutdown_connections 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 --2- 172.21.15.77:0/347128183 >> v2:172.21.15.158:3300/0 conn(0x7fbb000d16e0 0x7fbb000d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 --2- 172.21.15.77:0/347128183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015fa10 0x7fbb000ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 --2- 172.21.15.77:0/347128183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015bdd0 0x7fbb000c5610 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 -- 172.21.15.77:0/347128183 >> 172.21.15.77:0/347128183 conn(0x7fbb0009a9f0 msgr2=0x7fbb0009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 -- 172.21.15.77:0/347128183 shutdown_connections 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.924+0000 7fbb05efb640 1 -- 172.21.15.77:0/347128183 wait complete. 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.925+0000 7fbb05efb640 1 Processor -- start 2024-03-06T07:21:54.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.925+0000 7fbb05efb640 1 -- start start 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbb05efb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb000d16e0 0x7fbb00154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbb05efb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015bdd0 0x7fbb000ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbb05efb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015fa10 0x7fbb000caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbb05efb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000cc5d0 con 0x7fbb0015fa10 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbb05efb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000d2070 con 0x7fbb000d16e0 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbb05efb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0015c1b0 con 0x7fbb0015bdd0 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015fa10 0x7fbb000caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015fa10 0x7fbb000caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46726/0 (socket says 172.21.15.77:46726) 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 -- 172.21.15.77:0/3872097992 learned_addr learned my addr 172.21.15.77:0/3872097992 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbaff7fe640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb000d16e0 0x7fbb00154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafeffd640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015bdd0 0x7fbb000ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 -- 172.21.15.77:0/3872097992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015bdd0 msgr2=0x7fbb000ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015bdd0 0x7fbb000ca570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 -- 172.21.15.77:0/3872097992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb000d16e0 msgr2=0x7fbb00154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb000d16e0 0x7fbb00154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 -- 172.21.15.77:0/3872097992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0015f380 con 0x7fbb0015fa10 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbafffff640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015fa10 0x7fbb000caab0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800efc0 tx=0x7fbaf800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbae7fff640 1 -- 172.21.15.77:0/3872097992 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8018020 con 0x7fbb0015fa10 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbae7fff640 1 -- 172.21.15.77:0/3872097992 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8014480 con 0x7fbb0015fa10 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0015d680 con 0x7fbb0015fa10 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbae7fff640 1 -- 172.21.15.77:0/3872097992 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8014870 con 0x7fbb0015fa10 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.926+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb000c9d50 con 0x7fbb0015fa10 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.927+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0004f330 con 0x7fbb0015fa10 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.928+0000 7fbae7fff640 1 -- 172.21.15.77:0/3872097992 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fbaf8005d10 con 0x7fbb0015fa10 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.928+0000 7fbae7fff640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbaec06b440 0x7fbaec06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.928+0000 7fbaff7fe640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbaec06b440 0x7fbaec06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.928+0000 7fbae7fff640 1 -- 172.21.15.77:0/3872097992 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbaf808a5e0 con 0x7fbb0015fa10 2024-03-06T07:21:54.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.929+0000 7fbaff7fe640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbaec06b440 0x7fbaec06d900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0036000 tx=0x7fbaf003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:54.933+0000 7fbae7fff640 1 -- 172.21.15.77:0/3872097992 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaf8022070 con 0x7fbb0015fa10 2024-03-06T07:21:55.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.106+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb001602d0 con 0x7fbb0015fa10 2024-03-06T07:21:55.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.107+0000 7fbae7fff640 1 -- 172.21.15.77:0/3872097992 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12885 (secure 0 0 0) 0x7fbaf8010040 con 0x7fbb0015fa10 2024-03-06T07:21:55.109 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:55.109 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:54.915585+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.108+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbaec06b440 msgr2=0x7fbaec06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.108+0000 7fbb05efb640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbaec06b440 0x7fbaec06d900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0036000 tx=0x7fbaf003d040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.108+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015fa10 msgr2=0x7fbb000caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.108+0000 7fbb05efb640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015fa10 0x7fbb000caab0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800efc0 tx=0x7fbaf800c490 comp rx=0 tx=0).stop 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.109+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 shutdown_connections 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.109+0000 7fbb05efb640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbaec06b440 0x7fbaec06d900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.109+0000 7fbb05efb640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb0015fa10 0x7fbb000caab0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.109+0000 7fbb05efb640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb0015bdd0 0x7fbb000ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.109+0000 7fbb05efb640 1 --2- 172.21.15.77:0/3872097992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb000d16e0 0x7fbb00154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.109+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 >> 172.21.15.77:0/3872097992 conn(0x7fbb0009a9f0 msgr2=0x7fbb00070170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.109+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 shutdown_connections 2024-03-06T07:21:55.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.109+0000 7fbb05efb640 1 -- 172.21.15.77:0/3872097992 wait complete. 2024-03-06T07:21:55.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:55.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:55.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.223+0000 7f1499ced640 1 Processor -- start 2024-03-06T07:21:55.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.223+0000 7f1499ced640 1 -- start start 2024-03-06T07:21:55.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.223+0000 7f1499ced640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f149415ad90 0x7f14940b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.223+0000 7f1499ced640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149415b260 con 0x7f1494071a70 2024-03-06T07:21:55.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.223+0000 7f1499ced640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149415b540 con 0x7f14940b4110 2024-03-06T07:21:55.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.223+0000 7f1499ced640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149415c020 con 0x7f149415ad90 2024-03-06T07:21:55.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f14937fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1494071a70 0x7f1494071e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54480/0 (socket says 172.21.15.77:54480) 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f14937fe640 1 -- 172.21.15.77:0/3919266074 learned_addr learned my addr 172.21.15.77:0/3919266074 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176752535 0 0) 0x7f149415b260 con 0x7f1494071a70 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1480003520 con 0x7f1494071a70 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f148affd640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f149415ad90 0x7f14940b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1743334441 0 0) 0x7f1480003520 con 0x7f1494071a70 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f149415ad90 msgr2=0x7f14940b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f149415ad90 0x7f14940b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 >> v1:172.21.15.158:6789/0 conn(0x7f14940b4110 legacy=0x7f149415c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149415b820 con 0x7f1494071a70 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f1484001670 con 0x7f1494071a70 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 >> v1:172.21.15.77:6789/0 conn(0x7f1494071a70 legacy=0x7f1494071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14800039e0 0x7f1480003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f149415ad90 0x7f1480004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1480004d40 0x7f14800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149415c020 con 0x7f1480004d40 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148000bb20 con 0x7f14800039e0 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148000df30 con 0x7f149415ad90 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1493fff640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1480004d40 0x7f14800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f14937fe640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14800039e0 0x7f1480003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f148affd640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f149415ad90 0x7f1480004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1493fff640 1 -- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f149415ad90 msgr2=0x7f1480004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1493fff640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f149415ad90 0x7f1480004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1493fff640 1 -- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14800039e0 msgr2=0x7f1480003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1493fff640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14800039e0 0x7f1480003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.224+0000 7f1493fff640 1 -- 172.21.15.77:0/3919266074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1480003520 con 0x7f1480004d40 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.225+0000 7f1493fff640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1480004d40 0x7f14800051c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f148c001e50 tx=0x7f1480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce922d03034cf03f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.225+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c001a70 con 0x7f1480004d40 2024-03-06T07:21:55.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.225+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c003b70 con 0x7f1480004d40 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.225+0000 7f1492ffd640 1 -- 172.21.15.77:0/3919266074 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c0050d0 con 0x7f1480004d40 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.226+0000 7f1499ced640 1 -- 172.21.15.77:0/3919266074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1480004d40 msgr2=0x7f14800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.226+0000 7f1499ced640 1 --2- 172.21.15.77:0/3919266074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1480004d40 0x7f14800051c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f148c001e50 tx=0x7f1480005830 comp rx=0 tx=0).stop 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.226+0000 7f1493fff640 1 -- 172.21.15.77:0/3919266074 reap_dead start 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.226+0000 7f1499ced640 1 -- 172.21.15.77:0/3919266074 shutdown_connections 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.226+0000 7f1499ced640 1 -- 172.21.15.77:0/3919266074 >> 172.21.15.77:0/3919266074 conn(0x7f14940b2660 msgr2=0x7f14940b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.226+0000 7f1499ced640 1 -- 172.21.15.77:0/3919266074 shutdown_connections 2024-03-06T07:21:55.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.226+0000 7f1499ced640 1 -- 172.21.15.77:0/3919266074 wait complete. 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.227+0000 7f1499ced640 1 Processor -- start 2024-03-06T07:21:55.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.227+0000 7f1499ced640 1 -- start start 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1499ced640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1480004d40 0x7f1494141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1499ced640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14941e41c0 0x7f14940d0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1499ced640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14941e5460 0x7f14941e58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1499ced640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14940b46e0 con 0x7f14941e5460 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1499ced640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14940b4560 con 0x7f14941e41c0 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1499ced640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14940b3bd0 con 0x7f1480004d40 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14941e5460 0x7f14941e58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14941e5460 0x7f14941e58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46742/0 (socket says 172.21.15.77:46742) 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 -- 172.21.15.77:0/2794250749 learned_addr learned my addr 172.21.15.77:0/2794250749 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 -- 172.21.15.77:0/2794250749 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1480004d40 msgr2=0x7f1494141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1480004d40 0x7f1494141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 -- 172.21.15.77:0/2794250749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14941e41c0 msgr2=0x7f14940d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14941e41c0 0x7f14940d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 -- 172.21.15.77:0/2794250749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14941e4730 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1493fff640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14941e5460 0x7f14941e58e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f148c001f70 tx=0x7f148c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1490ff9640 1 -- 172.21.15.77:0/2794250749 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c007db0 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1490ff9640 1 -- 172.21.15.77:0/2794250749 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c003970 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14941e4a10 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.228+0000 7f1490ff9640 1 -- 172.21.15.77:0/2794250749 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c02f6a0 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.229+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14940ce0e0 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.230+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14940cd8f0 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.230+0000 7f1490ff9640 1 -- 172.21.15.77:0/2794250749 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f148c02f840 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.230+0000 7f1490ff9640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f147406b440 0x7f147406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.230+0000 7f1490ff9640 1 -- 172.21.15.77:0/2794250749 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f148c0ab470 con 0x7f14941e5460 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.230+0000 7f14937fe640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f147406b440 0x7f147406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.231+0000 7f14937fe640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f147406b440 0x7f147406d900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f14840011e0 tx=0x7f1484001be0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.235+0000 7f1490ff9640 1 -- 172.21.15.77:0/2794250749 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14940cd8f0 con 0x7f14941e5460 2024-03-06T07:21:55.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.404+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14940ce2e0 con 0x7f14941e5460 2024-03-06T07:21:55.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.404+0000 7f1490ff9640 1 -- 172.21.15.77:0/2794250749 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14940ce2e0 con 0x7f14941e5460 2024-03-06T07:21:55.406 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:21:55.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f147406b440 msgr2=0x7f147406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f147406b440 0x7f147406d900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f14840011e0 tx=0x7f1484001be0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14941e5460 msgr2=0x7f14941e58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14941e5460 0x7f14941e58e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f148c001f70 tx=0x7f148c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 shutdown_connections 2024-03-06T07:21:55.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f147406b440 0x7f147406d900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14941e5460 0x7f14941e58e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14941e41c0 0x7f14940d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 --2- 172.21.15.77:0/2794250749 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1480004d40 0x7f1494141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 >> 172.21.15.77:0/2794250749 conn(0x7f14940b2660 msgr2=0x7f1494159fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 shutdown_connections 2024-03-06T07:21:55.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.406+0000 7f1499ced640 1 -- 172.21.15.77:0/2794250749 wait complete. 2024-03-06T07:21:55.419 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:21:55.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.517+0000 7f1f64b2b640 1 Processor -- start 2024-03-06T07:21:55.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.517+0000 7f1f64b2b640 1 -- start start 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f64b2b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f600d0660 0x7f1f600cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f64b2b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f600cf850 0x7f1f600b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f64b2b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6015e4d0 con 0x7f1f6015e0f0 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f64b2b640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f600b1410 con 0x7f1f600d0660 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f64b2b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6006c8b0 con 0x7f1f600cf850 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f5e59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1f6015e0f0 0x7f1f600cff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54496/0 (socket says 172.21.15.77:54496) 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f5e59c640 1 -- 172.21.15.77:0/751354224 learned_addr learned my addr 172.21.15.77:0/751354224 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646493486 0 0) 0x7f1f6015e4d0 con 0x7f1f6015e0f0 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f4c003520 con 0x7f1f6015e0f0 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f5ed9d640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f600cf850 0x7f1f600b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.518+0000 7f1f5dd9b640 1 --2- 172.21.15.77:0/751354224 >> v2:172.21.15.158:3300/0 conn(0x7f1f600d0660 0x7f1f600cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4047992808 0 0) 0x7f1f4c003520 con 0x7f1f6015e0f0 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f600cf850 msgr2=0x7f1f600b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f600cf850 0x7f1f600b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 >> v2:172.21.15.158:3300/0 conn(0x7f1f600d0660 msgr2=0x7f1f600cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 --2- 172.21.15.77:0/751354224 >> v2:172.21.15.158:3300/0 conn(0x7f1f600d0660 0x7f1f600cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f600cdac0 con 0x7f1f6015e0f0 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f1f500019f0 con 0x7f1f6015e0f0 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 >> v1:172.21.15.77:6789/0 conn(0x7f1f6015e0f0 legacy=0x7f1f600cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f600cf850 0x7f1f4c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c003f30 0x7f1f4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6006c8b0 con 0x7f1f4c004d40 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c00b9c0 con 0x7f1f4c003f30 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c00bb40 con 0x7f1f600cf850 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5ed9d640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5e59c640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f600cf850 0x7f1f4c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5dd9b640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c003f30 0x7f1f4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5ed9d640 1 -- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f600cf850 msgr2=0x7f1f4c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5ed9d640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f600cf850 0x7f1f4c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5ed9d640 1 -- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c003f30 msgr2=0x7f1f4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5ed9d640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c003f30 0x7f1f4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5ed9d640 1 -- 172.21.15.77:0/751354224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c003520 con 0x7f1f4c004d40 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.519+0000 7f1f5dd9b640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f4c003f30 0x7f1f4c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.520+0000 7f1f5ed9d640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f4c0051c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009ac0 tx=0x7f1f4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bca0283a7c61b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.520+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58039070 con 0x7f1f4c004d40 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.520+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5802c6b0 con 0x7f1f4c004d40 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.520+0000 7f1f5d59a640 1 -- 172.21.15.77:0/751354224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58009690 con 0x7f1f4c004d40 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.520+0000 7f1f64b2b640 1 -- 172.21.15.77:0/751354224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 msgr2=0x7f1f4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.520+0000 7f1f64b2b640 1 --2- 172.21.15.77:0/751354224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f4c0051c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009ac0 tx=0x7f1f4c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.521+0000 7f1f5ed9d640 1 -- 172.21.15.77:0/751354224 reap_dead start 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.521+0000 7f1f64b2b640 1 -- 172.21.15.77:0/751354224 shutdown_connections 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.521+0000 7f1f64b2b640 1 -- 172.21.15.77:0/751354224 >> 172.21.15.77:0/751354224 conn(0x7f1f600b35a0 msgr2=0x7f1f600b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.521+0000 7f1f64b2b640 1 -- 172.21.15.77:0/751354224 shutdown_connections 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.521+0000 7f1f64b2b640 1 -- 172.21.15.77:0/751354224 wait complete. 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.522+0000 7f1f64b2b640 1 Processor -- start 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.522+0000 7f1f64b2b640 1 -- start start 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f64b2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f600b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f64b2b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6015be00 0x7f1f600ce780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f64b2b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f6006cfb0 0x7f1f6009cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f64b2b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600d0cc0 con 0x7f1f4c004d40 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f64b2b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6006c540 con 0x7f1f6015be00 2024-03-06T07:21:55.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f64b2b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600b25b0 con 0x7f1f6006cfb0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f600b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f600b5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46772/0 (socket says 172.21.15.77:46772) 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 -- 172.21.15.77:0/4192029711 learned_addr learned my addr 172.21.15.77:0/4192029711 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5dd9b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6015be00 0x7f1f600ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5ed9d640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f6006cfb0 0x7f1f6009cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 -- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f6006cfb0 msgr2=0x7f1f6009cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f6006cfb0 0x7f1f6009cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 -- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6015be00 msgr2=0x7f1f600ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6015be00 0x7f1f600ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 -- 172.21.15.77:0/4192029711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f600cf060 con 0x7f1f4c004d40 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5dd9b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6015be00 0x7f1f600ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f5e59c640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f600b5e40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1f50007a50 tx=0x7f1f5000a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f56ffd640 1 -- 172.21.15.77:0/4192029711 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50017020 con 0x7f1f4c004d40 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f56ffd640 1 -- 172.21.15.77:0/4192029711 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5000ae40 con 0x7f1f4c004d40 2024-03-06T07:21:55.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f56ffd640 1 -- 172.21.15.77:0/4192029711 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50013840 con 0x7f1f4c004d40 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f600cf340 con 0x7f1f4c004d40 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.523+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f600b5660 con 0x7f1f4c004d40 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.524+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6004f330 con 0x7f1f4c004d40 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.525+0000 7f1f56ffd640 1 -- 172.21.15.77:0/4192029711 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f1f5001a020 con 0x7f1f4c004d40 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.525+0000 7f1f56ffd640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f4006b470 0x7f1f4006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.525+0000 7f1f5dd9b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f4006b470 0x7f1f4006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.525+0000 7f1f56ffd640 1 -- 172.21.15.77:0/4192029711 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1f5008aa90 con 0x7f1f4c004d40 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.526+0000 7f1f5dd9b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f4006b470 0x7f1f4006d930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f600b0c30 tx=0x7f1f48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.530+0000 7f1f56ffd640 1 -- 172.21.15.77:0/4192029711 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f50021020 con 0x7f1f4c004d40 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.718+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1f600cd560 con 0x7f1f4c004d40 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.718+0000 7f1f56ffd640 1 -- 172.21.15.77:0/4192029711 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1f50059d40 con 0x7f1f4c004d40 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:21:55.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f4006b470 msgr2=0x7f1f4006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f4006b470 0x7f1f4006d930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f600b0c30 tx=0x7f1f48008040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 msgr2=0x7f1f600b5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f600b5e40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1f50007a50 tx=0x7f1f5000a790 comp rx=0 tx=0).stop 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 shutdown_connections 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f6006cfb0 0x7f1f6009cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f6015be00 0x7f1f600ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f4006b470 0x7f1f4006d930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 --2- 172.21.15.77:0/4192029711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f4c004d40 0x7f1f600b5e40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 >> 172.21.15.77:0/4192029711 conn(0x7f1f600b35a0 msgr2=0x7f1f6006f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 shutdown_connections 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.720+0000 7f1f64b2b640 1 -- 172.21.15.77:0/4192029711 wait complete. 2024-03-06T07:21:55.733 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:21:55.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.839+0000 7fd09ed5e640 1 Processor -- start 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.839+0000 7fd09ed5e640 1 -- start start 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.840+0000 7fd09ed5e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd09815de10 0x7fd0980c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.840+0000 7fd09ed5e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd09806b460 0x7fd09806b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.840+0000 7fd09ed5e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0980cf8b0 con 0x7fd0980c5520 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.840+0000 7fd09ed5e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd098073420 con 0x7fd09806b460 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.840+0000 7fd09ed5e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0980cfee0 con 0x7fd09815de10 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.840+0000 7fd09d2fb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd09806b460 0x7fd09806b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.840+0000 7fd09d2fb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd09806b460 0x7fd09806b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46794/0 (socket says 172.21.15.77:46794) 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.840+0000 7fd09d2fb640 1 -- 172.21.15.77:0/990279103 learned_addr learned my addr 172.21.15.77:0/990279103 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.841+0000 7fd09d2fb640 1 -- 172.21.15.77:0/990279103 >> v2:172.21.15.158:3300/0 conn(0x7fd09815de10 msgr2=0x7fd0980c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.841+0000 7fd09d2fb640 1 --2- 172.21.15.77:0/990279103 >> v2:172.21.15.158:3300/0 conn(0x7fd09815de10 0x7fd0980c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.841+0000 7fd09d2fb640 1 -- 172.21.15.77:0/990279103 >> v1:172.21.15.158:6789/0 conn(0x7fd0980c5520 legacy=0x7fd0980d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.841+0000 7fd09d2fb640 1 -- 172.21.15.77:0/990279103 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09806c1a0 con 0x7fd09806b460 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.841+0000 7fd09d2fb640 1 --2- 172.21.15.77:0/990279103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd09806b460 0x7fd09806b840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd09806fa60 tx=0x7fd09002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a2b0c5a37d84f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.841+0000 7fd08f7fe640 1 -- 172.21.15.77:0/990279103 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09002cb20 con 0x7fd09806b460 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.841+0000 7fd08f7fe640 1 -- 172.21.15.77:0/990279103 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09002ccc0 con 0x7fd09806b460 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.841+0000 7fd08f7fe640 1 -- 172.21.15.77:0/990279103 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090034610 con 0x7fd09806b460 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.842+0000 7fd09ed5e640 1 -- 172.21.15.77:0/990279103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd09806b460 msgr2=0x7fd09806b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.842+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/990279103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd09806b460 0x7fd09806b840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd09806fa60 tx=0x7fd09002bf80 comp rx=0 tx=0).stop 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.842+0000 7fd09ed5e640 1 -- 172.21.15.77:0/990279103 shutdown_connections 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.842+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/990279103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd09806b460 0x7fd09806b840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.842+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/990279103 >> v2:172.21.15.158:3300/0 conn(0x7fd09815de10 0x7fd0980c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.842+0000 7fd09ed5e640 1 -- 172.21.15.77:0/990279103 >> 172.21.15.77:0/990279103 conn(0x7fd0980725f0 msgr2=0x7fd098072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.842+0000 7fd09ed5e640 1 -- 172.21.15.77:0/990279103 shutdown_connections 2024-03-06T07:21:55.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.842+0000 7fd09ed5e640 1 -- 172.21.15.77:0/990279103 wait complete. 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.844+0000 7fd09ed5e640 1 Processor -- start 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.844+0000 7fd09ed5e640 1 -- start start 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd09ed5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd09806b460 0x7fd098066db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd09ed5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980c5520 0x7fd09815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd09ed5e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815de10 0x7fd09815c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd09ed5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd098068a20 con 0x7fd0980c5520 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd09ed5e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09806cd00 con 0x7fd09815de10 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd09ed5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd098068ba0 con 0x7fd09806b460 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980c5520 0x7fd09815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980c5520 0x7fd09815c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46808/0 (socket says 172.21.15.77:46808) 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 -- 172.21.15.77:0/898738089 learned_addr learned my addr 172.21.15.77:0/898738089 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd09d2fb640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815de10 0x7fd09815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd09cafa640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd09806b460 0x7fd098066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 -- 172.21.15.77:0/898738089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd09806b460 msgr2=0x7fd098066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd09806b460 0x7fd098066db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 -- 172.21.15.77:0/898738089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815de10 msgr2=0x7fd09815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815de10 0x7fd09815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 -- 172.21.15.77:0/898738089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0981f5450 con 0x7fd0980c5520 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.845+0000 7fd08ffff640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980c5520 0x7fd09815c2c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c410 tx=0x7fd08800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.846+0000 7fd08d7fa640 1 -- 172.21.15.77:0/898738089 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd088017070 con 0x7fd0980c5520 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.846+0000 7fd08d7fa640 1 -- 172.21.15.77:0/898738089 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0880035f0 con 0x7fd0980c5520 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.846+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0981f5730 con 0x7fd0980c5520 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.846+0000 7fd08d7fa640 1 -- 172.21.15.77:0/898738089 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd088003a20 con 0x7fd0980c5520 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.846+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0981f5cf0 con 0x7fd0980c5520 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.847+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09804f330 con 0x7fd0980c5520 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.847+0000 7fd08d7fa640 1 -- 172.21.15.77:0/898738089 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fd08801a070 con 0x7fd0980c5520 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.847+0000 7fd08d7fa640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd08406b440 0x7fd08406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.848+0000 7fd08d7fa640 1 -- 172.21.15.77:0/898738089 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd088089170 con 0x7fd0980c5520 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.852+0000 7fd08d7fa640 1 -- 172.21.15.77:0/898738089 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0880010c0 con 0x7fd0980c5520 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.856+0000 7fd09cafa640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd08406b440 0x7fd08406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:55.856+0000 7fd09cafa640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd08406b440 0x7fd08406d900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd0940098e0 tx=0x7fd094009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.039+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd0980d1050 con 0x7fd0980c5520 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.930+0000 7fd08d7fa640 1 -- 172.21.15.77:0/898738089 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd088057dd0 con 0x7fd0980c5520 2024-03-06T07:21:56.932 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:21:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd08406b440 msgr2=0x7fd08406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd08406b440 0x7fd08406d900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd0940098e0 tx=0x7fd094009060 comp rx=0 tx=0).stop 2024-03-06T07:21:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980c5520 msgr2=0x7fd09815c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980c5520 0x7fd09815c2c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c410 tx=0x7fd08800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 shutdown_connections 2024-03-06T07:21:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd08406b440 0x7fd08406d900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980c5520 0x7fd09815c2c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815de10 0x7fd09815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 --2- 172.21.15.77:0/898738089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd09806b460 0x7fd098066db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 >> 172.21.15.77:0/898738089 conn(0x7fd0980725f0 msgr2=0x7fd0980d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 shutdown_connections 2024-03-06T07:21:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:56.933+0000 7fd09ed5e640 1 -- 172.21.15.77:0/898738089 wait complete. 2024-03-06T07:21:56.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:21:57.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.054+0000 7f2be97dd640 1 Processor -- start 2024-03-06T07:21:57.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.054+0000 7f2be97dd640 1 -- start start 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be97dd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2be41614d0 0x7f2be40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be97dd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be97dd640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be41618b0 con 0x7f2be40d1910 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be97dd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be40cf400 con 0x7f2be41614d0 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be97dd640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40cf6a0 con 0x7f2be40d24b0 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be2ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2be41614d0 0x7f2be40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be2ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2be41614d0 0x7f2be40d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46822/0 (socket says 172.21.15.77:46822) 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be2ffd640 1 -- 172.21.15.77:0/395640316 learned_addr learned my addr 172.21.15.77:0/395640316 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be37fe640 1 --2- 172.21.15.77:0/395640316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be2ffd640 1 -- 172.21.15.77:0/395640316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 msgr2=0x7f2be40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be2ffd640 1 --2- 172.21.15.77:0/395640316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be2ffd640 1 -- 172.21.15.77:0/395640316 >> v1:172.21.15.158:6789/0 conn(0x7f2be40d1910 legacy=0x7f2be40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be2ffd640 1 -- 172.21.15.77:0/395640316 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40cf980 con 0x7f2be41614d0 2024-03-06T07:21:57.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.055+0000 7f2be37fe640 1 --2- 172.21.15.77:0/395640316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:57.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.056+0000 7f2be2ffd640 1 --2- 172.21.15.77:0/395640316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40d4860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009870 tx=0x7f2bd402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30166e966113afa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.056+0000 7f2be1ffb640 1 -- 172.21.15.77:0/395640316 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4034070 con 0x7f2be41614d0 2024-03-06T07:21:57.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.056+0000 7f2be1ffb640 1 -- 172.21.15.77:0/395640316 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd4030840 con 0x7f2be41614d0 2024-03-06T07:21:57.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.056+0000 7f2be1ffb640 1 -- 172.21.15.77:0/395640316 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4030c10 con 0x7f2be41614d0 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.057+0000 7f2be97dd640 1 -- 172.21.15.77:0/395640316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 msgr2=0x7f2be40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.057+0000 7f2be97dd640 1 --2- 172.21.15.77:0/395640316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40d4860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009870 tx=0x7f2bd402b980 comp rx=0 tx=0).stop 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.057+0000 7f2be97dd640 1 -- 172.21.15.77:0/395640316 shutdown_connections 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.057+0000 7f2be97dd640 1 --2- 172.21.15.77:0/395640316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.057+0000 7f2be97dd640 1 --2- 172.21.15.77:0/395640316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40d4860 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.057+0000 7f2be97dd640 1 -- 172.21.15.77:0/395640316 >> 172.21.15.77:0/395640316 conn(0x7f2be40d3d40 msgr2=0x7f2be40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.057+0000 7f2be97dd640 1 -- 172.21.15.77:0/395640316 shutdown_connections 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.057+0000 7f2be97dd640 1 -- 172.21.15.77:0/395640316 wait complete. 2024-03-06T07:21:57.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.058+0000 7f2be97dd640 1 Processor -- start 2024-03-06T07:21:57.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.058+0000 7f2be97dd640 1 -- start start 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be97dd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2be40d1910 0x7f2be415f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be97dd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be97dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be97dd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40b1370 con 0x7f2be41614d0 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be97dd640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40b0910 con 0x7f2be40d1910 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be97dd640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40b14f0 con 0x7f2be40d24b0 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40b4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46830/0 (socket says 172.21.15.77:46830) 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 -- 172.21.15.77:0/2004276806 learned_addr learned my addr 172.21.15.77:0/2004276806 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be2ffd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2be40d1910 0x7f2be415f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be27fc640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 -- 172.21.15.77:0/2004276806 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 msgr2=0x7f2be40b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 -- 172.21.15.77:0/2004276806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2be40d1910 msgr2=0x7f2be415f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2be40d1910 0x7f2be415f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 -- 172.21.15.77:0/2004276806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40cc4e0 con 0x7f2be41614d0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be37fe640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40b4db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009fa0 tx=0x7f2bdc00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2bcb7fe640 1 -- 172.21.15.77:0/2004276806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc01a070 con 0x7f2be41614d0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2bcb7fe640 1 -- 172.21.15.77:0/2004276806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc003bb0 con 0x7f2be41614d0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be40cc7c0 con 0x7f2be41614d0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2bcb7fe640 1 -- 172.21.15.77:0/2004276806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc004030 con 0x7f2be41614d0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.059+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be40ccd80 con 0x7f2be41614d0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.060+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be4068ff0 con 0x7f2be41614d0 2024-03-06T07:21:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.061+0000 7f2bcb7fe640 1 -- 172.21.15.77:0/2004276806 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f2bdc01d070 con 0x7f2be41614d0 2024-03-06T07:21:57.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.061+0000 7f2bcb7fe640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bd006b440 0x7f2bd006d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.061+0000 7f2be2ffd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bd006b440 0x7f2bd006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.061+0000 7f2bcb7fe640 1 -- 172.21.15.77:0/2004276806 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2bdc08a100 con 0x7f2be41614d0 2024-03-06T07:21:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.062+0000 7f2be2ffd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bd006b440 0x7f2bd006d900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4036000 tx=0x7f2bd403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.066+0000 7f2bcb7fe640 1 -- 172.21.15.77:0/2004276806 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bdc010070 con 0x7f2be41614d0 2024-03-06T07:21:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.237+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2be40d0500 con 0x7f2be41614d0 2024-03-06T07:21:57.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.936+0000 7f2bcb7fe640 1 -- 172.21.15.77:0/2004276806 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2bdc058c80 con 0x7f2be41614d0 2024-03-06T07:21:57.938 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:21:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bd006b440 msgr2=0x7f2bd006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bd006b440 0x7f2bd006d900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4036000 tx=0x7f2bd403b040 comp rx=0 tx=0).stop 2024-03-06T07:21:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 msgr2=0x7f2be40b4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40b4db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009fa0 tx=0x7f2bdc00d560 comp rx=0 tx=0).stop 2024-03-06T07:21:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 shutdown_connections 2024-03-06T07:21:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bd006b440 0x7f2bd006d900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be41614d0 0x7f2be40b4db0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2be40d24b0 0x7f2be40b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 --2- 172.21.15.77:0/2004276806 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2be40d1910 0x7f2be415f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 >> 172.21.15.77:0/2004276806 conn(0x7f2be40d3d40 msgr2=0x7f2be40d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 shutdown_connections 2024-03-06T07:21:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:57.939+0000 7f2be97dd640 1 -- 172.21.15.77:0/2004276806 wait complete. 2024-03-06T07:21:57.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:21:58.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.063+0000 7fe8b935e640 1 Processor -- start 2024-03-06T07:21:58.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.063+0000 7fe8b935e640 1 -- start start 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b935e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8b415e0b0 0x7fe8b40b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b935e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe8b40b3df0 0x7fe8b40d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b935e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b40b4290 con 0x7fe8b40d28c0 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b935e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b40ce990 con 0x7fe8b415e0b0 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b935e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b40b4570 con 0x7fe8b40b3df0 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b2ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8b415e0b0 0x7fe8b40b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe8b40d28c0 0x7fe8b40d0790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54500/0 (socket says 172.21.15.77:54500) 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b37fe640 1 -- 172.21.15.77:0/640949318 learned_addr learned my addr 172.21.15.77:0/640949318 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8abfff640 1 --2- 172.21.15.77:0/640949318 >> v2:172.21.15.158:3300/0 conn(0x7fe8b40b3df0 0x7fe8b40d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b2ffd640 1 -- 172.21.15.77:0/640949318 >> v2:172.21.15.158:3300/0 conn(0x7fe8b40b3df0 msgr2=0x7fe8b40d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b2ffd640 1 --2- 172.21.15.77:0/640949318 >> v2:172.21.15.158:3300/0 conn(0x7fe8b40b3df0 0x7fe8b40d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b2ffd640 1 -- 172.21.15.77:0/640949318 >> v1:172.21.15.77:6789/0 conn(0x7fe8b40d28c0 legacy=0x7fe8b40d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b27fc640 1 -- 172.21.15.77:0/640949318 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884026076 0 0) 0x7fe8b40b4290 con 0x7fe8b40d28c0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.064+0000 7fe8b2ffd640 1 -- 172.21.15.77:0/640949318 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b40b4850 con 0x7fe8b415e0b0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.065+0000 7fe8b2ffd640 1 --2- 172.21.15.77:0/640949318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b415e0b0 0x7fe8b40b38b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8b408dbc0 tx=0x7fe8a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59671ef31ca6ef31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.065+0000 7fe8b27fc640 1 -- 172.21.15.77:0/640949318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4035020 con 0x7fe8b415e0b0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.065+0000 7fe8b27fc640 1 -- 172.21.15.77:0/640949318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a4031830 con 0x7fe8b415e0b0 2024-03-06T07:21:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.065+0000 7fe8b27fc640 1 -- 172.21.15.77:0/640949318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4031c00 con 0x7fe8b415e0b0 2024-03-06T07:21:58.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.066+0000 7fe8b935e640 1 -- 172.21.15.77:0/640949318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b415e0b0 msgr2=0x7fe8b40b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.066+0000 7fe8b935e640 1 --2- 172.21.15.77:0/640949318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b415e0b0 0x7fe8b40b38b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8b408dbc0 tx=0x7fe8a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.066+0000 7fe8b935e640 1 -- 172.21.15.77:0/640949318 shutdown_connections 2024-03-06T07:21:58.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.066+0000 7fe8b935e640 1 --2- 172.21.15.77:0/640949318 >> v2:172.21.15.158:3300/0 conn(0x7fe8b40b3df0 0x7fe8b40d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.066+0000 7fe8b935e640 1 --2- 172.21.15.77:0/640949318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b415e0b0 0x7fe8b40b38b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.066+0000 7fe8b935e640 1 -- 172.21.15.77:0/640949318 >> 172.21.15.77:0/640949318 conn(0x7fe8b408d6f0 msgr2=0x7fe8b409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.066+0000 7fe8b935e640 1 -- 172.21.15.77:0/640949318 shutdown_connections 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.066+0000 7fe8b935e640 1 -- 172.21.15.77:0/640949318 wait complete. 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.067+0000 7fe8b935e640 1 Processor -- start 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.067+0000 7fe8b935e640 1 -- start start 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b935e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b40b3df0 0x7fe8b41511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b935e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8b40d28c0 0x7fe8b40b9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b935e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8b415e0b0 0x7fe8b40b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b935e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40d1cd0 con 0x7fe8b40b3df0 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b935e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b415e490 con 0x7fe8b40d28c0 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b935e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40d1e50 con 0x7fe8b415e0b0 2024-03-06T07:21:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b40b3df0 0x7fe8b41511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b40b3df0 0x7fe8b41511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46860/0 (socket says 172.21.15.77:46860) 2024-03-06T07:21:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 -- 172.21.15.77:0/2967873063 learned_addr learned my addr 172.21.15.77:0/2967873063 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8abfff640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8b40d28c0 0x7fe8b40b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b37fe640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8b415e0b0 0x7fe8b40b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 -- 172.21.15.77:0/2967873063 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8b415e0b0 msgr2=0x7fe8b40b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8b415e0b0 0x7fe8b40b6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 -- 172.21.15.77:0/2967873063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8b40d28c0 msgr2=0x7fe8b40b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8b40d28c0 0x7fe8b40b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 -- 172.21.15.77:0/2967873063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b41601a0 con 0x7fe8b40b3df0 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b2ffd640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b40b3df0 0x7fe8b41511a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4000c00 tx=0x7fe8a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8ab7fe640 1 -- 172.21.15.77:0/2967873063 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4040070 con 0x7fe8b40b3df0 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8ab7fe640 1 -- 172.21.15.77:0/2967873063 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a4039a20 con 0x7fe8b40b3df0 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b4160480 con 0x7fe8b40b3df0 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8ab7fe640 1 -- 172.21.15.77:0/2967873063 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4035040 con 0x7fe8b40b3df0 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.068+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b4160a40 con 0x7fe8b40b3df0 2024-03-06T07:21:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.069+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b404f330 con 0x7fe8b40b3df0 2024-03-06T07:21:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.070+0000 7fe8ab7fe640 1 -- 172.21.15.77:0/2967873063 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8a404b060 con 0x7fe8b40b3df0 2024-03-06T07:21:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.070+0000 7fe8ab7fe640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe89406b4d0 0x7fe89406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.070+0000 7fe8ab7fe640 1 -- 172.21.15.77:0/2967873063 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe8a40abaa0 con 0x7fe8b40b3df0 2024-03-06T07:21:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.070+0000 7fe8abfff640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe89406b4d0 0x7fe89406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.070+0000 7fe8abfff640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe89406b4d0 0x7fe89406d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4161660 tx=0x7fe89c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.074+0000 7fe8ab7fe640 1 -- 172.21.15.77:0/2967873063 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8a407a980 con 0x7fe8b40b3df0 2024-03-06T07:21:58.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.284+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe8b40d2160 con 0x7fe8b40b3df0 2024-03-06T07:21:58.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.938+0000 7fe8ab7fe640 1 -- 172.21.15.77:0/2967873063 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe8a407a280 con 0x7fe8b40b3df0 2024-03-06T07:21:58.940 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:21:58.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.940+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe89406b4d0 msgr2=0x7fe89406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.940+0000 7fe8b935e640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe89406b4d0 0x7fe89406d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4161660 tx=0x7fe89c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.941+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b40b3df0 msgr2=0x7fe8b41511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.941+0000 7fe8b935e640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b40b3df0 0x7fe8b41511a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4000c00 tx=0x7fe8a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.942+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 shutdown_connections 2024-03-06T07:21:58.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.942+0000 7fe8b935e640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8b415e0b0 0x7fe8b40b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.942+0000 7fe8b935e640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8b40d28c0 0x7fe8b40b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.942+0000 7fe8b935e640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe89406b4d0 0x7fe89406d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.942+0000 7fe8b935e640 1 --2- 172.21.15.77:0/2967873063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b40b3df0 0x7fe8b41511a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.942+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 >> 172.21.15.77:0/2967873063 conn(0x7fe8b408d6f0 msgr2=0x7fe8b409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.942+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 shutdown_connections 2024-03-06T07:21:58.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:58.942+0000 7fe8b935e640 1 -- 172.21.15.77:0/2967873063 wait complete. 2024-03-06T07:21:58.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:59.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.075+0000 7f0a01c8d640 1 Processor -- start 2024-03-06T07:21:59.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.075+0000 7f0a01c8d640 1 -- start start 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f0a01c8d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f09fc0b4f70 0x7f09fc0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f0a01c8d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc06c6d0 0x7f09fc15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f0a01c8d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f09fc066d90 0x7f09fc0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f0a01c8d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc15e9f0 con 0x7f09fc0b4f70 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f0a01c8d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc0cda10 con 0x7f09fc066d90 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f0a01c8d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0b5890 con 0x7f09fc06c6d0 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fb7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f09fc0b4f70 0x7f09fc0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fb7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f09fc0b4f70 0x7f09fc0b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41122/0 (socket says 172.21.15.77:41122) 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fb7fe640 1 -- 172.21.15.77:0/478402579 learned_addr learned my addr 172.21.15.77:0/478402579 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fbfff640 1 --2- 172.21.15.77:0/478402579 >> v2:172.21.15.158:3300/0 conn(0x7f09fc066d90 0x7f09fc0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fb7fe640 1 -- 172.21.15.77:0/478402579 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc06c6d0 msgr2=0x7f09fc15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09faffd640 1 --2- 172.21.15.77:0/478402579 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc06c6d0 0x7f09fc15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fb7fe640 1 --2- 172.21.15.77:0/478402579 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc06c6d0 0x7f09fc15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fb7fe640 1 -- 172.21.15.77:0/478402579 >> v2:172.21.15.158:3300/0 conn(0x7f09fc066d90 msgr2=0x7f09fc0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fb7fe640 1 --2- 172.21.15.77:0/478402579 >> v2:172.21.15.158:3300/0 conn(0x7f09fc066d90 0x7f09fc0ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fb7fe640 1 -- 172.21.15.77:0/478402579 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0cd450 con 0x7f09fc0b4f70 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09faffd640 1 --2- 172.21.15.77:0/478402579 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc06c6d0 0x7f09fc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.076+0000 7f09fbfff640 1 --2- 172.21.15.77:0/478402579 >> v2:172.21.15.158:3300/0 conn(0x7f09fc066d90 0x7f09fc0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.077+0000 7f09fb7fe640 1 --2- 172.21.15.77:0/478402579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc0b4f70 0x7f09fc0b5350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0723c0 tx=0x7f09ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f21fb05e3491082 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.077+0000 7f09fa7fc640 1 -- 172.21.15.77:0/478402579 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec035020 con 0x7f09fc0b4f70 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.077+0000 7f09fa7fc640 1 -- 172.21.15.77:0/478402579 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec031870 con 0x7f09fc0b4f70 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.077+0000 7f09fa7fc640 1 -- 172.21.15.77:0/478402579 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec031c40 con 0x7f09fc0b4f70 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 -- 172.21.15.77:0/478402579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc0b4f70 msgr2=0x7f09fc0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/478402579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc0b4f70 0x7f09fc0b5350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0723c0 tx=0x7f09ec02b910 comp rx=0 tx=0).stop 2024-03-06T07:21:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 -- 172.21.15.77:0/478402579 shutdown_connections 2024-03-06T07:21:59.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/478402579 >> v2:172.21.15.158:3300/0 conn(0x7f09fc066d90 0x7f09fc0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/478402579 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc06c6d0 0x7f09fc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/478402579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc0b4f70 0x7f09fc0b5350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 -- 172.21.15.77:0/478402579 >> 172.21.15.77:0/478402579 conn(0x7f09fc0724d0 msgr2=0x7f09fc0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 -- 172.21.15.77:0/478402579 shutdown_connections 2024-03-06T07:21:59.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.078+0000 7f0a01c8d640 1 -- 172.21.15.77:0/478402579 wait complete. 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.079+0000 7f0a01c8d640 1 Processor -- start 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.079+0000 7f0a01c8d640 1 -- start start 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f0a01c8d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09fc066d90 0x7f09fc15d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f0a01c8d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc0b4f70 0x7f09fc15d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f0a01c8d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc153230 0x7f09fc15c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f0a01c8d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc15f3c0 con 0x7f09fc153230 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f0a01c8d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc15e680 con 0x7f09fc066d90 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f0a01c8d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0cd090 con 0x7f09fc0b4f70 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc153230 0x7f09fc15c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc153230 0x7f09fc15c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41136/0 (socket says 172.21.15.77:41136) 2024-03-06T07:21:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 -- 172.21.15.77:0/2814487569 learned_addr learned my addr 172.21.15.77:0/2814487569 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 -- 172.21.15.77:0/2814487569 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc0b4f70 msgr2=0x7f09fc15d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc0b4f70 0x7f09fc15d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 -- 172.21.15.77:0/2814487569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09fc066d90 msgr2=0x7f09fc15d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fb7fe640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09fc066d90 0x7f09fc15d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09fc066d90 0x7f09fc15d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 -- 172.21.15.77:0/2814487569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc15de90 con 0x7f09fc153230 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fb7fe640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09fc066d90 0x7f09fc15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09fbfff640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc153230 0x7f09fc15c720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f09f400cca0 tx=0x7f09f400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09e3fff640 1 -- 172.21.15.77:0/2814487569 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f400f680 con 0x7f09fc153230 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f09e3fff640 1 -- 172.21.15.77:0/2814487569 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f400f840 con 0x7f09fc153230 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.080+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09fc156e80 con 0x7f09fc153230 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.081+0000 7f09e3fff640 1 -- 172.21.15.77:0/2814487569 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f400fc30 con 0x7f09fc153230 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.081+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc157440 con 0x7f09fc153230 2024-03-06T07:21:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.081+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc04f330 con 0x7f09fc153230 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.082+0000 7f09e3fff640 1 -- 172.21.15.77:0/2814487569 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09f40213f0 con 0x7f09fc153230 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.082+0000 7f09e3fff640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09e806b4d0 0x7f09e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.082+0000 7f09e3fff640 1 -- 172.21.15.77:0/2814487569 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09f408af10 con 0x7f09fc153230 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.082+0000 7f09fb7fe640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09e806b4d0 0x7f09e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.083+0000 7f09fb7fe640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09e806b4d0 0x7f09e806d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f09ec036000 tx=0x7f09ec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.087+0000 7f09e3fff640 1 -- 172.21.15.77:0/2814487569 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09f4014030 con 0x7f09fc153230 2024-03-06T07:21:59.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:59.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:59.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.263+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09fc0ca140 con 0x7f09fc153230 2024-03-06T07:21:59.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.264+0000 7f09e3fff640 1 -- 172.21.15.77:0/2814487569 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16606 (secure 0 0 0) 0x7f09f4059d90 con 0x7f09fc153230 2024-03-06T07:21:59.266 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:59.266 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:58.285580+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09e806b4d0 msgr2=0x7f09e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09e806b4d0 0x7f09e806d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f09ec036000 tx=0x7f09ec008030 comp rx=0 tx=0).stop 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc153230 msgr2=0x7f09fc15c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc153230 0x7f09fc15c720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f09f400cca0 tx=0x7f09f400a650 comp rx=0 tx=0).stop 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 shutdown_connections 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09e806b4d0 0x7f09e806d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09fc153230 0x7f09fc15c720 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09fc0b4f70 0x7f09fc15d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 --2- 172.21.15.77:0/2814487569 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09fc066d90 0x7f09fc15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 >> 172.21.15.77:0/2814487569 conn(0x7f09fc0724d0 msgr2=0x7f09fc0ca770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 shutdown_connections 2024-03-06T07:21:59.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.266+0000 7f0a01c8d640 1 -- 172.21.15.77:0/2814487569 wait complete. 2024-03-06T07:21:59.279 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:59.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.376+0000 7f4637eed640 1 Processor -- start 2024-03-06T07:21:59.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f4637eed640 1 -- start start 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f4637eed640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f463015dfe0 0x7f46300cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f4637eed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300cf830 0x7f46300d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f4637eed640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f46300d27f0 0x7f46300d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f4637eed640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46300d2c50 con 0x7f46300d27f0 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f4637eed640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46300b31e0 con 0x7f463015dfe0 2024-03-06T07:21:59.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f4637eed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46300b3d60 con 0x7f46300cf830 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f463648a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f46300d27f0 0x7f46300d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f463648a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f46300d27f0 0x7f46300d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41140/0 (socket says 172.21.15.77:41140) 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.377+0000 7f463648a640 1 -- 172.21.15.77:0/3236924825 learned_addr learned my addr 172.21.15.77:0/3236924825 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f4635c89640 1 --2- 172.21.15.77:0/3236924825 >> v2:172.21.15.158:3300/0 conn(0x7f463015dfe0 0x7f46300cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f4635488640 1 --2- 172.21.15.77:0/3236924825 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300cf830 0x7f46300d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f463648a640 1 -- 172.21.15.77:0/3236924825 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300cf830 msgr2=0x7f46300d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f463648a640 1 --2- 172.21.15.77:0/3236924825 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300cf830 0x7f46300d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f463648a640 1 -- 172.21.15.77:0/3236924825 >> v2:172.21.15.158:3300/0 conn(0x7f463015dfe0 msgr2=0x7f46300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f463648a640 1 --2- 172.21.15.77:0/3236924825 >> v2:172.21.15.158:3300/0 conn(0x7f463015dfe0 0x7f46300cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f463648a640 1 -- 172.21.15.77:0/3236924825 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300b3430 con 0x7f46300d27f0 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f4635c89640 1 --2- 172.21.15.77:0/3236924825 >> v2:172.21.15.158:3300/0 conn(0x7f463015dfe0 0x7f46300cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f4635488640 1 --2- 172.21.15.77:0/3236924825 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300cf830 0x7f46300d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.378+0000 7f463648a640 1 --2- 172.21.15.77:0/3236924825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46300d27f0 0x7f46300d0300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4630067990 tx=0x7f462802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111a1336c95f0618 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4634c87640 1 -- 172.21.15.77:0/3236924825 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628035020 con 0x7f46300d27f0 2024-03-06T07:21:59.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4634c87640 1 -- 172.21.15.77:0/3236924825 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4628031870 con 0x7f46300d27f0 2024-03-06T07:21:59.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4634c87640 1 -- 172.21.15.77:0/3236924825 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628031c40 con 0x7f46300d27f0 2024-03-06T07:21:59.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 -- 172.21.15.77:0/3236924825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46300d27f0 msgr2=0x7f46300d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 --2- 172.21.15.77:0/3236924825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46300d27f0 0x7f46300d0300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4630067990 tx=0x7f462802b910 comp rx=0 tx=0).stop 2024-03-06T07:21:59.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 -- 172.21.15.77:0/3236924825 shutdown_connections 2024-03-06T07:21:59.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 --2- 172.21.15.77:0/3236924825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46300d27f0 0x7f46300d0300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 --2- 172.21.15.77:0/3236924825 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300cf830 0x7f46300d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 --2- 172.21.15.77:0/3236924825 >> v2:172.21.15.158:3300/0 conn(0x7f463015dfe0 0x7f46300cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 -- 172.21.15.77:0/3236924825 >> 172.21.15.77:0/3236924825 conn(0x7f46300d3cd0 msgr2=0x7f46300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 -- 172.21.15.77:0/3236924825 shutdown_connections 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.379+0000 7f4637eed640 1 -- 172.21.15.77:0/3236924825 wait complete. 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.380+0000 7f4637eed640 1 Processor -- start 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.380+0000 7f4637eed640 1 -- start start 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f4637eed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46300cf830 0x7f463015a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f4637eed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300d27f0 0x7f46300b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f4637eed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f463015dfe0 0x7f46300b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f4637eed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46300d1c70 con 0x7f463015dfe0 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f4637eed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46300b39f0 con 0x7f46300cf830 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f4637eed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46300d1df0 con 0x7f46300d27f0 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f463015dfe0 0x7f46300b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f463015dfe0 0x7f46300b6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41142/0 (socket says 172.21.15.77:41142) 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 -- 172.21.15.77:0/4254480178 learned_addr learned my addr 172.21.15.77:0/4254480178 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 -- 172.21.15.77:0/4254480178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300d27f0 msgr2=0x7f46300b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300d27f0 0x7f46300b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 -- 172.21.15.77:0/4254480178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46300cf830 msgr2=0x7f463015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46300cf830 0x7f463015a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 -- 172.21.15.77:0/4254480178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300b9af0 con 0x7f463015dfe0 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.381+0000 7f463648a640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f463015dfe0 0x7f46300b6e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4628009960 tx=0x7f46280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.382+0000 7f46267fc640 1 -- 172.21.15.77:0/4254480178 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628040070 con 0x7f463015dfe0 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.382+0000 7f46267fc640 1 -- 172.21.15.77:0/4254480178 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4628030c50 con 0x7f463015dfe0 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.382+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46300cd5e0 con 0x7f463015dfe0 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.382+0000 7f46267fc640 1 -- 172.21.15.77:0/4254480178 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628035040 con 0x7f463015dfe0 2024-03-06T07:21:59.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.382+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46300cdba0 con 0x7f463015dfe0 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.383+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463004f330 con 0x7f463015dfe0 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.383+0000 7f46267fc640 1 -- 172.21.15.77:0/4254480178 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4628038020 con 0x7f463015dfe0 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.383+0000 7f46267fc640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f461406b4d0 0x7f461406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.384+0000 7f4635c89640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f461406b4d0 0x7f461406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.384+0000 7f46267fc640 1 -- 172.21.15.77:0/4254480178 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46280ac030 con 0x7f463015dfe0 2024-03-06T07:21:59.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.384+0000 7f4635c89640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f461406b4d0 0x7f461406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f462c009eb0 tx=0x7f462c0094a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.388+0000 7f46267fc640 1 -- 172.21.15.77:0/4254480178 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f462807aeb0 con 0x7f463015dfe0 2024-03-06T07:21:59.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.561+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46300ceac0 con 0x7f463015dfe0 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.562+0000 7f46267fc640 1 -- 172.21.15.77:0/4254480178 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16606 (secure 0 0 0) 0x7f462807a7b0 con 0x7f463015dfe0 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:21:59.565 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:58.285580+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f461406b4d0 msgr2=0x7f461406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f461406b4d0 0x7f461406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f462c009eb0 tx=0x7f462c0094a0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f463015dfe0 msgr2=0x7f46300b6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f463015dfe0 0x7f46300b6e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4628009960 tx=0x7f46280046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 shutdown_connections 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f461406b4d0 0x7f461406d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f463015dfe0 0x7f46300b6e50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46300d27f0 0x7f46300b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 --2- 172.21.15.77:0/4254480178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46300cf830 0x7f463015a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 >> 172.21.15.77:0/4254480178 conn(0x7f46300d3cd0 msgr2=0x7f463006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 shutdown_connections 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.564+0000 7f4637eed640 1 -- 172.21.15.77:0/4254480178 wait complete. 2024-03-06T07:21:59.577 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.678+0000 7f16a231e640 1 Processor -- start 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.678+0000 7f16a231e640 1 -- start start 2024-03-06T07:21:59.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f16a231e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f169c15bbb0 0x7f169c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f16a231e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c15ee10 0x7f169c071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f16a231e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c0cea90 con 0x7f169c06b4f0 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f16a231e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169c161970 con 0x7f169c15bbb0 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f16a231e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c1624f0 con 0x7f169c15ee10 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f169bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f169c15bbb0 0x7f169c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f169bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f169c15bbb0 0x7f169c06afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41152/0 (socket says 172.21.15.77:41152) 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f169bfff640 1 -- 172.21.15.77:0/1838967923 learned_addr learned my addr 172.21.15.77:0/1838967923 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f16a08bb640 1 --2- 172.21.15.77:0/1838967923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c15ee10 0x7f169c071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f169bfff640 1 -- 172.21.15.77:0/1838967923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c15ee10 msgr2=0x7f169c071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f169bfff640 1 --2- 172.21.15.77:0/1838967923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c15ee10 0x7f169c071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f169bfff640 1 -- 172.21.15.77:0/1838967923 >> v1:172.21.15.158:6789/0 conn(0x7f169c06b4f0 legacy=0x7f169c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f169bfff640 1 -- 172.21.15.77:0/1838967923 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c161bc0 con 0x7f169c15bbb0 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.679+0000 7f16a08bb640 1 --2- 172.21.15.77:0/1838967923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c15ee10 0x7f169c071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.680+0000 7f169bfff640 1 --2- 172.21.15.77:0/1838967923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15bbb0 0x7f169c06afb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f168c009870 tx=0x7f168c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63d21ccd5fb28ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.680+0000 7f169affd640 1 -- 172.21.15.77:0/1838967923 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c034070 con 0x7f169c15bbb0 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.680+0000 7f169affd640 1 -- 172.21.15.77:0/1838967923 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c030840 con 0x7f169c15bbb0 2024-03-06T07:21:59.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.680+0000 7f169affd640 1 -- 172.21.15.77:0/1838967923 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c030c10 con 0x7f169c15bbb0 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.681+0000 7f16a231e640 1 -- 172.21.15.77:0/1838967923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15bbb0 msgr2=0x7f169c06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.681+0000 7f16a231e640 1 --2- 172.21.15.77:0/1838967923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15bbb0 0x7f169c06afb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f168c009870 tx=0x7f168c02b980 comp rx=0 tx=0).stop 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.681+0000 7f16a231e640 1 -- 172.21.15.77:0/1838967923 shutdown_connections 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.681+0000 7f16a231e640 1 --2- 172.21.15.77:0/1838967923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c15ee10 0x7f169c071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.681+0000 7f16a231e640 1 --2- 172.21.15.77:0/1838967923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15bbb0 0x7f169c06afb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.681+0000 7f16a231e640 1 -- 172.21.15.77:0/1838967923 >> 172.21.15.77:0/1838967923 conn(0x7f169c0b2660 msgr2=0x7f169c0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.681+0000 7f16a231e640 1 -- 172.21.15.77:0/1838967923 shutdown_connections 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.681+0000 7f16a231e640 1 -- 172.21.15.77:0/1838967923 wait complete. 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.682+0000 7f16a231e640 1 Processor -- start 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.682+0000 7f16a231e640 1 -- start start 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f16a231e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c06b4f0 0x7f169c154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f16a231e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15ee10 0x7f169c1471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f16a231e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f169c155e80 0x7f169c0cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f16a231e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0721e0 con 0x7f169c15ee10 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f16a231e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c162180 con 0x7f169c155e80 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f16a231e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c072360 con 0x7f169c06b4f0 2024-03-06T07:21:59.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15ee10 0x7f169c1471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15ee10 0x7f169c1471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41162/0 (socket says 172.21.15.77:41162) 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 -- 172.21.15.77:0/877661189 learned_addr learned my addr 172.21.15.77:0/877661189 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f16a08bb640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f169c155e80 0x7f169c0cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169bfff640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c06b4f0 0x7f169c154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 -- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c06b4f0 msgr2=0x7f169c154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c06b4f0 0x7f169c154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 -- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f169c155e80 msgr2=0x7f169c0cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f169c155e80 0x7f169c0cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 -- 172.21.15.77:0/877661189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c160440 con 0x7f169c15ee10 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f16a08bb640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f169c155e80 0x7f169c0cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f169b7fe640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15ee10 0x7f169c1471f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f169000c480 tx=0x7f169000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.683+0000 7f1698ff9640 1 -- 172.21.15.77:0/877661189 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16900052d0 con 0x7f169c15ee10 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.684+0000 7f1698ff9640 1 -- 172.21.15.77:0/877661189 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690005470 con 0x7f169c15ee10 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.684+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c147730 con 0x7f169c15ee10 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.684+0000 7f1698ff9640 1 -- 172.21.15.77:0/877661189 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690012870 con 0x7f169c15ee10 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.684+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c15f960 con 0x7f169c15ee10 2024-03-06T07:21:59.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.685+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c0733b0 con 0x7f169c15ee10 2024-03-06T07:21:59.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.685+0000 7f1698ff9640 1 -- 172.21.15.77:0/877661189 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1690012a10 con 0x7f169c15ee10 2024-03-06T07:21:59.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.685+0000 7f1698ff9640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f168806b4d0 0x7f168806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.685+0000 7f169bfff640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f168806b4d0 0x7f168806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.686+0000 7f1698ff9640 1 -- 172.21.15.77:0/877661189 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1690089dd0 con 0x7f169c15ee10 2024-03-06T07:21:59.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.686+0000 7f169bfff640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f168806b4d0 0x7f168806d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f168c036000 tx=0x7f168c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.690+0000 7f1698ff9640 1 -- 172.21.15.77:0/877661189 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f169c0cea90 con 0x7f169c15ee10 2024-03-06T07:21:59.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.899+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f169c0d0720 con 0x7f169c15ee10 2024-03-06T07:21:59.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.955+0000 7f1698ff9640 1 -- 172.21.15.77:0/877661189 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f169008b0b0 con 0x7f169c15ee10 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f168806b4d0 msgr2=0x7f168806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f168806b4d0 0x7f168806d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f168c036000 tx=0x7f168c03b040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15ee10 msgr2=0x7f169c1471f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15ee10 0x7f169c1471f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f169000c480 tx=0x7f169000c940 comp rx=0 tx=0).stop 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 shutdown_connections 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f168806b4d0 0x7f168806d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169c15ee10 0x7f169c1471f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f169c155e80 0x7f169c0cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 --2- 172.21.15.77:0/877661189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f169c06b4f0 0x7f169c154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.957+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 >> 172.21.15.77:0/877661189 conn(0x7f169c0b2660 msgr2=0x7f169c0cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.958+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 shutdown_connections 2024-03-06T07:21:59.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:21:59.958+0000 7f16a231e640 1 -- 172.21.15.77:0/877661189 wait complete. 2024-03-06T07:21:59.971 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:21:59.971 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.072+0000 7f17584df640 1 Processor -- start 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.072+0000 7f17584df640 1 -- start start 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f17584df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f17501614d0 0x7f17500d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f17584df640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17500d1910 0x7f17500d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f17584df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17500d22e0 0x7f17500d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f17584df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17500d2750 con 0x7f17501614d0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f17584df640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17501618b0 con 0x7f17500d1910 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f17584df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17500cf400 con 0x7f17500d22e0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f175627b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f17501614d0 0x7f17500d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f175627b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f17501614d0 0x7f17500d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41166/0 (socket says 172.21.15.77:41166) 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f175627b640 1 -- 172.21.15.77:0/2655208279 learned_addr learned my addr 172.21.15.77:0/2655208279 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f1756a7c640 1 --2- 172.21.15.77:0/2655208279 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17500d22e0 0x7f17500d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.073+0000 7f1755a7a640 1 --2- 172.21.15.77:0/2655208279 >> v2:172.21.15.158:3300/0 conn(0x7f17500d1910 0x7f17500d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.074+0000 7f175627b640 1 -- 172.21.15.77:0/2655208279 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17500d22e0 msgr2=0x7f17500d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.074+0000 7f175627b640 1 --2- 172.21.15.77:0/2655208279 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17500d22e0 0x7f17500d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.074+0000 7f175627b640 1 -- 172.21.15.77:0/2655208279 >> v2:172.21.15.158:3300/0 conn(0x7f17500d1910 msgr2=0x7f17500d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.074+0000 7f175627b640 1 --2- 172.21.15.77:0/2655208279 >> v2:172.21.15.158:3300/0 conn(0x7f17500d1910 0x7f17500d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.074+0000 7f175627b640 1 -- 172.21.15.77:0/2655208279 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17500cf6a0 con 0x7f17501614d0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.074+0000 7f175627b640 1 --2- 172.21.15.77:0/2655208279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17501614d0 0x7f17500d4860 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17500ce550 tx=0x7f174002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bc6c591fe9329f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f1755279640 1 -- 172.21.15.77:0/2655208279 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174002ca70 con 0x7f17501614d0 2024-03-06T07:22:00.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f1755279640 1 -- 172.21.15.77:0/2655208279 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174002cc10 con 0x7f17501614d0 2024-03-06T07:22:00.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f1755279640 1 -- 172.21.15.77:0/2655208279 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17400343f0 con 0x7f17501614d0 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 -- 172.21.15.77:0/2655208279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17501614d0 msgr2=0x7f17500d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 --2- 172.21.15.77:0/2655208279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17501614d0 0x7f17500d4860 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17500ce550 tx=0x7f174002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 -- 172.21.15.77:0/2655208279 shutdown_connections 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 --2- 172.21.15.77:0/2655208279 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17500d22e0 0x7f17500d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 --2- 172.21.15.77:0/2655208279 >> v2:172.21.15.158:3300/0 conn(0x7f17500d1910 0x7f17500d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 --2- 172.21.15.77:0/2655208279 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17501614d0 0x7f17500d4860 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 -- 172.21.15.77:0/2655208279 >> 172.21.15.77:0/2655208279 conn(0x7f17500d3d40 msgr2=0x7f17500d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 -- 172.21.15.77:0/2655208279 shutdown_connections 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.075+0000 7f17584df640 1 -- 172.21.15.77:0/2655208279 wait complete. 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.076+0000 7f17584df640 1 Processor -- start 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.076+0000 7f17584df640 1 -- start start 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f17584df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17500d1910 0x7f17500b9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f17584df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17500d22e0 0x7f17500c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f17584df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17501614d0 0x7f17500c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f17584df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17500b0910 con 0x7f17500d22e0 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f17584df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17500cfc60 con 0x7f17500d1910 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f17584df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17500d0be0 con 0x7f17501614d0 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1755a7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17500d22e0 0x7f17500c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f175627b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17500d1910 0x7f17500b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1755a7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17500d22e0 0x7f17500c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41180/0 (socket says 172.21.15.77:41180) 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1755a7a640 1 -- 172.21.15.77:0/3372065083 learned_addr learned my addr 172.21.15.77:0/3372065083 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1756a7c640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17501614d0 0x7f17500c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1755a7a640 1 -- 172.21.15.77:0/3372065083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17501614d0 msgr2=0x7f17500c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1755a7a640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17501614d0 0x7f17500c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1755a7a640 1 -- 172.21.15.77:0/3372065083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17500d1910 msgr2=0x7f17500b9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1755a7a640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17500d1910 0x7f17500b9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.077+0000 7f1755a7a640 1 -- 172.21.15.77:0/3372065083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17501f7840 con 0x7f17500d22e0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.078+0000 7f1755a7a640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17500d22e0 0x7f17500c96c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f174400e9b0 tx=0x7f174400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.078+0000 7f174effd640 1 -- 172.21.15.77:0/3372065083 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744017070 con 0x7f17500d22e0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.078+0000 7f174effd640 1 -- 172.21.15.77:0/3372065083 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174400c9a0 con 0x7f17500d22e0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.078+0000 7f174effd640 1 -- 172.21.15.77:0/3372065083 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174400cd90 con 0x7f17500d22e0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.078+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17500cd500 con 0x7f17500d22e0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.078+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17500cdac0 con 0x7f17500d22e0 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.079+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17500c6fb0 con 0x7f17500d22e0 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.079+0000 7f174effd640 1 -- 172.21.15.77:0/3372065083 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f174401a070 con 0x7f17500d22e0 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.079+0000 7f174effd640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173c06b500 0x7f173c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.080+0000 7f175627b640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173c06b500 0x7f173c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.080+0000 7f174effd640 1 -- 172.21.15.77:0/3372065083 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f174408a140 con 0x7f17500d22e0 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.080+0000 7f175627b640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173c06b500 0x7f173c06d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1740035000 tx=0x7f1740030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.084+0000 7f174effd640 1 -- 172.21.15.77:0/3372065083 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1744058970 con 0x7f17500d22e0 2024-03-06T07:22:00.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.290+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1750068ff0 con 0x7f17500d22e0 2024-03-06T07:22:00.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.956+0000 7f174effd640 1 -- 172.21.15.77:0/3372065083 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f174405ca10 con 0x7f17500d22e0 2024-03-06T07:22:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173c06b500 msgr2=0x7f173c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173c06b500 0x7f173c06d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1740035000 tx=0x7f1740030000 comp rx=0 tx=0).stop 2024-03-06T07:22:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17500d22e0 msgr2=0x7f17500c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17500d22e0 0x7f17500c96c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f174400e9b0 tx=0x7f174400ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 shutdown_connections 2024-03-06T07:22:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173c06b500 0x7f173c06d9c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17500d22e0 0x7f17500c96c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17501614d0 0x7f17500c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 --2- 172.21.15.77:0/3372065083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17500d1910 0x7f17500b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 >> 172.21.15.77:0/3372065083 conn(0x7f17500d3d40 msgr2=0x7f17500b48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 shutdown_connections 2024-03-06T07:22:00.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:00.959+0000 7f17584df640 1 -- 172.21.15.77:0/3372065083 wait complete. 2024-03-06T07:22:00.974 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:00.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:01.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.078+0000 7fe83e647640 1 Processor -- start 2024-03-06T07:22:01.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.078+0000 7fe83e647640 1 -- start start 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe83e647640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8381515e0 0x7fe8380cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe83e647640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe83815b430 0x7fe838071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe83e647640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83815b8b0 con 0x7fe8380cd7a0 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe83e647640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe838070a70 con 0x7fe83815b430 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe83e647640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe838070d50 con 0x7fe8381515e0 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe83cbe4640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe8380cd7a0 0x7fe8380cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35462/0 (socket says 172.21.15.77:35462) 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe83cbe4640 1 -- 172.21.15.77:0/1948853435 learned_addr learned my addr 172.21.15.77:0/1948853435 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe8377fe640 1 --2- 172.21.15.77:0/1948853435 >> v2:172.21.15.158:3300/0 conn(0x7fe83815b430 0x7fe838071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.079+0000 7fe837fff640 1 --2- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8381515e0 0x7fe8380cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.080+0000 7fe836ffd640 1 -- 172.21.15.77:0/1948853435 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165612688 0 0) 0x7fe83815b8b0 con 0x7fe8380cd7a0 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.080+0000 7fe836ffd640 1 -- 172.21.15.77:0/1948853435 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe820003520 con 0x7fe8380cd7a0 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.080+0000 7fe8377fe640 1 -- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8381515e0 msgr2=0x7fe8380cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.080+0000 7fe8377fe640 1 --2- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8381515e0 0x7fe8380cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.080+0000 7fe8377fe640 1 -- 172.21.15.77:0/1948853435 >> v1:172.21.15.77:6789/0 conn(0x7fe8380cd7a0 legacy=0x7fe8380cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.080+0000 7fe8377fe640 1 -- 172.21.15.77:0/1948853435 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe838071030 con 0x7fe83815b430 2024-03-06T07:22:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.080+0000 7fe837fff640 1 --2- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8381515e0 0x7fe8380cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.080+0000 7fe8377fe640 1 --2- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815b430 0x7fe838071830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0097a0 tx=0x7fe82c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a81ea566478bfd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe836ffd640 1 -- 172.21.15.77:0/1948853435 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c035020 con 0x7fe83815b430 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe836ffd640 1 -- 172.21.15.77:0/1948853435 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c031830 con 0x7fe83815b430 2024-03-06T07:22:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe836ffd640 1 -- 172.21.15.77:0/1948853435 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c031c00 con 0x7fe83815b430 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe83e647640 1 -- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815b430 msgr2=0x7fe838071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe83e647640 1 --2- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815b430 0x7fe838071830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0097a0 tx=0x7fe82c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe83e647640 1 -- 172.21.15.77:0/1948853435 shutdown_connections 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe83e647640 1 --2- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe83815b430 0x7fe838071830 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe83e647640 1 --2- 172.21.15.77:0/1948853435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8381515e0 0x7fe8380cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe83e647640 1 -- 172.21.15.77:0/1948853435 >> 172.21.15.77:0/1948853435 conn(0x7fe8380cf8a0 msgr2=0x7fe8380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe83e647640 1 -- 172.21.15.77:0/1948853435 shutdown_connections 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.081+0000 7fe83e647640 1 -- 172.21.15.77:0/1948853435 wait complete. 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.082+0000 7fe83e647640 1 Processor -- start 2024-03-06T07:22:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.082+0000 7fe83e647640 1 -- start start 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83e647640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8380cd7a0 0x7fe8381f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83e647640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8381515e0 0x7fe838152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83e647640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe83815b430 0x7fe83815dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83e647640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe838071ef0 con 0x7fe83815b430 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83e647640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe838071d70 con 0x7fe8381515e0 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83e647640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8380cc3a0 con 0x7fe8380cd7a0 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83cbe4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe83815b430 0x7fe83815dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83cbe4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe83815b430 0x7fe83815dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41186/0 (socket says 172.21.15.77:41186) 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe83cbe4640 1 -- 172.21.15.77:0/2856136966 learned_addr learned my addr 172.21.15.77:0/2856136966 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe837fff640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8380cd7a0 0x7fe8381f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.083+0000 7fe8377fe640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8381515e0 0x7fe838152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe83cbe4640 1 -- 172.21.15.77:0/2856136966 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8380cd7a0 msgr2=0x7fe8381f13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe83cbe4640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8380cd7a0 0x7fe8381f13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe83cbe4640 1 -- 172.21.15.77:0/2856136966 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8381515e0 msgr2=0x7fe838152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe83cbe4640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8381515e0 0x7fe838152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe83cbe4640 1 -- 172.21.15.77:0/2856136966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe838153a80 con 0x7fe83815b430 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe83cbe4640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe83815b430 0x7fe83815dcf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe83000cb10 tx=0x7fe83000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe834ff9640 1 -- 172.21.15.77:0/2856136966 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830017070 con 0x7fe83815b430 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe834ff9640 1 -- 172.21.15.77:0/2856136966 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83000ac70 con 0x7fe83815b430 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8381534c0 con 0x7fe83815b430 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe834ff9640 1 -- 172.21.15.77:0/2856136966 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83000ae60 con 0x7fe83815b430 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.084+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83813db40 con 0x7fe83815b430 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.085+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe838068840 con 0x7fe83815b430 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.085+0000 7fe834ff9640 1 -- 172.21.15.77:0/2856136966 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe83001a070 con 0x7fe83815b430 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.086+0000 7fe834ff9640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe82406b4d0 0x7fe82406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.086+0000 7fe834ff9640 1 -- 172.21.15.77:0/2856136966 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe830089e90 con 0x7fe83815b430 2024-03-06T07:22:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.086+0000 7fe837fff640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe82406b4d0 0x7fe82406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.086+0000 7fe837fff640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe82406b4d0 0x7fe82406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe82800a570 tx=0x7fe82800a2a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.090+0000 7fe834ff9640 1 -- 172.21.15.77:0/2856136966 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe830058750 con 0x7fe83815b430 2024-03-06T07:22:01.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.297+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8380d0710 con 0x7fe83815b430 2024-03-06T07:22:01.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.298+0000 7fe834ff9640 1 -- 172.21.15.77:0/2856136966 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fe8380d0710 con 0x7fe83815b430 2024-03-06T07:22:01.300 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:01.300 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.285900+0000","modified":"2024-03-06T07:22:00.291221+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-06T07:22:01.300 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.299+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe82406b4d0 msgr2=0x7fe82406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.299+0000 7fe83e647640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe82406b4d0 0x7fe82406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe82800a570 tx=0x7fe82800a2a0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.299+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe83815b430 msgr2=0x7fe83815dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.299+0000 7fe83e647640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe83815b430 0x7fe83815dcf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe83000cb10 tx=0x7fe83000a060 comp rx=0 tx=0).stop 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.299+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 shutdown_connections 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.299+0000 7fe83e647640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe82406b4d0 0x7fe82406d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.300+0000 7fe83e647640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe83815b430 0x7fe83815dcf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.300+0000 7fe83e647640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8381515e0 0x7fe838152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.300+0000 7fe83e647640 1 --2- 172.21.15.77:0/2856136966 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8380cd7a0 0x7fe8381f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.300+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 >> 172.21.15.77:0/2856136966 conn(0x7fe8380cf8a0 msgr2=0x7fe838150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.300+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 shutdown_connections 2024-03-06T07:22:01.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.300+0000 7fe83e647640 1 -- 172.21.15.77:0/2856136966 wait complete. 2024-03-06T07:22:01.312 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.410+0000 7fe72c925640 1 Processor -- start 2024-03-06T07:22:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.410+0000 7fe72c925640 1 -- start start 2024-03-06T07:22:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72c925640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7280b4f70 0x7fe7280b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72c925640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe72806c6d0 0x7fe72815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72c925640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7280b5890 con 0x7fe728066d90 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72c925640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72815e9f0 con 0x7fe7280b4f70 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72c925640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7280cda10 con 0x7fe72806c6d0 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7280b4f70 0x7fe7280b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7280b4f70 0x7fe7280b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41192/0 (socket says 172.21.15.77:41192) 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72659c640 1 -- 172.21.15.77:0/4283754695 learned_addr learned my addr 172.21.15.77:0/4283754695 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe71ffff640 1 --2- 172.21.15.77:0/4283754695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe72806c6d0 0x7fe72815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72659c640 1 -- 172.21.15.77:0/4283754695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe72806c6d0 msgr2=0x7fe72815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72659c640 1 --2- 172.21.15.77:0/4283754695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe72806c6d0 0x7fe72815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72659c640 1 -- 172.21.15.77:0/4283754695 >> v1:172.21.15.158:6789/0 conn(0x7fe728066d90 legacy=0x7fe7280ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.411+0000 7fe72659c640 1 -- 172.21.15.77:0/4283754695 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7280cd450 con 0x7fe7280b4f70 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72659c640 1 --2- 172.21.15.77:0/4283754695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7280b4f70 0x7fe7280b5350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe718009870 tx=0x7fe71802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a936d007764eb36e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe725d9b640 1 -- 172.21.15.77:0/4283754695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718034070 con 0x7fe7280b4f70 2024-03-06T07:22:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe725d9b640 1 -- 172.21.15.77:0/4283754695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe718030840 con 0x7fe7280b4f70 2024-03-06T07:22:01.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe725d9b640 1 -- 172.21.15.77:0/4283754695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718030c10 con 0x7fe7280b4f70 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72c925640 1 -- 172.21.15.77:0/4283754695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7280b4f70 msgr2=0x7fe7280b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72c925640 1 --2- 172.21.15.77:0/4283754695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7280b4f70 0x7fe7280b5350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe718009870 tx=0x7fe71802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72c925640 1 -- 172.21.15.77:0/4283754695 shutdown_connections 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72c925640 1 --2- 172.21.15.77:0/4283754695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe72806c6d0 0x7fe72815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72c925640 1 --2- 172.21.15.77:0/4283754695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7280b4f70 0x7fe7280b5350 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72c925640 1 -- 172.21.15.77:0/4283754695 >> 172.21.15.77:0/4283754695 conn(0x7fe7280724d0 msgr2=0x7fe7280728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72c925640 1 -- 172.21.15.77:0/4283754695 shutdown_connections 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.412+0000 7fe72c925640 1 -- 172.21.15.77:0/4283754695 wait complete. 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.414+0000 7fe72c925640 1 Processor -- start 2024-03-06T07:22:01.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.414+0000 7fe72c925640 1 -- start start 2024-03-06T07:22:01.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.414+0000 7fe72c925640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe728066d90 0x7fe72814e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.414+0000 7fe72c925640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7280b4f70 0x7fe728158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.414+0000 7fe72c925640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7281581d0 0x7fe728158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.414+0000 7fe72c925640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72815f3c0 con 0x7fe7281581d0 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.414+0000 7fe72c925640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72815e680 con 0x7fe7280b4f70 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.414+0000 7fe72c925640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe728161ad0 con 0x7fe728066d90 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe726d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7281581d0 0x7fe728158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe71ffff640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7280b4f70 0x7fe728158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe72659c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe728066d90 0x7fe72814e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe726d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7281581d0 0x7fe728158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41198/0 (socket says 172.21.15.77:41198) 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe71ffff640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7280b4f70 0x7fe728158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:43406/0 (socket says 172.21.15.77:43406) 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe72659c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe728066d90 0x7fe72814e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35610/0 (socket says 172.21.15.77:35610) 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe71ffff640 1 -- 172.21.15.77:0/2594810395 learned_addr learned my addr 172.21.15.77:0/2594810395 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe726d9d640 1 -- 172.21.15.77:0/2594810395 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe728066d90 msgr2=0x7fe72814e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe726d9d640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe728066d90 0x7fe72814e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe726d9d640 1 -- 172.21.15.77:0/2594810395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7280b4f70 msgr2=0x7fe728158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe726d9d640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7280b4f70 0x7fe728158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe726d9d640 1 -- 172.21.15.77:0/2594810395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7281586d0 con 0x7fe7281581d0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe726d9d640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7281581d0 0x7fe728158ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7280671e0 tx=0x7fe72000cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe71effd640 1 -- 172.21.15.77:0/2594810395 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720003b70 con 0x7fe7281581d0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe71effd640 1 -- 172.21.15.77:0/2594810395 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe720003d10 con 0x7fe7281581d0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72814fab0 con 0x7fe7281581d0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe71effd640 1 -- 172.21.15.77:0/2594810395 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720005310 con 0x7fe7281581d0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.415+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe728150070 con 0x7fe7281581d0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.416+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72804f330 con 0x7fe7281581d0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.416+0000 7fe71effd640 1 -- 172.21.15.77:0/2594810395 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7200054b0 con 0x7fe7281581d0 2024-03-06T07:22:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.417+0000 7fe71effd640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.417+0000 7fe71effd640 1 -- 172.21.15.77:0/2594810395 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe720089f00 con 0x7fe7281581d0 2024-03-06T07:22:01.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.417+0000 7fe72659c640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.417+0000 7fe72659c640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe70806b4d0 0x7fe70806d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe718036000 tx=0x7fe71803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.421+0000 7fe71effd640 1 -- 172.21.15.77:0/2594810395 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe720058740 con 0x7fe7281581d0 2024-03-06T07:22:01.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.596+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7280d1070 con 0x7fe7281581d0 2024-03-06T07:22:01.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.597+0000 7fe71effd640 1 -- 172.21.15.77:0/2594810395 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fe72005c7e0 con 0x7fe7281581d0 2024-03-06T07:22:01.600 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:01.600 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:01.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe70806b4d0 msgr2=0x7fe70806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe70806b4d0 0x7fe70806d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe718036000 tx=0x7fe71803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7281581d0 msgr2=0x7fe728158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7281581d0 0x7fe728158ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7280671e0 tx=0x7fe72000cb20 comp rx=0 tx=0).stop 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 shutdown_connections 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7281581d0 0x7fe728158ed0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7280b4f70 0x7fe728158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 --2- 172.21.15.77:0/2594810395 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe728066d90 0x7fe72814e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.599+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 >> 172.21.15.77:0/2594810395 conn(0x7fe7280724d0 msgr2=0x7fe72806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.600+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 shutdown_connections 2024-03-06T07:22:01.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.600+0000 7fe72c925640 1 -- 172.21.15.77:0/2594810395 wait complete. 2024-03-06T07:22:01.613 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:01.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:01.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f3422c71640 1 Processor -- start 2024-03-06T07:22:01.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f3422c71640 1 -- start start 2024-03-06T07:22:01.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f3422c71640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f341c15bdd0 0x7f341c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f3422c71640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f341c15fa10 0x7f341c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f3422c71640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c0cebe0 con 0x7f341c0d16e0 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f3422c71640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341c0ceec0 con 0x7f341c15bdd0 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f3422c71640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341c0cf1a0 con 0x7f341c15fa10 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f3420a0d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f341c15bdd0 0x7f341c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f342120e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f341c0d16e0 0x7f341c0d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35476/0 (socket says 172.21.15.77:35476) 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.712+0000 7f342120e640 1 -- 172.21.15.77:0/1176754692 learned_addr learned my addr 172.21.15.77:0/1176754692 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.713+0000 7f341bfff640 1 --2- 172.21.15.77:0/1176754692 >> v2:172.21.15.158:3300/0 conn(0x7f341c15fa10 0x7f341c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.713+0000 7f3420a0d640 1 -- 172.21.15.77:0/1176754692 >> v2:172.21.15.158:3300/0 conn(0x7f341c15fa10 msgr2=0x7f341c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.713+0000 7f3420a0d640 1 --2- 172.21.15.77:0/1176754692 >> v2:172.21.15.158:3300/0 conn(0x7f341c15fa10 0x7f341c0ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.713+0000 7f3420a0d640 1 -- 172.21.15.77:0/1176754692 >> v1:172.21.15.77:6789/0 conn(0x7f341c0d16e0 legacy=0x7f341c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.713+0000 7f341b7fe640 1 -- 172.21.15.77:0/1176754692 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117257738 0 0) 0x7f341c0cebe0 con 0x7f341c0d16e0 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.713+0000 7f3420a0d640 1 -- 172.21.15.77:0/1176754692 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c0cc8e0 con 0x7f341c15bdd0 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.713+0000 7f3420a0d640 1 --2- 172.21.15.77:0/1176754692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0c5610 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f341c0cdfd0 tx=0x7f341002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5387882c800a5f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f341b7fe640 1 -- 172.21.15.77:0/1176754692 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34100315a0 con 0x7f341c15bdd0 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f341b7fe640 1 -- 172.21.15.77:0/1176754692 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3410037070 con 0x7f341c15bdd0 2024-03-06T07:22:01.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f341b7fe640 1 -- 172.21.15.77:0/1176754692 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410031960 con 0x7f341c15bdd0 2024-03-06T07:22:01.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f3422c71640 1 -- 172.21.15.77:0/1176754692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 msgr2=0x7f341c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f3422c71640 1 --2- 172.21.15.77:0/1176754692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0c5610 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f341c0cdfd0 tx=0x7f341002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:01.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f3422c71640 1 -- 172.21.15.77:0/1176754692 shutdown_connections 2024-03-06T07:22:01.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f3422c71640 1 --2- 172.21.15.77:0/1176754692 >> v2:172.21.15.158:3300/0 conn(0x7f341c15fa10 0x7f341c0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f3422c71640 1 --2- 172.21.15.77:0/1176754692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0c5610 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f3422c71640 1 -- 172.21.15.77:0/1176754692 >> 172.21.15.77:0/1176754692 conn(0x7f341c09a9f0 msgr2=0x7f341c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f3422c71640 1 -- 172.21.15.77:0/1176754692 shutdown_connections 2024-03-06T07:22:01.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.714+0000 7f3422c71640 1 -- 172.21.15.77:0/1176754692 wait complete. 2024-03-06T07:22:01.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.715+0000 7f3422c71640 1 Processor -- start 2024-03-06T07:22:01.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.715+0000 7f3422c71640 1 -- start start 2024-03-06T07:22:01.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f3422c71640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c0d16e0 0x7f341c14d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f3422c71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f3422c71640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f341c15fa10 0x7f341c0b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f3422c71640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c15c1b0 con 0x7f341c15bdd0 2024-03-06T07:22:01.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f3422c71640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0cc5d0 con 0x7f341c0d16e0 2024-03-06T07:22:01.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f3422c71640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0ce7b0 con 0x7f341c15fa10 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f341bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f341bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41224/0 (socket says 172.21.15.77:41224) 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f341bfff640 1 -- 172.21.15.77:0/4136829024 learned_addr learned my addr 172.21.15.77:0/4136829024 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f3420a0d640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c0d16e0 0x7f341c14d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f342120e640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f341c15fa10 0x7f341c0b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.716+0000 7f341bfff640 1 -- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f341c15fa10 msgr2=0x7f341c0b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f341bfff640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f341c15fa10 0x7f341c0b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f341bfff640 1 -- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c0d16e0 msgr2=0x7f341c14d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f341bfff640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c0d16e0 0x7f341c14d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f341bfff640 1 -- 172.21.15.77:0/4136829024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c09ea80 con 0x7f341c15bdd0 2024-03-06T07:22:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f342120e640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f341c15fa10 0x7f341c0b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f341bfff640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0b6b40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f340800e980 tx=0x7f340800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f34197fa640 1 -- 172.21.15.77:0/4136829024 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408017070 con 0x7f341c15bdd0 2024-03-06T07:22:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f34197fa640 1 -- 172.21.15.77:0/4136829024 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340800ca00 con 0x7f341c15bdd0 2024-03-06T07:22:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f34197fa640 1 -- 172.21.15.77:0/4136829024 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340800cdd0 con 0x7f341c15bdd0 2024-03-06T07:22:01.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c09ed60 con 0x7f341c15bdd0 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.717+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c09f320 con 0x7f341c15bdd0 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.718+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c04f330 con 0x7f341c15bdd0 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.718+0000 7f34197fa640 1 -- 172.21.15.77:0/4136829024 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f340801a070 con 0x7f341c15bdd0 2024-03-06T07:22:01.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.719+0000 7f34197fa640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f340006b500 0x7f340006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.719+0000 7f34197fa640 1 -- 172.21.15.77:0/4136829024 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f340808a500 con 0x7f341c15bdd0 2024-03-06T07:22:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.719+0000 7f3420a0d640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f340006b500 0x7f340006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.719+0000 7f3420a0d640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f340006b500 0x7f340006d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3410036000 tx=0x7f341003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.723+0000 7f34197fa640 1 -- 172.21.15.77:0/4136829024 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3408058d30 con 0x7f341c15bdd0 2024-03-06T07:22:01.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.891+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f341c0b7a40 con 0x7f340006b500 2024-03-06T07:22:01.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.897+0000 7f34197fa640 1 -- 172.21.15.77:0/4136829024 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240003 (secure 0 0 0) 0x7f341c0b7a40 con 0x7f340006b500 2024-03-06T07:22:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:01.904 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:22:01.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.900+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f340006b500 msgr2=0x7f340006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.900+0000 7f3422c71640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f340006b500 0x7f340006d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3410036000 tx=0x7f341003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.900+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 msgr2=0x7f341c0b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.900+0000 7f3422c71640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0b6b40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f340800e980 tx=0x7f340800ee40 comp rx=0 tx=0).stop 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.901+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 shutdown_connections 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.901+0000 7f3422c71640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f340006b500 0x7f340006d9c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.901+0000 7f3422c71640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f341c15bdd0 0x7f341c0b6b40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.901+0000 7f3422c71640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f341c15fa10 0x7f341c0b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.901+0000 7f3422c71640 1 --2- 172.21.15.77:0/4136829024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c0d16e0 0x7f341c14d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.901+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 >> 172.21.15.77:0/4136829024 conn(0x7f341c09a9f0 msgr2=0x7f341c0d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.901+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 shutdown_connections 2024-03-06T07:22:01.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:01.901+0000 7f3422c71640 1 -- 172.21.15.77:0/4136829024 wait complete. 2024-03-06T07:22:01.914 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-06T07:22:01.384268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236144,"kb_used_data":2448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738576,"statfs":{"total":773094113280,"available":772852301824,"internally_reserved":0,"allocated":2506752,"data_stored":1573386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.446341"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938022+0000","last_change":"2024-03-06T07:21:56.938022+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938022+0000","last_undegraded":"2024-03-06T07:21:56.938022+0000","last_fullsized":"2024-03-06T07:21:56.938022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935261+0000","last_change":"2024-03-06T07:21:56.935261+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935261+0000","last_undegraded":"2024-03-06T07:21:56.935261+0000","last_fullsized":"2024-03-06T07:21:56.935261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937834+0000","last_change":"2024-03-06T07:21:56.937834+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937834+0000","last_undegraded":"2024-03-06T07:21:56.937834+0000","last_fullsized":"2024-03-06T07:21:56.937834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939073+0000","last_change":"2024-03-06T07:21:56.939073+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939073+0000","last_undegraded":"2024-03-06T07:21:56.939073+0000","last_fullsized":"2024-03-06T07:21:56.939073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935466+0000","last_change":"2024-03-06T07:21:56.935466+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935466+0000","last_undegraded":"2024-03-06T07:21:56.935466+0000","last_fullsized":"2024-03-06T07:21:56.935466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939057+0000","last_change":"2024-03-06T07:21:56.939057+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939057+0000","last_undegraded":"2024-03-06T07:21:56.939057+0000","last_fullsized":"2024-03-06T07:21:56.939057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.943522+0000","last_change":"2024-03-06T07:21:56.943522+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.943522+0000","last_undegraded":"2024-03-06T07:21:56.943522+0000","last_fullsized":"2024-03-06T07:21:56.943522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934052+0000","last_change":"2024-03-06T07:21:56.934052+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934052+0000","last_undegraded":"2024-03-06T07:21:56.934052+0000","last_fullsized":"2024-03-06T07:21:56.934052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936360+0000","last_change":"2024-03-06T07:21:56.936360+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936360+0000","last_undegraded":"2024-03-06T07:21:56.936360+0000","last_fullsized":"2024-03-06T07:21:56.936360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939604+0000","last_change":"2024-03-06T07:21:56.939604+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939604+0000","last_undegraded":"2024-03-06T07:21:56.939604+0000","last_fullsized":"2024-03-06T07:21:56.939604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934036+0000","last_change":"2024-03-06T07:21:56.934036+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934036+0000","last_undegraded":"2024-03-06T07:21:56.934036+0000","last_fullsized":"2024-03-06T07:21:56.934036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935178+0000","last_change":"2024-03-06T07:21:56.935178+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935178+0000","last_undegraded":"2024-03-06T07:21:56.935178+0000","last_fullsized":"2024-03-06T07:21:56.935178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934025+0000","last_change":"2024-03-06T07:21:56.934025+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934025+0000","last_undegraded":"2024-03-06T07:21:56.934025+0000","last_fullsized":"2024-03-06T07:21:56.934025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934892+0000","last_change":"2024-03-06T07:21:56.934892+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934892+0000","last_undegraded":"2024-03-06T07:21:56.934892+0000","last_fullsized":"2024-03-06T07:21:56.934892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938444+0000","last_change":"2024-03-06T07:21:56.938444+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938444+0000","last_undegraded":"2024-03-06T07:21:56.938444+0000","last_fullsized":"2024-03-06T07:21:56.938444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934948+0000","last_change":"2024-03-06T07:21:56.934948+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934948+0000","last_undegraded":"2024-03-06T07:21:56.934948+0000","last_fullsized":"2024-03-06T07:21:56.934948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.938785+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:21:56.938785+0000","last_peered":"2024-03-06T07:21:56.938785+0000","last_clean":"2024-03-06T07:21:56.938785+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:21:56.938785+0000","last_undegraded":"2024-03-06T07:21:56.938785+0000","last_fullsized":"2024-03-06T07:21:56.938785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937756+0000","last_change":"2024-03-06T07:21:56.937756+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937756+0000","last_undegraded":"2024-03-06T07:21:56.937756+0000","last_fullsized":"2024-03-06T07:21:56.937756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936880+0000","last_change":"2024-03-06T07:21:56.936880+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936880+0000","last_undegraded":"2024-03-06T07:21:56.936880+0000","last_fullsized":"2024-03-06T07:21:56.936880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935141+0000","last_change":"2024-03-06T07:21:56.935141+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935141+0000","last_undegraded":"2024-03-06T07:21:56.935141+0000","last_fullsized":"2024-03-06T07:21:56.935141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939122+0000","last_change":"2024-03-06T07:21:56.939122+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939122+0000","last_undegraded":"2024-03-06T07:21:56.939122+0000","last_fullsized":"2024-03-06T07:21:56.939122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937384+0000","last_change":"2024-03-06T07:21:56.937384+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937384+0000","last_undegraded":"2024-03-06T07:21:56.937384+0000","last_fullsized":"2024-03-06T07:21:56.937384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936846+0000","last_change":"2024-03-06T07:21:56.936846+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936846+0000","last_undegraded":"2024-03-06T07:21:56.936846+0000","last_fullsized":"2024-03-06T07:21:56.936846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938525+0000","last_change":"2024-03-06T07:21:56.938525+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938525+0000","last_undegraded":"2024-03-06T07:21:56.938525+0000","last_fullsized":"2024-03-06T07:21:56.938525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934028+0000","last_change":"2024-03-06T07:21:56.934028+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934028+0000","last_undegraded":"2024-03-06T07:21:56.934028+0000","last_fullsized":"2024-03-06T07:21:56.934028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937283+0000","last_change":"2024-03-06T07:21:56.937283+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937283+0000","last_undegraded":"2024-03-06T07:21:56.937283+0000","last_fullsized":"2024-03-06T07:21:56.937283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937385+0000","last_change":"2024-03-06T07:21:56.937385+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937385+0000","last_undegraded":"2024-03-06T07:21:56.937385+0000","last_fullsized":"2024-03-06T07:21:56.937385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938407+0000","last_change":"2024-03-06T07:21:56.938407+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938407+0000","last_undegraded":"2024-03-06T07:21:56.938407+0000","last_fullsized":"2024-03-06T07:21:56.938407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937833+0000","last_change":"2024-03-06T07:21:56.937833+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937833+0000","last_undegraded":"2024-03-06T07:21:56.937833+0000","last_fullsized":"2024-03-06T07:21:56.937833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937909+0000","last_change":"2024-03-06T07:21:56.937909+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937909+0000","last_undegraded":"2024-03-06T07:21:56.937909+0000","last_fullsized":"2024-03-06T07:21:56.937909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937270+0000","last_change":"2024-03-06T07:21:56.937270+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937270+0000","last_undegraded":"2024-03-06T07:21:56.937270+0000","last_fullsized":"2024-03-06T07:21:56.937270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937599+0000","last_change":"2024-03-06T07:21:56.937599+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937599+0000","last_undegraded":"2024-03-06T07:21:56.937599+0000","last_fullsized":"2024-03-06T07:21:56.937599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936300+0000","last_change":"2024-03-06T07:21:56.936300+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936300+0000","last_undegraded":"2024-03-06T07:21:56.936300+0000","last_fullsized":"2024-03-06T07:21:56.936300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":761856,"data_stored":640806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":155648,"data_stored":45011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31340,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340500,"statfs":{"total":96636764160,"available":96604672000,"internally_reserved":0,"allocated":155648,"data_stored":45011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":761856,"data_stored":640806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:01.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:02.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.052+0000 7fd437e4e640 1 Processor -- start 2024-03-06T07:22:02.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.052+0000 7fd437e4e640 1 -- start start 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd437e4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd437e4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd4300cf850 0x7fd4300cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd437e4e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4300d09e0 con 0x7fd4300ce1b0 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd437e4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd43006ca90 con 0x7fd4300cf850 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd437e4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43006df30 con 0x7fd4300d00c0 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd4363eb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd4300ce1b0 0x7fd4300ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35488/0 (socket says 172.21.15.77:35488) 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd4363eb640 1 -- 172.21.15.77:0/3339949044 learned_addr learned my addr 172.21.15.77:0/3339949044 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd4353e9640 1 --2- 172.21.15.77:0/3339949044 >> v2:172.21.15.158:3300/0 conn(0x7fd4300cf850 0x7fd4300cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd435bea640 1 --2- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd434be8640 1 -- 172.21.15.77:0/3339949044 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855009130 0 0) 0x7fd4300d09e0 con 0x7fd4300ce1b0 2024-03-06T07:22:02.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd434be8640 1 -- 172.21.15.77:0/3339949044 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd41c003520 con 0x7fd4300ce1b0 2024-03-06T07:22:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd4353e9640 1 -- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 msgr2=0x7fd4300d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd4353e9640 1 --2- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.053+0000 7fd4353e9640 1 -- 172.21.15.77:0/3339949044 >> v1:172.21.15.77:6789/0 conn(0x7fd4300ce1b0 legacy=0x7fd4300ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.054+0000 7fd4353e9640 1 -- 172.21.15.77:0/3339949044 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43015e380 con 0x7fd4300cf850 2024-03-06T07:22:02.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.054+0000 7fd435bea640 1 --2- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.054+0000 7fd4353e9640 1 --2- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 0x7fd4300cd3c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd4200097a0 tx=0x7fd42002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6442976d8ebe7d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.054+0000 7fd434be8640 1 -- 172.21.15.77:0/3339949044 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd420035020 con 0x7fd4300cf850 2024-03-06T07:22:02.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.054+0000 7fd434be8640 1 -- 172.21.15.77:0/3339949044 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd420031830 con 0x7fd4300cf850 2024-03-06T07:22:02.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.054+0000 7fd434be8640 1 -- 172.21.15.77:0/3339949044 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd420031c00 con 0x7fd4300cf850 2024-03-06T07:22:02.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.055+0000 7fd437e4e640 1 -- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 msgr2=0x7fd4300cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.055+0000 7fd437e4e640 1 --2- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 0x7fd4300cd3c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd4200097a0 tx=0x7fd42002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.055+0000 7fd437e4e640 1 -- 172.21.15.77:0/3339949044 shutdown_connections 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.055+0000 7fd437e4e640 1 --2- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 0x7fd4300cd3c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.055+0000 7fd437e4e640 1 --2- 172.21.15.77:0/3339949044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.055+0000 7fd437e4e640 1 -- 172.21.15.77:0/3339949044 >> 172.21.15.77:0/3339949044 conn(0x7fd4300b35a0 msgr2=0x7fd4300b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.055+0000 7fd437e4e640 1 -- 172.21.15.77:0/3339949044 shutdown_connections 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.055+0000 7fd437e4e640 1 -- 172.21.15.77:0/3339949044 wait complete. 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.056+0000 7fd437e4e640 1 Processor -- start 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.056+0000 7fd437e4e640 1 -- start start 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd437e4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4300ce1b0 0x7fd430151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd437e4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 0x7fd4300c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd437e4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd437e4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43006c440 con 0x7fd4300ce1b0 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd437e4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43006c160 con 0x7fd4300cf850 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd437e4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43006c720 con 0x7fd4300d00c0 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4300ce1b0 0x7fd430151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4300ce1b0 0x7fd430151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41238/0 (socket says 172.21.15.77:41238) 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 -- 172.21.15.77:0/2089461389 learned_addr learned my addr 172.21.15.77:0/2089461389 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd4363eb640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd4353e9640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 0x7fd4300c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 -- 172.21.15.77:0/2089461389 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 msgr2=0x7fd4300c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 -- 172.21.15.77:0/2089461389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 msgr2=0x7fd4300c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 0x7fd4300c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 -- 172.21.15.77:0/2089461389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4300c60d0 con 0x7fd4300ce1b0 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd435bea640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4300ce1b0 0x7fd430151fa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e960 tx=0x7fd42c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.057+0000 7fd4267fc640 1 -- 172.21.15.77:0/2089461389 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c017070 con 0x7fd4300ce1b0 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.058+0000 7fd4267fc640 1 -- 172.21.15.77:0/2089461389 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c00ca00 con 0x7fd4300ce1b0 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.058+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4300c63b0 con 0x7fd4300ce1b0 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.058+0000 7fd4267fc640 1 -- 172.21.15.77:0/2089461389 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c00cdd0 con 0x7fd4300ce1b0 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.058+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4300c6970 con 0x7fd4300ce1b0 2024-03-06T07:22:02.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.058+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd430068150 con 0x7fd4300ce1b0 2024-03-06T07:22:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.059+0000 7fd4267fc640 1 -- 172.21.15.77:0/2089461389 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd42c01a070 con 0x7fd4300ce1b0 2024-03-06T07:22:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.059+0000 7fd4267fc640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd41406b500 0x7fd41406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.059+0000 7fd4267fc640 1 -- 172.21.15.77:0/2089461389 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd42c08a410 con 0x7fd4300ce1b0 2024-03-06T07:22:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.059+0000 7fd4353e9640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd41406b500 0x7fd41406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.060+0000 7fd4353e9640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd41406b500 0x7fd41406d9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd420036000 tx=0x7fd420008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.064+0000 7fd4267fc640 1 -- 172.21.15.77:0/2089461389 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd430068150 con 0x7fd4300ce1b0 2024-03-06T07:22:02.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.231+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd43004f330 con 0x7fd41406b500 2024-03-06T07:22:02.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.236+0000 7fd4267fc640 1 -- 172.21.15.77:0/2089461389 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240003 (secure 0 0 0) 0x7fd4200849a0 con 0x7fd41406b500 2024-03-06T07:22:02.241 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:02.242 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd41406b500 msgr2=0x7fd41406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd41406b500 0x7fd41406d9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd420036000 tx=0x7fd420008030 comp rx=0 tx=0).stop 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4300ce1b0 msgr2=0x7fd430151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4300ce1b0 0x7fd430151fa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e960 tx=0x7fd42c00ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 shutdown_connections 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4300d00c0 0x7fd4300c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4300cf850 0x7fd4300c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd41406b500 0x7fd41406d9c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 --2- 172.21.15.77:0/2089461389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4300ce1b0 0x7fd430151fa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 >> 172.21.15.77:0/2089461389 conn(0x7fd4300b35a0 msgr2=0x7fd4300d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 shutdown_connections 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.239+0000 7fd437e4e640 1 -- 172.21.15.77:0/2089461389 wait complete. 2024-03-06T07:22:02.252 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-06T07:22:01.384268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236144,"kb_used_data":2448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738576,"statfs":{"total":773094113280,"available":772852301824,"internally_reserved":0,"allocated":2506752,"data_stored":1573386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.446341"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938022+0000","last_change":"2024-03-06T07:21:56.938022+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938022+0000","last_undegraded":"2024-03-06T07:21:56.938022+0000","last_fullsized":"2024-03-06T07:21:56.938022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935261+0000","last_change":"2024-03-06T07:21:56.935261+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935261+0000","last_undegraded":"2024-03-06T07:21:56.935261+0000","last_fullsized":"2024-03-06T07:21:56.935261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937834+0000","last_change":"2024-03-06T07:21:56.937834+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937834+0000","last_undegraded":"2024-03-06T07:21:56.937834+0000","last_fullsized":"2024-03-06T07:21:56.937834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939073+0000","last_change":"2024-03-06T07:21:56.939073+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939073+0000","last_undegraded":"2024-03-06T07:21:56.939073+0000","last_fullsized":"2024-03-06T07:21:56.939073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935466+0000","last_change":"2024-03-06T07:21:56.935466+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935466+0000","last_undegraded":"2024-03-06T07:21:56.935466+0000","last_fullsized":"2024-03-06T07:21:56.935466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939057+0000","last_change":"2024-03-06T07:21:56.939057+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939057+0000","last_undegraded":"2024-03-06T07:21:56.939057+0000","last_fullsized":"2024-03-06T07:21:56.939057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.943522+0000","last_change":"2024-03-06T07:21:56.943522+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.943522+0000","last_undegraded":"2024-03-06T07:21:56.943522+0000","last_fullsized":"2024-03-06T07:21:56.943522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934052+0000","last_change":"2024-03-06T07:21:56.934052+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934052+0000","last_undegraded":"2024-03-06T07:21:56.934052+0000","last_fullsized":"2024-03-06T07:21:56.934052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936360+0000","last_change":"2024-03-06T07:21:56.936360+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936360+0000","last_undegraded":"2024-03-06T07:21:56.936360+0000","last_fullsized":"2024-03-06T07:21:56.936360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939604+0000","last_change":"2024-03-06T07:21:56.939604+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939604+0000","last_undegraded":"2024-03-06T07:21:56.939604+0000","last_fullsized":"2024-03-06T07:21:56.939604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934036+0000","last_change":"2024-03-06T07:21:56.934036+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934036+0000","last_undegraded":"2024-03-06T07:21:56.934036+0000","last_fullsized":"2024-03-06T07:21:56.934036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935178+0000","last_change":"2024-03-06T07:21:56.935178+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935178+0000","last_undegraded":"2024-03-06T07:21:56.935178+0000","last_fullsized":"2024-03-06T07:21:56.935178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934025+0000","last_change":"2024-03-06T07:21:56.934025+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934025+0000","last_undegraded":"2024-03-06T07:21:56.934025+0000","last_fullsized":"2024-03-06T07:21:56.934025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934892+0000","last_change":"2024-03-06T07:21:56.934892+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934892+0000","last_undegraded":"2024-03-06T07:21:56.934892+0000","last_fullsized":"2024-03-06T07:21:56.934892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938444+0000","last_change":"2024-03-06T07:21:56.938444+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938444+0000","last_undegraded":"2024-03-06T07:21:56.938444+0000","last_fullsized":"2024-03-06T07:21:56.938444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934948+0000","last_change":"2024-03-06T07:21:56.934948+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934948+0000","last_undegraded":"2024-03-06T07:21:56.934948+0000","last_fullsized":"2024-03-06T07:21:56.934948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.938785+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:21:56.938785+0000","last_peered":"2024-03-06T07:21:56.938785+0000","last_clean":"2024-03-06T07:21:56.938785+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:21:56.938785+0000","last_undegraded":"2024-03-06T07:21:56.938785+0000","last_fullsized":"2024-03-06T07:21:56.938785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937756+0000","last_change":"2024-03-06T07:21:56.937756+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937756+0000","last_undegraded":"2024-03-06T07:21:56.937756+0000","last_fullsized":"2024-03-06T07:21:56.937756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936880+0000","last_change":"2024-03-06T07:21:56.936880+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936880+0000","last_undegraded":"2024-03-06T07:21:56.936880+0000","last_fullsized":"2024-03-06T07:21:56.936880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935141+0000","last_change":"2024-03-06T07:21:56.935141+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935141+0000","last_undegraded":"2024-03-06T07:21:56.935141+0000","last_fullsized":"2024-03-06T07:21:56.935141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939122+0000","last_change":"2024-03-06T07:21:56.939122+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939122+0000","last_undegraded":"2024-03-06T07:21:56.939122+0000","last_fullsized":"2024-03-06T07:21:56.939122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937384+0000","last_change":"2024-03-06T07:21:56.937384+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937384+0000","last_undegraded":"2024-03-06T07:21:56.937384+0000","last_fullsized":"2024-03-06T07:21:56.937384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936846+0000","last_change":"2024-03-06T07:21:56.936846+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936846+0000","last_undegraded":"2024-03-06T07:21:56.936846+0000","last_fullsized":"2024-03-06T07:21:56.936846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938525+0000","last_change":"2024-03-06T07:21:56.938525+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938525+0000","last_undegraded":"2024-03-06T07:21:56.938525+0000","last_fullsized":"2024-03-06T07:21:56.938525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934028+0000","last_change":"2024-03-06T07:21:56.934028+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934028+0000","last_undegraded":"2024-03-06T07:21:56.934028+0000","last_fullsized":"2024-03-06T07:21:56.934028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937283+0000","last_change":"2024-03-06T07:21:56.937283+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937283+0000","last_undegraded":"2024-03-06T07:21:56.937283+0000","last_fullsized":"2024-03-06T07:21:56.937283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937385+0000","last_change":"2024-03-06T07:21:56.937385+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937385+0000","last_undegraded":"2024-03-06T07:21:56.937385+0000","last_fullsized":"2024-03-06T07:21:56.937385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938407+0000","last_change":"2024-03-06T07:21:56.938407+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938407+0000","last_undegraded":"2024-03-06T07:21:56.938407+0000","last_fullsized":"2024-03-06T07:21:56.938407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937833+0000","last_change":"2024-03-06T07:21:56.937833+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937833+0000","last_undegraded":"2024-03-06T07:21:56.937833+0000","last_fullsized":"2024-03-06T07:21:56.937833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937909+0000","last_change":"2024-03-06T07:21:56.937909+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937909+0000","last_undegraded":"2024-03-06T07:21:56.937909+0000","last_fullsized":"2024-03-06T07:21:56.937909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937270+0000","last_change":"2024-03-06T07:21:56.937270+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937270+0000","last_undegraded":"2024-03-06T07:21:56.937270+0000","last_fullsized":"2024-03-06T07:21:56.937270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937599+0000","last_change":"2024-03-06T07:21:56.937599+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937599+0000","last_undegraded":"2024-03-06T07:21:56.937599+0000","last_fullsized":"2024-03-06T07:21:56.937599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936300+0000","last_change":"2024-03-06T07:21:56.936300+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936300+0000","last_undegraded":"2024-03-06T07:21:56.936300+0000","last_fullsized":"2024-03-06T07:21:56.936300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":761856,"data_stored":640806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":155648,"data_stored":45011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31340,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340500,"statfs":{"total":96636764160,"available":96604672000,"internally_reserved":0,"allocated":155648,"data_stored":45011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":761856,"data_stored":640806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:02.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.391+0000 7fc35eda0640 1 Processor -- start 2024-03-06T07:22:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35eda0640 1 -- start start 2024-03-06T07:22:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35eda0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc35815a9c0 0x7fc3580d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35eda0640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35806b710 con 0x7fc35806afb0 2024-03-06T07:22:02.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35eda0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35806b9f0 con 0x7fc35815b4b0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35eda0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35806bcd0 con 0x7fc35815a9c0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35dd9e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc35815a9c0 0x7fc3580d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35d59d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc35806afb0 0x7fc35815ada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35490/0 (socket says 172.21.15.77:35490) 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35d59d640 1 -- 172.21.15.77:0/2203798440 learned_addr learned my addr 172.21.15.77:0/2203798440 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.392+0000 7fc35e59f640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc35815b4b0 0x7fc35806ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58754/0 (socket says 172.21.15.77:58754) 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.393+0000 7fc35dd9e640 1 -- 172.21.15.77:0/2203798440 >> v1:172.21.15.158:6789/0 conn(0x7fc35815b4b0 legacy=0x7fc35806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.393+0000 7fc35dd9e640 1 -- 172.21.15.77:0/2203798440 >> v1:172.21.15.77:6789/0 conn(0x7fc35806afb0 legacy=0x7fc35815ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.393+0000 7fc35cd9c640 1 -- 172.21.15.77:0/2203798440 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602186029 0 0) 0x7fc35806b9f0 con 0x7fc35815b4b0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.393+0000 7fc35dd9e640 1 -- 172.21.15.77:0/2203798440 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35806bfb0 con 0x7fc35815a9c0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.393+0000 7fc35dd9e640 1 --2- 172.21.15.77:0/2203798440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580d0aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc354009d90 tx=0x7fc35402bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a8b62201f8b6c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35cd9c640 1 -- 172.21.15.77:0/2203798440 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354039070 con 0x7fc35815a9c0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35cd9c640 1 -- 172.21.15.77:0/2203798440 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35402ca20 con 0x7fc35815a9c0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35cd9c640 1 -- 172.21.15.77:0/2203798440 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35402cdf0 con 0x7fc35815a9c0 2024-03-06T07:22:02.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35eda0640 1 -- 172.21.15.77:0/2203798440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 msgr2=0x7fc3580d0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35eda0640 1 --2- 172.21.15.77:0/2203798440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580d0aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc354009d90 tx=0x7fc35402bcf0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35eda0640 1 -- 172.21.15.77:0/2203798440 shutdown_connections 2024-03-06T07:22:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35eda0640 1 --2- 172.21.15.77:0/2203798440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580d0aa0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35eda0640 1 -- 172.21.15.77:0/2203798440 >> 172.21.15.77:0/2203798440 conn(0x7fc3580b21c0 msgr2=0x7fc3580cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35eda0640 1 -- 172.21.15.77:0/2203798440 shutdown_connections 2024-03-06T07:22:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.394+0000 7fc35eda0640 1 -- 172.21.15.77:0/2203798440 wait complete. 2024-03-06T07:22:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.395+0000 7fc35eda0640 1 Processor -- start 2024-03-06T07:22:02.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.395+0000 7fc35eda0640 1 -- start start 2024-03-06T07:22:02.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35eda0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc35806afb0 0x7fc35809d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35eda0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35eda0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc35815b4b0 0x7fc35809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35eda0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35806c7b0 con 0x7fc35815a9c0 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35eda0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35806b410 con 0x7fc35806afb0 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35eda0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580c8620 con 0x7fc35815b4b0 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35d59d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35d59d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41264/0 (socket says 172.21.15.77:41264) 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35d59d640 1 -- 172.21.15.77:0/4214817858 learned_addr learned my addr 172.21.15.77:0/4214817858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35dd9e640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc35806afb0 0x7fc35809d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.396+0000 7fc35e59f640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc35815b4b0 0x7fc35809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc35d59d640 1 -- 172.21.15.77:0/4214817858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc35815b4b0 msgr2=0x7fc35809ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc35d59d640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc35815b4b0 0x7fc35809ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc35d59d640 1 -- 172.21.15.77:0/4214817858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc35806afb0 msgr2=0x7fc35809d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc35d59d640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc35806afb0 0x7fc35809d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc35d59d640 1 -- 172.21.15.77:0/4214817858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3580c8850 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc35d59d640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580c80e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc34800c780 tx=0x7fc34800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc34e7fc640 1 -- 172.21.15.77:0/4214817858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348018070 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc34e7fc640 1 -- 172.21.15.77:0/4214817858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3480072f0 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3580c8b30 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc34e7fc640 1 -- 172.21.15.77:0/4214817858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3480047e0 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.397+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3580c90f0 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.398+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc358068f80 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.398+0000 7fc34e7fc640 1 -- 172.21.15.77:0/4214817858 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc34801b070 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.399+0000 7fc34e7fc640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.399+0000 7fc35dd9e640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.399+0000 7fc34e7fc640 1 -- 172.21.15.77:0/4214817858 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc348089610 con 0x7fc35815a9c0 2024-03-06T07:22:02.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.399+0000 7fc35dd9e640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33c06b4d0 0x7fc33c06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc354036000 tx=0x7fc354030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.403+0000 7fc34e7fc640 1 -- 172.21.15.77:0/4214817858 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc348057e40 con 0x7fc35815a9c0 2024-03-06T07:22:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.572+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3580d0810 con 0x7fc33c06b4d0 2024-03-06T07:22:02.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.578+0000 7fc34e7fc640 1 -- 172.21.15.77:0/4214817858 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240003 (secure 0 0 0) 0x7fc3580d0810 con 0x7fc33c06b4d0 2024-03-06T07:22:02.582 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33c06b4d0 msgr2=0x7fc33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33c06b4d0 0x7fc33c06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc354036000 tx=0x7fc354030000 comp rx=0 tx=0).stop 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 msgr2=0x7fc3580c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580c80e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc34800c780 tx=0x7fc34800cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 shutdown_connections 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc35815a9c0 0x7fc3580c80e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc35815b4b0 0x7fc35809ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 --2- 172.21.15.77:0/4214817858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc35806afb0 0x7fc35809d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.580+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 >> 172.21.15.77:0/4214817858 conn(0x7fc3580b21c0 msgr2=0x7fc35815d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.581+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 shutdown_connections 2024-03-06T07:22:02.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.581+0000 7fc35eda0640 1 -- 172.21.15.77:0/4214817858 wait complete. 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-06T07:22:01.384268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236144,"kb_used_data":2448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738576,"statfs":{"total":773094113280,"available":772852301824,"internally_reserved":0,"allocated":2506752,"data_stored":1573386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.446341"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938022+0000","last_change":"2024-03-06T07:21:56.938022+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938022+0000","last_undegraded":"2024-03-06T07:21:56.938022+0000","last_fullsized":"2024-03-06T07:21:56.938022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935261+0000","last_change":"2024-03-06T07:21:56.935261+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935261+0000","last_undegraded":"2024-03-06T07:21:56.935261+0000","last_fullsized":"2024-03-06T07:21:56.935261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937834+0000","last_change":"2024-03-06T07:21:56.937834+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937834+0000","last_undegraded":"2024-03-06T07:21:56.937834+0000","last_fullsized":"2024-03-06T07:21:56.937834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939073+0000","last_change":"2024-03-06T07:21:56.939073+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939073+0000","last_undegraded":"2024-03-06T07:21:56.939073+0000","last_fullsized":"2024-03-06T07:21:56.939073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935466+0000","last_change":"2024-03-06T07:21:56.935466+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935466+0000","last_undegraded":"2024-03-06T07:21:56.935466+0000","last_fullsized":"2024-03-06T07:21:56.935466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939057+0000","last_change":"2024-03-06T07:21:56.939057+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939057+0000","last_undegraded":"2024-03-06T07:21:56.939057+0000","last_fullsized":"2024-03-06T07:21:56.939057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.943522+0000","last_change":"2024-03-06T07:21:56.943522+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.943522+0000","last_undegraded":"2024-03-06T07:21:56.943522+0000","last_fullsized":"2024-03-06T07:21:56.943522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934052+0000","last_change":"2024-03-06T07:21:56.934052+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934052+0000","last_undegraded":"2024-03-06T07:21:56.934052+0000","last_fullsized":"2024-03-06T07:21:56.934052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936360+0000","last_change":"2024-03-06T07:21:56.936360+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936360+0000","last_undegraded":"2024-03-06T07:21:56.936360+0000","last_fullsized":"2024-03-06T07:21:56.936360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939604+0000","last_change":"2024-03-06T07:21:56.939604+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939604+0000","last_undegraded":"2024-03-06T07:21:56.939604+0000","last_fullsized":"2024-03-06T07:21:56.939604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934036+0000","last_change":"2024-03-06T07:21:56.934036+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934036+0000","last_undegraded":"2024-03-06T07:21:56.934036+0000","last_fullsized":"2024-03-06T07:21:56.934036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935178+0000","last_change":"2024-03-06T07:21:56.935178+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935178+0000","last_undegraded":"2024-03-06T07:21:56.935178+0000","last_fullsized":"2024-03-06T07:21:56.935178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934025+0000","last_change":"2024-03-06T07:21:56.934025+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934025+0000","last_undegraded":"2024-03-06T07:21:56.934025+0000","last_fullsized":"2024-03-06T07:21:56.934025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934892+0000","last_change":"2024-03-06T07:21:56.934892+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934892+0000","last_undegraded":"2024-03-06T07:21:56.934892+0000","last_fullsized":"2024-03-06T07:21:56.934892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938444+0000","last_change":"2024-03-06T07:21:56.938444+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938444+0000","last_undegraded":"2024-03-06T07:21:56.938444+0000","last_fullsized":"2024-03-06T07:21:56.938444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934948+0000","last_change":"2024-03-06T07:21:56.934948+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934948+0000","last_undegraded":"2024-03-06T07:21:56.934948+0000","last_fullsized":"2024-03-06T07:21:56.934948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.938785+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:21:56.938785+0000","last_peered":"2024-03-06T07:21:56.938785+0000","last_clean":"2024-03-06T07:21:56.938785+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:21:56.938785+0000","last_undegraded":"2024-03-06T07:21:56.938785+0000","last_fullsized":"2024-03-06T07:21:56.938785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937756+0000","last_change":"2024-03-06T07:21:56.937756+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937756+0000","last_undegraded":"2024-03-06T07:21:56.937756+0000","last_fullsized":"2024-03-06T07:21:56.937756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936880+0000","last_change":"2024-03-06T07:21:56.936880+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936880+0000","last_undegraded":"2024-03-06T07:21:56.936880+0000","last_fullsized":"2024-03-06T07:21:56.936880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935141+0000","last_change":"2024-03-06T07:21:56.935141+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935141+0000","last_undegraded":"2024-03-06T07:21:56.935141+0000","last_fullsized":"2024-03-06T07:21:56.935141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939122+0000","last_change":"2024-03-06T07:21:56.939122+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939122+0000","last_undegraded":"2024-03-06T07:21:56.939122+0000","last_fullsized":"2024-03-06T07:21:56.939122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937384+0000","last_change":"2024-03-06T07:21:56.937384+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937384+0000","last_undegraded":"2024-03-06T07:21:56.937384+0000","last_fullsized":"2024-03-06T07:21:56.937384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936846+0000","last_change":"2024-03-06T07:21:56.936846+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936846+0000","last_undegraded":"2024-03-06T07:21:56.936846+0000","last_fullsized":"2024-03-06T07:21:56.936846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938525+0000","last_change":"2024-03-06T07:21:56.938525+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938525+0000","last_undegraded":"2024-03-06T07:21:56.938525+0000","last_fullsized":"2024-03-06T07:21:56.938525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934028+0000","last_change":"2024-03-06T07:21:56.934028+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934028+0000","last_undegraded":"2024-03-06T07:21:56.934028+0000","last_fullsized":"2024-03-06T07:21:56.934028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937283+0000","last_change":"2024-03-06T07:21:56.937283+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937283+0000","last_undegraded":"2024-03-06T07:21:56.937283+0000","last_fullsized":"2024-03-06T07:21:56.937283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937385+0000","last_change":"2024-03-06T07:21:56.937385+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937385+0000","last_undegraded":"2024-03-06T07:21:56.937385+0000","last_fullsized":"2024-03-06T07:21:56.937385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938407+0000","last_change":"2024-03-06T07:21:56.938407+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938407+0000","last_undegraded":"2024-03-06T07:21:56.938407+0000","last_fullsized":"2024-03-06T07:21:56.938407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937833+0000","last_change":"2024-03-06T07:21:56.937833+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937833+0000","last_undegraded":"2024-03-06T07:21:56.937833+0000","last_fullsized":"2024-03-06T07:21:56.937833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937909+0000","last_change":"2024-03-06T07:21:56.937909+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937909+0000","last_undegraded":"2024-03-06T07:21:56.937909+0000","last_fullsized":"2024-03-06T07:21:56.937909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937270+0000","last_change":"2024-03-06T07:21:56.937270+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937270+0000","last_undegraded":"2024-03-06T07:21:56.937270+0000","last_fullsized":"2024-03-06T07:21:56.937270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937599+0000","last_change":"2024-03-06T07:21:56.937599+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937599+0000","last_undegraded":"2024-03-06T07:21:56.937599+0000","last_fullsized":"2024-03-06T07:21:56.937599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936300+0000","last_change":"2024-03-06T07:21:56.936300+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936300+0000","last_undegraded":"2024-03-06T07:21:56.936300+0000","last_fullsized":"2024-03-06T07:21:56.936300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":761856,"data_stored":640806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":155648,"data_stored":45011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31340,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340500,"statfs":{"total":96636764160,"available":96604672000,"internally_reserved":0,"allocated":155648,"data_stored":45011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":761856,"data_stored":640806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:02.597 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3f is not active+clean 2024-03-06T07:22:02.597 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.597 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3e is not active+clean 2024-03-06T07:22:02.597 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.597 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3d is not active+clean 2024-03-06T07:22:02.597 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3c is not active+clean 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3b is not active+clean 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3a is not active+clean 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.39 is not active+clean 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.38 is not active+clean 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.37 is not active+clean 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.36 is not active+clean 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.35 is not active+clean 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.34 is not active+clean 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.33 is not active+clean 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.32 is not active+clean 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.31 is not active+clean 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.30 is not active+clean 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2f is not active+clean 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2e is not active+clean 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2d is not active+clean 2024-03-06T07:22:02.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2c is not active+clean 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2b is not active+clean 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2a is not active+clean 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.29 is not active+clean 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.28 is not active+clean 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.27 is not active+clean 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.26 is not active+clean 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.25 is not active+clean 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.24 is not active+clean 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.23 is not active+clean 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.22 is not active+clean 2024-03-06T07:22:02.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.21 is not active+clean 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.20 is not active+clean 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1a is not active+clean 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1b is not active+clean 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.938022+0000', 'last_change': '2024-03-06T07:21:56.938022+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.938022+0000', 'last_undegraded': '2024-03-06T07:21:56.938022+0000', 'last_fullsized': '2024-03-06T07:21:56.938022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1b is not active+clean 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1a is not active+clean 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.935261+0000', 'last_change': '2024-03-06T07:21:56.935261+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.935261+0000', 'last_undegraded': '2024-03-06T07:21:56.935261+0000', 'last_fullsized': '2024-03-06T07:21:56.935261+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.19 is not active+clean 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937834+0000', 'last_change': '2024-03-06T07:21:56.937834+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937834+0000', 'last_undegraded': '2024-03-06T07:21:56.937834+0000', 'last_fullsized': '2024-03-06T07:21:56.937834+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.18 is not active+clean 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.18 is not active+clean 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.939073+0000', 'last_change': '2024-03-06T07:21:56.939073+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.939073+0000', 'last_undegraded': '2024-03-06T07:21:56.939073+0000', 'last_fullsized': '2024-03-06T07:21:56.939073+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.19 is not active+clean 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.17 is not active+clean 2024-03-06T07:22:02.604 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.935466+0000', 'last_change': '2024-03-06T07:21:56.935466+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.935466+0000', 'last_undegraded': '2024-03-06T07:21:56.935466+0000', 'last_fullsized': '2024-03-06T07:21:56.935466+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.16 is not active+clean 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.16 is not active+clean 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.939057+0000', 'last_change': '2024-03-06T07:21:56.939057+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.939057+0000', 'last_undegraded': '2024-03-06T07:21:56.939057+0000', 'last_fullsized': '2024-03-06T07:21:56.939057+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.17 is not active+clean 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.15 is not active+clean 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.943522+0000', 'last_change': '2024-03-06T07:21:56.943522+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.943522+0000', 'last_undegraded': '2024-03-06T07:21:56.943522+0000', 'last_fullsized': '2024-03-06T07:21:56.943522+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.14 is not active+clean 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.605 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.14 is not active+clean 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.934052+0000', 'last_change': '2024-03-06T07:21:56.934052+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.934052+0000', 'last_undegraded': '2024-03-06T07:21:56.934052+0000', 'last_fullsized': '2024-03-06T07:21:56.934052+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.15 is not active+clean 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.13 is not active+clean 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.936360+0000', 'last_change': '2024-03-06T07:21:56.936360+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.936360+0000', 'last_undegraded': '2024-03-06T07:21:56.936360+0000', 'last_fullsized': '2024-03-06T07:21:56.936360+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.12 is not active+clean 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.12 is not active+clean 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.939604+0000', 'last_change': '2024-03-06T07:21:56.939604+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.939604+0000', 'last_undegraded': '2024-03-06T07:21:56.939604+0000', 'last_fullsized': '2024-03-06T07:21:56.939604+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.13 is not active+clean 2024-03-06T07:22:02.606 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.11 is not active+clean 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.934036+0000', 'last_change': '2024-03-06T07:21:56.934036+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.934036+0000', 'last_undegraded': '2024-03-06T07:21:56.934036+0000', 'last_fullsized': '2024-03-06T07:21:56.934036+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.10 is not active+clean 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.10 is not active+clean 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.935178+0000', 'last_change': '2024-03-06T07:21:56.935178+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.935178+0000', 'last_undegraded': '2024-03-06T07:21:56.935178+0000', 'last_fullsized': '2024-03-06T07:21:56.935178+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.11 is not active+clean 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.f is not active+clean 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.934025+0000', 'last_change': '2024-03-06T07:21:56.934025+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.934025+0000', 'last_undegraded': '2024-03-06T07:21:56.934025+0000', 'last_fullsized': '2024-03-06T07:21:56.934025+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:02.607 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.e is not active+clean 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.e is not active+clean 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.934892+0000', 'last_change': '2024-03-06T07:21:56.934892+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.934892+0000', 'last_undegraded': '2024-03-06T07:21:56.934892+0000', 'last_fullsized': '2024-03-06T07:21:56.934892+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.f is not active+clean 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.d is not active+clean 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.938444+0000', 'last_change': '2024-03-06T07:21:56.938444+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.938444+0000', 'last_undegraded': '2024-03-06T07:21:56.938444+0000', 'last_fullsized': '2024-03-06T07:21:56.938444+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.c is not active+clean 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.c is not active+clean 2024-03-06T07:22:02.608 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.934948+0000', 'last_change': '2024-03-06T07:21:56.934948+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.934948+0000', 'last_undegraded': '2024-03-06T07:21:56.934948+0000', 'last_fullsized': '2024-03-06T07:21:56.934948+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.d is not active+clean 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.3 is not active+clean 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937756+0000', 'last_change': '2024-03-06T07:21:56.937756+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937756+0000', 'last_undegraded': '2024-03-06T07:21:56.937756+0000', 'last_fullsized': '2024-03-06T07:21:56.937756+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2 is not active+clean 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.0 is not active+clean 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.936880+0000', 'last_change': '2024-03-06T07:21:56.936880+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.936880+0000', 'last_undegraded': '2024-03-06T07:21:56.936880+0000', 'last_fullsized': '2024-03-06T07:21:56.936880+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1 is not active+clean 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.609 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1 is not active+clean 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.935141+0000', 'last_change': '2024-03-06T07:21:56.935141+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.935141+0000', 'last_undegraded': '2024-03-06T07:21:56.935141+0000', 'last_fullsized': '2024-03-06T07:21:56.935141+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.0 is not active+clean 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.2 is not active+clean 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.939122+0000', 'last_change': '2024-03-06T07:21:56.939122+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.939122+0000', 'last_undegraded': '2024-03-06T07:21:56.939122+0000', 'last_fullsized': '2024-03-06T07:21:56.939122+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3 is not active+clean 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.4 is not active+clean 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937384+0000', 'last_change': '2024-03-06T07:21:56.937384+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937384+0000', 'last_undegraded': '2024-03-06T07:21:56.937384+0000', 'last_fullsized': '2024-03-06T07:21:56.937384+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.5 is not active+clean 2024-03-06T07:22:02.610 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.5 is not active+clean 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.936846+0000', 'last_change': '2024-03-06T07:21:56.936846+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.936846+0000', 'last_undegraded': '2024-03-06T07:21:56.936846+0000', 'last_fullsized': '2024-03-06T07:21:56.936846+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.4 is not active+clean 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.6 is not active+clean 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.938525+0000', 'last_change': '2024-03-06T07:21:56.938525+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.938525+0000', 'last_undegraded': '2024-03-06T07:21:56.938525+0000', 'last_fullsized': '2024-03-06T07:21:56.938525+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.7 is not active+clean 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.7 is not active+clean 2024-03-06T07:22:02.611 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.934028+0000', 'last_change': '2024-03-06T07:21:56.934028+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.934028+0000', 'last_undegraded': '2024-03-06T07:21:56.934028+0000', 'last_fullsized': '2024-03-06T07:21:56.934028+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.6 is not active+clean 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.8 is not active+clean 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937283+0000', 'last_change': '2024-03-06T07:21:56.937283+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937283+0000', 'last_undegraded': '2024-03-06T07:21:56.937283+0000', 'last_fullsized': '2024-03-06T07:21:56.937283+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.9 is not active+clean 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.9 is not active+clean 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937385+0000', 'last_change': '2024-03-06T07:21:56.937385+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937385+0000', 'last_undegraded': '2024-03-06T07:21:56.937385+0000', 'last_fullsized': '2024-03-06T07:21:56.937385+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.8 is not active+clean 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.612 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.a is not active+clean 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.938407+0000', 'last_change': '2024-03-06T07:21:56.938407+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.938407+0000', 'last_undegraded': '2024-03-06T07:21:56.938407+0000', 'last_fullsized': '2024-03-06T07:21:56.938407+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.b is not active+clean 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.b is not active+clean 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937833+0000', 'last_change': '2024-03-06T07:21:56.937833+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937833+0000', 'last_undegraded': '2024-03-06T07:21:56.937833+0000', 'last_fullsized': '2024-03-06T07:21:56.937833+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.a is not active+clean 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1d is not active+clean 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1c is not active+clean 2024-03-06T07:22:02.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937909+0000', 'last_change': '2024-03-06T07:21:56.937909+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937909+0000', 'last_undegraded': '2024-03-06T07:21:56.937909+0000', 'last_fullsized': '2024-03-06T07:21:56.937909+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1c is not active+clean 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1d is not active+clean 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937270+0000', 'last_change': '2024-03-06T07:21:56.937270+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937270+0000', 'last_undegraded': '2024-03-06T07:21:56.937270+0000', 'last_fullsized': '2024-03-06T07:21:56.937270+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1f is not active+clean 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1e is not active+clean 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.937599+0000', 'last_change': '2024-03-06T07:21:56.937599+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.937599+0000', 'last_undegraded': '2024-03-06T07:21:56.937599+0000', 'last_fullsized': '2024-03-06T07:21:56.937599+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1e is not active+clean 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:21:57.932426+0000', 'last_change': '2024-03-06T07:21:57.932426+0000', 'last_active': '2024-03-06T07:21:57.932426+0000', 'last_peered': '2024-03-06T07:21:57.932426+0000', 'last_clean': '2024-03-06T07:21:57.932426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:57.932426+0000', 'last_undegraded': '2024-03-06T07:21:57.932426+0000', 'last_fullsized': '2024-03-06T07:21:57.932426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:57.932426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:22:02.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1f is not active+clean 2024-03-06T07:22:02.615 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:56.936300+0000', 'last_change': '2024-03-06T07:21:56.936300+0000', 'last_active': '2024-03-06T07:21:56.927366+0000', 'last_peered': '2024-03-06T07:21:56.927366+0000', 'last_clean': '2024-03-06T07:21:56.927366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:56.936300+0000', 'last_undegraded': '2024-03-06T07:21:56.936300+0000', 'last_fullsized': '2024-03-06T07:21:56.936300+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:56.927366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:02.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.731+0000 7fc8af888640 1 Processor -- start 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.731+0000 7fc8af888640 1 -- start start 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8af888640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8a80b4c00 0x7fc8a80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8af888640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8a80b28a0 0x7fc8a80b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8af888640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a815e540 0x7fc8a815e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8af888640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a815e0b0 con 0x7fc8a80b28a0 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8af888640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a815d510 con 0x7fc8a80b4c00 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8af888640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a815d780 con 0x7fc8a815e540 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8ace23640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8a80b28a0 0x7fc8a80b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8ace23640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8a80b28a0 0x7fc8a80b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41278/0 (socket says 172.21.15.77:41278) 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8ace23640 1 -- 172.21.15.77:0/1210149324 learned_addr learned my addr 172.21.15.77:0/1210149324 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8ad624640 1 --2- 172.21.15.77:0/1210149324 >> v2:172.21.15.158:3300/0 conn(0x7fc8a80b4c00 0x7fc8a80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.732+0000 7fc8ade25640 1 --2- 172.21.15.77:0/1210149324 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a815e540 0x7fc8a815e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.733+0000 7fc8ace23640 1 -- 172.21.15.77:0/1210149324 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a815e540 msgr2=0x7fc8a815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.733+0000 7fc8ace23640 1 --2- 172.21.15.77:0/1210149324 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a815e540 0x7fc8a815e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.733+0000 7fc8ace23640 1 -- 172.21.15.77:0/1210149324 >> v2:172.21.15.158:3300/0 conn(0x7fc8a80b4c00 msgr2=0x7fc8a80b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.733+0000 7fc8ace23640 1 --2- 172.21.15.77:0/1210149324 >> v2:172.21.15.158:3300/0 conn(0x7fc8a80b4c00 0x7fc8a80b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.733+0000 7fc8ace23640 1 -- 172.21.15.77:0/1210149324 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a815da60 con 0x7fc8a80b28a0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.733+0000 7fc8ace23640 1 --2- 172.21.15.77:0/1210149324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a80b28a0 0x7fc8a80b2d10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80721d0 tx=0x7fc89c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf79741b1fafc0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.733+0000 7fc8a7fff640 1 -- 172.21.15.77:0/1210149324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c034070 con 0x7fc8a80b28a0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8a7fff640 1 -- 172.21.15.77:0/1210149324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89c030860 con 0x7fc8a80b28a0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8a7fff640 1 -- 172.21.15.77:0/1210149324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c030c30 con 0x7fc8a80b28a0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 -- 172.21.15.77:0/1210149324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a80b28a0 msgr2=0x7fc8a80b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 --2- 172.21.15.77:0/1210149324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a80b28a0 0x7fc8a80b2d10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80721d0 tx=0x7fc89c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 -- 172.21.15.77:0/1210149324 shutdown_connections 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 --2- 172.21.15.77:0/1210149324 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a815e540 0x7fc8a815e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 --2- 172.21.15.77:0/1210149324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a80b28a0 0x7fc8a80b2d10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 --2- 172.21.15.77:0/1210149324 >> v2:172.21.15.158:3300/0 conn(0x7fc8a80b4c00 0x7fc8a80b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 -- 172.21.15.77:0/1210149324 >> 172.21.15.77:0/1210149324 conn(0x7fc8a809a9f0 msgr2=0x7fc8a809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 -- 172.21.15.77:0/1210149324 shutdown_connections 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.734+0000 7fc8af888640 1 -- 172.21.15.77:0/1210149324 wait complete. 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.735+0000 7fc8af888640 1 Processor -- start 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.735+0000 7fc8af888640 1 -- start start 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8af888640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a80b28a0 0x7fc8a80b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8af888640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8a80b4c00 0x7fc8a8155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8af888640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a815e540 0x7fc8a8155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8af888640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a80b33d0 con 0x7fc8a815e540 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8af888640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a80b3250 con 0x7fc8a80b4c00 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8af888640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a815eec0 con 0x7fc8a80b28a0 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ade25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a815e540 0x7fc8a8155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ade25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a815e540 0x7fc8a8155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41290/0 (socket says 172.21.15.77:41290) 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ade25640 1 -- 172.21.15.77:0/187035198 learned_addr learned my addr 172.21.15.77:0/187035198 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ace23640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8a80b4c00 0x7fc8a8155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ad624640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a80b28a0 0x7fc8a80b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ade25640 1 -- 172.21.15.77:0/187035198 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a80b28a0 msgr2=0x7fc8a80b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ade25640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a80b28a0 0x7fc8a80b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ade25640 1 -- 172.21.15.77:0/187035198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8a80b4c00 msgr2=0x7fc8a8155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ade25640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8a80b4c00 0x7fc8a8155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.736+0000 7fc8ade25640 1 -- 172.21.15.77:0/187035198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a8152d60 con 0x7fc8a815e540 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.737+0000 7fc8ade25640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a815e540 0x7fc8a8155e70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000e7f0 tx=0x7fc8a000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.737+0000 7fc8a5ffb640 1 -- 172.21.15.77:0/187035198 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0017070 con 0x7fc8a815e540 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.737+0000 7fc8a5ffb640 1 -- 172.21.15.77:0/187035198 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a000ee40 con 0x7fc8a815e540 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.737+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a80b1090 con 0x7fc8a815e540 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.737+0000 7fc8a5ffb640 1 -- 172.21.15.77:0/187035198 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a000cb80 con 0x7fc8a815e540 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.737+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a8151fa0 con 0x7fc8a815e540 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.738+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a804f330 con 0x7fc8a815e540 2024-03-06T07:22:02.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.738+0000 7fc8a5ffb640 1 -- 172.21.15.77:0/187035198 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8a001a070 con 0x7fc8a815e540 2024-03-06T07:22:02.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.739+0000 7fc8a5ffb640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc89406b4d0 0x7fc89406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.739+0000 7fc8a5ffb640 1 -- 172.21.15.77:0/187035198 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8a008a070 con 0x7fc8a815e540 2024-03-06T07:22:02.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.739+0000 7fc8ad624640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc89406b4d0 0x7fc89406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.739+0000 7fc8ad624640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc89406b4d0 0x7fc89406d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc898011360 tx=0x7fc89800d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.743+0000 7fc8a5ffb640 1 -- 172.21.15.77:0/187035198 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8a00588a0 con 0x7fc8a815e540 2024-03-06T07:22:02.918 INFO:tasks.ceph.mds.b.smithi158.stderr:2024-03-06T07:22:02.915+0000 7f9c6a728640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:02.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.981+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc8a809b3d0 con 0x7fc8a815e540 2024-03-06T07:22:02.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.981+0000 7fc8a5ffb640 1 -- 172.21.15.77:0/187035198 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1107 (secure 0 0 0) 0x7fc8a005c940 con 0x7fc8a815e540 2024-03-06T07:22:02.983 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:02.983 INFO:teuthology.orchestra.run.smithi077.stdout:{"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":21,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":21,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709711,"num_in_osds":8,"osd_in_since":1709709703,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":241811456,"bytes_avail":772852301824,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":6,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":4282}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-03-06T07:21:41.249274+0000","services":{}},"progress_events":{}} 2024-03-06T07:22:02.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc89406b4d0 msgr2=0x7fc89406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc89406b4d0 0x7fc89406d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc898011360 tx=0x7fc89800d040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a815e540 msgr2=0x7fc8a8155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a815e540 0x7fc8a8155e70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000e7f0 tx=0x7fc8a000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 shutdown_connections 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc89406b4d0 0x7fc89406d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8a815e540 0x7fc8a8155e70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8a80b4c00 0x7fc8a8155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 --2- 172.21.15.77:0/187035198 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8a80b28a0 0x7fc8a80b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 >> 172.21.15.77:0/187035198 conn(0x7fc8a809a9f0 msgr2=0x7fc8a8159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 shutdown_connections 2024-03-06T07:22:02.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:02.983+0000 7fc8af888640 1 -- 172.21.15.77:0/187035198 wait complete. 2024-03-06T07:22:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:22:02.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.094+0000 7ff570446640 1 Processor -- start 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.094+0000 7ff570446640 1 -- start start 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.095+0000 7ff570446640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff56815fa10 0x7ff5680ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.095+0000 7ff570446640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5680cebe0 con 0x7ff5680d16e0 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.095+0000 7ff570446640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5680ceec0 con 0x7ff56815bdd0 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.095+0000 7ff570446640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5680cf1a0 con 0x7ff56815fa10 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.095+0000 7ff56d9e1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff56815fa10 0x7ff5680ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.095+0000 7ff56e9e3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff5680d16e0 0x7ff5680d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35492/0 (socket says 172.21.15.77:35492) 2024-03-06T07:22:03.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.095+0000 7ff56e9e3640 1 -- 172.21.15.77:0/1724866800 learned_addr learned my addr 172.21.15.77:0/1724866800 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.096+0000 7ff56d9e1640 1 -- 172.21.15.77:0/1724866800 >> v1:172.21.15.158:6789/0 conn(0x7ff56815bdd0 legacy=0x7ff5680c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.096+0000 7ff56d9e1640 1 -- 172.21.15.77:0/1724866800 >> v1:172.21.15.77:6789/0 conn(0x7ff5680d16e0 legacy=0x7ff5680d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.096+0000 7ff56d1e0640 1 -- 172.21.15.77:0/1724866800 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186862041 0 0) 0x7ff5680ceec0 con 0x7ff56815bdd0 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.096+0000 7ff56d9e1640 1 -- 172.21.15.77:0/1724866800 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5680cc8e0 con 0x7ff56815fa10 2024-03-06T07:22:03.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.096+0000 7ff56d9e1640 1 --2- 172.21.15.77:0/1724866800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815fa10 0x7ff5680ce270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff5580097a0 tx=0x7ff55802fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbd5a72f405bfd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.096+0000 7ff56d1e0640 1 -- 172.21.15.77:0/1724866800 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558031860 con 0x7ff56815fa10 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.096+0000 7ff56d1e0640 1 -- 172.21.15.77:0/1724866800 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff558031a00 con 0x7ff56815fa10 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.096+0000 7ff56d1e0640 1 -- 172.21.15.77:0/1724866800 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558031dd0 con 0x7ff56815fa10 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.097+0000 7ff570446640 1 -- 172.21.15.77:0/1724866800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815fa10 msgr2=0x7ff5680ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.097+0000 7ff570446640 1 --2- 172.21.15.77:0/1724866800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815fa10 0x7ff5680ce270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff5580097a0 tx=0x7ff55802fb20 comp rx=0 tx=0).stop 2024-03-06T07:22:03.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.097+0000 7ff570446640 1 -- 172.21.15.77:0/1724866800 shutdown_connections 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.097+0000 7ff570446640 1 --2- 172.21.15.77:0/1724866800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815fa10 0x7ff5680ce270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.097+0000 7ff570446640 1 -- 172.21.15.77:0/1724866800 >> 172.21.15.77:0/1724866800 conn(0x7ff56809a9f0 msgr2=0x7ff56809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.097+0000 7ff570446640 1 -- 172.21.15.77:0/1724866800 shutdown_connections 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.097+0000 7ff570446640 1 -- 172.21.15.77:0/1724866800 wait complete. 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.098+0000 7ff570446640 1 Processor -- start 2024-03-06T07:22:03.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.098+0000 7ff570446640 1 -- start start 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff570446640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5680d16e0 0x7ff568158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff570446640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815bdd0 0x7ff56815e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff570446640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56815fa10 0x7ff568152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff570446640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5680ce7b0 con 0x7ff56815bdd0 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff570446640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56815c1b0 con 0x7ff56815fa10 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff570446640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5680ce930 con 0x7ff5680d16e0 2024-03-06T07:22:03.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815bdd0 0x7ff56815e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815bdd0 0x7ff56815e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41306/0 (socket says 172.21.15.77:41306) 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 -- 172.21.15.77:0/1339780713 learned_addr learned my addr 172.21.15.77:0/1339780713 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56e9e3640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56815fa10 0x7ff568152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56e1e2640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5680d16e0 0x7ff568158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 -- 172.21.15.77:0/1339780713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5680d16e0 msgr2=0x7ff568158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5680d16e0 0x7ff568158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 -- 172.21.15.77:0/1339780713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56815fa10 msgr2=0x7ff568152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56815fa10 0x7ff568152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 -- 172.21.15.77:0/1339780713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56815d750 con 0x7ff56815bdd0 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.099+0000 7ff56d9e1640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815bdd0 0x7ff56815e790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5580098c0 tx=0x7ff558004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.100+0000 7ff55effd640 1 -- 172.21.15.77:0/1339780713 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55804b070 con 0x7ff56815bdd0 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.100+0000 7ff55effd640 1 -- 172.21.15.77:0/1339780713 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55803b400 con 0x7ff56815bdd0 2024-03-06T07:22:03.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.100+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff568153250 con 0x7ff56815bdd0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.100+0000 7ff55effd640 1 -- 172.21.15.77:0/1339780713 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55803b7f0 con 0x7ff56815bdd0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.100+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56815dcb0 con 0x7ff56815bdd0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.101+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5680d0960 con 0x7ff56815bdd0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.101+0000 7ff55effd640 1 -- 172.21.15.77:0/1339780713 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff55803b990 con 0x7ff56815bdd0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.101+0000 7ff55effd640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff54c06b4d0 0x7ff54c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.101+0000 7ff55effd640 1 -- 172.21.15.77:0/1339780713 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5580b00d0 con 0x7ff56815bdd0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.101+0000 7ff56e1e2640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff54c06b4d0 0x7ff54c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.102+0000 7ff56e1e2640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff54c06b4d0 0x7ff54c06d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5640012e0 tx=0x7ff564009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.106+0000 7ff55effd640 1 -- 172.21.15.77:0/1339780713 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff56815d750 con 0x7ff56815bdd0 2024-03-06T07:22:03.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.274+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff56806e0e0 con 0x7ff54c06b4d0 2024-03-06T07:22:03.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.280+0000 7ff55effd640 1 -- 172.21.15.77:0/1339780713 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240003 (secure 0 0 0) 0x7ff56806e0e0 con 0x7ff54c06b4d0 2024-03-06T07:22:03.284 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:03.285 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff54c06b4d0 msgr2=0x7ff54c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff54c06b4d0 0x7ff54c06d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5640012e0 tx=0x7ff564009040 comp rx=0 tx=0).stop 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815bdd0 msgr2=0x7ff56815e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815bdd0 0x7ff56815e790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5580098c0 tx=0x7ff558004730 comp rx=0 tx=0).stop 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 shutdown_connections 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff54c06b4d0 0x7ff54c06d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56815bdd0 0x7ff56815e790 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56815fa10 0x7ff568152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 --2- 172.21.15.77:0/1339780713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5680d16e0 0x7ff568158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.282+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 >> 172.21.15.77:0/1339780713 conn(0x7ff56809a9f0 msgr2=0x7ff5680c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.283+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 shutdown_connections 2024-03-06T07:22:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:03.283+0000 7ff570446640 1 -- 172.21.15.77:0/1339780713 wait complete. 2024-03-06T07:22:03.295 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-06T07:22:01.384268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236144,"kb_used_data":2448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738576,"statfs":{"total":773094113280,"available":772852301824,"internally_reserved":0,"allocated":2506752,"data_stored":1573386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.446341"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938022+0000","last_change":"2024-03-06T07:21:56.938022+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938022+0000","last_undegraded":"2024-03-06T07:21:56.938022+0000","last_fullsized":"2024-03-06T07:21:56.938022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935261+0000","last_change":"2024-03-06T07:21:56.935261+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935261+0000","last_undegraded":"2024-03-06T07:21:56.935261+0000","last_fullsized":"2024-03-06T07:21:56.935261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937834+0000","last_change":"2024-03-06T07:21:56.937834+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937834+0000","last_undegraded":"2024-03-06T07:21:56.937834+0000","last_fullsized":"2024-03-06T07:21:56.937834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939073+0000","last_change":"2024-03-06T07:21:56.939073+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939073+0000","last_undegraded":"2024-03-06T07:21:56.939073+0000","last_fullsized":"2024-03-06T07:21:56.939073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935466+0000","last_change":"2024-03-06T07:21:56.935466+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935466+0000","last_undegraded":"2024-03-06T07:21:56.935466+0000","last_fullsized":"2024-03-06T07:21:56.935466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939057+0000","last_change":"2024-03-06T07:21:56.939057+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939057+0000","last_undegraded":"2024-03-06T07:21:56.939057+0000","last_fullsized":"2024-03-06T07:21:56.939057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.943522+0000","last_change":"2024-03-06T07:21:56.943522+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.943522+0000","last_undegraded":"2024-03-06T07:21:56.943522+0000","last_fullsized":"2024-03-06T07:21:56.943522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934052+0000","last_change":"2024-03-06T07:21:56.934052+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934052+0000","last_undegraded":"2024-03-06T07:21:56.934052+0000","last_fullsized":"2024-03-06T07:21:56.934052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936360+0000","last_change":"2024-03-06T07:21:56.936360+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936360+0000","last_undegraded":"2024-03-06T07:21:56.936360+0000","last_fullsized":"2024-03-06T07:21:56.936360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939604+0000","last_change":"2024-03-06T07:21:56.939604+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939604+0000","last_undegraded":"2024-03-06T07:21:56.939604+0000","last_fullsized":"2024-03-06T07:21:56.939604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934036+0000","last_change":"2024-03-06T07:21:56.934036+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934036+0000","last_undegraded":"2024-03-06T07:21:56.934036+0000","last_fullsized":"2024-03-06T07:21:56.934036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935178+0000","last_change":"2024-03-06T07:21:56.935178+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935178+0000","last_undegraded":"2024-03-06T07:21:56.935178+0000","last_fullsized":"2024-03-06T07:21:56.935178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934025+0000","last_change":"2024-03-06T07:21:56.934025+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934025+0000","last_undegraded":"2024-03-06T07:21:56.934025+0000","last_fullsized":"2024-03-06T07:21:56.934025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934892+0000","last_change":"2024-03-06T07:21:56.934892+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934892+0000","last_undegraded":"2024-03-06T07:21:56.934892+0000","last_fullsized":"2024-03-06T07:21:56.934892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938444+0000","last_change":"2024-03-06T07:21:56.938444+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938444+0000","last_undegraded":"2024-03-06T07:21:56.938444+0000","last_fullsized":"2024-03-06T07:21:56.938444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934948+0000","last_change":"2024-03-06T07:21:56.934948+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934948+0000","last_undegraded":"2024-03-06T07:21:56.934948+0000","last_fullsized":"2024-03-06T07:21:56.934948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.938785+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:21:56.938785+0000","last_peered":"2024-03-06T07:21:56.938785+0000","last_clean":"2024-03-06T07:21:56.938785+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:21:56.938785+0000","last_undegraded":"2024-03-06T07:21:56.938785+0000","last_fullsized":"2024-03-06T07:21:56.938785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937756+0000","last_change":"2024-03-06T07:21:56.937756+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937756+0000","last_undegraded":"2024-03-06T07:21:56.937756+0000","last_fullsized":"2024-03-06T07:21:56.937756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936880+0000","last_change":"2024-03-06T07:21:56.936880+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936880+0000","last_undegraded":"2024-03-06T07:21:56.936880+0000","last_fullsized":"2024-03-06T07:21:56.936880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.935141+0000","last_change":"2024-03-06T07:21:56.935141+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.935141+0000","last_undegraded":"2024-03-06T07:21:56.935141+0000","last_fullsized":"2024-03-06T07:21:56.935141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.939122+0000","last_change":"2024-03-06T07:21:56.939122+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.939122+0000","last_undegraded":"2024-03-06T07:21:56.939122+0000","last_fullsized":"2024-03-06T07:21:56.939122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937384+0000","last_change":"2024-03-06T07:21:56.937384+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937384+0000","last_undegraded":"2024-03-06T07:21:56.937384+0000","last_fullsized":"2024-03-06T07:21:56.937384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936846+0000","last_change":"2024-03-06T07:21:56.936846+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936846+0000","last_undegraded":"2024-03-06T07:21:56.936846+0000","last_fullsized":"2024-03-06T07:21:56.936846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938525+0000","last_change":"2024-03-06T07:21:56.938525+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938525+0000","last_undegraded":"2024-03-06T07:21:56.938525+0000","last_fullsized":"2024-03-06T07:21:56.938525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.934028+0000","last_change":"2024-03-06T07:21:56.934028+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.934028+0000","last_undegraded":"2024-03-06T07:21:56.934028+0000","last_fullsized":"2024-03-06T07:21:56.934028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937283+0000","last_change":"2024-03-06T07:21:56.937283+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937283+0000","last_undegraded":"2024-03-06T07:21:56.937283+0000","last_fullsized":"2024-03-06T07:21:56.937283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937385+0000","last_change":"2024-03-06T07:21:56.937385+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937385+0000","last_undegraded":"2024-03-06T07:21:56.937385+0000","last_fullsized":"2024-03-06T07:21:56.937385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.938407+0000","last_change":"2024-03-06T07:21:56.938407+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.938407+0000","last_undegraded":"2024-03-06T07:21:56.938407+0000","last_fullsized":"2024-03-06T07:21:56.938407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937833+0000","last_change":"2024-03-06T07:21:56.937833+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937833+0000","last_undegraded":"2024-03-06T07:21:56.937833+0000","last_fullsized":"2024-03-06T07:21:56.937833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937909+0000","last_change":"2024-03-06T07:21:56.937909+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937909+0000","last_undegraded":"2024-03-06T07:21:56.937909+0000","last_fullsized":"2024-03-06T07:21:56.937909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937270+0000","last_change":"2024-03-06T07:21:56.937270+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937270+0000","last_undegraded":"2024-03-06T07:21:56.937270+0000","last_fullsized":"2024-03-06T07:21:56.937270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.937599+0000","last_change":"2024-03-06T07:21:56.937599+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.937599+0000","last_undegraded":"2024-03-06T07:21:56.937599+0000","last_fullsized":"2024-03-06T07:21:56.937599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:57.932426+0000","last_change":"2024-03-06T07:21:57.932426+0000","last_active":"2024-03-06T07:21:57.932426+0000","last_peered":"2024-03-06T07:21:57.932426+0000","last_clean":"2024-03-06T07:21:57.932426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:57.932426+0000","last_undegraded":"2024-03-06T07:21:57.932426+0000","last_fullsized":"2024-03-06T07:21:57.932426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-03-06T07:21:56.936300+0000","last_change":"2024-03-06T07:21:56.936300+0000","last_active":"2024-03-06T07:21:56.927366+0000","last_peered":"2024-03-06T07:21:56.927366+0000","last_clean":"2024-03-06T07:21:56.927366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:56.936300+0000","last_undegraded":"2024-03-06T07:21:56.936300+0000","last_fullsized":"2024-03-06T07:21:56.936300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":761856,"data_stored":640806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":155648,"data_stored":45011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31340,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340500,"statfs":{"total":96636764160,"available":96604672000,"internally_reserved":0,"allocated":155648,"data_stored":45011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":50438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":761856,"data_stored":640806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:06.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:06.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.411+0000 7f445c9c7640 1 Processor -- start 2024-03-06T07:22:06.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.411+0000 7f445c9c7640 1 -- start start 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.412+0000 7f445c9c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f445815bdd0 0x7f44580c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.412+0000 7f445c9c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 0x7f44580ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.412+0000 7f445c9c7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f44580d16e0 0x7f44580d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.412+0000 7f445c9c7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44580cebe0 con 0x7f445815bdd0 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.412+0000 7f445c9c7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44580ceec0 con 0x7f44580d16e0 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.412+0000 7f445c9c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580cf1a0 con 0x7f445815fa10 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f445659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f445815bdd0 0x7f44580c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f4455d9b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 0x7f44580ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f4456d9d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f44580d16e0 0x7f44580d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f445659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f445815bdd0 0x7f44580c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41308/0 (socket says 172.21.15.77:41308) 2024-03-06T07:22:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f445659c640 1 -- 172.21.15.77:0/52339915 learned_addr learned my addr 172.21.15.77:0/52339915 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f445659c640 1 -- 172.21.15.77:0/52339915 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 msgr2=0x7f44580ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f445659c640 1 --2- 172.21.15.77:0/52339915 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 0x7f44580ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f445659c640 1 -- 172.21.15.77:0/52339915 >> v2:172.21.15.158:3300/0 conn(0x7f44580d16e0 msgr2=0x7f44580d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f445659c640 1 --2- 172.21.15.77:0/52339915 >> v2:172.21.15.158:3300/0 conn(0x7f44580d16e0 0x7f44580d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.413+0000 7f445659c640 1 -- 172.21.15.77:0/52339915 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580cc8e0 con 0x7f445815bdd0 2024-03-06T07:22:06.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445659c640 1 --2- 172.21.15.77:0/52339915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44580c5610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f44580729e0 tx=0x7f444802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91b486277f125a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445559a640 1 -- 172.21.15.77:0/52339915 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444802c9e0 con 0x7f445815bdd0 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445559a640 1 -- 172.21.15.77:0/52339915 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444802cb80 con 0x7f445815bdd0 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445559a640 1 -- 172.21.15.77:0/52339915 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44480343f0 con 0x7f445815bdd0 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 -- 172.21.15.77:0/52339915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 msgr2=0x7f44580c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 --2- 172.21.15.77:0/52339915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44580c5610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f44580729e0 tx=0x7f444802be40 comp rx=0 tx=0).stop 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 -- 172.21.15.77:0/52339915 shutdown_connections 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 --2- 172.21.15.77:0/52339915 >> v2:172.21.15.158:3300/0 conn(0x7f44580d16e0 0x7f44580d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 --2- 172.21.15.77:0/52339915 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 0x7f44580ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 --2- 172.21.15.77:0/52339915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44580c5610 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 -- 172.21.15.77:0/52339915 >> 172.21.15.77:0/52339915 conn(0x7f445809a9f0 msgr2=0x7f445809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 -- 172.21.15.77:0/52339915 shutdown_connections 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.414+0000 7f445c9c7640 1 -- 172.21.15.77:0/52339915 wait complete. 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.416+0000 7f445c9c7640 1 Processor -- start 2024-03-06T07:22:06.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.416+0000 7f445c9c7640 1 -- start start 2024-03-06T07:22:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.416+0000 7f445c9c7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44580d16e0 0x7f44580c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.416+0000 7f445c9c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44581e5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.416+0000 7f445c9c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 0x7f44580cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f445c9c7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44580cc5d0 con 0x7f445815bdd0 2024-03-06T07:22:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f445c9c7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44580d2070 con 0x7f44580d16e0 2024-03-06T07:22:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f445c9c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445815c1b0 con 0x7f445815fa10 2024-03-06T07:22:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44581e5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44581e5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41320/0 (socket says 172.21.15.77:41320) 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 -- 172.21.15.77:0/1773875204 learned_addr learned my addr 172.21.15.77:0/1773875204 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f445659c640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44580d16e0 0x7f44580c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4456d9d640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 0x7f44580cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 -- 172.21.15.77:0/1773875204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 msgr2=0x7f44580cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 0x7f44580cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 -- 172.21.15.77:0/1773875204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44580d16e0 msgr2=0x7f44580c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44580d16e0 0x7f44580c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 -- 172.21.15.77:0/1773875204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44581e6900 con 0x7f445815bdd0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4455d9b640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44581e5fb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e870 tx=0x7f444c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4446ffd640 1 -- 172.21.15.77:0/1773875204 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c017070 con 0x7f445815bdd0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4446ffd640 1 -- 172.21.15.77:0/1773875204 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c00c940 con 0x7f445815bdd0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44581e6be0 con 0x7f445815bdd0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f4446ffd640 1 -- 172.21.15.77:0/1773875204 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c00cd30 con 0x7f445815bdd0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.417+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44581e7480 con 0x7f445815bdd0 2024-03-06T07:22:06.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.418+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44581e6db0 con 0x7f445815bdd0 2024-03-06T07:22:06.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.419+0000 7f4446ffd640 1 -- 172.21.15.77:0/1773875204 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f444c01a070 con 0x7f445815bdd0 2024-03-06T07:22:06.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.419+0000 7f4446ffd640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f443806b4d0 0x7f443806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.419+0000 7f4446ffd640 1 -- 172.21.15.77:0/1773875204 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f444c08a2a0 con 0x7f445815bdd0 2024-03-06T07:22:06.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.419+0000 7f445659c640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f443806b4d0 0x7f443806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.420+0000 7f445659c640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f443806b4d0 0x7f443806d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4448035000 tx=0x7f4448030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.424+0000 7f4446ffd640 1 -- 172.21.15.77:0/1773875204 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f444c058b60 con 0x7f445815bdd0 2024-03-06T07:22:06.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.593+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44580cbfe0 con 0x7f443806b4d0 2024-03-06T07:22:06.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.599+0000 7f4446ffd640 1 -- 172.21.15.77:0/1773875204 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251946 (secure 0 0 0) 0x7f44580cbfe0 con 0x7f443806b4d0 2024-03-06T07:22:06.603 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:06.605 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:22:06.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f443806b4d0 msgr2=0x7f443806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f443806b4d0 0x7f443806d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4448035000 tx=0x7f4448030000 comp rx=0 tx=0).stop 2024-03-06T07:22:06.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 msgr2=0x7f44581e5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44581e5fb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e870 tx=0x7f444c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:06.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 shutdown_connections 2024-03-06T07:22:06.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f443806b4d0 0x7f443806d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445815bdd0 0x7f44581e5fb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815fa10 0x7f44580cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 --2- 172.21.15.77:0/1773875204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44580d16e0 0x7f44580c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 >> 172.21.15.77:0/1773875204 conn(0x7f445809a9f0 msgr2=0x7f445815e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 shutdown_connections 2024-03-06T07:22:06.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.602+0000 7f445c9c7640 1 -- 172.21.15.77:0/1773875204 wait complete. 2024-03-06T07:22:06.615 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:22:05.385030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235860,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738860,"statfs":{"total":773094113280,"available":772852592640,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.447102"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004993+0000","last_change":"2024-03-06T07:21:58.964978+0000","last_active":"2024-03-06T07:22:01.004993+0000","last_peered":"2024-03-06T07:22:01.004993+0000","last_clean":"2024-03-06T07:22:01.004993+0000","last_became_active":"2024-03-06T07:21:58.964298+0000","last_became_peered":"2024-03-06T07:21:58.964298+0000","last_unstale":"2024-03-06T07:22:01.004993+0000","last_undegraded":"2024-03-06T07:22:01.004993+0000","last_fullsized":"2024-03-06T07:22:01.004993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:53.179429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942479+0000","last_change":"2024-03-06T07:21:58.960691+0000","last_active":"2024-03-06T07:21:59.942479+0000","last_peered":"2024-03-06T07:21:59.942479+0000","last_clean":"2024-03-06T07:21:59.942479+0000","last_became_active":"2024-03-06T07:21:58.959088+0000","last_became_peered":"2024-03-06T07:21:58.959088+0000","last_unstale":"2024-03-06T07:21:59.942479+0000","last_undegraded":"2024-03-06T07:21:59.942479+0000","last_fullsized":"2024-03-06T07:21:59.942479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549070+0000","last_change":"2024-03-06T07:21:58.953314+0000","last_active":"2024-03-06T07:22:00.549070+0000","last_peered":"2024-03-06T07:22:00.549070+0000","last_clean":"2024-03-06T07:22:00.549070+0000","last_became_active":"2024-03-06T07:21:58.953048+0000","last_became_peered":"2024-03-06T07:21:58.953048+0000","last_unstale":"2024-03-06T07:22:00.549070+0000","last_undegraded":"2024-03-06T07:22:00.549070+0000","last_fullsized":"2024-03-06T07:22:00.549070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004545+0000","last_change":"2024-03-06T07:21:58.965150+0000","last_active":"2024-03-06T07:22:01.004545+0000","last_peered":"2024-03-06T07:22:01.004545+0000","last_clean":"2024-03-06T07:22:01.004545+0000","last_became_active":"2024-03-06T07:21:58.964307+0000","last_became_peered":"2024-03-06T07:21:58.964307+0000","last_unstale":"2024-03-06T07:22:01.004545+0000","last_undegraded":"2024-03-06T07:22:01.004545+0000","last_fullsized":"2024-03-06T07:22:01.004545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004939+0000","last_change":"2024-03-06T07:21:58.957427+0000","last_active":"2024-03-06T07:22:01.004939+0000","last_peered":"2024-03-06T07:22:01.004939+0000","last_clean":"2024-03-06T07:22:01.004939+0000","last_became_active":"2024-03-06T07:21:58.957124+0000","last_became_peered":"2024-03-06T07:21:58.957124+0000","last_unstale":"2024-03-06T07:22:01.004939+0000","last_undegraded":"2024-03-06T07:22:01.004939+0000","last_fullsized":"2024-03-06T07:22:01.004939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:51.617926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004247+0000","last_change":"2024-03-06T07:21:58.969024+0000","last_active":"2024-03-06T07:22:01.004247+0000","last_peered":"2024-03-06T07:22:01.004247+0000","last_clean":"2024-03-06T07:22:01.004247+0000","last_became_active":"2024-03-06T07:21:58.963242+0000","last_became_peered":"2024-03-06T07:21:58.963242+0000","last_unstale":"2024-03-06T07:22:01.004247+0000","last_undegraded":"2024-03-06T07:22:01.004247+0000","last_fullsized":"2024-03-06T07:22:01.004247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004439+0000","last_change":"2024-03-06T07:21:58.958719+0000","last_active":"2024-03-06T07:22:01.004439+0000","last_peered":"2024-03-06T07:22:01.004439+0000","last_clean":"2024-03-06T07:22:01.004439+0000","last_became_active":"2024-03-06T07:21:58.958025+0000","last_became_peered":"2024-03-06T07:21:58.958025+0000","last_unstale":"2024-03-06T07:22:01.004439+0000","last_undegraded":"2024-03-06T07:22:01.004439+0000","last_fullsized":"2024-03-06T07:22:01.004439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004500+0000","last_change":"2024-03-06T07:21:58.963329+0000","last_active":"2024-03-06T07:22:01.004500+0000","last_peered":"2024-03-06T07:22:01.004500+0000","last_clean":"2024-03-06T07:22:01.004500+0000","last_became_active":"2024-03-06T07:21:58.963004+0000","last_became_peered":"2024-03-06T07:21:58.963004+0000","last_unstale":"2024-03-06T07:22:01.004500+0000","last_undegraded":"2024-03-06T07:22:01.004500+0000","last_fullsized":"2024-03-06T07:22:01.004500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005042+0000","last_change":"2024-03-06T07:21:58.969451+0000","last_active":"2024-03-06T07:22:01.005042+0000","last_peered":"2024-03-06T07:22:01.005042+0000","last_clean":"2024-03-06T07:22:01.005042+0000","last_became_active":"2024-03-06T07:21:58.969086+0000","last_became_peered":"2024-03-06T07:21:58.969086+0000","last_unstale":"2024-03-06T07:22:01.005042+0000","last_undegraded":"2024-03-06T07:22:01.005042+0000","last_fullsized":"2024-03-06T07:22:01.005042+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:00.578026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944576+0000","last_change":"2024-03-06T07:21:58.966581+0000","last_active":"2024-03-06T07:21:59.944576+0000","last_peered":"2024-03-06T07:21:59.944576+0000","last_clean":"2024-03-06T07:21:59.944576+0000","last_became_active":"2024-03-06T07:21:58.966309+0000","last_became_peered":"2024-03-06T07:21:58.966309+0000","last_unstale":"2024-03-06T07:21:59.944576+0000","last_undegraded":"2024-03-06T07:21:59.944576+0000","last_fullsized":"2024-03-06T07:21:59.944576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944493+0000","last_change":"2024-03-06T07:21:58.968147+0000","last_active":"2024-03-06T07:21:59.944493+0000","last_peered":"2024-03-06T07:21:59.944493+0000","last_clean":"2024-03-06T07:21:59.944493+0000","last_became_active":"2024-03-06T07:21:58.967870+0000","last_became_peered":"2024-03-06T07:21:58.967870+0000","last_unstale":"2024-03-06T07:21:59.944493+0000","last_undegraded":"2024-03-06T07:21:59.944493+0000","last_fullsized":"2024-03-06T07:21:59.944493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943944+0000","last_change":"2024-03-06T07:21:58.950183+0000","last_active":"2024-03-06T07:21:59.943944+0000","last_peered":"2024-03-06T07:21:59.943944+0000","last_clean":"2024-03-06T07:21:59.943944+0000","last_became_active":"2024-03-06T07:21:58.949892+0000","last_became_peered":"2024-03-06T07:21:58.949892+0000","last_unstale":"2024-03-06T07:21:59.943944+0000","last_undegraded":"2024-03-06T07:21:59.943944+0000","last_fullsized":"2024-03-06T07:21:59.943944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:41.545324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004776+0000","last_change":"2024-03-06T07:21:58.963200+0000","last_active":"2024-03-06T07:22:01.004776+0000","last_peered":"2024-03-06T07:22:01.004776+0000","last_clean":"2024-03-06T07:22:01.004776+0000","last_became_active":"2024-03-06T07:21:58.962833+0000","last_became_peered":"2024-03-06T07:21:58.962833+0000","last_unstale":"2024-03-06T07:22:01.004776+0000","last_undegraded":"2024-03-06T07:22:01.004776+0000","last_fullsized":"2024-03-06T07:22:01.004776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548832+0000","last_change":"2024-03-06T07:21:58.952781+0000","last_active":"2024-03-06T07:22:00.548832+0000","last_peered":"2024-03-06T07:22:00.548832+0000","last_clean":"2024-03-06T07:22:00.548832+0000","last_became_active":"2024-03-06T07:21:58.952504+0000","last_became_peered":"2024-03-06T07:21:58.952504+0000","last_unstale":"2024-03-06T07:22:00.548832+0000","last_undegraded":"2024-03-06T07:22:00.548832+0000","last_fullsized":"2024-03-06T07:22:00.548832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:25.614483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942188+0000","last_change":"2024-03-06T07:21:58.950082+0000","last_active":"2024-03-06T07:21:59.942188+0000","last_peered":"2024-03-06T07:21:59.942188+0000","last_clean":"2024-03-06T07:21:59.942188+0000","last_became_active":"2024-03-06T07:21:58.949595+0000","last_became_peered":"2024-03-06T07:21:58.949595+0000","last_unstale":"2024-03-06T07:21:59.942188+0000","last_undegraded":"2024-03-06T07:21:59.942188+0000","last_fullsized":"2024-03-06T07:21:59.942188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004141+0000","last_change":"2024-03-06T07:21:58.961225+0000","last_active":"2024-03-06T07:22:01.004141+0000","last_peered":"2024-03-06T07:22:01.004141+0000","last_clean":"2024-03-06T07:22:01.004141+0000","last_became_active":"2024-03-06T07:21:58.960902+0000","last_became_peered":"2024-03-06T07:21:58.960902+0000","last_unstale":"2024-03-06T07:22:01.004141+0000","last_undegraded":"2024-03-06T07:22:01.004141+0000","last_fullsized":"2024-03-06T07:22:01.004141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004963+0000","last_change":"2024-03-06T07:21:58.970233+0000","last_active":"2024-03-06T07:22:01.004963+0000","last_peered":"2024-03-06T07:22:01.004963+0000","last_clean":"2024-03-06T07:22:01.004963+0000","last_became_active":"2024-03-06T07:21:58.969346+0000","last_became_peered":"2024-03-06T07:21:58.969346+0000","last_unstale":"2024-03-06T07:22:01.004963+0000","last_undegraded":"2024-03-06T07:22:01.004963+0000","last_fullsized":"2024-03-06T07:22:01.004963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:42.427321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983310+0000","last_change":"2024-03-06T07:21:58.948614+0000","last_active":"2024-03-06T07:21:59.983310+0000","last_peered":"2024-03-06T07:21:59.983310+0000","last_clean":"2024-03-06T07:21:59.983310+0000","last_became_active":"2024-03-06T07:21:58.948333+0000","last_became_peered":"2024-03-06T07:21:58.948333+0000","last_unstale":"2024-03-06T07:21:59.983310+0000","last_undegraded":"2024-03-06T07:21:59.983310+0000","last_fullsized":"2024-03-06T07:21:59.983310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:42.473136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005274+0000","last_change":"2024-03-06T07:21:58.957566+0000","last_active":"2024-03-06T07:22:01.005274+0000","last_peered":"2024-03-06T07:22:01.005274+0000","last_clean":"2024-03-06T07:22:01.005274+0000","last_became_active":"2024-03-06T07:21:58.957272+0000","last_became_peered":"2024-03-06T07:21:58.957272+0000","last_unstale":"2024-03-06T07:22:01.005274+0000","last_undegraded":"2024-03-06T07:22:01.005274+0000","last_fullsized":"2024-03-06T07:22:01.005274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:13.279685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944669+0000","last_change":"2024-03-06T07:21:58.948431+0000","last_active":"2024-03-06T07:21:59.944669+0000","last_peered":"2024-03-06T07:21:59.944669+0000","last_clean":"2024-03-06T07:21:59.944669+0000","last_became_active":"2024-03-06T07:21:58.947689+0000","last_became_peered":"2024-03-06T07:21:58.947689+0000","last_unstale":"2024-03-06T07:21:59.944669+0000","last_undegraded":"2024-03-06T07:21:59.944669+0000","last_fullsized":"2024-03-06T07:21:59.944669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548975+0000","last_change":"2024-03-06T07:21:58.965611+0000","last_active":"2024-03-06T07:22:00.548975+0000","last_peered":"2024-03-06T07:22:00.548975+0000","last_clean":"2024-03-06T07:22:00.548975+0000","last_became_active":"2024-03-06T07:21:58.965261+0000","last_became_peered":"2024-03-06T07:21:58.965261+0000","last_unstale":"2024-03-06T07:22:00.548975+0000","last_undegraded":"2024-03-06T07:22:00.548975+0000","last_fullsized":"2024-03-06T07:22:00.548975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004252+0000","last_change":"2024-03-06T07:21:58.969256+0000","last_active":"2024-03-06T07:22:01.004252+0000","last_peered":"2024-03-06T07:22:01.004252+0000","last_clean":"2024-03-06T07:22:01.004252+0000","last_became_active":"2024-03-06T07:21:58.968670+0000","last_became_peered":"2024-03-06T07:21:58.968670+0000","last_unstale":"2024-03-06T07:22:01.004252+0000","last_undegraded":"2024-03-06T07:22:01.004252+0000","last_fullsized":"2024-03-06T07:22:01.004252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:25.392050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004556+0000","last_change":"2024-03-06T07:21:58.962061+0000","last_active":"2024-03-06T07:22:01.004556+0000","last_peered":"2024-03-06T07:22:01.004556+0000","last_clean":"2024-03-06T07:22:01.004556+0000","last_became_active":"2024-03-06T07:21:58.961754+0000","last_became_peered":"2024-03-06T07:21:58.961754+0000","last_unstale":"2024-03-06T07:22:01.004556+0000","last_undegraded":"2024-03-06T07:22:01.004556+0000","last_fullsized":"2024-03-06T07:22:01.004556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004201+0000","last_change":"2024-03-06T07:21:58.961280+0000","last_active":"2024-03-06T07:22:01.004201+0000","last_peered":"2024-03-06T07:22:01.004201+0000","last_clean":"2024-03-06T07:22:01.004201+0000","last_became_active":"2024-03-06T07:21:58.960288+0000","last_became_peered":"2024-03-06T07:21:58.960288+0000","last_unstale":"2024-03-06T07:22:01.004201+0000","last_undegraded":"2024-03-06T07:22:01.004201+0000","last_fullsized":"2024-03-06T07:22:01.004201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005074+0000","last_change":"2024-03-06T07:21:58.962545+0000","last_active":"2024-03-06T07:22:01.005074+0000","last_peered":"2024-03-06T07:22:01.005074+0000","last_clean":"2024-03-06T07:22:01.005074+0000","last_became_active":"2024-03-06T07:21:58.962037+0000","last_became_peered":"2024-03-06T07:21:58.962037+0000","last_unstale":"2024-03-06T07:22:01.005074+0000","last_undegraded":"2024-03-06T07:22:01.005074+0000","last_fullsized":"2024-03-06T07:22:01.005074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944648+0000","last_change":"2024-03-06T07:21:58.959744+0000","last_active":"2024-03-06T07:21:59.944648+0000","last_peered":"2024-03-06T07:21:59.944648+0000","last_clean":"2024-03-06T07:21:59.944648+0000","last_became_active":"2024-03-06T07:21:58.959459+0000","last_became_peered":"2024-03-06T07:21:58.959459+0000","last_unstale":"2024-03-06T07:21:59.944648+0000","last_undegraded":"2024-03-06T07:21:59.944648+0000","last_fullsized":"2024-03-06T07:21:59.944648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942254+0000","last_change":"2024-03-06T07:21:58.950538+0000","last_active":"2024-03-06T07:21:59.942254+0000","last_peered":"2024-03-06T07:21:59.942254+0000","last_clean":"2024-03-06T07:21:59.942254+0000","last_became_active":"2024-03-06T07:21:58.950225+0000","last_became_peered":"2024-03-06T07:21:58.950225+0000","last_unstale":"2024-03-06T07:21:59.942254+0000","last_undegraded":"2024-03-06T07:21:59.942254+0000","last_fullsized":"2024-03-06T07:21:59.942254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004948+0000","last_change":"2024-03-06T07:21:58.961060+0000","last_active":"2024-03-06T07:22:01.004948+0000","last_peered":"2024-03-06T07:22:01.004948+0000","last_clean":"2024-03-06T07:22:01.004948+0000","last_became_active":"2024-03-06T07:21:58.960291+0000","last_became_peered":"2024-03-06T07:21:58.960291+0000","last_unstale":"2024-03-06T07:22:01.004948+0000","last_undegraded":"2024-03-06T07:22:01.004948+0000","last_fullsized":"2024-03-06T07:22:01.004948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548986+0000","last_change":"2024-03-06T07:21:58.957460+0000","last_active":"2024-03-06T07:22:00.548986+0000","last_peered":"2024-03-06T07:22:00.548986+0000","last_clean":"2024-03-06T07:22:00.548986+0000","last_became_active":"2024-03-06T07:21:58.957194+0000","last_became_peered":"2024-03-06T07:21:58.957194+0000","last_unstale":"2024-03-06T07:22:00.548986+0000","last_undegraded":"2024-03-06T07:22:00.548986+0000","last_fullsized":"2024-03-06T07:22:00.548986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004388+0000","last_change":"2024-03-06T07:21:58.971347+0000","last_active":"2024-03-06T07:22:01.004388+0000","last_peered":"2024-03-06T07:22:01.004388+0000","last_clean":"2024-03-06T07:22:01.004388+0000","last_became_active":"2024-03-06T07:21:58.970575+0000","last_became_peered":"2024-03-06T07:21:58.970575+0000","last_unstale":"2024-03-06T07:22:01.004388+0000","last_undegraded":"2024-03-06T07:22:01.004388+0000","last_fullsized":"2024-03-06T07:22:01.004388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983258+0000","last_change":"2024-03-06T07:21:58.964710+0000","last_active":"2024-03-06T07:21:59.983258+0000","last_peered":"2024-03-06T07:21:59.983258+0000","last_clean":"2024-03-06T07:21:59.983258+0000","last_became_active":"2024-03-06T07:21:58.964426+0000","last_became_peered":"2024-03-06T07:21:58.964426+0000","last_unstale":"2024-03-06T07:21:59.983258+0000","last_undegraded":"2024-03-06T07:21:59.983258+0000","last_fullsized":"2024-03-06T07:21:59.983258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:54.202847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943842+0000","last_change":"2024-03-06T07:21:58.958320+0000","last_active":"2024-03-06T07:21:59.943842+0000","last_peered":"2024-03-06T07:21:59.943842+0000","last_clean":"2024-03-06T07:21:59.943842+0000","last_became_active":"2024-03-06T07:21:58.958023+0000","last_became_peered":"2024-03-06T07:21:58.958023+0000","last_unstale":"2024-03-06T07:21:59.943842+0000","last_undegraded":"2024-03-06T07:21:59.943842+0000","last_fullsized":"2024-03-06T07:21:59.943842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943853+0000","last_change":"2024-03-06T07:21:58.949671+0000","last_active":"2024-03-06T07:21:59.943853+0000","last_peered":"2024-03-06T07:21:59.943853+0000","last_clean":"2024-03-06T07:21:59.943853+0000","last_became_active":"2024-03-06T07:21:58.949364+0000","last_became_peered":"2024-03-06T07:21:58.949364+0000","last_unstale":"2024-03-06T07:21:59.943853+0000","last_undegraded":"2024-03-06T07:21:59.943853+0000","last_fullsized":"2024-03-06T07:21:59.943853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:54.826119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004867+0000","last_change":"2024-03-06T07:21:57.960062+0000","last_active":"2024-03-06T07:22:01.004867+0000","last_peered":"2024-03-06T07:22:01.004867+0000","last_clean":"2024-03-06T07:22:01.004867+0000","last_became_active":"2024-03-06T07:21:57.959790+0000","last_became_peered":"2024-03-06T07:21:57.959790+0000","last_unstale":"2024-03-06T07:22:01.004867+0000","last_undegraded":"2024-03-06T07:22:01.004867+0000","last_fullsized":"2024-03-06T07:22:01.004867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:02.949716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549132+0000","last_change":"2024-03-06T07:21:58.958514+0000","last_active":"2024-03-06T07:22:00.549132+0000","last_peered":"2024-03-06T07:22:00.549132+0000","last_clean":"2024-03-06T07:22:00.549132+0000","last_became_active":"2024-03-06T07:21:58.958247+0000","last_became_peered":"2024-03-06T07:21:58.958247+0000","last_unstale":"2024-03-06T07:22:00.549132+0000","last_undegraded":"2024-03-06T07:22:00.549132+0000","last_fullsized":"2024-03-06T07:22:00.549132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944544+0000","last_change":"2024-03-06T07:21:57.949106+0000","last_active":"2024-03-06T07:21:59.944544+0000","last_peered":"2024-03-06T07:21:59.944544+0000","last_clean":"2024-03-06T07:21:59.944544+0000","last_became_active":"2024-03-06T07:21:57.948807+0000","last_became_peered":"2024-03-06T07:21:57.948807+0000","last_unstale":"2024-03-06T07:21:59.944544+0000","last_undegraded":"2024-03-06T07:21:59.944544+0000","last_fullsized":"2024-03-06T07:21:59.944544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004531+0000","last_change":"2024-03-06T07:21:57.962231+0000","last_active":"2024-03-06T07:22:01.004531+0000","last_peered":"2024-03-06T07:22:01.004531+0000","last_clean":"2024-03-06T07:22:01.004531+0000","last_became_active":"2024-03-06T07:21:57.961979+0000","last_became_peered":"2024-03-06T07:21:57.961979+0000","last_unstale":"2024-03-06T07:22:01.004531+0000","last_undegraded":"2024-03-06T07:22:01.004531+0000","last_fullsized":"2024-03-06T07:22:01.004531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004545+0000","last_change":"2024-03-06T07:21:58.962490+0000","last_active":"2024-03-06T07:22:01.004545+0000","last_peered":"2024-03-06T07:22:01.004545+0000","last_clean":"2024-03-06T07:22:01.004545+0000","last_became_active":"2024-03-06T07:21:58.960981+0000","last_became_peered":"2024-03-06T07:21:58.960981+0000","last_unstale":"2024-03-06T07:22:01.004545+0000","last_undegraded":"2024-03-06T07:22:01.004545+0000","last_fullsized":"2024-03-06T07:22:01.004545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004198+0000","last_change":"2024-03-06T07:21:57.961193+0000","last_active":"2024-03-06T07:22:01.004198+0000","last_peered":"2024-03-06T07:22:01.004198+0000","last_clean":"2024-03-06T07:22:01.004198+0000","last_became_active":"2024-03-06T07:21:57.960916+0000","last_became_peered":"2024-03-06T07:21:57.960916+0000","last_unstale":"2024-03-06T07:22:01.004198+0000","last_undegraded":"2024-03-06T07:22:01.004198+0000","last_fullsized":"2024-03-06T07:22:01.004198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942173+0000","last_change":"2024-03-06T07:21:58.951489+0000","last_active":"2024-03-06T07:21:59.942173+0000","last_peered":"2024-03-06T07:21:59.942173+0000","last_clean":"2024-03-06T07:21:59.942173+0000","last_became_active":"2024-03-06T07:21:58.951202+0000","last_became_peered":"2024-03-06T07:21:58.951202+0000","last_unstale":"2024-03-06T07:21:59.942173+0000","last_undegraded":"2024-03-06T07:21:59.942173+0000","last_fullsized":"2024-03-06T07:21:59.942173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944528+0000","last_change":"2024-03-06T07:21:57.959918+0000","last_active":"2024-03-06T07:21:59.944528+0000","last_peered":"2024-03-06T07:21:59.944528+0000","last_clean":"2024-03-06T07:21:59.944528+0000","last_became_active":"2024-03-06T07:21:57.959611+0000","last_became_peered":"2024-03-06T07:21:57.959611+0000","last_unstale":"2024-03-06T07:21:59.944528+0000","last_undegraded":"2024-03-06T07:21:59.944528+0000","last_fullsized":"2024-03-06T07:21:59.944528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004566+0000","last_change":"2024-03-06T07:21:58.970643+0000","last_active":"2024-03-06T07:22:01.004566+0000","last_peered":"2024-03-06T07:22:01.004566+0000","last_clean":"2024-03-06T07:22:01.004566+0000","last_became_active":"2024-03-06T07:21:58.970039+0000","last_became_peered":"2024-03-06T07:21:58.970039+0000","last_unstale":"2024-03-06T07:22:01.004566+0000","last_undegraded":"2024-03-06T07:22:01.004566+0000","last_fullsized":"2024-03-06T07:22:01.004566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004616+0000","last_change":"2024-03-06T07:21:57.962638+0000","last_active":"2024-03-06T07:22:01.004616+0000","last_peered":"2024-03-06T07:22:01.004616+0000","last_clean":"2024-03-06T07:22:01.004616+0000","last_became_active":"2024-03-06T07:21:57.962348+0000","last_became_peered":"2024-03-06T07:21:57.962348+0000","last_unstale":"2024-03-06T07:22:01.004616+0000","last_undegraded":"2024-03-06T07:22:01.004616+0000","last_fullsized":"2024-03-06T07:22:01.004616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549050+0000","last_change":"2024-03-06T07:21:58.962712+0000","last_active":"2024-03-06T07:22:00.549050+0000","last_peered":"2024-03-06T07:22:00.549050+0000","last_clean":"2024-03-06T07:22:00.549050+0000","last_became_active":"2024-03-06T07:21:58.962398+0000","last_became_peered":"2024-03-06T07:21:58.962398+0000","last_unstale":"2024-03-06T07:22:00.549050+0000","last_undegraded":"2024-03-06T07:22:00.549050+0000","last_fullsized":"2024-03-06T07:22:00.549050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942294+0000","last_change":"2024-03-06T07:21:57.958869+0000","last_active":"2024-03-06T07:21:59.942294+0000","last_peered":"2024-03-06T07:21:59.942294+0000","last_clean":"2024-03-06T07:21:59.942294+0000","last_became_active":"2024-03-06T07:21:57.958605+0000","last_became_peered":"2024-03-06T07:21:57.958605+0000","last_unstale":"2024-03-06T07:21:59.942294+0000","last_undegraded":"2024-03-06T07:21:59.942294+0000","last_fullsized":"2024-03-06T07:21:59.942294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944175+0000","last_change":"2024-03-06T07:21:58.960482+0000","last_active":"2024-03-06T07:21:59.944175+0000","last_peered":"2024-03-06T07:21:59.944175+0000","last_clean":"2024-03-06T07:21:59.944175+0000","last_became_active":"2024-03-06T07:21:58.960187+0000","last_became_peered":"2024-03-06T07:21:58.960187+0000","last_unstale":"2024-03-06T07:21:59.944175+0000","last_undegraded":"2024-03-06T07:21:59.944175+0000","last_fullsized":"2024-03-06T07:21:59.944175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:54.202847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944275+0000","last_change":"2024-03-06T07:21:57.963435+0000","last_active":"2024-03-06T07:21:59.944275+0000","last_peered":"2024-03-06T07:21:59.944275+0000","last_clean":"2024-03-06T07:21:59.944275+0000","last_became_active":"2024-03-06T07:21:57.963195+0000","last_became_peered":"2024-03-06T07:21:57.963195+0000","last_unstale":"2024-03-06T07:21:59.944275+0000","last_undegraded":"2024-03-06T07:21:59.944275+0000","last_fullsized":"2024-03-06T07:21:59.944275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005119+0000","last_change":"2024-03-06T07:21:58.961185+0000","last_active":"2024-03-06T07:22:01.005119+0000","last_peered":"2024-03-06T07:22:01.005119+0000","last_clean":"2024-03-06T07:22:01.005119+0000","last_became_active":"2024-03-06T07:21:58.960928+0000","last_became_peered":"2024-03-06T07:21:58.960928+0000","last_unstale":"2024-03-06T07:22:01.005119+0000","last_undegraded":"2024-03-06T07:22:01.005119+0000","last_fullsized":"2024-03-06T07:22:01.005119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:42.427321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549040+0000","last_change":"2024-03-06T07:21:57.950617+0000","last_active":"2024-03-06T07:22:00.549040+0000","last_peered":"2024-03-06T07:22:00.549040+0000","last_clean":"2024-03-06T07:22:00.549040+0000","last_became_active":"2024-03-06T07:21:57.950347+0000","last_became_peered":"2024-03-06T07:21:57.950347+0000","last_unstale":"2024-03-06T07:22:00.549040+0000","last_undegraded":"2024-03-06T07:22:00.549040+0000","last_fullsized":"2024-03-06T07:22:00.549040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:10.110560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548899+0000","last_change":"2024-03-06T07:21:58.957844+0000","last_active":"2024-03-06T07:22:00.548899+0000","last_peered":"2024-03-06T07:22:00.548899+0000","last_clean":"2024-03-06T07:22:00.548899+0000","last_became_active":"2024-03-06T07:21:58.957575+0000","last_became_peered":"2024-03-06T07:21:58.957575+0000","last_unstale":"2024-03-06T07:22:00.548899+0000","last_undegraded":"2024-03-06T07:22:00.548899+0000","last_fullsized":"2024-03-06T07:22:00.548899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:41.058405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004312+0000","last_change":"2024-03-06T07:21:57.964121+0000","last_active":"2024-03-06T07:22:01.004312+0000","last_peered":"2024-03-06T07:22:01.004312+0000","last_clean":"2024-03-06T07:22:01.004312+0000","last_became_active":"2024-03-06T07:21:57.963752+0000","last_became_peered":"2024-03-06T07:21:57.963752+0000","last_unstale":"2024-03-06T07:22:01.004312+0000","last_undegraded":"2024-03-06T07:22:01.004312+0000","last_fullsized":"2024-03-06T07:22:01.004312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005124+0000","last_change":"2024-03-06T07:21:58.962110+0000","last_active":"2024-03-06T07:22:01.005124+0000","last_peered":"2024-03-06T07:22:01.005124+0000","last_clean":"2024-03-06T07:22:01.005124+0000","last_became_active":"2024-03-06T07:21:58.961800+0000","last_became_peered":"2024-03-06T07:21:58.961800+0000","last_unstale":"2024-03-06T07:22:01.005124+0000","last_undegraded":"2024-03-06T07:22:01.005124+0000","last_fullsized":"2024-03-06T07:22:01.005124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:13.761240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944044+0000","last_change":"2024-03-06T07:21:57.949180+0000","last_active":"2024-03-06T07:21:59.944044+0000","last_peered":"2024-03-06T07:21:59.944044+0000","last_clean":"2024-03-06T07:21:59.944044+0000","last_became_active":"2024-03-06T07:21:57.948897+0000","last_became_peered":"2024-03-06T07:21:57.948897+0000","last_unstale":"2024-03-06T07:21:59.944044+0000","last_undegraded":"2024-03-06T07:21:59.944044+0000","last_fullsized":"2024-03-06T07:21:59.944044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944254+0000","last_change":"2024-03-06T07:21:58.957923+0000","last_active":"2024-03-06T07:21:59.944254+0000","last_peered":"2024-03-06T07:21:59.944254+0000","last_clean":"2024-03-06T07:21:59.944254+0000","last_became_active":"2024-03-06T07:21:58.957454+0000","last_became_peered":"2024-03-06T07:21:58.957454+0000","last_unstale":"2024-03-06T07:21:59.944254+0000","last_undegraded":"2024-03-06T07:21:59.944254+0000","last_fullsized":"2024-03-06T07:21:59.944254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983080+0000","last_change":"2024-03-06T07:21:57.956795+0000","last_active":"2024-03-06T07:21:59.983080+0000","last_peered":"2024-03-06T07:21:59.983080+0000","last_clean":"2024-03-06T07:21:59.983080+0000","last_became_active":"2024-03-06T07:21:57.956529+0000","last_became_peered":"2024-03-06T07:21:57.956529+0000","last_unstale":"2024-03-06T07:21:59.983080+0000","last_undegraded":"2024-03-06T07:21:59.983080+0000","last_fullsized":"2024-03-06T07:21:59.983080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:02.949716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004551+0000","last_change":"2024-03-06T07:21:58.961675+0000","last_active":"2024-03-06T07:22:01.004551+0000","last_peered":"2024-03-06T07:22:01.004551+0000","last_clean":"2024-03-06T07:22:01.004551+0000","last_became_active":"2024-03-06T07:21:58.961169+0000","last_became_peered":"2024-03-06T07:21:58.961169+0000","last_unstale":"2024-03-06T07:22:01.004551+0000","last_undegraded":"2024-03-06T07:22:01.004551+0000","last_fullsized":"2024-03-06T07:22:01.004551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943737+0000","last_change":"2024-03-06T07:21:57.952110+0000","last_active":"2024-03-06T07:21:59.943737+0000","last_peered":"2024-03-06T07:21:59.943737+0000","last_clean":"2024-03-06T07:21:59.943737+0000","last_became_active":"2024-03-06T07:21:57.951787+0000","last_became_peered":"2024-03-06T07:21:57.951787+0000","last_unstale":"2024-03-06T07:21:59.943737+0000","last_undegraded":"2024-03-06T07:21:59.943737+0000","last_fullsized":"2024-03-06T07:21:59.943737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:34.102031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004914+0000","last_change":"2024-03-06T07:21:58.967742+0000","last_active":"2024-03-06T07:22:01.004914+0000","last_peered":"2024-03-06T07:22:01.004914+0000","last_clean":"2024-03-06T07:22:01.004914+0000","last_became_active":"2024-03-06T07:21:58.967476+0000","last_became_peered":"2024-03-06T07:21:58.967476+0000","last_unstale":"2024-03-06T07:22:01.004914+0000","last_undegraded":"2024-03-06T07:22:01.004914+0000","last_fullsized":"2024-03-06T07:22:01.004914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983255+0000","last_change":"2024-03-06T07:21:57.962699+0000","last_active":"2024-03-06T07:21:59.983255+0000","last_peered":"2024-03-06T07:21:59.983255+0000","last_clean":"2024-03-06T07:21:59.983255+0000","last_became_active":"2024-03-06T07:21:57.962401+0000","last_became_peered":"2024-03-06T07:21:57.962401+0000","last_unstale":"2024-03-06T07:21:59.983255+0000","last_undegraded":"2024-03-06T07:21:59.983255+0000","last_fullsized":"2024-03-06T07:21:59.983255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005163+0000","last_change":"2024-03-06T07:21:58.964631+0000","last_active":"2024-03-06T07:22:01.005163+0000","last_peered":"2024-03-06T07:22:01.005163+0000","last_clean":"2024-03-06T07:22:01.005163+0000","last_became_active":"2024-03-06T07:21:58.964187+0000","last_became_peered":"2024-03-06T07:21:58.964187+0000","last_unstale":"2024-03-06T07:22:01.005163+0000","last_undegraded":"2024-03-06T07:22:01.005163+0000","last_fullsized":"2024-03-06T07:22:01.005163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:34.814920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942356+0000","last_change":"2024-03-06T07:21:57.951889+0000","last_active":"2024-03-06T07:21:59.942356+0000","last_peered":"2024-03-06T07:21:59.942356+0000","last_clean":"2024-03-06T07:21:59.942356+0000","last_became_active":"2024-03-06T07:21:57.951637+0000","last_became_peered":"2024-03-06T07:21:57.951637+0000","last_unstale":"2024-03-06T07:21:59.942356+0000","last_undegraded":"2024-03-06T07:21:59.942356+0000","last_fullsized":"2024-03-06T07:21:59.942356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004803+0000","last_change":"2024-03-06T07:21:58.962144+0000","last_active":"2024-03-06T07:22:01.004803+0000","last_peered":"2024-03-06T07:22:01.004803+0000","last_clean":"2024-03-06T07:22:01.004803+0000","last_became_active":"2024-03-06T07:21:58.960533+0000","last_became_peered":"2024-03-06T07:21:58.960533+0000","last_unstale":"2024-03-06T07:22:01.004803+0000","last_undegraded":"2024-03-06T07:22:01.004803+0000","last_fullsized":"2024-03-06T07:22:01.004803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:28.792591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983285+0000","last_change":"2024-03-06T07:21:57.958443+0000","last_active":"2024-03-06T07:21:59.983285+0000","last_peered":"2024-03-06T07:21:59.983285+0000","last_clean":"2024-03-06T07:21:59.983285+0000","last_became_active":"2024-03-06T07:21:57.958162+0000","last_became_peered":"2024-03-06T07:21:57.958162+0000","last_unstale":"2024-03-06T07:21:59.983285+0000","last_undegraded":"2024-03-06T07:21:59.983285+0000","last_fullsized":"2024-03-06T07:21:59.983285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005108+0000","last_change":"2024-03-06T07:21:58.968776+0000","last_active":"2024-03-06T07:22:01.005108+0000","last_peered":"2024-03-06T07:22:01.005108+0000","last_clean":"2024-03-06T07:22:01.005108+0000","last_became_active":"2024-03-06T07:21:58.968522+0000","last_became_peered":"2024-03-06T07:21:58.968522+0000","last_unstale":"2024-03-06T07:22:01.005108+0000","last_undegraded":"2024-03-06T07:22:01.005108+0000","last_fullsized":"2024-03-06T07:22:01.005108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:28.792591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004412+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:22:01.004412+0000","last_peered":"2024-03-06T07:22:01.004412+0000","last_clean":"2024-03-06T07:22:01.004412+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:22:01.004412+0000","last_undegraded":"2024-03-06T07:22:01.004412+0000","last_fullsized":"2024-03-06T07:22:01.004412+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004292+0000","last_change":"2024-03-06T07:21:57.958220+0000","last_active":"2024-03-06T07:22:01.004292+0000","last_peered":"2024-03-06T07:22:01.004292+0000","last_clean":"2024-03-06T07:22:01.004292+0000","last_became_active":"2024-03-06T07:21:57.957561+0000","last_became_peered":"2024-03-06T07:21:57.957561+0000","last_unstale":"2024-03-06T07:22:01.004292+0000","last_undegraded":"2024-03-06T07:22:01.004292+0000","last_fullsized":"2024-03-06T07:22:01.004292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004721+0000","last_change":"2024-03-06T07:21:58.968424+0000","last_active":"2024-03-06T07:22:01.004721+0000","last_peered":"2024-03-06T07:22:01.004721+0000","last_clean":"2024-03-06T07:22:01.004721+0000","last_became_active":"2024-03-06T07:21:58.968235+0000","last_became_peered":"2024-03-06T07:21:58.968235+0000","last_unstale":"2024-03-06T07:22:01.004721+0000","last_undegraded":"2024-03-06T07:22:01.004721+0000","last_fullsized":"2024-03-06T07:22:01.004721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004311+0000","last_change":"2024-03-06T07:21:57.962310+0000","last_active":"2024-03-06T07:22:01.004311+0000","last_peered":"2024-03-06T07:22:01.004311+0000","last_clean":"2024-03-06T07:22:01.004311+0000","last_became_active":"2024-03-06T07:21:57.962015+0000","last_became_peered":"2024-03-06T07:21:57.962015+0000","last_unstale":"2024-03-06T07:22:01.004311+0000","last_undegraded":"2024-03-06T07:22:01.004311+0000","last_fullsized":"2024-03-06T07:22:01.004311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:04.878864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548793+0000","last_change":"2024-03-06T07:21:58.949526+0000","last_active":"2024-03-06T07:22:00.548793+0000","last_peered":"2024-03-06T07:22:00.548793+0000","last_clean":"2024-03-06T07:22:00.548793+0000","last_became_active":"2024-03-06T07:21:58.949247+0000","last_became_peered":"2024-03-06T07:21:58.949247+0000","last_unstale":"2024-03-06T07:22:00.548793+0000","last_undegraded":"2024-03-06T07:22:00.548793+0000","last_fullsized":"2024-03-06T07:22:00.548793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:11.328500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983204+0000","last_change":"2024-03-06T07:21:57.963056+0000","last_active":"2024-03-06T07:21:59.983204+0000","last_peered":"2024-03-06T07:21:59.983204+0000","last_clean":"2024-03-06T07:21:59.983204+0000","last_became_active":"2024-03-06T07:21:57.962785+0000","last_became_peered":"2024-03-06T07:21:57.962785+0000","last_unstale":"2024-03-06T07:21:59.983204+0000","last_undegraded":"2024-03-06T07:21:59.983204+0000","last_fullsized":"2024-03-06T07:21:59.983204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942195+0000","last_change":"2024-03-06T07:21:58.948764+0000","last_active":"2024-03-06T07:21:59.942195+0000","last_peered":"2024-03-06T07:21:59.942195+0000","last_clean":"2024-03-06T07:21:59.942195+0000","last_became_active":"2024-03-06T07:21:58.948489+0000","last_became_peered":"2024-03-06T07:21:58.948489+0000","last_unstale":"2024-03-06T07:21:59.942195+0000","last_undegraded":"2024-03-06T07:21:59.942195+0000","last_fullsized":"2024-03-06T07:21:59.942195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004402+0000","last_change":"2024-03-06T07:21:57.965008+0000","last_active":"2024-03-06T07:22:01.004402+0000","last_peered":"2024-03-06T07:22:01.004402+0000","last_clean":"2024-03-06T07:22:01.004402+0000","last_became_active":"2024-03-06T07:21:57.964754+0000","last_became_peered":"2024-03-06T07:21:57.964754+0000","last_unstale":"2024-03-06T07:22:01.004402+0000","last_undegraded":"2024-03-06T07:22:01.004402+0000","last_fullsized":"2024-03-06T07:22:01.004402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944174+0000","last_change":"2024-03-06T07:21:58.958773+0000","last_active":"2024-03-06T07:21:59.944174+0000","last_peered":"2024-03-06T07:21:59.944174+0000","last_clean":"2024-03-06T07:21:59.944174+0000","last_became_active":"2024-03-06T07:21:58.958489+0000","last_became_peered":"2024-03-06T07:21:58.958489+0000","last_unstale":"2024-03-06T07:21:59.944174+0000","last_undegraded":"2024-03-06T07:21:59.944174+0000","last_fullsized":"2024-03-06T07:21:59.944174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:29.955363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942256+0000","last_change":"2024-03-06T07:21:57.952000+0000","last_active":"2024-03-06T07:21:59.942256+0000","last_peered":"2024-03-06T07:21:59.942256+0000","last_clean":"2024-03-06T07:21:59.942256+0000","last_became_active":"2024-03-06T07:21:57.951703+0000","last_became_peered":"2024-03-06T07:21:57.951703+0000","last_unstale":"2024-03-06T07:21:59.942256+0000","last_undegraded":"2024-03-06T07:21:59.942256+0000","last_fullsized":"2024-03-06T07:21:59.942256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004691+0000","last_change":"2024-03-06T07:21:58.967480+0000","last_active":"2024-03-06T07:22:01.004691+0000","last_peered":"2024-03-06T07:22:01.004691+0000","last_clean":"2024-03-06T07:22:01.004691+0000","last_became_active":"2024-03-06T07:21:58.967104+0000","last_became_peered":"2024-03-06T07:21:58.967104+0000","last_unstale":"2024-03-06T07:22:01.004691+0000","last_undegraded":"2024-03-06T07:22:01.004691+0000","last_fullsized":"2024-03-06T07:22:01.004691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:53.179429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005275+0000","last_change":"2024-03-06T07:21:57.960485+0000","last_active":"2024-03-06T07:22:01.005275+0000","last_peered":"2024-03-06T07:22:01.005275+0000","last_clean":"2024-03-06T07:22:01.005275+0000","last_became_active":"2024-03-06T07:21:57.960144+0000","last_became_peered":"2024-03-06T07:21:57.960144+0000","last_unstale":"2024-03-06T07:22:01.005275+0000","last_undegraded":"2024-03-06T07:22:01.005275+0000","last_fullsized":"2024-03-06T07:22:01.005275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:41.468075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942320+0000","last_change":"2024-03-06T07:21:58.947670+0000","last_active":"2024-03-06T07:21:59.942320+0000","last_peered":"2024-03-06T07:21:59.942320+0000","last_clean":"2024-03-06T07:21:59.942320+0000","last_became_active":"2024-03-06T07:21:58.947435+0000","last_became_peered":"2024-03-06T07:21:58.947435+0000","last_unstale":"2024-03-06T07:21:59.942320+0000","last_undegraded":"2024-03-06T07:21:59.942320+0000","last_fullsized":"2024-03-06T07:21:59.942320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942445+0000","last_change":"2024-03-06T07:21:57.951589+0000","last_active":"2024-03-06T07:21:59.942445+0000","last_peered":"2024-03-06T07:21:59.942445+0000","last_clean":"2024-03-06T07:21:59.942445+0000","last_became_active":"2024-03-06T07:21:57.951313+0000","last_became_peered":"2024-03-06T07:21:57.951313+0000","last_unstale":"2024-03-06T07:21:59.942445+0000","last_undegraded":"2024-03-06T07:21:59.942445+0000","last_fullsized":"2024-03-06T07:21:59.942445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:53.197786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005072+0000","last_change":"2024-03-06T07:21:58.966146+0000","last_active":"2024-03-06T07:22:01.005072+0000","last_peered":"2024-03-06T07:22:01.005072+0000","last_clean":"2024-03-06T07:22:01.005072+0000","last_became_active":"2024-03-06T07:21:58.965839+0000","last_became_peered":"2024-03-06T07:21:58.965839+0000","last_unstale":"2024-03-06T07:22:01.005072+0000","last_undegraded":"2024-03-06T07:22:01.005072+0000","last_fullsized":"2024-03-06T07:22:01.005072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944510+0000","last_change":"2024-03-06T07:21:57.956954+0000","last_active":"2024-03-06T07:21:59.944510+0000","last_peered":"2024-03-06T07:21:59.944510+0000","last_clean":"2024-03-06T07:21:59.944510+0000","last_became_active":"2024-03-06T07:21:57.956181+0000","last_became_peered":"2024-03-06T07:21:57.956181+0000","last_unstale":"2024-03-06T07:21:59.944510+0000","last_undegraded":"2024-03-06T07:21:59.944510+0000","last_fullsized":"2024-03-06T07:21:59.944510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549033+0000","last_change":"2024-03-06T07:21:58.957111+0000","last_active":"2024-03-06T07:22:00.549033+0000","last_peered":"2024-03-06T07:22:00.549033+0000","last_clean":"2024-03-06T07:22:00.549033+0000","last_became_active":"2024-03-06T07:21:58.956788+0000","last_became_peered":"2024-03-06T07:21:58.956788+0000","last_unstale":"2024-03-06T07:22:00.549033+0000","last_undegraded":"2024-03-06T07:22:00.549033+0000","last_fullsized":"2024-03-06T07:22:00.549033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004332+0000","last_change":"2024-03-06T07:21:57.960912+0000","last_active":"2024-03-06T07:22:01.004332+0000","last_peered":"2024-03-06T07:22:01.004332+0000","last_clean":"2024-03-06T07:22:01.004332+0000","last_became_active":"2024-03-06T07:21:57.960619+0000","last_became_peered":"2024-03-06T07:21:57.960619+0000","last_unstale":"2024-03-06T07:22:01.004332+0000","last_undegraded":"2024-03-06T07:22:01.004332+0000","last_fullsized":"2024-03-06T07:22:01.004332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:41.836911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943984+0000","last_change":"2024-03-06T07:21:58.949723+0000","last_active":"2024-03-06T07:21:59.943984+0000","last_peered":"2024-03-06T07:21:59.943984+0000","last_clean":"2024-03-06T07:21:59.943984+0000","last_became_active":"2024-03-06T07:21:58.949415+0000","last_became_peered":"2024-03-06T07:21:58.949415+0000","last_unstale":"2024-03-06T07:21:59.943984+0000","last_undegraded":"2024-03-06T07:21:59.943984+0000","last_fullsized":"2024-03-06T07:21:59.943984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:48.783928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942225+0000","last_change":"2024-03-06T07:21:57.957156+0000","last_active":"2024-03-06T07:21:59.942225+0000","last_peered":"2024-03-06T07:21:59.942225+0000","last_clean":"2024-03-06T07:21:59.942225+0000","last_became_active":"2024-03-06T07:21:57.956893+0000","last_became_peered":"2024-03-06T07:21:57.956893+0000","last_unstale":"2024-03-06T07:21:59.942225+0000","last_undegraded":"2024-03-06T07:21:59.942225+0000","last_fullsized":"2024-03-06T07:21:59.942225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004691+0000","last_change":"2024-03-06T07:21:58.953837+0000","last_active":"2024-03-06T07:22:01.004691+0000","last_peered":"2024-03-06T07:22:01.004691+0000","last_clean":"2024-03-06T07:22:01.004691+0000","last_became_active":"2024-03-06T07:21:58.953594+0000","last_became_peered":"2024-03-06T07:21:58.953594+0000","last_unstale":"2024-03-06T07:22:01.004691+0000","last_undegraded":"2024-03-06T07:22:01.004691+0000","last_fullsized":"2024-03-06T07:22:01.004691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942272+0000","last_change":"2024-03-06T07:21:57.962804+0000","last_active":"2024-03-06T07:21:59.942272+0000","last_peered":"2024-03-06T07:21:59.942272+0000","last_clean":"2024-03-06T07:21:59.942272+0000","last_became_active":"2024-03-06T07:21:57.962366+0000","last_became_peered":"2024-03-06T07:21:57.962366+0000","last_unstale":"2024-03-06T07:21:59.942272+0000","last_undegraded":"2024-03-06T07:21:59.942272+0000","last_fullsized":"2024-03-06T07:21:59.942272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004802+0000","last_change":"2024-03-06T07:21:58.957659+0000","last_active":"2024-03-06T07:22:01.004802+0000","last_peered":"2024-03-06T07:22:01.004802+0000","last_clean":"2024-03-06T07:22:01.004802+0000","last_became_active":"2024-03-06T07:21:58.957349+0000","last_became_peered":"2024-03-06T07:21:58.957349+0000","last_unstale":"2024-03-06T07:22:01.004802+0000","last_undegraded":"2024-03-06T07:22:01.004802+0000","last_fullsized":"2024-03-06T07:22:01.004802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005015+0000","last_change":"2024-03-06T07:21:57.957600+0000","last_active":"2024-03-06T07:22:01.005015+0000","last_peered":"2024-03-06T07:22:01.005015+0000","last_clean":"2024-03-06T07:22:01.005015+0000","last_became_active":"2024-03-06T07:21:57.957347+0000","last_became_peered":"2024-03-06T07:21:57.957347+0000","last_unstale":"2024-03-06T07:22:01.005015+0000","last_undegraded":"2024-03-06T07:22:01.005015+0000","last_fullsized":"2024-03-06T07:22:01.005015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944432+0000","last_change":"2024-03-06T07:21:58.950912+0000","last_active":"2024-03-06T07:21:59.944432+0000","last_peered":"2024-03-06T07:21:59.944432+0000","last_clean":"2024-03-06T07:21:59.944432+0000","last_became_active":"2024-03-06T07:21:58.950531+0000","last_became_peered":"2024-03-06T07:21:58.950531+0000","last_unstale":"2024-03-06T07:21:59.944432+0000","last_undegraded":"2024-03-06T07:21:59.944432+0000","last_fullsized":"2024-03-06T07:21:59.944432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004726+0000","last_change":"2024-03-06T07:21:58.968927+0000","last_active":"2024-03-06T07:22:01.004726+0000","last_peered":"2024-03-06T07:22:01.004726+0000","last_clean":"2024-03-06T07:22:01.004726+0000","last_became_active":"2024-03-06T07:21:58.967167+0000","last_became_peered":"2024-03-06T07:21:58.967167+0000","last_unstale":"2024-03-06T07:22:01.004726+0000","last_undegraded":"2024-03-06T07:22:01.004726+0000","last_fullsized":"2024-03-06T07:22:01.004726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:00.578026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004797+0000","last_change":"2024-03-06T07:21:57.962444+0000","last_active":"2024-03-06T07:22:01.004797+0000","last_peered":"2024-03-06T07:22:01.004797+0000","last_clean":"2024-03-06T07:22:01.004797+0000","last_became_active":"2024-03-06T07:21:57.962157+0000","last_became_peered":"2024-03-06T07:21:57.962157+0000","last_unstale":"2024-03-06T07:22:01.004797+0000","last_undegraded":"2024-03-06T07:22:01.004797+0000","last_fullsized":"2024-03-06T07:22:01.004797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004672+0000","last_change":"2024-03-06T07:21:58.965441+0000","last_active":"2024-03-06T07:22:01.004672+0000","last_peered":"2024-03-06T07:22:01.004672+0000","last_clean":"2024-03-06T07:22:01.004672+0000","last_became_active":"2024-03-06T07:21:58.964894+0000","last_became_peered":"2024-03-06T07:21:58.964894+0000","last_unstale":"2024-03-06T07:22:01.004672+0000","last_undegraded":"2024-03-06T07:22:01.004672+0000","last_fullsized":"2024-03-06T07:22:01.004672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:51.617926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005175+0000","last_change":"2024-03-06T07:21:57.958429+0000","last_active":"2024-03-06T07:22:01.005175+0000","last_peered":"2024-03-06T07:22:01.005175+0000","last_clean":"2024-03-06T07:22:01.005175+0000","last_became_active":"2024-03-06T07:21:57.958163+0000","last_became_peered":"2024-03-06T07:21:57.958163+0000","last_unstale":"2024-03-06T07:22:01.005175+0000","last_undegraded":"2024-03-06T07:22:01.005175+0000","last_fullsized":"2024-03-06T07:22:01.005175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:53.197786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548917+0000","last_change":"2024-03-06T07:21:58.962796+0000","last_active":"2024-03-06T07:22:00.548917+0000","last_peered":"2024-03-06T07:22:00.548917+0000","last_clean":"2024-03-06T07:22:00.548917+0000","last_became_active":"2024-03-06T07:21:58.962454+0000","last_became_peered":"2024-03-06T07:21:58.962454+0000","last_unstale":"2024-03-06T07:22:00.548917+0000","last_undegraded":"2024-03-06T07:22:00.548917+0000","last_fullsized":"2024-03-06T07:22:00.548917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004974+0000","last_change":"2024-03-06T07:21:57.962433+0000","last_active":"2024-03-06T07:22:01.004974+0000","last_peered":"2024-03-06T07:22:01.004974+0000","last_clean":"2024-03-06T07:22:01.004974+0000","last_became_active":"2024-03-06T07:21:57.962149+0000","last_became_peered":"2024-03-06T07:21:57.962149+0000","last_unstale":"2024-03-06T07:22:01.004974+0000","last_undegraded":"2024-03-06T07:22:01.004974+0000","last_fullsized":"2024-03-06T07:22:01.004974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004944+0000","last_change":"2024-03-06T07:21:58.963553+0000","last_active":"2024-03-06T07:22:01.004944+0000","last_peered":"2024-03-06T07:22:01.004944+0000","last_clean":"2024-03-06T07:22:01.004944+0000","last_became_active":"2024-03-06T07:21:58.963282+0000","last_became_peered":"2024-03-06T07:21:58.963282+0000","last_unstale":"2024-03-06T07:22:01.004944+0000","last_undegraded":"2024-03-06T07:22:01.004944+0000","last_fullsized":"2024-03-06T07:22:01.004944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:25.392050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549082+0000","last_change":"2024-03-06T07:21:57.956370+0000","last_active":"2024-03-06T07:22:00.549082+0000","last_peered":"2024-03-06T07:22:00.549082+0000","last_clean":"2024-03-06T07:22:00.549082+0000","last_became_active":"2024-03-06T07:21:57.955872+0000","last_became_peered":"2024-03-06T07:21:57.955872+0000","last_unstale":"2024-03-06T07:22:00.549082+0000","last_undegraded":"2024-03-06T07:22:00.549082+0000","last_fullsized":"2024-03-06T07:22:00.549082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:46.175278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:06.619 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:06.619 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:22:06.619 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:22:06.619 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:22:06.620 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.753+0000 7f300142a640 1 Processor -- start 2024-03-06T07:22:06.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.753+0000 7f300142a640 1 -- start start 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f300142a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2ffc166af0 0x7f2ffc168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f300142a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc15e380 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f300142a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc0cdc10 con 0x7f2ffc0cf850 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f300142a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc06ca90 con 0x7f2ffc166af0 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f2ffb7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2ffc166af0 0x7f2ffc168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f2ffaffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2ffc0d00c0 0x7f2ffc0d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35508/0 (socket says 172.21.15.77:35508) 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f2ffaffd640 1 -- 172.21.15.77:0/1504265115 learned_addr learned my addr 172.21.15.77:0/1504265115 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f2ffb7fe640 1 -- 172.21.15.77:0/1504265115 >> v1:172.21.15.158:6789/0 conn(0x7f2ffc0cf850 legacy=0x7f2ffc0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f2ffb7fe640 1 -- 172.21.15.77:0/1504265115 >> v1:172.21.15.77:6789/0 conn(0x7f2ffc0d00c0 legacy=0x7f2ffc0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f2ffb7fe640 1 -- 172.21.15.77:0/1504265115 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc06c160 con 0x7f2ffc166af0 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.754+0000 7f2ff9ffb640 1 -- 172.21.15.77:0/1504265115 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605983094 0 0) 0x7f2ffc15e380 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.755+0000 7f2ffb7fe640 1 --2- 172.21.15.77:0/1504265115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc166af0 0x7f2ffc168ee0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40097a0 tx=0x7f2ff402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a8dba7d338dfb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.755+0000 7f2ff9ffb640 1 -- 172.21.15.77:0/1504265115 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4034070 con 0x7f2ffc166af0 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.755+0000 7f2ff9ffb640 1 -- 172.21.15.77:0/1504265115 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff4030610 con 0x7f2ffc166af0 2024-03-06T07:22:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.755+0000 7f2ff9ffb640 1 -- 172.21.15.77:0/1504265115 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40309e0 con 0x7f2ffc166af0 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.756+0000 7f300142a640 1 -- 172.21.15.77:0/1504265115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc166af0 msgr2=0x7f2ffc168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.756+0000 7f300142a640 1 --2- 172.21.15.77:0/1504265115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc166af0 0x7f2ffc168ee0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40097a0 tx=0x7f2ff402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.756+0000 7f300142a640 1 -- 172.21.15.77:0/1504265115 shutdown_connections 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.756+0000 7f300142a640 1 --2- 172.21.15.77:0/1504265115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc166af0 0x7f2ffc168ee0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.756+0000 7f300142a640 1 -- 172.21.15.77:0/1504265115 >> 172.21.15.77:0/1504265115 conn(0x7f2ffc0b35a0 msgr2=0x7f2ffc0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.756+0000 7f300142a640 1 -- 172.21.15.77:0/1504265115 shutdown_connections 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.756+0000 7f300142a640 1 -- 172.21.15.77:0/1504265115 wait complete. 2024-03-06T07:22:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.757+0000 7f300142a640 1 Processor -- start 2024-03-06T07:22:06.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.757+0000 7f300142a640 1 -- start start 2024-03-06T07:22:06.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f300142a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ffc0cf850 0x7f2ffc157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f300142a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc0d00c0 0x7f2ffc0c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f300142a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ffc166af0 0x7f2ffc0b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f300142a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc06c720 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f300142a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc06df30 con 0x7f2ffc166af0 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f300142a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc0b1030 con 0x7f2ffc0cf850 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc0d00c0 0x7f2ffc0c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc0d00c0 0x7f2ffc0c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41342/0 (socket says 172.21.15.77:41342) 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 -- 172.21.15.77:0/3333418350 learned_addr learned my addr 172.21.15.77:0/3333418350 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffb7fe640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ffc166af0 0x7f2ffc0b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffaffd640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ffc0cf850 0x7f2ffc157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 -- 172.21.15.77:0/3333418350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ffc0cf850 msgr2=0x7f2ffc157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ffc0cf850 0x7f2ffc157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 -- 172.21.15.77:0/3333418350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ffc166af0 msgr2=0x7f2ffc0b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ffc166af0 0x7f2ffc0b6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 -- 172.21.15.77:0/3333418350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc0b5c40 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2ffa7fc640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc0d00c0 0x7f2ffc0c79e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000c430 tx=0x7f2ff000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2fe37fe640 1 -- 172.21.15.77:0/3333418350 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0017070 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2fe37fe640 1 -- 172.21.15.77:0/3333418350 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff00052d0 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc0b5f20 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f2fe37fe640 1 -- 172.21.15.77:0/3333418350 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0003830 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.758+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc0b64e0 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.759+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc04f330 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.760+0000 7f2fe37fe640 1 -- 172.21.15.77:0/3333418350 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ff001a070 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.760+0000 7f2fe37fe640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fe806b4d0 0x7f2fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.760+0000 7f2ffaffd640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fe806b4d0 0x7f2fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.760+0000 7f2fe37fe640 1 -- 172.21.15.77:0/3333418350 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ff00894d0 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.761+0000 7f2ffaffd640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fe806b4d0 0x7f2fe806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2fec001920 tx=0x7f2fec0018b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.765+0000 7f2fe37fe640 1 -- 172.21.15.77:0/3333418350 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ff0057d00 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.987+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2ffc0b0820 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.990+0000 7f2fe37fe640 1 -- 172.21.15.77:0/3333418350 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171781 (secure 0 0 0) 0x7f2ff0087e20 con 0x7f2ffc0d00c0 2024-03-06T07:22:06.993 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fe806b4d0 msgr2=0x7f2fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fe806b4d0 0x7f2fe806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2fec001920 tx=0x7f2fec0018b0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc0d00c0 msgr2=0x7f2ffc0c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc0d00c0 0x7f2ffc0c79e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000c430 tx=0x7f2ff000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 shutdown_connections 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fe806b4d0 0x7f2fe806d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ffc0d00c0 0x7f2ffc0c79e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ffc166af0 0x7f2ffc0b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 --2- 172.21.15.77:0/3333418350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ffc0cf850 0x7f2ffc157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 >> 172.21.15.77:0/3333418350 conn(0x7f2ffc0b35a0 msgr2=0x7f2ffc0cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 shutdown_connections 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:06.992+0000 7f300142a640 1 -- 172.21.15.77:0/3333418350 wait complete. 2024-03-06T07:22:07.005 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":885960424},{"type":"v1","addr":"172.21.15.77:6833","nonce":885960424}]},"active_addr":"172.21.15.77:6833/885960424","active_change":"2024-03-06T07:21:45.375801+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":1155613375}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":455964598}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2540264753}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2341658297}]}]} 2024-03-06T07:22:07.008 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:22:07.008 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:22:07.008 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:07.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.107+0000 7f6ff9c34640 1 Processor -- start 2024-03-06T07:22:07.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.107+0000 7f6ff9c34640 1 -- start start 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff9c34640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6ff406b220 0x7f6ff415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff9c34640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ff40ce920 0x7f6ff4071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff9c34640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff415ea30 con 0x7f6ff4071ec0 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff9c34640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff4161b70 con 0x7f6ff40ce920 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff9c34640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff415ed10 con 0x7f6ff406b220 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff2ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ff40ce920 0x7f6ff4071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6ff4071ec0 0x7f6ff4162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35516/0 (socket says 172.21.15.77:35516) 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff3fff640 1 -- 172.21.15.77:0/26497459 learned_addr learned my addr 172.21.15.77:0/26497459 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff37fe640 1 --2- 172.21.15.77:0/26497459 >> v2:172.21.15.158:3300/0 conn(0x7f6ff406b220 0x7f6ff415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff2ffd640 1 -- 172.21.15.77:0/26497459 >> v2:172.21.15.158:3300/0 conn(0x7f6ff406b220 msgr2=0x7f6ff415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff2ffd640 1 --2- 172.21.15.77:0/26497459 >> v2:172.21.15.158:3300/0 conn(0x7f6ff406b220 0x7f6ff415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff2ffd640 1 -- 172.21.15.77:0/26497459 >> v1:172.21.15.77:6789/0 conn(0x7f6ff4071ec0 legacy=0x7f6ff4162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff2ffd640 1 -- 172.21.15.77:0/26497459 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff415eff0 con 0x7f6ff40ce920 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.108+0000 7f6ff27fc640 1 -- 172.21.15.77:0/26497459 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565678925 0 0) 0x7f6ff415ea30 con 0x7f6ff4071ec0 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff2ffd640 1 --2- 172.21.15.77:0/26497459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff40ce920 0x7f6ff4071980 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40d0980 tx=0x7f6fe402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1fe86be128cfab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff27fc640 1 -- 172.21.15.77:0/26497459 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4035020 con 0x7f6ff40ce920 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff27fc640 1 -- 172.21.15.77:0/26497459 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4031830 con 0x7f6ff40ce920 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff27fc640 1 -- 172.21.15.77:0/26497459 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4031c00 con 0x7f6ff40ce920 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff9c34640 1 -- 172.21.15.77:0/26497459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff40ce920 msgr2=0x7f6ff4071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/26497459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff40ce920 0x7f6ff4071980 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40d0980 tx=0x7f6fe402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff9c34640 1 -- 172.21.15.77:0/26497459 shutdown_connections 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/26497459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff40ce920 0x7f6ff4071980 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/26497459 >> v2:172.21.15.158:3300/0 conn(0x7f6ff406b220 0x7f6ff415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff9c34640 1 -- 172.21.15.77:0/26497459 >> 172.21.15.77:0/26497459 conn(0x7f6ff406c0c0 msgr2=0x7f6ff406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.109+0000 7f6ff9c34640 1 -- 172.21.15.77:0/26497459 shutdown_connections 2024-03-06T07:22:07.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.110+0000 7f6ff9c34640 1 -- 172.21.15.77:0/26497459 wait complete. 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.111+0000 7f6ff9c34640 1 Processor -- start 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.111+0000 7f6ff9c34640 1 -- start start 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.111+0000 7f6ff9c34640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406b220 0x7f6ff41f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.111+0000 7f6ff9c34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff4071ec0 0x7f6ff41f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.111+0000 7f6ff9c34640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ff40ce920 0x7f6ff40ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.111+0000 7f6ff9c34640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40ce380 con 0x7f6ff4071ec0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.111+0000 7f6ff9c34640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40ce200 con 0x7f6ff406b220 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.111+0000 7f6ff9c34640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff4162710 con 0x7f6ff40ce920 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff4071ec0 0x7f6ff41f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff4071ec0 0x7f6ff41f4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41368/0 (socket says 172.21.15.77:41368) 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 -- 172.21.15.77:0/1050101171 learned_addr learned my addr 172.21.15.77:0/1050101171 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff37fe640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406b220 0x7f6ff41f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff3fff640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ff40ce920 0x7f6ff40ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 -- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ff40ce920 msgr2=0x7f6ff40ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ff40ce920 0x7f6ff40ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 -- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406b220 msgr2=0x7f6ff41f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406b220 0x7f6ff41f6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 -- 172.21.15.77:0/1050101171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40aeb70 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff3fff640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ff40ce920 0x7f6ff40ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff2ffd640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff4071ec0 0x7f6ff41f4110 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4034040 tx=0x7f6fe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6fdbfff640 1 -- 172.21.15.77:0/1050101171 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4035040 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6fdbfff640 1 -- 172.21.15.77:0/1050101171 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4004230 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff40aee50 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6fdbfff640 1 -- 172.21.15.77:0/1050101171 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe403a640 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.112+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff40af410 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.113+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff404f330 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.114+0000 7f6fdbfff640 1 -- 172.21.15.77:0/1050101171 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fe4038020 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.114+0000 7f6fdbfff640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.114+0000 7f6fdbfff640 1 -- 172.21.15.77:0/1050101171 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fe40abe50 con 0x7f6ff4071ec0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.114+0000 7f6ff37fe640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.114+0000 7f6ff37fe640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe006b4d0 0x7f6fe006d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ff406f020 tx=0x7f6fe8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.118+0000 7f6fdbfff640 1 -- 172.21.15.77:0/1050101171 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fe407acd0 con 0x7f6ff4071ec0 2024-03-06T07:22:07.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.290+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ff40cdf00 con 0x7f6ff4071ec0 2024-03-06T07:22:07.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.291+0000 7f6fdbfff640 1 -- 172.21.15.77:0/1050101171 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f6fe407a5d0 con 0x7f6ff4071ec0 2024-03-06T07:22:07.293 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:07.293 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe006b4d0 msgr2=0x7f6fe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe006b4d0 0x7f6fe006d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ff406f020 tx=0x7f6fe8009190 comp rx=0 tx=0).stop 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff4071ec0 msgr2=0x7f6ff41f4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff4071ec0 0x7f6ff41f4110 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4034040 tx=0x7f6fe40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 shutdown_connections 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ff4071ec0 0x7f6ff41f4110 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ff40ce920 0x7f6ff40ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 --2- 172.21.15.77:0/1050101171 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ff406b220 0x7f6ff41f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 >> 172.21.15.77:0/1050101171 conn(0x7f6ff406c0c0 msgr2=0x7f6ff4162000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 shutdown_connections 2024-03-06T07:22:07.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.293+0000 7f6ff9c34640 1 -- 172.21.15.77:0/1050101171 wait complete. 2024-03-06T07:22:07.306 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:22:07.307 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:07.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.405+0000 7fd3e0039640 1 Processor -- start 2024-03-06T07:22:07.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.405+0000 7fd3e0039640 1 -- start start 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3e0039640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3d815fa10 0x7fd3d80ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3e0039640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80cebe0 con 0x7fd3d815bdd0 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3e0039640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80ceec0 con 0x7fd3d80d16e0 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3e0039640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d80cf1a0 con 0x7fd3d815fa10 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dddd5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd3d815bdd0 0x7fd3d80c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35532/0 (socket says 172.21.15.77:35532) 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dddd5640 1 -- 172.21.15.77:0/3423593674 learned_addr learned my addr 172.21.15.77:0/3423593674 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dd5d4640 1 --2- 172.21.15.77:0/3423593674 >> v2:172.21.15.77:3300/0 conn(0x7fd3d815fa10 0x7fd3d80ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dcdd3640 1 -- 172.21.15.77:0/3423593674 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380554432 0 0) 0x7fd3d80cebe0 con 0x7fd3d815bdd0 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dcdd3640 1 -- 172.21.15.77:0/3423593674 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c4003520 con 0x7fd3d815bdd0 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dcdd3640 1 -- 172.21.15.77:0/3423593674 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234440364 0 0) 0x7fd3d80ceec0 con 0x7fd3d80d16e0 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dcdd3640 1 -- 172.21.15.77:0/3423593674 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d80cebe0 con 0x7fd3d80d16e0 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dd5d4640 1 -- 172.21.15.77:0/3423593674 >> v1:172.21.15.158:6789/0 conn(0x7fd3d80d16e0 legacy=0x7fd3d80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dd5d4640 1 -- 172.21.15.77:0/3423593674 >> v1:172.21.15.77:6789/0 conn(0x7fd3d815bdd0 legacy=0x7fd3d80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.406+0000 7fd3dd5d4640 1 -- 172.21.15.77:0/3423593674 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d80cc8e0 con 0x7fd3d815fa10 2024-03-06T07:22:07.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.407+0000 7fd3dd5d4640 1 --2- 172.21.15.77:0/3423593674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815fa10 0x7fd3d80ce270 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80097a0 tx=0x7fd3c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c341755ef01368 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.407+0000 7fd3dcdd3640 1 -- 172.21.15.77:0/3423593674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8035020 con 0x7fd3d815fa10 2024-03-06T07:22:07.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.407+0000 7fd3dcdd3640 1 -- 172.21.15.77:0/3423593674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c8031830 con 0x7fd3d815fa10 2024-03-06T07:22:07.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.407+0000 7fd3dcdd3640 1 -- 172.21.15.77:0/3423593674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8031c00 con 0x7fd3d815fa10 2024-03-06T07:22:07.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.408+0000 7fd3e0039640 1 -- 172.21.15.77:0/3423593674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815fa10 msgr2=0x7fd3d80ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.408+0000 7fd3e0039640 1 --2- 172.21.15.77:0/3423593674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815fa10 0x7fd3d80ce270 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80097a0 tx=0x7fd3c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.408+0000 7fd3e0039640 1 -- 172.21.15.77:0/3423593674 shutdown_connections 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.408+0000 7fd3e0039640 1 --2- 172.21.15.77:0/3423593674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815fa10 0x7fd3d80ce270 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.408+0000 7fd3e0039640 1 -- 172.21.15.77:0/3423593674 >> 172.21.15.77:0/3423593674 conn(0x7fd3d809a9f0 msgr2=0x7fd3d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.408+0000 7fd3e0039640 1 -- 172.21.15.77:0/3423593674 shutdown_connections 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.408+0000 7fd3e0039640 1 -- 172.21.15.77:0/3423593674 wait complete. 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.409+0000 7fd3e0039640 1 Processor -- start 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.409+0000 7fd3e0039640 1 -- start start 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3e0039640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d16e0 0x7fd3d81573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3e0039640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815bdd0 0x7fd3d806f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3e0039640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d815fa10 0x7fd3d806fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3e0039640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80ce7b0 con 0x7fd3d815bdd0 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3e0039640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d815c1b0 con 0x7fd3d815fa10 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3e0039640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80ce930 con 0x7fd3d80d16e0 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815bdd0 0x7fd3d806f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815bdd0 0x7fd3d806f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41380/0 (socket says 172.21.15.77:41380) 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 -- 172.21.15.77:0/2234206955 learned_addr learned my addr 172.21.15.77:0/2234206955 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3de5d6640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d815fa10 0x7fd3d806fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dddd5640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d16e0 0x7fd3d81573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 -- 172.21.15.77:0/2234206955 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d16e0 msgr2=0x7fd3d81573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d16e0 0x7fd3d81573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 -- 172.21.15.77:0/2234206955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d815fa10 msgr2=0x7fd3d806fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d815fa10 0x7fd3d806fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 -- 172.21.15.77:0/2234206955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8157cc0 con 0x7fd3d815bdd0 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3dd5d4640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815bdd0 0x7fd3d806f990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd3c802bee0 tx=0x7fd3c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/2234206955 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8035040 con 0x7fd3d815bdd0 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/2234206955 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c803b920 con 0x7fd3d815bdd0 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d80b52c0 con 0x7fd3d815bdd0 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/2234206955 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c803bd10 con 0x7fd3d815bdd0 2024-03-06T07:22:07.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.410+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d8158240 con 0x7fd3d815bdd0 2024-03-06T07:22:07.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.411+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d804f330 con 0x7fd3d815bdd0 2024-03-06T07:22:07.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.412+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/2234206955 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3c8038020 con 0x7fd3d815bdd0 2024-03-06T07:22:07.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.412+0000 7fd3ce7fc640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.412+0000 7fd3dddd5640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.412+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/2234206955 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3c80ac160 con 0x7fd3d815bdd0 2024-03-06T07:22:07.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.413+0000 7fd3dddd5640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4000f60 tx=0x7fd3d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.417+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/2234206955 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3c807aa20 con 0x7fd3d815bdd0 2024-03-06T07:22:07.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.591+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3d80b7a40 con 0x7fd3d815bdd0 2024-03-06T07:22:07.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.592+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/2234206955 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fd3c807eac0 con 0x7fd3d815bdd0 2024-03-06T07:22:07.594 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:07.594 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:07.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 msgr2=0x7fd3bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4000f60 tx=0x7fd3d4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815bdd0 msgr2=0x7fd3d806f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815bdd0 0x7fd3d806f990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd3c802bee0 tx=0x7fd3c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 shutdown_connections 2024-03-06T07:22:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815bdd0 0x7fd3d806f990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d815fa10 0x7fd3d806fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 --2- 172.21.15.77:0/2234206955 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d16e0 0x7fd3d81573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 >> 172.21.15.77:0/2234206955 conn(0x7fd3d809a9f0 msgr2=0x7fd3d80d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 shutdown_connections 2024-03-06T07:22:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.594+0000 7fd3e0039640 1 -- 172.21.15.77:0/2234206955 wait complete. 2024-03-06T07:22:07.607 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:22:07.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:22:07.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:22:07.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:22:07.609 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:22:07.609 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:22:07.609 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:22:07.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:22:07.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.777+0000 7f0fc2715640 1 Processor -- start 2024-03-06T07:22:07.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.777+0000 7f0fc2715640 1 -- start start 2024-03-06T07:22:07.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.777+0000 7fd6c3fee640 1 Processor -- start 2024-03-06T07:22:07.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.777+0000 7fd6c3fee640 1 -- start start 2024-03-06T07:22:07.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f700a195640 1 Processor -- start 2024-03-06T07:22:07.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f700a195640 1 -- start start 2024-03-06T07:22:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f8d690ed640 1 Processor -- start 2024-03-06T07:22:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f8d690ed640 1 -- start start 2024-03-06T07:22:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f7e6c5c9640 1 Processor -- start 2024-03-06T07:22:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f0fc2715640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0fb40df420 0x7f0fb404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f0fc2715640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0fb400aaa0 0x7f0fb404ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f0fc2715640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb4038770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f0fc2715640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb40068a0 con 0x7f0fb400aaa0 2024-03-06T07:22:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f0fc2715640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb4006af0 con 0x7f0fb40df420 2024-03-06T07:22:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f0fc2715640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4006dd0 con 0x7f0fb40382c0 2024-03-06T07:22:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f7e6c5c9640 1 -- start start 2024-03-06T07:22:07.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f5994088640 1 Processor -- start 2024-03-06T07:22:07.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.778+0000 7f5994088640 1 -- start start 2024-03-06T07:22:07.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c3fee640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc14bb10 0x7fd6bc165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c3fee640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6bc0ce8d0 0x7fd6bc0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c3fee640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc0cc020 con 0x7fd6bc06afb0 2024-03-06T07:22:07.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c3fee640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6bc06e3c0 con 0x7fd6bc0ce8d0 2024-03-06T07:22:07.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c3fee640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1666d0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1f14640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb4038770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1f14640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb4038770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35732/0 (socket says 172.21.15.77:35732) 2024-03-06T07:22:07.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1f14640 1 -- 172.21.15.77:0/1167800770 learned_addr learned my addr 172.21.15.77:0/1167800770 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1713640 1 --2- 172.21.15.77:0/1167800770 >> v2:172.21.15.158:3300/0 conn(0x7f0fb40df420 0x7f0fb404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c1d8a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc14bb10 0x7fd6bc165fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd0501f7640 1 Processor -- start 2024-03-06T07:22:07.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c1589640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6bc0ce8d0 0x7fd6bc0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f5994088640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f598c06de20 0x7f598c06fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1f14640 1 -- 172.21.15.77:0/1167800770 >> v2:172.21.15.158:3300/0 conn(0x7f0fb40df420 msgr2=0x7f0fb404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1f14640 1 --2- 172.21.15.77:0/1167800770 >> v2:172.21.15.158:3300/0 conn(0x7f0fb40df420 0x7f0fb404fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1f14640 1 -- 172.21.15.77:0/1167800770 >> v2:172.21.15.77:3300/0 conn(0x7f0fb400aaa0 msgr2=0x7f0fb404ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c1d8a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc14bb10 0x7fd6bc165fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35746/0 (socket says 172.21.15.77:35746) 2024-03-06T07:22:07.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd6c1d8a640 1 -- 172.21.15.77:0/946061240 learned_addr learned my addr 172.21.15.77:0/946061240 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1f14640 1 --2- 172.21.15.77:0/1167800770 >> v2:172.21.15.77:3300/0 conn(0x7f0fb400aaa0 0x7f0fb404ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f0fc1f14640 1 -- 172.21.15.77:0/1167800770 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb40070b0 con 0x7f0fb40382c0 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f700a195640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7004070740 0x7f700415a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f700a195640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700415ab90 con 0x7f7004154060 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f700a195640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700415ae70 con 0x7f700406e830 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f700a195640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70040b3c00 con 0x7f7004070740 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7008992640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7004070740 0x7f700415a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7008992640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7004070740 0x7f700415a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35754/0 (socket says 172.21.15.77:35754) 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7008992640 1 -- 172.21.15.77:0/27047575 learned_addr learned my addr 172.21.15.77:0/27047575 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f8d690ed640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8d6415e000 0x7f8d640d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f8d690ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640d4810 0x7f8d640d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f8d690ed640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640d25e0 con 0x7f8d640c80e0 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f8d690ed640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d640c7c50 con 0x7f8d6415e000 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7f8d690ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640d2880 con 0x7f8d640d4810 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f8d62d9d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8d6415e000 0x7f8d640d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f8d6359e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8d640c80e0 0x7f8d640c8570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35594/0 (socket says 172.21.15.77:35594) 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f8d6359e640 1 -- 172.21.15.77:0/2076203867 learned_addr learned my addr 172.21.15.77:0/2076203867 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6c5c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7e640b48d0 0x7f7e64070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6c5c9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7e6406c8b0 0x7f7e6406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6c5c9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e64166a50 con 0x7f7e64165bf0 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6c5c9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e64072d20 con 0x7f7e640b48d0 2024-03-06T07:22:07.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6c5c9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e640cf780 con 0x7f7e6406c8b0 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6a365640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7e64165bf0 0x7f7e64165fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35580/0 (socket says 172.21.15.77:35580) 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6a365640 1 -- 172.21.15.77:0/3697094355 learned_addr learned my addr 172.21.15.77:0/3697094355 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6ab66640 1 --2- 172.21.15.77:0/3697094355 >> v2:172.21.15.158:3300/0 conn(0x7f7e6406c8b0 0x7f7e6406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7fd6c1d8a640 1 -- 172.21.15.77:0/946061240 >> v2:172.21.15.158:3300/0 conn(0x7fd6bc0ce8d0 msgr2=0x7fd6bc0cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7fd6c1d8a640 1 --2- 172.21.15.77:0/946061240 >> v2:172.21.15.158:3300/0 conn(0x7fd6bc0ce8d0 0x7fd6bc0cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7fd6c1d8a640 1 -- 172.21.15.77:0/946061240 >> v1:172.21.15.77:6789/0 conn(0x7fd6bc06afb0 legacy=0x7fd6bc071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7fd6c1d8a640 1 -- 172.21.15.77:0/946061240 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc1669b0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f5994088640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c162e80 con 0x7f598c1627a0 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f5994088640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c163160 con 0x7f598c163f90 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f5994088640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c163720 con 0x7f598c06de20 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f5992625640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f598c163f90 0x7f598c164440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58800/0 (socket says 172.21.15.77:58800) 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f5992625640 1 -- 172.21.15.77:0/290988602 learned_addr learned my addr 172.21.15.77:0/290988602 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f0fc1f14640 1 --2- 172.21.15.77:0/1167800770 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb4038770 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc07a850 tx=0x7f0fbc06a7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f5028da7214f69d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7008992640 1 -- 172.21.15.77:0/27047575 >> v1:172.21.15.158:6789/0 conn(0x7f700406e830 legacy=0x7f700406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7008992640 1 -- 172.21.15.77:0/27047575 >> v1:172.21.15.77:6789/0 conn(0x7f7004154060 legacy=0x7f70040ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7008992640 1 -- 172.21.15.77:0/27047575 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700415b150 con 0x7f7004070740 2024-03-06T07:22:07.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f5991e24640 1 --2- 172.21.15.77:0/290988602 >> v2:172.21.15.77:3300/0 conn(0x7f598c06de20 0x7f598c06fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6ab66640 1 -- 172.21.15.77:0/3697094355 >> v2:172.21.15.77:3300/0 conn(0x7f7e640b48d0 msgr2=0x7f7e64070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6ab66640 1 --2- 172.21.15.77:0/3697094355 >> v2:172.21.15.77:3300/0 conn(0x7f7e640b48d0 0x7f7e64070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f7e6ab66640 1 -- 172.21.15.77:0/3697094355 >> v1:172.21.15.77:6789/0 conn(0x7f7e64165bf0 legacy=0x7f7e64165fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f8d6259c640 1 --2- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640d4810 0x7f8d640d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f8d62d9d640 1 -- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640d4810 msgr2=0x7f8d640d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f8d62d9d640 1 --2- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640d4810 0x7f8d640d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f8d61d9b640 1 -- 172.21.15.77:0/2076203867 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899981967 0 0) 0x7f8d640d25e0 con 0x7f8d640c80e0 2024-03-06T07:22:07.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.780+0000 7f0fb3fff640 1 -- 172.21.15.77:0/1167800770 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0b8070 con 0x7f0fb40382c0 2024-03-06T07:22:07.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f0fb3fff640 1 -- 172.21.15.77:0/1167800770 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc0abc00 con 0x7f0fb40382c0 2024-03-06T07:22:07.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f8d62d9d640 1 -- 172.21.15.77:0/2076203867 >> v1:172.21.15.77:6789/0 conn(0x7f8d640c80e0 legacy=0x7f8d640c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f8d62d9d640 1 -- 172.21.15.77:0/2076203867 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640d2b40 con 0x7f8d6415e000 2024-03-06T07:22:07.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f8d6259c640 1 --2- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640d4810 0x7f8d640d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:07.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f7e69363640 1 -- 172.21.15.77:0/3697094355 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535997561 0 0) 0x7f7e60001110 con 0x7f7e64165bf0 2024-03-06T07:22:07.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f0fb3fff640 1 -- 172.21.15.77:0/1167800770 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0b3640 con 0x7f0fb40382c0 2024-03-06T07:22:07.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f4d473d7640 1 Processor -- start 2024-03-06T07:22:07.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f7e6ab66640 1 -- 172.21.15.77:0/3697094355 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64072760 con 0x7f7e6406c8b0 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f5991e24640 1 -- 172.21.15.77:0/290988602 >> v1:172.21.15.158:6789/0 conn(0x7f598c163f90 legacy=0x7f598c164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f5991e24640 1 -- 172.21.15.77:0/290988602 >> v1:172.21.15.77:6789/0 conn(0x7f598c1627a0 legacy=0x7f598c06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f5991e24640 1 -- 172.21.15.77:0/290988602 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c163440 con 0x7f598c06de20 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.779+0000 7fd0501f7640 1 -- start start 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd0501f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd04815e000 0x7fd0480d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd0501f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0481622e0 0x7fd0480d45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd0501f7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0480ceb70 con 0x7fd0480cf930 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd0501f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0480d4af0 con 0x7fd04815e000 2024-03-06T07:22:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd0501f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0480cee50 con 0x7fd0481622e0 2024-03-06T07:22:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f7008992640 1 --2- 172.21.15.77:0/27047575 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7004070740 0x7f700415a280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7000013180 tx=0x7f7000034230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00ebdfb8f7df2b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd6c1d8a640 1 --2- 172.21.15.77:0/946061240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc14bb10 0x7fd6bc165fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009e50 tx=0x7fd6b402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f73f5af6e404255 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd04d792640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd0480cf930 0x7fd048161bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58812/0 (socket says 172.21.15.77:58812) 2024-03-06T07:22:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd04d792640 1 -- 172.21.15.77:0/3104187255 learned_addr learned my addr 172.21.15.77:0/3104187255 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:07.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f6ffbfff640 1 -- 172.21.15.77:0/27047575 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000110d0 con 0x7f7004070740 2024-03-06T07:22:07.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f6ffbfff640 1 -- 172.21.15.77:0/27047575 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000011270 con 0x7f7004070740 2024-03-06T07:22:07.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f4d473d7640 1 -- start start 2024-03-06T07:22:07.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7fd6c0d88640 1 -- 172.21.15.77:0/946061240 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b402c9e0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.781+0000 7f6ffbfff640 1 -- 172.21.15.77:0/27047575 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000011620 con 0x7f7004070740 2024-03-06T07:22:07.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7fd6c0d88640 1 -- 172.21.15.77:0/946061240 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b402cb80 con 0x7fd6bc14bb10 2024-03-06T07:22:07.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 -- 172.21.15.77:0/1167800770 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 msgr2=0x7f0fb4038770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 --2- 172.21.15.77:0/1167800770 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb4038770 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc07a850 tx=0x7f0fbc06a7b0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 -- 172.21.15.77:0/1167800770 shutdown_connections 2024-03-06T07:22:07.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 --2- 172.21.15.77:0/1167800770 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb4038770 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 --2- 172.21.15.77:0/1167800770 >> v2:172.21.15.77:3300/0 conn(0x7f0fb400aaa0 0x7f0fb404ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 --2- 172.21.15.77:0/1167800770 >> v2:172.21.15.158:3300/0 conn(0x7f0fb40df420 0x7f0fb404fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 -- 172.21.15.77:0/1167800770 >> 172.21.15.77:0/1167800770 conn(0x7f0fb4009fb0 msgr2=0x7f0fb400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 -- 172.21.15.77:0/1167800770 shutdown_connections 2024-03-06T07:22:07.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f0fc2715640 1 -- 172.21.15.77:0/1167800770 wait complete. 2024-03-06T07:22:07.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7fd04e794640 1 --2- 172.21.15.77:0/3104187255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0481622e0 0x7fd0480d45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7fd6c0d88640 1 -- 172.21.15.77:0/946061240 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40343f0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f8d62d9d640 1 --2- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d6415e000 0x7f8d640d0680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8d580097a0 tx=0x7f8d5802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3fb4bcb7e6fc0b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f5991e24640 1 --2- 172.21.15.77:0/290988602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c06de20 0x7f598c06fe70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f597c0097a0 tx=0x7f597c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c526ed5de8df7c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7fd04cf91640 1 -- 172.21.15.77:0/3104187255 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164959625 0 0) 0x7fd0480ceb70 con 0x7fd0480cf930 2024-03-06T07:22:07.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7fd04cf91640 1 -- 172.21.15.77:0/3104187255 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd034003520 con 0x7fd0480cf930 2024-03-06T07:22:07.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7fd04e794640 1 -- 172.21.15.77:0/3104187255 >> v2:172.21.15.77:3300/0 conn(0x7fd04815e000 msgr2=0x7fd0480d1bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:07.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7fd04e794640 1 --2- 172.21.15.77:0/3104187255 >> v2:172.21.15.77:3300/0 conn(0x7fd04815e000 0x7fd0480d1bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f5990e22640 1 -- 172.21.15.77:0/290988602 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c035020 con 0x7f598c06de20 2024-03-06T07:22:07.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f8d61d9b640 1 -- 172.21.15.77:0/2076203867 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58034070 con 0x7f8d6415e000 2024-03-06T07:22:07.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f700a195640 1 -- 172.21.15.77:0/27047575 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7004070740 msgr2=0x7f700415a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f8d61d9b640 1 -- 172.21.15.77:0/2076203867 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d58030610 con 0x7f8d6415e000 2024-03-06T07:22:07.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f700a195640 1 --2- 172.21.15.77:0/27047575 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7004070740 0x7f700415a280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7000013180 tx=0x7f7000034230 comp rx=0 tx=0).stop 2024-03-06T07:22:07.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd04cf91640 1 -- 172.21.15.77:0/3104187255 <== mon.1 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2456483416 0 0) 0x7fd034003520 con 0x7fd0480cf930 2024-03-06T07:22:07.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f7e6ab66640 1 --2- 172.21.15.77:0/3697094355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e6406c8b0 0x7f7e6406ccc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e6406ea80 tx=0x7f7e5c02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=115ef7789e1685cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f7e69363640 1 -- 172.21.15.77:0/3697094355 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c039070 con 0x7f7e6406c8b0 2024-03-06T07:22:07.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f7e69363640 1 -- 172.21.15.77:0/3697094355 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c02cbf0 con 0x7f7e6406c8b0 2024-03-06T07:22:07.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd6c3fee640 1 -- 172.21.15.77:0/946061240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc14bb10 msgr2=0x7fd6bc165fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd6c3fee640 1 --2- 172.21.15.77:0/946061240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc14bb10 0x7fd6bc165fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009e50 tx=0x7fd6b402be40 comp rx=0 tx=0).stop 2024-03-06T07:22:07.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f4d473d7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d38006630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f4d473d7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d38050150 con 0x7f4d380df770 2024-03-06T07:22:07.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f4d473d7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d38006c30 con 0x7f4d38055d10 2024-03-06T07:22:07.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f4d473d7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d38006f10 con 0x7f4d38006220 2024-03-06T07:22:07.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f8d61d9b640 1 -- 172.21.15.77:0/2076203867 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58034070 con 0x7f8d6415e000 2024-03-06T07:22:07.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f7e69363640 1 -- 172.21.15.77:0/3697094355 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c034440 con 0x7f7e6406c8b0 2024-03-06T07:22:07.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd04e794640 1 -- 172.21.15.77:0/3104187255 >> v1:172.21.15.158:6789/0 conn(0x7fd0480cf930 legacy=0x7fd048161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd04e794640 1 -- 172.21.15.77:0/3104187255 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0480cf130 con 0x7fd0481622e0 2024-03-06T07:22:07.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f700a195640 1 -- 172.21.15.77:0/27047575 shutdown_connections 2024-03-06T07:22:07.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f700a195640 1 --2- 172.21.15.77:0/27047575 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7004070740 0x7f700415a280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f700a195640 1 -- 172.21.15.77:0/27047575 >> 172.21.15.77:0/27047575 conn(0x7f70040b16e0 msgr2=0x7f70040b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f4d45974640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d38006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f4d45974640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d38006630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35782/0 (socket says 172.21.15.77:35782) 2024-03-06T07:22:07.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f4d45974640 1 -- 172.21.15.77:0/3671365317 learned_addr learned my addr 172.21.15.77:0/3671365317 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.782+0000 7f5990e22640 1 -- 172.21.15.77:0/290988602 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c031830 con 0x7f598c06de20 2024-03-06T07:22:07.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f4d45974640 1 -- 172.21.15.77:0/3671365317 >> v1:172.21.15.158:6789/0 conn(0x7f4d38055d10 legacy=0x7f4d380366a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f4d45974640 1 -- 172.21.15.77:0/3671365317 >> v1:172.21.15.77:6789/0 conn(0x7f4d380df770 legacy=0x7f4d3804fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f700a195640 1 -- 172.21.15.77:0/27047575 shutdown_connections 2024-03-06T07:22:07.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f5994088640 1 -- 172.21.15.77:0/290988602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c06de20 msgr2=0x7f598c06fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f5990e22640 1 -- 172.21.15.77:0/290988602 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c035020 con 0x7f598c06de20 2024-03-06T07:22:07.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f0fc2715640 1 Processor -- start 2024-03-06T07:22:07.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f5994088640 1 --2- 172.21.15.77:0/290988602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c06de20 0x7f598c06fe70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f597c0097a0 tx=0x7f597c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f700a195640 1 -- 172.21.15.77:0/27047575 wait complete. 2024-03-06T07:22:07.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f8d690ed640 1 -- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d6415e000 msgr2=0x7f8d640d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f8d690ed640 1 --2- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d6415e000 0x7f8d640d0680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8d580097a0 tx=0x7f8d5802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:07.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7fd04e794640 1 --2- 172.21.15.77:0/3104187255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0481622e0 0x7fd0480d45b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd0440097a0 tx=0x7fd04402b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3551ae4735cbb129 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd6c3fee640 1 -- 172.21.15.77:0/946061240 shutdown_connections 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd6c3fee640 1 --2- 172.21.15.77:0/946061240 >> v2:172.21.15.158:3300/0 conn(0x7fd6bc0ce8d0 0x7fd6bc0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd6c3fee640 1 --2- 172.21.15.77:0/946061240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc14bb10 0x7fd6bc165fa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7fd6c3fee640 1 -- 172.21.15.77:0/946061240 >> 172.21.15.77:0/946061240 conn(0x7fd6bc0b1830 msgr2=0x7fd6bc0b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/3697094355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e6406c8b0 msgr2=0x7f7e6406ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.783+0000 7f7e6c5c9640 1 --2- 172.21.15.77:0/3697094355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e6406c8b0 0x7f7e6406ccc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e6406ea80 tx=0x7f7e5c02be40 comp rx=0 tx=0).stop 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/3697094355 shutdown_connections 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f7e6c5c9640 1 --2- 172.21.15.77:0/3697094355 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e6406c8b0 0x7f7e6406ccc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f7e6c5c9640 1 --2- 172.21.15.77:0/3697094355 >> v2:172.21.15.77:3300/0 conn(0x7f7e640b48d0 0x7f7e64070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/3697094355 >> 172.21.15.77:0/3697094355 conn(0x7f7e640b16c0 msgr2=0x7f7e640b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f8d690ed640 1 -- 172.21.15.77:0/2076203867 shutdown_connections 2024-03-06T07:22:07.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f8d690ed640 1 --2- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640d4810 0x7f8d640d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f8d690ed640 1 --2- 172.21.15.77:0/2076203867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d6415e000 0x7f8d640d0680 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f8d690ed640 1 -- 172.21.15.77:0/2076203867 >> 172.21.15.77:0/2076203867 conn(0x7f8d640d3cf0 msgr2=0x7f8d640d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/3697094355 shutdown_connections 2024-03-06T07:22:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f4d45974640 1 -- 172.21.15.77:0/3671365317 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d380071f0 con 0x7f4d38006220 2024-03-06T07:22:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7fd04cf91640 1 -- 172.21.15.77:0/3104187255 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044037070 con 0x7fd0481622e0 2024-03-06T07:22:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7fd04cf91640 1 -- 172.21.15.77:0/3104187255 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044030510 con 0x7fd0481622e0 2024-03-06T07:22:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7fd04cf91640 1 -- 172.21.15.77:0/3104187255 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038001670 con 0x7fd0481622e0 2024-03-06T07:22:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f5994088640 1 -- 172.21.15.77:0/290988602 shutdown_connections 2024-03-06T07:22:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f5994088640 1 --2- 172.21.15.77:0/290988602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c06de20 0x7f598c06fe70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f5994088640 1 -- 172.21.15.77:0/290988602 >> 172.21.15.77:0/290988602 conn(0x7f598c0d0a10 msgr2=0x7f598c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/3697094355 wait complete. 2024-03-06T07:22:07.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f0fc2715640 1 -- start start 2024-03-06T07:22:07.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7f8d690ed640 1 -- 172.21.15.77:0/2076203867 shutdown_connections 2024-03-06T07:22:07.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f4d45974640 1 --2- 172.21.15.77:0/3671365317 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d38006630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c0098a0 tx=0x7f4d3c02ba00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9227bbc814960146 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc2715640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fb400aaa0 0x7f0fb40485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f4d37fff640 1 -- 172.21.15.77:0/3671365317 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c02f820 con 0x7f4d38006220 2024-03-06T07:22:07.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7fd0501f7640 1 -- 172.21.15.77:0/3104187255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0481622e0 msgr2=0x7fd0480d45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7fd0501f7640 1 --2- 172.21.15.77:0/3104187255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0481622e0 0x7fd0480d45b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd0440097a0 tx=0x7fd04402b7b0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc2715640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb40cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f4d37fff640 1 -- 172.21.15.77:0/3671365317 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c02f9c0 con 0x7f4d38006220 2024-03-06T07:22:07.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f4d37fff640 1 -- 172.21.15.77:0/3671365317 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c02fd90 con 0x7f4d38006220 2024-03-06T07:22:07.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7fd0501f7640 1 -- 172.21.15.77:0/3104187255 shutdown_connections 2024-03-06T07:22:07.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc2715640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fb40df420 0x7f0fb404aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7fd0501f7640 1 --2- 172.21.15.77:0/3104187255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0481622e0 0x7fd0480d45b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7fd0501f7640 1 --2- 172.21.15.77:0/3104187255 >> v2:172.21.15.77:3300/0 conn(0x7fd04815e000 0x7fd0480d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7fd0501f7640 1 -- 172.21.15.77:0/3104187255 >> 172.21.15.77:0/3104187255 conn(0x7fd04809cfc0 msgr2=0x7fd0480d3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc0f12640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb40cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc0f12640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb40cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35798/0 (socket says 172.21.15.77:35798) 2024-03-06T07:22:07.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f8d690ed640 1 -- 172.21.15.77:0/2076203867 wait complete. 2024-03-06T07:22:07.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7fd0501f7640 1 -- 172.21.15.77:0/3104187255 shutdown_connections 2024-03-06T07:22:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc1f14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fb40df420 0x7f0fb404aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc1f14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fb40df420 0x7f0fb404aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41400/0 (socket says 172.21.15.77:41400) 2024-03-06T07:22:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc0f12640 1 -- 172.21.15.77:0/2834361822 learned_addr learned my addr 172.21.15.77:0/2834361822 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f0fc2715640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4052cc0 con 0x7f0fb40df420 2024-03-06T07:22:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f0fc2715640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4052b40 con 0x7f0fb400aaa0 2024-03-06T07:22:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f0fc2715640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4038cb0 con 0x7f0fb40382c0 2024-03-06T07:22:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f4d473d7640 1 -- 172.21.15.77:0/3671365317 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 msgr2=0x7f4d38006630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f4d473d7640 1 --2- 172.21.15.77:0/3671365317 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d38006630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c0098a0 tx=0x7f4d3c02ba00 comp rx=0 tx=0).stop 2024-03-06T07:22:07.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f7e6c5c9640 1 Processor -- start 2024-03-06T07:22:07.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f4d473d7640 1 -- 172.21.15.77:0/3671365317 shutdown_connections 2024-03-06T07:22:07.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f4d473d7640 1 --2- 172.21.15.77:0/3671365317 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d38006630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f4d473d7640 1 -- 172.21.15.77:0/3671365317 >> 172.21.15.77:0/3671365317 conn(0x7f4d3801e6c0 msgr2=0x7f4d380553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f4d473d7640 1 -- 172.21.15.77:0/3671365317 shutdown_connections 2024-03-06T07:22:07.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f0fc0f12640 1 -- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fb400aaa0 msgr2=0x7f0fb40485f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:07.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f0fc0f12640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fb400aaa0 0x7f0fb40485f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f0fc0f12640 1 -- 172.21.15.77:0/2834361822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fb40df420 msgr2=0x7f0fb404aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f4d473d7640 1 -- 172.21.15.77:0/3671365317 wait complete. 2024-03-06T07:22:07.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f0fc0f12640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fb40df420 0x7f0fb404aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f0fc0f12640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4048bd0 con 0x7f0fb40382c0 2024-03-06T07:22:07.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f8d690ed640 1 Processor -- start 2024-03-06T07:22:07.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7fd6c3fee640 1 -- 172.21.15.77:0/946061240 shutdown_connections 2024-03-06T07:22:07.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.784+0000 7fd6c3fee640 1 -- 172.21.15.77:0/946061240 wait complete. 2024-03-06T07:22:07.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7fd6c3fee640 1 Processor -- start 2024-03-06T07:22:07.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7fd6c3fee640 1 -- start start 2024-03-06T07:22:07.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f8d690ed640 1 -- start start 2024-03-06T07:22:07.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f0fc1f14640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fb40df420 0x7f0fb404aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7fd0501f7640 1 -- 172.21.15.77:0/3104187255 wait complete. 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f700a195640 1 Processor -- start 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.785+0000 7f700a195640 1 -- start start 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f700a195640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f700406e830 0x7f700414ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f700a195640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7004070740 0x7f7004156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f700a195640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7004154060 0x7f70041572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f700a195640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040af1d0 con 0x7f7004154060 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f700a195640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040af050 con 0x7f7004070740 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f700a195640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7004154490 con 0x7f700406e830 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f7009994640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7004154060 0x7f70041572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f7009994640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7004154060 0x7f70041572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41404/0 (socket says 172.21.15.77:41404) 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f7009994640 1 -- 172.21.15.77:0/1861576287 learned_addr learned my addr 172.21.15.77:0/1861576287 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7009994640 1 -- 172.21.15.77:0/1861576287 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f700406e830 msgr2=0x7f700414ca90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7009994640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f700406e830 0x7f700414ca90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7009994640 1 -- 172.21.15.77:0/1861576287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7004070740 msgr2=0x7f7004156db0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7009994640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7004070740 0x7f7004156db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7009994640 1 -- 172.21.15.77:0/1861576287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7004154e00 con 0x7f7004154060 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f5994088640 1 -- 172.21.15.77:0/290988602 shutdown_connections 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f0fc0f12640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb40cb550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00b480 tx=0x7f0fac00b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f5994088640 1 -- 172.21.15.77:0/290988602 wait complete. 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.786+0000 7f7e6c5c9640 1 -- start start 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6c5c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7e6406c8b0 0x7f7e641419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6c5c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e640b48d0 0x7f7e64159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6c5c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e64165bf0 0x7f7e64164780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6c5c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64071080 con 0x7f7e64165bf0 2024-03-06T07:22:07.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6c5c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640b3570 con 0x7f7e640b48d0 2024-03-06T07:22:07.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6c5c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640b3a90 con 0x7f7e6406c8b0 2024-03-06T07:22:07.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7009994640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7004154060 0x7f70041572f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00c780 tx=0x7f6ffc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac014070 con 0x7f0fb40382c0 2024-03-06T07:22:07.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd0501f7640 1 Processor -- start 2024-03-06T07:22:07.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6a365640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7e6406c8b0 0x7f7e641419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6a365640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7e6406c8b0 0x7f7e641419d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35814/0 (socket says 172.21.15.77:35814) 2024-03-06T07:22:07.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e69b64640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e640b48d0 0x7f7e64159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f4d473d7640 1 Processor -- start 2024-03-06T07:22:07.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f0fc2715640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb40e09d0 con 0x7f0fb40382c0 2024-03-06T07:22:07.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f7e6a365640 1 -- 172.21.15.77:0/507031568 learned_addr learned my addr 172.21.15.77:0/507031568 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7f4d473d7640 1 -- start start 2024-03-06T07:22:07.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f7e6ab66640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e64165bf0 0x7f7e64164780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f0fc2715640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb40e0f90 con 0x7f0fb40382c0 2024-03-06T07:22:07.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f7e6a365640 1 -- 172.21.15.77:0/507031568 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e640b48d0 msgr2=0x7f7e64159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f7e6a365640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7e640b48d0 0x7f7e64159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7fd0501f7640 1 -- start start 2024-03-06T07:22:07.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f7e6a365640 1 -- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e64165bf0 msgr2=0x7f7e64164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c3fee640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6bc06afb0 0x7fd6bc0b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c3fee640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc0ce8d0 0x7fd6bc1433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c3fee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6bc14bb10 0x7fd6bc1411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c3fee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc071770 con 0x7fd6bc14bb10 2024-03-06T07:22:07.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c3fee640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc071ac0 con 0x7fd6bc06afb0 2024-03-06T07:22:07.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c3fee640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc14bef0 con 0x7fd6bc0ce8d0 2024-03-06T07:22:07.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c258b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6bc14bb10 0x7fd6bc1411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c1589640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc0ce8d0 0x7fd6bc1433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c1589640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc0ce8d0 0x7fd6bc1433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35810/0 (socket says 172.21.15.77:35810) 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c258b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6bc14bb10 0x7fd6bc1411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41412/0 (socket says 172.21.15.77:41412) 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.787+0000 7fd6c1589640 1 -- 172.21.15.77:0/571197917 learned_addr learned my addr 172.21.15.77:0/571197917 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7fd6c258b640 1 -- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc0ce8d0 msgr2=0x7fd6bc1433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7fd6c258b640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6bc0ce8d0 0x7fd6bc1433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7fd6c258b640 1 -- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6bc06afb0 msgr2=0x7fd6bc0b2980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7fd6c258b640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6bc06afb0 0x7fd6bc0b2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7fd6c258b640 1 -- 172.21.15.77:0/571197917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc143da0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7fd6c258b640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6bc14bb10 0x7fd6bc1411a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd6b801c590 tx=0x7fd6b801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f7e6a365640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e64165bf0 0x7f7e64164780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f7e6a365640 1 -- 172.21.15.77:0/507031568 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64141f10 con 0x7f7e6406c8b0 2024-03-06T07:22:07.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f7e6ab66640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e64165bf0 0x7f7e64164780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:07.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac010870 con 0x7f0fb40382c0 2024-03-06T07:22:07.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac010c40 con 0x7f0fb40382c0 2024-03-06T07:22:07.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f4d473d7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d3804dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f4d473d7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d38055d10 0x7f4d380d6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f4d473d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d380df770 0x7f4d3804a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f4d473d7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380dfbe0 con 0x7f4d380df770 2024-03-06T07:22:07.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f4d473d7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380560f0 con 0x7f4d38055d10 2024-03-06T07:22:07.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.788+0000 7f4d473d7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3803a0a0 con 0x7f4d38006220 2024-03-06T07:22:07.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d690ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640c80e0 0x7f8d6414fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f7e6a365640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7e6406c8b0 0x7f7e641419d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000c780 tx=0x7f7e6000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45173640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d3804dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45974640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d380df770 0x7f4d3804a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45173640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d3804dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35818/0 (socket says 172.21.15.77:35818) 2024-03-06T07:22:07.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd0501f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0480cf930 0x7fd0481fab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd0501f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04815e000 0x7fd048200b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45173640 1 -- 172.21.15.77:0/2582591158 learned_addr learned my addr 172.21.15.77:0/2582591158 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60018070 con 0x7f7e6406c8b0 2024-03-06T07:22:07.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f5994088640 1 Processor -- start 2024-03-06T07:22:07.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/507031568 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e64159d70 con 0x7f7e6406c8b0 2024-03-06T07:22:07.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/507031568 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6415a330 con 0x7f7e6406c8b0 2024-03-06T07:22:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d690ed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d640d4810 0x7f8d6409f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d62d9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640c80e0 0x7f8d6414fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d62d9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640c80e0 0x7f8d6414fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35828/0 (socket says 172.21.15.77:35828) 2024-03-06T07:22:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd0501f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0481622e0 0x7fd0480c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d44972640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d38055d10 0x7f4d380d6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d62d9d640 1 -- 172.21.15.77:0/3136439242 learned_addr learned my addr 172.21.15.77:0/3136439242 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d6259c640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d640d4810 0x7f8d6409f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d690ed640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6415e000 0x7f8d6409da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d690ed640 1 -- 172.21.15.77:0/3136439242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6415e3e0 con 0x7f8d6415e000 2024-03-06T07:22:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d690ed640 1 -- 172.21.15.77:0/3136439242 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640d22d0 con 0x7f8d640d4810 2024-03-06T07:22:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f8d690ed640 1 -- 172.21.15.77:0/3136439242 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640cf7d0 con 0x7f8d640c80e0 2024-03-06T07:22:07.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e600072d0 con 0x7f7e6406c8b0 2024-03-06T07:22:07.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600047e0 con 0x7f7e6406c8b0 2024-03-06T07:22:07.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd0501f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0480ce790 con 0x7fd04815e000 2024-03-06T07:22:07.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd0501f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04815e3e0 con 0x7fd0481622e0 2024-03-06T07:22:07.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd0501f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0480ce910 con 0x7fd0480cf930 2024-03-06T07:22:07.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd04d792640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04815e000 0x7fd048200b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd04d792640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04815e000 0x7fd048200b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41440/0 (socket says 172.21.15.77:41440) 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd04d792640 1 -- 172.21.15.77:0/932778505 learned_addr learned my addr 172.21.15.77:0/932778505 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd04e794640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0481622e0 0x7fd0480c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7fd04df93640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0480cf930 0x7fd0481fab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd04d792640 1 -- 172.21.15.77:0/932778505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0480cf930 msgr2=0x7fd0481fab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd04d792640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0480cf930 0x7fd0481fab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd04d792640 1 -- 172.21.15.77:0/932778505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0481622e0 msgr2=0x7fd0480c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd04d792640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0481622e0 0x7fd0480c97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd04d792640 1 -- 172.21.15.77:0/932778505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0480b7d60 con 0x7fd04815e000 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd04d792640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04815e000 0x7fd048200b50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd03800c6f0 tx=0x7fd03800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f5994088640 1 -- start start 2024-03-06T07:22:07.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f0fc2715640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0fb4048eb0 con 0x7f0fb40382c0 2024-03-06T07:22:07.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f8d6359e640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6415e000 0x7f8d6409da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f8d62d9d640 1 -- 172.21.15.77:0/3136439242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d640d4810 msgr2=0x7f8d6409f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f8d62d9d640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d640d4810 0x7f8d6409f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f8d62d9d640 1 -- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6415e000 msgr2=0x7f8d6409da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fac017070 con 0x7f0fb40382c0 2024-03-06T07:22:07.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f8d62d9d640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6415e000 0x7f8d6409da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f8d62d9d640 1 -- 172.21.15.77:0/3136439242 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6414ceb0 con 0x7f8d640c80e0 2024-03-06T07:22:07.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f8d6359e640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6415e000 0x7f8d6409da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc017070 con 0x7f7004154060 2024-03-06T07:22:07.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc003d80 con 0x7f7004154060 2024-03-06T07:22:07.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0052b0 con 0x7f7004154060 2024-03-06T07:22:07.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7f8d62d9d640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640c80e0 0x7f8d6414fa90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d580098c0 tx=0x7f8d580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038005ad0 con 0x7fd04815e000 2024-03-06T07:22:07.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45974640 1 -- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 msgr2=0x7f4d3804dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45974640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d38006220 0x7f4d3804dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45974640 1 -- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d38055d10 msgr2=0x7f4d380d6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45974640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d38055d10 0x7f4d380d6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.789+0000 7f4d45974640 1 -- 172.21.15.77:0/2582591158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3804efa0 con 0x7f4d380df770 2024-03-06T07:22:07.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f700a195640 1 -- 172.21.15.77:0/1861576287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7004151500 con 0x7f7004154060 2024-03-06T07:22:07.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f700a195640 1 -- 172.21.15.77:0/1861576287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7004151ac0 con 0x7f7004154060 2024-03-06T07:22:07.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0481fb050 con 0x7fd04815e000 2024-03-06T07:22:07.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd048201740 con 0x7fd04815e000 2024-03-06T07:22:07.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e6001b070 con 0x7f7e6406c8b0 2024-03-06T07:22:07.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8028600 con 0x7fd6bc14bb10 2024-03-06T07:22:07.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b80287a0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.790+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8028b70 con 0x7fd6bc14bb10 2024-03-06T07:22:07.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc15b100 con 0x7fd6bc14bb10 2024-03-06T07:22:07.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc15b6c0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f0fb1ffb640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fa006b5a0 0x7f0fa006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fac089f50 con 0x7f0fb40382c0 2024-03-06T07:22:07.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f0fb1ffb640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f0fa0073820 0x7f0fa0075c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f4d45974640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d380df770 0x7f4d3804a720 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800e550 tx=0x7f4d3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fac01be50 con 0x7f0fa0073820 2024-03-06T07:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fac08c510 con 0x7f0fb40382c0 2024-03-06T07:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f0fc1713640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fa006b5a0 0x7f0fa006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f5994088640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f598c06de20 0x7f598c204b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f5994088640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c1627a0 0x7f598c164690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f5994088640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f598c163f90 0x7f598c15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f5994088640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c162c20 con 0x7f598c1627a0 2024-03-06T07:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f5994088640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c163b00 con 0x7f598c06de20 2024-03-06T07:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f5994088640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c06b930 con 0x7f598c163f90 2024-03-06T07:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd6bc14aed0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f5991e24640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f598c06de20 0x7f598c204b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.791+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/507031568 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e4c001030 con 0x7f7e6406c8b0 2024-03-06T07:22:07.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f0fc1f14640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f0fa0073820 0x7f0fa0075c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f5991623640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c1627a0 0x7f598c164690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f5991623640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c1627a0 0x7f598c164690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41470/0 (socket says 172.21.15.77:41470) 2024-03-06T07:22:07.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f5991623640 1 -- 172.21.15.77:0/1317882737 learned_addr learned my addr 172.21.15.77:0/1317882737 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:07.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f0fc1713640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fa006b5a0 0x7f0fa006da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800b7f0 tx=0x7f0fb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f8d4b7fe640 1 -- 172.21.15.77:0/3136439242 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58034050 con 0x7f8d640c80e0 2024-03-06T07:22:07.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f5992625640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f598c163f90 0x7f598c15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f700a195640 1 -- 172.21.15.77:0/1861576287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6fe4001030 con 0x7f7004154060 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f8d4b7fe640 1 -- 172.21.15.77:0/3136439242 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d58048070 con 0x7f8d640c80e0 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f8d690ed640 1 -- 172.21.15.77:0/3136439242 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6414d150 con 0x7f8d640c80e0 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f7e5affd640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7e4806b500 0x7f7e4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e60089710 con 0x7f7e6406c8b0 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f7e5affd640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f7e480736a0 0x7f7e48075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e6000ae50 con 0x7f7e480736a0 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.792+0000 7f7e69b64640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7e4806b500 0x7f7e4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f7e69b64640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7e4806b500 0x7f7e4806d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e6415c9b0 tx=0x7f7e54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e60054000 con 0x7f7e6406c8b0 2024-03-06T07:22:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f7e6ab66640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f7e480736a0 0x7f7e48075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f0fc1f14640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f0fa0073820 0x7f0fa0075c40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f5992625640 1 -- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f598c06de20 msgr2=0x7f598c204b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f5992625640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f598c06de20 0x7f598c204b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f5992625640 1 -- 172.21.15.77:0/1317882737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c1627a0 msgr2=0x7f598c164690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f5992625640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c1627a0 0x7f598c164690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f5992625640 1 -- 172.21.15.77:0/1317882737 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c2050b0 con 0x7f598c163f90 2024-03-06T07:22:07.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f5991623640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c1627a0 0x7f598c164690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:07.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c041400 con 0x7f4d380df770 2024-03-06T07:22:07.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c046070 con 0x7f4d380df770 2024-03-06T07:22:07.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c041400 con 0x7f4d380df770 2024-03-06T07:22:07.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d3804d120 con 0x7f4d380df770 2024-03-06T07:22:07.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f8d690ed640 1 -- 172.21.15.77:0/3136439242 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6414d710 con 0x7f8d640c80e0 2024-03-06T07:22:07.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f7e6ab66640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f7e480736a0 0x7f7e48075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3804d6e0 con 0x7f4d380df770 2024-03-06T07:22:07.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f8d4b7fe640 1 -- 172.21.15.77:0/3136439242 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58044ad0 con 0x7f8d640c80e0 2024-03-06T07:22:07.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f5992625640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f598c163f90 0x7f598c15b8e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f598400ca70 tx=0x7f598400cf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ffc01a070 con 0x7f7004154060 2024-03-06T07:22:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f6ff9ffb640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe806b520 0x7f6fe806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ffc089850 con 0x7f7004154060 2024-03-06T07:22:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd038005c70 con 0x7fd04815e000 2024-03-06T07:22:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038007270 con 0x7fd04815e000 2024-03-06T07:22:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6b8028d10 con 0x7fd6bc14bb10 2024-03-06T07:22:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7fd6b27fc640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a006b5a0 0x7fd6a006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6b809b4e0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7fd6b27fc640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7fd6a0073820 0x7fd6a0075c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6b801bf40 con 0x7fd6a0073820 2024-03-06T07:22:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6b809b8d0 con 0x7fd6bc14bb10 2024-03-06T07:22:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.793+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0480a1c30 con 0x7fd04815e000 2024-03-06T07:22:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 <== osd.5 v2:172.21.15.158:6820/3476303095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0fac01be50 con 0x7f0fa0073820 2024-03-06T07:22:07.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f6ff9ffb640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f6fe8073740 0x7f6fe8075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7f7009193640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe806b520 0x7f6fe806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984017070 con 0x7f598c163f90 2024-03-06T07:22:07.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd038007410 con 0x7fd04815e000 2024-03-06T07:22:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fe8076210 con 0x7f6fe8073740 2024-03-06T07:22:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ffc08e2a0 con 0x7f7004154060 2024-03-06T07:22:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598400ab70 con 0x7f598c163f90 2024-03-06T07:22:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.794+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984012400 con 0x7f598c163f90 2024-03-06T07:22:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7f5994088640 1 -- 172.21.15.77:0/1317882737 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c0ac480 con 0x7f598c163f90 2024-03-06T07:22:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.795+0000 7f5994088640 1 -- 172.21.15.77:0/1317882737 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c0aca40 con 0x7f598c163f90 2024-03-06T07:22:07.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7fd03e7fc640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd02c06b5a0 0x7fd02c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f7009193640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe806b520 0x7f6fe806d9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4000f60 tx=0x7f6ff40079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f7008992640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f6fe8073740 0x7f6fe8075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f8d4b7fe640 1 -- 172.21.15.77:0/3136439242 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d58037070 con 0x7f8d640c80e0 2024-03-06T07:22:07.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd03808a060 con 0x7fd04815e000 2024-03-06T07:22:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7fd03e7fc640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fd02c0737c0 0x7fd02c075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 --> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd038020e50 con 0x7fd02c0737c0 2024-03-06T07:22:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd038091050 con 0x7fd04815e000 2024-03-06T07:22:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7fd6c1589640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7fd6a0073820 0x7fd6a0075c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7fd04df93640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd02c06b5a0 0x7fd02c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5968001030 con 0x7f598c163f90 2024-03-06T07:22:07.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 <== osd.3 v2:172.21.15.77:6816/461341877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7e6000ae50 con 0x7f7e480736a0 2024-03-06T07:22:07.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f8d4b7fe640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d5006b5a0 0x7f8d5006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f7008992640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f6fe8073740 0x7f6fe8075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7fd04e794640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fd02c0737c0 0x7fd02c075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f8d6259c640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d5006b5a0 0x7f8d5006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7fd04df93640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd02c06b5a0 0x7fd02c06da60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd0480b1510 tx=0x7fd0400094d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4d38008800 con 0x7f4d380df770 2024-03-06T07:22:07.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.796+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d3c040070 con 0x7f4d380df770 2024-03-06T07:22:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f4d35ffb640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d2406b8d0 0x7f4d2406dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d3c037cc0 con 0x7f4d380df770 2024-03-06T07:22:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f4d35ffb640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f4d24073ad0 0x7f4d24075ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d3c042ec0 con 0x7f4d24073ad0 2024-03-06T07:22:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d3c0b3050 con 0x7f4d380df770 2024-03-06T07:22:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f4d44972640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f4d24073ad0 0x7f4d24075ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7f8d4b7fe640 1 -- 172.21.15.77:0/3136439242 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d580b4620 con 0x7f8d640c80e0 2024-03-06T07:22:07.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7f8d6259c640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d5006b5a0 0x7f8d5006da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d640cf330 tx=0x7f8d5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7f4d45173640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d2406b8d0 0x7f4d2406dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7fd6c1589640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7fd6a0073820 0x7fd6a0075c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7fd6c1d8a640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a006b5a0 0x7fd6a006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.797+0000 7fd04e794640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fd02c0737c0 0x7fd02c075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 <== osd.0 v2:172.21.15.77:6808/3119591583 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6fe8076210 con 0x7f6fe8073740 2024-03-06T07:22:07.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7f8d690ed640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f8d6406bab0 0x7f8d6406be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f598401a070 con 0x7f598c163f90 2024-03-06T07:22:07.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7f8d690ed640 1 -- 172.21.15.77:0/3136439242 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d6404f330 con 0x7f8d6406bab0 2024-03-06T07:22:07.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7fd6c1d8a640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a006b5a0 0x7fd6a006da60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4035000 tx=0x7fd6b4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 <== osd.4 v2:172.21.15.158:6800/2421130833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd6b801bf40 con 0x7fd6a0073820 2024-03-06T07:22:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7f8d6359e640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f8d6406bab0 0x7f8d6406be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7f4d44972640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f4d24073ad0 0x7f4d24075ef0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.798+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 <== osd.2 v2:172.21.15.77:6824/107698445 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd038020e50 con 0x7fd02c0737c0 2024-03-06T07:22:07.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f59827fc640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f597006b5d0 0x7f597006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f4d45173640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d2406b8d0 0x7f4d2406dd90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4d4007aee0 tx=0x7f4d4006d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f8d6359e640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f8d6406bab0 0x7f8d6406be90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5984089da0 con 0x7f598c163f90 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f59827fc640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f59700737f0 0x7f5970075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59700762c0 con 0x7f59700737f0 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f598408c400 con 0x7f598c163f90 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f5991e24640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f597006b5d0 0x7f597006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.799+0000 7f5991623640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f59700737f0 0x7f5970075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.800+0000 7f5991e24640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f597006b5d0 0x7f597006da90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f597c036000 tx=0x7f597c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.800+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 <== osd.7 v2:172.21.15.158:6816/3281717214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4d3c042ec0 con 0x7f4d24073ad0 2024-03-06T07:22:07.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.800+0000 7f8d4b7fe640 1 -- 172.21.15.77:0/3136439242 <== osd.1 v2:172.21.15.77:6800/1774367160 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d6404f330 con 0x7f8d6406bab0 2024-03-06T07:22:07.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.801+0000 7f5991623640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f59700737f0 0x7f5970075c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.803+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 <== osd.6 v2:172.21.15.158:6801/2971141295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f59700762c0 con 0x7f59700737f0 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.823+0000 7f7e6c5c9640 1 -- 172.21.15.77:0/507031568 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e4c002dd0 con 0x7f7e480736a0 2024-03-06T07:22:07.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.824+0000 7f7e5affd640 1 -- 172.21.15.77:0/507031568 <== osd.3 v2:172.21.15.77:6816/461341877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7e4c002dd0 con 0x7f7e480736a0 2024-03-06T07:22:07.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.824+0000 7f7e58ff9640 1 -- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f7e480736a0 msgr2=0x7f7e48075ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.824+0000 7f7e58ff9640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f7e480736a0 0x7f7e48075ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e58ff9640 1 -- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7e4806b500 msgr2=0x7f7e4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e58ff9640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7e4806b500 0x7f7e4806d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e6415c9b0 tx=0x7f7e54008040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e58ff9640 1 -- 172.21.15.77:0/507031568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7e6406c8b0 msgr2=0x7f7e641419d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e58ff9640 1 --2- 172.21.15.77:0/507031568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7e6406c8b0 0x7f7e641419d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000c780 tx=0x7f7e6000cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e6ab66640 1 -- 172.21.15.77:0/507031568 reap_dead start 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e58ff9640 1 -- 172.21.15.77:0/507031568 shutdown_connections 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e58ff9640 1 -- 172.21.15.77:0/507031568 >> 172.21.15.77:0/507031568 conn(0x7f7e640b16c0 msgr2=0x7f7e640b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e58ff9640 1 -- 172.21.15.77:0/507031568 shutdown_connections 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.825+0000 7f7e58ff9640 1 -- 172.21.15.77:0/507031568 wait complete. 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.829+0000 7f0fc2715640 1 -- 172.21.15.77:0/2834361822 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0fb4048eb0 con 0x7f0fa0073820 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.829+0000 7f0fb1ffb640 1 -- 172.21.15.77:0/2834361822 <== osd.5 v2:172.21.15.158:6820/3476303095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0fbc0b9040 con 0x7f0fa0073820 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.830+0000 7f0fa77fe640 1 -- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f0fa0073820 msgr2=0x7f0fa0075c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.830+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f0fa0073820 0x7f0fa0075c40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.830+0000 7f0fa77fe640 1 -- 172.21.15.77:0/2834361822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fa006b5a0 msgr2=0x7f0fa006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.830+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fa006b5a0 0x7f0fa006da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800b7f0 tx=0x7f0fb8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f0fa77fe640 1 -- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 msgr2=0x7f0fb40cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/2834361822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fb40382c0 0x7f0fb40cb550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00b480 tx=0x7f0fac00b940 comp rx=0 tx=0).stop 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f0fc1f14640 1 -- 172.21.15.77:0/2834361822 reap_dead start 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f0fa77fe640 1 -- 172.21.15.77:0/2834361822 shutdown_connections 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f0fa77fe640 1 -- 172.21.15.77:0/2834361822 >> 172.21.15.77:0/2834361822 conn(0x7f0fb4009fb0 msgr2=0x7f0fb4039080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 --> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0481e7bc0 con 0x7fd02c0737c0 2024-03-06T07:22:07.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f0fa77fe640 1 -- 172.21.15.77:0/2834361822 shutdown_connections 2024-03-06T07:22:07.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f0fa77fe640 1 -- 172.21.15.77:0/2834361822 wait complete. 2024-03-06T07:22:07.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4d3804d920 con 0x7f4d24073ad0 2024-03-06T07:22:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7f700a195640 1 -- 172.21.15.77:0/1861576287 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6fe4002dd0 con 0x7f6fe8073740 2024-03-06T07:22:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.831+0000 7fd03e7fc640 1 -- 172.21.15.77:0/932778505 <== osd.2 v2:172.21.15.77:6824/107698445 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd0481e7bc0 con 0x7fd02c0737c0 2024-03-06T07:22:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7f6ff9ffb640 1 -- 172.21.15.77:0/1861576287 <== osd.0 v2:172.21.15.77:6808/3119591583 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6fe4002dd0 con 0x7f6fe8073740 2024-03-06T07:22:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7f4d35ffb640 1 -- 172.21.15.77:0/2582591158 <== osd.7 v2:172.21.15.158:6816/3281717214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4d3c042ec0 con 0x7f4d24073ad0 2024-03-06T07:22:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fd02c0737c0 msgr2=0x7fd02c075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd0501f7640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fd02c0737c0 0x7fd02c075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd02c06b5a0 msgr2=0x7fd02c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd0501f7640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd02c06b5a0 0x7fd02c06da60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd0480b1510 tx=0x7fd0400094d0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04815e000 msgr2=0x7fd048200b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd0501f7640 1 --2- 172.21.15.77:0/932778505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04815e000 0x7fd048200b50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd03800c6f0 tx=0x7fd03800cbb0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd04e794640 1 -- 172.21.15.77:0/932778505 reap_dead start 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd6bc14aed0 con 0x7fd6a0073820 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 shutdown_connections 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.832+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 >> 172.21.15.77:0/932778505 conn(0x7fd04809cfc0 msgr2=0x7fd0481e73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f6fef7fe640 1 -- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f6fe8073740 msgr2=0x7f6fe8075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 shutdown_connections 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f6fef7fe640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f6fe8073740 0x7f6fe8075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7fd0501f7640 1 -- 172.21.15.77:0/932778505 wait complete. 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7fd6b27fc640 1 -- 172.21.15.77:0/571197917 <== osd.4 v2:172.21.15.158:6800/2421130833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd6ac0098c0 con 0x7fd6a0073820 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f4d24073ad0 msgr2=0x7f4d24075ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f4d473d7640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f4d24073ad0 0x7f4d24075ef0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d2406b8d0 msgr2=0x7f4d2406dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f4d473d7640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d2406b8d0 0x7f4d2406dd90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4d4007aee0 tx=0x7f4d4006d040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d380df770 msgr2=0x7f4d3804a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f4d473d7640 1 --2- 172.21.15.77:0/2582591158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d380df770 0x7f4d3804a720 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800e550 tx=0x7f4d3c004730 comp rx=0 tx=0).stop 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f4d45974640 1 -- 172.21.15.77:0/2582591158 reap_dead start 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 shutdown_connections 2024-03-06T07:22:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 >> 172.21.15.77:0/2582591158 conn(0x7f4d3801e6c0 msgr2=0x7f4d3800bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f6fef7fe640 1 -- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe806b520 msgr2=0x7f6fe806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f6fef7fe640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6fe806b520 0x7f6fe806d9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4000f60 tx=0x7f6ff40079b0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f6fef7fe640 1 -- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7004154060 msgr2=0x7f70041572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f6fef7fe640 1 --2- 172.21.15.77:0/1861576287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7004154060 0x7f70041572f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00c780 tx=0x7f6ffc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f7009994640 1 -- 172.21.15.77:0/1861576287 reap_dead start 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f6fef7fe640 1 -- 172.21.15.77:0/1861576287 shutdown_connections 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.833+0000 7f6fef7fe640 1 -- 172.21.15.77:0/1861576287 >> 172.21.15.77:0/1861576287 conn(0x7f70040b16e0 msgr2=0x7f700406d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f6fef7fe640 1 -- 172.21.15.77:0/1861576287 shutdown_connections 2024-03-06T07:22:07.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f6fef7fe640 1 -- 172.21.15.77:0/1861576287 wait complete. 2024-03-06T07:22:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7fd6a0073820 msgr2=0x7fd6a0075c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c3fee640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7fd6a0073820 0x7fd6a0075c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f8d690ed640 1 -- 172.21.15.77:0/3136439242 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8d640b9760 con 0x7f8d6406bab0 2024-03-06T07:22:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5968002dd0 con 0x7f59700737f0 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a006b5a0 msgr2=0x7fd6a006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c3fee640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a006b5a0 0x7fd6a006da60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4035000 tx=0x7fd6b4030000 comp rx=0 tx=0).stop 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6bc14bb10 msgr2=0x7fd6bc1411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c3fee640 1 --2- 172.21.15.77:0/571197917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6bc14bb10 0x7fd6bc1411a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd6b801c590 tx=0x7fd6b801ca50 comp rx=0 tx=0).stop 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 shutdown_connections 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c258b640 1 -- 172.21.15.77:0/571197917 reap_dead start 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f4d473d7640 1 -- 172.21.15.77:0/2582591158 wait complete. 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f59827fc640 1 -- 172.21.15.77:0/1317882737 <== osd.6 v2:172.21.15.158:6801/2971141295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5968002dd0 con 0x7f59700737f0 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7f8d4b7fe640 1 -- 172.21.15.77:0/3136439242 <== osd.1 v2:172.21.15.77:6800/1774367160 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8d640b9760 con 0x7f8d6406bab0 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 shutdown_connections 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.834+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 >> 172.21.15.77:0/571197917 conn(0x7fd6bc0b1830 msgr2=0x7fd6bc06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 shutdown_connections 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7fd6c3fee640 1 -- 172.21.15.77:0/571197917 wait complete. 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f59700737f0 msgr2=0x7f5970075c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7f596ffff640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f59700737f0 0x7f5970075c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f597006b5d0 msgr2=0x7f597006da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7f596ffff640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f597006b5d0 0x7f597006da90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f597c036000 tx=0x7f597c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f598c163f90 msgr2=0x7f598c15b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7f596ffff640 1 --2- 172.21.15.77:0/1317882737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f598c163f90 0x7f598c15b8e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f598400ca70 tx=0x7f598400cf30 comp rx=0 tx=0).stop 2024-03-06T07:22:07.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7f8d497fa640 1 -- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f8d6406bab0 msgr2=0x7f8d6406be90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.835+0000 7f8d497fa640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f8d6406bab0 0x7f8d6406be90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d497fa640 1 -- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d5006b5a0 msgr2=0x7f8d5006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d497fa640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d5006b5a0 0x7f8d5006da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d640cf330 tx=0x7f8d5c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:07.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d497fa640 1 -- 172.21.15.77:0/3136439242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640c80e0 msgr2=0x7f8d6414fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d497fa640 1 --2- 172.21.15.77:0/3136439242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d640c80e0 0x7f8d6414fa90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d580098c0 tx=0x7f8d580046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d6359e640 1 -- 172.21.15.77:0/3136439242 reap_dead start 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d497fa640 1 -- 172.21.15.77:0/3136439242 shutdown_connections 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d497fa640 1 -- 172.21.15.77:0/3136439242 >> 172.21.15.77:0/3136439242 conn(0x7f8d640d3cf0 msgr2=0x7f8d640d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d497fa640 1 -- 172.21.15.77:0/3136439242 shutdown_connections 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.836+0000 7f8d497fa640 1 -- 172.21.15.77:0/3136439242 wait complete. 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.837+0000 7f5992625640 1 -- 172.21.15.77:0/1317882737 reap_dead start 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.837+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 shutdown_connections 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.837+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 >> 172.21.15.77:0/1317882737 conn(0x7f598c0d0a10 msgr2=0x7f598c0b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.838+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 shutdown_connections 2024-03-06T07:22:07.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:07.838+0000 7f596ffff640 1 -- 172.21.15.77:0/1317882737 wait complete. 2024-03-06T07:22:07.888 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:07.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:07.889 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:07.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:07.890 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:07.891 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:07.891 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:07.892 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:07.892 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:07.893 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:07.893 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:07.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:07.895 INFO:teuthology.orchestra.run.smithi077.stdout:55834574853 2024-03-06T07:22:07.895 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:07.896 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:07.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.094+0000 7f98d18af640 1 Processor -- start 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.094+0000 7f98d18af640 1 -- start start 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.095+0000 7f98d18af640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f98cc165190 0x7f98cc071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.095+0000 7f98d18af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.095+0000 7f98d18af640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc16b3a0 con 0x7f98cc0cece0 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.095+0000 7f98d18af640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc06b950 con 0x7f98cc165190 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.095+0000 7f98d18af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc16bfb0 con 0x7f98cc06afb0 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.096+0000 7f98cb7fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f98cc0cece0 0x7f98cc0cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58836/0 (socket says 172.21.15.77:58836) 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.096+0000 7f98cb7fe640 1 -- 172.21.15.77:0/1225690720 learned_addr learned my addr 172.21.15.77:0/1225690720 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:08.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.096+0000 7f98c3fff640 1 --2- 172.21.15.77:0/1225690720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.096+0000 7f98caffd640 1 --2- 172.21.15.77:0/1225690720 >> v2:172.21.15.158:3300/0 conn(0x7f98cc165190 0x7f98cc071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.096+0000 7f98c3fff640 1 -- 172.21.15.77:0/1225690720 >> v2:172.21.15.158:3300/0 conn(0x7f98cc165190 msgr2=0x7f98cc071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.096+0000 7f98c3fff640 1 --2- 172.21.15.77:0/1225690720 >> v2:172.21.15.158:3300/0 conn(0x7f98cc165190 0x7f98cc071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.096+0000 7f98c3fff640 1 -- 172.21.15.77:0/1225690720 >> v1:172.21.15.158:6789/0 conn(0x7f98cc0cece0 legacy=0x7f98cc0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.096+0000 7f98c3fff640 1 -- 172.21.15.77:0/1225690720 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc16b680 con 0x7f98cc06afb0 2024-03-06T07:22:08.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7fd0b1beb640 1 Processor -- start 2024-03-06T07:22:08.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7fabaad55640 1 Processor -- start 2024-03-06T07:22:08.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7f98c3fff640 1 --2- 172.21.15.77:0/1225690720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc06b410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f98b400dbd0 tx=0x7f98b402b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eca541f4f8acd9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7fabaad55640 1 -- start start 2024-03-06T07:22:08.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7fd0b1beb640 1 -- start start 2024-03-06T07:22:08.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7f64c2489640 1 Processor -- start 2024-03-06T07:22:08.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7f98ca7fc640 1 -- 172.21.15.77:0/1225690720 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4039070 con 0x7f98cc06afb0 2024-03-06T07:22:08.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7f98ca7fc640 1 -- 172.21.15.77:0/1225690720 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b4031480 con 0x7f98cc06afb0 2024-03-06T07:22:08.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.097+0000 7f98ca7fc640 1 -- 172.21.15.77:0/1225690720 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4031850 con 0x7f98cc06afb0 2024-03-06T07:22:08.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7f64c2489640 1 -- start start 2024-03-06T07:22:08.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7ffba933e640 1 Processor -- start 2024-03-06T07:22:08.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7ffba933e640 1 -- start start 2024-03-06T07:22:08.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7f98d18af640 1 -- 172.21.15.77:0/1225690720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 msgr2=0x7f98cc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7f98d18af640 1 --2- 172.21.15.77:0/1225690720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc06b410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f98b400dbd0 tx=0x7f98b402b720 comp rx=0 tx=0).stop 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7fabaad55640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7fabaad55640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faba406ce30 0x7faba406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7fabaad55640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba406ded0 con 0x7faba406da60 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7fabaad55640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba406d290 con 0x7faba406ce30 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7fabaad55640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40717e0 con 0x7faba415ff70 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7faba3fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faba406ce30 0x7faba406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7faba8af1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7faba3fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faba406ce30 0x7faba406d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:43604/0 (socket says 172.21.15.77:43604) 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.098+0000 7faba3fff640 1 -- 172.21.15.77:0/371817351 learned_addr learned my addr 172.21.15.77:0/371817351 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7faba3fff640 1 -- 172.21.15.77:0/371817351 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 msgr2=0x7faba406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7faba3fff640 1 --2- 172.21.15.77:0/371817351 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0b1beb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0ac165b70 0x7fd0ac162580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0abfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0ac165b70 0x7fd0ac162580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0aaffd640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd0ac0b3ba0 0x7fd0ac06cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58850/0 (socket says 172.21.15.77:58850) 2024-03-06T07:22:08.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0aaffd640 1 -- 172.21.15.77:0/2443330089 learned_addr learned my addr 172.21.15.77:0/2443330089 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:08.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0b1beb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac06c850 con 0x7fd0ac15c5b0 2024-03-06T07:22:08.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0b1beb640 1 -- 172.21.15.77:0/2443330089 --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac15c990 con 0x7fd0ac0b3ba0 2024-03-06T07:22:08.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0b1beb640 1 -- 172.21.15.77:0/2443330089 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ac162b70 con 0x7fd0ac165b70 2024-03-06T07:22:08.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f98d18af640 1 -- 172.21.15.77:0/1225690720 shutdown_connections 2024-03-06T07:22:08.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f98d18af640 1 --2- 172.21.15.77:0/1225690720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc06b410 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f98d18af640 1 --2- 172.21.15.77:0/1225690720 >> v2:172.21.15.158:3300/0 conn(0x7f98cc165190 0x7f98cc071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f98d18af640 1 -- 172.21.15.77:0/1225690720 >> 172.21.15.77:0/1225690720 conn(0x7f98cc0b1830 msgr2=0x7f98cc0b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f64c2489640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc06baa0 0x7f64bc0b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7faba3fff640 1 -- 172.21.15.77:0/371817351 >> v1:172.21.15.77:6789/0 conn(0x7faba406da60 legacy=0x7faba40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7faba3fff640 1 -- 172.21.15.77:0/371817351 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba406e170 con 0x7faba406ce30 2024-03-06T07:22:08.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f64c1487640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc06baa0 0x7f64bc0b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f64c1487640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc06baa0 0x7f64bc0b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35876/0 (socket says 172.21.15.77:35876) 2024-03-06T07:22:08.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0abfff640 1 -- 172.21.15.77:0/2443330089 >> v1:172.21.15.158:6789/0 conn(0x7fd0ac0b3ba0 legacy=0x7fd0ac06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0abfff640 1 -- 172.21.15.77:0/2443330089 >> v1:172.21.15.77:6789/0 conn(0x7fd0ac15c5b0 legacy=0x7fd0ac0b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f64c0c86640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f64bc15da90 0x7f64bc070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35630/0 (socket says 172.21.15.77:35630) 2024-03-06T07:22:08.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f98d18af640 1 -- 172.21.15.77:0/1225690720 shutdown_connections 2024-03-06T07:22:08.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7ffba933e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffba40ce9e0 0x7ffba40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba27fc640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ffba40cedc0 0x7ffba406e0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58862/0 (socket says 172.21.15.77:58862) 2024-03-06T07:22:08.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba27fc640 1 -- 172.21.15.77:0/2614865697 learned_addr learned my addr 172.21.15.77:0/2614865697 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:08.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba933e640 1 -- 172.21.15.77:0/2614865697 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba406e7b0 con 0x7ffba415dda0 2024-03-06T07:22:08.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f98d18af640 1 -- 172.21.15.77:0/1225690720 wait complete. 2024-03-06T07:22:08.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba933e640 1 -- 172.21.15.77:0/2614865697 --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba4071fd0 con 0x7ffba40cedc0 2024-03-06T07:22:08.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba2ffd640 1 --2- 172.21.15.77:0/2614865697 >> v2:172.21.15.158:3300/0 conn(0x7ffba40ce9e0 0x7ffba40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba933e640 1 -- 172.21.15.77:0/2614865697 --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba40722b0 con 0x7ffba40ce9e0 2024-03-06T07:22:08.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba1ffb640 1 -- 172.21.15.77:0/2614865697 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812469839 0 0) 0x7ffba4071fd0 con 0x7ffba40cedc0 2024-03-06T07:22:08.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7faba3fff640 1 --2- 172.21.15.77:0/371817351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406ce30 0x7faba406d520 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab98009870 tx=0x7fab9802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f9885efeea940 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba1ffb640 1 -- 172.21.15.77:0/2614865697 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb8c003520 con 0x7ffba40cedc0 2024-03-06T07:22:08.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f64c2489640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64bc06c860 0x7f64bc06ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7f64c1487640 1 -- 172.21.15.77:0/2583097861 learned_addr learned my addr 172.21.15.77:0/2583097861 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7f64c1c88640 1 --2- 172.21.15.77:0/2583097861 >> v2:172.21.15.77:3300/0 conn(0x7f64bc06c860 0x7f64bc06ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba2ffd640 1 -- 172.21.15.77:0/2614865697 >> v1:172.21.15.158:6789/0 conn(0x7ffba40cedc0 legacy=0x7ffba406e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba2ffd640 1 -- 172.21.15.77:0/2614865697 >> v1:172.21.15.77:6789/0 conn(0x7ffba415dda0 legacy=0x7ffba415e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7ffba2ffd640 1 -- 172.21.15.77:0/2614865697 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba4072590 con 0x7ffba40ce9e0 2024-03-06T07:22:08.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7f64c2489640 1 -- 172.21.15.77:0/2583097861 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc15e2e0 con 0x7f64bc15da90 2024-03-06T07:22:08.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7f64c2489640 1 -- 172.21.15.77:0/2583097861 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc15a640 con 0x7f64bc06c860 2024-03-06T07:22:08.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7f64c2489640 1 -- 172.21.15.77:0/2583097861 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc15e5c0 con 0x7f64bc06baa0 2024-03-06T07:22:08.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7faba37fe640 1 -- 172.21.15.77:0/371817351 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98035020 con 0x7faba406ce30 2024-03-06T07:22:08.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7faba37fe640 1 -- 172.21.15.77:0/371817351 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab98031640 con 0x7faba406ce30 2024-03-06T07:22:08.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.100+0000 7faba37fe640 1 -- 172.21.15.77:0/371817351 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9802d040 con 0x7faba406ce30 2024-03-06T07:22:08.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.099+0000 7fd0abfff640 1 -- 172.21.15.77:0/2443330089 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac162e50 con 0x7fd0ac165b70 2024-03-06T07:22:08.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fd0abfff640 1 --2- 172.21.15.77:0/2443330089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 0x7fd0ac162580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b3230 tx=0x7fd0a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd586c96fb4d032 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fd0aa7fc640 1 -- 172.21.15.77:0/2443330089 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00315a0 con 0x7fd0ac165b70 2024-03-06T07:22:08.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fd0aa7fc640 1 -- 172.21.15.77:0/2443330089 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a0037070 con 0x7fd0ac165b70 2024-03-06T07:22:08.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fd0aa7fc640 1 -- 172.21.15.77:0/2443330089 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00319a0 con 0x7fd0ac165b70 2024-03-06T07:22:08.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7ffba2ffd640 1 --2- 172.21.15.77:0/2614865697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffba40ce9e0 0x7ffba40b2950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c009d90 tx=0x7ffb9c02c620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=628ad48606e3ae0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fc6565ac640 1 Processor -- start 2024-03-06T07:22:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fc6565ac640 1 -- start start 2024-03-06T07:22:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7f64c1487640 1 -- 172.21.15.77:0/2583097861 >> v2:172.21.15.77:3300/0 conn(0x7f64bc06c860 msgr2=0x7f64bc06ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7f64c1487640 1 --2- 172.21.15.77:0/2583097861 >> v2:172.21.15.77:3300/0 conn(0x7f64bc06c860 0x7f64bc06ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7f64c1487640 1 -- 172.21.15.77:0/2583097861 >> v1:172.21.15.77:6789/0 conn(0x7f64bc15da90 legacy=0x7f64bc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7f64c1487640 1 -- 172.21.15.77:0/2583097861 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc15e8a0 con 0x7f64bc06baa0 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fabaad55640 1 -- 172.21.15.77:0/371817351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406ce30 msgr2=0x7faba406d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fabaad55640 1 --2- 172.21.15.77:0/371817351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406ce30 0x7faba406d520 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab98009870 tx=0x7fab9802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fabaad55640 1 -- 172.21.15.77:0/371817351 shutdown_connections 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fabaad55640 1 --2- 172.21.15.77:0/371817351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406ce30 0x7faba406d520 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.101+0000 7fabaad55640 1 --2- 172.21.15.77:0/371817351 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fabaad55640 1 -- 172.21.15.77:0/371817351 >> 172.21.15.77:0/371817351 conn(0x7faba40b1c80 msgr2=0x7faba40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fabaad55640 1 -- 172.21.15.77:0/371817351 shutdown_connections 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7ffba1ffb640 1 -- 172.21.15.77:0/2614865697 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c007740 con 0x7ffba40ce9e0 2024-03-06T07:22:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fabaad55640 1 -- 172.21.15.77:0/371817351 wait complete. 2024-03-06T07:22:08.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7ffba1ffb640 1 -- 172.21.15.77:0/2614865697 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9c0078e0 con 0x7ffba40ce9e0 2024-03-06T07:22:08.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7ffba1ffb640 1 -- 172.21.15.77:0/2614865697 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c007cb0 con 0x7ffba40ce9e0 2024-03-06T07:22:08.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7f98d18af640 1 Processor -- start 2024-03-06T07:22:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fd0b1beb640 1 -- 172.21.15.77:0/2443330089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 msgr2=0x7fd0ac162580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/2443330089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 0x7fd0ac162580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b3230 tx=0x7fd0a002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fd0b1beb640 1 -- 172.21.15.77:0/2443330089 shutdown_connections 2024-03-06T07:22:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/2443330089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 0x7fd0ac162580 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fd0b1beb640 1 -- 172.21.15.77:0/2443330089 >> 172.21.15.77:0/2443330089 conn(0x7fd0ac0b1990 msgr2=0x7fd0ac0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fd0b1beb640 1 -- 172.21.15.77:0/2443330089 shutdown_connections 2024-03-06T07:22:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fd0b1beb640 1 -- 172.21.15.77:0/2443330089 wait complete. 2024-03-06T07:22:08.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fc6565ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6480501a0 0x7fc648038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7f64c1487640 1 --2- 172.21.15.77:0/2583097861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc06baa0 0x7f64bc0b44f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009870 tx=0x7f64b802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aee78c3d0bb91ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fc6565ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc648053c70 0x7fc648006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7fc6555aa640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc6480df220 0x7fc64804fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58864/0 (socket says 172.21.15.77:58864) 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7fc6555aa640 1 -- 172.21.15.77:0/2293545338 learned_addr learned my addr 172.21.15.77:0/2293545338 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7fc654da9640 1 --2- 172.21.15.77:0/2293545338 >> v2:172.21.15.77:3300/0 conn(0x7fc6480501a0 0x7fc648038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7fc655dab640 1 --2- 172.21.15.77:0/2293545338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc648053c70 0x7fc648006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7fc6565ac640 1 -- 172.21.15.77:0/2293545338 --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6480544c0 con 0x7fc6480df220 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7fc6565ac640 1 -- 172.21.15.77:0/2293545338 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6480541e0 con 0x7fc6480501a0 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7fc647fff640 1 -- 172.21.15.77:0/2293545338 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476998283 0 0) 0x7fc6480544c0 con 0x7fc6480df220 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7fc6565ac640 1 -- 172.21.15.77:0/2293545338 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc648005f30 con 0x7fc648053c70 2024-03-06T07:22:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7fc647fff640 1 -- 172.21.15.77:0/2293545338 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c003520 con 0x7fc6480df220 2024-03-06T07:22:08.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7ffba933e640 1 -- 172.21.15.77:0/2614865697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffba40ce9e0 msgr2=0x7ffba40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7ffba933e640 1 --2- 172.21.15.77:0/2614865697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffba40ce9e0 0x7ffba40b2950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c009d90 tx=0x7ffb9c02c620 comp rx=0 tx=0).stop 2024-03-06T07:22:08.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7f64b3fff640 1 -- 172.21.15.77:0/2583097861 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8034070 con 0x7f64bc06baa0 2024-03-06T07:22:08.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7f64b3fff640 1 -- 172.21.15.77:0/2583097861 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b8030840 con 0x7f64bc06baa0 2024-03-06T07:22:08.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7f64b3fff640 1 -- 172.21.15.77:0/2583097861 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8030c10 con 0x7f64bc06baa0 2024-03-06T07:22:08.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7f98d18af640 1 -- start start 2024-03-06T07:22:08.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7f98d18af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc08b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7fabaad55640 1 Processor -- start 2024-03-06T07:22:08.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7f98d18af640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cece0 0x7f98cc167fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.103+0000 7f98d18af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98cc165190 0x7f98cc169330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7f64c2489640 1 -- 172.21.15.77:0/2583097861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc06baa0 msgr2=0x7f64bc0b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7f64c2489640 1 --2- 172.21.15.77:0/2583097861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc06baa0 0x7f64bc0b44f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009870 tx=0x7f64b802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:08.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7fabaad55640 1 -- start start 2024-03-06T07:22:08.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7ff322455640 1 Processor -- start 2024-03-06T07:22:08.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.102+0000 7ff322455640 1 -- start start 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ff322455640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff3140df210 0x7ff314038070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ff322455640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff314050190 0x7ff314006a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ff322455640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3140544b0 con 0x7ff314053c60 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ff322455640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3140541d0 con 0x7ff3140df210 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ff322455640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff314005f20 con 0x7ff314050190 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7f64c2489640 1 -- 172.21.15.77:0/2583097861 shutdown_connections 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7f64c2489640 1 --2- 172.21.15.77:0/2583097861 >> v2:172.21.15.77:3300/0 conn(0x7f64bc06c860 0x7f64bc06ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7f64c2489640 1 --2- 172.21.15.77:0/2583097861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc06baa0 0x7f64bc0b44f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7f64c2489640 1 -- 172.21.15.77:0/2583097861 >> 172.21.15.77:0/2583097861 conn(0x7f64bc06afb0 msgr2=0x7f64bc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7fc654da9640 1 -- 172.21.15.77:0/2293545338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc648053c70 msgr2=0x7fc648006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7fc654da9640 1 --2- 172.21.15.77:0/2293545338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc648053c70 0x7fc648006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7fc654da9640 1 -- 172.21.15.77:0/2293545338 >> v1:172.21.15.158:6789/0 conn(0x7fc6480df220 legacy=0x7fc64804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7fd0b1beb640 1 Processor -- start 2024-03-06T07:22:08.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7fc654da9640 1 -- 172.21.15.77:0/2293545338 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc648006210 con 0x7fc6480501a0 2024-03-06T07:22:08.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7f64c2489640 1 -- 172.21.15.77:0/2583097861 shutdown_connections 2024-03-06T07:22:08.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ff3209f2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff314050190 0x7ff314006a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ff3209f2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff314050190 0x7ff314006a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:43618/0 (socket says 172.21.15.77:43618) 2024-03-06T07:22:08.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ff3209f2640 1 -- 172.21.15.77:0/1506876862 learned_addr learned my addr 172.21.15.77:0/1506876862 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7f64c2489640 1 -- 172.21.15.77:0/2583097861 wait complete. 2024-03-06T07:22:08.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ffba933e640 1 -- 172.21.15.77:0/2614865697 shutdown_connections 2024-03-06T07:22:08.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ffba933e640 1 --2- 172.21.15.77:0/2614865697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffba40ce9e0 0x7ffba40b2950 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7fd0b1beb640 1 -- start start 2024-03-06T07:22:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.104+0000 7ffba933e640 1 -- 172.21.15.77:0/2614865697 >> 172.21.15.77:0/2614865697 conn(0x7ffba40b1c80 msgr2=0x7ffba40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7ff3209f2640 1 -- 172.21.15.77:0/1506876862 >> v2:172.21.15.77:3300/0 conn(0x7ff3140df210 msgr2=0x7ff314038070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7ff3209f2640 1 --2- 172.21.15.77:0/1506876862 >> v2:172.21.15.77:3300/0 conn(0x7ff3140df210 0x7ff314038070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7ff3209f2640 1 -- 172.21.15.77:0/1506876862 >> v1:172.21.15.158:6789/0 conn(0x7ff314053c60 legacy=0x7ff31404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7ff3209f2640 1 -- 172.21.15.77:0/1506876862 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff314006200 con 0x7ff314050190 2024-03-06T07:22:08.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7f98c3fff640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cece0 0x7f98cc167fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7ffba933e640 1 -- 172.21.15.77:0/2614865697 shutdown_connections 2024-03-06T07:22:08.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7ffba933e640 1 -- 172.21.15.77:0/2614865697 wait complete. 2024-03-06T07:22:08.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7f98d18af640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc165570 con 0x7f98cc165190 2024-03-06T07:22:08.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7f98caffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc08b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7f98caffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc08b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35894/0 (socket says 172.21.15.77:35894) 2024-03-06T07:22:08.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7f98caffd640 1 -- 172.21.15.77:0/1854202513 learned_addr learned my addr 172.21.15.77:0/1854202513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7f98cb7fe640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98cc165190 0x7f98cc169330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7f98d18af640 1 -- 172.21.15.77:0/1854202513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc16bc40 con 0x7f98cc0cece0 2024-03-06T07:22:08.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7f98d18af640 1 -- 172.21.15.77:0/1854202513 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc16e880 con 0x7f98cc06afb0 2024-03-06T07:22:08.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7fabaad55640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba406ce30 0x7faba4157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.105+0000 7fabaad55640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406da60 0x7faba41fb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fabaad55640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba41f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fc654da9640 1 --2- 172.21.15.77:0/2293545338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 0x7fc648038080 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc6400097a0 tx=0x7fc64002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2867a1c4b5e95b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fabaad55640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba41665b0 con 0x7faba406ce30 2024-03-06T07:22:08.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fabaad55640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba41662d0 con 0x7faba406da60 2024-03-06T07:22:08.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fabaad55640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba4160350 con 0x7faba415ff70 2024-03-06T07:22:08.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba8af1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba406ce30 0x7faba4157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba8af1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba406ce30 0x7faba4157e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41514/0 (socket says 172.21.15.77:41514) 2024-03-06T07:22:08.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba8af1640 1 -- 172.21.15.77:0/1597739729 learned_addr learned my addr 172.21.15.77:0/1597739729 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba3fff640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406da60 0x7faba41fb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fd0b1beb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0ac0b3ba0 0x7fd0ac144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98c3fff640 1 -- 172.21.15.77:0/1854202513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 msgr2=0x7f98cc08b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98c3fff640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc08b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98c3fff640 1 -- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98cc165190 msgr2=0x7f98cc169330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98c3fff640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98cc165190 0x7f98cc169330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98c3fff640 1 -- 172.21.15.77:0/1854202513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc08b570 con 0x7f98cc0cece0 2024-03-06T07:22:08.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba92f2640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba41f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fd0b1beb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0ac15c5b0 0x7fd0ac1f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fc647fff640 1 -- 172.21.15.77:0/2293545338 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400315a0 con 0x7fc6480501a0 2024-03-06T07:22:08.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fc647fff640 1 -- 172.21.15.77:0/2293545338 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc640037070 con 0x7fc6480501a0 2024-03-06T07:22:08.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fc647fff640 1 -- 172.21.15.77:0/2293545338 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640031960 con 0x7fc6480501a0 2024-03-06T07:22:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98caffd640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc08b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98cb7fe640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98cc165190 0x7f98cc169330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98c3fff640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cece0 0x7f98cc167fb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f98b4033000 tx=0x7f98b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f64c2489640 1 Processor -- start 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fd0ab7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0ac0b3ba0 0x7fd0ac144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fd0ab7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0ac0b3ba0 0x7fd0ac144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35914/0 (socket says 172.21.15.77:35914) 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fd0ab7fe640 1 -- 172.21.15.77:0/1323823967 learned_addr learned my addr 172.21.15.77:0/1323823967 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba92f2640 1 -- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406da60 msgr2=0x7faba41fb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba92f2640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406da60 0x7faba41fb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba92f2640 1 -- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba406ce30 msgr2=0x7faba4157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba92f2640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba406ce30 0x7faba4157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba92f2640 1 -- 172.21.15.77:0/1597739729 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba41fb8e0 con 0x7faba415ff70 2024-03-06T07:22:08.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba8af1640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba406ce30 0x7faba4157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7faba3fff640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406da60 0x7faba41fb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 0x7fd0ac15a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7ff3209f2640 1 --2- 172.21.15.77:0/1506876862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff314050190 0x7ff314006a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff31c04fec0 tx=0x7ff31c0a8fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=421419a4e2efe16d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f64c2489640 1 -- start start 2024-03-06T07:22:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98c8ff9640 1 -- 172.21.15.77:0/1854202513 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4043030 con 0x7f98cc0cece0 2024-03-06T07:22:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98d18af640 1 -- 172.21.15.77:0/1854202513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc1684f0 con 0x7f98cc0cece0 2024-03-06T07:22:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.106+0000 7f98d18af640 1 -- 172.21.15.77:0/1854202513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc16d810 con 0x7f98cc0cece0 2024-03-06T07:22:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac0710b0 con 0x7fd0ac165b70 2024-03-06T07:22:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac163410 con 0x7fd0ac15c5b0 2024-03-06T07:22:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac06d440 con 0x7fd0ac0b3ba0 2024-03-06T07:22:08.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fd0abfff640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 0x7fd0ac15a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fd0abfff640 1 -- 172.21.15.77:0/1323823967 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0ac0b3ba0 msgr2=0x7fd0ac144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fd0abfff640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0ac0b3ba0 0x7fd0ac144960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fd0abfff640 1 -- 172.21.15.77:0/1323823967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0ac15c5b0 msgr2=0x7fd0ac1f7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fd0abfff640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0ac15c5b0 0x7fd0ac1f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fd0abfff640 1 -- 172.21.15.77:0/1323823967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac1f87a0 con 0x7fd0ac165b70 2024-03-06T07:22:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7ffba933e640 1 Processor -- start 2024-03-06T07:22:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7ffba933e640 1 -- start start 2024-03-06T07:22:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fc6565ac640 1 -- 172.21.15.77:0/2293545338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 msgr2=0x7fc648038080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fc6565ac640 1 --2- 172.21.15.77:0/2293545338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 0x7fc648038080 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc6400097a0 tx=0x7fc64002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fc6565ac640 1 -- 172.21.15.77:0/2293545338 shutdown_connections 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fc6565ac640 1 --2- 172.21.15.77:0/2293545338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc648053c70 0x7fc648006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fc6565ac640 1 --2- 172.21.15.77:0/2293545338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 0x7fc648038080 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7fc6565ac640 1 -- 172.21.15.77:0/2293545338 >> 172.21.15.77:0/2293545338 conn(0x7fc648009fb0 msgr2=0x7fc64800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fc6565ac640 1 -- 172.21.15.77:0/2293545338 shutdown_connections 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fc6565ac640 1 -- 172.21.15.77:0/2293545338 wait complete. 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7ff31affd640 1 -- 172.21.15.77:0/1506876862 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0af070 con 0x7ff314050190 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7ff31affd640 1 -- 172.21.15.77:0/1506876862 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c0a9e40 con 0x7ff314050190 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ff322455640 1 -- 172.21.15.77:0/1506876862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff314050190 msgr2=0x7ff314006a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ff322455640 1 --2- 172.21.15.77:0/1506876862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff314050190 0x7ff314006a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff31c04fec0 tx=0x7ff31c0a8fd0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7f98c8ff9640 1 -- 172.21.15.77:0/1854202513 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b4031070 con 0x7f98cc0cece0 2024-03-06T07:22:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7f98c8ff9640 1 -- 172.21.15.77:0/1854202513 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4039040 con 0x7f98cc0cece0 2024-03-06T07:22:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.107+0000 7faba92f2640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba41f7680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab9400c390 tx=0x7fab9400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ff322455640 1 -- 172.21.15.77:0/1506876862 shutdown_connections 2024-03-06T07:22:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ff322455640 1 --2- 172.21.15.77:0/1506876862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff314050190 0x7ff314006a00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fd0ab7fe640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0ac0b3ba0 0x7fd0ac144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f98d18af640 1 -- 172.21.15.77:0/1854202513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b0005290 con 0x7f98cc0cece0 2024-03-06T07:22:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ff322455640 1 --2- 172.21.15.77:0/1506876862 >> v2:172.21.15.77:3300/0 conn(0x7ff3140df210 0x7ff314038070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ff322455640 1 -- 172.21.15.77:0/1506876862 >> 172.21.15.77:0/1506876862 conn(0x7ff314009fa0 msgr2=0x7ff31400a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7faba17fa640 1 -- 172.21.15.77:0/1597739729 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94017070 con 0x7faba415ff70 2024-03-06T07:22:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7faba17fa640 1 -- 172.21.15.77:0/1597739729 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab940052d0 con 0x7faba415ff70 2024-03-06T07:22:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7faba17fa640 1 -- 172.21.15.77:0/1597739729 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94003850 con 0x7faba415ff70 2024-03-06T07:22:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba41fbb60 con 0x7faba415ff70 2024-03-06T07:22:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fd0abfff640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 0x7fd0ac15a6b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0033000 tx=0x7fd0a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba41fc120 con 0x7faba415ff70 2024-03-06T07:22:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ffba933e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffba40ce9e0 0x7ffba4136470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ffba933e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffba40cedc0 0x7ffba415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ffba933e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffba415dda0 0x7ffba415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ffba933e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba4140790 con 0x7ffba40cedc0 2024-03-06T07:22:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ffba933e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba4072b50 con 0x7ffba40ce9e0 2024-03-06T07:22:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7ffba933e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba406dd60 con 0x7ffba415dda0 2024-03-06T07:22:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7ff322455640 1 -- 172.21.15.77:0/1506876862 shutdown_connections 2024-03-06T07:22:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7fd097fff640 1 -- 172.21.15.77:0/1323823967 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0032840 con 0x7fd0ac165b70 2024-03-06T07:22:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7fd097fff640 1 -- 172.21.15.77:0/1323823967 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a00329e0 con 0x7fd0ac165b70 2024-03-06T07:22:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7fd097fff640 1 -- 172.21.15.77:0/1323823967 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0032dd0 con 0x7fd0ac165b70 2024-03-06T07:22:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ac1f8a80 con 0x7fd0ac165b70 2024-03-06T07:22:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7ffba37fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffba415dda0 0x7ffba415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7ffba37fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffba415dda0 0x7ffba415ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35926/0 (socket says 172.21.15.77:35926) 2024-03-06T07:22:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7ffba37fe640 1 -- 172.21.15.77:0/1901644162 learned_addr learned my addr 172.21.15.77:0/1901644162 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c2489640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64bc06baa0 0x7f64bc0af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c2489640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64bc06c860 0x7f64bc14ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c2489640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc15da90 0x7f64bc1530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c2489640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc15ac50 con 0x7f64bc06baa0 2024-03-06T07:22:08.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab84005290 con 0x7faba415ff70 2024-03-06T07:22:08.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7ffba27fc640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffba40cedc0 0x7ffba415a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c2489640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc15aad0 con 0x7f64bc06c860 2024-03-06T07:22:08.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7fc6565ac640 1 Processor -- start 2024-03-06T07:22:08.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7fc6565ac640 1 -- start start 2024-03-06T07:22:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7ffba37fe640 1 -- 172.21.15.77:0/1901644162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffba40ce9e0 msgr2=0x7ffba4136470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7ffba37fe640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffba40ce9e0 0x7ffba4136470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7ffba37fe640 1 -- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffba40cedc0 msgr2=0x7ffba415a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7ffba37fe640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffba40cedc0 0x7ffba415a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7ffba37fe640 1 -- 172.21.15.77:0/1901644162 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba413f0e0 con 0x7ffba415dda0 2024-03-06T07:22:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7ff322455640 1 -- 172.21.15.77:0/1506876862 wait complete. 2024-03-06T07:22:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c2489640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc15add0 con 0x7f64bc15da90 2024-03-06T07:22:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7faba17fa640 1 -- 172.21.15.77:0/1597739729 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab9401a070 con 0x7faba415ff70 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c0c86640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64bc06c860 0x7f64bc14ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c0c86640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64bc06c860 0x7f64bc14ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:43642/0 (socket says 172.21.15.77:43642) 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c0c86640 1 -- 172.21.15.77:0/2647780562 learned_addr learned my addr 172.21.15.77:0/2647780562 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.108+0000 7f64c1c88640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc15da90 0x7f64bc1530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7f64c1c88640 1 -- 172.21.15.77:0/2647780562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64bc06c860 msgr2=0x7f64bc14ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7f64c1c88640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64bc06c860 0x7f64bc14ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7ffba37fe640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffba415dda0 0x7ffba415ac30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400b490 tx=0x7ffb9400b950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7f64c1c88640 1 -- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64bc06baa0 msgr2=0x7f64bc0af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ac1f8300 con 0x7fd0ac165b70 2024-03-06T07:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7faba17fa640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab9006b5a0 0x7fab9006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7faba17fa640 1 -- 172.21.15.77:0/1597739729 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab940893c0 con 0x7faba415ff70 2024-03-06T07:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7faba8af1640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab9006b5a0 0x7fab9006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7fc6565ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 0x7fc64817cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.110+0000 7f98c8ff9640 1 -- 172.21.15.77:0/1854202513 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98b4046070 con 0x7f98cc0cece0 2024-03-06T07:22:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f9f60d3e640 1 Processor -- start 2024-03-06T07:22:08.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f98c8ff9640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f98ac06b5a0 0x7f98ac06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f9f60d3e640 1 -- start start 2024-03-06T07:22:08.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f98c8ff9640 1 -- 172.21.15.77:0/1854202513 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98b40ac170 con 0x7f98cc0cece0 2024-03-06T07:22:08.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f98caffd640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f98ac06b5a0 0x7f98ac06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7ff322455640 1 Processor -- start 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7f64c1c88640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64bc06baa0 0x7f64bc0af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7f64c1c88640 1 -- 172.21.15.77:0/2647780562 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc14b390 con 0x7f64bc15da90 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.109+0000 7f64c1c88640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc15da90 0x7f64bc1530e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f64b400efc0 tx=0x7f64b400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2647780562 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4018020 con 0x7f64bc15da90 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc0cefb0 con 0x7f64bc15da90 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc0b02b0 con 0x7f64bc15da90 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2647780562 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b4014480 con 0x7f64bc15da90 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2647780562 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4014870 con 0x7f64bc15da90 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc6565ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc648053c70 0x7fc64804bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc6565ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6480df220 0x7fc64804c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc6565ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc648037a90 con 0x7fc6480501a0 2024-03-06T07:22:08.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc6565ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc648037910 con 0x7fc648053c70 2024-03-06T07:22:08.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc6565ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6480df600 con 0x7fc6480df220 2024-03-06T07:22:08.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7ff322455640 1 -- start start 2024-03-06T07:22:08.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7f98caffd640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f98ac06b5a0 0x7f98ac06da60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0ce750 tx=0x7f98bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.111+0000 7fd097fff640 1 -- 172.21.15.77:0/1323823967 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0a0046450 con 0x7fd0ac165b70 2024-03-06T07:22:08.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7faba8af1640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab9006b5a0 0x7fab9006da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0094c0 tx=0x7fab9c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fd097fff640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd09006b5a0 0x7fd09006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc655dab640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6480df220 0x7fc64804c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc655dab640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6480df220 0x7fc64804c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35934/0 (socket says 172.21.15.77:35934) 2024-03-06T07:22:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc655dab640 1 -- 172.21.15.77:0/1619903948 learned_addr learned my addr 172.21.15.77:0/1619903948 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc654da9640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc648053c70 0x7fc64804bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ac068f10 con 0x7fd0ac165b70 2024-03-06T07:22:08.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fd097fff640 1 -- 172.21.15.77:0/1323823967 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0a00ac480 con 0x7fd0ac165b70 2024-03-06T07:22:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f60d3e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f5c06eb60 0x7f9f5c0b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f60d3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c14b920 0x7f9f5c0b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7fd0ab7fe640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd09006b5a0 0x7fd09006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc6555aa640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 0x7fc64817cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc655dab640 1 -- 172.21.15.77:0/1619903948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc648053c70 msgr2=0x7fc64804bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc655dab640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc648053c70 0x7fc64804bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc655dab640 1 -- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 msgr2=0x7fc64817cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc655dab640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 0x7fc64817cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.112+0000 7fc655dab640 1 -- 172.21.15.77:0/1619903948 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc648049110 con 0x7fc6480df220 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7fc6555aa640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 0x7fc64817cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7fc655dab640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6480df220 0x7fc64804c480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc6500645f0 tx=0x7fc6500691a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff322455640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff314053c60 0x7ff314180ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff322455640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3140df210 0x7ff3140e20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff322455640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3140d15f0 0x7ff3140d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff322455640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff314037a80 con 0x7ff3140df210 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff322455640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff314037900 con 0x7ff3140d15f0 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff322455640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140df5f0 con 0x7ff314053c60 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f60d3e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c070d50 con 0x7f9f5c06e780 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f60d3e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5c1494c0 con 0x7f9f5c06eb60 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f60d3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c071030 con 0x7f9f5c14b920 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f5ad9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c14b920 0x7f9f5c0b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f5a59c640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9f5c06e780 0x7f9f5c148db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58884/0 (socket says 172.21.15.77:58884) 2024-03-06T07:22:08.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f5a59c640 1 -- 172.21.15.77:0/1901822738 learned_addr learned my addr 172.21.15.77:0/1901822738 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:08.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c005290 con 0x7f64bc15da90 2024-03-06T07:22:08.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff3209f2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3140d15f0 0x7ff3140d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff3209f2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3140d15f0 0x7ff3140d1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:43666/0 (socket says 172.21.15.77:43666) 2024-03-06T07:22:08.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff3209f2640 1 -- 172.21.15.77:0/2665620155 learned_addr learned my addr 172.21.15.77:0/2665620155 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff31bfff640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff314053c60 0x7ff314180ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff31b7fe640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3140df210 0x7ff3140e20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f59d9b640 1 --2- 172.21.15.77:0/1901822738 >> v2:172.21.15.77:3300/0 conn(0x7f9f5c06eb60 0x7f9f5c0b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7fd0ab7fe640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd09006b5a0 0x7fd09006da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b1540 tx=0x7fd098009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f59d9b640 1 -- 172.21.15.77:0/1901822738 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c14b920 msgr2=0x7f9f5c0b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f59d9b640 1 --2- 172.21.15.77:0/1901822738 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c14b920 0x7f9f5c0b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f59d9b640 1 -- 172.21.15.77:0/1901822738 >> v1:172.21.15.158:6789/0 conn(0x7f9f5c06e780 legacy=0x7f9f5c148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7f9f59d9b640 1 -- 172.21.15.77:0/1901822738 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c071310 con 0x7f9f5c06eb60 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff3209f2640 1 -- 172.21.15.77:0/2665620155 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff314053c60 msgr2=0x7ff314180ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff3209f2640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff314053c60 0x7ff314180ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff3209f2640 1 -- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3140df210 msgr2=0x7ff3140e20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff3209f2640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3140df210 0x7ff3140e20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.113+0000 7ff3209f2640 1 -- 172.21.15.77:0/2665620155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff314181400 con 0x7ff3140d15f0 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.114+0000 7ff3209f2640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3140d15f0 0x7ff3140d1aa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff31c069700 tx=0x7ff31c069a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.114+0000 7ff318ff9640 1 -- 172.21.15.77:0/2665620155 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c070360 con 0x7ff3140d15f0 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.114+0000 7ff322455640 1 -- 172.21.15.77:0/2665620155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31404e030 con 0x7ff3140d15f0 2024-03-06T07:22:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.114+0000 7ff322455640 1 -- 172.21.15.77:0/2665620155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31404aa60 con 0x7ff3140d15f0 2024-03-06T07:22:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.114+0000 7f98c8ff9640 1 -- 172.21.15.77:0/1854202513 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98b404d070 con 0x7f98cc0cece0 2024-03-06T07:22:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.115+0000 7ff318ff9640 1 -- 172.21.15.77:0/2665620155 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c070500 con 0x7ff3140d15f0 2024-03-06T07:22:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.115+0000 7ff318ff9640 1 -- 172.21.15.77:0/2665620155 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0708b0 con 0x7ff3140d15f0 2024-03-06T07:22:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.115+0000 7f9f59d9b640 1 --2- 172.21.15.77:0/1901822738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c06eb60 0x7f9f5c0b4060 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9f500097a0 tx=0x7f9f5002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3a7bb2e035c0f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.115+0000 7ffb8b7fe640 1 -- 172.21.15.77:0/1901644162 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94015020 con 0x7ffba415dda0 2024-03-06T07:22:08.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.115+0000 7ff318ff9640 1 -- 172.21.15.77:0/2665620155 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff31c0b2490 con 0x7ff3140d15f0 2024-03-06T07:22:08.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.115+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba413f3c0 con 0x7ffba415dda0 2024-03-06T07:22:08.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba413c960 con 0x7ffba415dda0 2024-03-06T07:22:08.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7ff3067fc640 1 -- 172.21.15.77:0/2665620155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f4005290 con 0x7ff3140d15f0 2024-03-06T07:22:08.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7faba17fa640 1 -- 172.21.15.77:0/1597739729 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faba41fb8e0 con 0x7faba415ff70 2024-03-06T07:22:08.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7ffb8b7fe640 1 -- 172.21.15.77:0/1901644162 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94011850 con 0x7ffba415dda0 2024-03-06T07:22:08.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7ffb8b7fe640 1 -- 172.21.15.77:0/1901644162 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94011c40 con 0x7ffba415dda0 2024-03-06T07:22:08.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7fc645ffb640 1 -- 172.21.15.77:0/1619903948 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6500735e0 con 0x7fc6480df220 2024-03-06T07:22:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7fc645ffb640 1 -- 172.21.15.77:0/1619903948 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65008f070 con 0x7fc6480df220 2024-03-06T07:22:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7fc645ffb640 1 -- 172.21.15.77:0/1619903948 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc650094690 con 0x7fc6480df220 2024-03-06T07:22:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6480493f0 con 0x7fc6480df220 2024-03-06T07:22:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7ff318ff9640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff30006b4d0 0x7ff30006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7ff318ff9640 1 -- 172.21.15.77:0/2665620155 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff31c0c2070 con 0x7ff3140d15f0 2024-03-06T07:22:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.116+0000 7ff31bfff640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff30006b4d0 0x7ff30006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.117+0000 7ff31bfff640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff30006b4d0 0x7ff30006d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff310009990 tx=0x7ff310009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.117+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6480499b0 con 0x7fc6480df220 2024-03-06T07:22:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.117+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb84005290 con 0x7ffba415dda0 2024-03-06T07:22:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.117+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2647780562 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64b4005d10 con 0x7f64bc15da90 2024-03-06T07:22:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7f64b1ffb640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64a006b550 0x7f64a006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc648005080 con 0x7fc6480df220 2024-03-06T07:22:08.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7f64c1487640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64a006b550 0x7f64a006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2647780562 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64b408b4a0 con 0x7f64bc15da90 2024-03-06T07:22:08.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7f9f5959a640 1 -- 172.21.15.77:0/1901822738 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50035020 con 0x7f9f5c06eb60 2024-03-06T07:22:08.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7f9f5959a640 1 -- 172.21.15.77:0/1901822738 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f50031810 con 0x7f9f5c06eb60 2024-03-06T07:22:08.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7f9f5959a640 1 -- 172.21.15.77:0/1901822738 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50031be0 con 0x7f9f5c06eb60 2024-03-06T07:22:08.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7f64c1487640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64a006b550 0x7f64a006da10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f64b8036000 tx=0x7f64b803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.118+0000 7fd097fff640 1 -- 172.21.15.77:0/1323823967 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0a007b370 con 0x7fd0ac165b70 2024-03-06T07:22:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.119+0000 7f9f60d3e640 1 -- 172.21.15.77:0/1901822738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c06eb60 msgr2=0x7f9f5c0b4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.119+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/1901822738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c06eb60 0x7f9f5c0b4060 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9f500097a0 tx=0x7f9f5002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.120+0000 7f9f60d3e640 1 -- 172.21.15.77:0/1901822738 shutdown_connections 2024-03-06T07:22:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.120+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/1901822738 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c14b920 0x7f9f5c0b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.120+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/1901822738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c06eb60 0x7f9f5c0b4060 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.120+0000 7f9f60d3e640 1 -- 172.21.15.77:0/1901822738 >> 172.21.15.77:0/1901822738 conn(0x7f9f5c0b1c50 msgr2=0x7f9f5c0b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.120+0000 7f9f60d3e640 1 -- 172.21.15.77:0/1901822738 shutdown_connections 2024-03-06T07:22:08.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.120+0000 7f9f60d3e640 1 -- 172.21.15.77:0/1901822738 wait complete. 2024-03-06T07:22:08.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.120+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2647780562 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64b4022030 con 0x7f64bc15da90 2024-03-06T07:22:08.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.120+0000 7ff318ff9640 1 -- 172.21.15.77:0/2665620155 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff314181400 con 0x7ff3140d15f0 2024-03-06T07:22:08.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.122+0000 7ffb8b7fe640 1 -- 172.21.15.77:0/1901644162 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb94018020 con 0x7ffba415dda0 2024-03-06T07:22:08.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.122+0000 7f9f60d3e640 1 Processor -- start 2024-03-06T07:22:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.122+0000 7fc645ffb640 1 -- 172.21.15.77:0/1619903948 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc65009e3f0 con 0x7fc6480df220 2024-03-06T07:22:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.122+0000 7ffb8b7fe640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb9006b4d0 0x7ffb9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.122+0000 7fc645ffb640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc63406b4d0 0x7fc63406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.122+0000 7f9f60d3e640 1 -- start start 2024-03-06T07:22:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.123+0000 7ffb8b7fe640 1 -- 172.21.15.77:0/1901644162 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb9408a6f0 con 0x7ffba415dda0 2024-03-06T07:22:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.123+0000 7fc645ffb640 1 -- 172.21.15.77:0/1619903948 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6501053c0 con 0x7fc6480df220 2024-03-06T07:22:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.123+0000 7ffba2ffd640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb9006b4d0 0x7ffb9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.123+0000 7fc6555aa640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc63406b4d0 0x7fc63406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7ffba2ffd640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb9006b4d0 0x7ffb9006d990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c036000 tx=0x7ffb9c031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7fc6555aa640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc63406b4d0 0x7fc63406d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc64c007fd0 tx=0x7fc64c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f60d3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c06e780 0x7f9f5c163160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f60d3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c06eb60 0x7f9f5c1f8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f60d3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c14b920 0x7f9f5c1f8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f60d3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c071bb0 con 0x7f9f5c14b920 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f60d3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c0718d0 con 0x7f9f5c06e780 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f60d3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c14b3d0 con 0x7f9f5c06eb60 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c14b920 0x7f9f5c1f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5a59c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c06e780 0x7f9f5c163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f59d9b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c06eb60 0x7f9f5c1f8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c14b920 0x7f9f5c1f8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41564/0 (socket says 172.21.15.77:41564) 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5ad9d640 1 -- 172.21.15.77:0/2465906173 learned_addr learned my addr 172.21.15.77:0/2465906173 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5a59c640 1 -- 172.21.15.77:0/2465906173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c06eb60 msgr2=0x7f9f5c1f8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5a59c640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c06eb60 0x7f9f5c1f8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5a59c640 1 -- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c14b920 msgr2=0x7f9f5c1f8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5a59c640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c14b920 0x7f9f5c1f8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.124+0000 7f9f5a59c640 1 -- 172.21.15.77:0/2465906173 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c163b40 con 0x7f9f5c06e780 2024-03-06T07:22:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.125+0000 7f9f5a59c640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c06e780 0x7f9f5c163160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00c440 tx=0x7f9f4c001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.125+0000 7f9f4affd640 1 -- 172.21.15.77:0/2465906173 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c017070 con 0x7f9f5c06e780 2024-03-06T07:22:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.125+0000 7f9f4affd640 1 -- 172.21.15.77:0/2465906173 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c005ce0 con 0x7f9f5c06e780 2024-03-06T07:22:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.125+0000 7f9f4affd640 1 -- 172.21.15.77:0/2465906173 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c0072d0 con 0x7f9f5c06e780 2024-03-06T07:22:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.125+0000 7ffb8b7fe640 1 -- 172.21.15.77:0/1901644162 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb9401f020 con 0x7ffba415dda0 2024-03-06T07:22:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.125+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c163e20 con 0x7f9f5c06e780 2024-03-06T07:22:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.125+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c1643e0 con 0x7f9f5c06e780 2024-03-06T07:22:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.127+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c04f330 con 0x7f9f5c06e780 2024-03-06T07:22:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.127+0000 7f9f4affd640 1 -- 172.21.15.77:0/2465906173 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f4c01a070 con 0x7f9f5c06e780 2024-03-06T07:22:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.128+0000 7f9f4affd640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f3c06b550 0x7f9f3c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.128+0000 7f9f4affd640 1 -- 172.21.15.77:0/2465906173 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f4c0894b0 con 0x7f9f5c06e780 2024-03-06T07:22:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.128+0000 7f9f59d9b640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f3c06b550 0x7f9f3c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.129+0000 7f9f59d9b640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f3c06b550 0x7f9f3c06da10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9f50036000 tx=0x7f9f50008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.129+0000 7fc645ffb640 1 -- 172.21.15.77:0/1619903948 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc648005080 con 0x7fc6480df220 2024-03-06T07:22:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.132+0000 7f9f4affd640 1 -- 172.21.15.77:0/2465906173 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f4c057cb0 con 0x7f9f5c06e780 2024-03-06T07:22:08.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.511+0000 7f98d18af640 1 -- 172.21.15.77:0/1854202513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f98b0005850 con 0x7f98cc0cece0 2024-03-06T07:22:08.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.511+0000 7f98c8ff9640 1 -- 172.21.15.77:0/1854202513 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f98cc08b570 con 0x7f98cc0cece0 2024-03-06T07:22:08.513 INFO:teuthology.orchestra.run.smithi077.stdout:55834574852 2024-03-06T07:22:08.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 -- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f98ac06b5a0 msgr2=0x7f98ac06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f98ac06b5a0 0x7f98ac06da60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0ce750 tx=0x7f98bc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 -- 172.21.15.77:0/1854202513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cece0 msgr2=0x7f98cc167fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cece0 0x7f98cc167fb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f98b4033000 tx=0x7f98b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 -- 172.21.15.77:0/1854202513 shutdown_connections 2024-03-06T07:22:08.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f98ac06b5a0 0x7f98ac06da60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98cc165190 0x7f98cc169330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f98cc0cece0 0x7f98cc167fb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 --2- 172.21.15.77:0/1854202513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f98cc06afb0 0x7f98cc08b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.515+0000 7f98c1ffb640 1 -- 172.21.15.77:0/1854202513 >> 172.21.15.77:0/1854202513 conn(0x7f98cc0b1830 msgr2=0x7f98cc071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.520+0000 7f98c1ffb640 1 -- 172.21.15.77:0/1854202513 shutdown_connections 2024-03-06T07:22:08.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.520+0000 7f98c1ffb640 1 -- 172.21.15.77:0/1854202513 wait complete. 2024-03-06T07:22:08.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.526+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd0ac1658e0 con 0x7fd0ac165b70 2024-03-06T07:22:08.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.526+0000 7fd097fff640 1 -- 172.21.15.77:0/1323823967 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd0a007ac70 con 0x7fd0ac165b70 2024-03-06T07:22:08.528 INFO:teuthology.orchestra.run.smithi077.stdout:55834574852 2024-03-06T07:22:08.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.527+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9f5c148f50 con 0x7f9f5c06e780 2024-03-06T07:22:08.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.529+0000 7f9f4affd640 1 -- 172.21.15.77:0/2465906173 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9f4c05bd50 con 0x7f9f5c06e780 2024-03-06T07:22:08.531 INFO:teuthology.orchestra.run.smithi077.stdout:55834574852 2024-03-06T07:22:08.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.530+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fab84005580 con 0x7faba415ff70 2024-03-06T07:22:08.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.530+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd09006b5a0 msgr2=0x7fd09006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.531+0000 7faba17fa640 1 -- 172.21.15.77:0/1597739729 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fab94057ca0 con 0x7faba415ff70 2024-03-06T07:22:08.533 INFO:teuthology.orchestra.run.smithi077.stdout:55834574852 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.530+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd09006b5a0 0x7fd09006da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b1540 tx=0x7fd098009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.530+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 msgr2=0x7fd0ac15a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.530+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 0x7fd0ac15a6b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0033000 tx=0x7fd0a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.531+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 shutdown_connections 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.531+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd09006b5a0 0x7fd09006da60 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0b1540 tx=0x7fd098009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.531+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0ac165b70 0x7fd0ac15a6b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.531+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0ac15c5b0 0x7fd0ac1f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.531+0000 7fd0b1beb640 1 --2- 172.21.15.77:0/1323823967 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0ac0b3ba0 0x7fd0ac144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.531+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 >> 172.21.15.77:0/1323823967 conn(0x7fd0ac0b1990 msgr2=0x7fd0ac0b3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 shutdown_connections 2024-03-06T07:22:08.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f3c06b550 msgr2=0x7f9f3c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f3c06b550 0x7f9f3c06da10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9f50036000 tx=0x7f9f50008030 comp rx=0 tx=0).stop 2024-03-06T07:22:08.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7fd0b1beb640 1 -- 172.21.15.77:0/1323823967 wait complete. 2024-03-06T07:22:08.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c06e780 msgr2=0x7f9f5c163160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c06e780 0x7f9f5c163160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00c440 tx=0x7f9f4c001e70 comp rx=0 tx=0).stop 2024-03-06T07:22:08.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 shutdown_connections 2024-03-06T07:22:08.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f3c06b550 0x7f9f3c06da10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c14b920 0x7f9f5c1f8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f5c06eb60 0x7f9f5c1f8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 --2- 172.21.15.77:0/2465906173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f5c06e780 0x7f9f5c163160 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.532+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 >> 172.21.15.77:0/2465906173 conn(0x7f9f5c0b1c50 msgr2=0x7f9f5c145010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.533+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 shutdown_connections 2024-03-06T07:22:08.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.533+0000 7f9f60d3e640 1 -- 172.21.15.77:0/2465906173 wait complete. 2024-03-06T07:22:08.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.535+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f649c005850 con 0x7f64bc15da90 2024-03-06T07:22:08.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.535+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2647780562 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f64bc14b390 con 0x7f64bc15da90 2024-03-06T07:22:08.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.0 2024-03-06T07:22:08.538 INFO:teuthology.orchestra.run.smithi077.stdout:55834574851 2024-03-06T07:22:08.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64a006b550 msgr2=0x7f64a006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64a006b550 0x7f64a006da10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f64b8036000 tx=0x7f64b803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc15da90 msgr2=0x7f64bc1530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc15da90 0x7f64bc1530e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f64b400efc0 tx=0x7f64b400c490 comp rx=0 tx=0).stop 2024-03-06T07:22:08.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 shutdown_connections 2024-03-06T07:22:08.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64bc15da90 0x7f64bc1530e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64bc06c860 0x7f64bc14ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64a006b550 0x7f64a006da10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 --2- 172.21.15.77:0/2647780562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64bc06baa0 0x7f64bc0af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.538+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 >> 172.21.15.77:0/2647780562 conn(0x7f64bc06afb0 msgr2=0x7f64bc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.539+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ffb84005850 con 0x7ffba415dda0 2024-03-06T07:22:08.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.539+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab9006b5a0 msgr2=0x7fab9006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.539+0000 7fabaad55640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab9006b5a0 0x7fab9006da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0094c0 tx=0x7fab9c00b040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.539+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 msgr2=0x7faba41f7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.539+0000 7fabaad55640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba41f7680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab9400c390 tx=0x7fab9400c850 comp rx=0 tx=0).stop 2024-03-06T07:22:08.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.539+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 shutdown_connections 2024-03-06T07:22:08.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.539+0000 7f64c2489640 1 -- 172.21.15.77:0/2647780562 wait complete. 2024-03-06T07:22:08.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.539+0000 7ffb8b7fe640 1 -- 172.21.15.77:0/1901644162 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffb9400f2e0 con 0x7ffba415dda0 2024-03-06T07:22:08.543 INFO:teuthology.orchestra.run.smithi077.stdout:55834574852 2024-03-06T07:22:08.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.540+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 shutdown_connections 2024-03-06T07:22:08.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.540+0000 7fabaad55640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba415ff70 0x7faba41f7680 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.540+0000 7fabaad55640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba406da60 0x7faba41fb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.540+0000 7fabaad55640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab9006b5a0 0x7fab9006da60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.540+0000 7fabaad55640 1 --2- 172.21.15.77:0/1597739729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba406ce30 0x7faba4157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.540+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 >> 172.21.15.77:0/1597739729 conn(0x7faba40b1c80 msgr2=0x7faba40ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.540+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 shutdown_connections 2024-03-06T07:22:08.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.540+0000 7fabaad55640 1 -- 172.21.15.77:0/1597739729 wait complete. 2024-03-06T07:22:08.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.542+0000 7ff3067fc640 1 -- 172.21.15.77:0/2665620155 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff2f4005850 con 0x7ff3140d15f0 2024-03-06T07:22:08.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.542+0000 7ff318ff9640 1 -- 172.21.15.77:0/2665620155 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff31c0f5a00 con 0x7ff3140d15f0 2024-03-06T07:22:08.545 INFO:teuthology.orchestra.run.smithi077.stdout:55834574852 2024-03-06T07:22:08.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb9006b4d0 msgr2=0x7ffb9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb9006b4d0 0x7ffb9006d990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c036000 tx=0x7ffb9c031000 comp rx=0 tx=0).stop 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffba415dda0 msgr2=0x7ffba415ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffba415dda0 0x7ffba415ac30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400b490 tx=0x7ffb9400b950 comp rx=0 tx=0).stop 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 shutdown_connections 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb9006b4d0 0x7ffb9006d990 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffba40cedc0 0x7ffba415a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffba415dda0 0x7ffba415ac30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 --2- 172.21.15.77:0/1901644162 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffba40ce9e0 0x7ffba4136470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 >> 172.21.15.77:0/1901644162 conn(0x7ffba40b1c80 msgr2=0x7ffba4136a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 shutdown_connections 2024-03-06T07:22:08.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.543+0000 7ffba933e640 1 -- 172.21.15.77:0/1901644162 wait complete. 2024-03-06T07:22:08.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.545+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc6480344c0 con 0x7fc6480df220 2024-03-06T07:22:08.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.545+0000 7fc645ffb640 1 -- 172.21.15.77:0/1619903948 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc6500a3080 con 0x7fc6480df220 2024-03-06T07:22:08.548 INFO:teuthology.orchestra.run.smithi077.stdout:55834574852 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 -- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff30006b4d0 msgr2=0x7ff30006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff30006b4d0 0x7ff30006d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff310009990 tx=0x7ff310009110 comp rx=0 tx=0).stop 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 -- 172.21.15.77:0/2665620155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3140d15f0 msgr2=0x7ff3140d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3140d15f0 0x7ff3140d1aa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff31c069700 tx=0x7ff31c069a60 comp rx=0 tx=0).stop 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 -- 172.21.15.77:0/2665620155 shutdown_connections 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff30006b4d0 0x7ff30006d990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3140df210 0x7ff3140e20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff3140d15f0 0x7ff3140d1aa0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 --2- 172.21.15.77:0/2665620155 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff314053c60 0x7ff314180ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.546+0000 7ff3067fc640 1 -- 172.21.15.77:0/2665620155 >> 172.21.15.77:0/2665620155 conn(0x7ff314009fa0 msgr2=0x7ff314034a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.547+0000 7ff3067fc640 1 -- 172.21.15.77:0/2665620155 shutdown_connections 2024-03-06T07:22:08.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.547+0000 7ff3067fc640 1 -- 172.21.15.77:0/2665620155 wait complete. 2024-03-06T07:22:08.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.549+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc63406b4d0 msgr2=0x7fc63406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.549+0000 7fc6565ac640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc63406b4d0 0x7fc63406d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc64c007fd0 tx=0x7fc64c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.549+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6480df220 msgr2=0x7fc64804c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.549+0000 7fc6565ac640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6480df220 0x7fc64804c480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc6500645f0 tx=0x7fc6500691a0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.550+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 shutdown_connections 2024-03-06T07:22:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.550+0000 7fc6565ac640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6480df220 0x7fc64804c480 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.550+0000 7fc6565ac640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc648053c70 0x7fc64804bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.550+0000 7fc6565ac640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc63406b4d0 0x7fc63406d990 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.550+0000 7fc6565ac640 1 --2- 172.21.15.77:0/1619903948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6480501a0 0x7fc64817cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.550+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 >> 172.21.15.77:0/1619903948 conn(0x7fc648009fb0 msgr2=0x7fc648035280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.550+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 shutdown_connections 2024-03-06T07:22:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:08.550+0000 7fc6565ac640 1 -- 172.21.15.77:0/1619903948 wait complete. 2024-03-06T07:22:08.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.1 2024-03-06T07:22:08.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.7 2024-03-06T07:22:08.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.4 2024-03-06T07:22:08.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.6 2024-03-06T07:22:08.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.3 2024-03-06T07:22:08.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.2 2024-03-06T07:22:08.567 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.5 2024-03-06T07:22:09.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:09.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:09.557 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:09.562 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:09.565 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:09.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:09.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:09.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.692+0000 7f47599f6640 1 Processor -- start 2024-03-06T07:22:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.692+0000 7f47599f6640 1 -- start start 2024-03-06T07:22:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.693+0000 7f47599f6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f47540b4c00 0x7f47540b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.693+0000 7f47599f6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f47540b28a0 0x7f47540b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.694+0000 7f47599f6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475415d510 con 0x7f475415e540 2024-03-06T07:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.694+0000 7f47599f6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f475415e0b0 con 0x7f47540b28a0 2024-03-06T07:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.694+0000 7f47599f6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f475415d780 con 0x7f47540b4c00 2024-03-06T07:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.694+0000 7f47527fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f47540b28a0 0x7f47540b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.694+0000 7f47527fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f47540b28a0 0x7f47540b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56036/0 (socket says 172.21.15.77:56036) 2024-03-06T07:22:09.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.694+0000 7f47527fc640 1 -- 172.21.15.77:0/2838553952 learned_addr learned my addr 172.21.15.77:0/2838553952 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.695+0000 7f47527fc640 1 -- 172.21.15.77:0/2838553952 >> v2:172.21.15.158:3300/0 conn(0x7f47540b4c00 msgr2=0x7f47540b4fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.695+0000 7f47527fc640 1 --2- 172.21.15.77:0/2838553952 >> v2:172.21.15.158:3300/0 conn(0x7f47540b4c00 0x7f47540b4fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.695+0000 7f47527fc640 1 -- 172.21.15.77:0/2838553952 >> v1:172.21.15.158:6789/0 conn(0x7f475415e540 legacy=0x7f475415e9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.695+0000 7f47527fc640 1 -- 172.21.15.77:0/2838553952 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475415da60 con 0x7f47540b28a0 2024-03-06T07:22:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.696+0000 7f47527fc640 1 --2- 172.21.15.77:0/2838553952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f47540b2d10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f47540721d0 tx=0x7f474c02c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0dbc06052fa641f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.697+0000 7f4751ffb640 1 -- 172.21.15.77:0/2838553952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c02cb90 con 0x7f47540b28a0 2024-03-06T07:22:09.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.697+0000 7f4751ffb640 1 -- 172.21.15.77:0/2838553952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c02cd30 con 0x7f47540b28a0 2024-03-06T07:22:09.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.698+0000 7f47599f6640 1 -- 172.21.15.77:0/2838553952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 msgr2=0x7f47540b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.698+0000 7f47599f6640 1 --2- 172.21.15.77:0/2838553952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f47540b2d10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f47540721d0 tx=0x7f474c02c1f0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.698+0000 7f47599f6640 1 -- 172.21.15.77:0/2838553952 shutdown_connections 2024-03-06T07:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.698+0000 7f47599f6640 1 --2- 172.21.15.77:0/2838553952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f47540b2d10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.698+0000 7f47599f6640 1 --2- 172.21.15.77:0/2838553952 >> v2:172.21.15.158:3300/0 conn(0x7f47540b4c00 0x7f47540b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.698+0000 7f47599f6640 1 -- 172.21.15.77:0/2838553952 >> 172.21.15.77:0/2838553952 conn(0x7f475409a9f0 msgr2=0x7f475409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.699+0000 7f47599f6640 1 -- 172.21.15.77:0/2838553952 shutdown_connections 2024-03-06T07:22:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.699+0000 7f47599f6640 1 -- 172.21.15.77:0/2838553952 wait complete. 2024-03-06T07:22:09.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.701+0000 7f47599f6640 1 Processor -- start 2024-03-06T07:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.701+0000 7f47599f6640 1 -- start start 2024-03-06T07:22:09.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.702+0000 7f47599f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f4754151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.702+0000 7f47599f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47540b4c00 0x7f475414a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.702+0000 7f47599f6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f475415e540 0x7f475414a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.702+0000 7f47599f6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b33d0 con 0x7f47540b28a0 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.702+0000 7f47599f6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b3250 con 0x7f475415e540 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.702+0000 7f47599f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475415f0c0 con 0x7f47540b4c00 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.703+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f4754151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.703+0000 7f47537fe640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f475415e540 0x7f475414a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.703+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f4754151250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56040/0 (socket says 172.21.15.77:56040) 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.703+0000 7f4752ffd640 1 -- 172.21.15.77:0/2533402847 learned_addr learned my addr 172.21.15.77:0/2533402847 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.704+0000 7f47537fe640 1 -- 172.21.15.77:0/2533402847 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47540b4c00 msgr2=0x7f475414a350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.704+0000 7f47537fe640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47540b4c00 0x7f475414a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.704+0000 7f47537fe640 1 -- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 msgr2=0x7f4754151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.704+0000 7f47537fe640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f4754151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.704+0000 7f47537fe640 1 -- 172.21.15.77:0/2533402847 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4754153be0 con 0x7f475415e540 2024-03-06T07:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.704+0000 7f4752ffd640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f4754151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.705+0000 7f47537fe640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f475415e540 0x7f475414a890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f474800b3e0 tx=0x7f474800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.705+0000 7f4743fff640 1 -- 172.21.15.77:0/2533402847 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748015020 con 0x7f475415e540 2024-03-06T07:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.705+0000 7f47599f6640 1 -- 172.21.15.77:0/2533402847 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4754153ec0 con 0x7f475415e540 2024-03-06T07:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.705+0000 7f47599f6640 1 -- 172.21.15.77:0/2533402847 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4754151790 con 0x7f475415e540 2024-03-06T07:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.706+0000 7f4743fff640 1 -- 172.21.15.77:0/2533402847 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748011830 con 0x7f475415e540 2024-03-06T07:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.706+0000 7f4743fff640 1 -- 172.21.15.77:0/2533402847 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748011c20 con 0x7f475415e540 2024-03-06T07:22:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.707+0000 7f4743fff640 1 -- 172.21.15.77:0/2533402847 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4748018020 con 0x7f475415e540 2024-03-06T07:22:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.709+0000 7f4743fff640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f473806b5a0 0x7f473806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.709+0000 7f4752ffd640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f473806b5a0 0x7f473806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.710+0000 7f4752ffd640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f473806b5a0 0x7f473806da60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4744009e50 tx=0x7f4744009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.710+0000 7f4743fff640 1 -- 172.21.15.77:0/2533402847 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f474808a010 con 0x7f475415e540 2024-03-06T07:22:09.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.711+0000 7f47599f6640 1 -- 172.21.15.77:0/2533402847 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4730005290 con 0x7f475415e540 2024-03-06T07:22:09.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.716+0000 7f4743fff640 1 -- 172.21.15.77:0/2533402847 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f474801f020 con 0x7f475415e540 2024-03-06T07:22:09.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.748+0000 7fcfb8f75640 1 Processor -- start 2024-03-06T07:22:09.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.749+0000 7fcfb8f75640 1 -- start start 2024-03-06T07:22:09.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb8f75640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac033920 0x7fcfac100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fd1d7d7a640 1 Processor -- start 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb8f75640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcfac038060 0x7fcfac054430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb8f75640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac052cf0 con 0x7fcfac037c40 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb8f75640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac052f60 con 0x7fcfac038060 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb8f75640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac053240 con 0x7fcfac033920 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb37fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac033920 0x7fcfac100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb37fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac033920 0x7fcfac100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:47764/0 (socket says 172.21.15.77:47764) 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb37fe640 1 -- 172.21.15.77:0/361679000 learned_addr learned my addr 172.21.15.77:0/361679000 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.750+0000 7fcfb3fff640 1 --2- 172.21.15.77:0/361679000 >> v2:172.21.15.158:3300/0 conn(0x7fcfac038060 0x7fcfac054430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fd1d7d7a640 1 -- start start 2024-03-06T07:22:09.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fb834a55640 1 Processor -- start 2024-03-06T07:22:09.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fb834a55640 1 -- start start 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279470070 0 0) 0x7fcfac052cf0 con 0x7fcfac037c40 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa4003520 con 0x7fcfac037c40 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 639579877 0 0) 0x7fcfa4003520 con 0x7fcfac037c40 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac033920 msgr2=0x7fcfac100da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac033920 0x7fcfac100da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 >> v2:172.21.15.158:3300/0 conn(0x7fcfac038060 msgr2=0x7fcfac054430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 --2- 172.21.15.77:0/361679000 >> v2:172.21.15.158:3300/0 conn(0x7fcfac038060 0x7fcfac054430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac1012e0 con 0x7fcfac037c40 2024-03-06T07:22:09.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.751+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fcfa8001650 con 0x7fcfac037c40 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 >> v1:172.21.15.77:6789/0 conn(0x7fcfac037c40 legacy=0x7fcfac053d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb27fc640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa4003a10 0x7fcfa4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb27fc640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa4004800 0x7fcfa4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb27fc640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa40051c0 0x7fcfa400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac052f60 con 0x7fcfa40051c0 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac053240 con 0x7fcfa4003a10 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa401d130 con 0x7fcfa4004800 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb3fff640 1 --2- 172.21.15.77:0/361679000 >> v2:172.21.15.158:3300/0 conn(0x7fcfac038060 0x7fcfac054430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:09.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fd1d7d7a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd1c80dfed0 0x7fd1c80e39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fc517fff640 1 Processor -- start 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb2ffd640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa4004800 0x7fcfa4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fc517fff640 1 -- start start 2024-03-06T07:22:09.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fd1d5b16640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd1c80dfed0 0x7fd1c80e39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fd1d5b16640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd1c80dfed0 0x7fd1c80e39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:44102/0 (socket says 172.21.15.77:44102) 2024-03-06T07:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fcfb37fe640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa4003a10 0x7fcfa4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fd1d7d7a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd1c8050210 0x7fd1c80e3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fd1d7d7a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c80e02b0 con 0x7fd1c80e4420 2024-03-06T07:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fd1d7d7a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c80506f0 con 0x7fd1c8050210 2024-03-06T07:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.752+0000 7fd1d7d7a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c8050d50 con 0x7fd1c80dfed0 2024-03-06T07:22:09.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fd1d5b16640 1 -- 172.21.15.77:0/3620145328 learned_addr learned my addr 172.21.15.77:0/3620145328 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fcfb37fe640 1 -- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa4004800 msgr2=0x7fcfa4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fb834a55640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82804fa80 0x7fb82804fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fb834a55640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb828054260 0x7fb8280389c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fb834a55640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280503a0 con 0x7fb8280df410 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fb834a55640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb828006890 con 0x7fb828054260 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fb834a55640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb828006ae0 con 0x7fb82804fa80 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fb82e59c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82804fa80 0x7fb82804fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fb82e59c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82804fa80 0x7fb82804fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:47778/0 (socket says 172.21.15.77:47778) 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fb82e59c640 1 -- 172.21.15.77:0/41830428 learned_addr learned my addr 172.21.15.77:0/41830428 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fb82ed9d640 1 --2- 172.21.15.77:0/41830428 >> v2:172.21.15.158:3300/0 conn(0x7fb828054260 0x7fb8280389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fcfb37fe640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa4004800 0x7fcfa4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fcfb37fe640 1 -- 172.21.15.77:0/361679000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa40051c0 msgr2=0x7fcfa400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fcfb37fe640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa40051c0 0x7fcfa400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fcfb37fe640 1 -- 172.21.15.77:0/361679000 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa401d3f0 con 0x7fcfa4003a10 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fc517fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc510006b00 0x7fc510006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fc517fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc510050420 0x7fc510050880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc517fff640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc510053930 con 0x7fc5100eb080 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc517fff640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc510051780 con 0x7fc510050420 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc517fff640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc510007420 con 0x7fc510006b00 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc5177fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc5100eb080 0x7fc5100eb4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45552/0 (socket says 172.21.15.77:45552) 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc5177fe640 1 -- 172.21.15.77:0/2910831842 learned_addr learned my addr 172.21.15.77:0/2910831842 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:09.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fd1d5b16640 1 -- 172.21.15.77:0/3620145328 >> v2:172.21.15.77:3300/0 conn(0x7fd1c8050210 msgr2=0x7fd1c80e3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fd1d5b16640 1 --2- 172.21.15.77:0/3620145328 >> v2:172.21.15.77:3300/0 conn(0x7fd1c8050210 0x7fd1c80e3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fd1d5b16640 1 -- 172.21.15.77:0/3620145328 >> v1:172.21.15.158:6789/0 conn(0x7fd1c80e4420 legacy=0x7fd1c8053b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.753+0000 7fd1d5b16640 1 -- 172.21.15.77:0/3620145328 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c8051030 con 0x7fd1c80dfed0 2024-03-06T07:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7f4f193ac640 1 Processor -- start 2024-03-06T07:22:09.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fd1d5b16640 1 --2- 172.21.15.77:0/3620145328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c80dfed0 0x7fd1c80e39a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800e770 tx=0x7fd1cc02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b941c00a3249adc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345879090 0 0) 0x7fc510053930 con 0x7fc5100eb080 2024-03-06T07:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc5167fc640 1 --2- 172.21.15.77:0/2910831842 >> v2:172.21.15.158:3300/0 conn(0x7fc510050420 0x7fc510050880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc500003520 con 0x7fc5100eb080 2024-03-06T07:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7f4f193ac640 1 -- start start 2024-03-06T07:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fd1d4b14640 1 -- 172.21.15.77:0/3620145328 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc035020 con 0x7fd1c80dfed0 2024-03-06T07:22:09.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fc516ffd640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc510006b00 0x7fc510006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fb82e59c640 1 -- 172.21.15.77:0/41830428 >> v2:172.21.15.158:3300/0 conn(0x7fb828054260 msgr2=0x7fb8280389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fb82e59c640 1 --2- 172.21.15.77:0/41830428 >> v2:172.21.15.158:3300/0 conn(0x7fb828054260 0x7fb8280389c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fb82e59c640 1 -- 172.21.15.77:0/41830428 >> v1:172.21.15.77:6789/0 conn(0x7fb8280df410 legacy=0x7fb8280382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7fb82e59c640 1 -- 172.21.15.77:0/41830428 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828006dc0 con 0x7fb82804fa80 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fb82e59c640 1 --2- 172.21.15.77:0/41830428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82804fa80 0x7fb82804fe60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb820007360 tx=0x7fb82002db30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7026090be1d37e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fcfb37fe640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa4003a10 0x7fcfa4003e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcfac007a60 tx=0x7fcfa4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ee7666338499926 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.754+0000 7f0ebf59e640 1 Processor -- start 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1091748978 0 0) 0x7fc500003520 con 0x7fc5100eb080 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc510006b00 msgr2=0x7fc510006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc510006b00 0x7fc510006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 >> v2:172.21.15.158:3300/0 conn(0x7fc510050420 msgr2=0x7fc510050880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 --2- 172.21.15.77:0/2910831842 >> v2:172.21.15.158:3300/0 conn(0x7fc510050420 0x7fc510050880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc510050ee0 con 0x7fc5100eb080 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fc50c001670 con 0x7fc5100eb080 2024-03-06T07:22:09.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 >> v1:172.21.15.77:6789/0 conn(0x7fc5100eb080 legacy=0x7fc5100eb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5000039b0 0x7fc500003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5000051c0 0x7fc50000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc510051780 con 0x7fc5000051c0 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc510007420 con 0x7fc5000039b0 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50000bf00 con 0x7fc500004800 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc516ffd640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc510006b00 0x7fc510006ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc516ffd640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5000039b0 0x7fc500003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc5177fe640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5000051c0 0x7fc50000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fc5167fc640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fd1d4b14640 1 -- 172.21.15.77:0/3620145328 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc031800 con 0x7fd1c80dfed0 2024-03-06T07:22:09.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fd1d4b14640 1 -- 172.21.15.77:0/3620145328 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc031bb0 con 0x7fd1c80dfed0 2024-03-06T07:22:09.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7f4f193ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4f0c005e80 0x7f4f0c0062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7f4f193ac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0c053710 con 0x7f4f0c006800 2024-03-06T07:22:09.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7f4f193ac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0c0539f0 con 0x7f4f0c0eacf0 2024-03-06T07:22:09.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7f4f193ac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f0c054320 con 0x7f4f0c005e80 2024-03-06T07:22:09.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fb82d59a640 1 -- 172.21.15.77:0/41830428 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820008df0 con 0x7fb82804fa80 2024-03-06T07:22:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa003a070 con 0x7fcfa4003a10 2024-03-06T07:22:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa0035410 con 0x7fcfa4003a10 2024-03-06T07:22:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fcfb27fc640 1 -- 172.21.15.77:0/361679000 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00069a0 con 0x7fcfa4003a10 2024-03-06T07:22:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fb82d59a640 1 -- 172.21.15.77:0/41830428 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82000a0d0 con 0x7fb82804fa80 2024-03-06T07:22:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fb82d59a640 1 -- 172.21.15.77:0/41830428 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82000a4e0 con 0x7fb82804fa80 2024-03-06T07:22:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fc5177fe640 1 -- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc500004800 msgr2=0x7fc500004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fc5177fe640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fc5177fe640 1 -- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5000039b0 msgr2=0x7fc500003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fc5177fe640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5000039b0 0x7fc500003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fc5177fe640 1 -- 172.21.15.77:0/2910831842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc510050ee0 con 0x7fc5000051c0 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f4f12ffd640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f4f0c0eacf0 0x7f4f0c04fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:46182/0 (socket says 172.21.15.77:46182) 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f4f12ffd640 1 -- 172.21.15.77:0/3743083890 learned_addr learned my addr 172.21.15.77:0/3743083890 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fc5167fc640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc500004800 0x7fc500004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f4f127fc640 1 --2- 172.21.15.77:0/3743083890 >> v2:172.21.15.77:3300/0 conn(0x7f4f0c005e80 0x7f4f0c0062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682875202 0 0) 0x7f4f0c0539f0 con 0x7f4f0c0eacf0 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f00003520 con 0x7f4f0c0eacf0 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.755+0000 7f0ebf59e640 1 -- start start 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebf59e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec0161ed0 0x7f0ec01622b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebf59e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ec01627f0 0x7f0ec006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebf59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0ec00afae0 0x7f0ec006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebf59e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec00b3a90 con 0x7f0ec00afae0 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebf59e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec00b3d70 con 0x7f0ec01627f0 2024-03-06T07:22:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebf59e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00b57a0 con 0x7f0ec0161ed0 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fc516ffd640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5000039b0 0x7fc500003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3620145328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c80dfed0 msgr2=0x7fd1c80e39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3620145328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c80dfed0 0x7fd1c80e39a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800e770 tx=0x7fd1cc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3620145328 shutdown_connections 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3620145328 >> v2:172.21.15.77:3300/0 conn(0x7fd1c8050210 0x7fd1c80e3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3620145328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c80dfed0 0x7fd1c80e39a0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800e770 tx=0x7fd1cc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3620145328 >> 172.21.15.77:0/3620145328 conn(0x7fd1c8055bc0 msgr2=0x7fd1c8055fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 <== mon.1 v1:172.21.15.158:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 776993938 0 0) 0x7f4f00003520 con 0x7f4f0c0eacf0 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 >> v2:172.21.15.77:3300/0 conn(0x7f4f0c005e80 msgr2=0x7f4f0c0062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fb834a55640 1 -- 172.21.15.77:0/41830428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82804fa80 msgr2=0x7fb82804fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fb834a55640 1 --2- 172.21.15.77:0/41830428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82804fa80 0x7fb82804fe60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb820007360 tx=0x7fb82002db30 comp rx=0 tx=0).stop 2024-03-06T07:22:09.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebe59c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec0161ed0 0x7f0ec01622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebe59c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec0161ed0 0x7f0ec01622b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:47804/0 (socket says 172.21.15.77:47804) 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7f0ebe59c640 1 -- 172.21.15.77:0/583155159 learned_addr learned my addr 172.21.15.77:0/583155159 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f0ebdd9b640 1 --2- 172.21.15.77:0/583155159 >> v2:172.21.15.158:3300/0 conn(0x7f0ec01627f0 0x7f0ec006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f0ebed9d640 1 --2- 172.21.15.77:0/583155159 >> v2:172.21.15.77:3300/0 conn(0x7f0ec00afae0 0x7f0ec006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f0ebe59c640 1 -- 172.21.15.77:0/583155159 >> v2:172.21.15.158:3300/0 conn(0x7f0ec01627f0 msgr2=0x7f0ec006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f0ebe59c640 1 --2- 172.21.15.77:0/583155159 >> v2:172.21.15.158:3300/0 conn(0x7f0ec01627f0 0x7f0ec006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f0ebe59c640 1 -- 172.21.15.77:0/583155159 >> v2:172.21.15.77:3300/0 conn(0x7f0ec00afae0 msgr2=0x7f0ec006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.756+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3620145328 shutdown_connections 2024-03-06T07:22:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 --2- 172.21.15.77:0/3743083890 >> v2:172.21.15.77:3300/0 conn(0x7f4f0c005e80 0x7f4f0c0062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 >> v1:172.21.15.77:6789/0 conn(0x7f4f0c006800 legacy=0x7f4f0c006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3620145328 wait complete. 2024-03-06T07:22:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 --> v1:172.21.15.158:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c053cd0 con 0x7f4f0c0eacf0 2024-03-06T07:22:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 <== mon.1 v1:172.21.15.158:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f4f04001670 con 0x7f4f0c0eacf0 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fb834a55640 1 -- 172.21.15.77:0/41830428 shutdown_connections 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fb834a55640 1 --2- 172.21.15.77:0/41830428 >> v2:172.21.15.158:3300/0 conn(0x7fb828054260 0x7fb8280389c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fb834a55640 1 --2- 172.21.15.77:0/41830428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82804fa80 0x7fb82804fe60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fb834a55640 1 -- 172.21.15.77:0/41830428 >> 172.21.15.77:0/41830428 conn(0x7fb828009fa0 msgr2=0x7fb82800a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fb834a55640 1 -- 172.21.15.77:0/41830428 shutdown_connections 2024-03-06T07:22:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 >> v1:172.21.15.158:6789/0 conn(0x7f4f0c0eacf0 legacy=0x7f4f0c04fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f000039b0 0x7f4f00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f00004800 0x7f4f00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000051c0 0x7f4f0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fb834a55640 1 -- 172.21.15.77:0/41830428 wait complete. 2024-03-06T07:22:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fcfb8f75640 1 -- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa4003a10 msgr2=0x7fcfa4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fcfb8f75640 1 --2- 172.21.15.77:0/361679000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa4003a10 0x7fcfa4003e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcfac007a60 tx=0x7fcfa4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:09.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fcfb3fff640 1 -- 172.21.15.77:0/361679000 reap_dead start 2024-03-06T07:22:09.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c054320 con 0x7f4f00004800 2024-03-06T07:22:09.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fcfb8f75640 1 -- 172.21.15.77:0/361679000 shutdown_connections 2024-03-06T07:22:09.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fcfb8f75640 1 -- 172.21.15.77:0/361679000 >> 172.21.15.77:0/361679000 conn(0x7fcfac009fb0 msgr2=0x7fcfac00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0000c060 con 0x7f4f000051c0 2024-03-06T07:22:09.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f12ffd640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f000039b0 0x7f4f00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0000c260 con 0x7f4f000039b0 2024-03-06T07:22:09.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f127fc640 1 --2- 172.21.15.77:0/3743083890 >> v2:172.21.15.77:3300/0 conn(0x7f4f0c005e80 0x7f4f0c0062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7fc5177fe640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5000051c0 0x7fc50000b9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00c5d0 tx=0x7fc500005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ae51825a1d224a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f9bb5de9640 1 Processor -- start 2024-03-06T07:22:09.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f4f137fe640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000051c0 0x7f4f0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f4f127fc640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f00004800 0x7f4f00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f4f12ffd640 1 -- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000051c0 msgr2=0x7f4f0000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f4f12ffd640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000051c0 0x7f4f0000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f4f12ffd640 1 -- 172.21.15.77:0/3743083890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f00004800 msgr2=0x7f4f00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f4f12ffd640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f00004800 0x7f4f00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f4f12ffd640 1 -- 172.21.15.77:0/3743083890 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f00003520 con 0x7f4f000039b0 2024-03-06T07:22:09.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c007d70 con 0x7fc5000051c0 2024-03-06T07:22:09.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f4f127fc640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f00004800 0x7f4f00004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:09.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c030070 con 0x7fc5000051c0 2024-03-06T07:22:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.757+0000 7f9bb5de9640 1 -- start start 2024-03-06T07:22:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7fc515ffb640 1 -- 172.21.15.77:0/2910831842 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c007d70 con 0x7fc5000051c0 2024-03-06T07:22:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fc517fff640 1 -- 172.21.15.77:0/2910831842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5000051c0 msgr2=0x7fc50000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fc517fff640 1 --2- 172.21.15.77:0/2910831842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5000051c0 0x7fc50000b9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00c5d0 tx=0x7fc500005830 comp rx=0 tx=0).stop 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f0ebe59c640 1 --2- 172.21.15.77:0/583155159 >> v2:172.21.15.77:3300/0 conn(0x7f0ec00afae0 0x7f0ec006b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.758+0000 7f0ebe59c640 1 -- 172.21.15.77:0/583155159 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00b0270 con 0x7f0ec0161ed0 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f0ebed9d640 1 --2- 172.21.15.77:0/583155159 >> v2:172.21.15.77:3300/0 conn(0x7f0ec00afae0 0x7f0ec006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9bb5de9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba803e830 0x7f9ba8053be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9bb5de9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9ba8054120 0x7f9ba80068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9bb5de9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba8058560 con 0x7f9ba80e30b0 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9bb5de9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba8058840 con 0x7f9ba8054120 2024-03-06T07:22:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9bb5de9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba8058b20 con 0x7f9ba803e830 2024-03-06T07:22:09.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fc5177fe640 1 -- 172.21.15.77:0/2910831842 reap_dead start 2024-03-06T07:22:09.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f4f12ffd640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f000039b0 0x7f4f00003dd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400c190 tx=0x7f4f00005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ba5addf05de200 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fd1d7d7a640 1 Processor -- start 2024-03-06T07:22:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fb834a55640 1 Processor -- start 2024-03-06T07:22:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fc517fff640 1 -- 172.21.15.77:0/2910831842 shutdown_connections 2024-03-06T07:22:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fc517fff640 1 -- 172.21.15.77:0/2910831842 >> 172.21.15.77:0/2910831842 conn(0x7fc510055bc0 msgr2=0x7fc510055fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fd1d7d7a640 1 -- start start 2024-03-06T07:22:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7fb834a55640 1 -- start start 2024-03-06T07:22:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9baf7fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9ba80e30b0 0x7f9ba8057d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:46186/0 (socket says 172.21.15.77:46186) 2024-03-06T07:22:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9baf7fe640 1 -- 172.21.15.77:0/1011037542 learned_addr learned my addr 172.21.15.77:0/1011037542 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9baffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9ba8054120 0x7f9ba80068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04007c80 con 0x7f4f000039b0 2024-03-06T07:22:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9baeffd640 1 --2- 172.21.15.77:0/1011037542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba803e830 0x7f9ba8053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04030070 con 0x7f4f000039b0 2024-03-06T07:22:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f4f11ffb640 1 -- 172.21.15.77:0/3743083890 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04005d50 con 0x7f4f000039b0 2024-03-06T07:22:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9bae7fc640 1 -- 172.21.15.77:0/1011037542 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513299031 0 0) 0x7f9ba8058560 con 0x7f9ba80e30b0 2024-03-06T07:22:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9bae7fc640 1 -- 172.21.15.77:0/1011037542 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba0003520 con 0x7f9ba80e30b0 2024-03-06T07:22:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9baeffd640 1 -- 172.21.15.77:0/1011037542 >> v2:172.21.15.77:3300/0 conn(0x7f9ba8054120 msgr2=0x7f9ba80068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9baeffd640 1 --2- 172.21.15.77:0/1011037542 >> v2:172.21.15.77:3300/0 conn(0x7f9ba8054120 0x7f9ba80068c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9baeffd640 1 -- 172.21.15.77:0/1011037542 >> v1:172.21.15.158:6789/0 conn(0x7f9ba80e30b0 legacy=0x7f9ba8057d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.759+0000 7f9baeffd640 1 -- 172.21.15.77:0/1011037542 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba80063a0 con 0x7f9ba803e830 2024-03-06T07:22:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7f0ebe59c640 1 --2- 172.21.15.77:0/583155159 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec0161ed0 0x7f0ec01622b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00d18a0 tx=0x7f0eb802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6abfcd2302ab30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7f4f137fe640 1 -- 172.21.15.77:0/3743083890 reap_dead start 2024-03-06T07:22:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7f0ebd59a640 1 -- 172.21.15.77:0/583155159 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb802ca80 con 0x7f0ec0161ed0 2024-03-06T07:22:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb834a55640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82804fa80 0x7fb8280d8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7f0ebd59a640 1 -- 172.21.15.77:0/583155159 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb802cc20 con 0x7f0ec0161ed0 2024-03-06T07:22:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7f0ebd59a640 1 -- 172.21.15.77:0/583155159 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb80343f0 con 0x7f0ec0161ed0 2024-03-06T07:22:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb834a55640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb828054260 0x7fb82804b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb834a55640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8280df410 0x7fb8280df7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb834a55640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280546f0 con 0x7fb82804fa80 2024-03-06T07:22:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb834a55640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb828007380 con 0x7fb8280df410 2024-03-06T07:22:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb834a55640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb828054870 con 0x7fb828054260 2024-03-06T07:22:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7f4f193ac640 1 -- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f000039b0 msgr2=0x7f4f00003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7f4f193ac640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f000039b0 0x7f4f00003dd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400c190 tx=0x7f4f00005830 comp rx=0 tx=0).stop 2024-03-06T07:22:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb82e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82804fa80 0x7fb8280d8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb82dd9b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb828054260 0x7fb82804b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb82e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82804fa80 0x7fb8280d8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56134/0 (socket says 172.21.15.77:56134) 2024-03-06T07:22:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb82dd9b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb828054260 0x7fb82804b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:47832/0 (socket says 172.21.15.77:47832) 2024-03-06T07:22:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fb82e59c640 1 -- 172.21.15.77:0/1058417837 learned_addr learned my addr 172.21.15.77:0/1058417837 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fd1d7d7a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c8050210 0x7fd1c804b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fc517fff640 1 -- 172.21.15.77:0/2910831842 shutdown_connections 2024-03-06T07:22:09.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fcfb8f75640 1 -- 172.21.15.77:0/361679000 shutdown_connections 2024-03-06T07:22:09.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fcfb8f75640 1 -- 172.21.15.77:0/361679000 wait complete. 2024-03-06T07:22:09.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7f9baeffd640 1 --2- 172.21.15.77:0/1011037542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba803e830 0x7f9ba8053be0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9bb002d580 tx=0x7f9bb00a68f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c4c02f9d37f8b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.760+0000 7fc517fff640 1 -- 172.21.15.77:0/2910831842 wait complete. 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f4f193ac640 1 -- 172.21.15.77:0/3743083890 shutdown_connections 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f4f193ac640 1 --2- 172.21.15.77:0/3743083890 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f000039b0 0x7f4f00003dd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f4f193ac640 1 -- 172.21.15.77:0/3743083890 >> 172.21.15.77:0/3743083890 conn(0x7f4f0c009fa0 msgr2=0x7f4f0c00a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f4f193ac640 1 -- 172.21.15.77:0/3743083890 shutdown_connections 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb82e59c640 1 -- 172.21.15.77:0/1058417837 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb828054260 msgr2=0x7fb82804b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb82e59c640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb828054260 0x7fb82804b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb82e59c640 1 -- 172.21.15.77:0/1058417837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8280df410 msgr2=0x7fb8280df7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb82e59c640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8280df410 0x7fb8280df7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb82e59c640 1 -- 172.21.15.77:0/1058417837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8280c4cf0 con 0x7fb82804fa80 2024-03-06T07:22:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb82e59c640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82804fa80 0x7fb8280d8cd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb820000c00 tx=0x7fb8200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb81effd640 1 -- 172.21.15.77:0/1058417837 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82002f6f0 con 0x7fb82804fa80 2024-03-06T07:22:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb834a55640 1 -- 172.21.15.77:0/1058417837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8280d8a30 con 0x7fb82804fa80 2024-03-06T07:22:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb82dd9b640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb828054260 0x7fb82804b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7fb834a55640 1 -- 172.21.15.77:0/1058417837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8280d7370 con 0x7fb82804fa80 2024-03-06T07:22:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7fb81effd640 1 -- 172.21.15.77:0/1058417837 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82002f890 con 0x7fb82804fa80 2024-03-06T07:22:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7fb81effd640 1 -- 172.21.15.77:0/1058417837 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82002fc80 con 0x7fb82804fa80 2024-03-06T07:22:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7fcfb8f75640 1 Processor -- start 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 -- 172.21.15.77:0/583155159 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec0161ed0 msgr2=0x7f0ec01622b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/583155159 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec0161ed0 0x7f0ec01622b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00d18a0 tx=0x7f0eb802bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 -- 172.21.15.77:0/583155159 shutdown_connections 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/583155159 >> v2:172.21.15.77:3300/0 conn(0x7f0ec00afae0 0x7f0ec006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/583155159 >> v2:172.21.15.158:3300/0 conn(0x7f0ec01627f0 0x7f0ec006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/583155159 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec0161ed0 0x7f0ec01622b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 -- 172.21.15.77:0/583155159 >> 172.21.15.77:0/583155159 conn(0x7f0ec006c0c0 msgr2=0x7f0ec006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 -- 172.21.15.77:0/583155159 shutdown_connections 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7fc517fff640 1 Processor -- start 2024-03-06T07:22:09.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7fc517fff640 1 -- start start 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f9bae7fc640 1 -- 172.21.15.77:0/1011037542 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb00b0020 con 0x7f9ba803e830 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f9bae7fc640 1 -- 172.21.15.77:0/1011037542 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb00ac700 con 0x7f9ba803e830 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7f9bb5de9640 1 -- 172.21.15.77:0/1011037542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba803e830 msgr2=0x7f9ba8053be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7f9bae7fc640 1 -- 172.21.15.77:0/1011037542 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb00b0020 con 0x7f9ba803e830 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7f9bb5de9640 1 --2- 172.21.15.77:0/1011037542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba803e830 0x7f9ba8053be0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9bb002d580 tx=0x7f9bb00a68f0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7f9bb5de9640 1 -- 172.21.15.77:0/1011037542 shutdown_connections 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7f9bb5de9640 1 --2- 172.21.15.77:0/1011037542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba803e830 0x7f9ba8053be0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7f9bb5de9640 1 --2- 172.21.15.77:0/1011037542 >> v2:172.21.15.77:3300/0 conn(0x7f9ba8054120 0x7f9ba80068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7f9bb5de9640 1 -- 172.21.15.77:0/1011037542 >> 172.21.15.77:0/1011037542 conn(0x7f9ba80559d0 msgr2=0x7f9ba8055de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7f9bb5de9640 1 -- 172.21.15.77:0/1011037542 shutdown_connections 2024-03-06T07:22:09.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f0ebf59e640 1 -- 172.21.15.77:0/583155159 wait complete. 2024-03-06T07:22:09.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7f0ebf59e640 1 Processor -- start 2024-03-06T07:22:09.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fb81cff9640 1 -- 172.21.15.77:0/1058417837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb808005290 con 0x7fb82804fa80 2024-03-06T07:22:09.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.762+0000 7fcfb8f75640 1 -- start start 2024-03-06T07:22:09.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.761+0000 7f4f193ac640 1 -- 172.21.15.77:0/3743083890 wait complete. 2024-03-06T07:22:09.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7f9bb5de9640 1 -- 172.21.15.77:0/1011037542 wait complete. 2024-03-06T07:22:09.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7f4f193ac640 1 Processor -- start 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fcfb8f75640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfac032da0 0x7fcfac196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fcfb8f75640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac04ef50 0x7fcfac04f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fcfb8f75640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac035740 0x7fcfac035bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fcfb8f75640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac100da0 con 0x7fcfac032da0 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fcfb8f75640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac053800 con 0x7fcfac04ef50 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fcfb8f75640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac100f20 con 0x7fcfac035740 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac04ef50 0x7fcfac04f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb3fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac035740 0x7fcfac035bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac04ef50 0x7fcfac04f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:44180/0 (socket says 172.21.15.77:44180) 2024-03-06T07:22:09.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb2ffd640 1 -- 172.21.15.77:0/31179068 learned_addr learned my addr 172.21.15.77:0/31179068 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fc517fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc51017c760 0x7fc51004ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb3fff640 1 -- 172.21.15.77:0/31179068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac04ef50 msgr2=0x7fcfac04f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7f0ebf59e640 1 -- start start 2024-03-06T07:22:09.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb3fff640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac04ef50 0x7fcfac04f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb3fff640 1 -- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfac032da0 msgr2=0x7fcfac196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb3fff640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfac032da0 0x7fcfac196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb3fff640 1 -- 172.21.15.77:0/31179068 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac02c5e0 con 0x7fcfac035740 2024-03-06T07:22:09.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fcfb2ffd640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac04ef50 0x7fcfac04f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7f0ebf59e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00afae0 0x7f0ec013e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7f0ebf59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec0161ed0 0x7f0ec0143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fc517fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51004bf50 0x7fc51004ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7f9bb5de9640 1 Processor -- start 2024-03-06T07:22:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fcfb3fff640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac035740 0x7fcfac035bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40686a0 tx=0x7fcfb40683c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f0ebf59e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec01627f0 0x7f0ec0133b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f0ebf59e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec006f410 con 0x7f0ec0161ed0 2024-03-06T07:22:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f0ebf59e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec006f290 con 0x7f0ec00afae0 2024-03-06T07:22:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f0ebf59e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec006f590 con 0x7f0ec01627f0 2024-03-06T07:22:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f9bb5de9640 1 -- start start 2024-03-06T07:22:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7f4f193ac640 1 -- start start 2024-03-06T07:22:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f193ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000039b0 0x7f4f0c186310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f193ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f0c0e3d10 0x7f4f0c0e2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f193ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f0c0e5e30 0x7f4f0c0e62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f193ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c0547b0 con 0x7f4f0c0e3d10 2024-03-06T07:22:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f193ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c050190 con 0x7f4f000039b0 2024-03-06T07:22:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f193ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c054930 con 0x7f4f0c0e5e30 2024-03-06T07:22:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fb81effd640 1 -- 172.21.15.77:0/1058417837 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb820047060 con 0x7fb82804fa80 2024-03-06T07:22:09.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f0ebdd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec0161ed0 0x7f0ec0143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f12ffd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000039b0 0x7f4f0c186310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f12ffd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000039b0 0x7f4f0c186310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:44206/0 (socket says 172.21.15.77:44206) 2024-03-06T07:22:09.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f12ffd640 1 -- 172.21.15.77:0/944189184 learned_addr learned my addr 172.21.15.77:0/944189184 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fcf9ffff640 1 -- 172.21.15.77:0/31179068 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4073230 con 0x7fcfac035740 2024-03-06T07:22:09.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fcf9ffff640 1 -- 172.21.15.77:0/31179068 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb40733f0 con 0x7fcfac035740 2024-03-06T07:22:09.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fcf9ffff640 1 -- 172.21.15.77:0/31179068 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40811d0 con 0x7fcfac035740 2024-03-06T07:22:09.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f137fe640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f0c0e5e30 0x7f4f0c0e62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f0ebdd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec0161ed0 0x7f0ec0143740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56154/0 (socket says 172.21.15.77:56154) 2024-03-06T07:22:09.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f0ebdd9b640 1 -- 172.21.15.77:0/632820896 learned_addr learned my addr 172.21.15.77:0/632820896 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f12ffd640 1 -- 172.21.15.77:0/944189184 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f0c0e5e30 msgr2=0x7f4f0c0e62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f12ffd640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f0c0e5e30 0x7f4f0c0e62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f12ffd640 1 -- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f0c0e3d10 msgr2=0x7f4f0c0e2f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:09.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fb81effd640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb81006b550 0x7fb81006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f127fc640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f0c0e3d10 0x7f4f0c0e2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fb81effd640 1 -- 172.21.15.77:0/1058417837 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8200ab210 con 0x7fb82804fa80 2024-03-06T07:22:09.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fb82dd9b640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb81006b550 0x7fb81006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f12ffd640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f0c0e3d10 0x7f4f0c0e2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9bb5de9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba803e830 0x7f9ba804eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7f4f12ffd640 1 -- 172.21.15.77:0/944189184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c0e4490 con 0x7f4f000039b0 2024-03-06T07:22:09.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac02c8c0 con 0x7fcfac035740 2024-03-06T07:22:09.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac0fd380 con 0x7fcfac035740 2024-03-06T07:22:09.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9bb5de9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba8054120 0x7f9ba8038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9bb5de9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba80e30b0 0x7f9ba803b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9bb5de9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba8006e00 con 0x7f9ba803e830 2024-03-06T07:22:09.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9bb5de9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba80e3490 con 0x7f9ba80e30b0 2024-03-06T07:22:09.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9bb5de9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba8006f80 con 0x7f9ba8054120 2024-03-06T07:22:09.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f4f12ffd640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000039b0 0x7f4f0c186310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f0402c4e0 tx=0x7f4f04001970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f0ebe59c640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00afae0 0x7f0ec013e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f4f0b7fe640 1 -- 172.21.15.77:0/944189184 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04045070 con 0x7f4f000039b0 2024-03-06T07:22:09.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f0ebed9d640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec01627f0 0x7f0ec0133b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9baf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba803e830 0x7f9ba804eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9baf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba803e830 0x7f9ba804eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56180/0 (socket says 172.21.15.77:56180) 2024-03-06T07:22:09.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9baf7fe640 1 -- 172.21.15.77:0/205535511 learned_addr learned my addr 172.21.15.77:0/205535511 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f0ebdd9b640 1 -- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec01627f0 msgr2=0x7f0ec0133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f0ebdd9b640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec01627f0 0x7f0ec0133b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f0ebdd9b640 1 -- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00afae0 msgr2=0x7f0ec013e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f0ebdd9b640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00afae0 0x7f0ec013e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f0ebdd9b640 1 -- 172.21.15.77:0/632820896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec0150330 con 0x7f0ec0161ed0 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f0ebe59c640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00afae0 0x7f0ec013e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fd1d7d7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1c80e4420 0x7fd1c804ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fd1d7d7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c804e300 0x7fd1c80d4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fd1d7d7a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c8008f50 con 0x7fd1c804e300 2024-03-06T07:22:09.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fd1d7d7a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c8055770 con 0x7fd1c8050210 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.763+0000 7fd1d7d7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c80090d0 con 0x7fd1c80e4420 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5b16640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c8050210 0x7fd1c804b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5b16640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c8050210 0x7fd1c804b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:44166/0 (socket says 172.21.15.77:44166) 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5b16640 1 -- 172.21.15.77:0/3579956146 learned_addr learned my addr 172.21.15.77:0/3579956146 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5315640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1c80e4420 0x7fd1c804ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5b16640 1 -- 172.21.15.77:0/3579956146 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1c80e4420 msgr2=0x7fd1c804ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5b16640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1c80e4420 0x7fd1c804ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5b16640 1 -- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c804e300 msgr2=0x7fd1c80d4690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5b16640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c804e300 0x7fd1c80d4690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.764+0000 7fd1d5b16640 1 -- 172.21.15.77:0/3579956146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c804cfc0 con 0x7fd1c8050210 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fd1d5b16640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c8050210 0x7fd1c804b3e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0090b0 tx=0x7fd1cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fd1c67fc640 1 -- 172.21.15.77:0/3579956146 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc042340 con 0x7fd1c8050210 2024-03-06T07:22:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fd1c67fc640 1 -- 172.21.15.77:0/3579956146 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc0424e0 con 0x7fd1c8050210 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fd1c67fc640 1 -- 172.21.15.77:0/3579956146 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc0428b0 con 0x7fd1c8050210 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c804d2a0 con 0x7fd1c8050210 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c804d860 con 0x7fd1c8050210 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fc517fff640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc51004d3d0 0x7fc5101892d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fc517fff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc510049260 con 0x7fc51004bf50 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fc517fff640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100514a0 con 0x7fc51004d3d0 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fc517fff640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100e77c0 con 0x7fc51017c760 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc51004d3d0 0x7fc5101892d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc51004d3d0 0x7fc5101892d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:44228/0 (socket says 172.21.15.77:44228) 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7fc5177fe640 1 -- 172.21.15.77:0/3241497824 learned_addr learned my addr 172.21.15.77:0/3241497824 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7fc5177fe640 1 -- 172.21.15.77:0/3241497824 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc51017c760 msgr2=0x7fc51004ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7fc5167fc640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51004bf50 0x7fc51004ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9baeffd640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba8054120 0x7f9ba8038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f9baffff640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba80e30b0 0x7f9ba803b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9baeffd640 1 -- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba80e30b0 msgr2=0x7f9ba803b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9baeffd640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba80e30b0 0x7f9ba803b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9baeffd640 1 -- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba803e830 msgr2=0x7f9ba804eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9baeffd640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba803e830 0x7f9ba804eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9baeffd640 1 -- 172.21.15.77:0/205535511 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba80db5c0 con 0x7f9ba8054120 2024-03-06T07:22:09.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9baf7fe640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba803e830 0x7f9ba804eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:09.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7fb82dd9b640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb81006b550 0x7fb81006da10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb830064ac0 tx=0x7fb83006d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac0089d0 con 0x7fcfac035740 2024-03-06T07:22:09.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f0ebdd9b640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec0161ed0 0x7f0ec0143740 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0ea800d5c0 tx=0x7f0ea800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9baeffd640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba8054120 0x7f9ba8038ce0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0050b10 tx=0x7f9bb0069a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0c0e4770 con 0x7f4f000039b0 2024-03-06T07:22:09.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.766+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0c0e4d30 con 0x7f4f000039b0 2024-03-06T07:22:09.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c005360 con 0x7f4f000039b0 2024-03-06T07:22:09.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7f4f0b7fe640 1 -- 172.21.15.77:0/944189184 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04007d50 con 0x7f4f000039b0 2024-03-06T07:22:09.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7f4f0b7fe640 1 -- 172.21.15.77:0/944189184 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04004df0 con 0x7f4f000039b0 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.765+0000 7fc516ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc51017c760 0x7fc51004ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7fc5177fe640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc51017c760 0x7fc51004ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7fc5177fe640 1 -- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51004bf50 msgr2=0x7fc51004ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7fc5177fe640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51004bf50 0x7fc51004ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7fc5177fe640 1 -- 172.21.15.77:0/3241497824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51004dbf0 con 0x7fc51004d3d0 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7fc516ffd640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc51017c760 0x7fc51004ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7fc5177fe640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc51004d3d0 0x7fc5101892d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00c6f0 tx=0x7fc50c000fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7fc5167fc640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51004bf50 0x7fc51004ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9b9bfff640 1 -- 172.21.15.77:0/205535511 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb00bc070 con 0x7f9ba8054120 2024-03-06T07:22:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.767+0000 7f9bb5de9640 1 -- 172.21.15.77:0/205535511 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba80db860 con 0x7f9ba8054120 2024-03-06T07:22:09.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7f9b9bfff640 1 -- 172.21.15.77:0/205535511 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb00b5b00 con 0x7f9ba8054120 2024-03-06T07:22:09.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7f9bb5de9640 1 -- 172.21.15.77:0/205535511 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba80dbe20 con 0x7f9ba8054120 2024-03-06T07:22:09.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.768+0000 7f9b9bfff640 1 -- 172.21.15.77:0/205535511 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb00b0040 con 0x7f9ba8054120 2024-03-06T07:22:09.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.769+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b8005290 con 0x7fd1c8050210 2024-03-06T07:22:09.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.769+0000 7f9b99ffb640 1 -- 172.21.15.77:0/205535511 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba80057d0 con 0x7f9ba8054120 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.769+0000 7f4f0b7fe640 1 -- 172.21.15.77:0/944189184 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f04007070 con 0x7f4f000039b0 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7fcf9ffff640 1 -- 172.21.15.77:0/31179068 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfb40853c0 con 0x7fcfac035740 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7f9b9bfff640 1 -- 172.21.15.77:0/205535511 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bb00c7060 con 0x7f9ba8054120 2024-03-06T07:22:09.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7fd1c67fc640 1 -- 172.21.15.77:0/3579956146 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1cc038020 con 0x7fd1c8050210 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7f4f0b7fe640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ef406b500 0x7f4ef406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7f4f0b7fe640 1 -- 172.21.15.77:0/944189184 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f040abf40 con 0x7f4f000039b0 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7f9b9bfff640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b9406b500 0x7f9b9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7f9baf7fe640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b9406b500 0x7f9b9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7f4f127fc640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ef406b500 0x7f4ef406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf9806b550 0x7fcf9806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7fcf9ffff640 1 -- 172.21.15.77:0/31179068 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfb40f36b0 con 0x7fcfac035740 2024-03-06T07:22:09.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.770+0000 7fd1c67fc640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1b406b520 0x7fd1b406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.771+0000 7fcfb37fe640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf9806b550 0x7fcf9806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.771+0000 7fd1c67fc640 1 -- 172.21.15.77:0/3579956146 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1cc0bd040 con 0x7fd1c8050210 2024-03-06T07:22:09.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.771+0000 7fd1d5315640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1b406b520 0x7fd1b406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.771+0000 7f4f127fc640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ef406b500 0x7f4ef406d9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0e54c0 tx=0x7f4efc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.771+0000 7fcfb37fe640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf9806b550 0x7fcf9806da10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0037000 tx=0x7fcfa0032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.771+0000 7fd1d5315640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1b406b520 0x7fd1b406d9e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80e32f0 tx=0x7fd1c0009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.771+0000 7f9baf7fe640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b9406b500 0x7f9b9406d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4000f60 tx=0x7f9ba4007cf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.772+0000 7f0eb6ffd640 1 -- 172.21.15.77:0/632820896 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80125f0 con 0x7f0ec0161ed0 2024-03-06T07:22:09.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.772+0000 7f0eb6ffd640 1 -- 172.21.15.77:0/632820896 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea8019070 con 0x7f0ec0161ed0 2024-03-06T07:22:09.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.772+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec0150610 con 0x7f0ec0161ed0 2024-03-06T07:22:09.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.772+0000 7fb81effd640 1 -- 172.21.15.77:0/1058417837 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb82007a200 con 0x7fb82804fa80 2024-03-06T07:22:09.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.772+0000 7f9b9bfff640 1 -- 172.21.15.77:0/205535511 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bb00ac690 con 0x7f9ba8054120 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.773+0000 7f0eb6ffd640 1 -- 172.21.15.77:0/632820896 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80129d0 con 0x7f0ec0161ed0 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.773+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec0150bd0 con 0x7f0ec0161ed0 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.773+0000 7fcf9ffff640 1 -- 172.21.15.77:0/31179068 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfb408c070 con 0x7fcfac035740 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.773+0000 7f4f0b7fe640 1 -- 172.21.15.77:0/944189184 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f0407a970 con 0x7f4f000039b0 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.773+0000 7fc4fffff640 1 -- 172.21.15.77:0/3241497824 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c007d70 con 0x7fc51004d3d0 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.773+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51004ded0 con 0x7fc51004d3d0 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.773+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51004e490 con 0x7fc51004d3d0 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.774+0000 7f0eb6ffd640 1 -- 172.21.15.77:0/632820896 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ea8012b70 con 0x7f0ec0161ed0 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.774+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec0150f00 con 0x7f0ec0161ed0 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.774+0000 7f0eb6ffd640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ea006b4d0 0x7f0ea006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.774+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5100057d0 con 0x7fc51004d3d0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.775+0000 7f0ebe59c640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ea006b4d0 0x7f0ea006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.775+0000 7f0eb6ffd640 1 -- 172.21.15.77:0/632820896 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ea808e170 con 0x7f0ec0161ed0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.775+0000 7f0ebe59c640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ea006b4d0 0x7f0ea006d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8035000 tx=0x7f0eb8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.777+0000 7f9b9bfff640 1 -- 172.21.15.77:0/205535511 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bb00f5ae0 con 0x7f9ba8054120 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.777+0000 7fc4fffff640 1 -- 172.21.15.77:0/3241497824 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c030030 con 0x7fc51004d3d0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.777+0000 7fc4fffff640 1 -- 172.21.15.77:0/3241497824 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c001cb0 con 0x7fc51004d3d0 2024-03-06T07:22:09.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.778+0000 7fd1c67fc640 1 -- 172.21.15.77:0/3579956146 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1cc062020 con 0x7fd1c8050210 2024-03-06T07:22:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.779+0000 7fc4fffff640 1 -- 172.21.15.77:0/3241497824 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc50c007070 con 0x7fc51004d3d0 2024-03-06T07:22:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.779+0000 7fc4fffff640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc50406b500 0x7fc50406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.780+0000 7fc4fffff640 1 -- 172.21.15.77:0/3241497824 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc50c0ab070 con 0x7fc51004d3d0 2024-03-06T07:22:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.780+0000 7fc516ffd640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc50406b500 0x7fc50406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.781+0000 7fc516ffd640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc50406b500 0x7fc50406d9c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc518062b90 tx=0x7fc51808cf90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.782+0000 7fc4fffff640 1 -- 172.21.15.77:0/3241497824 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc50c040e90 con 0x7fc51004d3d0 2024-03-06T07:22:09.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:09.783+0000 7f0eb6ffd640 1 -- 172.21.15.77:0/632820896 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ea8058bc0 con 0x7f0ec0161ed0 2024-03-06T07:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.118+0000 7f47599f6640 1 -- 172.21.15.77:0/2533402847 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4730005580 con 0x7f475415e540 2024-03-06T07:22:10.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.119+0000 7f4743fff640 1 -- 172.21.15.77:0/2533402847 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4748058e70 con 0x7f475415e540 2024-03-06T07:22:10.121 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:10.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 -- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f473806b5a0 msgr2=0x7f473806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f473806b5a0 0x7f473806da60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4744009e50 tx=0x7f4744009190 comp rx=0 tx=0).stop 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 -- 172.21.15.77:0/2533402847 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f475415e540 msgr2=0x7f475414a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f475415e540 0x7f475414a890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f474800b3e0 tx=0x7f474800b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 -- 172.21.15.77:0/2533402847 shutdown_connections 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f475415e540 0x7f475414a890 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47540b4c00 0x7f475414a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f473806b5a0 0x7f473806da60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 --2- 172.21.15.77:0/2533402847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47540b28a0 0x7f4754151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.122+0000 7f4741ffb640 1 -- 172.21.15.77:0/2533402847 >> 172.21.15.77:0/2533402847 conn(0x7f475409a9f0 msgr2=0x7f475406f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.123+0000 7f4741ffb640 1 -- 172.21.15.77:0/2533402847 shutdown_connections 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.123+0000 7f4741ffb640 1 -- 172.21.15.77:0/2533402847 wait complete. 2024-03-06T07:22:10.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-03-06T07:22:10.141 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.156+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4f0c170d40 con 0x7f4f000039b0 2024-03-06T07:22:10.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.156+0000 7f4f0b7fe640 1 -- 172.21.15.77:0/944189184 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4f0407ea10 con 0x7f4f000039b0 2024-03-06T07:22:10.158 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.158+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ef406b500 msgr2=0x7f4ef406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.158+0000 7f4f193ac640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ef406b500 0x7f4ef406d9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0e54c0 tx=0x7f4efc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.158+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000039b0 msgr2=0x7f4f0c186310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.158+0000 7f4f193ac640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000039b0 0x7f4f0c186310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f0402c4e0 tx=0x7f4f04001970 comp rx=0 tx=0).stop 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.159+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 shutdown_connections 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.159+0000 7f4f193ac640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ef406b500 0x7f4ef406d9c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.159+0000 7f4f193ac640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f0c0e3d10 0x7f4f0c0e2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.159+0000 7f4f193ac640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f0c0e5e30 0x7f4f0c0e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.159+0000 7f4f193ac640 1 --2- 172.21.15.77:0/944189184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f000039b0 0x7f4f0c186310 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.159+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 >> 172.21.15.77:0/944189184 conn(0x7f4f0c009fa0 msgr2=0x7f4f0c007350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.159+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 shutdown_connections 2024-03-06T07:22:10.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.159+0000 7f4f193ac640 1 -- 172.21.15.77:0/944189184 wait complete. 2024-03-06T07:22:10.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2024-03-06T07:22:10.172 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.184+0000 7fb81cff9640 1 -- 172.21.15.77:0/1058417837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb808005850 con 0x7fb82804fa80 2024-03-06T07:22:10.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.185+0000 7fb81effd640 1 -- 172.21.15.77:0/1058417837 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb82002fe20 con 0x7fb82804fa80 2024-03-06T07:22:10.187 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.187+0000 7fb81cff9640 1 -- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb81006b550 msgr2=0x7fb81006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.187+0000 7fb81cff9640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb81006b550 0x7fb81006da10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb830064ac0 tx=0x7fb83006d040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.187+0000 7fb81cff9640 1 -- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82804fa80 msgr2=0x7fb8280d8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.187+0000 7fb81cff9640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82804fa80 0x7fb8280d8cd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb820000c00 tx=0x7fb8200046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fb81cff9640 1 -- 172.21.15.77:0/1058417837 shutdown_connections 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fb81cff9640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8280df410 0x7fb8280df7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fb81cff9640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb828054260 0x7fb82804b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fb81cff9640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb81006b550 0x7fb81006da10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fb81cff9640 1 --2- 172.21.15.77:0/1058417837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82804fa80 0x7fb8280d8cd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fb81cff9640 1 -- 172.21.15.77:0/1058417837 >> 172.21.15.77:0/1058417837 conn(0x7fb828009fa0 msgr2=0x7fb8280dc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fb81cff9640 1 -- 172.21.15.77:0/1058417837 shutdown_connections 2024-03-06T07:22:10.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fb81cff9640 1 -- 172.21.15.77:0/1058417837 wait complete. 2024-03-06T07:22:10.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fcfac005390 con 0x7fcfac035740 2024-03-06T07:22:10.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.188+0000 7fcf9ffff640 1 -- 172.21.15.77:0/31179068 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcfb40c25a0 con 0x7fcfac035740 2024-03-06T07:22:10.192 INFO:teuthology.orchestra.run.smithi077.stdout:55834574853 2024-03-06T07:22:10.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.190+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf9806b550 msgr2=0x7fcf9806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.190+0000 7fcfb8f75640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf9806b550 0x7fcf9806da10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0037000 tx=0x7fcfa0032000 comp rx=0 tx=0).stop 2024-03-06T07:22:10.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.190+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac035740 msgr2=0x7fcfac035bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.190+0000 7fcfb8f75640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac035740 0x7fcfac035bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40686a0 tx=0x7fcfb40683c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.191+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 shutdown_connections 2024-03-06T07:22:10.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.191+0000 7fcfb8f75640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfac035740 0x7fcfac035bf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.191+0000 7fcfb8f75640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfac04ef50 0x7fcfac04f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.191+0000 7fcfb8f75640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf9806b550 0x7fcf9806da10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.191+0000 7fcfb8f75640 1 --2- 172.21.15.77:0/31179068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfac032da0 0x7fcfac196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.191+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 >> 172.21.15.77:0/31179068 conn(0x7fcfac009fb0 msgr2=0x7fcfac0369b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.191+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 shutdown_connections 2024-03-06T07:22:10.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.191+0000 7fcfb8f75640 1 -- 172.21.15.77:0/31179068 wait complete. 2024-03-06T07:22:10.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.193+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd1b8005580 con 0x7fd1c8050210 2024-03-06T07:22:10.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.193+0000 7f9b99ffb640 1 -- 172.21.15.77:0/205535511 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9ba803d0f0 con 0x7f9ba8054120 2024-03-06T07:22:10.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.194+0000 7f9b9bfff640 1 -- 172.21.15.77:0/205535511 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9bb00f5ae0 con 0x7f9ba8054120 2024-03-06T07:22:10.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.194+0000 7fd1c67fc640 1 -- 172.21.15.77:0/3579956146 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd1cc062020 con 0x7fd1c8050210 2024-03-06T07:22:10.196 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:10.197 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:10.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.195+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0ec006b580 con 0x7f0ec0161ed0 2024-03-06T07:22:10.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.195+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc5100554f0 con 0x7fc51004d3d0 2024-03-06T07:22:10.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.195+0000 7fc4fffff640 1 -- 172.21.15.77:0/3241497824 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc50c079e50 con 0x7fc51004d3d0 2024-03-06T07:22:10.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.195+0000 7f0eb6ffd640 1 -- 172.21.15.77:0/632820896 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0ea80584c0 con 0x7f0ec0161ed0 2024-03-06T07:22:10.198 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:10.198 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2024-03-06T07:22:10.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1b406b520 msgr2=0x7fd1b406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1b406b520 0x7fd1b406d9e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80e32f0 tx=0x7fd1c0009110 comp rx=0 tx=0).stop 2024-03-06T07:22:10.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c8050210 msgr2=0x7fd1c804b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c8050210 0x7fd1c804b3e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0090b0 tx=0x7fd1cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7f9b99ffb640 1 -- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b9406b500 msgr2=0x7f9b9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7f9b99ffb640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b9406b500 0x7f9b9406d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4000f60 tx=0x7f9ba4007cf0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 shutdown_connections 2024-03-06T07:22:10.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7f9b99ffb640 1 -- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba8054120 msgr2=0x7f9ba8038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7f9b99ffb640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba8054120 0x7f9ba8038ce0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0050b10 tx=0x7f9bb0069a60 comp rx=0 tx=0).stop 2024-03-06T07:22:10.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1b406b520 0x7fd1b406d9e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1c804e300 0x7fd1c80d4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1c80e4420 0x7fd1c804ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.197+0000 7fd1d7d7a640 1 --2- 172.21.15.77:0/3579956146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c8050210 0x7fd1c804b3e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 >> 172.21.15.77:0/3579956146 conn(0x7fd1c8055bc0 msgr2=0x7fd1c80d8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 shutdown_connections 2024-03-06T07:22:10.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7fd1d7d7a640 1 -- 172.21.15.77:0/3579956146 wait complete. 2024-03-06T07:22:10.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f9b99ffb640 1 -- 172.21.15.77:0/205535511 shutdown_connections 2024-03-06T07:22:10.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f9b99ffb640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9ba80e30b0 0x7f9ba803b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f9b99ffb640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9ba8054120 0x7f9ba8038ce0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f9b99ffb640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b9406b500 0x7f9b9406d9c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f9b99ffb640 1 --2- 172.21.15.77:0/205535511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba803e830 0x7f9ba804eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f9b99ffb640 1 -- 172.21.15.77:0/205535511 >> 172.21.15.77:0/205535511 conn(0x7f9ba80559d0 msgr2=0x7f9ba803c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f9b99ffb640 1 -- 172.21.15.77:0/205535511 shutdown_connections 2024-03-06T07:22:10.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f9b99ffb640 1 -- 172.21.15.77:0/205535511 wait complete. 2024-03-06T07:22:10.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ea006b4d0 msgr2=0x7f0ea006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ea006b4d0 0x7f0ea006d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8035000 tx=0x7f0eb8030000 comp rx=0 tx=0).stop 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec0161ed0 msgr2=0x7f0ec0143740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec0161ed0 0x7f0ec0143740 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0ea800d5c0 tx=0x7f0ea800da80 comp rx=0 tx=0).stop 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 shutdown_connections 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ea006b4d0 0x7f0ea006d990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec0161ed0 0x7f0ec0143740 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ec01627f0 0x7f0ec0133b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 --2- 172.21.15.77:0/632820896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ec00afae0 0x7f0ec013e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 >> 172.21.15.77:0/632820896 conn(0x7f0ec006c0c0 msgr2=0x7f0ec00b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.198+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 shutdown_connections 2024-03-06T07:22:10.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7f0ebf59e640 1 -- 172.21.15.77:0/632820896 wait complete. 2024-03-06T07:22:10.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc50406b500 msgr2=0x7fc50406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc50406b500 0x7fc50406d9c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc518062b90 tx=0x7fc51808cf90 comp rx=0 tx=0).stop 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc51004d3d0 msgr2=0x7fc5101892d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc51004d3d0 0x7fc5101892d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00c6f0 tx=0x7fc50c000fe0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 shutdown_connections 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc50406b500 0x7fc50406d9c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc51004bf50 0x7fc51004ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc51004d3d0 0x7fc5101892d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 --2- 172.21.15.77:0/3241497824 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc51017c760 0x7fc51004ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 >> 172.21.15.77:0/3241497824 conn(0x7fc510055bc0 msgr2=0x7fc5100ea2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 shutdown_connections 2024-03-06T07:22:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.199+0000 7fc517fff640 1 -- 172.21.15.77:0/3241497824 wait complete. 2024-03-06T07:22:10.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-03-06T07:22:10.208 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.210 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-03-06T07:22:10.210 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.216 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-03-06T07:22:10.217 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2024-03-06T07:22:10.217 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-06T07:22:10.218 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-03-06T07:22:10.219 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:10.219 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:22:10.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:10.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.355+0000 7feef5e83640 1 Processor -- start 2024-03-06T07:22:10.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.355+0000 7feef5e83640 1 -- start start 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feef5e83640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef01515e0 0x7feef00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feef5e83640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feef015b430 0x7feef0071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feef5e83640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feef00cd7a0 0x7feef00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feef5e83640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef015b8b0 con 0x7feef015b430 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feef5e83640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef0070a70 con 0x7feef00cd7a0 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feef5e83640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef0070d50 con 0x7feef01515e0 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeeffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feef015b430 0x7feef0071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeeffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feef015b430 0x7feef0071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56182/0 (socket says 172.21.15.77:56182) 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeeffd640 1 -- 172.21.15.77:0/771521528 learned_addr learned my addr 172.21.15.77:0/771521528 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeffff640 1 --2- 172.21.15.77:0/771521528 >> v2:172.21.15.158:3300/0 conn(0x7feef00cd7a0 0x7feef00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeef7fe640 1 --2- 172.21.15.77:0/771521528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef01515e0 0x7feef00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeeffd640 1 -- 172.21.15.77:0/771521528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef01515e0 msgr2=0x7feef00cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeeffd640 1 --2- 172.21.15.77:0/771521528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef01515e0 0x7feef00cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeeffd640 1 -- 172.21.15.77:0/771521528 >> v2:172.21.15.158:3300/0 conn(0x7feef00cd7a0 msgr2=0x7feef00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeeffd640 1 --2- 172.21.15.77:0/771521528 >> v2:172.21.15.158:3300/0 conn(0x7feef00cd7a0 0x7feef00cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.356+0000 7feeeeffd640 1 -- 172.21.15.77:0/771521528 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef0071030 con 0x7feef015b430 2024-03-06T07:22:10.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.357+0000 7feeeeffd640 1 --2- 172.21.15.77:0/771521528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef0071830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7feef00b3d60 tx=0x7feee402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db42e96e9ed6ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.357+0000 7feeee7fc640 1 -- 172.21.15.77:0/771521528 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4035020 con 0x7feef015b430 2024-03-06T07:22:10.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.357+0000 7feeee7fc640 1 -- 172.21.15.77:0/771521528 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee4031870 con 0x7feef015b430 2024-03-06T07:22:10.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.357+0000 7feeee7fc640 1 -- 172.21.15.77:0/771521528 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4031c40 con 0x7feef015b430 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 -- 172.21.15.77:0/771521528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 msgr2=0x7feef0071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 --2- 172.21.15.77:0/771521528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef0071830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7feef00b3d60 tx=0x7feee402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 -- 172.21.15.77:0/771521528 shutdown_connections 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 --2- 172.21.15.77:0/771521528 >> v2:172.21.15.158:3300/0 conn(0x7feef00cd7a0 0x7feef00cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 --2- 172.21.15.77:0/771521528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef0071830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 --2- 172.21.15.77:0/771521528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef01515e0 0x7feef00cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 -- 172.21.15.77:0/771521528 >> 172.21.15.77:0/771521528 conn(0x7feef00cf8a0 msgr2=0x7feef00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 -- 172.21.15.77:0/771521528 shutdown_connections 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.358+0000 7feef5e83640 1 -- 172.21.15.77:0/771521528 wait complete. 2024-03-06T07:22:10.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.359+0000 7feef5e83640 1 Processor -- start 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.359+0000 7feef5e83640 1 -- start start 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feef5e83640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef00cd7a0 0x7feef0163970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feef5e83640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feef01515e0 0x7feef013f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feef5e83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef00c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feef5e83640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef0071d70 con 0x7feef015b430 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feef5e83640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef00ce110 con 0x7feef01515e0 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feef5e83640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef0071ef0 con 0x7feef00cd7a0 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef00c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef00c5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56184/0 (socket says 172.21.15.77:56184) 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 -- 172.21.15.77:0/1026558141 learned_addr learned my addr 172.21.15.77:0/1026558141 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeeffd640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feef01515e0 0x7feef013f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeef7fe640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef00cd7a0 0x7feef0163970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 -- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef00cd7a0 msgr2=0x7feef0163970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef00cd7a0 0x7feef0163970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 -- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feef01515e0 msgr2=0x7feef013f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feef01515e0 0x7feef013f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 -- 172.21.15.77:0/1026558141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00c4dc0 con 0x7feef015b430 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeef7fe640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef00cd7a0 0x7feef0163970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feeeffff640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef00c5e60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feee800ef00 tx=0x7feee800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feed7fff640 1 -- 172.21.15.77:0/1026558141 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8018020 con 0x7feef015b430 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feed7fff640 1 -- 172.21.15.77:0/1026558141 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee800bcc0 con 0x7feef015b430 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef00c50a0 con 0x7feef015b430 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feed7fff640 1 -- 172.21.15.77:0/1026558141 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8014640 con 0x7feef015b430 2024-03-06T07:22:10.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.360+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef00c5940 con 0x7feef015b430 2024-03-06T07:22:10.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.361+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef00c5270 con 0x7feef015b430 2024-03-06T07:22:10.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.362+0000 7feed7fff640 1 -- 172.21.15.77:0/1026558141 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feee8005d10 con 0x7feef015b430 2024-03-06T07:22:10.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.362+0000 7feed7fff640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feedc06b4d0 0x7feedc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.362+0000 7feed7fff640 1 -- 172.21.15.77:0/1026558141 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feee808a740 con 0x7feef015b430 2024-03-06T07:22:10.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.362+0000 7feeef7fe640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feedc06b4d0 0x7feedc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.362+0000 7feeef7fe640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feedc06b4d0 0x7feedc06d990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feee0009af0 tx=0x7feee0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.366+0000 7feed7fff640 1 -- 172.21.15.77:0/1026558141 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feee8022070 con 0x7feef015b430 2024-03-06T07:22:10.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.537+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feef0068840 con 0x7feedc06b4d0 2024-03-06T07:22:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.542+0000 7feed7fff640 1 -- 172.21.15.77:0/1026558141 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7feef0068840 con 0x7feedc06b4d0 2024-03-06T07:22:10.547 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:22:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feedc06b4d0 msgr2=0x7feedc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feedc06b4d0 0x7feedc06d990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feee0009af0 tx=0x7feee0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 msgr2=0x7feef00c5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef00c5e60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feee800ef00 tx=0x7feee800b480 comp rx=0 tx=0).stop 2024-03-06T07:22:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 shutdown_connections 2024-03-06T07:22:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feedc06b4d0 0x7feedc06d990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feef015b430 0x7feef00c5e60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feef01515e0 0x7feef013f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 --2- 172.21.15.77:0/1026558141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feef00cd7a0 0x7feef0163970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 >> 172.21.15.77:0/1026558141 conn(0x7feef00cf8a0 msgr2=0x7feef0150770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.545+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 shutdown_connections 2024-03-06T07:22:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.546+0000 7feef5e83640 1 -- 172.21.15.77:0/1026558141 wait complete. 2024-03-06T07:22:10.558 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:22:09.385710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235860,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738860,"statfs":{"total":773094113280,"available":772852592640,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001783"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004993+0000","last_change":"2024-03-06T07:21:58.964978+0000","last_active":"2024-03-06T07:22:01.004993+0000","last_peered":"2024-03-06T07:22:01.004993+0000","last_clean":"2024-03-06T07:22:01.004993+0000","last_became_active":"2024-03-06T07:21:58.964298+0000","last_became_peered":"2024-03-06T07:21:58.964298+0000","last_unstale":"2024-03-06T07:22:01.004993+0000","last_undegraded":"2024-03-06T07:22:01.004993+0000","last_fullsized":"2024-03-06T07:22:01.004993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:53.179429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942479+0000","last_change":"2024-03-06T07:21:58.960691+0000","last_active":"2024-03-06T07:21:59.942479+0000","last_peered":"2024-03-06T07:21:59.942479+0000","last_clean":"2024-03-06T07:21:59.942479+0000","last_became_active":"2024-03-06T07:21:58.959088+0000","last_became_peered":"2024-03-06T07:21:58.959088+0000","last_unstale":"2024-03-06T07:21:59.942479+0000","last_undegraded":"2024-03-06T07:21:59.942479+0000","last_fullsized":"2024-03-06T07:21:59.942479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549070+0000","last_change":"2024-03-06T07:21:58.953314+0000","last_active":"2024-03-06T07:22:00.549070+0000","last_peered":"2024-03-06T07:22:00.549070+0000","last_clean":"2024-03-06T07:22:00.549070+0000","last_became_active":"2024-03-06T07:21:58.953048+0000","last_became_peered":"2024-03-06T07:21:58.953048+0000","last_unstale":"2024-03-06T07:22:00.549070+0000","last_undegraded":"2024-03-06T07:22:00.549070+0000","last_fullsized":"2024-03-06T07:22:00.549070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004545+0000","last_change":"2024-03-06T07:21:58.965150+0000","last_active":"2024-03-06T07:22:01.004545+0000","last_peered":"2024-03-06T07:22:01.004545+0000","last_clean":"2024-03-06T07:22:01.004545+0000","last_became_active":"2024-03-06T07:21:58.964307+0000","last_became_peered":"2024-03-06T07:21:58.964307+0000","last_unstale":"2024-03-06T07:22:01.004545+0000","last_undegraded":"2024-03-06T07:22:01.004545+0000","last_fullsized":"2024-03-06T07:22:01.004545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004939+0000","last_change":"2024-03-06T07:21:58.957427+0000","last_active":"2024-03-06T07:22:01.004939+0000","last_peered":"2024-03-06T07:22:01.004939+0000","last_clean":"2024-03-06T07:22:01.004939+0000","last_became_active":"2024-03-06T07:21:58.957124+0000","last_became_peered":"2024-03-06T07:21:58.957124+0000","last_unstale":"2024-03-06T07:22:01.004939+0000","last_undegraded":"2024-03-06T07:22:01.004939+0000","last_fullsized":"2024-03-06T07:22:01.004939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:51.617926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004247+0000","last_change":"2024-03-06T07:21:58.969024+0000","last_active":"2024-03-06T07:22:01.004247+0000","last_peered":"2024-03-06T07:22:01.004247+0000","last_clean":"2024-03-06T07:22:01.004247+0000","last_became_active":"2024-03-06T07:21:58.963242+0000","last_became_peered":"2024-03-06T07:21:58.963242+0000","last_unstale":"2024-03-06T07:22:01.004247+0000","last_undegraded":"2024-03-06T07:22:01.004247+0000","last_fullsized":"2024-03-06T07:22:01.004247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004439+0000","last_change":"2024-03-06T07:21:58.958719+0000","last_active":"2024-03-06T07:22:01.004439+0000","last_peered":"2024-03-06T07:22:01.004439+0000","last_clean":"2024-03-06T07:22:01.004439+0000","last_became_active":"2024-03-06T07:21:58.958025+0000","last_became_peered":"2024-03-06T07:21:58.958025+0000","last_unstale":"2024-03-06T07:22:01.004439+0000","last_undegraded":"2024-03-06T07:22:01.004439+0000","last_fullsized":"2024-03-06T07:22:01.004439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004500+0000","last_change":"2024-03-06T07:21:58.963329+0000","last_active":"2024-03-06T07:22:01.004500+0000","last_peered":"2024-03-06T07:22:01.004500+0000","last_clean":"2024-03-06T07:22:01.004500+0000","last_became_active":"2024-03-06T07:21:58.963004+0000","last_became_peered":"2024-03-06T07:21:58.963004+0000","last_unstale":"2024-03-06T07:22:01.004500+0000","last_undegraded":"2024-03-06T07:22:01.004500+0000","last_fullsized":"2024-03-06T07:22:01.004500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005042+0000","last_change":"2024-03-06T07:21:58.969451+0000","last_active":"2024-03-06T07:22:01.005042+0000","last_peered":"2024-03-06T07:22:01.005042+0000","last_clean":"2024-03-06T07:22:01.005042+0000","last_became_active":"2024-03-06T07:21:58.969086+0000","last_became_peered":"2024-03-06T07:21:58.969086+0000","last_unstale":"2024-03-06T07:22:01.005042+0000","last_undegraded":"2024-03-06T07:22:01.005042+0000","last_fullsized":"2024-03-06T07:22:01.005042+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:00.578026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944576+0000","last_change":"2024-03-06T07:21:58.966581+0000","last_active":"2024-03-06T07:21:59.944576+0000","last_peered":"2024-03-06T07:21:59.944576+0000","last_clean":"2024-03-06T07:21:59.944576+0000","last_became_active":"2024-03-06T07:21:58.966309+0000","last_became_peered":"2024-03-06T07:21:58.966309+0000","last_unstale":"2024-03-06T07:21:59.944576+0000","last_undegraded":"2024-03-06T07:21:59.944576+0000","last_fullsized":"2024-03-06T07:21:59.944576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944493+0000","last_change":"2024-03-06T07:21:58.968147+0000","last_active":"2024-03-06T07:21:59.944493+0000","last_peered":"2024-03-06T07:21:59.944493+0000","last_clean":"2024-03-06T07:21:59.944493+0000","last_became_active":"2024-03-06T07:21:58.967870+0000","last_became_peered":"2024-03-06T07:21:58.967870+0000","last_unstale":"2024-03-06T07:21:59.944493+0000","last_undegraded":"2024-03-06T07:21:59.944493+0000","last_fullsized":"2024-03-06T07:21:59.944493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943944+0000","last_change":"2024-03-06T07:21:58.950183+0000","last_active":"2024-03-06T07:21:59.943944+0000","last_peered":"2024-03-06T07:21:59.943944+0000","last_clean":"2024-03-06T07:21:59.943944+0000","last_became_active":"2024-03-06T07:21:58.949892+0000","last_became_peered":"2024-03-06T07:21:58.949892+0000","last_unstale":"2024-03-06T07:21:59.943944+0000","last_undegraded":"2024-03-06T07:21:59.943944+0000","last_fullsized":"2024-03-06T07:21:59.943944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:41.545324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004776+0000","last_change":"2024-03-06T07:21:58.963200+0000","last_active":"2024-03-06T07:22:01.004776+0000","last_peered":"2024-03-06T07:22:01.004776+0000","last_clean":"2024-03-06T07:22:01.004776+0000","last_became_active":"2024-03-06T07:21:58.962833+0000","last_became_peered":"2024-03-06T07:21:58.962833+0000","last_unstale":"2024-03-06T07:22:01.004776+0000","last_undegraded":"2024-03-06T07:22:01.004776+0000","last_fullsized":"2024-03-06T07:22:01.004776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548832+0000","last_change":"2024-03-06T07:21:58.952781+0000","last_active":"2024-03-06T07:22:00.548832+0000","last_peered":"2024-03-06T07:22:00.548832+0000","last_clean":"2024-03-06T07:22:00.548832+0000","last_became_active":"2024-03-06T07:21:58.952504+0000","last_became_peered":"2024-03-06T07:21:58.952504+0000","last_unstale":"2024-03-06T07:22:00.548832+0000","last_undegraded":"2024-03-06T07:22:00.548832+0000","last_fullsized":"2024-03-06T07:22:00.548832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:25.614483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942188+0000","last_change":"2024-03-06T07:21:58.950082+0000","last_active":"2024-03-06T07:21:59.942188+0000","last_peered":"2024-03-06T07:21:59.942188+0000","last_clean":"2024-03-06T07:21:59.942188+0000","last_became_active":"2024-03-06T07:21:58.949595+0000","last_became_peered":"2024-03-06T07:21:58.949595+0000","last_unstale":"2024-03-06T07:21:59.942188+0000","last_undegraded":"2024-03-06T07:21:59.942188+0000","last_fullsized":"2024-03-06T07:21:59.942188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004141+0000","last_change":"2024-03-06T07:21:58.961225+0000","last_active":"2024-03-06T07:22:01.004141+0000","last_peered":"2024-03-06T07:22:01.004141+0000","last_clean":"2024-03-06T07:22:01.004141+0000","last_became_active":"2024-03-06T07:21:58.960902+0000","last_became_peered":"2024-03-06T07:21:58.960902+0000","last_unstale":"2024-03-06T07:22:01.004141+0000","last_undegraded":"2024-03-06T07:22:01.004141+0000","last_fullsized":"2024-03-06T07:22:01.004141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004963+0000","last_change":"2024-03-06T07:21:58.970233+0000","last_active":"2024-03-06T07:22:01.004963+0000","last_peered":"2024-03-06T07:22:01.004963+0000","last_clean":"2024-03-06T07:22:01.004963+0000","last_became_active":"2024-03-06T07:21:58.969346+0000","last_became_peered":"2024-03-06T07:21:58.969346+0000","last_unstale":"2024-03-06T07:22:01.004963+0000","last_undegraded":"2024-03-06T07:22:01.004963+0000","last_fullsized":"2024-03-06T07:22:01.004963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:42.427321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983310+0000","last_change":"2024-03-06T07:21:58.948614+0000","last_active":"2024-03-06T07:21:59.983310+0000","last_peered":"2024-03-06T07:21:59.983310+0000","last_clean":"2024-03-06T07:21:59.983310+0000","last_became_active":"2024-03-06T07:21:58.948333+0000","last_became_peered":"2024-03-06T07:21:58.948333+0000","last_unstale":"2024-03-06T07:21:59.983310+0000","last_undegraded":"2024-03-06T07:21:59.983310+0000","last_fullsized":"2024-03-06T07:21:59.983310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:42.473136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005274+0000","last_change":"2024-03-06T07:21:58.957566+0000","last_active":"2024-03-06T07:22:01.005274+0000","last_peered":"2024-03-06T07:22:01.005274+0000","last_clean":"2024-03-06T07:22:01.005274+0000","last_became_active":"2024-03-06T07:21:58.957272+0000","last_became_peered":"2024-03-06T07:21:58.957272+0000","last_unstale":"2024-03-06T07:22:01.005274+0000","last_undegraded":"2024-03-06T07:22:01.005274+0000","last_fullsized":"2024-03-06T07:22:01.005274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:13.279685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944669+0000","last_change":"2024-03-06T07:21:58.948431+0000","last_active":"2024-03-06T07:21:59.944669+0000","last_peered":"2024-03-06T07:21:59.944669+0000","last_clean":"2024-03-06T07:21:59.944669+0000","last_became_active":"2024-03-06T07:21:58.947689+0000","last_became_peered":"2024-03-06T07:21:58.947689+0000","last_unstale":"2024-03-06T07:21:59.944669+0000","last_undegraded":"2024-03-06T07:21:59.944669+0000","last_fullsized":"2024-03-06T07:21:59.944669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548975+0000","last_change":"2024-03-06T07:21:58.965611+0000","last_active":"2024-03-06T07:22:00.548975+0000","last_peered":"2024-03-06T07:22:00.548975+0000","last_clean":"2024-03-06T07:22:00.548975+0000","last_became_active":"2024-03-06T07:21:58.965261+0000","last_became_peered":"2024-03-06T07:21:58.965261+0000","last_unstale":"2024-03-06T07:22:00.548975+0000","last_undegraded":"2024-03-06T07:22:00.548975+0000","last_fullsized":"2024-03-06T07:22:00.548975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004252+0000","last_change":"2024-03-06T07:21:58.969256+0000","last_active":"2024-03-06T07:22:01.004252+0000","last_peered":"2024-03-06T07:22:01.004252+0000","last_clean":"2024-03-06T07:22:01.004252+0000","last_became_active":"2024-03-06T07:21:58.968670+0000","last_became_peered":"2024-03-06T07:21:58.968670+0000","last_unstale":"2024-03-06T07:22:01.004252+0000","last_undegraded":"2024-03-06T07:22:01.004252+0000","last_fullsized":"2024-03-06T07:22:01.004252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:25.392050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004556+0000","last_change":"2024-03-06T07:21:58.962061+0000","last_active":"2024-03-06T07:22:01.004556+0000","last_peered":"2024-03-06T07:22:01.004556+0000","last_clean":"2024-03-06T07:22:01.004556+0000","last_became_active":"2024-03-06T07:21:58.961754+0000","last_became_peered":"2024-03-06T07:21:58.961754+0000","last_unstale":"2024-03-06T07:22:01.004556+0000","last_undegraded":"2024-03-06T07:22:01.004556+0000","last_fullsized":"2024-03-06T07:22:01.004556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004201+0000","last_change":"2024-03-06T07:21:58.961280+0000","last_active":"2024-03-06T07:22:01.004201+0000","last_peered":"2024-03-06T07:22:01.004201+0000","last_clean":"2024-03-06T07:22:01.004201+0000","last_became_active":"2024-03-06T07:21:58.960288+0000","last_became_peered":"2024-03-06T07:21:58.960288+0000","last_unstale":"2024-03-06T07:22:01.004201+0000","last_undegraded":"2024-03-06T07:22:01.004201+0000","last_fullsized":"2024-03-06T07:22:01.004201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005074+0000","last_change":"2024-03-06T07:21:58.962545+0000","last_active":"2024-03-06T07:22:01.005074+0000","last_peered":"2024-03-06T07:22:01.005074+0000","last_clean":"2024-03-06T07:22:01.005074+0000","last_became_active":"2024-03-06T07:21:58.962037+0000","last_became_peered":"2024-03-06T07:21:58.962037+0000","last_unstale":"2024-03-06T07:22:01.005074+0000","last_undegraded":"2024-03-06T07:22:01.005074+0000","last_fullsized":"2024-03-06T07:22:01.005074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944648+0000","last_change":"2024-03-06T07:21:58.959744+0000","last_active":"2024-03-06T07:21:59.944648+0000","last_peered":"2024-03-06T07:21:59.944648+0000","last_clean":"2024-03-06T07:21:59.944648+0000","last_became_active":"2024-03-06T07:21:58.959459+0000","last_became_peered":"2024-03-06T07:21:58.959459+0000","last_unstale":"2024-03-06T07:21:59.944648+0000","last_undegraded":"2024-03-06T07:21:59.944648+0000","last_fullsized":"2024-03-06T07:21:59.944648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942254+0000","last_change":"2024-03-06T07:21:58.950538+0000","last_active":"2024-03-06T07:21:59.942254+0000","last_peered":"2024-03-06T07:21:59.942254+0000","last_clean":"2024-03-06T07:21:59.942254+0000","last_became_active":"2024-03-06T07:21:58.950225+0000","last_became_peered":"2024-03-06T07:21:58.950225+0000","last_unstale":"2024-03-06T07:21:59.942254+0000","last_undegraded":"2024-03-06T07:21:59.942254+0000","last_fullsized":"2024-03-06T07:21:59.942254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004948+0000","last_change":"2024-03-06T07:21:58.961060+0000","last_active":"2024-03-06T07:22:01.004948+0000","last_peered":"2024-03-06T07:22:01.004948+0000","last_clean":"2024-03-06T07:22:01.004948+0000","last_became_active":"2024-03-06T07:21:58.960291+0000","last_became_peered":"2024-03-06T07:21:58.960291+0000","last_unstale":"2024-03-06T07:22:01.004948+0000","last_undegraded":"2024-03-06T07:22:01.004948+0000","last_fullsized":"2024-03-06T07:22:01.004948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548986+0000","last_change":"2024-03-06T07:21:58.957460+0000","last_active":"2024-03-06T07:22:00.548986+0000","last_peered":"2024-03-06T07:22:00.548986+0000","last_clean":"2024-03-06T07:22:00.548986+0000","last_became_active":"2024-03-06T07:21:58.957194+0000","last_became_peered":"2024-03-06T07:21:58.957194+0000","last_unstale":"2024-03-06T07:22:00.548986+0000","last_undegraded":"2024-03-06T07:22:00.548986+0000","last_fullsized":"2024-03-06T07:22:00.548986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004388+0000","last_change":"2024-03-06T07:21:58.971347+0000","last_active":"2024-03-06T07:22:01.004388+0000","last_peered":"2024-03-06T07:22:01.004388+0000","last_clean":"2024-03-06T07:22:01.004388+0000","last_became_active":"2024-03-06T07:21:58.970575+0000","last_became_peered":"2024-03-06T07:21:58.970575+0000","last_unstale":"2024-03-06T07:22:01.004388+0000","last_undegraded":"2024-03-06T07:22:01.004388+0000","last_fullsized":"2024-03-06T07:22:01.004388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983258+0000","last_change":"2024-03-06T07:21:58.964710+0000","last_active":"2024-03-06T07:21:59.983258+0000","last_peered":"2024-03-06T07:21:59.983258+0000","last_clean":"2024-03-06T07:21:59.983258+0000","last_became_active":"2024-03-06T07:21:58.964426+0000","last_became_peered":"2024-03-06T07:21:58.964426+0000","last_unstale":"2024-03-06T07:21:59.983258+0000","last_undegraded":"2024-03-06T07:21:59.983258+0000","last_fullsized":"2024-03-06T07:21:59.983258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:54.202847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943842+0000","last_change":"2024-03-06T07:21:58.958320+0000","last_active":"2024-03-06T07:21:59.943842+0000","last_peered":"2024-03-06T07:21:59.943842+0000","last_clean":"2024-03-06T07:21:59.943842+0000","last_became_active":"2024-03-06T07:21:58.958023+0000","last_became_peered":"2024-03-06T07:21:58.958023+0000","last_unstale":"2024-03-06T07:21:59.943842+0000","last_undegraded":"2024-03-06T07:21:59.943842+0000","last_fullsized":"2024-03-06T07:21:59.943842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943853+0000","last_change":"2024-03-06T07:21:58.949671+0000","last_active":"2024-03-06T07:21:59.943853+0000","last_peered":"2024-03-06T07:21:59.943853+0000","last_clean":"2024-03-06T07:21:59.943853+0000","last_became_active":"2024-03-06T07:21:58.949364+0000","last_became_peered":"2024-03-06T07:21:58.949364+0000","last_unstale":"2024-03-06T07:21:59.943853+0000","last_undegraded":"2024-03-06T07:21:59.943853+0000","last_fullsized":"2024-03-06T07:21:59.943853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:54.826119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004867+0000","last_change":"2024-03-06T07:21:57.960062+0000","last_active":"2024-03-06T07:22:01.004867+0000","last_peered":"2024-03-06T07:22:01.004867+0000","last_clean":"2024-03-06T07:22:01.004867+0000","last_became_active":"2024-03-06T07:21:57.959790+0000","last_became_peered":"2024-03-06T07:21:57.959790+0000","last_unstale":"2024-03-06T07:22:01.004867+0000","last_undegraded":"2024-03-06T07:22:01.004867+0000","last_fullsized":"2024-03-06T07:22:01.004867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:02.949716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549132+0000","last_change":"2024-03-06T07:21:58.958514+0000","last_active":"2024-03-06T07:22:00.549132+0000","last_peered":"2024-03-06T07:22:00.549132+0000","last_clean":"2024-03-06T07:22:00.549132+0000","last_became_active":"2024-03-06T07:21:58.958247+0000","last_became_peered":"2024-03-06T07:21:58.958247+0000","last_unstale":"2024-03-06T07:22:00.549132+0000","last_undegraded":"2024-03-06T07:22:00.549132+0000","last_fullsized":"2024-03-06T07:22:00.549132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944544+0000","last_change":"2024-03-06T07:21:57.949106+0000","last_active":"2024-03-06T07:21:59.944544+0000","last_peered":"2024-03-06T07:21:59.944544+0000","last_clean":"2024-03-06T07:21:59.944544+0000","last_became_active":"2024-03-06T07:21:57.948807+0000","last_became_peered":"2024-03-06T07:21:57.948807+0000","last_unstale":"2024-03-06T07:21:59.944544+0000","last_undegraded":"2024-03-06T07:21:59.944544+0000","last_fullsized":"2024-03-06T07:21:59.944544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004531+0000","last_change":"2024-03-06T07:21:57.962231+0000","last_active":"2024-03-06T07:22:01.004531+0000","last_peered":"2024-03-06T07:22:01.004531+0000","last_clean":"2024-03-06T07:22:01.004531+0000","last_became_active":"2024-03-06T07:21:57.961979+0000","last_became_peered":"2024-03-06T07:21:57.961979+0000","last_unstale":"2024-03-06T07:22:01.004531+0000","last_undegraded":"2024-03-06T07:22:01.004531+0000","last_fullsized":"2024-03-06T07:22:01.004531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004545+0000","last_change":"2024-03-06T07:21:58.962490+0000","last_active":"2024-03-06T07:22:01.004545+0000","last_peered":"2024-03-06T07:22:01.004545+0000","last_clean":"2024-03-06T07:22:01.004545+0000","last_became_active":"2024-03-06T07:21:58.960981+0000","last_became_peered":"2024-03-06T07:21:58.960981+0000","last_unstale":"2024-03-06T07:22:01.004545+0000","last_undegraded":"2024-03-06T07:22:01.004545+0000","last_fullsized":"2024-03-06T07:22:01.004545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004198+0000","last_change":"2024-03-06T07:21:57.961193+0000","last_active":"2024-03-06T07:22:01.004198+0000","last_peered":"2024-03-06T07:22:01.004198+0000","last_clean":"2024-03-06T07:22:01.004198+0000","last_became_active":"2024-03-06T07:21:57.960916+0000","last_became_peered":"2024-03-06T07:21:57.960916+0000","last_unstale":"2024-03-06T07:22:01.004198+0000","last_undegraded":"2024-03-06T07:22:01.004198+0000","last_fullsized":"2024-03-06T07:22:01.004198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942173+0000","last_change":"2024-03-06T07:21:58.951489+0000","last_active":"2024-03-06T07:21:59.942173+0000","last_peered":"2024-03-06T07:21:59.942173+0000","last_clean":"2024-03-06T07:21:59.942173+0000","last_became_active":"2024-03-06T07:21:58.951202+0000","last_became_peered":"2024-03-06T07:21:58.951202+0000","last_unstale":"2024-03-06T07:21:59.942173+0000","last_undegraded":"2024-03-06T07:21:59.942173+0000","last_fullsized":"2024-03-06T07:21:59.942173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944528+0000","last_change":"2024-03-06T07:21:57.959918+0000","last_active":"2024-03-06T07:21:59.944528+0000","last_peered":"2024-03-06T07:21:59.944528+0000","last_clean":"2024-03-06T07:21:59.944528+0000","last_became_active":"2024-03-06T07:21:57.959611+0000","last_became_peered":"2024-03-06T07:21:57.959611+0000","last_unstale":"2024-03-06T07:21:59.944528+0000","last_undegraded":"2024-03-06T07:21:59.944528+0000","last_fullsized":"2024-03-06T07:21:59.944528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004566+0000","last_change":"2024-03-06T07:21:58.970643+0000","last_active":"2024-03-06T07:22:01.004566+0000","last_peered":"2024-03-06T07:22:01.004566+0000","last_clean":"2024-03-06T07:22:01.004566+0000","last_became_active":"2024-03-06T07:21:58.970039+0000","last_became_peered":"2024-03-06T07:21:58.970039+0000","last_unstale":"2024-03-06T07:22:01.004566+0000","last_undegraded":"2024-03-06T07:22:01.004566+0000","last_fullsized":"2024-03-06T07:22:01.004566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004616+0000","last_change":"2024-03-06T07:21:57.962638+0000","last_active":"2024-03-06T07:22:01.004616+0000","last_peered":"2024-03-06T07:22:01.004616+0000","last_clean":"2024-03-06T07:22:01.004616+0000","last_became_active":"2024-03-06T07:21:57.962348+0000","last_became_peered":"2024-03-06T07:21:57.962348+0000","last_unstale":"2024-03-06T07:22:01.004616+0000","last_undegraded":"2024-03-06T07:22:01.004616+0000","last_fullsized":"2024-03-06T07:22:01.004616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549050+0000","last_change":"2024-03-06T07:21:58.962712+0000","last_active":"2024-03-06T07:22:00.549050+0000","last_peered":"2024-03-06T07:22:00.549050+0000","last_clean":"2024-03-06T07:22:00.549050+0000","last_became_active":"2024-03-06T07:21:58.962398+0000","last_became_peered":"2024-03-06T07:21:58.962398+0000","last_unstale":"2024-03-06T07:22:00.549050+0000","last_undegraded":"2024-03-06T07:22:00.549050+0000","last_fullsized":"2024-03-06T07:22:00.549050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942294+0000","last_change":"2024-03-06T07:21:57.958869+0000","last_active":"2024-03-06T07:21:59.942294+0000","last_peered":"2024-03-06T07:21:59.942294+0000","last_clean":"2024-03-06T07:21:59.942294+0000","last_became_active":"2024-03-06T07:21:57.958605+0000","last_became_peered":"2024-03-06T07:21:57.958605+0000","last_unstale":"2024-03-06T07:21:59.942294+0000","last_undegraded":"2024-03-06T07:21:59.942294+0000","last_fullsized":"2024-03-06T07:21:59.942294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944175+0000","last_change":"2024-03-06T07:21:58.960482+0000","last_active":"2024-03-06T07:21:59.944175+0000","last_peered":"2024-03-06T07:21:59.944175+0000","last_clean":"2024-03-06T07:21:59.944175+0000","last_became_active":"2024-03-06T07:21:58.960187+0000","last_became_peered":"2024-03-06T07:21:58.960187+0000","last_unstale":"2024-03-06T07:21:59.944175+0000","last_undegraded":"2024-03-06T07:21:59.944175+0000","last_fullsized":"2024-03-06T07:21:59.944175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:54.202847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944275+0000","last_change":"2024-03-06T07:21:57.963435+0000","last_active":"2024-03-06T07:21:59.944275+0000","last_peered":"2024-03-06T07:21:59.944275+0000","last_clean":"2024-03-06T07:21:59.944275+0000","last_became_active":"2024-03-06T07:21:57.963195+0000","last_became_peered":"2024-03-06T07:21:57.963195+0000","last_unstale":"2024-03-06T07:21:59.944275+0000","last_undegraded":"2024-03-06T07:21:59.944275+0000","last_fullsized":"2024-03-06T07:21:59.944275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005119+0000","last_change":"2024-03-06T07:21:58.961185+0000","last_active":"2024-03-06T07:22:01.005119+0000","last_peered":"2024-03-06T07:22:01.005119+0000","last_clean":"2024-03-06T07:22:01.005119+0000","last_became_active":"2024-03-06T07:21:58.960928+0000","last_became_peered":"2024-03-06T07:21:58.960928+0000","last_unstale":"2024-03-06T07:22:01.005119+0000","last_undegraded":"2024-03-06T07:22:01.005119+0000","last_fullsized":"2024-03-06T07:22:01.005119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:42.427321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549040+0000","last_change":"2024-03-06T07:21:57.950617+0000","last_active":"2024-03-06T07:22:00.549040+0000","last_peered":"2024-03-06T07:22:00.549040+0000","last_clean":"2024-03-06T07:22:00.549040+0000","last_became_active":"2024-03-06T07:21:57.950347+0000","last_became_peered":"2024-03-06T07:21:57.950347+0000","last_unstale":"2024-03-06T07:22:00.549040+0000","last_undegraded":"2024-03-06T07:22:00.549040+0000","last_fullsized":"2024-03-06T07:22:00.549040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:10.110560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548899+0000","last_change":"2024-03-06T07:21:58.957844+0000","last_active":"2024-03-06T07:22:00.548899+0000","last_peered":"2024-03-06T07:22:00.548899+0000","last_clean":"2024-03-06T07:22:00.548899+0000","last_became_active":"2024-03-06T07:21:58.957575+0000","last_became_peered":"2024-03-06T07:21:58.957575+0000","last_unstale":"2024-03-06T07:22:00.548899+0000","last_undegraded":"2024-03-06T07:22:00.548899+0000","last_fullsized":"2024-03-06T07:22:00.548899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:41.058405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004312+0000","last_change":"2024-03-06T07:21:57.964121+0000","last_active":"2024-03-06T07:22:01.004312+0000","last_peered":"2024-03-06T07:22:01.004312+0000","last_clean":"2024-03-06T07:22:01.004312+0000","last_became_active":"2024-03-06T07:21:57.963752+0000","last_became_peered":"2024-03-06T07:21:57.963752+0000","last_unstale":"2024-03-06T07:22:01.004312+0000","last_undegraded":"2024-03-06T07:22:01.004312+0000","last_fullsized":"2024-03-06T07:22:01.004312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005124+0000","last_change":"2024-03-06T07:21:58.962110+0000","last_active":"2024-03-06T07:22:01.005124+0000","last_peered":"2024-03-06T07:22:01.005124+0000","last_clean":"2024-03-06T07:22:01.005124+0000","last_became_active":"2024-03-06T07:21:58.961800+0000","last_became_peered":"2024-03-06T07:21:58.961800+0000","last_unstale":"2024-03-06T07:22:01.005124+0000","last_undegraded":"2024-03-06T07:22:01.005124+0000","last_fullsized":"2024-03-06T07:22:01.005124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:13.761240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944044+0000","last_change":"2024-03-06T07:21:57.949180+0000","last_active":"2024-03-06T07:21:59.944044+0000","last_peered":"2024-03-06T07:21:59.944044+0000","last_clean":"2024-03-06T07:21:59.944044+0000","last_became_active":"2024-03-06T07:21:57.948897+0000","last_became_peered":"2024-03-06T07:21:57.948897+0000","last_unstale":"2024-03-06T07:21:59.944044+0000","last_undegraded":"2024-03-06T07:21:59.944044+0000","last_fullsized":"2024-03-06T07:21:59.944044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944254+0000","last_change":"2024-03-06T07:21:58.957923+0000","last_active":"2024-03-06T07:21:59.944254+0000","last_peered":"2024-03-06T07:21:59.944254+0000","last_clean":"2024-03-06T07:21:59.944254+0000","last_became_active":"2024-03-06T07:21:58.957454+0000","last_became_peered":"2024-03-06T07:21:58.957454+0000","last_unstale":"2024-03-06T07:21:59.944254+0000","last_undegraded":"2024-03-06T07:21:59.944254+0000","last_fullsized":"2024-03-06T07:21:59.944254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983080+0000","last_change":"2024-03-06T07:21:57.956795+0000","last_active":"2024-03-06T07:21:59.983080+0000","last_peered":"2024-03-06T07:21:59.983080+0000","last_clean":"2024-03-06T07:21:59.983080+0000","last_became_active":"2024-03-06T07:21:57.956529+0000","last_became_peered":"2024-03-06T07:21:57.956529+0000","last_unstale":"2024-03-06T07:21:59.983080+0000","last_undegraded":"2024-03-06T07:21:59.983080+0000","last_fullsized":"2024-03-06T07:21:59.983080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:02.949716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004551+0000","last_change":"2024-03-06T07:21:58.961675+0000","last_active":"2024-03-06T07:22:01.004551+0000","last_peered":"2024-03-06T07:22:01.004551+0000","last_clean":"2024-03-06T07:22:01.004551+0000","last_became_active":"2024-03-06T07:21:58.961169+0000","last_became_peered":"2024-03-06T07:21:58.961169+0000","last_unstale":"2024-03-06T07:22:01.004551+0000","last_undegraded":"2024-03-06T07:22:01.004551+0000","last_fullsized":"2024-03-06T07:22:01.004551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943737+0000","last_change":"2024-03-06T07:21:57.952110+0000","last_active":"2024-03-06T07:21:59.943737+0000","last_peered":"2024-03-06T07:21:59.943737+0000","last_clean":"2024-03-06T07:21:59.943737+0000","last_became_active":"2024-03-06T07:21:57.951787+0000","last_became_peered":"2024-03-06T07:21:57.951787+0000","last_unstale":"2024-03-06T07:21:59.943737+0000","last_undegraded":"2024-03-06T07:21:59.943737+0000","last_fullsized":"2024-03-06T07:21:59.943737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:34.102031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004914+0000","last_change":"2024-03-06T07:21:58.967742+0000","last_active":"2024-03-06T07:22:01.004914+0000","last_peered":"2024-03-06T07:22:01.004914+0000","last_clean":"2024-03-06T07:22:01.004914+0000","last_became_active":"2024-03-06T07:21:58.967476+0000","last_became_peered":"2024-03-06T07:21:58.967476+0000","last_unstale":"2024-03-06T07:22:01.004914+0000","last_undegraded":"2024-03-06T07:22:01.004914+0000","last_fullsized":"2024-03-06T07:22:01.004914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983255+0000","last_change":"2024-03-06T07:21:57.962699+0000","last_active":"2024-03-06T07:21:59.983255+0000","last_peered":"2024-03-06T07:21:59.983255+0000","last_clean":"2024-03-06T07:21:59.983255+0000","last_became_active":"2024-03-06T07:21:57.962401+0000","last_became_peered":"2024-03-06T07:21:57.962401+0000","last_unstale":"2024-03-06T07:21:59.983255+0000","last_undegraded":"2024-03-06T07:21:59.983255+0000","last_fullsized":"2024-03-06T07:21:59.983255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005163+0000","last_change":"2024-03-06T07:21:58.964631+0000","last_active":"2024-03-06T07:22:01.005163+0000","last_peered":"2024-03-06T07:22:01.005163+0000","last_clean":"2024-03-06T07:22:01.005163+0000","last_became_active":"2024-03-06T07:21:58.964187+0000","last_became_peered":"2024-03-06T07:21:58.964187+0000","last_unstale":"2024-03-06T07:22:01.005163+0000","last_undegraded":"2024-03-06T07:22:01.005163+0000","last_fullsized":"2024-03-06T07:22:01.005163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:34.814920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942356+0000","last_change":"2024-03-06T07:21:57.951889+0000","last_active":"2024-03-06T07:21:59.942356+0000","last_peered":"2024-03-06T07:21:59.942356+0000","last_clean":"2024-03-06T07:21:59.942356+0000","last_became_active":"2024-03-06T07:21:57.951637+0000","last_became_peered":"2024-03-06T07:21:57.951637+0000","last_unstale":"2024-03-06T07:21:59.942356+0000","last_undegraded":"2024-03-06T07:21:59.942356+0000","last_fullsized":"2024-03-06T07:21:59.942356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004803+0000","last_change":"2024-03-06T07:21:58.962144+0000","last_active":"2024-03-06T07:22:01.004803+0000","last_peered":"2024-03-06T07:22:01.004803+0000","last_clean":"2024-03-06T07:22:01.004803+0000","last_became_active":"2024-03-06T07:21:58.960533+0000","last_became_peered":"2024-03-06T07:21:58.960533+0000","last_unstale":"2024-03-06T07:22:01.004803+0000","last_undegraded":"2024-03-06T07:22:01.004803+0000","last_fullsized":"2024-03-06T07:22:01.004803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:28.792591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983285+0000","last_change":"2024-03-06T07:21:57.958443+0000","last_active":"2024-03-06T07:21:59.983285+0000","last_peered":"2024-03-06T07:21:59.983285+0000","last_clean":"2024-03-06T07:21:59.983285+0000","last_became_active":"2024-03-06T07:21:57.958162+0000","last_became_peered":"2024-03-06T07:21:57.958162+0000","last_unstale":"2024-03-06T07:21:59.983285+0000","last_undegraded":"2024-03-06T07:21:59.983285+0000","last_fullsized":"2024-03-06T07:21:59.983285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005108+0000","last_change":"2024-03-06T07:21:58.968776+0000","last_active":"2024-03-06T07:22:01.005108+0000","last_peered":"2024-03-06T07:22:01.005108+0000","last_clean":"2024-03-06T07:22:01.005108+0000","last_became_active":"2024-03-06T07:21:58.968522+0000","last_became_peered":"2024-03-06T07:21:58.968522+0000","last_unstale":"2024-03-06T07:22:01.005108+0000","last_undegraded":"2024-03-06T07:22:01.005108+0000","last_fullsized":"2024-03-06T07:22:01.005108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:28.792591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004412+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:22:01.004412+0000","last_peered":"2024-03-06T07:22:01.004412+0000","last_clean":"2024-03-06T07:22:01.004412+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:22:01.004412+0000","last_undegraded":"2024-03-06T07:22:01.004412+0000","last_fullsized":"2024-03-06T07:22:01.004412+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004292+0000","last_change":"2024-03-06T07:21:57.958220+0000","last_active":"2024-03-06T07:22:01.004292+0000","last_peered":"2024-03-06T07:22:01.004292+0000","last_clean":"2024-03-06T07:22:01.004292+0000","last_became_active":"2024-03-06T07:21:57.957561+0000","last_became_peered":"2024-03-06T07:21:57.957561+0000","last_unstale":"2024-03-06T07:22:01.004292+0000","last_undegraded":"2024-03-06T07:22:01.004292+0000","last_fullsized":"2024-03-06T07:22:01.004292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004721+0000","last_change":"2024-03-06T07:21:58.968424+0000","last_active":"2024-03-06T07:22:01.004721+0000","last_peered":"2024-03-06T07:22:01.004721+0000","last_clean":"2024-03-06T07:22:01.004721+0000","last_became_active":"2024-03-06T07:21:58.968235+0000","last_became_peered":"2024-03-06T07:21:58.968235+0000","last_unstale":"2024-03-06T07:22:01.004721+0000","last_undegraded":"2024-03-06T07:22:01.004721+0000","last_fullsized":"2024-03-06T07:22:01.004721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004311+0000","last_change":"2024-03-06T07:21:57.962310+0000","last_active":"2024-03-06T07:22:01.004311+0000","last_peered":"2024-03-06T07:22:01.004311+0000","last_clean":"2024-03-06T07:22:01.004311+0000","last_became_active":"2024-03-06T07:21:57.962015+0000","last_became_peered":"2024-03-06T07:21:57.962015+0000","last_unstale":"2024-03-06T07:22:01.004311+0000","last_undegraded":"2024-03-06T07:22:01.004311+0000","last_fullsized":"2024-03-06T07:22:01.004311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:04.878864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548793+0000","last_change":"2024-03-06T07:21:58.949526+0000","last_active":"2024-03-06T07:22:00.548793+0000","last_peered":"2024-03-06T07:22:00.548793+0000","last_clean":"2024-03-06T07:22:00.548793+0000","last_became_active":"2024-03-06T07:21:58.949247+0000","last_became_peered":"2024-03-06T07:21:58.949247+0000","last_unstale":"2024-03-06T07:22:00.548793+0000","last_undegraded":"2024-03-06T07:22:00.548793+0000","last_fullsized":"2024-03-06T07:22:00.548793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:11.328500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983204+0000","last_change":"2024-03-06T07:21:57.963056+0000","last_active":"2024-03-06T07:21:59.983204+0000","last_peered":"2024-03-06T07:21:59.983204+0000","last_clean":"2024-03-06T07:21:59.983204+0000","last_became_active":"2024-03-06T07:21:57.962785+0000","last_became_peered":"2024-03-06T07:21:57.962785+0000","last_unstale":"2024-03-06T07:21:59.983204+0000","last_undegraded":"2024-03-06T07:21:59.983204+0000","last_fullsized":"2024-03-06T07:21:59.983204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942195+0000","last_change":"2024-03-06T07:21:58.948764+0000","last_active":"2024-03-06T07:21:59.942195+0000","last_peered":"2024-03-06T07:21:59.942195+0000","last_clean":"2024-03-06T07:21:59.942195+0000","last_became_active":"2024-03-06T07:21:58.948489+0000","last_became_peered":"2024-03-06T07:21:58.948489+0000","last_unstale":"2024-03-06T07:21:59.942195+0000","last_undegraded":"2024-03-06T07:21:59.942195+0000","last_fullsized":"2024-03-06T07:21:59.942195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004402+0000","last_change":"2024-03-06T07:21:57.965008+0000","last_active":"2024-03-06T07:22:01.004402+0000","last_peered":"2024-03-06T07:22:01.004402+0000","last_clean":"2024-03-06T07:22:01.004402+0000","last_became_active":"2024-03-06T07:21:57.964754+0000","last_became_peered":"2024-03-06T07:21:57.964754+0000","last_unstale":"2024-03-06T07:22:01.004402+0000","last_undegraded":"2024-03-06T07:22:01.004402+0000","last_fullsized":"2024-03-06T07:22:01.004402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944174+0000","last_change":"2024-03-06T07:21:58.958773+0000","last_active":"2024-03-06T07:21:59.944174+0000","last_peered":"2024-03-06T07:21:59.944174+0000","last_clean":"2024-03-06T07:21:59.944174+0000","last_became_active":"2024-03-06T07:21:58.958489+0000","last_became_peered":"2024-03-06T07:21:58.958489+0000","last_unstale":"2024-03-06T07:21:59.944174+0000","last_undegraded":"2024-03-06T07:21:59.944174+0000","last_fullsized":"2024-03-06T07:21:59.944174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:29.955363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942256+0000","last_change":"2024-03-06T07:21:57.952000+0000","last_active":"2024-03-06T07:21:59.942256+0000","last_peered":"2024-03-06T07:21:59.942256+0000","last_clean":"2024-03-06T07:21:59.942256+0000","last_became_active":"2024-03-06T07:21:57.951703+0000","last_became_peered":"2024-03-06T07:21:57.951703+0000","last_unstale":"2024-03-06T07:21:59.942256+0000","last_undegraded":"2024-03-06T07:21:59.942256+0000","last_fullsized":"2024-03-06T07:21:59.942256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004691+0000","last_change":"2024-03-06T07:21:58.967480+0000","last_active":"2024-03-06T07:22:01.004691+0000","last_peered":"2024-03-06T07:22:01.004691+0000","last_clean":"2024-03-06T07:22:01.004691+0000","last_became_active":"2024-03-06T07:21:58.967104+0000","last_became_peered":"2024-03-06T07:21:58.967104+0000","last_unstale":"2024-03-06T07:22:01.004691+0000","last_undegraded":"2024-03-06T07:22:01.004691+0000","last_fullsized":"2024-03-06T07:22:01.004691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:53.179429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005275+0000","last_change":"2024-03-06T07:21:57.960485+0000","last_active":"2024-03-06T07:22:01.005275+0000","last_peered":"2024-03-06T07:22:01.005275+0000","last_clean":"2024-03-06T07:22:01.005275+0000","last_became_active":"2024-03-06T07:21:57.960144+0000","last_became_peered":"2024-03-06T07:21:57.960144+0000","last_unstale":"2024-03-06T07:22:01.005275+0000","last_undegraded":"2024-03-06T07:22:01.005275+0000","last_fullsized":"2024-03-06T07:22:01.005275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:41.468075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942320+0000","last_change":"2024-03-06T07:21:58.947670+0000","last_active":"2024-03-06T07:21:59.942320+0000","last_peered":"2024-03-06T07:21:59.942320+0000","last_clean":"2024-03-06T07:21:59.942320+0000","last_became_active":"2024-03-06T07:21:58.947435+0000","last_became_peered":"2024-03-06T07:21:58.947435+0000","last_unstale":"2024-03-06T07:21:59.942320+0000","last_undegraded":"2024-03-06T07:21:59.942320+0000","last_fullsized":"2024-03-06T07:21:59.942320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942445+0000","last_change":"2024-03-06T07:21:57.951589+0000","last_active":"2024-03-06T07:21:59.942445+0000","last_peered":"2024-03-06T07:21:59.942445+0000","last_clean":"2024-03-06T07:21:59.942445+0000","last_became_active":"2024-03-06T07:21:57.951313+0000","last_became_peered":"2024-03-06T07:21:57.951313+0000","last_unstale":"2024-03-06T07:21:59.942445+0000","last_undegraded":"2024-03-06T07:21:59.942445+0000","last_fullsized":"2024-03-06T07:21:59.942445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:53.197786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005072+0000","last_change":"2024-03-06T07:21:58.966146+0000","last_active":"2024-03-06T07:22:01.005072+0000","last_peered":"2024-03-06T07:22:01.005072+0000","last_clean":"2024-03-06T07:22:01.005072+0000","last_became_active":"2024-03-06T07:21:58.965839+0000","last_became_peered":"2024-03-06T07:21:58.965839+0000","last_unstale":"2024-03-06T07:22:01.005072+0000","last_undegraded":"2024-03-06T07:22:01.005072+0000","last_fullsized":"2024-03-06T07:22:01.005072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944510+0000","last_change":"2024-03-06T07:21:57.956954+0000","last_active":"2024-03-06T07:21:59.944510+0000","last_peered":"2024-03-06T07:21:59.944510+0000","last_clean":"2024-03-06T07:21:59.944510+0000","last_became_active":"2024-03-06T07:21:57.956181+0000","last_became_peered":"2024-03-06T07:21:57.956181+0000","last_unstale":"2024-03-06T07:21:59.944510+0000","last_undegraded":"2024-03-06T07:21:59.944510+0000","last_fullsized":"2024-03-06T07:21:59.944510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549033+0000","last_change":"2024-03-06T07:21:58.957111+0000","last_active":"2024-03-06T07:22:00.549033+0000","last_peered":"2024-03-06T07:22:00.549033+0000","last_clean":"2024-03-06T07:22:00.549033+0000","last_became_active":"2024-03-06T07:21:58.956788+0000","last_became_peered":"2024-03-06T07:21:58.956788+0000","last_unstale":"2024-03-06T07:22:00.549033+0000","last_undegraded":"2024-03-06T07:22:00.549033+0000","last_fullsized":"2024-03-06T07:22:00.549033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004332+0000","last_change":"2024-03-06T07:21:57.960912+0000","last_active":"2024-03-06T07:22:01.004332+0000","last_peered":"2024-03-06T07:22:01.004332+0000","last_clean":"2024-03-06T07:22:01.004332+0000","last_became_active":"2024-03-06T07:21:57.960619+0000","last_became_peered":"2024-03-06T07:21:57.960619+0000","last_unstale":"2024-03-06T07:22:01.004332+0000","last_undegraded":"2024-03-06T07:22:01.004332+0000","last_fullsized":"2024-03-06T07:22:01.004332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:41.836911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943984+0000","last_change":"2024-03-06T07:21:58.949723+0000","last_active":"2024-03-06T07:21:59.943984+0000","last_peered":"2024-03-06T07:21:59.943984+0000","last_clean":"2024-03-06T07:21:59.943984+0000","last_became_active":"2024-03-06T07:21:58.949415+0000","last_became_peered":"2024-03-06T07:21:58.949415+0000","last_unstale":"2024-03-06T07:21:59.943984+0000","last_undegraded":"2024-03-06T07:21:59.943984+0000","last_fullsized":"2024-03-06T07:21:59.943984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:48.783928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942225+0000","last_change":"2024-03-06T07:21:57.957156+0000","last_active":"2024-03-06T07:21:59.942225+0000","last_peered":"2024-03-06T07:21:59.942225+0000","last_clean":"2024-03-06T07:21:59.942225+0000","last_became_active":"2024-03-06T07:21:57.956893+0000","last_became_peered":"2024-03-06T07:21:57.956893+0000","last_unstale":"2024-03-06T07:21:59.942225+0000","last_undegraded":"2024-03-06T07:21:59.942225+0000","last_fullsized":"2024-03-06T07:21:59.942225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004691+0000","last_change":"2024-03-06T07:21:58.953837+0000","last_active":"2024-03-06T07:22:01.004691+0000","last_peered":"2024-03-06T07:22:01.004691+0000","last_clean":"2024-03-06T07:22:01.004691+0000","last_became_active":"2024-03-06T07:21:58.953594+0000","last_became_peered":"2024-03-06T07:21:58.953594+0000","last_unstale":"2024-03-06T07:22:01.004691+0000","last_undegraded":"2024-03-06T07:22:01.004691+0000","last_fullsized":"2024-03-06T07:22:01.004691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942272+0000","last_change":"2024-03-06T07:21:57.962804+0000","last_active":"2024-03-06T07:21:59.942272+0000","last_peered":"2024-03-06T07:21:59.942272+0000","last_clean":"2024-03-06T07:21:59.942272+0000","last_became_active":"2024-03-06T07:21:57.962366+0000","last_became_peered":"2024-03-06T07:21:57.962366+0000","last_unstale":"2024-03-06T07:21:59.942272+0000","last_undegraded":"2024-03-06T07:21:59.942272+0000","last_fullsized":"2024-03-06T07:21:59.942272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004802+0000","last_change":"2024-03-06T07:21:58.957659+0000","last_active":"2024-03-06T07:22:01.004802+0000","last_peered":"2024-03-06T07:22:01.004802+0000","last_clean":"2024-03-06T07:22:01.004802+0000","last_became_active":"2024-03-06T07:21:58.957349+0000","last_became_peered":"2024-03-06T07:21:58.957349+0000","last_unstale":"2024-03-06T07:22:01.004802+0000","last_undegraded":"2024-03-06T07:22:01.004802+0000","last_fullsized":"2024-03-06T07:22:01.004802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005015+0000","last_change":"2024-03-06T07:21:57.957600+0000","last_active":"2024-03-06T07:22:01.005015+0000","last_peered":"2024-03-06T07:22:01.005015+0000","last_clean":"2024-03-06T07:22:01.005015+0000","last_became_active":"2024-03-06T07:21:57.957347+0000","last_became_peered":"2024-03-06T07:21:57.957347+0000","last_unstale":"2024-03-06T07:22:01.005015+0000","last_undegraded":"2024-03-06T07:22:01.005015+0000","last_fullsized":"2024-03-06T07:22:01.005015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944432+0000","last_change":"2024-03-06T07:21:58.950912+0000","last_active":"2024-03-06T07:21:59.944432+0000","last_peered":"2024-03-06T07:21:59.944432+0000","last_clean":"2024-03-06T07:21:59.944432+0000","last_became_active":"2024-03-06T07:21:58.950531+0000","last_became_peered":"2024-03-06T07:21:58.950531+0000","last_unstale":"2024-03-06T07:21:59.944432+0000","last_undegraded":"2024-03-06T07:21:59.944432+0000","last_fullsized":"2024-03-06T07:21:59.944432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004726+0000","last_change":"2024-03-06T07:21:58.968927+0000","last_active":"2024-03-06T07:22:01.004726+0000","last_peered":"2024-03-06T07:22:01.004726+0000","last_clean":"2024-03-06T07:22:01.004726+0000","last_became_active":"2024-03-06T07:21:58.967167+0000","last_became_peered":"2024-03-06T07:21:58.967167+0000","last_unstale":"2024-03-06T07:22:01.004726+0000","last_undegraded":"2024-03-06T07:22:01.004726+0000","last_fullsized":"2024-03-06T07:22:01.004726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:00.578026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004797+0000","last_change":"2024-03-06T07:21:57.962444+0000","last_active":"2024-03-06T07:22:01.004797+0000","last_peered":"2024-03-06T07:22:01.004797+0000","last_clean":"2024-03-06T07:22:01.004797+0000","last_became_active":"2024-03-06T07:21:57.962157+0000","last_became_peered":"2024-03-06T07:21:57.962157+0000","last_unstale":"2024-03-06T07:22:01.004797+0000","last_undegraded":"2024-03-06T07:22:01.004797+0000","last_fullsized":"2024-03-06T07:22:01.004797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004672+0000","last_change":"2024-03-06T07:21:58.965441+0000","last_active":"2024-03-06T07:22:01.004672+0000","last_peered":"2024-03-06T07:22:01.004672+0000","last_clean":"2024-03-06T07:22:01.004672+0000","last_became_active":"2024-03-06T07:21:58.964894+0000","last_became_peered":"2024-03-06T07:21:58.964894+0000","last_unstale":"2024-03-06T07:22:01.004672+0000","last_undegraded":"2024-03-06T07:22:01.004672+0000","last_fullsized":"2024-03-06T07:22:01.004672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:51.617926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005175+0000","last_change":"2024-03-06T07:21:57.958429+0000","last_active":"2024-03-06T07:22:01.005175+0000","last_peered":"2024-03-06T07:22:01.005175+0000","last_clean":"2024-03-06T07:22:01.005175+0000","last_became_active":"2024-03-06T07:21:57.958163+0000","last_became_peered":"2024-03-06T07:21:57.958163+0000","last_unstale":"2024-03-06T07:22:01.005175+0000","last_undegraded":"2024-03-06T07:22:01.005175+0000","last_fullsized":"2024-03-06T07:22:01.005175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:53.197786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548917+0000","last_change":"2024-03-06T07:21:58.962796+0000","last_active":"2024-03-06T07:22:00.548917+0000","last_peered":"2024-03-06T07:22:00.548917+0000","last_clean":"2024-03-06T07:22:00.548917+0000","last_became_active":"2024-03-06T07:21:58.962454+0000","last_became_peered":"2024-03-06T07:21:58.962454+0000","last_unstale":"2024-03-06T07:22:00.548917+0000","last_undegraded":"2024-03-06T07:22:00.548917+0000","last_fullsized":"2024-03-06T07:22:00.548917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004974+0000","last_change":"2024-03-06T07:21:57.962433+0000","last_active":"2024-03-06T07:22:01.004974+0000","last_peered":"2024-03-06T07:22:01.004974+0000","last_clean":"2024-03-06T07:22:01.004974+0000","last_became_active":"2024-03-06T07:21:57.962149+0000","last_became_peered":"2024-03-06T07:21:57.962149+0000","last_unstale":"2024-03-06T07:22:01.004974+0000","last_undegraded":"2024-03-06T07:22:01.004974+0000","last_fullsized":"2024-03-06T07:22:01.004974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004944+0000","last_change":"2024-03-06T07:21:58.963553+0000","last_active":"2024-03-06T07:22:01.004944+0000","last_peered":"2024-03-06T07:22:01.004944+0000","last_clean":"2024-03-06T07:22:01.004944+0000","last_became_active":"2024-03-06T07:21:58.963282+0000","last_became_peered":"2024-03-06T07:21:58.963282+0000","last_unstale":"2024-03-06T07:22:01.004944+0000","last_undegraded":"2024-03-06T07:22:01.004944+0000","last_fullsized":"2024-03-06T07:22:01.004944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:25.392050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549082+0000","last_change":"2024-03-06T07:21:57.956370+0000","last_active":"2024-03-06T07:22:00.549082+0000","last_peered":"2024-03-06T07:22:00.549082+0000","last_clean":"2024-03-06T07:22:00.549082+0000","last_became_active":"2024-03-06T07:21:57.955872+0000","last_became_peered":"2024-03-06T07:21:57.955872+0000","last_unstale":"2024-03-06T07:22:00.549082+0000","last_undegraded":"2024-03-06T07:22:00.549082+0000","last_fullsized":"2024-03-06T07:22:00.549082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:46.175278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:10.562 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:10.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.697+0000 7f158d502640 1 Processor -- start 2024-03-06T07:22:10.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.697+0000 7f158d502640 1 -- start start 2024-03-06T07:22:10.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f158d502640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15880d0660 0x7f15880cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f158d502640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f15880cf850 0x7f15880b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f158d502640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158815e4d0 con 0x7f158815e0f0 2024-03-06T07:22:10.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f158d502640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15880b1410 con 0x7f15880d0660 2024-03-06T07:22:10.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f158d502640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f158806c8b0 con 0x7f15880cf850 2024-03-06T07:22:10.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f15867fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15880d0660 0x7f15880cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f15867fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15880d0660 0x7f15880cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56192/0 (socket says 172.21.15.77:56192) 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f15867fc640 1 -- 172.21.15.77:0/3985630347 learned_addr learned my addr 172.21.15.77:0/3985630347 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f15877fe640 1 --2- 172.21.15.77:0/3985630347 >> v2:172.21.15.158:3300/0 conn(0x7f15880cf850 0x7f15880b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f15867fc640 1 -- 172.21.15.77:0/3985630347 >> v2:172.21.15.158:3300/0 conn(0x7f15880cf850 msgr2=0x7f15880b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f15867fc640 1 --2- 172.21.15.77:0/3985630347 >> v2:172.21.15.158:3300/0 conn(0x7f15880cf850 0x7f15880b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f15867fc640 1 -- 172.21.15.77:0/3985630347 >> v1:172.21.15.158:6789/0 conn(0x7f158815e0f0 legacy=0x7f15880cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.698+0000 7f15867fc640 1 -- 172.21.15.77:0/3985630347 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15880cdac0 con 0x7f15880d0660 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.699+0000 7f15867fc640 1 --2- 172.21.15.77:0/3985630347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15880d0660 0x7f15880cd3c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1588074c90 tx=0x7f157c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40426c25108b7a35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.699+0000 7f1585ffb640 1 -- 172.21.15.77:0/3985630347 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c035020 con 0x7f15880d0660 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.699+0000 7f1585ffb640 1 -- 172.21.15.77:0/3985630347 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c031870 con 0x7f15880d0660 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.699+0000 7f1585ffb640 1 -- 172.21.15.77:0/3985630347 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c031c40 con 0x7f15880d0660 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.700+0000 7f158d502640 1 -- 172.21.15.77:0/3985630347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15880d0660 msgr2=0x7f15880cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.700+0000 7f158d502640 1 --2- 172.21.15.77:0/3985630347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15880d0660 0x7f15880cd3c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1588074c90 tx=0x7f157c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.700+0000 7f158d502640 1 -- 172.21.15.77:0/3985630347 shutdown_connections 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.700+0000 7f158d502640 1 --2- 172.21.15.77:0/3985630347 >> v2:172.21.15.158:3300/0 conn(0x7f15880cf850 0x7f15880b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.700+0000 7f158d502640 1 --2- 172.21.15.77:0/3985630347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15880d0660 0x7f15880cd3c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.700+0000 7f158d502640 1 -- 172.21.15.77:0/3985630347 >> 172.21.15.77:0/3985630347 conn(0x7f15880b35a0 msgr2=0x7f15880b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.700+0000 7f158d502640 1 -- 172.21.15.77:0/3985630347 shutdown_connections 2024-03-06T07:22:10.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.700+0000 7f158d502640 1 -- 172.21.15.77:0/3985630347 wait complete. 2024-03-06T07:22:10.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.701+0000 7f158d502640 1 Processor -- start 2024-03-06T07:22:10.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.701+0000 7f158d502640 1 -- start start 2024-03-06T07:22:10.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f158d502640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15880cf850 0x7f15880b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f158d502640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15880d0660 0x7f15880b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f158d502640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158815e0f0 0x7f15880b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f158d502640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15880d0cc0 con 0x7f158815e0f0 2024-03-06T07:22:10.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f158d502640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158806c540 con 0x7f15880cf850 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f158d502640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15880b2af0 con 0x7f15880d0660 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158815e0f0 0x7f15880b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158815e0f0 0x7f15880b67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56204/0 (socket says 172.21.15.77:56204) 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 -- 172.21.15.77:0/3162847847 learned_addr learned my addr 172.21.15.77:0/3162847847 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f1586ffd640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15880cf850 0x7f15880b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15867fc640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15880d0660 0x7f15880b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 -- 172.21.15.77:0/3162847847 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15880d0660 msgr2=0x7f15880b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15880d0660 0x7f15880b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 -- 172.21.15.77:0/3162847847 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15880cf850 msgr2=0x7f15880b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15880cf850 0x7f15880b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 -- 172.21.15.77:0/3162847847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1588160520 con 0x7f158815e0f0 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.702+0000 7f15877fe640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158815e0f0 0x7f15880b67b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f158000e930 tx=0x7f158000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.703+0000 7f156f7fe640 1 -- 172.21.15.77:0/3162847847 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580017070 con 0x7f158815e0f0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.703+0000 7f156f7fe640 1 -- 172.21.15.77:0/3162847847 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158000c980 con 0x7f158815e0f0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.703+0000 7f156f7fe640 1 -- 172.21.15.77:0/3162847847 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158000cd70 con 0x7f158815e0f0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.703+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1588160800 con 0x7f158815e0f0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.703+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1588160dc0 con 0x7f158815e0f0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.704+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f158804f330 con 0x7f158815e0f0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.704+0000 7f156f7fe640 1 -- 172.21.15.77:0/3162847847 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f158001a070 con 0x7f158815e0f0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.704+0000 7f156f7fe640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157406b4d0 0x7f157406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.705+0000 7f156f7fe640 1 -- 172.21.15.77:0/3162847847 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f158008a480 con 0x7f158815e0f0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.705+0000 7f1586ffd640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157406b4d0 0x7f157406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.705+0000 7f1586ffd640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157406b4d0 0x7f157406d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1578001e70 tx=0x7f1578009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.709+0000 7f156f7fe640 1 -- 172.21.15.77:0/3162847847 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1580058e00 con 0x7f158815e0f0 2024-03-06T07:22:10.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.875+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15880b3db0 con 0x7f157406b4d0 2024-03-06T07:22:10.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.881+0000 7f156f7fe640 1 -- 172.21.15.77:0/3162847847 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7f15880b3db0 con 0x7f157406b4d0 2024-03-06T07:22:10.885 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:10.887 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:22:10.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157406b4d0 msgr2=0x7f157406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157406b4d0 0x7f157406d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1578001e70 tx=0x7f1578009040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158815e0f0 msgr2=0x7f15880b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158815e0f0 0x7f15880b67b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f158000e930 tx=0x7f158000edf0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 shutdown_connections 2024-03-06T07:22:10.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157406b4d0 0x7f157406d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158815e0f0 0x7f15880b67b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15880d0660 0x7f15880b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 --2- 172.21.15.77:0/3162847847 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15880cf850 0x7f15880b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 >> 172.21.15.77:0/3162847847 conn(0x7f15880b35a0 msgr2=0x7f158806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 shutdown_connections 2024-03-06T07:22:10.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:10.884+0000 7f158d502640 1 -- 172.21.15.77:0/3162847847 wait complete. 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:22:09.385710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235860,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738860,"statfs":{"total":773094113280,"available":772852592640,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001783"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004993+0000","last_change":"2024-03-06T07:21:58.964978+0000","last_active":"2024-03-06T07:22:01.004993+0000","last_peered":"2024-03-06T07:22:01.004993+0000","last_clean":"2024-03-06T07:22:01.004993+0000","last_became_active":"2024-03-06T07:21:58.964298+0000","last_became_peered":"2024-03-06T07:21:58.964298+0000","last_unstale":"2024-03-06T07:22:01.004993+0000","last_undegraded":"2024-03-06T07:22:01.004993+0000","last_fullsized":"2024-03-06T07:22:01.004993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:53.179429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942479+0000","last_change":"2024-03-06T07:21:58.960691+0000","last_active":"2024-03-06T07:21:59.942479+0000","last_peered":"2024-03-06T07:21:59.942479+0000","last_clean":"2024-03-06T07:21:59.942479+0000","last_became_active":"2024-03-06T07:21:58.959088+0000","last_became_peered":"2024-03-06T07:21:58.959088+0000","last_unstale":"2024-03-06T07:21:59.942479+0000","last_undegraded":"2024-03-06T07:21:59.942479+0000","last_fullsized":"2024-03-06T07:21:59.942479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549070+0000","last_change":"2024-03-06T07:21:58.953314+0000","last_active":"2024-03-06T07:22:00.549070+0000","last_peered":"2024-03-06T07:22:00.549070+0000","last_clean":"2024-03-06T07:22:00.549070+0000","last_became_active":"2024-03-06T07:21:58.953048+0000","last_became_peered":"2024-03-06T07:21:58.953048+0000","last_unstale":"2024-03-06T07:22:00.549070+0000","last_undegraded":"2024-03-06T07:22:00.549070+0000","last_fullsized":"2024-03-06T07:22:00.549070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004545+0000","last_change":"2024-03-06T07:21:58.965150+0000","last_active":"2024-03-06T07:22:01.004545+0000","last_peered":"2024-03-06T07:22:01.004545+0000","last_clean":"2024-03-06T07:22:01.004545+0000","last_became_active":"2024-03-06T07:21:58.964307+0000","last_became_peered":"2024-03-06T07:21:58.964307+0000","last_unstale":"2024-03-06T07:22:01.004545+0000","last_undegraded":"2024-03-06T07:22:01.004545+0000","last_fullsized":"2024-03-06T07:22:01.004545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004939+0000","last_change":"2024-03-06T07:21:58.957427+0000","last_active":"2024-03-06T07:22:01.004939+0000","last_peered":"2024-03-06T07:22:01.004939+0000","last_clean":"2024-03-06T07:22:01.004939+0000","last_became_active":"2024-03-06T07:21:58.957124+0000","last_became_peered":"2024-03-06T07:21:58.957124+0000","last_unstale":"2024-03-06T07:22:01.004939+0000","last_undegraded":"2024-03-06T07:22:01.004939+0000","last_fullsized":"2024-03-06T07:22:01.004939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:51.617926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004247+0000","last_change":"2024-03-06T07:21:58.969024+0000","last_active":"2024-03-06T07:22:01.004247+0000","last_peered":"2024-03-06T07:22:01.004247+0000","last_clean":"2024-03-06T07:22:01.004247+0000","last_became_active":"2024-03-06T07:21:58.963242+0000","last_became_peered":"2024-03-06T07:21:58.963242+0000","last_unstale":"2024-03-06T07:22:01.004247+0000","last_undegraded":"2024-03-06T07:22:01.004247+0000","last_fullsized":"2024-03-06T07:22:01.004247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004439+0000","last_change":"2024-03-06T07:21:58.958719+0000","last_active":"2024-03-06T07:22:01.004439+0000","last_peered":"2024-03-06T07:22:01.004439+0000","last_clean":"2024-03-06T07:22:01.004439+0000","last_became_active":"2024-03-06T07:21:58.958025+0000","last_became_peered":"2024-03-06T07:21:58.958025+0000","last_unstale":"2024-03-06T07:22:01.004439+0000","last_undegraded":"2024-03-06T07:22:01.004439+0000","last_fullsized":"2024-03-06T07:22:01.004439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004500+0000","last_change":"2024-03-06T07:21:58.963329+0000","last_active":"2024-03-06T07:22:01.004500+0000","last_peered":"2024-03-06T07:22:01.004500+0000","last_clean":"2024-03-06T07:22:01.004500+0000","last_became_active":"2024-03-06T07:21:58.963004+0000","last_became_peered":"2024-03-06T07:21:58.963004+0000","last_unstale":"2024-03-06T07:22:01.004500+0000","last_undegraded":"2024-03-06T07:22:01.004500+0000","last_fullsized":"2024-03-06T07:22:01.004500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005042+0000","last_change":"2024-03-06T07:21:58.969451+0000","last_active":"2024-03-06T07:22:01.005042+0000","last_peered":"2024-03-06T07:22:01.005042+0000","last_clean":"2024-03-06T07:22:01.005042+0000","last_became_active":"2024-03-06T07:21:58.969086+0000","last_became_peered":"2024-03-06T07:21:58.969086+0000","last_unstale":"2024-03-06T07:22:01.005042+0000","last_undegraded":"2024-03-06T07:22:01.005042+0000","last_fullsized":"2024-03-06T07:22:01.005042+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:00.578026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944576+0000","last_change":"2024-03-06T07:21:58.966581+0000","last_active":"2024-03-06T07:21:59.944576+0000","last_peered":"2024-03-06T07:21:59.944576+0000","last_clean":"2024-03-06T07:21:59.944576+0000","last_became_active":"2024-03-06T07:21:58.966309+0000","last_became_peered":"2024-03-06T07:21:58.966309+0000","last_unstale":"2024-03-06T07:21:59.944576+0000","last_undegraded":"2024-03-06T07:21:59.944576+0000","last_fullsized":"2024-03-06T07:21:59.944576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944493+0000","last_change":"2024-03-06T07:21:58.968147+0000","last_active":"2024-03-06T07:21:59.944493+0000","last_peered":"2024-03-06T07:21:59.944493+0000","last_clean":"2024-03-06T07:21:59.944493+0000","last_became_active":"2024-03-06T07:21:58.967870+0000","last_became_peered":"2024-03-06T07:21:58.967870+0000","last_unstale":"2024-03-06T07:21:59.944493+0000","last_undegraded":"2024-03-06T07:21:59.944493+0000","last_fullsized":"2024-03-06T07:21:59.944493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943944+0000","last_change":"2024-03-06T07:21:58.950183+0000","last_active":"2024-03-06T07:21:59.943944+0000","last_peered":"2024-03-06T07:21:59.943944+0000","last_clean":"2024-03-06T07:21:59.943944+0000","last_became_active":"2024-03-06T07:21:58.949892+0000","last_became_peered":"2024-03-06T07:21:58.949892+0000","last_unstale":"2024-03-06T07:21:59.943944+0000","last_undegraded":"2024-03-06T07:21:59.943944+0000","last_fullsized":"2024-03-06T07:21:59.943944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:41.545324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004776+0000","last_change":"2024-03-06T07:21:58.963200+0000","last_active":"2024-03-06T07:22:01.004776+0000","last_peered":"2024-03-06T07:22:01.004776+0000","last_clean":"2024-03-06T07:22:01.004776+0000","last_became_active":"2024-03-06T07:21:58.962833+0000","last_became_peered":"2024-03-06T07:21:58.962833+0000","last_unstale":"2024-03-06T07:22:01.004776+0000","last_undegraded":"2024-03-06T07:22:01.004776+0000","last_fullsized":"2024-03-06T07:22:01.004776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548832+0000","last_change":"2024-03-06T07:21:58.952781+0000","last_active":"2024-03-06T07:22:00.548832+0000","last_peered":"2024-03-06T07:22:00.548832+0000","last_clean":"2024-03-06T07:22:00.548832+0000","last_became_active":"2024-03-06T07:21:58.952504+0000","last_became_peered":"2024-03-06T07:21:58.952504+0000","last_unstale":"2024-03-06T07:22:00.548832+0000","last_undegraded":"2024-03-06T07:22:00.548832+0000","last_fullsized":"2024-03-06T07:22:00.548832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:25.614483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942188+0000","last_change":"2024-03-06T07:21:58.950082+0000","last_active":"2024-03-06T07:21:59.942188+0000","last_peered":"2024-03-06T07:21:59.942188+0000","last_clean":"2024-03-06T07:21:59.942188+0000","last_became_active":"2024-03-06T07:21:58.949595+0000","last_became_peered":"2024-03-06T07:21:58.949595+0000","last_unstale":"2024-03-06T07:21:59.942188+0000","last_undegraded":"2024-03-06T07:21:59.942188+0000","last_fullsized":"2024-03-06T07:21:59.942188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004141+0000","last_change":"2024-03-06T07:21:58.961225+0000","last_active":"2024-03-06T07:22:01.004141+0000","last_peered":"2024-03-06T07:22:01.004141+0000","last_clean":"2024-03-06T07:22:01.004141+0000","last_became_active":"2024-03-06T07:21:58.960902+0000","last_became_peered":"2024-03-06T07:21:58.960902+0000","last_unstale":"2024-03-06T07:22:01.004141+0000","last_undegraded":"2024-03-06T07:22:01.004141+0000","last_fullsized":"2024-03-06T07:22:01.004141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004963+0000","last_change":"2024-03-06T07:21:58.970233+0000","last_active":"2024-03-06T07:22:01.004963+0000","last_peered":"2024-03-06T07:22:01.004963+0000","last_clean":"2024-03-06T07:22:01.004963+0000","last_became_active":"2024-03-06T07:21:58.969346+0000","last_became_peered":"2024-03-06T07:21:58.969346+0000","last_unstale":"2024-03-06T07:22:01.004963+0000","last_undegraded":"2024-03-06T07:22:01.004963+0000","last_fullsized":"2024-03-06T07:22:01.004963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:42.427321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983310+0000","last_change":"2024-03-06T07:21:58.948614+0000","last_active":"2024-03-06T07:21:59.983310+0000","last_peered":"2024-03-06T07:21:59.983310+0000","last_clean":"2024-03-06T07:21:59.983310+0000","last_became_active":"2024-03-06T07:21:58.948333+0000","last_became_peered":"2024-03-06T07:21:58.948333+0000","last_unstale":"2024-03-06T07:21:59.983310+0000","last_undegraded":"2024-03-06T07:21:59.983310+0000","last_fullsized":"2024-03-06T07:21:59.983310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:42.473136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005274+0000","last_change":"2024-03-06T07:21:58.957566+0000","last_active":"2024-03-06T07:22:01.005274+0000","last_peered":"2024-03-06T07:22:01.005274+0000","last_clean":"2024-03-06T07:22:01.005274+0000","last_became_active":"2024-03-06T07:21:58.957272+0000","last_became_peered":"2024-03-06T07:21:58.957272+0000","last_unstale":"2024-03-06T07:22:01.005274+0000","last_undegraded":"2024-03-06T07:22:01.005274+0000","last_fullsized":"2024-03-06T07:22:01.005274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:13.279685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944669+0000","last_change":"2024-03-06T07:21:58.948431+0000","last_active":"2024-03-06T07:21:59.944669+0000","last_peered":"2024-03-06T07:21:59.944669+0000","last_clean":"2024-03-06T07:21:59.944669+0000","last_became_active":"2024-03-06T07:21:58.947689+0000","last_became_peered":"2024-03-06T07:21:58.947689+0000","last_unstale":"2024-03-06T07:21:59.944669+0000","last_undegraded":"2024-03-06T07:21:59.944669+0000","last_fullsized":"2024-03-06T07:21:59.944669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548975+0000","last_change":"2024-03-06T07:21:58.965611+0000","last_active":"2024-03-06T07:22:00.548975+0000","last_peered":"2024-03-06T07:22:00.548975+0000","last_clean":"2024-03-06T07:22:00.548975+0000","last_became_active":"2024-03-06T07:21:58.965261+0000","last_became_peered":"2024-03-06T07:21:58.965261+0000","last_unstale":"2024-03-06T07:22:00.548975+0000","last_undegraded":"2024-03-06T07:22:00.548975+0000","last_fullsized":"2024-03-06T07:22:00.548975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004252+0000","last_change":"2024-03-06T07:21:58.969256+0000","last_active":"2024-03-06T07:22:01.004252+0000","last_peered":"2024-03-06T07:22:01.004252+0000","last_clean":"2024-03-06T07:22:01.004252+0000","last_became_active":"2024-03-06T07:21:58.968670+0000","last_became_peered":"2024-03-06T07:21:58.968670+0000","last_unstale":"2024-03-06T07:22:01.004252+0000","last_undegraded":"2024-03-06T07:22:01.004252+0000","last_fullsized":"2024-03-06T07:22:01.004252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:25.392050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004556+0000","last_change":"2024-03-06T07:21:58.962061+0000","last_active":"2024-03-06T07:22:01.004556+0000","last_peered":"2024-03-06T07:22:01.004556+0000","last_clean":"2024-03-06T07:22:01.004556+0000","last_became_active":"2024-03-06T07:21:58.961754+0000","last_became_peered":"2024-03-06T07:21:58.961754+0000","last_unstale":"2024-03-06T07:22:01.004556+0000","last_undegraded":"2024-03-06T07:22:01.004556+0000","last_fullsized":"2024-03-06T07:22:01.004556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004201+0000","last_change":"2024-03-06T07:21:58.961280+0000","last_active":"2024-03-06T07:22:01.004201+0000","last_peered":"2024-03-06T07:22:01.004201+0000","last_clean":"2024-03-06T07:22:01.004201+0000","last_became_active":"2024-03-06T07:21:58.960288+0000","last_became_peered":"2024-03-06T07:21:58.960288+0000","last_unstale":"2024-03-06T07:22:01.004201+0000","last_undegraded":"2024-03-06T07:22:01.004201+0000","last_fullsized":"2024-03-06T07:22:01.004201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005074+0000","last_change":"2024-03-06T07:21:58.962545+0000","last_active":"2024-03-06T07:22:01.005074+0000","last_peered":"2024-03-06T07:22:01.005074+0000","last_clean":"2024-03-06T07:22:01.005074+0000","last_became_active":"2024-03-06T07:21:58.962037+0000","last_became_peered":"2024-03-06T07:21:58.962037+0000","last_unstale":"2024-03-06T07:22:01.005074+0000","last_undegraded":"2024-03-06T07:22:01.005074+0000","last_fullsized":"2024-03-06T07:22:01.005074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944648+0000","last_change":"2024-03-06T07:21:58.959744+0000","last_active":"2024-03-06T07:21:59.944648+0000","last_peered":"2024-03-06T07:21:59.944648+0000","last_clean":"2024-03-06T07:21:59.944648+0000","last_became_active":"2024-03-06T07:21:58.959459+0000","last_became_peered":"2024-03-06T07:21:58.959459+0000","last_unstale":"2024-03-06T07:21:59.944648+0000","last_undegraded":"2024-03-06T07:21:59.944648+0000","last_fullsized":"2024-03-06T07:21:59.944648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:23.492051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942254+0000","last_change":"2024-03-06T07:21:58.950538+0000","last_active":"2024-03-06T07:21:59.942254+0000","last_peered":"2024-03-06T07:21:59.942254+0000","last_clean":"2024-03-06T07:21:59.942254+0000","last_became_active":"2024-03-06T07:21:58.950225+0000","last_became_peered":"2024-03-06T07:21:58.950225+0000","last_unstale":"2024-03-06T07:21:59.942254+0000","last_undegraded":"2024-03-06T07:21:59.942254+0000","last_fullsized":"2024-03-06T07:21:59.942254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004948+0000","last_change":"2024-03-06T07:21:58.961060+0000","last_active":"2024-03-06T07:22:01.004948+0000","last_peered":"2024-03-06T07:22:01.004948+0000","last_clean":"2024-03-06T07:22:01.004948+0000","last_became_active":"2024-03-06T07:21:58.960291+0000","last_became_peered":"2024-03-06T07:21:58.960291+0000","last_unstale":"2024-03-06T07:22:01.004948+0000","last_undegraded":"2024-03-06T07:22:01.004948+0000","last_fullsized":"2024-03-06T07:22:01.004948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548986+0000","last_change":"2024-03-06T07:21:58.957460+0000","last_active":"2024-03-06T07:22:00.548986+0000","last_peered":"2024-03-06T07:22:00.548986+0000","last_clean":"2024-03-06T07:22:00.548986+0000","last_became_active":"2024-03-06T07:21:58.957194+0000","last_became_peered":"2024-03-06T07:21:58.957194+0000","last_unstale":"2024-03-06T07:22:00.548986+0000","last_undegraded":"2024-03-06T07:22:00.548986+0000","last_fullsized":"2024-03-06T07:22:00.548986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004388+0000","last_change":"2024-03-06T07:21:58.971347+0000","last_active":"2024-03-06T07:22:01.004388+0000","last_peered":"2024-03-06T07:22:01.004388+0000","last_clean":"2024-03-06T07:22:01.004388+0000","last_became_active":"2024-03-06T07:21:58.970575+0000","last_became_peered":"2024-03-06T07:21:58.970575+0000","last_unstale":"2024-03-06T07:22:01.004388+0000","last_undegraded":"2024-03-06T07:22:01.004388+0000","last_fullsized":"2024-03-06T07:22:01.004388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:07.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983258+0000","last_change":"2024-03-06T07:21:58.964710+0000","last_active":"2024-03-06T07:21:59.983258+0000","last_peered":"2024-03-06T07:21:59.983258+0000","last_clean":"2024-03-06T07:21:59.983258+0000","last_became_active":"2024-03-06T07:21:58.964426+0000","last_became_peered":"2024-03-06T07:21:58.964426+0000","last_unstale":"2024-03-06T07:21:59.983258+0000","last_undegraded":"2024-03-06T07:21:59.983258+0000","last_fullsized":"2024-03-06T07:21:59.983258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:54.202847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943842+0000","last_change":"2024-03-06T07:21:58.958320+0000","last_active":"2024-03-06T07:21:59.943842+0000","last_peered":"2024-03-06T07:21:59.943842+0000","last_clean":"2024-03-06T07:21:59.943842+0000","last_became_active":"2024-03-06T07:21:58.958023+0000","last_became_peered":"2024-03-06T07:21:58.958023+0000","last_unstale":"2024-03-06T07:21:59.943842+0000","last_undegraded":"2024-03-06T07:21:59.943842+0000","last_fullsized":"2024-03-06T07:21:59.943842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943853+0000","last_change":"2024-03-06T07:21:58.949671+0000","last_active":"2024-03-06T07:21:59.943853+0000","last_peered":"2024-03-06T07:21:59.943853+0000","last_clean":"2024-03-06T07:21:59.943853+0000","last_became_active":"2024-03-06T07:21:58.949364+0000","last_became_peered":"2024-03-06T07:21:58.949364+0000","last_unstale":"2024-03-06T07:21:59.943853+0000","last_undegraded":"2024-03-06T07:21:59.943853+0000","last_fullsized":"2024-03-06T07:21:59.943853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:54.826119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004867+0000","last_change":"2024-03-06T07:21:57.960062+0000","last_active":"2024-03-06T07:22:01.004867+0000","last_peered":"2024-03-06T07:22:01.004867+0000","last_clean":"2024-03-06T07:22:01.004867+0000","last_became_active":"2024-03-06T07:21:57.959790+0000","last_became_peered":"2024-03-06T07:21:57.959790+0000","last_unstale":"2024-03-06T07:22:01.004867+0000","last_undegraded":"2024-03-06T07:22:01.004867+0000","last_fullsized":"2024-03-06T07:22:01.004867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:02.949716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549132+0000","last_change":"2024-03-06T07:21:58.958514+0000","last_active":"2024-03-06T07:22:00.549132+0000","last_peered":"2024-03-06T07:22:00.549132+0000","last_clean":"2024-03-06T07:22:00.549132+0000","last_became_active":"2024-03-06T07:21:58.958247+0000","last_became_peered":"2024-03-06T07:21:58.958247+0000","last_unstale":"2024-03-06T07:22:00.549132+0000","last_undegraded":"2024-03-06T07:22:00.549132+0000","last_fullsized":"2024-03-06T07:22:00.549132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944544+0000","last_change":"2024-03-06T07:21:57.949106+0000","last_active":"2024-03-06T07:21:59.944544+0000","last_peered":"2024-03-06T07:21:59.944544+0000","last_clean":"2024-03-06T07:21:59.944544+0000","last_became_active":"2024-03-06T07:21:57.948807+0000","last_became_peered":"2024-03-06T07:21:57.948807+0000","last_unstale":"2024-03-06T07:21:59.944544+0000","last_undegraded":"2024-03-06T07:21:59.944544+0000","last_fullsized":"2024-03-06T07:21:59.944544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004531+0000","last_change":"2024-03-06T07:21:57.962231+0000","last_active":"2024-03-06T07:22:01.004531+0000","last_peered":"2024-03-06T07:22:01.004531+0000","last_clean":"2024-03-06T07:22:01.004531+0000","last_became_active":"2024-03-06T07:21:57.961979+0000","last_became_peered":"2024-03-06T07:21:57.961979+0000","last_unstale":"2024-03-06T07:22:01.004531+0000","last_undegraded":"2024-03-06T07:22:01.004531+0000","last_fullsized":"2024-03-06T07:22:01.004531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004545+0000","last_change":"2024-03-06T07:21:58.962490+0000","last_active":"2024-03-06T07:22:01.004545+0000","last_peered":"2024-03-06T07:22:01.004545+0000","last_clean":"2024-03-06T07:22:01.004545+0000","last_became_active":"2024-03-06T07:21:58.960981+0000","last_became_peered":"2024-03-06T07:21:58.960981+0000","last_unstale":"2024-03-06T07:22:01.004545+0000","last_undegraded":"2024-03-06T07:22:01.004545+0000","last_fullsized":"2024-03-06T07:22:01.004545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004198+0000","last_change":"2024-03-06T07:21:57.961193+0000","last_active":"2024-03-06T07:22:01.004198+0000","last_peered":"2024-03-06T07:22:01.004198+0000","last_clean":"2024-03-06T07:22:01.004198+0000","last_became_active":"2024-03-06T07:21:57.960916+0000","last_became_peered":"2024-03-06T07:21:57.960916+0000","last_unstale":"2024-03-06T07:22:01.004198+0000","last_undegraded":"2024-03-06T07:22:01.004198+0000","last_fullsized":"2024-03-06T07:22:01.004198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942173+0000","last_change":"2024-03-06T07:21:58.951489+0000","last_active":"2024-03-06T07:21:59.942173+0000","last_peered":"2024-03-06T07:21:59.942173+0000","last_clean":"2024-03-06T07:21:59.942173+0000","last_became_active":"2024-03-06T07:21:58.951202+0000","last_became_peered":"2024-03-06T07:21:58.951202+0000","last_unstale":"2024-03-06T07:21:59.942173+0000","last_undegraded":"2024-03-06T07:21:59.942173+0000","last_fullsized":"2024-03-06T07:21:59.942173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944528+0000","last_change":"2024-03-06T07:21:57.959918+0000","last_active":"2024-03-06T07:21:59.944528+0000","last_peered":"2024-03-06T07:21:59.944528+0000","last_clean":"2024-03-06T07:21:59.944528+0000","last_became_active":"2024-03-06T07:21:57.959611+0000","last_became_peered":"2024-03-06T07:21:57.959611+0000","last_unstale":"2024-03-06T07:21:59.944528+0000","last_undegraded":"2024-03-06T07:21:59.944528+0000","last_fullsized":"2024-03-06T07:21:59.944528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004566+0000","last_change":"2024-03-06T07:21:58.970643+0000","last_active":"2024-03-06T07:22:01.004566+0000","last_peered":"2024-03-06T07:22:01.004566+0000","last_clean":"2024-03-06T07:22:01.004566+0000","last_became_active":"2024-03-06T07:21:58.970039+0000","last_became_peered":"2024-03-06T07:21:58.970039+0000","last_unstale":"2024-03-06T07:22:01.004566+0000","last_undegraded":"2024-03-06T07:22:01.004566+0000","last_fullsized":"2024-03-06T07:22:01.004566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004616+0000","last_change":"2024-03-06T07:21:57.962638+0000","last_active":"2024-03-06T07:22:01.004616+0000","last_peered":"2024-03-06T07:22:01.004616+0000","last_clean":"2024-03-06T07:22:01.004616+0000","last_became_active":"2024-03-06T07:21:57.962348+0000","last_became_peered":"2024-03-06T07:21:57.962348+0000","last_unstale":"2024-03-06T07:22:01.004616+0000","last_undegraded":"2024-03-06T07:22:01.004616+0000","last_fullsized":"2024-03-06T07:22:01.004616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549050+0000","last_change":"2024-03-06T07:21:58.962712+0000","last_active":"2024-03-06T07:22:00.549050+0000","last_peered":"2024-03-06T07:22:00.549050+0000","last_clean":"2024-03-06T07:22:00.549050+0000","last_became_active":"2024-03-06T07:21:58.962398+0000","last_became_peered":"2024-03-06T07:21:58.962398+0000","last_unstale":"2024-03-06T07:22:00.549050+0000","last_undegraded":"2024-03-06T07:22:00.549050+0000","last_fullsized":"2024-03-06T07:22:00.549050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942294+0000","last_change":"2024-03-06T07:21:57.958869+0000","last_active":"2024-03-06T07:21:59.942294+0000","last_peered":"2024-03-06T07:21:59.942294+0000","last_clean":"2024-03-06T07:21:59.942294+0000","last_became_active":"2024-03-06T07:21:57.958605+0000","last_became_peered":"2024-03-06T07:21:57.958605+0000","last_unstale":"2024-03-06T07:21:59.942294+0000","last_undegraded":"2024-03-06T07:21:59.942294+0000","last_fullsized":"2024-03-06T07:21:59.942294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944175+0000","last_change":"2024-03-06T07:21:58.960482+0000","last_active":"2024-03-06T07:21:59.944175+0000","last_peered":"2024-03-06T07:21:59.944175+0000","last_clean":"2024-03-06T07:21:59.944175+0000","last_became_active":"2024-03-06T07:21:58.960187+0000","last_became_peered":"2024-03-06T07:21:58.960187+0000","last_unstale":"2024-03-06T07:21:59.944175+0000","last_undegraded":"2024-03-06T07:21:59.944175+0000","last_fullsized":"2024-03-06T07:21:59.944175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:54.202847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944275+0000","last_change":"2024-03-06T07:21:57.963435+0000","last_active":"2024-03-06T07:21:59.944275+0000","last_peered":"2024-03-06T07:21:59.944275+0000","last_clean":"2024-03-06T07:21:59.944275+0000","last_became_active":"2024-03-06T07:21:57.963195+0000","last_became_peered":"2024-03-06T07:21:57.963195+0000","last_unstale":"2024-03-06T07:21:59.944275+0000","last_undegraded":"2024-03-06T07:21:59.944275+0000","last_fullsized":"2024-03-06T07:21:59.944275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005119+0000","last_change":"2024-03-06T07:21:58.961185+0000","last_active":"2024-03-06T07:22:01.005119+0000","last_peered":"2024-03-06T07:22:01.005119+0000","last_clean":"2024-03-06T07:22:01.005119+0000","last_became_active":"2024-03-06T07:21:58.960928+0000","last_became_peered":"2024-03-06T07:21:58.960928+0000","last_unstale":"2024-03-06T07:22:01.005119+0000","last_undegraded":"2024-03-06T07:22:01.005119+0000","last_fullsized":"2024-03-06T07:22:01.005119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:42.427321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549040+0000","last_change":"2024-03-06T07:21:57.950617+0000","last_active":"2024-03-06T07:22:00.549040+0000","last_peered":"2024-03-06T07:22:00.549040+0000","last_clean":"2024-03-06T07:22:00.549040+0000","last_became_active":"2024-03-06T07:21:57.950347+0000","last_became_peered":"2024-03-06T07:21:57.950347+0000","last_unstale":"2024-03-06T07:22:00.549040+0000","last_undegraded":"2024-03-06T07:22:00.549040+0000","last_fullsized":"2024-03-06T07:22:00.549040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:10.110560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548899+0000","last_change":"2024-03-06T07:21:58.957844+0000","last_active":"2024-03-06T07:22:00.548899+0000","last_peered":"2024-03-06T07:22:00.548899+0000","last_clean":"2024-03-06T07:22:00.548899+0000","last_became_active":"2024-03-06T07:21:58.957575+0000","last_became_peered":"2024-03-06T07:21:58.957575+0000","last_unstale":"2024-03-06T07:22:00.548899+0000","last_undegraded":"2024-03-06T07:22:00.548899+0000","last_fullsized":"2024-03-06T07:22:00.548899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:41.058405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004312+0000","last_change":"2024-03-06T07:21:57.964121+0000","last_active":"2024-03-06T07:22:01.004312+0000","last_peered":"2024-03-06T07:22:01.004312+0000","last_clean":"2024-03-06T07:22:01.004312+0000","last_became_active":"2024-03-06T07:21:57.963752+0000","last_became_peered":"2024-03-06T07:21:57.963752+0000","last_unstale":"2024-03-06T07:22:01.004312+0000","last_undegraded":"2024-03-06T07:22:01.004312+0000","last_fullsized":"2024-03-06T07:22:01.004312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005124+0000","last_change":"2024-03-06T07:21:58.962110+0000","last_active":"2024-03-06T07:22:01.005124+0000","last_peered":"2024-03-06T07:22:01.005124+0000","last_clean":"2024-03-06T07:22:01.005124+0000","last_became_active":"2024-03-06T07:21:58.961800+0000","last_became_peered":"2024-03-06T07:21:58.961800+0000","last_unstale":"2024-03-06T07:22:01.005124+0000","last_undegraded":"2024-03-06T07:22:01.005124+0000","last_fullsized":"2024-03-06T07:22:01.005124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:13.761240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944044+0000","last_change":"2024-03-06T07:21:57.949180+0000","last_active":"2024-03-06T07:21:59.944044+0000","last_peered":"2024-03-06T07:21:59.944044+0000","last_clean":"2024-03-06T07:21:59.944044+0000","last_became_active":"2024-03-06T07:21:57.948897+0000","last_became_peered":"2024-03-06T07:21:57.948897+0000","last_unstale":"2024-03-06T07:21:59.944044+0000","last_undegraded":"2024-03-06T07:21:59.944044+0000","last_fullsized":"2024-03-06T07:21:59.944044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944254+0000","last_change":"2024-03-06T07:21:58.957923+0000","last_active":"2024-03-06T07:21:59.944254+0000","last_peered":"2024-03-06T07:21:59.944254+0000","last_clean":"2024-03-06T07:21:59.944254+0000","last_became_active":"2024-03-06T07:21:58.957454+0000","last_became_peered":"2024-03-06T07:21:58.957454+0000","last_unstale":"2024-03-06T07:21:59.944254+0000","last_undegraded":"2024-03-06T07:21:59.944254+0000","last_fullsized":"2024-03-06T07:21:59.944254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.836178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983080+0000","last_change":"2024-03-06T07:21:57.956795+0000","last_active":"2024-03-06T07:21:59.983080+0000","last_peered":"2024-03-06T07:21:59.983080+0000","last_clean":"2024-03-06T07:21:59.983080+0000","last_became_active":"2024-03-06T07:21:57.956529+0000","last_became_peered":"2024-03-06T07:21:57.956529+0000","last_unstale":"2024-03-06T07:21:59.983080+0000","last_undegraded":"2024-03-06T07:21:59.983080+0000","last_fullsized":"2024-03-06T07:21:59.983080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:02.949716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004551+0000","last_change":"2024-03-06T07:21:58.961675+0000","last_active":"2024-03-06T07:22:01.004551+0000","last_peered":"2024-03-06T07:22:01.004551+0000","last_clean":"2024-03-06T07:22:01.004551+0000","last_became_active":"2024-03-06T07:21:58.961169+0000","last_became_peered":"2024-03-06T07:21:58.961169+0000","last_unstale":"2024-03-06T07:22:01.004551+0000","last_undegraded":"2024-03-06T07:22:01.004551+0000","last_fullsized":"2024-03-06T07:22:01.004551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:16.085125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943737+0000","last_change":"2024-03-06T07:21:57.952110+0000","last_active":"2024-03-06T07:21:59.943737+0000","last_peered":"2024-03-06T07:21:59.943737+0000","last_clean":"2024-03-06T07:21:59.943737+0000","last_became_active":"2024-03-06T07:21:57.951787+0000","last_became_peered":"2024-03-06T07:21:57.951787+0000","last_unstale":"2024-03-06T07:21:59.943737+0000","last_undegraded":"2024-03-06T07:21:59.943737+0000","last_fullsized":"2024-03-06T07:21:59.943737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:34.102031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004914+0000","last_change":"2024-03-06T07:21:58.967742+0000","last_active":"2024-03-06T07:22:01.004914+0000","last_peered":"2024-03-06T07:22:01.004914+0000","last_clean":"2024-03-06T07:22:01.004914+0000","last_became_active":"2024-03-06T07:21:58.967476+0000","last_became_peered":"2024-03-06T07:21:58.967476+0000","last_unstale":"2024-03-06T07:22:01.004914+0000","last_undegraded":"2024-03-06T07:22:01.004914+0000","last_fullsized":"2024-03-06T07:22:01.004914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983255+0000","last_change":"2024-03-06T07:21:57.962699+0000","last_active":"2024-03-06T07:21:59.983255+0000","last_peered":"2024-03-06T07:21:59.983255+0000","last_clean":"2024-03-06T07:21:59.983255+0000","last_became_active":"2024-03-06T07:21:57.962401+0000","last_became_peered":"2024-03-06T07:21:57.962401+0000","last_unstale":"2024-03-06T07:21:59.983255+0000","last_undegraded":"2024-03-06T07:21:59.983255+0000","last_fullsized":"2024-03-06T07:21:59.983255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005163+0000","last_change":"2024-03-06T07:21:58.964631+0000","last_active":"2024-03-06T07:22:01.005163+0000","last_peered":"2024-03-06T07:22:01.005163+0000","last_clean":"2024-03-06T07:22:01.005163+0000","last_became_active":"2024-03-06T07:21:58.964187+0000","last_became_peered":"2024-03-06T07:21:58.964187+0000","last_unstale":"2024-03-06T07:22:01.005163+0000","last_undegraded":"2024-03-06T07:22:01.005163+0000","last_fullsized":"2024-03-06T07:22:01.005163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:34.814920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942356+0000","last_change":"2024-03-06T07:21:57.951889+0000","last_active":"2024-03-06T07:21:59.942356+0000","last_peered":"2024-03-06T07:21:59.942356+0000","last_clean":"2024-03-06T07:21:59.942356+0000","last_became_active":"2024-03-06T07:21:57.951637+0000","last_became_peered":"2024-03-06T07:21:57.951637+0000","last_unstale":"2024-03-06T07:21:59.942356+0000","last_undegraded":"2024-03-06T07:21:59.942356+0000","last_fullsized":"2024-03-06T07:21:59.942356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004803+0000","last_change":"2024-03-06T07:21:58.962144+0000","last_active":"2024-03-06T07:22:01.004803+0000","last_peered":"2024-03-06T07:22:01.004803+0000","last_clean":"2024-03-06T07:22:01.004803+0000","last_became_active":"2024-03-06T07:21:58.960533+0000","last_became_peered":"2024-03-06T07:21:58.960533+0000","last_unstale":"2024-03-06T07:22:01.004803+0000","last_undegraded":"2024-03-06T07:22:01.004803+0000","last_fullsized":"2024-03-06T07:22:01.004803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:28.792591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983285+0000","last_change":"2024-03-06T07:21:57.958443+0000","last_active":"2024-03-06T07:21:59.983285+0000","last_peered":"2024-03-06T07:21:59.983285+0000","last_clean":"2024-03-06T07:21:59.983285+0000","last_became_active":"2024-03-06T07:21:57.958162+0000","last_became_peered":"2024-03-06T07:21:57.958162+0000","last_unstale":"2024-03-06T07:21:59.983285+0000","last_undegraded":"2024-03-06T07:21:59.983285+0000","last_fullsized":"2024-03-06T07:21:59.983285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005108+0000","last_change":"2024-03-06T07:21:58.968776+0000","last_active":"2024-03-06T07:22:01.005108+0000","last_peered":"2024-03-06T07:22:01.005108+0000","last_clean":"2024-03-06T07:22:01.005108+0000","last_became_active":"2024-03-06T07:21:58.968522+0000","last_became_peered":"2024-03-06T07:21:58.968522+0000","last_unstale":"2024-03-06T07:22:01.005108+0000","last_undegraded":"2024-03-06T07:22:01.005108+0000","last_fullsized":"2024-03-06T07:22:01.005108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:28.792591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004412+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:22:01.004412+0000","last_peered":"2024-03-06T07:22:01.004412+0000","last_clean":"2024-03-06T07:22:01.004412+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:22:01.004412+0000","last_undegraded":"2024-03-06T07:22:01.004412+0000","last_fullsized":"2024-03-06T07:22:01.004412+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004292+0000","last_change":"2024-03-06T07:21:57.958220+0000","last_active":"2024-03-06T07:22:01.004292+0000","last_peered":"2024-03-06T07:22:01.004292+0000","last_clean":"2024-03-06T07:22:01.004292+0000","last_became_active":"2024-03-06T07:21:57.957561+0000","last_became_peered":"2024-03-06T07:21:57.957561+0000","last_unstale":"2024-03-06T07:22:01.004292+0000","last_undegraded":"2024-03-06T07:22:01.004292+0000","last_fullsized":"2024-03-06T07:22:01.004292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004721+0000","last_change":"2024-03-06T07:21:58.968424+0000","last_active":"2024-03-06T07:22:01.004721+0000","last_peered":"2024-03-06T07:22:01.004721+0000","last_clean":"2024-03-06T07:22:01.004721+0000","last_became_active":"2024-03-06T07:21:58.968235+0000","last_became_peered":"2024-03-06T07:21:58.968235+0000","last_unstale":"2024-03-06T07:22:01.004721+0000","last_undegraded":"2024-03-06T07:22:01.004721+0000","last_fullsized":"2024-03-06T07:22:01.004721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004311+0000","last_change":"2024-03-06T07:21:57.962310+0000","last_active":"2024-03-06T07:22:01.004311+0000","last_peered":"2024-03-06T07:22:01.004311+0000","last_clean":"2024-03-06T07:22:01.004311+0000","last_became_active":"2024-03-06T07:21:57.962015+0000","last_became_peered":"2024-03-06T07:21:57.962015+0000","last_unstale":"2024-03-06T07:22:01.004311+0000","last_undegraded":"2024-03-06T07:22:01.004311+0000","last_fullsized":"2024-03-06T07:22:01.004311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:04.878864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548793+0000","last_change":"2024-03-06T07:21:58.949526+0000","last_active":"2024-03-06T07:22:00.548793+0000","last_peered":"2024-03-06T07:22:00.548793+0000","last_clean":"2024-03-06T07:22:00.548793+0000","last_became_active":"2024-03-06T07:21:58.949247+0000","last_became_peered":"2024-03-06T07:21:58.949247+0000","last_unstale":"2024-03-06T07:22:00.548793+0000","last_undegraded":"2024-03-06T07:22:00.548793+0000","last_fullsized":"2024-03-06T07:22:00.548793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:11.328500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.983204+0000","last_change":"2024-03-06T07:21:57.963056+0000","last_active":"2024-03-06T07:21:59.983204+0000","last_peered":"2024-03-06T07:21:59.983204+0000","last_clean":"2024-03-06T07:21:59.983204+0000","last_became_active":"2024-03-06T07:21:57.962785+0000","last_became_peered":"2024-03-06T07:21:57.962785+0000","last_unstale":"2024-03-06T07:21:59.983204+0000","last_undegraded":"2024-03-06T07:21:59.983204+0000","last_fullsized":"2024-03-06T07:21:59.983204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942195+0000","last_change":"2024-03-06T07:21:58.948764+0000","last_active":"2024-03-06T07:21:59.942195+0000","last_peered":"2024-03-06T07:21:59.942195+0000","last_clean":"2024-03-06T07:21:59.942195+0000","last_became_active":"2024-03-06T07:21:58.948489+0000","last_became_peered":"2024-03-06T07:21:58.948489+0000","last_unstale":"2024-03-06T07:21:59.942195+0000","last_undegraded":"2024-03-06T07:21:59.942195+0000","last_fullsized":"2024-03-06T07:21:59.942195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004402+0000","last_change":"2024-03-06T07:21:57.965008+0000","last_active":"2024-03-06T07:22:01.004402+0000","last_peered":"2024-03-06T07:22:01.004402+0000","last_clean":"2024-03-06T07:22:01.004402+0000","last_became_active":"2024-03-06T07:21:57.964754+0000","last_became_peered":"2024-03-06T07:21:57.964754+0000","last_unstale":"2024-03-06T07:22:01.004402+0000","last_undegraded":"2024-03-06T07:22:01.004402+0000","last_fullsized":"2024-03-06T07:22:01.004402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944174+0000","last_change":"2024-03-06T07:21:58.958773+0000","last_active":"2024-03-06T07:21:59.944174+0000","last_peered":"2024-03-06T07:21:59.944174+0000","last_clean":"2024-03-06T07:21:59.944174+0000","last_became_active":"2024-03-06T07:21:58.958489+0000","last_became_peered":"2024-03-06T07:21:58.958489+0000","last_unstale":"2024-03-06T07:21:59.944174+0000","last_undegraded":"2024-03-06T07:21:59.944174+0000","last_fullsized":"2024-03-06T07:21:59.944174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:29.955363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942256+0000","last_change":"2024-03-06T07:21:57.952000+0000","last_active":"2024-03-06T07:21:59.942256+0000","last_peered":"2024-03-06T07:21:59.942256+0000","last_clean":"2024-03-06T07:21:59.942256+0000","last_became_active":"2024-03-06T07:21:57.951703+0000","last_became_peered":"2024-03-06T07:21:57.951703+0000","last_unstale":"2024-03-06T07:21:59.942256+0000","last_undegraded":"2024-03-06T07:21:59.942256+0000","last_fullsized":"2024-03-06T07:21:59.942256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:53.821059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004691+0000","last_change":"2024-03-06T07:21:58.967480+0000","last_active":"2024-03-06T07:22:01.004691+0000","last_peered":"2024-03-06T07:22:01.004691+0000","last_clean":"2024-03-06T07:22:01.004691+0000","last_became_active":"2024-03-06T07:21:58.967104+0000","last_became_peered":"2024-03-06T07:21:58.967104+0000","last_unstale":"2024-03-06T07:22:01.004691+0000","last_undegraded":"2024-03-06T07:22:01.004691+0000","last_fullsized":"2024-03-06T07:22:01.004691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:53.179429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005275+0000","last_change":"2024-03-06T07:21:57.960485+0000","last_active":"2024-03-06T07:22:01.005275+0000","last_peered":"2024-03-06T07:22:01.005275+0000","last_clean":"2024-03-06T07:22:01.005275+0000","last_became_active":"2024-03-06T07:21:57.960144+0000","last_became_peered":"2024-03-06T07:21:57.960144+0000","last_unstale":"2024-03-06T07:22:01.005275+0000","last_undegraded":"2024-03-06T07:22:01.005275+0000","last_fullsized":"2024-03-06T07:22:01.005275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:41.468075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942320+0000","last_change":"2024-03-06T07:21:58.947670+0000","last_active":"2024-03-06T07:21:59.942320+0000","last_peered":"2024-03-06T07:21:59.942320+0000","last_clean":"2024-03-06T07:21:59.942320+0000","last_became_active":"2024-03-06T07:21:58.947435+0000","last_became_peered":"2024-03-06T07:21:58.947435+0000","last_unstale":"2024-03-06T07:21:59.942320+0000","last_undegraded":"2024-03-06T07:21:59.942320+0000","last_fullsized":"2024-03-06T07:21:59.942320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:24.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942445+0000","last_change":"2024-03-06T07:21:57.951589+0000","last_active":"2024-03-06T07:21:59.942445+0000","last_peered":"2024-03-06T07:21:59.942445+0000","last_clean":"2024-03-06T07:21:59.942445+0000","last_became_active":"2024-03-06T07:21:57.951313+0000","last_became_peered":"2024-03-06T07:21:57.951313+0000","last_unstale":"2024-03-06T07:21:59.942445+0000","last_undegraded":"2024-03-06T07:21:59.942445+0000","last_fullsized":"2024-03-06T07:21:59.942445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:53.197786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005072+0000","last_change":"2024-03-06T07:21:58.966146+0000","last_active":"2024-03-06T07:22:01.005072+0000","last_peered":"2024-03-06T07:22:01.005072+0000","last_clean":"2024-03-06T07:22:01.005072+0000","last_became_active":"2024-03-06T07:21:58.965839+0000","last_became_peered":"2024-03-06T07:21:58.965839+0000","last_unstale":"2024-03-06T07:22:01.005072+0000","last_undegraded":"2024-03-06T07:22:01.005072+0000","last_fullsized":"2024-03-06T07:22:01.005072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:02.584924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944510+0000","last_change":"2024-03-06T07:21:57.956954+0000","last_active":"2024-03-06T07:21:59.944510+0000","last_peered":"2024-03-06T07:21:59.944510+0000","last_clean":"2024-03-06T07:21:59.944510+0000","last_became_active":"2024-03-06T07:21:57.956181+0000","last_became_peered":"2024-03-06T07:21:57.956181+0000","last_unstale":"2024-03-06T07:21:59.944510+0000","last_undegraded":"2024-03-06T07:21:59.944510+0000","last_fullsized":"2024-03-06T07:21:59.944510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:40.540264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549033+0000","last_change":"2024-03-06T07:21:58.957111+0000","last_active":"2024-03-06T07:22:00.549033+0000","last_peered":"2024-03-06T07:22:00.549033+0000","last_clean":"2024-03-06T07:22:00.549033+0000","last_became_active":"2024-03-06T07:21:58.956788+0000","last_became_peered":"2024-03-06T07:21:58.956788+0000","last_unstale":"2024-03-06T07:22:00.549033+0000","last_undegraded":"2024-03-06T07:22:00.549033+0000","last_fullsized":"2024-03-06T07:22:00.549033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004332+0000","last_change":"2024-03-06T07:21:57.960912+0000","last_active":"2024-03-06T07:22:01.004332+0000","last_peered":"2024-03-06T07:22:01.004332+0000","last_clean":"2024-03-06T07:22:01.004332+0000","last_became_active":"2024-03-06T07:21:57.960619+0000","last_became_peered":"2024-03-06T07:21:57.960619+0000","last_unstale":"2024-03-06T07:22:01.004332+0000","last_undegraded":"2024-03-06T07:22:01.004332+0000","last_fullsized":"2024-03-06T07:22:01.004332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:41.836911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.943984+0000","last_change":"2024-03-06T07:21:58.949723+0000","last_active":"2024-03-06T07:21:59.943984+0000","last_peered":"2024-03-06T07:21:59.943984+0000","last_clean":"2024-03-06T07:21:59.943984+0000","last_became_active":"2024-03-06T07:21:58.949415+0000","last_became_peered":"2024-03-06T07:21:58.949415+0000","last_unstale":"2024-03-06T07:21:59.943984+0000","last_undegraded":"2024-03-06T07:21:59.943984+0000","last_fullsized":"2024-03-06T07:21:59.943984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:48.783928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942225+0000","last_change":"2024-03-06T07:21:57.957156+0000","last_active":"2024-03-06T07:21:59.942225+0000","last_peered":"2024-03-06T07:21:59.942225+0000","last_clean":"2024-03-06T07:21:59.942225+0000","last_became_active":"2024-03-06T07:21:57.956893+0000","last_became_peered":"2024-03-06T07:21:57.956893+0000","last_unstale":"2024-03-06T07:21:59.942225+0000","last_undegraded":"2024-03-06T07:21:59.942225+0000","last_fullsized":"2024-03-06T07:21:59.942225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004691+0000","last_change":"2024-03-06T07:21:58.953837+0000","last_active":"2024-03-06T07:22:01.004691+0000","last_peered":"2024-03-06T07:22:01.004691+0000","last_clean":"2024-03-06T07:22:01.004691+0000","last_became_active":"2024-03-06T07:21:58.953594+0000","last_became_peered":"2024-03-06T07:21:58.953594+0000","last_unstale":"2024-03-06T07:22:01.004691+0000","last_undegraded":"2024-03-06T07:22:01.004691+0000","last_fullsized":"2024-03-06T07:22:01.004691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:13.480572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.942272+0000","last_change":"2024-03-06T07:21:57.962804+0000","last_active":"2024-03-06T07:21:59.942272+0000","last_peered":"2024-03-06T07:21:59.942272+0000","last_clean":"2024-03-06T07:21:59.942272+0000","last_became_active":"2024-03-06T07:21:57.962366+0000","last_became_peered":"2024-03-06T07:21:57.962366+0000","last_unstale":"2024-03-06T07:21:59.942272+0000","last_undegraded":"2024-03-06T07:21:59.942272+0000","last_fullsized":"2024-03-06T07:21:59.942272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.715690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004802+0000","last_change":"2024-03-06T07:21:58.957659+0000","last_active":"2024-03-06T07:22:01.004802+0000","last_peered":"2024-03-06T07:22:01.004802+0000","last_clean":"2024-03-06T07:22:01.004802+0000","last_became_active":"2024-03-06T07:21:58.957349+0000","last_became_peered":"2024-03-06T07:21:58.957349+0000","last_unstale":"2024-03-06T07:22:01.004802+0000","last_undegraded":"2024-03-06T07:22:01.004802+0000","last_fullsized":"2024-03-06T07:22:01.004802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:11.091138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005015+0000","last_change":"2024-03-06T07:21:57.957600+0000","last_active":"2024-03-06T07:22:01.005015+0000","last_peered":"2024-03-06T07:22:01.005015+0000","last_clean":"2024-03-06T07:22:01.005015+0000","last_became_active":"2024-03-06T07:21:57.957347+0000","last_became_peered":"2024-03-06T07:21:57.957347+0000","last_unstale":"2024-03-06T07:22:01.005015+0000","last_undegraded":"2024-03-06T07:22:01.005015+0000","last_fullsized":"2024-03-06T07:22:01.005015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:28.950302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:59.944432+0000","last_change":"2024-03-06T07:21:58.950912+0000","last_active":"2024-03-06T07:21:59.944432+0000","last_peered":"2024-03-06T07:21:59.944432+0000","last_clean":"2024-03-06T07:21:59.944432+0000","last_became_active":"2024-03-06T07:21:58.950531+0000","last_became_peered":"2024-03-06T07:21:58.950531+0000","last_unstale":"2024-03-06T07:21:59.944432+0000","last_undegraded":"2024-03-06T07:21:59.944432+0000","last_fullsized":"2024-03-06T07:21:59.944432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:08.529550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004726+0000","last_change":"2024-03-06T07:21:58.968927+0000","last_active":"2024-03-06T07:22:01.004726+0000","last_peered":"2024-03-06T07:22:01.004726+0000","last_clean":"2024-03-06T07:22:01.004726+0000","last_became_active":"2024-03-06T07:21:58.967167+0000","last_became_peered":"2024-03-06T07:21:58.967167+0000","last_unstale":"2024-03-06T07:22:01.004726+0000","last_undegraded":"2024-03-06T07:22:01.004726+0000","last_fullsized":"2024-03-06T07:22:01.004726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:00.578026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004797+0000","last_change":"2024-03-06T07:21:57.962444+0000","last_active":"2024-03-06T07:22:01.004797+0000","last_peered":"2024-03-06T07:22:01.004797+0000","last_clean":"2024-03-06T07:22:01.004797+0000","last_became_active":"2024-03-06T07:21:57.962157+0000","last_became_peered":"2024-03-06T07:21:57.962157+0000","last_unstale":"2024-03-06T07:22:01.004797+0000","last_undegraded":"2024-03-06T07:22:01.004797+0000","last_fullsized":"2024-03-06T07:22:01.004797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:47.778867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004672+0000","last_change":"2024-03-06T07:21:58.965441+0000","last_active":"2024-03-06T07:22:01.004672+0000","last_peered":"2024-03-06T07:22:01.004672+0000","last_clean":"2024-03-06T07:22:01.004672+0000","last_became_active":"2024-03-06T07:21:58.964894+0000","last_became_peered":"2024-03-06T07:21:58.964894+0000","last_unstale":"2024-03-06T07:22:01.004672+0000","last_undegraded":"2024-03-06T07:22:01.004672+0000","last_fullsized":"2024-03-06T07:22:01.004672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:51.617926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.005175+0000","last_change":"2024-03-06T07:21:57.958429+0000","last_active":"2024-03-06T07:22:01.005175+0000","last_peered":"2024-03-06T07:22:01.005175+0000","last_clean":"2024-03-06T07:22:01.005175+0000","last_became_active":"2024-03-06T07:21:57.958163+0000","last_became_peered":"2024-03-06T07:21:57.958163+0000","last_unstale":"2024-03-06T07:22:01.005175+0000","last_undegraded":"2024-03-06T07:22:01.005175+0000","last_fullsized":"2024-03-06T07:22:01.005175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:53.197786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.548917+0000","last_change":"2024-03-06T07:21:58.962796+0000","last_active":"2024-03-06T07:22:00.548917+0000","last_peered":"2024-03-06T07:22:00.548917+0000","last_clean":"2024-03-06T07:22:00.548917+0000","last_became_active":"2024-03-06T07:21:58.962454+0000","last_became_peered":"2024-03-06T07:21:58.962454+0000","last_unstale":"2024-03-06T07:22:00.548917+0000","last_undegraded":"2024-03-06T07:22:00.548917+0000","last_fullsized":"2024-03-06T07:22:00.548917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:29.411955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004974+0000","last_change":"2024-03-06T07:21:57.962433+0000","last_active":"2024-03-06T07:22:01.004974+0000","last_peered":"2024-03-06T07:22:01.004974+0000","last_clean":"2024-03-06T07:22:01.004974+0000","last_became_active":"2024-03-06T07:21:57.962149+0000","last_became_peered":"2024-03-06T07:21:57.962149+0000","last_unstale":"2024-03-06T07:22:01.004974+0000","last_undegraded":"2024-03-06T07:22:01.004974+0000","last_fullsized":"2024-03-06T07:22:01.004974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:27.779709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:01.004944+0000","last_change":"2024-03-06T07:21:58.963553+0000","last_active":"2024-03-06T07:22:01.004944+0000","last_peered":"2024-03-06T07:22:01.004944+0000","last_clean":"2024-03-06T07:22:01.004944+0000","last_became_active":"2024-03-06T07:21:58.963282+0000","last_became_peered":"2024-03-06T07:21:58.963282+0000","last_unstale":"2024-03-06T07:22:01.004944+0000","last_undegraded":"2024-03-06T07:22:01.004944+0000","last_fullsized":"2024-03-06T07:22:01.004944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:57.932426+0000","last_clean_scrub_stamp":"2024-03-06T07:21:57.932426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:25.392050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:22:00.549082+0000","last_change":"2024-03-06T07:21:57.956370+0000","last_active":"2024-03-06T07:22:00.549082+0000","last_peered":"2024-03-06T07:22:00.549082+0000","last_clean":"2024-03-06T07:22:00.549082+0000","last_became_active":"2024-03-06T07:21:57.955872+0000","last_became_peered":"2024-03-06T07:21:57.955872+0000","last_unstale":"2024-03-06T07:22:00.549082+0000","last_undegraded":"2024-03-06T07:22:00.549082+0000","last_fullsized":"2024-03-06T07:22:00.549082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:56.927366+0000","last_clean_scrub_stamp":"2024-03-06T07:21:56.927366+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:46.175278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:10.900 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:22:10.900 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:22:10.900 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:22:10.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:22:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.036+0000 7f8c16615640 1 Processor -- start 2024-03-06T07:22:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.036+0000 7f8c16615640 1 -- start start 2024-03-06T07:22:11.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.036+0000 7f8c16615640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8c101614a0 0x7f8c1006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.036+0000 7f8c16615640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8c100b1a20 0x7f8c100b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.036+0000 7f8c16615640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c100af4b0 con 0x7f8c100b0270 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.036+0000 7f8c16615640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c100af790 con 0x7f8c101614a0 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.036+0000 7f8c16615640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c100afa70 con 0x7f8c100b1a20 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c0ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8c101614a0 0x7f8c1006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c0f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8c100b0270 0x7f8c100b1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45566/0 (socket says 172.21.15.77:45566) 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c0f7fe640 1 -- 172.21.15.77:0/2085564868 learned_addr learned my addr 172.21.15.77:0/2085564868 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c14bb2640 1 --2- 172.21.15.77:0/2085564868 >> v2:172.21.15.158:3300/0 conn(0x7f8c100b1a20 0x7f8c100b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c0ffff640 1 -- 172.21.15.77:0/2085564868 >> v2:172.21.15.158:3300/0 conn(0x7f8c100b1a20 msgr2=0x7f8c100b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c0ffff640 1 --2- 172.21.15.77:0/2085564868 >> v2:172.21.15.158:3300/0 conn(0x7f8c100b1a20 0x7f8c100b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c0ffff640 1 -- 172.21.15.77:0/2085564868 >> v1:172.21.15.77:6789/0 conn(0x7f8c100b0270 legacy=0x7f8c100b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c0ffff640 1 -- 172.21.15.77:0/2085564868 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c100afd50 con 0x7f8c101614a0 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.037+0000 7f8c0effd640 1 -- 172.21.15.77:0/2085564868 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571511503 0 0) 0x7f8c100af4b0 con 0x7f8c100b0270 2024-03-06T07:22:11.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c0ffff640 1 --2- 172.21.15.77:0/2085564868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c101614a0 0x7f8c1006d3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8c100b34f0 tx=0x7f8c0002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314f15c9b91e90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c0effd640 1 -- 172.21.15.77:0/2085564868 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000315a0 con 0x7f8c101614a0 2024-03-06T07:22:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c0effd640 1 -- 172.21.15.77:0/2085564868 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c00037070 con 0x7f8c101614a0 2024-03-06T07:22:11.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c0effd640 1 -- 172.21.15.77:0/2085564868 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c00031960 con 0x7f8c101614a0 2024-03-06T07:22:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c16615640 1 -- 172.21.15.77:0/2085564868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c101614a0 msgr2=0x7f8c1006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c16615640 1 --2- 172.21.15.77:0/2085564868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c101614a0 0x7f8c1006d3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8c100b34f0 tx=0x7f8c0002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c16615640 1 -- 172.21.15.77:0/2085564868 shutdown_connections 2024-03-06T07:22:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c16615640 1 --2- 172.21.15.77:0/2085564868 >> v2:172.21.15.158:3300/0 conn(0x7f8c100b1a20 0x7f8c100b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c16615640 1 --2- 172.21.15.77:0/2085564868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c101614a0 0x7f8c1006d3c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c16615640 1 -- 172.21.15.77:0/2085564868 >> 172.21.15.77:0/2085564868 conn(0x7f8c1009a9f0 msgr2=0x7f8c1009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c16615640 1 -- 172.21.15.77:0/2085564868 shutdown_connections 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.038+0000 7f8c16615640 1 -- 172.21.15.77:0/2085564868 wait complete. 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.039+0000 7f8c16615640 1 Processor -- start 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c16615640 1 -- start start 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c16615640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c100b0270 0x7f8c100cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c16615640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c100b1a20 0x7f8c10153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c16615640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c101614a0 0x7f8c10155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c16615640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c100b1e00 con 0x7f8c100b1a20 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c16615640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c100b06e0 con 0x7f8c100b0270 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c16615640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c100b1f80 con 0x7f8c101614a0 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c0f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c100b1a20 0x7f8c10153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c0f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c100b1a20 0x7f8c10153c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56216/0 (socket says 172.21.15.77:56216) 2024-03-06T07:22:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c0f7fe640 1 -- 172.21.15.77:0/3671557329 learned_addr learned my addr 172.21.15.77:0/3671557329 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c0ffff640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c100b0270 0x7f8c100cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.040+0000 7f8c14bb2640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c101614a0 0x7f8c10155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0f7fe640 1 -- 172.21.15.77:0/3671557329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c101614a0 msgr2=0x7f8c10155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0f7fe640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c101614a0 0x7f8c10155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0f7fe640 1 -- 172.21.15.77:0/3671557329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c100b0270 msgr2=0x7f8c100cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0f7fe640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c100b0270 0x7f8c100cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0f7fe640 1 -- 172.21.15.77:0/3671557329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c101534a0 con 0x7f8c100b1a20 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0f7fe640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c100b1a20 0x7f8c10153c60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c460 tx=0x7f8c0400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0cff9640 1 -- 172.21.15.77:0/3671557329 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04018070 con 0x7f8c100b1a20 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0cff9640 1 -- 172.21.15.77:0/3671557329 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04007500 con 0x7f8c100b1a20 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c10150d90 con 0x7f8c100b1a20 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c0cff9640 1 -- 172.21.15.77:0/3671557329 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04004a20 con 0x7f8c100b1a20 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.041+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c10151350 con 0x7f8c100b1a20 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.042+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1004f330 con 0x7f8c100b1a20 2024-03-06T07:22:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.042+0000 7f8c0cff9640 1 -- 172.21.15.77:0/3671557329 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c0401b070 con 0x7f8c100b1a20 2024-03-06T07:22:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.043+0000 7f8c0cff9640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bfc06b4d0 0x7f8bfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.043+0000 7f8c0cff9640 1 -- 172.21.15.77:0/3671557329 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c04089390 con 0x7f8c100b1a20 2024-03-06T07:22:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.043+0000 7f8c0ffff640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bfc06b4d0 0x7f8bfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.043+0000 7f8c0ffff640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bfc06b4d0 0x7f8bfc06d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8c00036000 tx=0x7f8c0003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.047+0000 7f8c0cff9640 1 -- 172.21.15.77:0/3671557329 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c04057bc0 con 0x7f8c100b1a20 2024-03-06T07:22:11.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.252+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8c1009af00 con 0x7f8c100b1a20 2024-03-06T07:22:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.253+0000 7f8c0cff9640 1 -- 172.21.15.77:0/3671557329 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f8c0405bc60 con 0x7f8c100b1a20 2024-03-06T07:22:11.256 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:11.256 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.254+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bfc06b4d0 msgr2=0x7f8bfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.254+0000 7f8c16615640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bfc06b4d0 0x7f8bfc06d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8c00036000 tx=0x7f8c0003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.254+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c100b1a20 msgr2=0x7f8c10153c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.254+0000 7f8c16615640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c100b1a20 0x7f8c10153c60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c460 tx=0x7f8c0400c920 comp rx=0 tx=0).stop 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.255+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 shutdown_connections 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.255+0000 7f8c16615640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bfc06b4d0 0x7f8bfc06d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.255+0000 7f8c16615640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c100b1a20 0x7f8c10153c60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.255+0000 7f8c16615640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c101614a0 0x7f8c10155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.255+0000 7f8c16615640 1 --2- 172.21.15.77:0/3671557329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c100b0270 0x7f8c100cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.255+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 >> 172.21.15.77:0/3671557329 conn(0x7f8c1009a9f0 msgr2=0x7f8c1006ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.255+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 shutdown_connections 2024-03-06T07:22:11.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.255+0000 7f8c16615640 1 -- 172.21.15.77:0/3671557329 wait complete. 2024-03-06T07:22:11.268 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:22:11.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:11.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.366+0000 7f598c66f640 1 Processor -- start 2024-03-06T07:22:11.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.366+0000 7f598c66f640 1 -- start start 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598c66f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f598406b220 0x7f598415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598c66f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59840ce920 0x7f5984071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598c66f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5984071ec0 0x7f5984162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598c66f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5984161b70 con 0x7f5984071ec0 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598c66f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598415ea30 con 0x7f598406b220 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598c66f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598415ed10 con 0x7f59840ce920 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598ac0c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5984071ec0 0x7f5984162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598ac0c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5984071ec0 0x7f5984162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56220/0 (socket says 172.21.15.77:56220) 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598ac0c640 1 -- 172.21.15.77:0/1031605639 learned_addr learned my addr 172.21.15.77:0/1031605639 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:11.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598a40b640 1 --2- 172.21.15.77:0/1031605639 >> v2:172.21.15.158:3300/0 conn(0x7f598406b220 0x7f598415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f5989c0a640 1 --2- 172.21.15.77:0/1031605639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59840ce920 0x7f5984071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598ac0c640 1 -- 172.21.15.77:0/1031605639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59840ce920 msgr2=0x7f5984071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598ac0c640 1 --2- 172.21.15.77:0/1031605639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59840ce920 0x7f5984071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598ac0c640 1 -- 172.21.15.77:0/1031605639 >> v2:172.21.15.158:3300/0 conn(0x7f598406b220 msgr2=0x7f598415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598ac0c640 1 --2- 172.21.15.77:0/1031605639 >> v2:172.21.15.158:3300/0 conn(0x7f598406b220 0x7f598415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.367+0000 7f598ac0c640 1 -- 172.21.15.77:0/1031605639 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598415eff0 con 0x7f5984071ec0 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.368+0000 7f598ac0c640 1 --2- 172.21.15.77:0/1031605639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5984071ec0 0x7f5984162000 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f59840d0980 tx=0x7f597c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d8f8cee60b325d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.368+0000 7f5989409640 1 -- 172.21.15.77:0/1031605639 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c02ca70 con 0x7f5984071ec0 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.368+0000 7f5989409640 1 -- 172.21.15.77:0/1031605639 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c02cc10 con 0x7f5984071ec0 2024-03-06T07:22:11.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.368+0000 7f5989409640 1 -- 172.21.15.77:0/1031605639 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c0343f0 con 0x7f5984071ec0 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 -- 172.21.15.77:0/1031605639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5984071ec0 msgr2=0x7f5984162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 --2- 172.21.15.77:0/1031605639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5984071ec0 0x7f5984162000 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f59840d0980 tx=0x7f597c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 -- 172.21.15.77:0/1031605639 shutdown_connections 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 --2- 172.21.15.77:0/1031605639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5984071ec0 0x7f5984162000 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 --2- 172.21.15.77:0/1031605639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59840ce920 0x7f5984071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 --2- 172.21.15.77:0/1031605639 >> v2:172.21.15.158:3300/0 conn(0x7f598406b220 0x7f598415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 -- 172.21.15.77:0/1031605639 >> 172.21.15.77:0/1031605639 conn(0x7f598406c0c0 msgr2=0x7f598406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 -- 172.21.15.77:0/1031605639 shutdown_connections 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.369+0000 7f598c66f640 1 -- 172.21.15.77:0/1031605639 wait complete. 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.370+0000 7f598c66f640 1 Processor -- start 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.370+0000 7f598c66f640 1 -- start start 2024-03-06T07:22:11.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.370+0000 7f598c66f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598406b220 0x7f5984158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.370+0000 7f598c66f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5984071ec0 0x7f59841607b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.370+0000 7f598c66f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59840ce920 0x7f59840b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.370+0000 7f598c66f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840ce380 con 0x7f598406b220 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.370+0000 7f598c66f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840ce200 con 0x7f59840ce920 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.370+0000 7f598c66f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598406b600 con 0x7f5984071ec0 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598406b220 0x7f5984158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598406b220 0x7f5984158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56230/0 (socket says 172.21.15.77:56230) 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 -- 172.21.15.77:0/3556949375 learned_addr learned my addr 172.21.15.77:0/3556949375 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f5989c0a640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5984071ec0 0x7f59841607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598ac0c640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59840ce920 0x7f59840b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 -- 172.21.15.77:0/3556949375 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5984071ec0 msgr2=0x7f59841607b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5984071ec0 0x7f59841607b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 -- 172.21.15.77:0/3556949375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59840ce920 msgr2=0x7f59840b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59840ce920 0x7f59840b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 -- 172.21.15.77:0/3556949375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840ae9b0 con 0x7f598406b220 2024-03-06T07:22:11.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598a40b640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598406b220 0x7f5984158dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9e0 tx=0x7f598000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f597affd640 1 -- 172.21.15.77:0/3556949375 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5980017070 con 0x7f598406b220 2024-03-06T07:22:11.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f597affd640 1 -- 172.21.15.77:0/3556949375 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598000c9d0 con 0x7f598406b220 2024-03-06T07:22:11.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f597affd640 1 -- 172.21.15.77:0/3556949375 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598000cdc0 con 0x7f598406b220 2024-03-06T07:22:11.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59840aec90 con 0x7f598406b220 2024-03-06T07:22:11.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.371+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59840af250 con 0x7f598406b220 2024-03-06T07:22:11.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.372+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598404f330 con 0x7f598406b220 2024-03-06T07:22:11.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.372+0000 7f597affd640 1 -- 172.21.15.77:0/3556949375 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f598001a070 con 0x7f598406b220 2024-03-06T07:22:11.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.373+0000 7f597affd640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f596806b4d0 0x7f596806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.373+0000 7f597affd640 1 -- 172.21.15.77:0/3556949375 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f598008a620 con 0x7f598406b220 2024-03-06T07:22:11.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.373+0000 7f5989c0a640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f596806b4d0 0x7f596806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.373+0000 7f5989c0a640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f596806b4d0 0x7f596806d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f59840b1830 tx=0x7f5974009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.377+0000 7f597affd640 1 -- 172.21.15.77:0/3556949375 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5980058dc0 con 0x7f598406b220 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.551+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59840cdf00 con 0x7f598406b220 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.552+0000 7f597affd640 1 -- 172.21.15.77:0/3556949375 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f598005ce60 con 0x7f598406b220 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:11.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f596806b4d0 msgr2=0x7f596806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f596806b4d0 0x7f596806d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f59840b1830 tx=0x7f5974009190 comp rx=0 tx=0).stop 2024-03-06T07:22:11.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598406b220 msgr2=0x7f5984158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598406b220 0x7f5984158dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9e0 tx=0x7f598000eea0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 shutdown_connections 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59840ce920 0x7f59840b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5984071ec0 0x7f59841607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f596806b4d0 0x7f596806d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 --2- 172.21.15.77:0/3556949375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598406b220 0x7f5984158dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 >> 172.21.15.77:0/3556949375 conn(0x7f598406c0c0 msgr2=0x7f598406e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 shutdown_connections 2024-03-06T07:22:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.554+0000 7f598c66f640 1 -- 172.21.15.77:0/3556949375 wait complete. 2024-03-06T07:22:11.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:11.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:11.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.668+0000 7f704bf3a640 1 Processor -- start 2024-03-06T07:22:11.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.669+0000 7f704bf3a640 1 -- start start 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.669+0000 7f704bf3a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70441586e0 0x7f70440aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f704bf3a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f70440d01a0 0x7f70440b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f704bf3a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7044158ac0 con 0x7f70440aef80 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f704bf3a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704415b2a0 con 0x7f70441586e0 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f704bf3a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704415b540 con 0x7f70440d01a0 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f7049cd6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70441586e0 0x7f70440aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f704a4d7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f70440d01a0 0x7f70440b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f7049cd6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70441586e0 0x7f70440aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56246/0 (socket says 172.21.15.77:56246) 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f7049cd6640 1 -- 172.21.15.77:0/3493008231 learned_addr learned my addr 172.21.15.77:0/3493008231 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f7049cd6640 1 -- 172.21.15.77:0/3493008231 >> v2:172.21.15.158:3300/0 conn(0x7f70440d01a0 msgr2=0x7f70440b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f7049cd6640 1 --2- 172.21.15.77:0/3493008231 >> v2:172.21.15.158:3300/0 conn(0x7f70440d01a0 0x7f70440b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f7049cd6640 1 -- 172.21.15.77:0/3493008231 >> v1:172.21.15.158:6789/0 conn(0x7f70440aef80 legacy=0x7f70440cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.670+0000 7f7049cd6640 1 -- 172.21.15.77:0/3493008231 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704415b820 con 0x7f70441586e0 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f7049cd6640 1 --2- 172.21.15.77:0/3493008231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70441586e0 0x7f70440aea40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f70440cc350 tx=0x7f703402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603ddfc67b66ac32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f7048cd4640 1 -- 172.21.15.77:0/3493008231 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034035020 con 0x7f70441586e0 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f7048cd4640 1 -- 172.21.15.77:0/3493008231 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7034031870 con 0x7f70441586e0 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f7048cd4640 1 -- 172.21.15.77:0/3493008231 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034031c40 con 0x7f70441586e0 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f704bf3a640 1 -- 172.21.15.77:0/3493008231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70441586e0 msgr2=0x7f70440aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f704bf3a640 1 --2- 172.21.15.77:0/3493008231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70441586e0 0x7f70440aea40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f70440cc350 tx=0x7f703402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f704bf3a640 1 -- 172.21.15.77:0/3493008231 shutdown_connections 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f704bf3a640 1 --2- 172.21.15.77:0/3493008231 >> v2:172.21.15.158:3300/0 conn(0x7f70440d01a0 0x7f70440b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f704bf3a640 1 --2- 172.21.15.77:0/3493008231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70441586e0 0x7f70440aea40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.671+0000 7f704bf3a640 1 -- 172.21.15.77:0/3493008231 >> 172.21.15.77:0/3493008231 conn(0x7f704406b190 msgr2=0x7f704406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.672+0000 7f704bf3a640 1 -- 172.21.15.77:0/3493008231 shutdown_connections 2024-03-06T07:22:11.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.672+0000 7f704bf3a640 1 -- 172.21.15.77:0/3493008231 wait complete. 2024-03-06T07:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f704bf3a640 1 Processor -- start 2024-03-06T07:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f704bf3a640 1 -- start start 2024-03-06T07:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f704bf3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70440aef80 0x7f7044154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f704bf3a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70440d01a0 0x7f7044155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f704bf3a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70441586e0 0x7f7044148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f704bf3a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7044071350 con 0x7f70440aef80 2024-03-06T07:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f704bf3a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704415bde0 con 0x7f70441586e0 2024-03-06T07:22:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f704bf3a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70440714d0 con 0x7f70440d01a0 2024-03-06T07:22:11.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f7049cd6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70440aef80 0x7f7044154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f7049cd6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70440aef80 0x7f7044154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56262/0 (socket says 172.21.15.77:56262) 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f7049cd6640 1 -- 172.21.15.77:0/1701315500 learned_addr learned my addr 172.21.15.77:0/1701315500 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.673+0000 7f70494d5640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70440d01a0 0x7f7044155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f704a4d7640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70441586e0 0x7f7044148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f7049cd6640 1 -- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70440d01a0 msgr2=0x7f7044155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f7049cd6640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70440d01a0 0x7f7044155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f7049cd6640 1 -- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70441586e0 msgr2=0x7f7044148b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f7049cd6640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70441586e0 0x7f7044148b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f7049cd6640 1 -- 172.21.15.77:0/1701315500 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7044155800 con 0x7f70440aef80 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f704a4d7640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70441586e0 0x7f7044148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f7049cd6640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70440aef80 0x7f7044154b40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7034000c00 tx=0x7f70340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f703a7fc640 1 -- 172.21.15.77:0/1701315500 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034035040 con 0x7f70440aef80 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f703a7fc640 1 -- 172.21.15.77:0/1701315500 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7034030d00 con 0x7f70440aef80 2024-03-06T07:22:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7044155ae0 con 0x7f70440aef80 2024-03-06T07:22:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f703a7fc640 1 -- 172.21.15.77:0/1701315500 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70340456a0 con 0x7f70440aef80 2024-03-06T07:22:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.674+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7044146da0 con 0x7f70440aef80 2024-03-06T07:22:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.675+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70440732b0 con 0x7f70440aef80 2024-03-06T07:22:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.675+0000 7f703a7fc640 1 -- 172.21.15.77:0/1701315500 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7034038020 con 0x7f70440aef80 2024-03-06T07:22:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.676+0000 7f703a7fc640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f702806b4d0 0x7f702806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.676+0000 7f70494d5640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f702806b4d0 0x7f702806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.676+0000 7f703a7fc640 1 -- 172.21.15.77:0/1701315500 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70340ac3d0 con 0x7f70440aef80 2024-03-06T07:22:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.676+0000 7f70494d5640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f702806b4d0 0x7f702806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f70440b40d0 tx=0x7f7040009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.680+0000 7f703a7fc640 1 -- 172.21.15.77:0/1701315500 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f703407b2e0 con 0x7f70440aef80 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.851+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7044068ff0 con 0x7f70440aef80 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.852+0000 7f703a7fc640 1 -- 172.21.15.77:0/1701315500 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f703407abe0 con 0x7f70440aef80 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f702806b4d0 msgr2=0x7f702806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f702806b4d0 0x7f702806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f70440b40d0 tx=0x7f7040009040 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70440aef80 msgr2=0x7f7044154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70440aef80 0x7f7044154b40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7034000c00 tx=0x7f70340046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 shutdown_connections 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70441586e0 0x7f7044148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70440d01a0 0x7f7044155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f702806b4d0 0x7f702806d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 --2- 172.21.15.77:0/1701315500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70440aef80 0x7f7044154b40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 >> 172.21.15.77:0/1701315500 conn(0x7f704406b190 msgr2=0x7f70440cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 shutdown_connections 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.854+0000 7f704bf3a640 1 -- 172.21.15.77:0/1701315500 wait complete. 2024-03-06T07:22:11.867 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:11.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.967+0000 7f0032c88640 1 Processor -- start 2024-03-06T07:22:11.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.967+0000 7f0032c88640 1 -- start start 2024-03-06T07:22:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.968+0000 7f0032c88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c15bbb0 0x7f002c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.968+0000 7f0032c88640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c0cea90 con 0x7f002c06b4f0 2024-03-06T07:22:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.968+0000 7f0032c88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c161bc0 con 0x7f002c15ee10 2024-03-06T07:22:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.968+0000 7f0032c88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c161970 con 0x7f002c15bbb0 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.968+0000 7f002bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f002c06b4f0 0x7f002c15e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45578/0 (socket says 172.21.15.77:45578) 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.968+0000 7f002bfff640 1 -- 172.21.15.77:0/4259754466 learned_addr learned my addr 172.21.15.77:0/4259754466 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.968+0000 7f0030a24640 1 --2- 172.21.15.77:0/4259754466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c15bbb0 0x7f002c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.968+0000 7f002b7fe640 1 -- 172.21.15.77:0/4259754466 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555721923 0 0) 0x7f0020001110 con 0x7f002c06b4f0 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.969+0000 7f002b7fe640 1 -- 172.21.15.77:0/4259754466 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0014003520 con 0x7f002c06b4f0 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.969+0000 7f002b7fe640 1 -- 172.21.15.77:0/4259754466 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628804695 0 0) 0x7f002c161bc0 con 0x7f002c15ee10 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.969+0000 7f002b7fe640 1 -- 172.21.15.77:0/4259754466 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0020001110 con 0x7f002c15ee10 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.969+0000 7f0030a24640 1 -- 172.21.15.77:0/4259754466 >> v1:172.21.15.158:6789/0 conn(0x7f002c15ee10 legacy=0x7f002c071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.969+0000 7f0030a24640 1 -- 172.21.15.77:0/4259754466 >> v1:172.21.15.77:6789/0 conn(0x7f002c06b4f0 legacy=0x7f002c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.969+0000 7f0030a24640 1 -- 172.21.15.77:0/4259754466 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c161ea0 con 0x7f002c15bbb0 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.969+0000 7f0030a24640 1 --2- 172.21.15.77:0/4259754466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c15bbb0 0x7f002c06afb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f001c0097d0 tx=0x7f001c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ce4a839dc30bb3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f002b7fe640 1 -- 172.21.15.77:0/4259754466 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c035020 con 0x7f002c15bbb0 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f002b7fe640 1 -- 172.21.15.77:0/4259754466 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c031890 con 0x7f002c15bbb0 2024-03-06T07:22:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f002b7fe640 1 -- 172.21.15.77:0/4259754466 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c031c60 con 0x7f002c15bbb0 2024-03-06T07:22:11.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f0032c88640 1 -- 172.21.15.77:0/4259754466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c15bbb0 msgr2=0x7f002c06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f0032c88640 1 --2- 172.21.15.77:0/4259754466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c15bbb0 0x7f002c06afb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f001c0097d0 tx=0x7f001c02b930 comp rx=0 tx=0).stop 2024-03-06T07:22:11.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f0032c88640 1 -- 172.21.15.77:0/4259754466 shutdown_connections 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f0032c88640 1 --2- 172.21.15.77:0/4259754466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c15bbb0 0x7f002c06afb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f0032c88640 1 -- 172.21.15.77:0/4259754466 >> 172.21.15.77:0/4259754466 conn(0x7f002c0b2660 msgr2=0x7f002c0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f0032c88640 1 -- 172.21.15.77:0/4259754466 shutdown_connections 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.970+0000 7f0032c88640 1 -- 172.21.15.77:0/4259754466 wait complete. 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.971+0000 7f0032c88640 1 Processor -- start 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.971+0000 7f0032c88640 1 -- start start 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0032c88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c06b4f0 0x7f002c150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0032c88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f002c15ee10 0x7f002c15f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0032c88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f002c0cc4d0 0x7f002c1600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0032c88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0723b0 con 0x7f002c15ee10 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0032c88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c1624f0 con 0x7f002c0cc4d0 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0032c88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c072530 con 0x7f002c06b4f0 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f002bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f002c15ee10 0x7f002c15f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f002bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f002c15ee10 0x7f002c15f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56270/0 (socket says 172.21.15.77:56270) 2024-03-06T07:22:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f002bfff640 1 -- 172.21.15.77:0/1522177508 learned_addr learned my addr 172.21.15.77:0/1522177508 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0030a24640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c06b4f0 0x7f002c150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0031225640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f002c0cc4d0 0x7f002c1600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f002bfff640 1 -- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c06b4f0 msgr2=0x7f002c150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f002bfff640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c06b4f0 0x7f002c150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f002bfff640 1 -- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f002c0cc4d0 msgr2=0x7f002c1600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f002bfff640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f002c0cc4d0 0x7f002c1600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f002bfff640 1 -- 172.21.15.77:0/1522177508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c0cb430 con 0x7f002c15ee10 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.972+0000 7f0031225640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f002c0cc4d0 0x7f002c1600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.973+0000 7f002bfff640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f002c15ee10 0x7f002c15f430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f002000c680 tx=0x7f002000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.973+0000 7f00297fa640 1 -- 172.21.15.77:0/1522177508 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020017070 con 0x7f002c15ee10 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.973+0000 7f00297fa640 1 -- 172.21.15.77:0/1522177508 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0020005ab0 con 0x7f002c15ee10 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.973+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002c0cb710 con 0x7f002c15ee10 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.973+0000 7f00297fa640 1 -- 172.21.15.77:0/1522177508 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020007070 con 0x7f002c15ee10 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.973+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c0cbcd0 con 0x7f002c15ee10 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.974+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002c0733b0 con 0x7f002c15ee10 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.974+0000 7f00297fa640 1 -- 172.21.15.77:0/1522177508 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f002001a070 con 0x7f002c15ee10 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.974+0000 7f00297fa640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f001806b500 0x7f001806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.975+0000 7f00297fa640 1 -- 172.21.15.77:0/1522177508 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0020089660 con 0x7f002c15ee10 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.975+0000 7f0030a24640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f001806b500 0x7f001806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.975+0000 7f0030a24640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f001806b500 0x7f001806d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f001c036000 tx=0x7f001c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:11.979+0000 7f00297fa640 1 -- 172.21.15.77:0/1522177508 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f002c0cea90 con 0x7f002c15ee10 2024-03-06T07:22:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.151+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f002c068ff0 con 0x7f002c15ee10 2024-03-06T07:22:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.151+0000 7f00297fa640 1 -- 172.21.15.77:0/1522177508 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0020057f20 con 0x7f002c15ee10 2024-03-06T07:22:12.155 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:12.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f001806b500 msgr2=0x7f001806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f001806b500 0x7f001806d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f001c036000 tx=0x7f001c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f002c15ee10 msgr2=0x7f002c15f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f002c15ee10 0x7f002c15f430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f002000c680 tx=0x7f002000cb40 comp rx=0 tx=0).stop 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 shutdown_connections 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f001806b500 0x7f001806d9c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f002c15ee10 0x7f002c15f430 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f002c0cc4d0 0x7f002c1600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 --2- 172.21.15.77:0/1522177508 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f002c06b4f0 0x7f002c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 >> 172.21.15.77:0/1522177508 conn(0x7f002c0b2660 msgr2=0x7f002c06dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 shutdown_connections 2024-03-06T07:22:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.153+0000 7f0032c88640 1 -- 172.21.15.77:0/1522177508 wait complete. 2024-03-06T07:22:12.167 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:12.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.267+0000 7fcc39d43640 1 Processor -- start 2024-03-06T07:22:12.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.267+0000 7fcc39d43640 1 -- start start 2024-03-06T07:22:12.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc39d43640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc34145590 0x7fcc3406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc39d43640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcc3406b4f0 0x7fcc3406efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc39d43640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3406b970 con 0x7fcc3406f510 2024-03-06T07:22:12.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc39d43640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc34145970 con 0x7fcc3406b4f0 2024-03-06T07:22:12.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc39d43640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc34160110 con 0x7fcc34145590 2024-03-06T07:22:12.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc32ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcc3406b4f0 0x7fcc3406efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc337fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc34145590 0x7fcc3406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc33fff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcc3406f510 0x7fcc3406daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:46208/0 (socket says 172.21.15.77:46208) 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc33fff640 1 -- 172.21.15.77:0/1309828152 learned_addr learned my addr 172.21.15.77:0/1309828152 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc337fe640 1 -- 172.21.15.77:0/1309828152 >> v2:172.21.15.158:3300/0 conn(0x7fcc3406b4f0 msgr2=0x7fcc3406efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc337fe640 1 --2- 172.21.15.77:0/1309828152 >> v2:172.21.15.158:3300/0 conn(0x7fcc3406b4f0 0x7fcc3406efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc337fe640 1 -- 172.21.15.77:0/1309828152 >> v1:172.21.15.158:6789/0 conn(0x7fcc3406f510 legacy=0x7fcc3406daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc337fe640 1 -- 172.21.15.77:0/1309828152 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3415f7e0 con 0x7fcc34145590 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.268+0000 7fcc32ffd640 1 --2- 172.21.15.77:0/1309828152 >> v2:172.21.15.158:3300/0 conn(0x7fcc3406b4f0 0x7fcc3406efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.269+0000 7fcc337fe640 1 --2- 172.21.15.77:0/1309828152 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc34145590 0x7fcc3406afb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc240097a0 tx=0x7fcc2402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe6c09bcd43a5ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.269+0000 7fcc327fc640 1 -- 172.21.15.77:0/1309828152 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24035020 con 0x7fcc34145590 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.269+0000 7fcc327fc640 1 -- 172.21.15.77:0/1309828152 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc24031830 con 0x7fcc34145590 2024-03-06T07:22:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.269+0000 7fcc327fc640 1 -- 172.21.15.77:0/1309828152 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24031c00 con 0x7fcc34145590 2024-03-06T07:22:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.270+0000 7fcc39d43640 1 -- 172.21.15.77:0/1309828152 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc34145590 msgr2=0x7fcc3406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.270+0000 7fcc39d43640 1 --2- 172.21.15.77:0/1309828152 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc34145590 0x7fcc3406afb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc240097a0 tx=0x7fcc2402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.270+0000 7fcc39d43640 1 -- 172.21.15.77:0/1309828152 shutdown_connections 2024-03-06T07:22:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.270+0000 7fcc39d43640 1 --2- 172.21.15.77:0/1309828152 >> v2:172.21.15.158:3300/0 conn(0x7fcc3406b4f0 0x7fcc3406efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.270+0000 7fcc39d43640 1 --2- 172.21.15.77:0/1309828152 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc34145590 0x7fcc3406afb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.270+0000 7fcc39d43640 1 -- 172.21.15.77:0/1309828152 >> 172.21.15.77:0/1309828152 conn(0x7fcc340b2660 msgr2=0x7fcc340b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.270+0000 7fcc39d43640 1 -- 172.21.15.77:0/1309828152 shutdown_connections 2024-03-06T07:22:12.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.270+0000 7fcc39d43640 1 -- 172.21.15.77:0/1309828152 wait complete. 2024-03-06T07:22:12.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.271+0000 7fcc39d43640 1 Processor -- start 2024-03-06T07:22:12.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.271+0000 7fcc39d43640 1 -- start start 2024-03-06T07:22:12.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc39d43640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3406b4f0 0x7fcc340b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc39d43640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc3406f510 0x7fcc3415b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc39d43640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc34145590 0x7fcc341f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc39d43640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc340cd5d0 con 0x7fcc3406f510 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc39d43640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3415fda0 con 0x7fcc34145590 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc39d43640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc341605a0 con 0x7fcc3406b4f0 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc32ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc3406f510 0x7fcc3415b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc32ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc3406f510 0x7fcc3415b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56280/0 (socket says 172.21.15.77:56280) 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc32ffd640 1 -- 172.21.15.77:0/2697307619 learned_addr learned my addr 172.21.15.77:0/2697307619 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc337fe640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3406b4f0 0x7fcc340b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc33fff640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc34145590 0x7fcc341f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc32ffd640 1 -- 172.21.15.77:0/2697307619 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3406b4f0 msgr2=0x7fcc340b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc32ffd640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3406b4f0 0x7fcc340b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc32ffd640 1 -- 172.21.15.77:0/2697307619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc34145590 msgr2=0x7fcc341f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc32ffd640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc34145590 0x7fcc341f4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.272+0000 7fcc32ffd640 1 -- 172.21.15.77:0/2697307619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc340b43d0 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.273+0000 7fcc32ffd640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc3406f510 0x7fcc3415b560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800e960 tx=0x7fcc2800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.273+0000 7fcc1bfff640 1 -- 172.21.15.77:0/2697307619 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28017070 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.273+0000 7fcc1bfff640 1 -- 172.21.15.77:0/2697307619 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2800ca00 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.273+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc340b46b0 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.273+0000 7fcc1bfff640 1 -- 172.21.15.77:0/2697307619 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800cdd0 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.273+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc340b4c70 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.274+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc340b50b0 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.274+0000 7fcc1bfff640 1 -- 172.21.15.77:0/2697307619 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc2801a070 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.275+0000 7fcc1bfff640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2006b4d0 0x7fcc2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.275+0000 7fcc1bfff640 1 -- 172.21.15.77:0/2697307619 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc2808a330 con 0x7fcc3406f510 2024-03-06T07:22:12.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.275+0000 7fcc337fe640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2006b4d0 0x7fcc2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.275+0000 7fcc337fe640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2006b4d0 0x7fcc2006d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc24036000 tx=0x7fcc24008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.279+0000 7fcc1bfff640 1 -- 172.21.15.77:0/2697307619 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc28058bf0 con 0x7fcc3406f510 2024-03-06T07:22:12.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.449+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc340733b0 con 0x7fcc3406f510 2024-03-06T07:22:12.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.450+0000 7fcc1bfff640 1 -- 172.21.15.77:0/2697307619 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc2805cc90 con 0x7fcc3406f510 2024-03-06T07:22:12.453 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2006b4d0 msgr2=0x7fcc2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2006b4d0 0x7fcc2006d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc24036000 tx=0x7fcc24008030 comp rx=0 tx=0).stop 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc3406f510 msgr2=0x7fcc3415b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc3406f510 0x7fcc3415b560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800e960 tx=0x7fcc2800ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 shutdown_connections 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2006b4d0 0x7fcc2006d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc3406f510 0x7fcc3415b560 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc34145590 0x7fcc341f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 --2- 172.21.15.77:0/2697307619 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3406b4f0 0x7fcc340b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 >> 172.21.15.77:0/2697307619 conn(0x7fcc340b2660 msgr2=0x7fcc340b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 shutdown_connections 2024-03-06T07:22:12.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.452+0000 7fcc39d43640 1 -- 172.21.15.77:0/2697307619 wait complete. 2024-03-06T07:22:12.465 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:12.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.562+0000 7f3b8062a640 1 Processor -- start 2024-03-06T07:22:12.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.562+0000 7f3b8062a640 1 -- start start 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b8062a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b780ce920 0x7f3b7815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b8062a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3b7806b600 0x7f3b78162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b8062a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b7815ea30 con 0x7f3b7806b220 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b8062a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b78161b70 con 0x7f3b780ce920 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b8062a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b7815ed10 con 0x7f3b7806b600 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7dbc5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b780ce920 0x7f3b7815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7dbc5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b780ce920 0x7f3b7815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56286/0 (socket says 172.21.15.77:56286) 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7dbc5640 1 -- 172.21.15.77:0/3318444416 learned_addr learned my addr 172.21.15.77:0/3318444416 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7ebc7640 1 --2- 172.21.15.77:0/3318444416 >> v2:172.21.15.158:3300/0 conn(0x7f3b7806b600 0x7f3b78162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7dbc5640 1 -- 172.21.15.77:0/3318444416 >> v2:172.21.15.158:3300/0 conn(0x7f3b7806b600 msgr2=0x7f3b78162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7dbc5640 1 --2- 172.21.15.77:0/3318444416 >> v2:172.21.15.158:3300/0 conn(0x7f3b7806b600 0x7f3b78162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7dbc5640 1 -- 172.21.15.77:0/3318444416 >> v1:172.21.15.158:6789/0 conn(0x7f3b7806b220 legacy=0x7f3b78071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7dbc5640 1 -- 172.21.15.77:0/3318444416 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7815eff0 con 0x7f3b780ce920 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.563+0000 7f3b7ebc7640 1 --2- 172.21.15.77:0/3318444416 >> v2:172.21.15.158:3300/0 conn(0x7f3b7806b600 0x7f3b78162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b7dbc5640 1 --2- 172.21.15.77:0/3318444416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b780ce920 0x7f3b7815e450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3b7806c040 tx=0x7f3b6802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485fd245f44deb12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b7d3c4640 1 -- 172.21.15.77:0/3318444416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b68034070 con 0x7f3b780ce920 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b7d3c4640 1 -- 172.21.15.77:0/3318444416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b68030860 con 0x7f3b780ce920 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b7d3c4640 1 -- 172.21.15.77:0/3318444416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b68030c30 con 0x7f3b780ce920 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b8062a640 1 -- 172.21.15.77:0/3318444416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b780ce920 msgr2=0x7f3b7815e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3318444416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b780ce920 0x7f3b7815e450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3b7806c040 tx=0x7f3b6802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b8062a640 1 -- 172.21.15.77:0/3318444416 shutdown_connections 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3318444416 >> v2:172.21.15.158:3300/0 conn(0x7f3b7806b600 0x7f3b78162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.564+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3318444416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b780ce920 0x7f3b7815e450 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.565+0000 7f3b8062a640 1 -- 172.21.15.77:0/3318444416 >> 172.21.15.77:0/3318444416 conn(0x7f3b7806c0c0 msgr2=0x7f3b7806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.565+0000 7f3b8062a640 1 -- 172.21.15.77:0/3318444416 shutdown_connections 2024-03-06T07:22:12.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.565+0000 7f3b8062a640 1 -- 172.21.15.77:0/3318444416 wait complete. 2024-03-06T07:22:12.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b8062a640 1 Processor -- start 2024-03-06T07:22:12.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b8062a640 1 -- start start 2024-03-06T07:22:12.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b8062a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b7806b220 0x7f3b781f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b8062a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7806b600 0x7f3b780ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b8062a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b780ce920 0x7f3b780adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b8062a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b78072090 con 0x7f3b7806b600 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b8062a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b780ce200 con 0x7f3b7806b220 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b8062a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b780b0370 con 0x7f3b780ce920 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b7dbc5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7806b600 0x7f3b780ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.566+0000 7f3b7dbc5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7806b600 0x7f3b780ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56290/0 (socket says 172.21.15.77:56290) 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7dbc5640 1 -- 172.21.15.77:0/3922014532 learned_addr learned my addr 172.21.15.77:0/3922014532 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7e3c6640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b7806b220 0x7f3b781f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7ebc7640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b780ce920 0x7f3b780adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7dbc5640 1 -- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b780ce920 msgr2=0x7f3b780adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7dbc5640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b780ce920 0x7f3b780adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7dbc5640 1 -- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b7806b220 msgr2=0x7f3b781f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7dbc5640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b7806b220 0x7f3b781f3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7dbc5640 1 -- 172.21.15.77:0/3922014532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b780afa80 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7e3c6640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b7806b220 0x7f3b781f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b7dbc5640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7806b600 0x7f3b780ad860 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3b6802b9e0 tx=0x7f3b68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b6effd640 1 -- 172.21.15.77:0/3922014532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b68048070 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b6effd640 1 -- 172.21.15.77:0/3922014532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b68030660 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b78160ca0 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b6effd640 1 -- 172.21.15.77:0/3922014532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b68034040 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.567+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b780b0000 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.568+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7804f330 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.569+0000 7f3b6effd640 1 -- 172.21.15.77:0/3922014532 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b6804b070 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.569+0000 7f3b6effd640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b5c06b4d0 0x7f3b5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.569+0000 7f3b6effd640 1 -- 172.21.15.77:0/3922014532 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b680ac130 con 0x7f3b7806b600 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.569+0000 7f3b7e3c6640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b5c06b4d0 0x7f3b5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.570+0000 7f3b7e3c6640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b5c06b4d0 0x7f3b5c06d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b74001ad0 tx=0x7f3b74009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.574+0000 7f3b6effd640 1 -- 172.21.15.77:0/3922014532 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b6807a960 con 0x7f3b7806b600 2024-03-06T07:22:12.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.781+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b780cdf00 con 0x7f3b7806b600 2024-03-06T07:22:12.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.782+0000 7f3b6effd640 1 -- 172.21.15.77:0/3922014532 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f3b6807ea00 con 0x7f3b7806b600 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.285900+0000","modified":"2024-03-06T07:22:00.291221+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-06T07:22:12.785 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b5c06b4d0 msgr2=0x7f3b5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b5c06b4d0 0x7f3b5c06d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b74001ad0 tx=0x7f3b74009040 comp rx=0 tx=0).stop 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7806b600 msgr2=0x7f3b780ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7806b600 0x7f3b780ad860 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3b6802b9e0 tx=0x7f3b68004730 comp rx=0 tx=0).stop 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 shutdown_connections 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b5c06b4d0 0x7f3b5c06d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7806b600 0x7f3b780ad860 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b780ce920 0x7f3b780adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 --2- 172.21.15.77:0/3922014532 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b7806b220 0x7f3b781f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.783+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 >> 172.21.15.77:0/3922014532 conn(0x7f3b7806c0c0 msgr2=0x7f3b7806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.784+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 shutdown_connections 2024-03-06T07:22:12.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.784+0000 7f3b8062a640 1 -- 172.21.15.77:0/3922014532 wait complete. 2024-03-06T07:22:12.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:12.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.895+0000 7fbba6c78640 1 Processor -- start 2024-03-06T07:22:12.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.895+0000 7fbba6c78640 1 -- start start 2024-03-06T07:22:12.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbba6c78640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbba00b3060 0x7fbba015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbba6c78640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba015f580 con 0x7fbba013dc10 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbba6c78640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba015b450 con 0x7fbba015bf30 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbba6c78640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba00cf3f0 con 0x7fbba00b3060 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbba4a14640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbba013dc10 0x7fbba00b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45588/0 (socket says 172.21.15.77:45588) 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbba4a14640 1 -- 172.21.15.77:0/2242558717 learned_addr learned my addr 172.21.15.77:0/2242558717 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbb9ffff640 1 --2- 172.21.15.77:0/2242558717 >> v2:172.21.15.158:3300/0 conn(0x7fbba00b3060 0x7fbba015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbb9f7fe640 1 -- 172.21.15.77:0/2242558717 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973509233 0 0) 0x7fbba015f580 con 0x7fbba013dc10 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbb9f7fe640 1 -- 172.21.15.77:0/2242558717 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb90003520 con 0x7fbba013dc10 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbb9ffff640 1 -- 172.21.15.77:0/2242558717 >> v1:172.21.15.158:6789/0 conn(0x7fbba015bf30 legacy=0x7fbba015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbb9ffff640 1 -- 172.21.15.77:0/2242558717 >> v1:172.21.15.77:6789/0 conn(0x7fbba013dc10 legacy=0x7fbba00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.896+0000 7fbb9ffff640 1 -- 172.21.15.77:0/2242558717 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba015b730 con 0x7fbba00b3060 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbb9ffff640 1 --2- 172.21.15.77:0/2242558717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba00b3060 0x7fbba015e850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0097a0 tx=0x7fbb8c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dcbd8b56cd1e52e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbb9f7fe640 1 -- 172.21.15.77:0/2242558717 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c035020 con 0x7fbba00b3060 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbb9f7fe640 1 -- 172.21.15.77:0/2242558717 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c031830 con 0x7fbba00b3060 2024-03-06T07:22:12.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbb9f7fe640 1 -- 172.21.15.77:0/2242558717 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c031c00 con 0x7fbba00b3060 2024-03-06T07:22:12.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbba6c78640 1 -- 172.21.15.77:0/2242558717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba00b3060 msgr2=0x7fbba015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbba6c78640 1 --2- 172.21.15.77:0/2242558717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba00b3060 0x7fbba015e850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0097a0 tx=0x7fbb8c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbba6c78640 1 -- 172.21.15.77:0/2242558717 shutdown_connections 2024-03-06T07:22:12.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbba6c78640 1 --2- 172.21.15.77:0/2242558717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba00b3060 0x7fbba015e850 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbba6c78640 1 -- 172.21.15.77:0/2242558717 >> 172.21.15.77:0/2242558717 conn(0x7fbba00cf8a0 msgr2=0x7fbba00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbba6c78640 1 -- 172.21.15.77:0/2242558717 shutdown_connections 2024-03-06T07:22:12.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.897+0000 7fbba6c78640 1 -- 172.21.15.77:0/2242558717 wait complete. 2024-03-06T07:22:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.899+0000 7fbba6c78640 1 Processor -- start 2024-03-06T07:22:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.899+0000 7fbba6c78640 1 -- start start 2024-03-06T07:22:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.899+0000 7fbba6c78640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbba00b3060 0x7fbba013a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.899+0000 7fbba6c78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbba013dc10 0x7fbba01675d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.899+0000 7fbba6c78640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba015bf30 0x7fbba0159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.899+0000 7fbba6c78640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba00718e0 con 0x7fbba013dc10 2024-03-06T07:22:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.899+0000 7fbba6c78640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba015b210 con 0x7fbba015bf30 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.899+0000 7fbba6c78640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba0071a60 con 0x7fbba00b3060 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbba013dc10 0x7fbba01675d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbba4a14640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbba00b3060 0x7fbba013a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbba013dc10 0x7fbba01675d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56294/0 (socket says 172.21.15.77:56294) 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbba4a14640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbba00b3060 0x7fbba013a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:48050/0 (socket says 172.21.15.77:48050) 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 -- 172.21.15.77:0/3507376323 learned_addr learned my addr 172.21.15.77:0/3507376323 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbba5215640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba015bf30 0x7fbba0159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 -- 172.21.15.77:0/3507376323 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbba00b3060 msgr2=0x7fbba013a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbba00b3060 0x7fbba013a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 -- 172.21.15.77:0/3507376323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba015bf30 msgr2=0x7fbba0159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba015bf30 0x7fbba0159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 -- 172.21.15.77:0/3507376323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba013ac70 con 0x7fbba013dc10 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9ffff640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbba013dc10 0x7fbba01675d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c02bee0 tx=0x7fbb8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9d7fa640 1 -- 172.21.15.77:0/3507376323 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c035040 con 0x7fbba013dc10 2024-03-06T07:22:12.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9d7fa640 1 -- 172.21.15.77:0/3507376323 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c03b920 con 0x7fbba013dc10 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba01403d0 con 0x7fbba013dc10 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbb9d7fa640 1 -- 172.21.15.77:0/3507376323 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c03bd10 con 0x7fbba013dc10 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.900+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba0168610 con 0x7fbba013dc10 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.901+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba004f330 con 0x7fbba013dc10 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.901+0000 7fbb9d7fa640 1 -- 172.21.15.77:0/3507376323 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb8c038020 con 0x7fbba013dc10 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.902+0000 7fbb9d7fa640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb8406b4d0 0x7fbb8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.902+0000 7fbb9d7fa640 1 -- 172.21.15.77:0/3507376323 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb8c0ac1d0 con 0x7fbba013dc10 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.902+0000 7fbba4a14640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb8406b4d0 0x7fbb8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.902+0000 7fbba4a14640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb8406b4d0 0x7fbb8406d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbb940011e0 tx=0x7fbb94007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:12.906+0000 7fbb9d7fa640 1 -- 172.21.15.77:0/3507376323 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb8c031800 con 0x7fbba013dc10 2024-03-06T07:22:13.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.078+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbba006eb00 con 0x7fbba013dc10 2024-03-06T07:22:13.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.079+0000 7fbb9d7fa640 1 -- 172.21.15.77:0/3507376323 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fbb8c07aa10 con 0x7fbba013dc10 2024-03-06T07:22:13.083 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:13.083 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.081+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb8406b4d0 msgr2=0x7fbb8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.081+0000 7fbba6c78640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb8406b4d0 0x7fbb8406d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbb940011e0 tx=0x7fbb94007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.081+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbba013dc10 msgr2=0x7fbba01675d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.081+0000 7fbba6c78640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbba013dc10 0x7fbba01675d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c02bee0 tx=0x7fbb8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.082+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 shutdown_connections 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.082+0000 7fbba6c78640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb8406b4d0 0x7fbb8406d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.082+0000 7fbba6c78640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbba013dc10 0x7fbba01675d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.082+0000 7fbba6c78640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba015bf30 0x7fbba0159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.082+0000 7fbba6c78640 1 --2- 172.21.15.77:0/3507376323 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbba00b3060 0x7fbba013a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.082+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 >> 172.21.15.77:0/3507376323 conn(0x7fbba00cf8a0 msgr2=0x7fbba006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.082+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 shutdown_connections 2024-03-06T07:22:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.082+0000 7fbba6c78640 1 -- 172.21.15.77:0/3507376323 wait complete. 2024-03-06T07:22:13.096 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:13.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:13.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.195+0000 7fadab1f7640 1 Processor -- start 2024-03-06T07:22:13.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.195+0000 7fadab1f7640 1 -- start start 2024-03-06T07:22:13.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.196+0000 7fadab1f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada40cf200 0x7fada40cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.196+0000 7fadab1f7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada41600a0 con 0x7fada40d06e0 2024-03-06T07:22:13.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.196+0000 7fadab1f7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada40d47f0 con 0x7fada415fc30 2024-03-06T07:22:13.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.196+0000 7fadab1f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada40ced70 con 0x7fada40cf200 2024-03-06T07:22:13.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.196+0000 7fada8f93640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fada40d06e0 0x7fada40d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45596/0 (socket says 172.21.15.77:45596) 2024-03-06T07:22:13.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.196+0000 7fada8f93640 1 -- 172.21.15.77:0/2874138549 learned_addr learned my addr 172.21.15.77:0/2874138549 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:13.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.196+0000 7fada9794640 1 --2- 172.21.15.77:0/2874138549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada40cf200 0x7fada40cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.197+0000 7fada37fe640 1 -- 172.21.15.77:0/2874138549 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866211357 0 0) 0x7fada41600a0 con 0x7fada40d06e0 2024-03-06T07:22:13.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.197+0000 7fada37fe640 1 -- 172.21.15.77:0/2874138549 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad8c003520 con 0x7fada40d06e0 2024-03-06T07:22:13.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.197+0000 7fada37fe640 1 -- 172.21.15.77:0/2874138549 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243668440 0 0) 0x7fada40d47f0 con 0x7fada415fc30 2024-03-06T07:22:13.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.197+0000 7fada37fe640 1 -- 172.21.15.77:0/2874138549 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada41600a0 con 0x7fada415fc30 2024-03-06T07:22:13.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.197+0000 7fada9794640 1 -- 172.21.15.77:0/2874138549 >> v1:172.21.15.158:6789/0 conn(0x7fada415fc30 legacy=0x7fada40b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.197+0000 7fada9794640 1 -- 172.21.15.77:0/2874138549 >> v1:172.21.15.77:6789/0 conn(0x7fada40d06e0 legacy=0x7fada40d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.197+0000 7fada9794640 1 -- 172.21.15.77:0/2874138549 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40d1ab0 con 0x7fada40cf200 2024-03-06T07:22:13.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.197+0000 7fada9794640 1 --2- 172.21.15.77:0/2874138549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada40cf200 0x7fada40cf690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009dc0 tx=0x7fad9c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db4ef08649e9cbd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fada37fe640 1 -- 172.21.15.77:0/2874138549 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c039070 con 0x7fada40cf200 2024-03-06T07:22:13.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fada37fe640 1 -- 172.21.15.77:0/2874138549 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c02ccd0 con 0x7fada40cf200 2024-03-06T07:22:13.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fada37fe640 1 -- 172.21.15.77:0/2874138549 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c034640 con 0x7fada40cf200 2024-03-06T07:22:13.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fadab1f7640 1 -- 172.21.15.77:0/2874138549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada40cf200 msgr2=0x7fada40cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fadab1f7640 1 --2- 172.21.15.77:0/2874138549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada40cf200 0x7fada40cf690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009dc0 tx=0x7fad9c02bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:13.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fadab1f7640 1 -- 172.21.15.77:0/2874138549 shutdown_connections 2024-03-06T07:22:13.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fadab1f7640 1 --2- 172.21.15.77:0/2874138549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada40cf200 0x7fada40cf690 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fadab1f7640 1 -- 172.21.15.77:0/2874138549 >> 172.21.15.77:0/2874138549 conn(0x7fada40d3cd0 msgr2=0x7fada40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fadab1f7640 1 -- 172.21.15.77:0/2874138549 shutdown_connections 2024-03-06T07:22:13.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.198+0000 7fadab1f7640 1 -- 172.21.15.77:0/2874138549 wait complete. 2024-03-06T07:22:13.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.199+0000 7fadab1f7640 1 Processor -- start 2024-03-06T07:22:13.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.199+0000 7fadab1f7640 1 -- start start 2024-03-06T07:22:13.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fadab1f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fada40cf200 0x7fada40b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fadab1f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fada40d06e0 0x7fada40c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fadab1f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada415fc30 0x7fada40c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fadab1f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada40ce990 con 0x7fada40cf200 2024-03-06T07:22:13.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fadab1f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada40d4ab0 con 0x7fada40d06e0 2024-03-06T07:22:13.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fadab1f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada40b44d0 con 0x7fada415fc30 2024-03-06T07:22:13.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fada8f93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fada40cf200 0x7fada40b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fada8f93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fada40cf200 0x7fada40b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56296/0 (socket says 172.21.15.77:56296) 2024-03-06T07:22:13.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fada8f93640 1 -- 172.21.15.77:0/3901287089 learned_addr learned my addr 172.21.15.77:0/3901287089 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:13.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fada3fff640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fada40d06e0 0x7fada40c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.200+0000 7fada9794640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada415fc30 0x7fada40c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada8f93640 1 -- 172.21.15.77:0/3901287089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada415fc30 msgr2=0x7fada40c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada8f93640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada415fc30 0x7fada40c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada8f93640 1 -- 172.21.15.77:0/3901287089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fada40d06e0 msgr2=0x7fada40c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada8f93640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fada40d06e0 0x7fada40c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada8f93640 1 -- 172.21.15.77:0/3901287089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40b9740 con 0x7fada40cf200 2024-03-06T07:22:13.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada8f93640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fada40cf200 0x7fada40b2be0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fad9400c410 tx=0x7fad9400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada17fa640 1 -- 172.21.15.77:0/3901287089 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94017070 con 0x7fada40cf200 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada17fa640 1 -- 172.21.15.77:0/3901287089 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad94005ce0 con 0x7fada40cf200 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada40b15e0 con 0x7fada40cf200 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fada17fa640 1 -- 172.21.15.77:0/3901287089 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad940072d0 con 0x7fada40cf200 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.201+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada40b1ba0 con 0x7fada40cf200 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.202+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada40a1c80 con 0x7fada40cf200 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.202+0000 7fada17fa640 1 -- 172.21.15.77:0/3901287089 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad9401a070 con 0x7fada40cf200 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.203+0000 7fada17fa640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9006b4d0 0x7fad9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.203+0000 7fada3fff640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9006b4d0 0x7fad9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.203+0000 7fada17fa640 1 -- 172.21.15.77:0/3901287089 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad940894f0 con 0x7fada40cf200 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.203+0000 7fada3fff640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9006b4d0 0x7fad9006d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fada40ce270 tx=0x7fad98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.207+0000 7fada17fa640 1 -- 172.21.15.77:0/3901287089 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad94057de0 con 0x7fada40cf200 2024-03-06T07:22:13.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.380+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fada4161b10 con 0x7fada40cf200 2024-03-06T07:22:13.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.381+0000 7fada17fa640 1 -- 172.21.15.77:0/3901287089 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad9405be80 con 0x7fada40cf200 2024-03-06T07:22:13.384 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.382+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9006b4d0 msgr2=0x7fad9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.382+0000 7fadab1f7640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9006b4d0 0x7fad9006d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fada40ce270 tx=0x7fad98009040 comp rx=0 tx=0).stop 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.382+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fada40cf200 msgr2=0x7fada40b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.382+0000 7fadab1f7640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fada40cf200 0x7fada40b2be0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fad9400c410 tx=0x7fad9400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.383+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 shutdown_connections 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.383+0000 7fadab1f7640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fada415fc30 0x7fada40c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.383+0000 7fadab1f7640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fada40d06e0 0x7fada40c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.383+0000 7fadab1f7640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9006b4d0 0x7fad9006d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.383+0000 7fadab1f7640 1 --2- 172.21.15.77:0/3901287089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fada40cf200 0x7fada40b2be0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.383+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 >> 172.21.15.77:0/3901287089 conn(0x7fada40d3cd0 msgr2=0x7fada40b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.383+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 shutdown_connections 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.383+0000 7fadab1f7640 1 -- 172.21.15.77:0/3901287089 wait complete. 2024-03-06T07:22:13.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.495+0000 7fe4bb46f640 1 Processor -- start 2024-03-06T07:22:13.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.496+0000 7fe4bb46f640 1 -- start start 2024-03-06T07:22:13.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.496+0000 7fe4bb46f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4b41614d0 0x7fe4b40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.496+0000 7fe4bb46f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b40d24b0 0x7fe4b40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.496+0000 7fe4bb46f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b41618b0 con 0x7fe4b40d1910 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.496+0000 7fe4bb46f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b40cf400 con 0x7fe4b41614d0 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.496+0000 7fe4bb46f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40cf6a0 con 0x7fe4b40d24b0 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.497+0000 7fe4b920b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4b41614d0 0x7fe4b40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.497+0000 7fe4b920b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4b41614d0 0x7fe4b40d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56304/0 (socket says 172.21.15.77:56304) 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.497+0000 7fe4b920b640 1 -- 172.21.15.77:0/771281092 learned_addr learned my addr 172.21.15.77:0/771281092 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.497+0000 7fe4b9a0c640 1 --2- 172.21.15.77:0/771281092 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b40d24b0 0x7fe4b40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.497+0000 7fe4b920b640 1 -- 172.21.15.77:0/771281092 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b40d24b0 msgr2=0x7fe4b40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.497+0000 7fe4b920b640 1 --2- 172.21.15.77:0/771281092 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b40d24b0 0x7fe4b40b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.497+0000 7fe4b920b640 1 -- 172.21.15.77:0/771281092 >> v1:172.21.15.158:6789/0 conn(0x7fe4b40d1910 legacy=0x7fe4b40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.497+0000 7fe4b920b640 1 -- 172.21.15.77:0/771281092 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40cf980 con 0x7fe4b41614d0 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4b920b640 1 --2- 172.21.15.77:0/771281092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b41614d0 0x7fe4b40d4860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009870 tx=0x7fe4a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143d850454da1737 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4abfff640 1 -- 172.21.15.77:0/771281092 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4034070 con 0x7fe4b41614d0 2024-03-06T07:22:13.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4abfff640 1 -- 172.21.15.77:0/771281092 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a4030840 con 0x7fe4b41614d0 2024-03-06T07:22:13.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4abfff640 1 -- 172.21.15.77:0/771281092 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4030c10 con 0x7fe4b41614d0 2024-03-06T07:22:13.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4bb46f640 1 -- 172.21.15.77:0/771281092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b41614d0 msgr2=0x7fe4b40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/771281092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b41614d0 0x7fe4b40d4860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009870 tx=0x7fe4a402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4bb46f640 1 -- 172.21.15.77:0/771281092 shutdown_connections 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/771281092 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b40d24b0 0x7fe4b40b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/771281092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b41614d0 0x7fe4b40d4860 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4bb46f640 1 -- 172.21.15.77:0/771281092 >> 172.21.15.77:0/771281092 conn(0x7fe4b40d3d40 msgr2=0x7fe4b40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4bb46f640 1 -- 172.21.15.77:0/771281092 shutdown_connections 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.498+0000 7fe4bb46f640 1 -- 172.21.15.77:0/771281092 wait complete. 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.499+0000 7fe4bb46f640 1 Processor -- start 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.499+0000 7fe4bb46f640 1 -- start start 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4bb46f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b40d1910 0x7fe4b4159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4bb46f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b40d24b0 0x7fe4b40b9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4bb46f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b41614d0 0x7fe4b40d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4bb46f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40b1370 con 0x7fe4b40d1910 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4bb46f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40b0910 con 0x7fe4b40d24b0 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4bb46f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40b14f0 con 0x7fe4b41614d0 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4b920b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b40d1910 0x7fe4b4159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4b920b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b40d1910 0x7fe4b4159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56310/0 (socket says 172.21.15.77:56310) 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4b920b640 1 -- 172.21.15.77:0/3409230484 learned_addr learned my addr 172.21.15.77:0/3409230484 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4b8a0a640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b40d24b0 0x7fe4b40b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.500+0000 7fe4b9a0c640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b41614d0 0x7fe4b40d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4b920b640 1 -- 172.21.15.77:0/3409230484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b41614d0 msgr2=0x7fe4b40d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4b920b640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b41614d0 0x7fe4b40d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4b920b640 1 -- 172.21.15.77:0/3409230484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b40d24b0 msgr2=0x7fe4b40b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4b920b640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b40d24b0 0x7fe4b40b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4b920b640 1 -- 172.21.15.77:0/3409230484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40b88c0 con 0x7fe4b40d1910 2024-03-06T07:22:13.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4b920b640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b40d1910 0x7fe4b4159df0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4000c00 tx=0x7fe4a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4a9ffb640 1 -- 172.21.15.77:0/3409230484 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4034040 con 0x7fe4b40d1910 2024-03-06T07:22:13.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4a9ffb640 1 -- 172.21.15.77:0/3409230484 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a4039d90 con 0x7fe4b40d1910 2024-03-06T07:22:13.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b41f4760 con 0x7fe4b40d1910 2024-03-06T07:22:13.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4a9ffb640 1 -- 172.21.15.77:0/3409230484 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4043630 con 0x7fe4b40d1910 2024-03-06T07:22:13.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.501+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b415a330 con 0x7fe4b40d1910 2024-03-06T07:22:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.502+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b4068ff0 con 0x7fe4b40d1910 2024-03-06T07:22:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.503+0000 7fe4a9ffb640 1 -- 172.21.15.77:0/3409230484 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4a404a060 con 0x7fe4b40d1910 2024-03-06T07:22:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.503+0000 7fe4a9ffb640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe49806b4d0 0x7fe49806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.503+0000 7fe4b8a0a640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe49806b4d0 0x7fe49806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.503+0000 7fe4a9ffb640 1 -- 172.21.15.77:0/3409230484 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4a40ac040 con 0x7fe4b40d1910 2024-03-06T07:22:13.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.503+0000 7fe4b8a0a640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe49806b4d0 0x7fe49806d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe4b406c3d0 tx=0x7fe4b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.507+0000 7fe4a9ffb640 1 -- 172.21.15.77:0/3409230484 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4a403e070 con 0x7fe4b40d1910 2024-03-06T07:22:13.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.678+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4b40747a0 con 0x7fe4b40d1910 2024-03-06T07:22:13.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.678+0000 7fe4a9ffb640 1 -- 172.21.15.77:0/3409230484 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe4a407a870 con 0x7fe4b40d1910 2024-03-06T07:22:13.682 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.680+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe49806b4d0 msgr2=0x7fe49806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.680+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe49806b4d0 0x7fe49806d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe4b406c3d0 tx=0x7fe4b0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.680+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b40d1910 msgr2=0x7fe4b4159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.680+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b40d1910 0x7fe4b4159df0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4000c00 tx=0x7fe4a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.681+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 shutdown_connections 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.681+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4b41614d0 0x7fe4b40d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.681+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4b40d24b0 0x7fe4b40b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.681+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe49806b4d0 0x7fe49806d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.681+0000 7fe4bb46f640 1 --2- 172.21.15.77:0/3409230484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4b40d1910 0x7fe4b4159df0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.681+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 >> 172.21.15.77:0/3409230484 conn(0x7fe4b40d3d40 msgr2=0x7fe4b406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.681+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 shutdown_connections 2024-03-06T07:22:13.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.681+0000 7fe4bb46f640 1 -- 172.21.15.77:0/3409230484 wait complete. 2024-03-06T07:22:13.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:13.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.794+0000 7fe650481640 1 Processor -- start 2024-03-06T07:22:13.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.794+0000 7fe650481640 1 -- start start 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe650481640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe64815bbb0 0x7fe64806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe650481640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe64806b4f0 0x7fe64815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe650481640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6480cea90 con 0x7fe64815ec40 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe650481640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe648161970 con 0x7fe64806b4f0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe650481640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6481624f0 con 0x7fe64815bbb0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe64da1c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe64806b4f0 0x7fe64815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe64da1c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe64806b4f0 0x7fe64815e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56322/0 (socket says 172.21.15.77:56322) 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe64da1c640 1 -- 172.21.15.77:0/3968188333 learned_addr learned my addr 172.21.15.77:0/3968188333 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe64e21d640 1 --2- 172.21.15.77:0/3968188333 >> v2:172.21.15.158:3300/0 conn(0x7fe64815bbb0 0x7fe64806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe64da1c640 1 -- 172.21.15.77:0/3968188333 >> v2:172.21.15.158:3300/0 conn(0x7fe64815bbb0 msgr2=0x7fe64806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe64da1c640 1 --2- 172.21.15.77:0/3968188333 >> v2:172.21.15.158:3300/0 conn(0x7fe64815bbb0 0x7fe64806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe64da1c640 1 -- 172.21.15.77:0/3968188333 >> v1:172.21.15.158:6789/0 conn(0x7fe64815ec40 legacy=0x7fe6480cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.795+0000 7fe64da1c640 1 -- 172.21.15.77:0/3968188333 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648161bc0 con 0x7fe64806b4f0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.796+0000 7fe64da1c640 1 --2- 172.21.15.77:0/3968188333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64815e700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe6480ce420 tx=0x7fe63802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9bd7e20287cea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.796+0000 7fe64d21b640 1 -- 172.21.15.77:0/3968188333 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638034070 con 0x7fe64806b4f0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.796+0000 7fe64d21b640 1 -- 172.21.15.77:0/3968188333 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe638030860 con 0x7fe64806b4f0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.796+0000 7fe64d21b640 1 -- 172.21.15.77:0/3968188333 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638030c30 con 0x7fe64806b4f0 2024-03-06T07:22:13.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.797+0000 7fe650481640 1 -- 172.21.15.77:0/3968188333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 msgr2=0x7fe64815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.797+0000 7fe650481640 1 --2- 172.21.15.77:0/3968188333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64815e700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe6480ce420 tx=0x7fe63802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.797+0000 7fe650481640 1 -- 172.21.15.77:0/3968188333 shutdown_connections 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.797+0000 7fe650481640 1 --2- 172.21.15.77:0/3968188333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64815e700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.797+0000 7fe650481640 1 --2- 172.21.15.77:0/3968188333 >> v2:172.21.15.158:3300/0 conn(0x7fe64815bbb0 0x7fe64806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.797+0000 7fe650481640 1 -- 172.21.15.77:0/3968188333 >> 172.21.15.77:0/3968188333 conn(0x7fe6480b2660 msgr2=0x7fe6480b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.797+0000 7fe650481640 1 -- 172.21.15.77:0/3968188333 shutdown_connections 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.797+0000 7fe650481640 1 -- 172.21.15.77:0/3968188333 wait complete. 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.798+0000 7fe650481640 1 Processor -- start 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.798+0000 7fe650481640 1 -- start start 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe650481640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64814d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe650481640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe64815ec40 0x7fe6481f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe650481640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6481556c0 0x7fe648155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe650481640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe648072cb0 con 0x7fe64806b4f0 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe650481640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe648162180 con 0x7fe64815ec40 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe650481640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64815f0d0 con 0x7fe6481556c0 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64814d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64814d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56330/0 (socket says 172.21.15.77:56330) 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 -- 172.21.15.77:0/3907932634 learned_addr learned my addr 172.21.15.77:0/3907932634 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64da1c640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe64815ec40 0x7fe6481f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64ea1e640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6481556c0 0x7fe648155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 -- 172.21.15.77:0/3907932634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6481556c0 msgr2=0x7fe648155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6481556c0 0x7fe648155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 -- 172.21.15.77:0/3907932634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe64815ec40 msgr2=0x7fe6481f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe64815ec40 0x7fe6481f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 -- 172.21.15.77:0/3907932634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6481600e0 con 0x7fe64806b4f0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe64e21d640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64814d070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe64400efc0 tx=0x7fe64400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe63effd640 1 -- 172.21.15.77:0/3907932634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644018020 con 0x7fe64806b4f0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe63effd640 1 -- 172.21.15.77:0/3907932634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644014480 con 0x7fe64806b4f0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6481603c0 con 0x7fe64806b4f0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.799+0000 7fe63effd640 1 -- 172.21.15.77:0/3907932634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644014850 con 0x7fe64806b4f0 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.800+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64814b9d0 con 0x7fe64806b4f0 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.800+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6480733b0 con 0x7fe64806b4f0 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.801+0000 7fe63effd640 1 -- 172.21.15.77:0/3907932634 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe644005d10 con 0x7fe64806b4f0 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.801+0000 7fe63effd640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe62c06b4d0 0x7fe62c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.801+0000 7fe63effd640 1 -- 172.21.15.77:0/3907932634 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe64408ab60 con 0x7fe64806b4f0 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.801+0000 7fe64da1c640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe62c06b4d0 0x7fe62c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.802+0000 7fe64da1c640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe62c06b4d0 0x7fe62c06d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe638036000 tx=0x7fe63803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.805+0000 7fe63effd640 1 -- 172.21.15.77:0/3907932634 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe644022070 con 0x7fe64806b4f0 2024-03-06T07:22:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.975+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe648068ff0 con 0x7fe64806b4f0 2024-03-06T07:22:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.976+0000 7fe63effd640 1 -- 172.21.15.77:0/3907932634 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe644010040 con 0x7fe64806b4f0 2024-03-06T07:22:13.979 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe62c06b4d0 msgr2=0x7fe62c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe62c06b4d0 0x7fe62c06d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe638036000 tx=0x7fe63803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 msgr2=0x7fe64814d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64814d070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe64400efc0 tx=0x7fe64400c490 comp rx=0 tx=0).stop 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 shutdown_connections 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6481556c0 0x7fe648155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe64815ec40 0x7fe6481f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe62c06b4d0 0x7fe62c06d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 --2- 172.21.15.77:0/3907932634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe64806b4f0 0x7fe64814d070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 >> 172.21.15.77:0/3907932634 conn(0x7fe6480b2660 msgr2=0x7fe6480cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 shutdown_connections 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:13.978+0000 7fe650481640 1 -- 172.21.15.77:0/3907932634 wait complete. 2024-03-06T07:22:13.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.087+0000 7fde624fa640 1 Processor -- start 2024-03-06T07:22:14.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.088+0000 7fde624fa640 1 -- start start 2024-03-06T07:22:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.088+0000 7fde624fa640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c168f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.088+0000 7fde624fa640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde5c0cfd80 con 0x7fde5c158400 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.088+0000 7fde624fa640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde5c15e7d0 con 0x7fde5c1587e0 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.088+0000 7fde624fa640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c15e590 con 0x7fde5c06b6c0 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.088+0000 7fde5bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fde5c158400 0x7fde5c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45604/0 (socket says 172.21.15.77:45604) 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.088+0000 7fde5bfff640 1 -- 172.21.15.77:0/2892523497 learned_addr learned my addr 172.21.15.77:0/2892523497 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.088+0000 7fde60a97640 1 --2- 172.21.15.77:0/2892523497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c168f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.089+0000 7fde5affd640 1 -- 172.21.15.77:0/2892523497 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782350541 0 0) 0x7fde5c0cfd80 con 0x7fde5c158400 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.089+0000 7fde5affd640 1 -- 172.21.15.77:0/2892523497 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde48003520 con 0x7fde5c158400 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.089+0000 7fde5affd640 1 -- 172.21.15.77:0/2892523497 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533792672 0 0) 0x7fde5c15e7d0 con 0x7fde5c1587e0 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.089+0000 7fde5affd640 1 -- 172.21.15.77:0/2892523497 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde5c0cfd80 con 0x7fde5c1587e0 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.089+0000 7fde60a97640 1 -- 172.21.15.77:0/2892523497 >> v1:172.21.15.158:6789/0 conn(0x7fde5c1587e0 legacy=0x7fde5c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.089+0000 7fde60a97640 1 -- 172.21.15.77:0/2892523497 >> v1:172.21.15.77:6789/0 conn(0x7fde5c158400 legacy=0x7fde5c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.089+0000 7fde60a97640 1 -- 172.21.15.77:0/2892523497 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c15eab0 con 0x7fde5c06b6c0 2024-03-06T07:22:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde60a97640 1 --2- 172.21.15.77:0/2892523497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c168f60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde54009dc0 tx=0x7fde5402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c914445b82040c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde5affd640 1 -- 172.21.15.77:0/2892523497 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54039070 con 0x7fde5c06b6c0 2024-03-06T07:22:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde5affd640 1 -- 172.21.15.77:0/2892523497 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5402cc40 con 0x7fde5c06b6c0 2024-03-06T07:22:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde5affd640 1 -- 172.21.15.77:0/2892523497 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54034440 con 0x7fde5c06b6c0 2024-03-06T07:22:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde624fa640 1 -- 172.21.15.77:0/2892523497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 msgr2=0x7fde5c168f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde624fa640 1 --2- 172.21.15.77:0/2892523497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c168f60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde54009dc0 tx=0x7fde5402be90 comp rx=0 tx=0).stop 2024-03-06T07:22:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde624fa640 1 -- 172.21.15.77:0/2892523497 shutdown_connections 2024-03-06T07:22:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde624fa640 1 --2- 172.21.15.77:0/2892523497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c168f60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde624fa640 1 -- 172.21.15.77:0/2892523497 >> 172.21.15.77:0/2892523497 conn(0x7fde5c06c0c0 msgr2=0x7fde5c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde624fa640 1 -- 172.21.15.77:0/2892523497 shutdown_connections 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.090+0000 7fde624fa640 1 -- 172.21.15.77:0/2892523497 wait complete. 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.091+0000 7fde624fa640 1 Processor -- start 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde624fa640 1 -- start start 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde624fa640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde624fa640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde5c158400 0x7fde5c1f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde624fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde5c1587e0 0x7fde5c154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde624fa640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c0cfa40 con 0x7fde5c1587e0 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde624fa640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c0d00a0 con 0x7fde5c158400 2024-03-06T07:22:14.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde624fa640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c0cd260 con 0x7fde5c06b6c0 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde60a97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde5c1587e0 0x7fde5c154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde5bfff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde60a97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde5c1587e0 0x7fde5c154d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56332/0 (socket says 172.21.15.77:56332) 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde60a97640 1 -- 172.21.15.77:0/3426805548 learned_addr learned my addr 172.21.15.77:0/3426805548 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.092+0000 7fde5b7fe640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde5c158400 0x7fde5c1f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde60a97640 1 -- 172.21.15.77:0/3426805548 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 msgr2=0x7fde5c1528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde60a97640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c1528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde60a97640 1 -- 172.21.15.77:0/3426805548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde5c158400 msgr2=0x7fde5c1f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde60a97640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde5c158400 0x7fde5c1f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde60a97640 1 -- 172.21.15.77:0/3426805548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c161220 con 0x7fde5c1587e0 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde60a97640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde5c1587e0 0x7fde5c154d70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fde54037040 tx=0x7fde54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde58ff9640 1 -- 172.21.15.77:0/3426805548 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54039050 con 0x7fde5c1587e0 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde58ff9640 1 -- 172.21.15.77:0/3426805548 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54045070 con 0x7fde5c1587e0 2024-03-06T07:22:14.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c161500 con 0x7fde5c1587e0 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde58ff9640 1 -- 172.21.15.77:0/3426805548 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54040690 con 0x7fde5c1587e0 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.093+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c0b2a30 con 0x7fde5c1587e0 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.094+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5c04f330 con 0x7fde5c1587e0 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.094+0000 7fde58ff9640 1 -- 172.21.15.77:0/3426805548 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde54040830 con 0x7fde5c1587e0 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.095+0000 7fde58ff9640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde3c06b4d0 0x7fde3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.095+0000 7fde58ff9640 1 -- 172.21.15.77:0/3426805548 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde540aca90 con 0x7fde5c1587e0 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.095+0000 7fde5bfff640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde3c06b4d0 0x7fde3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.095+0000 7fde5bfff640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde3c06b4d0 0x7fde3c06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde4c000f60 tx=0x7fde4c007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.100+0000 7fde58ff9640 1 -- 172.21.15.77:0/3426805548 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde5407b230 con 0x7fde5c1587e0 2024-03-06T07:22:14.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.305+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde5c0cdf60 con 0x7fde5c1587e0 2024-03-06T07:22:14.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.306+0000 7fde58ff9640 1 -- 172.21.15.77:0/3426805548 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fde5407f2d0 con 0x7fde5c1587e0 2024-03-06T07:22:14.309 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:14.309 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.285900+0000","modified":"2024-03-06T07:22:00.291221+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-06T07:22:14.311 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-06T07:22:14.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde3c06b4d0 msgr2=0x7fde3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde3c06b4d0 0x7fde3c06d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde4c000f60 tx=0x7fde4c007d70 comp rx=0 tx=0).stop 2024-03-06T07:22:14.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde5c1587e0 msgr2=0x7fde5c154d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde5c1587e0 0x7fde5c154d70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fde54037040 tx=0x7fde54004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:14.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 shutdown_connections 2024-03-06T07:22:14.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde3c06b4d0 0x7fde3c06d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde5c1587e0 0x7fde5c154d70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde5c158400 0x7fde5c1f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 --2- 172.21.15.77:0/3426805548 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde5c06b6c0 0x7fde5c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.307+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 >> 172.21.15.77:0/3426805548 conn(0x7fde5c06c0c0 msgr2=0x7fde5c0cf330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.308+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 shutdown_connections 2024-03-06T07:22:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.308+0000 7fde624fa640 1 -- 172.21.15.77:0/3426805548 wait complete. 2024-03-06T07:22:14.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.419+0000 7facd0bbc640 1 Processor -- start 2024-03-06T07:22:14.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.419+0000 7facd0bbc640 1 -- start start 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facd0bbc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faccc1424f0 0x7faccc0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facd0bbc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0cc980 con 0x7faccc06da40 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facd0bbc640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0ccc60 con 0x7faccc0cd7a0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facd0bbc640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faccc0cd220 con 0x7faccc1424f0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7faccad9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faccc06da40 0x7faccc071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45620/0 (socket says 172.21.15.77:45620) 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7faccad9d640 1 -- 172.21.15.77:0/1083323912 learned_addr learned my addr 172.21.15.77:0/1083323912 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facca59c640 1 --2- 172.21.15.77:0/1083323912 >> v2:172.21.15.158:3300/0 conn(0x7faccc1424f0 0x7faccc0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facc9d9b640 1 -- 172.21.15.77:0/1083323912 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270992001 0 0) 0x7faccc0cc980 con 0x7faccc06da40 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facc9d9b640 1 -- 172.21.15.77:0/1083323912 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8003520 con 0x7faccc06da40 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facca59c640 1 -- 172.21.15.77:0/1083323912 >> v1:172.21.15.158:6789/0 conn(0x7faccc0cd7a0 legacy=0x7faccc06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facc9d9b640 1 -- 172.21.15.77:0/1083323912 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607891315 0 0) 0x7faccc0ccc60 con 0x7faccc0cd7a0 2024-03-06T07:22:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facca59c640 1 -- 172.21.15.77:0/1083323912 >> v1:172.21.15.77:6789/0 conn(0x7faccc06da40 legacy=0x7faccc071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facca59c640 1 -- 172.21.15.77:0/1083323912 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0ccf40 con 0x7faccc1424f0 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facca59c640 1 --2- 172.21.15.77:0/1083323912 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 0x7faccc0cc2f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7facbc0097a0 tx=0x7facbc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca523e32919d321e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facc9d9b640 1 -- 172.21.15.77:0/1083323912 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc034070 con 0x7faccc1424f0 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facc9d9b640 1 -- 172.21.15.77:0/1083323912 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc030610 con 0x7faccc1424f0 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.421+0000 7facc9d9b640 1 -- 172.21.15.77:0/1083323912 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc0309e0 con 0x7faccc1424f0 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.422+0000 7facd0bbc640 1 -- 172.21.15.77:0/1083323912 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 msgr2=0x7faccc0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.422+0000 7facd0bbc640 1 --2- 172.21.15.77:0/1083323912 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 0x7faccc0cc2f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7facbc0097a0 tx=0x7facbc02b960 comp rx=0 tx=0).stop 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.422+0000 7facd0bbc640 1 -- 172.21.15.77:0/1083323912 shutdown_connections 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.422+0000 7facd0bbc640 1 --2- 172.21.15.77:0/1083323912 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 0x7faccc0cc2f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.422+0000 7facd0bbc640 1 -- 172.21.15.77:0/1083323912 >> 172.21.15.77:0/1083323912 conn(0x7faccc0cf8a0 msgr2=0x7faccc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.422+0000 7facd0bbc640 1 -- 172.21.15.77:0/1083323912 shutdown_connections 2024-03-06T07:22:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.422+0000 7facd0bbc640 1 -- 172.21.15.77:0/1083323912 wait complete. 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.423+0000 7facd0bbc640 1 Processor -- start 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.423+0000 7facd0bbc640 1 -- start start 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facd0bbc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faccc06da40 0x7faccc15c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facd0bbc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faccc0cd7a0 0x7faccc156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facd0bbc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 0x7faccc1574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facd0bbc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc15d070 con 0x7faccc06da40 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facd0bbc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc15cef0 con 0x7faccc1424f0 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facd0bbc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faccc15d1f0 con 0x7faccc0cd7a0 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faccc06da40 0x7faccc15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faccc06da40 0x7faccc15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56336/0 (socket says 172.21.15.77:56336) 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 -- 172.21.15.77:0/2746860446 learned_addr learned my addr 172.21.15.77:0/2746860446 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7faccad9d640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 0x7faccc1574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facc3fff640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faccc0cd7a0 0x7faccc156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 -- 172.21.15.77:0/2746860446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faccc0cd7a0 msgr2=0x7faccc156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faccc0cd7a0 0x7faccc156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 -- 172.21.15.77:0/2746860446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 msgr2=0x7faccc1574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 0x7faccc1574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 -- 172.21.15.77:0/2746860446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc140350 con 0x7faccc06da40 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facca59c640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faccc06da40 0x7faccc15c570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7facbc0098c0 tx=0x7facbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facc2ffd640 1 -- 172.21.15.77:0/2746860446 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc049070 con 0x7faccc06da40 2024-03-06T07:22:14.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facc2ffd640 1 -- 172.21.15.77:0/2746860446 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc03a890 con 0x7faccc06da40 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facc2ffd640 1 -- 172.21.15.77:0/2746860446 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc034040 con 0x7faccc06da40 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc140630 con 0x7faccc06da40 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.424+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc140bf0 con 0x7faccc06da40 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.425+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc15bf10 con 0x7faccc06da40 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.426+0000 7facc2ffd640 1 -- 172.21.15.77:0/2746860446 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facbc04c070 con 0x7faccc06da40 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.426+0000 7facc2ffd640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7facac06b4d0 0x7facac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.426+0000 7facc2ffd640 1 -- 172.21.15.77:0/2746860446 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facbc0abe10 con 0x7faccc06da40 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.426+0000 7facc3fff640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7facac06b4d0 0x7facac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.426+0000 7facc3fff640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7facac06b4d0 0x7facac06d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faccc159eb0 tx=0x7facb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.431+0000 7facc2ffd640 1 -- 172.21.15.77:0/2746860446 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faccc15bf10 con 0x7faccc06da40 2024-03-06T07:22:14.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.603+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faccc04f330 con 0x7faccc06da40 2024-03-06T07:22:14.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.604+0000 7facc2ffd640 1 -- 172.21.15.77:0/2746860446 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7facbc03f070 con 0x7faccc06da40 2024-03-06T07:22:14.608 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:14.608 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7facac06b4d0 msgr2=0x7facac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7facac06b4d0 0x7facac06d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faccc159eb0 tx=0x7facb4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faccc06da40 msgr2=0x7faccc15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faccc06da40 0x7faccc15c570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7facbc0098c0 tx=0x7facbc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 shutdown_connections 2024-03-06T07:22:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faccc1424f0 0x7faccc1574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faccc0cd7a0 0x7faccc156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7facac06b4d0 0x7facac06d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 --2- 172.21.15.77:0/2746860446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faccc06da40 0x7faccc15c570 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 >> 172.21.15.77:0/2746860446 conn(0x7faccc0cf8a0 msgr2=0x7faccc0cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.606+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 shutdown_connections 2024-03-06T07:22:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.607+0000 7facd0bbc640 1 -- 172.21.15.77:0/2746860446 wait complete. 2024-03-06T07:22:14.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.718+0000 7f8e56676640 1 Processor -- start 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.718+0000 7f8e56676640 1 -- start start 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e56676640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e500b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e56676640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8e5015ef60 0x7f8e500b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e56676640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5015f580 con 0x7f8e5015bf30 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e56676640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e500cf3f0 con 0x7f8e5015ef60 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e56676640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5015b210 con 0x7f8e5013dc10 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e4ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e500b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e54c13640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8e5015ef60 0x7f8e500b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e4f7fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8e5015bf30 0x7f8e5015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:46262/0 (socket says 172.21.15.77:46262) 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e4f7fe640 1 -- 172.21.15.77:0/4164587094 learned_addr learned my addr 172.21.15.77:0/4164587094 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e4ffff640 1 -- 172.21.15.77:0/4164587094 >> v2:172.21.15.158:3300/0 conn(0x7f8e5015ef60 msgr2=0x7f8e500b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e4ffff640 1 --2- 172.21.15.77:0/4164587094 >> v2:172.21.15.158:3300/0 conn(0x7f8e5015ef60 0x7f8e500b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e4ffff640 1 -- 172.21.15.77:0/4164587094 >> v1:172.21.15.158:6789/0 conn(0x7f8e5015bf30 legacy=0x7f8e5015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e4effd640 1 -- 172.21.15.77:0/4164587094 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531650591 0 0) 0x7f8e5015f580 con 0x7f8e5015bf30 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e4ffff640 1 -- 172.21.15.77:0/4164587094 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5015b450 con 0x7f8e5013dc10 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.719+0000 7f8e54c13640 1 --2- 172.21.15.77:0/4164587094 >> v2:172.21.15.158:3300/0 conn(0x7f8e5015ef60 0x7f8e500b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.720+0000 7f8e4ffff640 1 --2- 172.21.15.77:0/4164587094 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e500b2950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8e400097a0 tx=0x7f8e4002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24a4b1e2dea2fbdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.720+0000 7f8e4effd640 1 -- 172.21.15.77:0/4164587094 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e400315a0 con 0x7f8e5013dc10 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.720+0000 7f8e4effd640 1 -- 172.21.15.77:0/4164587094 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e40037070 con 0x7f8e5013dc10 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.720+0000 7f8e4effd640 1 -- 172.21.15.77:0/4164587094 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40031960 con 0x7f8e5013dc10 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.721+0000 7f8e56676640 1 -- 172.21.15.77:0/4164587094 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 msgr2=0x7f8e500b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.721+0000 7f8e56676640 1 --2- 172.21.15.77:0/4164587094 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e500b2950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8e400097a0 tx=0x7f8e4002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.721+0000 7f8e56676640 1 -- 172.21.15.77:0/4164587094 shutdown_connections 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.721+0000 7f8e56676640 1 --2- 172.21.15.77:0/4164587094 >> v2:172.21.15.158:3300/0 conn(0x7f8e5015ef60 0x7f8e500b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.721+0000 7f8e56676640 1 --2- 172.21.15.77:0/4164587094 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e500b2950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.721+0000 7f8e56676640 1 -- 172.21.15.77:0/4164587094 >> 172.21.15.77:0/4164587094 conn(0x7f8e500cf8a0 msgr2=0x7f8e500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.721+0000 7f8e56676640 1 -- 172.21.15.77:0/4164587094 shutdown_connections 2024-03-06T07:22:14.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.721+0000 7f8e56676640 1 -- 172.21.15.77:0/4164587094 wait complete. 2024-03-06T07:22:14.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.722+0000 7f8e56676640 1 Processor -- start 2024-03-06T07:22:14.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.722+0000 7f8e56676640 1 -- start start 2024-03-06T07:22:14.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e56676640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e501f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e56676640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e5015bf30 0x7f8e501efc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e56676640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e5015ef60 0x7f8e50071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e56676640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500cd7a0 con 0x7f8e5015bf30 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e56676640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5013dff0 con 0x7f8e5015ef60 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e56676640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500cd920 con 0x7f8e5013dc10 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e5015bf30 0x7f8e501efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e5015bf30 0x7f8e501efc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56350/0 (socket says 172.21.15.77:56350) 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 -- 172.21.15.77:0/1368091606 learned_addr learned my addr 172.21.15.77:0/1368091606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4ffff640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e501f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e54c13640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e5015ef60 0x7f8e50071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 -- 172.21.15.77:0/1368091606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 msgr2=0x7f8e501f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e501f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 -- 172.21.15.77:0/1368091606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e5015ef60 msgr2=0x7f8e50071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e5015ef60 0x7f8e50071cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 -- 172.21.15.77:0/1368091606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e501f3170 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.723+0000 7f8e4f7fe640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e5015bf30 0x7f8e501efc20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400c4a0 tx=0x7f8e4400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.724+0000 7f8e4cff9640 1 -- 172.21.15.77:0/1368091606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e440074e0 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.724+0000 7f8e4cff9640 1 -- 172.21.15.77:0/1368091606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e440047e0 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.724+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e501f3450 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.724+0000 7f8e4cff9640 1 -- 172.21.15.77:0/1368091606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44004c50 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.724+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e501ef700 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.725+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e501ef4f0 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.725+0000 7f8e4cff9640 1 -- 172.21.15.77:0/1368091606 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e44011070 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.725+0000 7f8e4cff9640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e3c06b4d0 0x7f8e3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.725+0000 7f8e4cff9640 1 -- 172.21.15.77:0/1368091606 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e44089cc0 con 0x7f8e5015bf30 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.725+0000 7f8e4ffff640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e3c06b4d0 0x7f8e3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.726+0000 7f8e4ffff640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e3c06b4d0 0x7f8e3c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8e40036000 tx=0x7f8e40008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.730+0000 7f8e4cff9640 1 -- 172.21.15.77:0/1368091606 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e44058bd0 con 0x7f8e5015bf30 2024-03-06T07:22:14.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.937+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e5004f330 con 0x7f8e5015bf30 2024-03-06T07:22:14.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.937+0000 7f8e4cff9640 1 -- 172.21.15.77:0/1368091606 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f8e440584d0 con 0x7f8e5015bf30 2024-03-06T07:22:14.939 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:14.939 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.285900+0000","modified":"2024-03-06T07:22:00.291221+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-06T07:22:14.940 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-06T07:22:14.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e3c06b4d0 msgr2=0x7f8e3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e3c06b4d0 0x7f8e3c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8e40036000 tx=0x7f8e40008030 comp rx=0 tx=0).stop 2024-03-06T07:22:14.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e5015bf30 msgr2=0x7f8e501efc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e5015bf30 0x7f8e501efc20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400c4a0 tx=0x7f8e4400c8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 shutdown_connections 2024-03-06T07:22:14.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e3c06b4d0 0x7f8e3c06d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e5015bf30 0x7f8e501efc20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e5015ef60 0x7f8e50071cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 --2- 172.21.15.77:0/1368091606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e5013dc10 0x7f8e501f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 >> 172.21.15.77:0/1368091606 conn(0x7f8e500cf8a0 msgr2=0x7f8e5015d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 shutdown_connections 2024-03-06T07:22:14.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:14.938+0000 7f8e56676640 1 -- 172.21.15.77:0/1368091606 wait complete. 2024-03-06T07:22:15.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:58.285900+0000', 'modified': '2024-03-06T07:22:00.291221+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/3488131752', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6834', 'nonce': 3488131752}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 3488131752}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4282, 'qdb_cluster': [4282]} 2024-03-06T07:22:15.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:15.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:22:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.457+0000 7f413c15c640 1 Processor -- start 2024-03-06T07:22:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.457+0000 7f413c15c640 1 -- start start 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f413c15c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41341444c0 0x7f413406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f413c15c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4134070fe0 0x7f41340cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f413c15c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413406d370 con 0x7f413415b2d0 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f413c15c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41341448a0 con 0x7f41341444c0 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f413c15c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413406d650 con 0x7f4134070fe0 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f4139ef8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41341444c0 0x7f413406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f4139ef8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f41341444c0 0x7f413406ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56354/0 (socket says 172.21.15.77:56354) 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f4139ef8640 1 -- 172.21.15.77:0/3956306997 learned_addr learned my addr 172.21.15.77:0/3956306997 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f413a6f9640 1 --2- 172.21.15.77:0/3956306997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4134070fe0 0x7f41340cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f4139ef8640 1 -- 172.21.15.77:0/3956306997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4134070fe0 msgr2=0x7f41340cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f4139ef8640 1 --2- 172.21.15.77:0/3956306997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4134070fe0 0x7f41340cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f4139ef8640 1 -- 172.21.15.77:0/3956306997 >> v1:172.21.15.158:6789/0 conn(0x7f413415b2d0 legacy=0x7f41340708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.458+0000 7f4139ef8640 1 -- 172.21.15.77:0/3956306997 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413406d930 con 0x7f41341444c0 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.459+0000 7f413a6f9640 1 --2- 172.21.15.77:0/3956306997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4134070fe0 0x7f41340cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.459+0000 7f4139ef8640 1 --2- 172.21.15.77:0/3956306997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41341444c0 0x7f413406ca00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4130009e50 tx=0x7f413002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911db18da4731bcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.459+0000 7f4138ef6640 1 -- 172.21.15.77:0/3956306997 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413002cb00 con 0x7f41341444c0 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.459+0000 7f4138ef6640 1 -- 172.21.15.77:0/3956306997 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413002cca0 con 0x7f41341444c0 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.459+0000 7f4138ef6640 1 -- 172.21.15.77:0/3956306997 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130034610 con 0x7f41341444c0 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.460+0000 7f413c15c640 1 -- 172.21.15.77:0/3956306997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41341444c0 msgr2=0x7f413406ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.460+0000 7f413c15c640 1 --2- 172.21.15.77:0/3956306997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41341444c0 0x7f413406ca00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4130009e50 tx=0x7f413002bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.460+0000 7f413c15c640 1 -- 172.21.15.77:0/3956306997 shutdown_connections 2024-03-06T07:22:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.460+0000 7f413c15c640 1 --2- 172.21.15.77:0/3956306997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4134070fe0 0x7f41340cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.460+0000 7f413c15c640 1 --2- 172.21.15.77:0/3956306997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41341444c0 0x7f413406ca00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.460+0000 7f413c15c640 1 -- 172.21.15.77:0/3956306997 >> 172.21.15.77:0/3956306997 conn(0x7f41340b1c80 msgr2=0x7f41340cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.460+0000 7f413c15c640 1 -- 172.21.15.77:0/3956306997 shutdown_connections 2024-03-06T07:22:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.460+0000 7f413c15c640 1 -- 172.21.15.77:0/3956306997 wait complete. 2024-03-06T07:22:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.461+0000 7f413c15c640 1 Processor -- start 2024-03-06T07:22:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.461+0000 7f413c15c640 1 -- start start 2024-03-06T07:22:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f413c15c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4134070fe0 0x7f413413e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f413c15c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41341444c0 0x7f413413d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f413c15c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f413415b2d0 0x7f413415a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f413c15c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413406cf40 con 0x7f4134070fe0 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f413c15c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413406def0 con 0x7f413415b2d0 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f413c15c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413406d0c0 con 0x7f41341444c0 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4134070fe0 0x7f413413e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4134070fe0 0x7f413413e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56362/0 (socket says 172.21.15.77:56362) 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 -- 172.21.15.77:0/2321352954 learned_addr learned my addr 172.21.15.77:0/2321352954 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f41396f7640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41341444c0 0x7f413413d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f413a6f9640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f413415b2d0 0x7f413415a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 -- 172.21.15.77:0/2321352954 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41341444c0 msgr2=0x7f413413d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41341444c0 0x7f413413d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 -- 172.21.15.77:0/2321352954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f413415b2d0 msgr2=0x7f413415a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f413415b2d0 0x7f413415a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 -- 172.21.15.77:0/2321352954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413413de50 con 0x7f4134070fe0 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.462+0000 7f4139ef8640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4134070fe0 0x7f413413e580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4130000f80 tx=0x7f4130004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.463+0000 7f412a7fc640 1 -- 172.21.15.77:0/2321352954 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130043020 con 0x7f4134070fe0 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.463+0000 7f412a7fc640 1 -- 172.21.15.77:0/2321352954 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413002ce50 con 0x7f4134070fe0 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.463+0000 7f413c15c640 1 -- 172.21.15.77:0/2321352954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413413ec10 con 0x7f4134070fe0 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.463+0000 7f412a7fc640 1 -- 172.21.15.77:0/2321352954 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130009510 con 0x7f4134070fe0 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.463+0000 7f413c15c640 1 -- 172.21.15.77:0/2321352954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413413f1d0 con 0x7f4134070fe0 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.464+0000 7f412a7fc640 1 -- 172.21.15.77:0/2321352954 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4130046020 con 0x7f4134070fe0 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.464+0000 7f412a7fc640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f411806b4d0 0x7f411806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.464+0000 7f41396f7640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f411806b4d0 0x7f411806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.464+0000 7f412a7fc640 1 -- 172.21.15.77:0/2321352954 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41300ac7c0 con 0x7f4134070fe0 2024-03-06T07:22:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.465+0000 7f41396f7640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f411806b4d0 0x7f411806d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f413415abc0 tx=0x7f4124009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.465+0000 7f413ee13740 1 Processor -- start 2024-03-06T07:22:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.465+0000 7f413ee13740 1 -- start start 2024-03-06T07:22:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f413ee13740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5da5870 0x561bc5da5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f413ee13740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5da98c0 0x561bc5dab360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f413ee13740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f413ee13740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bc5d0f6b0 con 0x561bc5da98c0 2024-03-06T07:22:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f413ee13740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bc5cb8a60 con 0x561bc5da5870 2024-03-06T07:22:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f413ee13740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x561bc5d0b750 con 0x561bc5dd5600 2024-03-06T07:22:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f41396f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5da98c0 0x561bc5dab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f41396f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5da98c0 0x561bc5dab360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56374/0 (socket says 172.21.15.77:56374) 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f41396f7640 1 -- 172.21.15.77:0/1847012756 learned_addr learned my addr 172.21.15.77:0/1847012756 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f4139ef8640 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5da5870 0x561bc5da5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f413a6f9640 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f41396f7640 1 -- 172.21.15.77:0/1847012756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 msgr2=0x561bc5dd5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f41396f7640 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f41396f7640 1 -- 172.21.15.77:0/1847012756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5da5870 msgr2=0x561bc5da5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f41396f7640 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5da5870 0x561bc5da5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.466+0000 7f41396f7640 1 -- 172.21.15.77:0/1847012756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bc5cd6f30 con 0x561bc5da98c0 2024-03-06T07:22:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f41396f7640 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5da98c0 0x561bc5dab360 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f412400fb40 tx=0x7f4124012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa002e273be1d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f41177fe640 1 -- 172.21.15.77:0/1847012756 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124026070 con 0x561bc5da98c0 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f41177fe640 1 -- 172.21.15.77:0/1847012756 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41240075b0 con 0x561bc5da98c0 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f41177fe640 1 -- 172.21.15.77:0/1847012756 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41240069f0 con 0x561bc5da98c0 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f413ee13740 1 -- 172.21.15.77:0/1847012756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5da98c0 msgr2=0x561bc5dab360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f413ee13740 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5da98c0 0x561bc5dab360 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f412400fb40 tx=0x7f4124012000 comp rx=0 tx=0).stop 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f413ee13740 1 -- 172.21.15.77:0/1847012756 shutdown_connections 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f413ee13740 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5da98c0 0x561bc5dab360 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f413ee13740 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f413ee13740 1 --2- 172.21.15.77:0/1847012756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5da5870 0x561bc5da5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f413ee13740 1 -- 172.21.15.77:0/1847012756 >> 172.21.15.77:0/1847012756 conn(0x561bc5da78f0 msgr2=0x561bc5da91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.467+0000 7f413ee13740 1 -- 172.21.15.77:0/1847012756 shutdown_connections 2024-03-06T07:22:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.468+0000 7f413ee13740 1 -- 172.21.15.77:0/1847012756 wait complete. 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.468+0000 7f413ee13740 1 -- start start 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.468+0000 7f413ee13740 1 Processor -- start 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413ee13740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5dcfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413ee13740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5dd86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413ee13740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413ee13740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bc5da9d50 con 0x561bc5dd5600 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413ee13740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bc5ccbea0 con 0x561bc5dd7150 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413ee13740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x561bc5daa030 con 0x561bc5dd7530 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f4139ef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5dcfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413ee13740 10 client.0 _ll_register_callbacks cb 0x561bc5ccb870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f4139ef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5dcfbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56388/0 (socket says 172.21.15.77:56388) 2024-03-06T07:22:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f4139ef8640 1 -- 172.21.15.77:0/2853791721 learned_addr learned my addr 172.21.15.77:0/2853791721 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f41396f7640 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5dd86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413a6f9640 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f4139ef8640 1 -- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 msgr2=0x561bc5dd8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f4139ef8640 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f4139ef8640 1 -- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 msgr2=0x561bc5dd86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f4139ef8640 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5dd86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f4139ef8640 1 -- 172.21.15.77:0/2853791721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561bc5ccc270 con 0x561bc5dd5600 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.469+0000 7f413a6f9640 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f4139ef8640 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5dcfbd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f413002c570 tx=0x7f4130009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915d995e6a8c1178 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f4114ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f4114ff9640 1 -- 172.21.15.77:0/2853791721 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130097060 con 0x561bc5dd5600 2024-03-06T07:22:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f4114ff9640 1 -- 172.21.15.77:0/2853791721 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4130083260 con 0x561bc5dd5600 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 1 -- 172.21.15.77:0/2853791721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41240075b0 con 0x561bc5dd5600 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f4114ff9640 1 -- 172.21.15.77:0/2853791721 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4130092070 con 0x561bc5dd5600 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 10 client.4626 fetch_fsmap learned FSMap version 6 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 1 -- 172.21.15.77:0/2853791721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x561bc5a242d0 con 0x561bc5dd5600 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f4114ff9640 1 -- 172.21.15.77:0/2853791721 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41240075b0 con 0x561bc5dd5600 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f4114ff9640 1 -- 172.21.15.77:0/2853791721 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 6) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f413007a4e0 con 0x561bc5dd5600 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 10 client.4626 fetch_fsmap finished waiting for FSMap version 6 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 10 client.4626 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 20 client.4626 populate_metadata read hostname 'smithi077' 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] conn(0x561bc5e08f90 0x561bc5e13cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 4 client.4626 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f413ee13740 1 -- 172.21.15.77:0/2853791721 --> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561bc5d1cbf0 con 0x561bc5e08f90 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.470+0000 7f41396f7640 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] conn(0x561bc5e08f90 0x561bc5e13cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f4114ff9640 10 client.4626 ms_handle_connect on v2:172.21.15.158:6834/3488131752 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f41396f7640 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] conn(0x561bc5e08f90 0x561bc5e13cd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83064dba19cd7c8 server_cookie=18cb914c563280b3 in_seq=0 out_seq=0 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f4114ff9640 1 -- 172.21.15.77:0/2853791721 <== mds.0 v2:172.21.15.158:6834/3488131752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561bc5d1cbf0 con 0x561bc5e08f90 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f4114ff9640 10 client.4626 handle_command_reply: tid=0 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 client.4626 shutdown 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 -- 172.21.15.77:0/2853791721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 msgr2=0x561bc5dcfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5dcfbd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f413002c570 tx=0x7f4130009810 comp rx=0 tx=0).stop 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 -- 172.21.15.77:0/2853791721 shutdown_connections 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] conn(0x561bc5e08f90 0x561bc5e13cd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5dd86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f4114ff9640 0 client.4626 ms_handle_reset on v2:172.21.15.158:6834/3488131752 2024-03-06T07:22:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 --2- 172.21.15.77:0/2853791721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5dcfbd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.471+0000 7f413ee13740 1 -- 172.21.15.77:0/2853791721 >> 172.21.15.77:0/2853791721 conn(0x561bc5da5870 msgr2=0x561bc5dd02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.472+0000 7f413ee13740 1 -- 172.21.15.77:0/2853791721 shutdown_connections 2024-03-06T07:22:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.472+0000 7f413ee13740 1 -- 172.21.15.77:0/2853791721 wait complete. 2024-03-06T07:22:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.472+0000 7f413ee13740 20 client.4626 trim_cache size 0 max 16384 2024-03-06T07:22:15.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.488+0000 7f413ee13740 1 Processor -- start 2024-03-06T07:22:15.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.488+0000 7f413ee13740 1 -- start start 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413ee13740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413ee13740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5dd6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413ee13740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd7530 0x561bc5def880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413ee13740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41341fbd30 con 0x561bc5dd7530 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413ee13740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413413c0a0 con 0x561bc5dd7150 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413ee13740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41341f0b20 con 0x561bc5dd5600 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd7530 0x561bc5def880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd7530 0x561bc5def880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56398/0 (socket says 172.21.15.77:56398) 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 -- 172.21.15.77:0/3295283842 learned_addr learned my addr 172.21.15.77:0/3295283842 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f4139ef8640 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f41396f7640 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5dd6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 -- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 msgr2=0x561bc5dd5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 -- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 msgr2=0x561bc5dd6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5dd6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 -- 172.21.15.77:0/3295283842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bc5aee6b0 con 0x561bc5dd7530 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f4139ef8640 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.489+0000 7f413a6f9640 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd7530 0x561bc5def880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f412c009a30 tx=0x7f412c009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9185a48d73fed18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f4116ffd640 1 -- 172.21.15.77:0/3295283842 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c00f040 con 0x561bc5dd7530 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f4116ffd640 1 -- 172.21.15.77:0/3295283842 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c01b030 con 0x561bc5dd7530 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f4116ffd640 1 -- 172.21.15.77:0/3295283842 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c004030 con 0x561bc5dd7530 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 -- 172.21.15.77:0/3295283842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd7530 msgr2=0x561bc5def880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd7530 0x561bc5def880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f412c009a30 tx=0x7f412c009ef0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 -- 172.21.15.77:0/3295283842 shutdown_connections 2024-03-06T07:22:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd7530 0x561bc5def880 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5dd6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 --2- 172.21.15.77:0/3295283842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd5600 0x561bc5dd5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 -- 172.21.15.77:0/3295283842 >> 172.21.15.77:0/3295283842 conn(0x561bc5da5870 msgr2=0x561bc5da9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 -- 172.21.15.77:0/3295283842 shutdown_connections 2024-03-06T07:22:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.490+0000 7f413ee13740 1 -- 172.21.15.77:0/3295283842 wait complete. 2024-03-06T07:22:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.491+0000 7f413ee13740 1 -- start start 2024-03-06T07:22:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.491+0000 7f413ee13740 1 Processor -- start 2024-03-06T07:22:15.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5e67fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5def880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bc5deaf10 con 0x561bc5dd5600 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413413baa0 con 0x561bc5dd7150 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x561bc5def4a0 con 0x561bc5dd7530 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 10 client.0 _ll_register_callbacks cb 0x7f4130077420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5e67fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5e67fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56412/0 (socket says 172.21.15.77:56412) 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 -- 172.21.15.77:0/3537520450 learned_addr learned my addr 172.21.15.77:0/3537520450 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f41396f7640 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5def880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413a6f9640 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 -- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 msgr2=0x561bc5dd8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 -- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 msgr2=0x561bc5def880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5def880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 -- 172.21.15.77:0/3537520450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561bc5df4210 con 0x561bc5dd5600 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4139ef8640 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5e67fe0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4130039020 tx=0x7f41300aabc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889ba113a06be1ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4114ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4114ff9640 1 -- 172.21.15.77:0/3537520450 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41300aac30 con 0x561bc5dd5600 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4114ff9640 1 -- 172.21.15.77:0/3537520450 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4130098250 con 0x561bc5dd5600 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 1 -- 172.21.15.77:0/3537520450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f412c01b030 con 0x561bc5dd5600 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f4114ff9640 1 -- 172.21.15.77:0/3537520450 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4130097020 con 0x561bc5dd5600 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 10 client.4632 fetch_fsmap learned FSMap version 6 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.492+0000 7f413ee13740 1 -- 172.21.15.77:0/3537520450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x561bc5df4210 con 0x561bc5dd5600 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f4114ff9640 1 -- 172.21.15.77:0/3537520450 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c01b030 con 0x561bc5dd5600 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f4114ff9640 1 -- 172.21.15.77:0/3537520450 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 6) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f41300810c0 con 0x561bc5dd5600 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 10 client.4632 fetch_fsmap finished waiting for FSMap version 6 2024-03-06T07:22:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 10 client.4632 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 20 client.4632 populate_metadata read hostname 'smithi077' 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] conn(0x561bc5da4810 0x561bc5e17cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 4 client.4632 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 -- 172.21.15.77:0/3537520450 --> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561bc5da8050 con 0x561bc5da4810 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f41396f7640 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] conn(0x561bc5da4810 0x561bc5e17cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f4114ff9640 10 client.4632 ms_handle_connect on v2:172.21.15.158:6834/3488131752 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f41396f7640 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] conn(0x561bc5da4810 0x561bc5e17cf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b45f7d1f00ad8cd server_cookie=1af527c33e36e651 in_seq=0 out_seq=0 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f4114ff9640 1 -- 172.21.15.77:0/3537520450 <== mds.0 v2:172.21.15.158:6834/3488131752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x561bc5da8050 con 0x561bc5da4810 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f4114ff9640 10 client.4632 handle_command_reply: tid=0 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 client.4632 shutdown 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 -- 172.21.15.77:0/3537520450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 msgr2=0x561bc5e67fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5e67fe0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4130039020 tx=0x7f41300aabc0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 -- 172.21.15.77:0/3537520450 shutdown_connections 2024-03-06T07:22:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:6834/3488131752,v1:172.21.15.158:6835/3488131752] conn(0x561bc5da4810 0x561bc5e17cf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x561bc5dd7530 0x561bc5dd8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x561bc5dd7150 0x561bc5def880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 --2- 172.21.15.77:0/3537520450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561bc5dd5600 0x561bc5e67fe0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f4114ff9640 0 client.4632 ms_handle_reset on v2:172.21.15.158:6834/3488131752 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.493+0000 7f413ee13740 1 -- 172.21.15.77:0/3537520450 >> 172.21.15.77:0/3537520450 conn(0x561bc5da5870 msgr2=0x561bc5dd5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413ee13740 1 -- 172.21.15.77:0/3537520450 shutdown_connections 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413ee13740 1 -- 172.21.15.77:0/3537520450 wait complete. 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413ee13740 20 client.4632 trim_cache size 0 max 16384 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4282, 2024-03-06T07:22:15.503 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:22:15.504 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:22:15.504 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:22:15.504 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 16.554698385999998, 2024-03-06T07:22:15.504 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 6, 2024-03-06T07:22:15.504 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 21, 2024-03-06T07:22:15.504 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 21, 2024-03-06T07:22:15.504 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 23.572454572000002 2024-03-06T07:22:15.504 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:22:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 -- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f411806b4d0 msgr2=0x7f411806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f411806b4d0 0x7f411806d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f413415abc0 tx=0x7f4124009040 comp rx=0 tx=0).stop 2024-03-06T07:22:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 -- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4134070fe0 msgr2=0x7f413413e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4134070fe0 0x7f413413e580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4130000f80 tx=0x7f4130004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 -- 172.21.15.77:0/2321352954 shutdown_connections 2024-03-06T07:22:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f413415b2d0 0x7f413415a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41341444c0 0x7f413413d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f411806b4d0 0x7f411806d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 --2- 172.21.15.77:0/2321352954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4134070fe0 0x7f413413e580 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 -- 172.21.15.77:0/2321352954 >> 172.21.15.77:0/2321352954 conn(0x7f41340b1c80 msgr2=0x7f413415e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 -- 172.21.15.77:0/2321352954 shutdown_connections 2024-03-06T07:22:15.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.494+0000 7f413c15c640 1 -- 172.21.15.77:0/2321352954 wait complete. 2024-03-06T07:22:15.507 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-06T07:22:15.532 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-06T07:22:15.532 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:22:15.532 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-06T07:22:15.532 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:15.532 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:15.532 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-06T07:22:15.532 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:15.532 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:15.532 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-06T07:22:15.533 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:15.533 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:15.533 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-06T07:22:15.533 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:15.533 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:15.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.533 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2024-03-06T07:22:15.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.549 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link delete ceph-brx 2024-03-06T07:22:15.617 INFO:teuthology.orchestra.run.smithi158.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:15.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.619 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:22:15.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.634 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-03-06T07:22:15.703 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:15.705 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-06T07:22:15.705 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:15.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.844+0000 7f5dae9e9640 1 Processor -- start 2024-03-06T07:22:15.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.845+0000 7f5dae9e9640 1 -- start start 2024-03-06T07:22:15.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.845+0000 7f5dae9e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5da815bdd0 0x7f5da80c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.845+0000 7f5dae9e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5da815fa10 0x7f5da80ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.845+0000 7f5dae9e9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da80cebe0 con 0x7f5da80d16e0 2024-03-06T07:22:15.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.845+0000 7f5dae9e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da80ceec0 con 0x7f5da815bdd0 2024-03-06T07:22:15.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.845+0000 7f5dae9e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da80cf1a0 con 0x7f5da815fa10 2024-03-06T07:22:15.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5da815bdd0 0x7f5da80c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5da815bdd0 0x7f5da80c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56418/0 (socket says 172.21.15.77:56418) 2024-03-06T07:22:15.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da7fff640 1 -- 172.21.15.77:0/1931427333 learned_addr learned my addr 172.21.15.77:0/1931427333 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:15.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da77fe640 1 --2- 172.21.15.77:0/1931427333 >> v2:172.21.15.158:3300/0 conn(0x7f5da815fa10 0x7f5da80ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da7fff640 1 -- 172.21.15.77:0/1931427333 >> v2:172.21.15.158:3300/0 conn(0x7f5da815fa10 msgr2=0x7f5da80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da7fff640 1 --2- 172.21.15.77:0/1931427333 >> v2:172.21.15.158:3300/0 conn(0x7f5da815fa10 0x7f5da80ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da7fff640 1 -- 172.21.15.77:0/1931427333 >> v1:172.21.15.77:6789/0 conn(0x7f5da80d16e0 legacy=0x7f5da80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da7fff640 1 -- 172.21.15.77:0/1931427333 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da80cc8e0 con 0x7f5da815bdd0 2024-03-06T07:22:15.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.846+0000 7f5da6ffd640 1 -- 172.21.15.77:0/1931427333 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248698614 0 0) 0x7f5da80cebe0 con 0x7f5da80d16e0 2024-03-06T07:22:15.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5da7fff640 1 --2- 172.21.15.77:0/1931427333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80c5610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5da80cdfd0 tx=0x7f5d9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd974529b6225266 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5da6ffd640 1 -- 172.21.15.77:0/1931427333 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98035020 con 0x7f5da815bdd0 2024-03-06T07:22:15.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5da6ffd640 1 -- 172.21.15.77:0/1931427333 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d98031870 con 0x7f5da815bdd0 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5da6ffd640 1 -- 172.21.15.77:0/1931427333 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98031c40 con 0x7f5da815bdd0 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5dae9e9640 1 -- 172.21.15.77:0/1931427333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 msgr2=0x7f5da80c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/1931427333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80c5610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5da80cdfd0 tx=0x7f5d9802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5dae9e9640 1 -- 172.21.15.77:0/1931427333 shutdown_connections 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/1931427333 >> v2:172.21.15.158:3300/0 conn(0x7f5da815fa10 0x7f5da80ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/1931427333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80c5610 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.847+0000 7f5dae9e9640 1 -- 172.21.15.77:0/1931427333 >> 172.21.15.77:0/1931427333 conn(0x7f5da809a9f0 msgr2=0x7f5da809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.848+0000 7f5dae9e9640 1 -- 172.21.15.77:0/1931427333 shutdown_connections 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.848+0000 7f5dae9e9640 1 -- 172.21.15.77:0/1931427333 wait complete. 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.849+0000 7f5dae9e9640 1 Processor -- start 2024-03-06T07:22:15.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.849+0000 7f5dae9e9640 1 -- start start 2024-03-06T07:22:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.849+0000 7f5dae9e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5da80d16e0 0x7f5da814d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.849+0000 7f5dae9e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.849+0000 7f5dae9e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5da815fa10 0x7f5da80b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.849+0000 7f5dae9e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da815c1b0 con 0x7f5da815bdd0 2024-03-06T07:22:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5dae9e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da80cc5d0 con 0x7f5da815fa10 2024-03-06T07:22:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5dae9e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da80ce7b0 con 0x7f5da80d16e0 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56420/0 (socket says 172.21.15.77:56420) 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 -- 172.21.15.77:0/4109594157 learned_addr learned my addr 172.21.15.77:0/4109594157 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5dacf86640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5da815fa10 0x7f5da80b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da7fff640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5da80d16e0 0x7f5da814d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 -- 172.21.15.77:0/4109594157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5da80d16e0 msgr2=0x7f5da814d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5da80d16e0 0x7f5da814d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 -- 172.21.15.77:0/4109594157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5da815fa10 msgr2=0x7f5da80b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5da815fa10 0x7f5da80b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 -- 172.21.15.77:0/4109594157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da809ea80 con 0x7f5da815bdd0 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da77fe640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80b6b40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00e980 tx=0x7f5d9c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da4ff9640 1 -- 172.21.15.77:0/4109594157 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c017070 con 0x7f5da815bdd0 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da4ff9640 1 -- 172.21.15.77:0/4109594157 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c00ca00 con 0x7f5da815bdd0 2024-03-06T07:22:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da809ed60 con 0x7f5da815bdd0 2024-03-06T07:22:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5da4ff9640 1 -- 172.21.15.77:0/4109594157 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c00cdd0 con 0x7f5da815bdd0 2024-03-06T07:22:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.850+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da809f320 con 0x7f5da815bdd0 2024-03-06T07:22:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.851+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da804f330 con 0x7f5da815bdd0 2024-03-06T07:22:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.852+0000 7f5da4ff9640 1 -- 172.21.15.77:0/4109594157 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d9c01a070 con 0x7f5da815bdd0 2024-03-06T07:22:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.852+0000 7f5da4ff9640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b500 0x7f5d9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.852+0000 7f5da4ff9640 1 -- 172.21.15.77:0/4109594157 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d9c08a500 con 0x7f5da815bdd0 2024-03-06T07:22:15.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.852+0000 7f5da7fff640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b500 0x7f5d9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.852+0000 7f5da7fff640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b500 0x7f5d9406d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d98036000 tx=0x7f5d98008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:15.856+0000 7f5da4ff9640 1 -- 172.21.15.77:0/4109594157 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d9c058d30 con 0x7f5da815bdd0 2024-03-06T07:22:16.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.064+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5da80b7a40 con 0x7f5da815bdd0 2024-03-06T07:22:16.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.064+0000 7f5da4ff9640 1 -- 172.21.15.77:0/4109594157 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5d9c05cdd0 con 0x7f5da815bdd0 2024-03-06T07:22:16.067 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:16.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b500 msgr2=0x7f5d9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b500 0x7f5d9406d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d98036000 tx=0x7f5d98008030 comp rx=0 tx=0).stop 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 msgr2=0x7f5da80b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80b6b40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00e980 tx=0x7f5d9c00ee40 comp rx=0 tx=0).stop 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 shutdown_connections 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b500 0x7f5d9406d9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5da815bdd0 0x7f5da80b6b40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5da815fa10 0x7f5da80b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 --2- 172.21.15.77:0/4109594157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5da80d16e0 0x7f5da814d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.066+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 >> 172.21.15.77:0/4109594157 conn(0x7f5da809a9f0 msgr2=0x7f5da80d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.067+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 shutdown_connections 2024-03-06T07:22:16.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:16.067+0000 7f5dae9e9640 1 -- 172.21.15.77:0/4109594157 wait complete. 2024-03-06T07:22:16.079 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:16.079 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:16.079 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:22:16.079 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:22:16.079 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:16.080 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:16.080 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:16.080 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:16.080 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:16.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.080 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:22:16.095 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:16.095 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:16.095 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:16.095 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.095 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:16.095 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42661sec preferred_lft 42661sec 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:22:16.096 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:16.097 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:16.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.097 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:16.097 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:16.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:16.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:16.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2024-03-06T07:22:16.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:16.097 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:16.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.256 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:16.282 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:22:16.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.284 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:22:16.351 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:22:16.351 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:16.351 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:16.351 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:16.351 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:16.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.352 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:16.352 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:16.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:16.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:16.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:16.352 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:16.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.586 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:22:16.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.601 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-03-06T07:22:16.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.656 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:16.656 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:16.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:16.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:22:16.656 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:16.741 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:22:16.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.742 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:16.742 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:16.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:22:16.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:16.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:22:16.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:22:16.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:22:16.742 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:16.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:16.889 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:16.889 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:16.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:22:16.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:16.890 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:16.933 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:16.933 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:16.987 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:16.988 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:16.988 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:17.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:22:17.110 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:17.166 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:22:17.166 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:22:17.166 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:22:17.166 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:22:17.166 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:22:17.166 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:22:17.166 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:17.167 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.169 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:22:17.221 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:17.221 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:22:17.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:22:17.292 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:17.349 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:22:17.349 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:22:17.349 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:22:17.349 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:22:17.349 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:22:17.349 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:22:17.350 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:17.351 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:22:17.352 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.352 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.352 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:22:17.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:22:17.349+0000 7f590703d480 -1 init, newargv = 0x5641c332bea0 newargc=15 2024-03-06T07:22:17.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[46841]: starting ceph client 2024-03-06T07:22:17.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[46841]: starting fuse 2024-03-06T07:22:17.368 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:22:17.368 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:22:17.368 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:17.368 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:22:17.368 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:22:17.369 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:22:17.370 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:22:17.489 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.46841.asok 2024-03-06T07:22:17.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:17.500 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46841.asok 2024-03-06T07:22:17.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46841.asok status 2024-03-06T07:22:17.638 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:22:17.638 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "46841", 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4647, 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:22:17.639 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 4647 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3098329995 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:22:17.640 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3098329995 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.4647 192.168.0.1:0/3098329995", 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/3098329995", 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 6, 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 21, 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:22:17.641 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:22:17.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:17.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.746+0000 7f0850ec9640 1 Processor -- start 2024-03-06T07:22:17.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.746+0000 7f0850ec9640 1 -- start start 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f0850ec9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f084c0b4f70 0x7f084c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f0850ec9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c06c6d0 0x7f084c15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f0850ec9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c0b5890 con 0x7f084c066d90 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f0850ec9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c15e9f0 con 0x7f084c0b4f70 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f0850ec9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0cda10 con 0x7f084c06c6d0 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f084a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f084c0b4f70 0x7f084c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f084a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f084c0b4f70 0x7f084c0b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56428/0 (socket says 172.21.15.77:56428) 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f084a59c640 1 -- 172.21.15.77:0/400954675 learned_addr learned my addr 172.21.15.77:0/400954675 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f0843fff640 1 --2- 172.21.15.77:0/400954675 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c06c6d0 0x7f084c15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f084a59c640 1 -- 172.21.15.77:0/400954675 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c06c6d0 msgr2=0x7f084c15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f084a59c640 1 --2- 172.21.15.77:0/400954675 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c06c6d0 0x7f084c15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f084a59c640 1 -- 172.21.15.77:0/400954675 >> v1:172.21.15.158:6789/0 conn(0x7f084c066d90 legacy=0x7f084c0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f084a59c640 1 -- 172.21.15.77:0/400954675 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0cd450 con 0x7f084c0b4f70 2024-03-06T07:22:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.747+0000 7f0843fff640 1 --2- 172.21.15.77:0/400954675 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c06c6d0 0x7f084c15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.748+0000 7f084a59c640 1 --2- 172.21.15.77:0/400954675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c0b4f70 0x7f084c0b5350 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f083c009870 tx=0x7f083c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a21e8a94801ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.748+0000 7f0849d9b640 1 -- 172.21.15.77:0/400954675 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c034070 con 0x7f084c0b4f70 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.748+0000 7f0849d9b640 1 -- 172.21.15.77:0/400954675 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c030840 con 0x7f084c0b4f70 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.748+0000 7f0849d9b640 1 -- 172.21.15.77:0/400954675 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c030c10 con 0x7f084c0b4f70 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.749+0000 7f0850ec9640 1 -- 172.21.15.77:0/400954675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c0b4f70 msgr2=0x7f084c0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.749+0000 7f0850ec9640 1 --2- 172.21.15.77:0/400954675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c0b4f70 0x7f084c0b5350 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f083c009870 tx=0x7f083c02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.749+0000 7f0850ec9640 1 -- 172.21.15.77:0/400954675 shutdown_connections 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.749+0000 7f0850ec9640 1 --2- 172.21.15.77:0/400954675 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c06c6d0 0x7f084c15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.749+0000 7f0850ec9640 1 --2- 172.21.15.77:0/400954675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c0b4f70 0x7f084c0b5350 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.749+0000 7f0850ec9640 1 -- 172.21.15.77:0/400954675 >> 172.21.15.77:0/400954675 conn(0x7f084c0724d0 msgr2=0x7f084c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.749+0000 7f0850ec9640 1 -- 172.21.15.77:0/400954675 shutdown_connections 2024-03-06T07:22:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.749+0000 7f0850ec9640 1 -- 172.21.15.77:0/400954675 wait complete. 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.750+0000 7f0850ec9640 1 Processor -- start 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.750+0000 7f0850ec9640 1 -- start start 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0850ec9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c066d90 0x7f084c15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0850ec9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f084c0b4f70 0x7f084c0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0850ec9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c06e5e0 0x7f084c14b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0850ec9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15f3c0 con 0x7f084c06e5e0 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0850ec9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15e680 con 0x7f084c0b4f70 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0850ec9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c161ad0 con 0x7f084c066d90 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c06e5e0 0x7f084c14b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0843fff640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f084c0b4f70 0x7f084c0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c06e5e0 0x7f084c14b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56432/0 (socket says 172.21.15.77:56432) 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 -- 172.21.15.77:0/1884994987 learned_addr learned my addr 172.21.15.77:0/1884994987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084a59c640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c066d90 0x7f084c15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 -- 172.21.15.77:0/1884994987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c066d90 msgr2=0x7f084c15afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c066d90 0x7f084c15afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 -- 172.21.15.77:0/1884994987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f084c0b4f70 msgr2=0x7f084c0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f084c0b4f70 0x7f084c0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 -- 172.21.15.77:0/1884994987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c14bff0 con 0x7f084c06e5e0 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f084ad9d640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c06e5e0 0x7f084c14b420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f084400c730 tx=0x7f084400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0842ffd640 1 -- 172.21.15.77:0/1884994987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844003b70 con 0x7f084c06e5e0 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0842ffd640 1 -- 172.21.15.77:0/1884994987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0844003d10 con 0x7f084c06e5e0 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c14c260 con 0x7f084c06e5e0 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0842ffd640 1 -- 172.21.15.77:0/1884994987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084400aa90 con 0x7f084c06e5e0 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.751+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c0ce3d0 con 0x7f084c06e5e0 2024-03-06T07:22:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.752+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c04f330 con 0x7f084c06e5e0 2024-03-06T07:22:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.753+0000 7f0842ffd640 1 -- 172.21.15.77:0/1884994987 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f084400ac30 con 0x7f084c06e5e0 2024-03-06T07:22:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.753+0000 7f0842ffd640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f082c06b4d0 0x7f082c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.753+0000 7f0842ffd640 1 -- 172.21.15.77:0/1884994987 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0844089f90 con 0x7f084c06e5e0 2024-03-06T07:22:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.753+0000 7f084a59c640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f082c06b4d0 0x7f082c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.754+0000 7f084a59c640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f082c06b4d0 0x7f082c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f083c036000 tx=0x7f083c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.758+0000 7f0842ffd640 1 -- 172.21.15.77:0/1884994987 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08440587c0 con 0x7f084c06e5e0 2024-03-06T07:22:17.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.966+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f084c0b1860 con 0x7f084c06e5e0 2024-03-06T07:22:17.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.966+0000 7f0842ffd640 1 -- 172.21.15.77:0/1884994987 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f084405c860 con 0x7f084c06e5e0 2024-03-06T07:22:17.968 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f082c06b4d0 msgr2=0x7f082c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f082c06b4d0 0x7f082c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f083c036000 tx=0x7f083c03b040 comp rx=0 tx=0).stop 2024-03-06T07:22:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c06e5e0 msgr2=0x7f084c14b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c06e5e0 0x7f084c14b420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f084400c730 tx=0x7f084400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 shutdown_connections 2024-03-06T07:22:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f082c06b4d0 0x7f082c06d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c06e5e0 0x7f084c14b420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f084c0b4f70 0x7f084c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 --2- 172.21.15.77:0/1884994987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f084c066d90 0x7f084c15afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 >> 172.21.15.77:0/1884994987 conn(0x7f084c0724d0 msgr2=0x7f084c0ccbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 shutdown_connections 2024-03-06T07:22:17.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:17.968+0000 7f0850ec9640 1 -- 172.21.15.77:0/1884994987 wait complete. 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:17.981 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:17.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.982 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42659sec preferred_lft 42659sec 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link tentative 2024-03-06T07:22:17.998 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.999 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:17.999 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:17.999 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link tentative 2024-03-06T07:22:17.999 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:17.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:17.999 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:18.062 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:22:18.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.064 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:22:18.119 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:22:18.119 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:18.119 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:18.119 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:18.119 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:18.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.120 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:18.120 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:18.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:18.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:18.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:18.120 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:18.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.215 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:22:18.230 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:22:18.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.230 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-03-06T07:22:18.284 INFO:teuthology.orchestra.run.smithi077.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:22:18.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.285 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:18.285 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:18.285 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:18.285 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:22:18.285 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:18.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.368 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether c2:a5:e3:dd:f8:7b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::c0a5:e3ff:fedd:f87b/64 scope link 2024-03-06T07:22:18.435 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.437 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:18.504 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:18.504 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:18.504 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.504 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:18.504 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:18.505 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether c2:a5:e3:dd:f8:7b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:18.505 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:18.505 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.505 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::c0a5:e3ff:fedd:f87b/64 scope link 2024-03-06T07:22:18.505 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:18.506 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.506 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:18.506 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:18.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:22:18.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:18.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:22:18.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:22:18.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:22:18.507 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:18.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.633 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:22:18.633 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:22:18.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.1 up 2024-03-06T07:22:18.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:18.633 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:22:18.676 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-06T07:22:18.676 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:18.676 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:18.731 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:18.732 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:18.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:22:18.854 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:18.909 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:22:18.910 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:22:18.911 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:18.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:18.912 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:22:18.964 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:22:18.965 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-03-06T07:22:18.965 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-06T07:22:19.007 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:22:19.033 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.093 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:22:19.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi077.stderr:2024-03-06T07:22:19.090+0000 7fcbc1a44480 -1 init, newargv = 0x55a9b21985c0 newargc=15 2024-03-06T07:22:19.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi077.stderr:ceph-fuse[47336]: starting ceph client 2024-03-06T07:22:19.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi077.stderr:ceph-fuse[47336]: starting fuse 2024-03-06T07:22:19.108 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:22:19.108 INFO:teuthology.orchestra.run.smithi077.stdout:44 2024-03-06T07:22:19.109 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:22:19.109 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.2")) 2024-03-06T07:22:19.110 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:22:19.228 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.2.47336.asok 2024-03-06T07:22:19.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:19.238 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.47336.asok 2024-03-06T07:22:19.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.47336.asok status 2024-03-06T07:22:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:22:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:22:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "2", 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "47336", 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4662, 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:22:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 4662 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3115049788 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3115049788 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.4662 192.168.0.2:0/3115049788", 2024-03-06T07:22:19.375 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.2:0/3115049788", 2024-03-06T07:22:19.376 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:22:19.376 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 6, 2024-03-06T07:22:19.376 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 21, 2024-03-06T07:22:19.376 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:19.376 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:22:19.376 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:22:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:22:19.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.479+0000 7f3ebc0b5640 1 Processor -- start 2024-03-06T07:22:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.479+0000 7f3ebc0b5640 1 -- start start 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3ebc0b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb40b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3ebc0b5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3eb406c6d0 0x7f3eb415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3ebc0b5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3eb4066d90 0x7f3eb40ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3ebc0b5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb415e9f0 con 0x7f3eb4066d90 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3ebc0b5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb40cda10 con 0x7f3eb406c6d0 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3ebc0b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40b5890 con 0x7f3eb40b4f70 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eba652640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3eb4066d90 0x7f3eb40ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eb9650640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3eb406c6d0 0x7f3eb415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eb9e51640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb40b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eba652640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3eb4066d90 0x7f3eb40ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46278/0 (socket says 172.21.15.77:46278) 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eba652640 1 -- 172.21.15.77:0/4177856163 learned_addr learned my addr 172.21.15.77:0/4177856163 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eb9650640 1 -- 172.21.15.77:0/4177856163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 msgr2=0x7f3eb40b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eb9650640 1 --2- 172.21.15.77:0/4177856163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb40b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eb9650640 1 -- 172.21.15.77:0/4177856163 >> v2:172.21.15.77:3300/0 conn(0x7f3eb4066d90 msgr2=0x7f3eb40ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eb9650640 1 --2- 172.21.15.77:0/4177856163 >> v2:172.21.15.77:3300/0 conn(0x7f3eb4066d90 0x7f3eb40ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eb9650640 1 -- 172.21.15.77:0/4177856163 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40cd450 con 0x7f3eb406c6d0 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.480+0000 7f3eb9e51640 1 --2- 172.21.15.77:0/4177856163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb40b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.481+0000 7f3eb9650640 1 --2- 172.21.15.77:0/4177856163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb406c6d0 0x7f3eb415ee80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4068a90 tx=0x7f3ea402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c14543e7cfc86e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.481+0000 7f3eb8e4f640 1 -- 172.21.15.77:0/4177856163 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40315a0 con 0x7f3eb406c6d0 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.481+0000 7f3eb8e4f640 1 -- 172.21.15.77:0/4177856163 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4037070 con 0x7f3eb406c6d0 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.481+0000 7f3eb8e4f640 1 -- 172.21.15.77:0/4177856163 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4031960 con 0x7f3eb406c6d0 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.481+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/4177856163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb406c6d0 msgr2=0x7f3eb415ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.481+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/4177856163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb406c6d0 0x7f3eb415ee80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4068a90 tx=0x7f3ea402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.482+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/4177856163 shutdown_connections 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.482+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/4177856163 >> v2:172.21.15.77:3300/0 conn(0x7f3eb4066d90 0x7f3eb40ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.482+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/4177856163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb406c6d0 0x7f3eb415ee80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.482+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/4177856163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb40b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.482+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/4177856163 >> 172.21.15.77:0/4177856163 conn(0x7f3eb40724d0 msgr2=0x7f3eb40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.482+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/4177856163 shutdown_connections 2024-03-06T07:22:19.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.482+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/4177856163 wait complete. 2024-03-06T07:22:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.483+0000 7f3ebc0b5640 1 Processor -- start 2024-03-06T07:22:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.483+0000 7f3ebc0b5640 1 -- start start 2024-03-06T07:22:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.483+0000 7f3ebc0b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4066d90 0x7f3eb41553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.483+0000 7f3ebc0b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb415f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.483+0000 7f3ebc0b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb40aecb0 0x7f3eb415a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.483+0000 7f3ebc0b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb415f3c0 con 0x7f3eb4066d90 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.483+0000 7f3ebc0b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb415e680 con 0x7f3eb40aecb0 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.483+0000 7f3ebc0b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40cd090 con 0x7f3eb40b4f70 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4066d90 0x7f3eb41553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4066d90 0x7f3eb41553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46292/0 (socket says 172.21.15.77:46292) 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 -- 172.21.15.77:0/3852672478 learned_addr learned my addr 172.21.15.77:0/3852672478 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eba652640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb40aecb0 0x7f3eb415a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9650640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb415f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 -- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 msgr2=0x7f3eb415f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb415f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 -- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb40aecb0 msgr2=0x7f3eb415a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb40aecb0 0x7f3eb415a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 -- 172.21.15.77:0/3852672478 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb415ff40 con 0x7f3eb4066d90 2024-03-06T07:22:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eba652640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb40aecb0 0x7f3eb415a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eb9e51640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4066d90 0x7f3eb41553e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000ebe0 tx=0x7f3eb000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eaa7fc640 1 -- 172.21.15.77:0/3852672478 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0017070 con 0x7f3eb4066d90 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eaa7fc640 1 -- 172.21.15.77:0/3852672478 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb000cc30 con 0x7f3eb4066d90 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb41599a0 con 0x7f3eb4066d90 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3eaa7fc640 1 -- 172.21.15.77:0/3852672478 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0013470 con 0x7f3eb4066d90 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.484+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb4160220 con 0x7f3eb4066d90 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.485+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb404f330 con 0x7f3eb4066d90 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.486+0000 7f3eaa7fc640 1 -- 172.21.15.77:0/3852672478 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3eb001a070 con 0x7f3eb4066d90 2024-03-06T07:22:19.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.486+0000 7f3eaa7fc640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9806b4d0 0x7f3e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.486+0000 7f3eb9650640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9806b4d0 0x7f3e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.486+0000 7f3eaa7fc640 1 -- 172.21.15.77:0/3852672478 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3eb008a460 con 0x7f3eb4066d90 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.486+0000 7f3eb9650640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9806b4d0 0x7f3e9806d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4036000 tx=0x7f3ea403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.490+0000 7f3eaa7fc640 1 -- 172.21.15.77:0/3852672478 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3eb0058c90 con 0x7f3eb4066d90 2024-03-06T07:22:19.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.700+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3eb4159b70 con 0x7f3eb4066d90 2024-03-06T07:22:19.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.700+0000 7f3eaa7fc640 1 -- 172.21.15.77:0/3852672478 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3eb005cd30 con 0x7f3eb4066d90 2024-03-06T07:22:19.702 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9806b4d0 msgr2=0x7f3e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9806b4d0 0x7f3e9806d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4036000 tx=0x7f3ea403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4066d90 msgr2=0x7f3eb41553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4066d90 0x7f3eb41553e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000ebe0 tx=0x7f3eb000c520 comp rx=0 tx=0).stop 2024-03-06T07:22:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 shutdown_connections 2024-03-06T07:22:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb40aecb0 0x7f3eb415a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb40b4f70 0x7f3eb415f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9806b4d0 0x7f3e9806d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 --2- 172.21.15.77:0/3852672478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb4066d90 0x7f3eb41553e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 >> 172.21.15.77:0/3852672478 conn(0x7f3eb40724d0 msgr2=0x7f3eb406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 shutdown_connections 2024-03-06T07:22:19.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:19.702+0000 7f3ebc0b5640 1 -- 172.21.15.77:0/3852672478 wait complete. 2024-03-06T07:22:19.714 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:19.715 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:19.715 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2024-03-06T07:22:19.715 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:22:19.715 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:19.715 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:19.715 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:19.715 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:19.715 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:19.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.715 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: altname enp7s0f0 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi158.stdout: altname enp7s0f1 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 42692sec preferred_lft 42692sec 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::ec4:7aff:fe88:726f/64 scope link 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:19.733 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.733 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:19.734 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:19.734 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:19.734 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:19.734 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set ceph-brx up 2024-03-06T07:22:19.734 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:19.734 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:19.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.917 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:19.944 INFO:teuthology.orchestra.run.smithi158.stdout:1 2024-03-06T07:22:19.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:19.946 DEBUG:teuthology.orchestra.run.smithi158:> route 2024-03-06T07:22:20.013 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2024-03-06T07:22:20.014 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:20.014 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:20.014 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:20.014 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:20.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.014 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:20.014 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:20.014 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:20.014 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:20.014 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:20.015 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:20.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.301 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2024-03-06T07:22:20.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.316 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list-id 2024-03-06T07:22:20.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.371 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:20.371 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:20.371 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:20.371 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:22:20.371 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:20.453 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:22:20.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.453 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:20.453 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:20.453 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:22:20.453 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:20.453 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:22:20.453 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:22:20.453 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:22:20.453 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:20.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.608 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:20.608 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:20.608 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.0 up 2024-03-06T07:22:20.608 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:20.608 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:20.650 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-06T07:22:20.650 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:20.650 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:20.705 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:20.705 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:20.705 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:20.759 DEBUG:teuthology.orchestra.run.smithi158:> sudo modprobe fuse 2024-03-06T07:22:20.826 DEBUG:teuthology.orchestra.run.smithi158:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:20.882 INFO:teuthology.orchestra.run.smithi158.stdout:/proc 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/sys 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/dev 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/security 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/shm 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/pts 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/run 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/pstore 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/bpf 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/ 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/selinux 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/debug 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/mqueue 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/hugepages 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/tracing 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/config 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/run/user/1000 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:20.885 DEBUG:teuthology.orchestra.run.smithi158:> ls /sys/fs/fuse/connections 2024-03-06T07:22:20.937 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:20.937 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-06T07:22:20.980 DEBUG:teuthology.orchestra.run.smithi158:> sudo modprobe fuse 2024-03-06T07:22:21.010 DEBUG:teuthology.orchestra.run.smithi158:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/proc 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/sys 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/dev 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/security 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/shm 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/pts 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/run 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/pstore 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/bpf 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/ 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/selinux 2024-03-06T07:22:21.067 INFO:teuthology.orchestra.run.smithi158.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/debug 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/mqueue 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/hugepages 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/tracing 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/config 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/run/user/1000 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:21.068 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:21.069 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:21.069 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns 2024-03-06T07:22:21.069 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:21.069 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:21.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.069 DEBUG:teuthology.orchestra.run.smithi158:> ls /sys/fs/fuse/connections 2024-03-06T07:22:21.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi158.stderr:2024-03-06T07:22:21.067+0000 7fa05d435480 -1 init, newargv = 0x55ffc3ba8190 newargc=15 2024-03-06T07:22:21.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi158.stderr:ceph-fuse[43525]: starting ceph client 2024-03-06T07:22:21.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi158.stderr:ceph-fuse[43525]: starting fuse 2024-03-06T07:22:21.086 INFO:teuthology.orchestra.run.smithi158.stdout:41 2024-03-06T07:22:21.086 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:22:21.086 DEBUG:teuthology.orchestra.run.smithi158:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:21.086 DEBUG:teuthology.orchestra.run.smithi158:> import glob 2024-03-06T07:22:21.086 DEBUG:teuthology.orchestra.run.smithi158:> import re 2024-03-06T07:22:21.086 DEBUG:teuthology.orchestra.run.smithi158:> import os 2024-03-06T07:22:21.086 DEBUG:teuthology.orchestra.run.smithi158:> import subprocess 2024-03-06T07:22:21.086 DEBUG:teuthology.orchestra.run.smithi158:> 2024-03-06T07:22:21.086 DEBUG:teuthology.orchestra.run.smithi158:> def _find_admin_socket(client_name): 2024-03-06T07:22:21.086 DEBUG:teuthology.orchestra.run.smithi158:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> files = glob.glob(asok_path) 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> # Given a non-glob path, it better be there 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> if "*" not in asok_path: 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> assert(len(files) == 1) 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> return files[0] 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> for f in files: 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> contents = proc_f.read() 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> if mountpoint in contents: 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> return f 2024-03-06T07:22:21.087 DEBUG:teuthology.orchestra.run.smithi158:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi158:> 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi158:> print(_find_admin_socket("client.1")) 2024-03-06T07:22:21.088 DEBUG:teuthology.orchestra.run.smithi158:> ' 2024-03-06T07:22:21.212 INFO:teuthology.orchestra.run.smithi158.stdout:/var/run/ceph/ceph-client.1.43525.asok 2024-03-06T07:22:21.216 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:22:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:21.223 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43525.asok 2024-03-06T07:22:21.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.224 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43525.asok status 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "metadata": { 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "entity_id": "1", 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "hostname": "smithi158", 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "pid": "43525", 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "root": "/" 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2024-03-06T07:22:21.361 INFO:teuthology.orchestra.run.smithi158.stdout: "dentry_count": 0, 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 4696, 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "inst": { 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "name": { 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "client", 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "num": 4696 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": { 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "v1", 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "nonce": 2660458941 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: } 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": { 2024-03-06T07:22:21.362 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "v1", 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "nonce": 2660458941 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "inst_str": "client.4696 192.168.0.1:0/2660458941", 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "addr_str": "192.168.0.1:0/2660458941", 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "inode_count": 1, 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "mds_epoch": 6, 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "osd_epoch": 21, 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "blocklisted": false, 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs" 2024-03-06T07:22:21.363 INFO:teuthology.orchestra.run.smithi158.stdout:} 2024-03-06T07:22:21.369 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.479+0000 7fab3b42a640 1 Processor -- start 2024-03-06T07:22:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.479+0000 7fab3b42a640 1 -- start start 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab3b42a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab340d00c0 0x7fab340d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab3b42a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 0x7fab340cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab3b42a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab340d09e0 con 0x7fab340ce1b0 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab3b42a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3406ca90 con 0x7fab340d00c0 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab3b42a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3406df30 con 0x7fab340cf850 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab391c6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab340d00c0 0x7fab340d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab391c6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab340d00c0 0x7fab340d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46300/0 (socket says 172.21.15.77:46300) 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab391c6640 1 -- 172.21.15.77:0/1622954089 learned_addr learned my addr 172.21.15.77:0/1622954089 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.480+0000 7fab389c5640 1 --2- 172.21.15.77:0/1622954089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 0x7fab340cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.481+0000 7fab391c6640 1 -- 172.21.15.77:0/1622954089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 msgr2=0x7fab340cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.481+0000 7fab391c6640 1 --2- 172.21.15.77:0/1622954089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 0x7fab340cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.481+0000 7fab391c6640 1 -- 172.21.15.77:0/1622954089 >> v1:172.21.15.158:6789/0 conn(0x7fab340ce1b0 legacy=0x7fab340ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.481+0000 7fab391c6640 1 -- 172.21.15.77:0/1622954089 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3415e380 con 0x7fab340d00c0 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.481+0000 7fab391c6640 1 --2- 172.21.15.77:0/1622954089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab340d04a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab30009870 tx=0x7fab3002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465c31bb841aa178 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.481+0000 7fab2bfff640 1 -- 172.21.15.77:0/1622954089 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30034070 con 0x7fab340d00c0 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab2bfff640 1 -- 172.21.15.77:0/1622954089 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab30030840 con 0x7fab340d00c0 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab2bfff640 1 -- 172.21.15.77:0/1622954089 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30030c10 con 0x7fab340d00c0 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab3b42a640 1 -- 172.21.15.77:0/1622954089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 msgr2=0x7fab340d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab3b42a640 1 --2- 172.21.15.77:0/1622954089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab340d04a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab30009870 tx=0x7fab3002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab3b42a640 1 -- 172.21.15.77:0/1622954089 shutdown_connections 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab3b42a640 1 --2- 172.21.15.77:0/1622954089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 0x7fab340cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab3b42a640 1 --2- 172.21.15.77:0/1622954089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab340d04a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab3b42a640 1 -- 172.21.15.77:0/1622954089 >> 172.21.15.77:0/1622954089 conn(0x7fab340b35a0 msgr2=0x7fab340b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab3b42a640 1 -- 172.21.15.77:0/1622954089 shutdown_connections 2024-03-06T07:22:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.482+0000 7fab3b42a640 1 -- 172.21.15.77:0/1622954089 wait complete. 2024-03-06T07:22:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.483+0000 7fab3b42a640 1 Processor -- start 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.483+0000 7fab3b42a640 1 -- start start 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab3b42a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab340ce1b0 0x7fab340c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab3b42a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 0x7fab340b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab3b42a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab3414f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab3b42a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3406c440 con 0x7fab340d00c0 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab3b42a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3406c160 con 0x7fab340ce1b0 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab3b42a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3406c720 con 0x7fab340cf850 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab399c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab3414f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab399c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab3414f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46306/0 (socket says 172.21.15.77:46306) 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab399c7640 1 -- 172.21.15.77:0/80217918 learned_addr learned my addr 172.21.15.77:0/80217918 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab391c6640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab340ce1b0 0x7fab340c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.484+0000 7fab389c5640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 0x7fab340b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab399c7640 1 -- 172.21.15.77:0/80217918 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 msgr2=0x7fab340b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab399c7640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 0x7fab340b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab399c7640 1 -- 172.21.15.77:0/80217918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab340ce1b0 msgr2=0x7fab340c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab399c7640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab340ce1b0 0x7fab340c7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab399c7640 1 -- 172.21.15.77:0/80217918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab340ca600 con 0x7fab340d00c0 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab399c7640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab3414f930 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00c710 tx=0x7fab2c00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab29ffb640 1 -- 172.21.15.77:0/80217918 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c016020 con 0x7fab340d00c0 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab29ffb640 1 -- 172.21.15.77:0/80217918 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c006d50 con 0x7fab340d00c0 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab340ca8e0 con 0x7fab340d00c0 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab29ffb640 1 -- 172.21.15.77:0/80217918 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c003bb0 con 0x7fab340d00c0 2024-03-06T07:22:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.485+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab340caea0 con 0x7fab340d00c0 2024-03-06T07:22:21.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.486+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab34068150 con 0x7fab340d00c0 2024-03-06T07:22:21.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.486+0000 7fab29ffb640 1 -- 172.21.15.77:0/80217918 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab2c00a850 con 0x7fab340d00c0 2024-03-06T07:22:21.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.487+0000 7fab29ffb640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab1806b500 0x7fab1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.487+0000 7fab29ffb640 1 -- 172.21.15.77:0/80217918 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab2c089c60 con 0x7fab340d00c0 2024-03-06T07:22:21.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.487+0000 7fab391c6640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab1806b500 0x7fab1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.487+0000 7fab391c6640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab1806b500 0x7fab1806d9c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab30036000 tx=0x7fab3003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.491+0000 7fab29ffb640 1 -- 172.21.15.77:0/80217918 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab34068150 con 0x7fab340d00c0 2024-03-06T07:22:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.701+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fab340d17a0 con 0x7fab340d00c0 2024-03-06T07:22:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.702+0000 7fab29ffb640 1 -- 172.21.15.77:0/80217918 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7fab2c020020 con 0x7fab340d00c0 2024-03-06T07:22:21.704 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:21.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab1806b500 msgr2=0x7fab1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab1806b500 0x7fab1806d9c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab30036000 tx=0x7fab3003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 msgr2=0x7fab3414f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab3414f930 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00c710 tx=0x7fab2c00cbd0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 shutdown_connections 2024-03-06T07:22:21.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab1806b500 0x7fab1806d9c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab340d00c0 0x7fab3414f930 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab340cf850 0x7fab340b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 --2- 172.21.15.77:0/80217918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab340ce1b0 0x7fab340c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 >> 172.21.15.77:0/80217918 conn(0x7fab340b35a0 msgr2=0x7fab3406fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 shutdown_connections 2024-03-06T07:22:21.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:21.704+0000 7fab3b42a640 1 -- 172.21.15.77:0/80217918 wait complete. 2024-03-06T07:22:21.716 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:21.716 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:21.716 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi158.front.sepia.ceph.com 2024-03-06T07:22:21.716 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:22:21.716 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:21.716 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:21.717 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:21.717 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:21.717 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:21.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.717 DEBUG:teuthology.orchestra.run.smithi158:> ip addr 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: altname enp7s0f0 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:da:6b:d5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: altname enp7s0f1 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 0c:c4:7a:88:72:6f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: inet 172.21.15.158/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft 42690sec preferred_lft 42690sec 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::ec4:7aff:fe88:726f/64 scope link 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether f2:1e:fe:65:4d:95 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::5862:6cff:fe60:b57d/64 scope link 2024-03-06T07:22:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:21.735 INFO:teuthology.orchestra.run.smithi158.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:21.735 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether f2:1e:fe:65:4d:95 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:21.735 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::f01e:feff:fe65:4d95/64 scope link tentative 2024-03-06T07:22:21.735 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:21.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.735 DEBUG:teuthology.orchestra.run.smithi158:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:21.798 INFO:teuthology.orchestra.run.smithi158.stdout:1 2024-03-06T07:22:21.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.800 DEBUG:teuthology.orchestra.run.smithi158:> route 2024-03-06T07:22:21.856 INFO:teuthology.orchestra.run.smithi158.stdout:Kernel IP routing table 2024-03-06T07:22:21.856 INFO:teuthology.orchestra.run.smithi158.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:21.856 INFO:teuthology.orchestra.run.smithi158.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:21.856 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:21.856 INFO:teuthology.orchestra.run.smithi158.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:21.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.857 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:21.857 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:21.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:21.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:21.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:21.857 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:21.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.953 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2024-03-06T07:22:21.968 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:22:21.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:21.969 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list-id 2024-03-06T07:22:22.024 INFO:teuthology.orchestra.run.smithi158.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:22:22.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.025 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:22.025 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:22.025 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:22.025 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:22:22.025 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:22.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.108 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi158.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:22.175 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 2e:eb:a0:98:ea:a6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::2ceb:a0ff:fe98:eaa6/64 scope link 2024-03-06T07:22:22.176 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.177 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:22.244 INFO:teuthology.orchestra.run.smithi158.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: link/ether 2e:eb:a0:98:ea:a6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: inet6 fe80::2ceb:a0ff:fe98:eaa6/64 scope link 2024-03-06T07:22:22.245 INFO:teuthology.orchestra.run.smithi158.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:22.246 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:22:22.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.247 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:22.247 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:22.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:22:22.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:22.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:22:22.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:22:22.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:22:22.247 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:22.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.374 DEBUG:teuthology.orchestra.run.smithi158:> (cd / && exec bash -c ' 2024-03-06T07:22:22.374 DEBUG:teuthology.orchestra.run.smithi158:> set -e 2024-03-06T07:22:22.374 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set brx.1 up 2024-03-06T07:22:22.374 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:22.374 DEBUG:teuthology.orchestra.run.smithi158:> ') 2024-03-06T07:22:22.419 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-06T07:22:22.419 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:22.419 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run.smithi158.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:22.474 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:22.474 DEBUG:teuthology.orchestra.run.smithi158:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:22.528 DEBUG:teuthology.orchestra.run.smithi158:> sudo modprobe fuse 2024-03-06T07:22:22.595 DEBUG:teuthology.orchestra.run.smithi158:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:22.651 INFO:teuthology.orchestra.run.smithi158.stdout:/proc 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/sys 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/dev 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/security 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/shm 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/pts 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/run 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/pstore 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/bpf 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/ 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/selinux 2024-03-06T07:22:22.652 INFO:teuthology.orchestra.run.smithi158.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/debug 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/mqueue 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/hugepages 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/tracing 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/config 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/run/user/1000 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:22.653 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:22.654 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:22.654 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns 2024-03-06T07:22:22.654 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:22.654 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:22.654 INFO:teuthology.orchestra.run.smithi158.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:22.654 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:22.654 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:22.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.654 DEBUG:teuthology.orchestra.run.smithi158:> ls /sys/fs/fuse/connections 2024-03-06T07:22:22.706 INFO:teuthology.orchestra.run.smithi158.stdout:41 2024-03-06T07:22:22.706 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-06T07:22:22.706 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-06T07:22:22.749 DEBUG:teuthology.orchestra.run.smithi158:> sudo modprobe fuse 2024-03-06T07:22:22.776 DEBUG:teuthology.orchestra.run.smithi158:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:22.831 INFO:teuthology.orchestra.run.smithi158.stdout:/proc 2024-03-06T07:22:22.831 INFO:teuthology.orchestra.run.smithi158.stdout:/sys 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/dev 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/security 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/shm 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/pts 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/run 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/cgroup 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/pstore 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/bpf 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/ 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/selinux 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/debug 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/mqueue 2024-03-06T07:22:22.832 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/hugepages 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/tracing 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/sys/kernel/config 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/run/user/1000 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:22.833 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns 2024-03-06T07:22:22.834 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:22.834 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:22.834 INFO:teuthology.orchestra.run.smithi158.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:22.834 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:22.834 INFO:teuthology.orchestra.run.smithi158.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:22.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.835 DEBUG:teuthology.orchestra.run.smithi158:> ls /sys/fs/fuse/connections 2024-03-06T07:22:22.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi158.stderr:2024-03-06T07:22:22.833+0000 7f2314839480 -1 init, newargv = 0x55758112e860 newargc=15 2024-03-06T07:22:22.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi158.stderr:ceph-fuse[43977]: starting ceph client 2024-03-06T07:22:22.840 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi158.stderr:ceph-fuse[43977]: starting fuse 2024-03-06T07:22:22.854 INFO:teuthology.orchestra.run.smithi158.stdout:41 2024-03-06T07:22:22.854 INFO:teuthology.orchestra.run.smithi158.stdout:44 2024-03-06T07:22:22.854 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-06T07:22:22.854 DEBUG:teuthology.orchestra.run.smithi158:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:22.854 DEBUG:teuthology.orchestra.run.smithi158:> import glob 2024-03-06T07:22:22.854 DEBUG:teuthology.orchestra.run.smithi158:> import re 2024-03-06T07:22:22.854 DEBUG:teuthology.orchestra.run.smithi158:> import os 2024-03-06T07:22:22.854 DEBUG:teuthology.orchestra.run.smithi158:> import subprocess 2024-03-06T07:22:22.854 DEBUG:teuthology.orchestra.run.smithi158:> 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> def _find_admin_socket(client_name): 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> files = glob.glob(asok_path) 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> # Given a non-glob path, it better be there 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> if "*" not in asok_path: 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> assert(len(files) == 1) 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> return files[0] 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> for f in files: 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> contents = proc_f.read() 2024-03-06T07:22:22.855 DEBUG:teuthology.orchestra.run.smithi158:> if mountpoint in contents: 2024-03-06T07:22:22.856 DEBUG:teuthology.orchestra.run.smithi158:> return f 2024-03-06T07:22:22.856 DEBUG:teuthology.orchestra.run.smithi158:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:22.856 DEBUG:teuthology.orchestra.run.smithi158:> 2024-03-06T07:22:22.856 DEBUG:teuthology.orchestra.run.smithi158:> print(_find_admin_socket("client.3")) 2024-03-06T07:22:22.856 DEBUG:teuthology.orchestra.run.smithi158:> ' 2024-03-06T07:22:22.977 INFO:teuthology.orchestra.run.smithi158.stdout:/var/run/ceph/ceph-client.3.43977.asok 2024-03-06T07:22:22.980 INFO:teuthology.orchestra.run.smithi158.stderr:2024-03-06T07:22:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:22.987 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.43977.asok 2024-03-06T07:22:22.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.988 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.43977.asok status 2024-03-06T07:22:23.124 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2024-03-06T07:22:23.124 INFO:teuthology.orchestra.run.smithi158.stdout: "metadata": { 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "entity_id": "3", 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "hostname": "smithi158", 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "pid": "43977", 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "root": "/" 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "dentry_count": 0, 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:23.125 INFO:teuthology.orchestra.run.smithi158.stdout: "id": 4705, 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "inst": { 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "name": { 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "client", 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "num": 4705 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": { 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "v1", 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "nonce": 4207319311 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: } 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2024-03-06T07:22:23.126 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": { 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "v1", 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "nonce": 4207319311 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "inst_str": "client.4705 192.168.0.2:0/4207319311", 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "addr_str": "192.168.0.2:0/4207319311", 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "inode_count": 1, 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "mds_epoch": 6, 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "osd_epoch": 21, 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "blocklisted": false, 2024-03-06T07:22:23.127 INFO:teuthology.orchestra.run.smithi158.stdout: "fs_name": "cephfs" 2024-03-06T07:22:23.128 INFO:teuthology.orchestra.run.smithi158.stdout:} 2024-03-06T07:22:23.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.133 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:22:23.133 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:23.152 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:22:23.152 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:23.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.153 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:23.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.226 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:22:23.226 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:23.284 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:22:23.285 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:23.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.285 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:23.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.357 DEBUG:teuthology.orchestra.run.smithi158:> stat --file-system '--printf=%T 2024-03-06T07:22:23.358 DEBUG:teuthology.orchestra.run.smithi158:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:23.373 INFO:teuthology.orchestra.run.smithi158.stdout:fuseblk 2024-03-06T07:22:23.373 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:23.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.373 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:23.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.440 DEBUG:teuthology.orchestra.run.smithi158:> stat --file-system '--printf=%T 2024-03-06T07:22:23.440 DEBUG:teuthology.orchestra.run.smithi158:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:23.495 INFO:teuthology.orchestra.run.smithi158.stdout:fuseblk 2024-03-06T07:22:23.495 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:23.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:23.496 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:23.562 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:22:23.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:23.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.685+0000 7f6be3847640 1 Processor -- start 2024-03-06T07:22:23.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.685+0000 7f6be3847640 1 -- start start 2024-03-06T07:22:23.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be3847640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6bdc0b4f70 0x7f6bdc0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be3847640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc06c6d0 0x7f6bdc15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be3847640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6bdc066d90 0x7f6bdc0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be3847640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc15e9f0 con 0x7f6bdc0b4f70 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be3847640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc0cda10 con 0x7f6bdc066d90 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be3847640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0b5890 con 0x7f6bdc06c6d0 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be15e3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6bdc0b4f70 0x7f6bdc0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be15e3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6bdc0b4f70 0x7f6bdc0b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46310/0 (socket says 172.21.15.77:46310) 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be15e3640 1 -- 172.21.15.77:0/178265336 learned_addr learned my addr 172.21.15.77:0/178265336 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be0de2640 1 --2- 172.21.15.77:0/178265336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc06c6d0 0x7f6bdc15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.686+0000 7f6be1de4640 1 --2- 172.21.15.77:0/178265336 >> v2:172.21.15.158:3300/0 conn(0x7f6bdc066d90 0x7f6bdc0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.687+0000 7f6be15e3640 1 -- 172.21.15.77:0/178265336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc06c6d0 msgr2=0x7f6bdc15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.687+0000 7f6be15e3640 1 --2- 172.21.15.77:0/178265336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc06c6d0 0x7f6bdc15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.687+0000 7f6be15e3640 1 -- 172.21.15.77:0/178265336 >> v2:172.21.15.158:3300/0 conn(0x7f6bdc066d90 msgr2=0x7f6bdc0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.687+0000 7f6be15e3640 1 --2- 172.21.15.77:0/178265336 >> v2:172.21.15.158:3300/0 conn(0x7f6bdc066d90 0x7f6bdc0ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.687+0000 7f6be15e3640 1 -- 172.21.15.77:0/178265336 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0cd450 con 0x7f6bdc0b4f70 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.687+0000 7f6be15e3640 1 --2- 172.21.15.77:0/178265336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc0b4f70 0x7f6bdc0b5350 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0723c0 tx=0x7f6bd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9ea6f3d34b809e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.687+0000 7f6bd3fff640 1 -- 172.21.15.77:0/178265336 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8035020 con 0x7f6bdc0b4f70 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6bd3fff640 1 -- 172.21.15.77:0/178265336 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8031870 con 0x7f6bdc0b4f70 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6bd3fff640 1 -- 172.21.15.77:0/178265336 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8031c40 con 0x7f6bdc0b4f70 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 -- 172.21.15.77:0/178265336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc0b4f70 msgr2=0x7f6bdc0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 --2- 172.21.15.77:0/178265336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc0b4f70 0x7f6bdc0b5350 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0723c0 tx=0x7f6bd802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 -- 172.21.15.77:0/178265336 shutdown_connections 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 --2- 172.21.15.77:0/178265336 >> v2:172.21.15.158:3300/0 conn(0x7f6bdc066d90 0x7f6bdc0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 --2- 172.21.15.77:0/178265336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc06c6d0 0x7f6bdc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 --2- 172.21.15.77:0/178265336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc0b4f70 0x7f6bdc0b5350 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 -- 172.21.15.77:0/178265336 >> 172.21.15.77:0/178265336 conn(0x7f6bdc0724d0 msgr2=0x7f6bdc0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 -- 172.21.15.77:0/178265336 shutdown_connections 2024-03-06T07:22:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.688+0000 7f6be3847640 1 -- 172.21.15.77:0/178265336 wait complete. 2024-03-06T07:22:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.689+0000 7f6be3847640 1 Processor -- start 2024-03-06T07:22:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.689+0000 7f6be3847640 1 -- start start 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be3847640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bdc066d90 0x7f6bdc15a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be3847640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc0b4f70 0x7f6bdc15c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be3847640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc15ce90 0x7f6bdc156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be3847640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc15f3c0 con 0x7f6bdc15ce90 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be3847640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc15e680 con 0x7f6bdc066d90 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be3847640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0cd090 con 0x7f6bdc0b4f70 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be1de4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc15ce90 0x7f6bdc156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be1de4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc15ce90 0x7f6bdc156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46320/0 (socket says 172.21.15.77:46320) 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be1de4640 1 -- 172.21.15.77:0/286206350 learned_addr learned my addr 172.21.15.77:0/286206350 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be15e3640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bdc066d90 0x7f6bdc15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be0de2640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc0b4f70 0x7f6bdc15c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be1de4640 1 -- 172.21.15.77:0/286206350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc0b4f70 msgr2=0x7f6bdc15c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be1de4640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc0b4f70 0x7f6bdc15c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be1de4640 1 -- 172.21.15.77:0/286206350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bdc066d90 msgr2=0x7f6bdc15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.690+0000 7f6be1de4640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bdc066d90 0x7f6bdc15a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.691+0000 7f6be1de4640 1 -- 172.21.15.77:0/286206350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc157a40 con 0x7f6bdc15ce90 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.691+0000 7f6be1de4640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc15ce90 0x7f6bdc156bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400eed0 tx=0x7f6bd400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.691+0000 7f6bd1ffb640 1 -- 172.21.15.77:0/286206350 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4018020 con 0x7f6bdc15ce90 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.691+0000 7f6bd1ffb640 1 -- 172.21.15.77:0/286206350 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd400bd30 con 0x7f6bdc15ce90 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.691+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc1e78f0 con 0x7f6bdc15ce90 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.691+0000 7f6bd1ffb640 1 -- 172.21.15.77:0/286206350 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4014640 con 0x7f6bdc15ce90 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.691+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc15d810 con 0x7f6bdc15ce90 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.692+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc04f330 con 0x7f6bdc15ce90 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.692+0000 7f6bd1ffb640 1 -- 172.21.15.77:0/286206350 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bd4005d10 con 0x7f6bdc15ce90 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.693+0000 7f6bd1ffb640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bc006b4d0 0x7f6bc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.693+0000 7f6be15e3640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bc006b4d0 0x7f6bc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.693+0000 7f6bd1ffb640 1 -- 172.21.15.77:0/286206350 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bd408a980 con 0x7f6bdc15ce90 2024-03-06T07:22:23.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.693+0000 7f6be15e3640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bc006b4d0 0x7f6bc006d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8036000 tx=0x7f6bd803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.697+0000 7f6bd1ffb640 1 -- 172.21.15.77:0/286206350 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bd4022030 con 0x7f6bdc15ce90 2024-03-06T07:22:23.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.871+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bdc15b270 con 0x7f6bdc15ce90 2024-03-06T07:22:23.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.872+0000 7f6bd1ffb640 1 -- 172.21.15.77:0/286206350 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f6bdc157a40 con 0x7f6bdc15ce90 2024-03-06T07:22:23.875 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:23.875 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bc006b4d0 msgr2=0x7f6bc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bc006b4d0 0x7f6bc006d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8036000 tx=0x7f6bd803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc15ce90 msgr2=0x7f6bdc156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc15ce90 0x7f6bdc156bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400eed0 tx=0x7f6bd400b480 comp rx=0 tx=0).stop 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 shutdown_connections 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bc006b4d0 0x7f6bc006d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bdc15ce90 0x7f6bdc156bb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bdc0b4f70 0x7f6bdc15c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 --2- 172.21.15.77:0/286206350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bdc066d90 0x7f6bdc15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 >> 172.21.15.77:0/286206350 conn(0x7f6bdc0724d0 msgr2=0x7f6bdc0c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 shutdown_connections 2024-03-06T07:22:23.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.875+0000 7f6be3847640 1 -- 172.21.15.77:0/286206350 wait complete. 2024-03-06T07:22:23.888 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:23.888 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:23.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.989+0000 7f89ab4d4640 1 Processor -- start 2024-03-06T07:22:23.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.989+0000 7f89ab4d4640 1 -- start start 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89ab4d4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f89a40b55e0 0x7f89a4169140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89ab4d4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a406b710 con 0x7f89a4066d90 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89ab4d4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a406b150 con 0x7f89a40cf9a0 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89ab4d4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a40b61c0 con 0x7f89a40b55e0 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89a8a6f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f89a4066d90 0x7f89a40b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46606/0 (socket says 172.21.15.77:46606) 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89a8a6f640 1 -- 172.21.15.77:0/1447646693 learned_addr learned my addr 172.21.15.77:0/1447646693 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89a9a71640 1 --2- 172.21.15.77:0/1447646693 >> v2:172.21.15.158:3300/0 conn(0x7f89a40b55e0 0x7f89a4169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f899bfff640 1 -- 172.21.15.77:0/1447646693 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667563302 0 0) 0x7f89a406b710 con 0x7f89a4066d90 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f899bfff640 1 -- 172.21.15.77:0/1447646693 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8990003520 con 0x7f89a4066d90 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89a9a71640 1 -- 172.21.15.77:0/1447646693 >> v1:172.21.15.158:6789/0 conn(0x7f89a40cf9a0 legacy=0x7f89a40b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89a9a71640 1 -- 172.21.15.77:0/1447646693 >> v1:172.21.15.77:6789/0 conn(0x7f89a4066d90 legacy=0x7f89a40b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.990+0000 7f89a9a71640 1 -- 172.21.15.77:0/1447646693 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a406b430 con 0x7f89a40b55e0 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.991+0000 7f89a9a71640 1 --2- 172.21.15.77:0/1447646693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a40b55e0 0x7f89a4169140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f899c009d90 tx=0x7f899c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82a6a2a5896805e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.991+0000 7f899bfff640 1 -- 172.21.15.77:0/1447646693 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c039070 con 0x7f89a40b55e0 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.991+0000 7f899bfff640 1 -- 172.21.15.77:0/1447646693 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899c02cd00 con 0x7f89a40b55e0 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.991+0000 7f899bfff640 1 -- 172.21.15.77:0/1447646693 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c034660 con 0x7f89a40b55e0 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.992+0000 7f89ab4d4640 1 -- 172.21.15.77:0/1447646693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a40b55e0 msgr2=0x7f89a4169140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.992+0000 7f89ab4d4640 1 --2- 172.21.15.77:0/1447646693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a40b55e0 0x7f89a4169140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f899c009d90 tx=0x7f899c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.992+0000 7f89ab4d4640 1 -- 172.21.15.77:0/1447646693 shutdown_connections 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.992+0000 7f89ab4d4640 1 --2- 172.21.15.77:0/1447646693 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a40b55e0 0x7f89a4169140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.992+0000 7f89ab4d4640 1 -- 172.21.15.77:0/1447646693 >> 172.21.15.77:0/1447646693 conn(0x7f89a40b1d70 msgr2=0x7f89a40b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.992+0000 7f89ab4d4640 1 -- 172.21.15.77:0/1447646693 shutdown_connections 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.992+0000 7f89ab4d4640 1 -- 172.21.15.77:0/1447646693 wait complete. 2024-03-06T07:22:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.993+0000 7f89ab4d4640 1 Processor -- start 2024-03-06T07:22:24.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.993+0000 7f89ab4d4640 1 -- start start 2024-03-06T07:22:24.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89ab4d4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a4066d90 0x7f89a40ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89ab4d4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89a40b55e0 0x7f89a40cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89ab4d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a40cf9a0 0x7f89a40c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89ab4d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40cf050 con 0x7f89a40cf9a0 2024-03-06T07:22:24.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89ab4d4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40b0c50 con 0x7f89a4066d90 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89ab4d4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40cf1d0 con 0x7f89a40b55e0 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a40cf9a0 0x7f89a40c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a40cf9a0 0x7f89a40c6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46332/0 (socket says 172.21.15.77:46332) 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 -- 172.21.15.77:0/2910285786 learned_addr learned my addr 172.21.15.77:0/2910285786 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a8a6f640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89a40b55e0 0x7f89a40cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9270640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a4066d90 0x7f89a40ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 -- 172.21.15.77:0/2910285786 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89a40b55e0 msgr2=0x7f89a40cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89a40b55e0 0x7f89a40cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 -- 172.21.15.77:0/2910285786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a4066d90 msgr2=0x7f89a40ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a4066d90 0x7f89a40ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 -- 172.21.15.77:0/2910285786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a40c76f0 con 0x7f89a40cf9a0 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89a9a71640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a40cf9a0 0x7f89a40c6030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f899c009eb0 tx=0x7f899c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f8999ffb640 1 -- 172.21.15.77:0/2910285786 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c039050 con 0x7f89a40cf9a0 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f8999ffb640 1 -- 172.21.15.77:0/2910285786 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899c045070 con 0x7f89a40cf9a0 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f8999ffb640 1 -- 172.21.15.77:0/2910285786 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c009450 con 0x7f89a40cf9a0 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.994+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a40cacd0 con 0x7f89a40cf9a0 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.995+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a40c7cf0 con 0x7f89a40cf9a0 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.996+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a40caac0 con 0x7f89a40cf9a0 2024-03-06T07:22:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.996+0000 7f8999ffb640 1 -- 172.21.15.77:0/2910285786 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f899c03e850 con 0x7f89a40cf9a0 2024-03-06T07:22:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.996+0000 7f8999ffb640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f898806b500 0x7f898806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.996+0000 7f8999ffb640 1 -- 172.21.15.77:0/2910285786 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f899c0acec0 con 0x7f89a40cf9a0 2024-03-06T07:22:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.997+0000 7f89a9270640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f898806b500 0x7f898806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:23.997+0000 7f89a9270640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f898806b500 0x7f898806d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89a00016e0 tx=0x7f89a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.001+0000 7f8999ffb640 1 -- 172.21.15.77:0/2910285786 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f899c07b8a0 con 0x7f89a40cf9a0 2024-03-06T07:22:24.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.172+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89a409e3d0 con 0x7f89a40cf9a0 2024-03-06T07:22:24.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.173+0000 7f8999ffb640 1 -- 172.21.15.77:0/2910285786 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f899c07f940 con 0x7f89a40cf9a0 2024-03-06T07:22:24.175 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f898806b500 msgr2=0x7f898806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f898806b500 0x7f898806d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89a00016e0 tx=0x7f89a0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a40cf9a0 msgr2=0x7f89a40c6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a40cf9a0 0x7f89a40c6030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f899c009eb0 tx=0x7f899c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 shutdown_connections 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f898806b500 0x7f898806d9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a40cf9a0 0x7f89a40c6030 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89a40b55e0 0x7f89a40cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 --2- 172.21.15.77:0/2910285786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89a4066d90 0x7f89a40ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 >> 172.21.15.77:0/2910285786 conn(0x7f89a40b1d70 msgr2=0x7f89a406d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 shutdown_connections 2024-03-06T07:22:24.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.175+0000 7f89ab4d4640 1 -- 172.21.15.77:0/2910285786 wait complete. 2024-03-06T07:22:24.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:24.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.289+0000 7f9d38748640 1 Processor -- start 2024-03-06T07:22:24.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.289+0000 7f9d38748640 1 -- start start 2024-03-06T07:22:24.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d38748640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d300b0db0 0x7f9d30066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d38748640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d300ce750 0x7f9d300b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d38748640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3006b050 con 0x7f9d3015f4f0 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d38748640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3006b2c0 con 0x7f9d300b0db0 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d38748640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d300b0890 con 0x7f9d300ce750 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d35ce3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d300b0db0 0x7f9d30066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d35ce3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d300b0db0 0x7f9d30066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46334/0 (socket says 172.21.15.77:46334) 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d35ce3640 1 -- 172.21.15.77:0/3367979178 learned_addr learned my addr 172.21.15.77:0/3367979178 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d364e4640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9d3015f4f0 0x7f9d300ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46618/0 (socket says 172.21.15.77:46618) 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d36ce5640 1 --2- 172.21.15.77:0/3367979178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d300ce750 0x7f9d300b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d35ce3640 1 -- 172.21.15.77:0/3367979178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d300ce750 msgr2=0x7f9d300b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d35ce3640 1 --2- 172.21.15.77:0/3367979178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d300ce750 0x7f9d300b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d35ce3640 1 -- 172.21.15.77:0/3367979178 >> v1:172.21.15.77:6789/0 conn(0x7f9d3015f4f0 legacy=0x7f9d300ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d354e2640 1 -- 172.21.15.77:0/3367979178 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101591591 0 0) 0x7f9d3006b050 con 0x7f9d3015f4f0 2024-03-06T07:22:24.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.290+0000 7f9d35ce3640 1 -- 172.21.15.77:0/3367979178 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d300672e0 con 0x7f9d300b0db0 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.291+0000 7f9d35ce3640 1 --2- 172.21.15.77:0/3367979178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30066d00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d20009870 tx=0x7f9d2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cda17c5b162d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.291+0000 7f9d354e2640 1 -- 172.21.15.77:0/3367979178 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d20034070 con 0x7f9d300b0db0 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.291+0000 7f9d354e2640 1 -- 172.21.15.77:0/3367979178 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d20030840 con 0x7f9d300b0db0 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.291+0000 7f9d354e2640 1 -- 172.21.15.77:0/3367979178 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d20030c10 con 0x7f9d300b0db0 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.292+0000 7f9d38748640 1 -- 172.21.15.77:0/3367979178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 msgr2=0x7f9d30066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.292+0000 7f9d38748640 1 --2- 172.21.15.77:0/3367979178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30066d00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d20009870 tx=0x7f9d2002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.292+0000 7f9d38748640 1 -- 172.21.15.77:0/3367979178 shutdown_connections 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.292+0000 7f9d38748640 1 --2- 172.21.15.77:0/3367979178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d300ce750 0x7f9d300b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.292+0000 7f9d38748640 1 --2- 172.21.15.77:0/3367979178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30066d00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.292+0000 7f9d38748640 1 -- 172.21.15.77:0/3367979178 >> 172.21.15.77:0/3367979178 conn(0x7f9d300b1d70 msgr2=0x7f9d300b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.292+0000 7f9d38748640 1 -- 172.21.15.77:0/3367979178 shutdown_connections 2024-03-06T07:22:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.292+0000 7f9d38748640 1 -- 172.21.15.77:0/3367979178 wait complete. 2024-03-06T07:22:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.293+0000 7f9d38748640 1 Processor -- start 2024-03-06T07:22:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.293+0000 7f9d38748640 1 -- start start 2024-03-06T07:22:24.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d38748640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d38748640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d300ce750 0x7f9d301492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d38748640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d3015f4f0 0x7f9d3006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d38748640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3015f8d0 con 0x7f9d300b0db0 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d38748640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d300cebe0 con 0x7f9d300ce750 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d38748640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d30146f40 con 0x7f9d3015f4f0 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30146a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46344/0 (socket says 172.21.15.77:46344) 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 -- 172.21.15.77:0/3426090861 learned_addr learned my addr 172.21.15.77:0/3426090861 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d36ce5640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d3015f4f0 0x7f9d3006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d35ce3640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d300ce750 0x7f9d301492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 -- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d3015f4f0 msgr2=0x7f9d3006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d3015f4f0 0x7f9d3006d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 -- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d300ce750 msgr2=0x7f9d301492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d300ce750 0x7f9d301492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 -- 172.21.15.77:0/3426090861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30149d70 con 0x7f9d300b0db0 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d35ce3640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d300ce750 0x7f9d301492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:24.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d364e4640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30146a00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00c460 tx=0x7f9d2c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.294+0000 7f9d26ffd640 1 -- 172.21.15.77:0/3426090861 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c018070 con 0x7f9d300b0db0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.295+0000 7f9d26ffd640 1 -- 172.21.15.77:0/3426090861 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c007500 con 0x7f9d300b0db0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.295+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3014a050 con 0x7f9d300b0db0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.295+0000 7f9d26ffd640 1 -- 172.21.15.77:0/3426090861 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c004a20 con 0x7f9d300b0db0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.295+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d30147710 con 0x7f9d300b0db0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.295+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d30147220 con 0x7f9d300b0db0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.296+0000 7f9d26ffd640 1 -- 172.21.15.77:0/3426090861 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d2c01b070 con 0x7f9d300b0db0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.296+0000 7f9d26ffd640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d1406b4d0 0x7f9d1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.296+0000 7f9d26ffd640 1 -- 172.21.15.77:0/3426090861 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d2c089390 con 0x7f9d300b0db0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.296+0000 7f9d35ce3640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d1406b4d0 0x7f9d1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.297+0000 7f9d35ce3640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d1406b4d0 0x7f9d1406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d20036000 tx=0x7f9d2003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.300+0000 7f9d26ffd640 1 -- 172.21.15.77:0/3426090861 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d2c057c50 con 0x7f9d300b0db0 2024-03-06T07:22:24.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.471+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d3009e3d0 con 0x7f9d300b0db0 2024-03-06T07:22:24.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.472+0000 7f9d26ffd640 1 -- 172.21.15.77:0/3426090861 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9d2c05bcf0 con 0x7f9d300b0db0 2024-03-06T07:22:24.474 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.473+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d1406b4d0 msgr2=0x7f9d1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.473+0000 7f9d38748640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d1406b4d0 0x7f9d1406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d20036000 tx=0x7f9d2003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.473+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 msgr2=0x7f9d30146a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.473+0000 7f9d38748640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30146a00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00c460 tx=0x7f9d2c00c920 comp rx=0 tx=0).stop 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.474+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 shutdown_connections 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.474+0000 7f9d38748640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d3015f4f0 0x7f9d3006d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.474+0000 7f9d38748640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d300ce750 0x7f9d301492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.474+0000 7f9d38748640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d1406b4d0 0x7f9d1406d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.474+0000 7f9d38748640 1 --2- 172.21.15.77:0/3426090861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d300b0db0 0x7f9d30146a00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.474+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 >> 172.21.15.77:0/3426090861 conn(0x7f9d300b1d70 msgr2=0x7f9d3006e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.474+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 shutdown_connections 2024-03-06T07:22:24.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.474+0000 7f9d38748640 1 -- 172.21.15.77:0/3426090861 wait complete. 2024-03-06T07:22:24.486 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:24.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.588+0000 7efef5393640 1 Processor -- start 2024-03-06T07:22:24.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.588+0000 7efef5393640 1 -- start start 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efef5393640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efef00b1e40 0x7efef0168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efef5393640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef00d2730 con 0x7efef01614d0 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efef5393640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef01618b0 con 0x7efef00d4860 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efef5393640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef00d4cf0 con 0x7efef00b1e40 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeeeffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efef01614d0 0x7efef00d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46622/0 (socket says 172.21.15.77:46622) 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeeeffd640 1 -- 172.21.15.77:0/2981137309 learned_addr learned my addr 172.21.15.77:0/2981137309 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeef7fe640 1 --2- 172.21.15.77:0/2981137309 >> v2:172.21.15.158:3300/0 conn(0x7efef00b1e40 0x7efef0168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeedffb640 1 -- 172.21.15.77:0/2981137309 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496459682 0 0) 0x7efef00d2730 con 0x7efef01614d0 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeedffb640 1 -- 172.21.15.77:0/2981137309 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed8003520 con 0x7efef01614d0 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeef7fe640 1 -- 172.21.15.77:0/2981137309 >> v1:172.21.15.158:6789/0 conn(0x7efef00d4860 legacy=0x7efef00d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeedffb640 1 -- 172.21.15.77:0/2981137309 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116313425 0 0) 0x7efef01618b0 con 0x7efef00d4860 2024-03-06T07:22:24.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeef7fe640 1 -- 172.21.15.77:0/2981137309 >> v1:172.21.15.77:6789/0 conn(0x7efef01614d0 legacy=0x7efef00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.589+0000 7efeef7fe640 1 -- 172.21.15.77:0/2981137309 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef00cf400 con 0x7efef00b1e40 2024-03-06T07:22:24.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.590+0000 7efeef7fe640 1 --2- 172.21.15.77:0/2981137309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00b1e40 0x7efef0168b60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efef00cf5d0 tx=0x7efee802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5095afefe478130 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.590+0000 7efeedffb640 1 -- 172.21.15.77:0/2981137309 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8039070 con 0x7efef00b1e40 2024-03-06T07:22:24.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.590+0000 7efeedffb640 1 -- 172.21.15.77:0/2981137309 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee802cc50 con 0x7efef00b1e40 2024-03-06T07:22:24.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.590+0000 7efeedffb640 1 -- 172.21.15.77:0/2981137309 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8034440 con 0x7efef00b1e40 2024-03-06T07:22:24.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.591+0000 7efef5393640 1 -- 172.21.15.77:0/2981137309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00b1e40 msgr2=0x7efef0168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.591+0000 7efef5393640 1 --2- 172.21.15.77:0/2981137309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00b1e40 0x7efef0168b60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efef00cf5d0 tx=0x7efee802bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.591+0000 7efef5393640 1 -- 172.21.15.77:0/2981137309 shutdown_connections 2024-03-06T07:22:24.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.591+0000 7efef5393640 1 --2- 172.21.15.77:0/2981137309 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00b1e40 0x7efef0168b60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.591+0000 7efef5393640 1 -- 172.21.15.77:0/2981137309 >> 172.21.15.77:0/2981137309 conn(0x7efef00d3d40 msgr2=0x7efef00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.591+0000 7efef5393640 1 -- 172.21.15.77:0/2981137309 shutdown_connections 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.591+0000 7efef5393640 1 -- 172.21.15.77:0/2981137309 wait complete. 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.592+0000 7efef5393640 1 Processor -- start 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.592+0000 7efef5393640 1 -- start start 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efef5393640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef00b1e40 0x7efef00cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efef5393640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00d4860 0x7efef00c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efef5393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efef01614d0 0x7efef01f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efef5393640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00cfc60 con 0x7efef00b1e40 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efef5393640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00cf980 con 0x7efef00d4860 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efef5393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef00b0910 con 0x7efef01614d0 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef00b1e40 0x7efef00cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef00b1e40 0x7efef00cd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46356/0 (socket says 172.21.15.77:46356) 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 -- 172.21.15.77:0/1685547415 learned_addr learned my addr 172.21.15.77:0/1685547415 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeef7fe640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efef01614d0 0x7efef01f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeee7fc640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00d4860 0x7efef00c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 -- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efef01614d0 msgr2=0x7efef01f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efef01614d0 0x7efef01f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 -- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00d4860 msgr2=0x7efef00c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00d4860 0x7efef00c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 -- 172.21.15.77:0/1685547415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef01f44a0 con 0x7efef00b1e40 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeee7fc640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00d4860 0x7efef00c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efeeeffd640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef00b1e40 0x7efef00cd900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efee000c430 tx=0x7efee000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efed77fe640 1 -- 172.21.15.77:0/1685547415 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0017070 con 0x7efef00b1e40 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efed77fe640 1 -- 172.21.15.77:0/1685547415 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee0005ce0 con 0x7efef00b1e40 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef00c9d30 con 0x7efef00b1e40 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efed77fe640 1 -- 172.21.15.77:0/1685547415 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee00072f0 con 0x7efef00b1e40 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.593+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef00ca2f0 con 0x7efef00b1e40 2024-03-06T07:22:24.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.594+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef0068ff0 con 0x7efef00b1e40 2024-03-06T07:22:24.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.595+0000 7efed77fe640 1 -- 172.21.15.77:0/1685547415 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efee001a070 con 0x7efef00b1e40 2024-03-06T07:22:24.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.595+0000 7efed77fe640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efedc06b4d0 0x7efedc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.595+0000 7efed77fe640 1 -- 172.21.15.77:0/1685547415 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efee00895a0 con 0x7efef00b1e40 2024-03-06T07:22:24.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.595+0000 7efeee7fc640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efedc06b4d0 0x7efedc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.595+0000 7efeee7fc640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efedc06b4d0 0x7efedc06d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efee4000f60 tx=0x7efee4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.599+0000 7efed77fe640 1 -- 172.21.15.77:0/1685547415 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efef01f44a0 con 0x7efef00b1e40 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.771+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efef00747a0 con 0x7efef00b1e40 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.771+0000 7efed77fe640 1 -- 172.21.15.77:0/1685547415 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7efee005ce20 con 0x7efef00b1e40 2024-03-06T07:22:24.773 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:24.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efedc06b4d0 msgr2=0x7efedc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efedc06b4d0 0x7efedc06d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efee4000f60 tx=0x7efee4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef00b1e40 msgr2=0x7efef00cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef00b1e40 0x7efef00cd900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efee000c430 tx=0x7efee000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 shutdown_connections 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efef01614d0 0x7efef01f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efef00d4860 0x7efef00c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efedc06b4d0 0x7efedc06d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 --2- 172.21.15.77:0/1685547415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef00b1e40 0x7efef00cd900 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 >> 172.21.15.77:0/1685547415 conn(0x7efef00d3d40 msgr2=0x7efef0160600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 shutdown_connections 2024-03-06T07:22:24.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.773+0000 7efef5393640 1 -- 172.21.15.77:0/1685547415 wait complete. 2024-03-06T07:22:24.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:24.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.883+0000 7f8cc9484640 1 Processor -- start 2024-03-06T07:22:24.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.883+0000 7f8cc9484640 1 -- start start 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc9484640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc414bb80 0x7f8cc40b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc9484640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8cc40cc330 0x7f8cc406d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc9484640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40b3df0 con 0x7f8cc406d9d0 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc9484640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc40b4a00 con 0x7f8cc40cc330 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc9484640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40b40d0 con 0x7f8cc414bb80 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc27fc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8cc40cc330 0x7f8cc406d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc37fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8cc406d9d0 0x7f8cc4165e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:59860/0 (socket says 172.21.15.77:59860) 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc2ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc414bb80 0x7f8cc40b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc27fc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8cc40cc330 0x7f8cc406d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:33964/0 (socket says 172.21.15.77:33964) 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.884+0000 7f8cc37fe640 1 -- 172.21.15.77:0/3941052475 learned_addr learned my addr 172.21.15.77:0/3941052475 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:24.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.885+0000 7f8cc27fc640 1 -- 172.21.15.77:0/3941052475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc414bb80 msgr2=0x7f8cc40b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.885+0000 7f8cc27fc640 1 --2- 172.21.15.77:0/3941052475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc414bb80 0x7f8cc40b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.885+0000 7f8cc27fc640 1 -- 172.21.15.77:0/3941052475 >> v1:172.21.15.158:6789/0 conn(0x7f8cc406d9d0 legacy=0x7f8cc4165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.885+0000 7f8cc1ffb640 1 -- 172.21.15.77:0/3941052475 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124916076 0 0) 0x7f8cc40b3df0 con 0x7f8cc406d9d0 2024-03-06T07:22:24.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.885+0000 7f8cc27fc640 1 -- 172.21.15.77:0/3941052475 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40b43b0 con 0x7f8cc40cc330 2024-03-06T07:22:24.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.885+0000 7f8cc27fc640 1 --2- 172.21.15.77:0/3941052475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc40cc330 0x7f8cc406d490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80097a0 tx=0x7f8cb802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e157cd5811791cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc1ffb640 1 -- 172.21.15.77:0/3941052475 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8035020 con 0x7f8cc40cc330 2024-03-06T07:22:24.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc1ffb640 1 -- 172.21.15.77:0/3941052475 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb8031830 con 0x7f8cc40cc330 2024-03-06T07:22:24.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc1ffb640 1 -- 172.21.15.77:0/3941052475 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8031c00 con 0x7f8cc40cc330 2024-03-06T07:22:24.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc9484640 1 -- 172.21.15.77:0/3941052475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc40cc330 msgr2=0x7f8cc406d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3941052475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc40cc330 0x7f8cc406d490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80097a0 tx=0x7f8cb802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc9484640 1 -- 172.21.15.77:0/3941052475 shutdown_connections 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3941052475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc40cc330 0x7f8cc406d490 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3941052475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc414bb80 0x7f8cc40b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc9484640 1 -- 172.21.15.77:0/3941052475 >> 172.21.15.77:0/3941052475 conn(0x7f8cc40cf8a0 msgr2=0x7f8cc40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc9484640 1 -- 172.21.15.77:0/3941052475 shutdown_connections 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.886+0000 7f8cc9484640 1 -- 172.21.15.77:0/3941052475 wait complete. 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.887+0000 7f8cc9484640 1 Processor -- start 2024-03-06T07:22:24.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.887+0000 7f8cc9484640 1 -- start start 2024-03-06T07:22:24.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc9484640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc406d9d0 0x7f8cc40c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc9484640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cc40cc330 0x7f8cc4158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc9484640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc414bb80 0x7f8cc41491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc9484640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40cd4c0 con 0x7f8cc40cc330 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc9484640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40cd340 con 0x7f8cc414bb80 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc9484640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40ce840 con 0x7f8cc406d9d0 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cc40cc330 0x7f8cc4158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cc40cc330 0x7f8cc4158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46360/0 (socket says 172.21.15.77:46360) 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc27fc640 1 -- 172.21.15.77:0/3240340346 learned_addr learned my addr 172.21.15.77:0/3240340346 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc2ffd640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc406d9d0 0x7f8cc40c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.888+0000 7f8cc37fe640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc414bb80 0x7f8cc41491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cc27fc640 1 -- 172.21.15.77:0/3240340346 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc406d9d0 msgr2=0x7f8cc40c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cc27fc640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc406d9d0 0x7f8cc40c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cc27fc640 1 -- 172.21.15.77:0/3240340346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc414bb80 msgr2=0x7f8cc41491c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cc27fc640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc414bb80 0x7f8cc41491c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cc27fc640 1 -- 172.21.15.77:0/3240340346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40c7be0 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cc27fc640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cc40cc330 0x7f8cc4158960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8000c00 tx=0x7f8cb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cab7fe640 1 -- 172.21.15.77:0/3240340346 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8035070 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cab7fe640 1 -- 172.21.15.77:0/3240340346 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb8004230 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc40c7ec0 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cab7fe640 1 -- 172.21.15.77:0/3240340346 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb803a630 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.889+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc40c8480 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.890+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc404f330 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.890+0000 7f8cab7fe640 1 -- 172.21.15.77:0/3240340346 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cb8038020 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.891+0000 7f8cab7fe640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cb006b4d0 0x7f8cb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.891+0000 7f8cab7fe640 1 -- 172.21.15.77:0/3240340346 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cb80abc50 con 0x7f8cc40cc330 2024-03-06T07:22:24.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.891+0000 7f8cc2ffd640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cb006b4d0 0x7f8cb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.891+0000 7f8cc2ffd640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cb006b4d0 0x7f8cb006d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4011340 tx=0x7f8cb400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:24.895+0000 7f8cab7fe640 1 -- 172.21.15.77:0/3240340346 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8cb80b1020 con 0x7f8cc40cc330 2024-03-06T07:22:25.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.067+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cc40d0770 con 0x7f8cc40cc330 2024-03-06T07:22:25.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.068+0000 7f8cab7fe640 1 -- 172.21.15.77:0/3240340346 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f8cb8040070 con 0x7f8cc40cc330 2024-03-06T07:22:25.070 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:25.070 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:25.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cb006b4d0 msgr2=0x7f8cb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cb006b4d0 0x7f8cb006d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4011340 tx=0x7f8cb400b040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cc40cc330 msgr2=0x7f8cc4158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cc40cc330 0x7f8cc4158960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8000c00 tx=0x7f8cb80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 shutdown_connections 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cb006b4d0 0x7f8cb006d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cc40cc330 0x7f8cc4158960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8cc414bb80 0x7f8cc41491c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 --2- 172.21.15.77:0/3240340346 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8cc406d9d0 0x7f8cc40c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 >> 172.21.15.77:0/3240340346 conn(0x7f8cc40cf8a0 msgr2=0x7f8cc40c48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 shutdown_connections 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.070+0000 7f8cc9484640 1 -- 172.21.15.77:0/3240340346 wait complete. 2024-03-06T07:22:25.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:25.083 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:25.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.184+0000 7f92f8779640 1 Processor -- start 2024-03-06T07:22:25.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.184+0000 7f92f8779640 1 -- start start 2024-03-06T07:22:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f8779640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92f01614d0 0x7f92f00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f8779640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d1910 0x7f92f00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f8779640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f00d2740 con 0x7f92f00d22b0 2024-03-06T07:22:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f8779640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f01618b0 con 0x7f92f01614d0 2024-03-06T07:22:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f8779640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00cf400 con 0x7f92f00d1910 2024-03-06T07:22:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f6515640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92f01614d0 0x7f92f00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f5d14640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d1910 0x7f92f00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f6d16640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f92f00d22b0 0x7f92f00d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:59862/0 (socket says 172.21.15.77:59862) 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f6515640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92f01614d0 0x7f92f00d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:33978/0 (socket says 172.21.15.77:33978) 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f6d16640 1 -- 172.21.15.77:0/2802840238 learned_addr learned my addr 172.21.15.77:0/2802840238 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.185+0000 7f92f5d14640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d1910 0x7f92f00d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35204/0 (socket says 172.21.15.77:35204) 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.186+0000 7f92f6515640 1 -- 172.21.15.77:0/2802840238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d1910 msgr2=0x7f92f00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.186+0000 7f92f6515640 1 --2- 172.21.15.77:0/2802840238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d1910 0x7f92f00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.186+0000 7f92f6515640 1 -- 172.21.15.77:0/2802840238 >> v1:172.21.15.158:6789/0 conn(0x7f92f00d22b0 legacy=0x7f92f00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.186+0000 7f92f6515640 1 -- 172.21.15.77:0/2802840238 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f00cf6a0 con 0x7f92f01614d0 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.186+0000 7f92f5d14640 1 --2- 172.21.15.77:0/2802840238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d1910 0x7f92f00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f6515640 1 --2- 172.21.15.77:0/2802840238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f01614d0 0x7f92f00d4860 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f92ec009d90 tx=0x7f92ec02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a476baa64bd79bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f5513640 1 -- 172.21.15.77:0/2802840238 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec039070 con 0x7f92f01614d0 2024-03-06T07:22:25.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f5513640 1 -- 172.21.15.77:0/2802840238 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec02cc50 con 0x7f92f01614d0 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f5513640 1 -- 172.21.15.77:0/2802840238 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec034440 con 0x7f92f01614d0 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f8779640 1 -- 172.21.15.77:0/2802840238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f01614d0 msgr2=0x7f92f00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f8779640 1 --2- 172.21.15.77:0/2802840238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f01614d0 0x7f92f00d4860 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f92ec009d90 tx=0x7f92ec02bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f8779640 1 -- 172.21.15.77:0/2802840238 shutdown_connections 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f8779640 1 --2- 172.21.15.77:0/2802840238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d1910 0x7f92f00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f8779640 1 --2- 172.21.15.77:0/2802840238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f01614d0 0x7f92f00d4860 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f8779640 1 -- 172.21.15.77:0/2802840238 >> 172.21.15.77:0/2802840238 conn(0x7f92f00d3d40 msgr2=0x7f92f00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f8779640 1 -- 172.21.15.77:0/2802840238 shutdown_connections 2024-03-06T07:22:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.187+0000 7f92f8779640 1 -- 172.21.15.77:0/2802840238 wait complete. 2024-03-06T07:22:25.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.188+0000 7f92f8779640 1 Processor -- start 2024-03-06T07:22:25.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.188+0000 7f92f8779640 1 -- start start 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f8779640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f00d1910 0x7f92f00ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f8779640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d22b0 0x7f92f014ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f8779640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92f01614d0 0x7f92f00b9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f8779640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00b0910 con 0x7f92f01614d0 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f8779640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00cfc60 con 0x7f92f00d1910 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f8779640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00b1e40 con 0x7f92f00d22b0 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f6d16640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92f01614d0 0x7f92f00b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f6515640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f00d1910 0x7f92f00ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f6d16640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92f01614d0 0x7f92f00b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46374/0 (socket says 172.21.15.77:46374) 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f6d16640 1 -- 172.21.15.77:0/2250961361 learned_addr learned my addr 172.21.15.77:0/2250961361 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.189+0000 7f92f5d14640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d22b0 0x7f92f014ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f6d16640 1 -- 172.21.15.77:0/2250961361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d22b0 msgr2=0x7f92f014ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f6d16640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d22b0 0x7f92f014ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f6d16640 1 -- 172.21.15.77:0/2250961361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f00d1910 msgr2=0x7f92f00ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f6d16640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f00d1910 0x7f92f00ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f6d16640 1 -- 172.21.15.77:0/2250961361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f00cb250 con 0x7f92f01614d0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f5d14640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d22b0 0x7f92f014ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f6515640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f00d1910 0x7f92f00ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f6d16640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92f01614d0 0x7f92f00b9650 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f92e800c420 tx=0x7f92e800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92e6ffd640 1 -- 172.21.15.77:0/2250961361 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8017070 con 0x7f92f01614d0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92e6ffd640 1 -- 172.21.15.77:0/2250961361 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e8005ce0 con 0x7f92f01614d0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f00cb4f0 con 0x7f92f01614d0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92e6ffd640 1 -- 172.21.15.77:0/2250961361 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e80072d0 con 0x7f92f01614d0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.190+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f014b280 con 0x7f92f01614d0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.191+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f0068ff0 con 0x7f92f01614d0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.192+0000 7f92e6ffd640 1 -- 172.21.15.77:0/2250961361 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92e801a070 con 0x7f92f01614d0 2024-03-06T07:22:25.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.192+0000 7f92e6ffd640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92d406b500 0x7f92d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.192+0000 7f92e6ffd640 1 -- 172.21.15.77:0/2250961361 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92e8089520 con 0x7f92f01614d0 2024-03-06T07:22:25.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.192+0000 7f92f6515640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92d406b500 0x7f92d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.192+0000 7f92f6515640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92d406b500 0x7f92d406d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f92ec036000 tx=0x7f92ec030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.196+0000 7f92e6ffd640 1 -- 172.21.15.77:0/2250961361 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92f00d2740 con 0x7f92f01614d0 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.367+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f92f0074530 con 0x7f92f01614d0 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.367+0000 7f92e6ffd640 1 -- 172.21.15.77:0/2250961361 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f92e8057cc0 con 0x7f92f01614d0 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:25.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92d406b500 msgr2=0x7f92d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92d406b500 0x7f92d406d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f92ec036000 tx=0x7f92ec030000 comp rx=0 tx=0).stop 2024-03-06T07:22:25.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92f01614d0 msgr2=0x7f92f00b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92f01614d0 0x7f92f00b9650 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f92e800c420 tx=0x7f92e800c8e0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 shutdown_connections 2024-03-06T07:22:25.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92d406b500 0x7f92d406d9c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92f01614d0 0x7f92f00b9650 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f00d22b0 0x7f92f014ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 --2- 172.21.15.77:0/2250961361 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f00d1910 0x7f92f00ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 >> 172.21.15.77:0/2250961361 conn(0x7f92f00d3d40 msgr2=0x7f92f006c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 shutdown_connections 2024-03-06T07:22:25.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.369+0000 7f92f8779640 1 -- 172.21.15.77:0/2250961361 wait complete. 2024-03-06T07:22:25.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:25.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.483+0000 7f483bd01640 1 Processor -- start 2024-03-06T07:22:25.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.483+0000 7f483bd01640 1 -- start start 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f483bd01640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48341586e0 0x7f48340aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f483bd01640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4834158ac0 con 0x7f48340d01a0 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f483bd01640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483415b2a0 con 0x7f48340aef80 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f483bd01640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483415b540 con 0x7f48341586e0 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f483a29e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f48340d01a0 0x7f48340b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46632/0 (socket says 172.21.15.77:46632) 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f483a29e640 1 -- 172.21.15.77:0/3436006570 learned_addr learned my addr 172.21.15.77:0/3436006570 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f4839a9d640 1 --2- 172.21.15.77:0/3436006570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48341586e0 0x7f48340aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f4838a9b640 1 -- 172.21.15.77:0/3436006570 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142773696 0 0) 0x7f4834158ac0 con 0x7f48340d01a0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f4838a9b640 1 -- 172.21.15.77:0/3436006570 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4820003520 con 0x7f48340d01a0 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f4838a9b640 1 -- 172.21.15.77:0/3436006570 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37406610 0 0) 0x7f483415b2a0 con 0x7f48340aef80 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f4838a9b640 1 -- 172.21.15.77:0/3436006570 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4834158ac0 con 0x7f48340aef80 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f4839a9d640 1 -- 172.21.15.77:0/3436006570 >> v1:172.21.15.158:6789/0 conn(0x7f48340aef80 legacy=0x7f48340cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f4839a9d640 1 -- 172.21.15.77:0/3436006570 >> v1:172.21.15.77:6789/0 conn(0x7f48340d01a0 legacy=0x7f48340b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.484+0000 7f4839a9d640 1 -- 172.21.15.77:0/3436006570 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483415b820 con 0x7f48341586e0 2024-03-06T07:22:25.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.485+0000 7f4839a9d640 1 --2- 172.21.15.77:0/3436006570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48341586e0 0x7f48340aea40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f48300097d0 tx=0x7f483002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f19efe7fb4349955 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.485+0000 7f4838a9b640 1 -- 172.21.15.77:0/3436006570 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830035020 con 0x7f48341586e0 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.485+0000 7f4838a9b640 1 -- 172.21.15.77:0/3436006570 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4830031890 con 0x7f48341586e0 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.485+0000 7f4838a9b640 1 -- 172.21.15.77:0/3436006570 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830031c60 con 0x7f48341586e0 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.486+0000 7f483bd01640 1 -- 172.21.15.77:0/3436006570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48341586e0 msgr2=0x7f48340aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.486+0000 7f483bd01640 1 --2- 172.21.15.77:0/3436006570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48341586e0 0x7f48340aea40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f48300097d0 tx=0x7f483002b930 comp rx=0 tx=0).stop 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.486+0000 7f483bd01640 1 -- 172.21.15.77:0/3436006570 shutdown_connections 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.486+0000 7f483bd01640 1 --2- 172.21.15.77:0/3436006570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48341586e0 0x7f48340aea40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.486+0000 7f483bd01640 1 -- 172.21.15.77:0/3436006570 >> 172.21.15.77:0/3436006570 conn(0x7f483406b190 msgr2=0x7f483406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.486+0000 7f483bd01640 1 -- 172.21.15.77:0/3436006570 shutdown_connections 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.486+0000 7f483bd01640 1 -- 172.21.15.77:0/3436006570 wait complete. 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.487+0000 7f483bd01640 1 Processor -- start 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.487+0000 7f483bd01640 1 -- start start 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483bd01640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48340aef80 0x7f48341f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483bd01640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48340d01a0 0x7f48340aa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483bd01640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48341586e0 0x7f48340afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483bd01640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4834071350 con 0x7f48341586e0 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483bd01640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483415bde0 con 0x7f48340aef80 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483bd01640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48340714d0 con 0x7f48340d01a0 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48341586e0 0x7f48340afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48341586e0 0x7f48340afbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46390/0 (socket says 172.21.15.77:46390) 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 -- 172.21.15.77:0/2361939399 learned_addr learned my addr 172.21.15.77:0/2361939399 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f4839a9d640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48340aef80 0x7f48341f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483929c640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48340d01a0 0x7f48340aa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 -- 172.21.15.77:0/2361939399 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48340d01a0 msgr2=0x7f48340aa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48340d01a0 0x7f48340aa4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 -- 172.21.15.77:0/2361939399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48340aef80 msgr2=0x7f48341f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48340aef80 0x7f48341f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 -- 172.21.15.77:0/2361939399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48341f2080 con 0x7f48341586e0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f483a29e640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48341586e0 0x7f48340afbe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f482c00cbd0 tx=0x7f482c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f482a7fc640 1 -- 172.21.15.77:0/2361939399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c017070 con 0x7f48341586e0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.488+0000 7f482a7fc640 1 -- 172.21.15.77:0/2361939399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c00ace0 con 0x7f48341586e0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.489+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48340ab4b0 con 0x7f48341586e0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.489+0000 7f482a7fc640 1 -- 172.21.15.77:0/2361939399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c012620 con 0x7f48341586e0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.489+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48340ab1d0 con 0x7f48341586e0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.489+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48340681e0 con 0x7f48341586e0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.490+0000 7f482a7fc640 1 -- 172.21.15.77:0/2361939399 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f482c01a070 con 0x7f48341586e0 2024-03-06T07:22:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.490+0000 7f482a7fc640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f481806b4d0 0x7f481806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.490+0000 7f482a7fc640 1 -- 172.21.15.77:0/2361939399 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f482c08a190 con 0x7f48341586e0 2024-03-06T07:22:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.490+0000 7f4839a9d640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f481806b4d0 0x7f481806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.491+0000 7f4839a9d640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f481806b4d0 0x7f481806d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4830036000 tx=0x7f483003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.495+0000 7f482a7fc640 1 -- 172.21.15.77:0/2361939399 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48340681e0 con 0x7f48341586e0 2024-03-06T07:22:25.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.666+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f48340732b0 con 0x7f48341586e0 2024-03-06T07:22:25.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.666+0000 7f482a7fc640 1 -- 172.21.15.77:0/2361939399 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f482c0589c0 con 0x7f48341586e0 2024-03-06T07:22:25.668 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f481806b4d0 msgr2=0x7f481806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f481806b4d0 0x7f481806d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4830036000 tx=0x7f483003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48341586e0 msgr2=0x7f48340afbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48341586e0 0x7f48340afbe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f482c00cbd0 tx=0x7f482c00a430 comp rx=0 tx=0).stop 2024-03-06T07:22:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 shutdown_connections 2024-03-06T07:22:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f481806b4d0 0x7f481806d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48341586e0 0x7f48340afbe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48340d01a0 0x7f48340aa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 --2- 172.21.15.77:0/2361939399 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48340aef80 0x7f48341f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 >> 172.21.15.77:0/2361939399 conn(0x7f483406b190 msgr2=0x7f48340cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 shutdown_connections 2024-03-06T07:22:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.668+0000 7f483bd01640 1 -- 172.21.15.77:0/2361939399 wait complete. 2024-03-06T07:22:25.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:25.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.781+0000 7fcace1ee640 1 Processor -- start 2024-03-06T07:22:25.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.781+0000 7fcace1ee640 1 -- start start 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcace1ee640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac80b4970 0x7fcac806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcace1ee640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcac806d5c0 0x7fcac80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcace1ee640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac815ff90 con 0x7fcac815bb40 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcace1ee640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac80b4dd0 con 0x7fcac806d5c0 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcace1ee640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac806c910 con 0x7fcac80b4970 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcac7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcac806d5c0 0x7fcac80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcac77fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcac815bb40 0x7fcac815f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46634/0 (socket says 172.21.15.77:46634) 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcac77fe640 1 -- 172.21.15.77:0/1297858661 learned_addr learned my addr 172.21.15.77:0/1297858661 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.782+0000 7fcac6ffd640 1 --2- 172.21.15.77:0/1297858661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac80b4970 0x7fcac806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.783+0000 7fcac7fff640 1 -- 172.21.15.77:0/1297858661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac80b4970 msgr2=0x7fcac806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.783+0000 7fcac7fff640 1 --2- 172.21.15.77:0/1297858661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac80b4970 0x7fcac806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.783+0000 7fcac7fff640 1 -- 172.21.15.77:0/1297858661 >> v1:172.21.15.77:6789/0 conn(0x7fcac815bb40 legacy=0x7fcac815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.783+0000 7fcac7fff640 1 -- 172.21.15.77:0/1297858661 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8160270 con 0x7fcac806d5c0 2024-03-06T07:22:25.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.783+0000 7fcac67fc640 1 -- 172.21.15.77:0/1297858661 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973354581 0 0) 0x7fcac815ff90 con 0x7fcac815bb40 2024-03-06T07:22:25.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.783+0000 7fcac7fff640 1 --2- 172.21.15.77:0/1297858661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac806d5c0 0x7fcac80cfa90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009d90 tx=0x7fcac002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5ce314390d93a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcac67fc640 1 -- 172.21.15.77:0/1297858661 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0039070 con 0x7fcac806d5c0 2024-03-06T07:22:25.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcac67fc640 1 -- 172.21.15.77:0/1297858661 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac002cc50 con 0x7fcac806d5c0 2024-03-06T07:22:25.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcac67fc640 1 -- 172.21.15.77:0/1297858661 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0034440 con 0x7fcac806d5c0 2024-03-06T07:22:25.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcace1ee640 1 -- 172.21.15.77:0/1297858661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac806d5c0 msgr2=0x7fcac80cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcace1ee640 1 --2- 172.21.15.77:0/1297858661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac806d5c0 0x7fcac80cfa90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009d90 tx=0x7fcac002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcace1ee640 1 -- 172.21.15.77:0/1297858661 shutdown_connections 2024-03-06T07:22:25.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcace1ee640 1 --2- 172.21.15.77:0/1297858661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac806d5c0 0x7fcac80cfa90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcace1ee640 1 --2- 172.21.15.77:0/1297858661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac80b4970 0x7fcac806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcace1ee640 1 -- 172.21.15.77:0/1297858661 >> 172.21.15.77:0/1297858661 conn(0x7fcac806b190 msgr2=0x7fcac806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcace1ee640 1 -- 172.21.15.77:0/1297858661 shutdown_connections 2024-03-06T07:22:25.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.784+0000 7fcace1ee640 1 -- 172.21.15.77:0/1297858661 wait complete. 2024-03-06T07:22:25.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.785+0000 7fcace1ee640 1 Processor -- start 2024-03-06T07:22:25.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.785+0000 7fcace1ee640 1 -- start start 2024-03-06T07:22:25.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcace1ee640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcac806d5c0 0x7fcac8151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcace1ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac80b4970 0x7fcac8153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcace1ee640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac815bb40 0x7fcac80b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcace1ee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac815bf20 con 0x7fcac80b4970 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcace1ee640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac806cb60 con 0x7fcac806d5c0 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcace1ee640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac815c0a0 con 0x7fcac815bb40 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcac6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac80b4970 0x7fcac8153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcac6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac80b4970 0x7fcac8153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46414/0 (socket says 172.21.15.77:46414) 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcac6ffd640 1 -- 172.21.15.77:0/804102868 learned_addr learned my addr 172.21.15.77:0/804102868 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcac77fe640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcac806d5c0 0x7fcac8151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.786+0000 7fcac7fff640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac815bb40 0x7fcac80b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcac6ffd640 1 -- 172.21.15.77:0/804102868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac815bb40 msgr2=0x7fcac80b0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcac6ffd640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac815bb40 0x7fcac80b0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcac6ffd640 1 -- 172.21.15.77:0/804102868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcac806d5c0 msgr2=0x7fcac8151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcac6ffd640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcac806d5c0 0x7fcac8151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcac6ffd640 1 -- 172.21.15.77:0/804102868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8150560 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcac6ffd640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac80b4970 0x7fcac8153460 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00e9e0 tx=0x7fcabc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcaaffff640 1 -- 172.21.15.77:0/804102868 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc017070 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcaaffff640 1 -- 172.21.15.77:0/804102868 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcabc00c9d0 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac8150840 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcaaffff640 1 -- 172.21.15.77:0/804102868 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc00cdc0 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.787+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac80b0f10 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.788+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac80732b0 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.788+0000 7fcaaffff640 1 -- 172.21.15.77:0/804102868 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcabc01a070 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.789+0000 7fcaaffff640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.789+0000 7fcaaffff640 1 -- 172.21.15.77:0/804102868 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcabc08a590 con 0x7fcac80b4970 2024-03-06T07:22:25.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.789+0000 7fcac77fe640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.789+0000 7fcac77fe640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcab406b4d0 0x7fcab406d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcab8007fd0 tx=0x7fcab8007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.793+0000 7fcaaffff640 1 -- 172.21.15.77:0/804102868 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcabc058dc0 con 0x7fcac80b4970 2024-03-06T07:22:25.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.966+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcac8068ff0 con 0x7fcac80b4970 2024-03-06T07:22:25.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.966+0000 7fcaaffff640 1 -- 172.21.15.77:0/804102868 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcabc05ce60 con 0x7fcac80b4970 2024-03-06T07:22:25.968 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcab406b4d0 msgr2=0x7fcab406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcab406b4d0 0x7fcab406d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcab8007fd0 tx=0x7fcab8007de0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac80b4970 msgr2=0x7fcac8153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac80b4970 0x7fcac8153460 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00e9e0 tx=0x7fcabc00eea0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 shutdown_connections 2024-03-06T07:22:25.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac80b4970 0x7fcac8153460 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcac815bb40 0x7fcac80b0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 --2- 172.21.15.77:0/804102868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcac806d5c0 0x7fcac8151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 >> 172.21.15.77:0/804102868 conn(0x7fcac806b190 msgr2=0x7fcac8159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 shutdown_connections 2024-03-06T07:22:25.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:25.968+0000 7fcace1ee640 1 -- 172.21.15.77:0/804102868 wait complete. 2024-03-06T07:22:25.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.080+0000 7f45a6f56640 1 Processor -- start 2024-03-06T07:22:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.080+0000 7f45a6f56640 1 -- start start 2024-03-06T07:22:26.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f45a6f56640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a006c6d0 0x7f45a015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f45a6f56640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00cd450 con 0x7f45a0066d90 2024-03-06T07:22:26.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f45a6f56640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00cd730 con 0x7f45a00b4f70 2024-03-06T07:22:26.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f45a6f56640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a015e9f0 con 0x7f45a006c6d0 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f45a54f3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f45a0066d90 0x7f45a00ccb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46638/0 (socket says 172.21.15.77:46638) 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f45a54f3640 1 -- 172.21.15.77:0/450892707 learned_addr learned my addr 172.21.15.77:0/450892707 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f459ffff640 1 --2- 172.21.15.77:0/450892707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a006c6d0 0x7f45a015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f459f7fe640 1 -- 172.21.15.77:0/450892707 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781815132 0 0) 0x7f45a00cd450 con 0x7f45a0066d90 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f459f7fe640 1 -- 172.21.15.77:0/450892707 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4588003520 con 0x7f45a0066d90 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f459f7fe640 1 -- 172.21.15.77:0/450892707 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610298730 0 0) 0x7f45a00cd730 con 0x7f45a00b4f70 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f459f7fe640 1 -- 172.21.15.77:0/450892707 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a00cd450 con 0x7f45a00b4f70 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f459ffff640 1 -- 172.21.15.77:0/450892707 >> v1:172.21.15.158:6789/0 conn(0x7f45a00b4f70 legacy=0x7f45a00b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.081+0000 7f459ffff640 1 -- 172.21.15.77:0/450892707 >> v1:172.21.15.77:6789/0 conn(0x7f45a0066d90 legacy=0x7f45a00ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.082+0000 7f459ffff640 1 -- 172.21.15.77:0/450892707 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a015e680 con 0x7f45a006c6d0 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.082+0000 7f459ffff640 1 --2- 172.21.15.77:0/450892707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a006c6d0 0x7f45a015ee80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f45940097d0 tx=0x7f459402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df6e33cf7a619e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.082+0000 7f459f7fe640 1 -- 172.21.15.77:0/450892707 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594035020 con 0x7f45a006c6d0 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.082+0000 7f459f7fe640 1 -- 172.21.15.77:0/450892707 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4594031890 con 0x7f45a006c6d0 2024-03-06T07:22:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.082+0000 7f459f7fe640 1 -- 172.21.15.77:0/450892707 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594031c40 con 0x7f45a006c6d0 2024-03-06T07:22:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.083+0000 7f45a6f56640 1 -- 172.21.15.77:0/450892707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a006c6d0 msgr2=0x7f45a015ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.083+0000 7f45a6f56640 1 --2- 172.21.15.77:0/450892707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a006c6d0 0x7f45a015ee80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f45940097d0 tx=0x7f459402b930 comp rx=0 tx=0).stop 2024-03-06T07:22:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.083+0000 7f45a6f56640 1 -- 172.21.15.77:0/450892707 shutdown_connections 2024-03-06T07:22:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.083+0000 7f45a6f56640 1 --2- 172.21.15.77:0/450892707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a006c6d0 0x7f45a015ee80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.083+0000 7f45a6f56640 1 -- 172.21.15.77:0/450892707 >> 172.21.15.77:0/450892707 conn(0x7f45a00724d0 msgr2=0x7f45a00728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.083+0000 7f45a6f56640 1 -- 172.21.15.77:0/450892707 shutdown_connections 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.083+0000 7f45a6f56640 1 -- 172.21.15.77:0/450892707 wait complete. 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.084+0000 7f45a6f56640 1 Processor -- start 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.084+0000 7f45a6f56640 1 -- start start 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f45a6f56640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a0066d90 0x7f45a0150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f45a6f56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45a00b4f70 0x7f45a00cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f45a6f56640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45a00cf9f0 0x7f45a015d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f45a6f56640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0161ad0 con 0x7f45a00b4f70 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f45a6f56640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00cda10 con 0x7f45a00cf9f0 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f45a6f56640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0161c50 con 0x7f45a0066d90 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f459ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45a00b4f70 0x7f45a00cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f459ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45a00b4f70 0x7f45a00cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46420/0 (socket says 172.21.15.77:46420) 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f459ffff640 1 -- 172.21.15.77:0/2888188167 learned_addr learned my addr 172.21.15.77:0/2888188167 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f45a4cf2640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a0066d90 0x7f45a0150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f45a54f3640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45a00cf9f0 0x7f45a015d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f459ffff640 1 -- 172.21.15.77:0/2888188167 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a0066d90 msgr2=0x7f45a0150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f459ffff640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a0066d90 0x7f45a0150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f459ffff640 1 -- 172.21.15.77:0/2888188167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45a00cf9f0 msgr2=0x7f45a015d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f459ffff640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45a00cf9f0 0x7f45a015d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.085+0000 7f459ffff640 1 -- 172.21.15.77:0/2888188167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a00cfe90 con 0x7f45a00b4f70 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.086+0000 7f459ffff640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45a00b4f70 0x7f45a00cf4b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f45940098f0 tx=0x7f45940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.086+0000 7f459d7fa640 1 -- 172.21.15.77:0/2888188167 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594035040 con 0x7f45a00b4f70 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.086+0000 7f459d7fa640 1 -- 172.21.15.77:0/2888188167 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4594030b10 con 0x7f45a00b4f70 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.086+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a0157e70 con 0x7f45a00b4f70 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.086+0000 7f459d7fa640 1 -- 172.21.15.77:0/2888188167 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594044430 con 0x7f45a00b4f70 2024-03-06T07:22:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.086+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a0158430 con 0x7f45a00b4f70 2024-03-06T07:22:26.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.087+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a004f330 con 0x7f45a00b4f70 2024-03-06T07:22:26.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.087+0000 7f459d7fa640 1 -- 172.21.15.77:0/2888188167 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4594038020 con 0x7f45a00b4f70 2024-03-06T07:22:26.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.087+0000 7f459d7fa640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f458c06b4d0 0x7f458c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.088+0000 7f459d7fa640 1 -- 172.21.15.77:0/2888188167 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45940ac5e0 con 0x7f45a00b4f70 2024-03-06T07:22:26.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.088+0000 7f45a4cf2640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f458c06b4d0 0x7f458c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.088+0000 7f45a4cf2640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f458c06b4d0 0x7f458c06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4590007fd0 tx=0x7f4590009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.092+0000 7f459d7fa640 1 -- 172.21.15.77:0/2888188167 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f459402d040 con 0x7f45a00b4f70 2024-03-06T07:22:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.266+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45a00cc300 con 0x7f45a00b4f70 2024-03-06T07:22:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.267+0000 7f459d7fa640 1 -- 172.21.15.77:0/2888188167 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f459407ae10 con 0x7f45a00b4f70 2024-03-06T07:22:26.269 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:26.269 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:26.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f458c06b4d0 msgr2=0x7f458c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f458c06b4d0 0x7f458c06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4590007fd0 tx=0x7f4590009040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45a00b4f70 msgr2=0x7f45a00cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45a00b4f70 0x7f45a00cf4b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f45940098f0 tx=0x7f45940046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 shutdown_connections 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f458c06b4d0 0x7f458c06d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45a00b4f70 0x7f45a00cf4b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45a00cf9f0 0x7f45a015d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 --2- 172.21.15.77:0/2888188167 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45a0066d90 0x7f45a0150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 >> 172.21.15.77:0/2888188167 conn(0x7f45a00724d0 msgr2=0x7f45a00ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 shutdown_connections 2024-03-06T07:22:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.269+0000 7f45a6f56640 1 -- 172.21.15.77:0/2888188167 wait complete. 2024-03-06T07:22:26.282 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:26.283 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:26.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.384+0000 7fb957df7640 1 Processor -- start 2024-03-06T07:22:26.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.384+0000 7fb957df7640 1 -- start start 2024-03-06T07:22:26.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb957df7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9501586e0 0x7fb9500aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb957df7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb9500aef80 0x7fb9500cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb957df7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb950158ac0 con 0x7fb9500cffd0 2024-03-06T07:22:26.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb957df7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95015b2a0 con 0x7fb9501586e0 2024-03-06T07:22:26.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb957df7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95015b540 con 0x7fb9500aef80 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb955b93640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9501586e0 0x7fb9500aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb956394640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb9500cffd0 0x7fb95006da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46646/0 (socket says 172.21.15.77:46646) 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb956394640 1 -- 172.21.15.77:0/1490052687 learned_addr learned my addr 172.21.15.77:0/1490052687 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb955392640 1 --2- 172.21.15.77:0/1490052687 >> v2:172.21.15.158:3300/0 conn(0x7fb9500aef80 0x7fb9500cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb955b93640 1 -- 172.21.15.77:0/1490052687 >> v2:172.21.15.158:3300/0 conn(0x7fb9500aef80 msgr2=0x7fb9500cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.385+0000 7fb955b93640 1 --2- 172.21.15.77:0/1490052687 >> v2:172.21.15.158:3300/0 conn(0x7fb9500aef80 0x7fb9500cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.386+0000 7fb955b93640 1 -- 172.21.15.77:0/1490052687 >> v1:172.21.15.77:6789/0 conn(0x7fb9500cffd0 legacy=0x7fb95006da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.386+0000 7fb954b91640 1 -- 172.21.15.77:0/1490052687 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981967723 0 0) 0x7fb950158ac0 con 0x7fb9500cffd0 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.386+0000 7fb955b93640 1 -- 172.21.15.77:0/1490052687 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95015b820 con 0x7fb9501586e0 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.386+0000 7fb955b93640 1 --2- 172.21.15.77:0/1490052687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9501586e0 0x7fb9500aea40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb950072e30 tx=0x7fb94c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc7dcb3f7b843fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.386+0000 7fb954b91640 1 -- 172.21.15.77:0/1490052687 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c039070 con 0x7fb9501586e0 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.386+0000 7fb954b91640 1 -- 172.21.15.77:0/1490052687 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c02cc80 con 0x7fb9501586e0 2024-03-06T07:22:26.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.386+0000 7fb954b91640 1 -- 172.21.15.77:0/1490052687 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c034660 con 0x7fb9501586e0 2024-03-06T07:22:26.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.387+0000 7fb957df7640 1 -- 172.21.15.77:0/1490052687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9501586e0 msgr2=0x7fb9500aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.387+0000 7fb957df7640 1 --2- 172.21.15.77:0/1490052687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9501586e0 0x7fb9500aea40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb950072e30 tx=0x7fb94c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.387+0000 7fb957df7640 1 -- 172.21.15.77:0/1490052687 shutdown_connections 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.387+0000 7fb957df7640 1 --2- 172.21.15.77:0/1490052687 >> v2:172.21.15.158:3300/0 conn(0x7fb9500aef80 0x7fb9500cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.387+0000 7fb957df7640 1 --2- 172.21.15.77:0/1490052687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9501586e0 0x7fb9500aea40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.387+0000 7fb957df7640 1 -- 172.21.15.77:0/1490052687 >> 172.21.15.77:0/1490052687 conn(0x7fb95006b190 msgr2=0x7fb95006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.387+0000 7fb957df7640 1 -- 172.21.15.77:0/1490052687 shutdown_connections 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.387+0000 7fb957df7640 1 -- 172.21.15.77:0/1490052687 wait complete. 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.388+0000 7fb957df7640 1 Processor -- start 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.388+0000 7fb957df7640 1 -- start start 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb957df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9500aef80 0x7fb950153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb957df7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9500cffd0 0x7fb9500aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb957df7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9501586e0 0x7fb9500ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb957df7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb950071350 con 0x7fb9500aef80 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb957df7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015bde0 con 0x7fb9501586e0 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb957df7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500714d0 con 0x7fb9500cffd0 2024-03-06T07:22:26.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955b93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9500aef80 0x7fb950153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955b93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9500aef80 0x7fb950153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46428/0 (socket says 172.21.15.77:46428) 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955b93640 1 -- 172.21.15.77:0/2393538178 learned_addr learned my addr 172.21.15.77:0/2393538178 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955392640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9500cffd0 0x7fb9500aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb956394640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9501586e0 0x7fb9500ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955b93640 1 -- 172.21.15.77:0/2393538178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9500cffd0 msgr2=0x7fb9500aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955b93640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9500cffd0 0x7fb9500aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955b93640 1 -- 172.21.15.77:0/2393538178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9501586e0 msgr2=0x7fb9500ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955b93640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9501586e0 0x7fb9500ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.389+0000 7fb955b93640 1 -- 172.21.15.77:0/2393538178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9500ab550 con 0x7fb9500aef80 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.390+0000 7fb955b93640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9500aef80 0x7fb950153880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb94c037040 tx=0x7fb94c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.390+0000 7fb9467fc640 1 -- 172.21.15.77:0/2393538178 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c039050 con 0x7fb9500aef80 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.390+0000 7fb9467fc640 1 -- 172.21.15.77:0/2393538178 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c03a040 con 0x7fb9500aef80 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.390+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb950153dc0 con 0x7fb9500aef80 2024-03-06T07:22:26.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.390+0000 7fb9467fc640 1 -- 172.21.15.77:0/2393538178 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c003b90 con 0x7fb9500aef80 2024-03-06T07:22:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.390+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb950155a80 con 0x7fb9500aef80 2024-03-06T07:22:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.391+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9500b5b60 con 0x7fb9500aef80 2024-03-06T07:22:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.391+0000 7fb9467fc640 1 -- 172.21.15.77:0/2393538178 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb94c03c610 con 0x7fb9500aef80 2024-03-06T07:22:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.392+0000 7fb9467fc640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb93406b4d0 0x7fb93406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.392+0000 7fb955392640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb93406b4d0 0x7fb93406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.392+0000 7fb9467fc640 1 -- 172.21.15.77:0/2393538178 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb94c0adb60 con 0x7fb9500aef80 2024-03-06T07:22:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.392+0000 7fb955392640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb93406b4d0 0x7fb93406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb950157610 tx=0x7fb940009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.396+0000 7fb9467fc640 1 -- 172.21.15.77:0/2393538178 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9500b5b60 con 0x7fb9500aef80 2024-03-06T07:22:26.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.567+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb950159800 con 0x7fb9500aef80 2024-03-06T07:22:26.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.567+0000 7fb9467fc640 1 -- 172.21.15.77:0/2393538178 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb94c07c420 con 0x7fb9500aef80 2024-03-06T07:22:26.661 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb93406b4d0 msgr2=0x7fb93406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb93406b4d0 0x7fb93406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb950157610 tx=0x7fb940009190 comp rx=0 tx=0).stop 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9500aef80 msgr2=0x7fb950153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9500aef80 0x7fb950153880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb94c037040 tx=0x7fb94c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 shutdown_connections 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9501586e0 0x7fb9500ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9500cffd0 0x7fb9500aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb93406b4d0 0x7fb93406d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 --2- 172.21.15.77:0/2393538178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9500aef80 0x7fb950153880 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 >> 172.21.15.77:0/2393538178 conn(0x7fb95006b190 msgr2=0x7fb95015a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 shutdown_connections 2024-03-06T07:22:26.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.569+0000 7fb957df7640 1 -- 172.21.15.77:0/2393538178 wait complete. 2024-03-06T07:22:26.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.765+0000 7f01dcfeb640 1 Processor -- start 2024-03-06T07:22:26.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.765+0000 7f01dcfeb640 1 -- start start 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01dcfeb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01dcfeb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f01d815ec40 0x7f01d8071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01dcfeb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d80cea90 con 0x7f01d815bbb0 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01dcfeb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d8161970 con 0x7f01d815ec40 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01dcfeb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d81624f0 con 0x7f01d806b6c0 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d6d9d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f01d815ec40 0x7f01d8071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d659c640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f01d815bbb0 0x7f01d806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:59890/0 (socket says 172.21.15.77:59890) 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01cffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d659c640 1 -- 172.21.15.77:0/1735879433 learned_addr learned my addr 172.21.15.77:0/1735879433 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d5d9b640 1 -- 172.21.15.77:0/1735879433 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141952455 0 0) 0x7f01c8001110 con 0x7f01d815bbb0 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d5d9b640 1 -- 172.21.15.77:0/1735879433 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c4003520 con 0x7f01d815bbb0 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d6d9d640 1 -- 172.21.15.77:0/1735879433 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 msgr2=0x7f01d815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d6d9d640 1 --2- 172.21.15.77:0/1735879433 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d6d9d640 1 -- 172.21.15.77:0/1735879433 >> v1:172.21.15.158:6789/0 conn(0x7f01d815bbb0 legacy=0x7f01d806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01d6d9d640 1 -- 172.21.15.77:0/1735879433 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8161bc0 con 0x7f01d815ec40 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.766+0000 7f01cffff640 1 --2- 172.21.15.77:0/1735879433 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.767+0000 7f01d6d9d640 1 --2- 172.21.15.77:0/1735879433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 0x7f01d8071ca0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009d90 tx=0x7f01d002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cd563c167bd3670 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.767+0000 7f01d5d9b640 1 -- 172.21.15.77:0/1735879433 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d0039070 con 0x7f01d815ec40 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.767+0000 7f01d5d9b640 1 -- 172.21.15.77:0/1735879433 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d002cc50 con 0x7f01d815ec40 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.767+0000 7f01d5d9b640 1 -- 172.21.15.77:0/1735879433 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d0034440 con 0x7f01d815ec40 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.768+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1735879433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 msgr2=0x7f01d8071ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.768+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1735879433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 0x7f01d8071ca0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009d90 tx=0x7f01d002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.768+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1735879433 shutdown_connections 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.768+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1735879433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 0x7f01d8071ca0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.768+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1735879433 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.768+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1735879433 >> 172.21.15.77:0/1735879433 conn(0x7f01d80b2660 msgr2=0x7f01d80b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.768+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1735879433 shutdown_connections 2024-03-06T07:22:26.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.768+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1735879433 wait complete. 2024-03-06T07:22:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.769+0000 7f01dcfeb640 1 Processor -- start 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.769+0000 7f01dcfeb640 1 -- start start 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01dcfeb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d8160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01dcfeb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 0x7f01d8157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01dcfeb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d814fe80 0x7f01d81502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01dcfeb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d815f0f0 con 0x7f01d814fe80 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01dcfeb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d8162180 con 0x7f01d815ec40 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01dcfeb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80721e0 con 0x7f01d806b6c0 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d814fe80 0x7f01d81502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d814fe80 0x7f01d81502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46432/0 (socket says 172.21.15.77:46432) 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 -- 172.21.15.77:0/1519652580 learned_addr learned my addr 172.21.15.77:0/1519652580 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d659c640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d8160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01cffff640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 0x7f01d8157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 -- 172.21.15.77:0/1519652580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 msgr2=0x7f01d8160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d8160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 -- 172.21.15.77:0/1519652580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 msgr2=0x7f01d8157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 0x7f01d8157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 -- 172.21.15.77:0/1519652580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8150820 con 0x7f01d814fe80 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01d6d9d640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d814fe80 0x7f01d81502e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009d90 tx=0x7f01d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.770+0000 7f01ceffd640 1 -- 172.21.15.77:0/1519652580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d0044020 con 0x7f01d814fe80 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.771+0000 7f01ceffd640 1 -- 172.21.15.77:0/1519652580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d002ca50 con 0x7f01d814fe80 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.771+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d80b5fb0 con 0x7f01d814fe80 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.771+0000 7f01ceffd640 1 -- 172.21.15.77:0/1519652580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d0039040 con 0x7f01d814fe80 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.771+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d815dad0 con 0x7f01d814fe80 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.772+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d80733b0 con 0x7f01d814fe80 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.772+0000 7f01ceffd640 1 -- 172.21.15.77:0/1519652580 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01d0047020 con 0x7f01d814fe80 2024-03-06T07:22:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.772+0000 7f01ceffd640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01b806b4d0 0x7f01b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.773+0000 7f01d659c640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01b806b4d0 0x7f01b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.773+0000 7f01ceffd640 1 -- 172.21.15.77:0/1519652580 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01d00ac910 con 0x7f01d814fe80 2024-03-06T07:22:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.773+0000 7f01d659c640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01b806b4d0 0x7f01b806d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f01c8005880 tx=0x7f01c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.777+0000 7f01ceffd640 1 -- 172.21.15.77:0/1519652580 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01d004e030 con 0x7f01d814fe80 2024-03-06T07:22:26.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.947+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01d8068ff0 con 0x7f01d814fe80 2024-03-06T07:22:26.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.947+0000 7f01ceffd640 1 -- 172.21.15.77:0/1519652580 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f01d8150820 con 0x7f01d814fe80 2024-03-06T07:22:26.949 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:26.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01b806b4d0 msgr2=0x7f01b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01b806b4d0 0x7f01b806d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f01c8005880 tx=0x7f01c8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d814fe80 msgr2=0x7f01d81502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d814fe80 0x7f01d81502e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009d90 tx=0x7f01d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:26.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 shutdown_connections 2024-03-06T07:22:26.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01b806b4d0 0x7f01b806d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01d814fe80 0x7f01d81502e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01d815ec40 0x7f01d8157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 --2- 172.21.15.77:0/1519652580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01d806b6c0 0x7f01d8160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 >> 172.21.15.77:0/1519652580 conn(0x7f01d80b2660 msgr2=0x7f01d81588c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 shutdown_connections 2024-03-06T07:22:26.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:26.949+0000 7f01dcfeb640 1 -- 172.21.15.77:0/1519652580 wait complete. 2024-03-06T07:22:26.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:27.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.064+0000 7f7d65b3a640 1 Processor -- start 2024-03-06T07:22:27.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.064+0000 7f7d65b3a640 1 -- start start 2024-03-06T07:22:27.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.064+0000 7f7d65b3a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d6006fd10 0x7f7d600cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.064+0000 7f7d65b3a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d600d0af0 0x7f7d600ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.064+0000 7f7d65b3a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d600cd1e0 con 0x7f7d6015c050 2024-03-06T07:22:27.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.064+0000 7f7d65b3a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d600cd4b0 con 0x7f7d600d0af0 2024-03-06T07:22:27.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.064+0000 7f7d65b3a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d600cd790 con 0x7f7d6006fd10 2024-03-06T07:22:27.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d600d0af0 0x7f7d600ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7d6015c050 0x7f7d6006f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46662/0 (socket says 172.21.15.77:46662) 2024-03-06T07:22:27.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5f7fe640 1 -- 172.21.15.77:0/1472215272 learned_addr learned my addr 172.21.15.77:0/1472215272 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:27.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5effd640 1 --2- 172.21.15.77:0/1472215272 >> v2:172.21.15.158:3300/0 conn(0x7f7d6006fd10 0x7f7d600cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5ffff640 1 -- 172.21.15.77:0/1472215272 >> v2:172.21.15.158:3300/0 conn(0x7f7d6006fd10 msgr2=0x7f7d600cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5ffff640 1 --2- 172.21.15.77:0/1472215272 >> v2:172.21.15.158:3300/0 conn(0x7f7d6006fd10 0x7f7d600cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5ffff640 1 -- 172.21.15.77:0/1472215272 >> v1:172.21.15.77:6789/0 conn(0x7f7d6015c050 legacy=0x7f7d6006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5ffff640 1 -- 172.21.15.77:0/1472215272 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d600cda70 con 0x7f7d600d0af0 2024-03-06T07:22:27.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.065+0000 7f7d5e7fc640 1 -- 172.21.15.77:0/1472215272 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529584006 0 0) 0x7f7d600cd1e0 con 0x7f7d6015c050 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d5ffff640 1 --2- 172.21.15.77:0/1472215272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d600d0af0 0x7f7d600ccca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7d600b7dc0 tx=0x7f7d5802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9293064d1abb9b26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d5e7fc640 1 -- 172.21.15.77:0/1472215272 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d58039070 con 0x7f7d600d0af0 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d5e7fc640 1 -- 172.21.15.77:0/1472215272 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5802cbf0 con 0x7f7d600d0af0 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d5e7fc640 1 -- 172.21.15.77:0/1472215272 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d58034440 con 0x7f7d600d0af0 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d65b3a640 1 -- 172.21.15.77:0/1472215272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d600d0af0 msgr2=0x7f7d600ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/1472215272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d600d0af0 0x7f7d600ccca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7d600b7dc0 tx=0x7f7d5802be40 comp rx=0 tx=0).stop 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d65b3a640 1 -- 172.21.15.77:0/1472215272 shutdown_connections 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/1472215272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d600d0af0 0x7f7d600ccca0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/1472215272 >> v2:172.21.15.158:3300/0 conn(0x7f7d6006fd10 0x7f7d600cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d65b3a640 1 -- 172.21.15.77:0/1472215272 >> 172.21.15.77:0/1472215272 conn(0x7f7d600b3800 msgr2=0x7f7d600b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d65b3a640 1 -- 172.21.15.77:0/1472215272 shutdown_connections 2024-03-06T07:22:27.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.066+0000 7f7d65b3a640 1 -- 172.21.15.77:0/1472215272 wait complete. 2024-03-06T07:22:27.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.067+0000 7f7d65b3a640 1 Processor -- start 2024-03-06T07:22:27.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d65b3a640 1 -- start start 2024-03-06T07:22:27.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d65b3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d6006fd10 0x7f7d6013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d65b3a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d600d0af0 0x7f7d600b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d65b3a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d6015c050 0x7f7d60151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d65b3a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d60165c30 con 0x7f7d6006fd10 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d65b3a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d60165ab0 con 0x7f7d6015c050 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d65b3a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d60165db0 con 0x7f7d600d0af0 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d5f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d6006fd10 0x7f7d6013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d5f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d6006fd10 0x7f7d6013f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46446/0 (socket says 172.21.15.77:46446) 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.068+0000 7f7d5f7fe640 1 -- 172.21.15.77:0/2923871944 learned_addr learned my addr 172.21.15.77:0/2923871944 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5ffff640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d6015c050 0x7f7d60151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5effd640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d600d0af0 0x7f7d600b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5f7fe640 1 -- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d600d0af0 msgr2=0x7f7d600b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5f7fe640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d600d0af0 0x7f7d600b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5f7fe640 1 -- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d6015c050 msgr2=0x7f7d60151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5f7fe640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d6015c050 0x7f7d60151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5f7fe640 1 -- 172.21.15.77:0/2923871944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d600af760 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5effd640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d600d0af0 0x7f7d600b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d5f7fe640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d6006fd10 0x7f7d6013f4d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000c430 tx=0x7f7d5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d47fff640 1 -- 172.21.15.77:0/2923871944 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d500074e0 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d47fff640 1 -- 172.21.15.77:0/2923871944 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d500047e0 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d600afa40 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d47fff640 1 -- 172.21.15.77:0/2923871944 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d50004bd0 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.069+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d600b0000 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.070+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6009fe60 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.070+0000 7f7d47fff640 1 -- 172.21.15.77:0/2923871944 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d50011070 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.071+0000 7f7d47fff640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4c06b4d0 0x7f7d4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.071+0000 7f7d5effd640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4c06b4d0 0x7f7d4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.071+0000 7f7d47fff640 1 -- 172.21.15.77:0/2923871944 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d50089dd0 con 0x7f7d6006fd10 2024-03-06T07:22:27.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.071+0000 7f7d5effd640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4c06b4d0 0x7f7d4c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d6009fe00 tx=0x7f7d54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.077+0000 7f7d47fff640 1 -- 172.21.15.77:0/2923871944 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d50058c60 con 0x7f7d6006fd10 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.251+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d600b1f10 con 0x7f7d6006fd10 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.251+0000 7f7d47fff640 1 -- 172.21.15.77:0/2923871944 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d50058560 con 0x7f7d6006fd10 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:27.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4c06b4d0 msgr2=0x7f7d4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4c06b4d0 0x7f7d4c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d6009fe00 tx=0x7f7d54008040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d6006fd10 msgr2=0x7f7d6013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d6006fd10 0x7f7d6013f4d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000c430 tx=0x7f7d5000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 shutdown_connections 2024-03-06T07:22:27.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d6015c050 0x7f7d60151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d600d0af0 0x7f7d600b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4c06b4d0 0x7f7d4c06d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 --2- 172.21.15.77:0/2923871944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d6006fd10 0x7f7d6013f4d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 >> 172.21.15.77:0/2923871944 conn(0x7f7d600b3800 msgr2=0x7f7d600b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 shutdown_connections 2024-03-06T07:22:27.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.253+0000 7f7d65b3a640 1 -- 172.21.15.77:0/2923871944 wait complete. 2024-03-06T07:22:27.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:27.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.365+0000 7f2999e66640 1 Processor -- start 2024-03-06T07:22:27.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.365+0000 7f2999e66640 1 -- start start 2024-03-06T07:22:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2999e66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299413dc10 0x7f29940b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2999e66640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f299415ef60 0x7f29940b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2999e66640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299415f580 con 0x7f299415bf30 2024-03-06T07:22:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2999e66640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29940cf3f0 con 0x7f299415ef60 2024-03-06T07:22:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2999e66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299415b210 con 0x7f299413dc10 2024-03-06T07:22:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2993fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f299415ef60 0x7f29940b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f29937fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299413dc10 0x7f29940b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2993fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f299415ef60 0x7f29940b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:34072/0 (socket says 172.21.15.77:34072) 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2993fff640 1 -- 172.21.15.77:0/88746776 learned_addr learned my addr 172.21.15.77:0/88746776 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2992ffd640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f299415bf30 0x7f299415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:59902/0 (socket says 172.21.15.77:59902) 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f29937fe640 1 -- 172.21.15.77:0/88746776 >> v2:172.21.15.158:3300/0 conn(0x7f299415ef60 msgr2=0x7f29940b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f29937fe640 1 --2- 172.21.15.77:0/88746776 >> v2:172.21.15.158:3300/0 conn(0x7f299415ef60 0x7f29940b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f29937fe640 1 -- 172.21.15.77:0/88746776 >> v1:172.21.15.158:6789/0 conn(0x7f299415bf30 legacy=0x7f299415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f29937fe640 1 -- 172.21.15.77:0/88746776 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299415b450 con 0x7f299413dc10 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.366+0000 7f2993fff640 1 --2- 172.21.15.77:0/88746776 >> v2:172.21.15.158:3300/0 conn(0x7f299415ef60 0x7f29940b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.367+0000 7f29937fe640 1 --2- 172.21.15.77:0/88746776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299413dc10 0x7f29940b2950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f29840097a0 tx=0x7f298402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0504f6b8fddb99e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.367+0000 7f29927fc640 1 -- 172.21.15.77:0/88746776 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29840315a0 con 0x7f299413dc10 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.367+0000 7f29927fc640 1 -- 172.21.15.77:0/88746776 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2984037070 con 0x7f299413dc10 2024-03-06T07:22:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.367+0000 7f29927fc640 1 -- 172.21.15.77:0/88746776 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984031960 con 0x7f299413dc10 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.368+0000 7f2999e66640 1 -- 172.21.15.77:0/88746776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299413dc10 msgr2=0x7f29940b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.368+0000 7f2999e66640 1 --2- 172.21.15.77:0/88746776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299413dc10 0x7f29940b2950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f29840097a0 tx=0x7f298402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.368+0000 7f2999e66640 1 -- 172.21.15.77:0/88746776 shutdown_connections 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.368+0000 7f2999e66640 1 --2- 172.21.15.77:0/88746776 >> v2:172.21.15.158:3300/0 conn(0x7f299415ef60 0x7f29940b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.368+0000 7f2999e66640 1 --2- 172.21.15.77:0/88746776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299413dc10 0x7f29940b2950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.368+0000 7f2999e66640 1 -- 172.21.15.77:0/88746776 >> 172.21.15.77:0/88746776 conn(0x7f29940cf8a0 msgr2=0x7f29940cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.368+0000 7f2999e66640 1 -- 172.21.15.77:0/88746776 shutdown_connections 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.368+0000 7f2999e66640 1 -- 172.21.15.77:0/88746776 wait complete. 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.369+0000 7f2999e66640 1 Processor -- start 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.369+0000 7f2999e66640 1 -- start start 2024-03-06T07:22:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f2999e66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f299413dc10 0x7f2994142850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f2999e66640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f299415bf30 0x7f2994141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f2999e66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299415ef60 0x7f299406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f2999e66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29940708d0 con 0x7f299413dc10 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f2999e66640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299413dff0 con 0x7f299415bf30 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f2999e66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2994070a50 con 0x7f299415ef60 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f29937fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f299413dc10 0x7f2994142850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f29937fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f299413dc10 0x7f2994142850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46454/0 (socket says 172.21.15.77:46454) 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f29937fe640 1 -- 172.21.15.77:0/1540169312 learned_addr learned my addr 172.21.15.77:0/1540169312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f2992ffd640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f299415bf30 0x7f2994141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.370+0000 7f2993fff640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299415ef60 0x7f299406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f29937fe640 1 -- 172.21.15.77:0/1540169312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299415ef60 msgr2=0x7f299406d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f29937fe640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299415ef60 0x7f299406d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f29937fe640 1 -- 172.21.15.77:0/1540169312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f299415bf30 msgr2=0x7f2994141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f29937fe640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f299415bf30 0x7f2994141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f29937fe640 1 -- 172.21.15.77:0/1540169312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2994142d90 con 0x7f299413dc10 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f29937fe640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f299413dc10 0x7f2994142850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2984034040 tx=0x7f29840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f297bfff640 1 -- 172.21.15.77:0/1540169312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984032630 con 0x7f299413dc10 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f297bfff640 1 -- 172.21.15.77:0/1540169312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29840327f0 con 0x7f299413dc10 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299413e310 con 0x7f299413dc10 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f297bfff640 1 -- 172.21.15.77:0/1540169312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984032be0 con 0x7f299413dc10 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.371+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299413e8d0 con 0x7f299413dc10 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.372+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299404f330 con 0x7f299413dc10 2024-03-06T07:22:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.372+0000 7f297bfff640 1 -- 172.21.15.77:0/1540169312 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f298404e430 con 0x7f299413dc10 2024-03-06T07:22:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.373+0000 7f297bfff640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f298006b500 0x7f298006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.373+0000 7f297bfff640 1 -- 172.21.15.77:0/1540169312 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f298402f030 con 0x7f299413dc10 2024-03-06T07:22:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.373+0000 7f2992ffd640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f298006b500 0x7f298006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.373+0000 7f2992ffd640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f298006b500 0x7f298006d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2994139b10 tx=0x7f2988007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.377+0000 7f297bfff640 1 -- 172.21.15.77:0/1540169312 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2984040070 con 0x7f299413dc10 2024-03-06T07:22:27.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.550+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f299406eb00 con 0x7f299413dc10 2024-03-06T07:22:27.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.551+0000 7f297bfff640 1 -- 172.21.15.77:0/1540169312 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f298407a750 con 0x7f299413dc10 2024-03-06T07:22:27.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:27.553 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f298006b500 msgr2=0x7f298006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f298006b500 0x7f298006d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2994139b10 tx=0x7f2988007e80 comp rx=0 tx=0).stop 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f299413dc10 msgr2=0x7f2994142850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f299413dc10 0x7f2994142850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2984034040 tx=0x7f29840046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 shutdown_connections 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f299415ef60 0x7f299406d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f299415bf30 0x7f2994141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f298006b500 0x7f298006d9c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 --2- 172.21.15.77:0/1540169312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f299413dc10 0x7f2994142850 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 >> 172.21.15.77:0/1540169312 conn(0x7f29940cf8a0 msgr2=0x7f299415e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 shutdown_connections 2024-03-06T07:22:27.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.553+0000 7f2999e66640 1 -- 172.21.15.77:0/1540169312 wait complete. 2024-03-06T07:22:27.566 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:27.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.671+0000 7f4856234640 1 Processor -- start 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.671+0000 7f4856234640 1 -- start start 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.671+0000 7f4856234640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f48500d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.671+0000 7f4856234640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48501618b0 con 0x7f48500d1910 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.671+0000 7f4856234640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48500cf400 con 0x7f48500d2480 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.671+0000 7f4856234640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48500cf6a0 con 0x7f48501614d0 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f4847fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f48500d1910 0x7f48500d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46670/0 (socket says 172.21.15.77:46670) 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f4847fff640 1 -- 172.21.15.77:0/3802438541 learned_addr learned my addr 172.21.15.77:0/3802438541 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f484f7fe640 1 --2- 172.21.15.77:0/3802438541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f48500d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f484effd640 1 -- 172.21.15.77:0/3802438541 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377202835 0 0) 0x7f48500cf400 con 0x7f48500d2480 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f484effd640 1 -- 172.21.15.77:0/3802438541 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f483c003520 con 0x7f48500d2480 2024-03-06T07:22:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f484effd640 1 -- 172.21.15.77:0/3802438541 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755272762 0 0) 0x7f48501618b0 con 0x7f48500d1910 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f484effd640 1 -- 172.21.15.77:0/3802438541 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48500cf400 con 0x7f48500d1910 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f484f7fe640 1 -- 172.21.15.77:0/3802438541 >> v1:172.21.15.158:6789/0 conn(0x7f48500d2480 legacy=0x7f48500b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f484f7fe640 1 -- 172.21.15.77:0/3802438541 >> v1:172.21.15.77:6789/0 conn(0x7f48500d1910 legacy=0x7f48500d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.672+0000 7f484f7fe640 1 -- 172.21.15.77:0/3802438541 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48500cf980 con 0x7f48501614d0 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.673+0000 7f484f7fe640 1 --2- 172.21.15.77:0/3802438541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f48500d4860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f48400097d0 tx=0x7f484002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af3499a08fd44acd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.673+0000 7f484effd640 1 -- 172.21.15.77:0/3802438541 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4840035020 con 0x7f48501614d0 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.673+0000 7f484effd640 1 -- 172.21.15.77:0/3802438541 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4840031890 con 0x7f48501614d0 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.673+0000 7f484effd640 1 -- 172.21.15.77:0/3802438541 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4840031c60 con 0x7f48501614d0 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.674+0000 7f4856234640 1 -- 172.21.15.77:0/3802438541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 msgr2=0x7f48500d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.674+0000 7f4856234640 1 --2- 172.21.15.77:0/3802438541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f48500d4860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f48400097d0 tx=0x7f484002b930 comp rx=0 tx=0).stop 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.674+0000 7f4856234640 1 -- 172.21.15.77:0/3802438541 shutdown_connections 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.674+0000 7f4856234640 1 --2- 172.21.15.77:0/3802438541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f48500d4860 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.674+0000 7f4856234640 1 -- 172.21.15.77:0/3802438541 >> 172.21.15.77:0/3802438541 conn(0x7f48500d3d40 msgr2=0x7f48500d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.674+0000 7f4856234640 1 -- 172.21.15.77:0/3802438541 shutdown_connections 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.674+0000 7f4856234640 1 -- 172.21.15.77:0/3802438541 wait complete. 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.675+0000 7f4856234640 1 Processor -- start 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.675+0000 7f4856234640 1 -- start start 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f4856234640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48500d1910 0x7f48501542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f4856234640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48500d2480 0x7f485009f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f4856234640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f485009da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f4856234640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48500b1540 con 0x7f48500d1910 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f4856234640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48500b0910 con 0x7f48500d2480 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f4856234640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48500b16c0 con 0x7f48501614d0 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48500d1910 0x7f48501542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48500d1910 0x7f48501542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46464/0 (socket says 172.21.15.77:46464) 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484f7fe640 1 -- 172.21.15.77:0/1834540499 learned_addr learned my addr 172.21.15.77:0/1834540499 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f4847fff640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48500d2480 0x7f485009f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484ffff640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f485009da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484ffff640 1 -- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48500d2480 msgr2=0x7f485009f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484ffff640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48500d2480 0x7f485009f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484ffff640 1 -- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48500d1910 msgr2=0x7f48501542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484ffff640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48500d1910 0x7f48501542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484ffff640 1 -- 172.21.15.77:0/1834540499 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48500a03a0 con 0x7f48501614d0 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484f7fe640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48500d1910 0x7f48501542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484ffff640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f485009da90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4848002230 tx=0x7f484800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.676+0000 7f484cff9640 1 -- 172.21.15.77:0/1834540499 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848003b70 con 0x7f48501614d0 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.677+0000 7f484cff9640 1 -- 172.21.15.77:0/1834540499 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4848003d30 con 0x7f48501614d0 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.677+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48500a0680 con 0x7f48501614d0 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.677+0000 7f484cff9640 1 -- 172.21.15.77:0/1834540499 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848005310 con 0x7f48501614d0 2024-03-06T07:22:27.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.677+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48500a0c40 con 0x7f48501614d0 2024-03-06T07:22:27.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.678+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48500cea70 con 0x7f48501614d0 2024-03-06T07:22:27.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.678+0000 7f484cff9640 1 -- 172.21.15.77:0/1834540499 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48480056c0 con 0x7f48501614d0 2024-03-06T07:22:27.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.678+0000 7f484cff9640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f483006b4d0 0x7f483006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.678+0000 7f484cff9640 1 -- 172.21.15.77:0/1834540499 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f484808a1e0 con 0x7f48501614d0 2024-03-06T07:22:27.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.678+0000 7f484f7fe640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f483006b4d0 0x7f483006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.679+0000 7f484f7fe640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f483006b4d0 0x7f483006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4840009720 tx=0x7f484003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.683+0000 7f484cff9640 1 -- 172.21.15.77:0/1834540499 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4848058a10 con 0x7f48501614d0 2024-03-06T07:22:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.857+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4850068ff0 con 0x7f48501614d0 2024-03-06T07:22:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.857+0000 7f484cff9640 1 -- 172.21.15.77:0/1834540499 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f484805cab0 con 0x7f48501614d0 2024-03-06T07:22:27.860 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.859+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f483006b4d0 msgr2=0x7f483006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.859+0000 7f4856234640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f483006b4d0 0x7f483006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4840009720 tx=0x7f484003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.859+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 msgr2=0x7f485009da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.859+0000 7f4856234640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f485009da90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4848002230 tx=0x7f484800cb90 comp rx=0 tx=0).stop 2024-03-06T07:22:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.860+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 shutdown_connections 2024-03-06T07:22:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.860+0000 7f4856234640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48501614d0 0x7f485009da90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.860+0000 7f4856234640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48500d2480 0x7f485009f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.860+0000 7f4856234640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f483006b4d0 0x7f483006d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.860+0000 7f4856234640 1 --2- 172.21.15.77:0/1834540499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48500d1910 0x7f48501542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.860+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 >> 172.21.15.77:0/1834540499 conn(0x7f48500d3d40 msgr2=0x7f48500b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.860+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 shutdown_connections 2024-03-06T07:22:27.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.860+0000 7f4856234640 1 -- 172.21.15.77:0/1834540499 wait complete. 2024-03-06T07:22:27.873 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.974+0000 7fd47fedf640 1 Processor -- start 2024-03-06T07:22:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.974+0000 7fd47fedf640 1 -- start start 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47fedf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd4781614d0 0x7fd4780d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47fedf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd4780d1910 0x7fd4780d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47fedf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4781618b0 con 0x7fd4780d22e0 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47fedf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4780cf400 con 0x7fd4780d1910 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47fedf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4780d2710 con 0x7fd4781614d0 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47d47a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd4780d1910 0x7fd4780d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47e47c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd4780d22e0 0x7fd4780d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46676/0 (socket says 172.21.15.77:46676) 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47d47a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd4780d1910 0x7fd4780d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46480/0 (socket says 172.21.15.77:46480) 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47e47c640 1 -- 172.21.15.77:0/3779618973 learned_addr learned my addr 172.21.15.77:0/3779618973 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.975+0000 7fd47dc7b640 1 --2- 172.21.15.77:0/3779618973 >> v2:172.21.15.158:3300/0 conn(0x7fd4781614d0 0x7fd4780d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.976+0000 7fd47d47a640 1 -- 172.21.15.77:0/3779618973 >> v2:172.21.15.158:3300/0 conn(0x7fd4781614d0 msgr2=0x7fd4780d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.976+0000 7fd47d47a640 1 --2- 172.21.15.77:0/3779618973 >> v2:172.21.15.158:3300/0 conn(0x7fd4781614d0 0x7fd4780d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.976+0000 7fd47d47a640 1 -- 172.21.15.77:0/3779618973 >> v1:172.21.15.77:6789/0 conn(0x7fd4780d22e0 legacy=0x7fd4780d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.976+0000 7fd47d47a640 1 -- 172.21.15.77:0/3779618973 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4780cf6a0 con 0x7fd4780d1910 2024-03-06T07:22:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.976+0000 7fd47cc79640 1 -- 172.21.15.77:0/3779618973 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778819987 0 0) 0x7fd4781618b0 con 0x7fd4780d22e0 2024-03-06T07:22:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.976+0000 7fd47d47a640 1 --2- 172.21.15.77:0/3779618973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d1910 0x7fd4780d1da0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd47808c470 tx=0x7fd46802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb85f4e10143c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.976+0000 7fd47cc79640 1 -- 172.21.15.77:0/3779618973 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468035020 con 0x7fd4780d1910 2024-03-06T07:22:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47cc79640 1 -- 172.21.15.77:0/3779618973 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd468031830 con 0x7fd4780d1910 2024-03-06T07:22:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47cc79640 1 -- 172.21.15.77:0/3779618973 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468031c00 con 0x7fd4780d1910 2024-03-06T07:22:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47fedf640 1 -- 172.21.15.77:0/3779618973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d1910 msgr2=0x7fd4780d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3779618973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d1910 0x7fd4780d1da0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd47808c470 tx=0x7fd46802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47fedf640 1 -- 172.21.15.77:0/3779618973 shutdown_connections 2024-03-06T07:22:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3779618973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d1910 0x7fd4780d1da0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3779618973 >> v2:172.21.15.158:3300/0 conn(0x7fd4781614d0 0x7fd4780d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47fedf640 1 -- 172.21.15.77:0/3779618973 >> 172.21.15.77:0/3779618973 conn(0x7fd4780d3d40 msgr2=0x7fd4780d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47fedf640 1 -- 172.21.15.77:0/3779618973 shutdown_connections 2024-03-06T07:22:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.977+0000 7fd47fedf640 1 -- 172.21.15.77:0/3779618973 wait complete. 2024-03-06T07:22:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.978+0000 7fd47fedf640 1 Processor -- start 2024-03-06T07:22:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.978+0000 7fd47fedf640 1 -- start start 2024-03-06T07:22:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47fedf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4780d1910 0x7fd47815d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47fedf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d22e0 0x7fd4780cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47fedf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4781614d0 0x7fd4780cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47fedf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780b0910 con 0x7fd4780d22e0 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47fedf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780cfc60 con 0x7fd4781614d0 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47fedf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780b1e40 con 0x7fd4780d1910 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47d47a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d22e0 0x7fd4780cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47d47a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d22e0 0x7fd4780cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46494/0 (socket says 172.21.15.77:46494) 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47d47a640 1 -- 172.21.15.77:0/3438786250 learned_addr learned my addr 172.21.15.77:0/3438786250 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47dc7b640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4780d1910 0x7fd47815d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47e47c640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4781614d0 0x7fd4780cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47d47a640 1 -- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4780d1910 msgr2=0x7fd47815d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47d47a640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4780d1910 0x7fd47815d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.979+0000 7fd47d47a640 1 -- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4781614d0 msgr2=0x7fd4780cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd47d47a640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4781614d0 0x7fd4780cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd47d47a640 1 -- 172.21.15.77:0/3438786250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4780c79a0 con 0x7fd4780d22e0 2024-03-06T07:22:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd47e47c640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4781614d0 0x7fd4780cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd47d47a640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d22e0 0x7fd4780cc370 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd468000c00 tx=0x7fd4680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd46e7fc640 1 -- 172.21.15.77:0/3438786250 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468035070 con 0x7fd4780d22e0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd46e7fc640 1 -- 172.21.15.77:0/3438786250 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd468004250 con 0x7fd4780d22e0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4780c7c80 con 0x7fd4780d22e0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd46e7fc640 1 -- 172.21.15.77:0/3438786250 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46803a630 con 0x7fd4780d22e0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.980+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4780c8770 con 0x7fd4780d22e0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.981+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47808d7f0 con 0x7fd4780d22e0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.981+0000 7fd46e7fc640 1 -- 172.21.15.77:0/3438786250 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd468038020 con 0x7fd4780d22e0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.982+0000 7fd46e7fc640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45c06b4d0 0x7fd45c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.982+0000 7fd46e7fc640 1 -- 172.21.15.77:0/3438786250 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4680abda0 con 0x7fd4780d22e0 2024-03-06T07:22:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.982+0000 7fd47dc7b640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45c06b4d0 0x7fd45c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.982+0000 7fd47dc7b640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45c06b4d0 0x7fd45c06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd47815ec10 tx=0x7fd47400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:27.986+0000 7fd46e7fc640 1 -- 172.21.15.77:0/3438786250 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd47808d7f0 con 0x7fd4780d22e0 2024-03-06T07:22:28.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.157+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd4780b72f0 con 0x7fd4780d22e0 2024-03-06T07:22:28.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.157+0000 7fd46e7fc640 1 -- 172.21.15.77:0/3438786250 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd468040070 con 0x7fd4780d22e0 2024-03-06T07:22:28.159 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45c06b4d0 msgr2=0x7fd45c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45c06b4d0 0x7fd45c06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd47815ec10 tx=0x7fd47400b040 comp rx=0 tx=0).stop 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d22e0 msgr2=0x7fd4780cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d22e0 0x7fd4780cc370 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd468000c00 tx=0x7fd4680046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 shutdown_connections 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45c06b4d0 0x7fd45c06d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4780d22e0 0x7fd4780cc370 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd4781614d0 0x7fd4780cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 --2- 172.21.15.77:0/3438786250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd4780d1910 0x7fd47815d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 >> 172.21.15.77:0/3438786250 conn(0x7fd4780d3d40 msgr2=0x7fd478160800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 shutdown_connections 2024-03-06T07:22:28.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.159+0000 7fd47fedf640 1 -- 172.21.15.77:0/3438786250 wait complete. 2024-03-06T07:22:28.172 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:28.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.274+0000 7f2dee06b640 1 Processor -- start 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.274+0000 7f2dee06b640 1 -- start start 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.274+0000 7f2dee06b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2de815def0 0x7f2de80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.274+0000 7f2dee06b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2de80cea50 0x7f2de80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.274+0000 7f2dee06b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de806d030 con 0x7f2de806cab0 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.274+0000 7f2dee06b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de80cf1c0 con 0x7f2de80cea50 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.274+0000 7f2dee06b640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de80cc760 con 0x7f2de815def0 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2de80cea50 0x7f2de80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de6ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2de806cab0 0x7f2de80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46684/0 (socket says 172.21.15.77:46684) 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de6ffd640 1 -- 172.21.15.77:0/1752154028 learned_addr learned my addr 172.21.15.77:0/1752154028 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de77fe640 1 --2- 172.21.15.77:0/1752154028 >> v2:172.21.15.158:3300/0 conn(0x7f2de815def0 0x7f2de80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de7fff640 1 -- 172.21.15.77:0/1752154028 >> v2:172.21.15.158:3300/0 conn(0x7f2de815def0 msgr2=0x7f2de80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de7fff640 1 --2- 172.21.15.77:0/1752154028 >> v2:172.21.15.158:3300/0 conn(0x7f2de815def0 0x7f2de80b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de7fff640 1 -- 172.21.15.77:0/1752154028 >> v1:172.21.15.77:6789/0 conn(0x7f2de806cab0 legacy=0x7f2de80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de67fc640 1 -- 172.21.15.77:0/1752154028 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864210100 0 0) 0x7f2de806d030 con 0x7f2de806cab0 2024-03-06T07:22:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.275+0000 7f2de7fff640 1 -- 172.21.15.77:0/1752154028 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de806d310 con 0x7f2de80cea50 2024-03-06T07:22:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2de7fff640 1 --2- 172.21.15.77:0/1752154028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de80cea50 0x7f2de80ccbf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2de8071f60 tx=0x7f2de002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bdecd51b19d6a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2de67fc640 1 -- 172.21.15.77:0/1752154028 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0039070 con 0x7f2de80cea50 2024-03-06T07:22:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2de67fc640 1 -- 172.21.15.77:0/1752154028 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de002cd00 con 0x7f2de80cea50 2024-03-06T07:22:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2de67fc640 1 -- 172.21.15.77:0/1752154028 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0034660 con 0x7f2de80cea50 2024-03-06T07:22:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2dee06b640 1 -- 172.21.15.77:0/1752154028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de80cea50 msgr2=0x7f2de80ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2dee06b640 1 --2- 172.21.15.77:0/1752154028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de80cea50 0x7f2de80ccbf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2de8071f60 tx=0x7f2de002bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2dee06b640 1 -- 172.21.15.77:0/1752154028 shutdown_connections 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2dee06b640 1 --2- 172.21.15.77:0/1752154028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de80cea50 0x7f2de80ccbf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2dee06b640 1 --2- 172.21.15.77:0/1752154028 >> v2:172.21.15.158:3300/0 conn(0x7f2de815def0 0x7f2de80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2dee06b640 1 -- 172.21.15.77:0/1752154028 >> 172.21.15.77:0/1752154028 conn(0x7f2de80b3690 msgr2=0x7f2de80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2dee06b640 1 -- 172.21.15.77:0/1752154028 shutdown_connections 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.276+0000 7f2dee06b640 1 -- 172.21.15.77:0/1752154028 wait complete. 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.277+0000 7f2dee06b640 1 Processor -- start 2024-03-06T07:22:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2dee06b640 1 -- start start 2024-03-06T07:22:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2dee06b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de806cab0 0x7f2de81f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2dee06b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2de80cea50 0x7f2de815edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2dee06b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2de815def0 0x7f2de815f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2dee06b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de815e2d0 con 0x7f2de806cab0 2024-03-06T07:22:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2dee06b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80cc3f0 con 0x7f2de80cea50 2024-03-06T07:22:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2dee06b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de815e450 con 0x7f2de815def0 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2de77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de806cab0 0x7f2de81f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2de77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de806cab0 0x7f2de81f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46502/0 (socket says 172.21.15.77:46502) 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2de77fe640 1 -- 172.21.15.77:0/3281753245 learned_addr learned my addr 172.21.15.77:0/3281753245 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.278+0000 7f2de6ffd640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2de80cea50 0x7f2de815edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2de7fff640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2de815def0 0x7f2de815f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2de77fe640 1 -- 172.21.15.77:0/3281753245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2de815def0 msgr2=0x7f2de815f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2de77fe640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2de815def0 0x7f2de815f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2de77fe640 1 -- 172.21.15.77:0/3281753245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2de80cea50 msgr2=0x7f2de815edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2de77fe640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2de80cea50 0x7f2de815edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2de77fe640 1 -- 172.21.15.77:0/3281753245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de815f920 con 0x7f2de806cab0 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2de77fe640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de806cab0 0x7f2de81f6c90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00e870 tx=0x7f2ddc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2dcffff640 1 -- 172.21.15.77:0/3281753245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc017070 con 0x7f2de806cab0 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2dcffff640 1 -- 172.21.15.77:0/3281753245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc00c940 con 0x7f2de806cab0 2024-03-06T07:22:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de815fc00 con 0x7f2de806cab0 2024-03-06T07:22:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2dcffff640 1 -- 172.21.15.77:0/3281753245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc00cd30 con 0x7f2de806cab0 2024-03-06T07:22:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.279+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de81601c0 con 0x7f2de806cab0 2024-03-06T07:22:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.280+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de809fcf0 con 0x7f2de806cab0 2024-03-06T07:22:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.280+0000 7f2dcffff640 1 -- 172.21.15.77:0/3281753245 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ddc01a070 con 0x7f2de806cab0 2024-03-06T07:22:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.281+0000 7f2dcffff640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dd406b4d0 0x7f2dd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.281+0000 7f2dcffff640 1 -- 172.21.15.77:0/3281753245 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ddc08a3d0 con 0x7f2de806cab0 2024-03-06T07:22:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.281+0000 7f2de6ffd640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dd406b4d0 0x7f2dd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.281+0000 7f2de6ffd640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dd406b4d0 0x7f2dd406d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8007f70 tx=0x7f2dd8007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.285+0000 7f2dcffff640 1 -- 172.21.15.77:0/3281753245 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ddc058c00 con 0x7f2de806cab0 2024-03-06T07:22:28.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.457+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2de8068ff0 con 0x7f2de806cab0 2024-03-06T07:22:28.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.458+0000 7f2dcffff640 1 -- 172.21.15.77:0/3281753245 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2ddc05cca0 con 0x7f2de806cab0 2024-03-06T07:22:28.460 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dd406b4d0 msgr2=0x7f2dd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dd406b4d0 0x7f2dd406d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8007f70 tx=0x7f2dd8007f00 comp rx=0 tx=0).stop 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de806cab0 msgr2=0x7f2de81f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de806cab0 0x7f2de81f6c90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00e870 tx=0x7f2ddc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 shutdown_connections 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2de815def0 0x7f2de815f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2de80cea50 0x7f2de815edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dd406b4d0 0x7f2dd406d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 --2- 172.21.15.77:0/3281753245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2de806cab0 0x7f2de81f6c90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.459+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 >> 172.21.15.77:0/3281753245 conn(0x7f2de80b3690 msgr2=0x7f2de80ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.460+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 shutdown_connections 2024-03-06T07:22:28.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.460+0000 7f2dee06b640 1 -- 172.21.15.77:0/3281753245 wait complete. 2024-03-06T07:22:28.482 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:22:28.482 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:22:28.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:28.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.581+0000 7f9e75047640 1 Processor -- start 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.581+0000 7f9e75047640 1 -- start start 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e75047640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e701424f0 0x7f9e700cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e75047640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e700cc980 con 0x7f9e700cd7a0 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e75047640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e700ccc60 con 0x7f9e7006da40 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e75047640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700cd220 con 0x7f9e701424f0 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e67fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9e700cd7a0 0x7f9e7006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52416/0 (socket says 172.21.15.77:52416) 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e67fff640 1 -- 172.21.15.77:0/1677879728 learned_addr learned my addr 172.21.15.77:0/1677879728 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e6ed9d640 1 --2- 172.21.15.77:0/1677879728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e701424f0 0x7f9e700cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e6e59c640 1 -- 172.21.15.77:0/1677879728 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568402273 0 0) 0x7f9e700cc980 con 0x7f9e700cd7a0 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e6e59c640 1 -- 172.21.15.77:0/1677879728 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c003520 con 0x7f9e700cd7a0 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e6e59c640 1 -- 172.21.15.77:0/1677879728 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152833059 0 0) 0x7f9e700ccc60 con 0x7f9e7006da40 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e6e59c640 1 -- 172.21.15.77:0/1677879728 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e700cc980 con 0x7f9e7006da40 2024-03-06T07:22:28.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e6ed9d640 1 -- 172.21.15.77:0/1677879728 >> v1:172.21.15.158:6789/0 conn(0x7f9e7006da40 legacy=0x7f9e70071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e6ed9d640 1 -- 172.21.15.77:0/1677879728 >> v1:172.21.15.77:6789/0 conn(0x7f9e700cd7a0 legacy=0x7f9e7006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.582+0000 7f9e6ed9d640 1 -- 172.21.15.77:0/1677879728 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700ccf40 con 0x7f9e701424f0 2024-03-06T07:22:28.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.583+0000 7f9e6ed9d640 1 --2- 172.21.15.77:0/1677879728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e701424f0 0x7f9e700cc2f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e600097d0 tx=0x7f9e6002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d85d4e104c3b7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.583+0000 7f9e6e59c640 1 -- 172.21.15.77:0/1677879728 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60035020 con 0x7f9e701424f0 2024-03-06T07:22:28.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.583+0000 7f9e6e59c640 1 -- 172.21.15.77:0/1677879728 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60031890 con 0x7f9e701424f0 2024-03-06T07:22:28.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.583+0000 7f9e6e59c640 1 -- 172.21.15.77:0/1677879728 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60031c40 con 0x7f9e701424f0 2024-03-06T07:22:28.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.584+0000 7f9e75047640 1 -- 172.21.15.77:0/1677879728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e701424f0 msgr2=0x7f9e700cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.584+0000 7f9e75047640 1 --2- 172.21.15.77:0/1677879728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e701424f0 0x7f9e700cc2f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e600097d0 tx=0x7f9e6002b930 comp rx=0 tx=0).stop 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.584+0000 7f9e75047640 1 -- 172.21.15.77:0/1677879728 shutdown_connections 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.584+0000 7f9e75047640 1 --2- 172.21.15.77:0/1677879728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e701424f0 0x7f9e700cc2f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.584+0000 7f9e75047640 1 -- 172.21.15.77:0/1677879728 >> 172.21.15.77:0/1677879728 conn(0x7f9e700cf8a0 msgr2=0x7f9e700cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.584+0000 7f9e75047640 1 -- 172.21.15.77:0/1677879728 shutdown_connections 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.584+0000 7f9e75047640 1 -- 172.21.15.77:0/1677879728 wait complete. 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.585+0000 7f9e75047640 1 Processor -- start 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.585+0000 7f9e75047640 1 -- start start 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e75047640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e7006da40 0x7f9e70144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e75047640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e700cd7a0 0x7f9e70142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e75047640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e701424f0 0x7f9e7015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e75047640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7015d070 con 0x7f9e701424f0 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e75047640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7015cef0 con 0x7f9e7006da40 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e75047640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7015d1f0 con 0x7f9e700cd7a0 2024-03-06T07:22:28.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e701424f0 0x7f9e7015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e701424f0 0x7f9e7015b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42816/0 (socket says 172.21.15.77:42816) 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 -- 172.21.15.77:0/4179696360 learned_addr learned my addr 172.21.15.77:0/4179696360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6ed9d640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e7006da40 0x7f9e70144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e67fff640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e700cd7a0 0x7f9e70142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 -- 172.21.15.77:0/4179696360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e700cd7a0 msgr2=0x7f9e70142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e700cd7a0 0x7f9e70142cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 -- 172.21.15.77:0/4179696360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e7006da40 msgr2=0x7f9e70144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e7006da40 0x7f9e70144400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 -- 172.21.15.77:0/4179696360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7015bd00 con 0x7f9e701424f0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e6f59e640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e701424f0 0x7f9e7015b7c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c790 tx=0x7f9e6800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e677fe640 1 -- 172.21.15.77:0/4179696360 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68003b70 con 0x7f9e701424f0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e677fe640 1 -- 172.21.15.77:0/4179696360 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e68003d10 con 0x7f9e701424f0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.586+0000 7f9e677fe640 1 -- 172.21.15.77:0/4179696360 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800aa50 con 0x7f9e701424f0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.587+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e70143360 con 0x7f9e701424f0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.587+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e70143920 con 0x7f9e701424f0 2024-03-06T07:22:28.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.587+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7004f330 con 0x7f9e701424f0 2024-03-06T07:22:28.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.588+0000 7f9e677fe640 1 -- 172.21.15.77:0/4179696360 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e6800abf0 con 0x7f9e701424f0 2024-03-06T07:22:28.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.588+0000 7f9e677fe640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e5006b4d0 0x7f9e5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.588+0000 7f9e677fe640 1 -- 172.21.15.77:0/4179696360 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e6808a070 con 0x7f9e701424f0 2024-03-06T07:22:28.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.588+0000 7f9e6ed9d640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e5006b4d0 0x7f9e5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.589+0000 7f9e6ed9d640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e5006b4d0 0x7f9e5006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9e60036000 tx=0x7f9e6003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.592+0000 7f9e677fe640 1 -- 172.21.15.77:0/4179696360 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e7015bd00 con 0x7f9e701424f0 2024-03-06T07:22:28.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.765+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e70141b40 con 0x7f9e701424f0 2024-03-06T07:22:28.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.766+0000 7f9e677fe640 1 -- 172.21.15.77:0/4179696360 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f9e680588a0 con 0x7f9e701424f0 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:28.768 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e5006b4d0 msgr2=0x7f9e5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e5006b4d0 0x7f9e5006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9e60036000 tx=0x7f9e6003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e701424f0 msgr2=0x7f9e7015b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e701424f0 0x7f9e7015b7c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c790 tx=0x7f9e6800cc50 comp rx=0 tx=0).stop 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 shutdown_connections 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e5006b4d0 0x7f9e5006d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e701424f0 0x7f9e7015b7c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e700cd7a0 0x7f9e70142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 --2- 172.21.15.77:0/4179696360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e7006da40 0x7f9e70144400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 >> 172.21.15.77:0/4179696360 conn(0x7f9e700cf8a0 msgr2=0x7f9e700cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 shutdown_connections 2024-03-06T07:22:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.768+0000 7f9e75047640 1 -- 172.21.15.77:0/4179696360 wait complete. 2024-03-06T07:22:28.781 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:28.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.882+0000 7f8329393640 1 Processor -- start 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.882+0000 7f8329393640 1 -- start start 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f8329393640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f83240d4860 0x7f83240d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f8329393640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f83240b0e30 0x7f83240b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f8329393640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83241618b0 con 0x7f83241614d0 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f8329393640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83240d4cf0 con 0x7f83240d4860 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f8329393640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83240cf400 con 0x7f83240b0e30 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f83227fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f83240d4860 0x7f83240d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f83227fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f83240d4860 0x7f83240d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42830/0 (socket says 172.21.15.77:42830) 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f83227fc640 1 -- 172.21.15.77:0/4125803040 learned_addr learned my addr 172.21.15.77:0/4125803040 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f83237fe640 1 --2- 172.21.15.77:0/4125803040 >> v2:172.21.15.158:3300/0 conn(0x7f83240b0e30 0x7f83240b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f83227fc640 1 -- 172.21.15.77:0/4125803040 >> v2:172.21.15.158:3300/0 conn(0x7f83240b0e30 msgr2=0x7f83240b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f83227fc640 1 --2- 172.21.15.77:0/4125803040 >> v2:172.21.15.158:3300/0 conn(0x7f83240b0e30 0x7f83240b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f83227fc640 1 -- 172.21.15.77:0/4125803040 >> v1:172.21.15.158:6789/0 conn(0x7f83241614d0 legacy=0x7f83240d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.883+0000 7f83227fc640 1 -- 172.21.15.77:0/4125803040 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240cf6a0 con 0x7f83240d4860 2024-03-06T07:22:28.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.884+0000 7f83227fc640 1 --2- 172.21.15.77:0/4125803040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240d4860 0x7f83240d2020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f83240b2a70 tx=0x7f831802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b501d046435169 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.884+0000 7f8321ffb640 1 -- 172.21.15.77:0/4125803040 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318034070 con 0x7f83240d4860 2024-03-06T07:22:28.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.884+0000 7f8321ffb640 1 -- 172.21.15.77:0/4125803040 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8318030860 con 0x7f83240d4860 2024-03-06T07:22:28.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.884+0000 7f8321ffb640 1 -- 172.21.15.77:0/4125803040 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318030c30 con 0x7f83240d4860 2024-03-06T07:22:28.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.885+0000 7f8329393640 1 -- 172.21.15.77:0/4125803040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240d4860 msgr2=0x7f83240d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.885+0000 7f8329393640 1 --2- 172.21.15.77:0/4125803040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240d4860 0x7f83240d2020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f83240b2a70 tx=0x7f831802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.885+0000 7f8329393640 1 -- 172.21.15.77:0/4125803040 shutdown_connections 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.885+0000 7f8329393640 1 --2- 172.21.15.77:0/4125803040 >> v2:172.21.15.158:3300/0 conn(0x7f83240b0e30 0x7f83240b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.885+0000 7f8329393640 1 --2- 172.21.15.77:0/4125803040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240d4860 0x7f83240d2020 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.885+0000 7f8329393640 1 -- 172.21.15.77:0/4125803040 >> 172.21.15.77:0/4125803040 conn(0x7f83240d3d40 msgr2=0x7f83240d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.885+0000 7f8329393640 1 -- 172.21.15.77:0/4125803040 shutdown_connections 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.885+0000 7f8329393640 1 -- 172.21.15.77:0/4125803040 wait complete. 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.886+0000 7f8329393640 1 Processor -- start 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.886+0000 7f8329393640 1 -- start start 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8329393640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240b0e30 0x7f83240b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8329393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83240d4860 0x7f8324159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8329393640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83241614d0 0x7f83241e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8329393640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83240b0910 con 0x7f83240b0e30 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8329393640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83240cfc60 con 0x7f83241614d0 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8329393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83240d2560 con 0x7f83240d4860 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240b0e30 0x7f83240b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240b0e30 0x7f83240b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42834/0 (socket says 172.21.15.77:42834) 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 -- 172.21.15.77:0/564277980 learned_addr learned my addr 172.21.15.77:0/564277980 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f83227fc640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83240d4860 0x7f8324159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f83237fe640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83241614d0 0x7f83241e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 -- 172.21.15.77:0/564277980 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83240d4860 msgr2=0x7f8324159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83240d4860 0x7f8324159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 -- 172.21.15.77:0/564277980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83241614d0 msgr2=0x7f83241e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83241614d0 0x7f83241e9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 -- 172.21.15.77:0/564277980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240cb310 con 0x7f83240b0e30 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8322ffd640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240b0e30 0x7f83240b51b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f831400c340 tx=0x7f831400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f830b7fe640 1 -- 172.21.15.77:0/564277980 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314017070 con 0x7f83240b0e30 2024-03-06T07:22:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f830b7fe640 1 -- 172.21.15.77:0/564277980 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83140052d0 con 0x7f83240b0e30 2024-03-06T07:22:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83240cb5f0 con 0x7f83240b0e30 2024-03-06T07:22:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.887+0000 7f830b7fe640 1 -- 172.21.15.77:0/564277980 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314003850 con 0x7f83240b0e30 2024-03-06T07:22:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.888+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83240cbbb0 con 0x7f83240b0e30 2024-03-06T07:22:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.889+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8324068ff0 con 0x7f83240b0e30 2024-03-06T07:22:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.889+0000 7f830b7fe640 1 -- 172.21.15.77:0/564277980 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f831401a070 con 0x7f83240b0e30 2024-03-06T07:22:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.889+0000 7f830b7fe640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f831006b4d0 0x7f831006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.889+0000 7f830b7fe640 1 -- 172.21.15.77:0/564277980 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8314089460 con 0x7f83240b0e30 2024-03-06T07:22:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.889+0000 7f83227fc640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f831006b4d0 0x7f831006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.890+0000 7f83227fc640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f831006b4d0 0x7f831006d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8318036000 tx=0x7f8318008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:28.894+0000 7f830b7fe640 1 -- 172.21.15.77:0/564277980 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8314057c90 con 0x7f83240b0e30 2024-03-06T07:22:29.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.064+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83240cbdf0 con 0x7f83240b0e30 2024-03-06T07:22:29.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.064+0000 7f830b7fe640 1 -- 172.21.15.77:0/564277980 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f831405bd30 con 0x7f83240b0e30 2024-03-06T07:22:29.067 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f831006b4d0 msgr2=0x7f831006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f831006b4d0 0x7f831006d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8318036000 tx=0x7f8318008030 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240b0e30 msgr2=0x7f83240b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240b0e30 0x7f83240b51b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f831400c340 tx=0x7f831400c800 comp rx=0 tx=0).stop 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 shutdown_connections 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83241614d0 0x7f83241e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83240d4860 0x7f8324159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f831006b4d0 0x7f831006d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 --2- 172.21.15.77:0/564277980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83240b0e30 0x7f83240b51b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 >> 172.21.15.77:0/564277980 conn(0x7f83240d3d40 msgr2=0x7f83240cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 shutdown_connections 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.067+0000 7f8329393640 1 -- 172.21.15.77:0/564277980 wait complete. 2024-03-06T07:22:29.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:29.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.181+0000 7f3983484640 1 Processor -- start 2024-03-06T07:22:29.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.181+0000 7f3983484640 1 -- start start 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3983484640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f397c15def0 0x7f397c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3983484640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f397c06cab0 0x7f397c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3983484640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c06d030 con 0x7f397c0ce880 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3983484640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397c0cf1c0 con 0x7f397c15def0 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3983484640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397c0cc760 con 0x7f397c06cab0 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3981220640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f397c15def0 0x7f397c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3981220640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f397c15def0 0x7f397c0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42840/0 (socket says 172.21.15.77:42840) 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3981220640 1 -- 172.21.15.77:0/3943953295 learned_addr learned my addr 172.21.15.77:0/3943953295 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3980a1f640 1 --2- 172.21.15.77:0/3943953295 >> v2:172.21.15.158:3300/0 conn(0x7f397c06cab0 0x7f397c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3981220640 1 -- 172.21.15.77:0/3943953295 >> v2:172.21.15.158:3300/0 conn(0x7f397c06cab0 msgr2=0x7f397c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3981220640 1 --2- 172.21.15.77:0/3943953295 >> v2:172.21.15.158:3300/0 conn(0x7f397c06cab0 0x7f397c0ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3981220640 1 -- 172.21.15.77:0/3943953295 >> v1:172.21.15.158:6789/0 conn(0x7f397c0ce880 legacy=0x7f397c0cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3981220640 1 -- 172.21.15.77:0/3943953295 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c06d310 con 0x7f397c15def0 2024-03-06T07:22:29.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.182+0000 7f3980a1f640 1 --2- 172.21.15.77:0/3943953295 >> v2:172.21.15.158:3300/0 conn(0x7f397c06cab0 0x7f397c0ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3981220640 1 --2- 172.21.15.77:0/3943953295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c0b41b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f397c0b3660 tx=0x7f397802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fc255cc002c5c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3973fff640 1 -- 172.21.15.77:0/3943953295 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397802f850 con 0x7f397c15def0 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3973fff640 1 -- 172.21.15.77:0/3943953295 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397802f9f0 con 0x7f397c15def0 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3973fff640 1 -- 172.21.15.77:0/3943953295 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397802fdc0 con 0x7f397c15def0 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3983484640 1 -- 172.21.15.77:0/3943953295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 msgr2=0x7f397c0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3983484640 1 --2- 172.21.15.77:0/3943953295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c0b41b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f397c0b3660 tx=0x7f397802ba30 comp rx=0 tx=0).stop 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3983484640 1 -- 172.21.15.77:0/3943953295 shutdown_connections 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3983484640 1 --2- 172.21.15.77:0/3943953295 >> v2:172.21.15.158:3300/0 conn(0x7f397c06cab0 0x7f397c0ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3983484640 1 --2- 172.21.15.77:0/3943953295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c0b41b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.183+0000 7f3983484640 1 -- 172.21.15.77:0/3943953295 >> 172.21.15.77:0/3943953295 conn(0x7f397c0b3690 msgr2=0x7f397c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.184+0000 7f3983484640 1 -- 172.21.15.77:0/3943953295 shutdown_connections 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.184+0000 7f3983484640 1 -- 172.21.15.77:0/3943953295 wait complete. 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3983484640 1 Processor -- start 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3983484640 1 -- start start 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3983484640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f397c06cab0 0x7f397c0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3983484640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f397c0ce880 0x7f397c0b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3983484640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3983484640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0cc3f0 con 0x7f397c15def0 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3983484640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0cecb0 con 0x7f397c0ce880 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3983484640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397c15e2d0 con 0x7f397c06cab0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3981a21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3981a21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c160240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42844/0 (socket says 172.21.15.77:42844) 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.185+0000 7f3981a21640 1 -- 172.21.15.77:0/4083226868 learned_addr learned my addr 172.21.15.77:0/4083226868 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3981220640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f397c06cab0 0x7f397c0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3980a1f640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f397c0ce880 0x7f397c0b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3981a21640 1 -- 172.21.15.77:0/4083226868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f397c06cab0 msgr2=0x7f397c0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3981a21640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f397c06cab0 0x7f397c0cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3981a21640 1 -- 172.21.15.77:0/4083226868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f397c0ce880 msgr2=0x7f397c0b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3981a21640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f397c0ce880 0x7f397c0b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3981a21640 1 -- 172.21.15.77:0/4083226868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c0b49d0 con 0x7f397c15def0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3981a21640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c160240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f397400c7a0 tx=0x7f397400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3971ffb640 1 -- 172.21.15.77:0/4083226868 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974003b70 con 0x7f397c15def0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3971ffb640 1 -- 172.21.15.77:0/4083226868 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974003d30 con 0x7f397c15def0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c0b4cb0 con 0x7f397c15def0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3971ffb640 1 -- 172.21.15.77:0/4083226868 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397400aa90 con 0x7f397c15def0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.186+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c0b5270 con 0x7f397c15def0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.187+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c09fcf0 con 0x7f397c15def0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.188+0000 7f3971ffb640 1 -- 172.21.15.77:0/4083226868 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f397400ac30 con 0x7f397c15def0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.188+0000 7f3971ffb640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f396006b4d0 0x7f396006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.188+0000 7f3971ffb640 1 -- 172.21.15.77:0/4083226868 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3974089f70 con 0x7f397c15def0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.188+0000 7f3981220640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f396006b4d0 0x7f396006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.189+0000 7f3981220640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f396006b4d0 0x7f396006d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3978036000 tx=0x7f3978008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.193+0000 7f3971ffb640 1 -- 172.21.15.77:0/4083226868 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3974058830 con 0x7f397c15def0 2024-03-06T07:22:29.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:29.239 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:29.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.367+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f397c068ff0 con 0x7f397c15def0 2024-03-06T07:22:29.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.368+0000 7f3971ffb640 1 -- 172.21.15.77:0/4083226868 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f397405c8d0 con 0x7f397c15def0 2024-03-06T07:22:29.370 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f396006b4d0 msgr2=0x7f396006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f396006b4d0 0x7f396006d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3978036000 tx=0x7f3978008030 comp rx=0 tx=0).stop 2024-03-06T07:22:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 msgr2=0x7f397c160240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c160240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f397400c7a0 tx=0x7f397400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 shutdown_connections 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f396006b4d0 0x7f396006d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f397c15def0 0x7f397c160240 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f397c0ce880 0x7f397c0b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 --2- 172.21.15.77:0/4083226868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f397c06cab0 0x7f397c0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 >> 172.21.15.77:0/4083226868 conn(0x7f397c0b3690 msgr2=0x7f397c0d2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 shutdown_connections 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.370+0000 7f3983484640 1 -- 172.21.15.77:0/4083226868 wait complete. 2024-03-06T07:22:29.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:29.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.484+0000 7f338a53e640 1 Processor -- start 2024-03-06T07:22:29.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f338a53e640 1 -- start start 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f338a53e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33840d4860 0x7f33840d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f338a53e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f33840b0e30 0x7f33840b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f338a53e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33841618b0 con 0x7f33841614d0 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f338a53e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33840d4cf0 con 0x7f33840b0e30 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f338a53e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33840cf400 con 0x7f33840d4860 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f3388adb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f33840b0e30 0x7f33840b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f3383fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f33841614d0 0x7f33840d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52420/0 (socket says 172.21.15.77:52420) 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f3388adb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f33840b0e30 0x7f33840b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42858/0 (socket says 172.21.15.77:42858) 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.485+0000 7f3383fff640 1 -- 172.21.15.77:0/2502522028 learned_addr learned my addr 172.21.15.77:0/2502522028 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:29.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.486+0000 7f33837fe640 1 --2- 172.21.15.77:0/2502522028 >> v2:172.21.15.158:3300/0 conn(0x7f33840d4860 0x7f33840d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.486+0000 7f3388adb640 1 -- 172.21.15.77:0/2502522028 >> v2:172.21.15.158:3300/0 conn(0x7f33840d4860 msgr2=0x7f33840d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.486+0000 7f3388adb640 1 --2- 172.21.15.77:0/2502522028 >> v2:172.21.15.158:3300/0 conn(0x7f33840d4860 0x7f33840d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.486+0000 7f3388adb640 1 -- 172.21.15.77:0/2502522028 >> v1:172.21.15.77:6789/0 conn(0x7f33841614d0 legacy=0x7f33840d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.486+0000 7f3388adb640 1 -- 172.21.15.77:0/2502522028 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33840cf6a0 con 0x7f33840b0e30 2024-03-06T07:22:29.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.486+0000 7f3382ffd640 1 -- 172.21.15.77:0/2502522028 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115292755 0 0) 0x7f33841618b0 con 0x7f33841614d0 2024-03-06T07:22:29.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f3388adb640 1 --2- 172.21.15.77:0/2502522028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33840b0e30 0x7f33840b1260 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f33840ce550 tx=0x7f337c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1da9238220c15d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f3382ffd640 1 -- 172.21.15.77:0/2502522028 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c039070 con 0x7f33840b0e30 2024-03-06T07:22:29.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f3382ffd640 1 -- 172.21.15.77:0/2502522028 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c02cc80 con 0x7f33840b0e30 2024-03-06T07:22:29.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f3382ffd640 1 -- 172.21.15.77:0/2502522028 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c034660 con 0x7f33840b0e30 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f338a53e640 1 -- 172.21.15.77:0/2502522028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33840b0e30 msgr2=0x7f33840b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f338a53e640 1 --2- 172.21.15.77:0/2502522028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33840b0e30 0x7f33840b1260 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f33840ce550 tx=0x7f337c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f338a53e640 1 -- 172.21.15.77:0/2502522028 shutdown_connections 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f338a53e640 1 --2- 172.21.15.77:0/2502522028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33840b0e30 0x7f33840b1260 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f338a53e640 1 --2- 172.21.15.77:0/2502522028 >> v2:172.21.15.158:3300/0 conn(0x7f33840d4860 0x7f33840d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f338a53e640 1 -- 172.21.15.77:0/2502522028 >> 172.21.15.77:0/2502522028 conn(0x7f33840d3d40 msgr2=0x7f33840d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f338a53e640 1 -- 172.21.15.77:0/2502522028 shutdown_connections 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.487+0000 7f338a53e640 1 -- 172.21.15.77:0/2502522028 wait complete. 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.488+0000 7f338a53e640 1 Processor -- start 2024-03-06T07:22:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f338a53e640 1 -- start start 2024-03-06T07:22:29.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f338a53e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33840b0e30 0x7f33841f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f338a53e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33840d4860 0x7f338415a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f338a53e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33841614d0 0x7f33840cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f338a53e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840b0910 con 0x7f33841614d0 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f338a53e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840cfc60 con 0x7f33840d4860 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f338a53e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33840d2560 con 0x7f33840b0e30 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f3388adb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33841614d0 0x7f33840cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f3388adb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33841614d0 0x7f33840cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42870/0 (socket says 172.21.15.77:42870) 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f3388adb640 1 -- 172.21.15.77:0/1478829083 learned_addr learned my addr 172.21.15.77:0/1478829083 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f3383fff640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33840b0e30 0x7f33841f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.489+0000 7f33837fe640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33840d4860 0x7f338415a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3388adb640 1 -- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33840b0e30 msgr2=0x7f33841f7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3388adb640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33840b0e30 0x7f33841f7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3388adb640 1 -- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33840d4860 msgr2=0x7f338415a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3388adb640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33840d4860 0x7f338415a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3388adb640 1 -- 172.21.15.77:0/1478829083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33841f81a0 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f33837fe640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33840d4860 0x7f338415a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3388adb640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33841614d0 0x7f33840cc250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f337c009e80 tx=0x7f337c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3380ff9640 1 -- 172.21.15.77:0/1478829083 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c039050 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3380ff9640 1 -- 172.21.15.77:0/1478829083 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c045070 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33840cb1b0 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f3380ff9640 1 -- 172.21.15.77:0/1478829083 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c009450 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.490+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33840cb770 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.491+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338406c070 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.491+0000 7f3380ff9640 1 -- 172.21.15.77:0/1478829083 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f337c03e850 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.492+0000 7f3380ff9640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337006b500 0x7f337006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.492+0000 7f3380ff9640 1 -- 172.21.15.77:0/1478829083 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f337c0ad230 con 0x7f33841614d0 2024-03-06T07:22:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.492+0000 7f3383fff640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337006b500 0x7f337006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.492+0000 7f3383fff640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337006b500 0x7f337006d9c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3374000f60 tx=0x7f3374009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.496+0000 7f3380ff9640 1 -- 172.21.15.77:0/1478829083 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f338406c070 con 0x7f33841614d0 2024-03-06T07:22:29.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.667+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3384069e80 con 0x7f33841614d0 2024-03-06T07:22:29.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.667+0000 7f3380ff9640 1 -- 172.21.15.77:0/1478829083 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3384069e80 con 0x7f33841614d0 2024-03-06T07:22:29.670 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337006b500 msgr2=0x7f337006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337006b500 0x7f337006d9c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3374000f60 tx=0x7f3374009040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33841614d0 msgr2=0x7f33840cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33841614d0 0x7f33840cc250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f337c009e80 tx=0x7f337c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 shutdown_connections 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337006b500 0x7f337006d9c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33841614d0 0x7f33840cc250 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33840d4860 0x7f338415a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 --2- 172.21.15.77:0/1478829083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33840b0e30 0x7f33841f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.669+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 >> 172.21.15.77:0/1478829083 conn(0x7f33840d3d40 msgr2=0x7f338406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.670+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 shutdown_connections 2024-03-06T07:22:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.670+0000 7f338a53e640 1 -- 172.21.15.77:0/1478829083 wait complete. 2024-03-06T07:22:29.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:22:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.781+0000 7f6bd4655640 1 Processor -- start 2024-03-06T07:22:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.781+0000 7f6bd4655640 1 -- start start 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd4655640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd4655640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc070dc0 con 0x7f6bcc070930 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd4655640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc0b3bb0 con 0x7f6bcc144640 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd4655640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0ce5d0 con 0x7f6bcc15b7a0 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd2bf2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6bcc070930 0x7f6bcc0cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52434/0 (socket says 172.21.15.77:52434) 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd2bf2640 1 -- 172.21.15.77:0/2751255038 learned_addr learned my addr 172.21.15.77:0/2751255038 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd1bf0640 1 --2- 172.21.15.77:0/2751255038 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd13ef640 1 -- 172.21.15.77:0/2751255038 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163724841 0 0) 0x7f6bcc070dc0 con 0x7f6bcc070930 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd13ef640 1 -- 172.21.15.77:0/2751255038 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb8003520 con 0x7f6bcc070930 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd13ef640 1 -- 172.21.15.77:0/2751255038 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681196935 0 0) 0x7f6bcc0b3bb0 con 0x7f6bcc144640 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd13ef640 1 -- 172.21.15.77:0/2751255038 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bcc070dc0 con 0x7f6bcc144640 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd1bf0640 1 -- 172.21.15.77:0/2751255038 >> v1:172.21.15.158:6789/0 conn(0x7f6bcc144640 legacy=0x7f6bcc15b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.782+0000 7f6bd1bf0640 1 -- 172.21.15.77:0/2751255038 >> v1:172.21.15.77:6789/0 conn(0x7f6bcc070930 legacy=0x7f6bcc0cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.783+0000 7f6bd1bf0640 1 -- 172.21.15.77:0/2751255038 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0b3e90 con 0x7f6bcc15b7a0 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.783+0000 7f6bd1bf0640 1 --2- 172.21.15.77:0/2751255038 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b34d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0097d0 tx=0x7f6bbc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b3c6a47a9dba988 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.783+0000 7f6bd13ef640 1 -- 172.21.15.77:0/2751255038 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc035020 con 0x7f6bcc15b7a0 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.783+0000 7f6bd13ef640 1 -- 172.21.15.77:0/2751255038 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc031800 con 0x7f6bcc15b7a0 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.784+0000 7f6bd13ef640 1 -- 172.21.15.77:0/2751255038 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc031bb0 con 0x7f6bcc15b7a0 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.784+0000 7f6bd4655640 1 -- 172.21.15.77:0/2751255038 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 msgr2=0x7f6bcc0b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.784+0000 7f6bd4655640 1 --2- 172.21.15.77:0/2751255038 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b34d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0097d0 tx=0x7f6bbc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.784+0000 7f6bd4655640 1 -- 172.21.15.77:0/2751255038 shutdown_connections 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.784+0000 7f6bd4655640 1 --2- 172.21.15.77:0/2751255038 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b34d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.784+0000 7f6bd4655640 1 -- 172.21.15.77:0/2751255038 >> 172.21.15.77:0/2751255038 conn(0x7f6bcc06b3d0 msgr2=0x7f6bcc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.784+0000 7f6bd4655640 1 -- 172.21.15.77:0/2751255038 shutdown_connections 2024-03-06T07:22:29.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.784+0000 7f6bd4655640 1 -- 172.21.15.77:0/2751255038 wait complete. 2024-03-06T07:22:29.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.785+0000 7f6bd4655640 1 Processor -- start 2024-03-06T07:22:29.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.785+0000 7f6bd4655640 1 -- start start 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd4655640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bcc070930 0x7f6bcc13f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd4655640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc144640 0x7f6bcc0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd4655640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd4655640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0b4aa0 con 0x7f6bcc144640 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd4655640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0cdee0 con 0x7f6bcc070930 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd4655640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0b4c20 con 0x7f6bcc15b7a0 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd1bf0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc144640 0x7f6bcc0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd1bf0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc144640 0x7f6bcc0c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42882/0 (socket says 172.21.15.77:42882) 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd1bf0640 1 -- 172.21.15.77:0/3334679893 learned_addr learned my addr 172.21.15.77:0/3334679893 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd23f1640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bcc070930 0x7f6bcc13f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd2bf2640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd1bf0640 1 -- 172.21.15.77:0/3334679893 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 msgr2=0x7f6bcc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd1bf0640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd1bf0640 1 -- 172.21.15.77:0/3334679893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bcc070930 msgr2=0x7f6bcc13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd1bf0640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bcc070930 0x7f6bcc13f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.786+0000 7f6bd1bf0640 1 -- 172.21.15.77:0/3334679893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0c5e90 con 0x7f6bcc144640 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.787+0000 7f6bd1bf0640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc144640 0x7f6bcc0c5440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc000c00 tx=0x7f6bbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.787+0000 7f6bc2ffd640 1 -- 172.21.15.77:0/3334679893 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc040070 con 0x7f6bcc144640 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.787+0000 7f6bc2ffd640 1 -- 172.21.15.77:0/3334679893 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc031e30 con 0x7f6bcc144640 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.787+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bcc0c6170 con 0x7f6bcc144640 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.787+0000 7f6bc2ffd640 1 -- 172.21.15.77:0/3334679893 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc035040 con 0x7f6bcc144640 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.787+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bcc0c6730 con 0x7f6bcc144640 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.788+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bcc04f330 con 0x7f6bcc144640 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.788+0000 7f6bc2ffd640 1 -- 172.21.15.77:0/3334679893 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bbc04b060 con 0x7f6bcc144640 2024-03-06T07:22:29.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.789+0000 7f6bc2ffd640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bb006b4d0 0x7f6bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.789+0000 7f6bd23f1640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bb006b4d0 0x7f6bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.789+0000 7f6bc2ffd640 1 -- 172.21.15.77:0/3334679893 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bbc0abab0 con 0x7f6bcc144640 2024-03-06T07:22:29.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.789+0000 7f6bd23f1640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bb006b4d0 0x7f6bb006d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8007fd0 tx=0x7f6bc8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:29.793+0000 7f6bc2ffd640 1 -- 172.21.15.77:0/3334679893 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bbc07a2e0 con 0x7f6bcc144640 2024-03-06T07:22:30.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.032+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f6bcc15efe0 con 0x7f6bcc144640 2024-03-06T07:22:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.040+0000 7f6bc2ffd640 1 -- 172.21.15.77:0/3334679893 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7f6bbc07e380 con 0x7f6bcc144640 2024-03-06T07:22:30.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bb006b4d0 msgr2=0x7f6bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bb006b4d0 0x7f6bb006d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8007fd0 tx=0x7f6bc8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc144640 msgr2=0x7f6bcc0c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc144640 0x7f6bcc0c5440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc000c00 tx=0x7f6bbc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 shutdown_connections 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bb006b4d0 0x7f6bb006d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bcc144640 0x7f6bcc0c5440 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bcc15b7a0 0x7f6bcc0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 --2- 172.21.15.77:0/3334679893 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bcc070930 0x7f6bcc13f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.042+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 >> 172.21.15.77:0/3334679893 conn(0x7f6bcc06b3d0 msgr2=0x7f6bcc0b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.043+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 shutdown_connections 2024-03-06T07:22:30.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.043+0000 7f6bd4655640 1 -- 172.21.15.77:0/3334679893 wait complete. 2024-03-06T07:22:30.055 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:30.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.156+0000 7fb46de76640 1 Processor -- start 2024-03-06T07:22:30.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.156+0000 7fb46de76640 1 -- start start 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.156+0000 7fb46de76640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468066d90 0x7fb4680b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.156+0000 7fb46de76640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46806b710 con 0x7fb4680b5410 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.156+0000 7fb46de76640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46806b150 con 0x7fb4680cf9a0 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.156+0000 7fb46de76640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680b61c0 con 0x7fb468066d90 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb467fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb4680b5410 0x7fb4680ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52436/0 (socket says 172.21.15.77:52436) 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb467fff640 1 -- 172.21.15.77:0/14029877 learned_addr learned my addr 172.21.15.77:0/14029877 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb466ffd640 1 --2- 172.21.15.77:0/14029877 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468066d90 0x7fb4680b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb4667fc640 1 -- 172.21.15.77:0/14029877 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756467522 0 0) 0x7fb46806b710 con 0x7fb4680b5410 2024-03-06T07:22:30.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb4667fc640 1 -- 172.21.15.77:0/14029877 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb450003520 con 0x7fb4680b5410 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb4667fc640 1 -- 172.21.15.77:0/14029877 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161167386 0 0) 0x7fb46806b150 con 0x7fb4680cf9a0 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb4667fc640 1 -- 172.21.15.77:0/14029877 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46806b710 con 0x7fb4680cf9a0 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb466ffd640 1 -- 172.21.15.77:0/14029877 >> v1:172.21.15.158:6789/0 conn(0x7fb4680cf9a0 legacy=0x7fb4680b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb466ffd640 1 -- 172.21.15.77:0/14029877 >> v1:172.21.15.77:6789/0 conn(0x7fb4680b5410 legacy=0x7fb4680ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.157+0000 7fb466ffd640 1 -- 172.21.15.77:0/14029877 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46806b430 con 0x7fb468066d90 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.158+0000 7fb466ffd640 1 --2- 172.21.15.77:0/14029877 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468066d90 0x7fb4680b4ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb4580097d0 tx=0x7fb45802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a875fa3f112b4d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.158+0000 7fb4667fc640 1 -- 172.21.15.77:0/14029877 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458035020 con 0x7fb468066d90 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.158+0000 7fb4667fc640 1 -- 172.21.15.77:0/14029877 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb458031890 con 0x7fb468066d90 2024-03-06T07:22:30.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.158+0000 7fb4667fc640 1 -- 172.21.15.77:0/14029877 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458031c60 con 0x7fb468066d90 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.159+0000 7fb46de76640 1 -- 172.21.15.77:0/14029877 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468066d90 msgr2=0x7fb4680b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.159+0000 7fb46de76640 1 --2- 172.21.15.77:0/14029877 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468066d90 0x7fb4680b4ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb4580097d0 tx=0x7fb45802b930 comp rx=0 tx=0).stop 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.159+0000 7fb46de76640 1 -- 172.21.15.77:0/14029877 shutdown_connections 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.159+0000 7fb46de76640 1 --2- 172.21.15.77:0/14029877 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468066d90 0x7fb4680b4ed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.159+0000 7fb46de76640 1 -- 172.21.15.77:0/14029877 >> 172.21.15.77:0/14029877 conn(0x7fb4680b1d70 msgr2=0x7fb4680b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.159+0000 7fb46de76640 1 -- 172.21.15.77:0/14029877 shutdown_connections 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.159+0000 7fb46de76640 1 -- 172.21.15.77:0/14029877 wait complete. 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.160+0000 7fb46de76640 1 Processor -- start 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.160+0000 7fb46de76640 1 -- start start 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb46de76640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468066d90 0x7fb468151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb46de76640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680b5410 0x7fb46806df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb46de76640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cf9a0 0x7fb46806e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb46de76640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680cfd80 con 0x7fb468066d90 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb46de76640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680b0c50 con 0x7fb4680cf9a0 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb46de76640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680cff00 con 0x7fb4680b5410 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468066d90 0x7fb468151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468066d90 0x7fb468151580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42898/0 (socket says 172.21.15.77:42898) 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 -- 172.21.15.77:0/1318381254 learned_addr learned my addr 172.21.15.77:0/1318381254 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb467fff640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cf9a0 0x7fb46806e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb466ffd640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680b5410 0x7fb46806df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 -- 172.21.15.77:0/1318381254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680b5410 msgr2=0x7fb46806df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680b5410 0x7fb46806df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 -- 172.21.15.77:0/1318381254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cf9a0 msgr2=0x7fb46806e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cf9a0 0x7fb46806e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 -- 172.21.15.77:0/1318381254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46815c530 con 0x7fb468066d90 2024-03-06T07:22:30.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb4677fe640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468066d90 0x7fb468151580 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00c430 tx=0x7fb45c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb44ffff640 1 -- 172.21.15.77:0/1318381254 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c017070 con 0x7fb468066d90 2024-03-06T07:22:30.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb44ffff640 1 -- 172.21.15.77:0/1318381254 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c0052f0 con 0x7fb468066d90 2024-03-06T07:22:30.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46815c810 con 0x7fb468066d90 2024-03-06T07:22:30.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb44ffff640 1 -- 172.21.15.77:0/1318381254 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c003830 con 0x7fb468066d90 2024-03-06T07:22:30.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.161+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46815cdd0 con 0x7fb468066d90 2024-03-06T07:22:30.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.162+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46809e3d0 con 0x7fb468066d90 2024-03-06T07:22:30.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.163+0000 7fb44ffff640 1 -- 172.21.15.77:0/1318381254 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb45c01a070 con 0x7fb468066d90 2024-03-06T07:22:30.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.163+0000 7fb44ffff640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45406b4d0 0x7fb45406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.163+0000 7fb44ffff640 1 -- 172.21.15.77:0/1318381254 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb45c0895e0 con 0x7fb468066d90 2024-03-06T07:22:30.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.163+0000 7fb466ffd640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45406b4d0 0x7fb45406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.164+0000 7fb466ffd640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45406b4d0 0x7fb45406d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb458036000 tx=0x7fb45803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.167+0000 7fb44ffff640 1 -- 172.21.15.77:0/1318381254 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb45c057d80 con 0x7fb468066d90 2024-03-06T07:22:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.340+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb4680681a0 con 0x7fb468066d90 2024-03-06T07:22:30.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.345+0000 7fb44ffff640 1 -- 172.21.15.77:0/1318381254 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb45c05be20 con 0x7fb468066d90 2024-03-06T07:22:30.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45406b4d0 msgr2=0x7fb45406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45406b4d0 0x7fb45406d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb458036000 tx=0x7fb45803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468066d90 msgr2=0x7fb468151580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468066d90 0x7fb468151580 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00c430 tx=0x7fb45c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 shutdown_connections 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cf9a0 0x7fb46806e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680b5410 0x7fb46806df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45406b4d0 0x7fb45406d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 --2- 172.21.15.77:0/1318381254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468066d90 0x7fb468151580 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 >> 172.21.15.77:0/1318381254 conn(0x7fb4680b1d70 msgr2=0x7fb46815e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 shutdown_connections 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.347+0000 7fb46de76640 1 -- 172.21.15.77:0/1318381254 wait complete. 2024-03-06T07:22:30.360 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:30.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:30.398 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:30.443 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:30.444 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:22:30.444 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:30.459 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:22:30.460 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:30.515 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:30.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.653+0000 7fb4c993e640 1 Processor -- start 2024-03-06T07:22:30.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.653+0000 7fb4c993e640 1 -- start start 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c993e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb4c4162a00 0x7fb4c40d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c993e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb4c4162400 0x7fb4c40cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c993e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c40b1070 con 0x7fb4c4162020 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c993e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c40ce9d0 con 0x7fb4c4162400 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c993e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c40d2da0 con 0x7fb4c4162a00 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb4c4162400 0x7fb4c40cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb4c4162400 0x7fb4c40cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42906/0 (socket says 172.21.15.77:42906) 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c37fe640 1 -- 172.21.15.77:0/3921626219 learned_addr learned my addr 172.21.15.77:0/3921626219 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c27fc640 1 --2- 172.21.15.77:0/3921626219 >> v2:172.21.15.158:3300/0 conn(0x7fb4c4162a00 0x7fb4c40d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c37fe640 1 -- 172.21.15.77:0/3921626219 >> v2:172.21.15.158:3300/0 conn(0x7fb4c4162a00 msgr2=0x7fb4c40d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c37fe640 1 --2- 172.21.15.77:0/3921626219 >> v2:172.21.15.158:3300/0 conn(0x7fb4c4162a00 0x7fb4c40d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c37fe640 1 -- 172.21.15.77:0/3921626219 >> v1:172.21.15.158:6789/0 conn(0x7fb4c4162020 legacy=0x7fb4c40b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.654+0000 7fb4c37fe640 1 -- 172.21.15.77:0/3921626219 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c40d2500 con 0x7fb4c4162400 2024-03-06T07:22:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c37fe640 1 --2- 172.21.15.77:0/3921626219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162400 0x7fb4c40cf400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb4c408c0e0 tx=0x7fb4bc02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae39a89bde33e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c1ffb640 1 -- 172.21.15.77:0/3921626219 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc02cbb0 con 0x7fb4c4162400 2024-03-06T07:22:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c1ffb640 1 -- 172.21.15.77:0/3921626219 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4bc02cd50 con 0x7fb4c4162400 2024-03-06T07:22:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c1ffb640 1 -- 172.21.15.77:0/3921626219 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc034610 con 0x7fb4c4162400 2024-03-06T07:22:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c993e640 1 -- 172.21.15.77:0/3921626219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162400 msgr2=0x7fb4c40cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c993e640 1 --2- 172.21.15.77:0/3921626219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162400 0x7fb4c40cf400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb4c408c0e0 tx=0x7fb4bc02c010 comp rx=0 tx=0).stop 2024-03-06T07:22:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c993e640 1 -- 172.21.15.77:0/3921626219 shutdown_connections 2024-03-06T07:22:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c993e640 1 --2- 172.21.15.77:0/3921626219 >> v2:172.21.15.158:3300/0 conn(0x7fb4c4162a00 0x7fb4c40d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c993e640 1 --2- 172.21.15.77:0/3921626219 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162400 0x7fb4c40cf400 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c993e640 1 -- 172.21.15.77:0/3921626219 >> 172.21.15.77:0/3921626219 conn(0x7fb4c40d3d10 msgr2=0x7fb4c40d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c993e640 1 -- 172.21.15.77:0/3921626219 shutdown_connections 2024-03-06T07:22:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.655+0000 7fb4c993e640 1 -- 172.21.15.77:0/3921626219 wait complete. 2024-03-06T07:22:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.657+0000 7fb4c993e640 1 Processor -- start 2024-03-06T07:22:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.657+0000 7fb4c993e640 1 -- start start 2024-03-06T07:22:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.657+0000 7fb4c993e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4c4162020 0x7fb4c4156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.657+0000 7fb4c993e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4c4162400 0x7fb4c415fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.657+0000 7fb4c993e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162a00 0x7fb4c41602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.657+0000 7fb4c993e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c40cec00 con 0x7fb4c4162a00 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.657+0000 7fb4c993e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c40d2ac0 con 0x7fb4c4162400 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.657+0000 7fb4c993e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c40cf940 con 0x7fb4c4162020 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162a00 0x7fb4c41602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162a00 0x7fb4c41602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42916/0 (socket says 172.21.15.77:42916) 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 -- 172.21.15.77:0/911234000 learned_addr learned my addr 172.21.15.77:0/911234000 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c27fc640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4c4162400 0x7fb4c415fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c2ffd640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4c4162020 0x7fb4c4156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 -- 172.21.15.77:0/911234000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4c4162020 msgr2=0x7fb4c4156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4c4162020 0x7fb4c4156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 -- 172.21.15.77:0/911234000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4c4162400 msgr2=0x7fb4c415fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4c4162400 0x7fb4c415fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 -- 172.21.15.77:0/911234000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c4156ac0 con 0x7fb4c4162a00 2024-03-06T07:22:30.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c37fe640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162a00 0x7fb4c41602c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009f40 tx=0x7fb4bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4abfff640 1 -- 172.21.15.77:0/911234000 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc043020 con 0x7fb4c4162a00 2024-03-06T07:22:30.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4abfff640 1 -- 172.21.15.77:0/911234000 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4bc03e450 con 0x7fb4c4162a00 2024-03-06T07:22:30.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c40c96c0 con 0x7fb4c4162a00 2024-03-06T07:22:30.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4abfff640 1 -- 172.21.15.77:0/911234000 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc009510 con 0x7fb4c4162a00 2024-03-06T07:22:30.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.658+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c415da90 con 0x7fb4c4162a00 2024-03-06T07:22:30.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.659+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c404f330 con 0x7fb4c4162a00 2024-03-06T07:22:30.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.660+0000 7fb4abfff640 1 -- 172.21.15.77:0/911234000 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4bc046020 con 0x7fb4c4162a00 2024-03-06T07:22:30.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.660+0000 7fb4abfff640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b006b500 0x7fb4b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.660+0000 7fb4abfff640 1 -- 172.21.15.77:0/911234000 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4bc0ac830 con 0x7fb4c4162a00 2024-03-06T07:22:30.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.660+0000 7fb4c2ffd640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b006b500 0x7fb4b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.661+0000 7fb4c2ffd640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b006b500 0x7fb4b006d9c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb4c406c360 tx=0x7fb4b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.664+0000 7fb4abfff640 1 -- 172.21.15.77:0/911234000 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4bc04d030 con 0x7fb4c4162a00 2024-03-06T07:22:30.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.838+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4c4073380 con 0x7fb4c4162a00 2024-03-06T07:22:30.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.840+0000 7fb4abfff640 1 -- 172.21.15.77:0/911234000 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fb4bc07b180 con 0x7fb4c4162a00 2024-03-06T07:22:30.842 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:30.842 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.841+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b006b500 msgr2=0x7fb4b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.841+0000 7fb4c993e640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b006b500 0x7fb4b006d9c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb4c406c360 tx=0x7fb4b4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.841+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162a00 msgr2=0x7fb4c41602c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.841+0000 7fb4c993e640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162a00 0x7fb4c41602c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009f40 tx=0x7fb4bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.842+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 shutdown_connections 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.842+0000 7fb4c993e640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b006b500 0x7fb4b006d9c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.842+0000 7fb4c993e640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4c4162a00 0x7fb4c41602c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.842+0000 7fb4c993e640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4c4162400 0x7fb4c415fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.842+0000 7fb4c993e640 1 --2- 172.21.15.77:0/911234000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4c4162020 0x7fb4c4156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.842+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 >> 172.21.15.77:0/911234000 conn(0x7fb4c40d3d10 msgr2=0x7fb4c40b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.842+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 shutdown_connections 2024-03-06T07:22:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.842+0000 7fb4c993e640 1 -- 172.21.15.77:0/911234000 wait complete. 2024-03-06T07:22:30.855 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:30.855 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:30.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.957+0000 7f1add93e640 1 Processor -- start 2024-03-06T07:22:30.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.957+0000 7f1add93e640 1 -- start start 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.957+0000 7f1add93e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ad81614d0 0x7f1ad80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.957+0000 7f1add93e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ad80d1910 0x7f1ad80d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.957+0000 7f1add93e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d22e0 0x7f1ad80d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.957+0000 7f1add93e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad80d2750 con 0x7f1ad81614d0 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.957+0000 7f1add93e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad81618b0 con 0x7f1ad80d1910 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.957+0000 7f1add93e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80cf400 con 0x7f1ad80d22e0 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad6ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ad81614d0 0x7f1ad80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad6ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ad81614d0 0x7f1ad80d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42926/0 (socket says 172.21.15.77:42926) 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad6ffd640 1 -- 172.21.15.77:0/1452734453 learned_addr learned my addr 172.21.15.77:0/1452734453 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:30.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad77fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d22e0 0x7f1ad80d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad67fc640 1 --2- 172.21.15.77:0/1452734453 >> v2:172.21.15.158:3300/0 conn(0x7f1ad80d1910 0x7f1ad80d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad6ffd640 1 -- 172.21.15.77:0/1452734453 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d22e0 msgr2=0x7f1ad80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad6ffd640 1 --2- 172.21.15.77:0/1452734453 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d22e0 0x7f1ad80d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad6ffd640 1 -- 172.21.15.77:0/1452734453 >> v2:172.21.15.158:3300/0 conn(0x7f1ad80d1910 msgr2=0x7f1ad80d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad6ffd640 1 --2- 172.21.15.77:0/1452734453 >> v2:172.21.15.158:3300/0 conn(0x7f1ad80d1910 0x7f1ad80d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad6ffd640 1 -- 172.21.15.77:0/1452734453 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad80cf6a0 con 0x7f1ad81614d0 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.958+0000 7f1ad77fe640 1 --2- 172.21.15.77:0/1452734453 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d22e0 0x7f1ad80d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1ad6ffd640 1 --2- 172.21.15.77:0/1452734453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad81614d0 0x7f1ad80d4860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80ce550 tx=0x7f1ac802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b494e88e663396e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1ad5ffb640 1 -- 172.21.15.77:0/1452734453 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8035020 con 0x7f1ad81614d0 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1ad5ffb640 1 -- 172.21.15.77:0/1452734453 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac802d040 con 0x7f1ad81614d0 2024-03-06T07:22:30.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1ad5ffb640 1 -- 172.21.15.77:0/1452734453 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8031a80 con 0x7f1ad81614d0 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1add93e640 1 -- 172.21.15.77:0/1452734453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad81614d0 msgr2=0x7f1ad80d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1add93e640 1 --2- 172.21.15.77:0/1452734453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad81614d0 0x7f1ad80d4860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80ce550 tx=0x7f1ac802b900 comp rx=0 tx=0).stop 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1add93e640 1 -- 172.21.15.77:0/1452734453 shutdown_connections 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1add93e640 1 --2- 172.21.15.77:0/1452734453 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d22e0 0x7f1ad80d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1add93e640 1 --2- 172.21.15.77:0/1452734453 >> v2:172.21.15.158:3300/0 conn(0x7f1ad80d1910 0x7f1ad80d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1add93e640 1 --2- 172.21.15.77:0/1452734453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad81614d0 0x7f1ad80d4860 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1add93e640 1 -- 172.21.15.77:0/1452734453 >> 172.21.15.77:0/1452734453 conn(0x7f1ad80d3d40 msgr2=0x7f1ad80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.959+0000 7f1add93e640 1 -- 172.21.15.77:0/1452734453 shutdown_connections 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.960+0000 7f1add93e640 1 -- 172.21.15.77:0/1452734453 wait complete. 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.961+0000 7f1add93e640 1 Processor -- start 2024-03-06T07:22:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.961+0000 7f1add93e640 1 -- start start 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1add93e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d1910 0x7f1ad80c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1add93e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad80d22e0 0x7f1ad815a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1add93e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ad81614d0 0x7f1ad80c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1add93e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80b0910 con 0x7f1ad80d22e0 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1add93e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80cfc60 con 0x7f1ad81614d0 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1add93e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80d0be0 con 0x7f1ad80d1910 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad80d22e0 0x7f1ad815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad80d22e0 0x7f1ad815a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42928/0 (socket says 172.21.15.77:42928) 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 -- 172.21.15.77:0/1909667925 learned_addr learned my addr 172.21.15.77:0/1909667925 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad6ffd640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d1910 0x7f1ad80c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad77fe640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ad81614d0 0x7f1ad80c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 -- 172.21.15.77:0/1909667925 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d1910 msgr2=0x7f1ad80c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d1910 0x7f1ad80c8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 -- 172.21.15.77:0/1909667925 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ad81614d0 msgr2=0x7f1ad80c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ad81614d0 0x7f1ad80c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 -- 172.21.15.77:0/1909667925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad815b7a0 con 0x7f1ad80d22e0 2024-03-06T07:22:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1ad67fc640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad80d22e0 0x7f1ad815a660 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00e9e0 tx=0x7f1acc00ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.962+0000 7f1abffff640 1 -- 172.21.15.77:0/1909667925 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc017070 con 0x7f1ad80d22e0 2024-03-06T07:22:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.963+0000 7f1abffff640 1 -- 172.21.15.77:0/1909667925 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1acc00c940 con 0x7f1ad80d22e0 2024-03-06T07:22:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.963+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad80b74f0 con 0x7f1ad80d22e0 2024-03-06T07:22:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.963+0000 7f1abffff640 1 -- 172.21.15.77:0/1909667925 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc00cd30 con 0x7f1ad80d22e0 2024-03-06T07:22:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.963+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad80b7a70 con 0x7f1ad80d22e0 2024-03-06T07:22:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.964+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad8068ff0 con 0x7f1ad80d22e0 2024-03-06T07:22:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.964+0000 7f1abffff640 1 -- 172.21.15.77:0/1909667925 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1acc01a070 con 0x7f1ad80d22e0 2024-03-06T07:22:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.965+0000 7f1abffff640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.965+0000 7f1abffff640 1 -- 172.21.15.77:0/1909667925 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1acc08a0e0 con 0x7f1ad80d22e0 2024-03-06T07:22:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.965+0000 7f1ad6ffd640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.965+0000 7f1ad6ffd640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8036000 tx=0x7f1ac803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:30.969+0000 7f1abffff640 1 -- 172.21.15.77:0/1909667925 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1acc058910 con 0x7f1ad80d22e0 2024-03-06T07:22:31.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.142+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ad80c7f30 con 0x7f1ad80d22e0 2024-03-06T07:22:31.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.142+0000 7f1abffff640 1 -- 172.21.15.77:0/1909667925 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1acc05c9b0 con 0x7f1ad80d22e0 2024-03-06T07:22:31.144 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ac406b4d0 msgr2=0x7f1ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8036000 tx=0x7f1ac803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad80d22e0 msgr2=0x7f1ad815a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad80d22e0 0x7f1ad815a660 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00e9e0 tx=0x7f1acc00ede0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 shutdown_connections 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad80d22e0 0x7f1ad815a660 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ad81614d0 0x7f1ad80c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 --2- 172.21.15.77:0/1909667925 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ad80d1910 0x7f1ad80c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 >> 172.21.15.77:0/1909667925 conn(0x7f1ad80d3d40 msgr2=0x7f1ad80b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 shutdown_connections 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.144+0000 7f1add93e640 1 -- 172.21.15.77:0/1909667925 wait complete. 2024-03-06T07:22:31.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:31.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.259+0000 7ff650e5c640 1 Processor -- start 2024-03-06T07:22:31.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.259+0000 7ff650e5c640 1 -- start start 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.259+0000 7ff650e5c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff64c0cf9a0 0x7ff64c0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.259+0000 7ff650e5c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff64c066d90 0x7ff64c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.259+0000 7ff650e5c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c06b710 con 0x7ff64c0b5410 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.259+0000 7ff650e5c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64c0b5880 con 0x7ff64c0cf9a0 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.259+0000 7ff650e5c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64c06b150 con 0x7ff64c066d90 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.259+0000 7ff64a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff64c0cf9a0 0x7ff64c0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.260+0000 7ff64a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff64c0cf9a0 0x7ff64c0b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42936/0 (socket says 172.21.15.77:42936) 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.260+0000 7ff64a59c640 1 -- 172.21.15.77:0/146039939 learned_addr learned my addr 172.21.15.77:0/146039939 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.260+0000 7ff643fff640 1 --2- 172.21.15.77:0/146039939 >> v2:172.21.15.158:3300/0 conn(0x7ff64c066d90 0x7ff64c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.260+0000 7ff64a59c640 1 -- 172.21.15.77:0/146039939 >> v2:172.21.15.158:3300/0 conn(0x7ff64c066d90 msgr2=0x7ff64c0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.260+0000 7ff64a59c640 1 --2- 172.21.15.77:0/146039939 >> v2:172.21.15.158:3300/0 conn(0x7ff64c066d90 0x7ff64c0b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.260+0000 7ff64a59c640 1 -- 172.21.15.77:0/146039939 >> v1:172.21.15.77:6789/0 conn(0x7ff64c0b5410 legacy=0x7ff64c072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.260+0000 7ff64a59c640 1 -- 172.21.15.77:0/146039939 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c06b430 con 0x7ff64c0cf9a0 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff64a59c640 1 --2- 172.21.15.77:0/146039939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0cf9a0 0x7ff64c0b5ab0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0d11d0 tx=0x7ff63c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4569059ece3d091c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff649d9b640 1 -- 172.21.15.77:0/146039939 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c034070 con 0x7ff64c0cf9a0 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff649d9b640 1 -- 172.21.15.77:0/146039939 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c030860 con 0x7ff64c0cf9a0 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff649d9b640 1 -- 172.21.15.77:0/146039939 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c030c30 con 0x7ff64c0cf9a0 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff650e5c640 1 -- 172.21.15.77:0/146039939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0cf9a0 msgr2=0x7ff64c0b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff650e5c640 1 --2- 172.21.15.77:0/146039939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0cf9a0 0x7ff64c0b5ab0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0d11d0 tx=0x7ff63c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff650e5c640 1 -- 172.21.15.77:0/146039939 shutdown_connections 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff650e5c640 1 --2- 172.21.15.77:0/146039939 >> v2:172.21.15.158:3300/0 conn(0x7ff64c066d90 0x7ff64c0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff650e5c640 1 --2- 172.21.15.77:0/146039939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0cf9a0 0x7ff64c0b5ab0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff650e5c640 1 -- 172.21.15.77:0/146039939 >> 172.21.15.77:0/146039939 conn(0x7ff64c0b1d70 msgr2=0x7ff64c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff650e5c640 1 -- 172.21.15.77:0/146039939 shutdown_connections 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.261+0000 7ff650e5c640 1 -- 172.21.15.77:0/146039939 wait complete. 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff650e5c640 1 Processor -- start 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff650e5c640 1 -- start start 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff650e5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c066d90 0x7ff64c06e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff650e5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff64c0b5410 0x7ff64c09a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff650e5c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64c0cf9a0 0x7ff64c09d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff650e5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0b5ff0 con 0x7ff64c066d90 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff650e5c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0b0c50 con 0x7ff64c0cf9a0 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff650e5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0b6170 con 0x7ff64c0b5410 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff64a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c066d90 0x7ff64c06e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff64a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c066d90 0x7ff64c06e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42946/0 (socket says 172.21.15.77:42946) 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.263+0000 7ff64a59c640 1 -- 172.21.15.77:0/3447537563 learned_addr learned my addr 172.21.15.77:0/3447537563 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff643fff640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff64c0b5410 0x7ff64c09a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff64ad9d640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64c0cf9a0 0x7ff64c09d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff64a59c640 1 -- 172.21.15.77:0/3447537563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff64c0b5410 msgr2=0x7ff64c09a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff64a59c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff64c0b5410 0x7ff64c09a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff64a59c640 1 -- 172.21.15.77:0/3447537563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64c0cf9a0 msgr2=0x7ff64c09d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff64a59c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64c0cf9a0 0x7ff64c09d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff64a59c640 1 -- 172.21.15.77:0/3447537563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c0cd900 con 0x7ff64c066d90 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff64a59c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c066d90 0x7ff64c06e540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000c00 tx=0x7ff63c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff642ffd640 1 -- 172.21.15.77:0/3447537563 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c034040 con 0x7ff64c066d90 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff642ffd640 1 -- 172.21.15.77:0/3447537563 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c039d90 con 0x7ff64c066d90 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c09ce20 con 0x7ff64c066d90 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff642ffd640 1 -- 172.21.15.77:0/3447537563 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c043610 con 0x7ff64c066d90 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.264+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c0cd560 con 0x7ff64c066d90 2024-03-06T07:22:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.265+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c09e3d0 con 0x7ff64c066d90 2024-03-06T07:22:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.266+0000 7ff642ffd640 1 -- 172.21.15.77:0/3447537563 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff63c037070 con 0x7ff64c066d90 2024-03-06T07:22:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.266+0000 7ff642ffd640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff62c06b500 0x7ff62c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.266+0000 7ff642ffd640 1 -- 172.21.15.77:0/3447537563 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff63c0ac100 con 0x7ff64c066d90 2024-03-06T07:22:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.266+0000 7ff643fff640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff62c06b500 0x7ff62c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.266+0000 7ff643fff640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff62c06b500 0x7ff62c06d9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff64c158060 tx=0x7ff634009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.270+0000 7ff642ffd640 1 -- 172.21.15.77:0/3447537563 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff63c03e070 con 0x7ff64c066d90 2024-03-06T07:22:31.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.441+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff64c068ff0 con 0x7ff64c066d90 2024-03-06T07:22:31.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.441+0000 7ff642ffd640 1 -- 172.21.15.77:0/3447537563 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff63c07a9c0 con 0x7ff64c066d90 2024-03-06T07:22:31.443 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:31.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff62c06b500 msgr2=0x7ff62c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff62c06b500 0x7ff62c06d9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff64c158060 tx=0x7ff634009190 comp rx=0 tx=0).stop 2024-03-06T07:22:31.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c066d90 msgr2=0x7ff64c06e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c066d90 0x7ff64c06e540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff63c000c00 tx=0x7ff63c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 shutdown_connections 2024-03-06T07:22:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff64c0cf9a0 0x7ff64c09d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff64c0b5410 0x7ff64c09a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff62c06b500 0x7ff62c06d9c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 --2- 172.21.15.77:0/3447537563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c066d90 0x7ff64c06e540 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 >> 172.21.15.77:0/3447537563 conn(0x7ff64c0b1d70 msgr2=0x7ff64c0af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 shutdown_connections 2024-03-06T07:22:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.443+0000 7ff650e5c640 1 -- 172.21.15.77:0/3447537563 wait complete. 2024-03-06T07:22:31.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:31.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.558+0000 7fbd5973e640 1 Processor -- start 2024-03-06T07:22:31.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.558+0000 7fbd5973e640 1 -- start start 2024-03-06T07:22:31.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd5973e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbd541614d0 0x7fbd540d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd5973e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd541618b0 con 0x7fbd540d1910 2024-03-06T07:22:31.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd5973e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd540cf400 con 0x7fbd540d24b0 2024-03-06T07:22:31.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd5973e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd540cf6a0 con 0x7fbd541614d0 2024-03-06T07:22:31.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd4bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbd540d1910 0x7fbd540d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52454/0 (socket says 172.21.15.77:52454) 2024-03-06T07:22:31.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd4bfff640 1 -- 172.21.15.77:0/786976439 learned_addr learned my addr 172.21.15.77:0/786976439 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:31.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd52ffd640 1 --2- 172.21.15.77:0/786976439 >> v2:172.21.15.158:3300/0 conn(0x7fbd541614d0 0x7fbd540d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd527fc640 1 -- 172.21.15.77:0/786976439 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605741913 0 0) 0x7fbd541618b0 con 0x7fbd540d1910 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd527fc640 1 -- 172.21.15.77:0/786976439 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd40003520 con 0x7fbd540d1910 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd527fc640 1 -- 172.21.15.77:0/786976439 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832507164 0 0) 0x7fbd540cf400 con 0x7fbd540d24b0 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd527fc640 1 -- 172.21.15.77:0/786976439 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd541618b0 con 0x7fbd540d24b0 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd52ffd640 1 -- 172.21.15.77:0/786976439 >> v1:172.21.15.158:6789/0 conn(0x7fbd540d24b0 legacy=0x7fbd540b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd52ffd640 1 -- 172.21.15.77:0/786976439 >> v1:172.21.15.77:6789/0 conn(0x7fbd540d1910 legacy=0x7fbd540d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.559+0000 7fbd52ffd640 1 -- 172.21.15.77:0/786976439 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd540cf980 con 0x7fbd541614d0 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.560+0000 7fbd52ffd640 1 --2- 172.21.15.77:0/786976439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd541614d0 0x7fbd540d4860 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbd440097a0 tx=0x7fbd4402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cf68c88292ee8bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.560+0000 7fbd527fc640 1 -- 172.21.15.77:0/786976439 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44034070 con 0x7fbd541614d0 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.560+0000 7fbd527fc640 1 -- 172.21.15.77:0/786976439 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd44030610 con 0x7fbd541614d0 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.560+0000 7fbd527fc640 1 -- 172.21.15.77:0/786976439 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd440309c0 con 0x7fbd541614d0 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.561+0000 7fbd5973e640 1 -- 172.21.15.77:0/786976439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd541614d0 msgr2=0x7fbd540d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.561+0000 7fbd5973e640 1 --2- 172.21.15.77:0/786976439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd541614d0 0x7fbd540d4860 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbd440097a0 tx=0x7fbd4402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.561+0000 7fbd5973e640 1 -- 172.21.15.77:0/786976439 shutdown_connections 2024-03-06T07:22:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.561+0000 7fbd5973e640 1 --2- 172.21.15.77:0/786976439 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd541614d0 0x7fbd540d4860 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.561+0000 7fbd5973e640 1 -- 172.21.15.77:0/786976439 >> 172.21.15.77:0/786976439 conn(0x7fbd540d3d40 msgr2=0x7fbd540d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.561+0000 7fbd5973e640 1 -- 172.21.15.77:0/786976439 shutdown_connections 2024-03-06T07:22:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.561+0000 7fbd5973e640 1 -- 172.21.15.77:0/786976439 wait complete. 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.562+0000 7fbd5973e640 1 Processor -- start 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.562+0000 7fbd5973e640 1 -- start start 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.562+0000 7fbd5973e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd540d1910 0x7fbd5415af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd5973e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbd540d24b0 0x7fbd540b6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd5973e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd541614d0 0x7fbd5415f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd5973e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540b1540 con 0x7fbd541614d0 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd5973e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540b0910 con 0x7fbd540d1910 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd5973e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540b16c0 con 0x7fbd540d24b0 2024-03-06T07:22:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd541614d0 0x7fbd5415f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd541614d0 0x7fbd5415f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42950/0 (socket says 172.21.15.77:42950) 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 -- 172.21.15.77:0/2565044004 learned_addr learned my addr 172.21.15.77:0/2565044004 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd52ffd640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd540d1910 0x7fbd5415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd4bfff640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbd540d24b0 0x7fbd540b6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 -- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbd540d24b0 msgr2=0x7fbd540b6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbd540d24b0 0x7fbd540b6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 -- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd540d1910 msgr2=0x7fbd5415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd540d1910 0x7fbd5415af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 -- 172.21.15.77:0/2565044004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd540c7d00 con 0x7fbd541614d0 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd4bfff640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbd540d24b0 0x7fbd540b6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd537fe640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd541614d0 0x7fbd5415f600 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c770 tx=0x7fbd4c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd4b7fe640 1 -- 172.21.15.77:0/2565044004 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c003b70 con 0x7fbd541614d0 2024-03-06T07:22:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd4b7fe640 1 -- 172.21.15.77:0/2565044004 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c003d30 con 0x7fbd541614d0 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd540b8260 con 0x7fbd541614d0 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd4b7fe640 1 -- 172.21.15.77:0/2565044004 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c00aa90 con 0x7fbd541614d0 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.563+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd540b8820 con 0x7fbd541614d0 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.564+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd540cdfd0 con 0x7fbd541614d0 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.565+0000 7fbd4b7fe640 1 -- 172.21.15.77:0/2565044004 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd4c00ac30 con 0x7fbd541614d0 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.565+0000 7fbd4b7fe640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbd3406b4d0 0x7fbd3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.565+0000 7fbd4b7fe640 1 -- 172.21.15.77:0/2565044004 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd4c08a180 con 0x7fbd541614d0 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.565+0000 7fbd52ffd640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbd3406b4d0 0x7fbd3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.566+0000 7fbd52ffd640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbd3406b4d0 0x7fbd3406d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbd44036000 tx=0x7fbd4403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.569+0000 7fbd4b7fe640 1 -- 172.21.15.77:0/2565044004 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd4c058f80 con 0x7fbd541614d0 2024-03-06T07:22:31.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.740+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd54068ff0 con 0x7fbd541614d0 2024-03-06T07:22:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.741+0000 7fbd4b7fe640 1 -- 172.21.15.77:0/2565044004 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbd4c058880 con 0x7fbd541614d0 2024-03-06T07:22:31.743 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbd3406b4d0 msgr2=0x7fbd3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbd3406b4d0 0x7fbd3406d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbd44036000 tx=0x7fbd4403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd541614d0 msgr2=0x7fbd5415f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd541614d0 0x7fbd5415f600 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c770 tx=0x7fbd4c00cc30 comp rx=0 tx=0).stop 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 shutdown_connections 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbd3406b4d0 0x7fbd3406d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd541614d0 0x7fbd5415f600 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbd540d24b0 0x7fbd540b6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 --2- 172.21.15.77:0/2565044004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbd540d1910 0x7fbd5415af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 >> 172.21.15.77:0/2565044004 conn(0x7fbd540d3d40 msgr2=0x7fbd540ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 shutdown_connections 2024-03-06T07:22:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.743+0000 7fbd5973e640 1 -- 172.21.15.77:0/2565044004 wait complete. 2024-03-06T07:22:31.756 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:31.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.854+0000 7fe30e2df640 1 Processor -- start 2024-03-06T07:22:31.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.854+0000 7fe30e2df640 1 -- start start 2024-03-06T07:22:31.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.855+0000 7fe30e2df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe308162970 0x7fe3080d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.855+0000 7fe30e2df640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3080d22c0 con 0x7fe308161cf0 2024-03-06T07:22:31.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.855+0000 7fe30e2df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3080d25a0 con 0x7fe3080b4460 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.855+0000 7fe30e2df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe308161b00 con 0x7fe308162970 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.855+0000 7fe3077fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe308162970 0x7fe3080d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.855+0000 7fe3077fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe308162970 0x7fe3080d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42964/0 (socket says 172.21.15.77:42964) 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.855+0000 7fe30c87c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe308161cf0 0x7fe3080cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52458/0 (socket says 172.21.15.77:52458) 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.855+0000 7fe3077fe640 1 -- 172.21.15.77:0/2892934407 learned_addr learned my addr 172.21.15.77:0/2892934407 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.856+0000 7fe3077fe640 1 -- 172.21.15.77:0/2892934407 >> v1:172.21.15.158:6789/0 conn(0x7fe3080b4460 legacy=0x7fe3080b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.856+0000 7fe3077fe640 1 -- 172.21.15.77:0/2892934407 >> v1:172.21.15.77:6789/0 conn(0x7fe308161cf0 legacy=0x7fe3080cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.856+0000 7fe3077fe640 1 -- 172.21.15.77:0/2892934407 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080d2880 con 0x7fe308162970 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.856+0000 7fe3077fe640 1 --2- 172.21.15.77:0/2892934407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe308162970 0x7fe3080d4800 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009870 tx=0x7fe2fc02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc5d71fa446a1ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.856+0000 7fe306ffd640 1 -- 172.21.15.77:0/2892934407 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc02f840 con 0x7fe308162970 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.856+0000 7fe306ffd640 1 -- 172.21.15.77:0/2892934407 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc02f9e0 con 0x7fe308162970 2024-03-06T07:22:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.857+0000 7fe306ffd640 1 -- 172.21.15.77:0/2892934407 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc02fdb0 con 0x7fe308162970 2024-03-06T07:22:31.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.857+0000 7fe30e2df640 1 -- 172.21.15.77:0/2892934407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe308162970 msgr2=0x7fe3080d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.857+0000 7fe30e2df640 1 --2- 172.21.15.77:0/2892934407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe308162970 0x7fe3080d4800 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009870 tx=0x7fe2fc02ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:31.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.857+0000 7fe30e2df640 1 -- 172.21.15.77:0/2892934407 shutdown_connections 2024-03-06T07:22:31.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.857+0000 7fe30e2df640 1 --2- 172.21.15.77:0/2892934407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe308162970 0x7fe3080d4800 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.857+0000 7fe30e2df640 1 -- 172.21.15.77:0/2892934407 >> 172.21.15.77:0/2892934407 conn(0x7fe3080d3ce0 msgr2=0x7fe3080d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.857+0000 7fe30e2df640 1 -- 172.21.15.77:0/2892934407 shutdown_connections 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.857+0000 7fe30e2df640 1 -- 172.21.15.77:0/2892934407 wait complete. 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.858+0000 7fe30e2df640 1 Processor -- start 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.858+0000 7fe30e2df640 1 -- start start 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe30e2df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3080b4460 0x7fe3080cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe30e2df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe308161cf0 0x7fe3080c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe30e2df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe308162970 0x7fe3080c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe30e2df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080ce950 con 0x7fe3080b4460 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe30e2df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080cecc0 con 0x7fe308161cf0 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe30e2df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080b4840 con 0x7fe308162970 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe307fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3080b4460 0x7fe3080cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe307fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3080b4460 0x7fe3080cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42966/0 (socket says 172.21.15.77:42966) 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe307fff640 1 -- 172.21.15.77:0/3056978239 learned_addr learned my addr 172.21.15.77:0/3056978239 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:31.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe3077fe640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe308161cf0 0x7fe3080c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe30c87c640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe308162970 0x7fe3080c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe307fff640 1 -- 172.21.15.77:0/3056978239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe308162970 msgr2=0x7fe3080c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.859+0000 7fe307fff640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe308162970 0x7fe3080c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe307fff640 1 -- 172.21.15.77:0/3056978239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe308161cf0 msgr2=0x7fe3080c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe307fff640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe308161cf0 0x7fe3080c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe307fff640 1 -- 172.21.15.77:0/3056978239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080cccf0 con 0x7fe3080b4460 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe307fff640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3080b4460 0x7fe3080cc7b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800c3c0 tx=0x7fe2f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe304ff9640 1 -- 172.21.15.77:0/3056978239 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80052d0 con 0x7fe3080b4460 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe304ff9640 1 -- 172.21.15.77:0/3056978239 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f8005490 con 0x7fe3080b4460 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3081f4650 con 0x7fe3080b4460 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe304ff9640 1 -- 172.21.15.77:0/3056978239 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8007260 con 0x7fe3080b4460 2024-03-06T07:22:31.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.860+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3081f4c10 con 0x7fe3080b4460 2024-03-06T07:22:31.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.861+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30804f330 con 0x7fe3080b4460 2024-03-06T07:22:31.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.861+0000 7fe304ff9640 1 -- 172.21.15.77:0/3056978239 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2f8007400 con 0x7fe3080b4460 2024-03-06T07:22:31.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.862+0000 7fe304ff9640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.862+0000 7fe3077fe640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.862+0000 7fe304ff9640 1 -- 172.21.15.77:0/3056978239 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2f8089630 con 0x7fe3080b4460 2024-03-06T07:22:31.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.862+0000 7fe3077fe640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc036000 tx=0x7fe2fc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:31.866+0000 7fe304ff9640 1 -- 172.21.15.77:0/3056978239 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2f8057e60 con 0x7fe3080b4460 2024-03-06T07:22:32.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.050+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe3080ccf20 con 0x7fe3080b4460 2024-03-06T07:22:32.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.050+0000 7fe304ff9640 1 -- 172.21.15.77:0/3056978239 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe2f805ce20 con 0x7fe3080b4460 2024-03-06T07:22:32.052 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:22:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2f406b4d0 msgr2=0x7fe2f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc036000 tx=0x7fe2fc008030 comp rx=0 tx=0).stop 2024-03-06T07:22:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3080b4460 msgr2=0x7fe3080cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3080b4460 0x7fe3080cc7b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800c3c0 tx=0x7fe2f800c880 comp rx=0 tx=0).stop 2024-03-06T07:22:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 shutdown_connections 2024-03-06T07:22:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe308162970 0x7fe3080c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe308161cf0 0x7fe3080c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 --2- 172.21.15.77:0/3056978239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3080b4460 0x7fe3080cc7b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 >> 172.21.15.77:0/3056978239 conn(0x7fe3080d3ce0 msgr2=0x7fe3080d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 shutdown_connections 2024-03-06T07:22:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.052+0000 7fe30e2df640 1 -- 172.21.15.77:0/3056978239 wait complete. 2024-03-06T07:22:32.064 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:22:32.065 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:32.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:32.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.218+0000 7ff1aecdf640 1 Processor -- start 2024-03-06T07:22:32.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.218+0000 7ff1aecdf640 1 -- start start 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1aecdf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff1a80cdeb0 0x7ff1a815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1aecdf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a8071f40 con 0x7ff1a80cce40 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1aecdf640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a8070a70 con 0x7ff1a81515e0 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1aecdf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a80cd2f0 con 0x7ff1a80cdeb0 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1a7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff1a80cce40 0x7ff1a80cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52474/0 (socket says 172.21.15.77:52474) 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1a7fff640 1 -- 172.21.15.77:0/975636406 learned_addr learned my addr 172.21.15.77:0/975636406 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1ad27c640 1 --2- 172.21.15.77:0/975636406 >> v2:172.21.15.158:3300/0 conn(0x7ff1a80cdeb0 0x7ff1a815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1a77fe640 1 -- 172.21.15.77:0/975636406 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202559151 0 0) 0x7ff1a8071f40 con 0x7ff1a80cce40 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1a77fe640 1 -- 172.21.15.77:0/975636406 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff198003520 con 0x7ff1a80cce40 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1ad27c640 1 -- 172.21.15.77:0/975636406 >> v1:172.21.15.158:6789/0 conn(0x7ff1a81515e0 legacy=0x7ff1a8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1a77fe640 1 -- 172.21.15.77:0/975636406 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460293897 0 0) 0x7ff1a8070a70 con 0x7ff1a81515e0 2024-03-06T07:22:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1ad27c640 1 -- 172.21.15.77:0/975636406 >> v1:172.21.15.77:6789/0 conn(0x7ff1a80cce40 legacy=0x7ff1a80cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.219+0000 7ff1ad27c640 1 -- 172.21.15.77:0/975636406 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a8070d50 con 0x7ff1a80cdeb0 2024-03-06T07:22:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.220+0000 7ff1ad27c640 1 --2- 172.21.15.77:0/975636406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 0x7ff1a815b430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009d90 tx=0x7ff1a002bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed9c584bb0843df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.220+0000 7ff1a77fe640 1 -- 172.21.15.77:0/975636406 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0039070 con 0x7ff1a80cdeb0 2024-03-06T07:22:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.220+0000 7ff1a77fe640 1 -- 172.21.15.77:0/975636406 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a002c860 con 0x7ff1a80cdeb0 2024-03-06T07:22:32.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.220+0000 7ff1a77fe640 1 -- 172.21.15.77:0/975636406 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194001690 con 0x7ff1a80cdeb0 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.221+0000 7ff1aecdf640 1 -- 172.21.15.77:0/975636406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 msgr2=0x7ff1a815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.221+0000 7ff1aecdf640 1 --2- 172.21.15.77:0/975636406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 0x7ff1a815b430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009d90 tx=0x7ff1a002bd30 comp rx=0 tx=0).stop 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.221+0000 7ff1aecdf640 1 -- 172.21.15.77:0/975636406 shutdown_connections 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.221+0000 7ff1aecdf640 1 --2- 172.21.15.77:0/975636406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 0x7ff1a815b430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.221+0000 7ff1aecdf640 1 -- 172.21.15.77:0/975636406 >> 172.21.15.77:0/975636406 conn(0x7ff1a80cf8a0 msgr2=0x7ff1a80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.221+0000 7ff1aecdf640 1 -- 172.21.15.77:0/975636406 shutdown_connections 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.221+0000 7ff1aecdf640 1 -- 172.21.15.77:0/975636406 wait complete. 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.222+0000 7ff1aecdf640 1 Processor -- start 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.222+0000 7ff1aecdf640 1 -- start start 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aecdf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a80cce40 0x7ff1a814e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aecdf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 0x7ff1a8152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aecdf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1a81515e0 0x7ff1a814b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aecdf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80cc3a0 con 0x7ff1a80cce40 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aecdf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a8071310 con 0x7ff1a80cdeb0 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aecdf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80cc520 con 0x7ff1a81515e0 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a80cce40 0x7ff1a814e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a80cce40 0x7ff1a814e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42976/0 (socket says 172.21.15.77:42976) 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 -- 172.21.15.77:0/2459360394 learned_addr learned my addr 172.21.15.77:0/2459360394 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1a7fff640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 0x7ff1a8152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1ad27c640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1a81515e0 0x7ff1a814b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 -- 172.21.15.77:0/2459360394 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1a81515e0 msgr2=0x7ff1a814b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1a81515e0 0x7ff1a814b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 -- 172.21.15.77:0/2459360394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 msgr2=0x7ff1a8152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 0x7ff1a8152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 -- 172.21.15.77:0/2459360394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a814d4c0 con 0x7ff1a80cce40 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aca7b640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a80cce40 0x7ff1a814e280 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00c410 tx=0x7ff19c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1a57fa640 1 -- 172.21.15.77:0/2459360394 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c017070 con 0x7ff1a80cce40 2024-03-06T07:22:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1a57fa640 1 -- 172.21.15.77:0/2459360394 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c0035f0 con 0x7ff1a80cce40 2024-03-06T07:22:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a814d7a0 con 0x7ff1a80cce40 2024-03-06T07:22:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1a57fa640 1 -- 172.21.15.77:0/2459360394 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0039e0 con 0x7ff1a80cce40 2024-03-06T07:22:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.223+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a814dd60 con 0x7ff1a80cce40 2024-03-06T07:22:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.224+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a8068840 con 0x7ff1a80cce40 2024-03-06T07:22:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.225+0000 7ff1a57fa640 1 -- 172.21.15.77:0/2459360394 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff19c01a070 con 0x7ff1a80cce40 2024-03-06T07:22:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.225+0000 7ff1a57fa640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff18c06b4d0 0x7ff18c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.225+0000 7ff1a57fa640 1 -- 172.21.15.77:0/2459360394 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff19c0895a0 con 0x7ff1a80cce40 2024-03-06T07:22:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.225+0000 7ff1a7fff640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff18c06b4d0 0x7ff18c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.226+0000 7ff1a7fff640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff18c06b4d0 0x7ff18c06d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff194001eb0 tx=0x7ff1940048e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.229+0000 7ff1a57fa640 1 -- 172.21.15.77:0/2459360394 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff19c057dd0 con 0x7ff1a80cce40 2024-03-06T07:22:32.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.403+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1a804f330 con 0x7ff1a80cce40 2024-03-06T07:22:32.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.404+0000 7ff1a57fa640 1 -- 172.21.15.77:0/2459360394 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7ff19c05be70 con 0x7ff1a80cce40 2024-03-06T07:22:32.407 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:32.407 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:32.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.406+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff18c06b4d0 msgr2=0x7ff18c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.406+0000 7ff1aecdf640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff18c06b4d0 0x7ff18c06d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff194001eb0 tx=0x7ff1940048e0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.406+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a80cce40 msgr2=0x7ff1a814e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.406+0000 7ff1aecdf640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a80cce40 0x7ff1a814e280 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00c410 tx=0x7ff19c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.407+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 shutdown_connections 2024-03-06T07:22:32.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.407+0000 7ff1aecdf640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1a81515e0 0x7ff1a814b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.407+0000 7ff1aecdf640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1a80cdeb0 0x7ff1a8152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.407+0000 7ff1aecdf640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff18c06b4d0 0x7ff18c06d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.407+0000 7ff1aecdf640 1 --2- 172.21.15.77:0/2459360394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a80cce40 0x7ff1a814e280 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.407+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 >> 172.21.15.77:0/2459360394 conn(0x7ff1a80cf8a0 msgr2=0x7ff1a814f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.407+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 shutdown_connections 2024-03-06T07:22:32.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.407+0000 7ff1aecdf640 1 -- 172.21.15.77:0/2459360394 wait complete. 2024-03-06T07:22:32.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:32.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:32.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.521+0000 7f5de3713640 1 Processor -- start 2024-03-06T07:22:32.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.522+0000 7f5de3713640 1 -- start start 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.522+0000 7f5de3713640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc15ecd0 0x7f5ddc0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.522+0000 7f5de3713640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ddc0cfa50 con 0x7f5ddc0d10c0 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.522+0000 7f5de3713640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ddc0b45c0 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.522+0000 7f5de3713640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc15c050 con 0x7f5ddc15ecd0 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5de14af640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5ddc0d10c0 0x7f5ddc0d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52480/0 (socket says 172.21.15.77:52480) 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5de14af640 1 -- 172.21.15.77:0/1099874269 learned_addr learned my addr 172.21.15.77:0/1099874269 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5de1cb0640 1 --2- 172.21.15.77:0/1099874269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc15ecd0 0x7f5ddc0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5dd3fff640 1 -- 172.21.15.77:0/1099874269 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303105702 0 0) 0x7f5ddc0cfa50 con 0x7f5ddc0d10c0 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5dd3fff640 1 -- 172.21.15.77:0/1099874269 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc8003520 con 0x7f5ddc0d10c0 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5dd3fff640 1 -- 172.21.15.77:0/1099874269 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304473959 0 0) 0x7f5ddc0b45c0 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5dd3fff640 1 -- 172.21.15.77:0/1099874269 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ddc0cfa50 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5de1cb0640 1 -- 172.21.15.77:0/1099874269 >> v1:172.21.15.158:6789/0 conn(0x7f5ddc0d14a0 legacy=0x7f5ddc15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5de1cb0640 1 -- 172.21.15.77:0/1099874269 >> v1:172.21.15.77:6789/0 conn(0x7f5ddc0d10c0 legacy=0x7f5ddc0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.523+0000 7f5de1cb0640 1 -- 172.21.15.77:0/1099874269 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc15f160 con 0x7f5ddc15ecd0 2024-03-06T07:22:32.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5de1cb0640 1 --2- 172.21.15.77:0/1099874269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc15ecd0 0x7f5ddc0cae80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009dc0 tx=0x7f5dd402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12bcfe71a0d5f62e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5dd3fff640 1 -- 172.21.15.77:0/1099874269 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4039070 con 0x7f5ddc15ecd0 2024-03-06T07:22:32.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5dd3fff640 1 -- 172.21.15.77:0/1099874269 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd402ccd0 con 0x7f5ddc15ecd0 2024-03-06T07:22:32.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5dd3fff640 1 -- 172.21.15.77:0/1099874269 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4034640 con 0x7f5ddc15ecd0 2024-03-06T07:22:32.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5de3713640 1 -- 172.21.15.77:0/1099874269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc15ecd0 msgr2=0x7f5ddc0cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5de3713640 1 --2- 172.21.15.77:0/1099874269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc15ecd0 0x7f5ddc0cae80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009dc0 tx=0x7f5dd402bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:32.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5de3713640 1 -- 172.21.15.77:0/1099874269 shutdown_connections 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5de3713640 1 --2- 172.21.15.77:0/1099874269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc15ecd0 0x7f5ddc0cae80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.524+0000 7f5de3713640 1 -- 172.21.15.77:0/1099874269 >> 172.21.15.77:0/1099874269 conn(0x7f5ddc0b3800 msgr2=0x7f5ddc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.525+0000 7f5de3713640 1 -- 172.21.15.77:0/1099874269 shutdown_connections 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.525+0000 7f5de3713640 1 -- 172.21.15.77:0/1099874269 wait complete. 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de3713640 1 Processor -- start 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de3713640 1 -- start start 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de3713640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ddc0d10c0 0x7f5ddc14e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de3713640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ddc0d14a0 0x7f5ddc0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de3713640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc0b0b20 0x7f5ddc0b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de3713640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0cfcf0 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de3713640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0d0060 con 0x7f5ddc0d10c0 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de3713640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0cb3c0 con 0x7f5ddc0b0b20 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de0cae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ddc0d14a0 0x7f5ddc0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de0cae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ddc0d14a0 0x7f5ddc0b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42978/0 (socket says 172.21.15.77:42978) 2024-03-06T07:22:32.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de0cae640 1 -- 172.21.15.77:0/1816166488 learned_addr learned my addr 172.21.15.77:0/1816166488 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.526+0000 7f5de14af640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ddc0d10c0 0x7f5ddc14e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de1cb0640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc0b0b20 0x7f5ddc0b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de0cae640 1 -- 172.21.15.77:0/1816166488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc0b0b20 msgr2=0x7f5ddc0b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de0cae640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc0b0b20 0x7f5ddc0b63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de0cae640 1 -- 172.21.15.77:0/1816166488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ddc0d10c0 msgr2=0x7f5ddc14e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de0cae640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ddc0d10c0 0x7f5ddc14e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de0cae640 1 -- 172.21.15.77:0/1816166488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc0b7140 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de0cae640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ddc0d14a0 0x7f5ddc0b05e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00c500 tx=0x7f5dcc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5dd1ffb640 1 -- 172.21.15.77:0/1816166488 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc0052d0 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ddc0b7420 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5dd1ffb640 1 -- 172.21.15.77:0/1816166488 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc005490 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5dd1ffb640 1 -- 172.21.15.77:0/1816166488 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc012870 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.527+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ddc0cc3d0 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.528+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ddc0b0f30 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.529+0000 7f5dd1ffb640 1 -- 172.21.15.77:0/1816166488 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5dcc012a10 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.529+0000 7f5dd1ffb640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dc006b4d0 0x7f5dc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.529+0000 7f5dd1ffb640 1 -- 172.21.15.77:0/1816166488 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5dcc089e00 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.529+0000 7f5de14af640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dc006b4d0 0x7f5dc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.530+0000 7f5de14af640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dc006b4d0 0x7f5dc006d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80011e0 tx=0x7f5dd8007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.533+0000 7f5dd1ffb640 1 -- 172.21.15.77:0/1816166488 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ddc0b0f30 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.704+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ddc09fe60 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.704+0000 7f5dd1ffb640 1 -- 172.21.15.77:0/1816166488 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5dcc058640 con 0x7f5ddc0d14a0 2024-03-06T07:22:32.707 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:32.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dc006b4d0 msgr2=0x7f5dc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dc006b4d0 0x7f5dc006d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80011e0 tx=0x7f5dd8007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ddc0d14a0 msgr2=0x7f5ddc0b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ddc0d14a0 0x7f5ddc0b05e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00c500 tx=0x7f5dcc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 shutdown_connections 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dc006b4d0 0x7f5dc006d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ddc0d14a0 0x7f5ddc0b05e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ddc0b0b20 0x7f5ddc0b63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 --2- 172.21.15.77:0/1816166488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ddc0d10c0 0x7f5ddc14e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.706+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 >> 172.21.15.77:0/1816166488 conn(0x7f5ddc0b3800 msgr2=0x7f5ddc15e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.707+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 shutdown_connections 2024-03-06T07:22:32.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.707+0000 7f5de3713640 1 -- 172.21.15.77:0/1816166488 wait complete. 2024-03-06T07:22:32.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.821+0000 7fc5f23a0640 1 Processor -- start 2024-03-06T07:22:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.821+0000 7fc5f23a0640 1 -- start start 2024-03-06T07:22:32.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f23a0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc5ec15e010 0x7fc5ec0d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f23a0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d19d0 0x7fc5ec0d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f23a0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc5ec0d2660 0x7fc5ec0ce650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f23a0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec0b3570 con 0x7fc5ec15e010 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f23a0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec0d0ae0 con 0x7fc5ec0d2660 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f23a0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0d0dc0 con 0x7fc5ec0d19d0 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f139e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc5ec15e010 0x7fc5ec0d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f139e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc5ec15e010 0x7fc5ec0d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42982/0 (socket says 172.21.15.77:42982) 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f0b9d640 1 --2- 172.21.15.77:0/4175712024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d19d0 0x7fc5ec0d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f139e640 1 -- 172.21.15.77:0/4175712024 learned_addr learned my addr 172.21.15.77:0/4175712024 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.822+0000 7fc5f1b9f640 1 --2- 172.21.15.77:0/4175712024 >> v2:172.21.15.158:3300/0 conn(0x7fc5ec0d2660 0x7fc5ec0ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.823+0000 7fc5f0b9d640 1 -- 172.21.15.77:0/4175712024 >> v2:172.21.15.158:3300/0 conn(0x7fc5ec0d2660 msgr2=0x7fc5ec0ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.823+0000 7fc5f0b9d640 1 --2- 172.21.15.77:0/4175712024 >> v2:172.21.15.158:3300/0 conn(0x7fc5ec0d2660 0x7fc5ec0ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.823+0000 7fc5f0b9d640 1 -- 172.21.15.77:0/4175712024 >> v2:172.21.15.77:3300/0 conn(0x7fc5ec15e010 msgr2=0x7fc5ec0d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.823+0000 7fc5f0b9d640 1 --2- 172.21.15.77:0/4175712024 >> v2:172.21.15.77:3300/0 conn(0x7fc5ec15e010 0x7fc5ec0d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.823+0000 7fc5f0b9d640 1 -- 172.21.15.77:0/4175712024 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec0b2f20 con 0x7fc5ec0d19d0 2024-03-06T07:22:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.823+0000 7fc5f139e640 1 --2- 172.21.15.77:0/4175712024 >> v2:172.21.15.77:3300/0 conn(0x7fc5ec15e010 0x7fc5ec0d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:32.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.823+0000 7fc5f0b9d640 1 --2- 172.21.15.77:0/4175712024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d19d0 0x7fc5ec0d2120 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0097a0 tx=0x7fc5dc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=742b3721b1b6153c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5e3fff640 1 -- 172.21.15.77:0/4175712024 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc035020 con 0x7fc5ec0d19d0 2024-03-06T07:22:32.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5e3fff640 1 -- 172.21.15.77:0/4175712024 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5dc031830 con 0x7fc5ec0d19d0 2024-03-06T07:22:32.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5e3fff640 1 -- 172.21.15.77:0/4175712024 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc031c00 con 0x7fc5ec0d19d0 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 -- 172.21.15.77:0/4175712024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d19d0 msgr2=0x7fc5ec0d2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/4175712024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d19d0 0x7fc5ec0d2120 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0097a0 tx=0x7fc5dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 -- 172.21.15.77:0/4175712024 shutdown_connections 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/4175712024 >> v2:172.21.15.158:3300/0 conn(0x7fc5ec0d2660 0x7fc5ec0ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/4175712024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d19d0 0x7fc5ec0d2120 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/4175712024 >> v2:172.21.15.77:3300/0 conn(0x7fc5ec15e010 0x7fc5ec0d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 -- 172.21.15.77:0/4175712024 >> 172.21.15.77:0/4175712024 conn(0x7fc5ec09ce10 msgr2=0x7fc5ec0d3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 -- 172.21.15.77:0/4175712024 shutdown_connections 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.824+0000 7fc5f23a0640 1 -- 172.21.15.77:0/4175712024 wait complete. 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.825+0000 7fc5f23a0640 1 Processor -- start 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.825+0000 7fc5f23a0640 1 -- start start 2024-03-06T07:22:32.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f23a0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec0d19d0 0x7fc5ec1f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f23a0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d2660 0x7fc5ec1f7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f23a0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5ec15e010 0x7fc5ec1f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f23a0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0d0830 con 0x7fc5ec15e010 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f23a0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0d06b0 con 0x7fc5ec0d19d0 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f23a0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0ceb90 con 0x7fc5ec0d2660 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f1b9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5ec15e010 0x7fc5ec1f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f1b9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5ec15e010 0x7fc5ec1f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42992/0 (socket says 172.21.15.77:42992) 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f1b9f640 1 -- 172.21.15.77:0/45909670 learned_addr learned my addr 172.21.15.77:0/45909670 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f139e640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec0d19d0 0x7fc5ec1f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f0b9d640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d2660 0x7fc5ec1f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f1b9f640 1 -- 172.21.15.77:0/45909670 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d2660 msgr2=0x7fc5ec1f7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f1b9f640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d2660 0x7fc5ec1f7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f1b9f640 1 -- 172.21.15.77:0/45909670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec0d19d0 msgr2=0x7fc5ec1f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f1b9f640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec0d19d0 0x7fc5ec1f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.826+0000 7fc5f1b9f640 1 -- 172.21.15.77:0/45909670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec1f7b70 con 0x7fc5ec15e010 2024-03-06T07:22:32.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.827+0000 7fc5f1b9f640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5ec15e010 0x7fc5ec1f09b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400e830 tx=0x7fc5e400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.827+0000 7fc5e1ffb640 1 -- 172.21.15.77:0/45909670 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4017070 con 0x7fc5ec15e010 2024-03-06T07:22:32.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.827+0000 7fc5e1ffb640 1 -- 172.21.15.77:0/45909670 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e400c940 con 0x7fc5ec15e010 2024-03-06T07:22:32.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.827+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec1f3650 con 0x7fc5ec15e010 2024-03-06T07:22:32.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.827+0000 7fc5e1ffb640 1 -- 172.21.15.77:0/45909670 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e400cd30 con 0x7fc5ec15e010 2024-03-06T07:22:32.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.827+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec15e8e0 con 0x7fc5ec15e010 2024-03-06T07:22:32.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.828+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec068f10 con 0x7fc5ec15e010 2024-03-06T07:22:32.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.828+0000 7fc5e1ffb640 1 -- 172.21.15.77:0/45909670 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5e401a070 con 0x7fc5ec15e010 2024-03-06T07:22:32.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.829+0000 7fc5e1ffb640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5d006b4d0 0x7fc5d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.829+0000 7fc5e1ffb640 1 -- 172.21.15.77:0/45909670 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5e408a100 con 0x7fc5ec15e010 2024-03-06T07:22:32.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.829+0000 7fc5f139e640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5d006b4d0 0x7fc5d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.829+0000 7fc5f139e640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5d006b4d0 0x7fc5d006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0b4db0 tx=0x7fc5e800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:32.833+0000 7fc5e1ffb640 1 -- 172.21.15.77:0/45909670 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5ec1f7b70 con 0x7fc5ec15e010 2024-03-06T07:22:33.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.004+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc5ec0d35b0 con 0x7fc5ec15e010 2024-03-06T07:22:33.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.004+0000 7fc5e1ffb640 1 -- 172.21.15.77:0/45909670 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc5e4058930 con 0x7fc5ec15e010 2024-03-06T07:22:33.006 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:33.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5d006b4d0 msgr2=0x7fc5d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5d006b4d0 0x7fc5d006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0b4db0 tx=0x7fc5e800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5ec15e010 msgr2=0x7fc5ec1f09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5ec15e010 0x7fc5ec1f09b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400e830 tx=0x7fc5e400ecf0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 shutdown_connections 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5d006b4d0 0x7fc5d006d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5ec15e010 0x7fc5ec1f09b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc5ec0d2660 0x7fc5ec1f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 --2- 172.21.15.77:0/45909670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec0d19d0 0x7fc5ec1f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 >> 172.21.15.77:0/45909670 conn(0x7fc5ec09ce10 msgr2=0x7fc5ec0cee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 shutdown_connections 2024-03-06T07:22:33.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.006+0000 7fc5f23a0640 1 -- 172.21.15.77:0/45909670 wait complete. 2024-03-06T07:22:33.019 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:33.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.120+0000 7fe8625bc640 1 Processor -- start 2024-03-06T07:22:33.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.120+0000 7fe8625bc640 1 -- start start 2024-03-06T07:22:33.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.121+0000 7fe8625bc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe85c1586e0 0x7fe85c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.121+0000 7fe8625bc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c158ac0 con 0x7fe85c0aef80 2024-03-06T07:22:33.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.121+0000 7fe8625bc640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c15b2a0 con 0x7fe85c0d01a0 2024-03-06T07:22:33.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.121+0000 7fe8625bc640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85c15b540 con 0x7fe85c1586e0 2024-03-06T07:22:33.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.121+0000 7fe85b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe85c0aef80 0x7fe85c0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52492/0 (socket says 172.21.15.77:52492) 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.121+0000 7fe85b7fe640 1 -- 172.21.15.77:0/3958083664 learned_addr learned my addr 172.21.15.77:0/3958083664 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.122+0000 7fe85bfff640 1 --2- 172.21.15.77:0/3958083664 >> v2:172.21.15.158:3300/0 conn(0x7fe85c1586e0 0x7fe85c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.122+0000 7fe85affd640 1 -- 172.21.15.77:0/3958083664 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613321307 0 0) 0x7fe85c158ac0 con 0x7fe85c0aef80 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.122+0000 7fe85affd640 1 -- 172.21.15.77:0/3958083664 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe844003520 con 0x7fe85c0aef80 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.122+0000 7fe85bfff640 1 -- 172.21.15.77:0/3958083664 >> v1:172.21.15.158:6789/0 conn(0x7fe85c0d01a0 legacy=0x7fe85c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.122+0000 7fe85bfff640 1 -- 172.21.15.77:0/3958083664 >> v1:172.21.15.77:6789/0 conn(0x7fe85c0aef80 legacy=0x7fe85c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.122+0000 7fe85bfff640 1 -- 172.21.15.77:0/3958083664 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c15b820 con 0x7fe85c1586e0 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe85bfff640 1 --2- 172.21.15.77:0/3958083664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 0x7fe85c0aea40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0097a0 tx=0x7fe84c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88d5c635723b03de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe85affd640 1 -- 172.21.15.77:0/3958083664 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c035020 con 0x7fe85c1586e0 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe85affd640 1 -- 172.21.15.77:0/3958083664 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84c031830 con 0x7fe85c1586e0 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe85affd640 1 -- 172.21.15.77:0/3958083664 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c031c00 con 0x7fe85c1586e0 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe8625bc640 1 -- 172.21.15.77:0/3958083664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 msgr2=0x7fe85c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe8625bc640 1 --2- 172.21.15.77:0/3958083664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 0x7fe85c0aea40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0097a0 tx=0x7fe84c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe8625bc640 1 -- 172.21.15.77:0/3958083664 shutdown_connections 2024-03-06T07:22:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe8625bc640 1 --2- 172.21.15.77:0/3958083664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 0x7fe85c0aea40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe8625bc640 1 -- 172.21.15.77:0/3958083664 >> 172.21.15.77:0/3958083664 conn(0x7fe85c06b190 msgr2=0x7fe85c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe8625bc640 1 -- 172.21.15.77:0/3958083664 shutdown_connections 2024-03-06T07:22:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.123+0000 7fe8625bc640 1 -- 172.21.15.77:0/3958083664 wait complete. 2024-03-06T07:22:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.124+0000 7fe8625bc640 1 Processor -- start 2024-03-06T07:22:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe8625bc640 1 -- start start 2024-03-06T07:22:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe8625bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe85c0aef80 0x7fe85c153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe8625bc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe85c0d01a0 0x7fe85c0ad680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe8625bc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 0x7fe85c1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe8625bc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c071350 con 0x7fe85c0aef80 2024-03-06T07:22:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe8625bc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c15bde0 con 0x7fe85c1586e0 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe8625bc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c0714d0 con 0x7fe85c0d01a0 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe85c0aef80 0x7fe85c153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe85c0aef80 0x7fe85c153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43006/0 (socket says 172.21.15.77:43006) 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.125+0000 7fe85bfff640 1 -- 172.21.15.77:0/261848019 learned_addr learned my addr 172.21.15.77:0/261848019 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe85b7fe640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe85c0d01a0 0x7fe85c0ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe860b59640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 0x7fe85c1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe85bfff640 1 -- 172.21.15.77:0/261848019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe85c0d01a0 msgr2=0x7fe85c0ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe85bfff640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe85c0d01a0 0x7fe85c0ad680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe85bfff640 1 -- 172.21.15.77:0/261848019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 msgr2=0x7fe85c1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe85bfff640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 0x7fe85c1597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe85bfff640 1 -- 172.21.15.77:0/261848019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c0adbc0 con 0x7fe85c0aef80 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe85bfff640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe85c0aef80 0x7fe85c153c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe84c034040 tx=0x7fe84c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe858ff9640 1 -- 172.21.15.77:0/261848019 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c040070 con 0x7fe85c0aef80 2024-03-06T07:22:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe858ff9640 1 -- 172.21.15.77:0/261848019 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84c03a910 con 0x7fe85c0aef80 2024-03-06T07:22:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe858ff9640 1 -- 172.21.15.77:0/261848019 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c035040 con 0x7fe85c0aef80 2024-03-06T07:22:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c0ae400 con 0x7fe85c0aef80 2024-03-06T07:22:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.126+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c0ae120 con 0x7fe85c0aef80 2024-03-06T07:22:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.127+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c0732b0 con 0x7fe85c0aef80 2024-03-06T07:22:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.128+0000 7fe858ff9640 1 -- 172.21.15.77:0/261848019 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe84c038020 con 0x7fe85c0aef80 2024-03-06T07:22:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.128+0000 7fe858ff9640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe84806b4d0 0x7fe84806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.128+0000 7fe85b7fe640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe84806b4d0 0x7fe84806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.128+0000 7fe858ff9640 1 -- 172.21.15.77:0/261848019 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe84c0abb10 con 0x7fe85c0aef80 2024-03-06T07:22:33.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.129+0000 7fe85b7fe640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe84806b4d0 0x7fe84806d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe85c157be0 tx=0x7fe850007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.133+0000 7fe858ff9640 1 -- 172.21.15.77:0/261848019 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe85c0adbc0 con 0x7fe85c0aef80 2024-03-06T07:22:33.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.302+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe85c068ff0 con 0x7fe85c0aef80 2024-03-06T07:22:33.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.302+0000 7fe858ff9640 1 -- 172.21.15.77:0/261848019 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe84c031800 con 0x7fe85c0aef80 2024-03-06T07:22:33.305 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:33.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe84806b4d0 msgr2=0x7fe84806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe84806b4d0 0x7fe84806d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe85c157be0 tx=0x7fe850007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe85c0aef80 msgr2=0x7fe85c153c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe85c0aef80 0x7fe85c153c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe84c034040 tx=0x7fe84c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 shutdown_connections 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe85c1586e0 0x7fe85c1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe85c0d01a0 0x7fe85c0ad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe84806b4d0 0x7fe84806d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 --2- 172.21.15.77:0/261848019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe85c0aef80 0x7fe85c153c90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 >> 172.21.15.77:0/261848019 conn(0x7fe85c06b190 msgr2=0x7fe85c0cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 shutdown_connections 2024-03-06T07:22:33.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.304+0000 7fe8625bc640 1 -- 172.21.15.77:0/261848019 wait complete. 2024-03-06T07:22:33.317 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:33.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.416+0000 7ff3920c9640 1 Processor -- start 2024-03-06T07:22:33.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.416+0000 7ff3920c9640 1 -- start start 2024-03-06T07:22:33.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff3920c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 0x7ff38c15efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff3920c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff38c0b0240 0x7ff38c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff3920c9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff38c06b4f0 0x7ff38c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff3920c9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c158aa0 con 0x7ff38c0b0240 2024-03-06T07:22:33.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff3920c9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c15f520 con 0x7ff38c06b4f0 2024-03-06T07:22:33.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff3920c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c06b8f0 con 0x7ff38c1586c0 2024-03-06T07:22:33.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38affd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff38c0b0240 0x7ff38c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38affd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff38c0b0240 0x7ff38c06afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43018/0 (socket says 172.21.15.77:43018) 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38affd640 1 -- 172.21.15.77:0/530387858 learned_addr learned my addr 172.21.15.77:0/530387858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38bfff640 1 --2- 172.21.15.77:0/530387858 >> v2:172.21.15.158:3300/0 conn(0x7ff38c06b4f0 0x7ff38c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38b7fe640 1 --2- 172.21.15.77:0/530387858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 0x7ff38c15efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38affd640 1 -- 172.21.15.77:0/530387858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 msgr2=0x7ff38c15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38affd640 1 --2- 172.21.15.77:0/530387858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 0x7ff38c15efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38affd640 1 -- 172.21.15.77:0/530387858 >> v2:172.21.15.158:3300/0 conn(0x7ff38c06b4f0 msgr2=0x7ff38c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38affd640 1 --2- 172.21.15.77:0/530387858 >> v2:172.21.15.158:3300/0 conn(0x7ff38c06b4f0 0x7ff38c0cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.417+0000 7ff38affd640 1 -- 172.21.15.77:0/530387858 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c161cf0 con 0x7ff38c0b0240 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.418+0000 7ff38affd640 1 --2- 172.21.15.77:0/530387858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c06afb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff38c067cb0 tx=0x7ff38002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14461248b79cfa8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.418+0000 7ff38a7fc640 1 -- 172.21.15.77:0/530387858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380035020 con 0x7ff38c0b0240 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.418+0000 7ff38a7fc640 1 -- 172.21.15.77:0/530387858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff380031870 con 0x7ff38c0b0240 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.418+0000 7ff38a7fc640 1 -- 172.21.15.77:0/530387858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380031c40 con 0x7ff38c0b0240 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 -- 172.21.15.77:0/530387858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 msgr2=0x7ff38c06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 --2- 172.21.15.77:0/530387858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c06afb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff38c067cb0 tx=0x7ff38002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 -- 172.21.15.77:0/530387858 shutdown_connections 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 --2- 172.21.15.77:0/530387858 >> v2:172.21.15.158:3300/0 conn(0x7ff38c06b4f0 0x7ff38c0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 --2- 172.21.15.77:0/530387858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c06afb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 --2- 172.21.15.77:0/530387858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 0x7ff38c15efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 -- 172.21.15.77:0/530387858 >> 172.21.15.77:0/530387858 conn(0x7ff38c0d09d0 msgr2=0x7ff38c0d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 -- 172.21.15.77:0/530387858 shutdown_connections 2024-03-06T07:22:33.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.419+0000 7ff3920c9640 1 -- 172.21.15.77:0/530387858 wait complete. 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.420+0000 7ff3920c9640 1 Processor -- start 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.420+0000 7ff3920c9640 1 -- start start 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff3920c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff38c06b4f0 0x7ff38c149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff3920c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff3920c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 0x7ff38c160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff3920c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c162590 con 0x7ff38c0b0240 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff3920c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c1622b0 con 0x7ff38c06b4f0 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff3920c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c162870 con 0x7ff38c1586c0 2024-03-06T07:22:33.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c160950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43020/0 (socket says 172.21.15.77:43020) 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 -- 172.21.15.77:0/3861354758 learned_addr learned my addr 172.21.15.77:0/3861354758 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38bfff640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 0x7ff38c160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38b7fe640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff38c06b4f0 0x7ff38c149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 -- 172.21.15.77:0/3861354758 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 msgr2=0x7ff38c160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 0x7ff38c160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 -- 172.21.15.77:0/3861354758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff38c06b4f0 msgr2=0x7ff38c149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff38c06b4f0 0x7ff38c149480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 -- 172.21.15.77:0/3861354758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c1573b0 con 0x7ff38c0b0240 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.421+0000 7ff38affd640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c160950 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff38002b950 tx=0x7ff3800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.422+0000 7ff373fff640 1 -- 172.21.15.77:0/3861354758 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380040070 con 0x7ff38c0b0240 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.422+0000 7ff373fff640 1 -- 172.21.15.77:0/3861354758 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff380031e10 con 0x7ff38c0b0240 2024-03-06T07:22:33.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.422+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c15b960 con 0x7ff38c0b0240 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.422+0000 7ff373fff640 1 -- 172.21.15.77:0/3861354758 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380035040 con 0x7ff38c0b0240 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.422+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c15c320 con 0x7ff38c0b0240 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.423+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38c04f330 con 0x7ff38c0b0240 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.423+0000 7ff373fff640 1 -- 172.21.15.77:0/3861354758 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff380038020 con 0x7ff38c0b0240 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.423+0000 7ff373fff640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.423+0000 7ff373fff640 1 -- 172.21.15.77:0/3861354758 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3800abed0 con 0x7ff38c0b0240 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.423+0000 7ff38b7fe640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.424+0000 7ff38b7fe640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff37806b4d0 0x7ff37806d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009aa0 tx=0x7ff37c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.428+0000 7ff373fff640 1 -- 172.21.15.77:0/3861354758 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff38007a670 con 0x7ff38c0b0240 2024-03-06T07:22:33.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.613+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff38c0d17a0 con 0x7ff38c0b0240 2024-03-06T07:22:33.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.614+0000 7ff373fff640 1 -- 172.21.15.77:0/3861354758 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff38007e710 con 0x7ff38c0b0240 2024-03-06T07:22:33.616 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff37806b4d0 msgr2=0x7ff37806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff37806b4d0 0x7ff37806d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009aa0 tx=0x7ff37c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 msgr2=0x7ff38c160950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c160950 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff38002b950 tx=0x7ff3800046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 shutdown_connections 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0b0240 0x7ff38c160950 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff38c1586c0 0x7ff38c160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 --2- 172.21.15.77:0/3861354758 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff38c06b4f0 0x7ff38c149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 >> 172.21.15.77:0/3861354758 conn(0x7ff38c0d09d0 msgr2=0x7ff38c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 shutdown_connections 2024-03-06T07:22:33.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.616+0000 7ff3920c9640 1 -- 172.21.15.77:0/3861354758 wait complete. 2024-03-06T07:22:33.628 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:22:33.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:33.629 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:33.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:33.656 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:22:33.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:33.656 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:22:33.719 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:22:33.719 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:33.736 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:22:33.736 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:33.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:33.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.888+0000 7f9bffa2a640 1 Processor -- start 2024-03-06T07:22:33.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.889+0000 7f9bffa2a640 1 -- start start 2024-03-06T07:22:33.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.889+0000 7f9bffa2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf80b3060 0x7f9bf815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.889+0000 7f9bffa2a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9bf815bf30 0x7f9bf815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.889+0000 7f9bffa2a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf815f2d0 con 0x7f9bf813dc10 2024-03-06T07:22:33.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.889+0000 7f9bffa2a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf80cf3f0 con 0x7f9bf815bf30 2024-03-06T07:22:33.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.889+0000 7f9bffa2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf815b210 con 0x7f9bf80b3060 2024-03-06T07:22:33.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.890+0000 7f9bfdfc7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9bf815bf30 0x7f9bf815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.890+0000 7f9bfdfc7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9bf815bf30 0x7f9bf815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43036/0 (socket says 172.21.15.77:43036) 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.890+0000 7f9bfdfc7640 1 -- 172.21.15.77:0/4174287881 learned_addr learned my addr 172.21.15.77:0/4174287881 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.890+0000 7f9bfcfc5640 1 --2- 172.21.15.77:0/4174287881 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf80b3060 0x7f9bf815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.890+0000 7f9bfdfc7640 1 -- 172.21.15.77:0/4174287881 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf80b3060 msgr2=0x7f9bf815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.890+0000 7f9bfdfc7640 1 --2- 172.21.15.77:0/4174287881 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf80b3060 0x7f9bf815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.890+0000 7f9bfdfc7640 1 -- 172.21.15.77:0/4174287881 >> v1:172.21.15.158:6789/0 conn(0x7f9bf813dc10 legacy=0x7f9bf80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.890+0000 7f9bfdfc7640 1 -- 172.21.15.77:0/4174287881 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf815f580 con 0x7f9bf815bf30 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bfdfc7640 1 --2- 172.21.15.77:0/4174287881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf815bf30 0x7f9bf815ed90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009e50 tx=0x7f9bf002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3deb360ec2a81026 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bebfff640 1 -- 172.21.15.77:0/4174287881 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf002ca70 con 0x7f9bf815bf30 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bebfff640 1 -- 172.21.15.77:0/4174287881 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf002cc10 con 0x7f9bf815bf30 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bebfff640 1 -- 172.21.15.77:0/4174287881 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf00343f0 con 0x7f9bf815bf30 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bffa2a640 1 -- 172.21.15.77:0/4174287881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf815bf30 msgr2=0x7f9bf815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/4174287881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf815bf30 0x7f9bf815ed90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009e50 tx=0x7f9bf002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bffa2a640 1 -- 172.21.15.77:0/4174287881 shutdown_connections 2024-03-06T07:22:33.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/4174287881 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf80b3060 0x7f9bf815e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/4174287881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf815bf30 0x7f9bf815ed90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bffa2a640 1 -- 172.21.15.77:0/4174287881 >> 172.21.15.77:0/4174287881 conn(0x7f9bf80cf8a0 msgr2=0x7f9bf80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bffa2a640 1 -- 172.21.15.77:0/4174287881 shutdown_connections 2024-03-06T07:22:33.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.891+0000 7f9bffa2a640 1 -- 172.21.15.77:0/4174287881 wait complete. 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.892+0000 7f9bffa2a640 1 Processor -- start 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bffa2a640 1 -- start start 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bffa2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf80b3060 0x7f9bf8141f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bffa2a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bf813dc10 0x7f9bf8158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bffa2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf815bf30 0x7f9bf8071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bffa2a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf813dff0 con 0x7f9bf80b3060 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bffa2a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf815b730 con 0x7f9bf813dc10 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bffa2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80708d0 con 0x7f9bf815bf30 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bfd7c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf80b3060 0x7f9bf8141f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bfd7c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf80b3060 0x7f9bf8141f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43048/0 (socket says 172.21.15.77:43048) 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bfd7c6640 1 -- 172.21.15.77:0/2508566625 learned_addr learned my addr 172.21.15.77:0/2508566625 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bfdfc7640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf815bf30 0x7f9bf8071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.893+0000 7f9bfcfc5640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bf813dc10 0x7f9bf8158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bfd7c6640 1 -- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf815bf30 msgr2=0x7f9bf8071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bfd7c6640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf815bf30 0x7f9bf8071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bfd7c6640 1 -- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bf813dc10 msgr2=0x7f9bf8158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bfd7c6640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bf813dc10 0x7f9bf8158700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bfd7c6640 1 -- 172.21.15.77:0/2508566625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf8158c40 con 0x7f9bf80b3060 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bfcfc5640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bf813dc10 0x7f9bf8158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bfd7c6640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf80b3060 0x7f9bf8141f00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00e710 tx=0x7f9bec00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9be9ffb640 1 -- 172.21.15.77:0/2508566625 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0052d0 con 0x7f9bf80b3060 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9be9ffb640 1 -- 172.21.15.77:0/2508566625 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec005470 con 0x7f9bf80b3060 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf81424e0 con 0x7f9bf80b3060 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9be9ffb640 1 -- 172.21.15.77:0/2508566625 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec001b50 con 0x7f9bf80b3060 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.894+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf813f4e0 con 0x7f9bf80b3060 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.895+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf804f330 con 0x7f9bf80b3060 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.895+0000 7f9be9ffb640 1 -- 172.21.15.77:0/2508566625 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bec0035f0 con 0x7f9bf80b3060 2024-03-06T07:22:33.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.896+0000 7f9be9ffb640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.896+0000 7f9be9ffb640 1 -- 172.21.15.77:0/2508566625 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bec08c3c0 con 0x7f9bf80b3060 2024-03-06T07:22:33.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.896+0000 7f9bfcfc5640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.896+0000 7f9bfcfc5640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9bf813acc0 tx=0x7f9bf4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:33.900+0000 7f9be9ffb640 1 -- 172.21.15.77:0/2508566625 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bec016020 con 0x7f9bf80b3060 2024-03-06T07:22:34.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.072+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bf80ce9f0 con 0x7f9bf80b3060 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.073+0000 7f9be9ffb640 1 -- 172.21.15.77:0/2508566625 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f9bec00ee20 con 0x7f9bf80b3060 2024-03-06T07:22:34.075 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:34.076 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bdc06b4d0 msgr2=0x7f9bdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9bf813acc0 tx=0x7f9bf4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf80b3060 msgr2=0x7f9bf8141f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf80b3060 0x7f9bf8141f00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00e710 tx=0x7f9bec00ebd0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 shutdown_connections 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bf815bf30 0x7f9bf8071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bf813dc10 0x7f9bf8158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bdc06b4d0 0x7f9bdc06d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 --2- 172.21.15.77:0/2508566625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf80b3060 0x7f9bf8141f00 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 >> 172.21.15.77:0/2508566625 conn(0x7f9bf80cf8a0 msgr2=0x7f9bf80b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 shutdown_connections 2024-03-06T07:22:34.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.075+0000 7f9bffa2a640 1 -- 172.21.15.77:0/2508566625 wait complete. 2024-03-06T07:22:34.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:34.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:34.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.189+0000 7f999944e640 1 Processor -- start 2024-03-06T07:22:34.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.189+0000 7f999944e640 1 -- start start 2024-03-06T07:22:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f999944e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f99940aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f999944e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f99940aef80 0x7f99940cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f999944e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99940cffd0 0x7f999406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f999944e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99940d03f0 con 0x7f99940cffd0 2024-03-06T07:22:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f999944e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9994158ac0 con 0x7f99940aef80 2024-03-06T07:22:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f999944e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999415b2a0 con 0x7f99941586e0 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99937fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99940cffd0 0x7f999406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99937fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99940cffd0 0x7f999406da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43062/0 (socket says 172.21.15.77:43062) 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99937fe640 1 -- 172.21.15.77:0/2793661221 learned_addr learned my addr 172.21.15.77:0/2793661221 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99927fc640 1 --2- 172.21.15.77:0/2793661221 >> v2:172.21.15.158:3300/0 conn(0x7f99940aef80 0x7f99940cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f9992ffd640 1 --2- 172.21.15.77:0/2793661221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f99940aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99937fe640 1 -- 172.21.15.77:0/2793661221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 msgr2=0x7f99940aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99937fe640 1 --2- 172.21.15.77:0/2793661221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f99940aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99937fe640 1 -- 172.21.15.77:0/2793661221 >> v2:172.21.15.158:3300/0 conn(0x7f99940aef80 msgr2=0x7f99940cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99937fe640 1 --2- 172.21.15.77:0/2793661221 >> v2:172.21.15.158:3300/0 conn(0x7f99940aef80 0x7f99940cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.190+0000 7f99937fe640 1 -- 172.21.15.77:0/2793661221 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999415b540 con 0x7f99940cffd0 2024-03-06T07:22:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.191+0000 7f99937fe640 1 --2- 172.21.15.77:0/2793661221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940cffd0 0x7f999406da20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f99940d0610 tx=0x7f998c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4eaba1e695987 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.191+0000 7f9991ffb640 1 -- 172.21.15.77:0/2793661221 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c035020 con 0x7f99940cffd0 2024-03-06T07:22:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.191+0000 7f9991ffb640 1 -- 172.21.15.77:0/2793661221 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c031870 con 0x7f99940cffd0 2024-03-06T07:22:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.191+0000 7f9991ffb640 1 -- 172.21.15.77:0/2793661221 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c031c40 con 0x7f99940cffd0 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 -- 172.21.15.77:0/2793661221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940cffd0 msgr2=0x7f999406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 --2- 172.21.15.77:0/2793661221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940cffd0 0x7f999406da20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f99940d0610 tx=0x7f998c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 -- 172.21.15.77:0/2793661221 shutdown_connections 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 --2- 172.21.15.77:0/2793661221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940cffd0 0x7f999406da20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 --2- 172.21.15.77:0/2793661221 >> v2:172.21.15.158:3300/0 conn(0x7f99940aef80 0x7f99940cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 --2- 172.21.15.77:0/2793661221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f99940aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 -- 172.21.15.77:0/2793661221 >> 172.21.15.77:0/2793661221 conn(0x7f999406b190 msgr2=0x7f999406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 -- 172.21.15.77:0/2793661221 shutdown_connections 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.192+0000 7f999944e640 1 -- 172.21.15.77:0/2793661221 wait complete. 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.193+0000 7f999944e640 1 Processor -- start 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.193+0000 7f999944e640 1 -- start start 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f999944e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940aef80 0x7f999406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f999944e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99940cffd0 0x7f99940adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f999944e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f9994156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f999944e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999415bde0 con 0x7f99940aef80 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f999944e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999415bb00 con 0x7f99940cffd0 2024-03-06T07:22:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f999944e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9994071350 con 0x7f99941586e0 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940aef80 0x7f999406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940aef80 0x7f999406ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43072/0 (socket says 172.21.15.77:43072) 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 -- 172.21.15.77:0/4096947341 learned_addr learned my addr 172.21.15.77:0/4096947341 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f99927fc640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99940cffd0 0x7f99940adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f99937fe640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f9994156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 -- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 msgr2=0x7f9994156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f9994156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 -- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99940cffd0 msgr2=0x7f99940adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99940cffd0 0x7f99940adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 -- 172.21.15.77:0/4096947341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9994157790 con 0x7f99940aef80 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f99937fe640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f9994156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.194+0000 7f9992ffd640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940aef80 0x7f999406ca80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9984009d10 tx=0x7f998400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.195+0000 7f997b7fe640 1 -- 172.21.15.77:0/4096947341 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984018020 con 0x7f99940aef80 2024-03-06T07:22:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.195+0000 7f997b7fe640 1 -- 172.21.15.77:0/4096947341 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9984014480 con 0x7f99940aef80 2024-03-06T07:22:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.195+0000 7f997b7fe640 1 -- 172.21.15.77:0/4096947341 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984014830 con 0x7f99940aef80 2024-03-06T07:22:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.195+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99940ad140 con 0x7f99940aef80 2024-03-06T07:22:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.195+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99940ad700 con 0x7f99940aef80 2024-03-06T07:22:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.196+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99940732b0 con 0x7f99940aef80 2024-03-06T07:22:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.196+0000 7f997b7fe640 1 -- 172.21.15.77:0/4096947341 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9984005d10 con 0x7f99940aef80 2024-03-06T07:22:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.196+0000 7f997b7fe640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f998006b4d0 0x7f998006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.196+0000 7f997b7fe640 1 -- 172.21.15.77:0/4096947341 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f998408aea0 con 0x7f99940aef80 2024-03-06T07:22:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.196+0000 7f99927fc640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f998006b4d0 0x7f998006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.197+0000 7f99927fc640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f998006b4d0 0x7f998006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f999414c0a0 tx=0x7f9988009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.201+0000 7f997b7fe640 1 -- 172.21.15.77:0/4096947341 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9984022030 con 0x7f99940aef80 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.370+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9994068ff0 con 0x7f99940aef80 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.370+0000 7f997b7fe640 1 -- 172.21.15.77:0/4096947341 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9984010040 con 0x7f99940aef80 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:34.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f998006b4d0 msgr2=0x7f998006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f998006b4d0 0x7f998006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f999414c0a0 tx=0x7f9988009190 comp rx=0 tx=0).stop 2024-03-06T07:22:34.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940aef80 msgr2=0x7f999406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940aef80 0x7f999406ca80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9984009d10 tx=0x7f998400c490 comp rx=0 tx=0).stop 2024-03-06T07:22:34.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 shutdown_connections 2024-03-06T07:22:34.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99941586e0 0x7f9994156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99940cffd0 0x7f99940adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f998006b4d0 0x7f998006d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 --2- 172.21.15.77:0/4096947341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940aef80 0x7f999406ca80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 >> 172.21.15.77:0/4096947341 conn(0x7f999406b190 msgr2=0x7f99940cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 shutdown_connections 2024-03-06T07:22:34.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.372+0000 7f999944e640 1 -- 172.21.15.77:0/4096947341 wait complete. 2024-03-06T07:22:34.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.486+0000 7fca8a59a640 1 Processor -- start 2024-03-06T07:22:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.486+0000 7fca8a59a640 1 -- start start 2024-03-06T07:22:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.486+0000 7fca8a59a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca841586e0 0x7fca840aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.486+0000 7fca8a59a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fca840aef80 0x7fca840cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca8a59a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca840d03f0 con 0x7fca840cffd0 2024-03-06T07:22:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca8a59a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca84158ac0 con 0x7fca840aef80 2024-03-06T07:22:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca8a59a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8415b2a0 con 0x7fca841586e0 2024-03-06T07:22:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca837fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fca840aef80 0x7fca840cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca837fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fca840aef80 0x7fca840cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43074/0 (socket says 172.21.15.77:43074) 2024-03-06T07:22:34.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca837fe640 1 -- 172.21.15.77:0/903225511 learned_addr learned my addr 172.21.15.77:0/903225511 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:34.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca88b37640 1 --1- 172.21.15.77:0/903225511 >> v1:172.21.15.77:6789/0 conn(0x7fca840cffd0 0x7fca8406da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52504/0 (socket says 172.21.15.77:52504) 2024-03-06T07:22:34.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca83fff640 1 --2- 172.21.15.77:0/903225511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca841586e0 0x7fca840aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca837fe640 1 -- 172.21.15.77:0/903225511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca841586e0 msgr2=0x7fca840aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca837fe640 1 --2- 172.21.15.77:0/903225511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca841586e0 0x7fca840aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca837fe640 1 -- 172.21.15.77:0/903225511 >> v1:172.21.15.77:6789/0 conn(0x7fca840cffd0 legacy=0x7fca8406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.487+0000 7fca837fe640 1 -- 172.21.15.77:0/903225511 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8415b540 con 0x7fca840aef80 2024-03-06T07:22:34.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca837fe640 1 --2- 172.21.15.77:0/903225511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca840aef80 0x7fca840cfa90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fca78009870 tx=0x7fca7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5b566479a03c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca82ffd640 1 -- 172.21.15.77:0/903225511 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78034070 con 0x7fca840aef80 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca82ffd640 1 -- 172.21.15.77:0/903225511 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca78030840 con 0x7fca840aef80 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca82ffd640 1 -- 172.21.15.77:0/903225511 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78030c10 con 0x7fca840aef80 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca8a59a640 1 -- 172.21.15.77:0/903225511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca840aef80 msgr2=0x7fca840cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca8a59a640 1 --2- 172.21.15.77:0/903225511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca840aef80 0x7fca840cfa90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fca78009870 tx=0x7fca7802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca8a59a640 1 -- 172.21.15.77:0/903225511 shutdown_connections 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca8a59a640 1 --2- 172.21.15.77:0/903225511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca840aef80 0x7fca840cfa90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca8a59a640 1 --2- 172.21.15.77:0/903225511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca841586e0 0x7fca840aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca8a59a640 1 -- 172.21.15.77:0/903225511 >> 172.21.15.77:0/903225511 conn(0x7fca8406b190 msgr2=0x7fca8406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca8a59a640 1 -- 172.21.15.77:0/903225511 shutdown_connections 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.488+0000 7fca8a59a640 1 -- 172.21.15.77:0/903225511 wait complete. 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.490+0000 7fca8a59a640 1 Processor -- start 2024-03-06T07:22:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.490+0000 7fca8a59a640 1 -- start start 2024-03-06T07:22:34.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.490+0000 7fca8a59a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca840aef80 0x7fca8414d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.490+0000 7fca8a59a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca840cffd0 0x7fca840cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.490+0000 7fca8a59a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca841586e0 0x7fca841495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.490+0000 7fca8a59a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8415bde0 con 0x7fca841586e0 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.490+0000 7fca8a59a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8415bb00 con 0x7fca840cffd0 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.490+0000 7fca8a59a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca84071350 con 0x7fca840aef80 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca841586e0 0x7fca841495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca841586e0 0x7fca841495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43086/0 (socket says 172.21.15.77:43086) 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 -- 172.21.15.77:0/3073176541 learned_addr learned my addr 172.21.15.77:0/3073176541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca837fe640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca840cffd0 0x7fca840cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca83fff640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca840aef80 0x7fca8414d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 -- 172.21.15.77:0/3073176541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca840aef80 msgr2=0x7fca8414d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca840aef80 0x7fca8414d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 -- 172.21.15.77:0/3073176541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca840cffd0 msgr2=0x7fca840cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca840cffd0 0x7fca840cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 -- 172.21.15.77:0/3073176541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8406d7c0 con 0x7fca841586e0 2024-03-06T07:22:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca88b37640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca841586e0 0x7fca841495d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c780 tx=0x7fca7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca80ff9640 1 -- 172.21.15.77:0/3073176541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c017070 con 0x7fca841586e0 2024-03-06T07:22:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca80ff9640 1 -- 172.21.15.77:0/3073176541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c00a640 con 0x7fca841586e0 2024-03-06T07:22:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8406daa0 con 0x7fca841586e0 2024-03-06T07:22:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca80ff9640 1 -- 172.21.15.77:0/3073176541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c00a9f0 con 0x7fca841586e0 2024-03-06T07:22:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.491+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca84148af0 con 0x7fca841586e0 2024-03-06T07:22:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.492+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8415af90 con 0x7fca841586e0 2024-03-06T07:22:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.493+0000 7fca80ff9640 1 -- 172.21.15.77:0/3073176541 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca7c01a070 con 0x7fca841586e0 2024-03-06T07:22:34.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.493+0000 7fca80ff9640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7006b4d0 0x7fca7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.493+0000 7fca83fff640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7006b4d0 0x7fca7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.493+0000 7fca80ff9640 1 -- 172.21.15.77:0/3073176541 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca7c089730 con 0x7fca841586e0 2024-03-06T07:22:34.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.494+0000 7fca83fff640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7006b4d0 0x7fca7006d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fca74009aa0 tx=0x7fca74009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.498+0000 7fca80ff9640 1 -- 172.21.15.77:0/3073176541 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca8415af90 con 0x7fca841586e0 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.667+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca84068ff0 con 0x7fca841586e0 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.667+0000 7fca80ff9640 1 -- 172.21.15.77:0/3073176541 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fca7c057f60 con 0x7fca841586e0 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:34.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7006b4d0 msgr2=0x7fca7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7006b4d0 0x7fca7006d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fca74009aa0 tx=0x7fca74009190 comp rx=0 tx=0).stop 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca841586e0 msgr2=0x7fca841495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca841586e0 0x7fca841495d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c780 tx=0x7fca7c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 shutdown_connections 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7006b4d0 0x7fca7006d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca841586e0 0x7fca841495d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca840cffd0 0x7fca840cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 --2- 172.21.15.77:0/3073176541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca840aef80 0x7fca8414d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 >> 172.21.15.77:0/3073176541 conn(0x7fca8406b190 msgr2=0x7fca841572a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 shutdown_connections 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.669+0000 7fca8a59a640 1 -- 172.21.15.77:0/3073176541 wait complete. 2024-03-06T07:22:34.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.783+0000 7fceaad5c640 1 Processor -- start 2024-03-06T07:22:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.783+0000 7fceaad5c640 1 -- start start 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.783+0000 7fceaad5c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcea40ce3b0 0x7fcea4071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.783+0000 7fceaad5c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40ce890 con 0x7fcea415e2f0 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.783+0000 7fceaad5c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40ceb70 con 0x7fcea415df10 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.783+0000 7fceaad5c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea40cee50 con 0x7fcea40ce3b0 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.784+0000 7fcea3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcea40ce3b0 0x7fcea4071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.784+0000 7fcea3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcea40ce3b0 0x7fcea4071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43094/0 (socket says 172.21.15.77:43094) 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.784+0000 7fcea3fff640 1 -- 172.21.15.77:0/2467603629 learned_addr learned my addr 172.21.15.77:0/2467603629 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.784+0000 7fcea3fff640 1 -- 172.21.15.77:0/2467603629 >> v1:172.21.15.158:6789/0 conn(0x7fcea415df10 legacy=0x7fcea415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.784+0000 7fcea3fff640 1 -- 172.21.15.77:0/2467603629 >> v1:172.21.15.77:6789/0 conn(0x7fcea415e2f0 legacy=0x7fcea40b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.784+0000 7fcea37fe640 1 -- 172.21.15.77:0/2467603629 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719667058 0 0) 0x7fcea40ceb70 con 0x7fcea415df10 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.784+0000 7fcea3fff640 1 -- 172.21.15.77:0/2467603629 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40cf130 con 0x7fcea40ce3b0 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fcea3fff640 1 --2- 172.21.15.77:0/2467603629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea40ce3b0 0x7fcea4071f30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fce8c009870 tx=0x7fce8c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319810e2a6997f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fcea37fe640 1 -- 172.21.15.77:0/2467603629 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c034070 con 0x7fcea40ce3b0 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fcea37fe640 1 -- 172.21.15.77:0/2467603629 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c030860 con 0x7fcea40ce3b0 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fcea37fe640 1 -- 172.21.15.77:0/2467603629 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c030c30 con 0x7fcea40ce3b0 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fceaad5c640 1 -- 172.21.15.77:0/2467603629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea40ce3b0 msgr2=0x7fcea4071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fceaad5c640 1 --2- 172.21.15.77:0/2467603629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea40ce3b0 0x7fcea4071f30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fce8c009870 tx=0x7fce8c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fceaad5c640 1 -- 172.21.15.77:0/2467603629 shutdown_connections 2024-03-06T07:22:34.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fceaad5c640 1 --2- 172.21.15.77:0/2467603629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea40ce3b0 0x7fcea4071f30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fceaad5c640 1 -- 172.21.15.77:0/2467603629 >> 172.21.15.77:0/2467603629 conn(0x7fcea406b830 msgr2=0x7fcea40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fceaad5c640 1 -- 172.21.15.77:0/2467603629 shutdown_connections 2024-03-06T07:22:34.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.785+0000 7fceaad5c640 1 -- 172.21.15.77:0/2467603629 wait complete. 2024-03-06T07:22:34.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.786+0000 7fceaad5c640 1 Processor -- start 2024-03-06T07:22:34.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.786+0000 7fceaad5c640 1 -- start start 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fceaad5c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea40ce3b0 0x7fcea40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fceaad5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcea415df10 0x7fcea40b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fceaad5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea415e2f0 0x7fcea41f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fceaad5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40725f0 con 0x7fcea415e2f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fceaad5c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4072470 con 0x7fcea40ce3b0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fceaad5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4072770 con 0x7fcea415df10 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fcea92f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea415e2f0 0x7fcea41f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fcea92f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea415e2f0 0x7fcea41f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43098/0 (socket says 172.21.15.77:43098) 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.787+0000 7fcea92f9640 1 -- 172.21.15.77:0/252881131 learned_addr learned my addr 172.21.15.77:0/252881131 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea8af8640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea40ce3b0 0x7fcea40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea3fff640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcea415df10 0x7fcea40b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea92f9640 1 -- 172.21.15.77:0/252881131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcea415df10 msgr2=0x7fcea40b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea92f9640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcea415df10 0x7fcea40b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea92f9640 1 -- 172.21.15.77:0/252881131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea40ce3b0 msgr2=0x7fcea40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea92f9640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea40ce3b0 0x7fcea40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea92f9640 1 -- 172.21.15.77:0/252881131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40b3c00 con 0x7fcea415e2f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea92f9640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea415e2f0 0x7fcea41f42a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fce9400c390 tx=0x7fce9400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea17fa640 1 -- 172.21.15.77:0/252881131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94017070 con 0x7fcea415e2f0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea17fa640 1 -- 172.21.15.77:0/252881131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce940052d0 con 0x7fcea415e2f0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fcea17fa640 1 -- 172.21.15.77:0/252881131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94003810 con 0x7fcea415e2f0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea40b3ee0 con 0x7fcea415e2f0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.788+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea40b44a0 con 0x7fcea415e2f0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.789+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea40b3970 con 0x7fcea415e2f0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.789+0000 7fcea17fa640 1 -- 172.21.15.77:0/252881131 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce9401a070 con 0x7fcea415e2f0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.790+0000 7fcea17fa640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce8406b4d0 0x7fce8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.790+0000 7fcea17fa640 1 -- 172.21.15.77:0/252881131 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce94089400 con 0x7fcea415e2f0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.790+0000 7fcea8af8640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce8406b4d0 0x7fce8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.790+0000 7fcea8af8640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce8406b4d0 0x7fce8406d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0012e0 tx=0x7fce9c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.794+0000 7fcea17fa640 1 -- 172.21.15.77:0/252881131 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce94057cc0 con 0x7fcea415e2f0 2024-03-06T07:22:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.965+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcea406e0b0 con 0x7fcea415e2f0 2024-03-06T07:22:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.965+0000 7fcea17fa640 1 -- 172.21.15.77:0/252881131 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fce9405bd60 con 0x7fcea415e2f0 2024-03-06T07:22:34.967 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:34.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce8406b4d0 msgr2=0x7fce8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce8406b4d0 0x7fce8406d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0012e0 tx=0x7fce9c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea415e2f0 msgr2=0x7fcea41f42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea415e2f0 0x7fcea41f42a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fce9400c390 tx=0x7fce9400c850 comp rx=0 tx=0).stop 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 shutdown_connections 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce8406b4d0 0x7fce8406d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea415e2f0 0x7fcea41f42a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcea415df10 0x7fcea40b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 --2- 172.21.15.77:0/252881131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcea40ce3b0 0x7fcea40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 >> 172.21.15.77:0/252881131 conn(0x7fcea406b830 msgr2=0x7fcea4070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 shutdown_connections 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:34.967+0000 7fceaad5c640 1 -- 172.21.15.77:0/252881131 wait complete. 2024-03-06T07:22:34.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:35.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.080+0000 7fb4dc5d0640 1 Processor -- start 2024-03-06T07:22:35.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.080+0000 7fb4dc5d0640 1 -- start start 2024-03-06T07:22:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.081+0000 7fb4dc5d0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d40b4460 0x7fb4d40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.081+0000 7fb4dc5d0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb4d4162970 0x7fb4d40b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.081+0000 7fb4dc5d0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb4d4161cf0 0x7fb4d4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.081+0000 7fb4dc5d0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d4161b00 con 0x7fb4d4162970 2024-03-06T07:22:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.081+0000 7fb4dc5d0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d40cecc0 con 0x7fb4d4161cf0 2024-03-06T07:22:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.081+0000 7fb4dc5d0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40d22c0 con 0x7fb4d40b4460 2024-03-06T07:22:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.081+0000 7fb4d9b6b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb4d4162970 0x7fb4d40b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4d9b6b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb4d4162970 0x7fb4d40b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43100/0 (socket says 172.21.15.77:43100) 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4d9b6b640 1 -- 172.21.15.77:0/3929836700 learned_addr learned my addr 172.21.15.77:0/3929836700 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4dab6d640 1 --2- 172.21.15.77:0/3929836700 >> v2:172.21.15.158:3300/0 conn(0x7fb4d4161cf0 0x7fb4d4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4da36c640 1 --2- 172.21.15.77:0/3929836700 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d40b4460 0x7fb4d40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4d9b6b640 1 -- 172.21.15.77:0/3929836700 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d40b4460 msgr2=0x7fb4d40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4d9b6b640 1 --2- 172.21.15.77:0/3929836700 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d40b4460 0x7fb4d40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4d9b6b640 1 -- 172.21.15.77:0/3929836700 >> v2:172.21.15.158:3300/0 conn(0x7fb4d4161cf0 msgr2=0x7fb4d4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4d9b6b640 1 --2- 172.21.15.77:0/3929836700 >> v2:172.21.15.158:3300/0 conn(0x7fb4d4161cf0 0x7fb4d4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.082+0000 7fb4d9b6b640 1 -- 172.21.15.77:0/3929836700 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d40d25a0 con 0x7fb4d4162970 2024-03-06T07:22:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4d9b6b640 1 --2- 172.21.15.77:0/3929836700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4162970 0x7fb4d40b3580 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb4d40ce2c0 tx=0x7fb4c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f524dc37b95c1dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4d936a640 1 -- 172.21.15.77:0/3929836700 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4035020 con 0x7fb4d4162970 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4d936a640 1 -- 172.21.15.77:0/3929836700 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c4031870 con 0x7fb4d4162970 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4d936a640 1 -- 172.21.15.77:0/3929836700 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4031c40 con 0x7fb4d4162970 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/3929836700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4162970 msgr2=0x7fb4d40b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/3929836700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4162970 0x7fb4d40b3580 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb4d40ce2c0 tx=0x7fb4c402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/3929836700 shutdown_connections 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/3929836700 >> v2:172.21.15.158:3300/0 conn(0x7fb4d4161cf0 0x7fb4d4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/3929836700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4162970 0x7fb4d40b3580 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/3929836700 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d40b4460 0x7fb4d40d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/3929836700 >> 172.21.15.77:0/3929836700 conn(0x7fb4d40d3ce0 msgr2=0x7fb4d40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/3929836700 shutdown_connections 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.083+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/3929836700 wait complete. 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.084+0000 7fb4dc5d0640 1 Processor -- start 2024-03-06T07:22:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.084+0000 7fb4dc5d0640 1 -- start start 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4dc5d0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4d40b4460 0x7fb4d40ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4dc5d0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4161cf0 0x7fb4d41601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4dc5d0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d4162970 0x7fb4d40c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4dc5d0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40ce950 con 0x7fb4d4161cf0 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4dc5d0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40d2b60 con 0x7fb4d40b4460 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4dc5d0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40b3ac0 con 0x7fb4d4162970 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4d9b6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4161cf0 0x7fb4d41601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4d9b6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4161cf0 0x7fb4d41601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43110/0 (socket says 172.21.15.77:43110) 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4d9b6b640 1 -- 172.21.15.77:0/641864880 learned_addr learned my addr 172.21.15.77:0/641864880 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4dab6d640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d4162970 0x7fb4d40c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.085+0000 7fb4da36c640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4d40b4460 0x7fb4d40ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4d9b6b640 1 -- 172.21.15.77:0/641864880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d4162970 msgr2=0x7fb4d40c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4d9b6b640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d4162970 0x7fb4d40c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4d9b6b640 1 -- 172.21.15.77:0/641864880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4d40b4460 msgr2=0x7fb4d40ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4d9b6b640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4d40b4460 0x7fb4d40ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4d9b6b640 1 -- 172.21.15.77:0/641864880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d40b8670 con 0x7fb4d4161cf0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4d9b6b640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4161cf0 0x7fb4d41601b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402b950 tx=0x7fb4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4caffd640 1 -- 172.21.15.77:0/641864880 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4040070 con 0x7fb4d4161cf0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4caffd640 1 -- 172.21.15.77:0/641864880 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c4031e10 con 0x7fb4d4161cf0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4caffd640 1 -- 172.21.15.77:0/641864880 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4035040 con 0x7fb4d4161cf0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d40b8950 con 0x7fb4d4161cf0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.086+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d40b91f0 con 0x7fb4d4161cf0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.087+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d40b8d40 con 0x7fb4d4161cf0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.087+0000 7fb4caffd640 1 -- 172.21.15.77:0/641864880 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4c404b060 con 0x7fb4d4161cf0 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.088+0000 7fb4caffd640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b806b4d0 0x7fb4b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.088+0000 7fb4caffd640 1 -- 172.21.15.77:0/641864880 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4c40abcc0 con 0x7fb4d4161cf0 2024-03-06T07:22:35.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.088+0000 7fb4da36c640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b806b4d0 0x7fb4b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.088+0000 7fb4da36c640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b806b4d0 0x7fb4b806d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009aa0 tx=0x7fb4d0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.092+0000 7fb4caffd640 1 -- 172.21.15.77:0/641864880 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4c407a580 con 0x7fb4d4161cf0 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.274+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb4d404f330 con 0x7fb4d4161cf0 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.275+0000 7fb4caffd640 1 -- 172.21.15.77:0/641864880 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb4c407e620 con 0x7fb4d4161cf0 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:22:35.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b806b4d0 msgr2=0x7fb4b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b806b4d0 0x7fb4b806d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009aa0 tx=0x7fb4d0009190 comp rx=0 tx=0).stop 2024-03-06T07:22:35.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4161cf0 msgr2=0x7fb4d41601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4161cf0 0x7fb4d41601b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402b950 tx=0x7fb4c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 shutdown_connections 2024-03-06T07:22:35.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4b806b4d0 0x7fb4b806d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4d4161cf0 0x7fb4d41601b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4d4162970 0x7fb4d40c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 --2- 172.21.15.77:0/641864880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4d40b4460 0x7fb4d40ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 >> 172.21.15.77:0/641864880 conn(0x7fb4d40d3ce0 msgr2=0x7fb4d406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 shutdown_connections 2024-03-06T07:22:35.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.277+0000 7fb4dc5d0640 1 -- 172.21.15.77:0/641864880 wait complete. 2024-03-06T07:22:35.290 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:22:35.290 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:35.307 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:35.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.406+0000 7fc076ceb640 1 Processor -- start 2024-03-06T07:22:35.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.406+0000 7fc076ceb640 1 -- start start 2024-03-06T07:22:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.406+0000 7fc076ceb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015fa10 0x7fc0700ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.406+0000 7fc076ceb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0700cebe0 con 0x7fc0700d16e0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.406+0000 7fc076ceb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0700ceec0 con 0x7fc07015bdd0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.406+0000 7fc076ceb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700cf1a0 con 0x7fc07015fa10 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.406+0000 7fc075288640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc0700d16e0 0x7fc0700d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52518/0 (socket says 172.21.15.77:52518) 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.406+0000 7fc075288640 1 -- 172.21.15.77:0/2602073311 learned_addr learned my addr 172.21.15.77:0/2602073311 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.407+0000 7fc06ffff640 1 --2- 172.21.15.77:0/2602073311 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015fa10 0x7fc0700ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.407+0000 7fc06f7fe640 1 -- 172.21.15.77:0/2602073311 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970011481 0 0) 0x7fc0700cebe0 con 0x7fc0700d16e0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.407+0000 7fc06f7fe640 1 -- 172.21.15.77:0/2602073311 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc060003520 con 0x7fc0700d16e0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.407+0000 7fc06f7fe640 1 -- 172.21.15.77:0/2602073311 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067286127 0 0) 0x7fc0700ceec0 con 0x7fc07015bdd0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.407+0000 7fc06f7fe640 1 -- 172.21.15.77:0/2602073311 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0700cebe0 con 0x7fc07015bdd0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.407+0000 7fc06ffff640 1 -- 172.21.15.77:0/2602073311 >> v1:172.21.15.158:6789/0 conn(0x7fc07015bdd0 legacy=0x7fc0700c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.407+0000 7fc06ffff640 1 -- 172.21.15.77:0/2602073311 >> v1:172.21.15.77:6789/0 conn(0x7fc0700d16e0 legacy=0x7fc0700d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.407+0000 7fc06ffff640 1 -- 172.21.15.77:0/2602073311 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0700cc8e0 con 0x7fc07015fa10 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc06ffff640 1 --2- 172.21.15.77:0/2602073311 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015fa10 0x7fc0700ce270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0097d0 tx=0x7fc05c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ca028703b544f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc06f7fe640 1 -- 172.21.15.77:0/2602073311 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c035020 con 0x7fc07015fa10 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc06f7fe640 1 -- 172.21.15.77:0/2602073311 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c031800 con 0x7fc07015fa10 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc06f7fe640 1 -- 172.21.15.77:0/2602073311 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c031bd0 con 0x7fc07015fa10 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc076ceb640 1 -- 172.21.15.77:0/2602073311 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015fa10 msgr2=0x7fc0700ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc076ceb640 1 --2- 172.21.15.77:0/2602073311 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015fa10 0x7fc0700ce270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0097d0 tx=0x7fc05c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc076ceb640 1 -- 172.21.15.77:0/2602073311 shutdown_connections 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc076ceb640 1 --2- 172.21.15.77:0/2602073311 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015fa10 0x7fc0700ce270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc076ceb640 1 -- 172.21.15.77:0/2602073311 >> 172.21.15.77:0/2602073311 conn(0x7fc07009a9f0 msgr2=0x7fc07009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc076ceb640 1 -- 172.21.15.77:0/2602073311 shutdown_connections 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.408+0000 7fc076ceb640 1 -- 172.21.15.77:0/2602073311 wait complete. 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.410+0000 7fc076ceb640 1 Processor -- start 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.410+0000 7fc076ceb640 1 -- start start 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.410+0000 7fc076ceb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0700d16e0 0x7fc07015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.410+0000 7fc076ceb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015bdd0 0x7fc07015c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.410+0000 7fc076ceb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc07015fa10 0x7fc07015e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.410+0000 7fc076ceb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700ce7b0 con 0x7fc07015fa10 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.410+0000 7fc076ceb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07015c1b0 con 0x7fc0700d16e0 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.410+0000 7fc076ceb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700ce930 con 0x7fc07015bdd0 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc07015fa10 0x7fc07015e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc07015fa10 0x7fc07015e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43124/0 (socket says 172.21.15.77:43124) 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 -- 172.21.15.77:0/2146667541 learned_addr learned my addr 172.21.15.77:0/2146667541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc074a87640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0700d16e0 0x7fc07015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc06ffff640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015bdd0 0x7fc07015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 -- 172.21.15.77:0/2146667541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015bdd0 msgr2=0x7fc07015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015bdd0 0x7fc07015c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 -- 172.21.15.77:0/2146667541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0700d16e0 msgr2=0x7fc07015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0700d16e0 0x7fc07015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 -- 172.21.15.77:0/2146667541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0700b4f40 con 0x7fc07015fa10 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc075288640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc07015fa10 0x7fc07015e3d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc06800caf0 tx=0x7fc06800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc06d7fa640 1 -- 172.21.15.77:0/2146667541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068017070 con 0x7fc07015fa10 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc06d7fa640 1 -- 172.21.15.77:0/2146667541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06800aca0 con 0x7fc07015fa10 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc06d7fa640 1 -- 172.21.15.77:0/2146667541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068012620 con 0x7fc07015fa10 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0700b5220 con 0x7fc07015fa10 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.411+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0700b57e0 con 0x7fc07015fa10 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.412+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07004f330 con 0x7fc07015fa10 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.413+0000 7fc06d7fa640 1 -- 172.21.15.77:0/2146667541 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc06801a070 con 0x7fc07015fa10 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.413+0000 7fc06d7fa640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc05406b500 0x7fc05406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.413+0000 7fc06d7fa640 1 -- 172.21.15.77:0/2146667541 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc068089e80 con 0x7fc07015fa10 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.413+0000 7fc074a87640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc05406b500 0x7fc05406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.413+0000 7fc074a87640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc05406b500 0x7fc05406d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc064007fd0 tx=0x7fc064009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.417+0000 7fc06d7fa640 1 -- 172.21.15.77:0/2146667541 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0680586c0 con 0x7fc07015fa10 2024-03-06T07:22:35.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.591+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0700b7a40 con 0x7fc07015fa10 2024-03-06T07:22:35.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.592+0000 7fc06d7fa640 1 -- 172.21.15.77:0/2146667541 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fc06805c760 con 0x7fc07015fa10 2024-03-06T07:22:35.594 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:35.594 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:35.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc05406b500 msgr2=0x7fc05406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc05406b500 0x7fc05406d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc064007fd0 tx=0x7fc064009040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc07015fa10 msgr2=0x7fc07015e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc07015fa10 0x7fc07015e3d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc06800caf0 tx=0x7fc06800a060 comp rx=0 tx=0).stop 2024-03-06T07:22:35.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 shutdown_connections 2024-03-06T07:22:35.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc05406b500 0x7fc05406d9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc07015fa10 0x7fc07015e3d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc07015bdd0 0x7fc07015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 --2- 172.21.15.77:0/2146667541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0700d16e0 0x7fc07015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 >> 172.21.15.77:0/2146667541 conn(0x7fc07009a9f0 msgr2=0x7fc0700d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 shutdown_connections 2024-03-06T07:22:35.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.594+0000 7fc076ceb640 1 -- 172.21.15.77:0/2146667541 wait complete. 2024-03-06T07:22:35.607 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:35.607 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:35.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.710+0000 7f62d5965640 1 Processor -- start 2024-03-06T07:22:35.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.710+0000 7f62d5965640 1 -- start start 2024-03-06T07:22:35.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62d5965640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f62d01614d0 0x7f62d00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62d5965640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 0x7f62d00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62d5965640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d01618b0 con 0x7f62d00d1910 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62d5965640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d00cf400 con 0x7f62d01614d0 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62d5965640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d00cf6a0 con 0x7f62d00d24b0 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62c7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f62d00d1910 0x7f62d00d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52520/0 (socket says 172.21.15.77:52520) 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62c7fff640 1 -- 172.21.15.77:0/4032872808 learned_addr learned my addr 172.21.15.77:0/4032872808 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62ceffd640 1 --2- 172.21.15.77:0/4032872808 >> v2:172.21.15.158:3300/0 conn(0x7f62d01614d0 0x7f62d00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62cf7fe640 1 --2- 172.21.15.77:0/4032872808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 0x7f62d00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62ce7fc640 1 -- 172.21.15.77:0/4032872808 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421965465 0 0) 0x7f62d01618b0 con 0x7f62d00d1910 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62ce7fc640 1 -- 172.21.15.77:0/4032872808 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc003520 con 0x7f62d00d1910 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62ceffd640 1 -- 172.21.15.77:0/4032872808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 msgr2=0x7f62d00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62ceffd640 1 --2- 172.21.15.77:0/4032872808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 0x7f62d00b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62ceffd640 1 -- 172.21.15.77:0/4032872808 >> v1:172.21.15.77:6789/0 conn(0x7f62d00d1910 legacy=0x7f62d00d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.711+0000 7f62ceffd640 1 -- 172.21.15.77:0/4032872808 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d00cf980 con 0x7f62d01614d0 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.712+0000 7f62ceffd640 1 --2- 172.21.15.77:0/4032872808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 0x7f62d00d4860 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f62c00097a0 tx=0x7f62c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2702397e6a110561 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.712+0000 7f62ce7fc640 1 -- 172.21.15.77:0/4032872808 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0035020 con 0x7f62d01614d0 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.712+0000 7f62ce7fc640 1 -- 172.21.15.77:0/4032872808 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c0031830 con 0x7f62d01614d0 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.712+0000 7f62ce7fc640 1 -- 172.21.15.77:0/4032872808 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0031be0 con 0x7f62d01614d0 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.713+0000 7f62d5965640 1 -- 172.21.15.77:0/4032872808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 msgr2=0x7f62d00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.713+0000 7f62d5965640 1 --2- 172.21.15.77:0/4032872808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 0x7f62d00d4860 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f62c00097a0 tx=0x7f62c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.713+0000 7f62d5965640 1 -- 172.21.15.77:0/4032872808 shutdown_connections 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.713+0000 7f62d5965640 1 --2- 172.21.15.77:0/4032872808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 0x7f62d00b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.713+0000 7f62d5965640 1 --2- 172.21.15.77:0/4032872808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 0x7f62d00d4860 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.713+0000 7f62d5965640 1 -- 172.21.15.77:0/4032872808 >> 172.21.15.77:0/4032872808 conn(0x7f62d00d3d40 msgr2=0x7f62d00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.713+0000 7f62d5965640 1 -- 172.21.15.77:0/4032872808 shutdown_connections 2024-03-06T07:22:35.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.713+0000 7f62d5965640 1 -- 172.21.15.77:0/4032872808 wait complete. 2024-03-06T07:22:35.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.714+0000 7f62d5965640 1 Processor -- start 2024-03-06T07:22:35.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.714+0000 7f62d5965640 1 -- start start 2024-03-06T07:22:35.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62d5965640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d00d1910 0x7f62d01f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62d5965640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 0x7f62d01f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62d5965640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 0x7f62d00c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62d5965640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d00b1370 con 0x7f62d00d1910 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62d5965640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d00b0910 con 0x7f62d01614d0 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62d5965640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d00b14f0 con 0x7f62d00d24b0 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d00d1910 0x7f62d01f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d00d1910 0x7f62d01f44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43138/0 (socket says 172.21.15.77:43138) 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 -- 172.21.15.77:0/3482678589 learned_addr learned my addr 172.21.15.77:0/3482678589 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62c7fff640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 0x7f62d01f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62cf7fe640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 0x7f62d00c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 -- 172.21.15.77:0/3482678589 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 msgr2=0x7f62d01f78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 0x7f62d01f78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 -- 172.21.15.77:0/3482678589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 msgr2=0x7f62d00c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 0x7f62d00c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 -- 172.21.15.77:0/3482678589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d00c7b00 con 0x7f62d00d1910 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ceffd640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d00d1910 0x7f62d01f44c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f62c002b910 tx=0x7f62c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ccff9640 1 -- 172.21.15.77:0/3482678589 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0040070 con 0x7f62d00d1910 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ccff9640 1 -- 172.21.15.77:0/3482678589 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c003a910 con 0x7f62d00d1910 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d00c96b0 con 0x7f62d00d1910 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62ccff9640 1 -- 172.21.15.77:0/3482678589 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0035040 con 0x7f62d00d1910 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.715+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d01f70e0 con 0x7f62d00d1910 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.716+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d0068ff0 con 0x7f62d00d1910 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.717+0000 7f62ccff9640 1 -- 172.21.15.77:0/3482678589 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62c004b060 con 0x7f62d00d1910 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.717+0000 7f62ccff9640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62b006b4d0 0x7f62b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.717+0000 7f62ccff9640 1 -- 172.21.15.77:0/3482678589 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62c00ab7f0 con 0x7f62d00d1910 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.717+0000 7f62c7fff640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62b006b4d0 0x7f62b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.717+0000 7f62c7fff640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62b006b4d0 0x7f62b006d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f62d00b54a0 tx=0x7f62b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.721+0000 7f62ccff9640 1 -- 172.21.15.77:0/3482678589 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62c007a030 con 0x7f62d00d1910 2024-03-06T07:22:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.892+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62d00d3160 con 0x7f62d00d1910 2024-03-06T07:22:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.892+0000 7f62ccff9640 1 -- 172.21.15.77:0/3482678589 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62c007e0d0 con 0x7f62d00d1910 2024-03-06T07:22:35.894 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62b006b4d0 msgr2=0x7f62b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62b006b4d0 0x7f62b006d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f62d00b54a0 tx=0x7f62b8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d00d1910 msgr2=0x7f62d01f44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d00d1910 0x7f62d01f44c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f62c002b910 tx=0x7f62c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 shutdown_connections 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f62d01614d0 0x7f62d00c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62d00d24b0 0x7f62d01f78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62b006b4d0 0x7f62b006d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 --2- 172.21.15.77:0/3482678589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62d00d1910 0x7f62d01f44c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 >> 172.21.15.77:0/3482678589 conn(0x7f62d00d3d40 msgr2=0x7f62d00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 shutdown_connections 2024-03-06T07:22:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:35.894+0000 7f62d5965640 1 -- 172.21.15.77:0/3482678589 wait complete. 2024-03-06T07:22:35.907 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.008+0000 7f4cef3bd640 1 Processor -- start 2024-03-06T07:22:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.009+0000 7f4cef3bd640 1 -- start start 2024-03-06T07:22:36.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.009+0000 7f4cef3bd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4ce806fd10 0x7f4ce80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.009+0000 7f4cef3bd640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce80cd4b0 con 0x7f4ce815c050 2024-03-06T07:22:36.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.009+0000 7f4cef3bd640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce80cd790 con 0x7f4ce80d0af0 2024-03-06T07:22:36.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.009+0000 7f4cef3bd640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce80cda70 con 0x7f4ce806fd10 2024-03-06T07:22:36.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4ced159640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4ce815c050 0x7f4ce806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52526/0 (socket says 172.21.15.77:52526) 2024-03-06T07:22:36.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4ced159640 1 -- 172.21.15.77:0/2889934237 learned_addr learned my addr 172.21.15.77:0/2889934237 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:36.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4cec958640 1 --2- 172.21.15.77:0/2889934237 >> v2:172.21.15.158:3300/0 conn(0x7f4ce806fd10 0x7f4ce80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4cdffff640 1 -- 172.21.15.77:0/2889934237 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502005334 0 0) 0x7f4ce80cd4b0 con 0x7f4ce815c050 2024-03-06T07:22:36.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4cdffff640 1 -- 172.21.15.77:0/2889934237 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd4003520 con 0x7f4ce815c050 2024-03-06T07:22:36.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4cdffff640 1 -- 172.21.15.77:0/2889934237 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239310197 0 0) 0x7f4ce80cd790 con 0x7f4ce80d0af0 2024-03-06T07:22:36.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4cdffff640 1 -- 172.21.15.77:0/2889934237 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce80cd4b0 con 0x7f4ce80d0af0 2024-03-06T07:22:36.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4cec958640 1 -- 172.21.15.77:0/2889934237 >> v1:172.21.15.158:6789/0 conn(0x7f4ce80d0af0 legacy=0x7f4ce80ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4cec958640 1 -- 172.21.15.77:0/2889934237 >> v1:172.21.15.77:6789/0 conn(0x7f4ce815c050 legacy=0x7f4ce806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.010+0000 7f4cec958640 1 -- 172.21.15.77:0/2889934237 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce80cdd50 con 0x7f4ce806fd10 2024-03-06T07:22:36.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cec958640 1 --2- 172.21.15.77:0/2889934237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce806fd10 0x7f4ce80cc760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40097a0 tx=0x7f4ce402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12db86cd497074e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cdffff640 1 -- 172.21.15.77:0/2889934237 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4035020 con 0x7f4ce806fd10 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cdffff640 1 -- 172.21.15.77:0/2889934237 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce4031830 con 0x7f4ce806fd10 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cdffff640 1 -- 172.21.15.77:0/2889934237 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4031c00 con 0x7f4ce806fd10 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cef3bd640 1 -- 172.21.15.77:0/2889934237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce806fd10 msgr2=0x7f4ce80cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cef3bd640 1 --2- 172.21.15.77:0/2889934237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce806fd10 0x7f4ce80cc760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40097a0 tx=0x7f4ce402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cef3bd640 1 -- 172.21.15.77:0/2889934237 shutdown_connections 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cef3bd640 1 --2- 172.21.15.77:0/2889934237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce806fd10 0x7f4ce80cc760 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.011+0000 7f4cef3bd640 1 -- 172.21.15.77:0/2889934237 >> 172.21.15.77:0/2889934237 conn(0x7f4ce80b3800 msgr2=0x7f4ce80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.012+0000 7f4cef3bd640 1 -- 172.21.15.77:0/2889934237 shutdown_connections 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.012+0000 7f4cef3bd640 1 -- 172.21.15.77:0/2889934237 wait complete. 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cef3bd640 1 Processor -- start 2024-03-06T07:22:36.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cef3bd640 1 -- start start 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cef3bd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ce806fd10 0x7f4ce814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cef3bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce80d0af0 0x7f4ce81f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cef3bd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815c050 0x7f4ce8153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cef3bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce8166e50 con 0x7f4ce80d0af0 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cef3bd640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce8166cd0 con 0x7f4ce815c050 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cef3bd640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce8166fd0 con 0x7f4ce806fd10 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cec958640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce80d0af0 0x7f4ce81f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cec958640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce80d0af0 0x7f4ce81f5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43144/0 (socket says 172.21.15.77:43144) 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.013+0000 7f4cec958640 1 -- 172.21.15.77:0/3634332624 learned_addr learned my addr 172.21.15.77:0/3634332624 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:36.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4ced159640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ce806fd10 0x7f4ce814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4ced95a640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815c050 0x7f4ce8153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cec958640 1 -- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ce806fd10 msgr2=0x7f4ce814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cec958640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ce806fd10 0x7f4ce814f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cec958640 1 -- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815c050 msgr2=0x7f4ce8153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cec958640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815c050 0x7f4ce8153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cec958640 1 -- 172.21.15.77:0/3634332624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce814fec0 con 0x7f4ce80d0af0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4ced95a640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815c050 0x7f4ce8153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cec958640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce80d0af0 0x7f4ce81f5c80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40098c0 tx=0x7f4ce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cddffb640 1 -- 172.21.15.77:0/3634332624 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4040070 con 0x7f4ce80d0af0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cddffb640 1 -- 172.21.15.77:0/3634332624 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce403e070 con 0x7f4ce80d0af0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce8150660 con 0x7f4ce80d0af0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cddffb640 1 -- 172.21.15.77:0/3634332624 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4035040 con 0x7f4ce80d0af0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.014+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce8150c20 con 0x7f4ce80d0af0 2024-03-06T07:22:36.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.015+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce809fe60 con 0x7f4ce80d0af0 2024-03-06T07:22:36.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.016+0000 7f4cddffb640 1 -- 172.21.15.77:0/3634332624 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ce4038020 con 0x7f4ce80d0af0 2024-03-06T07:22:36.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.016+0000 7f4cddffb640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ccc06b4d0 0x7f4ccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.016+0000 7f4ced159640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ccc06b4d0 0x7f4ccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.016+0000 7f4cddffb640 1 -- 172.21.15.77:0/3634332624 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ce40ab5f0 con 0x7f4ce80d0af0 2024-03-06T07:22:36.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.017+0000 7f4ced159640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ccc06b4d0 0x7f4ccc06d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8000f60 tx=0x7f4cd8007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.021+0000 7f4cddffb640 1 -- 172.21.15.77:0/3634332624 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ce407a480 con 0x7f4ce80d0af0 2024-03-06T07:22:36.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.193+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4ce81580f0 con 0x7f4ce80d0af0 2024-03-06T07:22:36.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.194+0000 7f4cddffb640 1 -- 172.21.15.77:0/3634332624 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4ce4079d80 con 0x7f4ce80d0af0 2024-03-06T07:22:36.196 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ccc06b4d0 msgr2=0x7f4ccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ccc06b4d0 0x7f4ccc06d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8000f60 tx=0x7f4cd8007d70 comp rx=0 tx=0).stop 2024-03-06T07:22:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce80d0af0 msgr2=0x7f4ce81f5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce80d0af0 0x7f4ce81f5c80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40098c0 tx=0x7f4ce40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 shutdown_connections 2024-03-06T07:22:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ccc06b4d0 0x7f4ccc06d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ce80d0af0 0x7f4ce81f5c80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ce815c050 0x7f4ce8153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 --2- 172.21.15.77:0/3634332624 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ce806fd10 0x7f4ce814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 >> 172.21.15.77:0/3634332624 conn(0x7f4ce80b3800 msgr2=0x7f4ce80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 shutdown_connections 2024-03-06T07:22:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.196+0000 7f4cef3bd640 1 -- 172.21.15.77:0/3634332624 wait complete. 2024-03-06T07:22:36.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:36.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.310+0000 7f8f39f08640 1 Processor -- start 2024-03-06T07:22:36.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.310+0000 7f8f39f08640 1 -- start start 2024-03-06T07:22:36.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.310+0000 7f8f39f08640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f341614d0 0x7f8f340d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.310+0000 7f8f39f08640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f340d1910 0x7f8f340d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.310+0000 7f8f39f08640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f341618b0 con 0x7f8f340d22e0 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.310+0000 7f8f39f08640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f340cf400 con 0x7f8f340d1910 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.310+0000 7f8f39f08640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f340d2710 con 0x7f8f341614d0 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f32ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f340d1910 0x7f8f340d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f337fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f341614d0 0x7f8f340d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f32ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f340d1910 0x7f8f340d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43154/0 (socket says 172.21.15.77:43154) 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f337fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f341614d0 0x7f8f340d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:56466/0 (socket says 172.21.15.77:56466) 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f32ffd640 1 -- 172.21.15.77:0/1344574835 learned_addr learned my addr 172.21.15.77:0/1344574835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f32ffd640 1 -- 172.21.15.77:0/1344574835 >> v2:172.21.15.158:3300/0 conn(0x7f8f341614d0 msgr2=0x7f8f340d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f32ffd640 1 --2- 172.21.15.77:0/1344574835 >> v2:172.21.15.158:3300/0 conn(0x7f8f341614d0 0x7f8f340d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f32ffd640 1 -- 172.21.15.77:0/1344574835 >> v1:172.21.15.158:6789/0 conn(0x7f8f340d22e0 legacy=0x7f8f340d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.311+0000 7f8f32ffd640 1 -- 172.21.15.77:0/1344574835 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f340cf6a0 con 0x7f8f340d1910 2024-03-06T07:22:36.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f32ffd640 1 --2- 172.21.15.77:0/1344574835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d1910 0x7f8f340d1da0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8f3408c470 tx=0x7f8f2802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6945a79492e522a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f327fc640 1 -- 172.21.15.77:0/1344574835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28034070 con 0x7f8f340d1910 2024-03-06T07:22:36.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f327fc640 1 -- 172.21.15.77:0/1344574835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f28030860 con 0x7f8f340d1910 2024-03-06T07:22:36.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f327fc640 1 -- 172.21.15.77:0/1344574835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28030c30 con 0x7f8f340d1910 2024-03-06T07:22:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f39f08640 1 -- 172.21.15.77:0/1344574835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d1910 msgr2=0x7f8f340d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f39f08640 1 --2- 172.21.15.77:0/1344574835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d1910 0x7f8f340d1da0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8f3408c470 tx=0x7f8f2802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f39f08640 1 -- 172.21.15.77:0/1344574835 shutdown_connections 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f39f08640 1 --2- 172.21.15.77:0/1344574835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d1910 0x7f8f340d1da0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f39f08640 1 --2- 172.21.15.77:0/1344574835 >> v2:172.21.15.158:3300/0 conn(0x7f8f341614d0 0x7f8f340d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f39f08640 1 -- 172.21.15.77:0/1344574835 >> 172.21.15.77:0/1344574835 conn(0x7f8f340d3d40 msgr2=0x7f8f340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f39f08640 1 -- 172.21.15.77:0/1344574835 shutdown_connections 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.312+0000 7f8f39f08640 1 -- 172.21.15.77:0/1344574835 wait complete. 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.313+0000 7f8f39f08640 1 Processor -- start 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.314+0000 7f8f39f08640 1 -- start start 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.314+0000 7f8f39f08640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f340d1910 0x7f8f340cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.314+0000 7f8f39f08640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d22e0 0x7f8f3414f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.314+0000 7f8f39f08640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f341614d0 0x7f8f340b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.314+0000 7f8f39f08640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340b0910 con 0x7f8f340d22e0 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.314+0000 7f8f39f08640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340cfc60 con 0x7f8f341614d0 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.314+0000 7f8f39f08640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340b1e40 con 0x7f8f340d1910 2024-03-06T07:22:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d22e0 0x7f8f3414f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d22e0 0x7f8f3414f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43168/0 (socket says 172.21.15.77:43168) 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 -- 172.21.15.77:0/2322350683 learned_addr learned my addr 172.21.15.77:0/2322350683 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f33fff640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f341614d0 0x7f8f340b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f337fe640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f340d1910 0x7f8f340cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 -- 172.21.15.77:0/2322350683 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f340d1910 msgr2=0x7f8f340cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f340d1910 0x7f8f340cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 -- 172.21.15.77:0/2322350683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f341614d0 msgr2=0x7f8f340b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f341614d0 0x7f8f340b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 -- 172.21.15.77:0/2322350683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f340b63e0 con 0x7f8f340d22e0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f32ffd640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d22e0 0x7f8f3414f290 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8f28000c00 tx=0x7f8f28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f1bfff640 1 -- 172.21.15.77:0/2322350683 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28048070 con 0x7f8f340d22e0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f1bfff640 1 -- 172.21.15.77:0/2322350683 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f28030660 con 0x7f8f340d22e0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f340b5340 con 0x7f8f340d22e0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f1bfff640 1 -- 172.21.15.77:0/2322350683 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28034040 con 0x7f8f340d22e0 2024-03-06T07:22:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.315+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f340b5900 con 0x7f8f340d22e0 2024-03-06T07:22:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.316+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f34068ff0 con 0x7f8f340d22e0 2024-03-06T07:22:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.317+0000 7f8f1bfff640 1 -- 172.21.15.77:0/2322350683 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f2804b070 con 0x7f8f340d22e0 2024-03-06T07:22:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.317+0000 7f8f1bfff640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f2006b500 0x7f8f2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.317+0000 7f8f1bfff640 1 -- 172.21.15.77:0/2322350683 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f280ac190 con 0x7f8f340d22e0 2024-03-06T07:22:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.317+0000 7f8f337fe640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f2006b500 0x7f8f2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.318+0000 7f8f337fe640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f2006b500 0x7f8f2006d9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009aa0 tx=0x7f8f24009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.321+0000 7f8f1bfff640 1 -- 172.21.15.77:0/2322350683 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f28030830 con 0x7f8f340d22e0 2024-03-06T07:22:36.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.491+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f340caab0 con 0x7f8f340d22e0 2024-03-06T07:22:36.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.492+0000 7f8f1bfff640 1 -- 172.21.15.77:0/2322350683 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8f2807a9c0 con 0x7f8f340d22e0 2024-03-06T07:22:36.494 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f2006b500 msgr2=0x7f8f2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f2006b500 0x7f8f2006d9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009aa0 tx=0x7f8f24009190 comp rx=0 tx=0).stop 2024-03-06T07:22:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d22e0 msgr2=0x7f8f3414f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d22e0 0x7f8f3414f290 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8f28000c00 tx=0x7f8f28004730 comp rx=0 tx=0).stop 2024-03-06T07:22:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 shutdown_connections 2024-03-06T07:22:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f2006b500 0x7f8f2006d9c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f340d22e0 0x7f8f3414f290 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f341614d0 0x7f8f340b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 --2- 172.21.15.77:0/2322350683 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f340d1910 0x7f8f340cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 >> 172.21.15.77:0/2322350683 conn(0x7f8f340d3d40 msgr2=0x7f8f340cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 shutdown_connections 2024-03-06T07:22:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.494+0000 7f8f39f08640 1 -- 172.21.15.77:0/2322350683 wait complete. 2024-03-06T07:22:36.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:36.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.606+0000 7f4b7f6eb640 1 Processor -- start 2024-03-06T07:22:36.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.606+0000 7f4b7f6eb640 1 -- start start 2024-03-06T07:22:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7f6eb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b78072520 0x7f4b78066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7f6eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b780b5c30 0x7f4b78072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7f6eb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b78072e40 con 0x7f4b7815d4f0 2024-03-06T07:22:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7f6eb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b78067240 con 0x7f4b78072520 2024-03-06T07:22:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7f6eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7806c460 con 0x7f4b780b5c30 2024-03-06T07:22:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7d487640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b78072520 0x7f4b78066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7cc86640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b780b5c30 0x7f4b78072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7d487640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b78072520 0x7f4b78066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43176/0 (socket says 172.21.15.77:43176) 2024-03-06T07:22:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7d487640 1 -- 172.21.15.77:0/3191230774 learned_addr learned my addr 172.21.15.77:0/3191230774 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7cc86640 1 -- 172.21.15.77:0/3191230774 >> v2:172.21.15.77:3300/0 conn(0x7f4b78072520 msgr2=0x7f4b78066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7cc86640 1 --2- 172.21.15.77:0/3191230774 >> v2:172.21.15.77:3300/0 conn(0x7f4b78072520 0x7f4b78066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7cc86640 1 -- 172.21.15.77:0/3191230774 >> v1:172.21.15.77:6789/0 conn(0x7f4b7815d4f0 legacy=0x7f4b7815d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.607+0000 7f4b7cc86640 1 -- 172.21.15.77:0/3191230774 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7806c740 con 0x7f4b780b5c30 2024-03-06T07:22:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.608+0000 7f4b7cc86640 1 --2- 172.21.15.77:0/3191230774 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b780b5c30 0x7f4b78072900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b680097a0 tx=0x7f4b6802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7539770a4ab10d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.608+0000 7f4b6ffff640 1 -- 172.21.15.77:0/3191230774 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b680315a0 con 0x7f4b780b5c30 2024-03-06T07:22:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.608+0000 7f4b6ffff640 1 -- 172.21.15.77:0/3191230774 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b68037070 con 0x7f4b780b5c30 2024-03-06T07:22:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.608+0000 7f4b6ffff640 1 -- 172.21.15.77:0/3191230774 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68031960 con 0x7f4b780b5c30 2024-03-06T07:22:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.609+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/3191230774 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b780b5c30 msgr2=0x7f4b78072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.609+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/3191230774 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b780b5c30 0x7f4b78072900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b680097a0 tx=0x7f4b6802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.609+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/3191230774 shutdown_connections 2024-03-06T07:22:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.609+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/3191230774 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b780b5c30 0x7f4b78072900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.609+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/3191230774 >> v2:172.21.15.77:3300/0 conn(0x7f4b78072520 0x7f4b78066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.609+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/3191230774 >> 172.21.15.77:0/3191230774 conn(0x7f4b780b4e60 msgr2=0x7f4b780b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.609+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/3191230774 shutdown_connections 2024-03-06T07:22:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.609+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/3191230774 wait complete. 2024-03-06T07:22:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.610+0000 7f4b7f6eb640 1 Processor -- start 2024-03-06T07:22:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.610+0000 7f4b7f6eb640 1 -- start start 2024-03-06T07:22:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7f6eb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b78072520 0x7f4b7814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7f6eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b780b5c30 0x7f4b7809b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7f6eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b7815d4f0 0x7f4b7806e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7f6eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7815e050 con 0x7f4b780b5c30 2024-03-06T07:22:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7f6eb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7815fe30 con 0x7f4b78072520 2024-03-06T07:22:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7f6eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7815d040 con 0x7f4b7815d4f0 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b780b5c30 0x7f4b7809b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b780b5c30 0x7f4b7809b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43186/0 (socket says 172.21.15.77:43186) 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 -- 172.21.15.77:0/1306745431 learned_addr learned my addr 172.21.15.77:0/1306745431 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7d487640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b78072520 0x7f4b7814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7dc88640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b7815d4f0 0x7f4b7806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 -- 172.21.15.77:0/1306745431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b7815d4f0 msgr2=0x7f4b7806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b7815d4f0 0x7f4b7806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 -- 172.21.15.77:0/1306745431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b78072520 msgr2=0x7f4b7814fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b78072520 0x7f4b7814fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 -- 172.21.15.77:0/1306745431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7809bd10 con 0x7f4b780b5c30 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7cc86640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b780b5c30 0x7f4b7809b2e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4b680098c0 tx=0x7f4b680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b6dffb640 1 -- 172.21.15.77:0/1306745431 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68032630 con 0x7f4b780b5c30 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b6dffb640 1 -- 172.21.15.77:0/1306745431 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b680327d0 con 0x7f4b780b5c30 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b6dffb640 1 -- 172.21.15.77:0/1306745431 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68032bc0 con 0x7f4b780b5c30 2024-03-06T07:22:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.611+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7809bff0 con 0x7f4b780b5c30 2024-03-06T07:22:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.612+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7809c5b0 con 0x7f4b780b5c30 2024-03-06T07:22:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.612+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7804f330 con 0x7f4b780b5c30 2024-03-06T07:22:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.613+0000 7f4b6dffb640 1 -- 172.21.15.77:0/1306745431 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b6804e430 con 0x7f4b780b5c30 2024-03-06T07:22:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.613+0000 7f4b6dffb640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b5c06b4d0 0x7f4b5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.613+0000 7f4b6dffb640 1 -- 172.21.15.77:0/1306745431 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b6802f030 con 0x7f4b780b5c30 2024-03-06T07:22:36.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.613+0000 7f4b7d487640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b5c06b4d0 0x7f4b5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.614+0000 7f4b7d487640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b5c06b4d0 0x7f4b5c06d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009bd0 tx=0x7f4b74009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.618+0000 7f4b6dffb640 1 -- 172.21.15.77:0/1306745431 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b68040070 con 0x7f4b780b5c30 2024-03-06T07:22:36.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.802+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4b780b17d0 con 0x7f4b780b5c30 2024-03-06T07:22:36.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.802+0000 7f4b6dffb640 1 -- 172.21.15.77:0/1306745431 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4b6807a740 con 0x7f4b780b5c30 2024-03-06T07:22:36.804 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:22:36.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b5c06b4d0 msgr2=0x7f4b5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b5c06b4d0 0x7f4b5c06d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009bd0 tx=0x7f4b74009240 comp rx=0 tx=0).stop 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b780b5c30 msgr2=0x7f4b7809b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b780b5c30 0x7f4b7809b2e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4b680098c0 tx=0x7f4b680046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 shutdown_connections 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b5c06b4d0 0x7f4b5c06d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b780b5c30 0x7f4b7809b2e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4b7815d4f0 0x7f4b7806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 --2- 172.21.15.77:0/1306745431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4b78072520 0x7f4b7814fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 >> 172.21.15.77:0/1306745431 conn(0x7f4b780b4e60 msgr2=0x7f4b780ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 shutdown_connections 2024-03-06T07:22:36.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:36.804+0000 7f4b7f6eb640 1 -- 172.21.15.77:0/1306745431 wait complete. 2024-03-06T07:22:36.817 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi158.front.sepia.ceph.com... 2024-03-06T07:22:36.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:36.817 DEBUG:teuthology.orchestra.run.smithi158:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:36.846 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:36.846 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:22:36.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:36.846 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:22:36.911 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:22:36.911 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:36.927 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:22:36.927 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:36.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:37.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.121+0000 7fabd584e640 1 Processor -- start 2024-03-06T07:22:37.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.121+0000 7fabd584e640 1 -- start start 2024-03-06T07:22:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabd584e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fabd006b600 0x7fabd015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabd584e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd015ea30 con 0x7fabd00ce920 2024-03-06T07:22:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabd584e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd015ed10 con 0x7fabd006b220 2024-03-06T07:22:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabd584e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd0161b70 con 0x7fabd006b600 2024-03-06T07:22:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabcf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fabd006b600 0x7fabd015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabce7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fabd00ce920 0x7fabd0162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52550/0 (socket says 172.21.15.77:52550) 2024-03-06T07:22:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabce7fc640 1 -- 172.21.15.77:0/3219170790 learned_addr learned my addr 172.21.15.77:0/3219170790 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabcf7fe640 1 -- 172.21.15.77:0/3219170790 >> v1:172.21.15.158:6789/0 conn(0x7fabd006b220 legacy=0x7fabd0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabcf7fe640 1 -- 172.21.15.77:0/3219170790 >> v1:172.21.15.77:6789/0 conn(0x7fabd00ce920 legacy=0x7fabd0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.122+0000 7fabcf7fe640 1 -- 172.21.15.77:0/3219170790 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd015eff0 con 0x7fabd006b600 2024-03-06T07:22:37.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.123+0000 7fabcf7fe640 1 --2- 172.21.15.77:0/3219170790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd015e450 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fabd0161cf0 tx=0x7fabc802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9530b4be0c93dd7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.123+0000 7fabcdffb640 1 -- 172.21.15.77:0/3219170790 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc8034070 con 0x7fabd006b600 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.123+0000 7fabcdffb640 1 -- 172.21.15.77:0/3219170790 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc8030610 con 0x7fabd006b600 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.123+0000 7fabcdffb640 1 -- 172.21.15.77:0/3219170790 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc80309e0 con 0x7fabd006b600 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.124+0000 7fabd584e640 1 -- 172.21.15.77:0/3219170790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 msgr2=0x7fabd015e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.124+0000 7fabd584e640 1 --2- 172.21.15.77:0/3219170790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd015e450 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fabd0161cf0 tx=0x7fabc802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.124+0000 7fabd584e640 1 -- 172.21.15.77:0/3219170790 shutdown_connections 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.124+0000 7fabd584e640 1 --2- 172.21.15.77:0/3219170790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd015e450 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.124+0000 7fabd584e640 1 -- 172.21.15.77:0/3219170790 >> 172.21.15.77:0/3219170790 conn(0x7fabd006c0c0 msgr2=0x7fabd006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.124+0000 7fabd584e640 1 -- 172.21.15.77:0/3219170790 shutdown_connections 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.124+0000 7fabd584e640 1 -- 172.21.15.77:0/3219170790 wait complete. 2024-03-06T07:22:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.125+0000 7fabd584e640 1 Processor -- start 2024-03-06T07:22:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.125+0000 7fabd584e640 1 -- start start 2024-03-06T07:22:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabd584e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabd006b220 0x7fabd01508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabd584e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd0159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabd584e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd00ce920 0x7fabd015b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabd584e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0072090 con 0x7fabd006b600 2024-03-06T07:22:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabd584e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00ce200 con 0x7fabd00ce920 2024-03-06T07:22:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabd584e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0156740 con 0x7fabd006b220 2024-03-06T07:22:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd0159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd0159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43206/0 (socket says 172.21.15.77:43206) 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 -- 172.21.15.77:0/4201269066 learned_addr learned my addr 172.21.15.77:0/4201269066 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabceffd640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabd006b220 0x7fabd01508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabcf7fe640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd00ce920 0x7fabd015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 -- 172.21.15.77:0/4201269066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabd006b220 msgr2=0x7fabd01508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabd006b220 0x7fabd01508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 -- 172.21.15.77:0/4201269066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd00ce920 msgr2=0x7fabd015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd00ce920 0x7fabd015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 -- 172.21.15.77:0/4201269066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd0155b30 con 0x7fabd006b600 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabce7fc640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd0159690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fabc400c4a0 tx=0x7fabc400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabb7fff640 1 -- 172.21.15.77:0/4201269066 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc4017070 con 0x7fabd006b600 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabb7fff640 1 -- 172.21.15.77:0/4201269066 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc4005ce0 con 0x7fabd006b600 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd0155e10 con 0x7fabd006b600 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.126+0000 7fabb7fff640 1 -- 172.21.15.77:0/4201269066 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc4007270 con 0x7fabd006b600 2024-03-06T07:22:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.127+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd01563d0 con 0x7fabd006b600 2024-03-06T07:22:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.128+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd0156bb0 con 0x7fabd006b600 2024-03-06T07:22:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.128+0000 7fabb7fff640 1 -- 172.21.15.77:0/4201269066 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabc401a070 con 0x7fabd006b600 2024-03-06T07:22:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.128+0000 7fabb7fff640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabbc06b500 0x7fabbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.128+0000 7fabceffd640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabbc06b500 0x7fabbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.128+0000 7fabb7fff640 1 -- 172.21.15.77:0/4201269066 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabc40893d0 con 0x7fabd006b600 2024-03-06T07:22:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.129+0000 7fabceffd640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabbc06b500 0x7fabbc06d9c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fabc0001e70 tx=0x7fabc0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.132+0000 7fabb7fff640 1 -- 172.21.15.77:0/4201269066 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabd0156bb0 con 0x7fabd006b600 2024-03-06T07:22:37.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.307+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabd004f330 con 0x7fabd006b600 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.308+0000 7fabb7fff640 1 -- 172.21.15.77:0/4201269066 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fabc4057c00 con 0x7fabd006b600 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:37.310 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:37.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabbc06b500 msgr2=0x7fabbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabbc06b500 0x7fabbc06d9c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fabc0001e70 tx=0x7fabc0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 msgr2=0x7fabd0159690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd0159690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fabc400c4a0 tx=0x7fabc400c960 comp rx=0 tx=0).stop 2024-03-06T07:22:37.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 shutdown_connections 2024-03-06T07:22:37.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabbc06b500 0x7fabbc06d9c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabd006b600 0x7fabd0159690 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabd00ce920 0x7fabd015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 --2- 172.21.15.77:0/4201269066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabd006b220 0x7fabd01508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 >> 172.21.15.77:0/4201269066 conn(0x7fabd006c0c0 msgr2=0x7fabd00b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 shutdown_connections 2024-03-06T07:22:37.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.310+0000 7fabd584e640 1 -- 172.21.15.77:0/4201269066 wait complete. 2024-03-06T07:22:37.323 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:37.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.425+0000 7efc1c20e640 1 Processor -- start 2024-03-06T07:22:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.425+0000 7efc1c20e640 1 -- start start 2024-03-06T07:22:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc1c20e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efc140d4860 0x7efc140d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc1c20e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc140b0e30 0x7efc140b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc1c20e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc141618b0 con 0x7efc141614d0 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc1c20e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc140d4cf0 con 0x7efc140d4860 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc1c20e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc140cf400 con 0x7efc140b0e30 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc197a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efc140d4860 0x7efc140d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc197a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efc140d4860 0x7efc140d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43222/0 (socket says 172.21.15.77:43222) 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc19faa640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7efc141614d0 0x7efc140d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:47344/0 (socket says 172.21.15.77:47344) 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc197a9640 1 -- 172.21.15.77:0/1777490624 learned_addr learned my addr 172.21.15.77:0/1777490624 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.426+0000 7efc1a7ab640 1 --2- 172.21.15.77:0/1777490624 >> v2:172.21.15.158:3300/0 conn(0x7efc140b0e30 0x7efc140b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.427+0000 7efc197a9640 1 -- 172.21.15.77:0/1777490624 >> v2:172.21.15.158:3300/0 conn(0x7efc140b0e30 msgr2=0x7efc140b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.427+0000 7efc197a9640 1 --2- 172.21.15.77:0/1777490624 >> v2:172.21.15.158:3300/0 conn(0x7efc140b0e30 0x7efc140b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.427+0000 7efc197a9640 1 -- 172.21.15.77:0/1777490624 >> v1:172.21.15.158:6789/0 conn(0x7efc141614d0 legacy=0x7efc140d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.427+0000 7efc18fa8640 1 -- 172.21.15.77:0/1777490624 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795755708 0 0) 0x7efc141618b0 con 0x7efc141614d0 2024-03-06T07:22:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.427+0000 7efc197a9640 1 -- 172.21.15.77:0/1777490624 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc140cf6a0 con 0x7efc140d4860 2024-03-06T07:22:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.427+0000 7efc197a9640 1 --2- 172.21.15.77:0/1777490624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140d4860 0x7efc140d2020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efc140b2a70 tx=0x7efc0402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ca961f9f62215c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc18fa8640 1 -- 172.21.15.77:0/1777490624 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc04034070 con 0x7efc140d4860 2024-03-06T07:22:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc18fa8640 1 -- 172.21.15.77:0/1777490624 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc04030860 con 0x7efc140d4860 2024-03-06T07:22:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc18fa8640 1 -- 172.21.15.77:0/1777490624 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc04030c30 con 0x7efc140d4860 2024-03-06T07:22:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc1c20e640 1 -- 172.21.15.77:0/1777490624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140d4860 msgr2=0x7efc140d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc1c20e640 1 --2- 172.21.15.77:0/1777490624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140d4860 0x7efc140d2020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efc140b2a70 tx=0x7efc0402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc1c20e640 1 -- 172.21.15.77:0/1777490624 shutdown_connections 2024-03-06T07:22:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc1c20e640 1 --2- 172.21.15.77:0/1777490624 >> v2:172.21.15.158:3300/0 conn(0x7efc140b0e30 0x7efc140b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc1c20e640 1 --2- 172.21.15.77:0/1777490624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140d4860 0x7efc140d2020 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc1c20e640 1 -- 172.21.15.77:0/1777490624 >> 172.21.15.77:0/1777490624 conn(0x7efc140d3d40 msgr2=0x7efc140d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc1c20e640 1 -- 172.21.15.77:0/1777490624 shutdown_connections 2024-03-06T07:22:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.428+0000 7efc1c20e640 1 -- 172.21.15.77:0/1777490624 wait complete. 2024-03-06T07:22:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.429+0000 7efc1c20e640 1 Processor -- start 2024-03-06T07:22:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.429+0000 7efc1c20e640 1 -- start start 2024-03-06T07:22:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc1c20e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140b0e30 0x7efc140b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc1c20e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc140d4860 0x7efc1415dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc1c20e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc141614d0 0x7efc1415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc1c20e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc140b0910 con 0x7efc140b0e30 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc1c20e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc140cfc60 con 0x7efc140d4860 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc1c20e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc140d2560 con 0x7efc141614d0 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc19faa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140b0e30 0x7efc140b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc19faa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140b0e30 0x7efc140b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43238/0 (socket says 172.21.15.77:43238) 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc19faa640 1 -- 172.21.15.77:0/670209958 learned_addr learned my addr 172.21.15.77:0/670209958 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc197a9640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc140d4860 0x7efc1415dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc1a7ab640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc141614d0 0x7efc1415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc19faa640 1 -- 172.21.15.77:0/670209958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc141614d0 msgr2=0x7efc1415b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc19faa640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc141614d0 0x7efc1415b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc19faa640 1 -- 172.21.15.77:0/670209958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc140d4860 msgr2=0x7efc1415dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc19faa640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc140d4860 0x7efc1415dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.430+0000 7efc19faa640 1 -- 172.21.15.77:0/670209958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc140b5dd0 con 0x7efc140b0e30 2024-03-06T07:22:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.431+0000 7efc19faa640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140b0e30 0x7efc140b54d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efc1000c780 tx=0x7efc1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.431+0000 7efc0a7fc640 1 -- 172.21.15.77:0/670209958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10017070 con 0x7efc140b0e30 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.431+0000 7efc0a7fc640 1 -- 172.21.15.77:0/670209958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1000a640 con 0x7efc140b0e30 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.431+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc140b60b0 con 0x7efc140b0e30 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.431+0000 7efc0a7fc640 1 -- 172.21.15.77:0/670209958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1000aa30 con 0x7efc140b0e30 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.431+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc140cafb0 con 0x7efc140b0e30 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.432+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc14068ff0 con 0x7efc140b0e30 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.432+0000 7efc0a7fc640 1 -- 172.21.15.77:0/670209958 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc1001a070 con 0x7efc140b0e30 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.433+0000 7efc0a7fc640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf806b4d0 0x7efbf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.433+0000 7efc0a7fc640 1 -- 172.21.15.77:0/670209958 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc10089780 con 0x7efc140b0e30 2024-03-06T07:22:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.433+0000 7efc197a9640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf806b4d0 0x7efbf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.433+0000 7efc197a9640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf806b4d0 0x7efbf806d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efc04036000 tx=0x7efc04008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.437+0000 7efc0a7fc640 1 -- 172.21.15.77:0/670209958 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc10058040 con 0x7efc140b0e30 2024-03-06T07:22:37.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.606+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc1415f720 con 0x7efc140b0e30 2024-03-06T07:22:37.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.606+0000 7efc0a7fc640 1 -- 172.21.15.77:0/670209958 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc1005c0e0 con 0x7efc140b0e30 2024-03-06T07:22:37.608 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:37.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf806b4d0 msgr2=0x7efbf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf806b4d0 0x7efbf806d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efc04036000 tx=0x7efc04008030 comp rx=0 tx=0).stop 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140b0e30 msgr2=0x7efc140b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140b0e30 0x7efc140b54d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efc1000c780 tx=0x7efc1000cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 shutdown_connections 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc141614d0 0x7efc1415b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc140d4860 0x7efc1415dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf806b4d0 0x7efbf806d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 --2- 172.21.15.77:0/670209958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc140b0e30 0x7efc140b54d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.608+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 >> 172.21.15.77:0/670209958 conn(0x7efc140d3d40 msgr2=0x7efc1406c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.609+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 shutdown_connections 2024-03-06T07:22:37.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.609+0000 7efc1c20e640 1 -- 172.21.15.77:0/670209958 wait complete. 2024-03-06T07:22:37.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:37.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.723+0000 7fca35393640 1 Processor -- start 2024-03-06T07:22:37.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.723+0000 7fca35393640 1 -- start start 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.723+0000 7fca35393640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fca300cea50 0x7fca300b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca35393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 0x7fca300ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca35393640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3006d030 con 0x7fca3015def0 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca35393640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca300cf1c0 con 0x7fca300cea50 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca35393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca300cc760 con 0x7fca3006cab0 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca27fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fca300cea50 0x7fca300b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca27fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fca300cea50 0x7fca300b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43252/0 (socket says 172.21.15.77:43252) 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca27fff640 1 -- 172.21.15.77:0/1624703058 learned_addr learned my addr 172.21.15.77:0/1624703058 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:37.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca2f7fe640 1 --2- 172.21.15.77:0/1624703058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 0x7fca300ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca27fff640 1 -- 172.21.15.77:0/1624703058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 msgr2=0x7fca300ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca27fff640 1 --2- 172.21.15.77:0/1624703058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 0x7fca300ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca27fff640 1 -- 172.21.15.77:0/1624703058 >> v1:172.21.15.158:6789/0 conn(0x7fca3015def0 legacy=0x7fca300ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.724+0000 7fca27fff640 1 -- 172.21.15.77:0/1624703058 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3006d310 con 0x7fca300cea50 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca27fff640 1 --2- 172.21.15.77:0/1624703058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300b41b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fca18009870 tx=0x7fca1802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8692b205f82f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca2e7fc640 1 -- 172.21.15.77:0/1624703058 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1802f840 con 0x7fca300cea50 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca2e7fc640 1 -- 172.21.15.77:0/1624703058 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1802f9e0 con 0x7fca300cea50 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca2e7fc640 1 -- 172.21.15.77:0/1624703058 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1802fdb0 con 0x7fca300cea50 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca35393640 1 -- 172.21.15.77:0/1624703058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 msgr2=0x7fca300b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca35393640 1 --2- 172.21.15.77:0/1624703058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300b41b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fca18009870 tx=0x7fca1802ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca35393640 1 -- 172.21.15.77:0/1624703058 shutdown_connections 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca35393640 1 --2- 172.21.15.77:0/1624703058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 0x7fca300ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca35393640 1 --2- 172.21.15.77:0/1624703058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300b41b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.725+0000 7fca35393640 1 -- 172.21.15.77:0/1624703058 >> 172.21.15.77:0/1624703058 conn(0x7fca300b3690 msgr2=0x7fca300b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.726+0000 7fca35393640 1 -- 172.21.15.77:0/1624703058 shutdown_connections 2024-03-06T07:22:37.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.726+0000 7fca35393640 1 -- 172.21.15.77:0/1624703058 wait complete. 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca35393640 1 Processor -- start 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca35393640 1 -- start start 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca35393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 0x7fca3015bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca35393640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca35393640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca3015def0 0x7fca300b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca35393640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3015e2d0 con 0x7fca300cea50 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca35393640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca300cc3f0 con 0x7fca3015def0 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca35393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3015e450 con 0x7fca3006cab0 2024-03-06T07:22:37.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca27fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca27fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43268/0 (socket says 172.21.15.77:43268) 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca27fff640 1 -- 172.21.15.77:0/3840724185 learned_addr learned my addr 172.21.15.77:0/3840724185 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca2f7fe640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca3015def0 0x7fca300b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.727+0000 7fca2effd640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 0x7fca3015bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca27fff640 1 -- 172.21.15.77:0/3840724185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 msgr2=0x7fca3015bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca27fff640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 0x7fca3015bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca27fff640 1 -- 172.21.15.77:0/3840724185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca3015def0 msgr2=0x7fca300b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca27fff640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca3015def0 0x7fca300b6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca27fff640 1 -- 172.21.15.77:0/3840724185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca300b6ee0 con 0x7fca300cea50 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca27fff640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300c61a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fca18000c00 tx=0x7fca18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca277fe640 1 -- 172.21.15.77:0/3840724185 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18037bc0 con 0x7fca300cea50 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca277fe640 1 -- 172.21.15.77:0/3840724185 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca18046070 con 0x7fca300cea50 2024-03-06T07:22:37.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca300c56c0 con 0x7fca300cea50 2024-03-06T07:22:37.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca277fe640 1 -- 172.21.15.77:0/3840724185 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18041400 con 0x7fca300cea50 2024-03-06T07:22:37.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.728+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca300c5c80 con 0x7fca300cea50 2024-03-06T07:22:37.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.729+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3009fcf0 con 0x7fca300cea50 2024-03-06T07:22:37.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.729+0000 7fca277fe640 1 -- 172.21.15.77:0/3840724185 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca18040070 con 0x7fca300cea50 2024-03-06T07:22:37.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.730+0000 7fca277fe640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.730+0000 7fca277fe640 1 -- 172.21.15.77:0/3840724185 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca180ac6c0 con 0x7fca300cea50 2024-03-06T07:22:37.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.730+0000 7fca2effd640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.730+0000 7fca2effd640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca1006b4d0 0x7fca1006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fca200016e0 tx=0x7fca20009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.734+0000 7fca277fe640 1 -- 172.21.15.77:0/3840724185 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca18033030 con 0x7fca300cea50 2024-03-06T07:22:37.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.905+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca30068ff0 con 0x7fca300cea50 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.906+0000 7fca277fe640 1 -- 172.21.15.77:0/3840724185 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fca1807af80 con 0x7fca300cea50 2024-03-06T07:22:37.908 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.907+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca1006b4d0 msgr2=0x7fca1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.907+0000 7fca35393640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca1006b4d0 0x7fca1006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fca200016e0 tx=0x7fca20009040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.907+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 msgr2=0x7fca300c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.907+0000 7fca35393640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300c61a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fca18000c00 tx=0x7fca18004730 comp rx=0 tx=0).stop 2024-03-06T07:22:37.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.908+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 shutdown_connections 2024-03-06T07:22:37.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.908+0000 7fca35393640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.908+0000 7fca35393640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca300cea50 0x7fca300c61a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.908+0000 7fca35393640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca3015def0 0x7fca300b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.908+0000 7fca35393640 1 --2- 172.21.15.77:0/3840724185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca3006cab0 0x7fca3015bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.908+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 >> 172.21.15.77:0/3840724185 conn(0x7fca300b3690 msgr2=0x7fca300d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.908+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 shutdown_connections 2024-03-06T07:22:37.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:37.908+0000 7fca35393640 1 -- 172.21.15.77:0/3840724185 wait complete. 2024-03-06T07:22:37.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:38.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.022+0000 7f5799adf640 1 Processor -- start 2024-03-06T07:22:38.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.022+0000 7f5799adf640 1 -- start start 2024-03-06T07:22:38.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f5799adf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f57940d4860 0x7f57940d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f5799adf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57940b0e30 0x7f57940b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f5799adf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57941618b0 con 0x7f57941614d0 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f5799adf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57940d4cf0 con 0x7f57940b0e30 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f5799adf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57940cf400 con 0x7f57940d4860 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f5793fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57940b0e30 0x7f57940b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f57937fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f57941614d0 0x7f57940d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52564/0 (socket says 172.21.15.77:52564) 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f57937fe640 1 -- 172.21.15.77:0/3167027434 learned_addr learned my addr 172.21.15.77:0/3167027434 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.023+0000 7f578affd640 1 --2- 172.21.15.77:0/3167027434 >> v2:172.21.15.158:3300/0 conn(0x7f57940d4860 0x7f57940d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5793fff640 1 -- 172.21.15.77:0/3167027434 >> v2:172.21.15.158:3300/0 conn(0x7f57940d4860 msgr2=0x7f57940d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5793fff640 1 --2- 172.21.15.77:0/3167027434 >> v2:172.21.15.158:3300/0 conn(0x7f57940d4860 0x7f57940d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5793fff640 1 -- 172.21.15.77:0/3167027434 >> v1:172.21.15.77:6789/0 conn(0x7f57941614d0 legacy=0x7f57940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5793fff640 1 -- 172.21.15.77:0/3167027434 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57940cf6a0 con 0x7f57940b0e30 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5792ffd640 1 -- 172.21.15.77:0/3167027434 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909894634 0 0) 0x7f57941618b0 con 0x7f57941614d0 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5793fff640 1 --2- 172.21.15.77:0/3167027434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57940b0e30 0x7f57940b1260 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f57940ce550 tx=0x7f578c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926a2f2028d160fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5792ffd640 1 -- 172.21.15.77:0/3167027434 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c039070 con 0x7f57940b0e30 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5792ffd640 1 -- 172.21.15.77:0/3167027434 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c02cc80 con 0x7f57940b0e30 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.024+0000 7f5792ffd640 1 -- 172.21.15.77:0/3167027434 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c034660 con 0x7f57940b0e30 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.025+0000 7f5799adf640 1 -- 172.21.15.77:0/3167027434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57940b0e30 msgr2=0x7f57940b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.025+0000 7f5799adf640 1 --2- 172.21.15.77:0/3167027434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57940b0e30 0x7f57940b1260 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f57940ce550 tx=0x7f578c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.025+0000 7f5799adf640 1 -- 172.21.15.77:0/3167027434 shutdown_connections 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.025+0000 7f5799adf640 1 --2- 172.21.15.77:0/3167027434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57940b0e30 0x7f57940b1260 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.025+0000 7f5799adf640 1 --2- 172.21.15.77:0/3167027434 >> v2:172.21.15.158:3300/0 conn(0x7f57940d4860 0x7f57940d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.025+0000 7f5799adf640 1 -- 172.21.15.77:0/3167027434 >> 172.21.15.77:0/3167027434 conn(0x7f57940d3d40 msgr2=0x7f57940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.025+0000 7f5799adf640 1 -- 172.21.15.77:0/3167027434 shutdown_connections 2024-03-06T07:22:38.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.025+0000 7f5799adf640 1 -- 172.21.15.77:0/3167027434 wait complete. 2024-03-06T07:22:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.026+0000 7f5799adf640 1 Processor -- start 2024-03-06T07:22:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.026+0000 7f5799adf640 1 -- start start 2024-03-06T07:22:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5799adf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57940b0e30 0x7f57941f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5799adf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f57940d4860 0x7f579415a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5799adf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57941614d0 0x7f57940cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5799adf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940b0910 con 0x7f57941614d0 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5799adf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940cfc60 con 0x7f57940b0e30 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5799adf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57940d2560 con 0x7f57940d4860 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5793fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57941614d0 0x7f57940cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5793fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57941614d0 0x7f57940cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43292/0 (socket says 172.21.15.77:43292) 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f5793fff640 1 -- 172.21.15.77:0/1290241387 learned_addr learned my addr 172.21.15.77:0/1290241387 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f57937fe640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57940b0e30 0x7f57941f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.027+0000 7f578affd640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f57940d4860 0x7f579415a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5793fff640 1 -- 172.21.15.77:0/1290241387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f57940d4860 msgr2=0x7f579415a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5793fff640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f57940d4860 0x7f579415a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5793fff640 1 -- 172.21.15.77:0/1290241387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57940b0e30 msgr2=0x7f57941f7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5793fff640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57940b0e30 0x7f57941f7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5793fff640 1 -- 172.21.15.77:0/1290241387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57941f81a0 con 0x7f57941614d0 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5793fff640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57941614d0 0x7f57940cc250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f578c009e80 tx=0x7f578c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5790ff9640 1 -- 172.21.15.77:0/1290241387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c039050 con 0x7f57941614d0 2024-03-06T07:22:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5790ff9640 1 -- 172.21.15.77:0/1290241387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c045070 con 0x7f57941614d0 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5790ff9640 1 -- 172.21.15.77:0/1290241387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c009450 con 0x7f57941614d0 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57940cb1b0 con 0x7f57941614d0 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.028+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57940cb770 con 0x7f57941614d0 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.029+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f579406c070 con 0x7f57941614d0 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.029+0000 7f5790ff9640 1 -- 172.21.15.77:0/1290241387 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f578c03e850 con 0x7f57941614d0 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.030+0000 7f5790ff9640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f577406b500 0x7f577406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.030+0000 7f5790ff9640 1 -- 172.21.15.77:0/1290241387 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f578c0ad180 con 0x7f57941614d0 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.030+0000 7f57937fe640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f577406b500 0x7f577406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.030+0000 7f57937fe640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f577406b500 0x7f577406d9c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5784000f60 tx=0x7f5784009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.034+0000 7f5790ff9640 1 -- 172.21.15.77:0/1290241387 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f579406c070 con 0x7f57941614d0 2024-03-06T07:22:38.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.208+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5794069e80 con 0x7f57941614d0 2024-03-06T07:22:38.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.208+0000 7f5790ff9640 1 -- 172.21.15.77:0/1290241387 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5794069e80 con 0x7f57941614d0 2024-03-06T07:22:38.210 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.210+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f577406b500 msgr2=0x7f577406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.210+0000 7f5799adf640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f577406b500 0x7f577406d9c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5784000f60 tx=0x7f5784009040 comp rx=0 tx=0).stop 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.210+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57941614d0 msgr2=0x7f57940cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.210+0000 7f5799adf640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57941614d0 0x7f57940cc250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f578c009e80 tx=0x7f578c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.211+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 shutdown_connections 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.211+0000 7f5799adf640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f577406b500 0x7f577406d9c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.211+0000 7f5799adf640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57941614d0 0x7f57940cc250 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.211+0000 7f5799adf640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f57940d4860 0x7f579415a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.211+0000 7f5799adf640 1 --2- 172.21.15.77:0/1290241387 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f57940b0e30 0x7f57941f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.211+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 >> 172.21.15.77:0/1290241387 conn(0x7f57940d3d40 msgr2=0x7f579406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.211+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 shutdown_connections 2024-03-06T07:22:38.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.211+0000 7f5799adf640 1 -- 172.21.15.77:0/1290241387 wait complete. 2024-03-06T07:22:38.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:38.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.324+0000 7fa68ecc9640 1 Processor -- start 2024-03-06T07:22:38.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.325+0000 7fa68ecc9640 1 -- start start 2024-03-06T07:22:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ecc9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa6881614a0 0x7fa68806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ecc9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa6880b1a20 0x7fa6880b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ecc9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880af4b0 con 0x7fa6880b0270 2024-03-06T07:22:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ecc9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6880af790 con 0x7fa6881614a0 2024-03-06T07:22:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ecc9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6880afa70 con 0x7fa6880b1a20 2024-03-06T07:22:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ca65640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa6881614a0 0x7fa68806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ca65640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa6881614a0 0x7fa68806d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43294/0 (socket says 172.21.15.77:43294) 2024-03-06T07:22:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ca65640 1 -- 172.21.15.77:0/227247714 learned_addr learned my addr 172.21.15.77:0/227247714 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68d266640 1 --2- 172.21.15.77:0/227247714 >> v2:172.21.15.158:3300/0 conn(0x7fa6880b1a20 0x7fa6880b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ca65640 1 -- 172.21.15.77:0/227247714 >> v2:172.21.15.158:3300/0 conn(0x7fa6880b1a20 msgr2=0x7fa6880b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ca65640 1 --2- 172.21.15.77:0/227247714 >> v2:172.21.15.158:3300/0 conn(0x7fa6880b1a20 0x7fa6880b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ca65640 1 -- 172.21.15.77:0/227247714 >> v1:172.21.15.158:6789/0 conn(0x7fa6880b0270 legacy=0x7fa6880b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.326+0000 7fa68ca65640 1 -- 172.21.15.77:0/227247714 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6880afd50 con 0x7fa6881614a0 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.327+0000 7fa68ca65640 1 --2- 172.21.15.77:0/227247714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6881614a0 0x7fa68806d3c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6880b34f0 tx=0x7fa67c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b687d94c4a5c08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.327+0000 7fa6877fe640 1 -- 172.21.15.77:0/227247714 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c035020 con 0x7fa6881614a0 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.327+0000 7fa6877fe640 1 -- 172.21.15.77:0/227247714 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c031870 con 0x7fa6881614a0 2024-03-06T07:22:38.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.327+0000 7fa6877fe640 1 -- 172.21.15.77:0/227247714 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c031c40 con 0x7fa6881614a0 2024-03-06T07:22:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.328+0000 7fa68ecc9640 1 -- 172.21.15.77:0/227247714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6881614a0 msgr2=0x7fa68806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.328+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/227247714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6881614a0 0x7fa68806d3c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6880b34f0 tx=0x7fa67c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.328+0000 7fa68ecc9640 1 -- 172.21.15.77:0/227247714 shutdown_connections 2024-03-06T07:22:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.328+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/227247714 >> v2:172.21.15.158:3300/0 conn(0x7fa6880b1a20 0x7fa6880b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.328+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/227247714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6881614a0 0x7fa68806d3c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.328+0000 7fa68ecc9640 1 -- 172.21.15.77:0/227247714 >> 172.21.15.77:0/227247714 conn(0x7fa68809a9f0 msgr2=0x7fa68809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.328+0000 7fa68ecc9640 1 -- 172.21.15.77:0/227247714 shutdown_connections 2024-03-06T07:22:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.328+0000 7fa68ecc9640 1 -- 172.21.15.77:0/227247714 wait complete. 2024-03-06T07:22:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.330+0000 7fa68ecc9640 1 Processor -- start 2024-03-06T07:22:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.330+0000 7fa68ecc9640 1 -- start start 2024-03-06T07:22:38.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa68ecc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6880b0270 0x7fa68809f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa68ecc9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6880b1a20 0x7fa6881504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa68ecc9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6881614a0 0x7fa6881537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa68ecc9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6880b1e00 con 0x7fa6880b1a20 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa68ecc9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6880b06e0 con 0x7fa6881614a0 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa68ecc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6880b1f80 con 0x7fa6880b0270 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa687fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6880b1a20 0x7fa6881504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa687fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6880b1a20 0x7fa6881504b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43296/0 (socket says 172.21.15.77:43296) 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa687fff640 1 -- 172.21.15.77:0/713363090 learned_addr learned my addr 172.21.15.77:0/713363090 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa68d266640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6881614a0 0x7fa6881537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.331+0000 7fa68ca65640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6880b0270 0x7fa68809f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa687fff640 1 -- 172.21.15.77:0/713363090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6880b0270 msgr2=0x7fa68809f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa687fff640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6880b0270 0x7fa68809f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa687fff640 1 -- 172.21.15.77:0/713363090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6881614a0 msgr2=0x7fa6881537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa687fff640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6881614a0 0x7fa6881537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa687fff640 1 -- 172.21.15.77:0/713363090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa688151bb0 con 0x7fa6880b1a20 2024-03-06T07:22:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa687fff640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6880b1a20 0x7fa6881504b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa67400c3f0 tx=0x7fa67400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa6857fa640 1 -- 172.21.15.77:0/713363090 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6740052d0 con 0x7fa6880b1a20 2024-03-06T07:22:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa6857fa640 1 -- 172.21.15.77:0/713363090 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa674005490 con 0x7fa6880b1a20 2024-03-06T07:22:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa688151e50 con 0x7fa6880b1a20 2024-03-06T07:22:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa6857fa640 1 -- 172.21.15.77:0/713363090 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa674007260 con 0x7fa6880b1a20 2024-03-06T07:22:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.332+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6880cbc10 con 0x7fa6880b1a20 2024-03-06T07:22:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.333+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68804f330 con 0x7fa6880b1a20 2024-03-06T07:22:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.333+0000 7fa6857fa640 1 -- 172.21.15.77:0/713363090 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa674007400 con 0x7fa6880b1a20 2024-03-06T07:22:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.334+0000 7fa6857fa640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa66c06b4d0 0x7fa66c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.334+0000 7fa68ca65640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa66c06b4d0 0x7fa66c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.334+0000 7fa6857fa640 1 -- 172.21.15.77:0/713363090 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa674089630 con 0x7fa6880b1a20 2024-03-06T07:22:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.334+0000 7fa68ca65640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa66c06b4d0 0x7fa66c06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa67c036000 tx=0x7fa67c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.338+0000 7fa6857fa640 1 -- 172.21.15.77:0/713363090 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa674057e60 con 0x7fa6880b1a20 2024-03-06T07:22:38.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.530+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa6880ce1e0 con 0x7fa6880b1a20 2024-03-06T07:22:38.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.530+0000 7fa6857fa640 1 -- 172.21.15.77:0/713363090 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa67405ce20 con 0x7fa6880b1a20 2024-03-06T07:22:38.532 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:22:38.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.532+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa66c06b4d0 msgr2=0x7fa66c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.532+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa66c06b4d0 0x7fa66c06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa67c036000 tx=0x7fa67c03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.532+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6880b1a20 msgr2=0x7fa6881504b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.532+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6880b1a20 0x7fa6881504b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa67400c3f0 tx=0x7fa67400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.532+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 shutdown_connections 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.532+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa66c06b4d0 0x7fa66c06d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.533+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6880b1a20 0x7fa6881504b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.533+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6881614a0 0x7fa6881537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.533+0000 7fa68ecc9640 1 --2- 172.21.15.77:0/713363090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6880b0270 0x7fa68809f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.533+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 >> 172.21.15.77:0/713363090 conn(0x7fa68809a9f0 msgr2=0x7fa68806f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.533+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 shutdown_connections 2024-03-06T07:22:38.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.533+0000 7fa68ecc9640 1 -- 172.21.15.77:0/713363090 wait complete. 2024-03-06T07:22:38.546 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:22:38.546 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:38.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:38.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.702+0000 7f520caeb640 1 Processor -- start 2024-03-06T07:22:38.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.702+0000 7f520caeb640 1 -- start start 2024-03-06T07:22:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f520caeb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f52080cfff0 0x7f52080d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f520caeb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52080cc8b0 0x7f52080ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f520caeb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5208161980 con 0x7f520815bb20 2024-03-06T07:22:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f520caeb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52080d2500 con 0x7f52080cfff0 2024-03-06T07:22:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f520caeb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5208161c60 con 0x7f52080cc8b0 2024-03-06T07:22:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5205d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f52080cfff0 0x7f52080d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5205d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f52080cfff0 0x7f52080d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56572/0 (socket says 172.21.15.77:56572) 2024-03-06T07:22:38.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5205d9b640 1 -- 172.21.15.77:0/1063118637 learned_addr learned my addr 172.21.15.77:0/1063118637 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5206d9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52080cc8b0 0x7f52080ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f520659c640 1 --1- 172.21.15.77:0/1063118637 >> v1:172.21.15.158:6789/0 conn(0x7f520815bb20 0x7f52080d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:41792/0 (socket says 172.21.15.77:41792) 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5205d9b640 1 -- 172.21.15.77:0/1063118637 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52080cc8b0 msgr2=0x7f52080ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f520559a640 1 -- 172.21.15.77:0/1063118637 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978096621 0 0) 0x7f5208161980 con 0x7f520815bb20 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5205d9b640 1 --2- 172.21.15.77:0/1063118637 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52080cc8b0 0x7f52080ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5205d9b640 1 -- 172.21.15.77:0/1063118637 >> v1:172.21.15.158:6789/0 conn(0x7f520815bb20 legacy=0x7f52080d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5205d9b640 1 -- 172.21.15.77:0/1063118637 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5208161f40 con 0x7f52080cfff0 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.703+0000 7f5206d9d640 1 --2- 172.21.15.77:0/1063118637 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52080cc8b0 0x7f52080ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.704+0000 7f5205d9b640 1 --2- 172.21.15.77:0/1063118637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cfff0 0x7f52080d04a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009870 tx=0x7f51fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f8651d218dd72d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.704+0000 7f520559a640 1 -- 172.21.15.77:0/1063118637 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc034070 con 0x7f52080cfff0 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.704+0000 7f520559a640 1 -- 172.21.15.77:0/1063118637 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc030840 con 0x7f52080cfff0 2024-03-06T07:22:38.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.704+0000 7f520559a640 1 -- 172.21.15.77:0/1063118637 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc030c10 con 0x7f52080cfff0 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.705+0000 7f520caeb640 1 -- 172.21.15.77:0/1063118637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cfff0 msgr2=0x7f52080d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.705+0000 7f520caeb640 1 --2- 172.21.15.77:0/1063118637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cfff0 0x7f52080d04a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009870 tx=0x7f51fc02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.705+0000 7f520caeb640 1 -- 172.21.15.77:0/1063118637 shutdown_connections 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.705+0000 7f520caeb640 1 --2- 172.21.15.77:0/1063118637 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52080cc8b0 0x7f52080ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.705+0000 7f520caeb640 1 --2- 172.21.15.77:0/1063118637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cfff0 0x7f52080d04a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.705+0000 7f520caeb640 1 -- 172.21.15.77:0/1063118637 >> 172.21.15.77:0/1063118637 conn(0x7f520809a9f0 msgr2=0x7f520809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.705+0000 7f520caeb640 1 -- 172.21.15.77:0/1063118637 shutdown_connections 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.705+0000 7f520caeb640 1 -- 172.21.15.77:0/1063118637 wait complete. 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.706+0000 7f520caeb640 1 Processor -- start 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.706+0000 7f520caeb640 1 -- start start 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520caeb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cc8b0 0x7f52081481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520caeb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52080cfff0 0x7f52080c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520caeb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f520815bb20 0x7f52080c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520caeb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52080d1980 con 0x7f52080cc8b0 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520caeb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5208162500 con 0x7f52080cfff0 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520caeb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52080d1b00 con 0x7f520815bb20 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cc8b0 0x7f52081481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cc8b0 0x7f52081481e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56574/0 (socket says 172.21.15.77:56574) 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 -- 172.21.15.77:0/1521057600 learned_addr learned my addr 172.21.15.77:0/1521057600 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f5205d9b640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52080cfff0 0x7f52080c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f5206d9d640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f520815bb20 0x7f52080c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 -- 172.21.15.77:0/1521057600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f520815bb20 msgr2=0x7f52080c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f520815bb20 0x7f52080c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 -- 172.21.15.77:0/1521057600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52080cfff0 msgr2=0x7f52080c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52080cfff0 0x7f52080c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 -- 172.21.15.77:0/1521057600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52081e7b50 con 0x7f52080cc8b0 2024-03-06T07:22:38.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.707+0000 7f520659c640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cc8b0 0x7f52081481e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f51f800c460 tx=0x7f51f800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.708+0000 7f51f6ffd640 1 -- 172.21.15.77:0/1521057600 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f80074e0 con 0x7f52080cc8b0 2024-03-06T07:22:38.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.708+0000 7f51f6ffd640 1 -- 172.21.15.77:0/1521057600 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f80047e0 con 0x7f52080cc8b0 2024-03-06T07:22:38.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.708+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52081e7e30 con 0x7f52080cc8b0 2024-03-06T07:22:38.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.708+0000 7f51f6ffd640 1 -- 172.21.15.77:0/1521057600 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8004bd0 con 0x7f52080cc8b0 2024-03-06T07:22:38.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.708+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52081e83f0 con 0x7f52080cc8b0 2024-03-06T07:22:38.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.709+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520804f330 con 0x7f52080cc8b0 2024-03-06T07:22:38.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.709+0000 7f51f6ffd640 1 -- 172.21.15.77:0/1521057600 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51f8011070 con 0x7f52080cc8b0 2024-03-06T07:22:38.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.710+0000 7f51f6ffd640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51e806b4d0 0x7f51e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.710+0000 7f5205d9b640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51e806b4d0 0x7f51e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.710+0000 7f51f6ffd640 1 -- 172.21.15.77:0/1521057600 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51f8089d40 con 0x7f52080cc8b0 2024-03-06T07:22:38.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.710+0000 7f5205d9b640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51e806b4d0 0x7f51e806d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f51fc036000 tx=0x7f51fc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.714+0000 7f51f6ffd640 1 -- 172.21.15.77:0/1521057600 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51f8058bc0 con 0x7f52080cc8b0 2024-03-06T07:22:38.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.889+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f520806de70 con 0x7f52080cc8b0 2024-03-06T07:22:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.890+0000 7f51f6ffd640 1 -- 172.21.15.77:0/1521057600 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f51f80584c0 con 0x7f52080cc8b0 2024-03-06T07:22:38.892 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:38.892 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:38.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51e806b4d0 msgr2=0x7f51e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51e806b4d0 0x7f51e806d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f51fc036000 tx=0x7f51fc03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:38.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cc8b0 msgr2=0x7f52081481e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cc8b0 0x7f52081481e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f51f800c460 tx=0x7f51f800c860 comp rx=0 tx=0).stop 2024-03-06T07:22:38.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 shutdown_connections 2024-03-06T07:22:38.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f520815bb20 0x7f52080c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52080cfff0 0x7f52080c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51e806b4d0 0x7f51e806d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 --2- 172.21.15.77:0/1521057600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52080cc8b0 0x7f52081481e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 >> 172.21.15.77:0/1521057600 conn(0x7f520809a9f0 msgr2=0x7f52080aefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 shutdown_connections 2024-03-06T07:22:38.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:38.892+0000 7f520caeb640 1 -- 172.21.15.77:0/1521057600 wait complete. 2024-03-06T07:22:38.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:38.906 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:39.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.010+0000 7f6e8aa4e640 1 Processor -- start 2024-03-06T07:22:39.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.010+0000 7f6e8aa4e640 1 -- start start 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.010+0000 7f6e8aa4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e841614d0 0x7f6e840d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.010+0000 7f6e8aa4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840d1910 0x7f6e840d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.010+0000 7f6e8aa4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840d2740 con 0x7f6e840d22b0 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.010+0000 7f6e8aa4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e841618b0 con 0x7f6e841614d0 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.010+0000 7f6e8aa4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840cf400 con 0x7f6e840d1910 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840d1910 0x7f6e840d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e83fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e841614d0 0x7f6e840d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e88feb640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6e840d22b0 0x7f6e840d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:41800/0 (socket says 172.21.15.77:41800) 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e83fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e841614d0 0x7f6e840d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:60220/0 (socket says 172.21.15.77:60220) 2024-03-06T07:22:39.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e88feb640 1 -- 172.21.15.77:0/3236735290 learned_addr learned my addr 172.21.15.77:0/3236735290 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840d1910 0x7f6e840d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:34840/0 (socket says 172.21.15.77:34840) 2024-03-06T07:22:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e82ffd640 1 -- 172.21.15.77:0/3236735290 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211028376 0 0) 0x7f6e7c001490 con 0x7f6e840d22b0 2024-03-06T07:22:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e837fe640 1 -- 172.21.15.77:0/3236735290 >> v2:172.21.15.158:3300/0 conn(0x7f6e841614d0 msgr2=0x7f6e840d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e837fe640 1 --2- 172.21.15.77:0/3236735290 >> v2:172.21.15.158:3300/0 conn(0x7f6e841614d0 0x7f6e840d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e837fe640 1 -- 172.21.15.77:0/3236735290 >> v1:172.21.15.158:6789/0 conn(0x7f6e840d22b0 legacy=0x7f6e840d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.011+0000 7f6e837fe640 1 -- 172.21.15.77:0/3236735290 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e840cf6a0 con 0x7f6e840d1910 2024-03-06T07:22:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.012+0000 7f6e837fe640 1 --2- 172.21.15.77:0/3236735290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840d1910 0x7f6e840d1d70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e780097a0 tx=0x7f6e7802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7c8acc9b070cea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.012+0000 7f6e82ffd640 1 -- 172.21.15.77:0/3236735290 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78035020 con 0x7f6e840d1910 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.012+0000 7f6e82ffd640 1 -- 172.21.15.77:0/3236735290 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e78031830 con 0x7f6e840d1910 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.012+0000 7f6e82ffd640 1 -- 172.21.15.77:0/3236735290 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78031c00 con 0x7f6e840d1910 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.012+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/3236735290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840d1910 msgr2=0x7f6e840d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.012+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/3236735290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840d1910 0x7f6e840d1d70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e780097a0 tx=0x7f6e7802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.013+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/3236735290 shutdown_connections 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.013+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/3236735290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840d1910 0x7f6e840d1d70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.013+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/3236735290 >> v2:172.21.15.158:3300/0 conn(0x7f6e841614d0 0x7f6e840d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.013+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/3236735290 >> 172.21.15.77:0/3236735290 conn(0x7f6e840d3d40 msgr2=0x7f6e840d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.013+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/3236735290 shutdown_connections 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.013+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/3236735290 wait complete. 2024-03-06T07:22:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.014+0000 7f6e8aa4e640 1 Processor -- start 2024-03-06T07:22:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.014+0000 7f6e8aa4e640 1 -- start start 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.014+0000 7f6e8aa4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e840d1910 0x7f6e84153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.014+0000 7f6e8aa4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840d22b0 0x7f6e840b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.014+0000 7f6e8aa4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e841614d0 0x7f6e84151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e8aa4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840b0910 con 0x7f6e840d22b0 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e8aa4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840cfc60 con 0x7f6e840d1910 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e8aa4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840b1e40 con 0x7f6e841614d0 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840d22b0 0x7f6e840b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840d22b0 0x7f6e840b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56590/0 (socket says 172.21.15.77:56590) 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e837fe640 1 -- 172.21.15.77:0/845396342 learned_addr learned my addr 172.21.15.77:0/845396342 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e83fff640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e840d1910 0x7f6e84153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e88feb640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e841614d0 0x7f6e84151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e83fff640 1 -- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e841614d0 msgr2=0x7f6e84151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e83fff640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e841614d0 0x7f6e84151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e83fff640 1 -- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840d22b0 msgr2=0x7f6e840b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e83fff640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840d22b0 0x7f6e840b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e83fff640 1 -- 172.21.15.77:0/845396342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e840b9690 con 0x7f6e840d1910 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e837fe640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840d22b0 0x7f6e840b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e83fff640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e840d1910 0x7f6e84153300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400e870 tx=0x7f6e7400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e80ff9640 1 -- 172.21.15.77:0/845396342 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74017070 con 0x7f6e840d1910 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e80ff9640 1 -- 172.21.15.77:0/845396342 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7400ee60 con 0x7f6e840d1910 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e84150d20 con 0x7f6e840d1910 2024-03-06T07:22:39.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e80ff9640 1 -- 172.21.15.77:0/845396342 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7400cb60 con 0x7f6e840d1910 2024-03-06T07:22:39.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.015+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e841512e0 con 0x7f6e840d1910 2024-03-06T07:22:39.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.017+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e84150b30 con 0x7f6e840d1910 2024-03-06T07:22:39.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.017+0000 7f6e80ff9640 1 -- 172.21.15.77:0/845396342 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e7401a070 con 0x7f6e840d1910 2024-03-06T07:22:39.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.017+0000 7f6e80ff9640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7006b4d0 0x7f6e7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.017+0000 7f6e80ff9640 1 -- 172.21.15.77:0/845396342 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e74089f20 con 0x7f6e840d1910 2024-03-06T07:22:39.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.017+0000 7f6e837fe640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7006b4d0 0x7f6e7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.018+0000 7f6e837fe640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7006b4d0 0x7f6e7006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009560 tx=0x7f6e7803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.026+0000 7f6e80ff9640 1 -- 172.21.15.77:0/845396342 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e740587c0 con 0x7f6e840d1910 2024-03-06T07:22:39.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.205+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e84068ff0 con 0x7f6e840d1910 2024-03-06T07:22:39.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.206+0000 7f6e80ff9640 1 -- 172.21.15.77:0/845396342 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e7405c860 con 0x7f6e840d1910 2024-03-06T07:22:39.208 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:39.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7006b4d0 msgr2=0x7f6e7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7006b4d0 0x7f6e7006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009560 tx=0x7f6e7803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e840d1910 msgr2=0x7f6e84153300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e840d1910 0x7f6e84153300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400e870 tx=0x7f6e7400ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:39.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 shutdown_connections 2024-03-06T07:22:39.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7006b4d0 0x7f6e7006d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840d22b0 0x7f6e840b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e841614d0 0x7f6e84151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 --2- 172.21.15.77:0/845396342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e840d1910 0x7f6e84153300 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 >> 172.21.15.77:0/845396342 conn(0x7f6e840d3d40 msgr2=0x7f6e8406c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 shutdown_connections 2024-03-06T07:22:39.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.208+0000 7f6e8aa4e640 1 -- 172.21.15.77:0/845396342 wait complete. 2024-03-06T07:22:39.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:39.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.323+0000 7fa6933d0640 1 Processor -- start 2024-03-06T07:22:39.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.323+0000 7fa6933d0640 1 -- start start 2024-03-06T07:22:39.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa6933d0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa68c15dd00 0x7fa68c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa6933d0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c06b6c0 0x7fa68c0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa6933d0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa68c161bd0 con 0x7fa68c15e0e0 2024-03-06T07:22:39.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa6933d0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68c070220 con 0x7fa68c15dd00 2024-03-06T07:22:39.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa6933d0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c161e90 con 0x7fa68c06b6c0 2024-03-06T07:22:39.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa69116c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa68c15dd00 0x7fa68c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa69116c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa68c15dd00 0x7fa68c06e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56596/0 (socket says 172.21.15.77:56596) 2024-03-06T07:22:39.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa69116c640 1 -- 172.21.15.77:0/4094669812 learned_addr learned my addr 172.21.15.77:0/4094669812 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:39.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa69196d640 1 --2- 172.21.15.77:0/4094669812 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c06b6c0 0x7fa68c0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa69116c640 1 -- 172.21.15.77:0/4094669812 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c06b6c0 msgr2=0x7fa68c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa69116c640 1 --2- 172.21.15.77:0/4094669812 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c06b6c0 0x7fa68c0b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa69116c640 1 -- 172.21.15.77:0/4094669812 >> v1:172.21.15.158:6789/0 conn(0x7fa68c15e0e0 legacy=0x7fa68c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.324+0000 7fa69116c640 1 -- 172.21.15.77:0/4094669812 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c162170 con 0x7fa68c15dd00 2024-03-06T07:22:39.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.325+0000 7fa69116c640 1 --2- 172.21.15.77:0/4094669812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c15dd00 0x7fa68c06e620 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa688009870 tx=0x7fa68802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a956323e108516 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.325+0000 7fa683fff640 1 -- 172.21.15.77:0/4094669812 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688037070 con 0x7fa68c15dd00 2024-03-06T07:22:39.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.325+0000 7fa683fff640 1 -- 172.21.15.77:0/4094669812 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa688004230 con 0x7fa68c15dd00 2024-03-06T07:22:39.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.325+0000 7fa683fff640 1 -- 172.21.15.77:0/4094669812 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688003610 con 0x7fa68c15dd00 2024-03-06T07:22:39.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.326+0000 7fa6933d0640 1 -- 172.21.15.77:0/4094669812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c15dd00 msgr2=0x7fa68c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.326+0000 7fa6933d0640 1 --2- 172.21.15.77:0/4094669812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c15dd00 0x7fa68c06e620 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa688009870 tx=0x7fa68802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.326+0000 7fa6933d0640 1 -- 172.21.15.77:0/4094669812 shutdown_connections 2024-03-06T07:22:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.326+0000 7fa6933d0640 1 --2- 172.21.15.77:0/4094669812 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c06b6c0 0x7fa68c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.326+0000 7fa6933d0640 1 --2- 172.21.15.77:0/4094669812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c15dd00 0x7fa68c06e620 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.326+0000 7fa6933d0640 1 -- 172.21.15.77:0/4094669812 >> 172.21.15.77:0/4094669812 conn(0x7fa68c0b2660 msgr2=0x7fa68c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.326+0000 7fa6933d0640 1 -- 172.21.15.77:0/4094669812 shutdown_connections 2024-03-06T07:22:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.326+0000 7fa6933d0640 1 -- 172.21.15.77:0/4094669812 wait complete. 2024-03-06T07:22:39.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.327+0000 7fa6933d0640 1 Processor -- start 2024-03-06T07:22:39.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.327+0000 7fa6933d0640 1 -- start start 2024-03-06T07:22:39.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa6933d0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c06b6c0 0x7fa68c145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa6933d0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c15dd00 0x7fa68c15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa6933d0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa68c15fee0 0x7fa68c15aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa6933d0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c0b0180 con 0x7fa68c06b6c0 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa6933d0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c0b0000 con 0x7fa68c15fee0 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa6933d0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68c0b0300 con 0x7fa68c15dd00 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c06b6c0 0x7fa68c145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c06b6c0 0x7fa68c145460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56598/0 (socket says 172.21.15.77:56598) 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 -- 172.21.15.77:0/573115268 learned_addr learned my addr 172.21.15.77:0/573115268 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69196d640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa68c15fee0 0x7fa68c15aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69096b640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c15dd00 0x7fa68c15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 -- 172.21.15.77:0/573115268 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c15dd00 msgr2=0x7fa68c15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c15dd00 0x7fa68c15f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 -- 172.21.15.77:0/573115268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa68c15fee0 msgr2=0x7fa68c15aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa68c15fee0 0x7fa68c15aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 -- 172.21.15.77:0/573115268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c1459a0 con 0x7fa68c06b6c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa69116c640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c06b6c0 0x7fa68c145460 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa688000c00 tx=0x7fa6880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa681ffb640 1 -- 172.21.15.77:0/573115268 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688041070 con 0x7fa68c06b6c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa681ffb640 1 -- 172.21.15.77:0/573115268 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa688030510 con 0x7fa68c06b6c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa681ffb640 1 -- 172.21.15.77:0/573115268 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688037040 con 0x7fa68c06b6c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.328+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68c1e7730 con 0x7fa68c06b6c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.329+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68c1e7cf0 con 0x7fa68c06b6c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.329+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68c1602f0 con 0x7fa68c06b6c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.330+0000 7fa681ffb640 1 -- 172.21.15.77:0/573115268 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa688044070 con 0x7fa68c06b6c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.330+0000 7fa681ffb640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa67006b4d0 0x7fa67006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.330+0000 7fa681ffb640 1 -- 172.21.15.77:0/573115268 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6880abba0 con 0x7fa68c06b6c0 2024-03-06T07:22:39.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.331+0000 7fa69096b640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa67006b4d0 0x7fa67006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.331+0000 7fa69096b640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa67006b4d0 0x7fa67006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa68c1549d0 tx=0x7fa67c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.339+0000 7fa681ffb640 1 -- 172.21.15.77:0/573115268 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa68c1602f0 con 0x7fa68c06b6c0 2024-03-06T07:22:39.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.514+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa68c068ff0 con 0x7fa68c06b6c0 2024-03-06T07:22:39.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.514+0000 7fa681ffb640 1 -- 172.21.15.77:0/573115268 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa68c1459a0 con 0x7fa68c06b6c0 2024-03-06T07:22:39.517 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa67006b4d0 msgr2=0x7fa67006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa67006b4d0 0x7fa67006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa68c1549d0 tx=0x7fa67c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c06b6c0 msgr2=0x7fa68c145460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c06b6c0 0x7fa68c145460 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa688000c00 tx=0x7fa6880046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 shutdown_connections 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa68c15fee0 0x7fa68c15aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa68c15dd00 0x7fa68c15f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa67006b4d0 0x7fa67006d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 --2- 172.21.15.77:0/573115268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa68c06b6c0 0x7fa68c145460 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.516+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 >> 172.21.15.77:0/573115268 conn(0x7fa68c0b2660 msgr2=0x7fa68c071a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.517+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 shutdown_connections 2024-03-06T07:22:39.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.517+0000 7fa6933d0640 1 -- 172.21.15.77:0/573115268 wait complete. 2024-03-06T07:22:39.529 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.630+0000 7fdb7f765640 1 Processor -- start 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.630+0000 7fdb7f765640 1 -- start start 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.631+0000 7fdb7f765640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdb780b1e40 0x7fdb78168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.631+0000 7fdb7f765640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780d2730 con 0x7fdb780d4860 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.631+0000 7fdb7f765640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb781618b0 con 0x7fdb781614d0 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.631+0000 7fdb7f765640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb780d4cf0 con 0x7fdb780b1e40 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.631+0000 7fdb7cd00640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdb780d4860 0x7fdb780d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34244/0 (socket says 172.21.15.77:34244) 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.631+0000 7fdb7dd02640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdb780b1e40 0x7fdb78168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.631+0000 7fdb7cd00640 1 -- 172.21.15.77:0/1268432625 learned_addr learned my addr 172.21.15.77:0/1268432625 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.632+0000 7fdb7dd02640 1 -- 172.21.15.77:0/1268432625 >> v1:172.21.15.158:6789/0 conn(0x7fdb781614d0 legacy=0x7fdb780d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.632+0000 7fdb6ffff640 1 -- 172.21.15.77:0/1268432625 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817093516 0 0) 0x7fdb781618b0 con 0x7fdb781614d0 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.632+0000 7fdb7dd02640 1 -- 172.21.15.77:0/1268432625 >> v1:172.21.15.77:6789/0 conn(0x7fdb780d4860 legacy=0x7fdb780d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.632+0000 7fdb7dd02640 1 -- 172.21.15.77:0/1268432625 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780cf400 con 0x7fdb780b1e40 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.632+0000 7fdb7dd02640 1 --2- 172.21.15.77:0/1268432625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb78168b60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdb780cf5d0 tx=0x7fdb7002b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7ff1d453bd6ff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb6ffff640 1 -- 172.21.15.77:0/1268432625 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70037070 con 0x7fdb780b1e40 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb6ffff640 1 -- 172.21.15.77:0/1268432625 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb70030710 con 0x7fdb780b1e40 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb6ffff640 1 -- 172.21.15.77:0/1268432625 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70030ae0 con 0x7fdb780b1e40 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb7f765640 1 -- 172.21.15.77:0/1268432625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 msgr2=0x7fdb78168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb7f765640 1 --2- 172.21.15.77:0/1268432625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb78168b60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdb780cf5d0 tx=0x7fdb7002b6f0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb7f765640 1 -- 172.21.15.77:0/1268432625 shutdown_connections 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb7f765640 1 --2- 172.21.15.77:0/1268432625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb78168b60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb7f765640 1 -- 172.21.15.77:0/1268432625 >> 172.21.15.77:0/1268432625 conn(0x7fdb780d3d40 msgr2=0x7fdb780d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb7f765640 1 -- 172.21.15.77:0/1268432625 shutdown_connections 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.633+0000 7fdb7f765640 1 -- 172.21.15.77:0/1268432625 wait complete. 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.634+0000 7fdb7f765640 1 Processor -- start 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.634+0000 7fdb7f765640 1 -- start start 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7f765640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb780b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7f765640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb780d4860 0x7fdb7814aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7f765640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb781614d0 0x7fdb780b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7f765640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780cfc60 con 0x7fdb780b1e40 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7f765640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780cf980 con 0x7fdb780d4860 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7f765640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780b0910 con 0x7fdb781614d0 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7d501640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb780b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7d501640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb780b98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56612/0 (socket says 172.21.15.77:56612) 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7d501640 1 -- 172.21.15.77:0/3832412450 learned_addr learned my addr 172.21.15.77:0/3832412450 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7dd02640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb781614d0 0x7fdb780b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7cd00640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb780d4860 0x7fdb7814aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7d501640 1 -- 172.21.15.77:0/3832412450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb781614d0 msgr2=0x7fdb780b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7d501640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb781614d0 0x7fdb780b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7d501640 1 -- 172.21.15.77:0/3832412450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb780d4860 msgr2=0x7fdb7814aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7d501640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb780d4860 0x7fdb7814aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.635+0000 7fdb7d501640 1 -- 172.21.15.77:0/3832412450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7814e0e0 con 0x7fdb780b1e40 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.636+0000 7fdb7d501640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb780b98e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400c890 tx=0x7fdb7400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.636+0000 7fdb6dffb640 1 -- 172.21.15.77:0/3832412450 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74017070 con 0x7fdb780b1e40 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.636+0000 7fdb6dffb640 1 -- 172.21.15.77:0/3832412450 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb74003dc0 con 0x7fdb780b1e40 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.636+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb780b8e00 con 0x7fdb780b1e40 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.636+0000 7fdb6dffb640 1 -- 172.21.15.77:0/3832412450 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb740052b0 con 0x7fdb780b1e40 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.636+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb780b8a40 con 0x7fdb780b1e40 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.637+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb78068ff0 con 0x7fdb780b1e40 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.637+0000 7fdb6dffb640 1 -- 172.21.15.77:0/3832412450 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb7401a070 con 0x7fdb780b1e40 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.637+0000 7fdb6dffb640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb5c06b4d0 0x7fdb5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.638+0000 7fdb6dffb640 1 -- 172.21.15.77:0/3832412450 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb74089850 con 0x7fdb780b1e40 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.638+0000 7fdb7cd00640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb5c06b4d0 0x7fdb5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.638+0000 7fdb7cd00640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb5c06b4d0 0x7fdb5c06d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdb780d1620 tx=0x7fdb68007e20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.642+0000 7fdb6dffb640 1 -- 172.21.15.77:0/3832412450 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb74058080 con 0x7fdb780b1e40 2024-03-06T07:22:39.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.813+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdb7815feb0 con 0x7fdb780b1e40 2024-03-06T07:22:39.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.813+0000 7fdb6dffb640 1 -- 172.21.15.77:0/3832412450 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdb7405c120 con 0x7fdb780b1e40 2024-03-06T07:22:39.815 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:39.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb5c06b4d0 msgr2=0x7fdb5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb5c06b4d0 0x7fdb5c06d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdb780d1620 tx=0x7fdb68007e20 comp rx=0 tx=0).stop 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 msgr2=0x7fdb780b98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb780b98e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400c890 tx=0x7fdb7400cd50 comp rx=0 tx=0).stop 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 shutdown_connections 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb781614d0 0x7fdb780b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb780d4860 0x7fdb7814aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb5c06b4d0 0x7fdb5c06d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 --2- 172.21.15.77:0/3832412450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb780b1e40 0x7fdb780b98e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 >> 172.21.15.77:0/3832412450 conn(0x7fdb780d3d40 msgr2=0x7fdb780b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 shutdown_connections 2024-03-06T07:22:39.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.815+0000 7fdb7f765640 1 -- 172.21.15.77:0/3832412450 wait complete. 2024-03-06T07:22:39.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:39.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.928+0000 7f463d7e4640 1 Processor -- start 2024-03-06T07:22:39.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.928+0000 7f463d7e4640 1 -- start start 2024-03-06T07:22:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.928+0000 7f463d7e4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f46380b4460 0x7f46380d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.928+0000 7f463d7e4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4638162970 0x7f46380b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.928+0000 7f463d7e4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46380d22c0 con 0x7f4638161cf0 2024-03-06T07:22:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.928+0000 7f463d7e4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4638161b00 con 0x7f46380b4460 2024-03-06T07:22:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.928+0000 7f463d7e4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46380cecc0 con 0x7f4638162970 2024-03-06T07:22:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f4636ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f46380b4460 0x7f46380d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f46377fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4638161cf0 0x7f46380cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34256/0 (socket says 172.21.15.77:34256) 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f4636ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f46380b4460 0x7f46380d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56628/0 (socket says 172.21.15.77:56628) 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f46377fe640 1 -- 172.21.15.77:0/198854868 learned_addr learned my addr 172.21.15.77:0/198854868 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f46367fc640 1 --2- 172.21.15.77:0/198854868 >> v2:172.21.15.158:3300/0 conn(0x7f4638162970 0x7f46380b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f4636ffd640 1 -- 172.21.15.77:0/198854868 >> v2:172.21.15.158:3300/0 conn(0x7f4638162970 msgr2=0x7f46380b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f4636ffd640 1 --2- 172.21.15.77:0/198854868 >> v2:172.21.15.158:3300/0 conn(0x7f4638162970 0x7f46380b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f4636ffd640 1 -- 172.21.15.77:0/198854868 >> v1:172.21.15.77:6789/0 conn(0x7f4638161cf0 legacy=0x7f46380cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f4636ffd640 1 -- 172.21.15.77:0/198854868 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46380d25a0 con 0x7f46380b4460 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.929+0000 7f4635ffb640 1 -- 172.21.15.77:0/198854868 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069116646 0 0) 0x7f46380d22c0 con 0x7f4638161cf0 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f4636ffd640 1 --2- 172.21.15.77:0/198854868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380d4800 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f46380d17e0 tx=0x7f463002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2551bc581d022304 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f4635ffb640 1 -- 172.21.15.77:0/198854868 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630035020 con 0x7f46380b4460 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f4635ffb640 1 -- 172.21.15.77:0/198854868 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4630031830 con 0x7f46380b4460 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f4635ffb640 1 -- 172.21.15.77:0/198854868 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630031c00 con 0x7f46380b4460 2024-03-06T07:22:39.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f463d7e4640 1 -- 172.21.15.77:0/198854868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 msgr2=0x7f46380d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f463d7e4640 1 --2- 172.21.15.77:0/198854868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380d4800 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f46380d17e0 tx=0x7f463002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f463d7e4640 1 -- 172.21.15.77:0/198854868 shutdown_connections 2024-03-06T07:22:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f463d7e4640 1 --2- 172.21.15.77:0/198854868 >> v2:172.21.15.158:3300/0 conn(0x7f4638162970 0x7f46380b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f463d7e4640 1 --2- 172.21.15.77:0/198854868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380d4800 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f463d7e4640 1 -- 172.21.15.77:0/198854868 >> 172.21.15.77:0/198854868 conn(0x7f46380d3ce0 msgr2=0x7f46380d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f463d7e4640 1 -- 172.21.15.77:0/198854868 shutdown_connections 2024-03-06T07:22:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.930+0000 7f463d7e4640 1 -- 172.21.15.77:0/198854868 wait complete. 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f463d7e4640 1 Processor -- start 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f463d7e4640 1 -- start start 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f463d7e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f463d7e4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4638161cf0 0x7f46380c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f463d7e4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4638162970 0x7f46380b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f463d7e4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46380ce950 con 0x7f46380b4460 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f463d7e4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46380d2b60 con 0x7f4638162970 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f463d7e4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46380b3ac0 con 0x7f4638161cf0 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f4636ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f4636ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56644/0 (socket says 172.21.15.77:56644) 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.932+0000 7f4636ffd640 1 -- 172.21.15.77:0/201820867 learned_addr learned my addr 172.21.15.77:0/201820867 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f46377fe640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4638162970 0x7f46380b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f46367fc640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4638161cf0 0x7f46380c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f4636ffd640 1 -- 172.21.15.77:0/201820867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4638161cf0 msgr2=0x7f46380c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f4636ffd640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4638161cf0 0x7f46380c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f4636ffd640 1 -- 172.21.15.77:0/201820867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4638162970 msgr2=0x7f46380b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f4636ffd640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4638162970 0x7f46380b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f4636ffd640 1 -- 172.21.15.77:0/201820867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46380c9610 con 0x7f46380b4460 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f4636ffd640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380cc450 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4630000c00 tx=0x7f46300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f461f7fe640 1 -- 172.21.15.77:0/201820867 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630035040 con 0x7f46380b4460 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f461f7fe640 1 -- 172.21.15.77:0/201820867 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4630004250 con 0x7f46380b4460 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46380c98f0 con 0x7f46380b4460 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f461f7fe640 1 -- 172.21.15.77:0/201820867 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463003a640 con 0x7f46380b4460 2024-03-06T07:22:39.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.933+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46380c9eb0 con 0x7f46380b4460 2024-03-06T07:22:39.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.934+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463804f330 con 0x7f46380b4460 2024-03-06T07:22:39.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.935+0000 7f461f7fe640 1 -- 172.21.15.77:0/201820867 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4630038020 con 0x7f46380b4460 2024-03-06T07:22:39.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.935+0000 7f461f7fe640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f462406b4d0 0x7f462406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.935+0000 7f461f7fe640 1 -- 172.21.15.77:0/201820867 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46300abdc0 con 0x7f46380b4460 2024-03-06T07:22:39.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.935+0000 7f46367fc640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f462406b4d0 0x7f462406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.935+0000 7f46367fc640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f462406b4d0 0x7f462406d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f46380d47f0 tx=0x7f462c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:39.939+0000 7f461f7fe640 1 -- 172.21.15.77:0/201820867 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f463007a5f0 con 0x7f46380b4460 2024-03-06T07:22:40.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.124+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f463806b180 con 0x7f46380b4460 2024-03-06T07:22:40.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.124+0000 7f461f7fe640 1 -- 172.21.15.77:0/201820867 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f463007e690 con 0x7f46380b4460 2024-03-06T07:22:40.126 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:22:40.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f462406b4d0 msgr2=0x7f462406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f462406b4d0 0x7f462406d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f46380d47f0 tx=0x7f462c00b040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 msgr2=0x7f46380cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380cc450 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4630000c00 tx=0x7f46300046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 shutdown_connections 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4638162970 0x7f46380b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4638161cf0 0x7f46380c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f462406b4d0 0x7f462406d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 --2- 172.21.15.77:0/201820867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46380b4460 0x7f46380cc450 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 >> 172.21.15.77:0/201820867 conn(0x7f46380d3ce0 msgr2=0x7f46380d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 shutdown_connections 2024-03-06T07:22:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.126+0000 7f463d7e4640 1 -- 172.21.15.77:0/201820867 wait complete. 2024-03-06T07:22:40.139 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:22:40.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:40.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:40.166 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:40.166 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:22:40.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:40.167 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:22:40.222 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:22:40.222 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:40.238 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:22:40.238 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:40.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:40.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.393+0000 7f9b26f5c640 1 Processor -- start 2024-03-06T07:22:40.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.393+0000 7f9b26f5c640 1 -- start start 2024-03-06T07:22:40.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b26f5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b200d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b26f5c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9b200b41f0 0x7f9b200b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b26f5c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b200b3430 con 0x7f9b2015dfe0 2024-03-06T07:22:40.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b26f5c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b200d2c50 con 0x7f9b200b41f0 2024-03-06T07:22:40.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b26f5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b200b31e0 con 0x7f9b200d22b0 2024-03-06T07:22:40.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b254f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9b200b41f0 0x7f9b200b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b254f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9b200b41f0 0x7f9b200b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56654/0 (socket says 172.21.15.77:56654) 2024-03-06T07:22:40.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b254f9640 1 -- 172.21.15.77:0/736838951 learned_addr learned my addr 172.21.15.77:0/736838951 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:40.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b1ffff640 1 --2- 172.21.15.77:0/736838951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b200d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b254f9640 1 -- 172.21.15.77:0/736838951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 msgr2=0x7f9b200d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b254f9640 1 --2- 172.21.15.77:0/736838951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b200d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b254f9640 1 -- 172.21.15.77:0/736838951 >> v1:172.21.15.158:6789/0 conn(0x7f9b2015dfe0 legacy=0x7f9b200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b254f9640 1 -- 172.21.15.77:0/736838951 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b200b3710 con 0x7f9b200b41f0 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.394+0000 7f9b1ffff640 1 --2- 172.21.15.77:0/736838951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b200d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.395+0000 7f9b254f9640 1 --2- 172.21.15.77:0/736838951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b200b41f0 0x7f9b200b4650 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9b18009e50 tx=0x7f9b1802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684ad415702a01f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.395+0000 7f9b1f7fe640 1 -- 172.21.15.77:0/736838951 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1802cb00 con 0x7f9b200b41f0 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.395+0000 7f9b1f7fe640 1 -- 172.21.15.77:0/736838951 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1802cca0 con 0x7f9b200b41f0 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.395+0000 7f9b1f7fe640 1 -- 172.21.15.77:0/736838951 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18034610 con 0x7f9b200b41f0 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.395+0000 7f9b26f5c640 1 -- 172.21.15.77:0/736838951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b200b41f0 msgr2=0x7f9b200b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.395+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/736838951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b200b41f0 0x7f9b200b4650 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9b18009e50 tx=0x7f9b1802bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.396+0000 7f9b26f5c640 1 -- 172.21.15.77:0/736838951 shutdown_connections 2024-03-06T07:22:40.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.396+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/736838951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b200d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.396+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/736838951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b200b41f0 0x7f9b200b4650 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.396+0000 7f9b26f5c640 1 -- 172.21.15.77:0/736838951 >> 172.21.15.77:0/736838951 conn(0x7f9b200d3cd0 msgr2=0x7f9b200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.396+0000 7f9b26f5c640 1 -- 172.21.15.77:0/736838951 shutdown_connections 2024-03-06T07:22:40.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.396+0000 7f9b26f5c640 1 -- 172.21.15.77:0/736838951 wait complete. 2024-03-06T07:22:40.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.397+0000 7f9b26f5c640 1 Processor -- start 2024-03-06T07:22:40.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.397+0000 7f9b26f5c640 1 -- start start 2024-03-06T07:22:40.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b26f5c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b200b41f0 0x7f9b200cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b26f5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b2015c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b26f5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b2015dfe0 0x7f9b201f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b26f5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b200cfa00 con 0x7f9b2015dfe0 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b26f5c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b200b3d60 con 0x7f9b200b41f0 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b26f5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b200d1c70 con 0x7f9b200d22b0 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b2015dfe0 0x7f9b201f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b2015dfe0 0x7f9b201f3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56664/0 (socket says 172.21.15.77:56664) 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 -- 172.21.15.77:0/854120381 learned_addr learned my addr 172.21.15.77:0/854120381 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b24cf8640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b200b41f0 0x7f9b200cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b1ffff640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b2015c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 -- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 msgr2=0x7f9b2015c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b2015c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 -- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b200b41f0 msgr2=0x7f9b200cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b200b41f0 0x7f9b200cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 -- 172.21.15.77:0/854120381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b200cada0 con 0x7f9b2015dfe0 2024-03-06T07:22:40.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b1ffff640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b2015c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b254f9640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b2015dfe0 0x7f9b201f3820 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9b18036040 tx=0x7f9b18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b1d7fa640 1 -- 172.21.15.77:0/854120381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18043020 con 0x7f9b2015dfe0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b1d7fa640 1 -- 172.21.15.77:0/854120381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1802ce50 con 0x7f9b2015dfe0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b201f43b0 con 0x7f9b2015dfe0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b1d7fa640 1 -- 172.21.15.77:0/854120381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18009510 con 0x7f9b2015dfe0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.398+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b200cb340 con 0x7f9b2015dfe0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.399+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2004f330 con 0x7f9b2015dfe0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.400+0000 7f9b1d7fa640 1 -- 172.21.15.77:0/854120381 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b18005d10 con 0x7f9b2015dfe0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.400+0000 7f9b1d7fa640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b0406b4d0 0x7f9b0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.400+0000 7f9b1d7fa640 1 -- 172.21.15.77:0/854120381 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b180acaa0 con 0x7f9b2015dfe0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.400+0000 7f9b24cf8640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b0406b4d0 0x7f9b0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.400+0000 7f9b24cf8640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b0406b4d0 0x7f9b0406d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9b200cc480 tx=0x7f9b14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.404+0000 7f9b1d7fa640 1 -- 172.21.15.77:0/854120381 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b201f43b0 con 0x7f9b2015dfe0 2024-03-06T07:22:40.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.576+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b200cb510 con 0x7f9b2015dfe0 2024-03-06T07:22:40.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.577+0000 7f9b1d7fa640 1 -- 172.21.15.77:0/854120381 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f9b1807b920 con 0x7f9b2015dfe0 2024-03-06T07:22:40.579 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:40.579 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b0406b4d0 msgr2=0x7f9b0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b0406b4d0 0x7f9b0406d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9b200cc480 tx=0x7f9b14009040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b2015dfe0 msgr2=0x7f9b201f3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b2015dfe0 0x7f9b201f3820 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9b18036040 tx=0x7f9b18004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 shutdown_connections 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9b0406b4d0 0x7f9b0406d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b2015dfe0 0x7f9b201f3820 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b200d22b0 0x7f9b2015c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 --2- 172.21.15.77:0/854120381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b200b41f0 0x7f9b200cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 >> 172.21.15.77:0/854120381 conn(0x7f9b200d3cd0 msgr2=0x7f9b2006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 shutdown_connections 2024-03-06T07:22:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.579+0000 7f9b26f5c640 1 -- 172.21.15.77:0/854120381 wait complete. 2024-03-06T07:22:40.592 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:40.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:40.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.693+0000 7f3e0b14b640 1 Processor -- start 2024-03-06T07:22:40.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.693+0000 7f3e0b14b640 1 -- start start 2024-03-06T07:22:40.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e0b14b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3e0415bb40 0x7f3e040b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e0b14b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0415ff90 con 0x7f3e0415f7d0 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e0b14b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e04160270 con 0x7f3e040cfa90 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e0b14b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0406c910 con 0x7f3e0415bb40 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e08ee7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3e0415bb40 0x7f3e040b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e03fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3e0415f7d0 0x7f3e0406d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34266/0 (socket says 172.21.15.77:34266) 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e03fff640 1 -- 172.21.15.77:0/2005449199 learned_addr learned my addr 172.21.15.77:0/2005449199 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e08ee7640 1 -- 172.21.15.77:0/2005449199 >> v1:172.21.15.158:6789/0 conn(0x7f3e040cfa90 legacy=0x7f3e040cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e08ee7640 1 -- 172.21.15.77:0/2005449199 >> v1:172.21.15.77:6789/0 conn(0x7f3e0415f7d0 legacy=0x7f3e0406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e037fe640 1 -- 172.21.15.77:0/2005449199 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559925062 0 0) 0x7f3e0415ff90 con 0x7f3e0415f7d0 2024-03-06T07:22:40.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.694+0000 7f3e08ee7640 1 -- 172.21.15.77:0/2005449199 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04160550 con 0x7f3e0415bb40 2024-03-06T07:22:40.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.695+0000 7f3e08ee7640 1 --2- 172.21.15.77:0/2005449199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e040b4970 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3e0406ca90 tx=0x7f3df402b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832eb696d2f530f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.695+0000 7f3e037fe640 1 -- 172.21.15.77:0/2005449199 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4037070 con 0x7f3e0415bb40 2024-03-06T07:22:40.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.695+0000 7f3e037fe640 1 -- 172.21.15.77:0/2005449199 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df4030710 con 0x7f3e0415bb40 2024-03-06T07:22:40.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.695+0000 7f3e037fe640 1 -- 172.21.15.77:0/2005449199 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4030ae0 con 0x7f3e0415bb40 2024-03-06T07:22:40.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.696+0000 7f3e0b14b640 1 -- 172.21.15.77:0/2005449199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 msgr2=0x7f3e040b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.696+0000 7f3e0b14b640 1 --2- 172.21.15.77:0/2005449199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e040b4970 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3e0406ca90 tx=0x7f3df402b730 comp rx=0 tx=0).stop 2024-03-06T07:22:40.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.696+0000 7f3e0b14b640 1 -- 172.21.15.77:0/2005449199 shutdown_connections 2024-03-06T07:22:40.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.696+0000 7f3e0b14b640 1 --2- 172.21.15.77:0/2005449199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e040b4970 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.696+0000 7f3e0b14b640 1 -- 172.21.15.77:0/2005449199 >> 172.21.15.77:0/2005449199 conn(0x7f3e0406b190 msgr2=0x7f3e0406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.696+0000 7f3e0b14b640 1 -- 172.21.15.77:0/2005449199 shutdown_connections 2024-03-06T07:22:40.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.696+0000 7f3e0b14b640 1 -- 172.21.15.77:0/2005449199 wait complete. 2024-03-06T07:22:40.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.697+0000 7f3e0b14b640 1 Processor -- start 2024-03-06T07:22:40.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.697+0000 7f3e0b14b640 1 -- start start 2024-03-06T07:22:40.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e0b14b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e040cfa90 0x7f3e040add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e0b14b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e041f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e0b14b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0415f7d0 0x7f3e0415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e0b14b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0406d790 con 0x7f3e0415bb40 2024-03-06T07:22:40.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e0b14b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0415fc30 con 0x7f3e0415f7d0 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e0b14b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0415bf20 con 0x7f3e040cfa90 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e041f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e041f5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56678/0 (socket says 172.21.15.77:56678) 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 -- 172.21.15.77:0/3567481413 learned_addr learned my addr 172.21.15.77:0/3567481413 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e096e8640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0415f7d0 0x7f3e0415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e08ee7640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e040cfa90 0x7f3e040add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 -- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e040cfa90 msgr2=0x7f3e040add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e040cfa90 0x7f3e040add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 -- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0415f7d0 msgr2=0x7f3e0415ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0415f7d0 0x7f3e0415ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 -- 172.21.15.77:0/3567481413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04158de0 con 0x7f3e0415bb40 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e08ee7640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e040cfa90 0x7f3e040add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:40.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.698+0000 7f3e03fff640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e041f5ce0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3df800c470 tx=0x7f3df800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.699+0000 7f3e017fa640 1 -- 172.21.15.77:0/3567481413 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8018070 con 0x7f3e0415bb40 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.699+0000 7f3e017fa640 1 -- 172.21.15.77:0/3567481413 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df80074e0 con 0x7f3e0415bb40 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.699+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e041590c0 con 0x7f3e0415bb40 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.699+0000 7f3e017fa640 1 -- 172.21.15.77:0/3567481413 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8004a20 con 0x7f3e0415bb40 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.699+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e041f54e0 con 0x7f3e0415bb40 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.700+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e040732b0 con 0x7f3e0415bb40 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.700+0000 7f3e017fa640 1 -- 172.21.15.77:0/3567481413 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3df801b070 con 0x7f3e0415bb40 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.700+0000 7f3e017fa640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3df006b4d0 0x7f3df006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.700+0000 7f3e08ee7640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3df006b4d0 0x7f3df006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.700+0000 7f3e017fa640 1 -- 172.21.15.77:0/3567481413 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3df8089440 con 0x7f3e0415bb40 2024-03-06T07:22:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.701+0000 7f3e08ee7640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3df006b4d0 0x7f3df006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3df4034000 tx=0x7f3df4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.705+0000 7f3e017fa640 1 -- 172.21.15.77:0/3567481413 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3df8057c70 con 0x7f3e0415bb40 2024-03-06T07:22:40.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.874+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e04160020 con 0x7f3e0415bb40 2024-03-06T07:22:40.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.874+0000 7f3e017fa640 1 -- 172.21.15.77:0/3567481413 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3df805bd10 con 0x7f3e0415bb40 2024-03-06T07:22:40.876 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:40.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3df006b4d0 msgr2=0x7f3df006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3df006b4d0 0x7f3df006d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3df4034000 tx=0x7f3df4008030 comp rx=0 tx=0).stop 2024-03-06T07:22:40.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 msgr2=0x7f3e041f5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e041f5ce0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3df800c470 tx=0x7f3df800c930 comp rx=0 tx=0).stop 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 shutdown_connections 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3df006b4d0 0x7f3df006d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e0415bb40 0x7f3e041f5ce0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e0415f7d0 0x7f3e0415ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 --2- 172.21.15.77:0/3567481413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e040cfa90 0x7f3e040add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 >> 172.21.15.77:0/3567481413 conn(0x7f3e0406b190 msgr2=0x7f3e0415e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 shutdown_connections 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.876+0000 7f3e0b14b640 1 -- 172.21.15.77:0/3567481413 wait complete. 2024-03-06T07:22:40.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:40.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.990+0000 7ffa0dff7640 1 Processor -- start 2024-03-06T07:22:40.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.990+0000 7ffa0dff7640 1 -- start start 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.990+0000 7ffa0dff7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa080b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.990+0000 7ffa0dff7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffa080b55e0 0x7ffa080ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.990+0000 7ffa0dff7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0806b710 con 0x7ffa08066d90 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.990+0000 7ffa0dff7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa0806b150 con 0x7ffa080b55e0 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.990+0000 7ffa0dff7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0806b430 con 0x7ffa080cf9a0 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa06ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffa08066d90 0x7ffa080b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34282/0 (socket says 172.21.15.77:34282) 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa06ffd640 1 -- 172.21.15.77:0/2916860802 learned_addr learned my addr 172.21.15.77:0/2916860802 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa07fff640 1 --2- 172.21.15.77:0/2916860802 >> v2:172.21.15.158:3300/0 conn(0x7ffa080b55e0 0x7ffa080ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa077fe640 1 --2- 172.21.15.77:0/2916860802 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa080b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa067fc640 1 -- 172.21.15.77:0/2916860802 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61071951 0 0) 0x7ffa0806b710 con 0x7ffa08066d90 2024-03-06T07:22:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa067fc640 1 -- 172.21.15.77:0/2916860802 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f0003520 con 0x7ffa08066d90 2024-03-06T07:22:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa07fff640 1 -- 172.21.15.77:0/2916860802 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 msgr2=0x7ffa080b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa07fff640 1 --2- 172.21.15.77:0/2916860802 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa080b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa07fff640 1 -- 172.21.15.77:0/2916860802 >> v1:172.21.15.77:6789/0 conn(0x7ffa08066d90 legacy=0x7ffa080b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.991+0000 7ffa07fff640 1 -- 172.21.15.77:0/2916860802 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa080b0970 con 0x7ffa080b55e0 2024-03-06T07:22:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa07fff640 1 --2- 172.21.15.77:0/2916860802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 0x7ffa080ce040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009d90 tx=0x7ffa0002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cb8b65c32b71fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa067fc640 1 -- 172.21.15.77:0/2916860802 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00039070 con 0x7ffa080b55e0 2024-03-06T07:22:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa067fc640 1 -- 172.21.15.77:0/2916860802 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0002cc50 con 0x7ffa080b55e0 2024-03-06T07:22:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa067fc640 1 -- 172.21.15.77:0/2916860802 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00034440 con 0x7ffa080b55e0 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa0dff7640 1 -- 172.21.15.77:0/2916860802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 msgr2=0x7ffa080ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/2916860802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 0x7ffa080ce040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009d90 tx=0x7ffa0002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa0dff7640 1 -- 172.21.15.77:0/2916860802 shutdown_connections 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/2916860802 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 0x7ffa080ce040 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/2916860802 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa080b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.992+0000 7ffa0dff7640 1 -- 172.21.15.77:0/2916860802 >> 172.21.15.77:0/2916860802 conn(0x7ffa080b1d70 msgr2=0x7ffa080b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.993+0000 7ffa0dff7640 1 -- 172.21.15.77:0/2916860802 shutdown_connections 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.993+0000 7ffa0dff7640 1 -- 172.21.15.77:0/2916860802 wait complete. 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.994+0000 7ffa0dff7640 1 Processor -- start 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.994+0000 7ffa0dff7640 1 -- start start 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.994+0000 7ffa0dff7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08066d90 0x7ffa081567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.994+0000 7ffa0dff7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 0x7ffa080c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.994+0000 7ffa0dff7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa081f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.994+0000 7ffa0dff7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa080b6170 con 0x7ffa08066d90 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.994+0000 7ffa0dff7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa080b5ff0 con 0x7ffa080b55e0 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.994+0000 7ffa0dff7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa080cfd80 con 0x7ffa080cf9a0 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08066d90 0x7ffa081567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08066d90 0x7ffa081567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56694/0 (socket says 172.21.15.77:56694) 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 -- 172.21.15.77:0/3922286660 learned_addr learned my addr 172.21.15.77:0/3922286660 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa06ffd640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 0x7ffa080c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa07fff640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa081f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 -- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 msgr2=0x7ffa081f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa081f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 -- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 msgr2=0x7ffa080c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 0x7ffa080c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 -- 172.21.15.77:0/3922286660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa081f4660 con 0x7ffa08066d90 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa07fff640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa081f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa077fe640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08066d90 0x7ffa081567d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00e8a0 tx=0x7ff9fc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ff9effff640 1 -- 172.21.15.77:0/3922286660 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc017070 con 0x7ffa08066d90 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ff9effff640 1 -- 172.21.15.77:0/3922286660 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9fc00c940 con 0x7ffa08066d90 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa081f4940 con 0x7ffa08066d90 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ff9effff640 1 -- 172.21.15.77:0/3922286660 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc00cd30 con 0x7ffa08066d90 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.995+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa080c73b0 con 0x7ffa08066d90 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.996+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0809e3d0 con 0x7ffa08066d90 2024-03-06T07:22:41.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.997+0000 7ff9effff640 1 -- 172.21.15.77:0/3922286660 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9fc01a070 con 0x7ffa08066d90 2024-03-06T07:22:41.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.997+0000 7ff9effff640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9f406b4d0 0x7ff9f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.997+0000 7ff9effff640 1 -- 172.21.15.77:0/3922286660 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9fc08a410 con 0x7ffa08066d90 2024-03-06T07:22:41.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.997+0000 7ffa06ffd640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9f406b4d0 0x7ff9f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:40.997+0000 7ffa06ffd640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9f406b4d0 0x7ff9f406d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffa080cbd90 tx=0x7ff9f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.001+0000 7ff9effff640 1 -- 172.21.15.77:0/3922286660 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9fc058bb0 con 0x7ffa08066d90 2024-03-06T07:22:41.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.173+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa080c71d0 con 0x7ffa08066d90 2024-03-06T07:22:41.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.173+0000 7ff9effff640 1 -- 172.21.15.77:0/3922286660 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa080c71d0 con 0x7ffa08066d90 2024-03-06T07:22:41.175 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:41.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9f406b4d0 msgr2=0x7ff9f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9f406b4d0 0x7ff9f406d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffa080cbd90 tx=0x7ff9f8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08066d90 msgr2=0x7ffa081567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08066d90 0x7ffa081567d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00e8a0 tx=0x7ff9fc00ed60 comp rx=0 tx=0).stop 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 shutdown_connections 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa080cf9a0 0x7ffa081f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa080b55e0 0x7ffa080c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9f406b4d0 0x7ff9f406d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 --2- 172.21.15.77:0/3922286660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa08066d90 0x7ffa081567d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 >> 172.21.15.77:0/3922286660 conn(0x7ffa080b1d70 msgr2=0x7ffa0806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 shutdown_connections 2024-03-06T07:22:41.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.175+0000 7ffa0dff7640 1 -- 172.21.15.77:0/3922286660 wait complete. 2024-03-06T07:22:41.188 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.289+0000 7fbc37979640 1 Processor -- start 2024-03-06T07:22:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.289+0000 7fbc37979640 1 -- start start 2024-03-06T07:22:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc37979640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbc300cdd10 0x7fbc3015c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc37979640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbc3006e2c0 0x7fbc3006e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc37979640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3006ec00 con 0x7fbc30145700 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc37979640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc30071b60 con 0x7fbc300cdd10 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc37979640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc30070f50 con 0x7fbc3006e2c0 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc34f14640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbc300cdd10 0x7fbc3015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc34f14640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbc300cdd10 0x7fbc3015c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56702/0 (socket says 172.21.15.77:56702) 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc34f14640 1 -- 172.21.15.77:0/4120995719 learned_addr learned my addr 172.21.15.77:0/4120995719 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.290+0000 7fbc35f16640 1 --2- 172.21.15.77:0/4120995719 >> v2:172.21.15.158:3300/0 conn(0x7fbc3006e2c0 0x7fbc3006e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc34f14640 1 -- 172.21.15.77:0/4120995719 >> v2:172.21.15.158:3300/0 conn(0x7fbc3006e2c0 msgr2=0x7fbc3006e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc34f14640 1 --2- 172.21.15.77:0/4120995719 >> v2:172.21.15.158:3300/0 conn(0x7fbc3006e2c0 0x7fbc3006e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc34f14640 1 -- 172.21.15.77:0/4120995719 >> v1:172.21.15.158:6789/0 conn(0x7fbc30145700 legacy=0x7fbc300cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc34f14640 1 -- 172.21.15.77:0/4120995719 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30071230 con 0x7fbc300cdd10 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc35f16640 1 --2- 172.21.15.77:0/4120995719 >> v2:172.21.15.158:3300/0 conn(0x7fbc3006e2c0 0x7fbc3006e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc34f14640 1 --2- 172.21.15.77:0/4120995719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc3015c400 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbc300b2980 tx=0x7fbc2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff31715c116a970 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc27fff640 1 -- 172.21.15.77:0/4120995719 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20035020 con 0x7fbc300cdd10 2024-03-06T07:22:41.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc27fff640 1 -- 172.21.15.77:0/4120995719 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc20031870 con 0x7fbc300cdd10 2024-03-06T07:22:41.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.291+0000 7fbc27fff640 1 -- 172.21.15.77:0/4120995719 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20031c40 con 0x7fbc300cdd10 2024-03-06T07:22:41.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.292+0000 7fbc37979640 1 -- 172.21.15.77:0/4120995719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 msgr2=0x7fbc3015c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.292+0000 7fbc37979640 1 --2- 172.21.15.77:0/4120995719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc3015c400 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbc300b2980 tx=0x7fbc2002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:41.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.292+0000 7fbc37979640 1 -- 172.21.15.77:0/4120995719 shutdown_connections 2024-03-06T07:22:41.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.292+0000 7fbc37979640 1 --2- 172.21.15.77:0/4120995719 >> v2:172.21.15.158:3300/0 conn(0x7fbc3006e2c0 0x7fbc3006e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.292+0000 7fbc37979640 1 --2- 172.21.15.77:0/4120995719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc3015c400 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.292+0000 7fbc37979640 1 -- 172.21.15.77:0/4120995719 >> 172.21.15.77:0/4120995719 conn(0x7fbc3006b830 msgr2=0x7fbc300b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.292+0000 7fbc37979640 1 -- 172.21.15.77:0/4120995719 shutdown_connections 2024-03-06T07:22:41.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.292+0000 7fbc37979640 1 -- 172.21.15.77:0/4120995719 wait complete. 2024-03-06T07:22:41.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.293+0000 7fbc37979640 1 Processor -- start 2024-03-06T07:22:41.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.293+0000 7fbc37979640 1 -- start start 2024-03-06T07:22:41.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc37979640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc3006e2c0 0x7fbc301f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc37979640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc30160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc37979640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc30145700 0x7fbc30139c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc37979640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc300cfbc0 con 0x7fbc300cdd10 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc37979640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc300717f0 con 0x7fbc3006e2c0 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc37979640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30145ae0 con 0x7fbc30145700 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc34f14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc30160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc34f14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc30160460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56718/0 (socket says 172.21.15.77:56718) 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc34f14640 1 -- 172.21.15.77:0/1471601353 learned_addr learned my addr 172.21.15.77:0/1471601353 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc35715640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc3006e2c0 0x7fbc301f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc35f16640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc30145700 0x7fbc30139c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc34f14640 1 -- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc30145700 msgr2=0x7fbc30139c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc34f14640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc30145700 0x7fbc30139c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc34f14640 1 -- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc3006e2c0 msgr2=0x7fbc301f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc34f14640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc3006e2c0 0x7fbc301f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc34f14640 1 -- 172.21.15.77:0/1471601353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30140980 con 0x7fbc300cdd10 2024-03-06T07:22:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.294+0000 7fbc35715640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc3006e2c0 0x7fbc301f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.295+0000 7fbc34f14640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc30160460 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbc2002b950 tx=0x7fbc200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.295+0000 7fbc25ffb640 1 -- 172.21.15.77:0/1471601353 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20040070 con 0x7fbc300cdd10 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.295+0000 7fbc25ffb640 1 -- 172.21.15.77:0/1471601353 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc20030d00 con 0x7fbc300cdd10 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.295+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc30140c60 con 0x7fbc300cdd10 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.295+0000 7fbc25ffb640 1 -- 172.21.15.77:0/1471601353 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20035040 con 0x7fbc300cdd10 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.295+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3015c630 con 0x7fbc300cdd10 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.296+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3006e0b0 con 0x7fbc300cdd10 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.296+0000 7fbc25ffb640 1 -- 172.21.15.77:0/1471601353 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc20038020 con 0x7fbc300cdd10 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.297+0000 7fbc25ffb640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc1406b4d0 0x7fbc1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.297+0000 7fbc25ffb640 1 -- 172.21.15.77:0/1471601353 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc200ac070 con 0x7fbc300cdd10 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.297+0000 7fbc35715640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc1406b4d0 0x7fbc1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.297+0000 7fbc35715640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc1406b4d0 0x7fbc1406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c001d60 tx=0x7fbc2c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.301+0000 7fbc25ffb640 1 -- 172.21.15.77:0/1471601353 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc2007ae60 con 0x7fbc300cdd10 2024-03-06T07:22:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.472+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc300cc3e0 con 0x7fbc300cdd10 2024-03-06T07:22:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.472+0000 7fbc25ffb640 1 -- 172.21.15.77:0/1471601353 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc2007a760 con 0x7fbc300cdd10 2024-03-06T07:22:41.474 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc1406b4d0 msgr2=0x7fbc1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc1406b4d0 0x7fbc1406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c001d60 tx=0x7fbc2c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 msgr2=0x7fbc30160460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc30160460 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbc2002b950 tx=0x7fbc200046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 shutdown_connections 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc1406b4d0 0x7fbc1406d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc300cdd10 0x7fbc30160460 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc30145700 0x7fbc30139c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 --2- 172.21.15.77:0/1471601353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc3006e2c0 0x7fbc301f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 >> 172.21.15.77:0/1471601353 conn(0x7fbc3006b830 msgr2=0x7fbc301433f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.474+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 shutdown_connections 2024-03-06T07:22:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.475+0000 7fbc37979640 1 -- 172.21.15.77:0/1471601353 wait complete. 2024-03-06T07:22:41.487 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:41.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.585+0000 7f2705fbc640 1 Processor -- start 2024-03-06T07:22:41.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.586+0000 7f2705fbc640 1 -- start start 2024-03-06T07:22:41.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.586+0000 7f2705fbc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2700162970 0x7f27000d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.586+0000 7f2705fbc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700161cf0 0x7f2700162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.586+0000 7f2705fbc640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000d22c0 con 0x7f27000b4460 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.586+0000 7f2705fbc640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2700161b00 con 0x7f2700162970 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.586+0000 7f2705fbc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27000cecc0 con 0x7f2700161cf0 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.586+0000 7f26feffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2700162970 0x7f27000d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26feffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2700162970 0x7f27000d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56722/0 (socket says 172.21.15.77:56722) 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26feffd640 1 -- 172.21.15.77:0/2962074995 learned_addr learned my addr 172.21.15.77:0/2962074995 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26ff7fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f27000b4460 0x7f27000b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:41866/0 (socket says 172.21.15.77:41866) 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26fffff640 1 --2- 172.21.15.77:0/2962074995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700161cf0 0x7f2700162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26feffd640 1 -- 172.21.15.77:0/2962074995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700161cf0 msgr2=0x7f2700162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26feffd640 1 --2- 172.21.15.77:0/2962074995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700161cf0 0x7f2700162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26feffd640 1 -- 172.21.15.77:0/2962074995 >> v1:172.21.15.158:6789/0 conn(0x7f27000b4460 legacy=0x7f27000b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26fe7fc640 1 -- 172.21.15.77:0/2962074995 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472225039 0 0) 0x7f27000d22c0 con 0x7f27000b4460 2024-03-06T07:22:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.587+0000 7f26feffd640 1 -- 172.21.15.77:0/2962074995 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27000d25a0 con 0x7f2700162970 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f26feffd640 1 --2- 172.21.15.77:0/2962074995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2700162970 0x7f27000d4800 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009870 tx=0x7f26f402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49808cd28739be57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f26fe7fc640 1 -- 172.21.15.77:0/2962074995 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4034070 con 0x7f2700162970 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f26fe7fc640 1 -- 172.21.15.77:0/2962074995 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4030840 con 0x7f2700162970 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f26fe7fc640 1 -- 172.21.15.77:0/2962074995 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4030c10 con 0x7f2700162970 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f2705fbc640 1 -- 172.21.15.77:0/2962074995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2700162970 msgr2=0x7f27000d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f2705fbc640 1 --2- 172.21.15.77:0/2962074995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2700162970 0x7f27000d4800 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009870 tx=0x7f26f402b980 comp rx=0 tx=0).stop 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f2705fbc640 1 -- 172.21.15.77:0/2962074995 shutdown_connections 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f2705fbc640 1 --2- 172.21.15.77:0/2962074995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700161cf0 0x7f2700162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f2705fbc640 1 --2- 172.21.15.77:0/2962074995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2700162970 0x7f27000d4800 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f2705fbc640 1 -- 172.21.15.77:0/2962074995 >> 172.21.15.77:0/2962074995 conn(0x7f27000d3ce0 msgr2=0x7f27000d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f2705fbc640 1 -- 172.21.15.77:0/2962074995 shutdown_connections 2024-03-06T07:22:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.588+0000 7f2705fbc640 1 -- 172.21.15.77:0/2962074995 wait complete. 2024-03-06T07:22:41.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.589+0000 7f2705fbc640 1 Processor -- start 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f2705fbc640 1 -- start start 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f2705fbc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27000b4460 0x7f27000b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f2705fbc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2700161cf0 0x7f27000b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f2705fbc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700162970 0x7f2700154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f2705fbc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27000ce950 con 0x7f27000b4460 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f2705fbc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27000d2b60 con 0x7f2700161cf0 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f2705fbc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27000b4840 con 0x7f2700162970 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f26ff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27000b4460 0x7f27000b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f26ff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27000b4460 0x7f27000b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56736/0 (socket says 172.21.15.77:56736) 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f26ff7fe640 1 -- 172.21.15.77:0/1350223054 learned_addr learned my addr 172.21.15.77:0/1350223054 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f26fffff640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700162970 0x7f2700154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.590+0000 7f26feffd640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2700161cf0 0x7f27000b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26ff7fe640 1 -- 172.21.15.77:0/1350223054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700162970 msgr2=0x7f2700154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26ff7fe640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700162970 0x7f2700154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26ff7fe640 1 -- 172.21.15.77:0/1350223054 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2700161cf0 msgr2=0x7f27000b7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26ff7fe640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2700161cf0 0x7f27000b7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26ff7fe640 1 -- 172.21.15.77:0/1350223054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27000c8440 con 0x7f27000b4460 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26ff7fe640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27000b4460 0x7f27000b14c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f26f000c430 tx=0x7f26f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26e7fff640 1 -- 172.21.15.77:0/1350223054 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f00074e0 con 0x7f27000b4460 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26e7fff640 1 -- 172.21.15.77:0/1350223054 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f00047e0 con 0x7f27000b4460 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27000b8570 con 0x7f27000b4460 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f26e7fff640 1 -- 172.21.15.77:0/1350223054 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0004bd0 con 0x7f27000b4460 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.591+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27000b8b30 con 0x7f27000b4460 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.592+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27000c8610 con 0x7f27000b4460 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.593+0000 7f26e7fff640 1 -- 172.21.15.77:0/1350223054 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26f0011070 con 0x7f27000b4460 2024-03-06T07:22:41.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.593+0000 7f26e7fff640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f26ec06b4d0 0x7f26ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.593+0000 7f26e7fff640 1 -- 172.21.15.77:0/1350223054 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26f0089ca0 con 0x7f27000b4460 2024-03-06T07:22:41.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.593+0000 7f26feffd640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f26ec06b4d0 0x7f26ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.593+0000 7f26feffd640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f26ec06b4d0 0x7f26ec06d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f26f4036000 tx=0x7f26f403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.597+0000 7f26e7fff640 1 -- 172.21.15.77:0/1350223054 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26f00584d0 con 0x7f27000b4460 2024-03-06T07:22:41.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.781+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f270004f330 con 0x7f27000b4460 2024-03-06T07:22:41.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.781+0000 7f26e7fff640 1 -- 172.21.15.77:0/1350223054 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f26f005c570 con 0x7f27000b4460 2024-03-06T07:22:41.783 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f26ec06b4d0 msgr2=0x7f26ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f26ec06b4d0 0x7f26ec06d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f26f4036000 tx=0x7f26f403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27000b4460 msgr2=0x7f27000b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27000b4460 0x7f27000b14c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f26f000c430 tx=0x7f26f000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 shutdown_connections 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2700162970 0x7f2700154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2700161cf0 0x7f27000b7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f26ec06b4d0 0x7f26ec06d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 --2- 172.21.15.77:0/1350223054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27000b4460 0x7f27000b14c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.783+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 >> 172.21.15.77:0/1350223054 conn(0x7f27000d3ce0 msgr2=0x7f27001611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.784+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 shutdown_connections 2024-03-06T07:22:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.784+0000 7f2705fbc640 1 -- 172.21.15.77:0/1350223054 wait complete. 2024-03-06T07:22:41.797 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:22:41.797 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:41.813 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.911+0000 7feee7181640 1 Processor -- start 2024-03-06T07:22:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feee7181640 1 -- start start 2024-03-06T07:22:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feee7181640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feee015dfe0 0x7feee00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feee7181640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feee00d29c0 0x7feee00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feee7181640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee00b3430 con 0x7feee00cf830 2024-03-06T07:22:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feee7181640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee00b31e0 con 0x7feee00d29c0 2024-03-06T07:22:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feee7181640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee00b3d60 con 0x7feee015dfe0 2024-03-06T07:22:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feee571e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feee00d29c0 0x7feee00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feedffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7feee00cf830 0x7feee00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34298/0 (socket says 172.21.15.77:34298) 2024-03-06T07:22:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.912+0000 7feedffff640 1 -- 172.21.15.77:0/1144945875 learned_addr learned my addr 172.21.15.77:0/1144945875 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.913+0000 7feee4f1d640 1 --2- 172.21.15.77:0/1144945875 >> v2:172.21.15.158:3300/0 conn(0x7feee015dfe0 0x7feee00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.913+0000 7feee571e640 1 -- 172.21.15.77:0/1144945875 >> v2:172.21.15.158:3300/0 conn(0x7feee015dfe0 msgr2=0x7feee00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.913+0000 7feee571e640 1 --2- 172.21.15.77:0/1144945875 >> v2:172.21.15.158:3300/0 conn(0x7feee015dfe0 0x7feee00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.913+0000 7feee571e640 1 -- 172.21.15.77:0/1144945875 >> v1:172.21.15.77:6789/0 conn(0x7feee00cf830 legacy=0x7feee00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.913+0000 7feee571e640 1 -- 172.21.15.77:0/1144945875 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee00b3710 con 0x7feee00d29c0 2024-03-06T07:22:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.913+0000 7feedf7fe640 1 -- 172.21.15.77:0/1144945875 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209467219 0 0) 0x7feee00b3430 con 0x7feee00cf830 2024-03-06T07:22:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.913+0000 7feee571e640 1 --2- 172.21.15.77:0/1144945875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee00d29c0 0x7feee00b41f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feee0067990 tx=0x7feed002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460c1a0ef0d683c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feedf7fe640 1 -- 172.21.15.77:0/1144945875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0035020 con 0x7feee00d29c0 2024-03-06T07:22:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feedf7fe640 1 -- 172.21.15.77:0/1144945875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed0031830 con 0x7feee00d29c0 2024-03-06T07:22:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feedf7fe640 1 -- 172.21.15.77:0/1144945875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0031c00 con 0x7feee00d29c0 2024-03-06T07:22:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feee7181640 1 -- 172.21.15.77:0/1144945875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee00d29c0 msgr2=0x7feee00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feee7181640 1 --2- 172.21.15.77:0/1144945875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee00d29c0 0x7feee00b41f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feee0067990 tx=0x7feed002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feee7181640 1 -- 172.21.15.77:0/1144945875 shutdown_connections 2024-03-06T07:22:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feee7181640 1 --2- 172.21.15.77:0/1144945875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee00d29c0 0x7feee00b41f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feee7181640 1 --2- 172.21.15.77:0/1144945875 >> v2:172.21.15.158:3300/0 conn(0x7feee015dfe0 0x7feee00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feee7181640 1 -- 172.21.15.77:0/1144945875 >> 172.21.15.77:0/1144945875 conn(0x7feee00d3cd0 msgr2=0x7feee00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feee7181640 1 -- 172.21.15.77:0/1144945875 shutdown_connections 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.914+0000 7feee7181640 1 -- 172.21.15.77:0/1144945875 wait complete. 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.915+0000 7feee7181640 1 Processor -- start 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.915+0000 7feee7181640 1 -- start start 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee7181640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feee00cf830 0x7feee0160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee7181640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feee00d29c0 0x7feee00ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee7181640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee015dfe0 0x7feee01e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee7181640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee00d1df0 con 0x7feee015dfe0 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee7181640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee00d1c70 con 0x7feee00d29c0 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee7181640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee015e3c0 con 0x7feee00cf830 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee571e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee015dfe0 0x7feee01e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee571e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee015dfe0 0x7feee01e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56754/0 (socket says 172.21.15.77:56754) 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee571e640 1 -- 172.21.15.77:0/4192704242 learned_addr learned my addr 172.21.15.77:0/4192704242 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feedffff640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feee00d29c0 0x7feee00ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.916+0000 7feee4f1d640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feee00cf830 0x7feee0160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feee571e640 1 -- 172.21.15.77:0/4192704242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feee00cf830 msgr2=0x7feee0160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feee571e640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feee00cf830 0x7feee0160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feee571e640 1 -- 172.21.15.77:0/4192704242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feee00d29c0 msgr2=0x7feee00ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feee571e640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feee00d29c0 0x7feee00ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feee571e640 1 -- 172.21.15.77:0/4192704242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0160830 con 0x7feee015dfe0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feee571e640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee015dfe0 0x7feee01e8270 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7feed0009890 tx=0x7feed00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feedd7fa640 1 -- 172.21.15.77:0/4192704242 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0035070 con 0x7feee015dfe0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feedd7fa640 1 -- 172.21.15.77:0/4192704242 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed0004230 con 0x7feee015dfe0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feedd7fa640 1 -- 172.21.15.77:0/4192704242 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed003a630 con 0x7feee015dfe0 2024-03-06T07:22:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee0161300 con 0x7feee015dfe0 2024-03-06T07:22:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.917+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee01f76c0 con 0x7feee015dfe0 2024-03-06T07:22:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.918+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee004f330 con 0x7feee015dfe0 2024-03-06T07:22:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.918+0000 7feedd7fa640 1 -- 172.21.15.77:0/4192704242 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feed0038020 con 0x7feee015dfe0 2024-03-06T07:22:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.919+0000 7feedd7fa640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feec006b4d0 0x7feec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.919+0000 7feee4f1d640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feec006b4d0 0x7feec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.919+0000 7feedd7fa640 1 -- 172.21.15.77:0/4192704242 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feed00abd60 con 0x7feee015dfe0 2024-03-06T07:22:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.919+0000 7feee4f1d640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feec006b4d0 0x7feec006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7feed8011340 tx=0x7feed800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:41.923+0000 7feedd7fa640 1 -- 172.21.15.77:0/4192704242 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feed0040070 con 0x7feee015dfe0 2024-03-06T07:22:42.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.096+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feee00d0eb0 con 0x7feee015dfe0 2024-03-06T07:22:42.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.097+0000 7feedd7fa640 1 -- 172.21.15.77:0/4192704242 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7feed007a5a0 con 0x7feee015dfe0 2024-03-06T07:22:42.099 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:42.100 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feec006b4d0 msgr2=0x7feec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feec006b4d0 0x7feec006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7feed8011340 tx=0x7feed800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee015dfe0 msgr2=0x7feee01e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee015dfe0 0x7feee01e8270 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7feed0009890 tx=0x7feed00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 shutdown_connections 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feec006b4d0 0x7feec006d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feee015dfe0 0x7feee01e8270 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feee00d29c0 0x7feee00ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 --2- 172.21.15.77:0/4192704242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feee00cf830 0x7feee0160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 >> 172.21.15.77:0/4192704242 conn(0x7feee00d3cd0 msgr2=0x7feee00d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 shutdown_connections 2024-03-06T07:22:42.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.100+0000 7feee7181640 1 -- 172.21.15.77:0/4192704242 wait complete. 2024-03-06T07:22:42.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:42.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:42.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.214+0000 7fb3c6bd6640 1 Processor -- start 2024-03-06T07:22:42.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.214+0000 7fb3c6bd6640 1 -- start start 2024-03-06T07:22:42.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3c6bd6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb3c0146b80 0x7fb3c0147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3c6bd6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c00712b0 con 0x7fb3c0146f60 2024-03-06T07:22:42.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3c6bd6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c00b5080 con 0x7fb3c0070e20 2024-03-06T07:22:42.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3c6bd6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c00ad900 con 0x7fb3c0146b80 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3bffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb3c0146f60 0x7fb3c00b4870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34314/0 (socket says 172.21.15.77:34314) 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3bffff640 1 -- 172.21.15.77:0/17868067 learned_addr learned my addr 172.21.15.77:0/17868067 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3c4972640 1 --2- 172.21.15.77:0/17868067 >> v2:172.21.15.158:3300/0 conn(0x7fb3c0146b80 0x7fb3c0147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3bf7fe640 1 -- 172.21.15.77:0/17868067 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969738072 0 0) 0x7fb3c00712b0 con 0x7fb3c0146f60 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3bf7fe640 1 -- 172.21.15.77:0/17868067 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a8003520 con 0x7fb3c0146f60 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3bf7fe640 1 -- 172.21.15.77:0/17868067 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985973320 0 0) 0x7fb3c00b5080 con 0x7fb3c0070e20 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.215+0000 7fb3bf7fe640 1 -- 172.21.15.77:0/17868067 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c00712b0 con 0x7fb3c0070e20 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.216+0000 7fb3c4972640 1 -- 172.21.15.77:0/17868067 >> v1:172.21.15.158:6789/0 conn(0x7fb3c0070e20 legacy=0x7fb3c006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.216+0000 7fb3c4972640 1 -- 172.21.15.77:0/17868067 >> v1:172.21.15.77:6789/0 conn(0x7fb3c0146f60 legacy=0x7fb3c00b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.216+0000 7fb3c4972640 1 -- 172.21.15.77:0/17868067 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c00adbe0 con 0x7fb3c0146b80 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.216+0000 7fb3c4972640 1 --2- 172.21.15.77:0/17868067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 0x7fb3c0147890 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009d90 tx=0x7fb3b802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51e6735e19a7cdc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.216+0000 7fb3bf7fe640 1 -- 172.21.15.77:0/17868067 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8039070 con 0x7fb3c0146b80 2024-03-06T07:22:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3bf7fe640 1 -- 172.21.15.77:0/17868067 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b802cd00 con 0x7fb3c0146b80 2024-03-06T07:22:42.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3bf7fe640 1 -- 172.21.15.77:0/17868067 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8034640 con 0x7fb3c0146b80 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/17868067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 msgr2=0x7fb3c0147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3c6bd6640 1 --2- 172.21.15.77:0/17868067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 0x7fb3c0147890 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8009d90 tx=0x7fb3b802bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/17868067 shutdown_connections 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3c6bd6640 1 --2- 172.21.15.77:0/17868067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 0x7fb3c0147890 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/17868067 >> 172.21.15.77:0/17868067 conn(0x7fb3c006b830 msgr2=0x7fb3c00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/17868067 shutdown_connections 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.217+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/17868067 wait complete. 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.218+0000 7fb3c6bd6640 1 Processor -- start 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.218+0000 7fb3c6bd6640 1 -- start start 2024-03-06T07:22:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c6bd6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0070e20 0x7fb3c015d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c6bd6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 0x7fb3c01e7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c6bd6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb3c01e7670 0x7fb3c0145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c6bd6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c006e9d0 con 0x7fb3c0070e20 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c6bd6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c00b5360 con 0x7fb3c0146b80 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c6bd6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c006eb50 con 0x7fb3c01e7670 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c4972640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0070e20 0x7fb3c015d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c4972640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0070e20 0x7fb3c015d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56758/0 (socket says 172.21.15.77:56758) 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c4972640 1 -- 172.21.15.77:0/3795701959 learned_addr learned my addr 172.21.15.77:0/3795701959 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3bffff640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 0x7fb3c01e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c5173640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb3c01e7670 0x7fb3c0145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c4972640 1 -- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb3c01e7670 msgr2=0x7fb3c0145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c4972640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb3c01e7670 0x7fb3c0145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c4972640 1 -- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 msgr2=0x7fb3c01e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c4972640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 0x7fb3c01e7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c4972640 1 -- 172.21.15.77:0/3795701959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c01e7bb0 con 0x7fb3c0070e20 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.219+0000 7fb3c5173640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb3c01e7670 0x7fb3c0145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.220+0000 7fb3c4972640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0070e20 0x7fb3c015d9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8000f80 tx=0x7fb3b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.220+0000 7fb3bd7fa640 1 -- 172.21.15.77:0/3795701959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8039050 con 0x7fb3c0070e20 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.220+0000 7fb3bd7fa640 1 -- 172.21.15.77:0/3795701959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b8045070 con 0x7fb3c0070e20 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.220+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c01e7e90 con 0x7fb3c0070e20 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.220+0000 7fb3bd7fa640 1 -- 172.21.15.77:0/3795701959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8009450 con 0x7fb3c0070e20 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.220+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c01e8450 con 0x7fb3c0070e20 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.221+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c00cf740 con 0x7fb3c0070e20 2024-03-06T07:22:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.221+0000 7fb3bd7fa640 1 -- 172.21.15.77:0/3795701959 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3b803e850 con 0x7fb3c0070e20 2024-03-06T07:22:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.221+0000 7fb3bd7fa640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb3ac06b500 0x7fb3ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.222+0000 7fb3bd7fa640 1 -- 172.21.15.77:0/3795701959 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3b80acf50 con 0x7fb3c0070e20 2024-03-06T07:22:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.222+0000 7fb3bffff640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb3ac06b500 0x7fb3ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.222+0000 7fb3bffff640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb3ac06b500 0x7fb3ac06d9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0144ed0 tx=0x7fb3b4007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.226+0000 7fb3bd7fa640 1 -- 172.21.15.77:0/3795701959 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3b807bef0 con 0x7fb3c0070e20 2024-03-06T07:22:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.398+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3c0068ff0 con 0x7fb3c0070e20 2024-03-06T07:22:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.399+0000 7fb3bd7fa640 1 -- 172.21.15.77:0/3795701959 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb3b807b7f0 con 0x7fb3c0070e20 2024-03-06T07:22:42.401 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.400+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb3ac06b500 msgr2=0x7fb3ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.400+0000 7fb3c6bd6640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb3ac06b500 0x7fb3ac06d9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0144ed0 tx=0x7fb3b4007d70 comp rx=0 tx=0).stop 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.400+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0070e20 msgr2=0x7fb3c015d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.400+0000 7fb3c6bd6640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0070e20 0x7fb3c015d9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8000f80 tx=0x7fb3b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.401+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 shutdown_connections 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.401+0000 7fb3c6bd6640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb3c01e7670 0x7fb3c0145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.401+0000 7fb3c6bd6640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb3c0146b80 0x7fb3c01e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.401+0000 7fb3c6bd6640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb3ac06b500 0x7fb3ac06d9c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.401+0000 7fb3c6bd6640 1 --2- 172.21.15.77:0/3795701959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3c0070e20 0x7fb3c015d9c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.401+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 >> 172.21.15.77:0/3795701959 conn(0x7fb3c006b830 msgr2=0x7fb3c0071480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.401+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 shutdown_connections 2024-03-06T07:22:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.401+0000 7fb3c6bd6640 1 -- 172.21.15.77:0/3795701959 wait complete. 2024-03-06T07:22:42.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:42.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.515+0000 7f445e7e2640 1 Processor -- start 2024-03-06T07:22:42.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.515+0000 7f445e7e2640 1 -- start start 2024-03-06T07:22:42.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f445e7e2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f445815f140 0x7f44581625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f445e7e2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4458072870 con 0x7f445815bb50 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f445e7e2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4458072b50 con 0x7f445806e2c0 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f445e7e2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4458162160 con 0x7f445815f140 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f4457fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f445815bb50 0x7f4458071fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34326/0 (socket says 172.21.15.77:34326) 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f4457fff640 1 -- 172.21.15.77:0/4171014294 learned_addr learned my addr 172.21.15.77:0/4171014294 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f44577fe640 1 --2- 172.21.15.77:0/4171014294 >> v2:172.21.15.158:3300/0 conn(0x7f445815f140 0x7f44581625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f4456ffd640 1 -- 172.21.15.77:0/4171014294 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006908623 0 0) 0x7f4458072870 con 0x7f445815bb50 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f4456ffd640 1 -- 172.21.15.77:0/4171014294 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4440003520 con 0x7f445815bb50 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f44577fe640 1 -- 172.21.15.77:0/4171014294 >> v1:172.21.15.158:6789/0 conn(0x7f445806e2c0 legacy=0x7f44580ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f44577fe640 1 -- 172.21.15.77:0/4171014294 >> v1:172.21.15.77:6789/0 conn(0x7f445815bb50 legacy=0x7f4458071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.516+0000 7f44577fe640 1 -- 172.21.15.77:0/4171014294 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458161b10 con 0x7f445815f140 2024-03-06T07:22:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.517+0000 7f44577fe640 1 --2- 172.21.15.77:0/4171014294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815f140 0x7f44581625f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097a0 tx=0x7f444c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=671d0c8eabd43eb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.517+0000 7f4456ffd640 1 -- 172.21.15.77:0/4171014294 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c035020 con 0x7f445815f140 2024-03-06T07:22:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.517+0000 7f4456ffd640 1 -- 172.21.15.77:0/4171014294 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c031830 con 0x7f445815f140 2024-03-06T07:22:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.517+0000 7f4456ffd640 1 -- 172.21.15.77:0/4171014294 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c031be0 con 0x7f445815f140 2024-03-06T07:22:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.518+0000 7f445e7e2640 1 -- 172.21.15.77:0/4171014294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815f140 msgr2=0x7f44581625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.518+0000 7f445e7e2640 1 --2- 172.21.15.77:0/4171014294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815f140 0x7f44581625f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097a0 tx=0x7f444c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.518+0000 7f445e7e2640 1 -- 172.21.15.77:0/4171014294 shutdown_connections 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.518+0000 7f445e7e2640 1 --2- 172.21.15.77:0/4171014294 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815f140 0x7f44581625f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.518+0000 7f445e7e2640 1 -- 172.21.15.77:0/4171014294 >> 172.21.15.77:0/4171014294 conn(0x7f445806b830 msgr2=0x7f44580b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.518+0000 7f445e7e2640 1 -- 172.21.15.77:0/4171014294 shutdown_connections 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.518+0000 7f445e7e2640 1 -- 172.21.15.77:0/4171014294 wait complete. 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.519+0000 7f445e7e2640 1 Processor -- start 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.519+0000 7f445e7e2640 1 -- start start 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f445e7e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445806e2c0 0x7f44581f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f445e7e2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815bb50 0x7f44581f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f445e7e2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815f140 0x7f44580cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f445e7e2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445806e890 con 0x7f445806e2c0 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f445e7e2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445806e710 con 0x7f445815bb50 2024-03-06T07:22:42.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f445e7e2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445806ea10 con 0x7f445815f140 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f4457fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445806e2c0 0x7f44581f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f4457fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445806e2c0 0x7f44581f54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56766/0 (socket says 172.21.15.77:56766) 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f4457fff640 1 -- 172.21.15.77:0/4062349134 learned_addr learned my addr 172.21.15.77:0/4062349134 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f44577fe640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815bb50 0x7f44581f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f445cd7f640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815f140 0x7f44580cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f4457fff640 1 -- 172.21.15.77:0/4062349134 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815f140 msgr2=0x7f44580cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f4457fff640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815f140 0x7f44580cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f4457fff640 1 -- 172.21.15.77:0/4062349134 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815bb50 msgr2=0x7f44581f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f4457fff640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815bb50 0x7f44581f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.520+0000 7f4457fff640 1 -- 172.21.15.77:0/4062349134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44581f7b20 con 0x7f445806e2c0 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.521+0000 7f4457fff640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445806e2c0 0x7f44581f54a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f444800c500 tx=0x7f444800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.521+0000 7f4454ff9640 1 -- 172.21.15.77:0/4062349134 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4448017070 con 0x7f445806e2c0 2024-03-06T07:22:42.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.521+0000 7f4454ff9640 1 -- 172.21.15.77:0/4062349134 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4448005ce0 con 0x7f445806e2c0 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.521+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44581f7e00 con 0x7f445806e2c0 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.521+0000 7f4454ff9640 1 -- 172.21.15.77:0/4062349134 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4448007330 con 0x7f445806e2c0 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.521+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44581f83c0 con 0x7f445806e2c0 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.522+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44581f87a0 con 0x7f445806e2c0 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.522+0000 7f4454ff9640 1 -- 172.21.15.77:0/4062349134 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f444801a070 con 0x7f445806e2c0 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.523+0000 7f4454ff9640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f444406b500 0x7f444406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.523+0000 7f44577fe640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f444406b500 0x7f444406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.523+0000 7f4454ff9640 1 -- 172.21.15.77:0/4062349134 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4448089550 con 0x7f445806e2c0 2024-03-06T07:22:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.523+0000 7f44577fe640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f444406b500 0x7f444406d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f444c036000 tx=0x7f444c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.527+0000 7f4454ff9640 1 -- 172.21.15.77:0/4062349134 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44581f87a0 con 0x7f445806e2c0 2024-03-06T07:22:42.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.696+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f445806e0b0 con 0x7f445806e2c0 2024-03-06T07:22:42.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.697+0000 7f4454ff9640 1 -- 172.21.15.77:0/4062349134 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4448057ed0 con 0x7f445806e2c0 2024-03-06T07:22:42.699 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f444406b500 msgr2=0x7f444406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f444406b500 0x7f444406d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f444c036000 tx=0x7f444c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445806e2c0 msgr2=0x7f44581f54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445806e2c0 0x7f44581f54a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f444800c500 tx=0x7f444800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 shutdown_connections 2024-03-06T07:22:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f445815f140 0x7f44580cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f445815bb50 0x7f44581f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f444406b500 0x7f444406d9c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 --2- 172.21.15.77:0/4062349134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445806e2c0 0x7f44581f54a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 >> 172.21.15.77:0/4062349134 conn(0x7f445806b830 msgr2=0x7f4458070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 shutdown_connections 2024-03-06T07:22:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.699+0000 7f445e7e2640 1 -- 172.21.15.77:0/4062349134 wait complete. 2024-03-06T07:22:42.712 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:42.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.813+0000 7f4bfef6b640 1 Processor -- start 2024-03-06T07:22:42.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.813+0000 7f4bfef6b640 1 -- start start 2024-03-06T07:22:42.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bfef6b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4bf81614d0 0x7f4bf80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bfef6b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d24b0 0x7f4bf80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bfef6b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf81618b0 con 0x7f4bf80d1910 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bfef6b640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf80cf400 con 0x7f4bf81614d0 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bfef6b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80cf6a0 con 0x7f4bf80d24b0 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bfcd07640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4bf81614d0 0x7f4bf80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bf7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4bf80d1910 0x7f4bf80d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34332/0 (socket says 172.21.15.77:34332) 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bf7fff640 1 -- 172.21.15.77:0/4052538213 learned_addr learned my addr 172.21.15.77:0/4052538213 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.814+0000 7f4bfd508640 1 --2- 172.21.15.77:0/4052538213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d24b0 0x7f4bf80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.815+0000 7f4bf77fe640 1 -- 172.21.15.77:0/4052538213 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138504828 0 0) 0x7f4bf81618b0 con 0x7f4bf80d1910 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.815+0000 7f4bfcd07640 1 -- 172.21.15.77:0/4052538213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d24b0 msgr2=0x7f4bf80b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.815+0000 7f4bfcd07640 1 --2- 172.21.15.77:0/4052538213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d24b0 0x7f4bf80b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.815+0000 7f4bfcd07640 1 -- 172.21.15.77:0/4052538213 >> v1:172.21.15.77:6789/0 conn(0x7f4bf80d1910 legacy=0x7f4bf80d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.815+0000 7f4bfcd07640 1 -- 172.21.15.77:0/4052538213 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf80cf980 con 0x7f4bf81614d0 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.815+0000 7f4bfcd07640 1 --2- 172.21.15.77:0/4052538213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 0x7f4bf80d4860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4be80097a0 tx=0x7f4be802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64c25cb515a11aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bf77fe640 1 -- 172.21.15.77:0/4052538213 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8035020 con 0x7f4bf81614d0 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bf77fe640 1 -- 172.21.15.77:0/4052538213 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be8031830 con 0x7f4bf81614d0 2024-03-06T07:22:42.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bf77fe640 1 -- 172.21.15.77:0/4052538213 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8031c00 con 0x7f4bf81614d0 2024-03-06T07:22:42.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bfef6b640 1 -- 172.21.15.77:0/4052538213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 msgr2=0x7f4bf80d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/4052538213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 0x7f4bf80d4860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4be80097a0 tx=0x7f4be802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bfef6b640 1 -- 172.21.15.77:0/4052538213 shutdown_connections 2024-03-06T07:22:42.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/4052538213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d24b0 0x7f4bf80b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/4052538213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 0x7f4bf80d4860 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bfef6b640 1 -- 172.21.15.77:0/4052538213 >> 172.21.15.77:0/4052538213 conn(0x7f4bf80d3d40 msgr2=0x7f4bf80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bfef6b640 1 -- 172.21.15.77:0/4052538213 shutdown_connections 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.816+0000 7f4bfef6b640 1 -- 172.21.15.77:0/4052538213 wait complete. 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.817+0000 7f4bfef6b640 1 Processor -- start 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.817+0000 7f4bfef6b640 1 -- start start 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bfef6b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d1910 0x7f4bf80c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bfef6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf80d24b0 0x7f4bf80c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bfef6b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 0x7f4bf806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bfef6b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80b1370 con 0x7f4bf80d24b0 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bfef6b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80b0910 con 0x7f4bf81614d0 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bfef6b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf80b14f0 con 0x7f4bf80d1910 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf80d24b0 0x7f4bf80c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bf7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf80d24b0 0x7f4bf80c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56776/0 (socket says 172.21.15.77:56776) 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bf7fff640 1 -- 172.21.15.77:0/2355159534 learned_addr learned my addr 172.21.15.77:0/2355159534 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:42.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bfcd07640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d1910 0x7f4bf80c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.818+0000 7f4bfd508640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 0x7f4bf806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf7fff640 1 -- 172.21.15.77:0/2355159534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d1910 msgr2=0x7f4bf80c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf7fff640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d1910 0x7f4bf80c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf7fff640 1 -- 172.21.15.77:0/2355159534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 msgr2=0x7f4bf806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf7fff640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 0x7f4bf806c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf7fff640 1 -- 172.21.15.77:0/2355159534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf80c7dc0 con 0x7f4bf80d24b0 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf7fff640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf80d24b0 0x7f4bf80c93a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00c500 tx=0x7f4bec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf57fa640 1 -- 172.21.15.77:0/2355159534 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec018070 con 0x7f4bf80d24b0 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf57fa640 1 -- 172.21.15.77:0/2355159534 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bec007580 con 0x7f4bf80d24b0 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf80c80a0 con 0x7f4bf80d24b0 2024-03-06T07:22:42.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bf57fa640 1 -- 172.21.15.77:0/2355159534 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec004a20 con 0x7f4bf80d24b0 2024-03-06T07:22:42.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.819+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf80c8660 con 0x7f4bf80d24b0 2024-03-06T07:22:42.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.820+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf8068ff0 con 0x7f4bf80d24b0 2024-03-06T07:22:42.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.821+0000 7f4bf57fa640 1 -- 172.21.15.77:0/2355159534 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bec01b070 con 0x7f4bf80d24b0 2024-03-06T07:22:42.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.821+0000 7f4bf57fa640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4be406b4d0 0x7f4be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.821+0000 7f4bf57fa640 1 -- 172.21.15.77:0/2355159534 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bec089410 con 0x7f4bf80d24b0 2024-03-06T07:22:42.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.821+0000 7f4bfcd07640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4be406b4d0 0x7f4be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.821+0000 7f4bfcd07640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4be406b4d0 0x7f4be406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4be8036000 tx=0x7f4be803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.825+0000 7f4bf57fa640 1 -- 172.21.15.77:0/2355159534 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bec057cd0 con 0x7f4bf80d24b0 2024-03-06T07:22:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.998+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4bf8160a40 con 0x7f4bf80d24b0 2024-03-06T07:22:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:42.999+0000 7f4bf57fa640 1 -- 172.21.15.77:0/2355159534 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4bec05bd70 con 0x7f4bf80d24b0 2024-03-06T07:22:43.001 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.000+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4be406b4d0 msgr2=0x7f4be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.000+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4be406b4d0 0x7f4be406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4be8036000 tx=0x7f4be803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf80d24b0 msgr2=0x7f4bf80c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf80d24b0 0x7f4bf80c93a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00c500 tx=0x7f4bec00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 shutdown_connections 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4be406b4d0 0x7f4be406d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf80d24b0 0x7f4bf80c93a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf81614d0 0x7f4bf806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 --2- 172.21.15.77:0/2355159534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf80d1910 0x7f4bf80c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 >> 172.21.15.77:0/2355159534 conn(0x7f4bf80d3d40 msgr2=0x7f4bf80cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 shutdown_connections 2024-03-06T07:22:43.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.001+0000 7f4bfef6b640 1 -- 172.21.15.77:0/2355159534 wait complete. 2024-03-06T07:22:43.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:43.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.113+0000 7f4fac793640 1 Processor -- start 2024-03-06T07:22:43.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.113+0000 7f4fac793640 1 -- start start 2024-03-06T07:22:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.114+0000 7f4fac793640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa406d3c0 0x7f4fa40b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.114+0000 7f4fac793640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4fa40b1850 0x7f4fa40b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.114+0000 7f4fac793640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa40af4b0 con 0x7f4fa41614a0 2024-03-06T07:22:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.114+0000 7f4fac793640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa406d800 con 0x7f4fa40b1850 2024-03-06T07:22:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.114+0000 7f4fac793640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa40af790 con 0x7f4fa406d3c0 2024-03-06T07:22:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4faad30640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4fa40b1850 0x7f4fa40b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4faa52f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4fa41614a0 0x7f4fa40b0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34338/0 (socket says 172.21.15.77:34338) 2024-03-06T07:22:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4faa52f640 1 -- 172.21.15.77:0/1387067480 learned_addr learned my addr 172.21.15.77:0/1387067480 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4fa9d2e640 1 --2- 172.21.15.77:0/1387067480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa406d3c0 0x7f4fa40b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4fa952d640 1 -- 172.21.15.77:0/1387067480 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376475769 0 0) 0x7f4fa40af4b0 con 0x7f4fa41614a0 2024-03-06T07:22:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4fa952d640 1 -- 172.21.15.77:0/1387067480 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f90003520 con 0x7f4fa41614a0 2024-03-06T07:22:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4faad30640 1 -- 172.21.15.77:0/1387067480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa406d3c0 msgr2=0x7f4fa40b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4faad30640 1 --2- 172.21.15.77:0/1387067480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa406d3c0 0x7f4fa40b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4faad30640 1 -- 172.21.15.77:0/1387067480 >> v1:172.21.15.77:6789/0 conn(0x7f4fa41614a0 legacy=0x7f4fa40b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.115+0000 7f4faad30640 1 -- 172.21.15.77:0/1387067480 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa40afa70 con 0x7f4fa40b1850 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4faad30640 1 --2- 172.21.15.77:0/1387067480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa40b1850 0x7f4fa40b5510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009d90 tx=0x7f4f9c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea371739e4e9f9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fa952d640 1 -- 172.21.15.77:0/1387067480 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c039070 con 0x7f4fa40b1850 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fa952d640 1 -- 172.21.15.77:0/1387067480 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c02cc50 con 0x7f4fa40b1850 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fa952d640 1 -- 172.21.15.77:0/1387067480 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c034440 con 0x7f4fa40b1850 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fac793640 1 -- 172.21.15.77:0/1387067480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa40b1850 msgr2=0x7f4fa40b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fac793640 1 --2- 172.21.15.77:0/1387067480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa40b1850 0x7f4fa40b5510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009d90 tx=0x7f4f9c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fac793640 1 -- 172.21.15.77:0/1387067480 shutdown_connections 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fac793640 1 --2- 172.21.15.77:0/1387067480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa40b1850 0x7f4fa40b5510 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fac793640 1 --2- 172.21.15.77:0/1387067480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa406d3c0 0x7f4fa40b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.116+0000 7f4fac793640 1 -- 172.21.15.77:0/1387067480 >> 172.21.15.77:0/1387067480 conn(0x7f4fa409a9f0 msgr2=0x7f4fa409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.117+0000 7f4fac793640 1 -- 172.21.15.77:0/1387067480 shutdown_connections 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.117+0000 7f4fac793640 1 -- 172.21.15.77:0/1387067480 wait complete. 2024-03-06T07:22:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4fac793640 1 Processor -- start 2024-03-06T07:22:43.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4fac793640 1 -- start start 2024-03-06T07:22:43.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4fac793640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa406d3c0 0x7f4fa415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4fac793640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa40b1850 0x7f4fa415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4fac793640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fa41614a0 0x7f4fa415b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4fac793640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa40b1c80 con 0x7f4fa406d3c0 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4fac793640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4161880 con 0x7f4fa41614a0 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4fac793640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa40b1e00 con 0x7f4fa40b1850 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4faa52f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa406d3c0 0x7f4fa415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4faa52f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa406d3c0 0x7f4fa415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56790/0 (socket says 172.21.15.77:56790) 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.118+0000 7f4faa52f640 1 -- 172.21.15.77:0/3335424999 learned_addr learned my addr 172.21.15.77:0/3335424999 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4fa9d2e640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa40b1850 0x7f4fa415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4faad30640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fa41614a0 0x7f4fa415b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4faa52f640 1 -- 172.21.15.77:0/3335424999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa40b1850 msgr2=0x7f4fa415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4faa52f640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa40b1850 0x7f4fa415d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4faa52f640 1 -- 172.21.15.77:0/3335424999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fa41614a0 msgr2=0x7f4fa415b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4faa52f640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fa41614a0 0x7f4fa415b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4faa52f640 1 -- 172.21.15.77:0/3335424999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa415f480 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4faa52f640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa406d3c0 0x7f4fa415ef40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000c390 tx=0x7f4fa000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4f9affd640 1 -- 172.21.15.77:0/3335424999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0017070 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4f9affd640 1 -- 172.21.15.77:0/3335424999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa0005ce0 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa415dde0 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4f9affd640 1 -- 172.21.15.77:0/3335424999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00072f0 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.119+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa415aad0 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.120+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa404f330 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.121+0000 7f4f9affd640 1 -- 172.21.15.77:0/3335424999 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fa001a070 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.121+0000 7f4f9affd640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4f8806b4d0 0x7f4f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.121+0000 7f4f9affd640 1 -- 172.21.15.77:0/3335424999 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fa0089550 con 0x7f4fa406d3c0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.121+0000 7f4fa9d2e640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4f8806b4d0 0x7f4f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.121+0000 7f4fa9d2e640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4f8806b4d0 0x7f4f8806d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40b30c0 tx=0x7f4f94009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.125+0000 7f4f9affd640 1 -- 172.21.15.77:0/3335424999 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fa0057d80 con 0x7f4fa406d3c0 2024-03-06T07:22:43.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.307+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4fa415b4b0 con 0x7f4fa406d3c0 2024-03-06T07:22:43.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.307+0000 7f4f9affd640 1 -- 172.21.15.77:0/3335424999 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4fa005be20 con 0x7f4fa406d3c0 2024-03-06T07:22:43.309 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:22:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4f8806b4d0 msgr2=0x7f4f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4f8806b4d0 0x7f4f8806d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40b30c0 tx=0x7f4f94009190 comp rx=0 tx=0).stop 2024-03-06T07:22:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa406d3c0 msgr2=0x7f4fa415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa406d3c0 0x7f4fa415ef40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000c390 tx=0x7f4fa000c850 comp rx=0 tx=0).stop 2024-03-06T07:22:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 shutdown_connections 2024-03-06T07:22:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4fa41614a0 0x7f4fa415b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4fa40b1850 0x7f4fa415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4f8806b4d0 0x7f4f8806d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 --2- 172.21.15.77:0/3335424999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fa406d3c0 0x7f4fa415ef40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 >> 172.21.15.77:0/3335424999 conn(0x7f4fa409a9f0 msgr2=0x7f4fa40b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 shutdown_connections 2024-03-06T07:22:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.309+0000 7f4fac793640 1 -- 172.21.15.77:0/3335424999 wait complete. 2024-03-06T07:22:43.322 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi158.front.sepia.ceph.com... 2024-03-06T07:22:43.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:43.322 DEBUG:teuthology.orchestra.run.smithi158:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:43.354 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:43.354 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:22:43.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:43.354 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:22:43.410 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:22:43.410 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:43.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.508+0000 7fea14365640 1 Processor -- start 2024-03-06T07:22:43.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.508+0000 7fea14365640 1 -- start start 2024-03-06T07:22:43.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea14365640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c15dfe0 0x7fea0c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea14365640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fea0c0cf830 0x7fea0c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea14365640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea0c0b3430 con 0x7fea0c0d27f0 2024-03-06T07:22:43.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea14365640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0c0d2c50 con 0x7fea0c0cf830 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea14365640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c0b31e0 con 0x7fea0c15dfe0 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea12902640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fea0c0d27f0 0x7fea0c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34350/0 (socket says 172.21.15.77:34350) 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea12902640 1 -- 172.21.15.77:0/1321117568 learned_addr learned my addr 172.21.15.77:0/1321117568 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea12101640 1 --2- 172.21.15.77:0/1321117568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c15dfe0 0x7fea0c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.509+0000 7fea11900640 1 --2- 172.21.15.77:0/1321117568 >> v2:172.21.15.158:3300/0 conn(0x7fea0c0cf830 0x7fea0c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.510+0000 7fea110ff640 1 -- 172.21.15.77:0/1321117568 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231443423 0 0) 0x7fea0c0b3430 con 0x7fea0c0d27f0 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.510+0000 7fea110ff640 1 -- 172.21.15.77:0/1321117568 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f8003520 con 0x7fea0c0d27f0 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.510+0000 7fea11900640 1 -- 172.21.15.77:0/1321117568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c15dfe0 msgr2=0x7fea0c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.510+0000 7fea11900640 1 --2- 172.21.15.77:0/1321117568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c15dfe0 0x7fea0c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.510+0000 7fea11900640 1 -- 172.21.15.77:0/1321117568 >> v1:172.21.15.77:6789/0 conn(0x7fea0c0d27f0 legacy=0x7fea0c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.510+0000 7fea11900640 1 -- 172.21.15.77:0/1321117568 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c0b3710 con 0x7fea0c0cf830 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea11900640 1 --2- 172.21.15.77:0/1321117568 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 0x7fea0c0d22b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0097a0 tx=0x7fe9fc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8bb46367b15ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea110ff640 1 -- 172.21.15.77:0/1321117568 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc035020 con 0x7fea0c0cf830 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea110ff640 1 -- 172.21.15.77:0/1321117568 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9fc031830 con 0x7fea0c0cf830 2024-03-06T07:22:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea110ff640 1 -- 172.21.15.77:0/1321117568 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc031be0 con 0x7fea0c0cf830 2024-03-06T07:22:43.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea14365640 1 -- 172.21.15.77:0/1321117568 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 msgr2=0x7fea0c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea14365640 1 --2- 172.21.15.77:0/1321117568 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 0x7fea0c0d22b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0097a0 tx=0x7fe9fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea14365640 1 -- 172.21.15.77:0/1321117568 shutdown_connections 2024-03-06T07:22:43.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea14365640 1 --2- 172.21.15.77:0/1321117568 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 0x7fea0c0d22b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea14365640 1 --2- 172.21.15.77:0/1321117568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c15dfe0 0x7fea0c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea14365640 1 -- 172.21.15.77:0/1321117568 >> 172.21.15.77:0/1321117568 conn(0x7fea0c0d3cd0 msgr2=0x7fea0c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea14365640 1 -- 172.21.15.77:0/1321117568 shutdown_connections 2024-03-06T07:22:43.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.511+0000 7fea14365640 1 -- 172.21.15.77:0/1321117568 wait complete. 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.512+0000 7fea14365640 1 Processor -- start 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.512+0000 7fea14365640 1 -- start start 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea14365640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 0x7fea0c0cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea14365640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c0d27f0 0x7fea0c14a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea14365640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea0c15dfe0 0x7fea0c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea14365640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c0d1c70 con 0x7fea0c15dfe0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea14365640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c0b3d60 con 0x7fea0c0cf830 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea14365640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c0d1df0 con 0x7fea0c0d27f0 2024-03-06T07:22:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea12902640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea0c15dfe0 0x7fea0c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea12902640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea0c15dfe0 0x7fea0c15e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56794/0 (socket says 172.21.15.77:56794) 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea12902640 1 -- 172.21.15.77:0/2871016991 learned_addr learned my addr 172.21.15.77:0/2871016991 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea12101640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 0x7fea0c0cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.513+0000 7fea11900640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c0d27f0 0x7fea0c14a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea12902640 1 -- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c0d27f0 msgr2=0x7fea0c14a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea12902640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c0d27f0 0x7fea0c14a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea12902640 1 -- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 msgr2=0x7fea0c0cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea12902640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 0x7fea0c0cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea12902640 1 -- 172.21.15.77:0/2871016991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c160670 con 0x7fea0c15dfe0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea12101640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 0x7fea0c0cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea12902640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea0c15dfe0 0x7fea0c15e450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fea0400c460 tx=0x7fea0400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea02ffd640 1 -- 172.21.15.77:0/2871016991 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04017070 con 0x7fea0c15dfe0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea02ffd640 1 -- 172.21.15.77:0/2871016991 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea04005ce0 con 0x7fea0c15dfe0 2024-03-06T07:22:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0c160950 con 0x7fea0c15dfe0 2024-03-06T07:22:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea02ffd640 1 -- 172.21.15.77:0/2871016991 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea040072b0 con 0x7fea0c15dfe0 2024-03-06T07:22:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.514+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea0c160f10 con 0x7fea0c15dfe0 2024-03-06T07:22:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.515+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0c04f330 con 0x7fea0c15dfe0 2024-03-06T07:22:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.515+0000 7fea02ffd640 1 -- 172.21.15.77:0/2871016991 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea0401a070 con 0x7fea0c15dfe0 2024-03-06T07:22:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.516+0000 7fea02ffd640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9f006b4d0 0x7fe9f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.516+0000 7fea02ffd640 1 -- 172.21.15.77:0/2871016991 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea040894b0 con 0x7fea0c15dfe0 2024-03-06T07:22:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.516+0000 7fea12101640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9f006b4d0 0x7fe9f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.516+0000 7fea12101640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9f006b4d0 0x7fe9f006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea08009fd0 tx=0x7fea0800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.520+0000 7fea02ffd640 1 -- 172.21.15.77:0/2871016991 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea04057ce0 con 0x7fea0c15dfe0 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.728+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea0c0b0a60 con 0x7fea0c15dfe0 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.728+0000 7fea02ffd640 1 -- 172.21.15.77:0/2871016991 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fea0405bd80 con 0x7fea0c15dfe0 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:43.731 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.285900+0000","modified":"2024-03-06T07:22:00.291221+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-06T07:22:43.731 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-06T07:22:43.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9f006b4d0 msgr2=0x7fe9f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9f006b4d0 0x7fe9f006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea08009fd0 tx=0x7fea0800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea0c15dfe0 msgr2=0x7fea0c15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea0c15dfe0 0x7fea0c15e450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fea0400c460 tx=0x7fea0400c920 comp rx=0 tx=0).stop 2024-03-06T07:22:43.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 shutdown_connections 2024-03-06T07:22:43.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9f006b4d0 0x7fe9f006d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea0c15dfe0 0x7fea0c15e450 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea0c0d27f0 0x7fea0c14a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 --2- 172.21.15.77:0/2871016991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea0c0cf830 0x7fea0c0cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 >> 172.21.15.77:0/2871016991 conn(0x7fea0c0d3cd0 msgr2=0x7fea0c0d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 shutdown_connections 2024-03-06T07:22:43.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.730+0000 7fea14365640 1 -- 172.21.15.77:0/2871016991 wait complete. 2024-03-06T07:22:43.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:43.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.840+0000 7f0773f13640 1 Processor -- start 2024-03-06T07:22:43.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.840+0000 7f0773f13640 1 -- start start 2024-03-06T07:22:43.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.841+0000 7f0773f13640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f076c1424f0 0x7f076c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.841+0000 7f0773f13640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f076c0cd7a0 0x7f076c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.841+0000 7f0773f13640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c0cc980 con 0x7f076c06d330 2024-03-06T07:22:43.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.841+0000 7f0773f13640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076c0cd220 con 0x7f076c0cd7a0 2024-03-06T07:22:43.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.841+0000 7f0773f13640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076c0ccc60 con 0x7f076c1424f0 2024-03-06T07:22:43.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.841+0000 7f07714ae640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f076c0cd7a0 0x7f076c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.841+0000 7f07724b0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f076c06d330 0x7f076c071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34352/0 (socket says 172.21.15.77:34352) 2024-03-06T07:22:43.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.841+0000 7f07724b0640 1 -- 172.21.15.77:0/492055541 learned_addr learned my addr 172.21.15.77:0/492055541 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.842+0000 7f0771caf640 1 --2- 172.21.15.77:0/492055541 >> v2:172.21.15.158:3300/0 conn(0x7f076c1424f0 0x7f076c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.842+0000 7f07714ae640 1 -- 172.21.15.77:0/492055541 >> v2:172.21.15.158:3300/0 conn(0x7f076c1424f0 msgr2=0x7f076c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.842+0000 7f07714ae640 1 --2- 172.21.15.77:0/492055541 >> v2:172.21.15.158:3300/0 conn(0x7f076c1424f0 0x7f076c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.842+0000 7f07714ae640 1 -- 172.21.15.77:0/492055541 >> v1:172.21.15.77:6789/0 conn(0x7f076c06d330 legacy=0x7f076c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.842+0000 7f0770cad640 1 -- 172.21.15.77:0/492055541 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166872630 0 0) 0x7f076c0cc980 con 0x7f076c06d330 2024-03-06T07:22:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.842+0000 7f07714ae640 1 -- 172.21.15.77:0/492055541 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c0ccf40 con 0x7f076c0cd7a0 2024-03-06T07:22:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.842+0000 7f0771caf640 1 --2- 172.21.15.77:0/492055541 >> v2:172.21.15.158:3300/0 conn(0x7f076c1424f0 0x7f076c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f07714ae640 1 --2- 172.21.15.77:0/492055541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c0cd7a0 0x7f076c15cef0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f076c06c6d0 tx=0x7f076802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e719616eca38887 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0770cad640 1 -- 172.21.15.77:0/492055541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768035020 con 0x7f076c0cd7a0 2024-03-06T07:22:43.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0770cad640 1 -- 172.21.15.77:0/492055541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0768031830 con 0x7f076c0cd7a0 2024-03-06T07:22:43.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0770cad640 1 -- 172.21.15.77:0/492055541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768031c00 con 0x7f076c0cd7a0 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0773f13640 1 -- 172.21.15.77:0/492055541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c0cd7a0 msgr2=0x7f076c15cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0773f13640 1 --2- 172.21.15.77:0/492055541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c0cd7a0 0x7f076c15cef0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f076c06c6d0 tx=0x7f076802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0773f13640 1 -- 172.21.15.77:0/492055541 shutdown_connections 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0773f13640 1 --2- 172.21.15.77:0/492055541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c0cd7a0 0x7f076c15cef0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0773f13640 1 --2- 172.21.15.77:0/492055541 >> v2:172.21.15.158:3300/0 conn(0x7f076c1424f0 0x7f076c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0773f13640 1 -- 172.21.15.77:0/492055541 >> 172.21.15.77:0/492055541 conn(0x7f076c0cf8a0 msgr2=0x7f076c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0773f13640 1 -- 172.21.15.77:0/492055541 shutdown_connections 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.843+0000 7f0773f13640 1 -- 172.21.15.77:0/492055541 wait complete. 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.844+0000 7f0773f13640 1 Processor -- start 2024-03-06T07:22:43.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.844+0000 7f0773f13640 1 -- start start 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f0773f13640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f076c06d330 0x7f076c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f0773f13640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076c0cd7a0 0x7f076c15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f0773f13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c1424f0 0x7f076c15ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f0773f13640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c06d920 con 0x7f076c1424f0 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f0773f13640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c06d7a0 con 0x7f076c0cd7a0 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f0773f13640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c06daa0 con 0x7f076c06d330 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f07724b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c1424f0 0x7f076c15ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f07724b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c1424f0 0x7f076c15ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56812/0 (socket says 172.21.15.77:56812) 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f07724b0640 1 -- 172.21.15.77:0/132122761 learned_addr learned my addr 172.21.15.77:0/132122761 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f07714ae640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076c0cd7a0 0x7f076c15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.845+0000 7f0771caf640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f076c06d330 0x7f076c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07724b0640 1 -- 172.21.15.77:0/132122761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f076c06d330 msgr2=0x7f076c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07724b0640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f076c06d330 0x7f076c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07724b0640 1 -- 172.21.15.77:0/132122761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076c0cd7a0 msgr2=0x7f076c15f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07724b0640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076c0cd7a0 0x7f076c15f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07724b0640 1 -- 172.21.15.77:0/132122761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c1f3be0 con 0x7f076c1424f0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07724b0640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c1424f0 0x7f076c15ff30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f076400c7a0 tx=0x7f076400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07627fc640 1 -- 172.21.15.77:0/132122761 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764018070 con 0x7f076c1424f0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07627fc640 1 -- 172.21.15.77:0/132122761 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076400a640 con 0x7f076c1424f0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c1f4980 con 0x7f076c1424f0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f07627fc640 1 -- 172.21.15.77:0/132122761 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076400aab0 con 0x7f076c1424f0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.846+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c1f4180 con 0x7f076c1424f0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.847+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076c04f330 con 0x7f076c1424f0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.847+0000 7f07627fc640 1 -- 172.21.15.77:0/132122761 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f076401b070 con 0x7f076c1424f0 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.848+0000 7f07627fc640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f075006b4d0 0x7f075006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.848+0000 7f0771caf640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f075006b4d0 0x7f075006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.848+0000 7f07627fc640 1 -- 172.21.15.77:0/132122761 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0764089790 con 0x7f076c1424f0 2024-03-06T07:22:43.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.848+0000 7f0771caf640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f075006b4d0 0x7f075006d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f076c160720 tx=0x7f075c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:43.852+0000 7f07627fc640 1 -- 172.21.15.77:0/132122761 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0764057fc0 con 0x7f076c1424f0 2024-03-06T07:22:44.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.023+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f076c0d0620 con 0x7f076c1424f0 2024-03-06T07:22:44.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.024+0000 7f07627fc640 1 -- 172.21.15.77:0/132122761 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f076405c060 con 0x7f076c1424f0 2024-03-06T07:22:44.027 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:44.027 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:44.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f075006b4d0 msgr2=0x7f075006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f075006b4d0 0x7f075006d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f076c160720 tx=0x7f075c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c1424f0 msgr2=0x7f076c15ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c1424f0 0x7f076c15ff30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f076400c7a0 tx=0x7f076400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 shutdown_connections 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f075006b4d0 0x7f075006d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f076c1424f0 0x7f076c15ff30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f076c0cd7a0 0x7f076c15f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 --2- 172.21.15.77:0/132122761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f076c06d330 0x7f076c0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 >> 172.21.15.77:0/132122761 conn(0x7f076c0cf8a0 msgr2=0x7f076c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 shutdown_connections 2024-03-06T07:22:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.026+0000 7f0773f13640 1 -- 172.21.15.77:0/132122761 wait complete. 2024-03-06T07:22:44.039 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:56.057985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:57.255143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:44.040 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.142+0000 7f2bff516640 1 Processor -- start 2024-03-06T07:22:44.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bff516640 1 -- start start 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bff516640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2bf806fd10 0x7f2bf80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bff516640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80cd4b0 con 0x7f2bf815c050 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bff516640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80cd790 con 0x7f2bf80d0af0 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bff516640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf80cda70 con 0x7f2bf806fd10 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bfcab1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2bf806fd10 0x7f2bf80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bfd2b2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2bf815c050 0x7f2bf806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34354/0 (socket says 172.21.15.77:34354) 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bfcab1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2bf806fd10 0x7f2bf80cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56826/0 (socket says 172.21.15.77:56826) 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.143+0000 7f2bfd2b2640 1 -- 172.21.15.77:0/3751962917 learned_addr learned my addr 172.21.15.77:0/3751962917 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.144+0000 7f2bfcab1640 1 -- 172.21.15.77:0/3751962917 >> v1:172.21.15.158:6789/0 conn(0x7f2bf80d0af0 legacy=0x7f2bf80ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.144+0000 7f2bfcab1640 1 -- 172.21.15.77:0/3751962917 >> v1:172.21.15.77:6789/0 conn(0x7f2bf815c050 legacy=0x7f2bf806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.144+0000 7f2bfcab1640 1 -- 172.21.15.77:0/3751962917 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf80cdd50 con 0x7f2bf806fd10 2024-03-06T07:22:44.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.144+0000 7f2bf7fff640 1 -- 172.21.15.77:0/3751962917 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451221985 0 0) 0x7f2bf80cd4b0 con 0x7f2bf815c050 2024-03-06T07:22:44.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.144+0000 7f2bfcab1640 1 --2- 172.21.15.77:0/3751962917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf80cc760 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2bec0097a0 tx=0x7f2bec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a12777b9c7d364 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bf7fff640 1 -- 172.21.15.77:0/3751962917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec035020 con 0x7f2bf806fd10 2024-03-06T07:22:44.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bf7fff640 1 -- 172.21.15.77:0/3751962917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec031830 con 0x7f2bf806fd10 2024-03-06T07:22:44.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bf7fff640 1 -- 172.21.15.77:0/3751962917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec031c00 con 0x7f2bf806fd10 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bff516640 1 -- 172.21.15.77:0/3751962917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 msgr2=0x7f2bf80cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bff516640 1 --2- 172.21.15.77:0/3751962917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf80cc760 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2bec0097a0 tx=0x7f2bec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bff516640 1 -- 172.21.15.77:0/3751962917 shutdown_connections 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bff516640 1 --2- 172.21.15.77:0/3751962917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf80cc760 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bff516640 1 -- 172.21.15.77:0/3751962917 >> 172.21.15.77:0/3751962917 conn(0x7f2bf80b3800 msgr2=0x7f2bf80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bff516640 1 -- 172.21.15.77:0/3751962917 shutdown_connections 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.145+0000 7f2bff516640 1 -- 172.21.15.77:0/3751962917 wait complete. 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.146+0000 7f2bff516640 1 Processor -- start 2024-03-06T07:22:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.146+0000 7f2bff516640 1 -- start start 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bff516640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf81507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bff516640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bf80d0af0 0x7f2bf814c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bff516640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2bf815c050 0x7f2bf80b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bff516640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf8166e50 con 0x7f2bf806fd10 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bff516640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf8166cd0 con 0x7f2bf80d0af0 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bff516640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf8166fd0 con 0x7f2bf815c050 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfd2b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf81507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfd2b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf81507c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56834/0 (socket says 172.21.15.77:56834) 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfd2b2640 1 -- 172.21.15.77:0/3214322914 learned_addr learned my addr 172.21.15.77:0/3214322914 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfcab1640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bf80d0af0 0x7f2bf814c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfdab3640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2bf815c050 0x7f2bf80b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfd2b2640 1 -- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2bf815c050 msgr2=0x7f2bf80b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfd2b2640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2bf815c050 0x7f2bf80b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfd2b2640 1 -- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bf80d0af0 msgr2=0x7f2bf814c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfd2b2640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bf80d0af0 0x7f2bf814c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfd2b2640 1 -- 172.21.15.77:0/3214322914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf814cf60 con 0x7f2bf806fd10 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.147+0000 7f2bfdab3640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2bf815c050 0x7f2bf80b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.148+0000 7f2bfd2b2640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf81507c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000c4a0 tx=0x7f2bf000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.148+0000 7f2bf5ffb640 1 -- 172.21.15.77:0/3214322914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0018070 con 0x7f2bf806fd10 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.148+0000 7f2bf5ffb640 1 -- 172.21.15.77:0/3214322914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf00074e0 con 0x7f2bf806fd10 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.148+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf81544b0 con 0x7f2bf806fd10 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.148+0000 7f2bf5ffb640 1 -- 172.21.15.77:0/3214322914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0004a20 con 0x7f2bf806fd10 2024-03-06T07:22:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.148+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf814d1d0 con 0x7f2bf806fd10 2024-03-06T07:22:44.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.149+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf809fe60 con 0x7f2bf806fd10 2024-03-06T07:22:44.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.149+0000 7f2bf5ffb640 1 -- 172.21.15.77:0/3214322914 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2bf001b070 con 0x7f2bf806fd10 2024-03-06T07:22:44.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.150+0000 7f2bf5ffb640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2be406b4d0 0x7f2be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.150+0000 7f2bf5ffb640 1 -- 172.21.15.77:0/3214322914 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bf00893e0 con 0x7f2bf806fd10 2024-03-06T07:22:44.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.150+0000 7f2bfcab1640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2be406b4d0 0x7f2be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.150+0000 7f2bfcab1640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2be406b4d0 0x7f2be406d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2bec036000 tx=0x7f2bec03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.154+0000 7f2bf5ffb640 1 -- 172.21.15.77:0/3214322914 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bf0057c10 con 0x7f2bf806fd10 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.323+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2bf80b7d50 con 0x7f2bf806fd10 2024-03-06T07:22:44.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.323+0000 7f2bf5ffb640 1 -- 172.21.15.77:0/3214322914 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2bf005bcb0 con 0x7f2bf806fd10 2024-03-06T07:22:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.325+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2be406b4d0 msgr2=0x7f2be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.325+0000 7f2bff516640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2be406b4d0 0x7f2be406d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2bec036000 tx=0x7f2bec03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.325+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 msgr2=0x7f2bf81507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.325+0000 7f2bff516640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf81507c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000c4a0 tx=0x7f2bf000c960 comp rx=0 tx=0).stop 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.326+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 shutdown_connections 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.326+0000 7f2bff516640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2bf815c050 0x7f2bf80b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.326+0000 7f2bff516640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2bf80d0af0 0x7f2bf814c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.326+0000 7f2bff516640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2be406b4d0 0x7f2be406d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.326+0000 7f2bff516640 1 --2- 172.21.15.77:0/3214322914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf806fd10 0x7f2bf81507c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.326+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 >> 172.21.15.77:0/3214322914 conn(0x7f2bf80b3800 msgr2=0x7f2bf8070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.326+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 shutdown_connections 2024-03-06T07:22:44.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.326+0000 7f2bff516640 1 -- 172.21.15.77:0/3214322914 wait complete. 2024-03-06T07:22:44.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:44.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.440+0000 7f64ff90c640 1 Processor -- start 2024-03-06T07:22:44.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.440+0000 7f64ff90c640 1 -- start start 2024-03-06T07:22:44.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64ff90c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64f80cea50 0x7f64f80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64ff90c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f64f806cab0 0x7f64f80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64ff90c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f806d030 con 0x7f64f815def0 2024-03-06T07:22:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64ff90c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f80cf1c0 con 0x7f64f80cea50 2024-03-06T07:22:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64ff90c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f80cc760 con 0x7f64f806cab0 2024-03-06T07:22:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64fcea7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64f80cea50 0x7f64f80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64fcea7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64f80cea50 0x7f64f80b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56848/0 (socket says 172.21.15.77:56848) 2024-03-06T07:22:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64fcea7640 1 -- 172.21.15.77:0/3782240875 learned_addr learned my addr 172.21.15.77:0/3782240875 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.441+0000 7f64fdea9640 1 --2- 172.21.15.77:0/3782240875 >> v2:172.21.15.158:3300/0 conn(0x7f64f806cab0 0x7f64f80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.442+0000 7f64fcea7640 1 -- 172.21.15.77:0/3782240875 >> v2:172.21.15.158:3300/0 conn(0x7f64f806cab0 msgr2=0x7f64f80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.442+0000 7f64fcea7640 1 --2- 172.21.15.77:0/3782240875 >> v2:172.21.15.158:3300/0 conn(0x7f64f806cab0 0x7f64f80ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.442+0000 7f64fcea7640 1 -- 172.21.15.77:0/3782240875 >> v1:172.21.15.158:6789/0 conn(0x7f64f815def0 legacy=0x7f64f80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.442+0000 7f64fcea7640 1 -- 172.21.15.77:0/3782240875 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f806d310 con 0x7f64f80cea50 2024-03-06T07:22:44.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.442+0000 7f64fdea9640 1 --2- 172.21.15.77:0/3782240875 >> v2:172.21.15.158:3300/0 conn(0x7f64f806cab0 0x7f64f80ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.442+0000 7f64fcea7640 1 --2- 172.21.15.77:0/3782240875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f80cea50 0x7f64f80b41b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f64f80729a0 tx=0x7f64e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cdc9164564b82d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64effff640 1 -- 172.21.15.77:0/3782240875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8034070 con 0x7f64f80cea50 2024-03-06T07:22:44.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64effff640 1 -- 172.21.15.77:0/3782240875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e8030860 con 0x7f64f80cea50 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64effff640 1 -- 172.21.15.77:0/3782240875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8030c30 con 0x7f64f80cea50 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64ff90c640 1 -- 172.21.15.77:0/3782240875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f80cea50 msgr2=0x7f64f80b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3782240875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f80cea50 0x7f64f80b41b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f64f80729a0 tx=0x7f64e802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64ff90c640 1 -- 172.21.15.77:0/3782240875 shutdown_connections 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3782240875 >> v2:172.21.15.158:3300/0 conn(0x7f64f806cab0 0x7f64f80ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3782240875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f80cea50 0x7f64f80b41b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64ff90c640 1 -- 172.21.15.77:0/3782240875 >> 172.21.15.77:0/3782240875 conn(0x7f64f80b3690 msgr2=0x7f64f80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64ff90c640 1 -- 172.21.15.77:0/3782240875 shutdown_connections 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.443+0000 7f64ff90c640 1 -- 172.21.15.77:0/3782240875 wait complete. 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.444+0000 7f64ff90c640 1 Processor -- start 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.444+0000 7f64ff90c640 1 -- start start 2024-03-06T07:22:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64ff90c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64f806cab0 0x7f64f81f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64ff90c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64f80cea50 0x7f64f8151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64ff90c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f815def0 0x7f64f80c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64ff90c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f815e2d0 con 0x7f64f815def0 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64ff90c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80cc3f0 con 0x7f64f80cea50 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64ff90c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f815e450 con 0x7f64f806cab0 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fdea9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f815def0 0x7f64f80c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fdea9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f815def0 0x7f64f80c73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56862/0 (socket says 172.21.15.77:56862) 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fdea9640 1 -- 172.21.15.77:0/3044157227 learned_addr learned my addr 172.21.15.77:0/3044157227 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fcea7640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64f80cea50 0x7f64f8151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fd6a8640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64f806cab0 0x7f64f81f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fdea9640 1 -- 172.21.15.77:0/3044157227 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64f806cab0 msgr2=0x7f64f81f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fdea9640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64f806cab0 0x7f64f81f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fdea9640 1 -- 172.21.15.77:0/3044157227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64f80cea50 msgr2=0x7f64f8151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fdea9640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64f80cea50 0x7f64f8151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.445+0000 7f64fdea9640 1 -- 172.21.15.77:0/3044157227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f80c56e0 con 0x7f64f815def0 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.446+0000 7f64fdea9640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f815def0 0x7f64f80c73d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f64f000e960 tx=0x7f64f000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.446+0000 7f64edffb640 1 -- 172.21.15.77:0/3044157227 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f0017070 con 0x7f64f815def0 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.446+0000 7f64edffb640 1 -- 172.21.15.77:0/3044157227 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f0004020 con 0x7f64f815def0 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.446+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f80c59c0 con 0x7f64f815def0 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.446+0000 7f64edffb640 1 -- 172.21.15.77:0/3044157227 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f00043f0 con 0x7f64f815def0 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.446+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f80c5f80 con 0x7f64f815def0 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.447+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f809fcf0 con 0x7f64f815def0 2024-03-06T07:22:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.447+0000 7f64edffb640 1 -- 172.21.15.77:0/3044157227 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64f000ca00 con 0x7f64f815def0 2024-03-06T07:22:44.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.448+0000 7f64edffb640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64dc06b4d0 0x7f64dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.448+0000 7f64edffb640 1 -- 172.21.15.77:0/3044157227 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64f008a3a0 con 0x7f64f815def0 2024-03-06T07:22:44.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.448+0000 7f64fd6a8640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64dc06b4d0 0x7f64dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.448+0000 7f64fd6a8640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64dc06b4d0 0x7f64dc06d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f64f4001e70 tx=0x7f64f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.452+0000 7f64edffb640 1 -- 172.21.15.77:0/3044157227 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64f0058bd0 con 0x7f64f815def0 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.624+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f64f815f2f0 con 0x7f64f815def0 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.625+0000 7f64edffb640 1 -- 172.21.15.77:0/3044157227 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f64f005cc70 con 0x7f64f815def0 2024-03-06T07:22:44.627 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64dc06b4d0 msgr2=0x7f64dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64dc06b4d0 0x7f64dc06d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f64f4001e70 tx=0x7f64f4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f815def0 msgr2=0x7f64f80c73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f815def0 0x7f64f80c73d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f64f000e960 tx=0x7f64f000ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 shutdown_connections 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64dc06b4d0 0x7f64dc06d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64f815def0 0x7f64f80c73d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64f80cea50 0x7f64f8151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 --2- 172.21.15.77:0/3044157227 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64f806cab0 0x7f64f81f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 >> 172.21.15.77:0/3044157227 conn(0x7f64f80b3690 msgr2=0x7f64f80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 shutdown_connections 2024-03-06T07:22:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.627+0000 7f64ff90c640 1 -- 172.21.15.77:0/3044157227 wait complete. 2024-03-06T07:22:44.640 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:44.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.740+0000 7fdc2a00e640 1 Processor -- start 2024-03-06T07:22:44.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.740+0000 7fdc2a00e640 1 -- start start 2024-03-06T07:22:44.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.741+0000 7fdc2a00e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdc241614d0 0x7fdc240d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.741+0000 7fdc2a00e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc240d24b0 0x7fdc240b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.741+0000 7fdc2a00e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc241618b0 con 0x7fdc240d1910 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.741+0000 7fdc2a00e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc240cf400 con 0x7fdc241614d0 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.741+0000 7fdc2a00e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc240cf6a0 con 0x7fdc240d24b0 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.741+0000 7fdc237fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdc241614d0 0x7fdc240d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.741+0000 7fdc237fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdc241614d0 0x7fdc240d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56868/0 (socket says 172.21.15.77:56868) 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.741+0000 7fdc237fe640 1 -- 172.21.15.77:0/2903746339 learned_addr learned my addr 172.21.15.77:0/2903746339 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.742+0000 7fdc23fff640 1 --2- 172.21.15.77:0/2903746339 >> v2:172.21.15.158:3300/0 conn(0x7fdc240d24b0 0x7fdc240b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.742+0000 7fdc237fe640 1 -- 172.21.15.77:0/2903746339 >> v2:172.21.15.158:3300/0 conn(0x7fdc240d24b0 msgr2=0x7fdc240b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.742+0000 7fdc237fe640 1 --2- 172.21.15.77:0/2903746339 >> v2:172.21.15.158:3300/0 conn(0x7fdc240d24b0 0x7fdc240b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.742+0000 7fdc237fe640 1 -- 172.21.15.77:0/2903746339 >> v1:172.21.15.158:6789/0 conn(0x7fdc240d1910 legacy=0x7fdc240d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.742+0000 7fdc237fe640 1 -- 172.21.15.77:0/2903746339 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc240cf980 con 0x7fdc241614d0 2024-03-06T07:22:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.742+0000 7fdc237fe640 1 --2- 172.21.15.77:0/2903746339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240d4860 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdc240b2a70 tx=0x7fdc1402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba9eca373398392 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc22ffd640 1 -- 172.21.15.77:0/2903746339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14034070 con 0x7fdc241614d0 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc22ffd640 1 -- 172.21.15.77:0/2903746339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc14030860 con 0x7fdc241614d0 2024-03-06T07:22:44.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc22ffd640 1 -- 172.21.15.77:0/2903746339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14030c30 con 0x7fdc241614d0 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2903746339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 msgr2=0x7fdc240d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2903746339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240d4860 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdc240b2a70 tx=0x7fdc1402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2903746339 shutdown_connections 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2903746339 >> v2:172.21.15.158:3300/0 conn(0x7fdc240d24b0 0x7fdc240b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2903746339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240d4860 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2903746339 >> 172.21.15.77:0/2903746339 conn(0x7fdc240d3d40 msgr2=0x7fdc240d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2903746339 shutdown_connections 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.743+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2903746339 wait complete. 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.744+0000 7fdc2a00e640 1 Processor -- start 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc2a00e640 1 -- start start 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc2a00e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc240d1910 0x7fdc241edaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc2a00e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc240d24b0 0x7fdc240b7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc2a00e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc2a00e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240b1370 con 0x7fdc241614d0 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc2a00e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240b0910 con 0x7fdc240d24b0 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc2a00e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240b14f0 con 0x7fdc240d1910 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc23fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc23fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240b7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56872/0 (socket says 172.21.15.77:56872) 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc23fff640 1 -- 172.21.15.77:0/2068151357 learned_addr learned my addr 172.21.15.77:0/2068151357 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc237fe640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc240d1910 0x7fdc241edaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.745+0000 7fdc1affd640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc240d24b0 0x7fdc240b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc23fff640 1 -- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc240d1910 msgr2=0x7fdc241edaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc23fff640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc240d1910 0x7fdc241edaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc23fff640 1 -- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc240d24b0 msgr2=0x7fdc240b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc23fff640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc240d24b0 0x7fdc240b7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc23fff640 1 -- 172.21.15.77:0/2068151357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc240b6c40 con 0x7fdc241614d0 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc1affd640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc240d24b0 0x7fdc240b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc23fff640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240b7d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009dd0 tx=0x7fdc1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc20ff9640 1 -- 172.21.15.77:0/2068151357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c018020 con 0x7fdc241614d0 2024-03-06T07:22:44.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc20ff9640 1 -- 172.21.15.77:0/2068151357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c014480 con 0x7fdc241614d0 2024-03-06T07:22:44.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc240b83c0 con 0x7fdc241614d0 2024-03-06T07:22:44.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc20ff9640 1 -- 172.21.15.77:0/2068151357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c014870 con 0x7fdc241614d0 2024-03-06T07:22:44.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.746+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc240cb340 con 0x7fdc241614d0 2024-03-06T07:22:44.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.747+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc24068ff0 con 0x7fdc241614d0 2024-03-06T07:22:44.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.747+0000 7fdc20ff9640 1 -- 172.21.15.77:0/2068151357 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc1c005d10 con 0x7fdc241614d0 2024-03-06T07:22:44.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.748+0000 7fdc20ff9640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc0406b4d0 0x7fdc0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.748+0000 7fdc20ff9640 1 -- 172.21.15.77:0/2068151357 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc1c08af30 con 0x7fdc241614d0 2024-03-06T07:22:44.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.748+0000 7fdc237fe640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc0406b4d0 0x7fdc0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.748+0000 7fdc237fe640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc0406b4d0 0x7fdc0406d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdc14036000 tx=0x7fdc1403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.752+0000 7fdc20ff9640 1 -- 172.21.15.77:0/2068151357 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc1c022030 con 0x7fdc241614d0 2024-03-06T07:22:44.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.921+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdc240d0d10 con 0x7fdc241614d0 2024-03-06T07:22:44.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.921+0000 7fdc20ff9640 1 -- 172.21.15.77:0/2068151357 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdc1c010040 con 0x7fdc241614d0 2024-03-06T07:22:44.923 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:22:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc0406b4d0 msgr2=0x7fdc0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc0406b4d0 0x7fdc0406d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdc14036000 tx=0x7fdc1403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 msgr2=0x7fdc240b7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240b7d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009dd0 tx=0x7fdc1c00c490 comp rx=0 tx=0).stop 2024-03-06T07:22:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 shutdown_connections 2024-03-06T07:22:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc0406b4d0 0x7fdc0406d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc241614d0 0x7fdc240b7d90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc240d24b0 0x7fdc240b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 --2- 172.21.15.77:0/2068151357 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc240d1910 0x7fdc241edaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 >> 172.21.15.77:0/2068151357 conn(0x7fdc240d3d40 msgr2=0x7fdc240cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 shutdown_connections 2024-03-06T07:22:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:44.923+0000 7fdc2a00e640 1 -- 172.21.15.77:0/2068151357 wait complete. 2024-03-06T07:22:44.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:45.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.033+0000 7f6adc193640 1 Processor -- start 2024-03-06T07:22:45.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.033+0000 7f6adc193640 1 -- start start 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6adc193640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6ad40d0320 0x7f6ad40b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6adc193640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ad40b54b0 0x7f6ad40cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6adc193640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad415c270 con 0x7f6ad40d0700 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6adc193640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad40b04e0 con 0x7f6ad40b54b0 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6adc193640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad40d0e90 con 0x7f6ad40d0320 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6ada730640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ad40b54b0 0x7f6ad40cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6ad9f2f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6ad40d0320 0x7f6ad40b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6ada730640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ad40b54b0 0x7f6ad40cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56874/0 (socket says 172.21.15.77:56874) 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6ad972e640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6ad40d0700 0x7f6ad40ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:41924/0 (socket says 172.21.15.77:41924) 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6ada730640 1 -- 172.21.15.77:0/1452348507 learned_addr learned my addr 172.21.15.77:0/1452348507 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6ada730640 1 -- 172.21.15.77:0/1452348507 >> v2:172.21.15.158:3300/0 conn(0x7f6ad40d0320 msgr2=0x7f6ad40b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6ada730640 1 --2- 172.21.15.77:0/1452348507 >> v2:172.21.15.158:3300/0 conn(0x7f6ad40d0320 0x7f6ad40b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.034+0000 7f6ada730640 1 -- 172.21.15.77:0/1452348507 >> v1:172.21.15.158:6789/0 conn(0x7f6ad40d0700 legacy=0x7f6ad40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.035+0000 7f6ada730640 1 -- 172.21.15.77:0/1452348507 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad40b5ae0 con 0x7f6ad40b54b0 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.035+0000 7f6ada730640 1 --2- 172.21.15.77:0/1452348507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad40cd1f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40715b0 tx=0x7f6acc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1381fadf26f2f71f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.035+0000 7f6ad8f2d640 1 -- 172.21.15.77:0/1452348507 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc02cb20 con 0x7f6ad40b54b0 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.035+0000 7f6ad8f2d640 1 -- 172.21.15.77:0/1452348507 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc02ccc0 con 0x7f6ad40b54b0 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.035+0000 7f6ad8f2d640 1 -- 172.21.15.77:0/1452348507 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc034610 con 0x7f6ad40b54b0 2024-03-06T07:22:45.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.036+0000 7f6adc193640 1 -- 172.21.15.77:0/1452348507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 msgr2=0x7f6ad40cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.036+0000 7f6adc193640 1 --2- 172.21.15.77:0/1452348507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad40cd1f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40715b0 tx=0x7f6acc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:45.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.036+0000 7f6adc193640 1 -- 172.21.15.77:0/1452348507 shutdown_connections 2024-03-06T07:22:45.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.036+0000 7f6adc193640 1 --2- 172.21.15.77:0/1452348507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad40cd1f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.036+0000 7f6adc193640 1 --2- 172.21.15.77:0/1452348507 >> v2:172.21.15.158:3300/0 conn(0x7f6ad40d0320 0x7f6ad40b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.036+0000 7f6adc193640 1 -- 172.21.15.77:0/1452348507 >> 172.21.15.77:0/1452348507 conn(0x7f6ad40724d0 msgr2=0x7f6ad40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.036+0000 7f6adc193640 1 -- 172.21.15.77:0/1452348507 shutdown_connections 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.036+0000 7f6adc193640 1 -- 172.21.15.77:0/1452348507 wait complete. 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.037+0000 7f6adc193640 1 Processor -- start 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.037+0000 7f6adc193640 1 -- start start 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6adc193640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad414c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6adc193640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ad40d0320 0x7f6ad414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6adc193640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ad40d0700 0x7f6ad414fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6adc193640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40b60a0 con 0x7f6ad40b54b0 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6adc193640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40b0170 con 0x7f6ad40d0700 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6adc193640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40ca790 con 0x7f6ad40d0320 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad414c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad414c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56884/0 (socket says 172.21.15.77:56884) 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 -- 172.21.15.77:0/2180033592 learned_addr learned my addr 172.21.15.77:0/2180033592 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ada730640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ad40d0700 0x7f6ad414fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad972e640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ad40d0320 0x7f6ad414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 -- 172.21.15.77:0/2180033592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ad40d0320 msgr2=0x7f6ad414f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ad40d0320 0x7f6ad414f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 -- 172.21.15.77:0/2180033592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ad40d0700 msgr2=0x7f6ad414fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ad40d0700 0x7f6ad414fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 -- 172.21.15.77:0/2180033592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad409b1a0 con 0x7f6ad40b54b0 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.038+0000 7f6ad9f2f640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad414c530 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6ad000e9b0 tx=0x7f6ad000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.039+0000 7f6aca7fc640 1 -- 172.21.15.77:0/2180033592 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0017070 con 0x7f6ad40b54b0 2024-03-06T07:22:45.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.039+0000 7f6aca7fc640 1 -- 172.21.15.77:0/2180033592 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad000c9a0 con 0x7f6ad40b54b0 2024-03-06T07:22:45.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.039+0000 7f6aca7fc640 1 -- 172.21.15.77:0/2180033592 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad000cd90 con 0x7f6ad40b54b0 2024-03-06T07:22:45.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.039+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad409b480 con 0x7f6ad40b54b0 2024-03-06T07:22:45.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.039+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad4156010 con 0x7f6ad40b54b0 2024-03-06T07:22:45.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.039+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad404f330 con 0x7f6ad40b54b0 2024-03-06T07:22:45.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.040+0000 7f6aca7fc640 1 -- 172.21.15.77:0/2180033592 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ad001a070 con 0x7f6ad40b54b0 2024-03-06T07:22:45.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.040+0000 7f6aca7fc640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ab806b4d0 0x7f6ab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.040+0000 7f6aca7fc640 1 -- 172.21.15.77:0/2180033592 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ad008a4a0 con 0x7f6ad40b54b0 2024-03-06T07:22:45.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.040+0000 7f6ad972e640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ab806b4d0 0x7f6ab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.041+0000 7f6ad972e640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ab806b4d0 0x7f6ab806d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6ad41f5220 tx=0x7f6ac4007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.045+0000 7f6aca7fc640 1 -- 172.21.15.77:0/2180033592 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ad0058d60 con 0x7f6ad40b54b0 2024-03-06T07:22:45.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.251+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ad4156410 con 0x7f6ad40b54b0 2024-03-06T07:22:45.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.251+0000 7f6aca7fc640 1 -- 172.21.15.77:0/2180033592 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f6ad005ce00 con 0x7f6ad40b54b0 2024-03-06T07:22:45.253 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:45.253 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.285900+0000","modified":"2024-03-06T07:22:00.291221+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-06T07:22:45.254 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-06T07:22:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ab806b4d0 msgr2=0x7f6ab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ab806b4d0 0x7f6ab806d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6ad41f5220 tx=0x7f6ac4007e80 comp rx=0 tx=0).stop 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 msgr2=0x7f6ad414c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad414c530 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6ad000e9b0 tx=0x7f6ad000ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 shutdown_connections 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ad40d0700 0x7f6ad414fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ad40d0320 0x7f6ad414f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ab806b4d0 0x7f6ab806d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 --2- 172.21.15.77:0/2180033592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ad40b54b0 0x7f6ad414c530 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 >> 172.21.15.77:0/2180033592 conn(0x7f6ad40724d0 msgr2=0x7f6ad40af100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 shutdown_connections 2024-03-06T07:22:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.253+0000 7f6adc193640 1 -- 172.21.15.77:0/2180033592 wait complete. 2024-03-06T07:22:45.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:45.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.363+0000 7f7b749f7640 1 Processor -- start 2024-03-06T07:22:45.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.363+0000 7f7b749f7640 1 -- start start 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.364+0000 7f7b749f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b700b4f70 0x7f7b700b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.364+0000 7f7b749f7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b700b5890 con 0x7f7b70066d90 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.364+0000 7f7b749f7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b700cd450 con 0x7f7b7006c6d0 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.364+0000 7f7b749f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7015e9f0 con 0x7f7b700b4f70 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.364+0000 7f7b6e59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b700b4f70 0x7f7b700b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.364+0000 7f7b6ed9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7b70066d90 0x7f7b700ccb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34362/0 (socket says 172.21.15.77:34362) 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.364+0000 7f7b6ed9d640 1 -- 172.21.15.77:0/2258397755 learned_addr learned my addr 172.21.15.77:0/2258397755 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.365+0000 7f7b6e59c640 1 -- 172.21.15.77:0/2258397755 >> v1:172.21.15.158:6789/0 conn(0x7f7b7006c6d0 legacy=0x7f7b7015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.365+0000 7f7b6e59c640 1 -- 172.21.15.77:0/2258397755 >> v1:172.21.15.77:6789/0 conn(0x7f7b70066d90 legacy=0x7f7b700ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.365+0000 7f7b6e59c640 1 -- 172.21.15.77:0/2258397755 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b700cd730 con 0x7f7b700b4f70 2024-03-06T07:22:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.365+0000 7f7b6d59a640 1 -- 172.21.15.77:0/2258397755 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174665107 0 0) 0x7f7b700b5890 con 0x7f7b70066d90 2024-03-06T07:22:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.365+0000 7f7b6e59c640 1 --2- 172.21.15.77:0/2258397755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b700b4f70 0x7f7b700b5350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009d90 tx=0x7f7b6002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20bbcade0f18e30c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.365+0000 7f7b6d59a640 1 -- 172.21.15.77:0/2258397755 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60039070 con 0x7f7b700b4f70 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b6d59a640 1 -- 172.21.15.77:0/2258397755 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6002cd00 con 0x7f7b700b4f70 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b6d59a640 1 -- 172.21.15.77:0/2258397755 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60034660 con 0x7f7b700b4f70 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b749f7640 1 -- 172.21.15.77:0/2258397755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b700b4f70 msgr2=0x7f7b700b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b749f7640 1 --2- 172.21.15.77:0/2258397755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b700b4f70 0x7f7b700b5350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009d90 tx=0x7f7b6002bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b749f7640 1 -- 172.21.15.77:0/2258397755 shutdown_connections 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b749f7640 1 --2- 172.21.15.77:0/2258397755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b700b4f70 0x7f7b700b5350 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b749f7640 1 -- 172.21.15.77:0/2258397755 >> 172.21.15.77:0/2258397755 conn(0x7f7b700724d0 msgr2=0x7f7b700728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b749f7640 1 -- 172.21.15.77:0/2258397755 shutdown_connections 2024-03-06T07:22:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.366+0000 7f7b749f7640 1 -- 172.21.15.77:0/2258397755 wait complete. 2024-03-06T07:22:45.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.367+0000 7f7b749f7640 1 Processor -- start 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.367+0000 7f7b749f7640 1 -- start start 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b749f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b70066d90 0x7f7b701f7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b749f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b700b4f70 0x7f7b700adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b749f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b70159f40 0x7f7b700ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b749f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70161ad0 con 0x7f7b70159f40 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b749f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b700cda10 con 0x7f7b70066d90 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b749f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70161c50 con 0x7f7b700b4f70 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b6ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b70159f40 0x7f7b700ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b6ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b70159f40 0x7f7b700ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56896/0 (socket says 172.21.15.77:56896) 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b6ed9d640 1 -- 172.21.15.77:0/3677762370 learned_addr learned my addr 172.21.15.77:0/3677762370 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b6e59c640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b70066d90 0x7f7b701f7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.368+0000 7f7b6dd9b640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b700b4f70 0x7f7b700adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b6ed9d640 1 -- 172.21.15.77:0/3677762370 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b700b4f70 msgr2=0x7f7b700adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b6ed9d640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b700b4f70 0x7f7b700adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b6ed9d640 1 -- 172.21.15.77:0/3677762370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b70066d90 msgr2=0x7f7b701f7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b6ed9d640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b70066d90 0x7f7b701f7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b6ed9d640 1 -- 172.21.15.77:0/3677762370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b701599c0 con 0x7f7b70159f40 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b6ed9d640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b70159f40 0x7f7b700ae040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800c480 tx=0x7f7b6800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b5effd640 1 -- 172.21.15.77:0/3677762370 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68018070 con 0x7f7b70159f40 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b5effd640 1 -- 172.21.15.77:0/3677762370 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b680074e0 con 0x7f7b70159f40 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b5effd640 1 -- 172.21.15.77:0/3677762370 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68004a20 con 0x7f7b70159f40 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7015b530 con 0x7f7b70159f40 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.369+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7015a770 con 0x7f7b70159f40 2024-03-06T07:22:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.370+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7004f330 con 0x7f7b70159f40 2024-03-06T07:22:45.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.370+0000 7f7b5effd640 1 -- 172.21.15.77:0/3677762370 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b6801b070 con 0x7f7b70159f40 2024-03-06T07:22:45.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.371+0000 7f7b5effd640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b5006b4d0 0x7f7b5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.371+0000 7f7b5effd640 1 -- 172.21.15.77:0/3677762370 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b680893a0 con 0x7f7b70159f40 2024-03-06T07:22:45.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.371+0000 7f7b6e59c640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b5006b4d0 0x7f7b5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.371+0000 7f7b6e59c640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b5006b4d0 0x7f7b5006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7b60036000 tx=0x7f7b60030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.375+0000 7f7b5effd640 1 -- 172.21.15.77:0/3677762370 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b68057c60 con 0x7f7b70159f40 2024-03-06T07:22:45.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.555+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b700b1860 con 0x7f7b70159f40 2024-03-06T07:22:45.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.556+0000 7f7b5effd640 1 -- 172.21.15.77:0/3677762370 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f7b6805bd00 con 0x7f7b70159f40 2024-03-06T07:22:45.558 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:45.559 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:21:59.938887+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:45.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b5006b4d0 msgr2=0x7f7b5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b5006b4d0 0x7f7b5006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7b60036000 tx=0x7f7b60030000 comp rx=0 tx=0).stop 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b70159f40 msgr2=0x7f7b700ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b70159f40 0x7f7b700ae040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800c480 tx=0x7f7b6800c940 comp rx=0 tx=0).stop 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 shutdown_connections 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b5006b4d0 0x7f7b5006d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b70159f40 0x7f7b700ae040 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b700b4f70 0x7f7b700adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 --2- 172.21.15.77:0/3677762370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b70066d90 0x7f7b701f7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 >> 172.21.15.77:0/3677762370 conn(0x7f7b700724d0 msgr2=0x7f7b7006d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 shutdown_connections 2024-03-06T07:22:45.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.558+0000 7f7b749f7640 1 -- 172.21.15.77:0/3677762370 wait complete. 2024-03-06T07:22:45.572 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:22:45.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.670+0000 7f9cbfc37640 1 Processor -- start 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.670+0000 7f9cbfc37640 1 -- start start 2024-03-06T07:22:45.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbfc37640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb806c6d0 0x7f9cb815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbfc37640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9cb8066d90 0x7f9cb80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbfc37640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb80cd450 con 0x7f9cb80b4f70 2024-03-06T07:22:45.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbfc37640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb815e9f0 con 0x7f9cb8066d90 2024-03-06T07:22:45.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbfc37640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb80cda10 con 0x7f9cb806c6d0 2024-03-06T07:22:45.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbe1d4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9cb8066d90 0x7f9cb80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbd9d3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9cb80b4f70 0x7f9cb80b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34370/0 (socket says 172.21.15.77:34370) 2024-03-06T07:22:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbd9d3640 1 -- 172.21.15.77:0/3873140997 learned_addr learned my addr 172.21.15.77:0/3873140997 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbd1d2640 1 --2- 172.21.15.77:0/3873140997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb806c6d0 0x7f9cb815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbe1d4640 1 -- 172.21.15.77:0/3873140997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb806c6d0 msgr2=0x7f9cb815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbe1d4640 1 --2- 172.21.15.77:0/3873140997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb806c6d0 0x7f9cb815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbe1d4640 1 -- 172.21.15.77:0/3873140997 >> v1:172.21.15.77:6789/0 conn(0x7f9cb80b4f70 legacy=0x7f9cb80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbe1d4640 1 -- 172.21.15.77:0/3873140997 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80cd730 con 0x7f9cb8066d90 2024-03-06T07:22:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.671+0000 7f9cbc9d1640 1 -- 172.21.15.77:0/3873140997 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276071598 0 0) 0x7f9cb80cd450 con 0x7f9cb80b4f70 2024-03-06T07:22:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbe1d4640 1 --2- 172.21.15.77:0/3873140997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8066d90 0x7f9cb80ccb50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009d90 tx=0x7f9cb002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a4fdfa87a8eda9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbc9d1640 1 -- 172.21.15.77:0/3873140997 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0039070 con 0x7f9cb8066d90 2024-03-06T07:22:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbc9d1640 1 -- 172.21.15.77:0/3873140997 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb002cc50 con 0x7f9cb8066d90 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbc9d1640 1 -- 172.21.15.77:0/3873140997 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0034440 con 0x7f9cb8066d90 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbfc37640 1 -- 172.21.15.77:0/3873140997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8066d90 msgr2=0x7f9cb80ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/3873140997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8066d90 0x7f9cb80ccb50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009d90 tx=0x7f9cb002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbfc37640 1 -- 172.21.15.77:0/3873140997 shutdown_connections 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/3873140997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8066d90 0x7f9cb80ccb50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/3873140997 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb806c6d0 0x7f9cb815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.672+0000 7f9cbfc37640 1 -- 172.21.15.77:0/3873140997 >> 172.21.15.77:0/3873140997 conn(0x7f9cb80724d0 msgr2=0x7f9cb80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.673+0000 7f9cbfc37640 1 -- 172.21.15.77:0/3873140997 shutdown_connections 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.673+0000 7f9cbfc37640 1 -- 172.21.15.77:0/3873140997 wait complete. 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.674+0000 7f9cbfc37640 1 Processor -- start 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.674+0000 7f9cbfc37640 1 -- start start 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.674+0000 7f9cbfc37640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cb8066d90 0x7f9cb80ce070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.674+0000 7f9cbfc37640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb80b4f70 0x7f9cb806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.674+0000 7f9cbfc37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8159260 0x7f9cb8160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.674+0000 7f9cbfc37640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb80cd090 con 0x7f9cb8159260 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.674+0000 7f9cbfc37640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb815f3c0 con 0x7f9cb8066d90 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.674+0000 7f9cbfc37640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb80cd210 con 0x7f9cb80b4f70 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8159260 0x7f9cb8160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8159260 0x7f9cb8160a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56908/0 (socket says 172.21.15.77:56908) 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 -- 172.21.15.77:0/2982691583 learned_addr learned my addr 172.21.15.77:0/2982691583 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbd9d3640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cb8066d90 0x7f9cb80ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbd1d2640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb80b4f70 0x7f9cb806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 -- 172.21.15.77:0/2982691583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb80b4f70 msgr2=0x7f9cb806dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb80b4f70 0x7f9cb806dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 -- 172.21.15.77:0/2982691583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cb8066d90 msgr2=0x7f9cb80ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cb8066d90 0x7f9cb80ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 -- 172.21.15.77:0/2982691583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb815f9d0 con 0x7f9cb8159260 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbe1d4640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8159260 0x7f9cb8160a70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009d90 tx=0x7f9cb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cae7fc640 1 -- 172.21.15.77:0/2982691583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0039050 con 0x7f9cb8159260 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cae7fc640 1 -- 172.21.15.77:0/2982691583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0045070 con 0x7f9cb8159260 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cae7fc640 1 -- 172.21.15.77:0/2982691583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0009450 con 0x7f9cb8159260 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb815fcb0 con 0x7f9cb8159260 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.675+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb8160270 con 0x7f9cb8159260 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.676+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb804f330 con 0x7f9cb8159260 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.677+0000 7f9cae7fc640 1 -- 172.21.15.77:0/2982691583 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cb003e610 con 0x7f9cb8159260 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.677+0000 7f9cae7fc640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.677+0000 7f9cae7fc640 1 -- 172.21.15.77:0/2982691583 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cb00aca40 con 0x7f9cb8159260 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.677+0000 7f9cbd9d3640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.677+0000 7f9cbd9d3640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4007fd0 tx=0x7f9cb4007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.681+0000 7f9cae7fc640 1 -- 172.21.15.77:0/2982691583 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cb007b300 con 0x7f9cb8159260 2024-03-06T07:22:45.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.887+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cb8067590 con 0x7f9cb8159260 2024-03-06T07:22:45.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.887+0000 7f9cae7fc640 1 -- 172.21.15.77:0/2982691583 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f9cb00ace20 con 0x7f9cb8159260 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.285900+0000","modified":"2024-03-06T07:22:00.291221+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-06T07:22:45.890 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-06T07:22:45.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c9c06b4d0 msgr2=0x7f9c9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4007fd0 tx=0x7f9cb4007de0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8159260 msgr2=0x7f9cb8160a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8159260 0x7f9cb8160a70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009d90 tx=0x7f9cb0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 shutdown_connections 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cb8159260 0x7f9cb8160a70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9cb80b4f70 0x7f9cb806dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 --2- 172.21.15.77:0/2982691583 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9cb8066d90 0x7f9cb80ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 >> 172.21.15.77:0/2982691583 conn(0x7f9cb80724d0 msgr2=0x7f9cb80728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 shutdown_connections 2024-03-06T07:22:45.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:45.889+0000 7f9cbfc37640 1 -- 172.21.15.77:0/2982691583 wait complete. 2024-03-06T07:22:45.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:46.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.002+0000 7f6877cfb640 1 Processor -- start 2024-03-06T07:22:46.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.003+0000 7f6877cfb640 1 -- start start 2024-03-06T07:22:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.003+0000 7f6877cfb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f687015dfe0 0x7f68700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.003+0000 7f6877cfb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68700d29c0 0x7f68700b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.003+0000 7f6877cfb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68700b3430 con 0x7f68700cf830 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.003+0000 7f6877cfb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700b31e0 con 0x7f68700d29c0 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.003+0000 7f6877cfb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700b3d60 con 0x7f687015dfe0 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.003+0000 7f6876298640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68700d29c0 0x7f68700b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.004+0000 7f6876298640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68700d29c0 0x7f68700b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56910/0 (socket says 172.21.15.77:56910) 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.004+0000 7f6876298640 1 -- 172.21.15.77:0/1398837786 learned_addr learned my addr 172.21.15.77:0/1398837786 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.004+0000 7f6875a97640 1 --2- 172.21.15.77:0/1398837786 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f687015dfe0 0x7f68700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.004+0000 7f6876298640 1 -- 172.21.15.77:0/1398837786 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f687015dfe0 msgr2=0x7f68700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.004+0000 7f6876298640 1 --2- 172.21.15.77:0/1398837786 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f687015dfe0 0x7f68700cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.004+0000 7f6876298640 1 -- 172.21.15.77:0/1398837786 >> v1:172.21.15.158:6789/0 conn(0x7f68700cf830 legacy=0x7f68700d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.004+0000 7f6874a95640 1 -- 172.21.15.77:0/1398837786 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429395339 0 0) 0x7f68700b3430 con 0x7f68700cf830 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.004+0000 7f6876298640 1 -- 172.21.15.77:0/1398837786 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700b3710 con 0x7f68700d29c0 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6876298640 1 --2- 172.21.15.77:0/1398837786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f68700b41f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f686c009870 tx=0x7f686c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af186c541c810d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6874a95640 1 -- 172.21.15.77:0/1398837786 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c034070 con 0x7f68700d29c0 2024-03-06T07:22:46.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6874a95640 1 -- 172.21.15.77:0/1398837786 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c030840 con 0x7f68700d29c0 2024-03-06T07:22:46.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6874a95640 1 -- 172.21.15.77:0/1398837786 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c030c10 con 0x7f68700d29c0 2024-03-06T07:22:46.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6877cfb640 1 -- 172.21.15.77:0/1398837786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 msgr2=0x7f68700b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6877cfb640 1 --2- 172.21.15.77:0/1398837786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f68700b41f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f686c009870 tx=0x7f686c02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:46.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6877cfb640 1 -- 172.21.15.77:0/1398837786 shutdown_connections 2024-03-06T07:22:46.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6877cfb640 1 --2- 172.21.15.77:0/1398837786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f68700b41f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6877cfb640 1 --2- 172.21.15.77:0/1398837786 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f687015dfe0 0x7f68700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6877cfb640 1 -- 172.21.15.77:0/1398837786 >> 172.21.15.77:0/1398837786 conn(0x7f68700d3cd0 msgr2=0x7f68700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6877cfb640 1 -- 172.21.15.77:0/1398837786 shutdown_connections 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.005+0000 7f6877cfb640 1 -- 172.21.15.77:0/1398837786 wait complete. 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.006+0000 7f6877cfb640 1 Processor -- start 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.006+0000 7f6877cfb640 1 -- start start 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6877cfb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f68700cf830 0x7f68700cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6877cfb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f687014f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6877cfb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f687015dfe0 0x7f68700cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6877cfb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700d1df0 con 0x7f68700d29c0 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6877cfb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700d1c70 con 0x7f687015dfe0 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6877cfb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687015e3c0 con 0x7f68700cf830 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6875296640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f687014f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6875296640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f687014f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56920/0 (socket says 172.21.15.77:56920) 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6875296640 1 -- 172.21.15.77:0/2552647337 learned_addr learned my addr 172.21.15.77:0/2552647337 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6876298640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f687015dfe0 0x7f68700cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.007+0000 7f6875a97640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f68700cf830 0x7f68700cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6875296640 1 -- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f68700cf830 msgr2=0x7f68700cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6875296640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f68700cf830 0x7f68700cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6875296640 1 -- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f687015dfe0 msgr2=0x7f68700cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6875296640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f687015dfe0 0x7f68700cdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6875296640 1 -- 172.21.15.77:0/2552647337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68701f7350 con 0x7f68700d29c0 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6875a97640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f68700cf830 0x7f68700cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6875296640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f687014f090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f686000c460 tx=0x7f686000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f68667fc640 1 -- 172.21.15.77:0/2552647337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860005480 con 0x7f68700d29c0 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f68667fc640 1 -- 172.21.15.77:0/2552647337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68600035f0 con 0x7f68700d29c0 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68700cb3d0 con 0x7f68700d29c0 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f68667fc640 1 -- 172.21.15.77:0/2552647337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68600039e0 con 0x7f68700d29c0 2024-03-06T07:22:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.008+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68701f78d0 con 0x7f68700d29c0 2024-03-06T07:22:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.009+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68700b3360 con 0x7f68700d29c0 2024-03-06T07:22:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.009+0000 7f68667fc640 1 -- 172.21.15.77:0/2552647337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6860003b80 con 0x7f68700d29c0 2024-03-06T07:22:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.010+0000 7f68667fc640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f685406b6a0 0x7f685406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.010+0000 7f6875a97640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f685406b6a0 0x7f685406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.010+0000 7f68667fc640 1 -- 172.21.15.77:0/2552647337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6860089d10 con 0x7f68700d29c0 2024-03-06T07:22:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.010+0000 7f6875a97640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f685406b6a0 0x7f685406db60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6868009fd0 tx=0x7f686800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.014+0000 7f68667fc640 1 -- 172.21.15.77:0/2552647337 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68700b3360 con 0x7f68700d29c0 2024-03-06T07:22:46.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.222+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f687004f330 con 0x7f68700d29c0 2024-03-06T07:22:46.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.222+0000 7f68667fc640 1 -- 172.21.15.77:0/2552647337 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f6860058b90 con 0x7f68700d29c0 2024-03-06T07:22:46.224 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:46.224 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/516972049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":516972049},{"type":"v1","addr":"172.21.15.77:6836","nonce":516972049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2274067396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2274067396},{"type":"v1","addr":"172.21.15.77:6837","nonce":2274067396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/4098688505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4098688505},{"type":"v1","addr":"172.21.15.158:6833","nonce":4098688505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:58.285900+0000","modified":"2024-03-06T07:22:00.291221+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/3488131752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3488131752},{"type":"v1","addr":"172.21.15.158:6835","nonce":3488131752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-06T07:22:46.225 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-06T07:22:46.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f685406b6a0 msgr2=0x7f685406db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f685406b6a0 0x7f685406db60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6868009fd0 tx=0x7f686800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 msgr2=0x7f687014f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f687014f090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f686000c460 tx=0x7f686000c920 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 shutdown_connections 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f685406b6a0 0x7f685406db60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68700d29c0 0x7f687014f090 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f687015dfe0 0x7f68700cdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 --2- 172.21.15.77:0/2552647337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f68700cf830 0x7f68700cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 >> 172.21.15.77:0/2552647337 conn(0x7f68700d3cd0 msgr2=0x7f68700b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 shutdown_connections 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.224+0000 7f6877cfb640 1 -- 172.21.15.77:0/2552647337 wait complete. 2024-03-06T07:22:46.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:46.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.336+0000 7fecf9f16640 1 Processor -- start 2024-03-06T07:22:46.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.336+0000 7fecf9f16640 1 -- start start 2024-03-06T07:22:46.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.336+0000 7fecf9f16640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fecf40b4170 0x7fecf40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.336+0000 7fecf9f16640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf40d25d0 con 0x7fecf40d0a80 2024-03-06T07:22:46.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.336+0000 7fecf9f16640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf40d2870 con 0x7fecf40d4760 2024-03-06T07:22:46.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.336+0000 7fecf9f16640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf40d4bf0 con 0x7fecf40b4170 2024-03-06T07:22:46.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.337+0000 7fecf37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fecf40b4170 0x7fecf40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.337+0000 7fecf3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fecf40d0a80 0x7fecf40cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34374/0 (socket says 172.21.15.77:34374) 2024-03-06T07:22:46.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.337+0000 7fecf3fff640 1 -- 172.21.15.77:0/2186127525 learned_addr learned my addr 172.21.15.77:0/2186127525 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:46.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.337+0000 7fecf37fe640 1 -- 172.21.15.77:0/2186127525 >> v1:172.21.15.158:6789/0 conn(0x7fecf40d4760 legacy=0x7fecf40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.337+0000 7fecf37fe640 1 -- 172.21.15.77:0/2186127525 >> v1:172.21.15.77:6789/0 conn(0x7fecf40d0a80 legacy=0x7fecf40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.337+0000 7fecf37fe640 1 -- 172.21.15.77:0/2186127525 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40d2b50 con 0x7fecf40b4170 2024-03-06T07:22:46.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf37fe640 1 --2- 172.21.15.77:0/2186127525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40b4170 0x7fecf40b6a90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fece40097a0 tx=0x7fece402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da9544f99196a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf2ffd640 1 -- 172.21.15.77:0/2186127525 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4034070 con 0x7fecf40b4170 2024-03-06T07:22:46.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf2ffd640 1 -- 172.21.15.77:0/2186127525 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece4030610 con 0x7fecf40b4170 2024-03-06T07:22:46.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf2ffd640 1 -- 172.21.15.77:0/2186127525 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40309e0 con 0x7fecf40b4170 2024-03-06T07:22:46.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf9f16640 1 -- 172.21.15.77:0/2186127525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40b4170 msgr2=0x7fecf40b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf9f16640 1 --2- 172.21.15.77:0/2186127525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40b4170 0x7fecf40b6a90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fece40097a0 tx=0x7fece402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:46.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf9f16640 1 -- 172.21.15.77:0/2186127525 shutdown_connections 2024-03-06T07:22:46.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf9f16640 1 --2- 172.21.15.77:0/2186127525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40b4170 0x7fecf40b6a90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf9f16640 1 -- 172.21.15.77:0/2186127525 >> 172.21.15.77:0/2186127525 conn(0x7fecf40d3c40 msgr2=0x7fecf40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf9f16640 1 -- 172.21.15.77:0/2186127525 shutdown_connections 2024-03-06T07:22:46.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.338+0000 7fecf9f16640 1 -- 172.21.15.77:0/2186127525 wait complete. 2024-03-06T07:22:46.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.339+0000 7fecf9f16640 1 Processor -- start 2024-03-06T07:22:46.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf9f16640 1 -- start start 2024-03-06T07:22:46.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf9f16640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fecf40b4170 0x7fecf414aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf9f16640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40d0a80 0x7fecf4158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf9f16640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecf41f7600 0x7fecf41f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf9f16640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40b4dd0 con 0x7fecf40d0a80 2024-03-06T07:22:46.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf9f16640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40ce9e0 con 0x7fecf41f7600 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf9f16640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40b4f50 con 0x7fecf40b4170 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7feceaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40d0a80 0x7fecf4158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7feceaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40d0a80 0x7fecf4158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56940/0 (socket says 172.21.15.77:56940) 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7feceaffd640 1 -- 172.21.15.77:0/3334989973 learned_addr learned my addr 172.21.15.77:0/3334989973 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf37fe640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fecf40b4170 0x7fecf414aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.340+0000 7fecf3fff640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecf41f7600 0x7fecf41f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7feceaffd640 1 -- 172.21.15.77:0/3334989973 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fecf40b4170 msgr2=0x7fecf414aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7feceaffd640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fecf40b4170 0x7fecf414aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7feceaffd640 1 -- 172.21.15.77:0/3334989973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecf41f7600 msgr2=0x7fecf41f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7feceaffd640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecf41f7600 0x7fecf41f7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7feceaffd640 1 -- 172.21.15.77:0/3334989973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf41f80c0 con 0x7fecf40d0a80 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7feceaffd640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40d0a80 0x7fecf4158950 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00c3c0 tx=0x7fecdc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7fecf0ff9640 1 -- 172.21.15.77:0/3334989973 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc0052d0 con 0x7fecf40d0a80 2024-03-06T07:22:46.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7fecf0ff9640 1 -- 172.21.15.77:0/3334989973 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc005490 con 0x7fecf40d0a80 2024-03-06T07:22:46.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf41f83a0 con 0x7fecf40d0a80 2024-03-06T07:22:46.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7fecf0ff9640 1 -- 172.21.15.77:0/3334989973 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc007240 con 0x7fecf40d0a80 2024-03-06T07:22:46.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.341+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf4157700 con 0x7fecf40d0a80 2024-03-06T07:22:46.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.342+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf404f330 con 0x7fecf40d0a80 2024-03-06T07:22:46.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.342+0000 7fecf0ff9640 1 -- 172.21.15.77:0/3334989973 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecdc0073e0 con 0x7fecf40d0a80 2024-03-06T07:22:46.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.343+0000 7fecf0ff9640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecd406b500 0x7fecd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.343+0000 7fecf37fe640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecd406b500 0x7fecd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.343+0000 7fecf0ff9640 1 -- 172.21.15.77:0/3334989973 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fecdc089580 con 0x7fecf40d0a80 2024-03-06T07:22:46.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.343+0000 7fecf37fe640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecd406b500 0x7fecd406d9c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fece4036000 tx=0x7fece403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.347+0000 7fecf0ff9640 1 -- 172.21.15.77:0/3334989973 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fecdc057de0 con 0x7fecf40d0a80 2024-03-06T07:22:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.523+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecf40733b0 con 0x7fecf40d0a80 2024-03-06T07:22:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.524+0000 7fecf0ff9640 1 -- 172.21.15.77:0/3334989973 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16814 (secure 0 0 0) 0x7fecdc05be80 con 0x7fecf40d0a80 2024-03-06T07:22:46.527 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:46.527 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":22,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:46.081281+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:56.057985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:57.255143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecd406b500 msgr2=0x7fecd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecd406b500 0x7fecd406d9c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fece4036000 tx=0x7fece403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40d0a80 msgr2=0x7fecf4158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40d0a80 0x7fecf4158950 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00c3c0 tx=0x7fecdc00c880 comp rx=0 tx=0).stop 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 shutdown_connections 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecd406b500 0x7fecd406d9c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecf40d0a80 0x7fecf4158950 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fecf41f7600 0x7fecf41f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 --2- 172.21.15.77:0/3334989973 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fecf40b4170 0x7fecf414aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.526+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 >> 172.21.15.77:0/3334989973 conn(0x7fecf40d3c40 msgr2=0x7fecf40b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.527+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 shutdown_connections 2024-03-06T07:22:46.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.527+0000 7fecf9f16640 1 -- 172.21.15.77:0/3334989973 wait complete. 2024-03-06T07:22:46.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:22:46.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.639+0000 7f2c20fac640 1 Processor -- start 2024-03-06T07:22:46.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.639+0000 7f2c20fac640 1 -- start start 2024-03-06T07:22:46.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.640+0000 7f2c20fac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.640+0000 7f2c20fac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c0d22c0 con 0x7f2c1c162970 2024-03-06T07:22:46.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.640+0000 7f2c20fac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c0d25a0 con 0x7f2c1c161cf0 2024-03-06T07:22:46.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.640+0000 7f2c20fac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c161b00 con 0x7f2c1c0b4460 2024-03-06T07:22:46.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.640+0000 7f2c19d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2c1c162970 0x7f2c1c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34382/0 (socket says 172.21.15.77:34382) 2024-03-06T07:22:46.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.640+0000 7f2c19d9b640 1 -- 172.21.15.77:0/3852281986 learned_addr learned my addr 172.21.15.77:0/3852281986 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:46.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.640+0000 7f2c1a59c640 1 --2- 172.21.15.77:0/3852281986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.641+0000 7f2c1959a640 1 -- 172.21.15.77:0/3852281986 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534886262 0 0) 0x7f2c1c0d22c0 con 0x7f2c1c162970 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.641+0000 7f2c1959a640 1 -- 172.21.15.77:0/3852281986 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c04003520 con 0x7f2c1c162970 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.641+0000 7f2c1959a640 1 -- 172.21.15.77:0/3852281986 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164172522 0 0) 0x7f2c1c0d25a0 con 0x7f2c1c161cf0 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.641+0000 7f2c1959a640 1 -- 172.21.15.77:0/3852281986 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c1c0d22c0 con 0x7f2c1c161cf0 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.641+0000 7f2c1a59c640 1 -- 172.21.15.77:0/3852281986 >> v1:172.21.15.158:6789/0 conn(0x7f2c1c161cf0 legacy=0x7f2c1c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.641+0000 7f2c1a59c640 1 -- 172.21.15.77:0/3852281986 >> v1:172.21.15.77:6789/0 conn(0x7f2c1c162970 legacy=0x7f2c1c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.641+0000 7f2c1a59c640 1 -- 172.21.15.77:0/3852281986 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0d2880 con 0x7f2c1c0b4460 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c1a59c640 1 --2- 172.21.15.77:0/3852281986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0d4800 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009dc0 tx=0x7f2c0c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=103fa9b1463d3421 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c1959a640 1 -- 172.21.15.77:0/3852281986 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c039070 con 0x7f2c1c0b4460 2024-03-06T07:22:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c1959a640 1 -- 172.21.15.77:0/3852281986 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c02ccd0 con 0x7f2c1c0b4460 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c1959a640 1 -- 172.21.15.77:0/3852281986 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c034640 con 0x7f2c1c0b4460 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c20fac640 1 -- 172.21.15.77:0/3852281986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 msgr2=0x7f2c1c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c20fac640 1 --2- 172.21.15.77:0/3852281986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0d4800 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009dc0 tx=0x7f2c0c02bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c20fac640 1 -- 172.21.15.77:0/3852281986 shutdown_connections 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c20fac640 1 --2- 172.21.15.77:0/3852281986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0d4800 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c20fac640 1 -- 172.21.15.77:0/3852281986 >> 172.21.15.77:0/3852281986 conn(0x7f2c1c0d3ce0 msgr2=0x7f2c1c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c20fac640 1 -- 172.21.15.77:0/3852281986 shutdown_connections 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.642+0000 7f2c20fac640 1 -- 172.21.15.77:0/3852281986 wait complete. 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.644+0000 7f2c20fac640 1 Processor -- start 2024-03-06T07:22:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.644+0000 7f2c20fac640 1 -- start start 2024-03-06T07:22:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.644+0000 7f2c20fac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.644+0000 7f2c20fac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c161cf0 0x7f2c1c152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.644+0000 7f2c20fac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c1c162970 0x7f2c1c0b90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.644+0000 7f2c20fac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0ce950 con 0x7f2c1c161cf0 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.644+0000 7f2c20fac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0cecc0 con 0x7f2c1c162970 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.644+0000 7f2c20fac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0b4840 con 0x7f2c1c0b4460 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c161cf0 0x7f2c1c152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c1a59c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c161cf0 0x7f2c1c152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56946/0 (socket says 172.21.15.77:56946) 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c1ad9d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c1c162970 0x7f2c1c0b90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 -- 172.21.15.77:0/2470440750 learned_addr learned my addr 172.21.15.77:0/2470440750 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c1a59c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35164/0 (socket says 172.21.15.77:35164) 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 -- 172.21.15.77:0/2470440750 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 msgr2=0x7f2c1c0ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 -- 172.21.15.77:0/2470440750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c1c162970 msgr2=0x7f2c1c0b90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c1c162970 0x7f2c1c0b90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 -- 172.21.15.77:0/2470440750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0c9180 con 0x7f2c1c161cf0 2024-03-06T07:22:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c19d9b640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c161cf0 0x7f2c1c152240 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000c3e0 tx=0x7f2c1000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c0affd640 1 -- 172.21.15.77:0/2470440750 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10017070 con 0x7f2c1c161cf0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c0c9460 con 0x7f2c1c161cf0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c0affd640 1 -- 172.21.15.77:0/2470440750 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c10005d00 con 0x7f2c1c161cf0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c0affd640 1 -- 172.21.15.77:0/2470440750 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10007310 con 0x7f2c1c161cf0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.645+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c0c9a20 con 0x7f2c1c161cf0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.646+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c04f330 con 0x7f2c1c161cf0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.647+0000 7f2c0affd640 1 -- 172.21.15.77:0/2470440750 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c1001a070 con 0x7f2c1c161cf0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.647+0000 7f2c0affd640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bfc06b4d0 0x7f2bfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.647+0000 7f2c1a59c640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bfc06b4d0 0x7f2bfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.647+0000 7f2c0affd640 1 -- 172.21.15.77:0/2470440750 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f2c100896a0 con 0x7f2c1c161cf0 2024-03-06T07:22:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.647+0000 7f2c1a59c640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bfc06b4d0 0x7f2bfc06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c036000 tx=0x7f2c0c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.651+0000 7f2c0affd640 1 -- 172.21.15.77:0/2470440750 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c10057eb0 con 0x7f2c1c161cf0 2024-03-06T07:22:46.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.858+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2c1c06b180 con 0x7f2c1c161cf0 2024-03-06T07:22:46.861 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:22:46.859+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:46.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.860+0000 7f2c0affd640 1 -- 172.21.15.77:0/2470440750 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2c1005bf50 con 0x7f2c1c161cf0 2024-03-06T07:22:46.862 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bfc06b4d0 msgr2=0x7f2bfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bfc06b4d0 0x7f2bfc06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c036000 tx=0x7f2c0c030000 comp rx=0 tx=0).stop 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c161cf0 msgr2=0x7f2c1c152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c161cf0 0x7f2c1c152240 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000c3e0 tx=0x7f2c1000c8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 shutdown_connections 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2bfc06b4d0 0x7f2bfc06d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c161cf0 0x7f2c1c152240 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c1c162970 0x7f2c1c0b90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 --2- 172.21.15.77:0/2470440750 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c1c0b4460 0x7f2c1c0ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 >> 172.21.15.77:0/2470440750 conn(0x7f2c1c0d3ce0 msgr2=0x7f2c1c15f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 shutdown_connections 2024-03-06T07:22:46.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.863+0000 7f2c20fac640 1 -- 172.21.15.77:0/2470440750 wait complete. 2024-03-06T07:22:46.867 INFO:tasks.ceph.mds.b.smithi158.stderr: -2942> 2024-03-06T07:22:02.915+0000 7f9c6a728640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:46.876 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:22:46.894 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:22:46.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.979+0000 7fce7ebc9640 1 Processor -- start 2024-03-06T07:22:46.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.979+0000 7fce7ebc9640 1 -- start start 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7ebc9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce7815bbb0 0x7fce7806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7ebc9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce780cea90 con 0x7fce7815ee10 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7ebc9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce78161bc0 con 0x7fce7806b4f0 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7ebc9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce78161970 con 0x7fce7815bbb0 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7c965640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce7815bbb0 0x7fce7806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7d166640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fce7815ee10 0x7fce78071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:34392/0 (socket says 172.21.15.77:34392) 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7d166640 1 -- 172.21.15.77:0/4260442008 learned_addr learned my addr 172.21.15.77:0/4260442008 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7c965640 1 -- 172.21.15.77:0/4260442008 >> v1:172.21.15.158:6789/0 conn(0x7fce7806b4f0 legacy=0x7fce7815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7c965640 1 -- 172.21.15.77:0/4260442008 >> v1:172.21.15.77:6789/0 conn(0x7fce7815ee10 legacy=0x7fce78071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce7c965640 1 -- 172.21.15.77:0/4260442008 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce78161ea0 con 0x7fce7815bbb0 2024-03-06T07:22:46.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.980+0000 7fce777fe640 1 -- 172.21.15.77:0/4260442008 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214963476 0 0) 0x7fce70001490 con 0x7fce7815ee10 2024-03-06T07:22:46.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce7c965640 1 --2- 172.21.15.77:0/4260442008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815bbb0 0x7fce7806afb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0097a0 tx=0x7fce6c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef306a50ea96d66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce777fe640 1 -- 172.21.15.77:0/4260442008 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c034070 con 0x7fce7815bbb0 2024-03-06T07:22:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce777fe640 1 -- 172.21.15.77:0/4260442008 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c030610 con 0x7fce7815bbb0 2024-03-06T07:22:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce777fe640 1 -- 172.21.15.77:0/4260442008 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c0309e0 con 0x7fce7815bbb0 2024-03-06T07:22:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce7ebc9640 1 -- 172.21.15.77:0/4260442008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815bbb0 msgr2=0x7fce7806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce7ebc9640 1 --2- 172.21.15.77:0/4260442008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815bbb0 0x7fce7806afb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0097a0 tx=0x7fce6c02b960 comp rx=0 tx=0).stop 2024-03-06T07:22:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce7ebc9640 1 -- 172.21.15.77:0/4260442008 shutdown_connections 2024-03-06T07:22:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce7ebc9640 1 --2- 172.21.15.77:0/4260442008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815bbb0 0x7fce7806afb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce7ebc9640 1 -- 172.21.15.77:0/4260442008 >> 172.21.15.77:0/4260442008 conn(0x7fce780b2660 msgr2=0x7fce780b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.981+0000 7fce7ebc9640 1 -- 172.21.15.77:0/4260442008 shutdown_connections 2024-03-06T07:22:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.982+0000 7fce7ebc9640 1 -- 172.21.15.77:0/4260442008 wait complete. 2024-03-06T07:22:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.983+0000 7fce7ebc9640 1 Processor -- start 2024-03-06T07:22:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.983+0000 7fce7ebc9640 1 -- start start 2024-03-06T07:22:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.983+0000 7fce7ebc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce7806b4f0 0x7fce7814d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.983+0000 7fce7ebc9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815ee10 0x7fce781562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.983+0000 7fce7ebc9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce78160ee0 0x7fce780b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.983+0000 7fce7ebc9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce780723b0 con 0x7fce7815ee10 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.983+0000 7fce7ebc9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce781624f0 con 0x7fce78160ee0 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.983+0000 7fce7ebc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce78072530 con 0x7fce7806b4f0 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815ee10 0x7fce781562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815ee10 0x7fce781562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56976/0 (socket says 172.21.15.77:56976) 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 -- 172.21.15.77:0/1796232175 learned_addr learned my addr 172.21.15.77:0/1796232175 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce7c965640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce7806b4f0 0x7fce7814d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce7d166640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce78160ee0 0x7fce780b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 -- 172.21.15.77:0/1796232175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce7806b4f0 msgr2=0x7fce7814d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce7806b4f0 0x7fce7814d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 -- 172.21.15.77:0/1796232175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce78160ee0 msgr2=0x7fce780b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce78160ee0 0x7fce780b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 -- 172.21.15.77:0/1796232175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7814d930 con 0x7fce7815ee10 2024-03-06T07:22:46.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce77fff640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815ee10 0x7fce781562f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c450 tx=0x7fce6400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce757fa640 1 -- 172.21.15.77:0/1796232175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce64017070 con 0x7fce7815ee10 2024-03-06T07:22:46.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce757fa640 1 -- 172.21.15.77:0/1796232175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce640052d0 con 0x7fce7815ee10 2024-03-06T07:22:46.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce757fa640 1 -- 172.21.15.77:0/1796232175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce64003830 con 0x7fce7815ee10 2024-03-06T07:22:46.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce78156bb0 con 0x7fce7815ee10 2024-03-06T07:22:46.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.984+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7815f8b0 con 0x7fce7815ee10 2024-03-06T07:22:46.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.985+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce780733b0 con 0x7fce7815ee10 2024-03-06T07:22:46.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.986+0000 7fce757fa640 1 -- 172.21.15.77:0/1796232175 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce6401a070 con 0x7fce7815ee10 2024-03-06T07:22:46.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.986+0000 7fce757fa640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce5c06b4d0 0x7fce5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.986+0000 7fce757fa640 1 -- 172.21.15.77:0/1796232175 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fce64089600 con 0x7fce7815ee10 2024-03-06T07:22:46.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.986+0000 7fce7c965640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce5c06b4d0 0x7fce5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.986+0000 7fce7c965640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce5c06b4d0 0x7fce5c06d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce6c036000 tx=0x7fce6c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:46.990+0000 7fce757fa640 1 -- 172.21.15.77:0/1796232175 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce64058450 con 0x7fce7815ee10 2024-03-06T07:22:47.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.201+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fce78068ff0 con 0x7fce7815ee10 2024-03-06T07:22:47.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.862+0000 7fce757fa640 1 -- 172.21.15.77:0/1796232175 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7fce64057d50 con 0x7fce7815ee10 2024-03-06T07:22:47.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce5c06b4d0 msgr2=0x7fce5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce5c06b4d0 0x7fce5c06d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce6c036000 tx=0x7fce6c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:47.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815ee10 msgr2=0x7fce781562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815ee10 0x7fce781562f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c450 tx=0x7fce6400c910 comp rx=0 tx=0).stop 2024-03-06T07:22:47.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 shutdown_connections 2024-03-06T07:22:47.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce5c06b4d0 0x7fce5c06d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce7815ee10 0x7fce781562f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce78160ee0 0x7fce780b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 --2- 172.21.15.77:0/1796232175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce7806b4f0 0x7fce7814d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 >> 172.21.15.77:0/1796232175 conn(0x7fce780b2660 msgr2=0x7fce780b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 shutdown_connections 2024-03-06T07:22:47.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.864+0000 7fce7ebc9640 1 -- 172.21.15.77:0/1796232175 wait complete. 2024-03-06T07:22:47.878 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:22:47.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.980+0000 7fc61b0fa640 1 Processor -- start 2024-03-06T07:22:47.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.980+0000 7fc61b0fa640 1 -- start start 2024-03-06T07:22:47.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc61b0fa640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140ad940 0x7fc6140ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc61b0fa640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc614071d20 0x7fc614072190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc61b0fa640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61406d090 con 0x7fc614147720 2024-03-06T07:22:47.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc61b0fa640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6140726d0 con 0x7fc614071d20 2024-03-06T07:22:47.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc61b0fa640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61406d370 con 0x7fc6140ad940 2024-03-06T07:22:47.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc613fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140ad940 0x7fc6140ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc613fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140ad940 0x7fc6140ae750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:35186/0 (socket says 172.21.15.77:35186) 2024-03-06T07:22:47.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc613fff640 1 -- 172.21.15.77:0/3260591003 learned_addr learned my addr 172.21.15.77:0/3260591003 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:47.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.981+0000 7fc619697640 1 --2- 172.21.15.77:0/3260591003 >> v2:172.21.15.158:3300/0 conn(0x7fc614071d20 0x7fc614072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.982+0000 7fc613fff640 1 -- 172.21.15.77:0/3260591003 >> v2:172.21.15.158:3300/0 conn(0x7fc614071d20 msgr2=0x7fc614072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.982+0000 7fc613fff640 1 --2- 172.21.15.77:0/3260591003 >> v2:172.21.15.158:3300/0 conn(0x7fc614071d20 0x7fc614072190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.982+0000 7fc613fff640 1 -- 172.21.15.77:0/3260591003 >> v1:172.21.15.158:6789/0 conn(0x7fc614147720 legacy=0x7fc6140b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.982+0000 7fc613fff640 1 -- 172.21.15.77:0/3260591003 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61406d650 con 0x7fc6140ad940 2024-03-06T07:22:47.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.982+0000 7fc613fff640 1 --2- 172.21.15.77:0/3260591003 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140ad940 0x7fc6140ae750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc600009870 tx=0x7fc60002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef6e44ea2327715f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc6137fe640 1 -- 172.21.15.77:0/3260591003 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600034070 con 0x7fc6140ad940 2024-03-06T07:22:47.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc6137fe640 1 -- 172.21.15.77:0/3260591003 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc600030840 con 0x7fc6140ad940 2024-03-06T07:22:47.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc6137fe640 1 -- 172.21.15.77:0/3260591003 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600030c10 con 0x7fc6140ad940 2024-03-06T07:22:47.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc61b0fa640 1 -- 172.21.15.77:0/3260591003 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140ad940 msgr2=0x7fc6140ae750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/3260591003 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140ad940 0x7fc6140ae750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc600009870 tx=0x7fc60002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:47.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc61b0fa640 1 -- 172.21.15.77:0/3260591003 shutdown_connections 2024-03-06T07:22:47.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/3260591003 >> v2:172.21.15.158:3300/0 conn(0x7fc614071d20 0x7fc614072190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/3260591003 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140ad940 0x7fc6140ae750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc61b0fa640 1 -- 172.21.15.77:0/3260591003 >> 172.21.15.77:0/3260591003 conn(0x7fc6140b1a90 msgr2=0x7fc6140b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc61b0fa640 1 -- 172.21.15.77:0/3260591003 shutdown_connections 2024-03-06T07:22:47.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.983+0000 7fc61b0fa640 1 -- 172.21.15.77:0/3260591003 wait complete. 2024-03-06T07:22:47.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.984+0000 7fc61b0fa640 1 Processor -- start 2024-03-06T07:22:47.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.984+0000 7fc61b0fa640 1 -- start start 2024-03-06T07:22:47.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc61b0fa640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc614071d20 0x7fc614144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc61b0fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140ad940 0x7fc6140c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc61b0fa640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc614147720 0x7fc61415d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc61b0fa640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc614147b00 con 0x7fc6140ad940 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc61b0fa640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61406dc10 con 0x7fc614147720 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc61b0fa640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140aec90 con 0x7fc614071d20 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc613fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140ad940 0x7fc6140c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc613fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140ad940 0x7fc6140c87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56992/0 (socket says 172.21.15.77:56992) 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc613fff640 1 -- 172.21.15.77:0/121081588 learned_addr learned my addr 172.21.15.77:0/121081588 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc618e96640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc614071d20 0x7fc614144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.985+0000 7fc619697640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc614147720 0x7fc61415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc613fff640 1 -- 172.21.15.77:0/121081588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc614071d20 msgr2=0x7fc614144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc613fff640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc614071d20 0x7fc614144140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc613fff640 1 -- 172.21.15.77:0/121081588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc614147720 msgr2=0x7fc61415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc613fff640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc614147720 0x7fc61415d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc613fff640 1 -- 172.21.15.77:0/121081588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6141f7270 con 0x7fc6140ad940 2024-03-06T07:22:47.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc613fff640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140ad940 0x7fc6140c87c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc600000c00 tx=0x7fc600004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc6117fa640 1 -- 172.21.15.77:0/121081588 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600048070 con 0x7fc6140ad940 2024-03-06T07:22:47.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc6117fa640 1 -- 172.21.15.77:0/121081588 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc600030640 con 0x7fc6140ad940 2024-03-06T07:22:47.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6141f7550 con 0x7fc6140ad940 2024-03-06T07:22:47.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc6117fa640 1 -- 172.21.15.77:0/121081588 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600034040 con 0x7fc6140ad940 2024-03-06T07:22:47.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.986+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6141f7b10 con 0x7fc6140ad940 2024-03-06T07:22:47.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.987+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61404f330 con 0x7fc6140ad940 2024-03-06T07:22:47.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.987+0000 7fc6117fa640 1 -- 172.21.15.77:0/121081588 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc60004b070 con 0x7fc6140ad940 2024-03-06T07:22:47.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.988+0000 7fc6117fa640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.988+0000 7fc6117fa640 1 -- 172.21.15.77:0/121081588 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fc6000ac080 con 0x7fc6140ad940 2024-03-06T07:22:47.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.988+0000 7fc618e96640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.988+0000 7fc618e96640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f806b4d0 0x7fc5f806d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc6080016e0 tx=0x7fc608009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:47.992+0000 7fc6117fa640 1 -- 172.21.15.77:0/121081588 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc60007a7f0 con 0x7fc6140ad940 2024-03-06T07:22:48.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:48.166+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc6140c8d00 con 0x7fc6140ad940 2024-03-06T07:22:49.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.094+0000 7fc6117fa640 1 -- 172.21.15.77:0/121081588 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc60007e890 con 0x7fc6140ad940 2024-03-06T07:22:49.097 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f806b4d0 msgr2=0x7fc5f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f806b4d0 0x7fc5f806d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc6080016e0 tx=0x7fc608009040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140ad940 msgr2=0x7fc6140c87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140ad940 0x7fc6140c87c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc600000c00 tx=0x7fc600004730 comp rx=0 tx=0).stop 2024-03-06T07:22:49.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 shutdown_connections 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140ad940 0x7fc6140c87c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc614147720 0x7fc61415d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 --2- 172.21.15.77:0/121081588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc614071d20 0x7fc614144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 >> 172.21.15.77:0/121081588 conn(0x7fc6140b1a90 msgr2=0x7fc6141604f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 shutdown_connections 2024-03-06T07:22:49.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.098+0000 7fc61b0fa640 1 -- 172.21.15.77:0/121081588 wait complete. 2024-03-06T07:22:49.114 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:22:49.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.221+0000 7f92fdc17640 1 Processor -- start 2024-03-06T07:22:49.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.221+0000 7f92fdc17640 1 -- start start 2024-03-06T07:22:49.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92fdc17640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92f815a490 0x7f92f806df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92fdc17640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f806e4d0 con 0x7f92f80b3fd0 2024-03-06T07:22:49.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92fdc17640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f80b4490 con 0x7f92f80b3060 2024-03-06T07:22:49.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92fdc17640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f80cf4a0 con 0x7f92f815a490 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92f6ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f92f80b3fd0 0x7f92f80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48954/0 (socket says 172.21.15.77:48954) 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92f6ffd640 1 -- 172.21.15.77:0/1903406144 learned_addr learned my addr 172.21.15.77:0/1903406144 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438941587 0 0) 0x7f92ec001110 con 0x7f92f80b3fd0 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e0004630 con 0x7f92f80b3fd0 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.222+0000 7f92f77fe640 1 --2- 172.21.15.77:0/1903406144 >> v2:172.21.15.158:3300/0 conn(0x7f92f815a490 0x7f92f806df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2927650843 0 0) 0x7f92e0004630 con 0x7f92f80b3fd0 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 >> v2:172.21.15.158:3300/0 conn(0x7f92f815a490 msgr2=0x7f92f806df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 --2- 172.21.15.77:0/1903406144 >> v2:172.21.15.158:3300/0 conn(0x7f92f815a490 0x7f92f806df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 >> v1:172.21.15.158:6789/0 conn(0x7f92f80b3060 legacy=0x7f92f815dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f80b4750 con 0x7f92f80b3fd0 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f92ec001840 con 0x7f92f80b3fd0 2024-03-06T07:22:49.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 >> v1:172.21.15.77:6789/0 conn(0x7f92f80b3fd0 legacy=0x7f92f80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92e0004ac0 0x7f92e0004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92e0005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e00062d0 0x7f92e000cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80cf4a0 con 0x7f92e0005910 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e0003440 con 0x7f92e00062d0 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e0003640 con 0x7f92e0004ac0 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f6ffd640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92e0005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f77fe640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92e0004ac0 0x7f92e0004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f7fff640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e00062d0 0x7f92e000cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f6ffd640 1 -- 172.21.15.77:0/1903406144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92e0004ac0 msgr2=0x7f92e0004ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f6ffd640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92e0004ac0 0x7f92e0004ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f6ffd640 1 -- 172.21.15.77:0/1903406144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e00062d0 msgr2=0x7f92e000cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f6ffd640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92e00062d0 0x7f92e000cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f6ffd640 1 -- 172.21.15.77:0/1903406144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e0004630 con 0x7f92e0005910 2024-03-06T07:22:49.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.223+0000 7f92f7fff640 1 -- 172.21.15.77:0/1903406144 reap_dead start 2024-03-06T07:22:49.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92f6ffd640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92e0005d90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c4c0 tx=0x7f92e0006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289e8d57e0299067 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec00c0e0 con 0x7f92e0005910 2024-03-06T07:22:49.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec030070 con 0x7f92e0005910 2024-03-06T07:22:49.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92f67fc640 1 -- 172.21.15.77:0/1903406144 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0058d0 con 0x7f92e0005910 2024-03-06T07:22:49.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92fdc17640 1 -- 172.21.15.77:0/1903406144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 msgr2=0x7f92e0005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92fdc17640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92e0005d90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c4c0 tx=0x7f92e0006940 comp rx=0 tx=0).stop 2024-03-06T07:22:49.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92fdc17640 1 -- 172.21.15.77:0/1903406144 shutdown_connections 2024-03-06T07:22:49.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92fdc17640 1 --2- 172.21.15.77:0/1903406144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92e0005d90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.224+0000 7f92fdc17640 1 -- 172.21.15.77:0/1903406144 >> 172.21.15.77:0/1903406144 conn(0x7f92f80b1650 msgr2=0x7f92f80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.225+0000 7f92fdc17640 1 -- 172.21.15.77:0/1903406144 shutdown_connections 2024-03-06T07:22:49.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.225+0000 7f92fdc17640 1 -- 172.21.15.77:0/1903406144 wait complete. 2024-03-06T07:22:49.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.226+0000 7f92fdc17640 1 Processor -- start 2024-03-06T07:22:49.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.226+0000 7f92fdc17640 1 -- start start 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.226+0000 7f92fdc17640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92f8156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.226+0000 7f92fdc17640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f81efea0 0x7f92f80b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.226+0000 7f92fdc17640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f8145db0 0x7f92f8146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.226+0000 7f92fdc17640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80cf0e0 con 0x7f92e0005910 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.226+0000 7f92fdc17640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80b3490 con 0x7f92f81efea0 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.226+0000 7f92fdc17640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80cf260 con 0x7f92f8145db0 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92f8156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92f8156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42094/0 (socket says 172.21.15.77:42094) 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 -- 172.21.15.77:0/368280312 learned_addr learned my addr 172.21.15.77:0/368280312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f6ffd640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f81efea0 0x7f92f80b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f7fff640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f8145db0 0x7f92f8146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 -- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f8145db0 msgr2=0x7f92f8146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f8145db0 0x7f92f8146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 -- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f81efea0 msgr2=0x7f92f80b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f81efea0 0x7f92f80b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 -- 172.21.15.77:0/368280312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f8146780 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f7fff640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f8145db0 0x7f92f8146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92f77fe640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92f8156fe0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f92e80116d0 tx=0x7f92e8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92dffff640 1 -- 172.21.15.77:0/368280312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e800f9c0 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92dffff640 1 -- 172.21.15.77:0/368280312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e800fb60 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f8150e40 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92dffff640 1 -- 172.21.15.77:0/368280312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8015460 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.227+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f8157700 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.228+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f80687d0 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.229+0000 7f92dffff640 1 -- 172.21.15.77:0/368280312 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92e8015600 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.229+0000 7f92dffff640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92e406b4d0 0x7f92e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.229+0000 7f92dffff640 1 -- 172.21.15.77:0/368280312 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5452+0+0 (secure 0 0 0) 0x7f92e808ab00 con 0x7f92e0005910 2024-03-06T07:22:49.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.229+0000 7f92f6ffd640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92e406b4d0 0x7f92e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.230+0000 7f92f6ffd640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92e406b4d0 0x7f92e406d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c490 tx=0x7f92ec009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.233+0000 7f92dffff640 1 -- 172.21.15.77:0/368280312 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92e8019030 con 0x7f92e0005910 2024-03-06T07:22:49.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:49.404+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f92f804f330 con 0x7f92e0005910 2024-03-06T07:22:50.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.096+0000 7f92dffff640 1 -- 172.21.15.77:0/368280312 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f92e8059a80 con 0x7f92e0005910 2024-03-06T07:22:50.098 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.099+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92e406b4d0 msgr2=0x7f92e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.099+0000 7f92fdc17640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92e406b4d0 0x7f92e406d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c490 tx=0x7f92ec009030 comp rx=0 tx=0).stop 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.099+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 msgr2=0x7f92f8156fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.099+0000 7f92fdc17640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92f8156fe0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f92e80116d0 tx=0x7f92e8011b90 comp rx=0 tx=0).stop 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.100+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 shutdown_connections 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.100+0000 7f92fdc17640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92f8145db0 0x7f92f8146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.100+0000 7f92fdc17640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92f81efea0 0x7f92f80b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.100+0000 7f92fdc17640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92e406b4d0 0x7f92e406d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.100+0000 7f92fdc17640 1 --2- 172.21.15.77:0/368280312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0005910 0x7f92f8156fe0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.100+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 >> 172.21.15.77:0/368280312 conn(0x7f92f80b1650 msgr2=0x7f92f80cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.100+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 shutdown_connections 2024-03-06T07:22:50.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.100+0000 7f92fdc17640 1 -- 172.21.15.77:0/368280312 wait complete. 2024-03-06T07:22:50.116 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:50.116 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:50.116 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:50.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:50.117 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:50.117 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:50.117 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:50.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:50.117 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:50.117 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:50.118 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:50.118 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:50.118 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:50.118 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:50.118 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:50.118 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:50.127 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:50.127 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:50.128 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:50.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:50.130 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:50.130 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:50.131 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:50.131 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:50.169 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:50.170 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:50.170 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:50.171 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:50.173 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:50.173 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:50.174 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:50.174 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:50.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:50.248 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:22:50.248 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:22:50.253 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:22:50.253 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:22:50.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.279+0000 7f251500f640 1 Processor -- start 2024-03-06T07:22:50.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.279+0000 7f251500f640 1 -- start start 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.279+0000 7f251500f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 0x7f25100d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.279+0000 7f251500f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2510162970 0x7f25100b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.279+0000 7f251500f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2510161cf0 0x7f2510162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.279+0000 7f251500f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2510161b00 con 0x7f2510161cf0 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.279+0000 7f251500f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25100cecc0 con 0x7f2510162970 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.279+0000 7f251500f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25100d22c0 con 0x7f25100b4460 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.280+0000 7f250f59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2510161cf0 0x7f2510162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.280+0000 7f250f59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2510161cf0 0x7f2510162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42148/0 (socket says 172.21.15.77:42148) 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.280+0000 7f250f59e640 1 -- 172.21.15.77:0/3126344535 learned_addr learned my addr 172.21.15.77:0/3126344535 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:50.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.280+0000 7f250f59e640 1 -- 172.21.15.77:0/3126344535 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 msgr2=0x7f25100d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.280+0000 7f250f59e640 1 --2- 172.21.15.77:0/3126344535 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 0x7f25100d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.280+0000 7f250f59e640 1 -- 172.21.15.77:0/3126344535 >> v2:172.21.15.158:3300/0 conn(0x7f2510162970 msgr2=0x7f25100b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.280+0000 7f250f59e640 1 --2- 172.21.15.77:0/3126344535 >> v2:172.21.15.158:3300/0 conn(0x7f2510162970 0x7f25100b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.280+0000 7f250f59e640 1 -- 172.21.15.77:0/3126344535 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25100d25a0 con 0x7f2510161cf0 2024-03-06T07:22:50.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f250f59e640 1 --2- 172.21.15.77:0/3126344535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510161cf0 0x7f2510162180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f25100b6840 tx=0x7f250002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467c4f657c9132bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f250dd9b640 1 -- 172.21.15.77:0/3126344535 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500035020 con 0x7f2510161cf0 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f250dd9b640 1 -- 172.21.15.77:0/3126344535 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2500031870 con 0x7f2510161cf0 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f250dd9b640 1 -- 172.21.15.77:0/3126344535 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500031c40 con 0x7f2510161cf0 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 -- 172.21.15.77:0/3126344535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510161cf0 msgr2=0x7f2510162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 --2- 172.21.15.77:0/3126344535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510161cf0 0x7f2510162180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f25100b6840 tx=0x7f250002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 -- 172.21.15.77:0/3126344535 shutdown_connections 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 --2- 172.21.15.77:0/3126344535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510161cf0 0x7f2510162180 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 --2- 172.21.15.77:0/3126344535 >> v2:172.21.15.158:3300/0 conn(0x7f2510162970 0x7f25100b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 --2- 172.21.15.77:0/3126344535 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 0x7f25100d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 -- 172.21.15.77:0/3126344535 >> 172.21.15.77:0/3126344535 conn(0x7f25100d3ce0 msgr2=0x7f25100d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 -- 172.21.15.77:0/3126344535 shutdown_connections 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.281+0000 7f251500f640 1 -- 172.21.15.77:0/3126344535 wait complete. 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.283+0000 7f251500f640 1 Processor -- start 2024-03-06T07:22:50.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.283+0000 7f251500f640 1 -- start start 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.283+0000 7f251500f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 0x7f2510155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.283+0000 7f251500f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2510161cf0 0x7f25100cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.283+0000 7f251500f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510162970 0x7f25100c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.283+0000 7f251500f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25100ce950 con 0x7f2510162970 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.283+0000 7f251500f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25100d2b60 con 0x7f2510161cf0 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.283+0000 7f251500f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25100b3ac0 con 0x7f25100b4460 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510162970 0x7f25100c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510162970 0x7f25100c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42158/0 (socket says 172.21.15.77:42158) 2024-03-06T07:22:50.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 -- 172.21.15.77:0/177987369 learned_addr learned my addr 172.21.15.77:0/177987369 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250e59c640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2510161cf0 0x7f25100cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250ed9d640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 0x7f2510155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 -- 172.21.15.77:0/177987369 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 msgr2=0x7f2510155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 0x7f2510155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 -- 172.21.15.77:0/177987369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2510161cf0 msgr2=0x7f25100cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2510161cf0 0x7f25100cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 -- 172.21.15.77:0/177987369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25100c78b0 con 0x7f2510162970 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f250f59e640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510162970 0x7f25100c9c20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f250002b950 tx=0x7f25000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f24f77fe640 1 -- 172.21.15.77:0/177987369 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000443f0 con 0x7f2510162970 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f24f77fe640 1 -- 172.21.15.77:0/177987369 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25000445b0 con 0x7f2510162970 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25100c7b90 con 0x7f2510162970 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f24f77fe640 1 -- 172.21.15.77:0/177987369 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000449a0 con 0x7f2510162970 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.284+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25100c8150 con 0x7f2510162970 2024-03-06T07:22:50.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.285+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25100ce0f0 con 0x7f2510162970 2024-03-06T07:22:50.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.286+0000 7f24f77fe640 1 -- 172.21.15.77:0/177987369 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2500038020 con 0x7f2510162970 2024-03-06T07:22:50.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.286+0000 7f24f77fe640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24fc06b4d0 0x7f24fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.286+0000 7f250ed9d640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24fc06b4d0 0x7f24fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.286+0000 7f24f77fe640 1 -- 172.21.15.77:0/177987369 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f25000ac0e0 con 0x7f2510162970 2024-03-06T07:22:50.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.286+0000 7f250ed9d640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24fc06b4d0 0x7f24fc06d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f25080098e0 tx=0x7f2508009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.290+0000 7f24f77fe640 1 -- 172.21.15.77:0/177987369 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f250003f070 con 0x7f2510162970 2024-03-06T07:22:50.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:50.475+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f251004f330 con 0x7f2510162970 2024-03-06T07:22:51.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.115+0000 7f24f77fe640 1 -- 172.21.15.77:0/177987369 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7f250007ab80 con 0x7f2510162970 2024-03-06T07:22:51.117 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:22:51.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24fc06b4d0 msgr2=0x7f24fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24fc06b4d0 0x7f24fc06d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f25080098e0 tx=0x7f2508009060 comp rx=0 tx=0).stop 2024-03-06T07:22:51.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510162970 msgr2=0x7f25100c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510162970 0x7f25100c9c20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f250002b950 tx=0x7f25000046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 shutdown_connections 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24fc06b4d0 0x7f24fc06d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2510162970 0x7f25100c9c20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2510161cf0 0x7f25100cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 --2- 172.21.15.77:0/177987369 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25100b4460 0x7f2510155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 >> 172.21.15.77:0/177987369 conn(0x7f25100d3ce0 msgr2=0x7f25100d0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 shutdown_connections 2024-03-06T07:22:51.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.117+0000 7f251500f640 1 -- 172.21.15.77:0/177987369 wait complete. 2024-03-06T07:22:51.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:51.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.235+0000 7fb0cf6ed640 1 Processor -- start 2024-03-06T07:22:51.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.235+0000 7fb0cf6ed640 1 -- start start 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.235+0000 7fb0cf6ed640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb0c81515e0 0x7fb0c80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0cf6ed640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c8071f40 con 0x7fb0c815b430 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0cf6ed640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c815b8e0 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0cf6ed640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c8070a70 con 0x7fb0c81515e0 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0ccc88640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb0c815b430 0x7fb0c8071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48986/0 (socket says 172.21.15.77:48986) 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0ccc88640 1 -- 172.21.15.77:0/2264725754 learned_addr learned my addr 172.21.15.77:0/2264725754 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0cd489640 1 --2- 172.21.15.77:0/2264725754 >> v2:172.21.15.158:3300/0 conn(0x7fb0c81515e0 0x7fb0c80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0bbfff640 1 -- 172.21.15.77:0/2264725754 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965606288 0 0) 0x7fb0c8071f40 con 0x7fb0c815b430 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0bbfff640 1 -- 172.21.15.77:0/2264725754 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b4003520 con 0x7fb0c815b430 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0bbfff640 1 -- 172.21.15.77:0/2264725754 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29097243 0 0) 0x7fb0c815b8e0 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0bbfff640 1 -- 172.21.15.77:0/2264725754 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c8071f40 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0cd489640 1 -- 172.21.15.77:0/2264725754 >> v1:172.21.15.158:6789/0 conn(0x7fb0c80cd7a0 legacy=0x7fb0c80cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0cd489640 1 -- 172.21.15.77:0/2264725754 >> v1:172.21.15.77:6789/0 conn(0x7fb0c815b430 legacy=0x7fb0c8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0bbfff640 1 -- 172.21.15.77:0/2264725754 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294234661 0 0) 0x7fb0b4003520 con 0x7fb0c815b430 2024-03-06T07:22:51.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.236+0000 7fb0cd489640 1 -- 172.21.15.77:0/2264725754 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c8070d50 con 0x7fb0c81515e0 2024-03-06T07:22:51.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.237+0000 7fb0cd489640 1 --2- 172.21.15.77:0/2264725754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c81515e0 0x7fb0c80cce40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0097a0 tx=0x7fb0bc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ee717cd8ffc616f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.237+0000 7fb0bbfff640 1 -- 172.21.15.77:0/2264725754 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc038070 con 0x7fb0c81515e0 2024-03-06T07:22:51.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.237+0000 7fb0bbfff640 1 -- 172.21.15.77:0/2264725754 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc031500 con 0x7fb0c81515e0 2024-03-06T07:22:51.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.237+0000 7fb0bbfff640 1 -- 172.21.15.77:0/2264725754 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4001690 con 0x7fb0c81515e0 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.238+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2264725754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c81515e0 msgr2=0x7fb0c80cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.238+0000 7fb0cf6ed640 1 --2- 172.21.15.77:0/2264725754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c81515e0 0x7fb0c80cce40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0097a0 tx=0x7fb0bc02b820 comp rx=0 tx=0).stop 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.238+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2264725754 shutdown_connections 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.238+0000 7fb0cf6ed640 1 --2- 172.21.15.77:0/2264725754 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c81515e0 0x7fb0c80cce40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.238+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2264725754 >> 172.21.15.77:0/2264725754 conn(0x7fb0c80cf8a0 msgr2=0x7fb0c80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.238+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2264725754 shutdown_connections 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.238+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2264725754 wait complete. 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.239+0000 7fb0cf6ed640 1 Processor -- start 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.239+0000 7fb0cf6ed640 1 -- start start 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cf6ed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c80cd7a0 0x7fb0c8146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cf6ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0c81515e0 0x7fb0c8147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cf6ed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c815b430 0x7fb0c815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cf6ed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80cc3a0 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cf6ed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c8071310 con 0x7fb0c815b430 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cf6ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80cc520 con 0x7fb0c81515e0 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c80cd7a0 0x7fb0c8146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c80cd7a0 0x7fb0c8146c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42166/0 (socket says 172.21.15.77:42166) 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 -- 172.21.15.77:0/2983364764 learned_addr learned my addr 172.21.15.77:0/2983364764 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0ccc88640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0c81515e0 0x7fb0c8147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cdc8a640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c815b430 0x7fb0c815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 -- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0c81515e0 msgr2=0x7fb0c8147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0c81515e0 0x7fb0c8147cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 -- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c815b430 msgr2=0x7fb0c815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c815b430 0x7fb0c815dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 -- 172.21.15.77:0/2983364764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c812feb0 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cdc8a640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c815b430 0x7fb0c815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0ccc88640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0c81515e0 0x7fb0c8147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cd489640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c80cd7a0 0x7fb0c8146c00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc000c00 tx=0x7fb0bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0b9ffb640 1 -- 172.21.15.77:0/2983364764 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc043070 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0b9ffb640 1 -- 172.21.15.77:0/2983364764 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc03f870 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c8130190 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.240+0000 7fb0b9ffb640 1 -- 172.21.15.77:0/2983364764 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc038040 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.241+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c8130750 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.242+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c8068840 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.242+0000 7fb0b9ffb640 1 -- 172.21.15.77:0/2983364764 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0bc046070 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.242+0000 7fb0b9ffb640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0ac06b500 0x7fb0ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.243+0000 7fb0ccc88640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0ac06b500 0x7fb0ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.243+0000 7fb0b9ffb640 1 -- 172.21.15.77:0/2983364764 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb0bc0ab790 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.243+0000 7fb0ccc88640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0ac06b500 0x7fb0ac06d9c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4001800 tx=0x7fb0c40015f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.247+0000 7fb0b9ffb640 1 -- 172.21.15.77:0/2983364764 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0bc04d070 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.483+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb0c804f330 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.483+0000 7fb0b9ffb640 1 -- 172.21.15.77:0/2983364764 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fb0bc030070 con 0x7fb0c80cd7a0 2024-03-06T07:22:51.485 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:51.485 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:22:51.485 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:22:51.485 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:22:51.486 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:22:51.487 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.488 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:22:51.489 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:22:51.490 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.491 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:22:51.492 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.493 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:22:51.494 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:22:51.495 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.496 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQCFGehloRXSDhAAzHi0Jmg7rn6r/pevIi7fTQ==", 2024-03-06T07:22:51.497 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQCFGehlxx3SDhAA7O1GenGclC8dFVpanjUghw==", 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQCFGehltCTSDhAAkuPbFCmn1yQRM6MHjhvm7g==", 2024-03-06T07:22:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQCFGehlNCvSDhAAYw2OUEKcQT07s1wEhLAQMw==", 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQCFGehl+jHSDhAAposnbjXCxvoWe+lo5V4Mmg==", 2024-03-06T07:22:51.499 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQCFGehl+DjSDhAAQgdnV/orqoEm9cFafO10WQ==", 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:22:51.500 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:22:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.502 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:51.502 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:22:51.502 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:22:51.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0ac06b500 msgr2=0x7fb0ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0ac06b500 0x7fb0ac06d9c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4001800 tx=0x7fb0c40015f0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c80cd7a0 msgr2=0x7fb0c8146c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c80cd7a0 0x7fb0c8146c00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc000c00 tx=0x7fb0bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 shutdown_connections 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0c815b430 0x7fb0c815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0c81515e0 0x7fb0c8147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0ac06b500 0x7fb0ac06d9c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 --2- 172.21.15.77:0/2983364764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c80cd7a0 0x7fb0c8146c00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 >> 172.21.15.77:0/2983364764 conn(0x7fb0c80cf8a0 msgr2=0x7fb0c80ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 shutdown_connections 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.485+0000 7fb0cf6ed640 1 -- 172.21.15.77:0/2983364764 wait complete. 2024-03-06T07:22:51.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:22:51.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.604+0000 7f0eb028a640 1 Processor -- start 2024-03-06T07:22:51.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.604+0000 7f0eb028a640 1 -- start start 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.604+0000 7f0eb028a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ea8161670 0x7f0ea8072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eb028a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea8072a60 0x7f0ea8072ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eb028a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0ea80adf10 0x7f0ea80ae380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eb028a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea80b6170 con 0x7f0ea80adf10 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eb028a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea80ada80 con 0x7f0ea8161670 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eb028a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea8161a50 con 0x7f0ea8072a60 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae827640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0ea80adf10 0x7f0ea80ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae827640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0ea80adf10 0x7f0ea80ae380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42170/0 (socket says 172.21.15.77:42170) 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae827640 1 -- 172.21.15.77:0/1128950983 learned_addr learned my addr 172.21.15.77:0/1128950983 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0ead825640 1 --2- 172.21.15.77:0/1128950983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea8072a60 0x7f0ea8072ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae026640 1 --2- 172.21.15.77:0/1128950983 >> v2:172.21.15.158:3300/0 conn(0x7f0ea8161670 0x7f0ea8072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae827640 1 -- 172.21.15.77:0/1128950983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea8072a60 msgr2=0x7f0ea8072ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae827640 1 --2- 172.21.15.77:0/1128950983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea8072a60 0x7f0ea8072ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae827640 1 -- 172.21.15.77:0/1128950983 >> v2:172.21.15.158:3300/0 conn(0x7f0ea8161670 msgr2=0x7f0ea8072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae827640 1 --2- 172.21.15.77:0/1128950983 >> v2:172.21.15.158:3300/0 conn(0x7f0ea8161670 0x7f0ea8072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.605+0000 7f0eae827640 1 -- 172.21.15.77:0/1128950983 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea8066d00 con 0x7f0ea80adf10 2024-03-06T07:22:51.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.606+0000 7f0eae827640 1 --2- 172.21.15.77:0/1128950983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea80adf10 0x7f0ea80ae380 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8070a80 tx=0x7f0e9802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddb9f3803f60e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.606+0000 7f0ead024640 1 -- 172.21.15.77:0/1128950983 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98035020 con 0x7f0ea80adf10 2024-03-06T07:22:51.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.606+0000 7f0ead024640 1 -- 172.21.15.77:0/1128950983 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e98031640 con 0x7f0ea80adf10 2024-03-06T07:22:51.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.606+0000 7f0ead024640 1 -- 172.21.15.77:0/1128950983 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9802d040 con 0x7f0ea80adf10 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.606+0000 7f0eb028a640 1 -- 172.21.15.77:0/1128950983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea80adf10 msgr2=0x7f0ea80ae380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.606+0000 7f0eb028a640 1 --2- 172.21.15.77:0/1128950983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea80adf10 0x7f0ea80ae380 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8070a80 tx=0x7f0e9802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.607+0000 7f0eb028a640 1 -- 172.21.15.77:0/1128950983 shutdown_connections 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.607+0000 7f0eb028a640 1 --2- 172.21.15.77:0/1128950983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea80adf10 0x7f0ea80ae380 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.607+0000 7f0eb028a640 1 --2- 172.21.15.77:0/1128950983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea8072a60 0x7f0ea8072ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.607+0000 7f0eb028a640 1 --2- 172.21.15.77:0/1128950983 >> v2:172.21.15.158:3300/0 conn(0x7f0ea8161670 0x7f0ea8072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.607+0000 7f0eb028a640 1 -- 172.21.15.77:0/1128950983 >> 172.21.15.77:0/1128950983 conn(0x7f0ea80b4e60 msgr2=0x7f0ea80b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.607+0000 7f0eb028a640 1 -- 172.21.15.77:0/1128950983 shutdown_connections 2024-03-06T07:22:51.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.607+0000 7f0eb028a640 1 -- 172.21.15.77:0/1128950983 wait complete. 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.608+0000 7f0eb028a640 1 Processor -- start 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.608+0000 7f0eb028a640 1 -- start start 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.608+0000 7f0eb028a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ea8072a60 0x7f0ea814d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.608+0000 7f0eb028a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea80adf10 0x7f0ea81f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.608+0000 7f0eb028a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea8161670 0x7f0ea81f0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.608+0000 7f0eb028a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea806b330 con 0x7f0ea8161670 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.608+0000 7f0eb028a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea806b050 con 0x7f0ea8072a60 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.608+0000 7f0eb028a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea806b610 con 0x7f0ea80adf10 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea8161670 0x7f0ea81f0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea8161670 0x7f0ea81f0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42172/0 (socket says 172.21.15.77:42172) 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 -- 172.21.15.77:0/4215246751 learned_addr learned my addr 172.21.15.77:0/4215246751 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae026640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ea8072a60 0x7f0ea814d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0ead825640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea80adf10 0x7f0ea81f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 -- 172.21.15.77:0/4215246751 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea80adf10 msgr2=0x7f0ea81f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea80adf10 0x7f0ea81f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 -- 172.21.15.77:0/4215246751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ea8072a60 msgr2=0x7f0ea814d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ea8072a60 0x7f0ea814d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 -- 172.21.15.77:0/4215246751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea81f4800 con 0x7f0ea8161670 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eae827640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea8161670 0x7f0ea81f0c30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0e980097f0 tx=0x7f0e980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0ea6ffd640 1 -- 172.21.15.77:0/4215246751 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98040070 con 0x7f0ea8161670 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0ea6ffd640 1 -- 172.21.15.77:0/4215246751 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e980399f0 con 0x7f0ea8161670 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0ea6ffd640 1 -- 172.21.15.77:0/4215246751 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98035040 con 0x7f0ea8161670 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea814d700 con 0x7f0ea8161670 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.609+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea814dcc0 con 0x7f0ea8161670 2024-03-06T07:22:51.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.610+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea804f330 con 0x7f0ea8161670 2024-03-06T07:22:51.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.611+0000 7f0ea6ffd640 1 -- 172.21.15.77:0/4215246751 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e9804b060 con 0x7f0ea8161670 2024-03-06T07:22:51.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.611+0000 7f0ea6ffd640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e9406b500 0x7f0e9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.611+0000 7f0ea6ffd640 1 -- 172.21.15.77:0/4215246751 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e980ab470 con 0x7f0ea8161670 2024-03-06T07:22:51.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.611+0000 7f0eae026640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e9406b500 0x7f0e9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.611+0000 7f0eae026640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e9406b500 0x7f0e9406d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0011340 tx=0x7f0ea000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.615+0000 7f0ea6ffd640 1 -- 172.21.15.77:0/4215246751 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ea81f4800 con 0x7f0ea8161670 2024-03-06T07:22:51.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.852+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f0ea8067590 con 0x7f0ea8161670 2024-03-06T07:22:51.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.854+0000 7f0ea6ffd640 1 -- 172.21.15.77:0/4215246751 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0e9807a080 con 0x7f0ea8161670 2024-03-06T07:22:51.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e9406b500 msgr2=0x7f0e9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e9406b500 0x7f0e9406d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0011340 tx=0x7f0ea000b040 comp rx=0 tx=0).stop 2024-03-06T07:22:51.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea8161670 msgr2=0x7f0ea81f0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea8161670 0x7f0ea81f0c30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0e980097f0 tx=0x7f0e980046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 shutdown_connections 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e9406b500 0x7f0e9406d9c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea8161670 0x7f0ea81f0c30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ea80adf10 0x7f0ea81f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 --2- 172.21.15.77:0/4215246751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ea8072a60 0x7f0ea814d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 >> 172.21.15.77:0/4215246751 conn(0x7f0ea80b4e60 msgr2=0x7f0ea80b2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 shutdown_connections 2024-03-06T07:22:51.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.856+0000 7f0eb028a640 1 -- 172.21.15.77:0/4215246751 wait complete. 2024-03-06T07:22:51.869 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:22:51.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.968+0000 7f0d0e58a640 1 Processor -- start 2024-03-06T07:22:51.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.968+0000 7f0d0e58a640 1 -- start start 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d0e58a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0d080cc920 0x7f0d0815eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d0e58a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d080b4a50 0x7f0d08150b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d0e58a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0806e770 con 0x7f0d0806e000 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d0e58a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0815e660 con 0x7f0d080cc920 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d0e58a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0806e490 con 0x7f0d080b4a50 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d077fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0d0806e000 0x7f0d08072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48998/0 (socket says 172.21.15.77:48998) 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d077fe640 1 -- 172.21.15.77:0/2074750530 learned_addr learned my addr 172.21.15.77:0/2074750530 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d07fff640 1 --2- 172.21.15.77:0/2074750530 >> v2:172.21.15.158:3300/0 conn(0x7f0d080cc920 0x7f0d0815eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d0cb27640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d080b4a50 0x7f0d08150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167181187 0 0) 0x7f0d0806e770 con 0x7f0d0806e000 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf0003520 con 0x7f0d0806e000 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696251078 0 0) 0x7f0cf0003520 con 0x7f0d0806e000 2024-03-06T07:22:51.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d080b4a50 msgr2=0x7f0d08150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d080b4a50 0x7f0d08150b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 >> v2:172.21.15.158:3300/0 conn(0x7f0d080cc920 msgr2=0x7f0d0815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 --2- 172.21.15.77:0/2074750530 >> v2:172.21.15.158:3300/0 conn(0x7f0d080cc920 0x7f0d0815eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0815e010 con 0x7f0d0806e000 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f0cf8001690 con 0x7f0d0806e000 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 >> v1:172.21.15.77:6789/0 conn(0x7f0d0806e000 legacy=0x7f0d08072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.969+0000 7f0d06ffd640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d080b4a50 0x7f0cf00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d06ffd640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0cf0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d06ffd640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf0004d40 0x7f0cf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0815e660 con 0x7f0cf0003f30 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0806e490 con 0x7f0cf0004d40 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf000ba10 con 0x7f0d080b4a50 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d077fe640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0cf0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d07fff640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d080b4a50 0x7f0cf00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d0cb27640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf0004d40 0x7f0cf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d077fe640 1 -- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d080b4a50 msgr2=0x7f0cf00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d077fe640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d080b4a50 0x7f0cf00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d077fe640 1 -- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf0004d40 msgr2=0x7f0cf00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d077fe640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf0004d40 0x7f0cf00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d077fe640 1 -- 172.21.15.77:0/2074750530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0003520 con 0x7f0cf0003f30 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.970+0000 7f0d0cb27640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cf0004d40 0x7f0cf00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d077fe640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0cf0004800 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800c210 tx=0x7f0cf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249e230781a7785 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8007d70 con 0x7f0cf0003f30 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf8030070 con 0x7f0cf0003f30 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d06ffd640 1 -- 172.21.15.77:0/2074750530 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8005d30 con 0x7f0cf0003f30 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d0e58a640 1 -- 172.21.15.77:0/2074750530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 msgr2=0x7f0cf0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d0e58a640 1 --2- 172.21.15.77:0/2074750530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0cf0004800 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800c210 tx=0x7f0cf0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d0cb27640 1 -- 172.21.15.77:0/2074750530 reap_dead start 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d0e58a640 1 -- 172.21.15.77:0/2074750530 shutdown_connections 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d0e58a640 1 -- 172.21.15.77:0/2074750530 >> 172.21.15.77:0/2074750530 conn(0x7f0d080b1c80 msgr2=0x7f0d080cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d0e58a640 1 -- 172.21.15.77:0/2074750530 shutdown_connections 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.971+0000 7f0d0e58a640 1 -- 172.21.15.77:0/2074750530 wait complete. 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.972+0000 7f0d0e58a640 1 Processor -- start 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d0e58a640 1 -- start start 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d0e58a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0d0814b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d0e58a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d0806c650 0x7f0d0806cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d0e58a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d0815d120 0x7f0d0815d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d0e58a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080cce80 con 0x7f0cf0003f30 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d0e58a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080ccd00 con 0x7f0d0806c650 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d0e58a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08070940 con 0x7f0d0815d120 2024-03-06T07:22:51.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d07fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0d0814b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d07fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0d0814b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42196/0 (socket says 172.21.15.77:42196) 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d07fff640 1 -- 172.21.15.77:0/871295746 learned_addr learned my addr 172.21.15.77:0/871295746 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d0cb27640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d0815d120 0x7f0d0815d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.973+0000 7f0d077fe640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d0806c650 0x7f0d0806cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d07fff640 1 -- 172.21.15.77:0/871295746 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d0815d120 msgr2=0x7f0d0815d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d07fff640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d0815d120 0x7f0d0815d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d07fff640 1 -- 172.21.15.77:0/871295746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d0806c650 msgr2=0x7f0d0806cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d07fff640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d0806c650 0x7f0d0806cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d07fff640 1 -- 172.21.15.77:0/871295746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0815dae0 con 0x7f0cf0003f30 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d07fff640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0d0814b3f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00e990 tx=0x7f0cfc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d04ff9640 1 -- 172.21.15.77:0/871295746 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc00d9f0 con 0x7f0cf0003f30 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d04ff9640 1 -- 172.21.15.77:0/871295746 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc00db90 con 0x7f0cf0003f30 2024-03-06T07:22:51.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0814f290 con 0x7f0cf0003f30 2024-03-06T07:22:51.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d04ff9640 1 -- 172.21.15.77:0/871295746 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc01b450 con 0x7f0cf0003f30 2024-03-06T07:22:51.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.974+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0814f850 con 0x7f0cf0003f30 2024-03-06T07:22:51.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.975+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0804f330 con 0x7f0cf0003f30 2024-03-06T07:22:51.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.975+0000 7f0d04ff9640 1 -- 172.21.15.77:0/871295746 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cfc013470 con 0x7f0cf0003f30 2024-03-06T07:22:51.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.976+0000 7f0d04ff9640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cf406b4d0 0x7f0cf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.976+0000 7f0d04ff9640 1 -- 172.21.15.77:0/871295746 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0cfc08a9a0 con 0x7f0cf0003f30 2024-03-06T07:22:51.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.976+0000 7f0d077fe640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cf406b4d0 0x7f0cf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.976+0000 7f0d077fe640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cf406b4d0 0x7f0cf406d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8034000 tx=0x7f0cf8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:51.980+0000 7f0d04ff9640 1 -- 172.21.15.77:0/871295746 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cfc0595b0 con 0x7f0cf0003f30 2024-03-06T07:22:52.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.222+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f0d080b40d0 con 0x7f0cf0003f30 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.224+0000 7f0d04ff9640 1 -- 172.21.15.77:0/871295746 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0cfc05d650 con 0x7f0cf0003f30 2024-03-06T07:22:52.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cf406b4d0 msgr2=0x7f0cf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cf406b4d0 0x7f0cf406d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8034000 tx=0x7f0cf8009030 comp rx=0 tx=0).stop 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 msgr2=0x7f0d0814b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0d0814b3f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00e990 tx=0x7f0cfc00ee50 comp rx=0 tx=0).stop 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 shutdown_connections 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d0815d120 0x7f0d0815d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d0806c650 0x7f0d0806cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cf406b4d0 0x7f0cf406d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 --2- 172.21.15.77:0/871295746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf0003f30 0x7f0d0814b3f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.225+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 >> 172.21.15.77:0/871295746 conn(0x7f0d080b1c80 msgr2=0x7f0d080ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.226+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 shutdown_connections 2024-03-06T07:22:52.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.226+0000 7f0d0e58a640 1 -- 172.21.15.77:0/871295746 wait complete. 2024-03-06T07:22:52.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:22:52.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.338+0000 7f79db681640 1 Processor -- start 2024-03-06T07:22:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.338+0000 7f79db681640 1 -- start start 2024-03-06T07:22:52.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79db681640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415e000 0x7f79d40d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79db681640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d40d25e0 con 0x7f79d40cf7d0 2024-03-06T07:22:52.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79db681640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d40d2880 con 0x7f79d40d4810 2024-03-06T07:22:52.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79db681640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c7c50 con 0x7f79d415e000 2024-03-06T07:22:52.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79d9c1e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f79d40cf7d0 0x7f79d40c87f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49002/0 (socket says 172.21.15.77:49002) 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3583007069 learned_addr learned my addr 172.21.15.77:0/3583007069 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79d941d640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415e000 0x7f79d40d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995531091 0 0) 0x7f79d40d25e0 con 0x7f79d40cf7d0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.339+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c0003520 con 0x7f79d40cf7d0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2104848083 0 0) 0x7f79c0003520 con 0x7f79d40cf7d0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415e000 msgr2=0x7f79d40d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415e000 0x7f79d40d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 >> v1:172.21.15.158:6789/0 conn(0x7f79d40d4810 legacy=0x7f79d40c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40d2b40 con 0x7f79d40cf7d0 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f79cc0019f0 con 0x7f79d40cf7d0 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 >> v1:172.21.15.77:6789/0 conn(0x7f79d40cf7d0 legacy=0x7f79d40c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c00039e0 0x7f79c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415e000 0x7f79c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79c0004d40 0x7f79c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c7c50 con 0x7f79c0004d40 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c000b9c0 con 0x7f79c00039e0 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c000bbc0 con 0x7f79d415e000 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79d9c1e640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79c0004d40 0x7f79c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79d941d640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c00039e0 0x7f79c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79d8c1c640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415e000 0x7f79c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415e000 msgr2=0x7f79c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79d9c1e640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415e000 0x7f79c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c00039e0 msgr2=0x7f79c0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79d9c1e640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c00039e0 0x7f79c0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.340+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3583007069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c0003520 con 0x7f79c0004d40 2024-03-06T07:22:52.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.341+0000 7f79d9c1e640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79c0004d40 0x7f79c00051c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00a920 tx=0x7f79c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7a4ae385ac6e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.341+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00a4c0 con 0x7f79c0004d40 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.341+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc034650 con 0x7f79c0004d40 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.341+0000 7f79cbfff640 1 -- 172.21.15.77:0/3583007069 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc034a40 con 0x7f79c0004d40 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.341+0000 7f79db681640 1 -- 172.21.15.77:0/3583007069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79c0004d40 msgr2=0x7f79c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.341+0000 7f79db681640 1 --2- 172.21.15.77:0/3583007069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79c0004d40 0x7f79c00051c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00a920 tx=0x7f79c0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.341+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3583007069 reap_dead start 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.342+0000 7f79db681640 1 -- 172.21.15.77:0/3583007069 shutdown_connections 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.342+0000 7f79db681640 1 -- 172.21.15.77:0/3583007069 >> 172.21.15.77:0/3583007069 conn(0x7f79d40d3cf0 msgr2=0x7f79d40d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.342+0000 7f79db681640 1 -- 172.21.15.77:0/3583007069 shutdown_connections 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.342+0000 7f79db681640 1 -- 172.21.15.77:0/3583007069 wait complete. 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.343+0000 7f79db681640 1 Processor -- start 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.343+0000 7f79db681640 1 -- start start 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.343+0000 7f79db681640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004d40 0x7f79d4160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.343+0000 7f79db681640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415f6e0 0x7f79d40b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79db681640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d40b3900 0x7f79d41f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79db681640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d415e3e0 con 0x7f79d40b3900 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79db681640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40d22d0 con 0x7f79c0004d40 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79db681640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c8f00 con 0x7f79d415f6e0 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d40b3900 0x7f79d41f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d40b3900 0x7f79d41f75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42202/0 (socket says 172.21.15.77:42202) 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3814801466 learned_addr learned my addr 172.21.15.77:0/3814801466 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d8c1c640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415f6e0 0x7f79d40b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d941d640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004d40 0x7f79d4160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3814801466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415f6e0 msgr2=0x7f79d40b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415f6e0 0x7f79d40b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3814801466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004d40 msgr2=0x7f79d4160c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004d40 0x7f79d4160c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 -- 172.21.15.77:0/3814801466 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40b3d80 con 0x7f79d40b3900 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79d9c1e640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d40b3900 0x7f79d41f75d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00a920 tx=0x7f79cc001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79c9ffb640 1 -- 172.21.15.77:0/3814801466 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00a4c0 con 0x7f79d40b3900 2024-03-06T07:22:52.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79c9ffb640 1 -- 172.21.15.77:0/3814801466 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc044070 con 0x7f79d40b3900 2024-03-06T07:22:52.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d415fb60 con 0x7f79d40b3900 2024-03-06T07:22:52.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79c9ffb640 1 -- 172.21.15.77:0/3814801466 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc034c00 con 0x7f79d40b3900 2024-03-06T07:22:52.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.344+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d4160120 con 0x7f79d40b3900 2024-03-06T07:22:52.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.345+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d404f330 con 0x7f79d40b3900 2024-03-06T07:22:52.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.346+0000 7f79c9ffb640 1 -- 172.21.15.77:0/3814801466 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79cc049460 con 0x7f79d40b3900 2024-03-06T07:22:52.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.346+0000 7f79c9ffb640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f79b806b4d0 0x7f79b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.346+0000 7f79c9ffb640 1 -- 172.21.15.77:0/3814801466 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f79cc0ac2e0 con 0x7f79d40b3900 2024-03-06T07:22:52.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.346+0000 7f79d941d640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f79b806b4d0 0x7f79b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.347+0000 7f79d941d640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f79b806b4d0 0x7f79b806d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009bc0 tx=0x7f79d0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.351+0000 7f79c9ffb640 1 -- 172.21.15.77:0/3814801466 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79cc07b540 con 0x7f79d40b3900 2024-03-06T07:22:52.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.586+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f79d40c9bd0 con 0x7f79d40b3900 2024-03-06T07:22:52.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.587+0000 7f79c9ffb640 1 -- 172.21.15.77:0/3814801466 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f79cc07ae40 con 0x7f79d40b3900 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f79b806b4d0 msgr2=0x7f79b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f79b806b4d0 0x7f79b806d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009bc0 tx=0x7f79d0009190 comp rx=0 tx=0).stop 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d40b3900 msgr2=0x7f79d41f75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d40b3900 0x7f79d41f75d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00a920 tx=0x7f79cc001dc0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 shutdown_connections 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f79b806b4d0 0x7f79b806d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d40b3900 0x7f79d41f75d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79d415f6e0 0x7f79d40b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 --2- 172.21.15.77:0/3814801466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79c0004d40 0x7f79d4160c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 >> 172.21.15.77:0/3814801466 conn(0x7f79d40d3cf0 msgr2=0x7f79d415d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 shutdown_connections 2024-03-06T07:22:52.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.590+0000 7f79db681640 1 -- 172.21.15.77:0/3814801466 wait complete. 2024-03-06T07:22:52.602 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:22:52.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.702+0000 7f429c283640 1 Processor -- start 2024-03-06T07:22:52.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.702+0000 7f429c283640 1 -- start start 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.702+0000 7f429c283640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f42940cf560 0x7f429406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.702+0000 7f429c283640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f42940d06b0 0x7f42940adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.702+0000 7f429c283640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429406d860 con 0x7f429415f1c0 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429c283640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940cf940 con 0x7f42940d06b0 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429c283640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940ad860 con 0x7f42940cf560 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429a820640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f42940d06b0 0x7f42940adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429981e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f429415f1c0 0x7f42940cffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49018/0 (socket says 172.21.15.77:49018) 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429981e640 1 -- 172.21.15.77:0/707507988 learned_addr learned my addr 172.21.15.77:0/707507988 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429a01f640 1 --2- 172.21.15.77:0/707507988 >> v2:172.21.15.158:3300/0 conn(0x7f42940cf560 0x7f429406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429a820640 1 -- 172.21.15.77:0/707507988 >> v2:172.21.15.158:3300/0 conn(0x7f42940cf560 msgr2=0x7f429406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429a820640 1 --2- 172.21.15.77:0/707507988 >> v2:172.21.15.158:3300/0 conn(0x7f42940cf560 0x7f429406d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429a820640 1 -- 172.21.15.77:0/707507988 >> v1:172.21.15.77:6789/0 conn(0x7f429415f1c0 legacy=0x7f42940cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429a820640 1 -- 172.21.15.77:0/707507988 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429406db40 con 0x7f42940d06b0 2024-03-06T07:22:52.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.703+0000 7f429a01f640 1 --2- 172.21.15.77:0/707507988 >> v2:172.21.15.158:3300/0 conn(0x7f42940cf560 0x7f429406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:52.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429a820640 1 --2- 172.21.15.77:0/707507988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42940d06b0 0x7f42940adfd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4294068410 tx=0x7f428c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0db2bb906bdb533 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429901d640 1 -- 172.21.15.77:0/707507988 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c039070 con 0x7f42940d06b0 2024-03-06T07:22:52.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429901d640 1 -- 172.21.15.77:0/707507988 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428c02cb60 con 0x7f42940d06b0 2024-03-06T07:22:52.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429901d640 1 -- 172.21.15.77:0/707507988 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c034440 con 0x7f42940d06b0 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429c283640 1 -- 172.21.15.77:0/707507988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42940d06b0 msgr2=0x7f42940adfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429c283640 1 --2- 172.21.15.77:0/707507988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42940d06b0 0x7f42940adfd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4294068410 tx=0x7f428c02bdb0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429c283640 1 -- 172.21.15.77:0/707507988 shutdown_connections 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429c283640 1 --2- 172.21.15.77:0/707507988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42940d06b0 0x7f42940adfd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429c283640 1 --2- 172.21.15.77:0/707507988 >> v2:172.21.15.158:3300/0 conn(0x7f42940cf560 0x7f429406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.704+0000 7f429c283640 1 -- 172.21.15.77:0/707507988 >> 172.21.15.77:0/707507988 conn(0x7f429406b190 msgr2=0x7f429406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.705+0000 7f429c283640 1 -- 172.21.15.77:0/707507988 shutdown_connections 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.705+0000 7f429c283640 1 -- 172.21.15.77:0/707507988 wait complete. 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429c283640 1 Processor -- start 2024-03-06T07:22:52.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429c283640 1 -- start start 2024-03-06T07:22:52.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429c283640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42940cf560 0x7f4294146790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429c283640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42940d06b0 0x7f429415b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429c283640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429415f1c0 0x7f42941f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429c283640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429406d550 con 0x7f429415f1c0 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429c283640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42940adab0 con 0x7f42940d06b0 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429c283640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42940ae510 con 0x7f42940cf560 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429a820640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429415f1c0 0x7f42941f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.706+0000 7f429a820640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429415f1c0 0x7f42941f4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42234/0 (socket says 172.21.15.77:42234) 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429a820640 1 -- 172.21.15.77:0/4115955560 learned_addr learned my addr 172.21.15.77:0/4115955560 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429a01f640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42940cf560 0x7f4294146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429981e640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42940d06b0 0x7f429415b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429a820640 1 -- 172.21.15.77:0/4115955560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42940cf560 msgr2=0x7f4294146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429a820640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42940cf560 0x7f4294146790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429a820640 1 -- 172.21.15.77:0/4115955560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42940d06b0 msgr2=0x7f429415b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429a820640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42940d06b0 0x7f429415b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429a820640 1 -- 172.21.15.77:0/4115955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4294150bc0 con 0x7f429415f1c0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429a820640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429415f1c0 0x7f42941f4060 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f428c037040 tx=0x7f428c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f4286ffd640 1 -- 172.21.15.77:0/4115955560 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c039050 con 0x7f429415f1c0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f4286ffd640 1 -- 172.21.15.77:0/4115955560 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428c045070 con 0x7f429415f1c0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429415cc90 con 0x7f429415f1c0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f4286ffd640 1 -- 172.21.15.77:0/4115955560 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c03ea50 con 0x7f429415f1c0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.707+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429415d250 con 0x7f429415f1c0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.708+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429415da20 con 0x7f429415f1c0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.708+0000 7f4286ffd640 1 -- 172.21.15.77:0/4115955560 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f428c02cd10 con 0x7f429415f1c0 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.709+0000 7f4286ffd640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f427806b4d0 0x7f427806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.709+0000 7f429a01f640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f427806b4d0 0x7f427806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.709+0000 7f4286ffd640 1 -- 172.21.15.77:0/4115955560 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f428c0aca80 con 0x7f429415f1c0 2024-03-06T07:22:52.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.709+0000 7f429a01f640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f427806b4d0 0x7f427806d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f428800a990 tx=0x7f4288008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.713+0000 7f4286ffd640 1 -- 172.21.15.77:0/4115955560 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f428c032e20 con 0x7f429415f1c0 2024-03-06T07:22:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.951+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f429404f330 con 0x7f429415f1c0 2024-03-06T07:22:52.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.953+0000 7f4286ffd640 1 -- 172.21.15.77:0/4115955560 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f428c07bce0 con 0x7f429415f1c0 2024-03-06T07:22:52.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f427806b4d0 msgr2=0x7f427806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f427806b4d0 0x7f427806d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f428800a990 tx=0x7f4288008040 comp rx=0 tx=0).stop 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429415f1c0 msgr2=0x7f42941f4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429415f1c0 0x7f42941f4060 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f428c037040 tx=0x7f428c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 shutdown_connections 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f427806b4d0 0x7f427806d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429415f1c0 0x7f42941f4060 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42940d06b0 0x7f429415b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 --2- 172.21.15.77:0/4115955560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42940cf560 0x7f4294146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 >> 172.21.15.77:0/4115955560 conn(0x7f429406b190 msgr2=0x7f429406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 shutdown_connections 2024-03-06T07:22:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:52.955+0000 7f429c283640 1 -- 172.21.15.77:0/4115955560 wait complete. 2024-03-06T07:22:52.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:22:53.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.066+0000 7fdefcc55640 1 Processor -- start 2024-03-06T07:22:53.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.066+0000 7fdefcc55640 1 -- start start 2024-03-06T07:22:53.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.067+0000 7fdefcc55640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdef80d0d90 0x7fdef80ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.067+0000 7fdefcc55640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef80c90f0 0x7fdef80ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.067+0000 7fdefcc55640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef80d25e0 con 0x7fdef815e000 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.067+0000 7fdefcc55640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef80c7c50 con 0x7fdef80d0d90 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.067+0000 7fdefcc55640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80d2880 con 0x7fdef80c90f0 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef5d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdef80d0d90 0x7fdef80ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef5d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdef80d0d90 0x7fdef80ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42244/0 (socket says 172.21.15.77:42244) 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef5d9b640 1 -- 172.21.15.77:0/199966898 learned_addr learned my addr 172.21.15.77:0/199966898 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef6d9d640 1 --2- 172.21.15.77:0/199966898 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef80c90f0 0x7fdef80ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef5d9b640 1 -- 172.21.15.77:0/199966898 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef80c90f0 msgr2=0x7fdef80ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef5d9b640 1 --2- 172.21.15.77:0/199966898 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef80c90f0 0x7fdef80ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef5d9b640 1 -- 172.21.15.77:0/199966898 >> v1:172.21.15.158:6789/0 conn(0x7fdef815e000 legacy=0x7fdef80d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef5d9b640 1 -- 172.21.15.77:0/199966898 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef80d2b40 con 0x7fdef80d0d90 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.068+0000 7fdef6d9d640 1 --2- 172.21.15.77:0/199966898 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef80c90f0 0x7fdef80ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdef5d9b640 1 --2- 172.21.15.77:0/199966898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80d0d90 0x7fdef80ce7c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009870 tx=0x7fdeec02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1fcd9c936943a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdef559a640 1 -- 172.21.15.77:0/199966898 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec02f830 con 0x7fdef80d0d90 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdef559a640 1 -- 172.21.15.77:0/199966898 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec02f9d0 con 0x7fdef80d0d90 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdef559a640 1 -- 172.21.15.77:0/199966898 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec02fda0 con 0x7fdef80d0d90 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdefcc55640 1 -- 172.21.15.77:0/199966898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80d0d90 msgr2=0x7fdef80ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdefcc55640 1 --2- 172.21.15.77:0/199966898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80d0d90 0x7fdef80ce7c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009870 tx=0x7fdeec02ba10 comp rx=0 tx=0).stop 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdefcc55640 1 -- 172.21.15.77:0/199966898 shutdown_connections 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdefcc55640 1 --2- 172.21.15.77:0/199966898 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef80c90f0 0x7fdef80ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdefcc55640 1 --2- 172.21.15.77:0/199966898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80d0d90 0x7fdef80ce7c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdefcc55640 1 -- 172.21.15.77:0/199966898 >> 172.21.15.77:0/199966898 conn(0x7fdef80d3cf0 msgr2=0x7fdef80d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdefcc55640 1 -- 172.21.15.77:0/199966898 shutdown_connections 2024-03-06T07:22:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.069+0000 7fdefcc55640 1 -- 172.21.15.77:0/199966898 wait complete. 2024-03-06T07:22:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdefcc55640 1 Processor -- start 2024-03-06T07:22:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdefcc55640 1 -- start start 2024-03-06T07:22:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdefcc55640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80c90f0 0x7fdef81e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdefcc55640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef80d0d90 0x7fdef81e8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdefcc55640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef815e000 0x7fdef81e95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdefcc55640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef815e3e0 con 0x7fdef80c90f0 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdefcc55640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80d22d0 con 0x7fdef80d0d90 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdefcc55640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80d4810 con 0x7fdef815e000 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdef659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80c90f0 0x7fdef81e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdef659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80c90f0 0x7fdef81e43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42246/0 (socket says 172.21.15.77:42246) 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdef659c640 1 -- 172.21.15.77:0/3922756444 learned_addr learned my addr 172.21.15.77:0/3922756444 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdef6d9d640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef815e000 0x7fdef81e95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.071+0000 7fdef5d9b640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef80d0d90 0x7fdef81e8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdef659c640 1 -- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef815e000 msgr2=0x7fdef81e95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdef659c640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef815e000 0x7fdef81e95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdef659c640 1 -- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef80d0d90 msgr2=0x7fdef81e8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdef659c640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef80d0d90 0x7fdef81e8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdef659c640 1 -- 172.21.15.77:0/3922756444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef81e3310 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdef5d9b640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef80d0d90 0x7fdef81e8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdef659c640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80c90f0 0x7fdef81e43b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c3c0 tx=0x7fdef000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdee6ffd640 1 -- 172.21.15.77:0/3922756444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef00052d0 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdee6ffd640 1 -- 172.21.15.77:0/3922756444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef0005490 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef81e35f0 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdee6ffd640 1 -- 172.21.15.77:0/3922756444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0007260 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.072+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef81e3bb0 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.073+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef804f330 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.074+0000 7fdee6ffd640 1 -- 172.21.15.77:0/3922756444 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdef0007400 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.074+0000 7fdee6ffd640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.074+0000 7fdee6ffd640 1 -- 172.21.15.77:0/3922756444 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdef0089250 con 0x7fdef80c90f0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.074+0000 7fdef5d9b640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.074+0000 7fdef5d9b640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded806b4d0 0x7fded806d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdeec036000 tx=0x7fdeec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.078+0000 7fdee6ffd640 1 -- 172.21.15.77:0/3922756444 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdef0057e60 con 0x7fdef80c90f0 2024-03-06T07:22:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.316+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fdef80b6fe0 con 0x7fdef80c90f0 2024-03-06T07:22:53.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.318+0000 7fdee6ffd640 1 -- 172.21.15.77:0/3922756444 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdef005ce20 con 0x7fdef80c90f0 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded806b4d0 msgr2=0x7fded806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded806b4d0 0x7fded806d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdeec036000 tx=0x7fdeec008030 comp rx=0 tx=0).stop 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80c90f0 msgr2=0x7fdef81e43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80c90f0 0x7fdef81e43b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c3c0 tx=0x7fdef000c880 comp rx=0 tx=0).stop 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 shutdown_connections 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef815e000 0x7fdef81e95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef80d0d90 0x7fdef81e8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 --2- 172.21.15.77:0/3922756444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef80c90f0 0x7fdef81e43b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 >> 172.21.15.77:0/3922756444 conn(0x7fdef80d3cf0 msgr2=0x7fdef80b8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 shutdown_connections 2024-03-06T07:22:53.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.321+0000 7fdefcc55640 1 -- 172.21.15.77:0/3922756444 wait complete. 2024-03-06T07:22:53.334 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:22:53.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.434+0000 7f85d0765640 1 Processor -- start 2024-03-06T07:22:53.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.434+0000 7f85d0765640 1 -- start start 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85d0765640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85c8072520 0x7f85c8066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85d0765640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c80b5c30 0x7f85c8072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85d0765640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c8072e40 con 0x7f85c815d4f0 2024-03-06T07:22:53.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85d0765640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c8067240 con 0x7f85c8072520 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85d0765640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c806c460 con 0x7f85c80b5c30 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85ced02640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f85c815d4f0 0x7f85c815d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49022/0 (socket says 172.21.15.77:49022) 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85ced02640 1 -- 172.21.15.77:0/164685910 learned_addr learned my addr 172.21.15.77:0/164685910 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250118125 0 0) 0x7f85c8072e40 con 0x7f85c815d4f0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85b4003520 con 0x7f85c815d4f0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85ce501640 1 --2- 172.21.15.77:0/164685910 >> v2:172.21.15.158:3300/0 conn(0x7f85c8072520 0x7f85c8066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cdd00640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c80b5c30 0x7f85c8072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 703607299 0 0) 0x7f85b4003520 con 0x7f85c815d4f0 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c80b5c30 msgr2=0x7f85c8072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c80b5c30 0x7f85c8072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 >> v2:172.21.15.158:3300/0 conn(0x7f85c8072520 msgr2=0x7f85c8066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 --2- 172.21.15.77:0/164685910 >> v2:172.21.15.158:3300/0 conn(0x7f85c8072520 0x7f85c8066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c806c740 con 0x7f85c815d4f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f85c00019f0 con 0x7f85c815d4f0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 >> v1:172.21.15.77:6789/0 conn(0x7f85c815d4f0 legacy=0x7f85c815d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c80b5c30 0x7f85b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b4003f30 0x7f85b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c806c460 con 0x7f85b4004d40 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b400b9c0 con 0x7f85b4003f30 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.435+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b400bb90 con 0x7f85c80b5c30 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.436+0000 7f85ced02640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.436+0000 7f85ce501640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c80b5c30 0x7f85b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.436+0000 7f85cdd00640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b4003f30 0x7f85b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.436+0000 7f85ced02640 1 -- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c80b5c30 msgr2=0x7f85b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.436+0000 7f85ced02640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c80b5c30 0x7f85b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.436+0000 7f85ced02640 1 -- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b4003f30 msgr2=0x7f85b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.436+0000 7f85ced02640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b4003f30 0x7f85b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.436+0000 7f85ced02640 1 -- 172.21.15.77:0/164685910 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b4003520 con 0x7f85b4004d40 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85ced02640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85b40051c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f85c000a8e0 tx=0x7f85b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95e0a74148b1c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c000a4d0 con 0x7f85b4004d40 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c00072d0 con 0x7f85b4004d40 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85cd4ff640 1 -- 172.21.15.77:0/164685910 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c00076c0 con 0x7f85b4004d40 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85d0765640 1 -- 172.21.15.77:0/164685910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 msgr2=0x7f85b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85d0765640 1 --2- 172.21.15.77:0/164685910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85b40051c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f85c000a8e0 tx=0x7f85b4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85ced02640 1 -- 172.21.15.77:0/164685910 reap_dead start 2024-03-06T07:22:53.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85d0765640 1 -- 172.21.15.77:0/164685910 shutdown_connections 2024-03-06T07:22:53.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85d0765640 1 -- 172.21.15.77:0/164685910 >> 172.21.15.77:0/164685910 conn(0x7f85c80b4e60 msgr2=0x7f85c80b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85d0765640 1 -- 172.21.15.77:0/164685910 shutdown_connections 2024-03-06T07:22:53.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.437+0000 7f85d0765640 1 -- 172.21.15.77:0/164685910 wait complete. 2024-03-06T07:22:53.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.438+0000 7f85d0765640 1 Processor -- start 2024-03-06T07:22:53.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.438+0000 7f85d0765640 1 -- start start 2024-03-06T07:22:53.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85d0765640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85c81f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85d0765640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c81f12b0 0x7f85c81f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85d0765640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c80af790 0x7f85c80afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85d0765640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c815e050 con 0x7f85b4004d40 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85d0765640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c815fe30 con 0x7f85c80af790 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85d0765640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c815d040 con 0x7f85c81f12b0 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85ce501640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85c81f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85ce501640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85c81f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42270/0 (socket says 172.21.15.77:42270) 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85ce501640 1 -- 172.21.15.77:0/1091045677 learned_addr learned my addr 172.21.15.77:0/1091045677 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85ce501640 1 -- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c81f12b0 msgr2=0x7f85c81f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85ced02640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c80af790 0x7f85c80afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.439+0000 7f85cdd00640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c81f12b0 0x7f85c81f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85ce501640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c81f12b0 0x7f85c81f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85ce501640 1 -- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c80af790 msgr2=0x7f85c80afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85ce501640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c80af790 0x7f85c80afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85ce501640 1 -- 172.21.15.77:0/1091045677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c80b0530 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85ced02640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c80af790 0x7f85c80afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85cdd00640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c81f12b0 0x7f85c81f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85ce501640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85c81f4bd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f85c4011aa0 tx=0x7f85c4011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85beffd640 1 -- 172.21.15.77:0/1091045677 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c400fb80 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85beffd640 1 -- 172.21.15.77:0/1091045677 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c400fd20 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c80b0810 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85beffd640 1 -- 172.21.15.77:0/1091045677 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4014660 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.440+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c80b0dd0 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.441+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c80b1110 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.441+0000 7f85beffd640 1 -- 172.21.15.77:0/1091045677 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85c4012040 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.442+0000 7f85beffd640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85ac06b4d0 0x7f85ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.442+0000 7f85beffd640 1 -- 172.21.15.77:0/1091045677 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f85c408a580 con 0x7f85b4004d40 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.442+0000 7f85cdd00640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85ac06b4d0 0x7f85ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.442+0000 7f85cdd00640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85ac06b4d0 0x7f85ac06d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f85c81e7520 tx=0x7f85b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.446+0000 7f85beffd640 1 -- 172.21.15.77:0/1091045677 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85c80b1110 con 0x7f85b4004d40 2024-03-06T07:22:53.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.684+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f85c804f330 con 0x7f85b4004d40 2024-03-06T07:22:53.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.686+0000 7f85beffd640 1 -- 172.21.15.77:0/1091045677 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f85c4019030 con 0x7f85b4004d40 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85ac06b4d0 msgr2=0x7f85ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85ac06b4d0 0x7f85ac06d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f85c81e7520 tx=0x7f85b8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 msgr2=0x7f85c81f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85c81f4bd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f85c4011aa0 tx=0x7f85c4011f60 comp rx=0 tx=0).stop 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 shutdown_connections 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85c80af790 0x7f85c80afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85c81f12b0 0x7f85c81f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85ac06b4d0 0x7f85ac06d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 --2- 172.21.15.77:0/1091045677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b4004d40 0x7f85c81f4bd0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.688+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 >> 172.21.15.77:0/1091045677 conn(0x7f85c80b4e60 msgr2=0x7f85c806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.689+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 shutdown_connections 2024-03-06T07:22:53.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.689+0000 7f85d0765640 1 -- 172.21.15.77:0/1091045677 wait complete. 2024-03-06T07:22:53.701 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:53.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.799+0000 7f9c625c7640 1 Processor -- start 2024-03-06T07:22:53.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.799+0000 7f9c625c7640 1 -- start start 2024-03-06T07:22:53.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c625c7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c5c0cf100 0x7f9c5c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c625c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c0cc810 0x7f9c5c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c625c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c5c06e0a0 0x7f9c5c161a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c625c7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5c0ccc70 con 0x7f9c5c06e0a0 2024-03-06T07:22:53.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c625c7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5c15e440 con 0x7f9c5c0cf100 2024-03-06T07:22:53.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c625c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c15e700 con 0x7f9c5c0cc810 2024-03-06T07:22:53.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c60b64640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c5c06e0a0 0x7f9c5c161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c60b64640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c5c06e0a0 0x7f9c5c161a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42272/0 (socket says 172.21.15.77:42272) 2024-03-06T07:22:53.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c60b64640 1 -- 172.21.15.77:0/803598005 learned_addr learned my addr 172.21.15.77:0/803598005 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:53.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c60b64640 1 -- 172.21.15.77:0/803598005 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c0cc810 msgr2=0x7f9c5c06db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c60b64640 1 --2- 172.21.15.77:0/803598005 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c0cc810 0x7f9c5c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c60b64640 1 -- 172.21.15.77:0/803598005 >> v2:172.21.15.158:3300/0 conn(0x7f9c5c0cf100 msgr2=0x7f9c5c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c60b64640 1 --2- 172.21.15.77:0/803598005 >> v2:172.21.15.158:3300/0 conn(0x7f9c5c0cf100 0x7f9c5c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.800+0000 7f9c60b64640 1 -- 172.21.15.77:0/803598005 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c15e9e0 con 0x7f9c5c06e0a0 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.801+0000 7f9c60b64640 1 --2- 172.21.15.77:0/803598005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c06e0a0 0x7f9c5c161a30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c0cf800 tx=0x7f9c5402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c2a503ecb71802 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.801+0000 7f9c5affd640 1 -- 172.21.15.77:0/803598005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5402c9e0 con 0x7f9c5c06e0a0 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.801+0000 7f9c5affd640 1 -- 172.21.15.77:0/803598005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5402cb80 con 0x7f9c5c06e0a0 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.801+0000 7f9c5affd640 1 -- 172.21.15.77:0/803598005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c540343f0 con 0x7f9c5c06e0a0 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 -- 172.21.15.77:0/803598005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c06e0a0 msgr2=0x7f9c5c161a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 --2- 172.21.15.77:0/803598005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c06e0a0 0x7f9c5c161a30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c0cf800 tx=0x7f9c5402be40 comp rx=0 tx=0).stop 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 -- 172.21.15.77:0/803598005 shutdown_connections 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 --2- 172.21.15.77:0/803598005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c06e0a0 0x7f9c5c161a30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 --2- 172.21.15.77:0/803598005 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c0cc810 0x7f9c5c06db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 --2- 172.21.15.77:0/803598005 >> v2:172.21.15.158:3300/0 conn(0x7f9c5c0cf100 0x7f9c5c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 -- 172.21.15.77:0/803598005 >> 172.21.15.77:0/803598005 conn(0x7f9c5c0cf8a0 msgr2=0x7f9c5c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 -- 172.21.15.77:0/803598005 shutdown_connections 2024-03-06T07:22:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.802+0000 7f9c625c7640 1 -- 172.21.15.77:0/803598005 wait complete. 2024-03-06T07:22:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.803+0000 7f9c625c7640 1 Processor -- start 2024-03-06T07:22:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.803+0000 7f9c625c7640 1 -- start start 2024-03-06T07:22:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c625c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c06e0a0 0x7f9c5c1483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c625c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c0cc810 0x7f9c5c15d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c625c7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5c0cf100 0x7f9c5c1f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c625c7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c0cf4e0 con 0x7f9c5c0cc810 2024-03-06T07:22:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c625c7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c15efa0 con 0x7f9c5c0cf100 2024-03-06T07:22:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c625c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c06e510 con 0x7f9c5c06e0a0 2024-03-06T07:22:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c0cc810 0x7f9c5c15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c0cc810 0x7f9c5c15d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42288/0 (socket says 172.21.15.77:42288) 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 -- 172.21.15.77:0/1466437175 learned_addr learned my addr 172.21.15.77:0/1466437175 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5bfff640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c06e0a0 0x7f9c5c1483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c60b64640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5c0cf100 0x7f9c5c1f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 -- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c06e0a0 msgr2=0x7f9c5c1483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c06e0a0 0x7f9c5c1483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 -- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5c0cf100 msgr2=0x7f9c5c1f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5c0cf100 0x7f9c5c1f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 -- 172.21.15.77:0/1466437175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c157d30 con 0x7f9c5c0cc810 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c60b64640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5c0cf100 0x7f9c5c1f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c5b7fe640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c0cc810 0x7f9c5c15d050 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9c5000b440 tx=0x7f9c5000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c58ff9640 1 -- 172.21.15.77:0/1466437175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c50015020 con 0x7f9c5c0cc810 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c58ff9640 1 -- 172.21.15.77:0/1466437175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c50011830 con 0x7f9c5c0cc810 2024-03-06T07:22:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c58ff9640 1 -- 172.21.15.77:0/1466437175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c50011c20 con 0x7f9c5c0cc810 2024-03-06T07:22:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c158010 con 0x7f9c5c0cc810 2024-03-06T07:22:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.804+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c1585d0 con 0x7f9c5c0cc810 2024-03-06T07:22:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.805+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c0682b0 con 0x7f9c5c0cc810 2024-03-06T07:22:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.806+0000 7f9c58ff9640 1 -- 172.21.15.77:0/1466437175 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c50018020 con 0x7f9c5c0cc810 2024-03-06T07:22:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.806+0000 7f9c58ff9640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4806b500 0x7f9c4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.806+0000 7f9c58ff9640 1 -- 172.21.15.77:0/1466437175 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9c50089d20 con 0x7f9c5c0cc810 2024-03-06T07:22:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.806+0000 7f9c5bfff640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4806b500 0x7f9c4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.806+0000 7f9c5bfff640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4806b500 0x7f9c4806d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c1f75f0 tx=0x7f9c4c009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.810+0000 7f9c58ff9640 1 -- 172.21.15.77:0/1466437175 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c5c0682b0 con 0x7f9c5c0cc810 2024-03-06T07:22:53.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.983+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c5c04f330 con 0x7f9c5c0cc810 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.984+0000 7f9c58ff9640 1 -- 172.21.15.77:0/1466437175 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f9c5001f020 con 0x7f9c5c0cc810 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:53.986 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:51.126676+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4806b500 msgr2=0x7f9c4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4806b500 0x7f9c4806d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c1f75f0 tx=0x7f9c4c009110 comp rx=0 tx=0).stop 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c0cc810 msgr2=0x7f9c5c15d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c0cc810 0x7f9c5c15d050 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9c5000b440 tx=0x7f9c5000b840 comp rx=0 tx=0).stop 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 shutdown_connections 2024-03-06T07:22:53.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4806b500 0x7f9c4806d9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c5c0cc810 0x7f9c5c15d050 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c5c0cf100 0x7f9c5c1f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 --2- 172.21.15.77:0/1466437175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c5c06e0a0 0x7f9c5c1483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 >> 172.21.15.77:0/1466437175 conn(0x7f9c5c0cf8a0 msgr2=0x7f9c5c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 shutdown_connections 2024-03-06T07:22:53.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:53.986+0000 7f9c625c7640 1 -- 172.21.15.77:0/1466437175 wait complete. 2024-03-06T07:22:53.999 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:53.999 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.100+0000 7f7ec12af640 1 Processor -- start 2024-03-06T07:22:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.100+0000 7f7ec12af640 1 -- start start 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ec12af640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ebc0cf9a0 0x7f7ebc0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ec12af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ec12af640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc06b710 con 0x7f7ebc066d90 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ec12af640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ebc06b150 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ec12af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc06b430 con 0x7f7ebc0b55e0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ebb7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ebaffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7ebc0cf9a0 0x7f7ebc0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ebb7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:49178/0 (socket says 172.21.15.77:49178) 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.101+0000 7f7ebb7fe640 1 -- 172.21.15.77:0/1045946572 learned_addr learned my addr 172.21.15.77:0/1045946572 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.102+0000 7f7eb9ffb640 1 -- 172.21.15.77:0/1045946572 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756260213 0 0) 0x7f7ebc06b710 con 0x7f7ebc066d90 2024-03-06T07:22:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.102+0000 7f7eb9ffb640 1 -- 172.21.15.77:0/1045946572 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea4003520 con 0x7f7ebc066d90 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.102+0000 7f7ebb7fe640 1 -- 172.21.15.77:0/1045946572 >> v2:172.21.15.158:3300/0 conn(0x7f7ebc0cf9a0 msgr2=0x7f7ebc0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.102+0000 7f7ebb7fe640 1 --2- 172.21.15.77:0/1045946572 >> v2:172.21.15.158:3300/0 conn(0x7f7ebc0cf9a0 0x7f7ebc0b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.102+0000 7f7ebb7fe640 1 -- 172.21.15.77:0/1045946572 >> v1:172.21.15.158:6789/0 conn(0x7f7ebc066d90 legacy=0x7f7ebc0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.102+0000 7f7ebb7fe640 1 -- 172.21.15.77:0/1045946572 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc0b0970 con 0x7f7ebc0b55e0 2024-03-06T07:22:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.102+0000 7f7ebaffd640 1 --2- 172.21.15.77:0/1045946572 >> v2:172.21.15.158:3300/0 conn(0x7f7ebc0cf9a0 0x7f7ebc0b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ebb7fe640 1 --2- 172.21.15.77:0/1045946572 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0ce040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009e50 tx=0x7f7eb402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486ddc25276a65f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7eb9ffb640 1 -- 172.21.15.77:0/1045946572 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb402cb00 con 0x7f7ebc0b55e0 2024-03-06T07:22:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7eb9ffb640 1 -- 172.21.15.77:0/1045946572 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb402cca0 con 0x7f7ebc0b55e0 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7eb9ffb640 1 -- 172.21.15.77:0/1045946572 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb40345f0 con 0x7f7ebc0b55e0 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ec12af640 1 -- 172.21.15.77:0/1045946572 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 msgr2=0x7f7ebc0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ec12af640 1 --2- 172.21.15.77:0/1045946572 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0ce040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009e50 tx=0x7f7eb402bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ec12af640 1 -- 172.21.15.77:0/1045946572 shutdown_connections 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ec12af640 1 --2- 172.21.15.77:0/1045946572 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0ce040 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ec12af640 1 --2- 172.21.15.77:0/1045946572 >> v2:172.21.15.158:3300/0 conn(0x7f7ebc0cf9a0 0x7f7ebc0b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ec12af640 1 -- 172.21.15.77:0/1045946572 >> 172.21.15.77:0/1045946572 conn(0x7f7ebc0b1d70 msgr2=0x7f7ebc0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ec12af640 1 -- 172.21.15.77:0/1045946572 shutdown_connections 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.103+0000 7f7ec12af640 1 -- 172.21.15.77:0/1045946572 wait complete. 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ec12af640 1 Processor -- start 2024-03-06T07:22:54.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ec12af640 1 -- start start 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ec12af640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ebc066d90 0x7f7ebc160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ec12af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ec12af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ebc0cf9a0 0x7f7ebc0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ec12af640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc0b6170 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ec12af640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc0b5ff0 con 0x7f7ebc066d90 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ec12af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc0cfd80 con 0x7f7ebc0b55e0 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ebb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ebc0cf9a0 0x7f7ebc0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ebb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ebc0cf9a0 0x7f7ebc0c9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42296/0 (socket says 172.21.15.77:42296) 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.105+0000 7f7ebb7fe640 1 -- 172.21.15.77:0/2198249907 learned_addr learned my addr 172.21.15.77:0/2198249907 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ebaffd640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ebc066d90 0x7f7ebc160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7eba7fc640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ebb7fe640 1 -- 172.21.15.77:0/2198249907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 msgr2=0x7f7ebc0c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ebb7fe640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ebb7fe640 1 -- 172.21.15.77:0/2198249907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ebc066d90 msgr2=0x7f7ebc160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ebb7fe640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ebc066d90 0x7f7ebc160ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ebb7fe640 1 -- 172.21.15.77:0/2198249907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc0c89d0 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ebb7fe640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ebc0cf9a0 0x7f7ebc0c9a50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4036040 tx=0x7f7eb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ea37fe640 1 -- 172.21.15.77:0/2198249907 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4042020 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ea37fe640 1 -- 172.21.15.77:0/2198249907 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb402ce50 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc0c8cb0 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ea37fe640 1 -- 172.21.15.77:0/2198249907 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4009510 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.106+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc09bcc0 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.107+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ebc09e3d0 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.108+0000 7f7ea37fe640 1 -- 172.21.15.77:0/2198249907 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7eb4045020 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.108+0000 7f7ea37fe640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ea806b4d0 0x7f7ea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.108+0000 7f7ea37fe640 1 -- 172.21.15.77:0/2198249907 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7eb40abe60 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.108+0000 7f7ebaffd640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ea806b4d0 0x7f7ea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.108+0000 7f7ebaffd640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ea806b4d0 0x7f7ea806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009560 tx=0x7f7eac009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.112+0000 7f7ea37fe640 1 -- 172.21.15.77:0/2198249907 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7eb404c030 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.286+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ebc068ff0 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.286+0000 7f7ea37fe640 1 -- 172.21.15.77:0/2198249907 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7eb407ab30 con 0x7f7ebc0cf9a0 2024-03-06T07:22:54.288 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:22:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ea806b4d0 msgr2=0x7f7ea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ea806b4d0 0x7f7ea806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009560 tx=0x7f7eac009190 comp rx=0 tx=0).stop 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ebc0cf9a0 msgr2=0x7f7ebc0c9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ebc0cf9a0 0x7f7ebc0c9a50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4036040 tx=0x7f7eb4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 shutdown_connections 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ea806b4d0 0x7f7ea806d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ebc0cf9a0 0x7f7ebc0c9a50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7ebc0b55e0 0x7f7ebc0c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 --2- 172.21.15.77:0/2198249907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ebc066d90 0x7f7ebc160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 >> 172.21.15.77:0/2198249907 conn(0x7f7ebc0b1d70 msgr2=0x7f7ebc161670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 shutdown_connections 2024-03-06T07:22:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.288+0000 7f7ec12af640 1 -- 172.21.15.77:0/2198249907 wait complete. 2024-03-06T07:22:54.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:54.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.400+0000 7f213cdf7640 1 Processor -- start 2024-03-06T07:22:54.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.400+0000 7f213cdf7640 1 -- start start 2024-03-06T07:22:54.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213cdf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f213813dc10 0x7f21380b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213cdf7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f213815bf30 0x7f21380b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213cdf7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213815f580 con 0x7f213815e850 2024-03-06T07:22:54.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213cdf7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21380cf3f0 con 0x7f213813dc10 2024-03-06T07:22:54.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213cdf7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213815b210 con 0x7f213815bf30 2024-03-06T07:22:54.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f213813dc10 0x7f21380b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f2136d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f213815e850 0x7f213815ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49038/0 (socket says 172.21.15.77:49038) 2024-03-06T07:22:54.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f2136d9d640 1 -- 172.21.15.77:0/2973229567 learned_addr learned my addr 172.21.15.77:0/2973229567 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:54.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213659c640 1 -- 172.21.15.77:0/2973229567 >> v2:172.21.15.158:3300/0 conn(0x7f213815bf30 msgr2=0x7f21380b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213659c640 1 --2- 172.21.15.77:0/2973229567 >> v2:172.21.15.158:3300/0 conn(0x7f213815bf30 0x7f21380b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213659c640 1 -- 172.21.15.77:0/2973229567 >> v1:172.21.15.77:6789/0 conn(0x7f213815e850 legacy=0x7f213815ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f2135d9b640 1 -- 172.21.15.77:0/2973229567 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989355912 0 0) 0x7f213815f580 con 0x7f213815e850 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.401+0000 7f213659c640 1 -- 172.21.15.77:0/2973229567 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213815b450 con 0x7f213813dc10 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.402+0000 7f213659c640 1 --2- 172.21.15.77:0/2973229567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f21380b2950 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f213806c790 tx=0x7f212802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82537170df2dad3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.402+0000 7f2135d9b640 1 -- 172.21.15.77:0/2973229567 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21280315a0 con 0x7f213813dc10 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.402+0000 7f2135d9b640 1 -- 172.21.15.77:0/2973229567 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2128037070 con 0x7f213813dc10 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.402+0000 7f2135d9b640 1 -- 172.21.15.77:0/2973229567 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128031960 con 0x7f213813dc10 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.403+0000 7f213cdf7640 1 -- 172.21.15.77:0/2973229567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 msgr2=0x7f21380b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.403+0000 7f213cdf7640 1 --2- 172.21.15.77:0/2973229567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f21380b2950 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f213806c790 tx=0x7f212802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.403+0000 7f213cdf7640 1 -- 172.21.15.77:0/2973229567 shutdown_connections 2024-03-06T07:22:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.403+0000 7f213cdf7640 1 --2- 172.21.15.77:0/2973229567 >> v2:172.21.15.158:3300/0 conn(0x7f213815bf30 0x7f21380b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.403+0000 7f213cdf7640 1 --2- 172.21.15.77:0/2973229567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f21380b2950 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.403+0000 7f213cdf7640 1 -- 172.21.15.77:0/2973229567 >> 172.21.15.77:0/2973229567 conn(0x7f21380cf8a0 msgr2=0x7f21380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.403+0000 7f213cdf7640 1 -- 172.21.15.77:0/2973229567 shutdown_connections 2024-03-06T07:22:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.403+0000 7f213cdf7640 1 -- 172.21.15.77:0/2973229567 wait complete. 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.404+0000 7f213cdf7640 1 Processor -- start 2024-03-06T07:22:54.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.404+0000 7f213cdf7640 1 -- start start 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213cdf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f213813b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213cdf7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f213815bf30 0x7f21381545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213cdf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f213815e850 0x7f213813f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213cdf7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21380708d0 con 0x7f213813dc10 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213cdf7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213813dff0 con 0x7f213815bf30 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213cdf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2138070a50 con 0x7f213815e850 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f213813b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f213813b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42326/0 (socket says 172.21.15.77:42326) 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 -- 172.21.15.77:0/3577746779 learned_addr learned my addr 172.21.15.77:0/3577746779 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f2136d9d640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f213815e850 0x7f213813f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 -- 172.21.15.77:0/3577746779 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f213815e850 msgr2=0x7f213813f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f213815e850 0x7f213813f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 -- 172.21.15.77:0/3577746779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f213815bf30 msgr2=0x7f21381545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f213815bf30 0x7f21381545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 -- 172.21.15.77:0/3577746779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213813f930 con 0x7f213813dc10 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213659c640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f213813b8d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2128009890 tx=0x7f21280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f212effd640 1 -- 172.21.15.77:0/3577746779 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128032840 con 0x7f213813dc10 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f212effd640 1 -- 172.21.15.77:0/3577746779 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2128032a00 con 0x7f213813dc10 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213813bf20 con 0x7f213813dc10 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f212effd640 1 -- 172.21.15.77:0/3577746779 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128032df0 con 0x7f213813dc10 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.405+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213813c4e0 con 0x7f213813dc10 2024-03-06T07:22:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.406+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213804f330 con 0x7f213813dc10 2024-03-06T07:22:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.407+0000 7f212effd640 1 -- 172.21.15.77:0/3577746779 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f212804c020 con 0x7f213813dc10 2024-03-06T07:22:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.407+0000 7f212effd640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f211806b4d0 0x7f211806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.407+0000 7f212effd640 1 -- 172.21.15.77:0/3577746779 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f212802f020 con 0x7f213813dc10 2024-03-06T07:22:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.407+0000 7f212ffff640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f211806b4d0 0x7f211806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.408+0000 7f212ffff640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f211806b4d0 0x7f211806d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f21380722a0 tx=0x7f2120006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.411+0000 7f212effd640 1 -- 172.21.15.77:0/3577746779 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2128040070 con 0x7f213813dc10 2024-03-06T07:22:54.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.583+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f213806eb00 con 0x7f213813dc10 2024-03-06T07:22:54.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.584+0000 7f212effd640 1 -- 172.21.15.77:0/3577746779 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7f212807af50 con 0x7f213813dc10 2024-03-06T07:22:54.586 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:54.586 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:22:54.586 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:22:54.586 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:22:54.586 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:22:54.586 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:22:54.586 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:22:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f211806b4d0 msgr2=0x7f211806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f211806b4d0 0x7f211806d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f21380722a0 tx=0x7f2120006c90 comp rx=0 tx=0).stop 2024-03-06T07:22:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 msgr2=0x7f213813b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f213813b8d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2128009890 tx=0x7f21280046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 shutdown_connections 2024-03-06T07:22:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f213815e850 0x7f213813f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f213815bf30 0x7f21381545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f211806b4d0 0x7f211806d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 --2- 172.21.15.77:0/3577746779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f213813dc10 0x7f213813b8d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 >> 172.21.15.77:0/3577746779 conn(0x7f21380cf8a0 msgr2=0x7f2138070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 shutdown_connections 2024-03-06T07:22:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.586+0000 7f213cdf7640 1 -- 172.21.15.77:0/3577746779 wait complete. 2024-03-06T07:22:54.598 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:54.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:54.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.701+0000 7f3a36e2a640 1 Processor -- start 2024-03-06T07:22:54.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.701+0000 7f3a36e2a640 1 -- start start 2024-03-06T07:22:54.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a36e2a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a3015c5a0 0x7f3a3006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a36e2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a30071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a36e2a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3006ce50 con 0x7f3a300ce8c0 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a36e2a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a30072a20 con 0x7f3a3015c5a0 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a36e2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a300708b0 con 0x7f3a3015c980 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a34bc6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a3015c5a0 0x7f3a3006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a353c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a30071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a2ffff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3a300ce8c0 0x7f3a300cc330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:39476/0 (socket says 172.21.15.77:39476) 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a2ffff640 1 -- 172.21.15.77:0/969661899 learned_addr learned my addr 172.21.15.77:0/969661899 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.702+0000 7f3a353c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a30071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:49200/0 (socket says 172.21.15.77:49200) 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.703+0000 7f3a353c7640 1 -- 172.21.15.77:0/969661899 >> v2:172.21.15.158:3300/0 conn(0x7f3a3015c5a0 msgr2=0x7f3a3006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.703+0000 7f3a353c7640 1 --2- 172.21.15.77:0/969661899 >> v2:172.21.15.158:3300/0 conn(0x7f3a3015c5a0 0x7f3a3006c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.703+0000 7f3a353c7640 1 -- 172.21.15.77:0/969661899 >> v1:172.21.15.158:6789/0 conn(0x7f3a300ce8c0 legacy=0x7f3a300cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.703+0000 7f3a353c7640 1 -- 172.21.15.77:0/969661899 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a300ced50 con 0x7f3a3015c980 2024-03-06T07:22:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.703+0000 7f3a34bc6640 1 --2- 172.21.15.77:0/969661899 >> v2:172.21.15.158:3300/0 conn(0x7f3a3015c5a0 0x7f3a3006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:54.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.703+0000 7f3a353c7640 1 --2- 172.21.15.77:0/969661899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a30071300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009d90 tx=0x7f3a2802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=302d32188db3b94c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a2f7fe640 1 -- 172.21.15.77:0/969661899 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a28039070 con 0x7f3a3015c980 2024-03-06T07:22:54.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a2f7fe640 1 -- 172.21.15.77:0/969661899 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2802cbc0 con 0x7f3a3015c980 2024-03-06T07:22:54.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a2f7fe640 1 -- 172.21.15.77:0/969661899 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a28034440 con 0x7f3a3015c980 2024-03-06T07:22:54.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a36e2a640 1 -- 172.21.15.77:0/969661899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 msgr2=0x7f3a30071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/969661899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a30071300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009d90 tx=0x7f3a2802be10 comp rx=0 tx=0).stop 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a36e2a640 1 -- 172.21.15.77:0/969661899 shutdown_connections 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/969661899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a30071300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/969661899 >> v2:172.21.15.158:3300/0 conn(0x7f3a3015c5a0 0x7f3a3006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a36e2a640 1 -- 172.21.15.77:0/969661899 >> 172.21.15.77:0/969661899 conn(0x7f3a300b1650 msgr2=0x7f3a300b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a36e2a640 1 -- 172.21.15.77:0/969661899 shutdown_connections 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.704+0000 7f3a36e2a640 1 -- 172.21.15.77:0/969661899 wait complete. 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.705+0000 7f3a36e2a640 1 Processor -- start 2024-03-06T07:22:54.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.705+0000 7f3a36e2a640 1 -- start start 2024-03-06T07:22:54.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a36e2a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3015c5a0 0x7f3a301423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a36e2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a300b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a36e2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a300b0a20 0x7f3a301f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a36e2a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a30070b00 con 0x7f3a300b0a20 2024-03-06T07:22:54.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a36e2a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a30070e70 con 0x7f3a3015c5a0 2024-03-06T07:22:54.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a36e2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3015ce30 con 0x7f3a3015c980 2024-03-06T07:22:54.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a353c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a300b0a20 0x7f3a301f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a353c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a300b0a20 0x7f3a301f7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42330/0 (socket says 172.21.15.77:42330) 2024-03-06T07:22:54.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a353c7640 1 -- 172.21.15.77:0/601430586 learned_addr learned my addr 172.21.15.77:0/601430586 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:54.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a353c7640 1 -- 172.21.15.77:0/601430586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 msgr2=0x7f3a300b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a353c7640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a300b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a353c7640 1 -- 172.21.15.77:0/601430586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3015c5a0 msgr2=0x7f3a301423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a353c7640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3015c5a0 0x7f3a301423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.706+0000 7f3a353c7640 1 -- 172.21.15.77:0/601430586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a301f4780 con 0x7f3a300b0a20 2024-03-06T07:22:54.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.707+0000 7f3a353c7640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a300b0a20 0x7f3a301f7040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009eb0 tx=0x7f3a28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.707+0000 7f3a2d7fa640 1 -- 172.21.15.77:0/601430586 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a28039050 con 0x7f3a300b0a20 2024-03-06T07:22:54.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.707+0000 7f3a2d7fa640 1 -- 172.21.15.77:0/601430586 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a28045070 con 0x7f3a300b0a20 2024-03-06T07:22:54.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.707+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a301f39c0 con 0x7f3a300b0a20 2024-03-06T07:22:54.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.707+0000 7f3a2d7fa640 1 -- 172.21.15.77:0/601430586 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2803e850 con 0x7f3a300b0a20 2024-03-06T07:22:54.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.707+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a301f3f80 con 0x7f3a300b0a20 2024-03-06T07:22:54.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.708+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a300687d0 con 0x7f3a300b0a20 2024-03-06T07:22:54.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.708+0000 7f3a2d7fa640 1 -- 172.21.15.77:0/601430586 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a2803e9f0 con 0x7f3a300b0a20 2024-03-06T07:22:54.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.709+0000 7f3a2d7fa640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a1406b500 0x7f3a1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.709+0000 7f3a2d7fa640 1 -- 172.21.15.77:0/601430586 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3a280ac510 con 0x7f3a300b0a20 2024-03-06T07:22:54.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.709+0000 7f3a34bc6640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a1406b500 0x7f3a1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.709+0000 7f3a34bc6640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a1406b500 0x7f3a1406d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3a24006e80 tx=0x7f3a24006dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.713+0000 7f3a2d7fa640 1 -- 172.21.15.77:0/601430586 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a2804e020 con 0x7f3a300b0a20 2024-03-06T07:22:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:54.884+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3a3004f330 con 0x7f3a300b0a20 2024-03-06T07:22:55.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.321+0000 7f3a2d7fa640 1 -- 172.21.15.77:0/601430586 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3a280ac070 con 0x7f3a300b0a20 2024-03-06T07:22:55.324 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a1406b500 msgr2=0x7f3a1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a1406b500 0x7f3a1406d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3a24006e80 tx=0x7f3a24006dd0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a300b0a20 msgr2=0x7f3a301f7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a300b0a20 0x7f3a301f7040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009eb0 tx=0x7f3a28004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 shutdown_connections 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a1406b500 0x7f3a1406d9c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a300b0a20 0x7f3a301f7040 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a3015c980 0x7f3a300b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 --2- 172.21.15.77:0/601430586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a3015c5a0 0x7f3a301423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 >> 172.21.15.77:0/601430586 conn(0x7f3a300b1650 msgr2=0x7f3a3015e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 shutdown_connections 2024-03-06T07:22:55.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.324+0000 7f3a36e2a640 1 -- 172.21.15.77:0/601430586 wait complete. 2024-03-06T07:22:55.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:55.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.448+0000 7fac73193640 1 Processor -- start 2024-03-06T07:22:55.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.448+0000 7fac73193640 1 -- start start 2024-03-06T07:22:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac73193640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fac6c0cfa90 0x7fac6c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac73193640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c158ac0 con 0x7fac6c1586e0 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac73193640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c15b2a0 con 0x7fac6c0b33a0 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac73193640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c15b540 con 0x7fac6c0cfa90 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac70f2f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fac6c1586e0 0x7fac6c0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49046/0 (socket says 172.21.15.77:49046) 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac70f2f640 1 -- 172.21.15.77:0/1663387112 learned_addr learned my addr 172.21.15.77:0/1663387112 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac6bfff640 1 --2- 172.21.15.77:0/1663387112 >> v2:172.21.15.158:3300/0 conn(0x7fac6c0cfa90 0x7fac6c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297303900 0 0) 0x7fac6c158ac0 con 0x7fac6c1586e0 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac54003520 con 0x7fac6c1586e0 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.449+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975730903 0 0) 0x7fac54003520 con 0x7fac6c1586e0 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 >> v2:172.21.15.158:3300/0 conn(0x7fac6c0cfa90 msgr2=0x7fac6c0cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 --2- 172.21.15.77:0/1663387112 >> v2:172.21.15.158:3300/0 conn(0x7fac6c0cfa90 0x7fac6c0cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 >> v1:172.21.15.158:6789/0 conn(0x7fac6c0b33a0 legacy=0x7fac6c0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c15b820 con 0x7fac6c1586e0 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fac5c001670 con 0x7fac6c1586e0 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 >> v1:172.21.15.77:6789/0 conn(0x7fac6c1586e0 legacy=0x7fac6c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540039b0 0x7fac54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac54004800 0x7fac54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac540051c0 0x7fac5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c15b540 con 0x7fac540039b0 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5400c060 con 0x7fac540051c0 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5400c260 con 0x7fac54004800 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac70f2f640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540039b0 0x7fac54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac71730640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac540051c0 0x7fac5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6bfff640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac54004800 0x7fac54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac70f2f640 1 -- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac54004800 msgr2=0x7fac54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac70f2f640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac54004800 0x7fac54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac70f2f640 1 -- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac540051c0 msgr2=0x7fac5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac70f2f640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac540051c0 0x7fac5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac71730640 1 -- 172.21.15.77:0/1663387112 reap_dead start 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac70f2f640 1 -- 172.21.15.77:0/1663387112 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac54003520 con 0x7fac540039b0 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.450+0000 7fac6bfff640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac54004800 0x7fac54004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac70f2f640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540039b0 0x7fac54003dd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00c210 tx=0x7fac54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6b544f432c834a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c007d10 con 0x7fac540039b0 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c030070 con 0x7fac540039b0 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac6b7fe640 1 -- 172.21.15.77:0/1663387112 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c005d30 con 0x7fac540039b0 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac73193640 1 -- 172.21.15.77:0/1663387112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540039b0 msgr2=0x7fac54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac73193640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540039b0 0x7fac54003dd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00c210 tx=0x7fac54005830 comp rx=0 tx=0).stop 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac73193640 1 -- 172.21.15.77:0/1663387112 shutdown_connections 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac73193640 1 --2- 172.21.15.77:0/1663387112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540039b0 0x7fac54003dd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac73193640 1 -- 172.21.15.77:0/1663387112 >> 172.21.15.77:0/1663387112 conn(0x7fac6c06b190 msgr2=0x7fac6c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac73193640 1 -- 172.21.15.77:0/1663387112 shutdown_connections 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.451+0000 7fac73193640 1 -- 172.21.15.77:0/1663387112 wait complete. 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.453+0000 7fac73193640 1 Processor -- start 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.453+0000 7fac73193640 1 -- start start 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.453+0000 7fac73193640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac540039b0 0x7fac6c153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac73193640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac6c0c47e0 0x7fac6c1f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac73193640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac6c14e490 0x7fac6c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac73193640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c071350 con 0x7fac6c0c47e0 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac73193640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c15bde0 con 0x7fac6c14e490 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac73193640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0714d0 con 0x7fac540039b0 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac6c0c47e0 0x7fac6c1f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac6c0c47e0 0x7fac6c1f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42336/0 (socket says 172.21.15.77:42336) 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 -- 172.21.15.77:0/352603690 learned_addr learned my addr 172.21.15.77:0/352603690 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac70f2f640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac540039b0 0x7fac6c153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac71730640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac6c14e490 0x7fac6c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 -- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac540039b0 msgr2=0x7fac6c153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac540039b0 0x7fac6c153610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 -- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac6c14e490 msgr2=0x7fac6c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac6c14e490 0x7fac6c0ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 -- 172.21.15.77:0/352603690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0c4c40 con 0x7fac6c0c47e0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac70f2f640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac540039b0 0x7fac6c153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac6bfff640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac6c0c47e0 0x7fac6c1f7790 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fac600119f0 tx=0x7fac60011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac697fa640 1 -- 172.21.15.77:0/352603690 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6001e070 con 0x7fac6c0c47e0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac697fa640 1 -- 172.21.15.77:0/352603690 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6000fc80 con 0x7fac6c0c47e0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac697fa640 1 -- 172.21.15.77:0/352603690 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60019410 con 0x7fac6c0c47e0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c0c4ee0 con 0x7fac6c0c47e0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.454+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c1f6f90 con 0x7fac6c0c47e0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.455+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c0b4820 con 0x7fac6c0c47e0 2024-03-06T07:22:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.456+0000 7fac697fa640 1 -- 172.21.15.77:0/352603690 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac60021070 con 0x7fac6c0c47e0 2024-03-06T07:22:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.456+0000 7fac697fa640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac5806b4d0 0x7fac5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.456+0000 7fac697fa640 1 -- 172.21.15.77:0/352603690 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fac60089d70 con 0x7fac6c0c47e0 2024-03-06T07:22:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.456+0000 7fac70f2f640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac5806b4d0 0x7fac5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.457+0000 7fac70f2f640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac5806b4d0 0x7fac5806d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fac5c034000 tx=0x7fac5c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.461+0000 7fac697fa640 1 -- 172.21.15.77:0/352603690 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac6c0b4820 con 0x7fac6c0c47e0 2024-03-06T07:22:55.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:55.633+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fac6c068ff0 con 0x7fac6c0c47e0 2024-03-06T07:22:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.326+0000 7fac697fa640 1 -- 172.21.15.77:0/352603690 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7fac6000e070 con 0x7fac6c0c47e0 2024-03-06T07:22:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:22:56.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac5806b4d0 msgr2=0x7fac5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac5806b4d0 0x7fac5806d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fac5c034000 tx=0x7fac5c007000 comp rx=0 tx=0).stop 2024-03-06T07:22:56.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac6c0c47e0 msgr2=0x7fac6c1f7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac6c0c47e0 0x7fac6c1f7790 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fac600119f0 tx=0x7fac60011eb0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 shutdown_connections 2024-03-06T07:22:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac5806b4d0 0x7fac5806d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac6c0c47e0 0x7fac6c1f7790 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac6c14e490 0x7fac6c0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 --2- 172.21.15.77:0/352603690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac540039b0 0x7fac6c153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.329+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 >> 172.21.15.77:0/352603690 conn(0x7fac6c06b190 msgr2=0x7fac6c06e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.330+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 shutdown_connections 2024-03-06T07:22:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.330+0000 7fac73193640 1 -- 172.21.15.77:0/352603690 wait complete. 2024-03-06T07:22:56.354 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:56.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.456+0000 7f9c70cc2640 1 Processor -- start 2024-03-06T07:22:56.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.456+0000 7f9c70cc2640 1 -- start start 2024-03-06T07:22:56.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.457+0000 7f9c70cc2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c0d2380 0x7f9c6c0d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.457+0000 7f9c70cc2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0b3fc0 con 0x7f9c6c15e0b0 2024-03-06T07:22:56.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.457+0000 7f9c70cc2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0b4290 con 0x7f9c6c0cee00 2024-03-06T07:22:56.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.457+0000 7f9c70cc2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c0ce990 con 0x7f9c6c0d2380 2024-03-06T07:22:56.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.457+0000 7f9c6a59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9c6c15e0b0 0x7f9c6c0b38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49060/0 (socket says 172.21.15.77:49060) 2024-03-06T07:22:56.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.457+0000 7f9c6a59c640 1 -- 172.21.15.77:0/2517266235 learned_addr learned my addr 172.21.15.77:0/2517266235 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:56.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c69d9b640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c0d2380 0x7f9c6c0d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382498401 0 0) 0x7f9c6c0b3fc0 con 0x7f9c6c15e0b0 2024-03-06T07:22:56.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c54003520 con 0x7f9c6c15e0b0 2024-03-06T07:22:56.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2264191498 0 0) 0x7f9c54003520 con 0x7f9c6c15e0b0 2024-03-06T07:22:56.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c0d2380 msgr2=0x7f9c6c0d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c0d2380 0x7f9c6c0d27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 >> v1:172.21.15.158:6789/0 conn(0x7f9c6c0cee00 legacy=0x7f9c6c0cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c0b4570 con 0x7f9c6c15e0b0 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f9c5c0019f0 con 0x7f9c6c15e0b0 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 >> v1:172.21.15.77:6789/0 conn(0x7f9c6c15e0b0 legacy=0x7f9c6c0b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c540039e0 0x7f9c54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c54004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c0d2380 0x7f9c540050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5400b9c0 con 0x7f9c54004390 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5400bb40 con 0x7f9c540039e0 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5400bd40 con 0x7f9c6c0d2380 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c69d9b640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c54004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6a59c640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c540039e0 0x7f9c54003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c6ad9d640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c0d2380 0x7f9c540050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c69d9b640 1 -- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c0d2380 msgr2=0x7f9c540050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c69d9b640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c0d2380 0x7f9c540050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c69d9b640 1 -- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c540039e0 msgr2=0x7f9c54003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c69d9b640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c540039e0 0x7f9c54003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.458+0000 7f9c69d9b640 1 -- 172.21.15.77:0/2517266235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54003520 con 0x7f9c54004390 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.459+0000 7f9c69d9b640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c54004860 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009a90 tx=0x7f9c54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489ea663ad025459 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.459+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c60039070 con 0x7f9c54004390 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.459+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6002c680 con 0x7f9c54004390 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.459+0000 7f9c6959a640 1 -- 172.21.15.77:0/2517266235 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c60009690 con 0x7f9c54004390 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.460+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2517266235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 msgr2=0x7f9c54004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.460+0000 7f9c70cc2640 1 --2- 172.21.15.77:0/2517266235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c54004860 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009a90 tx=0x7f9c54005830 comp rx=0 tx=0).stop 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.460+0000 7f9c6ad9d640 1 -- 172.21.15.77:0/2517266235 reap_dead start 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.460+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2517266235 shutdown_connections 2024-03-06T07:22:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.460+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2517266235 >> 172.21.15.77:0/2517266235 conn(0x7f9c6c08d6f0 msgr2=0x7f9c6c09d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.460+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2517266235 shutdown_connections 2024-03-06T07:22:56.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.460+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2517266235 wait complete. 2024-03-06T07:22:56.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.461+0000 7f9c70cc2640 1 Processor -- start 2024-03-06T07:22:56.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.461+0000 7f9c70cc2640 1 -- start start 2024-03-06T07:22:56.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c70cc2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c6c156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c70cc2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c6c156fe0 0x7f9c6c0b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c70cc2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c157510 0x7f9c6c0b6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c70cc2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c15e490 con 0x7f9c54004390 2024-03-06T07:22:56.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c70cc2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c0b4b30 con 0x7f9c6c156fe0 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c70cc2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c0d1cd0 con 0x7f9c6c157510 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c6c156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c6c156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42346/0 (socket says 172.21.15.77:42346) 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 -- 172.21.15.77:0/2751006584 learned_addr learned my addr 172.21.15.77:0/2751006584 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6ad9d640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c157510 0x7f9c6c0b6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c69d9b640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c6c156fe0 0x7f9c6c0b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 -- 172.21.15.77:0/2751006584 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c157510 msgr2=0x7f9c6c0b6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c157510 0x7f9c6c0b6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 -- 172.21.15.77:0/2751006584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c6c156fe0 msgr2=0x7f9c6c0b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c6c156fe0 0x7f9c6c0b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 -- 172.21.15.77:0/2751006584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c0ccc40 con 0x7f9c54004390 2024-03-06T07:22:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.462+0000 7f9c6a59c640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c6c156840 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c007ae0 tx=0x7f9c5c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.463+0000 7f9c5affd640 1 -- 172.21.15.77:0/2751006584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c019070 con 0x7f9c54004390 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.463+0000 7f9c5affd640 1 -- 172.21.15.77:0/2751006584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c014480 con 0x7f9c54004390 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.463+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6c0ccf20 con 0x7f9c54004390 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.463+0000 7f9c5affd640 1 -- 172.21.15.77:0/2751006584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c014850 con 0x7f9c54004390 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.463+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6c0cd4e0 con 0x7f9c54004390 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.464+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6c068130 con 0x7f9c54004390 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.464+0000 7f9c5affd640 1 -- 172.21.15.77:0/2751006584 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c5c01c070 con 0x7f9c54004390 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.464+0000 7f9c5affd640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4c06b4d0 0x7f9c4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.464+0000 7f9c5affd640 1 -- 172.21.15.77:0/2751006584 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9c5c08a110 con 0x7f9c54004390 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.464+0000 7f9c69d9b640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4c06b4d0 0x7f9c4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.465+0000 7f9c69d9b640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4c06b4d0 0x7f9c4c06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009a90 tx=0x7f9c60030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.469+0000 7f9c5affd640 1 -- 172.21.15.77:0/2751006584 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c6c068130 con 0x7f9c54004390 2024-03-06T07:22:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:56.675+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9c6c04f330 con 0x7f9c54004390 2024-03-06T07:22:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.327+0000 7f9c5affd640 1 -- 172.21.15.77:0/2751006584 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9c5c0589a0 con 0x7f9c54004390 2024-03-06T07:22:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.330+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4c06b4d0 msgr2=0x7f9c4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.330+0000 7f9c70cc2640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4c06b4d0 0x7f9c4c06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009a90 tx=0x7f9c60030000 comp rx=0 tx=0).stop 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.330+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 msgr2=0x7f9c6c156840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.330+0000 7f9c70cc2640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c6c156840 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c007ae0 tx=0x7f9c5c00e530 comp rx=0 tx=0).stop 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.331+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 shutdown_connections 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.331+0000 7f9c70cc2640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c6c157510 0x7f9c6c0b6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.331+0000 7f9c70cc2640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c6c156fe0 0x7f9c6c0b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.331+0000 7f9c70cc2640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c4c06b4d0 0x7f9c4c06d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.331+0000 7f9c70cc2640 1 --2- 172.21.15.77:0/2751006584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c54004390 0x7f9c6c156840 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.331+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 >> 172.21.15.77:0/2751006584 conn(0x7f9c6c08d6f0 msgr2=0x7f9c6c073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.331+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 shutdown_connections 2024-03-06T07:22:57.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.331+0000 7f9c70cc2640 1 -- 172.21.15.77:0/2751006584 wait complete. 2024-03-06T07:22:57.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:57.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.459+0000 7f61935c7640 1 Processor -- start 2024-03-06T07:22:57.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.459+0000 7f61935c7640 1 -- start start 2024-03-06T07:22:57.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.460+0000 7f61935c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.460+0000 7f61935c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f618c15bf30 0x7f618c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.460+0000 7f61935c7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c15f580 con 0x7f618c15e850 2024-03-06T07:22:57.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.460+0000 7f61935c7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f618c0cf3f0 con 0x7f618c15bf30 2024-03-06T07:22:57.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.460+0000 7f61935c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618c15b210 con 0x7f618c13dc10 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.460+0000 7f6190b62640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f618c15bf30 0x7f618c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.460+0000 7f6190b62640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f618c15bf30 0x7f618c0b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42348/0 (socket says 172.21.15.77:42348) 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.460+0000 7f6190b62640 1 -- 172.21.15.77:0/1415386305 learned_addr learned my addr 172.21.15.77:0/1415386305 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.461+0000 7f6190b62640 1 -- 172.21.15.77:0/1415386305 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 msgr2=0x7f618c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.461+0000 7f6191363640 1 --2- 172.21.15.77:0/1415386305 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.461+0000 7f6190b62640 1 --2- 172.21.15.77:0/1415386305 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.461+0000 7f6190b62640 1 -- 172.21.15.77:0/1415386305 >> v1:172.21.15.158:6789/0 conn(0x7f618c15e850 legacy=0x7f618c15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.461+0000 7f6190b62640 1 -- 172.21.15.77:0/1415386305 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c15b450 con 0x7f618c15bf30 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.461+0000 7f6190b62640 1 --2- 172.21.15.77:0/1415386305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15bf30 0x7f618c0b2e90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f617c009870 tx=0x7f617c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d75a5f477efdf1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.461+0000 7f6183fff640 1 -- 172.21.15.77:0/1415386305 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617c0317c0 con 0x7f618c15bf30 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f6183fff640 1 -- 172.21.15.77:0/1415386305 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617c037070 con 0x7f618c15bf30 2024-03-06T07:22:57.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f6183fff640 1 -- 172.21.15.77:0/1415386305 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617c031b80 con 0x7f618c15bf30 2024-03-06T07:22:57.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f61935c7640 1 -- 172.21.15.77:0/1415386305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15bf30 msgr2=0x7f618c0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f61935c7640 1 --2- 172.21.15.77:0/1415386305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15bf30 0x7f618c0b2e90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f617c009870 tx=0x7f617c02b860 comp rx=0 tx=0).stop 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f61935c7640 1 -- 172.21.15.77:0/1415386305 shutdown_connections 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f61935c7640 1 --2- 172.21.15.77:0/1415386305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15bf30 0x7f618c0b2e90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f61935c7640 1 --2- 172.21.15.77:0/1415386305 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f61935c7640 1 -- 172.21.15.77:0/1415386305 >> 172.21.15.77:0/1415386305 conn(0x7f618c0cf8a0 msgr2=0x7f618c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f61935c7640 1 -- 172.21.15.77:0/1415386305 shutdown_connections 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.462+0000 7f61935c7640 1 -- 172.21.15.77:0/1415386305 wait complete. 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.463+0000 7f61935c7640 1 Processor -- start 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.463+0000 7f61935c7640 1 -- start start 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f61935c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f61935c7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f618c15bf30 0x7f618c140910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f61935c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15e850 0x7f618c13bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f61935c7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c0708d0 con 0x7f618c15e850 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f61935c7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c13dff0 con 0x7f618c15bf30 2024-03-06T07:22:57.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f61935c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618c070a50 con 0x7f618c13dc10 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191b64640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15e850 0x7f618c13bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191b64640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15e850 0x7f618c13bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42356/0 (socket says 172.21.15.77:42356) 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191b64640 1 -- 172.21.15.77:0/1092536443 learned_addr learned my addr 172.21.15.77:0/1092536443 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191363640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6190b62640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f618c15bf30 0x7f618c140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191b64640 1 -- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 msgr2=0x7f618c143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191b64640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191b64640 1 -- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f618c15bf30 msgr2=0x7f618c140910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191b64640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f618c15bf30 0x7f618c140910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191b64640 1 -- 172.21.15.77:0/1092536443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c143ad0 con 0x7f618c15e850 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6191363640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.464+0000 7f6190b62640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f618c15bf30 0x7f618c140910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.465+0000 7f6191b64640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15e850 0x7f618c13bf50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f618400b410 tx=0x7f618400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.465+0000 7f6181ffb640 1 -- 172.21.15.77:0/1092536443 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184015030 con 0x7f618c15e850 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.465+0000 7f6181ffb640 1 -- 172.21.15.77:0/1092536443 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6184004020 con 0x7f618c15e850 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.465+0000 7f6181ffb640 1 -- 172.21.15.77:0/1092536443 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184004410 con 0x7f618c15e850 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.465+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618c13f420 con 0x7f618c15e850 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.465+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618c13f9e0 con 0x7f618c15e850 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.466+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c06f270 con 0x7f618c15e850 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.466+0000 7f6181ffb640 1 -- 172.21.15.77:0/1092536443 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6184018070 con 0x7f618c15e850 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.467+0000 7f6181ffb640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f617006b4d0 0x7f617006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.467+0000 7f6181ffb640 1 -- 172.21.15.77:0/1092536443 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6184089ca0 con 0x7f618c15e850 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.467+0000 7f6191363640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f617006b4d0 0x7f617006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.467+0000 7f6191363640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f617006b4d0 0x7f617006d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f618c06e110 tx=0x7f6188007650 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.471+0000 7f6181ffb640 1 -- 172.21.15.77:0/1092536443 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f618c06f270 con 0x7f618c15e850 2024-03-06T07:22:57.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.644+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f618c0682b0 con 0x7f618c15e850 2024-03-06T07:22:57.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.645+0000 7f6181ffb640 1 -- 172.21.15.77:0/1092536443 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16670 (secure 0 0 0) 0x7f618c0682b0 con 0x7f618c15e850 2024-03-06T07:22:57.648 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:57.649 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:56.676439+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:54.903388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.652222+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f617006b4d0 msgr2=0x7f617006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f617006b4d0 0x7f617006d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f618c06e110 tx=0x7f6188007650 comp rx=0 tx=0).stop 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15e850 msgr2=0x7f618c13bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15e850 0x7f618c13bf50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f618400b410 tx=0x7f618400b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 shutdown_connections 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f617006b4d0 0x7f617006d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618c15e850 0x7f618c13bf50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f618c15bf30 0x7f618c140910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 --2- 172.21.15.77:0/1092536443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f618c13dc10 0x7f618c143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.647+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 >> 172.21.15.77:0/1092536443 conn(0x7f618c0cf8a0 msgr2=0x7f618c070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.648+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 shutdown_connections 2024-03-06T07:22:57.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.648+0000 7f61935c7640 1 -- 172.21.15.77:0/1092536443 wait complete. 2024-03-06T07:22:57.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.759+0000 7fd575793640 1 Processor -- start 2024-03-06T07:22:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.759+0000 7fd575793640 1 -- start start 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.760+0000 7fd575793640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd5700d22b0 0x7fd5700d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.760+0000 7fd575793640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700b3430 con 0x7fd5700b41f0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.760+0000 7fd575793640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700b3710 con 0x7fd57015dfe0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.760+0000 7fd575793640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5700d2c50 con 0x7fd5700d22b0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.760+0000 7fd56e7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd5700d22b0 0x7fd5700d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.760+0000 7fd56f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd5700b41f0 0x7fd5700b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49072/0 (socket says 172.21.15.77:49072) 2024-03-06T07:22:57.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.760+0000 7fd56f7fe640 1 -- 172.21.15.77:0/1382448763 learned_addr learned my addr 172.21.15.77:0/1382448763 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.761+0000 7fd56e7fc640 1 -- 172.21.15.77:0/1382448763 >> v1:172.21.15.158:6789/0 conn(0x7fd57015dfe0 legacy=0x7fd5700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.761+0000 7fd56e7fc640 1 -- 172.21.15.77:0/1382448763 >> v1:172.21.15.77:6789/0 conn(0x7fd5700b41f0 legacy=0x7fd5700b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.761+0000 7fd56dffb640 1 -- 172.21.15.77:0/1382448763 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614693814 0 0) 0x7fd5700b3430 con 0x7fd5700b41f0 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.761+0000 7fd56e7fc640 1 -- 172.21.15.77:0/1382448763 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700b39f0 con 0x7fd5700d22b0 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.761+0000 7fd56e7fc640 1 --2- 172.21.15.77:0/1382448763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700d2710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd5640097d0 tx=0x7fd56402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e359e0c2b6d483 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.761+0000 7fd56dffb640 1 -- 172.21.15.77:0/1382448763 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564035020 con 0x7fd5700d22b0 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.761+0000 7fd56dffb640 1 -- 172.21.15.77:0/1382448763 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564031830 con 0x7fd5700d22b0 2024-03-06T07:22:57.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.761+0000 7fd56dffb640 1 -- 172.21.15.77:0/1382448763 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564031c00 con 0x7fd5700d22b0 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.762+0000 7fd575793640 1 -- 172.21.15.77:0/1382448763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 msgr2=0x7fd5700d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.762+0000 7fd575793640 1 --2- 172.21.15.77:0/1382448763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700d2710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd5640097d0 tx=0x7fd56402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.762+0000 7fd575793640 1 -- 172.21.15.77:0/1382448763 shutdown_connections 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.762+0000 7fd575793640 1 --2- 172.21.15.77:0/1382448763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700d2710 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.762+0000 7fd575793640 1 -- 172.21.15.77:0/1382448763 >> 172.21.15.77:0/1382448763 conn(0x7fd5700d3cd0 msgr2=0x7fd5700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.762+0000 7fd575793640 1 -- 172.21.15.77:0/1382448763 shutdown_connections 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.762+0000 7fd575793640 1 -- 172.21.15.77:0/1382448763 wait complete. 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.763+0000 7fd575793640 1 Processor -- start 2024-03-06T07:22:57.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.763+0000 7fd575793640 1 -- start start 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd575793640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5700b41f0 0x7fd5700b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd575793640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700b7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd575793640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd57015dfe0 0x7fd5700b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd575793640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700b3d60 con 0x7fd5700d22b0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd575793640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700b31e0 con 0x7fd5700b41f0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd575793640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700d1c70 con 0x7fd57015dfe0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd56e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd56e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700b7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42360/0 (socket says 172.21.15.77:42360) 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd56e7fc640 1 -- 172.21.15.77:0/3068081779 learned_addr learned my addr 172.21.15.77:0/3068081779 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd56e7fc640 1 -- 172.21.15.77:0/3068081779 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd57015dfe0 msgr2=0x7fd5700b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd56e7fc640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd57015dfe0 0x7fd5700b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd56e7fc640 1 -- 172.21.15.77:0/3068081779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5700b41f0 msgr2=0x7fd5700b7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd56e7fc640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5700b41f0 0x7fd5700b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.764+0000 7fd56e7fc640 1 -- 172.21.15.77:0/3068081779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd570160100 con 0x7fd5700d22b0 2024-03-06T07:22:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.765+0000 7fd56e7fc640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700b7570 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd56402bee0 tx=0x7fd5640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.765+0000 7fd5577fe640 1 -- 172.21.15.77:0/3068081779 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56403ba80 con 0x7fd5700d22b0 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.765+0000 7fd5577fe640 1 -- 172.21.15.77:0/3068081779 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56403bc20 con 0x7fd5700d22b0 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.765+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5701603e0 con 0x7fd5700d22b0 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.765+0000 7fd5577fe640 1 -- 172.21.15.77:0/3068081779 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5640453f0 con 0x7fd5700d22b0 2024-03-06T07:22:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.765+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd570151600 con 0x7fd5700d22b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.766+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5701607d0 con 0x7fd5700d22b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.766+0000 7fd5577fe640 1 -- 172.21.15.77:0/3068081779 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd564038020 con 0x7fd5700d22b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.767+0000 7fd5577fe640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.767+0000 7fd5577fe640 1 -- 172.21.15.77:0/3068081779 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5640abf80 con 0x7fd5700d22b0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.767+0000 7fd56effd640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.767+0000 7fd56effd640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd55c06b4d0 0x7fd55c06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5700b9220 tx=0x7fd560009230 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.771+0000 7fd5577fe640 1 -- 172.21.15.77:0/3068081779 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd564040070 con 0x7fd5700d22b0 2024-03-06T07:22:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.943+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd57004f330 con 0x7fd5700d22b0 2024-03-06T07:22:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.945+0000 7fd5577fe640 1 -- 172.21.15.77:0/3068081779 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16670 (secure 0 0 0) 0x7fd56407ae00 con 0x7fd5700d22b0 2024-03-06T07:22:57.948 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:57.948 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:56.676439+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:54.903388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.652222+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:57.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd55c06b4d0 msgr2=0x7fd55c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd55c06b4d0 0x7fd55c06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5700b9220 tx=0x7fd560009230 comp rx=0 tx=0).stop 2024-03-06T07:22:57.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 msgr2=0x7fd5700b7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700b7570 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd56402bee0 tx=0x7fd5640046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 shutdown_connections 2024-03-06T07:22:57.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700d22b0 0x7fd5700b7570 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd57015dfe0 0x7fd5700b7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 --2- 172.21.15.77:0/3068081779 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5700b41f0 0x7fd5700b7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 >> 172.21.15.77:0/3068081779 conn(0x7fd5700d3cd0 msgr2=0x7fd5700d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 shutdown_connections 2024-03-06T07:22:57.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:57.947+0000 7fd575793640 1 -- 172.21.15.77:0/3068081779 wait complete. 2024-03-06T07:22:57.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:58.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.063+0000 7fe5e94c3640 1 Processor -- start 2024-03-06T07:22:58.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.063+0000 7fe5e94c3640 1 -- start start 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.063+0000 7fe5e94c3640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5e41614d0 0x7fe5e40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.063+0000 7fe5e94c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e40d1910 0x7fe5e40d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.063+0000 7fe5e94c3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5e40d22b0 0x7fe5e40d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.063+0000 7fe5e94c3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e40d2740 con 0x7fe5e40d22b0 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e94c3640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e41618b0 con 0x7fe5e41614d0 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e94c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40cf400 con 0x7fe5e40d1910 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5e40d22b0 0x7fe5e40d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe5e40d22b0 0x7fe5e40d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42364/0 (socket says 172.21.15.77:42364) 2024-03-06T07:22:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e37fe640 1 -- 172.21.15.77:0/1011939730 learned_addr learned my addr 172.21.15.77:0/1011939730 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e37fe640 1 -- 172.21.15.77:0/1011939730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e40d1910 msgr2=0x7fe5e40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e37fe640 1 --2- 172.21.15.77:0/1011939730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e40d1910 0x7fe5e40d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e37fe640 1 -- 172.21.15.77:0/1011939730 >> v2:172.21.15.158:3300/0 conn(0x7fe5e41614d0 msgr2=0x7fe5e40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e37fe640 1 --2- 172.21.15.77:0/1011939730 >> v2:172.21.15.158:3300/0 conn(0x7fe5e41614d0 0x7fe5e40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.064+0000 7fe5e37fe640 1 -- 172.21.15.77:0/1011939730 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e40cf6a0 con 0x7fe5e40d22b0 2024-03-06T07:22:58.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e37fe640 1 --2- 172.21.15.77:0/1011939730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40d06a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe5e408c470 tx=0x7fe5dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9702a17dff92dee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e27fc640 1 -- 172.21.15.77:0/1011939730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc02ca70 con 0x7fe5e40d22b0 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e27fc640 1 -- 172.21.15.77:0/1011939730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5dc02cc10 con 0x7fe5e40d22b0 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e27fc640 1 -- 172.21.15.77:0/1011939730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc0343f0 con 0x7fe5e40d22b0 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 -- 172.21.15.77:0/1011939730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 msgr2=0x7fe5e40d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/1011939730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40d06a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe5e408c470 tx=0x7fe5dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 -- 172.21.15.77:0/1011939730 shutdown_connections 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/1011939730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40d06a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/1011939730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e40d1910 0x7fe5e40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/1011939730 >> v2:172.21.15.158:3300/0 conn(0x7fe5e41614d0 0x7fe5e40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 -- 172.21.15.77:0/1011939730 >> 172.21.15.77:0/1011939730 conn(0x7fe5e40d3d40 msgr2=0x7fe5e40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 -- 172.21.15.77:0/1011939730 shutdown_connections 2024-03-06T07:22:58.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.065+0000 7fe5e94c3640 1 -- 172.21.15.77:0/1011939730 wait complete. 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5e94c3640 1 Processor -- start 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5e94c3640 1 -- start start 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5e94c3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e40d1910 0x7fe5e40ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5e94c3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5e94c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e41614d0 0x7fe5e40c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5e94c3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40b0910 con 0x7fe5e40d22b0 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5e94c3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40cfc60 con 0x7fe5e40d1910 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5e94c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40d0be0 con 0x7fe5e41614d0 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40b94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42380/0 (socket says 172.21.15.77:42380) 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.067+0000 7fe5dbfff640 1 -- 172.21.15.77:0/991684252 learned_addr learned my addr 172.21.15.77:0/991684252 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5e37fe640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e41614d0 0x7fe5e40c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5e2ffd640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e40d1910 0x7fe5e40ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5dbfff640 1 -- 172.21.15.77:0/991684252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e41614d0 msgr2=0x7fe5e40c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5dbfff640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e41614d0 0x7fe5e40c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5dbfff640 1 -- 172.21.15.77:0/991684252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e40d1910 msgr2=0x7fe5e40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5dbfff640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e40d1910 0x7fe5e40ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5dbfff640 1 -- 172.21.15.77:0/991684252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e40c8dc0 con 0x7fe5e40d22b0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5dbfff640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40b94d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00b4a0 tx=0x7fe5cc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5db7fe640 1 -- 172.21.15.77:0/991684252 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc014070 con 0x7fe5e40d22b0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5db7fe640 1 -- 172.21.15.77:0/991684252 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc010610 con 0x7fe5e40d22b0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e41f7560 con 0x7fe5e40d22b0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5db7fe640 1 -- 172.21.15.77:0/991684252 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc010a00 con 0x7fe5e40d22b0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.068+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e41f7b20 con 0x7fe5e40d22b0 2024-03-06T07:22:58.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.069+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e40d1d70 con 0x7fe5e40d22b0 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.069+0000 7fe5db7fe640 1 -- 172.21.15.77:0/991684252 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5cc017070 con 0x7fe5e40d22b0 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.070+0000 7fe5db7fe640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe5c406b500 0x7fe5c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.070+0000 7fe5db7fe640 1 -- 172.21.15.77:0/991684252 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5cc089b90 con 0x7fe5e40d22b0 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.070+0000 7fe5e2ffd640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe5c406b500 0x7fe5c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.070+0000 7fe5e2ffd640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe5c406b500 0x7fe5c406d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400b810 tx=0x7fe5d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.074+0000 7fe5db7fe640 1 -- 172.21.15.77:0/991684252 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5cc058aa0 con 0x7fe5e40d22b0 2024-03-06T07:22:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.281+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe5e4068ff0 con 0x7fe5e40d22b0 2024-03-06T07:22:58.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.341+0000 7fe5db7fe640 1 -- 172.21.15.77:0/991684252 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v12) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe5cc0583a0 con 0x7fe5e40d22b0 2024-03-06T07:22:58.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe5c406b500 msgr2=0x7fe5c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe5c406b500 0x7fe5c406d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400b810 tx=0x7fe5d4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 msgr2=0x7fe5e40b94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40b94d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00b4a0 tx=0x7fe5cc00b960 comp rx=0 tx=0).stop 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 shutdown_connections 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe5c406b500 0x7fe5c406d9c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5e40d22b0 0x7fe5e40b94d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5e41614d0 0x7fe5e40c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 --2- 172.21.15.77:0/991684252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5e40d1910 0x7fe5e40ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 >> 172.21.15.77:0/991684252 conn(0x7fe5e40d3d40 msgr2=0x7fe5e40b18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 shutdown_connections 2024-03-06T07:22:58.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.344+0000 7fe5e94c3640 1 -- 172.21.15.77:0/991684252 wait complete. 2024-03-06T07:22:58.358 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:58.358 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:58.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.458+0000 7fcd3b647640 1 Processor -- start 2024-03-06T07:22:58.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.458+0000 7fcd3b647640 1 -- start start 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd3b647640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415bb40 0x7fcd340b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd3b647640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd340cfa90 0x7fcd340cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd3b647640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3415ff90 con 0x7fcd3415f7d0 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd3b647640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3406c910 con 0x7fcd340cfa90 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd3b647640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd34160270 con 0x7fcd3415bb40 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd39be4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd340cfa90 0x7fcd340cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd39be4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd340cfa90 0x7fcd340cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52828/0 (socket says 172.21.15.77:52828) 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd39be4640 1 -- 172.21.15.77:0/3926853371 learned_addr learned my addr 172.21.15.77:0/3926853371 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:58.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd393e3640 1 --2- 172.21.15.77:0/3926853371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415bb40 0x7fcd340b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd39be4640 1 -- 172.21.15.77:0/3926853371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415bb40 msgr2=0x7fcd340b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd39be4640 1 --2- 172.21.15.77:0/3926853371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415bb40 0x7fcd340b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd39be4640 1 -- 172.21.15.77:0/3926853371 >> v1:172.21.15.158:6789/0 conn(0x7fcd3415f7d0 legacy=0x7fcd3406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.459+0000 7fcd39be4640 1 -- 172.21.15.77:0/3926853371 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd34160550 con 0x7fcd340cfa90 2024-03-06T07:22:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.460+0000 7fcd39be4640 1 --2- 172.21.15.77:0/3926853371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340cfa90 0x7fcd340cff20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009870 tx=0x7fcd2c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5602fb6463827760 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.460+0000 7fcd33fff640 1 -- 172.21.15.77:0/3926853371 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c034070 con 0x7fcd340cfa90 2024-03-06T07:22:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.460+0000 7fcd33fff640 1 -- 172.21.15.77:0/3926853371 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c030840 con 0x7fcd340cfa90 2024-03-06T07:22:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.460+0000 7fcd33fff640 1 -- 172.21.15.77:0/3926853371 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c030c10 con 0x7fcd340cfa90 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.461+0000 7fcd3b647640 1 -- 172.21.15.77:0/3926853371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340cfa90 msgr2=0x7fcd340cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.461+0000 7fcd3b647640 1 --2- 172.21.15.77:0/3926853371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340cfa90 0x7fcd340cff20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009870 tx=0x7fcd2c02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.461+0000 7fcd3b647640 1 -- 172.21.15.77:0/3926853371 shutdown_connections 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.461+0000 7fcd3b647640 1 --2- 172.21.15.77:0/3926853371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340cfa90 0x7fcd340cff20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.461+0000 7fcd3b647640 1 --2- 172.21.15.77:0/3926853371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415bb40 0x7fcd340b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.461+0000 7fcd3b647640 1 -- 172.21.15.77:0/3926853371 >> 172.21.15.77:0/3926853371 conn(0x7fcd3406b190 msgr2=0x7fcd3406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.461+0000 7fcd3b647640 1 -- 172.21.15.77:0/3926853371 shutdown_connections 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.461+0000 7fcd3b647640 1 -- 172.21.15.77:0/3926853371 wait complete. 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.462+0000 7fcd3b647640 1 Processor -- start 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.462+0000 7fcd3b647640 1 -- start start 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd3b647640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340cfa90 0x7fcd3414fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd3b647640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd3415bb40 0x7fcd3415e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd3b647640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415f7d0 0x7fcd34158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd3b647640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3406d790 con 0x7fcd3415bb40 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd3b647640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3415fc30 con 0x7fcd340cfa90 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd3b647640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3415bf20 con 0x7fcd3415f7d0 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd3415bb40 0x7fcd3415e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd3415bb40 0x7fcd3415e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52838/0 (socket says 172.21.15.77:52838) 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 -- 172.21.15.77:0/426974435 learned_addr learned my addr 172.21.15.77:0/426974435 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd393e3640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340cfa90 0x7fcd3414fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd39be4640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415f7d0 0x7fcd34158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 -- 172.21.15.77:0/426974435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415f7d0 msgr2=0x7fcd34158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415f7d0 0x7fcd34158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 -- 172.21.15.77:0/426974435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340cfa90 msgr2=0x7fcd3414fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340cfa90 0x7fcd3414fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 -- 172.21.15.77:0/426974435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3415f0f0 con 0x7fcd3415bb40 2024-03-06T07:22:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd38be2640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd3415bb40 0x7fcd3415e410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c3c0 tx=0x7fcd2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd31ffb640 1 -- 172.21.15.77:0/426974435 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd280052d0 con 0x7fcd3415bb40 2024-03-06T07:22:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd31ffb640 1 -- 172.21.15.77:0/426974435 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd28005490 con 0x7fcd3415bb40 2024-03-06T07:22:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.463+0000 7fcd31ffb640 1 -- 172.21.15.77:0/426974435 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28007260 con 0x7fcd3415bb40 2024-03-06T07:22:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.464+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3415ee10 con 0x7fcd3415bb40 2024-03-06T07:22:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.464+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd341f1890 con 0x7fcd3415bb40 2024-03-06T07:22:58.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.465+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd340732b0 con 0x7fcd3415bb40 2024-03-06T07:22:58.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.465+0000 7fcd31ffb640 1 -- 172.21.15.77:0/426974435 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd28007400 con 0x7fcd3415bb40 2024-03-06T07:22:58.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.465+0000 7fcd31ffb640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2006b4d0 0x7fcd2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.465+0000 7fcd393e3640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2006b4d0 0x7fcd2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.465+0000 7fcd31ffb640 1 -- 172.21.15.77:0/426974435 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd280895a0 con 0x7fcd3415bb40 2024-03-06T07:22:58.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.466+0000 7fcd393e3640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2006b4d0 0x7fcd2006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcd3415a3b0 tx=0x7fcd2400f040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.470+0000 7fcd31ffb640 1 -- 172.21.15.77:0/426974435 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd28057e60 con 0x7fcd3415bb40 2024-03-06T07:22:58.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:58.682+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcd34068ff0 con 0x7fcd3415bb40 2024-03-06T07:22:59.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.346+0000 7fcd31ffb640 1 -- 172.21.15.77:0/426974435 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v13) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcd2805ce20 con 0x7fcd3415bb40 2024-03-06T07:22:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2006b4d0 msgr2=0x7fcd2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2006b4d0 0x7fcd2006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcd3415a3b0 tx=0x7fcd2400f040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd3415bb40 msgr2=0x7fcd3415e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd3415bb40 0x7fcd3415e410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c3c0 tx=0x7fcd2800c880 comp rx=0 tx=0).stop 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 shutdown_connections 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2006b4d0 0x7fcd2006d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd3415bb40 0x7fcd3415e410 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd3415f7d0 0x7fcd34158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 --2- 172.21.15.77:0/426974435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340cfa90 0x7fcd3414fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 >> 172.21.15.77:0/426974435 conn(0x7fcd3406b190 msgr2=0x7fcd340af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 shutdown_connections 2024-03-06T07:22:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.348+0000 7fcd3b647640 1 -- 172.21.15.77:0/426974435 wait complete. 2024-03-06T07:22:59.362 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:59.363 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:59.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:59.364 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:59.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.503+0000 7fae9acaf640 1 Processor -- start 2024-03-06T07:22:59.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.503+0000 7fae9acaf640 1 -- start start 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae9acaf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fae9413dc10 0x7fae940b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae9acaf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae940b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae9acaf640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9415f580 con 0x7fae9415e850 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae9acaf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae940cf3f0 con 0x7fae9413dc10 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae9acaf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9415b210 con 0x7fae9415bf30 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae98a4b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fae9413dc10 0x7fae940b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae98a4b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fae9413dc10 0x7fae940b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52846/0 (socket says 172.21.15.77:52846) 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae98a4b640 1 -- 172.21.15.77:0/1888042859 learned_addr learned my addr 172.21.15.77:0/1888042859 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.504+0000 7fae93fff640 1 --2- 172.21.15.77:0/1888042859 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae940b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.505+0000 7fae98a4b640 1 -- 172.21.15.77:0/1888042859 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 msgr2=0x7fae940b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.505+0000 7fae98a4b640 1 --2- 172.21.15.77:0/1888042859 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae940b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.505+0000 7fae98a4b640 1 -- 172.21.15.77:0/1888042859 >> v1:172.21.15.158:6789/0 conn(0x7fae9415e850 legacy=0x7fae9415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.505+0000 7fae98a4b640 1 -- 172.21.15.77:0/1888042859 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9415b450 con 0x7fae9413dc10 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.505+0000 7fae98a4b640 1 --2- 172.21.15.77:0/1888042859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9413dc10 0x7fae940b2950 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fae84009870 tx=0x7fae8402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14edd87bc2bb671 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.505+0000 7fae937fe640 1 -- 172.21.15.77:0/1888042859 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84035020 con 0x7fae9413dc10 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.505+0000 7fae937fe640 1 -- 172.21.15.77:0/1888042859 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae84031640 con 0x7fae9413dc10 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.505+0000 7fae937fe640 1 -- 172.21.15.77:0/1888042859 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8402d040 con 0x7fae9413dc10 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.506+0000 7fae9acaf640 1 -- 172.21.15.77:0/1888042859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9413dc10 msgr2=0x7fae940b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.506+0000 7fae9acaf640 1 --2- 172.21.15.77:0/1888042859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9413dc10 0x7fae940b2950 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fae84009870 tx=0x7fae8402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.506+0000 7fae9acaf640 1 -- 172.21.15.77:0/1888042859 shutdown_connections 2024-03-06T07:22:59.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.506+0000 7fae9acaf640 1 --2- 172.21.15.77:0/1888042859 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae940b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.506+0000 7fae9acaf640 1 --2- 172.21.15.77:0/1888042859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9413dc10 0x7fae940b2950 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.506+0000 7fae9acaf640 1 -- 172.21.15.77:0/1888042859 >> 172.21.15.77:0/1888042859 conn(0x7fae940cf8a0 msgr2=0x7fae940cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.506+0000 7fae9acaf640 1 -- 172.21.15.77:0/1888042859 shutdown_connections 2024-03-06T07:22:59.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.506+0000 7fae9acaf640 1 -- 172.21.15.77:0/1888042859 wait complete. 2024-03-06T07:22:59.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.507+0000 7fae9acaf640 1 Processor -- start 2024-03-06T07:22:59.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.507+0000 7fae9acaf640 1 -- start start 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9acaf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae9413dc10 0x7fae94150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9acaf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae941341f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9acaf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9415e850 0x7fae94145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9acaf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940cd7a0 con 0x7fae9415e850 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9acaf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9413dff0 con 0x7fae9413dc10 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9acaf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae940cd920 con 0x7fae9415bf30 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9924c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9415e850 0x7fae94145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9924c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9415e850 0x7fae94145db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52860/0 (socket says 172.21.15.77:52860) 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9924c640 1 -- 172.21.15.77:0/2655736056 learned_addr learned my addr 172.21.15.77:0/2655736056 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae98a4b640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae9413dc10 0x7fae94150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae93fff640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae941341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9924c640 1 -- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 msgr2=0x7fae941341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9924c640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae941341f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9924c640 1 -- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae9413dc10 msgr2=0x7fae94150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9924c640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae9413dc10 0x7fae94150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae9924c640 1 -- 172.21.15.77:0/2655736056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94144a30 con 0x7fae9415e850 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.508+0000 7fae93fff640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae941341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.509+0000 7fae9924c640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9415e850 0x7fae94145db0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00c6e0 tx=0x7fae8c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.509+0000 7fae917fa640 1 -- 172.21.15.77:0/2655736056 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c003b70 con 0x7fae9415e850 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.509+0000 7fae917fa640 1 -- 172.21.15.77:0/2655736056 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c003d30 con 0x7fae9415e850 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.509+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae94144d10 con 0x7fae9415e850 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.509+0000 7fae917fa640 1 -- 172.21.15.77:0/2655736056 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c005310 con 0x7fae9415e850 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.509+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae941452d0 con 0x7fae9415e850 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.510+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9404f330 con 0x7fae9415e850 2024-03-06T07:22:59.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.510+0000 7fae917fa640 1 -- 172.21.15.77:0/2655736056 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae8c0054b0 con 0x7fae9415e850 2024-03-06T07:22:59.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.510+0000 7fae917fa640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fae8006b4d0 0x7fae8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.511+0000 7fae917fa640 1 -- 172.21.15.77:0/2655736056 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae8c08a020 con 0x7fae9415e850 2024-03-06T07:22:59.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.511+0000 7fae98a4b640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fae8006b4d0 0x7fae8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.511+0000 7fae98a4b640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fae8006b4d0 0x7fae8006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fae84036000 tx=0x7fae8403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.515+0000 7fae917fa640 1 -- 172.21.15.77:0/2655736056 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae8c058ea0 con 0x7fae9415e850 2024-03-06T07:22:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.723+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae94137000 con 0x7fae9415e850 2024-03-06T07:22:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.723+0000 7fae917fa640 1 -- 172.21.15.77:0/2655736056 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fae8c0587a0 con 0x7fae9415e850 2024-03-06T07:22:59.726 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:22:59.727 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:22:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:22:59.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fae8006b4d0 msgr2=0x7fae8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fae8006b4d0 0x7fae8006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fae84036000 tx=0x7fae8403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9415e850 msgr2=0x7fae94145db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9415e850 0x7fae94145db0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00c6e0 tx=0x7fae8c00cba0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 shutdown_connections 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fae8006b4d0 0x7fae8006d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae9415e850 0x7fae94145db0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fae9415bf30 0x7fae941341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 --2- 172.21.15.77:0/2655736056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fae9413dc10 0x7fae94150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 >> 172.21.15.77:0/2655736056 conn(0x7fae940cf8a0 msgr2=0x7fae940b28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.725+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 shutdown_connections 2024-03-06T07:22:59.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.726+0000 7fae9acaf640 1 -- 172.21.15.77:0/2655736056 wait complete. 2024-03-06T07:22:59.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:59.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.836+0000 7f6200803640 1 Processor -- start 2024-03-06T07:22:59.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.836+0000 7f6200803640 1 -- start start 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.837+0000 7f6200803640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61fc06b6c0 0x7f61fc0ada70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.837+0000 7f6200803640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc15e180 con 0x7f61fc0adfb0 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.837+0000 7f6200803640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc0ae3c0 con 0x7f61fc06f060 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.837+0000 7f6200803640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61fc0ae680 con 0x7f61fc06b6c0 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.837+0000 7f61fb59e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f61fc0adfb0 0x7f61fc0b38f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52146/0 (socket says 172.21.15.77:52146) 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.837+0000 7f61fb59e640 1 -- 172.21.15.77:0/1868248811 learned_addr learned my addr 172.21.15.77:0/1868248811 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.837+0000 7f61fa59c640 1 --2- 172.21.15.77:0/1868248811 >> v2:172.21.15.158:3300/0 conn(0x7f61fc06b6c0 0x7f61fc0ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.838+0000 7f61f9d9b640 1 -- 172.21.15.77:0/1868248811 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249244155 0 0) 0x7f61fc15e180 con 0x7f61fc0adfb0 2024-03-06T07:22:59.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.838+0000 7f61f9d9b640 1 -- 172.21.15.77:0/1868248811 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e4003520 con 0x7f61fc0adfb0 2024-03-06T07:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.838+0000 7f61f9d9b640 1 -- 172.21.15.77:0/1868248811 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969399336 0 0) 0x7f61fc0ae3c0 con 0x7f61fc06f060 2024-03-06T07:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.838+0000 7f61f9d9b640 1 -- 172.21.15.77:0/1868248811 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc15e180 con 0x7f61fc06f060 2024-03-06T07:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.838+0000 7f61fa59c640 1 -- 172.21.15.77:0/1868248811 >> v1:172.21.15.158:6789/0 conn(0x7f61fc06f060 legacy=0x7f61fc06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.838+0000 7f61f9d9b640 1 -- 172.21.15.77:0/1868248811 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 423903618 0 0) 0x7f61e4003520 con 0x7f61fc0adfb0 2024-03-06T07:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.838+0000 7f61fa59c640 1 -- 172.21.15.77:0/1868248811 >> v1:172.21.15.77:6789/0 conn(0x7f61fc0adfb0 legacy=0x7f61fc0b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.838+0000 7f61fa59c640 1 -- 172.21.15.77:0/1868248811 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc15e440 con 0x7f61fc06b6c0 2024-03-06T07:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f61fa59c640 1 --2- 172.21.15.77:0/1868248811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 0x7f61fc0ada70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f61f00097a0 tx=0x7f61f002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b5d885b45d09aac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f61f9d9b640 1 -- 172.21.15.77:0/1868248811 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0037070 con 0x7f61fc06b6c0 2024-03-06T07:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f61f9d9b640 1 -- 172.21.15.77:0/1868248811 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f0030510 con 0x7f61fc06b6c0 2024-03-06T07:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f61f9d9b640 1 -- 172.21.15.77:0/1868248811 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4001670 con 0x7f61fc06b6c0 2024-03-06T07:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f6200803640 1 -- 172.21.15.77:0/1868248811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 msgr2=0x7f61fc0ada70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f6200803640 1 --2- 172.21.15.77:0/1868248811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 0x7f61fc0ada70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f61f00097a0 tx=0x7f61f002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:59.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f6200803640 1 -- 172.21.15.77:0/1868248811 shutdown_connections 2024-03-06T07:22:59.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f6200803640 1 --2- 172.21.15.77:0/1868248811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 0x7f61fc0ada70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f6200803640 1 -- 172.21.15.77:0/1868248811 >> 172.21.15.77:0/1868248811 conn(0x7f61fc0d0860 msgr2=0x7f61fc0d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f6200803640 1 -- 172.21.15.77:0/1868248811 shutdown_connections 2024-03-06T07:22:59.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.839+0000 7f6200803640 1 -- 172.21.15.77:0/1868248811 wait complete. 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.841+0000 7f6200803640 1 Processor -- start 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.841+0000 7f6200803640 1 -- start start 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f6200803640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 0x7f61fc0cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f6200803640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61fc06f060 0x7f61fc14a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f6200803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61fc0adfb0 0x7f61fc1f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f6200803640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc15ef20 con 0x7f61fc0adfb0 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f6200803640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc15ea00 con 0x7f61fc06b6c0 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f6200803640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc15f0a0 con 0x7f61fc06f060 2024-03-06T07:22:59.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61fc0adfb0 0x7f61fc1f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61fc0adfb0 0x7f61fc1f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52864/0 (socket says 172.21.15.77:52864) 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 -- 172.21.15.77:0/520787987 learned_addr learned my addr 172.21.15.77:0/520787987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fa59c640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61fc06f060 0x7f61fc14a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fad9d640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 0x7f61fc0cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 -- 172.21.15.77:0/520787987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61fc06f060 msgr2=0x7f61fc14a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61fc06f060 0x7f61fc14a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 -- 172.21.15.77:0/520787987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 msgr2=0x7f61fc0cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 0x7f61fc0cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 -- 172.21.15.77:0/520787987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc0d0160 con 0x7f61fc0adfb0 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61fb59e640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61fc0adfb0 0x7f61fc1f3990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f61f400c500 tx=0x7f61f400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61e37fe640 1 -- 172.21.15.77:0/520787987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4005ce0 con 0x7f61fc0adfb0 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61e37fe640 1 -- 172.21.15.77:0/520787987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4007070 con 0x7f61fc0adfb0 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f61e37fe640 1 -- 172.21.15.77:0/520787987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4007460 con 0x7f61fc0adfb0 2024-03-06T07:22:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc159480 con 0x7f61fc0adfb0 2024-03-06T07:22:59.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.842+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc159a40 con 0x7f61fc0adfb0 2024-03-06T07:22:59.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.843+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61fc068f80 con 0x7f61fc0adfb0 2024-03-06T07:22:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.844+0000 7f61e37fe640 1 -- 172.21.15.77:0/520787987 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61f4007600 con 0x7f61fc0adfb0 2024-03-06T07:22:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.844+0000 7f61e37fe640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f61e806b4d0 0x7f61e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.844+0000 7f61e37fe640 1 -- 172.21.15.77:0/520787987 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61f4089d60 con 0x7f61fc0adfb0 2024-03-06T07:22:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.844+0000 7f61fad9d640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f61e806b4d0 0x7f61e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.845+0000 7f61fad9d640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f61e806b4d0 0x7f61e806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f61ec001560 tx=0x7f61ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:22:59.848+0000 7f61e37fe640 1 -- 172.21.15.77:0/520787987 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61fc0d0160 con 0x7f61fc0adfb0 2024-03-06T07:23:00.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.020+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61fc0adc80 con 0x7f61fc0adfb0 2024-03-06T07:23:00.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.021+0000 7f61e37fe640 1 -- 172.21.15.77:0/520787987 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f61f4058c70 con 0x7f61fc0adfb0 2024-03-06T07:23:00.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:00.024 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:58.329001+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:54.903388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.652222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f61e806b4d0 msgr2=0x7f61e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f61e806b4d0 0x7f61e806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f61ec001560 tx=0x7f61ec009040 comp rx=0 tx=0).stop 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61fc0adfb0 msgr2=0x7f61fc1f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61fc0adfb0 0x7f61fc1f3990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f61f400c500 tx=0x7f61f400c900 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 shutdown_connections 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f61e806b4d0 0x7f61e806d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61fc0adfb0 0x7f61fc1f3990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61fc06f060 0x7f61fc14a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 --2- 172.21.15.77:0/520787987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61fc06b6c0 0x7f61fc0cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 >> 172.21.15.77:0/520787987 conn(0x7f61fc0d0860 msgr2=0x7f61fc072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 shutdown_connections 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.023+0000 7f6200803640 1 -- 172.21.15.77:0/520787987 wait complete. 2024-03-06T07:23:00.037 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:00.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.133+0000 7f3bed55c640 1 Processor -- start 2024-03-06T07:23:00.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.133+0000 7f3bed55c640 1 -- start start 2024-03-06T07:23:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3bed55c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be81515e0 0x7f3be80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3bed55c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3be815b430 0x7f3be8071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3bed55c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be815b8b0 con 0x7f3be80cd7a0 2024-03-06T07:23:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3bed55c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be8070a70 con 0x7f3be815b430 2024-03-06T07:23:00.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3bed55c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be8070d50 con 0x7f3be81515e0 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3be67fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3be815b430 0x7f3be8071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3be67fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3be815b430 0x7f3be8071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52878/0 (socket says 172.21.15.77:52878) 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3be67fc640 1 -- 172.21.15.77:0/2485147124 learned_addr learned my addr 172.21.15.77:0/2485147124 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3be6ffd640 1 --2- 172.21.15.77:0/2485147124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be81515e0 0x7f3be80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.134+0000 7f3be67fc640 1 -- 172.21.15.77:0/2485147124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be81515e0 msgr2=0x7f3be80cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.135+0000 7f3be67fc640 1 --2- 172.21.15.77:0/2485147124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be81515e0 0x7f3be80cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.135+0000 7f3be67fc640 1 -- 172.21.15.77:0/2485147124 >> v1:172.21.15.158:6789/0 conn(0x7f3be80cd7a0 legacy=0x7f3be80cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.135+0000 7f3be67fc640 1 -- 172.21.15.77:0/2485147124 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be8071030 con 0x7f3be815b430 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.135+0000 7f3be67fc640 1 --2- 172.21.15.77:0/2485147124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be815b430 0x7f3be8071830 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009870 tx=0x7f3bdc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2b3e12b14406fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.135+0000 7f3be5ffb640 1 -- 172.21.15.77:0/2485147124 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc034070 con 0x7f3be815b430 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.135+0000 7f3be5ffb640 1 -- 172.21.15.77:0/2485147124 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc030840 con 0x7f3be815b430 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.135+0000 7f3be5ffb640 1 -- 172.21.15.77:0/2485147124 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc030c10 con 0x7f3be815b430 2024-03-06T07:23:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.136+0000 7f3bed55c640 1 -- 172.21.15.77:0/2485147124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be815b430 msgr2=0x7f3be8071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.136+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2485147124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be815b430 0x7f3be8071830 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009870 tx=0x7f3bdc02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.136+0000 7f3bed55c640 1 -- 172.21.15.77:0/2485147124 shutdown_connections 2024-03-06T07:23:00.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.136+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2485147124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be815b430 0x7f3be8071830 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.136+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2485147124 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be81515e0 0x7f3be80cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.136+0000 7f3bed55c640 1 -- 172.21.15.77:0/2485147124 >> 172.21.15.77:0/2485147124 conn(0x7f3be80cf8a0 msgr2=0x7f3be80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.136+0000 7f3bed55c640 1 -- 172.21.15.77:0/2485147124 shutdown_connections 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.136+0000 7f3bed55c640 1 -- 172.21.15.77:0/2485147124 wait complete. 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.137+0000 7f3bed55c640 1 Processor -- start 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.137+0000 7f3bed55c640 1 -- start start 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3bed55c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be80cd7a0 0x7f3be814a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3bed55c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be81515e0 0x7f3be80c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3bed55c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be815b430 0x7f3be80caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3bed55c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be8071ef0 con 0x7f3be80cd7a0 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3bed55c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be8071d70 con 0x7f3be81515e0 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3bed55c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be80cc3a0 con 0x7f3be815b430 2024-03-06T07:23:00.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be80cd7a0 0x7f3be814a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be80cd7a0 0x7f3be814a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52890/0 (socket says 172.21.15.77:52890) 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be6ffd640 1 -- 172.21.15.77:0/2143189460 learned_addr learned my addr 172.21.15.77:0/2143189460 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be67fc640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be81515e0 0x7f3be80c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be77fe640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be815b430 0x7f3be80caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be6ffd640 1 -- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be815b430 msgr2=0x7f3be80caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be6ffd640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be815b430 0x7f3be80caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be6ffd640 1 -- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be81515e0 msgr2=0x7f3be80c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be6ffd640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be81515e0 0x7f3be80c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be6ffd640 1 -- 172.21.15.77:0/2143189460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be80c8f20 con 0x7f3be80cd7a0 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be77fe640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be815b430 0x7f3be80caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.138+0000 7f3be67fc640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be81515e0 0x7f3be80c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.139+0000 7f3be6ffd640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be80cd7a0 0x7f3be814a8a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8009e30 tx=0x7f3bd80075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.139+0000 7f3bcf7fe640 1 -- 172.21.15.77:0/2143189460 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd80145f0 con 0x7f3be80cd7a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.139+0000 7f3bcf7fe640 1 -- 172.21.15.77:0/2143189460 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd8019070 con 0x7f3be80cd7a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.139+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be80c9200 con 0x7f3be80cd7a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.139+0000 7f3bcf7fe640 1 -- 172.21.15.77:0/2143189460 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8014990 con 0x7f3be80cd7a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.139+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be80c97c0 con 0x7f3be80cd7a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.140+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be8068840 con 0x7f3be80cd7a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.140+0000 7f3bcf7fe640 1 -- 172.21.15.77:0/2143189460 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bd8005d10 con 0x7f3be80cd7a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.140+0000 7f3bcf7fe640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd406b4d0 0x7f3bd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.141+0000 7f3bcf7fe640 1 -- 172.21.15.77:0/2143189460 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bd808a950 con 0x7f3be80cd7a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.141+0000 7f3be67fc640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd406b4d0 0x7f3bd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.141+0000 7f3be67fc640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd406b4d0 0x7f3bd406d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc036000 tx=0x7f3bdc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.145+0000 7f3bcf7fe640 1 -- 172.21.15.77:0/2143189460 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3bd8023020 con 0x7f3be80cd7a0 2024-03-06T07:23:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.313+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3be804f330 con 0x7f3bd406b4d0 2024-03-06T07:23:00.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.319+0000 7f3bcf7fe640 1 -- 172.21.15.77:0/2143189460 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251880 (secure 0 0 0) 0x7f3be804f330 con 0x7f3bd406b4d0 2024-03-06T07:23:00.324 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:00.326 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:23:00.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd406b4d0 msgr2=0x7f3bd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd406b4d0 0x7f3bd406d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc036000 tx=0x7f3bdc03b040 comp rx=0 tx=0).stop 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be80cd7a0 msgr2=0x7f3be814a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be80cd7a0 0x7f3be814a8a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8009e30 tx=0x7f3bd80075c0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 shutdown_connections 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be815b430 0x7f3be80caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be81515e0 0x7f3be80c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd406b4d0 0x7f3bd406d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 --2- 172.21.15.77:0/2143189460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be80cd7a0 0x7f3be814a8a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.321+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 >> 172.21.15.77:0/2143189460 conn(0x7f3be80cf8a0 msgr2=0x7f3be8072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.322+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 shutdown_connections 2024-03-06T07:23:00.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.322+0000 7f3bed55c640 1 -- 172.21.15.77:0/2143189460 wait complete. 2024-03-06T07:23:00.336 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-03-06T07:22:59.395082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236712,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738008,"statfs":{"total":773094113280,"available":772851720192,"internally_reserved":0,"allocated":3973120,"data_stored":2237296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.071546"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.340347+0000","last_change":"2024-03-06T07:22:57.338153+0000","last_active":"2024-03-06T07:22:57.340347+0000","last_peered":"2024-03-06T07:22:57.340347+0000","last_clean":"2024-03-06T07:22:57.340347+0000","last_became_active":"2024-03-06T07:22:57.337850+0000","last_became_peered":"2024-03-06T07:22:57.337850+0000","last_unstale":"2024-03-06T07:22:57.340347+0000","last_undegraded":"2024-03-06T07:22:57.340347+0000","last_fullsized":"2024-03-06T07:22:57.340347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:19.589051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350715+0000","last_change":"2024-03-06T07:22:57.340206+0000","last_active":"2024-03-06T07:22:57.350715+0000","last_peered":"2024-03-06T07:22:57.350715+0000","last_clean":"2024-03-06T07:22:57.350715+0000","last_became_active":"2024-03-06T07:22:57.339888+0000","last_became_peered":"2024-03-06T07:22:57.339888+0000","last_unstale":"2024-03-06T07:22:57.350715+0000","last_undegraded":"2024-03-06T07:22:57.350715+0000","last_fullsized":"2024-03-06T07:22:57.350715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:38.047531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356758+0000","last_change":"2024-03-06T07:22:57.355713+0000","last_active":"2024-03-06T07:22:57.356758+0000","last_peered":"2024-03-06T07:22:57.356758+0000","last_clean":"2024-03-06T07:22:57.356758+0000","last_became_active":"2024-03-06T07:22:57.354976+0000","last_became_peered":"2024-03-06T07:22:57.354976+0000","last_unstale":"2024-03-06T07:22:57.356758+0000","last_undegraded":"2024-03-06T07:22:57.356758+0000","last_fullsized":"2024-03-06T07:22:57.356758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:53.809337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357057+0000","last_change":"2024-03-06T07:22:57.349682+0000","last_active":"2024-03-06T07:22:57.357057+0000","last_peered":"2024-03-06T07:22:57.357057+0000","last_clean":"2024-03-06T07:22:57.357057+0000","last_became_active":"2024-03-06T07:22:57.349047+0000","last_became_peered":"2024-03-06T07:22:57.349047+0000","last_unstale":"2024-03-06T07:22:57.357057+0000","last_undegraded":"2024-03-06T07:22:57.357057+0000","last_fullsized":"2024-03-06T07:22:57.357057+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:30.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349306+0000","last_change":"2024-03-06T07:22:57.345033+0000","last_active":"2024-03-06T07:22:57.349306+0000","last_peered":"2024-03-06T07:22:57.349306+0000","last_clean":"2024-03-06T07:22:57.349306+0000","last_became_active":"2024-03-06T07:22:57.344748+0000","last_became_peered":"2024-03-06T07:22:57.344748+0000","last_unstale":"2024-03-06T07:22:57.349306+0000","last_undegraded":"2024-03-06T07:22:57.349306+0000","last_fullsized":"2024-03-06T07:22:57.349306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:11.670187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.341828+0000","last_change":"2024-03-06T07:22:57.338436+0000","last_active":"2024-03-06T07:22:57.341828+0000","last_peered":"2024-03-06T07:22:57.341828+0000","last_clean":"2024-03-06T07:22:57.341828+0000","last_became_active":"2024-03-06T07:22:57.338144+0000","last_became_peered":"2024-03-06T07:22:57.338144+0000","last_unstale":"2024-03-06T07:22:57.341828+0000","last_undegraded":"2024-03-06T07:22:57.341828+0000","last_fullsized":"2024-03-06T07:22:57.341828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:09.314442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356659+0000","last_change":"2024-03-06T07:22:57.352774+0000","last_active":"2024-03-06T07:22:57.356659+0000","last_peered":"2024-03-06T07:22:57.356659+0000","last_clean":"2024-03-06T07:22:57.356659+0000","last_became_active":"2024-03-06T07:22:57.352480+0000","last_became_peered":"2024-03-06T07:22:57.352480+0000","last_unstale":"2024-03-06T07:22:57.356659+0000","last_undegraded":"2024-03-06T07:22:57.356659+0000","last_fullsized":"2024-03-06T07:22:57.356659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.749433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356850+0000","last_change":"2024-03-06T07:22:57.353705+0000","last_active":"2024-03-06T07:22:57.356850+0000","last_peered":"2024-03-06T07:22:57.356850+0000","last_clean":"2024-03-06T07:22:57.356850+0000","last_became_active":"2024-03-06T07:22:57.353413+0000","last_became_peered":"2024-03-06T07:22:57.353413+0000","last_unstale":"2024-03-06T07:22:57.356850+0000","last_undegraded":"2024-03-06T07:22:57.356850+0000","last_fullsized":"2024-03-06T07:22:57.356850+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:05.481638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351746+0000","last_change":"2024-03-06T07:22:57.349290+0000","last_active":"2024-03-06T07:22:57.351746+0000","last_peered":"2024-03-06T07:22:57.351746+0000","last_clean":"2024-03-06T07:22:57.351746+0000","last_became_active":"2024-03-06T07:22:57.348972+0000","last_became_peered":"2024-03-06T07:22:57.348972+0000","last_unstale":"2024-03-06T07:22:57.351746+0000","last_undegraded":"2024-03-06T07:22:57.351746+0000","last_fullsized":"2024-03-06T07:22:57.351746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:36.463052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357662+0000","last_change":"2024-03-06T07:22:57.353712+0000","last_active":"2024-03-06T07:22:57.357662+0000","last_peered":"2024-03-06T07:22:57.357662+0000","last_clean":"2024-03-06T07:22:57.357662+0000","last_became_active":"2024-03-06T07:22:57.353455+0000","last_became_peered":"2024-03-06T07:22:57.353455+0000","last_unstale":"2024-03-06T07:22:57.357662+0000","last_undegraded":"2024-03-06T07:22:57.357662+0000","last_fullsized":"2024-03-06T07:22:57.357662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:14.067439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360107+0000","last_change":"2024-03-06T07:22:57.359109+0000","last_active":"2024-03-06T07:22:57.360107+0000","last_peered":"2024-03-06T07:22:57.360107+0000","last_clean":"2024-03-06T07:22:57.360107+0000","last_became_active":"2024-03-06T07:22:57.358828+0000","last_became_peered":"2024-03-06T07:22:57.358828+0000","last_unstale":"2024-03-06T07:22:57.360107+0000","last_undegraded":"2024-03-06T07:22:57.360107+0000","last_fullsized":"2024-03-06T07:22:57.360107+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:53.809337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349168+0000","last_change":"2024-03-06T07:22:57.341744+0000","last_active":"2024-03-06T07:22:57.349168+0000","last_peered":"2024-03-06T07:22:57.349168+0000","last_clean":"2024-03-06T07:22:57.349168+0000","last_became_active":"2024-03-06T07:22:57.341481+0000","last_became_peered":"2024-03-06T07:22:57.341481+0000","last_unstale":"2024-03-06T07:22:57.349168+0000","last_undegraded":"2024-03-06T07:22:57.349168+0000","last_fullsized":"2024-03-06T07:22:57.349168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:14.067439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351661+0000","last_change":"2024-03-06T07:22:57.346996+0000","last_active":"2024-03-06T07:22:57.351661+0000","last_peered":"2024-03-06T07:22:57.351661+0000","last_clean":"2024-03-06T07:22:57.351661+0000","last_became_active":"2024-03-06T07:22:57.346745+0000","last_became_peered":"2024-03-06T07:22:57.346745+0000","last_unstale":"2024-03-06T07:22:57.351661+0000","last_undegraded":"2024-03-06T07:22:57.351661+0000","last_fullsized":"2024-03-06T07:22:57.351661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:06.940047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357342+0000","last_change":"2024-03-06T07:22:57.353068+0000","last_active":"2024-03-06T07:22:57.357342+0000","last_peered":"2024-03-06T07:22:57.357342+0000","last_clean":"2024-03-06T07:22:57.357342+0000","last_became_active":"2024-03-06T07:22:57.352802+0000","last_became_peered":"2024-03-06T07:22:57.352802+0000","last_unstale":"2024-03-06T07:22:57.357342+0000","last_undegraded":"2024-03-06T07:22:57.357342+0000","last_fullsized":"2024-03-06T07:22:57.357342+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:40.685925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349061+0000","last_change":"2024-03-06T07:22:57.340091+0000","last_active":"2024-03-06T07:22:57.349061+0000","last_peered":"2024-03-06T07:22:57.349061+0000","last_clean":"2024-03-06T07:22:57.349061+0000","last_became_active":"2024-03-06T07:22:57.339831+0000","last_became_peered":"2024-03-06T07:22:57.339831+0000","last_unstale":"2024-03-06T07:22:57.349061+0000","last_undegraded":"2024-03-06T07:22:57.349061+0000","last_fullsized":"2024-03-06T07:22:57.349061+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.749433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360077+0000","last_change":"2024-03-06T07:22:57.358309+0000","last_active":"2024-03-06T07:22:57.360077+0000","last_peered":"2024-03-06T07:22:57.360077+0000","last_clean":"2024-03-06T07:22:57.360077+0000","last_became_active":"2024-03-06T07:22:57.358052+0000","last_became_peered":"2024-03-06T07:22:57.358052+0000","last_unstale":"2024-03-06T07:22:57.360077+0000","last_undegraded":"2024-03-06T07:22:57.360077+0000","last_fullsized":"2024-03-06T07:22:57.360077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:30.027283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357275+0000","last_change":"2024-03-06T07:22:57.356034+0000","last_active":"2024-03-06T07:22:57.357275+0000","last_peered":"2024-03-06T07:22:57.357275+0000","last_clean":"2024-03-06T07:22:57.357275+0000","last_became_active":"2024-03-06T07:22:57.355773+0000","last_became_peered":"2024-03-06T07:22:57.355773+0000","last_unstale":"2024-03-06T07:22:57.357275+0000","last_undegraded":"2024-03-06T07:22:57.357275+0000","last_fullsized":"2024-03-06T07:22:57.357275+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:38.047531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350735+0000","last_change":"2024-03-06T07:22:57.347346+0000","last_active":"2024-03-06T07:22:57.350735+0000","last_peered":"2024-03-06T07:22:57.350735+0000","last_clean":"2024-03-06T07:22:57.350735+0000","last_became_active":"2024-03-06T07:22:57.347089+0000","last_became_peered":"2024-03-06T07:22:57.347089+0000","last_unstale":"2024-03-06T07:22:57.350735+0000","last_undegraded":"2024-03-06T07:22:57.350735+0000","last_fullsized":"2024-03-06T07:22:57.350735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:06.909757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.358005+0000","last_change":"2024-03-06T07:22:57.355558+0000","last_active":"2024-03-06T07:22:57.358005+0000","last_peered":"2024-03-06T07:22:57.358005+0000","last_clean":"2024-03-06T07:22:57.358005+0000","last_became_active":"2024-03-06T07:22:57.355292+0000","last_became_peered":"2024-03-06T07:22:57.355292+0000","last_unstale":"2024-03-06T07:22:57.358005+0000","last_undegraded":"2024-03-06T07:22:57.358005+0000","last_fullsized":"2024-03-06T07:22:57.358005+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:08.116733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356227+0000","last_change":"2024-03-06T07:22:57.351445+0000","last_active":"2024-03-06T07:22:57.356227+0000","last_peered":"2024-03-06T07:22:57.356227+0000","last_clean":"2024-03-06T07:22:57.356227+0000","last_became_active":"2024-03-06T07:22:57.351179+0000","last_became_peered":"2024-03-06T07:22:57.351179+0000","last_unstale":"2024-03-06T07:22:57.356227+0000","last_undegraded":"2024-03-06T07:22:57.356227+0000","last_fullsized":"2024-03-06T07:22:57.356227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.352019+0000","last_change":"2024-03-06T07:22:57.349584+0000","last_active":"2024-03-06T07:22:57.352019+0000","last_peered":"2024-03-06T07:22:57.352019+0000","last_clean":"2024-03-06T07:22:57.352019+0000","last_became_active":"2024-03-06T07:22:57.349288+0000","last_became_peered":"2024-03-06T07:22:57.349288+0000","last_unstale":"2024-03-06T07:22:57.352019+0000","last_undegraded":"2024-03-06T07:22:57.352019+0000","last_fullsized":"2024-03-06T07:22:57.352019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:55.728332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.361189+0000","last_change":"2024-03-06T07:22:57.360454+0000","last_active":"2024-03-06T07:22:57.361189+0000","last_peered":"2024-03-06T07:22:57.361189+0000","last_clean":"2024-03-06T07:22:57.361189+0000","last_became_active":"2024-03-06T07:22:57.360141+0000","last_became_peered":"2024-03-06T07:22:57.360141+0000","last_unstale":"2024-03-06T07:22:57.361189+0000","last_undegraded":"2024-03-06T07:22:57.361189+0000","last_fullsized":"2024-03-06T07:22:57.361189+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:30.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.342931+0000","last_change":"2024-03-06T07:22:57.340224+0000","last_active":"2024-03-06T07:22:57.342931+0000","last_peered":"2024-03-06T07:22:57.342931+0000","last_clean":"2024-03-06T07:22:57.342931+0000","last_became_active":"2024-03-06T07:22:57.339961+0000","last_became_peered":"2024-03-06T07:22:57.339961+0000","last_unstale":"2024-03-06T07:22:57.342931+0000","last_undegraded":"2024-03-06T07:22:57.342931+0000","last_fullsized":"2024-03-06T07:22:57.342931+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:03.289829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350493+0000","last_change":"2024-03-06T07:22:57.341112+0000","last_active":"2024-03-06T07:22:57.350493+0000","last_peered":"2024-03-06T07:22:57.350493+0000","last_clean":"2024-03-06T07:22:57.350493+0000","last_became_active":"2024-03-06T07:22:57.340863+0000","last_became_peered":"2024-03-06T07:22:57.340863+0000","last_unstale":"2024-03-06T07:22:57.350493+0000","last_undegraded":"2024-03-06T07:22:57.350493+0000","last_fullsized":"2024-03-06T07:22:57.350493+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:05.481638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357052+0000","last_change":"2024-03-06T07:22:57.350460+0000","last_active":"2024-03-06T07:22:57.357052+0000","last_peered":"2024-03-06T07:22:57.357052+0000","last_clean":"2024-03-06T07:22:57.357052+0000","last_became_active":"2024-03-06T07:22:57.349599+0000","last_became_peered":"2024-03-06T07:22:57.349599+0000","last_unstale":"2024-03-06T07:22:57.357052+0000","last_undegraded":"2024-03-06T07:22:57.357052+0000","last_fullsized":"2024-03-06T07:22:57.357052+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:03.289829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357377+0000","last_change":"2024-03-06T07:22:57.351261+0000","last_active":"2024-03-06T07:22:57.357377+0000","last_peered":"2024-03-06T07:22:57.357377+0000","last_clean":"2024-03-06T07:22:57.357377+0000","last_became_active":"2024-03-06T07:22:57.350809+0000","last_became_peered":"2024-03-06T07:22:57.350809+0000","last_unstale":"2024-03-06T07:22:57.357377+0000","last_undegraded":"2024-03-06T07:22:57.357377+0000","last_fullsized":"2024-03-06T07:22:57.357377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.493445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357125+0000","last_change":"2024-03-06T07:22:57.351123+0000","last_active":"2024-03-06T07:22:57.357125+0000","last_peered":"2024-03-06T07:22:57.357125+0000","last_clean":"2024-03-06T07:22:57.357125+0000","last_became_active":"2024-03-06T07:22:57.350121+0000","last_became_peered":"2024-03-06T07:22:57.350121+0000","last_unstale":"2024-03-06T07:22:57.357125+0000","last_undegraded":"2024-03-06T07:22:57.357125+0000","last_fullsized":"2024-03-06T07:22:57.357125+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:55.745760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349147+0000","last_change":"2024-03-06T07:22:57.339490+0000","last_active":"2024-03-06T07:22:57.349147+0000","last_peered":"2024-03-06T07:22:57.349147+0000","last_clean":"2024-03-06T07:22:57.349147+0000","last_became_active":"2024-03-06T07:22:57.339235+0000","last_became_peered":"2024-03-06T07:22:57.339235+0000","last_unstale":"2024-03-06T07:22:57.349147+0000","last_undegraded":"2024-03-06T07:22:57.349147+0000","last_fullsized":"2024-03-06T07:22:57.349147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.914740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.361170+0000","last_change":"2024-03-06T07:22:57.360441+0000","last_active":"2024-03-06T07:22:57.361170+0000","last_peered":"2024-03-06T07:22:57.361170+0000","last_clean":"2024-03-06T07:22:57.361170+0000","last_became_active":"2024-03-06T07:22:57.360103+0000","last_became_peered":"2024-03-06T07:22:57.360103+0000","last_unstale":"2024-03-06T07:22:57.361170+0000","last_undegraded":"2024-03-06T07:22:57.361170+0000","last_fullsized":"2024-03-06T07:22:57.361170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:36.659361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356688+0000","last_change":"2024-03-06T07:22:57.351994+0000","last_active":"2024-03-06T07:22:57.356688+0000","last_peered":"2024-03-06T07:22:57.356688+0000","last_clean":"2024-03-06T07:22:57.356688+0000","last_became_active":"2024-03-06T07:22:57.351707+0000","last_became_peered":"2024-03-06T07:22:57.351707+0000","last_unstale":"2024-03-06T07:22:57.356688+0000","last_undegraded":"2024-03-06T07:22:57.356688+0000","last_fullsized":"2024-03-06T07:22:57.356688+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:55.745760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.346487+0000","last_change":"2024-03-06T07:22:57.344507+0000","last_active":"2024-03-06T07:22:57.346487+0000","last_peered":"2024-03-06T07:22:57.346487+0000","last_clean":"2024-03-06T07:22:57.346487+0000","last_became_active":"2024-03-06T07:22:57.344255+0000","last_became_peered":"2024-03-06T07:22:57.344255+0000","last_unstale":"2024-03-06T07:22:57.346487+0000","last_undegraded":"2024-03-06T07:22:57.346487+0000","last_fullsized":"2024-03-06T07:22:57.346487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:30.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356872+0000","last_change":"2024-03-06T07:22:57.353967+0000","last_active":"2024-03-06T07:22:57.356872+0000","last_peered":"2024-03-06T07:22:57.356872+0000","last_clean":"2024-03-06T07:22:57.356872+0000","last_became_active":"2024-03-06T07:22:57.353669+0000","last_became_peered":"2024-03-06T07:22:57.353669+0000","last_unstale":"2024-03-06T07:22:57.356872+0000","last_undegraded":"2024-03-06T07:22:57.356872+0000","last_fullsized":"2024-03-06T07:22:57.356872+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:14.067439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.355023+0000","last_change":"2024-03-06T07:22:57.348941+0000","last_active":"2024-03-06T07:22:57.355023+0000","last_peered":"2024-03-06T07:22:57.355023+0000","last_clean":"2024-03-06T07:22:57.355023+0000","last_became_active":"2024-03-06T07:22:57.348541+0000","last_became_peered":"2024-03-06T07:22:57.348541+0000","last_unstale":"2024-03-06T07:22:57.355023+0000","last_undegraded":"2024-03-06T07:22:57.355023+0000","last_fullsized":"2024-03-06T07:22:57.355023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.914740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.359259+0000","last_change":"2024-03-06T07:22:56.338084+0000","last_active":"2024-03-06T07:22:57.359259+0000","last_peered":"2024-03-06T07:22:57.359259+0000","last_clean":"2024-03-06T07:22:57.359259+0000","last_became_active":"2024-03-06T07:22:56.337792+0000","last_became_peered":"2024-03-06T07:22:56.337792+0000","last_unstale":"2024-03-06T07:22:57.359259+0000","last_undegraded":"2024-03-06T07:22:57.359259+0000","last_fullsized":"2024-03-06T07:22:57.359259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:54.726310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.347423+0000","last_change":"2024-03-06T07:22:57.346789+0000","last_active":"2024-03-06T07:22:57.347423+0000","last_peered":"2024-03-06T07:22:57.347423+0000","last_clean":"2024-03-06T07:22:57.347423+0000","last_became_active":"2024-03-06T07:22:57.345964+0000","last_became_peered":"2024-03-06T07:22:57.345964+0000","last_unstale":"2024-03-06T07:22:57.347423+0000","last_undegraded":"2024-03-06T07:22:57.347423+0000","last_fullsized":"2024-03-06T07:22:57.347423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:42.756008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356833+0000","last_change":"2024-03-06T07:22:56.346592+0000","last_active":"2024-03-06T07:22:57.356833+0000","last_peered":"2024-03-06T07:22:57.356833+0000","last_clean":"2024-03-06T07:22:57.356833+0000","last_became_active":"2024-03-06T07:22:56.346352+0000","last_became_peered":"2024-03-06T07:22:56.346352+0000","last_unstale":"2024-03-06T07:22:57.356833+0000","last_undegraded":"2024-03-06T07:22:57.356833+0000","last_fullsized":"2024-03-06T07:22:57.356833+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:18.587029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.361206+0000","last_change":"2024-03-06T07:22:57.360563+0000","last_active":"2024-03-06T07:22:57.361206+0000","last_peered":"2024-03-06T07:22:57.361206+0000","last_clean":"2024-03-06T07:22:57.361206+0000","last_became_active":"2024-03-06T07:22:57.360163+0000","last_became_peered":"2024-03-06T07:22:57.360163+0000","last_unstale":"2024-03-06T07:22:57.361206+0000","last_undegraded":"2024-03-06T07:22:57.361206+0000","last_fullsized":"2024-03-06T07:22:57.361206+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:42.756008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357627+0000","last_change":"2024-03-06T07:22:56.347367+0000","last_active":"2024-03-06T07:22:57.357627+0000","last_peered":"2024-03-06T07:22:57.357627+0000","last_clean":"2024-03-06T07:22:57.357627+0000","last_became_active":"2024-03-06T07:22:56.347086+0000","last_became_peered":"2024-03-06T07:22:56.347086+0000","last_unstale":"2024-03-06T07:22:57.357627+0000","last_undegraded":"2024-03-06T07:22:57.357627+0000","last_fullsized":"2024-03-06T07:22:57.357627+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:13.065416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357147+0000","last_change":"2024-03-06T07:22:57.351586+0000","last_active":"2024-03-06T07:22:57.357147+0000","last_peered":"2024-03-06T07:22:57.357147+0000","last_clean":"2024-03-06T07:22:57.357147+0000","last_became_active":"2024-03-06T07:22:57.350375+0000","last_became_peered":"2024-03-06T07:22:57.350375+0000","last_unstale":"2024-03-06T07:22:57.357147+0000","last_undegraded":"2024-03-06T07:22:57.357147+0000","last_fullsized":"2024-03-06T07:22:57.357147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.914740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357494+0000","last_change":"2024-03-06T07:22:56.346129+0000","last_active":"2024-03-06T07:22:57.357494+0000","last_peered":"2024-03-06T07:22:57.357494+0000","last_clean":"2024-03-06T07:22:57.357494+0000","last_became_active":"2024-03-06T07:22:56.345171+0000","last_became_peered":"2024-03-06T07:22:56.345171+0000","last_unstale":"2024-03-06T07:22:57.357494+0000","last_undegraded":"2024-03-06T07:22:57.357494+0000","last_fullsized":"2024-03-06T07:22:57.357494+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:39.815801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356798+0000","last_change":"2024-03-06T07:22:57.354530+0000","last_active":"2024-03-06T07:22:57.356798+0000","last_peered":"2024-03-06T07:22:57.356798+0000","last_clean":"2024-03-06T07:22:57.356798+0000","last_became_active":"2024-03-06T07:22:57.354209+0000","last_became_peered":"2024-03-06T07:22:57.354209+0000","last_unstale":"2024-03-06T07:22:57.356798+0000","last_undegraded":"2024-03-06T07:22:57.356798+0000","last_fullsized":"2024-03-06T07:22:57.356798+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:35.401911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360709+0000","last_change":"2024-03-06T07:22:56.342688+0000","last_active":"2024-03-06T07:22:57.360709+0000","last_peered":"2024-03-06T07:22:57.360709+0000","last_clean":"2024-03-06T07:22:57.360709+0000","last_became_active":"2024-03-06T07:22:56.342430+0000","last_became_peered":"2024-03-06T07:22:56.342430+0000","last_unstale":"2024-03-06T07:22:57.360709+0000","last_undegraded":"2024-03-06T07:22:57.360709+0000","last_fullsized":"2024-03-06T07:22:57.360709+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:01.120010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349005+0000","last_change":"2024-03-06T07:22:57.343177+0000","last_active":"2024-03-06T07:22:57.349005+0000","last_peered":"2024-03-06T07:22:57.349005+0000","last_clean":"2024-03-06T07:22:57.349005+0000","last_became_active":"2024-03-06T07:22:57.342911+0000","last_became_peered":"2024-03-06T07:22:57.342911+0000","last_unstale":"2024-03-06T07:22:57.349005+0000","last_undegraded":"2024-03-06T07:22:57.349005+0000","last_fullsized":"2024-03-06T07:22:57.349005+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:59.167118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360844+0000","last_change":"2024-03-06T07:22:56.343403+0000","last_active":"2024-03-06T07:22:57.360844+0000","last_peered":"2024-03-06T07:22:57.360844+0000","last_clean":"2024-03-06T07:22:57.360844+0000","last_became_active":"2024-03-06T07:22:56.342747+0000","last_became_peered":"2024-03-06T07:22:56.342747+0000","last_unstale":"2024-03-06T07:22:57.360844+0000","last_undegraded":"2024-03-06T07:22:57.360844+0000","last_fullsized":"2024-03-06T07:22:57.360844+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:56.789546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351877+0000","last_change":"2024-03-06T07:22:57.350429+0000","last_active":"2024-03-06T07:22:57.351877+0000","last_peered":"2024-03-06T07:22:57.351877+0000","last_clean":"2024-03-06T07:22:57.351877+0000","last_became_active":"2024-03-06T07:22:57.350169+0000","last_became_peered":"2024-03-06T07:22:57.350169+0000","last_unstale":"2024-03-06T07:22:57.351877+0000","last_undegraded":"2024-03-06T07:22:57.351877+0000","last_fullsized":"2024-03-06T07:22:57.351877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:37.755588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.331174+0000","last_change":"2024-03-06T07:22:56.342838+0000","last_active":"2024-03-06T07:22:57.331174+0000","last_peered":"2024-03-06T07:22:57.331174+0000","last_clean":"2024-03-06T07:22:57.331174+0000","last_became_active":"2024-03-06T07:22:56.342549+0000","last_became_peered":"2024-03-06T07:22:56.342549+0000","last_unstale":"2024-03-06T07:22:57.331174+0000","last_undegraded":"2024-03-06T07:22:57.331174+0000","last_fullsized":"2024-03-06T07:22:57.331174+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:54.912718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357246+0000","last_change":"2024-03-06T07:22:57.353296+0000","last_active":"2024-03-06T07:22:57.357246+0000","last_peered":"2024-03-06T07:22:57.357246+0000","last_clean":"2024-03-06T07:22:57.357246+0000","last_became_active":"2024-03-06T07:22:57.353050+0000","last_became_peered":"2024-03-06T07:22:57.353050+0000","last_unstale":"2024-03-06T07:22:57.357246+0000","last_undegraded":"2024-03-06T07:22:57.357246+0000","last_fullsized":"2024-03-06T07:22:57.357246+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:05.481638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.345337+0000","last_change":"2024-03-06T07:22:56.338593+0000","last_active":"2024-03-06T07:22:57.345337+0000","last_peered":"2024-03-06T07:22:57.345337+0000","last_clean":"2024-03-06T07:22:57.345337+0000","last_became_active":"2024-03-06T07:22:56.338301+0000","last_became_peered":"2024-03-06T07:22:56.338301+0000","last_unstale":"2024-03-06T07:22:57.345337+0000","last_undegraded":"2024-03-06T07:22:57.345337+0000","last_fullsized":"2024-03-06T07:22:57.345337+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:41.330236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350344+0000","last_change":"2024-03-06T07:22:57.342398+0000","last_active":"2024-03-06T07:22:57.350344+0000","last_peered":"2024-03-06T07:22:57.350344+0000","last_clean":"2024-03-06T07:22:57.350344+0000","last_became_active":"2024-03-06T07:22:57.342135+0000","last_became_peered":"2024-03-06T07:22:57.342135+0000","last_unstale":"2024-03-06T07:22:57.350344+0000","last_undegraded":"2024-03-06T07:22:57.350344+0000","last_fullsized":"2024-03-06T07:22:57.350344+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:35.401911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.331964+0000","last_change":"2024-03-06T07:22:56.347830+0000","last_active":"2024-03-06T07:22:57.331964+0000","last_peered":"2024-03-06T07:22:57.331964+0000","last_clean":"2024-03-06T07:22:57.331964+0000","last_became_active":"2024-03-06T07:22:56.346511+0000","last_became_peered":"2024-03-06T07:22:56.346511+0000","last_unstale":"2024-03-06T07:22:57.331964+0000","last_undegraded":"2024-03-06T07:22:57.331964+0000","last_fullsized":"2024-03-06T07:22:57.331964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:58.165095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351288+0000","last_change":"2024-03-06T07:22:57.346443+0000","last_active":"2024-03-06T07:22:57.351288+0000","last_peered":"2024-03-06T07:22:57.351288+0000","last_clean":"2024-03-06T07:22:57.351288+0000","last_became_active":"2024-03-06T07:22:57.346154+0000","last_became_peered":"2024-03-06T07:22:57.346154+0000","last_unstale":"2024-03-06T07:22:57.351288+0000","last_undegraded":"2024-03-06T07:22:57.351288+0000","last_fullsized":"2024-03-06T07:22:57.351288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:53.809337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.335081+0000","last_change":"2024-03-06T07:22:56.343614+0000","last_active":"2024-03-06T07:22:57.335081+0000","last_peered":"2024-03-06T07:22:57.335081+0000","last_clean":"2024-03-06T07:22:57.335081+0000","last_became_active":"2024-03-06T07:22:56.343371+0000","last_became_peered":"2024-03-06T07:22:56.343371+0000","last_unstale":"2024-03-06T07:22:57.335081+0000","last_undegraded":"2024-03-06T07:22:57.335081+0000","last_fullsized":"2024-03-06T07:22:57.335081+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:35.657338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.355709+0000","last_change":"2024-03-06T07:22:57.348645+0000","last_active":"2024-03-06T07:22:57.355709+0000","last_peered":"2024-03-06T07:22:57.355709+0000","last_clean":"2024-03-06T07:22:57.355709+0000","last_became_active":"2024-03-06T07:22:57.348393+0000","last_became_peered":"2024-03-06T07:22:57.348393+0000","last_unstale":"2024-03-06T07:22:57.355709+0000","last_undegraded":"2024-03-06T07:22:57.355709+0000","last_fullsized":"2024-03-06T07:22:57.355709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:09.156576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.359078+0000","last_change":"2024-03-06T07:22:56.346371+0000","last_active":"2024-03-06T07:22:57.359078+0000","last_peered":"2024-03-06T07:22:57.359078+0000","last_clean":"2024-03-06T07:22:57.359078+0000","last_became_active":"2024-03-06T07:22:56.346043+0000","last_became_peered":"2024-03-06T07:22:56.346043+0000","last_unstale":"2024-03-06T07:22:57.359078+0000","last_undegraded":"2024-03-06T07:22:57.359078+0000","last_fullsized":"2024-03-06T07:22:57.359078+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.743737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356095+0000","last_change":"2024-03-06T07:22:57.353131+0000","last_active":"2024-03-06T07:22:57.356095+0000","last_peered":"2024-03-06T07:22:57.356095+0000","last_clean":"2024-03-06T07:22:57.356095+0000","last_became_active":"2024-03-06T07:22:57.352879+0000","last_became_peered":"2024-03-06T07:22:57.352879+0000","last_unstale":"2024-03-06T07:22:57.356095+0000","last_undegraded":"2024-03-06T07:22:57.356095+0000","last_fullsized":"2024-03-06T07:22:57.356095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:38.047531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.365719+0000","last_change":"2024-03-06T07:22:56.352000+0000","last_active":"2024-03-06T07:22:57.365719+0000","last_peered":"2024-03-06T07:22:57.365719+0000","last_clean":"2024-03-06T07:22:57.365719+0000","last_became_active":"2024-03-06T07:22:56.351689+0000","last_became_peered":"2024-03-06T07:22:56.351689+0000","last_unstale":"2024-03-06T07:22:57.365719+0000","last_undegraded":"2024-03-06T07:22:57.365719+0000","last_fullsized":"2024-03-06T07:22:57.365719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:34.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357314+0000","last_change":"2024-03-06T07:22:57.353013+0000","last_active":"2024-03-06T07:22:57.357314+0000","last_peered":"2024-03-06T07:22:57.357314+0000","last_clean":"2024-03-06T07:22:57.357314+0000","last_became_active":"2024-03-06T07:22:57.352644+0000","last_became_peered":"2024-03-06T07:22:57.352644+0000","last_unstale":"2024-03-06T07:22:57.357314+0000","last_undegraded":"2024-03-06T07:22:57.357314+0000","last_fullsized":"2024-03-06T07:22:57.357314+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.749433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357815+0000","last_change":"2024-03-06T07:22:56.340575+0000","last_active":"2024-03-06T07:22:57.357815+0000","last_peered":"2024-03-06T07:22:57.357815+0000","last_clean":"2024-03-06T07:22:57.357815+0000","last_became_active":"2024-03-06T07:22:56.340206+0000","last_became_peered":"2024-03-06T07:22:56.340206+0000","last_unstale":"2024-03-06T07:22:57.357815+0000","last_undegraded":"2024-03-06T07:22:57.357815+0000","last_fullsized":"2024-03-06T07:22:57.357815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:50.551483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351417+0000","last_change":"2024-03-06T07:22:57.340938+0000","last_active":"2024-03-06T07:22:57.351417+0000","last_peered":"2024-03-06T07:22:57.351417+0000","last_clean":"2024-03-06T07:22:57.351417+0000","last_became_active":"2024-03-06T07:22:57.340661+0000","last_became_peered":"2024-03-06T07:22:57.340661+0000","last_unstale":"2024-03-06T07:22:57.351417+0000","last_undegraded":"2024-03-06T07:22:57.351417+0000","last_fullsized":"2024-03-06T07:22:57.351417+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:30.027283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.339219+0000","last_change":"2024-03-06T07:22:56.347343+0000","last_active":"2024-03-06T07:22:57.339219+0000","last_peered":"2024-03-06T07:22:57.339219+0000","last_clean":"2024-03-06T07:22:57.339219+0000","last_became_active":"2024-03-06T07:22:56.347086+0000","last_became_peered":"2024-03-06T07:22:56.347086+0000","last_unstale":"2024-03-06T07:22:57.339219+0000","last_undegraded":"2024-03-06T07:22:57.339219+0000","last_fullsized":"2024-03-06T07:22:57.339219+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:54.726310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356889+0000","last_change":"2024-03-06T07:22:57.354474+0000","last_active":"2024-03-06T07:22:57.356889+0000","last_peered":"2024-03-06T07:22:57.356889+0000","last_clean":"2024-03-06T07:22:57.356889+0000","last_became_active":"2024-03-06T07:22:57.354229+0000","last_became_peered":"2024-03-06T07:22:57.354229+0000","last_unstale":"2024-03-06T07:22:57.356889+0000","last_undegraded":"2024-03-06T07:22:57.356889+0000","last_fullsized":"2024-03-06T07:22:57.356889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:59.167118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.331406+0000","last_change":"2024-03-06T07:22:56.345499+0000","last_active":"2024-03-06T07:22:57.331406+0000","last_peered":"2024-03-06T07:22:57.331406+0000","last_clean":"2024-03-06T07:22:57.331406+0000","last_became_active":"2024-03-06T07:22:56.345241+0000","last_became_peered":"2024-03-06T07:22:56.345241+0000","last_unstale":"2024-03-06T07:22:57.331406+0000","last_undegraded":"2024-03-06T07:22:57.331406+0000","last_fullsized":"2024-03-06T07:22:57.331406+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:04.479616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.344669+0000","last_change":"2024-03-06T07:22:57.342753+0000","last_active":"2024-03-06T07:22:57.344669+0000","last_peered":"2024-03-06T07:22:57.344669+0000","last_clean":"2024-03-06T07:22:57.344669+0000","last_became_active":"2024-03-06T07:22:57.342484+0000","last_became_peered":"2024-03-06T07:22:57.342484+0000","last_unstale":"2024-03-06T07:22:57.344669+0000","last_undegraded":"2024-03-06T07:22:57.344669+0000","last_fullsized":"2024-03-06T07:22:57.344669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:36.659361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.358189+0000","last_change":"2024-03-06T07:22:56.343563+0000","last_active":"2024-03-06T07:22:57.358189+0000","last_peered":"2024-03-06T07:22:57.358189+0000","last_clean":"2024-03-06T07:22:57.358189+0000","last_became_active":"2024-03-06T07:22:56.343292+0000","last_became_peered":"2024-03-06T07:22:56.343292+0000","last_unstale":"2024-03-06T07:22:57.358189+0000","last_undegraded":"2024-03-06T07:22:57.358189+0000","last_fullsized":"2024-03-06T07:22:57.358189+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:29.747411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356531+0000","last_change":"2024-03-06T07:22:57.353792+0000","last_active":"2024-03-06T07:22:57.356531+0000","last_peered":"2024-03-06T07:22:57.356531+0000","last_clean":"2024-03-06T07:22:57.356531+0000","last_became_active":"2024-03-06T07:22:57.353526+0000","last_became_peered":"2024-03-06T07:22:57.353526+0000","last_unstale":"2024-03-06T07:22:57.356531+0000","last_undegraded":"2024-03-06T07:22:57.356531+0000","last_fullsized":"2024-03-06T07:22:57.356531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:19.589051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.359902+0000","last_change":"2024-03-06T07:22:56.349086+0000","last_active":"2024-03-06T07:22:57.359902+0000","last_peered":"2024-03-06T07:22:57.359902+0000","last_clean":"2024-03-06T07:22:57.359902+0000","last_became_active":"2024-03-06T07:22:56.348812+0000","last_became_peered":"2024-03-06T07:22:56.348812+0000","last_unstale":"2024-03-06T07:22:57.359902+0000","last_undegraded":"2024-03-06T07:22:57.359902+0000","last_fullsized":"2024-03-06T07:22:57.359902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:26.205859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350526+0000","last_change":"2024-03-06T07:22:57.343810+0000","last_active":"2024-03-06T07:22:57.350526+0000","last_peered":"2024-03-06T07:22:57.350526+0000","last_clean":"2024-03-06T07:22:57.350526+0000","last_became_active":"2024-03-06T07:22:57.343550+0000","last_became_peered":"2024-03-06T07:22:57.343550+0000","last_unstale":"2024-03-06T07:22:57.350526+0000","last_undegraded":"2024-03-06T07:22:57.350526+0000","last_fullsized":"2024-03-06T07:22:57.350526+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:09.314442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357061+0000","last_change":"2024-03-06T07:22:56.339376+0000","last_active":"2024-03-06T07:22:57.357061+0000","last_peered":"2024-03-06T07:22:57.357061+0000","last_clean":"2024-03-06T07:22:57.357061+0000","last_became_active":"2024-03-06T07:22:56.339116+0000","last_became_peered":"2024-03-06T07:22:56.339116+0000","last_unstale":"2024-03-06T07:22:57.357061+0000","last_undegraded":"2024-03-06T07:22:57.357061+0000","last_fullsized":"2024-03-06T07:22:57.357061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:09.071529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357079+0000","last_change":"2024-03-06T07:22:57.349952+0000","last_active":"2024-03-06T07:22:57.357079+0000","last_peered":"2024-03-06T07:22:57.357079+0000","last_clean":"2024-03-06T07:22:57.357079+0000","last_became_active":"2024-03-06T07:22:57.348790+0000","last_became_peered":"2024-03-06T07:22:57.348790+0000","last_unstale":"2024-03-06T07:22:57.357079+0000","last_undegraded":"2024-03-06T07:22:57.357079+0000","last_fullsized":"2024-03-06T07:22:57.357079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:42.756008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.362851+0000","last_change":"2024-03-06T07:22:56.343907+0000","last_active":"2024-03-06T07:22:57.362851+0000","last_peered":"2024-03-06T07:22:57.362851+0000","last_clean":"2024-03-06T07:22:57.362851+0000","last_became_active":"2024-03-06T07:22:56.343579+0000","last_became_peered":"2024-03-06T07:22:56.343579+0000","last_unstale":"2024-03-06T07:22:57.362851+0000","last_undegraded":"2024-03-06T07:22:57.362851+0000","last_fullsized":"2024-03-06T07:22:57.362851+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.668165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.354332+0000","last_change":"2024-03-06T07:22:57.349547+0000","last_active":"2024-03-06T07:22:57.354332+0000","last_peered":"2024-03-06T07:22:57.354332+0000","last_clean":"2024-03-06T07:22:57.354332+0000","last_became_active":"2024-03-06T07:22:57.349301+0000","last_became_peered":"2024-03-06T07:22:57.349301+0000","last_unstale":"2024-03-06T07:22:57.354332+0000","last_undegraded":"2024-03-06T07:22:57.354332+0000","last_fullsized":"2024-03-06T07:22:57.354332+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:55.283065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.339549+0000","last_change":"2024-03-06T07:22:56.344417+0000","last_active":"2024-03-06T07:22:57.339549+0000","last_peered":"2024-03-06T07:22:57.339549+0000","last_clean":"2024-03-06T07:22:57.339549+0000","last_became_active":"2024-03-06T07:22:56.344007+0000","last_became_peered":"2024-03-06T07:22:56.344007+0000","last_unstale":"2024-03-06T07:22:57.339549+0000","last_undegraded":"2024-03-06T07:22:57.339549+0000","last_fullsized":"2024-03-06T07:22:57.339549+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:54.281043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351694+0000","last_change":"2024-03-06T07:22:57.347532+0000","last_active":"2024-03-06T07:22:57.351694+0000","last_peered":"2024-03-06T07:22:57.351694+0000","last_clean":"2024-03-06T07:22:57.351694+0000","last_became_active":"2024-03-06T07:22:57.347291+0000","last_became_peered":"2024-03-06T07:22:57.347291+0000","last_unstale":"2024-03-06T07:22:57.351694+0000","last_undegraded":"2024-03-06T07:22:57.351694+0000","last_fullsized":"2024-03-06T07:22:57.351694+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:30.099116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.358051+0000","last_change":"2024-03-06T07:22:56.342208+0000","last_active":"2024-03-06T07:22:57.358051+0000","last_peered":"2024-03-06T07:22:57.358051+0000","last_clean":"2024-03-06T07:22:57.358051+0000","last_became_active":"2024-03-06T07:22:56.341821+0000","last_became_peered":"2024-03-06T07:22:56.341821+0000","last_unstale":"2024-03-06T07:22:57.358051+0000","last_undegraded":"2024-03-06T07:22:57.358051+0000","last_fullsized":"2024-03-06T07:22:57.358051+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:26.205859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349283+0000","last_change":"2024-03-06T07:22:57.343122+0000","last_active":"2024-03-06T07:22:57.349283+0000","last_peered":"2024-03-06T07:22:57.349283+0000","last_clean":"2024-03-06T07:22:57.349283+0000","last_became_active":"2024-03-06T07:22:57.342833+0000","last_became_peered":"2024-03-06T07:22:57.342833+0000","last_unstale":"2024-03-06T07:22:57.349283+0000","last_undegraded":"2024-03-06T07:22:57.349283+0000","last_fullsized":"2024-03-06T07:22:57.349283+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:12.151743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.334957+0000","last_change":"2024-03-06T07:22:56.341520+0000","last_active":"2024-03-06T07:22:57.334957+0000","last_peered":"2024-03-06T07:22:57.334957+0000","last_clean":"2024-03-06T07:22:57.334957+0000","last_became_active":"2024-03-06T07:22:56.341260+0000","last_became_peered":"2024-03-06T07:22:56.341260+0000","last_unstale":"2024-03-06T07:22:57.334957+0000","last_undegraded":"2024-03-06T07:22:57.334957+0000","last_fullsized":"2024-03-06T07:22:57.334957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:36.753566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349335+0000","last_change":"2024-03-06T07:22:57.343278+0000","last_active":"2024-03-06T07:22:57.349335+0000","last_peered":"2024-03-06T07:22:57.349335+0000","last_clean":"2024-03-06T07:22:57.349335+0000","last_became_active":"2024-03-06T07:22:57.342188+0000","last_became_peered":"2024-03-06T07:22:57.342188+0000","last_unstale":"2024-03-06T07:22:57.349335+0000","last_undegraded":"2024-03-06T07:22:57.349335+0000","last_fullsized":"2024-03-06T07:22:57.349335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:33.205423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.338394+0000","last_change":"2024-03-06T07:22:56.342902+0000","last_active":"2024-03-06T07:22:57.338394+0000","last_peered":"2024-03-06T07:22:57.338394+0000","last_clean":"2024-03-06T07:22:57.338394+0000","last_became_active":"2024-03-06T07:22:56.342598+0000","last_became_peered":"2024-03-06T07:22:56.342598+0000","last_unstale":"2024-03-06T07:22:57.338394+0000","last_undegraded":"2024-03-06T07:22:57.338394+0000","last_fullsized":"2024-03-06T07:22:57.338394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:05.938025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356638+0000","last_change":"2024-03-06T07:22:57.352208+0000","last_active":"2024-03-06T07:22:57.356638+0000","last_peered":"2024-03-06T07:22:57.356638+0000","last_clean":"2024-03-06T07:22:57.356638+0000","last_became_active":"2024-03-06T07:22:57.351950+0000","last_became_peered":"2024-03-06T07:22:57.351950+0000","last_unstale":"2024-03-06T07:22:57.356638+0000","last_undegraded":"2024-03-06T07:22:57.356638+0000","last_fullsized":"2024-03-06T07:22:57.356638+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.914740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.333407+0000","last_change":"2024-03-06T07:22:56.345321+0000","last_active":"2024-03-06T07:22:57.333407+0000","last_peered":"2024-03-06T07:22:57.333407+0000","last_clean":"2024-03-06T07:22:57.333407+0000","last_became_active":"2024-03-06T07:22:56.345047+0000","last_became_peered":"2024-03-06T07:22:56.345047+0000","last_unstale":"2024-03-06T07:22:57.333407+0000","last_undegraded":"2024-03-06T07:22:57.333407+0000","last_fullsized":"2024-03-06T07:22:57.333407+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:02.287807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.354301+0000","last_change":"2024-03-06T07:22:57.350188+0000","last_active":"2024-03-06T07:22:57.354301+0000","last_peered":"2024-03-06T07:22:57.354301+0000","last_clean":"2024-03-06T07:22:57.354301+0000","last_became_active":"2024-03-06T07:22:57.349936+0000","last_became_peered":"2024-03-06T07:22:57.349936+0000","last_unstale":"2024-03-06T07:22:57.354301+0000","last_undegraded":"2024-03-06T07:22:57.354301+0000","last_fullsized":"2024-03-06T07:22:57.354301+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:57.791569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360347+0000","last_change":"2024-03-06T07:22:56.345829+0000","last_active":"2024-03-06T07:22:57.360347+0000","last_peered":"2024-03-06T07:22:57.360347+0000","last_clean":"2024-03-06T07:22:57.360347+0000","last_became_active":"2024-03-06T07:22:56.345558+0000","last_became_peered":"2024-03-06T07:22:56.345558+0000","last_unstale":"2024-03-06T07:22:57.360347+0000","last_undegraded":"2024-03-06T07:22:57.360347+0000","last_fullsized":"2024-03-06T07:22:57.360347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:35.657338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.355994+0000","last_change":"2024-03-06T07:22:57.355348+0000","last_active":"2024-03-06T07:22:57.355994+0000","last_peered":"2024-03-06T07:22:57.355994+0000","last_clean":"2024-03-06T07:22:57.355994+0000","last_became_active":"2024-03-06T07:22:57.353228+0000","last_became_peered":"2024-03-06T07:22:57.353228+0000","last_unstale":"2024-03-06T07:22:57.355994+0000","last_undegraded":"2024-03-06T07:22:57.355994+0000","last_fullsized":"2024-03-06T07:22:57.355994+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:30.027283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.335217+0000","last_change":"2024-03-06T07:22:56.342200+0000","last_active":"2024-03-06T07:22:57.335217+0000","last_peered":"2024-03-06T07:22:57.335217+0000","last_clean":"2024-03-06T07:22:57.335217+0000","last_became_active":"2024-03-06T07:22:56.341853+0000","last_became_peered":"2024-03-06T07:22:56.341853+0000","last_unstale":"2024-03-06T07:22:57.335217+0000","last_undegraded":"2024-03-06T07:22:57.335217+0000","last_fullsized":"2024-03-06T07:22:57.335217+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:41.753985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.353821+0000","last_change":"2024-03-06T07:22:57.349672+0000","last_active":"2024-03-06T07:22:57.353821+0000","last_peered":"2024-03-06T07:22:57.353821+0000","last_clean":"2024-03-06T07:22:57.353821+0000","last_became_active":"2024-03-06T07:22:57.349386+0000","last_became_peered":"2024-03-06T07:22:57.349386+0000","last_unstale":"2024-03-06T07:22:57.353821+0000","last_undegraded":"2024-03-06T07:22:57.353821+0000","last_fullsized":"2024-03-06T07:22:57.353821+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.749433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.339442+0000","last_change":"2024-03-06T07:22:56.344783+0000","last_active":"2024-03-06T07:22:57.339442+0000","last_peered":"2024-03-06T07:22:57.339442+0000","last_clean":"2024-03-06T07:22:57.339442+0000","last_became_active":"2024-03-06T07:22:56.344532+0000","last_became_peered":"2024-03-06T07:22:56.344532+0000","last_unstale":"2024-03-06T07:22:57.339442+0000","last_undegraded":"2024-03-06T07:22:57.339442+0000","last_fullsized":"2024-03-06T07:22:57.339442+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:18.587029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.342892+0000","last_change":"2024-03-06T07:22:57.339944+0000","last_active":"2024-03-06T07:22:57.342892+0000","last_peered":"2024-03-06T07:22:57.342892+0000","last_clean":"2024-03-06T07:22:57.342892+0000","last_became_active":"2024-03-06T07:22:57.339644+0000","last_became_peered":"2024-03-06T07:22:57.339644+0000","last_unstale":"2024-03-06T07:22:57.342892+0000","last_undegraded":"2024-03-06T07:22:57.342892+0000","last_fullsized":"2024-03-06T07:22:57.342892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:27.207882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.335196+0000","last_change":"2024-03-06T07:22:56.336788+0000","last_active":"2024-03-06T07:22:57.335196+0000","last_peered":"2024-03-06T07:22:57.335196+0000","last_clean":"2024-03-06T07:22:57.335196+0000","last_became_active":"2024-03-06T07:22:56.336548+0000","last_became_peered":"2024-03-06T07:22:56.336548+0000","last_unstale":"2024-03-06T07:22:57.335196+0000","last_undegraded":"2024-03-06T07:22:57.335196+0000","last_fullsized":"2024-03-06T07:22:57.335196+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:29.184132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.348162+0000","last_change":"2024-03-06T07:22:57.340934+0000","last_active":"2024-03-06T07:22:57.348162+0000","last_peered":"2024-03-06T07:22:57.348162+0000","last_clean":"2024-03-06T07:22:57.348162+0000","last_became_active":"2024-03-06T07:22:57.340657+0000","last_became_peered":"2024-03-06T07:22:57.340657+0000","last_unstale":"2024-03-06T07:22:57.348162+0000","last_undegraded":"2024-03-06T07:22:57.348162+0000","last_fullsized":"2024-03-06T07:22:57.348162+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:55.745760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.339397+0000","last_change":"2024-03-06T07:22:56.346714+0000","last_active":"2024-03-06T07:22:57.339397+0000","last_peered":"2024-03-06T07:22:57.339397+0000","last_clean":"2024-03-06T07:22:57.339397+0000","last_became_active":"2024-03-06T07:22:56.346466+0000","last_became_peered":"2024-03-06T07:22:56.346466+0000","last_unstale":"2024-03-06T07:22:57.339397+0000","last_undegraded":"2024-03-06T07:22:57.339397+0000","last_fullsized":"2024-03-06T07:22:57.339397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:36.753566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351497+0000","last_change":"2024-03-06T07:22:57.347224+0000","last_active":"2024-03-06T07:22:57.351497+0000","last_peered":"2024-03-06T07:22:57.351497+0000","last_clean":"2024-03-06T07:22:57.351497+0000","last_became_active":"2024-03-06T07:22:57.346958+0000","last_became_peered":"2024-03-06T07:22:57.346958+0000","last_unstale":"2024-03-06T07:22:57.351497+0000","last_undegraded":"2024-03-06T07:22:57.351497+0000","last_fullsized":"2024-03-06T07:22:57.351497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:53.876326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.338543+0000","last_change":"2024-03-06T07:22:56.337441+0000","last_active":"2024-03-06T07:22:57.338543+0000","last_peered":"2024-03-06T07:22:57.338543+0000","last_clean":"2024-03-06T07:22:57.338543+0000","last_became_active":"2024-03-06T07:22:56.337176+0000","last_became_peered":"2024-03-06T07:22:56.337176+0000","last_unstale":"2024-03-06T07:22:57.338543+0000","last_undegraded":"2024-03-06T07:22:57.338543+0000","last_fullsized":"2024-03-06T07:22:57.338543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:29.097094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360633+0000","last_change":"2024-03-06T07:22:57.359815+0000","last_active":"2024-03-06T07:22:57.360633+0000","last_peered":"2024-03-06T07:22:57.360633+0000","last_clean":"2024-03-06T07:22:57.360633+0000","last_became_active":"2024-03-06T07:22:57.359512+0000","last_became_peered":"2024-03-06T07:22:57.359512+0000","last_unstale":"2024-03-06T07:22:57.360633+0000","last_undegraded":"2024-03-06T07:22:57.360633+0000","last_fullsized":"2024-03-06T07:22:57.360633+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:27.207882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.331151+0000","last_change":"2024-03-06T07:22:56.346553+0000","last_active":"2024-03-06T07:22:57.331151+0000","last_peered":"2024-03-06T07:22:57.331151+0000","last_clean":"2024-03-06T07:22:57.331151+0000","last_became_active":"2024-03-06T07:22:56.346283+0000","last_became_peered":"2024-03-06T07:22:56.346283+0000","last_unstale":"2024-03-06T07:22:57.331151+0000","last_undegraded":"2024-03-06T07:22:57.331151+0000","last_fullsized":"2024-03-06T07:22:57.331151+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.743737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356856+0000","last_change":"2024-03-06T07:22:57.355083+0000","last_active":"2024-03-06T07:22:57.356856+0000","last_peered":"2024-03-06T07:22:57.356856+0000","last_clean":"2024-03-06T07:22:57.356856+0000","last_became_active":"2024-03-06T07:22:57.353454+0000","last_became_peered":"2024-03-06T07:22:57.353454+0000","last_unstale":"2024-03-06T07:22:57.356856+0000","last_undegraded":"2024-03-06T07:22:57.356856+0000","last_fullsized":"2024-03-06T07:22:57.356856+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:09.314442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.333409+0000","last_change":"2024-03-06T07:22:56.337866+0000","last_active":"2024-03-06T07:22:57.333409+0000","last_peered":"2024-03-06T07:22:57.333409+0000","last_clean":"2024-03-06T07:22:57.333409+0000","last_became_active":"2024-03-06T07:22:56.337555+0000","last_became_peered":"2024-03-06T07:22:56.337555+0000","last_unstale":"2024-03-06T07:22:57.333409+0000","last_undegraded":"2024-03-06T07:22:57.333409+0000","last_fullsized":"2024-03-06T07:22:57.333409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:35.461029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":189,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.347173+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:22:57.347173+0000","last_peered":"2024-03-06T07:22:57.347173+0000","last_clean":"2024-03-06T07:22:57.347173+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:22:57.347173+0000","last_undegraded":"2024-03-06T07:22:57.347173+0000","last_fullsized":"2024-03-06T07:22:57.347173+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":722438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":722438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:00.340 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.473+0000 7fb83afac640 1 Processor -- start 2024-03-06T07:23:00.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.473+0000 7fb83afac640 1 -- start start 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb83afac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cce40 0x7fb83415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb83afac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb8340cd7a0 0x7fb8340cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb83afac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb834071f40 con 0x7fb8341515e0 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb83afac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8340cd2c0 con 0x7fb8340cd7a0 2024-03-06T07:23:00.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb83afac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb834070a70 con 0x7fb8340cce40 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb838d48640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb8341515e0 0x7fb834071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52158/0 (socket says 172.21.15.77:52158) 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb838d48640 1 -- 172.21.15.77:0/1014830144 learned_addr learned my addr 172.21.15.77:0/1014830144 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb833fff640 1 --2- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cce40 0x7fb83415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb839549640 1 --2- 172.21.15.77:0/1014830144 >> v2:172.21.15.158:3300/0 conn(0x7fb8340cd7a0 0x7fb8340cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb8337fe640 1 -- 172.21.15.77:0/1014830144 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876586024 0 0) 0x7fb834071f40 con 0x7fb8341515e0 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb8337fe640 1 -- 172.21.15.77:0/1014830144 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb81c003520 con 0x7fb8341515e0 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb8337fe640 1 -- 172.21.15.77:0/1014830144 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2673859236 0 0) 0x7fb81c003520 con 0x7fb8341515e0 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb839549640 1 -- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cce40 msgr2=0x7fb83415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb839549640 1 --2- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cce40 0x7fb83415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb839549640 1 -- 172.21.15.77:0/1014830144 >> v1:172.21.15.77:6789/0 conn(0x7fb8341515e0 legacy=0x7fb834071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb839549640 1 -- 172.21.15.77:0/1014830144 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834070d50 con 0x7fb8340cd7a0 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.474+0000 7fb833fff640 1 --2- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cce40 0x7fb83415b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.475+0000 7fb839549640 1 --2- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cd7a0 0x7fb8340cdbd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009d90 tx=0x7fb82c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf86272b02d566b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.475+0000 7fb8337fe640 1 -- 172.21.15.77:0/1014830144 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c039070 con 0x7fb8340cd7a0 2024-03-06T07:23:00.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.475+0000 7fb8337fe640 1 -- 172.21.15.77:0/1014830144 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c02c8a0 con 0x7fb8340cd7a0 2024-03-06T07:23:00.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.475+0000 7fb8337fe640 1 -- 172.21.15.77:0/1014830144 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824001670 con 0x7fb8340cd7a0 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.476+0000 7fb83afac640 1 -- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cd7a0 msgr2=0x7fb8340cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.476+0000 7fb83afac640 1 --2- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cd7a0 0x7fb8340cdbd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009d90 tx=0x7fb82c02bd70 comp rx=0 tx=0).stop 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.476+0000 7fb83afac640 1 -- 172.21.15.77:0/1014830144 shutdown_connections 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.476+0000 7fb83afac640 1 --2- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cd7a0 0x7fb8340cdbd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.476+0000 7fb83afac640 1 --2- 172.21.15.77:0/1014830144 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cce40 0x7fb83415b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.476+0000 7fb83afac640 1 -- 172.21.15.77:0/1014830144 >> 172.21.15.77:0/1014830144 conn(0x7fb8340cf8a0 msgr2=0x7fb8340cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.476+0000 7fb83afac640 1 -- 172.21.15.77:0/1014830144 shutdown_connections 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.476+0000 7fb83afac640 1 -- 172.21.15.77:0/1014830144 wait complete. 2024-03-06T07:23:00.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.477+0000 7fb83afac640 1 Processor -- start 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.477+0000 7fb83afac640 1 -- start start 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb83afac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cce40 0x7fb834147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb83afac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cd7a0 0x7fb8340b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb83afac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8341515e0 0x7fb8340b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb83afac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340ce110 con 0x7fb8341515e0 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb83afac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb834071310 con 0x7fb8340cce40 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb83afac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340cc3a0 con 0x7fb8340cd7a0 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb839549640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8341515e0 0x7fb8340b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb839549640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8341515e0 0x7fb8340b4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52902/0 (socket says 172.21.15.77:52902) 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb839549640 1 -- 172.21.15.77:0/851949796 learned_addr learned my addr 172.21.15.77:0/851949796 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb833fff640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cd7a0 0x7fb8340b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb838d48640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cce40 0x7fb834147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb839549640 1 -- 172.21.15.77:0/851949796 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cd7a0 msgr2=0x7fb8340b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb839549640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cd7a0 0x7fb8340b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb839549640 1 -- 172.21.15.77:0/851949796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cce40 msgr2=0x7fb834147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb839549640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cce40 0x7fb834147740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.478+0000 7fb839549640 1 -- 172.21.15.77:0/851949796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8340c5300 con 0x7fb8341515e0 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.479+0000 7fb839549640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8341515e0 0x7fb8340b4490 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb82c037040 tx=0x7fb82c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.479+0000 7fb8317fa640 1 -- 172.21.15.77:0/851949796 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c039050 con 0x7fb8341515e0 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.479+0000 7fb8317fa640 1 -- 172.21.15.77:0/851949796 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c045070 con 0x7fb8341515e0 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.479+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb834149e00 con 0x7fb8341515e0 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.479+0000 7fb8317fa640 1 -- 172.21.15.77:0/851949796 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c009450 con 0x7fb8341515e0 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.479+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8340c55a0 con 0x7fb8341515e0 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.480+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb834068840 con 0x7fb8341515e0 2024-03-06T07:23:00.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.480+0000 7fb8317fa640 1 -- 172.21.15.77:0/851949796 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb82c040690 con 0x7fb8341515e0 2024-03-06T07:23:00.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.481+0000 7fb8317fa640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb82006b4d0 0x7fb82006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.481+0000 7fb8317fa640 1 -- 172.21.15.77:0/851949796 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb82c0ac880 con 0x7fb8341515e0 2024-03-06T07:23:00.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.481+0000 7fb838d48640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb82006b4d0 0x7fb82006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.481+0000 7fb838d48640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb82006b4d0 0x7fb82006d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb824001320 tx=0x7fb8240012b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.485+0000 7fb8317fa640 1 -- 172.21.15.77:0/851949796 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb82c07b140 con 0x7fb8341515e0 2024-03-06T07:23:00.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.653+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb83404f330 con 0x7fb82006b4d0 2024-03-06T07:23:00.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.659+0000 7fb8317fa640 1 -- 172.21.15.77:0/851949796 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251880 (secure 0 0 0) 0x7fb83404f330 con 0x7fb82006b4d0 2024-03-06T07:23:00.664 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:00.666 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:23:00.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb82006b4d0 msgr2=0x7fb82006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb82006b4d0 0x7fb82006d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb824001320 tx=0x7fb8240012b0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8341515e0 msgr2=0x7fb8340b4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8341515e0 0x7fb8340b4490 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb82c037040 tx=0x7fb82c004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:00.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 shutdown_connections 2024-03-06T07:23:00.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb82006b4d0 0x7fb82006d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8341515e0 0x7fb8340b4490 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8340cd7a0 0x7fb8340b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 --2- 172.21.15.77:0/851949796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8340cce40 0x7fb834147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 >> 172.21.15.77:0/851949796 conn(0x7fb8340cf8a0 msgr2=0x7fb83414efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 shutdown_connections 2024-03-06T07:23:00.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.662+0000 7fb83afac640 1 -- 172.21.15.77:0/851949796 wait complete. 2024-03-06T07:23:00.676 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-03-06T07:22:59.395082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236712,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738008,"statfs":{"total":773094113280,"available":772851720192,"internally_reserved":0,"allocated":3973120,"data_stored":2237296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.071546"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.340347+0000","last_change":"2024-03-06T07:22:57.338153+0000","last_active":"2024-03-06T07:22:57.340347+0000","last_peered":"2024-03-06T07:22:57.340347+0000","last_clean":"2024-03-06T07:22:57.340347+0000","last_became_active":"2024-03-06T07:22:57.337850+0000","last_became_peered":"2024-03-06T07:22:57.337850+0000","last_unstale":"2024-03-06T07:22:57.340347+0000","last_undegraded":"2024-03-06T07:22:57.340347+0000","last_fullsized":"2024-03-06T07:22:57.340347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:19.589051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350715+0000","last_change":"2024-03-06T07:22:57.340206+0000","last_active":"2024-03-06T07:22:57.350715+0000","last_peered":"2024-03-06T07:22:57.350715+0000","last_clean":"2024-03-06T07:22:57.350715+0000","last_became_active":"2024-03-06T07:22:57.339888+0000","last_became_peered":"2024-03-06T07:22:57.339888+0000","last_unstale":"2024-03-06T07:22:57.350715+0000","last_undegraded":"2024-03-06T07:22:57.350715+0000","last_fullsized":"2024-03-06T07:22:57.350715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:38.047531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356758+0000","last_change":"2024-03-06T07:22:57.355713+0000","last_active":"2024-03-06T07:22:57.356758+0000","last_peered":"2024-03-06T07:22:57.356758+0000","last_clean":"2024-03-06T07:22:57.356758+0000","last_became_active":"2024-03-06T07:22:57.354976+0000","last_became_peered":"2024-03-06T07:22:57.354976+0000","last_unstale":"2024-03-06T07:22:57.356758+0000","last_undegraded":"2024-03-06T07:22:57.356758+0000","last_fullsized":"2024-03-06T07:22:57.356758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:53.809337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357057+0000","last_change":"2024-03-06T07:22:57.349682+0000","last_active":"2024-03-06T07:22:57.357057+0000","last_peered":"2024-03-06T07:22:57.357057+0000","last_clean":"2024-03-06T07:22:57.357057+0000","last_became_active":"2024-03-06T07:22:57.349047+0000","last_became_peered":"2024-03-06T07:22:57.349047+0000","last_unstale":"2024-03-06T07:22:57.357057+0000","last_undegraded":"2024-03-06T07:22:57.357057+0000","last_fullsized":"2024-03-06T07:22:57.357057+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:30.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349306+0000","last_change":"2024-03-06T07:22:57.345033+0000","last_active":"2024-03-06T07:22:57.349306+0000","last_peered":"2024-03-06T07:22:57.349306+0000","last_clean":"2024-03-06T07:22:57.349306+0000","last_became_active":"2024-03-06T07:22:57.344748+0000","last_became_peered":"2024-03-06T07:22:57.344748+0000","last_unstale":"2024-03-06T07:22:57.349306+0000","last_undegraded":"2024-03-06T07:22:57.349306+0000","last_fullsized":"2024-03-06T07:22:57.349306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:11.670187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.341828+0000","last_change":"2024-03-06T07:22:57.338436+0000","last_active":"2024-03-06T07:22:57.341828+0000","last_peered":"2024-03-06T07:22:57.341828+0000","last_clean":"2024-03-06T07:22:57.341828+0000","last_became_active":"2024-03-06T07:22:57.338144+0000","last_became_peered":"2024-03-06T07:22:57.338144+0000","last_unstale":"2024-03-06T07:22:57.341828+0000","last_undegraded":"2024-03-06T07:22:57.341828+0000","last_fullsized":"2024-03-06T07:22:57.341828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:09.314442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356659+0000","last_change":"2024-03-06T07:22:57.352774+0000","last_active":"2024-03-06T07:22:57.356659+0000","last_peered":"2024-03-06T07:22:57.356659+0000","last_clean":"2024-03-06T07:22:57.356659+0000","last_became_active":"2024-03-06T07:22:57.352480+0000","last_became_peered":"2024-03-06T07:22:57.352480+0000","last_unstale":"2024-03-06T07:22:57.356659+0000","last_undegraded":"2024-03-06T07:22:57.356659+0000","last_fullsized":"2024-03-06T07:22:57.356659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.749433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356850+0000","last_change":"2024-03-06T07:22:57.353705+0000","last_active":"2024-03-06T07:22:57.356850+0000","last_peered":"2024-03-06T07:22:57.356850+0000","last_clean":"2024-03-06T07:22:57.356850+0000","last_became_active":"2024-03-06T07:22:57.353413+0000","last_became_peered":"2024-03-06T07:22:57.353413+0000","last_unstale":"2024-03-06T07:22:57.356850+0000","last_undegraded":"2024-03-06T07:22:57.356850+0000","last_fullsized":"2024-03-06T07:22:57.356850+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:05.481638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351746+0000","last_change":"2024-03-06T07:22:57.349290+0000","last_active":"2024-03-06T07:22:57.351746+0000","last_peered":"2024-03-06T07:22:57.351746+0000","last_clean":"2024-03-06T07:22:57.351746+0000","last_became_active":"2024-03-06T07:22:57.348972+0000","last_became_peered":"2024-03-06T07:22:57.348972+0000","last_unstale":"2024-03-06T07:22:57.351746+0000","last_undegraded":"2024-03-06T07:22:57.351746+0000","last_fullsized":"2024-03-06T07:22:57.351746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:36.463052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357662+0000","last_change":"2024-03-06T07:22:57.353712+0000","last_active":"2024-03-06T07:22:57.357662+0000","last_peered":"2024-03-06T07:22:57.357662+0000","last_clean":"2024-03-06T07:22:57.357662+0000","last_became_active":"2024-03-06T07:22:57.353455+0000","last_became_peered":"2024-03-06T07:22:57.353455+0000","last_unstale":"2024-03-06T07:22:57.357662+0000","last_undegraded":"2024-03-06T07:22:57.357662+0000","last_fullsized":"2024-03-06T07:22:57.357662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:14.067439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360107+0000","last_change":"2024-03-06T07:22:57.359109+0000","last_active":"2024-03-06T07:22:57.360107+0000","last_peered":"2024-03-06T07:22:57.360107+0000","last_clean":"2024-03-06T07:22:57.360107+0000","last_became_active":"2024-03-06T07:22:57.358828+0000","last_became_peered":"2024-03-06T07:22:57.358828+0000","last_unstale":"2024-03-06T07:22:57.360107+0000","last_undegraded":"2024-03-06T07:22:57.360107+0000","last_fullsized":"2024-03-06T07:22:57.360107+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:53.809337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349168+0000","last_change":"2024-03-06T07:22:57.341744+0000","last_active":"2024-03-06T07:22:57.349168+0000","last_peered":"2024-03-06T07:22:57.349168+0000","last_clean":"2024-03-06T07:22:57.349168+0000","last_became_active":"2024-03-06T07:22:57.341481+0000","last_became_peered":"2024-03-06T07:22:57.341481+0000","last_unstale":"2024-03-06T07:22:57.349168+0000","last_undegraded":"2024-03-06T07:22:57.349168+0000","last_fullsized":"2024-03-06T07:22:57.349168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:14.067439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351661+0000","last_change":"2024-03-06T07:22:57.346996+0000","last_active":"2024-03-06T07:22:57.351661+0000","last_peered":"2024-03-06T07:22:57.351661+0000","last_clean":"2024-03-06T07:22:57.351661+0000","last_became_active":"2024-03-06T07:22:57.346745+0000","last_became_peered":"2024-03-06T07:22:57.346745+0000","last_unstale":"2024-03-06T07:22:57.351661+0000","last_undegraded":"2024-03-06T07:22:57.351661+0000","last_fullsized":"2024-03-06T07:22:57.351661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:06.940047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357342+0000","last_change":"2024-03-06T07:22:57.353068+0000","last_active":"2024-03-06T07:22:57.357342+0000","last_peered":"2024-03-06T07:22:57.357342+0000","last_clean":"2024-03-06T07:22:57.357342+0000","last_became_active":"2024-03-06T07:22:57.352802+0000","last_became_peered":"2024-03-06T07:22:57.352802+0000","last_unstale":"2024-03-06T07:22:57.357342+0000","last_undegraded":"2024-03-06T07:22:57.357342+0000","last_fullsized":"2024-03-06T07:22:57.357342+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:40.685925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349061+0000","last_change":"2024-03-06T07:22:57.340091+0000","last_active":"2024-03-06T07:22:57.349061+0000","last_peered":"2024-03-06T07:22:57.349061+0000","last_clean":"2024-03-06T07:22:57.349061+0000","last_became_active":"2024-03-06T07:22:57.339831+0000","last_became_peered":"2024-03-06T07:22:57.339831+0000","last_unstale":"2024-03-06T07:22:57.349061+0000","last_undegraded":"2024-03-06T07:22:57.349061+0000","last_fullsized":"2024-03-06T07:22:57.349061+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.749433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360077+0000","last_change":"2024-03-06T07:22:57.358309+0000","last_active":"2024-03-06T07:22:57.360077+0000","last_peered":"2024-03-06T07:22:57.360077+0000","last_clean":"2024-03-06T07:22:57.360077+0000","last_became_active":"2024-03-06T07:22:57.358052+0000","last_became_peered":"2024-03-06T07:22:57.358052+0000","last_unstale":"2024-03-06T07:22:57.360077+0000","last_undegraded":"2024-03-06T07:22:57.360077+0000","last_fullsized":"2024-03-06T07:22:57.360077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:30.027283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357275+0000","last_change":"2024-03-06T07:22:57.356034+0000","last_active":"2024-03-06T07:22:57.357275+0000","last_peered":"2024-03-06T07:22:57.357275+0000","last_clean":"2024-03-06T07:22:57.357275+0000","last_became_active":"2024-03-06T07:22:57.355773+0000","last_became_peered":"2024-03-06T07:22:57.355773+0000","last_unstale":"2024-03-06T07:22:57.357275+0000","last_undegraded":"2024-03-06T07:22:57.357275+0000","last_fullsized":"2024-03-06T07:22:57.357275+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:38.047531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350735+0000","last_change":"2024-03-06T07:22:57.347346+0000","last_active":"2024-03-06T07:22:57.350735+0000","last_peered":"2024-03-06T07:22:57.350735+0000","last_clean":"2024-03-06T07:22:57.350735+0000","last_became_active":"2024-03-06T07:22:57.347089+0000","last_became_peered":"2024-03-06T07:22:57.347089+0000","last_unstale":"2024-03-06T07:22:57.350735+0000","last_undegraded":"2024-03-06T07:22:57.350735+0000","last_fullsized":"2024-03-06T07:22:57.350735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:06.909757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.358005+0000","last_change":"2024-03-06T07:22:57.355558+0000","last_active":"2024-03-06T07:22:57.358005+0000","last_peered":"2024-03-06T07:22:57.358005+0000","last_clean":"2024-03-06T07:22:57.358005+0000","last_became_active":"2024-03-06T07:22:57.355292+0000","last_became_peered":"2024-03-06T07:22:57.355292+0000","last_unstale":"2024-03-06T07:22:57.358005+0000","last_undegraded":"2024-03-06T07:22:57.358005+0000","last_fullsized":"2024-03-06T07:22:57.358005+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:08.116733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356227+0000","last_change":"2024-03-06T07:22:57.351445+0000","last_active":"2024-03-06T07:22:57.356227+0000","last_peered":"2024-03-06T07:22:57.356227+0000","last_clean":"2024-03-06T07:22:57.356227+0000","last_became_active":"2024-03-06T07:22:57.351179+0000","last_became_peered":"2024-03-06T07:22:57.351179+0000","last_unstale":"2024-03-06T07:22:57.356227+0000","last_undegraded":"2024-03-06T07:22:57.356227+0000","last_fullsized":"2024-03-06T07:22:57.356227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:02.122033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.352019+0000","last_change":"2024-03-06T07:22:57.349584+0000","last_active":"2024-03-06T07:22:57.352019+0000","last_peered":"2024-03-06T07:22:57.352019+0000","last_clean":"2024-03-06T07:22:57.352019+0000","last_became_active":"2024-03-06T07:22:57.349288+0000","last_became_peered":"2024-03-06T07:22:57.349288+0000","last_unstale":"2024-03-06T07:22:57.352019+0000","last_undegraded":"2024-03-06T07:22:57.352019+0000","last_fullsized":"2024-03-06T07:22:57.352019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:55.728332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.361189+0000","last_change":"2024-03-06T07:22:57.360454+0000","last_active":"2024-03-06T07:22:57.361189+0000","last_peered":"2024-03-06T07:22:57.361189+0000","last_clean":"2024-03-06T07:22:57.361189+0000","last_became_active":"2024-03-06T07:22:57.360141+0000","last_became_peered":"2024-03-06T07:22:57.360141+0000","last_unstale":"2024-03-06T07:22:57.361189+0000","last_undegraded":"2024-03-06T07:22:57.361189+0000","last_fullsized":"2024-03-06T07:22:57.361189+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:30.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.342931+0000","last_change":"2024-03-06T07:22:57.340224+0000","last_active":"2024-03-06T07:22:57.342931+0000","last_peered":"2024-03-06T07:22:57.342931+0000","last_clean":"2024-03-06T07:22:57.342931+0000","last_became_active":"2024-03-06T07:22:57.339961+0000","last_became_peered":"2024-03-06T07:22:57.339961+0000","last_unstale":"2024-03-06T07:22:57.342931+0000","last_undegraded":"2024-03-06T07:22:57.342931+0000","last_fullsized":"2024-03-06T07:22:57.342931+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:03.289829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350493+0000","last_change":"2024-03-06T07:22:57.341112+0000","last_active":"2024-03-06T07:22:57.350493+0000","last_peered":"2024-03-06T07:22:57.350493+0000","last_clean":"2024-03-06T07:22:57.350493+0000","last_became_active":"2024-03-06T07:22:57.340863+0000","last_became_peered":"2024-03-06T07:22:57.340863+0000","last_unstale":"2024-03-06T07:22:57.350493+0000","last_undegraded":"2024-03-06T07:22:57.350493+0000","last_fullsized":"2024-03-06T07:22:57.350493+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:05.481638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357052+0000","last_change":"2024-03-06T07:22:57.350460+0000","last_active":"2024-03-06T07:22:57.357052+0000","last_peered":"2024-03-06T07:22:57.357052+0000","last_clean":"2024-03-06T07:22:57.357052+0000","last_became_active":"2024-03-06T07:22:57.349599+0000","last_became_peered":"2024-03-06T07:22:57.349599+0000","last_unstale":"2024-03-06T07:22:57.357052+0000","last_undegraded":"2024-03-06T07:22:57.357052+0000","last_fullsized":"2024-03-06T07:22:57.357052+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:03.289829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357377+0000","last_change":"2024-03-06T07:22:57.351261+0000","last_active":"2024-03-06T07:22:57.357377+0000","last_peered":"2024-03-06T07:22:57.357377+0000","last_clean":"2024-03-06T07:22:57.357377+0000","last_became_active":"2024-03-06T07:22:57.350809+0000","last_became_peered":"2024-03-06T07:22:57.350809+0000","last_unstale":"2024-03-06T07:22:57.357377+0000","last_undegraded":"2024-03-06T07:22:57.357377+0000","last_fullsized":"2024-03-06T07:22:57.357377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.493445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357125+0000","last_change":"2024-03-06T07:22:57.351123+0000","last_active":"2024-03-06T07:22:57.357125+0000","last_peered":"2024-03-06T07:22:57.357125+0000","last_clean":"2024-03-06T07:22:57.357125+0000","last_became_active":"2024-03-06T07:22:57.350121+0000","last_became_peered":"2024-03-06T07:22:57.350121+0000","last_unstale":"2024-03-06T07:22:57.357125+0000","last_undegraded":"2024-03-06T07:22:57.357125+0000","last_fullsized":"2024-03-06T07:22:57.357125+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:55.745760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349147+0000","last_change":"2024-03-06T07:22:57.339490+0000","last_active":"2024-03-06T07:22:57.349147+0000","last_peered":"2024-03-06T07:22:57.349147+0000","last_clean":"2024-03-06T07:22:57.349147+0000","last_became_active":"2024-03-06T07:22:57.339235+0000","last_became_peered":"2024-03-06T07:22:57.339235+0000","last_unstale":"2024-03-06T07:22:57.349147+0000","last_undegraded":"2024-03-06T07:22:57.349147+0000","last_fullsized":"2024-03-06T07:22:57.349147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.914740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.361170+0000","last_change":"2024-03-06T07:22:57.360441+0000","last_active":"2024-03-06T07:22:57.361170+0000","last_peered":"2024-03-06T07:22:57.361170+0000","last_clean":"2024-03-06T07:22:57.361170+0000","last_became_active":"2024-03-06T07:22:57.360103+0000","last_became_peered":"2024-03-06T07:22:57.360103+0000","last_unstale":"2024-03-06T07:22:57.361170+0000","last_undegraded":"2024-03-06T07:22:57.361170+0000","last_fullsized":"2024-03-06T07:22:57.361170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:36.659361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356688+0000","last_change":"2024-03-06T07:22:57.351994+0000","last_active":"2024-03-06T07:22:57.356688+0000","last_peered":"2024-03-06T07:22:57.356688+0000","last_clean":"2024-03-06T07:22:57.356688+0000","last_became_active":"2024-03-06T07:22:57.351707+0000","last_became_peered":"2024-03-06T07:22:57.351707+0000","last_unstale":"2024-03-06T07:22:57.356688+0000","last_undegraded":"2024-03-06T07:22:57.356688+0000","last_fullsized":"2024-03-06T07:22:57.356688+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:55.745760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.346487+0000","last_change":"2024-03-06T07:22:57.344507+0000","last_active":"2024-03-06T07:22:57.346487+0000","last_peered":"2024-03-06T07:22:57.346487+0000","last_clean":"2024-03-06T07:22:57.346487+0000","last_became_active":"2024-03-06T07:22:57.344255+0000","last_became_peered":"2024-03-06T07:22:57.344255+0000","last_unstale":"2024-03-06T07:22:57.346487+0000","last_undegraded":"2024-03-06T07:22:57.346487+0000","last_fullsized":"2024-03-06T07:22:57.346487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:30.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356872+0000","last_change":"2024-03-06T07:22:57.353967+0000","last_active":"2024-03-06T07:22:57.356872+0000","last_peered":"2024-03-06T07:22:57.356872+0000","last_clean":"2024-03-06T07:22:57.356872+0000","last_became_active":"2024-03-06T07:22:57.353669+0000","last_became_peered":"2024-03-06T07:22:57.353669+0000","last_unstale":"2024-03-06T07:22:57.356872+0000","last_undegraded":"2024-03-06T07:22:57.356872+0000","last_fullsized":"2024-03-06T07:22:57.356872+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:14.067439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.355023+0000","last_change":"2024-03-06T07:22:57.348941+0000","last_active":"2024-03-06T07:22:57.355023+0000","last_peered":"2024-03-06T07:22:57.355023+0000","last_clean":"2024-03-06T07:22:57.355023+0000","last_became_active":"2024-03-06T07:22:57.348541+0000","last_became_peered":"2024-03-06T07:22:57.348541+0000","last_unstale":"2024-03-06T07:22:57.355023+0000","last_undegraded":"2024-03-06T07:22:57.355023+0000","last_fullsized":"2024-03-06T07:22:57.355023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.914740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.359259+0000","last_change":"2024-03-06T07:22:56.338084+0000","last_active":"2024-03-06T07:22:57.359259+0000","last_peered":"2024-03-06T07:22:57.359259+0000","last_clean":"2024-03-06T07:22:57.359259+0000","last_became_active":"2024-03-06T07:22:56.337792+0000","last_became_peered":"2024-03-06T07:22:56.337792+0000","last_unstale":"2024-03-06T07:22:57.359259+0000","last_undegraded":"2024-03-06T07:22:57.359259+0000","last_fullsized":"2024-03-06T07:22:57.359259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:54.726310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.347423+0000","last_change":"2024-03-06T07:22:57.346789+0000","last_active":"2024-03-06T07:22:57.347423+0000","last_peered":"2024-03-06T07:22:57.347423+0000","last_clean":"2024-03-06T07:22:57.347423+0000","last_became_active":"2024-03-06T07:22:57.345964+0000","last_became_peered":"2024-03-06T07:22:57.345964+0000","last_unstale":"2024-03-06T07:22:57.347423+0000","last_undegraded":"2024-03-06T07:22:57.347423+0000","last_fullsized":"2024-03-06T07:22:57.347423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:42.756008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356833+0000","last_change":"2024-03-06T07:22:56.346592+0000","last_active":"2024-03-06T07:22:57.356833+0000","last_peered":"2024-03-06T07:22:57.356833+0000","last_clean":"2024-03-06T07:22:57.356833+0000","last_became_active":"2024-03-06T07:22:56.346352+0000","last_became_peered":"2024-03-06T07:22:56.346352+0000","last_unstale":"2024-03-06T07:22:57.356833+0000","last_undegraded":"2024-03-06T07:22:57.356833+0000","last_fullsized":"2024-03-06T07:22:57.356833+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:18.587029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.361206+0000","last_change":"2024-03-06T07:22:57.360563+0000","last_active":"2024-03-06T07:22:57.361206+0000","last_peered":"2024-03-06T07:22:57.361206+0000","last_clean":"2024-03-06T07:22:57.361206+0000","last_became_active":"2024-03-06T07:22:57.360163+0000","last_became_peered":"2024-03-06T07:22:57.360163+0000","last_unstale":"2024-03-06T07:22:57.361206+0000","last_undegraded":"2024-03-06T07:22:57.361206+0000","last_fullsized":"2024-03-06T07:22:57.361206+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:42.756008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357627+0000","last_change":"2024-03-06T07:22:56.347367+0000","last_active":"2024-03-06T07:22:57.357627+0000","last_peered":"2024-03-06T07:22:57.357627+0000","last_clean":"2024-03-06T07:22:57.357627+0000","last_became_active":"2024-03-06T07:22:56.347086+0000","last_became_peered":"2024-03-06T07:22:56.347086+0000","last_unstale":"2024-03-06T07:22:57.357627+0000","last_undegraded":"2024-03-06T07:22:57.357627+0000","last_fullsized":"2024-03-06T07:22:57.357627+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:13.065416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357147+0000","last_change":"2024-03-06T07:22:57.351586+0000","last_active":"2024-03-06T07:22:57.357147+0000","last_peered":"2024-03-06T07:22:57.357147+0000","last_clean":"2024-03-06T07:22:57.357147+0000","last_became_active":"2024-03-06T07:22:57.350375+0000","last_became_peered":"2024-03-06T07:22:57.350375+0000","last_unstale":"2024-03-06T07:22:57.357147+0000","last_undegraded":"2024-03-06T07:22:57.357147+0000","last_fullsized":"2024-03-06T07:22:57.357147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.914740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357494+0000","last_change":"2024-03-06T07:22:56.346129+0000","last_active":"2024-03-06T07:22:57.357494+0000","last_peered":"2024-03-06T07:22:57.357494+0000","last_clean":"2024-03-06T07:22:57.357494+0000","last_became_active":"2024-03-06T07:22:56.345171+0000","last_became_peered":"2024-03-06T07:22:56.345171+0000","last_unstale":"2024-03-06T07:22:57.357494+0000","last_undegraded":"2024-03-06T07:22:57.357494+0000","last_fullsized":"2024-03-06T07:22:57.357494+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:39.815801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356798+0000","last_change":"2024-03-06T07:22:57.354530+0000","last_active":"2024-03-06T07:22:57.356798+0000","last_peered":"2024-03-06T07:22:57.356798+0000","last_clean":"2024-03-06T07:22:57.356798+0000","last_became_active":"2024-03-06T07:22:57.354209+0000","last_became_peered":"2024-03-06T07:22:57.354209+0000","last_unstale":"2024-03-06T07:22:57.356798+0000","last_undegraded":"2024-03-06T07:22:57.356798+0000","last_fullsized":"2024-03-06T07:22:57.356798+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:35.401911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360709+0000","last_change":"2024-03-06T07:22:56.342688+0000","last_active":"2024-03-06T07:22:57.360709+0000","last_peered":"2024-03-06T07:22:57.360709+0000","last_clean":"2024-03-06T07:22:57.360709+0000","last_became_active":"2024-03-06T07:22:56.342430+0000","last_became_peered":"2024-03-06T07:22:56.342430+0000","last_unstale":"2024-03-06T07:22:57.360709+0000","last_undegraded":"2024-03-06T07:22:57.360709+0000","last_fullsized":"2024-03-06T07:22:57.360709+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:01.120010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349005+0000","last_change":"2024-03-06T07:22:57.343177+0000","last_active":"2024-03-06T07:22:57.349005+0000","last_peered":"2024-03-06T07:22:57.349005+0000","last_clean":"2024-03-06T07:22:57.349005+0000","last_became_active":"2024-03-06T07:22:57.342911+0000","last_became_peered":"2024-03-06T07:22:57.342911+0000","last_unstale":"2024-03-06T07:22:57.349005+0000","last_undegraded":"2024-03-06T07:22:57.349005+0000","last_fullsized":"2024-03-06T07:22:57.349005+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:59.167118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360844+0000","last_change":"2024-03-06T07:22:56.343403+0000","last_active":"2024-03-06T07:22:57.360844+0000","last_peered":"2024-03-06T07:22:57.360844+0000","last_clean":"2024-03-06T07:22:57.360844+0000","last_became_active":"2024-03-06T07:22:56.342747+0000","last_became_peered":"2024-03-06T07:22:56.342747+0000","last_unstale":"2024-03-06T07:22:57.360844+0000","last_undegraded":"2024-03-06T07:22:57.360844+0000","last_fullsized":"2024-03-06T07:22:57.360844+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:56.789546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351877+0000","last_change":"2024-03-06T07:22:57.350429+0000","last_active":"2024-03-06T07:22:57.351877+0000","last_peered":"2024-03-06T07:22:57.351877+0000","last_clean":"2024-03-06T07:22:57.351877+0000","last_became_active":"2024-03-06T07:22:57.350169+0000","last_became_peered":"2024-03-06T07:22:57.350169+0000","last_unstale":"2024-03-06T07:22:57.351877+0000","last_undegraded":"2024-03-06T07:22:57.351877+0000","last_fullsized":"2024-03-06T07:22:57.351877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:37.755588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.331174+0000","last_change":"2024-03-06T07:22:56.342838+0000","last_active":"2024-03-06T07:22:57.331174+0000","last_peered":"2024-03-06T07:22:57.331174+0000","last_clean":"2024-03-06T07:22:57.331174+0000","last_became_active":"2024-03-06T07:22:56.342549+0000","last_became_peered":"2024-03-06T07:22:56.342549+0000","last_unstale":"2024-03-06T07:22:57.331174+0000","last_undegraded":"2024-03-06T07:22:57.331174+0000","last_fullsized":"2024-03-06T07:22:57.331174+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:54.912718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357246+0000","last_change":"2024-03-06T07:22:57.353296+0000","last_active":"2024-03-06T07:22:57.357246+0000","last_peered":"2024-03-06T07:22:57.357246+0000","last_clean":"2024-03-06T07:22:57.357246+0000","last_became_active":"2024-03-06T07:22:57.353050+0000","last_became_peered":"2024-03-06T07:22:57.353050+0000","last_unstale":"2024-03-06T07:22:57.357246+0000","last_undegraded":"2024-03-06T07:22:57.357246+0000","last_fullsized":"2024-03-06T07:22:57.357246+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:05.481638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.345337+0000","last_change":"2024-03-06T07:22:56.338593+0000","last_active":"2024-03-06T07:22:57.345337+0000","last_peered":"2024-03-06T07:22:57.345337+0000","last_clean":"2024-03-06T07:22:57.345337+0000","last_became_active":"2024-03-06T07:22:56.338301+0000","last_became_peered":"2024-03-06T07:22:56.338301+0000","last_unstale":"2024-03-06T07:22:57.345337+0000","last_undegraded":"2024-03-06T07:22:57.345337+0000","last_fullsized":"2024-03-06T07:22:57.345337+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:41.330236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350344+0000","last_change":"2024-03-06T07:22:57.342398+0000","last_active":"2024-03-06T07:22:57.350344+0000","last_peered":"2024-03-06T07:22:57.350344+0000","last_clean":"2024-03-06T07:22:57.350344+0000","last_became_active":"2024-03-06T07:22:57.342135+0000","last_became_peered":"2024-03-06T07:22:57.342135+0000","last_unstale":"2024-03-06T07:22:57.350344+0000","last_undegraded":"2024-03-06T07:22:57.350344+0000","last_fullsized":"2024-03-06T07:22:57.350344+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:35.401911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.331964+0000","last_change":"2024-03-06T07:22:56.347830+0000","last_active":"2024-03-06T07:22:57.331964+0000","last_peered":"2024-03-06T07:22:57.331964+0000","last_clean":"2024-03-06T07:22:57.331964+0000","last_became_active":"2024-03-06T07:22:56.346511+0000","last_became_peered":"2024-03-06T07:22:56.346511+0000","last_unstale":"2024-03-06T07:22:57.331964+0000","last_undegraded":"2024-03-06T07:22:57.331964+0000","last_fullsized":"2024-03-06T07:22:57.331964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:58.165095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351288+0000","last_change":"2024-03-06T07:22:57.346443+0000","last_active":"2024-03-06T07:22:57.351288+0000","last_peered":"2024-03-06T07:22:57.351288+0000","last_clean":"2024-03-06T07:22:57.351288+0000","last_became_active":"2024-03-06T07:22:57.346154+0000","last_became_peered":"2024-03-06T07:22:57.346154+0000","last_unstale":"2024-03-06T07:22:57.351288+0000","last_undegraded":"2024-03-06T07:22:57.351288+0000","last_fullsized":"2024-03-06T07:22:57.351288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:53.809337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.335081+0000","last_change":"2024-03-06T07:22:56.343614+0000","last_active":"2024-03-06T07:22:57.335081+0000","last_peered":"2024-03-06T07:22:57.335081+0000","last_clean":"2024-03-06T07:22:57.335081+0000","last_became_active":"2024-03-06T07:22:56.343371+0000","last_became_peered":"2024-03-06T07:22:56.343371+0000","last_unstale":"2024-03-06T07:22:57.335081+0000","last_undegraded":"2024-03-06T07:22:57.335081+0000","last_fullsized":"2024-03-06T07:22:57.335081+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:35.657338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.355709+0000","last_change":"2024-03-06T07:22:57.348645+0000","last_active":"2024-03-06T07:22:57.355709+0000","last_peered":"2024-03-06T07:22:57.355709+0000","last_clean":"2024-03-06T07:22:57.355709+0000","last_became_active":"2024-03-06T07:22:57.348393+0000","last_became_peered":"2024-03-06T07:22:57.348393+0000","last_unstale":"2024-03-06T07:22:57.355709+0000","last_undegraded":"2024-03-06T07:22:57.355709+0000","last_fullsized":"2024-03-06T07:22:57.355709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:09.156576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.359078+0000","last_change":"2024-03-06T07:22:56.346371+0000","last_active":"2024-03-06T07:22:57.359078+0000","last_peered":"2024-03-06T07:22:57.359078+0000","last_clean":"2024-03-06T07:22:57.359078+0000","last_became_active":"2024-03-06T07:22:56.346043+0000","last_became_peered":"2024-03-06T07:22:56.346043+0000","last_unstale":"2024-03-06T07:22:57.359078+0000","last_undegraded":"2024-03-06T07:22:57.359078+0000","last_fullsized":"2024-03-06T07:22:57.359078+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.743737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356095+0000","last_change":"2024-03-06T07:22:57.353131+0000","last_active":"2024-03-06T07:22:57.356095+0000","last_peered":"2024-03-06T07:22:57.356095+0000","last_clean":"2024-03-06T07:22:57.356095+0000","last_became_active":"2024-03-06T07:22:57.352879+0000","last_became_peered":"2024-03-06T07:22:57.352879+0000","last_unstale":"2024-03-06T07:22:57.356095+0000","last_undegraded":"2024-03-06T07:22:57.356095+0000","last_fullsized":"2024-03-06T07:22:57.356095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:38.047531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.365719+0000","last_change":"2024-03-06T07:22:56.352000+0000","last_active":"2024-03-06T07:22:57.365719+0000","last_peered":"2024-03-06T07:22:57.365719+0000","last_clean":"2024-03-06T07:22:57.365719+0000","last_became_active":"2024-03-06T07:22:56.351689+0000","last_became_peered":"2024-03-06T07:22:56.351689+0000","last_unstale":"2024-03-06T07:22:57.365719+0000","last_undegraded":"2024-03-06T07:22:57.365719+0000","last_fullsized":"2024-03-06T07:22:57.365719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:34.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357314+0000","last_change":"2024-03-06T07:22:57.353013+0000","last_active":"2024-03-06T07:22:57.357314+0000","last_peered":"2024-03-06T07:22:57.357314+0000","last_clean":"2024-03-06T07:22:57.357314+0000","last_became_active":"2024-03-06T07:22:57.352644+0000","last_became_peered":"2024-03-06T07:22:57.352644+0000","last_unstale":"2024-03-06T07:22:57.357314+0000","last_undegraded":"2024-03-06T07:22:57.357314+0000","last_fullsized":"2024-03-06T07:22:57.357314+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.749433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357815+0000","last_change":"2024-03-06T07:22:56.340575+0000","last_active":"2024-03-06T07:22:57.357815+0000","last_peered":"2024-03-06T07:22:57.357815+0000","last_clean":"2024-03-06T07:22:57.357815+0000","last_became_active":"2024-03-06T07:22:56.340206+0000","last_became_peered":"2024-03-06T07:22:56.340206+0000","last_unstale":"2024-03-06T07:22:57.357815+0000","last_undegraded":"2024-03-06T07:22:57.357815+0000","last_fullsized":"2024-03-06T07:22:57.357815+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:50.551483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351417+0000","last_change":"2024-03-06T07:22:57.340938+0000","last_active":"2024-03-06T07:22:57.351417+0000","last_peered":"2024-03-06T07:22:57.351417+0000","last_clean":"2024-03-06T07:22:57.351417+0000","last_became_active":"2024-03-06T07:22:57.340661+0000","last_became_peered":"2024-03-06T07:22:57.340661+0000","last_unstale":"2024-03-06T07:22:57.351417+0000","last_undegraded":"2024-03-06T07:22:57.351417+0000","last_fullsized":"2024-03-06T07:22:57.351417+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:30.027283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.339219+0000","last_change":"2024-03-06T07:22:56.347343+0000","last_active":"2024-03-06T07:22:57.339219+0000","last_peered":"2024-03-06T07:22:57.339219+0000","last_clean":"2024-03-06T07:22:57.339219+0000","last_became_active":"2024-03-06T07:22:56.347086+0000","last_became_peered":"2024-03-06T07:22:56.347086+0000","last_unstale":"2024-03-06T07:22:57.339219+0000","last_undegraded":"2024-03-06T07:22:57.339219+0000","last_fullsized":"2024-03-06T07:22:57.339219+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:54.726310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356889+0000","last_change":"2024-03-06T07:22:57.354474+0000","last_active":"2024-03-06T07:22:57.356889+0000","last_peered":"2024-03-06T07:22:57.356889+0000","last_clean":"2024-03-06T07:22:57.356889+0000","last_became_active":"2024-03-06T07:22:57.354229+0000","last_became_peered":"2024-03-06T07:22:57.354229+0000","last_unstale":"2024-03-06T07:22:57.356889+0000","last_undegraded":"2024-03-06T07:22:57.356889+0000","last_fullsized":"2024-03-06T07:22:57.356889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:59.167118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.331406+0000","last_change":"2024-03-06T07:22:56.345499+0000","last_active":"2024-03-06T07:22:57.331406+0000","last_peered":"2024-03-06T07:22:57.331406+0000","last_clean":"2024-03-06T07:22:57.331406+0000","last_became_active":"2024-03-06T07:22:56.345241+0000","last_became_peered":"2024-03-06T07:22:56.345241+0000","last_unstale":"2024-03-06T07:22:57.331406+0000","last_undegraded":"2024-03-06T07:22:57.331406+0000","last_fullsized":"2024-03-06T07:22:57.331406+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:04.479616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.344669+0000","last_change":"2024-03-06T07:22:57.342753+0000","last_active":"2024-03-06T07:22:57.344669+0000","last_peered":"2024-03-06T07:22:57.344669+0000","last_clean":"2024-03-06T07:22:57.344669+0000","last_became_active":"2024-03-06T07:22:57.342484+0000","last_became_peered":"2024-03-06T07:22:57.342484+0000","last_unstale":"2024-03-06T07:22:57.344669+0000","last_undegraded":"2024-03-06T07:22:57.344669+0000","last_fullsized":"2024-03-06T07:22:57.344669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:36.659361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.358189+0000","last_change":"2024-03-06T07:22:56.343563+0000","last_active":"2024-03-06T07:22:57.358189+0000","last_peered":"2024-03-06T07:22:57.358189+0000","last_clean":"2024-03-06T07:22:57.358189+0000","last_became_active":"2024-03-06T07:22:56.343292+0000","last_became_peered":"2024-03-06T07:22:56.343292+0000","last_unstale":"2024-03-06T07:22:57.358189+0000","last_undegraded":"2024-03-06T07:22:57.358189+0000","last_fullsized":"2024-03-06T07:22:57.358189+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:29.747411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356531+0000","last_change":"2024-03-06T07:22:57.353792+0000","last_active":"2024-03-06T07:22:57.356531+0000","last_peered":"2024-03-06T07:22:57.356531+0000","last_clean":"2024-03-06T07:22:57.356531+0000","last_became_active":"2024-03-06T07:22:57.353526+0000","last_became_peered":"2024-03-06T07:22:57.353526+0000","last_unstale":"2024-03-06T07:22:57.356531+0000","last_undegraded":"2024-03-06T07:22:57.356531+0000","last_fullsized":"2024-03-06T07:22:57.356531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:19.589051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.359902+0000","last_change":"2024-03-06T07:22:56.349086+0000","last_active":"2024-03-06T07:22:57.359902+0000","last_peered":"2024-03-06T07:22:57.359902+0000","last_clean":"2024-03-06T07:22:57.359902+0000","last_became_active":"2024-03-06T07:22:56.348812+0000","last_became_peered":"2024-03-06T07:22:56.348812+0000","last_unstale":"2024-03-06T07:22:57.359902+0000","last_undegraded":"2024-03-06T07:22:57.359902+0000","last_fullsized":"2024-03-06T07:22:57.359902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:26.205859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.350526+0000","last_change":"2024-03-06T07:22:57.343810+0000","last_active":"2024-03-06T07:22:57.350526+0000","last_peered":"2024-03-06T07:22:57.350526+0000","last_clean":"2024-03-06T07:22:57.350526+0000","last_became_active":"2024-03-06T07:22:57.343550+0000","last_became_peered":"2024-03-06T07:22:57.343550+0000","last_unstale":"2024-03-06T07:22:57.350526+0000","last_undegraded":"2024-03-06T07:22:57.350526+0000","last_fullsized":"2024-03-06T07:22:57.350526+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:09.314442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357061+0000","last_change":"2024-03-06T07:22:56.339376+0000","last_active":"2024-03-06T07:22:57.357061+0000","last_peered":"2024-03-06T07:22:57.357061+0000","last_clean":"2024-03-06T07:22:57.357061+0000","last_became_active":"2024-03-06T07:22:56.339116+0000","last_became_peered":"2024-03-06T07:22:56.339116+0000","last_unstale":"2024-03-06T07:22:57.357061+0000","last_undegraded":"2024-03-06T07:22:57.357061+0000","last_fullsized":"2024-03-06T07:22:57.357061+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:09.071529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.357079+0000","last_change":"2024-03-06T07:22:57.349952+0000","last_active":"2024-03-06T07:22:57.357079+0000","last_peered":"2024-03-06T07:22:57.357079+0000","last_clean":"2024-03-06T07:22:57.357079+0000","last_became_active":"2024-03-06T07:22:57.348790+0000","last_became_peered":"2024-03-06T07:22:57.348790+0000","last_unstale":"2024-03-06T07:22:57.357079+0000","last_undegraded":"2024-03-06T07:22:57.357079+0000","last_fullsized":"2024-03-06T07:22:57.357079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:42.756008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.362851+0000","last_change":"2024-03-06T07:22:56.343907+0000","last_active":"2024-03-06T07:22:57.362851+0000","last_peered":"2024-03-06T07:22:57.362851+0000","last_clean":"2024-03-06T07:22:57.362851+0000","last_became_active":"2024-03-06T07:22:56.343579+0000","last_became_peered":"2024-03-06T07:22:56.343579+0000","last_unstale":"2024-03-06T07:22:57.362851+0000","last_undegraded":"2024-03-06T07:22:57.362851+0000","last_fullsized":"2024-03-06T07:22:57.362851+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.668165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.354332+0000","last_change":"2024-03-06T07:22:57.349547+0000","last_active":"2024-03-06T07:22:57.354332+0000","last_peered":"2024-03-06T07:22:57.354332+0000","last_clean":"2024-03-06T07:22:57.354332+0000","last_became_active":"2024-03-06T07:22:57.349301+0000","last_became_peered":"2024-03-06T07:22:57.349301+0000","last_unstale":"2024-03-06T07:22:57.354332+0000","last_undegraded":"2024-03-06T07:22:57.354332+0000","last_fullsized":"2024-03-06T07:22:57.354332+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:55.283065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.339549+0000","last_change":"2024-03-06T07:22:56.344417+0000","last_active":"2024-03-06T07:22:57.339549+0000","last_peered":"2024-03-06T07:22:57.339549+0000","last_clean":"2024-03-06T07:22:57.339549+0000","last_became_active":"2024-03-06T07:22:56.344007+0000","last_became_peered":"2024-03-06T07:22:56.344007+0000","last_unstale":"2024-03-06T07:22:57.339549+0000","last_undegraded":"2024-03-06T07:22:57.339549+0000","last_fullsized":"2024-03-06T07:22:57.339549+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:54.281043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351694+0000","last_change":"2024-03-06T07:22:57.347532+0000","last_active":"2024-03-06T07:22:57.351694+0000","last_peered":"2024-03-06T07:22:57.351694+0000","last_clean":"2024-03-06T07:22:57.351694+0000","last_became_active":"2024-03-06T07:22:57.347291+0000","last_became_peered":"2024-03-06T07:22:57.347291+0000","last_unstale":"2024-03-06T07:22:57.351694+0000","last_undegraded":"2024-03-06T07:22:57.351694+0000","last_fullsized":"2024-03-06T07:22:57.351694+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:30.099116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.358051+0000","last_change":"2024-03-06T07:22:56.342208+0000","last_active":"2024-03-06T07:22:57.358051+0000","last_peered":"2024-03-06T07:22:57.358051+0000","last_clean":"2024-03-06T07:22:57.358051+0000","last_became_active":"2024-03-06T07:22:56.341821+0000","last_became_peered":"2024-03-06T07:22:56.341821+0000","last_unstale":"2024-03-06T07:22:57.358051+0000","last_undegraded":"2024-03-06T07:22:57.358051+0000","last_fullsized":"2024-03-06T07:22:57.358051+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:26.205859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349283+0000","last_change":"2024-03-06T07:22:57.343122+0000","last_active":"2024-03-06T07:22:57.349283+0000","last_peered":"2024-03-06T07:22:57.349283+0000","last_clean":"2024-03-06T07:22:57.349283+0000","last_became_active":"2024-03-06T07:22:57.342833+0000","last_became_peered":"2024-03-06T07:22:57.342833+0000","last_unstale":"2024-03-06T07:22:57.349283+0000","last_undegraded":"2024-03-06T07:22:57.349283+0000","last_fullsized":"2024-03-06T07:22:57.349283+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:12.151743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.334957+0000","last_change":"2024-03-06T07:22:56.341520+0000","last_active":"2024-03-06T07:22:57.334957+0000","last_peered":"2024-03-06T07:22:57.334957+0000","last_clean":"2024-03-06T07:22:57.334957+0000","last_became_active":"2024-03-06T07:22:56.341260+0000","last_became_peered":"2024-03-06T07:22:56.341260+0000","last_unstale":"2024-03-06T07:22:57.334957+0000","last_undegraded":"2024-03-06T07:22:57.334957+0000","last_fullsized":"2024-03-06T07:22:57.334957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:36.753566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.349335+0000","last_change":"2024-03-06T07:22:57.343278+0000","last_active":"2024-03-06T07:22:57.349335+0000","last_peered":"2024-03-06T07:22:57.349335+0000","last_clean":"2024-03-06T07:22:57.349335+0000","last_became_active":"2024-03-06T07:22:57.342188+0000","last_became_peered":"2024-03-06T07:22:57.342188+0000","last_unstale":"2024-03-06T07:22:57.349335+0000","last_undegraded":"2024-03-06T07:22:57.349335+0000","last_fullsized":"2024-03-06T07:22:57.349335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:33.205423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.338394+0000","last_change":"2024-03-06T07:22:56.342902+0000","last_active":"2024-03-06T07:22:57.338394+0000","last_peered":"2024-03-06T07:22:57.338394+0000","last_clean":"2024-03-06T07:22:57.338394+0000","last_became_active":"2024-03-06T07:22:56.342598+0000","last_became_peered":"2024-03-06T07:22:56.342598+0000","last_unstale":"2024-03-06T07:22:57.338394+0000","last_undegraded":"2024-03-06T07:22:57.338394+0000","last_fullsized":"2024-03-06T07:22:57.338394+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:05.938025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356638+0000","last_change":"2024-03-06T07:22:57.352208+0000","last_active":"2024-03-06T07:22:57.356638+0000","last_peered":"2024-03-06T07:22:57.356638+0000","last_clean":"2024-03-06T07:22:57.356638+0000","last_became_active":"2024-03-06T07:22:57.351950+0000","last_became_peered":"2024-03-06T07:22:57.351950+0000","last_unstale":"2024-03-06T07:22:57.356638+0000","last_undegraded":"2024-03-06T07:22:57.356638+0000","last_fullsized":"2024-03-06T07:22:57.356638+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:55.914740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.333407+0000","last_change":"2024-03-06T07:22:56.345321+0000","last_active":"2024-03-06T07:22:57.333407+0000","last_peered":"2024-03-06T07:22:57.333407+0000","last_clean":"2024-03-06T07:22:57.333407+0000","last_became_active":"2024-03-06T07:22:56.345047+0000","last_became_peered":"2024-03-06T07:22:56.345047+0000","last_unstale":"2024-03-06T07:22:57.333407+0000","last_undegraded":"2024-03-06T07:22:57.333407+0000","last_fullsized":"2024-03-06T07:22:57.333407+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:02.287807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.354301+0000","last_change":"2024-03-06T07:22:57.350188+0000","last_active":"2024-03-06T07:22:57.354301+0000","last_peered":"2024-03-06T07:22:57.354301+0000","last_clean":"2024-03-06T07:22:57.354301+0000","last_became_active":"2024-03-06T07:22:57.349936+0000","last_became_peered":"2024-03-06T07:22:57.349936+0000","last_unstale":"2024-03-06T07:22:57.354301+0000","last_undegraded":"2024-03-06T07:22:57.354301+0000","last_fullsized":"2024-03-06T07:22:57.354301+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:57.791569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360347+0000","last_change":"2024-03-06T07:22:56.345829+0000","last_active":"2024-03-06T07:22:57.360347+0000","last_peered":"2024-03-06T07:22:57.360347+0000","last_clean":"2024-03-06T07:22:57.360347+0000","last_became_active":"2024-03-06T07:22:56.345558+0000","last_became_peered":"2024-03-06T07:22:56.345558+0000","last_unstale":"2024-03-06T07:22:57.360347+0000","last_undegraded":"2024-03-06T07:22:57.360347+0000","last_fullsized":"2024-03-06T07:22:57.360347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:35.657338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.355994+0000","last_change":"2024-03-06T07:22:57.355348+0000","last_active":"2024-03-06T07:22:57.355994+0000","last_peered":"2024-03-06T07:22:57.355994+0000","last_clean":"2024-03-06T07:22:57.355994+0000","last_became_active":"2024-03-06T07:22:57.353228+0000","last_became_peered":"2024-03-06T07:22:57.353228+0000","last_unstale":"2024-03-06T07:22:57.355994+0000","last_undegraded":"2024-03-06T07:22:57.355994+0000","last_fullsized":"2024-03-06T07:22:57.355994+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:30.027283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.335217+0000","last_change":"2024-03-06T07:22:56.342200+0000","last_active":"2024-03-06T07:22:57.335217+0000","last_peered":"2024-03-06T07:22:57.335217+0000","last_clean":"2024-03-06T07:22:57.335217+0000","last_became_active":"2024-03-06T07:22:56.341853+0000","last_became_peered":"2024-03-06T07:22:56.341853+0000","last_unstale":"2024-03-06T07:22:57.335217+0000","last_undegraded":"2024-03-06T07:22:57.335217+0000","last_fullsized":"2024-03-06T07:22:57.335217+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:41.753985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.353821+0000","last_change":"2024-03-06T07:22:57.349672+0000","last_active":"2024-03-06T07:22:57.353821+0000","last_peered":"2024-03-06T07:22:57.353821+0000","last_clean":"2024-03-06T07:22:57.353821+0000","last_became_active":"2024-03-06T07:22:57.349386+0000","last_became_peered":"2024-03-06T07:22:57.349386+0000","last_unstale":"2024-03-06T07:22:57.353821+0000","last_undegraded":"2024-03-06T07:22:57.353821+0000","last_fullsized":"2024-03-06T07:22:57.353821+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:30.749433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.339442+0000","last_change":"2024-03-06T07:22:56.344783+0000","last_active":"2024-03-06T07:22:57.339442+0000","last_peered":"2024-03-06T07:22:57.339442+0000","last_clean":"2024-03-06T07:22:57.339442+0000","last_became_active":"2024-03-06T07:22:56.344532+0000","last_became_peered":"2024-03-06T07:22:56.344532+0000","last_unstale":"2024-03-06T07:22:57.339442+0000","last_undegraded":"2024-03-06T07:22:57.339442+0000","last_fullsized":"2024-03-06T07:22:57.339442+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:18.587029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.342892+0000","last_change":"2024-03-06T07:22:57.339944+0000","last_active":"2024-03-06T07:22:57.342892+0000","last_peered":"2024-03-06T07:22:57.342892+0000","last_clean":"2024-03-06T07:22:57.342892+0000","last_became_active":"2024-03-06T07:22:57.339644+0000","last_became_peered":"2024-03-06T07:22:57.339644+0000","last_unstale":"2024-03-06T07:22:57.342892+0000","last_undegraded":"2024-03-06T07:22:57.342892+0000","last_fullsized":"2024-03-06T07:22:57.342892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:27.207882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.335196+0000","last_change":"2024-03-06T07:22:56.336788+0000","last_active":"2024-03-06T07:22:57.335196+0000","last_peered":"2024-03-06T07:22:57.335196+0000","last_clean":"2024-03-06T07:22:57.335196+0000","last_became_active":"2024-03-06T07:22:56.336548+0000","last_became_peered":"2024-03-06T07:22:56.336548+0000","last_unstale":"2024-03-06T07:22:57.335196+0000","last_undegraded":"2024-03-06T07:22:57.335196+0000","last_fullsized":"2024-03-06T07:22:57.335196+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:29.184132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.348162+0000","last_change":"2024-03-06T07:22:57.340934+0000","last_active":"2024-03-06T07:22:57.348162+0000","last_peered":"2024-03-06T07:22:57.348162+0000","last_clean":"2024-03-06T07:22:57.348162+0000","last_became_active":"2024-03-06T07:22:57.340657+0000","last_became_peered":"2024-03-06T07:22:57.340657+0000","last_unstale":"2024-03-06T07:22:57.348162+0000","last_undegraded":"2024-03-06T07:22:57.348162+0000","last_fullsized":"2024-03-06T07:22:57.348162+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:55.745760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.339397+0000","last_change":"2024-03-06T07:22:56.346714+0000","last_active":"2024-03-06T07:22:57.339397+0000","last_peered":"2024-03-06T07:22:57.339397+0000","last_clean":"2024-03-06T07:22:57.339397+0000","last_became_active":"2024-03-06T07:22:56.346466+0000","last_became_peered":"2024-03-06T07:22:56.346466+0000","last_unstale":"2024-03-06T07:22:57.339397+0000","last_undegraded":"2024-03-06T07:22:57.339397+0000","last_fullsized":"2024-03-06T07:22:57.339397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:36.753566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.351497+0000","last_change":"2024-03-06T07:22:57.347224+0000","last_active":"2024-03-06T07:22:57.351497+0000","last_peered":"2024-03-06T07:22:57.351497+0000","last_clean":"2024-03-06T07:22:57.351497+0000","last_became_active":"2024-03-06T07:22:57.346958+0000","last_became_peered":"2024-03-06T07:22:57.346958+0000","last_unstale":"2024-03-06T07:22:57.351497+0000","last_undegraded":"2024-03-06T07:22:57.351497+0000","last_fullsized":"2024-03-06T07:22:57.351497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:53.876326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.338543+0000","last_change":"2024-03-06T07:22:56.337441+0000","last_active":"2024-03-06T07:22:57.338543+0000","last_peered":"2024-03-06T07:22:57.338543+0000","last_clean":"2024-03-06T07:22:57.338543+0000","last_became_active":"2024-03-06T07:22:56.337176+0000","last_became_peered":"2024-03-06T07:22:56.337176+0000","last_unstale":"2024-03-06T07:22:57.338543+0000","last_undegraded":"2024-03-06T07:22:57.338543+0000","last_fullsized":"2024-03-06T07:22:57.338543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:29.097094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.360633+0000","last_change":"2024-03-06T07:22:57.359815+0000","last_active":"2024-03-06T07:22:57.360633+0000","last_peered":"2024-03-06T07:22:57.360633+0000","last_clean":"2024-03-06T07:22:57.360633+0000","last_became_active":"2024-03-06T07:22:57.359512+0000","last_became_peered":"2024-03-06T07:22:57.359512+0000","last_unstale":"2024-03-06T07:22:57.360633+0000","last_undegraded":"2024-03-06T07:22:57.360633+0000","last_fullsized":"2024-03-06T07:22:57.360633+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:27.207882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.331151+0000","last_change":"2024-03-06T07:22:56.346553+0000","last_active":"2024-03-06T07:22:57.331151+0000","last_peered":"2024-03-06T07:22:57.331151+0000","last_clean":"2024-03-06T07:22:57.331151+0000","last_became_active":"2024-03-06T07:22:56.346283+0000","last_became_peered":"2024-03-06T07:22:56.346283+0000","last_unstale":"2024-03-06T07:22:57.331151+0000","last_undegraded":"2024-03-06T07:22:57.331151+0000","last_fullsized":"2024-03-06T07:22:57.331151+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:54.743737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.356856+0000","last_change":"2024-03-06T07:22:57.355083+0000","last_active":"2024-03-06T07:22:57.356856+0000","last_peered":"2024-03-06T07:22:57.356856+0000","last_clean":"2024-03-06T07:22:57.356856+0000","last_became_active":"2024-03-06T07:22:57.353454+0000","last_became_peered":"2024-03-06T07:22:57.353454+0000","last_unstale":"2024-03-06T07:22:57.356856+0000","last_undegraded":"2024-03-06T07:22:57.356856+0000","last_fullsized":"2024-03-06T07:22:57.356856+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:56.322928+0000","last_clean_scrub_stamp":"2024-03-06T07:22:56.322928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:09.314442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.333409+0000","last_change":"2024-03-06T07:22:56.337866+0000","last_active":"2024-03-06T07:22:57.333409+0000","last_peered":"2024-03-06T07:22:57.333409+0000","last_clean":"2024-03-06T07:22:57.333409+0000","last_became_active":"2024-03-06T07:22:56.337555+0000","last_became_peered":"2024-03-06T07:22:56.337555+0000","last_unstale":"2024-03-06T07:22:57.333409+0000","last_undegraded":"2024-03-06T07:22:57.333409+0000","last_fullsized":"2024-03-06T07:22:57.333409+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:55.320906+0000","last_clean_scrub_stamp":"2024-03-06T07:22:55.320906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:35.461029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":189,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:57.347173+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:22:57.347173+0000","last_peered":"2024-03-06T07:22:57.347173+0000","last_clean":"2024-03-06T07:22:57.347173+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:22:57.347173+0000","last_undegraded":"2024-03-06T07:22:57.347173+0000","last_fullsized":"2024-03-06T07:22:57.347173+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":722438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":132070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":722438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:00.680 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:00.680 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:00.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9ace1640 1 Processor -- start 2024-03-06T07:23:00.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9ace1640 1 -- start start 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9ace1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1f940b0db0 0x7f1f94066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9ace1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f940b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9ace1640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9406b050 con 0x7f1f9415f4f0 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9ace1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9406b2c0 con 0x7f1f940b0db0 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9ace1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940b0890 con 0x7f1f940ce750 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f98a7d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1f9415f4f0 0x7f1f940ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52170/0 (socket says 172.21.15.77:52170) 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f98a7d640 1 -- 172.21.15.77:0/2402589080 learned_addr learned my addr 172.21.15.77:0/2402589080 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9927e640 1 --2- 172.21.15.77:0/2402589080 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f940b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f93fff640 1 --2- 172.21.15.77:0/2402589080 >> v2:172.21.15.158:3300/0 conn(0x7f1f940b0db0 0x7f1f94066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f937fe640 1 -- 172.21.15.77:0/2402589080 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628911122 0 0) 0x7f1f9406b050 con 0x7f1f9415f4f0 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f937fe640 1 -- 172.21.15.77:0/2402589080 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f84003520 con 0x7f1f9415f4f0 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9927e640 1 -- 172.21.15.77:0/2402589080 >> v2:172.21.15.158:3300/0 conn(0x7f1f940b0db0 msgr2=0x7f1f94066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9927e640 1 --2- 172.21.15.77:0/2402589080 >> v2:172.21.15.158:3300/0 conn(0x7f1f940b0db0 0x7f1f94066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9927e640 1 -- 172.21.15.77:0/2402589080 >> v1:172.21.15.77:6789/0 conn(0x7f1f9415f4f0 legacy=0x7f1f940ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.821+0000 7f1f9927e640 1 -- 172.21.15.77:0/2402589080 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f940672e0 con 0x7f1f940ce750 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.822+0000 7f1f9927e640 1 --2- 172.21.15.77:0/2402589080 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f940b4ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009d90 tx=0x7f1f8c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f522c101a4d900d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.822+0000 7f1f937fe640 1 -- 172.21.15.77:0/2402589080 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c039070 con 0x7f1f940ce750 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.822+0000 7f1f937fe640 1 -- 172.21.15.77:0/2402589080 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c02cc50 con 0x7f1f940ce750 2024-03-06T07:23:00.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.822+0000 7f1f937fe640 1 -- 172.21.15.77:0/2402589080 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c034440 con 0x7f1f940ce750 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.823+0000 7f1f9ace1640 1 -- 172.21.15.77:0/2402589080 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 msgr2=0x7f1f940b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.823+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/2402589080 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f940b4ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009d90 tx=0x7f1f8c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.823+0000 7f1f9ace1640 1 -- 172.21.15.77:0/2402589080 shutdown_connections 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.823+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/2402589080 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f940b4ed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.823+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/2402589080 >> v2:172.21.15.158:3300/0 conn(0x7f1f940b0db0 0x7f1f94066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.823+0000 7f1f9ace1640 1 -- 172.21.15.77:0/2402589080 >> 172.21.15.77:0/2402589080 conn(0x7f1f940b1d70 msgr2=0x7f1f940b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.823+0000 7f1f9ace1640 1 -- 172.21.15.77:0/2402589080 shutdown_connections 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.823+0000 7f1f9ace1640 1 -- 172.21.15.77:0/2402589080 wait complete. 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.824+0000 7f1f9ace1640 1 Processor -- start 2024-03-06T07:23:00.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.824+0000 7f1f9ace1640 1 -- start start 2024-03-06T07:23:00.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9ace1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f940b0db0 0x7f1f940c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9ace1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f94153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9ace1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415f4f0 0x7f1f94099630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9ace1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9415f8d0 con 0x7f1f940b0db0 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9ace1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940cebe0 con 0x7f1f9415f4f0 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9ace1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940c4fe0 con 0x7f1f940ce750 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f940b0db0 0x7f1f940c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f940b0db0 0x7f1f940c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52916/0 (socket says 172.21.15.77:52916) 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 -- 172.21.15.77:0/3449784752 learned_addr learned my addr 172.21.15.77:0/3449784752 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f93fff640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f94153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9927e640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415f4f0 0x7f1f94099630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 -- 172.21.15.77:0/3449784752 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 msgr2=0x7f1f94153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f94153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 -- 172.21.15.77:0/3449784752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415f4f0 msgr2=0x7f1f94099630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415f4f0 0x7f1f94099630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 -- 172.21.15.77:0/3449784752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f941f4db0 con 0x7f1f940b0db0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f98a7d640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f940b0db0 0x7f1f940c62d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800c430 tx=0x7f1f8800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f917fa640 1 -- 172.21.15.77:0/3449784752 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f88017070 con 0x7f1f940b0db0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f917fa640 1 -- 172.21.15.77:0/3449784752 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f88005ce0 con 0x7f1f940b0db0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f917fa640 1 -- 172.21.15.77:0/3449784752 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f880072f0 con 0x7f1f940b0db0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f94153ec0 con 0x7f1f940b0db0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.825+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f940c5510 con 0x7f1f940b0db0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.826+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9409e3d0 con 0x7f1f940b0db0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.827+0000 7f1f917fa640 1 -- 172.21.15.77:0/3449784752 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f8801a070 con 0x7f1f940b0db0 2024-03-06T07:23:00.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.827+0000 7f1f917fa640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f7806b4d0 0x7f1f7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.827+0000 7f1f917fa640 1 -- 172.21.15.77:0/3449784752 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f88089570 con 0x7f1f940b0db0 2024-03-06T07:23:00.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.827+0000 7f1f93fff640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f7806b4d0 0x7f1f7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.828+0000 7f1f93fff640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f7806b4d0 0x7f1f7806d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1f9415d9f0 tx=0x7f1f80009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:00.831+0000 7f1f917fa640 1 -- 172.21.15.77:0/3449784752 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f88057e30 con 0x7f1f940b0db0 2024-03-06T07:23:01.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.068+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f1f9406cd40 con 0x7f1f940b0db0 2024-03-06T07:23:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.069+0000 7f1f917fa640 1 -- 172.21.15.77:0/3449784752 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f1f8805bed0 con 0x7f1f940b0db0 2024-03-06T07:23:01.071 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:23:01.071 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:23:01.071 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:23:01.072 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:23:01.073 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f7806b4d0 msgr2=0x7f1f7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f7806b4d0 0x7f1f7806d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1f9415d9f0 tx=0x7f1f80009190 comp rx=0 tx=0).stop 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f940b0db0 msgr2=0x7f1f940c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f940b0db0 0x7f1f940c62d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800c430 tx=0x7f1f8800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 shutdown_connections 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1f9415f4f0 0x7f1f94099630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1f940ce750 0x7f1f94153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1f7806b4d0 0x7f1f7806d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 --2- 172.21.15.77:0/3449784752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f940b0db0 0x7f1f940c62d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 >> 172.21.15.77:0/3449784752 conn(0x7f1f940b1d70 msgr2=0x7f1f940cf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 shutdown_connections 2024-03-06T07:23:01.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.072+0000 7f1f9ace1640 1 -- 172.21.15.77:0/3449784752 wait complete. 2024-03-06T07:23:01.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.182+0000 7f8ea0023640 1 Processor -- start 2024-03-06T07:23:01.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.182+0000 7f8ea0023640 1 -- start start 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8ea0023640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8e980b3060 0x7f8e9815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8ea0023640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8e9815bf30 0x7f8e9815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8ea0023640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9815f2d0 con 0x7f8e9813dc10 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8ea0023640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e980cf3f0 con 0x7f8e9815bf30 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8ea0023640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9815b210 con 0x7f8e980b3060 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8e9e5c0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8e9815bf30 0x7f8e9815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8e9e5c0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8e9815bf30 0x7f8e9815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52932/0 (socket says 172.21.15.77:52932) 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8e9e5c0640 1 -- 172.21.15.77:0/1453376555 learned_addr learned my addr 172.21.15.77:0/1453376555 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.183+0000 7f8e9d5be640 1 --2- 172.21.15.77:0/1453376555 >> v2:172.21.15.158:3300/0 conn(0x7f8e980b3060 0x7f8e9815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.184+0000 7f8e9e5c0640 1 -- 172.21.15.77:0/1453376555 >> v2:172.21.15.158:3300/0 conn(0x7f8e980b3060 msgr2=0x7f8e9815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.184+0000 7f8e9e5c0640 1 --2- 172.21.15.77:0/1453376555 >> v2:172.21.15.158:3300/0 conn(0x7f8e980b3060 0x7f8e9815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.184+0000 7f8e9e5c0640 1 -- 172.21.15.77:0/1453376555 >> v1:172.21.15.158:6789/0 conn(0x7f8e9813dc10 legacy=0x7f8e980b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.184+0000 7f8e9e5c0640 1 -- 172.21.15.77:0/1453376555 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9815f580 con 0x7f8e9815bf30 2024-03-06T07:23:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.184+0000 7f8e9e5c0640 1 --2- 172.21.15.77:0/1453376555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e9815bf30 0x7f8e9815ed90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8e98068700 tx=0x7f8e9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590addc2cca0454d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8e9cdbd640 1 -- 172.21.15.77:0/1453376555 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90035020 con 0x7f8e9815bf30 2024-03-06T07:23:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8e9cdbd640 1 -- 172.21.15.77:0/1453376555 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e90031870 con 0x7f8e9815bf30 2024-03-06T07:23:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8e9cdbd640 1 -- 172.21.15.77:0/1453376555 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90031c40 con 0x7f8e9815bf30 2024-03-06T07:23:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8ea0023640 1 -- 172.21.15.77:0/1453376555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e9815bf30 msgr2=0x7f8e9815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8ea0023640 1 --2- 172.21.15.77:0/1453376555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e9815bf30 0x7f8e9815ed90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8e98068700 tx=0x7f8e9002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8ea0023640 1 -- 172.21.15.77:0/1453376555 shutdown_connections 2024-03-06T07:23:01.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8ea0023640 1 --2- 172.21.15.77:0/1453376555 >> v2:172.21.15.158:3300/0 conn(0x7f8e980b3060 0x7f8e9815e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8ea0023640 1 --2- 172.21.15.77:0/1453376555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e9815bf30 0x7f8e9815ed90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8ea0023640 1 -- 172.21.15.77:0/1453376555 >> 172.21.15.77:0/1453376555 conn(0x7f8e980cf8a0 msgr2=0x7f8e980cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8ea0023640 1 -- 172.21.15.77:0/1453376555 shutdown_connections 2024-03-06T07:23:01.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.185+0000 7f8ea0023640 1 -- 172.21.15.77:0/1453376555 wait complete. 2024-03-06T07:23:01.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.186+0000 7f8ea0023640 1 Processor -- start 2024-03-06T07:23:01.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.186+0000 7f8ea0023640 1 -- start start 2024-03-06T07:23:01.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8ea0023640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e980b3060 0x7f8e9813cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8ea0023640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e9813dc10 0x7f8e98159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8ea0023640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9815bf30 0x7f8e9815a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8ea0023640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9813dff0 con 0x7f8e980b3060 2024-03-06T07:23:01.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8ea0023640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9815b730 con 0x7f8e9815bf30 2024-03-06T07:23:01.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8ea0023640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980cd7a0 con 0x7f8e9813dc10 2024-03-06T07:23:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9ddbf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e980b3060 0x7f8e9813cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9ddbf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e980b3060 0x7f8e9813cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52946/0 (socket says 172.21.15.77:52946) 2024-03-06T07:23:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9ddbf640 1 -- 172.21.15.77:0/3865092377 learned_addr learned my addr 172.21.15.77:0/3865092377 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9e5c0640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9815bf30 0x7f8e9815a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9d5be640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e9813dc10 0x7f8e98159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9ddbf640 1 -- 172.21.15.77:0/3865092377 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e9813dc10 msgr2=0x7f8e98159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9ddbf640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e9813dc10 0x7f8e98159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9ddbf640 1 -- 172.21.15.77:0/3865092377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9815bf30 msgr2=0x7f8e9815a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.187+0000 7f8e9ddbf640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9815bf30 0x7f8e9815a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.188+0000 7f8e9ddbf640 1 -- 172.21.15.77:0/3865092377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e98167850 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.188+0000 7f8e9ddbf640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e980b3060 0x7f8e9813cfc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400c7c0 tx=0x7f8e9400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.188+0000 7f8e8e7fc640 1 -- 172.21.15.77:0/3865092377 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94016020 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.188+0000 7f8e8e7fc640 1 -- 172.21.15.77:0/3865092377 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e94006d50 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.188+0000 7f8e8e7fc640 1 -- 172.21.15.77:0/3865092377 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94003bb0 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.188+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e98167b30 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.188+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e981680f0 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.189+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9804f330 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.189+0000 7f8e8e7fc640 1 -- 172.21.15.77:0/3865092377 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e9400a850 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.190+0000 7f8e8e7fc640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e7c06b4d0 0x7f8e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.190+0000 7f8e8e7fc640 1 -- 172.21.15.77:0/3865092377 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e94089d30 con 0x7f8e980b3060 2024-03-06T07:23:01.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.190+0000 7f8e9d5be640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e7c06b4d0 0x7f8e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.190+0000 7f8e9d5be640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e7c06b4d0 0x7f8e7c06d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8e98151f10 tx=0x7f8e88009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.194+0000 7f8e8e7fc640 1 -- 172.21.15.77:0/3865092377 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e94020020 con 0x7f8e980b3060 2024-03-06T07:23:01.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.406+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e98151c60 con 0x7f8e980b3060 2024-03-06T07:23:01.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.406+0000 7f8e8e7fc640 1 -- 172.21.15.77:0/3865092377 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f8e940585f0 con 0x7f8e980b3060 2024-03-06T07:23:01.409 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:01.409 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.408+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e7c06b4d0 msgr2=0x7f8e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.408+0000 7f8ea0023640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e7c06b4d0 0x7f8e7c06d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8e98151f10 tx=0x7f8e88009190 comp rx=0 tx=0).stop 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.408+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e980b3060 msgr2=0x7f8e9813cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.408+0000 7f8ea0023640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e980b3060 0x7f8e9813cfc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400c7c0 tx=0x7f8e9400cc80 comp rx=0 tx=0).stop 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.408+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 shutdown_connections 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.408+0000 7f8ea0023640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e9815bf30 0x7f8e9815a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.408+0000 7f8ea0023640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e9813dc10 0x7f8e98159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.409+0000 7f8ea0023640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e7c06b4d0 0x7f8e7c06d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.409+0000 7f8ea0023640 1 --2- 172.21.15.77:0/3865092377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e980b3060 0x7f8e9813cfc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.409+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 >> 172.21.15.77:0/3865092377 conn(0x7f8e980cf8a0 msgr2=0x7f8e980715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.409+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 shutdown_connections 2024-03-06T07:23:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.409+0000 7f8ea0023640 1 -- 172.21.15.77:0/3865092377 wait complete. 2024-03-06T07:23:01.824 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:56.676746+0000', 'modified': '2024-03-06T07:22:58.684340+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5239}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5239': {'gid': 5239, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6834/929839050', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6832', 'nonce': 929839050}, {'type': 'v1', 'addr': '172.21.15.158:6834', 'nonce': 929839050}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5239, 'qdb_cluster': [5239]} 2024-03-06T07:23:01.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:01.825 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:23:01.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.926+0000 7f2ab1634640 1 Processor -- start 2024-03-06T07:23:01.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.926+0000 7f2ab1634640 1 -- start start 2024-03-06T07:23:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2ab1634640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac1444c0 0x7f2aac06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2ab1634640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2aac070fe0 0x7f2aac0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2ab1634640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac06d370 con 0x7f2aac15b2d0 2024-03-06T07:23:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2ab1634640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac1448a0 con 0x7f2aac070fe0 2024-03-06T07:23:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2ab1634640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac06d650 con 0x7f2aac1444c0 2024-03-06T07:23:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2aab7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2aac070fe0 0x7f2aac0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2aab7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2aac070fe0 0x7f2aac0cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52952/0 (socket says 172.21.15.77:52952) 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2aab7fe640 1 -- 172.21.15.77:0/3464433438 learned_addr learned my addr 172.21.15.77:0/3464433438 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/3464433438 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac1444c0 0x7f2aac06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2aab7fe640 1 -- 172.21.15.77:0/3464433438 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac1444c0 msgr2=0x7f2aac06ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/3464433438 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac1444c0 0x7f2aac06ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2aab7fe640 1 -- 172.21.15.77:0/3464433438 >> v1:172.21.15.158:6789/0 conn(0x7f2aac15b2d0 legacy=0x7f2aac0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.927+0000 7f2aab7fe640 1 -- 172.21.15.77:0/3464433438 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac06d930 con 0x7f2aac070fe0 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.928+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/3464433438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac070fe0 0x7f2aac0cda60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009870 tx=0x7f2aa402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd22d0d37cf06d6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.928+0000 7f2aa9ffb640 1 -- 172.21.15.77:0/3464433438 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4034070 con 0x7f2aac070fe0 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.928+0000 7f2aa9ffb640 1 -- 172.21.15.77:0/3464433438 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa4030840 con 0x7f2aac070fe0 2024-03-06T07:23:01.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.928+0000 7f2aa9ffb640 1 -- 172.21.15.77:0/3464433438 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4030c10 con 0x7f2aac070fe0 2024-03-06T07:23:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.929+0000 7f2ab1634640 1 -- 172.21.15.77:0/3464433438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac070fe0 msgr2=0x7f2aac0cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.929+0000 7f2ab1634640 1 --2- 172.21.15.77:0/3464433438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac070fe0 0x7f2aac0cda60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009870 tx=0x7f2aa402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.929+0000 7f2ab1634640 1 -- 172.21.15.77:0/3464433438 shutdown_connections 2024-03-06T07:23:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.929+0000 7f2ab1634640 1 --2- 172.21.15.77:0/3464433438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac070fe0 0x7f2aac0cda60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.929+0000 7f2ab1634640 1 --2- 172.21.15.77:0/3464433438 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac1444c0 0x7f2aac06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.929+0000 7f2ab1634640 1 -- 172.21.15.77:0/3464433438 >> 172.21.15.77:0/3464433438 conn(0x7f2aac0b1c80 msgr2=0x7f2aac0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.929+0000 7f2ab1634640 1 -- 172.21.15.77:0/3464433438 shutdown_connections 2024-03-06T07:23:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.929+0000 7f2ab1634640 1 -- 172.21.15.77:0/3464433438 wait complete. 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.930+0000 7f2ab1634640 1 Processor -- start 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.930+0000 7f2ab1634640 1 -- start start 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2ab1634640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac070fe0 0x7f2aac1404a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2ab1634640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aac1444c0 0x7f2aac13d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2ab1634640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac15b2d0 0x7f2aac1f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2ab1634640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac06cf40 con 0x7f2aac15b2d0 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2ab1634640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac06def0 con 0x7f2aac1444c0 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2ab1634640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac06d0c0 con 0x7f2aac070fe0 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac15b2d0 0x7f2aac1f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac15b2d0 0x7f2aac1f4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52962/0 (socket says 172.21.15.77:52962) 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 -- 172.21.15.77:0/4029582783 learned_addr learned my addr 172.21.15.77:0/4029582783 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac070fe0 0x7f2aac1404a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aac1444c0 0x7f2aac13d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 -- 172.21.15.77:0/4029582783 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac070fe0 msgr2=0x7f2aac1404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac070fe0 0x7f2aac1404a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 -- 172.21.15.77:0/4029582783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aac1444c0 msgr2=0x7f2aac13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aac1444c0 0x7f2aac13d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 -- 172.21.15.77:0/4029582783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0b3a90 con 0x7f2aac15b2d0 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.931+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac15b2d0 0x7f2aac1f4cd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009870 tx=0x7f2aa4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.932+0000 7f2a937fe640 1 -- 172.21.15.77:0/4029582783 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4048070 con 0x7f2aac15b2d0 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.932+0000 7f2a937fe640 1 -- 172.21.15.77:0/4029582783 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa4030640 con 0x7f2aac15b2d0 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.932+0000 7f2ab1634640 1 -- 172.21.15.77:0/4029582783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac1f5520 con 0x7f2aac15b2d0 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.932+0000 7f2a937fe640 1 -- 172.21.15.77:0/4029582783 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4034040 con 0x7f2aac15b2d0 2024-03-06T07:23:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.932+0000 7f2ab1634640 1 -- 172.21.15.77:0/4029582783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac0b3d00 con 0x7f2aac15b2d0 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.933+0000 7f2a937fe640 1 -- 172.21.15.77:0/4029582783 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2aa404b070 con 0x7f2aac15b2d0 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.933+0000 7f2a937fe640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a9806b4d0 0x7f2a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.934+0000 7f2a937fe640 1 -- 172.21.15.77:0/4029582783 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2aa40abfa0 con 0x7f2aac15b2d0 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.934+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a9806b4d0 0x7f2a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.934+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a9806b4d0 0x7f2a9806d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0113e0 tx=0x7f2a9c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.934+0000 7f2ab4218740 1 Processor -- start 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2ab4218740 1 -- start start 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332b6870 0x562e332b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332ba8c0 0x562e332bc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2ab4218740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e332206b0 con 0x562e332ba8c0 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2ab4218740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e331c9a60 con 0x562e332e6600 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2ab4218740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e3321c750 con 0x562e332b6870 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2aaa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332ba8c0 0x562e332bc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2aaa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332ba8c0 0x562e332bc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52964/0 (socket says 172.21.15.77:52964) 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/1232356468 learned_addr learned my addr 172.21.15.77:0/1232356468 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.935+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332b6870 0x562e332b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332b6870 msgr2=0x562e332b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332b6870 0x562e332b6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 msgr2=0x562e332e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/1232356468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e331e7f30 con 0x562e332ba8c0 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332ba8c0 0x562e332bc360 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0001720 tx=0x7f2aa0007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99863e5612b66fe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2a90ff9640 1 -- 172.21.15.77:0/1232356468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa001a070 con 0x562e332ba8c0 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2a90ff9640 1 -- 172.21.15.77:0/1232356468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa0005320 con 0x562e332ba8c0 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.936+0000 7f2a90ff9640 1 -- 172.21.15.77:0/1232356468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0007290 con 0x562e332ba8c0 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 -- 172.21.15.77:0/1232356468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332ba8c0 msgr2=0x562e332bc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332ba8c0 0x562e332bc360 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0001720 tx=0x7f2aa0007e50 comp rx=0 tx=0).stop 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 -- 172.21.15.77:0/1232356468 shutdown_connections 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332ba8c0 0x562e332bc360 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1232356468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332b6870 0x562e332b6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 -- 172.21.15.77:0/1232356468 >> 172.21.15.77:0/1232356468 conn(0x562e332b88f0 msgr2=0x562e332ba1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 -- 172.21.15.77:0/1232356468 shutdown_connections 2024-03-06T07:23:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 -- 172.21.15.77:0/1232356468 wait complete. 2024-03-06T07:23:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 -- start start 2024-03-06T07:23:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.937+0000 7f2ab4218740 1 Processor -- start 2024-03-06T07:23:01.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332db0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332e27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e2d00 0x562e332e7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2ab4218740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e332bad50 con 0x562e332dc350 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2ab4218740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e331dcea0 con 0x562e332e6600 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2ab4218740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e332bb030 con 0x562e332e2d00 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2ab4218740 10 client.0 _ll_register_callbacks cb 0x562e331dc870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332e27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332e27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52980/0 (socket says 172.21.15.77:52980) 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/3186433339 learned_addr learned my addr 172.21.15.77:0/3186433339 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332db0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e2d00 0x562e332e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e2d00 msgr2=0x562e332e7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e2d00 0x562e332e7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 msgr2=0x562e332db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332db0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.938+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/3186433339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562e331dd270 con 0x562e332dc350 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332e27c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0013000 tx=0x7f2aa0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aab6a574cf89556 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2a89ffb640 1 -- 172.21.15.77:0/3186433339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0023020 con 0x562e332dc350 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2a89ffb640 1 -- 172.21.15.77:0/3186433339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2aa00067d0 con 0x562e332dc350 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 1 -- 172.21.15.77:0/3186433339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2aa0005320 con 0x562e332dc350 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2a89ffb640 1 -- 172.21.15.77:0/3186433339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2aa002d050 con 0x562e332dc350 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 10 client.5376 fetch_fsmap learned FSMap version 13 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 1 -- 172.21.15.77:0/3186433339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x562e32f352d0 con 0x562e332dc350 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2a89ffb640 1 -- 172.21.15.77:0/3186433339 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa0005320 con 0x562e332dc350 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2a89ffb640 1 -- 172.21.15.77:0/3186433339 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 13) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2aa0030020 con 0x562e332dc350 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 10 client.5376 fetch_fsmap finished waiting for FSMap version 13 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 10 client.5376 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:01.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 20 client.5376 populate_metadata read hostname 'smithi077' 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] conn(0x562e332e94f0 0x562e332e98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 4 client.5376 mds_command: new command op to 5239 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2ab4218740 1 -- 172.21.15.77:0/3186433339 --> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e3322dbf0 con 0x562e332e94f0 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.939+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] conn(0x562e332e94f0 0x562e332e98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.940+0000 7f2a89ffb640 10 client.5376 ms_handle_connect on v2:172.21.15.158:6832/929839050 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.940+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] conn(0x562e332e94f0 0x562e332e98d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c67009cab1745a4 server_cookie=60165421b38320fd in_seq=0 out_seq=0 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2a89ffb640 1 -- 172.21.15.77:0/3186433339 <== mds.0 v2:172.21.15.158:6832/929839050 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562e3322dbf0 con 0x562e332e94f0 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2a89ffb640 10 client.5376 handle_command_reply: tid=0 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 client.5376 shutdown 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 -- 172.21.15.77:0/3186433339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 msgr2=0x562e332e27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332e27c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0013000 tx=0x7f2aa0000bc0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 -- 172.21.15.77:0/3186433339 shutdown_connections 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] conn(0x562e332e94f0 0x562e332e98d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332e27c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e2d00 0x562e332e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2a89ffb640 0 client.5376 ms_handle_reset on v2:172.21.15.158:6832/929839050 2024-03-06T07:23:01.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3186433339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332db0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 -- 172.21.15.77:0/3186433339 >> 172.21.15.77:0/3186433339 conn(0x562e332b6870 msgr2=0x562e332b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 -- 172.21.15.77:0/3186433339 shutdown_connections 2024-03-06T07:23:01.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 1 -- 172.21.15.77:0/3186433339 wait complete. 2024-03-06T07:23:01.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.941+0000 7f2ab4218740 20 client.5376 trim_cache size 0 max 16384 2024-03-06T07:23:01.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.957+0000 7f2ab4218740 1 Processor -- start 2024-03-06T07:23:01.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2ab4218740 1 -- start start 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332dc350 0x562e332e2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332e2d00 0x562e3331b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332bc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2ab4218740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac1fdfa0 con 0x562e332e2d00 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2ab4218740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac1fd0e0 con 0x562e332e6600 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2ab4218740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0cd0c0 con 0x562e332dc350 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2aaa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332e2d00 0x562e3331b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2aaa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332e2d00 0x562e3331b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52990/0 (socket says 172.21.15.77:52990) 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/3079514549 learned_addr learned my addr 172.21.15.77:0/3079514549 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332dc350 0x562e332e2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.958+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332bc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/3079514549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332dc350 msgr2=0x562e332e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332dc350 0x562e332e2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/3079514549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 msgr2=0x562e332bc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332bc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2aaa7fc640 1 -- 172.21.15.77:0/3079514549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e32fff6b0 con 0x562e332e2d00 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332e2d00 0x562e3331b920 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0006c30 tx=0x7f2aa001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ea5e702215466b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2a8bfff640 1 -- 172.21.15.77:0/3079514549 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0036070 con 0x562e332e2d00 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2a8bfff640 1 -- 172.21.15.77:0/3079514549 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa0006cb0 con 0x562e332e2d00 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2a8bfff640 1 -- 172.21.15.77:0/3079514549 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa001a040 con 0x562e332e2d00 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 -- 172.21.15.77:0/3079514549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332e2d00 msgr2=0x562e3331b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332e2d00 0x562e3331b920 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0006c30 tx=0x7f2aa001e000 comp rx=0 tx=0).stop 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 -- 172.21.15.77:0/3079514549 shutdown_connections 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332e2d00 0x562e3331b920 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e6600 0x562e332bc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 --2- 172.21.15.77:0/3079514549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332dc350 0x562e332e2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 -- 172.21.15.77:0/3079514549 >> 172.21.15.77:0/3079514549 conn(0x562e332b6870 msgr2=0x562e332daef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 -- 172.21.15.77:0/3079514549 shutdown_connections 2024-03-06T07:23:01.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.959+0000 7f2ab4218740 1 -- 172.21.15.77:0/3079514549 wait complete. 2024-03-06T07:23:01.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.960+0000 7f2ab4218740 1 -- start start 2024-03-06T07:23:01.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.960+0000 7f2ab4218740 1 Processor -- start 2024-03-06T07:23:01.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332dfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e2d00 0x562e332e0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2ab4218740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e6600 0x562e332d9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2ab4218740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e332df020 con 0x562e332dc350 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2ab4218740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0b4ac0 con 0x562e332e2d00 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2ab4218740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e332df680 con 0x562e332e6600 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2ab4218740 10 client.0 _ll_register_callbacks cb 0x7f2a9c009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332dfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332dfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52996/0 (socket says 172.21.15.77:52996) 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 -- 172.21.15.77:0/1957973129 learned_addr learned my addr 172.21.15.77:0/1957973129 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e2d00 0x562e332e0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aab7fe640 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e6600 0x562e332d9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 -- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e6600 msgr2=0x562e332d9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e6600 0x562e332d9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 -- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e2d00 msgr2=0x562e332e0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e2d00 0x562e332e0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 -- 172.21.15.77:0/1957973129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562e33305b00 con 0x562e332dc350 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2aaaffd640 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332dfb00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x562e332eacf0 tx=0x7f2a9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b14219a810519c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2a89ffb640 1 -- 172.21.15.77:0/1957973129 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c030070 con 0x562e332dc350 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2a89ffb640 1 -- 172.21.15.77:0/1957973129 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a9c02c9a0 con 0x562e332dc350 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2ab4218740 1 -- 172.21.15.77:0/1957973129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2aa0006cb0 con 0x562e332dc350 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.961+0000 7f2a89ffb640 1 -- 172.21.15.77:0/1957973129 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a9c03b050 con 0x562e332dc350 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2ab4218740 10 client.5382 fetch_fsmap learned FSMap version 13 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2ab4218740 1 -- 172.21.15.77:0/1957973129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x562e332e0700 con 0x562e332dc350 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2a89ffb640 1 -- 172.21.15.77:0/1957973129 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa0006cb0 con 0x562e332dc350 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2a89ffb640 1 -- 172.21.15.77:0/1957973129 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 13) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2a9c03e020 con 0x562e332dc350 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2ab4218740 10 client.5382 fetch_fsmap finished waiting for FSMap version 13 2024-03-06T07:23:01.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2ab4218740 10 client.5382 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2ab4218740 20 client.5382 populate_metadata read hostname 'smithi077' 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] conn(0x562e332da5a0 0x562e332e5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2ab4218740 4 client.5382 mds_command: new command op to 5239 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2ab4218740 1 -- 172.21.15.77:0/1957973129 --> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562e332b9050 con 0x562e332da5a0 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] conn(0x562e332da5a0 0x562e332e5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2aaa7fc640 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] conn(0x562e332da5a0 0x562e332e5470 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93bc079f6c4335b0 server_cookie=e1cc2345bb127c56 in_seq=0 out_seq=0 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.962+0000 7f2a89ffb640 10 client.5382 ms_handle_connect on v2:172.21.15.158:6832/929839050 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2a89ffb640 1 -- 172.21.15.77:0/1957973129 <== mds.0 v2:172.21.15.158:6832/929839050 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x562e332b9050 con 0x562e332da5a0 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2a89ffb640 10 client.5382 handle_command_reply: tid=0 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 client.5382 shutdown 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 -- 172.21.15.77:0/1957973129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 msgr2=0x562e332dfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332dfb00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x562e332eacf0 tx=0x7f2a9c004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 -- 172.21.15.77:0/1957973129 shutdown_connections 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:6832/929839050,v1:172.21.15.158:6834/929839050] conn(0x562e332da5a0 0x562e332e5470 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562e332e6600 0x562e332d9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562e332e2d00 0x562e332e0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2a89ffb640 0 client.5382 ms_handle_reset on v2:172.21.15.158:6832/929839050 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 --2- 172.21.15.77:0/1957973129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e332dc350 0x562e332dfb00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 -- 172.21.15.77:0/1957973129 >> 172.21.15.77:0/1957973129 conn(0x562e332b6870 msgr2=0x562e332bcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 -- 172.21.15.77:0/1957973129 shutdown_connections 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 1 -- 172.21.15.77:0/1957973129 wait complete. 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab4218740 20 client.5382 trim_cache size 0 max 16384 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:23:01.971 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5239, 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 4.6326947839999999, 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 13, 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 33, 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 33, 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 11.709973578 2024-03-06T07:23:01.972 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 -- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a9806b4d0 msgr2=0x7f2a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a9806b4d0 0x7f2a9806d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0113e0 tx=0x7f2a9c00b040 comp rx=0 tx=0).stop 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 -- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac15b2d0 msgr2=0x7f2aac1f4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac15b2d0 0x7f2aac1f4cd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009870 tx=0x7f2aa4004730 comp rx=0 tx=0).stop 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 -- 172.21.15.77:0/4029582783 shutdown_connections 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a9806b4d0 0x7f2a9806d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aac15b2d0 0x7f2aac1f4cd0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aac1444c0 0x7f2aac13d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 --2- 172.21.15.77:0/4029582783 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aac070fe0 0x7f2aac1404a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 -- 172.21.15.77:0/4029582783 >> 172.21.15.77:0/4029582783 conn(0x7f2aac0b1c80 msgr2=0x7f2aac142c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 -- 172.21.15.77:0/4029582783 shutdown_connections 2024-03-06T07:23:01.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:01.963+0000 7f2ab1634640 1 -- 172.21.15.77:0/4029582783 wait complete. 2024-03-06T07:23:01.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:02.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.074+0000 7f258d9a2640 1 Processor -- start 2024-03-06T07:23:02.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.074+0000 7f258d9a2640 1 -- start start 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f258d9a2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f25880cf190 0x7f25880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f258d9a2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880b4020 0x7f25880b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f258d9a2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25880b2fa0 con 0x7f25880d20c0 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f258d9a2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25880b3260 con 0x7f25880cf190 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f258d9a2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25880b3540 con 0x7f25880b4020 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f2587fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f25880cf190 0x7f25880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f258c9a0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f25880d20c0 0x7f25880d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52172/0 (socket says 172.21.15.77:52172) 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f258c9a0640 1 -- 172.21.15.77:0/2257039252 learned_addr learned my addr 172.21.15.77:0/2257039252 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f258d1a1640 1 --2- 172.21.15.77:0/2257039252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880b4020 0x7f25880b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f2587fff640 1 -- 172.21.15.77:0/2257039252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880b4020 msgr2=0x7f25880b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f2587fff640 1 --2- 172.21.15.77:0/2257039252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880b4020 0x7f25880b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f2587fff640 1 -- 172.21.15.77:0/2257039252 >> v1:172.21.15.77:6789/0 conn(0x7f25880d20c0 legacy=0x7f25880d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.075+0000 7f25877fe640 1 -- 172.21.15.77:0/2257039252 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087882104 0 0) 0x7f25880b2fa0 con 0x7f25880d20c0 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.076+0000 7f2587fff640 1 -- 172.21.15.77:0/2257039252 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880b3820 con 0x7f25880cf190 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.076+0000 7f2587fff640 1 --2- 172.21.15.77:0/2257039252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880cf190 0x7f25880cf590 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f257c0097a0 tx=0x7f257c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac83e7be5dba36b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.076+0000 7f25877fe640 1 -- 172.21.15.77:0/2257039252 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c034070 con 0x7f25880cf190 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.076+0000 7f25877fe640 1 -- 172.21.15.77:0/2257039252 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c030610 con 0x7f25880cf190 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.076+0000 7f25877fe640 1 -- 172.21.15.77:0/2257039252 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c0309e0 con 0x7f25880cf190 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.077+0000 7f258d9a2640 1 -- 172.21.15.77:0/2257039252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880cf190 msgr2=0x7f25880cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.077+0000 7f258d9a2640 1 --2- 172.21.15.77:0/2257039252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880cf190 0x7f25880cf590 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f257c0097a0 tx=0x7f257c02b960 comp rx=0 tx=0).stop 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.077+0000 7f258d9a2640 1 -- 172.21.15.77:0/2257039252 shutdown_connections 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.077+0000 7f258d9a2640 1 --2- 172.21.15.77:0/2257039252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880b4020 0x7f25880b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.077+0000 7f258d9a2640 1 --2- 172.21.15.77:0/2257039252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880cf190 0x7f25880cf590 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.077+0000 7f258d9a2640 1 -- 172.21.15.77:0/2257039252 >> 172.21.15.77:0/2257039252 conn(0x7f258809ce20 msgr2=0x7f25880d3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.077+0000 7f258d9a2640 1 -- 172.21.15.77:0/2257039252 shutdown_connections 2024-03-06T07:23:02.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.077+0000 7f258d9a2640 1 -- 172.21.15.77:0/2257039252 wait complete. 2024-03-06T07:23:02.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.078+0000 7f258d9a2640 1 Processor -- start 2024-03-06T07:23:02.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.078+0000 7f258d9a2640 1 -- start start 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258d9a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880b4020 0x7f25880b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258d9a2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f25880cf190 0x7f25880cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258d9a2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880d20c0 0x7f25881560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258d9a2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2588161630 con 0x7f25880b4020 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258d9a2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25880d1af0 con 0x7f25880cf190 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258d9a2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25881617b0 con 0x7f25880d20c0 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880b4020 0x7f25880b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880b4020 0x7f25880b21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53028/0 (socket says 172.21.15.77:53028) 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 -- 172.21.15.77:0/3512901608 learned_addr learned my addr 172.21.15.77:0/3512901608 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258d1a1640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880d20c0 0x7f25881560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f2587fff640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f25880cf190 0x7f25880cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 -- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880d20c0 msgr2=0x7f25881560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880d20c0 0x7f25881560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 -- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f25880cf190 msgr2=0x7f25880cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f25880cf190 0x7f25880cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 -- 172.21.15.77:0/3512901608 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880cb320 con 0x7f25880b4020 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258d1a1640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880d20c0 0x7f25881560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f2587fff640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f25880cf190 0x7f25880cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.079+0000 7f258c9a0640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880b4020 0x7f25880b21f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f257800c4d0 tx=0x7f257800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.080+0000 7f25857fa640 1 -- 172.21.15.77:0/3512901608 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578018070 con 0x7f25880b4020 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.080+0000 7f25857fa640 1 -- 172.21.15.77:0/3512901608 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578007500 con 0x7f25880b4020 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.080+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25880cb600 con 0x7f25880b4020 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.080+0000 7f25857fa640 1 -- 172.21.15.77:0/3512901608 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578004a20 con 0x7f25880b4020 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.080+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25880cbbc0 con 0x7f25880b4020 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.081+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25880b7340 con 0x7f25880b4020 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.081+0000 7f25857fa640 1 -- 172.21.15.77:0/3512901608 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f257801b070 con 0x7f25880b4020 2024-03-06T07:23:02.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.081+0000 7f25857fa640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f257406b4d0 0x7f257406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.081+0000 7f25857fa640 1 -- 172.21.15.77:0/3512901608 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25780894b0 con 0x7f25880b4020 2024-03-06T07:23:02.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.081+0000 7f2587fff640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f257406b4d0 0x7f257406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.082+0000 7f2587fff640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f257406b4d0 0x7f257406d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f257c036000 tx=0x7f257c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.086+0000 7f25857fa640 1 -- 172.21.15.77:0/3512901608 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25780583c0 con 0x7f25880b4020 2024-03-06T07:23:02.120 INFO:tasks.ceph.mds.b.smithi158.stderr:2024-03-06T07:23:02.119+0000 7fbaaf4aa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:02.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.294+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f25880d1ea0 con 0x7f25880b4020 2024-03-06T07:23:02.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.295+0000 7f25857fa640 1 -- 172.21.15.77:0/3512901608 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2578057cc0 con 0x7f25880b4020 2024-03-06T07:23:02.297 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f257406b4d0 msgr2=0x7f257406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f257406b4d0 0x7f257406d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f257c036000 tx=0x7f257c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880b4020 msgr2=0x7f25880b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880b4020 0x7f25880b21f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f257800c4d0 tx=0x7f257800c990 comp rx=0 tx=0).stop 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 shutdown_connections 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f25880d20c0 0x7f25881560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f25880cf190 0x7f25880cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f257406b4d0 0x7f257406d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 --2- 172.21.15.77:0/3512901608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25880b4020 0x7f25880b21f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 >> 172.21.15.77:0/3512901608 conn(0x7f258809ce20 msgr2=0x7f25880d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 shutdown_connections 2024-03-06T07:23:02.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:02.297+0000 7f258d9a2640 1 -- 172.21.15.77:0/3512901608 wait complete. 2024-03-06T07:23:02.310 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:02.310 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:02.310 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:23:02.310 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:02.310 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:02.310 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:02.311 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:02.311 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:02.311 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:02.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:02.312 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:23:02.327 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:02.327 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:02.327 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:02.327 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:02.327 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:02.327 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42615sec preferred_lft 42615sec 2024-03-06T07:23:02.328 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:02.329 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:23:02.330 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:02.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:02.330 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:02.394 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:23:02.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:02.395 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:23:02.450 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:23:02.450 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:02.450 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:02.450 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:02.450 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:02.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:02.451 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:23:02.451 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:23:02.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:02.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:02.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:02.451 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:23:02.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:02.546 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:23:02.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:23:02.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:02.561 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:02.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:02.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:23:02.631 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:02.631 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:02.631 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:02.685 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:02.686 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:02.686 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:02.740 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:23:02.808 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:02.864 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:23:02.865 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:02.866 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:02.867 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:02.867 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:02.867 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:23:02.867 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:02.867 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:02.867 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:02.867 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:02.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:02.867 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:23:02.919 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:02.919 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:02.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:23:02.989 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:03.044 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:23:03.044 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:23:03.044 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:23:03.044 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:23:03.044 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:23:03.044 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:03.045 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:03.046 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:03.047 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:03.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.047 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:23:03.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:23:03.046+0000 7fb77be40480 -1 init, newargv = 0x55d182bf8620 newargc=15 2024-03-06T07:23:03.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[52454]: starting ceph client 2024-03-06T07:23:03.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[52454]: starting fuse 2024-03-06T07:23:03.063 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:23:03.063 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:23:03.063 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:03.063 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:23:03.063 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:23:03.063 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:23:03.064 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:03.065 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:23:03.183 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.52454.asok 2024-03-06T07:23:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:03.193 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52454.asok 2024-03-06T07:23:03.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.193 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52454.asok status 2024-03-06T07:23:03.327 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "52454", 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:23:03.328 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5397, 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 5397 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 2865844203 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:03.329 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 2865844203 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.5397 192.168.0.1:0/2865844203", 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/2865844203", 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 13, 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 33, 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:23:03.330 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:23:03.331 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:23:03.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.336 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:23:03.336 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:03.351 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:23:03.351 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:03.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:03.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> subdir = "delete_me" 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> size = 1024 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> file_count = 1000 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:23:03.420 DEBUG:teuthology.orchestra.run.smithi077:> for i in range(0, file_count): 2024-03-06T07:23:03.421 DEBUG:teuthology.orchestra.run.smithi077:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:23:03.421 DEBUG:teuthology.orchestra.run.smithi077:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:23:03.421 DEBUG:teuthology.orchestra.run.smithi077:> f.write(size * '"'"'x'"'"') 2024-03-06T07:23:03.421 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:23:05.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:05.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.396+0000 7f94d69b6640 1 Processor -- start 2024-03-06T07:23:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.396+0000 7f94d69b6640 1 -- start start 2024-03-06T07:23:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d69b6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f94d00d22b0 0x7f94d00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d69b6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f94d00b41f0 0x7f94d00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d69b6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d00b3430 con 0x7f94d015dfe0 2024-03-06T07:23:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d69b6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d00d2c50 con 0x7f94d00b41f0 2024-03-06T07:23:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d69b6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d00b31e0 con 0x7f94d00d22b0 2024-03-06T07:23:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d4f53640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f94d00b41f0 0x7f94d00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94cffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f94d015dfe0 0x7f94d00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52174/0 (socket says 172.21.15.77:52174) 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94cffff640 1 -- 172.21.15.77:0/3913385522 learned_addr learned my addr 172.21.15.77:0/3913385522 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94cf7fe640 1 --2- 172.21.15.77:0/3913385522 >> v2:172.21.15.158:3300/0 conn(0x7f94d00d22b0 0x7f94d00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d4f53640 1 -- 172.21.15.77:0/3913385522 >> v2:172.21.15.158:3300/0 conn(0x7f94d00d22b0 msgr2=0x7f94d00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d4f53640 1 --2- 172.21.15.77:0/3913385522 >> v2:172.21.15.158:3300/0 conn(0x7f94d00d22b0 0x7f94d00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94d4f53640 1 -- 172.21.15.77:0/3913385522 >> v1:172.21.15.77:6789/0 conn(0x7f94d015dfe0 legacy=0x7f94d00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.398+0000 7f94d4f53640 1 -- 172.21.15.77:0/3913385522 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d00b3710 con 0x7f94d00b41f0 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.397+0000 7f94ceffd640 1 -- 172.21.15.77:0/3913385522 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583361592 0 0) 0x7f94d00b3430 con 0x7f94d015dfe0 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.398+0000 7f94d4f53640 1 --2- 172.21.15.77:0/3913385522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d00b41f0 0x7f94d00b4680 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009d90 tx=0x7f94c802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b99de93376135b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.398+0000 7f94ceffd640 1 -- 172.21.15.77:0/3913385522 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8039070 con 0x7f94d00b41f0 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.398+0000 7f94ceffd640 1 -- 172.21.15.77:0/3913385522 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c802cc60 con 0x7f94d00b41f0 2024-03-06T07:23:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.398+0000 7f94ceffd640 1 -- 172.21.15.77:0/3913385522 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8034660 con 0x7f94d00b41f0 2024-03-06T07:23:05.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.399+0000 7f94d69b6640 1 -- 172.21.15.77:0/3913385522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d00b41f0 msgr2=0x7f94d00b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.399+0000 7f94d69b6640 1 --2- 172.21.15.77:0/3913385522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d00b41f0 0x7f94d00b4680 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009d90 tx=0x7f94c802beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.399+0000 7f94d69b6640 1 -- 172.21.15.77:0/3913385522 shutdown_connections 2024-03-06T07:23:05.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.399+0000 7f94d69b6640 1 --2- 172.21.15.77:0/3913385522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d00b41f0 0x7f94d00b4680 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.399+0000 7f94d69b6640 1 --2- 172.21.15.77:0/3913385522 >> v2:172.21.15.158:3300/0 conn(0x7f94d00d22b0 0x7f94d00d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.399+0000 7f94d69b6640 1 -- 172.21.15.77:0/3913385522 >> 172.21.15.77:0/3913385522 conn(0x7f94d00d3cd0 msgr2=0x7f94d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.399+0000 7f94d69b6640 1 -- 172.21.15.77:0/3913385522 shutdown_connections 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.399+0000 7f94d69b6640 1 -- 172.21.15.77:0/3913385522 wait complete. 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.400+0000 7f94d69b6640 1 Processor -- start 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.400+0000 7f94d69b6640 1 -- start start 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d69b6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94d00b41f0 0x7f94d015ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d69b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94d00d22b0 0x7f94d00cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d69b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d015dfe0 0x7f94d00b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d69b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d00cfa00 con 0x7f94d015dfe0 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d69b6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d00b3d60 con 0x7f94d00b41f0 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d69b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d00d1c70 con 0x7f94d00d22b0 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d015dfe0 0x7f94d00b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d015dfe0 0x7f94d00b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53052/0 (socket says 172.21.15.77:53052) 2024-03-06T07:23:05.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 -- 172.21.15.77:0/4155275723 learned_addr learned my addr 172.21.15.77:0/4155275723 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94cffff640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94d00b41f0 0x7f94d015ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94cf7fe640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94d00d22b0 0x7f94d00cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 -- 172.21.15.77:0/4155275723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94d00d22b0 msgr2=0x7f94d00cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94d00d22b0 0x7f94d00cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 -- 172.21.15.77:0/4155275723 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94d00b41f0 msgr2=0x7f94d015ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94d00b41f0 0x7f94d015ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 -- 172.21.15.77:0/4155275723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d00cc820 con 0x7f94d015dfe0 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.401+0000 7f94d4f53640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d015dfe0 0x7f94d00b7540 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009d90 tx=0x7f94c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.402+0000 7f94ccff9640 1 -- 172.21.15.77:0/4155275723 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8044020 con 0x7f94d015dfe0 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.402+0000 7f94ccff9640 1 -- 172.21.15.77:0/4155275723 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c803a040 con 0x7f94d015dfe0 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.402+0000 7f94ccff9640 1 -- 172.21.15.77:0/4155275723 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8039050 con 0x7f94d015dfe0 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.402+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d00ccb00 con 0x7f94d015dfe0 2024-03-06T07:23:05.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.402+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d00cd3a0 con 0x7f94d015dfe0 2024-03-06T07:23:05.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.403+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d00cccd0 con 0x7f94d015dfe0 2024-03-06T07:23:05.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.403+0000 7f94ccff9640 1 -- 172.21.15.77:0/4155275723 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94c8003bb0 con 0x7f94d015dfe0 2024-03-06T07:23:05.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.404+0000 7f94ccff9640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94bc06b4d0 0x7f94bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.404+0000 7f94ccff9640 1 -- 172.21.15.77:0/4155275723 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94c80ac650 con 0x7f94d015dfe0 2024-03-06T07:23:05.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.404+0000 7f94cffff640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94bc06b4d0 0x7f94bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.404+0000 7f94cffff640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94bc06b4d0 0x7f94bc06d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f94c0007fd0 tx=0x7f94c0007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.408+0000 7f94ccff9640 1 -- 172.21.15.77:0/4155275723 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94c804e060 con 0x7f94d015dfe0 2024-03-06T07:23:05.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.617+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94d004f330 con 0x7f94d015dfe0 2024-03-06T07:23:05.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.617+0000 7f94ccff9640 1 -- 172.21.15.77:0/4155275723 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f94c807af10 con 0x7f94d015dfe0 2024-03-06T07:23:05.620 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:05.620 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:05.621 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:05.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94bc06b4d0 msgr2=0x7f94bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94bc06b4d0 0x7f94bc06d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f94c0007fd0 tx=0x7f94c0007de0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d015dfe0 msgr2=0x7f94d00b7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d015dfe0 0x7f94d00b7540 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009d90 tx=0x7f94c8004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 shutdown_connections 2024-03-06T07:23:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94bc06b4d0 0x7f94bc06d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d015dfe0 0x7f94d00b7540 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94d00d22b0 0x7f94d00cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 --2- 172.21.15.77:0/4155275723 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94d00b41f0 0x7f94d015ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 >> 172.21.15.77:0/4155275723 conn(0x7f94d00d3cd0 msgr2=0x7f94d00d0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 shutdown_connections 2024-03-06T07:23:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05.619+0000 7f94d69b6640 1 -- 172.21.15.77:0/4155275723 wait complete. 2024-03-06T07:23:05.632 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:23:05.912 INFO:teuthology.orchestra.run.smithi158.stdout:{"message":"","return_code":0} 2024-03-06T07:23:05.912 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:23:05.913 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:23:05.913 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:23:05.913 DEBUG:teuthology.orchestra.run.smithi077:> import stat 2024-03-06T07:23:05.913 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:05.913 DEBUG:teuthology.orchestra.run.smithi077:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:23:05.913 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:23:05.977 INFO:teuthology.orchestra.run.smithi077.stdout:1099511627776 2024-03-06T07:23:05.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:05.986 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973663cf40 1 Processor -- start 2024-03-06T07:23:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973663cf40 1 -- start start 2024-03-06T07:23:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973663cf40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da121f0 0x56544daa7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973663cf40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da125d0 0x56544da8ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973663cf40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da13940 0x56544daa69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973663cf40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56544d975b70 con 0x56544da121f0 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973663cf40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x56544d979ad0 con 0x56544da13940 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973663cf40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x56544da16cf0 con 0x56544da125d0 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973399a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da121f0 0x56544daa7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973399a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da121f0 0x56544daa7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53062/0 (socket says 172.21.15.77:53062) 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.065+0000 7f973399a640 1 -- 172.21.15.77:0/3589223701 learned_addr learned my addr 172.21.15.77:0/3589223701 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973419b640 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da13940 0x56544daa69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f9733199640 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da125d0 0x56544da8ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973399a640 1 -- 172.21.15.77:0/3589223701 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da125d0 msgr2=0x56544da8ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973399a640 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da125d0 0x56544da8ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973399a640 1 -- 172.21.15.77:0/3589223701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da13940 msgr2=0x56544daa69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973399a640 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da13940 0x56544daa69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973399a640 1 -- 172.21.15.77:0/3589223701 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56544daa6080 con 0x56544da121f0 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973399a640 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da121f0 0x56544daa7b20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f972401e3c0 tx=0x7f9724051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb182e4df4b56f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f9732197640 1 -- 172.21.15.77:0/3589223701 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724067070 con 0x56544da121f0 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f9732197640 1 -- 172.21.15.77:0/3589223701 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972404ec00 con 0x56544da121f0 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f9732197640 1 -- 172.21.15.77:0/3589223701 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972404f470 con 0x56544da121f0 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973663cf40 1 -- 172.21.15.77:0/3589223701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da121f0 msgr2=0x56544daa7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973663cf40 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da121f0 0x56544daa7b20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f972401e3c0 tx=0x7f9724051000 comp rx=0 tx=0).stop 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973663cf40 1 -- 172.21.15.77:0/3589223701 shutdown_connections 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973663cf40 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da13940 0x56544daa69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973663cf40 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da125d0 0x56544da8ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973663cf40 1 --2- 172.21.15.77:0/3589223701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da121f0 0x56544daa7b20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.066+0000 7f973663cf40 1 -- 172.21.15.77:0/3589223701 >> 172.21.15.77:0/3589223701 conn(0x56544d9fa350 msgr2=0x56544da182f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.067+0000 7f973663cf40 1 -- 172.21.15.77:0/3589223701 shutdown_connections 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.067+0000 7f973663cf40 1 -- 172.21.15.77:0/3589223701 wait complete. 2024-03-06T07:23:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.067+0000 7f973663cf40 1 Processor -- start 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.067+0000 7f973663cf40 1 -- start start 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973663cf40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da121f0 0x56544da86c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973663cf40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da125d0 0x56544daa48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973663cf40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da13940 0x56544da87ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973663cf40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56544da8cc80 con 0x56544da13940 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973663cf40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x56544daa53f0 con 0x56544da125d0 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973663cf40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x56544daa5de0 con 0x56544da121f0 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da13940 0x56544da87ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da13940 0x56544da87ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53072/0 (socket says 172.21.15.77:53072) 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 -- 172.21.15.77:0/3629407936 learned_addr learned my addr 172.21.15.77:0/3629407936 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973399a640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da121f0 0x56544da86c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f9733199640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da125d0 0x56544daa48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 -- 172.21.15.77:0/3629407936 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da121f0 msgr2=0x56544da86c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da121f0 0x56544da86c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 -- 172.21.15.77:0/3629407936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da125d0 msgr2=0x56544daa48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56544da125d0 0x56544daa48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 -- 172.21.15.77:0/3629407936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56544da8f360 con 0x56544da13940 2024-03-06T07:23:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973399a640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56544da121f0 0x56544da86c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f973419b640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da13940 0x56544da87ff0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f972c006530 tx=0x7f972c001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.068+0000 7f9731195640 1 -- 172.21.15.77:0/3629407936 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c005e30 con 0x56544da13940 2024-03-06T07:23:06.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.069+0000 7f9731195640 1 -- 172.21.15.77:0/3629407936 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c017070 con 0x56544da13940 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.069+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56544da871a0 con 0x56544da13940 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.069+0000 7f9731195640 1 -- 172.21.15.77:0/3629407936 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c011780 con 0x56544da13940 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.069+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56544daa8f30 con 0x56544da13940 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.070+0000 7f9731195640 1 -- 172.21.15.77:0/3629407936 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f972c011920 con 0x56544da13940 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.070+0000 7f9731195640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9720069ee0 0x7f972006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.070+0000 7f9731195640 1 -- 172.21.15.77:0/3629407936 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f972c088a80 con 0x56544da13940 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.070+0000 7f973399a640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9720069ee0 0x7f972006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.070+0000 7f973663cf40 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x56544daa7100 0x56544da8a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.070+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x56544da85390 con 0x56544daa7100 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f9733199640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x56544daa7100 0x56544da8a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973399a640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9720069ee0 0x7f972006c3a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9724064000 tx=0x7f9724053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f9733199640 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x56544daa7100 0x56544da8a4f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f9733199640 1 -- 172.21.15.77:0/3629407936 <== osd.3 v2:172.21.15.77:6816/461341877 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7f9728001930 con 0x56544daa7100 2024-03-06T07:23:06.080 INFO:teuthology.orchestra.run.smithi077.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:23:05.000000+0000, size 0 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x56544daa7100 msgr2=0x56544da8a4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x56544daa7100 0x56544da8a4f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9720069ee0 msgr2=0x7f972006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9720069ee0 0x7f972006c3a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9724064000 tx=0x7f9724053000 comp rx=0 tx=0).stop 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da13940 msgr2=0x56544da87ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 --2- 172.21.15.77:0/3629407936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56544da13940 0x56544da87ff0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f972c006530 tx=0x7f972c001a20 comp rx=0 tx=0).stop 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973419b640 1 -- 172.21.15.77:0/3629407936 reap_dead start 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 shutdown_connections 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 >> 172.21.15.77:0/3629407936 conn(0x56544d9fa350 msgr2=0x56544daa69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 shutdown_connections 2024-03-06T07:23:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:06.071+0000 7f973663cf40 1 -- 172.21.15.77:0/3629407936 wait complete. 2024-03-06T07:23:06.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:06.083 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:23:08.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:08.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.317+0000 7faf711eb640 1 Processor -- start 2024-03-06T07:23:08.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.318+0000 7faf711eb640 1 -- start start 2024-03-06T07:23:08.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.318+0000 7faf711eb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faf6c0d2710 0x7faf6c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.318+0000 7faf711eb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0b3430 con 0x7faf6c0d22b0 2024-03-06T07:23:08.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.318+0000 7faf711eb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0b3710 con 0x7faf6c15dfe0 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.318+0000 7faf711eb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6c0b31e0 con 0x7faf6c0d2710 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.318+0000 7faf63fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faf6c0d22b0 0x7faf6c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52188/0 (socket says 172.21.15.77:52188) 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.318+0000 7faf63fff640 1 -- 172.21.15.77:0/3791325433 learned_addr learned my addr 172.21.15.77:0/3791325433 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.319+0000 7faf6b59e640 1 --2- 172.21.15.77:0/3791325433 >> v2:172.21.15.158:3300/0 conn(0x7faf6c0d2710 0x7faf6c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.319+0000 7faf6a59c640 1 -- 172.21.15.77:0/3791325433 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222414493 0 0) 0x7faf6c0b3430 con 0x7faf6c0d22b0 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.319+0000 7faf6a59c640 1 -- 172.21.15.77:0/3791325433 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf58003520 con 0x7faf6c0d22b0 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.319+0000 7faf6a59c640 1 -- 172.21.15.77:0/3791325433 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719898775 0 0) 0x7faf6c0b3710 con 0x7faf6c15dfe0 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.319+0000 7faf6a59c640 1 -- 172.21.15.77:0/3791325433 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c0b3430 con 0x7faf6c15dfe0 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.319+0000 7faf6b59e640 1 -- 172.21.15.77:0/3791325433 >> v1:172.21.15.158:6789/0 conn(0x7faf6c15dfe0 legacy=0x7faf6c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.319+0000 7faf6b59e640 1 -- 172.21.15.77:0/3791325433 >> v1:172.21.15.77:6789/0 conn(0x7faf6c0d22b0 legacy=0x7faf6c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.319+0000 7faf6b59e640 1 -- 172.21.15.77:0/3791325433 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c0b39f0 con 0x7faf6c0d2710 2024-03-06T07:23:08.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf6b59e640 1 --2- 172.21.15.77:0/3791325433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c0d2710 0x7faf6c0b4900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faf64009dc0 tx=0x7faf6402bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba228e537ce034f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf6a59c640 1 -- 172.21.15.77:0/3791325433 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64039070 con 0x7faf6c0d2710 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf6a59c640 1 -- 172.21.15.77:0/3791325433 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6402ccd0 con 0x7faf6c0d2710 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf6a59c640 1 -- 172.21.15.77:0/3791325433 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64034640 con 0x7faf6c0d2710 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf711eb640 1 -- 172.21.15.77:0/3791325433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c0d2710 msgr2=0x7faf6c0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf711eb640 1 --2- 172.21.15.77:0/3791325433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c0d2710 0x7faf6c0b4900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faf64009dc0 tx=0x7faf6402bf20 comp rx=0 tx=0).stop 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf711eb640 1 -- 172.21.15.77:0/3791325433 shutdown_connections 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf711eb640 1 --2- 172.21.15.77:0/3791325433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c0d2710 0x7faf6c0b4900 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf711eb640 1 -- 172.21.15.77:0/3791325433 >> 172.21.15.77:0/3791325433 conn(0x7faf6c0d3cd0 msgr2=0x7faf6c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf711eb640 1 -- 172.21.15.77:0/3791325433 shutdown_connections 2024-03-06T07:23:08.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.320+0000 7faf711eb640 1 -- 172.21.15.77:0/3791325433 wait complete. 2024-03-06T07:23:08.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf711eb640 1 Processor -- start 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf711eb640 1 -- start start 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf711eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf6c0d22b0 0x7faf6c0b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf711eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf6c0d2710 0x7faf6c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf711eb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c15dfe0 0x7faf6c1f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf711eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0cfa00 con 0x7faf6c0d22b0 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf711eb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0b3d60 con 0x7faf6c15dfe0 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf711eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0d1c70 con 0x7faf6c0d2710 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf6ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf6c0d22b0 0x7faf6c0b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf6ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf6c0d22b0 0x7faf6c0b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53074/0 (socket says 172.21.15.77:53074) 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf6ad9d640 1 -- 172.21.15.77:0/3848581980 learned_addr learned my addr 172.21.15.77:0/3848581980 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf63fff640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf6c0d2710 0x7faf6c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.322+0000 7faf6b59e640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c15dfe0 0x7faf6c1f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf6ad9d640 1 -- 172.21.15.77:0/3848581980 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf6c0d2710 msgr2=0x7faf6c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf6ad9d640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf6c0d2710 0x7faf6c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf6ad9d640 1 -- 172.21.15.77:0/3848581980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c15dfe0 msgr2=0x7faf6c1f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf6ad9d640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c15dfe0 0x7faf6c1f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf6ad9d640 1 -- 172.21.15.77:0/3848581980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c0cb7f0 con 0x7faf6c0d22b0 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf6ad9d640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf6c0d22b0 0x7faf6c0b8220 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00c430 tx=0x7faf5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf637fe640 1 -- 172.21.15.77:0/3848581980 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c017070 con 0x7faf6c0d22b0 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf637fe640 1 -- 172.21.15.77:0/3848581980 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c0052f0 con 0x7faf6c0d22b0 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c0cbad0 con 0x7faf6c0d22b0 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf637fe640 1 -- 172.21.15.77:0/3848581980 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c003830 con 0x7faf6c0d22b0 2024-03-06T07:23:08.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.323+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c0cc370 con 0x7faf6c0d22b0 2024-03-06T07:23:08.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.324+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c04f330 con 0x7faf6c0d22b0 2024-03-06T07:23:08.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.325+0000 7faf637fe640 1 -- 172.21.15.77:0/3848581980 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf5c01a070 con 0x7faf6c0d22b0 2024-03-06T07:23:08.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.325+0000 7faf637fe640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf4c06b4d0 0x7faf4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.325+0000 7faf637fe640 1 -- 172.21.15.77:0/3848581980 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf5c089510 con 0x7faf6c0d22b0 2024-03-06T07:23:08.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.325+0000 7faf63fff640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf4c06b4d0 0x7faf4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.325+0000 7faf63fff640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf4c06b4d0 0x7faf4c06d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faf6c06cd40 tx=0x7faf54007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.329+0000 7faf637fe640 1 -- 172.21.15.77:0/3848581980 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf5c057d40 con 0x7faf6c0d22b0 2024-03-06T07:23:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.548+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf6c15d770 con 0x7faf6c0d22b0 2024-03-06T07:23:08.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.548+0000 7faf637fe640 1 -- 172.21.15.77:0/3848581980 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7faf5c05bde0 con 0x7faf6c0d22b0 2024-03-06T07:23:08.551 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:08.551 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:08.551 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:08.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf4c06b4d0 msgr2=0x7faf4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf4c06b4d0 0x7faf4c06d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faf6c06cd40 tx=0x7faf54007d30 comp rx=0 tx=0).stop 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf6c0d22b0 msgr2=0x7faf6c0b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf6c0d22b0 0x7faf6c0b8220 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00c430 tx=0x7faf5c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 shutdown_connections 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf6c15dfe0 0x7faf6c1f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf6c0d2710 0x7faf6c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf4c06b4d0 0x7faf4c06d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 --2- 172.21.15.77:0/3848581980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf6c0d22b0 0x7faf6c0b8220 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.550+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 >> 172.21.15.77:0/3848581980 conn(0x7faf6c0d3cd0 msgr2=0x7faf6c0b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.551+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 shutdown_connections 2024-03-06T07:23:08.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.551+0000 7faf711eb640 1 -- 172.21.15.77:0/3848581980 wait complete. 2024-03-06T07:23:08.563 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:23:08.858 INFO:teuthology.orchestra.run.smithi158.stdout:{"message":"","return_code":0} 2024-03-06T07:23:08.859 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:23:08.859 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.957+0000 7f536c9b6640 1 Processor -- start 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.957+0000 7f536c9b6640 1 -- start start 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f536c9b6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53680d22b0 0x7f53680d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f536c9b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 0x7f53680b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f536c9b6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680b3430 con 0x7f536815dfe0 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f536c9b6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53680d2c50 con 0x7f53680d22b0 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f536c9b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53680b31e0 con 0x7f53680b41f0 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f536659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f536815dfe0 0x7f53680cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40862/0 (socket says 172.21.15.77:40862) 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f536659c640 1 -- 172.21.15.77:0/3723948713 learned_addr learned my addr 172.21.15.77:0/3723948713 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f5365d9b640 1 --2- 172.21.15.77:0/3723948713 >> v2:172.21.15.77:3300/0 conn(0x7f53680d22b0 0x7f53680d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.958+0000 7f5366d9d640 1 --2- 172.21.15.77:0/3723948713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 0x7f53680b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.959+0000 7f536559a640 1 -- 172.21.15.77:0/3723948713 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544012370 0 0) 0x7f53680b3430 con 0x7f536815dfe0 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.959+0000 7f536559a640 1 -- 172.21.15.77:0/3723948713 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5350003520 con 0x7f536815dfe0 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.959+0000 7f5365d9b640 1 -- 172.21.15.77:0/3723948713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 msgr2=0x7f53680b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.959+0000 7f5365d9b640 1 --2- 172.21.15.77:0/3723948713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 0x7f53680b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.959+0000 7f5365d9b640 1 -- 172.21.15.77:0/3723948713 >> v1:172.21.15.77:6789/0 conn(0x7f536815dfe0 legacy=0x7f53680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.959+0000 7f5365d9b640 1 -- 172.21.15.77:0/3723948713 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53680b3710 con 0x7f53680d22b0 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.959+0000 7f5365d9b640 1 --2- 172.21.15.77:0/3723948713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53680d2710 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f535c0097a0 tx=0x7f535c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb94da8664dccbfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536559a640 1 -- 172.21.15.77:0/3723948713 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c035020 con 0x7f53680d22b0 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536559a640 1 -- 172.21.15.77:0/3723948713 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c031830 con 0x7f53680d22b0 2024-03-06T07:23:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536559a640 1 -- 172.21.15.77:0/3723948713 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c031c00 con 0x7f53680d22b0 2024-03-06T07:23:08.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536c9b6640 1 -- 172.21.15.77:0/3723948713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 msgr2=0x7f53680d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536c9b6640 1 --2- 172.21.15.77:0/3723948713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53680d2710 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f535c0097a0 tx=0x7f535c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536c9b6640 1 -- 172.21.15.77:0/3723948713 shutdown_connections 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536c9b6640 1 --2- 172.21.15.77:0/3723948713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 0x7f53680b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536c9b6640 1 --2- 172.21.15.77:0/3723948713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53680d2710 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536c9b6640 1 -- 172.21.15.77:0/3723948713 >> 172.21.15.77:0/3723948713 conn(0x7f53680d3cd0 msgr2=0x7f53680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536c9b6640 1 -- 172.21.15.77:0/3723948713 shutdown_connections 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.960+0000 7f536c9b6640 1 -- 172.21.15.77:0/3723948713 wait complete. 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.961+0000 7f536c9b6640 1 Processor -- start 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f536c9b6640 1 -- start start 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f536c9b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 0x7f53680b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f536c9b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53681530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f536c9b6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536815dfe0 0x7f536815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f536c9b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53680cfa00 con 0x7f53680d22b0 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f536c9b6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53680b3d60 con 0x7f536815dfe0 2024-03-06T07:23:08.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f536c9b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53680d1c70 con 0x7f53680b41f0 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f5365d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53681530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f5365d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53681530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55252/0 (socket says 172.21.15.77:55252) 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f5365d9b640 1 -- 172.21.15.77:0/122149849 learned_addr learned my addr 172.21.15.77:0/122149849 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f536659c640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 0x7f53680b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.962+0000 7f5366d9d640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536815dfe0 0x7f536815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5365d9b640 1 -- 172.21.15.77:0/122149849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 msgr2=0x7f53680b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5365d9b640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 0x7f53680b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5365d9b640 1 -- 172.21.15.77:0/122149849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536815dfe0 msgr2=0x7f536815f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5365d9b640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536815dfe0 0x7f536815f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5365d9b640 1 -- 172.21.15.77:0/122149849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536815e710 con 0x7f53680d22b0 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5365d9b640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53681530f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f535c02bee0 tx=0x7f535c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5356ffd640 1 -- 172.21.15.77:0/122149849 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c035040 con 0x7f53680d22b0 2024-03-06T07:23:08.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5356ffd640 1 -- 172.21.15.77:0/122149849 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c03b920 con 0x7f53680d22b0 2024-03-06T07:23:08.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536815e9f0 con 0x7f53680d22b0 2024-03-06T07:23:08.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f5356ffd640 1 -- 172.21.15.77:0/122149849 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c03bd10 con 0x7f53680d22b0 2024-03-06T07:23:08.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.963+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536815efb0 con 0x7f53680d22b0 2024-03-06T07:23:08.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.964+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536804f330 con 0x7f53680d22b0 2024-03-06T07:23:08.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.964+0000 7f5356ffd640 1 -- 172.21.15.77:0/122149849 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f535c038020 con 0x7f53680d22b0 2024-03-06T07:23:08.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.965+0000 7f5356ffd640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.965+0000 7f5356ffd640 1 -- 172.21.15.77:0/122149849 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f535c0ac180 con 0x7f53680d22b0 2024-03-06T07:23:08.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.965+0000 7f536659c640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.965+0000 7f536659c640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5358005880 tx=0x7f5358009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:08.969+0000 7f5356ffd640 1 -- 172.21.15.77:0/122149849 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f535c07a9b0 con 0x7f53680d22b0 2024-03-06T07:23:09.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.175+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f536806c390 con 0x7f53680d22b0 2024-03-06T07:23:09.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.175+0000 7f5356ffd640 1 -- 172.21.15.77:0/122149849 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f535c07ea50 con 0x7f53680d22b0 2024-03-06T07:23:09.177 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:09.177 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:09.178 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 msgr2=0x7f534806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5358005880 tx=0x7f5358009040 comp rx=0 tx=0).stop 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 msgr2=0x7f53681530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53681530f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f535c02bee0 tx=0x7f535c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 shutdown_connections 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53680d22b0 0x7f53681530f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536815dfe0 0x7f536815f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 --2- 172.21.15.77:0/122149849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680b41f0 0x7f53680b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 >> 172.21.15.77:0/122149849 conn(0x7f53680d3cd0 msgr2=0x7f53680b2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 shutdown_connections 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.177+0000 7f536c9b6640 1 -- 172.21.15.77:0/122149849 wait complete. 2024-03-06T07:23:09.190 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:23:09.309 INFO:teuthology.orchestra.run.smithi158.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:23:09.310 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:23:09.310 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:09.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.427+0000 7fe343bb6640 1 Processor -- start 2024-03-06T07:23:09.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.428+0000 7fe343bb6640 1 -- start start 2024-03-06T07:23:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe343bb6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe33c15bdd0 0x7fe33c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe343bb6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe33c0d16e0 0x7fe33c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe341952640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe33c15bdd0 0x7fe33c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe341952640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe33c15bdd0 0x7fe33c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:39734/0 (socket says 172.21.15.77:39734) 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe341952640 1 -- 172.21.15.77:0/2600006365 learned_addr learned my addr 172.21.15.77:0/2600006365 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe343bb6640 1 -- 172.21.15.77:0/2600006365 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33c0cebe0 con 0x7fe33c15fa10 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe343bb6640 1 -- 172.21.15.77:0/2600006365 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33c0ce980 con 0x7fe33c0d16e0 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe343bb6640 1 -- 172.21.15.77:0/2600006365 --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33c0ceec0 con 0x7fe33c15bdd0 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.429+0000 7fe342153640 1 --2- 172.21.15.77:0/2600006365 >> v2:172.21.15.77:3300/0 conn(0x7fe33c0d16e0 0x7fe33c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.430+0000 7fe340950640 1 -- 172.21.15.77:0/2600006365 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222546726 0 0) 0x7fe33c0cebe0 con 0x7fe33c15fa10 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.430+0000 7fe340950640 1 -- 172.21.15.77:0/2600006365 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe328003520 con 0x7fe33c15fa10 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.430+0000 7fe341952640 1 -- 172.21.15.77:0/2600006365 >> v2:172.21.15.77:3300/0 conn(0x7fe33c0d16e0 msgr2=0x7fe33c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.430+0000 7fe341952640 1 --2- 172.21.15.77:0/2600006365 >> v2:172.21.15.77:3300/0 conn(0x7fe33c0d16e0 0x7fe33c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.430+0000 7fe341952640 1 -- 172.21.15.77:0/2600006365 >> v1:172.21.15.77:6789/0 conn(0x7fe33c15fa10 legacy=0x7fe33c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.430+0000 7fe341952640 1 -- 172.21.15.77:0/2600006365 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c0cf1a0 con 0x7fe33c15bdd0 2024-03-06T07:23:09.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.431+0000 7fe341952640 1 --2- 172.21.15.77:0/2600006365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c15bdd0 0x7fe33c0c5610 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe33c06e3c0 tx=0x7fe33402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e17fcb9794467d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.431+0000 7fe340950640 1 -- 172.21.15.77:0/2600006365 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33402ca80 con 0x7fe33c15bdd0 2024-03-06T07:23:09.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.432+0000 7fe340950640 1 -- 172.21.15.77:0/2600006365 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33402cc20 con 0x7fe33c15bdd0 2024-03-06T07:23:09.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.432+0000 7fe340950640 1 -- 172.21.15.77:0/2600006365 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3340343f0 con 0x7fe33c15bdd0 2024-03-06T07:23:09.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.432+0000 7fe343bb6640 1 -- 172.21.15.77:0/2600006365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c15bdd0 msgr2=0x7fe33c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.432+0000 7fe343bb6640 1 --2- 172.21.15.77:0/2600006365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c15bdd0 0x7fe33c0c5610 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe33c06e3c0 tx=0x7fe33402bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.433+0000 7fe343bb6640 1 -- 172.21.15.77:0/2600006365 shutdown_connections 2024-03-06T07:23:09.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.433+0000 7fe343bb6640 1 --2- 172.21.15.77:0/2600006365 >> v2:172.21.15.77:3300/0 conn(0x7fe33c0d16e0 0x7fe33c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.433+0000 7fe343bb6640 1 --2- 172.21.15.77:0/2600006365 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c15bdd0 0x7fe33c0c5610 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.433+0000 7fe343bb6640 1 -- 172.21.15.77:0/2600006365 >> 172.21.15.77:0/2600006365 conn(0x7fe33c09a9f0 msgr2=0x7fe33c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.433+0000 7fe343bb6640 1 -- 172.21.15.77:0/2600006365 shutdown_connections 2024-03-06T07:23:09.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.433+0000 7fe343bb6640 1 -- 172.21.15.77:0/2600006365 wait complete. 2024-03-06T07:23:09.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.435+0000 7fe343bb6640 1 Processor -- start 2024-03-06T07:23:09.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.435+0000 7fe343bb6640 1 -- start start 2024-03-06T07:23:09.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe343bb6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c0d16e0 0x7fe33c15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe343bb6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe33c15bdd0 0x7fe33c0ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe343bb6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33c15fa10 0x7fe33c15cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe343bb6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c0cc5d0 con 0x7fe33c15fa10 2024-03-06T07:23:09.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe343bb6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c0c5b50 con 0x7fe33c0d16e0 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe343bb6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c15c1b0 con 0x7fe33c15bdd0 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe342153640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33c15fa10 0x7fe33c15cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe342153640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33c15fa10 0x7fe33c15cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55278/0 (socket says 172.21.15.77:55278) 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe342153640 1 -- 172.21.15.77:0/1339496191 learned_addr learned my addr 172.21.15.77:0/1339496191 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe341952640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c0d16e0 0x7fe33c15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe341151640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe33c15bdd0 0x7fe33c0ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe342153640 1 -- 172.21.15.77:0/1339496191 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe33c15bdd0 msgr2=0x7fe33c0ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe342153640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe33c15bdd0 0x7fe33c0ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe342153640 1 -- 172.21.15.77:0/1339496191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c0d16e0 msgr2=0x7fe33c15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe342153640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c0d16e0 0x7fe33c15b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.436+0000 7fe342153640 1 -- 172.21.15.77:0/1339496191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c15d740 con 0x7fe33c15fa10 2024-03-06T07:23:09.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.437+0000 7fe342153640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33c15fa10 0x7fe33c15cdf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00e820 tx=0x7fe32c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.437+0000 7fe3327fc640 1 -- 172.21.15.77:0/1339496191 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c017070 con 0x7fe33c15fa10 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.437+0000 7fe3327fc640 1 -- 172.21.15.77:0/1339496191 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32c00ee70 con 0x7fe33c15fa10 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.437+0000 7fe3327fc640 1 -- 172.21.15.77:0/1339496191 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c00cb60 con 0x7fe33c15fa10 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.437+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33c15de20 con 0x7fe33c15fa10 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.437+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33c15e3e0 con 0x7fe33c15fa10 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.439+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33c15e5e0 con 0x7fe33c15fa10 2024-03-06T07:23:09.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.439+0000 7fe3327fc640 1 -- 172.21.15.77:0/1339496191 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe32c01a070 con 0x7fe33c15fa10 2024-03-06T07:23:09.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.440+0000 7fe3327fc640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe32006b550 0x7fe32006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.440+0000 7fe3327fc640 1 -- 172.21.15.77:0/1339496191 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe32c08a180 con 0x7fe33c15fa10 2024-03-06T07:23:09.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.440+0000 7fe341952640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe32006b550 0x7fe32006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.441+0000 7fe341952640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe32006b550 0x7fe32006da10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe334035000 tx=0x7fe334030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.446+0000 7fe3327fc640 1 -- 172.21.15.77:0/1339496191 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe33c15e5e0 con 0x7fe33c15fa10 2024-03-06T07:23:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.673+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe33c04f330 con 0x7fe33c15fa10 2024-03-06T07:23:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.673+0000 7fe3327fc640 1 -- 172.21.15.77:0/1339496191 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fe32c058b20 con 0x7fe33c15fa10 2024-03-06T07:23:09.675 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:09.675 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:09.676 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe32006b550 msgr2=0x7fe32006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe32006b550 0x7fe32006da10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe334035000 tx=0x7fe334030000 comp rx=0 tx=0).stop 2024-03-06T07:23:09.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33c15fa10 msgr2=0x7fe33c15cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33c15fa10 0x7fe33c15cdf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00e820 tx=0x7fe32c00ec20 comp rx=0 tx=0).stop 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 shutdown_connections 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe32006b550 0x7fe32006da10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33c15fa10 0x7fe33c15cdf0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe33c15bdd0 0x7fe33c0ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 --2- 172.21.15.77:0/1339496191 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe33c0d16e0 0x7fe33c15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 >> 172.21.15.77:0/1339496191 conn(0x7fe33c09a9f0 msgr2=0x7fe33c09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 shutdown_connections 2024-03-06T07:23:09.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:09.675+0000 7fe343bb6640 1 -- 172.21.15.77:0/1339496191 wait complete. 2024-03-06T07:23:09.688 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:09.852 INFO:teuthology.orchestra.run.smithi158.stdout:{"mds_cache":{"strays_enqueued":739}} 2024-03-06T07:23:09.853 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 739 } } 2024-03-06T07:23:09.853 DEBUG:tasks.ceph_test_case:wait_until_equal: 739 != 1001, waiting (timeout=600)... 2024-03-06T07:23:14.854 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:14.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.963+0000 7f74e4d1e640 1 Processor -- start 2024-03-06T07:23:14.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.963+0000 7f74e4d1e640 1 -- start start 2024-03-06T07:23:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.964+0000 7f74e4d1e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74e015dfe0 0x7f74e00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.964+0000 7f74e4d1e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 0x7f74e00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.964+0000 7f74e4d1e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e00b3430 con 0x7f74e00d27f0 2024-03-06T07:23:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.964+0000 7f74e4d1e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e00d2c50 con 0x7f74e015dfe0 2024-03-06T07:23:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.964+0000 7f74e4d1e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00b31e0 con 0x7f74e00cf830 2024-03-06T07:23:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.965+0000 7f74de59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74e015dfe0 0x7f74e00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.965+0000 7f74de59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74e015dfe0 0x7f74e00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55284/0 (socket says 172.21.15.77:55284) 2024-03-06T07:23:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.965+0000 7f74de59c640 1 -- 172.21.15.77:0/724209712 learned_addr learned my addr 172.21.15.77:0/724209712 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.965+0000 7f74ddd9b640 1 --2- 172.21.15.77:0/724209712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 0x7f74e00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.965+0000 7f74de59c640 1 -- 172.21.15.77:0/724209712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 msgr2=0x7f74e00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.965+0000 7f74de59c640 1 --2- 172.21.15.77:0/724209712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 0x7f74e00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.965+0000 7f74de59c640 1 -- 172.21.15.77:0/724209712 >> v1:172.21.15.158:6789/0 conn(0x7f74e00d27f0 legacy=0x7f74e00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.965+0000 7f74de59c640 1 -- 172.21.15.77:0/724209712 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00b3710 con 0x7f74e015dfe0 2024-03-06T07:23:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74de59c640 1 --2- 172.21.15.77:0/724209712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cf2f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009e50 tx=0x7f74d002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f149eb8312e5e681 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74dd59a640 1 -- 172.21.15.77:0/724209712 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d002cb00 con 0x7f74e015dfe0 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74dd59a640 1 -- 172.21.15.77:0/724209712 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d002cca0 con 0x7f74e015dfe0 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74dd59a640 1 -- 172.21.15.77:0/724209712 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0034610 con 0x7f74e015dfe0 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74e4d1e640 1 -- 172.21.15.77:0/724209712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 msgr2=0x7f74e00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/724209712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cf2f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009e50 tx=0x7f74d002bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74e4d1e640 1 -- 172.21.15.77:0/724209712 shutdown_connections 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/724209712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 0x7f74e00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/724209712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cf2f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.966+0000 7f74e4d1e640 1 -- 172.21.15.77:0/724209712 >> 172.21.15.77:0/724209712 conn(0x7f74e00d3cd0 msgr2=0x7f74e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.967+0000 7f74e4d1e640 1 -- 172.21.15.77:0/724209712 shutdown_connections 2024-03-06T07:23:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.967+0000 7f74e4d1e640 1 -- 172.21.15.77:0/724209712 wait complete. 2024-03-06T07:23:14.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.968+0000 7f74e4d1e640 1 Processor -- start 2024-03-06T07:23:14.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.968+0000 7f74e4d1e640 1 -- start start 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74e4d1e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 0x7f74e014e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74e4d1e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74e00d27f0 0x7f74e00cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74e4d1e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74e4d1e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00d1c70 con 0x7f74e015dfe0 2024-03-06T07:23:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74e4d1e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00b3d60 con 0x7f74e00d27f0 2024-03-06T07:23:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74e4d1e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00d1df0 con 0x7f74e00cf830 2024-03-06T07:23:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55286/0 (socket says 172.21.15.77:55286) 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 -- 172.21.15.77:0/4283168194 learned_addr learned my addr 172.21.15.77:0/4283168194 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ddd9b640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74e00d27f0 0x7f74e00cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74de59c640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 0x7f74e014e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 -- 172.21.15.77:0/4283168194 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 msgr2=0x7f74e014e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 0x7f74e014e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 -- 172.21.15.77:0/4283168194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74e00d27f0 msgr2=0x7f74e00cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74e00d27f0 0x7f74e00cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 -- 172.21.15.77:0/4283168194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e014e6c0 con 0x7f74e015dfe0 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ded9d640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cd690 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f74d8001eb0 tx=0x7f74d800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ceffd640 1 -- 172.21.15.77:0/4283168194 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8017070 con 0x7f74e015dfe0 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ceffd640 1 -- 172.21.15.77:0/4283168194 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d80052d0 con 0x7f74e015dfe0 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e00b7900 con 0x7f74e015dfe0 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74ceffd640 1 -- 172.21.15.77:0/4283168194 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8003810 con 0x7f74e015dfe0 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.969+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e00b7ec0 con 0x7f74e015dfe0 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.970+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e00ce3c0 con 0x7f74e015dfe0 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.971+0000 7f74ceffd640 1 -- 172.21.15.77:0/4283168194 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74d801a070 con 0x7f74e015dfe0 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.971+0000 7f74ceffd640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f74c006b4d0 0x7f74c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.971+0000 7f74ceffd640 1 -- 172.21.15.77:0/4283168194 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74d80894f0 con 0x7f74e015dfe0 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.971+0000 7f74de59c640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f74c006b4d0 0x7f74c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.972+0000 7f74de59c640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f74c006b4d0 0x7f74c006d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f74d0035000 tx=0x7f74d0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:14.976+0000 7f74ceffd640 1 -- 172.21.15.77:0/4283168194 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74e00ce3c0 con 0x7f74e015dfe0 2024-03-06T07:23:15.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.181+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74e00cf5f0 con 0x7f74e015dfe0 2024-03-06T07:23:15.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.182+0000 7f74ceffd640 1 -- 172.21.15.77:0/4283168194 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f74e00cf5f0 con 0x7f74e015dfe0 2024-03-06T07:23:15.184 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:15.184 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:15.184 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f74c006b4d0 msgr2=0x7f74c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f74c006b4d0 0x7f74c006d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f74d0035000 tx=0x7f74d0030000 comp rx=0 tx=0).stop 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 msgr2=0x7f74e00cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cd690 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f74d8001eb0 tx=0x7f74d800c890 comp rx=0 tx=0).stop 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 shutdown_connections 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f74c006b4d0 0x7f74c006d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e015dfe0 0x7f74e00cd690 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f74e00d27f0 0x7f74e00cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 --2- 172.21.15.77:0/4283168194 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f74e00cf830 0x7f74e014e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 >> 172.21.15.77:0/4283168194 conn(0x7f74e00d3cd0 msgr2=0x7f74e006ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 shutdown_connections 2024-03-06T07:23:15.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.184+0000 7f74e4d1e640 1 -- 172.21.15.77:0/4283168194 wait complete. 2024-03-06T07:23:15.197 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:15.316 INFO:teuthology.orchestra.run.smithi158.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:23:15.316 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:23:15.317 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.416+0000 7f14ce9e9640 1 Processor -- start 2024-03-06T07:23:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.416+0000 7f14ce9e9640 1 -- start start 2024-03-06T07:23:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14ce9e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f14c806da40 0x7f14c80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14ce9e9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c80cc980 con 0x7f14c80cd7a0 2024-03-06T07:23:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14ce9e9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c80ccc60 con 0x7f14c81424f0 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14ce9e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c80cd220 con 0x7f14c806da40 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14c77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f14c806da40 0x7f14c80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14ccf86640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f14c80cd7a0 0x7f14c8071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40870/0 (socket says 172.21.15.77:40870) 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14ccf86640 1 -- 172.21.15.77:0/3777325157 learned_addr learned my addr 172.21.15.77:0/3777325157 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14c77fe640 1 --2- 172.21.15.77:0/3777325157 >> v2:172.21.15.77:3300/0 conn(0x7f14c806da40 0x7f14c80cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55288/0 (socket says 172.21.15.77:55288) 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14c77fe640 1 -- 172.21.15.77:0/3777325157 >> v1:172.21.15.158:6789/0 conn(0x7f14c81424f0 legacy=0x7f14c806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14c77fe640 1 -- 172.21.15.77:0/3777325157 >> v1:172.21.15.77:6789/0 conn(0x7f14c80cd7a0 legacy=0x7f14c8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14c6ffd640 1 -- 172.21.15.77:0/3777325157 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130780972 0 0) 0x7f14c80cc980 con 0x7f14c80cd7a0 2024-03-06T07:23:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.417+0000 7f14c77fe640 1 -- 172.21.15.77:0/3777325157 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c80ccf40 con 0x7f14c806da40 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.418+0000 7f14c77fe640 1 --2- 172.21.15.77:0/3777325157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c806da40 0x7f14c80cc2f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0097a0 tx=0x7f14bc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623e19692cf5c05e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.418+0000 7f14c6ffd640 1 -- 172.21.15.77:0/3777325157 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc034070 con 0x7f14c806da40 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.418+0000 7f14c6ffd640 1 -- 172.21.15.77:0/3777325157 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc030610 con 0x7f14c806da40 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.418+0000 7f14c6ffd640 1 -- 172.21.15.77:0/3777325157 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0309e0 con 0x7f14c806da40 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.419+0000 7f14ce9e9640 1 -- 172.21.15.77:0/3777325157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c806da40 msgr2=0x7f14c80cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.419+0000 7f14ce9e9640 1 --2- 172.21.15.77:0/3777325157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c806da40 0x7f14c80cc2f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0097a0 tx=0x7f14bc02b960 comp rx=0 tx=0).stop 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.419+0000 7f14ce9e9640 1 -- 172.21.15.77:0/3777325157 shutdown_connections 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.419+0000 7f14ce9e9640 1 --2- 172.21.15.77:0/3777325157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c806da40 0x7f14c80cc2f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.419+0000 7f14ce9e9640 1 -- 172.21.15.77:0/3777325157 >> 172.21.15.77:0/3777325157 conn(0x7f14c80cf8a0 msgr2=0x7f14c80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.419+0000 7f14ce9e9640 1 -- 172.21.15.77:0/3777325157 shutdown_connections 2024-03-06T07:23:15.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.419+0000 7f14ce9e9640 1 -- 172.21.15.77:0/3777325157 wait complete. 2024-03-06T07:23:15.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.420+0000 7f14ce9e9640 1 Processor -- start 2024-03-06T07:23:15.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.420+0000 7f14ce9e9640 1 -- start start 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ce9e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14c806da40 0x7f14c80af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ce9e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14c80cd7a0 0x7f14c80b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ce9e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c81424f0 0x7f14c8135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ce9e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c815d070 con 0x7f14c81424f0 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ce9e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c815cef0 con 0x7f14c806da40 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ce9e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c815d1f0 con 0x7f14c80cd7a0 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c81424f0 0x7f14c8135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c81424f0 0x7f14c8135100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55300/0 (socket says 172.21.15.77:55300) 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 -- 172.21.15.77:0/2722940947 learned_addr learned my addr 172.21.15.77:0/2722940947 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14c7fff640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14c806da40 0x7f14c80af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14c77fe640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14c80cd7a0 0x7f14c80b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 -- 172.21.15.77:0/2722940947 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14c80cd7a0 msgr2=0x7f14c80b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14c80cd7a0 0x7f14c80b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 -- 172.21.15.77:0/2722940947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14c806da40 msgr2=0x7f14c80af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14c806da40 0x7f14c80af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 -- 172.21.15.77:0/2722940947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c8135640 con 0x7f14c81424f0 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.421+0000 7f14ccf86640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c81424f0 0x7f14c8135100 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f14c000c7a0 tx=0x7f14c000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.422+0000 7f14c4ff9640 1 -- 172.21.15.77:0/2722940947 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c0018070 con 0x7f14c81424f0 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.422+0000 7f14c4ff9640 1 -- 172.21.15.77:0/2722940947 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c000a640 con 0x7f14c81424f0 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.422+0000 7f14c4ff9640 1 -- 172.21.15.77:0/2722940947 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c000aa70 con 0x7f14c81424f0 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.422+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c80afd10 con 0x7f14c81424f0 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.422+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c80b02d0 con 0x7f14c81424f0 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.423+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c804f330 con 0x7f14c81424f0 2024-03-06T07:23:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.423+0000 7f14c4ff9640 1 -- 172.21.15.77:0/2722940947 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14c001b070 con 0x7f14c81424f0 2024-03-06T07:23:15.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.424+0000 7f14c4ff9640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.424+0000 7f14c7fff640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.424+0000 7f14c4ff9640 1 -- 172.21.15.77:0/2722940947 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14c00897e0 con 0x7f14c81424f0 2024-03-06T07:23:15.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.424+0000 7f14c7fff640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14b406b4d0 0x7f14b406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000f60 tx=0x7f14b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.428+0000 7f14c4ff9640 1 -- 172.21.15.77:0/2722940947 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14c0057f80 con 0x7f14c81424f0 2024-03-06T07:23:15.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.638+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14c80d0620 con 0x7f14c81424f0 2024-03-06T07:23:15.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.639+0000 7f14c4ff9640 1 -- 172.21.15.77:0/2722940947 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f14c005c020 con 0x7f14c81424f0 2024-03-06T07:23:15.641 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:15.641 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:15.641 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.640+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14b406b4d0 msgr2=0x7f14b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.640+0000 7f14ce9e9640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14b406b4d0 0x7f14b406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000f60 tx=0x7f14b8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.640+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c81424f0 msgr2=0x7f14c8135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.640+0000 7f14ce9e9640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c81424f0 0x7f14c8135100 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f14c000c7a0 tx=0x7f14c000cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.640+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 shutdown_connections 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.640+0000 7f14ce9e9640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.641+0000 7f14ce9e9640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14c81424f0 0x7f14c8135100 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.641+0000 7f14ce9e9640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14c80cd7a0 0x7f14c80b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.641+0000 7f14ce9e9640 1 --2- 172.21.15.77:0/2722940947 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14c806da40 0x7f14c80af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.641+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 >> 172.21.15.77:0/2722940947 conn(0x7f14c80cf8a0 msgr2=0x7f14c80ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.641+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 shutdown_connections 2024-03-06T07:23:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.641+0000 7f14ce9e9640 1 -- 172.21.15.77:0/2722940947 wait complete. 2024-03-06T07:23:15.653 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:23:15.769 INFO:teuthology.orchestra.run.smithi158.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:23:15.770 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:23:15.770 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:15.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.809+0000 7f43f9a35f40 1 Processor -- start 2024-03-06T07:23:15.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.809+0000 7f43f9a35f40 1 -- start start 2024-03-06T07:23:15.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f9a35f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 0x55f77e4ecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f9a35f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 0x55f77e4ea260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f9a35f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e4ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f9a35f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f77e3d9b70 con 0x55f77e47bdc0 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f9a35f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f77e4330f0 con 0x55f77e47b9e0 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f9a35f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f77e4eafc0 con 0x55f77e432590 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e4ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f65bf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 0x55f77e4ea260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f6dc0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 0x55f77e4ecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e4ea7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55302/0 (socket says 172.21.15.77:55302) 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f65bf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 0x55f77e4ea260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:39792/0 (socket says 172.21.15.77:39792) 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 -- 172.21.15.77:0/279557230 learned_addr learned my addr 172.21.15.77:0/279557230 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:15.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f6dc0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 0x55f77e4ecbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:60588/0 (socket says 172.21.15.77:60588) 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 -- 172.21.15.77:0/279557230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 msgr2=0x55f77e4ecbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 --2- 172.21.15.77:0/279557230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 0x55f77e4ecbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 -- 172.21.15.77:0/279557230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 msgr2=0x55f77e4ea260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 --2- 172.21.15.77:0/279557230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 0x55f77e4ea260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 -- 172.21.15.77:0/279557230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f77e4eef40 con 0x55f77e47bdc0 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f75c1640 1 --2- 172.21.15.77:0/279557230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e4ea7a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f43f0001d60 tx=0x7f43f0001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5137e81519a493 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.810+0000 7f43f55bd640 1 -- 172.21.15.77:0/279557230 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0008030 con 0x55f77e47bdc0 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f55bd640 1 -- 172.21.15.77:0/279557230 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f0009030 con 0x55f77e47bdc0 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f55bd640 1 -- 172.21.15.77:0/279557230 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0005070 con 0x55f77e47bdc0 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 -- 172.21.15.77:0/279557230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 msgr2=0x55f77e4ea7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 --2- 172.21.15.77:0/279557230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e4ea7a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f43f0001d60 tx=0x7f43f0001e50 comp rx=0 tx=0).stop 2024-03-06T07:23:15.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 -- 172.21.15.77:0/279557230 shutdown_connections 2024-03-06T07:23:15.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 --2- 172.21.15.77:0/279557230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e4ea7a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 --2- 172.21.15.77:0/279557230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 0x55f77e4ea260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 --2- 172.21.15.77:0/279557230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 0x55f77e4ecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 -- 172.21.15.77:0/279557230 >> 172.21.15.77:0/279557230 conn(0x55f77e45e350 msgr2=0x55f77e461910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 -- 172.21.15.77:0/279557230 shutdown_connections 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.811+0000 7f43f9a35f40 1 -- 172.21.15.77:0/279557230 wait complete. 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f9a35f40 1 Processor -- start 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f9a35f40 1 -- start start 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f9a35f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 0x55f77e45fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f9a35f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 0x55f77e508400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f9a35f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e506690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f9a35f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f77e4ef480 con 0x55f77e47bdc0 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f9a35f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f77e4ef1e0 con 0x55f77e47b9e0 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f9a35f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f77e460e20 con 0x55f77e432590 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f75c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e506690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f75c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e506690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55318/0 (socket says 172.21.15.77:55318) 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f75c1640 1 -- 172.21.15.77:0/719171588 learned_addr learned my addr 172.21.15.77:0/719171588 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f6dc0640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 0x55f77e45fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.812+0000 7f43f65bf640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 0x55f77e508400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43f75c1640 1 -- 172.21.15.77:0/719171588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 msgr2=0x55f77e45fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43f75c1640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f77e432590 0x55f77e45fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43f75c1640 1 -- 172.21.15.77:0/719171588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 msgr2=0x55f77e508400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43f75c1640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f77e47b9e0 0x55f77e508400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43f75c1640 1 -- 172.21.15.77:0/719171588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f77e508b30 con 0x55f77e47bdc0 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43f75c1640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e506690 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f43f0001c50 tx=0x7f43f0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43effff640 1 -- 172.21.15.77:0/719171588 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0016640 con 0x55f77e47bdc0 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43effff640 1 -- 172.21.15.77:0/719171588 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f0016800 con 0x55f77e47bdc0 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f77e508e10 con 0x55f77e47bdc0 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43effff640 1 -- 172.21.15.77:0/719171588 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0016bf0 con 0x55f77e47bdc0 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.813+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f77e506bd0 con 0x55f77e47bdc0 2024-03-06T07:23:15.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.814+0000 7f43effff640 1 -- 172.21.15.77:0/719171588 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43f0020070 con 0x55f77e47bdc0 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.815+0000 7f43effff640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43d8069eb0 0x7f43d806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.815+0000 7f43effff640 1 -- 172.21.15.77:0/719171588 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43f008a490 con 0x55f77e47bdc0 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.815+0000 7f43f6dc0640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43d8069eb0 0x7f43d806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.815+0000 7f43f9a35f40 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x55f77e5071e0 0x55f77e5075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.815+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x55f77e507b00 con 0x55f77e5071e0 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.815+0000 7f43f65bf640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x55f77e5071e0 0x55f77e5075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.815+0000 7f43f6dc0640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43d8069eb0 0x7f43d806c370 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f43e8005fd0 tx=0x7f43e8007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.815+0000 7f43f65bf640 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x55f77e5071e0 0x55f77e5075c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f65bf640 1 -- 172.21.15.77:0/719171588 <== osd.3 v2:172.21.15.77:6816/461341877 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f43e005a040 con 0x55f77e5071e0 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x55f77e5071e0 msgr2=0x55f77e5075c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x55f77e5071e0 0x55f77e5075c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43d8069eb0 msgr2=0x7f43d806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43d8069eb0 0x7f43d806c370 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f43e8005fd0 tx=0x7f43e8007040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 msgr2=0x55f77e506690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 --2- 172.21.15.77:0/719171588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77e47bdc0 0x55f77e506690 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f43f0001c50 tx=0x7f43f0000ed0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f75c1640 1 -- 172.21.15.77:0/719171588 reap_dead start 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 shutdown_connections 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 >> 172.21.15.77:0/719171588 conn(0x55f77e45e350 msgr2=0x55f77e4ea260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 shutdown_connections 2024-03-06T07:23:15.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.816+0000 7f43f9a35f40 1 -- 172.21.15.77:0/719171588 wait complete. 2024-03-06T07:23:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:15.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:23:15.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.965+0000 7f2950042640 1 Processor -- start 2024-03-06T07:23:15.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.965+0000 7f2950042640 1 -- start start 2024-03-06T07:23:15.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f2950042640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2948071e00 0x7f29480cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f2950042640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f29480cd7c0 0x7f29480cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f2950042640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29480ce120 con 0x7f294815bbf0 2024-03-06T07:23:15.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f2950042640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2948072250 con 0x7f2948071e00 2024-03-06T07:23:15.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f2950042640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29480cf100 con 0x7f29480cd7c0 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294d5dd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2948071e00 0x7f29480cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294ddde640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f294815bbf0 0x7f294815bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40886/0 (socket says 172.21.15.77:40886) 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294ddde640 1 -- 172.21.15.77:0/1719288204 learned_addr learned my addr 172.21.15.77:0/1719288204 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294e5df640 1 --2- 172.21.15.77:0/1719288204 >> v2:172.21.15.158:3300/0 conn(0x7f29480cd7c0 0x7f29480cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294d5dd640 1 -- 172.21.15.77:0/1719288204 >> v2:172.21.15.158:3300/0 conn(0x7f29480cd7c0 msgr2=0x7f29480cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294d5dd640 1 --2- 172.21.15.77:0/1719288204 >> v2:172.21.15.158:3300/0 conn(0x7f29480cd7c0 0x7f29480cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294d5dd640 1 -- 172.21.15.77:0/1719288204 >> v1:172.21.15.77:6789/0 conn(0x7f294815bbf0 legacy=0x7f294815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294d5dd640 1 -- 172.21.15.77:0/1719288204 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948072490 con 0x7f2948071e00 2024-03-06T07:23:15.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.966+0000 7f294cddc640 1 -- 172.21.15.77:0/1719288204 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170171715 0 0) 0x7f29480ce120 con 0x7f294815bbf0 2024-03-06T07:23:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.967+0000 7f294d5dd640 1 --2- 172.21.15.77:0/1719288204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2948071e00 0x7f29480cebc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f294806ec30 tx=0x7f293802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c71095fd07a9da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.967+0000 7f294cddc640 1 -- 172.21.15.77:0/1719288204 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938035020 con 0x7f2948071e00 2024-03-06T07:23:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.967+0000 7f294cddc640 1 -- 172.21.15.77:0/1719288204 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938031830 con 0x7f2948071e00 2024-03-06T07:23:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.967+0000 7f294cddc640 1 -- 172.21.15.77:0/1719288204 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938031c00 con 0x7f2948071e00 2024-03-06T07:23:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.968+0000 7f2950042640 1 -- 172.21.15.77:0/1719288204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2948071e00 msgr2=0x7f29480cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.968+0000 7f2950042640 1 --2- 172.21.15.77:0/1719288204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2948071e00 0x7f29480cebc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f294806ec30 tx=0x7f293802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.968+0000 7f2950042640 1 -- 172.21.15.77:0/1719288204 shutdown_connections 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.968+0000 7f2950042640 1 --2- 172.21.15.77:0/1719288204 >> v2:172.21.15.158:3300/0 conn(0x7f29480cd7c0 0x7f29480cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.968+0000 7f2950042640 1 --2- 172.21.15.77:0/1719288204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2948071e00 0x7f29480cebc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.968+0000 7f2950042640 1 -- 172.21.15.77:0/1719288204 >> 172.21.15.77:0/1719288204 conn(0x7f29480b1c80 msgr2=0x7f29480cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.968+0000 7f2950042640 1 -- 172.21.15.77:0/1719288204 shutdown_connections 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.968+0000 7f2950042640 1 -- 172.21.15.77:0/1719288204 wait complete. 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.969+0000 7f2950042640 1 Processor -- start 2024-03-06T07:23:15.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.969+0000 7f2950042640 1 -- start start 2024-03-06T07:23:15.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f2950042640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2948071e00 0x7f294814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f2950042640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29480cd7c0 0x7f29480cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f2950042640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294815bbf0 0x7f2948154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f2950042640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294815c6e0 con 0x7f294815bbf0 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f2950042640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29480d0650 con 0x7f29480cd7c0 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f2950042640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29480b3680 con 0x7f2948071e00 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294815bbf0 0x7f2948154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294815bbf0 0x7f2948154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55348/0 (socket says 172.21.15.77:55348) 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 -- 172.21.15.77:0/38400180 learned_addr learned my addr 172.21.15.77:0/38400180 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294d5dd640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29480cd7c0 0x7f29480cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294ddde640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2948071e00 0x7f294814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 -- 172.21.15.77:0/38400180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2948071e00 msgr2=0x7f294814f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2948071e00 0x7f294814f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 -- 172.21.15.77:0/38400180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29480cd7c0 msgr2=0x7f29480cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29480cd7c0 0x7f29480cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 -- 172.21.15.77:0/38400180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29481f2130 con 0x7f294815bbf0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f294e5df640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294815bbf0 0x7f2948154c00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f294000efc0 tx=0x7f294000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f293e7fc640 1 -- 172.21.15.77:0/38400180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940018020 con 0x7f294815bbf0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f293e7fc640 1 -- 172.21.15.77:0/38400180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940014480 con 0x7f294815bbf0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f293e7fc640 1 -- 172.21.15.77:0/38400180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940014870 con 0x7f294815bbf0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29481f23f0 con 0x7f294815bbf0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.970+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29481f29b0 con 0x7f294815bbf0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.971+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294804f330 con 0x7f294815bbf0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.972+0000 7f293e7fc640 1 -- 172.21.15.77:0/38400180 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2940005d10 con 0x7f294815bbf0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.972+0000 7f293e7fc640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f292c06b4d0 0x7f292c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.972+0000 7f294ddde640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f292c06b4d0 0x7f292c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.972+0000 7f293e7fc640 1 -- 172.21.15.77:0/38400180 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f294008abe0 con 0x7f294815bbf0 2024-03-06T07:23:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.973+0000 7f294ddde640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f292c06b4d0 0x7f292c06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2944007fd0 tx=0x7f2944007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:15.977+0000 7f293e7fc640 1 -- 172.21.15.77:0/38400180 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2940022070 con 0x7f294815bbf0 2024-03-06T07:23:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.217+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f29480cf810 con 0x7f294815bbf0 2024-03-06T07:23:16.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.217+0000 7f293e7fc640 1 -- 172.21.15.77:0/38400180 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1658 (secure 0 0 0) 0x7f2940010040 con 0x7f294815bbf0 2024-03-06T07:23:16.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:16.219 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "stats": { 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "total_avail_bytes": 772818993152, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_bytes": 275120128, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_raw_bytes": 275120128, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_raw_ratio": 0.00035586886224336922, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "num_osds": 8, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "num_per_pool_osds": 8, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "stats_by_class": { 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "ssd": { 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:16.220 INFO:teuthology.orchestra.run.smithi077.stdout: "total_avail_bytes": 772818993152, 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_bytes": 275120128, 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_raw_bytes": 275120128, 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_raw_ratio": 0.00035586886224336922 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "pools": [ 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "name": ".mgr", 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 1, 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "stats": { 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "stored": 590368, 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "objects": 2, 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "kb_used": 1160, 2024-03-06T07:23:16.221 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_used": 1187840, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "percent_used": 1.6180464399440098e-06, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "max_avail": 367059337216 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephfs_metadata", 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "stats": { 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "stored": 3520234, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "objects": 23, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "kb_used": 6936, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_used": 7102464, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "percent_used": 9.6747235147631727e-06, 2024-03-06T07:23:16.222 INFO:teuthology.orchestra.run.smithi077.stdout: "max_avail": 367059337216 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephfs_data", 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5, 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "stats": { 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "stored": 0, 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "objects": 0, 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "kb_used": 0, 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_used": 0, 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "percent_used": 0, 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: "max_avail": 367059337216 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:16.223 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:23:16.224 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:23:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f292c06b4d0 msgr2=0x7f292c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f292c06b4d0 0x7f292c06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2944007fd0 tx=0x7f2944007de0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294815bbf0 msgr2=0x7f2948154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294815bbf0 0x7f2948154c00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f294000efc0 tx=0x7f294000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 shutdown_connections 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f292c06b4d0 0x7f292c06d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294815bbf0 0x7f2948154c00 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29480cd7c0 0x7f29480cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 --2- 172.21.15.77:0/38400180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2948071e00 0x7f294814f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 >> 172.21.15.77:0/38400180 conn(0x7f29480b1c80 msgr2=0x7f29481579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 shutdown_connections 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.219+0000 7f2950042640 1 -- 172.21.15.77:0/38400180 wait complete. 2024-03-06T07:23:16.232 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:23:16.232 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:16.232 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:16.283 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:16.323 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:16.323 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:23:16.323 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:16.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.477+0000 7f43bc064640 1 Processor -- start 2024-03-06T07:23:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.477+0000 7f43bc064640 1 -- start start 2024-03-06T07:23:16.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43bc064640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f43b40d02c0 0x7f43b40cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43bc064640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f43b40b6550 0x7f43b40cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43bc064640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b415c620 con 0x7f43b40d06a0 2024-03-06T07:23:16.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43bc064640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40b50b0 con 0x7f43b40d02c0 2024-03-06T07:23:16.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43bc064640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40cd800 con 0x7f43b40b6550 2024-03-06T07:23:16.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43b9e00640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f43b40d02c0 0x7f43b40cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43b95ff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f43b40d06a0 0x7f43b40ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40892/0 (socket says 172.21.15.77:40892) 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43b95ff640 1 -- 172.21.15.77:0/1041251737 learned_addr learned my addr 172.21.15.77:0/1041251737 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43b9e00640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f43b40d02c0 0x7f43b40cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55356/0 (socket says 172.21.15.77:55356) 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.478+0000 7f43ba601640 1 --2- 172.21.15.77:0/1041251737 >> v2:172.21.15.158:3300/0 conn(0x7f43b40b6550 0x7f43b40cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b9e00640 1 -- 172.21.15.77:0/1041251737 >> v2:172.21.15.158:3300/0 conn(0x7f43b40b6550 msgr2=0x7f43b40cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b9e00640 1 --2- 172.21.15.77:0/1041251737 >> v2:172.21.15.158:3300/0 conn(0x7f43b40b6550 0x7f43b40cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b9e00640 1 -- 172.21.15.77:0/1041251737 >> v1:172.21.15.77:6789/0 conn(0x7f43b40d06a0 legacy=0x7f43b40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b9e00640 1 -- 172.21.15.77:0/1041251737 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40b4a60 con 0x7f43b40d02c0 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b8dfe640 1 -- 172.21.15.77:0/1041251737 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922996103 0 0) 0x7f43b415c620 con 0x7f43b40d06a0 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43ba601640 1 --2- 172.21.15.77:0/1041251737 >> v2:172.21.15.158:3300/0 conn(0x7f43b40b6550 0x7f43b40cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b9e00640 1 --2- 172.21.15.77:0/1041251737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b40cf850 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f43b40cc050 tx=0x7f43b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b60f60907ee1756 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b8dfe640 1 -- 172.21.15.77:0/1041251737 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b0035020 con 0x7f43b40d02c0 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b8dfe640 1 -- 172.21.15.77:0/1041251737 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b0031830 con 0x7f43b40d02c0 2024-03-06T07:23:16.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.479+0000 7f43b8dfe640 1 -- 172.21.15.77:0/1041251737 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b0031c00 con 0x7f43b40d02c0 2024-03-06T07:23:16.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.480+0000 7f43bc064640 1 -- 172.21.15.77:0/1041251737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 msgr2=0x7f43b40cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.480+0000 7f43bc064640 1 --2- 172.21.15.77:0/1041251737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b40cf850 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f43b40cc050 tx=0x7f43b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.480+0000 7f43bc064640 1 -- 172.21.15.77:0/1041251737 shutdown_connections 2024-03-06T07:23:16.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.480+0000 7f43bc064640 1 --2- 172.21.15.77:0/1041251737 >> v2:172.21.15.158:3300/0 conn(0x7f43b40b6550 0x7f43b40cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.480+0000 7f43bc064640 1 --2- 172.21.15.77:0/1041251737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b40cf850 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.480+0000 7f43bc064640 1 -- 172.21.15.77:0/1041251737 >> 172.21.15.77:0/1041251737 conn(0x7f43b40b35a0 msgr2=0x7f43b40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.480+0000 7f43bc064640 1 -- 172.21.15.77:0/1041251737 shutdown_connections 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.480+0000 7f43bc064640 1 -- 172.21.15.77:0/1041251737 wait complete. 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.481+0000 7f43bc064640 1 Processor -- start 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.481+0000 7f43bc064640 1 -- start start 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43bc064640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b40b6550 0x7f43b4156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43bc064640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b4159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43bc064640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43b40d06a0 0x7f43b40c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43bc064640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40ce9c0 con 0x7f43b40d02c0 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43bc064640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40ce840 con 0x7f43b40d06a0 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43bc064640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40ceb40 con 0x7f43b40b6550 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b95ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b4159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b95ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b4159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55368/0 (socket says 172.21.15.77:55368) 2024-03-06T07:23:16.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b95ff640 1 -- 172.21.15.77:0/3577642864 learned_addr learned my addr 172.21.15.77:0/3577642864 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43ba601640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43b40d06a0 0x7f43b40c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b9e00640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b40b6550 0x7f43b4156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b95ff640 1 -- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b40b6550 msgr2=0x7f43b4156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b95ff640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b40b6550 0x7f43b4156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b95ff640 1 -- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43b40d06a0 msgr2=0x7f43b40c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b95ff640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43b40d06a0 0x7f43b40c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b95ff640 1 -- 172.21.15.77:0/3577642864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b415a4f0 con 0x7f43b40d02c0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.482+0000 7f43b9e00640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b40b6550 0x7f43b4156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.483+0000 7f43b95ff640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b4159bf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f43a400c430 tx=0x7f43a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.483+0000 7f43aa7fc640 1 -- 172.21.15.77:0/3577642864 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40074e0 con 0x7f43b40d02c0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.483+0000 7f43aa7fc640 1 -- 172.21.15.77:0/3577642864 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a40047e0 con 0x7f43b40d02c0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.483+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b415a7d0 con 0x7f43b40d02c0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.483+0000 7f43aa7fc640 1 -- 172.21.15.77:0/3577642864 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4004bd0 con 0x7f43b40d02c0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.483+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b40c5640 con 0x7f43b40d02c0 2024-03-06T07:23:16.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.484+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b404f330 con 0x7f43b40d02c0 2024-03-06T07:23:16.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.484+0000 7f43aa7fc640 1 -- 172.21.15.77:0/3577642864 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43a4011070 con 0x7f43b40d02c0 2024-03-06T07:23:16.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.485+0000 7f43aa7fc640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.485+0000 7f43aa7fc640 1 -- 172.21.15.77:0/3577642864 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43a4089e40 con 0x7f43b40d02c0 2024-03-06T07:23:16.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.485+0000 7f43b9e00640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.485+0000 7f43b9e00640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439806b4d0 0x7f439806d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f43b0036000 tx=0x7f43b0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.489+0000 7f43aa7fc640 1 -- 172.21.15.77:0/3577642864 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43a4058c40 con 0x7f43b40d02c0 2024-03-06T07:23:16.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.660+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43b40cf3d0 con 0x7f43b40d02c0 2024-03-06T07:23:16.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.661+0000 7f43aa7fc640 1 -- 172.21.15.77:0/3577642864 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f43a4058540 con 0x7f43b40d02c0 2024-03-06T07:23:16.663 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:16.663 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:58.329001+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:54.903388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.652222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:16.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439806b4d0 msgr2=0x7f439806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439806b4d0 0x7f439806d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f43b0036000 tx=0x7f43b0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:16.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 msgr2=0x7f43b4159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b4159bf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f43a400c430 tx=0x7f43a400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 shutdown_connections 2024-03-06T07:23:16.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b40d02c0 0x7f43b4159bf0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43b40d06a0 0x7f43b40c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 --2- 172.21.15.77:0/3577642864 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b40b6550 0x7f43b4156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 >> 172.21.15.77:0/3577642864 conn(0x7f43b40b35a0 msgr2=0x7f43b4159450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 shutdown_connections 2024-03-06T07:23:16.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.663+0000 7f43bc064640 1 -- 172.21.15.77:0/3577642864 wait complete. 2024-03-06T07:23:16.676 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:54.903388+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:55.652222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:16.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:16.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.778+0000 7f08e2a39640 1 Processor -- start 2024-03-06T07:23:16.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.778+0000 7f08e2a39640 1 -- start start 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.778+0000 7f08e2a39640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f08dc1614d0 0x7f08dc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.778+0000 7f08e2a39640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 0x7f08dc0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.778+0000 7f08e2a39640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08dc0d2740 con 0x7f08dc0d22b0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.778+0000 7f08e2a39640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08dc1618b0 con 0x7f08dc1614d0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.778+0000 7f08e2a39640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08dc0cf400 con 0x7f08dc0d1910 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.779+0000 7f08dbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f08dc1614d0 0x7f08dc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.779+0000 7f08dbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f08dc1614d0 0x7f08dc0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55376/0 (socket says 172.21.15.77:55376) 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.779+0000 7f08dbfff640 1 -- 172.21.15.77:0/1329378636 learned_addr learned my addr 172.21.15.77:0/1329378636 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.779+0000 7f08db7fe640 1 --2- 172.21.15.77:0/1329378636 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 0x7f08dc0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.779+0000 7f08dbfff640 1 -- 172.21.15.77:0/1329378636 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 msgr2=0x7f08dc0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.779+0000 7f08dbfff640 1 --2- 172.21.15.77:0/1329378636 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 0x7f08dc0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.779+0000 7f08dbfff640 1 -- 172.21.15.77:0/1329378636 >> v1:172.21.15.158:6789/0 conn(0x7f08dc0d22b0 legacy=0x7f08dc0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.779+0000 7f08dbfff640 1 -- 172.21.15.77:0/1329378636 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc0cf6a0 con 0x7f08dc1614d0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08dbfff640 1 --2- 172.21.15.77:0/1329378636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc1614d0 0x7f08dc0d4860 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009870 tx=0x7f08cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b40bd7b21b4d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08daffd640 1 -- 172.21.15.77:0/1329378636 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc034070 con 0x7f08dc1614d0 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08daffd640 1 -- 172.21.15.77:0/1329378636 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc030840 con 0x7f08dc1614d0 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08daffd640 1 -- 172.21.15.77:0/1329378636 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc030c10 con 0x7f08dc1614d0 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08e2a39640 1 -- 172.21.15.77:0/1329378636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc1614d0 msgr2=0x7f08dc0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08e2a39640 1 --2- 172.21.15.77:0/1329378636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc1614d0 0x7f08dc0d4860 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009870 tx=0x7f08cc02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08e2a39640 1 -- 172.21.15.77:0/1329378636 shutdown_connections 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08e2a39640 1 --2- 172.21.15.77:0/1329378636 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 0x7f08dc0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08e2a39640 1 --2- 172.21.15.77:0/1329378636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc1614d0 0x7f08dc0d4860 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08e2a39640 1 -- 172.21.15.77:0/1329378636 >> 172.21.15.77:0/1329378636 conn(0x7f08dc0d3d40 msgr2=0x7f08dc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08e2a39640 1 -- 172.21.15.77:0/1329378636 shutdown_connections 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.780+0000 7f08e2a39640 1 -- 172.21.15.77:0/1329378636 wait complete. 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.781+0000 7f08e2a39640 1 Processor -- start 2024-03-06T07:23:16.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.782+0000 7f08e2a39640 1 -- start start 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.782+0000 7f08e2a39640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 0x7f08dc1f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.782+0000 7f08e2a39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc0d22b0 0x7f08dc1f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.782+0000 7f08e2a39640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08dc1614d0 0x7f08dc0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.782+0000 7f08e2a39640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08dc0b0910 con 0x7f08dc0d22b0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.782+0000 7f08e2a39640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08dc0cfc60 con 0x7f08dc1614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.782+0000 7f08e2a39640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08dc0b1e40 con 0x7f08dc0d1910 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.782+0000 7f08db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc0d22b0 0x7f08dc1f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc0d22b0 0x7f08dc1f10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55386/0 (socket says 172.21.15.77:55386) 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08db7fe640 1 -- 172.21.15.77:0/3904916576 learned_addr learned my addr 172.21.15.77:0/3904916576 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08dbfff640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 0x7f08dc1f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08e0fd6640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08dc1614d0 0x7f08dc0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08db7fe640 1 -- 172.21.15.77:0/3904916576 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 msgr2=0x7f08dc1f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08db7fe640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 0x7f08dc1f46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08db7fe640 1 -- 172.21.15.77:0/3904916576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08dc1614d0 msgr2=0x7f08dc0b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08db7fe640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08dc1614d0 0x7f08dc0b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08db7fe640 1 -- 172.21.15.77:0/3904916576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc1f4be0 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08db7fe640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc0d22b0 0x7f08dc1f10a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f08d000e9b0 tx=0x7f08d000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08d8ff9640 1 -- 172.21.15.77:0/3904916576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0017070 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08d8ff9640 1 -- 172.21.15.77:0/3904916576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d000c9a0 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08dc0cb140 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08d8ff9640 1 -- 172.21.15.77:0/3904916576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d000cd90 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.783+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08dc0cb700 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.784+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08dc068ff0 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.785+0000 7f08d8ff9640 1 -- 172.21.15.77:0/3904916576 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08d001a070 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.785+0000 7f08d8ff9640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08c806b4d0 0x7f08c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.785+0000 7f08d8ff9640 1 -- 172.21.15.77:0/3904916576 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08d008a190 con 0x7f08dc0d22b0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.785+0000 7f08dbfff640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08c806b4d0 0x7f08c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.786+0000 7f08dbfff640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08c806b4d0 0x7f08c806d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f08cc036000 tx=0x7f08cc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.789+0000 7f08d8ff9640 1 -- 172.21.15.77:0/3904916576 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08dc1f4be0 con 0x7f08dc0d22b0 2024-03-06T07:23:16.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.961+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08dc0c7260 con 0x7f08dc0d22b0 2024-03-06T07:23:16.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.962+0000 7f08d8ff9640 1 -- 172.21.15.77:0/3904916576 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f08d00589c0 con 0x7f08dc0d22b0 2024-03-06T07:23:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.963+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08c806b4d0 msgr2=0x7f08c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.963+0000 7f08e2a39640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08c806b4d0 0x7f08c806d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f08cc036000 tx=0x7f08cc03b040 comp rx=0 tx=0).stop 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.963+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc0d22b0 msgr2=0x7f08dc1f10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.963+0000 7f08e2a39640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc0d22b0 0x7f08dc1f10a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f08d000e9b0 tx=0x7f08d000ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.964+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 shutdown_connections 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.964+0000 7f08e2a39640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08c806b4d0 0x7f08c806d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.964+0000 7f08e2a39640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08dc0d22b0 0x7f08dc1f10a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.964+0000 7f08e2a39640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08dc1614d0 0x7f08dc0b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.964+0000 7f08e2a39640 1 --2- 172.21.15.77:0/3904916576 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08dc0d1910 0x7f08dc1f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.964+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 >> 172.21.15.77:0/3904916576 conn(0x7f08dc0d3d40 msgr2=0x7f08dc0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.964+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 shutdown_connections 2024-03-06T07:23:16.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:16.964+0000 7f08e2a39640 1 -- 172.21.15.77:0/3904916576 wait complete. 2024-03-06T07:23:16.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:17.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.077+0000 7f022237a640 1 Processor -- start 2024-03-06T07:23:17.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.077+0000 7f022237a640 1 -- start start 2024-03-06T07:23:17.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.078+0000 7f022237a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f021c0b1e40 0x7f021c168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.078+0000 7f022237a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021c0d2730 con 0x7f021c0d4860 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.078+0000 7f022237a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021c1618b0 con 0x7f021c1614d0 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.078+0000 7f022237a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021c0d4cf0 con 0x7f021c0b1e40 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.078+0000 7f0220917640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f021c0b1e40 0x7f021c168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.078+0000 7f021b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f021c0d4860 0x7f021c0d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40908/0 (socket says 172.21.15.77:40908) 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.078+0000 7f021b7fe640 1 -- 172.21.15.77:0/2881246649 learned_addr learned my addr 172.21.15.77:0/2881246649 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.079+0000 7f0220917640 1 -- 172.21.15.77:0/2881246649 >> v1:172.21.15.158:6789/0 conn(0x7f021c1614d0 legacy=0x7f021c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.079+0000 7f0220917640 1 -- 172.21.15.77:0/2881246649 >> v1:172.21.15.77:6789/0 conn(0x7f021c0d4860 legacy=0x7f021c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.079+0000 7f021affd640 1 -- 172.21.15.77:0/2881246649 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078878846 0 0) 0x7f021c0d2730 con 0x7f021c0d4860 2024-03-06T07:23:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.079+0000 7f0220917640 1 -- 172.21.15.77:0/2881246649 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c0cf400 con 0x7f021c0b1e40 2024-03-06T07:23:17.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.079+0000 7f0220917640 1 --2- 172.21.15.77:0/2881246649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c168b60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f021c0cf5d0 tx=0x7f021402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e82f2aed43e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.079+0000 7f021affd640 1 -- 172.21.15.77:0/2881246649 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214039070 con 0x7f021c0b1e40 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.079+0000 7f021affd640 1 -- 172.21.15.77:0/2881246649 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021402cc50 con 0x7f021c0b1e40 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.079+0000 7f021affd640 1 -- 172.21.15.77:0/2881246649 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214034440 con 0x7f021c0b1e40 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.080+0000 7f022237a640 1 -- 172.21.15.77:0/2881246649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 msgr2=0x7f021c168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.080+0000 7f022237a640 1 --2- 172.21.15.77:0/2881246649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c168b60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f021c0cf5d0 tx=0x7f021402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.080+0000 7f022237a640 1 -- 172.21.15.77:0/2881246649 shutdown_connections 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.080+0000 7f022237a640 1 --2- 172.21.15.77:0/2881246649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c168b60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.080+0000 7f022237a640 1 -- 172.21.15.77:0/2881246649 >> 172.21.15.77:0/2881246649 conn(0x7f021c0d3d40 msgr2=0x7f021c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.080+0000 7f022237a640 1 -- 172.21.15.77:0/2881246649 shutdown_connections 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.080+0000 7f022237a640 1 -- 172.21.15.77:0/2881246649 wait complete. 2024-03-06T07:23:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.081+0000 7f022237a640 1 Processor -- start 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.081+0000 7f022237a640 1 -- start start 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f022237a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c0b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f022237a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f021c0d4860 0x7f021c1f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f022237a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f021c1614d0 0x7f021c1f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f022237a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0cfc60 con 0x7f021c0b1e40 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f022237a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0cf980 con 0x7f021c0d4860 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f022237a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0b0910 con 0x7f021c1614d0 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c0b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c0b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55400/0 (socket says 172.21.15.77:55400) 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021bfff640 1 -- 172.21.15.77:0/4262651079 learned_addr learned my addr 172.21.15.77:0/4262651079 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021b7fe640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f021c0d4860 0x7f021c1f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f0220917640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f021c1614d0 0x7f021c1f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021bfff640 1 -- 172.21.15.77:0/4262651079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f021c1614d0 msgr2=0x7f021c1f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021bfff640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f021c1614d0 0x7f021c1f46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021bfff640 1 -- 172.21.15.77:0/4262651079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f021c0d4860 msgr2=0x7f021c1f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021bfff640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f021c0d4860 0x7f021c1f4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.082+0000 7f021bfff640 1 -- 172.21.15.77:0/4262651079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c0b72a0 con 0x7f021c0b1e40 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.083+0000 7f021bfff640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c0b6930 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f020c00c450 tx=0x7f020c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.083+0000 7f0218ff9640 1 -- 172.21.15.77:0/4262651079 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c0052d0 con 0x7f021c0b1e40 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.083+0000 7f0218ff9640 1 -- 172.21.15.77:0/4262651079 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c005470 con 0x7f021c0b1e40 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.083+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c0b7580 con 0x7f021c0b1e40 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.083+0000 7f0218ff9640 1 -- 172.21.15.77:0/4262651079 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c012870 con 0x7f021c0b1e40 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.083+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c0b7b40 con 0x7f021c0b1e40 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.084+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c068ff0 con 0x7f021c0b1e40 2024-03-06T07:23:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.084+0000 7f0218ff9640 1 -- 172.21.15.77:0/4262651079 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f020c012a10 con 0x7f021c0b1e40 2024-03-06T07:23:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.085+0000 7f0218ff9640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f020806b4d0 0x7f020806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.085+0000 7f0218ff9640 1 -- 172.21.15.77:0/4262651079 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f020c089c20 con 0x7f021c0b1e40 2024-03-06T07:23:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.085+0000 7f021b7fe640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f020806b4d0 0x7f020806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.085+0000 7f021b7fe640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f020806b4d0 0x7f020806d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0210007f70 tx=0x7f0210007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.089+0000 7f0218ff9640 1 -- 172.21.15.77:0/4262651079 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f020c0584e0 con 0x7f021c0b1e40 2024-03-06T07:23:17.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.258+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f021c0b5550 con 0x7f021c0b1e40 2024-03-06T07:23:17.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.259+0000 7f0218ff9640 1 -- 172.21.15.77:0/4262651079 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f020c05c580 con 0x7f021c0b1e40 2024-03-06T07:23:17.261 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:23:17.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f020806b4d0 msgr2=0x7f020806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f020806b4d0 0x7f020806d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0210007f70 tx=0x7f0210007f00 comp rx=0 tx=0).stop 2024-03-06T07:23:17.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 msgr2=0x7f021c0b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c0b6930 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f020c00c450 tx=0x7f020c00c910 comp rx=0 tx=0).stop 2024-03-06T07:23:17.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 shutdown_connections 2024-03-06T07:23:17.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f021c1614d0 0x7f021c1f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f021c0d4860 0x7f021c1f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f020806b4d0 0x7f020806d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 --2- 172.21.15.77:0/4262651079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f021c0b1e40 0x7f021c0b6930 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 >> 172.21.15.77:0/4262651079 conn(0x7f021c0d3d40 msgr2=0x7f021c15f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 shutdown_connections 2024-03-06T07:23:17.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.261+0000 7f022237a640 1 -- 172.21.15.77:0/4262651079 wait complete. 2024-03-06T07:23:17.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:17.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.374+0000 7f0193181640 1 Processor -- start 2024-03-06T07:23:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.374+0000 7f0193181640 1 -- start start 2024-03-06T07:23:17.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.375+0000 7f0193181640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c15c050 0x7f018c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.375+0000 7f0193181640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018c0cd4b0 con 0x7f018c06fb40 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.375+0000 7f0193181640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018c0cd790 con 0x7f018c0d0af0 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.375+0000 7f0193181640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c06ff80 con 0x7f018c15c050 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.375+0000 7f018bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f018c06fb40 0x7f018c0cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40920/0 (socket says 172.21.15.77:40920) 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.375+0000 7f018bfff640 1 -- 172.21.15.77:0/4175868387 learned_addr learned my addr 172.21.15.77:0/4175868387 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.376+0000 7f0190f1d640 1 --2- 172.21.15.77:0/4175868387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c15c050 0x7f018c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.376+0000 7f018b7fe640 1 -- 172.21.15.77:0/4175868387 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073714349 0 0) 0x7f018c0cd4b0 con 0x7f018c06fb40 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.376+0000 7f018b7fe640 1 -- 172.21.15.77:0/4175868387 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0174003520 con 0x7f018c06fb40 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.376+0000 7f018b7fe640 1 -- 172.21.15.77:0/4175868387 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210359464 0 0) 0x7f018c0cd790 con 0x7f018c0d0af0 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.376+0000 7f018b7fe640 1 -- 172.21.15.77:0/4175868387 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018c0cd4b0 con 0x7f018c0d0af0 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.376+0000 7f0190f1d640 1 -- 172.21.15.77:0/4175868387 >> v1:172.21.15.158:6789/0 conn(0x7f018c0d0af0 legacy=0x7f018c165ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.376+0000 7f0190f1d640 1 -- 172.21.15.77:0/4175868387 >> v1:172.21.15.77:6789/0 conn(0x7f018c06fb40 legacy=0x7f018c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.376+0000 7f0190f1d640 1 -- 172.21.15.77:0/4175868387 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c0cda70 con 0x7f018c15c050 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f0190f1d640 1 --2- 172.21.15.77:0/4175868387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c15c050 0x7f018c06f600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f017c0097d0 tx=0x7f017c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e8c2d9eafddfe5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f018b7fe640 1 -- 172.21.15.77:0/4175868387 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c035020 con 0x7f018c15c050 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f018b7fe640 1 -- 172.21.15.77:0/4175868387 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c031800 con 0x7f018c15c050 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f018b7fe640 1 -- 172.21.15.77:0/4175868387 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c031bb0 con 0x7f018c15c050 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f0193181640 1 -- 172.21.15.77:0/4175868387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c15c050 msgr2=0x7f018c06f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f0193181640 1 --2- 172.21.15.77:0/4175868387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c15c050 0x7f018c06f600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f017c0097d0 tx=0x7f017c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f0193181640 1 -- 172.21.15.77:0/4175868387 shutdown_connections 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f0193181640 1 --2- 172.21.15.77:0/4175868387 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c15c050 0x7f018c06f600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f0193181640 1 -- 172.21.15.77:0/4175868387 >> 172.21.15.77:0/4175868387 conn(0x7f018c0b3800 msgr2=0x7f018c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f0193181640 1 -- 172.21.15.77:0/4175868387 shutdown_connections 2024-03-06T07:23:17.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.377+0000 7f0193181640 1 -- 172.21.15.77:0/4175868387 wait complete. 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.378+0000 7f0193181640 1 Processor -- start 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.378+0000 7f0193181640 1 -- start start 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f0193181640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c06fb40 0x7f018c0b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f0193181640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c0d0af0 0x7f018c150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f0193181640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f018c15c050 0x7f018c1f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f0193181640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0ccff0 con 0x7f018c15c050 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f0193181640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0cce70 con 0x7f018c06fb40 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f0193181640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0cd170 con 0x7f018c0d0af0 2024-03-06T07:23:17.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f019171e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f018c15c050 0x7f018c1f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f019171e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f018c15c050 0x7f018c1f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55402/0 (socket says 172.21.15.77:55402) 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f0190f1d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c06fb40 0x7f018c0b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f019171e640 1 -- 172.21.15.77:0/2106181987 learned_addr learned my addr 172.21.15.77:0/2106181987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.379+0000 7f018bfff640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c0d0af0 0x7f018c150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f019171e640 1 -- 172.21.15.77:0/2106181987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c0d0af0 msgr2=0x7f018c150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f019171e640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c0d0af0 0x7f018c150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f019171e640 1 -- 172.21.15.77:0/2106181987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c06fb40 msgr2=0x7f018c0b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f019171e640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c06fb40 0x7f018c0b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f019171e640 1 -- 172.21.15.77:0/2106181987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c0b4a70 con 0x7f018c15c050 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f019171e640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f018c15c050 0x7f018c1f73a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f018400c6b0 tx=0x7f018400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f01897fa640 1 -- 172.21.15.77:0/2106181987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184003b70 con 0x7f018c15c050 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f01897fa640 1 -- 172.21.15.77:0/2106181987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0184003d10 con 0x7f018c15c050 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f01897fa640 1 -- 172.21.15.77:0/2106181987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184005310 con 0x7f018c15c050 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018c0b4d50 con 0x7f018c15c050 2024-03-06T07:23:17.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.380+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c0b5310 con 0x7f018c15c050 2024-03-06T07:23:17.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.381+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c09fe60 con 0x7f018c15c050 2024-03-06T07:23:17.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.381+0000 7f01897fa640 1 -- 172.21.15.77:0/2106181987 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01840054b0 con 0x7f018c15c050 2024-03-06T07:23:17.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.382+0000 7f01897fa640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f017806b4d0 0x7f017806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.382+0000 7f01897fa640 1 -- 172.21.15.77:0/2106181987 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0184089f90 con 0x7f018c15c050 2024-03-06T07:23:17.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.382+0000 7f0190f1d640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f017806b4d0 0x7f017806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.382+0000 7f0190f1d640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f017806b4d0 0x7f017806d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f017c036000 tx=0x7f017c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.386+0000 7f01897fa640 1 -- 172.21.15.77:0/2106181987 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01840587c0 con 0x7f018c15c050 2024-03-06T07:23:17.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.561+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f018c068ff0 con 0x7f018c15c050 2024-03-06T07:23:17.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.561+0000 7f01897fa640 1 -- 172.21.15.77:0/2106181987 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7f018405c860 con 0x7f018c15c050 2024-03-06T07:23:17.563 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:23:17.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f017806b4d0 msgr2=0x7f017806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f017806b4d0 0x7f017806d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f017c036000 tx=0x7f017c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:17.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f018c15c050 msgr2=0x7f018c1f73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f018c15c050 0x7f018c1f73a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f018400c6b0 tx=0x7f018400cb70 comp rx=0 tx=0).stop 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 shutdown_connections 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f017806b4d0 0x7f017806d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f018c15c050 0x7f018c1f73a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f018c0d0af0 0x7f018c150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 --2- 172.21.15.77:0/2106181987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f018c06fb40 0x7f018c0b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 >> 172.21.15.77:0/2106181987 conn(0x7f018c0b3800 msgr2=0x7f018c0b2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 shutdown_connections 2024-03-06T07:23:17.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.563+0000 7f0193181640 1 -- 172.21.15.77:0/2106181987 wait complete. 2024-03-06T07:23:17.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:17.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.675+0000 7fdfb5881640 1 Processor -- start 2024-03-06T07:23:17.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.675+0000 7fdfb5881640 1 -- start start 2024-03-06T07:23:17.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfb5881640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdfb00b4460 0x7fdfb00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfb5881640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0162970 0x7fdfb00b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfb5881640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00d22c0 con 0x7fdfb0161cf0 2024-03-06T07:23:17.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfb5881640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb0161b00 con 0x7fdfb00b4460 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfb5881640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00cecc0 con 0x7fdfb0162970 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfaf7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdfb0161cf0 0x7fdfb00cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:40930/0 (socket says 172.21.15.77:40930) 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfaf7fe640 1 -- 172.21.15.77:0/792720664 learned_addr learned my addr 172.21.15.77:0/792720664 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfaeffd640 1 --2- 172.21.15.77:0/792720664 >> v2:172.21.15.158:3300/0 conn(0x7fdfb00b4460 0x7fdfb00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfae7fc640 1 --2- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0162970 0x7fdfb00b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.676+0000 7fdfadffb640 1 -- 172.21.15.77:0/792720664 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790088336 0 0) 0x7fdfb00d22c0 con 0x7fdfb0161cf0 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.677+0000 7fdfadffb640 1 -- 172.21.15.77:0/792720664 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9c003520 con 0x7fdfb0161cf0 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.677+0000 7fdfaeffd640 1 -- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0162970 msgr2=0x7fdfb00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.677+0000 7fdfaeffd640 1 --2- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0162970 0x7fdfb00b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.677+0000 7fdfaeffd640 1 -- 172.21.15.77:0/792720664 >> v1:172.21.15.77:6789/0 conn(0x7fdfb0161cf0 legacy=0x7fdfb00cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.677+0000 7fdfaeffd640 1 -- 172.21.15.77:0/792720664 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb00d25a0 con 0x7fdfb00b4460 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.677+0000 7fdfae7fc640 1 --2- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0162970 0x7fdfb00b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.677+0000 7fdfaeffd640 1 --2- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb00b4460 0x7fdfb00d4800 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009d90 tx=0x7fdfa802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd1db1475c83716 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfadffb640 1 -- 172.21.15.77:0/792720664 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8039070 con 0x7fdfb00b4460 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfadffb640 1 -- 172.21.15.77:0/792720664 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa802cc50 con 0x7fdfb00b4460 2024-03-06T07:23:17.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfadffb640 1 -- 172.21.15.77:0/792720664 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8034440 con 0x7fdfb00b4460 2024-03-06T07:23:17.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfb5881640 1 -- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb00b4460 msgr2=0x7fdfb00d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfb5881640 1 --2- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb00b4460 0x7fdfb00d4800 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009d90 tx=0x7fdfa802bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfb5881640 1 -- 172.21.15.77:0/792720664 shutdown_connections 2024-03-06T07:23:17.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfb5881640 1 --2- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0162970 0x7fdfb00b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfb5881640 1 --2- 172.21.15.77:0/792720664 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb00b4460 0x7fdfb00d4800 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfb5881640 1 -- 172.21.15.77:0/792720664 >> 172.21.15.77:0/792720664 conn(0x7fdfb00d3ce0 msgr2=0x7fdfb00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfb5881640 1 -- 172.21.15.77:0/792720664 shutdown_connections 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.678+0000 7fdfb5881640 1 -- 172.21.15.77:0/792720664 wait complete. 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.679+0000 7fdfb5881640 1 Processor -- start 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.679+0000 7fdfb5881640 1 -- start start 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfb5881640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb00b4460 0x7fdfb01511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfb5881640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0161cf0 0x7fdfb00c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfb5881640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb0162970 0x7fdfb01f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfb5881640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00ce950 con 0x7fdfb00b4460 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfb5881640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00d2b60 con 0x7fdfb0162970 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfb5881640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00b3ac0 con 0x7fdfb0161cf0 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfaeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb00b4460 0x7fdfb01511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfaeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb00b4460 0x7fdfb01511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55412/0 (socket says 172.21.15.77:55412) 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfaeffd640 1 -- 172.21.15.77:0/999792613 learned_addr learned my addr 172.21.15.77:0/999792613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfae7fc640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0161cf0 0x7fdfb00c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.680+0000 7fdfaf7fe640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb0162970 0x7fdfb01f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfaeffd640 1 -- 172.21.15.77:0/999792613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0161cf0 msgr2=0x7fdfb00c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfaeffd640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0161cf0 0x7fdfb00c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfaeffd640 1 -- 172.21.15.77:0/999792613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb0162970 msgr2=0x7fdfb01f72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfaeffd640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb0162970 0x7fdfb01f72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfaeffd640 1 -- 172.21.15.77:0/999792613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb00b9530 con 0x7fdfb00b4460 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfaeffd640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb00b4460 0x7fdfb01511c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009d90 tx=0x7fdfa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfa7fff640 1 -- 172.21.15.77:0/999792613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8039050 con 0x7fdfb00b4460 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfa7fff640 1 -- 172.21.15.77:0/999792613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa8045070 con 0x7fdfb00b4460 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb00b9810 con 0x7fdfb00b4460 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfa7fff640 1 -- 172.21.15.77:0/999792613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8009450 con 0x7fdfb00b4460 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.681+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb00ca600 con 0x7fdfb00b4460 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.682+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb004f330 con 0x7fdfb00b4460 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.682+0000 7fdfa7fff640 1 -- 172.21.15.77:0/999792613 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfa803e610 con 0x7fdfb00b4460 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.683+0000 7fdfa7fff640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf9006b4d0 0x7fdf9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.683+0000 7fdfa7fff640 1 -- 172.21.15.77:0/999792613 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfa80aca10 con 0x7fdfb00b4460 2024-03-06T07:23:17.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.683+0000 7fdfae7fc640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf9006b4d0 0x7fdf9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.683+0000 7fdfae7fc640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf9006b4d0 0x7fdf9006d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009560 tx=0x7fdf98009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.687+0000 7fdfa7fff640 1 -- 172.21.15.77:0/999792613 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfa807b390 con 0x7fdfb00b4460 2024-03-06T07:23:17.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.871+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdfb006b180 con 0x7fdfb00b4460 2024-03-06T07:23:17.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.871+0000 7fdfa7fff640 1 -- 172.21.15.77:0/999792613 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdfa807f430 con 0x7fdfb00b4460 2024-03-06T07:23:17.873 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf9006b4d0 msgr2=0x7fdf9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf9006b4d0 0x7fdf9006d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009560 tx=0x7fdf98009190 comp rx=0 tx=0).stop 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb00b4460 msgr2=0x7fdfb01511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb00b4460 0x7fdfb01511c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009d90 tx=0x7fdfa8004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:17.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 shutdown_connections 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdfb0162970 0x7fdfb01f72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdfb0161cf0 0x7fdfb00c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf9006b4d0 0x7fdf9006d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 --2- 172.21.15.77:0/999792613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdfb00b4460 0x7fdfb01511c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 >> 172.21.15.77:0/999792613 conn(0x7fdfb00d3ce0 msgr2=0x7fdfb006c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 shutdown_connections 2024-03-06T07:23:17.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:17.873+0000 7fdfb5881640 1 -- 172.21.15.77:0/999792613 wait complete. 2024-03-06T07:23:17.886 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:23:17.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:17.886 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:17.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[52454]: fuse finished with error 0 and tester_r 0 2024-03-06T07:23:17.913 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:23:17.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:17.913 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:18.055 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:23:18.055 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:18.071 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:18.071 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:23:18.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:18.071 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:23:18.127 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:23:18.127 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:18.143 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:18.143 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:23:18.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:18.143 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:18.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:18.198 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:23:18.199 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:23:18.199 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:18.214 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:18.214 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:23:18.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:18.214 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:18.269 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:23:18.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.366+0000 7f5e00ceb640 1 Processor -- start 2024-03-06T07:23:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.366+0000 7f5e00ceb640 1 -- start start 2024-03-06T07:23:18.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.367+0000 7f5e00ceb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5dfc0b3060 0x7f5dfc15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.367+0000 7f5e00ceb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.367+0000 7f5e00ceb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dfc15f2d0 con 0x7f5dfc13dc10 2024-03-06T07:23:18.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.367+0000 7f5e00ceb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dfc0cf3f0 con 0x7f5dfc0b3060 2024-03-06T07:23:18.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.367+0000 7f5e00ceb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc15b210 con 0x7f5dfc15bf30 2024-03-06T07:23:18.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:60688/0 (socket says 172.21.15.77:60688) 2024-03-06T07:23:18.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5dfad9d640 1 -- 172.21.15.77:0/2308166008 learned_addr learned my addr 172.21.15.77:0/2308166008 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:18.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5df9d9b640 1 --2- 172.21.15.77:0/2308166008 >> v2:172.21.15.158:3300/0 conn(0x7f5dfc0b3060 0x7f5dfc15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5dfad9d640 1 -- 172.21.15.77:0/2308166008 >> v2:172.21.15.158:3300/0 conn(0x7f5dfc0b3060 msgr2=0x7f5dfc15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5dfad9d640 1 --2- 172.21.15.77:0/2308166008 >> v2:172.21.15.158:3300/0 conn(0x7f5dfc0b3060 0x7f5dfc15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5dfad9d640 1 -- 172.21.15.77:0/2308166008 >> v1:172.21.15.158:6789/0 conn(0x7f5dfc13dc10 legacy=0x7f5dfc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5df959a640 1 -- 172.21.15.77:0/2308166008 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210426971 0 0) 0x7f5dfc15f2d0 con 0x7f5dfc13dc10 2024-03-06T07:23:18.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.368+0000 7f5dfad9d640 1 -- 172.21.15.77:0/2308166008 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc15f580 con 0x7f5dfc15bf30 2024-03-06T07:23:18.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5dfad9d640 1 --2- 172.21.15.77:0/2308166008 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc15ed90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009e50 tx=0x7f5df402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b829ecb4b056dca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5df959a640 1 -- 172.21.15.77:0/2308166008 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df402ca70 con 0x7f5dfc15bf30 2024-03-06T07:23:18.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5df959a640 1 -- 172.21.15.77:0/2308166008 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df402cc10 con 0x7f5dfc15bf30 2024-03-06T07:23:18.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5df959a640 1 -- 172.21.15.77:0/2308166008 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df40343f0 con 0x7f5dfc15bf30 2024-03-06T07:23:18.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5e00ceb640 1 -- 172.21.15.77:0/2308166008 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 msgr2=0x7f5dfc15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/2308166008 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc15ed90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009e50 tx=0x7f5df402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5e00ceb640 1 -- 172.21.15.77:0/2308166008 shutdown_connections 2024-03-06T07:23:18.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/2308166008 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc15ed90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/2308166008 >> v2:172.21.15.158:3300/0 conn(0x7f5dfc0b3060 0x7f5dfc15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.369+0000 7f5e00ceb640 1 -- 172.21.15.77:0/2308166008 >> 172.21.15.77:0/2308166008 conn(0x7f5dfc0cf8a0 msgr2=0x7f5dfc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.370+0000 7f5e00ceb640 1 -- 172.21.15.77:0/2308166008 shutdown_connections 2024-03-06T07:23:18.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.370+0000 7f5e00ceb640 1 -- 172.21.15.77:0/2308166008 wait complete. 2024-03-06T07:23:18.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.371+0000 7f5e00ceb640 1 Processor -- start 2024-03-06T07:23:18.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.371+0000 7f5e00ceb640 1 -- start start 2024-03-06T07:23:18.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.371+0000 7f5e00ceb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dfc0b3060 0x7f5dfc143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.371+0000 7f5e00ceb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dfc13dc10 0x7f5dfc13a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.371+0000 7f5e00ceb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc13c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.371+0000 7f5e00ceb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc13dff0 con 0x7f5dfc0b3060 2024-03-06T07:23:18.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.371+0000 7f5e00ceb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc15b730 con 0x7f5dfc13dc10 2024-03-06T07:23:18.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.371+0000 7f5e00ceb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc0708d0 con 0x7f5dfc15bf30 2024-03-06T07:23:18.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dfc0b3060 0x7f5dfc143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dfc0b3060 0x7f5dfc143710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55418/0 (socket says 172.21.15.77:55418) 2024-03-06T07:23:18.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 -- 172.21.15.77:0/1333225946 learned_addr learned my addr 172.21.15.77:0/1333225946 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:18.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5df9d9b640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dfc13dc10 0x7f5dfc13a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfad9d640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc13c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 -- 172.21.15.77:0/1333225946 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 msgr2=0x7f5dfc13c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc13c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 -- 172.21.15.77:0/1333225946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dfc13dc10 msgr2=0x7f5dfc13a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dfc13dc10 0x7f5dfc13a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 -- 172.21.15.77:0/1333225946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc143f60 con 0x7f5dfc0b3060 2024-03-06T07:23:18.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5dfa59c640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dfc0b3060 0x7f5dfc143710 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5de400c4b0 tx=0x7f5de400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1333225946 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4017070 con 0x7f5dfc0b3060 2024-03-06T07:23:18.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1333225946 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de4005480 con 0x7f5dfc0b3060 2024-03-06T07:23:18.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dfc1444b0 con 0x7f5dfc0b3060 2024-03-06T07:23:18.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1333225946 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4003830 con 0x7f5dfc0b3060 2024-03-06T07:23:18.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.372+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dfc1441d0 con 0x7f5dfc0b3060 2024-03-06T07:23:18.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.373+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dfc13a650 con 0x7f5dfc0b3060 2024-03-06T07:23:18.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.374+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1333225946 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5de401a070 con 0x7f5dfc0b3060 2024-03-06T07:23:18.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.374+0000 7f5df2ffd640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ddc06b4d0 0x7f5ddc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.374+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1333225946 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5de4089500 con 0x7f5dfc0b3060 2024-03-06T07:23:18.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.374+0000 7f5df9d9b640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ddc06b4d0 0x7f5ddc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.375+0000 7f5df9d9b640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ddc06b4d0 0x7f5ddc06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc071200 tx=0x7f5dec009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.378+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1333225946 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5dfc13a650 con 0x7f5dfc0b3060 2024-03-06T07:23:18.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.586+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dfc0b3eb0 con 0x7f5dfc0b3060 2024-03-06T07:23:18.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.586+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1333225946 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f5dfc0b3eb0 con 0x7f5dfc0b3060 2024-03-06T07:23:18.588 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:18.589 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:18.589 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ddc06b4d0 msgr2=0x7f5ddc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ddc06b4d0 0x7f5ddc06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc071200 tx=0x7f5dec009190 comp rx=0 tx=0).stop 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dfc0b3060 msgr2=0x7f5dfc143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dfc0b3060 0x7f5dfc143710 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5de400c4b0 tx=0x7f5de400c970 comp rx=0 tx=0).stop 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 shutdown_connections 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dfc15bf30 0x7f5dfc13c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dfc13dc10 0x7f5dfc13a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ddc06b4d0 0x7f5ddc06d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 --2- 172.21.15.77:0/1333225946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dfc0b3060 0x7f5dfc143710 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 >> 172.21.15.77:0/1333225946 conn(0x7f5dfc0cf8a0 msgr2=0x7f5dfc0b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 shutdown_connections 2024-03-06T07:23:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.588+0000 7f5e00ceb640 1 -- 172.21.15.77:0/1333225946 wait complete. 2024-03-06T07:23:18.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:18.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.699+0000 7fd62beec640 1 Processor -- start 2024-03-06T07:23:18.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd62beec640 1 -- start start 2024-03-06T07:23:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd62beec640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd62415dfe0 0x7fd6240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd62beec640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6240d29c0 0x7fd6240b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd62beec640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6240b3430 con 0x7fd6240cf830 2024-03-06T07:23:18.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd62beec640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6240b31e0 con 0x7fd62415dfe0 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd62beec640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6240b3d60 con 0x7fd6240d29c0 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd629c88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd62415dfe0 0x7fd6240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd629487640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd6240cf830 0x7fd6240d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49566/0 (socket says 172.21.15.77:49566) 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.700+0000 7fd629487640 1 -- 172.21.15.77:0/295567142 learned_addr learned my addr 172.21.15.77:0/295567142 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.701+0000 7fd62a489640 1 --2- 172.21.15.77:0/295567142 >> v2:172.21.15.158:3300/0 conn(0x7fd6240d29c0 0x7fd6240b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.701+0000 7fd629c88640 1 -- 172.21.15.77:0/295567142 >> v2:172.21.15.158:3300/0 conn(0x7fd6240d29c0 msgr2=0x7fd6240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.701+0000 7fd629c88640 1 --2- 172.21.15.77:0/295567142 >> v2:172.21.15.158:3300/0 conn(0x7fd6240d29c0 0x7fd6240b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.701+0000 7fd629c88640 1 -- 172.21.15.77:0/295567142 >> v1:172.21.15.77:6789/0 conn(0x7fd6240cf830 legacy=0x7fd6240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.701+0000 7fd629c88640 1 -- 172.21.15.77:0/295567142 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6240b3710 con 0x7fd62415dfe0 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.701+0000 7fd628c86640 1 -- 172.21.15.77:0/295567142 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247409415 0 0) 0x7fd6240b3430 con 0x7fd6240cf830 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.701+0000 7fd629c88640 1 --2- 172.21.15.77:0/295567142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62415dfe0 0x7fd6240cf2f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd6240b3cd0 tx=0x7fd62002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226d2ab1d067d26b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd628c86640 1 -- 172.21.15.77:0/295567142 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620039070 con 0x7fd62415dfe0 2024-03-06T07:23:18.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd628c86640 1 -- 172.21.15.77:0/295567142 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62002cc80 con 0x7fd62415dfe0 2024-03-06T07:23:18.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd628c86640 1 -- 172.21.15.77:0/295567142 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620034660 con 0x7fd62415dfe0 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd62beec640 1 -- 172.21.15.77:0/295567142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62415dfe0 msgr2=0x7fd6240cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd62beec640 1 --2- 172.21.15.77:0/295567142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62415dfe0 0x7fd6240cf2f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd6240b3cd0 tx=0x7fd62002bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd62beec640 1 -- 172.21.15.77:0/295567142 shutdown_connections 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd62beec640 1 --2- 172.21.15.77:0/295567142 >> v2:172.21.15.158:3300/0 conn(0x7fd6240d29c0 0x7fd6240b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd62beec640 1 --2- 172.21.15.77:0/295567142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62415dfe0 0x7fd6240cf2f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd62beec640 1 -- 172.21.15.77:0/295567142 >> 172.21.15.77:0/295567142 conn(0x7fd6240d3cd0 msgr2=0x7fd6240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd62beec640 1 -- 172.21.15.77:0/295567142 shutdown_connections 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.702+0000 7fd62beec640 1 -- 172.21.15.77:0/295567142 wait complete. 2024-03-06T07:23:18.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.703+0000 7fd62beec640 1 Processor -- start 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.703+0000 7fd62beec640 1 -- start start 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd62beec640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6240cf830 0x7fd6241f78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd62beec640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6240d29c0 0x7fd6240b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd62beec640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd62415dfe0 0x7fd6240b90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd62beec640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240d1df0 con 0x7fd6240cf830 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd62beec640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240d1c70 con 0x7fd62415dfe0 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd62beec640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62415e3c0 con 0x7fd6240d29c0 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd629c88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6240cf830 0x7fd6241f78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd629c88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6240cf830 0x7fd6241f78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35848/0 (socket says 172.21.15.77:35848) 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd629c88640 1 -- 172.21.15.77:0/59148595 learned_addr learned my addr 172.21.15.77:0/59148595 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd62a489640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd62415dfe0 0x7fd6240b90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.704+0000 7fd629487640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6240d29c0 0x7fd6240b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd629c88640 1 -- 172.21.15.77:0/59148595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6240d29c0 msgr2=0x7fd6240b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd629c88640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6240d29c0 0x7fd6240b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd629c88640 1 -- 172.21.15.77:0/59148595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd62415dfe0 msgr2=0x7fd6240b90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd629c88640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd62415dfe0 0x7fd6240b90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd629c88640 1 -- 172.21.15.77:0/59148595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6241f4910 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd629c88640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6240cf830 0x7fd6241f78c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd620037040 tx=0x7fd620004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd61a7fc640 1 -- 172.21.15.77:0/59148595 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620039050 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd61a7fc640 1 -- 172.21.15.77:0/59148595 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62003a040 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6240b80b0 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd61a7fc640 1 -- 172.21.15.77:0/59148595 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620003990 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.705+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6240b7d10 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.706+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62404f330 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.706+0000 7fd61a7fc640 1 -- 172.21.15.77:0/59148595 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd620047030 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.707+0000 7fd61a7fc640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd60806b4d0 0x7fd60806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.707+0000 7fd61a7fc640 1 -- 172.21.15.77:0/59148595 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6200ac370 con 0x7fd6240cf830 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.707+0000 7fd629487640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd60806b4d0 0x7fd60806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.707+0000 7fd629487640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd60806b4d0 0x7fd60806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd614007f70 tx=0x7fd614007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.711+0000 7fd61a7fc640 1 -- 172.21.15.77:0/59148595 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd62004e060 con 0x7fd6240cf830 2024-03-06T07:23:18.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.887+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6240d0e20 con 0x7fd6240cf830 2024-03-06T07:23:18.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.888+0000 7fd61a7fc640 1 -- 172.21.15.77:0/59148595 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7fd62007ac30 con 0x7fd6240cf830 2024-03-06T07:23:18.890 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:18.891 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:58.329001+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:54.903388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.652222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:18.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd60806b4d0 msgr2=0x7fd60806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd60806b4d0 0x7fd60806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd614007f70 tx=0x7fd614007f00 comp rx=0 tx=0).stop 2024-03-06T07:23:18.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6240cf830 msgr2=0x7fd6241f78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6240cf830 0x7fd6241f78c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd620037040 tx=0x7fd620004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 shutdown_connections 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd62415dfe0 0x7fd6240b90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6240d29c0 0x7fd6240b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd60806b4d0 0x7fd60806d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 --2- 172.21.15.77:0/59148595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6240cf830 0x7fd6241f78c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 >> 172.21.15.77:0/59148595 conn(0x7fd6240d3cd0 msgr2=0x7fd62406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 shutdown_connections 2024-03-06T07:23:18.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:18.890+0000 7fd62beec640 1 -- 172.21.15.77:0/59148595 wait complete. 2024-03-06T07:23:18.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:54.903388+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:55.652222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:18.904 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.005+0000 7f2fe59e9640 1 Processor -- start 2024-03-06T07:23:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.005+0000 7f2fe59e9640 1 -- start start 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fe59e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2fe01614d0 0x7f2fe00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fe59e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d1910 0x7f2fe00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fe59e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2fe00d22b0 0x7f2fe00d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fe59e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe00d2740 con 0x7f2fe01614d0 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fe59e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe01618b0 con 0x7f2fe00d22b0 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fe59e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00cf400 con 0x7f2fe00d1910 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fdeffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2fe01614d0 0x7f2fe00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fdeffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2fe01614d0 0x7f2fe00d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35858/0 (socket says 172.21.15.77:35858) 2024-03-06T07:23:19.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fdeffd640 1 -- 172.21.15.77:0/3837997012 learned_addr learned my addr 172.21.15.77:0/3837997012 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fdf7fe640 1 --2- 172.21.15.77:0/3837997012 >> v2:172.21.15.158:3300/0 conn(0x7f2fe00d22b0 0x7f2fe00d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.006+0000 7f2fde7fc640 1 --2- 172.21.15.77:0/3837997012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d1910 0x7f2fe00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fdeffd640 1 -- 172.21.15.77:0/3837997012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d1910 msgr2=0x7f2fe00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fdeffd640 1 --2- 172.21.15.77:0/3837997012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d1910 0x7f2fe00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fdeffd640 1 -- 172.21.15.77:0/3837997012 >> v2:172.21.15.158:3300/0 conn(0x7f2fe00d22b0 msgr2=0x7f2fe00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fdeffd640 1 --2- 172.21.15.77:0/3837997012 >> v2:172.21.15.158:3300/0 conn(0x7f2fe00d22b0 0x7f2fe00d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fdeffd640 1 -- 172.21.15.77:0/3837997012 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe00cf6a0 con 0x7f2fe01614d0 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fdeffd640 1 --2- 172.21.15.77:0/3837997012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe01614d0 0x7f2fe00d4860 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00b57d0 tx=0x7f2fd002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4d468e4d71b16d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fddffb640 1 -- 172.21.15.77:0/3837997012 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd002ca70 con 0x7f2fe01614d0 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fddffb640 1 -- 172.21.15.77:0/3837997012 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd002cc10 con 0x7f2fe01614d0 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.007+0000 7f2fddffb640 1 -- 172.21.15.77:0/3837997012 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd00343f0 con 0x7f2fe01614d0 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3837997012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe01614d0 msgr2=0x7f2fe00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3837997012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe01614d0 0x7f2fe00d4860 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00b57d0 tx=0x7f2fd002bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3837997012 shutdown_connections 2024-03-06T07:23:19.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3837997012 >> v2:172.21.15.158:3300/0 conn(0x7f2fe00d22b0 0x7f2fe00d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3837997012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d1910 0x7f2fe00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3837997012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe01614d0 0x7f2fe00d4860 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3837997012 >> 172.21.15.77:0/3837997012 conn(0x7f2fe00d3d40 msgr2=0x7f2fe00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3837997012 shutdown_connections 2024-03-06T07:23:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.008+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3837997012 wait complete. 2024-03-06T07:23:19.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.009+0000 7f2fe59e9640 1 Processor -- start 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.009+0000 7f2fe59e9640 1 -- start start 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fe59e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00d1910 0x7f2fe014c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fe59e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d22b0 0x7f2fe00cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fe59e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fe01614d0 0x7f2fe00b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fe59e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00b0910 con 0x7f2fe00d1910 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fe59e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00cfc60 con 0x7f2fe01614d0 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fe59e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00d0be0 con 0x7f2fe00d22b0 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00d1910 0x7f2fe014c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00d1910 0x7f2fe014c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35866/0 (socket says 172.21.15.77:35866) 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdeffd640 1 -- 172.21.15.77:0/3369442313 learned_addr learned my addr 172.21.15.77:0/3369442313 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdf7fe640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fe01614d0 0x7f2fe00b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fde7fc640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d22b0 0x7f2fe00cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdeffd640 1 -- 172.21.15.77:0/3369442313 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d22b0 msgr2=0x7f2fe00cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdeffd640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d22b0 0x7f2fe00cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdeffd640 1 -- 172.21.15.77:0/3369442313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fe01614d0 msgr2=0x7f2fe00b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdeffd640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fe01614d0 0x7f2fe00b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.010+0000 7f2fdeffd640 1 -- 172.21.15.77:0/3369442313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe00c8320 con 0x7f2fe00d1910 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.011+0000 7f2fdeffd640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00d1910 0x7f2fe014c1d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0000f80 tx=0x7f2fd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.011+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3369442313 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0043020 con 0x7f2fe00d1910 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.011+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3369442313 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd003e450 con 0x7f2fe00d1910 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.011+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe00c85c0 con 0x7f2fe00d1910 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.011+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3369442313 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0009510 con 0x7f2fe00d1910 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.011+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe00c8b80 con 0x7f2fe00d1910 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.012+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe00cf140 con 0x7f2fe00d1910 2024-03-06T07:23:19.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.012+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3369442313 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fd0046020 con 0x7f2fe00d1910 2024-03-06T07:23:19.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.012+0000 7f2fc7fff640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fcc06b4d0 0x7f2fcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.013+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3369442313 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fd00ac760 con 0x7f2fe00d1910 2024-03-06T07:23:19.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.013+0000 7f2fde7fc640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fcc06b4d0 0x7f2fcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.013+0000 7f2fde7fc640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fcc06b4d0 0x7f2fcc06d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00cc5b0 tx=0x7f2fd4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.017+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3369442313 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fd004d030 con 0x7f2fe00d1910 2024-03-06T07:23:19.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.187+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2fe0068ff0 con 0x7f2fe00d1910 2024-03-06T07:23:19.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.188+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3369442313 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2fd007b020 con 0x7f2fe00d1910 2024-03-06T07:23:19.189 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:23:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fcc06b4d0 msgr2=0x7f2fcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fcc06b4d0 0x7f2fcc06d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00cc5b0 tx=0x7f2fd4009190 comp rx=0 tx=0).stop 2024-03-06T07:23:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00d1910 msgr2=0x7f2fe014c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00d1910 0x7f2fe014c1d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0000f80 tx=0x7f2fd0004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 shutdown_connections 2024-03-06T07:23:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fe01614d0 0x7f2fe00b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe00d22b0 0x7f2fe00cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fcc06b4d0 0x7f2fcc06d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 --2- 172.21.15.77:0/3369442313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00d1910 0x7f2fe014c1d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.189+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 >> 172.21.15.77:0/3369442313 conn(0x7f2fe00d3d40 msgr2=0x7f2fe006c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.190+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 shutdown_connections 2024-03-06T07:23:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.190+0000 7f2fe59e9640 1 -- 172.21.15.77:0/3369442313 wait complete. 2024-03-06T07:23:19.203 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:19.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.304+0000 7fa09cb3e640 1 Processor -- start 2024-03-06T07:23:19.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.304+0000 7fa09cb3e640 1 -- start start 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.305+0000 7fa09cb3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09815a220 0x7fa0980cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.305+0000 7fa09cb3e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa0980712d0 0x7fa098071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.305+0000 7fa09cb3e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa09815de60 0x7fa09815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.305+0000 7fa09cb3e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0980cf610 con 0x7fa09815de60 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.305+0000 7fa09cb3e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa098071c70 con 0x7fa0980712d0 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.305+0000 7fa09cb3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0980b3320 con 0x7fa09815a220 2024-03-06T07:23:19.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa096d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa09815de60 0x7fa09815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa096d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa09815de60 0x7fa09815e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35882/0 (socket says 172.21.15.77:35882) 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa096d9d640 1 -- 172.21.15.77:0/866115538 learned_addr learned my addr 172.21.15.77:0/866115538 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa09659c640 1 --2- 172.21.15.77:0/866115538 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09815a220 0x7fa0980cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa095d9b640 1 --2- 172.21.15.77:0/866115538 >> v2:172.21.15.158:3300/0 conn(0x7fa0980712d0 0x7fa098071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa096d9d640 1 -- 172.21.15.77:0/866115538 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09815a220 msgr2=0x7fa0980cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa096d9d640 1 --2- 172.21.15.77:0/866115538 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09815a220 0x7fa0980cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa096d9d640 1 -- 172.21.15.77:0/866115538 >> v2:172.21.15.158:3300/0 conn(0x7fa0980712d0 msgr2=0x7fa098071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa096d9d640 1 --2- 172.21.15.77:0/866115538 >> v2:172.21.15.158:3300/0 conn(0x7fa0980712d0 0x7fa098071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.306+0000 7fa096d9d640 1 -- 172.21.15.77:0/866115538 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0980728d0 con 0x7fa09815de60 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa096d9d640 1 --2- 172.21.15.77:0/866115538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09815e2f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa09806eac0 tx=0x7fa09002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9621987296bb0e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09559a640 1 -- 172.21.15.77:0/866115538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09002c9e0 con 0x7fa09815de60 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09559a640 1 -- 172.21.15.77:0/866115538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09002cb80 con 0x7fa09815de60 2024-03-06T07:23:19.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09559a640 1 -- 172.21.15.77:0/866115538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900343f0 con 0x7fa09815de60 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 -- 172.21.15.77:0/866115538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 msgr2=0x7fa09815e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/866115538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09815e2f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa09806eac0 tx=0x7fa09002be40 comp rx=0 tx=0).stop 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 -- 172.21.15.77:0/866115538 shutdown_connections 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/866115538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09815e2f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/866115538 >> v2:172.21.15.158:3300/0 conn(0x7fa0980712d0 0x7fa098071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/866115538 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09815a220 0x7fa0980cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 -- 172.21.15.77:0/866115538 >> 172.21.15.77:0/866115538 conn(0x7fa09806b190 msgr2=0x7fa09806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 -- 172.21.15.77:0/866115538 shutdown_connections 2024-03-06T07:23:19.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.307+0000 7fa09cb3e640 1 -- 172.21.15.77:0/866115538 wait complete. 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.309+0000 7fa09cb3e640 1 Processor -- start 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.309+0000 7fa09cb3e640 1 -- start start 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.309+0000 7fa09cb3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0980712d0 0x7fa098149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.309+0000 7fa09cb3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09815a220 0x7fa098148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.309+0000 7fa09cb3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09806d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.309+0000 7fa09cb3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa098072030 con 0x7fa09815de60 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.309+0000 7fa09cb3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa098072bb0 con 0x7fa09815a220 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.309+0000 7fa09cb3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa098072530 con 0x7fa0980712d0 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa09659c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0980712d0 0x7fa098149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa096d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09806d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa095d9b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09815a220 0x7fa098148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa095d9b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09815a220 0x7fa098148810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:40264/0 (socket says 172.21.15.77:40264) 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa095d9b640 1 -- 172.21.15.77:0/1368814458 learned_addr learned my addr 172.21.15.77:0/1368814458 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa09659c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0980712d0 0x7fa098149180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:54646/0 (socket says 172.21.15.77:54646) 2024-03-06T07:23:19.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa096d9d640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09806d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35886/0 (socket says 172.21.15.77:35886) 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa096d9d640 1 -- 172.21.15.77:0/1368814458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0980712d0 msgr2=0x7fa098149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa096d9d640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0980712d0 0x7fa098149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa096d9d640 1 -- 172.21.15.77:0/1368814458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09815a220 msgr2=0x7fa098148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa096d9d640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09815a220 0x7fa098148810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa096d9d640 1 -- 172.21.15.77:0/1368814458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0981499d0 con 0x7fa09815de60 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa096d9d640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09806d880 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa090032000 tx=0x7fa090004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa08effd640 1 -- 172.21.15.77:0/1368814458 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090043020 con 0x7fa09815de60 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa08effd640 1 -- 172.21.15.77:0/1368814458 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09003e450 con 0x7fa09815de60 2024-03-06T07:23:19.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa098149cb0 con 0x7fa09815de60 2024-03-06T07:23:19.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa08effd640 1 -- 172.21.15.77:0/1368814458 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090009510 con 0x7fa09815de60 2024-03-06T07:23:19.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.310+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09814a270 con 0x7fa09815de60 2024-03-06T07:23:19.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.311+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0980732b0 con 0x7fa09815de60 2024-03-06T07:23:19.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.312+0000 7fa08effd640 1 -- 172.21.15.77:0/1368814458 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa090046020 con 0x7fa09815de60 2024-03-06T07:23:19.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.312+0000 7fa08effd640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa07806b500 0x7fa07806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.312+0000 7fa08effd640 1 -- 172.21.15.77:0/1368814458 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0900ac3a0 con 0x7fa09815de60 2024-03-06T07:23:19.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.312+0000 7fa09659c640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa07806b500 0x7fa07806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.313+0000 7fa09659c640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa07806b500 0x7fa07806d9c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa088005bc0 tx=0x7fa08800b750 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.317+0000 7fa08effd640 1 -- 172.21.15.77:0/1368814458 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa09004d070 con 0x7fa09815de60 2024-03-06T07:23:19.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.486+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa098068ff0 con 0x7fa09815de60 2024-03-06T07:23:19.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.486+0000 7fa08effd640 1 -- 172.21.15.77:0/1368814458 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa09007ac60 con 0x7fa09815de60 2024-03-06T07:23:19.488 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:23:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa07806b500 msgr2=0x7fa07806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa07806b500 0x7fa07806d9c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa088005bc0 tx=0x7fa08800b750 comp rx=0 tx=0).stop 2024-03-06T07:23:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 msgr2=0x7fa09806d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09806d880 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa090032000 tx=0x7fa090004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 shutdown_connections 2024-03-06T07:23:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa07806b500 0x7fa07806d9c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09815de60 0x7fa09806d880 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09815a220 0x7fa098148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 --2- 172.21.15.77:0/1368814458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0980712d0 0x7fa098149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 >> 172.21.15.77:0/1368814458 conn(0x7fa09806b190 msgr2=0x7fa0980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 shutdown_connections 2024-03-06T07:23:19.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.488+0000 7fa09cb3e640 1 -- 172.21.15.77:0/1368814458 wait complete. 2024-03-06T07:23:19.501 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.602+0000 7fe47959a640 1 Processor -- start 2024-03-06T07:23:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.602+0000 7fe47959a640 1 -- start start 2024-03-06T07:23:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe47959a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe4740b4970 0x7fe47406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe47959a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47415ff90 con 0x7fe47415bb40 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe47959a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe474160270 con 0x7fe47406d5c0 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe47959a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4740b4dd0 con 0x7fe4740b4970 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe472ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe47415bb40 0x7fe47415f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49582/0 (socket says 172.21.15.77:49582) 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe472ffd640 1 -- 172.21.15.77:0/4087075004 learned_addr learned my addr 172.21.15.77:0/4087075004 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe4727fc640 1 --2- 172.21.15.77:0/4087075004 >> v2:172.21.15.158:3300/0 conn(0x7fe4740b4970 0x7fe47406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe471ffb640 1 -- 172.21.15.77:0/4087075004 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750475427 0 0) 0x7fe47415ff90 con 0x7fe47415bb40 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe471ffb640 1 -- 172.21.15.77:0/4087075004 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe45c003520 con 0x7fe47415bb40 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe471ffb640 1 -- 172.21.15.77:0/4087075004 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757964639 0 0) 0x7fe474160270 con 0x7fe47406d5c0 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe471ffb640 1 -- 172.21.15.77:0/4087075004 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe47415ff90 con 0x7fe47406d5c0 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe4727fc640 1 -- 172.21.15.77:0/4087075004 >> v1:172.21.15.158:6789/0 conn(0x7fe47406d5c0 legacy=0x7fe4740cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe4727fc640 1 -- 172.21.15.77:0/4087075004 >> v1:172.21.15.77:6789/0 conn(0x7fe47415bb40 legacy=0x7fe47415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.603+0000 7fe4727fc640 1 -- 172.21.15.77:0/4087075004 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe474160550 con 0x7fe4740b4970 2024-03-06T07:23:19.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.604+0000 7fe4727fc640 1 --2- 172.21.15.77:0/4087075004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4740b4970 0x7fe47406d080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe4680097a0 tx=0x7fe46802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14762c5fe2de4a64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.604+0000 7fe471ffb640 1 -- 172.21.15.77:0/4087075004 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468034070 con 0x7fe4740b4970 2024-03-06T07:23:19.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.604+0000 7fe471ffb640 1 -- 172.21.15.77:0/4087075004 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468030610 con 0x7fe4740b4970 2024-03-06T07:23:19.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.604+0000 7fe471ffb640 1 -- 172.21.15.77:0/4087075004 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4680309c0 con 0x7fe4740b4970 2024-03-06T07:23:19.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.605+0000 7fe47959a640 1 -- 172.21.15.77:0/4087075004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4740b4970 msgr2=0x7fe47406d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.605+0000 7fe47959a640 1 --2- 172.21.15.77:0/4087075004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4740b4970 0x7fe47406d080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe4680097a0 tx=0x7fe46802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:19.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.605+0000 7fe47959a640 1 -- 172.21.15.77:0/4087075004 shutdown_connections 2024-03-06T07:23:19.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.605+0000 7fe47959a640 1 --2- 172.21.15.77:0/4087075004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4740b4970 0x7fe47406d080 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.605+0000 7fe47959a640 1 -- 172.21.15.77:0/4087075004 >> 172.21.15.77:0/4087075004 conn(0x7fe47406b190 msgr2=0x7fe47406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.605+0000 7fe47959a640 1 -- 172.21.15.77:0/4087075004 shutdown_connections 2024-03-06T07:23:19.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.605+0000 7fe47959a640 1 -- 172.21.15.77:0/4087075004 wait complete. 2024-03-06T07:23:19.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.606+0000 7fe47959a640 1 Processor -- start 2024-03-06T07:23:19.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.606+0000 7fe47959a640 1 -- start start 2024-03-06T07:23:19.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe47959a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe47406d5c0 0x7fe474148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe47959a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4740b4970 0x7fe4741482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe47959a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe47415bb40 0x7fe47415ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe47959a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47415bf20 con 0x7fe47415bb40 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe47959a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47406c910 con 0x7fe47406d5c0 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe47959a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47415c0a0 con 0x7fe4740b4970 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe47415bb40 0x7fe47415ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe47415bb40 0x7fe47415ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35890/0 (socket says 172.21.15.77:35890) 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4737fe640 1 -- 172.21.15.77:0/251193004 learned_addr learned my addr 172.21.15.77:0/251193004 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe472ffd640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe47406d5c0 0x7fe474148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4727fc640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4740b4970 0x7fe4741482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4737fe640 1 -- 172.21.15.77:0/251193004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4740b4970 msgr2=0x7fe4741482a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4737fe640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4740b4970 0x7fe4741482a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4737fe640 1 -- 172.21.15.77:0/251193004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe47406d5c0 msgr2=0x7fe474148b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4737fe640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe47406d5c0 0x7fe474148b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.607+0000 7fe4737fe640 1 -- 172.21.15.77:0/251193004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe474149090 con 0x7fe47415bb40 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.608+0000 7fe4737fe640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe47415bb40 0x7fe47415ea80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00c400 tx=0x7fe46c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.608+0000 7fe45b7fe640 1 -- 172.21.15.77:0/251193004 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c017070 con 0x7fe47415bb40 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.608+0000 7fe45b7fe640 1 -- 172.21.15.77:0/251193004 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c0052d0 con 0x7fe47415bb40 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.608+0000 7fe45b7fe640 1 -- 172.21.15.77:0/251193004 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c0037f0 con 0x7fe47415bb40 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.608+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe474070d60 con 0x7fe47415bb40 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.608+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe474071320 con 0x7fe47415bb40 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.609+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4740732b0 con 0x7fe47415bb40 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.609+0000 7fe45b7fe640 1 -- 172.21.15.77:0/251193004 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe46c01a070 con 0x7fe47415bb40 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.610+0000 7fe45b7fe640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46006b4d0 0x7fe46006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.610+0000 7fe472ffd640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46006b4d0 0x7fe46006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.610+0000 7fe45b7fe640 1 -- 172.21.15.77:0/251193004 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe46c089440 con 0x7fe47415bb40 2024-03-06T07:23:19.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.610+0000 7fe472ffd640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46006b4d0 0x7fe46006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe464001560 tx=0x7fe4640019f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.614+0000 7fe45b7fe640 1 -- 172.21.15.77:0/251193004 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe46c057d00 con 0x7fe47415bb40 2024-03-06T07:23:19.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.786+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe47406e130 con 0x7fe47415bb40 2024-03-06T07:23:19.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.786+0000 7fe45b7fe640 1 -- 172.21.15.77:0/251193004 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe46c05bda0 con 0x7fe47415bb40 2024-03-06T07:23:19.788 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:23:19.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46006b4d0 msgr2=0x7fe46006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46006b4d0 0x7fe46006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe464001560 tx=0x7fe4640019f0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe47415bb40 msgr2=0x7fe47415ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe47415bb40 0x7fe47415ea80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00c400 tx=0x7fe46c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 shutdown_connections 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46006b4d0 0x7fe46006d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe47415bb40 0x7fe47415ea80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4740b4970 0x7fe4741482a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 --2- 172.21.15.77:0/251193004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe47406d5c0 0x7fe474148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.788+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 >> 172.21.15.77:0/251193004 conn(0x7fe47406b190 msgr2=0x7fe474154bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.789+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 shutdown_connections 2024-03-06T07:23:19.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.789+0000 7fe47959a640 1 -- 172.21.15.77:0/251193004 wait complete. 2024-03-06T07:23:19.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.899+0000 7fdda15c7640 1 Processor -- start 2024-03-06T07:23:19.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.899+0000 7fdda15c7640 1 -- start start 2024-03-06T07:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdda15c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdd9c0b3060 0x7fdd9c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdda15c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c15bf30 0x7fdd9c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdda15c7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c15f2d0 con 0x7fdd9c13dc10 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdda15c7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd9c0cf3f0 con 0x7fdd9c0b3060 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdda15c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c15b210 con 0x7fdd9c15bf30 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd9a7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdd9c0b3060 0x7fdd9c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd9a7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdd9c0b3060 0x7fdd9c15e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35892/0 (socket says 172.21.15.77:35892) 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd9a7fc640 1 -- 172.21.15.77:0/118564660 learned_addr learned my addr 172.21.15.77:0/118564660 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd9b7fe640 1 --2- 172.21.15.77:0/118564660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c15bf30 0x7fdd9c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd9a7fc640 1 -- 172.21.15.77:0/118564660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c15bf30 msgr2=0x7fdd9c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd9a7fc640 1 --2- 172.21.15.77:0/118564660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c15bf30 0x7fdd9c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd9a7fc640 1 -- 172.21.15.77:0/118564660 >> v1:172.21.15.158:6789/0 conn(0x7fdd9c13dc10 legacy=0x7fdd9c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd9a7fc640 1 -- 172.21.15.77:0/118564660 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c15f580 con 0x7fdd9c0b3060 2024-03-06T07:23:19.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.900+0000 7fdd99ffb640 1 -- 172.21.15.77:0/118564660 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254334961 0 0) 0x7fdd9c15f2d0 con 0x7fdd9c13dc10 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.901+0000 7fdd9a7fc640 1 --2- 172.21.15.77:0/118564660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c0b3060 0x7fdd9c15e850 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000bb10 tx=0x7fdd9002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc053fb489219c52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.901+0000 7fdd99ffb640 1 -- 172.21.15.77:0/118564660 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9003b070 con 0x7fdd9c0b3060 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.901+0000 7fdd99ffb640 1 -- 172.21.15.77:0/118564660 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd90036440 con 0x7fdd9c0b3060 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.901+0000 7fdd99ffb640 1 -- 172.21.15.77:0/118564660 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90036810 con 0x7fdd9c0b3060 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.902+0000 7fdda15c7640 1 -- 172.21.15.77:0/118564660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c0b3060 msgr2=0x7fdd9c15e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.902+0000 7fdda15c7640 1 --2- 172.21.15.77:0/118564660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c0b3060 0x7fdd9c15e850 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000bb10 tx=0x7fdd9002e0c0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.902+0000 7fdda15c7640 1 -- 172.21.15.77:0/118564660 shutdown_connections 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.902+0000 7fdda15c7640 1 --2- 172.21.15.77:0/118564660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c15bf30 0x7fdd9c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.902+0000 7fdda15c7640 1 --2- 172.21.15.77:0/118564660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c0b3060 0x7fdd9c15e850 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.902+0000 7fdda15c7640 1 -- 172.21.15.77:0/118564660 >> 172.21.15.77:0/118564660 conn(0x7fdd9c0cf8a0 msgr2=0x7fdd9c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.902+0000 7fdda15c7640 1 -- 172.21.15.77:0/118564660 shutdown_connections 2024-03-06T07:23:19.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.902+0000 7fdda15c7640 1 -- 172.21.15.77:0/118564660 wait complete. 2024-03-06T07:23:19.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.903+0000 7fdda15c7640 1 Processor -- start 2024-03-06T07:23:19.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.903+0000 7fdda15c7640 1 -- start start 2024-03-06T07:23:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdda15c7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd9c0b3060 0x7fdd9c152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdda15c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c13dc10 0x7fdd9c12f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdda15c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c15bf30 0x7fdd9c071d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdda15c7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c13dff0 con 0x7fdd9c15bf30 2024-03-06T07:23:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdda15c7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c15b730 con 0x7fdd9c0b3060 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdda15c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c0cd7a0 con 0x7fdd9c13dc10 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c15bf30 0x7fdd9c071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c15bf30 0x7fdd9c071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35898/0 (socket says 172.21.15.77:35898) 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 -- 172.21.15.77:0/3930208370 learned_addr learned my addr 172.21.15.77:0/3930208370 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9a7fc640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c13dc10 0x7fdd9c12f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9affd640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd9c0b3060 0x7fdd9c152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 -- 172.21.15.77:0/3930208370 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c13dc10 msgr2=0x7fdd9c12f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c13dc10 0x7fdd9c12f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 -- 172.21.15.77:0/3930208370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd9c0b3060 msgr2=0x7fdd9c152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd9c0b3060 0x7fdd9c152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 -- 172.21.15.77:0/3930208370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c152960 con 0x7fdd9c15bf30 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.904+0000 7fdd9b7fe640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c15bf30 0x7fdd9c071d00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400e820 tx=0x7fdd9400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.905+0000 7fdd837fe640 1 -- 172.21.15.77:0/3930208370 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd94017070 con 0x7fdd9c15bf30 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.905+0000 7fdd837fe640 1 -- 172.21.15.77:0/3930208370 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9400ee70 con 0x7fdd9c15bf30 2024-03-06T07:23:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.905+0000 7fdd837fe640 1 -- 172.21.15.77:0/3930208370 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9400cb60 con 0x7fdd9c15bf30 2024-03-06T07:23:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.905+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c152c40 con 0x7fdd9c15bf30 2024-03-06T07:23:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.905+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c153200 con 0x7fdd9c15bf30 2024-03-06T07:23:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.906+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c141300 con 0x7fdd9c15bf30 2024-03-06T07:23:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.906+0000 7fdd837fe640 1 -- 172.21.15.77:0/3930208370 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd9401a070 con 0x7fdd9c15bf30 2024-03-06T07:23:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.906+0000 7fdd837fe640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd8806b4d0 0x7fdd8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.906+0000 7fdd837fe640 1 -- 172.21.15.77:0/3930208370 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd9408a320 con 0x7fdd9c15bf30 2024-03-06T07:23:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.906+0000 7fdd9affd640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd8806b4d0 0x7fdd8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.907+0000 7fdd9affd640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd8806b4d0 0x7fdd8806d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0012e0 tx=0x7fdd8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:19.911+0000 7fdd837fe640 1 -- 172.21.15.77:0/3930208370 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd9c141300 con 0x7fdd9c15bf30 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.117+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd9c06eb00 con 0x7fdd9c15bf30 2024-03-06T07:23:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.117+0000 7fdd837fe640 1 -- 172.21.15.77:0/3930208370 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fdd94058be0 con 0x7fdd9c15bf30 2024-03-06T07:23:20.119 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:20.119 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:20.120 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd8806b4d0 msgr2=0x7fdd8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd8806b4d0 0x7fdd8806d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0012e0 tx=0x7fdd8c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c15bf30 msgr2=0x7fdd9c071d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c15bf30 0x7fdd9c071d00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400e820 tx=0x7fdd9400ec20 comp rx=0 tx=0).stop 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 shutdown_connections 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd8806b4d0 0x7fdd8806d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd9c15bf30 0x7fdd9c071d00 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd9c13dc10 0x7fdd9c12f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 --2- 172.21.15.77:0/3930208370 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd9c0b3060 0x7fdd9c152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 >> 172.21.15.77:0/3930208370 conn(0x7fdd9c0cf8a0 msgr2=0x7fdd9c13ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 shutdown_connections 2024-03-06T07:23:20.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.119+0000 7fdda15c7640 1 -- 172.21.15.77:0/3930208370 wait complete. 2024-03-06T07:23:20.132 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:20.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.230+0000 7f60c4b77640 1 Processor -- start 2024-03-06T07:23:20.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.230+0000 7f60c4b77640 1 -- start start 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60c4b77640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60c006da40 0x7f60c00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60c4b77640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c00cc980 con 0x7f60c00cd7a0 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60c4b77640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c00ccc60 con 0x7f60c01424f0 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60c4b77640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c00cd220 con 0x7f60c006da40 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60b7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60c006da40 0x7f60c00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60bed9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f60c00cd7a0 0x7f60c0071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49588/0 (socket says 172.21.15.77:49588) 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60bed9d640 1 -- 172.21.15.77:0/256810081 learned_addr learned my addr 172.21.15.77:0/256810081 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60b7fff640 1 -- 172.21.15.77:0/256810081 >> v1:172.21.15.158:6789/0 conn(0x7f60c01424f0 legacy=0x7f60c006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60b7fff640 1 -- 172.21.15.77:0/256810081 >> v1:172.21.15.77:6789/0 conn(0x7f60c00cd7a0 legacy=0x7f60c0071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60bdd9b640 1 -- 172.21.15.77:0/256810081 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814516230 0 0) 0x7f60c00ccc60 con 0x7f60c01424f0 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.231+0000 7f60b7fff640 1 -- 172.21.15.77:0/256810081 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c00ccf40 con 0x7f60c006da40 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.232+0000 7f60b7fff640 1 --2- 172.21.15.77:0/256810081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c006da40 0x7f60c00cc2f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f60a80097a0 tx=0x7f60a802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4eda463c7d42c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.232+0000 7f60bdd9b640 1 -- 172.21.15.77:0/256810081 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8037070 con 0x7f60c006da40 2024-03-06T07:23:20.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.232+0000 7f60bdd9b640 1 -- 172.21.15.77:0/256810081 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a8030710 con 0x7f60c006da40 2024-03-06T07:23:20.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.232+0000 7f60bdd9b640 1 -- 172.21.15.77:0/256810081 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8030ae0 con 0x7f60c006da40 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.233+0000 7f60c4b77640 1 -- 172.21.15.77:0/256810081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c006da40 msgr2=0x7f60c00cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.233+0000 7f60c4b77640 1 --2- 172.21.15.77:0/256810081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c006da40 0x7f60c00cc2f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f60a80097a0 tx=0x7f60a802b790 comp rx=0 tx=0).stop 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.233+0000 7f60c4b77640 1 -- 172.21.15.77:0/256810081 shutdown_connections 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.233+0000 7f60c4b77640 1 --2- 172.21.15.77:0/256810081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c006da40 0x7f60c00cc2f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.233+0000 7f60c4b77640 1 -- 172.21.15.77:0/256810081 >> 172.21.15.77:0/256810081 conn(0x7f60c00cf8a0 msgr2=0x7f60c00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.233+0000 7f60c4b77640 1 -- 172.21.15.77:0/256810081 shutdown_connections 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.233+0000 7f60c4b77640 1 -- 172.21.15.77:0/256810081 wait complete. 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.234+0000 7f60c4b77640 1 Processor -- start 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.234+0000 7f60c4b77640 1 -- start start 2024-03-06T07:23:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60c4b77640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006da40 0x7f60c01593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60c4b77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c00cd7a0 0x7f60c0134cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60c4b77640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60c01424f0 0x7f60c01351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60c4b77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c015d070 con 0x7f60c00cd7a0 2024-03-06T07:23:20.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60c4b77640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c015cef0 con 0x7f60c006da40 2024-03-06T07:23:20.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60c4b77640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c015d1f0 con 0x7f60c01424f0 2024-03-06T07:23:20.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c00cd7a0 0x7f60c0134cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c00cd7a0 0x7f60c0134cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35920/0 (socket says 172.21.15.77:35920) 2024-03-06T07:23:20.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 -- 172.21.15.77:0/3715424212 learned_addr learned my addr 172.21.15.77:0/3715424212 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60bed9d640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60c01424f0 0x7f60c01351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60be59c640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006da40 0x7f60c01593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 -- 172.21.15.77:0/3715424212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60c01424f0 msgr2=0x7f60c01351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60c01424f0 0x7f60c01351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 -- 172.21.15.77:0/3715424212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006da40 msgr2=0x7f60c01593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006da40 0x7f60c01593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 -- 172.21.15.77:0/3715424212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c0155b40 con 0x7f60c00cd7a0 2024-03-06T07:23:20.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b7fff640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c00cd7a0 0x7f60c0134cb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f60a80097a0 tx=0x7f60a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b6ffd640 1 -- 172.21.15.77:0/3715424212 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8037040 con 0x7f60c00cd7a0 2024-03-06T07:23:20.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b6ffd640 1 -- 172.21.15.77:0/3715424212 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a80329c0 con 0x7f60c00cd7a0 2024-03-06T07:23:20.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c0155e20 con 0x7f60c00cd7a0 2024-03-06T07:23:20.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60b6ffd640 1 -- 172.21.15.77:0/3715424212 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8032d90 con 0x7f60c00cd7a0 2024-03-06T07:23:20.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.235+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c01563e0 con 0x7f60c00cd7a0 2024-03-06T07:23:20.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.237+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c004f330 con 0x7f60c00cd7a0 2024-03-06T07:23:20.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.237+0000 7f60b6ffd640 1 -- 172.21.15.77:0/3715424212 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60a8004030 con 0x7f60c00cd7a0 2024-03-06T07:23:20.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.237+0000 7f60b6ffd640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f60a006b4d0 0x7f60a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.237+0000 7f60b6ffd640 1 -- 172.21.15.77:0/3715424212 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60a80abdc0 con 0x7f60c00cd7a0 2024-03-06T07:23:20.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.237+0000 7f60be59c640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f60a006b4d0 0x7f60a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.238+0000 7f60be59c640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f60a006b4d0 0x7f60a006d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f60b0000f60 tx=0x7f60b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.241+0000 7f60b6ffd640 1 -- 172.21.15.77:0/3715424212 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60a8042020 con 0x7f60c00cd7a0 2024-03-06T07:23:20.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.417+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60c006d280 con 0x7f60c00cd7a0 2024-03-06T07:23:20.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.418+0000 7f60b6ffd640 1 -- 172.21.15.77:0/3715424212 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f60a807a680 con 0x7f60c00cd7a0 2024-03-06T07:23:20.421 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:20.421 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:58.329001+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:54.903388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.652222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f60a006b4d0 msgr2=0x7f60a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f60a006b4d0 0x7f60a006d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f60b0000f60 tx=0x7f60b0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c00cd7a0 msgr2=0x7f60c0134cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c00cd7a0 0x7f60c0134cb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f60a80097a0 tx=0x7f60a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 shutdown_connections 2024-03-06T07:23:20.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f60a006b4d0 0x7f60a006d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60c00cd7a0 0x7f60c0134cb0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60c01424f0 0x7f60c01351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 --2- 172.21.15.77:0/3715424212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60c006da40 0x7f60c01593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 >> 172.21.15.77:0/3715424212 conn(0x7f60c00cf8a0 msgr2=0x7f60c015f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 shutdown_connections 2024-03-06T07:23:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.420+0000 7f60c4b77640 1 -- 172.21.15.77:0/3715424212 wait complete. 2024-03-06T07:23:20.433 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:20.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:20.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.532+0000 7f02dd29a640 1 Processor -- start 2024-03-06T07:23:20.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.533+0000 7f02dd29a640 1 -- start start 2024-03-06T07:23:20.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.533+0000 7f02dd29a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02d80d4920 0x7f02d80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.533+0000 7f02dd29a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d80ceac0 con 0x7f02d81625a0 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.533+0000 7f02dd29a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d80d2dc0 con 0x7f02d81621c0 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.533+0000 7f02dd29a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d80cf690 con 0x7f02d80d4920 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.533+0000 7f02d67fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02d80d4920 0x7f02d80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.533+0000 7f02d77fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f02d81625a0 0x7f02d815dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49594/0 (socket says 172.21.15.77:49594) 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.534+0000 7f02d77fe640 1 -- 172.21.15.77:0/543087457 learned_addr learned my addr 172.21.15.77:0/543087457 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.534+0000 7f02d67fc640 1 -- 172.21.15.77:0/543087457 >> v1:172.21.15.158:6789/0 conn(0x7f02d81621c0 legacy=0x7f02d80d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.534+0000 7f02d67fc640 1 -- 172.21.15.77:0/543087457 >> v1:172.21.15.77:6789/0 conn(0x7f02d81625a0 legacy=0x7f02d815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.534+0000 7f02d67fc640 1 -- 172.21.15.77:0/543087457 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d80ced60 con 0x7f02d80d4920 2024-03-06T07:23:20.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.534+0000 7f02d67fc640 1 --2- 172.21.15.77:0/543087457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d80d4920 0x7f02d80cfb20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0097a0 tx=0x7f02cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb378cc16b2a64b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02d5ffb640 1 -- 172.21.15.77:0/543087457 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc035020 con 0x7f02d80d4920 2024-03-06T07:23:20.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02d5ffb640 1 -- 172.21.15.77:0/543087457 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc031830 con 0x7f02d80d4920 2024-03-06T07:23:20.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02d5ffb640 1 -- 172.21.15.77:0/543087457 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc031c00 con 0x7f02d80d4920 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02dd29a640 1 -- 172.21.15.77:0/543087457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d80d4920 msgr2=0x7f02d80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02dd29a640 1 --2- 172.21.15.77:0/543087457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d80d4920 0x7f02d80cfb20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0097a0 tx=0x7f02cc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02dd29a640 1 -- 172.21.15.77:0/543087457 shutdown_connections 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02dd29a640 1 --2- 172.21.15.77:0/543087457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d80d4920 0x7f02d80cfb20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02dd29a640 1 -- 172.21.15.77:0/543087457 >> 172.21.15.77:0/543087457 conn(0x7f02d80d3d10 msgr2=0x7f02d80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02dd29a640 1 -- 172.21.15.77:0/543087457 shutdown_connections 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.535+0000 7f02dd29a640 1 -- 172.21.15.77:0/543087457 wait complete. 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.536+0000 7f02dd29a640 1 Processor -- start 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02dd29a640 1 -- start start 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02dd29a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f02d80d4920 0x7f02d81f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02dd29a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d81621c0 0x7f02d80b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02dd29a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d81625a0 0x7f02d806c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02dd29a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d815e450 con 0x7f02d81621c0 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02dd29a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d80cf320 con 0x7f02d81625a0 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02dd29a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d815e5d0 con 0x7f02d80d4920 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02d67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d81621c0 0x7f02d80b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02d67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d81621c0 0x7f02d80b65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35948/0 (socket says 172.21.15.77:35948) 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02d67fc640 1 -- 172.21.15.77:0/3564286610 learned_addr learned my addr 172.21.15.77:0/3564286610 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02d77fe640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d81625a0 0x7f02d806c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.537+0000 7f02d6ffd640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f02d80d4920 0x7f02d81f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02d67fc640 1 -- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f02d80d4920 msgr2=0x7f02d81f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02d67fc640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f02d80d4920 0x7f02d81f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02d67fc640 1 -- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d81625a0 msgr2=0x7f02d806c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02d67fc640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d81625a0 0x7f02d806c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02d67fc640 1 -- 172.21.15.77:0/3564286610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d80b78f0 con 0x7f02d81621c0 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02d6ffd640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f02d80d4920 0x7f02d81f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02d67fc640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d81621c0 0x7f02d80b65d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f02cc02bee0 tx=0x7f02cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3564286610 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc041070 con 0x7f02d81621c0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3564286610 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc03a950 con 0x7f02d81621c0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d80b7050 con 0x7f02d81621c0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3564286610 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc035040 con 0x7f02d81621c0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.538+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d80b7610 con 0x7f02d81621c0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.539+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d804f330 con 0x7f02d81621c0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.539+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3564286610 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02cc04c060 con 0x7f02d81621c0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.540+0000 7f02bf7fe640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02c406b4d0 0x7f02c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.540+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3564286610 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02cc0abb60 con 0x7f02d81621c0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.540+0000 7f02d6ffd640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02c406b4d0 0x7f02c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.540+0000 7f02d6ffd640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02c406b4d0 0x7f02c406d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02d815f870 tx=0x7f02c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.544+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3564286610 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02cc07aa70 con 0x7f02d81621c0 2024-03-06T07:23:20.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.752+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02d8073380 con 0x7f02d81621c0 2024-03-06T07:23:20.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.752+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3564286610 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f02cc07a370 con 0x7f02d81621c0 2024-03-06T07:23:20.754 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:20.755 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02c406b4d0 msgr2=0x7f02c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02c406b4d0 0x7f02c406d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02d815f870 tx=0x7f02c8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d81621c0 msgr2=0x7f02d80b65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d81621c0 0x7f02d80b65d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f02cc02bee0 tx=0x7f02cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 shutdown_connections 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02c406b4d0 0x7f02c406d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02d81621c0 0x7f02d80b65d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f02d81625a0 0x7f02d806c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 --2- 172.21.15.77:0/3564286610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f02d80d4920 0x7f02d81f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 >> 172.21.15.77:0/3564286610 conn(0x7f02d80d3d10 msgr2=0x7f02d8160d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.754+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 shutdown_connections 2024-03-06T07:23:20.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.755+0000 7f02dd29a640 1 -- 172.21.15.77:0/3564286610 wait complete. 2024-03-06T07:23:20.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:20.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.865+0000 7f5ab1bac640 1 Processor -- start 2024-03-06T07:23:20.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.865+0000 7f5ab1bac640 1 -- start start 2024-03-06T07:23:20.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.865+0000 7f5ab1bac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5aac1515e0 0x7f5aac0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5ab1bac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5aac0cd7a0 0x7f5aac0cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5ab1bac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac071f40 con 0x7f5aac15b430 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5ab1bac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aac15b8e0 con 0x7f5aac1515e0 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5ab1bac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aac070a70 con 0x7f5aac0cd7a0 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5aab7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5aac1515e0 0x7f5aac0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5aab7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5aac1515e0 0x7f5aac0cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35962/0 (socket says 172.21.15.77:35962) 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5aab7fe640 1 -- 172.21.15.77:0/398023770 learned_addr learned my addr 172.21.15.77:0/398023770 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5aabfff640 1 --2- 172.21.15.77:0/398023770 >> v2:172.21.15.158:3300/0 conn(0x7f5aac0cd7a0 0x7f5aac0cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5aab7fe640 1 -- 172.21.15.77:0/398023770 >> v2:172.21.15.158:3300/0 conn(0x7f5aac0cd7a0 msgr2=0x7f5aac0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5aab7fe640 1 --2- 172.21.15.77:0/398023770 >> v2:172.21.15.158:3300/0 conn(0x7f5aac0cd7a0 0x7f5aac0cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5aab7fe640 1 -- 172.21.15.77:0/398023770 >> v1:172.21.15.158:6789/0 conn(0x7f5aac15b430 legacy=0x7f5aac071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.866+0000 7f5aab7fe640 1 -- 172.21.15.77:0/398023770 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac070d50 con 0x7f5aac1515e0 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5aab7fe640 1 --2- 172.21.15.77:0/398023770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac0cce40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0cc340 tx=0x7f5aa002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f23ddaefb05b2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5aaa7fc640 1 -- 172.21.15.77:0/398023770 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0034070 con 0x7f5aac1515e0 2024-03-06T07:23:20.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5aaa7fc640 1 -- 172.21.15.77:0/398023770 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa0030860 con 0x7f5aac1515e0 2024-03-06T07:23:20.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5aaa7fc640 1 -- 172.21.15.77:0/398023770 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0030c30 con 0x7f5aac1515e0 2024-03-06T07:23:20.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5ab1bac640 1 -- 172.21.15.77:0/398023770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 msgr2=0x7f5aac0cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/398023770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac0cce40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0cc340 tx=0x7f5aa002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5ab1bac640 1 -- 172.21.15.77:0/398023770 shutdown_connections 2024-03-06T07:23:20.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/398023770 >> v2:172.21.15.158:3300/0 conn(0x7f5aac0cd7a0 0x7f5aac0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/398023770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac0cce40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5ab1bac640 1 -- 172.21.15.77:0/398023770 >> 172.21.15.77:0/398023770 conn(0x7f5aac0cf8a0 msgr2=0x7f5aac0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5ab1bac640 1 -- 172.21.15.77:0/398023770 shutdown_connections 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.867+0000 7f5ab1bac640 1 -- 172.21.15.77:0/398023770 wait complete. 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.868+0000 7f5ab1bac640 1 Processor -- start 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.869+0000 7f5ab1bac640 1 -- start start 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.869+0000 7f5ab1bac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5aac0cd7a0 0x7f5aac140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.869+0000 7f5ab1bac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.869+0000 7f5ab1bac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aac15b430 0x7f5aac159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.869+0000 7f5ab1bac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0ce0c0 con 0x7f5aac1515e0 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.869+0000 7f5ab1bac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac071310 con 0x7f5aac15b430 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.869+0000 7f5ab1bac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0cc3a0 con 0x7f5aac0cd7a0 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.869+0000 7f5aaaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aaaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35970/0 (socket says 172.21.15.77:35970) 2024-03-06T07:23:20.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aaaffd640 1 -- 172.21.15.77:0/3662137873 learned_addr learned my addr 172.21.15.77:0/3662137873 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aab7fe640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5aac0cd7a0 0x7f5aac140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aabfff640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aac15b430 0x7f5aac159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aaaffd640 1 -- 172.21.15.77:0/3662137873 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5aac0cd7a0 msgr2=0x7f5aac140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aaaffd640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5aac0cd7a0 0x7f5aac140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aaaffd640 1 -- 172.21.15.77:0/3662137873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aac15b430 msgr2=0x7f5aac159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aaaffd640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aac15b430 0x7f5aac159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aaaffd640 1 -- 172.21.15.77:0/3662137873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac1662b0 con 0x7f5aac1515e0 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5aaaffd640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac159400 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00c340 tx=0x7f5a9c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5a93fff640 1 -- 172.21.15.77:0/3662137873 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c017070 con 0x7f5aac1515e0 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5a93fff640 1 -- 172.21.15.77:0/3662137873 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c0052d0 con 0x7f5aac1515e0 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac166590 con 0x7f5aac1515e0 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5a93fff640 1 -- 172.21.15.77:0/3662137873 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c003850 con 0x7f5aac1515e0 2024-03-06T07:23:20.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.870+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac166b50 con 0x7f5aac1515e0 2024-03-06T07:23:20.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.871+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aac068840 con 0x7f5aac1515e0 2024-03-06T07:23:20.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.872+0000 7f5a93fff640 1 -- 172.21.15.77:0/3662137873 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a9c01a070 con 0x7f5aac1515e0 2024-03-06T07:23:20.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.872+0000 7f5a93fff640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a9806b4d0 0x7f5a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.872+0000 7f5a93fff640 1 -- 172.21.15.77:0/3662137873 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a9c0894f0 con 0x7f5aac1515e0 2024-03-06T07:23:20.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.872+0000 7f5aab7fe640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a9806b4d0 0x7f5a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.872+0000 7f5aab7fe640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a9806b4d0 0x7f5a9806d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0036000 tx=0x7f5aa0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:20.876+0000 7f5a93fff640 1 -- 172.21.15.77:0/3662137873 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a9c057c90 con 0x7f5aac1515e0 2024-03-06T07:23:21.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.083+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5aac04f330 con 0x7f5aac1515e0 2024-03-06T07:23:21.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.084+0000 7f5a93fff640 1 -- 172.21.15.77:0/3662137873 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f5a9c05bd30 con 0x7f5aac1515e0 2024-03-06T07:23:21.086 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:21.086 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:56.676746+0000","modified":"2024-03-06T07:22:58.684340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/929839050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":929839050},{"type":"v1","addr":"172.21.15.158:6834","nonce":929839050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5239,"qdb_cluster":[5239]},"id":2}]} 2024-03-06T07:23:21.086 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-06T07:23:21.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a9806b4d0 msgr2=0x7f5a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a9806b4d0 0x7f5a9806d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0036000 tx=0x7f5aa0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:21.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 msgr2=0x7f5aac159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac159400 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00c340 tx=0x7f5a9c00c800 comp rx=0 tx=0).stop 2024-03-06T07:23:21.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 shutdown_connections 2024-03-06T07:23:21.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a9806b4d0 0x7f5a9806d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac1515e0 0x7f5aac159400 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5aac15b430 0x7f5aac159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 --2- 172.21.15.77:0/3662137873 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5aac0cd7a0 0x7f5aac140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 >> 172.21.15.77:0/3662137873 conn(0x7f5aac0cf8a0 msgr2=0x7f5aac0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 shutdown_connections 2024-03-06T07:23:21.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.086+0000 7f5ab1bac640 1 -- 172.21.15.77:0/3662137873 wait complete. 2024-03-06T07:23:21.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.198+0000 7f11c00a4640 1 Processor -- start 2024-03-06T07:23:21.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.198+0000 7f11c00a4640 1 -- start start 2024-03-06T07:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11c00a4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11b815bf30 0x7f11b815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11c00a4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b815f580 con 0x7f11b813dc10 2024-03-06T07:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11c00a4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b815b450 con 0x7f11b80b3060 2024-03-06T07:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11c00a4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b80cf3f0 con 0x7f11b815bf30 2024-03-06T07:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11be641640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11b815bf30 0x7f11b815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11bde40640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f11b813dc10 0x7f11b80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49598/0 (socket says 172.21.15.77:49598) 2024-03-06T07:23:21.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11bde40640 1 -- 172.21.15.77:0/1600140022 learned_addr learned my addr 172.21.15.77:0/1600140022 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:21.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11be641640 1 -- 172.21.15.77:0/1600140022 >> v1:172.21.15.158:6789/0 conn(0x7f11b80b3060 legacy=0x7f11b815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11be641640 1 -- 172.21.15.77:0/1600140022 >> v1:172.21.15.77:6789/0 conn(0x7f11b813dc10 legacy=0x7f11b80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11bce3e640 1 -- 172.21.15.77:0/1600140022 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313408094 0 0) 0x7f11b815f580 con 0x7f11b813dc10 2024-03-06T07:23:21.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.199+0000 7f11be641640 1 -- 172.21.15.77:0/1600140022 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b815b730 con 0x7f11b815bf30 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.200+0000 7f11be641640 1 --2- 172.21.15.77:0/1600140022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b815ef60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009d90 tx=0x7f11b002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c43bbabf4ab088 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.200+0000 7f11bce3e640 1 -- 172.21.15.77:0/1600140022 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0039070 con 0x7f11b815bf30 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.200+0000 7f11bce3e640 1 -- 172.21.15.77:0/1600140022 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b002cc60 con 0x7f11b815bf30 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.200+0000 7f11bce3e640 1 -- 172.21.15.77:0/1600140022 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0034660 con 0x7f11b815bf30 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.201+0000 7f11c00a4640 1 -- 172.21.15.77:0/1600140022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 msgr2=0x7f11b815ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.201+0000 7f11c00a4640 1 --2- 172.21.15.77:0/1600140022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b815ef60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009d90 tx=0x7f11b002beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.201+0000 7f11c00a4640 1 -- 172.21.15.77:0/1600140022 shutdown_connections 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.201+0000 7f11c00a4640 1 --2- 172.21.15.77:0/1600140022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b815ef60 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.201+0000 7f11c00a4640 1 -- 172.21.15.77:0/1600140022 >> 172.21.15.77:0/1600140022 conn(0x7f11b80cf8a0 msgr2=0x7f11b80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.201+0000 7f11c00a4640 1 -- 172.21.15.77:0/1600140022 shutdown_connections 2024-03-06T07:23:21.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.201+0000 7f11c00a4640 1 -- 172.21.15.77:0/1600140022 wait complete. 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.202+0000 7f11c00a4640 1 Processor -- start 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.202+0000 7f11c00a4640 1 -- start start 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.202+0000 7f11c00a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11b80b3060 0x7f11b813cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11c00a4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11b813dc10 0x7f11b8142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11c00a4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b8142dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11c00a4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80718e0 con 0x7f11b815bf30 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11c00a4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b815b210 con 0x7f11b813dc10 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11c00a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8071a60 con 0x7f11b80b3060 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b8142dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b8142dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35990/0 (socket says 172.21.15.77:35990) 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 -- 172.21.15.77:0/1317828242 learned_addr learned my addr 172.21.15.77:0/1317828242 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11bd63f640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11b813dc10 0x7f11b8142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11bde40640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11b80b3060 0x7f11b813cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 -- 172.21.15.77:0/1317828242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11b80b3060 msgr2=0x7f11b813cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11b80b3060 0x7f11b813cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 -- 172.21.15.77:0/1317828242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11b813dc10 msgr2=0x7f11b8142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11b813dc10 0x7f11b8142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 -- 172.21.15.77:0/1317828242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b813d7c0 con 0x7f11b815bf30 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11be641640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b8142dd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009d90 tx=0x7f11b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11ae7fc640 1 -- 172.21.15.77:0/1317828242 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0039050 con 0x7f11b815bf30 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11ae7fc640 1 -- 172.21.15.77:0/1317828242 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b003a040 con 0x7f11b815bf30 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b8143450 con 0x7f11b815bf30 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11ae7fc640 1 -- 172.21.15.77:0/1317828242 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b003e610 con 0x7f11b815bf30 2024-03-06T07:23:21.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.203+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b813f090 con 0x7f11b815bf30 2024-03-06T07:23:21.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.204+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b804f330 con 0x7f11b815bf30 2024-03-06T07:23:21.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.205+0000 7f11ae7fc640 1 -- 172.21.15.77:0/1317828242 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11b0047020 con 0x7f11b815bf30 2024-03-06T07:23:21.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.205+0000 7f11ae7fc640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f119c06b4d0 0x7f119c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.205+0000 7f11bde40640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f119c06b4d0 0x7f119c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.205+0000 7f11ae7fc640 1 -- 172.21.15.77:0/1317828242 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11b00ac450 con 0x7f11b815bf30 2024-03-06T07:23:21.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.206+0000 7f11bde40640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f119c06b4d0 0x7f119c06d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f11b4007f70 tx=0x7f11b4007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.210+0000 7f11ae7fc640 1 -- 172.21.15.77:0/1317828242 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11b004e020 con 0x7f11b815bf30 2024-03-06T07:23:21.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.385+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11b806eb00 con 0x7f11b815bf30 2024-03-06T07:23:21.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.386+0000 7f11ae7fc640 1 -- 172.21.15.77:0/1317828242 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f11b8143450 con 0x7f11b815bf30 2024-03-06T07:23:21.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:21.388 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:22:58.329001+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:54.903388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:55.652222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:21.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f119c06b4d0 msgr2=0x7f119c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f119c06b4d0 0x7f119c06d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f11b4007f70 tx=0x7f11b4007f00 comp rx=0 tx=0).stop 2024-03-06T07:23:21.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 msgr2=0x7f11b8142dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b8142dd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009d90 tx=0x7f11b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 shutdown_connections 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f119c06b4d0 0x7f119c06d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11b815bf30 0x7f11b8142dd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11b813dc10 0x7f11b8142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 --2- 172.21.15.77:0/1317828242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11b80b3060 0x7f11b813cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 >> 172.21.15.77:0/1317828242 conn(0x7f11b80cf8a0 msgr2=0x7f11b806c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 shutdown_connections 2024-03-06T07:23:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.388+0000 7f11c00a4640 1 -- 172.21.15.77:0/1317828242 wait complete. 2024-03-06T07:23:21.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:21.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.502+0000 7fb04e437640 1 Processor -- start 2024-03-06T07:23:21.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.502+0000 7fb04e437640 1 -- start start 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb04e437640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb04815e010 0x7fb0480cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb04e437640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0480b3240 0x7fb0480b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb04e437640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb0480b2910 0x7fb0480d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb04e437640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0480b1f40 con 0x7fb0480b3240 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb04e437640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04815e3f0 con 0x7fb0480b2910 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb04e437640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0480b1c10 con 0x7fb04815e010 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb0477fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0480b3240 0x7fb0480b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb0477fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0480b3240 0x7fb0480b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36006/0 (socket says 172.21.15.77:36006) 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb0477fe640 1 -- 172.21.15.77:0/1370952320 learned_addr learned my addr 172.21.15.77:0/1370952320 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb04c9d4640 1 --2- 172.21.15.77:0/1370952320 >> v2:172.21.15.158:3300/0 conn(0x7fb0480b2910 0x7fb0480d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.503+0000 7fb047fff640 1 --2- 172.21.15.77:0/1370952320 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb04815e010 0x7fb0480cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.504+0000 7fb0477fe640 1 -- 172.21.15.77:0/1370952320 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb04815e010 msgr2=0x7fb0480cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.504+0000 7fb0477fe640 1 --2- 172.21.15.77:0/1370952320 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb04815e010 0x7fb0480cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.504+0000 7fb0477fe640 1 -- 172.21.15.77:0/1370952320 >> v2:172.21.15.158:3300/0 conn(0x7fb0480b2910 msgr2=0x7fb0480d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.504+0000 7fb0477fe640 1 --2- 172.21.15.77:0/1370952320 >> v2:172.21.15.158:3300/0 conn(0x7fb0480b2910 0x7fb0480d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.504+0000 7fb0477fe640 1 -- 172.21.15.77:0/1370952320 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04806d1b0 con 0x7fb0480b3240 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.504+0000 7fb0477fe640 1 --2- 172.21.15.77:0/1370952320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0480b3240 0x7fb0480b23d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009870 tx=0x7fb03c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205670bb94ad2d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb046ffd640 1 -- 172.21.15.77:0/1370952320 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c0317c0 con 0x7fb0480b3240 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb046ffd640 1 -- 172.21.15.77:0/1370952320 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c037070 con 0x7fb0480b3240 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb046ffd640 1 -- 172.21.15.77:0/1370952320 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c031b80 con 0x7fb0480b3240 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 -- 172.21.15.77:0/1370952320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0480b3240 msgr2=0x7fb0480b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 --2- 172.21.15.77:0/1370952320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0480b3240 0x7fb0480b23d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009870 tx=0x7fb03c02b860 comp rx=0 tx=0).stop 2024-03-06T07:23:21.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 -- 172.21.15.77:0/1370952320 shutdown_connections 2024-03-06T07:23:21.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 --2- 172.21.15.77:0/1370952320 >> v2:172.21.15.158:3300/0 conn(0x7fb0480b2910 0x7fb0480d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 --2- 172.21.15.77:0/1370952320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0480b3240 0x7fb0480b23d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 --2- 172.21.15.77:0/1370952320 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb04815e010 0x7fb0480cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 -- 172.21.15.77:0/1370952320 >> 172.21.15.77:0/1370952320 conn(0x7fb04809a9f0 msgr2=0x7fb04809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 -- 172.21.15.77:0/1370952320 shutdown_connections 2024-03-06T07:23:21.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.505+0000 7fb04e437640 1 -- 172.21.15.77:0/1370952320 wait complete. 2024-03-06T07:23:21.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.506+0000 7fb04e437640 1 Processor -- start 2024-03-06T07:23:21.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.506+0000 7fb04e437640 1 -- start start 2024-03-06T07:23:21.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb04e437640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04815e010 0x7fb0480c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb04e437640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb048148d10 0x7fb0480c6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb04e437640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0480d0600 0x7fb0480d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb04e437640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0480cf310 con 0x7fb04815e010 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb04e437640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04806ced0 con 0x7fb048148d10 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb04e437640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0480d2010 con 0x7fb0480d0600 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb047fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04815e010 0x7fb0480c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb047fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04815e010 0x7fb0480c6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36016/0 (socket says 172.21.15.77:36016) 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb047fff640 1 -- 172.21.15.77:0/2578724850 learned_addr learned my addr 172.21.15.77:0/2578724850 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb04c9d4640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0480d0600 0x7fb0480d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb0477fe640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb048148d10 0x7fb0480c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb047fff640 1 -- 172.21.15.77:0/2578724850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0480d0600 msgr2=0x7fb0480d0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.507+0000 7fb047fff640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0480d0600 0x7fb0480d0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb047fff640 1 -- 172.21.15.77:0/2578724850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb048148d10 msgr2=0x7fb0480c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb047fff640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb048148d10 0x7fb0480c6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb047fff640 1 -- 172.21.15.77:0/2578724850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0480c6650 con 0x7fb04815e010 2024-03-06T07:23:21.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb047fff640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04815e010 0x7fb0480c6f30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb03800e8a0 tx=0x7fb03800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb044ff9640 1 -- 172.21.15.77:0/2578724850 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038017070 con 0x7fb04815e010 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb044ff9640 1 -- 172.21.15.77:0/2578724850 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03800c8e0 con 0x7fb04815e010 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb044ff9640 1 -- 172.21.15.77:0/2578724850 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03800ccd0 con 0x7fb04815e010 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0480c8690 con 0x7fb04815e010 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.508+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0480c78d0 con 0x7fb04815e010 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.509+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04804f330 con 0x7fb04815e010 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.509+0000 7fb044ff9640 1 -- 172.21.15.77:0/2578724850 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb03801a070 con 0x7fb04815e010 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.510+0000 7fb044ff9640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb03406b500 0x7fb03406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.510+0000 7fb044ff9640 1 -- 172.21.15.77:0/2578724850 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb038089fd0 con 0x7fb04815e010 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.510+0000 7fb0477fe640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb03406b500 0x7fb03406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.510+0000 7fb0477fe640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb03406b500 0x7fb03406d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb03c036000 tx=0x7fb03c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.514+0000 7fb044ff9640 1 -- 172.21.15.77:0/2578724850 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb038058800 con 0x7fb04815e010 2024-03-06T07:23:21.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.721+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb0480d2980 con 0x7fb04815e010 2024-03-06T07:23:21.723 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:23:21.721+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:21.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.722+0000 7fb044ff9640 1 -- 172.21.15.77:0/2578724850 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v14) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb03805c8a0 con 0x7fb04815e010 2024-03-06T07:23:21.724 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb03406b500 msgr2=0x7fb03406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb03406b500 0x7fb03406d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb03c036000 tx=0x7fb03c03d040 comp rx=0 tx=0).stop 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04815e010 msgr2=0x7fb0480c6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04815e010 0x7fb0480c6f30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb03800e8a0 tx=0x7fb03800ed60 comp rx=0 tx=0).stop 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 shutdown_connections 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0480d0600 0x7fb0480d0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb048148d10 0x7fb0480c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb03406b500 0x7fb03406d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 --2- 172.21.15.77:0/2578724850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb04815e010 0x7fb0480c6f30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 >> 172.21.15.77:0/2578724850 conn(0x7fb04809a9f0 msgr2=0x7fb0480ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 shutdown_connections 2024-03-06T07:23:21.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.725+0000 7fb04e437640 1 -- 172.21.15.77:0/2578724850 wait complete. 2024-03-06T07:23:21.738 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:21.765 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:23:21.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.838+0000 7f5f3835e640 1 Processor -- start 2024-03-06T07:23:21.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.838+0000 7f5f3835e640 1 -- start start 2024-03-06T07:23:21.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f3835e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3015bb40 0x7f5f300b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f3835e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f3015f7d0 0x7f5f3006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f3835e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3015ff90 con 0x7f5f3006d5c0 2024-03-06T07:23:21.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f3835e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3006c910 con 0x7f5f3015f7d0 2024-03-06T07:23:21.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f3835e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f30160270 con 0x7f5f3015bb40 2024-03-06T07:23:21.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f358f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f3015f7d0 0x7f5f3006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f358f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f3015f7d0 0x7f5f3006d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36020/0 (socket says 172.21.15.77:36020) 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f358f9640 1 -- 172.21.15.77:0/3457539204 learned_addr learned my addr 172.21.15.77:0/3457539204 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.839+0000 7f5f360fa640 1 --2- 172.21.15.77:0/3457539204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3015bb40 0x7f5f300b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.840+0000 7f5f358f9640 1 -- 172.21.15.77:0/3457539204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3015bb40 msgr2=0x7f5f300b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.840+0000 7f5f358f9640 1 --2- 172.21.15.77:0/3457539204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3015bb40 0x7f5f300b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.840+0000 7f5f358f9640 1 -- 172.21.15.77:0/3457539204 >> v1:172.21.15.158:6789/0 conn(0x7f5f3006d5c0 legacy=0x7f5f300cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.840+0000 7f5f358f9640 1 -- 172.21.15.77:0/3457539204 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f30160550 con 0x7f5f3015f7d0 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.840+0000 7f5f358f9640 1 --2- 172.21.15.77:0/3457539204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f3006d080 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009870 tx=0x7f5f2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabeb91373da3d10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.840+0000 7f5f350f8640 1 -- 172.21.15.77:0/3457539204 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20034070 con 0x7f5f3015f7d0 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.840+0000 7f5f350f8640 1 -- 172.21.15.77:0/3457539204 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f20030840 con 0x7f5f3015f7d0 2024-03-06T07:23:21.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.840+0000 7f5f350f8640 1 -- 172.21.15.77:0/3457539204 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20030c10 con 0x7f5f3015f7d0 2024-03-06T07:23:21.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.841+0000 7f5f3835e640 1 -- 172.21.15.77:0/3457539204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 msgr2=0x7f5f3006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.841+0000 7f5f3835e640 1 --2- 172.21.15.77:0/3457539204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f3006d080 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009870 tx=0x7f5f2002b980 comp rx=0 tx=0).stop 2024-03-06T07:23:21.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.841+0000 7f5f3835e640 1 -- 172.21.15.77:0/3457539204 shutdown_connections 2024-03-06T07:23:21.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.841+0000 7f5f3835e640 1 --2- 172.21.15.77:0/3457539204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f3006d080 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.841+0000 7f5f3835e640 1 --2- 172.21.15.77:0/3457539204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3015bb40 0x7f5f300b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.841+0000 7f5f3835e640 1 -- 172.21.15.77:0/3457539204 >> 172.21.15.77:0/3457539204 conn(0x7f5f3006b190 msgr2=0x7f5f3006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.841+0000 7f5f3835e640 1 -- 172.21.15.77:0/3457539204 shutdown_connections 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.841+0000 7f5f3835e640 1 -- 172.21.15.77:0/3457539204 wait complete. 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.842+0000 7f5f3835e640 1 Processor -- start 2024-03-06T07:23:21.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.842+0000 7f5f3835e640 1 -- start start 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f3835e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3006d5c0 0x7f5f3015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f3835e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f3015bb40 0x7f5f30141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f3835e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f301423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f3835e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3015bf20 con 0x7f5f3015f7d0 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f3835e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3015fc30 con 0x7f5f3015bb40 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f3835e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3015c0a0 con 0x7f5f3006d5c0 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f368fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f301423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f368fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f301423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36034/0 (socket says 172.21.15.77:36034) 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f368fb640 1 -- 172.21.15.77:0/4093745688 learned_addr learned my addr 172.21.15.77:0/4093745688 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:21.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f360fa640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3006d5c0 0x7f5f3015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f358f9640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f3015bb40 0x7f5f30141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f368fb640 1 -- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3006d5c0 msgr2=0x7f5f3015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f368fb640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3006d5c0 0x7f5f3015a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f368fb640 1 -- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f3015bb40 msgr2=0x7f5f30141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f368fb640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f3015bb40 0x7f5f30141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f368fb640 1 -- 172.21.15.77:0/4093745688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3015d0a0 con 0x7f5f3015f7d0 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.843+0000 7f5f360fa640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3006d5c0 0x7f5f3015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.844+0000 7f5f368fb640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f301423e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c7a0 tx=0x7f5f2800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.844+0000 7f5f26ffd640 1 -- 172.21.15.77:0/4093745688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28003b70 con 0x7f5f3015f7d0 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.844+0000 7f5f26ffd640 1 -- 172.21.15.77:0/4093745688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f28003d30 con 0x7f5f3015f7d0 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.844+0000 7f5f26ffd640 1 -- 172.21.15.77:0/4093745688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2800aa90 con 0x7f5f3015f7d0 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.844+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3015d380 con 0x7f5f3015f7d0 2024-03-06T07:23:21.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.844+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3015d940 con 0x7f5f3015f7d0 2024-03-06T07:23:21.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.845+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f300732b0 con 0x7f5f3015f7d0 2024-03-06T07:23:21.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.845+0000 7f5f26ffd640 1 -- 172.21.15.77:0/4093745688 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f2800ac30 con 0x7f5f3015f7d0 2024-03-06T07:23:21.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.845+0000 7f5f26ffd640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f1406b500 0x7f5f1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.846+0000 7f5f26ffd640 1 -- 172.21.15.77:0/4093745688 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5f2808a1b0 con 0x7f5f3015f7d0 2024-03-06T07:23:21.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.846+0000 7f5f360fa640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f1406b500 0x7f5f1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.846+0000 7f5f360fa640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f1406b500 0x7f5f1406d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009b90 tx=0x7f5f2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:21.850+0000 7f5f26ffd640 1 -- 172.21.15.77:0/4093745688 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f28058f30 con 0x7f5f3015f7d0 2024-03-06T07:23:22.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.059+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5f30068ff0 con 0x7f5f3015f7d0 2024-03-06T07:23:22.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.724+0000 7f5f26ffd640 1 -- 172.21.15.77:0/4093745688 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5f28058830 con 0x7f5f3015f7d0 2024-03-06T07:23:22.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f1406b500 msgr2=0x7f5f1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f1406b500 0x7f5f1406d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009b90 tx=0x7f5f2c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 msgr2=0x7f5f301423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f301423e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c7a0 tx=0x7f5f2800cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 shutdown_connections 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f1406b500 0x7f5f1406d9c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f3015f7d0 0x7f5f301423e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f3015bb40 0x7f5f30141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.726+0000 7f5f3835e640 1 --2- 172.21.15.77:0/4093745688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f3006d5c0 0x7f5f3015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.727+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 >> 172.21.15.77:0/4093745688 conn(0x7f5f3006b190 msgr2=0x7f5f30159ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.727+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 shutdown_connections 2024-03-06T07:23:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.727+0000 7f5f3835e640 1 -- 172.21.15.77:0/4093745688 wait complete. 2024-03-06T07:23:22.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:22.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.847+0000 7fce66ec9640 1 Processor -- start 2024-03-06T07:23:22.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.847+0000 7fce66ec9640 1 -- start start 2024-03-06T07:23:22.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce66ec9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce600b5ca0 0x7fce600b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce66ec9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce60068b00 con 0x7fce6006be80 2024-03-06T07:23:22.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce66ec9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6015e4e0 con 0x7fce6015f9d0 2024-03-06T07:23:22.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce66ec9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6015edf0 con 0x7fce600b5ca0 2024-03-06T07:23:22.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce5ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce600b5ca0 0x7fce600b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce64c65640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fce6006be80 0x7fce6006c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49610/0 (socket says 172.21.15.77:49610) 2024-03-06T07:23:22.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce64c65640 1 -- 172.21.15.77:0/747568879 learned_addr learned my addr 172.21.15.77:0/747568879 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:22.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce5ffff640 1 -- 172.21.15.77:0/747568879 >> v1:172.21.15.158:6789/0 conn(0x7fce6015f9d0 legacy=0x7fce6015fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce5ffff640 1 -- 172.21.15.77:0/747568879 >> v1:172.21.15.77:6789/0 conn(0x7fce6006be80 legacy=0x7fce6006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.848+0000 7fce5ffff640 1 -- 172.21.15.77:0/747568879 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce600b1900 con 0x7fce600b5ca0 2024-03-06T07:23:22.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.849+0000 7fce5ffff640 1 --2- 172.21.15.77:0/747568879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce600b5ca0 0x7fce600b11b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fce6015ef70 tx=0x7fce5402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f7293a618e413f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.849+0000 7fce5f7fe640 1 -- 172.21.15.77:0/747568879 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce540315a0 con 0x7fce600b5ca0 2024-03-06T07:23:22.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.849+0000 7fce5f7fe640 1 -- 172.21.15.77:0/747568879 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce54037070 con 0x7fce600b5ca0 2024-03-06T07:23:22.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.849+0000 7fce5f7fe640 1 -- 172.21.15.77:0/747568879 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce54031960 con 0x7fce600b5ca0 2024-03-06T07:23:22.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.850+0000 7fce66ec9640 1 -- 172.21.15.77:0/747568879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce600b5ca0 msgr2=0x7fce600b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.850+0000 7fce66ec9640 1 --2- 172.21.15.77:0/747568879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce600b5ca0 0x7fce600b11b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fce6015ef70 tx=0x7fce5402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:22.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.850+0000 7fce66ec9640 1 -- 172.21.15.77:0/747568879 shutdown_connections 2024-03-06T07:23:22.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.850+0000 7fce66ec9640 1 --2- 172.21.15.77:0/747568879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce600b5ca0 0x7fce600b11b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.850+0000 7fce66ec9640 1 -- 172.21.15.77:0/747568879 >> 172.21.15.77:0/747568879 conn(0x7fce600b4ed0 msgr2=0x7fce600b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.850+0000 7fce66ec9640 1 -- 172.21.15.77:0/747568879 shutdown_connections 2024-03-06T07:23:22.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.850+0000 7fce66ec9640 1 -- 172.21.15.77:0/747568879 wait complete. 2024-03-06T07:23:22.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.851+0000 7fce66ec9640 1 Processor -- start 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.851+0000 7fce66ec9640 1 -- start start 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce66ec9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce6006be80 0x7fce601525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce66ec9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce600b5ca0 0x7fce600cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce66ec9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6015f9d0 0x7fce600cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce66ec9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6015ea80 con 0x7fce6015f9d0 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce66ec9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce601618a0 con 0x7fce6006be80 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce66ec9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6015f280 con 0x7fce600b5ca0 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce65466640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6015f9d0 0x7fce600cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce65466640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6015f9d0 0x7fce600cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36046/0 (socket says 172.21.15.77:36046) 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce65466640 1 -- 172.21.15.77:0/804463977 learned_addr learned my addr 172.21.15.77:0/804463977 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce64c65640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce6006be80 0x7fce601525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce5ffff640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce600b5ca0 0x7fce600cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce65466640 1 -- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce600b5ca0 msgr2=0x7fce600cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce65466640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce600b5ca0 0x7fce600cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce65466640 1 -- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce6006be80 msgr2=0x7fce601525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce65466640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce6006be80 0x7fce601525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce65466640 1 -- 172.21.15.77:0/804463977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6015b690 con 0x7fce6015f9d0 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.852+0000 7fce64c65640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce6006be80 0x7fce601525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.853+0000 7fce65466640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6015f9d0 0x7fce600cff20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fce5800b990 tx=0x7fce5800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.853+0000 7fce5d7fa640 1 -- 172.21.15.77:0/804463977 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58019070 con 0x7fce6015f9d0 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.853+0000 7fce5d7fa640 1 -- 172.21.15.77:0/804463977 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5800cba0 con 0x7fce6015f9d0 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.853+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce600cdb90 con 0x7fce6015f9d0 2024-03-06T07:23:22.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.853+0000 7fce5d7fa640 1 -- 172.21.15.77:0/804463977 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58014440 con 0x7fce6015f9d0 2024-03-06T07:23:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.853+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce600cf140 con 0x7fce6015f9d0 2024-03-06T07:23:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.854+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6004f330 con 0x7fce6015f9d0 2024-03-06T07:23:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.854+0000 7fce5d7fa640 1 -- 172.21.15.77:0/804463977 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce5801c070 con 0x7fce6015f9d0 2024-03-06T07:23:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.854+0000 7fce5d7fa640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce4c06b500 0x7fce4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.854+0000 7fce5d7fa640 1 -- 172.21.15.77:0/804463977 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fce58089f10 con 0x7fce6015f9d0 2024-03-06T07:23:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.855+0000 7fce64c65640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce4c06b500 0x7fce4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.855+0000 7fce64c65640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce4c06b500 0x7fce4c06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fce50000f60 tx=0x7fce50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:22.859+0000 7fce5d7fa640 1 -- 172.21.15.77:0/804463977 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce58058dc0 con 0x7fce6015f9d0 2024-03-06T07:23:23.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.029+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fce600cddc0 con 0x7fce6015f9d0 2024-03-06T07:23:23.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.728+0000 7fce5d7fa640 1 -- 172.21.15.77:0/804463977 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) v1 ==== 181+0+0 (secure 0 0 0) 0x7fce580586c0 con 0x7fce6015f9d0 2024-03-06T07:23:23.730 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:23.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce4c06b500 msgr2=0x7fce4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce4c06b500 0x7fce4c06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fce50000f60 tx=0x7fce50009040 comp rx=0 tx=0).stop 2024-03-06T07:23:23.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6015f9d0 msgr2=0x7fce600cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6015f9d0 0x7fce600cff20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fce5800b990 tx=0x7fce5800be50 comp rx=0 tx=0).stop 2024-03-06T07:23:23.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 shutdown_connections 2024-03-06T07:23:23.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce4c06b500 0x7fce4c06d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce6015f9d0 0x7fce600cff20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce600b5ca0 0x7fce600cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 --2- 172.21.15.77:0/804463977 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce6006be80 0x7fce601525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 >> 172.21.15.77:0/804463977 conn(0x7fce600b4ed0 msgr2=0x7fce6015fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 shutdown_connections 2024-03-06T07:23:23.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.732+0000 7fce66ec9640 1 -- 172.21.15.77:0/804463977 wait complete. 2024-03-06T07:23:23.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.858+0000 7fc1db078640 1 Processor -- start 2024-03-06T07:23:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.858+0000 7fc1db078640 1 -- start start 2024-03-06T07:23:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1db078640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc1d40708b0 0x7fc1d4070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1db078640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d415afe0 con 0x7fc1d415ac00 2024-03-06T07:23:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1db078640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d40cd000 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1db078640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d4071250 con 0x7fc1d40708b0 2024-03-06T07:23:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1d3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc1d40708b0 0x7fc1d4070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1d8e14640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc1d415ac00 0x7fc1d406c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49620/0 (socket says 172.21.15.77:49620) 2024-03-06T07:23:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1d8e14640 1 -- 172.21.15.77:0/4081635507 learned_addr learned my addr 172.21.15.77:0/4081635507 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1d3fff640 1 -- 172.21.15.77:0/4081635507 >> v1:172.21.15.158:6789/0 conn(0x7fc1d40ce0a0 legacy=0x7fc1d40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1d3fff640 1 -- 172.21.15.77:0/4081635507 >> v1:172.21.15.77:6789/0 conn(0x7fc1d415ac00 legacy=0x7fc1d406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1d3fff640 1 -- 172.21.15.77:0/4081635507 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40cd2e0 con 0x7fc1d40708b0 2024-03-06T07:23:23.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.859+0000 7fc1d37fe640 1 -- 172.21.15.77:0/4081635507 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976997353 0 0) 0x7fc1d415afe0 con 0x7fc1d415ac00 2024-03-06T07:23:23.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.860+0000 7fc1d3fff640 1 --2- 172.21.15.77:0/4081635507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40708b0 0x7fc1d4070d10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00097a0 tx=0x7fc1c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255414fc510bae6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.860+0000 7fc1d37fe640 1 -- 172.21.15.77:0/4081635507 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00315a0 con 0x7fc1d40708b0 2024-03-06T07:23:23.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.860+0000 7fc1d37fe640 1 -- 172.21.15.77:0/4081635507 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c0037070 con 0x7fc1d40708b0 2024-03-06T07:23:23.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.860+0000 7fc1d37fe640 1 -- 172.21.15.77:0/4081635507 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0031960 con 0x7fc1d40708b0 2024-03-06T07:23:23.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.861+0000 7fc1db078640 1 -- 172.21.15.77:0/4081635507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40708b0 msgr2=0x7fc1d4070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.861+0000 7fc1db078640 1 --2- 172.21.15.77:0/4081635507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40708b0 0x7fc1d4070d10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00097a0 tx=0x7fc1c002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.861+0000 7fc1db078640 1 -- 172.21.15.77:0/4081635507 shutdown_connections 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.861+0000 7fc1db078640 1 --2- 172.21.15.77:0/4081635507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40708b0 0x7fc1d4070d10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.861+0000 7fc1db078640 1 -- 172.21.15.77:0/4081635507 >> 172.21.15.77:0/4081635507 conn(0x7fc1d40b1650 msgr2=0x7fc1d40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.861+0000 7fc1db078640 1 -- 172.21.15.77:0/4081635507 shutdown_connections 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.861+0000 7fc1db078640 1 -- 172.21.15.77:0/4081635507 wait complete. 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.862+0000 7fc1db078640 1 Processor -- start 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.862+0000 7fc1db078640 1 -- start start 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1db078640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1d40708b0 0x7fc1d415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1db078640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40ce0a0 0x7fc1d41f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1db078640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1d415ac00 0x7fc1d40b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1db078640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40cd8a0 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1db078640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40cdc10 con 0x7fc1d415ac00 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1db078640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40cf5d0 con 0x7fc1d40708b0 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40ce0a0 0x7fc1d41f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40ce0a0 0x7fc1d41f3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36054/0 (socket says 172.21.15.77:36054) 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 -- 172.21.15.77:0/2228581774 learned_addr learned my addr 172.21.15.77:0/2228581774 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d8e14640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1d40708b0 0x7fc1d415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d9615640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1d415ac00 0x7fc1d40b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 -- 172.21.15.77:0/2228581774 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1d40708b0 msgr2=0x7fc1d415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1d40708b0 0x7fc1d415afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 -- 172.21.15.77:0/2228581774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1d415ac00 msgr2=0x7fc1d40b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1d415ac00 0x7fc1d40b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 -- 172.21.15.77:0/2228581774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d415b810 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d3fff640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40ce0a0 0x7fc1d41f3390 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00097a0 tx=0x7fc1c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d17fa640 1 -- 172.21.15.77:0/2228581774 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0004030 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d17fa640 1 -- 172.21.15.77:0/2228581774 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c00041f0 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1d17fa640 1 -- 172.21.15.77:0/2228581774 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0032640 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d415baf0 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.863+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d415c0b0 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.865+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d40687d0 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.865+0000 7fc1d17fa640 1 -- 172.21.15.77:0/2228581774 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1c00327e0 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.865+0000 7fc1d17fa640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1b806b4d0 0x7fc1b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.865+0000 7fc1d17fa640 1 -- 172.21.15.77:0/2228581774 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fc1c002f030 con 0x7fc1d40ce0a0 2024-03-06T07:23:23.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.865+0000 7fc1d8e14640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1b806b4d0 0x7fc1b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.866+0000 7fc1d8e14640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1b806b4d0 0x7fc1b806d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8000f60 tx=0x7fc1c8007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:23.870+0000 7fc1d17fa640 1 -- 172.21.15.77:0/2228581774 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1c0041070 con 0x7fc1d40ce0a0 2024-03-06T07:23:24.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.039+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc1d41e6fe0 con 0x7fc1d40ce0a0 2024-03-06T07:23:24.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.729+0000 7fc1d17fa640 1 -- 172.21.15.77:0/2228581774 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc1c003ae20 con 0x7fc1d40ce0a0 2024-03-06T07:23:24.732 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:24.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.733+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1b806b4d0 msgr2=0x7fc1b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.733+0000 7fc1db078640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1b806b4d0 0x7fc1b806d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8000f60 tx=0x7fc1c8007b50 comp rx=0 tx=0).stop 2024-03-06T07:23:24.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40ce0a0 msgr2=0x7fc1d41f3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40ce0a0 0x7fc1d41f3390 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00097a0 tx=0x7fc1c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 shutdown_connections 2024-03-06T07:23:24.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1b806b4d0 0x7fc1b806d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1d40ce0a0 0x7fc1d41f3390 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1d415ac00 0x7fc1d40b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 --2- 172.21.15.77:0/2228581774 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1d40708b0 0x7fc1d415afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 >> 172.21.15.77:0/2228581774 conn(0x7fc1d40b1650 msgr2=0x7fc1d41555f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 shutdown_connections 2024-03-06T07:23:24.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.734+0000 7fc1db078640 1 -- 172.21.15.77:0/2228581774 wait complete. 2024-03-06T07:23:24.752 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.856+0000 7f4bf6278640 1 Processor -- start 2024-03-06T07:23:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.856+0000 7f4bf6278640 1 -- start start 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4bf6278640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4bf015fc30 0x7f4bf0163160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4bf6278640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4bf00ae6b0 0x7f4bf00b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4bf6278640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf00af170 con 0x7f4bf00cfa90 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4bf6278640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf01629a0 con 0x7f4bf015fc30 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4bf6278640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf00aebb0 con 0x7f4bf00ae6b0 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4beffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4bf015fc30 0x7f4bf0163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4bf4815640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4bf00cfa90 0x7f4bf006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49634/0 (socket says 172.21.15.77:49634) 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4bf4815640 1 -- 172.21.15.77:0/4168637354 learned_addr learned my addr 172.21.15.77:0/4168637354 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4bef7fe640 1 --2- 172.21.15.77:0/4168637354 >> v2:172.21.15.158:3300/0 conn(0x7f4bf00ae6b0 0x7f4bf00b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4beffff640 1 -- 172.21.15.77:0/4168637354 >> v2:172.21.15.158:3300/0 conn(0x7f4bf00ae6b0 msgr2=0x7f4bf00b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4beffff640 1 --2- 172.21.15.77:0/4168637354 >> v2:172.21.15.158:3300/0 conn(0x7f4bf00ae6b0 0x7f4bf00b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4beffff640 1 -- 172.21.15.77:0/4168637354 >> v1:172.21.15.77:6789/0 conn(0x7f4bf00cfa90 legacy=0x7f4bf006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4beeffd640 1 -- 172.21.15.77:0/4168637354 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809359454 0 0) 0x7f4bf00af170 con 0x7f4bf00cfa90 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.857+0000 7f4beffff640 1 -- 172.21.15.77:0/4168637354 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf00aee90 con 0x7f4bf015fc30 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.858+0000 7f4beffff640 1 --2- 172.21.15.77:0/4168637354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf0163160 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00b4240 tx=0x7f4be002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ccc44625a0b91c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.858+0000 7f4beeffd640 1 -- 172.21.15.77:0/4168637354 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0035020 con 0x7f4bf015fc30 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.858+0000 7f4beeffd640 1 -- 172.21.15.77:0/4168637354 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be0031830 con 0x7f4bf015fc30 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.858+0000 7f4beeffd640 1 -- 172.21.15.77:0/4168637354 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0031c00 con 0x7f4bf015fc30 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.859+0000 7f4bf6278640 1 -- 172.21.15.77:0/4168637354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 msgr2=0x7f4bf0163160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.859+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4168637354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf0163160 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00b4240 tx=0x7f4be002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.859+0000 7f4bf6278640 1 -- 172.21.15.77:0/4168637354 shutdown_connections 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.859+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4168637354 >> v2:172.21.15.158:3300/0 conn(0x7f4bf00ae6b0 0x7f4bf00b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.859+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4168637354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf0163160 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.859+0000 7f4bf6278640 1 -- 172.21.15.77:0/4168637354 >> 172.21.15.77:0/4168637354 conn(0x7f4bf006b190 msgr2=0x7f4bf006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.859+0000 7f4bf6278640 1 -- 172.21.15.77:0/4168637354 shutdown_connections 2024-03-06T07:23:24.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.859+0000 7f4bf6278640 1 -- 172.21.15.77:0/4168637354 wait complete. 2024-03-06T07:23:24.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.860+0000 7f4bf6278640 1 Processor -- start 2024-03-06T07:23:24.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.860+0000 7f4bf6278640 1 -- start start 2024-03-06T07:23:24.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf6278640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf00ae6b0 0x7f4bf00b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf6278640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf00cfa90 0x7f4bf01ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf6278640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf015a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf6278640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00b4c40 con 0x7f4bf015fc30 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf6278640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf01636a0 con 0x7f4bf00ae6b0 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf6278640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00b4dc0 con 0x7f4bf00cfa90 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf015a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf015a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36070/0 (socket says 172.21.15.77:36070) 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 -- 172.21.15.77:0/4066766298 learned_addr learned my addr 172.21.15.77:0/4066766298 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bef7fe640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf00cfa90 0x7f4bf01ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4beffff640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf00ae6b0 0x7f4bf00b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 -- 172.21.15.77:0/4066766298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf00cfa90 msgr2=0x7f4bf01ec620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf00cfa90 0x7f4bf01ec620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 -- 172.21.15.77:0/4066766298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf00ae6b0 msgr2=0x7f4bf00b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf00ae6b0 0x7f4bf00b0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 -- 172.21.15.77:0/4066766298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf01ed000 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4bf4815640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf015a160 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4be800c700 tx=0x7f4be800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4becff9640 1 -- 172.21.15.77:0/4066766298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8018070 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.861+0000 7f4becff9640 1 -- 172.21.15.77:0/4066766298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be800ce10 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.862+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf01ed2e0 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.862+0000 7f4becff9640 1 -- 172.21.15.77:0/4066766298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be800a820 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.862+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf01ed8a0 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.862+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf00732b0 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.863+0000 7f4becff9640 1 -- 172.21.15.77:0/4066766298 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4be801b070 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.863+0000 7f4becff9640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.863+0000 7f4becff9640 1 -- 172.21.15.77:0/4066766298 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4be80893a0 con 0x7f4bf015fc30 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.863+0000 7f4beffff640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.864+0000 7f4beffff640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4be0036000 tx=0x7f4be0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:24.868+0000 7f4becff9640 1 -- 172.21.15.77:0/4066766298 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4be80585b0 con 0x7f4bf015fc30 2024-03-06T07:23:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.053+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4bf0068ff0 con 0x7f4bf015fc30 2024-03-06T07:23:25.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.056+0000 7f4becff9640 1 -- 172.21.15.77:0/4066766298 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4be8057eb0 con 0x7f4bf015fc30 2024-03-06T07:23:25.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4bdc06b4d0 msgr2=0x7f4bdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4be0036000 tx=0x7f4be0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 msgr2=0x7f4bf015a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf015a160 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4be800c700 tx=0x7f4be800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 shutdown_connections 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4bdc06b4d0 0x7f4bdc06d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf015fc30 0x7f4bf015a160 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bf00cfa90 0x7f4bf01ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 --2- 172.21.15.77:0/4066766298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bf00ae6b0 0x7f4bf00b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 >> 172.21.15.77:0/4066766298 conn(0x7f4bf006b190 msgr2=0x7f4bf00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 shutdown_connections 2024-03-06T07:23:25.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.058+0000 7f4bf6278640 1 -- 172.21.15.77:0/4066766298 wait complete. 2024-03-06T07:23:25.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:23:25.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.169+0000 7f0405956640 1 Processor -- start 2024-03-06T07:23:25.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.169+0000 7f0405956640 1 -- start start 2024-03-06T07:23:25.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f0405956640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 0x7f04000d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f0405956640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f04000b41f0 0x7f04000b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f0405956640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04000b3430 con 0x7f040015dfe0 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f0405956640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04000d2c50 con 0x7f04000b41f0 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f0405956640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04000b31e0 con 0x7f04000d22b0 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03feffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f040015dfe0 0x7f04000cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49642/0 (socket says 172.21.15.77:49642) 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03feffd640 1 -- 172.21.15.77:0/3111003870 learned_addr learned my addr 172.21.15.77:0/3111003870 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395427975 0 0) 0x7f04000b3430 con 0x7f040015dfe0 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e8003520 con 0x7f040015dfe0 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03ff7fe640 1 --2- 172.21.15.77:0/3111003870 >> v2:172.21.15.158:3300/0 conn(0x7f04000b41f0 0x7f04000b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fe7fc640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 0x7f04000d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3341873051 0 0) 0x7f03e8003520 con 0x7f040015dfe0 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 msgr2=0x7f04000d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 0x7f04000d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 >> v2:172.21.15.158:3300/0 conn(0x7f04000b41f0 msgr2=0x7f04000b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 --2- 172.21.15.77:0/3111003870 >> v2:172.21.15.158:3300/0 conn(0x7f04000b41f0 0x7f04000b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04000b3710 con 0x7f040015dfe0 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f03f0001670 con 0x7f040015dfe0 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 >> v1:172.21.15.77:6789/0 conn(0x7f040015dfe0 legacy=0x7f04000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f03e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 0x7f03e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e8004d40 0x7f03e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04000b31e0 con 0x7f03e80039b0 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e800b9c0 con 0x7f03e8004d40 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e800bb40 con 0x7f04000d22b0 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.170+0000 7f03feffd640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f03e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03fe7fc640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 0x7f03e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03ff7fe640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e8004d40 0x7f03e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03feffd640 1 -- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 msgr2=0x7f03e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03feffd640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 0x7f03e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03feffd640 1 -- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e8004d40 msgr2=0x7f03e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03feffd640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f03e8004d40 0x7f03e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03feffd640 1 -- 172.21.15.77:0/3111003870 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e8003520 con 0x7f03e80039b0 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03fe7fc640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000d22b0 0x7f03e8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03feffd640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f03e8003dd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f03f000c290 tx=0x7f03e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c7325e445cace6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f0007d50 con 0x7f03e80039b0 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.171+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f0030070 con 0x7f03e80039b0 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.172+0000 7f03fdffb640 1 -- 172.21.15.77:0/3111003870 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f002cd80 con 0x7f03e80039b0 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.172+0000 7f0405956640 1 -- 172.21.15.77:0/3111003870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 msgr2=0x7f03e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.172+0000 7f0405956640 1 --2- 172.21.15.77:0/3111003870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f03e8003dd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f03f000c290 tx=0x7f03e8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.172+0000 7f03ff7fe640 1 -- 172.21.15.77:0/3111003870 reap_dead start 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.172+0000 7f0405956640 1 -- 172.21.15.77:0/3111003870 shutdown_connections 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.172+0000 7f0405956640 1 -- 172.21.15.77:0/3111003870 >> 172.21.15.77:0/3111003870 conn(0x7f04000d3cd0 msgr2=0x7f04000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.172+0000 7f0405956640 1 -- 172.21.15.77:0/3111003870 shutdown_connections 2024-03-06T07:23:25.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.172+0000 7f0405956640 1 -- 172.21.15.77:0/3111003870 wait complete. 2024-03-06T07:23:25.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.173+0000 7f0405956640 1 Processor -- start 2024-03-06T07:23:25.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.173+0000 7f0405956640 1 -- start start 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f0405956640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f04001f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f0405956640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040015b360 0x7f04000b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f0405956640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000b18e0 0x7f04000b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f0405956640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04000cfa00 con 0x7f03e80039b0 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f0405956640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04000b3d60 con 0x7f040015b360 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f0405956640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04000d1c70 con 0x7f04000b18e0 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03feffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f04001f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03feffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f04001f4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36082/0 (socket says 172.21.15.77:36082) 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03feffd640 1 -- 172.21.15.77:0/3261864415 learned_addr learned my addr 172.21.15.77:0/3261864415 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03feffd640 1 -- 172.21.15.77:0/3261864415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000b18e0 msgr2=0x7f04000b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03feffd640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000b18e0 0x7f04000b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03fe7fc640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040015b360 0x7f04000b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03feffd640 1 -- 172.21.15.77:0/3261864415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040015b360 msgr2=0x7f04000b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03feffd640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040015b360 0x7f04000b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03feffd640 1 -- 172.21.15.77:0/3261864415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04000b8310 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.174+0000 7f03fe7fc640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040015b360 0x7f04000b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.175+0000 7f03feffd640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f04001f4820 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f03f0001970 tx=0x7f04000cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.175+0000 7f03e7fff640 1 -- 172.21.15.77:0/3261864415 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f0045070 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.175+0000 7f03e7fff640 1 -- 172.21.15.77:0/3261864415 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f0004bf0 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.175+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04000b6e10 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.175+0000 7f03e7fff640 1 -- 172.21.15.77:0/3261864415 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f0001c80 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.175+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04000b73c0 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.176+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040004f330 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.176+0000 7f03e7fff640 1 -- 172.21.15.77:0/3261864415 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03f0007070 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.176+0000 7f03e7fff640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03ec06b4d0 0x7f03ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.177+0000 7f03fe7fc640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03ec06b4d0 0x7f03ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.177+0000 7f03e7fff640 1 -- 172.21.15.77:0/3261864415 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f03f00aafe0 con 0x7f03e80039b0 2024-03-06T07:23:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.177+0000 7f03fe7fc640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03ec06b4d0 0x7f03ec06d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f04000b8120 tx=0x7f03f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.181+0000 7f03e7fff640 1 -- 172.21.15.77:0/3261864415 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03f007a240 con 0x7f03e80039b0 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.418+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f04000b09c0 con 0x7f03e80039b0 2024-03-06T07:23:25.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.737+0000 7f03e7fff640 1 -- 172.21.15.77:0/3261864415 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v97) v1 ==== 132+0+0 (secure 0 0 0) 0x7f03f0079b40 con 0x7f03e80039b0 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03ec06b4d0 msgr2=0x7f03ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03ec06b4d0 0x7f03ec06d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f04000b8120 tx=0x7f03f4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 msgr2=0x7f04001f4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f04001f4820 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f03f0001970 tx=0x7f04000cfd40 comp rx=0 tx=0).stop 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 shutdown_connections 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04000b18e0 0x7f04000b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f040015b360 0x7f04000b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03ec06b4d0 0x7f03ec06d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 --2- 172.21.15.77:0/3261864415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03e80039b0 0x7f04001f4820 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 >> 172.21.15.77:0/3261864415 conn(0x7f04000d3cd0 msgr2=0x7f040006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 shutdown_connections 2024-03-06T07:23:25.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.740+0000 7f0405956640 1 -- 172.21.15.77:0/3261864415 wait complete. 2024-03-06T07:23:25.754 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:23:25.754 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:23:25.754 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:23:25.755 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.856+0000 7ff9786cf640 1 Processor -- start 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.856+0000 7ff9786cf640 1 -- start start 2024-03-06T07:23:25.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff9786cf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff97013dc10 0x7ff9700b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff9786cf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff97015ef60 0x7ff9700b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff9786cf640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97015f580 con 0x7ff97015bf30 2024-03-06T07:23:25.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff9786cf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9700cf3f0 con 0x7ff97013dc10 2024-03-06T07:23:25.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff9786cf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff97015b210 con 0x7ff97015ef60 2024-03-06T07:23:25.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff97646b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff97013dc10 0x7ff9700b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff97646b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff97013dc10 0x7ff9700b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36096/0 (socket says 172.21.15.77:36096) 2024-03-06T07:23:25.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff97646b640 1 -- 172.21.15.77:0/3479080218 learned_addr learned my addr 172.21.15.77:0/3479080218 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:25.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff976c6c640 1 --2- 172.21.15.77:0/3479080218 >> v2:172.21.15.158:3300/0 conn(0x7ff97015ef60 0x7ff9700b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff97646b640 1 -- 172.21.15.77:0/3479080218 >> v2:172.21.15.158:3300/0 conn(0x7ff97015ef60 msgr2=0x7ff9700b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff97646b640 1 --2- 172.21.15.77:0/3479080218 >> v2:172.21.15.158:3300/0 conn(0x7ff97015ef60 0x7ff9700b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff97646b640 1 -- 172.21.15.77:0/3479080218 >> v1:172.21.15.158:6789/0 conn(0x7ff97015bf30 legacy=0x7ff97015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.857+0000 7ff97646b640 1 -- 172.21.15.77:0/3479080218 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97015b450 con 0x7ff97013dc10 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.858+0000 7ff97646b640 1 --2- 172.21.15.77:0/3479080218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97013dc10 0x7ff9700b2950 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff970068700 tx=0x7ff96c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4878d936ca4e20a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.858+0000 7ff975469640 1 -- 172.21.15.77:0/3479080218 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c035020 con 0x7ff97013dc10 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.858+0000 7ff975469640 1 -- 172.21.15.77:0/3479080218 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c031870 con 0x7ff97013dc10 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.858+0000 7ff975469640 1 -- 172.21.15.77:0/3479080218 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c031c40 con 0x7ff97013dc10 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.858+0000 7ff9786cf640 1 -- 172.21.15.77:0/3479080218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97013dc10 msgr2=0x7ff9700b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.858+0000 7ff9786cf640 1 --2- 172.21.15.77:0/3479080218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97013dc10 0x7ff9700b2950 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff970068700 tx=0x7ff96c02b910 comp rx=0 tx=0).stop 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.859+0000 7ff9786cf640 1 -- 172.21.15.77:0/3479080218 shutdown_connections 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.859+0000 7ff9786cf640 1 --2- 172.21.15.77:0/3479080218 >> v2:172.21.15.158:3300/0 conn(0x7ff97015ef60 0x7ff9700b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.859+0000 7ff9786cf640 1 --2- 172.21.15.77:0/3479080218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97013dc10 0x7ff9700b2950 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.859+0000 7ff9786cf640 1 -- 172.21.15.77:0/3479080218 >> 172.21.15.77:0/3479080218 conn(0x7ff9700cf8a0 msgr2=0x7ff9700cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.859+0000 7ff9786cf640 1 -- 172.21.15.77:0/3479080218 shutdown_connections 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.859+0000 7ff9786cf640 1 -- 172.21.15.77:0/3479080218 wait complete. 2024-03-06T07:23:25.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.860+0000 7ff9786cf640 1 Processor -- start 2024-03-06T07:23:25.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.860+0000 7ff9786cf640 1 -- start start 2024-03-06T07:23:25.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.860+0000 7ff9786cf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff97013dc10 0x7ff970153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.860+0000 7ff9786cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97015bf30 0x7ff9701f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.860+0000 7ff9786cf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff97015ef60 0x7ff97013c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.860+0000 7ff9786cf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700708d0 con 0x7ff97015bf30 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.860+0000 7ff9786cf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97013dff0 con 0x7ff97015ef60 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.860+0000 7ff9786cf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff970070a50 con 0x7ff97013dc10 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97015bf30 0x7ff9701f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97015bf30 0x7ff9701f10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36104/0 (socket says 172.21.15.77:36104) 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 -- 172.21.15.77:0/1664341023 learned_addr learned my addr 172.21.15.77:0/1664341023 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff97646b640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff97013dc10 0x7ff970153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff976c6c640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff97015ef60 0x7ff97013c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 -- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff97013dc10 msgr2=0x7ff970153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff97013dc10 0x7ff970153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 -- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff97015ef60 msgr2=0x7ff97013c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff97015ef60 0x7ff97013c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 -- 172.21.15.77:0/1664341023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9701403f0 con 0x7ff97015bf30 2024-03-06T07:23:25.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff976c6c640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff97015ef60 0x7ff97013c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff975c6a640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97015bf30 0x7ff9701f10a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff96000e710 tx=0x7ff96000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff966ffd640 1 -- 172.21.15.77:0/1664341023 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9600052d0 con 0x7ff97015bf30 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff966ffd640 1 -- 172.21.15.77:0/1664341023 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff960005470 con 0x7ff97015bf30 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9701406d0 con 0x7ff97015bf30 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff966ffd640 1 -- 172.21.15.77:0/1664341023 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960001b50 con 0x7ff97015bf30 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.861+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff970141810 con 0x7ff97015bf30 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.862+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97004f330 con 0x7ff97015bf30 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.863+0000 7ff966ffd640 1 -- 172.21.15.77:0/1664341023 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9600035f0 con 0x7ff97015bf30 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.863+0000 7ff966ffd640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff95406b4d0 0x7ff95406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.863+0000 7ff966ffd640 1 -- 172.21.15.77:0/1664341023 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff96008c0d0 con 0x7ff97015bf30 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.863+0000 7ff97646b640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff95406b4d0 0x7ff95406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.863+0000 7ff97646b640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff95406b4d0 0x7ff95406d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff96c036000 tx=0x7ff96c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:25.867+0000 7ff966ffd640 1 -- 172.21.15.77:0/1664341023 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff960016020 con 0x7ff97015bf30 2024-03-06T07:23:26.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.041+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9700b3570 con 0x7ff97015bf30 2024-03-06T07:23:26.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.041+0000 7ff966ffd640 1 -- 172.21.15.77:0/1664341023 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13008 (secure 0 0 0) 0x7ff96000ee20 con 0x7ff97015bf30 2024-03-06T07:23:26.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:26.044 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:24.729200+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6834/929839050":"2024-03-07T07:23:21.722321+0000","172.21.15.158:6832/929839050":"2024-03-07T07:23:21.722321+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff95406b4d0 msgr2=0x7ff95406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff95406b4d0 0x7ff95406d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff96c036000 tx=0x7ff96c03c040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97015bf30 msgr2=0x7ff9701f10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97015bf30 0x7ff9701f10a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff96000e710 tx=0x7ff96000ebd0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 shutdown_connections 2024-03-06T07:23:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff95406b4d0 0x7ff95406d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff97015bf30 0x7ff9701f10a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff97015ef60 0x7ff97013c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 --2- 172.21.15.77:0/1664341023 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff97013dc10 0x7ff970153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.043+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 >> 172.21.15.77:0/1664341023 conn(0x7ff9700cf8a0 msgr2=0x7ff970072140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.044+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 shutdown_connections 2024-03-06T07:23:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.044+0000 7ff9786cf640 1 -- 172.21.15.77:0/1664341023 wait complete. 2024-03-06T07:23:26.057 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:26.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.158+0000 7f69c7c81640 1 Processor -- start 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.158+0000 7f69c7c81640 1 -- start start 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.158+0000 7f69c7c81640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69c01614d0 0x7f69c00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c7c81640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69c00d24b0 0x7f69c00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c7c81640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c01618b0 con 0x7f69c00d1910 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c7c81640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c00cf400 con 0x7f69c00d24b0 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c7c81640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c00cf6a0 con 0x7f69c01614d0 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c621e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69c00d24b0 0x7f69c00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c621e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69c00d24b0 0x7f69c00b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36118/0 (socket says 172.21.15.77:36118) 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c621e640 1 -- 172.21.15.77:0/949153777 learned_addr learned my addr 172.21.15.77:0/949153777 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c621e640 1 -- 172.21.15.77:0/949153777 >> v2:172.21.15.158:3300/0 conn(0x7f69c01614d0 msgr2=0x7f69c00d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c621e640 1 --2- 172.21.15.77:0/949153777 >> v2:172.21.15.158:3300/0 conn(0x7f69c01614d0 0x7f69c00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c621e640 1 -- 172.21.15.77:0/949153777 >> v1:172.21.15.158:6789/0 conn(0x7f69c00d1910 legacy=0x7f69c00d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.159+0000 7f69c621e640 1 -- 172.21.15.77:0/949153777 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c00cf980 con 0x7f69c00d24b0 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c621e640 1 --2- 172.21.15.77:0/949153777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b0e30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f69c008c470 tx=0x7f69b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80319785eee16c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c4a1b640 1 -- 172.21.15.77:0/949153777 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b802ca80 con 0x7f69c00d24b0 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c4a1b640 1 -- 172.21.15.77:0/949153777 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b802cc20 con 0x7f69c00d24b0 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c4a1b640 1 -- 172.21.15.77:0/949153777 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80343f0 con 0x7f69c00d24b0 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c7c81640 1 -- 172.21.15.77:0/949153777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 msgr2=0x7f69c00b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c7c81640 1 --2- 172.21.15.77:0/949153777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b0e30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f69c008c470 tx=0x7f69b802bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c7c81640 1 -- 172.21.15.77:0/949153777 shutdown_connections 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c7c81640 1 --2- 172.21.15.77:0/949153777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b0e30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c7c81640 1 --2- 172.21.15.77:0/949153777 >> v2:172.21.15.158:3300/0 conn(0x7f69c01614d0 0x7f69c00d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c7c81640 1 -- 172.21.15.77:0/949153777 >> 172.21.15.77:0/949153777 conn(0x7f69c00d3d40 msgr2=0x7f69c00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c7c81640 1 -- 172.21.15.77:0/949153777 shutdown_connections 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.160+0000 7f69c7c81640 1 -- 172.21.15.77:0/949153777 wait complete. 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c7c81640 1 Processor -- start 2024-03-06T07:23:26.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c7c81640 1 -- start start 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c7c81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69c00d1910 0x7f69c014e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c7c81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c7c81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69c01614d0 0x7f69c00c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c7c81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00b1370 con 0x7f69c00d24b0 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c7c81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00b0910 con 0x7f69c00d1910 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c7c81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00b14f0 con 0x7f69c01614d0 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c521c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c521c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36128/0 (socket says 172.21.15.77:36128) 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.162+0000 7f69c521c640 1 -- 172.21.15.77:0/2466681086 learned_addr learned my addr 172.21.15.77:0/2466681086 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c521c640 1 -- 172.21.15.77:0/2466681086 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69c01614d0 msgr2=0x7f69c00c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c521c640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69c01614d0 0x7f69c00c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c521c640 1 -- 172.21.15.77:0/2466681086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69c00d1910 msgr2=0x7f69c014e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c5a1d640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69c00d1910 0x7f69c014e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c521c640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69c00d1910 0x7f69c014e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c521c640 1 -- 172.21.15.77:0/2466681086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c00c8310 con 0x7f69c00d24b0 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c521c640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b8d70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00c440 tx=0x7f69bc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69b27fc640 1 -- 172.21.15.77:0/2466681086 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc017070 con 0x7f69c00d24b0 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69b27fc640 1 -- 172.21.15.77:0/2466681086 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc0035f0 con 0x7f69c00d24b0 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c00c85f0 con 0x7f69c00d24b0 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69b27fc640 1 -- 172.21.15.77:0/2466681086 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0039e0 con 0x7f69c00d24b0 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.163+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c00ccae0 con 0x7f69c00d24b0 2024-03-06T07:23:26.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.164+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c00d1450 con 0x7f69c00d24b0 2024-03-06T07:23:26.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.164+0000 7f69b27fc640 1 -- 172.21.15.77:0/2466681086 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69bc01a070 con 0x7f69c00d24b0 2024-03-06T07:23:26.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.165+0000 7f69b27fc640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69a406b500 0x7f69a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.165+0000 7f69b27fc640 1 -- 172.21.15.77:0/2466681086 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f69bc0892c0 con 0x7f69c00d24b0 2024-03-06T07:23:26.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.165+0000 7f69c5a1d640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69a406b500 0x7f69a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.165+0000 7f69c5a1d640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69a406b500 0x7f69a406d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f69b400a770 tx=0x7f69b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.169+0000 7f69b27fc640 1 -- 172.21.15.77:0/2466681086 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69c00d1450 con 0x7f69c00d24b0 2024-03-06T07:23:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.339+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f69c00cccb0 con 0x7f69c00d24b0 2024-03-06T07:23:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.339+0000 7f69b27fc640 1 -- 172.21.15.77:0/2466681086 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f69bc0583b0 con 0x7f69c00d24b0 2024-03-06T07:23:26.341 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:23:26.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69a406b500 msgr2=0x7f69a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69a406b500 0x7f69a406d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f69b400a770 tx=0x7f69b4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 msgr2=0x7f69c00b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b8d70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00c440 tx=0x7f69bc00c900 comp rx=0 tx=0).stop 2024-03-06T07:23:26.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 shutdown_connections 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69a406b500 0x7f69a406d9c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69c00d24b0 0x7f69c00b8d70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69c01614d0 0x7f69c00c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 --2- 172.21.15.77:0/2466681086 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69c00d1910 0x7f69c014e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 >> 172.21.15.77:0/2466681086 conn(0x7f69c00d3d40 msgr2=0x7f69c00d1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 shutdown_connections 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.341+0000 7f69c7c81640 1 -- 172.21.15.77:0/2466681086 wait complete. 2024-03-06T07:23:26.354 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:23:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.453+0000 7fa029ab5640 1 Processor -- start 2024-03-06T07:23:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.453+0000 7fa029ab5640 1 -- start start 2024-03-06T07:23:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.453+0000 7fa029ab5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0240d2380 0x7fa0240d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.453+0000 7fa029ab5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0240cee00 0x7fa0240cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.453+0000 7fa029ab5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240b3fc0 con 0x7fa02415e0b0 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.453+0000 7fa029ab5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0240ce990 con 0x7fa0240d2380 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.453+0000 7fa029ab5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0240d2d20 con 0x7fa0240cee00 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.454+0000 7fa022ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0240d2380 0x7fa0240d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.454+0000 7fa022ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0240d2380 0x7fa0240d27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36134/0 (socket says 172.21.15.77:36134) 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.454+0000 7fa022ffd640 1 -- 172.21.15.77:0/1222573347 learned_addr learned my addr 172.21.15.77:0/1222573347 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.454+0000 7fa023fff640 1 --2- 172.21.15.77:0/1222573347 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0240cee00 0x7fa0240cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.454+0000 7fa022ffd640 1 -- 172.21.15.77:0/1222573347 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0240cee00 msgr2=0x7fa0240cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.454+0000 7fa022ffd640 1 --2- 172.21.15.77:0/1222573347 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0240cee00 0x7fa0240cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.454+0000 7fa022ffd640 1 -- 172.21.15.77:0/1222573347 >> v1:172.21.15.158:6789/0 conn(0x7fa02415e0b0 legacy=0x7fa0240b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.454+0000 7fa022ffd640 1 -- 172.21.15.77:0/1222573347 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0240b4290 con 0x7fa0240d2380 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa022ffd640 1 --2- 172.21.15.77:0/1222573347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240d2380 0x7fa0240d27e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009870 tx=0x7fa00c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578c2898074ae456 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa0227fc640 1 -- 172.21.15.77:0/1222573347 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c034070 con 0x7fa0240d2380 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa0227fc640 1 -- 172.21.15.77:0/1222573347 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c030840 con 0x7fa0240d2380 2024-03-06T07:23:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa0227fc640 1 -- 172.21.15.77:0/1222573347 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c030c10 con 0x7fa0240d2380 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa029ab5640 1 -- 172.21.15.77:0/1222573347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240d2380 msgr2=0x7fa0240d27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa029ab5640 1 --2- 172.21.15.77:0/1222573347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240d2380 0x7fa0240d27e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009870 tx=0x7fa00c02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa029ab5640 1 -- 172.21.15.77:0/1222573347 shutdown_connections 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa029ab5640 1 --2- 172.21.15.77:0/1222573347 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0240cee00 0x7fa0240cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa029ab5640 1 --2- 172.21.15.77:0/1222573347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240d2380 0x7fa0240d27e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa029ab5640 1 -- 172.21.15.77:0/1222573347 >> 172.21.15.77:0/1222573347 conn(0x7fa02408d6f0 msgr2=0x7fa02409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa029ab5640 1 -- 172.21.15.77:0/1222573347 shutdown_connections 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.455+0000 7fa029ab5640 1 -- 172.21.15.77:0/1222573347 wait complete. 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.456+0000 7fa029ab5640 1 Processor -- start 2024-03-06T07:23:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.457+0000 7fa029ab5640 1 -- start start 2024-03-06T07:23:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.457+0000 7fa029ab5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240cee00 0x7fa0240ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.457+0000 7fa029ab5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0240d2380 0x7fa02414b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.457+0000 7fa029ab5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa02415e0b0 0x7fa02415f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.457+0000 7fa029ab5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0240b4b30 con 0x7fa0240cee00 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.457+0000 7fa029ab5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0240b4850 con 0x7fa0240d2380 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.457+0000 7fa029ab5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02415e490 con 0x7fa02415e0b0 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.457+0000 7fa0237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240cee00 0x7fa0240ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa0237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240cee00 0x7fa0240ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36138/0 (socket says 172.21.15.77:36138) 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa0237fe640 1 -- 172.21.15.77:0/3221610287 learned_addr learned my addr 172.21.15.77:0/3221610287 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa023fff640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa02415e0b0 0x7fa02415f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa022ffd640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0240d2380 0x7fa02414b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa0237fe640 1 -- 172.21.15.77:0/3221610287 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa02415e0b0 msgr2=0x7fa02415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa0237fe640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa02415e0b0 0x7fa02415f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa0237fe640 1 -- 172.21.15.77:0/3221610287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0240d2380 msgr2=0x7fa02414b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa0237fe640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0240d2380 0x7fa02414b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa0237fe640 1 -- 172.21.15.77:0/3221610287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02414de50 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa0237fe640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240cee00 0x7fa0240ca170 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa01400c7c0 tx=0x7fa01400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa01bfff640 1 -- 172.21.15.77:0/3221610287 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014016020 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa01bfff640 1 -- 172.21.15.77:0/3221610287 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa014006d50 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0240b6bd0 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa01bfff640 1 -- 172.21.15.77:0/3221610287 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014003bb0 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.458+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa024160280 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.459+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0240cdfc0 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.460+0000 7fa01bfff640 1 -- 172.21.15.77:0/3221610287 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa01400a850 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.460+0000 7fa01bfff640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa00406b4d0 0x7fa00406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.460+0000 7fa01bfff640 1 -- 172.21.15.77:0/3221610287 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa014089c90 con 0x7fa0240cee00 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.460+0000 7fa022ffd640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa00406b4d0 0x7fa00406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.460+0000 7fa022ffd640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa00406b4d0 0x7fa00406d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa00c036000 tx=0x7fa00c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.464+0000 7fa01bfff640 1 -- 172.21.15.77:0/3221610287 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0240cdfc0 con 0x7fa0240cee00 2024-03-06T07:23:26.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.707+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7fa024160480 con 0x7fa0240cee00 2024-03-06T07:23:26.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.739+0000 7fa01bfff640 1 -- 172.21.15.77:0/3221610287 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v98) v1 ==== 136+0+0 (secure 0 0 0) 0x7fa024160480 con 0x7fa0240cee00 2024-03-06T07:23:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa00406b4d0 msgr2=0x7fa00406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa00406b4d0 0x7fa00406d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa00c036000 tx=0x7fa00c03b040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240cee00 msgr2=0x7fa0240ca170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240cee00 0x7fa0240ca170 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa01400c7c0 tx=0x7fa01400cc80 comp rx=0 tx=0).stop 2024-03-06T07:23:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 shutdown_connections 2024-03-06T07:23:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa02415e0b0 0x7fa02415f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0240d2380 0x7fa02414b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa00406b4d0 0x7fa00406d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 --2- 172.21.15.77:0/3221610287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0240cee00 0x7fa0240ca170 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 >> 172.21.15.77:0/3221610287 conn(0x7fa02408d6f0 msgr2=0x7fa02406cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 shutdown_connections 2024-03-06T07:23:26.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.741+0000 7fa029ab5640 1 -- 172.21.15.77:0/3221610287 wait complete. 2024-03-06T07:23:26.755 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:26.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.856+0000 7fda93c76640 1 Processor -- start 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.856+0000 7fda93c76640 1 -- start start 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda93c76640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fda8c15a220 0x7fda8c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda93c76640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fda8c0712d0 0x7fda8c071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda93c76640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda93c76640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c0cf610 con 0x7fda8c0712d0 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda93c76640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c071c70 con 0x7fda8c15a220 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda93c76640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0b3320 con 0x7fda8c15de60 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91211640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fda8c0712d0 0x7fda8c071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91211640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fda8c0712d0 0x7fda8c071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36140/0 (socket says 172.21.15.77:36140) 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91211640 1 -- 172.21.15.77:0/104355952 learned_addr learned my addr 172.21.15.77:0/104355952 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda92213640 1 --2- 172.21.15.77:0/104355952 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91a12640 1 --2- 172.21.15.77:0/104355952 >> v2:172.21.15.158:3300/0 conn(0x7fda8c15a220 0x7fda8c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91211640 1 -- 172.21.15.77:0/104355952 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 msgr2=0x7fda8c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91211640 1 --2- 172.21.15.77:0/104355952 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c15e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91211640 1 -- 172.21.15.77:0/104355952 >> v2:172.21.15.158:3300/0 conn(0x7fda8c15a220 msgr2=0x7fda8c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91211640 1 --2- 172.21.15.77:0/104355952 >> v2:172.21.15.158:3300/0 conn(0x7fda8c15a220 0x7fda8c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.857+0000 7fda91211640 1 -- 172.21.15.77:0/104355952 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c0728d0 con 0x7fda8c0712d0 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.858+0000 7fda91211640 1 --2- 172.21.15.77:0/104355952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c071730 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fda8c06ebb0 tx=0x7fda7c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f4c442ad2f5e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.858+0000 7fda90a10640 1 -- 172.21.15.77:0/104355952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c02e070 con 0x7fda8c0712d0 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.858+0000 7fda90a10640 1 -- 172.21.15.77:0/104355952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c037070 con 0x7fda8c0712d0 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.858+0000 7fda90a10640 1 -- 172.21.15.77:0/104355952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c031a00 con 0x7fda8c0712d0 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.858+0000 7fda93c76640 1 -- 172.21.15.77:0/104355952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 msgr2=0x7fda8c071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.858+0000 7fda93c76640 1 --2- 172.21.15.77:0/104355952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c071730 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fda8c06ebb0 tx=0x7fda7c02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.859+0000 7fda93c76640 1 -- 172.21.15.77:0/104355952 shutdown_connections 2024-03-06T07:23:26.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.859+0000 7fda93c76640 1 --2- 172.21.15.77:0/104355952 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c15e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.859+0000 7fda93c76640 1 --2- 172.21.15.77:0/104355952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c071730 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.859+0000 7fda93c76640 1 --2- 172.21.15.77:0/104355952 >> v2:172.21.15.158:3300/0 conn(0x7fda8c15a220 0x7fda8c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.859+0000 7fda93c76640 1 -- 172.21.15.77:0/104355952 >> 172.21.15.77:0/104355952 conn(0x7fda8c06b190 msgr2=0x7fda8c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.859+0000 7fda93c76640 1 -- 172.21.15.77:0/104355952 shutdown_connections 2024-03-06T07:23:26.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.859+0000 7fda93c76640 1 -- 172.21.15.77:0/104355952 wait complete. 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.860+0000 7fda93c76640 1 Processor -- start 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.860+0000 7fda93c76640 1 -- start start 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.860+0000 7fda93c76640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.860+0000 7fda93c76640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda8c15a220 0x7fda8c14a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.860+0000 7fda93c76640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c0ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.860+0000 7fda93c76640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c072030 con 0x7fda8c0712d0 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda93c76640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c072bb0 con 0x7fda8c15a220 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda93c76640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c072530 con 0x7fda8c15de60 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c147d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36144/0 (socket says 172.21.15.77:36144) 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 -- 172.21.15.77:0/124454226 learned_addr learned my addr 172.21.15.77:0/124454226 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda92213640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c0ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91211640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda8c15a220 0x7fda8c14a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 -- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 msgr2=0x7fda8c0ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c0ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 -- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda8c15a220 msgr2=0x7fda8c14a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda8c15a220 0x7fda8c14a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 -- 172.21.15.77:0/124454226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c1483e0 con 0x7fda8c0712d0 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda92213640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c0ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91211640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda8c15a220 0x7fda8c14a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda91a12640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c147d20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fda8800e890 tx=0x7fda8800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda827fc640 1 -- 172.21.15.77:0/124454226 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88017070 con 0x7fda8c0712d0 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda827fc640 1 -- 172.21.15.77:0/124454226 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8800c940 con 0x7fda8c0712d0 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c1486c0 con 0x7fda8c0712d0 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda827fc640 1 -- 172.21.15.77:0/124454226 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8800cd10 con 0x7fda8c0712d0 2024-03-06T07:23:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.861+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c148c80 con 0x7fda8c0712d0 2024-03-06T07:23:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.862+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c0732b0 con 0x7fda8c0712d0 2024-03-06T07:23:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.863+0000 7fda827fc640 1 -- 172.21.15.77:0/124454226 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda8801a070 con 0x7fda8c0712d0 2024-03-06T07:23:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.863+0000 7fda827fc640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda7006b4d0 0x7fda7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.863+0000 7fda91211640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda7006b4d0 0x7fda7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.863+0000 7fda827fc640 1 -- 172.21.15.77:0/124454226 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fda8808a2b0 con 0x7fda8c0712d0 2024-03-06T07:23:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.863+0000 7fda91211640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda7006b4d0 0x7fda7006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fda7c036000 tx=0x7fda7c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:26.867+0000 7fda827fc640 1 -- 172.21.15.77:0/124454226 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda880593a0 con 0x7fda8c0712d0 2024-03-06T07:23:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.038+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fda8c068ff0 con 0x7fda8c0712d0 2024-03-06T07:23:27.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.041+0000 7fda827fc640 1 -- 172.21.15.77:0/124454226 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fda8808a0d0 con 0x7fda8c0712d0 2024-03-06T07:23:27.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda7006b4d0 msgr2=0x7fda7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda7006b4d0 0x7fda7006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fda7c036000 tx=0x7fda7c03d040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 msgr2=0x7fda8c147d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c147d20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fda8800e890 tx=0x7fda8800ed50 comp rx=0 tx=0).stop 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 shutdown_connections 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda8c15de60 0x7fda8c0ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda8c15a220 0x7fda8c14a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda7006b4d0 0x7fda7006d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 --2- 172.21.15.77:0/124454226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c0712d0 0x7fda8c147d20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 >> 172.21.15.77:0/124454226 conn(0x7fda8c06b190 msgr2=0x7fda8c159230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 shutdown_connections 2024-03-06T07:23:27.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.043+0000 7fda93c76640 1 -- 172.21.15.77:0/124454226 wait complete. 2024-03-06T07:23:27.057 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:27.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:27.098 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:27.153 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:27.154 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:23:27.154 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:27.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:27.169 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:23:27.169 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:27.191 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:27.191 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:23:27.191 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:27.225 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:27.225 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:23:27.225 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:27.249 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:27.249 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:23:27.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:27.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.363+0000 7f2a54a8d640 1 Processor -- start 2024-03-06T07:23:27.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.363+0000 7f2a54a8d640 1 -- start start 2024-03-06T07:23:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a54a8d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a501515e0 0x7f2a500cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a54a8d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a50071f40 con 0x7f2a5015b430 2024-03-06T07:23:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a54a8d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5015b8b0 con 0x7f2a500cd7a0 2024-03-06T07:23:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a54a8d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a50070a70 con 0x7f2a501515e0 2024-03-06T07:23:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4dd9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2a5015b430 0x7f2a50071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49658/0 (socket says 172.21.15.77:49658) 2024-03-06T07:23:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4dd9b640 1 -- 172.21.15.77:0/4107807211 learned_addr learned my addr 172.21.15.77:0/4107807211 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051100063 0 0) 0x7f2a50071f40 con 0x7f2a5015b430 2024-03-06T07:23:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a38003520 con 0x7f2a5015b430 2024-03-06T07:23:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4e59c640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a501515e0 0x7f2a500cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1591878673 0 0) 0x7f2a38003520 con 0x7f2a5015b430 2024-03-06T07:23:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a501515e0 msgr2=0x7f2a500cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a501515e0 0x7f2a500cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 >> v1:172.21.15.158:6789/0 conn(0x7f2a500cd7a0 legacy=0x7f2a500cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a50070d50 con 0x7f2a5015b430 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f2a44001670 con 0x7f2a5015b430 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 >> v1:172.21.15.77:6789/0 conn(0x7f2a5015b430 legacy=0x7f2a50071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a501515e0 0x7f2a380039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a38004010 0x7f2a38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a38004d40 0x7f2a380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a50070a70 con 0x7f2a38004010 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3800bb80 con 0x7f2a38004d40 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3800bd80 con 0x7f2a501515e0 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4dd9b640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a38004010 0x7f2a38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.364+0000 7f2a4e59c640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a501515e0 0x7f2a380039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.365+0000 7f2a4ed9d640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a38004d40 0x7f2a380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.365+0000 7f2a4dd9b640 1 -- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a501515e0 msgr2=0x7f2a380039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.365+0000 7f2a4dd9b640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a501515e0 0x7f2a380039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.365+0000 7f2a4dd9b640 1 -- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a38004d40 msgr2=0x7f2a380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.365+0000 7f2a4dd9b640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a38004d40 0x7f2a380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.365+0000 7f2a4dd9b640 1 -- 172.21.15.77:0/4107807211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a38003520 con 0x7f2a38004010 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.365+0000 7f2a4ed9d640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a38004d40 0x7f2a380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.365+0000 7f2a4dd9b640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a38004010 0x7f2a38004800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400c210 tx=0x7f2a38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf24ed62865ac85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a44007d30 con 0x7f2a38004010 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a44030070 con 0x7f2a38004010 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a4d59a640 1 -- 172.21.15.77:0/4107807211 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4402cda0 con 0x7f2a38004010 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a54a8d640 1 -- 172.21.15.77:0/4107807211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a38004010 msgr2=0x7f2a38004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a54a8d640 1 --2- 172.21.15.77:0/4107807211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a38004010 0x7f2a38004800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400c210 tx=0x7f2a38005830 comp rx=0 tx=0).stop 2024-03-06T07:23:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a4ed9d640 1 -- 172.21.15.77:0/4107807211 reap_dead start 2024-03-06T07:23:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a54a8d640 1 -- 172.21.15.77:0/4107807211 shutdown_connections 2024-03-06T07:23:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a54a8d640 1 -- 172.21.15.77:0/4107807211 >> 172.21.15.77:0/4107807211 conn(0x7f2a500cf8a0 msgr2=0x7f2a500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a54a8d640 1 -- 172.21.15.77:0/4107807211 shutdown_connections 2024-03-06T07:23:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.366+0000 7f2a54a8d640 1 -- 172.21.15.77:0/4107807211 wait complete. 2024-03-06T07:23:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.367+0000 7f2a54a8d640 1 Processor -- start 2024-03-06T07:23:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a54a8d640 1 -- start start 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a54a8d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a38004010 0x7f2a5015df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a54a8d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a50143ec0 0x7f2a50144340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a54a8d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50146120 0x7f2a501465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a54a8d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a500cce40 con 0x7f2a50146120 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a54a8d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a50071310 con 0x7f2a50143ec0 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a54a8d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a500ccfc0 con 0x7f2a38004010 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a4ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50146120 0x7f2a501465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a4ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50146120 0x7f2a501465a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36154/0 (socket says 172.21.15.77:36154) 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.368+0000 7f2a4ed9d640 1 -- 172.21.15.77:0/2443607185 learned_addr learned my addr 172.21.15.77:0/2443607185 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a4ed9d640 1 -- 172.21.15.77:0/2443607185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a38004010 msgr2=0x7f2a5015df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a4ed9d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a38004010 0x7f2a5015df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a4ed9d640 1 -- 172.21.15.77:0/2443607185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a50143ec0 msgr2=0x7f2a50144340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a4ed9d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a50143ec0 0x7f2a50144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a4ed9d640 1 -- 172.21.15.77:0/2443607185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a50146de0 con 0x7f2a50146120 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a4ed9d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50146120 0x7f2a501465a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2a48000f80 tx=0x7f2a4800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a3effd640 1 -- 172.21.15.77:0/2443607185 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48006b50 con 0x7f2a50146120 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a3effd640 1 -- 172.21.15.77:0/2443607185 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a48006d10 con 0x7f2a50146120 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a501470c0 con 0x7f2a50146120 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a3effd640 1 -- 172.21.15.77:0/2443607185 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48003970 con 0x7f2a50146120 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.369+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a50147680 con 0x7f2a50146120 2024-03-06T07:23:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.370+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a50068840 con 0x7f2a50146120 2024-03-06T07:23:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.370+0000 7f2a3effd640 1 -- 172.21.15.77:0/2443607185 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a4801a020 con 0x7f2a50146120 2024-03-06T07:23:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.371+0000 7f2a3effd640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a3006b500 0x7f2a3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.371+0000 7f2a4e59c640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a3006b500 0x7f2a3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.371+0000 7f2a3effd640 1 -- 172.21.15.77:0/2443607185 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2a48089520 con 0x7f2a50146120 2024-03-06T07:23:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.371+0000 7f2a4e59c640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a3006b500 0x7f2a3006d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2a40006a90 tx=0x7f2a40006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.375+0000 7f2a3effd640 1 -- 172.21.15.77:0/2443607185 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a48006f00 con 0x7f2a50146120 2024-03-06T07:23:27.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.583+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a5004f330 con 0x7f2a50146120 2024-03-06T07:23:27.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.583+0000 7f2a3effd640 1 -- 172.21.15.77:0/2443607185 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+3698 (secure 0 0 0) 0x7f2a480586a0 con 0x7f2a50146120 2024-03-06T07:23:27.586 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:27.586 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":15,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3501932573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3501932573},{"type":"v1","addr":"172.21.15.158:6835","nonce":3501932573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3774716138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3774716138},{"type":"v1","addr":"172.21.15.158:6834","nonce":3774716138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":15},{"gid":5202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1588016475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1588016475},{"type":"v1","addr":"172.21.15.77:6836","nonce":1588016475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5205,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/79210774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":79210774},{"type":"v1","addr":"172.21.15.77:6837","nonce":79210774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-06T07:23:27.586 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 15 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a3006b500 msgr2=0x7f2a3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a3006b500 0x7f2a3006d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2a40006a90 tx=0x7f2a40006a20 comp rx=0 tx=0).stop 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50146120 msgr2=0x7f2a501465a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50146120 0x7f2a501465a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2a48000f80 tx=0x7f2a4800eb90 comp rx=0 tx=0).stop 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 shutdown_connections 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a3006b500 0x7f2a3006d9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a50146120 0x7f2a501465a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a50143ec0 0x7f2a50144340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.585+0000 7f2a54a8d640 1 --2- 172.21.15.77:0/2443607185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a38004010 0x7f2a5015df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.586+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 >> 172.21.15.77:0/2443607185 conn(0x7f2a500cf8a0 msgr2=0x7f2a5014f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.586+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 shutdown_connections 2024-03-06T07:23:27.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.586+0000 7f2a54a8d640 1 -- 172.21.15.77:0/2443607185 wait complete. 2024-03-06T07:23:27.599 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:27.599 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:27.599 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:27.599 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:27.599 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:27.599 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:27.600 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:27.600 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:27.600 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:27.600 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:27.600 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:27.600 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:27.600 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:27.600 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:27.601 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:27.601 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:27.609 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:27.609 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:27.610 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:27.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:27.612 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:27.612 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:27.613 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:27.613 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:27.652 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:27.652 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:27.653 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:27.653 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:27.655 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:27.655 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:27.656 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:27.656 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:27.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:27.728 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:23:27.729 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:23:27.738 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:23:27.738 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:23:27.739 INFO:tasks.ceph.mds.d.smithi158.stderr:2024-03-06T07:23:27.736+0000 7f0ca178d640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:23:27.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.760+0000 7ff1ff53a640 1 Processor -- start 2024-03-06T07:23:27.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.760+0000 7ff1ff53a640 1 -- start start 2024-03-06T07:23:27.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1ff53a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff1f80d0d90 0x7ff1f80ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1ff53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f80c90f0 0x7ff1f80ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1ff53a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80d25e0 con 0x7ff1f815e000 2024-03-06T07:23:27.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1ff53a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f80c7c50 con 0x7ff1f80d0d90 2024-03-06T07:23:27.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1ff53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80d2880 con 0x7ff1f80c90f0 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1fcad5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff1f80d0d90 0x7ff1f80ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1fcad5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff1f80d0d90 0x7ff1f80ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36184/0 (socket says 172.21.15.77:36184) 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1fd2d6640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff1f815e000 0x7ff1f80d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49670/0 (socket says 172.21.15.77:49670) 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1fcad5640 1 -- 172.21.15.77:0/1130798054 learned_addr learned my addr 172.21.15.77:0/1130798054 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1fcad5640 1 -- 172.21.15.77:0/1130798054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f80c90f0 msgr2=0x7ff1f80ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1fcad5640 1 --2- 172.21.15.77:0/1130798054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f80c90f0 0x7ff1f80ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1fcad5640 1 -- 172.21.15.77:0/1130798054 >> v1:172.21.15.77:6789/0 conn(0x7ff1f815e000 legacy=0x7ff1f80d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.761+0000 7ff1fcad5640 1 -- 172.21.15.77:0/1130798054 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80d2b40 con 0x7ff1f80d0d90 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.762+0000 7ff1fcad5640 1 --2- 172.21.15.77:0/1130798054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80ce7c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009870 tx=0x7ff1e802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92421ed61d2186f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.762+0000 7ff1effff640 1 -- 172.21.15.77:0/1130798054 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e802f840 con 0x7ff1f80d0d90 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.762+0000 7ff1effff640 1 -- 172.21.15.77:0/1130798054 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e802f9e0 con 0x7ff1f80d0d90 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.762+0000 7ff1effff640 1 -- 172.21.15.77:0/1130798054 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e802fdb0 con 0x7ff1f80d0d90 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.763+0000 7ff1ff53a640 1 -- 172.21.15.77:0/1130798054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 msgr2=0x7ff1f80ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.763+0000 7ff1ff53a640 1 --2- 172.21.15.77:0/1130798054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80ce7c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009870 tx=0x7ff1e802ba20 comp rx=0 tx=0).stop 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.763+0000 7ff1ff53a640 1 -- 172.21.15.77:0/1130798054 shutdown_connections 2024-03-06T07:23:27.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.763+0000 7ff1ff53a640 1 --2- 172.21.15.77:0/1130798054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f80c90f0 0x7ff1f80ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.763+0000 7ff1ff53a640 1 --2- 172.21.15.77:0/1130798054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80ce7c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.763+0000 7ff1ff53a640 1 -- 172.21.15.77:0/1130798054 >> 172.21.15.77:0/1130798054 conn(0x7ff1f80d3cf0 msgr2=0x7ff1f80d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.763+0000 7ff1ff53a640 1 -- 172.21.15.77:0/1130798054 shutdown_connections 2024-03-06T07:23:27.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.763+0000 7ff1ff53a640 1 -- 172.21.15.77:0/1130798054 wait complete. 2024-03-06T07:23:27.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.764+0000 7ff1ff53a640 1 Processor -- start 2024-03-06T07:23:27.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.764+0000 7ff1ff53a640 1 -- start start 2024-03-06T07:23:27.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1ff53a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1f80c90f0 0x7ff1f815b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1ff53a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1ff53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f815e000 0x7ff1f80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1ff53a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f815e3e0 con 0x7ff1f80d0d90 2024-03-06T07:23:27.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1ff53a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80d22d0 con 0x7ff1f80c90f0 2024-03-06T07:23:27.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1ff53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80d4810 con 0x7ff1f815e000 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36194/0 (socket says 172.21.15.77:36194) 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 -- 172.21.15.77:0/2426095359 learned_addr learned my addr 172.21.15.77:0/2426095359 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fd2d6640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1f80c90f0 0x7ff1f815b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fdad7640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f815e000 0x7ff1f80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 -- 172.21.15.77:0/2426095359 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f815e000 msgr2=0x7ff1f80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f815e000 0x7ff1f80cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 -- 172.21.15.77:0/2426095359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1f80c90f0 msgr2=0x7ff1f815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1f80c90f0 0x7ff1f815b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 -- 172.21.15.77:0/2426095359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80b1510 con 0x7ff1f80d0d90 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1fcad5640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80cd280 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009870 tx=0x7ff1e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1edffb640 1 -- 172.21.15.77:0/2426095359 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8042480 con 0x7ff1f80d0d90 2024-03-06T07:23:27.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1edffb640 1 -- 172.21.15.77:0/2426095359 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e8047070 con 0x7ff1f80d0d90 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1ff53a640 1 -- 172.21.15.77:0/2426095359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f80b17f0 con 0x7ff1f80d0d90 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.765+0000 7ff1edffb640 1 -- 172.21.15.77:0/2426095359 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8042880 con 0x7ff1f80d0d90 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.766+0000 7ff1ff53a640 1 -- 172.21.15.77:0/2426095359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f80b1db0 con 0x7ff1f80d0d90 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.766+0000 7ff1e37fe640 1 -- 172.21.15.77:0/2426095359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d0005290 con 0x7ff1f80d0d90 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.767+0000 7ff1edffb640 1 -- 172.21.15.77:0/2426095359 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1e8040050 con 0x7ff1f80d0d90 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.767+0000 7ff1edffb640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1dc06b4d0 0x7ff1dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.767+0000 7ff1edffb640 1 -- 172.21.15.77:0/2426095359 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff1e80accc0 con 0x7ff1f80d0d90 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.767+0000 7ff1fd2d6640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1dc06b4d0 0x7ff1dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.768+0000 7ff1fd2d6640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1dc06b4d0 0x7ff1dc06d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4000f60 tx=0x7ff1f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.771+0000 7ff1edffb640 1 -- 172.21.15.77:0/2426095359 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1e80b3080 con 0x7ff1f80d0d90 2024-03-06T07:23:27.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:27.954+0000 7ff1e37fe640 1 -- 172.21.15.77:0/2426095359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff1d0005580 con 0x7ff1f80d0d90 2024-03-06T07:23:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.066+0000 7ff1edffb640 1 -- 172.21.15.77:0/2426095359 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v41) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff1e8051020 con 0x7ff1f80d0d90 2024-03-06T07:23:29.068 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 -- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1dc06b4d0 msgr2=0x7ff1dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1dc06b4d0 0x7ff1dc06d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4000f60 tx=0x7ff1f4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 -- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 msgr2=0x7ff1f80cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80cd280 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009870 tx=0x7ff1e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 -- 172.21.15.77:0/2426095359 shutdown_connections 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1dc06b4d0 0x7ff1dc06d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f80d0d90 0x7ff1f80cd280 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1f815e000 0x7ff1f80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 --2- 172.21.15.77:0/2426095359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1f80c90f0 0x7ff1f815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 -- 172.21.15.77:0/2426095359 >> 172.21.15.77:0/2426095359 conn(0x7ff1f80d3cf0 msgr2=0x7ff1f80d18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 -- 172.21.15.77:0/2426095359 shutdown_connections 2024-03-06T07:23:29.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.069+0000 7ff1e37fe640 1 -- 172.21.15.77:0/2426095359 wait complete. 2024-03-06T07:23:29.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.187+0000 7f066f9f7640 1 Processor -- start 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066f9f7640 1 -- start start 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066f9f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f066815dfe0 0x7f06680cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066f9f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06680cf830 0x7f06680d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066f9f7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06680b3430 con 0x7f06680d27f0 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066f9f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06680d2c50 con 0x7f066815dfe0 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066f9f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06680b31e0 con 0x7f06680cf830 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066d793640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f066815dfe0 0x7f06680cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066df94640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f06680d27f0 0x7f06680d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56200/0 (socket says 172.21.15.77:56200) 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.188+0000 7f066df94640 1 -- 172.21.15.77:0/1866989953 learned_addr learned my addr 172.21.15.77:0/1866989953 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.189+0000 7f066d793640 1 -- 172.21.15.77:0/1866989953 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06680cf830 msgr2=0x7f06680d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.189+0000 7f066d793640 1 --2- 172.21.15.77:0/1866989953 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06680cf830 0x7f06680d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.189+0000 7f066d793640 1 -- 172.21.15.77:0/1866989953 >> v1:172.21.15.77:6789/0 conn(0x7f06680d27f0 legacy=0x7f06680d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.189+0000 7f065ffff640 1 -- 172.21.15.77:0/1866989953 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111039955 0 0) 0x7f06680b3430 con 0x7f06680d27f0 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.189+0000 7f066d793640 1 -- 172.21.15.77:0/1866989953 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06680b3710 con 0x7f066815dfe0 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.189+0000 7f066d793640 1 --2- 172.21.15.77:0/1866989953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680cf2f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0664009d90 tx=0x7f066402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb8810aad61c285 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f065ffff640 1 -- 172.21.15.77:0/1866989953 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664039070 con 0x7f066815dfe0 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f065ffff640 1 -- 172.21.15.77:0/1866989953 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066402cc50 con 0x7f066815dfe0 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f065ffff640 1 -- 172.21.15.77:0/1866989953 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664034440 con 0x7f066815dfe0 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f066f9f7640 1 -- 172.21.15.77:0/1866989953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 msgr2=0x7f06680cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f066f9f7640 1 --2- 172.21.15.77:0/1866989953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680cf2f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0664009d90 tx=0x7f066402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f066f9f7640 1 -- 172.21.15.77:0/1866989953 shutdown_connections 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f066f9f7640 1 --2- 172.21.15.77:0/1866989953 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06680cf830 0x7f06680d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f066f9f7640 1 --2- 172.21.15.77:0/1866989953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680cf2f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f066f9f7640 1 -- 172.21.15.77:0/1866989953 >> 172.21.15.77:0/1866989953 conn(0x7f06680d3cd0 msgr2=0x7f06680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f066f9f7640 1 -- 172.21.15.77:0/1866989953 shutdown_connections 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.190+0000 7f066f9f7640 1 -- 172.21.15.77:0/1866989953 wait complete. 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.191+0000 7f066f9f7640 1 Processor -- start 2024-03-06T07:23:29.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066f9f7640 1 -- start start 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066f9f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f06680cf830 0x7f0668160990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066f9f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06680d27f0 0x7f06680cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066f9f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066f9f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06680d1c70 con 0x7f066815dfe0 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066f9f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06680b3d60 con 0x7f06680cf830 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066f9f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06680d1df0 con 0x7f06680d27f0 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066df94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066df94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33136/0 (socket says 172.21.15.77:33136) 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.192+0000 7f066df94640 1 -- 172.21.15.77:0/3087701401 learned_addr learned my addr 172.21.15.77:0/3087701401 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f066df94640 1 -- 172.21.15.77:0/3087701401 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06680d27f0 msgr2=0x7f06680cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f066df94640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06680d27f0 0x7f06680cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f066df94640 1 -- 172.21.15.77:0/3087701401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f06680cf830 msgr2=0x7f0668160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f066df94640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f06680cf830 0x7f0668160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f066df94640 1 -- 172.21.15.77:0/3087701401 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06680b9770 con 0x7f066815dfe0 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f066df94640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680b68a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f066000c480 tx=0x7f066000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f065dffb640 1 -- 172.21.15.77:0/3087701401 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660018070 con 0x7f066815dfe0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f065dffb640 1 -- 172.21.15.77:0/3087701401 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0660007500 con 0x7f066815dfe0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06680b9a50 con 0x7f066815dfe0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f065dffb640 1 -- 172.21.15.77:0/3087701401 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660004a20 con 0x7f066815dfe0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.193+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066815fe80 con 0x7f066815dfe0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.194+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066804f330 con 0x7f066815dfe0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.194+0000 7f065dffb640 1 -- 172.21.15.77:0/3087701401 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f066001b070 con 0x7f066815dfe0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.195+0000 7f065dffb640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f064c06b500 0x7f064c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.195+0000 7f065dffb640 1 -- 172.21.15.77:0/3087701401 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0660089170 con 0x7f066815dfe0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.195+0000 7f066d793640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f064c06b500 0x7f064c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.195+0000 7f066d793640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f064c06b500 0x7f064c06d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0664036000 tx=0x7f066402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.199+0000 7f065dffb640 1 -- 172.21.15.77:0/3087701401 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06600583d0 con 0x7f066815dfe0 2024-03-06T07:23:29.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.437+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f06680747a0 con 0x7f066815dfe0 2024-03-06T07:23:29.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.438+0000 7f065dffb640 1 -- 172.21.15.77:0/3087701401 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0660057cd0 con 0x7f066815dfe0 2024-03-06T07:23:29.440 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:29.440 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:23:29.440 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:23:29.440 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.440 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:23:29.441 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.442 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.443 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.444 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:23:29.445 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:23:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.447 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-03-06T07:23:29.448 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.449 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:23:29.450 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.451 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:29.452 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:23:29.453 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.453 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:29.453 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:23:29.453 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:23:29.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.439+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f064c06b500 msgr2=0x7f064c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.439+0000 7f066f9f7640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f064c06b500 0x7f064c06d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0664036000 tx=0x7f066402e000 comp rx=0 tx=0).stop 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.439+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 msgr2=0x7f06680b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.439+0000 7f066f9f7640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680b68a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f066000c480 tx=0x7f066000c940 comp rx=0 tx=0).stop 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.440+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 shutdown_connections 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.440+0000 7f066f9f7640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f064c06b500 0x7f064c06d9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.440+0000 7f066f9f7640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f066815dfe0 0x7f06680b68a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.440+0000 7f066f9f7640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06680d27f0 0x7f06680cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.440+0000 7f066f9f7640 1 --2- 172.21.15.77:0/3087701401 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f06680cf830 0x7f0668160990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.440+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 >> 172.21.15.77:0/3087701401 conn(0x7f06680d3cd0 msgr2=0x7f066815d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.440+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 shutdown_connections 2024-03-06T07:23:29.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.440+0000 7f066f9f7640 1 -- 172.21.15.77:0/3087701401 wait complete. 2024-03-06T07:23:29.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:29.484 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:29.501 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:29.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.555+0000 7f5762ab5640 1 Processor -- start 2024-03-06T07:23:29.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.555+0000 7f5762ab5640 1 -- start start 2024-03-06T07:23:29.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5762ab5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f575c13dc10 0x7f575c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5762ab5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15ef60 0x7f575c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5762ab5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c15f580 con 0x7f575c15bf30 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5762ab5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c0cf3f0 con 0x7f575c13dc10 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5762ab5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c15b210 con 0x7f575c15ef60 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5760851640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f575c13dc10 0x7f575c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f575bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f575c15bf30 0x7f575c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56212/0 (socket says 172.21.15.77:56212) 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5760851640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f575c13dc10 0x7f575c0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33146/0 (socket says 172.21.15.77:33146) 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f575bfff640 1 -- 172.21.15.77:0/1164873502 learned_addr learned my addr 172.21.15.77:0/1164873502 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5760851640 1 -- 172.21.15.77:0/1164873502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15ef60 msgr2=0x7f575c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5760851640 1 --2- 172.21.15.77:0/1164873502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15ef60 0x7f575c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5760851640 1 -- 172.21.15.77:0/1164873502 >> v1:172.21.15.77:6789/0 conn(0x7f575c15bf30 legacy=0x7f575c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f5760851640 1 -- 172.21.15.77:0/1164873502 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c15b450 con 0x7f575c13dc10 2024-03-06T07:23:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.556+0000 7f575b7fe640 1 -- 172.21.15.77:0/1164873502 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738701272 0 0) 0x7f575c15f580 con 0x7f575c15bf30 2024-03-06T07:23:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.557+0000 7f5760851640 1 --2- 172.21.15.77:0/1164873502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c0b2950 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f57500097a0 tx=0x7f575002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aee7570af07d77a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.557+0000 7f575b7fe640 1 -- 172.21.15.77:0/1164873502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57500315a0 con 0x7f575c13dc10 2024-03-06T07:23:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.557+0000 7f575b7fe640 1 -- 172.21.15.77:0/1164873502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5750037070 con 0x7f575c13dc10 2024-03-06T07:23:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.557+0000 7f575b7fe640 1 -- 172.21.15.77:0/1164873502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750031960 con 0x7f575c13dc10 2024-03-06T07:23:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.558+0000 7f5762ab5640 1 -- 172.21.15.77:0/1164873502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 msgr2=0x7f575c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.558+0000 7f5762ab5640 1 --2- 172.21.15.77:0/1164873502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c0b2950 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f57500097a0 tx=0x7f575002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.558+0000 7f5762ab5640 1 -- 172.21.15.77:0/1164873502 shutdown_connections 2024-03-06T07:23:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.558+0000 7f5762ab5640 1 --2- 172.21.15.77:0/1164873502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15ef60 0x7f575c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.558+0000 7f5762ab5640 1 --2- 172.21.15.77:0/1164873502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c0b2950 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.558+0000 7f5762ab5640 1 -- 172.21.15.77:0/1164873502 >> 172.21.15.77:0/1164873502 conn(0x7f575c0cf8a0 msgr2=0x7f575c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.558+0000 7f5762ab5640 1 -- 172.21.15.77:0/1164873502 shutdown_connections 2024-03-06T07:23:29.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.558+0000 7f5762ab5640 1 -- 172.21.15.77:0/1164873502 wait complete. 2024-03-06T07:23:29.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.559+0000 7f5762ab5640 1 Processor -- start 2024-03-06T07:23:29.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.559+0000 7f5762ab5640 1 -- start start 2024-03-06T07:23:29.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5762ab5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c1441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5762ab5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15bf30 0x7f575c1541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5762ab5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c15ef60 0x7f575c1f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5762ab5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0708d0 con 0x7f575c13dc10 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5762ab5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c13dff0 con 0x7f575c15ef60 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5762ab5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c070a50 con 0x7f575c15bf30 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5760851640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c1441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5760851640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c1441a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33156/0 (socket says 172.21.15.77:33156) 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5760851640 1 -- 172.21.15.77:0/609790958 learned_addr learned my addr 172.21.15.77:0/609790958 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f575bfff640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15bf30 0x7f575c1541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5761052640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c15ef60 0x7f575c1f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5760851640 1 -- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15bf30 msgr2=0x7f575c1541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5760851640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15bf30 0x7f575c1541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5760851640 1 -- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c15ef60 msgr2=0x7f575c1f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5760851640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c15ef60 0x7f575c1f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f5760851640 1 -- 172.21.15.77:0/609790958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c141d00 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.560+0000 7f575bfff640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15bf30 0x7f575c1541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.561+0000 7f5760851640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c1441a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f57500097a0 tx=0x7f57500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.561+0000 7f57597fa640 1 -- 172.21.15.77:0/609790958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750032840 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.561+0000 7f57597fa640 1 -- 172.21.15.77:0/609790958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57500329e0 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.561+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c141fe0 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.561+0000 7f57597fa640 1 -- 172.21.15.77:0/609790958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750032dd0 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.561+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c1425a0 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.562+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c04f330 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.562+0000 7f57597fa640 1 -- 172.21.15.77:0/609790958 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f575003b3f0 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.562+0000 7f57597fa640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f574006b500 0x7f574006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.563+0000 7f57597fa640 1 -- 172.21.15.77:0/609790958 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f575002f030 con 0x7f575c13dc10 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.563+0000 7f575bfff640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f574006b500 0x7f574006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.563+0000 7f575bfff640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f574006b500 0x7f574006d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f575c1f3aa0 tx=0x7f5748007c10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.567+0000 7f57597fa640 1 -- 172.21.15.77:0/609790958 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5750040070 con 0x7f575c13dc10 2024-03-06T07:23:29.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.739+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f575c06eb00 con 0x7f575c13dc10 2024-03-06T07:23:29.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.740+0000 7f57597fa640 1 -- 172.21.15.77:0/609790958 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f575007b080 con 0x7f575c13dc10 2024-03-06T07:23:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:29.742 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:29.064179+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:29.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f574006b500 msgr2=0x7f574006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f574006b500 0x7f574006d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f575c1f3aa0 tx=0x7f5748007c10 comp rx=0 tx=0).stop 2024-03-06T07:23:29.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 msgr2=0x7f575c1441a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c1441a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f57500097a0 tx=0x7f57500046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 shutdown_connections 2024-03-06T07:23:29.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f575c15ef60 0x7f575c1f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f575c15bf30 0x7f575c1541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f574006b500 0x7f574006d9c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 --2- 172.21.15.77:0/609790958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f575c13dc10 0x7f575c1441a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 >> 172.21.15.77:0/609790958 conn(0x7f575c0cf8a0 msgr2=0x7f575c0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 shutdown_connections 2024-03-06T07:23:29.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.742+0000 7f5762ab5640 1 -- 172.21.15.77:0/609790958 wait complete. 2024-03-06T07:23:29.755 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:29.756 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.896+0000 7f2fa1250640 1 Processor -- start 2024-03-06T07:23:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.896+0000 7f2fa1250640 1 -- start start 2024-03-06T07:23:29.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.897+0000 7f2fa1250640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f9c0b1e40 0x7f2f9c168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.897+0000 7f2fa1250640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0d2730 con 0x7f2f9c0d4860 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.897+0000 7f2fa1250640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c1618b0 con 0x7f2f9c1614d0 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.897+0000 7f2fa1250640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f9c0d4cf0 con 0x7f2f9c0b1e40 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.897+0000 7f2f9b59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f9c0b1e40 0x7f2f9c168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.897+0000 7f2f9a59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2f9c0d4860 0x7f2f9c0d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56214/0 (socket says 172.21.15.77:56214) 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.897+0000 7f2f9a59c640 1 -- 172.21.15.77:0/1731275318 learned_addr learned my addr 172.21.15.77:0/1731275318 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.898+0000 7f2f9b59e640 1 -- 172.21.15.77:0/1731275318 >> v1:172.21.15.158:6789/0 conn(0x7f2f9c1614d0 legacy=0x7f2f9c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.898+0000 7f2f9b59e640 1 -- 172.21.15.77:0/1731275318 >> v1:172.21.15.77:6789/0 conn(0x7f2f9c0d4860 legacy=0x7f2f9c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.898+0000 7f2f9b59e640 1 -- 172.21.15.77:0/1731275318 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c0cf400 con 0x7f2f9c0b1e40 2024-03-06T07:23:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.898+0000 7f2f9b59e640 1 --2- 172.21.15.77:0/1731275318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0b1e40 0x7f2f9c168b60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0cf5d0 tx=0x7f2f8c02bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772801016a15d167 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.898+0000 7f2f99d9b640 1 -- 172.21.15.77:0/1731275318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c039070 con 0x7f2f9c0b1e40 2024-03-06T07:23:29.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2f99d9b640 1 -- 172.21.15.77:0/1731275318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c02c980 con 0x7f2f9c0b1e40 2024-03-06T07:23:29.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2f99d9b640 1 -- 172.21.15.77:0/1731275318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c02cd50 con 0x7f2f9c0b1e40 2024-03-06T07:23:29.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2fa1250640 1 -- 172.21.15.77:0/1731275318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0b1e40 msgr2=0x7f2f9c168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2fa1250640 1 --2- 172.21.15.77:0/1731275318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0b1e40 0x7f2f9c168b60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0cf5d0 tx=0x7f2f8c02bc50 comp rx=0 tx=0).stop 2024-03-06T07:23:29.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2fa1250640 1 -- 172.21.15.77:0/1731275318 shutdown_connections 2024-03-06T07:23:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2fa1250640 1 --2- 172.21.15.77:0/1731275318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0b1e40 0x7f2f9c168b60 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2fa1250640 1 -- 172.21.15.77:0/1731275318 >> 172.21.15.77:0/1731275318 conn(0x7f2f9c0d3d40 msgr2=0x7f2f9c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2fa1250640 1 -- 172.21.15.77:0/1731275318 shutdown_connections 2024-03-06T07:23:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.899+0000 7f2fa1250640 1 -- 172.21.15.77:0/1731275318 wait complete. 2024-03-06T07:23:29.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.900+0000 7f2fa1250640 1 Processor -- start 2024-03-06T07:23:29.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.900+0000 7f2fa1250640 1 -- start start 2024-03-06T07:23:29.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2fa1250640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f9c0b1e40 0x7f2f9c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2fa1250640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0d4860 0x7f2f9c0b8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2fa1250640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f9c1614d0 0x7f2f9c0b9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2fa1250640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c0cfc60 con 0x7f2f9c0d4860 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2fa1250640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c0cf980 con 0x7f2f9c0b1e40 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2fa1250640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c0b0910 con 0x7f2f9c1614d0 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2f9a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0d4860 0x7f2f9c0b8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2f9a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0d4860 0x7f2f9c0b8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33184/0 (socket says 172.21.15.77:33184) 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.901+0000 7f2f9a59c640 1 -- 172.21.15.77:0/2574580872 learned_addr learned my addr 172.21.15.77:0/2574580872 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f9a59c640 1 -- 172.21.15.77:0/2574580872 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f9c1614d0 msgr2=0x7f2f9c0b9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f9a59c640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f9c1614d0 0x7f2f9c0b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f9a59c640 1 -- 172.21.15.77:0/2574580872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f9c0b1e40 msgr2=0x7f2f9c0cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f9a59c640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f9c0b1e40 0x7f2f9c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f9a59c640 1 -- 172.21.15.77:0/2574580872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c1f4040 con 0x7f2f9c0d4860 2024-03-06T07:23:29.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f9a59c640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0d4860 0x7f2f9c0b8ed0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000c780 tx=0x7f2f9000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f837fe640 1 -- 172.21.15.77:0/2574580872 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f90018070 con 0x7f2f9c0d4860 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f837fe640 1 -- 172.21.15.77:0/2574580872 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f900072d0 con 0x7f2f9c0d4860 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c0c9290 con 0x7f2f9c0d4860 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2f837fe640 1 -- 172.21.15.77:0/2574580872 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f900047e0 con 0x7f2f9c0d4860 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.902+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c0c9ae0 con 0x7f2f9c0d4860 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.903+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9c068ff0 con 0x7f2f9c0d4860 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.903+0000 7f2f837fe640 1 -- 172.21.15.77:0/2574580872 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f9001b070 con 0x7f2f9c0d4860 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.904+0000 7f2f837fe640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f8806b4d0 0x7f2f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.904+0000 7f2f837fe640 1 -- 172.21.15.77:0/2574580872 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2f90089350 con 0x7f2f9c0d4860 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.904+0000 7f2f9ad9d640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f8806b4d0 0x7f2f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.904+0000 7f2f9ad9d640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f8806b4d0 0x7f2f8806d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2f94003cc0 tx=0x7f2f940079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:29.908+0000 7f2f837fe640 1 -- 172.21.15.77:0/2574580872 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f90058520 con 0x7f2f9c0d4860 2024-03-06T07:23:30.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.083+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f9c069e80 con 0x7f2f9c0d4860 2024-03-06T07:23:30.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.083+0000 7f2f837fe640 1 -- 172.21.15.77:0/2574580872 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f90057e20 con 0x7f2f9c0d4860 2024-03-06T07:23:30.085 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:23:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f8806b4d0 msgr2=0x7f2f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f8806b4d0 0x7f2f8806d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2f94003cc0 tx=0x7f2f940079b0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0d4860 msgr2=0x7f2f9c0b8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0d4860 0x7f2f9c0b8ed0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000c780 tx=0x7f2f9000cc40 comp rx=0 tx=0).stop 2024-03-06T07:23:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 shutdown_connections 2024-03-06T07:23:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f8806b4d0 0x7f2f8806d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f9c0d4860 0x7f2f9c0b8ed0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f9c1614d0 0x7f2f9c0b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 --2- 172.21.15.77:0/2574580872 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f9c0b1e40 0x7f2f9c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 >> 172.21.15.77:0/2574580872 conn(0x7f2f9c0d3d40 msgr2=0x7f2f9c15e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 shutdown_connections 2024-03-06T07:23:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.085+0000 7f2fa1250640 1 -- 172.21.15.77:0/2574580872 wait complete. 2024-03-06T07:23:30.100 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:30.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.200+0000 7fa35e39a640 1 Processor -- start 2024-03-06T07:23:30.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.200+0000 7fa35e39a640 1 -- start start 2024-03-06T07:23:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.200+0000 7fa35e39a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa35813dc10 0x7fa3580b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.200+0000 7fa35e39a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35815f580 con 0x7fa35815bf30 2024-03-06T07:23:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.200+0000 7fa35e39a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35815b450 con 0x7fa35815ef60 2024-03-06T07:23:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa35e39a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580cf3f0 con 0x7fa35813dc10 2024-03-06T07:23:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa3577fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa35815bf30 0x7fa35815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56216/0 (socket says 172.21.15.77:56216) 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa3577fe640 1 -- 172.21.15.77:0/226161494 learned_addr learned my addr 172.21.15.77:0/226161494 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa357fff640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa35813dc10 0x7fa3580b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364219487 0 0) 0x7fa35815f580 con 0x7fa35815bf30 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa340003520 con 0x7fa35815bf30 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3014258974 0 0) 0x7fa340003520 con 0x7fa35815bf30 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa35813dc10 msgr2=0x7fa3580b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa35813dc10 0x7fa3580b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 >> v1:172.21.15.158:6789/0 conn(0x7fa35815ef60 legacy=0x7fa3580708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35815b730 con 0x7fa35815bf30 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fa34c001670 con 0x7fa35815bf30 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 >> v1:172.21.15.77:6789/0 conn(0x7fa35815bf30 legacy=0x7fa35815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3400039e0 0x7fa340003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa35813dc10 0x7fa340004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa340004d40 0x7fa3400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580cf3f0 con 0x7fa3400039e0 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34000ba70 con 0x7fa340004d40 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34000bc70 con 0x7fa35813dc10 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa357fff640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3400039e0 0x7fa340003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.201+0000 7fa3577fe640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa35813dc10 0x7fa340004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa35c937640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa340004d40 0x7fa3400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa357fff640 1 -- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa35813dc10 msgr2=0x7fa340004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa357fff640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa35813dc10 0x7fa340004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa357fff640 1 -- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa340004d40 msgr2=0x7fa3400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa357fff640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa340004d40 0x7fa3400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa357fff640 1 -- 172.21.15.77:0/226161494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa340003520 con 0x7fa3400039e0 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa357fff640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3400039e0 0x7fa340003dc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa348009ac0 tx=0x7fa340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f301f59e3f5e8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348039070 con 0x7fa3400039e0 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.202+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34802c6b0 con 0x7fa3400039e0 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.203+0000 7fa356ffd640 1 -- 172.21.15.77:0/226161494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348009690 con 0x7fa3400039e0 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.203+0000 7fa35e39a640 1 -- 172.21.15.77:0/226161494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3400039e0 msgr2=0x7fa340003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.203+0000 7fa35e39a640 1 --2- 172.21.15.77:0/226161494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3400039e0 0x7fa340003dc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa348009ac0 tx=0x7fa340005830 comp rx=0 tx=0).stop 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.203+0000 7fa35c937640 1 -- 172.21.15.77:0/226161494 reap_dead start 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.203+0000 7fa35e39a640 1 -- 172.21.15.77:0/226161494 shutdown_connections 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.203+0000 7fa35e39a640 1 -- 172.21.15.77:0/226161494 >> 172.21.15.77:0/226161494 conn(0x7fa3580cf8a0 msgr2=0x7fa3580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.203+0000 7fa35e39a640 1 -- 172.21.15.77:0/226161494 shutdown_connections 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.203+0000 7fa35e39a640 1 -- 172.21.15.77:0/226161494 wait complete. 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.204+0000 7fa35e39a640 1 Processor -- start 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.204+0000 7fa35e39a640 1 -- start start 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa35e39a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3400039e0 0x7fa3581f5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa35e39a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa358154380 0x7fa358157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa35e39a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa358154800 0x7fa3580cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa35e39a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa358071ff0 con 0x7fa358154380 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa35e39a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35815b210 con 0x7fa3400039e0 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa35e39a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa358072170 con 0x7fa358154800 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa3577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa358154380 0x7fa358157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa3577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa358154380 0x7fa358157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33196/0 (socket says 172.21.15.77:33196) 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa3577fe640 1 -- 172.21.15.77:0/3788061223 learned_addr learned my addr 172.21.15.77:0/3788061223 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa357fff640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3400039e0 0x7fa3581f5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa35c937640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa358154800 0x7fa3580cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa3577fe640 1 -- 172.21.15.77:0/3788061223 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa358154800 msgr2=0x7fa3580cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa3577fe640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa358154800 0x7fa3580cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa3577fe640 1 -- 172.21.15.77:0/3788061223 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3400039e0 msgr2=0x7fa3581f5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa3577fe640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3400039e0 0x7fa3581f5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.205+0000 7fa3577fe640 1 -- 172.21.15.77:0/3788061223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa358157c70 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.206+0000 7fa3577fe640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa358154380 0x7fa358157220 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00e700 tx=0x7fa34c00ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.206+0000 7fa354ff9640 1 -- 172.21.15.77:0/3788061223 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c00ee10 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.206+0000 7fa354ff9640 1 -- 172.21.15.77:0/3788061223 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c018070 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.206+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa358157f50 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.206+0000 7fa354ff9640 1 -- 172.21.15.77:0/3788061223 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c00cb10 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.206+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3581f5510 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.207+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa358158120 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.207+0000 7fa354ff9640 1 -- 172.21.15.77:0/3788061223 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa34c00ccb0 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.208+0000 7fa354ff9640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34406b4d0 0x7fa34406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.208+0000 7fa354ff9640 1 -- 172.21.15.77:0/3788061223 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa34c089c30 con 0x7fa358154380 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.208+0000 7fa357fff640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34406b4d0 0x7fa34406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.208+0000 7fa357fff640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34406b4d0 0x7fa34406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa348036000 tx=0x7fa348030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.212+0000 7fa354ff9640 1 -- 172.21.15.77:0/3788061223 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa34c058e00 con 0x7fa358154380 2024-03-06T07:23:30.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.385+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa35804f330 con 0x7fa358154380 2024-03-06T07:23:30.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.386+0000 7fa354ff9640 1 -- 172.21.15.77:0/3788061223 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa34c058700 con 0x7fa358154380 2024-03-06T07:23:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:30.388 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:23:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:23:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:23:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:23:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:30.388 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:23:30.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34406b4d0 msgr2=0x7fa34406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34406b4d0 0x7fa34406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa348036000 tx=0x7fa348030000 comp rx=0 tx=0).stop 2024-03-06T07:23:30.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa358154380 msgr2=0x7fa358157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa358154380 0x7fa358157220 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00e700 tx=0x7fa34c00ebc0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 shutdown_connections 2024-03-06T07:23:30.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34406b4d0 0x7fa34406d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa358154380 0x7fa358157220 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa358154800 0x7fa3580cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 --2- 172.21.15.77:0/3788061223 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3400039e0 0x7fa3581f5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 >> 172.21.15.77:0/3788061223 conn(0x7fa3580cf8a0 msgr2=0x7fa3580b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 shutdown_connections 2024-03-06T07:23:30.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.388+0000 7fa35e39a640 1 -- 172.21.15.77:0/3788061223 wait complete. 2024-03-06T07:23:30.400 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:30.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:30.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.502+0000 7f0898eed640 1 Processor -- start 2024-03-06T07:23:30.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.502+0000 7f0898eed640 1 -- start start 2024-03-06T07:23:30.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f0898eed640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f08940708b0 0x7f0894070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f0898eed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940ce0a0 0x7f08940ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f0898eed640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089415afe0 con 0x7f089415ac00 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f0898eed640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0894071250 con 0x7f08940708b0 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f0898eed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08940cdc10 con 0x7f08940ce0a0 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f089259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f089415ac00 0x7f089406c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56222/0 (socket says 172.21.15.77:56222) 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f089259c640 1 -- 172.21.15.77:0/3443046819 learned_addr learned my addr 172.21.15.77:0/3443046819 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f0891d9b640 1 --2- 172.21.15.77:0/3443046819 >> v2:172.21.15.158:3300/0 conn(0x7f08940708b0 0x7f0894070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f0892d9d640 1 --2- 172.21.15.77:0/3443046819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940ce0a0 0x7f08940ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f089159a640 1 -- 172.21.15.77:0/3443046819 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198038098 0 0) 0x7f089415afe0 con 0x7f089415ac00 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.503+0000 7f089159a640 1 -- 172.21.15.77:0/3443046819 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087c003520 con 0x7f089415ac00 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.504+0000 7f089159a640 1 -- 172.21.15.77:0/3443046819 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1341660254 0 0) 0x7f087c003520 con 0x7f089415ac00 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.504+0000 7f0891d9b640 1 -- 172.21.15.77:0/3443046819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940ce0a0 msgr2=0x7f08940ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.504+0000 7f0891d9b640 1 --2- 172.21.15.77:0/3443046819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940ce0a0 0x7f08940ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.504+0000 7f0891d9b640 1 -- 172.21.15.77:0/3443046819 >> v1:172.21.15.77:6789/0 conn(0x7f089415ac00 legacy=0x7f089406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.504+0000 7f0891d9b640 1 -- 172.21.15.77:0/3443046819 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08940cd000 con 0x7f08940708b0 2024-03-06T07:23:30.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0891d9b640 1 --2- 172.21.15.77:0/3443046819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940708b0 0x7f0894070d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08880097a0 tx=0x7f088802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c86003660d818b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f089159a640 1 -- 172.21.15.77:0/3443046819 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888037070 con 0x7f08940708b0 2024-03-06T07:23:30.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f089159a640 1 -- 172.21.15.77:0/3443046819 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0888030510 con 0x7f08940708b0 2024-03-06T07:23:30.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f089159a640 1 -- 172.21.15.77:0/3443046819 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08840019f0 con 0x7f08940708b0 2024-03-06T07:23:30.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0898eed640 1 -- 172.21.15.77:0/3443046819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940708b0 msgr2=0x7f0894070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0898eed640 1 --2- 172.21.15.77:0/3443046819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940708b0 0x7f0894070d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08880097a0 tx=0x7f088802b710 comp rx=0 tx=0).stop 2024-03-06T07:23:30.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0898eed640 1 -- 172.21.15.77:0/3443046819 shutdown_connections 2024-03-06T07:23:30.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0898eed640 1 --2- 172.21.15.77:0/3443046819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940ce0a0 0x7f08940ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0898eed640 1 --2- 172.21.15.77:0/3443046819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940708b0 0x7f0894070d10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0898eed640 1 -- 172.21.15.77:0/3443046819 >> 172.21.15.77:0/3443046819 conn(0x7f08940b1650 msgr2=0x7f08940b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0898eed640 1 -- 172.21.15.77:0/3443046819 shutdown_connections 2024-03-06T07:23:30.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.505+0000 7f0898eed640 1 -- 172.21.15.77:0/3443046819 wait complete. 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.507+0000 7f0898eed640 1 Processor -- start 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.507+0000 7f0898eed640 1 -- start start 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.507+0000 7f0898eed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940708b0 0x7f089414f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.507+0000 7f0898eed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940ce0a0 0x7f08941505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.507+0000 7f0898eed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089415ac00 0x7f089412f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.507+0000 7f0898eed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08940cd8a0 con 0x7f089415ac00 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.507+0000 7f0898eed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08940cd5c0 con 0x7f08940ce0a0 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.507+0000 7f0898eed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08940cf5d0 con 0x7f08940708b0 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089415ac00 0x7f089412f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089415ac00 0x7f089412f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33212/0 (socket says 172.21.15.77:33212) 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 -- 172.21.15.77:0/102818224 learned_addr learned my addr 172.21.15.77:0/102818224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f089259c640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940708b0 0x7f089414f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0891d9b640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940ce0a0 0x7f08941505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 -- 172.21.15.77:0/102818224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940708b0 msgr2=0x7f089414f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940708b0 0x7f089414f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 -- 172.21.15.77:0/102818224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940ce0a0 msgr2=0x7f08941505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940ce0a0 0x7f08941505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 -- 172.21.15.77:0/102818224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894144f30 con 0x7f089415ac00 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0892d9d640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089415ac00 0x7f089412f010 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f088c00e930 tx=0x7f088c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0882ffd640 1 -- 172.21.15.77:0/102818224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c017070 con 0x7f089415ac00 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0882ffd640 1 -- 172.21.15.77:0/102818224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c00c980 con 0x7f089415ac00 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0882ffd640 1 -- 172.21.15.77:0/102818224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c00cd70 con 0x7f089415ac00 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0894145210 con 0x7f089415ac00 2024-03-06T07:23:30.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.508+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089414fb10 con 0x7f089415ac00 2024-03-06T07:23:30.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.509+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089404f330 con 0x7f089415ac00 2024-03-06T07:23:30.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.510+0000 7f0882ffd640 1 -- 172.21.15.77:0/102818224 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f088c01a070 con 0x7f089415ac00 2024-03-06T07:23:30.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.510+0000 7f0882ffd640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f087406b4d0 0x7f087406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.510+0000 7f0882ffd640 1 -- 172.21.15.77:0/102818224 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f088c08a130 con 0x7f089415ac00 2024-03-06T07:23:30.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.510+0000 7f089259c640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f087406b4d0 0x7f087406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.510+0000 7f089259c640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f087406b4d0 0x7f087406d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0884002230 tx=0x7f0884009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.514+0000 7f0882ffd640 1 -- 172.21.15.77:0/102818224 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f088c058e00 con 0x7f089415ac00 2024-03-06T07:23:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:30.684+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f089414def0 con 0x7f089415ac00 2024-03-06T07:23:31.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.072+0000 7f0882ffd640 1 -- 172.21.15.77:0/102818224 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) v1 ==== 138+0+0 (secure 0 0 0) 0x7f088c05cea0 con 0x7f089415ac00 2024-03-06T07:23:31.074 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f087406b4d0 msgr2=0x7f087406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f087406b4d0 0x7f087406d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0884002230 tx=0x7f0884009040 comp rx=0 tx=0).stop 2024-03-06T07:23:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089415ac00 msgr2=0x7f089412f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089415ac00 0x7f089412f010 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f088c00e930 tx=0x7f088c00edf0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 shutdown_connections 2024-03-06T07:23:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f087406b4d0 0x7f087406d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f089415ac00 0x7f089412f010 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08940ce0a0 0x7f08941505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 --2- 172.21.15.77:0/102818224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08940708b0 0x7f089414f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 >> 172.21.15.77:0/102818224 conn(0x7f08940b1650 msgr2=0x7f08940b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 shutdown_connections 2024-03-06T07:23:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.075+0000 7f0898eed640 1 -- 172.21.15.77:0/102818224 wait complete. 2024-03-06T07:23:31.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:31.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.197+0000 7fbe8a6b5640 1 Processor -- start 2024-03-06T07:23:31.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.197+0000 7fbe8a6b5640 1 -- start start 2024-03-06T07:23:31.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe8a6b5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbe841586e0 0x7fbe840aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe8a6b5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbe840aef80 0x7fbe840cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe8a6b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 0x7fbe8406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe8a6b5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe84158ac0 con 0x7fbe841586e0 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe8a6b5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8415b2a0 con 0x7fbe840aef80 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe8a6b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8415b540 con 0x7fbe840cffd0 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe83fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbe841586e0 0x7fbe840aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe83fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbe841586e0 0x7fbe840aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33224/0 (socket says 172.21.15.77:33224) 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe83fff640 1 -- 172.21.15.77:0/1719013666 learned_addr learned my addr 172.21.15.77:0/1719013666 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe83fff640 1 -- 172.21.15.77:0/1719013666 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 msgr2=0x7fbe8406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe83fff640 1 --2- 172.21.15.77:0/1719013666 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 0x7fbe8406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe83fff640 1 -- 172.21.15.77:0/1719013666 >> v2:172.21.15.158:3300/0 conn(0x7fbe840aef80 msgr2=0x7fbe840cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe83fff640 1 --2- 172.21.15.77:0/1719013666 >> v2:172.21.15.158:3300/0 conn(0x7fbe840aef80 0x7fbe840cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.198+0000 7fbe83fff640 1 -- 172.21.15.77:0/1719013666 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8415b820 con 0x7fbe841586e0 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.199+0000 7fbe83fff640 1 --2- 172.21.15.77:0/1719013666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe841586e0 0x7fbe840aea40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbe84072e30 tx=0x7fbe7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf25af12299a6b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.199+0000 7fbe82ffd640 1 -- 172.21.15.77:0/1719013666 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74035020 con 0x7fbe841586e0 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.199+0000 7fbe82ffd640 1 -- 172.21.15.77:0/1719013666 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe74031870 con 0x7fbe841586e0 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.199+0000 7fbe82ffd640 1 -- 172.21.15.77:0/1719013666 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74031c40 con 0x7fbe841586e0 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1719013666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe841586e0 msgr2=0x7fbe840aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1719013666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe841586e0 0x7fbe840aea40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbe84072e30 tx=0x7fbe7402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1719013666 shutdown_connections 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1719013666 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 0x7fbe8406da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1719013666 >> v2:172.21.15.158:3300/0 conn(0x7fbe840aef80 0x7fbe840cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1719013666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe841586e0 0x7fbe840aea40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1719013666 >> 172.21.15.77:0/1719013666 conn(0x7fbe8406b190 msgr2=0x7fbe8406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1719013666 shutdown_connections 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.200+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1719013666 wait complete. 2024-03-06T07:23:31.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.201+0000 7fbe8a6b5640 1 Processor -- start 2024-03-06T07:23:31.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.201+0000 7fbe8a6b5640 1 -- start start 2024-03-06T07:23:31.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe8a6b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe840aef80 0x7fbe84148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe8a6b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 0x7fbe841455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe8a6b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe841586e0 0x7fbe8406c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe8a6b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe84071350 con 0x7fbe840aef80 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe8a6b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8415bde0 con 0x7fbe841586e0 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe8a6b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840714d0 con 0x7fbe840cffd0 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe840aef80 0x7fbe84148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe840aef80 0x7fbe84148800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33228/0 (socket says 172.21.15.77:33228) 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 -- 172.21.15.77:0/1969407445 learned_addr learned my addr 172.21.15.77:0/1969407445 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe837fe640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 0x7fbe841455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 -- 172.21.15.77:0/1969407445 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 msgr2=0x7fbe841455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 0x7fbe841455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 -- 172.21.15.77:0/1969407445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe841586e0 msgr2=0x7fbe8406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe841586e0 0x7fbe8406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 -- 172.21.15.77:0/1969407445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe840a9450 con 0x7fbe840aef80 2024-03-06T07:23:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe83fff640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe840aef80 0x7fbe84148800 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbe74000c00 tx=0x7fbe740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.202+0000 7fbe80ff9640 1 -- 172.21.15.77:0/1969407445 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe740443f0 con 0x7fbe840aef80 2024-03-06T07:23:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.203+0000 7fbe80ff9640 1 -- 172.21.15.77:0/1969407445 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe74044590 con 0x7fbe840aef80 2024-03-06T07:23:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.203+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe840a99d0 con 0x7fbe840aef80 2024-03-06T07:23:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.203+0000 7fbe80ff9640 1 -- 172.21.15.77:0/1969407445 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74044960 con 0x7fbe840aef80 2024-03-06T07:23:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.203+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe840a9f90 con 0x7fbe840aef80 2024-03-06T07:23:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.204+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe840732b0 con 0x7fbe840aef80 2024-03-06T07:23:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.204+0000 7fbe80ff9640 1 -- 172.21.15.77:0/1969407445 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe7404b060 con 0x7fbe840aef80 2024-03-06T07:23:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.204+0000 7fbe80ff9640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbe7006b4d0 0x7fbe7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.205+0000 7fbe80ff9640 1 -- 172.21.15.77:0/1969407445 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fbe740ac190 con 0x7fbe840aef80 2024-03-06T07:23:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.205+0000 7fbe837fe640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbe7006b4d0 0x7fbe7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.205+0000 7fbe837fe640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbe7006b4d0 0x7fbe7006d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe8406bb80 tx=0x7fbe78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.209+0000 7fbe80ff9640 1 -- 172.21.15.77:0/1969407445 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe7403f070 con 0x7fbe840aef80 2024-03-06T07:23:31.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:31.381+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbe840aa160 con 0x7fbe840aef80 2024-03-06T07:23:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.077+0000 7fbe80ff9640 1 -- 172.21.15.77:0/1969407445 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbe7402f320 con 0x7fbe840aef80 2024-03-06T07:23:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.080+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbe7006b4d0 msgr2=0x7fbe7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbe7006b4d0 0x7fbe7006d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe8406bb80 tx=0x7fbe78008040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe840aef80 msgr2=0x7fbe84148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe840aef80 0x7fbe84148800 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbe74000c00 tx=0x7fbe740046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 shutdown_connections 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbe841586e0 0x7fbe8406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbe840cffd0 0x7fbe841455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbe7006b4d0 0x7fbe7006d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 --2- 172.21.15.77:0/1969407445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe840aef80 0x7fbe84148800 secure :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbe74000c00 tx=0x7fbe740046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.081+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 >> 172.21.15.77:0/1969407445 conn(0x7fbe8406b190 msgr2=0x7fbe84072210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.082+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 shutdown_connections 2024-03-06T07:23:32.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.082+0000 7fbe8a6b5640 1 -- 172.21.15.77:0/1969407445 wait complete. 2024-03-06T07:23:32.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.207+0000 7fd65d6d4640 1 Processor -- start 2024-03-06T07:23:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.207+0000 7fd65d6d4640 1 -- start start 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.207+0000 7fd65d6d4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd658161460 0x7fd6580b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.207+0000 7fd65d6d4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6580d0d40 0x7fd6580b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.207+0000 7fd65d6d4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6580cf540 con 0x7fd658161840 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.207+0000 7fd65d6d4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6580cf2f0 con 0x7fd658161460 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.207+0000 7fd65d6d4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6580b6b20 con 0x7fd6580d0d40 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd656ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd658161460 0x7fd6580b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd656ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd658161460 0x7fd6580b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33234/0 (socket says 172.21.15.77:33234) 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd656ffd640 1 -- 172.21.15.77:0/3150424546 learned_addr learned my addr 172.21.15.77:0/3150424546 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd6577fe640 1 --2- 172.21.15.77:0/3150424546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6580d0d40 0x7fd6580b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd656ffd640 1 -- 172.21.15.77:0/3150424546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6580d0d40 msgr2=0x7fd6580b6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd656ffd640 1 --2- 172.21.15.77:0/3150424546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6580d0d40 0x7fd6580b6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd656ffd640 1 -- 172.21.15.77:0/3150424546 >> v1:172.21.15.158:6789/0 conn(0x7fd658161840 legacy=0x7fd6580d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd656ffd640 1 -- 172.21.15.77:0/3150424546 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6580cf820 con 0x7fd658161460 2024-03-06T07:23:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.208+0000 7fd6577fe640 1 --2- 172.21.15.77:0/3150424546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6580d0d40 0x7fd6580b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:32.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd656ffd640 1 --2- 172.21.15.77:0/3150424546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580b4c00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd648009870 tx=0x7fd64802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b752dd0a665b0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd655ffb640 1 -- 172.21.15.77:0/3150424546 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648035020 con 0x7fd658161460 2024-03-06T07:23:32.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd655ffb640 1 -- 172.21.15.77:0/3150424546 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd648031640 con 0x7fd658161460 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd655ffb640 1 -- 172.21.15.77:0/3150424546 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64802d040 con 0x7fd658161460 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd65d6d4640 1 -- 172.21.15.77:0/3150424546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 msgr2=0x7fd6580b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/3150424546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580b4c00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd648009870 tx=0x7fd64802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd65d6d4640 1 -- 172.21.15.77:0/3150424546 shutdown_connections 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/3150424546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6580d0d40 0x7fd6580b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/3150424546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580b4c00 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd65d6d4640 1 -- 172.21.15.77:0/3150424546 >> 172.21.15.77:0/3150424546 conn(0x7fd6580d3cd0 msgr2=0x7fd6580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd65d6d4640 1 -- 172.21.15.77:0/3150424546 shutdown_connections 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.209+0000 7fd65d6d4640 1 -- 172.21.15.77:0/3150424546 wait complete. 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd65d6d4640 1 Processor -- start 2024-03-06T07:23:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd65d6d4640 1 -- start start 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd65d6d4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6580d0d40 0x7fd65815b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd65d6d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580d40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd65d6d4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd658161840 0x7fd6580d45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd65d6d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6580d12d0 con 0x7fd658161460 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd65d6d4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6580b5140 con 0x7fd6580d0d40 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd65d6d4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6580d1450 con 0x7fd658161840 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.211+0000 7fd6567fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580d40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6567fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580d40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33236/0 (socket says 172.21.15.77:33236) 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6567fc640 1 -- 172.21.15.77:0/2333517375 learned_addr learned my addr 172.21.15.77:0/2333517375 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:32.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd656ffd640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6580d0d40 0x7fd65815b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6577fe640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd658161840 0x7fd6580d45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6567fc640 1 -- 172.21.15.77:0/2333517375 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd658161840 msgr2=0x7fd6580d45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6567fc640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd658161840 0x7fd6580d45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6567fc640 1 -- 172.21.15.77:0/2333517375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6580d0d40 msgr2=0x7fd65815b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6567fc640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6580d0d40 0x7fd65815b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6567fc640 1 -- 172.21.15.77:0/2333517375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6581591c0 con 0x7fd658161460 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd6567fc640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580d40b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00c470 tx=0x7fd64c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd63f7fe640 1 -- 172.21.15.77:0/2333517375 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c0052d0 con 0x7fd658161460 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd63f7fe640 1 -- 172.21.15.77:0/2333517375 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c005470 con 0x7fd658161460 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd63f7fe640 1 -- 172.21.15.77:0/2333517375 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c007260 con 0x7fd658161460 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65815ab10 con 0x7fd658161460 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.212+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65815d1b0 con 0x7fd658161460 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.213+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65804f330 con 0x7fd658161460 2024-03-06T07:23:32.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.214+0000 7fd63f7fe640 1 -- 172.21.15.77:0/2333517375 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd64c007400 con 0x7fd658161460 2024-03-06T07:23:32.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.214+0000 7fd63f7fe640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd64406b4d0 0x7fd64406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.214+0000 7fd63f7fe640 1 -- 172.21.15.77:0/2333517375 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd64c089570 con 0x7fd658161460 2024-03-06T07:23:32.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.214+0000 7fd656ffd640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd64406b4d0 0x7fd64406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.214+0000 7fd656ffd640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd64406b4d0 0x7fd64406d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd648036000 tx=0x7fd64803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.219+0000 7fd63f7fe640 1 -- 172.21.15.77:0/2333517375 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd64c057e90 con 0x7fd658161460 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:32.427+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd6580b4940 con 0x7fd658161460 2024-03-06T07:23:33.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.078+0000 7fd63f7fe640 1 -- 172.21.15.77:0/2333517375 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v17) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd64c05bf30 con 0x7fd658161460 2024-03-06T07:23:33.081 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:23:33.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.083+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd64406b4d0 msgr2=0x7fd64406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.083+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd64406b4d0 0x7fd64406d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd648036000 tx=0x7fd64803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.083+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 msgr2=0x7fd6580d40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.083+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580d40b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00c470 tx=0x7fd64c001e70 comp rx=0 tx=0).stop 2024-03-06T07:23:33.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.085+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 shutdown_connections 2024-03-06T07:23:33.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.085+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd64406b4d0 0x7fd64406d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.085+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd658161460 0x7fd6580d40b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.085+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd658161840 0x7fd6580d45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.085+0000 7fd65d6d4640 1 --2- 172.21.15.77:0/2333517375 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6580d0d40 0x7fd65815b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.085+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 >> 172.21.15.77:0/2333517375 conn(0x7fd6580d3cd0 msgr2=0x7fd6580b7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.092+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 shutdown_connections 2024-03-06T07:23:33.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.097+0000 7fd65d6d4640 1 -- 172.21.15.77:0/2333517375 wait complete. 2024-03-06T07:23:33.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:33.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.219+0000 7f5fda439640 1 Processor -- start 2024-03-06T07:23:33.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.219+0000 7f5fda439640 1 -- start start 2024-03-06T07:23:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fda439640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fda439640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd415f580 con 0x7f5fd415ef60 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fda439640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd415b450 con 0x7f5fd415bf30 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fda439640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd40cf3f0 con 0x7f5fd413dc10 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd89d6640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5fd415ef60 0x7f5fd40708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56224/0 (socket says 172.21.15.77:56224) 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd89d6640 1 -- 172.21.15.77:0/2383994582 learned_addr learned my addr 172.21.15.77:0/2383994582 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd3fff640 1 --2- 172.21.15.77:0/2383994582 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd2ffd640 1 -- 172.21.15.77:0/2383994582 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925709791 0 0) 0x7f5fd415f580 con 0x7f5fd415ef60 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd2ffd640 1 -- 172.21.15.77:0/2383994582 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fbc003520 con 0x7f5fd415ef60 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd2ffd640 1 -- 172.21.15.77:0/2383994582 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637280714 0 0) 0x7f5fd415b450 con 0x7f5fd415bf30 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd2ffd640 1 -- 172.21.15.77:0/2383994582 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd415f580 con 0x7f5fd415bf30 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd3fff640 1 -- 172.21.15.77:0/2383994582 >> v1:172.21.15.158:6789/0 conn(0x7f5fd415bf30 legacy=0x7f5fd415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd2ffd640 1 -- 172.21.15.77:0/2383994582 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902671408 0 0) 0x7f5fbc003520 con 0x7f5fd415ef60 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd3fff640 1 -- 172.21.15.77:0/2383994582 >> v1:172.21.15.77:6789/0 conn(0x7f5fd415ef60 legacy=0x7f5fd40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.220+0000 7f5fd3fff640 1 -- 172.21.15.77:0/2383994582 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd415b730 con 0x7f5fd413dc10 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.221+0000 7f5fd3fff640 1 --2- 172.21.15.77:0/2383994582 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd40b2950 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40097d0 tx=0x7f5fc402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba5f5e7a2ceccb46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.221+0000 7f5fd2ffd640 1 -- 172.21.15.77:0/2383994582 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4037070 con 0x7f5fd413dc10 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.221+0000 7f5fd2ffd640 1 -- 172.21.15.77:0/2383994582 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4030510 con 0x7f5fd413dc10 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.221+0000 7f5fd2ffd640 1 -- 172.21.15.77:0/2383994582 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0019f0 con 0x7f5fd413dc10 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.222+0000 7f5fda439640 1 -- 172.21.15.77:0/2383994582 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 msgr2=0x7f5fd40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.222+0000 7f5fda439640 1 --2- 172.21.15.77:0/2383994582 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd40b2950 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40097d0 tx=0x7f5fc402b770 comp rx=0 tx=0).stop 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.222+0000 7f5fda439640 1 -- 172.21.15.77:0/2383994582 shutdown_connections 2024-03-06T07:23:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.222+0000 7f5fda439640 1 --2- 172.21.15.77:0/2383994582 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd40b2950 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.222+0000 7f5fda439640 1 -- 172.21.15.77:0/2383994582 >> 172.21.15.77:0/2383994582 conn(0x7f5fd40cf8a0 msgr2=0x7f5fd40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.222+0000 7f5fda439640 1 -- 172.21.15.77:0/2383994582 shutdown_connections 2024-03-06T07:23:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.222+0000 7f5fda439640 1 -- 172.21.15.77:0/2383994582 wait complete. 2024-03-06T07:23:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.223+0000 7f5fda439640 1 Processor -- start 2024-03-06T07:23:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.223+0000 7f5fda439640 1 -- start start 2024-03-06T07:23:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fda439640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd4141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fda439640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fd415bf30 0x7f5fd413ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fda439640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd415ef60 0x7f5fd4159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fda439640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd40b2e90 con 0x7f5fd415ef60 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fda439640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd415b210 con 0x7f5fd415bf30 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fda439640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd40b3010 con 0x7f5fd413dc10 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd89d6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd415ef60 0x7f5fd4159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd89d6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd415ef60 0x7f5fd4159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33238/0 (socket says 172.21.15.77:33238) 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd89d6640 1 -- 172.21.15.77:0/3623385468 learned_addr learned my addr 172.21.15.77:0/3623385468 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd3fff640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd4141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd37fe640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fd415bf30 0x7f5fd413ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd89d6640 1 -- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 msgr2=0x7f5fd4141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd89d6640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd4141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd89d6640 1 -- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fd415bf30 msgr2=0x7f5fd413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd89d6640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fd415bf30 0x7f5fd413ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd89d6640 1 -- 172.21.15.77:0/3623385468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd413b740 con 0x7f5fd415ef60 2024-03-06T07:23:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.224+0000 7f5fd37fe640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fd415bf30 0x7f5fd413ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.225+0000 7f5fd89d6640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd415ef60 0x7f5fd4159d30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00cc30 tx=0x7f5fcc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.225+0000 7f5fd0ff9640 1 -- 172.21.15.77:0/3623385468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc017070 con 0x7f5fd415ef60 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.225+0000 7f5fd0ff9640 1 -- 172.21.15.77:0/3623385468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc00ce40 con 0x7f5fd415ef60 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.225+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd4142120 con 0x7f5fd415ef60 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.225+0000 7f5fd0ff9640 1 -- 172.21.15.77:0/3623385468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc012620 con 0x7f5fd415ef60 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.225+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd41426e0 con 0x7f5fd415ef60 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.226+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd404f330 con 0x7f5fd415ef60 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.226+0000 7f5fd0ff9640 1 -- 172.21.15.77:0/3623385468 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fcc01a070 con 0x7f5fd415ef60 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.227+0000 7f5fd0ff9640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5fc006b4d0 0x7f5fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.227+0000 7f5fd0ff9640 1 -- 172.21.15.77:0/3623385468 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fcc08a0b0 con 0x7f5fd415ef60 2024-03-06T07:23:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.227+0000 7f5fd3fff640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5fc006b4d0 0x7f5fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.227+0000 7f5fd3fff640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5fc006b4d0 0x7f5fc006d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4034000 tx=0x7f5fc4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.231+0000 7f5fd0ff9640 1 -- 172.21.15.77:0/3623385468 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fcc0588e0 con 0x7f5fd415ef60 2024-03-06T07:23:33.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.405+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fd40b2c10 con 0x7f5fd415ef60 2024-03-06T07:23:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.406+0000 7f5fd0ff9640 1 -- 172.21.15.77:0/3623385468 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16614 (secure 0 0 0) 0x7f5fcc05c980 con 0x7f5fd415ef60 2024-03-06T07:23:33.408 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:33.408 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":44,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:32.429067+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:30.703498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:31.400205+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:33.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5fc006b4d0 msgr2=0x7f5fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5fc006b4d0 0x7f5fc006d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4034000 tx=0x7f5fc4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:33.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd415ef60 msgr2=0x7f5fd4159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd415ef60 0x7f5fd4159d30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00cc30 tx=0x7f5fcc00a430 comp rx=0 tx=0).stop 2024-03-06T07:23:33.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 shutdown_connections 2024-03-06T07:23:33.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5fc006b4d0 0x7f5fc006d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd415ef60 0x7f5fd4159d30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5fd415bf30 0x7f5fd413ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 --2- 172.21.15.77:0/3623385468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fd413dc10 0x7f5fd4141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 >> 172.21.15.77:0/3623385468 conn(0x7f5fd40cf8a0 msgr2=0x7f5fd415e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 shutdown_connections 2024-03-06T07:23:33.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.408+0000 7f5fda439640 1 -- 172.21.15.77:0/3623385468 wait complete. 2024-03-06T07:23:33.421 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:33.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.518+0000 7fc58350f640 1 Processor -- start 2024-03-06T07:23:33.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.519+0000 7fc58350f640 1 -- start start 2024-03-06T07:23:33.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.519+0000 7fc58350f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc57c13dc10 0x7fc57c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.519+0000 7fc58350f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c15bf30 0x7fc57c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.519+0000 7fc58350f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc57c15e850 0x7fc57c15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.519+0000 7fc58350f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57c0cf3f0 con 0x7fc57c15e850 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.519+0000 7fc58350f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57c15b210 con 0x7fc57c13dc10 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.519+0000 7fc58350f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c15f580 con 0x7fc57c15bf30 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc581aac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc57c15e850 0x7fc57c15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc581aac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc57c15e850 0x7fc57c15eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33244/0 (socket says 172.21.15.77:33244) 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc581aac640 1 -- 172.21.15.77:0/516641764 learned_addr learned my addr 172.21.15.77:0/516641764 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc5812ab640 1 --2- 172.21.15.77:0/516641764 >> v2:172.21.15.158:3300/0 conn(0x7fc57c13dc10 0x7fc57c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc580aaa640 1 --2- 172.21.15.77:0/516641764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c15bf30 0x7fc57c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc581aac640 1 -- 172.21.15.77:0/516641764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c15bf30 msgr2=0x7fc57c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc581aac640 1 --2- 172.21.15.77:0/516641764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c15bf30 0x7fc57c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc581aac640 1 -- 172.21.15.77:0/516641764 >> v2:172.21.15.158:3300/0 conn(0x7fc57c13dc10 msgr2=0x7fc57c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc581aac640 1 --2- 172.21.15.77:0/516641764 >> v2:172.21.15.158:3300/0 conn(0x7fc57c13dc10 0x7fc57c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc581aac640 1 -- 172.21.15.77:0/516641764 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c15b450 con 0x7fc57c15e850 2024-03-06T07:23:33.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.520+0000 7fc580aaa640 1 --2- 172.21.15.77:0/516641764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c15bf30 0x7fc57c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:33.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc581aac640 1 --2- 172.21.15.77:0/516641764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15e850 0x7fc57c15eca0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0731a0 tx=0x7fc57402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795230b420b0365e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc56ffff640 1 -- 172.21.15.77:0/516641764 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57402c9e0 con 0x7fc57c15e850 2024-03-06T07:23:33.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc56ffff640 1 -- 172.21.15.77:0/516641764 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57402cb80 con 0x7fc57c15e850 2024-03-06T07:23:33.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc56ffff640 1 -- 172.21.15.77:0/516641764 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5740343f0 con 0x7fc57c15e850 2024-03-06T07:23:33.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 -- 172.21.15.77:0/516641764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15e850 msgr2=0x7fc57c15eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 --2- 172.21.15.77:0/516641764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15e850 0x7fc57c15eca0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0731a0 tx=0x7fc57402be40 comp rx=0 tx=0).stop 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 -- 172.21.15.77:0/516641764 shutdown_connections 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 --2- 172.21.15.77:0/516641764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15e850 0x7fc57c15eca0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 --2- 172.21.15.77:0/516641764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c15bf30 0x7fc57c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 --2- 172.21.15.77:0/516641764 >> v2:172.21.15.158:3300/0 conn(0x7fc57c13dc10 0x7fc57c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 -- 172.21.15.77:0/516641764 >> 172.21.15.77:0/516641764 conn(0x7fc57c0cf8a0 msgr2=0x7fc57c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 -- 172.21.15.77:0/516641764 shutdown_connections 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.521+0000 7fc58350f640 1 -- 172.21.15.77:0/516641764 wait complete. 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc58350f640 1 Processor -- start 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc58350f640 1 -- start start 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc58350f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c13dc10 0x7fc57c15c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc58350f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15bf30 0x7fc57c13a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc58350f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc57c15e850 0x7fc57c159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc58350f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c15f1e0 con 0x7fc57c15bf30 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc58350f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c13dff0 con 0x7fc57c15e850 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc58350f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c15f360 con 0x7fc57c13dc10 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc580aaa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15bf30 0x7fc57c13a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc580aaa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15bf30 0x7fc57c13a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33246/0 (socket says 172.21.15.77:33246) 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.523+0000 7fc580aaa640 1 -- 172.21.15.77:0/1698713250 learned_addr learned my addr 172.21.15.77:0/1698713250 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc5812ab640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c13dc10 0x7fc57c15c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc581aac640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc57c15e850 0x7fc57c159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc580aaa640 1 -- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c13dc10 msgr2=0x7fc57c15c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc580aaa640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c13dc10 0x7fc57c15c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc580aaa640 1 -- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc57c15e850 msgr2=0x7fc57c159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc580aaa640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc57c15e850 0x7fc57c159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc580aaa640 1 -- 172.21.15.77:0/1698713250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c15ca20 con 0x7fc57c15bf30 2024-03-06T07:23:33.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc581aac640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc57c15e850 0x7fc57c159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc580aaa640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15bf30 0x7fc57c13a470 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc57000c820 tx=0x7fc57000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc56dffb640 1 -- 172.21.15.77:0/1698713250 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570015070 con 0x7fc57c15bf30 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc56dffb640 1 -- 172.21.15.77:0/1698713250 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570007a30 con 0x7fc57c15bf30 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57c15cd00 con 0x7fc57c15bf30 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc56dffb640 1 -- 172.21.15.77:0/1698713250 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570007e00 con 0x7fc57c15bf30 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.524+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57c15d2c0 con 0x7fc57c15bf30 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.525+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57c1f3b50 con 0x7fc57c15bf30 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.525+0000 7fc56dffb640 1 -- 172.21.15.77:0/1698713250 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc570018070 con 0x7fc57c15bf30 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.526+0000 7fc56dffb640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc56006b4d0 0x7fc56006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.526+0000 7fc56dffb640 1 -- 172.21.15.77:0/1698713250 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc57008a1e0 con 0x7fc57c15bf30 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.526+0000 7fc5812ab640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc56006b4d0 0x7fc56006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.526+0000 7fc5812ab640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc56006b4d0 0x7fc56006d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc578006fd0 tx=0x7fc578008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.530+0000 7fc56dffb640 1 -- 172.21.15.77:0/1698713250 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc570059060 con 0x7fc57c15bf30 2024-03-06T07:23:33.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.701+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc57c04f330 con 0x7fc57c15bf30 2024-03-06T07:23:33.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.702+0000 7fc56dffb640 1 -- 172.21.15.77:0/1698713250 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16614 (secure 0 0 0) 0x7fc570058960 con 0x7fc57c15bf30 2024-03-06T07:23:33.704 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:33.705 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":44,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:32.429067+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:30.703498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:31.400205+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:33.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc56006b4d0 msgr2=0x7fc56006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc56006b4d0 0x7fc56006d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc578006fd0 tx=0x7fc578008040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15bf30 msgr2=0x7fc57c13a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15bf30 0x7fc57c13a470 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc57000c820 tx=0x7fc57000cc20 comp rx=0 tx=0).stop 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 shutdown_connections 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc56006b4d0 0x7fc56006d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc57c15bf30 0x7fc57c13a470 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc57c15e850 0x7fc57c159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 --2- 172.21.15.77:0/1698713250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc57c13dc10 0x7fc57c15c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.704+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 >> 172.21.15.77:0/1698713250 conn(0x7fc57c0cf8a0 msgr2=0x7fc57c0b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.705+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 shutdown_connections 2024-03-06T07:23:33.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.705+0000 7fc58350f640 1 -- 172.21.15.77:0/1698713250 wait complete. 2024-03-06T07:23:33.718 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:33.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.817+0000 7f36ef37a640 1 Processor -- start 2024-03-06T07:23:33.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.818+0000 7f36ef37a640 1 -- start start 2024-03-06T07:23:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.818+0000 7f36ef37a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80cf0d0 0x7f36e815a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.818+0000 7f36ef37a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36e80719e0 0x7f36e81651f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.818+0000 7f36ef37a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e80b3320 con 0x7f36e80cfa90 2024-03-06T07:23:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.818+0000 7f36ef37a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e815a760 con 0x7f36e80719e0 2024-03-06T07:23:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.818+0000 7f36ef37a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80728d0 con 0x7f36e80cf0d0 2024-03-06T07:23:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ec915640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f36e80cfa90 0x7f36e80712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56240/0 (socket says 172.21.15.77:56240) 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ec915640 1 -- 172.21.15.77:0/3946563733 learned_addr learned my addr 172.21.15.77:0/3946563733 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ed116640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80cf0d0 0x7f36e815a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ed917640 1 --2- 172.21.15.77:0/3946563733 >> v2:172.21.15.158:3300/0 conn(0x7f36e80719e0 0x7f36e81651f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400110985 0 0) 0x7f36e80b3320 con 0x7f36e80cfa90 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d4003520 con 0x7f36e80cfa90 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478527050 0 0) 0x7f36d4003520 con 0x7f36e80cfa90 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80cf0d0 msgr2=0x7f36e815a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80cf0d0 0x7f36e815a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 >> v2:172.21.15.158:3300/0 conn(0x7f36e80719e0 msgr2=0x7f36e81651f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 --2- 172.21.15.77:0/3946563733 >> v2:172.21.15.158:3300/0 conn(0x7f36e80719e0 0x7f36e81651f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80b3590 con 0x7f36e80cfa90 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f36d8001670 con 0x7f36e80cfa90 2024-03-06T07:23:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 >> v1:172.21.15.77:6789/0 conn(0x7f36e80cfa90 legacy=0x7f36e80712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d40039b0 0x7f36d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d4004800 0x7f36d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80cf0d0 0x7f36d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e815a760 con 0x7f36d40039b0 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80728d0 con 0x7f36d4004800 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d400b9c0 con 0x7f36e80cf0d0 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ed116640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d40039b0 0x7f36d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ec915640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d4004800 0x7f36d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ed917640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80cf0d0 0x7f36d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ed116640 1 -- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80cf0d0 msgr2=0x7f36d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.819+0000 7f36ed116640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80cf0d0 0x7f36d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.820+0000 7f36ed116640 1 -- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d4004800 msgr2=0x7f36d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.820+0000 7f36ed116640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d4004800 0x7f36d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.820+0000 7f36ed116640 1 -- 172.21.15.77:0/3946563733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4003520 con 0x7f36d40039b0 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.820+0000 7f36ec915640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d4004800 0x7f36d4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.820+0000 7f36ed116640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d40039b0 0x7f36d4003dd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f36e0009af0 tx=0x7f36d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d74146649ff6d87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.820+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0039070 con 0x7f36d40039b0 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.820+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e002c6b0 con 0x7f36d40039b0 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.820+0000 7f36dffff640 1 -- 172.21.15.77:0/3946563733 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0009690 con 0x7f36d40039b0 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.821+0000 7f36ef37a640 1 -- 172.21.15.77:0/3946563733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d40039b0 msgr2=0x7f36d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.821+0000 7f36ef37a640 1 --2- 172.21.15.77:0/3946563733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d40039b0 0x7f36d4003dd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f36e0009af0 tx=0x7f36d4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.821+0000 7f36ed917640 1 -- 172.21.15.77:0/3946563733 reap_dead start 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.821+0000 7f36ef37a640 1 -- 172.21.15.77:0/3946563733 shutdown_connections 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.821+0000 7f36ef37a640 1 -- 172.21.15.77:0/3946563733 >> 172.21.15.77:0/3946563733 conn(0x7f36e806b190 msgr2=0x7f36e806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.821+0000 7f36ef37a640 1 -- 172.21.15.77:0/3946563733 shutdown_connections 2024-03-06T07:23:33.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.821+0000 7f36ef37a640 1 -- 172.21.15.77:0/3946563733 wait complete. 2024-03-06T07:23:33.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.822+0000 7f36ef37a640 1 Processor -- start 2024-03-06T07:23:33.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.822+0000 7f36ef37a640 1 -- start start 2024-03-06T07:23:33.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ef37a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d40039b0 0x7f36e814dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ef37a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80ac4b0 0x7f36e81f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ef37a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e80c8bf0 0x7f36e80c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ef37a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e8072030 con 0x7f36e80c8bf0 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ef37a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80cf4b0 con 0x7f36d40039b0 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ef37a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e8072530 con 0x7f36e80ac4b0 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e80c8bf0 0x7f36e80c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e80c8bf0 0x7f36e80c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33264/0 (socket says 172.21.15.77:33264) 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 -- 172.21.15.77:0/3752643563 learned_addr learned my addr 172.21.15.77:0/3752643563 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed116640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d40039b0 0x7f36e814dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ec915640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80ac4b0 0x7f36e81f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 -- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80ac4b0 msgr2=0x7f36e81f35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80ac4b0 0x7f36e81f35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 -- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d40039b0 msgr2=0x7f36e814dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d40039b0 0x7f36e814dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 -- 172.21.15.77:0/3752643563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80ac930 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed116640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d40039b0 0x7f36e814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ed917640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e80c8bf0 0x7f36e80c9070 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f36e8071220 tx=0x7f36e40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.823+0000 7f36ddffb640 1 -- 172.21.15.77:0/3752643563 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4003990 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.824+0000 7f36ddffb640 1 -- 172.21.15.77:0/3752643563 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e401b070 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.824+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e80acbe0 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.824+0000 7f36ddffb640 1 -- 172.21.15.77:0/3752643563 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4003b70 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.824+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e80ad1a0 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.825+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e80732b0 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.825+0000 7f36ddffb640 1 -- 172.21.15.77:0/3752643563 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36e4005d10 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.825+0000 7f36ddffb640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36cc06b500 0x7f36cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.826+0000 7f36ddffb640 1 -- 172.21.15.77:0/3752643563 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36e408a570 con 0x7f36e80c8bf0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.826+0000 7f36ed116640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36cc06b500 0x7f36cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.826+0000 7f36ed116640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36cc06b500 0x7f36cc06d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f36e0036000 tx=0x7f36e0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:33.830+0000 7f36ddffb640 1 -- 172.21.15.77:0/3752643563 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36e4059480 con 0x7f36e80c8bf0 2024-03-06T07:23:34.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.037+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f36e8068ff0 con 0x7f36e80c8bf0 2024-03-06T07:23:34.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.091+0000 7f36ddffb640 1 -- 172.21.15.77:0/3752643563 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f36e4058d80 con 0x7f36e80c8bf0 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36cc06b500 msgr2=0x7f36cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36cc06b500 0x7f36cc06d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f36e0036000 tx=0x7f36e0030000 comp rx=0 tx=0).stop 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e80c8bf0 msgr2=0x7f36e80c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e80c8bf0 0x7f36e80c9070 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f36e8071220 tx=0x7f36e40115a0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 shutdown_connections 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36cc06b500 0x7f36cc06d9c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e80c8bf0 0x7f36e80c9070 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e80ac4b0 0x7f36e81f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 --2- 172.21.15.77:0/3752643563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36d40039b0 0x7f36e814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 >> 172.21.15.77:0/3752643563 conn(0x7f36e806b190 msgr2=0x7f36e80b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 shutdown_connections 2024-03-06T07:23:34.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.094+0000 7f36ef37a640 1 -- 172.21.15.77:0/3752643563 wait complete. 2024-03-06T07:23:34.107 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:34.107 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:34.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.208+0000 7fccaa495640 1 Processor -- start 2024-03-06T07:23:34.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.208+0000 7fccaa495640 1 -- start start 2024-03-06T07:23:34.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fccaa495640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcca40b1e40 0x7fcca4168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fccaa495640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca40d2730 con 0x7fcca40d4860 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fccaa495640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca41618b0 con 0x7fcca41614d0 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fccaa495640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca40d4cf0 con 0x7fcca40b1e40 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fcca37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcca40d4860 0x7fcca40d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56252/0 (socket says 172.21.15.77:56252) 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fcca37fe640 1 -- 172.21.15.77:0/1325586671 learned_addr learned my addr 172.21.15.77:0/1325586671 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fcca8a32640 1 --2- 172.21.15.77:0/1325586671 >> v2:172.21.15.158:3300/0 conn(0x7fcca40b1e40 0x7fcca4168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fcca2ffd640 1 -- 172.21.15.77:0/1325586671 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430084361 0 0) 0x7fcca40d2730 con 0x7fcca40d4860 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.209+0000 7fcca2ffd640 1 -- 172.21.15.77:0/1325586671 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc8c003520 con 0x7fcca40d4860 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.210+0000 7fcca2ffd640 1 -- 172.21.15.77:0/1325586671 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2218834828 0 0) 0x7fcc8c003520 con 0x7fcca40d4860 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.210+0000 7fcca8a32640 1 -- 172.21.15.77:0/1325586671 >> v1:172.21.15.158:6789/0 conn(0x7fcca41614d0 legacy=0x7fcca40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.210+0000 7fcca8a32640 1 -- 172.21.15.77:0/1325586671 >> v1:172.21.15.77:6789/0 conn(0x7fcca40d4860 legacy=0x7fcca40d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.210+0000 7fcca8a32640 1 -- 172.21.15.77:0/1325586671 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40cf400 con 0x7fcca40b1e40 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.210+0000 7fcca8a32640 1 --2- 172.21.15.77:0/1325586671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 0x7fcca4168b60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcca40cf5d0 tx=0x7fcc9c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d492817a6aaa11a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fcca2ffd640 1 -- 172.21.15.77:0/1325586671 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c039070 con 0x7fcca40b1e40 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fcca2ffd640 1 -- 172.21.15.77:0/1325586671 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c02c8a0 con 0x7fcca40b1e40 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fcca2ffd640 1 -- 172.21.15.77:0/1325586671 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc98001670 con 0x7fcca40b1e40 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fccaa495640 1 -- 172.21.15.77:0/1325586671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 msgr2=0x7fcca4168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fccaa495640 1 --2- 172.21.15.77:0/1325586671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 0x7fcca4168b60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcca40cf5d0 tx=0x7fcc9c02bd70 comp rx=0 tx=0).stop 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fccaa495640 1 -- 172.21.15.77:0/1325586671 shutdown_connections 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fccaa495640 1 --2- 172.21.15.77:0/1325586671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 0x7fcca4168b60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fccaa495640 1 -- 172.21.15.77:0/1325586671 >> 172.21.15.77:0/1325586671 conn(0x7fcca40d3d40 msgr2=0x7fcca40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fccaa495640 1 -- 172.21.15.77:0/1325586671 shutdown_connections 2024-03-06T07:23:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.211+0000 7fccaa495640 1 -- 172.21.15.77:0/1325586671 wait complete. 2024-03-06T07:23:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.213+0000 7fccaa495640 1 Processor -- start 2024-03-06T07:23:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.213+0000 7fccaa495640 1 -- start start 2024-03-06T07:23:34.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.214+0000 7fccaa495640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 0x7fcca4147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.214+0000 7fccaa495640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcca40d4860 0x7fcca40b9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.214+0000 7fccaa495640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcca41614d0 0x7fcca40b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.214+0000 7fccaa495640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40cfc60 con 0x7fcca40d4860 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.214+0000 7fccaa495640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40cf980 con 0x7fcca40b1e40 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.214+0000 7fccaa495640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40b0910 con 0x7fcca41614d0 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.214+0000 7fcca37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcca40d4860 0x7fcca40b9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcca40d4860 0x7fcca40b9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33278/0 (socket says 172.21.15.77:33278) 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca37fe640 1 -- 172.21.15.77:0/4009182261 learned_addr learned my addr 172.21.15.77:0/4009182261 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca3fff640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 0x7fcca4147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca8a32640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcca41614d0 0x7fcca40b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca37fe640 1 -- 172.21.15.77:0/4009182261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcca41614d0 msgr2=0x7fcca40b5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca37fe640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcca41614d0 0x7fcca40b5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca37fe640 1 -- 172.21.15.77:0/4009182261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 msgr2=0x7fcca4147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca37fe640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 0x7fcca4147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca37fe640 1 -- 172.21.15.77:0/4009182261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40b65c0 con 0x7fcca40d4860 2024-03-06T07:23:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca37fe640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcca40d4860 0x7fcca40b9280 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800c730 tx=0x7fcc9800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca0ff9640 1 -- 172.21.15.77:0/4009182261 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc98017070 con 0x7fcca40d4860 2024-03-06T07:23:34.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca0ff9640 1 -- 172.21.15.77:0/4009182261 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc98005af0 con 0x7fcca40d4860 2024-03-06T07:23:34.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca40cbf00 con 0x7fcca40d4860 2024-03-06T07:23:34.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fcca0ff9640 1 -- 172.21.15.77:0/4009182261 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc98007070 con 0x7fcca40d4860 2024-03-06T07:23:34.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.215+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca40cc4c0 con 0x7fcca40d4860 2024-03-06T07:23:34.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.216+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca4068ff0 con 0x7fcca40d4860 2024-03-06T07:23:34.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.217+0000 7fcca0ff9640 1 -- 172.21.15.77:0/4009182261 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc9801a070 con 0x7fcca40d4860 2024-03-06T07:23:34.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.217+0000 7fcca0ff9640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc9006b4d0 0x7fcc9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.217+0000 7fcca0ff9640 1 -- 172.21.15.77:0/4009182261 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc980895e0 con 0x7fcca40d4860 2024-03-06T07:23:34.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.217+0000 7fcca3fff640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc9006b4d0 0x7fcc9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.218+0000 7fcca3fff640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc9006b4d0 0x7fcc9006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000f60 tx=0x7fcc94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.221+0000 7fcca0ff9640 1 -- 172.21.15.77:0/4009182261 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc98057ea0 con 0x7fcca40d4860 2024-03-06T07:23:34.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:34.433+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcca40a1c10 con 0x7fcca40d4860 2024-03-06T07:23:35.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.095+0000 7fcca0ff9640 1 -- 172.21.15.77:0/4009182261 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v19) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcc9805bf40 con 0x7fcca40d4860 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc9006b4d0 msgr2=0x7fcc9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc9006b4d0 0x7fcc9006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000f60 tx=0x7fcc94009040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcca40d4860 msgr2=0x7fcca40b9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcca40d4860 0x7fcca40b9280 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800c730 tx=0x7fcc9800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 shutdown_connections 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc9006b4d0 0x7fcc9006d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcca40d4860 0x7fcca40b9280 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcca41614d0 0x7fcca40b5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 --2- 172.21.15.77:0/4009182261 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcca40b1e40 0x7fcca4147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 >> 172.21.15.77:0/4009182261 conn(0x7fcca40d3d40 msgr2=0x7fcca40b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 shutdown_connections 2024-03-06T07:23:35.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.098+0000 7fccaa495640 1 -- 172.21.15.77:0/4009182261 wait complete. 2024-03-06T07:23:35.110 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:35.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:35.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.209+0000 7f01383a1640 1 Processor -- start 2024-03-06T07:23:35.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.209+0000 7f01383a1640 1 -- start start 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.210+0000 7f01383a1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f01300d2710 0x7f01300b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.210+0000 7f01383a1640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01300b3430 con 0x7f013015dfe0 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.210+0000 7f01383a1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01300b3710 con 0x7f01300d22b0 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.210+0000 7f01383a1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01300b31e0 con 0x7f01300d2710 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.210+0000 7f013613d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f013015dfe0 0x7f01300cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56256/0 (socket says 172.21.15.77:56256) 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.210+0000 7f013613d640 1 -- 172.21.15.77:0/3553204943 learned_addr learned my addr 172.21.15.77:0/3553204943 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013693e640 1 --2- 172.21.15.77:0/3553204943 >> v2:172.21.15.158:3300/0 conn(0x7f01300d2710 0x7f01300b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908759778 0 0) 0x7f01300b3430 con 0x7f013015dfe0 2024-03-06T07:23:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c003520 con 0x7f013015dfe0 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2688392914 0 0) 0x7f011c003520 con 0x7f013015dfe0 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 >> v2:172.21.15.158:3300/0 conn(0x7f01300d2710 msgr2=0x7f01300b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 --2- 172.21.15.77:0/3553204943 >> v2:172.21.15.158:3300/0 conn(0x7f01300d2710 0x7f01300b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 >> v1:172.21.15.158:6789/0 conn(0x7f01300d22b0 legacy=0x7f01300b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300b39f0 con 0x7f013015dfe0 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f012c001670 con 0x7f013015dfe0 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 >> v1:172.21.15.77:6789/0 conn(0x7f013015dfe0 legacy=0x7f01300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f011c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f011c004800 0x7f011c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f011c0051c0 0x7f011c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300b31e0 con 0x7f011c0039e0 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c00c010 con 0x7f011c0051c0 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c00c210 con 0x7f011c004800 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013613d640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f011c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013593c640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f011c004800 0x7f011c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013693e640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f011c0051c0 0x7f011c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013613d640 1 -- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f011c004800 msgr2=0x7f011c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013613d640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f011c004800 0x7f011c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013613d640 1 -- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f011c0051c0 msgr2=0x7f011c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013613d640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f011c0051c0 0x7f011c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013613d640 1 -- 172.21.15.77:0/3553204943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c003520 con 0x7f011c0039e0 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013693e640 1 -- 172.21.15.77:0/3553204943 reap_dead start 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.211+0000 7f013593c640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f011c004800 0x7f011c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.212+0000 7f013613d640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f011c003dc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f012c00c240 tx=0x7f011c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3134f494557c0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.212+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c007d30 con 0x7f011c0039e0 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.212+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c030070 con 0x7f011c0039e0 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.212+0000 7f013513b640 1 -- 172.21.15.77:0/3553204943 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c005d70 con 0x7f011c0039e0 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.213+0000 7f01383a1640 1 -- 172.21.15.77:0/3553204943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 msgr2=0x7f011c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.213+0000 7f01383a1640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f011c003dc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f012c00c240 tx=0x7f011c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.213+0000 7f01383a1640 1 -- 172.21.15.77:0/3553204943 shutdown_connections 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.213+0000 7f01383a1640 1 --2- 172.21.15.77:0/3553204943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f011c003dc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.213+0000 7f01383a1640 1 -- 172.21.15.77:0/3553204943 >> 172.21.15.77:0/3553204943 conn(0x7f01300d3cd0 msgr2=0x7f01300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.213+0000 7f01383a1640 1 -- 172.21.15.77:0/3553204943 shutdown_connections 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.213+0000 7f01383a1640 1 -- 172.21.15.77:0/3553204943 wait complete. 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.214+0000 7f01383a1640 1 Processor -- start 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.214+0000 7f01383a1640 1 -- start start 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f01383a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f01300a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f01383a1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01300a0810 0x7f01301f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f01383a1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01301f2f30 0x7f01301f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f01383a1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300d1c70 con 0x7f011c0039e0 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f01383a1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300b3d60 con 0x7f01301f2f30 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f01383a1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01300d1df0 con 0x7f01300a0810 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f01300a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f01300a02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33302/0 (socket says 172.21.15.77:33302) 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 -- 172.21.15.77:0/2677628088 learned_addr learned my addr 172.21.15.77:0/2677628088 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 -- 172.21.15.77:0/2677628088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01300a0810 msgr2=0x7f01301f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01300a0810 0x7f01301f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 -- 172.21.15.77:0/2677628088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01301f2f30 msgr2=0x7f01301f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01301f2f30 0x7f01301f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 -- 172.21.15.77:0/2677628088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300a0c90 con 0x7f011c0039e0 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f013613d640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f01300a02d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f012c00c360 tx=0x7f012c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f0126ffd640 1 -- 172.21.15.77:0/2677628088 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c007d30 con 0x7f011c0039e0 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.215+0000 7f0126ffd640 1 -- 172.21.15.77:0/2677628088 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c004c10 con 0x7f011c0039e0 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.216+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01300cccf0 con 0x7f011c0039e0 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.216+0000 7f0126ffd640 1 -- 172.21.15.77:0/2677628088 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c007070 con 0x7f011c0039e0 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.216+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013009de50 con 0x7f011c0039e0 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.217+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013004f330 con 0x7f011c0039e0 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.217+0000 7f0126ffd640 1 -- 172.21.15.77:0/2677628088 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f012c048020 con 0x7f011c0039e0 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.217+0000 7f0126ffd640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f011406b4d0 0x7f011406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.217+0000 7f0126ffd640 1 -- 172.21.15.77:0/2677628088 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f012c0ab490 con 0x7f011c0039e0 2024-03-06T07:23:35.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.217+0000 7f013593c640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f011406b4d0 0x7f011406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.218+0000 7f013593c640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f011406b4d0 0x7f011406d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0120007fd0 tx=0x7f0120009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.222+0000 7f0126ffd640 1 -- 172.21.15.77:0/2677628088 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f012c07a480 con 0x7f011c0039e0 2024-03-06T07:23:35.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.436+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f013006bea0 con 0x7f011c0039e0 2024-03-06T07:23:35.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.437+0000 7f0126ffd640 1 -- 172.21.15.77:0/2677628088 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f012c079d80 con 0x7f011c0039e0 2024-03-06T07:23:35.440 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:35.440 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1388591691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1388591691},{"type":"v1","addr":"172.21.15.158:6834","nonce":1388591691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/870048529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":870048529},{"type":"v1","addr":"172.21.15.158:6835","nonce":870048529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/650607050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":650607050},{"type":"v1","addr":"172.21.15.77:6836","nonce":650607050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:32.429377+0000","modified":"2024-03-06T07:23:34.434503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2240574726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2240574726},{"type":"v1","addr":"172.21.15.77:6837","nonce":2240574726}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":3}]} 2024-03-06T07:23:35.440 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 19 2024-03-06T07:23:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f011406b4d0 msgr2=0x7f011406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f011406b4d0 0x7f011406d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0120007fd0 tx=0x7f0120009040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 msgr2=0x7f01300a02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f01300a02d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f012c00c360 tx=0x7f012c02e070 comp rx=0 tx=0).stop 2024-03-06T07:23:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 shutdown_connections 2024-03-06T07:23:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01301f2f30 0x7f01301f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01300a0810 0x7f01301f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f011406b4d0 0x7f011406d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 --2- 172.21.15.77:0/2677628088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f011c0039e0 0x7f01300a02d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 >> 172.21.15.77:0/2677628088 conn(0x7f01300d3cd0 msgr2=0x7f01300b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 shutdown_connections 2024-03-06T07:23:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.439+0000 7f01383a1640 1 -- 172.21.15.77:0/2677628088 wait complete. 2024-03-06T07:23:35.455 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.555+0000 7fc25110f640 1 Processor -- start 2024-03-06T07:23:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.555+0000 7fc25110f640 1 -- start start 2024-03-06T07:23:35.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc25110f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc24c15dfe0 0x7fc24c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc25110f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0cf830 0x7fc24c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc25110f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc24c0d27f0 0x7fc24c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc25110f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24c0d2c50 con 0x7fc24c15dfe0 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc25110f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24c0b31e0 con 0x7fc24c0d27f0 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc25110f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c0b3d60 con 0x7fc24c0cf830 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc24c15dfe0 0x7fc24c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc24c15dfe0 0x7fc24c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33308/0 (socket says 172.21.15.77:33308) 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24ad9d640 1 -- 172.21.15.77:0/760564152 learned_addr learned my addr 172.21.15.77:0/760564152 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24a59c640 1 --2- 172.21.15.77:0/760564152 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0cf830 0x7fc24c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24b59e640 1 --2- 172.21.15.77:0/760564152 >> v2:172.21.15.158:3300/0 conn(0x7fc24c0d27f0 0x7fc24c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24ad9d640 1 -- 172.21.15.77:0/760564152 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0cf830 msgr2=0x7fc24c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24ad9d640 1 --2- 172.21.15.77:0/760564152 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0cf830 0x7fc24c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24ad9d640 1 -- 172.21.15.77:0/760564152 >> v2:172.21.15.158:3300/0 conn(0x7fc24c0d27f0 msgr2=0x7fc24c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24ad9d640 1 --2- 172.21.15.77:0/760564152 >> v2:172.21.15.158:3300/0 conn(0x7fc24c0d27f0 0x7fc24c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.556+0000 7fc24ad9d640 1 -- 172.21.15.77:0/760564152 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c0b3430 con 0x7fc24c15dfe0 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.557+0000 7fc24ad9d640 1 --2- 172.21.15.77:0/760564152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c0cf2f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc24c067ad0 tx=0x7fc24002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85739e8ebdcc939 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.557+0000 7fc249d9b640 1 -- 172.21.15.77:0/760564152 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240035020 con 0x7fc24c15dfe0 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.557+0000 7fc249d9b640 1 -- 172.21.15.77:0/760564152 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc240031870 con 0x7fc24c15dfe0 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.557+0000 7fc249d9b640 1 -- 172.21.15.77:0/760564152 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240031c40 con 0x7fc24c15dfe0 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 -- 172.21.15.77:0/760564152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 msgr2=0x7fc24c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 --2- 172.21.15.77:0/760564152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c0cf2f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc24c067ad0 tx=0x7fc24002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 -- 172.21.15.77:0/760564152 shutdown_connections 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 --2- 172.21.15.77:0/760564152 >> v2:172.21.15.158:3300/0 conn(0x7fc24c0d27f0 0x7fc24c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 --2- 172.21.15.77:0/760564152 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0cf830 0x7fc24c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 --2- 172.21.15.77:0/760564152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c0cf2f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 -- 172.21.15.77:0/760564152 >> 172.21.15.77:0/760564152 conn(0x7fc24c0d3cd0 msgr2=0x7fc24c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 -- 172.21.15.77:0/760564152 shutdown_connections 2024-03-06T07:23:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.558+0000 7fc25110f640 1 -- 172.21.15.77:0/760564152 wait complete. 2024-03-06T07:23:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.559+0000 7fc25110f640 1 Processor -- start 2024-03-06T07:23:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.559+0000 7fc25110f640 1 -- start start 2024-03-06T07:23:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc25110f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc24c0cf830 0x7fc24c1f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc25110f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0d27f0 0x7fc24c160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc25110f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c15fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc25110f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c0d1c70 con 0x7fc24c15dfe0 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc25110f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c0b39f0 con 0x7fc24c0cf830 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc25110f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c0d1df0 con 0x7fc24c0d27f0 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c15fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c15fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33312/0 (socket says 172.21.15.77:33312) 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24b59e640 1 -- 172.21.15.77:0/3887576091 learned_addr learned my addr 172.21.15.77:0/3887576091 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24ad9d640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc24c0cf830 0x7fc24c1f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24a59c640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0d27f0 0x7fc24c160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24b59e640 1 -- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0d27f0 msgr2=0x7fc24c160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24b59e640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0d27f0 0x7fc24c160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24b59e640 1 -- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc24c0cf830 msgr2=0x7fc24c1f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24b59e640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc24c0cf830 0x7fc24c1f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24b59e640 1 -- 172.21.15.77:0/3887576091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c0b9740 con 0x7fc24c15dfe0 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.560+0000 7fc24a59c640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0d27f0 0x7fc24c160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.561+0000 7fc24b59e640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c15fcb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc244007e40 tx=0x7fc24400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.561+0000 7fc2337fe640 1 -- 172.21.15.77:0/3887576091 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244014620 con 0x7fc24c15dfe0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.561+0000 7fc2337fe640 1 -- 172.21.15.77:0/3887576091 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc244019070 con 0x7fc24c15dfe0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.561+0000 7fc2337fe640 1 -- 172.21.15.77:0/3887576091 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244014a20 con 0x7fc24c15dfe0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.561+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24c0b9a20 con 0x7fc24c15dfe0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.561+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24c0cceb0 con 0x7fc24c15dfe0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.562+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24c0b5200 con 0x7fc24c15dfe0 2024-03-06T07:23:35.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.562+0000 7fc2337fe640 1 -- 172.21.15.77:0/3887576091 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc244014bc0 con 0x7fc24c15dfe0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.563+0000 7fc2337fe640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc23806b4d0 0x7fc23806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.563+0000 7fc24ad9d640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc23806b4d0 0x7fc23806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.563+0000 7fc2337fe640 1 -- 172.21.15.77:0/3887576091 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc24408aad0 con 0x7fc24c15dfe0 2024-03-06T07:23:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.563+0000 7fc24ad9d640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc23806b4d0 0x7fc23806d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc240036000 tx=0x7fc24003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.567+0000 7fc2337fe640 1 -- 172.21.15.77:0/3887576091 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc24c0b5200 con 0x7fc24c15dfe0 2024-03-06T07:23:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.740+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc24c04f330 con 0x7fc24c15dfe0 2024-03-06T07:23:35.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.741+0000 7fc2337fe640 1 -- 172.21.15.77:0/3887576091 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16601 (secure 0 0 0) 0x7fc244059950 con 0x7fc24c15dfe0 2024-03-06T07:23:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:35.743 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":45,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:34.087201+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:30.703498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:31.400205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:35.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc23806b4d0 msgr2=0x7fc23806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc23806b4d0 0x7fc23806d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc240036000 tx=0x7fc24003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 msgr2=0x7fc24c15fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c15fcb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc244007e40 tx=0x7fc24400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 shutdown_connections 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc23806b4d0 0x7fc23806d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc24c15dfe0 0x7fc24c15fcb0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc24c0d27f0 0x7fc24c160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 --2- 172.21.15.77:0/3887576091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc24c0cf830 0x7fc24c1f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 >> 172.21.15.77:0/3887576091 conn(0x7fc24c0d3cd0 msgr2=0x7fc24c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 shutdown_connections 2024-03-06T07:23:35.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.743+0000 7fc25110f640 1 -- 172.21.15.77:0/3887576091 wait complete. 2024-03-06T07:23:35.757 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:35.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:35.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.855+0000 7f87dce08640 1 Processor -- start 2024-03-06T07:23:35.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.855+0000 7f87dce08640 1 -- start start 2024-03-06T07:23:35.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87dce08640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d815dfe0 0x7f87d80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87dce08640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d80cf830 0x7f87d80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87dce08640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d80b3430 con 0x7f87d80d27f0 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87dce08640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d80d2c50 con 0x7f87d80cf830 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87dce08640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d80b31e0 con 0x7f87d815dfe0 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d5d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d80cf830 0x7f87d80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d5d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87d80cf830 0x7f87d80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33320/0 (socket says 172.21.15.77:33320) 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d5d9b640 1 -- 172.21.15.77:0/3698971213 learned_addr learned my addr 172.21.15.77:0/3698971213 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d659c640 1 --2- 172.21.15.77:0/3698971213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d815dfe0 0x7f87d80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d5d9b640 1 -- 172.21.15.77:0/3698971213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d815dfe0 msgr2=0x7f87d80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d5d9b640 1 --2- 172.21.15.77:0/3698971213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d815dfe0 0x7f87d80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d5d9b640 1 -- 172.21.15.77:0/3698971213 >> v1:172.21.15.77:6789/0 conn(0x7f87d80d27f0 legacy=0x7f87d80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d5d9b640 1 -- 172.21.15.77:0/3698971213 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d80b3710 con 0x7f87d80cf830 2024-03-06T07:23:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.857+0000 7f87d559a640 1 -- 172.21.15.77:0/3698971213 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424973886 0 0) 0x7f87d80b3430 con 0x7f87d80d27f0 2024-03-06T07:23:35.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.858+0000 7f87d5d9b640 1 --2- 172.21.15.77:0/3698971213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d80d22b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009870 tx=0x7f87cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476caf69c32f00db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.858+0000 7f87d559a640 1 -- 172.21.15.77:0/3698971213 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc034070 con 0x7f87d80cf830 2024-03-06T07:23:35.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.858+0000 7f87d559a640 1 -- 172.21.15.77:0/3698971213 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc030840 con 0x7f87d80cf830 2024-03-06T07:23:35.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.858+0000 7f87d559a640 1 -- 172.21.15.77:0/3698971213 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc030c10 con 0x7f87d80cf830 2024-03-06T07:23:35.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.859+0000 7f87dce08640 1 -- 172.21.15.77:0/3698971213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 msgr2=0x7f87d80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.859+0000 7f87dce08640 1 --2- 172.21.15.77:0/3698971213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d80d22b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009870 tx=0x7f87cc02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:35.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.859+0000 7f87dce08640 1 -- 172.21.15.77:0/3698971213 shutdown_connections 2024-03-06T07:23:35.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.859+0000 7f87dce08640 1 --2- 172.21.15.77:0/3698971213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d80d22b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.859+0000 7f87dce08640 1 --2- 172.21.15.77:0/3698971213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d815dfe0 0x7f87d80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.859+0000 7f87dce08640 1 -- 172.21.15.77:0/3698971213 >> 172.21.15.77:0/3698971213 conn(0x7f87d80d3cd0 msgr2=0x7f87d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.859+0000 7f87dce08640 1 -- 172.21.15.77:0/3698971213 shutdown_connections 2024-03-06T07:23:35.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.859+0000 7f87dce08640 1 -- 172.21.15.77:0/3698971213 wait complete. 2024-03-06T07:23:35.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.860+0000 7f87dce08640 1 Processor -- start 2024-03-06T07:23:35.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.860+0000 7f87dce08640 1 -- start start 2024-03-06T07:23:35.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87dce08640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d8161130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87dce08640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d80d27f0 0x7f87d80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87dce08640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87d815dfe0 0x7f87d814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87dce08640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d80d1c70 con 0x7f87d80cf830 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87dce08640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d80b3d60 con 0x7f87d815dfe0 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87dce08640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d80d1df0 con 0x7f87d80d27f0 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d8161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d8161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33328/0 (socket says 172.21.15.77:33328) 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 -- 172.21.15.77:0/4127897189 learned_addr learned my addr 172.21.15.77:0/4127897189 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d6d9d640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87d815dfe0 0x7f87d814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d5d9b640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d80d27f0 0x7f87d80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 -- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d80d27f0 msgr2=0x7f87d80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d80d27f0 0x7f87d80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 -- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87d815dfe0 msgr2=0x7f87d814f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87d815dfe0 0x7f87d814f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 -- 172.21.15.77:0/4127897189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d80cb5f0 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d6d9d640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87d815dfe0 0x7f87d814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d5d9b640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d80d27f0 0x7f87d80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.861+0000 7f87d659c640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d8161130 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f87c800d5f0 tx=0x7f87c800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.862+0000 7f87c6ffd640 1 -- 172.21.15.77:0/4127897189 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c80125f0 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.862+0000 7f87c6ffd640 1 -- 172.21.15.77:0/4127897189 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8019070 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.862+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d80cb8d0 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.862+0000 7f87c6ffd640 1 -- 172.21.15.77:0/4127897189 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c80129d0 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.862+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d80cb2e0 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.863+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d804f330 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.863+0000 7f87c6ffd640 1 -- 172.21.15.77:0/4127897189 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87c8012b70 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.864+0000 7f87c6ffd640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f87b806b4d0 0x7f87b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.864+0000 7f87c6ffd640 1 -- 172.21.15.77:0/4127897189 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87c808a5f0 con 0x7f87d80cf830 2024-03-06T07:23:35.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.864+0000 7f87d5d9b640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f87b806b4d0 0x7f87b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:35.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.864+0000 7f87d5d9b640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f87b806b4d0 0x7f87b806d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87cc036000 tx=0x7f87cc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:35.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:35.868+0000 7f87c6ffd640 1 -- 172.21.15.77:0/4127897189 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87c8059470 con 0x7f87d80cf830 2024-03-06T07:23:36.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.153+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87d80cbdf0 con 0x7f87b806b4d0 2024-03-06T07:23:36.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.159+0000 7f87c6ffd640 1 -- 172.21.15.77:0/4127897189 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246636 (secure 0 0 0) 0x7f87d80cbdf0 con 0x7f87b806b4d0 2024-03-06T07:23:36.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:36.166 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:23:36.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f87b806b4d0 msgr2=0x7f87b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f87b806b4d0 0x7f87b806d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87cc036000 tx=0x7f87cc03b040 comp rx=0 tx=0).stop 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 msgr2=0x7f87d8161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d8161130 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f87c800d5f0 tx=0x7f87c800dab0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 shutdown_connections 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f87d815dfe0 0x7f87d814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f87d80d27f0 0x7f87d80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f87b806b4d0 0x7f87b806d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 --2- 172.21.15.77:0/4127897189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87d80cf830 0x7f87d8161130 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 >> 172.21.15.77:0/4127897189 conn(0x7f87d80d3cd0 msgr2=0x7f87d806ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 shutdown_connections 2024-03-06T07:23:36.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.162+0000 7f87dce08640 1 -- 172.21.15.77:0/4127897189 wait complete. 2024-03-06T07:23:36.175 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-03-06T07:23:35.402131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258624,"kb_used_data":5012,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754716096,"statfs":{"total":773094113280,"available":772829282304,"internally_reserved":0,"allocated":5132288,"data_stored":2741483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.328272"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086421+0000","last_change":"2024-03-06T07:23:32.086421+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086421+0000","last_undegraded":"2024-03-06T07:23:32.086421+0000","last_fullsized":"2024-03-06T07:23:32.086421+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085015+0000","last_change":"2024-03-06T07:23:32.085015+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085015+0000","last_undegraded":"2024-03-06T07:23:32.085015+0000","last_fullsized":"2024-03-06T07:23:32.085015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084977+0000","last_change":"2024-03-06T07:23:32.084977+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084977+0000","last_undegraded":"2024-03-06T07:23:32.084977+0000","last_fullsized":"2024-03-06T07:23:32.084977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087731+0000","last_change":"2024-03-06T07:23:32.087731+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087731+0000","last_undegraded":"2024-03-06T07:23:32.087731+0000","last_fullsized":"2024-03-06T07:23:32.087731+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085010+0000","last_change":"2024-03-06T07:23:32.085010+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085010+0000","last_undegraded":"2024-03-06T07:23:32.085010+0000","last_fullsized":"2024-03-06T07:23:32.085010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.092100+0000","last_change":"2024-03-06T07:23:32.092100+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.092100+0000","last_undegraded":"2024-03-06T07:23:32.092100+0000","last_fullsized":"2024-03-06T07:23:32.092100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084934+0000","last_change":"2024-03-06T07:23:32.084934+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084934+0000","last_undegraded":"2024-03-06T07:23:32.084934+0000","last_fullsized":"2024-03-06T07:23:32.084934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088106+0000","last_change":"2024-03-06T07:23:32.088106+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088106+0000","last_undegraded":"2024-03-06T07:23:32.088106+0000","last_fullsized":"2024-03-06T07:23:32.088106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087051+0000","last_change":"2024-03-06T07:23:32.087051+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087051+0000","last_undegraded":"2024-03-06T07:23:32.087051+0000","last_fullsized":"2024-03-06T07:23:32.087051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088416+0000","last_change":"2024-03-06T07:23:32.088416+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088416+0000","last_undegraded":"2024-03-06T07:23:32.088416+0000","last_fullsized":"2024-03-06T07:23:32.088416+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087393+0000","last_change":"2024-03-06T07:23:32.087393+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087393+0000","last_undegraded":"2024-03-06T07:23:32.087393+0000","last_fullsized":"2024-03-06T07:23:32.087393+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086312+0000","last_change":"2024-03-06T07:23:32.086312+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086312+0000","last_undegraded":"2024-03-06T07:23:32.086312+0000","last_fullsized":"2024-03-06T07:23:32.086312+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085227+0000","last_change":"2024-03-06T07:23:32.085227+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085227+0000","last_undegraded":"2024-03-06T07:23:32.085227+0000","last_fullsized":"2024-03-06T07:23:32.085227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087620+0000","last_change":"2024-03-06T07:23:32.087620+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087620+0000","last_undegraded":"2024-03-06T07:23:32.087620+0000","last_fullsized":"2024-03-06T07:23:32.087620+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087300+0000","last_change":"2024-03-06T07:23:32.087300+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087300+0000","last_undegraded":"2024-03-06T07:23:32.087300+0000","last_fullsized":"2024-03-06T07:23:32.087300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082117+0000","last_change":"2024-03-06T07:23:32.082117+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082117+0000","last_undegraded":"2024-03-06T07:23:32.082117+0000","last_fullsized":"2024-03-06T07:23:32.082117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086672+0000","last_change":"2024-03-06T07:23:32.086672+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086672+0000","last_undegraded":"2024-03-06T07:23:32.086672+0000","last_fullsized":"2024-03-06T07:23:32.086672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085949+0000","last_change":"2024-03-06T07:23:32.085949+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085949+0000","last_undegraded":"2024-03-06T07:23:32.085949+0000","last_fullsized":"2024-03-06T07:23:32.085949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082809+0000","last_change":"2024-03-06T07:23:32.082809+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082809+0000","last_undegraded":"2024-03-06T07:23:32.082809+0000","last_fullsized":"2024-03-06T07:23:32.082809+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085204+0000","last_change":"2024-03-06T07:23:32.085204+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085204+0000","last_undegraded":"2024-03-06T07:23:32.085204+0000","last_fullsized":"2024-03-06T07:23:32.085204+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.092218+0000","last_change":"2024-03-06T07:23:32.092218+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.092218+0000","last_undegraded":"2024-03-06T07:23:32.092218+0000","last_fullsized":"2024-03-06T07:23:32.092218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087214+0000","last_change":"2024-03-06T07:23:32.087214+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087214+0000","last_undegraded":"2024-03-06T07:23:32.087214+0000","last_fullsized":"2024-03-06T07:23:32.087214+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081010+0000","last_change":"2024-03-06T07:23:32.081010+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081010+0000","last_undegraded":"2024-03-06T07:23:32.081010+0000","last_fullsized":"2024-03-06T07:23:32.081010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081561+0000","last_change":"2024-03-06T07:23:32.081561+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081561+0000","last_undegraded":"2024-03-06T07:23:32.081561+0000","last_fullsized":"2024-03-06T07:23:32.081561+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084558+0000","last_change":"2024-03-06T07:23:32.084558+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084558+0000","last_undegraded":"2024-03-06T07:23:32.084558+0000","last_fullsized":"2024-03-06T07:23:32.084558+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084689+0000","last_change":"2024-03-06T07:23:32.084689+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084689+0000","last_undegraded":"2024-03-06T07:23:32.084689+0000","last_fullsized":"2024-03-06T07:23:32.084689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081376+0000","last_change":"2024-03-06T07:23:32.081376+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081376+0000","last_undegraded":"2024-03-06T07:23:32.081376+0000","last_fullsized":"2024-03-06T07:23:32.081376+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085933+0000","last_change":"2024-03-06T07:23:32.085933+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085933+0000","last_undegraded":"2024-03-06T07:23:32.085933+0000","last_fullsized":"2024-03-06T07:23:32.085933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083052+0000","last_change":"2024-03-06T07:23:32.083052+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083052+0000","last_undegraded":"2024-03-06T07:23:32.083052+0000","last_fullsized":"2024-03-06T07:23:32.083052+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083219+0000","last_change":"2024-03-06T07:23:32.083219+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083219+0000","last_undegraded":"2024-03-06T07:23:32.083219+0000","last_fullsized":"2024-03-06T07:23:32.083219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086871+0000","last_change":"2024-03-06T07:23:32.086871+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086871+0000","last_undegraded":"2024-03-06T07:23:32.086871+0000","last_fullsized":"2024-03-06T07:23:32.086871+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084351+0000","last_change":"2024-03-06T07:23:32.084351+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084351+0000","last_undegraded":"2024-03-06T07:23:32.084351+0000","last_fullsized":"2024-03-06T07:23:32.084351+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084795+0000","last_change":"2024-03-06T07:23:32.084795+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084795+0000","last_undegraded":"2024-03-06T07:23:32.084795+0000","last_fullsized":"2024-03-06T07:23:32.084795+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098181+0000","last_change":"2024-03-06T07:23:32.096250+0000","last_active":"2024-03-06T07:23:32.098181+0000","last_peered":"2024-03-06T07:23:32.098181+0000","last_clean":"2024-03-06T07:23:32.098181+0000","last_became_active":"2024-03-06T07:23:32.095974+0000","last_became_peered":"2024-03-06T07:23:32.095974+0000","last_unstale":"2024-03-06T07:23:32.098181+0000","last_undegraded":"2024-03-06T07:23:32.098181+0000","last_fullsized":"2024-03-06T07:23:32.098181+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:00.492894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084125+0000","last_change":"2024-03-06T07:23:32.084125+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084125+0000","last_undegraded":"2024-03-06T07:23:32.084125+0000","last_fullsized":"2024-03-06T07:23:32.084125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.095292+0000","last_change":"2024-03-06T07:23:32.091961+0000","last_active":"2024-03-06T07:23:32.095292+0000","last_peered":"2024-03-06T07:23:32.095292+0000","last_clean":"2024-03-06T07:23:32.095292+0000","last_became_active":"2024-03-06T07:23:32.091129+0000","last_became_peered":"2024-03-06T07:23:32.091129+0000","last_unstale":"2024-03-06T07:23:32.095292+0000","last_undegraded":"2024-03-06T07:23:32.095292+0000","last_fullsized":"2024-03-06T07:23:32.095292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.717543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084960+0000","last_change":"2024-03-06T07:23:32.084960+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084960+0000","last_undegraded":"2024-03-06T07:23:32.084960+0000","last_fullsized":"2024-03-06T07:23:32.084960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.095230+0000","last_change":"2024-03-06T07:23:32.091042+0000","last_active":"2024-03-06T07:23:32.095230+0000","last_peered":"2024-03-06T07:23:32.095230+0000","last_clean":"2024-03-06T07:23:32.095230+0000","last_became_active":"2024-03-06T07:23:32.090683+0000","last_became_peered":"2024-03-06T07:23:32.090683+0000","last_unstale":"2024-03-06T07:23:32.095230+0000","last_undegraded":"2024-03-06T07:23:32.095230+0000","last_fullsized":"2024-03-06T07:23:32.095230+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:41.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084970+0000","last_change":"2024-03-06T07:23:32.084970+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084970+0000","last_undegraded":"2024-03-06T07:23:32.084970+0000","last_fullsized":"2024-03-06T07:23:32.084970+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097212+0000","last_change":"2024-03-06T07:23:32.093816+0000","last_active":"2024-03-06T07:23:32.097212+0000","last_peered":"2024-03-06T07:23:32.097212+0000","last_clean":"2024-03-06T07:23:32.097212+0000","last_became_active":"2024-03-06T07:23:32.093538+0000","last_became_peered":"2024-03-06T07:23:32.093538+0000","last_unstale":"2024-03-06T07:23:32.097212+0000","last_undegraded":"2024-03-06T07:23:32.097212+0000","last_fullsized":"2024-03-06T07:23:32.097212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:23.944371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082422+0000","last_change":"2024-03-06T07:23:32.082422+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082422+0000","last_undegraded":"2024-03-06T07:23:32.082422+0000","last_fullsized":"2024-03-06T07:23:32.082422+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100838+0000","last_change":"2024-03-06T07:23:32.099457+0000","last_active":"2024-03-06T07:23:32.100838+0000","last_peered":"2024-03-06T07:23:32.100838+0000","last_clean":"2024-03-06T07:23:32.100838+0000","last_became_active":"2024-03-06T07:23:32.099188+0000","last_became_peered":"2024-03-06T07:23:32.099188+0000","last_unstale":"2024-03-06T07:23:32.100838+0000","last_undegraded":"2024-03-06T07:23:32.100838+0000","last_fullsized":"2024-03-06T07:23:32.100838+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:04.671716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.089376+0000","last_change":"2024-03-06T07:23:32.089376+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.089376+0000","last_undegraded":"2024-03-06T07:23:32.089376+0000","last_fullsized":"2024-03-06T07:23:32.089376+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.092153+0000","last_change":"2024-03-06T07:23:32.090283+0000","last_active":"2024-03-06T07:23:32.092153+0000","last_peered":"2024-03-06T07:23:32.092153+0000","last_clean":"2024-03-06T07:23:32.092153+0000","last_became_active":"2024-03-06T07:23:32.089989+0000","last_became_peered":"2024-03-06T07:23:32.089989+0000","last_unstale":"2024-03-06T07:23:32.092153+0000","last_undegraded":"2024-03-06T07:23:32.092153+0000","last_fullsized":"2024-03-06T07:23:32.092153+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:21.231136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087568+0000","last_change":"2024-03-06T07:23:32.087568+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087568+0000","last_undegraded":"2024-03-06T07:23:32.087568+0000","last_fullsized":"2024-03-06T07:23:32.087568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097985+0000","last_change":"2024-03-06T07:23:32.095613+0000","last_active":"2024-03-06T07:23:32.097985+0000","last_peered":"2024-03-06T07:23:32.097985+0000","last_clean":"2024-03-06T07:23:32.097985+0000","last_became_active":"2024-03-06T07:23:32.095182+0000","last_became_peered":"2024-03-06T07:23:32.095182+0000","last_unstale":"2024-03-06T07:23:32.097985+0000","last_undegraded":"2024-03-06T07:23:32.097985+0000","last_fullsized":"2024-03-06T07:23:32.097985+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:17.223200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082182+0000","last_change":"2024-03-06T07:23:32.082182+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082182+0000","last_undegraded":"2024-03-06T07:23:32.082182+0000","last_fullsized":"2024-03-06T07:23:32.082182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100297+0000","last_change":"2024-03-06T07:23:32.099418+0000","last_active":"2024-03-06T07:23:32.100297+0000","last_peered":"2024-03-06T07:23:32.100297+0000","last_clean":"2024-03-06T07:23:32.100297+0000","last_became_active":"2024-03-06T07:23:32.099121+0000","last_became_peered":"2024-03-06T07:23:32.099121+0000","last_unstale":"2024-03-06T07:23:32.100297+0000","last_undegraded":"2024-03-06T07:23:32.100297+0000","last_fullsized":"2024-03-06T07:23:32.100297+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:34.981739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085484+0000","last_change":"2024-03-06T07:23:32.085484+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085484+0000","last_undegraded":"2024-03-06T07:23:32.085484+0000","last_fullsized":"2024-03-06T07:23:32.085484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.091583+0000","last_change":"2024-03-06T07:23:32.087920+0000","last_active":"2024-03-06T07:23:32.091583+0000","last_peered":"2024-03-06T07:23:32.091583+0000","last_clean":"2024-03-06T07:23:32.091583+0000","last_became_active":"2024-03-06T07:23:32.087584+0000","last_became_peered":"2024-03-06T07:23:32.087584+0000","last_unstale":"2024-03-06T07:23:32.091583+0000","last_undegraded":"2024-03-06T07:23:32.091583+0000","last_fullsized":"2024-03-06T07:23:32.091583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:40.708193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085997+0000","last_change":"2024-03-06T07:23:32.085997+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085997+0000","last_undegraded":"2024-03-06T07:23:32.085997+0000","last_fullsized":"2024-03-06T07:23:32.085997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099448+0000","last_change":"2024-03-06T07:23:32.097214+0000","last_active":"2024-03-06T07:23:32.099448+0000","last_peered":"2024-03-06T07:23:32.099448+0000","last_clean":"2024-03-06T07:23:32.099448+0000","last_became_active":"2024-03-06T07:23:32.096930+0000","last_became_peered":"2024-03-06T07:23:32.096930+0000","last_unstale":"2024-03-06T07:23:32.099448+0000","last_undegraded":"2024-03-06T07:23:32.099448+0000","last_fullsized":"2024-03-06T07:23:32.099448+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:38.093438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087625+0000","last_change":"2024-03-06T07:23:32.087625+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087625+0000","last_undegraded":"2024-03-06T07:23:32.087625+0000","last_fullsized":"2024-03-06T07:23:32.087625+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097744+0000","last_change":"2024-03-06T07:23:32.096162+0000","last_active":"2024-03-06T07:23:32.097744+0000","last_peered":"2024-03-06T07:23:32.097744+0000","last_clean":"2024-03-06T07:23:32.097744+0000","last_became_active":"2024-03-06T07:23:32.095663+0000","last_became_peered":"2024-03-06T07:23:32.095663+0000","last_unstale":"2024-03-06T07:23:32.097744+0000","last_undegraded":"2024-03-06T07:23:32.097744+0000","last_fullsized":"2024-03-06T07:23:32.097744+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086333+0000","last_change":"2024-03-06T07:23:32.086333+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086333+0000","last_undegraded":"2024-03-06T07:23:32.086333+0000","last_fullsized":"2024-03-06T07:23:32.086333+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098226+0000","last_change":"2024-03-06T07:23:32.093863+0000","last_active":"2024-03-06T07:23:32.098226+0000","last_peered":"2024-03-06T07:23:32.098226+0000","last_clean":"2024-03-06T07:23:32.098226+0000","last_became_active":"2024-03-06T07:23:32.093600+0000","last_became_peered":"2024-03-06T07:23:32.093600+0000","last_unstale":"2024-03-06T07:23:32.098226+0000","last_undegraded":"2024-03-06T07:23:32.098226+0000","last_fullsized":"2024-03-06T07:23:32.098226+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:33.913245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081445+0000","last_change":"2024-03-06T07:23:32.081445+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081445+0000","last_undegraded":"2024-03-06T07:23:32.081445+0000","last_fullsized":"2024-03-06T07:23:32.081445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099946+0000","last_change":"2024-03-06T07:23:32.097368+0000","last_active":"2024-03-06T07:23:32.099946+0000","last_peered":"2024-03-06T07:23:32.099946+0000","last_clean":"2024-03-06T07:23:32.099946+0000","last_became_active":"2024-03-06T07:23:32.097100+0000","last_became_peered":"2024-03-06T07:23:32.097100+0000","last_unstale":"2024-03-06T07:23:32.099946+0000","last_undegraded":"2024-03-06T07:23:32.099946+0000","last_fullsized":"2024-03-06T07:23:32.099946+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:56.501180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088639+0000","last_change":"2024-03-06T07:23:32.088639+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088639+0000","last_undegraded":"2024-03-06T07:23:32.088639+0000","last_fullsized":"2024-03-06T07:23:32.088639+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097589+0000","last_change":"2024-03-06T07:23:32.094421+0000","last_active":"2024-03-06T07:23:32.097589+0000","last_peered":"2024-03-06T07:23:32.097589+0000","last_clean":"2024-03-06T07:23:32.097589+0000","last_became_active":"2024-03-06T07:23:32.094161+0000","last_became_peered":"2024-03-06T07:23:32.094161+0000","last_unstale":"2024-03-06T07:23:32.097589+0000","last_undegraded":"2024-03-06T07:23:32.097589+0000","last_fullsized":"2024-03-06T07:23:32.097589+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.777241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086688+0000","last_change":"2024-03-06T07:23:32.086688+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086688+0000","last_undegraded":"2024-03-06T07:23:32.086688+0000","last_fullsized":"2024-03-06T07:23:32.086688+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098325+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:32.098325+0000","last_peered":"2024-03-06T07:23:32.098325+0000","last_clean":"2024-03-06T07:23:32.098325+0000","last_became_active":"2024-03-06T07:23:32.096097+0000","last_became_peered":"2024-03-06T07:23:32.096097+0000","last_unstale":"2024-03-06T07:23:32.098325+0000","last_undegraded":"2024-03-06T07:23:32.098325+0000","last_fullsized":"2024-03-06T07:23:32.098325+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:11.255144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084439+0000","last_change":"2024-03-06T07:23:32.084439+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084439+0000","last_undegraded":"2024-03-06T07:23:32.084439+0000","last_fullsized":"2024-03-06T07:23:32.084439+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098473+0000","last_change":"2024-03-06T07:23:32.095330+0000","last_active":"2024-03-06T07:23:32.098473+0000","last_peered":"2024-03-06T07:23:32.098473+0000","last_clean":"2024-03-06T07:23:32.098473+0000","last_became_active":"2024-03-06T07:23:32.094988+0000","last_became_peered":"2024-03-06T07:23:32.094988+0000","last_unstale":"2024-03-06T07:23:32.098473+0000","last_undegraded":"2024-03-06T07:23:32.098473+0000","last_fullsized":"2024-03-06T07:23:32.098473+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:40.708193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084193+0000","last_change":"2024-03-06T07:23:32.084193+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084193+0000","last_undegraded":"2024-03-06T07:23:32.084193+0000","last_fullsized":"2024-03-06T07:23:32.084193+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098698+0000","last_change":"2024-03-06T07:23:32.095660+0000","last_active":"2024-03-06T07:23:32.098698+0000","last_peered":"2024-03-06T07:23:32.098698+0000","last_clean":"2024-03-06T07:23:32.098698+0000","last_became_active":"2024-03-06T07:23:32.094528+0000","last_became_peered":"2024-03-06T07:23:32.094528+0000","last_unstale":"2024-03-06T07:23:32.098698+0000","last_undegraded":"2024-03-06T07:23:32.098698+0000","last_fullsized":"2024-03-06T07:23:32.098698+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:44.060570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086601+0000","last_change":"2024-03-06T07:23:32.086601+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086601+0000","last_undegraded":"2024-03-06T07:23:32.086601+0000","last_fullsized":"2024-03-06T07:23:32.086601+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.094285+0000","last_change":"2024-03-06T07:23:32.091549+0000","last_active":"2024-03-06T07:23:32.094285+0000","last_peered":"2024-03-06T07:23:32.094285+0000","last_clean":"2024-03-06T07:23:32.094285+0000","last_became_active":"2024-03-06T07:23:32.091217+0000","last_became_peered":"2024-03-06T07:23:32.091217+0000","last_unstale":"2024-03-06T07:23:32.094285+0000","last_undegraded":"2024-03-06T07:23:32.094285+0000","last_fullsized":"2024-03-06T07:23:32.094285+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:38.093438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081027+0000","last_change":"2024-03-06T07:23:32.081027+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081027+0000","last_undegraded":"2024-03-06T07:23:32.081027+0000","last_fullsized":"2024-03-06T07:23:32.081027+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097934+0000","last_change":"2024-03-06T07:23:32.096137+0000","last_active":"2024-03-06T07:23:32.097934+0000","last_peered":"2024-03-06T07:23:32.097934+0000","last_clean":"2024-03-06T07:23:32.097934+0000","last_became_active":"2024-03-06T07:23:32.095785+0000","last_became_peered":"2024-03-06T07:23:32.095785+0000","last_unstale":"2024-03-06T07:23:32.097934+0000","last_undegraded":"2024-03-06T07:23:32.097934+0000","last_fullsized":"2024-03-06T07:23:32.097934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.777241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083252+0000","last_change":"2024-03-06T07:23:32.083252+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083252+0000","last_undegraded":"2024-03-06T07:23:32.083252+0000","last_fullsized":"2024-03-06T07:23:32.083252+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098782+0000","last_change":"2024-03-06T07:23:32.095556+0000","last_active":"2024-03-06T07:23:32.098782+0000","last_peered":"2024-03-06T07:23:32.098782+0000","last_clean":"2024-03-06T07:23:32.098782+0000","last_became_active":"2024-03-06T07:23:32.094330+0000","last_became_peered":"2024-03-06T07:23:32.094330+0000","last_unstale":"2024-03-06T07:23:32.098782+0000","last_undegraded":"2024-03-06T07:23:32.098782+0000","last_fullsized":"2024-03-06T07:23:32.098782+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.090883+0000","last_change":"2024-03-06T07:23:32.090883+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.090883+0000","last_undegraded":"2024-03-06T07:23:32.090883+0000","last_fullsized":"2024-03-06T07:23:32.090883+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100819+0000","last_change":"2024-03-06T07:23:32.099162+0000","last_active":"2024-03-06T07:23:32.100819+0000","last_peered":"2024-03-06T07:23:32.100819+0000","last_clean":"2024-03-06T07:23:32.100819+0000","last_became_active":"2024-03-06T07:23:32.098923+0000","last_became_peered":"2024-03-06T07:23:32.098923+0000","last_unstale":"2024-03-06T07:23:32.100819+0000","last_undegraded":"2024-03-06T07:23:32.100819+0000","last_fullsized":"2024-03-06T07:23:32.100819+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:24.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084391+0000","last_change":"2024-03-06T07:23:32.084391+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084391+0000","last_undegraded":"2024-03-06T07:23:32.084391+0000","last_fullsized":"2024-03-06T07:23:32.084391+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.096792+0000","last_change":"2024-03-06T07:23:32.093466+0000","last_active":"2024-03-06T07:23:32.096792+0000","last_peered":"2024-03-06T07:23:32.096792+0000","last_clean":"2024-03-06T07:23:32.096792+0000","last_became_active":"2024-03-06T07:23:32.093178+0000","last_became_peered":"2024-03-06T07:23:32.093178+0000","last_unstale":"2024-03-06T07:23:32.096792+0000","last_undegraded":"2024-03-06T07:23:32.096792+0000","last_fullsized":"2024-03-06T07:23:32.096792+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:38.144217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083685+0000","last_change":"2024-03-06T07:23:32.083685+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083685+0000","last_undegraded":"2024-03-06T07:23:32.083685+0000","last_fullsized":"2024-03-06T07:23:32.083685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099451+0000","last_change":"2024-03-06T07:23:32.098048+0000","last_active":"2024-03-06T07:23:32.099451+0000","last_peered":"2024-03-06T07:23:32.099451+0000","last_clean":"2024-03-06T07:23:32.099451+0000","last_became_active":"2024-03-06T07:23:32.097599+0000","last_became_peered":"2024-03-06T07:23:32.097599+0000","last_unstale":"2024-03-06T07:23:32.099451+0000","last_undegraded":"2024-03-06T07:23:32.099451+0000","last_fullsized":"2024-03-06T07:23:32.099451+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.717543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086396+0000","last_change":"2024-03-06T07:23:32.086396+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086396+0000","last_undegraded":"2024-03-06T07:23:32.086396+0000","last_fullsized":"2024-03-06T07:23:32.086396+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097628+0000","last_change":"2024-03-06T07:23:32.094948+0000","last_active":"2024-03-06T07:23:32.097628+0000","last_peered":"2024-03-06T07:23:32.097628+0000","last_clean":"2024-03-06T07:23:32.097628+0000","last_became_active":"2024-03-06T07:23:32.094671+0000","last_became_peered":"2024-03-06T07:23:32.094671+0000","last_unstale":"2024-03-06T07:23:32.097628+0000","last_undegraded":"2024-03-06T07:23:32.097628+0000","last_fullsized":"2024-03-06T07:23:32.097628+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:26.788560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083723+0000","last_change":"2024-03-06T07:23:32.083723+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083723+0000","last_undegraded":"2024-03-06T07:23:32.083723+0000","last_fullsized":"2024-03-06T07:23:32.083723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099967+0000","last_change":"2024-03-06T07:23:32.097587+0000","last_active":"2024-03-06T07:23:32.099967+0000","last_peered":"2024-03-06T07:23:32.099967+0000","last_clean":"2024-03-06T07:23:32.099967+0000","last_became_active":"2024-03-06T07:23:32.096686+0000","last_became_peered":"2024-03-06T07:23:32.096686+0000","last_unstale":"2024-03-06T07:23:32.099967+0000","last_undegraded":"2024-03-06T07:23:32.099967+0000","last_fullsized":"2024-03-06T07:23:32.099967+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:11.079702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086545+0000","last_change":"2024-03-06T07:23:32.086545+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086545+0000","last_undegraded":"2024-03-06T07:23:32.086545+0000","last_fullsized":"2024-03-06T07:23:32.086545+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097723+0000","last_change":"2024-03-06T07:23:32.095927+0000","last_active":"2024-03-06T07:23:32.097723+0000","last_peered":"2024-03-06T07:23:32.097723+0000","last_clean":"2024-03-06T07:23:32.097723+0000","last_became_active":"2024-03-06T07:23:32.095354+0000","last_became_peered":"2024-03-06T07:23:32.095354+0000","last_unstale":"2024-03-06T07:23:32.097723+0000","last_undegraded":"2024-03-06T07:23:32.097723+0000","last_fullsized":"2024-03-06T07:23:32.097723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:13.768735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085807+0000","last_change":"2024-03-06T07:23:32.085807+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085807+0000","last_undegraded":"2024-03-06T07:23:32.085807+0000","last_fullsized":"2024-03-06T07:23:32.085807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098853+0000","last_change":"2024-03-06T07:23:32.094186+0000","last_active":"2024-03-06T07:23:32.098853+0000","last_peered":"2024-03-06T07:23:32.098853+0000","last_clean":"2024-03-06T07:23:32.098853+0000","last_became_active":"2024-03-06T07:23:32.093861+0000","last_became_peered":"2024-03-06T07:23:32.093861+0000","last_unstale":"2024-03-06T07:23:32.098853+0000","last_undegraded":"2024-03-06T07:23:32.098853+0000","last_fullsized":"2024-03-06T07:23:32.098853+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.750831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088864+0000","last_change":"2024-03-06T07:23:32.088864+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088864+0000","last_undegraded":"2024-03-06T07:23:32.088864+0000","last_fullsized":"2024-03-06T07:23:32.088864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097714+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:32.097714+0000","last_peered":"2024-03-06T07:23:32.097714+0000","last_clean":"2024-03-06T07:23:32.097714+0000","last_became_active":"2024-03-06T07:23:32.094821+0000","last_became_peered":"2024-03-06T07:23:32.094821+0000","last_unstale":"2024-03-06T07:23:32.097714+0000","last_undegraded":"2024-03-06T07:23:32.097714+0000","last_fullsized":"2024-03-06T07:23:32.097714+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:17.223200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.091300+0000","last_change":"2024-03-06T07:23:32.091300+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.091300+0000","last_undegraded":"2024-03-06T07:23:32.091300+0000","last_fullsized":"2024-03-06T07:23:32.091300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.096070+0000","last_change":"2024-03-06T07:23:32.095102+0000","last_active":"2024-03-06T07:23:32.096070+0000","last_peered":"2024-03-06T07:23:32.096070+0000","last_clean":"2024-03-06T07:23:32.096070+0000","last_became_active":"2024-03-06T07:23:32.094829+0000","last_became_peered":"2024-03-06T07:23:32.094829+0000","last_unstale":"2024-03-06T07:23:32.096070+0000","last_undegraded":"2024-03-06T07:23:32.096070+0000","last_fullsized":"2024-03-06T07:23:32.096070+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:34.981739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084715+0000","last_change":"2024-03-06T07:23:32.084715+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084715+0000","last_undegraded":"2024-03-06T07:23:32.084715+0000","last_fullsized":"2024-03-06T07:23:32.084715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.094318+0000","last_change":"2024-03-06T07:23:32.091753+0000","last_active":"2024-03-06T07:23:32.094318+0000","last_peered":"2024-03-06T07:23:32.094318+0000","last_clean":"2024-03-06T07:23:32.094318+0000","last_became_active":"2024-03-06T07:23:32.091465+0000","last_became_peered":"2024-03-06T07:23:32.091465+0000","last_unstale":"2024-03-06T07:23:32.094318+0000","last_undegraded":"2024-03-06T07:23:32.094318+0000","last_fullsized":"2024-03-06T07:23:32.094318+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:57.007860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084094+0000","last_change":"2024-03-06T07:23:32.084094+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084094+0000","last_undegraded":"2024-03-06T07:23:32.084094+0000","last_fullsized":"2024-03-06T07:23:32.084094+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098495+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:32.098495+0000","last_peered":"2024-03-06T07:23:32.098495+0000","last_clean":"2024-03-06T07:23:32.098495+0000","last_became_active":"2024-03-06T07:23:32.096065+0000","last_became_peered":"2024-03-06T07:23:32.096065+0000","last_unstale":"2024-03-06T07:23:32.098495+0000","last_undegraded":"2024-03-06T07:23:32.098495+0000","last_fullsized":"2024-03-06T07:23:32.098495+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:21.231136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088028+0000","last_change":"2024-03-06T07:23:32.088028+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088028+0000","last_undegraded":"2024-03-06T07:23:32.088028+0000","last_fullsized":"2024-03-06T07:23:32.088028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097054+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:32.097054+0000","last_peered":"2024-03-06T07:23:32.097054+0000","last_clean":"2024-03-06T07:23:32.097054+0000","last_became_active":"2024-03-06T07:23:32.094784+0000","last_became_peered":"2024-03-06T07:23:32.094784+0000","last_unstale":"2024-03-06T07:23:32.097054+0000","last_undegraded":"2024-03-06T07:23:32.097054+0000","last_fullsized":"2024-03-06T07:23:32.097054+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:23.944371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.088815+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:23:32.088815+0000","last_peered":"2024-03-06T07:23:32.088815+0000","last_clean":"2024-03-06T07:23:32.088815+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:23:32.088815+0000","last_undegraded":"2024-03-06T07:23:32.088815+0000","last_fullsized":"2024-03-06T07:23:32.088815+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":784678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":200577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":784678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:36.183 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:36.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.314+0000 7fec89b13640 1 Processor -- start 2024-03-06T07:23:36.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.314+0000 7fec89b13640 1 -- start start 2024-03-06T07:23:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec89b13640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fec8415dfe0 0x7fec840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec89b13640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec840cf830 0x7fec840d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec89b13640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec840b3430 con 0x7fec840d27f0 2024-03-06T07:23:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec89b13640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec840d2c50 con 0x7fec8415dfe0 2024-03-06T07:23:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec89b13640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec840b31e0 con 0x7fec840cf830 2024-03-06T07:23:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec837fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fec8415dfe0 0x7fec840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec82ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec840cf830 0x7fec840d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec83fff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fec840d27f0 0x7fec840d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:42462/0 (socket says 172.21.15.77:42462) 2024-03-06T07:23:36.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.315+0000 7fec83fff640 1 -- 172.21.15.77:0/735684202 learned_addr learned my addr 172.21.15.77:0/735684202 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:36.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.316+0000 7fec827fc640 1 -- 172.21.15.77:0/735684202 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860442316 0 0) 0x7fec840b3430 con 0x7fec840d27f0 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.316+0000 7fec827fc640 1 -- 172.21.15.77:0/735684202 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec6c003520 con 0x7fec840d27f0 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.316+0000 7fec837fe640 1 -- 172.21.15.77:0/735684202 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec840cf830 msgr2=0x7fec840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.316+0000 7fec837fe640 1 --2- 172.21.15.77:0/735684202 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec840cf830 0x7fec840d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.316+0000 7fec837fe640 1 -- 172.21.15.77:0/735684202 >> v1:172.21.15.158:6789/0 conn(0x7fec840d27f0 legacy=0x7fec840d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.316+0000 7fec837fe640 1 -- 172.21.15.77:0/735684202 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec840b3710 con 0x7fec8415dfe0 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.316+0000 7fec82ffd640 1 --2- 172.21.15.77:0/735684202 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec840cf830 0x7fec840d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec837fe640 1 --2- 172.21.15.77:0/735684202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec8415dfe0 0x7fec840cf2f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fec740097a0 tx=0x7fec7402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70d47530d563af25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec827fc640 1 -- 172.21.15.77:0/735684202 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74035020 con 0x7fec8415dfe0 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec827fc640 1 -- 172.21.15.77:0/735684202 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec74031830 con 0x7fec8415dfe0 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec827fc640 1 -- 172.21.15.77:0/735684202 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74031be0 con 0x7fec8415dfe0 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec89b13640 1 -- 172.21.15.77:0/735684202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec8415dfe0 msgr2=0x7fec840cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec89b13640 1 --2- 172.21.15.77:0/735684202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec8415dfe0 0x7fec840cf2f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fec740097a0 tx=0x7fec7402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec89b13640 1 -- 172.21.15.77:0/735684202 shutdown_connections 2024-03-06T07:23:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec89b13640 1 --2- 172.21.15.77:0/735684202 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec840cf830 0x7fec840d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec89b13640 1 --2- 172.21.15.77:0/735684202 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec8415dfe0 0x7fec840cf2f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec89b13640 1 -- 172.21.15.77:0/735684202 >> 172.21.15.77:0/735684202 conn(0x7fec840d3cd0 msgr2=0x7fec840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec89b13640 1 -- 172.21.15.77:0/735684202 shutdown_connections 2024-03-06T07:23:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.317+0000 7fec89b13640 1 -- 172.21.15.77:0/735684202 wait complete. 2024-03-06T07:23:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.318+0000 7fec89b13640 1 Processor -- start 2024-03-06T07:23:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.319+0000 7fec89b13640 1 -- start start 2024-03-06T07:23:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.319+0000 7fec89b13640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec840cf830 0x7fec8415ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.319+0000 7fec89b13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec840d27f0 0x7fec840cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.319+0000 7fec89b13640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec8415dfe0 0x7fec840b9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.319+0000 7fec89b13640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840d1c70 con 0x7fec840d27f0 2024-03-06T07:23:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.319+0000 7fec89b13640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840b3d60 con 0x7fec840cf830 2024-03-06T07:23:36.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.319+0000 7fec89b13640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec840d1df0 con 0x7fec8415dfe0 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.319+0000 7fec82ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec840d27f0 0x7fec840cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec82ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec840d27f0 0x7fec840cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33330/0 (socket says 172.21.15.77:33330) 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec82ffd640 1 -- 172.21.15.77:0/4240840539 learned_addr learned my addr 172.21.15.77:0/4240840539 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec837fe640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec840cf830 0x7fec8415ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec83fff640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec8415dfe0 0x7fec840b9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec82ffd640 1 -- 172.21.15.77:0/4240840539 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec8415dfe0 msgr2=0x7fec840b9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec82ffd640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec8415dfe0 0x7fec840b9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec82ffd640 1 -- 172.21.15.77:0/4240840539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec840cf830 msgr2=0x7fec8415ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec82ffd640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec840cf830 0x7fec8415ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec82ffd640 1 -- 172.21.15.77:0/4240840539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec840cc750 con 0x7fec840d27f0 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec82ffd640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec840d27f0 0x7fec840cc050 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fec7800e9b0 tx=0x7fec7800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec6bfff640 1 -- 172.21.15.77:0/4240840539 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78017070 con 0x7fec840d27f0 2024-03-06T07:23:36.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec6bfff640 1 -- 172.21.15.77:0/4240840539 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7800ca00 con 0x7fec840d27f0 2024-03-06T07:23:36.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec6bfff640 1 -- 172.21.15.77:0/4240840539 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7800cdf0 con 0x7fec840d27f0 2024-03-06T07:23:36.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec840cca30 con 0x7fec840d27f0 2024-03-06T07:23:36.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.320+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec840ccff0 con 0x7fec840d27f0 2024-03-06T07:23:36.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.321+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8404f330 con 0x7fec840d27f0 2024-03-06T07:23:36.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.322+0000 7fec6bfff640 1 -- 172.21.15.77:0/4240840539 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec7801a070 con 0x7fec840d27f0 2024-03-06T07:23:36.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.322+0000 7fec6bfff640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.322+0000 7fec6bfff640 1 -- 172.21.15.77:0/4240840539 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec7808a0d0 con 0x7fec840d27f0 2024-03-06T07:23:36.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.322+0000 7fec837fe640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.322+0000 7fec837fe640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7006b4d0 0x7fec7006d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fec74036000 tx=0x7fec7403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.326+0000 7fec6bfff640 1 -- 172.21.15.77:0/4240840539 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec78058910 con 0x7fec840d27f0 2024-03-06T07:23:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.495+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec840cd330 con 0x7fec7006b4d0 2024-03-06T07:23:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.501+0000 7fec6bfff640 1 -- 172.21.15.77:0/4240840539 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246636 (secure 0 0 0) 0x7fec74084380 con 0x7fec7006b4d0 2024-03-06T07:23:36.505 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:23:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7006b4d0 msgr2=0x7fec7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7006b4d0 0x7fec7006d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fec74036000 tx=0x7fec7403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec840d27f0 msgr2=0x7fec840cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec840d27f0 0x7fec840cc050 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fec7800e9b0 tx=0x7fec7800ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 shutdown_connections 2024-03-06T07:23:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec840d27f0 0x7fec840cc050 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec8415dfe0 0x7fec840b9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 --2- 172.21.15.77:0/4240840539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec840cf830 0x7fec8415ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 >> 172.21.15.77:0/4240840539 conn(0x7fec840d3cd0 msgr2=0x7fec840b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 shutdown_connections 2024-03-06T07:23:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.504+0000 7fec89b13640 1 -- 172.21.15.77:0/4240840539 wait complete. 2024-03-06T07:23:36.517 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-03-06T07:23:35.402131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258624,"kb_used_data":5012,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754716096,"statfs":{"total":773094113280,"available":772829282304,"internally_reserved":0,"allocated":5132288,"data_stored":2741483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.328272"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086421+0000","last_change":"2024-03-06T07:23:32.086421+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086421+0000","last_undegraded":"2024-03-06T07:23:32.086421+0000","last_fullsized":"2024-03-06T07:23:32.086421+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085015+0000","last_change":"2024-03-06T07:23:32.085015+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085015+0000","last_undegraded":"2024-03-06T07:23:32.085015+0000","last_fullsized":"2024-03-06T07:23:32.085015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084977+0000","last_change":"2024-03-06T07:23:32.084977+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084977+0000","last_undegraded":"2024-03-06T07:23:32.084977+0000","last_fullsized":"2024-03-06T07:23:32.084977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087731+0000","last_change":"2024-03-06T07:23:32.087731+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087731+0000","last_undegraded":"2024-03-06T07:23:32.087731+0000","last_fullsized":"2024-03-06T07:23:32.087731+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085010+0000","last_change":"2024-03-06T07:23:32.085010+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085010+0000","last_undegraded":"2024-03-06T07:23:32.085010+0000","last_fullsized":"2024-03-06T07:23:32.085010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.092100+0000","last_change":"2024-03-06T07:23:32.092100+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.092100+0000","last_undegraded":"2024-03-06T07:23:32.092100+0000","last_fullsized":"2024-03-06T07:23:32.092100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084934+0000","last_change":"2024-03-06T07:23:32.084934+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084934+0000","last_undegraded":"2024-03-06T07:23:32.084934+0000","last_fullsized":"2024-03-06T07:23:32.084934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088106+0000","last_change":"2024-03-06T07:23:32.088106+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088106+0000","last_undegraded":"2024-03-06T07:23:32.088106+0000","last_fullsized":"2024-03-06T07:23:32.088106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087051+0000","last_change":"2024-03-06T07:23:32.087051+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087051+0000","last_undegraded":"2024-03-06T07:23:32.087051+0000","last_fullsized":"2024-03-06T07:23:32.087051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088416+0000","last_change":"2024-03-06T07:23:32.088416+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088416+0000","last_undegraded":"2024-03-06T07:23:32.088416+0000","last_fullsized":"2024-03-06T07:23:32.088416+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087393+0000","last_change":"2024-03-06T07:23:32.087393+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087393+0000","last_undegraded":"2024-03-06T07:23:32.087393+0000","last_fullsized":"2024-03-06T07:23:32.087393+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086312+0000","last_change":"2024-03-06T07:23:32.086312+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086312+0000","last_undegraded":"2024-03-06T07:23:32.086312+0000","last_fullsized":"2024-03-06T07:23:32.086312+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085227+0000","last_change":"2024-03-06T07:23:32.085227+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085227+0000","last_undegraded":"2024-03-06T07:23:32.085227+0000","last_fullsized":"2024-03-06T07:23:32.085227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087620+0000","last_change":"2024-03-06T07:23:32.087620+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087620+0000","last_undegraded":"2024-03-06T07:23:32.087620+0000","last_fullsized":"2024-03-06T07:23:32.087620+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087300+0000","last_change":"2024-03-06T07:23:32.087300+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087300+0000","last_undegraded":"2024-03-06T07:23:32.087300+0000","last_fullsized":"2024-03-06T07:23:32.087300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082117+0000","last_change":"2024-03-06T07:23:32.082117+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082117+0000","last_undegraded":"2024-03-06T07:23:32.082117+0000","last_fullsized":"2024-03-06T07:23:32.082117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086672+0000","last_change":"2024-03-06T07:23:32.086672+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086672+0000","last_undegraded":"2024-03-06T07:23:32.086672+0000","last_fullsized":"2024-03-06T07:23:32.086672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085949+0000","last_change":"2024-03-06T07:23:32.085949+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085949+0000","last_undegraded":"2024-03-06T07:23:32.085949+0000","last_fullsized":"2024-03-06T07:23:32.085949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082809+0000","last_change":"2024-03-06T07:23:32.082809+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082809+0000","last_undegraded":"2024-03-06T07:23:32.082809+0000","last_fullsized":"2024-03-06T07:23:32.082809+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085204+0000","last_change":"2024-03-06T07:23:32.085204+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085204+0000","last_undegraded":"2024-03-06T07:23:32.085204+0000","last_fullsized":"2024-03-06T07:23:32.085204+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.092218+0000","last_change":"2024-03-06T07:23:32.092218+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.092218+0000","last_undegraded":"2024-03-06T07:23:32.092218+0000","last_fullsized":"2024-03-06T07:23:32.092218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087214+0000","last_change":"2024-03-06T07:23:32.087214+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087214+0000","last_undegraded":"2024-03-06T07:23:32.087214+0000","last_fullsized":"2024-03-06T07:23:32.087214+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081010+0000","last_change":"2024-03-06T07:23:32.081010+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081010+0000","last_undegraded":"2024-03-06T07:23:32.081010+0000","last_fullsized":"2024-03-06T07:23:32.081010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081561+0000","last_change":"2024-03-06T07:23:32.081561+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081561+0000","last_undegraded":"2024-03-06T07:23:32.081561+0000","last_fullsized":"2024-03-06T07:23:32.081561+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084558+0000","last_change":"2024-03-06T07:23:32.084558+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084558+0000","last_undegraded":"2024-03-06T07:23:32.084558+0000","last_fullsized":"2024-03-06T07:23:32.084558+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084689+0000","last_change":"2024-03-06T07:23:32.084689+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084689+0000","last_undegraded":"2024-03-06T07:23:32.084689+0000","last_fullsized":"2024-03-06T07:23:32.084689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081376+0000","last_change":"2024-03-06T07:23:32.081376+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081376+0000","last_undegraded":"2024-03-06T07:23:32.081376+0000","last_fullsized":"2024-03-06T07:23:32.081376+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085933+0000","last_change":"2024-03-06T07:23:32.085933+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085933+0000","last_undegraded":"2024-03-06T07:23:32.085933+0000","last_fullsized":"2024-03-06T07:23:32.085933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083052+0000","last_change":"2024-03-06T07:23:32.083052+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083052+0000","last_undegraded":"2024-03-06T07:23:32.083052+0000","last_fullsized":"2024-03-06T07:23:32.083052+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083219+0000","last_change":"2024-03-06T07:23:32.083219+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083219+0000","last_undegraded":"2024-03-06T07:23:32.083219+0000","last_fullsized":"2024-03-06T07:23:32.083219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086871+0000","last_change":"2024-03-06T07:23:32.086871+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086871+0000","last_undegraded":"2024-03-06T07:23:32.086871+0000","last_fullsized":"2024-03-06T07:23:32.086871+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084351+0000","last_change":"2024-03-06T07:23:32.084351+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084351+0000","last_undegraded":"2024-03-06T07:23:32.084351+0000","last_fullsized":"2024-03-06T07:23:32.084351+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084795+0000","last_change":"2024-03-06T07:23:32.084795+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084795+0000","last_undegraded":"2024-03-06T07:23:32.084795+0000","last_fullsized":"2024-03-06T07:23:32.084795+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098181+0000","last_change":"2024-03-06T07:23:32.096250+0000","last_active":"2024-03-06T07:23:32.098181+0000","last_peered":"2024-03-06T07:23:32.098181+0000","last_clean":"2024-03-06T07:23:32.098181+0000","last_became_active":"2024-03-06T07:23:32.095974+0000","last_became_peered":"2024-03-06T07:23:32.095974+0000","last_unstale":"2024-03-06T07:23:32.098181+0000","last_undegraded":"2024-03-06T07:23:32.098181+0000","last_fullsized":"2024-03-06T07:23:32.098181+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:00.492894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084125+0000","last_change":"2024-03-06T07:23:32.084125+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084125+0000","last_undegraded":"2024-03-06T07:23:32.084125+0000","last_fullsized":"2024-03-06T07:23:32.084125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.095292+0000","last_change":"2024-03-06T07:23:32.091961+0000","last_active":"2024-03-06T07:23:32.095292+0000","last_peered":"2024-03-06T07:23:32.095292+0000","last_clean":"2024-03-06T07:23:32.095292+0000","last_became_active":"2024-03-06T07:23:32.091129+0000","last_became_peered":"2024-03-06T07:23:32.091129+0000","last_unstale":"2024-03-06T07:23:32.095292+0000","last_undegraded":"2024-03-06T07:23:32.095292+0000","last_fullsized":"2024-03-06T07:23:32.095292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.717543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084960+0000","last_change":"2024-03-06T07:23:32.084960+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084960+0000","last_undegraded":"2024-03-06T07:23:32.084960+0000","last_fullsized":"2024-03-06T07:23:32.084960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.095230+0000","last_change":"2024-03-06T07:23:32.091042+0000","last_active":"2024-03-06T07:23:32.095230+0000","last_peered":"2024-03-06T07:23:32.095230+0000","last_clean":"2024-03-06T07:23:32.095230+0000","last_became_active":"2024-03-06T07:23:32.090683+0000","last_became_peered":"2024-03-06T07:23:32.090683+0000","last_unstale":"2024-03-06T07:23:32.095230+0000","last_undegraded":"2024-03-06T07:23:32.095230+0000","last_fullsized":"2024-03-06T07:23:32.095230+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:41.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084970+0000","last_change":"2024-03-06T07:23:32.084970+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084970+0000","last_undegraded":"2024-03-06T07:23:32.084970+0000","last_fullsized":"2024-03-06T07:23:32.084970+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097212+0000","last_change":"2024-03-06T07:23:32.093816+0000","last_active":"2024-03-06T07:23:32.097212+0000","last_peered":"2024-03-06T07:23:32.097212+0000","last_clean":"2024-03-06T07:23:32.097212+0000","last_became_active":"2024-03-06T07:23:32.093538+0000","last_became_peered":"2024-03-06T07:23:32.093538+0000","last_unstale":"2024-03-06T07:23:32.097212+0000","last_undegraded":"2024-03-06T07:23:32.097212+0000","last_fullsized":"2024-03-06T07:23:32.097212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:23.944371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082422+0000","last_change":"2024-03-06T07:23:32.082422+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082422+0000","last_undegraded":"2024-03-06T07:23:32.082422+0000","last_fullsized":"2024-03-06T07:23:32.082422+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100838+0000","last_change":"2024-03-06T07:23:32.099457+0000","last_active":"2024-03-06T07:23:32.100838+0000","last_peered":"2024-03-06T07:23:32.100838+0000","last_clean":"2024-03-06T07:23:32.100838+0000","last_became_active":"2024-03-06T07:23:32.099188+0000","last_became_peered":"2024-03-06T07:23:32.099188+0000","last_unstale":"2024-03-06T07:23:32.100838+0000","last_undegraded":"2024-03-06T07:23:32.100838+0000","last_fullsized":"2024-03-06T07:23:32.100838+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:04.671716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.089376+0000","last_change":"2024-03-06T07:23:32.089376+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.089376+0000","last_undegraded":"2024-03-06T07:23:32.089376+0000","last_fullsized":"2024-03-06T07:23:32.089376+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.092153+0000","last_change":"2024-03-06T07:23:32.090283+0000","last_active":"2024-03-06T07:23:32.092153+0000","last_peered":"2024-03-06T07:23:32.092153+0000","last_clean":"2024-03-06T07:23:32.092153+0000","last_became_active":"2024-03-06T07:23:32.089989+0000","last_became_peered":"2024-03-06T07:23:32.089989+0000","last_unstale":"2024-03-06T07:23:32.092153+0000","last_undegraded":"2024-03-06T07:23:32.092153+0000","last_fullsized":"2024-03-06T07:23:32.092153+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:21.231136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087568+0000","last_change":"2024-03-06T07:23:32.087568+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087568+0000","last_undegraded":"2024-03-06T07:23:32.087568+0000","last_fullsized":"2024-03-06T07:23:32.087568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097985+0000","last_change":"2024-03-06T07:23:32.095613+0000","last_active":"2024-03-06T07:23:32.097985+0000","last_peered":"2024-03-06T07:23:32.097985+0000","last_clean":"2024-03-06T07:23:32.097985+0000","last_became_active":"2024-03-06T07:23:32.095182+0000","last_became_peered":"2024-03-06T07:23:32.095182+0000","last_unstale":"2024-03-06T07:23:32.097985+0000","last_undegraded":"2024-03-06T07:23:32.097985+0000","last_fullsized":"2024-03-06T07:23:32.097985+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:17.223200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082182+0000","last_change":"2024-03-06T07:23:32.082182+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082182+0000","last_undegraded":"2024-03-06T07:23:32.082182+0000","last_fullsized":"2024-03-06T07:23:32.082182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100297+0000","last_change":"2024-03-06T07:23:32.099418+0000","last_active":"2024-03-06T07:23:32.100297+0000","last_peered":"2024-03-06T07:23:32.100297+0000","last_clean":"2024-03-06T07:23:32.100297+0000","last_became_active":"2024-03-06T07:23:32.099121+0000","last_became_peered":"2024-03-06T07:23:32.099121+0000","last_unstale":"2024-03-06T07:23:32.100297+0000","last_undegraded":"2024-03-06T07:23:32.100297+0000","last_fullsized":"2024-03-06T07:23:32.100297+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:34.981739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085484+0000","last_change":"2024-03-06T07:23:32.085484+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085484+0000","last_undegraded":"2024-03-06T07:23:32.085484+0000","last_fullsized":"2024-03-06T07:23:32.085484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.091583+0000","last_change":"2024-03-06T07:23:32.087920+0000","last_active":"2024-03-06T07:23:32.091583+0000","last_peered":"2024-03-06T07:23:32.091583+0000","last_clean":"2024-03-06T07:23:32.091583+0000","last_became_active":"2024-03-06T07:23:32.087584+0000","last_became_peered":"2024-03-06T07:23:32.087584+0000","last_unstale":"2024-03-06T07:23:32.091583+0000","last_undegraded":"2024-03-06T07:23:32.091583+0000","last_fullsized":"2024-03-06T07:23:32.091583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:40.708193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085997+0000","last_change":"2024-03-06T07:23:32.085997+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085997+0000","last_undegraded":"2024-03-06T07:23:32.085997+0000","last_fullsized":"2024-03-06T07:23:32.085997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099448+0000","last_change":"2024-03-06T07:23:32.097214+0000","last_active":"2024-03-06T07:23:32.099448+0000","last_peered":"2024-03-06T07:23:32.099448+0000","last_clean":"2024-03-06T07:23:32.099448+0000","last_became_active":"2024-03-06T07:23:32.096930+0000","last_became_peered":"2024-03-06T07:23:32.096930+0000","last_unstale":"2024-03-06T07:23:32.099448+0000","last_undegraded":"2024-03-06T07:23:32.099448+0000","last_fullsized":"2024-03-06T07:23:32.099448+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:38.093438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087625+0000","last_change":"2024-03-06T07:23:32.087625+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087625+0000","last_undegraded":"2024-03-06T07:23:32.087625+0000","last_fullsized":"2024-03-06T07:23:32.087625+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097744+0000","last_change":"2024-03-06T07:23:32.096162+0000","last_active":"2024-03-06T07:23:32.097744+0000","last_peered":"2024-03-06T07:23:32.097744+0000","last_clean":"2024-03-06T07:23:32.097744+0000","last_became_active":"2024-03-06T07:23:32.095663+0000","last_became_peered":"2024-03-06T07:23:32.095663+0000","last_unstale":"2024-03-06T07:23:32.097744+0000","last_undegraded":"2024-03-06T07:23:32.097744+0000","last_fullsized":"2024-03-06T07:23:32.097744+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086333+0000","last_change":"2024-03-06T07:23:32.086333+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086333+0000","last_undegraded":"2024-03-06T07:23:32.086333+0000","last_fullsized":"2024-03-06T07:23:32.086333+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098226+0000","last_change":"2024-03-06T07:23:32.093863+0000","last_active":"2024-03-06T07:23:32.098226+0000","last_peered":"2024-03-06T07:23:32.098226+0000","last_clean":"2024-03-06T07:23:32.098226+0000","last_became_active":"2024-03-06T07:23:32.093600+0000","last_became_peered":"2024-03-06T07:23:32.093600+0000","last_unstale":"2024-03-06T07:23:32.098226+0000","last_undegraded":"2024-03-06T07:23:32.098226+0000","last_fullsized":"2024-03-06T07:23:32.098226+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:33.913245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081445+0000","last_change":"2024-03-06T07:23:32.081445+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081445+0000","last_undegraded":"2024-03-06T07:23:32.081445+0000","last_fullsized":"2024-03-06T07:23:32.081445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099946+0000","last_change":"2024-03-06T07:23:32.097368+0000","last_active":"2024-03-06T07:23:32.099946+0000","last_peered":"2024-03-06T07:23:32.099946+0000","last_clean":"2024-03-06T07:23:32.099946+0000","last_became_active":"2024-03-06T07:23:32.097100+0000","last_became_peered":"2024-03-06T07:23:32.097100+0000","last_unstale":"2024-03-06T07:23:32.099946+0000","last_undegraded":"2024-03-06T07:23:32.099946+0000","last_fullsized":"2024-03-06T07:23:32.099946+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:56.501180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088639+0000","last_change":"2024-03-06T07:23:32.088639+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088639+0000","last_undegraded":"2024-03-06T07:23:32.088639+0000","last_fullsized":"2024-03-06T07:23:32.088639+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097589+0000","last_change":"2024-03-06T07:23:32.094421+0000","last_active":"2024-03-06T07:23:32.097589+0000","last_peered":"2024-03-06T07:23:32.097589+0000","last_clean":"2024-03-06T07:23:32.097589+0000","last_became_active":"2024-03-06T07:23:32.094161+0000","last_became_peered":"2024-03-06T07:23:32.094161+0000","last_unstale":"2024-03-06T07:23:32.097589+0000","last_undegraded":"2024-03-06T07:23:32.097589+0000","last_fullsized":"2024-03-06T07:23:32.097589+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.777241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086688+0000","last_change":"2024-03-06T07:23:32.086688+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086688+0000","last_undegraded":"2024-03-06T07:23:32.086688+0000","last_fullsized":"2024-03-06T07:23:32.086688+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098325+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:32.098325+0000","last_peered":"2024-03-06T07:23:32.098325+0000","last_clean":"2024-03-06T07:23:32.098325+0000","last_became_active":"2024-03-06T07:23:32.096097+0000","last_became_peered":"2024-03-06T07:23:32.096097+0000","last_unstale":"2024-03-06T07:23:32.098325+0000","last_undegraded":"2024-03-06T07:23:32.098325+0000","last_fullsized":"2024-03-06T07:23:32.098325+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:11.255144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084439+0000","last_change":"2024-03-06T07:23:32.084439+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084439+0000","last_undegraded":"2024-03-06T07:23:32.084439+0000","last_fullsized":"2024-03-06T07:23:32.084439+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098473+0000","last_change":"2024-03-06T07:23:32.095330+0000","last_active":"2024-03-06T07:23:32.098473+0000","last_peered":"2024-03-06T07:23:32.098473+0000","last_clean":"2024-03-06T07:23:32.098473+0000","last_became_active":"2024-03-06T07:23:32.094988+0000","last_became_peered":"2024-03-06T07:23:32.094988+0000","last_unstale":"2024-03-06T07:23:32.098473+0000","last_undegraded":"2024-03-06T07:23:32.098473+0000","last_fullsized":"2024-03-06T07:23:32.098473+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:40.708193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084193+0000","last_change":"2024-03-06T07:23:32.084193+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084193+0000","last_undegraded":"2024-03-06T07:23:32.084193+0000","last_fullsized":"2024-03-06T07:23:32.084193+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098698+0000","last_change":"2024-03-06T07:23:32.095660+0000","last_active":"2024-03-06T07:23:32.098698+0000","last_peered":"2024-03-06T07:23:32.098698+0000","last_clean":"2024-03-06T07:23:32.098698+0000","last_became_active":"2024-03-06T07:23:32.094528+0000","last_became_peered":"2024-03-06T07:23:32.094528+0000","last_unstale":"2024-03-06T07:23:32.098698+0000","last_undegraded":"2024-03-06T07:23:32.098698+0000","last_fullsized":"2024-03-06T07:23:32.098698+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:44.060570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086601+0000","last_change":"2024-03-06T07:23:32.086601+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086601+0000","last_undegraded":"2024-03-06T07:23:32.086601+0000","last_fullsized":"2024-03-06T07:23:32.086601+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.094285+0000","last_change":"2024-03-06T07:23:32.091549+0000","last_active":"2024-03-06T07:23:32.094285+0000","last_peered":"2024-03-06T07:23:32.094285+0000","last_clean":"2024-03-06T07:23:32.094285+0000","last_became_active":"2024-03-06T07:23:32.091217+0000","last_became_peered":"2024-03-06T07:23:32.091217+0000","last_unstale":"2024-03-06T07:23:32.094285+0000","last_undegraded":"2024-03-06T07:23:32.094285+0000","last_fullsized":"2024-03-06T07:23:32.094285+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:38.093438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081027+0000","last_change":"2024-03-06T07:23:32.081027+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081027+0000","last_undegraded":"2024-03-06T07:23:32.081027+0000","last_fullsized":"2024-03-06T07:23:32.081027+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097934+0000","last_change":"2024-03-06T07:23:32.096137+0000","last_active":"2024-03-06T07:23:32.097934+0000","last_peered":"2024-03-06T07:23:32.097934+0000","last_clean":"2024-03-06T07:23:32.097934+0000","last_became_active":"2024-03-06T07:23:32.095785+0000","last_became_peered":"2024-03-06T07:23:32.095785+0000","last_unstale":"2024-03-06T07:23:32.097934+0000","last_undegraded":"2024-03-06T07:23:32.097934+0000","last_fullsized":"2024-03-06T07:23:32.097934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.777241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083252+0000","last_change":"2024-03-06T07:23:32.083252+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083252+0000","last_undegraded":"2024-03-06T07:23:32.083252+0000","last_fullsized":"2024-03-06T07:23:32.083252+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098782+0000","last_change":"2024-03-06T07:23:32.095556+0000","last_active":"2024-03-06T07:23:32.098782+0000","last_peered":"2024-03-06T07:23:32.098782+0000","last_clean":"2024-03-06T07:23:32.098782+0000","last_became_active":"2024-03-06T07:23:32.094330+0000","last_became_peered":"2024-03-06T07:23:32.094330+0000","last_unstale":"2024-03-06T07:23:32.098782+0000","last_undegraded":"2024-03-06T07:23:32.098782+0000","last_fullsized":"2024-03-06T07:23:32.098782+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.090883+0000","last_change":"2024-03-06T07:23:32.090883+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.090883+0000","last_undegraded":"2024-03-06T07:23:32.090883+0000","last_fullsized":"2024-03-06T07:23:32.090883+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100819+0000","last_change":"2024-03-06T07:23:32.099162+0000","last_active":"2024-03-06T07:23:32.100819+0000","last_peered":"2024-03-06T07:23:32.100819+0000","last_clean":"2024-03-06T07:23:32.100819+0000","last_became_active":"2024-03-06T07:23:32.098923+0000","last_became_peered":"2024-03-06T07:23:32.098923+0000","last_unstale":"2024-03-06T07:23:32.100819+0000","last_undegraded":"2024-03-06T07:23:32.100819+0000","last_fullsized":"2024-03-06T07:23:32.100819+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:24.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084391+0000","last_change":"2024-03-06T07:23:32.084391+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084391+0000","last_undegraded":"2024-03-06T07:23:32.084391+0000","last_fullsized":"2024-03-06T07:23:32.084391+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.096792+0000","last_change":"2024-03-06T07:23:32.093466+0000","last_active":"2024-03-06T07:23:32.096792+0000","last_peered":"2024-03-06T07:23:32.096792+0000","last_clean":"2024-03-06T07:23:32.096792+0000","last_became_active":"2024-03-06T07:23:32.093178+0000","last_became_peered":"2024-03-06T07:23:32.093178+0000","last_unstale":"2024-03-06T07:23:32.096792+0000","last_undegraded":"2024-03-06T07:23:32.096792+0000","last_fullsized":"2024-03-06T07:23:32.096792+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:38.144217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083685+0000","last_change":"2024-03-06T07:23:32.083685+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083685+0000","last_undegraded":"2024-03-06T07:23:32.083685+0000","last_fullsized":"2024-03-06T07:23:32.083685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099451+0000","last_change":"2024-03-06T07:23:32.098048+0000","last_active":"2024-03-06T07:23:32.099451+0000","last_peered":"2024-03-06T07:23:32.099451+0000","last_clean":"2024-03-06T07:23:32.099451+0000","last_became_active":"2024-03-06T07:23:32.097599+0000","last_became_peered":"2024-03-06T07:23:32.097599+0000","last_unstale":"2024-03-06T07:23:32.099451+0000","last_undegraded":"2024-03-06T07:23:32.099451+0000","last_fullsized":"2024-03-06T07:23:32.099451+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.717543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086396+0000","last_change":"2024-03-06T07:23:32.086396+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086396+0000","last_undegraded":"2024-03-06T07:23:32.086396+0000","last_fullsized":"2024-03-06T07:23:32.086396+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097628+0000","last_change":"2024-03-06T07:23:32.094948+0000","last_active":"2024-03-06T07:23:32.097628+0000","last_peered":"2024-03-06T07:23:32.097628+0000","last_clean":"2024-03-06T07:23:32.097628+0000","last_became_active":"2024-03-06T07:23:32.094671+0000","last_became_peered":"2024-03-06T07:23:32.094671+0000","last_unstale":"2024-03-06T07:23:32.097628+0000","last_undegraded":"2024-03-06T07:23:32.097628+0000","last_fullsized":"2024-03-06T07:23:32.097628+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:26.788560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083723+0000","last_change":"2024-03-06T07:23:32.083723+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083723+0000","last_undegraded":"2024-03-06T07:23:32.083723+0000","last_fullsized":"2024-03-06T07:23:32.083723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099967+0000","last_change":"2024-03-06T07:23:32.097587+0000","last_active":"2024-03-06T07:23:32.099967+0000","last_peered":"2024-03-06T07:23:32.099967+0000","last_clean":"2024-03-06T07:23:32.099967+0000","last_became_active":"2024-03-06T07:23:32.096686+0000","last_became_peered":"2024-03-06T07:23:32.096686+0000","last_unstale":"2024-03-06T07:23:32.099967+0000","last_undegraded":"2024-03-06T07:23:32.099967+0000","last_fullsized":"2024-03-06T07:23:32.099967+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:11.079702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086545+0000","last_change":"2024-03-06T07:23:32.086545+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086545+0000","last_undegraded":"2024-03-06T07:23:32.086545+0000","last_fullsized":"2024-03-06T07:23:32.086545+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097723+0000","last_change":"2024-03-06T07:23:32.095927+0000","last_active":"2024-03-06T07:23:32.097723+0000","last_peered":"2024-03-06T07:23:32.097723+0000","last_clean":"2024-03-06T07:23:32.097723+0000","last_became_active":"2024-03-06T07:23:32.095354+0000","last_became_peered":"2024-03-06T07:23:32.095354+0000","last_unstale":"2024-03-06T07:23:32.097723+0000","last_undegraded":"2024-03-06T07:23:32.097723+0000","last_fullsized":"2024-03-06T07:23:32.097723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:13.768735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085807+0000","last_change":"2024-03-06T07:23:32.085807+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085807+0000","last_undegraded":"2024-03-06T07:23:32.085807+0000","last_fullsized":"2024-03-06T07:23:32.085807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098853+0000","last_change":"2024-03-06T07:23:32.094186+0000","last_active":"2024-03-06T07:23:32.098853+0000","last_peered":"2024-03-06T07:23:32.098853+0000","last_clean":"2024-03-06T07:23:32.098853+0000","last_became_active":"2024-03-06T07:23:32.093861+0000","last_became_peered":"2024-03-06T07:23:32.093861+0000","last_unstale":"2024-03-06T07:23:32.098853+0000","last_undegraded":"2024-03-06T07:23:32.098853+0000","last_fullsized":"2024-03-06T07:23:32.098853+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.750831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088864+0000","last_change":"2024-03-06T07:23:32.088864+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088864+0000","last_undegraded":"2024-03-06T07:23:32.088864+0000","last_fullsized":"2024-03-06T07:23:32.088864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097714+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:32.097714+0000","last_peered":"2024-03-06T07:23:32.097714+0000","last_clean":"2024-03-06T07:23:32.097714+0000","last_became_active":"2024-03-06T07:23:32.094821+0000","last_became_peered":"2024-03-06T07:23:32.094821+0000","last_unstale":"2024-03-06T07:23:32.097714+0000","last_undegraded":"2024-03-06T07:23:32.097714+0000","last_fullsized":"2024-03-06T07:23:32.097714+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:17.223200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.091300+0000","last_change":"2024-03-06T07:23:32.091300+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.091300+0000","last_undegraded":"2024-03-06T07:23:32.091300+0000","last_fullsized":"2024-03-06T07:23:32.091300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.096070+0000","last_change":"2024-03-06T07:23:32.095102+0000","last_active":"2024-03-06T07:23:32.096070+0000","last_peered":"2024-03-06T07:23:32.096070+0000","last_clean":"2024-03-06T07:23:32.096070+0000","last_became_active":"2024-03-06T07:23:32.094829+0000","last_became_peered":"2024-03-06T07:23:32.094829+0000","last_unstale":"2024-03-06T07:23:32.096070+0000","last_undegraded":"2024-03-06T07:23:32.096070+0000","last_fullsized":"2024-03-06T07:23:32.096070+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:34.981739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084715+0000","last_change":"2024-03-06T07:23:32.084715+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084715+0000","last_undegraded":"2024-03-06T07:23:32.084715+0000","last_fullsized":"2024-03-06T07:23:32.084715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.094318+0000","last_change":"2024-03-06T07:23:32.091753+0000","last_active":"2024-03-06T07:23:32.094318+0000","last_peered":"2024-03-06T07:23:32.094318+0000","last_clean":"2024-03-06T07:23:32.094318+0000","last_became_active":"2024-03-06T07:23:32.091465+0000","last_became_peered":"2024-03-06T07:23:32.091465+0000","last_unstale":"2024-03-06T07:23:32.094318+0000","last_undegraded":"2024-03-06T07:23:32.094318+0000","last_fullsized":"2024-03-06T07:23:32.094318+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:57.007860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084094+0000","last_change":"2024-03-06T07:23:32.084094+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084094+0000","last_undegraded":"2024-03-06T07:23:32.084094+0000","last_fullsized":"2024-03-06T07:23:32.084094+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098495+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:32.098495+0000","last_peered":"2024-03-06T07:23:32.098495+0000","last_clean":"2024-03-06T07:23:32.098495+0000","last_became_active":"2024-03-06T07:23:32.096065+0000","last_became_peered":"2024-03-06T07:23:32.096065+0000","last_unstale":"2024-03-06T07:23:32.098495+0000","last_undegraded":"2024-03-06T07:23:32.098495+0000","last_fullsized":"2024-03-06T07:23:32.098495+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:21.231136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088028+0000","last_change":"2024-03-06T07:23:32.088028+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088028+0000","last_undegraded":"2024-03-06T07:23:32.088028+0000","last_fullsized":"2024-03-06T07:23:32.088028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097054+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:32.097054+0000","last_peered":"2024-03-06T07:23:32.097054+0000","last_clean":"2024-03-06T07:23:32.097054+0000","last_became_active":"2024-03-06T07:23:32.094784+0000","last_became_peered":"2024-03-06T07:23:32.094784+0000","last_unstale":"2024-03-06T07:23:32.097054+0000","last_undegraded":"2024-03-06T07:23:32.097054+0000","last_fullsized":"2024-03-06T07:23:32.097054+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:23.944371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.088815+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:23:32.088815+0000","last_peered":"2024-03-06T07:23:32.088815+0000","last_clean":"2024-03-06T07:23:32.088815+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:23:32.088815+0000","last_undegraded":"2024-03-06T07:23:32.088815+0000","last_fullsized":"2024-03-06T07:23:32.088815+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":784678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":200577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":784678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:36.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.657+0000 7f2328b3e640 1 Processor -- start 2024-03-06T07:23:36.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.657+0000 7f2328b3e640 1 -- start start 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2328b3e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f232415dfe0 0x7f23240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2328b3e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23240d29c0 0x7f23240b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2328b3e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23240b3430 con 0x7f23240cf830 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2328b3e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23240b31e0 con 0x7f23240d29c0 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2328b3e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23240b3d60 con 0x7f232415dfe0 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2322d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23240d29c0 0x7f23240b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2321d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f23240cf830 0x7f23240d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56278/0 (socket says 172.21.15.77:56278) 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2321d9b640 1 -- 172.21.15.77:0/3663742222 learned_addr learned my addr 172.21.15.77:0/3663742222 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f232259c640 1 --2- 172.21.15.77:0/3663742222 >> v2:172.21.15.158:3300/0 conn(0x7f232415dfe0 0x7f23240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2322d9d640 1 -- 172.21.15.77:0/3663742222 >> v2:172.21.15.158:3300/0 conn(0x7f232415dfe0 msgr2=0x7f23240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2322d9d640 1 --2- 172.21.15.77:0/3663742222 >> v2:172.21.15.158:3300/0 conn(0x7f232415dfe0 0x7f23240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.658+0000 7f2322d9d640 1 -- 172.21.15.77:0/3663742222 >> v1:172.21.15.77:6789/0 conn(0x7f23240cf830 legacy=0x7f23240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.659+0000 7f2322d9d640 1 -- 172.21.15.77:0/3663742222 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240b3710 con 0x7f23240d29c0 2024-03-06T07:23:36.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.659+0000 7f232159a640 1 -- 172.21.15.77:0/3663742222 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145434377 0 0) 0x7f23240b3430 con 0x7f23240cf830 2024-03-06T07:23:36.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.659+0000 7f232259c640 1 --2- 172.21.15.77:0/3663742222 >> v2:172.21.15.158:3300/0 conn(0x7f232415dfe0 0x7f23240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:36.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2322d9d640 1 --2- 172.21.15.77:0/3663742222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240d29c0 0x7f23240b41f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2324067990 tx=0x7f231402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15b7c82152b46b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f232159a640 1 -- 172.21.15.77:0/3663742222 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314035020 con 0x7f23240d29c0 2024-03-06T07:23:36.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f232159a640 1 -- 172.21.15.77:0/3663742222 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2314031830 con 0x7f23240d29c0 2024-03-06T07:23:36.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f232159a640 1 -- 172.21.15.77:0/3663742222 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314031c00 con 0x7f23240d29c0 2024-03-06T07:23:36.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2328b3e640 1 -- 172.21.15.77:0/3663742222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240d29c0 msgr2=0x7f23240b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2328b3e640 1 --2- 172.21.15.77:0/3663742222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240d29c0 0x7f23240b41f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2324067990 tx=0x7f231402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2328b3e640 1 -- 172.21.15.77:0/3663742222 shutdown_connections 2024-03-06T07:23:36.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2328b3e640 1 --2- 172.21.15.77:0/3663742222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240d29c0 0x7f23240b41f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2328b3e640 1 --2- 172.21.15.77:0/3663742222 >> v2:172.21.15.158:3300/0 conn(0x7f232415dfe0 0x7f23240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2328b3e640 1 -- 172.21.15.77:0/3663742222 >> 172.21.15.77:0/3663742222 conn(0x7f23240d3cd0 msgr2=0x7f23240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2328b3e640 1 -- 172.21.15.77:0/3663742222 shutdown_connections 2024-03-06T07:23:36.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.660+0000 7f2328b3e640 1 -- 172.21.15.77:0/3663742222 wait complete. 2024-03-06T07:23:36.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f2328b3e640 1 Processor -- start 2024-03-06T07:23:36.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f2328b3e640 1 -- start start 2024-03-06T07:23:36.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f2328b3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240cf830 0x7f23241610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f2328b3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23240d29c0 0x7f23240cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f2328b3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f232415dfe0 0x7f232415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f2328b3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23240d1df0 con 0x7f23240cf830 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f2328b3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23240d1c70 con 0x7f23240d29c0 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f2328b3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232415e3c0 con 0x7f232415dfe0 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.662+0000 7f232259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240cf830 0x7f23241610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f232259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240cf830 0x7f23241610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33352/0 (socket says 172.21.15.77:33352) 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f232259c640 1 -- 172.21.15.77:0/1203074079 learned_addr learned my addr 172.21.15.77:0/1203074079 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f2321d9b640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23240d29c0 0x7f23240cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f232259c640 1 -- 172.21.15.77:0/1203074079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f232415dfe0 msgr2=0x7f232415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f232259c640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f232415dfe0 0x7f232415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f232259c640 1 -- 172.21.15.77:0/1203074079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23240d29c0 msgr2=0x7f23240cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f232259c640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23240d29c0 0x7f23240cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f232259c640 1 -- 172.21.15.77:0/1203074079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240cbf00 con 0x7f23240cf830 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f232259c640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240cf830 0x7f23241610c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f231c00ce00 tx=0x7f231c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f2312ffd640 1 -- 172.21.15.77:0/1203074079 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c00f890 con 0x7f23240cf830 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f2312ffd640 1 -- 172.21.15.77:0/1203074079 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c00fa30 con 0x7f23240cf830 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23240cc1e0 con 0x7f23240cf830 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f2312ffd640 1 -- 172.21.15.77:0/1203074079 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c00fe40 con 0x7f23240cf830 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.663+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23240cc7a0 con 0x7f23240cf830 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.664+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232404f330 con 0x7f23240cf830 2024-03-06T07:23:36.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.665+0000 7f2312ffd640 1 -- 172.21.15.77:0/1203074079 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f231c018400 con 0x7f23240cf830 2024-03-06T07:23:36.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.665+0000 7f2312ffd640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f230406b4d0 0x7f230406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.665+0000 7f2321d9b640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f230406b4d0 0x7f230406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.665+0000 7f2312ffd640 1 -- 172.21.15.77:0/1203074079 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f231c08b0e0 con 0x7f23240cf830 2024-03-06T07:23:36.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.666+0000 7f2321d9b640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f230406b4d0 0x7f230406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f23240cdf10 tx=0x7f2318009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.670+0000 7f2312ffd640 1 -- 172.21.15.77:0/1203074079 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f231c014030 con 0x7f23240cf830 2024-03-06T07:23:36.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.845+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23241e8b20 con 0x7f230406b4d0 2024-03-06T07:23:36.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.851+0000 7f2312ffd640 1 -- 172.21.15.77:0/1203074079 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246636 (secure 0 0 0) 0x7f23241e8b20 con 0x7f230406b4d0 2024-03-06T07:23:36.855 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:36.856 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:23:36.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.853+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f230406b4d0 msgr2=0x7f230406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.853+0000 7f2328b3e640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f230406b4d0 0x7f230406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f23240cdf10 tx=0x7f2318009040 comp rx=0 tx=0).stop 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.853+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240cf830 msgr2=0x7f23241610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.853+0000 7f2328b3e640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240cf830 0x7f23241610c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f231c00ce00 tx=0x7f231c00a650 comp rx=0 tx=0).stop 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.854+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 shutdown_connections 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.854+0000 7f2328b3e640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f232415dfe0 0x7f232415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.854+0000 7f2328b3e640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23240d29c0 0x7f23240cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.854+0000 7f2328b3e640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f230406b4d0 0x7f230406d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.854+0000 7f2328b3e640 1 --2- 172.21.15.77:0/1203074079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23240cf830 0x7f23241610c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.854+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 >> 172.21.15.77:0/1203074079 conn(0x7f23240d3cd0 msgr2=0x7f23240cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.854+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 shutdown_connections 2024-03-06T07:23:36.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:36.854+0000 7f2328b3e640 1 -- 172.21.15.77:0/1203074079 wait complete. 2024-03-06T07:23:36.870 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-03-06T07:23:35.402131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258624,"kb_used_data":5012,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754716096,"statfs":{"total":773094113280,"available":772829282304,"internally_reserved":0,"allocated":5132288,"data_stored":2741483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.328272"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086421+0000","last_change":"2024-03-06T07:23:32.086421+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086421+0000","last_undegraded":"2024-03-06T07:23:32.086421+0000","last_fullsized":"2024-03-06T07:23:32.086421+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085015+0000","last_change":"2024-03-06T07:23:32.085015+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085015+0000","last_undegraded":"2024-03-06T07:23:32.085015+0000","last_fullsized":"2024-03-06T07:23:32.085015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084977+0000","last_change":"2024-03-06T07:23:32.084977+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084977+0000","last_undegraded":"2024-03-06T07:23:32.084977+0000","last_fullsized":"2024-03-06T07:23:32.084977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087731+0000","last_change":"2024-03-06T07:23:32.087731+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087731+0000","last_undegraded":"2024-03-06T07:23:32.087731+0000","last_fullsized":"2024-03-06T07:23:32.087731+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085010+0000","last_change":"2024-03-06T07:23:32.085010+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085010+0000","last_undegraded":"2024-03-06T07:23:32.085010+0000","last_fullsized":"2024-03-06T07:23:32.085010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.092100+0000","last_change":"2024-03-06T07:23:32.092100+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.092100+0000","last_undegraded":"2024-03-06T07:23:32.092100+0000","last_fullsized":"2024-03-06T07:23:32.092100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084934+0000","last_change":"2024-03-06T07:23:32.084934+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084934+0000","last_undegraded":"2024-03-06T07:23:32.084934+0000","last_fullsized":"2024-03-06T07:23:32.084934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088106+0000","last_change":"2024-03-06T07:23:32.088106+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088106+0000","last_undegraded":"2024-03-06T07:23:32.088106+0000","last_fullsized":"2024-03-06T07:23:32.088106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087051+0000","last_change":"2024-03-06T07:23:32.087051+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087051+0000","last_undegraded":"2024-03-06T07:23:32.087051+0000","last_fullsized":"2024-03-06T07:23:32.087051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088416+0000","last_change":"2024-03-06T07:23:32.088416+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088416+0000","last_undegraded":"2024-03-06T07:23:32.088416+0000","last_fullsized":"2024-03-06T07:23:32.088416+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087393+0000","last_change":"2024-03-06T07:23:32.087393+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087393+0000","last_undegraded":"2024-03-06T07:23:32.087393+0000","last_fullsized":"2024-03-06T07:23:32.087393+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086312+0000","last_change":"2024-03-06T07:23:32.086312+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086312+0000","last_undegraded":"2024-03-06T07:23:32.086312+0000","last_fullsized":"2024-03-06T07:23:32.086312+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085227+0000","last_change":"2024-03-06T07:23:32.085227+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085227+0000","last_undegraded":"2024-03-06T07:23:32.085227+0000","last_fullsized":"2024-03-06T07:23:32.085227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087620+0000","last_change":"2024-03-06T07:23:32.087620+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087620+0000","last_undegraded":"2024-03-06T07:23:32.087620+0000","last_fullsized":"2024-03-06T07:23:32.087620+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087300+0000","last_change":"2024-03-06T07:23:32.087300+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087300+0000","last_undegraded":"2024-03-06T07:23:32.087300+0000","last_fullsized":"2024-03-06T07:23:32.087300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082117+0000","last_change":"2024-03-06T07:23:32.082117+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082117+0000","last_undegraded":"2024-03-06T07:23:32.082117+0000","last_fullsized":"2024-03-06T07:23:32.082117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086672+0000","last_change":"2024-03-06T07:23:32.086672+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086672+0000","last_undegraded":"2024-03-06T07:23:32.086672+0000","last_fullsized":"2024-03-06T07:23:32.086672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085949+0000","last_change":"2024-03-06T07:23:32.085949+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085949+0000","last_undegraded":"2024-03-06T07:23:32.085949+0000","last_fullsized":"2024-03-06T07:23:32.085949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082809+0000","last_change":"2024-03-06T07:23:32.082809+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082809+0000","last_undegraded":"2024-03-06T07:23:32.082809+0000","last_fullsized":"2024-03-06T07:23:32.082809+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085204+0000","last_change":"2024-03-06T07:23:32.085204+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085204+0000","last_undegraded":"2024-03-06T07:23:32.085204+0000","last_fullsized":"2024-03-06T07:23:32.085204+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.092218+0000","last_change":"2024-03-06T07:23:32.092218+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.092218+0000","last_undegraded":"2024-03-06T07:23:32.092218+0000","last_fullsized":"2024-03-06T07:23:32.092218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087214+0000","last_change":"2024-03-06T07:23:32.087214+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087214+0000","last_undegraded":"2024-03-06T07:23:32.087214+0000","last_fullsized":"2024-03-06T07:23:32.087214+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081010+0000","last_change":"2024-03-06T07:23:32.081010+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081010+0000","last_undegraded":"2024-03-06T07:23:32.081010+0000","last_fullsized":"2024-03-06T07:23:32.081010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081561+0000","last_change":"2024-03-06T07:23:32.081561+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081561+0000","last_undegraded":"2024-03-06T07:23:32.081561+0000","last_fullsized":"2024-03-06T07:23:32.081561+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084558+0000","last_change":"2024-03-06T07:23:32.084558+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084558+0000","last_undegraded":"2024-03-06T07:23:32.084558+0000","last_fullsized":"2024-03-06T07:23:32.084558+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084689+0000","last_change":"2024-03-06T07:23:32.084689+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084689+0000","last_undegraded":"2024-03-06T07:23:32.084689+0000","last_fullsized":"2024-03-06T07:23:32.084689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081376+0000","last_change":"2024-03-06T07:23:32.081376+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081376+0000","last_undegraded":"2024-03-06T07:23:32.081376+0000","last_fullsized":"2024-03-06T07:23:32.081376+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085933+0000","last_change":"2024-03-06T07:23:32.085933+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085933+0000","last_undegraded":"2024-03-06T07:23:32.085933+0000","last_fullsized":"2024-03-06T07:23:32.085933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083052+0000","last_change":"2024-03-06T07:23:32.083052+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083052+0000","last_undegraded":"2024-03-06T07:23:32.083052+0000","last_fullsized":"2024-03-06T07:23:32.083052+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083219+0000","last_change":"2024-03-06T07:23:32.083219+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083219+0000","last_undegraded":"2024-03-06T07:23:32.083219+0000","last_fullsized":"2024-03-06T07:23:32.083219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086871+0000","last_change":"2024-03-06T07:23:32.086871+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086871+0000","last_undegraded":"2024-03-06T07:23:32.086871+0000","last_fullsized":"2024-03-06T07:23:32.086871+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084351+0000","last_change":"2024-03-06T07:23:32.084351+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084351+0000","last_undegraded":"2024-03-06T07:23:32.084351+0000","last_fullsized":"2024-03-06T07:23:32.084351+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084795+0000","last_change":"2024-03-06T07:23:32.084795+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084795+0000","last_undegraded":"2024-03-06T07:23:32.084795+0000","last_fullsized":"2024-03-06T07:23:32.084795+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098181+0000","last_change":"2024-03-06T07:23:32.096250+0000","last_active":"2024-03-06T07:23:32.098181+0000","last_peered":"2024-03-06T07:23:32.098181+0000","last_clean":"2024-03-06T07:23:32.098181+0000","last_became_active":"2024-03-06T07:23:32.095974+0000","last_became_peered":"2024-03-06T07:23:32.095974+0000","last_unstale":"2024-03-06T07:23:32.098181+0000","last_undegraded":"2024-03-06T07:23:32.098181+0000","last_fullsized":"2024-03-06T07:23:32.098181+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:00.492894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084125+0000","last_change":"2024-03-06T07:23:32.084125+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084125+0000","last_undegraded":"2024-03-06T07:23:32.084125+0000","last_fullsized":"2024-03-06T07:23:32.084125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.095292+0000","last_change":"2024-03-06T07:23:32.091961+0000","last_active":"2024-03-06T07:23:32.095292+0000","last_peered":"2024-03-06T07:23:32.095292+0000","last_clean":"2024-03-06T07:23:32.095292+0000","last_became_active":"2024-03-06T07:23:32.091129+0000","last_became_peered":"2024-03-06T07:23:32.091129+0000","last_unstale":"2024-03-06T07:23:32.095292+0000","last_undegraded":"2024-03-06T07:23:32.095292+0000","last_fullsized":"2024-03-06T07:23:32.095292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.717543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084960+0000","last_change":"2024-03-06T07:23:32.084960+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084960+0000","last_undegraded":"2024-03-06T07:23:32.084960+0000","last_fullsized":"2024-03-06T07:23:32.084960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.095230+0000","last_change":"2024-03-06T07:23:32.091042+0000","last_active":"2024-03-06T07:23:32.095230+0000","last_peered":"2024-03-06T07:23:32.095230+0000","last_clean":"2024-03-06T07:23:32.095230+0000","last_became_active":"2024-03-06T07:23:32.090683+0000","last_became_peered":"2024-03-06T07:23:32.090683+0000","last_unstale":"2024-03-06T07:23:32.095230+0000","last_undegraded":"2024-03-06T07:23:32.095230+0000","last_fullsized":"2024-03-06T07:23:32.095230+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:41.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084970+0000","last_change":"2024-03-06T07:23:32.084970+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084970+0000","last_undegraded":"2024-03-06T07:23:32.084970+0000","last_fullsized":"2024-03-06T07:23:32.084970+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097212+0000","last_change":"2024-03-06T07:23:32.093816+0000","last_active":"2024-03-06T07:23:32.097212+0000","last_peered":"2024-03-06T07:23:32.097212+0000","last_clean":"2024-03-06T07:23:32.097212+0000","last_became_active":"2024-03-06T07:23:32.093538+0000","last_became_peered":"2024-03-06T07:23:32.093538+0000","last_unstale":"2024-03-06T07:23:32.097212+0000","last_undegraded":"2024-03-06T07:23:32.097212+0000","last_fullsized":"2024-03-06T07:23:32.097212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:23.944371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082422+0000","last_change":"2024-03-06T07:23:32.082422+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082422+0000","last_undegraded":"2024-03-06T07:23:32.082422+0000","last_fullsized":"2024-03-06T07:23:32.082422+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100838+0000","last_change":"2024-03-06T07:23:32.099457+0000","last_active":"2024-03-06T07:23:32.100838+0000","last_peered":"2024-03-06T07:23:32.100838+0000","last_clean":"2024-03-06T07:23:32.100838+0000","last_became_active":"2024-03-06T07:23:32.099188+0000","last_became_peered":"2024-03-06T07:23:32.099188+0000","last_unstale":"2024-03-06T07:23:32.100838+0000","last_undegraded":"2024-03-06T07:23:32.100838+0000","last_fullsized":"2024-03-06T07:23:32.100838+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:04.671716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.089376+0000","last_change":"2024-03-06T07:23:32.089376+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.089376+0000","last_undegraded":"2024-03-06T07:23:32.089376+0000","last_fullsized":"2024-03-06T07:23:32.089376+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.092153+0000","last_change":"2024-03-06T07:23:32.090283+0000","last_active":"2024-03-06T07:23:32.092153+0000","last_peered":"2024-03-06T07:23:32.092153+0000","last_clean":"2024-03-06T07:23:32.092153+0000","last_became_active":"2024-03-06T07:23:32.089989+0000","last_became_peered":"2024-03-06T07:23:32.089989+0000","last_unstale":"2024-03-06T07:23:32.092153+0000","last_undegraded":"2024-03-06T07:23:32.092153+0000","last_fullsized":"2024-03-06T07:23:32.092153+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:21.231136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087568+0000","last_change":"2024-03-06T07:23:32.087568+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087568+0000","last_undegraded":"2024-03-06T07:23:32.087568+0000","last_fullsized":"2024-03-06T07:23:32.087568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097985+0000","last_change":"2024-03-06T07:23:32.095613+0000","last_active":"2024-03-06T07:23:32.097985+0000","last_peered":"2024-03-06T07:23:32.097985+0000","last_clean":"2024-03-06T07:23:32.097985+0000","last_became_active":"2024-03-06T07:23:32.095182+0000","last_became_peered":"2024-03-06T07:23:32.095182+0000","last_unstale":"2024-03-06T07:23:32.097985+0000","last_undegraded":"2024-03-06T07:23:32.097985+0000","last_fullsized":"2024-03-06T07:23:32.097985+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:17.223200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082182+0000","last_change":"2024-03-06T07:23:32.082182+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082182+0000","last_undegraded":"2024-03-06T07:23:32.082182+0000","last_fullsized":"2024-03-06T07:23:32.082182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100297+0000","last_change":"2024-03-06T07:23:32.099418+0000","last_active":"2024-03-06T07:23:32.100297+0000","last_peered":"2024-03-06T07:23:32.100297+0000","last_clean":"2024-03-06T07:23:32.100297+0000","last_became_active":"2024-03-06T07:23:32.099121+0000","last_became_peered":"2024-03-06T07:23:32.099121+0000","last_unstale":"2024-03-06T07:23:32.100297+0000","last_undegraded":"2024-03-06T07:23:32.100297+0000","last_fullsized":"2024-03-06T07:23:32.100297+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:34.981739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085484+0000","last_change":"2024-03-06T07:23:32.085484+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085484+0000","last_undegraded":"2024-03-06T07:23:32.085484+0000","last_fullsized":"2024-03-06T07:23:32.085484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.091583+0000","last_change":"2024-03-06T07:23:32.087920+0000","last_active":"2024-03-06T07:23:32.091583+0000","last_peered":"2024-03-06T07:23:32.091583+0000","last_clean":"2024-03-06T07:23:32.091583+0000","last_became_active":"2024-03-06T07:23:32.087584+0000","last_became_peered":"2024-03-06T07:23:32.087584+0000","last_unstale":"2024-03-06T07:23:32.091583+0000","last_undegraded":"2024-03-06T07:23:32.091583+0000","last_fullsized":"2024-03-06T07:23:32.091583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:40.708193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085997+0000","last_change":"2024-03-06T07:23:32.085997+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085997+0000","last_undegraded":"2024-03-06T07:23:32.085997+0000","last_fullsized":"2024-03-06T07:23:32.085997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099448+0000","last_change":"2024-03-06T07:23:32.097214+0000","last_active":"2024-03-06T07:23:32.099448+0000","last_peered":"2024-03-06T07:23:32.099448+0000","last_clean":"2024-03-06T07:23:32.099448+0000","last_became_active":"2024-03-06T07:23:32.096930+0000","last_became_peered":"2024-03-06T07:23:32.096930+0000","last_unstale":"2024-03-06T07:23:32.099448+0000","last_undegraded":"2024-03-06T07:23:32.099448+0000","last_fullsized":"2024-03-06T07:23:32.099448+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:38.093438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087625+0000","last_change":"2024-03-06T07:23:32.087625+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087625+0000","last_undegraded":"2024-03-06T07:23:32.087625+0000","last_fullsized":"2024-03-06T07:23:32.087625+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097744+0000","last_change":"2024-03-06T07:23:32.096162+0000","last_active":"2024-03-06T07:23:32.097744+0000","last_peered":"2024-03-06T07:23:32.097744+0000","last_clean":"2024-03-06T07:23:32.097744+0000","last_became_active":"2024-03-06T07:23:32.095663+0000","last_became_peered":"2024-03-06T07:23:32.095663+0000","last_unstale":"2024-03-06T07:23:32.097744+0000","last_undegraded":"2024-03-06T07:23:32.097744+0000","last_fullsized":"2024-03-06T07:23:32.097744+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086333+0000","last_change":"2024-03-06T07:23:32.086333+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086333+0000","last_undegraded":"2024-03-06T07:23:32.086333+0000","last_fullsized":"2024-03-06T07:23:32.086333+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098226+0000","last_change":"2024-03-06T07:23:32.093863+0000","last_active":"2024-03-06T07:23:32.098226+0000","last_peered":"2024-03-06T07:23:32.098226+0000","last_clean":"2024-03-06T07:23:32.098226+0000","last_became_active":"2024-03-06T07:23:32.093600+0000","last_became_peered":"2024-03-06T07:23:32.093600+0000","last_unstale":"2024-03-06T07:23:32.098226+0000","last_undegraded":"2024-03-06T07:23:32.098226+0000","last_fullsized":"2024-03-06T07:23:32.098226+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:33.913245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081445+0000","last_change":"2024-03-06T07:23:32.081445+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081445+0000","last_undegraded":"2024-03-06T07:23:32.081445+0000","last_fullsized":"2024-03-06T07:23:32.081445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099946+0000","last_change":"2024-03-06T07:23:32.097368+0000","last_active":"2024-03-06T07:23:32.099946+0000","last_peered":"2024-03-06T07:23:32.099946+0000","last_clean":"2024-03-06T07:23:32.099946+0000","last_became_active":"2024-03-06T07:23:32.097100+0000","last_became_peered":"2024-03-06T07:23:32.097100+0000","last_unstale":"2024-03-06T07:23:32.099946+0000","last_undegraded":"2024-03-06T07:23:32.099946+0000","last_fullsized":"2024-03-06T07:23:32.099946+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:56.501180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088639+0000","last_change":"2024-03-06T07:23:32.088639+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088639+0000","last_undegraded":"2024-03-06T07:23:32.088639+0000","last_fullsized":"2024-03-06T07:23:32.088639+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097589+0000","last_change":"2024-03-06T07:23:32.094421+0000","last_active":"2024-03-06T07:23:32.097589+0000","last_peered":"2024-03-06T07:23:32.097589+0000","last_clean":"2024-03-06T07:23:32.097589+0000","last_became_active":"2024-03-06T07:23:32.094161+0000","last_became_peered":"2024-03-06T07:23:32.094161+0000","last_unstale":"2024-03-06T07:23:32.097589+0000","last_undegraded":"2024-03-06T07:23:32.097589+0000","last_fullsized":"2024-03-06T07:23:32.097589+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.777241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086688+0000","last_change":"2024-03-06T07:23:32.086688+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086688+0000","last_undegraded":"2024-03-06T07:23:32.086688+0000","last_fullsized":"2024-03-06T07:23:32.086688+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098325+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:32.098325+0000","last_peered":"2024-03-06T07:23:32.098325+0000","last_clean":"2024-03-06T07:23:32.098325+0000","last_became_active":"2024-03-06T07:23:32.096097+0000","last_became_peered":"2024-03-06T07:23:32.096097+0000","last_unstale":"2024-03-06T07:23:32.098325+0000","last_undegraded":"2024-03-06T07:23:32.098325+0000","last_fullsized":"2024-03-06T07:23:32.098325+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:11.255144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084439+0000","last_change":"2024-03-06T07:23:32.084439+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084439+0000","last_undegraded":"2024-03-06T07:23:32.084439+0000","last_fullsized":"2024-03-06T07:23:32.084439+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098473+0000","last_change":"2024-03-06T07:23:32.095330+0000","last_active":"2024-03-06T07:23:32.098473+0000","last_peered":"2024-03-06T07:23:32.098473+0000","last_clean":"2024-03-06T07:23:32.098473+0000","last_became_active":"2024-03-06T07:23:32.094988+0000","last_became_peered":"2024-03-06T07:23:32.094988+0000","last_unstale":"2024-03-06T07:23:32.098473+0000","last_undegraded":"2024-03-06T07:23:32.098473+0000","last_fullsized":"2024-03-06T07:23:32.098473+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:40.708193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084193+0000","last_change":"2024-03-06T07:23:32.084193+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084193+0000","last_undegraded":"2024-03-06T07:23:32.084193+0000","last_fullsized":"2024-03-06T07:23:32.084193+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098698+0000","last_change":"2024-03-06T07:23:32.095660+0000","last_active":"2024-03-06T07:23:32.098698+0000","last_peered":"2024-03-06T07:23:32.098698+0000","last_clean":"2024-03-06T07:23:32.098698+0000","last_became_active":"2024-03-06T07:23:32.094528+0000","last_became_peered":"2024-03-06T07:23:32.094528+0000","last_unstale":"2024-03-06T07:23:32.098698+0000","last_undegraded":"2024-03-06T07:23:32.098698+0000","last_fullsized":"2024-03-06T07:23:32.098698+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:44.060570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086601+0000","last_change":"2024-03-06T07:23:32.086601+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086601+0000","last_undegraded":"2024-03-06T07:23:32.086601+0000","last_fullsized":"2024-03-06T07:23:32.086601+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.094285+0000","last_change":"2024-03-06T07:23:32.091549+0000","last_active":"2024-03-06T07:23:32.094285+0000","last_peered":"2024-03-06T07:23:32.094285+0000","last_clean":"2024-03-06T07:23:32.094285+0000","last_became_active":"2024-03-06T07:23:32.091217+0000","last_became_peered":"2024-03-06T07:23:32.091217+0000","last_unstale":"2024-03-06T07:23:32.094285+0000","last_undegraded":"2024-03-06T07:23:32.094285+0000","last_fullsized":"2024-03-06T07:23:32.094285+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:38.093438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081027+0000","last_change":"2024-03-06T07:23:32.081027+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081027+0000","last_undegraded":"2024-03-06T07:23:32.081027+0000","last_fullsized":"2024-03-06T07:23:32.081027+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097934+0000","last_change":"2024-03-06T07:23:32.096137+0000","last_active":"2024-03-06T07:23:32.097934+0000","last_peered":"2024-03-06T07:23:32.097934+0000","last_clean":"2024-03-06T07:23:32.097934+0000","last_became_active":"2024-03-06T07:23:32.095785+0000","last_became_peered":"2024-03-06T07:23:32.095785+0000","last_unstale":"2024-03-06T07:23:32.097934+0000","last_undegraded":"2024-03-06T07:23:32.097934+0000","last_fullsized":"2024-03-06T07:23:32.097934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.777241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083252+0000","last_change":"2024-03-06T07:23:32.083252+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083252+0000","last_undegraded":"2024-03-06T07:23:32.083252+0000","last_fullsized":"2024-03-06T07:23:32.083252+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098782+0000","last_change":"2024-03-06T07:23:32.095556+0000","last_active":"2024-03-06T07:23:32.098782+0000","last_peered":"2024-03-06T07:23:32.098782+0000","last_clean":"2024-03-06T07:23:32.098782+0000","last_became_active":"2024-03-06T07:23:32.094330+0000","last_became_peered":"2024-03-06T07:23:32.094330+0000","last_unstale":"2024-03-06T07:23:32.098782+0000","last_undegraded":"2024-03-06T07:23:32.098782+0000","last_fullsized":"2024-03-06T07:23:32.098782+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.090883+0000","last_change":"2024-03-06T07:23:32.090883+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.090883+0000","last_undegraded":"2024-03-06T07:23:32.090883+0000","last_fullsized":"2024-03-06T07:23:32.090883+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100819+0000","last_change":"2024-03-06T07:23:32.099162+0000","last_active":"2024-03-06T07:23:32.100819+0000","last_peered":"2024-03-06T07:23:32.100819+0000","last_clean":"2024-03-06T07:23:32.100819+0000","last_became_active":"2024-03-06T07:23:32.098923+0000","last_became_peered":"2024-03-06T07:23:32.098923+0000","last_unstale":"2024-03-06T07:23:32.100819+0000","last_undegraded":"2024-03-06T07:23:32.100819+0000","last_fullsized":"2024-03-06T07:23:32.100819+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:24.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084391+0000","last_change":"2024-03-06T07:23:32.084391+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084391+0000","last_undegraded":"2024-03-06T07:23:32.084391+0000","last_fullsized":"2024-03-06T07:23:32.084391+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.096792+0000","last_change":"2024-03-06T07:23:32.093466+0000","last_active":"2024-03-06T07:23:32.096792+0000","last_peered":"2024-03-06T07:23:32.096792+0000","last_clean":"2024-03-06T07:23:32.096792+0000","last_became_active":"2024-03-06T07:23:32.093178+0000","last_became_peered":"2024-03-06T07:23:32.093178+0000","last_unstale":"2024-03-06T07:23:32.096792+0000","last_undegraded":"2024-03-06T07:23:32.096792+0000","last_fullsized":"2024-03-06T07:23:32.096792+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:38.144217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083685+0000","last_change":"2024-03-06T07:23:32.083685+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083685+0000","last_undegraded":"2024-03-06T07:23:32.083685+0000","last_fullsized":"2024-03-06T07:23:32.083685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099451+0000","last_change":"2024-03-06T07:23:32.098048+0000","last_active":"2024-03-06T07:23:32.099451+0000","last_peered":"2024-03-06T07:23:32.099451+0000","last_clean":"2024-03-06T07:23:32.099451+0000","last_became_active":"2024-03-06T07:23:32.097599+0000","last_became_peered":"2024-03-06T07:23:32.097599+0000","last_unstale":"2024-03-06T07:23:32.099451+0000","last_undegraded":"2024-03-06T07:23:32.099451+0000","last_fullsized":"2024-03-06T07:23:32.099451+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.717543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086396+0000","last_change":"2024-03-06T07:23:32.086396+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086396+0000","last_undegraded":"2024-03-06T07:23:32.086396+0000","last_fullsized":"2024-03-06T07:23:32.086396+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097628+0000","last_change":"2024-03-06T07:23:32.094948+0000","last_active":"2024-03-06T07:23:32.097628+0000","last_peered":"2024-03-06T07:23:32.097628+0000","last_clean":"2024-03-06T07:23:32.097628+0000","last_became_active":"2024-03-06T07:23:32.094671+0000","last_became_peered":"2024-03-06T07:23:32.094671+0000","last_unstale":"2024-03-06T07:23:32.097628+0000","last_undegraded":"2024-03-06T07:23:32.097628+0000","last_fullsized":"2024-03-06T07:23:32.097628+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:26.788560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083723+0000","last_change":"2024-03-06T07:23:32.083723+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083723+0000","last_undegraded":"2024-03-06T07:23:32.083723+0000","last_fullsized":"2024-03-06T07:23:32.083723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099967+0000","last_change":"2024-03-06T07:23:32.097587+0000","last_active":"2024-03-06T07:23:32.099967+0000","last_peered":"2024-03-06T07:23:32.099967+0000","last_clean":"2024-03-06T07:23:32.099967+0000","last_became_active":"2024-03-06T07:23:32.096686+0000","last_became_peered":"2024-03-06T07:23:32.096686+0000","last_unstale":"2024-03-06T07:23:32.099967+0000","last_undegraded":"2024-03-06T07:23:32.099967+0000","last_fullsized":"2024-03-06T07:23:32.099967+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:11.079702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086545+0000","last_change":"2024-03-06T07:23:32.086545+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086545+0000","last_undegraded":"2024-03-06T07:23:32.086545+0000","last_fullsized":"2024-03-06T07:23:32.086545+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097723+0000","last_change":"2024-03-06T07:23:32.095927+0000","last_active":"2024-03-06T07:23:32.097723+0000","last_peered":"2024-03-06T07:23:32.097723+0000","last_clean":"2024-03-06T07:23:32.097723+0000","last_became_active":"2024-03-06T07:23:32.095354+0000","last_became_peered":"2024-03-06T07:23:32.095354+0000","last_unstale":"2024-03-06T07:23:32.097723+0000","last_undegraded":"2024-03-06T07:23:32.097723+0000","last_fullsized":"2024-03-06T07:23:32.097723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:13.768735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085807+0000","last_change":"2024-03-06T07:23:32.085807+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085807+0000","last_undegraded":"2024-03-06T07:23:32.085807+0000","last_fullsized":"2024-03-06T07:23:32.085807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098853+0000","last_change":"2024-03-06T07:23:32.094186+0000","last_active":"2024-03-06T07:23:32.098853+0000","last_peered":"2024-03-06T07:23:32.098853+0000","last_clean":"2024-03-06T07:23:32.098853+0000","last_became_active":"2024-03-06T07:23:32.093861+0000","last_became_peered":"2024-03-06T07:23:32.093861+0000","last_unstale":"2024-03-06T07:23:32.098853+0000","last_undegraded":"2024-03-06T07:23:32.098853+0000","last_fullsized":"2024-03-06T07:23:32.098853+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.750831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088864+0000","last_change":"2024-03-06T07:23:32.088864+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088864+0000","last_undegraded":"2024-03-06T07:23:32.088864+0000","last_fullsized":"2024-03-06T07:23:32.088864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097714+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:32.097714+0000","last_peered":"2024-03-06T07:23:32.097714+0000","last_clean":"2024-03-06T07:23:32.097714+0000","last_became_active":"2024-03-06T07:23:32.094821+0000","last_became_peered":"2024-03-06T07:23:32.094821+0000","last_unstale":"2024-03-06T07:23:32.097714+0000","last_undegraded":"2024-03-06T07:23:32.097714+0000","last_fullsized":"2024-03-06T07:23:32.097714+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:17.223200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.091300+0000","last_change":"2024-03-06T07:23:32.091300+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.091300+0000","last_undegraded":"2024-03-06T07:23:32.091300+0000","last_fullsized":"2024-03-06T07:23:32.091300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.096070+0000","last_change":"2024-03-06T07:23:32.095102+0000","last_active":"2024-03-06T07:23:32.096070+0000","last_peered":"2024-03-06T07:23:32.096070+0000","last_clean":"2024-03-06T07:23:32.096070+0000","last_became_active":"2024-03-06T07:23:32.094829+0000","last_became_peered":"2024-03-06T07:23:32.094829+0000","last_unstale":"2024-03-06T07:23:32.096070+0000","last_undegraded":"2024-03-06T07:23:32.096070+0000","last_fullsized":"2024-03-06T07:23:32.096070+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:34.981739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084715+0000","last_change":"2024-03-06T07:23:32.084715+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084715+0000","last_undegraded":"2024-03-06T07:23:32.084715+0000","last_fullsized":"2024-03-06T07:23:32.084715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.094318+0000","last_change":"2024-03-06T07:23:32.091753+0000","last_active":"2024-03-06T07:23:32.094318+0000","last_peered":"2024-03-06T07:23:32.094318+0000","last_clean":"2024-03-06T07:23:32.094318+0000","last_became_active":"2024-03-06T07:23:32.091465+0000","last_became_peered":"2024-03-06T07:23:32.091465+0000","last_unstale":"2024-03-06T07:23:32.094318+0000","last_undegraded":"2024-03-06T07:23:32.094318+0000","last_fullsized":"2024-03-06T07:23:32.094318+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:57.007860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084094+0000","last_change":"2024-03-06T07:23:32.084094+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084094+0000","last_undegraded":"2024-03-06T07:23:32.084094+0000","last_fullsized":"2024-03-06T07:23:32.084094+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098495+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:32.098495+0000","last_peered":"2024-03-06T07:23:32.098495+0000","last_clean":"2024-03-06T07:23:32.098495+0000","last_became_active":"2024-03-06T07:23:32.096065+0000","last_became_peered":"2024-03-06T07:23:32.096065+0000","last_unstale":"2024-03-06T07:23:32.098495+0000","last_undegraded":"2024-03-06T07:23:32.098495+0000","last_fullsized":"2024-03-06T07:23:32.098495+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:21.231136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088028+0000","last_change":"2024-03-06T07:23:32.088028+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088028+0000","last_undegraded":"2024-03-06T07:23:32.088028+0000","last_fullsized":"2024-03-06T07:23:32.088028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097054+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:32.097054+0000","last_peered":"2024-03-06T07:23:32.097054+0000","last_clean":"2024-03-06T07:23:32.097054+0000","last_became_active":"2024-03-06T07:23:32.094784+0000","last_became_peered":"2024-03-06T07:23:32.094784+0000","last_unstale":"2024-03-06T07:23:32.097054+0000","last_undegraded":"2024-03-06T07:23:32.097054+0000","last_fullsized":"2024-03-06T07:23:32.097054+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:23.944371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.088815+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:23:32.088815+0000","last_peered":"2024-03-06T07:23:32.088815+0000","last_clean":"2024-03-06T07:23:32.088815+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:23:32.088815+0000","last_undegraded":"2024-03-06T07:23:32.088815+0000","last_fullsized":"2024-03-06T07:23:32.088815+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":784678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":200577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":784678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:36.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2024-03-06T07:23:36.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086421+0000', 'last_change': '2024-03-06T07:23:32.086421+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086421+0000', 'last_undegraded': '2024-03-06T07:23:32.086421+0000', 'last_fullsized': '2024-03-06T07:23:32.086421+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:36.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2024-03-06T07:23:36.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085015+0000', 'last_change': '2024-03-06T07:23:32.085015+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085015+0000', 'last_undegraded': '2024-03-06T07:23:32.085015+0000', 'last_fullsized': '2024-03-06T07:23:32.085015+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:36.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2024-03-06T07:23:36.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084977+0000', 'last_change': '2024-03-06T07:23:32.084977+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084977+0000', 'last_undegraded': '2024-03-06T07:23:32.084977+0000', 'last_fullsized': '2024-03-06T07:23:32.084977+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.087731+0000', 'last_change': '2024-03-06T07:23:32.087731+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.087731+0000', 'last_undegraded': '2024-03-06T07:23:32.087731+0000', 'last_fullsized': '2024-03-06T07:23:32.087731+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085010+0000', 'last_change': '2024-03-06T07:23:32.085010+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085010+0000', 'last_undegraded': '2024-03-06T07:23:32.085010+0000', 'last_fullsized': '2024-03-06T07:23:32.085010+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.092100+0000', 'last_change': '2024-03-06T07:23:32.092100+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.092100+0000', 'last_undegraded': '2024-03-06T07:23:32.092100+0000', 'last_fullsized': '2024-03-06T07:23:32.092100+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084934+0000', 'last_change': '2024-03-06T07:23:32.084934+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084934+0000', 'last_undegraded': '2024-03-06T07:23:32.084934+0000', 'last_fullsized': '2024-03-06T07:23:32.084934+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.088106+0000', 'last_change': '2024-03-06T07:23:32.088106+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.088106+0000', 'last_undegraded': '2024-03-06T07:23:32.088106+0000', 'last_fullsized': '2024-03-06T07:23:32.088106+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.087051+0000', 'last_change': '2024-03-06T07:23:32.087051+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.087051+0000', 'last_undegraded': '2024-03-06T07:23:32.087051+0000', 'last_fullsized': '2024-03-06T07:23:32.087051+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.088416+0000', 'last_change': '2024-03-06T07:23:32.088416+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.088416+0000', 'last_undegraded': '2024-03-06T07:23:32.088416+0000', 'last_fullsized': '2024-03-06T07:23:32.088416+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.087393+0000', 'last_change': '2024-03-06T07:23:32.087393+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.087393+0000', 'last_undegraded': '2024-03-06T07:23:32.087393+0000', 'last_fullsized': '2024-03-06T07:23:32.087393+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086312+0000', 'last_change': '2024-03-06T07:23:32.086312+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086312+0000', 'last_undegraded': '2024-03-06T07:23:32.086312+0000', 'last_fullsized': '2024-03-06T07:23:32.086312+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085227+0000', 'last_change': '2024-03-06T07:23:32.085227+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085227+0000', 'last_undegraded': '2024-03-06T07:23:32.085227+0000', 'last_fullsized': '2024-03-06T07:23:32.085227+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2024-03-06T07:23:36.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.087620+0000', 'last_change': '2024-03-06T07:23:32.087620+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.087620+0000', 'last_undegraded': '2024-03-06T07:23:32.087620+0000', 'last_fullsized': '2024-03-06T07:23:32.087620+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.087300+0000', 'last_change': '2024-03-06T07:23:32.087300+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.087300+0000', 'last_undegraded': '2024-03-06T07:23:32.087300+0000', 'last_fullsized': '2024-03-06T07:23:32.087300+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.082117+0000', 'last_change': '2024-03-06T07:23:32.082117+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.082117+0000', 'last_undegraded': '2024-03-06T07:23:32.082117+0000', 'last_fullsized': '2024-03-06T07:23:32.082117+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086672+0000', 'last_change': '2024-03-06T07:23:32.086672+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086672+0000', 'last_undegraded': '2024-03-06T07:23:32.086672+0000', 'last_fullsized': '2024-03-06T07:23:32.086672+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085949+0000', 'last_change': '2024-03-06T07:23:32.085949+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085949+0000', 'last_undegraded': '2024-03-06T07:23:32.085949+0000', 'last_fullsized': '2024-03-06T07:23:32.085949+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2024-03-06T07:23:36.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.082809+0000', 'last_change': '2024-03-06T07:23:32.082809+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.082809+0000', 'last_undegraded': '2024-03-06T07:23:32.082809+0000', 'last_fullsized': '2024-03-06T07:23:32.082809+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085204+0000', 'last_change': '2024-03-06T07:23:32.085204+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085204+0000', 'last_undegraded': '2024-03-06T07:23:32.085204+0000', 'last_fullsized': '2024-03-06T07:23:32.085204+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.092218+0000', 'last_change': '2024-03-06T07:23:32.092218+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.092218+0000', 'last_undegraded': '2024-03-06T07:23:32.092218+0000', 'last_fullsized': '2024-03-06T07:23:32.092218+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.087214+0000', 'last_change': '2024-03-06T07:23:32.087214+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.087214+0000', 'last_undegraded': '2024-03-06T07:23:32.087214+0000', 'last_fullsized': '2024-03-06T07:23:32.087214+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.081010+0000', 'last_change': '2024-03-06T07:23:32.081010+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.081010+0000', 'last_undegraded': '2024-03-06T07:23:32.081010+0000', 'last_fullsized': '2024-03-06T07:23:32.081010+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.081561+0000', 'last_change': '2024-03-06T07:23:32.081561+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.081561+0000', 'last_undegraded': '2024-03-06T07:23:32.081561+0000', 'last_fullsized': '2024-03-06T07:23:32.081561+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084558+0000', 'last_change': '2024-03-06T07:23:32.084558+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084558+0000', 'last_undegraded': '2024-03-06T07:23:32.084558+0000', 'last_fullsized': '2024-03-06T07:23:32.084558+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084689+0000', 'last_change': '2024-03-06T07:23:32.084689+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084689+0000', 'last_undegraded': '2024-03-06T07:23:32.084689+0000', 'last_fullsized': '2024-03-06T07:23:32.084689+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.081376+0000', 'last_change': '2024-03-06T07:23:32.081376+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.081376+0000', 'last_undegraded': '2024-03-06T07:23:32.081376+0000', 'last_fullsized': '2024-03-06T07:23:32.081376+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085933+0000', 'last_change': '2024-03-06T07:23:32.085933+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085933+0000', 'last_undegraded': '2024-03-06T07:23:32.085933+0000', 'last_fullsized': '2024-03-06T07:23:32.085933+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.083052+0000', 'last_change': '2024-03-06T07:23:32.083052+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.083052+0000', 'last_undegraded': '2024-03-06T07:23:32.083052+0000', 'last_fullsized': '2024-03-06T07:23:32.083052+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2024-03-06T07:23:36.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.083219+0000', 'last_change': '2024-03-06T07:23:32.083219+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.083219+0000', 'last_undegraded': '2024-03-06T07:23:32.083219+0000', 'last_fullsized': '2024-03-06T07:23:32.083219+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086871+0000', 'last_change': '2024-03-06T07:23:32.086871+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086871+0000', 'last_undegraded': '2024-03-06T07:23:32.086871+0000', 'last_fullsized': '2024-03-06T07:23:32.086871+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084351+0000', 'last_change': '2024-03-06T07:23:32.084351+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084351+0000', 'last_undegraded': '2024-03-06T07:23:32.084351+0000', 'last_fullsized': '2024-03-06T07:23:32.084351+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084795+0000', 'last_change': '2024-03-06T07:23:32.084795+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084795+0000', 'last_undegraded': '2024-03-06T07:23:32.084795+0000', 'last_fullsized': '2024-03-06T07:23:32.084795+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084125+0000', 'last_change': '2024-03-06T07:23:32.084125+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084125+0000', 'last_undegraded': '2024-03-06T07:23:32.084125+0000', 'last_fullsized': '2024-03-06T07:23:32.084125+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084960+0000', 'last_change': '2024-03-06T07:23:32.084960+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084960+0000', 'last_undegraded': '2024-03-06T07:23:32.084960+0000', 'last_fullsized': '2024-03-06T07:23:32.084960+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084970+0000', 'last_change': '2024-03-06T07:23:32.084970+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084970+0000', 'last_undegraded': '2024-03-06T07:23:32.084970+0000', 'last_fullsized': '2024-03-06T07:23:32.084970+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.082422+0000', 'last_change': '2024-03-06T07:23:32.082422+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.082422+0000', 'last_undegraded': '2024-03-06T07:23:32.082422+0000', 'last_fullsized': '2024-03-06T07:23:32.082422+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.089376+0000', 'last_change': '2024-03-06T07:23:32.089376+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.089376+0000', 'last_undegraded': '2024-03-06T07:23:32.089376+0000', 'last_fullsized': '2024-03-06T07:23:32.089376+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.087568+0000', 'last_change': '2024-03-06T07:23:32.087568+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.087568+0000', 'last_undegraded': '2024-03-06T07:23:32.087568+0000', 'last_fullsized': '2024-03-06T07:23:32.087568+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-03-06T07:23:36.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.082182+0000', 'last_change': '2024-03-06T07:23:32.082182+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.082182+0000', 'last_undegraded': '2024-03-06T07:23:32.082182+0000', 'last_fullsized': '2024-03-06T07:23:32.082182+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085484+0000', 'last_change': '2024-03-06T07:23:32.085484+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085484+0000', 'last_undegraded': '2024-03-06T07:23:32.085484+0000', 'last_fullsized': '2024-03-06T07:23:32.085484+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085997+0000', 'last_change': '2024-03-06T07:23:32.085997+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085997+0000', 'last_undegraded': '2024-03-06T07:23:32.085997+0000', 'last_fullsized': '2024-03-06T07:23:32.085997+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.087625+0000', 'last_change': '2024-03-06T07:23:32.087625+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.087625+0000', 'last_undegraded': '2024-03-06T07:23:32.087625+0000', 'last_fullsized': '2024-03-06T07:23:32.087625+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086333+0000', 'last_change': '2024-03-06T07:23:32.086333+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086333+0000', 'last_undegraded': '2024-03-06T07:23:32.086333+0000', 'last_fullsized': '2024-03-06T07:23:32.086333+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.081445+0000', 'last_change': '2024-03-06T07:23:32.081445+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.081445+0000', 'last_undegraded': '2024-03-06T07:23:32.081445+0000', 'last_fullsized': '2024-03-06T07:23:32.081445+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-03-06T07:23:36.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.088639+0000', 'last_change': '2024-03-06T07:23:32.088639+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.088639+0000', 'last_undegraded': '2024-03-06T07:23:32.088639+0000', 'last_fullsized': '2024-03-06T07:23:32.088639+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086688+0000', 'last_change': '2024-03-06T07:23:32.086688+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086688+0000', 'last_undegraded': '2024-03-06T07:23:32.086688+0000', 'last_fullsized': '2024-03-06T07:23:32.086688+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084439+0000', 'last_change': '2024-03-06T07:23:32.084439+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084439+0000', 'last_undegraded': '2024-03-06T07:23:32.084439+0000', 'last_fullsized': '2024-03-06T07:23:32.084439+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084193+0000', 'last_change': '2024-03-06T07:23:32.084193+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084193+0000', 'last_undegraded': '2024-03-06T07:23:32.084193+0000', 'last_fullsized': '2024-03-06T07:23:32.084193+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086601+0000', 'last_change': '2024-03-06T07:23:32.086601+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086601+0000', 'last_undegraded': '2024-03-06T07:23:32.086601+0000', 'last_fullsized': '2024-03-06T07:23:32.086601+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-03-06T07:23:36.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.081027+0000', 'last_change': '2024-03-06T07:23:32.081027+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.081027+0000', 'last_undegraded': '2024-03-06T07:23:32.081027+0000', 'last_fullsized': '2024-03-06T07:23:32.081027+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.083252+0000', 'last_change': '2024-03-06T07:23:32.083252+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.083252+0000', 'last_undegraded': '2024-03-06T07:23:32.083252+0000', 'last_fullsized': '2024-03-06T07:23:32.083252+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.090883+0000', 'last_change': '2024-03-06T07:23:32.090883+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.090883+0000', 'last_undegraded': '2024-03-06T07:23:32.090883+0000', 'last_fullsized': '2024-03-06T07:23:32.090883+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084391+0000', 'last_change': '2024-03-06T07:23:32.084391+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084391+0000', 'last_undegraded': '2024-03-06T07:23:32.084391+0000', 'last_fullsized': '2024-03-06T07:23:32.084391+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.083685+0000', 'last_change': '2024-03-06T07:23:32.083685+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.083685+0000', 'last_undegraded': '2024-03-06T07:23:32.083685+0000', 'last_fullsized': '2024-03-06T07:23:32.083685+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-03-06T07:23:36.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086396+0000', 'last_change': '2024-03-06T07:23:32.086396+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086396+0000', 'last_undegraded': '2024-03-06T07:23:32.086396+0000', 'last_fullsized': '2024-03-06T07:23:32.086396+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.083723+0000', 'last_change': '2024-03-06T07:23:32.083723+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.083723+0000', 'last_undegraded': '2024-03-06T07:23:32.083723+0000', 'last_fullsized': '2024-03-06T07:23:32.083723+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.086545+0000', 'last_change': '2024-03-06T07:23:32.086545+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.086545+0000', 'last_undegraded': '2024-03-06T07:23:32.086545+0000', 'last_fullsized': '2024-03-06T07:23:32.086545+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.085807+0000', 'last_change': '2024-03-06T07:23:32.085807+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.085807+0000', 'last_undegraded': '2024-03-06T07:23:32.085807+0000', 'last_fullsized': '2024-03-06T07:23:32.085807+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.088864+0000', 'last_change': '2024-03-06T07:23:32.088864+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.088864+0000', 'last_undegraded': '2024-03-06T07:23:32.088864+0000', 'last_fullsized': '2024-03-06T07:23:32.088864+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.091300+0000', 'last_change': '2024-03-06T07:23:32.091300+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.091300+0000', 'last_undegraded': '2024-03-06T07:23:32.091300+0000', 'last_fullsized': '2024-03-06T07:23:32.091300+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:36.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-03-06T07:23:36.886 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084715+0000', 'last_change': '2024-03-06T07:23:32.084715+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084715+0000', 'last_undegraded': '2024-03-06T07:23:32.084715+0000', 'last_fullsized': '2024-03-06T07:23:32.084715+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:36.886 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-03-06T07:23:36.886 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.084094+0000', 'last_change': '2024-03-06T07:23:32.084094+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.084094+0000', 'last_undegraded': '2024-03-06T07:23:32.084094+0000', 'last_fullsized': '2024-03-06T07:23:32.084094+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:36.886 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-03-06T07:23:36.886 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:32.088028+0000', 'last_change': '2024-03-06T07:23:32.088028+0000', 'last_active': '2024-03-06T07:23:32.073823+0000', 'last_peered': '2024-03-06T07:23:32.073823+0000', 'last_clean': '2024-03-06T07:23:32.073823+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:32.088028+0000', 'last_undegraded': '2024-03-06T07:23:32.088028+0000', 'last_fullsized': '2024-03-06T07:23:32.088028+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:32.073823+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:36.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:23:37.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.008+0000 7f53c66af640 1 Processor -- start 2024-03-06T07:23:37.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.008+0000 7f53c66af640 1 -- start start 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53c66af640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53c0162970 0x7f53c00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53c66af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 0x7f53c0162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53c66af640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c00d22c0 con 0x7f53c00b4460 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53c66af640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c0161b00 con 0x7f53c0162970 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53c66af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00cecc0 con 0x7f53c0161cf0 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53bf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53c0162970 0x7f53c00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53bf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53c0162970 0x7f53c00d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33356/0 (socket says 172.21.15.77:33356) 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53bf7fe640 1 -- 172.21.15.77:0/1406168213 learned_addr learned my addr 172.21.15.77:0/1406168213 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53c4c4c640 1 --2- 172.21.15.77:0/1406168213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 0x7f53c0162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53bf7fe640 1 -- 172.21.15.77:0/1406168213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 msgr2=0x7f53c0162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:37.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53bf7fe640 1 --2- 172.21.15.77:0/1406168213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 0x7f53c0162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53bf7fe640 1 -- 172.21.15.77:0/1406168213 >> v1:172.21.15.158:6789/0 conn(0x7f53c00b4460 legacy=0x7f53c00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.009+0000 7f53bf7fe640 1 -- 172.21.15.77:0/1406168213 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00d25a0 con 0x7f53c0162970 2024-03-06T07:23:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.010+0000 7f53bf7fe640 1 --2- 172.21.15.77:0/1406168213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c0162970 0x7f53c00d4800 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009870 tx=0x7f53b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499311800678e111 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.010+0000 7f53beffd640 1 -- 172.21.15.77:0/1406168213 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0034070 con 0x7f53c0162970 2024-03-06T07:23:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.010+0000 7f53beffd640 1 -- 172.21.15.77:0/1406168213 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b0030840 con 0x7f53c0162970 2024-03-06T07:23:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.010+0000 7f53beffd640 1 -- 172.21.15.77:0/1406168213 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0030c10 con 0x7f53c0162970 2024-03-06T07:23:37.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.011+0000 7f53c66af640 1 -- 172.21.15.77:0/1406168213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c0162970 msgr2=0x7f53c00d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.011+0000 7f53c66af640 1 --2- 172.21.15.77:0/1406168213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c0162970 0x7f53c00d4800 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009870 tx=0x7f53b002b980 comp rx=0 tx=0).stop 2024-03-06T07:23:37.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.011+0000 7f53c66af640 1 -- 172.21.15.77:0/1406168213 shutdown_connections 2024-03-06T07:23:37.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.011+0000 7f53c66af640 1 --2- 172.21.15.77:0/1406168213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 0x7f53c0162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.011+0000 7f53c66af640 1 --2- 172.21.15.77:0/1406168213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c0162970 0x7f53c00d4800 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.011+0000 7f53c66af640 1 -- 172.21.15.77:0/1406168213 >> 172.21.15.77:0/1406168213 conn(0x7f53c00d3ce0 msgr2=0x7f53c00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.011+0000 7f53c66af640 1 -- 172.21.15.77:0/1406168213 shutdown_connections 2024-03-06T07:23:37.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.011+0000 7f53c66af640 1 -- 172.21.15.77:0/1406168213 wait complete. 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.012+0000 7f53c66af640 1 Processor -- start 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.012+0000 7f53c66af640 1 -- start start 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53c66af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c00b4460 0x7f53c00b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53c66af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 0x7f53c0158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53c66af640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53c0162970 0x7f53c0158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53c66af640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00ce950 con 0x7f53c00b4460 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53c66af640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00d2b60 con 0x7f53c0162970 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53c66af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00b4840 con 0x7f53c0161cf0 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c00b4460 0x7f53c00b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c00b4460 0x7f53c00b1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33372/0 (socket says 172.21.15.77:33372) 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 -- 172.21.15.77:0/237613006 learned_addr learned my addr 172.21.15.77:0/237613006 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53c4c4c640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53c0162970 0x7f53c0158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bf7fe640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 0x7f53c0158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 -- 172.21.15.77:0/237613006 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 msgr2=0x7f53c0158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 0x7f53c0158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 -- 172.21.15.77:0/237613006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53c0162970 msgr2=0x7f53c0158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53c0162970 0x7f53c0158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 -- 172.21.15.77:0/237613006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00b8ea0 con 0x7f53c00b4460 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bffff640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c00b4460 0x7f53c00b1e10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f53b400c3c0 tx=0x7f53b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bcff9640 1 -- 172.21.15.77:0/237613006 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b40052d0 con 0x7f53c00b4460 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.013+0000 7f53bcff9640 1 -- 172.21.15.77:0/237613006 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b4005490 con 0x7f53c00b4460 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.014+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c00c7810 con 0x7f53c00b4460 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.014+0000 7f53bcff9640 1 -- 172.21.15.77:0/237613006 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4007260 con 0x7f53c00b4460 2024-03-06T07:23:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.014+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c00c7dd0 con 0x7f53c00b4460 2024-03-06T07:23:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.015+0000 7f53bcff9640 1 -- 172.21.15.77:0/237613006 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53b4007400 con 0x7f53c00b4460 2024-03-06T07:23:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.015+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c004f330 con 0x7f53c00b4460 2024-03-06T07:23:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.015+0000 7f53bcff9640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53ac06b4d0 0x7f53ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.015+0000 7f53bcff9640 1 -- 172.21.15.77:0/237613006 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53b4089650 con 0x7f53c00b4460 2024-03-06T07:23:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.015+0000 7f53bf7fe640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53ac06b4d0 0x7f53ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.016+0000 7f53bf7fe640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53ac06b4d0 0x7f53ac06d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f53b0036000 tx=0x7f53b003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.020+0000 7f53bcff9640 1 -- 172.21.15.77:0/237613006 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53b4058560 con 0x7f53c00b4460 2024-03-06T07:23:37.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.292+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f53c006b180 con 0x7f53c00b4460 2024-03-06T07:23:37.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.293+0000 7f53bcff9640 1 -- 172.21.15.77:0/237613006 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1582 (secure 0 0 0) 0x7f53b4057e60 con 0x7f53c00b4460 2024-03-06T07:23:37.295 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:37.295 INFO:teuthology.orchestra.run.smithi077.stdout:{"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":116,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709711,"num_in_osds":8,"osd_in_since":1709709703,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":264830976,"bytes_avail":772829282304,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":19,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":5667}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-06T07:22:53.394118+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.294+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53ac06b4d0 msgr2=0x7f53ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.294+0000 7f53c66af640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53ac06b4d0 0x7f53ac06d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f53b0036000 tx=0x7f53b003b040 comp rx=0 tx=0).stop 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c00b4460 msgr2=0x7f53c00b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c00b4460 0x7f53c00b1e10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f53b400c3c0 tx=0x7f53b400c880 comp rx=0 tx=0).stop 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 shutdown_connections 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53c0162970 0x7f53c0158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53c0161cf0 0x7f53c0158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53ac06b4d0 0x7f53ac06d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 --2- 172.21.15.77:0/237613006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53c00b4460 0x7f53c00b1e10 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 >> 172.21.15.77:0/237613006 conn(0x7f53c00d3ce0 msgr2=0x7f53c00b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 shutdown_connections 2024-03-06T07:23:37.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.295+0000 7f53c66af640 1 -- 172.21.15.77:0/237613006 wait complete. 2024-03-06T07:23:37.311 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:23:37.312 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:37.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.410+0000 7f69760a4640 1 Processor -- start 2024-03-06T07:23:37.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.411+0000 7f69760a4640 1 -- start start 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f69760a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69700d22b0 0x7f69700d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f69760a4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700b3430 con 0x7f69700b41f0 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f69760a4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700b3710 con 0x7f697015dfe0 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f69760a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69700d2c50 con 0x7f69700d22b0 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f69700b41f0 0x7f69700b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56282/0 (socket says 172.21.15.77:56282) 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696ffff640 1 -- 172.21.15.77:0/417276761 learned_addr learned my addr 172.21.15.77:0/417276761 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696effd640 1 --2- 172.21.15.77:0/417276761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69700d22b0 0x7f69700d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696e7fc640 1 -- 172.21.15.77:0/417276761 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025191518 0 0) 0x7f69700b3430 con 0x7f69700b41f0 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696e7fc640 1 -- 172.21.15.77:0/417276761 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f695c003520 con 0x7f69700b41f0 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696e7fc640 1 -- 172.21.15.77:0/417276761 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137634086 0 0) 0x7f69700b3710 con 0x7f697015dfe0 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696e7fc640 1 -- 172.21.15.77:0/417276761 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69700b3430 con 0x7f697015dfe0 2024-03-06T07:23:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696effd640 1 -- 172.21.15.77:0/417276761 >> v1:172.21.15.158:6789/0 conn(0x7f697015dfe0 legacy=0x7f69700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696effd640 1 -- 172.21.15.77:0/417276761 >> v1:172.21.15.77:6789/0 conn(0x7f69700b41f0 legacy=0x7f69700b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696e7fc640 1 -- 172.21.15.77:0/417276761 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1689309147 0 0) 0x7f695c003520 con 0x7f69700b41f0 2024-03-06T07:23:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.412+0000 7f696effd640 1 -- 172.21.15.77:0/417276761 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69700b39f0 con 0x7f69700d22b0 2024-03-06T07:23:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.413+0000 7f696effd640 1 --2- 172.21.15.77:0/417276761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69700d22b0 0x7f69700d2710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f69580097d0 tx=0x7f695802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c117763ed24fcf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.413+0000 7f696e7fc640 1 -- 172.21.15.77:0/417276761 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958037070 con 0x7f69700d22b0 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.413+0000 7f696e7fc640 1 -- 172.21.15.77:0/417276761 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958030510 con 0x7f69700d22b0 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.413+0000 7f696e7fc640 1 -- 172.21.15.77:0/417276761 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968001670 con 0x7f69700d22b0 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.414+0000 7f69760a4640 1 -- 172.21.15.77:0/417276761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69700d22b0 msgr2=0x7f69700d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.414+0000 7f69760a4640 1 --2- 172.21.15.77:0/417276761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69700d22b0 0x7f69700d2710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f69580097d0 tx=0x7f695802b800 comp rx=0 tx=0).stop 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.414+0000 7f69760a4640 1 -- 172.21.15.77:0/417276761 shutdown_connections 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.414+0000 7f69760a4640 1 --2- 172.21.15.77:0/417276761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69700d22b0 0x7f69700d2710 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.414+0000 7f69760a4640 1 -- 172.21.15.77:0/417276761 >> 172.21.15.77:0/417276761 conn(0x7f69700d3cd0 msgr2=0x7f69700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.414+0000 7f69760a4640 1 -- 172.21.15.77:0/417276761 shutdown_connections 2024-03-06T07:23:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.414+0000 7f69760a4640 1 -- 172.21.15.77:0/417276761 wait complete. 2024-03-06T07:23:37.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.415+0000 7f69760a4640 1 Processor -- start 2024-03-06T07:23:37.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.415+0000 7f69760a4640 1 -- start start 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f69760a4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69700b41f0 0x7f69700b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f69760a4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700d22b0 0x7f697015ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f69760a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f697015dfe0 0x7f69700b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f69760a4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69700b3d60 con 0x7f69700b41f0 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f69760a4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69700b31e0 con 0x7f69700d22b0 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f69760a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69700d1c70 con 0x7f697015dfe0 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f696f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69700b41f0 0x7f69700b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f696f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69700b41f0 0x7f69700b6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33376/0 (socket says 172.21.15.77:33376) 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f696f7fe640 1 -- 172.21.15.77:0/1849907631 learned_addr learned my addr 172.21.15.77:0/1849907631 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:37.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f696effd640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700d22b0 0x7f697015ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.416+0000 7f696ffff640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f697015dfe0 0x7f69700b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f696f7fe640 1 -- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f697015dfe0 msgr2=0x7f69700b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f696f7fe640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f697015dfe0 0x7f69700b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f696f7fe640 1 -- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700d22b0 msgr2=0x7f697015ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f696f7fe640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700d22b0 0x7f697015ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f696f7fe640 1 -- 172.21.15.77:0/1849907631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69700b9bc0 con 0x7f69700b41f0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f696ffff640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f697015dfe0 0x7f69700b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f696f7fe640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69700b41f0 0x7f69700b6d00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f696000c7d0 tx=0x7f696000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f6967fff640 1 -- 172.21.15.77:0/1849907631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960003b70 con 0x7f69700b41f0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f6967fff640 1 -- 172.21.15.77:0/1849907631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960003d10 con 0x7f69700b41f0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69700cba20 con 0x7f69700b41f0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f6967fff640 1 -- 172.21.15.77:0/1849907631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000aa90 con 0x7f69700b41f0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.417+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69700cbcc0 con 0x7f69700b41f0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.418+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697004f330 con 0x7f69700b41f0 2024-03-06T07:23:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.418+0000 7f6967fff640 1 -- 172.21.15.77:0/1849907631 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f696000ac30 con 0x7f69700b41f0 2024-03-06T07:23:37.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.419+0000 7f6967fff640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f695006b4d0 0x7f695006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:37.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.419+0000 7f6967fff640 1 -- 172.21.15.77:0/1849907631 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f696008a070 con 0x7f69700b41f0 2024-03-06T07:23:37.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.419+0000 7f696effd640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f695006b4d0 0x7f695006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:37.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.419+0000 7f696effd640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f695006b4d0 0x7f695006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6958034000 tx=0x7f6958008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:37.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.423+0000 7f6967fff640 1 -- 172.21.15.77:0/1849907631 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6960058f80 con 0x7f69700b41f0 2024-03-06T07:23:37.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.591+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f697015d3c0 con 0x7f695006b4d0 2024-03-06T07:23:37.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.597+0000 7f6967fff640 1 -- 172.21.15.77:0/1849907631 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246636 (secure 0 0 0) 0x7f697015d3c0 con 0x7f695006b4d0 2024-03-06T07:23:37.601 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:37.603 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:23:37.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f695006b4d0 msgr2=0x7f695006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f695006b4d0 0x7f695006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6958034000 tx=0x7f6958008030 comp rx=0 tx=0).stop 2024-03-06T07:23:37.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69700b41f0 msgr2=0x7f69700b6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:37.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69700b41f0 0x7f69700b6d00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f696000c7d0 tx=0x7f696000cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:37.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 shutdown_connections 2024-03-06T07:23:37.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f697015dfe0 0x7f69700b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69700d22b0 0x7f697015ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f695006b4d0 0x7f695006d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 --2- 172.21.15.77:0/1849907631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69700b41f0 0x7f69700b6d00 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:37.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 >> 172.21.15.77:0/1849907631 conn(0x7f69700d3cd0 msgr2=0x7f697006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:37.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 shutdown_connections 2024-03-06T07:23:37.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:37.601+0000 7f69760a4640 1 -- 172.21.15.77:0/1849907631 wait complete. 2024-03-06T07:23:37.614 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-03-06T07:23:37.402502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258624,"kb_used_data":5012,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754716096,"statfs":{"total":773094113280,"available":772829282304,"internally_reserved":0,"allocated":5132288,"data_stored":2741483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001071"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086421+0000","last_change":"2024-03-06T07:23:32.086421+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086421+0000","last_undegraded":"2024-03-06T07:23:32.086421+0000","last_fullsized":"2024-03-06T07:23:32.086421+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085015+0000","last_change":"2024-03-06T07:23:32.085015+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085015+0000","last_undegraded":"2024-03-06T07:23:32.085015+0000","last_fullsized":"2024-03-06T07:23:32.085015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084977+0000","last_change":"2024-03-06T07:23:32.084977+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084977+0000","last_undegraded":"2024-03-06T07:23:32.084977+0000","last_fullsized":"2024-03-06T07:23:32.084977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087731+0000","last_change":"2024-03-06T07:23:32.087731+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087731+0000","last_undegraded":"2024-03-06T07:23:32.087731+0000","last_fullsized":"2024-03-06T07:23:32.087731+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085010+0000","last_change":"2024-03-06T07:23:32.085010+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085010+0000","last_undegraded":"2024-03-06T07:23:32.085010+0000","last_fullsized":"2024-03-06T07:23:32.085010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.092100+0000","last_change":"2024-03-06T07:23:32.092100+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.092100+0000","last_undegraded":"2024-03-06T07:23:32.092100+0000","last_fullsized":"2024-03-06T07:23:32.092100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084934+0000","last_change":"2024-03-06T07:23:32.084934+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084934+0000","last_undegraded":"2024-03-06T07:23:32.084934+0000","last_fullsized":"2024-03-06T07:23:32.084934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088106+0000","last_change":"2024-03-06T07:23:32.088106+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088106+0000","last_undegraded":"2024-03-06T07:23:32.088106+0000","last_fullsized":"2024-03-06T07:23:32.088106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087051+0000","last_change":"2024-03-06T07:23:32.087051+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087051+0000","last_undegraded":"2024-03-06T07:23:32.087051+0000","last_fullsized":"2024-03-06T07:23:32.087051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088416+0000","last_change":"2024-03-06T07:23:32.088416+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088416+0000","last_undegraded":"2024-03-06T07:23:32.088416+0000","last_fullsized":"2024-03-06T07:23:32.088416+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087393+0000","last_change":"2024-03-06T07:23:32.087393+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087393+0000","last_undegraded":"2024-03-06T07:23:32.087393+0000","last_fullsized":"2024-03-06T07:23:32.087393+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086312+0000","last_change":"2024-03-06T07:23:32.086312+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086312+0000","last_undegraded":"2024-03-06T07:23:32.086312+0000","last_fullsized":"2024-03-06T07:23:32.086312+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085227+0000","last_change":"2024-03-06T07:23:32.085227+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085227+0000","last_undegraded":"2024-03-06T07:23:32.085227+0000","last_fullsized":"2024-03-06T07:23:32.085227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087620+0000","last_change":"2024-03-06T07:23:32.087620+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087620+0000","last_undegraded":"2024-03-06T07:23:32.087620+0000","last_fullsized":"2024-03-06T07:23:32.087620+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087300+0000","last_change":"2024-03-06T07:23:32.087300+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087300+0000","last_undegraded":"2024-03-06T07:23:32.087300+0000","last_fullsized":"2024-03-06T07:23:32.087300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082117+0000","last_change":"2024-03-06T07:23:32.082117+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082117+0000","last_undegraded":"2024-03-06T07:23:32.082117+0000","last_fullsized":"2024-03-06T07:23:32.082117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086672+0000","last_change":"2024-03-06T07:23:32.086672+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086672+0000","last_undegraded":"2024-03-06T07:23:32.086672+0000","last_fullsized":"2024-03-06T07:23:32.086672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085949+0000","last_change":"2024-03-06T07:23:32.085949+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085949+0000","last_undegraded":"2024-03-06T07:23:32.085949+0000","last_fullsized":"2024-03-06T07:23:32.085949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082809+0000","last_change":"2024-03-06T07:23:32.082809+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082809+0000","last_undegraded":"2024-03-06T07:23:32.082809+0000","last_fullsized":"2024-03-06T07:23:32.082809+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085204+0000","last_change":"2024-03-06T07:23:32.085204+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085204+0000","last_undegraded":"2024-03-06T07:23:32.085204+0000","last_fullsized":"2024-03-06T07:23:32.085204+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.092218+0000","last_change":"2024-03-06T07:23:32.092218+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.092218+0000","last_undegraded":"2024-03-06T07:23:32.092218+0000","last_fullsized":"2024-03-06T07:23:32.092218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087214+0000","last_change":"2024-03-06T07:23:32.087214+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087214+0000","last_undegraded":"2024-03-06T07:23:32.087214+0000","last_fullsized":"2024-03-06T07:23:32.087214+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081010+0000","last_change":"2024-03-06T07:23:32.081010+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081010+0000","last_undegraded":"2024-03-06T07:23:32.081010+0000","last_fullsized":"2024-03-06T07:23:32.081010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081561+0000","last_change":"2024-03-06T07:23:32.081561+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081561+0000","last_undegraded":"2024-03-06T07:23:32.081561+0000","last_fullsized":"2024-03-06T07:23:32.081561+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084558+0000","last_change":"2024-03-06T07:23:32.084558+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084558+0000","last_undegraded":"2024-03-06T07:23:32.084558+0000","last_fullsized":"2024-03-06T07:23:32.084558+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084689+0000","last_change":"2024-03-06T07:23:32.084689+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084689+0000","last_undegraded":"2024-03-06T07:23:32.084689+0000","last_fullsized":"2024-03-06T07:23:32.084689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081376+0000","last_change":"2024-03-06T07:23:32.081376+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081376+0000","last_undegraded":"2024-03-06T07:23:32.081376+0000","last_fullsized":"2024-03-06T07:23:32.081376+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085933+0000","last_change":"2024-03-06T07:23:32.085933+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085933+0000","last_undegraded":"2024-03-06T07:23:32.085933+0000","last_fullsized":"2024-03-06T07:23:32.085933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083052+0000","last_change":"2024-03-06T07:23:32.083052+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083052+0000","last_undegraded":"2024-03-06T07:23:32.083052+0000","last_fullsized":"2024-03-06T07:23:32.083052+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083219+0000","last_change":"2024-03-06T07:23:32.083219+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083219+0000","last_undegraded":"2024-03-06T07:23:32.083219+0000","last_fullsized":"2024-03-06T07:23:32.083219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086871+0000","last_change":"2024-03-06T07:23:32.086871+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086871+0000","last_undegraded":"2024-03-06T07:23:32.086871+0000","last_fullsized":"2024-03-06T07:23:32.086871+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084351+0000","last_change":"2024-03-06T07:23:32.084351+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084351+0000","last_undegraded":"2024-03-06T07:23:32.084351+0000","last_fullsized":"2024-03-06T07:23:32.084351+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084795+0000","last_change":"2024-03-06T07:23:32.084795+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084795+0000","last_undegraded":"2024-03-06T07:23:32.084795+0000","last_fullsized":"2024-03-06T07:23:32.084795+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098181+0000","last_change":"2024-03-06T07:23:32.096250+0000","last_active":"2024-03-06T07:23:32.098181+0000","last_peered":"2024-03-06T07:23:32.098181+0000","last_clean":"2024-03-06T07:23:32.098181+0000","last_became_active":"2024-03-06T07:23:32.095974+0000","last_became_peered":"2024-03-06T07:23:32.095974+0000","last_unstale":"2024-03-06T07:23:32.098181+0000","last_undegraded":"2024-03-06T07:23:32.098181+0000","last_fullsized":"2024-03-06T07:23:32.098181+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:00.492894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084125+0000","last_change":"2024-03-06T07:23:32.084125+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084125+0000","last_undegraded":"2024-03-06T07:23:32.084125+0000","last_fullsized":"2024-03-06T07:23:32.084125+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.095292+0000","last_change":"2024-03-06T07:23:32.091961+0000","last_active":"2024-03-06T07:23:32.095292+0000","last_peered":"2024-03-06T07:23:32.095292+0000","last_clean":"2024-03-06T07:23:32.095292+0000","last_became_active":"2024-03-06T07:23:32.091129+0000","last_became_peered":"2024-03-06T07:23:32.091129+0000","last_unstale":"2024-03-06T07:23:32.095292+0000","last_undegraded":"2024-03-06T07:23:32.095292+0000","last_fullsized":"2024-03-06T07:23:32.095292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.717543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084960+0000","last_change":"2024-03-06T07:23:32.084960+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084960+0000","last_undegraded":"2024-03-06T07:23:32.084960+0000","last_fullsized":"2024-03-06T07:23:32.084960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.095230+0000","last_change":"2024-03-06T07:23:32.091042+0000","last_active":"2024-03-06T07:23:32.095230+0000","last_peered":"2024-03-06T07:23:32.095230+0000","last_clean":"2024-03-06T07:23:32.095230+0000","last_became_active":"2024-03-06T07:23:32.090683+0000","last_became_peered":"2024-03-06T07:23:32.090683+0000","last_unstale":"2024-03-06T07:23:32.095230+0000","last_undegraded":"2024-03-06T07:23:32.095230+0000","last_fullsized":"2024-03-06T07:23:32.095230+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:41.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084970+0000","last_change":"2024-03-06T07:23:32.084970+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084970+0000","last_undegraded":"2024-03-06T07:23:32.084970+0000","last_fullsized":"2024-03-06T07:23:32.084970+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097212+0000","last_change":"2024-03-06T07:23:32.093816+0000","last_active":"2024-03-06T07:23:32.097212+0000","last_peered":"2024-03-06T07:23:32.097212+0000","last_clean":"2024-03-06T07:23:32.097212+0000","last_became_active":"2024-03-06T07:23:32.093538+0000","last_became_peered":"2024-03-06T07:23:32.093538+0000","last_unstale":"2024-03-06T07:23:32.097212+0000","last_undegraded":"2024-03-06T07:23:32.097212+0000","last_fullsized":"2024-03-06T07:23:32.097212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:23.944371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082422+0000","last_change":"2024-03-06T07:23:32.082422+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082422+0000","last_undegraded":"2024-03-06T07:23:32.082422+0000","last_fullsized":"2024-03-06T07:23:32.082422+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100838+0000","last_change":"2024-03-06T07:23:32.099457+0000","last_active":"2024-03-06T07:23:32.100838+0000","last_peered":"2024-03-06T07:23:32.100838+0000","last_clean":"2024-03-06T07:23:32.100838+0000","last_became_active":"2024-03-06T07:23:32.099188+0000","last_became_peered":"2024-03-06T07:23:32.099188+0000","last_unstale":"2024-03-06T07:23:32.100838+0000","last_undegraded":"2024-03-06T07:23:32.100838+0000","last_fullsized":"2024-03-06T07:23:32.100838+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:04.671716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.089376+0000","last_change":"2024-03-06T07:23:32.089376+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.089376+0000","last_undegraded":"2024-03-06T07:23:32.089376+0000","last_fullsized":"2024-03-06T07:23:32.089376+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.092153+0000","last_change":"2024-03-06T07:23:32.090283+0000","last_active":"2024-03-06T07:23:32.092153+0000","last_peered":"2024-03-06T07:23:32.092153+0000","last_clean":"2024-03-06T07:23:32.092153+0000","last_became_active":"2024-03-06T07:23:32.089989+0000","last_became_peered":"2024-03-06T07:23:32.089989+0000","last_unstale":"2024-03-06T07:23:32.092153+0000","last_undegraded":"2024-03-06T07:23:32.092153+0000","last_fullsized":"2024-03-06T07:23:32.092153+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:21.231136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087568+0000","last_change":"2024-03-06T07:23:32.087568+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087568+0000","last_undegraded":"2024-03-06T07:23:32.087568+0000","last_fullsized":"2024-03-06T07:23:32.087568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097985+0000","last_change":"2024-03-06T07:23:32.095613+0000","last_active":"2024-03-06T07:23:32.097985+0000","last_peered":"2024-03-06T07:23:32.097985+0000","last_clean":"2024-03-06T07:23:32.097985+0000","last_became_active":"2024-03-06T07:23:32.095182+0000","last_became_peered":"2024-03-06T07:23:32.095182+0000","last_unstale":"2024-03-06T07:23:32.097985+0000","last_undegraded":"2024-03-06T07:23:32.097985+0000","last_fullsized":"2024-03-06T07:23:32.097985+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:17.223200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.082182+0000","last_change":"2024-03-06T07:23:32.082182+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.082182+0000","last_undegraded":"2024-03-06T07:23:32.082182+0000","last_fullsized":"2024-03-06T07:23:32.082182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100297+0000","last_change":"2024-03-06T07:23:32.099418+0000","last_active":"2024-03-06T07:23:32.100297+0000","last_peered":"2024-03-06T07:23:32.100297+0000","last_clean":"2024-03-06T07:23:32.100297+0000","last_became_active":"2024-03-06T07:23:32.099121+0000","last_became_peered":"2024-03-06T07:23:32.099121+0000","last_unstale":"2024-03-06T07:23:32.100297+0000","last_undegraded":"2024-03-06T07:23:32.100297+0000","last_fullsized":"2024-03-06T07:23:32.100297+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:34.981739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085484+0000","last_change":"2024-03-06T07:23:32.085484+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085484+0000","last_undegraded":"2024-03-06T07:23:32.085484+0000","last_fullsized":"2024-03-06T07:23:32.085484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.091583+0000","last_change":"2024-03-06T07:23:32.087920+0000","last_active":"2024-03-06T07:23:32.091583+0000","last_peered":"2024-03-06T07:23:32.091583+0000","last_clean":"2024-03-06T07:23:32.091583+0000","last_became_active":"2024-03-06T07:23:32.087584+0000","last_became_peered":"2024-03-06T07:23:32.087584+0000","last_unstale":"2024-03-06T07:23:32.091583+0000","last_undegraded":"2024-03-06T07:23:32.091583+0000","last_fullsized":"2024-03-06T07:23:32.091583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:40.708193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085997+0000","last_change":"2024-03-06T07:23:32.085997+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085997+0000","last_undegraded":"2024-03-06T07:23:32.085997+0000","last_fullsized":"2024-03-06T07:23:32.085997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099448+0000","last_change":"2024-03-06T07:23:32.097214+0000","last_active":"2024-03-06T07:23:32.099448+0000","last_peered":"2024-03-06T07:23:32.099448+0000","last_clean":"2024-03-06T07:23:32.099448+0000","last_became_active":"2024-03-06T07:23:32.096930+0000","last_became_peered":"2024-03-06T07:23:32.096930+0000","last_unstale":"2024-03-06T07:23:32.099448+0000","last_undegraded":"2024-03-06T07:23:32.099448+0000","last_fullsized":"2024-03-06T07:23:32.099448+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:38.093438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.087625+0000","last_change":"2024-03-06T07:23:32.087625+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.087625+0000","last_undegraded":"2024-03-06T07:23:32.087625+0000","last_fullsized":"2024-03-06T07:23:32.087625+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097744+0000","last_change":"2024-03-06T07:23:32.096162+0000","last_active":"2024-03-06T07:23:32.097744+0000","last_peered":"2024-03-06T07:23:32.097744+0000","last_clean":"2024-03-06T07:23:32.097744+0000","last_became_active":"2024-03-06T07:23:32.095663+0000","last_became_peered":"2024-03-06T07:23:32.095663+0000","last_unstale":"2024-03-06T07:23:32.097744+0000","last_undegraded":"2024-03-06T07:23:32.097744+0000","last_fullsized":"2024-03-06T07:23:32.097744+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086333+0000","last_change":"2024-03-06T07:23:32.086333+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086333+0000","last_undegraded":"2024-03-06T07:23:32.086333+0000","last_fullsized":"2024-03-06T07:23:32.086333+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098226+0000","last_change":"2024-03-06T07:23:32.093863+0000","last_active":"2024-03-06T07:23:32.098226+0000","last_peered":"2024-03-06T07:23:32.098226+0000","last_clean":"2024-03-06T07:23:32.098226+0000","last_became_active":"2024-03-06T07:23:32.093600+0000","last_became_peered":"2024-03-06T07:23:32.093600+0000","last_unstale":"2024-03-06T07:23:32.098226+0000","last_undegraded":"2024-03-06T07:23:32.098226+0000","last_fullsized":"2024-03-06T07:23:32.098226+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:33.913245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081445+0000","last_change":"2024-03-06T07:23:32.081445+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081445+0000","last_undegraded":"2024-03-06T07:23:32.081445+0000","last_fullsized":"2024-03-06T07:23:32.081445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099946+0000","last_change":"2024-03-06T07:23:32.097368+0000","last_active":"2024-03-06T07:23:32.099946+0000","last_peered":"2024-03-06T07:23:32.099946+0000","last_clean":"2024-03-06T07:23:32.099946+0000","last_became_active":"2024-03-06T07:23:32.097100+0000","last_became_peered":"2024-03-06T07:23:32.097100+0000","last_unstale":"2024-03-06T07:23:32.099946+0000","last_undegraded":"2024-03-06T07:23:32.099946+0000","last_fullsized":"2024-03-06T07:23:32.099946+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:56.501180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088639+0000","last_change":"2024-03-06T07:23:32.088639+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088639+0000","last_undegraded":"2024-03-06T07:23:32.088639+0000","last_fullsized":"2024-03-06T07:23:32.088639+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097589+0000","last_change":"2024-03-06T07:23:32.094421+0000","last_active":"2024-03-06T07:23:32.097589+0000","last_peered":"2024-03-06T07:23:32.097589+0000","last_clean":"2024-03-06T07:23:32.097589+0000","last_became_active":"2024-03-06T07:23:32.094161+0000","last_became_peered":"2024-03-06T07:23:32.094161+0000","last_unstale":"2024-03-06T07:23:32.097589+0000","last_undegraded":"2024-03-06T07:23:32.097589+0000","last_fullsized":"2024-03-06T07:23:32.097589+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.777241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086688+0000","last_change":"2024-03-06T07:23:32.086688+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086688+0000","last_undegraded":"2024-03-06T07:23:32.086688+0000","last_fullsized":"2024-03-06T07:23:32.086688+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098325+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:32.098325+0000","last_peered":"2024-03-06T07:23:32.098325+0000","last_clean":"2024-03-06T07:23:32.098325+0000","last_became_active":"2024-03-06T07:23:32.096097+0000","last_became_peered":"2024-03-06T07:23:32.096097+0000","last_unstale":"2024-03-06T07:23:32.098325+0000","last_undegraded":"2024-03-06T07:23:32.098325+0000","last_fullsized":"2024-03-06T07:23:32.098325+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:11.255144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084439+0000","last_change":"2024-03-06T07:23:32.084439+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084439+0000","last_undegraded":"2024-03-06T07:23:32.084439+0000","last_fullsized":"2024-03-06T07:23:32.084439+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098473+0000","last_change":"2024-03-06T07:23:32.095330+0000","last_active":"2024-03-06T07:23:32.098473+0000","last_peered":"2024-03-06T07:23:32.098473+0000","last_clean":"2024-03-06T07:23:32.098473+0000","last_became_active":"2024-03-06T07:23:32.094988+0000","last_became_peered":"2024-03-06T07:23:32.094988+0000","last_unstale":"2024-03-06T07:23:32.098473+0000","last_undegraded":"2024-03-06T07:23:32.098473+0000","last_fullsized":"2024-03-06T07:23:32.098473+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:40.708193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084193+0000","last_change":"2024-03-06T07:23:32.084193+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084193+0000","last_undegraded":"2024-03-06T07:23:32.084193+0000","last_fullsized":"2024-03-06T07:23:32.084193+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098698+0000","last_change":"2024-03-06T07:23:32.095660+0000","last_active":"2024-03-06T07:23:32.098698+0000","last_peered":"2024-03-06T07:23:32.098698+0000","last_clean":"2024-03-06T07:23:32.098698+0000","last_became_active":"2024-03-06T07:23:32.094528+0000","last_became_peered":"2024-03-06T07:23:32.094528+0000","last_unstale":"2024-03-06T07:23:32.098698+0000","last_undegraded":"2024-03-06T07:23:32.098698+0000","last_fullsized":"2024-03-06T07:23:32.098698+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:44.060570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086601+0000","last_change":"2024-03-06T07:23:32.086601+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086601+0000","last_undegraded":"2024-03-06T07:23:32.086601+0000","last_fullsized":"2024-03-06T07:23:32.086601+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.094285+0000","last_change":"2024-03-06T07:23:32.091549+0000","last_active":"2024-03-06T07:23:32.094285+0000","last_peered":"2024-03-06T07:23:32.094285+0000","last_clean":"2024-03-06T07:23:32.094285+0000","last_became_active":"2024-03-06T07:23:32.091217+0000","last_became_peered":"2024-03-06T07:23:32.091217+0000","last_unstale":"2024-03-06T07:23:32.094285+0000","last_undegraded":"2024-03-06T07:23:32.094285+0000","last_fullsized":"2024-03-06T07:23:32.094285+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:38.093438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.081027+0000","last_change":"2024-03-06T07:23:32.081027+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.081027+0000","last_undegraded":"2024-03-06T07:23:32.081027+0000","last_fullsized":"2024-03-06T07:23:32.081027+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097934+0000","last_change":"2024-03-06T07:23:32.096137+0000","last_active":"2024-03-06T07:23:32.097934+0000","last_peered":"2024-03-06T07:23:32.097934+0000","last_clean":"2024-03-06T07:23:32.097934+0000","last_became_active":"2024-03-06T07:23:32.095785+0000","last_became_peered":"2024-03-06T07:23:32.095785+0000","last_unstale":"2024-03-06T07:23:32.097934+0000","last_undegraded":"2024-03-06T07:23:32.097934+0000","last_fullsized":"2024-03-06T07:23:32.097934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.777241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083252+0000","last_change":"2024-03-06T07:23:32.083252+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083252+0000","last_undegraded":"2024-03-06T07:23:32.083252+0000","last_fullsized":"2024-03-06T07:23:32.083252+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098782+0000","last_change":"2024-03-06T07:23:32.095556+0000","last_active":"2024-03-06T07:23:32.098782+0000","last_peered":"2024-03-06T07:23:32.098782+0000","last_clean":"2024-03-06T07:23:32.098782+0000","last_became_active":"2024-03-06T07:23:32.094330+0000","last_became_peered":"2024-03-06T07:23:32.094330+0000","last_unstale":"2024-03-06T07:23:32.098782+0000","last_undegraded":"2024-03-06T07:23:32.098782+0000","last_fullsized":"2024-03-06T07:23:32.098782+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.090883+0000","last_change":"2024-03-06T07:23:32.090883+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.090883+0000","last_undegraded":"2024-03-06T07:23:32.090883+0000","last_fullsized":"2024-03-06T07:23:32.090883+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.100819+0000","last_change":"2024-03-06T07:23:32.099162+0000","last_active":"2024-03-06T07:23:32.100819+0000","last_peered":"2024-03-06T07:23:32.100819+0000","last_clean":"2024-03-06T07:23:32.100819+0000","last_became_active":"2024-03-06T07:23:32.098923+0000","last_became_peered":"2024-03-06T07:23:32.098923+0000","last_unstale":"2024-03-06T07:23:32.100819+0000","last_undegraded":"2024-03-06T07:23:32.100819+0000","last_fullsized":"2024-03-06T07:23:32.100819+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:24.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084391+0000","last_change":"2024-03-06T07:23:32.084391+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084391+0000","last_undegraded":"2024-03-06T07:23:32.084391+0000","last_fullsized":"2024-03-06T07:23:32.084391+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.096792+0000","last_change":"2024-03-06T07:23:32.093466+0000","last_active":"2024-03-06T07:23:32.096792+0000","last_peered":"2024-03-06T07:23:32.096792+0000","last_clean":"2024-03-06T07:23:32.096792+0000","last_became_active":"2024-03-06T07:23:32.093178+0000","last_became_peered":"2024-03-06T07:23:32.093178+0000","last_unstale":"2024-03-06T07:23:32.096792+0000","last_undegraded":"2024-03-06T07:23:32.096792+0000","last_fullsized":"2024-03-06T07:23:32.096792+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:38.144217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083685+0000","last_change":"2024-03-06T07:23:32.083685+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083685+0000","last_undegraded":"2024-03-06T07:23:32.083685+0000","last_fullsized":"2024-03-06T07:23:32.083685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099451+0000","last_change":"2024-03-06T07:23:32.098048+0000","last_active":"2024-03-06T07:23:32.099451+0000","last_peered":"2024-03-06T07:23:32.099451+0000","last_clean":"2024-03-06T07:23:32.099451+0000","last_became_active":"2024-03-06T07:23:32.097599+0000","last_became_peered":"2024-03-06T07:23:32.097599+0000","last_unstale":"2024-03-06T07:23:32.099451+0000","last_undegraded":"2024-03-06T07:23:32.099451+0000","last_fullsized":"2024-03-06T07:23:32.099451+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.717543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086396+0000","last_change":"2024-03-06T07:23:32.086396+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086396+0000","last_undegraded":"2024-03-06T07:23:32.086396+0000","last_fullsized":"2024-03-06T07:23:32.086396+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097628+0000","last_change":"2024-03-06T07:23:32.094948+0000","last_active":"2024-03-06T07:23:32.097628+0000","last_peered":"2024-03-06T07:23:32.097628+0000","last_clean":"2024-03-06T07:23:32.097628+0000","last_became_active":"2024-03-06T07:23:32.094671+0000","last_became_peered":"2024-03-06T07:23:32.094671+0000","last_unstale":"2024-03-06T07:23:32.097628+0000","last_undegraded":"2024-03-06T07:23:32.097628+0000","last_fullsized":"2024-03-06T07:23:32.097628+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:26.788560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.083723+0000","last_change":"2024-03-06T07:23:32.083723+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.083723+0000","last_undegraded":"2024-03-06T07:23:32.083723+0000","last_fullsized":"2024-03-06T07:23:32.083723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.099967+0000","last_change":"2024-03-06T07:23:32.097587+0000","last_active":"2024-03-06T07:23:32.099967+0000","last_peered":"2024-03-06T07:23:32.099967+0000","last_clean":"2024-03-06T07:23:32.099967+0000","last_became_active":"2024-03-06T07:23:32.096686+0000","last_became_peered":"2024-03-06T07:23:32.096686+0000","last_unstale":"2024-03-06T07:23:32.099967+0000","last_undegraded":"2024-03-06T07:23:32.099967+0000","last_fullsized":"2024-03-06T07:23:32.099967+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:11.079702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.086545+0000","last_change":"2024-03-06T07:23:32.086545+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.086545+0000","last_undegraded":"2024-03-06T07:23:32.086545+0000","last_fullsized":"2024-03-06T07:23:32.086545+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097723+0000","last_change":"2024-03-06T07:23:32.095927+0000","last_active":"2024-03-06T07:23:32.097723+0000","last_peered":"2024-03-06T07:23:32.097723+0000","last_clean":"2024-03-06T07:23:32.097723+0000","last_became_active":"2024-03-06T07:23:32.095354+0000","last_became_peered":"2024-03-06T07:23:32.095354+0000","last_unstale":"2024-03-06T07:23:32.097723+0000","last_undegraded":"2024-03-06T07:23:32.097723+0000","last_fullsized":"2024-03-06T07:23:32.097723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:13.768735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.085807+0000","last_change":"2024-03-06T07:23:32.085807+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.085807+0000","last_undegraded":"2024-03-06T07:23:32.085807+0000","last_fullsized":"2024-03-06T07:23:32.085807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098853+0000","last_change":"2024-03-06T07:23:32.094186+0000","last_active":"2024-03-06T07:23:32.098853+0000","last_peered":"2024-03-06T07:23:32.098853+0000","last_clean":"2024-03-06T07:23:32.098853+0000","last_became_active":"2024-03-06T07:23:32.093861+0000","last_became_peered":"2024-03-06T07:23:32.093861+0000","last_unstale":"2024-03-06T07:23:32.098853+0000","last_undegraded":"2024-03-06T07:23:32.098853+0000","last_fullsized":"2024-03-06T07:23:32.098853+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.750831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088864+0000","last_change":"2024-03-06T07:23:32.088864+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088864+0000","last_undegraded":"2024-03-06T07:23:32.088864+0000","last_fullsized":"2024-03-06T07:23:32.088864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097714+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:32.097714+0000","last_peered":"2024-03-06T07:23:32.097714+0000","last_clean":"2024-03-06T07:23:32.097714+0000","last_became_active":"2024-03-06T07:23:32.094821+0000","last_became_peered":"2024-03-06T07:23:32.094821+0000","last_unstale":"2024-03-06T07:23:32.097714+0000","last_undegraded":"2024-03-06T07:23:32.097714+0000","last_fullsized":"2024-03-06T07:23:32.097714+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:17.223200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.091300+0000","last_change":"2024-03-06T07:23:32.091300+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.091300+0000","last_undegraded":"2024-03-06T07:23:32.091300+0000","last_fullsized":"2024-03-06T07:23:32.091300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.096070+0000","last_change":"2024-03-06T07:23:32.095102+0000","last_active":"2024-03-06T07:23:32.096070+0000","last_peered":"2024-03-06T07:23:32.096070+0000","last_clean":"2024-03-06T07:23:32.096070+0000","last_became_active":"2024-03-06T07:23:32.094829+0000","last_became_peered":"2024-03-06T07:23:32.094829+0000","last_unstale":"2024-03-06T07:23:32.096070+0000","last_undegraded":"2024-03-06T07:23:32.096070+0000","last_fullsized":"2024-03-06T07:23:32.096070+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:34.981739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084715+0000","last_change":"2024-03-06T07:23:32.084715+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084715+0000","last_undegraded":"2024-03-06T07:23:32.084715+0000","last_fullsized":"2024-03-06T07:23:32.084715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.094318+0000","last_change":"2024-03-06T07:23:32.091753+0000","last_active":"2024-03-06T07:23:32.094318+0000","last_peered":"2024-03-06T07:23:32.094318+0000","last_clean":"2024-03-06T07:23:32.094318+0000","last_became_active":"2024-03-06T07:23:32.091465+0000","last_became_peered":"2024-03-06T07:23:32.091465+0000","last_unstale":"2024-03-06T07:23:32.094318+0000","last_undegraded":"2024-03-06T07:23:32.094318+0000","last_fullsized":"2024-03-06T07:23:32.094318+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:57.007860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.084094+0000","last_change":"2024-03-06T07:23:32.084094+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.084094+0000","last_undegraded":"2024-03-06T07:23:32.084094+0000","last_fullsized":"2024-03-06T07:23:32.084094+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.098495+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:32.098495+0000","last_peered":"2024-03-06T07:23:32.098495+0000","last_clean":"2024-03-06T07:23:32.098495+0000","last_became_active":"2024-03-06T07:23:32.096065+0000","last_became_peered":"2024-03-06T07:23:32.096065+0000","last_unstale":"2024-03-06T07:23:32.098495+0000","last_undegraded":"2024-03-06T07:23:32.098495+0000","last_fullsized":"2024-03-06T07:23:32.098495+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:21.231136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-03-06T07:23:32.088028+0000","last_change":"2024-03-06T07:23:32.088028+0000","last_active":"2024-03-06T07:23:32.073823+0000","last_peered":"2024-03-06T07:23:32.073823+0000","last_clean":"2024-03-06T07:23:32.073823+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:32.088028+0000","last_undegraded":"2024-03-06T07:23:32.088028+0000","last_fullsized":"2024-03-06T07:23:32.088028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.097054+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:32.097054+0000","last_peered":"2024-03-06T07:23:32.097054+0000","last_clean":"2024-03-06T07:23:32.097054+0000","last_became_active":"2024-03-06T07:23:32.094784+0000","last_became_peered":"2024-03-06T07:23:32.094784+0000","last_unstale":"2024-03-06T07:23:32.097054+0000","last_undegraded":"2024-03-06T07:23:32.097054+0000","last_fullsized":"2024-03-06T07:23:32.097054+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:23.944371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-06T07:23:32.088815+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:23:32.088815+0000","last_peered":"2024-03-06T07:23:32.088815+0000","last_clean":"2024-03-06T07:23:32.088815+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:23:32.088815+0000","last_undegraded":"2024-03-06T07:23:32.088815+0000","last_fullsized":"2024-03-06T07:23:32.088815+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":784678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":200577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":491520,"data_stored":194310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":784678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:38.756 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-03-06T07:23:38.754+0000 7fee9a1e8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:40.619 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:40.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.720+0000 7fe2207d7640 1 Processor -- start 2024-03-06T07:23:40.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.720+0000 7fe2207d7640 1 -- start start 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe2207d7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe2180d4760 0x7fe2180b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe2207d7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2180d0a80 0x7fe2180cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe2207d7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2180d25d0 con 0x7fe2180b4170 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe2207d7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2180d4bf0 con 0x7fe2180d0a80 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe2207d7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2180ce9e0 con 0x7fe2180d4760 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21ed74640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2180d0a80 0x7fe2180cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21ed74640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2180d0a80 0x7fe2180cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49162/0 (socket says 172.21.15.77:49162) 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21ed74640 1 -- 172.21.15.77:0/2073500413 learned_addr learned my addr 172.21.15.77:0/2073500413 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21e573640 1 --1- 172.21.15.77:0/2073500413 >> v1:172.21.15.77:6789/0 conn(0x7fe2180b4170 0x7fe2180d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58584/0 (socket says 172.21.15.77:58584) 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21dd72640 1 --2- 172.21.15.77:0/2073500413 >> v2:172.21.15.158:3300/0 conn(0x7fe2180d4760 0x7fe2180b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21ed74640 1 -- 172.21.15.77:0/2073500413 >> v2:172.21.15.158:3300/0 conn(0x7fe2180d4760 msgr2=0x7fe2180b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21ed74640 1 --2- 172.21.15.77:0/2073500413 >> v2:172.21.15.158:3300/0 conn(0x7fe2180d4760 0x7fe2180b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21ed74640 1 -- 172.21.15.77:0/2073500413 >> v1:172.21.15.77:6789/0 conn(0x7fe2180b4170 legacy=0x7fe2180d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21ed74640 1 -- 172.21.15.77:0/2073500413 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2180d2870 con 0x7fe2180d0a80 2024-03-06T07:23:40.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.721+0000 7fe21d571640 1 -- 172.21.15.77:0/2073500413 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016468813 0 0) 0x7fe2180d25d0 con 0x7fe2180b4170 2024-03-06T07:23:40.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.722+0000 7fe21ed74640 1 --2- 172.21.15.77:0/2073500413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180d0a80 0x7fe2180cee70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe218067920 tx=0x7fe21002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e0ae353618257f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.722+0000 7fe21d571640 1 -- 172.21.15.77:0/2073500413 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21002cb20 con 0x7fe2180d0a80 2024-03-06T07:23:40.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.722+0000 7fe21d571640 1 -- 172.21.15.77:0/2073500413 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21002ccc0 con 0x7fe2180d0a80 2024-03-06T07:23:40.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.722+0000 7fe21d571640 1 -- 172.21.15.77:0/2073500413 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210034610 con 0x7fe2180d0a80 2024-03-06T07:23:40.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.722+0000 7fe2207d7640 1 -- 172.21.15.77:0/2073500413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180d0a80 msgr2=0x7fe2180cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.722+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2073500413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180d0a80 0x7fe2180cee70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe218067920 tx=0x7fe21002bf80 comp rx=0 tx=0).stop 2024-03-06T07:23:40.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.723+0000 7fe2207d7640 1 -- 172.21.15.77:0/2073500413 shutdown_connections 2024-03-06T07:23:40.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.723+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2073500413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180d0a80 0x7fe2180cee70 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.723+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2073500413 >> v2:172.21.15.158:3300/0 conn(0x7fe2180d4760 0x7fe2180b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.723+0000 7fe2207d7640 1 -- 172.21.15.77:0/2073500413 >> 172.21.15.77:0/2073500413 conn(0x7fe2180d3c40 msgr2=0x7fe2180d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.723+0000 7fe2207d7640 1 -- 172.21.15.77:0/2073500413 shutdown_connections 2024-03-06T07:23:40.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.723+0000 7fe2207d7640 1 -- 172.21.15.77:0/2073500413 wait complete. 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe2207d7640 1 Processor -- start 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe2207d7640 1 -- start start 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe2207d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180b4170 0x7fe21814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe2207d7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2180d0a80 0x7fe2180ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe2207d7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2180cd520 0x7fe21815d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe2207d7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2180b4dd0 con 0x7fe2180b4170 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe2207d7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2180b4550 con 0x7fe2180cd520 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe2207d7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2180b4f50 con 0x7fe2180d0a80 2024-03-06T07:23:40.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.724+0000 7fe21e573640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180b4170 0x7fe21814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21e573640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180b4170 0x7fe21814f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49170/0 (socket says 172.21.15.77:49170) 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21e573640 1 -- 172.21.15.77:0/2941158165 learned_addr learned my addr 172.21.15.77:0/2941158165 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21ed74640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2180cd520 0x7fe21815d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21dd72640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2180d0a80 0x7fe2180ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21e573640 1 -- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2180d0a80 msgr2=0x7fe2180ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21e573640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2180d0a80 0x7fe2180ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21e573640 1 -- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2180cd520 msgr2=0x7fe21815d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21e573640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2180cd520 0x7fe21815d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21e573640 1 -- 172.21.15.77:0/2941158165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21814f5d0 con 0x7fe2180b4170 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21ed74640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2180cd520 0x7fe21815d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe21e573640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180b4170 0x7fe21814f090 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe21400c510 tx=0x7fe21400c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe20effd640 1 -- 172.21.15.77:0/2941158165 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214018070 con 0x7fe2180b4170 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe20effd640 1 -- 172.21.15.77:0/2941158165 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2140074a0 con 0x7fe2180b4170 2024-03-06T07:23:40.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21815bff0 con 0x7fe2180b4170 2024-03-06T07:23:40.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe20effd640 1 -- 172.21.15.77:0/2941158165 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2140047e0 con 0x7fe2180b4170 2024-03-06T07:23:40.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.725+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21815c5b0 con 0x7fe2180b4170 2024-03-06T07:23:40.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.726+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21804f330 con 0x7fe2180b4170 2024-03-06T07:23:40.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.727+0000 7fe20effd640 1 -- 172.21.15.77:0/2941158165 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe21401b070 con 0x7fe2180b4170 2024-03-06T07:23:40.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.727+0000 7fe20effd640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.727+0000 7fe20effd640 1 -- 172.21.15.77:0/2941158165 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe214089560 con 0x7fe2180b4170 2024-03-06T07:23:40.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.727+0000 7fe21dd72640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.727+0000 7fe21dd72640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe208006fd0 tx=0x7fe208008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.736+0000 7fe20effd640 1 -- 172.21.15.77:0/2941158165 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe21814f5d0 con 0x7fe2180b4170 2024-03-06T07:23:40.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.908+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2180b9450 con 0x7fe1fc06b4d0 2024-03-06T07:23:40.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.920+0000 7fe20effd640 1 -- 172.21.15.77:0/2941158165 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251958 (secure 0 0 0) 0x7fe2180b9450 con 0x7fe1fc06b4d0 2024-03-06T07:23:40.924 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:40.925 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:23:40.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1fc06b4d0 msgr2=0x7fe1fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe208006fd0 tx=0x7fe208008040 comp rx=0 tx=0).stop 2024-03-06T07:23:40.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180b4170 msgr2=0x7fe21814f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180b4170 0x7fe21814f090 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe21400c510 tx=0x7fe21400c9d0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 shutdown_connections 2024-03-06T07:23:40.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2180cd520 0x7fe21815d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2180d0a80 0x7fe2180ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 --2- 172.21.15.77:0/2941158165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2180b4170 0x7fe21814f090 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.922+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 >> 172.21.15.77:0/2941158165 conn(0x7fe2180d3c40 msgr2=0x7fe2180b7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.923+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 shutdown_connections 2024-03-06T07:23:40.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:40.923+0000 7fe2207d7640 1 -- 172.21.15.77:0/2941158165 wait complete. 2024-03-06T07:23:40.935 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-03-06T07:23:39.402816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258976,"kb_used_data":5352,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754715744,"statfs":{"total":773094113280,"available":772828921856,"internally_reserved":0,"allocated":5480448,"data_stored":2894772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.324263"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783645+0000","last_change":"2024-03-06T07:23:33.100649+0000","last_active":"2024-03-06T07:23:34.783645+0000","last_peered":"2024-03-06T07:23:34.783645+0000","last_clean":"2024-03-06T07:23:34.783645+0000","last_became_active":"2024-03-06T07:23:33.100379+0000","last_became_peered":"2024-03-06T07:23:33.100379+0000","last_unstale":"2024-03-06T07:23:34.783645+0000","last_undegraded":"2024-03-06T07:23:34.783645+0000","last_fullsized":"2024-03-06T07:23:34.783645+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:31.076210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782662+0000","last_change":"2024-03-06T07:23:33.095132+0000","last_active":"2024-03-06T07:23:35.782662+0000","last_peered":"2024-03-06T07:23:35.782662+0000","last_clean":"2024-03-06T07:23:35.782662+0000","last_became_active":"2024-03-06T07:23:33.094675+0000","last_became_peered":"2024-03-06T07:23:33.094675+0000","last_unstale":"2024-03-06T07:23:35.782662+0000","last_undegraded":"2024-03-06T07:23:35.782662+0000","last_fullsized":"2024-03-06T07:23:35.782662+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:43.911725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095032+0000","last_change":"2024-03-06T07:23:33.086648+0000","last_active":"2024-03-06T07:23:34.095032+0000","last_peered":"2024-03-06T07:23:34.095032+0000","last_clean":"2024-03-06T07:23:34.095032+0000","last_became_active":"2024-03-06T07:23:33.086358+0000","last_became_peered":"2024-03-06T07:23:33.086358+0000","last_unstale":"2024-03-06T07:23:34.095032+0000","last_undegraded":"2024-03-06T07:23:34.095032+0000","last_fullsized":"2024-03-06T07:23:34.095032+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:01.782008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859099+0000","last_change":"2024-03-06T07:23:33.099893+0000","last_active":"2024-03-06T07:23:34.859099+0000","last_peered":"2024-03-06T07:23:34.859099+0000","last_clean":"2024-03-06T07:23:34.859099+0000","last_became_active":"2024-03-06T07:23:33.099626+0000","last_became_peered":"2024-03-06T07:23:33.099626+0000","last_unstale":"2024-03-06T07:23:34.859099+0000","last_undegraded":"2024-03-06T07:23:34.859099+0000","last_fullsized":"2024-03-06T07:23:34.859099+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:46.609566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782623+0000","last_change":"2024-03-06T07:23:33.094266+0000","last_active":"2024-03-06T07:23:35.782623+0000","last_peered":"2024-03-06T07:23:35.782623+0000","last_clean":"2024-03-06T07:23:35.782623+0000","last_became_active":"2024-03-06T07:23:33.093838+0000","last_became_peered":"2024-03-06T07:23:33.093838+0000","last_unstale":"2024-03-06T07:23:35.782623+0000","last_undegraded":"2024-03-06T07:23:35.782623+0000","last_fullsized":"2024-03-06T07:23:35.782623+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:46.609566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085463+0000","last_change":"2024-03-06T07:23:33.104112+0000","last_active":"2024-03-06T07:23:35.085463+0000","last_peered":"2024-03-06T07:23:35.085463+0000","last_clean":"2024-03-06T07:23:35.085463+0000","last_became_active":"2024-03-06T07:23:33.103846+0000","last_became_peered":"2024-03-06T07:23:33.103846+0000","last_unstale":"2024-03-06T07:23:35.085463+0000","last_undegraded":"2024-03-06T07:23:35.085463+0000","last_fullsized":"2024-03-06T07:23:35.085463+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:56.292318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783852+0000","last_change":"2024-03-06T07:23:33.101320+0000","last_active":"2024-03-06T07:23:34.783852+0000","last_peered":"2024-03-06T07:23:34.783852+0000","last_clean":"2024-03-06T07:23:34.783852+0000","last_became_active":"2024-03-06T07:23:33.100857+0000","last_became_peered":"2024-03-06T07:23:33.100857+0000","last_unstale":"2024-03-06T07:23:34.783852+0000","last_undegraded":"2024-03-06T07:23:34.783852+0000","last_fullsized":"2024-03-06T07:23:34.783852+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:31.956740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858720+0000","last_change":"2024-03-06T07:23:33.096238+0000","last_active":"2024-03-06T07:23:34.858720+0000","last_peered":"2024-03-06T07:23:34.858720+0000","last_clean":"2024-03-06T07:23:34.858720+0000","last_became_active":"2024-03-06T07:23:33.096011+0000","last_became_peered":"2024-03-06T07:23:33.096011+0000","last_unstale":"2024-03-06T07:23:34.858720+0000","last_undegraded":"2024-03-06T07:23:34.858720+0000","last_fullsized":"2024-03-06T07:23:34.858720+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:57.505947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859290+0000","last_change":"2024-03-06T07:23:33.108077+0000","last_active":"2024-03-06T07:23:34.859290+0000","last_peered":"2024-03-06T07:23:34.859290+0000","last_clean":"2024-03-06T07:23:34.859290+0000","last_became_active":"2024-03-06T07:23:33.107628+0000","last_became_peered":"2024-03-06T07:23:33.107628+0000","last_unstale":"2024-03-06T07:23:34.859290+0000","last_undegraded":"2024-03-06T07:23:34.859290+0000","last_fullsized":"2024-03-06T07:23:34.859290+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:31.076210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859267+0000","last_change":"2024-03-06T07:23:33.096657+0000","last_active":"2024-03-06T07:23:34.859267+0000","last_peered":"2024-03-06T07:23:34.859267+0000","last_clean":"2024-03-06T07:23:34.859267+0000","last_became_active":"2024-03-06T07:23:33.096390+0000","last_became_peered":"2024-03-06T07:23:33.096390+0000","last_unstale":"2024-03-06T07:23:34.859267+0000","last_undegraded":"2024-03-06T07:23:34.859267+0000","last_fullsized":"2024-03-06T07:23:34.859267+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:50.522229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859135+0000","last_change":"2024-03-06T07:23:33.107412+0000","last_active":"2024-03-06T07:23:34.859135+0000","last_peered":"2024-03-06T07:23:34.859135+0000","last_clean":"2024-03-06T07:23:34.859135+0000","last_became_active":"2024-03-06T07:23:33.105969+0000","last_became_peered":"2024-03-06T07:23:33.105969+0000","last_unstale":"2024-03-06T07:23:34.859135+0000","last_undegraded":"2024-03-06T07:23:34.859135+0000","last_fullsized":"2024-03-06T07:23:34.859135+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:43.911725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782690+0000","last_change":"2024-03-06T07:23:33.096505+0000","last_active":"2024-03-06T07:23:35.782690+0000","last_peered":"2024-03-06T07:23:35.782690+0000","last_clean":"2024-03-06T07:23:35.782690+0000","last_became_active":"2024-03-06T07:23:33.096259+0000","last_became_peered":"2024-03-06T07:23:33.096259+0000","last_unstale":"2024-03-06T07:23:35.782690+0000","last_undegraded":"2024-03-06T07:23:35.782690+0000","last_fullsized":"2024-03-06T07:23:35.782690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:10.964086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.094773+0000","last_change":"2024-03-06T07:23:33.103036+0000","last_active":"2024-03-06T07:23:34.094773+0000","last_peered":"2024-03-06T07:23:34.094773+0000","last_clean":"2024-03-06T07:23:34.094773+0000","last_became_active":"2024-03-06T07:23:33.102526+0000","last_became_peered":"2024-03-06T07:23:33.102526+0000","last_unstale":"2024-03-06T07:23:34.094773+0000","last_undegraded":"2024-03-06T07:23:34.094773+0000","last_fullsized":"2024-03-06T07:23:34.094773+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:19.238308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858820+0000","last_change":"2024-03-06T07:23:33.102053+0000","last_active":"2024-03-06T07:23:34.858820+0000","last_peered":"2024-03-06T07:23:34.858820+0000","last_clean":"2024-03-06T07:23:34.858820+0000","last_became_active":"2024-03-06T07:23:33.101650+0000","last_became_peered":"2024-03-06T07:23:33.101650+0000","last_unstale":"2024-03-06T07:23:34.858820+0000","last_undegraded":"2024-03-06T07:23:34.858820+0000","last_fullsized":"2024-03-06T07:23:34.858820+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:05.676483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085540+0000","last_change":"2024-03-06T07:23:33.100002+0000","last_active":"2024-03-06T07:23:35.085540+0000","last_peered":"2024-03-06T07:23:35.085540+0000","last_clean":"2024-03-06T07:23:35.085540+0000","last_became_active":"2024-03-06T07:23:33.098132+0000","last_became_peered":"2024-03-06T07:23:33.098132+0000","last_unstale":"2024-03-06T07:23:35.085540+0000","last_undegraded":"2024-03-06T07:23:35.085540+0000","last_fullsized":"2024-03-06T07:23:35.085540+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:02.392531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095114+0000","last_change":"2024-03-06T07:23:33.093669+0000","last_active":"2024-03-06T07:23:34.095114+0000","last_peered":"2024-03-06T07:23:34.095114+0000","last_clean":"2024-03-06T07:23:34.095114+0000","last_became_active":"2024-03-06T07:23:33.093389+0000","last_became_peered":"2024-03-06T07:23:33.093389+0000","last_unstale":"2024-03-06T07:23:34.095114+0000","last_undegraded":"2024-03-06T07:23:34.095114+0000","last_fullsized":"2024-03-06T07:23:34.095114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:42.504411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095490+0000","last_change":"2024-03-06T07:23:33.094773+0000","last_active":"2024-03-06T07:23:34.095490+0000","last_peered":"2024-03-06T07:23:34.095490+0000","last_clean":"2024-03-06T07:23:34.095490+0000","last_became_active":"2024-03-06T07:23:33.094510+0000","last_became_peered":"2024-03-06T07:23:33.094510+0000","last_unstale":"2024-03-06T07:23:34.095490+0000","last_undegraded":"2024-03-06T07:23:34.095490+0000","last_fullsized":"2024-03-06T07:23:34.095490+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:57.505947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095444+0000","last_change":"2024-03-06T07:23:33.098749+0000","last_active":"2024-03-06T07:23:34.095444+0000","last_peered":"2024-03-06T07:23:34.095444+0000","last_clean":"2024-03-06T07:23:34.095444+0000","last_became_active":"2024-03-06T07:23:33.098484+0000","last_became_peered":"2024-03-06T07:23:33.098484+0000","last_unstale":"2024-03-06T07:23:34.095444+0000","last_undegraded":"2024-03-06T07:23:34.095444+0000","last_fullsized":"2024-03-06T07:23:34.095444+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:01.497661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.096433+0000","last_change":"2024-03-06T07:23:33.098078+0000","last_active":"2024-03-06T07:23:34.096433+0000","last_peered":"2024-03-06T07:23:34.096433+0000","last_clean":"2024-03-06T07:23:34.096433+0000","last_became_active":"2024-03-06T07:23:33.097829+0000","last_became_peered":"2024-03-06T07:23:33.097829+0000","last_unstale":"2024-03-06T07:23:34.096433+0000","last_undegraded":"2024-03-06T07:23:34.096433+0000","last_fullsized":"2024-03-06T07:23:34.096433+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:44.037518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.094662+0000","last_change":"2024-03-06T07:23:33.094839+0000","last_active":"2024-03-06T07:23:34.094662+0000","last_peered":"2024-03-06T07:23:34.094662+0000","last_clean":"2024-03-06T07:23:34.094662+0000","last_became_active":"2024-03-06T07:23:33.094583+0000","last_became_peered":"2024-03-06T07:23:33.094583+0000","last_unstale":"2024-03-06T07:23:34.094662+0000","last_undegraded":"2024-03-06T07:23:34.094662+0000","last_fullsized":"2024-03-06T07:23:34.094662+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:35.986506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085501+0000","last_change":"2024-03-06T07:23:33.103514+0000","last_active":"2024-03-06T07:23:35.085501+0000","last_peered":"2024-03-06T07:23:35.085501+0000","last_clean":"2024-03-06T07:23:35.085501+0000","last_became_active":"2024-03-06T07:23:33.103222+0000","last_became_peered":"2024-03-06T07:23:33.103222+0000","last_unstale":"2024-03-06T07:23:35.085501+0000","last_undegraded":"2024-03-06T07:23:35.085501+0000","last_fullsized":"2024-03-06T07:23:35.085501+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:27.427288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858861+0000","last_change":"2024-03-06T07:23:33.095878+0000","last_active":"2024-03-06T07:23:34.858861+0000","last_peered":"2024-03-06T07:23:34.858861+0000","last_clean":"2024-03-06T07:23:34.858861+0000","last_became_active":"2024-03-06T07:23:33.095641+0000","last_became_peered":"2024-03-06T07:23:33.095641+0000","last_unstale":"2024-03-06T07:23:34.858861+0000","last_undegraded":"2024-03-06T07:23:34.858861+0000","last_fullsized":"2024-03-06T07:23:34.858861+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:56.447466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.096008+0000","last_change":"2024-03-06T07:23:33.091015+0000","last_active":"2024-03-06T07:23:34.096008+0000","last_peered":"2024-03-06T07:23:34.096008+0000","last_clean":"2024-03-06T07:23:34.096008+0000","last_became_active":"2024-03-06T07:23:33.090027+0000","last_became_peered":"2024-03-06T07:23:33.090027+0000","last_unstale":"2024-03-06T07:23:34.096008+0000","last_undegraded":"2024-03-06T07:23:34.096008+0000","last_fullsized":"2024-03-06T07:23:34.096008+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:44.445726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095200+0000","last_change":"2024-03-06T07:23:33.091101+0000","last_active":"2024-03-06T07:23:34.095200+0000","last_peered":"2024-03-06T07:23:34.095200+0000","last_clean":"2024-03-06T07:23:34.095200+0000","last_became_active":"2024-03-06T07:23:33.090823+0000","last_became_peered":"2024-03-06T07:23:33.090823+0000","last_unstale":"2024-03-06T07:23:34.095200+0000","last_undegraded":"2024-03-06T07:23:34.095200+0000","last_fullsized":"2024-03-06T07:23:34.095200+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:58.012627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.096239+0000","last_change":"2024-03-06T07:23:33.102162+0000","last_active":"2024-03-06T07:23:34.096239+0000","last_peered":"2024-03-06T07:23:34.096239+0000","last_clean":"2024-03-06T07:23:34.096239+0000","last_became_active":"2024-03-06T07:23:33.101740+0000","last_became_peered":"2024-03-06T07:23:33.101740+0000","last_unstale":"2024-03-06T07:23:34.096239+0000","last_undegraded":"2024-03-06T07:23:34.096239+0000","last_fullsized":"2024-03-06T07:23:34.096239+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:39.148984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.094432+0000","last_change":"2024-03-06T07:23:33.101516+0000","last_active":"2024-03-06T07:23:34.094432+0000","last_peered":"2024-03-06T07:23:34.094432+0000","last_clean":"2024-03-06T07:23:34.094432+0000","last_became_active":"2024-03-06T07:23:33.101224+0000","last_became_peered":"2024-03-06T07:23:33.101224+0000","last_unstale":"2024-03-06T07:23:34.094432+0000","last_undegraded":"2024-03-06T07:23:34.094432+0000","last_fullsized":"2024-03-06T07:23:34.094432+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:22.235903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095220+0000","last_change":"2024-03-06T07:23:33.093728+0000","last_active":"2024-03-06T07:23:34.095220+0000","last_peered":"2024-03-06T07:23:34.095220+0000","last_clean":"2024-03-06T07:23:34.095220+0000","last_became_active":"2024-03-06T07:23:33.093374+0000","last_became_peered":"2024-03-06T07:23:33.093374+0000","last_unstale":"2024-03-06T07:23:34.095220+0000","last_undegraded":"2024-03-06T07:23:34.095220+0000","last_fullsized":"2024-03-06T07:23:34.095220+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:14.773502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782789+0000","last_change":"2024-03-06T07:23:33.095874+0000","last_active":"2024-03-06T07:23:35.782789+0000","last_peered":"2024-03-06T07:23:35.782789+0000","last_clean":"2024-03-06T07:23:35.782789+0000","last_became_active":"2024-03-06T07:23:33.095589+0000","last_became_peered":"2024-03-06T07:23:33.095589+0000","last_unstale":"2024-03-06T07:23:35.782789+0000","last_undegraded":"2024-03-06T07:23:35.782789+0000","last_fullsized":"2024-03-06T07:23:35.782789+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:50.522229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095696+0000","last_change":"2024-03-06T07:23:33.097122+0000","last_active":"2024-03-06T07:23:34.095696+0000","last_peered":"2024-03-06T07:23:34.095696+0000","last_clean":"2024-03-06T07:23:34.095696+0000","last_became_active":"2024-03-06T07:23:33.096241+0000","last_became_peered":"2024-03-06T07:23:33.096241+0000","last_unstale":"2024-03-06T07:23:34.095696+0000","last_undegraded":"2024-03-06T07:23:34.095696+0000","last_fullsized":"2024-03-06T07:23:34.095696+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:57.755598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095181+0000","last_change":"2024-03-06T07:23:33.091795+0000","last_active":"2024-03-06T07:23:34.095181+0000","last_peered":"2024-03-06T07:23:34.095181+0000","last_clean":"2024-03-06T07:23:34.095181+0000","last_became_active":"2024-03-06T07:23:33.091530+0000","last_became_peered":"2024-03-06T07:23:33.091530+0000","last_unstale":"2024-03-06T07:23:34.095181+0000","last_undegraded":"2024-03-06T07:23:34.095181+0000","last_fullsized":"2024-03-06T07:23:34.095181+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:27.793327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085585+0000","last_change":"2024-03-06T07:23:33.101146+0000","last_active":"2024-03-06T07:23:35.085585+0000","last_peered":"2024-03-06T07:23:35.085585+0000","last_clean":"2024-03-06T07:23:35.085585+0000","last_became_active":"2024-03-06T07:23:33.100857+0000","last_became_peered":"2024-03-06T07:23:33.100857+0000","last_unstale":"2024-03-06T07:23:35.085585+0000","last_undegraded":"2024-03-06T07:23:35.085585+0000","last_fullsized":"2024-03-06T07:23:35.085585+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:26.000523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.094410+0000","last_change":"2024-03-06T07:23:33.094298+0000","last_active":"2024-03-06T07:23:34.094410+0000","last_peered":"2024-03-06T07:23:34.094410+0000","last_clean":"2024-03-06T07:23:34.094410+0000","last_became_active":"2024-03-06T07:23:33.094043+0000","last_became_peered":"2024-03-06T07:23:33.094043+0000","last_unstale":"2024-03-06T07:23:34.094410+0000","last_undegraded":"2024-03-06T07:23:34.094410+0000","last_fullsized":"2024-03-06T07:23:34.094410+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:41.712960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.714710+0000","last_change":"2024-03-06T07:23:33.109233+0000","last_active":"2024-03-06T07:23:34.714710+0000","last_peered":"2024-03-06T07:23:34.714710+0000","last_clean":"2024-03-06T07:23:34.714710+0000","last_became_active":"2024-03-06T07:23:33.108921+0000","last_became_peered":"2024-03-06T07:23:33.108921+0000","last_unstale":"2024-03-06T07:23:34.714710+0000","last_undegraded":"2024-03-06T07:23:34.714710+0000","last_fullsized":"2024-03-06T07:23:34.714710+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:18.227967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085883+0000","last_change":"2024-03-06T07:23:32.096250+0000","last_active":"2024-03-06T07:23:35.085883+0000","last_peered":"2024-03-06T07:23:35.085883+0000","last_clean":"2024-03-06T07:23:35.085883+0000","last_became_active":"2024-03-06T07:23:32.095974+0000","last_became_peered":"2024-03-06T07:23:32.095974+0000","last_unstale":"2024-03-06T07:23:35.085883+0000","last_undegraded":"2024-03-06T07:23:35.085883+0000","last_fullsized":"2024-03-06T07:23:35.085883+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:55.442699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.096360+0000","last_change":"2024-03-06T07:23:33.095125+0000","last_active":"2024-03-06T07:23:34.096360+0000","last_peered":"2024-03-06T07:23:34.096360+0000","last_clean":"2024-03-06T07:23:34.096360+0000","last_became_active":"2024-03-06T07:23:33.094845+0000","last_became_peered":"2024-03-06T07:23:33.094845+0000","last_unstale":"2024-03-06T07:23:34.096360+0000","last_undegraded":"2024-03-06T07:23:34.096360+0000","last_fullsized":"2024-03-06T07:23:34.096360+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:15.790448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095624+0000","last_change":"2024-03-06T07:23:32.091961+0000","last_active":"2024-03-06T07:23:34.095624+0000","last_peered":"2024-03-06T07:23:34.095624+0000","last_clean":"2024-03-06T07:23:34.095624+0000","last_became_active":"2024-03-06T07:23:32.091129+0000","last_became_peered":"2024-03-06T07:23:32.091129+0000","last_unstale":"2024-03-06T07:23:34.095624+0000","last_undegraded":"2024-03-06T07:23:34.095624+0000","last_fullsized":"2024-03-06T07:23:34.095624+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:41.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782873+0000","last_change":"2024-03-06T07:23:33.097400+0000","last_active":"2024-03-06T07:23:35.782873+0000","last_peered":"2024-03-06T07:23:35.782873+0000","last_clean":"2024-03-06T07:23:35.782873+0000","last_became_active":"2024-03-06T07:23:33.097056+0000","last_became_peered":"2024-03-06T07:23:33.097056+0000","last_unstale":"2024-03-06T07:23:35.782873+0000","last_undegraded":"2024-03-06T07:23:35.782873+0000","last_fullsized":"2024-03-06T07:23:35.782873+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:31.076210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.714759+0000","last_change":"2024-03-06T07:23:32.091042+0000","last_active":"2024-03-06T07:23:34.714759+0000","last_peered":"2024-03-06T07:23:34.714759+0000","last_clean":"2024-03-06T07:23:34.714759+0000","last_became_active":"2024-03-06T07:23:32.090683+0000","last_became_peered":"2024-03-06T07:23:32.090683+0000","last_unstale":"2024-03-06T07:23:34.714759+0000","last_undegraded":"2024-03-06T07:23:34.714759+0000","last_fullsized":"2024-03-06T07:23:34.714759+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:15.362850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.714805+0000","last_change":"2024-03-06T07:23:33.097834+0000","last_active":"2024-03-06T07:23:34.714805+0000","last_peered":"2024-03-06T07:23:34.714805+0000","last_clean":"2024-03-06T07:23:34.714805+0000","last_became_active":"2024-03-06T07:23:33.097591+0000","last_became_peered":"2024-03-06T07:23:33.097591+0000","last_unstale":"2024-03-06T07:23:34.714805+0000","last_undegraded":"2024-03-06T07:23:34.714805+0000","last_fullsized":"2024-03-06T07:23:34.714805+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:27.793327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783573+0000","last_change":"2024-03-06T07:23:32.093816+0000","last_active":"2024-03-06T07:23:34.783573+0000","last_peered":"2024-03-06T07:23:34.783573+0000","last_clean":"2024-03-06T07:23:34.783573+0000","last_became_active":"2024-03-06T07:23:32.093538+0000","last_became_peered":"2024-03-06T07:23:32.093538+0000","last_unstale":"2024-03-06T07:23:34.783573+0000","last_undegraded":"2024-03-06T07:23:34.783573+0000","last_fullsized":"2024-03-06T07:23:34.783573+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:56.501180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095371+0000","last_change":"2024-03-06T07:23:33.096568+0000","last_active":"2024-03-06T07:23:34.095371+0000","last_peered":"2024-03-06T07:23:34.095371+0000","last_clean":"2024-03-06T07:23:34.095371+0000","last_became_active":"2024-03-06T07:23:33.096314+0000","last_became_peered":"2024-03-06T07:23:33.096314+0000","last_unstale":"2024-03-06T07:23:34.095371+0000","last_undegraded":"2024-03-06T07:23:34.095371+0000","last_fullsized":"2024-03-06T07:23:34.095371+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:59.952549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085718+0000","last_change":"2024-03-06T07:23:32.099457+0000","last_active":"2024-03-06T07:23:35.085718+0000","last_peered":"2024-03-06T07:23:35.085718+0000","last_clean":"2024-03-06T07:23:35.085718+0000","last_became_active":"2024-03-06T07:23:32.099188+0000","last_became_peered":"2024-03-06T07:23:32.099188+0000","last_unstale":"2024-03-06T07:23:35.085718+0000","last_undegraded":"2024-03-06T07:23:35.085718+0000","last_fullsized":"2024-03-06T07:23:35.085718+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:09.959319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859028+0000","last_change":"2024-03-06T07:23:33.098949+0000","last_active":"2024-03-06T07:23:34.859028+0000","last_peered":"2024-03-06T07:23:34.859028+0000","last_clean":"2024-03-06T07:23:34.859028+0000","last_became_active":"2024-03-06T07:23:33.094540+0000","last_became_peered":"2024-03-06T07:23:33.094540+0000","last_unstale":"2024-03-06T07:23:34.859028+0000","last_undegraded":"2024-03-06T07:23:34.859028+0000","last_fullsized":"2024-03-06T07:23:34.859028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:36.887363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095711+0000","last_change":"2024-03-06T07:23:32.090283+0000","last_active":"2024-03-06T07:23:34.095711+0000","last_peered":"2024-03-06T07:23:34.095711+0000","last_clean":"2024-03-06T07:23:34.095711+0000","last_became_active":"2024-03-06T07:23:32.089989+0000","last_became_peered":"2024-03-06T07:23:32.089989+0000","last_unstale":"2024-03-06T07:23:34.095711+0000","last_undegraded":"2024-03-06T07:23:34.095711+0000","last_fullsized":"2024-03-06T07:23:34.095711+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:26.122927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858771+0000","last_change":"2024-03-06T07:23:33.096094+0000","last_active":"2024-03-06T07:23:34.858771+0000","last_peered":"2024-03-06T07:23:34.858771+0000","last_clean":"2024-03-06T07:23:34.858771+0000","last_became_active":"2024-03-06T07:23:33.095709+0000","last_became_peered":"2024-03-06T07:23:33.095709+0000","last_unstale":"2024-03-06T07:23:34.858771+0000","last_undegraded":"2024-03-06T07:23:34.858771+0000","last_fullsized":"2024-03-06T07:23:34.858771+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:12.084469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782412+0000","last_change":"2024-03-06T07:23:32.095613+0000","last_active":"2024-03-06T07:23:35.782412+0000","last_peered":"2024-03-06T07:23:35.782412+0000","last_clean":"2024-03-06T07:23:35.782412+0000","last_became_active":"2024-03-06T07:23:32.095182+0000","last_became_peered":"2024-03-06T07:23:32.095182+0000","last_unstale":"2024-03-06T07:23:35.782412+0000","last_undegraded":"2024-03-06T07:23:35.782412+0000","last_fullsized":"2024-03-06T07:23:35.782412+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:13.768735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.096588+0000","last_change":"2024-03-06T07:23:33.090202+0000","last_active":"2024-03-06T07:23:34.096588+0000","last_peered":"2024-03-06T07:23:34.096588+0000","last_clean":"2024-03-06T07:23:34.096588+0000","last_became_active":"2024-03-06T07:23:33.089901+0000","last_became_peered":"2024-03-06T07:23:33.089901+0000","last_unstale":"2024-03-06T07:23:34.096588+0000","last_undegraded":"2024-03-06T07:23:34.096588+0000","last_fullsized":"2024-03-06T07:23:34.096588+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:18.227967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858844+0000","last_change":"2024-03-06T07:23:32.099418+0000","last_active":"2024-03-06T07:23:34.858844+0000","last_peered":"2024-03-06T07:23:34.858844+0000","last_clean":"2024-03-06T07:23:34.858844+0000","last_became_active":"2024-03-06T07:23:32.099121+0000","last_became_peered":"2024-03-06T07:23:32.099121+0000","last_unstale":"2024-03-06T07:23:34.858844+0000","last_undegraded":"2024-03-06T07:23:34.858844+0000","last_fullsized":"2024-03-06T07:23:34.858844+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:26.122927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095465+0000","last_change":"2024-03-06T07:23:33.097338+0000","last_active":"2024-03-06T07:23:34.095465+0000","last_peered":"2024-03-06T07:23:34.095465+0000","last_clean":"2024-03-06T07:23:34.095465+0000","last_became_active":"2024-03-06T07:23:33.097059+0000","last_became_peered":"2024-03-06T07:23:33.097059+0000","last_unstale":"2024-03-06T07:23:34.095465+0000","last_undegraded":"2024-03-06T07:23:34.095465+0000","last_fullsized":"2024-03-06T07:23:34.095465+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:12.259910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095606+0000","last_change":"2024-03-06T07:23:32.087920+0000","last_active":"2024-03-06T07:23:34.095606+0000","last_peered":"2024-03-06T07:23:34.095606+0000","last_clean":"2024-03-06T07:23:34.095606+0000","last_became_active":"2024-03-06T07:23:32.087584+0000","last_became_peered":"2024-03-06T07:23:32.087584+0000","last_unstale":"2024-03-06T07:23:34.095606+0000","last_undegraded":"2024-03-06T07:23:34.095606+0000","last_fullsized":"2024-03-06T07:23:34.095606+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:57.007860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783673+0000","last_change":"2024-03-06T07:23:33.101745+0000","last_active":"2024-03-06T07:23:34.783673+0000","last_peered":"2024-03-06T07:23:34.783673+0000","last_clean":"2024-03-06T07:23:34.783673+0000","last_became_active":"2024-03-06T07:23:33.101477+0000","last_became_peered":"2024-03-06T07:23:33.101477+0000","last_unstale":"2024-03-06T07:23:34.783673+0000","last_undegraded":"2024-03-06T07:23:34.783673+0000","last_fullsized":"2024-03-06T07:23:34.783673+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:35.867592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859322+0000","last_change":"2024-03-06T07:23:32.097214+0000","last_active":"2024-03-06T07:23:34.859322+0000","last_peered":"2024-03-06T07:23:34.859322+0000","last_clean":"2024-03-06T07:23:34.859322+0000","last_became_active":"2024-03-06T07:23:32.096930+0000","last_became_peered":"2024-03-06T07:23:32.096930+0000","last_unstale":"2024-03-06T07:23:34.859322+0000","last_undegraded":"2024-03-06T07:23:34.859322+0000","last_fullsized":"2024-03-06T07:23:34.859322+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:57.007860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085623+0000","last_change":"2024-03-06T07:23:33.100148+0000","last_active":"2024-03-06T07:23:35.085623+0000","last_peered":"2024-03-06T07:23:35.085623+0000","last_clean":"2024-03-06T07:23:35.085623+0000","last_became_active":"2024-03-06T07:23:33.099902+0000","last_became_peered":"2024-03-06T07:23:33.099902+0000","last_unstale":"2024-03-06T07:23:35.085623+0000","last_undegraded":"2024-03-06T07:23:35.085623+0000","last_fullsized":"2024-03-06T07:23:35.085623+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:08.700182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783807+0000","last_change":"2024-03-06T07:23:32.096162+0000","last_active":"2024-03-06T07:23:34.783807+0000","last_peered":"2024-03-06T07:23:34.783807+0000","last_clean":"2024-03-06T07:23:34.783807+0000","last_became_active":"2024-03-06T07:23:32.095663+0000","last_became_peered":"2024-03-06T07:23:32.095663+0000","last_unstale":"2024-03-06T07:23:34.783807+0000","last_undegraded":"2024-03-06T07:23:34.783807+0000","last_fullsized":"2024-03-06T07:23:34.783807+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:11.079702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783664+0000","last_change":"2024-03-06T07:23:33.104586+0000","last_active":"2024-03-06T07:23:34.783664+0000","last_peered":"2024-03-06T07:23:34.783664+0000","last_clean":"2024-03-06T07:23:34.783664+0000","last_became_active":"2024-03-06T07:23:33.104307+0000","last_became_peered":"2024-03-06T07:23:33.104307+0000","last_unstale":"2024-03-06T07:23:34.783664+0000","last_undegraded":"2024-03-06T07:23:34.783664+0000","last_fullsized":"2024-03-06T07:23:34.783664+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:06.017956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.094612+0000","last_change":"2024-03-06T07:23:32.093863+0000","last_active":"2024-03-06T07:23:34.094612+0000","last_peered":"2024-03-06T07:23:34.094612+0000","last_clean":"2024-03-06T07:23:34.094612+0000","last_became_active":"2024-03-06T07:23:32.093600+0000","last_became_peered":"2024-03-06T07:23:32.093600+0000","last_unstale":"2024-03-06T07:23:34.094612+0000","last_undegraded":"2024-03-06T07:23:34.094612+0000","last_fullsized":"2024-03-06T07:23:34.094612+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:04.773410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.096574+0000","last_change":"2024-03-06T07:23:33.098933+0000","last_active":"2024-03-06T07:23:34.096574+0000","last_peered":"2024-03-06T07:23:34.096574+0000","last_clean":"2024-03-06T07:23:34.096574+0000","last_became_active":"2024-03-06T07:23:33.098660+0000","last_became_peered":"2024-03-06T07:23:33.098660+0000","last_unstale":"2024-03-06T07:23:34.096574+0000","last_undegraded":"2024-03-06T07:23:34.096574+0000","last_fullsized":"2024-03-06T07:23:34.096574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:24.949138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085738+0000","last_change":"2024-03-06T07:23:32.097368+0000","last_active":"2024-03-06T07:23:35.085738+0000","last_peered":"2024-03-06T07:23:35.085738+0000","last_clean":"2024-03-06T07:23:35.085738+0000","last_became_active":"2024-03-06T07:23:32.097100+0000","last_became_peered":"2024-03-06T07:23:32.097100+0000","last_unstale":"2024-03-06T07:23:35.085738+0000","last_undegraded":"2024-03-06T07:23:35.085738+0000","last_fullsized":"2024-03-06T07:23:35.085738+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:45.604799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859029+0000","last_change":"2024-03-06T07:23:33.094069+0000","last_active":"2024-03-06T07:23:34.859029+0000","last_peered":"2024-03-06T07:23:34.859029+0000","last_clean":"2024-03-06T07:23:34.859029+0000","last_became_active":"2024-03-06T07:23:33.093790+0000","last_became_peered":"2024-03-06T07:23:33.093790+0000","last_unstale":"2024-03-06T07:23:34.859029+0000","last_undegraded":"2024-03-06T07:23:34.859029+0000","last_fullsized":"2024-03-06T07:23:34.859029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:15.315599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783472+0000","last_change":"2024-03-06T07:23:32.094421+0000","last_active":"2024-03-06T07:23:34.783472+0000","last_peered":"2024-03-06T07:23:34.783472+0000","last_clean":"2024-03-06T07:23:34.783472+0000","last_became_active":"2024-03-06T07:23:32.094161+0000","last_became_peered":"2024-03-06T07:23:32.094161+0000","last_unstale":"2024-03-06T07:23:34.783472+0000","last_undegraded":"2024-03-06T07:23:34.783472+0000","last_fullsized":"2024-03-06T07:23:34.783472+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:00.492894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085425+0000","last_change":"2024-03-06T07:23:33.095705+0000","last_active":"2024-03-06T07:23:35.085425+0000","last_peered":"2024-03-06T07:23:35.085425+0000","last_clean":"2024-03-06T07:23:35.085425+0000","last_became_active":"2024-03-06T07:23:33.095446+0000","last_became_peered":"2024-03-06T07:23:33.095446+0000","last_unstale":"2024-03-06T07:23:35.085425+0000","last_undegraded":"2024-03-06T07:23:35.085425+0000","last_fullsized":"2024-03-06T07:23:35.085425+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:09.517197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085697+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:35.085697+0000","last_peered":"2024-03-06T07:23:35.085697+0000","last_clean":"2024-03-06T07:23:35.085697+0000","last_became_active":"2024-03-06T07:23:32.096097+0000","last_became_peered":"2024-03-06T07:23:32.096097+0000","last_unstale":"2024-03-06T07:23:35.085697+0000","last_undegraded":"2024-03-06T07:23:35.085697+0000","last_fullsized":"2024-03-06T07:23:35.085697+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:14.310832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783887+0000","last_change":"2024-03-06T07:23:33.097365+0000","last_active":"2024-03-06T07:23:34.783887+0000","last_peered":"2024-03-06T07:23:34.783887+0000","last_clean":"2024-03-06T07:23:34.783887+0000","last_became_active":"2024-03-06T07:23:33.097107+0000","last_became_peered":"2024-03-06T07:23:33.097107+0000","last_unstale":"2024-03-06T07:23:34.783887+0000","last_undegraded":"2024-03-06T07:23:34.783887+0000","last_fullsized":"2024-03-06T07:23:34.783887+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:01.549347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858888+0000","last_change":"2024-03-06T07:23:32.095330+0000","last_active":"2024-03-06T07:23:34.858888+0000","last_peered":"2024-03-06T07:23:34.858888+0000","last_clean":"2024-03-06T07:23:34.858888+0000","last_became_active":"2024-03-06T07:23:32.094988+0000","last_became_peered":"2024-03-06T07:23:32.094988+0000","last_unstale":"2024-03-06T07:23:34.858888+0000","last_undegraded":"2024-03-06T07:23:34.858888+0000","last_fullsized":"2024-03-06T07:23:34.858888+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:16.414419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.094583+0000","last_change":"2024-03-06T07:23:33.093873+0000","last_active":"2024-03-06T07:23:34.094583+0000","last_peered":"2024-03-06T07:23:34.094583+0000","last_clean":"2024-03-06T07:23:34.094583+0000","last_became_active":"2024-03-06T07:23:33.093578+0000","last_became_peered":"2024-03-06T07:23:33.093578+0000","last_unstale":"2024-03-06T07:23:34.094583+0000","last_undegraded":"2024-03-06T07:23:34.094583+0000","last_fullsized":"2024-03-06T07:23:34.094583+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:27.722310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.094428+0000","last_change":"2024-03-06T07:23:32.095660+0000","last_active":"2024-03-06T07:23:34.094428+0000","last_peered":"2024-03-06T07:23:34.094428+0000","last_clean":"2024-03-06T07:23:34.094428+0000","last_became_active":"2024-03-06T07:23:32.094528+0000","last_became_peered":"2024-03-06T07:23:32.094528+0000","last_unstale":"2024-03-06T07:23:34.094428+0000","last_undegraded":"2024-03-06T07:23:34.094428+0000","last_fullsized":"2024-03-06T07:23:34.094428+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:12.793659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859353+0000","last_change":"2024-03-06T07:23:33.107990+0000","last_active":"2024-03-06T07:23:34.859353+0000","last_peered":"2024-03-06T07:23:34.859353+0000","last_clean":"2024-03-06T07:23:34.859353+0000","last_became_active":"2024-03-06T07:23:33.107615+0000","last_became_peered":"2024-03-06T07:23:33.107615+0000","last_unstale":"2024-03-06T07:23:34.859353+0000","last_undegraded":"2024-03-06T07:23:34.859353+0000","last_fullsized":"2024-03-06T07:23:34.859353+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:06.017956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095566+0000","last_change":"2024-03-06T07:23:32.091549+0000","last_active":"2024-03-06T07:23:34.095566+0000","last_peered":"2024-03-06T07:23:34.095566+0000","last_clean":"2024-03-06T07:23:34.095566+0000","last_became_active":"2024-03-06T07:23:32.091217+0000","last_became_peered":"2024-03-06T07:23:32.091217+0000","last_unstale":"2024-03-06T07:23:34.095566+0000","last_undegraded":"2024-03-06T07:23:34.095566+0000","last_fullsized":"2024-03-06T07:23:34.095566+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:16.414419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095934+0000","last_change":"2024-03-06T07:23:33.097443+0000","last_active":"2024-03-06T07:23:34.095934+0000","last_peered":"2024-03-06T07:23:34.095934+0000","last_clean":"2024-03-06T07:23:34.095934+0000","last_became_active":"2024-03-06T07:23:33.096688+0000","last_became_peered":"2024-03-06T07:23:33.096688+0000","last_unstale":"2024-03-06T07:23:34.095934+0000","last_undegraded":"2024-03-06T07:23:34.095934+0000","last_fullsized":"2024-03-06T07:23:34.095934+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:16.367617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782795+0000","last_change":"2024-03-06T07:23:32.096137+0000","last_active":"2024-03-06T07:23:35.782795+0000","last_peered":"2024-03-06T07:23:35.782795+0000","last_clean":"2024-03-06T07:23:35.782795+0000","last_became_active":"2024-03-06T07:23:32.095785+0000","last_became_peered":"2024-03-06T07:23:32.095785+0000","last_unstale":"2024-03-06T07:23:35.782795+0000","last_undegraded":"2024-03-06T07:23:35.782795+0000","last_fullsized":"2024-03-06T07:23:35.782795+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:58.947782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.096164+0000","last_change":"2024-03-06T07:23:33.099659+0000","last_active":"2024-03-06T07:23:34.096164+0000","last_peered":"2024-03-06T07:23:34.096164+0000","last_clean":"2024-03-06T07:23:34.096164+0000","last_became_active":"2024-03-06T07:23:33.099392+0000","last_became_peered":"2024-03-06T07:23:33.099392+0000","last_unstale":"2024-03-06T07:23:34.096164+0000","last_undegraded":"2024-03-06T07:23:34.096164+0000","last_fullsized":"2024-03-06T07:23:34.096164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:04.722479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085698+0000","last_change":"2024-03-06T07:23:32.095556+0000","last_active":"2024-03-06T07:23:35.085698+0000","last_peered":"2024-03-06T07:23:35.085698+0000","last_clean":"2024-03-06T07:23:35.085698+0000","last_became_active":"2024-03-06T07:23:32.094330+0000","last_became_peered":"2024-03-06T07:23:32.094330+0000","last_unstale":"2024-03-06T07:23:35.085698+0000","last_undegraded":"2024-03-06T07:23:35.085698+0000","last_fullsized":"2024-03-06T07:23:35.085698+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:35.882596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085471+0000","last_change":"2024-03-06T07:23:33.099607+0000","last_active":"2024-03-06T07:23:35.085471+0000","last_peered":"2024-03-06T07:23:35.085471+0000","last_clean":"2024-03-06T07:23:35.085471+0000","last_became_active":"2024-03-06T07:23:33.099423+0000","last_became_peered":"2024-03-06T07:23:33.099423+0000","last_unstale":"2024-03-06T07:23:35.085471+0000","last_undegraded":"2024-03-06T07:23:35.085471+0000","last_fullsized":"2024-03-06T07:23:35.085471+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:41.273144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085840+0000","last_change":"2024-03-06T07:23:32.099162+0000","last_active":"2024-03-06T07:23:35.085840+0000","last_peered":"2024-03-06T07:23:35.085840+0000","last_clean":"2024-03-06T07:23:35.085840+0000","last_became_active":"2024-03-06T07:23:32.098923+0000","last_became_peered":"2024-03-06T07:23:32.098923+0000","last_unstale":"2024-03-06T07:23:35.085840+0000","last_undegraded":"2024-03-06T07:23:35.085840+0000","last_fullsized":"2024-03-06T07:23:35.085840+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:42.906958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.714619+0000","last_change":"2024-03-06T07:23:33.094603+0000","last_active":"2024-03-06T07:23:34.714619+0000","last_peered":"2024-03-06T07:23:34.714619+0000","last_clean":"2024-03-06T07:23:34.714619+0000","last_became_active":"2024-03-06T07:23:33.094180+0000","last_became_peered":"2024-03-06T07:23:33.094180+0000","last_unstale":"2024-03-06T07:23:34.714619+0000","last_undegraded":"2024-03-06T07:23:34.714619+0000","last_fullsized":"2024-03-06T07:23:34.714619+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:01.782008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782426+0000","last_change":"2024-03-06T07:23:32.093466+0000","last_active":"2024-03-06T07:23:35.782426+0000","last_peered":"2024-03-06T07:23:35.782426+0000","last_clean":"2024-03-06T07:23:35.782426+0000","last_became_active":"2024-03-06T07:23:32.093178+0000","last_became_peered":"2024-03-06T07:23:32.093178+0000","last_unstale":"2024-03-06T07:23:35.782426+0000","last_undegraded":"2024-03-06T07:23:35.782426+0000","last_fullsized":"2024-03-06T07:23:35.782426+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:26.788560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.094566+0000","last_change":"2024-03-06T07:23:33.090510+0000","last_active":"2024-03-06T07:23:34.094566+0000","last_peered":"2024-03-06T07:23:34.094566+0000","last_clean":"2024-03-06T07:23:34.094566+0000","last_became_active":"2024-03-06T07:23:33.090236+0000","last_became_peered":"2024-03-06T07:23:33.090236+0000","last_unstale":"2024-03-06T07:23:34.094566+0000","last_undegraded":"2024-03-06T07:23:34.094566+0000","last_fullsized":"2024-03-06T07:23:34.094566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:39.098205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858808+0000","last_change":"2024-03-06T07:23:32.098048+0000","last_active":"2024-03-06T07:23:34.858808+0000","last_peered":"2024-03-06T07:23:34.858808+0000","last_clean":"2024-03-06T07:23:34.858808+0000","last_became_active":"2024-03-06T07:23:32.097599+0000","last_became_peered":"2024-03-06T07:23:32.097599+0000","last_unstale":"2024-03-06T07:23:34.858808+0000","last_undegraded":"2024-03-06T07:23:34.858808+0000","last_fullsized":"2024-03-06T07:23:34.858808+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:18.233541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858771+0000","last_change":"2024-03-06T07:23:33.105482+0000","last_active":"2024-03-06T07:23:34.858771+0000","last_peered":"2024-03-06T07:23:34.858771+0000","last_clean":"2024-03-06T07:23:34.858771+0000","last_became_active":"2024-03-06T07:23:33.105220+0000","last_became_peered":"2024-03-06T07:23:33.105220+0000","last_unstale":"2024-03-06T07:23:34.858771+0000","last_undegraded":"2024-03-06T07:23:34.858771+0000","last_fullsized":"2024-03-06T07:23:34.858771+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:25.093558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783387+0000","last_change":"2024-03-06T07:23:32.094948+0000","last_active":"2024-03-06T07:23:34.783387+0000","last_peered":"2024-03-06T07:23:34.783387+0000","last_clean":"2024-03-06T07:23:34.783387+0000","last_became_active":"2024-03-06T07:23:32.094671+0000","last_became_peered":"2024-03-06T07:23:32.094671+0000","last_unstale":"2024-03-06T07:23:34.783387+0000","last_undegraded":"2024-03-06T07:23:34.783387+0000","last_fullsized":"2024-03-06T07:23:34.783387+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:11.255144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.096581+0000","last_change":"2024-03-06T07:23:33.102092+0000","last_active":"2024-03-06T07:23:34.096581+0000","last_peered":"2024-03-06T07:23:34.096581+0000","last_clean":"2024-03-06T07:23:34.096581+0000","last_became_active":"2024-03-06T07:23:33.100960+0000","last_became_peered":"2024-03-06T07:23:33.100960+0000","last_unstale":"2024-03-06T07:23:34.096581+0000","last_undegraded":"2024-03-06T07:23:34.096581+0000","last_fullsized":"2024-03-06T07:23:34.096581+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:01.782008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085753+0000","last_change":"2024-03-06T07:23:32.097587+0000","last_active":"2024-03-06T07:23:35.085753+0000","last_peered":"2024-03-06T07:23:35.085753+0000","last_clean":"2024-03-06T07:23:35.085753+0000","last_became_active":"2024-03-06T07:23:32.096686+0000","last_became_peered":"2024-03-06T07:23:32.096686+0000","last_unstale":"2024-03-06T07:23:35.085753+0000","last_undegraded":"2024-03-06T07:23:35.085753+0000","last_fullsized":"2024-03-06T07:23:35.085753+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:49.517462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859152+0000","last_change":"2024-03-06T07:23:33.105943+0000","last_active":"2024-03-06T07:23:34.859152+0000","last_peered":"2024-03-06T07:23:34.859152+0000","last_clean":"2024-03-06T07:23:34.859152+0000","last_became_active":"2024-03-06T07:23:33.105695+0000","last_became_peered":"2024-03-06T07:23:33.105695+0000","last_unstale":"2024-03-06T07:23:34.859152+0000","last_undegraded":"2024-03-06T07:23:34.859152+0000","last_fullsized":"2024-03-06T07:23:34.859152+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:10.964086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783877+0000","last_change":"2024-03-06T07:23:32.095927+0000","last_active":"2024-03-06T07:23:34.783877+0000","last_peered":"2024-03-06T07:23:34.783877+0000","last_clean":"2024-03-06T07:23:34.783877+0000","last_became_active":"2024-03-06T07:23:32.095354+0000","last_became_peered":"2024-03-06T07:23:32.095354+0000","last_unstale":"2024-03-06T07:23:34.783877+0000","last_undegraded":"2024-03-06T07:23:34.783877+0000","last_fullsized":"2024-03-06T07:23:34.783877+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:24.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783774+0000","last_change":"2024-03-06T07:23:33.096731+0000","last_active":"2024-03-06T07:23:34.783774+0000","last_peered":"2024-03-06T07:23:34.783774+0000","last_clean":"2024-03-06T07:23:34.783774+0000","last_became_active":"2024-03-06T07:23:33.096462+0000","last_became_peered":"2024-03-06T07:23:33.096462+0000","last_unstale":"2024-03-06T07:23:34.783774+0000","last_undegraded":"2024-03-06T07:23:34.783774+0000","last_fullsized":"2024-03-06T07:23:34.783774+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:41.506777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.714577+0000","last_change":"2024-03-06T07:23:32.094186+0000","last_active":"2024-03-06T07:23:34.714577+0000","last_peered":"2024-03-06T07:23:34.714577+0000","last_clean":"2024-03-06T07:23:34.714577+0000","last_became_active":"2024-03-06T07:23:32.093861+0000","last_became_peered":"2024-03-06T07:23:32.093861+0000","last_unstale":"2024-03-06T07:23:34.714577+0000","last_undegraded":"2024-03-06T07:23:34.714577+0000","last_fullsized":"2024-03-06T07:23:34.714577+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:03.717712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085691+0000","last_change":"2024-03-06T07:23:33.104667+0000","last_active":"2024-03-06T07:23:35.085691+0000","last_peered":"2024-03-06T07:23:35.085691+0000","last_clean":"2024-03-06T07:23:35.085691+0000","last_became_active":"2024-03-06T07:23:33.104416+0000","last_became_peered":"2024-03-06T07:23:33.104416+0000","last_unstale":"2024-03-06T07:23:35.085691+0000","last_undegraded":"2024-03-06T07:23:35.085691+0000","last_fullsized":"2024-03-06T07:23:35.085691+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:55.306014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783590+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:34.783590+0000","last_peered":"2024-03-06T07:23:34.783590+0000","last_clean":"2024-03-06T07:23:34.783590+0000","last_became_active":"2024-03-06T07:23:32.094821+0000","last_became_peered":"2024-03-06T07:23:32.094821+0000","last_unstale":"2024-03-06T07:23:34.783590+0000","last_undegraded":"2024-03-06T07:23:34.783590+0000","last_fullsized":"2024-03-06T07:23:34.783590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:04.671716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.085633+0000","last_change":"2024-03-06T07:23:33.098812+0000","last_active":"2024-03-06T07:23:35.085633+0000","last_peered":"2024-03-06T07:23:35.085633+0000","last_clean":"2024-03-06T07:23:35.085633+0000","last_became_active":"2024-03-06T07:23:33.098567+0000","last_became_peered":"2024-03-06T07:23:33.098567+0000","last_unstale":"2024-03-06T07:23:35.085633+0000","last_undegraded":"2024-03-06T07:23:35.085633+0000","last_fullsized":"2024-03-06T07:23:35.085633+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:58.112016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095601+0000","last_change":"2024-03-06T07:23:32.095102+0000","last_active":"2024-03-06T07:23:34.095601+0000","last_peered":"2024-03-06T07:23:34.095601+0000","last_clean":"2024-03-06T07:23:34.095601+0000","last_became_active":"2024-03-06T07:23:32.094829+0000","last_became_peered":"2024-03-06T07:23:32.094829+0000","last_unstale":"2024-03-06T07:23:34.095601+0000","last_undegraded":"2024-03-06T07:23:34.095601+0000","last_fullsized":"2024-03-06T07:23:34.095601+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:18.233541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782492+0000","last_change":"2024-03-06T07:23:33.103093+0000","last_active":"2024-03-06T07:23:35.782492+0000","last_peered":"2024-03-06T07:23:35.782492+0000","last_clean":"2024-03-06T07:23:35.782492+0000","last_became_active":"2024-03-06T07:23:33.102827+0000","last_became_peered":"2024-03-06T07:23:33.102827+0000","last_unstale":"2024-03-06T07:23:35.782492+0000","last_undegraded":"2024-03-06T07:23:35.782492+0000","last_fullsized":"2024-03-06T07:23:35.782492+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:15.315599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.714507+0000","last_change":"2024-03-06T07:23:32.091753+0000","last_active":"2024-03-06T07:23:34.714507+0000","last_peered":"2024-03-06T07:23:34.714507+0000","last_clean":"2024-03-06T07:23:34.714507+0000","last_became_active":"2024-03-06T07:23:32.091465+0000","last_became_peered":"2024-03-06T07:23:32.091465+0000","last_unstale":"2024-03-06T07:23:34.714507+0000","last_undegraded":"2024-03-06T07:23:34.714507+0000","last_fullsized":"2024-03-06T07:23:34.714507+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:43.440959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.095063+0000","last_change":"2024-03-06T07:23:33.097099+0000","last_active":"2024-03-06T07:23:34.095063+0000","last_peered":"2024-03-06T07:23:34.095063+0000","last_clean":"2024-03-06T07:23:34.095063+0000","last_became_active":"2024-03-06T07:23:33.096806+0000","last_became_peered":"2024-03-06T07:23:33.096806+0000","last_unstale":"2024-03-06T07:23:34.095063+0000","last_undegraded":"2024-03-06T07:23:34.095063+0000","last_fullsized":"2024-03-06T07:23:34.095063+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:18.227967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.858945+0000","last_change":"2024-03-06T07:23:32.096349+0000","last_active":"2024-03-06T07:23:34.858945+0000","last_peered":"2024-03-06T07:23:34.858945+0000","last_clean":"2024-03-06T07:23:34.858945+0000","last_became_active":"2024-03-06T07:23:32.096065+0000","last_became_peered":"2024-03-06T07:23:32.096065+0000","last_unstale":"2024-03-06T07:23:34.858945+0000","last_undegraded":"2024-03-06T07:23:34.858945+0000","last_fullsized":"2024-03-06T07:23:34.858945+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:41.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.859358+0000","last_change":"2024-03-06T07:23:33.098785+0000","last_active":"2024-03-06T07:23:34.859358+0000","last_peered":"2024-03-06T07:23:34.859358+0000","last_clean":"2024-03-06T07:23:34.859358+0000","last_became_active":"2024-03-06T07:23:33.096979+0000","last_became_peered":"2024-03-06T07:23:33.096979+0000","last_unstale":"2024-03-06T07:23:34.859358+0000","last_undegraded":"2024-03-06T07:23:34.859358+0000","last_fullsized":"2024-03-06T07:23:34.859358+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:32.073823+0000","last_clean_scrub_stamp":"2024-03-06T07:23:32.073823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:35.867592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:35.782853+0000","last_change":"2024-03-06T07:23:32.095066+0000","last_active":"2024-03-06T07:23:35.782853+0000","last_peered":"2024-03-06T07:23:35.782853+0000","last_clean":"2024-03-06T07:23:35.782853+0000","last_became_active":"2024-03-06T07:23:32.094784+0000","last_became_peered":"2024-03-06T07:23:32.094784+0000","last_unstale":"2024-03-06T07:23:35.782853+0000","last_undegraded":"2024-03-06T07:23:35.782853+0000","last_fullsized":"2024-03-06T07:23:35.782853+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:31.069056+0000","last_clean_scrub_stamp":"2024-03-06T07:23:31.069056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:00.492894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":210,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:34.783397+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:23:34.783397+0000","last_peered":"2024-03-06T07:23:34.783397+0000","last_clean":"2024-03-06T07:23:34.783397+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:23:34.783397+0000","last_undegraded":"2024-03-06T07:23:34.783397+0000","last_fullsized":"2024-03-06T07:23:34.783397+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":804110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":536576,"data_stored":214324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":213722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35872,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335968,"statfs":{"total":96636764160,"available":96600031232,"internally_reserved":0,"allocated":540672,"data_stored":215264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":214290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":215286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":213722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":804054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:40.939 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:40.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:41.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.081+0000 7fc9700e1640 1 Processor -- start 2024-03-06T07:23:41.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.082+0000 7fc9700e1640 1 -- start start 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.082+0000 7fc9700e1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc9680d4860 0x7fc9680d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.082+0000 7fc9700e1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc9680b0e30 0x7fc9680b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.082+0000 7fc9700e1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9681618b0 con 0x7fc9681614d0 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.082+0000 7fc9700e1640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9680d4cf0 con 0x7fc9680d4860 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.082+0000 7fc9700e1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9680cf400 con 0x7fc9680b0e30 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.082+0000 7fc96d67c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc9680d4860 0x7fc9680d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.083+0000 7fc96d67c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc9680d4860 0x7fc9680d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49182/0 (socket says 172.21.15.77:49182) 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.083+0000 7fc96d67c640 1 -- 172.21.15.77:0/1949456100 learned_addr learned my addr 172.21.15.77:0/1949456100 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.083+0000 7fc96e67e640 1 --2- 172.21.15.77:0/1949456100 >> v2:172.21.15.158:3300/0 conn(0x7fc9680b0e30 0x7fc9680b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.083+0000 7fc96d67c640 1 -- 172.21.15.77:0/1949456100 >> v2:172.21.15.158:3300/0 conn(0x7fc9680b0e30 msgr2=0x7fc9680b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.083+0000 7fc96d67c640 1 --2- 172.21.15.77:0/1949456100 >> v2:172.21.15.158:3300/0 conn(0x7fc9680b0e30 0x7fc9680b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.083+0000 7fc96d67c640 1 -- 172.21.15.77:0/1949456100 >> v1:172.21.15.158:6789/0 conn(0x7fc9681614d0 legacy=0x7fc9680d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.083+0000 7fc96d67c640 1 -- 172.21.15.77:0/1949456100 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680cf6a0 con 0x7fc9680d4860 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.084+0000 7fc96d67c640 1 --2- 172.21.15.77:0/1949456100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9680d4860 0x7fc9680d2020 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc9680ce550 tx=0x7fc95802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a1892d21e4f7ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.084+0000 7fc96ce7b640 1 -- 172.21.15.77:0/1949456100 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958035020 con 0x7fc9680d4860 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.084+0000 7fc96ce7b640 1 -- 172.21.15.77:0/1949456100 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc958031640 con 0x7fc9680d4860 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.084+0000 7fc96ce7b640 1 -- 172.21.15.77:0/1949456100 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95802d040 con 0x7fc9680d4860 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.085+0000 7fc9700e1640 1 -- 172.21.15.77:0/1949456100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9680d4860 msgr2=0x7fc9680d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.085+0000 7fc9700e1640 1 --2- 172.21.15.77:0/1949456100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9680d4860 0x7fc9680d2020 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc9680ce550 tx=0x7fc95802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.085+0000 7fc9700e1640 1 -- 172.21.15.77:0/1949456100 shutdown_connections 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.085+0000 7fc9700e1640 1 --2- 172.21.15.77:0/1949456100 >> v2:172.21.15.158:3300/0 conn(0x7fc9680b0e30 0x7fc9680b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.085+0000 7fc9700e1640 1 --2- 172.21.15.77:0/1949456100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9680d4860 0x7fc9680d2020 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.085+0000 7fc9700e1640 1 -- 172.21.15.77:0/1949456100 >> 172.21.15.77:0/1949456100 conn(0x7fc9680d3d40 msgr2=0x7fc9680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.085+0000 7fc9700e1640 1 -- 172.21.15.77:0/1949456100 shutdown_connections 2024-03-06T07:23:41.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.085+0000 7fc9700e1640 1 -- 172.21.15.77:0/1949456100 wait complete. 2024-03-06T07:23:41.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.086+0000 7fc9700e1640 1 Processor -- start 2024-03-06T07:23:41.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.086+0000 7fc9700e1640 1 -- start start 2024-03-06T07:23:41.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc9700e1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9680b0e30 0x7fc9680c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc9700e1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc9680d4860 0x7fc9680ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc9700e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9681614d0 0x7fc9680cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc9700e1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680b0910 con 0x7fc9681614d0 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc9700e1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680cfc60 con 0x7fc9680b0e30 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc9700e1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680d2560 con 0x7fc9680d4860 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9681614d0 0x7fc9680cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9681614d0 0x7fc9680cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49198/0 (socket says 172.21.15.77:49198) 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 -- 172.21.15.77:0/3839200458 learned_addr learned my addr 172.21.15.77:0/3839200458 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96d67c640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc9680d4860 0x7fc9680ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96de7d640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9680b0e30 0x7fc9680c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 -- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc9680d4860 msgr2=0x7fc9680ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc9680d4860 0x7fc9680ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 -- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9680b0e30 msgr2=0x7fc9680c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9680b0e30 0x7fc9680c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 -- 172.21.15.77:0/3839200458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680b4760 con 0x7fc9681614d0 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96d67c640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc9680d4860 0x7fc9680ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc96e67e640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9681614d0 0x7fc9680cd8a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc964007e40 tx=0x7fc96400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc95e7fc640 1 -- 172.21.15.77:0/3839200458 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964014620 con 0x7fc9681614d0 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc95e7fc640 1 -- 172.21.15.77:0/3839200458 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc964019070 con 0x7fc9681614d0 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc95e7fc640 1 -- 172.21.15.77:0/3839200458 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964014a20 con 0x7fc9681614d0 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9680b4a40 con 0x7fc9681614d0 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.087+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9680b5000 con 0x7fc9681614d0 2024-03-06T07:23:41.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.089+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9680b5f90 con 0x7fc9681614d0 2024-03-06T07:23:41.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.089+0000 7fc95e7fc640 1 -- 172.21.15.77:0/3839200458 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc964014bc0 con 0x7fc9681614d0 2024-03-06T07:23:41.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.089+0000 7fc95e7fc640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc94c06b4d0 0x7fc94c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.090+0000 7fc95e7fc640 1 -- 172.21.15.77:0/3839200458 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc96408a9f0 con 0x7fc9681614d0 2024-03-06T07:23:41.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.090+0000 7fc96de7d640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc94c06b4d0 0x7fc94c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.090+0000 7fc96de7d640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc94c06b4d0 0x7fc94c06d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc960006fd0 tx=0x7fc960008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.095+0000 7fc95e7fc640 1 -- 172.21.15.77:0/3839200458 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc964059900 con 0x7fc9681614d0 2024-03-06T07:23:41.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.341+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc968068ff0 con 0x7fc9681614d0 2024-03-06T07:23:41.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.343+0000 7fc95e7fc640 1 -- 172.21.15.77:0/3839200458 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc964059200 con 0x7fc9681614d0 2024-03-06T07:23:41.345 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:23:41.345 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:23:41.345 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:23:41.345 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:23:41.345 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:41.346 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:23:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc94c06b4d0 msgr2=0x7fc94c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc94c06b4d0 0x7fc94c06d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc960006fd0 tx=0x7fc960008040 comp rx=0 tx=0).stop 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9681614d0 msgr2=0x7fc9680cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9681614d0 0x7fc9680cd8a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc964007e40 tx=0x7fc96400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 shutdown_connections 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc94c06b4d0 0x7fc94c06d990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9681614d0 0x7fc9680cd8a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc9680d4860 0x7fc9680ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 --2- 172.21.15.77:0/3839200458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc9680b0e30 0x7fc9680c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 >> 172.21.15.77:0/3839200458 conn(0x7fc9680d3d40 msgr2=0x7fc9680d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 shutdown_connections 2024-03-06T07:23:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.346+0000 7fc9700e1640 1 -- 172.21.15.77:0/3839200458 wait complete. 2024-03-06T07:23:41.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:41.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.488+0000 7feb50fee640 1 Processor -- start 2024-03-06T07:23:41.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.488+0000 7feb50fee640 1 -- start start 2024-03-06T07:23:41.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb50fee640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb4c13dc10 0x7feb4c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb50fee640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb4c15bf30 0x7feb4c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb50fee640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c15f580 con 0x7feb4c15e850 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb50fee640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c0cf3f0 con 0x7feb4c13dc10 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb50fee640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c15b210 con 0x7feb4c15bf30 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb4c13dc10 0x7feb4c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4ad9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7feb4c15e850 0x7feb4c15ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58594/0 (socket says 172.21.15.77:58594) 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb4c13dc10 0x7feb4c0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49206/0 (socket says 172.21.15.77:49206) 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4ad9d640 1 -- 172.21.15.77:0/2552941570 learned_addr learned my addr 172.21.15.77:0/2552941570 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4a59c640 1 -- 172.21.15.77:0/2552941570 >> v2:172.21.15.158:3300/0 conn(0x7feb4c15bf30 msgr2=0x7feb4c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4a59c640 1 --2- 172.21.15.77:0/2552941570 >> v2:172.21.15.158:3300/0 conn(0x7feb4c15bf30 0x7feb4c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4a59c640 1 -- 172.21.15.77:0/2552941570 >> v1:172.21.15.77:6789/0 conn(0x7feb4c15e850 legacy=0x7feb4c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4a59c640 1 -- 172.21.15.77:0/2552941570 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c15b450 con 0x7feb4c13dc10 2024-03-06T07:23:41.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.489+0000 7feb4959a640 1 -- 172.21.15.77:0/2552941570 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831932940 0 0) 0x7feb4c15f580 con 0x7feb4c15e850 2024-03-06T07:23:41.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.490+0000 7feb4a59c640 1 --2- 172.21.15.77:0/2552941570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c13dc10 0x7feb4c0b2950 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7feb4c06c790 tx=0x7feb4002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14abe1c16e359d4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.490+0000 7feb4959a640 1 -- 172.21.15.77:0/2552941570 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400315a0 con 0x7feb4c13dc10 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.490+0000 7feb4959a640 1 -- 172.21.15.77:0/2552941570 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb40037070 con 0x7feb4c13dc10 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.490+0000 7feb4959a640 1 -- 172.21.15.77:0/2552941570 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40031960 con 0x7feb4c13dc10 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.491+0000 7feb50fee640 1 -- 172.21.15.77:0/2552941570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c13dc10 msgr2=0x7feb4c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.491+0000 7feb50fee640 1 --2- 172.21.15.77:0/2552941570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c13dc10 0x7feb4c0b2950 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7feb4c06c790 tx=0x7feb4002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.491+0000 7feb50fee640 1 -- 172.21.15.77:0/2552941570 shutdown_connections 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.491+0000 7feb50fee640 1 --2- 172.21.15.77:0/2552941570 >> v2:172.21.15.158:3300/0 conn(0x7feb4c15bf30 0x7feb4c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.491+0000 7feb50fee640 1 --2- 172.21.15.77:0/2552941570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c13dc10 0x7feb4c0b2950 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.491+0000 7feb50fee640 1 -- 172.21.15.77:0/2552941570 >> 172.21.15.77:0/2552941570 conn(0x7feb4c0cf8a0 msgr2=0x7feb4c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.491+0000 7feb50fee640 1 -- 172.21.15.77:0/2552941570 shutdown_connections 2024-03-06T07:23:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.491+0000 7feb50fee640 1 -- 172.21.15.77:0/2552941570 wait complete. 2024-03-06T07:23:41.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.492+0000 7feb50fee640 1 Processor -- start 2024-03-06T07:23:41.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.493+0000 7feb50fee640 1 -- start start 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb50fee640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb4c13dc10 0x7feb4c1331e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb50fee640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4c15bf30 0x7feb4c159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb50fee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c15e850 0x7feb4c1685a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb50fee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0cd7a0 con 0x7feb4c15e850 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb50fee640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c13dff0 con 0x7feb4c15bf30 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb50fee640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0cd920 con 0x7feb4c13dc10 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c15e850 0x7feb4c1685a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c15e850 0x7feb4c1685a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49214/0 (socket says 172.21.15.77:49214) 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 -- 172.21.15.77:0/452827298 learned_addr learned my addr 172.21.15.77:0/452827298 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 -- 172.21.15.77:0/452827298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb4c13dc10 msgr2=0x7feb4c1331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb4c13dc10 0x7feb4c1331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 -- 172.21.15.77:0/452827298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4c15bf30 msgr2=0x7feb4c159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb49d9b640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4c15bf30 0x7feb4c159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4c15bf30 0x7feb4c159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 -- 172.21.15.77:0/452827298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c1677e0 con 0x7feb4c15e850 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb4ad9d640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c15e850 0x7feb4c1685a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7feb4400c700 tx=0x7feb4400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb3affd640 1 -- 172.21.15.77:0/452827298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44018070 con 0x7feb4c15e850 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb3affd640 1 -- 172.21.15.77:0/452827298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4400ce10 con 0x7feb4c15e850 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4c167ac0 con 0x7feb4c15e850 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb3affd640 1 -- 172.21.15.77:0/452827298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4400a860 con 0x7feb4c15e850 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.494+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c168080 con 0x7feb4c15e850 2024-03-06T07:23:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.495+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4c06a1b0 con 0x7feb4c15e850 2024-03-06T07:23:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.496+0000 7feb3affd640 1 -- 172.21.15.77:0/452827298 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb4401b070 con 0x7feb4c15e850 2024-03-06T07:23:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.497+0000 7feb3affd640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2c06b4d0 0x7feb2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.497+0000 7feb4a59c640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2c06b4d0 0x7feb2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.497+0000 7feb3affd640 1 -- 172.21.15.77:0/452827298 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb44089a80 con 0x7feb4c15e850 2024-03-06T07:23:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.497+0000 7feb4a59c640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2c06b4d0 0x7feb2c06d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7feb40036000 tx=0x7feb40030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.501+0000 7feb3affd640 1 -- 172.21.15.77:0/452827298 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb4c06a1b0 con 0x7feb4c15e850 2024-03-06T07:23:41.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.709+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb4c04f330 con 0x7feb4c15e850 2024-03-06T07:23:41.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.709+0000 7feb3affd640 1 -- 172.21.15.77:0/452827298 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4977 (secure 0 0 0) 0x7feb440589e0 con 0x7feb4c15e850 2024-03-06T07:23:41.711 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:41.711 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1388591691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1388591691},{"type":"v1","addr":"172.21.15.158:6834","nonce":1388591691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/870048529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":870048529},{"type":"v1","addr":"172.21.15.158:6835","nonce":870048529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/650607050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":650607050},{"type":"v1","addr":"172.21.15.77:6836","nonce":650607050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:32.429377+0000","modified":"2024-03-06T07:23:34.434503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2240574726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2240574726},{"type":"v1","addr":"172.21.15.77:6837","nonce":2240574726}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":3}]} 2024-03-06T07:23:41.711 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 19 2024-03-06T07:23:41.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2c06b4d0 msgr2=0x7feb2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2c06b4d0 0x7feb2c06d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7feb40036000 tx=0x7feb40030000 comp rx=0 tx=0).stop 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c15e850 msgr2=0x7feb4c1685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c15e850 0x7feb4c1685a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7feb4400c700 tx=0x7feb4400cbc0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 shutdown_connections 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2c06b4d0 0x7feb2c06d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4c15e850 0x7feb4c1685a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4c15bf30 0x7feb4c159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 --2- 172.21.15.77:0/452827298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb4c13dc10 0x7feb4c1331e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 >> 172.21.15.77:0/452827298 conn(0x7feb4c0cf8a0 msgr2=0x7feb4c1337c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 shutdown_connections 2024-03-06T07:23:41.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:41.711+0000 7feb50fee640 1 -- 172.21.15.77:0/452827298 wait complete. 2024-03-06T07:23:42.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:32.429377+0000', 'modified': '2024-03-06T07:23:34.434503+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5667}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5667': {'gid': 5667, 'name': 'c', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2240574726', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6835', 'nonce': 2240574726}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2240574726}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5667, 'qdb_cluster': [5667]} 2024-03-06T07:23:42.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:42.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.229+0000 7f3e297ac640 1 Processor -- start 2024-03-06T07:23:42.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.229+0000 7f3e297ac640 1 -- start start 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.230+0000 7f3e297ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e2406e7b0 0x7f3e240cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.230+0000 7f3e297ac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e240b4450 con 0x7f3e2406ee10 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.230+0000 7f3e297ac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e240b5b60 con 0x7f3e24144030 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.230+0000 7f3e297ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240b4a60 con 0x7f3e2406e7b0 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.230+0000 7f3e22ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3e2406ee10 0x7f3e2406f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58606/0 (socket says 172.21.15.77:58606) 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.230+0000 7f3e22ffd640 1 -- 172.21.15.77:0/1658076875 learned_addr learned my addr 172.21.15.77:0/1658076875 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e2406e7b0 0x7f3e240cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67589518 0 0) 0x7f3e240b4450 con 0x7f3e2406ee10 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e10005730 con 0x7f3e2406ee10 2024-03-06T07:23:42.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3619171831 0 0) 0x7f3e10005730 con 0x7f3e2406ee10 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e2406e7b0 msgr2=0x7f3e240cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e2406e7b0 0x7f3e240cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 >> v1:172.21.15.158:6789/0 conn(0x7f3e24144030 legacy=0x7f3e240b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240b5e40 con 0x7f3e2406ee10 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3e14001670 con 0x7f3e2406ee10 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 >> v1:172.21.15.77:6789/0 conn(0x7f3e2406ee10 legacy=0x7f3e2406f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e2406e7b0 0x7f3e10005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e10006170 0x7f3e10006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e10006f50 0x7f3e100073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240b4a60 con 0x7f3e10006f50 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1000dcb0 con 0x7f3e10006170 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1000deb0 con 0x7f3e2406e7b0 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e10006f50 0x7f3e100073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e2406e7b0 0x7f3e10005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e10006170 0x7f3e10006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e237fe640 1 -- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e2406e7b0 msgr2=0x7f3e10005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e2406e7b0 0x7f3e10005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e237fe640 1 -- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e10006170 msgr2=0x7f3e10006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e10006170 0x7f3e10006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e237fe640 1 -- 172.21.15.77:0/1658076875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10005730 con 0x7f3e10006f50 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.231+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e10006170 0x7f3e10006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.232+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e10006f50 0x7f3e100073d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009fc0 tx=0x7f3e10007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e00c891f2df0d68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.232+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c039070 con 0x7f3e10006f50 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.232+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c02cc20 con 0x7f3e10006f50 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.232+0000 7f3e227fc640 1 -- 172.21.15.77:0/1658076875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c009c10 con 0x7f3e10006f50 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.233+0000 7f3e297ac640 1 -- 172.21.15.77:0/1658076875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e10006f50 msgr2=0x7f3e100073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.233+0000 7f3e297ac640 1 --2- 172.21.15.77:0/1658076875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e10006f50 0x7f3e100073d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009fc0 tx=0x7f3e10007a40 comp rx=0 tx=0).stop 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.233+0000 7f3e237fe640 1 -- 172.21.15.77:0/1658076875 reap_dead start 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.233+0000 7f3e297ac640 1 -- 172.21.15.77:0/1658076875 shutdown_connections 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.233+0000 7f3e297ac640 1 -- 172.21.15.77:0/1658076875 >> 172.21.15.77:0/1658076875 conn(0x7f3e240d0a10 msgr2=0x7f3e240d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.233+0000 7f3e297ac640 1 -- 172.21.15.77:0/1658076875 shutdown_connections 2024-03-06T07:23:42.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.233+0000 7f3e297ac640 1 -- 172.21.15.77:0/1658076875 wait complete. 2024-03-06T07:23:42.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.234+0000 7f3e297ac640 1 Processor -- start 2024-03-06T07:23:42.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.234+0000 7f3e297ac640 1 -- start start 2024-03-06T07:23:42.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e297ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e240ce050 0x7f3e240c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e297ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e240c8660 0x7f3e240ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e297ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e240ce970 0x7f3e240cee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e297ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24143c40 con 0x7f3e240ce050 2024-03-06T07:23:42.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e297ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24144840 con 0x7f3e240ce970 2024-03-06T07:23:42.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e297ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24143dc0 con 0x7f3e240c8660 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e240ce050 0x7f3e240c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e240ce050 0x7f3e240c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49228/0 (socket says 172.21.15.77:49228) 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 -- 172.21.15.77:0/3870743958 learned_addr learned my addr 172.21.15.77:0/3870743958 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e237fe640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e240ce970 0x7f3e240cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e240c8660 0x7f3e240ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 -- 172.21.15.77:0/3870743958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e240c8660 msgr2=0x7f3e240ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e240c8660 0x7f3e240ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 -- 172.21.15.77:0/3870743958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e240ce970 msgr2=0x7f3e240cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e240ce970 0x7f3e240cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 -- 172.21.15.77:0/3870743958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240cf4e0 con 0x7f3e240ce050 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e240ce050 0x7f3e240c8120 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3e140072c0 tx=0x7f3e14007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e1b7fe640 1 -- 172.21.15.77:0/3870743958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1400f9a0 con 0x7f3e240ce050 2024-03-06T07:23:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e1b7fe640 1 -- 172.21.15.77:0/3870743958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e14018070 con 0x7f3e240ce050 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e1b7fe640 1 -- 172.21.15.77:0/3870743958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1400fda0 con 0x7f3e240ce050 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e297ac640 1 -- 172.21.15.77:0/3870743958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e241d5350 con 0x7f3e240ce050 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.235+0000 7f3e297ac640 1 -- 172.21.15.77:0/3870743958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e241d5910 con 0x7f3e240ce050 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.237+0000 7f3e1b7fe640 1 -- 172.21.15.77:0/3870743958 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e1401d460 con 0x7f3e240ce050 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.237+0000 7f3e1b7fe640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e0406b500 0x7f3e0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.237+0000 7f3e1b7fe640 1 -- 172.21.15.77:0/3870743958 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e1408a3e0 con 0x7f3e240ce050 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.237+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e0406b500 0x7f3e0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.238+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e0406b500 0x7f3e0406d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e241d8300 tx=0x7f3e0c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.238+0000 7f3e2c392740 1 Processor -- start 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.238+0000 7f3e2c392740 1 -- start start 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca06c870 0x5589ca06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca0708c0 0x5589ca072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca09c600 0x5589ca09ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e2c392740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c9fd66b0 con 0x5589ca0708c0 2024-03-06T07:23:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e2c392740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c9f7fa60 con 0x5589ca09c600 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e2c392740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5589c9fd2750 con 0x5589ca06c870 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca0708c0 0x5589ca072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca0708c0 0x5589ca072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49242/0 (socket says 172.21.15.77:49242) 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e1bfff640 1 -- 172.21.15.77:0/2811022396 learned_addr learned my addr 172.21.15.77:0/2811022396 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca06c870 0x5589ca06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e237fe640 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca09c600 0x5589ca09ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e1bfff640 1 -- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca06c870 msgr2=0x5589ca06cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca06c870 0x5589ca06cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e1bfff640 1 -- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca09c600 msgr2=0x5589ca09ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca09c600 0x5589ca09ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e1bfff640 1 -- 172.21.15.77:0/2811022396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589c9f9df30 con 0x5589ca0708c0 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.239+0000 7f3e237fe640 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca09c600 0x5589ca09ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca0708c0 0x5589ca072360 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0015d0 tx=0x7f3e0c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4e2163ffaf9668 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e18ff9640 1 -- 172.21.15.77:0/2811022396 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c027070 con 0x5589ca0708c0 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e18ff9640 1 -- 172.21.15.77:0/2811022396 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0c014800 con 0x5589ca0708c0 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e18ff9640 1 -- 172.21.15.77:0/2811022396 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c014bb0 con 0x5589ca0708c0 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e2c392740 1 -- 172.21.15.77:0/2811022396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca0708c0 msgr2=0x5589ca072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca0708c0 0x5589ca072360 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0015d0 tx=0x7f3e0c013000 comp rx=0 tx=0).stop 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e2c392740 1 -- 172.21.15.77:0/2811022396 shutdown_connections 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca0708c0 0x5589ca072360 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca09c600 0x5589ca09ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2811022396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca06c870 0x5589ca06cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.240+0000 7f3e2c392740 1 -- 172.21.15.77:0/2811022396 >> 172.21.15.77:0/2811022396 conn(0x5589ca06e8f0 msgr2=0x5589ca0701b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.241+0000 7f3e2c392740 1 -- 172.21.15.77:0/2811022396 shutdown_connections 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.241+0000 7f3e2c392740 1 -- 172.21.15.77:0/2811022396 wait complete. 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.241+0000 7f3e2c392740 1 -- start start 2024-03-06T07:23:42.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.241+0000 7f3e2c392740 1 Processor -- start 2024-03-06T07:23:42.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca080560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca098190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 0x5589ca09d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e2c392740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589ca070d50 con 0x5589ca09c600 2024-03-06T07:23:42.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e2c392740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c9f92ea0 con 0x5589ca07d580 2024-03-06T07:23:42.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e2c392740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5589ca071030 con 0x5589ca0986d0 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e2c392740 10 client.0 _ll_register_callbacks cb 0x5589c9f92870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e22ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca080560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e22ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca080560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49250/0 (socket says 172.21.15.77:49250) 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e22ffd640 1 -- 172.21.15.77:0/1917605079 learned_addr learned my addr 172.21.15.77:0/1917605079 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 0x5589ca09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca098190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e22ffd640 1 -- 172.21.15.77:0/1917605079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 msgr2=0x5589ca09d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 0x5589ca09d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e22ffd640 1 -- 172.21.15.77:0/1917605079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 msgr2=0x5589ca098190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca098190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.242+0000 7f3e22ffd640 1 -- 172.21.15.77:0/1917605079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589c9f93270 con 0x5589ca09c600 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca080560 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400f5a0 tx=0x7f3e14059260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a619dc0140ce84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:23:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e01ffb640 1 -- 172.21.15.77:0/1917605079 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14061270 con 0x5589ca09c600 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e01ffb640 1 -- 172.21.15.77:0/1917605079 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e1405dbd0 con 0x5589ca09c600 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 1 -- 172.21.15.77:0/1917605079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e0c014800 con 0x5589ca09c600 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e01ffb640 1 -- 172.21.15.77:0/1917605079 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e1405ddd0 con 0x5589ca09c600 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 10 client.5847 fetch_fsmap learned FSMap version 19 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 1 -- 172.21.15.77:0/1917605079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x5589c9ceb2d0 con 0x5589ca09c600 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e01ffb640 1 -- 172.21.15.77:0/1917605079 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0c014800 con 0x5589ca09c600 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e01ffb640 1 -- 172.21.15.77:0/1917605079 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 19) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3e14060de0 con 0x5589ca09c600 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 10 client.5847 fetch_fsmap finished waiting for FSMap version 19 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 10 client.5847 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 20 client.5847 populate_metadata read hostname 'smithi077' 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] conn(0x5589ca09eae0 0x5589ca0d6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 4 client.5847 mds_command: new command op to 5667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e2c392740 1 -- 172.21.15.77:0/1917605079 --> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5589c9fe3bf0 con 0x5589ca09eae0 2024-03-06T07:23:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.243+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] conn(0x5589ca09eae0 0x5589ca0d6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.244+0000 7f3e01ffb640 10 client.5847 ms_handle_connect on v2:172.21.15.77:6835/2240574726 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.244+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] conn(0x5589ca09eae0 0x5589ca0d6f30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97e3f4ed0e3c7ada server_cookie=5099311530674857 in_seq=0 out_seq=0 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e01ffb640 1 -- 172.21.15.77:0/1917605079 <== mds.0 v2:172.21.15.77:6835/2240574726 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5589c9fe3bf0 con 0x5589ca09eae0 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e01ffb640 10 client.5847 handle_command_reply: tid=0 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 client.5847 shutdown 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 -- 172.21.15.77:0/1917605079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 msgr2=0x5589ca080560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca080560 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400f5a0 tx=0x7f3e14059260 comp rx=0 tx=0).stop 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 -- 172.21.15.77:0/1917605079 shutdown_connections 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] conn(0x5589ca09eae0 0x5589ca0d6f30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 0x5589ca09d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca098190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1917605079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca080560 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 -- 172.21.15.77:0/1917605079 >> 172.21.15.77:0/1917605079 conn(0x5589ca06c870 msgr2=0x5589ca06b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e01ffb640 0 client.5847 ms_handle_reset on v2:172.21.15.77:6835/2240574726 2024-03-06T07:23:42.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 -- 172.21.15.77:0/1917605079 shutdown_connections 2024-03-06T07:23:42.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 1 -- 172.21.15.77:0/1917605079 wait complete. 2024-03-06T07:23:42.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.245+0000 7f3e2c392740 20 client.5847 trim_cache size 0 max 16384 2024-03-06T07:23:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e2c392740 1 Processor -- start 2024-03-06T07:23:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e2c392740 1 -- start start 2024-03-06T07:23:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca0920c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 0x5589ca06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e2c392740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e241ffd20 con 0x5589ca09c600 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e2c392740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e241e4a80 con 0x5589ca07d580 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e2c392740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e241ff750 con 0x5589ca0986d0 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca072390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49256/0 (socket says 172.21.15.77:49256) 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.262+0000 7f3e237fe640 1 -- 172.21.15.77:0/1692373464 learned_addr learned my addr 172.21.15.77:0/1692373464 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca0920c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e1bfff640 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 0x5589ca06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e237fe640 1 -- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 msgr2=0x5589ca06b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 0x5589ca06b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e237fe640 1 -- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 msgr2=0x5589ca0920c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca0920c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e237fe640 1 -- 172.21.15.77:0/1692373464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589ca098500 con 0x5589ca09c600 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca0920c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e237fe640 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca072390 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c02c690 tx=0x7f3e1c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f6650f50f3f056 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e03fff640 1 -- 172.21.15.77:0/1692373464 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c039070 con 0x5589ca09c600 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e03fff640 1 -- 172.21.15.77:0/1692373464 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c03a040 con 0x5589ca09c600 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e03fff640 1 -- 172.21.15.77:0/1692373464 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c02cc20 con 0x5589ca09c600 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e2c392740 1 -- 172.21.15.77:0/1692373464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 msgr2=0x5589ca072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.263+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca072390 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c02c690 tx=0x7f3e1c02e070 comp rx=0 tx=0).stop 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.264+0000 7f3e2c392740 1 -- 172.21.15.77:0/1692373464 shutdown_connections 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.264+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca072390 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.264+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca0986d0 0x5589ca06b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.264+0000 7f3e2c392740 1 --2- 172.21.15.77:0/1692373464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca07d580 0x5589ca0920c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.264+0000 7f3e2c392740 1 -- 172.21.15.77:0/1692373464 >> 172.21.15.77:0/1692373464 conn(0x5589ca06c870 msgr2=0x5589ca0935b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.264+0000 7f3e2c392740 1 -- 172.21.15.77:0/1692373464 shutdown_connections 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.264+0000 7f3e2c392740 1 -- 172.21.15.77:0/1692373464 wait complete. 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.264+0000 7f3e2c392740 1 -- start start 2024-03-06T07:23:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.265+0000 7f3e2c392740 1 Processor -- start 2024-03-06T07:23:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca07d580 0x5589ca126c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca0986d0 0x5589ca070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e2c392740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca0714a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e2c392740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589ca094170 con 0x5589ca09c600 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e2c392740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589ca094f40 con 0x5589ca0986d0 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e2c392740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5589ca095800 con 0x5589ca07d580 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e2c392740 10 client.0 _ll_register_callbacks cb 0x7f3e140586e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca0714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca0714a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49262/0 (socket says 172.21.15.77:49262) 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 -- 172.21.15.77:0/2113105120 learned_addr learned my addr 172.21.15.77:0/2113105120 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 -- 172.21.15.77:0/2113105120 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca07d580 msgr2=0x5589ca126c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca07d580 0x5589ca126c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 -- 172.21.15.77:0/2113105120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca0986d0 msgr2=0x5589ca070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca0986d0 0x5589ca070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 -- 172.21.15.77:0/2113105120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589c9db56b0 con 0x5589ca09c600 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e237fe640 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca0714a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c02d000 tx=0x7f3e1c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bf53830f0d9baf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e01ffb640 1 -- 172.21.15.77:0/2113105120 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c039070 con 0x5589ca09c600 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e01ffb640 1 -- 172.21.15.77:0/2113105120 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e1c003ba0 con 0x5589ca09c600 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.266+0000 7f3e2c392740 1 -- 172.21.15.77:0/2113105120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e1c03a040 con 0x5589ca09c600 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e01ffb640 1 -- 172.21.15.77:0/2113105120 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e1c052050 con 0x5589ca09c600 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 10 client.5853 fetch_fsmap learned FSMap version 19 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 1 -- 172.21.15.77:0/2113105120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x5589ca0dbbb0 con 0x5589ca09c600 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e01ffb640 1 -- 172.21.15.77:0/2113105120 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c03a040 con 0x5589ca09c600 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e01ffb640 1 -- 172.21.15.77:0/2113105120 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 19) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3e1c055020 con 0x5589ca09c600 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 10 client.5853 fetch_fsmap finished waiting for FSMap version 19 2024-03-06T07:23:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 10 client.5853 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 20 client.5853 populate_metadata read hostname 'smithi077' 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] conn(0x5589ca12a840 0x5589ca12ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 4 client.5853 mds_command: new command op to 5667 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 1 -- 172.21.15.77:0/2113105120 --> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5589c9fddf40 con 0x5589ca12a840 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] conn(0x5589ca12a840 0x5589ca12ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e22ffd640 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] conn(0x5589ca12a840 0x5589ca12ac20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e460a3b2f7b7a723 server_cookie=732d9d28229ca3f3 in_seq=0 out_seq=0 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e01ffb640 10 client.5853 ms_handle_connect on v2:172.21.15.77:6835/2240574726 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e01ffb640 1 -- 172.21.15.77:0/2113105120 <== mds.0 v2:172.21.15.77:6835/2240574726 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5589c9fddf40 con 0x5589ca12a840 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e01ffb640 10 client.5853 handle_command_reply: tid=0 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.267+0000 7f3e2c392740 1 client.5853 shutdown 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 -- 172.21.15.77:0/2113105120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 msgr2=0x5589ca0714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca0714a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c02d000 tx=0x7f3e1c004ab0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 -- 172.21.15.77:0/2113105120 shutdown_connections 2024-03-06T07:23:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.77:6835/2240574726,v1:172.21.15.77:6837/2240574726] conn(0x5589ca12a840 0x5589ca12ac20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca09c600 0x5589ca0714a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5589ca0986d0 0x5589ca070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e01ffb640 0 client.5853 ms_handle_reset on v2:172.21.15.77:6835/2240574726 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 --2- 172.21.15.77:0/2113105120 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5589ca07d580 0x5589ca126c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 -- 172.21.15.77:0/2113105120 >> 172.21.15.77:0/2113105120 conn(0x5589ca06c870 msgr2=0x5589ca0935b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 -- 172.21.15.77:0/2113105120 shutdown_connections 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 1 -- 172.21.15.77:0/2113105120 wait complete. 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e2c392740 20 client.5853 trim_cache size 0 max 16384 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:23:42.276 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5667, 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 9.1883585970000006, 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 19, 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 45, 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 45, 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 14.538956922000001 2024-03-06T07:23:42.277 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:23:42.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 -- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e0406b500 msgr2=0x7f3e0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e0406b500 0x7f3e0406d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e241d8300 tx=0x7f3e0c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:42.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 -- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e240ce050 msgr2=0x7f3e240c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e240ce050 0x7f3e240c8120 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3e140072c0 tx=0x7f3e14007780 comp rx=0 tx=0).stop 2024-03-06T07:23:42.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 -- 172.21.15.77:0/3870743958 shutdown_connections 2024-03-06T07:23:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e240ce970 0x7f3e240cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e240c8660 0x7f3e240ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e0406b500 0x7f3e0406d9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 --2- 172.21.15.77:0/3870743958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e240ce050 0x7f3e240c8120 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 -- 172.21.15.77:0/3870743958 >> 172.21.15.77:0/3870743958 conn(0x7f3e240d0a10 msgr2=0x7f3e24141210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 -- 172.21.15.77:0/3870743958 shutdown_connections 2024-03-06T07:23:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.268+0000 7f3e297ac640 1 -- 172.21.15.77:0/3870743958 wait complete. 2024-03-06T07:23:42.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:42.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.388+0000 7f1d7f01c640 1 Processor -- start 2024-03-06T07:23:42.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.388+0000 7f1d7f01c640 1 -- start start 2024-03-06T07:23:42.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d7f01c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d780cf850 0x7f1d780cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d7f01c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1d7806cf20 0x7f1d7806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d7f01c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7815e380 con 0x7f1d780d00c0 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d7f01c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7806ca90 con 0x7f1d780cf850 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d7f01c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d780cdc10 con 0x7f1d7806cf20 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d77fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d780cf850 0x7f1d780cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d77fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d780cf850 0x7f1d780cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49266/0 (socket says 172.21.15.77:49266) 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d77fff640 1 -- 172.21.15.77:0/3097038561 learned_addr learned my addr 172.21.15.77:0/3097038561 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d7d5b9640 1 --2- 172.21.15.77:0/3097038561 >> v2:172.21.15.158:3300/0 conn(0x7f1d7806cf20 0x7f1d7806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d77fff640 1 -- 172.21.15.77:0/3097038561 >> v2:172.21.15.158:3300/0 conn(0x7f1d7806cf20 msgr2=0x7f1d7806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d77fff640 1 --2- 172.21.15.77:0/3097038561 >> v2:172.21.15.158:3300/0 conn(0x7f1d7806cf20 0x7f1d7806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d77fff640 1 -- 172.21.15.77:0/3097038561 >> v1:172.21.15.158:6789/0 conn(0x7f1d780d00c0 legacy=0x7f1d780d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.389+0000 7f1d77fff640 1 -- 172.21.15.77:0/3097038561 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7806c160 con 0x7f1d780cf850 2024-03-06T07:23:42.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.390+0000 7f1d77fff640 1 --2- 172.21.15.77:0/3097038561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d780cf850 0x7f1d780cd3c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1d78071f50 tx=0x7f1d6802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a543401331f803 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d777fe640 1 -- 172.21.15.77:0/3097038561 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68034070 con 0x7f1d780cf850 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d777fe640 1 -- 172.21.15.77:0/3097038561 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d68030860 con 0x7f1d780cf850 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d777fe640 1 -- 172.21.15.77:0/3097038561 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68030c30 con 0x7f1d780cf850 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d7f01c640 1 -- 172.21.15.77:0/3097038561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d780cf850 msgr2=0x7f1d780cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/3097038561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d780cf850 0x7f1d780cd3c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1d78071f50 tx=0x7f1d6802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d7f01c640 1 -- 172.21.15.77:0/3097038561 shutdown_connections 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/3097038561 >> v2:172.21.15.158:3300/0 conn(0x7f1d7806cf20 0x7f1d7806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/3097038561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d780cf850 0x7f1d780cd3c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d7f01c640 1 -- 172.21.15.77:0/3097038561 >> 172.21.15.77:0/3097038561 conn(0x7f1d780b35a0 msgr2=0x7f1d780b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d7f01c640 1 -- 172.21.15.77:0/3097038561 shutdown_connections 2024-03-06T07:23:42.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.391+0000 7f1d7f01c640 1 -- 172.21.15.77:0/3097038561 wait complete. 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.392+0000 7f1d7f01c640 1 Processor -- start 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.392+0000 7f1d7f01c640 1 -- start start 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7f01c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d7806cf20 0x7f1d78159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7f01c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d780cf850 0x7f1d7815a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7f01c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d780d00c0 0x7f1d780ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7f01c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d780cd900 con 0x7f1d7806cf20 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7f01c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7806c720 con 0x7f1d780cf850 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7f01c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d780b1030 con 0x7f1d780d00c0 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7cdb8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d7806cf20 0x7f1d78159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7cdb8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d7806cf20 0x7f1d78159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49272/0 (socket says 172.21.15.77:49272) 2024-03-06T07:23:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.393+0000 7f1d7cdb8640 1 -- 172.21.15.77:0/1433624010 learned_addr learned my addr 172.21.15.77:0/1433624010 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7d5b9640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d780d00c0 0x7f1d780ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d77fff640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d780cf850 0x7f1d7815a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7cdb8640 1 -- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d780d00c0 msgr2=0x7f1d780ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7cdb8640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d780d00c0 0x7f1d780ae810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7cdb8640 1 -- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d780cf850 msgr2=0x7f1d7815a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7cdb8640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d780cf850 0x7f1d7815a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7cdb8640 1 -- 172.21.15.77:0/1433624010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d781f5050 con 0x7f1d7806cf20 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d77fff640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d780cf850 0x7f1d7815a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7cdb8640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d7806cf20 0x7f1d78159b40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1d7000c730 tx=0x7f1d7000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d757fa640 1 -- 172.21.15.77:0/1433624010 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70016020 con 0x7f1d7806cf20 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d757fa640 1 -- 172.21.15.77:0/1433624010 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d70006d50 con 0x7f1d7806cf20 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d781f5330 con 0x7f1d7806cf20 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d757fa640 1 -- 172.21.15.77:0/1433624010 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70003bb0 con 0x7f1d7806cf20 2024-03-06T07:23:42.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.394+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d781f58f0 con 0x7f1d7806cf20 2024-03-06T07:23:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.395+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7804f330 con 0x7f1d7806cf20 2024-03-06T07:23:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.395+0000 7f1d757fa640 1 -- 172.21.15.77:0/1433624010 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d7000a850 con 0x7f1d7806cf20 2024-03-06T07:23:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.396+0000 7f1d757fa640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d5806b500 0x7f1d5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.396+0000 7f1d757fa640 1 -- 172.21.15.77:0/1433624010 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d70089da0 con 0x7f1d7806cf20 2024-03-06T07:23:42.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.396+0000 7f1d77fff640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d5806b500 0x7f1d5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.396+0000 7f1d77fff640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d5806b500 0x7f1d5806d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1d68036000 tx=0x7f1d6803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.404+0000 7f1d757fa640 1 -- 172.21.15.77:0/1433624010 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d70020020 con 0x7f1d7806cf20 2024-03-06T07:23:42.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.615+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1d780ced60 con 0x7f1d7806cf20 2024-03-06T07:23:42.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.615+0000 7f1d757fa640 1 -- 172.21.15.77:0/1433624010 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1d70019310 con 0x7f1d7806cf20 2024-03-06T07:23:42.617 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:42.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d5806b500 msgr2=0x7f1d5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d5806b500 0x7f1d5806d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1d68036000 tx=0x7f1d6803b040 comp rx=0 tx=0).stop 2024-03-06T07:23:42.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d7806cf20 msgr2=0x7f1d78159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d7806cf20 0x7f1d78159b40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1d7000c730 tx=0x7f1d7000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 shutdown_connections 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d780d00c0 0x7f1d780ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d780cf850 0x7f1d7815a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d5806b500 0x7f1d5806d9c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 --2- 172.21.15.77:0/1433624010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d7806cf20 0x7f1d78159b40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 >> 172.21.15.77:0/1433624010 conn(0x7f1d780b35a0 msgr2=0x7f1d780702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 shutdown_connections 2024-03-06T07:23:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:42.617+0000 7f1d7f01c640 1 -- 172.21.15.77:0/1433624010 wait complete. 2024-03-06T07:23:42.630 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:42.630 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:42.630 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:23:42.630 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:42.630 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:42.631 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:42.631 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:42.631 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:42.631 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:42.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:42.631 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:23:42.646 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:42.646 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:42.646 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:42.646 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:42.646 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:42.647 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42575sec preferred_lft 42575sec 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:42.648 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:42.649 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:23:42.649 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:42.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:42.649 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:42.715 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:23:42.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:42.717 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:23:42.772 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:23:42.772 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:42.772 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:42.772 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:42.772 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:42.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:42.773 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:23:42.773 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:23:42.773 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:42.773 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:42.773 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:42.773 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:23:42.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:42.868 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:23:42.882 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:23:42.882 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:42.882 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:42.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:42.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:23:42.951 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:42.951 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:42.951 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:43.005 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:43.006 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:43.006 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:43.059 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:23:43.127 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:43.184 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:23:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:43.186 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:43.187 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:43.187 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:23:43.187 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:43.187 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:43.187 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:43.187 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:43.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:43.187 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:23:43.239 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:43.239 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:43.282 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:23:43.311 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:43.366 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:23:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:43.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:43.369 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:23:43.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:23:43.365+0000 7f9446666480 -1 init, newargv = 0x5586c0bb1770 newargc=15 2024-03-06T07:23:43.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[55644]: starting ceph client 2024-03-06T07:23:43.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[55644]: starting fuse 2024-03-06T07:23:43.385 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:23:43.385 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:43.386 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:43.387 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:23:43.506 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.55644.asok 2024-03-06T07:23:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:43.517 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55644.asok 2024-03-06T07:23:43.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:43.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55644.asok status 2024-03-06T07:23:43.653 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:23:43.653 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:23:43.653 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "55644", 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5868, 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 5868 2024-03-06T07:23:43.654 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1343174997 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1343174997 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.5868 192.168.0.1:0/1343174997", 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/1343174997", 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:23:43.655 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 19, 2024-03-06T07:23:43.656 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 45, 2024-03-06T07:23:43.656 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:43.656 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:23:43.656 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:23:43.656 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:23:43.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:43.662 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:23:43.662 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:43.677 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:23:43.677 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:43.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:43.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:43.745 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:23:43.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.889+0000 7f3e6411e640 1 Processor -- start 2024-03-06T07:23:43.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.890+0000 7f3e6411e640 1 -- start start 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.890+0000 7f3e6411e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3e5c0d10c0 0x7f3e5c0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.890+0000 7f3e6411e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5c0cfa50 con 0x7f3e5c0d14a0 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.890+0000 7f3e6411e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5c0d0a30 con 0x7f3e5c15ecd0 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.890+0000 7f3e6411e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5c15c050 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.891+0000 7f3e61eba640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3e5c0d10c0 0x7f3e5c0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.891+0000 7f3e616b9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3e5c0d14a0 0x7f3e5c15e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58620/0 (socket says 172.21.15.77:58620) 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.891+0000 7f3e616b9640 1 -- 172.21.15.77:0/2440272756 learned_addr learned my addr 172.21.15.77:0/2440272756 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.891+0000 7f3e61eba640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3e5c0d10c0 0x7f3e5c0d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49276/0 (socket says 172.21.15.77:49276) 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.891+0000 7f3e61eba640 1 -- 172.21.15.77:0/2440272756 >> v1:172.21.15.158:6789/0 conn(0x7f3e5c15ecd0 legacy=0x7f3e5c0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.891+0000 7f3e61eba640 1 -- 172.21.15.77:0/2440272756 >> v1:172.21.15.77:6789/0 conn(0x7f3e5c0d14a0 legacy=0x7f3e5c15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.891+0000 7f3e61eba640 1 -- 172.21.15.77:0/2440272756 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0b45c0 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.891+0000 7f3e60eb8640 1 -- 172.21.15.77:0/2440272756 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190976341 0 0) 0x7f3e5c0cfa50 con 0x7f3e5c0d14a0 2024-03-06T07:23:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.892+0000 7f3e61eba640 1 --2- 172.21.15.77:0/2440272756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0d04f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3e580097a0 tx=0x7f3e5802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd96cc42f820a76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.892+0000 7f3e60eb8640 1 -- 172.21.15.77:0/2440272756 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58034070 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.892+0000 7f3e60eb8640 1 -- 172.21.15.77:0/2440272756 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e58030610 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.892+0000 7f3e60eb8640 1 -- 172.21.15.77:0/2440272756 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e580309e0 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.893+0000 7f3e6411e640 1 -- 172.21.15.77:0/2440272756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 msgr2=0x7f3e5c0d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.893+0000 7f3e6411e640 1 --2- 172.21.15.77:0/2440272756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0d04f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3e580097a0 tx=0x7f3e5802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.893+0000 7f3e6411e640 1 -- 172.21.15.77:0/2440272756 shutdown_connections 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.893+0000 7f3e6411e640 1 --2- 172.21.15.77:0/2440272756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0d04f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.893+0000 7f3e6411e640 1 -- 172.21.15.77:0/2440272756 >> 172.21.15.77:0/2440272756 conn(0x7f3e5c0b3800 msgr2=0x7f3e5c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.893+0000 7f3e6411e640 1 -- 172.21.15.77:0/2440272756 shutdown_connections 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.893+0000 7f3e6411e640 1 -- 172.21.15.77:0/2440272756 wait complete. 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.894+0000 7f3e6411e640 1 Processor -- start 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.894+0000 7f3e6411e640 1 -- start start 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.894+0000 7f3e6411e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.894+0000 7f3e6411e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e5c0d14a0 0x7f3e5c1f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.894+0000 7f3e6411e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5c0ccb90 0x7f3e5c0b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.894+0000 7f3e6411e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c15f440 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e6411e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0d0060 con 0x7f3e5c0ccb90 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e6411e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0cfcf0 con 0x7f3e5c0d14a0 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49282/0 (socket says 172.21.15.77:49282) 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 -- 172.21.15.77:0/2513838169 learned_addr learned my addr 172.21.15.77:0/2513838169 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e616b9640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e5c0d14a0 0x7f3e5c1f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e626bb640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5c0ccb90 0x7f3e5c0b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 -- 172.21.15.77:0/2513838169 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e5c0d14a0 msgr2=0x7f3e5c1f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e5c0d14a0 0x7f3e5c1f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 -- 172.21.15.77:0/2513838169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5c0ccb90 msgr2=0x7f3e5c0b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5c0ccb90 0x7f3e5c0b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 -- 172.21.15.77:0/2513838169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0cd090 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e61eba640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0ce9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3e580097a0 tx=0x7f3e580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e527fc640 1 -- 172.21.15.77:0/2513838169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58047070 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e527fc640 1 -- 172.21.15.77:0/2513838169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e58004030 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5c0cd370 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e527fc640 1 -- 172.21.15.77:0/2513838169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58034040 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.895+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5c0cd930 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.896+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5c09fe60 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.897+0000 7f3e527fc640 1 -- 172.21.15.77:0/2513838169 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e5804a070 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.897+0000 7f3e527fc640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e4006b4d0 0x7f3e4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.897+0000 7f3e616b9640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e4006b4d0 0x7f3e4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.897+0000 7f3e527fc640 1 -- 172.21.15.77:0/2513838169 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e580abc60 con 0x7f3e5c0d10c0 2024-03-06T07:23:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.898+0000 7f3e616b9640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e4006b4d0 0x7f3e4006d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c0705c0 tx=0x7f3e4c007d00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:43.901+0000 7f3e527fc640 1 -- 172.21.15.77:0/2513838169 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e5807ab70 con 0x7f3e5c0d10c0 2024-03-06T07:23:44.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.078+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f3e5c070a50 con 0x7f3e5c0d10c0 2024-03-06T07:23:44.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.081+0000 7f3e527fc640 1 -- 172.21.15.77:0/2513838169 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3e5807a470 con 0x7f3e5c0d10c0 2024-03-06T07:23:44.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e4006b4d0 msgr2=0x7f3e4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e4006b4d0 0x7f3e4006d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c0705c0 tx=0x7f3e4c007d00 comp rx=0 tx=0).stop 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 msgr2=0x7f3e5c0ce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0ce9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3e580097a0 tx=0x7f3e580046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 shutdown_connections 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3e5c0ccb90 0x7f3e5c0b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3e5c0d14a0 0x7f3e5c1f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e4006b4d0 0x7f3e4006d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 --2- 172.21.15.77:0/2513838169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c0d10c0 0x7f3e5c0ce9c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 >> 172.21.15.77:0/2513838169 conn(0x7f3e5c0b3800 msgr2=0x7f3e5c0b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 shutdown_connections 2024-03-06T07:23:44.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:44.083+0000 7f3e6411e640 1 -- 172.21.15.77:0/2513838169 wait complete. 2024-03-06T07:23:54.097 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:23:54.097 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:54.113 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:23:54.113 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:23:54.113 DEBUG:teuthology.orchestra.run.smithi077:> import uuid 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> n = 51 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = False 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> if 0 > 0: 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = True 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> dpath = os.path.dirname(path) 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> fnameprefix = os.path.basename(path) 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi077:> try: 2024-03-06T07:23:54.115 DEBUG:teuthology.orchestra.run.smithi077:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:23:54.115 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:23:54.115 DEBUG:teuthology.orchestra.run.smithi077:> for i in range(n): 2024-03-06T07:23:54.115 DEBUG:teuthology.orchestra.run.smithi077:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:23:54.115 DEBUG:teuthology.orchestra.run.smithi077:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:23:54.115 DEBUG:teuthology.orchestra.run.smithi077:> f.write(f"{i}") 2024-03-06T07:23:54.115 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:23:54.115 DEBUG:teuthology.orchestra.run.smithi077:> f.flush() 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(f.fileno()) 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> os.unlink(fpath) 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> if create_hard_links: 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> for j in range(0): 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> if True: 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> finally: 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> os.close(dirfd) 2024-03-06T07:23:54.116 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:23:54.341 INFO:teuthology.orchestra.run.smithi077.stderr:Traceback (most recent call last): 2024-03-06T07:23:54.341 INFO:teuthology.orchestra.run.smithi077.stderr: File "", line 20, in 2024-03-06T07:23:54.341 INFO:teuthology.orchestra.run.smithi077.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_50' 2024-03-06T07:23:54.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:23:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:54.353 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:54.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:54.432 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:54.478 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:54.478 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:23:54.478 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:54.494 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:54.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.635+0000 7fce05ce1640 1 Processor -- start 2024-03-06T07:23:54.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.635+0000 7fce05ce1640 1 -- start start 2024-03-06T07:23:54.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.636+0000 7fce05ce1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce0015dfe0 0x7fce000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fce05ce1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce000d29c0 0x7fce000b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fce05ce1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce000b3430 con 0x7fce000cf830 2024-03-06T07:23:54.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fce05ce1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce000b31e0 con 0x7fce000d29c0 2024-03-06T07:23:54.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fce05ce1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b3d60 con 0x7fce0015dfe0 2024-03-06T07:23:54.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdff7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce0015dfe0 0x7fce000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdfffff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce000d29c0 0x7fce000b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdff7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce0015dfe0 0x7fce000cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:47158/0 (socket says 172.21.15.77:47158) 2024-03-06T07:23:54.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdff7fe640 1 -- 172.21.15.77:0/1516718184 learned_addr learned my addr 172.21.15.77:0/1516718184 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:54.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdfffff640 1 -- 172.21.15.77:0/1516718184 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce0015dfe0 msgr2=0x7fce000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdfffff640 1 --2- 172.21.15.77:0/1516718184 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce0015dfe0 0x7fce000cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdfffff640 1 -- 172.21.15.77:0/1516718184 >> v1:172.21.15.158:6789/0 conn(0x7fce000cf830 legacy=0x7fce000d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdfffff640 1 -- 172.21.15.77:0/1516718184 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce000b3710 con 0x7fce000d29c0 2024-03-06T07:23:54.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.637+0000 7fcdfe7fc640 1 -- 172.21.15.77:0/1516718184 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877487556 0 0) 0x7fce000b3430 con 0x7fce000cf830 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.638+0000 7fcdfffff640 1 --2- 172.21.15.77:0/1516718184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce000d29c0 0x7fce000b41f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009d90 tx=0x7fcdf802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a528c5fb9d7f40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.638+0000 7fcdfe7fc640 1 -- 172.21.15.77:0/1516718184 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8039070 con 0x7fce000d29c0 2024-03-06T07:23:54.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.638+0000 7fcdfe7fc640 1 -- 172.21.15.77:0/1516718184 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf802cc50 con 0x7fce000d29c0 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.638+0000 7fcdfe7fc640 1 -- 172.21.15.77:0/1516718184 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8034440 con 0x7fce000d29c0 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.639+0000 7fce05ce1640 1 -- 172.21.15.77:0/1516718184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce000d29c0 msgr2=0x7fce000b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.639+0000 7fce05ce1640 1 --2- 172.21.15.77:0/1516718184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce000d29c0 0x7fce000b41f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009d90 tx=0x7fcdf802bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.639+0000 7fce05ce1640 1 -- 172.21.15.77:0/1516718184 shutdown_connections 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.639+0000 7fce05ce1640 1 --2- 172.21.15.77:0/1516718184 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce000d29c0 0x7fce000b41f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.639+0000 7fce05ce1640 1 --2- 172.21.15.77:0/1516718184 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce0015dfe0 0x7fce000cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.639+0000 7fce05ce1640 1 -- 172.21.15.77:0/1516718184 >> 172.21.15.77:0/1516718184 conn(0x7fce000d3cd0 msgr2=0x7fce000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.639+0000 7fce05ce1640 1 -- 172.21.15.77:0/1516718184 shutdown_connections 2024-03-06T07:23:54.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.639+0000 7fce05ce1640 1 -- 172.21.15.77:0/1516718184 wait complete. 2024-03-06T07:23:54.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.641+0000 7fce05ce1640 1 Processor -- start 2024-03-06T07:23:54.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.641+0000 7fce05ce1640 1 -- start start 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fce05ce1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce000cf830 0x7fce00154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fce05ce1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce000d29c0 0x7fce0006caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fce05ce1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce0015dfe0 0x7fce0006d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fce05ce1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce000d1df0 con 0x7fce000d29c0 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fce05ce1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce000d1c70 con 0x7fce0015dfe0 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fce05ce1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0015e3c0 con 0x7fce000cf830 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fcdfeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce000d29c0 0x7fce0006caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fcdfeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce000d29c0 0x7fce0006caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41604/0 (socket says 172.21.15.77:41604) 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.642+0000 7fcdfeffd640 1 -- 172.21.15.77:0/3364578087 learned_addr learned my addr 172.21.15.77:0/3364578087 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcdfffff640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce0015dfe0 0x7fce0006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcdff7fe640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce000cf830 0x7fce00154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcdfeffd640 1 -- 172.21.15.77:0/3364578087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce000cf830 msgr2=0x7fce00154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcdfeffd640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce000cf830 0x7fce00154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcdfeffd640 1 -- 172.21.15.77:0/3364578087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce0015dfe0 msgr2=0x7fce0006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcdfeffd640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce0015dfe0 0x7fce0006d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcdfeffd640 1 -- 172.21.15.77:0/3364578087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce001f7130 con 0x7fce000d29c0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcdfeffd640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce000d29c0 0x7fce0006caf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400c550 tx=0x7fcdf400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcde7fff640 1 -- 172.21.15.77:0/3364578087 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4017070 con 0x7fce000d29c0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce001f7410 con 0x7fce000d29c0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcde7fff640 1 -- 172.21.15.77:0/3364578087 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf40035f0 con 0x7fce000d29c0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fcde7fff640 1 -- 172.21.15.77:0/3364578087 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40039e0 con 0x7fce000d29c0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.643+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce001f79d0 con 0x7fce000d29c0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.645+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0004f330 con 0x7fce000d29c0 2024-03-06T07:23:54.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.645+0000 7fcde7fff640 1 -- 172.21.15.77:0/3364578087 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdf401a070 con 0x7fce000d29c0 2024-03-06T07:23:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.645+0000 7fcde7fff640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdec06b4d0 0x7fcdec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.645+0000 7fcde7fff640 1 -- 172.21.15.77:0/3364578087 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcdf4089600 con 0x7fce000d29c0 2024-03-06T07:23:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.645+0000 7fcdff7fe640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdec06b4d0 0x7fcdec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.645+0000 7fcdff7fe640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdec06b4d0 0x7fcdec06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009b30 tx=0x7fcdf0009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.650+0000 7fcde7fff640 1 -- 172.21.15.77:0/3364578087 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdf4057e00 con 0x7fce000d29c0 2024-03-06T07:23:54.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.823+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce000b2df0 con 0x7fce000d29c0 2024-03-06T07:23:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.824+0000 7fcde7fff640 1 -- 172.21.15.77:0/3364578087 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16753 (secure 0 0 0) 0x7fcdf405bea0 con 0x7fce000d29c0 2024-03-06T07:23:54.827 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:54.827 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:48.090066+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:30.703498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:31.400205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3b","mappings":[{"from":0,"to":4}]},{"pgid":"7.3c","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.826+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdec06b4d0 msgr2=0x7fcdec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.826+0000 7fce05ce1640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdec06b4d0 0x7fcdec06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009b30 tx=0x7fcdf0009240 comp rx=0 tx=0).stop 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.826+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce000d29c0 msgr2=0x7fce0006caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.826+0000 7fce05ce1640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce000d29c0 0x7fce0006caf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400c550 tx=0x7fcdf400ca10 comp rx=0 tx=0).stop 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.827+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 shutdown_connections 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.827+0000 7fce05ce1640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdec06b4d0 0x7fcdec06d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.827+0000 7fce05ce1640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce000d29c0 0x7fce0006caf0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.827+0000 7fce05ce1640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce0015dfe0 0x7fce0006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.827+0000 7fce05ce1640 1 --2- 172.21.15.77:0/3364578087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce000cf830 0x7fce00154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.827+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 >> 172.21.15.77:0/3364578087 conn(0x7fce000d3cd0 msgr2=0x7fce000d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.827+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 shutdown_connections 2024-03-06T07:23:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.827+0000 7fce05ce1640 1 -- 172.21.15.77:0/3364578087 wait complete. 2024-03-06T07:23:54.840 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:30.703498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:31.400205+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:54.841 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.947+0000 7f7335de9640 1 Processor -- start 2024-03-06T07:23:54.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.947+0000 7f7335de9640 1 -- start start 2024-03-06T07:23:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f7335de9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f73300d14a0 0x7f733015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f7335de9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 0x7f73300cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f7335de9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73300cfa50 con 0x7f73300d10c0 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f7335de9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733015c050 con 0x7f73300d14a0 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f7335de9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300d0060 con 0x7f733015eb00 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f732effd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f73300d14a0 0x7f733015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f732ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 0x7f73300cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f732effd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f73300d14a0 0x7f733015e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:58210/0 (socket says 172.21.15.77:58210) 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f732f7fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f73300d10c0 0x7f73300d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:43528/0 (socket says 172.21.15.77:43528) 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.948+0000 7f732effd640 1 -- 172.21.15.77:0/3553254429 learned_addr learned my addr 172.21.15.77:0/3553254429 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.949+0000 7f732effd640 1 -- 172.21.15.77:0/3553254429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 msgr2=0x7f73300cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.949+0000 7f732effd640 1 --2- 172.21.15.77:0/3553254429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 0x7f73300cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.949+0000 7f732effd640 1 -- 172.21.15.77:0/3553254429 >> v1:172.21.15.158:6789/0 conn(0x7f73300d10c0 legacy=0x7f73300d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.949+0000 7f732effd640 1 -- 172.21.15.77:0/3553254429 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73300b45c0 con 0x7f73300d14a0 2024-03-06T07:23:54.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.949+0000 7f732e7fc640 1 -- 172.21.15.77:0/3553254429 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416439574 0 0) 0x7f73300cfa50 con 0x7f73300d10c0 2024-03-06T07:23:54.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f732effd640 1 --2- 172.21.15.77:0/3553254429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 0x7f733015e5c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7318009870 tx=0x7f731802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ae37fdb7032d34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f732e7fc640 1 -- 172.21.15.77:0/3553254429 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318034070 con 0x7f73300d14a0 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f732e7fc640 1 -- 172.21.15.77:0/3553254429 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7318030840 con 0x7f73300d14a0 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f732e7fc640 1 -- 172.21.15.77:0/3553254429 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318030c10 con 0x7f73300d14a0 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f7335de9640 1 -- 172.21.15.77:0/3553254429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 msgr2=0x7f733015e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f7335de9640 1 --2- 172.21.15.77:0/3553254429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 0x7f733015e5c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7318009870 tx=0x7f731802b980 comp rx=0 tx=0).stop 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f7335de9640 1 -- 172.21.15.77:0/3553254429 shutdown_connections 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f7335de9640 1 --2- 172.21.15.77:0/3553254429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 0x7f73300cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f7335de9640 1 --2- 172.21.15.77:0/3553254429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 0x7f733015e5c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f7335de9640 1 -- 172.21.15.77:0/3553254429 >> 172.21.15.77:0/3553254429 conn(0x7f73300b3800 msgr2=0x7f73300b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f7335de9640 1 -- 172.21.15.77:0/3553254429 shutdown_connections 2024-03-06T07:23:54.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.950+0000 7f7335de9640 1 -- 172.21.15.77:0/3553254429 wait complete. 2024-03-06T07:23:54.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.951+0000 7f7335de9640 1 Processor -- start 2024-03-06T07:23:54.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f7335de9640 1 -- start start 2024-03-06T07:23:54.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f7335de9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73300d10c0 0x7f733014dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f7335de9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 0x7f73300cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f7335de9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 0x7f73300afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f7335de9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73300cfcf0 con 0x7f73300d10c0 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f7335de9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733015f440 con 0x7f73300d14a0 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f7335de9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300cb3c0 con 0x7f733015eb00 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f732f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73300d10c0 0x7f733014dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f732f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73300d10c0 0x7f733014dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41612/0 (socket says 172.21.15.77:41612) 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.952+0000 7f732f7fe640 1 -- 172.21.15.77:0/3003594875 learned_addr learned my addr 172.21.15.77:0/3003594875 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732effd640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 0x7f73300cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732ffff640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 0x7f73300afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732f7fe640 1 -- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 msgr2=0x7f73300afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732f7fe640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 0x7f73300afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732f7fe640 1 -- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 msgr2=0x7f73300cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732f7fe640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 0x7f73300cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732f7fe640 1 -- 172.21.15.77:0/3003594875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733014e020 con 0x7f73300d10c0 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732effd640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 0x7f73300cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f732f7fe640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73300d10c0 0x7f733014dae0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f732000c890 tx=0x7f732000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f7327fff640 1 -- 172.21.15.77:0/3003594875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320018070 con 0x7f73300d10c0 2024-03-06T07:23:54.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f7327fff640 1 -- 172.21.15.77:0/3003594875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732000a5e0 con 0x7f73300d10c0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f7327fff640 1 -- 172.21.15.77:0/3003594875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732000a9d0 con 0x7f73300d10c0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73300b5550 con 0x7f73300d10c0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.953+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73300b5b10 con 0x7f73300d10c0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.954+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733009fe60 con 0x7f73300d10c0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.955+0000 7f7327fff640 1 -- 172.21.15.77:0/3003594875 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f732001b070 con 0x7f73300d10c0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.955+0000 7f7327fff640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f731006b500 0x7f731006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.955+0000 7f7327fff640 1 -- 172.21.15.77:0/3003594875 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7320089850 con 0x7f73300d10c0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.955+0000 7f732effd640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f731006b500 0x7f731006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.955+0000 7f732effd640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f731006b500 0x7f731006d9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7318036000 tx=0x7f7318008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:54.959+0000 7f7327fff640 1 -- 172.21.15.77:0/3003594875 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7320058690 con 0x7f73300d10c0 2024-03-06T07:23:55.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.137+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7330068ff0 con 0x7f73300d10c0 2024-03-06T07:23:55.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.137+0000 7f7327fff640 1 -- 172.21.15.77:0/3003594875 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7320057f90 con 0x7f73300d10c0 2024-03-06T07:23:55.139 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:23:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f731006b500 msgr2=0x7f731006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f731006b500 0x7f731006d9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7318036000 tx=0x7f7318008030 comp rx=0 tx=0).stop 2024-03-06T07:23:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73300d10c0 msgr2=0x7f733014dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73300d10c0 0x7f733014dae0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f732000c890 tx=0x7f732000cd50 comp rx=0 tx=0).stop 2024-03-06T07:23:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 shutdown_connections 2024-03-06T07:23:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f733015eb00 0x7f73300afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73300d14a0 0x7f73300cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f731006b500 0x7f731006d9c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 --2- 172.21.15.77:0/3003594875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73300d10c0 0x7f733014dae0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 >> 172.21.15.77:0/3003594875 conn(0x7f73300b3800 msgr2=0x7f73300b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 shutdown_connections 2024-03-06T07:23:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.139+0000 7f7335de9640 1 -- 172.21.15.77:0/3003594875 wait complete. 2024-03-06T07:23:55.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:55.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.254+0000 7f542a779640 1 Processor -- start 2024-03-06T07:23:55.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.254+0000 7f542a779640 1 -- start start 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.255+0000 7f542a779640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f542415bb40 0x7f54240b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.255+0000 7f542a779640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f54240cfa90 0x7f54240cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.255+0000 7f542a779640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542415ff90 con 0x7f542415f7d0 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.255+0000 7f542a779640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240d0440 con 0x7f542415bb40 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.255+0000 7f542a779640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542406c910 con 0x7f54240cfa90 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.255+0000 7f5423fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f542415bb40 0x7f54240b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.255+0000 7f5423fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f542415bb40 0x7f54240b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41624/0 (socket says 172.21.15.77:41624) 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.255+0000 7f5423fff640 1 -- 172.21.15.77:0/2069252494 learned_addr learned my addr 172.21.15.77:0/2069252494 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.256+0000 7f5428d16640 1 --2- 172.21.15.77:0/2069252494 >> v2:172.21.15.158:3300/0 conn(0x7f54240cfa90 0x7f54240cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.256+0000 7f5423fff640 1 -- 172.21.15.77:0/2069252494 >> v2:172.21.15.158:3300/0 conn(0x7f54240cfa90 msgr2=0x7f54240cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.256+0000 7f5423fff640 1 --2- 172.21.15.77:0/2069252494 >> v2:172.21.15.158:3300/0 conn(0x7f54240cfa90 0x7f54240cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.256+0000 7f5423fff640 1 -- 172.21.15.77:0/2069252494 >> v1:172.21.15.158:6789/0 conn(0x7f542415f7d0 legacy=0x7f542406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.256+0000 7f5423fff640 1 -- 172.21.15.77:0/2069252494 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5424160270 con 0x7f542415bb40 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.257+0000 7f5423fff640 1 --2- 172.21.15.77:0/2069252494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415bb40 0x7f54240b4970 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5424072c60 tx=0x7f541802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4203a0fa706dc093 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.257+0000 7f5422ffd640 1 -- 172.21.15.77:0/2069252494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418035020 con 0x7f542415bb40 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.257+0000 7f5422ffd640 1 -- 172.21.15.77:0/2069252494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5418031870 con 0x7f542415bb40 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.257+0000 7f5422ffd640 1 -- 172.21.15.77:0/2069252494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418031c40 con 0x7f542415bb40 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.257+0000 7f542a779640 1 -- 172.21.15.77:0/2069252494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415bb40 msgr2=0x7f54240b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.257+0000 7f542a779640 1 --2- 172.21.15.77:0/2069252494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415bb40 0x7f54240b4970 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5424072c60 tx=0x7f541802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.257+0000 7f542a779640 1 -- 172.21.15.77:0/2069252494 shutdown_connections 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.258+0000 7f542a779640 1 --2- 172.21.15.77:0/2069252494 >> v2:172.21.15.158:3300/0 conn(0x7f54240cfa90 0x7f54240cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.258+0000 7f542a779640 1 --2- 172.21.15.77:0/2069252494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415bb40 0x7f54240b4970 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.258+0000 7f542a779640 1 -- 172.21.15.77:0/2069252494 >> 172.21.15.77:0/2069252494 conn(0x7f542406b190 msgr2=0x7f542406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.258+0000 7f542a779640 1 -- 172.21.15.77:0/2069252494 shutdown_connections 2024-03-06T07:23:55.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.258+0000 7f542a779640 1 -- 172.21.15.77:0/2069252494 wait complete. 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.259+0000 7f542a779640 1 Processor -- start 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.259+0000 7f542a779640 1 -- start start 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f542a779640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54240cfa90 0x7f5424141650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f542a779640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f542415bb40 0x7f5424141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f542a779640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415f7d0 0x7f5424158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f542a779640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542415fc30 con 0x7f542415f7d0 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f542a779640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542406cb60 con 0x7f542415bb40 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f542a779640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542406d790 con 0x7f54240cfa90 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415f7d0 0x7f5424158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415f7d0 0x7f5424158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41626/0 (socket says 172.21.15.77:41626) 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 -- 172.21.15.77:0/1928685410 learned_addr learned my addr 172.21.15.77:0/1928685410 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5423fff640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54240cfa90 0x7f5424141650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f54237fe640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f542415bb40 0x7f5424141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 -- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54240cfa90 msgr2=0x7f5424141650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54240cfa90 0x7f5424141650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 -- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f542415bb40 msgr2=0x7f5424141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f542415bb40 0x7f5424141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 -- 172.21.15.77:0/1928685410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5424144d00 con 0x7f542415f7d0 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5423fff640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54240cfa90 0x7f5424141650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5428d16640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415f7d0 0x7f5424158d90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f541c007e90 tx=0x7f541c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5420ff9640 1 -- 172.21.15.77:0/1928685410 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c014420 con 0x7f542415f7d0 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5420ff9640 1 -- 172.21.15.77:0/1928685410 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c019070 con 0x7f542415f7d0 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f5420ff9640 1 -- 172.21.15.77:0/1928685410 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c014800 con 0x7f542415f7d0 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54241440f0 con 0x7f542415f7d0 2024-03-06T07:23:55.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.260+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54241446b0 con 0x7f542415f7d0 2024-03-06T07:23:55.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.261+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54240732b0 con 0x7f542415f7d0 2024-03-06T07:23:55.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.262+0000 7f5420ff9640 1 -- 172.21.15.77:0/1928685410 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f541c0149a0 con 0x7f542415f7d0 2024-03-06T07:23:55.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.263+0000 7f5420ff9640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f541006b500 0x7f541006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.263+0000 7f5420ff9640 1 -- 172.21.15.77:0/1928685410 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f541c08aa70 con 0x7f542415f7d0 2024-03-06T07:23:55.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.263+0000 7f5423fff640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f541006b500 0x7f541006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.263+0000 7f5423fff640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f541006b500 0x7f541006d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5418036000 tx=0x7f541803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.267+0000 7f5420ff9640 1 -- 172.21.15.77:0/1928685410 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f541c0598b0 con 0x7f542415f7d0 2024-03-06T07:23:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.442+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5424068ff0 con 0x7f542415f7d0 2024-03-06T07:23:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.442+0000 7f5420ff9640 1 -- 172.21.15.77:0/1928685410 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) v1 ==== 104+0+11 (secure 0 0 0) 0x7f541c0591b0 con 0x7f542415f7d0 2024-03-06T07:23:55.445 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.444+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f541006b500 msgr2=0x7f541006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.444+0000 7f542a779640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f541006b500 0x7f541006d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5418036000 tx=0x7f541803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.444+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415f7d0 msgr2=0x7f5424158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.444+0000 7f542a779640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415f7d0 0x7f5424158d90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f541c007e90 tx=0x7f541c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.445+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 shutdown_connections 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.445+0000 7f542a779640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f541006b500 0x7f541006d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.445+0000 7f542a779640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f542415f7d0 0x7f5424158d90 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.445+0000 7f542a779640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f542415bb40 0x7f5424141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.445+0000 7f542a779640 1 --2- 172.21.15.77:0/1928685410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54240cfa90 0x7f5424141650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.445+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 >> 172.21.15.77:0/1928685410 conn(0x7f542406b190 msgr2=0x7f54240b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.445+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 shutdown_connections 2024-03-06T07:23:55.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.445+0000 7f542a779640 1 -- 172.21.15.77:0/1928685410 wait complete. 2024-03-06T07:23:55.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:55.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.562+0000 7f48b98c2640 1 Processor -- start 2024-03-06T07:23:55.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.562+0000 7f48b98c2640 1 -- start start 2024-03-06T07:23:55.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.563+0000 7f48b98c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48b40cf200 0x7f48b40cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.563+0000 7f48b98c2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b41600a0 con 0x7f48b40d06e0 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.563+0000 7f48b98c2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b40d47f0 con 0x7f48b415fc30 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.563+0000 7f48b98c2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b40ced70 con 0x7f48b40cf200 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.563+0000 7f48b37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48b40cf200 0x7f48b40cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.563+0000 7f48b2ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f48b40d06e0 0x7f48b40d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51320/0 (socket says 172.21.15.77:51320) 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.563+0000 7f48b2ffd640 1 -- 172.21.15.77:0/1303241755 learned_addr learned my addr 172.21.15.77:0/1303241755 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.564+0000 7f48b37fe640 1 -- 172.21.15.77:0/1303241755 >> v1:172.21.15.158:6789/0 conn(0x7f48b415fc30 legacy=0x7f48b40b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.564+0000 7f48b37fe640 1 -- 172.21.15.77:0/1303241755 >> v1:172.21.15.77:6789/0 conn(0x7f48b40d06e0 legacy=0x7f48b40d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.564+0000 7f48b27fc640 1 -- 172.21.15.77:0/1303241755 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47532243 0 0) 0x7f48b41600a0 con 0x7f48b40d06e0 2024-03-06T07:23:55.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.564+0000 7f48b37fe640 1 -- 172.21.15.77:0/1303241755 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b40d1ab0 con 0x7f48b40cf200 2024-03-06T07:23:55.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.564+0000 7f48b37fe640 1 --2- 172.21.15.77:0/1303241755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b40cf670 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f48ac009d90 tx=0x7f48ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87b8111a6c2a1b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b27fc640 1 -- 172.21.15.77:0/1303241755 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac039070 con 0x7f48b40cf200 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b27fc640 1 -- 172.21.15.77:0/1303241755 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac02cc60 con 0x7f48b40cf200 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b27fc640 1 -- 172.21.15.77:0/1303241755 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac034660 con 0x7f48b40cf200 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b98c2640 1 -- 172.21.15.77:0/1303241755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 msgr2=0x7f48b40cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b98c2640 1 --2- 172.21.15.77:0/1303241755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b40cf670 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f48ac009d90 tx=0x7f48ac02beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b98c2640 1 -- 172.21.15.77:0/1303241755 shutdown_connections 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b98c2640 1 --2- 172.21.15.77:0/1303241755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b40cf670 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b98c2640 1 -- 172.21.15.77:0/1303241755 >> 172.21.15.77:0/1303241755 conn(0x7f48b40d3cd0 msgr2=0x7f48b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b98c2640 1 -- 172.21.15.77:0/1303241755 shutdown_connections 2024-03-06T07:23:55.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.565+0000 7f48b98c2640 1 -- 172.21.15.77:0/1303241755 wait complete. 2024-03-06T07:23:55.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.566+0000 7f48b98c2640 1 Processor -- start 2024-03-06T07:23:55.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.566+0000 7f48b98c2640 1 -- start start 2024-03-06T07:23:55.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b98c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b4151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b98c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48b40d06e0 0x7f48b40b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b98c2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48b415fc30 0x7f48b4153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b98c2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b40ce990 con 0x7f48b40cf200 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b98c2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b40d4ab0 con 0x7f48b415fc30 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b98c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b40b44d0 con 0x7f48b40d06e0 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b4151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b4151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41638/0 (socket says 172.21.15.77:41638) 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b2ffd640 1 -- 172.21.15.77:0/1302064542 learned_addr learned my addr 172.21.15.77:0/1302064542 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48abfff640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48b40d06e0 0x7f48b40b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b37fe640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48b415fc30 0x7f48b4153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b2ffd640 1 -- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48b40d06e0 msgr2=0x7f48b40b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b2ffd640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48b40d06e0 0x7f48b40b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b2ffd640 1 -- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48b415fc30 msgr2=0x7f48b4153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b2ffd640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48b415fc30 0x7f48b4153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b2ffd640 1 -- 172.21.15.77:0/1302064542 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b40b7ba0 con 0x7f48b40cf200 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48abfff640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48b40d06e0 0x7f48b40b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.567+0000 7f48b37fe640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48b415fc30 0x7f48b4153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.568+0000 7f48b2ffd640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b4151f90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f48a400c460 tx=0x7f48a400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.568+0000 7f48b0ff9640 1 -- 172.21.15.77:0/1302064542 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a40074e0 con 0x7f48b40cf200 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.568+0000 7f48b0ff9640 1 -- 172.21.15.77:0/1302064542 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a40047e0 con 0x7f48b40cf200 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.568+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b40b7e80 con 0x7f48b40cf200 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.568+0000 7f48b0ff9640 1 -- 172.21.15.77:0/1302064542 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4004bd0 con 0x7f48b40cf200 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.568+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b40b8440 con 0x7f48b40cf200 2024-03-06T07:23:55.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.569+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b40a1c80 con 0x7f48b40cf200 2024-03-06T07:23:55.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.569+0000 7f48b0ff9640 1 -- 172.21.15.77:0/1302064542 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48a4011070 con 0x7f48b40cf200 2024-03-06T07:23:55.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.570+0000 7f48b0ff9640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f489406b4d0 0x7f489406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.570+0000 7f48abfff640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f489406b4d0 0x7f489406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.570+0000 7f48b0ff9640 1 -- 172.21.15.77:0/1302064542 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f48a4089e00 con 0x7f48b40cf200 2024-03-06T07:23:55.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.570+0000 7f48abfff640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f489406b4d0 0x7f489406d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f48b40d0ca0 tx=0x7f489c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.574+0000 7f48b0ff9640 1 -- 172.21.15.77:0/1302064542 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48a4058c40 con 0x7f48b40cf200 2024-03-06T07:23:55.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.751+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48b4068ff0 con 0x7f48b40cf200 2024-03-06T07:23:55.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.751+0000 7f48b0ff9640 1 -- 172.21.15.77:0/1302064542 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48a4058540 con 0x7f48b40cf200 2024-03-06T07:23:55.753 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:23:55.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f489406b4d0 msgr2=0x7f489406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f489406b4d0 0x7f489406d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f48b40d0ca0 tx=0x7f489c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:55.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 msgr2=0x7f48b4151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b4151f90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f48a400c460 tx=0x7f48a400c920 comp rx=0 tx=0).stop 2024-03-06T07:23:55.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 shutdown_connections 2024-03-06T07:23:55.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48b415fc30 0x7f48b4153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48b40d06e0 0x7f48b40b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f489406b4d0 0x7f489406d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 --2- 172.21.15.77:0/1302064542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48b40cf200 0x7f48b4151f90 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 >> 172.21.15.77:0/1302064542 conn(0x7f48b40d3cd0 msgr2=0x7f48b40b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 shutdown_connections 2024-03-06T07:23:55.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.753+0000 7f48b98c2640 1 -- 172.21.15.77:0/1302064542 wait complete. 2024-03-06T07:23:55.766 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.865+0000 7fdd68a47640 1 Processor -- start 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.865+0000 7fdd68a47640 1 -- start start 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd68a47640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdd64162400 0x7fdd640d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd68a47640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640d22c0 con 0x7fdd640b4460 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd68a47640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640d25a0 con 0x7fdd64162970 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd68a47640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd64161b00 con 0x7fdd64162400 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd62d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdd64162400 0x7fdd640d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd6259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdd640b4460 0x7fdd640b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51328/0 (socket says 172.21.15.77:51328) 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd6259c640 1 -- 172.21.15.77:0/999036379 learned_addr learned my addr 172.21.15.77:0/999036379 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd62d9d640 1 -- 172.21.15.77:0/999036379 >> v1:172.21.15.158:6789/0 conn(0x7fdd64162970 legacy=0x7fdd64161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd62d9d640 1 -- 172.21.15.77:0/999036379 >> v1:172.21.15.77:6789/0 conn(0x7fdd640b4460 legacy=0x7fdd640b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd62d9d640 1 -- 172.21.15.77:0/999036379 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd640d2880 con 0x7fdd64162400 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.867+0000 7fdd61d9b640 1 -- 172.21.15.77:0/999036379 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673223495 0 0) 0x7fdd640d22c0 con 0x7fdd640b4460 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.868+0000 7fdd62d9d640 1 --2- 172.21.15.77:0/999036379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd64162400 0x7fdd640d4800 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009d90 tx=0x7fdd5c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac08a1e62594b40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.868+0000 7fdd61d9b640 1 -- 172.21.15.77:0/999036379 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c039070 con 0x7fdd64162400 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.868+0000 7fdd61d9b640 1 -- 172.21.15.77:0/999036379 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c02cc60 con 0x7fdd64162400 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.868+0000 7fdd61d9b640 1 -- 172.21.15.77:0/999036379 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c034660 con 0x7fdd64162400 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.869+0000 7fdd68a47640 1 -- 172.21.15.77:0/999036379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd64162400 msgr2=0x7fdd640d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.869+0000 7fdd68a47640 1 --2- 172.21.15.77:0/999036379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd64162400 0x7fdd640d4800 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009d90 tx=0x7fdd5c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.869+0000 7fdd68a47640 1 -- 172.21.15.77:0/999036379 shutdown_connections 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.869+0000 7fdd68a47640 1 --2- 172.21.15.77:0/999036379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd64162400 0x7fdd640d4800 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.869+0000 7fdd68a47640 1 -- 172.21.15.77:0/999036379 >> 172.21.15.77:0/999036379 conn(0x7fdd640d3ce0 msgr2=0x7fdd640d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.869+0000 7fdd68a47640 1 -- 172.21.15.77:0/999036379 shutdown_connections 2024-03-06T07:23:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.869+0000 7fdd68a47640 1 -- 172.21.15.77:0/999036379 wait complete. 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.870+0000 7fdd68a47640 1 Processor -- start 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.870+0000 7fdd68a47640 1 -- start start 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd68a47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd640b4460 0x7fdd640c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd68a47640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd64162400 0x7fdd64155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd68a47640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd64162970 0x7fdd640c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd68a47640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640ce950 con 0x7fdd640b4460 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd68a47640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640cecc0 con 0x7fdd64162970 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd68a47640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640b4840 con 0x7fdd64162400 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd640b4460 0x7fdd640c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd640b4460 0x7fdd640c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41668/0 (socket says 172.21.15.77:41668) 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 -- 172.21.15.77:0/2977302459 learned_addr learned my addr 172.21.15.77:0/2977302459 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 -- 172.21.15.77:0/2977302459 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd64162400 msgr2=0x7fdd64155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd64162400 0x7fdd64155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 -- 172.21.15.77:0/2977302459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd64162970 msgr2=0x7fdd640c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd64162970 0x7fdd640c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 -- 172.21.15.77:0/2977302459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64156250 con 0x7fdd640b4460 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd6259c640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd640b4460 0x7fdd640c9420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400c4d0 tx=0x7fdd5400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd5affd640 1 -- 172.21.15.77:0/2977302459 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54018070 con 0x7fdd640b4460 2024-03-06T07:23:55.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd5affd640 1 -- 172.21.15.77:0/2977302459 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd540074e0 con 0x7fdd640b4460 2024-03-06T07:23:55.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd640c8380 con 0x7fdd640b4460 2024-03-06T07:23:55.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd5affd640 1 -- 172.21.15.77:0/2977302459 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54004a20 con 0x7fdd640b4460 2024-03-06T07:23:55.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.871+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd640c8940 con 0x7fdd640b4460 2024-03-06T07:23:55.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.873+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6404f330 con 0x7fdd640b4460 2024-03-06T07:23:55.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.873+0000 7fdd5affd640 1 -- 172.21.15.77:0/2977302459 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd5401b070 con 0x7fdd640b4460 2024-03-06T07:23:55.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.873+0000 7fdd5affd640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd4406b4d0 0x7fdd4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.873+0000 7fdd5affd640 1 -- 172.21.15.77:0/2977302459 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdd54089530 con 0x7fdd640b4460 2024-03-06T07:23:55.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.873+0000 7fdd5bfff640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd4406b4d0 0x7fdd4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.874+0000 7fdd5bfff640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd4406b4d0 0x7fdd4406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006fd0 tx=0x7fdd4c006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:55.877+0000 7fdd5affd640 1 -- 172.21.15.77:0/2977302459 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd54058400 con 0x7fdd640b4460 2024-03-06T07:23:56.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.100+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdd6406b180 con 0x7fdd640b4460 2024-03-06T07:23:56.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.101+0000 7fdd5affd640 1 -- 172.21.15.77:0/2977302459 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v47) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdd54057d00 con 0x7fdd640b4460 2024-03-06T07:23:56.103 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd4406b4d0 msgr2=0x7fdd4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd4406b4d0 0x7fdd4406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006fd0 tx=0x7fdd4c006bc0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd640b4460 msgr2=0x7fdd640c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd640b4460 0x7fdd640c9420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400c4d0 tx=0x7fdd5400c990 comp rx=0 tx=0).stop 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 shutdown_connections 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd64162970 0x7fdd640c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdd64162400 0x7fdd64155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdd4406b4d0 0x7fdd4406d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 --2- 172.21.15.77:0/2977302459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd640b4460 0x7fdd640c9420 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 >> 172.21.15.77:0/2977302459 conn(0x7fdd640d3ce0 msgr2=0x7fdd640d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 shutdown_connections 2024-03-06T07:23:56.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.103+0000 7fdd68a47640 1 -- 172.21.15.77:0/2977302459 wait complete. 2024-03-06T07:23:56.119 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:23:56.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:56.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:56.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[55644]: fuse finished with error 0 and tester_r 0 2024-03-06T07:23:56.148 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:23:56.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:56.148 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:56.330 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:23:56.330 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:56.348 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:56.348 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:23:56.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:56.349 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:56.405 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:23:56.405 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:23:56.405 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:56.420 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:56.420 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:23:56.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:56.420 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:56.475 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:23:56.476 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:23:56.476 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:56.493 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:56.494 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:23:56.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:56.494 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:56.550 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:23:56.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:56.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.649+0000 7efe0a7f7640 1 Processor -- start 2024-03-06T07:23:56.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.649+0000 7efe0a7f7640 1 -- start start 2024-03-06T07:23:56.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe0a7f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efe040d22b0 0x7efe040d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe0a7f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efe040b41f0 0x7efe040b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe0a7f7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe040b3430 con 0x7efe0415dfe0 2024-03-06T07:23:56.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe0a7f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe040d2c50 con 0x7efe040b41f0 2024-03-06T07:23:56.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe0a7f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe040b31e0 con 0x7efe040d22b0 2024-03-06T07:23:56.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe08d94640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efe040b41f0 0x7efe040b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe08d94640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efe040b41f0 0x7efe040b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41682/0 (socket says 172.21.15.77:41682) 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe08d94640 1 -- 172.21.15.77:0/204652677 learned_addr learned my addr 172.21.15.77:0/204652677 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe08d94640 1 -- 172.21.15.77:0/204652677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efe040d22b0 msgr2=0x7efe040d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe08d94640 1 --2- 172.21.15.77:0/204652677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efe040d22b0 0x7efe040d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe08d94640 1 -- 172.21.15.77:0/204652677 >> v1:172.21.15.158:6789/0 conn(0x7efe0415dfe0 legacy=0x7efe040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.650+0000 7efe08d94640 1 -- 172.21.15.77:0/204652677 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe040b3710 con 0x7efe040b41f0 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.651+0000 7efe08d94640 1 --2- 172.21.15.77:0/204652677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe040b4650 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efdfc009e50 tx=0x7efdfc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadd81d2fd79e092 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.651+0000 7efe02ffd640 1 -- 172.21.15.77:0/204652677 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc02ca70 con 0x7efe040b41f0 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.651+0000 7efe02ffd640 1 -- 172.21.15.77:0/204652677 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc02cc10 con 0x7efe040b41f0 2024-03-06T07:23:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.651+0000 7efe02ffd640 1 -- 172.21.15.77:0/204652677 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0343f0 con 0x7efe040b41f0 2024-03-06T07:23:56.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.652+0000 7efe0a7f7640 1 -- 172.21.15.77:0/204652677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 msgr2=0x7efe040b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.652+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/204652677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe040b4650 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efdfc009e50 tx=0x7efdfc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.652+0000 7efe0a7f7640 1 -- 172.21.15.77:0/204652677 shutdown_connections 2024-03-06T07:23:56.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.652+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/204652677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efe040d22b0 0x7efe040d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.652+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/204652677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe040b4650 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.652+0000 7efe0a7f7640 1 -- 172.21.15.77:0/204652677 >> 172.21.15.77:0/204652677 conn(0x7efe040d3cd0 msgr2=0x7efe040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.652+0000 7efe0a7f7640 1 -- 172.21.15.77:0/204652677 shutdown_connections 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.652+0000 7efe0a7f7640 1 -- 172.21.15.77:0/204652677 wait complete. 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.653+0000 7efe0a7f7640 1 Processor -- start 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.653+0000 7efe0a7f7640 1 -- start start 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe0a7f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe04160ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe0a7f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe040d22b0 0x7efe040b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe0a7f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efe0415dfe0 0x7efe041f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe0a7f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe040cfa00 con 0x7efe040b41f0 2024-03-06T07:23:56.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe0a7f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe040b3d60 con 0x7efe040d22b0 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe0a7f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe040d1c70 con 0x7efe0415dfe0 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe04160ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe04160ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41698/0 (socket says 172.21.15.77:41698) 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 -- 172.21.15.77:0/1201280555 learned_addr learned my addr 172.21.15.77:0/1201280555 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 -- 172.21.15.77:0/1201280555 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efe0415dfe0 msgr2=0x7efe041f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efe0415dfe0 0x7efe041f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 -- 172.21.15.77:0/1201280555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe040d22b0 msgr2=0x7efe040b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe040d22b0 0x7efe040b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 -- 172.21.15.77:0/1201280555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe040b6de0 con 0x7efe040b41f0 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe03fff640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe04160ea0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efdf800b3e0 tx=0x7efdf800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.654+0000 7efe00ff9640 1 -- 172.21.15.77:0/1201280555 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8015020 con 0x7efe040b41f0 2024-03-06T07:23:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.655+0000 7efe00ff9640 1 -- 172.21.15.77:0/1201280555 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf8011850 con 0x7efe040b41f0 2024-03-06T07:23:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.655+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe040b7080 con 0x7efe040b41f0 2024-03-06T07:23:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.655+0000 7efe00ff9640 1 -- 172.21.15.77:0/1201280555 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8011c40 con 0x7efe040b41f0 2024-03-06T07:23:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.655+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe040b7640 con 0x7efe040b41f0 2024-03-06T07:23:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.655+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe040d3740 con 0x7efe040b41f0 2024-03-06T07:23:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.656+0000 7efe00ff9640 1 -- 172.21.15.77:0/1201280555 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdf8018020 con 0x7efe040b41f0 2024-03-06T07:23:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.656+0000 7efe00ff9640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efdf006b500 0x7efdf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.657+0000 7efe00ff9640 1 -- 172.21.15.77:0/1201280555 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efdf808a190 con 0x7efe040b41f0 2024-03-06T07:23:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.657+0000 7efe037fe640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efdf006b500 0x7efdf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.657+0000 7efe037fe640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efdf006b500 0x7efdf006d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efe040a1ce0 tx=0x7efdf4006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.661+0000 7efe00ff9640 1 -- 172.21.15.77:0/1201280555 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe040d3740 con 0x7efe040b41f0 2024-03-06T07:23:56.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.872+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe0404f330 con 0x7efe040b41f0 2024-03-06T07:23:56.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.872+0000 7efe00ff9640 1 -- 172.21.15.77:0/1201280555 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4977 (secure 0 0 0) 0x7efdf801f020 con 0x7efe040b41f0 2024-03-06T07:23:56.874 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:56.875 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1388591691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1388591691},{"type":"v1","addr":"172.21.15.158:6834","nonce":1388591691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/870048529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":870048529},{"type":"v1","addr":"172.21.15.158:6835","nonce":870048529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/650607050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":650607050},{"type":"v1","addr":"172.21.15.77:6836","nonce":650607050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:32.429377+0000","modified":"2024-03-06T07:23:34.434503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2240574726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2240574726},{"type":"v1","addr":"172.21.15.77:6837","nonce":2240574726}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":3}]} 2024-03-06T07:23:56.875 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 19 2024-03-06T07:23:56.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efdf006b500 msgr2=0x7efdf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efdf006b500 0x7efdf006d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efe040a1ce0 tx=0x7efdf4006c90 comp rx=0 tx=0).stop 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 msgr2=0x7efe04160ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe04160ea0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efdf800b3e0 tx=0x7efdf800b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 shutdown_connections 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efe0415dfe0 0x7efe041f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efe040d22b0 0x7efe040b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efdf006b500 0x7efdf006d9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 --2- 172.21.15.77:0/1201280555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe040b41f0 0x7efe04160ea0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.874+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 >> 172.21.15.77:0/1201280555 conn(0x7efe040d3cd0 msgr2=0x7efe040cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.875+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 shutdown_connections 2024-03-06T07:23:56.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.875+0000 7efe0a7f7640 1 -- 172.21.15.77:0/1201280555 wait complete. 2024-03-06T07:23:56.891 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:56.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.991+0000 7fcfd470c640 1 Processor -- start 2024-03-06T07:23:56.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.991+0000 7fcfd470c640 1 -- start start 2024-03-06T07:23:56.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd470c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc1515e0 0x7fcfcc0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd470c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcfcc15b430 0x7fcfcc071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd470c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfcc0cd7a0 0x7fcfcc0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd470c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfcc15b8b0 con 0x7fcfcc0cd7a0 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd470c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfcc070a70 con 0x7fcfcc15b430 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd470c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc070d50 con 0x7fcfcc1515e0 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd2ca9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfcc0cd7a0 0x7fcfcc0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd2ca9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfcc0cd7a0 0x7fcfcc0cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41702/0 (socket says 172.21.15.77:41702) 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd2ca9640 1 -- 172.21.15.77:0/3168568674 learned_addr learned my addr 172.21.15.77:0/3168568674 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd24a8640 1 --2- 172.21.15.77:0/3168568674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc1515e0 0x7fcfcc0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd1ca7640 1 --2- 172.21.15.77:0/3168568674 >> v2:172.21.15.158:3300/0 conn(0x7fcfcc15b430 0x7fcfcc071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd2ca9640 1 -- 172.21.15.77:0/3168568674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc1515e0 msgr2=0x7fcfcc0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd2ca9640 1 --2- 172.21.15.77:0/3168568674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc1515e0 0x7fcfcc0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd2ca9640 1 -- 172.21.15.77:0/3168568674 >> v2:172.21.15.158:3300/0 conn(0x7fcfcc15b430 msgr2=0x7fcfcc071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd2ca9640 1 --2- 172.21.15.77:0/3168568674 >> v2:172.21.15.158:3300/0 conn(0x7fcfcc15b430 0x7fcfcc071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.992+0000 7fcfd2ca9640 1 -- 172.21.15.77:0/3168568674 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc071030 con 0x7fcfcc0cd7a0 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.993+0000 7fcfd2ca9640 1 --2- 172.21.15.77:0/3168568674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc0cd7a0 0x7fcfcc0cdbd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc073220 tx=0x7fcfc402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ce31683ae63ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.993+0000 7fcfd14a6640 1 -- 172.21.15.77:0/3168568674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc402ca70 con 0x7fcfcc0cd7a0 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.993+0000 7fcfd14a6640 1 -- 172.21.15.77:0/3168568674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc402cc10 con 0x7fcfcc0cd7a0 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.993+0000 7fcfd14a6640 1 -- 172.21.15.77:0/3168568674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40343f0 con 0x7fcfcc0cd7a0 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 -- 172.21.15.77:0/3168568674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc0cd7a0 msgr2=0x7fcfcc0cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 --2- 172.21.15.77:0/3168568674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc0cd7a0 0x7fcfcc0cdbd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc073220 tx=0x7fcfc402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 -- 172.21.15.77:0/3168568674 shutdown_connections 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 --2- 172.21.15.77:0/3168568674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc0cd7a0 0x7fcfcc0cdbd0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 --2- 172.21.15.77:0/3168568674 >> v2:172.21.15.158:3300/0 conn(0x7fcfcc15b430 0x7fcfcc071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 --2- 172.21.15.77:0/3168568674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc1515e0 0x7fcfcc0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 -- 172.21.15.77:0/3168568674 >> 172.21.15.77:0/3168568674 conn(0x7fcfcc0cf8a0 msgr2=0x7fcfcc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 -- 172.21.15.77:0/3168568674 shutdown_connections 2024-03-06T07:23:56.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.994+0000 7fcfd470c640 1 -- 172.21.15.77:0/3168568674 wait complete. 2024-03-06T07:23:56.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.996+0000 7fcfd470c640 1 Processor -- start 2024-03-06T07:23:56.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.996+0000 7fcfd470c640 1 -- start start 2024-03-06T07:23:57.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd470c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfcc0cd7a0 0x7fcfcc14f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd470c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc1515e0 0x7fcfcc1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd470c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc15b430 0x7fcfcc1f7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd470c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc071d70 con 0x7fcfcc1515e0 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd470c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc0ce110 con 0x7fcfcc0cd7a0 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd470c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc071ef0 con 0x7fcfcc15b430 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd1ca7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc1515e0 0x7fcfcc1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd1ca7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc1515e0 0x7fcfcc1599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41706/0 (socket says 172.21.15.77:41706) 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd1ca7640 1 -- 172.21.15.77:0/1016714151 learned_addr learned my addr 172.21.15.77:0/1016714151 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd2ca9640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc15b430 0x7fcfcc1f7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.997+0000 7fcfd24a8640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfcc0cd7a0 0x7fcfcc14f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfd1ca7640 1 -- 172.21.15.77:0/1016714151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc15b430 msgr2=0x7fcfcc1f7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfd1ca7640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc15b430 0x7fcfcc1f7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfd1ca7640 1 -- 172.21.15.77:0/1016714151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfcc0cd7a0 msgr2=0x7fcfcc14f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfd1ca7640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfcc0cd7a0 0x7fcfcc14f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfd1ca7640 1 -- 172.21.15.77:0/1016714151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc152d80 con 0x7fcfcc1515e0 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfd1ca7640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc1515e0 0x7fcfcc1599a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800d5f0 tx=0x7fcfc800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfc2ffd640 1 -- 172.21.15.77:0/1016714151 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80125f0 con 0x7fcfcc1515e0 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfc2ffd640 1 -- 172.21.15.77:0/1016714151 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc8019070 con 0x7fcfcc1515e0 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc153060 con 0x7fcfcc1515e0 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfc2ffd640 1 -- 172.21.15.77:0/1016714151 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80129d0 con 0x7fcfcc1515e0 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.998+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc15a460 con 0x7fcfcc1515e0 2024-03-06T07:23:57.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.999+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc068840 con 0x7fcfcc1515e0 2024-03-06T07:23:57.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:56.999+0000 7fcfc2ffd640 1 -- 172.21.15.77:0/1016714151 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfc8012b70 con 0x7fcfcc1515e0 2024-03-06T07:23:57.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.000+0000 7fcfc2ffd640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcfb006b4d0 0x7fcfb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.000+0000 7fcfc2ffd640 1 -- 172.21.15.77:0/1016714151 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcfc808a5e0 con 0x7fcfcc1515e0 2024-03-06T07:23:57.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.000+0000 7fcfd24a8640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcfb006b4d0 0x7fcfb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.000+0000 7fcfd24a8640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcfb006b4d0 0x7fcfb006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc003610 tx=0x7fcfbc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.004+0000 7fcfc2ffd640 1 -- 172.21.15.77:0/1016714151 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfc8058d40 con 0x7fcfcc1515e0 2024-03-06T07:23:57.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.201+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfcc04f330 con 0x7fcfcc1515e0 2024-03-06T07:23:57.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.202+0000 7fcfc2ffd640 1 -- 172.21.15.77:0/1016714151 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16753 (secure 0 0 0) 0x7fcfc805cde0 con 0x7fcfcc1515e0 2024-03-06T07:23:57.204 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:57.205 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:48.090066+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:30.703498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:31.400205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3b","mappings":[{"from":0,"to":4}]},{"pgid":"7.3c","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:57.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcfb006b4d0 msgr2=0x7fcfb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcfb006b4d0 0x7fcfb006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc003610 tx=0x7fcfbc008040 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc1515e0 msgr2=0x7fcfcc1599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc1515e0 0x7fcfcc1599a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800d5f0 tx=0x7fcfc800dab0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 shutdown_connections 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcfb006b4d0 0x7fcfb006d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfcc1515e0 0x7fcfcc1599a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfcc15b430 0x7fcfcc1f7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 --2- 172.21.15.77:0/1016714151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfcc0cd7a0 0x7fcfcc14f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 >> 172.21.15.77:0/1016714151 conn(0x7fcfcc0cf8a0 msgr2=0x7fcfcc0720a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 shutdown_connections 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.204+0000 7fcfd470c640 1 -- 172.21.15.77:0/1016714151 wait complete. 2024-03-06T07:23:57.217 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:30.703498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:31.400205+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:57.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:57.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.319+0000 7fcc4f2fb640 1 Processor -- start 2024-03-06T07:23:57.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.319+0000 7fcc4f2fb640 1 -- start start 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.320+0000 7fcc4f2fb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcc481614d0 0x7fcc480d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.320+0000 7fcc4f2fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d1910 0x7fcc480d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.320+0000 7fcc4f2fb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc480d2740 con 0x7fcc480d22b0 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.320+0000 7fcc4f2fb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc481618b0 con 0x7fcc481614d0 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.320+0000 7fcc4f2fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc480cf400 con 0x7fcc480d1910 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4c896640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d1910 0x7fcc480d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4d097640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcc481614d0 0x7fcc480d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4c896640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d1910 0x7fcc480d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:47258/0 (socket says 172.21.15.77:47258) 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4c896640 1 -- 172.21.15.77:0/2013325928 learned_addr learned my addr 172.21.15.77:0/2013325928 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc3ffff640 1 -- 172.21.15.77:0/2013325928 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894093835 0 0) 0x7fcc40001490 con 0x7fcc480d22b0 2024-03-06T07:23:57.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc3ffff640 1 -- 172.21.15.77:0/2013325928 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc34003520 con 0x7fcc480d22b0 2024-03-06T07:23:57.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4c896640 1 -- 172.21.15.77:0/2013325928 >> v2:172.21.15.158:3300/0 conn(0x7fcc481614d0 msgr2=0x7fcc480d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4c896640 1 --2- 172.21.15.77:0/2013325928 >> v2:172.21.15.158:3300/0 conn(0x7fcc481614d0 0x7fcc480d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4c896640 1 -- 172.21.15.77:0/2013325928 >> v1:172.21.15.158:6789/0 conn(0x7fcc480d22b0 legacy=0x7fcc480d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4c896640 1 -- 172.21.15.77:0/2013325928 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc480cf6a0 con 0x7fcc480d1910 2024-03-06T07:23:57.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.321+0000 7fcc4d097640 1 --2- 172.21.15.77:0/2013325928 >> v2:172.21.15.158:3300/0 conn(0x7fcc481614d0 0x7fcc480d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.322+0000 7fcc4c896640 1 --2- 172.21.15.77:0/2013325928 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d1910 0x7fcc480d1d70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009870 tx=0x7fcc4402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d37fa34173896e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.322+0000 7fcc3ffff640 1 -- 172.21.15.77:0/2013325928 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44034070 con 0x7fcc480d1910 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.322+0000 7fcc3ffff640 1 -- 172.21.15.77:0/2013325928 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc44030840 con 0x7fcc480d1910 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.322+0000 7fcc3ffff640 1 -- 172.21.15.77:0/2013325928 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44030bf0 con 0x7fcc480d1910 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.322+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/2013325928 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d1910 msgr2=0x7fcc480d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.322+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/2013325928 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d1910 0x7fcc480d1d70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009870 tx=0x7fcc4402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.323+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/2013325928 shutdown_connections 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.323+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/2013325928 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d1910 0x7fcc480d1d70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.323+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/2013325928 >> v2:172.21.15.158:3300/0 conn(0x7fcc481614d0 0x7fcc480d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.323+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/2013325928 >> 172.21.15.77:0/2013325928 conn(0x7fcc480d3d40 msgr2=0x7fcc480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.323+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/2013325928 shutdown_connections 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.323+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/2013325928 wait complete. 2024-03-06T07:23:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.324+0000 7fcc4f2fb640 1 Processor -- start 2024-03-06T07:23:57.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.324+0000 7fcc4f2fb640 1 -- start start 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.324+0000 7fcc4f2fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc480d1910 0x7fcc480b7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.324+0000 7fcc4f2fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d22b0 0x7fcc480c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.324+0000 7fcc4f2fb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc481614d0 0x7fcc480c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.324+0000 7fcc4f2fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc480b0910 con 0x7fcc480d1910 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.324+0000 7fcc4f2fb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc480cfc60 con 0x7fcc481614d0 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.324+0000 7fcc4f2fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc480b1e40 con 0x7fcc480d22b0 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc480d1910 0x7fcc480b7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc480d1910 0x7fcc480b7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41722/0 (socket says 172.21.15.77:41722) 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 -- 172.21.15.77:0/4244334653 learned_addr learned my addr 172.21.15.77:0/4244334653 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4c896640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d22b0 0x7fcc480c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d898640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc481614d0 0x7fcc480c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 -- 172.21.15.77:0/4244334653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d22b0 msgr2=0x7fcc480c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d22b0 0x7fcc480c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 -- 172.21.15.77:0/4244334653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc481614d0 msgr2=0x7fcc480c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc481614d0 0x7fcc480c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 -- 172.21.15.77:0/4244334653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc480c91b0 con 0x7fcc480d1910 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4d097640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc480d1910 0x7fcc480b7440 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcc3800e960 tx=0x7fcc3800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc3dffb640 1 -- 172.21.15.77:0/4244334653 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38017070 con 0x7fcc480d1910 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc3dffb640 1 -- 172.21.15.77:0/4244334653 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3800ca00 con 0x7fcc480d1910 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc3dffb640 1 -- 172.21.15.77:0/4244334653 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3800cdd0 con 0x7fcc480d1910 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc480cb060 con 0x7fcc480d1910 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.325+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc480c9e20 con 0x7fcc480d1910 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.327+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc480d1060 con 0x7fcc480d1910 2024-03-06T07:23:57.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.327+0000 7fcc3dffb640 1 -- 172.21.15.77:0/4244334653 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc3801a070 con 0x7fcc480d1910 2024-03-06T07:23:57.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.327+0000 7fcc3dffb640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2c06b4d0 0x7fcc2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.328+0000 7fcc4c896640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2c06b4d0 0x7fcc2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.328+0000 7fcc3dffb640 1 -- 172.21.15.77:0/4244334653 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcc3808a370 con 0x7fcc480d1910 2024-03-06T07:23:57.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.328+0000 7fcc4c896640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2c06b4d0 0x7fcc2c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcc44036000 tx=0x7fcc44008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.332+0000 7fcc3dffb640 1 -- 172.21.15.77:0/4244334653 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc38058bf0 con 0x7fcc480d1910 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.508+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc48068ff0 con 0x7fcc480d1910 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.509+0000 7fcc3dffb640 1 -- 172.21.15.77:0/4244334653 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc3805cc90 con 0x7fcc480d1910 2024-03-06T07:23:57.511 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:23:57.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.510+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2c06b4d0 msgr2=0x7fcc2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.510+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2c06b4d0 0x7fcc2c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcc44036000 tx=0x7fcc44008030 comp rx=0 tx=0).stop 2024-03-06T07:23:57.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.510+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc480d1910 msgr2=0x7fcc480b7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.510+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc480d1910 0x7fcc480b7440 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcc3800e960 tx=0x7fcc3800ee20 comp rx=0 tx=0).stop 2024-03-06T07:23:57.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.511+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 shutdown_connections 2024-03-06T07:23:57.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.511+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc481614d0 0x7fcc480c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.511+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc480d22b0 0x7fcc480c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.511+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc2c06b4d0 0x7fcc2c06d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.511+0000 7fcc4f2fb640 1 --2- 172.21.15.77:0/4244334653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc480d1910 0x7fcc480b7440 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.511+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 >> 172.21.15.77:0/4244334653 conn(0x7fcc480d3d40 msgr2=0x7fcc48073fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.511+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 shutdown_connections 2024-03-06T07:23:57.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.511+0000 7fcc4f2fb640 1 -- 172.21.15.77:0/4244334653 wait complete. 2024-03-06T07:23:57.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:57.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.627+0000 7fc0c39bc640 1 Processor -- start 2024-03-06T07:23:57.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.628+0000 7fc0c39bc640 1 -- start start 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c39bc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0bc15ecd0 0x7fc0bc0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c39bc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc0cfa50 con 0x7fc0bc0d14a0 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c39bc640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc0b45c0 con 0x7fc0bc0d10c0 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c39bc640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0bc15c050 con 0x7fc0bc15ecd0 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c1f59640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0bc15ecd0 0x7fc0bc0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c1f59640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0bc15ecd0 0x7fc0bc0cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41732/0 (socket says 172.21.15.77:41732) 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c0f57640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc0bc0d14a0 0x7fc0bc15e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51330/0 (socket says 172.21.15.77:51330) 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c1f59640 1 -- 172.21.15.77:0/1225939659 learned_addr learned my addr 172.21.15.77:0/1225939659 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c1f59640 1 -- 172.21.15.77:0/1225939659 >> v1:172.21.15.158:6789/0 conn(0x7fc0bc0d10c0 legacy=0x7fc0bc0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c1f59640 1 -- 172.21.15.77:0/1225939659 >> v1:172.21.15.77:6789/0 conn(0x7fc0bc0d14a0 legacy=0x7fc0bc15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.629+0000 7fc0c1f59640 1 -- 172.21.15.77:0/1225939659 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc15f160 con 0x7fc0bc15ecd0 2024-03-06T07:23:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.630+0000 7fc0c1f59640 1 --2- 172.21.15.77:0/1225939659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc15ecd0 0x7fc0bc0cae80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009e50 tx=0x7fc0b402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d884f2909fd5e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.630+0000 7fc0b3fff640 1 -- 172.21.15.77:0/1225939659 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b402cbb0 con 0x7fc0bc15ecd0 2024-03-06T07:23:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.630+0000 7fc0b3fff640 1 -- 172.21.15.77:0/1225939659 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b402cd50 con 0x7fc0bc15ecd0 2024-03-06T07:23:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.630+0000 7fc0b3fff640 1 -- 172.21.15.77:0/1225939659 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4034610 con 0x7fc0bc15ecd0 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.631+0000 7fc0c39bc640 1 -- 172.21.15.77:0/1225939659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc15ecd0 msgr2=0x7fc0bc0cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.631+0000 7fc0c39bc640 1 --2- 172.21.15.77:0/1225939659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc15ecd0 0x7fc0bc0cae80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009e50 tx=0x7fc0b402c010 comp rx=0 tx=0).stop 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.631+0000 7fc0c39bc640 1 -- 172.21.15.77:0/1225939659 shutdown_connections 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.631+0000 7fc0c39bc640 1 --2- 172.21.15.77:0/1225939659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc15ecd0 0x7fc0bc0cae80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.631+0000 7fc0c39bc640 1 -- 172.21.15.77:0/1225939659 >> 172.21.15.77:0/1225939659 conn(0x7fc0bc0b3800 msgr2=0x7fc0bc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.631+0000 7fc0c39bc640 1 -- 172.21.15.77:0/1225939659 shutdown_connections 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.631+0000 7fc0c39bc640 1 -- 172.21.15.77:0/1225939659 wait complete. 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.632+0000 7fc0c39bc640 1 Processor -- start 2024-03-06T07:23:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.632+0000 7fc0c39bc640 1 -- start start 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c39bc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0bc0d10c0 0x7fc0bc0b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c39bc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0bc0d14a0 0x7fc0bc0b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c39bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc14e490 0x7fc0bc1f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c39bc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc0cfcf0 con 0x7fc0bc14e490 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c39bc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc0d0060 con 0x7fc0bc0d10c0 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c39bc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc0cb3c0 con 0x7fc0bc0d14a0 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc14e490 0x7fc0bc1f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc14e490 0x7fc0bc1f42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41742/0 (socket says 172.21.15.77:41742) 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 -- 172.21.15.77:0/2778655512 learned_addr learned my addr 172.21.15.77:0/2778655512 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1758640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0bc0d10c0 0x7fc0bc0b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c0f57640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0bc0d14a0 0x7fc0bc0b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 -- 172.21.15.77:0/2778655512 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0bc0d14a0 msgr2=0x7fc0bc0b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0bc0d14a0 0x7fc0bc0b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 -- 172.21.15.77:0/2778655512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0bc0d10c0 msgr2=0x7fc0bc0b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0bc0d10c0 0x7fc0bc0b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 -- 172.21.15.77:0/2778655512 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc0b58b0 con 0x7fc0bc14e490 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0c1f59640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc14e490 0x7fc0bc1f42c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009e50 tx=0x7fc0b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.633+0000 7fc0b1ffb640 1 -- 172.21.15.77:0/2778655512 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4043020 con 0x7fc0bc14e490 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.634+0000 7fc0b1ffb640 1 -- 172.21.15.77:0/2778655512 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b403e450 con 0x7fc0bc14e490 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.634+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0bc0b7580 con 0x7fc0bc14e490 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.634+0000 7fc0b1ffb640 1 -- 172.21.15.77:0/2778655512 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4009510 con 0x7fc0bc14e490 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.634+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc0afe10 con 0x7fc0bc14e490 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.635+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0bc09fe60 con 0x7fc0bc14e490 2024-03-06T07:23:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.635+0000 7fc0b1ffb640 1 -- 172.21.15.77:0/2778655512 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0b4046020 con 0x7fc0bc14e490 2024-03-06T07:23:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.636+0000 7fc0b1ffb640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc0a006b4d0 0x7fc0a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.636+0000 7fc0c1758640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc0a006b4d0 0x7fc0a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.636+0000 7fc0b1ffb640 1 -- 172.21.15.77:0/2778655512 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc0b40acae0 con 0x7fc0bc14e490 2024-03-06T07:23:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.636+0000 7fc0c1758640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc0a006b4d0 0x7fc0a006d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8000f60 tx=0x7fc0b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.640+0000 7fc0b1ffb640 1 -- 172.21.15.77:0/2778655512 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0b404d030 con 0x7fc0bc14e490 2024-03-06T07:23:57.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.817+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0bc06d930 con 0x7fc0bc14e490 2024-03-06T07:23:57.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.818+0000 7fc0b1ffb640 1 -- 172.21.15.77:0/2778655512 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc0b407b920 con 0x7fc0bc14e490 2024-03-06T07:23:57.820 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:23:57.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc0a006b4d0 msgr2=0x7fc0a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc0a006b4d0 0x7fc0a006d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8000f60 tx=0x7fc0b8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:57.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc14e490 msgr2=0x7fc0bc1f42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc14e490 0x7fc0bc1f42c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009e50 tx=0x7fc0b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 shutdown_connections 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc0a006b4d0 0x7fc0a006d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0bc14e490 0x7fc0bc1f42c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0bc0d14a0 0x7fc0bc0b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 --2- 172.21.15.77:0/2778655512 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc0bc0d10c0 0x7fc0bc0b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 >> 172.21.15.77:0/2778655512 conn(0x7fc0bc0b3800 msgr2=0x7fc0bc15e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 shutdown_connections 2024-03-06T07:23:57.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.820+0000 7fc0c39bc640 1 -- 172.21.15.77:0/2778655512 wait complete. 2024-03-06T07:23:57.833 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.935+0000 7f13bf642640 1 Processor -- start 2024-03-06T07:23:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.935+0000 7f13bf642640 1 -- start start 2024-03-06T07:23:57.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bf642640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bf642640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13b80d2480 0x7f13b80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bf642640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b81618b0 con 0x7f13b80d1910 2024-03-06T07:23:57.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bf642640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b80cf400 con 0x7f13b80d2480 2024-03-06T07:23:57.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bf642640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80cf6a0 con 0x7f13b81614d0 2024-03-06T07:23:57.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bdbdf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13b80d2480 0x7f13b80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bdbdf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13b80d2480 0x7f13b80b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41744/0 (socket says 172.21.15.77:41744) 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bdbdf640 1 -- 172.21.15.77:0/2544896733 learned_addr learned my addr 172.21.15.77:0/2544896733 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bd3de640 1 --2- 172.21.15.77:0/2544896733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bdbdf640 1 -- 172.21.15.77:0/2544896733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 msgr2=0x7f13b80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bdbdf640 1 --2- 172.21.15.77:0/2544896733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b80d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bdbdf640 1 -- 172.21.15.77:0/2544896733 >> v1:172.21.15.158:6789/0 conn(0x7f13b80d1910 legacy=0x7f13b80d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.936+0000 7f13bdbdf640 1 -- 172.21.15.77:0/2544896733 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b80cf980 con 0x7f13b80d2480 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.937+0000 7f13bdbdf640 1 --2- 172.21.15.77:0/2544896733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d2480 0x7f13b80b0e30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009e50 tx=0x7f13b002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e2b91af0461d0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.937+0000 7f13affff640 1 -- 172.21.15.77:0/2544896733 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b002cb00 con 0x7f13b80d2480 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.937+0000 7f13affff640 1 -- 172.21.15.77:0/2544896733 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b002cca0 con 0x7f13b80d2480 2024-03-06T07:23:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.937+0000 7f13affff640 1 -- 172.21.15.77:0/2544896733 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0034610 con 0x7f13b80d2480 2024-03-06T07:23:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.938+0000 7f13bf642640 1 -- 172.21.15.77:0/2544896733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d2480 msgr2=0x7f13b80b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.938+0000 7f13bf642640 1 --2- 172.21.15.77:0/2544896733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d2480 0x7f13b80b0e30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009e50 tx=0x7f13b002bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.938+0000 7f13bf642640 1 -- 172.21.15.77:0/2544896733 shutdown_connections 2024-03-06T07:23:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.938+0000 7f13bf642640 1 --2- 172.21.15.77:0/2544896733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d2480 0x7f13b80b0e30 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.938+0000 7f13bf642640 1 --2- 172.21.15.77:0/2544896733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.938+0000 7f13bf642640 1 -- 172.21.15.77:0/2544896733 >> 172.21.15.77:0/2544896733 conn(0x7f13b80d3d40 msgr2=0x7f13b80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.938+0000 7f13bf642640 1 -- 172.21.15.77:0/2544896733 shutdown_connections 2024-03-06T07:23:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.938+0000 7f13bf642640 1 -- 172.21.15.77:0/2544896733 wait complete. 2024-03-06T07:23:57.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.940+0000 7f13bf642640 1 Processor -- start 2024-03-06T07:23:57.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.940+0000 7f13bf642640 1 -- start start 2024-03-06T07:23:57.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bf642640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d1910 0x7f13b81f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bf642640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13b80d2480 0x7f13b81f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bf642640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b81f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bf642640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80b1370 con 0x7f13b80d1910 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bf642640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80b0910 con 0x7f13b80d2480 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bf642640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80b14f0 con 0x7f13b81614d0 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d1910 0x7f13b81f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d1910 0x7f13b81f54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41746/0 (socket says 172.21.15.77:41746) 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 -- 172.21.15.77:0/1245312686 learned_addr learned my addr 172.21.15.77:0/1245312686 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bdbdf640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b81f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bcbdd640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13b80d2480 0x7f13b81f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 -- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 msgr2=0x7f13b81f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b81f4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 -- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13b80d2480 msgr2=0x7f13b81f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13b80d2480 0x7f13b81f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 -- 172.21.15.77:0/1245312686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b81f7640 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bcbdd640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13b80d2480 0x7f13b81f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bdbdf640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b81f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13bd3de640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d1910 0x7f13b81f54b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f13b400e870 tx=0x7f13b400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13adffb640 1 -- 172.21.15.77:0/1245312686 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4017070 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.941+0000 7f13adffb640 1 -- 172.21.15.77:0/1245312686 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b400c940 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.942+0000 7f13adffb640 1 -- 172.21.15.77:0/1245312686 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400cd30 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.942+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b80b80c0 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.942+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b80c9580 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.943+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b8068ff0 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.943+0000 7f13adffb640 1 -- 172.21.15.77:0/1245312686 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13b401a070 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.944+0000 7f13adffb640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f139c06b4d0 0x7f139c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.944+0000 7f13adffb640 1 -- 172.21.15.77:0/1245312686 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f13b408a020 con 0x7f13b80d1910 2024-03-06T07:23:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.944+0000 7f13bcbdd640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f139c06b4d0 0x7f139c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.944+0000 7f13bcbdd640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f139c06b4d0 0x7f139c06d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f13b815ca50 tx=0x7f13a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:57.948+0000 7f13adffb640 1 -- 172.21.15.77:0/1245312686 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13b4058ef0 con 0x7f13b80d1910 2024-03-06T07:23:58.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.124+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f13b80ce810 con 0x7f13b80d1910 2024-03-06T07:23:58.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.124+0000 7f13adffb640 1 -- 172.21.15.77:0/1245312686 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) v1 ==== 100+0+11 (secure 0 0 0) 0x7f13b40587f0 con 0x7f13b80d1910 2024-03-06T07:23:58.126 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:23:58.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f139c06b4d0 msgr2=0x7f139c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f139c06b4d0 0x7f139c06d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f13b815ca50 tx=0x7f13a8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:58.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d1910 msgr2=0x7f13b81f54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d1910 0x7f13b81f54b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f13b400e870 tx=0x7f13b400ed30 comp rx=0 tx=0).stop 2024-03-06T07:23:58.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 shutdown_connections 2024-03-06T07:23:58.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13b81614d0 0x7f13b81f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13b80d2480 0x7f13b81f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f139c06b4d0 0x7f139c06d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 --2- 172.21.15.77:0/1245312686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13b80d1910 0x7f13b81f54b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 >> 172.21.15.77:0/1245312686 conn(0x7f13b80d3d40 msgr2=0x7f13b80d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 shutdown_connections 2024-03-06T07:23:58.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.126+0000 7f13bf642640 1 -- 172.21.15.77:0/1245312686 wait complete. 2024-03-06T07:23:58.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:58.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.238+0000 7f1b5f720640 1 Processor -- start 2024-03-06T07:23:58.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.238+0000 7f1b5f720640 1 -- start start 2024-03-06T07:23:58.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5f720640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 0x7f1b580d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5f720640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1b580cfff0 0x7f1b580d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5f720640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b580ce4b0 0x7f1b580ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5f720640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b58161980 con 0x7f1b580ce4b0 2024-03-06T07:23:58.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5f720640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b58161c60 con 0x7f1b580cfff0 2024-03-06T07:23:58.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5f720640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58161f40 con 0x7f1b5815bb20 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5dcbd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b580ce4b0 0x7f1b580ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5dcbd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b580ce4b0 0x7f1b580ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41754/0 (socket says 172.21.15.77:41754) 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5dcbd640 1 -- 172.21.15.77:0/2634992914 learned_addr learned my addr 172.21.15.77:0/2634992914 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5dcbd640 1 -- 172.21.15.77:0/2634992914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 msgr2=0x7f1b580d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5dcbd640 1 --2- 172.21.15.77:0/2634992914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 0x7f1b580d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5dcbd640 1 -- 172.21.15.77:0/2634992914 >> v2:172.21.15.158:3300/0 conn(0x7f1b580cfff0 msgr2=0x7f1b580d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5dcbd640 1 --2- 172.21.15.77:0/2634992914 >> v2:172.21.15.158:3300/0 conn(0x7f1b580cfff0 0x7f1b580d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.239+0000 7f1b5dcbd640 1 -- 172.21.15.77:0/2634992914 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b58162220 con 0x7f1b580ce4b0 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b5dcbd640 1 --2- 172.21.15.77:0/2634992914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580ce4b0 0x7f1b580ce940 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1b5806bbf0 tx=0x7f1b5002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234f0ecda96e176b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b4ffff640 1 -- 172.21.15.77:0/2634992914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5002ca70 con 0x7f1b580ce4b0 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b4ffff640 1 -- 172.21.15.77:0/2634992914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5002cc10 con 0x7f1b580ce4b0 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b4ffff640 1 -- 172.21.15.77:0/2634992914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b500343f0 con 0x7f1b580ce4b0 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b5f720640 1 -- 172.21.15.77:0/2634992914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580ce4b0 msgr2=0x7f1b580ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b5f720640 1 --2- 172.21.15.77:0/2634992914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580ce4b0 0x7f1b580ce940 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1b5806bbf0 tx=0x7f1b5002bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b5f720640 1 -- 172.21.15.77:0/2634992914 shutdown_connections 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b5f720640 1 --2- 172.21.15.77:0/2634992914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580ce4b0 0x7f1b580ce940 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b5f720640 1 --2- 172.21.15.77:0/2634992914 >> v2:172.21.15.158:3300/0 conn(0x7f1b580cfff0 0x7f1b580d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b5f720640 1 --2- 172.21.15.77:0/2634992914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 0x7f1b580d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.240+0000 7f1b5f720640 1 -- 172.21.15.77:0/2634992914 >> 172.21.15.77:0/2634992914 conn(0x7f1b5809a9f0 msgr2=0x7f1b5809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.241+0000 7f1b5f720640 1 -- 172.21.15.77:0/2634992914 shutdown_connections 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.241+0000 7f1b5f720640 1 -- 172.21.15.77:0/2634992914 wait complete. 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5f720640 1 Processor -- start 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5f720640 1 -- start start 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5f720640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b580ce4b0 0x7f1b580c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5f720640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580cfff0 0x7f1b5806ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5f720640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 0x7f1b5806f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5f720640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580d24b0 con 0x7f1b580cfff0 2024-03-06T07:23:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5f720640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580d2330 con 0x7f1b580ce4b0 2024-03-06T07:23:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5f720640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580d1980 con 0x7f1b5815bb20 2024-03-06T07:23:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5ccbb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580cfff0 0x7f1b5806ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5ccbb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580cfff0 0x7f1b5806ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41770/0 (socket says 172.21.15.77:41770) 2024-03-06T07:23:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.242+0000 7f1b5ccbb640 1 -- 172.21.15.77:0/3072017562 learned_addr learned my addr 172.21.15.77:0/3072017562 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5dcbd640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 0x7f1b5806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5d4bc640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b580ce4b0 0x7f1b580c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5ccbb640 1 -- 172.21.15.77:0/3072017562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 msgr2=0x7f1b5806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5ccbb640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 0x7f1b5806f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5ccbb640 1 -- 172.21.15.77:0/3072017562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b580ce4b0 msgr2=0x7f1b580c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5ccbb640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b580ce4b0 0x7f1b580c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5ccbb640 1 -- 172.21.15.77:0/3072017562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5815f5b0 con 0x7f1b580cfff0 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5ccbb640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580cfff0 0x7f1b5806ebf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800b4a0 tx=0x7f1b4800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b4dffb640 1 -- 172.21.15.77:0/3072017562 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48014070 con 0x7f1b580cfff0 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b4dffb640 1 -- 172.21.15.77:0/3072017562 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b48010610 con 0x7f1b580cfff0 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b4dffb640 1 -- 172.21.15.77:0/3072017562 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48010a00 con 0x7f1b580cfff0 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5815e510 con 0x7f1b580cfff0 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.243+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5815ead0 con 0x7f1b580cfff0 2024-03-06T07:23:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.244+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5804f330 con 0x7f1b580cfff0 2024-03-06T07:23:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.245+0000 7f1b4dffb640 1 -- 172.21.15.77:0/3072017562 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b48017070 con 0x7f1b580cfff0 2024-03-06T07:23:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.245+0000 7f1b4dffb640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b3c06b500 0x7f1b3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.245+0000 7f1b5d4bc640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b3c06b500 0x7f1b3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.245+0000 7f1b4dffb640 1 -- 172.21.15.77:0/3072017562 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1b48089b90 con 0x7f1b580cfff0 2024-03-06T07:23:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.246+0000 7f1b5d4bc640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b3c06b500 0x7f1b3c06d9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400b810 tx=0x7f1b54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.250+0000 7f1b4dffb640 1 -- 172.21.15.77:0/3072017562 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b48058a60 con 0x7f1b580cfff0 2024-03-06T07:23:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.465+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b580b0740 con 0x7f1b580cfff0 2024-03-06T07:23:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.466+0000 7f1b4dffb640 1 -- 172.21.15.77:0/3072017562 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f1b48058360 con 0x7f1b580cfff0 2024-03-06T07:23:58.468 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:58.468 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1388591691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1388591691},{"type":"v1","addr":"172.21.15.158:6834","nonce":1388591691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/870048529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":870048529},{"type":"v1","addr":"172.21.15.158:6835","nonce":870048529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/650607050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":650607050},{"type":"v1","addr":"172.21.15.77:6836","nonce":650607050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:32.429377+0000","modified":"2024-03-06T07:23:34.434503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2240574726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2240574726},{"type":"v1","addr":"172.21.15.77:6837","nonce":2240574726}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":3}]} 2024-03-06T07:23:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 19 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.467+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b3c06b500 msgr2=0x7f1b3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b3c06b500 0x7f1b3c06d9c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400b810 tx=0x7f1b54008040 comp rx=0 tx=0).stop 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580cfff0 msgr2=0x7f1b5806ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580cfff0 0x7f1b5806ebf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800b4a0 tx=0x7f1b4800b960 comp rx=0 tx=0).stop 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 shutdown_connections 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b3c06b500 0x7f1b3c06d9c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b580cfff0 0x7f1b5806ebf0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b5815bb20 0x7f1b5806f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 --2- 172.21.15.77:0/3072017562 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b580ce4b0 0x7f1b580c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 >> 172.21.15.77:0/3072017562 conn(0x7f1b5809a9f0 msgr2=0x7f1b5809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 shutdown_connections 2024-03-06T07:23:58.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.468+0000 7f1b5f720640 1 -- 172.21.15.77:0/3072017562 wait complete. 2024-03-06T07:23:58.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:58.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.581+0000 7f3c3d5f6640 1 Processor -- start 2024-03-06T07:23:58.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.581+0000 7f3c3d5f6640 1 -- start start 2024-03-06T07:23:58.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.582+0000 7f3c3d5f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c380b4170 0x7f3c380b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.582+0000 7f3c3d5f6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3c380d4760 0x7f3c380d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c3d5f6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c380d25d0 con 0x7f3c380b4dd0 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c3d5f6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c380d4bf0 con 0x7f3c380d4760 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c3d5f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380ce9e0 con 0x7f3c380b4170 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c377fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3c380b4dd0 0x7f3c380cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45510/0 (socket says 172.21.15.77:45510) 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c377fe640 1 -- 172.21.15.77:0/252225467 learned_addr learned my addr 172.21.15.77:0/252225467 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c36ffd640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c380b4170 0x7f3c380b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c367fc640 1 --2- 172.21.15.77:0/252225467 >> v2:172.21.15.158:3300/0 conn(0x7f3c380d4760 0x7f3c380d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268260413 0 0) 0x7f3c380d25d0 con 0x7f3c380b4dd0 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c20003520 con 0x7f3c380b4dd0 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187996459 0 0) 0x7f3c20003520 con 0x7f3c380b4dd0 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c380b4170 msgr2=0x7f3c380b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c380b4170 0x7f3c380b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 >> v2:172.21.15.158:3300/0 conn(0x7f3c380d4760 msgr2=0x7f3c380d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 --2- 172.21.15.77:0/252225467 >> v2:172.21.15.158:3300/0 conn(0x7f3c380d4760 0x7f3c380d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c380d2870 con 0x7f3c380b4dd0 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3c28001670 con 0x7f3c380b4dd0 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 >> v1:172.21.15.77:6789/0 conn(0x7f3c380b4dd0 legacy=0x7f3c380cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c380b4170 0x7f3c200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c20004d40 0x7f3c200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380d4bf0 con 0x7f3c20003f30 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380ce9e0 con 0x7f3c20004d40 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.583+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2000b9c0 con 0x7f3c380b4170 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c367fc640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c36ffd640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c380b4170 0x7f3c200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c377fe640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c20004d40 0x7f3c200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c367fc640 1 -- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c380b4170 msgr2=0x7f3c200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c367fc640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c380b4170 0x7f3c200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c367fc640 1 -- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c20004d40 msgr2=0x7f3c200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c367fc640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c20004d40 0x7f3c200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c367fc640 1 -- 172.21.15.77:0/252225467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c20003520 con 0x7f3c20003f30 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.584+0000 7f3c367fc640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c20004800 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009fc0 tx=0x7f3c20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775847c3ac3fcd52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c039070 con 0x7f3c20003f30 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c02cc40 con 0x7f3c20003f30 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c35ffb640 1 -- 172.21.15.77:0/252225467 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c009c10 con 0x7f3c20003f30 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/252225467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 msgr2=0x7f3c20004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c3d5f6640 1 --2- 172.21.15.77:0/252225467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c20004800 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009fc0 tx=0x7f3c20005830 comp rx=0 tx=0).stop 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c377fe640 1 -- 172.21.15.77:0/252225467 reap_dead start 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/252225467 shutdown_connections 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/252225467 >> 172.21.15.77:0/252225467 conn(0x7f3c380d3c40 msgr2=0x7f3c380d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/252225467 shutdown_connections 2024-03-06T07:23:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.585+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/252225467 wait complete. 2024-03-06T07:23:58.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.586+0000 7f3c3d5f6640 1 Processor -- start 2024-03-06T07:23:58.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c3d5f6640 1 -- start start 2024-03-06T07:23:58.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c3d5f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c3814e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c3d5f6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c3815b290 0x7f3c3815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c3d5f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c381598c0 0x7f3c38159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c3d5f6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380b47d0 con 0x7f3c20003f30 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c3d5f6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380b4550 con 0x7f3c3815b290 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c3d5f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380b4950 con 0x7f3c381598c0 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c36ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c3814e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c36ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c3814e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50274/0 (socket says 172.21.15.77:50274) 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c36ffd640 1 -- 172.21.15.77:0/935140082 learned_addr learned my addr 172.21.15.77:0/935140082 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c377fe640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c381598c0 0x7f3c38159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.587+0000 7f3c367fc640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c3815b290 0x7f3c3815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c36ffd640 1 -- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c381598c0 msgr2=0x7f3c38159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c36ffd640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c381598c0 0x7f3c38159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c36ffd640 1 -- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c3815b290 msgr2=0x7f3c3815b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c36ffd640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c3815b290 0x7f3c3815b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c36ffd640 1 -- 172.21.15.77:0/935140082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3815a680 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c367fc640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c3815b290 0x7f3c3815b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c36ffd640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c3814e180 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3c300119d0 tx=0x7f3c30011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c1f7fe640 1 -- 172.21.15.77:0/935140082 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3000fb50 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c1f7fe640 1 -- 172.21.15.77:0/935140082 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3000fcf0 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c380cb9c0 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c1f7fe640 1 -- 172.21.15.77:0/935140082 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3000fea0 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.588+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c380cbf80 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.589+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3804f330 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.589+0000 7f3c1f7fe640 1 -- 172.21.15.77:0/935140082 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c30012040 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.590+0000 7f3c1f7fe640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c2406b4d0 0x7f3c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.590+0000 7f3c1f7fe640 1 -- 172.21.15.77:0/935140082 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3c3008a8b0 con 0x7f3c20003f30 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.590+0000 7f3c367fc640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c2406b4d0 0x7f3c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.590+0000 7f3c367fc640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c2406b4d0 0x7f3c2406d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c036000 tx=0x7f3c2c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.595+0000 7f3c1f7fe640 1 -- 172.21.15.77:0/935140082 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c30019030 con 0x7f3c20003f30 2024-03-06T07:23:58.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.793+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c380671c0 con 0x7f3c20003f30 2024-03-06T07:23:58.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.794+0000 7f3c1f7fe640 1 -- 172.21.15.77:0/935140082 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16753 (secure 0 0 0) 0x7f3c300596f0 con 0x7f3c20003f30 2024-03-06T07:23:58.796 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:58.796 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:48.090066+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:30.703498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:31.400205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3b","mappings":[{"from":0,"to":4}]},{"pgid":"7.3c","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c2406b4d0 msgr2=0x7f3c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c2406b4d0 0x7f3c2406d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c036000 tx=0x7f3c2c02e000 comp rx=0 tx=0).stop 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 msgr2=0x7f3c3814e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c3814e180 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3c300119d0 tx=0x7f3c30011e90 comp rx=0 tx=0).stop 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 shutdown_connections 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c381598c0 0x7f3c38159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c3815b290 0x7f3c3815b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c2406b4d0 0x7f3c2406d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 --2- 172.21.15.77:0/935140082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c20003f30 0x7f3c3814e180 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 >> 172.21.15.77:0/935140082 conn(0x7f3c380d3c40 msgr2=0x7f3c380cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 shutdown_connections 2024-03-06T07:23:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.796+0000 7f3c3d5f6640 1 -- 172.21.15.77:0/935140082 wait complete. 2024-03-06T07:23:58.809 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:58.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.908+0000 7f8a12a39640 1 Processor -- start 2024-03-06T07:23:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.909+0000 7f8a12a39640 1 -- start start 2024-03-06T07:23:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.909+0000 7f8a12a39640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8a0c0cf790 0x7f8a0c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.909+0000 7f8a12a39640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c0ae240 con 0x7f8a0c0adda0 2024-03-06T07:23:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.909+0000 7f8a12a39640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c0ae520 con 0x7f8a0c0723f0 2024-03-06T07:23:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.909+0000 7f8a12a39640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0c06cf70 con 0x7f8a0c0cf790 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.909+0000 7f8a0b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8a0c0adda0 0x7f8a0c071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45522/0 (socket says 172.21.15.77:45522) 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.909+0000 7f8a0b7fe640 1 -- 172.21.15.77:0/4223989939 learned_addr learned my addr 172.21.15.77:0/4223989939 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0bfff640 1 --2- 172.21.15.77:0/4223989939 >> v2:172.21.15.158:3300/0 conn(0x7f8a0c0cf790 0x7f8a0c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965860074 0 0) 0x7f8a0c0ae240 con 0x7f8a0c0adda0 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f4003520 con 0x7f8a0c0adda0 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 498646890 0 0) 0x7f89f4003520 con 0x7f8a0c0adda0 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 >> v2:172.21.15.158:3300/0 conn(0x7f8a0c0cf790 msgr2=0x7f8a0c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 --2- 172.21.15.77:0/4223989939 >> v2:172.21.15.158:3300/0 conn(0x7f8a0c0cf790 0x7f8a0c0ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 >> v1:172.21.15.158:6789/0 conn(0x7f8a0c0723f0 legacy=0x7f8a0c06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c0ae800 con 0x7f8a0c0adda0 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f8a00001690 con 0x7f8a0c0adda0 2024-03-06T07:23:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 >> v1:172.21.15.77:6789/0 conn(0x7f8a0c0adda0 legacy=0x7f8a0c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40039b0 0x7f89f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f4004800 0x7f89f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f40051c0 0x7f89f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c06cf70 con 0x7f89f40051c0 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f400c060 con 0x7f89f40039b0 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f400c260 con 0x7f89f4004800 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a10fd6640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f40051c0 0x7f89f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0bfff640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40039b0 0x7f89f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a0b7fe640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f4004800 0x7f89f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a10fd6640 1 -- 172.21.15.77:0/4223989939 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f4004800 msgr2=0x7f89f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.910+0000 7f8a10fd6640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f4004800 0x7f89f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.911+0000 7f8a10fd6640 1 -- 172.21.15.77:0/4223989939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40039b0 msgr2=0x7f89f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.911+0000 7f8a10fd6640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89f40039b0 0x7f89f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.911+0000 7f8a10fd6640 1 -- 172.21.15.77:0/4223989939 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f4003520 con 0x7f89f40051c0 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a10fd6640 1 -- 172.21.15.77:0/4223989939 reap_dead start 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a10fd6640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f40051c0 0x7f89f400b9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c074940 tx=0x7f89f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a44190250b604a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04001a70 con 0x7f89f40051c0 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a04003b70 con 0x7f89f40051c0 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a0affd640 1 -- 172.21.15.77:0/4223989939 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a040050d0 con 0x7f89f40051c0 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a12a39640 1 -- 172.21.15.77:0/4223989939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f40051c0 msgr2=0x7f89f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a12a39640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f40051c0 0x7f89f400b9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c074940 tx=0x7f89f4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a12a39640 1 -- 172.21.15.77:0/4223989939 shutdown_connections 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a12a39640 1 --2- 172.21.15.77:0/4223989939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89f40051c0 0x7f89f400b9c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a12a39640 1 -- 172.21.15.77:0/4223989939 >> 172.21.15.77:0/4223989939 conn(0x7f8a0c0b1a90 msgr2=0x7f8a0c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a12a39640 1 -- 172.21.15.77:0/4223989939 shutdown_connections 2024-03-06T07:23:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.912+0000 7f8a12a39640 1 -- 172.21.15.77:0/4223989939 wait complete. 2024-03-06T07:23:58.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a12a39640 1 Processor -- start 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a12a39640 1 -- start start 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a12a39640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f40051c0 0x7f8a0c151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a12a39640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a0c14a8a0 0x7f8a0c140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a12a39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a0c155f20 0x7f8a0c1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a12a39640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c15e700 con 0x7f8a0c155f20 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a12a39640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c15e580 con 0x7f8a0c14a8a0 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a12a39640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0ad860 con 0x7f89f40051c0 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a10fd6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a0c155f20 0x7f8a0c1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a10fd6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a0c155f20 0x7f8a0c1563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50294/0 (socket says 172.21.15.77:50294) 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.914+0000 7f8a10fd6640 1 -- 172.21.15.77:0/3306819406 learned_addr learned my addr 172.21.15.77:0/3306819406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a0b7fe640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a0c14a8a0 0x7f8a0c140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a0bfff640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f40051c0 0x7f8a0c151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a10fd6640 1 -- 172.21.15.77:0/3306819406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f40051c0 msgr2=0x7f8a0c151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a10fd6640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f40051c0 0x7f8a0c151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a10fd6640 1 -- 172.21.15.77:0/3306819406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a0c14a8a0 msgr2=0x7f8a0c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a10fd6640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a0c14a8a0 0x7f8a0c140660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a10fd6640 1 -- 172.21.15.77:0/3306819406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c153900 con 0x7f8a0c155f20 2024-03-06T07:23:58.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a10fd6640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a0c155f20 0x7f8a0c1563a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8a0402c020 tx=0x7f8a040014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a08ff9640 1 -- 172.21.15.77:0/3306819406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04006b50 con 0x7f8a0c155f20 2024-03-06T07:23:58.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a08ff9640 1 -- 172.21.15.77:0/3306819406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a04006cf0 con 0x7f8a0c155f20 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0c153be0 con 0x7f8a0c155f20 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a08ff9640 1 -- 172.21.15.77:0/3306819406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04007250 con 0x7f8a0c155f20 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.915+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c1541a0 con 0x7f8a0c155f20 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.916+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0c1568e0 con 0x7f8a0c155f20 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.917+0000 7f8a08ff9640 1 -- 172.21.15.77:0/3306819406 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a040073f0 con 0x7f8a0c155f20 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.917+0000 7f8a08ff9640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f89f806b4d0 0x7f89f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.917+0000 7f8a0bfff640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f89f806b4d0 0x7f89f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.917+0000 7f8a08ff9640 1 -- 172.21.15.77:0/3306819406 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8a040ab3c0 con 0x7f8a0c155f20 2024-03-06T07:23:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.918+0000 7f8a0bfff640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f89f806b4d0 0x7f89f806d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009b40 tx=0x7f89fc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:58.922+0000 7f8a08ff9640 1 -- 172.21.15.77:0/3306819406 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a04049020 con 0x7f8a0c155f20 2024-03-06T07:23:59.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.135+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a0c04f330 con 0x7f8a0c155f20 2024-03-06T07:23:59.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.136+0000 7f8a08ff9640 1 -- 172.21.15.77:0/3306819406 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f8a0403f070 con 0x7f8a0c155f20 2024-03-06T07:23:59.138 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:59.138 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1388591691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1388591691},{"type":"v1","addr":"172.21.15.158:6834","nonce":1388591691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/870048529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":870048529},{"type":"v1","addr":"172.21.15.158:6835","nonce":870048529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/650607050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":650607050},{"type":"v1","addr":"172.21.15.77:6836","nonce":650607050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:32.429377+0000","modified":"2024-03-06T07:23:34.434503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2240574726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2240574726},{"type":"v1","addr":"172.21.15.77:6837","nonce":2240574726}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":3}]} 2024-03-06T07:23:59.138 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 19 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f89f806b4d0 msgr2=0x7f89f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f89f806b4d0 0x7f89f806d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009b40 tx=0x7f89fc009190 comp rx=0 tx=0).stop 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a0c155f20 msgr2=0x7f8a0c1563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a0c155f20 0x7f8a0c1563a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8a0402c020 tx=0x7f8a040014a0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 shutdown_connections 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f89f806b4d0 0x7f89f806d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a0c155f20 0x7f8a0c1563a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a0c14a8a0 0x7f8a0c140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 --2- 172.21.15.77:0/3306819406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89f40051c0 0x7f8a0c151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 >> 172.21.15.77:0/3306819406 conn(0x7f8a0c0b1a90 msgr2=0x7f8a0c15e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 shutdown_connections 2024-03-06T07:23:59.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.138+0000 7f8a12a39640 1 -- 172.21.15.77:0/3306819406 wait complete. 2024-03-06T07:23:59.151 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:59.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.249+0000 7f555fdd7640 1 Processor -- start 2024-03-06T07:23:59.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.249+0000 7f555fdd7640 1 -- start start 2024-03-06T07:23:59.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555fdd7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f55580d22b0 0x7f55580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555fdd7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55580b3430 con 0x7f555815dfe0 2024-03-06T07:23:59.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555fdd7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55580b3710 con 0x7f55580b41f0 2024-03-06T07:23:59.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555fdd7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55580d2c50 con 0x7f55580d22b0 2024-03-06T07:23:59.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555d372640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f55580d22b0 0x7f55580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555d372640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f55580d22b0 0x7f55580d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50298/0 (socket says 172.21.15.77:50298) 2024-03-06T07:23:59.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555db73640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f555815dfe0 0x7f55580cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45524/0 (socket says 172.21.15.77:45524) 2024-03-06T07:23:59.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555d372640 1 -- 172.21.15.77:0/3591020043 learned_addr learned my addr 172.21.15.77:0/3591020043 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:59.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555d372640 1 -- 172.21.15.77:0/3591020043 >> v1:172.21.15.158:6789/0 conn(0x7f55580b41f0 legacy=0x7f55580b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555d372640 1 -- 172.21.15.77:0/3591020043 >> v1:172.21.15.77:6789/0 conn(0x7f555815dfe0 legacy=0x7f55580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.250+0000 7f555d372640 1 -- 172.21.15.77:0/3591020043 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55580b39f0 con 0x7f55580d22b0 2024-03-06T07:23:59.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.251+0000 7f555d372640 1 --2- 172.21.15.77:0/3591020043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580d22b0 0x7f55580d2710 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f55480098a0 tx=0x7f554802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b4dcee31574a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.251+0000 7f555cb71640 1 -- 172.21.15.77:0/3591020043 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554802f840 con 0x7f55580d22b0 2024-03-06T07:23:59.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.251+0000 7f555cb71640 1 -- 172.21.15.77:0/3591020043 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554802f9e0 con 0x7f55580d22b0 2024-03-06T07:23:59.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.251+0000 7f555cb71640 1 -- 172.21.15.77:0/3591020043 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554802fdb0 con 0x7f55580d22b0 2024-03-06T07:23:59.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.252+0000 7f555fdd7640 1 -- 172.21.15.77:0/3591020043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580d22b0 msgr2=0x7f55580d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.252+0000 7f555fdd7640 1 --2- 172.21.15.77:0/3591020043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580d22b0 0x7f55580d2710 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f55480098a0 tx=0x7f554802ba20 comp rx=0 tx=0).stop 2024-03-06T07:23:59.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.252+0000 7f555fdd7640 1 -- 172.21.15.77:0/3591020043 shutdown_connections 2024-03-06T07:23:59.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.252+0000 7f555fdd7640 1 --2- 172.21.15.77:0/3591020043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580d22b0 0x7f55580d2710 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.252+0000 7f555fdd7640 1 -- 172.21.15.77:0/3591020043 >> 172.21.15.77:0/3591020043 conn(0x7f55580d3cd0 msgr2=0x7f55580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.252+0000 7f555fdd7640 1 -- 172.21.15.77:0/3591020043 shutdown_connections 2024-03-06T07:23:59.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.252+0000 7f555fdd7640 1 -- 172.21.15.77:0/3591020043 wait complete. 2024-03-06T07:23:59.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.253+0000 7f555fdd7640 1 Processor -- start 2024-03-06T07:23:59.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.253+0000 7f555fdd7640 1 -- start start 2024-03-06T07:23:59.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555fdd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580b41f0 0x7f55580cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555fdd7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55580d22b0 0x7f555815a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555fdd7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f555815dfe0 0x7f55580cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555fdd7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55580b3d60 con 0x7f55580b41f0 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555fdd7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55580b31e0 con 0x7f55580d22b0 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555fdd7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55580d1c70 con 0x7f555815dfe0 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580b41f0 0x7f55580cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580b41f0 0x7f55580cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50310/0 (socket says 172.21.15.77:50310) 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 -- 172.21.15.77:0/715157396 learned_addr learned my addr 172.21.15.77:0/715157396 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555d372640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55580d22b0 0x7f555815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555e374640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f555815dfe0 0x7f55580cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 -- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f555815dfe0 msgr2=0x7f55580cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f555815dfe0 0x7f55580cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 -- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55580d22b0 msgr2=0x7f555815a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55580d22b0 0x7f555815a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 -- 172.21.15.77:0/715157396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5558161650 con 0x7f55580b41f0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555e374640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f555815dfe0 0x7f55580cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555d372640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55580d22b0 0x7f555815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f555db73640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580b41f0 0x7f55580cada0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f555400c410 tx=0x7f555400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.254+0000 7f554e7fc640 1 -- 172.21.15.77:0/715157396 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55540074e0 con 0x7f55580b41f0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.255+0000 7f554e7fc640 1 -- 172.21.15.77:0/715157396 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5554007680 con 0x7f55580b41f0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.255+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5558161090 con 0x7f55580b41f0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.255+0000 7f554e7fc640 1 -- 172.21.15.77:0/715157396 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55540049e0 con 0x7f55580b41f0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.255+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55580cd180 con 0x7f55580b41f0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.256+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555804f330 con 0x7f55580b41f0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.256+0000 7f554e7fc640 1 -- 172.21.15.77:0/715157396 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f555401a020 con 0x7f55580b41f0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.257+0000 7f554e7fc640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f553c06b500 0x7f553c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.257+0000 7f555d372640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f553c06b500 0x7f553c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.257+0000 7f554e7fc640 1 -- 172.21.15.77:0/715157396 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f55540899a0 con 0x7f55580b41f0 2024-03-06T07:23:59.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.257+0000 7f555d372640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f553c06b500 0x7f553c06d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5548036000 tx=0x7f5548039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.261+0000 7f554e7fc640 1 -- 172.21.15.77:0/715157396 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5554010300 con 0x7f55580b41f0 2024-03-06T07:23:59.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.476+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55580ce9f0 con 0x7f55580b41f0 2024-03-06T07:23:59.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.477+0000 7f554e7fc640 1 -- 172.21.15.77:0/715157396 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f5554058870 con 0x7f55580b41f0 2024-03-06T07:23:59.479 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:59.479 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1388591691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1388591691},{"type":"v1","addr":"172.21.15.158:6834","nonce":1388591691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/870048529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":870048529},{"type":"v1","addr":"172.21.15.158:6835","nonce":870048529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/650607050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":650607050},{"type":"v1","addr":"172.21.15.77:6836","nonce":650607050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:32.429377+0000","modified":"2024-03-06T07:23:34.434503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"c","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2240574726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2240574726},{"type":"v1","addr":"172.21.15.77:6837","nonce":2240574726}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":3}]} 2024-03-06T07:23:59.479 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 19 2024-03-06T07:23:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f553c06b500 msgr2=0x7f553c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f553c06b500 0x7f553c06d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5548036000 tx=0x7f5548039040 comp rx=0 tx=0).stop 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580b41f0 msgr2=0x7f55580cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580b41f0 0x7f55580cada0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f555400c410 tx=0x7f555400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 shutdown_connections 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f555815dfe0 0x7f55580cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55580d22b0 0x7f555815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f553c06b500 0x7f553c06d9c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 --2- 172.21.15.77:0/715157396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55580b41f0 0x7f55580cada0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.478+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 >> 172.21.15.77:0/715157396 conn(0x7f55580d3cd0 msgr2=0x7f555806cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.479+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 shutdown_connections 2024-03-06T07:23:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.479+0000 7f555fdd7640 1 -- 172.21.15.77:0/715157396 wait complete. 2024-03-06T07:23:59.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.590+0000 7f42125f6640 1 Processor -- start 2024-03-06T07:23:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.590+0000 7f42125f6640 1 -- start start 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f42125f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c15dfe0 0x7f420c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f42125f6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f420c0d29c0 0x7f420c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f42125f6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c0b3430 con 0x7f420c0cf830 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f42125f6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420c0b31e0 con 0x7f420c0d29c0 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f42125f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0b3d60 con 0x7f420c15dfe0 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f4210b93640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f420c0d29c0 0x7f420c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f420b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f420c0cf830 0x7f420c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45526/0 (socket says 172.21.15.77:45526) 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f420b7fe640 1 -- 172.21.15.77:0/4032745660 learned_addr learned my addr 172.21.15.77:0/4032745660 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f420bfff640 1 --2- 172.21.15.77:0/4032745660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c15dfe0 0x7f420c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f4210b93640 1 -- 172.21.15.77:0/4032745660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c15dfe0 msgr2=0x7f420c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f4210b93640 1 --2- 172.21.15.77:0/4032745660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c15dfe0 0x7f420c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f4210b93640 1 -- 172.21.15.77:0/4032745660 >> v1:172.21.15.77:6789/0 conn(0x7f420c0cf830 legacy=0x7f420c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f4210b93640 1 -- 172.21.15.77:0/4032745660 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c0b3710 con 0x7f420c0d29c0 2024-03-06T07:23:59.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f420affd640 1 -- 172.21.15.77:0/4032745660 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396074036 0 0) 0x7f420c0b3430 con 0x7f420c0cf830 2024-03-06T07:23:59.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.592+0000 7f420bfff640 1 --2- 172.21.15.77:0/4032745660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c15dfe0 0x7f420c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:59.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.593+0000 7f4210b93640 1 --2- 172.21.15.77:0/4032745660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c0d29c0 0x7f420c0b41f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4204009d90 tx=0x7f420402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db443df74d7af49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.593+0000 7f420affd640 1 -- 172.21.15.77:0/4032745660 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204039070 con 0x7f420c0d29c0 2024-03-06T07:23:59.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.593+0000 7f420affd640 1 -- 172.21.15.77:0/4032745660 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420402cc50 con 0x7f420c0d29c0 2024-03-06T07:23:59.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.593+0000 7f420affd640 1 -- 172.21.15.77:0/4032745660 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204034440 con 0x7f420c0d29c0 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.594+0000 7f42125f6640 1 -- 172.21.15.77:0/4032745660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c0d29c0 msgr2=0x7f420c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.594+0000 7f42125f6640 1 --2- 172.21.15.77:0/4032745660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c0d29c0 0x7f420c0b41f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4204009d90 tx=0x7f420402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.594+0000 7f42125f6640 1 -- 172.21.15.77:0/4032745660 shutdown_connections 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.594+0000 7f42125f6640 1 --2- 172.21.15.77:0/4032745660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c0d29c0 0x7f420c0b41f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.594+0000 7f42125f6640 1 --2- 172.21.15.77:0/4032745660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c15dfe0 0x7f420c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.594+0000 7f42125f6640 1 -- 172.21.15.77:0/4032745660 >> 172.21.15.77:0/4032745660 conn(0x7f420c0d3cd0 msgr2=0x7f420c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.594+0000 7f42125f6640 1 -- 172.21.15.77:0/4032745660 shutdown_connections 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.594+0000 7f42125f6640 1 -- 172.21.15.77:0/4032745660 wait complete. 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.595+0000 7f42125f6640 1 Processor -- start 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.595+0000 7f42125f6640 1 -- start start 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f42125f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c0cf830 0x7f420c15cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f42125f6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0d29c0 0x7f420c09e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f42125f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c15dfe0 0x7f420c0cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f42125f6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0d1df0 con 0x7f420c15dfe0 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f42125f6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0d1c70 con 0x7f420c0d29c0 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f42125f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c15e3c0 con 0x7f420c0cf830 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c15dfe0 0x7f420c0cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c15dfe0 0x7f420c0cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50334/0 (socket says 172.21.15.77:50334) 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 -- 172.21.15.77:0/951900858 learned_addr learned my addr 172.21.15.77:0/951900858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f420bfff640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c0cf830 0x7f420c15cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f420b7fe640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0d29c0 0x7f420c09e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 -- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c0cf830 msgr2=0x7f420c15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c0cf830 0x7f420c15cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 -- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0d29c0 msgr2=0x7f420c09e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0d29c0 0x7f420c09e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 -- 172.21.15.77:0/951900858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c15e990 con 0x7f420c15dfe0 2024-03-06T07:23:59.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f420b7fe640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0d29c0 0x7f420c09e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4210b93640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c15dfe0 0x7f420c0cce50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4204009d90 tx=0x7f4204004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4208ff9640 1 -- 172.21.15.77:0/951900858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204044020 con 0x7f420c15dfe0 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4208ff9640 1 -- 172.21.15.77:0/951900858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420403a040 con 0x7f420c15dfe0 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420c15ec70 con 0x7f420c15dfe0 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f4208ff9640 1 -- 172.21.15.77:0/951900858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204039050 con 0x7f420c15dfe0 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.596+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420c15f230 con 0x7f420c15dfe0 2024-03-06T07:23:59.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.597+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420c04f330 con 0x7f420c15dfe0 2024-03-06T07:23:59.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.598+0000 7f4208ff9640 1 -- 172.21.15.77:0/951900858 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4204003990 con 0x7f420c15dfe0 2024-03-06T07:23:59.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.598+0000 7f4208ff9640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f41f806b4d0 0x7f41f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.598+0000 7f4208ff9640 1 -- 172.21.15.77:0/951900858 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f42040ac620 con 0x7f420c15dfe0 2024-03-06T07:23:59.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.598+0000 7f420bfff640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f41f806b4d0 0x7f41f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.599+0000 7f420bfff640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f41f806b4d0 0x7f41f806d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0096d0 tx=0x7f41fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.602+0000 7f4208ff9640 1 -- 172.21.15.77:0/951900858 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f420404e060 con 0x7f420c15dfe0 2024-03-06T07:23:59.728 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:59.730 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:59.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.782+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f420c06a190 con 0x7f420c15dfe0 2024-03-06T07:23:59.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.783+0000 7f4208ff9640 1 -- 172.21.15.77:0/951900858 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16753 (secure 0 0 0) 0x7f420407b4d0 con 0x7f420c15dfe0 2024-03-06T07:23:59.785 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:23:59.785 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:23:48.090066+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:30.703498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:31.400205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3b","mappings":[{"from":0,"to":4}]},{"pgid":"7.3c","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:59.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f41f806b4d0 msgr2=0x7f41f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f41f806b4d0 0x7f41f806d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0096d0 tx=0x7f41fc008040 comp rx=0 tx=0).stop 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c15dfe0 msgr2=0x7f420c0cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c15dfe0 0x7f420c0cce50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4204009d90 tx=0x7f4204004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 shutdown_connections 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f41f806b4d0 0x7f41f806d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f420c15dfe0 0x7f420c0cce50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f420c0d29c0 0x7f420c09e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 --2- 172.21.15.77:0/951900858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f420c0cf830 0x7f420c15cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 >> 172.21.15.77:0/951900858 conn(0x7f420c0d3cd0 msgr2=0x7f420c0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 shutdown_connections 2024-03-06T07:23:59.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.785+0000 7f42125f6640 1 -- 172.21.15.77:0/951900858 wait complete. 2024-03-06T07:23:59.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:59.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.938+0000 7f51cb0c0640 1 Processor -- start 2024-03-06T07:23:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.938+0000 7f51cb0c0640 1 -- start start 2024-03-06T07:23:59.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.939+0000 7f51cb0c0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f51c4162400 0x7f51c40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.939+0000 7f51cb0c0640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40d22c0 con 0x7f51c40b4460 2024-03-06T07:23:59.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.939+0000 7f51cb0c0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40d25a0 con 0x7f51c4162970 2024-03-06T07:23:59.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.939+0000 7f51cb0c0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c4161b00 con 0x7f51c4162400 2024-03-06T07:23:59.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.939+0000 7f51c8e5c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f51c40b4460 0x7f51c40b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45538/0 (socket says 172.21.15.77:45538) 2024-03-06T07:23:59.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.939+0000 7f51c8e5c640 1 -- 172.21.15.77:0/1972789052 learned_addr learned my addr 172.21.15.77:0/1972789052 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c965d640 1 --2- 172.21.15.77:0/1972789052 >> v2:172.21.15.158:3300/0 conn(0x7f51c4162400 0x7f51c40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903282442 0 0) 0x7f51c40d22c0 con 0x7f51c40b4460 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b0003520 con 0x7f51c40b4460 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1411523317 0 0) 0x7f51b0003520 con 0x7f51c40b4460 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 >> v2:172.21.15.158:3300/0 conn(0x7f51c4162400 msgr2=0x7f51c40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 --2- 172.21.15.77:0/1972789052 >> v2:172.21.15.158:3300/0 conn(0x7f51c4162400 0x7f51c40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 >> v1:172.21.15.158:6789/0 conn(0x7f51c4162970 legacy=0x7f51c4161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40d2880 con 0x7f51c40b4460 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f51bc0019f0 con 0x7f51c40b4460 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 >> v1:172.21.15.77:6789/0 conn(0x7f51c40b4460 legacy=0x7f51c40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51b0004800 0x7f51b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51b00051c0 0x7f51b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4161b00 con 0x7f51b00039b0 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b000c060 con 0x7f51b0004800 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b000c260 con 0x7f51b00051c0 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c8e5c640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c3fff640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51b0004800 0x7f51b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c965d640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51b00051c0 0x7f51b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c8e5c640 1 -- 172.21.15.77:0/1972789052 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51b00051c0 msgr2=0x7f51b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c8e5c640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51b00051c0 0x7f51b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c8e5c640 1 -- 172.21.15.77:0/1972789052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51b0004800 msgr2=0x7f51b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c8e5c640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51b0004800 0x7f51b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c8e5c640 1 -- 172.21.15.77:0/1972789052 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b0003520 con 0x7f51b00039b0 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.940+0000 7f51c965d640 1 -- 172.21.15.77:0/1972789052 reap_dead start 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.941+0000 7f51c8e5c640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51b0003dd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00a860 tx=0x7f51b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea49518c3deff8ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.941+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc00a480 con 0x7f51b00039b0 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.941+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51bc034630 con 0x7f51b00039b0 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.941+0000 7f51c37fe640 1 -- 172.21.15.77:0/1972789052 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc034aa0 con 0x7f51b00039b0 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.942+0000 7f51cb0c0640 1 -- 172.21.15.77:0/1972789052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 msgr2=0x7f51b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.942+0000 7f51cb0c0640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51b0003dd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00a860 tx=0x7f51b0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.942+0000 7f51cb0c0640 1 -- 172.21.15.77:0/1972789052 shutdown_connections 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.942+0000 7f51cb0c0640 1 --2- 172.21.15.77:0/1972789052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51b0003dd0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.942+0000 7f51cb0c0640 1 -- 172.21.15.77:0/1972789052 >> 172.21.15.77:0/1972789052 conn(0x7f51c40d3ce0 msgr2=0x7f51c40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.942+0000 7f51cb0c0640 1 -- 172.21.15.77:0/1972789052 shutdown_connections 2024-03-06T07:23:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.942+0000 7f51cb0c0640 1 -- 172.21.15.77:0/1972789052 wait complete. 2024-03-06T07:23:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.943+0000 7f51cb0c0640 1 Processor -- start 2024-03-06T07:23:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.943+0000 7f51cb0c0640 1 -- start start 2024-03-06T07:23:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51cb0c0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51c4156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51cb0c0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51c40c9820 0x7f51c40c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51cb0c0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51c40ccc90 0x7f51c40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51cb0c0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40ce950 con 0x7f51b00039b0 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51cb0c0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40cecc0 con 0x7f51c40ccc90 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51cb0c0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40b4840 con 0x7f51c40c9820 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c8e5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51c4156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c8e5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51c4156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50352/0 (socket says 172.21.15.77:50352) 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c8e5c640 1 -- 172.21.15.77:0/73892392 learned_addr learned my addr 172.21.15.77:0/73892392 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c965d640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51c40ccc90 0x7f51c40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c3fff640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51c40c9820 0x7f51c40c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c8e5c640 1 -- 172.21.15.77:0/73892392 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51c40c9820 msgr2=0x7f51c40c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c8e5c640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51c40c9820 0x7f51c40c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c8e5c640 1 -- 172.21.15.77:0/73892392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51c40ccc90 msgr2=0x7f51c40cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c8e5c640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51c40ccc90 0x7f51c40cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.944+0000 7f51c8e5c640 1 -- 172.21.15.77:0/73892392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40b6a60 con 0x7f51b00039b0 2024-03-06T07:23:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.945+0000 7f51c8e5c640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51c4156130 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00a980 tx=0x7f51c40b3c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.945+0000 7f51c17fa640 1 -- 172.21.15.77:0/73892392 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc041020 con 0x7f51b00039b0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.945+0000 7f51c17fa640 1 -- 172.21.15.77:0/73892392 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51bc034430 con 0x7f51b00039b0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.945+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c40b6d40 con 0x7f51b00039b0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.945+0000 7f51c17fa640 1 -- 172.21.15.77:0/73892392 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc004d60 con 0x7f51b00039b0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.945+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c40b7300 con 0x7f51b00039b0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.946+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c404f330 con 0x7f51b00039b0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.946+0000 7f51c17fa640 1 -- 172.21.15.77:0/73892392 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51bc044020 con 0x7f51b00039b0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.947+0000 7f51c17fa640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51a406b500 0x7f51a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.947+0000 7f51c3fff640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51a406b500 0x7f51a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.947+0000 7f51c17fa640 1 -- 172.21.15.77:0/73892392 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f51bc0abfc0 con 0x7f51b00039b0 2024-03-06T07:23:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.947+0000 7f51c3fff640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51a406b500 0x7f51a406d9c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f51b4000c00 tx=0x7f51b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:23:59.951+0000 7f51c17fa640 1 -- 172.21.15.77:0/73892392 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51bc04b030 con 0x7f51b00039b0 2024-03-06T07:24:00.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.341+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f51c406b180 con 0x7f51b00039b0 2024-03-06T07:24:00.343 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:24:00.342+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:00.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.343+0000 7f51c17fa640 1 -- 172.21.15.77:0/73892392 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) v1 ==== 161+0+0 (secure 0 0 0) 0x7f51bc07ae70 con 0x7f51b00039b0 2024-03-06T07:24:00.345 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.345+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51a406b500 msgr2=0x7f51a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.345+0000 7f51cb0c0640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51a406b500 0x7f51a406d9c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f51b4000c00 tx=0x7f51b4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.345+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 msgr2=0x7f51c4156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.345+0000 7f51cb0c0640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51c4156130 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00a980 tx=0x7f51c40b3c90 comp rx=0 tx=0).stop 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.346+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 shutdown_connections 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.346+0000 7f51cb0c0640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51c40ccc90 0x7f51c40cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.346+0000 7f51cb0c0640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51c40c9820 0x7f51c40c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.346+0000 7f51cb0c0640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f51a406b500 0x7f51a406d9c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.346+0000 7f51cb0c0640 1 --2- 172.21.15.77:0/73892392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51b00039b0 0x7f51c4156130 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.346+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 >> 172.21.15.77:0/73892392 conn(0x7f51c40d3ce0 msgr2=0x7f51c4161270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.346+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 shutdown_connections 2024-03-06T07:24:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.346+0000 7f51cb0c0640 1 -- 172.21.15.77:0/73892392 wait complete. 2024-03-06T07:24:00.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:00.395 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:24:00.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.461+0000 7f2983257640 1 Processor -- start 2024-03-06T07:24:00.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.461+0000 7f2983257640 1 -- start start 2024-03-06T07:24:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f2983257640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f297c15a220 0x7f297c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f2983257640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f297c0712d0 0x7f297c071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f2983257640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297c0b3320 con 0x7f297c15de60 2024-03-06T07:24:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f2983257640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f297c0cf610 con 0x7f297c0712d0 2024-03-06T07:24:00.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f2983257640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f297c071c70 con 0x7f297c15a220 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f297bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f297c0712d0 0x7f297c071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f297bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f297c0712d0 0x7f297c071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50380/0 (socket says 172.21.15.77:50380) 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f297bfff640 1 -- 172.21.15.77:0/862620406 learned_addr learned my addr 172.21.15.77:0/862620406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.462+0000 7f2980ff3640 1 --2- 172.21.15.77:0/862620406 >> v2:172.21.15.158:3300/0 conn(0x7f297c15a220 0x7f297c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.463+0000 7f297bfff640 1 -- 172.21.15.77:0/862620406 >> v2:172.21.15.158:3300/0 conn(0x7f297c15a220 msgr2=0x7f297c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.463+0000 7f297bfff640 1 --2- 172.21.15.77:0/862620406 >> v2:172.21.15.158:3300/0 conn(0x7f297c15a220 0x7f297c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.463+0000 7f297bfff640 1 -- 172.21.15.77:0/862620406 >> v1:172.21.15.158:6789/0 conn(0x7f297c15de60 legacy=0x7f297c15e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.463+0000 7f297bfff640 1 -- 172.21.15.77:0/862620406 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c0728d0 con 0x7f297c0712d0 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.463+0000 7f2980ff3640 1 --2- 172.21.15.77:0/862620406 >> v2:172.21.15.158:3300/0 conn(0x7f297c15a220 0x7f297c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.463+0000 7f297bfff640 1 --2- 172.21.15.77:0/862620406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c0712d0 0x7f297c071730 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f297c06ebb0 tx=0x7f297002deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b6eace790b5544 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.463+0000 7f297b7fe640 1 -- 172.21.15.77:0/862620406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970034070 con 0x7f297c0712d0 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.463+0000 7f297b7fe640 1 -- 172.21.15.77:0/862620406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297002ec60 con 0x7f297c0712d0 2024-03-06T07:24:00.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f297b7fe640 1 -- 172.21.15.77:0/862620406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970038620 con 0x7f297c0712d0 2024-03-06T07:24:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f2983257640 1 -- 172.21.15.77:0/862620406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c0712d0 msgr2=0x7f297c071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f2983257640 1 --2- 172.21.15.77:0/862620406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c0712d0 0x7f297c071730 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f297c06ebb0 tx=0x7f297002deb0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f2983257640 1 -- 172.21.15.77:0/862620406 shutdown_connections 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f2983257640 1 --2- 172.21.15.77:0/862620406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c0712d0 0x7f297c071730 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f2983257640 1 --2- 172.21.15.77:0/862620406 >> v2:172.21.15.158:3300/0 conn(0x7f297c15a220 0x7f297c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f2983257640 1 -- 172.21.15.77:0/862620406 >> 172.21.15.77:0/862620406 conn(0x7f297c06b190 msgr2=0x7f297c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f2983257640 1 -- 172.21.15.77:0/862620406 shutdown_connections 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.464+0000 7f2983257640 1 -- 172.21.15.77:0/862620406 wait complete. 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.465+0000 7f2983257640 1 Processor -- start 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.465+0000 7f2983257640 1 -- start start 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f2983257640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f297c0712d0 0x7f297c14e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f2983257640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f297c15a220 0x7f297c152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f2983257640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c15de60 0x7f297c15e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f2983257640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c072030 con 0x7f297c15de60 2024-03-06T07:24:00.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f2983257640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c072bb0 con 0x7f297c0712d0 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f2983257640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297c072530 con 0x7f297c15a220 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f29817f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c15de60 0x7f297c15e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f29817f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c15de60 0x7f297c15e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50384/0 (socket says 172.21.15.77:50384) 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f29817f4640 1 -- 172.21.15.77:0/492129945 learned_addr learned my addr 172.21.15.77:0/492129945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f297bfff640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f297c15a220 0x7f297c152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f2980ff3640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f297c0712d0 0x7f297c14e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f29817f4640 1 -- 172.21.15.77:0/492129945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f297c15a220 msgr2=0x7f297c152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f29817f4640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f297c15a220 0x7f297c152610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f29817f4640 1 -- 172.21.15.77:0/492129945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f297c0712d0 msgr2=0x7f297c14e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f29817f4640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f297c0712d0 0x7f297c14e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.466+0000 7f29817f4640 1 -- 172.21.15.77:0/492129945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c152ea0 con 0x7f297c15de60 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.467+0000 7f29817f4640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c15de60 0x7f297c15e2b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f297400c6e0 tx=0x7f297400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.467+0000 7f29797fa640 1 -- 172.21.15.77:0/492129945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2974003b70 con 0x7f297c15de60 2024-03-06T07:24:00.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.467+0000 7f29797fa640 1 -- 172.21.15.77:0/492129945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2974003d30 con 0x7f297c15de60 2024-03-06T07:24:00.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.467+0000 7f29797fa640 1 -- 172.21.15.77:0/492129945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2974005310 con 0x7f297c15de60 2024-03-06T07:24:00.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.467+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297c153180 con 0x7f297c15de60 2024-03-06T07:24:00.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.467+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297c153740 con 0x7f297c15de60 2024-03-06T07:24:00.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.468+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297c0b2990 con 0x7f297c15de60 2024-03-06T07:24:00.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.468+0000 7f29797fa640 1 -- 172.21.15.77:0/492129945 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29740054b0 con 0x7f297c15de60 2024-03-06T07:24:00.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.469+0000 7f29797fa640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f296806b500 0x7f296806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.469+0000 7f29797fa640 1 -- 172.21.15.77:0/492129945 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f297408a0d0 con 0x7f297c15de60 2024-03-06T07:24:00.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.469+0000 7f2980ff3640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f296806b500 0x7f296806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.469+0000 7f2980ff3640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f296806b500 0x7f296806d9c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f296c006fd0 tx=0x7f296c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.473+0000 7f29797fa640 1 -- 172.21.15.77:0/492129945 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f297c0b2990 con 0x7f297c15de60 2024-03-06T07:24:00.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:00.680+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f297c068ff0 con 0x7f297c15de60 2024-03-06T07:24:01.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.344+0000 7f29797fa640 1 -- 172.21.15.77:0/492129945 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2974058ec0 con 0x7f297c15de60 2024-03-06T07:24:01.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f296806b500 msgr2=0x7f296806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f296806b500 0x7f296806d9c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f296c006fd0 tx=0x7f296c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:01.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c15de60 msgr2=0x7f297c15e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c15de60 0x7f297c15e2b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f297400c6e0 tx=0x7f297400cba0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 shutdown_connections 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f296806b500 0x7f296806d9c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f297c15de60 0x7f297c15e2b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f297c15a220 0x7f297c152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 --2- 172.21.15.77:0/492129945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f297c0712d0 0x7f297c14e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 >> 172.21.15.77:0/492129945 conn(0x7f297c06b190 msgr2=0x7f297c0cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 shutdown_connections 2024-03-06T07:24:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.347+0000 7f2983257640 1 -- 172.21.15.77:0/492129945 wait complete. 2024-03-06T07:24:01.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:01.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.510+0000 7fed1fd20640 1 Processor -- start 2024-03-06T07:24:01.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.511+0000 7fed1fd20640 1 -- start start 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.511+0000 7fed1fd20640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fed1815ac00 0x7fed1806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.511+0000 7fed1fd20640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 0x7fed18070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.511+0000 7fed1fd20640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed1815e7a0 0x7fed1815ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.511+0000 7fed1fd20640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed18071250 con 0x7fed1815e7a0 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.511+0000 7fed1fd20640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed180cdc10 con 0x7fed1815ac00 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.511+0000 7fed1fd20640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1815afe0 con 0x7fed180708b0 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1e2bd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed1815e7a0 0x7fed1815ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1e2bd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed1815e7a0 0x7fed1815ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50388/0 (socket says 172.21.15.77:50388) 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1e2bd640 1 -- 172.21.15.77:0/3253386797 learned_addr learned my addr 172.21.15.77:0/3253386797 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1e2bd640 1 -- 172.21.15.77:0/3253386797 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 msgr2=0x7fed18070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1e2bd640 1 --2- 172.21.15.77:0/3253386797 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 0x7fed18070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1dabc640 1 --2- 172.21.15.77:0/3253386797 >> v2:172.21.15.158:3300/0 conn(0x7fed1815ac00 0x7fed1806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1e2bd640 1 -- 172.21.15.77:0/3253386797 >> v2:172.21.15.158:3300/0 conn(0x7fed1815ac00 msgr2=0x7fed1806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1e2bd640 1 --2- 172.21.15.77:0/3253386797 >> v2:172.21.15.158:3300/0 conn(0x7fed1815ac00 0x7fed1806c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1e2bd640 1 -- 172.21.15.77:0/3253386797 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed180cd000 con 0x7fed1815e7a0 2024-03-06T07:24:01.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.512+0000 7fed1dabc640 1 --2- 172.21.15.77:0/3253386797 >> v2:172.21.15.158:3300/0 conn(0x7fed1815ac00 0x7fed1806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:01.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1e2bd640 1 --2- 172.21.15.77:0/3253386797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed1815ec20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fed180730c0 tx=0x7fed1002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c06c5b45121448c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1caba640 1 -- 172.21.15.77:0/3253386797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1002c9e0 con 0x7fed1815e7a0 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1caba640 1 -- 172.21.15.77:0/3253386797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1002cb80 con 0x7fed1815e7a0 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1caba640 1 -- 172.21.15.77:0/3253386797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100343f0 con 0x7fed1815e7a0 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 -- 172.21.15.77:0/3253386797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 msgr2=0x7fed1815ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 --2- 172.21.15.77:0/3253386797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed1815ec20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fed180730c0 tx=0x7fed1002be40 comp rx=0 tx=0).stop 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 -- 172.21.15.77:0/3253386797 shutdown_connections 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 --2- 172.21.15.77:0/3253386797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed1815ec20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 --2- 172.21.15.77:0/3253386797 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 0x7fed18070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 --2- 172.21.15.77:0/3253386797 >> v2:172.21.15.158:3300/0 conn(0x7fed1815ac00 0x7fed1806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 -- 172.21.15.77:0/3253386797 >> 172.21.15.77:0/3253386797 conn(0x7fed180b1650 msgr2=0x7fed180b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 -- 172.21.15.77:0/3253386797 shutdown_connections 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.513+0000 7fed1fd20640 1 -- 172.21.15.77:0/3253386797 wait complete. 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1fd20640 1 Processor -- start 2024-03-06T07:24:01.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1fd20640 1 -- start start 2024-03-06T07:24:01.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1fd20640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 0x7fed1814a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1fd20640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed1815ac00 0x7fed1814d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1fd20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed18154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1fd20640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed180cd8a0 con 0x7fed1815e7a0 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1fd20640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed180cd5c0 con 0x7fed1815ac00 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1fd20640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1806ce50 con 0x7fed180708b0 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1e2bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed18154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1e2bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed18154490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50404/0 (socket says 172.21.15.77:50404) 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.515+0000 7fed1e2bd640 1 -- 172.21.15.77:0/1514912704 learned_addr learned my addr 172.21.15.77:0/1514912704 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1d2bb640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed1815ac00 0x7fed1814d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1dabc640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 0x7fed1814a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1e2bd640 1 -- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 msgr2=0x7fed1814a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1e2bd640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 0x7fed1814a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1e2bd640 1 -- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed1815ac00 msgr2=0x7fed1814d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1e2bd640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed1815ac00 0x7fed1814d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1e2bd640 1 -- 172.21.15.77:0/1514912704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1814a960 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1d2bb640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed1815ac00 0x7fed1814d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1e2bd640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed18154490 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fed1002c450 tx=0x7fed10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed0e7fc640 1 -- 172.21.15.77:0/1514912704 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10044020 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed0e7fc640 1 -- 172.21.15.77:0/1514912704 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed10040690 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1815c010 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed0e7fc640 1 -- 172.21.15.77:0/1514912704 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10040a80 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.516+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1815c5d0 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.517+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1804f330 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.517+0000 7fed0e7fc640 1 -- 172.21.15.77:0/1514912704 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed10047020 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.518+0000 7fed0e7fc640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecfc06b500 0x7fecfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.518+0000 7fed0e7fc640 1 -- 172.21.15.77:0/1514912704 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fed100accb0 con 0x7fed1815e7a0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.518+0000 7fed1dabc640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecfc06b500 0x7fecfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.518+0000 7fed1dabc640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecfc06b500 0x7fecfc06d9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fed1400a990 tx=0x7fed14008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.522+0000 7fed0e7fc640 1 -- 172.21.15.77:0/1514912704 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed1004e070 con 0x7fed1815e7a0 2024-03-06T07:24:01.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:01.694+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fed1806df60 con 0x7fed1815e7a0 2024-03-06T07:24:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.349+0000 7fed0e7fc640 1 -- 172.21.15.77:0/1514912704 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v49) v1 ==== 181+0+0 (secure 0 0 0) 0x7fed1814a960 con 0x7fed1815e7a0 2024-03-06T07:24:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.352+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecfc06b500 msgr2=0x7fecfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.352+0000 7fed1fd20640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecfc06b500 0x7fecfc06d9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fed1400a990 tx=0x7fed14008040 comp rx=0 tx=0).stop 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.352+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 msgr2=0x7fed18154490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.352+0000 7fed1fd20640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed18154490 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fed1002c450 tx=0x7fed10004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.353+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 shutdown_connections 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.353+0000 7fed1fd20640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fecfc06b500 0x7fecfc06d9c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.353+0000 7fed1fd20640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1815e7a0 0x7fed18154490 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.353+0000 7fed1fd20640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed1815ac00 0x7fed1814d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.353+0000 7fed1fd20640 1 --2- 172.21.15.77:0/1514912704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed180708b0 0x7fed1814a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.353+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 >> 172.21.15.77:0/1514912704 conn(0x7fed180b1650 msgr2=0x7fed180af630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.353+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 shutdown_connections 2024-03-06T07:24:02.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.353+0000 7fed1fd20640 1 -- 172.21.15.77:0/1514912704 wait complete. 2024-03-06T07:24:02.368 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:02.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.473+0000 7f53bb7b5640 1 Processor -- start 2024-03-06T07:24:02.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.473+0000 7f53bb7b5640 1 -- start start 2024-03-06T07:24:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53bb7b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b415ac00 0x7f53b406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53bb7b5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53b4070d10 0x7f53b415eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53bb7b5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b415afe0 con 0x7f53b40708b0 2024-03-06T07:24:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53bb7b5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b40cdc10 con 0x7f53b4070d10 2024-03-06T07:24:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53bb7b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b40cd000 con 0x7f53b415ac00 2024-03-06T07:24:02.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53b9d52640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53b4070d10 0x7f53b415eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53b9d52640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53b4070d10 0x7f53b415eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50406/0 (socket says 172.21.15.77:50406) 2024-03-06T07:24:02.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53b9d52640 1 -- 172.21.15.77:0/252210047 learned_addr learned my addr 172.21.15.77:0/252210047 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:02.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53b9d52640 1 -- 172.21.15.77:0/252210047 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b415ac00 msgr2=0x7f53b406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53b9d52640 1 --2- 172.21.15.77:0/252210047 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b415ac00 0x7f53b406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53b9d52640 1 -- 172.21.15.77:0/252210047 >> v1:172.21.15.158:6789/0 conn(0x7f53b40708b0 legacy=0x7f53b415e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.474+0000 7f53b9d52640 1 -- 172.21.15.77:0/252210047 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40cd2e0 con 0x7f53b4070d10 2024-03-06T07:24:02.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.475+0000 7f53b9d52640 1 --2- 172.21.15.77:0/252210047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b4070d10 0x7f53b415eeb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009e50 tx=0x7f53ac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bf8c753a4016f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.475+0000 7f53abfff640 1 -- 172.21.15.77:0/252210047 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac02c9e0 con 0x7f53b4070d10 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.475+0000 7f53abfff640 1 -- 172.21.15.77:0/252210047 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac02cb80 con 0x7f53b4070d10 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.475+0000 7f53abfff640 1 -- 172.21.15.77:0/252210047 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0343f0 con 0x7f53b4070d10 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.476+0000 7f53bb7b5640 1 -- 172.21.15.77:0/252210047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b4070d10 msgr2=0x7f53b415eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.476+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/252210047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b4070d10 0x7f53b415eeb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009e50 tx=0x7f53ac02be40 comp rx=0 tx=0).stop 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.476+0000 7f53bb7b5640 1 -- 172.21.15.77:0/252210047 shutdown_connections 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.476+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/252210047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b4070d10 0x7f53b415eeb0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.476+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/252210047 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b415ac00 0x7f53b406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.476+0000 7f53bb7b5640 1 -- 172.21.15.77:0/252210047 >> 172.21.15.77:0/252210047 conn(0x7f53b40b1650 msgr2=0x7f53b40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.476+0000 7f53bb7b5640 1 -- 172.21.15.77:0/252210047 shutdown_connections 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.476+0000 7f53bb7b5640 1 -- 172.21.15.77:0/252210047 wait complete. 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.477+0000 7f53bb7b5640 1 Processor -- start 2024-03-06T07:24:02.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.477+0000 7f53bb7b5640 1 -- start start 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.477+0000 7f53bb7b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b40708b0 0x7f53b414b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.477+0000 7f53bb7b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b4070d10 0x7f53b414dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.477+0000 7f53bb7b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b415ac00 0x7f53b414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.477+0000 7f53bb7b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4071190 con 0x7f53b40708b0 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.477+0000 7f53bb7b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b40cd8a0 con 0x7f53b415ac00 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.477+0000 7f53bb7b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b406ce50 con 0x7f53b4070d10 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b40708b0 0x7f53b414b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b40708b0 0x7f53b414b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50422/0 (socket says 172.21.15.77:50422) 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 -- 172.21.15.77:0/2675302849 learned_addr learned my addr 172.21.15.77:0/2675302849 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b8d50640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b4070d10 0x7f53b414dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9d52640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b415ac00 0x7f53b414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 -- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b4070d10 msgr2=0x7f53b414dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b4070d10 0x7f53b414dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 -- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b415ac00 msgr2=0x7f53b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b415ac00 0x7f53b414e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 -- 172.21.15.77:0/2675302849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b414c530 con 0x7f53b40708b0 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b8d50640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b4070d10 0x7f53b414dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53b9551640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b40708b0 0x7f53b414b220 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f53b000b4a0 tx=0x7f53b000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53a9ffb640 1 -- 172.21.15.77:0/2675302849 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0014070 con 0x7f53b40708b0 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53a9ffb640 1 -- 172.21.15.77:0/2675302849 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b0010610 con 0x7f53b40708b0 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b414c810 con 0x7f53b40708b0 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.478+0000 7f53a9ffb640 1 -- 172.21.15.77:0/2675302849 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0010a00 con 0x7f53b40708b0 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.479+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b414cdd0 con 0x7f53b40708b0 2024-03-06T07:24:02.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.480+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b40687d0 con 0x7f53b40708b0 2024-03-06T07:24:02.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.480+0000 7f53a9ffb640 1 -- 172.21.15.77:0/2675302849 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53b0017070 con 0x7f53b40708b0 2024-03-06T07:24:02.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.480+0000 7f53a9ffb640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.480+0000 7f53a9ffb640 1 -- 172.21.15.77:0/2675302849 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f53b0089bb0 con 0x7f53b40708b0 2024-03-06T07:24:02.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.480+0000 7f53b8d50640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.481+0000 7f53b8d50640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f539806b4d0 0x7f539806d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f53b41f3ff0 tx=0x7f53a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.485+0000 7f53a9ffb640 1 -- 172.21.15.77:0/2675302849 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53b0058b50 con 0x7f53b40708b0 2024-03-06T07:24:02.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:02.657+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f53b414e870 con 0x7f53b40708b0 2024-03-06T07:24:03.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.350+0000 7f53a9ffb640 1 -- 172.21.15.77:0/2675302849 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v50) v1 ==== 169+0+0 (secure 0 0 0) 0x7f53b0058450 con 0x7f53b40708b0 2024-03-06T07:24:03.352 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:03.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f539806b4d0 msgr2=0x7f539806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f539806b4d0 0x7f539806d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f53b41f3ff0 tx=0x7f53a4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:03.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b40708b0 msgr2=0x7f53b414b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b40708b0 0x7f53b414b220 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f53b000b4a0 tx=0x7f53b000b960 comp rx=0 tx=0).stop 2024-03-06T07:24:03.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 shutdown_connections 2024-03-06T07:24:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53b415ac00 0x7f53b414e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53b4070d10 0x7f53b414dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f539806b4d0 0x7f539806d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 --2- 172.21.15.77:0/2675302849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53b40708b0 0x7f53b414b220 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 >> 172.21.15.77:0/2675302849 conn(0x7f53b40b1650 msgr2=0x7f53b415f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 shutdown_connections 2024-03-06T07:24:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.354+0000 7f53bb7b5640 1 -- 172.21.15.77:0/2675302849 wait complete. 2024-03-06T07:24:03.369 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:24:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.477+0000 7fb1df95e640 1 Processor -- start 2024-03-06T07:24:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.477+0000 7fb1df95e640 1 -- start start 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1df95e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d80cf9a0 0x7fb1d80b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1df95e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1d80b55e0 0x7fb1d80ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1df95e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d806b710 con 0x7fb1d8066d90 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1df95e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d806b150 con 0x7fb1d80b55e0 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1df95e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d806b430 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1ddefb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1d80b55e0 0x7fb1d80ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1dd6fa640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d80cf9a0 0x7fb1d80b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1dd6fa640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d80cf9a0 0x7fb1d80b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:37328/0 (socket says 172.21.15.77:37328) 2024-03-06T07:24:03.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1dd6fa640 1 -- 172.21.15.77:0/1702867021 learned_addr learned my addr 172.21.15.77:0/1702867021 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:03.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1ddefb640 1 -- 172.21.15.77:0/1702867021 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d80cf9a0 msgr2=0x7fb1d80b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1ddefb640 1 --2- 172.21.15.77:0/1702867021 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d80cf9a0 0x7fb1d80b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1ddefb640 1 -- 172.21.15.77:0/1702867021 >> v1:172.21.15.158:6789/0 conn(0x7fb1d8066d90 legacy=0x7fb1d80b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1ddefb640 1 -- 172.21.15.77:0/1702867021 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d80b0970 con 0x7fb1d80b55e0 2024-03-06T07:24:03.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.478+0000 7fb1dd6fa640 1 --2- 172.21.15.77:0/1702867021 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d80cf9a0 0x7fb1d80b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:03.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.479+0000 7fb1ddefb640 1 --2- 172.21.15.77:0/1702867021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 0x7fb1d80ce040 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009d90 tx=0x7fb1d002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9913053f7b316b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.479+0000 7fb1cffff640 1 -- 172.21.15.77:0/1702867021 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0039070 con 0x7fb1d80b55e0 2024-03-06T07:24:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.479+0000 7fb1cffff640 1 -- 172.21.15.77:0/1702867021 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d002cc50 con 0x7fb1d80b55e0 2024-03-06T07:24:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.479+0000 7fb1cffff640 1 -- 172.21.15.77:0/1702867021 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0034440 con 0x7fb1d80b55e0 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.480+0000 7fb1df95e640 1 -- 172.21.15.77:0/1702867021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 msgr2=0x7fb1d80ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.480+0000 7fb1df95e640 1 --2- 172.21.15.77:0/1702867021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 0x7fb1d80ce040 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009d90 tx=0x7fb1d002bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.480+0000 7fb1df95e640 1 -- 172.21.15.77:0/1702867021 shutdown_connections 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.480+0000 7fb1df95e640 1 --2- 172.21.15.77:0/1702867021 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 0x7fb1d80ce040 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.480+0000 7fb1df95e640 1 --2- 172.21.15.77:0/1702867021 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d80cf9a0 0x7fb1d80b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.480+0000 7fb1df95e640 1 -- 172.21.15.77:0/1702867021 >> 172.21.15.77:0/1702867021 conn(0x7fb1d80b1d70 msgr2=0x7fb1d80b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.480+0000 7fb1df95e640 1 -- 172.21.15.77:0/1702867021 shutdown_connections 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.480+0000 7fb1df95e640 1 -- 172.21.15.77:0/1702867021 wait complete. 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.481+0000 7fb1df95e640 1 Processor -- start 2024-03-06T07:24:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.481+0000 7fb1df95e640 1 -- start start 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1df95e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d8066d90 0x7fb1d80cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1df95e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 0x7fb1d8152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1df95e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d80cf9a0 0x7fb1d80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1df95e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d80b6170 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1df95e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d80b5ff0 con 0x7fb1d80b55e0 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1df95e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d80cfd80 con 0x7fb1d8066d90 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d80cf9a0 0x7fb1d80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d80cf9a0 0x7fb1d80ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50424/0 (socket says 172.21.15.77:50424) 2024-03-06T07:24:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 -- 172.21.15.77:0/3273558541 learned_addr learned my addr 172.21.15.77:0/3273558541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1dcef9640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 0x7fb1d8152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1dd6fa640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d8066d90 0x7fb1d80cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 -- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d8066d90 msgr2=0x7fb1d80cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d8066d90 0x7fb1d80cb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 -- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 msgr2=0x7fb1d8152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 0x7fb1d8152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 -- 172.21.15.77:0/3273558541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d81f79c0 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1dd6fa640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d8066d90 0x7fb1d80cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1dcef9640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 0x7fb1d8152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.482+0000 7fb1ddefb640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d80cf9a0 0x7fb1d80ce7b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009eb0 tx=0x7fb1d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.483+0000 7fb1cdffb640 1 -- 172.21.15.77:0/3273558541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0039050 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.483+0000 7fb1cdffb640 1 -- 172.21.15.77:0/3273558541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d0045070 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.483+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d81f7ca0 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.483+0000 7fb1cdffb640 1 -- 172.21.15.77:0/3273558541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d003e610 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.483+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d81f8260 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.484+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d809e3d0 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.484+0000 7fb1cdffb640 1 -- 172.21.15.77:0/3273558541 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1d003e7b0 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.484+0000 7fb1cdffb640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.485+0000 7fb1cdffb640 1 -- 172.21.15.77:0/3273558541 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb1d00ac9c0 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.485+0000 7fb1dd6fa640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.485+0000 7fb1dd6fa640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009a20 tx=0x7fb1d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.489+0000 7fb1cdffb640 1 -- 172.21.15.77:0/3273558541 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1d00aee20 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.660+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fb1d8068ff0 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.664+0000 7fb1cdffb640 1 -- 172.21.15.77:0/3273558541 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v14) v1 ==== 108+0+0 (secure 0 0 0) 0x7fb1d007bcf0 con 0x7fb1d80cf9a0 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1bc06b500 msgr2=0x7fb1bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009a20 tx=0x7fb1d4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d80cf9a0 msgr2=0x7fb1d80ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d80cf9a0 0x7fb1d80ce7b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009eb0 tx=0x7fb1d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 shutdown_connections 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d80cf9a0 0x7fb1d80ce7b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d80b55e0 0x7fb1d8152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 --2- 172.21.15.77:0/3273558541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1d8066d90 0x7fb1d80cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 >> 172.21.15.77:0/3273558541 conn(0x7fb1d80b1d70 msgr2=0x7fb1d80b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 shutdown_connections 2024-03-06T07:24:03.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.666+0000 7fb1df95e640 1 -- 172.21.15.77:0/3273558541 wait complete. 2024-03-06T07:24:03.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:03.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.779+0000 7f7068bd0640 1 Processor -- start 2024-03-06T07:24:03.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.779+0000 7f7068bd0640 1 -- start start 2024-03-06T07:24:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.780+0000 7f7068bd0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70641586e0 0x7f70640aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.780+0000 7f7068bd0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f70640aef80 0x7f70640cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.780+0000 7f7068bd0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7064158ac0 con 0x7f70640cffd0 2024-03-06T07:24:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.780+0000 7f7068bd0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706415b2a0 con 0x7f70641586e0 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.780+0000 7f7068bd0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706415b540 con 0x7f70640aef80 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.780+0000 7f706259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70641586e0 0x7f70640aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.780+0000 7f706259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70641586e0 0x7f70640aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50426/0 (socket says 172.21.15.77:50426) 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.781+0000 7f706259c640 1 -- 172.21.15.77:0/1737596377 learned_addr learned my addr 172.21.15.77:0/1737596377 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.781+0000 7f7061d9b640 1 --2- 172.21.15.77:0/1737596377 >> v2:172.21.15.158:3300/0 conn(0x7f70640aef80 0x7f70640cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.781+0000 7f706259c640 1 -- 172.21.15.77:0/1737596377 >> v2:172.21.15.158:3300/0 conn(0x7f70640aef80 msgr2=0x7f70640cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.781+0000 7f706259c640 1 --2- 172.21.15.77:0/1737596377 >> v2:172.21.15.158:3300/0 conn(0x7f70640aef80 0x7f70640cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.781+0000 7f706259c640 1 -- 172.21.15.77:0/1737596377 >> v1:172.21.15.158:6789/0 conn(0x7f70640cffd0 legacy=0x7f706406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.781+0000 7f706259c640 1 -- 172.21.15.77:0/1737596377 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706415b820 con 0x7f70641586e0 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.781+0000 7f706259c640 1 --2- 172.21.15.77:0/1737596377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f70640aea40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7064072e30 tx=0x7f705c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e89fef1d17e010 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f706159a640 1 -- 172.21.15.77:0/1737596377 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c035020 con 0x7f70641586e0 2024-03-06T07:24:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f706159a640 1 -- 172.21.15.77:0/1737596377 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c031870 con 0x7f70641586e0 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f706159a640 1 -- 172.21.15.77:0/1737596377 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c031c40 con 0x7f70641586e0 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f7068bd0640 1 -- 172.21.15.77:0/1737596377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 msgr2=0x7f70640aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1737596377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f70640aea40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7064072e30 tx=0x7f705c02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f7068bd0640 1 -- 172.21.15.77:0/1737596377 shutdown_connections 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1737596377 >> v2:172.21.15.158:3300/0 conn(0x7f70640aef80 0x7f70640cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1737596377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f70640aea40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f7068bd0640 1 -- 172.21.15.77:0/1737596377 >> 172.21.15.77:0/1737596377 conn(0x7f706406b190 msgr2=0x7f706406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f7068bd0640 1 -- 172.21.15.77:0/1737596377 shutdown_connections 2024-03-06T07:24:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.782+0000 7f7068bd0640 1 -- 172.21.15.77:0/1737596377 wait complete. 2024-03-06T07:24:03.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7068bd0640 1 Processor -- start 2024-03-06T07:24:03.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7068bd0640 1 -- start start 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7068bd0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70640aef80 0x7f706414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7068bd0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70640cffd0 0x7f706406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7068bd0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f706406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7068bd0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7064071350 con 0x7f70641586e0 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7068bd0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706415bde0 con 0x7f70640aef80 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7068bd0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70640714d0 con 0x7f70640cffd0 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f706406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f706406dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50442/0 (socket says 172.21.15.77:50442) 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f7062d9d640 1 -- 172.21.15.77:0/1571859680 learned_addr learned my addr 172.21.15.77:0/1571859680 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.784+0000 7f706259c640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70640aef80 0x7f706414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7061d9b640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70640cffd0 0x7f706406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7062d9d640 1 -- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70640cffd0 msgr2=0x7f706406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7062d9d640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70640cffd0 0x7f706406d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7062d9d640 1 -- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70640aef80 msgr2=0x7f706414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7062d9d640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70640aef80 0x7f706414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7062d9d640 1 -- 172.21.15.77:0/1571859680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064145190 con 0x7f70641586e0 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7061d9b640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70640cffd0 0x7f706406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7062d9d640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f706406dea0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f705400c410 tx=0x7f705400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7052ffd640 1 -- 172.21.15.77:0/1571859680 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054011790 con 0x7f70641586e0 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7052ffd640 1 -- 172.21.15.77:0/1571859680 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054017070 con 0x7f70641586e0 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7064145470 con 0x7f70641586e0 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7052ffd640 1 -- 172.21.15.77:0/1571859680 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054011bb0 con 0x7f70641586e0 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.785+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7064145a30 con 0x7f70641586e0 2024-03-06T07:24:03.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.786+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70640732b0 con 0x7f70641586e0 2024-03-06T07:24:03.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.786+0000 7f7052ffd640 1 -- 172.21.15.77:0/1571859680 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f705401c460 con 0x7f70641586e0 2024-03-06T07:24:03.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.787+0000 7f7052ffd640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f704406b4d0 0x7f704406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.787+0000 7f7052ffd640 1 -- 172.21.15.77:0/1571859680 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7054089b30 con 0x7f70641586e0 2024-03-06T07:24:03.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.787+0000 7f706259c640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f704406b4d0 0x7f704406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.787+0000 7f706259c640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f704406b4d0 0x7f704406d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f705c036000 tx=0x7f705c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.791+0000 7f7052ffd640 1 -- 172.21.15.77:0/1571859680 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7054058d40 con 0x7f70641586e0 2024-03-06T07:24:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.967+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7064145c00 con 0x7f70641586e0 2024-03-06T07:24:03.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.970+0000 7f7052ffd640 1 -- 172.21.15.77:0/1571859680 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7054058640 con 0x7f70641586e0 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f704406b4d0 msgr2=0x7f704406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f704406b4d0 0x7f704406d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f705c036000 tx=0x7f705c03c040 comp rx=0 tx=0).stop 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 msgr2=0x7f706406dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f706406dea0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f705400c410 tx=0x7f705400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 shutdown_connections 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f704406b4d0 0x7f704406d990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70641586e0 0x7f706406dea0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70640cffd0 0x7f706406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 --2- 172.21.15.77:0/1571859680 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70640aef80 0x7f706414afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 >> 172.21.15.77:0/1571859680 conn(0x7f706406b190 msgr2=0x7f70640cee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 shutdown_connections 2024-03-06T07:24:03.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:03.972+0000 7f7068bd0640 1 -- 172.21.15.77:0/1571859680 wait complete. 2024-03-06T07:24:03.985 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:24:04.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.083+0000 7f85970ec640 1 Processor -- start 2024-03-06T07:24:04.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.083+0000 7f85970ec640 1 -- start start 2024-03-06T07:24:04.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f85970ec640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85900b3060 0x7f859015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f85970ec640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859015bf30 0x7f859015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f85970ec640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859015f2d0 con 0x7f859013dc10 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f85970ec640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85900cf3f0 con 0x7f85900b3060 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f85970ec640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859015b210 con 0x7f859015bf30 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f858ffff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85900b3060 0x7f859015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f8595689640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859015bf30 0x7f859015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f858ffff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85900b3060 0x7f859015e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:59742/0 (socket says 172.21.15.77:59742) 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f858ffff640 1 -- 172.21.15.77:0/3108093316 learned_addr learned my addr 172.21.15.77:0/3108093316 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.084+0000 7f8594e88640 1 --1- 172.21.15.77:0/3108093316 >> v1:172.21.15.158:6789/0 conn(0x7f859013dc10 0x7f85900b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:57770/0 (socket says 172.21.15.77:57770) 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.085+0000 7f858ffff640 1 -- 172.21.15.77:0/3108093316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859015bf30 msgr2=0x7f859015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.085+0000 7f858ffff640 1 --2- 172.21.15.77:0/3108093316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859015bf30 0x7f859015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.085+0000 7f858ffff640 1 -- 172.21.15.77:0/3108093316 >> v1:172.21.15.158:6789/0 conn(0x7f859013dc10 legacy=0x7f85900b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.085+0000 7f858ffff640 1 -- 172.21.15.77:0/3108093316 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859015f580 con 0x7f85900b3060 2024-03-06T07:24:04.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.085+0000 7f858f7fe640 1 -- 172.21.15.77:0/3108093316 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352615744 0 0) 0x7f859015f2d0 con 0x7f859013dc10 2024-03-06T07:24:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.085+0000 7f858ffff640 1 --2- 172.21.15.77:0/3108093316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85900b3060 0x7f859015e850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f857c009870 tx=0x7f857c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8619e7e6052d651 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f858f7fe640 1 -- 172.21.15.77:0/3108093316 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c035020 con 0x7f85900b3060 2024-03-06T07:24:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f858f7fe640 1 -- 172.21.15.77:0/3108093316 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857c031640 con 0x7f85900b3060 2024-03-06T07:24:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f858f7fe640 1 -- 172.21.15.77:0/3108093316 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c02d040 con 0x7f85900b3060 2024-03-06T07:24:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f85970ec640 1 -- 172.21.15.77:0/3108093316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85900b3060 msgr2=0x7f859015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f85970ec640 1 --2- 172.21.15.77:0/3108093316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85900b3060 0x7f859015e850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f857c009870 tx=0x7f857c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f85970ec640 1 -- 172.21.15.77:0/3108093316 shutdown_connections 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f85970ec640 1 --2- 172.21.15.77:0/3108093316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859015bf30 0x7f859015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f85970ec640 1 --2- 172.21.15.77:0/3108093316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85900b3060 0x7f859015e850 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f85970ec640 1 -- 172.21.15.77:0/3108093316 >> 172.21.15.77:0/3108093316 conn(0x7f85900cf8a0 msgr2=0x7f85900cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f85970ec640 1 -- 172.21.15.77:0/3108093316 shutdown_connections 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.086+0000 7f85970ec640 1 -- 172.21.15.77:0/3108093316 wait complete. 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.087+0000 7f85970ec640 1 Processor -- start 2024-03-06T07:24:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.087+0000 7f85970ec640 1 -- start start 2024-03-06T07:24:04.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f85970ec640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85900b3060 0x7f859013c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f85970ec640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859013dc10 0x7f859013a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f85970ec640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f859015bf30 0x7f859013a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f85970ec640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859013dff0 con 0x7f85900b3060 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f85970ec640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859015b730 con 0x7f859015bf30 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f85970ec640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85900708d0 con 0x7f859013dc10 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f8594e88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85900b3060 0x7f859013c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f8594e88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85900b3060 0x7f859013c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50448/0 (socket says 172.21.15.77:50448) 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f8594e88640 1 -- 172.21.15.77:0/341819197 learned_addr learned my addr 172.21.15.77:0/341819197 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f8595689640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f859015bf30 0x7f859013a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.088+0000 7f858ffff640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859013dc10 0x7f859013a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f8594e88640 1 -- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859013dc10 msgr2=0x7f859013a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f8594e88640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859013dc10 0x7f859013a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f8594e88640 1 -- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f859015bf30 msgr2=0x7f859013a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f8594e88640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f859015bf30 0x7f859013a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f8594e88640 1 -- 172.21.15.77:0/341819197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85901f4ff0 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f858ffff640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859013dc10 0x7f859013a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f8594e88640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85900b3060 0x7f859013c6e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f858400c500 tx=0x7f858400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f858d7fa640 1 -- 172.21.15.77:0/341819197 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8584018070 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f858d7fa640 1 -- 172.21.15.77:0/341819197 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8584007580 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85901f52d0 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f858d7fa640 1 -- 172.21.15.77:0/341819197 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8584004a20 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.089+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85901f5890 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.090+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859004f330 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.090+0000 7f858d7fa640 1 -- 172.21.15.77:0/341819197 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8584004bc0 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.091+0000 7f858d7fa640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f857406b4d0 0x7f857406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.091+0000 7f858d7fa640 1 -- 172.21.15.77:0/341819197 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8584089210 con 0x7f85900b3060 2024-03-06T07:24:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.091+0000 7f858ffff640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f857406b4d0 0x7f857406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.091+0000 7f858ffff640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f857406b4d0 0x7f857406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f857c036000 tx=0x7f857c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.095+0000 7f858d7fa640 1 -- 172.21.15.77:0/341819197 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85840583a0 con 0x7f85900b3060 2024-03-06T07:24:04.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.333+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f859006eb00 con 0x7f85900b3060 2024-03-06T07:24:04.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.355+0000 7f858d7fa640 1 -- 172.21.15.77:0/341819197 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v129) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8584057ca0 con 0x7f85900b3060 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.356+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f857406b4d0 msgr2=0x7f857406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.356+0000 7f85970ec640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f857406b4d0 0x7f857406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f857c036000 tx=0x7f857c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.356+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85900b3060 msgr2=0x7f859013c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.356+0000 7f85970ec640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85900b3060 0x7f859013c6e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f858400c500 tx=0x7f858400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.357+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 shutdown_connections 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.357+0000 7f85970ec640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f859015bf30 0x7f859013a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.357+0000 7f85970ec640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f859013dc10 0x7f859013a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.357+0000 7f85970ec640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f857406b4d0 0x7f857406d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.357+0000 7f85970ec640 1 --2- 172.21.15.77:0/341819197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85900b3060 0x7f859013c6e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.357+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 >> 172.21.15.77:0/341819197 conn(0x7f85900cf8a0 msgr2=0x7f85900b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.357+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 shutdown_connections 2024-03-06T07:24:04.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.357+0000 7f85970ec640 1 -- 172.21.15.77:0/341819197 wait complete. 2024-03-06T07:24:04.369 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:24:04.370 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:24:04.370 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:24:04.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:04.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.468+0000 7f21eb4b5640 1 Processor -- start 2024-03-06T07:24:04.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.469+0000 7f21eb4b5640 1 -- start start 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.469+0000 7f21eb4b5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f21e415bf30 0x7f21e415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.469+0000 7f21eb4b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e415f580 con 0x7f21e413dc10 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.469+0000 7f21eb4b5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e415b450 con 0x7f21e40b3060 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.469+0000 7f21eb4b5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e40cf3f0 con 0x7f21e415bf30 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.469+0000 7f21e9251640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f21e413dc10 0x7f21e40b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45560/0 (socket says 172.21.15.77:45560) 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9251640 1 -- 172.21.15.77:0/579547755 learned_addr learned my addr 172.21.15.77:0/579547755 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9a52640 1 --2- 172.21.15.77:0/579547755 >> v2:172.21.15.158:3300/0 conn(0x7f21e415bf30 0x7f21e415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554407370 0 0) 0x7f21e415f580 con 0x7f21e413dc10 2024-03-06T07:24:04.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d0003520 con 0x7f21e413dc10 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1014132718 0 0) 0x7f21d0003520 con 0x7f21e413dc10 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 >> v2:172.21.15.158:3300/0 conn(0x7f21e415bf30 msgr2=0x7f21e415ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 --2- 172.21.15.77:0/579547755 >> v2:172.21.15.158:3300/0 conn(0x7f21e415bf30 0x7f21e415ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 >> v1:172.21.15.158:6789/0 conn(0x7f21e40b3060 legacy=0x7f21e415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e415b730 con 0x7f21e413dc10 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f21e00019f0 con 0x7f21e413dc10 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 >> v1:172.21.15.77:6789/0 conn(0x7f21e413dc10 legacy=0x7f21e40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d00039b0 0x7f21d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d0004800 0x7f21d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d00051c0 0x7f21d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e40cf3f0 con 0x7f21d00039b0 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d000c160 con 0x7f21d00051c0 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d000c360 con 0x7f21d0004800 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9251640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d00039b0 0x7f21d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e8a50640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d0004800 0x7f21d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9a52640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d00051c0 0x7f21d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9251640 1 -- 172.21.15.77:0/579547755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d0004800 msgr2=0x7f21d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9251640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d0004800 0x7f21d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9251640 1 -- 172.21.15.77:0/579547755 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d00051c0 msgr2=0x7f21d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9251640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d00051c0 0x7f21d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9251640 1 -- 172.21.15.77:0/579547755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0003520 con 0x7f21d00039b0 2024-03-06T07:24:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.470+0000 7f21e9a52640 1 -- 172.21.15.77:0/579547755 reap_dead start 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.471+0000 7f21e9251640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d00039b0 0x7f21d0003dd0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f21e000a840 tx=0x7f21d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d5a795108719d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.471+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000a460 con 0x7f21d00039b0 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.471+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0034650 con 0x7f21d00039b0 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.471+0000 7f21dbfff640 1 -- 172.21.15.77:0/579547755 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0034a80 con 0x7f21d00039b0 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.472+0000 7f21eb4b5640 1 -- 172.21.15.77:0/579547755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d00039b0 msgr2=0x7f21d0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.472+0000 7f21eb4b5640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d00039b0 0x7f21d0003dd0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f21e000a840 tx=0x7f21d0005830 comp rx=0 tx=0).stop 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.472+0000 7f21eb4b5640 1 -- 172.21.15.77:0/579547755 shutdown_connections 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.472+0000 7f21eb4b5640 1 --2- 172.21.15.77:0/579547755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d00039b0 0x7f21d0003dd0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.472+0000 7f21eb4b5640 1 -- 172.21.15.77:0/579547755 >> 172.21.15.77:0/579547755 conn(0x7f21e40cf8a0 msgr2=0x7f21e40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.472+0000 7f21eb4b5640 1 -- 172.21.15.77:0/579547755 shutdown_connections 2024-03-06T07:24:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.472+0000 7f21eb4b5640 1 -- 172.21.15.77:0/579547755 wait complete. 2024-03-06T07:24:04.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.473+0000 7f21eb4b5640 1 Processor -- start 2024-03-06T07:24:04.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.473+0000 7f21eb4b5640 1 -- start start 2024-03-06T07:24:04.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21eb4b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d00039b0 0x7f21e406ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21eb4b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21e41501a0 0x7f21e4135dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21eb4b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21e4136310 0x7f21e413f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21eb4b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e40b2950 con 0x7f21e4136310 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21eb4b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e415b210 con 0x7f21e41501a0 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21eb4b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e40b2ad0 con 0x7f21d00039b0 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21e4136310 0x7f21e413f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21e4136310 0x7f21e413f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50460/0 (socket says 172.21.15.77:50460) 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 -- 172.21.15.77:0/3458632475 learned_addr learned my addr 172.21.15.77:0/3458632475 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e8a50640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21e41501a0 0x7f21e4135dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9251640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d00039b0 0x7f21e406ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 -- 172.21.15.77:0/3458632475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d00039b0 msgr2=0x7f21e406ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d00039b0 0x7f21e406ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 -- 172.21.15.77:0/3458632475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21e41501a0 msgr2=0x7f21e4135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21e41501a0 0x7f21e4135dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 -- 172.21.15.77:0/3458632475 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e4144710 con 0x7f21e4136310 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.474+0000 7f21e9a52640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21e4136310 0x7f21e413f540 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f21dc011490 tx=0x7f21dc011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.475+0000 7f21d9ffb640 1 -- 172.21.15.77:0/3458632475 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc019020 con 0x7f21e4136310 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.475+0000 7f21d9ffb640 1 -- 172.21.15.77:0/3458632475 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc004460 con 0x7f21e4136310 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.475+0000 7f21d9ffb640 1 -- 172.21.15.77:0/3458632475 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc003850 con 0x7f21e4136310 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.475+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e4144150 con 0x7f21e4136310 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.475+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e413ed40 con 0x7f21e4136310 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.476+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e4137920 con 0x7f21e4136310 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.476+0000 7f21d9ffb640 1 -- 172.21.15.77:0/3458632475 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21dc01c020 con 0x7f21e4136310 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.477+0000 7f21d9ffb640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c806b4d0 0x7f21c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.477+0000 7f21d9ffb640 1 -- 172.21.15.77:0/3458632475 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f21dc08a1b0 con 0x7f21e4136310 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.477+0000 7f21e9251640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c806b4d0 0x7f21c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.477+0000 7f21e9251640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c806b4d0 0x7f21c806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f21e0034000 tx=0x7f21e0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.481+0000 7f21d9ffb640 1 -- 172.21.15.77:0/3458632475 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21e4137920 con 0x7f21e4136310 2024-03-06T07:24:04.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.655+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e404f330 con 0x7f21e4136310 2024-03-06T07:24:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.655+0000 7f21d9ffb640 1 -- 172.21.15.77:0/3458632475 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13008 (secure 0 0 0) 0x7f21dc023020 con 0x7f21e4136310 2024-03-06T07:24:04.657 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:04.657 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":50,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:03.350237+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/2240574726":"2024-03-07T07:24:00.342700+0000","172.21.15.77:6835/2240574726":"2024-03-07T07:24:00.342700+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c806b4d0 msgr2=0x7f21c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c806b4d0 0x7f21c806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f21e0034000 tx=0x7f21e0009030 comp rx=0 tx=0).stop 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21e4136310 msgr2=0x7f21e413f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21e4136310 0x7f21e413f540 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f21dc011490 tx=0x7f21dc011950 comp rx=0 tx=0).stop 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 shutdown_connections 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c806b4d0 0x7f21c806d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21e4136310 0x7f21e413f540 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21e41501a0 0x7f21e4135dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 --2- 172.21.15.77:0/3458632475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d00039b0 0x7f21e406ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.657+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 >> 172.21.15.77:0/3458632475 conn(0x7f21e40cf8a0 msgr2=0x7f21e4159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.658+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 shutdown_connections 2024-03-06T07:24:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.658+0000 7f21eb4b5640 1 -- 172.21.15.77:0/3458632475 wait complete. 2024-03-06T07:24:04.670 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:04.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:04.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.771+0000 7f61a8034640 1 Processor -- start 2024-03-06T07:24:04.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.771+0000 7f61a8034640 1 -- start start 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a8034640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a015a220 0x7f61a00cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a8034640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61a006da00 0x7f61a0071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a8034640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a00b3320 con 0x7f61a00712d0 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a8034640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a00cf610 con 0x7f61a006da00 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a8034640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a0071c70 con 0x7f61a015a220 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a65d1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61a006da00 0x7f61a0071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a5dd0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a015a220 0x7f61a00cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a65d1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61a006da00 0x7f61a0071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:59776/0 (socket says 172.21.15.77:59776) 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a55cf640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f61a00712d0 0x7f61a015de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:57796/0 (socket says 172.21.15.77:57796) 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.772+0000 7f61a65d1640 1 -- 172.21.15.77:0/2799934764 learned_addr learned my addr 172.21.15.77:0/2799934764 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.773+0000 7f61a5dd0640 1 -- 172.21.15.77:0/2799934764 >> v2:172.21.15.158:3300/0 conn(0x7f61a006da00 msgr2=0x7f61a0071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.773+0000 7f61a5dd0640 1 --2- 172.21.15.77:0/2799934764 >> v2:172.21.15.158:3300/0 conn(0x7f61a006da00 0x7f61a0071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.773+0000 7f61a5dd0640 1 -- 172.21.15.77:0/2799934764 >> v1:172.21.15.158:6789/0 conn(0x7f61a00712d0 legacy=0x7f61a015de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.773+0000 7f61a5dd0640 1 -- 172.21.15.77:0/2799934764 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a00728d0 con 0x7f61a015a220 2024-03-06T07:24:04.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.773+0000 7f61a65d1640 1 --2- 172.21.15.77:0/2799934764 >> v2:172.21.15.158:3300/0 conn(0x7f61a006da00 0x7f61a0071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.773+0000 7f61a5dd0640 1 --2- 172.21.15.77:0/2799934764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a015a220 0x7f61a00cf0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f619c009d90 tx=0x7f619c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f8e9ef47ab8dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a4dce640 1 -- 172.21.15.77:0/2799934764 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c039070 con 0x7f61a015a220 2024-03-06T07:24:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a4dce640 1 -- 172.21.15.77:0/2799934764 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c02cbc0 con 0x7f61a015a220 2024-03-06T07:24:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a4dce640 1 -- 172.21.15.77:0/2799934764 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c034440 con 0x7f61a015a220 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a8034640 1 -- 172.21.15.77:0/2799934764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a015a220 msgr2=0x7f61a00cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a8034640 1 --2- 172.21.15.77:0/2799934764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a015a220 0x7f61a00cf0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f619c009d90 tx=0x7f619c02be10 comp rx=0 tx=0).stop 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a8034640 1 -- 172.21.15.77:0/2799934764 shutdown_connections 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a8034640 1 --2- 172.21.15.77:0/2799934764 >> v2:172.21.15.158:3300/0 conn(0x7f61a006da00 0x7f61a0071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a8034640 1 --2- 172.21.15.77:0/2799934764 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a015a220 0x7f61a00cf0d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a8034640 1 -- 172.21.15.77:0/2799934764 >> 172.21.15.77:0/2799934764 conn(0x7f61a006b190 msgr2=0x7f61a006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a8034640 1 -- 172.21.15.77:0/2799934764 shutdown_connections 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.774+0000 7f61a8034640 1 -- 172.21.15.77:0/2799934764 wait complete. 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.775+0000 7f61a8034640 1 Processor -- start 2024-03-06T07:24:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.775+0000 7f61a8034640 1 -- start start 2024-03-06T07:24:04.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a8034640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a006da00 0x7f61a014b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a8034640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61a00712d0 0x7f61a0149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a8034640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a015a220 0x7f61a014a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a8034640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a0072030 con 0x7f61a015a220 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a8034640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a0072bb0 con 0x7f61a00712d0 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a8034640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a0072530 con 0x7f61a006da00 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a65d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a015a220 0x7f61a014a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a65d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a015a220 0x7f61a014a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50468/0 (socket says 172.21.15.77:50468) 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a65d1640 1 -- 172.21.15.77:0/355700136 learned_addr learned my addr 172.21.15.77:0/355700136 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a5dd0640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a006da00 0x7f61a014b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.776+0000 7f61a55cf640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61a00712d0 0x7f61a0149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61a65d1640 1 -- 172.21.15.77:0/355700136 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a006da00 msgr2=0x7f61a014b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61a65d1640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a006da00 0x7f61a014b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61a65d1640 1 -- 172.21.15.77:0/355700136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61a00712d0 msgr2=0x7f61a0149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61a65d1640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61a00712d0 0x7f61a0149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61a65d1640 1 -- 172.21.15.77:0/355700136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a01477f0 con 0x7f61a015a220 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61a65d1640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a015a220 0x7f61a014a410 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f619800e980 tx=0x7f619800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61967fc640 1 -- 172.21.15.77:0/355700136 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198017070 con 0x7f61a015a220 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61967fc640 1 -- 172.21.15.77:0/355700136 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619800ca00 con 0x7f61a015a220 2024-03-06T07:24:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a0147ad0 con 0x7f61a015a220 2024-03-06T07:24:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61967fc640 1 -- 172.21.15.77:0/355700136 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619800ce10 con 0x7f61a015a220 2024-03-06T07:24:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.777+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a0148090 con 0x7f61a015a220 2024-03-06T07:24:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.778+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a00732b0 con 0x7f61a015a220 2024-03-06T07:24:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.778+0000 7f61967fc640 1 -- 172.21.15.77:0/355700136 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f619801a070 con 0x7f61a015a220 2024-03-06T07:24:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.779+0000 7f61967fc640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f618406b4d0 0x7f618406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.779+0000 7f61967fc640 1 -- 172.21.15.77:0/355700136 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f619808a080 con 0x7f61a015a220 2024-03-06T07:24:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.779+0000 7f61a5dd0640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f618406b4d0 0x7f618406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.779+0000 7f61a5dd0640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f618406b4d0 0x7f618406d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f619c036000 tx=0x7f619c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.783+0000 7f61967fc640 1 -- 172.21.15.77:0/355700136 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6198058c40 con 0x7f61a015a220 2024-03-06T07:24:04.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.952+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61a00ae7f0 con 0x7f61a015a220 2024-03-06T07:24:04.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.953+0000 7f61967fc640 1 -- 172.21.15.77:0/355700136 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f619805cce0 con 0x7f61a015a220 2024-03-06T07:24:04.955 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f618406b4d0 msgr2=0x7f618406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f618406b4d0 0x7f618406d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f619c036000 tx=0x7f619c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a015a220 msgr2=0x7f61a014a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a015a220 0x7f61a014a410 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f619800e980 tx=0x7f619800ee40 comp rx=0 tx=0).stop 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 shutdown_connections 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f618406b4d0 0x7f618406d990 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a015a220 0x7f61a014a410 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61a00712d0 0x7f61a0149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 --2- 172.21.15.77:0/355700136 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61a006da00 0x7f61a014b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.954+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 >> 172.21.15.77:0/355700136 conn(0x7f61a006b190 msgr2=0x7f61a00cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.955+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 shutdown_connections 2024-03-06T07:24:04.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:04.955+0000 7f61a8034640 1 -- 172.21.15.77:0/355700136 wait complete. 2024-03-06T07:24:04.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:24:05.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.065+0000 7fda3c2a4640 1 Processor -- start 2024-03-06T07:24:05.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.065+0000 7fda3c2a4640 1 -- start start 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.066+0000 7fda3c2a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 0x7fda340cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.066+0000 7fda3c2a4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fda340d29c0 0x7fda340b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.066+0000 7fda3c2a4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda340b3430 con 0x7fda340cf830 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.066+0000 7fda3c2a4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda340b31e0 con 0x7fda340d29c0 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.066+0000 7fda3c2a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda340b3d60 con 0x7fda3415dfe0 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.066+0000 7fda3a841640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fda340d29c0 0x7fda340b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.066+0000 7fda3a841640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fda340d29c0 0x7fda340b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50482/0 (socket says 172.21.15.77:50482) 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.066+0000 7fda3a841640 1 -- 172.21.15.77:0/3608862026 learned_addr learned my addr 172.21.15.77:0/3608862026 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:05.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.067+0000 7fda3a040640 1 --2- 172.21.15.77:0/3608862026 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 0x7fda340cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.067+0000 7fda3a841640 1 -- 172.21.15.77:0/3608862026 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 msgr2=0x7fda340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.067+0000 7fda3a841640 1 --2- 172.21.15.77:0/3608862026 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 0x7fda340cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.067+0000 7fda3a841640 1 -- 172.21.15.77:0/3608862026 >> v1:172.21.15.158:6789/0 conn(0x7fda340cf830 legacy=0x7fda340d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.067+0000 7fda3a841640 1 -- 172.21.15.77:0/3608862026 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda340b3710 con 0x7fda340d29c0 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.067+0000 7fda3a841640 1 --2- 172.21.15.77:0/3608862026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340b41f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009870 tx=0x7fda2c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894ea4826eecad9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3903e640 1 -- 172.21.15.77:0/3608862026 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c034070 con 0x7fda340d29c0 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3903e640 1 -- 172.21.15.77:0/3608862026 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c030840 con 0x7fda340d29c0 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3903e640 1 -- 172.21.15.77:0/3608862026 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c030c10 con 0x7fda340d29c0 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3c2a4640 1 -- 172.21.15.77:0/3608862026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 msgr2=0x7fda340b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/3608862026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340b41f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009870 tx=0x7fda2c02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3c2a4640 1 -- 172.21.15.77:0/3608862026 shutdown_connections 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/3608862026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340b41f0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/3608862026 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 0x7fda340cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3c2a4640 1 -- 172.21.15.77:0/3608862026 >> 172.21.15.77:0/3608862026 conn(0x7fda340d3cd0 msgr2=0x7fda340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3c2a4640 1 -- 172.21.15.77:0/3608862026 shutdown_connections 2024-03-06T07:24:05.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.068+0000 7fda3c2a4640 1 -- 172.21.15.77:0/3608862026 wait complete. 2024-03-06T07:24:05.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.069+0000 7fda3c2a4640 1 Processor -- start 2024-03-06T07:24:05.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.069+0000 7fda3c2a4640 1 -- start start 2024-03-06T07:24:05.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3c2a4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda340cf830 0x7fda340b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3c2a4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3c2a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 0x7fda340cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3c2a4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda340d1df0 con 0x7fda340d29c0 2024-03-06T07:24:05.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3c2a4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda340d1c70 con 0x7fda340cf830 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3c2a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3415e3c0 con 0x7fda3415dfe0 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3983f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3983f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50498/0 (socket says 172.21.15.77:50498) 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3983f640 1 -- 172.21.15.77:0/1432284845 learned_addr learned my addr 172.21.15.77:0/1432284845 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3983f640 1 -- 172.21.15.77:0/1432284845 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 msgr2=0x7fda340cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3983f640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 0x7fda340cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3983f640 1 -- 172.21.15.77:0/1432284845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda340cf830 msgr2=0x7fda340b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3983f640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda340cf830 0x7fda340b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.070+0000 7fda3983f640 1 -- 172.21.15.77:0/1432284845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda340b7100 con 0x7fda340d29c0 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.071+0000 7fda3983f640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340cc4a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c470 tx=0x7fda24001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.071+0000 7fda2affd640 1 -- 172.21.15.77:0/1432284845 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda240052d0 con 0x7fda340d29c0 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.071+0000 7fda2affd640 1 -- 172.21.15.77:0/1432284845 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda24005470 con 0x7fda340d29c0 2024-03-06T07:24:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.071+0000 7fda2affd640 1 -- 172.21.15.77:0/1432284845 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24007260 con 0x7fda340d29c0 2024-03-06T07:24:05.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.071+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda340b73e0 con 0x7fda340d29c0 2024-03-06T07:24:05.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.071+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda340b79a0 con 0x7fda340d29c0 2024-03-06T07:24:05.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.072+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3404f330 con 0x7fda340d29c0 2024-03-06T07:24:05.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.072+0000 7fda2affd640 1 -- 172.21.15.77:0/1432284845 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda24007400 con 0x7fda340d29c0 2024-03-06T07:24:05.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.072+0000 7fda2affd640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.072+0000 7fda2affd640 1 -- 172.21.15.77:0/1432284845 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fda240893a0 con 0x7fda340d29c0 2024-03-06T07:24:05.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.072+0000 7fda3a040640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.073+0000 7fda3a040640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b4d0 0x7fda1806d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda300070b0 tx=0x7fda30007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.077+0000 7fda2affd640 1 -- 172.21.15.77:0/1432284845 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda24058520 con 0x7fda340d29c0 2024-03-06T07:24:05.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.315+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fda3406c7a0 con 0x7fda340d29c0 2024-03-06T07:24:05.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.356+0000 7fda2affd640 1 -- 172.21.15.77:0/1432284845 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v130) v1 ==== 147+0+0 (secure 0 0 0) 0x7fda24057e20 con 0x7fda340d29c0 2024-03-06T07:24:05.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b4d0 msgr2=0x7fda1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b4d0 0x7fda1806d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda300070b0 tx=0x7fda30007040 comp rx=0 tx=0).stop 2024-03-06T07:24:05.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 msgr2=0x7fda340cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340cc4a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c470 tx=0x7fda24001e70 comp rx=0 tx=0).stop 2024-03-06T07:24:05.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 shutdown_connections 2024-03-06T07:24:05.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda340d29c0 0x7fda340cc4a0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda3415dfe0 0x7fda340cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 --2- 172.21.15.77:0/1432284845 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda340cf830 0x7fda340b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 >> 172.21.15.77:0/1432284845 conn(0x7fda340d3cd0 msgr2=0x7fda3406ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 shutdown_connections 2024-03-06T07:24:05.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.359+0000 7fda3c2a4640 1 -- 172.21.15.77:0/1432284845 wait complete. 2024-03-06T07:24:05.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:05.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.474+0000 7fea44afb640 1 Processor -- start 2024-03-06T07:24:05.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.474+0000 7fea44afb640 1 -- start start 2024-03-06T07:24:05.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.474+0000 7fea44afb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fea400d4860 0x7fea400d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.474+0000 7fea44afb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea400b0e30 0x7fea400b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.474+0000 7fea44afb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea401618b0 con 0x7fea401614d0 2024-03-06T07:24:05.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.474+0000 7fea44afb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea400d4cf0 con 0x7fea400d4860 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.474+0000 7fea44afb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea400cf400 con 0x7fea400b0e30 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3e59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fea401614d0 0x7fea400d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45566/0 (socket says 172.21.15.77:45566) 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3e59c640 1 -- 172.21.15.77:0/2650931041 learned_addr learned my addr 172.21.15.77:0/2650931041 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea37fff640 1 --2- 172.21.15.77:0/2650931041 >> v2:172.21.15.158:3300/0 conn(0x7fea400d4860 0x7fea400d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3ed9d640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea400b0e30 0x7fea400b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46365729 0 0) 0x7fea401618b0 con 0x7fea401614d0 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c003520 con 0x7fea401614d0 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885649188 0 0) 0x7fea2c003520 con 0x7fea401614d0 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea400b0e30 msgr2=0x7fea400b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea400b0e30 0x7fea400b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 >> v2:172.21.15.158:3300/0 conn(0x7fea400d4860 msgr2=0x7fea400d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 --2- 172.21.15.77:0/2650931041 >> v2:172.21.15.158:3300/0 conn(0x7fea400d4860 0x7fea400d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea400cf6a0 con 0x7fea401614d0 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fea30001670 con 0x7fea401614d0 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 >> v1:172.21.15.77:6789/0 conn(0x7fea401614d0 legacy=0x7fea400d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea400b0e30 0x7fea2c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea2c003f30 0x7fea2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea2c004d40 0x7fea2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400d4cf0 con 0x7fea2c004d40 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400cf400 con 0x7fea2c003f30 2024-03-06T07:24:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c00b9c0 con 0x7fea400b0e30 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3ed9d640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea2c004d40 0x7fea2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3e59c640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea400b0e30 0x7fea2c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea37fff640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea2c003f30 0x7fea2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3ed9d640 1 -- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea400b0e30 msgr2=0x7fea2c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3ed9d640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea400b0e30 0x7fea2c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3ed9d640 1 -- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea2c003f30 msgr2=0x7fea2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3ed9d640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea2c003f30 0x7fea2c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea3ed9d640 1 -- 172.21.15.77:0/2650931041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c003520 con 0x7fea2c004d40 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.475+0000 7fea37fff640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea2c003f30 0x7fea2c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.476+0000 7fea3ed9d640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea2c004d40 0x7fea2c0051c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fea38009fc0 tx=0x7fea2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be08860ecb133d74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.476+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38039070 con 0x7fea2c004d40 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.476+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3802cc00 con 0x7fea2c004d40 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.476+0000 7fea3dd9b640 1 -- 172.21.15.77:0/2650931041 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38009c10 con 0x7fea2c004d40 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.477+0000 7fea44afb640 1 -- 172.21.15.77:0/2650931041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea2c004d40 msgr2=0x7fea2c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.477+0000 7fea44afb640 1 --2- 172.21.15.77:0/2650931041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea2c004d40 0x7fea2c0051c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fea38009fc0 tx=0x7fea2c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.477+0000 7fea3ed9d640 1 -- 172.21.15.77:0/2650931041 reap_dead start 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.477+0000 7fea44afb640 1 -- 172.21.15.77:0/2650931041 shutdown_connections 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.477+0000 7fea44afb640 1 -- 172.21.15.77:0/2650931041 >> 172.21.15.77:0/2650931041 conn(0x7fea400d3d40 msgr2=0x7fea400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.477+0000 7fea44afb640 1 -- 172.21.15.77:0/2650931041 shutdown_connections 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.477+0000 7fea44afb640 1 -- 172.21.15.77:0/2650931041 wait complete. 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.478+0000 7fea44afb640 1 Processor -- start 2024-03-06T07:24:05.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.478+0000 7fea44afb640 1 -- start start 2024-03-06T07:24:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea44afb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea40157760 0x7fea400ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea44afb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea400c7870 0x7fea400c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea44afb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea4015fad0 0x7fea4015ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea44afb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400b0910 con 0x7fea40157760 2024-03-06T07:24:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea44afb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400cfc60 con 0x7fea400c7870 2024-03-06T07:24:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea44afb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea400d2560 con 0x7fea4015fad0 2024-03-06T07:24:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea3e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea40157760 0x7fea400ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea3e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea40157760 0x7fea400ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50524/0 (socket says 172.21.15.77:50524) 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea3e59c640 1 -- 172.21.15.77:0/3952248658 learned_addr learned my addr 172.21.15.77:0/3952248658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea3e59c640 1 -- 172.21.15.77:0/3952248658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea4015fad0 msgr2=0x7fea4015ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea3e59c640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea4015fad0 0x7fea4015ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea3e59c640 1 -- 172.21.15.77:0/3952248658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea400c7870 msgr2=0x7fea400c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea3e59c640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea400c7870 0x7fea400c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.479+0000 7fea3e59c640 1 -- 172.21.15.77:0/3952248658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea400c8310 con 0x7fea40157760 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.480+0000 7fea3e59c640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea40157760 0x7fea400ca3d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fea3000eb20 tx=0x7fea30005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.480+0000 7fea36ffd640 1 -- 172.21.15.77:0/3952248658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000c7f0 con 0x7fea40157760 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.480+0000 7fea36ffd640 1 -- 172.21.15.77:0/3952248658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30018070 con 0x7fea40157760 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.480+0000 7fea36ffd640 1 -- 172.21.15.77:0/3952248658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000cb90 con 0x7fea40157760 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.480+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea400c85f0 con 0x7fea40157760 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.480+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea400c9bd0 con 0x7fea40157760 2024-03-06T07:24:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.481+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea40160490 con 0x7fea40157760 2024-03-06T07:24:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.481+0000 7fea36ffd640 1 -- 172.21.15.77:0/3952248658 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea30013420 con 0x7fea40157760 2024-03-06T07:24:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.482+0000 7fea36ffd640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea2006b500 0x7fea2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.482+0000 7fea36ffd640 1 -- 172.21.15.77:0/3952248658 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fea30089b70 con 0x7fea40157760 2024-03-06T07:24:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.482+0000 7fea37fff640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea2006b500 0x7fea2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.482+0000 7fea37fff640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea2006b500 0x7fea2006d9c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fea4015b3b0 tx=0x7fea28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.486+0000 7fea36ffd640 1 -- 172.21.15.77:0/3952248658 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea30058cf0 con 0x7fea40157760 2024-03-06T07:24:05.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.658+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fea40068ff0 con 0x7fea40157760 2024-03-06T07:24:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.661+0000 7fea36ffd640 1 -- 172.21.15.77:0/3952248658 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fea300585f0 con 0x7fea40157760 2024-03-06T07:24:05.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea2006b500 msgr2=0x7fea2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea2006b500 0x7fea2006d9c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fea4015b3b0 tx=0x7fea28008040 comp rx=0 tx=0).stop 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea40157760 msgr2=0x7fea400ca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea40157760 0x7fea400ca3d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fea3000eb20 tx=0x7fea30005e10 comp rx=0 tx=0).stop 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 shutdown_connections 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea4015fad0 0x7fea4015ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea400c7870 0x7fea400c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea2006b500 0x7fea2006d9c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 --2- 172.21.15.77:0/3952248658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea40157760 0x7fea400ca3d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 >> 172.21.15.77:0/3952248658 conn(0x7fea400d3d40 msgr2=0x7fea4006c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 shutdown_connections 2024-03-06T07:24:05.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.663+0000 7fea44afb640 1 -- 172.21.15.77:0/3952248658 wait complete. 2024-03-06T07:24:05.679 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:05.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:05.720 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:05.778 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:05.779 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:05.779 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.794 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:05.794 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:24:05.794 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.812 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:05.812 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:05.813 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.849 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:05.850 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:24:05.850 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:05.869 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:05.869 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:24:05.870 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:05.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.987+0000 7f0ee0765640 1 Processor -- start 2024-03-06T07:24:05.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.987+0000 7f0ee0765640 1 -- start start 2024-03-06T07:24:05.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.988+0000 7f0ee0765640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0ed815dfe0 0x7f0ed80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.988+0000 7f0ee0765640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ed80cf830 0x7f0ed80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.988+0000 7f0ee0765640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed80b3430 con 0x7f0ed80d27f0 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.988+0000 7f0ee0765640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed80b31e0 con 0x7f0ed815dfe0 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.988+0000 7f0ee0765640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed80b3d60 con 0x7f0ed80cf830 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.988+0000 7f0ede501640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0ed815dfe0 0x7f0ed80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.988+0000 7f0eded02640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0ed80d27f0 0x7f0ed80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45568/0 (socket says 172.21.15.77:45568) 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.988+0000 7f0eded02640 1 -- 172.21.15.77:0/1180029038 learned_addr learned my addr 172.21.15.77:0/1180029038 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.989+0000 7f0ede501640 1 -- 172.21.15.77:0/1180029038 >> v2:172.21.15.158:3300/0 conn(0x7f0ed80cf830 msgr2=0x7f0ed80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.989+0000 7f0ede501640 1 --2- 172.21.15.77:0/1180029038 >> v2:172.21.15.158:3300/0 conn(0x7f0ed80cf830 0x7f0ed80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.989+0000 7f0ede501640 1 -- 172.21.15.77:0/1180029038 >> v1:172.21.15.77:6789/0 conn(0x7f0ed80d27f0 legacy=0x7f0ed80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.989+0000 7f0ede501640 1 -- 172.21.15.77:0/1180029038 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed80b3710 con 0x7f0ed815dfe0 2024-03-06T07:24:05.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.989+0000 7f0edd4ff640 1 -- 172.21.15.77:0/1180029038 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272731423 0 0) 0x7f0ed80b3430 con 0x7f0ed80d27f0 2024-03-06T07:24:05.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.989+0000 7f0ede501640 1 --2- 172.21.15.77:0/1180029038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed80cf2f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4009d90 tx=0x7f0ed402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a01e9fa5f94c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0edd4ff640 1 -- 172.21.15.77:0/1180029038 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4039070 con 0x7f0ed815dfe0 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0edd4ff640 1 -- 172.21.15.77:0/1180029038 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed402cc60 con 0x7f0ed815dfe0 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0edd4ff640 1 -- 172.21.15.77:0/1180029038 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4034660 con 0x7f0ed815dfe0 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0ee0765640 1 -- 172.21.15.77:0/1180029038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 msgr2=0x7f0ed80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0ee0765640 1 --2- 172.21.15.77:0/1180029038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed80cf2f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4009d90 tx=0x7f0ed402beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0ee0765640 1 -- 172.21.15.77:0/1180029038 shutdown_connections 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0ee0765640 1 --2- 172.21.15.77:0/1180029038 >> v2:172.21.15.158:3300/0 conn(0x7f0ed80cf830 0x7f0ed80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0ee0765640 1 --2- 172.21.15.77:0/1180029038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed80cf2f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0ee0765640 1 -- 172.21.15.77:0/1180029038 >> 172.21.15.77:0/1180029038 conn(0x7f0ed80d3cd0 msgr2=0x7f0ed80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0ee0765640 1 -- 172.21.15.77:0/1180029038 shutdown_connections 2024-03-06T07:24:05.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.990+0000 7f0ee0765640 1 -- 172.21.15.77:0/1180029038 wait complete. 2024-03-06T07:24:05.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.991+0000 7f0ee0765640 1 Processor -- start 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.991+0000 7f0ee0765640 1 -- start start 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0ee0765640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ed80cf830 0x7f0ed81f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0ee0765640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed80d27f0 0x7f0ed815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0ee0765640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0ee0765640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed80d1df0 con 0x7f0ed815dfe0 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0ee0765640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed80d1c70 con 0x7f0ed80d27f0 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0ee0765640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed815e3c0 con 0x7f0ed80cf830 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0eded02640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0eded02640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed815bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50542/0 (socket says 172.21.15.77:50542) 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0eded02640 1 -- 172.21.15.77:0/2508793164 learned_addr learned my addr 172.21.15.77:0/2508793164 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0eddd00640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed80d27f0 0x7f0ed815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.992+0000 7f0ede501640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ed80cf830 0x7f0ed81f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eded02640 1 -- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ed80cf830 msgr2=0x7f0ed81f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eded02640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ed80cf830 0x7f0ed81f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eded02640 1 -- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed80d27f0 msgr2=0x7f0ed815e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eded02640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed80d27f0 0x7f0ed815e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eded02640 1 -- 172.21.15.77:0/2508793164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed8160ce0 con 0x7f0ed815dfe0 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eddd00640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed80d27f0 0x7f0ed815e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eded02640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed815bb70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000c450 tx=0x7f0ed000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eceffd640 1 -- 172.21.15.77:0/2508793164 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0018070 con 0x7f0ed815dfe0 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eceffd640 1 -- 172.21.15.77:0/2508793164 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed0007500 con 0x7f0ed815dfe0 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed8160fc0 con 0x7f0ed815dfe0 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0eceffd640 1 -- 172.21.15.77:0/2508793164 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0004a20 con 0x7f0ed815dfe0 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.993+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed815ec30 con 0x7f0ed815dfe0 2024-03-06T07:24:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.994+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed804f330 con 0x7f0ed815dfe0 2024-03-06T07:24:05.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.994+0000 7f0eceffd640 1 -- 172.21.15.77:0/2508793164 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ed001b070 con 0x7f0ed815dfe0 2024-03-06T07:24:05.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.995+0000 7f0eceffd640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.995+0000 7f0eceffd640 1 -- 172.21.15.77:0/2508793164 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0ed00891b0 con 0x7f0ed815dfe0 2024-03-06T07:24:05.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.995+0000 7f0ede501640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.995+0000 7f0ede501640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4036000 tx=0x7f0ed4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:05.999+0000 7f0eceffd640 1 -- 172.21.15.77:0/2508793164 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ed00583f0 con 0x7f0ed815dfe0 2024-03-06T07:24:06.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.206+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ed80d3420 con 0x7f0ed815dfe0 2024-03-06T07:24:06.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.207+0000 7f0eceffd640 1 -- 172.21.15.77:0/2508793164 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f0ed0057cf0 con 0x7f0ed815dfe0 2024-03-06T07:24:06.209 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:06.209 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1388591691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1388591691},{"type":"v1","addr":"172.21.15.158:6834","nonce":1388591691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/870048529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":870048529},{"type":"v1","addr":"172.21.15.158:6835","nonce":870048529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/650607050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":650607050},{"type":"v1","addr":"172.21.15.77:6836","nonce":650607050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4187088983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":4187088983},{"type":"v1","addr":"172.21.15.77:6837","nonce":4187088983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[]} 2024-03-06T07:24:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.208+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ebc06b4d0 msgr2=0x7f0ebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.208+0000 7f0ee0765640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4036000 tx=0x7f0ed4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.208+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 msgr2=0x7f0ed815bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.208+0000 7f0ee0765640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed815bb70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000c450 tx=0x7f0ed000c910 comp rx=0 tx=0).stop 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.208+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 shutdown_connections 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.208+0000 7f0ee0765640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.209+0000 7f0ee0765640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed815dfe0 0x7f0ed815bb70 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.209+0000 7f0ee0765640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed80d27f0 0x7f0ed815e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.209+0000 7f0ee0765640 1 --2- 172.21.15.77:0/2508793164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0ed80cf830 0x7f0ed81f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.209+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 >> 172.21.15.77:0/2508793164 conn(0x7f0ed80d3cd0 msgr2=0x7f0ed806bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.209+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 shutdown_connections 2024-03-06T07:24:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.209+0000 7f0ee0765640 1 -- 172.21.15.77:0/2508793164 wait complete. 2024-03-06T07:24:06.222 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:06.222 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:06.222 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:06.222 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:06.222 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:06.222 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:06.222 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:06.223 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:06.223 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:06.223 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:06.223 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:06.223 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:06.223 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:06.223 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:06.224 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:06.224 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:06.231 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:06.232 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:06.232 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:06.233 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:06.235 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:06.235 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:06.236 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:06.236 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:06.274 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:06.274 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:06.275 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:06.275 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:06.278 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:06.278 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:06.279 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:06.279 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:06.280 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:06.350 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:24:06.351 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:24:06.368 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:24:06.368 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:24:06.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.383+0000 7f12ae965640 1 Processor -- start 2024-03-06T07:24:06.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.383+0000 7f12ae965640 1 -- start start 2024-03-06T07:24:06.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.384+0000 7f12ae965640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12a80d2ad0 0x7f12a816dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.384+0000 7f12ae965640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a81677e0 con 0x7f12a80d06f0 2024-03-06T07:24:06.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.384+0000 7f12ae965640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80d4c10 con 0x7f12a80b51d0 2024-03-06T07:24:06.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.384+0000 7f12ae965640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a80b4cb0 con 0x7f12a80d2ad0 2024-03-06T07:24:06.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.384+0000 7f12a7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f12a80d06f0 0x7f12a81669c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45576/0 (socket says 172.21.15.77:45576) 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.384+0000 7f12a7fff640 1 -- 172.21.15.77:0/399753615 learned_addr learned my addr 172.21.15.77:0/399753615 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12acf02640 1 --2- 172.21.15.77:0/399753615 >> v2:172.21.15.158:3300/0 conn(0x7f12a80d2ad0 0x7f12a816dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444752277 0 0) 0x7f12a81677e0 con 0x7f12a80d06f0 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1290003520 con 0x7f12a80d06f0 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1662382342 0 0) 0x7f1290003520 con 0x7f12a80d06f0 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 >> v2:172.21.15.158:3300/0 conn(0x7f12a80d2ad0 msgr2=0x7f12a816dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 --2- 172.21.15.77:0/399753615 >> v2:172.21.15.158:3300/0 conn(0x7f12a80d2ad0 0x7f12a816dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 >> v1:172.21.15.158:6789/0 conn(0x7f12a80b51d0 legacy=0x7f12a81670d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80d4930 con 0x7f12a80d06f0 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f1298001670 con 0x7f12a80d06f0 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 >> v1:172.21.15.77:6789/0 conn(0x7f12a80d06f0 legacy=0x7f12a81669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12900039b0 0x7f1290003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f1290004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1290005120 0x7f129000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80b4cb0 con 0x7f12900039b0 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129000c060 con 0x7f12900043b0 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129000c260 con 0x7f1290005120 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a7fff640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12900039b0 0x7f1290003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a77fe640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f1290004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12acf02640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1290005120 0x7f129000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a77fe640 1 -- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1290005120 msgr2=0x7f129000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a77fe640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1290005120 0x7f129000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a77fe640 1 -- 172.21.15.77:0/399753615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12900039b0 msgr2=0x7f1290003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a77fe640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12900039b0 0x7f1290003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a77fe640 1 -- 172.21.15.77:0/399753615 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1290003520 con 0x7f12900043b0 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12a7fff640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12900039b0 0x7f1290003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.385+0000 7f12acf02640 1 -- 172.21.15.77:0/399753615 reap_dead start 2024-03-06T07:24:06.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.386+0000 7f12a77fe640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f1290004880 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f129c0018e0 tx=0x7f1290005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b24ba598bf9ded4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.386+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c038070 con 0x7f12900043b0 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.386+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c033630 con 0x7f12900043b0 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.386+0000 7f12a6ffd640 1 -- 172.21.15.77:0/399753615 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c02f050 con 0x7f12900043b0 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.387+0000 7f12ae965640 1 -- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 msgr2=0x7f1290004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.387+0000 7f12ae965640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f1290004880 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f129c0018e0 tx=0x7f1290005830 comp rx=0 tx=0).stop 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.387+0000 7f12ae965640 1 -- 172.21.15.77:0/399753615 shutdown_connections 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.387+0000 7f12ae965640 1 --2- 172.21.15.77:0/399753615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f1290004880 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.387+0000 7f12ae965640 1 -- 172.21.15.77:0/399753615 >> 172.21.15.77:0/399753615 conn(0x7f12a80d3ce0 msgr2=0x7f12a80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.387+0000 7f12ae965640 1 -- 172.21.15.77:0/399753615 shutdown_connections 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.387+0000 7f12ae965640 1 -- 172.21.15.77:0/399753615 wait complete. 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.388+0000 7f12ae965640 1 Processor -- start 2024-03-06T07:24:06.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.388+0000 7f12ae965640 1 -- start start 2024-03-06T07:24:06.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12ae965640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f12a81e88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12ae965640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f12a80c8860 0x7f12a81d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12ae965640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a80c9e70 0x7f12a81e5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12ae965640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80cf810 con 0x7f12a80c9e70 2024-03-06T07:24:06.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12ae965640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80c6ff0 con 0x7f12900043b0 2024-03-06T07:24:06.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12ae965640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80c7170 con 0x7f12a80c8860 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12acf02640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a80c9e70 0x7f12a81e5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12acf02640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a80c9e70 0x7f12a81e5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50574/0 (socket says 172.21.15.77:50574) 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12acf02640 1 -- 172.21.15.77:0/1083729870 learned_addr learned my addr 172.21.15.77:0/1083729870 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12a7fff640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f12a81e88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.389+0000 7f12a77fe640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f12a80c8860 0x7f12a81d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12acf02640 1 -- 172.21.15.77:0/1083729870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f12a80c8860 msgr2=0x7f12a81d3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12acf02640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f12a80c8860 0x7f12a81d3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12acf02640 1 -- 172.21.15.77:0/1083729870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 msgr2=0x7f12a81e88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12acf02640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f12a81e88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12acf02640 1 -- 172.21.15.77:0/1083729870 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80c8dd0 con 0x7f12a80c9e70 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12acf02640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a80c9e70 0x7f12a81e5bc0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f12a806b370 tx=0x7f12a000f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12a4ff9640 1 -- 172.21.15.77:0/1083729870 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0003e60 con 0x7f12a80c9e70 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a80c90b0 con 0x7f12a80c9e70 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12a4ff9640 1 -- 172.21.15.77:0/1083729870 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a001c070 con 0x7f12a80c9e70 2024-03-06T07:24:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a80c9670 con 0x7f12a80c9e70 2024-03-06T07:24:06.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.390+0000 7f12a4ff9640 1 -- 172.21.15.77:0/1083729870 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a000fb10 con 0x7f12a80c9e70 2024-03-06T07:24:06.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.391+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a80ce480 con 0x7f12a80c9e70 2024-03-06T07:24:06.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.391+0000 7f12a4ff9640 1 -- 172.21.15.77:0/1083729870 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12a000fcb0 con 0x7f12a80c9e70 2024-03-06T07:24:06.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.392+0000 7f12a4ff9640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.392+0000 7f12a7fff640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.392+0000 7f12a4ff9640 1 -- 172.21.15.77:0/1083729870 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f12a008a810 con 0x7f12a80c9e70 2024-03-06T07:24:06.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.392+0000 7f12a7fff640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f129406b4d0 0x7f129406d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1298001200 tx=0x7f1298007b80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.396+0000 7f12a4ff9640 1 -- 172.21.15.77:0/1083729870 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12a80ce480 con 0x7f12a80c9e70 2024-03-06T07:24:06.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:06.580+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f12a804f330 con 0x7f12a80c9e70 2024-03-06T07:24:07.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.378+0000 7f12a4ff9640 1 -- 172.21.15.77:0/1083729870 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) v1 ==== 97+0+0 (secure 0 0 0) 0x7f12a0010040 con 0x7f12a80c9e70 2024-03-06T07:24:07.380 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:24:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f129406b4d0 msgr2=0x7f129406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f129406b4d0 0x7f129406d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1298001200 tx=0x7f1298007b80 comp rx=0 tx=0).stop 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a80c9e70 msgr2=0x7f12a81e5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a80c9e70 0x7f12a81e5bc0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f12a806b370 tx=0x7f12a000f1b0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 shutdown_connections 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a80c9e70 0x7f12a81e5bc0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f12a80c8860 0x7f12a81d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 --2- 172.21.15.77:0/1083729870 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f12900043b0 0x7f12a81e88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 >> 172.21.15.77:0/1083729870 conn(0x7f12a80d3ce0 msgr2=0x7f12a80cd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.380+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 shutdown_connections 2024-03-06T07:24:07.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.381+0000 7f12ae965640 1 -- 172.21.15.77:0/1083729870 wait complete. 2024-03-06T07:24:07.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:07.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.499+0000 7fca96f1e640 1 Processor -- start 2024-03-06T07:24:07.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.499+0000 7fca96f1e640 1 -- start start 2024-03-06T07:24:07.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca96f1e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013dc10 0x7fca900b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca96f1e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9015f580 con 0x7fca9015ef60 2024-03-06T07:24:07.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca96f1e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9015b450 con 0x7fca9015bf30 2024-03-06T07:24:07.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca96f1e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca900cf3f0 con 0x7fca9013dc10 2024-03-06T07:24:07.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca954bb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fca9015ef60 0x7fca900708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45592/0 (socket says 172.21.15.77:45592) 2024-03-06T07:24:07.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca954bb640 1 -- 172.21.15.77:0/1258156466 learned_addr learned my addr 172.21.15.77:0/1258156466 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:07.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152481261 0 0) 0x7fca9015f580 con 0x7fca9015ef60 2024-03-06T07:24:07.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca78003520 con 0x7fca9015ef60 2024-03-06T07:24:07.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.500+0000 7fca94cba640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013dc10 0x7fca900b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 870230719 0 0) 0x7fca78003520 con 0x7fca9015ef60 2024-03-06T07:24:07.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013dc10 msgr2=0x7fca900b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013dc10 0x7fca900b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 >> v1:172.21.15.158:6789/0 conn(0x7fca9015bf30 legacy=0x7fca9015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9015b730 con 0x7fca9015ef60 2024-03-06T07:24:07.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fca880019f0 con 0x7fca9015ef60 2024-03-06T07:24:07.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 >> v1:172.21.15.77:6789/0 conn(0x7fca9015ef60 legacy=0x7fca900708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013dc10 0x7fca780039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca78004d40 0x7fca780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca900cf3f0 con 0x7fca78003f60 2024-03-06T07:24:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7800dd60 con 0x7fca78004d40 2024-03-06T07:24:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7800df60 con 0x7fca9013dc10 2024-03-06T07:24:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8ffff640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca94cba640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013dc10 0x7fca780039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca954bb640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca78004d40 0x7fca780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8ffff640 1 -- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013dc10 msgr2=0x7fca780039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8ffff640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013dc10 0x7fca780039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8ffff640 1 -- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca78004d40 msgr2=0x7fca780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8ffff640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca78004d40 0x7fca780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.501+0000 7fca8ffff640 1 -- 172.21.15.77:0/1258156466 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca78003520 con 0x7fca78003f60 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca8ffff640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca78004800 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fca84001750 tx=0x7fca78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39d184ccedb32b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca84001370 con 0x7fca78003f60 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca840052d0 con 0x7fca78003f60 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca8f7fe640 1 -- 172.21.15.77:0/1258156466 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca84003830 con 0x7fca78003f60 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca96f1e640 1 -- 172.21.15.77:0/1258156466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 msgr2=0x7fca78004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca96f1e640 1 --2- 172.21.15.77:0/1258156466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca78004800 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fca84001750 tx=0x7fca78005830 comp rx=0 tx=0).stop 2024-03-06T07:24:07.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca954bb640 1 -- 172.21.15.77:0/1258156466 reap_dead start 2024-03-06T07:24:07.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca96f1e640 1 -- 172.21.15.77:0/1258156466 shutdown_connections 2024-03-06T07:24:07.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.502+0000 7fca96f1e640 1 -- 172.21.15.77:0/1258156466 >> 172.21.15.77:0/1258156466 conn(0x7fca900cf8a0 msgr2=0x7fca900cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.503+0000 7fca96f1e640 1 -- 172.21.15.77:0/1258156466 shutdown_connections 2024-03-06T07:24:07.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.503+0000 7fca96f1e640 1 -- 172.21.15.77:0/1258156466 wait complete. 2024-03-06T07:24:07.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.504+0000 7fca96f1e640 1 Processor -- start 2024-03-06T07:24:07.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.504+0000 7fca96f1e640 1 -- start start 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca96f1e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca9013cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca96f1e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013fa70 0x7fca90140aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca96f1e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca90150960 0x7fca90142f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca96f1e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca90071ff0 con 0x7fca78003f60 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca96f1e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9015b210 con 0x7fca90150960 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca96f1e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca90072170 con 0x7fca9013fa70 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca9013cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca9013cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50602/0 (socket says 172.21.15.77:50602) 2024-03-06T07:24:07.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 -- 172.21.15.77:0/1137338563 learned_addr learned my addr 172.21.15.77:0/1137338563 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca8ffff640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013fa70 0x7fca90140aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca954bb640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca90150960 0x7fca90142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 -- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013fa70 msgr2=0x7fca90140aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013fa70 0x7fca90140aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 -- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca90150960 msgr2=0x7fca90142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca90150960 0x7fca90142f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 -- 172.21.15.77:0/1137338563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9013ffa0 con 0x7fca78003f60 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca954bb640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca90150960 0x7fca90142f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca94cba640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca9013cc00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fca800119a0 tx=0x7fca80011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca8d7fa640 1 -- 172.21.15.77:0/1137338563 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8000faf0 con 0x7fca78003f60 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca8d7fa640 1 -- 172.21.15.77:0/1137338563 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8000fc90 con 0x7fca78003f60 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca8d7fa640 1 -- 172.21.15.77:0/1137338563 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80014440 con 0x7fca78003f60 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca9014fe80 con 0x7fca78003f60 2024-03-06T07:24:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.505+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca90150440 con 0x7fca78003f60 2024-03-06T07:24:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.506+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9004f330 con 0x7fca78003f60 2024-03-06T07:24:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.507+0000 7fca8d7fa640 1 -- 172.21.15.77:0/1137338563 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca80012040 con 0x7fca78003f60 2024-03-06T07:24:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.507+0000 7fca8d7fa640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7c06b500 0x7fca7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.507+0000 7fca8d7fa640 1 -- 172.21.15.77:0/1137338563 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fca8008a570 con 0x7fca78003f60 2024-03-06T07:24:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.507+0000 7fca8ffff640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7c06b500 0x7fca7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.508+0000 7fca8ffff640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7c06b500 0x7fca7c06d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fca84001750 tx=0x7fca84009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.511+0000 7fca8d7fa640 1 -- 172.21.15.77:0/1137338563 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca80019030 con 0x7fca78003f60 2024-03-06T07:24:07.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.755+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fca90144790 con 0x7fca78003f60 2024-03-06T07:24:07.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.756+0000 7fca8d7fa640 1 -- 172.21.15.77:0/1137338563 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fca800590f0 con 0x7fca78003f60 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.758 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:24:07.759 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:07.760 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.761 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.762 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.763 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:24:07.771 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:07.771 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:24:07.771 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:07.771 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:07.771 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.771 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:07.771 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:24:07.771 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:24:07.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7c06b500 msgr2=0x7fca7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7c06b500 0x7fca7c06d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fca84001750 tx=0x7fca84009030 comp rx=0 tx=0).stop 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 msgr2=0x7fca9013cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca9013cc00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fca800119a0 tx=0x7fca80011e60 comp rx=0 tx=0).stop 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 shutdown_connections 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca90150960 0x7fca90142f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca9013fa70 0x7fca90140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fca7c06b500 0x7fca7c06d9c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 --2- 172.21.15.77:0/1137338563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca78003f60 0x7fca9013cc00 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 >> 172.21.15.77:0/1137338563 conn(0x7fca900cf8a0 msgr2=0x7fca900707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 shutdown_connections 2024-03-06T07:24:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.758+0000 7fca96f1e640 1 -- 172.21.15.77:0/1137338563 wait complete. 2024-03-06T07:24:07.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:07.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.872+0000 7fc29de83640 1 Processor -- start 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.872+0000 7fc29de83640 1 -- start start 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.872+0000 7fc29de83640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc2980d4920 0x7fc2980cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.872+0000 7fc29de83640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc29815dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.872+0000 7fc29de83640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2980ceac0 con 0x7fc2981621c0 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.872+0000 7fc29de83640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2980cf690 con 0x7fc2980d4920 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.872+0000 7fc29de83640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2980d2dc0 con 0x7fc2981625a0 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc2977fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc2981621c0 0x7fc2980d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45600/0 (socket says 172.21.15.77:45600) 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc2977fe640 1 -- 172.21.15.77:0/4232017892 learned_addr learned my addr 172.21.15.77:0/4232017892 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc28effd640 1 --2- 172.21.15.77:0/4232017892 >> v2:172.21.15.77:3300/0 conn(0x7fc2980d4920 0x7fc2980cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc297fff640 1 --2- 172.21.15.77:0/4232017892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc29815dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc28effd640 1 -- 172.21.15.77:0/4232017892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 msgr2=0x7fc29815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc28effd640 1 --2- 172.21.15.77:0/4232017892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc29815dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc28effd640 1 -- 172.21.15.77:0/4232017892 >> v1:172.21.15.77:6789/0 conn(0x7fc2981621c0 legacy=0x7fc2980d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc28effd640 1 -- 172.21.15.77:0/4232017892 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2980ced60 con 0x7fc2980d4920 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc296ffd640 1 -- 172.21.15.77:0/4232017892 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066277119 0 0) 0x7fc2980ceac0 con 0x7fc2981621c0 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.873+0000 7fc297fff640 1 --2- 172.21.15.77:0/4232017892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc29815dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc28effd640 1 --2- 172.21.15.77:0/4232017892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc2980cfb20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc2800097a0 tx=0x7fc28002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8a4d8b7bab9a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc296ffd640 1 -- 172.21.15.77:0/4232017892 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280035020 con 0x7fc2980d4920 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc296ffd640 1 -- 172.21.15.77:0/4232017892 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc280031830 con 0x7fc2980d4920 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc296ffd640 1 -- 172.21.15.77:0/4232017892 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280031c00 con 0x7fc2980d4920 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc29de83640 1 -- 172.21.15.77:0/4232017892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 msgr2=0x7fc2980cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc29de83640 1 --2- 172.21.15.77:0/4232017892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc2980cfb20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc2800097a0 tx=0x7fc28002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc29de83640 1 -- 172.21.15.77:0/4232017892 shutdown_connections 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc29de83640 1 --2- 172.21.15.77:0/4232017892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc29815dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc29de83640 1 --2- 172.21.15.77:0/4232017892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc2980cfb20 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc29de83640 1 -- 172.21.15.77:0/4232017892 >> 172.21.15.77:0/4232017892 conn(0x7fc2980d3d10 msgr2=0x7fc2980d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc29de83640 1 -- 172.21.15.77:0/4232017892 shutdown_connections 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.874+0000 7fc29de83640 1 -- 172.21.15.77:0/4232017892 wait complete. 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.875+0000 7fc29de83640 1 Processor -- start 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc29de83640 1 -- start start 2024-03-06T07:24:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc29de83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc298157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc29de83640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2981621c0 0x7fc2980cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc29de83640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc2980cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc29de83640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29815e280 con 0x7fc2980d4920 2024-03-06T07:24:07.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc29de83640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2980cf320 con 0x7fc2981621c0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc29de83640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29815e400 con 0x7fc2981625a0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc2977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc298157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc2977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc298157400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50618/0 (socket says 172.21.15.77:50618) 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.876+0000 7fc2977fe640 1 -- 172.21.15.77:0/3715404792 learned_addr learned my addr 172.21.15.77:0/3715404792 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc297fff640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc2980cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc2977fe640 1 -- 172.21.15.77:0/3715404792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 msgr2=0x7fc2980cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc2977fe640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc2980cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc2977fe640 1 -- 172.21.15.77:0/3715404792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2981621c0 msgr2=0x7fc2980cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc2977fe640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2981621c0 0x7fc2980cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc2977fe640 1 -- 172.21.15.77:0/3715404792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2980cc3c0 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc2977fe640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc298157400 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc28800c500 tx=0x7fc28800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc294ff9640 1 -- 172.21.15.77:0/3715404792 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288018070 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc294ff9640 1 -- 172.21.15.77:0/3715404792 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc288007580 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2980cc6a0 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc294ff9640 1 -- 172.21.15.77:0/3715404792 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288004a20 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.877+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2980ccc60 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.878+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29804f330 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.878+0000 7fc294ff9640 1 -- 172.21.15.77:0/3715404792 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc28801b070 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.879+0000 7fc294ff9640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc27806b4d0 0x7fc27806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.879+0000 7fc28effd640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc27806b4d0 0x7fc27806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.879+0000 7fc294ff9640 1 -- 172.21.15.77:0/3715404792 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc288089170 con 0x7fc2980d4920 2024-03-06T07:24:07.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.879+0000 7fc28effd640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc27806b4d0 0x7fc27806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc280036000 tx=0x7fc280030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:07.883+0000 7fc294ff9640 1 -- 172.21.15.77:0/3715404792 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2880583d0 con 0x7fc2980d4920 2024-03-06T07:24:08.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.055+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc298073380 con 0x7fc2980d4920 2024-03-06T07:24:08.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.055+0000 7fc294ff9640 1 -- 172.21.15.77:0/3715404792 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+12879 (secure 0 0 0) 0x7fc288057cd0 con 0x7fc2980d4920 2024-03-06T07:24:08.057 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:08.058 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":54,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:07.392269+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc27806b4d0 msgr2=0x7fc27806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc27806b4d0 0x7fc27806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc280036000 tx=0x7fc280030000 comp rx=0 tx=0).stop 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 msgr2=0x7fc298157400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc298157400 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc28800c500 tx=0x7fc28800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 shutdown_connections 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2981625a0 0x7fc2980cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2981621c0 0x7fc2980cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc27806b4d0 0x7fc27806d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 --2- 172.21.15.77:0/3715404792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2980d4920 0x7fc298157400 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.057+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 >> 172.21.15.77:0/3715404792 conn(0x7fc2980d3d10 msgr2=0x7fc29815ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.058+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 shutdown_connections 2024-03-06T07:24:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.058+0000 7fc29de83640 1 -- 172.21.15.77:0/3715404792 wait complete. 2024-03-06T07:24:08.070 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:08.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.172+0000 7f7c29113640 1 Processor -- start 2024-03-06T07:24:08.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.172+0000 7f7c29113640 1 -- start start 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c29113640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415a220 0x7f7c240cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c29113640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7c240712d0 0x7f7c24071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c29113640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c2415de60 0x7f7c2415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c29113640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c240cf610 con 0x7f7c2415de60 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c29113640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c24071c70 con 0x7f7c240712d0 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c29113640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c240b3320 con 0x7f7c2415a220 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c2359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c2415de60 0x7f7c2415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c2359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c2415de60 0x7f7c2415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50624/0 (socket says 172.21.15.77:50624) 2024-03-06T07:24:08.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c2359e640 1 -- 172.21.15.77:0/2391497409 learned_addr learned my addr 172.21.15.77:0/2391497409 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c2359e640 1 -- 172.21.15.77:0/2391497409 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415a220 msgr2=0x7f7c240cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c22d9d640 1 --2- 172.21.15.77:0/2391497409 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415a220 0x7f7c240cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c1a59c640 1 --2- 172.21.15.77:0/2391497409 >> v2:172.21.15.158:3300/0 conn(0x7f7c240712d0 0x7f7c24071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c2359e640 1 --2- 172.21.15.77:0/2391497409 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415a220 0x7f7c240cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c2359e640 1 -- 172.21.15.77:0/2391497409 >> v2:172.21.15.158:3300/0 conn(0x7f7c240712d0 msgr2=0x7f7c24071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c2359e640 1 --2- 172.21.15.77:0/2391497409 >> v2:172.21.15.158:3300/0 conn(0x7f7c240712d0 0x7f7c24071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.173+0000 7f7c2359e640 1 -- 172.21.15.77:0/2391497409 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c240728d0 con 0x7f7c2415de60 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.174+0000 7f7c2359e640 1 --2- 172.21.15.77:0/2391497409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2415de60 0x7f7c2415e2f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7c2406eac0 tx=0x7f7c1c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3bcb14ee34a27ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.174+0000 7f7c2259c640 1 -- 172.21.15.77:0/2391497409 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c02c9e0 con 0x7f7c2415de60 2024-03-06T07:24:08.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.174+0000 7f7c2259c640 1 -- 172.21.15.77:0/2391497409 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c02cb80 con 0x7f7c2415de60 2024-03-06T07:24:08.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.174+0000 7f7c2259c640 1 -- 172.21.15.77:0/2391497409 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0343f0 con 0x7f7c2415de60 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 -- 172.21.15.77:0/2391497409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2415de60 msgr2=0x7f7c2415e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 --2- 172.21.15.77:0/2391497409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2415de60 0x7f7c2415e2f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7c2406eac0 tx=0x7f7c1c02be40 comp rx=0 tx=0).stop 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 -- 172.21.15.77:0/2391497409 shutdown_connections 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 --2- 172.21.15.77:0/2391497409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c2415de60 0x7f7c2415e2f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 --2- 172.21.15.77:0/2391497409 >> v2:172.21.15.158:3300/0 conn(0x7f7c240712d0 0x7f7c24071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 --2- 172.21.15.77:0/2391497409 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415a220 0x7f7c240cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 -- 172.21.15.77:0/2391497409 >> 172.21.15.77:0/2391497409 conn(0x7f7c2406b190 msgr2=0x7f7c2406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 -- 172.21.15.77:0/2391497409 shutdown_connections 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.175+0000 7f7c29113640 1 -- 172.21.15.77:0/2391497409 wait complete. 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.176+0000 7f7c29113640 1 Processor -- start 2024-03-06T07:24:08.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.176+0000 7f7c29113640 1 -- start start 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c29113640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c240712d0 0x7f7c240afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c29113640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c2415a220 0x7f7c241f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c29113640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415de60 0x7f7c2414fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c29113640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c24072030 con 0x7f7c240712d0 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c29113640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c24072bb0 con 0x7f7c2415a220 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c29113640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c24072530 con 0x7f7c2415de60 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c240712d0 0x7f7c240afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c240712d0 0x7f7c240afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50628/0 (socket says 172.21.15.77:50628) 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 -- 172.21.15.77:0/4223659889 learned_addr learned my addr 172.21.15.77:0/4223659889 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c2359e640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415de60 0x7f7c2414fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 -- 172.21.15.77:0/4223659889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415de60 msgr2=0x7f7c2414fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415de60 0x7f7c2414fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 -- 172.21.15.77:0/4223659889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c2415a220 msgr2=0x7f7c241f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c2415a220 0x7f7c241f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 -- 172.21.15.77:0/4223659889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c241f7ce0 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c2359e640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415de60 0x7f7c2414fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.177+0000 7f7c22d9d640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c240712d0 0x7f7c240afd50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400c7f0 tx=0x7f7c1400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.178+0000 7f7c1bfff640 1 -- 172.21.15.77:0/4223659889 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14015070 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.178+0000 7f7c1bfff640 1 -- 172.21.15.77:0/4223659889 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c14004240 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.178+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c241f7fc0 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.178+0000 7f7c1bfff640 1 -- 172.21.15.77:0/4223659889 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14007620 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.178+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c240c8e00 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.179+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c240c9000 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.179+0000 7f7c1bfff640 1 -- 172.21.15.77:0/4223659889 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c140079d0 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.179+0000 7f7c1bfff640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7c0406b4d0 0x7f7c0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.179+0000 7f7c1bfff640 1 -- 172.21.15.77:0/4223659889 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7c14089f60 con 0x7f7c240712d0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.179+0000 7f7c1a59c640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7c0406b4d0 0x7f7c0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.180+0000 7f7c1a59c640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7c0406b4d0 0x7f7c0406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c006b10 tx=0x7f7c0c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.184+0000 7f7c1bfff640 1 -- 172.21.15.77:0/4223659889 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c240c9000 con 0x7f7c240712d0 2024-03-06T07:24:08.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.355+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c240c8bf0 con 0x7f7c240712d0 2024-03-06T07:24:08.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.355+0000 7f7c1bfff640 1 -- 172.21.15.77:0/4223659889 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c14059130 con 0x7f7c240712d0 2024-03-06T07:24:08.357 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7c0406b4d0 msgr2=0x7f7c0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7c0406b4d0 0x7f7c0406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c006b10 tx=0x7f7c0c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c240712d0 msgr2=0x7f7c240afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c240712d0 0x7f7c240afd50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400c7f0 tx=0x7f7c1400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 shutdown_connections 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7c2415de60 0x7f7c2414fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7c2415a220 0x7f7c241f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7c0406b4d0 0x7f7c0406d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 --2- 172.21.15.77:0/4223659889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c240712d0 0x7f7c240afd50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.357+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 >> 172.21.15.77:0/4223659889 conn(0x7f7c2406b190 msgr2=0x7f7c240b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.358+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 shutdown_connections 2024-03-06T07:24:08.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.358+0000 7f7c29113640 1 -- 172.21.15.77:0/4223659889 wait complete. 2024-03-06T07:24:08.371 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.469+0000 7f6a601e2640 1 Processor -- start 2024-03-06T07:24:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.469+0000 7f6a601e2640 1 -- start start 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a601e2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6a581515e0 0x7f6a580cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a601e2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a5815b430 0x7f6a58071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a601e2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5815b8e0 con 0x7f6a580cd7a0 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a601e2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a58070a70 con 0x7f6a5815b430 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a601e2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a58070d50 con 0x7f6a581515e0 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a5d77d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a5815b430 0x7f6a58071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a5d77d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a5815b430 0x7f6a58071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56636/0 (socket says 172.21.15.77:56636) 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a5e77f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6a580cd7a0 0x7f6a580cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54896/0 (socket says 172.21.15.77:54896) 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a5d77d640 1 -- 172.21.15.77:0/3049457667 learned_addr learned my addr 172.21.15.77:0/3049457667 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a5df7e640 1 --2- 172.21.15.77:0/3049457667 >> v2:172.21.15.158:3300/0 conn(0x7f6a581515e0 0x7f6a580cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a5d77d640 1 -- 172.21.15.77:0/3049457667 >> v2:172.21.15.158:3300/0 conn(0x7f6a581515e0 msgr2=0x7f6a580cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a5d77d640 1 --2- 172.21.15.77:0/3049457667 >> v2:172.21.15.158:3300/0 conn(0x7f6a581515e0 0x7f6a580cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.470+0000 7f6a5d77d640 1 -- 172.21.15.77:0/3049457667 >> v1:172.21.15.77:6789/0 conn(0x7f6a580cd7a0 legacy=0x7f6a580cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.471+0000 7f6a5d77d640 1 -- 172.21.15.77:0/3049457667 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58071030 con 0x7f6a5815b430 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.471+0000 7f6a5cf7c640 1 -- 172.21.15.77:0/3049457667 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457225950 0 0) 0x7f6a5815b8e0 con 0x7f6a580cd7a0 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.471+0000 7f6a5df7e640 1 --2- 172.21.15.77:0/3049457667 >> v2:172.21.15.158:3300/0 conn(0x7f6a581515e0 0x7f6a580cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.471+0000 7f6a5d77d640 1 --2- 172.21.15.77:0/3049457667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a5815b430 0x7f6a58071830 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6a5806eb50 tx=0x7f6a4802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198875b935b04ff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.471+0000 7f6a5cf7c640 1 -- 172.21.15.77:0/3049457667 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48034070 con 0x7f6a5815b430 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.471+0000 7f6a5cf7c640 1 -- 172.21.15.77:0/3049457667 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a48030860 con 0x7f6a5815b430 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.471+0000 7f6a5cf7c640 1 -- 172.21.15.77:0/3049457667 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48030c30 con 0x7f6a5815b430 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.472+0000 7f6a601e2640 1 -- 172.21.15.77:0/3049457667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a5815b430 msgr2=0x7f6a58071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.472+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3049457667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a5815b430 0x7f6a58071830 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6a5806eb50 tx=0x7f6a4802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.472+0000 7f6a601e2640 1 -- 172.21.15.77:0/3049457667 shutdown_connections 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.472+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3049457667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a5815b430 0x7f6a58071830 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.472+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3049457667 >> v2:172.21.15.158:3300/0 conn(0x7f6a581515e0 0x7f6a580cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.472+0000 7f6a601e2640 1 -- 172.21.15.77:0/3049457667 >> 172.21.15.77:0/3049457667 conn(0x7f6a580cf8a0 msgr2=0x7f6a580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.472+0000 7f6a601e2640 1 -- 172.21.15.77:0/3049457667 shutdown_connections 2024-03-06T07:24:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.472+0000 7f6a601e2640 1 -- 172.21.15.77:0/3049457667 wait complete. 2024-03-06T07:24:08.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.473+0000 7f6a601e2640 1 Processor -- start 2024-03-06T07:24:08.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.473+0000 7f6a601e2640 1 -- start start 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a601e2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a580cd7a0 0x7f6a5814a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a601e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a581515e0 0x7f6a58150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a601e2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a5815b430 0x7f6a5814ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a601e2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a58071ef0 con 0x7f6a581515e0 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a601e2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a58071d70 con 0x7f6a5815b430 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a601e2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580cc3a0 con 0x7f6a580cd7a0 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a581515e0 0x7f6a58150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a581515e0 0x7f6a58150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56638/0 (socket says 172.21.15.77:56638) 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 -- 172.21.15.77:0/3708346454 learned_addr learned my addr 172.21.15.77:0/3708346454 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 -- 172.21.15.77:0/3708346454 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a580cd7a0 msgr2=0x7f6a5814a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a580cd7a0 0x7f6a5814a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 -- 172.21.15.77:0/3708346454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a5815b430 msgr2=0x7f6a5814ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a5815b430 0x7f6a5814ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 -- 172.21.15.77:0/3708346454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a581509b0 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.474+0000 7f6a5d77d640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a581515e0 0x7f6a58150040 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6a48000c00 tx=0x7f6a480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.475+0000 7f6a4e7fc640 1 -- 172.21.15.77:0/3708346454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48034040 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.475+0000 7f6a4e7fc640 1 -- 172.21.15.77:0/3708346454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4803ad90 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.475+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a580c5960 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.475+0000 7f6a4e7fc640 1 -- 172.21.15.77:0/3708346454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48004270 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.475+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a580c5f20 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.476+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a58068840 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.476+0000 7f6a4e7fc640 1 -- 172.21.15.77:0/3708346454 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a4804b060 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.476+0000 7f6a4e7fc640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a3c06b500 0x7f6a3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.476+0000 7f6a4e7fc640 1 -- 172.21.15.77:0/3708346454 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6a480abe60 con 0x7f6a581515e0 2024-03-06T07:24:08.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.476+0000 7f6a5df7e640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a3c06b500 0x7f6a3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.477+0000 7f6a5df7e640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a3c06b500 0x7f6a3c06d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6a54008ea0 tx=0x7f6a540083d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.481+0000 7f6a4e7fc640 1 -- 172.21.15.77:0/3708346454 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a4803f070 con 0x7f6a581515e0 2024-03-06T07:24:08.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.653+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6a5804f330 con 0x7f6a581515e0 2024-03-06T07:24:08.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.653+0000 7f6a4e7fc640 1 -- 172.21.15.77:0/3708346454 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6a4807b0c0 con 0x7f6a581515e0 2024-03-06T07:24:08.655 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:08.655 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:24:08.655 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:08.655 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:24:08.656 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:24:08.656 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:08.656 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:24:08.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a3c06b500 msgr2=0x7f6a3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a3c06b500 0x7f6a3c06d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6a54008ea0 tx=0x7f6a540083d0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a581515e0 msgr2=0x7f6a58150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a581515e0 0x7f6a58150040 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6a48000c00 tx=0x7f6a480046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 shutdown_connections 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a3c06b500 0x7f6a3c06d9c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a581515e0 0x7f6a58150040 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a5815b430 0x7f6a5814ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 --2- 172.21.15.77:0/3708346454 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a580cd7a0 0x7f6a5814a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 >> 172.21.15.77:0/3708346454 conn(0x7f6a580cf8a0 msgr2=0x7f6a5814ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 shutdown_connections 2024-03-06T07:24:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.655+0000 7f6a601e2640 1 -- 172.21.15.77:0/3708346454 wait complete. 2024-03-06T07:24:08.668 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:08.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:08.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.771+0000 7f348908f640 1 Processor -- start 2024-03-06T07:24:08.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.771+0000 7f348908f640 1 -- start start 2024-03-06T07:24:08.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.771+0000 7f348908f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f348406be80 0x7f34840b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.771+0000 7f348908f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f348415f9d0 0x7f348415fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.771+0000 7f348908f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3484068b00 con 0x7f34840b5ca0 2024-03-06T07:24:08.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.771+0000 7f348908f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348415edf0 con 0x7f348406be80 2024-03-06T07:24:08.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.771+0000 7f348908f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34841618a0 con 0x7f348415f9d0 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.771+0000 7f3482d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f348406be80 0x7f34840b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.772+0000 7f3482d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f348406be80 0x7f34840b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56652/0 (socket says 172.21.15.77:56652) 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.772+0000 7f3482d9d640 1 -- 172.21.15.77:0/2061112563 learned_addr learned my addr 172.21.15.77:0/2061112563 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.772+0000 7f348359e640 1 --2- 172.21.15.77:0/2061112563 >> v2:172.21.15.158:3300/0 conn(0x7f348415f9d0 0x7f348415fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.772+0000 7f3482d9d640 1 -- 172.21.15.77:0/2061112563 >> v2:172.21.15.158:3300/0 conn(0x7f348415f9d0 msgr2=0x7f348415fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.772+0000 7f3482d9d640 1 --2- 172.21.15.77:0/2061112563 >> v2:172.21.15.158:3300/0 conn(0x7f348415f9d0 0x7f348415fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.772+0000 7f3482d9d640 1 -- 172.21.15.77:0/2061112563 >> v1:172.21.15.158:6789/0 conn(0x7f34840b5ca0 legacy=0x7f348406c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.772+0000 7f3482d9d640 1 -- 172.21.15.77:0/2061112563 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348415e4e0 con 0x7f348406be80 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.772+0000 7f3482d9d640 1 --2- 172.21.15.77:0/2061112563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34840b11b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f348415ef70 tx=0x7f347402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e2d0e646770da0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f3481d9b640 1 -- 172.21.15.77:0/2061112563 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347402ca70 con 0x7f348406be80 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f3481d9b640 1 -- 172.21.15.77:0/2061112563 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347402cc10 con 0x7f348406be80 2024-03-06T07:24:08.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f3481d9b640 1 -- 172.21.15.77:0/2061112563 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34740343f0 con 0x7f348406be80 2024-03-06T07:24:08.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f348908f640 1 -- 172.21.15.77:0/2061112563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 msgr2=0x7f34840b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f348908f640 1 --2- 172.21.15.77:0/2061112563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34840b11b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f348415ef70 tx=0x7f347402bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f348908f640 1 -- 172.21.15.77:0/2061112563 shutdown_connections 2024-03-06T07:24:08.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f348908f640 1 --2- 172.21.15.77:0/2061112563 >> v2:172.21.15.158:3300/0 conn(0x7f348415f9d0 0x7f348415fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f348908f640 1 --2- 172.21.15.77:0/2061112563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34840b11b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f348908f640 1 -- 172.21.15.77:0/2061112563 >> 172.21.15.77:0/2061112563 conn(0x7f34840b4ed0 msgr2=0x7f34840b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f348908f640 1 -- 172.21.15.77:0/2061112563 shutdown_connections 2024-03-06T07:24:08.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.773+0000 7f348908f640 1 -- 172.21.15.77:0/2061112563 wait complete. 2024-03-06T07:24:08.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f348908f640 1 Processor -- start 2024-03-06T07:24:08.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f348908f640 1 -- start start 2024-03-06T07:24:08.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f348908f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34841eb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f348908f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34840b5ca0 0x7f34840cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f348908f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f348415f9d0 0x7f34840ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f348908f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348415ea80 con 0x7f348406be80 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f348908f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348415e7a0 con 0x7f34840b5ca0 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f348908f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348415f280 con 0x7f348415f9d0 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f3482d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34841eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f3482d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34841eb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56668/0 (socket says 172.21.15.77:56668) 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.775+0000 7f3482d9d640 1 -- 172.21.15.77:0/2102450397 learned_addr learned my addr 172.21.15.77:0/2102450397 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f348359e640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f348415f9d0 0x7f34840ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f348259c640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34840b5ca0 0x7f34840cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f3482d9d640 1 -- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f348415f9d0 msgr2=0x7f34840ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f3482d9d640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f348415f9d0 0x7f34840ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f3482d9d640 1 -- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34840b5ca0 msgr2=0x7f34840cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f3482d9d640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34840b5ca0 0x7f34840cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f3482d9d640 1 -- 172.21.15.77:0/2102450397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34840cae00 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f348259c640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34840b5ca0 0x7f34840cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f3482d9d640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34841eb230 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3474036040 tx=0x7f3474004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f346b7fe640 1 -- 172.21.15.77:0/2102450397 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474043020 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f346b7fe640 1 -- 172.21.15.77:0/2102450397 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347403e450 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f346b7fe640 1 -- 172.21.15.77:0/2102450397 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474009510 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34840cde20 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.776+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34840cb380 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.777+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348404f330 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.778+0000 7f346b7fe640 1 -- 172.21.15.77:0/2102450397 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3474046020 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.778+0000 7f346b7fe640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f347006b4d0 0x7f347006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.778+0000 7f346b7fe640 1 -- 172.21.15.77:0/2102450397 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f34740ac6f0 con 0x7f348406be80 2024-03-06T07:24:08.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.778+0000 7f348259c640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f347006b4d0 0x7f347006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.778+0000 7f348259c640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f347006b4d0 0x7f347006d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3478007fd0 tx=0x7f3478009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.782+0000 7f346b7fe640 1 -- 172.21.15.77:0/2102450397 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f347404d030 con 0x7f348406be80 2024-03-06T07:24:08.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:08.956+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f348406ec10 con 0x7f348406be80 2024-03-06T07:24:09.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.399+0000 7f346b7fe640 1 -- 172.21.15.77:0/2102450397 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) v1 ==== 138+0+0 (secure 0 0 0) 0x7f347407b950 con 0x7f348406be80 2024-03-06T07:24:09.401 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f347006b4d0 msgr2=0x7f347006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f347006b4d0 0x7f347006d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3478007fd0 tx=0x7f3478009040 comp rx=0 tx=0).stop 2024-03-06T07:24:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 msgr2=0x7f34841eb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34841eb230 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3474036040 tx=0x7f3474004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 shutdown_connections 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f348415f9d0 0x7f34840ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34840b5ca0 0x7f34840cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f347006b4d0 0x7f347006d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 --2- 172.21.15.77:0/2102450397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f348406be80 0x7f34841eb230 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 >> 172.21.15.77:0/2102450397 conn(0x7f34840b4ed0 msgr2=0x7f3484072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 shutdown_connections 2024-03-06T07:24:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.402+0000 7f348908f640 1 -- 172.21.15.77:0/2102450397 wait complete. 2024-03-06T07:24:09.419 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:09.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.525+0000 7fa2adfd7640 1 Processor -- start 2024-03-06T07:24:09.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.525+0000 7fa2adfd7640 1 -- start start 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2adfd7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815a220 0x7fa2a80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2adfd7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a80b3320 con 0x7fa2a806da00 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2adfd7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a815ec80 con 0x7fa2a80712d0 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2adfd7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a80cf610 con 0x7fa2a815a220 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2a7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa2a806da00 0x7fa2a815e570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54904/0 (socket says 172.21.15.77:54904) 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2a7fff640 1 -- 172.21.15.77:0/363288558 learned_addr learned my addr 172.21.15.77:0/363288558 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815a220 0x7fa2a80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861665268 0 0) 0x7fa2a80b3320 con 0x7fa2a806da00 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa290005730 con 0x7fa2a806da00 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.526+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011330037 0 0) 0x7fa2a815ec80 con 0x7fa2a80712d0 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a80b3320 con 0x7fa2a80712d0 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2136630752 0 0) 0x7fa290005730 con 0x7fa2a806da00 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815a220 msgr2=0x7fa2a80cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815a220 0x7fa2a80cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 >> v1:172.21.15.158:6789/0 conn(0x7fa2a80712d0 legacy=0x7fa2a815de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a80728d0 con 0x7fa2a806da00 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fa2a00019f0 con 0x7fa2a806da00 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 >> v1:172.21.15.77:6789/0 conn(0x7fa2a806da00 legacy=0x7fa2a815e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa290005d80 0x7fa290006180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290006a10 0x7fa290006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815a220 0x7fa290007430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a80cf610 con 0x7fa290005d80 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29000dd60 con 0x7fa290006a10 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29000df30 con 0x7fa2a815a220 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa290005d80 0x7fa290006180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a7fff640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815a220 0x7fa290007430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a6ffd640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290006a10 0x7fa290006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a77fe640 1 -- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815a220 msgr2=0x7fa290007430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815a220 0x7fa290007430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a77fe640 1 -- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290006a10 msgr2=0x7fa290006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290006a10 0x7fa290006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.527+0000 7fa2a77fe640 1 -- 172.21.15.77:0/363288558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa290005730 con 0x7fa290005d80 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.528+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa290005d80 0x7fa290006180 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa298009af0 tx=0x7fa290007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848feb83f721356 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.528+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa298039070 con 0x7fa290005d80 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.528+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29802c6b0 con 0x7fa290005d80 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.528+0000 7fa2a67fc640 1 -- 172.21.15.77:0/363288558 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa298009690 con 0x7fa290005d80 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.528+0000 7fa2adfd7640 1 -- 172.21.15.77:0/363288558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa290005d80 msgr2=0x7fa290006180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.528+0000 7fa2adfd7640 1 --2- 172.21.15.77:0/363288558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa290005d80 0x7fa290006180 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa298009af0 tx=0x7fa290007a40 comp rx=0 tx=0).stop 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.528+0000 7fa2a7fff640 1 -- 172.21.15.77:0/363288558 reap_dead start 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.529+0000 7fa2adfd7640 1 -- 172.21.15.77:0/363288558 shutdown_connections 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.529+0000 7fa2adfd7640 1 -- 172.21.15.77:0/363288558 >> 172.21.15.77:0/363288558 conn(0x7fa2a806b190 msgr2=0x7fa2a806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.529+0000 7fa2adfd7640 1 -- 172.21.15.77:0/363288558 shutdown_connections 2024-03-06T07:24:09.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.529+0000 7fa2adfd7640 1 -- 172.21.15.77:0/363288558 wait complete. 2024-03-06T07:24:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.530+0000 7fa2adfd7640 1 Processor -- start 2024-03-06T07:24:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.530+0000 7fa2adfd7640 1 -- start start 2024-03-06T07:24:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2adfd7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290005d80 0x7fa2a80ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2adfd7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815b8c0 0x7fa2a80aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2adfd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2a815bd40 0x7fa2a81f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2adfd7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8072030 con 0x7fa2a815bd40 2024-03-06T07:24:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2adfd7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8072bb0 con 0x7fa290005d80 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2adfd7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8072530 con 0x7fa2a815b8c0 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2a815bd40 0x7fa2a81f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2a815bd40 0x7fa2a81f01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56690/0 (socket says 172.21.15.77:56690) 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 -- 172.21.15.77:0/499305383 learned_addr learned my addr 172.21.15.77:0/499305383 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290005d80 0x7fa2a80ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a6ffd640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815b8c0 0x7fa2a80aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 -- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815b8c0 msgr2=0x7fa2a80aa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815b8c0 0x7fa2a80aa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 -- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290005d80 msgr2=0x7fa2a80ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290005d80 0x7fa2a80ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 -- 172.21.15.77:0/499305383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a80ab150 con 0x7fa2a815bd40 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290005d80 0x7fa2a80ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2a7fff640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2a815bd40 0x7fa2a81f01f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa2a815ef00 tx=0x7fa2a0007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa28ffff640 1 -- 172.21.15.77:0/499305383 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a000aaf0 con 0x7fa2a815bd40 2024-03-06T07:24:09.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa28ffff640 1 -- 172.21.15.77:0/499305383 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a0019070 con 0x7fa2a815bd40 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a80ab430 con 0x7fa2a815bd40 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa28ffff640 1 -- 172.21.15.77:0/499305383 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a00133f0 con 0x7fa2a815bd40 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.531+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a80ab9f0 con 0x7fa2a815bd40 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.532+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a80732b0 con 0x7fa2a815bd40 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.533+0000 7fa28ffff640 1 -- 172.21.15.77:0/499305383 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2a0013590 con 0x7fa2a815bd40 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.533+0000 7fa28ffff640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa29406b4d0 0x7fa29406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.533+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa29406b4d0 0x7fa29406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.533+0000 7fa28ffff640 1 -- 172.21.15.77:0/499305383 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fa2a008a630 con 0x7fa2a815bd40 2024-03-06T07:24:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.534+0000 7fa2a77fe640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa29406b4d0 0x7fa29406d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa298036000 tx=0x7fa298030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.538+0000 7fa28ffff640 1 -- 172.21.15.77:0/499305383 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2a0059610 con 0x7fa2a815bd40 2024-03-06T07:24:09.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:09.711+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa2a8068ff0 con 0x7fa2a815bd40 2024-03-06T07:24:10.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.404+0000 7fa28ffff640 1 -- 172.21.15.77:0/499305383 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa2a0058f10 con 0x7fa2a815bd40 2024-03-06T07:24:10.406 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:24:10.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa29406b4d0 msgr2=0x7fa29406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa29406b4d0 0x7fa29406d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa298036000 tx=0x7fa298030000 comp rx=0 tx=0).stop 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2a815bd40 msgr2=0x7fa2a81f01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2a815bd40 0x7fa2a81f01f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa2a815ef00 tx=0x7fa2a0007bd0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 shutdown_connections 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa29406b4d0 0x7fa29406d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2a815bd40 0x7fa2a81f01f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2a815b8c0 0x7fa2a80aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 --2- 172.21.15.77:0/499305383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa290005d80 0x7fa2a80ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.407+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 >> 172.21.15.77:0/499305383 conn(0x7fa2a806b190 msgr2=0x7fa2a815cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.408+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 shutdown_connections 2024-03-06T07:24:10.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.408+0000 7fa2adfd7640 1 -- 172.21.15.77:0/499305383 wait complete. 2024-03-06T07:24:10.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:10.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.538+0000 7fb00213e640 1 Processor -- start 2024-03-06T07:24:10.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.538+0000 7fb00213e640 1 -- start start 2024-03-06T07:24:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7fb00213e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faffc158800 0x7faffc15b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7fb00213e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc15e8a0 con 0x7faffc158420 2024-03-06T07:24:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7fb00213e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc15eb80 con 0x7faffc0b3cd0 2024-03-06T07:24:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7fb00213e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc15e650 con 0x7faffc158800 2024-03-06T07:24:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7faffb7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faffc158420 0x7faffc0b35c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54912/0 (socket says 172.21.15.77:54912) 2024-03-06T07:24:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7faffb7fe640 1 -- 172.21.15.77:0/1966129794 learned_addr learned my addr 172.21.15.77:0/1966129794 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727010197 0 0) 0x7faffc15e8a0 con 0x7faffc158420 2024-03-06T07:24:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7faffaffd640 1 --2- 172.21.15.77:0/1966129794 >> v2:172.21.15.158:3300/0 conn(0x7faffc158800 0x7faffc15b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.539+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe4003520 con 0x7faffc158420 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1892278198 0 0) 0x7fafe4003520 con 0x7faffc158420 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 >> v2:172.21.15.158:3300/0 conn(0x7faffc158800 msgr2=0x7faffc15b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 --2- 172.21.15.77:0/1966129794 >> v2:172.21.15.158:3300/0 conn(0x7faffc158800 0x7faffc15b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 >> v1:172.21.15.158:6789/0 conn(0x7faffc0b3cd0 legacy=0x7faffc15b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc15ee60 con 0x7faffc158420 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fafec001670 con 0x7faffc158420 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 >> v1:172.21.15.77:6789/0 conn(0x7faffc158420 legacy=0x7faffc0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fafe40039b0 0x7fafe4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafe4004800 0x7fafe4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafe40051c0 0x7fafe400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc15e650 con 0x7fafe40051c0 2024-03-06T07:24:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe400c160 con 0x7fafe4004800 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe400c360 con 0x7fafe40039b0 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffbfff640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafe40051c0 0x7fafe400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffaffd640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafe4004800 0x7fafe4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffb7fe640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fafe40039b0 0x7fafe4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffbfff640 1 -- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fafe40039b0 msgr2=0x7fafe4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffbfff640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fafe40039b0 0x7fafe4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffbfff640 1 -- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafe4004800 msgr2=0x7fafe4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffbfff640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafe4004800 0x7fafe4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffbfff640 1 -- 172.21.15.77:0/1966129794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe4003520 con 0x7fafe40051c0 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.540+0000 7faffb7fe640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fafe40039b0 0x7fafe4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.541+0000 7faffbfff640 1 -- 172.21.15.77:0/1966129794 reap_dead start 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.541+0000 7faffbfff640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafe40051c0 0x7fafe400b9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7faffc0b40e0 tx=0x7fafe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3870435cc2eccd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.541+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4001a70 con 0x7fafe40051c0 2024-03-06T07:24:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.541+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4003b70 con 0x7fafe40051c0 2024-03-06T07:24:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.541+0000 7faffa7fc640 1 -- 172.21.15.77:0/1966129794 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40050d0 con 0x7fafe40051c0 2024-03-06T07:24:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.542+0000 7fb00213e640 1 -- 172.21.15.77:0/1966129794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafe40051c0 msgr2=0x7fafe400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.542+0000 7fb00213e640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafe40051c0 0x7fafe400b9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7faffc0b40e0 tx=0x7fafe4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.542+0000 7fb00213e640 1 -- 172.21.15.77:0/1966129794 shutdown_connections 2024-03-06T07:24:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.542+0000 7fb00213e640 1 --2- 172.21.15.77:0/1966129794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafe40051c0 0x7fafe400b9c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.542+0000 7fb00213e640 1 -- 172.21.15.77:0/1966129794 >> 172.21.15.77:0/1966129794 conn(0x7faffc06afb0 msgr2=0x7faffc06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.542+0000 7fb00213e640 1 -- 172.21.15.77:0/1966129794 shutdown_connections 2024-03-06T07:24:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.542+0000 7fb00213e640 1 -- 172.21.15.77:0/1966129794 wait complete. 2024-03-06T07:24:10.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.543+0000 7fb00213e640 1 Processor -- start 2024-03-06T07:24:10.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.543+0000 7fb00213e640 1 -- start start 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fb00213e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafe40051c0 0x7faffc1f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fb00213e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faffc0cdd20 0x7faffc156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fb00213e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faffc0ce6f0 0x7faffc156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fb00213e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0b3740 con 0x7faffc0cdd20 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fb00213e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0b35c0 con 0x7fafe40051c0 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fb00213e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0b38c0 con 0x7faffc0ce6f0 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faffc0cdd20 0x7faffc156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faffc0cdd20 0x7faffc156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56708/0 (socket says 172.21.15.77:56708) 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 -- 172.21.15.77:0/4163954430 learned_addr learned my addr 172.21.15.77:0/4163954430 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 -- 172.21.15.77:0/4163954430 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faffc0ce6f0 msgr2=0x7faffc156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faffc0ce6f0 0x7faffc156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 -- 172.21.15.77:0/4163954430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafe40051c0 msgr2=0x7faffc1f3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafe40051c0 0x7faffc1f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 -- 172.21.15.77:0/4163954430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc1f3e90 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7faffaffd640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faffc0cdd20 0x7faffc156350 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faff0011910 tx=0x7faff0011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fafe3fff640 1 -- 172.21.15.77:0/4163954430 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff000fa90 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fafe3fff640 1 -- 172.21.15.77:0/4163954430 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff000fc30 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc0cecb0 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fafe3fff640 1 -- 172.21.15.77:0/4163954430 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0014440 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.544+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc0cf270 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.546+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc04f330 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.546+0000 7fafe3fff640 1 -- 172.21.15.77:0/4163954430 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faff0012040 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.546+0000 7fafe3fff640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fafe806b500 0x7fafe806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.546+0000 7fafe3fff640 1 -- 172.21.15.77:0/4163954430 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5609+0+0 (secure 0 0 0) 0x7faff008a6d0 con 0x7faffc0cdd20 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.546+0000 7faffb7fe640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fafe806b500 0x7fafe806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.547+0000 7faffb7fe640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fafe806b500 0x7fafe806d9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fafec001eb0 tx=0x7fafec001190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.551+0000 7fafe3fff640 1 -- 172.21.15.77:0/4163954430 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faff0019030 con 0x7faffc0cdd20 2024-03-06T07:24:10.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:10.759+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faffc0af150 con 0x7faffc0cdd20 2024-03-06T07:24:11.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.406+0000 7fafe3fff640 1 -- 172.21.15.77:0/4163954430 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v24) v1 ==== 172+0+0 (secure 0 0 0) 0x7faff00595b0 con 0x7faffc0cdd20 2024-03-06T07:24:11.408 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:24:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.414+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fafe806b500 msgr2=0x7fafe806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.414+0000 7fb00213e640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fafe806b500 0x7fafe806d9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fafec001eb0 tx=0x7fafec001190 comp rx=0 tx=0).stop 2024-03-06T07:24:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.414+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faffc0cdd20 msgr2=0x7faffc156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.414+0000 7fb00213e640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faffc0cdd20 0x7faffc156350 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faff0011910 tx=0x7faff0011dd0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.421+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 shutdown_connections 2024-03-06T07:24:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.421+0000 7fb00213e640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fafe806b500 0x7fafe806d9c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.421+0000 7fb00213e640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faffc0cdd20 0x7faffc156350 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.421+0000 7fb00213e640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faffc0ce6f0 0x7faffc156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.421+0000 7fb00213e640 1 --2- 172.21.15.77:0/4163954430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fafe40051c0 0x7faffc1f3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.421+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 >> 172.21.15.77:0/4163954430 conn(0x7faffc06afb0 msgr2=0x7faffc15dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.428+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 shutdown_connections 2024-03-06T07:24:11.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.429+0000 7fb00213e640 1 -- 172.21.15.77:0/4163954430 wait complete. 2024-03-06T07:24:11.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:11.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.544+0000 7f6b20f5e640 1 Processor -- start 2024-03-06T07:24:11.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.545+0000 7f6b20f5e640 1 -- start start 2024-03-06T07:24:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.545+0000 7f6b20f5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c0b3060 0x7f6b1c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.545+0000 7f6b20f5e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b1c15bf30 0x7f6b1c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.545+0000 7f6b20f5e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1c15f2d0 con 0x7f6b1c13dc10 2024-03-06T07:24:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.545+0000 7f6b20f5e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1c0cf3f0 con 0x7f6b1c15bf30 2024-03-06T07:24:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.545+0000 7f6b20f5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c15b210 con 0x7f6b1c0b3060 2024-03-06T07:24:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b1ad9d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b1c15bf30 0x7f6b1c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b13fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c0b3060 0x7f6b1c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b1a59c640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6b1c13dc10 0x7f6b1c0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44938/0 (socket says 172.21.15.77:44938) 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b1a59c640 1 -- 172.21.15.77:0/3690499476 learned_addr learned my addr 172.21.15.77:0/3690499476 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b19d9b640 1 -- 172.21.15.77:0/3690499476 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201401399 0 0) 0x7f6b1c15f2d0 con 0x7f6b1c13dc10 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b19d9b640 1 -- 172.21.15.77:0/3690499476 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b08003520 con 0x7f6b1c13dc10 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b13fff640 1 -- 172.21.15.77:0/3690499476 >> v2:172.21.15.158:3300/0 conn(0x7f6b1c15bf30 msgr2=0x7f6b1c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b13fff640 1 --2- 172.21.15.77:0/3690499476 >> v2:172.21.15.158:3300/0 conn(0x7f6b1c15bf30 0x7f6b1c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b13fff640 1 -- 172.21.15.77:0/3690499476 >> v1:172.21.15.158:6789/0 conn(0x7f6b1c13dc10 legacy=0x7f6b1c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b13fff640 1 -- 172.21.15.77:0/3690499476 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c15f580 con 0x7f6b1c0b3060 2024-03-06T07:24:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.546+0000 7f6b1ad9d640 1 --2- 172.21.15.77:0/3690499476 >> v2:172.21.15.158:3300/0 conn(0x7f6b1c15bf30 0x7f6b1c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.547+0000 7f6b13fff640 1 --2- 172.21.15.77:0/3690499476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c0b3060 0x7f6b1c15e850 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6b040097a0 tx=0x7f6b0402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb9b348ca6df542 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.547+0000 7f6b19d9b640 1 -- 172.21.15.77:0/3690499476 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b040315a0 con 0x7f6b1c0b3060 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.547+0000 7f6b19d9b640 1 -- 172.21.15.77:0/3690499476 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b04037070 con 0x7f6b1c0b3060 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.547+0000 7f6b19d9b640 1 -- 172.21.15.77:0/3690499476 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04031940 con 0x7f6b1c0b3060 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.548+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3690499476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c0b3060 msgr2=0x7f6b1c15e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.548+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3690499476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c0b3060 0x7f6b1c15e850 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6b040097a0 tx=0x7f6b0402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.548+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3690499476 shutdown_connections 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.548+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3690499476 >> v2:172.21.15.158:3300/0 conn(0x7f6b1c15bf30 0x7f6b1c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.548+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3690499476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c0b3060 0x7f6b1c15e850 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.548+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3690499476 >> 172.21.15.77:0/3690499476 conn(0x7f6b1c0cf8a0 msgr2=0x7f6b1c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.548+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3690499476 shutdown_connections 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.548+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3690499476 wait complete. 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.549+0000 7f6b20f5e640 1 Processor -- start 2024-03-06T07:24:11.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.549+0000 7f6b20f5e640 1 -- start start 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b20f5e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b1c0b3060 0x7f6b1c15db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b20f5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b1c13dc10 0x7f6b1c15d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b20f5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c15bf30 0x7f6b1c136630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b20f5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c13dff0 con 0x7f6b1c13dc10 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b20f5e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c15b730 con 0x7f6b1c0b3060 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b20f5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c0cd7a0 con 0x7f6b1c15bf30 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b1c13dc10 0x7f6b1c15d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b1c13dc10 0x7f6b1c15d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56720/0 (socket says 172.21.15.77:56720) 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 -- 172.21.15.77:0/3294016884 learned_addr learned my addr 172.21.15.77:0/3294016884 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b1a59c640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b1c0b3060 0x7f6b1c15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b1ad9d640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c15bf30 0x7f6b1c136630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 -- 172.21.15.77:0/3294016884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c15bf30 msgr2=0x7f6b1c136630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c15bf30 0x7f6b1c136630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 -- 172.21.15.77:0/3294016884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b1c0b3060 msgr2=0x7f6b1c15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b1c0b3060 0x7f6b1c15db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 -- 172.21.15.77:0/3294016884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c15e250 con 0x7f6b1c13dc10 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.550+0000 7f6b13fff640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b1c13dc10 0x7f6b1c15d2d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6b0402b880 tx=0x7f6b040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.551+0000 7f6b12ffd640 1 -- 172.21.15.77:0/3294016884 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04032630 con 0x7f6b1c13dc10 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.551+0000 7f6b12ffd640 1 -- 172.21.15.77:0/3294016884 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b040327f0 con 0x7f6b1c13dc10 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.551+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c15e530 con 0x7f6b1c13dc10 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.551+0000 7f6b12ffd640 1 -- 172.21.15.77:0/3294016884 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04032be0 con 0x7f6b1c13dc10 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.551+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c15eaf0 con 0x7f6b1c13dc10 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.552+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c0b0d00 con 0x7f6b1c13dc10 2024-03-06T07:24:11.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.552+0000 7f6b12ffd640 1 -- 172.21.15.77:0/3294016884 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b0403b3f0 con 0x7f6b1c13dc10 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.553+0000 7f6b12ffd640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6afc06b4d0 0x7f6afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.553+0000 7f6b1a59c640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6afc06b4d0 0x7f6afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.553+0000 7f6b12ffd640 1 -- 172.21.15.77:0/3294016884 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b0402f030 con 0x7f6b1c13dc10 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.553+0000 7f6b1a59c640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6afc06b4d0 0x7f6afc06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c005880 tx=0x7f6b0c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.557+0000 7f6b12ffd640 1 -- 172.21.15.77:0/3294016884 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b1c0b0d00 con 0x7f6b1c13dc10 2024-03-06T07:24:11.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.730+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b1c04f330 con 0x7f6b1c13dc10 2024-03-06T07:24:11.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.731+0000 7f6b12ffd640 1 -- 172.21.15.77:0/3294016884 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f6b04040070 con 0x7f6b1c13dc10 2024-03-06T07:24:11.733 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":57,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:10.760296+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:11.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6afc06b4d0 msgr2=0x7f6afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6afc06b4d0 0x7f6afc06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c005880 tx=0x7f6b0c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b1c13dc10 msgr2=0x7f6b1c15d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b1c13dc10 0x7f6b1c15d2d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6b0402b880 tx=0x7f6b040046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 shutdown_connections 2024-03-06T07:24:11.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6afc06b4d0 0x7f6afc06d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b1c13dc10 0x7f6b1c15d2d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b1c15bf30 0x7f6b1c136630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 --2- 172.21.15.77:0/3294016884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b1c0b3060 0x7f6b1c15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 >> 172.21.15.77:0/3294016884 conn(0x7f6b1c0cf8a0 msgr2=0x7f6b1c0711f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 shutdown_connections 2024-03-06T07:24:11.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.733+0000 7f6b20f5e640 1 -- 172.21.15.77:0/3294016884 wait complete. 2024-03-06T07:24:11.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:11.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.844+0000 7f158bfac640 1 Processor -- start 2024-03-06T07:24:11.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.844+0000 7f158bfac640 1 -- start start 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f158bfac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f158413dc10 0x7f15840b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f158bfac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415bf30 0x7f15840b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f158bfac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f158415e850 0x7f158415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f158bfac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15840cf3f0 con 0x7f158413dc10 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f158bfac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f158415b210 con 0x7f158415e850 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f158bfac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158415f580 con 0x7f158415bf30 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f1589d48640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f158413dc10 0x7f15840b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f1589d48640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f158413dc10 0x7f15840b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56730/0 (socket says 172.21.15.77:56730) 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f1589d48640 1 -- 172.21.15.77:0/3634093011 learned_addr learned my addr 172.21.15.77:0/3634093011 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f1589547640 1 --2- 172.21.15.77:0/3634093011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415bf30 0x7f15840b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.845+0000 7f158a549640 1 --2- 172.21.15.77:0/3634093011 >> v2:172.21.15.158:3300/0 conn(0x7f158415e850 0x7f158415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1589d48640 1 -- 172.21.15.77:0/3634093011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415bf30 msgr2=0x7f15840b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1589d48640 1 --2- 172.21.15.77:0/3634093011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415bf30 0x7f15840b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1589d48640 1 -- 172.21.15.77:0/3634093011 >> v2:172.21.15.158:3300/0 conn(0x7f158415e850 msgr2=0x7f158415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1589d48640 1 --2- 172.21.15.77:0/3634093011 >> v2:172.21.15.158:3300/0 conn(0x7f158415e850 0x7f158415eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1589d48640 1 -- 172.21.15.77:0/3634093011 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158415b450 con 0x7f158413dc10 2024-03-06T07:24:11.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1589d48640 1 --2- 172.21.15.77:0/3634093011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f15840b2950 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f158406c790 tx=0x7f158002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b0c39d593392ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1588d46640 1 -- 172.21.15.77:0/3634093011 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158002c9e0 con 0x7f158413dc10 2024-03-06T07:24:11.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1588d46640 1 -- 172.21.15.77:0/3634093011 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158002cb80 con 0x7f158413dc10 2024-03-06T07:24:11.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.846+0000 7f1588d46640 1 -- 172.21.15.77:0/3634093011 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15800343f0 con 0x7f158413dc10 2024-03-06T07:24:11.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 -- 172.21.15.77:0/3634093011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 msgr2=0x7f15840b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 --2- 172.21.15.77:0/3634093011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f15840b2950 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f158406c790 tx=0x7f158002be40 comp rx=0 tx=0).stop 2024-03-06T07:24:11.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 -- 172.21.15.77:0/3634093011 shutdown_connections 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 --2- 172.21.15.77:0/3634093011 >> v2:172.21.15.158:3300/0 conn(0x7f158415e850 0x7f158415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 --2- 172.21.15.77:0/3634093011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415bf30 0x7f15840b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 --2- 172.21.15.77:0/3634093011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f15840b2950 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 -- 172.21.15.77:0/3634093011 >> 172.21.15.77:0/3634093011 conn(0x7f15840cf8a0 msgr2=0x7f15840cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 -- 172.21.15.77:0/3634093011 shutdown_connections 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.847+0000 7f158bfac640 1 -- 172.21.15.77:0/3634093011 wait complete. 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.848+0000 7f158bfac640 1 Processor -- start 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.848+0000 7f158bfac640 1 -- start start 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f158bfac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f158412fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f158bfac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f158415bf30 0x7f15841540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f158bfac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415e850 0x7f158413bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f158bfac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158415f1e0 con 0x7f158413dc10 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f158bfac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158413dff0 con 0x7f158415bf30 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f158bfac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158415f360 con 0x7f158415e850 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f158412fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f158412fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56734/0 (socket says 172.21.15.77:56734) 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 -- 172.21.15.77:0/1996604674 learned_addr learned my addr 172.21.15.77:0/1996604674 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f158a549640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415e850 0x7f158413bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589547640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f158415bf30 0x7f15841540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 -- 172.21.15.77:0/1996604674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415e850 msgr2=0x7f158413bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415e850 0x7f158413bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 -- 172.21.15.77:0/1996604674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f158415bf30 msgr2=0x7f15841540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f158415bf30 0x7f15841540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 -- 172.21.15.77:0/1996604674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1584142100 con 0x7f158413dc10 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.849+0000 7f1589d48640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f158412fad0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1580000f80 tx=0x7f1580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.850+0000 7f157a7fc640 1 -- 172.21.15.77:0/1996604674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580043020 con 0x7f158413dc10 2024-03-06T07:24:11.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.850+0000 7f157a7fc640 1 -- 172.21.15.77:0/1996604674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158003e450 con 0x7f158413dc10 2024-03-06T07:24:11.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.850+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15841423e0 con 0x7f158413dc10 2024-03-06T07:24:11.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.850+0000 7f157a7fc640 1 -- 172.21.15.77:0/1996604674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580009510 con 0x7f158413dc10 2024-03-06T07:24:11.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.850+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15841429a0 con 0x7f158413dc10 2024-03-06T07:24:11.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.851+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f158404f330 con 0x7f158413dc10 2024-03-06T07:24:11.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.851+0000 7f157a7fc640 1 -- 172.21.15.77:0/1996604674 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1580046020 con 0x7f158413dc10 2024-03-06T07:24:11.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.851+0000 7f157a7fc640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156806b4d0 0x7f156806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.852+0000 7f1589547640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156806b4d0 0x7f156806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.852+0000 7f157a7fc640 1 -- 172.21.15.77:0/1996604674 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15800acb90 con 0x7f158413dc10 2024-03-06T07:24:11.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.852+0000 7f1589547640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156806b4d0 0x7f156806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1584140f80 tx=0x7f1574008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:11.856+0000 7f157a7fc640 1 -- 172.21.15.77:0/1996604674 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f158004d030 con 0x7f158413dc10 2024-03-06T07:24:12.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.029+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f158406eb00 con 0x7f158413dc10 2024-03-06T07:24:12.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.030+0000 7f157a7fc640 1 -- 172.21.15.77:0/1996604674 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f158007baa0 con 0x7f158413dc10 2024-03-06T07:24:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:12.032 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":57,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:10.760296+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:12.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156806b4d0 msgr2=0x7f156806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156806b4d0 0x7f156806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1584140f80 tx=0x7f1574008040 comp rx=0 tx=0).stop 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 msgr2=0x7f158412fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f158412fad0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1580000f80 tx=0x7f1580004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 shutdown_connections 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f158415e850 0x7f158413bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f158415bf30 0x7f15841540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156806b4d0 0x7f156806d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 --2- 172.21.15.77:0/1996604674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f158413dc10 0x7f158412fad0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 >> 172.21.15.77:0/1996604674 conn(0x7f15840cf8a0 msgr2=0x7f158415d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 shutdown_connections 2024-03-06T07:24:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.032+0000 7f158bfac640 1 -- 172.21.15.77:0/1996604674 wait complete. 2024-03-06T07:24:12.045 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:12.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.145+0000 7f276ab4b640 1 Processor -- start 2024-03-06T07:24:12.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.145+0000 7f276ab4b640 1 -- start start 2024-03-06T07:24:12.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f276ab4b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f27640b4970 0x7f276406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f276ab4b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276415ff90 con 0x7f276406d5c0 2024-03-06T07:24:12.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f276ab4b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2764160270 con 0x7f276415bb40 2024-03-06T07:24:12.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f276ab4b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27640b4dd0 con 0x7f27640b4970 2024-03-06T07:24:12.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f2763fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f27640b4970 0x7f276406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f27690e8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f276406d5c0 0x7f27640cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54926/0 (socket says 172.21.15.77:54926) 2024-03-06T07:24:12.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f2763fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f27640b4970 0x7f276406d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56740/0 (socket says 172.21.15.77:56740) 2024-03-06T07:24:12.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f27690e8640 1 -- 172.21.15.77:0/2814669891 learned_addr learned my addr 172.21.15.77:0/2814669891 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:12.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f2763fff640 1 -- 172.21.15.77:0/2814669891 >> v1:172.21.15.158:6789/0 conn(0x7f276415bb40 legacy=0x7f276415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f2763fff640 1 -- 172.21.15.77:0/2814669891 >> v1:172.21.15.77:6789/0 conn(0x7f276406d5c0 legacy=0x7f27640cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f2763fff640 1 -- 172.21.15.77:0/2814669891 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764160550 con 0x7f27640b4970 2024-03-06T07:24:12.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.146+0000 7f27637fe640 1 -- 172.21.15.77:0/2814669891 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972472234 0 0) 0x7f276415ff90 con 0x7f276406d5c0 2024-03-06T07:24:12.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.147+0000 7f2763fff640 1 --2- 172.21.15.77:0/2814669891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27640b4970 0x7f276406d080 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f27580097a0 tx=0x7f275802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf1187eb33de647 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.147+0000 7f27637fe640 1 -- 172.21.15.77:0/2814669891 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2758035020 con 0x7f27640b4970 2024-03-06T07:24:12.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.147+0000 7f27637fe640 1 -- 172.21.15.77:0/2814669891 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2758031830 con 0x7f27640b4970 2024-03-06T07:24:12.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.147+0000 7f27637fe640 1 -- 172.21.15.77:0/2814669891 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2758031c00 con 0x7f27640b4970 2024-03-06T07:24:12.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.148+0000 7f276ab4b640 1 -- 172.21.15.77:0/2814669891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27640b4970 msgr2=0x7f276406d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.148+0000 7f276ab4b640 1 --2- 172.21.15.77:0/2814669891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27640b4970 0x7f276406d080 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f27580097a0 tx=0x7f275802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.148+0000 7f276ab4b640 1 -- 172.21.15.77:0/2814669891 shutdown_connections 2024-03-06T07:24:12.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.148+0000 7f276ab4b640 1 --2- 172.21.15.77:0/2814669891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27640b4970 0x7f276406d080 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.148+0000 7f276ab4b640 1 -- 172.21.15.77:0/2814669891 >> 172.21.15.77:0/2814669891 conn(0x7f276406b190 msgr2=0x7f276406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.148+0000 7f276ab4b640 1 -- 172.21.15.77:0/2814669891 shutdown_connections 2024-03-06T07:24:12.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.148+0000 7f276ab4b640 1 -- 172.21.15.77:0/2814669891 wait complete. 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.149+0000 7f276ab4b640 1 Processor -- start 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.149+0000 7f276ab4b640 1 -- start start 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f276ab4b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f276406d5c0 0x7f27640b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f276ab4b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27640b4970 0x7f27640af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f276ab4b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f276415bb40 0x7f27640b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f276ab4b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276415bf20 con 0x7f276406d5c0 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f276ab4b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276406c910 con 0x7f276415bb40 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f276ab4b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276415c0a0 con 0x7f27640b4970 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f276406d5c0 0x7f27640b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f276406d5c0 0x7f27640b1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56752/0 (socket says 172.21.15.77:56752) 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 -- 172.21.15.77:0/704550996 learned_addr learned my addr 172.21.15.77:0/704550996 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 -- 172.21.15.77:0/704550996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27640b4970 msgr2=0x7f27640af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27640b4970 0x7f27640af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 -- 172.21.15.77:0/704550996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f276415bb40 msgr2=0x7f27640b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f276415bb40 0x7f27640b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 -- 172.21.15.77:0/704550996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27640afde0 con 0x7f276406d5c0 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27688e7640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f276406d5c0 0x7f27640b1cf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f275400b3e0 tx=0x7f275400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27617fa640 1 -- 172.21.15.77:0/704550996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2754015020 con 0x7f276406d5c0 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27617fa640 1 -- 172.21.15.77:0/704550996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2754011850 con 0x7f276406d5c0 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27640b00c0 con 0x7f276406d5c0 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f27617fa640 1 -- 172.21.15.77:0/704550996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2754011c40 con 0x7f276406d5c0 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.150+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27640b0680 con 0x7f276406d5c0 2024-03-06T07:24:12.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.151+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27640d0230 con 0x7f276406d5c0 2024-03-06T07:24:12.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.152+0000 7f27617fa640 1 -- 172.21.15.77:0/704550996 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2754018020 con 0x7f276406d5c0 2024-03-06T07:24:12.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.152+0000 7f27617fa640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f275006b4d0 0x7f275006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.152+0000 7f2763fff640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f275006b4d0 0x7f275006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.152+0000 7f27617fa640 1 -- 172.21.15.77:0/704550996 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f275408a1a0 con 0x7f276406d5c0 2024-03-06T07:24:12.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.153+0000 7f2763fff640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f275006b4d0 0x7f275006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2758036000 tx=0x7f2758030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.157+0000 7f27617fa640 1 -- 172.21.15.77:0/704550996 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27640d0230 con 0x7f276406d5c0 2024-03-06T07:24:12.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.367+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f27640732b0 con 0x7f276406d5c0 2024-03-06T07:24:12.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.419+0000 7f27617fa640 1 -- 172.21.15.77:0/704550996 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f275401f020 con 0x7f276406d5c0 2024-03-06T07:24:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f275006b4d0 msgr2=0x7f275006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f275006b4d0 0x7f275006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2758036000 tx=0x7f2758030000 comp rx=0 tx=0).stop 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f276406d5c0 msgr2=0x7f27640b1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f276406d5c0 0x7f27640b1cf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f275400b3e0 tx=0x7f275400b8a0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 shutdown_connections 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f276415bb40 0x7f27640b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27640b4970 0x7f27640af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f275006b4d0 0x7f275006d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 --2- 172.21.15.77:0/704550996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f276406d5c0 0x7f27640b1cf0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 >> 172.21.15.77:0/704550996 conn(0x7f276406b190 msgr2=0x7f27641593f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.422+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 shutdown_connections 2024-03-06T07:24:12.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.423+0000 7f276ab4b640 1 -- 172.21.15.77:0/704550996 wait complete. 2024-03-06T07:24:12.435 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:12.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.537+0000 7f41912c2640 1 Processor -- start 2024-03-06T07:24:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.537+0000 7f41912c2640 1 -- start start 2024-03-06T07:24:12.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f41912c2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f418c15def0 0x7f418c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f41912c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c0cea50 0x7f418c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f41912c2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c06d030 con 0x7f418c06cab0 2024-03-06T07:24:12.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f41912c2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418c0cf1c0 con 0x7f418c15def0 2024-03-06T07:24:12.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f41912c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0cc760 con 0x7f418c0cea50 2024-03-06T07:24:12.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f418b7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c0cea50 0x7f418c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f418affd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f418c15def0 0x7f418c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f418a7fc640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f418c06cab0 0x7f418c0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44964/0 (socket says 172.21.15.77:44964) 2024-03-06T07:24:12.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.538+0000 7f418a7fc640 1 -- 172.21.15.77:0/3385549732 learned_addr learned my addr 172.21.15.77:0/3385549732 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:12.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.539+0000 7f4189ffb640 1 -- 172.21.15.77:0/3385549732 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564970523 0 0) 0x7f418c06d030 con 0x7f418c06cab0 2024-03-06T07:24:12.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.539+0000 7f4189ffb640 1 -- 172.21.15.77:0/3385549732 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4178003520 con 0x7f418c06cab0 2024-03-06T07:24:12.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.539+0000 7f418affd640 1 -- 172.21.15.77:0/3385549732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c0cea50 msgr2=0x7f418c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.539+0000 7f418affd640 1 --2- 172.21.15.77:0/3385549732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c0cea50 0x7f418c0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.539+0000 7f418affd640 1 -- 172.21.15.77:0/3385549732 >> v1:172.21.15.158:6789/0 conn(0x7f418c06cab0 legacy=0x7f418c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.539+0000 7f418affd640 1 -- 172.21.15.77:0/3385549732 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c06d310 con 0x7f418c15def0 2024-03-06T07:24:12.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.539+0000 7f418b7fe640 1 --2- 172.21.15.77:0/3385549732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c0cea50 0x7f418c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.540+0000 7f418affd640 1 --2- 172.21.15.77:0/3385549732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 0x7f418c0b41b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f41740097a0 tx=0x7f417402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91f21fdc964d6c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.540+0000 7f4189ffb640 1 -- 172.21.15.77:0/3385549732 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174034070 con 0x7f418c15def0 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.540+0000 7f4189ffb640 1 -- 172.21.15.77:0/3385549732 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4174030610 con 0x7f418c15def0 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.540+0000 7f4189ffb640 1 -- 172.21.15.77:0/3385549732 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41740309e0 con 0x7f418c15def0 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.541+0000 7f41912c2640 1 -- 172.21.15.77:0/3385549732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 msgr2=0x7f418c0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.541+0000 7f41912c2640 1 --2- 172.21.15.77:0/3385549732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 0x7f418c0b41b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f41740097a0 tx=0x7f417402b960 comp rx=0 tx=0).stop 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.541+0000 7f41912c2640 1 -- 172.21.15.77:0/3385549732 shutdown_connections 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.541+0000 7f41912c2640 1 --2- 172.21.15.77:0/3385549732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c0cea50 0x7f418c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.541+0000 7f41912c2640 1 --2- 172.21.15.77:0/3385549732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 0x7f418c0b41b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.541+0000 7f41912c2640 1 -- 172.21.15.77:0/3385549732 >> 172.21.15.77:0/3385549732 conn(0x7f418c0b3690 msgr2=0x7f418c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.541+0000 7f41912c2640 1 -- 172.21.15.77:0/3385549732 shutdown_connections 2024-03-06T07:24:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.541+0000 7f41912c2640 1 -- 172.21.15.77:0/3385549732 wait complete. 2024-03-06T07:24:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.543+0000 7f41912c2640 1 Processor -- start 2024-03-06T07:24:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.543+0000 7f41912c2640 1 -- start start 2024-03-06T07:24:12.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.544+0000 7f41912c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c06cab0 0x7f418c0cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.544+0000 7f41912c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f418c0cea50 0x7f418c0b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.544+0000 7f41912c2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 0x7f418c14daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.544+0000 7f41912c2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c15e2d0 con 0x7f418c0cea50 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.544+0000 7f41912c2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0cc3f0 con 0x7f418c15def0 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.544+0000 7f41912c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c15e450 con 0x7f418c06cab0 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.544+0000 7f418a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f418c0cea50 0x7f418c0b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f418c0cea50 0x7f418c0b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56762/0 (socket says 172.21.15.77:56762) 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418a7fc640 1 -- 172.21.15.77:0/2073359470 learned_addr learned my addr 172.21.15.77:0/2073359470 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418affd640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c06cab0 0x7f418c0cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418b7fe640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 0x7f418c14daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418a7fc640 1 -- 172.21.15.77:0/2073359470 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c06cab0 msgr2=0x7f418c0cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418a7fc640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c06cab0 0x7f418c0cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418a7fc640 1 -- 172.21.15.77:0/2073359470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 msgr2=0x7f418c14daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418a7fc640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 0x7f418c14daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418a7fc640 1 -- 172.21.15.77:0/2073359470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0c9020 con 0x7f418c0cea50 2024-03-06T07:24:12.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f418a7fc640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f418c0cea50 0x7f418c0b5f30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f417c00c460 tx=0x7f417c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f41837fe640 1 -- 172.21.15.77:0/2073359470 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c017070 con 0x7f418c0cea50 2024-03-06T07:24:12.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f41837fe640 1 -- 172.21.15.77:0/2073359470 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c005ce0 con 0x7f418c0cea50 2024-03-06T07:24:12.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c0c9300 con 0x7f418c0cea50 2024-03-06T07:24:12.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f41837fe640 1 -- 172.21.15.77:0/2073359470 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c007290 con 0x7f418c0cea50 2024-03-06T07:24:12.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.545+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c0c98c0 con 0x7f418c0cea50 2024-03-06T07:24:12.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.546+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c09fcf0 con 0x7f418c0cea50 2024-03-06T07:24:12.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.547+0000 7f41837fe640 1 -- 172.21.15.77:0/2073359470 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f417c01a070 con 0x7f418c0cea50 2024-03-06T07:24:12.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.547+0000 7f41837fe640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f416c06b4d0 0x7f416c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.547+0000 7f41837fe640 1 -- 172.21.15.77:0/2073359470 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f417c089500 con 0x7f418c0cea50 2024-03-06T07:24:12.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.547+0000 7f418affd640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f416c06b4d0 0x7f416c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.548+0000 7f418affd640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f416c06b4d0 0x7f416c06d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4174036000 tx=0x7f417403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.551+0000 7f41837fe640 1 -- 172.21.15.77:0/2073359470 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f417c057d30 con 0x7f418c0cea50 2024-03-06T07:24:12.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:12.762+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f418c068ff0 con 0x7f418c0cea50 2024-03-06T07:24:13.382 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-03-06T07:24:13.381+0000 7efe98b48640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.426+0000 7f41837fe640 1 -- 172.21.15.77:0/2073359470 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f417c05bdd0 con 0x7f418c0cea50 2024-03-06T07:24:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f416c06b4d0 msgr2=0x7f416c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f416c06b4d0 0x7f416c06d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4174036000 tx=0x7f417403b040 comp rx=0 tx=0).stop 2024-03-06T07:24:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f418c0cea50 msgr2=0x7f418c0b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f418c0cea50 0x7f418c0b5f30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f417c00c460 tx=0x7f417c00c920 comp rx=0 tx=0).stop 2024-03-06T07:24:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 shutdown_connections 2024-03-06T07:24:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f416c06b4d0 0x7f416c06d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f418c0cea50 0x7f418c0b5f30 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f418c15def0 0x7f418c14daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 --2- 172.21.15.77:0/2073359470 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f418c06cab0 0x7f418c0cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 >> 172.21.15.77:0/2073359470 conn(0x7f418c0b3690 msgr2=0x7f418c0b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 shutdown_connections 2024-03-06T07:24:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.428+0000 7f41912c2640 1 -- 172.21.15.77:0/2073359470 wait complete. 2024-03-06T07:24:13.441 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:13.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:13.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.541+0000 7f2c7c0b5640 1 Processor -- start 2024-03-06T07:24:13.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.542+0000 7f2c7c0b5640 1 -- start start 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c7c0b5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2c741686d0 0x7f2c740b4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c7c0b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c740cdeb0 0x7f2c7408b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c7c0b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c740b3710 con 0x7f2c740b49b0 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c7c0b5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c740b3fe0 con 0x7f2c741686d0 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c7c0b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7416be30 con 0x7f2c740cdeb0 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c79650640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2c740b49b0 0x7f2c740cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54928/0 (socket says 172.21.15.77:54928) 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c79650640 1 -- 172.21.15.77:0/780760461 learned_addr learned my addr 172.21.15.77:0/780760461 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554351896 0 0) 0x7f2c64001110 con 0x7f2c740b49b0 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c60003520 con 0x7f2c740b49b0 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c7a652640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c740cdeb0 0x7f2c7408b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.543+0000 7f2c79e51640 1 --2- 172.21.15.77:0/780760461 >> v2:172.21.15.158:3300/0 conn(0x7f2c741686d0 0x7f2c740b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3885243608 0 0) 0x7f2c60003520 con 0x7f2c740b49b0 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c740cdeb0 msgr2=0x7f2c7408b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c740cdeb0 0x7f2c7408b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 >> v2:172.21.15.158:3300/0 conn(0x7f2c741686d0 msgr2=0x7f2c740b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 --2- 172.21.15.77:0/780760461 >> v2:172.21.15.158:3300/0 conn(0x7f2c741686d0 0x7f2c740b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c740b3990 con 0x7f2c740b49b0 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f2c64001840 con 0x7f2c740b49b0 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 >> v1:172.21.15.77:6789/0 conn(0x7f2c740b49b0 legacy=0x7f2c740cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60003b80 0x7f2c60005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60006680 0x7f2c60008b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60009230 0x7f2c6000b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c740b3fe0 con 0x7f2c60006680 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7416be30 con 0x7f2c60003b80 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.556+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6000be70 con 0x7f2c60009230 2024-03-06T07:24:13.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.557+0000 7f2c79650640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60006680 0x7f2c60008b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.557+0000 7f2c79e51640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60003b80 0x7f2c60005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.557+0000 7f2c7a652640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60009230 0x7f2c6000b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.557+0000 7f2c79650640 1 -- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60009230 msgr2=0x7f2c6000b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.557+0000 7f2c79650640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60009230 0x7f2c6000b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.557+0000 7f2c79650640 1 -- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60003b80 msgr2=0x7f2c60005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.557+0000 7f2c79650640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60003b80 0x7f2c60005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.557+0000 7f2c79650640 1 -- 172.21.15.77:0/780760461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60003520 con 0x7f2c60006680 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c79650640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60006680 0x7f2c60008b10 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400c3b0 tx=0x7f2c60003810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf9b423f35fdf19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6400c0e0 con 0x7f2c60006680 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c64030070 con 0x7f2c60006680 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c78e4f640 1 -- 172.21.15.77:0/780760461 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c640058d0 con 0x7f2c60006680 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/780760461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60006680 msgr2=0x7f2c60008b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c7c0b5640 1 --2- 172.21.15.77:0/780760461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60006680 0x7f2c60008b10 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400c3b0 tx=0x7f2c60003810 comp rx=0 tx=0).stop 2024-03-06T07:24:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c7a652640 1 -- 172.21.15.77:0/780760461 reap_dead start 2024-03-06T07:24:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/780760461 shutdown_connections 2024-03-06T07:24:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/780760461 >> 172.21.15.77:0/780760461 conn(0x7f2c740cf8a0 msgr2=0x7f2c740cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/780760461 shutdown_connections 2024-03-06T07:24:13.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.558+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/780760461 wait complete. 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.559+0000 7f2c7c0b5640 1 Processor -- start 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c7c0b5640 1 -- start start 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c7c0b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60006680 0x7f2c74203d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c7c0b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c7415c630 0x7f2c74162f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c7c0b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c741586e0 0x7f2c741634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c7c0b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7408b310 con 0x7f2c7415c630 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c7c0b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74168ab0 con 0x7f2c60006680 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c7c0b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7408b490 con 0x7f2c741586e0 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c79650640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c7415c630 0x7f2c74162f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c79650640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c7415c630 0x7f2c74162f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56784/0 (socket says 172.21.15.77:56784) 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c79650640 1 -- 172.21.15.77:0/2389232678 learned_addr learned my addr 172.21.15.77:0/2389232678 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c79e51640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60006680 0x7f2c74203d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.560+0000 7f2c7a652640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c741586e0 0x7f2c741634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c79650640 1 -- 172.21.15.77:0/2389232678 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c741586e0 msgr2=0x7f2c741634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c79650640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c741586e0 0x7f2c741634a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c79650640 1 -- 172.21.15.77:0/2389232678 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60006680 msgr2=0x7f2c74203d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c79650640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60006680 0x7f2c74203d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c79650640 1 -- 172.21.15.77:0/2389232678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c740713b0 con 0x7f2c7415c630 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c79650640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c7415c630 0x7f2c74162f60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400c4d0 tx=0x7f2c740b3b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c6a7fc640 1 -- 172.21.15.77:0/2389232678 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64004bf0 con 0x7f2c7415c630 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c6a7fc640 1 -- 172.21.15.77:0/2389232678 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c64004d90 con 0x7f2c7415c630 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c74204370 con 0x7f2c7415c630 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c6a7fc640 1 -- 172.21.15.77:0/2389232678 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c640072b0 con 0x7f2c7415c630 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.561+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c74204930 con 0x7f2c7415c630 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.562+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7404f330 con 0x7f2c7415c630 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.563+0000 7f2c6a7fc640 1 -- 172.21.15.77:0/2389232678 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c64007450 con 0x7f2c7415c630 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.563+0000 7f2c6a7fc640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c5806b500 0x7f2c5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.563+0000 7f2c79e51640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c5806b500 0x7f2c5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.563+0000 7f2c6a7fc640 1 -- 172.21.15.77:0/2389232678 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c640abd60 con 0x7f2c7415c630 2024-03-06T07:24:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.563+0000 7f2c79e51640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c5806b500 0x7f2c5806d9c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2c70006fd0 tx=0x7f2c70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.567+0000 7f2c6a7fc640 1 -- 172.21.15.77:0/2389232678 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c740b3710 con 0x7f2c7415c630 2024-03-06T07:24:13.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.780+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c7408b930 con 0x7f2c7415c630 2024-03-06T07:24:13.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.780+0000 7f2c6a7fc640 1 -- 172.21.15.77:0/2389232678 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f2c640343d0 con 0x7f2c7415c630 2024-03-06T07:24:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:13.782 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.760628+0000","modified":"2024-03-06T07:24:12.764047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/251375557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":251375557},{"type":"v1","addr":"172.21.15.77:6837","nonce":251375557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-03-06T07:24:13.783 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c5806b500 msgr2=0x7f2c5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c5806b500 0x7f2c5806d9c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2c70006fd0 tx=0x7f2c70008040 comp rx=0 tx=0).stop 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c7415c630 msgr2=0x7f2c74162f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c7415c630 0x7f2c74162f60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400c4d0 tx=0x7f2c740b3b60 comp rx=0 tx=0).stop 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 shutdown_connections 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c5806b500 0x7f2c5806d9c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c7415c630 0x7f2c74162f60 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c741586e0 0x7f2c741634a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 --2- 172.21.15.77:0/2389232678 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c60006680 0x7f2c74203d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.782+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 >> 172.21.15.77:0/2389232678 conn(0x7f2c740cf8a0 msgr2=0x7f2c740ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.783+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 shutdown_connections 2024-03-06T07:24:13.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.783+0000 7f2c7c0b5640 1 -- 172.21.15.77:0/2389232678 wait complete. 2024-03-06T07:24:13.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:13.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.895+0000 7f31becaf640 1 Processor -- start 2024-03-06T07:24:13.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.895+0000 7f31becaf640 1 -- start start 2024-03-06T07:24:13.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.895+0000 7f31becaf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b40c0 0x7f31b80d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.895+0000 7f31becaf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f31b80b22f0 0x7f31b80cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.895+0000 7f31becaf640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b80d0020 con 0x7f31b815c320 2024-03-06T07:24:13.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.895+0000 7f31becaf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b80b1000 con 0x7f31b80b22f0 2024-03-06T07:24:13.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.895+0000 7f31becaf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b80b0d20 con 0x7f31b80b40c0 2024-03-06T07:24:13.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31bd24c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f31b80b22f0 0x7f31b80cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31b7fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b40c0 0x7f31b80d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31bca4b640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f31b815c320 0x7f31b80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44970/0 (socket says 172.21.15.77:44970) 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31bca4b640 1 -- 172.21.15.77:0/1423178054 learned_addr learned my addr 172.21.15.77:0/1423178054 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31bd24c640 1 -- 172.21.15.77:0/1423178054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b40c0 msgr2=0x7f31b80d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31bd24c640 1 --2- 172.21.15.77:0/1423178054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b40c0 0x7f31b80d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31bd24c640 1 -- 172.21.15.77:0/1423178054 >> v1:172.21.15.158:6789/0 conn(0x7f31b815c320 legacy=0x7f31b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31bd24c640 1 -- 172.21.15.77:0/1423178054 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b80cfa60 con 0x7f31b80b22f0 2024-03-06T07:24:13.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.896+0000 7f31b7fff640 1 --2- 172.21.15.77:0/1423178054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b40c0 0x7f31b80d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.897+0000 7f31bd24c640 1 --2- 172.21.15.77:0/1423178054 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b80b22f0 0x7f31b80cc620 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009d90 tx=0x7f31b002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e68dfb24cc9204ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.897+0000 7f31b77fe640 1 -- 172.21.15.77:0/1423178054 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0039070 con 0x7f31b80b22f0 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.897+0000 7f31b77fe640 1 -- 172.21.15.77:0/1423178054 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b002cb30 con 0x7f31b80b22f0 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.897+0000 7f31b77fe640 1 -- 172.21.15.77:0/1423178054 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0034440 con 0x7f31b80b22f0 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.898+0000 7f31becaf640 1 -- 172.21.15.77:0/1423178054 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b80b22f0 msgr2=0x7f31b80cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.898+0000 7f31becaf640 1 --2- 172.21.15.77:0/1423178054 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b80b22f0 0x7f31b80cc620 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009d90 tx=0x7f31b002bd80 comp rx=0 tx=0).stop 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.898+0000 7f31becaf640 1 -- 172.21.15.77:0/1423178054 shutdown_connections 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.898+0000 7f31becaf640 1 --2- 172.21.15.77:0/1423178054 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b80b22f0 0x7f31b80cc620 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.898+0000 7f31becaf640 1 --2- 172.21.15.77:0/1423178054 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b40c0 0x7f31b80d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.898+0000 7f31becaf640 1 -- 172.21.15.77:0/1423178054 >> 172.21.15.77:0/1423178054 conn(0x7f31b80b35a0 msgr2=0x7f31b80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.898+0000 7f31becaf640 1 -- 172.21.15.77:0/1423178054 shutdown_connections 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.898+0000 7f31becaf640 1 -- 172.21.15.77:0/1423178054 wait complete. 2024-03-06T07:24:13.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.899+0000 7f31becaf640 1 Processor -- start 2024-03-06T07:24:13.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.899+0000 7f31becaf640 1 -- start start 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31becaf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b22f0 0x7f31b8157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31becaf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31b80b40c0 0x7f31b80c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31becaf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b815c320 0x7f31b80b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31becaf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b80cda00 con 0x7f31b80b40c0 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31becaf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b80cd880 con 0x7f31b815c320 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31becaf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b80b2780 con 0x7f31b80b22f0 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31b80b40c0 0x7f31b80c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31b80b40c0 0x7f31b80c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56798/0 (socket says 172.21.15.77:56798) 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31b7fff640 1 -- 172.21.15.77:0/2648110404 learned_addr learned my addr 172.21.15.77:0/2648110404 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:13.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31bca4b640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b22f0 0x7f31b8157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31bd24c640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b815c320 0x7f31b80b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31b7fff640 1 -- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b22f0 msgr2=0x7f31b8157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31b7fff640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b22f0 0x7f31b8157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31b7fff640 1 -- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b815c320 msgr2=0x7f31b80b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31b7fff640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b815c320 0x7f31b80b68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31b7fff640 1 -- 172.21.15.77:0/2648110404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b80c8e50 con 0x7f31b80b40c0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.900+0000 7f31bd24c640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b815c320 0x7f31b80b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.901+0000 7f31b7fff640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31b80b40c0 0x7f31b80c7a20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f31a400e960 tx=0x7f31a400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.901+0000 7f31b57fa640 1 -- 172.21.15.77:0/2648110404 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4017070 con 0x7f31b80b40c0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.901+0000 7f31b57fa640 1 -- 172.21.15.77:0/2648110404 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a400ca00 con 0x7f31b80b40c0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.901+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b80c9130 con 0x7f31b80b40c0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.901+0000 7f31b57fa640 1 -- 172.21.15.77:0/2648110404 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a400cdd0 con 0x7f31b80b40c0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.901+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b80c96f0 con 0x7f31b80b40c0 2024-03-06T07:24:13.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.902+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b804f330 con 0x7f31b80b40c0 2024-03-06T07:24:13.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.902+0000 7f31b57fa640 1 -- 172.21.15.77:0/2648110404 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31a401a070 con 0x7f31b80b40c0 2024-03-06T07:24:13.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.903+0000 7f31b57fa640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f319c06b4d0 0x7f319c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.903+0000 7f31b57fa640 1 -- 172.21.15.77:0/2648110404 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31a408a3c0 con 0x7f31b80b40c0 2024-03-06T07:24:13.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.903+0000 7f31bca4b640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f319c06b4d0 0x7f319c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.903+0000 7f31bca4b640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f319c06b4d0 0x7f319c06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f31ac007fd0 tx=0x7f31ac007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:13.907+0000 7f31b57fa640 1 -- 172.21.15.77:0/2648110404 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31a4058bf0 con 0x7f31b80b40c0 2024-03-06T07:24:14.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.167+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31b80b3bd0 con 0x7f31b80b40c0 2024-03-06T07:24:14.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.168+0000 7f31b57fa640 1 -- 172.21.15.77:0/2648110404 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f31a405cc90 con 0x7f31b80b40c0 2024-03-06T07:24:14.170 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:14.171 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:12.416501+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:14.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f319c06b4d0 msgr2=0x7f319c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f319c06b4d0 0x7f319c06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f31ac007fd0 tx=0x7f31ac007e80 comp rx=0 tx=0).stop 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31b80b40c0 msgr2=0x7f31b80c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31b80b40c0 0x7f31b80c7a20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f31a400e960 tx=0x7f31a400ee20 comp rx=0 tx=0).stop 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 shutdown_connections 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f319c06b4d0 0x7f319c06d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31b80b40c0 0x7f31b80c7a20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31b815c320 0x7f31b80b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 --2- 172.21.15.77:0/2648110404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31b80b22f0 0x7f31b8157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 >> 172.21.15.77:0/2648110404 conn(0x7f31b80b35a0 msgr2=0x7f31b806ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 shutdown_connections 2024-03-06T07:24:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.170+0000 7f31becaf640 1 -- 172.21.15.77:0/2648110404 wait complete. 2024-03-06T07:24:14.187 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:14.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:14.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.287+0000 7fd9cdbf7640 1 Processor -- start 2024-03-06T07:24:14.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.287+0000 7fd9cdbf7640 1 -- start start 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9cdbf7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd9c80d00c0 0x7fd9c80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9cdbf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80cf850 0x7fd9c80cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9cdbf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9c80ce1b0 0x7fd9c80ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9cdbf7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c806ca90 con 0x7fd9c80ce1b0 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9cdbf7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c80d09e0 con 0x7fd9c80d00c0 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9cdbf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c815e380 con 0x7fd9c80cf850 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9c80ce1b0 0x7fd9c80ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9c80ce1b0 0x7fd9c80ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56800/0 (socket says 172.21.15.77:56800) 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c7fff640 1 -- 172.21.15.77:0/1782923329 learned_addr learned my addr 172.21.15.77:0/1782923329 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9beffd640 1 --2- 172.21.15.77:0/1782923329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80cf850 0x7fd9c80cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c77fe640 1 --2- 172.21.15.77:0/1782923329 >> v2:172.21.15.158:3300/0 conn(0x7fd9c80d00c0 0x7fd9c80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c7fff640 1 -- 172.21.15.77:0/1782923329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80cf850 msgr2=0x7fd9c80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c7fff640 1 --2- 172.21.15.77:0/1782923329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80cf850 0x7fd9c80cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c7fff640 1 -- 172.21.15.77:0/1782923329 >> v2:172.21.15.158:3300/0 conn(0x7fd9c80d00c0 msgr2=0x7fd9c80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c7fff640 1 --2- 172.21.15.77:0/1782923329 >> v2:172.21.15.158:3300/0 conn(0x7fd9c80d00c0 0x7fd9c80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.288+0000 7fd9c7fff640 1 -- 172.21.15.77:0/1782923329 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c80cdc10 con 0x7fd9c80ce1b0 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.289+0000 7fd9c7fff640 1 --2- 172.21.15.77:0/1782923329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80ce1b0 0x7fd9c80ce640 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80d1a40 tx=0x7fd9c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ceb0c69e365b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.289+0000 7fd9c6ffd640 1 -- 172.21.15.77:0/1782923329 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c002ca70 con 0x7fd9c80ce1b0 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.289+0000 7fd9c6ffd640 1 -- 172.21.15.77:0/1782923329 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c002cc10 con 0x7fd9c80ce1b0 2024-03-06T07:24:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.289+0000 7fd9c6ffd640 1 -- 172.21.15.77:0/1782923329 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00343f0 con 0x7fd9c80ce1b0 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/1782923329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80ce1b0 msgr2=0x7fd9c80ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/1782923329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80ce1b0 0x7fd9c80ce640 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80d1a40 tx=0x7fd9c002bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/1782923329 shutdown_connections 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/1782923329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80ce1b0 0x7fd9c80ce640 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/1782923329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80cf850 0x7fd9c80cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/1782923329 >> v2:172.21.15.158:3300/0 conn(0x7fd9c80d00c0 0x7fd9c80d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/1782923329 >> 172.21.15.77:0/1782923329 conn(0x7fd9c80b35a0 msgr2=0x7fd9c80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/1782923329 shutdown_connections 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.290+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/1782923329 wait complete. 2024-03-06T07:24:14.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.291+0000 7fd9cdbf7640 1 Processor -- start 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.291+0000 7fd9cdbf7640 1 -- start start 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9cdbf7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c80ce1b0 0x7fd9c815d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9cdbf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80cf850 0x7fd9c80c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9cdbf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80d00c0 0x7fd9c80c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9cdbf7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c806c720 con 0x7fd9c80cf850 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9cdbf7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c806c440 con 0x7fd9c80ce1b0 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9cdbf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c80cd900 con 0x7fd9c80d00c0 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80cf850 0x7fd9c80c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80cf850 0x7fd9c80c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56814/0 (socket says 172.21.15.77:56814) 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 -- 172.21.15.77:0/2255495137 learned_addr learned my addr 172.21.15.77:0/2255495137 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 -- 172.21.15.77:0/2255495137 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80d00c0 msgr2=0x7fd9c80c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80d00c0 0x7fd9c80c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 -- 172.21.15.77:0/2255495137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c80ce1b0 msgr2=0x7fd9c815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c80ce1b0 0x7fd9c815d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 -- 172.21.15.77:0/2255495137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c80c90b0 con 0x7fd9c80cf850 2024-03-06T07:24:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.292+0000 7fd9beffd640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80cf850 0x7fd9c80c79e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000d800 tx=0x7fd9b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.293+0000 7fd9c4ff9640 1 -- 172.21.15.77:0/2255495137 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b000b6c0 con 0x7fd9c80cf850 2024-03-06T07:24:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.293+0000 7fd9c4ff9640 1 -- 172.21.15.77:0/2255495137 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b0019070 con 0x7fd9c80cf850 2024-03-06T07:24:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.293+0000 7fd9c4ff9640 1 -- 172.21.15.77:0/2255495137 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b000ba80 con 0x7fd9c80cf850 2024-03-06T07:24:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.293+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c81f7330 con 0x7fd9c80cf850 2024-03-06T07:24:14.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.293+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c81f78f0 con 0x7fd9c80cf850 2024-03-06T07:24:14.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.294+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c804f330 con 0x7fd9c80cf850 2024-03-06T07:24:14.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.294+0000 7fd9c4ff9640 1 -- 172.21.15.77:0/2255495137 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9b000bc20 con 0x7fd9c80cf850 2024-03-06T07:24:14.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.294+0000 7fd9c4ff9640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd9a806b4d0 0x7fd9a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.295+0000 7fd9c4ff9640 1 -- 172.21.15.77:0/2255495137 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9b008a620 con 0x7fd9c80cf850 2024-03-06T07:24:14.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.295+0000 7fd9c77fe640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd9a806b4d0 0x7fd9a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.295+0000 7fd9c77fe640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd9a806b4d0 0x7fd9a806d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800b7f0 tx=0x7fd9b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.303+0000 7fd9c4ff9640 1 -- 172.21.15.77:0/2255495137 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9b0059530 con 0x7fd9c80cf850 2024-03-06T07:24:14.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.477+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9c8070890 con 0x7fd9a806b4d0 2024-03-06T07:24:14.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.483+0000 7fd9c4ff9640 1 -- 172.21.15.77:0/2255495137 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251888 (secure 0 0 0) 0x7fd9c8070890 con 0x7fd9a806b4d0 2024-03-06T07:24:14.488 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:14.490 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:24:14.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd9a806b4d0 msgr2=0x7fd9a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd9a806b4d0 0x7fd9a806d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800b7f0 tx=0x7fd9b8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:14.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80cf850 msgr2=0x7fd9c80c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80cf850 0x7fd9c80c79e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000d800 tx=0x7fd9b000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 shutdown_connections 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd9a806b4d0 0x7fd9a806d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9c80cf850 0x7fd9c80c79e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9c80d00c0 0x7fd9c80c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 --2- 172.21.15.77:0/2255495137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9c80ce1b0 0x7fd9c815d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 >> 172.21.15.77:0/2255495137 conn(0x7fd9c80b35a0 msgr2=0x7fd9c8070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 shutdown_connections 2024-03-06T07:24:14.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.486+0000 7fd9cdbf7640 1 -- 172.21.15.77:0/2255495137 wait complete. 2024-03-06T07:24:14.499 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-06T07:24:13.408852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268548,"kb_used_data":6568,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754706172,"statfs":{"total":773094113280,"available":772819120128,"internally_reserved":0,"allocated":6725632,"data_stored":3447172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008121"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.442867+0000","last_change":"2024-03-06T07:24:11.435832+0000","last_active":"2024-03-06T07:24:11.442867+0000","last_peered":"2024-03-06T07:24:11.442867+0000","last_clean":"2024-03-06T07:24:11.442867+0000","last_became_active":"2024-03-06T07:24:11.435525+0000","last_became_peered":"2024-03-06T07:24:11.435525+0000","last_unstale":"2024-03-06T07:24:11.442867+0000","last_undegraded":"2024-03-06T07:24:11.442867+0000","last_fullsized":"2024-03-06T07:24:11.442867+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:38.279431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437713+0000","last_change":"2024-03-06T07:24:11.432423+0000","last_active":"2024-03-06T07:24:11.437713+0000","last_peered":"2024-03-06T07:24:11.437713+0000","last_clean":"2024-03-06T07:24:11.437713+0000","last_became_active":"2024-03-06T07:24:11.432124+0000","last_became_peered":"2024-03-06T07:24:11.432124+0000","last_unstale":"2024-03-06T07:24:11.437713+0000","last_undegraded":"2024-03-06T07:24:11.437713+0000","last_fullsized":"2024-03-06T07:24:11.437713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:08.308683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.431546+0000","last_change":"2024-03-06T07:24:11.429822+0000","last_active":"2024-03-06T07:24:11.431546+0000","last_peered":"2024-03-06T07:24:11.431546+0000","last_clean":"2024-03-06T07:24:11.431546+0000","last_became_active":"2024-03-06T07:24:11.429408+0000","last_became_peered":"2024-03-06T07:24:11.429408+0000","last_unstale":"2024-03-06T07:24:11.431546+0000","last_undegraded":"2024-03-06T07:24:11.431546+0000","last_fullsized":"2024-03-06T07:24:11.431546+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:28.743853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430799+0000","last_change":"2024-03-06T07:24:11.425461+0000","last_active":"2024-03-06T07:24:11.430799+0000","last_peered":"2024-03-06T07:24:11.430799+0000","last_clean":"2024-03-06T07:24:11.430799+0000","last_became_active":"2024-03-06T07:24:11.425197+0000","last_became_peered":"2024-03-06T07:24:11.425197+0000","last_unstale":"2024-03-06T07:24:11.430799+0000","last_undegraded":"2024-03-06T07:24:11.430799+0000","last_fullsized":"2024-03-06T07:24:11.430799+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:36.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429164+0000","last_change":"2024-03-06T07:24:11.424056+0000","last_active":"2024-03-06T07:24:11.429164+0000","last_peered":"2024-03-06T07:24:11.429164+0000","last_clean":"2024-03-06T07:24:11.429164+0000","last_became_active":"2024-03-06T07:24:11.423550+0000","last_became_peered":"2024-03-06T07:24:11.423550+0000","last_unstale":"2024-03-06T07:24:11.429164+0000","last_undegraded":"2024-03-06T07:24:11.429164+0000","last_fullsized":"2024-03-06T07:24:11.429164+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:56.554849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429086+0000","last_change":"2024-03-06T07:24:11.426715+0000","last_active":"2024-03-06T07:24:11.429086+0000","last_peered":"2024-03-06T07:24:11.429086+0000","last_clean":"2024-03-06T07:24:11.429086+0000","last_became_active":"2024-03-06T07:24:11.426430+0000","last_became_peered":"2024-03-06T07:24:11.426430+0000","last_unstale":"2024-03-06T07:24:11.429086+0000","last_undegraded":"2024-03-06T07:24:11.429086+0000","last_fullsized":"2024-03-06T07:24:11.429086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:28.743853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429178+0000","last_change":"2024-03-06T07:24:11.422112+0000","last_active":"2024-03-06T07:24:11.429178+0000","last_peered":"2024-03-06T07:24:11.429178+0000","last_clean":"2024-03-06T07:24:11.429178+0000","last_became_active":"2024-03-06T07:24:11.421821+0000","last_became_peered":"2024-03-06T07:24:11.421821+0000","last_unstale":"2024-03-06T07:24:11.429178+0000","last_undegraded":"2024-03-06T07:24:11.429178+0000","last_fullsized":"2024-03-06T07:24:11.429178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:04.327405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437533+0000","last_change":"2024-03-06T07:24:11.432026+0000","last_active":"2024-03-06T07:24:11.437533+0000","last_peered":"2024-03-06T07:24:11.437533+0000","last_clean":"2024-03-06T07:24:11.437533+0000","last_became_active":"2024-03-06T07:24:11.426164+0000","last_became_peered":"2024-03-06T07:24:11.426164+0000","last_unstale":"2024-03-06T07:24:11.437533+0000","last_undegraded":"2024-03-06T07:24:11.437533+0000","last_fullsized":"2024-03-06T07:24:11.437533+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:46.409586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425849+0000","last_change":"2024-03-06T07:24:11.420846+0000","last_active":"2024-03-06T07:24:11.425849+0000","last_peered":"2024-03-06T07:24:11.425849+0000","last_clean":"2024-03-06T07:24:11.425849+0000","last_became_active":"2024-03-06T07:24:11.420527+0000","last_became_peered":"2024-03-06T07:24:11.420527+0000","last_unstale":"2024-03-06T07:24:11.425849+0000","last_undegraded":"2024-03-06T07:24:11.425849+0000","last_fullsized":"2024-03-06T07:24:11.425849+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:40.719413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.422204+0000","last_change":"2024-03-06T07:24:11.417417+0000","last_active":"2024-03-06T07:24:11.422204+0000","last_peered":"2024-03-06T07:24:11.422204+0000","last_clean":"2024-03-06T07:24:11.422204+0000","last_became_active":"2024-03-06T07:24:11.417151+0000","last_became_peered":"2024-03-06T07:24:11.417151+0000","last_unstale":"2024-03-06T07:24:11.422204+0000","last_undegraded":"2024-03-06T07:24:11.422204+0000","last_fullsized":"2024-03-06T07:24:11.422204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:19.833659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432126+0000","last_change":"2024-03-06T07:24:11.426440+0000","last_active":"2024-03-06T07:24:11.432126+0000","last_peered":"2024-03-06T07:24:11.432126+0000","last_clean":"2024-03-06T07:24:11.432126+0000","last_became_active":"2024-03-06T07:24:11.426193+0000","last_became_peered":"2024-03-06T07:24:11.426193+0000","last_unstale":"2024-03-06T07:24:11.432126+0000","last_undegraded":"2024-03-06T07:24:11.432126+0000","last_fullsized":"2024-03-06T07:24:11.432126+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:12.524885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425010+0000","last_change":"2024-03-06T07:24:11.423393+0000","last_active":"2024-03-06T07:24:11.425010+0000","last_peered":"2024-03-06T07:24:11.425010+0000","last_clean":"2024-03-06T07:24:11.425010+0000","last_became_active":"2024-03-06T07:24:11.422350+0000","last_became_peered":"2024-03-06T07:24:11.422350+0000","last_unstale":"2024-03-06T07:24:11.425010+0000","last_undegraded":"2024-03-06T07:24:11.425010+0000","last_fullsized":"2024-03-06T07:24:11.425010+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:10.247025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.424941+0000","last_change":"2024-03-06T07:24:11.423245+0000","last_active":"2024-03-06T07:24:11.424941+0000","last_peered":"2024-03-06T07:24:11.424941+0000","last_clean":"2024-03-06T07:24:11.424941+0000","last_became_active":"2024-03-06T07:24:11.422831+0000","last_became_peered":"2024-03-06T07:24:11.422831+0000","last_unstale":"2024-03-06T07:24:11.424941+0000","last_undegraded":"2024-03-06T07:24:11.424941+0000","last_fullsized":"2024-03-06T07:24:11.424941+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:09.403092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429112+0000","last_change":"2024-03-06T07:24:11.424127+0000","last_active":"2024-03-06T07:24:11.429112+0000","last_peered":"2024-03-06T07:24:11.429112+0000","last_clean":"2024-03-06T07:24:11.429112+0000","last_became_active":"2024-03-06T07:24:11.423850+0000","last_became_peered":"2024-03-06T07:24:11.423850+0000","last_unstale":"2024-03-06T07:24:11.429112+0000","last_undegraded":"2024-03-06T07:24:11.429112+0000","last_fullsized":"2024-03-06T07:24:11.429112+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:36.438898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429204+0000","last_change":"2024-03-06T07:24:11.424747+0000","last_active":"2024-03-06T07:24:11.429204+0000","last_peered":"2024-03-06T07:24:11.429204+0000","last_clean":"2024-03-06T07:24:11.429204+0000","last_became_active":"2024-03-06T07:24:11.423961+0000","last_became_peered":"2024-03-06T07:24:11.423961+0000","last_unstale":"2024-03-06T07:24:11.429204+0000","last_undegraded":"2024-03-06T07:24:11.429204+0000","last_fullsized":"2024-03-06T07:24:11.429204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:47.027064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.420232+0000","last_change":"2024-03-06T07:24:11.419413+0000","last_active":"2024-03-06T07:24:11.420232+0000","last_peered":"2024-03-06T07:24:11.420232+0000","last_clean":"2024-03-06T07:24:11.420232+0000","last_became_active":"2024-03-06T07:24:11.419171+0000","last_became_peered":"2024-03-06T07:24:11.419171+0000","last_unstale":"2024-03-06T07:24:11.420232+0000","last_undegraded":"2024-03-06T07:24:11.420232+0000","last_fullsized":"2024-03-06T07:24:11.420232+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:24.782748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434743+0000","last_change":"2024-03-06T07:24:11.427275+0000","last_active":"2024-03-06T07:24:11.434743+0000","last_peered":"2024-03-06T07:24:11.434743+0000","last_clean":"2024-03-06T07:24:11.434743+0000","last_became_active":"2024-03-06T07:24:11.427020+0000","last_became_peered":"2024-03-06T07:24:11.427020+0000","last_unstale":"2024-03-06T07:24:11.434743+0000","last_undegraded":"2024-03-06T07:24:11.434743+0000","last_fullsized":"2024-03-06T07:24:11.434743+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:05.100065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.431019+0000","last_change":"2024-03-06T07:24:11.428604+0000","last_active":"2024-03-06T07:24:11.431019+0000","last_peered":"2024-03-06T07:24:11.431019+0000","last_clean":"2024-03-06T07:24:11.431019+0000","last_became_active":"2024-03-06T07:24:11.428200+0000","last_became_peered":"2024-03-06T07:24:11.428200+0000","last_unstale":"2024-03-06T07:24:11.431019+0000","last_undegraded":"2024-03-06T07:24:11.431019+0000","last_fullsized":"2024-03-06T07:24:11.431019+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:54.799719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437553+0000","last_change":"2024-03-06T07:24:11.432581+0000","last_active":"2024-03-06T07:24:11.437553+0000","last_peered":"2024-03-06T07:24:11.437553+0000","last_clean":"2024-03-06T07:24:11.437553+0000","last_became_active":"2024-03-06T07:24:11.432303+0000","last_became_peered":"2024-03-06T07:24:11.432303+0000","last_unstale":"2024-03-06T07:24:11.437553+0000","last_undegraded":"2024-03-06T07:24:11.437553+0000","last_fullsized":"2024-03-06T07:24:11.437553+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:53.100384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434932+0000","last_change":"2024-03-06T07:24:11.428281+0000","last_active":"2024-03-06T07:24:11.434932+0000","last_peered":"2024-03-06T07:24:11.434932+0000","last_clean":"2024-03-06T07:24:11.434932+0000","last_became_active":"2024-03-06T07:24:11.427998+0000","last_became_peered":"2024-03-06T07:24:11.427998+0000","last_unstale":"2024-03-06T07:24:11.434932+0000","last_undegraded":"2024-03-06T07:24:11.434932+0000","last_fullsized":"2024-03-06T07:24:11.434932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:59.026745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423338+0000","last_change":"2024-03-06T07:24:11.422232+0000","last_active":"2024-03-06T07:24:11.423338+0000","last_peered":"2024-03-06T07:24:11.423338+0000","last_clean":"2024-03-06T07:24:11.423338+0000","last_became_active":"2024-03-06T07:24:11.421943+0000","last_became_peered":"2024-03-06T07:24:11.421943+0000","last_unstale":"2024-03-06T07:24:11.423338+0000","last_undegraded":"2024-03-06T07:24:11.423338+0000","last_fullsized":"2024-03-06T07:24:11.423338+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:25.046426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435001+0000","last_change":"2024-03-06T07:24:11.430560+0000","last_active":"2024-03-06T07:24:11.435001+0000","last_peered":"2024-03-06T07:24:11.435001+0000","last_clean":"2024-03-06T07:24:11.435001+0000","last_became_active":"2024-03-06T07:24:11.430292+0000","last_became_peered":"2024-03-06T07:24:11.430292+0000","last_unstale":"2024-03-06T07:24:11.435001+0000","last_undegraded":"2024-03-06T07:24:11.435001+0000","last_fullsized":"2024-03-06T07:24:11.435001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:12.846672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435734+0000","last_change":"2024-03-06T07:24:11.434276+0000","last_active":"2024-03-06T07:24:11.435734+0000","last_peered":"2024-03-06T07:24:11.435734+0000","last_clean":"2024-03-06T07:24:11.435734+0000","last_became_active":"2024-03-06T07:24:11.433993+0000","last_became_peered":"2024-03-06T07:24:11.433993+0000","last_unstale":"2024-03-06T07:24:11.435734+0000","last_undegraded":"2024-03-06T07:24:11.435734+0000","last_fullsized":"2024-03-06T07:24:11.435734+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:45.319669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.424136+0000","last_change":"2024-03-06T07:24:11.422288+0000","last_active":"2024-03-06T07:24:11.424136+0000","last_peered":"2024-03-06T07:24:11.424136+0000","last_clean":"2024-03-06T07:24:11.424136+0000","last_became_active":"2024-03-06T07:24:11.422024+0000","last_became_peered":"2024-03-06T07:24:11.422024+0000","last_unstale":"2024-03-06T07:24:11.424136+0000","last_undegraded":"2024-03-06T07:24:11.424136+0000","last_fullsized":"2024-03-06T07:24:11.424136+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:28.849111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429058+0000","last_change":"2024-03-06T07:24:11.427663+0000","last_active":"2024-03-06T07:24:11.429058+0000","last_peered":"2024-03-06T07:24:11.429058+0000","last_clean":"2024-03-06T07:24:11.429058+0000","last_became_active":"2024-03-06T07:24:11.427402+0000","last_became_peered":"2024-03-06T07:24:11.427402+0000","last_unstale":"2024-03-06T07:24:11.429058+0000","last_undegraded":"2024-03-06T07:24:11.429058+0000","last_fullsized":"2024-03-06T07:24:11.429058+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:06.120209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430261+0000","last_change":"2024-03-06T07:24:11.424554+0000","last_active":"2024-03-06T07:24:11.430261+0000","last_peered":"2024-03-06T07:24:11.430261+0000","last_clean":"2024-03-06T07:24:11.430261+0000","last_became_active":"2024-03-06T07:24:11.423360+0000","last_became_peered":"2024-03-06T07:24:11.423360+0000","last_unstale":"2024-03-06T07:24:11.430261+0000","last_undegraded":"2024-03-06T07:24:11.430261+0000","last_fullsized":"2024-03-06T07:24:11.430261+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:24.782748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425501+0000","last_change":"2024-03-06T07:24:11.423213+0000","last_active":"2024-03-06T07:24:11.425501+0000","last_peered":"2024-03-06T07:24:11.425501+0000","last_clean":"2024-03-06T07:24:11.425501+0000","last_became_active":"2024-03-06T07:24:11.422933+0000","last_became_peered":"2024-03-06T07:24:11.422933+0000","last_unstale":"2024-03-06T07:24:11.425501+0000","last_undegraded":"2024-03-06T07:24:11.425501+0000","last_fullsized":"2024-03-06T07:24:11.425501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:36.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429191+0000","last_change":"2024-03-06T07:24:11.423029+0000","last_active":"2024-03-06T07:24:11.429191+0000","last_peered":"2024-03-06T07:24:11.429191+0000","last_clean":"2024-03-06T07:24:11.429191+0000","last_became_active":"2024-03-06T07:24:11.422757+0000","last_became_peered":"2024-03-06T07:24:11.422757+0000","last_unstale":"2024-03-06T07:24:11.429191+0000","last_undegraded":"2024-03-06T07:24:11.429191+0000","last_fullsized":"2024-03-06T07:24:11.429191+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:03.276020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437991+0000","last_change":"2024-03-06T07:24:11.434259+0000","last_active":"2024-03-06T07:24:11.437991+0000","last_peered":"2024-03-06T07:24:11.437991+0000","last_clean":"2024-03-06T07:24:11.437991+0000","last_became_active":"2024-03-06T07:24:11.433670+0000","last_became_peered":"2024-03-06T07:24:11.433670+0000","last_unstale":"2024-03-06T07:24:11.437991+0000","last_undegraded":"2024-03-06T07:24:11.437991+0000","last_fullsized":"2024-03-06T07:24:11.437991+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:52.465571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432148+0000","last_change":"2024-03-06T07:24:11.429862+0000","last_active":"2024-03-06T07:24:11.432148+0000","last_peered":"2024-03-06T07:24:11.432148+0000","last_clean":"2024-03-06T07:24:11.432148+0000","last_became_active":"2024-03-06T07:24:11.429513+0000","last_became_peered":"2024-03-06T07:24:11.429513+0000","last_unstale":"2024-03-06T07:24:11.432148+0000","last_undegraded":"2024-03-06T07:24:11.432148+0000","last_fullsized":"2024-03-06T07:24:11.432148+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:35.886714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428891+0000","last_change":"2024-03-06T07:24:11.423793+0000","last_active":"2024-03-06T07:24:11.428891+0000","last_peered":"2024-03-06T07:24:11.428891+0000","last_clean":"2024-03-06T07:24:11.428891+0000","last_became_active":"2024-03-06T07:24:11.423542+0000","last_became_peered":"2024-03-06T07:24:11.423542+0000","last_unstale":"2024-03-06T07:24:11.428891+0000","last_undegraded":"2024-03-06T07:24:11.428891+0000","last_fullsized":"2024-03-06T07:24:11.428891+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:40.108890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437501+0000","last_change":"2024-03-06T07:24:11.433607+0000","last_active":"2024-03-06T07:24:11.437501+0000","last_peered":"2024-03-06T07:24:11.437501+0000","last_clean":"2024-03-06T07:24:11.437501+0000","last_became_active":"2024-03-06T07:24:11.433344+0000","last_became_peered":"2024-03-06T07:24:11.433344+0000","last_unstale":"2024-03-06T07:24:11.437501+0000","last_undegraded":"2024-03-06T07:24:11.437501+0000","last_fullsized":"2024-03-06T07:24:11.437501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:28.626770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428989+0000","last_change":"2024-03-06T07:24:11.426829+0000","last_active":"2024-03-06T07:24:11.428989+0000","last_peered":"2024-03-06T07:24:11.428989+0000","last_clean":"2024-03-06T07:24:11.428989+0000","last_became_active":"2024-03-06T07:24:11.426562+0000","last_became_peered":"2024-03-06T07:24:11.426562+0000","last_unstale":"2024-03-06T07:24:11.428989+0000","last_undegraded":"2024-03-06T07:24:11.428989+0000","last_fullsized":"2024-03-06T07:24:11.428989+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:47.024923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.431141+0000","last_change":"2024-03-06T07:24:10.421827+0000","last_active":"2024-03-06T07:24:11.431141+0000","last_peered":"2024-03-06T07:24:11.431141+0000","last_clean":"2024-03-06T07:24:11.431141+0000","last_became_active":"2024-03-06T07:24:10.421568+0000","last_became_peered":"2024-03-06T07:24:10.421568+0000","last_unstale":"2024-03-06T07:24:11.431141+0000","last_undegraded":"2024-03-06T07:24:11.431141+0000","last_fullsized":"2024-03-06T07:24:11.431141+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:19.231070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430140+0000","last_change":"2024-03-06T07:24:11.424371+0000","last_active":"2024-03-06T07:24:11.430140+0000","last_peered":"2024-03-06T07:24:11.430140+0000","last_clean":"2024-03-06T07:24:11.430140+0000","last_became_active":"2024-03-06T07:24:11.424085+0000","last_became_peered":"2024-03-06T07:24:11.424085+0000","last_unstale":"2024-03-06T07:24:11.430140+0000","last_undegraded":"2024-03-06T07:24:11.430140+0000","last_fullsized":"2024-03-06T07:24:11.430140+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:22.655505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435719+0000","last_change":"2024-03-06T07:24:10.423271+0000","last_active":"2024-03-06T07:24:11.435719+0000","last_peered":"2024-03-06T07:24:11.435719+0000","last_clean":"2024-03-06T07:24:11.435719+0000","last_became_active":"2024-03-06T07:24:10.422817+0000","last_became_peered":"2024-03-06T07:24:10.422817+0000","last_unstale":"2024-03-06T07:24:11.435719+0000","last_undegraded":"2024-03-06T07:24:11.435719+0000","last_fullsized":"2024-03-06T07:24:11.435719+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:38.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434817+0000","last_change":"2024-03-06T07:24:11.429802+0000","last_active":"2024-03-06T07:24:11.434817+0000","last_peered":"2024-03-06T07:24:11.434817+0000","last_clean":"2024-03-06T07:24:11.434817+0000","last_became_active":"2024-03-06T07:24:11.429323+0000","last_became_peered":"2024-03-06T07:24:11.429323+0000","last_unstale":"2024-03-06T07:24:11.434817+0000","last_undegraded":"2024-03-06T07:24:11.434817+0000","last_fullsized":"2024-03-06T07:24:11.434817+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:45.319669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432682+0000","last_change":"2024-03-06T07:24:10.425453+0000","last_active":"2024-03-06T07:24:11.432682+0000","last_peered":"2024-03-06T07:24:11.432682+0000","last_clean":"2024-03-06T07:24:11.432682+0000","last_became_active":"2024-03-06T07:24:10.425162+0000","last_became_peered":"2024-03-06T07:24:10.425162+0000","last_unstale":"2024-03-06T07:24:11.432682+0000","last_undegraded":"2024-03-06T07:24:11.432682+0000","last_fullsized":"2024-03-06T07:24:11.432682+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:44.871546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425536+0000","last_change":"2024-03-06T07:24:11.424501+0000","last_active":"2024-03-06T07:24:11.425536+0000","last_peered":"2024-03-06T07:24:11.425536+0000","last_clean":"2024-03-06T07:24:11.425536+0000","last_became_active":"2024-03-06T07:24:11.424235+0000","last_became_peered":"2024-03-06T07:24:11.424235+0000","last_unstale":"2024-03-06T07:24:11.425536+0000","last_undegraded":"2024-03-06T07:24:11.425536+0000","last_fullsized":"2024-03-06T07:24:11.425536+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:43.231154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435737+0000","last_change":"2024-03-06T07:24:10.422388+0000","last_active":"2024-03-06T07:24:11.435737+0000","last_peered":"2024-03-06T07:24:11.435737+0000","last_clean":"2024-03-06T07:24:11.435737+0000","last_became_active":"2024-03-06T07:24:10.422123+0000","last_became_peered":"2024-03-06T07:24:10.422123+0000","last_unstale":"2024-03-06T07:24:11.435737+0000","last_undegraded":"2024-03-06T07:24:11.435737+0000","last_fullsized":"2024-03-06T07:24:11.435737+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:58.063204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434954+0000","last_change":"2024-03-06T07:24:11.428108+0000","last_active":"2024-03-06T07:24:11.434954+0000","last_peered":"2024-03-06T07:24:11.434954+0000","last_clean":"2024-03-06T07:24:11.434954+0000","last_became_active":"2024-03-06T07:24:11.427547+0000","last_became_peered":"2024-03-06T07:24:11.427547+0000","last_unstale":"2024-03-06T07:24:11.434954+0000","last_undegraded":"2024-03-06T07:24:11.434954+0000","last_fullsized":"2024-03-06T07:24:11.434954+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:46.808499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437760+0000","last_change":"2024-03-06T07:24:10.420772+0000","last_active":"2024-03-06T07:24:11.437760+0000","last_peered":"2024-03-06T07:24:11.437760+0000","last_clean":"2024-03-06T07:24:11.437760+0000","last_became_active":"2024-03-06T07:24:10.420336+0000","last_became_peered":"2024-03-06T07:24:10.420336+0000","last_unstale":"2024-03-06T07:24:11.437760+0000","last_undegraded":"2024-03-06T07:24:11.437760+0000","last_fullsized":"2024-03-06T07:24:11.437760+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:46.842242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.440342+0000","last_change":"2024-03-06T07:24:11.434475+0000","last_active":"2024-03-06T07:24:11.440342+0000","last_peered":"2024-03-06T07:24:11.440342+0000","last_clean":"2024-03-06T07:24:11.440342+0000","last_became_active":"2024-03-06T07:24:11.433960+0000","last_became_peered":"2024-03-06T07:24:11.433960+0000","last_unstale":"2024-03-06T07:24:11.440342+0000","last_undegraded":"2024-03-06T07:24:11.440342+0000","last_fullsized":"2024-03-06T07:24:11.440342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:35.073901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428992+0000","last_change":"2024-03-06T07:24:10.412729+0000","last_active":"2024-03-06T07:24:11.428992+0000","last_peered":"2024-03-06T07:24:11.428992+0000","last_clean":"2024-03-06T07:24:11.428992+0000","last_became_active":"2024-03-06T07:24:10.412472+0000","last_became_peered":"2024-03-06T07:24:10.412472+0000","last_unstale":"2024-03-06T07:24:11.428992+0000","last_undegraded":"2024-03-06T07:24:11.428992+0000","last_fullsized":"2024-03-06T07:24:11.428992+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:58.063204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428600+0000","last_change":"2024-03-06T07:24:11.425388+0000","last_active":"2024-03-06T07:24:11.428600+0000","last_peered":"2024-03-06T07:24:11.428600+0000","last_clean":"2024-03-06T07:24:11.428600+0000","last_became_active":"2024-03-06T07:24:11.424948+0000","last_became_peered":"2024-03-06T07:24:11.424948+0000","last_unstale":"2024-03-06T07:24:11.428600+0000","last_undegraded":"2024-03-06T07:24:11.428600+0000","last_fullsized":"2024-03-06T07:24:11.428600+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:44.344838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.415117+0000","last_change":"2024-03-06T07:24:10.418951+0000","last_active":"2024-03-06T07:24:11.415117+0000","last_peered":"2024-03-06T07:24:11.415117+0000","last_clean":"2024-03-06T07:24:11.415117+0000","last_became_active":"2024-03-06T07:24:10.418659+0000","last_became_peered":"2024-03-06T07:24:10.418659+0000","last_unstale":"2024-03-06T07:24:11.415117+0000","last_undegraded":"2024-03-06T07:24:11.415117+0000","last_fullsized":"2024-03-06T07:24:11.415117+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:04.860899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435293+0000","last_change":"2024-03-06T07:24:11.427075+0000","last_active":"2024-03-06T07:24:11.435293+0000","last_peered":"2024-03-06T07:24:11.435293+0000","last_clean":"2024-03-06T07:24:11.435293+0000","last_became_active":"2024-03-06T07:24:11.426794+0000","last_became_peered":"2024-03-06T07:24:11.426794+0000","last_unstale":"2024-03-06T07:24:11.435293+0000","last_undegraded":"2024-03-06T07:24:11.435293+0000","last_fullsized":"2024-03-06T07:24:11.435293+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:16.489811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.410459+0000","last_change":"2024-03-06T07:24:10.416980+0000","last_active":"2024-03-06T07:24:11.410459+0000","last_peered":"2024-03-06T07:24:11.410459+0000","last_clean":"2024-03-06T07:24:11.410459+0000","last_became_active":"2024-03-06T07:24:10.416694+0000","last_became_peered":"2024-03-06T07:24:10.416694+0000","last_unstale":"2024-03-06T07:24:11.410459+0000","last_undegraded":"2024-03-06T07:24:11.410459+0000","last_fullsized":"2024-03-06T07:24:11.410459+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:11.523049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.427058+0000","last_change":"2024-03-06T07:24:11.423891+0000","last_active":"2024-03-06T07:24:11.427058+0000","last_peered":"2024-03-06T07:24:11.427058+0000","last_clean":"2024-03-06T07:24:11.427058+0000","last_became_active":"2024-03-06T07:24:11.423601+0000","last_became_peered":"2024-03-06T07:24:11.423601+0000","last_unstale":"2024-03-06T07:24:11.427058+0000","last_undegraded":"2024-03-06T07:24:11.427058+0000","last_fullsized":"2024-03-06T07:24:11.427058+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:05.754170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.419547+0000","last_change":"2024-03-06T07:24:10.425652+0000","last_active":"2024-03-06T07:24:11.419547+0000","last_peered":"2024-03-06T07:24:11.419547+0000","last_clean":"2024-03-06T07:24:11.419547+0000","last_became_active":"2024-03-06T07:24:10.425353+0000","last_became_peered":"2024-03-06T07:24:10.425353+0000","last_unstale":"2024-03-06T07:24:11.419547+0000","last_undegraded":"2024-03-06T07:24:11.419547+0000","last_fullsized":"2024-03-06T07:24:11.419547+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:31.665843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432103+0000","last_change":"2024-03-06T07:24:11.430568+0000","last_active":"2024-03-06T07:24:11.432103+0000","last_peered":"2024-03-06T07:24:11.432103+0000","last_clean":"2024-03-06T07:24:11.432103+0000","last_became_active":"2024-03-06T07:24:11.429116+0000","last_became_peered":"2024-03-06T07:24:11.429116+0000","last_unstale":"2024-03-06T07:24:11.432103+0000","last_undegraded":"2024-03-06T07:24:11.432103+0000","last_fullsized":"2024-03-06T07:24:11.432103+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:47.024923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.417386+0000","last_change":"2024-03-06T07:24:10.421332+0000","last_active":"2024-03-06T07:24:11.417386+0000","last_peered":"2024-03-06T07:24:11.417386+0000","last_clean":"2024-03-06T07:24:11.417386+0000","last_became_active":"2024-03-06T07:24:10.421063+0000","last_became_peered":"2024-03-06T07:24:10.421063+0000","last_unstale":"2024-03-06T07:24:11.417386+0000","last_undegraded":"2024-03-06T07:24:11.417386+0000","last_fullsized":"2024-03-06T07:24:11.417386+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:39.879733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423634+0000","last_change":"2024-03-06T07:24:11.418632+0000","last_active":"2024-03-06T07:24:11.423634+0000","last_peered":"2024-03-06T07:24:11.423634+0000","last_clean":"2024-03-06T07:24:11.423634+0000","last_became_active":"2024-03-06T07:24:11.418372+0000","last_became_peered":"2024-03-06T07:24:11.418372+0000","last_unstale":"2024-03-06T07:24:11.423634+0000","last_undegraded":"2024-03-06T07:24:11.423634+0000","last_fullsized":"2024-03-06T07:24:11.423634+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:39.876229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.438381+0000","last_change":"2024-03-06T07:24:10.426470+0000","last_active":"2024-03-06T07:24:11.438381+0000","last_peered":"2024-03-06T07:24:11.438381+0000","last_clean":"2024-03-06T07:24:11.438381+0000","last_became_active":"2024-03-06T07:24:10.426209+0000","last_became_peered":"2024-03-06T07:24:10.426209+0000","last_unstale":"2024-03-06T07:24:11.438381+0000","last_undegraded":"2024-03-06T07:24:11.438381+0000","last_fullsized":"2024-03-06T07:24:11.438381+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:09.245188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430737+0000","last_change":"2024-03-06T07:24:11.426807+0000","last_active":"2024-03-06T07:24:11.430737+0000","last_peered":"2024-03-06T07:24:11.430737+0000","last_clean":"2024-03-06T07:24:11.430737+0000","last_became_active":"2024-03-06T07:24:11.425815+0000","last_became_peered":"2024-03-06T07:24:11.425815+0000","last_unstale":"2024-03-06T07:24:11.430737+0000","last_undegraded":"2024-03-06T07:24:11.430737+0000","last_fullsized":"2024-03-06T07:24:11.430737+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:24.782748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.440527+0000","last_change":"2024-03-06T07:24:10.425734+0000","last_active":"2024-03-06T07:24:11.440527+0000","last_peered":"2024-03-06T07:24:11.440527+0000","last_clean":"2024-03-06T07:24:11.440527+0000","last_became_active":"2024-03-06T07:24:10.425352+0000","last_became_peered":"2024-03-06T07:24:10.425352+0000","last_unstale":"2024-03-06T07:24:11.440527+0000","last_undegraded":"2024-03-06T07:24:11.440527+0000","last_fullsized":"2024-03-06T07:24:11.440527+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:33.617363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430802+0000","last_change":"2024-03-06T07:24:11.426827+0000","last_active":"2024-03-06T07:24:11.430802+0000","last_peered":"2024-03-06T07:24:11.430802+0000","last_clean":"2024-03-06T07:24:11.430802+0000","last_became_active":"2024-03-06T07:24:11.426553+0000","last_became_peered":"2024-03-06T07:24:11.426553+0000","last_unstale":"2024-03-06T07:24:11.430802+0000","last_undegraded":"2024-03-06T07:24:11.430802+0000","last_fullsized":"2024-03-06T07:24:11.430802+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:43.231154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.443042+0000","last_change":"2024-03-06T07:24:10.427532+0000","last_active":"2024-03-06T07:24:11.443042+0000","last_peered":"2024-03-06T07:24:11.443042+0000","last_clean":"2024-03-06T07:24:11.443042+0000","last_became_active":"2024-03-06T07:24:10.427218+0000","last_became_peered":"2024-03-06T07:24:10.427218+0000","last_unstale":"2024-03-06T07:24:11.443042+0000","last_undegraded":"2024-03-06T07:24:11.443042+0000","last_fullsized":"2024-03-06T07:24:11.443042+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:44.317833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429675+0000","last_change":"2024-03-06T07:24:11.424481+0000","last_active":"2024-03-06T07:24:11.429675+0000","last_peered":"2024-03-06T07:24:11.429675+0000","last_clean":"2024-03-06T07:24:11.429675+0000","last_became_active":"2024-03-06T07:24:11.424215+0000","last_became_peered":"2024-03-06T07:24:11.424215+0000","last_unstale":"2024-03-06T07:24:11.429675+0000","last_undegraded":"2024-03-06T07:24:11.429675+0000","last_fullsized":"2024-03-06T07:24:11.429675+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:15.909456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.410374+0000","last_change":"2024-03-06T07:24:10.419249+0000","last_active":"2024-03-06T07:24:11.410374+0000","last_peered":"2024-03-06T07:24:11.410374+0000","last_clean":"2024-03-06T07:24:11.410374+0000","last_became_active":"2024-03-06T07:24:10.418976+0000","last_became_peered":"2024-03-06T07:24:10.418976+0000","last_unstale":"2024-03-06T07:24:11.410374+0000","last_undegraded":"2024-03-06T07:24:11.410374+0000","last_fullsized":"2024-03-06T07:24:11.410374+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:21.236770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.427450+0000","last_change":"2024-03-06T07:24:11.425410+0000","last_active":"2024-03-06T07:24:11.427450+0000","last_peered":"2024-03-06T07:24:11.427450+0000","last_clean":"2024-03-06T07:24:11.427450+0000","last_became_active":"2024-03-06T07:24:11.425059+0000","last_became_peered":"2024-03-06T07:24:11.425059+0000","last_unstale":"2024-03-06T07:24:11.427450+0000","last_undegraded":"2024-03-06T07:24:11.427450+0000","last_fullsized":"2024-03-06T07:24:11.427450+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:33.632896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.415268+0000","last_change":"2024-03-06T07:24:10.421385+0000","last_active":"2024-03-06T07:24:11.415268+0000","last_peered":"2024-03-06T07:24:11.415268+0000","last_clean":"2024-03-06T07:24:11.415268+0000","last_became_active":"2024-03-06T07:24:10.421122+0000","last_became_peered":"2024-03-06T07:24:10.421122+0000","last_unstale":"2024-03-06T07:24:11.415268+0000","last_undegraded":"2024-03-06T07:24:11.415268+0000","last_fullsized":"2024-03-06T07:24:11.415268+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:26.211315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435753+0000","last_change":"2024-03-06T07:24:11.434366+0000","last_active":"2024-03-06T07:24:11.435753+0000","last_peered":"2024-03-06T07:24:11.435753+0000","last_clean":"2024-03-06T07:24:11.435753+0000","last_became_active":"2024-03-06T07:24:11.434100+0000","last_became_peered":"2024-03-06T07:24:11.434100+0000","last_unstale":"2024-03-06T07:24:11.435753+0000","last_undegraded":"2024-03-06T07:24:11.435753+0000","last_fullsized":"2024-03-06T07:24:11.435753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:28.626770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.436835+0000","last_change":"2024-03-06T07:24:10.422051+0000","last_active":"2024-03-06T07:24:11.436835+0000","last_peered":"2024-03-06T07:24:11.436835+0000","last_clean":"2024-03-06T07:24:11.436835+0000","last_became_active":"2024-03-06T07:24:10.421782+0000","last_became_peered":"2024-03-06T07:24:10.421782+0000","last_unstale":"2024-03-06T07:24:11.436835+0000","last_undegraded":"2024-03-06T07:24:11.436835+0000","last_fullsized":"2024-03-06T07:24:11.436835+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:19.231070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.431045+0000","last_change":"2024-03-06T07:24:11.428771+0000","last_active":"2024-03-06T07:24:11.431045+0000","last_peered":"2024-03-06T07:24:11.431045+0000","last_clean":"2024-03-06T07:24:11.431045+0000","last_became_active":"2024-03-06T07:24:11.428425+0000","last_became_peered":"2024-03-06T07:24:11.428425+0000","last_unstale":"2024-03-06T07:24:11.431045+0000","last_undegraded":"2024-03-06T07:24:11.431045+0000","last_fullsized":"2024-03-06T07:24:11.431045+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:43.231154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.443048+0000","last_change":"2024-03-06T07:24:10.420499+0000","last_active":"2024-03-06T07:24:11.443048+0000","last_peered":"2024-03-06T07:24:11.443048+0000","last_clean":"2024-03-06T07:24:11.443048+0000","last_became_active":"2024-03-06T07:24:10.420247+0000","last_became_peered":"2024-03-06T07:24:10.420247+0000","last_unstale":"2024-03-06T07:24:11.443048+0000","last_undegraded":"2024-03-06T07:24:11.443048+0000","last_fullsized":"2024-03-06T07:24:11.443048+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:46.023087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423708+0000","last_change":"2024-03-06T07:24:11.422945+0000","last_active":"2024-03-06T07:24:11.423708+0000","last_peered":"2024-03-06T07:24:11.423708+0000","last_clean":"2024-03-06T07:24:11.423708+0000","last_became_active":"2024-03-06T07:24:11.422643+0000","last_became_peered":"2024-03-06T07:24:11.422643+0000","last_unstale":"2024-03-06T07:24:11.423708+0000","last_undegraded":"2024-03-06T07:24:11.423708+0000","last_fullsized":"2024-03-06T07:24:11.423708+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:32.667680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432864+0000","last_change":"2024-03-06T07:24:10.422477+0000","last_active":"2024-03-06T07:24:11.432864+0000","last_peered":"2024-03-06T07:24:11.432864+0000","last_clean":"2024-03-06T07:24:11.432864+0000","last_became_active":"2024-03-06T07:24:10.422208+0000","last_became_peered":"2024-03-06T07:24:10.422208+0000","last_unstale":"2024-03-06T07:24:11.432864+0000","last_undegraded":"2024-03-06T07:24:11.432864+0000","last_fullsized":"2024-03-06T07:24:11.432864+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:31.665843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434882+0000","last_change":"2024-03-06T07:24:11.427613+0000","last_active":"2024-03-06T07:24:11.434882+0000","last_peered":"2024-03-06T07:24:11.434882+0000","last_clean":"2024-03-06T07:24:11.434882+0000","last_became_active":"2024-03-06T07:24:11.427342+0000","last_became_peered":"2024-03-06T07:24:11.427342+0000","last_unstale":"2024-03-06T07:24:11.434882+0000","last_undegraded":"2024-03-06T07:24:11.434882+0000","last_fullsized":"2024-03-06T07:24:11.434882+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:27.823929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435810+0000","last_change":"2024-03-06T07:24:10.422632+0000","last_active":"2024-03-06T07:24:11.435810+0000","last_peered":"2024-03-06T07:24:11.435810+0000","last_clean":"2024-03-06T07:24:11.435810+0000","last_became_active":"2024-03-06T07:24:10.422213+0000","last_became_peered":"2024-03-06T07:24:10.422213+0000","last_unstale":"2024-03-06T07:24:11.435810+0000","last_undegraded":"2024-03-06T07:24:11.435810+0000","last_fullsized":"2024-03-06T07:24:11.435810+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:19.231070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430761+0000","last_change":"2024-03-06T07:24:11.427477+0000","last_active":"2024-03-06T07:24:11.430761+0000","last_peered":"2024-03-06T07:24:11.430761+0000","last_clean":"2024-03-06T07:24:11.430761+0000","last_became_active":"2024-03-06T07:24:11.427193+0000","last_became_peered":"2024-03-06T07:24:11.427193+0000","last_unstale":"2024-03-06T07:24:11.430761+0000","last_undegraded":"2024-03-06T07:24:11.430761+0000","last_fullsized":"2024-03-06T07:24:11.430761+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:36.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.418287+0000","last_change":"2024-03-06T07:24:10.420892+0000","last_active":"2024-03-06T07:24:11.418287+0000","last_peered":"2024-03-06T07:24:11.418287+0000","last_clean":"2024-03-06T07:24:11.418287+0000","last_became_active":"2024-03-06T07:24:10.420637+0000","last_became_peered":"2024-03-06T07:24:10.420637+0000","last_unstale":"2024-03-06T07:24:11.418287+0000","last_undegraded":"2024-03-06T07:24:11.418287+0000","last_fullsized":"2024-03-06T07:24:11.418287+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:24.044590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429783+0000","last_change":"2024-03-06T07:24:11.427212+0000","last_active":"2024-03-06T07:24:11.429783+0000","last_peered":"2024-03-06T07:24:11.429783+0000","last_clean":"2024-03-06T07:24:11.429783+0000","last_became_active":"2024-03-06T07:24:11.426411+0000","last_became_peered":"2024-03-06T07:24:11.426411+0000","last_unstale":"2024-03-06T07:24:11.429783+0000","last_undegraded":"2024-03-06T07:24:11.429783+0000","last_fullsized":"2024-03-06T07:24:11.429783+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:15.909456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437795+0000","last_change":"2024-03-06T07:24:10.422534+0000","last_active":"2024-03-06T07:24:11.437795+0000","last_peered":"2024-03-06T07:24:11.437795+0000","last_clean":"2024-03-06T07:24:11.437795+0000","last_became_active":"2024-03-06T07:24:10.422011+0000","last_became_peered":"2024-03-06T07:24:10.422011+0000","last_unstale":"2024-03-06T07:24:11.437795+0000","last_undegraded":"2024-03-06T07:24:11.437795+0000","last_fullsized":"2024-03-06T07:24:11.437795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:27.624933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.420209+0000","last_change":"2024-03-06T07:24:11.418776+0000","last_active":"2024-03-06T07:24:11.420209+0000","last_peered":"2024-03-06T07:24:11.420209+0000","last_clean":"2024-03-06T07:24:11.420209+0000","last_became_active":"2024-03-06T07:24:11.418026+0000","last_became_peered":"2024-03-06T07:24:11.418026+0000","last_unstale":"2024-03-06T07:24:11.420209+0000","last_undegraded":"2024-03-06T07:24:11.420209+0000","last_fullsized":"2024-03-06T07:24:11.420209+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:15.909456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.409849+0000","last_change":"2024-03-06T07:24:10.425123+0000","last_active":"2024-03-06T07:24:11.409849+0000","last_peered":"2024-03-06T07:24:11.409849+0000","last_clean":"2024-03-06T07:24:11.409849+0000","last_became_active":"2024-03-06T07:24:10.424854+0000","last_became_peered":"2024-03-06T07:24:10.424854+0000","last_unstale":"2024-03-06T07:24:11.409849+0000","last_undegraded":"2024-03-06T07:24:11.409849+0000","last_fullsized":"2024-03-06T07:24:11.409849+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:08.401255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429267+0000","last_change":"2024-03-06T07:24:11.425789+0000","last_active":"2024-03-06T07:24:11.429267+0000","last_peered":"2024-03-06T07:24:11.429267+0000","last_clean":"2024-03-06T07:24:11.429267+0000","last_became_active":"2024-03-06T07:24:11.425542+0000","last_became_peered":"2024-03-06T07:24:11.425542+0000","last_unstale":"2024-03-06T07:24:11.429267+0000","last_undegraded":"2024-03-06T07:24:11.429267+0000","last_fullsized":"2024-03-06T07:24:11.429267+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:35.886714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.414642+0000","last_change":"2024-03-06T07:24:10.421384+0000","last_active":"2024-03-06T07:24:11.414642+0000","last_peered":"2024-03-06T07:24:11.414642+0000","last_clean":"2024-03-06T07:24:11.414642+0000","last_became_active":"2024-03-06T07:24:10.421149+0000","last_became_peered":"2024-03-06T07:24:10.421149+0000","last_unstale":"2024-03-06T07:24:11.414642+0000","last_undegraded":"2024-03-06T07:24:11.414642+0000","last_fullsized":"2024-03-06T07:24:11.414642+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:32.631060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430281+0000","last_change":"2024-03-06T07:24:11.425511+0000","last_active":"2024-03-06T07:24:11.430281+0000","last_peered":"2024-03-06T07:24:11.430281+0000","last_clean":"2024-03-06T07:24:11.430281+0000","last_became_active":"2024-03-06T07:24:11.425242+0000","last_became_peered":"2024-03-06T07:24:11.425242+0000","last_unstale":"2024-03-06T07:24:11.430281+0000","last_undegraded":"2024-03-06T07:24:11.430281+0000","last_fullsized":"2024-03-06T07:24:11.430281+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:54.799719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.433553+0000","last_change":"2024-03-06T07:24:10.425758+0000","last_active":"2024-03-06T07:24:11.433553+0000","last_peered":"2024-03-06T07:24:11.433553+0000","last_clean":"2024-03-06T07:24:11.433553+0000","last_became_active":"2024-03-06T07:24:10.425489+0000","last_became_peered":"2024-03-06T07:24:10.425489+0000","last_unstale":"2024-03-06T07:24:11.433553+0000","last_undegraded":"2024-03-06T07:24:11.433553+0000","last_fullsized":"2024-03-06T07:24:11.433553+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:43.343002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428508+0000","last_change":"2024-03-06T07:24:11.424876+0000","last_active":"2024-03-06T07:24:11.428508+0000","last_peered":"2024-03-06T07:24:11.428508+0000","last_clean":"2024-03-06T07:24:11.428508+0000","last_became_active":"2024-03-06T07:24:11.424567+0000","last_became_peered":"2024-03-06T07:24:11.424567+0000","last_unstale":"2024-03-06T07:24:11.428508+0000","last_undegraded":"2024-03-06T07:24:11.428508+0000","last_fullsized":"2024-03-06T07:24:11.428508+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:25.046426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.418213+0000","last_change":"2024-03-06T07:24:10.420338+0000","last_active":"2024-03-06T07:24:11.418213+0000","last_peered":"2024-03-06T07:24:11.418213+0000","last_clean":"2024-03-06T07:24:11.418213+0000","last_became_active":"2024-03-06T07:24:10.419569+0000","last_became_peered":"2024-03-06T07:24:10.419569+0000","last_unstale":"2024-03-06T07:24:11.418213+0000","last_undegraded":"2024-03-06T07:24:11.418213+0000","last_fullsized":"2024-03-06T07:24:11.418213+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:09.245188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423668+0000","last_change":"2024-03-06T07:24:11.419306+0000","last_active":"2024-03-06T07:24:11.423668+0000","last_peered":"2024-03-06T07:24:11.423668+0000","last_clean":"2024-03-06T07:24:11.423668+0000","last_became_active":"2024-03-06T07:24:11.419000+0000","last_became_peered":"2024-03-06T07:24:11.419000+0000","last_unstale":"2024-03-06T07:24:11.423668+0000","last_undegraded":"2024-03-06T07:24:11.423668+0000","last_fullsized":"2024-03-06T07:24:11.423668+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:04.127474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.411812+0000","last_change":"2024-03-06T07:24:10.417309+0000","last_active":"2024-03-06T07:24:11.411812+0000","last_peered":"2024-03-06T07:24:11.411812+0000","last_clean":"2024-03-06T07:24:11.411812+0000","last_became_active":"2024-03-06T07:24:10.417037+0000","last_became_peered":"2024-03-06T07:24:10.417037+0000","last_unstale":"2024-03-06T07:24:11.411812+0000","last_undegraded":"2024-03-06T07:24:11.411812+0000","last_fullsized":"2024-03-06T07:24:11.411812+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:48.289131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425521+0000","last_change":"2024-03-06T07:24:11.424052+0000","last_active":"2024-03-06T07:24:11.425521+0000","last_peered":"2024-03-06T07:24:11.425521+0000","last_clean":"2024-03-06T07:24:11.425521+0000","last_became_active":"2024-03-06T07:24:11.423779+0000","last_became_peered":"2024-03-06T07:24:11.423779+0000","last_unstale":"2024-03-06T07:24:11.425521+0000","last_undegraded":"2024-03-06T07:24:11.425521+0000","last_fullsized":"2024-03-06T07:24:11.425521+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:54.799719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.415614+0000","last_change":"2024-03-06T07:24:10.419644+0000","last_active":"2024-03-06T07:24:11.415614+0000","last_peered":"2024-03-06T07:24:11.415614+0000","last_clean":"2024-03-06T07:24:11.415614+0000","last_became_active":"2024-03-06T07:24:10.418589+0000","last_became_peered":"2024-03-06T07:24:10.418589+0000","last_unstale":"2024-03-06T07:24:11.415614+0000","last_undegraded":"2024-03-06T07:24:11.415614+0000","last_fullsized":"2024-03-06T07:24:11.415614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:44.871546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434977+0000","last_change":"2024-03-06T07:24:11.428681+0000","last_active":"2024-03-06T07:24:11.434977+0000","last_peered":"2024-03-06T07:24:11.434977+0000","last_clean":"2024-03-06T07:24:11.434977+0000","last_became_active":"2024-03-06T07:24:11.427698+0000","last_became_peered":"2024-03-06T07:24:11.427698+0000","last_unstale":"2024-03-06T07:24:11.434977+0000","last_undegraded":"2024-03-06T07:24:11.434977+0000","last_fullsized":"2024-03-06T07:24:11.434977+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:23.177374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.409688+0000","last_change":"2024-03-06T07:24:10.424491+0000","last_active":"2024-03-06T07:24:11.409688+0000","last_peered":"2024-03-06T07:24:11.409688+0000","last_clean":"2024-03-06T07:24:11.409688+0000","last_became_active":"2024-03-06T07:24:10.424234+0000","last_became_peered":"2024-03-06T07:24:10.424234+0000","last_unstale":"2024-03-06T07:24:11.409688+0000","last_undegraded":"2024-03-06T07:24:11.409688+0000","last_fullsized":"2024-03-06T07:24:11.409688+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:27.847275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.436118+0000","last_change":"2024-03-06T07:24:11.435296+0000","last_active":"2024-03-06T07:24:11.436118+0000","last_peered":"2024-03-06T07:24:11.436118+0000","last_clean":"2024-03-06T07:24:11.436118+0000","last_became_active":"2024-03-06T07:24:11.434928+0000","last_became_peered":"2024-03-06T07:24:11.434928+0000","last_unstale":"2024-03-06T07:24:11.436118+0000","last_undegraded":"2024-03-06T07:24:11.436118+0000","last_fullsized":"2024-03-06T07:24:11.436118+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:16.500427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.412395+0000","last_change":"2024-03-06T07:24:10.419415+0000","last_active":"2024-03-06T07:24:11.412395+0000","last_peered":"2024-03-06T07:24:11.412395+0000","last_clean":"2024-03-06T07:24:11.412395+0000","last_became_active":"2024-03-06T07:24:10.419147+0000","last_became_peered":"2024-03-06T07:24:10.419147+0000","last_unstale":"2024-03-06T07:24:11.412395+0000","last_undegraded":"2024-03-06T07:24:11.412395+0000","last_fullsized":"2024-03-06T07:24:11.412395+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:18.598190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423189+0000","last_change":"2024-03-06T07:24:11.418454+0000","last_active":"2024-03-06T07:24:11.423189+0000","last_peered":"2024-03-06T07:24:11.423189+0000","last_clean":"2024-03-06T07:24:11.423189+0000","last_became_active":"2024-03-06T07:24:11.418157+0000","last_became_peered":"2024-03-06T07:24:11.418157+0000","last_unstale":"2024-03-06T07:24:11.423189+0000","last_undegraded":"2024-03-06T07:24:11.423189+0000","last_fullsized":"2024-03-06T07:24:11.423189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:10.283622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.409944+0000","last_change":"2024-03-06T07:24:10.426732+0000","last_active":"2024-03-06T07:24:11.409944+0000","last_peered":"2024-03-06T07:24:11.409944+0000","last_clean":"2024-03-06T07:24:11.409944+0000","last_became_active":"2024-03-06T07:24:10.426456+0000","last_became_peered":"2024-03-06T07:24:10.426456+0000","last_unstale":"2024-03-06T07:24:11.409944+0000","last_undegraded":"2024-03-06T07:24:11.409944+0000","last_fullsized":"2024-03-06T07:24:11.409944+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:13.192638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435271+0000","last_change":"2024-03-06T07:24:11.426613+0000","last_active":"2024-03-06T07:24:11.435271+0000","last_peered":"2024-03-06T07:24:11.435271+0000","last_clean":"2024-03-06T07:24:11.435271+0000","last_became_active":"2024-03-06T07:24:11.426063+0000","last_became_peered":"2024-03-06T07:24:11.426063+0000","last_unstale":"2024-03-06T07:24:11.435271+0000","last_undegraded":"2024-03-06T07:24:11.435271+0000","last_fullsized":"2024-03-06T07:24:11.435271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:18.624608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.416716+0000","last_change":"2024-03-06T07:24:10.421360+0000","last_active":"2024-03-06T07:24:11.416716+0000","last_peered":"2024-03-06T07:24:11.416716+0000","last_clean":"2024-03-06T07:24:11.416716+0000","last_became_active":"2024-03-06T07:24:10.421075+0000","last_became_peered":"2024-03-06T07:24:10.421075+0000","last_unstale":"2024-03-06T07:24:11.416716+0000","last_undegraded":"2024-03-06T07:24:11.416716+0000","last_fullsized":"2024-03-06T07:24:11.416716+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:15.498591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435049+0000","last_change":"2024-03-06T07:24:11.431571+0000","last_active":"2024-03-06T07:24:11.435049+0000","last_peered":"2024-03-06T07:24:11.435049+0000","last_clean":"2024-03-06T07:24:11.435049+0000","last_became_active":"2024-03-06T07:24:11.426693+0000","last_became_peered":"2024-03-06T07:24:11.426693+0000","last_unstale":"2024-03-06T07:24:11.435049+0000","last_undegraded":"2024-03-06T07:24:11.435049+0000","last_fullsized":"2024-03-06T07:24:11.435049+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:35.653124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.410126+0000","last_change":"2024-03-06T07:24:10.421129+0000","last_active":"2024-03-06T07:24:11.410126+0000","last_peered":"2024-03-06T07:24:11.410126+0000","last_clean":"2024-03-06T07:24:11.410126+0000","last_became_active":"2024-03-06T07:24:10.419860+0000","last_became_peered":"2024-03-06T07:24:10.419860+0000","last_unstale":"2024-03-06T07:24:11.410126+0000","last_undegraded":"2024-03-06T07:24:11.410126+0000","last_fullsized":"2024-03-06T07:24:11.410126+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:23.934612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":233,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.416219+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:24:11.416219+0000","last_peered":"2024-03-06T07:24:11.416219+0000","last_clean":"2024-03-06T07:24:11.416219+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:24:11.416219+0000","last_undegraded":"2024-03-06T07:24:11.416219+0000","last_fullsized":"2024-03-06T07:24:11.416219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36600,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335240,"statfs":{"total":96636764160,"available":96599285760,"internally_reserved":0,"allocated":1286144,"data_stored":873230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36016,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335824,"statfs":{"total":96636764160,"available":96599883776,"internally_reserved":0,"allocated":684032,"data_stored":282702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":283354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574878,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36028,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335812,"statfs":{"total":96636764160,"available":96599871488,"internally_reserved":0,"allocated":696320,"data_stored":284314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":282758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":282758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":688128,"data_stored":282792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":875264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:14.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:14.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.638+0000 7f8bf3bd0640 1 Processor -- start 2024-03-06T07:24:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.638+0000 7f8bf3bd0640 1 -- start start 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf3bd0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8bec15bb20 0x7f8bec0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf3bd0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 0x7f8bec0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf3bd0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8bec0ce4b0 0x7f8bec0ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf3bd0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bec161980 con 0x7f8bec0ce4b0 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf3bd0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bec161c60 con 0x7f8bec15bb20 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf3bd0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bec161f40 con 0x7f8bec0cfff0 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf216d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8bec0ce4b0 0x7f8bec0ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf216d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8bec0ce4b0 0x7f8bec0ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56816/0 (socket says 172.21.15.77:56816) 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.639+0000 7f8bf216d640 1 -- 172.21.15.77:0/1767892677 learned_addr learned my addr 172.21.15.77:0/1767892677 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.640+0000 7f8bf196c640 1 --2- 172.21.15.77:0/1767892677 >> v2:172.21.15.158:3300/0 conn(0x7f8bec15bb20 0x7f8bec0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.640+0000 7f8bf116b640 1 --2- 172.21.15.77:0/1767892677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 0x7f8bec0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.640+0000 7f8bf216d640 1 -- 172.21.15.77:0/1767892677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 msgr2=0x7f8bec0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.640+0000 7f8bf216d640 1 --2- 172.21.15.77:0/1767892677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 0x7f8bec0d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.640+0000 7f8bf216d640 1 -- 172.21.15.77:0/1767892677 >> v2:172.21.15.158:3300/0 conn(0x7f8bec15bb20 msgr2=0x7f8bec0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.640+0000 7f8bf216d640 1 --2- 172.21.15.77:0/1767892677 >> v2:172.21.15.158:3300/0 conn(0x7f8bec15bb20 0x7f8bec0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.640+0000 7f8bf216d640 1 -- 172.21.15.77:0/1767892677 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bec162220 con 0x7f8bec0ce4b0 2024-03-06T07:24:14.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.640+0000 7f8bf196c640 1 --2- 172.21.15.77:0/1767892677 >> v2:172.21.15.158:3300/0 conn(0x7f8bec15bb20 0x7f8bec0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:14.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf216d640 1 --2- 172.21.15.77:0/1767892677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0ce4b0 0x7f8bec0ce940 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0d2bc0 tx=0x7f8bdc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5b9db852346605 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf096a640 1 -- 172.21.15.77:0/1767892677 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc02ca70 con 0x7f8bec0ce4b0 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf096a640 1 -- 172.21.15.77:0/1767892677 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bdc02cc10 con 0x7f8bec0ce4b0 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf096a640 1 -- 172.21.15.77:0/1767892677 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc0343f0 con 0x7f8bec0ce4b0 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/1767892677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0ce4b0 msgr2=0x7f8bec0ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/1767892677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0ce4b0 0x7f8bec0ce940 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0d2bc0 tx=0x7f8bdc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/1767892677 shutdown_connections 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/1767892677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec0ce4b0 0x7f8bec0ce940 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/1767892677 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 0x7f8bec0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/1767892677 >> v2:172.21.15.158:3300/0 conn(0x7f8bec15bb20 0x7f8bec0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.641+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/1767892677 >> 172.21.15.77:0/1767892677 conn(0x7f8bec09a9f0 msgr2=0x7f8bec09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.642+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/1767892677 shutdown_connections 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.642+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/1767892677 wait complete. 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf3bd0640 1 Processor -- start 2024-03-06T07:24:14.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf3bd0640 1 -- start start 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf3bd0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bec0ce4b0 0x7f8bec06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf3bd0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 0x7f8bec06d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf3bd0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec15bb20 0x7f8bec15d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf3bd0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bec0d24b0 con 0x7f8bec15bb20 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf3bd0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bec0d2330 con 0x7f8bec0ce4b0 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf3bd0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bec0d1980 con 0x7f8bec0cfff0 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf216d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec15bb20 0x7f8bec15d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf216d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec15bb20 0x7f8bec15d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56820/0 (socket says 172.21.15.77:56820) 2024-03-06T07:24:14.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.643+0000 7f8bf216d640 1 -- 172.21.15.77:0/2790900174 learned_addr learned my addr 172.21.15.77:0/2790900174 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf196c640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bec0ce4b0 0x7f8bec06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf116b640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 0x7f8bec06d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf216d640 1 -- 172.21.15.77:0/2790900174 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 msgr2=0x7f8bec06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf216d640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 0x7f8bec06d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf216d640 1 -- 172.21.15.77:0/2790900174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bec0ce4b0 msgr2=0x7f8bec06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf216d640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bec0ce4b0 0x7f8bec06cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf216d640 1 -- 172.21.15.77:0/2790900174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bec1e7880 con 0x7f8bec15bb20 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf216d640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec15bb20 0x7f8bec15d6c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc036040 tx=0x7f8bdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bea7fc640 1 -- 172.21.15.77:0/2790900174 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc043020 con 0x7f8bec15bb20 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bea7fc640 1 -- 172.21.15.77:0/2790900174 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bdc03e450 con 0x7f8bec15bb20 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bec1e7b60 con 0x7f8bec15bb20 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bea7fc640 1 -- 172.21.15.77:0/2790900174 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc009510 con 0x7f8bec15bb20 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.644+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bec1e8120 con 0x7f8bec15bb20 2024-03-06T07:24:14.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.645+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bec0b0cb0 con 0x7f8bec15bb20 2024-03-06T07:24:14.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.646+0000 7f8bea7fc640 1 -- 172.21.15.77:0/2790900174 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bdc046020 con 0x7f8bec15bb20 2024-03-06T07:24:14.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.646+0000 7f8bea7fc640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bd806b4d0 0x7f8bd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.646+0000 7f8bf196c640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bd806b4d0 0x7f8bd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.646+0000 7f8bea7fc640 1 -- 172.21.15.77:0/2790900174 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bdc0acc50 con 0x7f8bec15bb20 2024-03-06T07:24:14.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.647+0000 7f8bf196c640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bd806b4d0 0x7f8bd806d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8bec06ddb0 tx=0x7f8be4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.651+0000 7f8bea7fc640 1 -- 172.21.15.77:0/2790900174 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bec0b0cb0 con 0x7f8bec15bb20 2024-03-06T07:24:14.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.819+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8bec04f330 con 0x7f8bd806b4d0 2024-03-06T07:24:14.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.825+0000 7f8bea7fc640 1 -- 172.21.15.77:0/2790900174 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251888 (secure 0 0 0) 0x7f8bec04f330 con 0x7f8bd806b4d0 2024-03-06T07:24:14.830 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:14.831 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:24:14.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bd806b4d0 msgr2=0x7f8bd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bd806b4d0 0x7f8bd806d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8bec06ddb0 tx=0x7f8be4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:14.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec15bb20 msgr2=0x7f8bec15d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec15bb20 0x7f8bec15d6c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc036040 tx=0x7f8bdc004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 shutdown_connections 2024-03-06T07:24:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bd806b4d0 0x7f8bd806d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bec15bb20 0x7f8bec15d6c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bec0cfff0 0x7f8bec06d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 --2- 172.21.15.77:0/2790900174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bec0ce4b0 0x7f8bec06cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 >> 172.21.15.77:0/2790900174 conn(0x7f8bec09a9f0 msgr2=0x7f8bec0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 shutdown_connections 2024-03-06T07:24:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.828+0000 7f8bf3bd0640 1 -- 172.21.15.77:0/2790900174 wait complete. 2024-03-06T07:24:14.841 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-06T07:24:13.408852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268548,"kb_used_data":6568,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754706172,"statfs":{"total":773094113280,"available":772819120128,"internally_reserved":0,"allocated":6725632,"data_stored":3447172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008121"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.442867+0000","last_change":"2024-03-06T07:24:11.435832+0000","last_active":"2024-03-06T07:24:11.442867+0000","last_peered":"2024-03-06T07:24:11.442867+0000","last_clean":"2024-03-06T07:24:11.442867+0000","last_became_active":"2024-03-06T07:24:11.435525+0000","last_became_peered":"2024-03-06T07:24:11.435525+0000","last_unstale":"2024-03-06T07:24:11.442867+0000","last_undegraded":"2024-03-06T07:24:11.442867+0000","last_fullsized":"2024-03-06T07:24:11.442867+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:38.279431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437713+0000","last_change":"2024-03-06T07:24:11.432423+0000","last_active":"2024-03-06T07:24:11.437713+0000","last_peered":"2024-03-06T07:24:11.437713+0000","last_clean":"2024-03-06T07:24:11.437713+0000","last_became_active":"2024-03-06T07:24:11.432124+0000","last_became_peered":"2024-03-06T07:24:11.432124+0000","last_unstale":"2024-03-06T07:24:11.437713+0000","last_undegraded":"2024-03-06T07:24:11.437713+0000","last_fullsized":"2024-03-06T07:24:11.437713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:08.308683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.431546+0000","last_change":"2024-03-06T07:24:11.429822+0000","last_active":"2024-03-06T07:24:11.431546+0000","last_peered":"2024-03-06T07:24:11.431546+0000","last_clean":"2024-03-06T07:24:11.431546+0000","last_became_active":"2024-03-06T07:24:11.429408+0000","last_became_peered":"2024-03-06T07:24:11.429408+0000","last_unstale":"2024-03-06T07:24:11.431546+0000","last_undegraded":"2024-03-06T07:24:11.431546+0000","last_fullsized":"2024-03-06T07:24:11.431546+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:28.743853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430799+0000","last_change":"2024-03-06T07:24:11.425461+0000","last_active":"2024-03-06T07:24:11.430799+0000","last_peered":"2024-03-06T07:24:11.430799+0000","last_clean":"2024-03-06T07:24:11.430799+0000","last_became_active":"2024-03-06T07:24:11.425197+0000","last_became_peered":"2024-03-06T07:24:11.425197+0000","last_unstale":"2024-03-06T07:24:11.430799+0000","last_undegraded":"2024-03-06T07:24:11.430799+0000","last_fullsized":"2024-03-06T07:24:11.430799+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:36.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429164+0000","last_change":"2024-03-06T07:24:11.424056+0000","last_active":"2024-03-06T07:24:11.429164+0000","last_peered":"2024-03-06T07:24:11.429164+0000","last_clean":"2024-03-06T07:24:11.429164+0000","last_became_active":"2024-03-06T07:24:11.423550+0000","last_became_peered":"2024-03-06T07:24:11.423550+0000","last_unstale":"2024-03-06T07:24:11.429164+0000","last_undegraded":"2024-03-06T07:24:11.429164+0000","last_fullsized":"2024-03-06T07:24:11.429164+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:56.554849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429086+0000","last_change":"2024-03-06T07:24:11.426715+0000","last_active":"2024-03-06T07:24:11.429086+0000","last_peered":"2024-03-06T07:24:11.429086+0000","last_clean":"2024-03-06T07:24:11.429086+0000","last_became_active":"2024-03-06T07:24:11.426430+0000","last_became_peered":"2024-03-06T07:24:11.426430+0000","last_unstale":"2024-03-06T07:24:11.429086+0000","last_undegraded":"2024-03-06T07:24:11.429086+0000","last_fullsized":"2024-03-06T07:24:11.429086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:28.743853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429178+0000","last_change":"2024-03-06T07:24:11.422112+0000","last_active":"2024-03-06T07:24:11.429178+0000","last_peered":"2024-03-06T07:24:11.429178+0000","last_clean":"2024-03-06T07:24:11.429178+0000","last_became_active":"2024-03-06T07:24:11.421821+0000","last_became_peered":"2024-03-06T07:24:11.421821+0000","last_unstale":"2024-03-06T07:24:11.429178+0000","last_undegraded":"2024-03-06T07:24:11.429178+0000","last_fullsized":"2024-03-06T07:24:11.429178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:04.327405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437533+0000","last_change":"2024-03-06T07:24:11.432026+0000","last_active":"2024-03-06T07:24:11.437533+0000","last_peered":"2024-03-06T07:24:11.437533+0000","last_clean":"2024-03-06T07:24:11.437533+0000","last_became_active":"2024-03-06T07:24:11.426164+0000","last_became_peered":"2024-03-06T07:24:11.426164+0000","last_unstale":"2024-03-06T07:24:11.437533+0000","last_undegraded":"2024-03-06T07:24:11.437533+0000","last_fullsized":"2024-03-06T07:24:11.437533+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:46.409586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425849+0000","last_change":"2024-03-06T07:24:11.420846+0000","last_active":"2024-03-06T07:24:11.425849+0000","last_peered":"2024-03-06T07:24:11.425849+0000","last_clean":"2024-03-06T07:24:11.425849+0000","last_became_active":"2024-03-06T07:24:11.420527+0000","last_became_peered":"2024-03-06T07:24:11.420527+0000","last_unstale":"2024-03-06T07:24:11.425849+0000","last_undegraded":"2024-03-06T07:24:11.425849+0000","last_fullsized":"2024-03-06T07:24:11.425849+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:40.719413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.422204+0000","last_change":"2024-03-06T07:24:11.417417+0000","last_active":"2024-03-06T07:24:11.422204+0000","last_peered":"2024-03-06T07:24:11.422204+0000","last_clean":"2024-03-06T07:24:11.422204+0000","last_became_active":"2024-03-06T07:24:11.417151+0000","last_became_peered":"2024-03-06T07:24:11.417151+0000","last_unstale":"2024-03-06T07:24:11.422204+0000","last_undegraded":"2024-03-06T07:24:11.422204+0000","last_fullsized":"2024-03-06T07:24:11.422204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:19.833659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432126+0000","last_change":"2024-03-06T07:24:11.426440+0000","last_active":"2024-03-06T07:24:11.432126+0000","last_peered":"2024-03-06T07:24:11.432126+0000","last_clean":"2024-03-06T07:24:11.432126+0000","last_became_active":"2024-03-06T07:24:11.426193+0000","last_became_peered":"2024-03-06T07:24:11.426193+0000","last_unstale":"2024-03-06T07:24:11.432126+0000","last_undegraded":"2024-03-06T07:24:11.432126+0000","last_fullsized":"2024-03-06T07:24:11.432126+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:12.524885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425010+0000","last_change":"2024-03-06T07:24:11.423393+0000","last_active":"2024-03-06T07:24:11.425010+0000","last_peered":"2024-03-06T07:24:11.425010+0000","last_clean":"2024-03-06T07:24:11.425010+0000","last_became_active":"2024-03-06T07:24:11.422350+0000","last_became_peered":"2024-03-06T07:24:11.422350+0000","last_unstale":"2024-03-06T07:24:11.425010+0000","last_undegraded":"2024-03-06T07:24:11.425010+0000","last_fullsized":"2024-03-06T07:24:11.425010+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:10.247025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.424941+0000","last_change":"2024-03-06T07:24:11.423245+0000","last_active":"2024-03-06T07:24:11.424941+0000","last_peered":"2024-03-06T07:24:11.424941+0000","last_clean":"2024-03-06T07:24:11.424941+0000","last_became_active":"2024-03-06T07:24:11.422831+0000","last_became_peered":"2024-03-06T07:24:11.422831+0000","last_unstale":"2024-03-06T07:24:11.424941+0000","last_undegraded":"2024-03-06T07:24:11.424941+0000","last_fullsized":"2024-03-06T07:24:11.424941+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:09.403092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429112+0000","last_change":"2024-03-06T07:24:11.424127+0000","last_active":"2024-03-06T07:24:11.429112+0000","last_peered":"2024-03-06T07:24:11.429112+0000","last_clean":"2024-03-06T07:24:11.429112+0000","last_became_active":"2024-03-06T07:24:11.423850+0000","last_became_peered":"2024-03-06T07:24:11.423850+0000","last_unstale":"2024-03-06T07:24:11.429112+0000","last_undegraded":"2024-03-06T07:24:11.429112+0000","last_fullsized":"2024-03-06T07:24:11.429112+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:36.438898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429204+0000","last_change":"2024-03-06T07:24:11.424747+0000","last_active":"2024-03-06T07:24:11.429204+0000","last_peered":"2024-03-06T07:24:11.429204+0000","last_clean":"2024-03-06T07:24:11.429204+0000","last_became_active":"2024-03-06T07:24:11.423961+0000","last_became_peered":"2024-03-06T07:24:11.423961+0000","last_unstale":"2024-03-06T07:24:11.429204+0000","last_undegraded":"2024-03-06T07:24:11.429204+0000","last_fullsized":"2024-03-06T07:24:11.429204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:47.027064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.420232+0000","last_change":"2024-03-06T07:24:11.419413+0000","last_active":"2024-03-06T07:24:11.420232+0000","last_peered":"2024-03-06T07:24:11.420232+0000","last_clean":"2024-03-06T07:24:11.420232+0000","last_became_active":"2024-03-06T07:24:11.419171+0000","last_became_peered":"2024-03-06T07:24:11.419171+0000","last_unstale":"2024-03-06T07:24:11.420232+0000","last_undegraded":"2024-03-06T07:24:11.420232+0000","last_fullsized":"2024-03-06T07:24:11.420232+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:24.782748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434743+0000","last_change":"2024-03-06T07:24:11.427275+0000","last_active":"2024-03-06T07:24:11.434743+0000","last_peered":"2024-03-06T07:24:11.434743+0000","last_clean":"2024-03-06T07:24:11.434743+0000","last_became_active":"2024-03-06T07:24:11.427020+0000","last_became_peered":"2024-03-06T07:24:11.427020+0000","last_unstale":"2024-03-06T07:24:11.434743+0000","last_undegraded":"2024-03-06T07:24:11.434743+0000","last_fullsized":"2024-03-06T07:24:11.434743+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:05.100065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.431019+0000","last_change":"2024-03-06T07:24:11.428604+0000","last_active":"2024-03-06T07:24:11.431019+0000","last_peered":"2024-03-06T07:24:11.431019+0000","last_clean":"2024-03-06T07:24:11.431019+0000","last_became_active":"2024-03-06T07:24:11.428200+0000","last_became_peered":"2024-03-06T07:24:11.428200+0000","last_unstale":"2024-03-06T07:24:11.431019+0000","last_undegraded":"2024-03-06T07:24:11.431019+0000","last_fullsized":"2024-03-06T07:24:11.431019+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:54.799719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437553+0000","last_change":"2024-03-06T07:24:11.432581+0000","last_active":"2024-03-06T07:24:11.437553+0000","last_peered":"2024-03-06T07:24:11.437553+0000","last_clean":"2024-03-06T07:24:11.437553+0000","last_became_active":"2024-03-06T07:24:11.432303+0000","last_became_peered":"2024-03-06T07:24:11.432303+0000","last_unstale":"2024-03-06T07:24:11.437553+0000","last_undegraded":"2024-03-06T07:24:11.437553+0000","last_fullsized":"2024-03-06T07:24:11.437553+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:53.100384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434932+0000","last_change":"2024-03-06T07:24:11.428281+0000","last_active":"2024-03-06T07:24:11.434932+0000","last_peered":"2024-03-06T07:24:11.434932+0000","last_clean":"2024-03-06T07:24:11.434932+0000","last_became_active":"2024-03-06T07:24:11.427998+0000","last_became_peered":"2024-03-06T07:24:11.427998+0000","last_unstale":"2024-03-06T07:24:11.434932+0000","last_undegraded":"2024-03-06T07:24:11.434932+0000","last_fullsized":"2024-03-06T07:24:11.434932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:59.026745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423338+0000","last_change":"2024-03-06T07:24:11.422232+0000","last_active":"2024-03-06T07:24:11.423338+0000","last_peered":"2024-03-06T07:24:11.423338+0000","last_clean":"2024-03-06T07:24:11.423338+0000","last_became_active":"2024-03-06T07:24:11.421943+0000","last_became_peered":"2024-03-06T07:24:11.421943+0000","last_unstale":"2024-03-06T07:24:11.423338+0000","last_undegraded":"2024-03-06T07:24:11.423338+0000","last_fullsized":"2024-03-06T07:24:11.423338+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:25.046426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435001+0000","last_change":"2024-03-06T07:24:11.430560+0000","last_active":"2024-03-06T07:24:11.435001+0000","last_peered":"2024-03-06T07:24:11.435001+0000","last_clean":"2024-03-06T07:24:11.435001+0000","last_became_active":"2024-03-06T07:24:11.430292+0000","last_became_peered":"2024-03-06T07:24:11.430292+0000","last_unstale":"2024-03-06T07:24:11.435001+0000","last_undegraded":"2024-03-06T07:24:11.435001+0000","last_fullsized":"2024-03-06T07:24:11.435001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:12.846672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435734+0000","last_change":"2024-03-06T07:24:11.434276+0000","last_active":"2024-03-06T07:24:11.435734+0000","last_peered":"2024-03-06T07:24:11.435734+0000","last_clean":"2024-03-06T07:24:11.435734+0000","last_became_active":"2024-03-06T07:24:11.433993+0000","last_became_peered":"2024-03-06T07:24:11.433993+0000","last_unstale":"2024-03-06T07:24:11.435734+0000","last_undegraded":"2024-03-06T07:24:11.435734+0000","last_fullsized":"2024-03-06T07:24:11.435734+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:45.319669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.424136+0000","last_change":"2024-03-06T07:24:11.422288+0000","last_active":"2024-03-06T07:24:11.424136+0000","last_peered":"2024-03-06T07:24:11.424136+0000","last_clean":"2024-03-06T07:24:11.424136+0000","last_became_active":"2024-03-06T07:24:11.422024+0000","last_became_peered":"2024-03-06T07:24:11.422024+0000","last_unstale":"2024-03-06T07:24:11.424136+0000","last_undegraded":"2024-03-06T07:24:11.424136+0000","last_fullsized":"2024-03-06T07:24:11.424136+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:28.849111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429058+0000","last_change":"2024-03-06T07:24:11.427663+0000","last_active":"2024-03-06T07:24:11.429058+0000","last_peered":"2024-03-06T07:24:11.429058+0000","last_clean":"2024-03-06T07:24:11.429058+0000","last_became_active":"2024-03-06T07:24:11.427402+0000","last_became_peered":"2024-03-06T07:24:11.427402+0000","last_unstale":"2024-03-06T07:24:11.429058+0000","last_undegraded":"2024-03-06T07:24:11.429058+0000","last_fullsized":"2024-03-06T07:24:11.429058+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:06.120209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430261+0000","last_change":"2024-03-06T07:24:11.424554+0000","last_active":"2024-03-06T07:24:11.430261+0000","last_peered":"2024-03-06T07:24:11.430261+0000","last_clean":"2024-03-06T07:24:11.430261+0000","last_became_active":"2024-03-06T07:24:11.423360+0000","last_became_peered":"2024-03-06T07:24:11.423360+0000","last_unstale":"2024-03-06T07:24:11.430261+0000","last_undegraded":"2024-03-06T07:24:11.430261+0000","last_fullsized":"2024-03-06T07:24:11.430261+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:24.782748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425501+0000","last_change":"2024-03-06T07:24:11.423213+0000","last_active":"2024-03-06T07:24:11.425501+0000","last_peered":"2024-03-06T07:24:11.425501+0000","last_clean":"2024-03-06T07:24:11.425501+0000","last_became_active":"2024-03-06T07:24:11.422933+0000","last_became_peered":"2024-03-06T07:24:11.422933+0000","last_unstale":"2024-03-06T07:24:11.425501+0000","last_undegraded":"2024-03-06T07:24:11.425501+0000","last_fullsized":"2024-03-06T07:24:11.425501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:36.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429191+0000","last_change":"2024-03-06T07:24:11.423029+0000","last_active":"2024-03-06T07:24:11.429191+0000","last_peered":"2024-03-06T07:24:11.429191+0000","last_clean":"2024-03-06T07:24:11.429191+0000","last_became_active":"2024-03-06T07:24:11.422757+0000","last_became_peered":"2024-03-06T07:24:11.422757+0000","last_unstale":"2024-03-06T07:24:11.429191+0000","last_undegraded":"2024-03-06T07:24:11.429191+0000","last_fullsized":"2024-03-06T07:24:11.429191+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:03.276020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437991+0000","last_change":"2024-03-06T07:24:11.434259+0000","last_active":"2024-03-06T07:24:11.437991+0000","last_peered":"2024-03-06T07:24:11.437991+0000","last_clean":"2024-03-06T07:24:11.437991+0000","last_became_active":"2024-03-06T07:24:11.433670+0000","last_became_peered":"2024-03-06T07:24:11.433670+0000","last_unstale":"2024-03-06T07:24:11.437991+0000","last_undegraded":"2024-03-06T07:24:11.437991+0000","last_fullsized":"2024-03-06T07:24:11.437991+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:52.465571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432148+0000","last_change":"2024-03-06T07:24:11.429862+0000","last_active":"2024-03-06T07:24:11.432148+0000","last_peered":"2024-03-06T07:24:11.432148+0000","last_clean":"2024-03-06T07:24:11.432148+0000","last_became_active":"2024-03-06T07:24:11.429513+0000","last_became_peered":"2024-03-06T07:24:11.429513+0000","last_unstale":"2024-03-06T07:24:11.432148+0000","last_undegraded":"2024-03-06T07:24:11.432148+0000","last_fullsized":"2024-03-06T07:24:11.432148+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:35.886714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428891+0000","last_change":"2024-03-06T07:24:11.423793+0000","last_active":"2024-03-06T07:24:11.428891+0000","last_peered":"2024-03-06T07:24:11.428891+0000","last_clean":"2024-03-06T07:24:11.428891+0000","last_became_active":"2024-03-06T07:24:11.423542+0000","last_became_peered":"2024-03-06T07:24:11.423542+0000","last_unstale":"2024-03-06T07:24:11.428891+0000","last_undegraded":"2024-03-06T07:24:11.428891+0000","last_fullsized":"2024-03-06T07:24:11.428891+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:40.108890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437501+0000","last_change":"2024-03-06T07:24:11.433607+0000","last_active":"2024-03-06T07:24:11.437501+0000","last_peered":"2024-03-06T07:24:11.437501+0000","last_clean":"2024-03-06T07:24:11.437501+0000","last_became_active":"2024-03-06T07:24:11.433344+0000","last_became_peered":"2024-03-06T07:24:11.433344+0000","last_unstale":"2024-03-06T07:24:11.437501+0000","last_undegraded":"2024-03-06T07:24:11.437501+0000","last_fullsized":"2024-03-06T07:24:11.437501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:28.626770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428989+0000","last_change":"2024-03-06T07:24:11.426829+0000","last_active":"2024-03-06T07:24:11.428989+0000","last_peered":"2024-03-06T07:24:11.428989+0000","last_clean":"2024-03-06T07:24:11.428989+0000","last_became_active":"2024-03-06T07:24:11.426562+0000","last_became_peered":"2024-03-06T07:24:11.426562+0000","last_unstale":"2024-03-06T07:24:11.428989+0000","last_undegraded":"2024-03-06T07:24:11.428989+0000","last_fullsized":"2024-03-06T07:24:11.428989+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:47.024923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.431141+0000","last_change":"2024-03-06T07:24:10.421827+0000","last_active":"2024-03-06T07:24:11.431141+0000","last_peered":"2024-03-06T07:24:11.431141+0000","last_clean":"2024-03-06T07:24:11.431141+0000","last_became_active":"2024-03-06T07:24:10.421568+0000","last_became_peered":"2024-03-06T07:24:10.421568+0000","last_unstale":"2024-03-06T07:24:11.431141+0000","last_undegraded":"2024-03-06T07:24:11.431141+0000","last_fullsized":"2024-03-06T07:24:11.431141+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:19.231070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430140+0000","last_change":"2024-03-06T07:24:11.424371+0000","last_active":"2024-03-06T07:24:11.430140+0000","last_peered":"2024-03-06T07:24:11.430140+0000","last_clean":"2024-03-06T07:24:11.430140+0000","last_became_active":"2024-03-06T07:24:11.424085+0000","last_became_peered":"2024-03-06T07:24:11.424085+0000","last_unstale":"2024-03-06T07:24:11.430140+0000","last_undegraded":"2024-03-06T07:24:11.430140+0000","last_fullsized":"2024-03-06T07:24:11.430140+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:22.655505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435719+0000","last_change":"2024-03-06T07:24:10.423271+0000","last_active":"2024-03-06T07:24:11.435719+0000","last_peered":"2024-03-06T07:24:11.435719+0000","last_clean":"2024-03-06T07:24:11.435719+0000","last_became_active":"2024-03-06T07:24:10.422817+0000","last_became_peered":"2024-03-06T07:24:10.422817+0000","last_unstale":"2024-03-06T07:24:11.435719+0000","last_undegraded":"2024-03-06T07:24:11.435719+0000","last_fullsized":"2024-03-06T07:24:11.435719+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:38.044581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434817+0000","last_change":"2024-03-06T07:24:11.429802+0000","last_active":"2024-03-06T07:24:11.434817+0000","last_peered":"2024-03-06T07:24:11.434817+0000","last_clean":"2024-03-06T07:24:11.434817+0000","last_became_active":"2024-03-06T07:24:11.429323+0000","last_became_peered":"2024-03-06T07:24:11.429323+0000","last_unstale":"2024-03-06T07:24:11.434817+0000","last_undegraded":"2024-03-06T07:24:11.434817+0000","last_fullsized":"2024-03-06T07:24:11.434817+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:45.319669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432682+0000","last_change":"2024-03-06T07:24:10.425453+0000","last_active":"2024-03-06T07:24:11.432682+0000","last_peered":"2024-03-06T07:24:11.432682+0000","last_clean":"2024-03-06T07:24:11.432682+0000","last_became_active":"2024-03-06T07:24:10.425162+0000","last_became_peered":"2024-03-06T07:24:10.425162+0000","last_unstale":"2024-03-06T07:24:11.432682+0000","last_undegraded":"2024-03-06T07:24:11.432682+0000","last_fullsized":"2024-03-06T07:24:11.432682+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:44.871546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425536+0000","last_change":"2024-03-06T07:24:11.424501+0000","last_active":"2024-03-06T07:24:11.425536+0000","last_peered":"2024-03-06T07:24:11.425536+0000","last_clean":"2024-03-06T07:24:11.425536+0000","last_became_active":"2024-03-06T07:24:11.424235+0000","last_became_peered":"2024-03-06T07:24:11.424235+0000","last_unstale":"2024-03-06T07:24:11.425536+0000","last_undegraded":"2024-03-06T07:24:11.425536+0000","last_fullsized":"2024-03-06T07:24:11.425536+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:43.231154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435737+0000","last_change":"2024-03-06T07:24:10.422388+0000","last_active":"2024-03-06T07:24:11.435737+0000","last_peered":"2024-03-06T07:24:11.435737+0000","last_clean":"2024-03-06T07:24:11.435737+0000","last_became_active":"2024-03-06T07:24:10.422123+0000","last_became_peered":"2024-03-06T07:24:10.422123+0000","last_unstale":"2024-03-06T07:24:11.435737+0000","last_undegraded":"2024-03-06T07:24:11.435737+0000","last_fullsized":"2024-03-06T07:24:11.435737+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:58.063204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434954+0000","last_change":"2024-03-06T07:24:11.428108+0000","last_active":"2024-03-06T07:24:11.434954+0000","last_peered":"2024-03-06T07:24:11.434954+0000","last_clean":"2024-03-06T07:24:11.434954+0000","last_became_active":"2024-03-06T07:24:11.427547+0000","last_became_peered":"2024-03-06T07:24:11.427547+0000","last_unstale":"2024-03-06T07:24:11.434954+0000","last_undegraded":"2024-03-06T07:24:11.434954+0000","last_fullsized":"2024-03-06T07:24:11.434954+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:46.808499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437760+0000","last_change":"2024-03-06T07:24:10.420772+0000","last_active":"2024-03-06T07:24:11.437760+0000","last_peered":"2024-03-06T07:24:11.437760+0000","last_clean":"2024-03-06T07:24:11.437760+0000","last_became_active":"2024-03-06T07:24:10.420336+0000","last_became_peered":"2024-03-06T07:24:10.420336+0000","last_unstale":"2024-03-06T07:24:11.437760+0000","last_undegraded":"2024-03-06T07:24:11.437760+0000","last_fullsized":"2024-03-06T07:24:11.437760+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:46.842242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.440342+0000","last_change":"2024-03-06T07:24:11.434475+0000","last_active":"2024-03-06T07:24:11.440342+0000","last_peered":"2024-03-06T07:24:11.440342+0000","last_clean":"2024-03-06T07:24:11.440342+0000","last_became_active":"2024-03-06T07:24:11.433960+0000","last_became_peered":"2024-03-06T07:24:11.433960+0000","last_unstale":"2024-03-06T07:24:11.440342+0000","last_undegraded":"2024-03-06T07:24:11.440342+0000","last_fullsized":"2024-03-06T07:24:11.440342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:35.073901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428992+0000","last_change":"2024-03-06T07:24:10.412729+0000","last_active":"2024-03-06T07:24:11.428992+0000","last_peered":"2024-03-06T07:24:11.428992+0000","last_clean":"2024-03-06T07:24:11.428992+0000","last_became_active":"2024-03-06T07:24:10.412472+0000","last_became_peered":"2024-03-06T07:24:10.412472+0000","last_unstale":"2024-03-06T07:24:11.428992+0000","last_undegraded":"2024-03-06T07:24:11.428992+0000","last_fullsized":"2024-03-06T07:24:11.428992+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:58.063204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428600+0000","last_change":"2024-03-06T07:24:11.425388+0000","last_active":"2024-03-06T07:24:11.428600+0000","last_peered":"2024-03-06T07:24:11.428600+0000","last_clean":"2024-03-06T07:24:11.428600+0000","last_became_active":"2024-03-06T07:24:11.424948+0000","last_became_peered":"2024-03-06T07:24:11.424948+0000","last_unstale":"2024-03-06T07:24:11.428600+0000","last_undegraded":"2024-03-06T07:24:11.428600+0000","last_fullsized":"2024-03-06T07:24:11.428600+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:44.344838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.415117+0000","last_change":"2024-03-06T07:24:10.418951+0000","last_active":"2024-03-06T07:24:11.415117+0000","last_peered":"2024-03-06T07:24:11.415117+0000","last_clean":"2024-03-06T07:24:11.415117+0000","last_became_active":"2024-03-06T07:24:10.418659+0000","last_became_peered":"2024-03-06T07:24:10.418659+0000","last_unstale":"2024-03-06T07:24:11.415117+0000","last_undegraded":"2024-03-06T07:24:11.415117+0000","last_fullsized":"2024-03-06T07:24:11.415117+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:04.860899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435293+0000","last_change":"2024-03-06T07:24:11.427075+0000","last_active":"2024-03-06T07:24:11.435293+0000","last_peered":"2024-03-06T07:24:11.435293+0000","last_clean":"2024-03-06T07:24:11.435293+0000","last_became_active":"2024-03-06T07:24:11.426794+0000","last_became_peered":"2024-03-06T07:24:11.426794+0000","last_unstale":"2024-03-06T07:24:11.435293+0000","last_undegraded":"2024-03-06T07:24:11.435293+0000","last_fullsized":"2024-03-06T07:24:11.435293+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:16.489811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.410459+0000","last_change":"2024-03-06T07:24:10.416980+0000","last_active":"2024-03-06T07:24:11.410459+0000","last_peered":"2024-03-06T07:24:11.410459+0000","last_clean":"2024-03-06T07:24:11.410459+0000","last_became_active":"2024-03-06T07:24:10.416694+0000","last_became_peered":"2024-03-06T07:24:10.416694+0000","last_unstale":"2024-03-06T07:24:11.410459+0000","last_undegraded":"2024-03-06T07:24:11.410459+0000","last_fullsized":"2024-03-06T07:24:11.410459+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:11.523049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.427058+0000","last_change":"2024-03-06T07:24:11.423891+0000","last_active":"2024-03-06T07:24:11.427058+0000","last_peered":"2024-03-06T07:24:11.427058+0000","last_clean":"2024-03-06T07:24:11.427058+0000","last_became_active":"2024-03-06T07:24:11.423601+0000","last_became_peered":"2024-03-06T07:24:11.423601+0000","last_unstale":"2024-03-06T07:24:11.427058+0000","last_undegraded":"2024-03-06T07:24:11.427058+0000","last_fullsized":"2024-03-06T07:24:11.427058+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:05.754170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.419547+0000","last_change":"2024-03-06T07:24:10.425652+0000","last_active":"2024-03-06T07:24:11.419547+0000","last_peered":"2024-03-06T07:24:11.419547+0000","last_clean":"2024-03-06T07:24:11.419547+0000","last_became_active":"2024-03-06T07:24:10.425353+0000","last_became_peered":"2024-03-06T07:24:10.425353+0000","last_unstale":"2024-03-06T07:24:11.419547+0000","last_undegraded":"2024-03-06T07:24:11.419547+0000","last_fullsized":"2024-03-06T07:24:11.419547+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:31.665843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432103+0000","last_change":"2024-03-06T07:24:11.430568+0000","last_active":"2024-03-06T07:24:11.432103+0000","last_peered":"2024-03-06T07:24:11.432103+0000","last_clean":"2024-03-06T07:24:11.432103+0000","last_became_active":"2024-03-06T07:24:11.429116+0000","last_became_peered":"2024-03-06T07:24:11.429116+0000","last_unstale":"2024-03-06T07:24:11.432103+0000","last_undegraded":"2024-03-06T07:24:11.432103+0000","last_fullsized":"2024-03-06T07:24:11.432103+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:47.024923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.417386+0000","last_change":"2024-03-06T07:24:10.421332+0000","last_active":"2024-03-06T07:24:11.417386+0000","last_peered":"2024-03-06T07:24:11.417386+0000","last_clean":"2024-03-06T07:24:11.417386+0000","last_became_active":"2024-03-06T07:24:10.421063+0000","last_became_peered":"2024-03-06T07:24:10.421063+0000","last_unstale":"2024-03-06T07:24:11.417386+0000","last_undegraded":"2024-03-06T07:24:11.417386+0000","last_fullsized":"2024-03-06T07:24:11.417386+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:39.879733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423634+0000","last_change":"2024-03-06T07:24:11.418632+0000","last_active":"2024-03-06T07:24:11.423634+0000","last_peered":"2024-03-06T07:24:11.423634+0000","last_clean":"2024-03-06T07:24:11.423634+0000","last_became_active":"2024-03-06T07:24:11.418372+0000","last_became_peered":"2024-03-06T07:24:11.418372+0000","last_unstale":"2024-03-06T07:24:11.423634+0000","last_undegraded":"2024-03-06T07:24:11.423634+0000","last_fullsized":"2024-03-06T07:24:11.423634+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:39.876229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.438381+0000","last_change":"2024-03-06T07:24:10.426470+0000","last_active":"2024-03-06T07:24:11.438381+0000","last_peered":"2024-03-06T07:24:11.438381+0000","last_clean":"2024-03-06T07:24:11.438381+0000","last_became_active":"2024-03-06T07:24:10.426209+0000","last_became_peered":"2024-03-06T07:24:10.426209+0000","last_unstale":"2024-03-06T07:24:11.438381+0000","last_undegraded":"2024-03-06T07:24:11.438381+0000","last_fullsized":"2024-03-06T07:24:11.438381+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:09.245188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430737+0000","last_change":"2024-03-06T07:24:11.426807+0000","last_active":"2024-03-06T07:24:11.430737+0000","last_peered":"2024-03-06T07:24:11.430737+0000","last_clean":"2024-03-06T07:24:11.430737+0000","last_became_active":"2024-03-06T07:24:11.425815+0000","last_became_peered":"2024-03-06T07:24:11.425815+0000","last_unstale":"2024-03-06T07:24:11.430737+0000","last_undegraded":"2024-03-06T07:24:11.430737+0000","last_fullsized":"2024-03-06T07:24:11.430737+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:24.782748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.440527+0000","last_change":"2024-03-06T07:24:10.425734+0000","last_active":"2024-03-06T07:24:11.440527+0000","last_peered":"2024-03-06T07:24:11.440527+0000","last_clean":"2024-03-06T07:24:11.440527+0000","last_became_active":"2024-03-06T07:24:10.425352+0000","last_became_peered":"2024-03-06T07:24:10.425352+0000","last_unstale":"2024-03-06T07:24:11.440527+0000","last_undegraded":"2024-03-06T07:24:11.440527+0000","last_fullsized":"2024-03-06T07:24:11.440527+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:33.617363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430802+0000","last_change":"2024-03-06T07:24:11.426827+0000","last_active":"2024-03-06T07:24:11.430802+0000","last_peered":"2024-03-06T07:24:11.430802+0000","last_clean":"2024-03-06T07:24:11.430802+0000","last_became_active":"2024-03-06T07:24:11.426553+0000","last_became_peered":"2024-03-06T07:24:11.426553+0000","last_unstale":"2024-03-06T07:24:11.430802+0000","last_undegraded":"2024-03-06T07:24:11.430802+0000","last_fullsized":"2024-03-06T07:24:11.430802+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:43.231154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.443042+0000","last_change":"2024-03-06T07:24:10.427532+0000","last_active":"2024-03-06T07:24:11.443042+0000","last_peered":"2024-03-06T07:24:11.443042+0000","last_clean":"2024-03-06T07:24:11.443042+0000","last_became_active":"2024-03-06T07:24:10.427218+0000","last_became_peered":"2024-03-06T07:24:10.427218+0000","last_unstale":"2024-03-06T07:24:11.443042+0000","last_undegraded":"2024-03-06T07:24:11.443042+0000","last_fullsized":"2024-03-06T07:24:11.443042+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:44.317833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429675+0000","last_change":"2024-03-06T07:24:11.424481+0000","last_active":"2024-03-06T07:24:11.429675+0000","last_peered":"2024-03-06T07:24:11.429675+0000","last_clean":"2024-03-06T07:24:11.429675+0000","last_became_active":"2024-03-06T07:24:11.424215+0000","last_became_peered":"2024-03-06T07:24:11.424215+0000","last_unstale":"2024-03-06T07:24:11.429675+0000","last_undegraded":"2024-03-06T07:24:11.429675+0000","last_fullsized":"2024-03-06T07:24:11.429675+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:15.909456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.410374+0000","last_change":"2024-03-06T07:24:10.419249+0000","last_active":"2024-03-06T07:24:11.410374+0000","last_peered":"2024-03-06T07:24:11.410374+0000","last_clean":"2024-03-06T07:24:11.410374+0000","last_became_active":"2024-03-06T07:24:10.418976+0000","last_became_peered":"2024-03-06T07:24:10.418976+0000","last_unstale":"2024-03-06T07:24:11.410374+0000","last_undegraded":"2024-03-06T07:24:11.410374+0000","last_fullsized":"2024-03-06T07:24:11.410374+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:21.236770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.427450+0000","last_change":"2024-03-06T07:24:11.425410+0000","last_active":"2024-03-06T07:24:11.427450+0000","last_peered":"2024-03-06T07:24:11.427450+0000","last_clean":"2024-03-06T07:24:11.427450+0000","last_became_active":"2024-03-06T07:24:11.425059+0000","last_became_peered":"2024-03-06T07:24:11.425059+0000","last_unstale":"2024-03-06T07:24:11.427450+0000","last_undegraded":"2024-03-06T07:24:11.427450+0000","last_fullsized":"2024-03-06T07:24:11.427450+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:33.632896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.415268+0000","last_change":"2024-03-06T07:24:10.421385+0000","last_active":"2024-03-06T07:24:11.415268+0000","last_peered":"2024-03-06T07:24:11.415268+0000","last_clean":"2024-03-06T07:24:11.415268+0000","last_became_active":"2024-03-06T07:24:10.421122+0000","last_became_peered":"2024-03-06T07:24:10.421122+0000","last_unstale":"2024-03-06T07:24:11.415268+0000","last_undegraded":"2024-03-06T07:24:11.415268+0000","last_fullsized":"2024-03-06T07:24:11.415268+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:26.211315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435753+0000","last_change":"2024-03-06T07:24:11.434366+0000","last_active":"2024-03-06T07:24:11.435753+0000","last_peered":"2024-03-06T07:24:11.435753+0000","last_clean":"2024-03-06T07:24:11.435753+0000","last_became_active":"2024-03-06T07:24:11.434100+0000","last_became_peered":"2024-03-06T07:24:11.434100+0000","last_unstale":"2024-03-06T07:24:11.435753+0000","last_undegraded":"2024-03-06T07:24:11.435753+0000","last_fullsized":"2024-03-06T07:24:11.435753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:28.626770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.436835+0000","last_change":"2024-03-06T07:24:10.422051+0000","last_active":"2024-03-06T07:24:11.436835+0000","last_peered":"2024-03-06T07:24:11.436835+0000","last_clean":"2024-03-06T07:24:11.436835+0000","last_became_active":"2024-03-06T07:24:10.421782+0000","last_became_peered":"2024-03-06T07:24:10.421782+0000","last_unstale":"2024-03-06T07:24:11.436835+0000","last_undegraded":"2024-03-06T07:24:11.436835+0000","last_fullsized":"2024-03-06T07:24:11.436835+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:19.231070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.431045+0000","last_change":"2024-03-06T07:24:11.428771+0000","last_active":"2024-03-06T07:24:11.431045+0000","last_peered":"2024-03-06T07:24:11.431045+0000","last_clean":"2024-03-06T07:24:11.431045+0000","last_became_active":"2024-03-06T07:24:11.428425+0000","last_became_peered":"2024-03-06T07:24:11.428425+0000","last_unstale":"2024-03-06T07:24:11.431045+0000","last_undegraded":"2024-03-06T07:24:11.431045+0000","last_fullsized":"2024-03-06T07:24:11.431045+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:43.231154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.443048+0000","last_change":"2024-03-06T07:24:10.420499+0000","last_active":"2024-03-06T07:24:11.443048+0000","last_peered":"2024-03-06T07:24:11.443048+0000","last_clean":"2024-03-06T07:24:11.443048+0000","last_became_active":"2024-03-06T07:24:10.420247+0000","last_became_peered":"2024-03-06T07:24:10.420247+0000","last_unstale":"2024-03-06T07:24:11.443048+0000","last_undegraded":"2024-03-06T07:24:11.443048+0000","last_fullsized":"2024-03-06T07:24:11.443048+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:46.023087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423708+0000","last_change":"2024-03-06T07:24:11.422945+0000","last_active":"2024-03-06T07:24:11.423708+0000","last_peered":"2024-03-06T07:24:11.423708+0000","last_clean":"2024-03-06T07:24:11.423708+0000","last_became_active":"2024-03-06T07:24:11.422643+0000","last_became_peered":"2024-03-06T07:24:11.422643+0000","last_unstale":"2024-03-06T07:24:11.423708+0000","last_undegraded":"2024-03-06T07:24:11.423708+0000","last_fullsized":"2024-03-06T07:24:11.423708+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:32.667680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.432864+0000","last_change":"2024-03-06T07:24:10.422477+0000","last_active":"2024-03-06T07:24:11.432864+0000","last_peered":"2024-03-06T07:24:11.432864+0000","last_clean":"2024-03-06T07:24:11.432864+0000","last_became_active":"2024-03-06T07:24:10.422208+0000","last_became_peered":"2024-03-06T07:24:10.422208+0000","last_unstale":"2024-03-06T07:24:11.432864+0000","last_undegraded":"2024-03-06T07:24:11.432864+0000","last_fullsized":"2024-03-06T07:24:11.432864+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:31.665843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434882+0000","last_change":"2024-03-06T07:24:11.427613+0000","last_active":"2024-03-06T07:24:11.434882+0000","last_peered":"2024-03-06T07:24:11.434882+0000","last_clean":"2024-03-06T07:24:11.434882+0000","last_became_active":"2024-03-06T07:24:11.427342+0000","last_became_peered":"2024-03-06T07:24:11.427342+0000","last_unstale":"2024-03-06T07:24:11.434882+0000","last_undegraded":"2024-03-06T07:24:11.434882+0000","last_fullsized":"2024-03-06T07:24:11.434882+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:27.823929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435810+0000","last_change":"2024-03-06T07:24:10.422632+0000","last_active":"2024-03-06T07:24:11.435810+0000","last_peered":"2024-03-06T07:24:11.435810+0000","last_clean":"2024-03-06T07:24:11.435810+0000","last_became_active":"2024-03-06T07:24:10.422213+0000","last_became_peered":"2024-03-06T07:24:10.422213+0000","last_unstale":"2024-03-06T07:24:11.435810+0000","last_undegraded":"2024-03-06T07:24:11.435810+0000","last_fullsized":"2024-03-06T07:24:11.435810+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:19.231070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430761+0000","last_change":"2024-03-06T07:24:11.427477+0000","last_active":"2024-03-06T07:24:11.430761+0000","last_peered":"2024-03-06T07:24:11.430761+0000","last_clean":"2024-03-06T07:24:11.430761+0000","last_became_active":"2024-03-06T07:24:11.427193+0000","last_became_peered":"2024-03-06T07:24:11.427193+0000","last_unstale":"2024-03-06T07:24:11.430761+0000","last_undegraded":"2024-03-06T07:24:11.430761+0000","last_fullsized":"2024-03-06T07:24:11.430761+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:36.884328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.418287+0000","last_change":"2024-03-06T07:24:10.420892+0000","last_active":"2024-03-06T07:24:11.418287+0000","last_peered":"2024-03-06T07:24:11.418287+0000","last_clean":"2024-03-06T07:24:11.418287+0000","last_became_active":"2024-03-06T07:24:10.420637+0000","last_became_peered":"2024-03-06T07:24:10.420637+0000","last_unstale":"2024-03-06T07:24:11.418287+0000","last_undegraded":"2024-03-06T07:24:11.418287+0000","last_fullsized":"2024-03-06T07:24:11.418287+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:24.044590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429783+0000","last_change":"2024-03-06T07:24:11.427212+0000","last_active":"2024-03-06T07:24:11.429783+0000","last_peered":"2024-03-06T07:24:11.429783+0000","last_clean":"2024-03-06T07:24:11.429783+0000","last_became_active":"2024-03-06T07:24:11.426411+0000","last_became_peered":"2024-03-06T07:24:11.426411+0000","last_unstale":"2024-03-06T07:24:11.429783+0000","last_undegraded":"2024-03-06T07:24:11.429783+0000","last_fullsized":"2024-03-06T07:24:11.429783+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:15.909456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.437795+0000","last_change":"2024-03-06T07:24:10.422534+0000","last_active":"2024-03-06T07:24:11.437795+0000","last_peered":"2024-03-06T07:24:11.437795+0000","last_clean":"2024-03-06T07:24:11.437795+0000","last_became_active":"2024-03-06T07:24:10.422011+0000","last_became_peered":"2024-03-06T07:24:10.422011+0000","last_unstale":"2024-03-06T07:24:11.437795+0000","last_undegraded":"2024-03-06T07:24:11.437795+0000","last_fullsized":"2024-03-06T07:24:11.437795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:27.624933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.420209+0000","last_change":"2024-03-06T07:24:11.418776+0000","last_active":"2024-03-06T07:24:11.420209+0000","last_peered":"2024-03-06T07:24:11.420209+0000","last_clean":"2024-03-06T07:24:11.420209+0000","last_became_active":"2024-03-06T07:24:11.418026+0000","last_became_peered":"2024-03-06T07:24:11.418026+0000","last_unstale":"2024-03-06T07:24:11.420209+0000","last_undegraded":"2024-03-06T07:24:11.420209+0000","last_fullsized":"2024-03-06T07:24:11.420209+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:15.909456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.409849+0000","last_change":"2024-03-06T07:24:10.425123+0000","last_active":"2024-03-06T07:24:11.409849+0000","last_peered":"2024-03-06T07:24:11.409849+0000","last_clean":"2024-03-06T07:24:11.409849+0000","last_became_active":"2024-03-06T07:24:10.424854+0000","last_became_peered":"2024-03-06T07:24:10.424854+0000","last_unstale":"2024-03-06T07:24:11.409849+0000","last_undegraded":"2024-03-06T07:24:11.409849+0000","last_fullsized":"2024-03-06T07:24:11.409849+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:08.401255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.429267+0000","last_change":"2024-03-06T07:24:11.425789+0000","last_active":"2024-03-06T07:24:11.429267+0000","last_peered":"2024-03-06T07:24:11.429267+0000","last_clean":"2024-03-06T07:24:11.429267+0000","last_became_active":"2024-03-06T07:24:11.425542+0000","last_became_peered":"2024-03-06T07:24:11.425542+0000","last_unstale":"2024-03-06T07:24:11.429267+0000","last_undegraded":"2024-03-06T07:24:11.429267+0000","last_fullsized":"2024-03-06T07:24:11.429267+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:35.886714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.414642+0000","last_change":"2024-03-06T07:24:10.421384+0000","last_active":"2024-03-06T07:24:11.414642+0000","last_peered":"2024-03-06T07:24:11.414642+0000","last_clean":"2024-03-06T07:24:11.414642+0000","last_became_active":"2024-03-06T07:24:10.421149+0000","last_became_peered":"2024-03-06T07:24:10.421149+0000","last_unstale":"2024-03-06T07:24:11.414642+0000","last_undegraded":"2024-03-06T07:24:11.414642+0000","last_fullsized":"2024-03-06T07:24:11.414642+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:32.631060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.430281+0000","last_change":"2024-03-06T07:24:11.425511+0000","last_active":"2024-03-06T07:24:11.430281+0000","last_peered":"2024-03-06T07:24:11.430281+0000","last_clean":"2024-03-06T07:24:11.430281+0000","last_became_active":"2024-03-06T07:24:11.425242+0000","last_became_peered":"2024-03-06T07:24:11.425242+0000","last_unstale":"2024-03-06T07:24:11.430281+0000","last_undegraded":"2024-03-06T07:24:11.430281+0000","last_fullsized":"2024-03-06T07:24:11.430281+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:54.799719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.433553+0000","last_change":"2024-03-06T07:24:10.425758+0000","last_active":"2024-03-06T07:24:11.433553+0000","last_peered":"2024-03-06T07:24:11.433553+0000","last_clean":"2024-03-06T07:24:11.433553+0000","last_became_active":"2024-03-06T07:24:10.425489+0000","last_became_peered":"2024-03-06T07:24:10.425489+0000","last_unstale":"2024-03-06T07:24:11.433553+0000","last_undegraded":"2024-03-06T07:24:11.433553+0000","last_fullsized":"2024-03-06T07:24:11.433553+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:43.343002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.428508+0000","last_change":"2024-03-06T07:24:11.424876+0000","last_active":"2024-03-06T07:24:11.428508+0000","last_peered":"2024-03-06T07:24:11.428508+0000","last_clean":"2024-03-06T07:24:11.428508+0000","last_became_active":"2024-03-06T07:24:11.424567+0000","last_became_peered":"2024-03-06T07:24:11.424567+0000","last_unstale":"2024-03-06T07:24:11.428508+0000","last_undegraded":"2024-03-06T07:24:11.428508+0000","last_fullsized":"2024-03-06T07:24:11.428508+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:25.046426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.418213+0000","last_change":"2024-03-06T07:24:10.420338+0000","last_active":"2024-03-06T07:24:11.418213+0000","last_peered":"2024-03-06T07:24:11.418213+0000","last_clean":"2024-03-06T07:24:11.418213+0000","last_became_active":"2024-03-06T07:24:10.419569+0000","last_became_peered":"2024-03-06T07:24:10.419569+0000","last_unstale":"2024-03-06T07:24:11.418213+0000","last_undegraded":"2024-03-06T07:24:11.418213+0000","last_fullsized":"2024-03-06T07:24:11.418213+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:09.245188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423668+0000","last_change":"2024-03-06T07:24:11.419306+0000","last_active":"2024-03-06T07:24:11.423668+0000","last_peered":"2024-03-06T07:24:11.423668+0000","last_clean":"2024-03-06T07:24:11.423668+0000","last_became_active":"2024-03-06T07:24:11.419000+0000","last_became_peered":"2024-03-06T07:24:11.419000+0000","last_unstale":"2024-03-06T07:24:11.423668+0000","last_undegraded":"2024-03-06T07:24:11.423668+0000","last_fullsized":"2024-03-06T07:24:11.423668+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:04.127474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.411812+0000","last_change":"2024-03-06T07:24:10.417309+0000","last_active":"2024-03-06T07:24:11.411812+0000","last_peered":"2024-03-06T07:24:11.411812+0000","last_clean":"2024-03-06T07:24:11.411812+0000","last_became_active":"2024-03-06T07:24:10.417037+0000","last_became_peered":"2024-03-06T07:24:10.417037+0000","last_unstale":"2024-03-06T07:24:11.411812+0000","last_undegraded":"2024-03-06T07:24:11.411812+0000","last_fullsized":"2024-03-06T07:24:11.411812+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:48.289131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.425521+0000","last_change":"2024-03-06T07:24:11.424052+0000","last_active":"2024-03-06T07:24:11.425521+0000","last_peered":"2024-03-06T07:24:11.425521+0000","last_clean":"2024-03-06T07:24:11.425521+0000","last_became_active":"2024-03-06T07:24:11.423779+0000","last_became_peered":"2024-03-06T07:24:11.423779+0000","last_unstale":"2024-03-06T07:24:11.425521+0000","last_undegraded":"2024-03-06T07:24:11.425521+0000","last_fullsized":"2024-03-06T07:24:11.425521+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:54.799719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.415614+0000","last_change":"2024-03-06T07:24:10.419644+0000","last_active":"2024-03-06T07:24:11.415614+0000","last_peered":"2024-03-06T07:24:11.415614+0000","last_clean":"2024-03-06T07:24:11.415614+0000","last_became_active":"2024-03-06T07:24:10.418589+0000","last_became_peered":"2024-03-06T07:24:10.418589+0000","last_unstale":"2024-03-06T07:24:11.415614+0000","last_undegraded":"2024-03-06T07:24:11.415614+0000","last_fullsized":"2024-03-06T07:24:11.415614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:44.871546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.434977+0000","last_change":"2024-03-06T07:24:11.428681+0000","last_active":"2024-03-06T07:24:11.434977+0000","last_peered":"2024-03-06T07:24:11.434977+0000","last_clean":"2024-03-06T07:24:11.434977+0000","last_became_active":"2024-03-06T07:24:11.427698+0000","last_became_peered":"2024-03-06T07:24:11.427698+0000","last_unstale":"2024-03-06T07:24:11.434977+0000","last_undegraded":"2024-03-06T07:24:11.434977+0000","last_fullsized":"2024-03-06T07:24:11.434977+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:23.177374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.409688+0000","last_change":"2024-03-06T07:24:10.424491+0000","last_active":"2024-03-06T07:24:11.409688+0000","last_peered":"2024-03-06T07:24:11.409688+0000","last_clean":"2024-03-06T07:24:11.409688+0000","last_became_active":"2024-03-06T07:24:10.424234+0000","last_became_peered":"2024-03-06T07:24:10.424234+0000","last_unstale":"2024-03-06T07:24:11.409688+0000","last_undegraded":"2024-03-06T07:24:11.409688+0000","last_fullsized":"2024-03-06T07:24:11.409688+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:27.847275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.436118+0000","last_change":"2024-03-06T07:24:11.435296+0000","last_active":"2024-03-06T07:24:11.436118+0000","last_peered":"2024-03-06T07:24:11.436118+0000","last_clean":"2024-03-06T07:24:11.436118+0000","last_became_active":"2024-03-06T07:24:11.434928+0000","last_became_peered":"2024-03-06T07:24:11.434928+0000","last_unstale":"2024-03-06T07:24:11.436118+0000","last_undegraded":"2024-03-06T07:24:11.436118+0000","last_fullsized":"2024-03-06T07:24:11.436118+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:16.500427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.412395+0000","last_change":"2024-03-06T07:24:10.419415+0000","last_active":"2024-03-06T07:24:11.412395+0000","last_peered":"2024-03-06T07:24:11.412395+0000","last_clean":"2024-03-06T07:24:11.412395+0000","last_became_active":"2024-03-06T07:24:10.419147+0000","last_became_peered":"2024-03-06T07:24:10.419147+0000","last_unstale":"2024-03-06T07:24:11.412395+0000","last_undegraded":"2024-03-06T07:24:11.412395+0000","last_fullsized":"2024-03-06T07:24:11.412395+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:18.598190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.423189+0000","last_change":"2024-03-06T07:24:11.418454+0000","last_active":"2024-03-06T07:24:11.423189+0000","last_peered":"2024-03-06T07:24:11.423189+0000","last_clean":"2024-03-06T07:24:11.423189+0000","last_became_active":"2024-03-06T07:24:11.418157+0000","last_became_peered":"2024-03-06T07:24:11.418157+0000","last_unstale":"2024-03-06T07:24:11.423189+0000","last_undegraded":"2024-03-06T07:24:11.423189+0000","last_fullsized":"2024-03-06T07:24:11.423189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:10.283622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.409944+0000","last_change":"2024-03-06T07:24:10.426732+0000","last_active":"2024-03-06T07:24:11.409944+0000","last_peered":"2024-03-06T07:24:11.409944+0000","last_clean":"2024-03-06T07:24:11.409944+0000","last_became_active":"2024-03-06T07:24:10.426456+0000","last_became_peered":"2024-03-06T07:24:10.426456+0000","last_unstale":"2024-03-06T07:24:11.409944+0000","last_undegraded":"2024-03-06T07:24:11.409944+0000","last_fullsized":"2024-03-06T07:24:11.409944+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:13.192638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435271+0000","last_change":"2024-03-06T07:24:11.426613+0000","last_active":"2024-03-06T07:24:11.435271+0000","last_peered":"2024-03-06T07:24:11.435271+0000","last_clean":"2024-03-06T07:24:11.435271+0000","last_became_active":"2024-03-06T07:24:11.426063+0000","last_became_peered":"2024-03-06T07:24:11.426063+0000","last_unstale":"2024-03-06T07:24:11.435271+0000","last_undegraded":"2024-03-06T07:24:11.435271+0000","last_fullsized":"2024-03-06T07:24:11.435271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:18.624608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.416716+0000","last_change":"2024-03-06T07:24:10.421360+0000","last_active":"2024-03-06T07:24:11.416716+0000","last_peered":"2024-03-06T07:24:11.416716+0000","last_clean":"2024-03-06T07:24:11.416716+0000","last_became_active":"2024-03-06T07:24:10.421075+0000","last_became_peered":"2024-03-06T07:24:10.421075+0000","last_unstale":"2024-03-06T07:24:11.416716+0000","last_undegraded":"2024-03-06T07:24:11.416716+0000","last_fullsized":"2024-03-06T07:24:11.416716+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:15.498591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.435049+0000","last_change":"2024-03-06T07:24:11.431571+0000","last_active":"2024-03-06T07:24:11.435049+0000","last_peered":"2024-03-06T07:24:11.435049+0000","last_clean":"2024-03-06T07:24:11.435049+0000","last_became_active":"2024-03-06T07:24:11.426693+0000","last_became_peered":"2024-03-06T07:24:11.426693+0000","last_unstale":"2024-03-06T07:24:11.435049+0000","last_undegraded":"2024-03-06T07:24:11.435049+0000","last_fullsized":"2024-03-06T07:24:11.435049+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.400705+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.400705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:35.653124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.410126+0000","last_change":"2024-03-06T07:24:10.421129+0000","last_active":"2024-03-06T07:24:11.410126+0000","last_peered":"2024-03-06T07:24:11.410126+0000","last_clean":"2024-03-06T07:24:11.410126+0000","last_became_active":"2024-03-06T07:24:10.419860+0000","last_became_peered":"2024-03-06T07:24:10.419860+0000","last_unstale":"2024-03-06T07:24:11.410126+0000","last_undegraded":"2024-03-06T07:24:11.410126+0000","last_fullsized":"2024-03-06T07:24:11.410126+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.398869+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.398869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:23.934612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":233,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:11.416219+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:24:11.416219+0000","last_peered":"2024-03-06T07:24:11.416219+0000","last_clean":"2024-03-06T07:24:11.416219+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:24:11.416219+0000","last_undegraded":"2024-03-06T07:24:11.416219+0000","last_fullsized":"2024-03-06T07:24:11.416219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36600,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335240,"statfs":{"total":96636764160,"available":96599285760,"internally_reserved":0,"allocated":1286144,"data_stored":873230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36016,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335824,"statfs":{"total":96636764160,"available":96599883776,"internally_reserved":0,"allocated":684032,"data_stored":282702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":283354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574878,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36028,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335812,"statfs":{"total":96636764160,"available":96599871488,"internally_reserved":0,"allocated":696320,"data_stored":284314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":282758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":282758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":688128,"data_stored":282792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":875264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:14.845 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:14.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:14.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.987+0000 7fe1d98c9640 1 Processor -- start 2024-03-06T07:24:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.987+0000 7fe1d98c9640 1 -- start start 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.987+0000 7fe1d98c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1d40cf200 0x7fe1d40cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.987+0000 7fe1d98c9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d41600a0 con 0x7fe1d40d06e0 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.987+0000 7fe1d98c9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d40d47f0 con 0x7fe1d415fc30 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.987+0000 7fe1d98c9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d40ced70 con 0x7fe1d40cf200 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.988+0000 7fe1d37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1d40cf200 0x7fe1d40cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.988+0000 7fe1d2ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe1d40d06e0 0x7fe1d40d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54930/0 (socket says 172.21.15.77:54930) 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.988+0000 7fe1d2ffd640 1 -- 172.21.15.77:0/1667590433 learned_addr learned my addr 172.21.15.77:0/1667590433 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.988+0000 7fe1d37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1d40cf200 0x7fe1d40cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56822/0 (socket says 172.21.15.77:56822) 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.988+0000 7fe1d37fe640 1 -- 172.21.15.77:0/1667590433 >> v1:172.21.15.158:6789/0 conn(0x7fe1d415fc30 legacy=0x7fe1d40b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.988+0000 7fe1d37fe640 1 -- 172.21.15.77:0/1667590433 >> v1:172.21.15.77:6789/0 conn(0x7fe1d40d06e0 legacy=0x7fe1d40d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.988+0000 7fe1d1ffb640 1 -- 172.21.15.77:0/1667590433 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413559272 0 0) 0x7fe1d41600a0 con 0x7fe1d40d06e0 2024-03-06T07:24:14.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.988+0000 7fe1d37fe640 1 -- 172.21.15.77:0/1667590433 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d40d1ab0 con 0x7fe1d40cf200 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.989+0000 7fe1d37fe640 1 --2- 172.21.15.77:0/1667590433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40cf670 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009d90 tx=0x7fe1cc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdf042eb79d0aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.989+0000 7fe1d1ffb640 1 -- 172.21.15.77:0/1667590433 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc039070 con 0x7fe1d40cf200 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.989+0000 7fe1d1ffb640 1 -- 172.21.15.77:0/1667590433 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc02cc60 con 0x7fe1d40cf200 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.989+0000 7fe1d1ffb640 1 -- 172.21.15.77:0/1667590433 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc034660 con 0x7fe1d40cf200 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.989+0000 7fe1d98c9640 1 -- 172.21.15.77:0/1667590433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 msgr2=0x7fe1d40cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.989+0000 7fe1d98c9640 1 --2- 172.21.15.77:0/1667590433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40cf670 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009d90 tx=0x7fe1cc02beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.990+0000 7fe1d98c9640 1 -- 172.21.15.77:0/1667590433 shutdown_connections 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.990+0000 7fe1d98c9640 1 --2- 172.21.15.77:0/1667590433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40cf670 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.990+0000 7fe1d98c9640 1 -- 172.21.15.77:0/1667590433 >> 172.21.15.77:0/1667590433 conn(0x7fe1d40d3cd0 msgr2=0x7fe1d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.990+0000 7fe1d98c9640 1 -- 172.21.15.77:0/1667590433 shutdown_connections 2024-03-06T07:24:14.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.990+0000 7fe1d98c9640 1 -- 172.21.15.77:0/1667590433 wait complete. 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.991+0000 7fe1d98c9640 1 Processor -- start 2024-03-06T07:24:14.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.992+0000 7fe1d98c9640 1 -- start start 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d98c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d98c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d40d06e0 0x7fe1d4151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d98c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe1d415fc30 0x7fe1d40b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d98c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40ce990 con 0x7fe1d40cf200 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d98c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40d4ab0 con 0x7fe1d40d06e0 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d98c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40b44d0 con 0x7fe1d415fc30 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56836/0 (socket says 172.21.15.77:56836) 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 -- 172.21.15.77:0/2273838575 learned_addr learned my addr 172.21.15.77:0/2273838575 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 -- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe1d415fc30 msgr2=0x7fe1d40b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d37fe640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe1d415fc30 0x7fe1d40b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d27fc640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d40d06e0 0x7fe1d4151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe1d415fc30 0x7fe1d40b1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 -- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d40d06e0 msgr2=0x7fe1d4151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d40d06e0 0x7fe1d4151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 -- 172.21.15.77:0/2273838575 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d41f48b0 con 0x7fe1d40cf200 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d27fc640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d40d06e0 0x7fe1d4151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d2ffd640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40c7f90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400c430 tx=0x7fe1c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1cbfff640 1 -- 172.21.15.77:0/2273838575 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c40074e0 con 0x7fe1d40cf200 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1cbfff640 1 -- 172.21.15.77:0/2273838575 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c40047e0 con 0x7fe1d40cf200 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1cbfff640 1 -- 172.21.15.77:0/2273838575 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4004bd0 con 0x7fe1d40cf200 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.993+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d40b2150 con 0x7fe1d40cf200 2024-03-06T07:24:14.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.994+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d40b1b90 con 0x7fe1d40cf200 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.995+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d40a1c80 con 0x7fe1d40cf200 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.995+0000 7fe1cbfff640 1 -- 172.21.15.77:0/2273838575 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1c4011070 con 0x7fe1d40cf200 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.995+0000 7fe1cbfff640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1b406b4d0 0x7fe1b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.995+0000 7fe1cbfff640 1 -- 172.21.15.77:0/2273838575 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1c4089e30 con 0x7fe1d40cf200 2024-03-06T07:24:14.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.995+0000 7fe1d27fc640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1b406b4d0 0x7fe1b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:14.996+0000 7fe1d27fc640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1b406b4d0 0x7fe1b406d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe1d41f7ee0 tx=0x7fe1bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.000+0000 7fe1cbfff640 1 -- 172.21.15.77:0/2273838575 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1c4058d40 con 0x7fe1d40cf200 2024-03-06T07:24:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.245+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe1d40b4f40 con 0x7fe1d40cf200 2024-03-06T07:24:15.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.247+0000 7fe1cbfff640 1 -- 172.21.15.77:0/2273838575 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe1c4058640 con 0x7fe1d40cf200 2024-03-06T07:24:15.248 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:24:15.249 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:24:15.249 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:24:15.249 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:24:15.249 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:15.249 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1b406b4d0 msgr2=0x7fe1b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1b406b4d0 0x7fe1b406d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe1d41f7ee0 tx=0x7fe1bc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 msgr2=0x7fe1d40c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40c7f90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400c430 tx=0x7fe1c400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 shutdown_connections 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe1d415fc30 0x7fe1d40b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1d40d06e0 0x7fe1d4151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe1b406b4d0 0x7fe1b406d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 --2- 172.21.15.77:0/2273838575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1d40cf200 0x7fe1d40c7f90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 >> 172.21.15.77:0/2273838575 conn(0x7fe1d40d3cd0 msgr2=0x7fe1d41624c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 shutdown_connections 2024-03-06T07:24:15.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.249+0000 7fe1d98c9640 1 -- 172.21.15.77:0/2273838575 wait complete. 2024-03-06T07:24:15.262 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:15.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.361+0000 7f7f8ce47640 1 Processor -- start 2024-03-06T07:24:15.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.361+0000 7f7f8ce47640 1 -- start start 2024-03-06T07:24:15.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.362+0000 7f7f8ce47640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7f880d22b0 0x7f7f880d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.362+0000 7f7f8ce47640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f880b3430 con 0x7f7f8815dfe0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.362+0000 7f7f8ce47640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f880b3710 con 0x7f7f880b41f0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.362+0000 7f7f8ce47640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f880d2c50 con 0x7f7f880d22b0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.362+0000 7f7f8659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7f8815dfe0 0x7f7f880cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54942/0 (socket says 172.21.15.77:54942) 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.362+0000 7f7f8659c640 1 -- 172.21.15.77:0/1283194828 learned_addr learned my addr 172.21.15.77:0/1283194828 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.363+0000 7f7f85d9b640 1 --2- 172.21.15.77:0/1283194828 >> v2:172.21.15.158:3300/0 conn(0x7f7f880d22b0 0x7f7f880d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.363+0000 7f7f8559a640 1 -- 172.21.15.77:0/1283194828 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744364492 0 0) 0x7f7f880b3430 con 0x7f7f8815dfe0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.363+0000 7f7f8559a640 1 -- 172.21.15.77:0/1283194828 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f70003520 con 0x7f7f8815dfe0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.363+0000 7f7f8559a640 1 -- 172.21.15.77:0/1283194828 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291028682 0 0) 0x7f7f880b3710 con 0x7f7f880b41f0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.363+0000 7f7f8559a640 1 -- 172.21.15.77:0/1283194828 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f880b3430 con 0x7f7f880b41f0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.363+0000 7f7f85d9b640 1 -- 172.21.15.77:0/1283194828 >> v1:172.21.15.158:6789/0 conn(0x7f7f880b41f0 legacy=0x7f7f880b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.363+0000 7f7f85d9b640 1 -- 172.21.15.77:0/1283194828 >> v1:172.21.15.77:6789/0 conn(0x7f7f8815dfe0 legacy=0x7f7f880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.363+0000 7f7f85d9b640 1 -- 172.21.15.77:0/1283194828 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f880b39f0 con 0x7f7f880d22b0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.364+0000 7f7f85d9b640 1 --2- 172.21.15.77:0/1283194828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880d22b0 0x7f7f880d2710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0097d0 tx=0x7f7f7c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f850a2dd0ac35ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.364+0000 7f7f8559a640 1 -- 172.21.15.77:0/1283194828 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c034070 con 0x7f7f880d22b0 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.364+0000 7f7f8559a640 1 -- 172.21.15.77:0/1283194828 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c030610 con 0x7f7f880d22b0 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.364+0000 7f7f8559a640 1 -- 172.21.15.77:0/1283194828 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c0309e0 con 0x7f7f880d22b0 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.364+0000 7f7f8ce47640 1 -- 172.21.15.77:0/1283194828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880d22b0 msgr2=0x7f7f880d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.364+0000 7f7f8ce47640 1 --2- 172.21.15.77:0/1283194828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880d22b0 0x7f7f880d2710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0097d0 tx=0x7f7f7c02b960 comp rx=0 tx=0).stop 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.365+0000 7f7f8ce47640 1 -- 172.21.15.77:0/1283194828 shutdown_connections 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.365+0000 7f7f8ce47640 1 --2- 172.21.15.77:0/1283194828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880d22b0 0x7f7f880d2710 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.365+0000 7f7f8ce47640 1 -- 172.21.15.77:0/1283194828 >> 172.21.15.77:0/1283194828 conn(0x7f7f880d3cd0 msgr2=0x7f7f880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.365+0000 7f7f8ce47640 1 -- 172.21.15.77:0/1283194828 shutdown_connections 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.365+0000 7f7f8ce47640 1 -- 172.21.15.77:0/1283194828 wait complete. 2024-03-06T07:24:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.366+0000 7f7f8ce47640 1 Processor -- start 2024-03-06T07:24:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.366+0000 7f7f8ce47640 1 -- start start 2024-03-06T07:24:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f8ce47640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880b41f0 0x7f7f88161230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f8ce47640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f880d22b0 0x7f7f88151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f8ce47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8815dfe0 0x7f7f880b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f8ce47640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880b3d60 con 0x7f7f8815dfe0 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f8ce47640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880b31e0 con 0x7f7f880b41f0 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f8ce47640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880d1c70 con 0x7f7f880d22b0 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8815dfe0 0x7f7f880b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8815dfe0 0x7f7f880b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56852/0 (socket says 172.21.15.77:56852) 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 -- 172.21.15.77:0/3590407203 learned_addr learned my addr 172.21.15.77:0/3590407203 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f8659c640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880b41f0 0x7f7f88161230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f85d9b640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f880d22b0 0x7f7f88151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 -- 172.21.15.77:0/3590407203 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f880d22b0 msgr2=0x7f7f88151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f880d22b0 0x7f7f88151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 -- 172.21.15.77:0/3590407203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880b41f0 msgr2=0x7f7f88161230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880b41f0 0x7f7f88161230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 -- 172.21.15.77:0/3590407203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f880b8ff0 con 0x7f7f8815dfe0 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f86d9d640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8815dfe0 0x7f7f880b68a0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c7a0 tx=0x7f7f8000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f76ffd640 1 -- 172.21.15.77:0/3590407203 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80003b70 con 0x7f7f8815dfe0 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f76ffd640 1 -- 172.21.15.77:0/3590407203 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f80003d30 con 0x7f7f8815dfe0 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.367+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f880b92d0 con 0x7f7f8815dfe0 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.368+0000 7f7f76ffd640 1 -- 172.21.15.77:0/3590407203 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8000aa90 con 0x7f7f8815dfe0 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.368+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f880b9890 con 0x7f7f8815dfe0 2024-03-06T07:24:15.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.369+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f880670d0 con 0x7f7f8815dfe0 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.369+0000 7f7f76ffd640 1 -- 172.21.15.77:0/3590407203 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f8000ac30 con 0x7f7f8815dfe0 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.369+0000 7f7f76ffd640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f6806b4d0 0x7f7f6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.370+0000 7f7f76ffd640 1 -- 172.21.15.77:0/3590407203 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f8008a180 con 0x7f7f8815dfe0 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.370+0000 7f7f8659c640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f6806b4d0 0x7f7f6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.370+0000 7f7f8659c640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f6806b4d0 0x7f7f6806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7f780011e0 tx=0x7f7f78009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.374+0000 7f7f76ffd640 1 -- 172.21.15.77:0/3590407203 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f880670d0 con 0x7f7f8815dfe0 2024-03-06T07:24:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.605+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f8804f330 con 0x7f7f8815dfe0 2024-03-06T07:24:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.606+0000 7f7f76ffd640 1 -- 172.21.15.77:0/3590407203 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f7f80059000 con 0x7f7f8815dfe0 2024-03-06T07:24:15.608 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:15.608 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.760628+0000","modified":"2024-03-06T07:24:12.764047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/251375557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":251375557},{"type":"v1","addr":"172.21.15.77:6837","nonce":251375557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-03-06T07:24:15.608 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-06T07:24:15.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f6806b4d0 msgr2=0x7f7f6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f6806b4d0 0x7f7f6806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7f780011e0 tx=0x7f7f78009040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8815dfe0 msgr2=0x7f7f880b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8815dfe0 0x7f7f880b68a0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c7a0 tx=0x7f7f8000cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:15.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 shutdown_connections 2024-03-06T07:24:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f6806b4d0 0x7f7f6806d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8815dfe0 0x7f7f880b68a0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f880d22b0 0x7f7f88151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 --2- 172.21.15.77:0/3590407203 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f880b41f0 0x7f7f88161230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.607+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 >> 172.21.15.77:0/3590407203 conn(0x7f7f880d3cd0 msgr2=0x7f7f880d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.608+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 shutdown_connections 2024-03-06T07:24:15.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:15.608+0000 7f7f8ce47640 1 -- 172.21.15.77:0/3590407203 wait complete. 2024-03-06T07:24:16.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:10.760628+0000', 'modified': '2024-03-06T07:24:12.764047+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6054': {'gid': 6054, 'name': 'c', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/251375557', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6835', 'nonce': 251375557}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 251375557}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6054, 'qdb_cluster': [6054]} 2024-03-06T07:24:16.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:16.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:24:16.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.127+0000 7f3f6bb8a640 1 Processor -- start 2024-03-06T07:24:16.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.128+0000 7f3f6bb8a640 1 -- start start 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f6bb8a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f64162970 0x7f3f640d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f6bb8a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640d22c0 con 0x7f3f64161cf0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f6bb8a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640d25a0 con 0x7f3f640b4460 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f6bb8a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f64161b00 con 0x7f3f64162970 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f69125640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f64162970 0x7f3f640d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f69125640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f64162970 0x7f3f640d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56866/0 (socket says 172.21.15.77:56866) 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f69125640 1 -- 172.21.15.77:0/1259101685 learned_addr learned my addr 172.21.15.77:0/1259101685 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f6a127640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3f64161cf0 0x7f3f640cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54944/0 (socket says 172.21.15.77:54944) 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.129+0000 7f3f69125640 1 -- 172.21.15.77:0/1259101685 >> v1:172.21.15.158:6789/0 conn(0x7f3f640b4460 legacy=0x7f3f640b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.130+0000 7f3f69125640 1 -- 172.21.15.77:0/1259101685 >> v1:172.21.15.77:6789/0 conn(0x7f3f64161cf0 legacy=0x7f3f640cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.130+0000 7f3f68924640 1 -- 172.21.15.77:0/1259101685 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400910518 0 0) 0x7f3f640d22c0 con 0x7f3f64161cf0 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.130+0000 7f3f69125640 1 -- 172.21.15.77:0/1259101685 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f640d2880 con 0x7f3f64162970 2024-03-06T07:24:16.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f69125640 1 --2- 172.21.15.77:0/1259101685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64162970 0x7f3f640d4800 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009870 tx=0x7f3f5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba048be2190fe28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f68924640 1 -- 172.21.15.77:0/1259101685 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54034070 con 0x7f3f64162970 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f68924640 1 -- 172.21.15.77:0/1259101685 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54030860 con 0x7f3f64162970 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f68924640 1 -- 172.21.15.77:0/1259101685 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54030c30 con 0x7f3f64162970 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/1259101685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64162970 msgr2=0x7f3f640d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f6bb8a640 1 --2- 172.21.15.77:0/1259101685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64162970 0x7f3f640d4800 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009870 tx=0x7f3f5402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/1259101685 shutdown_connections 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f6bb8a640 1 --2- 172.21.15.77:0/1259101685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64162970 0x7f3f640d4800 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/1259101685 >> 172.21.15.77:0/1259101685 conn(0x7f3f640d3ce0 msgr2=0x7f3f640d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/1259101685 shutdown_connections 2024-03-06T07:24:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.131+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/1259101685 wait complete. 2024-03-06T07:24:16.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.133+0000 7f3f6bb8a640 1 Processor -- start 2024-03-06T07:24:16.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.133+0000 7f3f6bb8a640 1 -- start start 2024-03-06T07:24:16.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.134+0000 7f3f6bb8a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f640b4460 0x7f3f640c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.134+0000 7f3f6bb8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64161cf0 0x7f3f640c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.134+0000 7f3f6bb8a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f64162970 0x7f3f641565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.134+0000 7f3f6bb8a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640ce950 con 0x7f3f64161cf0 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.134+0000 7f3f6bb8a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640cecc0 con 0x7f3f640b4460 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.134+0000 7f3f6bb8a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640b4840 con 0x7f3f64162970 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64161cf0 0x7f3f640c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64161cf0 0x7f3f640c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56870/0 (socket says 172.21.15.77:56870) 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 -- 172.21.15.77:0/3372164242 learned_addr learned my addr 172.21.15.77:0/3372164242 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69926640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f640b4460 0x7f3f640c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f6a127640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f64162970 0x7f3f641565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 -- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f64162970 msgr2=0x7f3f641565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f64162970 0x7f3f641565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 -- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f640b4460 msgr2=0x7f3f640c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f640b4460 0x7f3f640c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 -- 172.21.15.77:0/3372164242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64156b00 con 0x7f3f64161cf0 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69926640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f640b4460 0x7f3f640c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f6a127640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f64162970 0x7f3f641565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f69125640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64161cf0 0x7f3f640c7810 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009870 tx=0x7f3f540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f5a7fc640 1 -- 172.21.15.77:0/3372164242 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54034040 con 0x7f3f64161cf0 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f5a7fc640 1 -- 172.21.15.77:0/3372164242 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54030660 con 0x7f3f64161cf0 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/3372164242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f640b76d0 con 0x7f3f64161cf0 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f5a7fc640 1 -- 172.21.15.77:0/3372164242 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54039d20 con 0x7f3f64161cf0 2024-03-06T07:24:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.135+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/3372164242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f640c8420 con 0x7f3f64161cf0 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.137+0000 7f3f5a7fc640 1 -- 172.21.15.77:0/3372164242 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f54037070 con 0x7f3f64161cf0 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.137+0000 7f3f5a7fc640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f4806b4d0 0x7f3f4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.137+0000 7f3f5a7fc640 1 -- 172.21.15.77:0/3372164242 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f540ac180 con 0x7f3f64161cf0 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.137+0000 7f3f69926640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f4806b4d0 0x7f3f4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.138+0000 7f3f69926640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f4806b4d0 0x7f3f4806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c009fc0 tx=0x7f3f5c009550 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.138+0000 7f3f6e775740 1 Processor -- start 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.138+0000 7f3f6e775740 1 -- start start 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78714870 0x562c78714c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c787188c0 0x562c7871a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c78744600 0x562c78744a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f6e775740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7867e6b0 con 0x562c78714870 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f6e775740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c78627a60 con 0x562c78744600 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f6e775740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x562c7867a750 con 0x562c787188c0 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78714870 0x562c78714c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78714870 0x562c78714c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56884/0 (socket says 172.21.15.77:56884) 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 -- 172.21.15.77:0/3039023537 learned_addr learned my addr 172.21.15.77:0/3039023537 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69125640 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c787188c0 0x562c7871a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f6a127640 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c78744600 0x562c78744a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 -- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c787188c0 msgr2=0x562c7871a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c787188c0 0x562c7871a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 -- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c78744600 msgr2=0x562c78744a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c78744600 0x562c78744a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 -- 172.21.15.77:0/3039023537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c78645f30 con 0x562c78714870 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f6a127640 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c78744600 0x562c78744a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69125640 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c787188c0 0x562c7871a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.139+0000 7f3f69926640 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78714870 0x562c78714c50 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c018800 tx=0x7f3f5c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0e9e8b8b5e9aad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f477fe640 1 -- 172.21.15.77:0/3039023537 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c018030 con 0x562c78714870 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f477fe640 1 -- 172.21.15.77:0/3039023537 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c0181d0 con 0x562c78714870 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f477fe640 1 -- 172.21.15.77:0/3039023537 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c024430 con 0x562c78714870 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 -- 172.21.15.77:0/3039023537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78714870 msgr2=0x562c78714c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78714870 0x562c78714c50 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c018800 tx=0x7f3f5c011000 comp rx=0 tx=0).stop 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 -- 172.21.15.77:0/3039023537 shutdown_connections 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c78744600 0x562c78744a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c787188c0 0x562c7871a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3039023537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78714870 0x562c78714c50 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 -- 172.21.15.77:0/3039023537 >> 172.21.15.77:0/3039023537 conn(0x562c787168f0 msgr2=0x562c787181b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 -- 172.21.15.77:0/3039023537 shutdown_connections 2024-03-06T07:24:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.140+0000 7f3f6e775740 1 -- 172.21.15.77:0/3039023537 wait complete. 2024-03-06T07:24:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.141+0000 7f3f6e775740 1 -- start start 2024-03-06T07:24:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.141+0000 7f3f6e775740 1 Processor -- start 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7875d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78760120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c7875c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f6e775740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c78718d50 con 0x562c7873d010 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f6e775740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7863aea0 con 0x562c7875c500 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f6e775740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x562c78719030 con 0x562c78744600 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f6e775740 10 client.0 _ll_register_callbacks cb 0x562c7863a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78760120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78760120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56894/0 (socket says 172.21.15.77:56894) 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 -- 172.21.15.77:0/2958394138 learned_addr learned my addr 172.21.15.77:0/2958394138 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69926640 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7875d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f6a127640 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c7875c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 -- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 msgr2=0x562c7875d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7875d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 -- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 msgr2=0x562c7875c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c7875c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 -- 172.21.15.77:0/2958394138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c7863b270 con 0x562c7873d010 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69926640 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7875d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f69125640 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78760120 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3f540090b0 tx=0x7f3f5407a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890a6b4d71ee0504 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.142+0000 7f3f44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f44ff9640 1 -- 172.21.15.77:0/2958394138 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5407bda0 con 0x562c7873d010 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f44ff9640 1 -- 172.21.15.77:0/2958394138 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f54079cc0 con 0x562c7873d010 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 1 -- 172.21.15.77:0/2958394138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f5c0181d0 con 0x562c7873d010 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f44ff9640 1 -- 172.21.15.77:0/2958394138 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f5407a870 con 0x562c7873d010 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 10 client.6192 fetch_fsmap learned FSMap version 26 2024-03-06T07:24:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 1 -- 172.21.15.77:0/2958394138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x562c783932d0 con 0x562c7873d010 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f44ff9640 1 -- 172.21.15.77:0/2958394138 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c0181d0 con 0x562c7873d010 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f44ff9640 1 -- 172.21.15.77:0/2958394138 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3f540aa160 con 0x562c7873d010 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 10 client.6192 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 10 client.6192 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 20 client.6192 populate_metadata read hostname 'smithi077' 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] conn(0x562c78747d20 0x562c7877e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 4 client.6192 mds_command: new command op to 6054 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f6e775740 1 -- 172.21.15.77:0/2958394138 --> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c7868bbf0 con 0x562c78747d20 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f69926640 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] conn(0x562c78747d20 0x562c7877e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f44ff9640 10 client.6192 ms_handle_connect on v2:172.21.15.77:6835/251375557 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.143+0000 7f3f69926640 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] conn(0x562c78747d20 0x562c7877e6a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdce12db513a95d5 server_cookie=e2fca613e607b71a in_seq=0 out_seq=0 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f44ff9640 1 -- 172.21.15.77:0/2958394138 <== mds.0 v2:172.21.15.77:6835/251375557 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562c7868bbf0 con 0x562c78747d20 2024-03-06T07:24:16.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f44ff9640 10 client.6192 handle_command_reply: tid=0 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 client.6192 shutdown 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 -- 172.21.15.77:0/2958394138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 msgr2=0x562c78760120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78760120 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3f540090b0 tx=0x7f3f5407a290 comp rx=0 tx=0).stop 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 -- 172.21.15.77:0/2958394138 shutdown_connections 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] conn(0x562c78747d20 0x562c7877e6a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78760120 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f44ff9640 0 client.6192 ms_handle_reset on v2:172.21.15.77:6835/251375557 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c7875c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 --2- 172.21.15.77:0/2958394138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7875d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 -- 172.21.15.77:0/2958394138 >> 172.21.15.77:0/2958394138 conn(0x562c78714870 msgr2=0x562c787425c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.144+0000 7f3f6e775740 1 -- 172.21.15.77:0/2958394138 shutdown_connections 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.145+0000 7f3f6e775740 1 -- 172.21.15.77:0/2958394138 wait complete. 2024-03-06T07:24:16.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.145+0000 7f3f6e775740 20 client.6192 trim_cache size 0 max 16384 2024-03-06T07:24:16.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.176+0000 7f3f6e775740 1 Processor -- start 2024-03-06T07:24:16.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.176+0000 7f3f6e775740 1 -- start start 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c7873d010 0x562c7875d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78744600 0x562c7875db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787449e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f6e775740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f641fffb0 con 0x562c78744600 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f6e775740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f641ff110 con 0x562c7875c500 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f6e775740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640cc710 con 0x562c7873d010 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78744600 0x562c7875db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78744600 0x562c7875db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56910/0 (socket says 172.21.15.77:56910) 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 -- 172.21.15.77:0/3294562766 learned_addr learned my addr 172.21.15.77:0/3294562766 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69926640 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c7873d010 0x562c7875d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 -- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c7873d010 msgr2=0x562c7875d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c7873d010 0x562c7875d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f6a127640 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 -- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 msgr2=0x562c787449e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787449e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 -- 172.21.15.77:0/3294562766 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c7845d6b0 con 0x562c78744600 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69926640 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c7873d010 0x562c7875d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f6a127640 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f69125640 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78744600 0x562c7875db10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3f54078da0 tx=0x7f3f5404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8998ac8ce6a782e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f46ffd640 1 -- 172.21.15.77:0/3294562766 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54083010 con 0x562c78744600 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f46ffd640 1 -- 172.21.15.77:0/3294562766 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5407bba0 con 0x562c78744600 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.178+0000 7f3f46ffd640 1 -- 172.21.15.77:0/3294562766 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54079cc0 con 0x562c78744600 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 -- 172.21.15.77:0/3294562766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78744600 msgr2=0x562c7875db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78744600 0x562c7875db10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3f54078da0 tx=0x7f3f5404e000 comp rx=0 tx=0).stop 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 -- 172.21.15.77:0/3294562766 shutdown_connections 2024-03-06T07:24:16.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c78744600 0x562c7875db10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3294562766 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c7873d010 0x562c7875d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 -- 172.21.15.77:0/3294562766 >> 172.21.15.77:0/3294562766 conn(0x562c78714870 msgr2=0x562c7870fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 -- 172.21.15.77:0/3294562766 shutdown_connections 2024-03-06T07:24:16.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.179+0000 7f3f6e775740 1 -- 172.21.15.77:0/3294562766 wait complete. 2024-03-06T07:24:16.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.180+0000 7f3f6e775740 1 -- start start 2024-03-06T07:24:16.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.180+0000 7f3f6e775740 1 Processor -- start 2024-03-06T07:24:16.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.181+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78742300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.181+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7873bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.181+0000 7f3f6e775740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.181+0000 7f3f6e775740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7875e610 con 0x562c7873d010 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.181+0000 7f3f6e775740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f641feb10 con 0x562c7875c500 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.181+0000 7f3f6e775740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x562c78747050 con 0x562c78744600 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78742300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f6e775740 10 client.0 _ll_register_callbacks cb 0x7f3f54039ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78742300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56922/0 (socket says 172.21.15.77:56922) 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 -- 172.21.15.77:0/3782916850 learned_addr learned my addr 172.21.15.77:0/3782916850 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69125640 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7873bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f6a127640 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 -- 172.21.15.77:0/3782916850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 msgr2=0x562c7873bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7873bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 -- 172.21.15.77:0/3782916850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 msgr2=0x562c787ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 -- 172.21.15.77:0/3782916850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c7875a010 con 0x562c7873d010 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f69926640 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78742300 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x562c78748cf0 tx=0x7f3f5c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e82b6c6156643f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f44ff9640 1 -- 172.21.15.77:0/3782916850 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c031170 con 0x562c7873d010 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f44ff9640 1 -- 172.21.15.77:0/3782916850 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f5c01e040 con 0x562c7873d010 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f6e775740 1 -- 172.21.15.77:0/3782916850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f5407bba0 con 0x562c7873d010 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f44ff9640 1 -- 172.21.15.77:0/3782916850 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f5c040050 con 0x562c7873d010 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f6e775740 10 client.6198 fetch_fsmap learned FSMap version 26 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f6e775740 1 -- 172.21.15.77:0/3782916850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x562c7875ad70 con 0x562c7873d010 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.182+0000 7f3f44ff9640 1 -- 172.21.15.77:0/3782916850 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5407bba0 con 0x562c7873d010 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f44ff9640 1 -- 172.21.15.77:0/3782916850 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3f5c043020 con 0x562c7873d010 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 10 client.6198 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:24:16.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 10 client.6198 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 20 client.6198 populate_metadata read hostname 'smithi077' 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] conn(0x562c787450a0 0x562c78772070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 4 client.6198 mds_command: new command op to 6054 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 -- 172.21.15.77:0/3782916850 --> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562c78717050 con 0x562c787450a0 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f69125640 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] conn(0x562c787450a0 0x562c78772070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f44ff9640 10 client.6198 ms_handle_connect on v2:172.21.15.77:6835/251375557 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f69125640 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] conn(0x562c787450a0 0x562c78772070 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c6bfe553a5e067e server_cookie=46bf9a10ff3a8b15 in_seq=0 out_seq=0 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f44ff9640 1 -- 172.21.15.77:0/3782916850 <== mds.0 v2:172.21.15.77:6835/251375557 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x562c78717050 con 0x562c787450a0 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f44ff9640 10 client.6198 handle_command_reply: tid=0 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 client.6198 shutdown 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 -- 172.21.15.77:0/3782916850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 msgr2=0x562c78742300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78742300 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x562c78748cf0 tx=0x7f3f5c016000 comp rx=0 tx=0).stop 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 -- 172.21.15.77:0/3782916850 shutdown_connections 2024-03-06T07:24:16.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.77:6835/251375557,v1:172.21.15.77:6837/251375557] conn(0x562c787450a0 0x562c78772070 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562c7875c500 0x562c787ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x562c78744600 0x562c7873bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f44ff9640 0 client.6198 ms_handle_reset on v2:172.21.15.77:6835/251375557 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 --2- 172.21.15.77:0/3782916850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562c7873d010 0x562c78742300 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.183+0000 7f3f6e775740 1 -- 172.21.15.77:0/3782916850 >> 172.21.15.77:0/3782916850 conn(0x562c78714870 msgr2=0x562c78756fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6e775740 1 -- 172.21.15.77:0/3782916850 shutdown_connections 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6e775740 1 -- 172.21.15.77:0/3782916850 wait complete. 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6e775740 20 client.6198 trim_cache size 0 max 16384 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:24:16.191 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 6054, 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 4.7769272660000004, 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 58, 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 58, 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 9.8333572230000001 2024-03-06T07:24:16.192 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:24:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f4806b4d0 msgr2=0x7f3f4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f4806b4d0 0x7f3f4806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c009fc0 tx=0x7f3f5c009550 comp rx=0 tx=0).stop 2024-03-06T07:24:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64161cf0 msgr2=0x7f3f640c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64161cf0 0x7f3f640c7810 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009870 tx=0x7f3f540046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/3372164242 shutdown_connections 2024-03-06T07:24:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f4806b4d0 0x7f3f4806d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f64161cf0 0x7f3f640c7810 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f64162970 0x7f3f641565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 --2- 172.21.15.77:0/3372164242 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f640b4460 0x7f3f640c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/3372164242 >> 172.21.15.77:0/3372164242 conn(0x7f3f640d3ce0 msgr2=0x7f3f640d0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/3372164242 shutdown_connections 2024-03-06T07:24:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.184+0000 7f3f6bb8a640 1 -- 172.21.15.77:0/3372164242 wait complete. 2024-03-06T07:24:16.197 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:16.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.296+0000 7fadbc7eb640 1 Processor -- start 2024-03-06T07:24:16.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.296+0000 7fadbc7eb640 1 -- start start 2024-03-06T07:24:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.297+0000 7fadbc7eb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fadb415dfe0 0x7fadb40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.297+0000 7fadbc7eb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fadb40d29c0 0x7fadb40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.297+0000 7fadbc7eb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb40b3430 con 0x7fadb40cf830 2024-03-06T07:24:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.297+0000 7fadbc7eb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb40b31e0 con 0x7fadb415dfe0 2024-03-06T07:24:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.297+0000 7fadbc7eb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb40b3d60 con 0x7fadb40d29c0 2024-03-06T07:24:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.297+0000 7fadba587640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fadb415dfe0 0x7fadb40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadb9d86640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fadb40cf830 0x7fadb40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54946/0 (socket says 172.21.15.77:54946) 2024-03-06T07:24:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadba587640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fadb415dfe0 0x7fadb40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56928/0 (socket says 172.21.15.77:56928) 2024-03-06T07:24:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadb9d86640 1 -- 172.21.15.77:0/2360501059 learned_addr learned my addr 172.21.15.77:0/2360501059 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadbad88640 1 --2- 172.21.15.77:0/2360501059 >> v2:172.21.15.158:3300/0 conn(0x7fadb40d29c0 0x7fadb40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadba587640 1 -- 172.21.15.77:0/2360501059 >> v2:172.21.15.158:3300/0 conn(0x7fadb40d29c0 msgr2=0x7fadb40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadba587640 1 --2- 172.21.15.77:0/2360501059 >> v2:172.21.15.158:3300/0 conn(0x7fadb40d29c0 0x7fadb40b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadba587640 1 -- 172.21.15.77:0/2360501059 >> v1:172.21.15.77:6789/0 conn(0x7fadb40cf830 legacy=0x7fadb40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadba587640 1 -- 172.21.15.77:0/2360501059 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb40b3710 con 0x7fadb415dfe0 2024-03-06T07:24:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.298+0000 7fadb9585640 1 -- 172.21.15.77:0/2360501059 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699662953 0 0) 0x7fadb40b3430 con 0x7fadb40cf830 2024-03-06T07:24:16.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadba587640 1 --2- 172.21.15.77:0/2360501059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb415dfe0 0x7fadb40cf2f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fadb40b3cd0 tx=0x7fadb002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212fbb8b04c4fc85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadb9585640 1 -- 172.21.15.77:0/2360501059 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0035020 con 0x7fadb415dfe0 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadb9585640 1 -- 172.21.15.77:0/2360501059 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb0031830 con 0x7fadb415dfe0 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadb9585640 1 -- 172.21.15.77:0/2360501059 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0031c00 con 0x7fadb415dfe0 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadbc7eb640 1 -- 172.21.15.77:0/2360501059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb415dfe0 msgr2=0x7fadb40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/2360501059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb415dfe0 0x7fadb40cf2f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fadb40b3cd0 tx=0x7fadb002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadbc7eb640 1 -- 172.21.15.77:0/2360501059 shutdown_connections 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/2360501059 >> v2:172.21.15.158:3300/0 conn(0x7fadb40d29c0 0x7fadb40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/2360501059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb415dfe0 0x7fadb40cf2f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadbc7eb640 1 -- 172.21.15.77:0/2360501059 >> 172.21.15.77:0/2360501059 conn(0x7fadb40d3cd0 msgr2=0x7fadb40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadbc7eb640 1 -- 172.21.15.77:0/2360501059 shutdown_connections 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.299+0000 7fadbc7eb640 1 -- 172.21.15.77:0/2360501059 wait complete. 2024-03-06T07:24:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.301+0000 7fadbc7eb640 1 Processor -- start 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.301+0000 7fadbc7eb640 1 -- start start 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.302+0000 7fadbc7eb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadb40cf830 0x7fadb414bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.302+0000 7fadbc7eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb40d29c0 0x7fadb4148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.302+0000 7fadbc7eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadb415dfe0 0x7fadb40b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.302+0000 7fadbc7eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb40d1df0 con 0x7fadb40d29c0 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.302+0000 7fadbc7eb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb40d1c70 con 0x7fadb40cf830 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.302+0000 7fadbc7eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb415e3c0 con 0x7fadb415dfe0 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb40d29c0 0x7fadb4148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb40d29c0 0x7fadb4148c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56936/0 (socket says 172.21.15.77:56936) 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 -- 172.21.15.77:0/99633342 learned_addr learned my addr 172.21.15.77:0/99633342 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadba587640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadb40cf830 0x7fadb414bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadbad88640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadb415dfe0 0x7fadb40b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 -- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadb415dfe0 msgr2=0x7fadb40b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadb415dfe0 0x7fadb40b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 -- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadb40cf830 msgr2=0x7fadb414bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadb40cf830 0x7fadb414bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 -- 172.21.15.77:0/99633342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb414c470 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadba587640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadb40cf830 0x7fadb414bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadb9d86640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb40d29c0 0x7fadb4148c70 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fada400c4e0 tx=0x7fada400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadaaffd640 1 -- 172.21.15.77:0/99633342 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada4018070 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadaaffd640 1 -- 172.21.15.77:0/99633342 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada4007500 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb41491b0 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadaaffd640 1 -- 172.21.15.77:0/99633342 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada4004a20 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.303+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb40b6e50 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.304+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb404f330 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.305+0000 7fadaaffd640 1 -- 172.21.15.77:0/99633342 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fada401b070 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.305+0000 7fadaaffd640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9806b4d0 0x7fad9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.306+0000 7fadaaffd640 1 -- 172.21.15.77:0/99633342 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fada40894d0 con 0x7fadb40d29c0 2024-03-06T07:24:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.306+0000 7fadba587640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9806b4d0 0x7fad9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.306+0000 7fadba587640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9806b4d0 0x7fad9806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fadb0036000 tx=0x7fadb003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.310+0000 7fadaaffd640 1 -- 172.21.15.77:0/99633342 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadb41491b0 con 0x7fadb40d29c0 2024-03-06T07:24:16.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.534+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fadb41e6cb0 con 0x7fadb40d29c0 2024-03-06T07:24:16.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.535+0000 7fadaaffd640 1 -- 172.21.15.77:0/99633342 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7fada4087e20 con 0x7fadb40d29c0 2024-03-06T07:24:16.537 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:16.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9806b4d0 msgr2=0x7fad9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9806b4d0 0x7fad9806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fadb0036000 tx=0x7fadb003c040 comp rx=0 tx=0).stop 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb40d29c0 msgr2=0x7fadb4148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb40d29c0 0x7fadb4148c70 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fada400c4e0 tx=0x7fada400c9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 shutdown_connections 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad9806b4d0 0x7fad9806d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadb40d29c0 0x7fadb4148c70 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadb415dfe0 0x7fadb40b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 --2- 172.21.15.77:0/99633342 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadb40cf830 0x7fadb414bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.537+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 >> 172.21.15.77:0/99633342 conn(0x7fadb40d3cd0 msgr2=0x7fadb40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.538+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 shutdown_connections 2024-03-06T07:24:16.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:16.538+0000 7fadbc7eb640 1 -- 172.21.15.77:0/99633342 wait complete. 2024-03-06T07:24:16.550 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:16.550 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:16.550 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:24:16.550 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:16.551 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:16.551 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:16.551 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:16.551 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:16.551 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:16.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:16.551 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42541sec preferred_lft 42541sec 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:16.568 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:16.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:16.570 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:16.632 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:24:16.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:16.634 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:24:16.690 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:24:16.690 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:16.690 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:24:16.690 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:24:16.690 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:16.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:16.691 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:24:16.691 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:24:16.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:24:16.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:24:16.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:24:16.691 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:24:16.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:16.789 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:24:16.804 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:16.804 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:16.804 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:16.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:16.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:24:16.872 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:16.872 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:16.872 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:16.927 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:16.927 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:16.927 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:16.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:24:17.049 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:17.105 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:24:17.105 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:24:17.105 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:24:17.106 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:17.107 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:17.108 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:17.108 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:17.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:17.109 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:24:17.165 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:17.166 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:17.208 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:24:17.235 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:17.291 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:24:17.291 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:24:17.291 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:24:17.291 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:17.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:17.294 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:17.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:17.294 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:24:17.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:24:17.292+0000 7fe0d8237480 -1 init, newargv = 0x5569f769b5d0 newargc=15 2024-03-06T07:24:17.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[58383]: starting ceph client 2024-03-06T07:24:17.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[58383]: starting fuse 2024-03-06T07:24:17.309 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:24:17.310 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:24:17.310 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:17.311 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:24:17.430 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.58383.asok 2024-03-06T07:24:17.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:17.441 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58383.asok 2024-03-06T07:24:17.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:17.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58383.asok status 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "58383", 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:24:17.578 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 6210, 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 6210 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1618858578 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:17.579 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1618858578 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.6210 192.168.0.1:0/1618858578", 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/1618858578", 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 26, 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 58, 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:24:17.580 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:24:17.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:17.586 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:24:17.586 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:17.602 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:24:17.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:17.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:17.602 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:17.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:24:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.812+0000 7f75a6b3e640 1 Processor -- start 2024-03-06T07:24:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.812+0000 7f75a6b3e640 1 -- start start 2024-03-06T07:24:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.813+0000 7f75a6b3e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f75a00d4860 0x7f75a00d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.813+0000 7f75a6b3e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f75a00b0e30 0x7f75a00b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.813+0000 7f75a6b3e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a01618b0 con 0x7f75a01614d0 2024-03-06T07:24:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.813+0000 7f75a6b3e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a00d4cf0 con 0x7f75a00d4860 2024-03-06T07:24:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.813+0000 7f75a6b3e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a00cf400 con 0x7f75a00b0e30 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.813+0000 7f759ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f75a00d4860 0x7f75a00d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.813+0000 7f75a48da640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f75a01614d0 0x7f75a00d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54962/0 (socket says 172.21.15.77:54962) 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.813+0000 7f75a48da640 1 -- 172.21.15.77:0/406034695 learned_addr learned my addr 172.21.15.77:0/406034695 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.814+0000 7f75a50db640 1 --2- 172.21.15.77:0/406034695 >> v2:172.21.15.158:3300/0 conn(0x7f75a00b0e30 0x7f75a00b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.814+0000 7f759ffff640 1 -- 172.21.15.77:0/406034695 >> v2:172.21.15.158:3300/0 conn(0x7f75a00b0e30 msgr2=0x7f75a00b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.814+0000 7f759ffff640 1 --2- 172.21.15.77:0/406034695 >> v2:172.21.15.158:3300/0 conn(0x7f75a00b0e30 0x7f75a00b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.814+0000 7f759ffff640 1 -- 172.21.15.77:0/406034695 >> v1:172.21.15.77:6789/0 conn(0x7f75a01614d0 legacy=0x7f75a00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.814+0000 7f759f7fe640 1 -- 172.21.15.77:0/406034695 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248769320 0 0) 0x7f75a01618b0 con 0x7f75a01614d0 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.814+0000 7f759ffff640 1 -- 172.21.15.77:0/406034695 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a00cf6a0 con 0x7f75a00d4860 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.814+0000 7f75a50db640 1 --2- 172.21.15.77:0/406034695 >> v2:172.21.15.158:3300/0 conn(0x7f75a00b0e30 0x7f75a00b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.814+0000 7f759ffff640 1 --2- 172.21.15.77:0/406034695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a00d4860 0x7f75a00d2020 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f75a00b2a70 tx=0x7f758c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c26d7b2639c05cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f759f7fe640 1 -- 172.21.15.77:0/406034695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c035020 con 0x7f75a00d4860 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f759f7fe640 1 -- 172.21.15.77:0/406034695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c031830 con 0x7f75a00d4860 2024-03-06T07:24:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f759f7fe640 1 -- 172.21.15.77:0/406034695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c031c00 con 0x7f75a00d4860 2024-03-06T07:24:17.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f75a6b3e640 1 -- 172.21.15.77:0/406034695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a00d4860 msgr2=0x7f75a00d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/406034695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a00d4860 0x7f75a00d2020 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f75a00b2a70 tx=0x7f758c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f75a6b3e640 1 -- 172.21.15.77:0/406034695 shutdown_connections 2024-03-06T07:24:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/406034695 >> v2:172.21.15.158:3300/0 conn(0x7f75a00b0e30 0x7f75a00b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/406034695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a00d4860 0x7f75a00d2020 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f75a6b3e640 1 -- 172.21.15.77:0/406034695 >> 172.21.15.77:0/406034695 conn(0x7f75a00d3d40 msgr2=0x7f75a00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f75a6b3e640 1 -- 172.21.15.77:0/406034695 shutdown_connections 2024-03-06T07:24:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.815+0000 7f75a6b3e640 1 -- 172.21.15.77:0/406034695 wait complete. 2024-03-06T07:24:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.816+0000 7f75a6b3e640 1 Processor -- start 2024-03-06T07:24:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.816+0000 7f75a6b3e640 1 -- start start 2024-03-06T07:24:17.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a6b3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75a00b0e30 0x7f75a01521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a6b3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75a00d4860 0x7f75a00c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a6b3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a01614d0 0x7f75a00b8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a6b3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00b0910 con 0x7f75a01614d0 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a6b3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00cfc60 con 0x7f75a00b0e30 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a6b3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00d2560 con 0x7f75a00d4860 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a50db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a01614d0 0x7f75a00b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a50db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a01614d0 0x7f75a00b8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56944/0 (socket says 172.21.15.77:56944) 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.817+0000 7f75a50db640 1 -- 172.21.15.77:0/485714638 learned_addr learned my addr 172.21.15.77:0/485714638 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a48da640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75a00b0e30 0x7f75a01521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f759ffff640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75a00d4860 0x7f75a00c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a50db640 1 -- 172.21.15.77:0/485714638 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75a00d4860 msgr2=0x7f75a00c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a50db640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75a00d4860 0x7f75a00c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a50db640 1 -- 172.21.15.77:0/485714638 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75a00b0e30 msgr2=0x7f75a01521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a50db640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75a00b0e30 0x7f75a01521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a50db640 1 -- 172.21.15.77:0/485714638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a01526e0 con 0x7f75a01614d0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a50db640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a01614d0 0x7f75a00b8430 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f759800cf20 tx=0x7f759800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f759d7fa640 1 -- 172.21.15.77:0/485714638 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759801a070 con 0x7f75a01614d0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f759d7fa640 1 -- 172.21.15.77:0/485714638 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759800d840 con 0x7f75a01614d0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a00c7db0 con 0x7f75a01614d0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f759d7fa640 1 -- 172.21.15.77:0/485714638 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759800dc30 con 0x7f75a01614d0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.818+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a00cab80 con 0x7f75a01614d0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.819+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a00d1c20 con 0x7f75a01614d0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.820+0000 7f759d7fa640 1 -- 172.21.15.77:0/485714638 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f759801d070 con 0x7f75a01614d0 2024-03-06T07:24:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.820+0000 7f759d7fa640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f758406b4d0 0x7f758406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.820+0000 7f759d7fa640 1 -- 172.21.15.77:0/485714638 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f759808aa40 con 0x7f75a01614d0 2024-03-06T07:24:17.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.820+0000 7f75a48da640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f758406b4d0 0x7f758406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.820+0000 7f75a48da640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f758406b4d0 0x7f758406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7594000f60 tx=0x7f7594009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.824+0000 7f759d7fa640 1 -- 172.21.15.77:0/485714638 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7598059270 con 0x7f75a01614d0 2024-03-06T07:24:18.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:17.998+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f75a00cee60 con 0x7f75a01614d0 2024-03-06T07:24:18.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.002+0000 7f759d7fa640 1 -- 172.21.15.77:0/485714638 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7f759808ae20 con 0x7f75a01614d0 2024-03-06T07:24:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f758406b4d0 msgr2=0x7f758406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f758406b4d0 0x7f758406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7594000f60 tx=0x7f7594009040 comp rx=0 tx=0).stop 2024-03-06T07:24:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a01614d0 msgr2=0x7f75a00b8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a01614d0 0x7f75a00b8430 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f759800cf20 tx=0x7f759800a7d0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 shutdown_connections 2024-03-06T07:24:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f758406b4d0 0x7f758406d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75a01614d0 0x7f75a00b8430 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75a00d4860 0x7f75a00c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 --2- 172.21.15.77:0/485714638 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75a00b0e30 0x7f75a01521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 >> 172.21.15.77:0/485714638 conn(0x7f75a00d3d40 msgr2=0x7f75a015fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 shutdown_connections 2024-03-06T07:24:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.004+0000 7f75a6b3e640 1 -- 172.21.15.77:0/485714638 wait complete. 2024-03-06T07:24:18.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:24:18.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.119+0000 7f727915e640 1 Processor -- start 2024-03-06T07:24:18.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.119+0000 7f727915e640 1 -- start start 2024-03-06T07:24:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727915e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72740d4860 0x7f72740d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727915e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72741618b0 con 0x7f72740b0e30 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727915e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72740cf400 con 0x7f72741614d0 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727915e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72740d4cf0 con 0x7f72740d4860 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727359e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f72740b0e30 0x7f72740b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54972/0 (socket says 172.21.15.77:54972) 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727359e640 1 -- 172.21.15.77:0/3816063091 learned_addr learned my addr 172.21.15.77:0/3816063091 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f726bfff640 1 --2- 172.21.15.77:0/3816063091 >> v2:172.21.15.158:3300/0 conn(0x7f72740d4860 0x7f72740d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745009084 0 0) 0x7f72741618b0 con 0x7f72740b0e30 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7260003520 con 0x7f72740b0e30 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105192436 0 0) 0x7f7260003520 con 0x7f72740b0e30 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 >> v2:172.21.15.158:3300/0 conn(0x7f72740d4860 msgr2=0x7f72740d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 --2- 172.21.15.77:0/3816063091 >> v2:172.21.15.158:3300/0 conn(0x7f72740d4860 0x7f72740d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 >> v1:172.21.15.158:6789/0 conn(0x7f72741614d0 legacy=0x7f72740d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72740cf6a0 con 0x7f72740b0e30 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f726c001a10 con 0x7f72740b0e30 2024-03-06T07:24:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 >> v1:172.21.15.77:6789/0 conn(0x7f72740b0e30 legacy=0x7f72740b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72600039b0 0x7f7260003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7260004800 0x7f7260004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600051c0 0x7f726000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72740d4cf0 con 0x7f72600039b0 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726000c060 con 0x7f7260004800 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.120+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726000c260 con 0x7f72600051c0 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f7272d9d640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72600039b0 0x7f7260003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f726bfff640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7260004800 0x7f7260004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f727359e640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600051c0 0x7f726000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f7272d9d640 1 -- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600051c0 msgr2=0x7f726000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f7272d9d640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600051c0 0x7f726000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f7272d9d640 1 -- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7260004800 msgr2=0x7f7260004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f7272d9d640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7260004800 0x7f7260004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f7272d9d640 1 -- 172.21.15.77:0/3816063091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7260003520 con 0x7f72600039b0 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f727359e640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600051c0 0x7f726000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.121+0000 7f727359e640 1 -- 172.21.15.77:0/3816063091 reap_dead start 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f7272d9d640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72600039b0 0x7f7260003dd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7264001750 tx=0x7f7260005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2293f1e3b38bfd07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7264001370 con 0x7f72600039b0 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72640052d0 con 0x7f72600039b0 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727259c640 1 -- 172.21.15.77:0/3816063091 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7264003830 con 0x7f72600039b0 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727915e640 1 -- 172.21.15.77:0/3816063091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72600039b0 msgr2=0x7f7260003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727915e640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72600039b0 0x7f7260003dd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7264001750 tx=0x7f7260005830 comp rx=0 tx=0).stop 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727915e640 1 -- 172.21.15.77:0/3816063091 shutdown_connections 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727915e640 1 --2- 172.21.15.77:0/3816063091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72600039b0 0x7f7260003dd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727915e640 1 -- 172.21.15.77:0/3816063091 >> 172.21.15.77:0/3816063091 conn(0x7f72740d3d40 msgr2=0x7f72740d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727915e640 1 -- 172.21.15.77:0/3816063091 shutdown_connections 2024-03-06T07:24:18.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.122+0000 7f727915e640 1 -- 172.21.15.77:0/3816063091 wait complete. 2024-03-06T07:24:18.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727915e640 1 Processor -- start 2024-03-06T07:24:18.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727915e640 1 -- start start 2024-03-06T07:24:18.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727915e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600039b0 0x7f7274154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727915e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72740cfea0 0x7f72740d0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727915e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72740c9a40 0x7f72740c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727915e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72740b0910 con 0x7f72740c9a40 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727915e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72740cfc60 con 0x7f72740cfea0 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727915e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72740d1910 con 0x7f72600039b0 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72740c9a40 0x7f72740c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72740c9a40 0x7f72740c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56960/0 (socket says 172.21.15.77:56960) 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.124+0000 7f727359e640 1 -- 172.21.15.77:0/2940352043 learned_addr learned my addr 172.21.15.77:0/2940352043 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f726bfff640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72740cfea0 0x7f72740d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f7272d9d640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600039b0 0x7f7274154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f727359e640 1 -- 172.21.15.77:0/2940352043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600039b0 msgr2=0x7f7274154130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f727359e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600039b0 0x7f7274154130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f727359e640 1 -- 172.21.15.77:0/2940352043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72740cfea0 msgr2=0x7f72740d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f727359e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72740cfea0 0x7f72740d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f727359e640 1 -- 172.21.15.77:0/2940352043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72740ccda0 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f727359e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72740c9a40 0x7f72740c9ea0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f726c001370 tx=0x7f726c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f726b7fe640 1 -- 172.21.15.77:0/2940352043 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c00eb90 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f726b7fe640 1 -- 172.21.15.77:0/2940352043 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c00ed50 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72740cbfe0 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f726b7fe640 1 -- 172.21.15.77:0/2940352043 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0146c0 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.125+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72740cc5a0 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.126+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7274068ff0 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.127+0000 7f726b7fe640 1 -- 172.21.15.77:0/2940352043 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f726c01b060 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.127+0000 7f726b7fe640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f725406b500 0x7f725406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.127+0000 7f726b7fe640 1 -- 172.21.15.77:0/2940352043 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f726c089680 con 0x7f72740c9a40 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.127+0000 7f7272d9d640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f725406b500 0x7f725406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.128+0000 7f7272d9d640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f725406b500 0x7f725406d9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7264033000 tx=0x7f7264009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.135+0000 7f726b7fe640 1 -- 172.21.15.77:0/2940352043 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f726c058590 con 0x7f72740c9a40 2024-03-06T07:24:18.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.311+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f72740008d0 con 0x7f72740c9a40 2024-03-06T07:24:18.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.314+0000 7f726b7fe640 1 -- 172.21.15.77:0/2940352043 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f726c057e90 con 0x7f72740c9a40 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f725406b500 msgr2=0x7f725406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f725406b500 0x7f725406d9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7264033000 tx=0x7f7264009030 comp rx=0 tx=0).stop 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72740c9a40 msgr2=0x7f72740c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72740c9a40 0x7f72740c9ea0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f726c001370 tx=0x7f726c00e530 comp rx=0 tx=0).stop 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 shutdown_connections 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f725406b500 0x7f725406d9c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72740c9a40 0x7f72740c9ea0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72740cfea0 0x7f72740d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 --2- 172.21.15.77:0/2940352043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72600039b0 0x7f7274154130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 >> 172.21.15.77:0/2940352043 conn(0x7f72740d3d40 msgr2=0x7f72740b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 shutdown_connections 2024-03-06T07:24:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:18.317+0000 7f727915e640 1 -- 172.21.15.77:0/2940352043 wait complete. 2024-03-06T07:24:28.330 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:28.330 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> import uuid 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> n = 50 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = False 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> if 0 > 0: 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = True 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> dpath = os.path.dirname(path) 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:28.346 DEBUG:teuthology.orchestra.run.smithi077:> try: 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> for i in range(n): 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> f.write(f"{i}") 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> f.flush() 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(f.fileno()) 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> os.unlink(fpath) 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> if create_hard_links: 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> for j in range(0): 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:28.347 DEBUG:teuthology.orchestra.run.smithi077:> if True: 2024-03-06T07:24:28.348 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:24:28.348 DEBUG:teuthology.orchestra.run.smithi077:> finally: 2024-03-06T07:24:28.348 DEBUG:teuthology.orchestra.run.smithi077:> os.close(dirfd) 2024-03-06T07:24:28.348 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:24:28.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:28.563 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:28.563 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:28.619 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:28.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.758+0000 7f1753d1e640 1 Processor -- start 2024-03-06T07:24:28.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.758+0000 7f1753d1e640 1 -- start start 2024-03-06T07:24:28.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f1753d1e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c13dc10 0x7f174c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f1753d1e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f174c15bf30 0x7f174c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f1753d1e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f174c15e850 0x7f174c15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f1753d1e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f174c0cf3f0 con 0x7f174c15bf30 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f1753d1e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f174c15b210 con 0x7f174c15e850 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f1753d1e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c15f580 con 0x7f174c13dc10 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17512b9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f174c15bf30 0x7f174c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17512b9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f174c15bf30 0x7f174c0b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46074/0 (socket says 172.21.15.77:46074) 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17512b9640 1 -- 172.21.15.77:0/1813850033 learned_addr learned my addr 172.21.15.77:0/1813850033 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17522bb640 1 --2- 172.21.15.77:0/1813850033 >> v2:172.21.15.158:3300/0 conn(0x7f174c15e850 0x7f174c15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f1751aba640 1 --2- 172.21.15.77:0/1813850033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c13dc10 0x7f174c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17512b9640 1 -- 172.21.15.77:0/1813850033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c13dc10 msgr2=0x7f174c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17512b9640 1 --2- 172.21.15.77:0/1813850033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c13dc10 0x7f174c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17512b9640 1 -- 172.21.15.77:0/1813850033 >> v2:172.21.15.158:3300/0 conn(0x7f174c15e850 msgr2=0x7f174c15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17512b9640 1 --2- 172.21.15.77:0/1813850033 >> v2:172.21.15.158:3300/0 conn(0x7f174c15e850 0x7f174c15eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17512b9640 1 -- 172.21.15.77:0/1813850033 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c15b450 con 0x7f174c15bf30 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f17522bb640 1 --2- 172.21.15.77:0/1813850033 >> v2:172.21.15.158:3300/0 conn(0x7f174c15e850 0x7f174c15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.759+0000 7f1751aba640 1 --2- 172.21.15.77:0/1813850033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c13dc10 0x7f174c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:28.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.760+0000 7f17512b9640 1 --2- 172.21.15.77:0/1813850033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c0b2e90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f174c06c790 tx=0x7f173c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3fe5ae11032e39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1750ab8640 1 -- 172.21.15.77:0/1813850033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c0317c0 con 0x7f174c15bf30 2024-03-06T07:24:28.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1750ab8640 1 -- 172.21.15.77:0/1813850033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c037070 con 0x7f174c15bf30 2024-03-06T07:24:28.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1750ab8640 1 -- 172.21.15.77:0/1813850033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c031b80 con 0x7f174c15bf30 2024-03-06T07:24:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 -- 172.21.15.77:0/1813850033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 msgr2=0x7f174c0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 --2- 172.21.15.77:0/1813850033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c0b2e90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f174c06c790 tx=0x7f173c02b860 comp rx=0 tx=0).stop 2024-03-06T07:24:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 -- 172.21.15.77:0/1813850033 shutdown_connections 2024-03-06T07:24:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 --2- 172.21.15.77:0/1813850033 >> v2:172.21.15.158:3300/0 conn(0x7f174c15e850 0x7f174c15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 --2- 172.21.15.77:0/1813850033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c0b2e90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 --2- 172.21.15.77:0/1813850033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c13dc10 0x7f174c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 -- 172.21.15.77:0/1813850033 >> 172.21.15.77:0/1813850033 conn(0x7f174c0cf8a0 msgr2=0x7f174c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 -- 172.21.15.77:0/1813850033 shutdown_connections 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.761+0000 7f1753d1e640 1 -- 172.21.15.77:0/1813850033 wait complete. 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.763+0000 7f1753d1e640 1 Processor -- start 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.763+0000 7f1753d1e640 1 -- start start 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f1753d1e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c13dc10 0x7f174c1546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f1753d1e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f1753d1e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c15e850 0x7f174c13a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f1753d1e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c15f1e0 con 0x7f174c15bf30 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f1753d1e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c13dff0 con 0x7f174c13dc10 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f1753d1e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c15f360 con 0x7f174c15e850 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f17512b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f17512b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c139c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46076/0 (socket says 172.21.15.77:46076) 2024-03-06T07:24:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.764+0000 7f17512b9640 1 -- 172.21.15.77:0/646268818 learned_addr learned my addr 172.21.15.77:0/646268818 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f1751aba640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c13dc10 0x7f174c1546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17522bb640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c15e850 0x7f174c13a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17512b9640 1 -- 172.21.15.77:0/646268818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c15e850 msgr2=0x7f174c13a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17512b9640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c15e850 0x7f174c13a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17512b9640 1 -- 172.21.15.77:0/646268818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c13dc10 msgr2=0x7f174c1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17512b9640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c13dc10 0x7f174c1546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17512b9640 1 -- 172.21.15.77:0/646268818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c155cc0 con 0x7f174c15bf30 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17512b9640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c139c80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f173c000c00 tx=0x7f173c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17427fc640 1 -- 172.21.15.77:0/646268818 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c032860 con 0x7f174c15bf30 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17427fc640 1 -- 172.21.15.77:0/646268818 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c032a00 con 0x7f174c15bf30 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f17427fc640 1 -- 172.21.15.77:0/646268818 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c032df0 con 0x7f174c15bf30 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174c155fa0 con 0x7f174c15bf30 2024-03-06T07:24:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.765+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c1f20b0 con 0x7f174c15bf30 2024-03-06T07:24:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.766+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174c04f330 con 0x7f174c15bf30 2024-03-06T07:24:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.767+0000 7f17427fc640 1 -- 172.21.15.77:0/646268818 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f173c04c060 con 0x7f174c15bf30 2024-03-06T07:24:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.767+0000 7f17427fc640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173006b4d0 0x7f173006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.767+0000 7f17427fc640 1 -- 172.21.15.77:0/646268818 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f173c0ac060 con 0x7f174c15bf30 2024-03-06T07:24:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.767+0000 7f1751aba640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173006b4d0 0x7f173006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.768+0000 7f1751aba640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173006b4d0 0x7f173006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f174800a980 tx=0x7f1748008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.772+0000 7f17427fc640 1 -- 172.21.15.77:0/646268818 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f173c040070 con 0x7f174c15bf30 2024-03-06T07:24:28.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.950+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f174c06eb00 con 0x7f174c15bf30 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.951+0000 7f17427fc640 1 -- 172.21.15.77:0/646268818 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f173c07a890 con 0x7f174c15bf30 2024-03-06T07:24:28.953 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:28.954 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:12.416501+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173006b4d0 msgr2=0x7f173006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173006b4d0 0x7f173006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f174800a980 tx=0x7f1748008040 comp rx=0 tx=0).stop 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 msgr2=0x7f174c139c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c139c80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f173c000c00 tx=0x7f173c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 shutdown_connections 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f173006b4d0 0x7f173006d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f174c15bf30 0x7f174c139c80 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f174c15e850 0x7f174c13a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 --2- 172.21.15.77:0/646268818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f174c13dc10 0x7f174c1546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.953+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 >> 172.21.15.77:0/646268818 conn(0x7f174c0cf8a0 msgr2=0x7f174c15de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.954+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 shutdown_connections 2024-03-06T07:24:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:28.954+0000 7f1753d1e640 1 -- 172.21.15.77:0/646268818 wait complete. 2024-03-06T07:24:28.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:08.975524+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.729967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:28.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:29.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.070+0000 7f7d426b5640 1 Processor -- start 2024-03-06T07:24:29.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.070+0000 7f7d426b5640 1 -- start start 2024-03-06T07:24:29.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d426b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0cfa90 0x7f7d3c0cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d426b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c158ac0 con 0x7f7d3c1586e0 2024-03-06T07:24:29.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d426b5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c15b2a0 con 0x7f7d3c0b33a0 2024-03-06T07:24:29.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d426b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c15b540 con 0x7f7d3c0cfa90 2024-03-06T07:24:29.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d3bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7d3c1586e0 0x7f7d3c0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45512/0 (socket says 172.21.15.77:45512) 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d3bfff640 1 -- 172.21.15.77:0/3668136707 learned_addr learned my addr 172.21.15.77:0/3668136707 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d3b7fe640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0cfa90 0x7f7d3c0cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089867670 0 0) 0x7f7d3c158ac0 con 0x7f7d3c1586e0 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.071+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d24003520 con 0x7f7d3c1586e0 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133972622 0 0) 0x7f7d3c15b2a0 con 0x7f7d3c0b33a0 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d3c158ac0 con 0x7f7d3c0b33a0 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095911638 0 0) 0x7f7d24003520 con 0x7f7d3c1586e0 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0cfa90 msgr2=0x7f7d3c0cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0cfa90 0x7f7d3c0cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 >> v1:172.21.15.158:6789/0 conn(0x7f7d3c0b33a0 legacy=0x7f7d3c0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c15b820 con 0x7f7d3c1586e0 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f7d30001670 con 0x7f7d3c1586e0 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 >> v1:172.21.15.77:6789/0 conn(0x7f7d3c1586e0 legacy=0x7f7d3c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d24003b70 0x7f7d24003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0cfa90 0x7f7d24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24004d40 0x7f7d240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c15b540 con 0x7f7d24003b70 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2400baa0 con 0x7f7d24004d40 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2400bca0 con 0x7f7d3c0cfa90 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3bfff640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d24003b70 0x7f7d24003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3b7fe640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0cfa90 0x7f7d24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d40c52640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24004d40 0x7f7d240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3bfff640 1 -- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0cfa90 msgr2=0x7f7d24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3bfff640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0cfa90 0x7f7d24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3bfff640 1 -- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24004d40 msgr2=0x7f7d240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3bfff640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24004d40 0x7f7d240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.072+0000 7f7d3bfff640 1 -- 172.21.15.77:0/3668136707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d24003520 con 0x7f7d24003b70 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.073+0000 7f7d3bfff640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d24003b70 0x7f7d24003f70 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000c0e0 tx=0x7f7d24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b0a1fe31f7de3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.073+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30007c00 con 0x7f7d24003b70 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.073+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d30030070 con 0x7f7d24003b70 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.073+0000 7f7d3affd640 1 -- 172.21.15.77:0/3668136707 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30005d10 con 0x7f7d24003b70 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.074+0000 7f7d426b5640 1 -- 172.21.15.77:0/3668136707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d24003b70 msgr2=0x7f7d24003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.074+0000 7f7d426b5640 1 --2- 172.21.15.77:0/3668136707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d24003b70 0x7f7d24003f70 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000c0e0 tx=0x7f7d24005830 comp rx=0 tx=0).stop 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.074+0000 7f7d40c52640 1 -- 172.21.15.77:0/3668136707 reap_dead start 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.074+0000 7f7d426b5640 1 -- 172.21.15.77:0/3668136707 shutdown_connections 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.074+0000 7f7d426b5640 1 -- 172.21.15.77:0/3668136707 >> 172.21.15.77:0/3668136707 conn(0x7f7d3c06b190 msgr2=0x7f7d3c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.074+0000 7f7d426b5640 1 -- 172.21.15.77:0/3668136707 shutdown_connections 2024-03-06T07:24:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.074+0000 7f7d426b5640 1 -- 172.21.15.77:0/3668136707 wait complete. 2024-03-06T07:24:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.075+0000 7f7d426b5640 1 Processor -- start 2024-03-06T07:24:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.075+0000 7f7d426b5640 1 -- start start 2024-03-06T07:24:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d426b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24003b70 0x7f7d3c06cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d426b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c148130 0x7f7d3c1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d426b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c147120 0x7f7d3c1475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d426b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0af150 con 0x7f7d3c148130 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d426b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c15bde0 con 0x7f7d24003b70 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d426b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c071350 con 0x7f7d3c147120 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c148130 0x7f7d3c1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c148130 0x7f7d3c1485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46088/0 (socket says 172.21.15.77:46088) 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 -- 172.21.15.77:0/126542495 learned_addr learned my addr 172.21.15.77:0/126542495 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3bfff640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24003b70 0x7f7d3c06cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d40c52640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c147120 0x7f7d3c1475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 -- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c147120 msgr2=0x7f7d3c1475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c147120 0x7f7d3c1475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 -- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24003b70 msgr2=0x7f7d3c06cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24003b70 0x7f7d3c06cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 -- 172.21.15.77:0/126542495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c1f55e0 con 0x7f7d3c148130 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3bfff640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24003b70 0x7f7d3c06cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d3b7fe640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c148130 0x7f7d3c1485b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c011600 tx=0x7f7d2c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d38ff9640 1 -- 172.21.15.77:0/126542495 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00f920 con 0x7f7d3c148130 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d38ff9640 1 -- 172.21.15.77:0/126542495 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c00fac0 con 0x7f7d3c148130 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c1f58c0 con 0x7f7d3c148130 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.076+0000 7f7d38ff9640 1 -- 172.21.15.77:0/126542495 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00fe70 con 0x7f7d3c148130 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.077+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c1f5e80 con 0x7f7d3c148130 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.078+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c1f6050 con 0x7f7d3c148130 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.078+0000 7f7d38ff9640 1 -- 172.21.15.77:0/126542495 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d2c015460 con 0x7f7d3c148130 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.078+0000 7f7d38ff9640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d2806b4d0 0x7f7d2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.078+0000 7f7d38ff9640 1 -- 172.21.15.77:0/126542495 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d2c08ab50 con 0x7f7d3c148130 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.078+0000 7f7d3bfff640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d2806b4d0 0x7f7d2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.079+0000 7f7d3bfff640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d2806b4d0 0x7f7d2806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7d30034000 tx=0x7f7d30009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.082+0000 7f7d38ff9640 1 -- 172.21.15.77:0/126542495 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d3c1f6050 con 0x7f7d3c148130 2024-03-06T07:24:29.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.254+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d3c0ae280 con 0x7f7d3c148130 2024-03-06T07:24:29.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.254+0000 7f7d38ff9640 1 -- 172.21.15.77:0/126542495 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d2c019030 con 0x7f7d3c148130 2024-03-06T07:24:29.256 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:24:29.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d2806b4d0 msgr2=0x7f7d2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d2806b4d0 0x7f7d2806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7d30034000 tx=0x7f7d30009030 comp rx=0 tx=0).stop 2024-03-06T07:24:29.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c148130 msgr2=0x7f7d3c1485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c148130 0x7f7d3c1485b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c011600 tx=0x7f7d2c011ac0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 shutdown_connections 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d2806b4d0 0x7f7d2806d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c148130 0x7f7d3c1485b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c147120 0x7f7d3c1475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 --2- 172.21.15.77:0/126542495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d24003b70 0x7f7d3c06cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 >> 172.21.15.77:0/126542495 conn(0x7f7d3c06b190 msgr2=0x7f7d3c0b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 shutdown_connections 2024-03-06T07:24:29.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.256+0000 7f7d426b5640 1 -- 172.21.15.77:0/126542495 wait complete. 2024-03-06T07:24:29.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.371+0000 7fd55d40e640 1 Processor -- start 2024-03-06T07:24:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.371+0000 7fd55d40e640 1 -- start start 2024-03-06T07:24:29.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd55d40e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd55806fd10 0x7fd5580cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd55d40e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5580d0af0 0x7fd5580ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd55d40e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5580cd1e0 con 0x7fd55815c050 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd55d40e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5580cd4b0 con 0x7fd5580d0af0 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd55d40e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5580cd790 con 0x7fd55806fd10 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd5577fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5580d0af0 0x7fd5580ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd5567fc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd55806fd10 0x7fd5580cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd556ffd640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fd55815c050 0x7fd55806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:48114/0 (socket says 172.21.15.77:48114) 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd556ffd640 1 -- 172.21.15.77:0/643242437 learned_addr learned my addr 172.21.15.77:0/643242437 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.372+0000 7fd5577fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5580d0af0 0x7fd5580ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:36988/0 (socket says 172.21.15.77:36988) 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.373+0000 7fd5577fe640 1 -- 172.21.15.77:0/643242437 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd55806fd10 msgr2=0x7fd5580cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.373+0000 7fd5577fe640 1 --2- 172.21.15.77:0/643242437 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd55806fd10 0x7fd5580cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.373+0000 7fd5577fe640 1 -- 172.21.15.77:0/643242437 >> v1:172.21.15.158:6789/0 conn(0x7fd55815c050 legacy=0x7fd55806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.373+0000 7fd5577fe640 1 -- 172.21.15.77:0/643242437 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580cda70 con 0x7fd5580d0af0 2024-03-06T07:24:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.373+0000 7fd555ffb640 1 -- 172.21.15.77:0/643242437 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721357834 0 0) 0x7fd5580cd1e0 con 0x7fd55815c050 2024-03-06T07:24:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd5577fe640 1 --2- 172.21.15.77:0/643242437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5580d0af0 0x7fd5580ccca0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd550009d90 tx=0x7fd55002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c16f0021b72a8516 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd555ffb640 1 -- 172.21.15.77:0/643242437 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550039070 con 0x7fd5580d0af0 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd555ffb640 1 -- 172.21.15.77:0/643242437 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55002cbc0 con 0x7fd5580d0af0 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd555ffb640 1 -- 172.21.15.77:0/643242437 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550034440 con 0x7fd5580d0af0 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd55d40e640 1 -- 172.21.15.77:0/643242437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5580d0af0 msgr2=0x7fd5580ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd55d40e640 1 --2- 172.21.15.77:0/643242437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5580d0af0 0x7fd5580ccca0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd550009d90 tx=0x7fd55002be10 comp rx=0 tx=0).stop 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd55d40e640 1 -- 172.21.15.77:0/643242437 shutdown_connections 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd55d40e640 1 --2- 172.21.15.77:0/643242437 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5580d0af0 0x7fd5580ccca0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd55d40e640 1 --2- 172.21.15.77:0/643242437 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd55806fd10 0x7fd5580cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd55d40e640 1 -- 172.21.15.77:0/643242437 >> 172.21.15.77:0/643242437 conn(0x7fd5580b3800 msgr2=0x7fd5580b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd55d40e640 1 -- 172.21.15.77:0/643242437 shutdown_connections 2024-03-06T07:24:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.374+0000 7fd55d40e640 1 -- 172.21.15.77:0/643242437 wait complete. 2024-03-06T07:24:29.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd55d40e640 1 Processor -- start 2024-03-06T07:24:29.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd55d40e640 1 -- start start 2024-03-06T07:24:29.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd55d40e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd55806fd10 0x7fd5580b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd55d40e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5580d0af0 0x7fd558150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd55d40e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd55815c050 0x7fd55813a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd55d40e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558165c30 con 0x7fd55815c050 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd55d40e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558165ab0 con 0x7fd55806fd10 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd55d40e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd558165db0 con 0x7fd5580d0af0 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd55815c050 0x7fd55813a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd5577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd55815c050 0x7fd55813a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46096/0 (socket says 172.21.15.77:46096) 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd5577fe640 1 -- 172.21.15.77:0/1271498014 learned_addr learned my addr 172.21.15.77:0/1271498014 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.376+0000 7fd556ffd640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd55806fd10 0x7fd5580b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd5567fc640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5580d0af0 0x7fd558150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd5577fe640 1 -- 172.21.15.77:0/1271498014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5580d0af0 msgr2=0x7fd558150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd5577fe640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5580d0af0 0x7fd558150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd5577fe640 1 -- 172.21.15.77:0/1271498014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd55806fd10 msgr2=0x7fd5580b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd5577fe640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd55806fd10 0x7fd5580b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd5577fe640 1 -- 172.21.15.77:0/1271498014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55814c7c0 con 0x7fd55815c050 2024-03-06T07:24:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd5577fe640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd55815c050 0x7fd55813a4f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd550009eb0 tx=0x7fd550004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd53f7fe640 1 -- 172.21.15.77:0/1271498014 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550044020 con 0x7fd55815c050 2024-03-06T07:24:29.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd53f7fe640 1 -- 172.21.15.77:0/1271498014 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55003f470 con 0x7fd55815c050 2024-03-06T07:24:29.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd53f7fe640 1 -- 172.21.15.77:0/1271498014 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550039040 con 0x7fd55815c050 2024-03-06T07:24:29.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55814caa0 con 0x7fd55815c050 2024-03-06T07:24:29.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.377+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55814d060 con 0x7fd55815c050 2024-03-06T07:24:29.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.378+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55809fe60 con 0x7fd55815c050 2024-03-06T07:24:29.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.379+0000 7fd53f7fe640 1 -- 172.21.15.77:0/1271498014 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd550047020 con 0x7fd55815c050 2024-03-06T07:24:29.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.379+0000 7fd53f7fe640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd54406b4d0 0x7fd54406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.379+0000 7fd53f7fe640 1 -- 172.21.15.77:0/1271498014 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5500acae0 con 0x7fd55815c050 2024-03-06T07:24:29.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.379+0000 7fd556ffd640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd54406b4d0 0x7fd54406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.380+0000 7fd556ffd640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd54406b4d0 0x7fd54406d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd548000f60 tx=0x7fd548007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.384+0000 7fd53f7fe640 1 -- 172.21.15.77:0/1271498014 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd55004e030 con 0x7fd55815c050 2024-03-06T07:24:29.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.562+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5580b1380 con 0x7fd55815c050 2024-03-06T07:24:29.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.562+0000 7fd53f7fe640 1 -- 172.21.15.77:0/1271498014 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd55007b3a0 con 0x7fd55815c050 2024-03-06T07:24:29.782 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:24:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd54406b4d0 msgr2=0x7fd54406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd54406b4d0 0x7fd54406d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd548000f60 tx=0x7fd548007d50 comp rx=0 tx=0).stop 2024-03-06T07:24:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd55815c050 msgr2=0x7fd55813a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd55815c050 0x7fd55813a4f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd550009eb0 tx=0x7fd550004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 shutdown_connections 2024-03-06T07:24:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd54406b4d0 0x7fd54406d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd55815c050 0x7fd55813a4f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5580d0af0 0x7fd558150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 --2- 172.21.15.77:0/1271498014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd55806fd10 0x7fd5580b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 >> 172.21.15.77:0/1271498014 conn(0x7fd5580b3800 msgr2=0x7fd5580b2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 shutdown_connections 2024-03-06T07:24:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.564+0000 7fd55d40e640 1 -- 172.21.15.77:0/1271498014 wait complete. 2024-03-06T07:24:29.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:29.796 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:29.829 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:29.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.890+0000 7fed46155640 1 Processor -- start 2024-03-06T07:24:29.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.890+0000 7fed46155640 1 -- start start 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.890+0000 7fed46155640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed400d4860 0x7fed400d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed46155640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed401618b0 con 0x7fed400b0e30 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed46155640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed400cf400 con 0x7fed401614d0 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed46155640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed400d4cf0 con 0x7fed400d4860 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed3effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed400d4860 0x7fed400d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed3ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fed400b0e30 0x7fed400b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45516/0 (socket says 172.21.15.77:45516) 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed3ffff640 1 -- 172.21.15.77:0/3638826181 learned_addr learned my addr 172.21.15.77:0/3638826181 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed3effd640 1 -- 172.21.15.77:0/3638826181 >> v1:172.21.15.158:6789/0 conn(0x7fed401614d0 legacy=0x7fed400d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed3effd640 1 -- 172.21.15.77:0/3638826181 >> v1:172.21.15.77:6789/0 conn(0x7fed400b0e30 legacy=0x7fed400b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed3e7fc640 1 -- 172.21.15.77:0/3638826181 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833377922 0 0) 0x7fed401618b0 con 0x7fed400b0e30 2024-03-06T07:24:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.891+0000 7fed3effd640 1 -- 172.21.15.77:0/3638826181 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed400cf6a0 con 0x7fed400d4860 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed3effd640 1 --2- 172.21.15.77:0/3638826181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed400d4860 0x7fed400d2020 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fed340097a0 tx=0x7fed3402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd848ad59b78185 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed3e7fc640 1 -- 172.21.15.77:0/3638826181 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed34034070 con 0x7fed400d4860 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed3e7fc640 1 -- 172.21.15.77:0/3638826181 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed34030610 con 0x7fed400d4860 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed3e7fc640 1 -- 172.21.15.77:0/3638826181 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed340309e0 con 0x7fed400d4860 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed46155640 1 -- 172.21.15.77:0/3638826181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed400d4860 msgr2=0x7fed400d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed46155640 1 --2- 172.21.15.77:0/3638826181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed400d4860 0x7fed400d2020 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fed340097a0 tx=0x7fed3402b960 comp rx=0 tx=0).stop 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed46155640 1 -- 172.21.15.77:0/3638826181 shutdown_connections 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed46155640 1 --2- 172.21.15.77:0/3638826181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed400d4860 0x7fed400d2020 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed46155640 1 -- 172.21.15.77:0/3638826181 >> 172.21.15.77:0/3638826181 conn(0x7fed400d3d40 msgr2=0x7fed400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.892+0000 7fed46155640 1 -- 172.21.15.77:0/3638826181 shutdown_connections 2024-03-06T07:24:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.893+0000 7fed46155640 1 -- 172.21.15.77:0/3638826181 wait complete. 2024-03-06T07:24:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.894+0000 7fed46155640 1 Processor -- start 2024-03-06T07:24:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.894+0000 7fed46155640 1 -- start start 2024-03-06T07:24:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.894+0000 7fed46155640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed400b0e30 0x7fed400cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed46155640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed400d4860 0x7fed40156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed46155640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed401614d0 0x7fed401f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed46155640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed400b0910 con 0x7fed401614d0 2024-03-06T07:24:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed46155640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed400cfc60 con 0x7fed400b0e30 2024-03-06T07:24:29.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed46155640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed400d2560 con 0x7fed400d4860 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed401614d0 0x7fed401f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed401614d0 0x7fed401f7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46110/0 (socket says 172.21.15.77:46110) 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 -- 172.21.15.77:0/3949143895 learned_addr learned my addr 172.21.15.77:0/3949143895 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3f7fe640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed400b0e30 0x7fed400cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3effd640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed400d4860 0x7fed40156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 -- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed400d4860 msgr2=0x7fed40156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed400d4860 0x7fed40156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 -- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed400b0e30 msgr2=0x7fed400cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed400b0e30 0x7fed400cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 -- 172.21.15.77:0/3949143895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed400ca020 con 0x7fed401614d0 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3f7fe640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed400b0e30 0x7fed400cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:29.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3effd640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed400d4860 0x7fed40156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:29.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed3ffff640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed401614d0 0x7fed401f7000 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fed3800cb20 tx=0x7fed3800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed27fff640 1 -- 172.21.15.77:0/3949143895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3800aca0 con 0x7fed401614d0 2024-03-06T07:24:29.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed27fff640 1 -- 172.21.15.77:0/3949143895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed38012400 con 0x7fed401614d0 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed27fff640 1 -- 172.21.15.77:0/3949143895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3800ae60 con 0x7fed401614d0 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed400cbee0 con 0x7fed401614d0 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.895+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed400cc4a0 con 0x7fed401614d0 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.897+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed40068ff0 con 0x7fed401614d0 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.897+0000 7fed27fff640 1 -- 172.21.15.77:0/3949143895 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed380127f0 con 0x7fed401614d0 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.897+0000 7fed27fff640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed2c06b4d0 0x7fed2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.898+0000 7fed3f7fe640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed2c06b4d0 0x7fed2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.898+0000 7fed27fff640 1 -- 172.21.15.77:0/3949143895 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed3808a7b0 con 0x7fed401614d0 2024-03-06T07:24:29.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.898+0000 7fed3f7fe640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed2c06b4d0 0x7fed2c06d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fed30007fd0 tx=0x7fed30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:29.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:29.902+0000 7fed27fff640 1 -- 172.21.15.77:0/3949143895 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed380596c0 con 0x7fed401614d0 2024-03-06T07:24:30.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.130+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed400d0af0 con 0x7fed401614d0 2024-03-06T07:24:30.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.131+0000 7fed27fff640 1 -- 172.21.15.77:0/3949143895 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed38058fc0 con 0x7fed401614d0 2024-03-06T07:24:30.134 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed2c06b4d0 msgr2=0x7fed2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed2c06b4d0 0x7fed2c06d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fed30007fd0 tx=0x7fed30009040 comp rx=0 tx=0).stop 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed401614d0 msgr2=0x7fed401f7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed401614d0 0x7fed401f7000 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fed3800cb20 tx=0x7fed3800a060 comp rx=0 tx=0).stop 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 shutdown_connections 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed2c06b4d0 0x7fed2c06d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed401614d0 0x7fed401f7000 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed400d4860 0x7fed40156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 --2- 172.21.15.77:0/3949143895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed400b0e30 0x7fed400cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 >> 172.21.15.77:0/3949143895 conn(0x7fed400d3d40 msgr2=0x7fed400d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 shutdown_connections 2024-03-06T07:24:30.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.133+0000 7fed46155640 1 -- 172.21.15.77:0/3949143895 wait complete. 2024-03-06T07:24:30.147 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:30.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.285+0000 7f9f09e39640 1 Processor -- start 2024-03-06T07:24:30.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.285+0000 7f9f09e39640 1 -- start start 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.286+0000 7f9f09e39640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9f040b4460 0x7f9f040d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.286+0000 7f9f09e39640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f04161cf0 0x7f9f04162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.286+0000 7f9f09e39640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f040d22c0 con 0x7f9f04162970 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f09e39640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f04161b00 con 0x7f9f04161cf0 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f09e39640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f040cecc0 con 0x7f9f040b4460 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f03fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f04161cf0 0x7f9f04162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f03fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f04161cf0 0x7f9f04162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46116/0 (socket says 172.21.15.77:46116) 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f03fff640 1 -- 172.21.15.77:0/3790156491 learned_addr learned my addr 172.21.15.77:0/3790156491 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f02ffd640 1 --1- 172.21.15.77:0/3790156491 >> v1:172.21.15.77:6789/0 conn(0x7f9f04162970 0x7f9f040b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45520/0 (socket says 172.21.15.77:45520) 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f03fff640 1 -- 172.21.15.77:0/3790156491 >> v2:172.21.15.158:3300/0 conn(0x7f9f040b4460 msgr2=0x7f9f040d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f03fff640 1 --2- 172.21.15.77:0/3790156491 >> v2:172.21.15.158:3300/0 conn(0x7f9f040b4460 0x7f9f040d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f03fff640 1 -- 172.21.15.77:0/3790156491 >> v1:172.21.15.77:6789/0 conn(0x7f9f04162970 legacy=0x7f9f040b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.287+0000 7f9f03fff640 1 -- 172.21.15.77:0/3790156491 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f040d25a0 con 0x7f9f04161cf0 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f03fff640 1 --2- 172.21.15.77:0/3790156491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f04161cf0 0x7f9f04162180 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9f0408ddd0 tx=0x7f9efc02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78d4aab7c10b541 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f027fc640 1 -- 172.21.15.77:0/3790156491 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc02c8d0 con 0x7f9f04161cf0 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f027fc640 1 -- 172.21.15.77:0/3790156491 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc02ca70 con 0x7f9f04161cf0 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f027fc640 1 -- 172.21.15.77:0/3790156491 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc02ce40 con 0x7f9f04161cf0 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f09e39640 1 -- 172.21.15.77:0/3790156491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f04161cf0 msgr2=0x7f9f04162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f09e39640 1 --2- 172.21.15.77:0/3790156491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f04161cf0 0x7f9f04162180 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9f0408ddd0 tx=0x7f9efc02bdb0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f09e39640 1 -- 172.21.15.77:0/3790156491 shutdown_connections 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f09e39640 1 --2- 172.21.15.77:0/3790156491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f04161cf0 0x7f9f04162180 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f09e39640 1 --2- 172.21.15.77:0/3790156491 >> v2:172.21.15.158:3300/0 conn(0x7f9f040b4460 0x7f9f040d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f09e39640 1 -- 172.21.15.77:0/3790156491 >> 172.21.15.77:0/3790156491 conn(0x7f9f040d3ce0 msgr2=0x7f9f040d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f09e39640 1 -- 172.21.15.77:0/3790156491 shutdown_connections 2024-03-06T07:24:30.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.288+0000 7f9f09e39640 1 -- 172.21.15.77:0/3790156491 wait complete. 2024-03-06T07:24:30.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.290+0000 7f9f09e39640 1 Processor -- start 2024-03-06T07:24:30.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.290+0000 7f9f09e39640 1 -- start start 2024-03-06T07:24:30.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f09e39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f040b4460 0x7f9f0415dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f09e39640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f04161cf0 0x7f9f040b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f09e39640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f04162970 0x7f9f040b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f09e39640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040ce950 con 0x7f9f040b4460 2024-03-06T07:24:30.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f09e39640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040d2b60 con 0x7f9f04162970 2024-03-06T07:24:30.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f09e39640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040b4840 con 0x7f9f04161cf0 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f040b4460 0x7f9f0415dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f040b4460 0x7f9f0415dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46126/0 (socket says 172.21.15.77:46126) 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 -- 172.21.15.77:0/563530420 learned_addr learned my addr 172.21.15.77:0/563530420 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 -- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f04161cf0 msgr2=0x7f9f040b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f02ffd640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f04161cf0 0x7f9f040b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f03fff640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f04162970 0x7f9f040b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f04161cf0 0x7f9f040b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 -- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f04162970 msgr2=0x7f9f040b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f04162970 0x7f9f040b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 -- 172.21.15.77:0/563530420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f040b1ae0 con 0x7f9f040b4460 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f02ffd640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f04161cf0 0x7f9f040b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f03fff640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f04162970 0x7f9f040b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f037fe640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f040b4460 0x7f9f0415dff0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800b440 tx=0x7f9ef800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9eebfff640 1 -- 172.21.15.77:0/563530420 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8015020 con 0x7f9f040b4460 2024-03-06T07:24:30.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9eebfff640 1 -- 172.21.15.77:0/563530420 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8011890 con 0x7f9f040b4460 2024-03-06T07:24:30.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f040c87f0 con 0x7f9f040b4460 2024-03-06T07:24:30.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9eebfff640 1 -- 172.21.15.77:0/563530420 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8011c80 con 0x7f9f040b4460 2024-03-06T07:24:30.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.292+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f040c9090 con 0x7f9f040b4460 2024-03-06T07:24:30.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.294+0000 7f9eebfff640 1 -- 172.21.15.77:0/563530420 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ef8018020 con 0x7f9f040b4460 2024-03-06T07:24:30.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.294+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0404f330 con 0x7f9f040b4460 2024-03-06T07:24:30.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.294+0000 7f9eebfff640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef006b4d0 0x7f9ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.294+0000 7f9eebfff640 1 -- 172.21.15.77:0/563530420 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ef808a260 con 0x7f9f040b4460 2024-03-06T07:24:30.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.294+0000 7f9f02ffd640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef006b4d0 0x7f9ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.294+0000 7f9f02ffd640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef006b4d0 0x7f9ef006d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9f0406c980 tx=0x7f9ef4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.299+0000 7f9eebfff640 1 -- 172.21.15.77:0/563530420 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ef801f020 con 0x7f9f040b4460 2024-03-06T07:24:30.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.485+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9f040c89c0 con 0x7f9f040b4460 2024-03-06T07:24:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.486+0000 7f9eebfff640 1 -- 172.21.15.77:0/563530420 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9ef800d040 con 0x7f9f040b4460 2024-03-06T07:24:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.487+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef006b4d0 msgr2=0x7f9ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.487+0000 7f9f09e39640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef006b4d0 0x7f9ef006d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9f0406c980 tx=0x7f9ef4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.487+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f040b4460 msgr2=0x7f9f0415dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.487+0000 7f9f09e39640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f040b4460 0x7f9f0415dff0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800b440 tx=0x7f9ef800b900 comp rx=0 tx=0).stop 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.487+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 shutdown_connections 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.487+0000 7f9f09e39640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f04162970 0x7f9f040b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.487+0000 7f9f09e39640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f04161cf0 0x7f9f040b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.487+0000 7f9f09e39640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef006b4d0 0x7f9ef006d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.488+0000 7f9f09e39640 1 --2- 172.21.15.77:0/563530420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f040b4460 0x7f9f0415dff0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.488+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 >> 172.21.15.77:0/563530420 conn(0x7f9f040d3ce0 msgr2=0x7f9f0406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.488+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 shutdown_connections 2024-03-06T07:24:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.488+0000 7f9f09e39640 1 -- 172.21.15.77:0/563530420 wait complete. 2024-03-06T07:24:30.501 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:30.501 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:30.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:30.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.654+0000 7f2c8b77a640 1 Processor -- start 2024-03-06T07:24:30.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.655+0000 7f2c8b77a640 1 -- start start 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.655+0000 7f2c8b77a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.655+0000 7f2c8b77a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c840d2920 0x7f2c840b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.655+0000 7f2c8b77a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8415e470 con 0x7f2c840b3b20 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.655+0000 7f2c8b77a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c840d18f0 con 0x7f2c840d2920 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.655+0000 7f2c8b77a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c840d1b90 con 0x7f2c8415e090 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.656+0000 7f2c88d15640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c840d2920 0x7f2c840b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.656+0000 7f2c88d15640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c840d2920 0x7f2c840b35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46142/0 (socket says 172.21.15.77:46142) 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.656+0000 7f2c88d15640 1 -- 172.21.15.77:0/2957889360 learned_addr learned my addr 172.21.15.77:0/2957889360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.656+0000 7f2c89516640 1 --2- 172.21.15.77:0/2957889360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.656+0000 7f2c88d15640 1 -- 172.21.15.77:0/2957889360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 msgr2=0x7f2c840d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.656+0000 7f2c88d15640 1 --2- 172.21.15.77:0/2957889360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.656+0000 7f2c88d15640 1 -- 172.21.15.77:0/2957889360 >> v1:172.21.15.158:6789/0 conn(0x7f2c840b3b20 legacy=0x7f2c840d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.656+0000 7f2c88d15640 1 -- 172.21.15.77:0/2957889360 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840ceb50 con 0x7f2c840d2920 2024-03-06T07:24:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c88d15640 1 --2- 172.21.15.77:0/2957889360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c840b35e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009870 tx=0x7f2c7402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c4e06aa1c11b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c7bfff640 1 -- 172.21.15.77:0/2957889360 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74034070 con 0x7f2c840d2920 2024-03-06T07:24:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c7bfff640 1 -- 172.21.15.77:0/2957889360 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c74030840 con 0x7f2c840d2920 2024-03-06T07:24:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c7bfff640 1 -- 172.21.15.77:0/2957889360 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74030c10 con 0x7f2c840d2920 2024-03-06T07:24:30.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c8b77a640 1 -- 172.21.15.77:0/2957889360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 msgr2=0x7f2c840b35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/2957889360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c840b35e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009870 tx=0x7f2c7402b980 comp rx=0 tx=0).stop 2024-03-06T07:24:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c8b77a640 1 -- 172.21.15.77:0/2957889360 shutdown_connections 2024-03-06T07:24:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/2957889360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c840b35e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/2957889360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c8b77a640 1 -- 172.21.15.77:0/2957889360 >> 172.21.15.77:0/2957889360 conn(0x7f2c840d3c80 msgr2=0x7f2c840d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c8b77a640 1 -- 172.21.15.77:0/2957889360 shutdown_connections 2024-03-06T07:24:30.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.657+0000 7f2c8b77a640 1 -- 172.21.15.77:0/2957889360 wait complete. 2024-03-06T07:24:30.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.659+0000 7f2c8b77a640 1 Processor -- start 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.659+0000 7f2c8b77a640 1 -- start start 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c8b77a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c8414e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c8b77a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c8b77a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c840b92e0 0x7f2c840b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c8b77a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c840ce800 con 0x7f2c840d2920 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c8b77a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c840b2df0 con 0x7f2c840b92e0 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c8b77a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c840d0d60 con 0x7f2c8415e090 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89516640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c8414e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89516640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c8414e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46158/0 (socket says 172.21.15.77:46158) 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89516640 1 -- 172.21.15.77:0/1268783536 learned_addr learned my addr 172.21.15.77:0/1268783536 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c88d15640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89d17640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c840b92e0 0x7f2c840b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89516640 1 -- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 msgr2=0x7f2c840cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89516640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89516640 1 -- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c840b92e0 msgr2=0x7f2c840b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89516640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c840b92e0 0x7f2c840b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89516640 1 -- 172.21.15.77:0/1268783536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840cd260 con 0x7f2c840d2920 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c88d15640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.660+0000 7f2c89d17640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c840b92e0 0x7f2c840b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.661+0000 7f2c89516640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c8414e110 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000d600 tx=0x7f2c8000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.661+0000 7f2c79ffb640 1 -- 172.21.15.77:0/1268783536 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c800125f0 con 0x7f2c840d2920 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.661+0000 7f2c79ffb640 1 -- 172.21.15.77:0/1268783536 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c80019070 con 0x7f2c840d2920 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.661+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c840cd540 con 0x7f2c840d2920 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.661+0000 7f2c79ffb640 1 -- 172.21.15.77:0/1268783536 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c800129f0 con 0x7f2c840d2920 2024-03-06T07:24:30.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.661+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c840cdb00 con 0x7f2c840d2920 2024-03-06T07:24:30.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.662+0000 7f2c79ffb640 1 -- 172.21.15.77:0/1268783536 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c80012b90 con 0x7f2c840d2920 2024-03-06T07:24:30.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.662+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c8404f330 con 0x7f2c840d2920 2024-03-06T07:24:30.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.663+0000 7f2c79ffb640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.663+0000 7f2c88d15640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.663+0000 7f2c79ffb640 1 -- 172.21.15.77:0/1268783536 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c8008a5f0 con 0x7f2c840d2920 2024-03-06T07:24:30.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.663+0000 7f2c88d15640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c6806b4d0 0x7f2c6806d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2c74036000 tx=0x7f2c7403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.667+0000 7f2c79ffb640 1 -- 172.21.15.77:0/1268783536 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c80059500 con 0x7f2c840d2920 2024-03-06T07:24:30.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.841+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c840d1230 con 0x7f2c840d2920 2024-03-06T07:24:30.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.843+0000 7f2c79ffb640 1 -- 172.21.15.77:0/1268783536 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f2c80058e00 con 0x7f2c840d2920 2024-03-06T07:24:30.846 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:30.846 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:12.416501+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c6806b4d0 msgr2=0x7f2c6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c6806b4d0 0x7f2c6806d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2c74036000 tx=0x7f2c7403b040 comp rx=0 tx=0).stop 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 msgr2=0x7f2c8414e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c8414e110 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000d600 tx=0x7f2c8000dac0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 shutdown_connections 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c840b92e0 0x7f2c840b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c8415e090 0x7f2c840cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 --2- 172.21.15.77:0/1268783536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c840d2920 0x7f2c8414e110 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 >> 172.21.15.77:0/1268783536 conn(0x7f2c840d3c80 msgr2=0x7f2c8415bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 shutdown_connections 2024-03-06T07:24:30.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.845+0000 7f2c8b77a640 1 -- 172.21.15.77:0/1268783536 wait complete. 2024-03-06T07:24:30.859 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:08.975524+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.729967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:30.859 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.960+0000 7f08c4a71640 1 Processor -- start 2024-03-06T07:24:30.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.960+0000 7f08c4a71640 1 -- start start 2024-03-06T07:24:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.961+0000 7f08c4a71640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c01614d0 0x7f08c00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.961+0000 7f08c4a71640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c01618b0 con 0x7f08c00d1910 2024-03-06T07:24:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.961+0000 7f08c4a71640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c00cf400 con 0x7f08c00d2480 2024-03-06T07:24:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.961+0000 7f08c4a71640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c00cf6a0 con 0x7f08c01614d0 2024-03-06T07:24:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.961+0000 7f08bdd9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f08c00d1910 0x7f08c00d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45534/0 (socket says 172.21.15.77:45534) 2024-03-06T07:24:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.961+0000 7f08bdd9b640 1 -- 172.21.15.77:0/159391901 learned_addr learned my addr 172.21.15.77:0/159391901 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.961+0000 7f08be59c640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c01614d0 0x7f08c00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032396907 0 0) 0x7f08c01618b0 con 0x7f08c00d1910 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a8003520 con 0x7f08c00d1910 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 826481929 0 0) 0x7f08a8003520 con 0x7f08c00d1910 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c01614d0 msgr2=0x7f08c00d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c01614d0 0x7f08c00d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 >> v1:172.21.15.158:6789/0 conn(0x7f08c00d2480 legacy=0x7f08c00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c00cf980 con 0x7f08c00d1910 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f08b4001670 con 0x7f08c00d1910 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 >> v1:172.21.15.77:6789/0 conn(0x7f08c00d1910 legacy=0x7f08c00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a80039e0 0x7f08a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c01614d0 0x7f08a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08a8004d40 0x7f08a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c00cf6a0 con 0x7f08a8004d40 2024-03-06T07:24:30.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a800b9c0 con 0x7f08a80039e0 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a800bbc0 con 0x7f08c01614d0 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bed9d640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08a8004d40 0x7f08a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08be59c640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a80039e0 0x7f08a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bdd9b640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c01614d0 0x7f08a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bed9d640 1 -- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c01614d0 msgr2=0x7f08a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bed9d640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c01614d0 0x7f08a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bed9d640 1 -- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a80039e0 msgr2=0x7f08a8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bed9d640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a80039e0 0x7f08a8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.962+0000 7f08bed9d640 1 -- 172.21.15.77:0/159391901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a8003520 con 0x7f08a8004d40 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.963+0000 7f08bed9d640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08a8004d40 0x7f08a80051c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f08b8001f30 tx=0x7f08a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a77e7360227383 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.963+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8001b30 con 0x7f08a8004d40 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.963+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b802d6e0 con 0x7f08a8004d40 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.963+0000 7f08bd59a640 1 -- 172.21.15.77:0/159391901 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b802dad0 con 0x7f08a8004d40 2024-03-06T07:24:30.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.964+0000 7f08c4a71640 1 -- 172.21.15.77:0/159391901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08a8004d40 msgr2=0x7f08a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.964+0000 7f08c4a71640 1 --2- 172.21.15.77:0/159391901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08a8004d40 0x7f08a80051c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f08b8001f30 tx=0x7f08a8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:30.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.964+0000 7f08bed9d640 1 -- 172.21.15.77:0/159391901 reap_dead start 2024-03-06T07:24:30.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.964+0000 7f08c4a71640 1 -- 172.21.15.77:0/159391901 shutdown_connections 2024-03-06T07:24:30.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.964+0000 7f08c4a71640 1 -- 172.21.15.77:0/159391901 >> 172.21.15.77:0/159391901 conn(0x7f08c00d3d40 msgr2=0x7f08c00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:30.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.964+0000 7f08c4a71640 1 -- 172.21.15.77:0/159391901 shutdown_connections 2024-03-06T07:24:30.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.964+0000 7f08c4a71640 1 -- 172.21.15.77:0/159391901 wait complete. 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.965+0000 7f08c4a71640 1 Processor -- start 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.965+0000 7f08c4a71640 1 -- start start 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08c4a71640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a8004d40 0x7f08c015a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08c4a71640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c0158440 0x7f08c0159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08c4a71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08c01551f0 0x7f08c00c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08c4a71640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c00d4860 con 0x7f08c01551f0 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08c4a71640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c00b0910 con 0x7f08a8004d40 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08c4a71640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c00d49e0 con 0x7f08c0158440 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08c01551f0 0x7f08c00c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08c01551f0 0x7f08c00c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46168/0 (socket says 172.21.15.77:46168) 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 -- 172.21.15.77:0/856256061 learned_addr learned my addr 172.21.15.77:0/856256061 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bdd9b640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c0158440 0x7f08c0159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08be59c640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a8004d40 0x7f08c015a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 -- 172.21.15.77:0/856256061 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c0158440 msgr2=0x7f08c0159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c0158440 0x7f08c0159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 -- 172.21.15.77:0/856256061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a8004d40 msgr2=0x7f08c015a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a8004d40 0x7f08c015a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 -- 172.21.15.77:0/856256061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0159940 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.966+0000 7f08bed9d640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08c01551f0 0x7f08c00c78e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f08b8002050 tx=0x7f08b80015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.967+0000 7f08aeffd640 1 -- 172.21.15.77:0/856256061 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8001b30 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.967+0000 7f08aeffd640 1 -- 172.21.15.77:0/856256061 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b8043070 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.967+0000 7f08aeffd640 1 -- 172.21.15.77:0/856256061 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8003bb0 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.967+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c0159c20 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.967+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c015a1e0 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.968+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c0068ff0 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.968+0000 7f08aeffd640 1 -- 172.21.15.77:0/856256061 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08b8005070 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.969+0000 7f08aeffd640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.969+0000 7f08aeffd640 1 -- 172.21.15.77:0/856256061 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08b80abef0 con 0x7f08c01551f0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.969+0000 7f08be59c640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.969+0000 7f08be59c640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08a006b4d0 0x7f08a006d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f08b0009b00 tx=0x7f08b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:30.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:30.973+0000 7f08aeffd640 1 -- 172.21.15.77:0/856256061 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08b807ad70 con 0x7f08c01551f0 2024-03-06T07:24:31.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.146+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08c0160af0 con 0x7f08c01551f0 2024-03-06T07:24:31.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.146+0000 7f08aeffd640 1 -- 172.21.15.77:0/856256061 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f08b807a670 con 0x7f08c01551f0 2024-03-06T07:24:31.149 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:24:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08a006b4d0 msgr2=0x7f08a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08a006b4d0 0x7f08a006d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f08b0009b00 tx=0x7f08b0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08c01551f0 msgr2=0x7f08c00c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08c01551f0 0x7f08c00c78e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f08b8002050 tx=0x7f08b80015b0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 shutdown_connections 2024-03-06T07:24:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08c01551f0 0x7f08c00c78e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08c0158440 0x7f08c0159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 --2- 172.21.15.77:0/856256061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08a8004d40 0x7f08c015a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 >> 172.21.15.77:0/856256061 conn(0x7f08c00d3d40 msgr2=0x7f08c00c9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 shutdown_connections 2024-03-06T07:24:31.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.149+0000 7f08c4a71640 1 -- 172.21.15.77:0/856256061 wait complete. 2024-03-06T07:24:31.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:31.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.263+0000 7f5cfe6a3640 1 Processor -- start 2024-03-06T07:24:31.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.263+0000 7f5cfe6a3640 1 -- start start 2024-03-06T07:24:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.263+0000 7f5cfe6a3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf815e0e0 0x7f5cf806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.263+0000 7f5cfe6a3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf8161bd0 con 0x7f5cf806b6c0 2024-03-06T07:24:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cfe6a3640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf8070220 con 0x7f5cf815dd00 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cfe6a3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8161e90 con 0x7f5cf815e0e0 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cfcc40640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5cf806b6c0 0x7f5cf80b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45550/0 (socket says 172.21.15.77:45550) 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2210506989 learned_addr learned my addr 172.21.15.77:0/2210506989 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449232351 0 0) 0x7f5cf8161bd0 con 0x7f5cf806b6c0 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce0003520 con 0x7f5cf806b6c0 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf77fe640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf815e0e0 0x7f5cf806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009386994 0 0) 0x7f5ce0003520 con 0x7f5cf806b6c0 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf815e0e0 msgr2=0x7f5cf806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf815e0e0 0x7f5cf806e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 >> v1:172.21.15.158:6789/0 conn(0x7f5cf815dd00 legacy=0x7f5cf806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8162170 con 0x7f5cf806b6c0 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f5cf00019f0 con 0x7f5cf806b6c0 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 >> v1:172.21.15.77:6789/0 conn(0x7f5cf806b6c0 legacy=0x7f5cf80b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ce00039e0 0x7f5ce0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ce0004340 0x7f5ce0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ce0005020 0x7f5ce000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8161e90 con 0x7f5ce0005020 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce000bf00 con 0x7f5ce0004340 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce000c0b0 con 0x7f5ce00039e0 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cfcc40640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ce0005020 0x7f5ce000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf7fff640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ce00039e0 0x7f5ce0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.264+0000 7f5cf77fe640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ce0004340 0x7f5ce0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.265+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ce00039e0 msgr2=0x7f5ce0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.265+0000 7f5cfcc40640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ce00039e0 0x7f5ce0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.265+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ce0004340 msgr2=0x7f5ce0004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.265+0000 7f5cfcc40640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ce0004340 0x7f5ce0004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.265+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2210506989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce0003520 con 0x7f5ce0005020 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.265+0000 7f5cfcc40640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ce0005020 0x7f5ce000b9c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000a860 tx=0x7f5ce0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba728e1e83d97cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf000a480 con 0x7f5ce0005020 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf0034630 con 0x7f5ce0005020 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cf6ffd640 1 -- 172.21.15.77:0/2210506989 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0034aa0 con 0x7f5ce0005020 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2210506989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ce0005020 msgr2=0x7f5ce000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cfe6a3640 1 --2- 172.21.15.77:0/2210506989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ce0005020 0x7f5ce000b9c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000a860 tx=0x7f5ce0005830 comp rx=0 tx=0).stop 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2210506989 reap_dead start 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2210506989 shutdown_connections 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2210506989 >> 172.21.15.77:0/2210506989 conn(0x7f5cf80b2660 msgr2=0x7f5cf80b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2210506989 shutdown_connections 2024-03-06T07:24:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.266+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2210506989 wait complete. 2024-03-06T07:24:31.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.267+0000 7f5cfe6a3640 1 Processor -- start 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.267+0000 7f5cfe6a3640 1 -- start start 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfe6a3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf81503e0 0x7f5cf815edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfe6a3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf81507c0 0x7f5cf8155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfe6a3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf81598e0 0x7f5cf8155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfe6a3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf80b0180 con 0x7f5cf81598e0 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfe6a3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf80b0000 con 0x7f5cf81507c0 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfe6a3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf80b0300 con 0x7f5cf81503e0 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfcc40640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf81598e0 0x7f5cf8155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfcc40640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf81598e0 0x7f5cf8155ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46192/0 (socket says 172.21.15.77:46192) 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2847690513 learned_addr learned my addr 172.21.15.77:0/2847690513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cf77fe640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf81507c0 0x7f5cf8155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cf7fff640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf81503e0 0x7f5cf815edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2847690513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf81503e0 msgr2=0x7f5cf815edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfcc40640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf81503e0 0x7f5cf815edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2847690513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf81507c0 msgr2=0x7f5cf8155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfcc40640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf81507c0 0x7f5cf8155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.268+0000 7f5cfcc40640 1 -- 172.21.15.77:0/2847690513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8153a60 con 0x7f5cf81598e0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.269+0000 7f5cfcc40640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf81598e0 0x7f5cf8155ff0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5cf002cc70 tx=0x7f5cf80703f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.269+0000 7f5cf4ff9640 1 -- 172.21.15.77:0/2847690513 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0045070 con 0x7f5cf81598e0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.269+0000 7f5cf4ff9640 1 -- 172.21.15.77:0/2847690513 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf003a040 con 0x7f5cf81598e0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.269+0000 7f5cf4ff9640 1 -- 172.21.15.77:0/2847690513 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf000a480 con 0x7f5cf81598e0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.269+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf8153eb0 con 0x7f5cf81598e0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.269+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf8154470 con 0x7f5cf81598e0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.270+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf8073390 con 0x7f5cf81598e0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.270+0000 7f5cf4ff9640 1 -- 172.21.15.77:0/2847690513 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cf0048070 con 0x7f5cf81598e0 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.271+0000 7f5cf4ff9640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ce406b4d0 0x7f5ce406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.271+0000 7f5cf4ff9640 1 -- 172.21.15.77:0/2847690513 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cf00abc00 con 0x7f5cf81598e0 2024-03-06T07:24:31.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.271+0000 7f5cf7fff640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ce406b4d0 0x7f5ce406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.271+0000 7f5cf7fff640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ce406b4d0 0x7f5ce406d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8005590 tx=0x7f5ce8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.275+0000 7f5cf4ff9640 1 -- 172.21.15.77:0/2847690513 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cf00b2080 con 0x7f5cf81598e0 2024-03-06T07:24:31.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.488+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5cf80cfea0 con 0x7f5cf81598e0 2024-03-06T07:24:31.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.488+0000 7f5cf4ff9640 1 -- 172.21.15.77:0/2847690513 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5cf007a430 con 0x7f5cf81598e0 2024-03-06T07:24:31.491 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ce406b4d0 msgr2=0x7f5ce406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ce406b4d0 0x7f5ce406d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8005590 tx=0x7f5ce8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:31.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf81598e0 msgr2=0x7f5cf8155ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf81598e0 0x7f5cf8155ff0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5cf002cc70 tx=0x7f5cf80703f0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 shutdown_connections 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ce406b4d0 0x7f5ce406d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf81598e0 0x7f5cf8155ff0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5cf81507c0 0x7f5cf8155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 --2- 172.21.15.77:0/2847690513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5cf81503e0 0x7f5cf815edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 >> 172.21.15.77:0/2847690513 conn(0x7f5cf80b2660 msgr2=0x7f5cf80b5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 shutdown_connections 2024-03-06T07:24:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.490+0000 7f5cfe6a3640 1 -- 172.21.15.77:0/2847690513 wait complete. 2024-03-06T07:24:31.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:31.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.606+0000 7f158646d640 1 Processor -- start 2024-03-06T07:24:31.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.606+0000 7f158646d640 1 -- start start 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.606+0000 7f158646d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.606+0000 7f158646d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15800b0e30 0x7f15800b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.606+0000 7f158646d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15801618b0 con 0x7f15801614d0 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.606+0000 7f158646d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15800d4cc0 con 0x7f15800b0e30 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.606+0000 7f158646d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15800cf400 con 0x7f15800d4860 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f1584a0a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15800b0e30 0x7f15800b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f157ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f15801614d0 0x7f15800d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45564/0 (socket says 172.21.15.77:45564) 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f1584a0a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15800b0e30 0x7f15800b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46202/0 (socket says 172.21.15.77:46202) 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f157ffff640 1 -- 172.21.15.77:0/1427140515 learned_addr learned my addr 172.21.15.77:0/1427140515 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f157f7fe640 1 --2- 172.21.15.77:0/1427140515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f1584a0a640 1 -- 172.21.15.77:0/1427140515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 msgr2=0x7f15800d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f1584a0a640 1 --2- 172.21.15.77:0/1427140515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f1584a0a640 1 -- 172.21.15.77:0/1427140515 >> v1:172.21.15.77:6789/0 conn(0x7f15801614d0 legacy=0x7f15800d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f1584a0a640 1 -- 172.21.15.77:0/1427140515 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15800cf6a0 con 0x7f15800b0e30 2024-03-06T07:24:31.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f157effd640 1 -- 172.21.15.77:0/1427140515 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934854964 0 0) 0x7f15801618b0 con 0x7f15801614d0 2024-03-06T07:24:31.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.607+0000 7f157f7fe640 1 --2- 172.21.15.77:0/1427140515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:31.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f1584a0a640 1 --2- 172.21.15.77:0/1427140515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15800b0e30 0x7f15800b12c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1578009d90 tx=0x7f157802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15613e72f14af0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f157effd640 1 -- 172.21.15.77:0/1427140515 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1578039070 con 0x7f15800b0e30 2024-03-06T07:24:31.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f157effd640 1 -- 172.21.15.77:0/1427140515 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157802cc50 con 0x7f15800b0e30 2024-03-06T07:24:31.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f157effd640 1 -- 172.21.15.77:0/1427140515 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1578034440 con 0x7f15800b0e30 2024-03-06T07:24:31.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f158646d640 1 -- 172.21.15.77:0/1427140515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15800b0e30 msgr2=0x7f15800b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f158646d640 1 --2- 172.21.15.77:0/1427140515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15800b0e30 0x7f15800b12c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1578009d90 tx=0x7f157802bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f158646d640 1 -- 172.21.15.77:0/1427140515 shutdown_connections 2024-03-06T07:24:31.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f158646d640 1 --2- 172.21.15.77:0/1427140515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15800b0e30 0x7f15800b12c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f158646d640 1 --2- 172.21.15.77:0/1427140515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f158646d640 1 -- 172.21.15.77:0/1427140515 >> 172.21.15.77:0/1427140515 conn(0x7f15800d3d40 msgr2=0x7f15800d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f158646d640 1 -- 172.21.15.77:0/1427140515 shutdown_connections 2024-03-06T07:24:31.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.608+0000 7f158646d640 1 -- 172.21.15.77:0/1427140515 wait complete. 2024-03-06T07:24:31.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.610+0000 7f158646d640 1 Processor -- start 2024-03-06T07:24:31.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.610+0000 7f158646d640 1 -- start start 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f158646d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15800b0e30 0x7f15801f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f158646d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f158646d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15801614d0 0x7f15800b1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f158646d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15800b0910 con 0x7f15801614d0 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f158646d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15800cfc60 con 0x7f15800b0e30 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f158646d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15800d2560 con 0x7f15800d4860 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15801614d0 0x7f15800b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15801614d0 0x7f15800b1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46210/0 (socket says 172.21.15.77:46210) 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 -- 172.21.15.77:0/1833334174 learned_addr learned my addr 172.21.15.77:0/1833334174 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f157ffff640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15800b0e30 0x7f15801f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f157f7fe640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 -- 172.21.15.77:0/1833334174 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 msgr2=0x7f15800b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800b1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 -- 172.21.15.77:0/1833334174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15800b0e30 msgr2=0x7f15801f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15800b0e30 0x7f15801f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 -- 172.21.15.77:0/1833334174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158014cde0 con 0x7f15801614d0 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f1584a0a640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15801614d0 0x7f15800b1750 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1578009eb0 tx=0x7f1578004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f157cff9640 1 -- 172.21.15.77:0/1833334174 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1578039050 con 0x7f15801614d0 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f157cff9640 1 -- 172.21.15.77:0/1833334174 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1578045070 con 0x7f15801614d0 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15801f3e70 con 0x7f15801614d0 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f157cff9640 1 -- 172.21.15.77:0/1833334174 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157803e610 con 0x7f15801614d0 2024-03-06T07:24:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.612+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15801f4430 con 0x7f15801614d0 2024-03-06T07:24:31.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.613+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1580068ff0 con 0x7f15801614d0 2024-03-06T07:24:31.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.614+0000 7f157cff9640 1 -- 172.21.15.77:0/1833334174 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f157803e7b0 con 0x7f15801614d0 2024-03-06T07:24:31.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.615+0000 7f157cff9640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156c06b4d0 0x7f156c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.615+0000 7f157cff9640 1 -- 172.21.15.77:0/1833334174 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15780acf90 con 0x7f15801614d0 2024-03-06T07:24:31.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.615+0000 7f157ffff640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156c06b4d0 0x7f156c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.615+0000 7f157ffff640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156c06b4d0 0x7f156c06d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1570007fd0 tx=0x7f1570007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.619+0000 7f157cff9640 1 -- 172.21.15.77:0/1833334174 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f157807b8e0 con 0x7f15801614d0 2024-03-06T07:24:31.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.797+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f15800d3370 con 0x7f15801614d0 2024-03-06T07:24:31.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.797+0000 7f157cff9640 1 -- 172.21.15.77:0/1833334174 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f157807f980 con 0x7f15801614d0 2024-03-06T07:24:31.800 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:24:31.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156c06b4d0 msgr2=0x7f156c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156c06b4d0 0x7f156c06d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1570007fd0 tx=0x7f1570007de0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15801614d0 msgr2=0x7f15800b1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15801614d0 0x7f15800b1750 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1578009eb0 tx=0x7f1578004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 shutdown_connections 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f156c06b4d0 0x7f156c06d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15801614d0 0x7f15800b1750 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f15800d4860 0x7f15800b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 --2- 172.21.15.77:0/1833334174 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15800b0e30 0x7f15801f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 >> 172.21.15.77:0/1833334174 conn(0x7f15800d3d40 msgr2=0x7f15800d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 shutdown_connections 2024-03-06T07:24:31.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.799+0000 7f158646d640 1 -- 172.21.15.77:0/1833334174 wait complete. 2024-03-06T07:24:31.813 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:31.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.911+0000 7fe16411e640 1 Processor -- start 2024-03-06T07:24:31.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.912+0000 7fe16411e640 1 -- start start 2024-03-06T07:24:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.912+0000 7fe16411e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b4460 0x7fe15c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.912+0000 7fe16411e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe15c161cf0 0x7fe15c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.912+0000 7fe16411e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15c0d22c0 con 0x7fe15c162970 2024-03-06T07:24:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.912+0000 7fe16411e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c161b00 con 0x7fe15c161cf0 2024-03-06T07:24:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.912+0000 7fe16411e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0cecc0 con 0x7fe15c0b4460 2024-03-06T07:24:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe1616b9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe15c162970 0x7fe15c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45570/0 (socket says 172.21.15.77:45570) 2024-03-06T07:24:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe1616b9640 1 -- 172.21.15.77:0/2629928147 learned_addr learned my addr 172.21.15.77:0/2629928147 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe161eba640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b4460 0x7fe15c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe1626bb640 1 --2- 172.21.15.77:0/2629928147 >> v2:172.21.15.158:3300/0 conn(0x7fe15c161cf0 0x7fe15c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004602582 0 0) 0x7fe15c0d22c0 con 0x7fe15c162970 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe148003520 con 0x7fe15c162970 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533778160 0 0) 0x7fe148003520 con 0x7fe15c162970 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b4460 msgr2=0x7fe15c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b4460 0x7fe15c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 >> v2:172.21.15.158:3300/0 conn(0x7fe15c161cf0 msgr2=0x7fe15c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 --2- 172.21.15.77:0/2629928147 >> v2:172.21.15.158:3300/0 conn(0x7fe15c161cf0 0x7fe15c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c0d25a0 con 0x7fe15c162970 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fe14c001670 con 0x7fe15c162970 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 >> v1:172.21.15.77:6789/0 conn(0x7fe15c162970 legacy=0x7fe15c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1480039b0 0x7fe148003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe148004800 0x7fe148004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b4460 0x7fe1480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c161b00 con 0x7fe148004800 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0cecc0 con 0x7fe1480039b0 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14800b9c0 con 0x7fe15c0b4460 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe1616b9640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe148004800 0x7fe148004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe161eba640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1480039b0 0x7fe148003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe1626bb640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b4460 0x7fe1480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.913+0000 7fe1616b9640 1 -- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b4460 msgr2=0x7fe1480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.914+0000 7fe1616b9640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b4460 0x7fe1480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.914+0000 7fe1616b9640 1 -- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1480039b0 msgr2=0x7fe148003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.914+0000 7fe1616b9640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe1480039b0 0x7fe148003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.914+0000 7fe1616b9640 1 -- 172.21.15.77:0/2629928147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe148003520 con 0x7fe148004800 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.914+0000 7fe1616b9640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe148004800 0x7fe148004c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe14c001050 tx=0x7fe148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9330f7641ac2116e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.914+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c007d70 con 0x7fe148004800 2024-03-06T07:24:31.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.914+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c030070 con 0x7fe148004800 2024-03-06T07:24:31.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.914+0000 7fe160eb8640 1 -- 172.21.15.77:0/2629928147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c02cd40 con 0x7fe148004800 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.915+0000 7fe16411e640 1 -- 172.21.15.77:0/2629928147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe148004800 msgr2=0x7fe148004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.915+0000 7fe16411e640 1 --2- 172.21.15.77:0/2629928147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe148004800 0x7fe148004c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe14c001050 tx=0x7fe148005830 comp rx=0 tx=0).stop 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.915+0000 7fe1626bb640 1 -- 172.21.15.77:0/2629928147 reap_dead start 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.915+0000 7fe16411e640 1 -- 172.21.15.77:0/2629928147 shutdown_connections 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.915+0000 7fe16411e640 1 -- 172.21.15.77:0/2629928147 >> 172.21.15.77:0/2629928147 conn(0x7fe15c0d3ce0 msgr2=0x7fe15c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.915+0000 7fe16411e640 1 -- 172.21.15.77:0/2629928147 shutdown_connections 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.915+0000 7fe16411e640 1 -- 172.21.15.77:0/2629928147 wait complete. 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.916+0000 7fe16411e640 1 Processor -- start 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.916+0000 7fe16411e640 1 -- start start 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe16411e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe148004800 0x7fe15c1560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe16411e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b1960 0x7fe15c0b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe16411e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe15c156cf0 0x7fe15c157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe16411e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0ce950 con 0x7fe15c156cf0 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe16411e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0d2b60 con 0x7fe148004800 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe16411e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0b4840 con 0x7fe15c0b1960 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe15c156cf0 0x7fe15c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe15c156cf0 0x7fe15c157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46240/0 (socket says 172.21.15.77:46240) 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 -- 172.21.15.77:0/3284880918 learned_addr learned my addr 172.21.15.77:0/3284880918 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1616b9640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b1960 0x7fe15c0b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe161eba640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe148004800 0x7fe15c1560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 -- 172.21.15.77:0/3284880918 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b1960 msgr2=0x7fe15c0b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b1960 0x7fe15c0b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 -- 172.21.15.77:0/3284880918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe148004800 msgr2=0x7fe15c1560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe148004800 0x7fe15c1560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 -- 172.21.15.77:0/3284880918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c1576a0 con 0x7fe15c156cf0 2024-03-06T07:24:31.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1626bb640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe15c156cf0 0x7fe15c157160 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe154007d10 tx=0x7fe1540115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.917+0000 7fe1527fc640 1 -- 172.21.15.77:0/3284880918 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154011e40 con 0x7fe15c156cf0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.918+0000 7fe1527fc640 1 -- 172.21.15.77:0/3284880918 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15401b070 con 0x7fe15c156cf0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.918+0000 7fe1527fc640 1 -- 172.21.15.77:0/3284880918 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154003990 con 0x7fe15c156cf0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.918+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c1565e0 con 0x7fe15c156cf0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.918+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c157c20 con 0x7fe15c156cf0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.919+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c04f330 con 0x7fe15c156cf0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.919+0000 7fe1527fc640 1 -- 172.21.15.77:0/3284880918 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe154005d10 con 0x7fe15c156cf0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.919+0000 7fe1527fc640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe14006b4d0 0x7fe14006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.919+0000 7fe1527fc640 1 -- 172.21.15.77:0/3284880918 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe15408a620 con 0x7fe15c156cf0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.919+0000 7fe161eba640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe14006b4d0 0x7fe14006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.920+0000 7fe161eba640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe14006b4d0 0x7fe14006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe15c06d6c0 tx=0x7fe158008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:31.924+0000 7fe1527fc640 1 -- 172.21.15.77:0/3284880918 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe154014350 con 0x7fe15c156cf0 2024-03-06T07:24:32.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.109+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe15c06a1b0 con 0x7fe15c156cf0 2024-03-06T07:24:32.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.110+0000 7fe1527fc640 1 -- 172.21.15.77:0/3284880918 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe154059530 con 0x7fe15c156cf0 2024-03-06T07:24:32.113 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe14006b4d0 msgr2=0x7fe14006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe14006b4d0 0x7fe14006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe15c06d6c0 tx=0x7fe158008040 comp rx=0 tx=0).stop 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe15c156cf0 msgr2=0x7fe15c157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe15c156cf0 0x7fe15c157160 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe154007d10 tx=0x7fe1540115a0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 shutdown_connections 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe14006b4d0 0x7fe14006d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe15c156cf0 0x7fe15c157160 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe15c0b1960 0x7fe15c0b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 --2- 172.21.15.77:0/3284880918 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe148004800 0x7fe15c1560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 >> 172.21.15.77:0/3284880918 conn(0x7fe15c0d3ce0 msgr2=0x7fe15c0d4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 shutdown_connections 2024-03-06T07:24:32.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:32.112+0000 7fe16411e640 1 -- 172.21.15.77:0/3284880918 wait complete. 2024-03-06T07:24:32.125 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:24:32.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:32.125 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:32.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[58383]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:32.152 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:38.254 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:24:38.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:38.255 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:38.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:38.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.409+0000 7fbcd8bf6640 1 Processor -- start 2024-03-06T07:24:38.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.409+0000 7fbcd8bf6640 1 -- start start 2024-03-06T07:24:38.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd8bf6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40d22b0 0x7fbcd40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd8bf6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd40b3430 con 0x7fbcd40b41f0 2024-03-06T07:24:38.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd8bf6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd40b3710 con 0x7fbcd415dfe0 2024-03-06T07:24:38.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd8bf6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40d2c50 con 0x7fbcd40d22b0 2024-03-06T07:24:38.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd2d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbcd40b41f0 0x7fbcd40b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45582/0 (socket says 172.21.15.77:45582) 2024-03-06T07:24:38.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd2d9d640 1 -- 172.21.15.77:0/2903510704 learned_addr learned my addr 172.21.15.77:0/2903510704 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:38.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961755211 0 0) 0x7fbcd40b3430 con 0x7fbcd40b41f0 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd1d9b640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40d22b0 0x7fbcd40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcbc003520 con 0x7fbcd40b41f0 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834867021 0 0) 0x7fbcbc003520 con 0x7fbcd40b41f0 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40d22b0 msgr2=0x7fbcd40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40d22b0 0x7fbcd40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 >> v1:172.21.15.158:6789/0 conn(0x7fbcd415dfe0 legacy=0x7fbcd40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd40b39f0 con 0x7fbcd40b41f0 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fbccc0019f0 con 0x7fbcd40b41f0 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 >> v1:172.21.15.77:6789/0 conn(0x7fbcd40b41f0 legacy=0x7fbcd40b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40d22b0 0x7fbcbc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc003f60 0x7fbcbc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcbc004d40 0x7fbcbc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc00ba70 con 0x7fbcbc004d40 2024-03-06T07:24:38.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc00bc20 con 0x7fbcbc003f60 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.410+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc00be20 con 0x7fbcd40d22b0 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.411+0000 7fbcd2d9d640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcbc004d40 0x7fbcbc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.411+0000 7fbcd259c640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40d22b0 0x7fbcbc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.411+0000 7fbcd1d9b640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc003f60 0x7fbcbc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.411+0000 7fbcd2d9d640 1 -- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40d22b0 msgr2=0x7fbcbc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.411+0000 7fbcd2d9d640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40d22b0 0x7fbcbc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.411+0000 7fbcd2d9d640 1 -- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc003f60 msgr2=0x7fbcbc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.411+0000 7fbcd2d9d640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc003f60 0x7fbcbc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.411+0000 7fbcd2d9d640 1 -- 172.21.15.77:0/2903510704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc003520 con 0x7fbcbc004d40 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd2d9d640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcbc004d40 0x7fbcbc0051c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00a860 tx=0x7fbcbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9657cf2a6467078b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc00a480 con 0x7fbcbc004d40 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc034630 con 0x7fbcbc004d40 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd159a640 1 -- 172.21.15.77:0/2903510704 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc034aa0 con 0x7fbcbc004d40 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/2903510704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcbc004d40 msgr2=0x7fbcbc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd8bf6640 1 --2- 172.21.15.77:0/2903510704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcbc004d40 0x7fbcbc0051c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00a860 tx=0x7fbcbc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:38.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd2d9d640 1 -- 172.21.15.77:0/2903510704 reap_dead start 2024-03-06T07:24:38.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/2903510704 shutdown_connections 2024-03-06T07:24:38.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/2903510704 >> 172.21.15.77:0/2903510704 conn(0x7fbcd40d3cd0 msgr2=0x7fbcd40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/2903510704 shutdown_connections 2024-03-06T07:24:38.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.412+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/2903510704 wait complete. 2024-03-06T07:24:38.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd8bf6640 1 Processor -- start 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd8bf6640 1 -- start start 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd8bf6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc004d40 0x7fbcd415c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd8bf6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcd409f2a0 0x7fbcd40cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd8bf6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40a0310 0x7fbcd40a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd8bf6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40b3d60 con 0x7fbcd409f2a0 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd8bf6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40b31e0 con 0x7fbcbc004d40 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd8bf6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40d1c70 con 0x7fbcd40a0310 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.414+0000 7fbcd1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcd409f2a0 0x7fbcd40cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcd409f2a0 0x7fbcd40cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46268/0 (socket says 172.21.15.77:46268) 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd1d9b640 1 -- 172.21.15.77:0/4231029111 learned_addr learned my addr 172.21.15.77:0/4231029111 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd1d9b640 1 -- 172.21.15.77:0/4231029111 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40a0310 msgr2=0x7fbcd40a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd1d9b640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40a0310 0x7fbcd40a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd1d9b640 1 -- 172.21.15.77:0/4231029111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc004d40 msgr2=0x7fbcd415c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd259c640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc004d40 0x7fbcd415c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd1d9b640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc004d40 0x7fbcd415c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd1d9b640 1 -- 172.21.15.77:0/4231029111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd40cd0c0 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd259c640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc004d40 0x7fbcd415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd1d9b640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcd409f2a0 0x7fbcd40cc800 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8011920 tx=0x7fbcc8011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcc2ffd640 1 -- 172.21.15.77:0/4231029111 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc800faf0 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcc2ffd640 1 -- 172.21.15.77:0/4231029111 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc800fc90 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd40cd3a0 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcc2ffd640 1 -- 172.21.15.77:0/4231029111 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8014440 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.415+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd41f3bc0 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.416+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd404f330 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.417+0000 7fbcc2ffd640 1 -- 172.21.15.77:0/4231029111 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcc8012040 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.417+0000 7fbcc2ffd640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcb406b4d0 0x7fbcb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.417+0000 7fbcc2ffd640 1 -- 172.21.15.77:0/4231029111 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcc808a8c0 con 0x7fbcd409f2a0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.417+0000 7fbcd259c640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcb406b4d0 0x7fbcb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.417+0000 7fbcd259c640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcb406b4d0 0x7fbcb406d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40b9c90 tx=0x7fbcc4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.422+0000 7fbcc2ffd640 1 -- 172.21.15.77:0/4231029111 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcc8019030 con 0x7fbcd409f2a0 2024-03-06T07:24:38.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.647+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcd40b2e10 con 0x7fbcd409f2a0 2024-03-06T07:24:38.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.647+0000 7fbcc2ffd640 1 -- 172.21.15.77:0/4231029111 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fbcc8059740 con 0x7fbcd409f2a0 2024-03-06T07:24:38.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:38.650 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.760628+0000","modified":"2024-03-06T07:24:12.764047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/251375557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":251375557},{"type":"v1","addr":"172.21.15.77:6837","nonce":251375557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-03-06T07:24:38.650 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcb406b4d0 msgr2=0x7fbcb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcb406b4d0 0x7fbcb406d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40b9c90 tx=0x7fbcc4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcd409f2a0 msgr2=0x7fbcd40cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcd409f2a0 0x7fbcd40cc800 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8011920 tx=0x7fbcc8011de0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 shutdown_connections 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcb406b4d0 0x7fbcb406d990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcd409f2a0 0x7fbcd40cc800 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcd40a0310 0x7fbcd40a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 --2- 172.21.15.77:0/4231029111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcbc004d40 0x7fbcd415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.649+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 >> 172.21.15.77:0/4231029111 conn(0x7fbcd40d3cd0 msgr2=0x7fbcd41603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.650+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 shutdown_connections 2024-03-06T07:24:38.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.650+0000 7fbcd8bf6640 1 -- 172.21.15.77:0/4231029111 wait complete. 2024-03-06T07:24:38.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:24:38.782 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:24:38.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:38.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.881+0000 7f5948888640 1 Processor -- start 2024-03-06T07:24:38.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.881+0000 7f5948888640 1 -- start start 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5948888640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f59440cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5948888640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59440cf830 0x7f59440d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5948888640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59440d27f0 0x7f59440d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5948888640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59440d2c50 con 0x7f59440d27f0 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5948888640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59440b31e0 con 0x7f59440cf830 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5948888640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59440b3d60 con 0x7f594415dfe0 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5942d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59440d27f0 0x7f59440d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5942d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59440d27f0 0x7f59440d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44102/0 (socket says 172.21.15.77:44102) 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5942d9d640 1 -- 172.21.15.77:0/3182208522 learned_addr learned my addr 172.21.15.77:0/3182208522 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f594259c640 1 --2- 172.21.15.77:0/3182208522 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f59440cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5941d9b640 1 --2- 172.21.15.77:0/3182208522 >> v2:172.21.15.158:3300/0 conn(0x7f59440cf830 0x7f59440d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5942d9d640 1 -- 172.21.15.77:0/3182208522 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 msgr2=0x7f59440cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5942d9d640 1 --2- 172.21.15.77:0/3182208522 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f59440cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5942d9d640 1 -- 172.21.15.77:0/3182208522 >> v2:172.21.15.158:3300/0 conn(0x7f59440cf830 msgr2=0x7f59440d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5942d9d640 1 --2- 172.21.15.77:0/3182208522 >> v2:172.21.15.158:3300/0 conn(0x7f59440cf830 0x7f59440d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5942d9d640 1 -- 172.21.15.77:0/3182208522 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59440b3430 con 0x7f59440d27f0 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.882+0000 7f5941d9b640 1 --2- 172.21.15.77:0/3182208522 >> v2:172.21.15.158:3300/0 conn(0x7f59440cf830 0x7f59440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.883+0000 7f5942d9d640 1 --2- 172.21.15.77:0/3182208522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440d27f0 0x7f59440d0300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f594408cba0 tx=0x7f593402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c3b40acfb5bf49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.883+0000 7f594159a640 1 -- 172.21.15.77:0/3182208522 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934035020 con 0x7f59440d27f0 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.883+0000 7f594159a640 1 -- 172.21.15.77:0/3182208522 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5934031870 con 0x7f59440d27f0 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.883+0000 7f594159a640 1 -- 172.21.15.77:0/3182208522 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934031c40 con 0x7f59440d27f0 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 -- 172.21.15.77:0/3182208522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440d27f0 msgr2=0x7f59440d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 --2- 172.21.15.77:0/3182208522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440d27f0 0x7f59440d0300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f594408cba0 tx=0x7f593402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 -- 172.21.15.77:0/3182208522 shutdown_connections 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 --2- 172.21.15.77:0/3182208522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440d27f0 0x7f59440d0300 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 --2- 172.21.15.77:0/3182208522 >> v2:172.21.15.158:3300/0 conn(0x7f59440cf830 0x7f59440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 --2- 172.21.15.77:0/3182208522 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f59440cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 -- 172.21.15.77:0/3182208522 >> 172.21.15.77:0/3182208522 conn(0x7f59440d3cd0 msgr2=0x7f59440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 -- 172.21.15.77:0/3182208522 shutdown_connections 2024-03-06T07:24:38.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.884+0000 7f5948888640 1 -- 172.21.15.77:0/3182208522 wait complete. 2024-03-06T07:24:38.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.885+0000 7f5948888640 1 Processor -- start 2024-03-06T07:24:38.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.885+0000 7f5948888640 1 -- start start 2024-03-06T07:24:38.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5948888640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440cf830 0x7f5944156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5948888640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59440d27f0 0x7f594415f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5948888640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f594415f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5948888640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59440d1c70 con 0x7f59440cf830 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5948888640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59440b39f0 con 0x7f59440d27f0 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5948888640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59440d1df0 con 0x7f594415dfe0 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f594259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440cf830 0x7f5944156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f594259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440cf830 0x7f5944156900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44110/0 (socket says 172.21.15.77:44110) 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f594259c640 1 -- 172.21.15.77:0/2723767823 learned_addr learned my addr 172.21.15.77:0/2723767823 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5941d9b640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59440d27f0 0x7f594415f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5942d9d640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f594415f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f594259c640 1 -- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 msgr2=0x7f594415f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f594259c640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f594415f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f594259c640 1 -- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59440d27f0 msgr2=0x7f594415f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f594259c640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59440d27f0 0x7f594415f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f594259c640 1 -- 172.21.15.77:0/2723767823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594415ec10 con 0x7f59440cf830 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5941d9b640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59440d27f0 0x7f594415f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.886+0000 7f5942d9d640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f594415f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:38.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.887+0000 7f594259c640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440cf830 0x7f5944156900 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f593c007e70 tx=0x7f593c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.887+0000 7f5932ffd640 1 -- 172.21.15.77:0/2723767823 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c014620 con 0x7f59440cf830 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.887+0000 7f5932ffd640 1 -- 172.21.15.77:0/2723767823 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c019070 con 0x7f59440cf830 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.887+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594415eef0 con 0x7f59440cf830 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.887+0000 7f5932ffd640 1 -- 172.21.15.77:0/2723767823 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c014a00 con 0x7f59440cf830 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.887+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59440b6d30 con 0x7f59440cf830 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.888+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594404f330 con 0x7f59440cf830 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.888+0000 7f5932ffd640 1 -- 172.21.15.77:0/2723767823 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f593c014ba0 con 0x7f59440cf830 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.888+0000 7f5932ffd640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f592406b4d0 0x7f592406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.889+0000 7f5932ffd640 1 -- 172.21.15.77:0/2723767823 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f593c08ab00 con 0x7f59440cf830 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.889+0000 7f5941d9b640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f592406b4d0 0x7f592406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.889+0000 7f5941d9b640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f592406b4d0 0x7f592406d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f59441f7090 tx=0x7f5938008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:38.897+0000 7f5932ffd640 1 -- 172.21.15.77:0/2723767823 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f593c0598f0 con 0x7f59440cf830 2024-03-06T07:24:39.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.186+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f594415d4c0 con 0x7f59440cf830 2024-03-06T07:24:39.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.186+0000 7f5932ffd640 1 -- 172.21.15.77:0/2723767823 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f593c0591f0 con 0x7f59440cf830 2024-03-06T07:24:39.189 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:39.189 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.760628+0000","modified":"2024-03-06T07:24:12.764047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/251375557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":251375557},{"type":"v1","addr":"172.21.15.77:6837","nonce":251375557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-03-06T07:24:39.189 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-06T07:24:39.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f592406b4d0 msgr2=0x7f592406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f592406b4d0 0x7f592406d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f59441f7090 tx=0x7f5938008040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440cf830 msgr2=0x7f5944156900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440cf830 0x7f5944156900 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f593c007e70 tx=0x7f593c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 shutdown_connections 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f594415dfe0 0x7f594415f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59440d27f0 0x7f594415f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f592406b4d0 0x7f592406d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 --2- 172.21.15.77:0/2723767823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59440cf830 0x7f5944156900 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 >> 172.21.15.77:0/2723767823 conn(0x7f59440d3cd0 msgr2=0x7f594406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 shutdown_connections 2024-03-06T07:24:39.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.188+0000 7f5948888640 1 -- 172.21.15.77:0/2723767823 wait complete. 2024-03-06T07:24:39.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:24:39.333 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:24:39.333 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:24:39.333 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:39.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.433+0000 7f6c91247640 1 Processor -- start 2024-03-06T07:24:39.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.433+0000 7f6c91247640 1 -- start start 2024-03-06T07:24:39.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.433+0000 7f6c91247640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c8c15dfe0 0x7f6c8c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.433+0000 7f6c91247640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c8c0d29c0 0x7f6c8c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.433+0000 7f6c91247640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8c0b3430 con 0x7f6c8c0cf830 2024-03-06T07:24:39.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.433+0000 7f6c91247640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c0b31e0 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.433+0000 7f6c91247640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c0b3d60 con 0x7f6c8c15dfe0 2024-03-06T07:24:39.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.434+0000 7f6c8b59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c8c0d29c0 0x7f6c8c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.434+0000 7f6c8b59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c8c0d29c0 0x7f6c8c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44112/0 (socket says 172.21.15.77:44112) 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.434+0000 7f6c8b59e640 1 -- 172.21.15.77:0/1898027343 learned_addr learned my addr 172.21.15.77:0/1898027343 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.434+0000 7f6c8ad9d640 1 --2- 172.21.15.77:0/1898027343 >> v2:172.21.15.158:3300/0 conn(0x7f6c8c15dfe0 0x7f6c8c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.434+0000 7f6c8b59e640 1 -- 172.21.15.77:0/1898027343 >> v2:172.21.15.158:3300/0 conn(0x7f6c8c15dfe0 msgr2=0x7f6c8c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.434+0000 7f6c8b59e640 1 --2- 172.21.15.77:0/1898027343 >> v2:172.21.15.158:3300/0 conn(0x7f6c8c15dfe0 0x7f6c8c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.434+0000 7f6c8b59e640 1 -- 172.21.15.77:0/1898027343 >> v1:172.21.15.158:6789/0 conn(0x7f6c8c0cf830 legacy=0x7f6c8c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.434+0000 7f6c8b59e640 1 -- 172.21.15.77:0/1898027343 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c0b3710 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c8b59e640 1 --2- 172.21.15.77:0/1898027343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b41f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c067990 tx=0x7f6c8402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104d116ca631e13c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c89d9b640 1 -- 172.21.15.77:0/1898027343 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8402cb20 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c89d9b640 1 -- 172.21.15.77:0/1898027343 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8402ccc0 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c89d9b640 1 -- 172.21.15.77:0/1898027343 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84034610 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c91247640 1 -- 172.21.15.77:0/1898027343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 msgr2=0x7f6c8c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c91247640 1 --2- 172.21.15.77:0/1898027343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b41f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c067990 tx=0x7f6c8402bf80 comp rx=0 tx=0).stop 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c91247640 1 -- 172.21.15.77:0/1898027343 shutdown_connections 2024-03-06T07:24:39.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c91247640 1 --2- 172.21.15.77:0/1898027343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b41f0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c91247640 1 --2- 172.21.15.77:0/1898027343 >> v2:172.21.15.158:3300/0 conn(0x7f6c8c15dfe0 0x7f6c8c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c91247640 1 -- 172.21.15.77:0/1898027343 >> 172.21.15.77:0/1898027343 conn(0x7f6c8c0d3cd0 msgr2=0x7f6c8c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c91247640 1 -- 172.21.15.77:0/1898027343 shutdown_connections 2024-03-06T07:24:39.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.435+0000 7f6c91247640 1 -- 172.21.15.77:0/1898027343 wait complete. 2024-03-06T07:24:39.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.437+0000 7f6c91247640 1 Processor -- start 2024-03-06T07:24:39.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.437+0000 7f6c91247640 1 -- start start 2024-03-06T07:24:39.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.437+0000 7f6c91247640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c0cf830 0x7f6c8c150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.437+0000 7f6c91247640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.437+0000 7f6c91247640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c8c15dfe0 0x7f6c8c0b93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.437+0000 7f6c91247640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0d1df0 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.437+0000 7f6c91247640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0d1c70 con 0x7f6c8c0cf830 2024-03-06T07:24:39.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.437+0000 7f6c91247640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c15e3c0 con 0x7f6c8c15dfe0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44116/0 (socket says 172.21.15.77:44116) 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 -- 172.21.15.77:0/3668799752 learned_addr learned my addr 172.21.15.77:0/3668799752 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8ad9d640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c0cf830 0x7f6c8c150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8b59e640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c8c15dfe0 0x7f6c8c0b93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 -- 172.21.15.77:0/3668799752 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c8c15dfe0 msgr2=0x7f6c8c0b93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c8c15dfe0 0x7f6c8c0b93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 -- 172.21.15.77:0/3668799752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c0cf830 msgr2=0x7f6c8c150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c0cf830 0x7f6c8c150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 -- 172.21.15.77:0/3668799752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c0cbe00 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c8a59c640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b8eb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000c440 tx=0x7f6c80001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c737fe640 1 -- 172.21.15.77:0/3668799752 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c800052d0 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c737fe640 1 -- 172.21.15.77:0/3668799752 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80005470 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c0cc0e0 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c737fe640 1 -- 172.21.15.77:0/3668799752 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80007260 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.438+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c0cc6a0 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.439+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c0cca80 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.440+0000 7f6c737fe640 1 -- 172.21.15.77:0/3668799752 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c80007400 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.440+0000 7f6c737fe640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6c7806b4d0 0x7f6c7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.440+0000 7f6c737fe640 1 -- 172.21.15.77:0/3668799752 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c80089710 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.440+0000 7f6c8ad9d640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6c7806b4d0 0x7f6c7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.441+0000 7f6c8ad9d640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6c7806b4d0 0x7f6c7806d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c006fd0 tx=0x7f6c7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.445+0000 7f6c737fe640 1 -- 172.21.15.77:0/3668799752 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c80058590 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.660+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6c8c0cebe0 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.661+0000 7f6c737fe640 1 -- 172.21.15.77:0/3668799752 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6c80057e90 con 0x7f6c8c0d29c0 2024-03-06T07:24:39.663 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6c7806b4d0 msgr2=0x7f6c7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6c7806b4d0 0x7f6c7806d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c006fd0 tx=0x7f6c7c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 msgr2=0x7f6c8c0b8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b8eb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000c440 tx=0x7f6c80001e70 comp rx=0 tx=0).stop 2024-03-06T07:24:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 shutdown_connections 2024-03-06T07:24:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6c7806b4d0 0x7f6c7806d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c0d29c0 0x7f6c8c0b8eb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c8c15dfe0 0x7f6c8c0b93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 --2- 172.21.15.77:0/3668799752 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c8c0cf830 0x7f6c8c150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 >> 172.21.15.77:0/3668799752 conn(0x7f6c8c0d3cd0 msgr2=0x7f6c8c06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 shutdown_connections 2024-03-06T07:24:39.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:39.663+0000 7f6c91247640 1 -- 172.21.15.77:0/3668799752 wait complete. 2024-03-06T07:24:39.675 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:39.676 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:39.676 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:24:39.676 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:39.676 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:39.676 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:39.676 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:39.676 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:39.676 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:39.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:39.676 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:24:39.691 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:39.691 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:39.691 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:39.691 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:39.691 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:39.691 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42517sec preferred_lft 42517sec 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:39.692 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:39.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:39.693 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:39.757 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:24:39.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:39.758 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:24:39.814 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:24:39.814 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:39.815 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:24:39.815 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:24:39.815 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:39.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:39.815 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:24:39.815 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:24:39.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:24:39.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:24:39.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:24:39.815 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:24:39.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:39.910 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:24:39.925 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:39.925 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:39.925 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:39.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:39.925 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:24:39.993 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:39.993 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:39.993 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:40.048 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:40.048 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:40.048 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:40.102 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:24:40.169 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:40.225 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:24:40.225 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:24:40.225 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:24:40.225 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:24:40.225 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:24:40.225 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:24:40.225 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:40.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:40.228 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:24:40.280 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:40.281 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:40.322 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:24:40.349 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:24:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:24:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:24:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:24:40.405 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:24:40.406 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:40.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:40.408 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:24:40.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:24:40.410+0000 7fd21ae46480 -1 init, newargv = 0x55d1f4e945b0 newargc=15 2024-03-06T07:24:40.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[59547]: starting ceph client 2024-03-06T07:24:40.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[59547]: starting fuse 2024-03-06T07:24:40.427 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:24:40.427 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:24:40.428 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:40.429 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:24:40.547 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.59547.asok 2024-03-06T07:24:40.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:40.557 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59547.asok 2024-03-06T07:24:40.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:40.557 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59547.asok status 2024-03-06T07:24:40.695 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:24:40.695 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:24:40.695 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:40.695 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:40.695 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:24:40.695 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:24:40.695 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:40.695 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "59547", 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 6312, 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 6312 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 280364852 2024-03-06T07:24:40.696 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 280364852 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.6312 192.168.0.1:0/280364852", 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/280364852", 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 26, 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 58, 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:24:40.697 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:24:40.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:40.703 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:24:40.704 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:40.719 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:24:40.719 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:40.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:40.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:40.786 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:40.786 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> import uuid 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> n = 75 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = False 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> if 0 > 0: 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = True 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:40.842 DEBUG:teuthology.orchestra.run.smithi077:> dpath = os.path.dirname(path) 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> try: 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> for i in range(n): 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> f.write(f"{i}") 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> f.flush() 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(f.fileno()) 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> os.unlink(fpath) 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:24:40.843 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:24:40.844 DEBUG:teuthology.orchestra.run.smithi077:> if create_hard_links: 2024-03-06T07:24:40.844 DEBUG:teuthology.orchestra.run.smithi077:> for j in range(0): 2024-03-06T07:24:40.844 DEBUG:teuthology.orchestra.run.smithi077:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:40.844 DEBUG:teuthology.orchestra.run.smithi077:> if True: 2024-03-06T07:24:40.844 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:24:40.844 DEBUG:teuthology.orchestra.run.smithi077:> finally: 2024-03-06T07:24:40.844 DEBUG:teuthology.orchestra.run.smithi077:> os.close(dirfd) 2024-03-06T07:24:40.844 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:24:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:41.293 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:41.294 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:41.372 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:41.415 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:41.416 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:41.416 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:41.432 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:41.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.570+0000 7fabefd0d640 1 Processor -- start 2024-03-06T07:24:41.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.570+0000 7fabefd0d640 1 -- start start 2024-03-06T07:24:41.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabefd0d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabefd0d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fabe80cf830 0x7fabe80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabefd0d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe80b3430 con 0x7fabe80d27f0 2024-03-06T07:24:41.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabefd0d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabe80d2c50 con 0x7fabe80cf830 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabefd0d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80b31e0 con 0x7fabe815dfe0 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabedaa9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabed2a8640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fabe80cf830 0x7fabe80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabee2aa640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fabe80d27f0 0x7fabe80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:36030/0 (socket says 172.21.15.77:36030) 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.571+0000 7fabee2aa640 1 -- 172.21.15.77:0/3562577897 learned_addr learned my addr 172.21.15.77:0/3562577897 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.572+0000 7fabedaa9640 1 -- 172.21.15.77:0/3562577897 >> v2:172.21.15.158:3300/0 conn(0x7fabe80cf830 msgr2=0x7fabe80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.572+0000 7fabedaa9640 1 --2- 172.21.15.77:0/3562577897 >> v2:172.21.15.158:3300/0 conn(0x7fabe80cf830 0x7fabe80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.572+0000 7fabedaa9640 1 -- 172.21.15.77:0/3562577897 >> v1:172.21.15.158:6789/0 conn(0x7fabe80d27f0 legacy=0x7fabe80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.572+0000 7fabedaa9640 1 -- 172.21.15.77:0/3562577897 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80b3710 con 0x7fabe815dfe0 2024-03-06T07:24:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.572+0000 7fabed2a8640 1 --2- 172.21.15.77:0/3562577897 >> v2:172.21.15.158:3300/0 conn(0x7fabe80cf830 0x7fabe80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:41.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.572+0000 7fabedaa9640 1 --2- 172.21.15.77:0/3562577897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe80cf2f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fabe40097a0 tx=0x7fabe402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bedafab9399d7f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabecaa7640 1 -- 172.21.15.77:0/3562577897 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4035020 con 0x7fabe815dfe0 2024-03-06T07:24:41.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabecaa7640 1 -- 172.21.15.77:0/3562577897 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe4031830 con 0x7fabe815dfe0 2024-03-06T07:24:41.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabecaa7640 1 -- 172.21.15.77:0/3562577897 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4031c00 con 0x7fabe815dfe0 2024-03-06T07:24:41.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabefd0d640 1 -- 172.21.15.77:0/3562577897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 msgr2=0x7fabe80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3562577897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe80cf2f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fabe40097a0 tx=0x7fabe402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabefd0d640 1 -- 172.21.15.77:0/3562577897 shutdown_connections 2024-03-06T07:24:41.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3562577897 >> v2:172.21.15.158:3300/0 conn(0x7fabe80cf830 0x7fabe80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3562577897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe80cf2f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabefd0d640 1 -- 172.21.15.77:0/3562577897 >> 172.21.15.77:0/3562577897 conn(0x7fabe80d3cd0 msgr2=0x7fabe80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabefd0d640 1 -- 172.21.15.77:0/3562577897 shutdown_connections 2024-03-06T07:24:41.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.573+0000 7fabefd0d640 1 -- 172.21.15.77:0/3562577897 wait complete. 2024-03-06T07:24:41.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.574+0000 7fabefd0d640 1 Processor -- start 2024-03-06T07:24:41.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.574+0000 7fabefd0d640 1 -- start start 2024-03-06T07:24:41.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabefd0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabe80cf830 0x7fabe80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabefd0d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabe80d27f0 0x7fabe80b8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabefd0d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe815b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabefd0d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80d1c70 con 0x7fabe80cf830 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabefd0d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80b3d60 con 0x7fabe80d27f0 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabefd0d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80d1df0 con 0x7fabe815dfe0 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabedaa9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabe80cf830 0x7fabe80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabedaa9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabe80cf830 0x7fabe80cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44128/0 (socket says 172.21.15.77:44128) 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabedaa9640 1 -- 172.21.15.77:0/3094036318 learned_addr learned my addr 172.21.15.77:0/3094036318 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabee2aa640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe815b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabedaa9640 1 -- 172.21.15.77:0/3094036318 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 msgr2=0x7fabe815b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabed2a8640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabe80d27f0 0x7fabe80b8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabedaa9640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe815b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabedaa9640 1 -- 172.21.15.77:0/3094036318 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabe80d27f0 msgr2=0x7fabe80b8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabedaa9640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabe80d27f0 0x7fabe80b8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.575+0000 7fabedaa9640 1 -- 172.21.15.77:0/3094036318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80b7d30 con 0x7fabe80cf830 2024-03-06T07:24:41.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.576+0000 7fabedaa9640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabe80cf830 0x7fabe80cd7c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fabe4034040 tx=0x7fabe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.576+0000 7fabde7fc640 1 -- 172.21.15.77:0/3094036318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4040070 con 0x7fabe80cf830 2024-03-06T07:24:41.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.576+0000 7fabde7fc640 1 -- 172.21.15.77:0/3094036318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe403a910 con 0x7fabe80cf830 2024-03-06T07:24:41.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.576+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe80cba00 con 0x7fabe80cf830 2024-03-06T07:24:41.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.576+0000 7fabde7fc640 1 -- 172.21.15.77:0/3094036318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4035040 con 0x7fabe80cf830 2024-03-06T07:24:41.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.576+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe815c3c0 con 0x7fabe80cf830 2024-03-06T07:24:41.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.577+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe804f330 con 0x7fabe80cf830 2024-03-06T07:24:41.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.578+0000 7fabde7fc640 1 -- 172.21.15.77:0/3094036318 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabe4038020 con 0x7fabe80cf830 2024-03-06T07:24:41.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.578+0000 7fabde7fc640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabcc06b4d0 0x7fabcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.578+0000 7fabed2a8640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabcc06b4d0 0x7fabcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.578+0000 7fabde7fc640 1 -- 172.21.15.77:0/3094036318 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabe40abbd0 con 0x7fabe80cf830 2024-03-06T07:24:41.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.579+0000 7fabed2a8640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabcc06b4d0 0x7fabcc06d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009560 tx=0x7fabd8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.583+0000 7fabde7fc640 1 -- 172.21.15.77:0/3094036318 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabe4031800 con 0x7fabe80cf830 2024-03-06T07:24:41.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.761+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabe80d0d20 con 0x7fabe80cf830 2024-03-06T07:24:41.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.762+0000 7fabde7fc640 1 -- 172.21.15.77:0/3094036318 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fabe407aae0 con 0x7fabe80cf830 2024-03-06T07:24:41.764 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:41.765 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:12.416501+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabcc06b4d0 msgr2=0x7fabcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabcc06b4d0 0x7fabcc06d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009560 tx=0x7fabd8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabe80cf830 msgr2=0x7fabe80cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabe80cf830 0x7fabe80cd7c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fabe4034040 tx=0x7fabe40046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 shutdown_connections 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fabe815dfe0 0x7fabe815b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabe80d27f0 0x7fabe80b8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fabcc06b4d0 0x7fabcc06d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 --2- 172.21.15.77:0/3094036318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabe80cf830 0x7fabe80cd7c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 >> 172.21.15.77:0/3094036318 conn(0x7fabe80d3cd0 msgr2=0x7fabe806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 shutdown_connections 2024-03-06T07:24:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.764+0000 7fabefd0d640 1 -- 172.21.15.77:0/3094036318 wait complete. 2024-03-06T07:24:41.777 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:08.975524+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.729967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:41.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:41.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.880+0000 7faa2f684640 1 Processor -- start 2024-03-06T07:24:41.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.880+0000 7faa2f684640 1 -- start start 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2f684640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faa281586e0 0x7faa280aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2f684640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa28158ac0 con 0x7faa280aef80 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2f684640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2815b2a0 con 0x7faa280d01a0 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2f684640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2815b540 con 0x7faa281586e0 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2cc1f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faa280aef80 0x7faa280cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41226/0 (socket says 172.21.15.77:41226) 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2cc1f640 1 -- 172.21.15.77:0/620957185 learned_addr learned my addr 172.21.15.77:0/620957185 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2d420640 1 --2- 172.21.15.77:0/620957185 >> v2:172.21.15.158:3300/0 conn(0x7faa281586e0 0x7faa280aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054047583 0 0) 0x7faa28158ac0 con 0x7faa280aef80 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa14005730 con 0x7faa280aef80 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661894890 0 0) 0x7faa2815b2a0 con 0x7faa280d01a0 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa28158ac0 con 0x7faa280d01a0 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4001755964 0 0) 0x7faa14005730 con 0x7faa280aef80 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 >> v2:172.21.15.158:3300/0 conn(0x7faa281586e0 msgr2=0x7faa280aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 --2- 172.21.15.77:0/620957185 >> v2:172.21.15.158:3300/0 conn(0x7faa281586e0 0x7faa280aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 >> v1:172.21.15.158:6789/0 conn(0x7faa280d01a0 legacy=0x7faa280b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2815b820 con 0x7faa280aef80 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7faa18001670 con 0x7faa280aef80 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 >> v1:172.21.15.77:6789/0 conn(0x7faa280aef80 legacy=0x7faa280cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa14005d50 0x7faa14006150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa14006a10 0x7faa14006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa140073d0 0x7faa1400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2815b540 con 0x7faa140073d0 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1400e300 con 0x7faa14005d50 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1400e500 con 0x7faa14006a10 2024-03-06T07:24:41.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2d420640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa14005d50 0x7faa14006150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2dc21640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa140073d0 0x7faa1400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.881+0000 7faa2cc1f640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa14006a10 0x7faa14006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.882+0000 7faa2dc21640 1 -- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa14006a10 msgr2=0x7faa14006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.882+0000 7faa2dc21640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa14006a10 0x7faa14006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.882+0000 7faa2dc21640 1 -- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa14005d50 msgr2=0x7faa14006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.882+0000 7faa2dc21640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa14005d50 0x7faa14006150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.882+0000 7faa2dc21640 1 -- 172.21.15.77:0/620957185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14005730 con 0x7faa140073d0 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.882+0000 7faa2d420640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa14005d50 0x7faa14006150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.883+0000 7faa2dc21640 1 -- 172.21.15.77:0/620957185 reap_dead start 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.883+0000 7faa2dc21640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa140073d0 0x7faa1400dbd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7faa20001fd0 tx=0x7faa14007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe679a3d8de42704 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.883+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24009690 con 0x7faa140073d0 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.883+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa20003b70 con 0x7faa140073d0 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.883+0000 7faa1ffff640 1 -- 172.21.15.77:0/620957185 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20001bf0 con 0x7faa140073d0 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.883+0000 7faa2f684640 1 -- 172.21.15.77:0/620957185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa140073d0 msgr2=0x7faa1400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.883+0000 7faa2f684640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa140073d0 0x7faa1400dbd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7faa20001fd0 tx=0x7faa14007a40 comp rx=0 tx=0).stop 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.884+0000 7faa2f684640 1 -- 172.21.15.77:0/620957185 shutdown_connections 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.884+0000 7faa2f684640 1 --2- 172.21.15.77:0/620957185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa140073d0 0x7faa1400dbd0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.884+0000 7faa2f684640 1 -- 172.21.15.77:0/620957185 >> 172.21.15.77:0/620957185 conn(0x7faa2806b190 msgr2=0x7faa2806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.884+0000 7faa2f684640 1 -- 172.21.15.77:0/620957185 shutdown_connections 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.884+0000 7faa2f684640 1 -- 172.21.15.77:0/620957185 wait complete. 2024-03-06T07:24:41.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.885+0000 7faa2f684640 1 Processor -- start 2024-03-06T07:24:41.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.885+0000 7faa2f684640 1 -- start start 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.885+0000 7faa2f684640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa140073d0 0x7faa281471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.885+0000 7faa2f684640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa28150230 0x7faa281f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.885+0000 7faa2f684640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa281526c0 0x7faa28152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.885+0000 7faa2f684640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28071350 con 0x7faa281526c0 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.885+0000 7faa2f684640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2815bde0 con 0x7faa28150230 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.885+0000 7faa2f684640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa280714d0 con 0x7faa140073d0 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa281526c0 0x7faa28152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa281526c0 0x7faa28152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44138/0 (socket says 172.21.15.77:44138) 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 -- 172.21.15.77:0/1233578791 learned_addr learned my addr 172.21.15.77:0/1233578791 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2cc1f640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa28150230 0x7faa281f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2d420640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa140073d0 0x7faa281471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 -- 172.21.15.77:0/1233578791 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa140073d0 msgr2=0x7faa281471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa140073d0 0x7faa281471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 -- 172.21.15.77:0/1233578791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa28150230 msgr2=0x7faa281f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa28150230 0x7faa281f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 -- 172.21.15.77:0/1233578791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa281561b0 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2dc21640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa281526c0 0x7faa28152b30 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7faa200012e0 tx=0x7faa20001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa1dffb640 1 -- 172.21.15.77:0/1233578791 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20001bf0 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa1dffb640 1 -- 172.21.15.77:0/1233578791 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa20003970 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa281f3880 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa1dffb640 1 -- 172.21.15.77:0/1233578791 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20005810 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.886+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa281f35a0 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.887+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa280732b0 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.888+0000 7faa1dffb640 1 -- 172.21.15.77:0/1233578791 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa2002c570 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.888+0000 7faa1dffb640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa0c06b4d0 0x7faa0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.888+0000 7faa1dffb640 1 -- 172.21.15.77:0/1233578791 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa200aaf50 con 0x7faa281526c0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.888+0000 7faa2d420640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa0c06b4d0 0x7faa0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.889+0000 7faa2d420640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa0c06b4d0 0x7faa0c06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faa2815abc0 tx=0x7faa24009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:41.893+0000 7faa1dffb640 1 -- 172.21.15.77:0/1233578791 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa20079e60 con 0x7faa281526c0 2024-03-06T07:24:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.125+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa2806b8f0 con 0x7faa281526c0 2024-03-06T07:24:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.125+0000 7faa1dffb640 1 -- 172.21.15.77:0/1233578791 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa20079760 con 0x7faa281526c0 2024-03-06T07:24:42.127 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:24:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa0c06b4d0 msgr2=0x7faa0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa0c06b4d0 0x7faa0c06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faa2815abc0 tx=0x7faa24009190 comp rx=0 tx=0).stop 2024-03-06T07:24:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa281526c0 msgr2=0x7faa28152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa281526c0 0x7faa28152b30 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7faa200012e0 tx=0x7faa20001650 comp rx=0 tx=0).stop 2024-03-06T07:24:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 shutdown_connections 2024-03-06T07:24:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa0c06b4d0 0x7faa0c06d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa281526c0 0x7faa28152b30 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa28150230 0x7faa281f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 --2- 172.21.15.77:0/1233578791 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa140073d0 0x7faa281471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 >> 172.21.15.77:0/1233578791 conn(0x7faa2806b190 msgr2=0x7faa280b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 shutdown_connections 2024-03-06T07:24:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.127+0000 7faa2f684640 1 -- 172.21.15.77:0/1233578791 wait complete. 2024-03-06T07:24:42.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.242+0000 7f087697a640 1 Processor -- start 2024-03-06T07:24:42.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.242+0000 7f087697a640 1 -- start start 2024-03-06T07:24:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.242+0000 7f087697a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08700d4860 0x7f08700d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.242+0000 7f087697a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08701618b0 con 0x7f08700b0e30 2024-03-06T07:24:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.242+0000 7f087697a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08700cf400 con 0x7f08701614d0 2024-03-06T07:24:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.242+0000 7f087697a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08700d4cc0 con 0x7f08700d4860 2024-03-06T07:24:42.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f0874f17640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f08700b0e30 0x7f08700b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41230/0 (socket says 172.21.15.77:41230) 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f0874f17640 1 -- 172.21.15.77:0/1990386981 learned_addr learned my addr 172.21.15.77:0/1990386981 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086f7fe640 1 --2- 172.21.15.77:0/1990386981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08700d4860 0x7f08700d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086effd640 1 -- 172.21.15.77:0/1990386981 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078937915 0 0) 0x7f08701618b0 con 0x7f08700b0e30 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086effd640 1 -- 172.21.15.77:0/1990386981 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0858003520 con 0x7f08700b0e30 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086effd640 1 -- 172.21.15.77:0/1990386981 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087285342 0 0) 0x7f08700cf400 con 0x7f08701614d0 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086effd640 1 -- 172.21.15.77:0/1990386981 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08701618b0 con 0x7f08701614d0 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086effd640 1 -- 172.21.15.77:0/1990386981 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804366818 0 0) 0x7f0858003520 con 0x7f08700b0e30 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086f7fe640 1 -- 172.21.15.77:0/1990386981 >> v1:172.21.15.158:6789/0 conn(0x7f08701614d0 legacy=0x7f08700d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086f7fe640 1 -- 172.21.15.77:0/1990386981 >> v1:172.21.15.77:6789/0 conn(0x7f08700b0e30 legacy=0x7f08700b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.243+0000 7f086f7fe640 1 -- 172.21.15.77:0/1990386981 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08700cf6a0 con 0x7f08700d4860 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.244+0000 7f086f7fe640 1 --2- 172.21.15.77:0/1990386981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08700d4860 0x7f08700d2020 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f08640097d0 tx=0x7f086402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afdfd94ee2eaff6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.244+0000 7f086effd640 1 -- 172.21.15.77:0/1990386981 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864037070 con 0x7f08700d4860 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.244+0000 7f086effd640 1 -- 172.21.15.77:0/1990386981 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0864030510 con 0x7f08700d4860 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.244+0000 7f086effd640 1 -- 172.21.15.77:0/1990386981 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868001970 con 0x7f08700d4860 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.244+0000 7f087697a640 1 -- 172.21.15.77:0/1990386981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08700d4860 msgr2=0x7f08700d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.244+0000 7f087697a640 1 --2- 172.21.15.77:0/1990386981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08700d4860 0x7f08700d2020 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f08640097d0 tx=0x7f086402b800 comp rx=0 tx=0).stop 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.245+0000 7f087697a640 1 -- 172.21.15.77:0/1990386981 shutdown_connections 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.245+0000 7f087697a640 1 --2- 172.21.15.77:0/1990386981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08700d4860 0x7f08700d2020 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.245+0000 7f087697a640 1 -- 172.21.15.77:0/1990386981 >> 172.21.15.77:0/1990386981 conn(0x7f08700d3d40 msgr2=0x7f08700d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.245+0000 7f087697a640 1 -- 172.21.15.77:0/1990386981 shutdown_connections 2024-03-06T07:24:42.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.245+0000 7f087697a640 1 -- 172.21.15.77:0/1990386981 wait complete. 2024-03-06T07:24:42.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.246+0000 7f087697a640 1 Processor -- start 2024-03-06T07:24:42.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.246+0000 7f087697a640 1 -- start start 2024-03-06T07:24:42.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f087697a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08700b0e30 0x7f08700c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f087697a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08700d4860 0x7f0870156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f087697a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08701614d0 0x7f08700ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f087697a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08700b0910 con 0x7f08700b0e30 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f087697a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08700cfc60 con 0x7f08700d4860 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f087697a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08700d2560 con 0x7f08701614d0 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08700b0e30 0x7f08700c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08700b0e30 0x7f08700c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44154/0 (socket says 172.21.15.77:44154) 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 -- 172.21.15.77:0/3253977298 learned_addr learned my addr 172.21.15.77:0/3253977298 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086f7fe640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08700d4860 0x7f0870156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f0874f17640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08701614d0 0x7f08700ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 -- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08701614d0 msgr2=0x7f08700ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08701614d0 0x7f08700ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 -- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08700d4860 msgr2=0x7f0870156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08700d4860 0x7f0870156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 -- 172.21.15.77:0/3253977298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08700ca6a0 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f0874f17640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08701614d0 0x7f08700ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086ffff640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08700b0e30 0x7f08700c88c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f086000c4d0 tx=0x7f086000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086cff9640 1 -- 172.21.15.77:0/3253977298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08600052d0 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086cff9640 1 -- 172.21.15.77:0/3253977298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0860005490 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08700c8ea0 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f086cff9640 1 -- 172.21.15.77:0/3253977298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860012870 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.247+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08700c9460 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.248+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0870068ff0 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.249+0000 7f086cff9640 1 -- 172.21.15.77:0/3253977298 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0860012a10 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.249+0000 7f086cff9640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f085c06b4d0 0x7f085c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.249+0000 7f086cff9640 1 -- 172.21.15.77:0/3253977298 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0860089e60 con 0x7f08700b0e30 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.249+0000 7f086f7fe640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f085c06b4d0 0x7f085c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.250+0000 7f086f7fe640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f085c06b4d0 0x7f085c06d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0864034000 tx=0x7f0864008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.254+0000 7f086cff9640 1 -- 172.21.15.77:0/3253977298 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0860058720 con 0x7f08700b0e30 2024-03-06T07:24:42.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.450+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08700d41c0 con 0x7f08700b0e30 2024-03-06T07:24:42.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.450+0000 7f086cff9640 1 -- 172.21.15.77:0/3253977298 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f086005c7c0 con 0x7f08700b0e30 2024-03-06T07:24:42.452 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:24:42.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f085c06b4d0 msgr2=0x7f085c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f085c06b4d0 0x7f085c06d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0864034000 tx=0x7f0864008030 comp rx=0 tx=0).stop 2024-03-06T07:24:42.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08700b0e30 msgr2=0x7f08700c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08700b0e30 0x7f08700c88c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f086000c4d0 tx=0x7f086000c990 comp rx=0 tx=0).stop 2024-03-06T07:24:42.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 shutdown_connections 2024-03-06T07:24:42.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f08701614d0 0x7f08700ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f08700d4860 0x7f0870156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f085c06b4d0 0x7f085c06d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 --2- 172.21.15.77:0/3253977298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08700b0e30 0x7f08700c88c0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 >> 172.21.15.77:0/3253977298 conn(0x7f08700d3d40 msgr2=0x7f087006b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 shutdown_connections 2024-03-06T07:24:42.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.452+0000 7f087697a640 1 -- 172.21.15.77:0/3253977298 wait complete. 2024-03-06T07:24:42.468 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.570+0000 7f7de69ac640 1 Processor -- start 2024-03-06T07:24:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.571+0000 7f7de69ac640 1 -- start start 2024-03-06T07:24:42.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.571+0000 7f7de69ac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7de00d0af0 0x7f7de00cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.571+0000 7f7de69ac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de00cd4b0 con 0x7f7de015c050 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.571+0000 7f7de69ac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de00cd790 con 0x7f7de006fd10 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.571+0000 7f7de69ac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de00cda70 con 0x7f7de00d0af0 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.571+0000 7f7ddffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7de015c050 0x7f7de006f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41238/0 (socket says 172.21.15.77:41238) 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddffff640 1 -- 172.21.15.77:0/4065605327 learned_addr learned my addr 172.21.15.77:0/4065605327 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7de4f49640 1 --2- 172.21.15.77:0/4065605327 >> v2:172.21.15.158:3300/0 conn(0x7f7de00d0af0 0x7f7de00cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020255292 0 0) 0x7f7de00cd4b0 con 0x7f7de015c050 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dcc003520 con 0x7f7de015c050 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152957565 0 0) 0x7f7de00cd790 con 0x7f7de006fd10 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de00cd4b0 con 0x7f7de006fd10 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383808114 0 0) 0x7f7dcc003520 con 0x7f7de015c050 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 >> v2:172.21.15.158:3300/0 conn(0x7f7de00d0af0 msgr2=0x7f7de00cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 --2- 172.21.15.77:0/4065605327 >> v2:172.21.15.158:3300/0 conn(0x7f7de00d0af0 0x7f7de00cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 >> v1:172.21.15.158:6789/0 conn(0x7f7de006fd10 legacy=0x7f7de00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de00cdd50 con 0x7f7de015c050 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f7dd0001670 con 0x7f7de015c050 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 >> v1:172.21.15.77:6789/0 conn(0x7f7de015c050 legacy=0x7f7de006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc003b40 0x7f7dcc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7dcc004800 0x7f7dcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00cda70 con 0x7f7dcc0051c0 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc00c0f0 con 0x7f7dcc003b40 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc00c2f0 con 0x7f7dcc004800 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7de4f49640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7ddffff640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc003b40 0x7f7dcc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7dd7fff640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7dcc004800 0x7f7dcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7de4f49640 1 -- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7dcc004800 msgr2=0x7f7dcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7de4f49640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7dcc004800 0x7f7dcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.572+0000 7f7de4f49640 1 -- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc003b40 msgr2=0x7f7dcc003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.573+0000 7f7de4f49640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc003b40 0x7f7dcc003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.573+0000 7f7de4f49640 1 -- 172.21.15.77:0/4065605327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc003520 con 0x7f7dcc0051c0 2024-03-06T07:24:42.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.573+0000 7f7dd7fff640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7dcc004800 0x7f7dcc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:42.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de4f49640 1 -- 172.21.15.77:0/4065605327 reap_dead start 2024-03-06T07:24:42.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de4f49640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009f90 tx=0x7f7dcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3740ebc23351ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8039070 con 0x7f7dcc0051c0 2024-03-06T07:24:42.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd802cc20 con 0x7f7dcc0051c0 2024-03-06T07:24:42.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7ddf7fe640 1 -- 172.21.15.77:0/4065605327 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8009c10 con 0x7f7dcc0051c0 2024-03-06T07:24:42.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de69ac640 1 -- 172.21.15.77:0/4065605327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dcc0051c0 msgr2=0x7f7dcc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de69ac640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009f90 tx=0x7f7dcc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:42.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de69ac640 1 -- 172.21.15.77:0/4065605327 shutdown_connections 2024-03-06T07:24:42.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de69ac640 1 --2- 172.21.15.77:0/4065605327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dcc0051c0 0x7f7dcc00b9c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de69ac640 1 -- 172.21.15.77:0/4065605327 >> 172.21.15.77:0/4065605327 conn(0x7f7de00b3800 msgr2=0x7f7de00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de69ac640 1 -- 172.21.15.77:0/4065605327 shutdown_connections 2024-03-06T07:24:42.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.574+0000 7f7de69ac640 1 -- 172.21.15.77:0/4065605327 wait complete. 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.575+0000 7f7de69ac640 1 Processor -- start 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7de69ac640 1 -- start start 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7de69ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc0051c0 0x7f7de00c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7de69ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de00b6390 0x7f7de015a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7de69ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7de00b6810 0x7f7de015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7de69ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de006f990 con 0x7f7de00b6390 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7de69ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de006f810 con 0x7f7dcc0051c0 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7de69ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de006fb10 con 0x7f7de00b6810 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7dd7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de00b6390 0x7f7de015a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7dd7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de00b6390 0x7f7de015a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44168/0 (socket says 172.21.15.77:44168) 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7dd7fff640 1 -- 172.21.15.77:0/3632680314 learned_addr learned my addr 172.21.15.77:0/3632680314 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7ddffff640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc0051c0 0x7f7de00c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.576+0000 7f7de4f49640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7de00b6810 0x7f7de015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7dd7fff640 1 -- 172.21.15.77:0/3632680314 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7de00b6810 msgr2=0x7f7de015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7dd7fff640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7de00b6810 0x7f7de015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7dd7fff640 1 -- 172.21.15.77:0/3632680314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc0051c0 msgr2=0x7f7de00c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7dd7fff640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc0051c0 0x7f7de00c6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7dd7fff640 1 -- 172.21.15.77:0/3632680314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de009da30 con 0x7f7de00b6390 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7dd7fff640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de00b6390 0x7f7de015a6a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800e580 tx=0x7f7dc800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7ddd7fa640 1 -- 172.21.15.77:0/3632680314 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80052e0 con 0x7f7de00b6390 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7ddd7fa640 1 -- 172.21.15.77:0/3632680314 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc8007020 con 0x7f7de00b6390 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7ddd7fa640 1 -- 172.21.15.77:0/3632680314 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80071f0 con 0x7f7de00b6390 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de009dd10 con 0x7f7de00b6390 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.577+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de009e2d0 con 0x7f7de00b6390 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.578+0000 7f7ddd7fa640 1 -- 172.21.15.77:0/3632680314 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dc801a020 con 0x7f7de00b6390 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.578+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de009fe60 con 0x7f7de00b6390 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.579+0000 7f7ddd7fa640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7dc006b500 0x7f7dc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.579+0000 7f7ddd7fa640 1 -- 172.21.15.77:0/3632680314 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dc8089810 con 0x7f7de00b6390 2024-03-06T07:24:42.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.579+0000 7f7ddffff640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7dc006b500 0x7f7dc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.579+0000 7f7ddffff640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7dc006b500 0x7f7dc006d9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0000f60 tx=0x7f7dd0000ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.583+0000 7f7ddd7fa640 1 -- 172.21.15.77:0/3632680314 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dc80586a0 con 0x7f7de00b6390 2024-03-06T07:24:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.760+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7de0068ff0 con 0x7f7de00b6390 2024-03-06T07:24:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.760+0000 7f7ddd7fa640 1 -- 172.21.15.77:0/3632680314 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7dc8057fa0 con 0x7f7de00b6390 2024-03-06T07:24:42.762 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:24:42.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7dc006b500 msgr2=0x7f7dc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7dc006b500 0x7f7dc006d9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0000f60 tx=0x7f7dd0000ef0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de00b6390 msgr2=0x7f7de015a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de00b6390 0x7f7de015a6a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800e580 tx=0x7f7dc800ea40 comp rx=0 tx=0).stop 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 shutdown_connections 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7dc006b500 0x7f7dc006d9c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7de00b6390 0x7f7de015a6a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7de00b6810 0x7f7de015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 --2- 172.21.15.77:0/3632680314 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7dcc0051c0 0x7f7de00c6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 >> 172.21.15.77:0/3632680314 conn(0x7f7de00b3800 msgr2=0x7f7de00cc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 shutdown_connections 2024-03-06T07:24:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.762+0000 7f7de69ac640 1 -- 172.21.15.77:0/3632680314 wait complete. 2024-03-06T07:24:42.775 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.874+0000 7fa7f0688640 1 Processor -- start 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.874+0000 7fa7f0688640 1 -- start start 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.875+0000 7fa7f0688640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7e8162400 0x7fa7e80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.875+0000 7fa7f0688640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e80d22c0 con 0x7fa7e8162970 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.875+0000 7fa7f0688640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e80d25a0 con 0x7fa7e80b4460 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.875+0000 7fa7f0688640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e8161b00 con 0x7fa7e8162400 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.875+0000 7fa7eec25640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7e8162400 0x7fa7e80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.875+0000 7fa7edc23640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa7e8162970 0x7fa7e8161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41254/0 (socket says 172.21.15.77:41254) 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.875+0000 7fa7edc23640 1 -- 172.21.15.77:0/1240003934 learned_addr learned my addr 172.21.15.77:0/1240003934 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.876+0000 7fa7eec25640 1 -- 172.21.15.77:0/1240003934 >> v1:172.21.15.158:6789/0 conn(0x7fa7e80b4460 legacy=0x7fa7e80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.876+0000 7fa7eec25640 1 -- 172.21.15.77:0/1240003934 >> v1:172.21.15.77:6789/0 conn(0x7fa7e8162970 legacy=0x7fa7e8161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.876+0000 7fa7eec25640 1 -- 172.21.15.77:0/1240003934 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80d2880 con 0x7fa7e8162400 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.876+0000 7fa7eec25640 1 --2- 172.21.15.77:0/1240003934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162400 0x7fa7e80d4800 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009d90 tx=0x7fa7e002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1024214f277d600b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.876+0000 7fa7ed422640 1 -- 172.21.15.77:0/1240003934 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0039070 con 0x7fa7e8162400 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7ed422640 1 -- 172.21.15.77:0/1240003934 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e002cd00 con 0x7fa7e8162400 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7ed422640 1 -- 172.21.15.77:0/1240003934 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0034660 con 0x7fa7e8162400 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7f0688640 1 -- 172.21.15.77:0/1240003934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162400 msgr2=0x7fa7e80d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7f0688640 1 --2- 172.21.15.77:0/1240003934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162400 0x7fa7e80d4800 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009d90 tx=0x7fa7e002bf50 comp rx=0 tx=0).stop 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7f0688640 1 -- 172.21.15.77:0/1240003934 shutdown_connections 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7f0688640 1 --2- 172.21.15.77:0/1240003934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162400 0x7fa7e80d4800 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7f0688640 1 -- 172.21.15.77:0/1240003934 >> 172.21.15.77:0/1240003934 conn(0x7fa7e80d3ce0 msgr2=0x7fa7e80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7f0688640 1 -- 172.21.15.77:0/1240003934 shutdown_connections 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.877+0000 7fa7f0688640 1 -- 172.21.15.77:0/1240003934 wait complete. 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.878+0000 7fa7f0688640 1 Processor -- start 2024-03-06T07:24:42.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.878+0000 7fa7f0688640 1 -- start start 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7f0688640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7e80b4460 0x7fa7e8155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7f0688640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7e8162400 0x7fa7e80cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7f0688640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162970 0x7fa7e815ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7f0688640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80ce950 con 0x7fa7e8162970 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7f0688640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80cecc0 con 0x7fa7e80b4460 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7f0688640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80b4840 con 0x7fa7e8162400 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7eec25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162970 0x7fa7e815ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7eec25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162970 0x7fa7e815ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44184/0 (socket says 172.21.15.77:44184) 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7eec25640 1 -- 172.21.15.77:0/183381698 learned_addr learned my addr 172.21.15.77:0/183381698 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7edc23640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7e8162400 0x7fa7e80cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7ee424640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7e80b4460 0x7fa7e8155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7eec25640 1 -- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7e8162400 msgr2=0x7fa7e80cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7eec25640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7e8162400 0x7fa7e80cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7eec25640 1 -- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7e80b4460 msgr2=0x7fa7e8155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7eec25640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7e80b4460 0x7fa7e8155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7eec25640 1 -- 172.21.15.77:0/183381698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e8155980 con 0x7fa7e8162970 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7ee424640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7e80b4460 0x7fa7e8155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.879+0000 7fa7edc23640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7e8162400 0x7fa7e80cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.880+0000 7fa7eec25640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162970 0x7fa7e815ece0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009d90 tx=0x7fa7e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.880+0000 7fa7deffd640 1 -- 172.21.15.77:0/183381698 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0039050 con 0x7fa7e8162970 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.880+0000 7fa7deffd640 1 -- 172.21.15.77:0/183381698 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e0045070 con 0x7fa7e8162970 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.880+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e80b8960 con 0x7fa7e8162970 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.880+0000 7fa7deffd640 1 -- 172.21.15.77:0/183381698 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e003e830 con 0x7fa7e8162970 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.880+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e80cadb0 con 0x7fa7e8162970 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.881+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e804f330 con 0x7fa7e8162970 2024-03-06T07:24:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.881+0000 7fa7deffd640 1 -- 172.21.15.77:0/183381698 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7e003e9d0 con 0x7fa7e8162970 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.881+0000 7fa7deffd640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.881+0000 7fa7deffd640 1 -- 172.21.15.77:0/183381698 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7e00acf60 con 0x7fa7e8162970 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.882+0000 7fa7ee424640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.882+0000 7fa7ee424640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80015e0 tx=0x7fa7d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:42.886+0000 7fa7deffd640 1 -- 172.21.15.77:0/183381698 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7e007bf00 con 0x7fa7e8162970 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.098+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa7e806b180 con 0x7fa7e8162970 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.099+0000 7fa7deffd640 1 -- 172.21.15.77:0/183381698 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa7e007b800 con 0x7fa7e8162970 2024-03-06T07:24:43.101 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:24:43.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7cc06b4d0 msgr2=0x7fa7cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80015e0 tx=0x7fa7d8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:43.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162970 msgr2=0x7fa7e815ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162970 0x7fa7e815ece0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009d90 tx=0x7fa7e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:43.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 shutdown_connections 2024-03-06T07:24:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e8162970 0x7fa7e815ece0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7e8162400 0x7fa7e80cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 --2- 172.21.15.77:0/183381698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7e80b4460 0x7fa7e8155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 >> 172.21.15.77:0/183381698 conn(0x7fa7e80d3ce0 msgr2=0x7fa7e80cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 shutdown_connections 2024-03-06T07:24:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.101+0000 7fa7f0688640 1 -- 172.21.15.77:0/183381698 wait complete. 2024-03-06T07:24:43.113 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:24:43.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:43.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:43.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[59547]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:43.141 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:24:43.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:43.141 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:43.314 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:24:43.314 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:43.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:43.332 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:24:43.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:43.333 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:24:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:43.390 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:24:43.390 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:43.390 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:43.406 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:43.406 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:24:43.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:43.406 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:24:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:43.461 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:24:43.462 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:24:43.462 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:43.479 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:43.480 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:24:43.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:43.481 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:24:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:43.536 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:24:43.536 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.635+0000 7ff8360e1640 1 Processor -- start 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.635+0000 7ff8360e1640 1 -- start start 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff8360e1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff830072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff8360e1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff830162030 0x7ff8301624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff8360e1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83015eb40 con 0x7ff83015e3a0 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff8360e1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff830161ba0 con 0x7ff830162030 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff8360e1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83015ee20 con 0x7ff830158600 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff82f7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff830072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff82ffff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff830162030 0x7ff8301624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff82effd640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff83015e3a0 0x7ff83006d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:36060/0 (socket says 172.21.15.77:36060) 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.636+0000 7ff82effd640 1 -- 172.21.15.77:0/3070720056 learned_addr learned my addr 172.21.15.77:0/3070720056 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82ffff640 1 -- 172.21.15.77:0/3070720056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 msgr2=0x7ff830072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82ffff640 1 --2- 172.21.15.77:0/3070720056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff830072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82ffff640 1 -- 172.21.15.77:0/3070720056 >> v1:172.21.15.158:6789/0 conn(0x7ff83015e3a0 legacy=0x7ff83006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82e7fc640 1 -- 172.21.15.77:0/3070720056 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66401225 0 0) 0x7ff83015eb40 con 0x7ff83015e3a0 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82ffff640 1 -- 172.21.15.77:0/3070720056 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83015f100 con 0x7ff830162030 2024-03-06T07:24:43.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82f7fe640 1 --2- 172.21.15.77:0/3070720056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff830072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:43.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82ffff640 1 --2- 172.21.15.77:0/3070720056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 0x7ff8301624c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff828009d90 tx=0x7ff82802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614229c8f8835976 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82e7fc640 1 -- 172.21.15.77:0/3070720056 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828039070 con 0x7ff830162030 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.637+0000 7ff82e7fc640 1 -- 172.21.15.77:0/3070720056 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82802cbc0 con 0x7ff830162030 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff82e7fc640 1 -- 172.21.15.77:0/3070720056 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828034440 con 0x7ff830162030 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff8360e1640 1 -- 172.21.15.77:0/3070720056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 msgr2=0x7ff8301624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3070720056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 0x7ff8301624c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff828009d90 tx=0x7ff82802be10 comp rx=0 tx=0).stop 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff8360e1640 1 -- 172.21.15.77:0/3070720056 shutdown_connections 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3070720056 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 0x7ff8301624c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3070720056 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff830072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff8360e1640 1 -- 172.21.15.77:0/3070720056 >> 172.21.15.77:0/3070720056 conn(0x7ff83006c0c0 msgr2=0x7ff83006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff8360e1640 1 -- 172.21.15.77:0/3070720056 shutdown_connections 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.638+0000 7ff8360e1640 1 -- 172.21.15.77:0/3070720056 wait complete. 2024-03-06T07:24:43.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.639+0000 7ff8360e1640 1 Processor -- start 2024-03-06T07:24:43.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.639+0000 7ff8360e1640 1 -- start start 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff8360e1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff83014d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff8360e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff83015e3a0 0x7ff830153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff8360e1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 0x7ff8301544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff8360e1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83006e080 con 0x7ff83015e3a0 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff8360e1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83006afb0 con 0x7ff830162030 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff8360e1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff830153b20 con 0x7ff830158600 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff83015e3a0 0x7ff830153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff83015e3a0 0x7ff830153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44194/0 (socket says 172.21.15.77:44194) 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82effd640 1 -- 172.21.15.77:0/3712091389 learned_addr learned my addr 172.21.15.77:0/3712091389 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82ffff640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 0x7ff8301544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82f7fe640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff83014d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82effd640 1 -- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 msgr2=0x7ff83014d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82effd640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff83014d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82effd640 1 -- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 msgr2=0x7ff8301544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82effd640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 0x7ff8301544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82effd640 1 -- 172.21.15.77:0/3712091389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83014df00 con 0x7ff83015e3a0 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.640+0000 7ff82ffff640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 0x7ff8301544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.641+0000 7ff82effd640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff83015e3a0 0x7ff830153f90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff82400c4d0 tx=0x7ff82400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.641+0000 7ff817fff640 1 -- 172.21.15.77:0/3712091389 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824018070 con 0x7ff83015e3a0 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.641+0000 7ff817fff640 1 -- 172.21.15.77:0/3712091389 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff824007500 con 0x7ff83015e3a0 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.641+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8301e87a0 con 0x7ff83015e3a0 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.641+0000 7ff817fff640 1 -- 172.21.15.77:0/3712091389 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824004a20 con 0x7ff83015e3a0 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.641+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8301e8d60 con 0x7ff83015e3a0 2024-03-06T07:24:43.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.642+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83004f330 con 0x7ff83015e3a0 2024-03-06T07:24:43.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.643+0000 7ff817fff640 1 -- 172.21.15.77:0/3712091389 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff82401b070 con 0x7ff83015e3a0 2024-03-06T07:24:43.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.643+0000 7ff817fff640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff81c06b4d0 0x7ff81c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.643+0000 7ff817fff640 1 -- 172.21.15.77:0/3712091389 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff824089480 con 0x7ff83015e3a0 2024-03-06T07:24:43.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.643+0000 7ff82f7fe640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff81c06b4d0 0x7ff81c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.644+0000 7ff82f7fe640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff81c06b4d0 0x7ff81c06d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff820009560 tx=0x7ff820008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.647+0000 7ff817fff640 1 -- 172.21.15.77:0/3712091389 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff824058390 con 0x7ff83015e3a0 2024-03-06T07:24:43.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.875+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8300d06f0 con 0x7ff83015e3a0 2024-03-06T07:24:43.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.875+0000 7ff817fff640 1 -- 172.21.15.77:0/3712091389 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7ff824057c90 con 0x7ff83015e3a0 2024-03-06T07:24:43.877 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:43.877 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.760628+0000","modified":"2024-03-06T07:24:12.764047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/251375557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":251375557},{"type":"v1","addr":"172.21.15.77:6837","nonce":251375557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-03-06T07:24:43.878 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.877+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff81c06b4d0 msgr2=0x7ff81c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.877+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff81c06b4d0 0x7ff81c06d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff820009560 tx=0x7ff820008040 comp rx=0 tx=0).stop 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.877+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff83015e3a0 msgr2=0x7ff830153f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.877+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff83015e3a0 0x7ff830153f90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff82400c4d0 tx=0x7ff82400c990 comp rx=0 tx=0).stop 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.877+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 shutdown_connections 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.878+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff81c06b4d0 0x7ff81c06d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.878+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff83015e3a0 0x7ff830153f90 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.878+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff830162030 0x7ff8301544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.878+0000 7ff8360e1640 1 --2- 172.21.15.77:0/3712091389 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff830158600 0x7ff83014d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.878+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 >> 172.21.15.77:0/3712091389 conn(0x7ff83006c0c0 msgr2=0x7ff8300b0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.878+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 shutdown_connections 2024-03-06T07:24:43.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.878+0000 7ff8360e1640 1 -- 172.21.15.77:0/3712091389 wait complete. 2024-03-06T07:24:43.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:43.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.989+0000 7fe47e38a640 1 Processor -- start 2024-03-06T07:24:43.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.990+0000 7fe47e38a640 1 -- start start 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe47e38a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780d22b0 0x7fe4780d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe47e38a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe4780b41f0 0x7fe4780b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe47e38a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4780b3430 con 0x7fe47815dfe0 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe47e38a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4780d2c50 con 0x7fe4780b41f0 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe47e38a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780b31e0 con 0x7fe4780d22b0 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe477fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe47815dfe0 0x7fe4780cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41270/0 (socket says 172.21.15.77:41270) 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe477fff640 1 -- 172.21.15.77:0/3472641403 learned_addr learned my addr 172.21.15.77:0/3472641403 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939475078 0 0) 0x7fe4780b3430 con 0x7fe47815dfe0 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe4777fe640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780d22b0 0x7fe4780d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe460003520 con 0x7fe47815dfe0 2024-03-06T07:24:43.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe47c927640 1 --2- 172.21.15.77:0/3472641403 >> v2:172.21.15.158:3300/0 conn(0x7fe4780b41f0 0x7fe4780b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 444868251 0 0) 0x7fe460003520 con 0x7fe47815dfe0 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780d22b0 msgr2=0x7fe4780d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780d22b0 0x7fe4780d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 >> v2:172.21.15.158:3300/0 conn(0x7fe4780b41f0 msgr2=0x7fe4780b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 --2- 172.21.15.77:0/3472641403 >> v2:172.21.15.158:3300/0 conn(0x7fe4780b41f0 0x7fe4780b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780b3710 con 0x7fe47815dfe0 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.991+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fe468001670 con 0x7fe47815dfe0 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 >> v1:172.21.15.77:6789/0 conn(0x7fe47815dfe0 legacy=0x7fe4780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe476ffd640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4600039b0 0x7fe460003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe476ffd640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe460004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe476ffd640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780d22b0 0x7fe4600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780b31e0 con 0x7fe460004800 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46000b9c0 con 0x7fe4600039b0 2024-03-06T07:24:43.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46000bb40 con 0x7fe4780d22b0 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe4777fe640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe460004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe477fff640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4600039b0 0x7fe460003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe47c927640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780d22b0 0x7fe4600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe4777fe640 1 -- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780d22b0 msgr2=0x7fe4600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe4777fe640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780d22b0 0x7fe4600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe4777fe640 1 -- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4600039b0 msgr2=0x7fe460003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe4777fe640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4600039b0 0x7fe460003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.992+0000 7fe4777fe640 1 -- 172.21.15.77:0/3472641403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460003520 con 0x7fe460004800 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.993+0000 7fe4777fe640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe460004c80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009ac0 tx=0x7fe460005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71668ab040a0b5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.993+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c039070 con 0x7fe460004800 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.993+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c02c6b0 con 0x7fe460004800 2024-03-06T07:24:43.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.993+0000 7fe476ffd640 1 -- 172.21.15.77:0/3472641403 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c009690 con 0x7fe460004800 2024-03-06T07:24:44.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.994+0000 7fe47e38a640 1 -- 172.21.15.77:0/3472641403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 msgr2=0x7fe460004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.994+0000 7fe47e38a640 1 --2- 172.21.15.77:0/3472641403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe460004c80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009ac0 tx=0x7fe460005830 comp rx=0 tx=0).stop 2024-03-06T07:24:44.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.994+0000 7fe47c927640 1 -- 172.21.15.77:0/3472641403 reap_dead start 2024-03-06T07:24:44.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.994+0000 7fe47e38a640 1 -- 172.21.15.77:0/3472641403 shutdown_connections 2024-03-06T07:24:44.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.994+0000 7fe47e38a640 1 -- 172.21.15.77:0/3472641403 >> 172.21.15.77:0/3472641403 conn(0x7fe4780d3cd0 msgr2=0x7fe4780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.994+0000 7fe47e38a640 1 -- 172.21.15.77:0/3472641403 shutdown_connections 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.994+0000 7fe47e38a640 1 -- 172.21.15.77:0/3472641403 wait complete. 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.995+0000 7fe47e38a640 1 Processor -- start 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.995+0000 7fe47e38a640 1 -- start start 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe47e38a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe47815f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe47e38a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780b9600 0x7fe4780cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe47e38a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4780ccac0 0x7fe4780ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe47e38a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780cfa00 con 0x7fe460004800 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe47e38a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780b3d60 con 0x7fe4780ccac0 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe47e38a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780d1c70 con 0x7fe4780b9600 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe47815f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe47815f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44214/0 (socket says 172.21.15.77:44214) 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 -- 172.21.15.77:0/1295260601 learned_addr learned my addr 172.21.15.77:0/1295260601 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 -- 172.21.15.77:0/1295260601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780b9600 msgr2=0x7fe4780cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780b9600 0x7fe4780cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 -- 172.21.15.77:0/1295260601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4780ccac0 msgr2=0x7fe4780ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4780ccac0 0x7fe4780ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 -- 172.21.15.77:0/1295260601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780cd480 con 0x7fe460004800 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe477fff640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe47815f410 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe468007340 tx=0x7fe468007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe474ff9640 1 -- 172.21.15.77:0/1295260601 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46800f8e0 con 0x7fe460004800 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe474ff9640 1 -- 172.21.15.77:0/1295260601 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468018070 con 0x7fe460004800 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe478160e20 con 0x7fe460004800 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe474ff9640 1 -- 172.21.15.77:0/1295260601 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46800fce0 con 0x7fe460004800 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.997+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe478160340 con 0x7fe460004800 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.999+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47804f330 con 0x7fe460004800 2024-03-06T07:24:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.999+0000 7fe474ff9640 1 -- 172.21.15.77:0/1295260601 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe46801c410 con 0x7fe460004800 2024-03-06T07:24:44.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.999+0000 7fe474ff9640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46406b500 0x7fe46406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.999+0000 7fe474ff9640 1 -- 172.21.15.77:0/1295260601 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe468089d60 con 0x7fe460004800 2024-03-06T07:24:44.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:43.999+0000 7fe4777fe640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46406b500 0x7fe46406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.000+0000 7fe4777fe640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46406b500 0x7fe46406d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe46c036000 tx=0x7fe46c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.004+0000 7fe474ff9640 1 -- 172.21.15.77:0/1295260601 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe468058be0 con 0x7fe460004800 2024-03-06T07:24:44.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.189+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4780d41f0 con 0x7fe460004800 2024-03-06T07:24:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.190+0000 7fe474ff9640 1 -- 172.21.15.77:0/1295260601 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fe4680584e0 con 0x7fe460004800 2024-03-06T07:24:44.192 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:44.192 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:12.416501+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46406b500 msgr2=0x7fe46406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46406b500 0x7fe46406d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe46c036000 tx=0x7fe46c02e000 comp rx=0 tx=0).stop 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 msgr2=0x7fe47815f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe47815f410 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe468007340 tx=0x7fe468007800 comp rx=0 tx=0).stop 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 shutdown_connections 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe4780ccac0 0x7fe4780ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe4780b9600 0x7fe4780cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe46406b500 0x7fe46406d9c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 --2- 172.21.15.77:0/1295260601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe460004800 0x7fe47815f410 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 >> 172.21.15.77:0/1295260601 conn(0x7fe4780d3cd0 msgr2=0x7fe4780d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 shutdown_connections 2024-03-06T07:24:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.192+0000 7fe47e38a640 1 -- 172.21.15.77:0/1295260601 wait complete. 2024-03-06T07:24:44.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:08.975524+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.729967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:44.207 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:44.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.310+0000 7f9e21102640 1 Processor -- start 2024-03-06T07:24:44.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.310+0000 7f9e21102640 1 -- start start 2024-03-06T07:24:44.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e21102640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e1c1614d0 0x7f9e1c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e21102640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e1c0d1910 0x7f9e1c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e21102640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e1c0d22b0 0x7f9e1c0d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e21102640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c0d2740 con 0x7f9e1c0d1910 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e21102640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c1618b0 con 0x7f9e1c0d22b0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e21102640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0cf400 con 0x7f9e1c1614d0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e13fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e1c0d1910 0x7f9e1c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e13fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e1c0d1910 0x7f9e1c0d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44228/0 (socket says 172.21.15.77:44228) 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e13fff640 1 -- 172.21.15.77:0/1085405158 learned_addr learned my addr 172.21.15.77:0/1085405158 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.311+0000 7f9e13fff640 1 -- 172.21.15.77:0/1085405158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e1c1614d0 msgr2=0x7f9e1c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.312+0000 7f9e13fff640 1 --2- 172.21.15.77:0/1085405158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e1c1614d0 0x7f9e1c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.312+0000 7f9e13fff640 1 -- 172.21.15.77:0/1085405158 >> v2:172.21.15.158:3300/0 conn(0x7f9e1c0d22b0 msgr2=0x7f9e1c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.312+0000 7f9e13fff640 1 --2- 172.21.15.77:0/1085405158 >> v2:172.21.15.158:3300/0 conn(0x7f9e1c0d22b0 0x7f9e1c0d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.312+0000 7f9e13fff640 1 -- 172.21.15.77:0/1085405158 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c0cf6a0 con 0x7f9e1c0d1910 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.312+0000 7f9e13fff640 1 --2- 172.21.15.77:0/1085405158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0d1d70 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0b57d0 tx=0x7f9e0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810b269201f82c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.312+0000 7f9e1a59c640 1 -- 172.21.15.77:0/1085405158 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04035020 con 0x7f9e1c0d1910 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.312+0000 7f9e1a59c640 1 -- 172.21.15.77:0/1085405158 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e04031870 con 0x7f9e1c0d1910 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.312+0000 7f9e1a59c640 1 -- 172.21.15.77:0/1085405158 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04031c40 con 0x7f9e1c0d1910 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 -- 172.21.15.77:0/1085405158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 msgr2=0x7f9e1c0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 --2- 172.21.15.77:0/1085405158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0d1d70 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0b57d0 tx=0x7f9e0402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 -- 172.21.15.77:0/1085405158 shutdown_connections 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 --2- 172.21.15.77:0/1085405158 >> v2:172.21.15.158:3300/0 conn(0x7f9e1c0d22b0 0x7f9e1c0d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 --2- 172.21.15.77:0/1085405158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0d1d70 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 --2- 172.21.15.77:0/1085405158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e1c1614d0 0x7f9e1c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 -- 172.21.15.77:0/1085405158 >> 172.21.15.77:0/1085405158 conn(0x7f9e1c0d3d40 msgr2=0x7f9e1c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 -- 172.21.15.77:0/1085405158 shutdown_connections 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.313+0000 7f9e21102640 1 -- 172.21.15.77:0/1085405158 wait complete. 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.314+0000 7f9e21102640 1 Processor -- start 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.314+0000 7f9e21102640 1 -- start start 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e21102640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e21102640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e1c0d22b0 0x7f9e1c151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e21102640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e1c1614d0 0x7f9e1c0b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e21102640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0b0910 con 0x7f9e1c0d1910 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e21102640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0cfc60 con 0x7f9e1c1614d0 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e21102640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0d0be0 con 0x7f9e1c0d22b0 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44238/0 (socket says 172.21.15.77:44238) 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 -- 172.21.15.77:0/3937419196 learned_addr learned my addr 172.21.15.77:0/3937419196 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 -- 172.21.15.77:0/3937419196 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e1c0d22b0 msgr2=0x7f9e1c151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e1c0d22b0 0x7f9e1c151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 -- 172.21.15.77:0/3937419196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e1c1614d0 msgr2=0x7f9e1c0b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e1c1614d0 0x7f9e1c0b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 -- 172.21.15.77:0/3937419196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c151880 con 0x7f9e1c0d1910 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.315+0000 7f9e1ad9d640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0cb0b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00b440 tx=0x7f9e0c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.316+0000 7f9e137fe640 1 -- 172.21.15.77:0/3937419196 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c015020 con 0x7f9e1c0d1910 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.316+0000 7f9e137fe640 1 -- 172.21.15.77:0/3937419196 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c011890 con 0x7f9e1c0d1910 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.316+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c0cddb0 con 0x7f9e1c0d1910 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.316+0000 7f9e137fe640 1 -- 172.21.15.77:0/3937419196 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c011c80 con 0x7f9e1c0d1910 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.316+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c0b7180 con 0x7f9e1c0d1910 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.317+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c068ff0 con 0x7f9e1c0d1910 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.317+0000 7f9e137fe640 1 -- 172.21.15.77:0/3937419196 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e0c018020 con 0x7f9e1c0d1910 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.318+0000 7f9e137fe640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9dfc06b500 0x7f9dfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.318+0000 7f9e13fff640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9dfc06b500 0x7f9dfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.318+0000 7f9e137fe640 1 -- 172.21.15.77:0/3937419196 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e0c08a2c0 con 0x7f9e1c0d1910 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.318+0000 7f9e13fff640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9dfc06b500 0x7f9dfc06d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9e04036000 tx=0x7f9e04030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.322+0000 7f9e137fe640 1 -- 172.21.15.77:0/3937419196 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e0c01f020 con 0x7f9e1c0d1910 2024-03-06T07:24:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.500+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e1c0d46c0 con 0x7f9e1c0d1910 2024-03-06T07:24:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.501+0000 7f9e137fe640 1 -- 172.21.15.77:0/3937419196 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e1c151880 con 0x7f9e1c0d1910 2024-03-06T07:24:44.503 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:24:44.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9dfc06b500 msgr2=0x7f9dfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9dfc06b500 0x7f9dfc06d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9e04036000 tx=0x7f9e04030000 comp rx=0 tx=0).stop 2024-03-06T07:24:44.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 msgr2=0x7f9e1c0cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0cb0b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00b440 tx=0x7f9e0c00b900 comp rx=0 tx=0).stop 2024-03-06T07:24:44.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 shutdown_connections 2024-03-06T07:24:44.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e1c1614d0 0x7f9e1c0b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e1c0d22b0 0x7f9e1c151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9dfc06b500 0x7f9dfc06d9c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 --2- 172.21.15.77:0/3937419196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1c0d1910 0x7f9e1c0cb0b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 >> 172.21.15.77:0/3937419196 conn(0x7f9e1c0d3d40 msgr2=0x7f9e1c0c9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 shutdown_connections 2024-03-06T07:24:44.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.503+0000 7f9e21102640 1 -- 172.21.15.77:0/3937419196 wait complete. 2024-03-06T07:24:44.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.619+0000 7f910d15c640 1 Processor -- start 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.619+0000 7f910d15c640 1 -- start start 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910d15c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f910806fd10 0x7f91080cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910d15c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91080d0af0 0x7f91080ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910d15c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91080cd1e0 con 0x7f910815c050 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910d15c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91080cd4b0 con 0x7f91080d0af0 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910d15c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91080cd790 con 0x7f910806fd10 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f9106d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f910815c050 0x7f910806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41282/0 (socket says 172.21.15.77:41282) 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f9106d9d640 1 -- 172.21.15.77:0/533329338 learned_addr learned my addr 172.21.15.77:0/533329338 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910659c640 1 --2- 172.21.15.77:0/533329338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f910806fd10 0x7f91080cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910759e640 1 --2- 172.21.15.77:0/533329338 >> v2:172.21.15.158:3300/0 conn(0x7f91080d0af0 0x7f91080ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f9105d9b640 1 -- 172.21.15.77:0/533329338 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562811520 0 0) 0x7f91080cd1e0 con 0x7f910815c050 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f9105d9b640 1 -- 172.21.15.77:0/533329338 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90f0003520 con 0x7f910815c050 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910759e640 1 -- 172.21.15.77:0/533329338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f910806fd10 msgr2=0x7f91080cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910759e640 1 --2- 172.21.15.77:0/533329338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f910806fd10 0x7f91080cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910759e640 1 -- 172.21.15.77:0/533329338 >> v1:172.21.15.77:6789/0 conn(0x7f910815c050 legacy=0x7f910806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f9105d9b640 1 -- 172.21.15.77:0/533329338 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2839704284 0 0) 0x7f90f0003520 con 0x7f910815c050 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.620+0000 7f910759e640 1 -- 172.21.15.77:0/533329338 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91080cda70 con 0x7f91080d0af0 2024-03-06T07:24:44.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.621+0000 7f910759e640 1 --2- 172.21.15.77:0/533329338 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91080d0af0 0x7f91080ccca0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9100009d90 tx=0x7f910002bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d53586a2057749d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.621+0000 7f9105d9b640 1 -- 172.21.15.77:0/533329338 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100039070 con 0x7f91080d0af0 2024-03-06T07:24:44.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.621+0000 7f9105d9b640 1 -- 172.21.15.77:0/533329338 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910002c810 con 0x7f91080d0af0 2024-03-06T07:24:44.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.621+0000 7f9105d9b640 1 -- 172.21.15.77:0/533329338 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc001670 con 0x7f91080d0af0 2024-03-06T07:24:44.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.622+0000 7f910d15c640 1 -- 172.21.15.77:0/533329338 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91080d0af0 msgr2=0x7f91080ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.622+0000 7f910d15c640 1 --2- 172.21.15.77:0/533329338 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91080d0af0 0x7f91080ccca0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9100009d90 tx=0x7f910002bce0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.622+0000 7f910d15c640 1 -- 172.21.15.77:0/533329338 shutdown_connections 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.622+0000 7f910d15c640 1 --2- 172.21.15.77:0/533329338 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f91080d0af0 0x7f91080ccca0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.622+0000 7f910d15c640 1 --2- 172.21.15.77:0/533329338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f910806fd10 0x7f91080cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.622+0000 7f910d15c640 1 -- 172.21.15.77:0/533329338 >> 172.21.15.77:0/533329338 conn(0x7f91080b3800 msgr2=0x7f91080b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.622+0000 7f910d15c640 1 -- 172.21.15.77:0/533329338 shutdown_connections 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.622+0000 7f910d15c640 1 -- 172.21.15.77:0/533329338 wait complete. 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.623+0000 7f910d15c640 1 Processor -- start 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.623+0000 7f910d15c640 1 -- start start 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910d15c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f910806fd10 0x7f9108159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910d15c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91080d0af0 0x7f91080b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910d15c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f910815c050 0x7f910814f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910d15c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9108165c30 con 0x7f910815c050 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910d15c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9108165ab0 con 0x7f910806fd10 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910d15c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9108165db0 con 0x7f91080d0af0 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f910815c050 0x7f910814f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f910815c050 0x7f910814f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44248/0 (socket says 172.21.15.77:44248) 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910759e640 1 -- 172.21.15.77:0/520154710 learned_addr learned my addr 172.21.15.77:0/520154710 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f9106d9d640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f910806fd10 0x7f9108159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.624+0000 7f910659c640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91080d0af0 0x7f91080b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910759e640 1 -- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91080d0af0 msgr2=0x7f91080b7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910759e640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91080d0af0 0x7f91080b7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910759e640 1 -- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f910806fd10 msgr2=0x7f9108159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910759e640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f910806fd10 0x7f9108159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910759e640 1 -- 172.21.15.77:0/520154710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910806c650 con 0x7f910815c050 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f9106d9d640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f910806fd10 0x7f9108159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910659c640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91080d0af0 0x7f91080b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910759e640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f910815c050 0x7f910814f310 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9100009d90 tx=0x7f9100004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f90ef7fe640 1 -- 172.21.15.77:0/520154710 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100044020 con 0x7f910815c050 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f90ef7fe640 1 -- 172.21.15.77:0/520154710 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9100034ce0 con 0x7f910815c050 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910806c930 con 0x7f910815c050 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f90ef7fe640 1 -- 172.21.15.77:0/520154710 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100039040 con 0x7f910815c050 2024-03-06T07:24:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.625+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910806cef0 con 0x7f910815c050 2024-03-06T07:24:44.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.626+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910809fe60 con 0x7f910815c050 2024-03-06T07:24:44.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.626+0000 7f90ef7fe640 1 -- 172.21.15.77:0/520154710 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9100047020 con 0x7f910815c050 2024-03-06T07:24:44.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.627+0000 7f90ef7fe640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90f406b4d0 0x7f90f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.627+0000 7f90ef7fe640 1 -- 172.21.15.77:0/520154710 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91000ac870 con 0x7f910815c050 2024-03-06T07:24:44.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.627+0000 7f9106d9d640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90f406b4d0 0x7f90f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.627+0000 7f9106d9d640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90f406b4d0 0x7f90f406d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f90fc007a00 tx=0x7f90fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.631+0000 7f90ef7fe640 1 -- 172.21.15.77:0/520154710 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f910004e030 con 0x7f910815c050 2024-03-06T07:24:44.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.806+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9108068ff0 con 0x7f910815c050 2024-03-06T07:24:44.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.807+0000 7f90ef7fe640 1 -- 172.21.15.77:0/520154710 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f910007b670 con 0x7f910815c050 2024-03-06T07:24:44.809 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:24:44.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90f406b4d0 msgr2=0x7f90f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90f406b4d0 0x7f90f406d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f90fc007a00 tx=0x7f90fc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f910815c050 msgr2=0x7f910814f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f910815c050 0x7f910814f310 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9100009d90 tx=0x7f9100004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 shutdown_connections 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90f406b4d0 0x7f90f406d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f910815c050 0x7f910814f310 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91080d0af0 0x7f91080b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 --2- 172.21.15.77:0/520154710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f910806fd10 0x7f9108159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 >> 172.21.15.77:0/520154710 conn(0x7f91080b3800 msgr2=0x7f910806f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 shutdown_connections 2024-03-06T07:24:44.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.809+0000 7f910d15c640 1 -- 172.21.15.77:0/520154710 wait complete. 2024-03-06T07:24:44.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.927+0000 7ff23e11e640 1 Processor -- start 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.927+0000 7ff23e11e640 1 -- start start 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff23e11e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff2380b1e40 0x7ff238168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff23e11e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2380d2730 con 0x7ff2381614d0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff23e11e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2381618b0 con 0x7ff2380d4860 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff23e11e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2380d4cf0 con 0x7ff2380b1e40 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff237fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff2380b1e40 0x7ff238168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff2377fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff2381614d0 0x7ff2380d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41294/0 (socket says 172.21.15.77:41294) 2024-03-06T07:24:44.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff2377fe640 1 -- 172.21.15.77:0/1250829702 learned_addr learned my addr 172.21.15.77:0/1250829702 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:44.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff237fff640 1 -- 172.21.15.77:0/1250829702 >> v1:172.21.15.158:6789/0 conn(0x7ff2380d4860 legacy=0x7ff2380d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff237fff640 1 -- 172.21.15.77:0/1250829702 >> v1:172.21.15.77:6789/0 conn(0x7ff2381614d0 legacy=0x7ff2380d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff237fff640 1 -- 172.21.15.77:0/1250829702 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2380cf400 con 0x7ff2380b1e40 2024-03-06T07:24:44.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.928+0000 7ff236ffd640 1 -- 172.21.15.77:0/1250829702 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72062164 0 0) 0x7ff2380d2730 con 0x7ff2381614d0 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.929+0000 7ff237fff640 1 --2- 172.21.15.77:0/1250829702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff238168b60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ff2380cf5d0 tx=0x7ff23002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefd3511275bca99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.929+0000 7ff236ffd640 1 -- 172.21.15.77:0/1250829702 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230039070 con 0x7ff2380b1e40 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.929+0000 7ff236ffd640 1 -- 172.21.15.77:0/1250829702 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23002cc50 con 0x7ff2380b1e40 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.929+0000 7ff236ffd640 1 -- 172.21.15.77:0/1250829702 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230034440 con 0x7ff2380b1e40 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.930+0000 7ff23e11e640 1 -- 172.21.15.77:0/1250829702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 msgr2=0x7ff238168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.930+0000 7ff23e11e640 1 --2- 172.21.15.77:0/1250829702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff238168b60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ff2380cf5d0 tx=0x7ff23002bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.930+0000 7ff23e11e640 1 -- 172.21.15.77:0/1250829702 shutdown_connections 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.930+0000 7ff23e11e640 1 --2- 172.21.15.77:0/1250829702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff238168b60 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.930+0000 7ff23e11e640 1 -- 172.21.15.77:0/1250829702 >> 172.21.15.77:0/1250829702 conn(0x7ff2380d3d40 msgr2=0x7ff2380d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.930+0000 7ff23e11e640 1 -- 172.21.15.77:0/1250829702 shutdown_connections 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.930+0000 7ff23e11e640 1 -- 172.21.15.77:0/1250829702 wait complete. 2024-03-06T07:24:44.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.932+0000 7ff23e11e640 1 Processor -- start 2024-03-06T07:24:44.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.932+0000 7ff23e11e640 1 -- start start 2024-03-06T07:24:44.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff23e11e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff2380c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff23e11e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2380d4860 0x7ff23815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff23e11e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2381614d0 0x7ff2380b8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff23e11e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380cfc60 con 0x7ff2380b1e40 2024-03-06T07:24:44.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff23e11e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380cf980 con 0x7ff2381614d0 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff23e11e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380b0910 con 0x7ff2380d4860 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff2377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff2380c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff2377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff2380c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44262/0 (socket says 172.21.15.77:44262) 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff2377fe640 1 -- 172.21.15.77:0/3741114956 learned_addr learned my addr 172.21.15.77:0/3741114956 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff237fff640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2381614d0 0x7ff2380b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.933+0000 7ff22effd640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2380d4860 0x7ff23815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff2377fe640 1 -- 172.21.15.77:0/3741114956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2380d4860 msgr2=0x7ff23815b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff2377fe640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2380d4860 0x7ff23815b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff2377fe640 1 -- 172.21.15.77:0/3741114956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2381614d0 msgr2=0x7ff2380b8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff2377fe640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2381614d0 0x7ff2380b8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff2377fe640 1 -- 172.21.15.77:0/3741114956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2380c8510 con 0x7ff2380b1e40 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff2377fe640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff2380c95b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff22800c4d0 tx=0x7ff22800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff234ff9640 1 -- 172.21.15.77:0/3741114956 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228018070 con 0x7ff2380b1e40 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff234ff9640 1 -- 172.21.15.77:0/3741114956 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2280074e0 con 0x7ff2380b1e40 2024-03-06T07:24:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2380c87f0 con 0x7ff2380b1e40 2024-03-06T07:24:44.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff234ff9640 1 -- 172.21.15.77:0/3741114956 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228004a20 con 0x7ff2380b1e40 2024-03-06T07:24:44.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.934+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2380c8db0 con 0x7ff2380b1e40 2024-03-06T07:24:44.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.935+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2380ca9c0 con 0x7ff2380b1e40 2024-03-06T07:24:44.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.935+0000 7ff234ff9640 1 -- 172.21.15.77:0/3741114956 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff22801b070 con 0x7ff2380b1e40 2024-03-06T07:24:44.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.936+0000 7ff234ff9640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff21806b4d0 0x7ff21806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.936+0000 7ff234ff9640 1 -- 172.21.15.77:0/3741114956 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2280894b0 con 0x7ff2380b1e40 2024-03-06T07:24:44.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.936+0000 7ff22effd640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff21806b4d0 0x7ff21806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.936+0000 7ff22effd640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff21806b4d0 0x7ff21806d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff23815c330 tx=0x7ff220009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:44.940+0000 7ff234ff9640 1 -- 172.21.15.77:0/3741114956 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff228057ce0 con 0x7ff2380b1e40 2024-03-06T07:24:45.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.130+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff238068ff0 con 0x7ff2380b1e40 2024-03-06T07:24:45.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.131+0000 7ff234ff9640 1 -- 172.21.15.77:0/3741114956 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff22805bd80 con 0x7ff2380b1e40 2024-03-06T07:24:45.133 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:24:45.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff21806b4d0 msgr2=0x7ff21806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff21806b4d0 0x7ff21806d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff23815c330 tx=0x7ff220009040 comp rx=0 tx=0).stop 2024-03-06T07:24:45.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 msgr2=0x7ff2380c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff2380c95b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff22800c4d0 tx=0x7ff22800c990 comp rx=0 tx=0).stop 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 shutdown_connections 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2381614d0 0x7ff2380b8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2380d4860 0x7ff23815b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff21806b4d0 0x7ff21806d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 --2- 172.21.15.77:0/3741114956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2380b1e40 0x7ff2380c95b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 >> 172.21.15.77:0/3741114956 conn(0x7ff2380d3d40 msgr2=0x7ff2380cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 shutdown_connections 2024-03-06T07:24:45.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.133+0000 7ff23e11e640 1 -- 172.21.15.77:0/3741114956 wait complete. 2024-03-06T07:24:45.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:45.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.250+0000 7febb64a4640 1 Processor -- start 2024-03-06T07:24:45.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.250+0000 7febb64a4640 1 -- start start 2024-03-06T07:24:45.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febb64a4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7febb00ce920 0x7febb015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febb64a4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7febb006b600 0x7febb0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febb64a4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb015ea30 con 0x7febb006b220 2024-03-06T07:24:45.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febb64a4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb0161b70 con 0x7febb00ce920 2024-03-06T07:24:45.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febb64a4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb015ed10 con 0x7febb006b600 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febaf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7febb00ce920 0x7febb015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febaffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7febb006b220 0x7febb0071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41302/0 (socket says 172.21.15.77:41302) 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febaf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7febb00ce920 0x7febb015e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44278/0 (socket says 172.21.15.77:44278) 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febaffff640 1 -- 172.21.15.77:0/1625855752 learned_addr learned my addr 172.21.15.77:0/1625855752 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febb4a41640 1 --2- 172.21.15.77:0/1625855752 >> v2:172.21.15.158:3300/0 conn(0x7febb006b600 0x7febb0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febaf7fe640 1 -- 172.21.15.77:0/1625855752 >> v2:172.21.15.158:3300/0 conn(0x7febb006b600 msgr2=0x7febb0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febaf7fe640 1 --2- 172.21.15.77:0/1625855752 >> v2:172.21.15.158:3300/0 conn(0x7febb006b600 0x7febb0162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.251+0000 7febaf7fe640 1 -- 172.21.15.77:0/1625855752 >> v1:172.21.15.77:6789/0 conn(0x7febb006b220 legacy=0x7febb0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.252+0000 7febaf7fe640 1 -- 172.21.15.77:0/1625855752 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb015eff0 con 0x7febb00ce920 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.252+0000 7febaeffd640 1 -- 172.21.15.77:0/1625855752 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809365550 0 0) 0x7febb015ea30 con 0x7febb006b220 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.252+0000 7febb4a41640 1 --2- 172.21.15.77:0/1625855752 >> v2:172.21.15.158:3300/0 conn(0x7febb006b600 0x7febb0162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:45.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.252+0000 7febaf7fe640 1 --2- 172.21.15.77:0/1625855752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb00ce920 0x7febb015e450 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7febb006c040 tx=0x7feba002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c4b4c83f5edf54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.252+0000 7febaeffd640 1 -- 172.21.15.77:0/1625855752 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0035020 con 0x7febb00ce920 2024-03-06T07:24:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.252+0000 7febaeffd640 1 -- 172.21.15.77:0/1625855752 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba0031830 con 0x7febb00ce920 2024-03-06T07:24:45.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.252+0000 7febaeffd640 1 -- 172.21.15.77:0/1625855752 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0031c00 con 0x7febb00ce920 2024-03-06T07:24:45.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.253+0000 7febb64a4640 1 -- 172.21.15.77:0/1625855752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb00ce920 msgr2=0x7febb015e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.253+0000 7febb64a4640 1 --2- 172.21.15.77:0/1625855752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb00ce920 0x7febb015e450 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7febb006c040 tx=0x7feba002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.253+0000 7febb64a4640 1 -- 172.21.15.77:0/1625855752 shutdown_connections 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.253+0000 7febb64a4640 1 --2- 172.21.15.77:0/1625855752 >> v2:172.21.15.158:3300/0 conn(0x7febb006b600 0x7febb0162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.253+0000 7febb64a4640 1 --2- 172.21.15.77:0/1625855752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb00ce920 0x7febb015e450 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.253+0000 7febb64a4640 1 -- 172.21.15.77:0/1625855752 >> 172.21.15.77:0/1625855752 conn(0x7febb006c0c0 msgr2=0x7febb006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.253+0000 7febb64a4640 1 -- 172.21.15.77:0/1625855752 shutdown_connections 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.253+0000 7febb64a4640 1 -- 172.21.15.77:0/1625855752 wait complete. 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.254+0000 7febb64a4640 1 Processor -- start 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.254+0000 7febb64a4640 1 -- start start 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febb64a4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb006b220 0x7febb00adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febb64a4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006b600 0x7febb00b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febb64a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febb00ce920 0x7febb014a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febb64a4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb0072090 con 0x7febb006b220 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febb64a4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb00ce200 con 0x7febb006b600 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febb64a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb00b0d00 con 0x7febb00ce920 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb006b220 0x7febb00adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb006b220 0x7febb00adc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44284/0 (socket says 172.21.15.77:44284) 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaffff640 1 -- 172.21.15.77:0/1667638771 learned_addr learned my addr 172.21.15.77:0/1667638771 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaf7fe640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006b600 0x7febb00b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febb4a41640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febb00ce920 0x7febb014a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaffff640 1 -- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febb00ce920 msgr2=0x7febb014a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaffff640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febb00ce920 0x7febb014a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaffff640 1 -- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006b600 msgr2=0x7febb00b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaffff640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006b600 0x7febb00b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febaffff640 1 -- 172.21.15.77:0/1667638771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb00b0fe0 con 0x7febb006b220 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.255+0000 7febb4a41640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febb00ce920 0x7febb014a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:45.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.256+0000 7febaffff640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb006b220 0x7febb00adc10 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7feba400c430 tx=0x7feba400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.256+0000 7febacff9640 1 -- 172.21.15.77:0/1667638771 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40074e0 con 0x7febb006b220 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.256+0000 7febacff9640 1 -- 172.21.15.77:0/1667638771 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba40047e0 con 0x7febb006b220 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.256+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb00b12c0 con 0x7febb006b220 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.256+0000 7febacff9640 1 -- 172.21.15.77:0/1667638771 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4004bd0 con 0x7febb006b220 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.256+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb00b1880 con 0x7febb006b220 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.257+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb006da40 con 0x7febb006b220 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.257+0000 7febacff9640 1 -- 172.21.15.77:0/1667638771 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feba4011070 con 0x7febb006b220 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.258+0000 7febacff9640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb9c06b4d0 0x7feb9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.258+0000 7febacff9640 1 -- 172.21.15.77:0/1667638771 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feba4089e20 con 0x7febb006b220 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.258+0000 7febaf7fe640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb9c06b4d0 0x7feb9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.258+0000 7febaf7fe640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb9c06b4d0 0x7feb9c06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feba0036000 tx=0x7feba0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.266+0000 7febacff9640 1 -- 172.21.15.77:0/1667638771 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febb006da40 con 0x7febb006b220 2024-03-06T07:24:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.497+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febb004f330 con 0x7febb006b220 2024-03-06T07:24:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.497+0000 7febacff9640 1 -- 172.21.15.77:0/1667638771 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7feba4058ca0 con 0x7febb006b220 2024-03-06T07:24:45.499 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:45.499 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.760628+0000","modified":"2024-03-06T07:24:12.764047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/251375557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":251375557},{"type":"v1","addr":"172.21.15.77:6837","nonce":251375557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-03-06T07:24:45.500 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-06T07:24:45.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb9c06b4d0 msgr2=0x7feb9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb9c06b4d0 0x7feb9c06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feba0036000 tx=0x7feba0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb006b220 msgr2=0x7febb00adc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb006b220 0x7febb00adc10 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7feba400c430 tx=0x7feba400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 shutdown_connections 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febb00ce920 0x7febb014a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febb006b600 0x7febb00b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb9c06b4d0 0x7feb9c06d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 --2- 172.21.15.77:0/1667638771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febb006b220 0x7febb00adc10 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.499+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 >> 172.21.15.77:0/1667638771 conn(0x7febb006c0c0 msgr2=0x7febb006e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.500+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 shutdown_connections 2024-03-06T07:24:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.500+0000 7febb64a4640 1 -- 172.21.15.77:0/1667638771 wait complete. 2024-03-06T07:24:45.513 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:45.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.611+0000 7fe8a2678640 1 Processor -- start 2024-03-06T07:24:45.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.612+0000 7fe8a2678640 1 -- start start 2024-03-06T07:24:45.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.612+0000 7fe8a2678640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe89c15fa10 0x7fe89c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.612+0000 7fe8a2678640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe89c0d16e0 0x7fe89c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.612+0000 7fe8a2678640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0cebe0 con 0x7fe89c15bdd0 2024-03-06T07:24:45.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.612+0000 7fe8a2678640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c0ceec0 con 0x7fe89c15fa10 2024-03-06T07:24:45.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.612+0000 7fe8a2678640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c0cf1a0 con 0x7fe89c0d16e0 2024-03-06T07:24:45.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe89c15fa10 0x7fe89c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe89c15fa10 0x7fe89c0ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44290/0 (socket says 172.21.15.77:44290) 2024-03-06T07:24:45.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe89c15bdd0 0x7fe89c0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41314/0 (socket says 172.21.15.77:41314) 2024-03-06T07:24:45.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89b7fe640 1 -- 172.21.15.77:0/3963356894 learned_addr learned my addr 172.21.15.77:0/3963356894 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:45.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89b7fe640 1 -- 172.21.15.77:0/3963356894 >> v2:172.21.15.158:3300/0 conn(0x7fe89c0d16e0 msgr2=0x7fe89c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe8a0c15640 1 --2- 172.21.15.77:0/3963356894 >> v2:172.21.15.158:3300/0 conn(0x7fe89c0d16e0 0x7fe89c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89b7fe640 1 --2- 172.21.15.77:0/3963356894 >> v2:172.21.15.158:3300/0 conn(0x7fe89c0d16e0 0x7fe89c0d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89affd640 1 -- 172.21.15.77:0/3963356894 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344556046 0 0) 0x7fe89c0cebe0 con 0x7fe89c15bdd0 2024-03-06T07:24:45.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89b7fe640 1 -- 172.21.15.77:0/3963356894 >> v1:172.21.15.77:6789/0 conn(0x7fe89c15bdd0 legacy=0x7fe89c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.613+0000 7fe89b7fe640 1 -- 172.21.15.77:0/3963356894 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0cc8e0 con 0x7fe89c15fa10 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe89b7fe640 1 --2- 172.21.15.77:0/3963356894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c0ce270 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0c5520 tx=0x7fe89002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9e50c13319e0a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe89affd640 1 -- 172.21.15.77:0/3963356894 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890035020 con 0x7fe89c15fa10 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe89affd640 1 -- 172.21.15.77:0/3963356894 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe890031870 con 0x7fe89c15fa10 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe89affd640 1 -- 172.21.15.77:0/3963356894 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890031c40 con 0x7fe89c15fa10 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe8a2678640 1 -- 172.21.15.77:0/3963356894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 msgr2=0x7fe89c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3963356894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c0ce270 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0c5520 tx=0x7fe89002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe8a2678640 1 -- 172.21.15.77:0/3963356894 shutdown_connections 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3963356894 >> v2:172.21.15.158:3300/0 conn(0x7fe89c0d16e0 0x7fe89c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3963356894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c0ce270 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe8a2678640 1 -- 172.21.15.77:0/3963356894 >> 172.21.15.77:0/3963356894 conn(0x7fe89c09a9f0 msgr2=0x7fe89c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe8a2678640 1 -- 172.21.15.77:0/3963356894 shutdown_connections 2024-03-06T07:24:45.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.614+0000 7fe8a2678640 1 -- 172.21.15.77:0/3963356894 wait complete. 2024-03-06T07:24:45.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.616+0000 7fe8a2678640 1 Processor -- start 2024-03-06T07:24:45.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.616+0000 7fe8a2678640 1 -- start start 2024-03-06T07:24:45.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.616+0000 7fe8a2678640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c0d16e0 0x7fe89c0c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.616+0000 7fe8a2678640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c15bdd0 0x7fe89c0cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.616+0000 7fe8a2678640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c1e65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.616+0000 7fe8a2678640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0d2030 con 0x7fe89c15fa10 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.616+0000 7fe8a2678640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c15c1b0 con 0x7fe89c0d16e0 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.616+0000 7fe8a2678640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0ce7b0 con 0x7fe89c15bdd0 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c1e65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c1e65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44302/0 (socket says 172.21.15.77:44302) 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 -- 172.21.15.77:0/3811978897 learned_addr learned my addr 172.21.15.77:0/3811978897 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe89bfff640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c0d16e0 0x7fe89c0c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe89b7fe640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c15bdd0 0x7fe89c0cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 -- 172.21.15.77:0/3811978897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c15bdd0 msgr2=0x7fe89c0cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c15bdd0 0x7fe89c0cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 -- 172.21.15.77:0/3811978897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c0d16e0 msgr2=0x7fe89c0c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c0d16e0 0x7fe89c0c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 -- 172.21.15.77:0/3811978897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c06d500 con 0x7fe89c15fa10 2024-03-06T07:24:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a0c15640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c1e65b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe89400cd70 tx=0x7fe89400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe898ff9640 1 -- 172.21.15.77:0/3811978897 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894003b90 con 0x7fe89c15fa10 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe898ff9640 1 -- 172.21.15.77:0/3811978897 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894003d30 con 0x7fe89c15fa10 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c06d7e0 con 0x7fe89c15fa10 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe898ff9640 1 -- 172.21.15.77:0/3811978897 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894004240 con 0x7fe89c15fa10 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.617+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c06dda0 con 0x7fe89c15fa10 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.618+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c06e7d0 con 0x7fe89c15fa10 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.619+0000 7fe898ff9640 1 -- 172.21.15.77:0/3811978897 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8940043e0 con 0x7fe89c15fa10 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.619+0000 7fe898ff9640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88806b500 0x7fe88806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.619+0000 7fe898ff9640 1 -- 172.21.15.77:0/3811978897 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe89408ab50 con 0x7fe89c15fa10 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.619+0000 7fe89bfff640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88806b500 0x7fe88806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.620+0000 7fe89bfff640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88806b500 0x7fe88806d9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000f60 tx=0x7fe88c007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.623+0000 7fe898ff9640 1 -- 172.21.15.77:0/3811978897 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe89c06e7d0 con 0x7fe89c15fa10 2024-03-06T07:24:45.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.801+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe89c04f330 con 0x7fe89c15fa10 2024-03-06T07:24:45.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.802+0000 7fe898ff9640 1 -- 172.21.15.77:0/3811978897 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fe894059410 con 0x7fe89c15fa10 2024-03-06T07:24:45.804 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:45.805 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:12.416501+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:45.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88806b500 msgr2=0x7fe88806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88806b500 0x7fe88806d9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000f60 tx=0x7fe88c007d50 comp rx=0 tx=0).stop 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 msgr2=0x7fe89c1e65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c1e65b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe89400cd70 tx=0x7fe89400a660 comp rx=0 tx=0).stop 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 shutdown_connections 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88806b500 0x7fe88806d9c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe89c15fa10 0x7fe89c1e65b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c15bdd0 0x7fe89c0cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 --2- 172.21.15.77:0/3811978897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c0d16e0 0x7fe89c0c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.804+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 >> 172.21.15.77:0/3811978897 conn(0x7fe89c09a9f0 msgr2=0x7fe89c06f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.805+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 shutdown_connections 2024-03-06T07:24:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.805+0000 7fe8a2678640 1 -- 172.21.15.77:0/3811978897 wait complete. 2024-03-06T07:24:45.818 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:45.818 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:45.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.920+0000 7fbb435bd640 1 Processor -- start 2024-03-06T07:24:45.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.920+0000 7fbb435bd640 1 -- start start 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb435bd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c15dfe0 0x7fbb3c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb435bd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb3c0cf830 0x7fbb3c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb435bd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbb3c0d27f0 0x7fbb3c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb435bd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3c0d2c50 con 0x7fbb3c0cf830 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb435bd640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3c0b31e0 con 0x7fbb3c0d27f0 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb435bd640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c0b3d60 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb40b58640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb3c0cf830 0x7fbb3c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb40b58640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb3c0cf830 0x7fbb3c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44316/0 (socket says 172.21.15.77:44316) 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.921+0000 7fbb40b58640 1 -- 172.21.15.77:0/3706120417 learned_addr learned my addr 172.21.15.77:0/3706120417 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb41b5a640 1 --2- 172.21.15.77:0/3706120417 >> v2:172.21.15.158:3300/0 conn(0x7fbb3c0d27f0 0x7fbb3c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb41359640 1 --2- 172.21.15.77:0/3706120417 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c15dfe0 0x7fbb3c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb40b58640 1 -- 172.21.15.77:0/3706120417 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c15dfe0 msgr2=0x7fbb3c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb40b58640 1 --2- 172.21.15.77:0/3706120417 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c15dfe0 0x7fbb3c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb40b58640 1 -- 172.21.15.77:0/3706120417 >> v2:172.21.15.158:3300/0 conn(0x7fbb3c0d27f0 msgr2=0x7fbb3c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb40b58640 1 --2- 172.21.15.77:0/3706120417 >> v2:172.21.15.158:3300/0 conn(0x7fbb3c0d27f0 0x7fbb3c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb40b58640 1 -- 172.21.15.77:0/3706120417 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c0b3430 con 0x7fbb3c0cf830 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb41b5a640 1 --2- 172.21.15.77:0/3706120417 >> v2:172.21.15.158:3300/0 conn(0x7fbb3c0d27f0 0x7fbb3c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.922+0000 7fbb40b58640 1 --2- 172.21.15.77:0/3706120417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c0cf830 0x7fbb3c0d22b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c067ad0 tx=0x7fbb3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5053de04a2dbd3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb2ffff640 1 -- 172.21.15.77:0/3706120417 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30035020 con 0x7fbb3c0cf830 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb2ffff640 1 -- 172.21.15.77:0/3706120417 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb30031870 con 0x7fbb3c0cf830 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb2ffff640 1 -- 172.21.15.77:0/3706120417 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30031c40 con 0x7fbb3c0cf830 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 -- 172.21.15.77:0/3706120417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c0cf830 msgr2=0x7fbb3c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 --2- 172.21.15.77:0/3706120417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c0cf830 0x7fbb3c0d22b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c067ad0 tx=0x7fbb3002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 -- 172.21.15.77:0/3706120417 shutdown_connections 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 --2- 172.21.15.77:0/3706120417 >> v2:172.21.15.158:3300/0 conn(0x7fbb3c0d27f0 0x7fbb3c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 --2- 172.21.15.77:0/3706120417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c0cf830 0x7fbb3c0d22b0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 --2- 172.21.15.77:0/3706120417 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c15dfe0 0x7fbb3c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 -- 172.21.15.77:0/3706120417 >> 172.21.15.77:0/3706120417 conn(0x7fbb3c0d3cd0 msgr2=0x7fbb3c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 -- 172.21.15.77:0/3706120417 shutdown_connections 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.923+0000 7fbb435bd640 1 -- 172.21.15.77:0/3706120417 wait complete. 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.924+0000 7fbb435bd640 1 Processor -- start 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.924+0000 7fbb435bd640 1 -- start start 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb435bd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c0cf830 0x7fbb3c160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb435bd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3c0d27f0 0x7fbb3c1f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb435bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c15dfe0 0x7fbb3c1f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb435bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c0d1c70 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb435bd640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c0b39f0 con 0x7fbb3c0d27f0 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb435bd640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c0d1df0 con 0x7fbb3c0cf830 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb41b5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c15dfe0 0x7fbb3c1f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb41b5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c15dfe0 0x7fbb3c1f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44328/0 (socket says 172.21.15.77:44328) 2024-03-06T07:24:45.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb41b5a640 1 -- 172.21.15.77:0/1628768887 learned_addr learned my addr 172.21.15.77:0/1628768887 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb40b58640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3c0d27f0 0x7fbb3c1f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.925+0000 7fbb41359640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c0cf830 0x7fbb3c160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb41b5a640 1 -- 172.21.15.77:0/1628768887 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c0cf830 msgr2=0x7fbb3c160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb41b5a640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c0cf830 0x7fbb3c160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb41b5a640 1 -- 172.21.15.77:0/1628768887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3c0d27f0 msgr2=0x7fbb3c1f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb41b5a640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3c0d27f0 0x7fbb3c1f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb41b5a640 1 -- 172.21.15.77:0/1628768887 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c15f490 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb41b5a640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c15dfe0 0x7fbb3c1f71b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbb34007e10 tx=0x7fbb3400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb2dffb640 1 -- 172.21.15.77:0/1628768887 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3401a070 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb2dffb640 1 -- 172.21.15.77:0/1628768887 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb34003b90 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c15f770 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb2dffb640 1 -- 172.21.15.77:0/1628768887 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb34004030 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.926+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c1614d0 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.927+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3c04f330 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.927+0000 7fbb2dffb640 1 -- 172.21.15.77:0/1628768887 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb3401d070 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.928+0000 7fbb2dffb640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb2006b4d0 0x7fbb2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.928+0000 7fbb2dffb640 1 -- 172.21.15.77:0/1628768887 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb3408a3c0 con 0x7fbb3c15dfe0 2024-03-06T07:24:45.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.928+0000 7fbb41359640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb2006b4d0 0x7fbb2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.928+0000 7fbb41359640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb2006b4d0 0x7fbb2006d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbb380079a0 tx=0x7fbb38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:45.932+0000 7fbb2dffb640 1 -- 172.21.15.77:0/1628768887 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb34010070 con 0x7fbb3c15dfe0 2024-03-06T07:24:46.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.143+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb3c0747a0 con 0x7fbb3c15dfe0 2024-03-06T07:24:46.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.143+0000 7fbb2dffb640 1 -- 172.21.15.77:0/1628768887 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fbb34058c80 con 0x7fbb3c15dfe0 2024-03-06T07:24:46.145 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:46.145 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.760628+0000","modified":"2024-03-06T07:24:12.764047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/251375557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":251375557},{"type":"v1","addr":"172.21.15.77:6837","nonce":251375557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-03-06T07:24:46.146 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-06T07:24:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb2006b4d0 msgr2=0x7fbb2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb2006b4d0 0x7fbb2006d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbb380079a0 tx=0x7fbb38008040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c15dfe0 msgr2=0x7fbb3c1f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c15dfe0 0x7fbb3c1f71b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbb34007e10 tx=0x7fbb3400d4a0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 shutdown_connections 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb2006b4d0 0x7fbb2006d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb3c15dfe0 0x7fbb3c1f71b0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3c0d27f0 0x7fbb3c1f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 --2- 172.21.15.77:0/1628768887 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb3c0cf830 0x7fbb3c160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 >> 172.21.15.77:0/1628768887 conn(0x7fbb3c0d3cd0 msgr2=0x7fbb3c06ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 shutdown_connections 2024-03-06T07:24:46.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.146+0000 7fbb435bd640 1 -- 172.21.15.77:0/1628768887 wait complete. 2024-03-06T07:24:46.159 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:46.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.258+0000 7f4cd7284640 1 Processor -- start 2024-03-06T07:24:46.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.258+0000 7f4cd7284640 1 -- start start 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd7284640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4cd015dfe0 0x7f4cd00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd7284640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4cd00cf830 0x7f4cd00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd7284640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd00b3430 con 0x7f4cd00d27f0 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd7284640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd00b31e0 con 0x7f4cd015dfe0 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd7284640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd00b3d60 con 0x7f4cd00cf830 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd5020640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4cd015dfe0 0x7f4cd00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd5821640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4cd00d27f0 0x7f4cd00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41324/0 (socket says 172.21.15.77:41324) 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd5821640 1 -- 172.21.15.77:0/3026700047 learned_addr learned my addr 172.21.15.77:0/3026700047 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd5020640 1 -- 172.21.15.77:0/3026700047 >> v2:172.21.15.158:3300/0 conn(0x7f4cd00cf830 msgr2=0x7f4cd00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd481f640 1 --2- 172.21.15.77:0/3026700047 >> v2:172.21.15.158:3300/0 conn(0x7f4cd00cf830 0x7f4cd00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd5020640 1 --2- 172.21.15.77:0/3026700047 >> v2:172.21.15.158:3300/0 conn(0x7f4cd00cf830 0x7f4cd00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd5020640 1 -- 172.21.15.77:0/3026700047 >> v1:172.21.15.77:6789/0 conn(0x7f4cd00d27f0 legacy=0x7f4cd00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd5020640 1 -- 172.21.15.77:0/3026700047 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd00b3710 con 0x7f4cd015dfe0 2024-03-06T07:24:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cc7fff640 1 -- 172.21.15.77:0/3026700047 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353235671 0 0) 0x7f4cd00b3430 con 0x7f4cd00d27f0 2024-03-06T07:24:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.259+0000 7f4cd481f640 1 --2- 172.21.15.77:0/3026700047 >> v2:172.21.15.158:3300/0 conn(0x7f4cd00cf830 0x7f4cd00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.260+0000 7f4cd5020640 1 --2- 172.21.15.77:0/3026700047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd015dfe0 0x7f4cd00cf2f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0097a0 tx=0x7f4ccc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6f7e8ddbff999e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.260+0000 7f4cc7fff640 1 -- 172.21.15.77:0/3026700047 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc035020 con 0x7f4cd015dfe0 2024-03-06T07:24:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.260+0000 7f4cc7fff640 1 -- 172.21.15.77:0/3026700047 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc031830 con 0x7f4cd015dfe0 2024-03-06T07:24:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.260+0000 7f4cc7fff640 1 -- 172.21.15.77:0/3026700047 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc031c00 con 0x7f4cd015dfe0 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.261+0000 7f4cd7284640 1 -- 172.21.15.77:0/3026700047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd015dfe0 msgr2=0x7f4cd00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.261+0000 7f4cd7284640 1 --2- 172.21.15.77:0/3026700047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd015dfe0 0x7f4cd00cf2f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0097a0 tx=0x7f4ccc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.261+0000 7f4cd7284640 1 -- 172.21.15.77:0/3026700047 shutdown_connections 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.261+0000 7f4cd7284640 1 --2- 172.21.15.77:0/3026700047 >> v2:172.21.15.158:3300/0 conn(0x7f4cd00cf830 0x7f4cd00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.261+0000 7f4cd7284640 1 --2- 172.21.15.77:0/3026700047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd015dfe0 0x7f4cd00cf2f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.261+0000 7f4cd7284640 1 -- 172.21.15.77:0/3026700047 >> 172.21.15.77:0/3026700047 conn(0x7f4cd00d3cd0 msgr2=0x7f4cd00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.261+0000 7f4cd7284640 1 -- 172.21.15.77:0/3026700047 shutdown_connections 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.261+0000 7f4cd7284640 1 -- 172.21.15.77:0/3026700047 wait complete. 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.262+0000 7f4cd7284640 1 Processor -- start 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.262+0000 7f4cd7284640 1 -- start start 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd7284640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd00cf830 0x7f4cd00b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd7284640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cd00d27f0 0x7f4cd00b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd7284640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cd015dfe0 0x7f4cd00b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd7284640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd00d1df0 con 0x7f4cd00cf830 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd7284640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd00d1c70 con 0x7f4cd015dfe0 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd7284640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015e3c0 con 0x7f4cd00d27f0 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd00cf830 0x7f4cd00b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd00cf830 0x7f4cd00b91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44350/0 (socket says 172.21.15.77:44350) 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 -- 172.21.15.77:0/4105816852 learned_addr learned my addr 172.21.15.77:0/4105816852 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 -- 172.21.15.77:0/4105816852 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cd00d27f0 msgr2=0x7f4cd00b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cd00d27f0 0x7f4cd00b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 -- 172.21.15.77:0/4105816852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cd015dfe0 msgr2=0x7f4cd00b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cd015dfe0 0x7f4cd00b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 -- 172.21.15.77:0/4105816852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd00b8430 con 0x7f4cd00cf830 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd5020640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd00cf830 0x7f4cd00b91f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009890 tx=0x7f4ccc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cc5ffb640 1 -- 172.21.15.77:0/4105816852 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc041070 con 0x7f4cd00cf830 2024-03-06T07:24:46.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cc5ffb640 1 -- 172.21.15.77:0/4105816852 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc03aa30 con 0x7f4cd00cf830 2024-03-06T07:24:46.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd00b8710 con 0x7f4cd00cf830 2024-03-06T07:24:46.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.263+0000 7f4cc5ffb640 1 -- 172.21.15.77:0/4105816852 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc035040 con 0x7f4cd00cf830 2024-03-06T07:24:46.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.264+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd00b8cd0 con 0x7f4cd00cf830 2024-03-06T07:24:46.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.265+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd004f330 con 0x7f4cd00cf830 2024-03-06T07:24:46.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.265+0000 7f4cc5ffb640 1 -- 172.21.15.77:0/4105816852 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ccc04c060 con 0x7f4cd00cf830 2024-03-06T07:24:46.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.265+0000 7f4cc5ffb640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb406b4d0 0x7f4cb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.266+0000 7f4cc5ffb640 1 -- 172.21.15.77:0/4105816852 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ccc0abc40 con 0x7f4cd00cf830 2024-03-06T07:24:46.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.266+0000 7f4cd481f640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb406b4d0 0x7f4cb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.266+0000 7f4cd481f640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb406b4d0 0x7f4cb406d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0006fd0 tx=0x7f4cc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.274+0000 7f4cc5ffb640 1 -- 172.21.15.77:0/4105816852 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ccc07aac0 con 0x7f4cd00cf830 2024-03-06T07:24:46.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.493+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd006a1b0 con 0x7f4cd00cf830 2024-03-06T07:24:46.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.494+0000 7f4cc5ffb640 1 -- 172.21.15.77:0/4105816852 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f4ccc07a3c0 con 0x7f4cd00cf830 2024-03-06T07:24:46.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:46.496 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:10.760628+0000","modified":"2024-03-06T07:24:12.764047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/251375557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":251375557},{"type":"v1","addr":"172.21.15.77:6837","nonce":251375557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-03-06T07:24:46.496 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.495+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb406b4d0 msgr2=0x7f4cb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.495+0000 7f4cd7284640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb406b4d0 0x7f4cb406d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0006fd0 tx=0x7f4cc0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.495+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd00cf830 msgr2=0x7f4cd00b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.495+0000 7f4cd7284640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd00cf830 0x7f4cd00b91f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009890 tx=0x7f4ccc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.496+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 shutdown_connections 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.496+0000 7f4cd7284640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cd015dfe0 0x7f4cd00b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.496+0000 7f4cd7284640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cd00d27f0 0x7f4cd00b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.496+0000 7f4cd7284640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb406b4d0 0x7f4cb406d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.496+0000 7f4cd7284640 1 --2- 172.21.15.77:0/4105816852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cd00cf830 0x7f4cd00b91f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.496+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 >> 172.21.15.77:0/4105816852 conn(0x7f4cd00d3cd0 msgr2=0x7f4cd00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.496+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 shutdown_connections 2024-03-06T07:24:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.496+0000 7f4cd7284640 1 -- 172.21.15.77:0/4105816852 wait complete. 2024-03-06T07:24:46.508 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:46.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.607+0000 7f69e268f640 1 Processor -- start 2024-03-06T07:24:46.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.607+0000 7f69e268f640 1 -- start start 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69e268f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69dc15dfe0 0x7f69dc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69e268f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69dc0cf830 0x7f69dc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69e268f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0d27f0 0x7f69dc0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69e268f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69dc0b31e0 con 0x7f69dc0cf830 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69e268f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69dc0b3d60 con 0x7f69dc15dfe0 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69e268f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0b3430 con 0x7f69dc0d27f0 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69db7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69dc0cf830 0x7f69dc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69db7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69dc0cf830 0x7f69dc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44354/0 (socket says 172.21.15.77:44354) 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69db7fe640 1 -- 172.21.15.77:0/2232146905 learned_addr learned my addr 172.21.15.77:0/2232146905 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69e0c2c640 1 --2- 172.21.15.77:0/2232146905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0d27f0 0x7f69dc0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69dbfff640 1 --2- 172.21.15.77:0/2232146905 >> v2:172.21.15.158:3300/0 conn(0x7f69dc15dfe0 0x7f69dc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69db7fe640 1 -- 172.21.15.77:0/2232146905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0d27f0 msgr2=0x7f69dc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69db7fe640 1 --2- 172.21.15.77:0/2232146905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0d27f0 0x7f69dc0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69db7fe640 1 -- 172.21.15.77:0/2232146905 >> v2:172.21.15.158:3300/0 conn(0x7f69dc15dfe0 msgr2=0x7f69dc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69db7fe640 1 --2- 172.21.15.77:0/2232146905 >> v2:172.21.15.158:3300/0 conn(0x7f69dc15dfe0 0x7f69dc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69db7fe640 1 -- 172.21.15.77:0/2232146905 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc0b3710 con 0x7f69dc0cf830 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69dbfff640 1 --2- 172.21.15.77:0/2232146905 >> v2:172.21.15.158:3300/0 conn(0x7f69dc15dfe0 0x7f69dc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.608+0000 7f69e0c2c640 1 --2- 172.21.15.77:0/2232146905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0d27f0 0x7f69dc0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.609+0000 7f69db7fe640 1 --2- 172.21.15.77:0/2232146905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0cf830 0x7f69dc0d22b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f69dc067990 tx=0x7f69d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc84e3b4fef23fe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.609+0000 7f69daffd640 1 -- 172.21.15.77:0/2232146905 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d0035020 con 0x7f69dc0cf830 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.609+0000 7f69daffd640 1 -- 172.21.15.77:0/2232146905 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d0031870 con 0x7f69dc0cf830 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.609+0000 7f69daffd640 1 -- 172.21.15.77:0/2232146905 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d0031c40 con 0x7f69dc0cf830 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 -- 172.21.15.77:0/2232146905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0cf830 msgr2=0x7f69dc0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 --2- 172.21.15.77:0/2232146905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0cf830 0x7f69dc0d22b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f69dc067990 tx=0x7f69d002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 -- 172.21.15.77:0/2232146905 shutdown_connections 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 --2- 172.21.15.77:0/2232146905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0d27f0 0x7f69dc0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 --2- 172.21.15.77:0/2232146905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0cf830 0x7f69dc0d22b0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 --2- 172.21.15.77:0/2232146905 >> v2:172.21.15.158:3300/0 conn(0x7f69dc15dfe0 0x7f69dc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 -- 172.21.15.77:0/2232146905 >> 172.21.15.77:0/2232146905 conn(0x7f69dc0d3cd0 msgr2=0x7f69dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 -- 172.21.15.77:0/2232146905 shutdown_connections 2024-03-06T07:24:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.610+0000 7f69e268f640 1 -- 172.21.15.77:0/2232146905 wait complete. 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.611+0000 7f69e268f640 1 Processor -- start 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.611+0000 7f69e268f640 1 -- start start 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e268f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0cf830 0x7f69dc0b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e268f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc0d27f0 0x7f69dc0b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e268f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc15dfe0 0x7f69dc1f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e268f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0d1df0 con 0x7f69dc15dfe0 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e268f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0d1c70 con 0x7f69dc0d27f0 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e268f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc15e3c0 con 0x7f69dc0cf830 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc15dfe0 0x7f69dc1f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc15dfe0 0x7f69dc1f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44370/0 (socket says 172.21.15.77:44370) 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 -- 172.21.15.77:0/3877366109 learned_addr learned my addr 172.21.15.77:0/3877366109 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69dbfff640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0cf830 0x7f69dc0b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69db7fe640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc0d27f0 0x7f69dc0b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 -- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0cf830 msgr2=0x7f69dc0b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0cf830 0x7f69dc0b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 -- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc0d27f0 msgr2=0x7f69dc0b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc0d27f0 0x7f69dc0b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 -- 172.21.15.77:0/3877366109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc1f4960 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69dbfff640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0cf830 0x7f69dc0b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e0c2c640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc15dfe0 0x7f69dc1f3630 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f69d400cef0 tx=0x7f69d400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69d8ff9640 1 -- 172.21.15.77:0/3877366109 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d401a070 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69d8ff9640 1 -- 172.21.15.77:0/3877366109 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d400d840 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69d8ff9640 1 -- 172.21.15.77:0/3877366109 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d400dc30 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc1f4c40 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.612+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc0b86b0 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.613+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc04f330 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.614+0000 7f69d8ff9640 1 -- 172.21.15.77:0/3877366109 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69d401d070 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.615+0000 7f69d8ff9640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c806b4d0 0x7f69c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.615+0000 7f69d8ff9640 1 -- 172.21.15.77:0/3877366109 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69d408a9a0 con 0x7f69dc15dfe0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.615+0000 7f69dbfff640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c806b4d0 0x7f69c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.615+0000 7f69dbfff640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c806b4d0 0x7f69c806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006fd0 tx=0x7f69cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.619+0000 7f69d8ff9640 1 -- 172.21.15.77:0/3877366109 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69d40598b0 con 0x7f69dc15dfe0 2024-03-06T07:24:46.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.831+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69dc0ce9b0 con 0x7f69dc15dfe0 2024-03-06T07:24:46.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.832+0000 7f69d8ff9640 1 -- 172.21.15.77:0/3877366109 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f69d40591b0 con 0x7f69dc15dfe0 2024-03-06T07:24:46.834 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:46.834 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:12.416501+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:08.975524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.729967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:46.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c806b4d0 msgr2=0x7f69c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c806b4d0 0x7f69c806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006fd0 tx=0x7f69cc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc15dfe0 msgr2=0x7f69dc1f3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc15dfe0 0x7f69dc1f3630 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f69d400cef0 tx=0x7f69d400a710 comp rx=0 tx=0).stop 2024-03-06T07:24:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 shutdown_connections 2024-03-06T07:24:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c806b4d0 0x7f69c806d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc15dfe0 0x7f69dc1f3630 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc0d27f0 0x7f69dc0b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 --2- 172.21.15.77:0/3877366109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0cf830 0x7f69dc0b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 >> 172.21.15.77:0/3877366109 conn(0x7f69dc0d3cd0 msgr2=0x7f69dc06bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 shutdown_connections 2024-03-06T07:24:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.834+0000 7f69e268f640 1 -- 172.21.15.77:0/3877366109 wait complete. 2024-03-06T07:24:46.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:46.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.948+0000 7f53acd93640 1 Processor -- start 2024-03-06T07:24:46.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.948+0000 7f53acd93640 1 -- start start 2024-03-06T07:24:46.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53acd93640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53a81444c0 0x7f53a806ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53acd93640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a80708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53acd93640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a806d370 con 0x7f53a8070e10 2024-03-06T07:24:46.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53acd93640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a81448a0 con 0x7f53a81444c0 2024-03-06T07:24:46.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53acd93640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8071260 con 0x7f53a815b2d0 2024-03-06T07:24:46.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f539ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a80708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53a659c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53a81444c0 0x7f53a806ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f539ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a80708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:53986/0 (socket says 172.21.15.77:53986) 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53a659c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53a81444c0 0x7f53a806ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:59038/0 (socket says 172.21.15.77:59038) 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53a6d9d640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f53a8070e10 0x7f53a815e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:36066/0 (socket says 172.21.15.77:36066) 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f539ffff640 1 -- 172.21.15.77:0/771894251 learned_addr learned my addr 172.21.15.77:0/771894251 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f539ffff640 1 -- 172.21.15.77:0/771894251 >> v2:172.21.15.158:3300/0 conn(0x7f53a81444c0 msgr2=0x7f53a806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f539ffff640 1 --2- 172.21.15.77:0/771894251 >> v2:172.21.15.158:3300/0 conn(0x7f53a81444c0 0x7f53a806ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f539ffff640 1 -- 172.21.15.77:0/771894251 >> v1:172.21.15.158:6789/0 conn(0x7f53a8070e10 legacy=0x7f53a815e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f539ffff640 1 -- 172.21.15.77:0/771894251 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a806d650 con 0x7f53a815b2d0 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.949+0000 7f53a659c640 1 --2- 172.21.15.77:0/771894251 >> v2:172.21.15.158:3300/0 conn(0x7f53a81444c0 0x7f53a806ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.950+0000 7f539ffff640 1 --2- 172.21.15.77:0/771894251 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a80708d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5390009870 tx=0x7f539002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbddf1d480eb0688 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.950+0000 7f53a5d9b640 1 -- 172.21.15.77:0/771894251 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390034070 con 0x7f53a815b2d0 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.950+0000 7f53a5d9b640 1 -- 172.21.15.77:0/771894251 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5390030840 con 0x7f53a815b2d0 2024-03-06T07:24:46.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.950+0000 7f53a5d9b640 1 -- 172.21.15.77:0/771894251 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390030c10 con 0x7f53a815b2d0 2024-03-06T07:24:46.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.951+0000 7f53acd93640 1 -- 172.21.15.77:0/771894251 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 msgr2=0x7f53a80708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.951+0000 7f53acd93640 1 --2- 172.21.15.77:0/771894251 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a80708d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5390009870 tx=0x7f539002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:46.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.951+0000 7f53acd93640 1 -- 172.21.15.77:0/771894251 shutdown_connections 2024-03-06T07:24:46.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.951+0000 7f53acd93640 1 --2- 172.21.15.77:0/771894251 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a80708d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.951+0000 7f53acd93640 1 --2- 172.21.15.77:0/771894251 >> v2:172.21.15.158:3300/0 conn(0x7f53a81444c0 0x7f53a806ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.951+0000 7f53acd93640 1 -- 172.21.15.77:0/771894251 >> 172.21.15.77:0/771894251 conn(0x7f53a80b1c80 msgr2=0x7f53a80cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.951+0000 7f53acd93640 1 -- 172.21.15.77:0/771894251 shutdown_connections 2024-03-06T07:24:46.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.951+0000 7f53acd93640 1 -- 172.21.15.77:0/771894251 wait complete. 2024-03-06T07:24:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.952+0000 7f53acd93640 1 Processor -- start 2024-03-06T07:24:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.952+0000 7f53acd93640 1 -- start start 2024-03-06T07:24:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f53acd93640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53a8070e10 0x7f53a815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f53acd93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53a81444c0 0x7f53a80ab260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f53acd93640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a81f1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f53acd93640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a806def0 con 0x7f53a81444c0 2024-03-06T07:24:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f53acd93640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a806dc10 con 0x7f53a8070e10 2024-03-06T07:24:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f53acd93640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a806cf40 con 0x7f53a815b2d0 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f539ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53a81444c0 0x7f53a80ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f539ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53a81444c0 0x7f53a80ab260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44384/0 (socket says 172.21.15.77:44384) 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f539ffff640 1 -- 172.21.15.77:0/1489745733 learned_addr learned my addr 172.21.15.77:0/1489745733 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f53a659c640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53a8070e10 0x7f53a815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.953+0000 7f53a6d9d640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a81f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539ffff640 1 -- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 msgr2=0x7f53a81f1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539ffff640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a81f1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539ffff640 1 -- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53a8070e10 msgr2=0x7f53a815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539ffff640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53a8070e10 0x7f53a815a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539ffff640 1 -- 172.21.15.77:0/1489745733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a81f2140 con 0x7f53a81444c0 2024-03-06T07:24:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f53a6d9d640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a81f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539ffff640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53a81444c0 0x7f53a80ab260 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f53900090b0 tx=0x7f53900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539effd640 1 -- 172.21.15.77:0/1489745733 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390034040 con 0x7f53a81444c0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539effd640 1 -- 172.21.15.77:0/1489745733 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539003ada0 con 0x7f53a81444c0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a81f2420 con 0x7f53a81444c0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f539effd640 1 -- 172.21.15.77:0/1489745733 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390044610 con 0x7f53a81444c0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.954+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a80aad40 con 0x7f53a81444c0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.955+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a804f330 con 0x7f53a81444c0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.956+0000 7f539effd640 1 -- 172.21.15.77:0/1489745733 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f539004b060 con 0x7f53a81444c0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.956+0000 7f539effd640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f538806b4d0 0x7f538806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.956+0000 7f539effd640 1 -- 172.21.15.77:0/1489745733 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53900ac180 con 0x7f53a81444c0 2024-03-06T07:24:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.956+0000 7f53a659c640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f538806b4d0 0x7f538806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.957+0000 7f53a659c640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f538806b4d0 0x7f538806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5398009560 tx=0x7f5398008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:46.961+0000 7f539effd640 1 -- 172.21.15.77:0/1489745733 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f539003f070 con 0x7f53a81444c0 2024-03-06T07:24:47.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.228+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f53a8070970 con 0x7f53a81444c0 2024-03-06T07:24:47.230 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:24:47.229+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.230+0000 7f539effd640 1 -- 172.21.15.77:0/1489745733 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f539007b010 con 0x7f53a81444c0 2024-03-06T07:24:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:47.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f538806b4d0 msgr2=0x7f538806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f538806b4d0 0x7f538806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5398009560 tx=0x7f5398008040 comp rx=0 tx=0).stop 2024-03-06T07:24:47.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53a81444c0 msgr2=0x7f53a80ab260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53a81444c0 0x7f53a80ab260 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f53900090b0 tx=0x7f53900046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 shutdown_connections 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f538806b4d0 0x7f538806d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53a81444c0 0x7f53a80ab260 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53a815b2d0 0x7f53a81f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 --2- 172.21.15.77:0/1489745733 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53a8070e10 0x7f53a815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 >> 172.21.15.77:0/1489745733 conn(0x7f53a80b1c80 msgr2=0x7f53a8142d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 shutdown_connections 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.233+0000 7f53acd93640 1 -- 172.21.15.77:0/1489745733 wait complete. 2024-03-06T07:24:47.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:47.277 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:24:47.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.351+0000 7fb636e9d640 1 Processor -- start 2024-03-06T07:24:47.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.351+0000 7fb636e9d640 1 -- start start 2024-03-06T07:24:47.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.352+0000 7fb636e9d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6301614d0 0x7fb6300d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.352+0000 7fb636e9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb6300d1910 0x7fb6300d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.352+0000 7fb636e9d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6301618b0 con 0x7fb6300d22e0 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.352+0000 7fb636e9d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6300cf400 con 0x7fb6300d1910 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.352+0000 7fb636e9d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6300d2710 con 0x7fb6301614d0 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.352+0000 7fb62ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb6300d1910 0x7fb6300d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.352+0000 7fb63543a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb6300d22e0 0x7fb6300d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41342/0 (socket says 172.21.15.77:41342) 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.352+0000 7fb63543a640 1 -- 172.21.15.77:0/2782191960 learned_addr learned my addr 172.21.15.77:0/2782191960 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb634c39640 1 --2- 172.21.15.77:0/2782191960 >> v2:172.21.15.158:3300/0 conn(0x7fb6301614d0 0x7fb6300d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62ffff640 1 -- 172.21.15.77:0/2782191960 >> v2:172.21.15.158:3300/0 conn(0x7fb6301614d0 msgr2=0x7fb6300d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62ffff640 1 --2- 172.21.15.77:0/2782191960 >> v2:172.21.15.158:3300/0 conn(0x7fb6301614d0 0x7fb6300d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62ffff640 1 -- 172.21.15.77:0/2782191960 >> v1:172.21.15.77:6789/0 conn(0x7fb6300d22e0 legacy=0x7fb6300d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62f7fe640 1 -- 172.21.15.77:0/2782191960 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374643269 0 0) 0x7fb6301618b0 con 0x7fb6300d22e0 2024-03-06T07:24:47.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62ffff640 1 -- 172.21.15.77:0/2782191960 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300cf6a0 con 0x7fb6300d1910 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62ffff640 1 --2- 172.21.15.77:0/2782191960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb6300d1da0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb63008c470 tx=0x7fb62402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ab1d0d6b58186e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62f7fe640 1 -- 172.21.15.77:0/2782191960 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb624035020 con 0x7fb6300d1910 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62f7fe640 1 -- 172.21.15.77:0/2782191960 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb624031830 con 0x7fb6300d1910 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.353+0000 7fb62f7fe640 1 -- 172.21.15.77:0/2782191960 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb624031c00 con 0x7fb6300d1910 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.354+0000 7fb636e9d640 1 -- 172.21.15.77:0/2782191960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 msgr2=0x7fb6300d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.354+0000 7fb636e9d640 1 --2- 172.21.15.77:0/2782191960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb6300d1da0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb63008c470 tx=0x7fb62402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.354+0000 7fb636e9d640 1 -- 172.21.15.77:0/2782191960 shutdown_connections 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.354+0000 7fb636e9d640 1 --2- 172.21.15.77:0/2782191960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb6300d1da0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.354+0000 7fb636e9d640 1 --2- 172.21.15.77:0/2782191960 >> v2:172.21.15.158:3300/0 conn(0x7fb6301614d0 0x7fb6300d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.354+0000 7fb636e9d640 1 -- 172.21.15.77:0/2782191960 >> 172.21.15.77:0/2782191960 conn(0x7fb6300d3d40 msgr2=0x7fb6300d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.354+0000 7fb636e9d640 1 -- 172.21.15.77:0/2782191960 shutdown_connections 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.354+0000 7fb636e9d640 1 -- 172.21.15.77:0/2782191960 wait complete. 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.355+0000 7fb636e9d640 1 Processor -- start 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.355+0000 7fb636e9d640 1 -- start start 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb636e9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb63015be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb636e9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6300d22e0 0x7fb6300b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb636e9d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6301614d0 0x7fb63015dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb636e9d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6300b0910 con 0x7fb6300d1910 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb636e9d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6300cfc60 con 0x7fb6301614d0 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb636e9d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6300b1e40 con 0x7fb6300d22e0 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb634c39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb63015be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb634c39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb63015be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44414/0 (socket says 172.21.15.77:44414) 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb634c39640 1 -- 172.21.15.77:0/3036571810 learned_addr learned my addr 172.21.15.77:0/3036571810 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb634c39640 1 -- 172.21.15.77:0/3036571810 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6300d22e0 msgr2=0x7fb6300b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb634c39640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6300d22e0 0x7fb6300b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb634c39640 1 -- 172.21.15.77:0/3036571810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6301614d0 msgr2=0x7fb63015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb634c39640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6301614d0 0x7fb63015dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.356+0000 7fb634c39640 1 -- 172.21.15.77:0/3036571810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300cb1d0 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.357+0000 7fb634c39640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb63015be80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb62000c820 tx=0x7fb62000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.357+0000 7fb62d7fa640 1 -- 172.21.15.77:0/3036571810 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620015070 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.357+0000 7fb62d7fa640 1 -- 172.21.15.77:0/3036571810 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb620004020 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.357+0000 7fb62d7fa640 1 -- 172.21.15.77:0/3036571810 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6200043f0 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.357+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6300cb4b0 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.357+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6300cba70 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.358+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6300d1da0 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.358+0000 7fb62d7fa640 1 -- 172.21.15.77:0/3036571810 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb620007830 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.359+0000 7fb62d7fa640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb61c06b4d0 0x7fb61c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.359+0000 7fb62d7fa640 1 -- 172.21.15.77:0/3036571810 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb620089ff0 con 0x7fb6300d1910 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.359+0000 7fb62ffff640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb61c06b4d0 0x7fb61c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.359+0000 7fb62ffff640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb61c06b4d0 0x7fb61c06d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb624036000 tx=0x7fb624030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.363+0000 7fb62d7fa640 1 -- 172.21.15.77:0/3036571810 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb620058ea0 con 0x7fb6300d1910 2024-03-06T07:24:47.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:47.575+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb630068ff0 con 0x7fb6300d1910 2024-03-06T07:24:48.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.235+0000 7fb62d7fa640 1 -- 172.21.15.77:0/3036571810 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb6200587a0 con 0x7fb6300d1910 2024-03-06T07:24:48.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb61c06b4d0 msgr2=0x7fb61c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb61c06b4d0 0x7fb61c06d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb624036000 tx=0x7fb624030000 comp rx=0 tx=0).stop 2024-03-06T07:24:48.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 msgr2=0x7fb63015be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb63015be80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb62000c820 tx=0x7fb62000cce0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 shutdown_connections 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6301614d0 0x7fb63015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6300d22e0 0x7fb6300b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb61c06b4d0 0x7fb61c06d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 --2- 172.21.15.77:0/3036571810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6300d1910 0x7fb63015be80 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.237+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 >> 172.21.15.77:0/3036571810 conn(0x7fb6300d3d40 msgr2=0x7fb63006b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.238+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 shutdown_connections 2024-03-06T07:24:48.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.238+0000 7fb636e9d640 1 -- 172.21.15.77:0/3036571810 wait complete. 2024-03-06T07:24:48.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:48.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.399+0000 7f71d92c2640 1 Processor -- start 2024-03-06T07:24:48.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.399+0000 7f71d92c2640 1 -- start start 2024-03-06T07:24:48.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d92c2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71d40d4630 0x7f71d40d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d92c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71d40d2820 0x7f71d40cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d92c2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d415e380 con 0x7f71d415dfa0 2024-03-06T07:24:48.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d92c2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d40d2c70 con 0x7f71d40d2820 2024-03-06T07:24:48.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d92c2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d40d2100 con 0x7f71d40d4630 2024-03-06T07:24:48.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71d40d2820 0x7f71d40cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71d40d2820 0x7f71d40cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44424/0 (socket says 172.21.15.77:44424) 2024-03-06T07:24:48.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d37fe640 1 -- 172.21.15.77:0/2689435057 learned_addr learned my addr 172.21.15.77:0/2689435057 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:48.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71cbfff640 1 --2- 172.21.15.77:0/2689435057 >> v2:172.21.15.158:3300/0 conn(0x7f71d40d4630 0x7f71d40d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d37fe640 1 -- 172.21.15.77:0/2689435057 >> v2:172.21.15.158:3300/0 conn(0x7f71d40d4630 msgr2=0x7f71d40d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d37fe640 1 --2- 172.21.15.77:0/2689435057 >> v2:172.21.15.158:3300/0 conn(0x7f71d40d4630 0x7f71d40d22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d37fe640 1 -- 172.21.15.77:0/2689435057 >> v1:172.21.15.158:6789/0 conn(0x7f71d415dfa0 legacy=0x7f71d40d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71d37fe640 1 -- 172.21.15.77:0/2689435057 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d40cf470 con 0x7f71d40d2820 2024-03-06T07:24:48.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.400+0000 7f71cbfff640 1 --2- 172.21.15.77:0/2689435057 >> v2:172.21.15.158:3300/0 conn(0x7f71d40d4630 0x7f71d40d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:48.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.401+0000 7f71d37fe640 1 --2- 172.21.15.77:0/2689435057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d40d2820 0x7f71d40cb190 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f71d40b6910 tx=0x7f71cc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96909928842e6531 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.401+0000 7f71d27fc640 1 -- 172.21.15.77:0/2689435057 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc02ca80 con 0x7f71d40d2820 2024-03-06T07:24:48.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.401+0000 7f71d27fc640 1 -- 172.21.15.77:0/2689435057 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc02cc20 con 0x7f71d40d2820 2024-03-06T07:24:48.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.401+0000 7f71d27fc640 1 -- 172.21.15.77:0/2689435057 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc0343f0 con 0x7f71d40d2820 2024-03-06T07:24:48.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.402+0000 7f71d92c2640 1 -- 172.21.15.77:0/2689435057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d40d2820 msgr2=0x7f71d40cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.402+0000 7f71d92c2640 1 --2- 172.21.15.77:0/2689435057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d40d2820 0x7f71d40cb190 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f71d40b6910 tx=0x7f71cc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.402+0000 7f71d92c2640 1 -- 172.21.15.77:0/2689435057 shutdown_connections 2024-03-06T07:24:48.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.402+0000 7f71d92c2640 1 --2- 172.21.15.77:0/2689435057 >> v2:172.21.15.158:3300/0 conn(0x7f71d40d4630 0x7f71d40d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.402+0000 7f71d92c2640 1 --2- 172.21.15.77:0/2689435057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d40d2820 0x7f71d40cb190 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.402+0000 7f71d92c2640 1 -- 172.21.15.77:0/2689435057 >> 172.21.15.77:0/2689435057 conn(0x7f71d409d000 msgr2=0x7f71d40d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.402+0000 7f71d92c2640 1 -- 172.21.15.77:0/2689435057 shutdown_connections 2024-03-06T07:24:48.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.402+0000 7f71d92c2640 1 -- 172.21.15.77:0/2689435057 wait complete. 2024-03-06T07:24:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.403+0000 7f71d92c2640 1 Processor -- start 2024-03-06T07:24:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.403+0000 7f71d92c2640 1 -- start start 2024-03-06T07:24:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d92c2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71d40d2820 0x7f71d415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d92c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71d40d4630 0x7f71d40c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d92c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d415dfa0 0x7f71d40c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d92c2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40cb6d0 con 0x7f71d415dfa0 2024-03-06T07:24:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d92c2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40cfa30 con 0x7f71d40d2820 2024-03-06T07:24:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d92c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40cb850 con 0x7f71d40d4630 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d415dfa0 0x7f71d40c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d415dfa0 0x7f71d40c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44432/0 (socket says 172.21.15.77:44432) 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 -- 172.21.15.77:0/3276676295 learned_addr learned my addr 172.21.15.77:0/3276676295 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71cbfff640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71d40d4630 0x7f71d40c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d2ffd640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71d40d2820 0x7f71d415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 -- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71d40d4630 msgr2=0x7f71d40c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71d40d4630 0x7f71d40c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 -- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71d40d2820 msgr2=0x7f71d415c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71d40d2820 0x7f71d415c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 -- 172.21.15.77:0/3276676295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d40b6a60 con 0x7f71d415dfa0 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d2ffd640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71d40d2820 0x7f71d415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d37fe640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d415dfa0 0x7f71d40c9f60 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009f40 tx=0x7f71cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71cb7fe640 1 -- 172.21.15.77:0/3276676295 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc043020 con 0x7f71d415dfa0 2024-03-06T07:24:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71cb7fe640 1 -- 172.21.15.77:0/3276676295 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc03e450 con 0x7f71d415dfa0 2024-03-06T07:24:48.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d40b6d40 con 0x7f71d415dfa0 2024-03-06T07:24:48.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71cb7fe640 1 -- 172.21.15.77:0/3276676295 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc009510 con 0x7f71d415dfa0 2024-03-06T07:24:48.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.404+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d40b7300 con 0x7f71d415dfa0 2024-03-06T07:24:48.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.405+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d404f330 con 0x7f71d415dfa0 2024-03-06T07:24:48.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.406+0000 7f71cb7fe640 1 -- 172.21.15.77:0/3276676295 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71cc046020 con 0x7f71d415dfa0 2024-03-06T07:24:48.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.406+0000 7f71cb7fe640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71b406b4d0 0x7f71b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.406+0000 7f71d2ffd640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71b406b4d0 0x7f71b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.406+0000 7f71cb7fe640 1 -- 172.21.15.77:0/3276676295 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f71cc0aca70 con 0x7f71d415dfa0 2024-03-06T07:24:48.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.407+0000 7f71d2ffd640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71b406b4d0 0x7f71b406d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f71c4001660 tx=0x7f71c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.411+0000 7f71cb7fe640 1 -- 172.21.15.77:0/3276676295 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71cc04d030 con 0x7f71d415dfa0 2024-03-06T07:24:48.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:48.586+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f71d4160c30 con 0x7f71d415dfa0 2024-03-06T07:24:49.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.236+0000 7f71cb7fe640 1 -- 172.21.15.77:0/3276676295 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) v1 ==== 181+0+0 (secure 0 0 0) 0x7f71cc07b890 con 0x7f71d415dfa0 2024-03-06T07:24:49.238 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:49.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.239+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71b406b4d0 msgr2=0x7f71b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.239+0000 7f71d92c2640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71b406b4d0 0x7f71b406d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f71c4001660 tx=0x7f71c4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.239+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d415dfa0 msgr2=0x7f71d40c9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.239+0000 7f71d92c2640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d415dfa0 0x7f71d40c9f60 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009f40 tx=0x7f71cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.240+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 shutdown_connections 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.240+0000 7f71d92c2640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71b406b4d0 0x7f71b406d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.240+0000 7f71d92c2640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d415dfa0 0x7f71d40c9f60 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.240+0000 7f71d92c2640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71d40d4630 0x7f71d40c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.240+0000 7f71d92c2640 1 --2- 172.21.15.77:0/3276676295 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71d40d2820 0x7f71d415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.240+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 >> 172.21.15.77:0/3276676295 conn(0x7f71d409d000 msgr2=0x7f71d406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.240+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 shutdown_connections 2024-03-06T07:24:49.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.240+0000 7f71d92c2640 1 -- 172.21.15.77:0/3276676295 wait complete. 2024-03-06T07:24:49.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:49.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.360+0000 7f7387ac2640 1 Processor -- start 2024-03-06T07:24:49.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.360+0000 7f7387ac2640 1 -- start start 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f7387ac2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f73800d4630 0x7f73800d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f7387ac2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73800d2820 0x7f73800cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f7387ac2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738015e380 con 0x7f738015dfa0 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f7387ac2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73800d2c70 con 0x7f73800d2820 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f7387ac2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73800d2100 con 0x7f73800d4630 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f738605f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73800d2820 0x7f73800cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f738605f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73800d2820 0x7f73800cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38854/0 (socket says 172.21.15.77:38854) 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f738605f640 1 -- 172.21.15.77:0/2086153825 learned_addr learned my addr 172.21.15.77:0/2086153825 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f738605f640 1 -- 172.21.15.77:0/2086153825 >> v2:172.21.15.158:3300/0 conn(0x7f73800d4630 msgr2=0x7f73800d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f738605f640 1 --2- 172.21.15.77:0/2086153825 >> v2:172.21.15.158:3300/0 conn(0x7f73800d4630 0x7f73800d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f738605f640 1 -- 172.21.15.77:0/2086153825 >> v1:172.21.15.158:6789/0 conn(0x7f738015dfa0 legacy=0x7f73800d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.361+0000 7f738605f640 1 -- 172.21.15.77:0/2086153825 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73800cf470 con 0x7f73800d2820 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f738605f640 1 --2- 172.21.15.77:0/2086153825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d2820 0x7f73800cb190 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f73800b6910 tx=0x7f737802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0cf29953e2c9a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f738485c640 1 -- 172.21.15.77:0/2086153825 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737802c9f0 con 0x7f73800d2820 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f738485c640 1 -- 172.21.15.77:0/2086153825 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737802cb90 con 0x7f73800d2820 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f738485c640 1 -- 172.21.15.77:0/2086153825 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73780343f0 con 0x7f73800d2820 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f7387ac2640 1 -- 172.21.15.77:0/2086153825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d2820 msgr2=0x7f73800cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f7387ac2640 1 --2- 172.21.15.77:0/2086153825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d2820 0x7f73800cb190 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f73800b6910 tx=0x7f737802be50 comp rx=0 tx=0).stop 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f7387ac2640 1 -- 172.21.15.77:0/2086153825 shutdown_connections 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f7387ac2640 1 --2- 172.21.15.77:0/2086153825 >> v2:172.21.15.158:3300/0 conn(0x7f73800d4630 0x7f73800d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f7387ac2640 1 --2- 172.21.15.77:0/2086153825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d2820 0x7f73800cb190 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f7387ac2640 1 -- 172.21.15.77:0/2086153825 >> 172.21.15.77:0/2086153825 conn(0x7f738009d000 msgr2=0x7f73800d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.362+0000 7f7387ac2640 1 -- 172.21.15.77:0/2086153825 shutdown_connections 2024-03-06T07:24:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.363+0000 7f7387ac2640 1 -- 172.21.15.77:0/2086153825 wait complete. 2024-03-06T07:24:49.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f7387ac2640 1 Processor -- start 2024-03-06T07:24:49.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f7387ac2640 1 -- start start 2024-03-06T07:24:49.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f7387ac2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73800d2820 0x7f73801501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f7387ac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d4630 0x7f73800b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f7387ac2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f738015dfa0 0x7f738006c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f7387ac2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800cb6d0 con 0x7f73800d4630 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f7387ac2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800cfa30 con 0x7f73800d2820 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f7387ac2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73800cb850 con 0x7f738015dfa0 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f738505d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d4630 0x7f73800b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f738505d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d4630 0x7f73800b0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38862/0 (socket says 172.21.15.77:38862) 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.364+0000 7f738505d640 1 -- 172.21.15.77:0/3813646252 learned_addr learned my addr 172.21.15.77:0/3813646252 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f738585e640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73800d2820 0x7f73801501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f738605f640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f738015dfa0 0x7f738006c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f738505d640 1 -- 172.21.15.77:0/3813646252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f738015dfa0 msgr2=0x7f738006c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f738505d640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f738015dfa0 0x7f738006c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f738505d640 1 -- 172.21.15.77:0/3813646252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73800d2820 msgr2=0x7f73801501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f738505d640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73800d2820 0x7f73801501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f738505d640 1 -- 172.21.15.77:0/3813646252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73800b1050 con 0x7f73800d4630 2024-03-06T07:24:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f738505d640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d4630 0x7f73800b0b10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f737c00b450 tx=0x7f737c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f73767fc640 1 -- 172.21.15.77:0/3813646252 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c015020 con 0x7f73800d4630 2024-03-06T07:24:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f73767fc640 1 -- 172.21.15.77:0/3813646252 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c011890 con 0x7f73800d4630 2024-03-06T07:24:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73800b3d30 con 0x7f73800d4630 2024-03-06T07:24:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f73767fc640 1 -- 172.21.15.77:0/3813646252 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c011c80 con 0x7f73800d4630 2024-03-06T07:24:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.365+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73800c7a30 con 0x7f73800d4630 2024-03-06T07:24:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.366+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738004f330 con 0x7f73800d4630 2024-03-06T07:24:49.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.366+0000 7f73767fc640 1 -- 172.21.15.77:0/3813646252 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f737c018020 con 0x7f73800d4630 2024-03-06T07:24:49.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.367+0000 7f73767fc640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f736406b4d0 0x7f736406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.367+0000 7f73767fc640 1 -- 172.21.15.77:0/3813646252 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f737c08a2c0 con 0x7f73800d4630 2024-03-06T07:24:49.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.367+0000 7f738585e640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f736406b4d0 0x7f736406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.367+0000 7f738585e640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f736406b4d0 0x7f736406d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7370006fd0 tx=0x7f7370008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.371+0000 7f73767fc640 1 -- 172.21.15.77:0/3813646252 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f737c01f020 con 0x7f73800d4630 2024-03-06T07:24:49.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:49.542+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f73800d2560 con 0x7f73800d4630 2024-03-06T07:24:50.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.241+0000 7f73767fc640 1 -- 172.21.15.77:0/3813646252 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) v1 ==== 169+0+0 (secure 0 0 0) 0x7f737c00d040 con 0x7f73800d4630 2024-03-06T07:24:50.243 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:50.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f736406b4d0 msgr2=0x7f736406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f736406b4d0 0x7f736406d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7370006fd0 tx=0x7f7370008040 comp rx=0 tx=0).stop 2024-03-06T07:24:50.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d4630 msgr2=0x7f73800b0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d4630 0x7f73800b0b10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f737c00b450 tx=0x7f737c00b910 comp rx=0 tx=0).stop 2024-03-06T07:24:50.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 shutdown_connections 2024-03-06T07:24:50.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f736406b4d0 0x7f736406d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73800d4630 0x7f73800b0b10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f738015dfa0 0x7f738006c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 --2- 172.21.15.77:0/3813646252 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73800d2820 0x7f73801501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 >> 172.21.15.77:0/3813646252 conn(0x7f738009d000 msgr2=0x7f73800ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 shutdown_connections 2024-03-06T07:24:50.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.244+0000 7f7387ac2640 1 -- 172.21.15.77:0/3813646252 wait complete. 2024-03-06T07:24:50.262 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:24:50.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.367+0000 7f0c6e7f7640 1 Processor -- start 2024-03-06T07:24:50.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.367+0000 7f0c6e7f7640 1 -- start start 2024-03-06T07:24:50.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c6e7f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0c6815f4f0 0x7f0c68066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c6e7f7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0c680b0db0 0x7f0c680ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c6e7f7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6806b050 con 0x7f0c680ce580 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c6e7f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6806b2c0 con 0x7f0c6815f4f0 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c6e7f7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c680b0890 con 0x7f0c680b0db0 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c67fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0c6815f4f0 0x7f0c68066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c6cd94640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0c680ce580 0x7f0c68072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39818/0 (socket says 172.21.15.77:39818) 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c6cd94640 1 -- 172.21.15.77:0/1273601538 learned_addr learned my addr 172.21.15.77:0/1273601538 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c67fff640 1 -- 172.21.15.77:0/1273601538 >> v2:172.21.15.158:3300/0 conn(0x7f0c680b0db0 msgr2=0x7f0c680ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c67fff640 1 --2- 172.21.15.77:0/1273601538 >> v2:172.21.15.158:3300/0 conn(0x7f0c680b0db0 0x7f0c680ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.368+0000 7f0c67fff640 1 -- 172.21.15.77:0/1273601538 >> v1:172.21.15.77:6789/0 conn(0x7f0c680ce580 legacy=0x7f0c68072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.369+0000 7f0c66ffd640 1 -- 172.21.15.77:0/1273601538 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038635183 0 0) 0x7f0c6806b050 con 0x7f0c680ce580 2024-03-06T07:24:50.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.369+0000 7f0c67fff640 1 -- 172.21.15.77:0/1273601538 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680672e0 con 0x7f0c6815f4f0 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.369+0000 7f0c67fff640 1 --2- 172.21.15.77:0/1273601538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68066d00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0c680d11d0 tx=0x7f0c5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514dbfdbc9a86397 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.369+0000 7f0c66ffd640 1 -- 172.21.15.77:0/1273601538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58035020 con 0x7f0c6815f4f0 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.369+0000 7f0c66ffd640 1 -- 172.21.15.77:0/1273601538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c58031830 con 0x7f0c6815f4f0 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.369+0000 7f0c66ffd640 1 -- 172.21.15.77:0/1273601538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58031c00 con 0x7f0c6815f4f0 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.370+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/1273601538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 msgr2=0x7f0c68066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.370+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/1273601538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68066d00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0c680d11d0 tx=0x7f0c5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.370+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/1273601538 shutdown_connections 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.370+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/1273601538 >> v2:172.21.15.158:3300/0 conn(0x7f0c680b0db0 0x7f0c680ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.370+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/1273601538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68066d00 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.370+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/1273601538 >> 172.21.15.77:0/1273601538 conn(0x7f0c680b1d70 msgr2=0x7f0c680b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.370+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/1273601538 shutdown_connections 2024-03-06T07:24:50.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.370+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/1273601538 wait complete. 2024-03-06T07:24:50.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.371+0000 7f0c6e7f7640 1 Processor -- start 2024-03-06T07:24:50.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.371+0000 7f0c6e7f7640 1 -- start start 2024-03-06T07:24:50.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6e7f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0c680b0db0 0x7f0c681533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6e7f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c680ce580 0x7f0c681465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6e7f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6e7f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6815f8d0 con 0x7f0c6815f4f0 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6e7f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6806b5a0 con 0x7f0c680ce580 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6e7f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c680aeab0 con 0x7f0c680b0db0 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6cd94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6cd94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68146b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38886/0 (socket says 172.21.15.77:38886) 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6cd94640 1 -- 172.21.15.77:0/2755482600 learned_addr learned my addr 172.21.15.77:0/2755482600 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c677fe640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c680ce580 0x7f0c681465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6cd94640 1 -- 172.21.15.77:0/2755482600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0c680b0db0 msgr2=0x7f0c681533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6cd94640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0c680b0db0 0x7f0c681533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6cd94640 1 -- 172.21.15.77:0/2755482600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c680ce580 msgr2=0x7f0c681465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6cd94640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c680ce580 0x7f0c681465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.372+0000 7f0c6cd94640 1 -- 172.21.15.77:0/2755482600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680adea0 con 0x7f0c6815f4f0 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.373+0000 7f0c677fe640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c680ce580 0x7f0c681465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.373+0000 7f0c6cd94640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68146b10 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000c7d0 tx=0x7f0c6000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.373+0000 7f0c64ff9640 1 -- 172.21.15.77:0/2755482600 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60018070 con 0x7f0c6815f4f0 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.373+0000 7f0c64ff9640 1 -- 172.21.15.77:0/2755482600 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6000a640 con 0x7f0c6815f4f0 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.373+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c680ae180 con 0x7f0c6815f4f0 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.373+0000 7f0c64ff9640 1 -- 172.21.15.77:0/2755482600 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6000aa70 con 0x7f0c6815f4f0 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.373+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c680ae740 con 0x7f0c6815f4f0 2024-03-06T07:24:50.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.374+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6809e3d0 con 0x7f0c6815f4f0 2024-03-06T07:24:50.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.374+0000 7f0c64ff9640 1 -- 172.21.15.77:0/2755482600 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c6001b070 con 0x7f0c6815f4f0 2024-03-06T07:24:50.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.375+0000 7f0c64ff9640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0c5406b4d0 0x7f0c5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.375+0000 7f0c67fff640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0c5406b4d0 0x7f0c5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.375+0000 7f0c64ff9640 1 -- 172.21.15.77:0/2755482600 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0c600894e0 con 0x7f0c6815f4f0 2024-03-06T07:24:50.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.375+0000 7f0c67fff640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0c5406b4d0 0x7f0c5406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0c58036000 tx=0x7f0c58030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.379+0000 7f0c64ff9640 1 -- 172.21.15.77:0/2755482600 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c60058660 con 0x7f0c6815f4f0 2024-03-06T07:24:50.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.554+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f0c6806cd40 con 0x7f0c6815f4f0 2024-03-06T07:24:50.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.557+0000 7f0c64ff9640 1 -- 172.21.15.77:0/2755482600 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v19) v1 ==== 108+0+0 (secure 0 0 0) 0x7f0c60057f60 con 0x7f0c6815f4f0 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0c5406b4d0 msgr2=0x7f0c5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0c5406b4d0 0x7f0c5406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0c58036000 tx=0x7f0c58030000 comp rx=0 tx=0).stop 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 msgr2=0x7f0c68146b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68146b10 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000c7d0 tx=0x7f0c6000cc90 comp rx=0 tx=0).stop 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 shutdown_connections 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0c5406b4d0 0x7f0c5406d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c6815f4f0 0x7f0c68146b10 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0c680ce580 0x7f0c681465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 --2- 172.21.15.77:0/2755482600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0c680b0db0 0x7f0c681533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 >> 172.21.15.77:0/2755482600 conn(0x7f0c680b1d70 msgr2=0x7f0c6806dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 shutdown_connections 2024-03-06T07:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.559+0000 7f0c6e7f7640 1 -- 172.21.15.77:0/2755482600 wait complete. 2024-03-06T07:24:50.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:24:50.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.672+0000 7fc22ef56640 1 Processor -- start 2024-03-06T07:24:50.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.672+0000 7fc22ef56640 1 -- start start 2024-03-06T07:24:50.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.672+0000 7fc22ef56640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc22815a220 0x7fc2280cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.672+0000 7fc22ef56640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2280b3320 con 0x7fc2280712d0 2024-03-06T07:24:50.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.672+0000 7fc22ef56640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22815ec80 con 0x7fc22806da00 2024-03-06T07:24:50.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.672+0000 7fc22ef56640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2280cf610 con 0x7fc22815a220 2024-03-06T07:24:50.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc227fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc2280712d0 0x7fc22815de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39834/0 (socket says 172.21.15.77:39834) 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc227fff640 1 -- 172.21.15.77:0/890632157 learned_addr learned my addr 172.21.15.77:0/890632157 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc22ccf2640 1 --2- 172.21.15.77:0/890632157 >> v2:172.21.15.158:3300/0 conn(0x7fc22815a220 0x7fc2280cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812417279 0 0) 0x7fc2280b3320 con 0x7fc2280712d0 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc218004630 con 0x7fc2280712d0 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3949204499 0 0) 0x7fc218004630 con 0x7fc2280712d0 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 >> v2:172.21.15.158:3300/0 conn(0x7fc22815a220 msgr2=0x7fc2280cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 --2- 172.21.15.77:0/890632157 >> v2:172.21.15.158:3300/0 conn(0x7fc22815a220 0x7fc2280cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 >> v1:172.21.15.158:6789/0 conn(0x7fc22806da00 legacy=0x7fc22815e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2280728d0 con 0x7fc2280712d0 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fc214001670 con 0x7fc2280712d0 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 >> v1:172.21.15.77:6789/0 conn(0x7fc2280712d0 legacy=0x7fc22815de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc218004ac0 0x7fc218004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc218005910 0x7fc218005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2180062d0 0x7fc21800cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2280cf610 con 0x7fc218004ac0 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21800d270 con 0x7fc2180062d0 2024-03-06T07:24:50.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21800d470 con 0x7fc218005910 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc22ccf2640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc218004ac0 0x7fc218004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc227fff640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc218005910 0x7fc218005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.673+0000 7fc22d4f3640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2180062d0 0x7fc21800cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc22ccf2640 1 -- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc218005910 msgr2=0x7fc218005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc22ccf2640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc218005910 0x7fc218005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc22ccf2640 1 -- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2180062d0 msgr2=0x7fc21800cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc22ccf2640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2180062d0 0x7fc21800cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc22ccf2640 1 -- 172.21.15.77:0/890632157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc218004630 con 0x7fc218004ac0 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc22d4f3640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2180062d0 0x7fc21800cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc227fff640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc218005910 0x7fc218005d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc22d4f3640 1 -- 172.21.15.77:0/890632157 reap_dead start 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc22ccf2640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc218004ac0 0x7fc218004ee0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009ac0 tx=0x7fc218006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33999f830708dd18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c039070 con 0x7fc218004ac0 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c02c6b0 con 0x7fc218004ac0 2024-03-06T07:24:50.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.674+0000 7fc2277fe640 1 -- 172.21.15.77:0/890632157 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c009690 con 0x7fc218004ac0 2024-03-06T07:24:50.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.675+0000 7fc22ef56640 1 -- 172.21.15.77:0/890632157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc218004ac0 msgr2=0x7fc218004ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.675+0000 7fc22ef56640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc218004ac0 0x7fc218004ee0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009ac0 tx=0x7fc218006940 comp rx=0 tx=0).stop 2024-03-06T07:24:50.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.675+0000 7fc22ef56640 1 -- 172.21.15.77:0/890632157 shutdown_connections 2024-03-06T07:24:50.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.675+0000 7fc22ef56640 1 --2- 172.21.15.77:0/890632157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc218004ac0 0x7fc218004ee0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.675+0000 7fc22ef56640 1 -- 172.21.15.77:0/890632157 >> 172.21.15.77:0/890632157 conn(0x7fc22806b190 msgr2=0x7fc22806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.675+0000 7fc22ef56640 1 -- 172.21.15.77:0/890632157 shutdown_connections 2024-03-06T07:24:50.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.675+0000 7fc22ef56640 1 -- 172.21.15.77:0/890632157 wait complete. 2024-03-06T07:24:50.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.676+0000 7fc22ef56640 1 Processor -- start 2024-03-06T07:24:50.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.676+0000 7fc22ef56640 1 -- start start 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22ef56640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc218004ac0 0x7fc22814b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22ef56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22815afc0 0x7fc2280a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22ef56640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2280a9990 0x7fc2280adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22ef56640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc228072030 con 0x7fc22815afc0 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22ef56640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc228072bb0 con 0x7fc218004ac0 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22ef56640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc228072530 con 0x7fc2280a9990 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc227fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22815afc0 0x7fc2280a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc227fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22815afc0 0x7fc2280a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38906/0 (socket says 172.21.15.77:38906) 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc227fff640 1 -- 172.21.15.77:0/2956204617 learned_addr learned my addr 172.21.15.77:0/2956204617 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22ccf2640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc218004ac0 0x7fc22814b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22d4f3640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2280a9990 0x7fc2280adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc227fff640 1 -- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2280a9990 msgr2=0x7fc2280adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc227fff640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2280a9990 0x7fc2280adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc227fff640 1 -- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc218004ac0 msgr2=0x7fc22814b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc227fff640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc218004ac0 0x7fc22814b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc227fff640 1 -- 172.21.15.77:0/2956204617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2280aca90 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.677+0000 7fc22d4f3640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2280a9990 0x7fc2280adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.678+0000 7fc227fff640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22815afc0 0x7fc2280a9450 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc21400e680 tx=0x7fc21400eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.678+0000 7fc2257fa640 1 -- 172.21.15.77:0/2956204617 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21400c6c0 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.678+0000 7fc2257fa640 1 -- 172.21.15.77:0/2956204617 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc214018070 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.678+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2280acd70 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.678+0000 7fc2257fa640 1 -- 172.21.15.77:0/2956204617 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21400ca80 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.678+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2280ad330 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.679+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2280732b0 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.679+0000 7fc2257fa640 1 -- 172.21.15.77:0/2956204617 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc21400cc20 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.679+0000 7fc2257fa640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.680+0000 7fc2257fa640 1 -- 172.21.15.77:0/2956204617 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc214089cc0 con 0x7fc22815afc0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.680+0000 7fc22ccf2640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.680+0000 7fc22ccf2640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc21c036000 tx=0x7fc21c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.684+0000 7fc2257fa640 1 -- 172.21.15.77:0/2956204617 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc214058e40 con 0x7fc22815afc0 2024-03-06T07:24:50.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.855+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7fc228068ff0 con 0x7fc22815afc0 2024-03-06T07:24:50.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.860+0000 7fc2257fa640 1 -- 172.21.15.77:0/2956204617 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v20) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc214058740 con 0x7fc22815afc0 2024-03-06T07:24:50.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 msgr2=0x7fc20c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc21c036000 tx=0x7fc21c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22815afc0 msgr2=0x7fc2280a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22815afc0 0x7fc2280a9450 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc21400e680 tx=0x7fc21400eb40 comp rx=0 tx=0).stop 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 shutdown_connections 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22815afc0 0x7fc2280a9450 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2280a9990 0x7fc2280adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 --2- 172.21.15.77:0/2956204617 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc218004ac0 0x7fc22814b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 >> 172.21.15.77:0/2956204617 conn(0x7fc22806b190 msgr2=0x7fc2280b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 shutdown_connections 2024-03-06T07:24:50.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.862+0000 7fc22ef56640 1 -- 172.21.15.77:0/2956204617 wait complete. 2024-03-06T07:24:50.876 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:50.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.976+0000 7f86ea0fb640 1 Processor -- start 2024-03-06T07:24:50.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.976+0000 7f86ea0fb640 1 -- start start 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.976+0000 7f86ea0fb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86e415f4f0 0x7f86e4066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.976+0000 7f86ea0fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86e40b0db0 0x7f86e40ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.976+0000 7f86ea0fb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e406b050 con 0x7f86e40ce580 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.976+0000 7f86ea0fb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e406b2c0 con 0x7f86e415f4f0 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.976+0000 7f86ea0fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40b0890 con 0x7f86e40b0db0 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f86e40ce580 0x7f86e4072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39844/0 (socket says 172.21.15.77:39844) 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e3fff640 1 -- 172.21.15.77:0/3846634696 learned_addr learned my addr 172.21.15.77:0/3846634696 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e37fe640 1 --2- 172.21.15.77:0/3846634696 >> v2:172.21.15.158:3300/0 conn(0x7f86e415f4f0 0x7f86e4066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e2ffd640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86e40b0db0 0x7f86e40ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337623881 0 0) 0x7f86e406b050 con 0x7f86e40ce580 2024-03-06T07:24:50.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86cc003520 con 0x7f86e40ce580 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3489046045 0 0) 0x7f86cc003520 con 0x7f86e40ce580 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86e40b0db0 msgr2=0x7f86e40ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86e40b0db0 0x7f86e40ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 >> v2:172.21.15.158:3300/0 conn(0x7f86e415f4f0 msgr2=0x7f86e4066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 --2- 172.21.15.77:0/3846634696 >> v2:172.21.15.158:3300/0 conn(0x7f86e415f4f0 0x7f86e4066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e40672e0 con 0x7f86e40ce580 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f86dc0019f0 con 0x7f86e40ce580 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e37fe640 1 --2- 172.21.15.77:0/3846634696 >> v2:172.21.15.158:3300/0 conn(0x7f86e415f4f0 0x7f86e4066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 >> v1:172.21.15.77:6789/0 conn(0x7f86e40ce580 legacy=0x7f86e4072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86cc0039b0 0x7f86cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86cc0043a0 0x7f86cc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86cc004ef0 0x7f86cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e406b2c0 con 0x7f86cc0043a0 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40b0890 con 0x7f86cc004ef0 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc00bf00 con 0x7f86cc0039b0 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e2ffd640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86e40b0db0 0x7f86e40ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e2ffd640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86cc0043a0 0x7f86cc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e37fe640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86cc0039b0 0x7f86cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.977+0000 7f86e3fff640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86cc004ef0 0x7f86cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.978+0000 7f86e2ffd640 1 -- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86cc0039b0 msgr2=0x7f86cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.978+0000 7f86e2ffd640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86cc0039b0 0x7f86cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.978+0000 7f86e2ffd640 1 -- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86cc004ef0 msgr2=0x7f86cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.978+0000 7f86e2ffd640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86cc004ef0 0x7f86cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.978+0000 7f86e2ffd640 1 -- 172.21.15.77:0/3846634696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc003520 con 0x7f86cc0043a0 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.978+0000 7f86e2ffd640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86cc0043a0 0x7f86cc004880 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f86e406baa0 tx=0x7f86cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125b3545faa08cb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.978+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d80097d0 con 0x7f86cc0043a0 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d8004030 con 0x7f86cc0043a0 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86e27fc640 1 -- 172.21.15.77:0/3846634696 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8004400 con 0x7f86cc0043a0 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3846634696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86cc0043a0 msgr2=0x7f86cc004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86ea0fb640 1 --2- 172.21.15.77:0/3846634696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86cc0043a0 0x7f86cc004880 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f86e406baa0 tx=0x7f86cc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86e3fff640 1 -- 172.21.15.77:0/3846634696 reap_dead start 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3846634696 shutdown_connections 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3846634696 >> 172.21.15.77:0/3846634696 conn(0x7f86e40b1d70 msgr2=0x7f86e40b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3846634696 shutdown_connections 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.979+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3846634696 wait complete. 2024-03-06T07:24:50.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.980+0000 7f86ea0fb640 1 Processor -- start 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.980+0000 7f86ea0fb640 1 -- start start 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86ea0fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86cc0043a0 0x7f86e414c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86ea0fb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86e40ae400 0x7f86e4146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86ea0fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e40ae860 0x7f86e41471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86ea0fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e415f8d0 con 0x7f86e40ae860 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86ea0fb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e406b5a0 con 0x7f86e40ae400 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86ea0fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40ad9d0 con 0x7f86cc0043a0 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86e3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e40ae860 0x7f86e41471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86e3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e40ae860 0x7f86e41471c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38916/0 (socket says 172.21.15.77:38916) 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.981+0000 7f86e3fff640 1 -- 172.21.15.77:0/3870770659 learned_addr learned my addr 172.21.15.77:0/3870770659 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86e3fff640 1 -- 172.21.15.77:0/3870770659 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86cc0043a0 msgr2=0x7f86e414c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86e3fff640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86cc0043a0 0x7f86e414c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86e3fff640 1 -- 172.21.15.77:0/3870770659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86e40ae400 msgr2=0x7f86e4146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86e3fff640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86e40ae400 0x7f86e4146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86e3fff640 1 -- 172.21.15.77:0/3870770659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e40c7210 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86e3fff640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e40ae860 0x7f86e41471c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f86e40afb30 tx=0x7f86dc007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86cbfff640 1 -- 172.21.15.77:0/3870770659 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc00ab20 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86cbfff640 1 -- 172.21.15.77:0/3870770659 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc019070 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e414cae0 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86cbfff640 1 -- 172.21.15.77:0/3870770659 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0133f0 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.982+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e40adc00 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.983+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e409e3d0 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.983+0000 7f86cbfff640 1 -- 172.21.15.77:0/3870770659 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86dc013590 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.984+0000 7f86cbfff640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86d006b4d0 0x7f86d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.984+0000 7f86cbfff640 1 -- 172.21.15.77:0/3870770659 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f86dc08a490 con 0x7f86e40ae860 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.984+0000 7f86e37fe640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86d006b4d0 0x7f86d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.984+0000 7f86e37fe640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86d006b4d0 0x7f86d006d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009c40 tx=0x7f86d4009760 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:50.988+0000 7f86cbfff640 1 -- 172.21.15.77:0/3870770659 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86dc059610 con 0x7f86e40ae860 2024-03-06T07:24:51.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.165+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f86e40cf410 con 0x7f86e40ae860 2024-03-06T07:24:51.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.168+0000 7f86cbfff640 1 -- 172.21.15.77:0/3870770659 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f86dc058f10 con 0x7f86e40ae860 2024-03-06T07:24:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86d006b4d0 msgr2=0x7f86d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86d006b4d0 0x7f86d006d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009c40 tx=0x7f86d4009760 comp rx=0 tx=0).stop 2024-03-06T07:24:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e40ae860 msgr2=0x7f86e41471c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e40ae860 0x7f86e41471c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f86e40afb30 tx=0x7f86dc007a90 comp rx=0 tx=0).stop 2024-03-06T07:24:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 shutdown_connections 2024-03-06T07:24:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86d006b4d0 0x7f86d006d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e40ae860 0x7f86e41471c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86e40ae400 0x7f86e4146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 --2- 172.21.15.77:0/3870770659 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86cc0043a0 0x7f86e414c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 >> 172.21.15.77:0/3870770659 conn(0x7f86e40b1d70 msgr2=0x7f86e40afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 shutdown_connections 2024-03-06T07:24:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.170+0000 7f86ea0fb640 1 -- 172.21.15.77:0/3870770659 wait complete. 2024-03-06T07:24:51.183 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:24:51.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.282+0000 7f327d17a640 1 Processor -- start 2024-03-06T07:24:51.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.282+0000 7f327d17a640 1 -- start start 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327d17a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f327806da40 0x7f32780cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327d17a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f32780cd7a0 0x7f327815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327d17a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780cc980 con 0x7f32781424f0 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327d17a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780cd220 con 0x7f32780cd7a0 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327d17a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32780ccc60 con 0x7f327806da40 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f32780cd7a0 0x7f327815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f32780cd7a0 0x7f327815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38932/0 (socket says 172.21.15.77:38932) 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327759e640 1 -- 172.21.15.77:0/1987059384 learned_addr learned my addr 172.21.15.77:0/1987059384 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f326ffff640 1 --2- 172.21.15.77:0/1987059384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f327806da40 0x7f32780cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327759e640 1 -- 172.21.15.77:0/1987059384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f327806da40 msgr2=0x7f32780cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327759e640 1 --2- 172.21.15.77:0/1987059384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f327806da40 0x7f32780cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327759e640 1 -- 172.21.15.77:0/1987059384 >> v1:172.21.15.158:6789/0 conn(0x7f32781424f0 legacy=0x7f327806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f327759e640 1 -- 172.21.15.77:0/1987059384 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780ccf40 con 0x7f32780cd7a0 2024-03-06T07:24:51.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.283+0000 7f326ffff640 1 --2- 172.21.15.77:0/1987059384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f327806da40 0x7f32780cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:51.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.284+0000 7f327759e640 1 --2- 172.21.15.77:0/1987059384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32780cd7a0 0x7f327815cef0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3270009e50 tx=0x7f327002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51763bc00cc1dfa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.284+0000 7f327659c640 1 -- 172.21.15.77:0/1987059384 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327002ca70 con 0x7f32780cd7a0 2024-03-06T07:24:51.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.284+0000 7f327659c640 1 -- 172.21.15.77:0/1987059384 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327002cc10 con 0x7f32780cd7a0 2024-03-06T07:24:51.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.284+0000 7f327659c640 1 -- 172.21.15.77:0/1987059384 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32700343f0 con 0x7f32780cd7a0 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.284+0000 7f327d17a640 1 -- 172.21.15.77:0/1987059384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32780cd7a0 msgr2=0x7f327815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.284+0000 7f327d17a640 1 --2- 172.21.15.77:0/1987059384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32780cd7a0 0x7f327815cef0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3270009e50 tx=0x7f327002bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.285+0000 7f327d17a640 1 -- 172.21.15.77:0/1987059384 shutdown_connections 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.285+0000 7f327d17a640 1 --2- 172.21.15.77:0/1987059384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f327806da40 0x7f32780cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.285+0000 7f327d17a640 1 --2- 172.21.15.77:0/1987059384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32780cd7a0 0x7f327815cef0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.285+0000 7f327d17a640 1 -- 172.21.15.77:0/1987059384 >> 172.21.15.77:0/1987059384 conn(0x7f32780cf8a0 msgr2=0x7f32780cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.285+0000 7f327d17a640 1 -- 172.21.15.77:0/1987059384 shutdown_connections 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.285+0000 7f327d17a640 1 -- 172.21.15.77:0/1987059384 wait complete. 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.286+0000 7f327d17a640 1 Processor -- start 2024-03-06T07:24:51.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.286+0000 7f327d17a640 1 -- start start 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.286+0000 7f327d17a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f327806da40 0x7f327813ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327d17a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32780cd7a0 0x7f3278145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327d17a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32781424f0 0x7f32780b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327d17a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278071ce0 con 0x7f32781424f0 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327d17a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278071b60 con 0x7f327806da40 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327d17a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3278071e60 con 0x7f32780cd7a0 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32781424f0 0x7f32780b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32781424f0 0x7f32780b3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38944/0 (socket says 172.21.15.77:38944) 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 -- 172.21.15.77:0/1624197096 learned_addr learned my addr 172.21.15.77:0/1624197096 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f326ffff640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32780cd7a0 0x7f3278145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f3276d9d640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f327806da40 0x7f327813ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 -- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32780cd7a0 msgr2=0x7f3278145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32780cd7a0 0x7f3278145320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 -- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f327806da40 msgr2=0x7f327813ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f327806da40 0x7f327813ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 -- 172.21.15.77:0/1624197096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3278144280 con 0x7f32781424f0 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f326ffff640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32780cd7a0 0x7f3278145320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327759e640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32781424f0 0x7f32780b3790 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3270036040 tx=0x7f3270004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f326f7fe640 1 -- 172.21.15.77:0/1624197096 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270043020 con 0x7f32781424f0 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f326f7fe640 1 -- 172.21.15.77:0/1624197096 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327003e450 con 0x7f32781424f0 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3278144560 con 0x7f32781424f0 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f326f7fe640 1 -- 172.21.15.77:0/1624197096 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327003e820 con 0x7f32781424f0 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.287+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3278144b20 con 0x7f32781424f0 2024-03-06T07:24:51.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.288+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327804f330 con 0x7f32781424f0 2024-03-06T07:24:51.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.289+0000 7f326f7fe640 1 -- 172.21.15.77:0/1624197096 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3270046020 con 0x7f32781424f0 2024-03-06T07:24:51.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.289+0000 7f326f7fe640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f325806b4d0 0x7f325806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.289+0000 7f3276d9d640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f325806b4d0 0x7f325806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.289+0000 7f326f7fe640 1 -- 172.21.15.77:0/1624197096 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f32700ac7b0 con 0x7f32781424f0 2024-03-06T07:24:51.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.290+0000 7f3276d9d640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f325806b4d0 0x7f325806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3268006fd0 tx=0x7f3268008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.294+0000 7f326f7fe640 1 -- 172.21.15.77:0/1624197096 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f327004d030 con 0x7f32781424f0 2024-03-06T07:24:51.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:51.543+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f32780d0620 con 0x7f32781424f0 2024-03-06T07:24:52.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.249+0000 7f326f7fe640 1 -- 172.21.15.77:0/1624197096 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v168) v1 ==== 144+0+0 (secure 0 0 0) 0x7f327007b9c0 con 0x7f32781424f0 2024-03-06T07:24:52.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f325806b4d0 msgr2=0x7f325806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f325806b4d0 0x7f325806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3268006fd0 tx=0x7f3268008040 comp rx=0 tx=0).stop 2024-03-06T07:24:52.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32781424f0 msgr2=0x7f32780b3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32781424f0 0x7f32780b3790 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3270036040 tx=0x7f3270004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 shutdown_connections 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f325806b4d0 0x7f325806d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32781424f0 0x7f32780b3790 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32780cd7a0 0x7f3278145320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 --2- 172.21.15.77:0/1624197096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f327806da40 0x7f327813ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 >> 172.21.15.77:0/1624197096 conn(0x7f32780cf8a0 msgr2=0x7f32780afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.252+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 shutdown_connections 2024-03-06T07:24:52.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.253+0000 7f327d17a640 1 -- 172.21.15.77:0/1624197096 wait complete. 2024-03-06T07:24:52.268 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:24:52.268 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:24:52.268 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:24:52.268 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:52.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.371+0000 7fe7f1b7a640 1 Processor -- start 2024-03-06T07:24:52.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.371+0000 7fe7f1b7a640 1 -- start start 2024-03-06T07:24:52.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.372+0000 7fe7f1b7a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe7ec15dfe0 0x7fe7ec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.372+0000 7fe7f1b7a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ec0b3430 con 0x7fe7ec0d29c0 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.372+0000 7fe7f1b7a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ec0b3710 con 0x7fe7ec0cf830 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.372+0000 7fe7f1b7a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ec0b31e0 con 0x7fe7ec15dfe0 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.372+0000 7fe7ebfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe7ec0d29c0 0x7fe7ec0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39846/0 (socket says 172.21.15.77:39846) 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.372+0000 7fe7ebfff640 1 -- 172.21.15.77:0/3556805983 learned_addr learned my addr 172.21.15.77:0/3556805983 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955309781 0 0) 0x7fe7ec0b3430 con 0x7fe7ec0d29c0 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d8003520 con 0x7fe7ec0d29c0 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eb7fe640 1 --2- 172.21.15.77:0/3556805983 >> v2:172.21.15.158:3300/0 conn(0x7fe7ec15dfe0 0x7fe7ec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2748492556 0 0) 0x7fe7d8003520 con 0x7fe7ec0d29c0 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 >> v2:172.21.15.158:3300/0 conn(0x7fe7ec15dfe0 msgr2=0x7fe7ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 --2- 172.21.15.77:0/3556805983 >> v2:172.21.15.158:3300/0 conn(0x7fe7ec15dfe0 0x7fe7ec0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 >> v1:172.21.15.158:6789/0 conn(0x7fe7ec0cf830 legacy=0x7fe7ec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec0b39f0 con 0x7fe7ec0d29c0 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fe7e40019f0 con 0x7fe7ec0d29c0 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 >> v1:172.21.15.77:6789/0 conn(0x7fe7ec0d29c0 legacy=0x7fe7ec0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7d80039b0 0x7fe7d8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7d8004800 0x7fe7d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d8005210 0x7fe7d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec0b31e0 con 0x7fe7d80039b0 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d800c140 con 0x7fe7d8004800 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d800c340 con 0x7fe7d8005210 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eb7fe640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7d80039b0 0x7fe7d8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7e2ffd640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7d8004800 0x7fe7d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7ebfff640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d8005210 0x7fe7d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eb7fe640 1 -- 172.21.15.77:0/3556805983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d8005210 msgr2=0x7fe7d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eb7fe640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d8005210 0x7fe7d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eb7fe640 1 -- 172.21.15.77:0/3556805983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7d8004800 msgr2=0x7fe7d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eb7fe640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7d8004800 0x7fe7d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7eb7fe640 1 -- 172.21.15.77:0/3556805983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d8003520 con 0x7fe7d80039b0 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.373+0000 7fe7ebfff640 1 -- 172.21.15.77:0/3556805983 reap_dead start 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.374+0000 7fe7eb7fe640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7d80039b0 0x7fe7d8003df0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00ba30 tx=0x7fe7d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2bc78ba9c4519f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.374+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc036020 con 0x7fe7d80039b0 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.374+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc032640 con 0x7fe7d80039b0 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.374+0000 7fe7eaffd640 1 -- 172.21.15.77:0/3556805983 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc00b600 con 0x7fe7d80039b0 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.375+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3556805983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7d80039b0 msgr2=0x7fe7d8003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.375+0000 7fe7f1b7a640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7d80039b0 0x7fe7d8003df0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00ba30 tx=0x7fe7d8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:52.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.375+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3556805983 shutdown_connections 2024-03-06T07:24:52.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.375+0000 7fe7f1b7a640 1 --2- 172.21.15.77:0/3556805983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7d80039b0 0x7fe7d8003df0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.375+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3556805983 >> 172.21.15.77:0/3556805983 conn(0x7fe7ec0d3cd0 msgr2=0x7fe7ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.375+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3556805983 shutdown_connections 2024-03-06T07:24:52.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.375+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3556805983 wait complete. 2024-03-06T07:24:52.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.376+0000 7fe7f1b7a640 1 Processor -- start 2024-03-06T07:24:52.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.376+0000 7fe7f1b7a640 1 -- start start 2024-03-06T07:24:52.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7f1b7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d80039b0 0x7fe7ec0b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7f1b7a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec0b6c30 0x7fe7ec0b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7f1b7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7ec0b8c80 0x7fe7ec0b9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7f1b7a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec15e3c0 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7f1b7a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec0b3d60 con 0x7fe7ec0b6c30 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7f1b7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec15e540 con 0x7fe7d80039b0 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7ec0b8c80 0x7fe7ec0b9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7ec0b8c80 0x7fe7ec0b9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38976/0 (socket says 172.21.15.77:38976) 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 -- 172.21.15.77:0/3704611796 learned_addr learned my addr 172.21.15.77:0/3704611796 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7eb7fe640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d80039b0 0x7fe7ec0b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 -- 172.21.15.77:0/3704611796 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d80039b0 msgr2=0x7fe7ec0b1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d80039b0 0x7fe7ec0b1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7e2ffd640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec0b6c30 0x7fe7ec0b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 -- 172.21.15.77:0/3704611796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec0b6c30 msgr2=0x7fe7ec0b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec0b6c30 0x7fe7ec0b70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 -- 172.21.15.77:0/3704611796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec158ae0 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7ebfff640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7ec0b8c80 0x7fe7ec0b9130 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4007960 tx=0x7fe7e400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7e8ff9640 1 -- 172.21.15.77:0/3704611796 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e400ebc0 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.377+0000 7fe7e8ff9640 1 -- 172.21.15.77:0/3704611796 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e400ed80 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.378+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ec158dc0 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.378+0000 7fe7e8ff9640 1 -- 172.21.15.77:0/3704611796 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4014670 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.378+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ec159380 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.378+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec159580 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.379+0000 7fe7e8ff9640 1 -- 172.21.15.77:0/3704611796 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7e4014a20 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.379+0000 7fe7e8ff9640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7cc06b500 0x7fe7cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.379+0000 7fe7e8ff9640 1 -- 172.21.15.77:0/3704611796 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe7e4089e70 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.379+0000 7fe7eb7fe640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7cc06b500 0x7fe7cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.380+0000 7fe7eb7fe640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7cc06b500 0x7fe7cc06d9c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00ba00 tx=0x7fe7dc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.384+0000 7fe7e8ff9640 1 -- 172.21.15.77:0/3704611796 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7ec159580 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.560+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7ec04f330 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.561+0000 7fe7e8ff9640 1 -- 172.21.15.77:0/3704611796 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+13006 (secure 0 0 0) 0x7fe7e4058ff0 con 0x7fe7ec0b8c80 2024-03-06T07:24:52.563 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:52.563 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":61,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:50.237969+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/251375557":"2024-03-07T07:24:47.229896+0000","172.21.15.77:6835/251375557":"2024-03-07T07:24:47.229896+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7cc06b500 msgr2=0x7fe7cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7cc06b500 0x7fe7cc06d9c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00ba00 tx=0x7fe7dc030000 comp rx=0 tx=0).stop 2024-03-06T07:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7ec0b8c80 msgr2=0x7fe7ec0b9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7ec0b8c80 0x7fe7ec0b9130 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4007960 tx=0x7fe7e400e530 comp rx=0 tx=0).stop 2024-03-06T07:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 shutdown_connections 2024-03-06T07:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7cc06b500 0x7fe7cc06d9c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7ec0b8c80 0x7fe7ec0b9130 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec0b6c30 0x7fe7ec0b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 --2- 172.21.15.77:0/3704611796 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7d80039b0 0x7fe7ec0b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 >> 172.21.15.77:0/3704611796 conn(0x7fe7ec0d3cd0 msgr2=0x7fe7ec0b4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 shutdown_connections 2024-03-06T07:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.563+0000 7fe7f1b7a640 1 -- 172.21.15.77:0/3704611796 wait complete. 2024-03-06T07:24:52.576 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:52.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.678+0000 7f200a4fb640 1 Processor -- start 2024-03-06T07:24:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.678+0000 7f200a4fb640 1 -- start start 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.678+0000 7f200a4fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20041586e0 0x7f20040aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.678+0000 7f200a4fb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20040d01a0 0x7f20040b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.678+0000 7f200a4fb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2004158ac0 con 0x7f20040aef80 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.678+0000 7f200a4fb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200415b2a0 con 0x7f20040d01a0 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.678+0000 7f200a4fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200415b540 con 0x7f20041586e0 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2003fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20041586e0 0x7f20040aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2008a98640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20040d01a0 0x7f20040b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2003fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20041586e0 0x7f20040aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:43348/0 (socket says 172.21.15.77:43348) 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2003fff640 1 -- 172.21.15.77:0/2912510515 learned_addr learned my addr 172.21.15.77:0/2912510515 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2008a98640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20040d01a0 0x7f20040b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:52988/0 (socket says 172.21.15.77:52988) 2024-03-06T07:24:52.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f20037fe640 1 --1- 172.21.15.77:0/2912510515 >> v1:172.21.15.158:6789/0 conn(0x7f20040aef80 0x7f20040cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:33572/0 (socket says 172.21.15.77:33572) 2024-03-06T07:24:52.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2008a98640 1 -- 172.21.15.77:0/2912510515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20041586e0 msgr2=0x7f20040aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2008a98640 1 --2- 172.21.15.77:0/2912510515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20041586e0 0x7f20040aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2008a98640 1 -- 172.21.15.77:0/2912510515 >> v1:172.21.15.158:6789/0 conn(0x7f20040aef80 legacy=0x7f20040cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2008a98640 1 -- 172.21.15.77:0/2912510515 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200415b820 con 0x7f20040d01a0 2024-03-06T07:24:52.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.679+0000 7f2003fff640 1 --2- 172.21.15.77:0/2912510515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20041586e0 0x7f20040aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.680+0000 7f2008a98640 1 --2- 172.21.15.77:0/2912510515 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 0x7f20040b33a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009d90 tx=0x7f1ffc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ec38a061449430 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.680+0000 7f2002ffd640 1 -- 172.21.15.77:0/2912510515 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc039070 con 0x7f20040d01a0 2024-03-06T07:24:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.680+0000 7f2002ffd640 1 -- 172.21.15.77:0/2912510515 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc02cc50 con 0x7f20040d01a0 2024-03-06T07:24:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.680+0000 7f2002ffd640 1 -- 172.21.15.77:0/2912510515 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc034440 con 0x7f20040d01a0 2024-03-06T07:24:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.681+0000 7f200a4fb640 1 -- 172.21.15.77:0/2912510515 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 msgr2=0x7f20040b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.681+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2912510515 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 0x7f20040b33a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009d90 tx=0x7f1ffc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.681+0000 7f200a4fb640 1 -- 172.21.15.77:0/2912510515 shutdown_connections 2024-03-06T07:24:52.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.681+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2912510515 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 0x7f20040b33a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.681+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2912510515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20041586e0 0x7f20040aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.681+0000 7f200a4fb640 1 -- 172.21.15.77:0/2912510515 >> 172.21.15.77:0/2912510515 conn(0x7f200406b190 msgr2=0x7f200406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.681+0000 7f200a4fb640 1 -- 172.21.15.77:0/2912510515 shutdown_connections 2024-03-06T07:24:52.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.681+0000 7f200a4fb640 1 -- 172.21.15.77:0/2912510515 wait complete. 2024-03-06T07:24:52.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.682+0000 7f200a4fb640 1 Processor -- start 2024-03-06T07:24:52.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.682+0000 7f200a4fb640 1 -- start start 2024-03-06T07:24:52.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f200a4fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20040aef80 0x7f20040ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f200a4fb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 0x7f20041f8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f200a4fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20041586e0 0x7f20040adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f200a4fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2004071350 con 0x7f20041586e0 2024-03-06T07:24:52.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f200a4fb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200415bde0 con 0x7f20040d01a0 2024-03-06T07:24:52.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f200a4fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20040714d0 con 0x7f20040aef80 2024-03-06T07:24:52.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20041586e0 0x7f20040adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20041586e0 0x7f20040adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38988/0 (socket says 172.21.15.77:38988) 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 -- 172.21.15.77:0/2277569339 learned_addr learned my addr 172.21.15.77:0/2277569339 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f20037fe640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 0x7f20041f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2003fff640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20040aef80 0x7f20040ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 -- 172.21.15.77:0/2277569339 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20040aef80 msgr2=0x7f20040ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20040aef80 0x7f20040ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 -- 172.21.15.77:0/2277569339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 msgr2=0x7f20041f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 0x7f20041f8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 -- 172.21.15.77:0/2277569339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20040cd950 con 0x7f20041586e0 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2008a98640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20041586e0 0x7f20040adad0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009eb0 tx=0x7f1ffc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2000ff9640 1 -- 172.21.15.77:0/2277569339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc039050 con 0x7f20041586e0 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2000ff9640 1 -- 172.21.15.77:0/2277569339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc045070 con 0x7f20041586e0 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20040cdc30 con 0x7f20041586e0 2024-03-06T07:24:52.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f2000ff9640 1 -- 172.21.15.77:0/2277569339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc009450 con 0x7f20041586e0 2024-03-06T07:24:52.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.683+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20040ce1f0 con 0x7f20041586e0 2024-03-06T07:24:52.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.684+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200415c250 con 0x7f20041586e0 2024-03-06T07:24:52.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.685+0000 7f2000ff9640 1 -- 172.21.15.77:0/2277569339 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ffc03e610 con 0x7f20041586e0 2024-03-06T07:24:52.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.685+0000 7f2000ff9640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ff006b4d0 0x7f1ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.685+0000 7f2000ff9640 1 -- 172.21.15.77:0/2277569339 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1ffc0acb40 con 0x7f20041586e0 2024-03-06T07:24:52.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.685+0000 7f2003fff640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ff006b4d0 0x7f1ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.686+0000 7f2003fff640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ff006b4d0 0x7f1ff006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009ac0 tx=0x7f1ff8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.689+0000 7f2000ff9640 1 -- 172.21.15.77:0/2277569339 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f200415c250 con 0x7f20041586e0 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.859+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20040aeda0 con 0x7f20041586e0 2024-03-06T07:24:52.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.859+0000 7f2000ff9640 1 -- 172.21.15.77:0/2277569339 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7f20040aeda0 con 0x7f20041586e0 2024-03-06T07:24:52.862 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:24:52.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ff006b4d0 msgr2=0x7f1ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ff006b4d0 0x7f1ff006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009ac0 tx=0x7f1ff8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:52.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20041586e0 msgr2=0x7f20040adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20041586e0 0x7f20040adad0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009eb0 tx=0x7f1ffc004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:52.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 shutdown_connections 2024-03-06T07:24:52.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ff006b4d0 0x7f1ff006d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20041586e0 0x7f20040adad0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20040d01a0 0x7f20041f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 --2- 172.21.15.77:0/2277569339 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20040aef80 0x7f20040ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 >> 172.21.15.77:0/2277569339 conn(0x7f200406b190 msgr2=0x7f2004156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 shutdown_connections 2024-03-06T07:24:52.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.861+0000 7f200a4fb640 1 -- 172.21.15.77:0/2277569339 wait complete. 2024-03-06T07:24:52.874 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:24:52.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.973+0000 7f4bcced4640 1 Processor -- start 2024-03-06T07:24:52.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.973+0000 7f4bcced4640 1 -- start start 2024-03-06T07:24:52.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bcced4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc815dfe0 0x7f4bc80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bcced4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4bc80cf830 0x7f4bc80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bcced4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4bc80d27f0 0x7f4bc80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bcced4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc80d2c50 con 0x7f4bc80cf830 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bcced4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc80b31e0 con 0x7f4bc80d27f0 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bcced4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80b3d60 con 0x7f4bc815dfe0 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc5d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4bc80cf830 0x7f4bc80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc5d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4bc80cf830 0x7f4bc80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39000/0 (socket says 172.21.15.77:39000) 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc5d9b640 1 -- 172.21.15.77:0/3857138131 learned_addr learned my addr 172.21.15.77:0/3857138131 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc5d9b640 1 -- 172.21.15.77:0/3857138131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc815dfe0 msgr2=0x7f4bc80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc5d9b640 1 --2- 172.21.15.77:0/3857138131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc815dfe0 0x7f4bc80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc5d9b640 1 -- 172.21.15.77:0/3857138131 >> v2:172.21.15.158:3300/0 conn(0x7f4bc80d27f0 msgr2=0x7f4bc80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc6d9d640 1 --2- 172.21.15.77:0/3857138131 >> v2:172.21.15.158:3300/0 conn(0x7f4bc80d27f0 0x7f4bc80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc5d9b640 1 --2- 172.21.15.77:0/3857138131 >> v2:172.21.15.158:3300/0 conn(0x7f4bc80d27f0 0x7f4bc80d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.974+0000 7f4bc5d9b640 1 -- 172.21.15.77:0/3857138131 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80b3430 con 0x7f4bc80cf830 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.975+0000 7f4bc5d9b640 1 --2- 172.21.15.77:0/3857138131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80cf830 0x7f4bc80d22b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8067ad0 tx=0x7f4bbc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27d828db219dfdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.975+0000 7f4bc559a640 1 -- 172.21.15.77:0/3857138131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc035020 con 0x7f4bc80cf830 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.975+0000 7f4bc559a640 1 -- 172.21.15.77:0/3857138131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc031870 con 0x7f4bc80cf830 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.975+0000 7f4bc559a640 1 -- 172.21.15.77:0/3857138131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc031c40 con 0x7f4bc80cf830 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.975+0000 7f4bcced4640 1 -- 172.21.15.77:0/3857138131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80cf830 msgr2=0x7f4bc80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.975+0000 7f4bcced4640 1 --2- 172.21.15.77:0/3857138131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80cf830 0x7f4bc80d22b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8067ad0 tx=0x7f4bbc02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.976+0000 7f4bcced4640 1 -- 172.21.15.77:0/3857138131 shutdown_connections 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.976+0000 7f4bcced4640 1 --2- 172.21.15.77:0/3857138131 >> v2:172.21.15.158:3300/0 conn(0x7f4bc80d27f0 0x7f4bc80d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.976+0000 7f4bcced4640 1 --2- 172.21.15.77:0/3857138131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80cf830 0x7f4bc80d22b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.976+0000 7f4bcced4640 1 --2- 172.21.15.77:0/3857138131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc815dfe0 0x7f4bc80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.976+0000 7f4bcced4640 1 -- 172.21.15.77:0/3857138131 >> 172.21.15.77:0/3857138131 conn(0x7f4bc80d3cd0 msgr2=0x7f4bc80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.976+0000 7f4bcced4640 1 -- 172.21.15.77:0/3857138131 shutdown_connections 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.976+0000 7f4bcced4640 1 -- 172.21.15.77:0/3857138131 wait complete. 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.977+0000 7f4bcced4640 1 Processor -- start 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.977+0000 7f4bcced4640 1 -- start start 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bcced4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bc80cf830 0x7f4bc8160f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bcced4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc80d27f0 0x7f4bc80b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bcced4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc815dfe0 0x7f4bc8152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bcced4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80d1c70 con 0x7f4bc815dfe0 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bcced4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80b39f0 con 0x7f4bc80cf830 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bcced4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80d1df0 con 0x7f4bc80d27f0 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc815dfe0 0x7f4bc8152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc815dfe0 0x7f4bc8152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39010/0 (socket says 172.21.15.77:39010) 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 -- 172.21.15.77:0/1913243534 learned_addr learned my addr 172.21.15.77:0/1913243534 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc659c640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bc80cf830 0x7f4bc8160f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc5d9b640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc80d27f0 0x7f4bc80b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 -- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc80d27f0 msgr2=0x7f4bc80b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc80d27f0 0x7f4bc80b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 -- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bc80cf830 msgr2=0x7f4bc8160f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bc80cf830 0x7f4bc8160f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 -- 172.21.15.77:0/1913243534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80b7120 con 0x7f4bc815dfe0 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc659c640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bc80cf830 0x7f4bc8160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc5d9b640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc80d27f0 0x7f4bc80b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bc6d9d640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc815dfe0 0x7f4bc8152f30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000cd70 tx=0x7f4bc000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.978+0000 7f4bb6ffd640 1 -- 172.21.15.77:0/1913243534 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0003b90 con 0x7f4bc815dfe0 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.979+0000 7f4bb6ffd640 1 -- 172.21.15.77:0/1913243534 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc0003d30 con 0x7f4bc815dfe0 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.979+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc80b7400 con 0x7f4bc815dfe0 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.979+0000 7f4bb6ffd640 1 -- 172.21.15.77:0/1913243534 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0004240 con 0x7f4bc815dfe0 2024-03-06T07:24:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.979+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc80b79c0 con 0x7f4bc815dfe0 2024-03-06T07:24:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.979+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc804f330 con 0x7f4bc815dfe0 2024-03-06T07:24:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.980+0000 7f4bb6ffd640 1 -- 172.21.15.77:0/1913243534 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bc00043e0 con 0x7f4bc815dfe0 2024-03-06T07:24:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.980+0000 7f4bb6ffd640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ba806b4d0 0x7f4ba806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.980+0000 7f4bb6ffd640 1 -- 172.21.15.77:0/1913243534 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4bc008aab0 con 0x7f4bc815dfe0 2024-03-06T07:24:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.980+0000 7f4bc659c640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ba806b4d0 0x7f4ba806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.981+0000 7f4bc659c640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ba806b4d0 0x7f4ba806d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800a870 tx=0x7f4bb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:52.985+0000 7f4bb6ffd640 1 -- 172.21.15.77:0/1913243534 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bc0059ba0 con 0x7f4bc815dfe0 2024-03-06T07:24:53.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.222+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f4bc806c5d0 con 0x7f4bc815dfe0 2024-03-06T07:24:53.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.251+0000 7f4bb6ffd640 1 -- 172.21.15.77:0/1913243534 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v169) v1 ==== 143+0+0 (secure 0 0 0) 0x7f4bc00594a0 con 0x7f4bc815dfe0 2024-03-06T07:24:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ba806b4d0 msgr2=0x7f4ba806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ba806b4d0 0x7f4ba806d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800a870 tx=0x7f4bb8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc815dfe0 msgr2=0x7f4bc8152f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc815dfe0 0x7f4bc8152f30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000cd70 tx=0x7f4bc000a660 comp rx=0 tx=0).stop 2024-03-06T07:24:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 shutdown_connections 2024-03-06T07:24:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ba806b4d0 0x7f4ba806d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc815dfe0 0x7f4bc8152f30 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4bc80d27f0 0x7f4bc80b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 --2- 172.21.15.77:0/1913243534 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4bc80cf830 0x7f4bc8160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 >> 172.21.15.77:0/1913243534 conn(0x7f4bc80d3cd0 msgr2=0x7f4bc806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 shutdown_connections 2024-03-06T07:24:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.253+0000 7f4bcced4640 1 -- 172.21.15.77:0/1913243534 wait complete. 2024-03-06T07:24:53.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.366+0000 7f9c78981640 1 Processor -- start 2024-03-06T07:24:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.366+0000 7f9c78981640 1 -- start start 2024-03-06T07:24:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c78981640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c7415bb40 0x7f9c740b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c78981640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c7406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c78981640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7415ff90 con 0x7f9c7406d5c0 2024-03-06T07:24:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c78981640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7406c910 con 0x7f9c7415bb40 2024-03-06T07:24:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c78981640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74160270 con 0x7f9c7415f7d0 2024-03-06T07:24:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c71d9b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c7406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c7259c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c7415bb40 0x7f9c740b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c72d9d640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f9c7406d5c0 0x7f9c740cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:33574/0 (socket says 172.21.15.77:33574) 2024-03-06T07:24:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c72d9d640 1 -- 172.21.15.77:0/2981071661 learned_addr learned my addr 172.21.15.77:0/2981071661 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.367+0000 7f9c7259c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c7415bb40 0x7f9c740b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:53010/0 (socket says 172.21.15.77:53010) 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.368+0000 7f9c7159a640 1 -- 172.21.15.77:0/2981071661 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36472553 0 0) 0x7f9c7415ff90 con 0x7f9c7406d5c0 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.368+0000 7f9c7159a640 1 -- 172.21.15.77:0/2981071661 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c003520 con 0x7f9c7406d5c0 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.368+0000 7f9c7259c640 1 -- 172.21.15.77:0/2981071661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 msgr2=0x7f9c7406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.368+0000 7f9c7259c640 1 --2- 172.21.15.77:0/2981071661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c7406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.368+0000 7f9c7259c640 1 -- 172.21.15.77:0/2981071661 >> v1:172.21.15.158:6789/0 conn(0x7f9c7406d5c0 legacy=0x7f9c740cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.368+0000 7f9c7259c640 1 -- 172.21.15.77:0/2981071661 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c74160550 con 0x7f9c7415bb40 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.368+0000 7f9c71d9b640 1 --2- 172.21.15.77:0/2981071661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c7406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c7259c640 1 --2- 172.21.15.77:0/2981071661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 0x7f9c740b4970 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009d90 tx=0x7f9c6c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48be2fa1d95aba65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c7159a640 1 -- 172.21.15.77:0/2981071661 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c039070 con 0x7f9c7415bb40 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c7159a640 1 -- 172.21.15.77:0/2981071661 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c02cc50 con 0x7f9c7415bb40 2024-03-06T07:24:53.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c7159a640 1 -- 172.21.15.77:0/2981071661 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c034440 con 0x7f9c7415bb40 2024-03-06T07:24:53.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c78981640 1 -- 172.21.15.77:0/2981071661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 msgr2=0x7f9c740b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c78981640 1 --2- 172.21.15.77:0/2981071661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 0x7f9c740b4970 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009d90 tx=0x7f9c6c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c78981640 1 -- 172.21.15.77:0/2981071661 shutdown_connections 2024-03-06T07:24:53.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c78981640 1 --2- 172.21.15.77:0/2981071661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c7406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c78981640 1 --2- 172.21.15.77:0/2981071661 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 0x7f9c740b4970 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c78981640 1 -- 172.21.15.77:0/2981071661 >> 172.21.15.77:0/2981071661 conn(0x7f9c7406b190 msgr2=0x7f9c7406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c78981640 1 -- 172.21.15.77:0/2981071661 shutdown_connections 2024-03-06T07:24:53.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.369+0000 7f9c78981640 1 -- 172.21.15.77:0/2981071661 wait complete. 2024-03-06T07:24:53.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.371+0000 7f9c78981640 1 Processor -- start 2024-03-06T07:24:53.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.371+0000 7f9c78981640 1 -- start start 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.371+0000 7f9c78981640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c7406d5c0 0x7f9c741465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.371+0000 7f9c78981640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 0x7f9c74157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.371+0000 7f9c78981640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c741550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.371+0000 7f9c78981640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7415bf20 con 0x7f9c7406d5c0 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.371+0000 7f9c78981640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7415fc30 con 0x7f9c7415bb40 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.371+0000 7f9c78981640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7415c0a0 con 0x7f9c7415f7d0 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c7406d5c0 0x7f9c741465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c7406d5c0 0x7f9c741465a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39022/0 (socket says 172.21.15.77:39022) 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 -- 172.21.15.77:0/730863090 learned_addr learned my addr 172.21.15.77:0/730863090 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c72d9d640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c741550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c71d9b640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 0x7f9c74157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 -- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 msgr2=0x7f9c741550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c741550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 -- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 msgr2=0x7f9c74157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 0x7f9c74157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 -- 172.21.15.77:0/730863090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7415c3f0 con 0x7f9c7406d5c0 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c71d9b640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 0x7f9c74157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c7259c640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c7406d5c0 0x7f9c741465a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009eb0 tx=0x7f9c6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c62ffd640 1 -- 172.21.15.77:0/730863090 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c039050 con 0x7f9c7406d5c0 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c62ffd640 1 -- 172.21.15.77:0/730863090 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c045070 con 0x7f9c7406d5c0 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7415c6d0 con 0x7f9c7406d5c0 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c62ffd640 1 -- 172.21.15.77:0/730863090 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c009450 con 0x7f9c7406d5c0 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.372+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7415cc90 con 0x7f9c7406d5c0 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.373+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c740cf5f0 con 0x7f9c7406d5c0 2024-03-06T07:24:53.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.374+0000 7f9c62ffd640 1 -- 172.21.15.77:0/730863090 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c6c03e610 con 0x7f9c7406d5c0 2024-03-06T07:24:53.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.374+0000 7f9c62ffd640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.374+0000 7f9c71d9b640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.374+0000 7f9c62ffd640 1 -- 172.21.15.77:0/730863090 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9c6c0ac830 con 0x7f9c7406d5c0 2024-03-06T07:24:53.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.375+0000 7f9c71d9b640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5406b500 0x7f9c5406d9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9c740b0840 tx=0x7f9c68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.379+0000 7f9c62ffd640 1 -- 172.21.15.77:0/730863090 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c740cf5f0 con 0x7f9c7406d5c0 2024-03-06T07:24:53.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.551+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9c740732b0 con 0x7f9c7406d5c0 2024-03-06T07:24:53.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.554+0000 7f9c62ffd640 1 -- 172.21.15.77:0/730863090 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9c6c07b9b0 con 0x7f9c7406d5c0 2024-03-06T07:24:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5406b500 msgr2=0x7f9c5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5406b500 0x7f9c5406d9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9c740b0840 tx=0x7f9c68008040 comp rx=0 tx=0).stop 2024-03-06T07:24:53.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c7406d5c0 msgr2=0x7f9c741465a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c7406d5c0 0x7f9c741465a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009eb0 tx=0x7f9c6c004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 shutdown_connections 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c7415f7d0 0x7f9c741550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c7415bb40 0x7f9c74157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 --2- 172.21.15.77:0/730863090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c7406d5c0 0x7f9c741465a0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 >> 172.21.15.77:0/730863090 conn(0x7f9c7406b190 msgr2=0x7f9c74156d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 shutdown_connections 2024-03-06T07:24:53.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.556+0000 7f9c78981640 1 -- 172.21.15.77:0/730863090 wait complete. 2024-03-06T07:24:53.569 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:53.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:53.617 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:53.667 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:53.667 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:53.667 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:53.683 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:53.683 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:24:53.683 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:53.700 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:53.701 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:24:53.701 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:53.738 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:53.738 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:24:53.739 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:53.758 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:53.758 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:24:53.758 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:53.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.876+0000 7f7f73aed640 1 Processor -- start 2024-03-06T07:24:53.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.876+0000 7f7f73aed640 1 -- start start 2024-03-06T07:24:53.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f73aed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c06da40 0x7f7f6c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f73aed640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c0cc980 con 0x7f7f6c1424f0 2024-03-06T07:24:53.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f73aed640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c0ccc60 con 0x7f7f6c0cd7a0 2024-03-06T07:24:53.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f73aed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0cd220 con 0x7f7f6c06da40 2024-03-06T07:24:53.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f71889640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7f6c1424f0 0x7f7f6c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39860/0 (socket says 172.21.15.77:39860) 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f71889640 1 -- 172.21.15.77:0/1940729690 learned_addr learned my addr 172.21.15.77:0/1940729690 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669504107 0 0) 0x7f7f6c0cc980 con 0x7f7f6c1424f0 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f58003520 con 0x7f7f6c1424f0 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.877+0000 7f7f71088640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c06da40 0x7f7f6c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048498805 0 0) 0x7f7f58003520 con 0x7f7f6c1424f0 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c06da40 msgr2=0x7f7f6c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c06da40 0x7f7f6c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 >> v1:172.21.15.158:6789/0 conn(0x7f7f6c0cd7a0 legacy=0x7f7f6c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c0ccf40 con 0x7f7f6c1424f0 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f7f680019f0 con 0x7f7f6c1424f0 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 >> v1:172.21.15.77:6789/0 conn(0x7f7f6c1424f0 legacy=0x7f7f6c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c06da40 0x7f7f580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f58003f60 0x7f7f58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f58004d40 0x7f7f580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0cd220 con 0x7f7f58004d40 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5800baa0 con 0x7f7f58003f60 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5800bca0 con 0x7f7f6c06da40 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f7208a640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f58004d40 0x7f7f580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f71088640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f58003f60 0x7f7f58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f71889640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c06da40 0x7f7f580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f7208a640 1 -- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c06da40 msgr2=0x7f7f580039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f7208a640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c06da40 0x7f7f580039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f7208a640 1 -- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f58003f60 msgr2=0x7f7f58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f7208a640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f58003f60 0x7f7f58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.878+0000 7f7f7208a640 1 -- 172.21.15.77:0/1940729690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f58003520 con 0x7f7f58004d40 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.879+0000 7f7f7208a640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f58004d40 0x7f7f580051c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7f64001750 tx=0x7f7f58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513c224a40ef62f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.879+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64001370 con 0x7f7f58004d40 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.879+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f640052d0 con 0x7f7f58004d40 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.879+0000 7f7f70887640 1 -- 172.21.15.77:0/1940729690 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64003830 con 0x7f7f58004d40 2024-03-06T07:24:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.879+0000 7f7f73aed640 1 -- 172.21.15.77:0/1940729690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f58004d40 msgr2=0x7f7f580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.879+0000 7f7f73aed640 1 --2- 172.21.15.77:0/1940729690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f58004d40 0x7f7f580051c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7f64001750 tx=0x7f7f58005830 comp rx=0 tx=0).stop 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.880+0000 7f7f7208a640 1 -- 172.21.15.77:0/1940729690 reap_dead start 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.880+0000 7f7f73aed640 1 -- 172.21.15.77:0/1940729690 shutdown_connections 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.880+0000 7f7f73aed640 1 -- 172.21.15.77:0/1940729690 >> 172.21.15.77:0/1940729690 conn(0x7f7f6c0cf8a0 msgr2=0x7f7f6c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.880+0000 7f7f73aed640 1 -- 172.21.15.77:0/1940729690 shutdown_connections 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.880+0000 7f7f73aed640 1 -- 172.21.15.77:0/1940729690 wait complete. 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.881+0000 7f7f73aed640 1 Processor -- start 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.881+0000 7f7f73aed640 1 -- start start 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.881+0000 7f7f73aed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c13bbd0 0x7f7f6c153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.881+0000 7f7f73aed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f6c1593b0 0x7f7f6c1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.881+0000 7f7f73aed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c153cd0 0x7f7f6c155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.881+0000 7f7f73aed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0cc470 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.881+0000 7f7f73aed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0cc2f0 con 0x7f7f6c1593b0 2024-03-06T07:24:53.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f73aed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0cc5f0 con 0x7f7f6c153cd0 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c13bbd0 0x7f7f6c153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c13bbd0 0x7f7f6c153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39038/0 (socket says 172.21.15.77:39038) 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 -- 172.21.15.77:0/3042296491 learned_addr learned my addr 172.21.15.77:0/3042296491 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71088640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f6c1593b0 0x7f7f6c1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f7208a640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c153cd0 0x7f7f6c155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 -- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c153cd0 msgr2=0x7f7f6c155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c153cd0 0x7f7f6c155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 -- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f6c1593b0 msgr2=0x7f7f6c1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f6c1593b0 0x7f7f6c1551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 -- 172.21.15.77:0/3042296491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c13bfb0 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f7208a640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c153cd0 0x7f7f6c155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71088640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f6c1593b0 0x7f7f6c1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f71889640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c13bbd0 0x7f7f6c153790 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800a590 tx=0x7f7f6800aa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f627fc640 1 -- 172.21.15.77:0/3042296491 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f68013600 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f627fc640 1 -- 172.21.15.77:0/3042296491 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f68018070 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c154170 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f627fc640 1 -- 172.21.15.77:0/3042296491 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f680139c0 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.882+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c154730 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.883+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c04f330 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.884+0000 7f7f627fc640 1 -- 172.21.15.77:0/3042296491 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f68013b60 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.884+0000 7f7f627fc640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f5006b4d0 0x7f7f5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.884+0000 7f7f627fc640 1 -- 172.21.15.77:0/3042296491 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7f6808ac20 con 0x7f7f6c13bbd0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.884+0000 7f7f71088640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f5006b4d0 0x7f7f5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.884+0000 7f7f71088640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f5006b4d0 0x7f7f5006d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c1f57a0 tx=0x7f7f5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:53.888+0000 7f7f627fc640 1 -- 172.21.15.77:0/3042296491 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f68023250 con 0x7f7f6c13bbd0 2024-03-06T07:24:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.100+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f6c0d0620 con 0x7f7f6c13bbd0 2024-03-06T07:24:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.100+0000 7f7f627fc640 1 -- 172.21.15.77:0/3042296491 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f7f68059da0 con 0x7f7f6c13bbd0 2024-03-06T07:24:54.102 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:54.102 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/4080700420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":4080700420},{"type":"v1","addr":"172.21.15.158:6835","nonce":4080700420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1019401193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1019401193},{"type":"v1","addr":"172.21.15.158:6834","nonce":1019401193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6051,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1455054164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1455054164},{"type":"v1","addr":"172.21.15.77:6836","nonce":1455054164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2291401617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2291401617},{"type":"v1","addr":"172.21.15.77:6837","nonce":2291401617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[]} 2024-03-06T07:24:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 28 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.102+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f5006b4d0 msgr2=0x7f7f5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f5006b4d0 0x7f7f5006d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c1f57a0 tx=0x7f7f5c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c13bbd0 msgr2=0x7f7f6c153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c13bbd0 0x7f7f6c153790 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800a590 tx=0x7f7f6800aa50 comp rx=0 tx=0).stop 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 shutdown_connections 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f6c153cd0 0x7f7f6c155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f6c1593b0 0x7f7f6c1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f5006b4d0 0x7f7f5006d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 --2- 172.21.15.77:0/3042296491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c13bbd0 0x7f7f6c153790 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 >> 172.21.15.77:0/3042296491 conn(0x7f7f6c0cf8a0 msgr2=0x7f7f6c06cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 shutdown_connections 2024-03-06T07:24:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.103+0000 7f7f73aed640 1 -- 172.21.15.77:0/3042296491 wait complete. 2024-03-06T07:24:54.116 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:54.116 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:54.116 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:54.116 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:54.116 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:54.116 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:54.116 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:54.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:54.117 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:54.117 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:54.117 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:54.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:54.117 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:54.117 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:54.117 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:54.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:54.128 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:54.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:54.128 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:54.129 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:54.130 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:54.130 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:54.131 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:54.131 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:54.169 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:54.170 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:54.170 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:54.170 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:54.172 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:54.173 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:54.173 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:54.173 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:54.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:54.246 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:24:54.246 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:24:54.258 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:24:54.258 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:24:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.278+0000 7f7f9f3bc640 1 Processor -- start 2024-03-06T07:24:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.279+0000 7f7f9f3bc640 1 -- start start 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.279+0000 7f7f9f3bc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7f98162970 0x7f7f980d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.279+0000 7f7f9f3bc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f980d22c0 con 0x7f7f98161cf0 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.279+0000 7f7f9f3bc640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f980d25a0 con 0x7f7f980b4460 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f9f3bc640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f98161b00 con 0x7f7f98162970 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f9d959640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7f98161cf0 0x7f7f980cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39884/0 (socket says 172.21.15.77:39884) 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f9d959640 1 -- 172.21.15.77:0/266606428 learned_addr learned my addr 172.21.15.77:0/266606428 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f9c957640 1 --2- 172.21.15.77:0/266606428 >> v2:172.21.15.158:3300/0 conn(0x7f7f98162970 0x7f7f980d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861890221 0 0) 0x7f7f980d22c0 con 0x7f7f98161cf0 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f84003520 con 0x7f7f98161cf0 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508194810 0 0) 0x7f7f980d25a0 con 0x7f7f980b4460 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f980d22c0 con 0x7f7f980b4460 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1668471403 0 0) 0x7f7f84003520 con 0x7f7f98161cf0 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 >> v2:172.21.15.158:3300/0 conn(0x7f7f98162970 msgr2=0x7f7f980d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 --2- 172.21.15.77:0/266606428 >> v2:172.21.15.158:3300/0 conn(0x7f7f98162970 0x7f7f980d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 >> v1:172.21.15.158:6789/0 conn(0x7f7f980b4460 legacy=0x7f7f980b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f980d2880 con 0x7f7f98161cf0 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f7f90001970 con 0x7f7f98161cf0 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 >> v1:172.21.15.77:6789/0 conn(0x7f7f98161cf0 legacy=0x7f7f980cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f84003b90 0x7f7f84003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f84004800 0x7f7f84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f840051c0 0x7f7f8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f98161b00 con 0x7f7f840051c0 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8400c090 con 0x7f7f84003b90 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8400c290 con 0x7f7f84004800 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f9d959640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f840051c0 0x7f7f8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f9c957640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f84004800 0x7f7f84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.280+0000 7f7f9d158640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f84003b90 0x7f7f84003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.281+0000 7f7f9d959640 1 -- 172.21.15.77:0/266606428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f84004800 msgr2=0x7f7f84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.281+0000 7f7f9d959640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f84004800 0x7f7f84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.281+0000 7f7f9d959640 1 -- 172.21.15.77:0/266606428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f84003b90 msgr2=0x7f7f84003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.281+0000 7f7f9d959640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f84003b90 0x7f7f84003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.281+0000 7f7f9d959640 1 -- 172.21.15.77:0/266606428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f84003520 con 0x7f7f840051c0 2024-03-06T07:24:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.281+0000 7f7f9d959640 1 -- 172.21.15.77:0/266606428 reap_dead start 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.281+0000 7f7f9d959640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f840051c0 0x7f7f8400b9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000a6b0 tx=0x7f7f84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1343218eede662 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9000a2d0 con 0x7f7f840051c0 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f90007370 con 0x7f7f840051c0 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f8bfff640 1 -- 172.21.15.77:0/266606428 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90007760 con 0x7f7f840051c0 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/266606428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f840051c0 msgr2=0x7f7f8400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f9f3bc640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f840051c0 0x7f7f8400b9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000a6b0 tx=0x7f7f84005830 comp rx=0 tx=0).stop 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/266606428 shutdown_connections 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f9f3bc640 1 --2- 172.21.15.77:0/266606428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f840051c0 0x7f7f8400b9c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/266606428 >> 172.21.15.77:0/266606428 conn(0x7f7f980d3ce0 msgr2=0x7f7f980d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/266606428 shutdown_connections 2024-03-06T07:24:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.282+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/266606428 wait complete. 2024-03-06T07:24:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.283+0000 7f7f9f3bc640 1 Processor -- start 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.283+0000 7f7f9f3bc640 1 -- start start 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9f3bc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f840051c0 0x7f7f98156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9f3bc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f980ccaa0 0x7f7f9815e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9f3bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f9815ebe0 0x7f7f980cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9f3bc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f980ce950 con 0x7f7f9815ebe0 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9f3bc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f980cecc0 con 0x7f7f840051c0 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9f3bc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f980b4840 con 0x7f7f980ccaa0 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9c957640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f980ccaa0 0x7f7f9815e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9d959640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f9815ebe0 0x7f7f980cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9d158640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f840051c0 0x7f7f98156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9c957640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f980ccaa0 0x7f7f9815e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:43470/0 (socket says 172.21.15.77:43470) 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9c957640 1 -- 172.21.15.77:0/2994826999 learned_addr learned my addr 172.21.15.77:0/2994826999 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9d959640 1 -- 172.21.15.77:0/2994826999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f980ccaa0 msgr2=0x7f7f9815e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9d959640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f980ccaa0 0x7f7f9815e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9d959640 1 -- 172.21.15.77:0/2994826999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f840051c0 msgr2=0x7f7f98156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9d959640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f840051c0 0x7f7f98156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.284+0000 7f7f9d959640 1 -- 172.21.15.77:0/2994826999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f980cba00 con 0x7f7f9815ebe0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.285+0000 7f7f9d959640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f9815ebe0 0x7f7f980cadb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f7f980c7610 tx=0x7f7f980cec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.285+0000 7f7f89ffb640 1 -- 172.21.15.77:0/2994826999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90043070 con 0x7f7f9815ebe0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.285+0000 7f7f89ffb640 1 -- 172.21.15.77:0/2994826999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f90004b60 con 0x7f7f9815ebe0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.285+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f980cbce0 con 0x7f7f9815ebe0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.285+0000 7f7f89ffb640 1 -- 172.21.15.77:0/2994826999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90039040 con 0x7f7f9815ebe0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.285+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f980cc2a0 con 0x7f7f9815ebe0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.286+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9804f330 con 0x7f7f9815ebe0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.286+0000 7f7f89ffb640 1 -- 172.21.15.77:0/2994826999 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f90046070 con 0x7f7f9815ebe0 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.287+0000 7f7f89ffb640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.287+0000 7f7f89ffb640 1 -- 172.21.15.77:0/2994826999 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7f900ab1c0 con 0x7f7f9815ebe0 2024-03-06T07:24:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.287+0000 7f7f9d158640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.287+0000 7f7f9d158640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7f94001920 tx=0x7f7f94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.291+0000 7f7f89ffb640 1 -- 172.21.15.77:0/2994826999 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f90079c60 con 0x7f7f9815ebe0 2024-03-06T07:24:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:54.475+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7f9806b180 con 0x7f7f9815ebe0 2024-03-06T07:24:55.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.261+0000 7f7f89ffb640 1 -- 172.21.15.77:0/2994826999 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7f9007dd00 con 0x7f7f9815ebe0 2024-03-06T07:24:55.263 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:24:55.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f7c06b4d0 msgr2=0x7f7f7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7f94001920 tx=0x7f7f94009040 comp rx=0 tx=0).stop 2024-03-06T07:24:55.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f9815ebe0 msgr2=0x7f7f980cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f9815ebe0 0x7f7f980cadb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f7f980c7610 tx=0x7f7f980cec30 comp rx=0 tx=0).stop 2024-03-06T07:24:55.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 shutdown_connections 2024-03-06T07:24:55.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f9815ebe0 0x7f7f980cadb0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f980ccaa0 0x7f7f9815e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 --2- 172.21.15.77:0/2994826999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f840051c0 0x7f7f98156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 >> 172.21.15.77:0/2994826999 conn(0x7f7f980d3ce0 msgr2=0x7f7f980b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 shutdown_connections 2024-03-06T07:24:55.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.263+0000 7f7f9f3bc640 1 -- 172.21.15.77:0/2994826999 wait complete. 2024-03-06T07:24:55.280 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:55.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.382+0000 7f0e80edf640 1 Processor -- start 2024-03-06T07:24:55.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.382+0000 7f0e80edf640 1 -- start start 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e80edf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e7c0d22b0 0x7f0e7c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e80edf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0e7c0b41f0 0x7f0e7c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e80edf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c0b3430 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e80edf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7c0d2c50 con 0x7f0e7c0d22b0 2024-03-06T07:24:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e80edf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7c0b31e0 con 0x7f0e7c0b41f0 2024-03-06T07:24:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e79d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e7c0d22b0 0x7f0e7c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e7a59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0e7c15dfe0 0x7f0e7c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39900/0 (socket says 172.21.15.77:39900) 2024-03-06T07:24:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e7a59c640 1 -- 172.21.15.77:0/647478232 learned_addr learned my addr 172.21.15.77:0/647478232 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.383+0000 7f0e79d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e7c0d22b0 0x7f0e7c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39100/0 (socket says 172.21.15.77:39100) 2024-03-06T07:24:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e79d9b640 1 -- 172.21.15.77:0/647478232 >> v2:172.21.15.158:3300/0 conn(0x7f0e7c0b41f0 msgr2=0x7f0e7c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e7ad9d640 1 --2- 172.21.15.77:0/647478232 >> v2:172.21.15.158:3300/0 conn(0x7f0e7c0b41f0 0x7f0e7c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e79d9b640 1 --2- 172.21.15.77:0/647478232 >> v2:172.21.15.158:3300/0 conn(0x7f0e7c0b41f0 0x7f0e7c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e79d9b640 1 -- 172.21.15.77:0/647478232 >> v1:172.21.15.77:6789/0 conn(0x7f0e7c15dfe0 legacy=0x7f0e7c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e79d9b640 1 -- 172.21.15.77:0/647478232 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c0b3710 con 0x7f0e7c0d22b0 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e7959a640 1 -- 172.21.15.77:0/647478232 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346636680 0 0) 0x7f0e7c0b3430 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e7ad9d640 1 --2- 172.21.15.77:0/647478232 >> v2:172.21.15.158:3300/0 conn(0x7f0e7c0b41f0 0x7f0e7c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e79d9b640 1 --2- 172.21.15.77:0/647478232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c0d22b0 0x7f0e7c0d2710 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0e700097a0 tx=0x7f0e7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a73a4d4b304ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e7959a640 1 -- 172.21.15.77:0/647478232 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70035020 con 0x7f0e7c0d22b0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.384+0000 7f0e7959a640 1 -- 172.21.15.77:0/647478232 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e70031830 con 0x7f0e7c0d22b0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e7959a640 1 -- 172.21.15.77:0/647478232 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70031c00 con 0x7f0e7c0d22b0 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e80edf640 1 -- 172.21.15.77:0/647478232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c0d22b0 msgr2=0x7f0e7c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e80edf640 1 --2- 172.21.15.77:0/647478232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c0d22b0 0x7f0e7c0d2710 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0e700097a0 tx=0x7f0e7002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e80edf640 1 -- 172.21.15.77:0/647478232 shutdown_connections 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e80edf640 1 --2- 172.21.15.77:0/647478232 >> v2:172.21.15.158:3300/0 conn(0x7f0e7c0b41f0 0x7f0e7c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e80edf640 1 --2- 172.21.15.77:0/647478232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c0d22b0 0x7f0e7c0d2710 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e80edf640 1 -- 172.21.15.77:0/647478232 >> 172.21.15.77:0/647478232 conn(0x7f0e7c0d3cd0 msgr2=0x7f0e7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e80edf640 1 -- 172.21.15.77:0/647478232 shutdown_connections 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.385+0000 7f0e80edf640 1 -- 172.21.15.77:0/647478232 wait complete. 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.386+0000 7f0e80edf640 1 Processor -- start 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.386+0000 7f0e80edf640 1 -- start start 2024-03-06T07:24:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e80edf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7c0b41f0 0x7f0e7c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e80edf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e7c0d22b0 0x7f0e7c0b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e80edf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c15dfe0 0x7f0e7c0b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e80edf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c0cfa00 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e80edf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c0b3d60 con 0x7f0e7c0b41f0 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e80edf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c0d1c70 con 0x7f0e7c0d22b0 2024-03-06T07:24:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e7ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c15dfe0 0x7f0e7c0b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e7ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c15dfe0 0x7f0e7c0b7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39104/0 (socket says 172.21.15.77:39104) 2024-03-06T07:24:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e7ad9d640 1 -- 172.21.15.77:0/3325544330 learned_addr learned my addr 172.21.15.77:0/3325544330 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e7a59c640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7c0b41f0 0x7f0e7c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.387+0000 7f0e79d9b640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e7c0d22b0 0x7f0e7c0b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e7ad9d640 1 -- 172.21.15.77:0/3325544330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e7c0d22b0 msgr2=0x7f0e7c0b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e7ad9d640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e7c0d22b0 0x7f0e7c0b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e7ad9d640 1 -- 172.21.15.77:0/3325544330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7c0b41f0 msgr2=0x7f0e7c15e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e7ad9d640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7c0b41f0 0x7f0e7c15e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e7ad9d640 1 -- 172.21.15.77:0/3325544330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c15e900 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e7ad9d640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c15dfe0 0x7f0e7c0b7050 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400c7f0 tx=0x7f0e7400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e6affd640 1 -- 172.21.15.77:0/3325544330 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74015070 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e6affd640 1 -- 172.21.15.77:0/3325544330 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e74007820 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7c15ec30 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e6affd640 1 -- 172.21.15.77:0/3325544330 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74007c50 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.388+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7c15f470 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.389+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7c04f330 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.389+0000 7f0e6affd640 1 -- 172.21.15.77:0/3325544330 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e74018070 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.390+0000 7f0e6affd640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.390+0000 7f0e6affd640 1 -- 172.21.15.77:0/3325544330 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e74089f50 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.390+0000 7f0e7a59c640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.390+0000 7f0e7a59c640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c007ed0 tx=0x7f0e6c007e60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.394+0000 7f0e6affd640 1 -- 172.21.15.77:0/3325544330 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e7c15ec30 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.628+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0e7c0b1e60 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.628+0000 7f0e6affd640 1 -- 172.21.15.77:0/3325544330 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0e740591b0 con 0x7f0e7c15dfe0 2024-03-06T07:24:55.631 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:55.631 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:24:55.631 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:24:55.631 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.631 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:24:55.632 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:24:55.632 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.632 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:55.632 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:55.632 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:55.632 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.632 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.633 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.633 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.633 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:24:55.633 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:24:55.633 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.633 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:55.633 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:55.634 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:55.634 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.634 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.634 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.634 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.634 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:24:55.634 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:24:55.635 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.635 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:55.635 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:55.635 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:55.635 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.635 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.636 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.636 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.636 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:24:55.636 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:24:55.636 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.636 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:55.636 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:55.637 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:55.637 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.637 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.637 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.637 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.637 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:24:55.637 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:24:55.638 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:24:55.639 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.639 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:55.639 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:55.639 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.639 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.639 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.639 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.639 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:24:55.640 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:24:55.640 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.640 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:55.640 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:55.640 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.640 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.640 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.641 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.641 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:24:55.641 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:24:55.641 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.641 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:55.641 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:55.641 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.642 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.642 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.642 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.642 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:24:55.642 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:24:55.643 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.643 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:55.643 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:55.643 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.643 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.643 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.644 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.644 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:24:55.644 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:24:55.644 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.644 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:55.644 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:55.644 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.644 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.645 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.646 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.646 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.646 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:24:55.646 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:24:55.646 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.646 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:55.647 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:24:55.647 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.647 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.647 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.647 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.647 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:24:55.648 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:24:55.648 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.648 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:55.648 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-03-06T07:24:55.648 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:55.648 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.648 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.649 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.649 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:24:55.649 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:24:55.649 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.649 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:55.649 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:24:55.650 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:24:55.650 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:24:55.650 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.650 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.650 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.650 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:24:55.650 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:24:55.651 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.651 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:55.651 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:24:55.651 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:24:55.651 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:24:55.651 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.652 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.652 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.652 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:24:55.652 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:24:55.652 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.652 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:24:55.653 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:24:55.653 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:24:55.653 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:24:55.653 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.653 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.653 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.653 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:24:55.654 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:24:55.654 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.654 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:24:55.654 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:24:55.654 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:24:55.654 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.654 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.655 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.655 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.655 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:24:55.655 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:24:55.655 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.655 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:24:55.655 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:55.656 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.656 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.656 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:24:55.656 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:55.656 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:24:55.656 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:24:55.657 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:24:55.657 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:24:55.657 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:55.657 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:24:55.657 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.657 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:55.657 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:24:55.658 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:24:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5c06b4d0 msgr2=0x7f0e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c007ed0 tx=0x7f0e6c007e60 comp rx=0 tx=0).stop 2024-03-06T07:24:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c15dfe0 msgr2=0x7f0e7c0b7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c15dfe0 0x7f0e7c0b7050 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400c7f0 tx=0x7f0e7400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 shutdown_connections 2024-03-06T07:24:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5c06b4d0 0x7f0e5c06d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e7c15dfe0 0x7f0e7c0b7050 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e7c0d22b0 0x7f0e7c0b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 --2- 172.21.15.77:0/3325544330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7c0b41f0 0x7f0e7c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 >> 172.21.15.77:0/3325544330 conn(0x7f0e7c0d3cd0 msgr2=0x7f0e7c06cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 shutdown_connections 2024-03-06T07:24:55.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.630+0000 7f0e80edf640 1 -- 172.21.15.77:0/3325544330 wait complete. 2024-03-06T07:24:55.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:55.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.761+0000 7f9bec481640 1 Processor -- start 2024-03-06T07:24:55.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.761+0000 7f9bec481640 1 -- start start 2024-03-06T07:24:55.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9bec481640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9be40cce40 0x7f9be415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9bec481640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be4071f40 con 0x7f9be40cd7a0 2024-03-06T07:24:55.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9bec481640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be4070a70 con 0x7f9be41515e0 2024-03-06T07:24:55.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9bec481640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be40cd2f0 con 0x7f9be40cce40 2024-03-06T07:24:55.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9beaa1e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9be40cd7a0 0x7f9be40cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39912/0 (socket says 172.21.15.77:39912) 2024-03-06T07:24:55.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9beaa1e640 1 -- 172.21.15.77:0/703841852 learned_addr learned my addr 172.21.15.77:0/703841852 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:55.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557056588 0 0) 0x7f9be4071f40 con 0x7f9be40cd7a0 2024-03-06T07:24:55.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd0003440 con 0x7f9be40cd7a0 2024-03-06T07:24:55.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.762+0000 7f9be9a1c640 1 --2- 172.21.15.77:0/703841852 >> v2:172.21.15.158:3300/0 conn(0x7f9be40cce40 0x7f9be415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4203613954 0 0) 0x7f9bd0003440 con 0x7f9be40cd7a0 2024-03-06T07:24:55.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 >> v2:172.21.15.158:3300/0 conn(0x7f9be40cce40 msgr2=0x7f9be415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 --2- 172.21.15.77:0/703841852 >> v2:172.21.15.158:3300/0 conn(0x7f9be40cce40 0x7f9be415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 >> v1:172.21.15.158:6789/0 conn(0x7f9be41515e0 legacy=0x7f9be4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4070d50 con 0x7f9be40cd7a0 2024-03-06T07:24:55.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f9bdc001690 con 0x7f9be40cd7a0 2024-03-06T07:24:55.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 >> v1:172.21.15.77:6789/0 conn(0x7f9be40cd7a0 legacy=0x7f9be40cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd0003aa0 0x7f9bd0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9bd000acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bd000b4c0 0x7f9bd000d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be40cd2f0 con 0x7f9bd0008850 2024-03-06T07:24:55.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd000e260 con 0x7f9bd0003aa0 2024-03-06T07:24:55.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd000e460 con 0x7f9bd000b4c0 2024-03-06T07:24:55.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be9a1c640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9bd000acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9bea21d640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd0003aa0 0x7f9bd0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9beaa1e640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bd000b4c0 0x7f9bd000d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be9a1c640 1 -- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bd000b4c0 msgr2=0x7f9bd000d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be9a1c640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bd000b4c0 0x7f9bd000d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be9a1c640 1 -- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd0003aa0 msgr2=0x7f9bd0003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be9a1c640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9bd0003aa0 0x7f9bd0003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9be9a1c640 1 -- 172.21.15.77:0/703841852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd0003440 con 0x7f9bd0008850 2024-03-06T07:24:55.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9beaa1e640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9bd000b4c0 0x7f9bd000d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:55.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.763+0000 7f9beaa1e640 1 -- 172.21.15.77:0/703841852 reap_dead start 2024-03-06T07:24:55.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.764+0000 7f9be9a1c640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9bd000acc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009ac0 tx=0x7f9bd0003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdb69e1be9cb796 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.764+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4039070 con 0x7f9bd0008850 2024-03-06T07:24:55.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.764+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd402c6b0 con 0x7f9bd0008850 2024-03-06T07:24:55.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.764+0000 7f9be921b640 1 -- 172.21.15.77:0/703841852 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4009690 con 0x7f9bd0008850 2024-03-06T07:24:55.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.765+0000 7f9bec481640 1 -- 172.21.15.77:0/703841852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 msgr2=0x7f9bd000acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.765+0000 7f9bec481640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9bd000acc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009ac0 tx=0x7f9bd0003730 comp rx=0 tx=0).stop 2024-03-06T07:24:55.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.765+0000 7f9bec481640 1 -- 172.21.15.77:0/703841852 shutdown_connections 2024-03-06T07:24:55.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.765+0000 7f9bec481640 1 --2- 172.21.15.77:0/703841852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9bd000acc0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.765+0000 7f9bec481640 1 -- 172.21.15.77:0/703841852 >> 172.21.15.77:0/703841852 conn(0x7f9be40cf8a0 msgr2=0x7f9be40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.765+0000 7f9bec481640 1 -- 172.21.15.77:0/703841852 shutdown_connections 2024-03-06T07:24:55.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.765+0000 7f9bec481640 1 -- 172.21.15.77:0/703841852 wait complete. 2024-03-06T07:24:55.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.766+0000 7f9bec481640 1 Processor -- start 2024-03-06T07:24:55.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.766+0000 7f9bec481640 1 -- start start 2024-03-06T07:24:55.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bec481640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9be413c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bec481640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9be4153f30 0x7f9be41543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bec481640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9be4151a20 0x7f9be4151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bec481640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be415b430 con 0x7f9bd0008850 2024-03-06T07:24:55.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bec481640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be4071310 con 0x7f9be4153f30 2024-03-06T07:24:55.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bec481640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be415b5b0 con 0x7f9be4151a20 2024-03-06T07:24:55.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9be413c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9be413c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39122/0 (socket says 172.21.15.77:39122) 2024-03-06T07:24:55.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 -- 172.21.15.77:0/1286779874 learned_addr learned my addr 172.21.15.77:0/1286779874 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:55.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9beaa1e640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9be4151a20 0x7f9be4151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9be9a1c640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9be4153f30 0x7f9be41543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 -- 172.21.15.77:0/1286779874 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9be4151a20 msgr2=0x7f9be4151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9be4151a20 0x7f9be4151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 -- 172.21.15.77:0/1286779874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9be4153f30 msgr2=0x7f9be41543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9be4153f30 0x7f9be41543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 -- 172.21.15.77:0/1286779874 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4152750 con 0x7f9bd0008850 2024-03-06T07:24:55.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bea21d640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9be413c590 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f9be000fb90 tx=0x7f9be000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bdaffd640 1 -- 172.21.15.77:0/1286779874 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be000fe90 con 0x7f9bd0008850 2024-03-06T07:24:55.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bdaffd640 1 -- 172.21.15.77:0/1286779874 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be0018070 con 0x7f9bd0008850 2024-03-06T07:24:55.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be4152a30 con 0x7f9bd0008850 2024-03-06T07:24:55.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bdaffd640 1 -- 172.21.15.77:0/1286779874 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be000de50 con 0x7f9bd0008850 2024-03-06T07:24:55.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.767+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be4152ff0 con 0x7f9bd0008850 2024-03-06T07:24:55.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.768+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be4068840 con 0x7f9bd0008850 2024-03-06T07:24:55.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.769+0000 7f9bdaffd640 1 -- 172.21.15.77:0/1286779874 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9be0003970 con 0x7f9bd0008850 2024-03-06T07:24:55.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.769+0000 7f9bdaffd640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bc806b670 0x7f9bc806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.769+0000 7f9bdaffd640 1 -- 172.21.15.77:0/1286779874 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9be008a090 con 0x7f9bd0008850 2024-03-06T07:24:55.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.769+0000 7f9be9a1c640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bc806b670 0x7f9bc806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.770+0000 7f9be9a1c640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bc806b670 0x7f9bc806db30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4036000 tx=0x7f9bd4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.773+0000 7f9bdaffd640 1 -- 172.21.15.77:0/1286779874 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9be001a310 con 0x7f9bd0008850 2024-03-06T07:24:55.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.946+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9be4144450 con 0x7f9bd0008850 2024-03-06T07:24:55.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.947+0000 7f9bdaffd640 1 -- 172.21.15.77:0/1286779874 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f9be4144450 con 0x7f9bd0008850 2024-03-06T07:24:55.949 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:55.950 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":65,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:55.264683+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bc806b670 msgr2=0x7f9bc806db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bc806b670 0x7f9bc806db30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4036000 tx=0x7f9bd4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 msgr2=0x7f9be413c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9be413c590 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f9be000fb90 tx=0x7f9be000d520 comp rx=0 tx=0).stop 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 shutdown_connections 2024-03-06T07:24:55.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9be4151a20 0x7f9be4151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9be4153f30 0x7f9be41543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9bc806b670 0x7f9bc806db30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 --2- 172.21.15.77:0/1286779874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bd0008850 0x7f9be413c590 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 >> 172.21.15.77:0/1286779874 conn(0x7f9be40cf8a0 msgr2=0x7f9be4072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 shutdown_connections 2024-03-06T07:24:55.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:55.949+0000 7f9bec481640 1 -- 172.21.15.77:0/1286779874 wait complete. 2024-03-06T07:24:55.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:55.963 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:56.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.065+0000 7f6c11047640 1 Processor -- start 2024-03-06T07:24:56.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.065+0000 7f6c11047640 1 -- start start 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.065+0000 7f6c11047640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c0c1614d0 0x7f6c0c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.065+0000 7f6c11047640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6c0c0d24b0 0x7f6c0c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.065+0000 7f6c11047640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c1618b0 con 0x7f6c0c0d1910 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.065+0000 7f6c11047640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0c0cf400 con 0x7f6c0c1614d0 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.065+0000 7f6c11047640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0c0cf6a0 con 0x7f6c0c0d24b0 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.065+0000 7f6c0ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c0c1614d0 0x7f6c0c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c03fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6c0c0d1910 0x7f6c0c0d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39926/0 (socket says 172.21.15.77:39926) 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c0ad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c0c1614d0 0x7f6c0c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39124/0 (socket says 172.21.15.77:39124) 2024-03-06T07:24:56.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c03fff640 1 -- 172.21.15.77:0/600586088 learned_addr learned my addr 172.21.15.77:0/600586088 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c0b59e640 1 --2- 172.21.15.77:0/600586088 >> v2:172.21.15.158:3300/0 conn(0x7f6c0c0d24b0 0x7f6c0c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c0ad9d640 1 -- 172.21.15.77:0/600586088 >> v2:172.21.15.158:3300/0 conn(0x7f6c0c0d24b0 msgr2=0x7f6c0c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c0ad9d640 1 --2- 172.21.15.77:0/600586088 >> v2:172.21.15.158:3300/0 conn(0x7f6c0c0d24b0 0x7f6c0c0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c0ad9d640 1 -- 172.21.15.77:0/600586088 >> v1:172.21.15.77:6789/0 conn(0x7f6c0c0d1910 legacy=0x7f6c0c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c0a59c640 1 -- 172.21.15.77:0/600586088 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655356114 0 0) 0x7f6c0c1618b0 con 0x7f6c0c0d1910 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c0ad9d640 1 -- 172.21.15.77:0/600586088 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c0cf980 con 0x7f6c0c1614d0 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.066+0000 7f6c0b59e640 1 --2- 172.21.15.77:0/600586088 >> v2:172.21.15.158:3300/0 conn(0x7f6c0c0d24b0 0x7f6c0c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c0ad9d640 1 --2- 172.21.15.77:0/600586088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c0d4860 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0b2a70 tx=0x7f6bfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409ddab6f48de521 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c0a59c640 1 -- 172.21.15.77:0/600586088 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc035020 con 0x7f6c0c1614d0 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c0a59c640 1 -- 172.21.15.77:0/600586088 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc031830 con 0x7f6c0c1614d0 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c0a59c640 1 -- 172.21.15.77:0/600586088 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc031c00 con 0x7f6c0c1614d0 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c11047640 1 -- 172.21.15.77:0/600586088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 msgr2=0x7f6c0c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c11047640 1 --2- 172.21.15.77:0/600586088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c0d4860 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0b2a70 tx=0x7f6bfc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c11047640 1 -- 172.21.15.77:0/600586088 shutdown_connections 2024-03-06T07:24:56.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c11047640 1 --2- 172.21.15.77:0/600586088 >> v2:172.21.15.158:3300/0 conn(0x7f6c0c0d24b0 0x7f6c0c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c11047640 1 --2- 172.21.15.77:0/600586088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c0d4860 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c11047640 1 -- 172.21.15.77:0/600586088 >> 172.21.15.77:0/600586088 conn(0x7f6c0c0d3d40 msgr2=0x7f6c0c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c11047640 1 -- 172.21.15.77:0/600586088 shutdown_connections 2024-03-06T07:24:56.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.067+0000 7f6c11047640 1 -- 172.21.15.77:0/600586088 wait complete. 2024-03-06T07:24:56.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c11047640 1 Processor -- start 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c11047640 1 -- start start 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c11047640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c0c0d1910 0x7f6c0c1f8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c11047640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0c0d24b0 0x7f6c0c0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c11047640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c06cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c11047640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0b1370 con 0x7f6c0c1614d0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c11047640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0b0910 con 0x7f6c0c0d24b0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c11047640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0b14f0 con 0x7f6c0c0d1910 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c0b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c06cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c0b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c06cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39138/0 (socket says 172.21.15.77:39138) 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.069+0000 7f6c0b59e640 1 -- 172.21.15.77:0/563239040 learned_addr learned my addr 172.21.15.77:0/563239040 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c0ad9d640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c0c0d1910 0x7f6c0c1f8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c03fff640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0c0d24b0 0x7f6c0c0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c0b59e640 1 -- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c0c0d1910 msgr2=0x7f6c0c1f8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c0b59e640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c0c0d1910 0x7f6c0c1f8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c0b59e640 1 -- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0c0d24b0 msgr2=0x7f6c0c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c0b59e640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0c0d24b0 0x7f6c0c0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c0b59e640 1 -- 172.21.15.77:0/563239040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c1f76b0 con 0x7f6c0c1614d0 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c03fff640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0c0d24b0 0x7f6c0c0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c0b59e640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c06cad0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400cf20 tx=0x7f6c0400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c037fe640 1 -- 172.21.15.77:0/563239040 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0401a070 con 0x7f6c0c1614d0 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c037fe640 1 -- 172.21.15.77:0/563239040 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0400d840 con 0x7f6c0c1614d0 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c0cd420 con 0x7f6c0c1614d0 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c037fe640 1 -- 172.21.15.77:0/563239040 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400dc30 con 0x7f6c0c1614d0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.070+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c1f7970 con 0x7f6c0c1614d0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.071+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0c0b7b10 con 0x7f6c0c1614d0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.071+0000 7f6c037fe640 1 -- 172.21.15.77:0/563239040 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c0401d070 con 0x7f6c0c1614d0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.072+0000 7f6c037fe640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bec06b4d0 0x7f6bec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.072+0000 7f6c0ad9d640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bec06b4d0 0x7f6bec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.072+0000 7f6c037fe640 1 -- 172.21.15.77:0/563239040 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6c0408a760 con 0x7f6c0c1614d0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.072+0000 7f6c0ad9d640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bec06b4d0 0x7f6bec06d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc036000 tx=0x7f6bfc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.076+0000 7f6c037fe640 1 -- 172.21.15.77:0/563239040 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c04059930 con 0x7f6c0c1614d0 2024-03-06T07:24:56.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.248+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c0c068ff0 con 0x7f6c0c1614d0 2024-03-06T07:24:56.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.248+0000 7f6c037fe640 1 -- 172.21.15.77:0/563239040 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c04059230 con 0x7f6c0c1614d0 2024-03-06T07:24:56.250 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:24:56.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bec06b4d0 msgr2=0x7f6bec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bec06b4d0 0x7f6bec06d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc036000 tx=0x7f6bfc03c040 comp rx=0 tx=0).stop 2024-03-06T07:24:56.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 msgr2=0x7f6c0c06cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c06cad0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400cf20 tx=0x7f6c0400a7d0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 shutdown_connections 2024-03-06T07:24:56.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bec06b4d0 0x7f6bec06d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c1614d0 0x7f6c0c06cad0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6c0c0d24b0 0x7f6c0c0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 --2- 172.21.15.77:0/563239040 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6c0c0d1910 0x7f6c0c1f8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.250+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 >> 172.21.15.77:0/563239040 conn(0x7f6c0c0d3d40 msgr2=0x7f6c0c0b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.251+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 shutdown_connections 2024-03-06T07:24:56.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.251+0000 7f6c11047640 1 -- 172.21.15.77:0/563239040 wait complete. 2024-03-06T07:24:56.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:56.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.362+0000 7fbc3a42a640 1 Processor -- start 2024-03-06T07:24:56.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.362+0000 7fbc3a42a640 1 -- start start 2024-03-06T07:24:56.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.363+0000 7fbc3a42a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340d22b0 0x7fbc340d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.363+0000 7fbc3a42a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc340b3430 con 0x7fbc340b41f0 2024-03-06T07:24:56.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.363+0000 7fbc3a42a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc340b3710 con 0x7fbc3415dfe0 2024-03-06T07:24:56.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.363+0000 7fbc3a42a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340d2c50 con 0x7fbc340d22b0 2024-03-06T07:24:56.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.363+0000 7fbc389c7640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbc340b41f0 0x7fbc340b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39934/0 (socket says 172.21.15.77:39934) 2024-03-06T07:24:56.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.363+0000 7fbc389c7640 1 -- 172.21.15.77:0/2150812537 learned_addr learned my addr 172.21.15.77:0/2150812537 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:56.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.363+0000 7fbc337fe640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340d22b0 0x7fbc340d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215337440 0 0) 0x7fbc340b3430 con 0x7fbc340b41f0 2024-03-06T07:24:56.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc1c003520 con 0x7fbc340b41f0 2024-03-06T07:24:56.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2806292121 0 0) 0x7fbc1c003520 con 0x7fbc340b41f0 2024-03-06T07:24:56.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340d22b0 msgr2=0x7fbc340d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340d22b0 0x7fbc340d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 >> v1:172.21.15.158:6789/0 conn(0x7fbc3415dfe0 legacy=0x7fbc340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc340b39f0 con 0x7fbc340b41f0 2024-03-06T07:24:56.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fbc2c0019f0 con 0x7fbc340b41f0 2024-03-06T07:24:56.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 >> v1:172.21.15.77:6789/0 conn(0x7fbc340b41f0 legacy=0x7fbc340b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc1c0039e0 0x7fbc1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c004390 0x7fbc1c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340d22b0 0x7fbc1c0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c00b9c0 con 0x7fbc1c0039e0 2024-03-06T07:24:56.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c00bb40 con 0x7fbc1c004390 2024-03-06T07:24:56.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c00bd40 con 0x7fbc340d22b0 2024-03-06T07:24:56.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc33fff640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc1c0039e0 0x7fbc1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc337fe640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c004390 0x7fbc1c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc389c7640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340d22b0 0x7fbc1c0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc33fff640 1 -- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340d22b0 msgr2=0x7fbc1c0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc33fff640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340d22b0 0x7fbc1c0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc33fff640 1 -- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c004390 msgr2=0x7fbc1c004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc33fff640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c004390 0x7fbc1c004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc33fff640 1 -- 172.21.15.77:0/2150812537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c003520 con 0x7fbc1c0039e0 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.364+0000 7fbc337fe640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c004390 0x7fbc1c004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.365+0000 7fbc33fff640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc1c0039e0 0x7fbc1c003dc0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fbc24001860 tx=0x7fbc1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984c7a1c289a4981 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.365+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2402f050 con 0x7fbc1c0039e0 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.365+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc24033650 con 0x7fbc1c0039e0 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.365+0000 7fbc32ffd640 1 -- 172.21.15.77:0/2150812537 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24001430 con 0x7fbc1c0039e0 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.366+0000 7fbc3a42a640 1 -- 172.21.15.77:0/2150812537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc1c0039e0 msgr2=0x7fbc1c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.366+0000 7fbc3a42a640 1 --2- 172.21.15.77:0/2150812537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc1c0039e0 0x7fbc1c003dc0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fbc24001860 tx=0x7fbc1c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.366+0000 7fbc389c7640 1 -- 172.21.15.77:0/2150812537 reap_dead start 2024-03-06T07:24:56.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.366+0000 7fbc3a42a640 1 -- 172.21.15.77:0/2150812537 shutdown_connections 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.366+0000 7fbc3a42a640 1 -- 172.21.15.77:0/2150812537 >> 172.21.15.77:0/2150812537 conn(0x7fbc340d3cd0 msgr2=0x7fbc340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.366+0000 7fbc3a42a640 1 -- 172.21.15.77:0/2150812537 shutdown_connections 2024-03-06T07:24:56.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.366+0000 7fbc3a42a640 1 -- 172.21.15.77:0/2150812537 wait complete. 2024-03-06T07:24:56.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.367+0000 7fbc3a42a640 1 Processor -- start 2024-03-06T07:24:56.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.367+0000 7fbc3a42a640 1 -- start start 2024-03-06T07:24:56.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc3a42a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c0039e0 0x7fbc341530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc3a42a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc340b57c0 0x7fbc340b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc3a42a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340b5c40 0x7fbc340b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc3a42a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340b3d60 con 0x7fbc340b57c0 2024-03-06T07:24:56.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc3a42a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340b31e0 con 0x7fbc1c0039e0 2024-03-06T07:24:56.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc3a42a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340d1c70 con 0x7fbc340b5c40 2024-03-06T07:24:56.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc340b57c0 0x7fbc340b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc340b57c0 0x7fbc340b7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39166/0 (socket says 172.21.15.77:39166) 2024-03-06T07:24:56.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 -- 172.21.15.77:0/1576652126 learned_addr learned my addr 172.21.15.77:0/1576652126 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:56.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 -- 172.21.15.77:0/1576652126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340b5c40 msgr2=0x7fbc340b8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc389c7640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340b5c40 0x7fbc340b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc33fff640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c0039e0 0x7fbc341530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340b5c40 0x7fbc340b8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 -- 172.21.15.77:0/1576652126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c0039e0 msgr2=0x7fbc341530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c0039e0 0x7fbc341530d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 -- 172.21.15.77:0/1576652126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc340b6210 con 0x7fbc340b57c0 2024-03-06T07:24:56.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc337fe640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc340b57c0 0x7fbc340b7fe0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fbc28011700 tx=0x7fbc28011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc30ff9640 1 -- 172.21.15.77:0/1576652126 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2800f9c0 con 0x7fbc340b57c0 2024-03-06T07:24:56.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc30ff9640 1 -- 172.21.15.77:0/1576652126 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2800fb60 con 0x7fbc340b57c0 2024-03-06T07:24:56.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc340b64f0 con 0x7fbc340b57c0 2024-03-06T07:24:56.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc30ff9640 1 -- 172.21.15.77:0/1576652126 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28015460 con 0x7fbc340b57c0 2024-03-06T07:24:56.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.368+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc340b6ab0 con 0x7fbc340b57c0 2024-03-06T07:24:56.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.369+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3404f330 con 0x7fbc340b57c0 2024-03-06T07:24:56.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.370+0000 7fbc30ff9640 1 -- 172.21.15.77:0/1576652126 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc28015600 con 0x7fbc340b57c0 2024-03-06T07:24:56.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.370+0000 7fbc30ff9640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.370+0000 7fbc30ff9640 1 -- 172.21.15.77:0/1576652126 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbc2808a7b0 con 0x7fbc340b57c0 2024-03-06T07:24:56.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.370+0000 7fbc33fff640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.371+0000 7fbc33fff640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbc24001860 tx=0x7fbc24031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.375+0000 7fbc30ff9640 1 -- 172.21.15.77:0/1576652126 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc28019030 con 0x7fbc340b57c0 2024-03-06T07:24:56.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.549+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbc340b9aa0 con 0x7fbc340b57c0 2024-03-06T07:24:56.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.549+0000 7fbc30ff9640 1 -- 172.21.15.77:0/1576652126 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v65) v1 ==== 84+0+65 (secure 0 0 0) 0x7fbc28059980 con 0x7fbc340b57c0 2024-03-06T07:24:56.551 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:56.551 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:24:56.551 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:24:56.551 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:24:56.551 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:24:56.551 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:24:56.552 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:24:56.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc2006b4d0 msgr2=0x7fbc2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbc24001860 tx=0x7fbc24031000 comp rx=0 tx=0).stop 2024-03-06T07:24:56.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc340b57c0 msgr2=0x7fbc340b7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc340b57c0 0x7fbc340b7fe0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fbc28011700 tx=0x7fbc28011bc0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 shutdown_connections 2024-03-06T07:24:56.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc340b57c0 0x7fbc340b7fe0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbc340b5c40 0x7fbc340b8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 --2- 172.21.15.77:0/1576652126 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c0039e0 0x7fbc341530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 >> 172.21.15.77:0/1576652126 conn(0x7fbc340d3cd0 msgr2=0x7fbc341603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 shutdown_connections 2024-03-06T07:24:56.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.551+0000 7fbc3a42a640 1 -- 172.21.15.77:0/1576652126 wait complete. 2024-03-06T07:24:56.564 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:56.564 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:56.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.667+0000 7fc6198b5640 1 Processor -- start 2024-03-06T07:24:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.667+0000 7fc6198b5640 1 -- start start 2024-03-06T07:24:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.667+0000 7fc6198b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 0x7fc61406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.667+0000 7fc6198b5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6140cc6e0 0x7fc6141601a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.667+0000 7fc6198b5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61406f270 con 0x7fc61415bd60 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.667+0000 7fc6198b5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61415f750 con 0x7fc6140cc6e0 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.667+0000 7fc6198b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61415fd10 con 0x7fc6140b57b0 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.668+0000 7fc6137fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6140cc6e0 0x7fc6141601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.668+0000 7fc6137fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6140cc6e0 0x7fc6141601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39182/0 (socket says 172.21.15.77:39182) 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.668+0000 7fc6137fe640 1 -- 172.21.15.77:0/2227497052 learned_addr learned my addr 172.21.15.77:0/2227497052 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.668+0000 7fc6127fc640 1 --2- 172.21.15.77:0/2227497052 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 0x7fc61406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.668+0000 7fc6137fe640 1 -- 172.21.15.77:0/2227497052 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 msgr2=0x7fc61406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.668+0000 7fc6137fe640 1 --2- 172.21.15.77:0/2227497052 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 0x7fc61406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.668+0000 7fc6137fe640 1 -- 172.21.15.77:0/2227497052 >> v1:172.21.15.158:6789/0 conn(0x7fc61415bd60 legacy=0x7fc6140b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.668+0000 7fc6137fe640 1 -- 172.21.15.77:0/2227497052 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61406f550 con 0x7fc6140cc6e0 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6137fe640 1 --2- 172.21.15.77:0/2227497052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140cc6e0 0x7fc6141601a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009e50 tx=0x7fc60c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8eec1d602374455 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc611ffb640 1 -- 172.21.15.77:0/2227497052 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c02c9e0 con 0x7fc6140cc6e0 2024-03-06T07:24:56.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc611ffb640 1 -- 172.21.15.77:0/2227497052 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c02cb80 con 0x7fc6140cc6e0 2024-03-06T07:24:56.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc611ffb640 1 -- 172.21.15.77:0/2227497052 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0343f0 con 0x7fc6140cc6e0 2024-03-06T07:24:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6198b5640 1 -- 172.21.15.77:0/2227497052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140cc6e0 msgr2=0x7fc6141601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6198b5640 1 --2- 172.21.15.77:0/2227497052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140cc6e0 0x7fc6141601a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009e50 tx=0x7fc60c02be40 comp rx=0 tx=0).stop 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6198b5640 1 -- 172.21.15.77:0/2227497052 shutdown_connections 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6198b5640 1 --2- 172.21.15.77:0/2227497052 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 0x7fc61406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6198b5640 1 --2- 172.21.15.77:0/2227497052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6140cc6e0 0x7fc6141601a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6198b5640 1 -- 172.21.15.77:0/2227497052 >> 172.21.15.77:0/2227497052 conn(0x7fc6140b1a90 msgr2=0x7fc6140b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6198b5640 1 -- 172.21.15.77:0/2227497052 shutdown_connections 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.669+0000 7fc6198b5640 1 -- 172.21.15.77:0/2227497052 wait complete. 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.670+0000 7fc6198b5640 1 Processor -- start 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6198b5640 1 -- start start 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6198b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 0x7fc6141e61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6198b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6140cc6e0 0x7fc614157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6198b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61415bd60 0x7fc6141415b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6198b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140ce140 con 0x7fc61415bd60 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6198b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6141606e0 con 0x7fc6140cc6e0 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6198b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140ce2c0 con 0x7fc6140b57b0 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61415bd60 0x7fc6141415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61415bd60 0x7fc6141415b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39194/0 (socket says 172.21.15.77:39194) 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc6137fe640 1 -- 172.21.15.77:0/233846932 learned_addr learned my addr 172.21.15.77:0/233846932 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.671+0000 7fc612ffd640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 0x7fc6141e61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6127fc640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6140cc6e0 0x7fc614157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6137fe640 1 -- 172.21.15.77:0/233846932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 msgr2=0x7fc6141e61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6137fe640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 0x7fc6141e61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6137fe640 1 -- 172.21.15.77:0/233846932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6140cc6e0 msgr2=0x7fc614157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6137fe640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6140cc6e0 0x7fc614157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6137fe640 1 -- 172.21.15.77:0/233846932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614158670 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6137fe640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61415bd60 0x7fc6141415b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc60c036040 tx=0x7fc60c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc60bfff640 1 -- 172.21.15.77:0/233846932 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c043020 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc614158950 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc60bfff640 1 -- 172.21.15.77:0/233846932 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c03e450 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc60bfff640 1 -- 172.21.15.77:0/233846932 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c03e820 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.672+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc614158f10 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.673+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61404f330 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.673+0000 7fc60bfff640 1 -- 172.21.15.77:0/233846932 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc60c046020 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.674+0000 7fc60bfff640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f406b500 0x7fc5f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.674+0000 7fc60bfff640 1 -- 172.21.15.77:0/233846932 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc60c0ac6d0 con 0x7fc61415bd60 2024-03-06T07:24:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.674+0000 7fc612ffd640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f406b500 0x7fc5f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.674+0000 7fc612ffd640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f406b500 0x7fc5f406d9c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0016e0 tx=0x7fc5fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.679+0000 7fc60bfff640 1 -- 172.21.15.77:0/233846932 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc60c04d030 con 0x7fc61415bd60 2024-03-06T07:24:56.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:56.849+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc614072d30 con 0x7fc61415bd60 2024-03-06T07:24:57.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.268+0000 7fc60bfff640 1 -- 172.21.15.77:0/233846932 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v66) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc60c07b8b0 con 0x7fc61415bd60 2024-03-06T07:24:57.270 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f406b500 msgr2=0x7fc5f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f406b500 0x7fc5f406d9c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0016e0 tx=0x7fc5fc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61415bd60 msgr2=0x7fc6141415b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61415bd60 0x7fc6141415b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc60c036040 tx=0x7fc60c004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 shutdown_connections 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc5f406b500 0x7fc5f406d9c0 secure :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0016e0 tx=0x7fc5fc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61415bd60 0x7fc6141415b0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6140cc6e0 0x7fc614157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 --2- 172.21.15.77:0/233846932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b57b0 0x7fc6141e61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 >> 172.21.15.77:0/233846932 conn(0x7fc6140b1a90 msgr2=0x7fc6140ce680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 shutdown_connections 2024-03-06T07:24:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.271+0000 7fc6198b5640 1 -- 172.21.15.77:0/233846932 wait complete. 2024-03-06T07:24:57.286 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:57.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.392+0000 7f798899a640 1 Processor -- start 2024-03-06T07:24:57.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.392+0000 7f798899a640 1 -- start start 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f798899a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79840719e0 0x7f79840cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f798899a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f798406ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f798899a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79840b3320 con 0x7f798415a220 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f798899a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79840cf610 con 0x7f79840719e0 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f798899a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79840728d0 con 0x7f798406c9f0 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f7981d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79840719e0 0x7f79840cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f7981d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79840719e0 0x7f79840cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39198/0 (socket says 172.21.15.77:39198) 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f7981d9b640 1 -- 172.21.15.77:0/2736915290 learned_addr learned my addr 172.21.15.77:0/2736915290 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:57.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f7981d9b640 1 -- 172.21.15.77:0/2736915290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 msgr2=0x7f798406ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f7982d9d640 1 --2- 172.21.15.77:0/2736915290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f798406ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f7981d9b640 1 --2- 172.21.15.77:0/2736915290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f798406ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f7981d9b640 1 -- 172.21.15.77:0/2736915290 >> v1:172.21.15.158:6789/0 conn(0x7f798415a220 legacy=0x7f79840712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.393+0000 7f7981d9b640 1 -- 172.21.15.77:0/2736915290 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79840b3590 con 0x7f79840719e0 2024-03-06T07:24:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.394+0000 7f7981d9b640 1 --2- 172.21.15.77:0/2736915290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79840719e0 0x7f79840cf0d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7978009870 tx=0x7f797802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a878ea3892d42a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.394+0000 7f798159a640 1 -- 172.21.15.77:0/2736915290 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780317c0 con 0x7f79840719e0 2024-03-06T07:24:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.394+0000 7f798159a640 1 -- 172.21.15.77:0/2736915290 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978037070 con 0x7f79840719e0 2024-03-06T07:24:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.394+0000 7f798159a640 1 -- 172.21.15.77:0/2736915290 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978031b80 con 0x7f79840719e0 2024-03-06T07:24:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.395+0000 7f798899a640 1 -- 172.21.15.77:0/2736915290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79840719e0 msgr2=0x7f79840cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.395+0000 7f798899a640 1 --2- 172.21.15.77:0/2736915290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79840719e0 0x7f79840cf0d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7978009870 tx=0x7f797802b860 comp rx=0 tx=0).stop 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.395+0000 7f798899a640 1 -- 172.21.15.77:0/2736915290 shutdown_connections 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.395+0000 7f798899a640 1 --2- 172.21.15.77:0/2736915290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f798406ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.395+0000 7f798899a640 1 --2- 172.21.15.77:0/2736915290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79840719e0 0x7f79840cf0d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.395+0000 7f798899a640 1 -- 172.21.15.77:0/2736915290 >> 172.21.15.77:0/2736915290 conn(0x7f798406b190 msgr2=0x7f798406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.395+0000 7f798899a640 1 -- 172.21.15.77:0/2736915290 shutdown_connections 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.395+0000 7f798899a640 1 -- 172.21.15.77:0/2736915290 wait complete. 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.396+0000 7f798899a640 1 Processor -- start 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.396+0000 7f798899a640 1 -- start start 2024-03-06T07:24:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f798899a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f79840acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f798899a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79840719e0 0x7f7984152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f798899a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f798415a220 0x7f798415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f798899a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984072530 con 0x7f798415a220 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f798899a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984072030 con 0x7f79840719e0 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f798899a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79840726b0 con 0x7f798406c9f0 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f798415a220 0x7f798415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f798415a220 0x7f798415ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39204/0 (socket says 172.21.15.77:39204) 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 -- 172.21.15.77:0/4267760544 learned_addr learned my addr 172.21.15.77:0/4267760544 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 -- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 msgr2=0x7f79840acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7981d9b640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79840719e0 0x7f7984152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f798259c640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f79840acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f79840acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 -- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79840719e0 msgr2=0x7f7984152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79840719e0 0x7f7984152370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 -- 172.21.15.77:0/4267760544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79840ad4c0 con 0x7f798415a220 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7981d9b640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79840719e0 0x7f7984152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f798259c640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f79840acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.397+0000 7f7982d9d640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f798415a220 0x7f798415ca40 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f797c00c7f0 tx=0x7f797c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.398+0000 7f7972ffd640 1 -- 172.21.15.77:0/4267760544 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c015070 con 0x7f798415a220 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.398+0000 7f7972ffd640 1 -- 172.21.15.77:0/4267760544 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c004220 con 0x7f798415a220 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.398+0000 7f7972ffd640 1 -- 172.21.15.77:0/4267760544 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c007620 con 0x7f798415a220 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.398+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79840ad7a0 con 0x7f798415a220 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.398+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79840aaa20 con 0x7f798415a220 2024-03-06T07:24:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.399+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79840732b0 con 0x7f798415a220 2024-03-06T07:24:57.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.399+0000 7f7972ffd640 1 -- 172.21.15.77:0/4267760544 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f797c0079d0 con 0x7f798415a220 2024-03-06T07:24:57.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.399+0000 7f7972ffd640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f796406b4d0 0x7f796406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.400+0000 7f798259c640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f796406b4d0 0x7f796406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.400+0000 7f7972ffd640 1 -- 172.21.15.77:0/4267760544 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f797c08a040 con 0x7f798415a220 2024-03-06T07:24:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.400+0000 7f798259c640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f796406b4d0 0x7f796406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7974007eb0 tx=0x7f79740073c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.404+0000 7f7972ffd640 1 -- 172.21.15.77:0/4267760544 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f797c059130 con 0x7f798415a220 2024-03-06T07:24:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:57.576+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f79840aabf0 con 0x7f798415a220 2024-03-06T07:24:58.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.273+0000 7f7972ffd640 1 -- 172.21.15.77:0/4267760544 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v67) v1 ==== 170+0+0 (secure 0 0 0) 0x7f797c058a30 con 0x7f798415a220 2024-03-06T07:24:58.275 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:24:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f796406b4d0 msgr2=0x7f796406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f796406b4d0 0x7f796406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7974007eb0 tx=0x7f79740073c0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f798415a220 msgr2=0x7f798415ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f798415a220 0x7f798415ca40 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f797c00c7f0 tx=0x7f797c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 shutdown_connections 2024-03-06T07:24:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f796406b4d0 0x7f796406d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f798415a220 0x7f798415ca40 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79840719e0 0x7f7984152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 --2- 172.21.15.77:0/4267760544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f798406c9f0 0x7f79840acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.276+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 >> 172.21.15.77:0/4267760544 conn(0x7f798406b190 msgr2=0x7f7984153150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.277+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 shutdown_connections 2024-03-06T07:24:58.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.278+0000 7f798899a640 1 -- 172.21.15.77:0/4267760544 wait complete. 2024-03-06T07:24:58.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:58.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.404+0000 7f7f4585c640 1 Processor -- start 2024-03-06T07:24:58.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.405+0000 7f7f4585c640 1 -- start start 2024-03-06T07:24:58.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.405+0000 7f7f4585c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7f400d2380 0x7f7f400d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.405+0000 7f7f4585c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 0x7f7f400cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.405+0000 7f7f4585c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f400b3fc0 con 0x7f7f4015e0b0 2024-03-06T07:24:58.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.405+0000 7f7f4585c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f400ce990 con 0x7f7f400d2380 2024-03-06T07:24:58.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.405+0000 7f7f4585c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f400d2d20 con 0x7f7f400cee00 2024-03-06T07:24:58.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.405+0000 7f7f3effd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7f4015e0b0 0x7f7f400b38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39940/0 (socket says 172.21.15.77:39940) 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.405+0000 7f7f3effd640 1 -- 172.21.15.77:0/652884514 learned_addr learned my addr 172.21.15.77:0/652884514 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3e7fc640 1 --2- 172.21.15.77:0/652884514 >> v2:172.21.15.158:3300/0 conn(0x7f7f400d2380 0x7f7f400d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3f7fe640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 0x7f7f400cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37947844 0 0) 0x7f7f400b3fc0 con 0x7f7f4015e0b0 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f28003520 con 0x7f7f4015e0b0 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765440277 0 0) 0x7f7f28003520 con 0x7f7f4015e0b0 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 msgr2=0x7f7f400cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 0x7f7f400cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 >> v2:172.21.15.158:3300/0 conn(0x7f7f400d2380 msgr2=0x7f7f400d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 --2- 172.21.15.77:0/652884514 >> v2:172.21.15.158:3300/0 conn(0x7f7f400d2380 0x7f7f400d27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f400b4290 con 0x7f7f4015e0b0 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f7f300019f0 con 0x7f7f4015e0b0 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 >> v1:172.21.15.77:6789/0 conn(0x7f7f4015e0b0 legacy=0x7f7f400b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f280039b0 0x7f7f28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f28004800 0x7f7f28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 0x7f7f280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2800b9c0 con 0x7f7f280039b0 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2800bb40 con 0x7f7f28004800 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2800bcc0 con 0x7f7f400cee00 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3effd640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f280039b0 0x7f7f28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3f7fe640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 0x7f7f280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3e7fc640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f28004800 0x7f7f28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3effd640 1 -- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 msgr2=0x7f7f280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3effd640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 0x7f7f280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3effd640 1 -- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f28004800 msgr2=0x7f7f28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3effd640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f28004800 0x7f7f28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.406+0000 7f7f3effd640 1 -- 172.21.15.77:0/652884514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f28003520 con 0x7f7f280039b0 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.407+0000 7f7f3f7fe640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f400cee00 0x7f7f280051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.407+0000 7f7f3effd640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f280039b0 0x7f7f28003dd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000a8a0 tx=0x7f7f28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7a2364d96cb533 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.407+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3000a4c0 con 0x7f7f280039b0 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.407+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f30034650 con 0x7f7f280039b0 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.407+0000 7f7f3dffb640 1 -- 172.21.15.77:0/652884514 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30034a40 con 0x7f7f280039b0 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.408+0000 7f7f4585c640 1 -- 172.21.15.77:0/652884514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f280039b0 msgr2=0x7f7f28003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.408+0000 7f7f4585c640 1 --2- 172.21.15.77:0/652884514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f280039b0 0x7f7f28003dd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000a8a0 tx=0x7f7f28005830 comp rx=0 tx=0).stop 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.408+0000 7f7f3f7fe640 1 -- 172.21.15.77:0/652884514 reap_dead start 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.408+0000 7f7f4585c640 1 -- 172.21.15.77:0/652884514 shutdown_connections 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.408+0000 7f7f4585c640 1 -- 172.21.15.77:0/652884514 >> 172.21.15.77:0/652884514 conn(0x7f7f4008d6f0 msgr2=0x7f7f4009d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.408+0000 7f7f4585c640 1 -- 172.21.15.77:0/652884514 shutdown_connections 2024-03-06T07:24:58.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.408+0000 7f7f4585c640 1 -- 172.21.15.77:0/652884514 wait complete. 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.409+0000 7f7f4585c640 1 Processor -- start 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.409+0000 7f7f4585c640 1 -- start start 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f4585c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f4014f0c0 0x7f7f400b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f4585c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f4015f460 0x7f7f400cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f4585c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f4014f4a0 0x7f7f40160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f4585c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f400b4b30 con 0x7f7f4014f4a0 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f4585c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f400b4850 con 0x7f7f4014f0c0 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f4585c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4015e490 con 0x7f7f4015f460 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f4014f4a0 0x7f7f40160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f4014f4a0 0x7f7f40160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39224/0 (socket says 172.21.15.77:39224) 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3f7fe640 1 -- 172.21.15.77:0/1721234688 learned_addr learned my addr 172.21.15.77:0/1721234688 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3e7fc640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f4015f460 0x7f7f400cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3effd640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f4014f0c0 0x7f7f400b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3f7fe640 1 -- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f4015f460 msgr2=0x7f7f400cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3f7fe640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f4015f460 0x7f7f400cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3f7fe640 1 -- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f4014f0c0 msgr2=0x7f7f400b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3f7fe640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f4014f0c0 0x7f7f400b95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3f7fe640 1 -- 172.21.15.77:0/1721234688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f400b9b20 con 0x7f7f4014f4a0 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3e7fc640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f4015f460 0x7f7f400cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.410+0000 7f7f3effd640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f4014f0c0 0x7f7f400b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.411+0000 7f7f3f7fe640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f4014f4a0 0x7f7f40160ca0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7f400b51a0 tx=0x7f7f38011860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.411+0000 7f7f27fff640 1 -- 172.21.15.77:0/1721234688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38004440 con 0x7f7f4014f4a0 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.411+0000 7f7f27fff640 1 -- 172.21.15.77:0/1721234688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3801b070 con 0x7f7f4014f4a0 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.411+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4015fb10 con 0x7f7f4014f4a0 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.411+0000 7f7f27fff640 1 -- 172.21.15.77:0/1721234688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38003850 con 0x7f7f4014f4a0 2024-03-06T07:24:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.411+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f401604a0 con 0x7f7f4014f4a0 2024-03-06T07:24:58.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.412+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4004f330 con 0x7f7f4014f4a0 2024-03-06T07:24:58.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.412+0000 7f7f27fff640 1 -- 172.21.15.77:0/1721234688 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f380039f0 con 0x7f7f4014f4a0 2024-03-06T07:24:58.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.412+0000 7f7f27fff640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.413+0000 7f7f3effd640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.413+0000 7f7f27fff640 1 -- 172.21.15.77:0/1721234688 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7f3808a1e0 con 0x7f7f4014f4a0 2024-03-06T07:24:58.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.413+0000 7f7f3effd640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7f30034000 tx=0x7f7f30009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.417+0000 7f7f27fff640 1 -- 172.21.15.77:0/1721234688 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f38025020 con 0x7f7f4014f4a0 2024-03-06T07:24:58.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:58.625+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7f400d2160 con 0x7f7f4014f4a0 2024-03-06T07:24:59.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.274+0000 7f7f27fff640 1 -- 172.21.15.77:0/1721234688 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v31) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7f380590c0 con 0x7f7f4014f4a0 2024-03-06T07:24:59.276 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f2c06b500 msgr2=0x7f7f2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7f30034000 tx=0x7f7f30009030 comp rx=0 tx=0).stop 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f4014f4a0 msgr2=0x7f7f40160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f4014f4a0 0x7f7f40160ca0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7f400b51a0 tx=0x7f7f38011860 comp rx=0 tx=0).stop 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 shutdown_connections 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f4014f4a0 0x7f7f40160ca0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f4015f460 0x7f7f400cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 --2- 172.21.15.77:0/1721234688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f4014f0c0 0x7f7f400b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 >> 172.21.15.77:0/1721234688 conn(0x7f7f4008d6f0 msgr2=0x7f7f400b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 shutdown_connections 2024-03-06T07:24:59.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.278+0000 7f7f4585c640 1 -- 172.21.15.77:0/1721234688 wait complete. 2024-03-06T07:24:59.303 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:59.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.403+0000 7fa8b0c8f640 1 Processor -- start 2024-03-06T07:24:59.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.403+0000 7fa8b0c8f640 1 -- start start 2024-03-06T07:24:59.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8b0c8f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8ac15dfe0 0x7fa8ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8b0c8f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 0x7fa8ac0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8b0c8f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ac0b3430 con 0x7fa8ac0d27f0 2024-03-06T07:24:59.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8b0c8f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ac0d2c50 con 0x7fa8ac15dfe0 2024-03-06T07:24:59.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8b0c8f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0b31e0 con 0x7fa8ac0cf830 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8aad9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa8ac0d27f0 0x7fa8ac0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59330/0 (socket says 172.21.15.77:59330) 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8aad9d640 1 -- 172.21.15.77:0/199052713 learned_addr learned my addr 172.21.15.77:0/199052713 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8aa59c640 1 --2- 172.21.15.77:0/199052713 >> v2:172.21.15.158:3300/0 conn(0x7fa8ac15dfe0 0x7fa8ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a3fff640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 0x7fa8ac0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152966032 0 0) 0x7fa8ac0b3430 con 0x7fa8ac0d27f0 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa898003520 con 0x7fa8ac0d27f0 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1638697819 0 0) 0x7fa898003520 con 0x7fa8ac0d27f0 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 msgr2=0x7fa8ac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 0x7fa8ac0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 >> v2:172.21.15.158:3300/0 conn(0x7fa8ac15dfe0 msgr2=0x7fa8ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 --2- 172.21.15.77:0/199052713 >> v2:172.21.15.158:3300/0 conn(0x7fa8ac15dfe0 0x7fa8ac0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac0b3710 con 0x7fa8ac0d27f0 2024-03-06T07:24:59.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.404+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fa8a40019f0 con 0x7fa8ac0d27f0 2024-03-06T07:24:59.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 >> v1:172.21.15.77:6789/0 conn(0x7fa8ac0d27f0 legacy=0x7fa8ac0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8a9d9b640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980039b0 0x7fa898003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8a9d9b640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa898004800 0x7fa898004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8a9d9b640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 0x7fa8980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0b31e0 con 0x7fa8980039b0 2024-03-06T07:24:59.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89800b9c0 con 0x7fa898004800 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89800bb40 con 0x7fa8ac0cf830 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8aa59c640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980039b0 0x7fa898003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8a3fff640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa898004800 0x7fa898004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8aad9d640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 0x7fa8980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8aa59c640 1 -- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 msgr2=0x7fa8980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8aa59c640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 0x7fa8980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8aa59c640 1 -- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa898004800 msgr2=0x7fa898004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8aa59c640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa898004800 0x7fa898004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8aa59c640 1 -- 172.21.15.77:0/199052713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898003520 con 0x7fa8980039b0 2024-03-06T07:24:59.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.405+0000 7fa8aad9d640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cf830 0x7fa8980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8aa59c640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980039b0 0x7fa898003dd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009ac0 tx=0x7fa898005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3867a6b0d8437f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c039070 con 0x7fa8980039b0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c02c710 con 0x7fa8980039b0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8a9d9b640 1 -- 172.21.15.77:0/199052713 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c009690 con 0x7fa8980039b0 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/199052713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980039b0 msgr2=0x7fa898003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8b0c8f640 1 --2- 172.21.15.77:0/199052713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980039b0 0x7fa898003dd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009ac0 tx=0x7fa898005830 comp rx=0 tx=0).stop 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8aad9d640 1 -- 172.21.15.77:0/199052713 reap_dead start 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/199052713 shutdown_connections 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/199052713 >> 172.21.15.77:0/199052713 conn(0x7fa8ac0d3cd0 msgr2=0x7fa8ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/199052713 shutdown_connections 2024-03-06T07:24:59.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.406+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/199052713 wait complete. 2024-03-06T07:24:59.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8b0c8f640 1 Processor -- start 2024-03-06T07:24:59.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8b0c8f640 1 -- start start 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8b0c8f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8980039b0 0x7fa8ac0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8b0c8f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cb780 0x7fa8ac15c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8b0c8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8ac1f3870 0x7fa8ac1f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8b0c8f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0d1c70 con 0x7fa8ac1f3870 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8b0c8f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0b3d60 con 0x7fa8980039b0 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8b0c8f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0d1df0 con 0x7fa8ac0cb780 2024-03-06T07:24:59.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8aad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8ac1f3870 0x7fa8ac1f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.408+0000 7fa8aad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8ac1f3870 0x7fa8ac1f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44450/0 (socket says 172.21.15.77:44450) 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8aad9d640 1 -- 172.21.15.77:0/1255322647 learned_addr learned my addr 172.21.15.77:0/1255322647 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8aa59c640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8980039b0 0x7fa8ac0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8a3fff640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cb780 0x7fa8ac15c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8aad9d640 1 -- 172.21.15.77:0/1255322647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cb780 msgr2=0x7fa8ac15c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8aad9d640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cb780 0x7fa8ac15c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8aad9d640 1 -- 172.21.15.77:0/1255322647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8980039b0 msgr2=0x7fa8ac0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8aad9d640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8980039b0 0x7fa8ac0ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8aad9d640 1 -- 172.21.15.77:0/1255322647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac0cc830 con 0x7fa8ac1f3870 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8aad9d640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8ac1f3870 0x7fa8ac1f3cf0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0cff40 tx=0x7fa8a400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8a2ffd640 1 -- 172.21.15.77:0/1255322647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a400eb60 con 0x7fa8ac1f3870 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8a2ffd640 1 -- 172.21.15.77:0/1255322647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a400ed00 con 0x7fa8ac1f3870 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ac0cbf90 con 0x7fa8ac1f3870 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8a2ffd640 1 -- 172.21.15.77:0/1255322647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4014630 con 0x7fa8ac1f3870 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.409+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ac0cc550 con 0x7fa8ac1f3870 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.410+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ac04f330 con 0x7fa8ac1f3870 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.411+0000 7fa8a2ffd640 1 -- 172.21.15.77:0/1255322647 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8a40147d0 con 0x7fa8ac1f3870 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.411+0000 7fa8a2ffd640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa88c06b4d0 0x7fa88c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.411+0000 7fa8a2ffd640 1 -- 172.21.15.77:0/1255322647 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8a4089f10 con 0x7fa8ac1f3870 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.411+0000 7fa8aa59c640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa88c06b4d0 0x7fa88c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.411+0000 7fa8aa59c640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa88c06b4d0 0x7fa88c06d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa89c036000 tx=0x7fa89c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.415+0000 7fa8a2ffd640 1 -- 172.21.15.77:0/1255322647 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8a40586b0 con 0x7fa8ac1f3870 2024-03-06T07:24:59.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.588+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8ac0b4500 con 0x7fa8ac1f3870 2024-03-06T07:24:59.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.589+0000 7fa8a2ffd640 1 -- 172.21.15.77:0/1255322647 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16669 (secure 0 0 0) 0x7fa8a405c750 con 0x7fa8ac1f3870 2024-03-06T07:24:59.591 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:59.591 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":68,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:58.626734+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:56.868266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:57.595004+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa88c06b4d0 msgr2=0x7fa88c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa88c06b4d0 0x7fa88c06d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa89c036000 tx=0x7fa89c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8ac1f3870 msgr2=0x7fa8ac1f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8ac1f3870 0x7fa8ac1f3cf0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0cff40 tx=0x7fa8a400e530 comp rx=0 tx=0).stop 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 shutdown_connections 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa88c06b4d0 0x7fa88c06d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8ac1f3870 0x7fa8ac1f3cf0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8ac0cb780 0x7fa8ac15c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 --2- 172.21.15.77:0/1255322647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8980039b0 0x7fa8ac0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 >> 172.21.15.77:0/1255322647 conn(0x7fa8ac0d3cd0 msgr2=0x7fa8ac0b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 shutdown_connections 2024-03-06T07:24:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.591+0000 7fa8b0c8f640 1 -- 172.21.15.77:0/1255322647 wait complete. 2024-03-06T07:24:59.605 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:59.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.702+0000 7f8dcf593640 1 Processor -- start 2024-03-06T07:24:59.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.702+0000 7f8dcf593640 1 -- start start 2024-03-06T07:24:59.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcf593640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8dc80d00c0 0x7f8dc80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcf593640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc806cf20 0x7f8dc806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcf593640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc80d09e0 con 0x7f8dc80cf850 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcf593640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc806ca90 con 0x7f8dc80d00c0 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcf593640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc815e380 con 0x7f8dc806cf20 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcd32f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8dc80d00c0 0x7f8dc80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcd32f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8dc80d00c0 0x7f8dc80d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44452/0 (socket says 172.21.15.77:44452) 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcd32f640 1 -- 172.21.15.77:0/1106113500 learned_addr learned my addr 172.21.15.77:0/1106113500 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcdb30640 1 --2- 172.21.15.77:0/1106113500 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc806cf20 0x7f8dc806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcd32f640 1 -- 172.21.15.77:0/1106113500 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc806cf20 msgr2=0x7f8dc806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcd32f640 1 --2- 172.21.15.77:0/1106113500 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc806cf20 0x7f8dc806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcd32f640 1 -- 172.21.15.77:0/1106113500 >> v1:172.21.15.158:6789/0 conn(0x7f8dc80cf850 legacy=0x7f8dc80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.703+0000 7f8dcd32f640 1 -- 172.21.15.77:0/1106113500 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80cdc10 con 0x7f8dc80d00c0 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.704+0000 7f8dcd32f640 1 --2- 172.21.15.77:0/1106113500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc80d00c0 0x7f8dc80d04a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009e50 tx=0x7f8dc402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229e96d02904aeef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.704+0000 7f8dbffff640 1 -- 172.21.15.77:0/1106113500 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc402cb00 con 0x7f8dc80d00c0 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.704+0000 7f8dbffff640 1 -- 172.21.15.77:0/1106113500 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc402cca0 con 0x7f8dc80d00c0 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.704+0000 7f8dbffff640 1 -- 172.21.15.77:0/1106113500 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4034610 con 0x7f8dc80d00c0 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.705+0000 7f8dcf593640 1 -- 172.21.15.77:0/1106113500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc80d00c0 msgr2=0x7f8dc80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.705+0000 7f8dcf593640 1 --2- 172.21.15.77:0/1106113500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc80d00c0 0x7f8dc80d04a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009e50 tx=0x7f8dc402bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.705+0000 7f8dcf593640 1 -- 172.21.15.77:0/1106113500 shutdown_connections 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.705+0000 7f8dcf593640 1 --2- 172.21.15.77:0/1106113500 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc806cf20 0x7f8dc806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.705+0000 7f8dcf593640 1 --2- 172.21.15.77:0/1106113500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc80d00c0 0x7f8dc80d04a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.705+0000 7f8dcf593640 1 -- 172.21.15.77:0/1106113500 >> 172.21.15.77:0/1106113500 conn(0x7f8dc80b35a0 msgr2=0x7f8dc80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.705+0000 7f8dcf593640 1 -- 172.21.15.77:0/1106113500 shutdown_connections 2024-03-06T07:24:59.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.705+0000 7f8dcf593640 1 -- 172.21.15.77:0/1106113500 wait complete. 2024-03-06T07:24:59.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.706+0000 7f8dcf593640 1 Processor -- start 2024-03-06T07:24:59.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.706+0000 7f8dcf593640 1 -- start start 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcf593640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc806cf20 0x7f8dc80c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcf593640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc80cf850 0x7f8dc8151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcf593640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc80d00c0 0x7f8dc80c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcf593640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc806c720 con 0x7f8dc806cf20 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcf593640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc806c440 con 0x7f8dc80d00c0 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcf593640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80b1030 con 0x7f8dc80cf850 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc806cf20 0x7f8dc80c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc806cf20 0x7f8dc80c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44468/0 (socket says 172.21.15.77:44468) 2024-03-06T07:24:59.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 -- 172.21.15.77:0/712176586 learned_addr learned my addr 172.21.15.77:0/712176586 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcdb30640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc80d00c0 0x7f8dc80c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dccb2e640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc80cf850 0x7f8dc8151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 -- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc80cf850 msgr2=0x7f8dc8151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc80cf850 0x7f8dc8151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 -- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc80d00c0 msgr2=0x7f8dc80c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc80d00c0 0x7f8dc80c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 -- 172.21.15.77:0/712176586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80c9390 con 0x7f8dc806cf20 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcdb30640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc80d00c0 0x7f8dc80c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcd32f640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc806cf20 0x7f8dc80c8b40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4036040 tx=0x7f8dc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dbdffb640 1 -- 172.21.15.77:0/712176586 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4043020 con 0x7f8dc806cf20 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dbdffb640 1 -- 172.21.15.77:0/712176586 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc402ce50 con 0x7f8dc806cf20 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.707+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc80c9670 con 0x7f8dc806cf20 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.708+0000 7f8dbdffb640 1 -- 172.21.15.77:0/712176586 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4009510 con 0x7f8dc806cf20 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.708+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc80c9c30 con 0x7f8dc806cf20 2024-03-06T07:24:59.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.709+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc804f330 con 0x7f8dc806cf20 2024-03-06T07:24:59.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.709+0000 7f8dbdffb640 1 -- 172.21.15.77:0/712176586 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dc4046020 con 0x7f8dc806cf20 2024-03-06T07:24:59.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.709+0000 7f8dbdffb640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8dac06b4d0 0x7f8dac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.710+0000 7f8dbdffb640 1 -- 172.21.15.77:0/712176586 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dc40ac7c0 con 0x7f8dc806cf20 2024-03-06T07:24:59.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.710+0000 7f8dccb2e640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8dac06b4d0 0x7f8dac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.710+0000 7f8dccb2e640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8dac06b4d0 0x7f8dac06d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8070890 tx=0x7f8db8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.714+0000 7f8dbdffb640 1 -- 172.21.15.77:0/712176586 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8dc404d030 con 0x7f8dc806cf20 2024-03-06T07:24:59.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.887+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dc80b3d00 con 0x7f8dc806cf20 2024-03-06T07:24:59.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.888+0000 7f8dbdffb640 1 -- 172.21.15.77:0/712176586 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f8dc407b6d0 con 0x7f8dc806cf20 2024-03-06T07:24:59.890 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:24:59.890 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":68,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:24:58.626734+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:56.868266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:57.595004+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8dac06b4d0 msgr2=0x7f8dac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8dac06b4d0 0x7f8dac06d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8070890 tx=0x7f8db8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc806cf20 msgr2=0x7f8dc80c8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc806cf20 0x7f8dc80c8b40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4036040 tx=0x7f8dc4004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 shutdown_connections 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc80d00c0 0x7f8dc80c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc80cf850 0x7f8dc8151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8dac06b4d0 0x7f8dac06d990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 --2- 172.21.15.77:0/712176586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc806cf20 0x7f8dc80c8b40 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 >> 172.21.15.77:0/712176586 conn(0x7f8dc80b35a0 msgr2=0x7f8dc80cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 shutdown_connections 2024-03-06T07:24:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:24:59.890+0000 7f8dcf593640 1 -- 172.21.15.77:0/712176586 wait complete. 2024-03-06T07:24:59.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:59.903 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:59.904 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:00.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.046+0000 7fa762ceb640 1 Processor -- start 2024-03-06T07:25:00.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.046+0000 7fa762ceb640 1 -- start start 2024-03-06T07:25:00.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.047+0000 7fa762ceb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa75c071a70 0x7fa75c071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.047+0000 7fa762ceb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa75c0b42e0 0x7fa75c15c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.047+0000 7fa762ceb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c072390 con 0x7fa75c15ad90 2024-03-06T07:25:00.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.047+0000 7fa762ceb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75c15c020 con 0x7fa75c0b42e0 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.047+0000 7fa762ceb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75c0b4720 con 0x7fa75c071a70 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.047+0000 7fa761288640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa75c0b42e0 0x7fa75c15c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.047+0000 7fa761288640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa75c0b42e0 0x7fa75c15c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44472/0 (socket says 172.21.15.77:44472) 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.047+0000 7fa761288640 1 -- 172.21.15.77:0/3161783308 learned_addr learned my addr 172.21.15.77:0/3161783308 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.048+0000 7fa760a87640 1 --2- 172.21.15.77:0/3161783308 >> v2:172.21.15.158:3300/0 conn(0x7fa75c071a70 0x7fa75c071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.048+0000 7fa761288640 1 -- 172.21.15.77:0/3161783308 >> v2:172.21.15.158:3300/0 conn(0x7fa75c071a70 msgr2=0x7fa75c071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.048+0000 7fa761288640 1 --2- 172.21.15.77:0/3161783308 >> v2:172.21.15.158:3300/0 conn(0x7fa75c071a70 0x7fa75c071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.048+0000 7fa761288640 1 -- 172.21.15.77:0/3161783308 >> v1:172.21.15.158:6789/0 conn(0x7fa75c15ad90 legacy=0x7fa75c0b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.048+0000 7fa761288640 1 -- 172.21.15.77:0/3161783308 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c15b260 con 0x7fa75c0b42e0 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.048+0000 7fa761288640 1 --2- 172.21.15.77:0/3161783308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c15c210 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa75c06d050 tx=0x7fa74c02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34502b7e25a5a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.048+0000 7fa75b7fe640 1 -- 172.21.15.77:0/3161783308 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c03b070 con 0x7fa75c0b42e0 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa75b7fe640 1 -- 172.21.15.77:0/3161783308 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c02ee00 con 0x7fa75c0b42e0 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa75b7fe640 1 -- 172.21.15.77:0/3161783308 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c036660 con 0x7fa75c0b42e0 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa762ceb640 1 -- 172.21.15.77:0/3161783308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 msgr2=0x7fa75c15c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa762ceb640 1 --2- 172.21.15.77:0/3161783308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c15c210 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa75c06d050 tx=0x7fa74c02e050 comp rx=0 tx=0).stop 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa762ceb640 1 -- 172.21.15.77:0/3161783308 shutdown_connections 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa762ceb640 1 --2- 172.21.15.77:0/3161783308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c15c210 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa762ceb640 1 --2- 172.21.15.77:0/3161783308 >> v2:172.21.15.158:3300/0 conn(0x7fa75c071a70 0x7fa75c071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa762ceb640 1 -- 172.21.15.77:0/3161783308 >> 172.21.15.77:0/3161783308 conn(0x7fa75c0b2660 msgr2=0x7fa75c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa762ceb640 1 -- 172.21.15.77:0/3161783308 shutdown_connections 2024-03-06T07:25:00.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.049+0000 7fa762ceb640 1 -- 172.21.15.77:0/3161783308 wait complete. 2024-03-06T07:25:00.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.050+0000 7fa762ceb640 1 Processor -- start 2024-03-06T07:25:00.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.050+0000 7fa762ceb640 1 -- start start 2024-03-06T07:25:00.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa762ceb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa75c071a70 0x7fa75c0cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa762ceb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c1f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa762ceb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa75c15ad90 0x7fa75c0aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa762ceb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c15bb00 con 0x7fa75c0b42e0 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa762ceb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c15b820 con 0x7fa75c15ad90 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa762ceb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c15c750 con 0x7fa75c071a70 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa75bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c1f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa75bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c1f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44476/0 (socket says 172.21.15.77:44476) 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.051+0000 7fa75bfff640 1 -- 172.21.15.77:0/941951068 learned_addr learned my addr 172.21.15.77:0/941951068 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa760a87640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa75c071a70 0x7fa75c0cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa761288640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa75c15ad90 0x7fa75c0aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa75bfff640 1 -- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa75c071a70 msgr2=0x7fa75c0cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa75bfff640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa75c071a70 0x7fa75c0cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa75bfff640 1 -- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa75c15ad90 msgr2=0x7fa75c0aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa75bfff640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa75c15ad90 0x7fa75c0aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa75bfff640 1 -- 172.21.15.77:0/941951068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c0ce210 con 0x7fa75c0b42e0 2024-03-06T07:25:00.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa760a87640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa75c071a70 0x7fa75c0cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa75bfff640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c1f3b30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa75000b3e0 tx=0x7fa75000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa7597fa640 1 -- 172.21.15.77:0/941951068 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750015020 con 0x7fa75c0b42e0 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa7597fa640 1 -- 172.21.15.77:0/941951068 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa750011830 con 0x7fa75c0b42e0 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c0ce4b0 con 0x7fa75c0b42e0 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa7597fa640 1 -- 172.21.15.77:0/941951068 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750011c20 con 0x7fa75c0b42e0 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.052+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c0cea70 con 0x7fa75c0b42e0 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.053+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c073390 con 0x7fa75c0b42e0 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.054+0000 7fa7597fa640 1 -- 172.21.15.77:0/941951068 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa750018020 con 0x7fa75c0b42e0 2024-03-06T07:25:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.054+0000 7fa7597fa640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa74806b4d0 0x7fa74806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.054+0000 7fa7597fa640 1 -- 172.21.15.77:0/941951068 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa75008a130 con 0x7fa75c0b42e0 2024-03-06T07:25:00.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.054+0000 7fa760a87640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa74806b4d0 0x7fa74806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.054+0000 7fa760a87640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa74806b4d0 0x7fa74806d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa75c15df30 tx=0x7fa7540074c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.058+0000 7fa7597fa640 1 -- 172.21.15.77:0/941951068 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa75001f020 con 0x7fa75c0b42e0 2024-03-06T07:25:00.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.268+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa75c068ff0 con 0x7fa75c0b42e0 2024-03-06T07:25:00.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.287+0000 7fa7597fa640 1 -- 172.21.15.77:0/941951068 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa7500172c0 con 0x7fa75c0b42e0 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.289+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa74806b4d0 msgr2=0x7fa74806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.289+0000 7fa762ceb640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa74806b4d0 0x7fa74806d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa75c15df30 tx=0x7fa7540074c0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.289+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 msgr2=0x7fa75c1f3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.289+0000 7fa762ceb640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c1f3b30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa75000b3e0 tx=0x7fa75000b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.290+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 shutdown_connections 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.290+0000 7fa762ceb640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa74806b4d0 0x7fa74806d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.290+0000 7fa762ceb640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c0b42e0 0x7fa75c1f3b30 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.290+0000 7fa762ceb640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa75c15ad90 0x7fa75c0aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.290+0000 7fa762ceb640 1 --2- 172.21.15.77:0/941951068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa75c071a70 0x7fa75c0cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.290+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 >> 172.21.15.77:0/941951068 conn(0x7fa75c0b2660 msgr2=0x7fa75c06da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.290+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 shutdown_connections 2024-03-06T07:25:00.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.290+0000 7fa762ceb640 1 -- 172.21.15.77:0/941951068 wait complete. 2024-03-06T07:25:00.306 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:00.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:00.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.450+0000 7fd8ea881640 1 Processor -- start 2024-03-06T07:25:00.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.450+0000 7fd8ea881640 1 -- start start 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.451+0000 7fd8ea881640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd8e40d10c0 0x7fd8e40d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.451+0000 7fd8ea881640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd8e415ecd0 0x7fd8e40cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.451+0000 7fd8ea881640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e40cfa50 con 0x7fd8e40d14a0 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.451+0000 7fd8ea881640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e415c050 con 0x7fd8e40d10c0 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.451+0000 7fd8ea881640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e40d0060 con 0x7fd8e415ecd0 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.451+0000 7fd8e3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd8e40d10c0 0x7fd8e40d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.451+0000 7fd8e37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd8e40d14a0 0x7fd8e415e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59342/0 (socket says 172.21.15.77:59342) 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.451+0000 7fd8e37fe640 1 -- 172.21.15.77:0/1008504790 learned_addr learned my addr 172.21.15.77:0/1008504790 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.452+0000 7fd8e8e1e640 1 --2- 172.21.15.77:0/1008504790 >> v2:172.21.15.158:3300/0 conn(0x7fd8e415ecd0 0x7fd8e40cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.452+0000 7fd8e3fff640 1 -- 172.21.15.77:0/1008504790 >> v2:172.21.15.158:3300/0 conn(0x7fd8e415ecd0 msgr2=0x7fd8e40cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.452+0000 7fd8e3fff640 1 --2- 172.21.15.77:0/1008504790 >> v2:172.21.15.158:3300/0 conn(0x7fd8e415ecd0 0x7fd8e40cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.452+0000 7fd8e3fff640 1 -- 172.21.15.77:0/1008504790 >> v1:172.21.15.77:6789/0 conn(0x7fd8e40d14a0 legacy=0x7fd8e415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.452+0000 7fd8e2ffd640 1 -- 172.21.15.77:0/1008504790 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033091382 0 0) 0x7fd8e40cfa50 con 0x7fd8e40d14a0 2024-03-06T07:25:00.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.452+0000 7fd8e3fff640 1 -- 172.21.15.77:0/1008504790 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e40d0a30 con 0x7fd8e40d10c0 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8e3fff640 1 --2- 172.21.15.77:0/1008504790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e40d04f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40b1720 tx=0x7fd8d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76eb51e6cb263c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8e2ffd640 1 -- 172.21.15.77:0/1008504790 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4035020 con 0x7fd8e40d10c0 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8e2ffd640 1 -- 172.21.15.77:0/1008504790 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d4031830 con 0x7fd8e40d10c0 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8e2ffd640 1 -- 172.21.15.77:0/1008504790 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4031c00 con 0x7fd8e40d10c0 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8ea881640 1 -- 172.21.15.77:0/1008504790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 msgr2=0x7fd8e40d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8ea881640 1 --2- 172.21.15.77:0/1008504790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e40d04f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40b1720 tx=0x7fd8d402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8ea881640 1 -- 172.21.15.77:0/1008504790 shutdown_connections 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8ea881640 1 --2- 172.21.15.77:0/1008504790 >> v2:172.21.15.158:3300/0 conn(0x7fd8e415ecd0 0x7fd8e40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8ea881640 1 --2- 172.21.15.77:0/1008504790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e40d04f0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8ea881640 1 -- 172.21.15.77:0/1008504790 >> 172.21.15.77:0/1008504790 conn(0x7fd8e40b3800 msgr2=0x7fd8e40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8ea881640 1 -- 172.21.15.77:0/1008504790 shutdown_connections 2024-03-06T07:25:00.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.453+0000 7fd8ea881640 1 -- 172.21.15.77:0/1008504790 wait complete. 2024-03-06T07:25:00.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.454+0000 7fd8ea881640 1 Processor -- start 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8ea881640 1 -- start start 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8ea881640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e41f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8ea881640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8e40d14a0 0x7fd8e40b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8ea881640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8e40b48f0 0x7fd8e41f7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8ea881640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e415f440 con 0x7fd8e40d10c0 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8ea881640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e415f160 con 0x7fd8e40b48f0 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8ea881640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40cfcf0 con 0x7fd8e40d14a0 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8e3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e41f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8e3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e41f4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44502/0 (socket says 172.21.15.77:44502) 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.455+0000 7fd8e3fff640 1 -- 172.21.15.77:0/584059043 learned_addr learned my addr 172.21.15.77:0/584059043 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e37fe640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8e40d14a0 0x7fd8e40b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e3fff640 1 -- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8e40d14a0 msgr2=0x7fd8e40b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e8e1e640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8e40b48f0 0x7fd8e41f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e3fff640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8e40d14a0 0x7fd8e40b1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e3fff640 1 -- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8e40b48f0 msgr2=0x7fd8e41f7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e3fff640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8e40b48f0 0x7fd8e41f7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e3fff640 1 -- 172.21.15.77:0/584059043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e41f4b60 con 0x7fd8e40d10c0 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e37fe640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8e40d14a0 0x7fd8e40b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e3fff640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e41f4620 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4000c00 tx=0x7fd8d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e0ff9640 1 -- 172.21.15.77:0/584059043 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4040070 con 0x7fd8e40d10c0 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e0ff9640 1 -- 172.21.15.77:0/584059043 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d4039a20 con 0x7fd8e40d10c0 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8e0ff9640 1 -- 172.21.15.77:0/584059043 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4035040 con 0x7fd8e40d10c0 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e414d3d0 con 0x7fd8e40d10c0 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.456+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e414d990 con 0x7fd8e40d10c0 2024-03-06T07:25:00.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.457+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e409fe60 con 0x7fd8e40d10c0 2024-03-06T07:25:00.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.458+0000 7fd8e0ff9640 1 -- 172.21.15.77:0/584059043 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8d404b060 con 0x7fd8e40d10c0 2024-03-06T07:25:00.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.458+0000 7fd8e0ff9640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd8d006b4d0 0x7fd8d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.458+0000 7fd8e0ff9640 1 -- 172.21.15.77:0/584059043 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8d40aba70 con 0x7fd8e40d10c0 2024-03-06T07:25:00.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.458+0000 7fd8e37fe640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd8d006b4d0 0x7fd8d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.459+0000 7fd8e37fe640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd8d006b4d0 0x7fd8d006d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd8e415bdf0 tx=0x7fd8d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.462+0000 7fd8e0ff9640 1 -- 172.21.15.77:0/584059043 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8d407a980 con 0x7fd8e40d10c0 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:00.679+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd8e4068ff0 con 0x7fd8e40d10c0 2024-03-06T07:25:01.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.288+0000 7fd8e0ff9640 1 -- 172.21.15.77:0/584059043 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd8d407a280 con 0x7fd8e40d10c0 2024-03-06T07:25:01.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd8d006b4d0 msgr2=0x7fd8d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd8d006b4d0 0x7fd8d006d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd8e415bdf0 tx=0x7fd8d8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:01.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 msgr2=0x7fd8e41f4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e41f4620 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4000c00 tx=0x7fd8d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 shutdown_connections 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8e40b48f0 0x7fd8e41f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8e40d14a0 0x7fd8e40b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd8d006b4d0 0x7fd8d006d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 --2- 172.21.15.77:0/584059043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e40d10c0 0x7fd8e41f4620 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 >> 172.21.15.77:0/584059043 conn(0x7fd8e40b3800 msgr2=0x7fd8e40b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 shutdown_connections 2024-03-06T07:25:01.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.290+0000 7fd8ea881640 1 -- 172.21.15.77:0/584059043 wait complete. 2024-03-06T07:25:01.303 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:01.303 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:01.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.402+0000 7f58c1a15640 1 Processor -- start 2024-03-06T07:25:01.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.402+0000 7f58c1a15640 1 -- start start 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.403+0000 7f58c1a15640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f58bc0b4170 0x7f58bc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.403+0000 7f58c1a15640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0d4760 0x7f58bc0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.403+0000 7f58c1a15640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0d25d0 con 0x7f58bc0b4dd0 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.403+0000 7f58c1a15640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc0d4bf0 con 0x7f58bc0b4170 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.403+0000 7f58c1a15640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0ce9e0 con 0x7f58bc0d4760 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.403+0000 7f58b27fc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0d4760 0x7f58bc0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.403+0000 7f58baffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f58bc0b4170 0x7f58bc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.404+0000 7f58b27fc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0d4760 0x7f58bc0d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:45878/0 (socket says 172.21.15.77:45878) 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.404+0000 7f58b27fc640 1 -- 172.21.15.77:0/2191365935 learned_addr learned my addr 172.21.15.77:0/2191365935 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.404+0000 7f58bb7fe640 1 --1- 172.21.15.77:0/2191365935 >> v1:172.21.15.158:6789/0 conn(0x7f58bc0b4dd0 0x7f58bc0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:32976/0 (socket says 172.21.15.77:32976) 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.404+0000 7f58b27fc640 1 -- 172.21.15.77:0/2191365935 >> v2:172.21.15.158:3300/0 conn(0x7f58bc0b4170 msgr2=0x7f58bc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.404+0000 7f58b27fc640 1 --2- 172.21.15.77:0/2191365935 >> v2:172.21.15.158:3300/0 conn(0x7f58bc0b4170 0x7f58bc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.404+0000 7f58b27fc640 1 -- 172.21.15.77:0/2191365935 >> v1:172.21.15.158:6789/0 conn(0x7f58bc0b4dd0 legacy=0x7f58bc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.404+0000 7f58b27fc640 1 -- 172.21.15.77:0/2191365935 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc0d2870 con 0x7f58bc0d4760 2024-03-06T07:25:01.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58b27fc640 1 --2- 172.21.15.77:0/2191365935 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0d4760 0x7f58bc0d0370 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009870 tx=0x7f58a402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c77743bdb026ea5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58ba7fc640 1 -- 172.21.15.77:0/2191365935 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4034070 con 0x7f58bc0d4760 2024-03-06T07:25:01.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58ba7fc640 1 -- 172.21.15.77:0/2191365935 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4030840 con 0x7f58bc0d4760 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58ba7fc640 1 -- 172.21.15.77:0/2191365935 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4030c10 con 0x7f58bc0d4760 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58c1a15640 1 -- 172.21.15.77:0/2191365935 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0d4760 msgr2=0x7f58bc0d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58c1a15640 1 --2- 172.21.15.77:0/2191365935 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0d4760 0x7f58bc0d0370 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009870 tx=0x7f58a402b980 comp rx=0 tx=0).stop 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58c1a15640 1 -- 172.21.15.77:0/2191365935 shutdown_connections 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58c1a15640 1 --2- 172.21.15.77:0/2191365935 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0d4760 0x7f58bc0d0370 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58c1a15640 1 --2- 172.21.15.77:0/2191365935 >> v2:172.21.15.158:3300/0 conn(0x7f58bc0b4170 0x7f58bc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58c1a15640 1 -- 172.21.15.77:0/2191365935 >> 172.21.15.77:0/2191365935 conn(0x7f58bc0d3c40 msgr2=0x7f58bc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58c1a15640 1 -- 172.21.15.77:0/2191365935 shutdown_connections 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.405+0000 7f58c1a15640 1 -- 172.21.15.77:0/2191365935 wait complete. 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.406+0000 7f58c1a15640 1 Processor -- start 2024-03-06T07:25:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.407+0000 7f58c1a15640 1 -- start start 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.407+0000 7f58c1a15640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0b4170 0x7f58bc1f7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.407+0000 7f58c1a15640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b4dd0 0x7f58bc1f0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.407+0000 7f58c1a15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0c91c0 0x7f58bc06c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.407+0000 7f58c1a15640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0b47d0 con 0x7f58bc0c91c0 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.407+0000 7f58c1a15640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0b4550 con 0x7f58bc0b4dd0 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.407+0000 7f58c1a15640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0b4950 con 0x7f58bc0b4170 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0c91c0 0x7f58bc06c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0c91c0 0x7f58bc06c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44512/0 (socket says 172.21.15.77:44512) 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58baffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0b4170 0x7f58bc1f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 -- 172.21.15.77:0/158179332 learned_addr learned my addr 172.21.15.77:0/158179332 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58b27fc640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b4dd0 0x7f58bc1f0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 -- 172.21.15.77:0/158179332 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0b4170 msgr2=0x7f58bc1f7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0b4170 0x7f58bc1f7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 -- 172.21.15.77:0/158179332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b4dd0 msgr2=0x7f58bc1f0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b4dd0 0x7f58bc1f0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 -- 172.21.15.77:0/158179332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc1f3b30 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58bb7fe640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0c91c0 0x7f58bc06c820 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f58b400c7b0 tx=0x7f58b400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58b8ff9640 1 -- 172.21.15.77:0/158179332 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4017070 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58b8ff9640 1 -- 172.21.15.77:0/158179332 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b400a640 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58b8ff9640 1 -- 172.21.15.77:0/158179332 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b400a9f0 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc1f3e10 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.408+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc1f43d0 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.409+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc04f330 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.410+0000 7f58b8ff9640 1 -- 172.21.15.77:0/158179332 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58b401a070 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.410+0000 7f58b8ff9640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.410+0000 7f58b8ff9640 1 -- 172.21.15.77:0/158179332 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58b4089760 con 0x7f58bc0c91c0 2024-03-06T07:25:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.410+0000 7f58baffd640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.410+0000 7f58baffd640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f589c06b4d0 0x7f589c06d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f58bc06d7f0 tx=0x7f58ac009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.417+0000 7f58b8ff9640 1 -- 172.21.15.77:0/158179332 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58b4057f90 con 0x7f58bc0c91c0 2024-03-06T07:25:01.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.632+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58bc0733b0 con 0x7f58bc0c91c0 2024-03-06T07:25:01.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.633+0000 7f58b8ff9640 1 -- 172.21.15.77:0/158179332 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f58b405c030 con 0x7f58bc0c91c0 2024-03-06T07:25:01.634 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:01.634 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:01.635 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:01.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f589c06b4d0 msgr2=0x7f589c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f589c06b4d0 0x7f589c06d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f58bc06d7f0 tx=0x7f58ac009190 comp rx=0 tx=0).stop 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0c91c0 msgr2=0x7f58bc06c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0c91c0 0x7f58bc06c820 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f58b400c7b0 tx=0x7f58b400cc70 comp rx=0 tx=0).stop 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 shutdown_connections 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0c91c0 0x7f58bc06c820 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b4dd0 0x7f58bc1f0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 --2- 172.21.15.77:0/158179332 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0b4170 0x7f58bc1f7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.634+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 >> 172.21.15.77:0/158179332 conn(0x7f58bc0d3c40 msgr2=0x7f58bc0b5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.635+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 shutdown_connections 2024-03-06T07:25:01.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.635+0000 7f58c1a15640 1 -- 172.21.15.77:0/158179332 wait complete. 2024-03-06T07:25:01.648 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:01.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.746+0000 7fc231a9a640 1 Processor -- start 2024-03-06T07:25:01.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.746+0000 7fc231a9a640 1 -- start start 2024-03-06T07:25:01.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.747+0000 7fc231a9a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc22c15bdd0 0x7fc22c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.747+0000 7fc231a9a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc22c0d16e0 0x7fc22c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.747+0000 7fc231a9a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c0cebe0 con 0x7fc22c15fa10 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.747+0000 7fc231a9a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc22c0ce980 con 0x7fc22c0d16e0 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.747+0000 7fc231a9a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc22c0ceec0 con 0x7fc22c15bdd0 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.747+0000 7fc22bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc22c0d16e0 0x7fc22c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.747+0000 7fc22bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc22c0d16e0 0x7fc22c0d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44522/0 (socket says 172.21.15.77:44522) 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.747+0000 7fc22bfff640 1 -- 172.21.15.77:0/1642735340 learned_addr learned my addr 172.21.15.77:0/1642735340 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.748+0000 7fc22b7fe640 1 --2- 172.21.15.77:0/1642735340 >> v2:172.21.15.158:3300/0 conn(0x7fc22c15bdd0 0x7fc22c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.748+0000 7fc22bfff640 1 -- 172.21.15.77:0/1642735340 >> v2:172.21.15.158:3300/0 conn(0x7fc22c15bdd0 msgr2=0x7fc22c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.748+0000 7fc22bfff640 1 --2- 172.21.15.77:0/1642735340 >> v2:172.21.15.158:3300/0 conn(0x7fc22c15bdd0 0x7fc22c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.748+0000 7fc22bfff640 1 -- 172.21.15.77:0/1642735340 >> v1:172.21.15.158:6789/0 conn(0x7fc22c15fa10 legacy=0x7fc22c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.748+0000 7fc22bfff640 1 -- 172.21.15.77:0/1642735340 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0cf1a0 con 0x7fc22c0d16e0 2024-03-06T07:25:01.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.748+0000 7fc22bfff640 1 --2- 172.21.15.77:0/1642735340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c0d16e0 0x7fc22c0d1af0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0c5520 tx=0x7fc22402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bb4ca0fe5beb27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc22a7fc640 1 -- 172.21.15.77:0/1642735340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22402c9e0 con 0x7fc22c0d16e0 2024-03-06T07:25:01.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc22a7fc640 1 -- 172.21.15.77:0/1642735340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22402cb80 con 0x7fc22c0d16e0 2024-03-06T07:25:01.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc22a7fc640 1 -- 172.21.15.77:0/1642735340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2240343f0 con 0x7fc22c0d16e0 2024-03-06T07:25:01.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc231a9a640 1 -- 172.21.15.77:0/1642735340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c0d16e0 msgr2=0x7fc22c0d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc231a9a640 1 --2- 172.21.15.77:0/1642735340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c0d16e0 0x7fc22c0d1af0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0c5520 tx=0x7fc22402be40 comp rx=0 tx=0).stop 2024-03-06T07:25:01.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc231a9a640 1 -- 172.21.15.77:0/1642735340 shutdown_connections 2024-03-06T07:25:01.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc231a9a640 1 --2- 172.21.15.77:0/1642735340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c0d16e0 0x7fc22c0d1af0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc231a9a640 1 --2- 172.21.15.77:0/1642735340 >> v2:172.21.15.158:3300/0 conn(0x7fc22c15bdd0 0x7fc22c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc231a9a640 1 -- 172.21.15.77:0/1642735340 >> 172.21.15.77:0/1642735340 conn(0x7fc22c09a9f0 msgr2=0x7fc22c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc231a9a640 1 -- 172.21.15.77:0/1642735340 shutdown_connections 2024-03-06T07:25:01.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.749+0000 7fc231a9a640 1 -- 172.21.15.77:0/1642735340 wait complete. 2024-03-06T07:25:01.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.750+0000 7fc231a9a640 1 Processor -- start 2024-03-06T07:25:01.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.750+0000 7fc231a9a640 1 -- start start 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc231a9a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc22c0d16e0 0x7fc22c0b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc231a9a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c15bdd0 0x7fc22c0b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc231a9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc22c15fa10 0x7fc22c0b5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc231a9a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c0cc5d0 con 0x7fc22c15bdd0 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc231a9a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c0c5b50 con 0x7fc22c0d16e0 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc231a9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c15c1b0 con 0x7fc22c15fa10 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c15bdd0 0x7fc22c0b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c15bdd0 0x7fc22c0b58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44528/0 (socket says 172.21.15.77:44528) 2024-03-06T07:25:01.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22affd640 1 -- 172.21.15.77:0/3260275635 learned_addr learned my addr 172.21.15.77:0/3260275635 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22bfff640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc22c15fa10 0x7fc22c0b5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22b7fe640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc22c0d16e0 0x7fc22c0b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22affd640 1 -- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc22c15fa10 msgr2=0x7fc22c0b5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22affd640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc22c15fa10 0x7fc22c0b5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22affd640 1 -- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc22c0d16e0 msgr2=0x7fc22c0b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22affd640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc22c0d16e0 0x7fc22c0b6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22affd640 1 -- 172.21.15.77:0/3260275635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0c9b30 con 0x7fc22c15bdd0 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.751+0000 7fc22bfff640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc22c15fa10 0x7fc22c0b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.752+0000 7fc22b7fe640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc22c0d16e0 0x7fc22c0b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.752+0000 7fc22affd640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c15bdd0 0x7fc22c0b58c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc21400b3f0 tx=0x7fc21400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.752+0000 7fc223fff640 1 -- 172.21.15.77:0/3260275635 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214015020 con 0x7fc22c15bdd0 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.752+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22c15eff0 con 0x7fc22c15bdd0 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.752+0000 7fc223fff640 1 -- 172.21.15.77:0/3260275635 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc214011830 con 0x7fc22c15bdd0 2024-03-06T07:25:01.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.752+0000 7fc223fff640 1 -- 172.21.15.77:0/3260275635 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214011c20 con 0x7fc22c15bdd0 2024-03-06T07:25:01.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.752+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22c0c9df0 con 0x7fc22c15bdd0 2024-03-06T07:25:01.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.753+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc22c0ce770 con 0x7fc22c15bdd0 2024-03-06T07:25:01.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.753+0000 7fc223fff640 1 -- 172.21.15.77:0/3260275635 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc214018020 con 0x7fc22c15bdd0 2024-03-06T07:25:01.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.754+0000 7fc223fff640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.754+0000 7fc22b7fe640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.754+0000 7fc223fff640 1 -- 172.21.15.77:0/3260275635 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc214089e10 con 0x7fc22c15bdd0 2024-03-06T07:25:01.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.754+0000 7fc22b7fe640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00b820 tx=0x7fc21c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.762+0000 7fc223fff640 1 -- 172.21.15.77:0/3260275635 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc22c0ce770 con 0x7fc22c15bdd0 2024-03-06T07:25:01.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.951+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc22c0b7a40 con 0x7fc22c15bdd0 2024-03-06T07:25:01.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.952+0000 7fc223fff640 1 -- 172.21.15.77:0/3260275635 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16656 (secure 0 0 0) 0x7fc21401f030 con 0x7fc22c15bdd0 2024-03-06T07:25:01.954 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:01.954 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":69,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:00.275386+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:56.868266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:57.595004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 msgr2=0x7fc20c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00b820 tx=0x7fc21c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c15bdd0 msgr2=0x7fc22c0b58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c15bdd0 0x7fc22c0b58c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc21400b3f0 tx=0x7fc21400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 shutdown_connections 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc20c06b4d0 0x7fc20c06d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc22c15bdd0 0x7fc22c0b58c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc22c15fa10 0x7fc22c0b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 --2- 172.21.15.77:0/3260275635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc22c0d16e0 0x7fc22c0b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 >> 172.21.15.77:0/3260275635 conn(0x7fc22c09a9f0 msgr2=0x7fc22c0ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 shutdown_connections 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:01.954+0000 7fc231a9a640 1 -- 172.21.15.77:0/3260275635 wait complete. 2024-03-06T07:25:01.967 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:01.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:02.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.103+0000 7f58a74c2640 1 Processor -- start 2024-03-06T07:25:02.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.103+0000 7f58a74c2640 1 -- start start 2024-03-06T07:25:02.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a74c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f58a013dc10 0x7f58a00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a74c2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f58a015ef60 0x7f58a00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a74c2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a015f580 con 0x7f58a015bf30 2024-03-06T07:25:02.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a74c2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a00cf3f0 con 0x7f58a013dc10 2024-03-06T07:25:02.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a74c2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a015b210 con 0x7f58a015ef60 2024-03-06T07:25:02.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a525e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f58a013dc10 0x7f58a00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a525e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f58a013dc10 0x7f58a00b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44536/0 (socket says 172.21.15.77:44536) 2024-03-06T07:25:02.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a525e640 1 -- 172.21.15.77:0/2203636811 learned_addr learned my addr 172.21.15.77:0/2203636811 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:02.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a4a5d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f58a015bf30 0x7f58a015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59348/0 (socket says 172.21.15.77:59348) 2024-03-06T07:25:02.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a5a5f640 1 --2- 172.21.15.77:0/2203636811 >> v2:172.21.15.158:3300/0 conn(0x7f58a015ef60 0x7f58a00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a525e640 1 -- 172.21.15.77:0/2203636811 >> v2:172.21.15.158:3300/0 conn(0x7f58a015ef60 msgr2=0x7f58a00b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a525e640 1 --2- 172.21.15.77:0/2203636811 >> v2:172.21.15.158:3300/0 conn(0x7f58a015ef60 0x7f58a00b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a525e640 1 -- 172.21.15.77:0/2203636811 >> v1:172.21.15.77:6789/0 conn(0x7f58a015bf30 legacy=0x7f58a015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f58a525e640 1 -- 172.21.15.77:0/2203636811 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a015b450 con 0x7f58a013dc10 2024-03-06T07:25:02.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.104+0000 7f5897fff640 1 -- 172.21.15.77:0/2203636811 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279274932 0 0) 0x7f58a015f580 con 0x7f58a015bf30 2024-03-06T07:25:02.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.105+0000 7f58a525e640 1 --2- 172.21.15.77:0/2203636811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a00b2950 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f58a0068700 tx=0x7f589c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796bf5b2d7f16389 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.105+0000 7f5897fff640 1 -- 172.21.15.77:0/2203636811 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c03b070 con 0x7f58a013dc10 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.105+0000 7f5897fff640 1 -- 172.21.15.77:0/2203636811 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c036440 con 0x7f58a013dc10 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.105+0000 7f5897fff640 1 -- 172.21.15.77:0/2203636811 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c036810 con 0x7f58a013dc10 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.106+0000 7f58a74c2640 1 -- 172.21.15.77:0/2203636811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 msgr2=0x7f58a00b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.106+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2203636811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a00b2950 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f58a0068700 tx=0x7f589c02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.106+0000 7f58a74c2640 1 -- 172.21.15.77:0/2203636811 shutdown_connections 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.106+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2203636811 >> v2:172.21.15.158:3300/0 conn(0x7f58a015ef60 0x7f58a00b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.106+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2203636811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a00b2950 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.106+0000 7f58a74c2640 1 -- 172.21.15.77:0/2203636811 >> 172.21.15.77:0/2203636811 conn(0x7f58a00cf8a0 msgr2=0x7f58a00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.106+0000 7f58a74c2640 1 -- 172.21.15.77:0/2203636811 shutdown_connections 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.106+0000 7f58a74c2640 1 -- 172.21.15.77:0/2203636811 wait complete. 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.107+0000 7f58a74c2640 1 Processor -- start 2024-03-06T07:25:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.107+0000 7f58a74c2640 1 -- start start 2024-03-06T07:25:02.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.107+0000 7f58a74c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a013ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.107+0000 7f58a74c2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58a015bf30 0x7f58a0139e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a74c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58a015ef60 0x7f58a0159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a74c2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00cd7a0 con 0x7f58a013dc10 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a74c2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a013dff0 con 0x7f58a015bf30 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a74c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00cd920 con 0x7f58a015ef60 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a013ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a013ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44542/0 (socket says 172.21.15.77:44542) 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 -- 172.21.15.77:0/2764355390 learned_addr learned my addr 172.21.15.77:0/2764355390 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a5a5f640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58a015ef60 0x7f58a0159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a4a5d640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58a015bf30 0x7f58a0139e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 -- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58a015ef60 msgr2=0x7f58a0159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58a015ef60 0x7f58a0159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 -- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58a015bf30 msgr2=0x7f58a0139e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58a015bf30 0x7f58a0139e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 -- 172.21.15.77:0/2764355390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a013b460 con 0x7f58a013dc10 2024-03-06T07:25:02.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a5a5f640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58a015ef60 0x7f58a0159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a4a5d640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58a015bf30 0x7f58a0139e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a525e640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a013ac10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f589c000c00 tx=0x7f589c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f5895ffb640 1 -- 172.21.15.77:0/2764355390 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c046070 con 0x7f58a013dc10 2024-03-06T07:25:02.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f5895ffb640 1 -- 172.21.15.77:0/2764355390 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c004230 con 0x7f58a013dc10 2024-03-06T07:25:02.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a013b740 con 0x7f58a013dc10 2024-03-06T07:25:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f5895ffb640 1 -- 172.21.15.77:0/2764355390 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c03b070 con 0x7f58a013dc10 2024-03-06T07:25:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.108+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a013bd00 con 0x7f58a013dc10 2024-03-06T07:25:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.109+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a015d730 con 0x7f58a013dc10 2024-03-06T07:25:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.110+0000 7f5895ffb640 1 -- 172.21.15.77:0/2764355390 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f589c049070 con 0x7f58a013dc10 2024-03-06T07:25:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.110+0000 7f5895ffb640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f588406b4d0 0x7f588406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.110+0000 7f58a4a5d640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f588406b4d0 0x7f588406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.110+0000 7f5895ffb640 1 -- 172.21.15.77:0/2764355390 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f589c0ac320 con 0x7f58a013dc10 2024-03-06T07:25:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.111+0000 7f58a4a5d640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f588406b4d0 0x7f588406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f58a01f7490 tx=0x7f5890009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.114+0000 7f5895ffb640 1 -- 172.21.15.77:0/2764355390 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58a015d730 con 0x7f58a013dc10 2024-03-06T07:25:02.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.325+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58a013bf40 con 0x7f588406b4d0 2024-03-06T07:25:02.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.330+0000 7f5895ffb640 1 -- 172.21.15.77:0/2764355390 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240120 (secure 0 0 0) 0x7f58a013bf40 con 0x7f588406b4d0 2024-03-06T07:25:02.337 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:02.339 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f588406b4d0 msgr2=0x7f588406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f588406b4d0 0x7f588406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f58a01f7490 tx=0x7f5890009040 comp rx=0 tx=0).stop 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 msgr2=0x7f58a013ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a013ac10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f589c000c00 tx=0x7f589c004730 comp rx=0 tx=0).stop 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 shutdown_connections 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58a015ef60 0x7f58a0159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58a015bf30 0x7f58a0139e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f588406b4d0 0x7f588406d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 --2- 172.21.15.77:0/2764355390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a013dc10 0x7f58a013ac10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 >> 172.21.15.77:0/2764355390 conn(0x7f58a00cf8a0 msgr2=0x7f58a015de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 shutdown_connections 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.333+0000 7f58a74c2640 1 -- 172.21.15.77:0/2764355390 wait complete. 2024-03-06T07:25:02.347 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-03-06T07:25:01.416822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":269296,"kb_used_data":7336,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754705424,"statfs":{"total":773094113280,"available":772818354176,"internally_reserved":0,"allocated":7512064,"data_stored":3798464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.147288"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273762+0000","last_change":"2024-03-06T07:24:57.273762+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273762+0000","last_undegraded":"2024-03-06T07:24:57.273762+0000","last_fullsized":"2024-03-06T07:24:57.273762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271721+0000","last_change":"2024-03-06T07:24:57.271721+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271721+0000","last_undegraded":"2024-03-06T07:24:57.271721+0000","last_fullsized":"2024-03-06T07:24:57.271721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271713+0000","last_change":"2024-03-06T07:24:57.271713+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271713+0000","last_undegraded":"2024-03-06T07:24:57.271713+0000","last_fullsized":"2024-03-06T07:24:57.271713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273236+0000","last_change":"2024-03-06T07:24:57.273236+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273236+0000","last_undegraded":"2024-03-06T07:24:57.273236+0000","last_fullsized":"2024-03-06T07:24:57.273236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275272+0000","last_change":"2024-03-06T07:24:57.275272+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275272+0000","last_undegraded":"2024-03-06T07:24:57.275272+0000","last_fullsized":"2024-03-06T07:24:57.275272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274442+0000","last_change":"2024-03-06T07:24:57.274442+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274442+0000","last_undegraded":"2024-03-06T07:24:57.274442+0000","last_fullsized":"2024-03-06T07:24:57.274442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274528+0000","last_change":"2024-03-06T07:24:57.274528+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274528+0000","last_undegraded":"2024-03-06T07:24:57.274528+0000","last_fullsized":"2024-03-06T07:24:57.274528+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273778+0000","last_change":"2024-03-06T07:24:57.273778+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273778+0000","last_undegraded":"2024-03-06T07:24:57.273778+0000","last_fullsized":"2024-03-06T07:24:57.273778+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271654+0000","last_change":"2024-03-06T07:24:57.271654+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271654+0000","last_undegraded":"2024-03-06T07:24:57.271654+0000","last_fullsized":"2024-03-06T07:24:57.271654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273525+0000","last_change":"2024-03-06T07:24:57.273525+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273525+0000","last_undegraded":"2024-03-06T07:24:57.273525+0000","last_fullsized":"2024-03-06T07:24:57.273525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275295+0000","last_change":"2024-03-06T07:24:57.275295+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275295+0000","last_undegraded":"2024-03-06T07:24:57.275295+0000","last_fullsized":"2024-03-06T07:24:57.275295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274130+0000","last_change":"2024-03-06T07:24:57.274130+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274130+0000","last_undegraded":"2024-03-06T07:24:57.274130+0000","last_fullsized":"2024-03-06T07:24:57.274130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274039+0000","last_change":"2024-03-06T07:24:57.274039+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274039+0000","last_undegraded":"2024-03-06T07:24:57.274039+0000","last_fullsized":"2024-03-06T07:24:57.274039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272658+0000","last_change":"2024-03-06T07:24:57.272658+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272658+0000","last_undegraded":"2024-03-06T07:24:57.272658+0000","last_fullsized":"2024-03-06T07:24:57.272658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.276403+0000","last_change":"2024-03-06T07:24:57.276403+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.276403+0000","last_undegraded":"2024-03-06T07:24:57.276403+0000","last_fullsized":"2024-03-06T07:24:57.276403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271563+0000","last_change":"2024-03-06T07:24:57.271563+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271563+0000","last_undegraded":"2024-03-06T07:24:57.271563+0000","last_fullsized":"2024-03-06T07:24:57.271563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274961+0000","last_change":"2024-03-06T07:24:57.274961+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274961+0000","last_undegraded":"2024-03-06T07:24:57.274961+0000","last_fullsized":"2024-03-06T07:24:57.274961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271570+0000","last_change":"2024-03-06T07:24:57.271570+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271570+0000","last_undegraded":"2024-03-06T07:24:57.271570+0000","last_fullsized":"2024-03-06T07:24:57.271570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273579+0000","last_change":"2024-03-06T07:24:57.273579+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273579+0000","last_undegraded":"2024-03-06T07:24:57.273579+0000","last_fullsized":"2024-03-06T07:24:57.273579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273735+0000","last_change":"2024-03-06T07:24:57.273735+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273735+0000","last_undegraded":"2024-03-06T07:24:57.273735+0000","last_fullsized":"2024-03-06T07:24:57.273735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272097+0000","last_change":"2024-03-06T07:24:57.272097+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272097+0000","last_undegraded":"2024-03-06T07:24:57.272097+0000","last_fullsized":"2024-03-06T07:24:57.272097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275762+0000","last_change":"2024-03-06T07:24:57.275762+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275762+0000","last_undegraded":"2024-03-06T07:24:57.275762+0000","last_fullsized":"2024-03-06T07:24:57.275762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271724+0000","last_change":"2024-03-06T07:24:57.271724+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271724+0000","last_undegraded":"2024-03-06T07:24:57.271724+0000","last_fullsized":"2024-03-06T07:24:57.271724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273839+0000","last_change":"2024-03-06T07:24:57.273839+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273839+0000","last_undegraded":"2024-03-06T07:24:57.273839+0000","last_fullsized":"2024-03-06T07:24:57.273839+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275785+0000","last_change":"2024-03-06T07:24:57.275785+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275785+0000","last_undegraded":"2024-03-06T07:24:57.275785+0000","last_fullsized":"2024-03-06T07:24:57.275785+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273811+0000","last_change":"2024-03-06T07:24:57.273811+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273811+0000","last_undegraded":"2024-03-06T07:24:57.273811+0000","last_fullsized":"2024-03-06T07:24:57.273811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271723+0000","last_change":"2024-03-06T07:24:57.271723+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271723+0000","last_undegraded":"2024-03-06T07:24:57.271723+0000","last_fullsized":"2024-03-06T07:24:57.271723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273793+0000","last_change":"2024-03-06T07:24:57.273793+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273793+0000","last_undegraded":"2024-03-06T07:24:57.273793+0000","last_fullsized":"2024-03-06T07:24:57.273793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274063+0000","last_change":"2024-03-06T07:24:57.274063+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274063+0000","last_undegraded":"2024-03-06T07:24:57.274063+0000","last_fullsized":"2024-03-06T07:24:57.274063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271950+0000","last_change":"2024-03-06T07:24:57.271950+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271950+0000","last_undegraded":"2024-03-06T07:24:57.271950+0000","last_fullsized":"2024-03-06T07:24:57.271950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272662+0000","last_change":"2024-03-06T07:24:57.272662+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272662+0000","last_undegraded":"2024-03-06T07:24:57.272662+0000","last_fullsized":"2024-03-06T07:24:57.272662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272231+0000","last_change":"2024-03-06T07:24:57.272231+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272231+0000","last_undegraded":"2024-03-06T07:24:57.272231+0000","last_fullsized":"2024-03-06T07:24:57.272231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":250,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.274072+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:24:57.274072+0000","last_peered":"2024-03-06T07:24:57.274072+0000","last_clean":"2024-03-06T07:24:57.274072+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:24:57.274072+0000","last_undegraded":"2024-03-06T07:24:57.274072+0000","last_fullsized":"2024-03-06T07:24:57.274072+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1384448,"data_stored":917489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36124,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335716,"statfs":{"total":96636764160,"available":96599773184,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36124,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335716,"statfs":{"total":96636764160,"available":96599773184,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":778240,"data_stored":322064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1384448,"data_stored":917489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:02.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:02.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.486+0000 7fb426d16640 1 Processor -- start 2024-03-06T07:25:02.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.486+0000 7fb426d16640 1 -- start start 2024-03-06T07:25:02.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb426d16640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb42015bb20 0x7fb4200d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb426d16640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb4200cfff0 0x7fb4200d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb426d16640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4200ce4b0 0x7fb4200ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb426d16640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb420161980 con 0x7fb42015bb20 2024-03-06T07:25:02.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb426d16640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb420161c60 con 0x7fb4200cfff0 2024-03-06T07:25:02.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb426d16640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb420161f40 con 0x7fb4200ce4b0 2024-03-06T07:25:02.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb424ab2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb42015bb20 0x7fb4200d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb424ab2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb42015bb20 0x7fb4200d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44552/0 (socket says 172.21.15.77:44552) 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.487+0000 7fb424ab2640 1 -- 172.21.15.77:0/2298764368 learned_addr learned my addr 172.21.15.77:0/2298764368 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.488+0000 7fb424ab2640 1 -- 172.21.15.77:0/2298764368 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4200ce4b0 msgr2=0x7fb4200ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.488+0000 7fb4252b3640 1 --2- 172.21.15.77:0/2298764368 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4200ce4b0 0x7fb4200ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.488+0000 7fb424ab2640 1 --2- 172.21.15.77:0/2298764368 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4200ce4b0 0x7fb4200ce920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.488+0000 7fb424ab2640 1 -- 172.21.15.77:0/2298764368 >> v2:172.21.15.158:3300/0 conn(0x7fb4200cfff0 msgr2=0x7fb4200d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.488+0000 7fb417fff640 1 --2- 172.21.15.77:0/2298764368 >> v2:172.21.15.158:3300/0 conn(0x7fb4200cfff0 0x7fb4200d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.488+0000 7fb424ab2640 1 --2- 172.21.15.77:0/2298764368 >> v2:172.21.15.158:3300/0 conn(0x7fb4200cfff0 0x7fb4200d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.488+0000 7fb424ab2640 1 -- 172.21.15.77:0/2298764368 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420162220 con 0x7fb42015bb20 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.488+0000 7fb424ab2640 1 --2- 172.21.15.77:0/2298764368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42015bb20 0x7fb4200d1df0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb420071290 tx=0x7fb41c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6326312934362db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb4177fe640 1 -- 172.21.15.77:0/2298764368 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c035020 con 0x7fb42015bb20 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb4177fe640 1 -- 172.21.15.77:0/2298764368 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c031870 con 0x7fb42015bb20 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb4177fe640 1 -- 172.21.15.77:0/2298764368 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c031c40 con 0x7fb42015bb20 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 -- 172.21.15.77:0/2298764368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42015bb20 msgr2=0x7fb4200d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 --2- 172.21.15.77:0/2298764368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42015bb20 0x7fb4200d1df0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb420071290 tx=0x7fb41c02b910 comp rx=0 tx=0).stop 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 -- 172.21.15.77:0/2298764368 shutdown_connections 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 --2- 172.21.15.77:0/2298764368 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4200ce4b0 0x7fb4200ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 --2- 172.21.15.77:0/2298764368 >> v2:172.21.15.158:3300/0 conn(0x7fb4200cfff0 0x7fb4200d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 --2- 172.21.15.77:0/2298764368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42015bb20 0x7fb4200d1df0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 -- 172.21.15.77:0/2298764368 >> 172.21.15.77:0/2298764368 conn(0x7fb42009a9f0 msgr2=0x7fb42009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 -- 172.21.15.77:0/2298764368 shutdown_connections 2024-03-06T07:25:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.489+0000 7fb426d16640 1 -- 172.21.15.77:0/2298764368 wait complete. 2024-03-06T07:25:02.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.490+0000 7fb426d16640 1 Processor -- start 2024-03-06T07:25:02.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb426d16640 1 -- start start 2024-03-06T07:25:02.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb426d16640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4200ce4b0 0x7fb420157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb426d16640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4200cfff0 0x7fb4200c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb426d16640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb42015bb20 0x7fb4200c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb426d16640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4200d24b0 con 0x7fb4200cfff0 2024-03-06T07:25:02.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb426d16640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4200d2330 con 0x7fb4200ce4b0 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb426d16640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4200d1980 con 0x7fb42015bb20 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb417fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4200cfff0 0x7fb4200c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.491+0000 7fb417fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4200cfff0 0x7fb4200c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44556/0 (socket says 172.21.15.77:44556) 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb417fff640 1 -- 172.21.15.77:0/2909040240 learned_addr learned my addr 172.21.15.77:0/2909040240 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb4252b3640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb42015bb20 0x7fb4200c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb424ab2640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4200ce4b0 0x7fb420157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb417fff640 1 -- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb42015bb20 msgr2=0x7fb4200c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb417fff640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb42015bb20 0x7fb4200c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb417fff640 1 -- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4200ce4b0 msgr2=0x7fb420157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb417fff640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4200ce4b0 0x7fb420157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb417fff640 1 -- 172.21.15.77:0/2909040240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200c6db0 con 0x7fb4200cfff0 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb424ab2640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4200ce4b0 0x7fb420157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb417fff640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4200cfff0 0x7fb4200c75d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c7f0 tx=0x7fb41000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb4157fa640 1 -- 172.21.15.77:0/2909040240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb410015070 con 0x7fb4200cfff0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb4157fa640 1 -- 172.21.15.77:0/2909040240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb410004240 con 0x7fb4200cfff0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb4157fa640 1 -- 172.21.15.77:0/2909040240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb410007620 con 0x7fb4200cfff0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4200c5f50 con 0x7fb4200cfff0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.492+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4200c6510 con 0x7fb4200cfff0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.493+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42004f330 con 0x7fb4200cfff0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.493+0000 7fb4157fa640 1 -- 172.21.15.77:0/2909040240 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4100079d0 con 0x7fb4200cfff0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.494+0000 7fb4157fa640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb40c06b4d0 0x7fb40c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.494+0000 7fb4157fa640 1 -- 172.21.15.77:0/2909040240 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb41008a610 con 0x7fb4200cfff0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.494+0000 7fb424ab2640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb40c06b4d0 0x7fb40c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.494+0000 7fb424ab2640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb40c06b4d0 0x7fb40c06d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb41c036000 tx=0x7fb41c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.502+0000 7fb4157fa640 1 -- 172.21.15.77:0/2909040240 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb410059490 con 0x7fb4200cfff0 2024-03-06T07:25:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.677+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4200d0690 con 0x7fb40c06b4d0 2024-03-06T07:25:02.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.682+0000 7fb4157fa640 1 -- 172.21.15.77:0/2909040240 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240120 (secure 0 0 0) 0x7fb4200d0690 con 0x7fb40c06b4d0 2024-03-06T07:25:02.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:02.688 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:25:02.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb40c06b4d0 msgr2=0x7fb40c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb40c06b4d0 0x7fb40c06d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb41c036000 tx=0x7fb41c03c040 comp rx=0 tx=0).stop 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4200cfff0 msgr2=0x7fb4200c75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4200cfff0 0x7fb4200c75d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c7f0 tx=0x7fb41000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 shutdown_connections 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb40c06b4d0 0x7fb40c06d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4200cfff0 0x7fb4200c75d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb42015bb20 0x7fb4200c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 --2- 172.21.15.77:0/2909040240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4200ce4b0 0x7fb420157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.685+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 >> 172.21.15.77:0/2909040240 conn(0x7fb42009a9f0 msgr2=0x7fb4200b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.686+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 shutdown_connections 2024-03-06T07:25:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.686+0000 7fb426d16640 1 -- 172.21.15.77:0/2909040240 wait complete. 2024-03-06T07:25:02.699 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-03-06T07:25:01.416822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":269296,"kb_used_data":7336,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754705424,"statfs":{"total":773094113280,"available":772818354176,"internally_reserved":0,"allocated":7512064,"data_stored":3798464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.147288"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273762+0000","last_change":"2024-03-06T07:24:57.273762+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273762+0000","last_undegraded":"2024-03-06T07:24:57.273762+0000","last_fullsized":"2024-03-06T07:24:57.273762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271721+0000","last_change":"2024-03-06T07:24:57.271721+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271721+0000","last_undegraded":"2024-03-06T07:24:57.271721+0000","last_fullsized":"2024-03-06T07:24:57.271721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271713+0000","last_change":"2024-03-06T07:24:57.271713+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271713+0000","last_undegraded":"2024-03-06T07:24:57.271713+0000","last_fullsized":"2024-03-06T07:24:57.271713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273236+0000","last_change":"2024-03-06T07:24:57.273236+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273236+0000","last_undegraded":"2024-03-06T07:24:57.273236+0000","last_fullsized":"2024-03-06T07:24:57.273236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275272+0000","last_change":"2024-03-06T07:24:57.275272+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275272+0000","last_undegraded":"2024-03-06T07:24:57.275272+0000","last_fullsized":"2024-03-06T07:24:57.275272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274442+0000","last_change":"2024-03-06T07:24:57.274442+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274442+0000","last_undegraded":"2024-03-06T07:24:57.274442+0000","last_fullsized":"2024-03-06T07:24:57.274442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274528+0000","last_change":"2024-03-06T07:24:57.274528+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274528+0000","last_undegraded":"2024-03-06T07:24:57.274528+0000","last_fullsized":"2024-03-06T07:24:57.274528+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273778+0000","last_change":"2024-03-06T07:24:57.273778+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273778+0000","last_undegraded":"2024-03-06T07:24:57.273778+0000","last_fullsized":"2024-03-06T07:24:57.273778+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271654+0000","last_change":"2024-03-06T07:24:57.271654+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271654+0000","last_undegraded":"2024-03-06T07:24:57.271654+0000","last_fullsized":"2024-03-06T07:24:57.271654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273525+0000","last_change":"2024-03-06T07:24:57.273525+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273525+0000","last_undegraded":"2024-03-06T07:24:57.273525+0000","last_fullsized":"2024-03-06T07:24:57.273525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275295+0000","last_change":"2024-03-06T07:24:57.275295+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275295+0000","last_undegraded":"2024-03-06T07:24:57.275295+0000","last_fullsized":"2024-03-06T07:24:57.275295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274130+0000","last_change":"2024-03-06T07:24:57.274130+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274130+0000","last_undegraded":"2024-03-06T07:24:57.274130+0000","last_fullsized":"2024-03-06T07:24:57.274130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274039+0000","last_change":"2024-03-06T07:24:57.274039+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274039+0000","last_undegraded":"2024-03-06T07:24:57.274039+0000","last_fullsized":"2024-03-06T07:24:57.274039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272658+0000","last_change":"2024-03-06T07:24:57.272658+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272658+0000","last_undegraded":"2024-03-06T07:24:57.272658+0000","last_fullsized":"2024-03-06T07:24:57.272658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.276403+0000","last_change":"2024-03-06T07:24:57.276403+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.276403+0000","last_undegraded":"2024-03-06T07:24:57.276403+0000","last_fullsized":"2024-03-06T07:24:57.276403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271563+0000","last_change":"2024-03-06T07:24:57.271563+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271563+0000","last_undegraded":"2024-03-06T07:24:57.271563+0000","last_fullsized":"2024-03-06T07:24:57.271563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274961+0000","last_change":"2024-03-06T07:24:57.274961+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274961+0000","last_undegraded":"2024-03-06T07:24:57.274961+0000","last_fullsized":"2024-03-06T07:24:57.274961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271570+0000","last_change":"2024-03-06T07:24:57.271570+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271570+0000","last_undegraded":"2024-03-06T07:24:57.271570+0000","last_fullsized":"2024-03-06T07:24:57.271570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273579+0000","last_change":"2024-03-06T07:24:57.273579+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273579+0000","last_undegraded":"2024-03-06T07:24:57.273579+0000","last_fullsized":"2024-03-06T07:24:57.273579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273735+0000","last_change":"2024-03-06T07:24:57.273735+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273735+0000","last_undegraded":"2024-03-06T07:24:57.273735+0000","last_fullsized":"2024-03-06T07:24:57.273735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272097+0000","last_change":"2024-03-06T07:24:57.272097+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272097+0000","last_undegraded":"2024-03-06T07:24:57.272097+0000","last_fullsized":"2024-03-06T07:24:57.272097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275762+0000","last_change":"2024-03-06T07:24:57.275762+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275762+0000","last_undegraded":"2024-03-06T07:24:57.275762+0000","last_fullsized":"2024-03-06T07:24:57.275762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271724+0000","last_change":"2024-03-06T07:24:57.271724+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271724+0000","last_undegraded":"2024-03-06T07:24:57.271724+0000","last_fullsized":"2024-03-06T07:24:57.271724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273839+0000","last_change":"2024-03-06T07:24:57.273839+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273839+0000","last_undegraded":"2024-03-06T07:24:57.273839+0000","last_fullsized":"2024-03-06T07:24:57.273839+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275785+0000","last_change":"2024-03-06T07:24:57.275785+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275785+0000","last_undegraded":"2024-03-06T07:24:57.275785+0000","last_fullsized":"2024-03-06T07:24:57.275785+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273811+0000","last_change":"2024-03-06T07:24:57.273811+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273811+0000","last_undegraded":"2024-03-06T07:24:57.273811+0000","last_fullsized":"2024-03-06T07:24:57.273811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271723+0000","last_change":"2024-03-06T07:24:57.271723+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271723+0000","last_undegraded":"2024-03-06T07:24:57.271723+0000","last_fullsized":"2024-03-06T07:24:57.271723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273793+0000","last_change":"2024-03-06T07:24:57.273793+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273793+0000","last_undegraded":"2024-03-06T07:24:57.273793+0000","last_fullsized":"2024-03-06T07:24:57.273793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274063+0000","last_change":"2024-03-06T07:24:57.274063+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274063+0000","last_undegraded":"2024-03-06T07:24:57.274063+0000","last_fullsized":"2024-03-06T07:24:57.274063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271950+0000","last_change":"2024-03-06T07:24:57.271950+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271950+0000","last_undegraded":"2024-03-06T07:24:57.271950+0000","last_fullsized":"2024-03-06T07:24:57.271950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272662+0000","last_change":"2024-03-06T07:24:57.272662+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272662+0000","last_undegraded":"2024-03-06T07:24:57.272662+0000","last_fullsized":"2024-03-06T07:24:57.272662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272231+0000","last_change":"2024-03-06T07:24:57.272231+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272231+0000","last_undegraded":"2024-03-06T07:24:57.272231+0000","last_fullsized":"2024-03-06T07:24:57.272231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":250,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.274072+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:24:57.274072+0000","last_peered":"2024-03-06T07:24:57.274072+0000","last_clean":"2024-03-06T07:24:57.274072+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:24:57.274072+0000","last_undegraded":"2024-03-06T07:24:57.274072+0000","last_fullsized":"2024-03-06T07:24:57.274072+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1384448,"data_stored":917489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36124,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335716,"statfs":{"total":96636764160,"available":96599773184,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36124,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335716,"statfs":{"total":96636764160,"available":96599773184,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":778240,"data_stored":322064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1384448,"data_stored":917489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:02.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:02.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.839+0000 7f81467bc640 1 Processor -- start 2024-03-06T07:25:02.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.839+0000 7f81467bc640 1 -- start start 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.840+0000 7f81467bc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f81400d22b0 0x7f81400d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.840+0000 7f81467bc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81400b3430 con 0x7f81400b41f0 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.840+0000 7f81467bc640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81400b3710 con 0x7f814015dfe0 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.840+0000 7f81467bc640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81400d2c50 con 0x7f81400d22b0 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.840+0000 7f8144d59640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f81400b41f0 0x7f81400b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59350/0 (socket says 172.21.15.77:59350) 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.840+0000 7f8144d59640 1 -- 172.21.15.77:0/518335792 learned_addr learned my addr 172.21.15.77:0/518335792 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793210310 0 0) 0x7f81400b3430 con 0x7f81400b41f0 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c003520 con 0x7f81400b41f0 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813f7fe640 1 --2- 172.21.15.77:0/518335792 >> v2:172.21.15.158:3300/0 conn(0x7f81400d22b0 0x7f81400d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3597204531 0 0) 0x7f812c003520 con 0x7f81400b41f0 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 >> v2:172.21.15.158:3300/0 conn(0x7f81400d22b0 msgr2=0x7f81400d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 --2- 172.21.15.77:0/518335792 >> v2:172.21.15.158:3300/0 conn(0x7f81400d22b0 0x7f81400d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 >> v1:172.21.15.158:6789/0 conn(0x7f814015dfe0 legacy=0x7f81400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81400b39f0 con 0x7f81400b41f0 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f81380019f0 con 0x7f81400b41f0 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 >> v1:172.21.15.77:6789/0 conn(0x7f81400b41f0 legacy=0x7f81400b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f812c0039b0 0x7f812c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c004800 0x7f812c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f812c0051c0 0x7f812c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c00c030 con 0x7f812c0039b0 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c00c1e0 con 0x7f812c004800 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812c00c3e0 con 0x7f812c0051c0 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813ffff640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f812c0039b0 0x7f812c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813f7fe640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c004800 0x7f812c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f8144d59640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f812c0051c0 0x7f812c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813ffff640 1 -- 172.21.15.77:0/518335792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f812c0051c0 msgr2=0x7f812c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813ffff640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f812c0051c0 0x7f812c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813ffff640 1 -- 172.21.15.77:0/518335792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c004800 msgr2=0x7f812c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813ffff640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c004800 0x7f812c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f8144d59640 1 -- 172.21.15.77:0/518335792 reap_dead start 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.841+0000 7f813ffff640 1 -- 172.21.15.77:0/518335792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c003520 con 0x7f812c0039b0 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.842+0000 7f813ffff640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f812c0039b0 0x7f812c003dd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8130001770 tx=0x7f812c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8aacaa00745ae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.842+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130001390 con 0x7f812c0039b0 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.842+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81300052d0 con 0x7f812c0039b0 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.842+0000 7f813effd640 1 -- 172.21.15.77:0/518335792 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130003830 con 0x7f812c0039b0 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.843+0000 7f81467bc640 1 -- 172.21.15.77:0/518335792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f812c0039b0 msgr2=0x7f812c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.843+0000 7f81467bc640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f812c0039b0 0x7f812c003dd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8130001770 tx=0x7f812c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:02.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.843+0000 7f81467bc640 1 -- 172.21.15.77:0/518335792 shutdown_connections 2024-03-06T07:25:02.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.843+0000 7f81467bc640 1 --2- 172.21.15.77:0/518335792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f812c0039b0 0x7f812c003dd0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.843+0000 7f81467bc640 1 -- 172.21.15.77:0/518335792 >> 172.21.15.77:0/518335792 conn(0x7f81400d3cd0 msgr2=0x7f81400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.843+0000 7f81467bc640 1 -- 172.21.15.77:0/518335792 shutdown_connections 2024-03-06T07:25:02.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.843+0000 7f81467bc640 1 -- 172.21.15.77:0/518335792 wait complete. 2024-03-06T07:25:02.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.844+0000 7f81467bc640 1 Processor -- start 2024-03-06T07:25:02.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.844+0000 7f81467bc640 1 -- start start 2024-03-06T07:25:02.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f81467bc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c0039b0 0x7f81400cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f81467bc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f81400b98e0 0x7f814014f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f81467bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81400b7480 0x7f81401602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f81467bc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81400b3d60 con 0x7f81400b7480 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f81467bc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81400b31e0 con 0x7f812c0039b0 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f81467bc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814015e3c0 con 0x7f81400b98e0 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f8144d59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81400b7480 0x7f81401602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f8144d59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81400b7480 0x7f81401602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44564/0 (socket says 172.21.15.77:44564) 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f8144d59640 1 -- 172.21.15.77:0/2784230894 learned_addr learned my addr 172.21.15.77:0/2784230894 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f813ffff640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c0039b0 0x7f81400cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f813f7fe640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f81400b98e0 0x7f814014f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f8144d59640 1 -- 172.21.15.77:0/2784230894 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f81400b98e0 msgr2=0x7f814014f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f8144d59640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f81400b98e0 0x7f814014f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f8144d59640 1 -- 172.21.15.77:0/2784230894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c0039b0 msgr2=0x7f81400cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f8144d59640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c0039b0 0x7f81400cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.845+0000 7f8144d59640 1 -- 172.21.15.77:0/2784230894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81400b7d60 con 0x7f81400b7480 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.846+0000 7f8144d59640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81400b7480 0x7f81401602f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f814008c940 tx=0x7f813800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.846+0000 7f813cff9640 1 -- 172.21.15.77:0/2784230894 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813800eb90 con 0x7f81400b7480 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.846+0000 7f813cff9640 1 -- 172.21.15.77:0/2784230894 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8138018070 con 0x7f81400b7480 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.846+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81400caf20 con 0x7f81400b7480 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.846+0000 7f813cff9640 1 -- 172.21.15.77:0/2784230894 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138013420 con 0x7f81400b7480 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.846+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81400cb4e0 con 0x7f81400b7480 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.847+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814004f330 con 0x7f81400b7480 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.847+0000 7f813cff9640 1 -- 172.21.15.77:0/2784230894 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81380135c0 con 0x7f81400b7480 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.848+0000 7f813cff9640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f812006b500 0x7f812006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.848+0000 7f813ffff640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f812006b500 0x7f812006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.848+0000 7f813cff9640 1 -- 172.21.15.77:0/2784230894 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f813808a570 con 0x7f81400b7480 2024-03-06T07:25:02.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.848+0000 7f813ffff640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f812006b500 0x7f812006d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8130033000 tx=0x7f8130009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:02.852+0000 7f813cff9640 1 -- 172.21.15.77:0/2784230894 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8138059350 con 0x7f81400b7480 2024-03-06T07:25:03.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.057+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81400cb6b0 con 0x7f812006b500 2024-03-06T07:25:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.062+0000 7f813cff9640 1 -- 172.21.15.77:0/2784230894 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240120 (secure 0 0 0) 0x7f81400cb6b0 con 0x7f812006b500 2024-03-06T07:25:03.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:03.068 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:25:03.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f812006b500 msgr2=0x7f812006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f812006b500 0x7f812006d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8130033000 tx=0x7f8130009030 comp rx=0 tx=0).stop 2024-03-06T07:25:03.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81400b7480 msgr2=0x7f81401602f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81400b7480 0x7f81401602f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f814008c940 tx=0x7f813800e530 comp rx=0 tx=0).stop 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 shutdown_connections 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f812006b500 0x7f812006d9c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81400b7480 0x7f81401602f0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f81400b98e0 0x7f814014f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 --2- 172.21.15.77:0/2784230894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f812c0039b0 0x7f81400cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 >> 172.21.15.77:0/2784230894 conn(0x7f81400d3cd0 msgr2=0x7f814006c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 shutdown_connections 2024-03-06T07:25:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.065+0000 7f81467bc640 1 -- 172.21.15.77:0/2784230894 wait complete. 2024-03-06T07:25:03.078 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-03-06T07:25:01.416822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":269296,"kb_used_data":7336,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754705424,"statfs":{"total":773094113280,"available":772818354176,"internally_reserved":0,"allocated":7512064,"data_stored":3798464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.147288"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273762+0000","last_change":"2024-03-06T07:24:57.273762+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273762+0000","last_undegraded":"2024-03-06T07:24:57.273762+0000","last_fullsized":"2024-03-06T07:24:57.273762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271721+0000","last_change":"2024-03-06T07:24:57.271721+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271721+0000","last_undegraded":"2024-03-06T07:24:57.271721+0000","last_fullsized":"2024-03-06T07:24:57.271721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271713+0000","last_change":"2024-03-06T07:24:57.271713+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271713+0000","last_undegraded":"2024-03-06T07:24:57.271713+0000","last_fullsized":"2024-03-06T07:24:57.271713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273236+0000","last_change":"2024-03-06T07:24:57.273236+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273236+0000","last_undegraded":"2024-03-06T07:24:57.273236+0000","last_fullsized":"2024-03-06T07:24:57.273236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275272+0000","last_change":"2024-03-06T07:24:57.275272+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275272+0000","last_undegraded":"2024-03-06T07:24:57.275272+0000","last_fullsized":"2024-03-06T07:24:57.275272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274442+0000","last_change":"2024-03-06T07:24:57.274442+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274442+0000","last_undegraded":"2024-03-06T07:24:57.274442+0000","last_fullsized":"2024-03-06T07:24:57.274442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274528+0000","last_change":"2024-03-06T07:24:57.274528+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274528+0000","last_undegraded":"2024-03-06T07:24:57.274528+0000","last_fullsized":"2024-03-06T07:24:57.274528+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273778+0000","last_change":"2024-03-06T07:24:57.273778+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273778+0000","last_undegraded":"2024-03-06T07:24:57.273778+0000","last_fullsized":"2024-03-06T07:24:57.273778+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271654+0000","last_change":"2024-03-06T07:24:57.271654+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271654+0000","last_undegraded":"2024-03-06T07:24:57.271654+0000","last_fullsized":"2024-03-06T07:24:57.271654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273525+0000","last_change":"2024-03-06T07:24:57.273525+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273525+0000","last_undegraded":"2024-03-06T07:24:57.273525+0000","last_fullsized":"2024-03-06T07:24:57.273525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275295+0000","last_change":"2024-03-06T07:24:57.275295+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275295+0000","last_undegraded":"2024-03-06T07:24:57.275295+0000","last_fullsized":"2024-03-06T07:24:57.275295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274130+0000","last_change":"2024-03-06T07:24:57.274130+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274130+0000","last_undegraded":"2024-03-06T07:24:57.274130+0000","last_fullsized":"2024-03-06T07:24:57.274130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274039+0000","last_change":"2024-03-06T07:24:57.274039+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274039+0000","last_undegraded":"2024-03-06T07:24:57.274039+0000","last_fullsized":"2024-03-06T07:24:57.274039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272658+0000","last_change":"2024-03-06T07:24:57.272658+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272658+0000","last_undegraded":"2024-03-06T07:24:57.272658+0000","last_fullsized":"2024-03-06T07:24:57.272658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.276403+0000","last_change":"2024-03-06T07:24:57.276403+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.276403+0000","last_undegraded":"2024-03-06T07:24:57.276403+0000","last_fullsized":"2024-03-06T07:24:57.276403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271563+0000","last_change":"2024-03-06T07:24:57.271563+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271563+0000","last_undegraded":"2024-03-06T07:24:57.271563+0000","last_fullsized":"2024-03-06T07:24:57.271563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274961+0000","last_change":"2024-03-06T07:24:57.274961+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274961+0000","last_undegraded":"2024-03-06T07:24:57.274961+0000","last_fullsized":"2024-03-06T07:24:57.274961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271570+0000","last_change":"2024-03-06T07:24:57.271570+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271570+0000","last_undegraded":"2024-03-06T07:24:57.271570+0000","last_fullsized":"2024-03-06T07:24:57.271570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273579+0000","last_change":"2024-03-06T07:24:57.273579+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273579+0000","last_undegraded":"2024-03-06T07:24:57.273579+0000","last_fullsized":"2024-03-06T07:24:57.273579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273735+0000","last_change":"2024-03-06T07:24:57.273735+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273735+0000","last_undegraded":"2024-03-06T07:24:57.273735+0000","last_fullsized":"2024-03-06T07:24:57.273735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272097+0000","last_change":"2024-03-06T07:24:57.272097+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272097+0000","last_undegraded":"2024-03-06T07:24:57.272097+0000","last_fullsized":"2024-03-06T07:24:57.272097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275762+0000","last_change":"2024-03-06T07:24:57.275762+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275762+0000","last_undegraded":"2024-03-06T07:24:57.275762+0000","last_fullsized":"2024-03-06T07:24:57.275762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271724+0000","last_change":"2024-03-06T07:24:57.271724+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271724+0000","last_undegraded":"2024-03-06T07:24:57.271724+0000","last_fullsized":"2024-03-06T07:24:57.271724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273839+0000","last_change":"2024-03-06T07:24:57.273839+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273839+0000","last_undegraded":"2024-03-06T07:24:57.273839+0000","last_fullsized":"2024-03-06T07:24:57.273839+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.275785+0000","last_change":"2024-03-06T07:24:57.275785+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.275785+0000","last_undegraded":"2024-03-06T07:24:57.275785+0000","last_fullsized":"2024-03-06T07:24:57.275785+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273811+0000","last_change":"2024-03-06T07:24:57.273811+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273811+0000","last_undegraded":"2024-03-06T07:24:57.273811+0000","last_fullsized":"2024-03-06T07:24:57.273811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271723+0000","last_change":"2024-03-06T07:24:57.271723+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271723+0000","last_undegraded":"2024-03-06T07:24:57.271723+0000","last_fullsized":"2024-03-06T07:24:57.271723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.273793+0000","last_change":"2024-03-06T07:24:57.273793+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.273793+0000","last_undegraded":"2024-03-06T07:24:57.273793+0000","last_fullsized":"2024-03-06T07:24:57.273793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.274063+0000","last_change":"2024-03-06T07:24:57.274063+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.274063+0000","last_undegraded":"2024-03-06T07:24:57.274063+0000","last_fullsized":"2024-03-06T07:24:57.274063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.271950+0000","last_change":"2024-03-06T07:24:57.271950+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.271950+0000","last_undegraded":"2024-03-06T07:24:57.271950+0000","last_fullsized":"2024-03-06T07:24:57.271950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272662+0000","last_change":"2024-03-06T07:24:57.272662+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272662+0000","last_undegraded":"2024-03-06T07:24:57.272662+0000","last_fullsized":"2024-03-06T07:24:57.272662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:58.269595+0000","last_change":"2024-03-06T07:24:58.269595+0000","last_active":"2024-03-06T07:24:58.269595+0000","last_peered":"2024-03-06T07:24:58.269595+0000","last_clean":"2024-03-06T07:24:58.269595+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:58.269595+0000","last_undegraded":"2024-03-06T07:24:58.269595+0000","last_fullsized":"2024-03-06T07:24:58.269595+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-03-06T07:24:57.272231+0000","last_change":"2024-03-06T07:24:57.272231+0000","last_active":"2024-03-06T07:24:57.267765+0000","last_peered":"2024-03-06T07:24:57.267765+0000","last_clean":"2024-03-06T07:24:57.267765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:57.272231+0000","last_undegraded":"2024-03-06T07:24:57.272231+0000","last_fullsized":"2024-03-06T07:24:57.272231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":250,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.274072+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:24:57.274072+0000","last_peered":"2024-03-06T07:24:57.274072+0000","last_clean":"2024-03-06T07:24:57.274072+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:24:57.274072+0000","last_undegraded":"2024-03-06T07:24:57.274072+0000","last_fullsized":"2024-03-06T07:24:57.274072+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1384448,"data_stored":917489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36124,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335716,"statfs":{"total":96636764160,"available":96599773184,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36124,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335716,"statfs":{"total":96636764160,"available":96599773184,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":778240,"data_stored":322064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":327121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":332938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1384448,"data_stored":917489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3f is not active+clean 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3e is not active+clean 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3d is not active+clean 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3c is not active+clean 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3b is not active+clean 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3a is not active+clean 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.39 is not active+clean 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.38 is not active+clean 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.37 is not active+clean 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.36 is not active+clean 2024-03-06T07:25:03.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.35 is not active+clean 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.34 is not active+clean 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.33 is not active+clean 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.32 is not active+clean 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.31 is not active+clean 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.30 is not active+clean 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2f is not active+clean 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2e is not active+clean 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2d is not active+clean 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2c is not active+clean 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2b is not active+clean 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2a is not active+clean 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.29 is not active+clean 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.28 is not active+clean 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.27 is not active+clean 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.26 is not active+clean 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.25 is not active+clean 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.24 is not active+clean 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.23 is not active+clean 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.22 is not active+clean 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.21 is not active+clean 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.20 is not active+clean 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1f is not active+clean 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273762+0000', 'last_change': '2024-03-06T07:24:57.273762+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273762+0000', 'last_undegraded': '2024-03-06T07:24:57.273762+0000', 'last_fullsized': '2024-03-06T07:24:57.273762+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1e is not active+clean 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.271721+0000', 'last_change': '2024-03-06T07:24:57.271721+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.271721+0000', 'last_undegraded': '2024-03-06T07:24:57.271721+0000', 'last_fullsized': '2024-03-06T07:24:57.271721+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1d is not active+clean 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.271713+0000', 'last_change': '2024-03-06T07:24:57.271713+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.271713+0000', 'last_undegraded': '2024-03-06T07:24:57.271713+0000', 'last_fullsized': '2024-03-06T07:24:57.271713+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-03-06T07:25:03.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1c is not active+clean 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273236+0000', 'last_change': '2024-03-06T07:24:57.273236+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273236+0000', 'last_undegraded': '2024-03-06T07:24:57.273236+0000', 'last_fullsized': '2024-03-06T07:24:57.273236+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1b is not active+clean 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.275272+0000', 'last_change': '2024-03-06T07:24:57.275272+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.275272+0000', 'last_undegraded': '2024-03-06T07:24:57.275272+0000', 'last_fullsized': '2024-03-06T07:24:57.275272+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1a is not active+clean 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.274442+0000', 'last_change': '2024-03-06T07:24:57.274442+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.274442+0000', 'last_undegraded': '2024-03-06T07:24:57.274442+0000', 'last_fullsized': '2024-03-06T07:24:57.274442+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-03-06T07:25:03.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.19 is not active+clean 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.274528+0000', 'last_change': '2024-03-06T07:24:57.274528+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.274528+0000', 'last_undegraded': '2024-03-06T07:24:57.274528+0000', 'last_fullsized': '2024-03-06T07:24:57.274528+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.18 is not active+clean 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273778+0000', 'last_change': '2024-03-06T07:24:57.273778+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273778+0000', 'last_undegraded': '2024-03-06T07:24:57.273778+0000', 'last_fullsized': '2024-03-06T07:24:57.273778+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.17 is not active+clean 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.271654+0000', 'last_change': '2024-03-06T07:24:57.271654+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.271654+0000', 'last_undegraded': '2024-03-06T07:24:57.271654+0000', 'last_fullsized': '2024-03-06T07:24:57.271654+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-03-06T07:25:03.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.16 is not active+clean 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273525+0000', 'last_change': '2024-03-06T07:24:57.273525+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273525+0000', 'last_undegraded': '2024-03-06T07:24:57.273525+0000', 'last_fullsized': '2024-03-06T07:24:57.273525+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.15 is not active+clean 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.275295+0000', 'last_change': '2024-03-06T07:24:57.275295+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.275295+0000', 'last_undegraded': '2024-03-06T07:24:57.275295+0000', 'last_fullsized': '2024-03-06T07:24:57.275295+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.14 is not active+clean 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.274130+0000', 'last_change': '2024-03-06T07:24:57.274130+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.274130+0000', 'last_undegraded': '2024-03-06T07:24:57.274130+0000', 'last_fullsized': '2024-03-06T07:24:57.274130+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.13 is not active+clean 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.274039+0000', 'last_change': '2024-03-06T07:24:57.274039+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.274039+0000', 'last_undegraded': '2024-03-06T07:24:57.274039+0000', 'last_fullsized': '2024-03-06T07:24:57.274039+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.12 is not active+clean 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.272658+0000', 'last_change': '2024-03-06T07:24:57.272658+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.272658+0000', 'last_undegraded': '2024-03-06T07:24:57.272658+0000', 'last_fullsized': '2024-03-06T07:24:57.272658+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-03-06T07:25:03.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.11 is not active+clean 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.276403+0000', 'last_change': '2024-03-06T07:24:57.276403+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.276403+0000', 'last_undegraded': '2024-03-06T07:24:57.276403+0000', 'last_fullsized': '2024-03-06T07:24:57.276403+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.10 is not active+clean 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.271563+0000', 'last_change': '2024-03-06T07:24:57.271563+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.271563+0000', 'last_undegraded': '2024-03-06T07:24:57.271563+0000', 'last_fullsized': '2024-03-06T07:24:57.271563+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.f is not active+clean 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.274961+0000', 'last_change': '2024-03-06T07:24:57.274961+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.274961+0000', 'last_undegraded': '2024-03-06T07:24:57.274961+0000', 'last_fullsized': '2024-03-06T07:24:57.274961+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.e is not active+clean 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.271570+0000', 'last_change': '2024-03-06T07:24:57.271570+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.271570+0000', 'last_undegraded': '2024-03-06T07:24:57.271570+0000', 'last_fullsized': '2024-03-06T07:24:57.271570+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.d is not active+clean 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273579+0000', 'last_change': '2024-03-06T07:24:57.273579+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273579+0000', 'last_undegraded': '2024-03-06T07:24:57.273579+0000', 'last_fullsized': '2024-03-06T07:24:57.273579+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.c is not active+clean 2024-03-06T07:25:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273735+0000', 'last_change': '2024-03-06T07:24:57.273735+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273735+0000', 'last_undegraded': '2024-03-06T07:24:57.273735+0000', 'last_fullsized': '2024-03-06T07:24:57.273735+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.a is not active+clean 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.272097+0000', 'last_change': '2024-03-06T07:24:57.272097+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.272097+0000', 'last_undegraded': '2024-03-06T07:24:57.272097+0000', 'last_fullsized': '2024-03-06T07:24:57.272097+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.9 is not active+clean 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.275762+0000', 'last_change': '2024-03-06T07:24:57.275762+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.275762+0000', 'last_undegraded': '2024-03-06T07:24:57.275762+0000', 'last_fullsized': '2024-03-06T07:24:57.275762+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.8 is not active+clean 2024-03-06T07:25:03.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.271724+0000', 'last_change': '2024-03-06T07:24:57.271724+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.271724+0000', 'last_undegraded': '2024-03-06T07:24:57.271724+0000', 'last_fullsized': '2024-03-06T07:24:57.271724+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.7 is not active+clean 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273839+0000', 'last_change': '2024-03-06T07:24:57.273839+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273839+0000', 'last_undegraded': '2024-03-06T07:24:57.273839+0000', 'last_fullsized': '2024-03-06T07:24:57.273839+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.6 is not active+clean 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.275785+0000', 'last_change': '2024-03-06T07:24:57.275785+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.275785+0000', 'last_undegraded': '2024-03-06T07:24:57.275785+0000', 'last_fullsized': '2024-03-06T07:24:57.275785+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.5 is not active+clean 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273811+0000', 'last_change': '2024-03-06T07:24:57.273811+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273811+0000', 'last_undegraded': '2024-03-06T07:24:57.273811+0000', 'last_fullsized': '2024-03-06T07:24:57.273811+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.4 is not active+clean 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.271723+0000', 'last_change': '2024-03-06T07:24:57.271723+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.271723+0000', 'last_undegraded': '2024-03-06T07:24:57.271723+0000', 'last_fullsized': '2024-03-06T07:24:57.271723+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3 is not active+clean 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.273793+0000', 'last_change': '2024-03-06T07:24:57.273793+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.273793+0000', 'last_undegraded': '2024-03-06T07:24:57.273793+0000', 'last_fullsized': '2024-03-06T07:24:57.273793+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-03-06T07:25:03.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2 is not active+clean 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.274063+0000', 'last_change': '2024-03-06T07:24:57.274063+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.274063+0000', 'last_undegraded': '2024-03-06T07:24:57.274063+0000', 'last_fullsized': '2024-03-06T07:24:57.274063+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1 is not active+clean 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.271950+0000', 'last_change': '2024-03-06T07:24:57.271950+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.271950+0000', 'last_undegraded': '2024-03-06T07:24:57.271950+0000', 'last_fullsized': '2024-03-06T07:24:57.271950+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.0 is not active+clean 2024-03-06T07:25:03.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.272662+0000', 'last_change': '2024-03-06T07:24:57.272662+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.272662+0000', 'last_undegraded': '2024-03-06T07:24:57.272662+0000', 'last_fullsized': '2024-03-06T07:24:57.272662+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-03-06T07:25:03.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:58.269595+0000', 'last_change': '2024-03-06T07:24:58.269595+0000', 'last_active': '2024-03-06T07:24:58.269595+0000', 'last_peered': '2024-03-06T07:24:58.269595+0000', 'last_clean': '2024-03-06T07:24:58.269595+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:58.269595+0000', 'last_undegraded': '2024-03-06T07:24:58.269595+0000', 'last_fullsized': '2024-03-06T07:24:58.269595+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:58.269595+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:03.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.b is not active+clean 2024-03-06T07:25:03.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 66, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:57.272231+0000', 'last_change': '2024-03-06T07:24:57.272231+0000', 'last_active': '2024-03-06T07:24:57.267765+0000', 'last_peered': '2024-03-06T07:24:57.267765+0000', 'last_clean': '2024-03-06T07:24:57.267765+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:57.272231+0000', 'last_undegraded': '2024-03-06T07:24:57.272231+0000', 'last_fullsized': '2024-03-06T07:24:57.272231+0000', 'mapping_epoch': 66, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 66, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:57.267765+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:03.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:25:03.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.217+0000 7fc6efb5e640 1 Processor -- start 2024-03-06T07:25:03.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6efb5e640 1 -- start start 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6efb5e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6e80b4460 0x7fc6e80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6efb5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 0x7fc6e80b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6efb5e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e80d22c0 con 0x7fc6e8161cf0 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6efb5e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e8161b00 con 0x7fc6e80b4460 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6efb5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e80cecc0 con 0x7fc6e8162970 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6ed8fa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6e80b4460 0x7fc6e80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6ed8fa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6e80b4460 0x7fc6e80d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44580/0 (socket says 172.21.15.77:44580) 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.219+0000 7fc6ed8fa640 1 -- 172.21.15.77:0/3025095884 learned_addr learned my addr 172.21.15.77:0/3025095884 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.218+0000 7fc6ee0fb640 1 --1- 172.21.15.77:0/3025095884 >> v1:172.21.15.77:6789/0 conn(0x7fc6e8161cf0 0x7fc6e80cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59362/0 (socket says 172.21.15.77:59362) 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.219+0000 7fc6ed0f9640 1 --2- 172.21.15.77:0/3025095884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 0x7fc6e80b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.219+0000 7fc6ed8fa640 1 -- 172.21.15.77:0/3025095884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 msgr2=0x7fc6e80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.219+0000 7fc6ed8fa640 1 --2- 172.21.15.77:0/3025095884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 0x7fc6e80b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.219+0000 7fc6ed8fa640 1 -- 172.21.15.77:0/3025095884 >> v1:172.21.15.77:6789/0 conn(0x7fc6e8161cf0 legacy=0x7fc6e80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.219+0000 7fc6ed8fa640 1 -- 172.21.15.77:0/3025095884 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e80d25a0 con 0x7fc6e80b4460 2024-03-06T07:25:03.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.219+0000 7fc6ec8f8640 1 -- 172.21.15.77:0/3025095884 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473615727 0 0) 0x7fc6e80d22c0 con 0x7fc6e8161cf0 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6ed8fa640 1 --2- 172.21.15.77:0/3025095884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e80b4460 0x7fc6e80d4800 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009870 tx=0x7fc6e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda927f1025a48c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6ec8f8640 1 -- 172.21.15.77:0/3025095884 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0034070 con 0x7fc6e80b4460 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6ec8f8640 1 -- 172.21.15.77:0/3025095884 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0030840 con 0x7fc6e80b4460 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6ec8f8640 1 -- 172.21.15.77:0/3025095884 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0030c10 con 0x7fc6e80b4460 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6efb5e640 1 -- 172.21.15.77:0/3025095884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e80b4460 msgr2=0x7fc6e80d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/3025095884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e80b4460 0x7fc6e80d4800 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009870 tx=0x7fc6e002b980 comp rx=0 tx=0).stop 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6efb5e640 1 -- 172.21.15.77:0/3025095884 shutdown_connections 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/3025095884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 0x7fc6e80b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/3025095884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e80b4460 0x7fc6e80d4800 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6efb5e640 1 -- 172.21.15.77:0/3025095884 >> 172.21.15.77:0/3025095884 conn(0x7fc6e80d3ce0 msgr2=0x7fc6e80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6efb5e640 1 -- 172.21.15.77:0/3025095884 shutdown_connections 2024-03-06T07:25:03.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.220+0000 7fc6efb5e640 1 -- 172.21.15.77:0/3025095884 wait complete. 2024-03-06T07:25:03.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.221+0000 7fc6efb5e640 1 Processor -- start 2024-03-06T07:25:03.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.221+0000 7fc6efb5e640 1 -- start start 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6efb5e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6e80b4460 0x7fc6e81562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6efb5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e8161cf0 0x7fc6e815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6efb5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 0x7fc6e815f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6efb5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e80ce950 con 0x7fc6e8161cf0 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6efb5e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e80d2b60 con 0x7fc6e80b4460 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6efb5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e80b3ac0 con 0x7fc6e8162970 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6ed0f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e8161cf0 0x7fc6e815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6ed0f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e8161cf0 0x7fc6e815ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44594/0 (socket says 172.21.15.77:44594) 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6ed0f9640 1 -- 172.21.15.77:0/1121446787 learned_addr learned my addr 172.21.15.77:0/1121446787 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:03.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.222+0000 7fc6ee0fb640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 0x7fc6e815f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6ed0f9640 1 -- 172.21.15.77:0/1121446787 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 msgr2=0x7fc6e815f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6ed0f9640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 0x7fc6e815f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6ed8fa640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6e80b4460 0x7fc6e81562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6ed0f9640 1 -- 172.21.15.77:0/1121446787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6e80b4460 msgr2=0x7fc6e81562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6ed0f9640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6e80b4460 0x7fc6e81562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6ed0f9640 1 -- 172.21.15.77:0/1121446787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e80c9620 con 0x7fc6e8161cf0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6ed0f9640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e8161cf0 0x7fc6e815ecd0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc007ec0 tx=0x7fc6dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6e67fc640 1 -- 172.21.15.77:0/1121446787 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc014620 con 0x7fc6e8161cf0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6e67fc640 1 -- 172.21.15.77:0/1121446787 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc019070 con 0x7fc6e8161cf0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e815bff0 con 0x7fc6e8161cf0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6e67fc640 1 -- 172.21.15.77:0/1121446787 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc014a00 con 0x7fc6e8161cf0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.223+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e815ba30 con 0x7fc6e8161cf0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.224+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e806a090 con 0x7fc6e8161cf0 2024-03-06T07:25:03.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.224+0000 7fc6e67fc640 1 -- 172.21.15.77:0/1121446787 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6dc014ba0 con 0x7fc6e8161cf0 2024-03-06T07:25:03.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.225+0000 7fc6e67fc640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc6d406b4d0 0x7fc6d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.225+0000 7fc6e67fc640 1 -- 172.21.15.77:0/1121446787 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6dc08aa30 con 0x7fc6e8161cf0 2024-03-06T07:25:03.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.225+0000 7fc6ed8fa640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc6d406b4d0 0x7fc6d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.225+0000 7fc6ed8fa640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc6d406b4d0 0x7fc6d406d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0036000 tx=0x7fc6e003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.229+0000 7fc6e67fc640 1 -- 172.21.15.77:0/1121446787 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6e806a090 con 0x7fc6e8161cf0 2024-03-06T07:25:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.477+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc6e804f330 con 0x7fc6e8161cf0 2024-03-06T07:25:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.477+0000 7fc6e67fc640 1 -- 172.21.15.77:0/1121446787 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1528 (secure 0 0 0) 0x7fc6e80c9620 con 0x7fc6e8161cf0 2024-03-06T07:25:03.479 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:03.479 INFO:teuthology.orchestra.run.smithi077.stdout:{"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":202,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":69,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709711,"num_in_osds":8,"osd_in_since":1709709703,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":275759104,"bytes_avail":772818354176,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":33,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"c","status":"up:active","gid":6507}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-06T07:24:47.414902+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc6d406b4d0 msgr2=0x7fc6d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc6d406b4d0 0x7fc6d406d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0036000 tx=0x7fc6e003b040 comp rx=0 tx=0).stop 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e8161cf0 msgr2=0x7fc6e815ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e8161cf0 0x7fc6e815ecd0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc007ec0 tx=0x7fc6dc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 shutdown_connections 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc6d406b4d0 0x7fc6d406d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6e8161cf0 0x7fc6e815ecd0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6e8162970 0x7fc6e815f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 --2- 172.21.15.77:0/1121446787 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6e80b4460 0x7fc6e81562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 >> 172.21.15.77:0/1121446787 conn(0x7fc6e80d3ce0 msgr2=0x7fc6e80d08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 shutdown_connections 2024-03-06T07:25:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.479+0000 7fc6efb5e640 1 -- 172.21.15.77:0/1121446787 wait complete. 2024-03-06T07:25:03.492 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:25:03.492 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.590+0000 7f927f26d640 1 Processor -- start 2024-03-06T07:25:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.590+0000 7f927f26d640 1 -- start start 2024-03-06T07:25:03.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927f26d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 0x7f92780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927f26d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92780cf830 0x7f92780d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927f26d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92780b3430 con 0x7f92780d27f0 2024-03-06T07:25:03.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927f26d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92780d2c50 con 0x7f92780cf830 2024-03-06T07:25:03.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927f26d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92780b31e0 con 0x7f927815dfe0 2024-03-06T07:25:03.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927c808640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92780cf830 0x7f92780d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927c808640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92780cf830 0x7f92780d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44602/0 (socket says 172.21.15.77:44602) 2024-03-06T07:25:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927c808640 1 -- 172.21.15.77:0/1775283925 learned_addr learned my addr 172.21.15.77:0/1775283925 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927d009640 1 --2- 172.21.15.77:0/1775283925 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 0x7f92780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927c808640 1 -- 172.21.15.77:0/1775283925 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 msgr2=0x7f92780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927c808640 1 --2- 172.21.15.77:0/1775283925 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 0x7f92780cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927c808640 1 -- 172.21.15.77:0/1775283925 >> v1:172.21.15.158:6789/0 conn(0x7f92780d27f0 legacy=0x7f92780d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.592+0000 7f927c808640 1 -- 172.21.15.77:0/1775283925 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92780b3710 con 0x7f92780cf830 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.593+0000 7f927c808640 1 --2- 172.21.15.77:0/1775283925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780cf830 0x7f92780d22b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9268009870 tx=0x7f926802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901a508f1ed9a7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.593+0000 7f926ffff640 1 -- 172.21.15.77:0/1775283925 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9268035020 con 0x7f92780cf830 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.593+0000 7f926ffff640 1 -- 172.21.15.77:0/1775283925 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9268031640 con 0x7f92780cf830 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.593+0000 7f926ffff640 1 -- 172.21.15.77:0/1775283925 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926802d040 con 0x7f92780cf830 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.594+0000 7f927f26d640 1 -- 172.21.15.77:0/1775283925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780cf830 msgr2=0x7f92780d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.594+0000 7f927f26d640 1 --2- 172.21.15.77:0/1775283925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780cf830 0x7f92780d22b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9268009870 tx=0x7f926802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.594+0000 7f927f26d640 1 -- 172.21.15.77:0/1775283925 shutdown_connections 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.594+0000 7f927f26d640 1 --2- 172.21.15.77:0/1775283925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780cf830 0x7f92780d22b0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.594+0000 7f927f26d640 1 --2- 172.21.15.77:0/1775283925 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 0x7f92780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.594+0000 7f927f26d640 1 -- 172.21.15.77:0/1775283925 >> 172.21.15.77:0/1775283925 conn(0x7f92780d3cd0 msgr2=0x7f92780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.594+0000 7f927f26d640 1 -- 172.21.15.77:0/1775283925 shutdown_connections 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.594+0000 7f927f26d640 1 -- 172.21.15.77:0/1775283925 wait complete. 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.595+0000 7f927f26d640 1 Processor -- start 2024-03-06T07:25:03.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.595+0000 7f927f26d640 1 -- start start 2024-03-06T07:25:03.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927f26d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92780cf830 0x7f9278155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927f26d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780d27f0 0x7f92780cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927f26d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 0x7f927815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927f26d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92780d1c70 con 0x7f92780d27f0 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927f26d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92780b3d60 con 0x7f92780cf830 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927f26d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92780d1df0 con 0x7f927815dfe0 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780d27f0 0x7f92780cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780d27f0 0x7f92780cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44606/0 (socket says 172.21.15.77:44606) 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 -- 172.21.15.77:0/1344491329 learned_addr learned my addr 172.21.15.77:0/1344491329 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927d80a640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 0x7f927815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927d009640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92780cf830 0x7f9278155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 -- 172.21.15.77:0/1344491329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 msgr2=0x7f927815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 0x7f927815b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 -- 172.21.15.77:0/1344491329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92780cf830 msgr2=0x7f9278155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92780cf830 0x7f9278155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 -- 172.21.15.77:0/1344491329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92780cc890 con 0x7f92780d27f0 2024-03-06T07:25:03.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927c808640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780d27f0 0x7f92780cbe40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f926802b930 tx=0x7f92680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f926dffb640 1 -- 172.21.15.77:0/1344491329 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9268040070 con 0x7f92780d27f0 2024-03-06T07:25:03.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f926dffb640 1 -- 172.21.15.77:0/1344491329 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9268039bf0 con 0x7f92780d27f0 2024-03-06T07:25:03.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927815e680 con 0x7f92780d27f0 2024-03-06T07:25:03.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.596+0000 7f926dffb640 1 -- 172.21.15.77:0/1344491329 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9268035040 con 0x7f92780d27f0 2024-03-06T07:25:03.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.597+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92781557c0 con 0x7f92780d27f0 2024-03-06T07:25:03.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.598+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927804f330 con 0x7f92780d27f0 2024-03-06T07:25:03.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.598+0000 7f926dffb640 1 -- 172.21.15.77:0/1344491329 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f926804b060 con 0x7f92780d27f0 2024-03-06T07:25:03.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.599+0000 7f926dffb640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f925c06b4d0 0x7f925c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.599+0000 7f927d009640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f925c06b4d0 0x7f925c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.599+0000 7f926dffb640 1 -- 172.21.15.77:0/1344491329 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92680abd80 con 0x7f92780d27f0 2024-03-06T07:25:03.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.599+0000 7f927d009640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f925c06b4d0 0x7f925c06d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9274006fd0 tx=0x7f9274008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.603+0000 7f926dffb640 1 -- 172.21.15.77:0/1344491329 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f926802f300 con 0x7f92780d27f0 2024-03-06T07:25:03.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.777+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92780ceb70 con 0x7f925c06b4d0 2024-03-06T07:25:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.783+0000 7f926dffb640 1 -- 172.21.15.77:0/1344491329 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252067 (secure 0 0 0) 0x7f92780ceb70 con 0x7f925c06b4d0 2024-03-06T07:25:03.787 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:25:03.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f925c06b4d0 msgr2=0x7f925c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f925c06b4d0 0x7f925c06d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9274006fd0 tx=0x7f9274008040 comp rx=0 tx=0).stop 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780d27f0 msgr2=0x7f92780cbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780d27f0 0x7f92780cbe40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f926802b930 tx=0x7f92680046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 shutdown_connections 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f925c06b4d0 0x7f925c06d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92780d27f0 0x7f92780cbe40 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f927815dfe0 0x7f927815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 --2- 172.21.15.77:0/1344491329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92780cf830 0x7f9278155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 >> 172.21.15.77:0/1344491329 conn(0x7f92780d3cd0 msgr2=0x7f92780b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 shutdown_connections 2024-03-06T07:25:03.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:03.786+0000 7f927f26d640 1 -- 172.21.15.77:0/1344491329 wait complete. 2024-03-06T07:25:03.798 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-03-06T07:25:03.417044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":273876,"kb_used_data":7784,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754700844,"statfs":{"total":773094113280,"available":772813664256,"internally_reserved":0,"allocated":7970816,"data_stored":4003860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000734"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695917+0000","last_change":"2024-03-06T07:24:59.302574+0000","last_active":"2024-03-06T07:25:00.695917+0000","last_peered":"2024-03-06T07:25:00.695917+0000","last_clean":"2024-03-06T07:25:00.695917+0000","last_became_active":"2024-03-06T07:24:59.302188+0000","last_became_peered":"2024-03-06T07:24:59.302188+0000","last_unstale":"2024-03-06T07:25:00.695917+0000","last_undegraded":"2024-03-06T07:25:00.695917+0000","last_fullsized":"2024-03-06T07:25:00.695917+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:31.250047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280766+0000","last_change":"2024-03-06T07:24:59.293289+0000","last_active":"2024-03-06T07:25:00.280766+0000","last_peered":"2024-03-06T07:25:00.280766+0000","last_clean":"2024-03-06T07:25:00.280766+0000","last_became_active":"2024-03-06T07:24:59.292991+0000","last_became_peered":"2024-03-06T07:24:59.292991+0000","last_unstale":"2024-03-06T07:25:00.280766+0000","last_undegraded":"2024-03-06T07:25:00.280766+0000","last_fullsized":"2024-03-06T07:25:00.280766+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:04.812363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496552+0000","last_change":"2024-03-06T07:24:59.296498+0000","last_active":"2024-03-06T07:25:00.496552+0000","last_peered":"2024-03-06T07:25:00.496552+0000","last_clean":"2024-03-06T07:25:00.496552+0000","last_became_active":"2024-03-06T07:24:59.296227+0000","last_became_peered":"2024-03-06T07:24:59.296227+0000","last_unstale":"2024-03-06T07:25:00.496552+0000","last_undegraded":"2024-03-06T07:25:00.496552+0000","last_fullsized":"2024-03-06T07:25:00.496552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:36.220103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689993+0000","last_change":"2024-03-06T07:24:59.306562+0000","last_active":"2024-03-06T07:25:01.689993+0000","last_peered":"2024-03-06T07:25:01.689993+0000","last_clean":"2024-03-06T07:25:01.689993+0000","last_became_active":"2024-03-06T07:24:59.306269+0000","last_became_peered":"2024-03-06T07:24:59.306269+0000","last_unstale":"2024-03-06T07:25:01.689993+0000","last_undegraded":"2024-03-06T07:25:01.689993+0000","last_fullsized":"2024-03-06T07:25:01.689993+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:15.692819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278668+0000","last_change":"2024-03-06T07:24:59.292427+0000","last_active":"2024-03-06T07:25:00.278668+0000","last_peered":"2024-03-06T07:25:00.278668+0000","last_clean":"2024-03-06T07:25:00.278668+0000","last_became_active":"2024-03-06T07:24:59.291543+0000","last_became_peered":"2024-03-06T07:24:59.291543+0000","last_unstale":"2024-03-06T07:25:00.278668+0000","last_undegraded":"2024-03-06T07:25:00.278668+0000","last_fullsized":"2024-03-06T07:25:00.278668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:12.915316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280763+0000","last_change":"2024-03-06T07:24:59.296561+0000","last_active":"2024-03-06T07:25:00.280763+0000","last_peered":"2024-03-06T07:25:00.280763+0000","last_clean":"2024-03-06T07:25:00.280763+0000","last_became_active":"2024-03-06T07:24:59.296297+0000","last_became_peered":"2024-03-06T07:24:59.296297+0000","last_unstale":"2024-03-06T07:25:00.280763+0000","last_undegraded":"2024-03-06T07:25:00.280763+0000","last_fullsized":"2024-03-06T07:25:00.280763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:31.496570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189514+0000","last_change":"2024-03-06T07:24:59.293417+0000","last_active":"2024-03-06T07:25:01.189514+0000","last_peered":"2024-03-06T07:25:01.189514+0000","last_clean":"2024-03-06T07:25:01.189514+0000","last_became_active":"2024-03-06T07:24:59.292763+0000","last_became_peered":"2024-03-06T07:24:59.292763+0000","last_unstale":"2024-03-06T07:25:01.189514+0000","last_undegraded":"2024-03-06T07:25:01.189514+0000","last_fullsized":"2024-03-06T07:25:01.189514+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:00.346652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.281003+0000","last_change":"2024-03-06T07:24:59.300531+0000","last_active":"2024-03-06T07:25:00.281003+0000","last_peered":"2024-03-06T07:25:00.281003+0000","last_clean":"2024-03-06T07:25:00.281003+0000","last_became_active":"2024-03-06T07:24:59.299791+0000","last_became_peered":"2024-03-06T07:24:59.299791+0000","last_unstale":"2024-03-06T07:25:00.281003+0000","last_undegraded":"2024-03-06T07:25:00.281003+0000","last_fullsized":"2024-03-06T07:25:00.281003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:50.718497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278860+0000","last_change":"2024-03-06T07:24:59.287790+0000","last_active":"2024-03-06T07:25:00.278860+0000","last_peered":"2024-03-06T07:25:00.278860+0000","last_clean":"2024-03-06T07:25:00.278860+0000","last_became_active":"2024-03-06T07:24:59.287515+0000","last_became_peered":"2024-03-06T07:24:59.287515+0000","last_unstale":"2024-03-06T07:25:00.278860+0000","last_undegraded":"2024-03-06T07:25:00.278860+0000","last_fullsized":"2024-03-06T07:25:00.278860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:42.668609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280350+0000","last_change":"2024-03-06T07:24:59.292646+0000","last_active":"2024-03-06T07:25:00.280350+0000","last_peered":"2024-03-06T07:25:00.280350+0000","last_clean":"2024-03-06T07:25:00.280350+0000","last_became_active":"2024-03-06T07:24:59.292366+0000","last_became_peered":"2024-03-06T07:24:59.292366+0000","last_unstale":"2024-03-06T07:25:00.280350+0000","last_undegraded":"2024-03-06T07:25:00.280350+0000","last_fullsized":"2024-03-06T07:25:00.280350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:18.709021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695749+0000","last_change":"2024-03-06T07:24:59.295856+0000","last_active":"2024-03-06T07:25:00.695749+0000","last_peered":"2024-03-06T07:25:00.695749+0000","last_clean":"2024-03-06T07:25:00.695749+0000","last_became_active":"2024-03-06T07:24:59.295587+0000","last_became_peered":"2024-03-06T07:24:59.295587+0000","last_unstale":"2024-03-06T07:25:00.695749+0000","last_undegraded":"2024-03-06T07:25:00.695749+0000","last_fullsized":"2024-03-06T07:25:00.695749+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:16.578112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.690073+0000","last_change":"2024-03-06T07:24:59.299041+0000","last_active":"2024-03-06T07:25:01.690073+0000","last_peered":"2024-03-06T07:25:01.690073+0000","last_clean":"2024-03-06T07:25:01.690073+0000","last_became_active":"2024-03-06T07:24:59.298798+0000","last_became_peered":"2024-03-06T07:24:59.298798+0000","last_unstale":"2024-03-06T07:25:01.690073+0000","last_undegraded":"2024-03-06T07:25:01.690073+0000","last_fullsized":"2024-03-06T07:25:01.690073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:34.677390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284433+0000","last_change":"2024-03-06T07:24:59.297747+0000","last_active":"2024-03-06T07:25:01.284433+0000","last_peered":"2024-03-06T07:25:01.284433+0000","last_clean":"2024-03-06T07:25:01.284433+0000","last_became_active":"2024-03-06T07:24:59.297402+0000","last_became_peered":"2024-03-06T07:24:59.297402+0000","last_unstale":"2024-03-06T07:25:01.284433+0000","last_undegraded":"2024-03-06T07:25:01.284433+0000","last_fullsized":"2024-03-06T07:25:01.284433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:39.856757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278978+0000","last_change":"2024-03-06T07:24:59.290253+0000","last_active":"2024-03-06T07:25:00.278978+0000","last_peered":"2024-03-06T07:25:00.278978+0000","last_clean":"2024-03-06T07:25:00.278978+0000","last_became_active":"2024-03-06T07:24:59.289967+0000","last_became_peered":"2024-03-06T07:24:59.289967+0000","last_unstale":"2024-03-06T07:25:00.278978+0000","last_undegraded":"2024-03-06T07:25:00.278978+0000","last_fullsized":"2024-03-06T07:25:00.278978+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:16.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284353+0000","last_change":"2024-03-06T07:24:59.296686+0000","last_active":"2024-03-06T07:25:01.284353+0000","last_peered":"2024-03-06T07:25:01.284353+0000","last_clean":"2024-03-06T07:25:01.284353+0000","last_became_active":"2024-03-06T07:24:59.296329+0000","last_became_peered":"2024-03-06T07:24:59.296329+0000","last_unstale":"2024-03-06T07:25:01.284353+0000","last_undegraded":"2024-03-06T07:25:01.284353+0000","last_fullsized":"2024-03-06T07:25:01.284353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:27.080068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279506+0000","last_change":"2024-03-06T07:24:59.299688+0000","last_active":"2024-03-06T07:25:00.279506+0000","last_peered":"2024-03-06T07:25:00.279506+0000","last_clean":"2024-03-06T07:25:00.279506+0000","last_became_active":"2024-03-06T07:24:59.299410+0000","last_became_peered":"2024-03-06T07:24:59.299410+0000","last_unstale":"2024-03-06T07:25:00.279506+0000","last_undegraded":"2024-03-06T07:25:00.279506+0000","last_fullsized":"2024-03-06T07:25:00.279506+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:04.812363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278876+0000","last_change":"2024-03-06T07:24:59.286850+0000","last_active":"2024-03-06T07:25:00.278876+0000","last_peered":"2024-03-06T07:25:00.278876+0000","last_clean":"2024-03-06T07:25:00.278876+0000","last_became_active":"2024-03-06T07:24:59.286604+0000","last_became_peered":"2024-03-06T07:24:59.286604+0000","last_unstale":"2024-03-06T07:25:00.278876+0000","last_undegraded":"2024-03-06T07:25:00.278876+0000","last_fullsized":"2024-03-06T07:25:00.278876+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:31.100044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.690132+0000","last_change":"2024-03-06T07:24:59.305671+0000","last_active":"2024-03-06T07:25:01.690132+0000","last_peered":"2024-03-06T07:25:01.690132+0000","last_clean":"2024-03-06T07:25:01.690132+0000","last_became_active":"2024-03-06T07:24:59.305411+0000","last_became_peered":"2024-03-06T07:24:59.305411+0000","last_unstale":"2024-03-06T07:25:01.690132+0000","last_undegraded":"2024-03-06T07:25:01.690132+0000","last_fullsized":"2024-03-06T07:25:01.690132+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:46.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280591+0000","last_change":"2024-03-06T07:24:59.299801+0000","last_active":"2024-03-06T07:25:00.280591+0000","last_peered":"2024-03-06T07:25:00.280591+0000","last_clean":"2024-03-06T07:25:00.280591+0000","last_became_active":"2024-03-06T07:24:59.299499+0000","last_became_peered":"2024-03-06T07:24:59.299499+0000","last_unstale":"2024-03-06T07:25:00.280591+0000","last_undegraded":"2024-03-06T07:25:00.280591+0000","last_fullsized":"2024-03-06T07:25:00.280591+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:45.209746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284588+0000","last_change":"2024-03-06T07:24:59.291143+0000","last_active":"2024-03-06T07:25:01.284588+0000","last_peered":"2024-03-06T07:25:01.284588+0000","last_clean":"2024-03-06T07:25:01.284588+0000","last_became_active":"2024-03-06T07:24:59.290881+0000","last_became_peered":"2024-03-06T07:24:59.290881+0000","last_unstale":"2024-03-06T07:25:01.284588+0000","last_undegraded":"2024-03-06T07:25:01.284588+0000","last_fullsized":"2024-03-06T07:25:01.284588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:45.289459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279945+0000","last_change":"2024-03-06T07:24:59.296968+0000","last_active":"2024-03-06T07:25:00.279945+0000","last_peered":"2024-03-06T07:25:00.279945+0000","last_clean":"2024-03-06T07:25:00.279945+0000","last_became_active":"2024-03-06T07:24:59.296711+0000","last_became_peered":"2024-03-06T07:24:59.296711+0000","last_unstale":"2024-03-06T07:25:00.279945+0000","last_undegraded":"2024-03-06T07:25:00.279945+0000","last_fullsized":"2024-03-06T07:25:00.279945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:31.295986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279914+0000","last_change":"2024-03-06T07:24:59.286100+0000","last_active":"2024-03-06T07:25:00.279914+0000","last_peered":"2024-03-06T07:25:00.279914+0000","last_clean":"2024-03-06T07:25:00.279914+0000","last_became_active":"2024-03-06T07:24:59.285771+0000","last_became_peered":"2024-03-06T07:24:59.285771+0000","last_unstale":"2024-03-06T07:25:00.279914+0000","last_undegraded":"2024-03-06T07:25:00.279914+0000","last_fullsized":"2024-03-06T07:25:00.279914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:45.878592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695438+0000","last_change":"2024-03-06T07:24:59.294002+0000","last_active":"2024-03-06T07:25:00.695438+0000","last_peered":"2024-03-06T07:25:00.695438+0000","last_clean":"2024-03-06T07:25:00.695438+0000","last_became_active":"2024-03-06T07:24:59.293512+0000","last_became_peered":"2024-03-06T07:24:59.293512+0000","last_unstale":"2024-03-06T07:25:00.695438+0000","last_undegraded":"2024-03-06T07:25:00.695438+0000","last_fullsized":"2024-03-06T07:25:00.695438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:32.810212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695389+0000","last_change":"2024-03-06T07:24:59.301374+0000","last_active":"2024-03-06T07:25:00.695389+0000","last_peered":"2024-03-06T07:25:00.695389+0000","last_clean":"2024-03-06T07:25:00.695389+0000","last_became_active":"2024-03-06T07:24:59.300966+0000","last_became_peered":"2024-03-06T07:24:59.300966+0000","last_unstale":"2024-03-06T07:25:00.695389+0000","last_undegraded":"2024-03-06T07:25:00.695389+0000","last_fullsized":"2024-03-06T07:25:00.695389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:43.197457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279699+0000","last_change":"2024-03-06T07:24:59.295435+0000","last_active":"2024-03-06T07:25:00.279699+0000","last_peered":"2024-03-06T07:25:00.279699+0000","last_clean":"2024-03-06T07:25:00.279699+0000","last_became_active":"2024-03-06T07:24:59.295170+0000","last_became_peered":"2024-03-06T07:24:59.295170+0000","last_unstale":"2024-03-06T07:25:00.279699+0000","last_undegraded":"2024-03-06T07:25:00.279699+0000","last_fullsized":"2024-03-06T07:25:00.279699+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:50.718497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189750+0000","last_change":"2024-03-06T07:24:59.302154+0000","last_active":"2024-03-06T07:25:01.189750+0000","last_peered":"2024-03-06T07:25:01.189750+0000","last_clean":"2024-03-06T07:25:01.189750+0000","last_became_active":"2024-03-06T07:24:59.301886+0000","last_became_peered":"2024-03-06T07:24:59.301886+0000","last_unstale":"2024-03-06T07:25:01.189750+0000","last_undegraded":"2024-03-06T07:25:01.189750+0000","last_fullsized":"2024-03-06T07:25:01.189750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:42.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496555+0000","last_change":"2024-03-06T07:24:59.288408+0000","last_active":"2024-03-06T07:25:00.496555+0000","last_peered":"2024-03-06T07:25:00.496555+0000","last_clean":"2024-03-06T07:25:00.496555+0000","last_became_active":"2024-03-06T07:24:59.285862+0000","last_became_peered":"2024-03-06T07:24:59.285862+0000","last_unstale":"2024-03-06T07:25:00.496555+0000","last_undegraded":"2024-03-06T07:25:00.496555+0000","last_fullsized":"2024-03-06T07:25:00.496555+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:05.383872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280759+0000","last_change":"2024-03-06T07:24:59.291942+0000","last_active":"2024-03-06T07:25:00.280759+0000","last_peered":"2024-03-06T07:25:00.280759+0000","last_clean":"2024-03-06T07:25:00.280759+0000","last_became_active":"2024-03-06T07:24:59.290198+0000","last_became_peered":"2024-03-06T07:24:59.290198+0000","last_unstale":"2024-03-06T07:25:00.280759+0000","last_undegraded":"2024-03-06T07:25:00.280759+0000","last_fullsized":"2024-03-06T07:25:00.280759+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:53.460638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496443+0000","last_change":"2024-03-06T07:24:59.290512+0000","last_active":"2024-03-06T07:25:00.496443+0000","last_peered":"2024-03-06T07:25:00.496443+0000","last_clean":"2024-03-06T07:25:00.496443+0000","last_became_active":"2024-03-06T07:24:59.290282+0000","last_became_peered":"2024-03-06T07:24:59.290282+0000","last_unstale":"2024-03-06T07:25:00.496443+0000","last_undegraded":"2024-03-06T07:25:00.496443+0000","last_fullsized":"2024-03-06T07:25:00.496443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:57.853489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189530+0000","last_change":"2024-03-06T07:24:59.294729+0000","last_active":"2024-03-06T07:25:01.189530+0000","last_peered":"2024-03-06T07:25:01.189530+0000","last_clean":"2024-03-06T07:25:01.189530+0000","last_became_active":"2024-03-06T07:24:59.294468+0000","last_became_peered":"2024-03-06T07:24:59.294468+0000","last_unstale":"2024-03-06T07:25:01.189530+0000","last_undegraded":"2024-03-06T07:25:01.189530+0000","last_fullsized":"2024-03-06T07:25:01.189530+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:42.167568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189748+0000","last_change":"2024-03-06T07:24:59.293641+0000","last_active":"2024-03-06T07:25:01.189748+0000","last_peered":"2024-03-06T07:25:01.189748+0000","last_clean":"2024-03-06T07:25:01.189748+0000","last_became_active":"2024-03-06T07:24:59.293154+0000","last_became_peered":"2024-03-06T07:24:59.293154+0000","last_unstale":"2024-03-06T07:25:01.189748+0000","last_undegraded":"2024-03-06T07:25:01.189748+0000","last_fullsized":"2024-03-06T07:25:01.189748+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:27.080068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.281046+0000","last_change":"2024-03-06T07:24:59.298323+0000","last_active":"2024-03-06T07:25:00.281046+0000","last_peered":"2024-03-06T07:25:00.281046+0000","last_clean":"2024-03-06T07:25:00.281046+0000","last_became_active":"2024-03-06T07:24:59.298057+0000","last_became_peered":"2024-03-06T07:24:59.298057+0000","last_unstale":"2024-03-06T07:25:00.281046+0000","last_undegraded":"2024-03-06T07:25:00.281046+0000","last_fullsized":"2024-03-06T07:25:00.281046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:45.878592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695849+0000","last_change":"2024-03-06T07:24:59.303884+0000","last_active":"2024-03-06T07:25:00.695849+0000","last_peered":"2024-03-06T07:25:00.695849+0000","last_clean":"2024-03-06T07:25:00.695849+0000","last_became_active":"2024-03-06T07:24:59.303210+0000","last_became_peered":"2024-03-06T07:24:59.303210+0000","last_unstale":"2024-03-06T07:25:00.695849+0000","last_undegraded":"2024-03-06T07:25:00.695849+0000","last_fullsized":"2024-03-06T07:25:00.695849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:49.805086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.696032+0000","last_change":"2024-03-06T07:24:58.292643+0000","last_active":"2024-03-06T07:25:00.696032+0000","last_peered":"2024-03-06T07:25:00.696032+0000","last_clean":"2024-03-06T07:25:00.696032+0000","last_became_active":"2024-03-06T07:24:58.292289+0000","last_became_peered":"2024-03-06T07:24:58.292289+0000","last_unstale":"2024-03-06T07:25:00.696032+0000","last_undegraded":"2024-03-06T07:25:00.696032+0000","last_fullsized":"2024-03-06T07:25:00.696032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:10.684873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278962+0000","last_change":"2024-03-06T07:24:59.285713+0000","last_active":"2024-03-06T07:25:00.278962+0000","last_peered":"2024-03-06T07:25:00.278962+0000","last_clean":"2024-03-06T07:25:00.278962+0000","last_became_active":"2024-03-06T07:24:59.285460+0000","last_became_peered":"2024-03-06T07:24:59.285460+0000","last_unstale":"2024-03-06T07:25:00.278962+0000","last_undegraded":"2024-03-06T07:25:00.278962+0000","last_fullsized":"2024-03-06T07:25:00.278962+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:23.755605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284609+0000","last_change":"2024-03-06T07:24:58.295724+0000","last_active":"2024-03-06T07:25:01.284609+0000","last_peered":"2024-03-06T07:25:01.284609+0000","last_clean":"2024-03-06T07:25:01.284609+0000","last_became_active":"2024-03-06T07:24:58.295420+0000","last_became_peered":"2024-03-06T07:24:58.295420+0000","last_unstale":"2024-03-06T07:25:01.284609+0000","last_undegraded":"2024-03-06T07:25:01.284609+0000","last_fullsized":"2024-03-06T07:25:01.284609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:30.699089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280112+0000","last_change":"2024-03-06T07:24:59.293098+0000","last_active":"2024-03-06T07:25:00.280112+0000","last_peered":"2024-03-06T07:25:00.280112+0000","last_clean":"2024-03-06T07:25:00.280112+0000","last_became_active":"2024-03-06T07:24:59.290216+0000","last_became_peered":"2024-03-06T07:24:59.290216+0000","last_unstale":"2024-03-06T07:25:00.280112+0000","last_undegraded":"2024-03-06T07:25:00.280112+0000","last_fullsized":"2024-03-06T07:25:00.280112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:31.650270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279659+0000","last_change":"2024-03-06T07:24:58.295696+0000","last_active":"2024-03-06T07:25:00.279659+0000","last_peered":"2024-03-06T07:25:00.279659+0000","last_clean":"2024-03-06T07:25:00.279659+0000","last_became_active":"2024-03-06T07:24:58.295083+0000","last_became_peered":"2024-03-06T07:24:58.295083+0000","last_unstale":"2024-03-06T07:25:00.279659+0000","last_undegraded":"2024-03-06T07:25:00.279659+0000","last_fullsized":"2024-03-06T07:25:00.279659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:33.675559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695844+0000","last_change":"2024-03-06T07:24:59.297688+0000","last_active":"2024-03-06T07:25:00.695844+0000","last_peered":"2024-03-06T07:25:00.695844+0000","last_clean":"2024-03-06T07:25:00.695844+0000","last_became_active":"2024-03-06T07:24:59.297313+0000","last_became_peered":"2024-03-06T07:24:59.297313+0000","last_unstale":"2024-03-06T07:25:00.695844+0000","last_undegraded":"2024-03-06T07:25:00.695844+0000","last_fullsized":"2024-03-06T07:25:00.695844+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:27.478963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280172+0000","last_change":"2024-03-06T07:24:58.290508+0000","last_active":"2024-03-06T07:25:00.280172+0000","last_peered":"2024-03-06T07:25:00.280172+0000","last_clean":"2024-03-06T07:25:00.280172+0000","last_became_active":"2024-03-06T07:24:58.286582+0000","last_became_peered":"2024-03-06T07:24:58.286582+0000","last_unstale":"2024-03-06T07:25:00.280172+0000","last_undegraded":"2024-03-06T07:25:00.280172+0000","last_fullsized":"2024-03-06T07:25:00.280172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:15.493830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189730+0000","last_change":"2024-03-06T07:24:59.292011+0000","last_active":"2024-03-06T07:25:01.189730+0000","last_peered":"2024-03-06T07:25:01.189730+0000","last_clean":"2024-03-06T07:25:01.189730+0000","last_became_active":"2024-03-06T07:24:59.291667+0000","last_became_peered":"2024-03-06T07:24:59.291667+0000","last_unstale":"2024-03-06T07:25:01.189730+0000","last_undegraded":"2024-03-06T07:25:01.189730+0000","last_fullsized":"2024-03-06T07:25:01.189730+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:36.866907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496493+0000","last_change":"2024-03-06T07:24:58.286577+0000","last_active":"2024-03-06T07:25:00.496493+0000","last_peered":"2024-03-06T07:25:00.496493+0000","last_clean":"2024-03-06T07:25:00.496493+0000","last_became_active":"2024-03-06T07:24:58.286259+0000","last_became_peered":"2024-03-06T07:24:58.286259+0000","last_unstale":"2024-03-06T07:25:00.496493+0000","last_undegraded":"2024-03-06T07:25:00.496493+0000","last_fullsized":"2024-03-06T07:25:00.496493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:15.576282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279957+0000","last_change":"2024-03-06T07:24:59.286948+0000","last_active":"2024-03-06T07:25:00.279957+0000","last_peered":"2024-03-06T07:25:00.279957+0000","last_clean":"2024-03-06T07:25:00.279957+0000","last_became_active":"2024-03-06T07:24:59.286660+0000","last_became_peered":"2024-03-06T07:24:59.286660+0000","last_unstale":"2024-03-06T07:25:00.279957+0000","last_undegraded":"2024-03-06T07:25:00.279957+0000","last_fullsized":"2024-03-06T07:25:00.279957+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:35.117491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280842+0000","last_change":"2024-03-06T07:24:58.290810+0000","last_active":"2024-03-06T07:25:00.280842+0000","last_peered":"2024-03-06T07:25:00.280842+0000","last_clean":"2024-03-06T07:25:00.280842+0000","last_became_active":"2024-03-06T07:24:58.290521+0000","last_became_peered":"2024-03-06T07:24:58.290521+0000","last_unstale":"2024-03-06T07:25:00.280842+0000","last_undegraded":"2024-03-06T07:25:00.280842+0000","last_fullsized":"2024-03-06T07:25:00.280842+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:27.748630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278542+0000","last_change":"2024-03-06T07:24:59.282271+0000","last_active":"2024-03-06T07:25:00.278542+0000","last_peered":"2024-03-06T07:25:00.278542+0000","last_clean":"2024-03-06T07:25:00.278542+0000","last_became_active":"2024-03-06T07:24:59.282018+0000","last_became_peered":"2024-03-06T07:24:59.282018+0000","last_unstale":"2024-03-06T07:25:00.278542+0000","last_undegraded":"2024-03-06T07:25:00.278542+0000","last_fullsized":"2024-03-06T07:25:00.278542+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:58.115915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689815+0000","last_change":"2024-03-06T07:24:58.286735+0000","last_active":"2024-03-06T07:25:01.689815+0000","last_peered":"2024-03-06T07:25:01.689815+0000","last_clean":"2024-03-06T07:25:01.689815+0000","last_became_active":"2024-03-06T07:24:58.285523+0000","last_became_peered":"2024-03-06T07:24:58.285523+0000","last_unstale":"2024-03-06T07:25:01.689815+0000","last_undegraded":"2024-03-06T07:25:01.689815+0000","last_fullsized":"2024-03-06T07:25:01.689815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:33.891983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284418+0000","last_change":"2024-03-06T07:24:59.292380+0000","last_active":"2024-03-06T07:25:01.284418+0000","last_peered":"2024-03-06T07:25:01.284418+0000","last_clean":"2024-03-06T07:25:01.284418+0000","last_became_active":"2024-03-06T07:24:59.292123+0000","last_became_peered":"2024-03-06T07:24:59.292123+0000","last_unstale":"2024-03-06T07:25:01.284418+0000","last_undegraded":"2024-03-06T07:25:01.284418+0000","last_fullsized":"2024-03-06T07:25:01.284418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:25.492967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278455+0000","last_change":"2024-03-06T07:24:58.284398+0000","last_active":"2024-03-06T07:25:00.278455+0000","last_peered":"2024-03-06T07:25:00.278455+0000","last_clean":"2024-03-06T07:25:00.278455+0000","last_became_active":"2024-03-06T07:24:58.284101+0000","last_became_peered":"2024-03-06T07:24:58.284101+0000","last_unstale":"2024-03-06T07:25:00.278455+0000","last_undegraded":"2024-03-06T07:25:00.278455+0000","last_fullsized":"2024-03-06T07:25:00.278455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:06.467086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189834+0000","last_change":"2024-03-06T07:24:59.288497+0000","last_active":"2024-03-06T07:25:01.189834+0000","last_peered":"2024-03-06T07:25:01.189834+0000","last_clean":"2024-03-06T07:25:01.189834+0000","last_became_active":"2024-03-06T07:24:59.288161+0000","last_became_peered":"2024-03-06T07:24:59.288161+0000","last_unstale":"2024-03-06T07:25:01.189834+0000","last_undegraded":"2024-03-06T07:25:01.189834+0000","last_fullsized":"2024-03-06T07:25:01.189834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:25.492967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279734+0000","last_change":"2024-03-06T07:24:58.283835+0000","last_active":"2024-03-06T07:25:00.279734+0000","last_peered":"2024-03-06T07:25:00.279734+0000","last_clean":"2024-03-06T07:25:00.279734+0000","last_became_active":"2024-03-06T07:24:58.283535+0000","last_became_peered":"2024-03-06T07:24:58.283535+0000","last_unstale":"2024-03-06T07:25:00.279734+0000","last_undegraded":"2024-03-06T07:25:00.279734+0000","last_fullsized":"2024-03-06T07:25:00.279734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:10.044434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695976+0000","last_change":"2024-03-06T07:24:59.304238+0000","last_active":"2024-03-06T07:25:00.695976+0000","last_peered":"2024-03-06T07:25:00.695976+0000","last_clean":"2024-03-06T07:25:00.695976+0000","last_became_active":"2024-03-06T07:24:59.303930+0000","last_became_peered":"2024-03-06T07:24:59.303930+0000","last_unstale":"2024-03-06T07:25:00.695976+0000","last_undegraded":"2024-03-06T07:25:00.695976+0000","last_fullsized":"2024-03-06T07:25:00.695976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:37.414063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280830+0000","last_change":"2024-03-06T07:24:58.294620+0000","last_active":"2024-03-06T07:25:00.280830+0000","last_peered":"2024-03-06T07:25:00.280830+0000","last_clean":"2024-03-06T07:25:00.280830+0000","last_became_active":"2024-03-06T07:24:58.294312+0000","last_became_peered":"2024-03-06T07:24:58.294312+0000","last_unstale":"2024-03-06T07:25:00.280830+0000","last_undegraded":"2024-03-06T07:25:00.280830+0000","last_fullsized":"2024-03-06T07:25:00.280830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:03.367487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280510+0000","last_change":"2024-03-06T07:24:59.293316+0000","last_active":"2024-03-06T07:25:00.280510+0000","last_peered":"2024-03-06T07:25:00.280510+0000","last_clean":"2024-03-06T07:25:00.280510+0000","last_became_active":"2024-03-06T07:24:59.292835+0000","last_became_peered":"2024-03-06T07:24:59.292835+0000","last_unstale":"2024-03-06T07:25:00.280510+0000","last_undegraded":"2024-03-06T07:25:00.280510+0000","last_fullsized":"2024-03-06T07:25:00.280510+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:50.166642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496613+0000","last_change":"2024-03-06T07:24:58.288380+0000","last_active":"2024-03-06T07:25:00.496613+0000","last_peered":"2024-03-06T07:25:00.496613+0000","last_clean":"2024-03-06T07:25:00.496613+0000","last_became_active":"2024-03-06T07:24:58.287762+0000","last_became_peered":"2024-03-06T07:24:58.287762+0000","last_unstale":"2024-03-06T07:25:00.496613+0000","last_undegraded":"2024-03-06T07:25:00.496613+0000","last_fullsized":"2024-03-06T07:25:00.496613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:39.536983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278709+0000","last_change":"2024-03-06T07:24:59.295751+0000","last_active":"2024-03-06T07:25:00.278709+0000","last_peered":"2024-03-06T07:25:00.278709+0000","last_clean":"2024-03-06T07:25:00.278709+0000","last_became_active":"2024-03-06T07:24:59.295470+0000","last_became_peered":"2024-03-06T07:24:59.295470+0000","last_unstale":"2024-03-06T07:25:00.278709+0000","last_undegraded":"2024-03-06T07:25:00.278709+0000","last_fullsized":"2024-03-06T07:25:00.278709+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:12.651638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.690189+0000","last_change":"2024-03-06T07:24:58.283050+0000","last_active":"2024-03-06T07:25:01.690189+0000","last_peered":"2024-03-06T07:25:01.690189+0000","last_clean":"2024-03-06T07:25:01.690189+0000","last_became_active":"2024-03-06T07:24:58.282778+0000","last_became_peered":"2024-03-06T07:24:58.282778+0000","last_unstale":"2024-03-06T07:25:01.690189+0000","last_undegraded":"2024-03-06T07:25:01.690189+0000","last_fullsized":"2024-03-06T07:25:01.690189+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:15.493830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695599+0000","last_change":"2024-03-06T07:24:59.291367+0000","last_active":"2024-03-06T07:25:00.695599+0000","last_peered":"2024-03-06T07:25:00.695599+0000","last_clean":"2024-03-06T07:25:00.695599+0000","last_became_active":"2024-03-06T07:24:59.289436+0000","last_became_peered":"2024-03-06T07:24:59.289436+0000","last_unstale":"2024-03-06T07:25:00.695599+0000","last_undegraded":"2024-03-06T07:25:00.695599+0000","last_fullsized":"2024-03-06T07:25:00.695599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:40.538813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689861+0000","last_change":"2024-03-06T07:24:58.289949+0000","last_active":"2024-03-06T07:25:01.689861+0000","last_peered":"2024-03-06T07:25:01.689861+0000","last_clean":"2024-03-06T07:25:01.689861+0000","last_became_active":"2024-03-06T07:24:58.289646+0000","last_became_peered":"2024-03-06T07:24:58.289646+0000","last_unstale":"2024-03-06T07:25:01.689861+0000","last_undegraded":"2024-03-06T07:25:01.689861+0000","last_fullsized":"2024-03-06T07:25:01.689861+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:15.610913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189641+0000","last_change":"2024-03-06T07:24:59.291252+0000","last_active":"2024-03-06T07:25:01.189641+0000","last_peered":"2024-03-06T07:25:01.189641+0000","last_clean":"2024-03-06T07:25:01.189641+0000","last_became_active":"2024-03-06T07:24:59.290989+0000","last_became_peered":"2024-03-06T07:24:59.290989+0000","last_unstale":"2024-03-06T07:25:01.189641+0000","last_undegraded":"2024-03-06T07:25:01.189641+0000","last_fullsized":"2024-03-06T07:25:01.189641+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:09.656864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280852+0000","last_change":"2024-03-06T07:24:58.294391+0000","last_active":"2024-03-06T07:25:00.280852+0000","last_peered":"2024-03-06T07:25:00.280852+0000","last_clean":"2024-03-06T07:25:00.280852+0000","last_became_active":"2024-03-06T07:24:58.294053+0000","last_became_peered":"2024-03-06T07:24:58.294053+0000","last_unstale":"2024-03-06T07:25:00.280852+0000","last_undegraded":"2024-03-06T07:25:00.280852+0000","last_fullsized":"2024-03-06T07:25:00.280852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:05.491668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284412+0000","last_change":"2024-03-06T07:24:59.301281+0000","last_active":"2024-03-06T07:25:01.284412+0000","last_peered":"2024-03-06T07:25:01.284412+0000","last_clean":"2024-03-06T07:25:01.284412+0000","last_became_active":"2024-03-06T07:24:59.300861+0000","last_became_peered":"2024-03-06T07:24:59.300861+0000","last_unstale":"2024-03-06T07:25:01.284412+0000","last_undegraded":"2024-03-06T07:25:01.284412+0000","last_fullsized":"2024-03-06T07:25:01.284412+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:42.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189457+0000","last_change":"2024-03-06T07:24:58.289673+0000","last_active":"2024-03-06T07:25:01.189457+0000","last_peered":"2024-03-06T07:25:01.189457+0000","last_clean":"2024-03-06T07:25:01.189457+0000","last_became_active":"2024-03-06T07:24:58.289390+0000","last_became_peered":"2024-03-06T07:24:58.289390+0000","last_unstale":"2024-03-06T07:25:01.189457+0000","last_undegraded":"2024-03-06T07:25:01.189457+0000","last_fullsized":"2024-03-06T07:25:01.189457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:03.810533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278685+0000","last_change":"2024-03-06T07:24:59.291316+0000","last_active":"2024-03-06T07:25:00.278685+0000","last_peered":"2024-03-06T07:25:00.278685+0000","last_clean":"2024-03-06T07:25:00.278685+0000","last_became_active":"2024-03-06T07:24:59.291076+0000","last_became_peered":"2024-03-06T07:24:59.291076+0000","last_unstale":"2024-03-06T07:25:00.278685+0000","last_undegraded":"2024-03-06T07:25:00.278685+0000","last_fullsized":"2024-03-06T07:25:00.278685+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:03.778346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279476+0000","last_change":"2024-03-06T07:24:58.288346+0000","last_active":"2024-03-06T07:25:00.279476+0000","last_peered":"2024-03-06T07:25:00.279476+0000","last_clean":"2024-03-06T07:25:00.279476+0000","last_became_active":"2024-03-06T07:24:58.288014+0000","last_became_peered":"2024-03-06T07:24:58.288014+0000","last_unstale":"2024-03-06T07:25:00.279476+0000","last_undegraded":"2024-03-06T07:25:00.279476+0000","last_fullsized":"2024-03-06T07:25:00.279476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:45.893805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280579+0000","last_change":"2024-03-06T07:24:59.295750+0000","last_active":"2024-03-06T07:25:00.280579+0000","last_peered":"2024-03-06T07:25:00.280579+0000","last_clean":"2024-03-06T07:25:00.280579+0000","last_became_active":"2024-03-06T07:24:59.295222+0000","last_became_peered":"2024-03-06T07:24:59.295222+0000","last_unstale":"2024-03-06T07:25:00.280579+0000","last_undegraded":"2024-03-06T07:25:00.280579+0000","last_fullsized":"2024-03-06T07:25:00.280579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:56.781976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280454+0000","last_change":"2024-03-06T07:24:58.290475+0000","last_active":"2024-03-06T07:25:00.280454+0000","last_peered":"2024-03-06T07:25:00.280454+0000","last_clean":"2024-03-06T07:25:00.280454+0000","last_became_active":"2024-03-06T07:24:58.290156+0000","last_became_peered":"2024-03-06T07:24:58.290156+0000","last_unstale":"2024-03-06T07:25:00.280454+0000","last_undegraded":"2024-03-06T07:25:00.280454+0000","last_fullsized":"2024-03-06T07:25:00.280454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:22.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.696083+0000","last_change":"2024-03-06T07:24:59.297552+0000","last_active":"2024-03-06T07:25:00.696083+0000","last_peered":"2024-03-06T07:25:00.696083+0000","last_clean":"2024-03-06T07:25:00.696083+0000","last_became_active":"2024-03-06T07:24:59.296465+0000","last_became_peered":"2024-03-06T07:24:59.296465+0000","last_unstale":"2024-03-06T07:25:00.696083+0000","last_undegraded":"2024-03-06T07:25:00.696083+0000","last_fullsized":"2024-03-06T07:25:00.696083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:04.276418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280654+0000","last_change":"2024-03-06T07:24:58.295406+0000","last_active":"2024-03-06T07:25:00.280654+0000","last_peered":"2024-03-06T07:25:00.280654+0000","last_clean":"2024-03-06T07:25:00.280654+0000","last_became_active":"2024-03-06T07:24:58.294948+0000","last_became_peered":"2024-03-06T07:24:58.294948+0000","last_unstale":"2024-03-06T07:25:00.280654+0000","last_undegraded":"2024-03-06T07:25:00.280654+0000","last_fullsized":"2024-03-06T07:25:00.280654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:22.520184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.690139+0000","last_change":"2024-03-06T07:24:59.299022+0000","last_active":"2024-03-06T07:25:01.690139+0000","last_peered":"2024-03-06T07:25:01.690139+0000","last_clean":"2024-03-06T07:25:01.690139+0000","last_became_active":"2024-03-06T07:24:59.298613+0000","last_became_peered":"2024-03-06T07:24:59.298613+0000","last_unstale":"2024-03-06T07:25:01.690139+0000","last_undegraded":"2024-03-06T07:25:01.690139+0000","last_fullsized":"2024-03-06T07:25:01.690139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:11.046264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280658+0000","last_change":"2024-03-06T07:24:58.292520+0000","last_active":"2024-03-06T07:25:00.280658+0000","last_peered":"2024-03-06T07:25:00.280658+0000","last_clean":"2024-03-06T07:25:00.280658+0000","last_became_active":"2024-03-06T07:24:58.291165+0000","last_became_peered":"2024-03-06T07:24:58.291165+0000","last_unstale":"2024-03-06T07:25:00.280658+0000","last_undegraded":"2024-03-06T07:25:00.280658+0000","last_fullsized":"2024-03-06T07:25:00.280658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:33.891983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278808+0000","last_change":"2024-03-06T07:24:59.286682+0000","last_active":"2024-03-06T07:25:00.278808+0000","last_peered":"2024-03-06T07:25:00.278808+0000","last_clean":"2024-03-06T07:25:00.278808+0000","last_became_active":"2024-03-06T07:24:59.286403+0000","last_became_peered":"2024-03-06T07:24:59.286403+0000","last_unstale":"2024-03-06T07:25:00.278808+0000","last_undegraded":"2024-03-06T07:25:00.278808+0000","last_fullsized":"2024-03-06T07:25:00.278808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:24.753218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695606+0000","last_change":"2024-03-06T07:24:58.289658+0000","last_active":"2024-03-06T07:25:00.695606+0000","last_peered":"2024-03-06T07:25:00.695606+0000","last_clean":"2024-03-06T07:25:00.695606+0000","last_became_active":"2024-03-06T07:24:58.289381+0000","last_became_peered":"2024-03-06T07:24:58.289381+0000","last_unstale":"2024-03-06T07:25:00.695606+0000","last_undegraded":"2024-03-06T07:25:00.695606+0000","last_fullsized":"2024-03-06T07:25:00.695606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:30.699089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496542+0000","last_change":"2024-03-06T07:24:59.294536+0000","last_active":"2024-03-06T07:25:00.496542+0000","last_peered":"2024-03-06T07:25:00.496542+0000","last_clean":"2024-03-06T07:25:00.496542+0000","last_became_active":"2024-03-06T07:24:59.294221+0000","last_became_peered":"2024-03-06T07:24:59.294221+0000","last_unstale":"2024-03-06T07:25:00.496542+0000","last_undegraded":"2024-03-06T07:25:00.496542+0000","last_fullsized":"2024-03-06T07:25:00.496542+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:54.917133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280957+0000","last_change":"2024-03-06T07:24:58.286439+0000","last_active":"2024-03-06T07:25:00.280957+0000","last_peered":"2024-03-06T07:25:00.280957+0000","last_clean":"2024-03-06T07:25:00.280957+0000","last_became_active":"2024-03-06T07:24:58.285996+0000","last_became_peered":"2024-03-06T07:24:58.285996+0000","last_unstale":"2024-03-06T07:25:00.280957+0000","last_undegraded":"2024-03-06T07:25:00.280957+0000","last_fullsized":"2024-03-06T07:25:00.280957+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:03.356871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496503+0000","last_change":"2024-03-06T07:24:59.300417+0000","last_active":"2024-03-06T07:25:00.496503+0000","last_peered":"2024-03-06T07:25:00.496503+0000","last_clean":"2024-03-06T07:25:00.496503+0000","last_became_active":"2024-03-06T07:24:59.300084+0000","last_became_peered":"2024-03-06T07:24:59.300084+0000","last_unstale":"2024-03-06T07:25:00.496503+0000","last_undegraded":"2024-03-06T07:25:00.496503+0000","last_fullsized":"2024-03-06T07:25:00.496503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:51.966784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189725+0000","last_change":"2024-03-06T07:24:58.293625+0000","last_active":"2024-03-06T07:25:01.189725+0000","last_peered":"2024-03-06T07:25:01.189725+0000","last_clean":"2024-03-06T07:25:01.189725+0000","last_became_active":"2024-03-06T07:24:58.293328+0000","last_became_peered":"2024-03-06T07:24:58.293328+0000","last_unstale":"2024-03-06T07:25:01.189725+0000","last_undegraded":"2024-03-06T07:25:01.189725+0000","last_fullsized":"2024-03-06T07:25:01.189725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:49.716667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496460+0000","last_change":"2024-03-06T07:24:59.300218+0000","last_active":"2024-03-06T07:25:00.496460+0000","last_peered":"2024-03-06T07:25:00.496460+0000","last_clean":"2024-03-06T07:25:00.496460+0000","last_became_active":"2024-03-06T07:24:59.299977+0000","last_became_peered":"2024-03-06T07:24:59.299977+0000","last_unstale":"2024-03-06T07:25:00.496460+0000","last_undegraded":"2024-03-06T07:25:00.496460+0000","last_fullsized":"2024-03-06T07:25:00.496460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:02.788779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284221+0000","last_change":"2024-03-06T07:24:58.295781+0000","last_active":"2024-03-06T07:25:01.284221+0000","last_peered":"2024-03-06T07:25:01.284221+0000","last_clean":"2024-03-06T07:25:01.284221+0000","last_became_active":"2024-03-06T07:24:58.295430+0000","last_became_peered":"2024-03-06T07:24:58.295430+0000","last_unstale":"2024-03-06T07:25:01.284221+0000","last_undegraded":"2024-03-06T07:25:01.284221+0000","last_fullsized":"2024-03-06T07:25:01.284221+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:54.331409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284646+0000","last_change":"2024-03-06T07:24:59.299253+0000","last_active":"2024-03-06T07:25:01.284646+0000","last_peered":"2024-03-06T07:25:01.284646+0000","last_clean":"2024-03-06T07:25:01.284646+0000","last_became_active":"2024-03-06T07:24:59.298992+0000","last_became_peered":"2024-03-06T07:24:59.298992+0000","last_unstale":"2024-03-06T07:25:01.284646+0000","last_undegraded":"2024-03-06T07:25:01.284646+0000","last_fullsized":"2024-03-06T07:25:01.284646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:50.745366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280721+0000","last_change":"2024-03-06T07:24:58.293223+0000","last_active":"2024-03-06T07:25:00.280721+0000","last_peered":"2024-03-06T07:25:00.280721+0000","last_clean":"2024-03-06T07:25:00.280721+0000","last_became_active":"2024-03-06T07:24:58.292940+0000","last_became_peered":"2024-03-06T07:24:58.292940+0000","last_unstale":"2024-03-06T07:25:00.280721+0000","last_undegraded":"2024-03-06T07:25:00.280721+0000","last_fullsized":"2024-03-06T07:25:00.280721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:15.610913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189696+0000","last_change":"2024-03-06T07:24:59.296017+0000","last_active":"2024-03-06T07:25:01.189696+0000","last_peered":"2024-03-06T07:25:01.189696+0000","last_clean":"2024-03-06T07:25:01.189696+0000","last_became_active":"2024-03-06T07:24:59.295730+0000","last_became_peered":"2024-03-06T07:24:59.295730+0000","last_unstale":"2024-03-06T07:25:01.189696+0000","last_undegraded":"2024-03-06T07:25:01.189696+0000","last_fullsized":"2024-03-06T07:25:01.189696+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:06.435738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189837+0000","last_change":"2024-03-06T07:24:58.290422+0000","last_active":"2024-03-06T07:25:01.189837+0000","last_peered":"2024-03-06T07:25:01.189837+0000","last_clean":"2024-03-06T07:25:01.189837+0000","last_became_active":"2024-03-06T07:24:58.290146+0000","last_became_peered":"2024-03-06T07:24:58.290146+0000","last_unstale":"2024-03-06T07:25:01.189837+0000","last_undegraded":"2024-03-06T07:25:01.189837+0000","last_fullsized":"2024-03-06T07:25:01.189837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:44.876762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695806+0000","last_change":"2024-03-06T07:24:59.297699+0000","last_active":"2024-03-06T07:25:00.695806+0000","last_peered":"2024-03-06T07:25:00.695806+0000","last_clean":"2024-03-06T07:25:00.695806+0000","last_became_active":"2024-03-06T07:24:59.297414+0000","last_became_peered":"2024-03-06T07:24:59.297414+0000","last_unstale":"2024-03-06T07:25:00.695806+0000","last_undegraded":"2024-03-06T07:25:00.695806+0000","last_fullsized":"2024-03-06T07:25:00.695806+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:24.707279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278754+0000","last_change":"2024-03-06T07:24:58.291745+0000","last_active":"2024-03-06T07:25:00.278754+0000","last_peered":"2024-03-06T07:25:00.278754+0000","last_clean":"2024-03-06T07:25:00.278754+0000","last_became_active":"2024-03-06T07:24:58.291465+0000","last_became_peered":"2024-03-06T07:24:58.291465+0000","last_unstale":"2024-03-06T07:25:00.278754+0000","last_undegraded":"2024-03-06T07:25:00.278754+0000","last_fullsized":"2024-03-06T07:25:00.278754+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:51.194465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695919+0000","last_change":"2024-03-06T07:24:59.293638+0000","last_active":"2024-03-06T07:25:00.695919+0000","last_peered":"2024-03-06T07:25:00.695919+0000","last_clean":"2024-03-06T07:25:00.695919+0000","last_became_active":"2024-03-06T07:24:59.293372+0000","last_became_peered":"2024-03-06T07:24:59.293372+0000","last_unstale":"2024-03-06T07:25:00.695919+0000","last_undegraded":"2024-03-06T07:25:00.695919+0000","last_fullsized":"2024-03-06T07:25:00.695919+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:19.927845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284493+0000","last_change":"2024-03-06T07:24:58.292455+0000","last_active":"2024-03-06T07:25:01.284493+0000","last_peered":"2024-03-06T07:25:01.284493+0000","last_clean":"2024-03-06T07:25:01.284493+0000","last_became_active":"2024-03-06T07:24:58.292096+0000","last_became_peered":"2024-03-06T07:24:58.292096+0000","last_unstale":"2024-03-06T07:25:01.284493+0000","last_undegraded":"2024-03-06T07:25:01.284493+0000","last_fullsized":"2024-03-06T07:25:01.284493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:52.458808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279112+0000","last_change":"2024-03-06T07:24:59.293894+0000","last_active":"2024-03-06T07:25:00.279112+0000","last_peered":"2024-03-06T07:25:00.279112+0000","last_clean":"2024-03-06T07:25:00.279112+0000","last_became_active":"2024-03-06T07:24:59.293362+0000","last_became_peered":"2024-03-06T07:24:59.293362+0000","last_unstale":"2024-03-06T07:25:00.279112+0000","last_undegraded":"2024-03-06T07:25:00.279112+0000","last_fullsized":"2024-03-06T07:25:00.279112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:34.893813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278591+0000","last_change":"2024-03-06T07:24:58.290189+0000","last_active":"2024-03-06T07:25:00.278591+0000","last_peered":"2024-03-06T07:25:00.278591+0000","last_clean":"2024-03-06T07:25:00.278591+0000","last_became_active":"2024-03-06T07:24:58.289929+0000","last_became_peered":"2024-03-06T07:24:58.289929+0000","last_unstale":"2024-03-06T07:25:00.278591+0000","last_undegraded":"2024-03-06T07:25:00.278591+0000","last_fullsized":"2024-03-06T07:25:00.278591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:27.586473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279915+0000","last_change":"2024-03-06T07:24:59.289846+0000","last_active":"2024-03-06T07:25:00.279915+0000","last_peered":"2024-03-06T07:25:00.279915+0000","last_clean":"2024-03-06T07:25:00.279915+0000","last_became_active":"2024-03-06T07:24:59.289374+0000","last_became_peered":"2024-03-06T07:24:59.289374+0000","last_unstale":"2024-03-06T07:25:00.279915+0000","last_undegraded":"2024-03-06T07:25:00.279915+0000","last_fullsized":"2024-03-06T07:25:00.279915+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:40.334461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689745+0000","last_change":"2024-03-06T07:24:58.287733+0000","last_active":"2024-03-06T07:25:01.689745+0000","last_peered":"2024-03-06T07:25:01.689745+0000","last_clean":"2024-03-06T07:25:01.689745+0000","last_became_active":"2024-03-06T07:24:58.287475+0000","last_became_peered":"2024-03-06T07:24:58.287475+0000","last_unstale":"2024-03-06T07:25:01.689745+0000","last_undegraded":"2024-03-06T07:25:01.689745+0000","last_fullsized":"2024-03-06T07:25:01.689745+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:32.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689882+0000","last_change":"2024-03-06T07:24:59.301355+0000","last_active":"2024-03-06T07:25:01.689882+0000","last_peered":"2024-03-06T07:25:01.689882+0000","last_clean":"2024-03-06T07:25:01.689882+0000","last_became_active":"2024-03-06T07:24:59.301070+0000","last_became_peered":"2024-03-06T07:24:59.301070+0000","last_unstale":"2024-03-06T07:25:01.689882+0000","last_undegraded":"2024-03-06T07:25:01.689882+0000","last_fullsized":"2024-03-06T07:25:01.689882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:52.968955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280117+0000","last_change":"2024-03-06T07:24:58.287276+0000","last_active":"2024-03-06T07:25:00.280117+0000","last_peered":"2024-03-06T07:25:00.280117+0000","last_clean":"2024-03-06T07:25:00.280117+0000","last_became_active":"2024-03-06T07:24:58.287015+0000","last_became_peered":"2024-03-06T07:24:58.287015+0000","last_unstale":"2024-03-06T07:25:00.280117+0000","last_undegraded":"2024-03-06T07:25:00.280117+0000","last_fullsized":"2024-03-06T07:25:00.280117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:59.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280049+0000","last_change":"2024-03-06T07:24:59.293984+0000","last_active":"2024-03-06T07:25:00.280049+0000","last_peered":"2024-03-06T07:25:00.280049+0000","last_clean":"2024-03-06T07:25:00.280049+0000","last_became_active":"2024-03-06T07:24:59.293725+0000","last_became_peered":"2024-03-06T07:24:59.293725+0000","last_unstale":"2024-03-06T07:25:00.280049+0000","last_undegraded":"2024-03-06T07:25:00.280049+0000","last_fullsized":"2024-03-06T07:25:00.280049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:22.942791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.281040+0000","last_change":"2024-03-06T07:24:58.293026+0000","last_active":"2024-03-06T07:25:00.281040+0000","last_peered":"2024-03-06T07:25:00.281040+0000","last_clean":"2024-03-06T07:25:00.281040+0000","last_became_active":"2024-03-06T07:24:58.292754+0000","last_became_peered":"2024-03-06T07:24:58.292754+0000","last_unstale":"2024-03-06T07:25:00.281040+0000","last_undegraded":"2024-03-06T07:25:00.281040+0000","last_fullsized":"2024-03-06T07:25:00.281040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:25.913477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496716+0000","last_change":"2024-03-06T07:24:59.301173+0000","last_active":"2024-03-06T07:25:00.496716+0000","last_peered":"2024-03-06T07:25:00.496716+0000","last_clean":"2024-03-06T07:25:00.496716+0000","last_became_active":"2024-03-06T07:24:59.300231+0000","last_became_peered":"2024-03-06T07:24:59.300231+0000","last_unstale":"2024-03-06T07:25:00.496716+0000","last_undegraded":"2024-03-06T07:25:00.496716+0000","last_fullsized":"2024-03-06T07:25:00.496716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:58.080499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279858+0000","last_change":"2024-03-06T07:24:58.281091+0000","last_active":"2024-03-06T07:25:00.279858+0000","last_peered":"2024-03-06T07:25:00.279858+0000","last_clean":"2024-03-06T07:25:00.279858+0000","last_became_active":"2024-03-06T07:24:58.280818+0000","last_became_peered":"2024-03-06T07:24:58.280818+0000","last_unstale":"2024-03-06T07:25:00.279858+0000","last_undegraded":"2024-03-06T07:25:00.279858+0000","last_fullsized":"2024-03-06T07:25:00.279858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:32.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":256,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695377+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:25:00.695377+0000","last_peered":"2024-03-06T07:25:00.695377+0000","last_clean":"2024-03-06T07:25:00.695377+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:25:00.695377+0000","last_undegraded":"2024-03-06T07:25:00.695377+0000","last_fullsized":"2024-03-06T07:25:00.695377+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1441792,"data_stored":943272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36168,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335672,"statfs":{"total":96636764160,"available":96599728128,"internally_reserved":0,"allocated":847872,"data_stored":352380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36164,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335676,"statfs":{"total":96636764160,"available":96599732224,"internally_reserved":0,"allocated":843776,"data_stored":352378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36168,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335672,"statfs":{"total":96636764160,"available":96599728128,"internally_reserved":0,"allocated":843776,"data_stored":352358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":352400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":353900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":354516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1433600,"data_stored":942656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:04.562 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-03-06T07:25:04.560+0000 7f0717e05640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:06.803 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:06.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.903+0000 7f2fcde4e640 1 Processor -- start 2024-03-06T07:25:06.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.903+0000 7f2fcde4e640 1 -- start start 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fcde4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc806b220 0x7f2fc815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fcde4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2fc80ce920 0x7f2fc8071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fcde4e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc815ea30 con 0x7f2fc8071ec0 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fcde4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc8161b70 con 0x7f2fc80ce920 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fcde4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc815ed10 con 0x7f2fc806b220 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2fc8071ec0 0x7f2fc8162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59372/0 (socket says 172.21.15.77:59372) 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3018791072 learned_addr learned my addr 172.21.15.77:0/3018791072 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc806b220 0x7f2fc815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc6ffd640 1 --2- 172.21.15.77:0/3018791072 >> v2:172.21.15.158:3300/0 conn(0x7f2fc80ce920 0x7f2fc8071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037354553 0 0) 0x7f2fc815ea30 con 0x7f2fc8071ec0 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb0003520 con 0x7f2fc8071ec0 2024-03-06T07:25:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 297395294 0 0) 0x7f2fb0003520 con 0x7f2fc8071ec0 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc806b220 msgr2=0x7f2fc815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc806b220 0x7f2fc815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 >> v2:172.21.15.158:3300/0 conn(0x7f2fc80ce920 msgr2=0x7f2fc8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 --2- 172.21.15.77:0/3018791072 >> v2:172.21.15.158:3300/0 conn(0x7f2fc80ce920 0x7f2fc8071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc815eff0 con 0x7f2fc8071ec0 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f2fc00019f0 con 0x7f2fc8071ec0 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 >> v1:172.21.15.77:6789/0 conn(0x7f2fc8071ec0 legacy=0x7f2fc8162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00039b0 0x7f2fb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.904+0000 7f2fc67fc640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc806b220 0x7f2fb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc67fc640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb0004d40 0x7f2fb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc8161b70 con 0x7f2fb00039b0 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc815ed10 con 0x7f2fb0004d40 2024-03-06T07:25:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb000b9c0 con 0x7f2fc806b220 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00039b0 0x7f2fb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc7fff640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb0004d40 0x7f2fb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc6ffd640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc806b220 0x7f2fb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc77fe640 1 -- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc806b220 msgr2=0x7f2fb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc806b220 0x7f2fb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc77fe640 1 -- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb0004d40 msgr2=0x7f2fb00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb0004d40 0x7f2fb00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc77fe640 1 -- 172.21.15.77:0/3018791072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0003520 con 0x7f2fb00039b0 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.905+0000 7f2fc7fff640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb0004d40 0x7f2fb00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.906+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00039b0 0x7f2fb0003dd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009ac0 tx=0x7f2fb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b664ba71c489033c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.906+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc039070 con 0x7f2fb00039b0 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.906+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc02c6b0 con 0x7f2fb00039b0 2024-03-06T07:25:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.906+0000 7f2fc67fc640 1 -- 172.21.15.77:0/3018791072 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc009690 con 0x7f2fb00039b0 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.906+0000 7f2fcde4e640 1 -- 172.21.15.77:0/3018791072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00039b0 msgr2=0x7f2fb0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.907+0000 7f2fcde4e640 1 --2- 172.21.15.77:0/3018791072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fb00039b0 0x7f2fb0003dd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009ac0 tx=0x7f2fb0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.907+0000 7f2fc7fff640 1 -- 172.21.15.77:0/3018791072 reap_dead start 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.907+0000 7f2fcde4e640 1 -- 172.21.15.77:0/3018791072 shutdown_connections 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.907+0000 7f2fcde4e640 1 -- 172.21.15.77:0/3018791072 >> 172.21.15.77:0/3018791072 conn(0x7f2fc806c0c0 msgr2=0x7f2fc806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.907+0000 7f2fcde4e640 1 -- 172.21.15.77:0/3018791072 shutdown_connections 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.907+0000 7f2fcde4e640 1 -- 172.21.15.77:0/3018791072 wait complete. 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.908+0000 7f2fcde4e640 1 Processor -- start 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.908+0000 7f2fcde4e640 1 -- start start 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fcde4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb00039b0 0x7f2fc813f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fcde4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fc80b55e0 0x7f2fc815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fcde4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc815aef0 0x7f2fc815b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fcde4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80ce380 con 0x7f2fc80b55e0 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fcde4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80ce200 con 0x7f2fb00039b0 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fcde4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc8162710 con 0x7f2fc815aef0 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fc80b55e0 0x7f2fc815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fc80b55e0 0x7f2fc815fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44634/0 (socket says 172.21.15.77:44634) 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 -- 172.21.15.77:0/2858802788 learned_addr learned my addr 172.21.15.77:0/2858802788 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb00039b0 0x7f2fc813f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc7fff640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc815aef0 0x7f2fc815b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 -- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc815aef0 msgr2=0x7f2fc815b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc815aef0 0x7f2fc815b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 -- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb00039b0 msgr2=0x7f2fc813f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb00039b0 0x7f2fc813f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 -- 172.21.15.77:0/2858802788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc813f760 con 0x7f2fc80b55e0 2024-03-06T07:25:06.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb00039b0 0x7f2fc813f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fc6ffd640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fc80b55e0 0x7f2fc815fab0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8011730 tx=0x7f2fb8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2faffff640 1 -- 172.21.15.77:0/2858802788 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800f9c0 con 0x7f2fc80b55e0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2faffff640 1 -- 172.21.15.77:0/2858802788 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb800fb60 con 0x7f2fc80b55e0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc813fa30 con 0x7f2fc80b55e0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2faffff640 1 -- 172.21.15.77:0/2858802788 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8015460 con 0x7f2fc80b55e0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.909+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc813fff0 con 0x7f2fc80b55e0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.910+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc815b890 con 0x7f2fc80b55e0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.911+0000 7f2faffff640 1 -- 172.21.15.77:0/2858802788 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fb8015600 con 0x7f2fc80b55e0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.911+0000 7f2faffff640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fb406b4d0 0x7f2fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.912+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fb406b4d0 0x7f2fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.912+0000 7f2faffff640 1 -- 172.21.15.77:0/2858802788 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fb808ab30 con 0x7f2fc80b55e0 2024-03-06T07:25:06.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.912+0000 7f2fc77fe640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fb406b4d0 0x7f2fb406d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc036000 tx=0x7f2fbc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:06.916+0000 7f2faffff640 1 -- 172.21.15.77:0/2858802788 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fb8019030 con 0x7f2fc80b55e0 2024-03-06T07:25:07.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.084+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2fc813cee0 con 0x7f2fb406b4d0 2024-03-06T07:25:07.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.090+0000 7f2faffff640 1 -- 172.21.15.77:0/2858802788 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252067 (secure 0 0 0) 0x7f2fc813cee0 con 0x7f2fb406b4d0 2024-03-06T07:25:07.094 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:07.095 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:25:07.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fb406b4d0 msgr2=0x7f2fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fb406b4d0 0x7f2fb406d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc036000 tx=0x7f2fbc030000 comp rx=0 tx=0).stop 2024-03-06T07:25:07.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fc80b55e0 msgr2=0x7f2fc815fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fc80b55e0 0x7f2fc815fab0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8011730 tx=0x7f2fb8011bf0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 shutdown_connections 2024-03-06T07:25:07.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fb406b4d0 0x7f2fb406d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fc80b55e0 0x7f2fc815fab0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fc815aef0 0x7f2fc815b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 --2- 172.21.15.77:0/2858802788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fb00039b0 0x7f2fc813f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 >> 172.21.15.77:0/2858802788 conn(0x7f2fc806c0c0 msgr2=0x7f2fc8072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 shutdown_connections 2024-03-06T07:25:07.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.093+0000 7f2fcde4e640 1 -- 172.21.15.77:0/2858802788 wait complete. 2024-03-06T07:25:07.106 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-03-06T07:25:05.417493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":273876,"kb_used_data":7784,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754700844,"statfs":{"total":773094113280,"available":772813664256,"internally_reserved":0,"allocated":7970816,"data_stored":4003860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.142882"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695917+0000","last_change":"2024-03-06T07:24:59.302574+0000","last_active":"2024-03-06T07:25:00.695917+0000","last_peered":"2024-03-06T07:25:00.695917+0000","last_clean":"2024-03-06T07:25:00.695917+0000","last_became_active":"2024-03-06T07:24:59.302188+0000","last_became_peered":"2024-03-06T07:24:59.302188+0000","last_unstale":"2024-03-06T07:25:00.695917+0000","last_undegraded":"2024-03-06T07:25:00.695917+0000","last_fullsized":"2024-03-06T07:25:00.695917+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:31.250047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280766+0000","last_change":"2024-03-06T07:24:59.293289+0000","last_active":"2024-03-06T07:25:00.280766+0000","last_peered":"2024-03-06T07:25:00.280766+0000","last_clean":"2024-03-06T07:25:00.280766+0000","last_became_active":"2024-03-06T07:24:59.292991+0000","last_became_peered":"2024-03-06T07:24:59.292991+0000","last_unstale":"2024-03-06T07:25:00.280766+0000","last_undegraded":"2024-03-06T07:25:00.280766+0000","last_fullsized":"2024-03-06T07:25:00.280766+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:04.812363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496552+0000","last_change":"2024-03-06T07:24:59.296498+0000","last_active":"2024-03-06T07:25:00.496552+0000","last_peered":"2024-03-06T07:25:00.496552+0000","last_clean":"2024-03-06T07:25:00.496552+0000","last_became_active":"2024-03-06T07:24:59.296227+0000","last_became_peered":"2024-03-06T07:24:59.296227+0000","last_unstale":"2024-03-06T07:25:00.496552+0000","last_undegraded":"2024-03-06T07:25:00.496552+0000","last_fullsized":"2024-03-06T07:25:00.496552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:36.220103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689993+0000","last_change":"2024-03-06T07:24:59.306562+0000","last_active":"2024-03-06T07:25:01.689993+0000","last_peered":"2024-03-06T07:25:01.689993+0000","last_clean":"2024-03-06T07:25:01.689993+0000","last_became_active":"2024-03-06T07:24:59.306269+0000","last_became_peered":"2024-03-06T07:24:59.306269+0000","last_unstale":"2024-03-06T07:25:01.689993+0000","last_undegraded":"2024-03-06T07:25:01.689993+0000","last_fullsized":"2024-03-06T07:25:01.689993+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:15.692819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278668+0000","last_change":"2024-03-06T07:24:59.292427+0000","last_active":"2024-03-06T07:25:00.278668+0000","last_peered":"2024-03-06T07:25:00.278668+0000","last_clean":"2024-03-06T07:25:00.278668+0000","last_became_active":"2024-03-06T07:24:59.291543+0000","last_became_peered":"2024-03-06T07:24:59.291543+0000","last_unstale":"2024-03-06T07:25:00.278668+0000","last_undegraded":"2024-03-06T07:25:00.278668+0000","last_fullsized":"2024-03-06T07:25:00.278668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:12.915316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280763+0000","last_change":"2024-03-06T07:24:59.296561+0000","last_active":"2024-03-06T07:25:00.280763+0000","last_peered":"2024-03-06T07:25:00.280763+0000","last_clean":"2024-03-06T07:25:00.280763+0000","last_became_active":"2024-03-06T07:24:59.296297+0000","last_became_peered":"2024-03-06T07:24:59.296297+0000","last_unstale":"2024-03-06T07:25:00.280763+0000","last_undegraded":"2024-03-06T07:25:00.280763+0000","last_fullsized":"2024-03-06T07:25:00.280763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:31.496570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189514+0000","last_change":"2024-03-06T07:24:59.293417+0000","last_active":"2024-03-06T07:25:01.189514+0000","last_peered":"2024-03-06T07:25:01.189514+0000","last_clean":"2024-03-06T07:25:01.189514+0000","last_became_active":"2024-03-06T07:24:59.292763+0000","last_became_peered":"2024-03-06T07:24:59.292763+0000","last_unstale":"2024-03-06T07:25:01.189514+0000","last_undegraded":"2024-03-06T07:25:01.189514+0000","last_fullsized":"2024-03-06T07:25:01.189514+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:00.346652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.281003+0000","last_change":"2024-03-06T07:24:59.300531+0000","last_active":"2024-03-06T07:25:00.281003+0000","last_peered":"2024-03-06T07:25:00.281003+0000","last_clean":"2024-03-06T07:25:00.281003+0000","last_became_active":"2024-03-06T07:24:59.299791+0000","last_became_peered":"2024-03-06T07:24:59.299791+0000","last_unstale":"2024-03-06T07:25:00.281003+0000","last_undegraded":"2024-03-06T07:25:00.281003+0000","last_fullsized":"2024-03-06T07:25:00.281003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:50.718497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278860+0000","last_change":"2024-03-06T07:24:59.287790+0000","last_active":"2024-03-06T07:25:00.278860+0000","last_peered":"2024-03-06T07:25:00.278860+0000","last_clean":"2024-03-06T07:25:00.278860+0000","last_became_active":"2024-03-06T07:24:59.287515+0000","last_became_peered":"2024-03-06T07:24:59.287515+0000","last_unstale":"2024-03-06T07:25:00.278860+0000","last_undegraded":"2024-03-06T07:25:00.278860+0000","last_fullsized":"2024-03-06T07:25:00.278860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:42.668609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280350+0000","last_change":"2024-03-06T07:24:59.292646+0000","last_active":"2024-03-06T07:25:00.280350+0000","last_peered":"2024-03-06T07:25:00.280350+0000","last_clean":"2024-03-06T07:25:00.280350+0000","last_became_active":"2024-03-06T07:24:59.292366+0000","last_became_peered":"2024-03-06T07:24:59.292366+0000","last_unstale":"2024-03-06T07:25:00.280350+0000","last_undegraded":"2024-03-06T07:25:00.280350+0000","last_fullsized":"2024-03-06T07:25:00.280350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:18.709021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695749+0000","last_change":"2024-03-06T07:24:59.295856+0000","last_active":"2024-03-06T07:25:00.695749+0000","last_peered":"2024-03-06T07:25:00.695749+0000","last_clean":"2024-03-06T07:25:00.695749+0000","last_became_active":"2024-03-06T07:24:59.295587+0000","last_became_peered":"2024-03-06T07:24:59.295587+0000","last_unstale":"2024-03-06T07:25:00.695749+0000","last_undegraded":"2024-03-06T07:25:00.695749+0000","last_fullsized":"2024-03-06T07:25:00.695749+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:16.578112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.690073+0000","last_change":"2024-03-06T07:24:59.299041+0000","last_active":"2024-03-06T07:25:01.690073+0000","last_peered":"2024-03-06T07:25:01.690073+0000","last_clean":"2024-03-06T07:25:01.690073+0000","last_became_active":"2024-03-06T07:24:59.298798+0000","last_became_peered":"2024-03-06T07:24:59.298798+0000","last_unstale":"2024-03-06T07:25:01.690073+0000","last_undegraded":"2024-03-06T07:25:01.690073+0000","last_fullsized":"2024-03-06T07:25:01.690073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:34.677390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284433+0000","last_change":"2024-03-06T07:24:59.297747+0000","last_active":"2024-03-06T07:25:01.284433+0000","last_peered":"2024-03-06T07:25:01.284433+0000","last_clean":"2024-03-06T07:25:01.284433+0000","last_became_active":"2024-03-06T07:24:59.297402+0000","last_became_peered":"2024-03-06T07:24:59.297402+0000","last_unstale":"2024-03-06T07:25:01.284433+0000","last_undegraded":"2024-03-06T07:25:01.284433+0000","last_fullsized":"2024-03-06T07:25:01.284433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:39.856757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278978+0000","last_change":"2024-03-06T07:24:59.290253+0000","last_active":"2024-03-06T07:25:00.278978+0000","last_peered":"2024-03-06T07:25:00.278978+0000","last_clean":"2024-03-06T07:25:00.278978+0000","last_became_active":"2024-03-06T07:24:59.289967+0000","last_became_peered":"2024-03-06T07:24:59.289967+0000","last_unstale":"2024-03-06T07:25:00.278978+0000","last_undegraded":"2024-03-06T07:25:00.278978+0000","last_fullsized":"2024-03-06T07:25:00.278978+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:16.612743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284353+0000","last_change":"2024-03-06T07:24:59.296686+0000","last_active":"2024-03-06T07:25:01.284353+0000","last_peered":"2024-03-06T07:25:01.284353+0000","last_clean":"2024-03-06T07:25:01.284353+0000","last_became_active":"2024-03-06T07:24:59.296329+0000","last_became_peered":"2024-03-06T07:24:59.296329+0000","last_unstale":"2024-03-06T07:25:01.284353+0000","last_undegraded":"2024-03-06T07:25:01.284353+0000","last_fullsized":"2024-03-06T07:25:01.284353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:27.080068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279506+0000","last_change":"2024-03-06T07:24:59.299688+0000","last_active":"2024-03-06T07:25:00.279506+0000","last_peered":"2024-03-06T07:25:00.279506+0000","last_clean":"2024-03-06T07:25:00.279506+0000","last_became_active":"2024-03-06T07:24:59.299410+0000","last_became_peered":"2024-03-06T07:24:59.299410+0000","last_unstale":"2024-03-06T07:25:00.279506+0000","last_undegraded":"2024-03-06T07:25:00.279506+0000","last_fullsized":"2024-03-06T07:25:00.279506+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:04.812363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278876+0000","last_change":"2024-03-06T07:24:59.286850+0000","last_active":"2024-03-06T07:25:00.278876+0000","last_peered":"2024-03-06T07:25:00.278876+0000","last_clean":"2024-03-06T07:25:00.278876+0000","last_became_active":"2024-03-06T07:24:59.286604+0000","last_became_peered":"2024-03-06T07:24:59.286604+0000","last_unstale":"2024-03-06T07:25:00.278876+0000","last_undegraded":"2024-03-06T07:25:00.278876+0000","last_fullsized":"2024-03-06T07:25:00.278876+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:31.100044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.690132+0000","last_change":"2024-03-06T07:24:59.305671+0000","last_active":"2024-03-06T07:25:01.690132+0000","last_peered":"2024-03-06T07:25:01.690132+0000","last_clean":"2024-03-06T07:25:01.690132+0000","last_became_active":"2024-03-06T07:24:59.305411+0000","last_became_peered":"2024-03-06T07:24:59.305411+0000","last_unstale":"2024-03-06T07:25:01.690132+0000","last_undegraded":"2024-03-06T07:25:01.690132+0000","last_fullsized":"2024-03-06T07:25:01.690132+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:46.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280591+0000","last_change":"2024-03-06T07:24:59.299801+0000","last_active":"2024-03-06T07:25:00.280591+0000","last_peered":"2024-03-06T07:25:00.280591+0000","last_clean":"2024-03-06T07:25:00.280591+0000","last_became_active":"2024-03-06T07:24:59.299499+0000","last_became_peered":"2024-03-06T07:24:59.299499+0000","last_unstale":"2024-03-06T07:25:00.280591+0000","last_undegraded":"2024-03-06T07:25:00.280591+0000","last_fullsized":"2024-03-06T07:25:00.280591+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:45.209746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284588+0000","last_change":"2024-03-06T07:24:59.291143+0000","last_active":"2024-03-06T07:25:01.284588+0000","last_peered":"2024-03-06T07:25:01.284588+0000","last_clean":"2024-03-06T07:25:01.284588+0000","last_became_active":"2024-03-06T07:24:59.290881+0000","last_became_peered":"2024-03-06T07:24:59.290881+0000","last_unstale":"2024-03-06T07:25:01.284588+0000","last_undegraded":"2024-03-06T07:25:01.284588+0000","last_fullsized":"2024-03-06T07:25:01.284588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:45.289459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279945+0000","last_change":"2024-03-06T07:24:59.296968+0000","last_active":"2024-03-06T07:25:00.279945+0000","last_peered":"2024-03-06T07:25:00.279945+0000","last_clean":"2024-03-06T07:25:00.279945+0000","last_became_active":"2024-03-06T07:24:59.296711+0000","last_became_peered":"2024-03-06T07:24:59.296711+0000","last_unstale":"2024-03-06T07:25:00.279945+0000","last_undegraded":"2024-03-06T07:25:00.279945+0000","last_fullsized":"2024-03-06T07:25:00.279945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:31.295986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279914+0000","last_change":"2024-03-06T07:24:59.286100+0000","last_active":"2024-03-06T07:25:00.279914+0000","last_peered":"2024-03-06T07:25:00.279914+0000","last_clean":"2024-03-06T07:25:00.279914+0000","last_became_active":"2024-03-06T07:24:59.285771+0000","last_became_peered":"2024-03-06T07:24:59.285771+0000","last_unstale":"2024-03-06T07:25:00.279914+0000","last_undegraded":"2024-03-06T07:25:00.279914+0000","last_fullsized":"2024-03-06T07:25:00.279914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:45.878592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695438+0000","last_change":"2024-03-06T07:24:59.294002+0000","last_active":"2024-03-06T07:25:00.695438+0000","last_peered":"2024-03-06T07:25:00.695438+0000","last_clean":"2024-03-06T07:25:00.695438+0000","last_became_active":"2024-03-06T07:24:59.293512+0000","last_became_peered":"2024-03-06T07:24:59.293512+0000","last_unstale":"2024-03-06T07:25:00.695438+0000","last_undegraded":"2024-03-06T07:25:00.695438+0000","last_fullsized":"2024-03-06T07:25:00.695438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:32.810212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695389+0000","last_change":"2024-03-06T07:24:59.301374+0000","last_active":"2024-03-06T07:25:00.695389+0000","last_peered":"2024-03-06T07:25:00.695389+0000","last_clean":"2024-03-06T07:25:00.695389+0000","last_became_active":"2024-03-06T07:24:59.300966+0000","last_became_peered":"2024-03-06T07:24:59.300966+0000","last_unstale":"2024-03-06T07:25:00.695389+0000","last_undegraded":"2024-03-06T07:25:00.695389+0000","last_fullsized":"2024-03-06T07:25:00.695389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:43.197457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279699+0000","last_change":"2024-03-06T07:24:59.295435+0000","last_active":"2024-03-06T07:25:00.279699+0000","last_peered":"2024-03-06T07:25:00.279699+0000","last_clean":"2024-03-06T07:25:00.279699+0000","last_became_active":"2024-03-06T07:24:59.295170+0000","last_became_peered":"2024-03-06T07:24:59.295170+0000","last_unstale":"2024-03-06T07:25:00.279699+0000","last_undegraded":"2024-03-06T07:25:00.279699+0000","last_fullsized":"2024-03-06T07:25:00.279699+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:50.718497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189750+0000","last_change":"2024-03-06T07:24:59.302154+0000","last_active":"2024-03-06T07:25:01.189750+0000","last_peered":"2024-03-06T07:25:01.189750+0000","last_clean":"2024-03-06T07:25:01.189750+0000","last_became_active":"2024-03-06T07:24:59.301886+0000","last_became_peered":"2024-03-06T07:24:59.301886+0000","last_unstale":"2024-03-06T07:25:01.189750+0000","last_undegraded":"2024-03-06T07:25:01.189750+0000","last_fullsized":"2024-03-06T07:25:01.189750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:42.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496555+0000","last_change":"2024-03-06T07:24:59.288408+0000","last_active":"2024-03-06T07:25:00.496555+0000","last_peered":"2024-03-06T07:25:00.496555+0000","last_clean":"2024-03-06T07:25:00.496555+0000","last_became_active":"2024-03-06T07:24:59.285862+0000","last_became_peered":"2024-03-06T07:24:59.285862+0000","last_unstale":"2024-03-06T07:25:00.496555+0000","last_undegraded":"2024-03-06T07:25:00.496555+0000","last_fullsized":"2024-03-06T07:25:00.496555+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:05.383872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280759+0000","last_change":"2024-03-06T07:24:59.291942+0000","last_active":"2024-03-06T07:25:00.280759+0000","last_peered":"2024-03-06T07:25:00.280759+0000","last_clean":"2024-03-06T07:25:00.280759+0000","last_became_active":"2024-03-06T07:24:59.290198+0000","last_became_peered":"2024-03-06T07:24:59.290198+0000","last_unstale":"2024-03-06T07:25:00.280759+0000","last_undegraded":"2024-03-06T07:25:00.280759+0000","last_fullsized":"2024-03-06T07:25:00.280759+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:53.460638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496443+0000","last_change":"2024-03-06T07:24:59.290512+0000","last_active":"2024-03-06T07:25:00.496443+0000","last_peered":"2024-03-06T07:25:00.496443+0000","last_clean":"2024-03-06T07:25:00.496443+0000","last_became_active":"2024-03-06T07:24:59.290282+0000","last_became_peered":"2024-03-06T07:24:59.290282+0000","last_unstale":"2024-03-06T07:25:00.496443+0000","last_undegraded":"2024-03-06T07:25:00.496443+0000","last_fullsized":"2024-03-06T07:25:00.496443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:57.853489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189530+0000","last_change":"2024-03-06T07:24:59.294729+0000","last_active":"2024-03-06T07:25:01.189530+0000","last_peered":"2024-03-06T07:25:01.189530+0000","last_clean":"2024-03-06T07:25:01.189530+0000","last_became_active":"2024-03-06T07:24:59.294468+0000","last_became_peered":"2024-03-06T07:24:59.294468+0000","last_unstale":"2024-03-06T07:25:01.189530+0000","last_undegraded":"2024-03-06T07:25:01.189530+0000","last_fullsized":"2024-03-06T07:25:01.189530+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:42.167568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189748+0000","last_change":"2024-03-06T07:24:59.293641+0000","last_active":"2024-03-06T07:25:01.189748+0000","last_peered":"2024-03-06T07:25:01.189748+0000","last_clean":"2024-03-06T07:25:01.189748+0000","last_became_active":"2024-03-06T07:24:59.293154+0000","last_became_peered":"2024-03-06T07:24:59.293154+0000","last_unstale":"2024-03-06T07:25:01.189748+0000","last_undegraded":"2024-03-06T07:25:01.189748+0000","last_fullsized":"2024-03-06T07:25:01.189748+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:27.080068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.281046+0000","last_change":"2024-03-06T07:24:59.298323+0000","last_active":"2024-03-06T07:25:00.281046+0000","last_peered":"2024-03-06T07:25:00.281046+0000","last_clean":"2024-03-06T07:25:00.281046+0000","last_became_active":"2024-03-06T07:24:59.298057+0000","last_became_peered":"2024-03-06T07:24:59.298057+0000","last_unstale":"2024-03-06T07:25:00.281046+0000","last_undegraded":"2024-03-06T07:25:00.281046+0000","last_fullsized":"2024-03-06T07:25:00.281046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:45.878592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695849+0000","last_change":"2024-03-06T07:24:59.303884+0000","last_active":"2024-03-06T07:25:00.695849+0000","last_peered":"2024-03-06T07:25:00.695849+0000","last_clean":"2024-03-06T07:25:00.695849+0000","last_became_active":"2024-03-06T07:24:59.303210+0000","last_became_peered":"2024-03-06T07:24:59.303210+0000","last_unstale":"2024-03-06T07:25:00.695849+0000","last_undegraded":"2024-03-06T07:25:00.695849+0000","last_fullsized":"2024-03-06T07:25:00.695849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:49.805086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.696032+0000","last_change":"2024-03-06T07:24:58.292643+0000","last_active":"2024-03-06T07:25:00.696032+0000","last_peered":"2024-03-06T07:25:00.696032+0000","last_clean":"2024-03-06T07:25:00.696032+0000","last_became_active":"2024-03-06T07:24:58.292289+0000","last_became_peered":"2024-03-06T07:24:58.292289+0000","last_unstale":"2024-03-06T07:25:00.696032+0000","last_undegraded":"2024-03-06T07:25:00.696032+0000","last_fullsized":"2024-03-06T07:25:00.696032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:10.684873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278962+0000","last_change":"2024-03-06T07:24:59.285713+0000","last_active":"2024-03-06T07:25:00.278962+0000","last_peered":"2024-03-06T07:25:00.278962+0000","last_clean":"2024-03-06T07:25:00.278962+0000","last_became_active":"2024-03-06T07:24:59.285460+0000","last_became_peered":"2024-03-06T07:24:59.285460+0000","last_unstale":"2024-03-06T07:25:00.278962+0000","last_undegraded":"2024-03-06T07:25:00.278962+0000","last_fullsized":"2024-03-06T07:25:00.278962+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:23.755605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284609+0000","last_change":"2024-03-06T07:24:58.295724+0000","last_active":"2024-03-06T07:25:01.284609+0000","last_peered":"2024-03-06T07:25:01.284609+0000","last_clean":"2024-03-06T07:25:01.284609+0000","last_became_active":"2024-03-06T07:24:58.295420+0000","last_became_peered":"2024-03-06T07:24:58.295420+0000","last_unstale":"2024-03-06T07:25:01.284609+0000","last_undegraded":"2024-03-06T07:25:01.284609+0000","last_fullsized":"2024-03-06T07:25:01.284609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:30.699089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280112+0000","last_change":"2024-03-06T07:24:59.293098+0000","last_active":"2024-03-06T07:25:00.280112+0000","last_peered":"2024-03-06T07:25:00.280112+0000","last_clean":"2024-03-06T07:25:00.280112+0000","last_became_active":"2024-03-06T07:24:59.290216+0000","last_became_peered":"2024-03-06T07:24:59.290216+0000","last_unstale":"2024-03-06T07:25:00.280112+0000","last_undegraded":"2024-03-06T07:25:00.280112+0000","last_fullsized":"2024-03-06T07:25:00.280112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:31.650270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279659+0000","last_change":"2024-03-06T07:24:58.295696+0000","last_active":"2024-03-06T07:25:00.279659+0000","last_peered":"2024-03-06T07:25:00.279659+0000","last_clean":"2024-03-06T07:25:00.279659+0000","last_became_active":"2024-03-06T07:24:58.295083+0000","last_became_peered":"2024-03-06T07:24:58.295083+0000","last_unstale":"2024-03-06T07:25:00.279659+0000","last_undegraded":"2024-03-06T07:25:00.279659+0000","last_fullsized":"2024-03-06T07:25:00.279659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:33.675559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695844+0000","last_change":"2024-03-06T07:24:59.297688+0000","last_active":"2024-03-06T07:25:00.695844+0000","last_peered":"2024-03-06T07:25:00.695844+0000","last_clean":"2024-03-06T07:25:00.695844+0000","last_became_active":"2024-03-06T07:24:59.297313+0000","last_became_peered":"2024-03-06T07:24:59.297313+0000","last_unstale":"2024-03-06T07:25:00.695844+0000","last_undegraded":"2024-03-06T07:25:00.695844+0000","last_fullsized":"2024-03-06T07:25:00.695844+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:27.478963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280172+0000","last_change":"2024-03-06T07:24:58.290508+0000","last_active":"2024-03-06T07:25:00.280172+0000","last_peered":"2024-03-06T07:25:00.280172+0000","last_clean":"2024-03-06T07:25:00.280172+0000","last_became_active":"2024-03-06T07:24:58.286582+0000","last_became_peered":"2024-03-06T07:24:58.286582+0000","last_unstale":"2024-03-06T07:25:00.280172+0000","last_undegraded":"2024-03-06T07:25:00.280172+0000","last_fullsized":"2024-03-06T07:25:00.280172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:15.493830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189730+0000","last_change":"2024-03-06T07:24:59.292011+0000","last_active":"2024-03-06T07:25:01.189730+0000","last_peered":"2024-03-06T07:25:01.189730+0000","last_clean":"2024-03-06T07:25:01.189730+0000","last_became_active":"2024-03-06T07:24:59.291667+0000","last_became_peered":"2024-03-06T07:24:59.291667+0000","last_unstale":"2024-03-06T07:25:01.189730+0000","last_undegraded":"2024-03-06T07:25:01.189730+0000","last_fullsized":"2024-03-06T07:25:01.189730+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:36.866907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496493+0000","last_change":"2024-03-06T07:24:58.286577+0000","last_active":"2024-03-06T07:25:00.496493+0000","last_peered":"2024-03-06T07:25:00.496493+0000","last_clean":"2024-03-06T07:25:00.496493+0000","last_became_active":"2024-03-06T07:24:58.286259+0000","last_became_peered":"2024-03-06T07:24:58.286259+0000","last_unstale":"2024-03-06T07:25:00.496493+0000","last_undegraded":"2024-03-06T07:25:00.496493+0000","last_fullsized":"2024-03-06T07:25:00.496493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:15.576282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279957+0000","last_change":"2024-03-06T07:24:59.286948+0000","last_active":"2024-03-06T07:25:00.279957+0000","last_peered":"2024-03-06T07:25:00.279957+0000","last_clean":"2024-03-06T07:25:00.279957+0000","last_became_active":"2024-03-06T07:24:59.286660+0000","last_became_peered":"2024-03-06T07:24:59.286660+0000","last_unstale":"2024-03-06T07:25:00.279957+0000","last_undegraded":"2024-03-06T07:25:00.279957+0000","last_fullsized":"2024-03-06T07:25:00.279957+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:35.117491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280842+0000","last_change":"2024-03-06T07:24:58.290810+0000","last_active":"2024-03-06T07:25:00.280842+0000","last_peered":"2024-03-06T07:25:00.280842+0000","last_clean":"2024-03-06T07:25:00.280842+0000","last_became_active":"2024-03-06T07:24:58.290521+0000","last_became_peered":"2024-03-06T07:24:58.290521+0000","last_unstale":"2024-03-06T07:25:00.280842+0000","last_undegraded":"2024-03-06T07:25:00.280842+0000","last_fullsized":"2024-03-06T07:25:00.280842+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:27.748630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278542+0000","last_change":"2024-03-06T07:24:59.282271+0000","last_active":"2024-03-06T07:25:00.278542+0000","last_peered":"2024-03-06T07:25:00.278542+0000","last_clean":"2024-03-06T07:25:00.278542+0000","last_became_active":"2024-03-06T07:24:59.282018+0000","last_became_peered":"2024-03-06T07:24:59.282018+0000","last_unstale":"2024-03-06T07:25:00.278542+0000","last_undegraded":"2024-03-06T07:25:00.278542+0000","last_fullsized":"2024-03-06T07:25:00.278542+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:58.115915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689815+0000","last_change":"2024-03-06T07:24:58.286735+0000","last_active":"2024-03-06T07:25:01.689815+0000","last_peered":"2024-03-06T07:25:01.689815+0000","last_clean":"2024-03-06T07:25:01.689815+0000","last_became_active":"2024-03-06T07:24:58.285523+0000","last_became_peered":"2024-03-06T07:24:58.285523+0000","last_unstale":"2024-03-06T07:25:01.689815+0000","last_undegraded":"2024-03-06T07:25:01.689815+0000","last_fullsized":"2024-03-06T07:25:01.689815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:33.891983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284418+0000","last_change":"2024-03-06T07:24:59.292380+0000","last_active":"2024-03-06T07:25:01.284418+0000","last_peered":"2024-03-06T07:25:01.284418+0000","last_clean":"2024-03-06T07:25:01.284418+0000","last_became_active":"2024-03-06T07:24:59.292123+0000","last_became_peered":"2024-03-06T07:24:59.292123+0000","last_unstale":"2024-03-06T07:25:01.284418+0000","last_undegraded":"2024-03-06T07:25:01.284418+0000","last_fullsized":"2024-03-06T07:25:01.284418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:25.492967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278455+0000","last_change":"2024-03-06T07:24:58.284398+0000","last_active":"2024-03-06T07:25:00.278455+0000","last_peered":"2024-03-06T07:25:00.278455+0000","last_clean":"2024-03-06T07:25:00.278455+0000","last_became_active":"2024-03-06T07:24:58.284101+0000","last_became_peered":"2024-03-06T07:24:58.284101+0000","last_unstale":"2024-03-06T07:25:00.278455+0000","last_undegraded":"2024-03-06T07:25:00.278455+0000","last_fullsized":"2024-03-06T07:25:00.278455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:06.467086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189834+0000","last_change":"2024-03-06T07:24:59.288497+0000","last_active":"2024-03-06T07:25:01.189834+0000","last_peered":"2024-03-06T07:25:01.189834+0000","last_clean":"2024-03-06T07:25:01.189834+0000","last_became_active":"2024-03-06T07:24:59.288161+0000","last_became_peered":"2024-03-06T07:24:59.288161+0000","last_unstale":"2024-03-06T07:25:01.189834+0000","last_undegraded":"2024-03-06T07:25:01.189834+0000","last_fullsized":"2024-03-06T07:25:01.189834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:25.492967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279734+0000","last_change":"2024-03-06T07:24:58.283835+0000","last_active":"2024-03-06T07:25:00.279734+0000","last_peered":"2024-03-06T07:25:00.279734+0000","last_clean":"2024-03-06T07:25:00.279734+0000","last_became_active":"2024-03-06T07:24:58.283535+0000","last_became_peered":"2024-03-06T07:24:58.283535+0000","last_unstale":"2024-03-06T07:25:00.279734+0000","last_undegraded":"2024-03-06T07:25:00.279734+0000","last_fullsized":"2024-03-06T07:25:00.279734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:10.044434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695976+0000","last_change":"2024-03-06T07:24:59.304238+0000","last_active":"2024-03-06T07:25:00.695976+0000","last_peered":"2024-03-06T07:25:00.695976+0000","last_clean":"2024-03-06T07:25:00.695976+0000","last_became_active":"2024-03-06T07:24:59.303930+0000","last_became_peered":"2024-03-06T07:24:59.303930+0000","last_unstale":"2024-03-06T07:25:00.695976+0000","last_undegraded":"2024-03-06T07:25:00.695976+0000","last_fullsized":"2024-03-06T07:25:00.695976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:37.414063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280830+0000","last_change":"2024-03-06T07:24:58.294620+0000","last_active":"2024-03-06T07:25:00.280830+0000","last_peered":"2024-03-06T07:25:00.280830+0000","last_clean":"2024-03-06T07:25:00.280830+0000","last_became_active":"2024-03-06T07:24:58.294312+0000","last_became_peered":"2024-03-06T07:24:58.294312+0000","last_unstale":"2024-03-06T07:25:00.280830+0000","last_undegraded":"2024-03-06T07:25:00.280830+0000","last_fullsized":"2024-03-06T07:25:00.280830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:03.367487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280510+0000","last_change":"2024-03-06T07:24:59.293316+0000","last_active":"2024-03-06T07:25:00.280510+0000","last_peered":"2024-03-06T07:25:00.280510+0000","last_clean":"2024-03-06T07:25:00.280510+0000","last_became_active":"2024-03-06T07:24:59.292835+0000","last_became_peered":"2024-03-06T07:24:59.292835+0000","last_unstale":"2024-03-06T07:25:00.280510+0000","last_undegraded":"2024-03-06T07:25:00.280510+0000","last_fullsized":"2024-03-06T07:25:00.280510+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:50.166642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496613+0000","last_change":"2024-03-06T07:24:58.288380+0000","last_active":"2024-03-06T07:25:00.496613+0000","last_peered":"2024-03-06T07:25:00.496613+0000","last_clean":"2024-03-06T07:25:00.496613+0000","last_became_active":"2024-03-06T07:24:58.287762+0000","last_became_peered":"2024-03-06T07:24:58.287762+0000","last_unstale":"2024-03-06T07:25:00.496613+0000","last_undegraded":"2024-03-06T07:25:00.496613+0000","last_fullsized":"2024-03-06T07:25:00.496613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:39.536983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278709+0000","last_change":"2024-03-06T07:24:59.295751+0000","last_active":"2024-03-06T07:25:00.278709+0000","last_peered":"2024-03-06T07:25:00.278709+0000","last_clean":"2024-03-06T07:25:00.278709+0000","last_became_active":"2024-03-06T07:24:59.295470+0000","last_became_peered":"2024-03-06T07:24:59.295470+0000","last_unstale":"2024-03-06T07:25:00.278709+0000","last_undegraded":"2024-03-06T07:25:00.278709+0000","last_fullsized":"2024-03-06T07:25:00.278709+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:12.651638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.690189+0000","last_change":"2024-03-06T07:24:58.283050+0000","last_active":"2024-03-06T07:25:01.690189+0000","last_peered":"2024-03-06T07:25:01.690189+0000","last_clean":"2024-03-06T07:25:01.690189+0000","last_became_active":"2024-03-06T07:24:58.282778+0000","last_became_peered":"2024-03-06T07:24:58.282778+0000","last_unstale":"2024-03-06T07:25:01.690189+0000","last_undegraded":"2024-03-06T07:25:01.690189+0000","last_fullsized":"2024-03-06T07:25:01.690189+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:15.493830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695599+0000","last_change":"2024-03-06T07:24:59.291367+0000","last_active":"2024-03-06T07:25:00.695599+0000","last_peered":"2024-03-06T07:25:00.695599+0000","last_clean":"2024-03-06T07:25:00.695599+0000","last_became_active":"2024-03-06T07:24:59.289436+0000","last_became_peered":"2024-03-06T07:24:59.289436+0000","last_unstale":"2024-03-06T07:25:00.695599+0000","last_undegraded":"2024-03-06T07:25:00.695599+0000","last_fullsized":"2024-03-06T07:25:00.695599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:40.538813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689861+0000","last_change":"2024-03-06T07:24:58.289949+0000","last_active":"2024-03-06T07:25:01.689861+0000","last_peered":"2024-03-06T07:25:01.689861+0000","last_clean":"2024-03-06T07:25:01.689861+0000","last_became_active":"2024-03-06T07:24:58.289646+0000","last_became_peered":"2024-03-06T07:24:58.289646+0000","last_unstale":"2024-03-06T07:25:01.689861+0000","last_undegraded":"2024-03-06T07:25:01.689861+0000","last_fullsized":"2024-03-06T07:25:01.689861+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:15.610913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189641+0000","last_change":"2024-03-06T07:24:59.291252+0000","last_active":"2024-03-06T07:25:01.189641+0000","last_peered":"2024-03-06T07:25:01.189641+0000","last_clean":"2024-03-06T07:25:01.189641+0000","last_became_active":"2024-03-06T07:24:59.290989+0000","last_became_peered":"2024-03-06T07:24:59.290989+0000","last_unstale":"2024-03-06T07:25:01.189641+0000","last_undegraded":"2024-03-06T07:25:01.189641+0000","last_fullsized":"2024-03-06T07:25:01.189641+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:09.656864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280852+0000","last_change":"2024-03-06T07:24:58.294391+0000","last_active":"2024-03-06T07:25:00.280852+0000","last_peered":"2024-03-06T07:25:00.280852+0000","last_clean":"2024-03-06T07:25:00.280852+0000","last_became_active":"2024-03-06T07:24:58.294053+0000","last_became_peered":"2024-03-06T07:24:58.294053+0000","last_unstale":"2024-03-06T07:25:00.280852+0000","last_undegraded":"2024-03-06T07:25:00.280852+0000","last_fullsized":"2024-03-06T07:25:00.280852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:05.491668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284412+0000","last_change":"2024-03-06T07:24:59.301281+0000","last_active":"2024-03-06T07:25:01.284412+0000","last_peered":"2024-03-06T07:25:01.284412+0000","last_clean":"2024-03-06T07:25:01.284412+0000","last_became_active":"2024-03-06T07:24:59.300861+0000","last_became_peered":"2024-03-06T07:24:59.300861+0000","last_unstale":"2024-03-06T07:25:01.284412+0000","last_undegraded":"2024-03-06T07:25:01.284412+0000","last_fullsized":"2024-03-06T07:25:01.284412+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:42.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189457+0000","last_change":"2024-03-06T07:24:58.289673+0000","last_active":"2024-03-06T07:25:01.189457+0000","last_peered":"2024-03-06T07:25:01.189457+0000","last_clean":"2024-03-06T07:25:01.189457+0000","last_became_active":"2024-03-06T07:24:58.289390+0000","last_became_peered":"2024-03-06T07:24:58.289390+0000","last_unstale":"2024-03-06T07:25:01.189457+0000","last_undegraded":"2024-03-06T07:25:01.189457+0000","last_fullsized":"2024-03-06T07:25:01.189457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:03.810533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278685+0000","last_change":"2024-03-06T07:24:59.291316+0000","last_active":"2024-03-06T07:25:00.278685+0000","last_peered":"2024-03-06T07:25:00.278685+0000","last_clean":"2024-03-06T07:25:00.278685+0000","last_became_active":"2024-03-06T07:24:59.291076+0000","last_became_peered":"2024-03-06T07:24:59.291076+0000","last_unstale":"2024-03-06T07:25:00.278685+0000","last_undegraded":"2024-03-06T07:25:00.278685+0000","last_fullsized":"2024-03-06T07:25:00.278685+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:03.778346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279476+0000","last_change":"2024-03-06T07:24:58.288346+0000","last_active":"2024-03-06T07:25:00.279476+0000","last_peered":"2024-03-06T07:25:00.279476+0000","last_clean":"2024-03-06T07:25:00.279476+0000","last_became_active":"2024-03-06T07:24:58.288014+0000","last_became_peered":"2024-03-06T07:24:58.288014+0000","last_unstale":"2024-03-06T07:25:00.279476+0000","last_undegraded":"2024-03-06T07:25:00.279476+0000","last_fullsized":"2024-03-06T07:25:00.279476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:45.893805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280579+0000","last_change":"2024-03-06T07:24:59.295750+0000","last_active":"2024-03-06T07:25:00.280579+0000","last_peered":"2024-03-06T07:25:00.280579+0000","last_clean":"2024-03-06T07:25:00.280579+0000","last_became_active":"2024-03-06T07:24:59.295222+0000","last_became_peered":"2024-03-06T07:24:59.295222+0000","last_unstale":"2024-03-06T07:25:00.280579+0000","last_undegraded":"2024-03-06T07:25:00.280579+0000","last_fullsized":"2024-03-06T07:25:00.280579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:56.781976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280454+0000","last_change":"2024-03-06T07:24:58.290475+0000","last_active":"2024-03-06T07:25:00.280454+0000","last_peered":"2024-03-06T07:25:00.280454+0000","last_clean":"2024-03-06T07:25:00.280454+0000","last_became_active":"2024-03-06T07:24:58.290156+0000","last_became_peered":"2024-03-06T07:24:58.290156+0000","last_unstale":"2024-03-06T07:25:00.280454+0000","last_undegraded":"2024-03-06T07:25:00.280454+0000","last_fullsized":"2024-03-06T07:25:00.280454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:22.753774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.696083+0000","last_change":"2024-03-06T07:24:59.297552+0000","last_active":"2024-03-06T07:25:00.696083+0000","last_peered":"2024-03-06T07:25:00.696083+0000","last_clean":"2024-03-06T07:25:00.696083+0000","last_became_active":"2024-03-06T07:24:59.296465+0000","last_became_peered":"2024-03-06T07:24:59.296465+0000","last_unstale":"2024-03-06T07:25:00.696083+0000","last_undegraded":"2024-03-06T07:25:00.696083+0000","last_fullsized":"2024-03-06T07:25:00.696083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:04.276418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280654+0000","last_change":"2024-03-06T07:24:58.295406+0000","last_active":"2024-03-06T07:25:00.280654+0000","last_peered":"2024-03-06T07:25:00.280654+0000","last_clean":"2024-03-06T07:25:00.280654+0000","last_became_active":"2024-03-06T07:24:58.294948+0000","last_became_peered":"2024-03-06T07:24:58.294948+0000","last_unstale":"2024-03-06T07:25:00.280654+0000","last_undegraded":"2024-03-06T07:25:00.280654+0000","last_fullsized":"2024-03-06T07:25:00.280654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:22.520184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.690139+0000","last_change":"2024-03-06T07:24:59.299022+0000","last_active":"2024-03-06T07:25:01.690139+0000","last_peered":"2024-03-06T07:25:01.690139+0000","last_clean":"2024-03-06T07:25:01.690139+0000","last_became_active":"2024-03-06T07:24:59.298613+0000","last_became_peered":"2024-03-06T07:24:59.298613+0000","last_unstale":"2024-03-06T07:25:01.690139+0000","last_undegraded":"2024-03-06T07:25:01.690139+0000","last_fullsized":"2024-03-06T07:25:01.690139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:11.046264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280658+0000","last_change":"2024-03-06T07:24:58.292520+0000","last_active":"2024-03-06T07:25:00.280658+0000","last_peered":"2024-03-06T07:25:00.280658+0000","last_clean":"2024-03-06T07:25:00.280658+0000","last_became_active":"2024-03-06T07:24:58.291165+0000","last_became_peered":"2024-03-06T07:24:58.291165+0000","last_unstale":"2024-03-06T07:25:00.280658+0000","last_undegraded":"2024-03-06T07:25:00.280658+0000","last_fullsized":"2024-03-06T07:25:00.280658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:33.891983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278808+0000","last_change":"2024-03-06T07:24:59.286682+0000","last_active":"2024-03-06T07:25:00.278808+0000","last_peered":"2024-03-06T07:25:00.278808+0000","last_clean":"2024-03-06T07:25:00.278808+0000","last_became_active":"2024-03-06T07:24:59.286403+0000","last_became_peered":"2024-03-06T07:24:59.286403+0000","last_unstale":"2024-03-06T07:25:00.278808+0000","last_undegraded":"2024-03-06T07:25:00.278808+0000","last_fullsized":"2024-03-06T07:25:00.278808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:24.753218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695606+0000","last_change":"2024-03-06T07:24:58.289658+0000","last_active":"2024-03-06T07:25:00.695606+0000","last_peered":"2024-03-06T07:25:00.695606+0000","last_clean":"2024-03-06T07:25:00.695606+0000","last_became_active":"2024-03-06T07:24:58.289381+0000","last_became_peered":"2024-03-06T07:24:58.289381+0000","last_unstale":"2024-03-06T07:25:00.695606+0000","last_undegraded":"2024-03-06T07:25:00.695606+0000","last_fullsized":"2024-03-06T07:25:00.695606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:30.699089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496542+0000","last_change":"2024-03-06T07:24:59.294536+0000","last_active":"2024-03-06T07:25:00.496542+0000","last_peered":"2024-03-06T07:25:00.496542+0000","last_clean":"2024-03-06T07:25:00.496542+0000","last_became_active":"2024-03-06T07:24:59.294221+0000","last_became_peered":"2024-03-06T07:24:59.294221+0000","last_unstale":"2024-03-06T07:25:00.496542+0000","last_undegraded":"2024-03-06T07:25:00.496542+0000","last_fullsized":"2024-03-06T07:25:00.496542+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:54.917133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280957+0000","last_change":"2024-03-06T07:24:58.286439+0000","last_active":"2024-03-06T07:25:00.280957+0000","last_peered":"2024-03-06T07:25:00.280957+0000","last_clean":"2024-03-06T07:25:00.280957+0000","last_became_active":"2024-03-06T07:24:58.285996+0000","last_became_peered":"2024-03-06T07:24:58.285996+0000","last_unstale":"2024-03-06T07:25:00.280957+0000","last_undegraded":"2024-03-06T07:25:00.280957+0000","last_fullsized":"2024-03-06T07:25:00.280957+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:03.356871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496503+0000","last_change":"2024-03-06T07:24:59.300417+0000","last_active":"2024-03-06T07:25:00.496503+0000","last_peered":"2024-03-06T07:25:00.496503+0000","last_clean":"2024-03-06T07:25:00.496503+0000","last_became_active":"2024-03-06T07:24:59.300084+0000","last_became_peered":"2024-03-06T07:24:59.300084+0000","last_unstale":"2024-03-06T07:25:00.496503+0000","last_undegraded":"2024-03-06T07:25:00.496503+0000","last_fullsized":"2024-03-06T07:25:00.496503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:51.966784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189725+0000","last_change":"2024-03-06T07:24:58.293625+0000","last_active":"2024-03-06T07:25:01.189725+0000","last_peered":"2024-03-06T07:25:01.189725+0000","last_clean":"2024-03-06T07:25:01.189725+0000","last_became_active":"2024-03-06T07:24:58.293328+0000","last_became_peered":"2024-03-06T07:24:58.293328+0000","last_unstale":"2024-03-06T07:25:01.189725+0000","last_undegraded":"2024-03-06T07:25:01.189725+0000","last_fullsized":"2024-03-06T07:25:01.189725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:49.716667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496460+0000","last_change":"2024-03-06T07:24:59.300218+0000","last_active":"2024-03-06T07:25:00.496460+0000","last_peered":"2024-03-06T07:25:00.496460+0000","last_clean":"2024-03-06T07:25:00.496460+0000","last_became_active":"2024-03-06T07:24:59.299977+0000","last_became_peered":"2024-03-06T07:24:59.299977+0000","last_unstale":"2024-03-06T07:25:00.496460+0000","last_undegraded":"2024-03-06T07:25:00.496460+0000","last_fullsized":"2024-03-06T07:25:00.496460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:02.788779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284221+0000","last_change":"2024-03-06T07:24:58.295781+0000","last_active":"2024-03-06T07:25:01.284221+0000","last_peered":"2024-03-06T07:25:01.284221+0000","last_clean":"2024-03-06T07:25:01.284221+0000","last_became_active":"2024-03-06T07:24:58.295430+0000","last_became_peered":"2024-03-06T07:24:58.295430+0000","last_unstale":"2024-03-06T07:25:01.284221+0000","last_undegraded":"2024-03-06T07:25:01.284221+0000","last_fullsized":"2024-03-06T07:25:01.284221+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:54.331409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284646+0000","last_change":"2024-03-06T07:24:59.299253+0000","last_active":"2024-03-06T07:25:01.284646+0000","last_peered":"2024-03-06T07:25:01.284646+0000","last_clean":"2024-03-06T07:25:01.284646+0000","last_became_active":"2024-03-06T07:24:59.298992+0000","last_became_peered":"2024-03-06T07:24:59.298992+0000","last_unstale":"2024-03-06T07:25:01.284646+0000","last_undegraded":"2024-03-06T07:25:01.284646+0000","last_fullsized":"2024-03-06T07:25:01.284646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:50.745366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280721+0000","last_change":"2024-03-06T07:24:58.293223+0000","last_active":"2024-03-06T07:25:00.280721+0000","last_peered":"2024-03-06T07:25:00.280721+0000","last_clean":"2024-03-06T07:25:00.280721+0000","last_became_active":"2024-03-06T07:24:58.292940+0000","last_became_peered":"2024-03-06T07:24:58.292940+0000","last_unstale":"2024-03-06T07:25:00.280721+0000","last_undegraded":"2024-03-06T07:25:00.280721+0000","last_fullsized":"2024-03-06T07:25:00.280721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:15.610913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189696+0000","last_change":"2024-03-06T07:24:59.296017+0000","last_active":"2024-03-06T07:25:01.189696+0000","last_peered":"2024-03-06T07:25:01.189696+0000","last_clean":"2024-03-06T07:25:01.189696+0000","last_became_active":"2024-03-06T07:24:59.295730+0000","last_became_peered":"2024-03-06T07:24:59.295730+0000","last_unstale":"2024-03-06T07:25:01.189696+0000","last_undegraded":"2024-03-06T07:25:01.189696+0000","last_fullsized":"2024-03-06T07:25:01.189696+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:06.435738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.189837+0000","last_change":"2024-03-06T07:24:58.290422+0000","last_active":"2024-03-06T07:25:01.189837+0000","last_peered":"2024-03-06T07:25:01.189837+0000","last_clean":"2024-03-06T07:25:01.189837+0000","last_became_active":"2024-03-06T07:24:58.290146+0000","last_became_peered":"2024-03-06T07:24:58.290146+0000","last_unstale":"2024-03-06T07:25:01.189837+0000","last_undegraded":"2024-03-06T07:25:01.189837+0000","last_fullsized":"2024-03-06T07:25:01.189837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:44.876762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695806+0000","last_change":"2024-03-06T07:24:59.297699+0000","last_active":"2024-03-06T07:25:00.695806+0000","last_peered":"2024-03-06T07:25:00.695806+0000","last_clean":"2024-03-06T07:25:00.695806+0000","last_became_active":"2024-03-06T07:24:59.297414+0000","last_became_peered":"2024-03-06T07:24:59.297414+0000","last_unstale":"2024-03-06T07:25:00.695806+0000","last_undegraded":"2024-03-06T07:25:00.695806+0000","last_fullsized":"2024-03-06T07:25:00.695806+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:24.707279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278754+0000","last_change":"2024-03-06T07:24:58.291745+0000","last_active":"2024-03-06T07:25:00.278754+0000","last_peered":"2024-03-06T07:25:00.278754+0000","last_clean":"2024-03-06T07:25:00.278754+0000","last_became_active":"2024-03-06T07:24:58.291465+0000","last_became_peered":"2024-03-06T07:24:58.291465+0000","last_unstale":"2024-03-06T07:25:00.278754+0000","last_undegraded":"2024-03-06T07:25:00.278754+0000","last_fullsized":"2024-03-06T07:25:00.278754+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:51.194465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695919+0000","last_change":"2024-03-06T07:24:59.293638+0000","last_active":"2024-03-06T07:25:00.695919+0000","last_peered":"2024-03-06T07:25:00.695919+0000","last_clean":"2024-03-06T07:25:00.695919+0000","last_became_active":"2024-03-06T07:24:59.293372+0000","last_became_peered":"2024-03-06T07:24:59.293372+0000","last_unstale":"2024-03-06T07:25:00.695919+0000","last_undegraded":"2024-03-06T07:25:00.695919+0000","last_fullsized":"2024-03-06T07:25:00.695919+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:19.927845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.284493+0000","last_change":"2024-03-06T07:24:58.292455+0000","last_active":"2024-03-06T07:25:01.284493+0000","last_peered":"2024-03-06T07:25:01.284493+0000","last_clean":"2024-03-06T07:25:01.284493+0000","last_became_active":"2024-03-06T07:24:58.292096+0000","last_became_peered":"2024-03-06T07:24:58.292096+0000","last_unstale":"2024-03-06T07:25:01.284493+0000","last_undegraded":"2024-03-06T07:25:01.284493+0000","last_fullsized":"2024-03-06T07:25:01.284493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:52.458808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279112+0000","last_change":"2024-03-06T07:24:59.293894+0000","last_active":"2024-03-06T07:25:00.279112+0000","last_peered":"2024-03-06T07:25:00.279112+0000","last_clean":"2024-03-06T07:25:00.279112+0000","last_became_active":"2024-03-06T07:24:59.293362+0000","last_became_peered":"2024-03-06T07:24:59.293362+0000","last_unstale":"2024-03-06T07:25:00.279112+0000","last_undegraded":"2024-03-06T07:25:00.279112+0000","last_fullsized":"2024-03-06T07:25:00.279112+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:34.893813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.278591+0000","last_change":"2024-03-06T07:24:58.290189+0000","last_active":"2024-03-06T07:25:00.278591+0000","last_peered":"2024-03-06T07:25:00.278591+0000","last_clean":"2024-03-06T07:25:00.278591+0000","last_became_active":"2024-03-06T07:24:58.289929+0000","last_became_peered":"2024-03-06T07:24:58.289929+0000","last_unstale":"2024-03-06T07:25:00.278591+0000","last_undegraded":"2024-03-06T07:25:00.278591+0000","last_fullsized":"2024-03-06T07:25:00.278591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:27.586473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279915+0000","last_change":"2024-03-06T07:24:59.289846+0000","last_active":"2024-03-06T07:25:00.279915+0000","last_peered":"2024-03-06T07:25:00.279915+0000","last_clean":"2024-03-06T07:25:00.279915+0000","last_became_active":"2024-03-06T07:24:59.289374+0000","last_became_peered":"2024-03-06T07:24:59.289374+0000","last_unstale":"2024-03-06T07:25:00.279915+0000","last_undegraded":"2024-03-06T07:25:00.279915+0000","last_fullsized":"2024-03-06T07:25:00.279915+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:40.334461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689745+0000","last_change":"2024-03-06T07:24:58.287733+0000","last_active":"2024-03-06T07:25:01.689745+0000","last_peered":"2024-03-06T07:25:01.689745+0000","last_clean":"2024-03-06T07:25:01.689745+0000","last_became_active":"2024-03-06T07:24:58.287475+0000","last_became_peered":"2024-03-06T07:24:58.287475+0000","last_unstale":"2024-03-06T07:25:01.689745+0000","last_undegraded":"2024-03-06T07:25:01.689745+0000","last_fullsized":"2024-03-06T07:25:01.689745+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:32.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:01.689882+0000","last_change":"2024-03-06T07:24:59.301355+0000","last_active":"2024-03-06T07:25:01.689882+0000","last_peered":"2024-03-06T07:25:01.689882+0000","last_clean":"2024-03-06T07:25:01.689882+0000","last_became_active":"2024-03-06T07:24:59.301070+0000","last_became_peered":"2024-03-06T07:24:59.301070+0000","last_unstale":"2024-03-06T07:25:01.689882+0000","last_undegraded":"2024-03-06T07:25:01.689882+0000","last_fullsized":"2024-03-06T07:25:01.689882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:08:52.968955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280117+0000","last_change":"2024-03-06T07:24:58.287276+0000","last_active":"2024-03-06T07:25:00.280117+0000","last_peered":"2024-03-06T07:25:00.280117+0000","last_clean":"2024-03-06T07:25:00.280117+0000","last_became_active":"2024-03-06T07:24:58.287015+0000","last_became_peered":"2024-03-06T07:24:58.287015+0000","last_unstale":"2024-03-06T07:25:00.280117+0000","last_undegraded":"2024-03-06T07:25:00.280117+0000","last_fullsized":"2024-03-06T07:25:00.280117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:59.713732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.280049+0000","last_change":"2024-03-06T07:24:59.293984+0000","last_active":"2024-03-06T07:25:00.280049+0000","last_peered":"2024-03-06T07:25:00.280049+0000","last_clean":"2024-03-06T07:25:00.280049+0000","last_became_active":"2024-03-06T07:24:59.293725+0000","last_became_peered":"2024-03-06T07:24:59.293725+0000","last_unstale":"2024-03-06T07:25:00.280049+0000","last_undegraded":"2024-03-06T07:25:00.280049+0000","last_fullsized":"2024-03-06T07:25:00.280049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:22.942791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.281040+0000","last_change":"2024-03-06T07:24:58.293026+0000","last_active":"2024-03-06T07:25:00.281040+0000","last_peered":"2024-03-06T07:25:00.281040+0000","last_clean":"2024-03-06T07:25:00.281040+0000","last_became_active":"2024-03-06T07:24:58.292754+0000","last_became_peered":"2024-03-06T07:24:58.292754+0000","last_unstale":"2024-03-06T07:25:00.281040+0000","last_undegraded":"2024-03-06T07:25:00.281040+0000","last_fullsized":"2024-03-06T07:25:00.281040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:25.913477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.496716+0000","last_change":"2024-03-06T07:24:59.301173+0000","last_active":"2024-03-06T07:25:00.496716+0000","last_peered":"2024-03-06T07:25:00.496716+0000","last_clean":"2024-03-06T07:25:00.496716+0000","last_became_active":"2024-03-06T07:24:59.300231+0000","last_became_peered":"2024-03-06T07:24:59.300231+0000","last_unstale":"2024-03-06T07:25:00.496716+0000","last_undegraded":"2024-03-06T07:25:00.496716+0000","last_fullsized":"2024-03-06T07:25:00.496716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:58.269595+0000","last_clean_scrub_stamp":"2024-03-06T07:24:58.269595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:58.080499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.279858+0000","last_change":"2024-03-06T07:24:58.281091+0000","last_active":"2024-03-06T07:25:00.279858+0000","last_peered":"2024-03-06T07:25:00.279858+0000","last_clean":"2024-03-06T07:25:00.279858+0000","last_became_active":"2024-03-06T07:24:58.280818+0000","last_became_peered":"2024-03-06T07:24:58.280818+0000","last_unstale":"2024-03-06T07:25:00.279858+0000","last_undegraded":"2024-03-06T07:25:00.279858+0000","last_fullsized":"2024-03-06T07:25:00.279858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:57.267765+0000","last_clean_scrub_stamp":"2024-03-06T07:24:57.267765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:32.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":256,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:00.695377+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:25:00.695377+0000","last_peered":"2024-03-06T07:25:00.695377+0000","last_clean":"2024-03-06T07:25:00.695377+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:25:00.695377+0000","last_undegraded":"2024-03-06T07:25:00.695377+0000","last_fullsized":"2024-03-06T07:25:00.695377+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1441792,"data_stored":943272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36168,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335672,"statfs":{"total":96636764160,"available":96599728128,"internally_reserved":0,"allocated":847872,"data_stored":352380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36164,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335676,"statfs":{"total":96636764160,"available":96599732224,"internally_reserved":0,"allocated":843776,"data_stored":352378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36168,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335672,"statfs":{"total":96636764160,"available":96599728128,"internally_reserved":0,"allocated":843776,"data_stored":352358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":352400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":353900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":354516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1433600,"data_stored":942656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:07.110 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:07.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.253+0000 7f315c3f7640 1 Processor -- start 2024-03-06T07:25:07.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.253+0000 7f315c3f7640 1 -- start start 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f315c3f7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f31541614d0 0x7f31540d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f315c3f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31540d24b0 0x7f31540b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f315c3f7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31541618b0 con 0x7f31540d1910 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f315c3f7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31540cf400 con 0x7f31541614d0 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f315c3f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cf6a0 con 0x7f31540d24b0 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159992640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f31540d1910 0x7f31540d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59384/0 (socket says 172.21.15.77:59384) 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159992640 1 -- 172.21.15.77:0/3725773996 learned_addr learned my addr 172.21.15.77:0/3725773996 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612877494 0 0) 0x7f31541618b0 con 0x7f31540d1910 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f315a994640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31540d24b0 0x7f31540b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f315a193640 1 --2- 172.21.15.77:0/3725773996 >> v2:172.21.15.158:3300/0 conn(0x7f31541614d0 0x7f31540d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3140003520 con 0x7f31540d1910 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 841225776 0 0) 0x7f3140003520 con 0x7f31540d1910 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31540d24b0 msgr2=0x7f31540b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31540d24b0 0x7f31540b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 >> v2:172.21.15.158:3300/0 conn(0x7f31541614d0 msgr2=0x7f31540d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 --2- 172.21.15.77:0/3725773996 >> v2:172.21.15.158:3300/0 conn(0x7f31541614d0 0x7f31540d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540cf980 con 0x7f31540d1910 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3144001670 con 0x7f31540d1910 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 >> v1:172.21.15.77:6789/0 conn(0x7f31540d1910 legacy=0x7f31540d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.254+0000 7f3159191640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31540d24b0 0x7f31400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159191640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3140003f30 0x7f3140004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159191640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3140004d40 0x7f31400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cf400 con 0x7f3140003f30 2024-03-06T07:25:07.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cf6a0 con 0x7f3140004d40 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314000b9c0 con 0x7f31540d24b0 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159992640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3140003f30 0x7f3140004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f315a193640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31540d24b0 0x7f31400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f315a994640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3140004d40 0x7f31400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159992640 1 -- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31540d24b0 msgr2=0x7f31400039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159992640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31540d24b0 0x7f31400039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159992640 1 -- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3140004d40 msgr2=0x7f31400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159992640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3140004d40 0x7f31400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f3159992640 1 -- 172.21.15.77:0/3725773996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3140003520 con 0x7f3140003f30 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.255+0000 7f315a994640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3140004d40 0x7f31400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.256+0000 7f3159992640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3140003f30 0x7f3140004800 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3144001050 tx=0x7f3140005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0bccc423323036 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.256+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144007d70 con 0x7f3140003f30 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.256+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3144030070 con 0x7f3140003f30 2024-03-06T07:25:07.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.256+0000 7f3159191640 1 -- 172.21.15.77:0/3725773996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314402cd70 con 0x7f3140003f30 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.257+0000 7f315c3f7640 1 -- 172.21.15.77:0/3725773996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3140003f30 msgr2=0x7f3140004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.257+0000 7f315c3f7640 1 --2- 172.21.15.77:0/3725773996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3140003f30 0x7f3140004800 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3144001050 tx=0x7f3140005830 comp rx=0 tx=0).stop 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.257+0000 7f315a994640 1 -- 172.21.15.77:0/3725773996 reap_dead start 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.257+0000 7f315c3f7640 1 -- 172.21.15.77:0/3725773996 shutdown_connections 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.257+0000 7f315c3f7640 1 -- 172.21.15.77:0/3725773996 >> 172.21.15.77:0/3725773996 conn(0x7f31540d3d40 msgr2=0x7f31540d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.257+0000 7f315c3f7640 1 -- 172.21.15.77:0/3725773996 shutdown_connections 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.257+0000 7f315c3f7640 1 -- 172.21.15.77:0/3725773996 wait complete. 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.258+0000 7f315c3f7640 1 Processor -- start 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.258+0000 7f315c3f7640 1 -- start start 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315c3f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3140003f30 0x7f3154150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315c3f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31540b8840 0x7f315414db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315c3f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31540b3540 0x7f31540b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315c3f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540b0e30 con 0x7f31540b8840 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315c3f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540b0910 con 0x7f31540b3540 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315c3f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540b0fb0 con 0x7f3140003f30 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31540b8840 0x7f315414db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31540b8840 0x7f315414db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44640/0 (socket says 172.21.15.77:44640) 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 -- 172.21.15.77:0/2599698188 learned_addr learned my addr 172.21.15.77:0/2599698188 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315a193640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3140003f30 0x7f3154150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315a994640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31540b3540 0x7f31540b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 -- 172.21.15.77:0/2599698188 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3140003f30 msgr2=0x7f3154150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3140003f30 0x7f3154150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 -- 172.21.15.77:0/2599698188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31540b3540 msgr2=0x7f31540b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31540b3540 0x7f31540b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 -- 172.21.15.77:0/2599698188 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540b9bf0 con 0x7f31540b8840 2024-03-06T07:25:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f3159992640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31540b8840 0x7f315414db00 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f314400c300 tx=0x7f3144000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f314affd640 1 -- 172.21.15.77:0/2599698188 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144004bf0 con 0x7f31540b8840 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f314affd640 1 -- 172.21.15.77:0/2599698188 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3144004d90 con 0x7f31540b8840 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f314affd640 1 -- 172.21.15.77:0/2599698188 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144007270 con 0x7f31540b8840 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31540b9070 con 0x7f31540b8840 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.259+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31540b9630 con 0x7f31540b8840 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.260+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31540b3f00 con 0x7f31540b8840 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.261+0000 7f314affd640 1 -- 172.21.15.77:0/2599698188 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3144047020 con 0x7f31540b8840 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.261+0000 7f314affd640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f313806b4d0 0x7f313806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.261+0000 7f314affd640 1 -- 172.21.15.77:0/2599698188 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31440ab480 con 0x7f31540b8840 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.261+0000 7f315a193640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f313806b4d0 0x7f313806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.262+0000 7f315a193640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f313806b4d0 0x7f313806d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3150009b80 tx=0x7f3150008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.266+0000 7f314affd640 1 -- 172.21.15.77:0/2599698188 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f314407a300 con 0x7f31540b8840 2024-03-06T07:25:07.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.534+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f31540d0b00 con 0x7f31540b8840 2024-03-06T07:25:07.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.536+0000 7f314affd640 1 -- 172.21.15.77:0/2599698188 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3144079c00 con 0x7f31540b8840 2024-03-06T07:25:07.538 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:25:07.538 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:25:07.538 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:25:07.538 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:25:07.538 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:07.538 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:25:07.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f313806b4d0 msgr2=0x7f313806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f313806b4d0 0x7f313806d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3150009b80 tx=0x7f3150008040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31540b8840 msgr2=0x7f315414db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31540b8840 0x7f315414db00 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f314400c300 tx=0x7f3144000fe0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 shutdown_connections 2024-03-06T07:25:07.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f313806b4d0 0x7f313806d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31540b8840 0x7f315414db00 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31540b3540 0x7f31540b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 --2- 172.21.15.77:0/2599698188 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3140003f30 0x7f3154150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 >> 172.21.15.77:0/2599698188 conn(0x7f31540d3d40 msgr2=0x7f315406c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 shutdown_connections 2024-03-06T07:25:07.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.538+0000 7f315c3f7640 1 -- 172.21.15.77:0/2599698188 wait complete. 2024-03-06T07:25:07.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:07.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.652+0000 7f45c57f7640 1 Processor -- start 2024-03-06T07:25:07.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.652+0000 7f45c57f7640 1 -- start start 2024-03-06T07:25:07.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45c57f7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f45c00ce3c0 0x7f45c006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45c57f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45c006d0d0 0x7f45c006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45c57f7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c006efb0 con 0x7f45c015ac70 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45c57f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c006b170 con 0x7f45c006d0d0 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45c57f7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c006ec60 con 0x7f45c00ce3c0 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45bf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45c006d0d0 0x7f45c006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45bf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45c006d0d0 0x7f45c006bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44644/0 (socket says 172.21.15.77:44644) 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45bf7fe640 1 -- 172.21.15.77:0/719519788 learned_addr learned my addr 172.21.15.77:0/719519788 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45be7fc640 1 --2- 172.21.15.77:0/719519788 >> v2:172.21.15.158:3300/0 conn(0x7f45c00ce3c0 0x7f45c006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45bf7fe640 1 -- 172.21.15.77:0/719519788 >> v2:172.21.15.158:3300/0 conn(0x7f45c00ce3c0 msgr2=0x7f45c006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45bf7fe640 1 --2- 172.21.15.77:0/719519788 >> v2:172.21.15.158:3300/0 conn(0x7f45c00ce3c0 0x7f45c006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45bf7fe640 1 -- 172.21.15.77:0/719519788 >> v1:172.21.15.158:6789/0 conn(0x7f45c015ac70 legacy=0x7f45c00cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.653+0000 7f45bf7fe640 1 -- 172.21.15.77:0/719519788 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c006e6e0 con 0x7f45c006d0d0 2024-03-06T07:25:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.654+0000 7f45bf7fe640 1 --2- 172.21.15.77:0/719519788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c006bb40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f45c006f180 tx=0x7f45b802e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd47489270c16550 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.654+0000 7f45bdffb640 1 -- 172.21.15.77:0/719519788 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b802ea00 con 0x7f45c006d0d0 2024-03-06T07:25:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.654+0000 7f45bdffb640 1 -- 172.21.15.77:0/719519788 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b802eba0 con 0x7f45c006d0d0 2024-03-06T07:25:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.654+0000 7f45bdffb640 1 -- 172.21.15.77:0/719519788 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b8009600 con 0x7f45c006d0d0 2024-03-06T07:25:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.655+0000 7f45c57f7640 1 -- 172.21.15.77:0/719519788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 msgr2=0x7f45c006bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.655+0000 7f45c57f7640 1 --2- 172.21.15.77:0/719519788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c006bb40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f45c006f180 tx=0x7f45b802e060 comp rx=0 tx=0).stop 2024-03-06T07:25:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.655+0000 7f45c57f7640 1 -- 172.21.15.77:0/719519788 shutdown_connections 2024-03-06T07:25:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.655+0000 7f45c57f7640 1 --2- 172.21.15.77:0/719519788 >> v2:172.21.15.158:3300/0 conn(0x7f45c00ce3c0 0x7f45c006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.655+0000 7f45c57f7640 1 --2- 172.21.15.77:0/719519788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c006bb40 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.655+0000 7f45c57f7640 1 -- 172.21.15.77:0/719519788 >> 172.21.15.77:0/719519788 conn(0x7f45c00d00e0 msgr2=0x7f45c00d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.655+0000 7f45c57f7640 1 -- 172.21.15.77:0/719519788 shutdown_connections 2024-03-06T07:25:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.655+0000 7f45c57f7640 1 -- 172.21.15.77:0/719519788 wait complete. 2024-03-06T07:25:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.656+0000 7f45c57f7640 1 Processor -- start 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.656+0000 7f45c57f7640 1 -- start start 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45c57f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c015db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45c57f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45c00ce3c0 0x7f45c014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45c57f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45c015ac70 0x7f45c009bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45c57f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c006e910 con 0x7f45c006d0d0 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45c57f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c006e400 con 0x7f45c015ac70 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45c57f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c009dec0 con 0x7f45c00ce3c0 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c015db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44652/0 (socket says 172.21.15.77:44652) 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45beffd640 1 -- 172.21.15.77:0/1559569976 learned_addr learned my addr 172.21.15.77:0/1559569976 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45be7fc640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45c00ce3c0 0x7f45c014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45bf7fe640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45c015ac70 0x7f45c009bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45beffd640 1 -- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45c00ce3c0 msgr2=0x7f45c014e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45beffd640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45c00ce3c0 0x7f45c014e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45beffd640 1 -- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45c015ac70 msgr2=0x7f45c009bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45beffd640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45c015ac70 0x7f45c009bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45beffd640 1 -- 172.21.15.77:0/1559569976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c009a9f0 con 0x7f45c006d0d0 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45bf7fe640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45c015ac70 0x7f45c009bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.657+0000 7f45be7fc640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45c00ce3c0 0x7f45c014e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.658+0000 7f45beffd640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c015db30 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f45b000c450 tx=0x7f45b000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.658+0000 7f45a77fe640 1 -- 172.21.15.77:0/1559569976 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b00052d0 con 0x7f45c006d0d0 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.658+0000 7f45a77fe640 1 -- 172.21.15.77:0/1559569976 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b0005470 con 0x7f45c006d0d0 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.658+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c009acb0 con 0x7f45c006d0d0 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.658+0000 7f45a77fe640 1 -- 172.21.15.77:0/1559569976 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b0012870 con 0x7f45c006d0d0 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.658+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c009b270 con 0x7f45c006d0d0 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.659+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c004f330 con 0x7f45c006d0d0 2024-03-06T07:25:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.659+0000 7f45a77fe640 1 -- 172.21.15.77:0/1559569976 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45b0012a10 con 0x7f45c006d0d0 2024-03-06T07:25:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.659+0000 7f45a77fe640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f45ac06b4d0 0x7f45ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.660+0000 7f45be7fc640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f45ac06b4d0 0x7f45ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.660+0000 7f45a77fe640 1 -- 172.21.15.77:0/1559569976 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45b0089e30 con 0x7f45c006d0d0 2024-03-06T07:25:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.660+0000 7f45be7fc640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f45ac06b4d0 0x7f45ac06d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f45b4006fd0 tx=0x7f45b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.664+0000 7f45a77fe640 1 -- 172.21.15.77:0/1559569976 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45b0058cb0 con 0x7f45c006d0d0 2024-03-06T07:25:07.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.876+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45c00b5fa0 con 0x7f45c006d0d0 2024-03-06T07:25:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.877+0000 7f45a77fe640 1 -- 172.21.15.77:0/1559569976 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f45b00585b0 con 0x7f45c006d0d0 2024-03-06T07:25:07.879 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:07.879 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:07.879 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.879+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f45ac06b4d0 msgr2=0x7f45ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f45ac06b4d0 0x7f45ac06d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f45b4006fd0 tx=0x7f45b4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 msgr2=0x7f45c015db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c015db30 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f45b000c450 tx=0x7f45b000c910 comp rx=0 tx=0).stop 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 shutdown_connections 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45c015ac70 0x7f45c009bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45c00ce3c0 0x7f45c014e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f45ac06b4d0 0x7f45ac06d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 --2- 172.21.15.77:0/1559569976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45c006d0d0 0x7f45c015db30 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 >> 172.21.15.77:0/1559569976 conn(0x7f45c00d00e0 msgr2=0x7f45c00724a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 shutdown_connections 2024-03-06T07:25:07.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:07.880+0000 7f45c57f7640 1 -- 172.21.15.77:0/1559569976 wait complete. 2024-03-06T07:25:08.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:58.627086+0000', 'modified': '2024-03-06T07:25:00.680393+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6507}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6507': {'gid': 6507, 'name': 'c', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/3491727415', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6835', 'nonce': 3491727415}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 3491727415}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6507, 'qdb_cluster': [6507]} 2024-03-06T07:25:08.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:08.296 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:25:08.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.395+0000 7f0f2ee4e640 1 Processor -- start 2024-03-06T07:25:08.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.395+0000 7f0f2ee4e640 1 -- start start 2024-03-06T07:25:08.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2ee4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f28162970 0x7f0f280d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2ee4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0f28161cf0 0x7f0f28162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2ee4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f280d22c0 con 0x7f0f280b4460 2024-03-06T07:25:08.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2ee4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f28161b00 con 0x7f0f28161cf0 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2ee4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280cecc0 con 0x7f0f28162970 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2d3eb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0f28161cf0 0x7f0f28162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f27fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f28162970 0x7f0f280d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2cbea640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0f280b4460 0x7f0f280b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:33040/0 (socket says 172.21.15.77:33040) 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2cbea640 1 -- 172.21.15.77:0/573144591 learned_addr learned my addr 172.21.15.77:0/573144591 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f27fff640 1 -- 172.21.15.77:0/573144591 >> v2:172.21.15.158:3300/0 conn(0x7f0f28161cf0 msgr2=0x7f0f28162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f27fff640 1 --2- 172.21.15.77:0/573144591 >> v2:172.21.15.158:3300/0 conn(0x7f0f28161cf0 0x7f0f28162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f27fff640 1 -- 172.21.15.77:0/573144591 >> v1:172.21.15.158:6789/0 conn(0x7f0f280b4460 legacy=0x7f0f280b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f27fff640 1 -- 172.21.15.77:0/573144591 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f280d25a0 con 0x7f0f28162970 2024-03-06T07:25:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.396+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/573144591 >> v2:172.21.15.158:3300/0 conn(0x7f0f28161cf0 0x7f0f28162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:08.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.397+0000 7f0f27fff640 1 --2- 172.21.15.77:0/573144591 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f28162970 0x7f0f280d4800 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0f140097a0 tx=0x7f0f1402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e14b2e3099e7af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.397+0000 7f0f277fe640 1 -- 172.21.15.77:0/573144591 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14035020 con 0x7f0f28162970 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f277fe640 1 -- 172.21.15.77:0/573144591 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14031830 con 0x7f0f28162970 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f277fe640 1 -- 172.21.15.77:0/573144591 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14031c00 con 0x7f0f28162970 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/573144591 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f28162970 msgr2=0x7f0f280d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/573144591 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f28162970 0x7f0f280d4800 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0f140097a0 tx=0x7f0f1402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/573144591 shutdown_connections 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/573144591 >> v2:172.21.15.158:3300/0 conn(0x7f0f28161cf0 0x7f0f28162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/573144591 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f28162970 0x7f0f280d4800 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/573144591 >> 172.21.15.77:0/573144591 conn(0x7f0f280d3ce0 msgr2=0x7f0f280d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/573144591 shutdown_connections 2024-03-06T07:25:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.398+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/573144591 wait complete. 2024-03-06T07:25:08.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.400+0000 7f0f2ee4e640 1 Processor -- start 2024-03-06T07:25:08.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.400+0000 7f0f2ee4e640 1 -- start start 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2ee4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f280b4460 0x7f0f280ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2ee4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f28161cf0 0x7f0f281f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2ee4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f28162970 0x7f0f280cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2ee4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280ce950 con 0x7f0f28161cf0 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2ee4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280d2b60 con 0x7f0f28162970 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2ee4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280b4840 con 0x7f0f280b4460 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f28161cf0 0x7f0f281f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f28161cf0 0x7f0f281f3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44660/0 (socket says 172.21.15.77:44660) 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 -- 172.21.15.77:0/3417269735 learned_addr learned my addr 172.21.15.77:0/3417269735 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f280b4460 0x7f0f280ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f28162970 0x7f0f280cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 -- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f280b4460 msgr2=0x7f0f280ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f280b4460 0x7f0f280ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 -- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f28162970 msgr2=0x7f0f280cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f28162970 0x7f0f280cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 -- 172.21.15.77:0/3417269735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f280cd610 con 0x7f0f28161cf0 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f280b4460 0x7f0f280ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f28162970 0x7f0f280cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f27fff640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f28161cf0 0x7f0f281f3db0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0f140098c0 tx=0x7f0f140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f257fa640 1 -- 172.21.15.77:0/3417269735 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14041070 con 0x7f0f28161cf0 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f257fa640 1 -- 172.21.15.77:0/3417269735 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1403b880 con 0x7f0f28161cf0 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/3417269735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f281f44e0 con 0x7f0f28161cf0 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f257fa640 1 -- 172.21.15.77:0/3417269735 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14035040 con 0x7f0f28161cf0 2024-03-06T07:25:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.402+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/3417269735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f281f4aa0 con 0x7f0f28161cf0 2024-03-06T07:25:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.404+0000 7f0f257fa640 1 -- 172.21.15.77:0/3417269735 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f1404c060 con 0x7f0f28161cf0 2024-03-06T07:25:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.405+0000 7f0f257fa640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.405+0000 7f0f257fa640 1 -- 172.21.15.77:0/3417269735 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f140abca0 con 0x7f0f28161cf0 2024-03-06T07:25:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.405+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.405+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0f280cbfc0 tx=0x7f0f1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.405+0000 7f0f31a36740 1 Processor -- start 2024-03-06T07:25:08.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.405+0000 7f0f31a36740 1 -- start start 2024-03-06T07:25:08.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99506870 0x55ff99506c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff9950a8c0 0x55ff9950c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 0x55ff99536a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f31a36740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff994706b0 con 0x55ff99506870 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f31a36740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff99419a60 con 0x55ff99536600 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f31a36740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff9946c750 con 0x55ff9950a8c0 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99506870 0x55ff99506c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99506870 0x55ff99506c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44672/0 (socket says 172.21.15.77:44672) 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 -- 172.21.15.77:0/2011769729 learned_addr learned my addr 172.21.15.77:0/2011769729 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 -- 172.21.15.77:0/2011769729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff9950a8c0 msgr2=0x55ff9950c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/2011769729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff9950a8c0 0x55ff9950c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 -- 172.21.15.77:0/2011769729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 msgr2=0x55ff99536a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/2011769729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 0x55ff99536a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 -- 172.21.15.77:0/2011769729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff99437f30 con 0x55ff99506870 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.406+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/2011769729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99506870 0x55ff99506c50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c005cc0 tx=0x7f0f1c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c69f185ed37c188 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f127fc640 1 -- 172.21.15.77:0/2011769729 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c00d990 con 0x55ff99506870 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f127fc640 1 -- 172.21.15.77:0/2011769729 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c00db30 con 0x55ff99506870 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f127fc640 1 -- 172.21.15.77:0/2011769729 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c020450 con 0x55ff99506870 2024-03-06T07:25:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 -- 172.21.15.77:0/2011769729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99506870 msgr2=0x55ff99506c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2011769729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99506870 0x55ff99506c50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c005cc0 tx=0x7f0f1c010000 comp rx=0 tx=0).stop 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 -- 172.21.15.77:0/2011769729 shutdown_connections 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2011769729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 0x55ff99536a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2011769729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff9950a8c0 0x55ff9950c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2011769729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99506870 0x55ff99506c50 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 -- 172.21.15.77:0/2011769729 >> 172.21.15.77:0/2011769729 conn(0x55ff995088f0 msgr2=0x55ff9950a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 -- 172.21.15.77:0/2011769729 shutdown_connections 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.407+0000 7f0f31a36740 1 -- 172.21.15.77:0/2011769729 wait complete. 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.408+0000 7f0f31a36740 1 -- start start 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.408+0000 7f0f31a36740 1 Processor -- start 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff99536600 0x55ff99537e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff9952e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff995296c0 0x55ff9952eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f31a36740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9950ad50 con 0x55ff995296c0 2024-03-06T07:25:08.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f31a36740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9942cea0 con 0x55ff99528ce0 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f31a36740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff9950b030 con 0x55ff99536600 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f31a36740 10 client.0 _ll_register_callbacks cb 0x55ff9942c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff995296c0 0x55ff9952eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff995296c0 0x55ff9952eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44674/0 (socket says 172.21.15.77:44674) 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 -- 172.21.15.77:0/2091697240 learned_addr learned my addr 172.21.15.77:0/2091697240 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff99536600 0x55ff99537e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f27fff640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff9952e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 -- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff99536600 msgr2=0x55ff99537e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff99536600 0x55ff99537e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 -- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 msgr2=0x55ff9952e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff9952e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 -- 172.21.15.77:0/2091697240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff9942d270 con 0x55ff995296c0 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff99536600 0x55ff99537e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f27fff640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff9952e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:08.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff995296c0 0x55ff9952eaf0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0f20007e10 tx=0x7f0f2000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1567b11e403aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.409+0000 7f0f077fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f077fe640 1 -- 172.21.15.77:0/2091697240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000fd50 con 0x55ff995296c0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f077fe640 1 -- 172.21.15.77:0/2091697240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f20016420 con 0x55ff995296c0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 1 -- 172.21.15.77:0/2091697240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f1c00db30 con 0x55ff995296c0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f077fe640 1 -- 172.21.15.77:0/2091697240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f20016620 con 0x55ff995296c0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 10 client.6672 fetch_fsmap learned FSMap version 33 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 1 -- 172.21.15.77:0/2091697240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55ff991852d0 con 0x55ff995296c0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f077fe640 1 -- 172.21.15.77:0/2091697240 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c00db30 con 0x55ff995296c0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f077fe640 1 -- 172.21.15.77:0/2091697240 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0f20026080 con 0x55ff995296c0 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 10 client.6672 fetch_fsmap finished waiting for FSMap version 33 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 10 client.6672 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 20 client.6672 populate_metadata read hostname 'smithi077' 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] conn(0x55ff995329a0 0x55ff99570710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 4 client.6672 mds_command: new command op to 6507 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f31a36740 1 -- 172.21.15.77:0/2091697240 --> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff9947dbf0 con 0x55ff995329a0 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.410+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] conn(0x55ff995329a0 0x55ff99570710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f077fe640 10 client.6672 ms_handle_connect on v2:172.21.15.77:6835/3491727415 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] conn(0x55ff995329a0 0x55ff99570710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70b1d3fa12e1f01b server_cookie=5c913c3c13cfc48a in_seq=0 out_seq=0 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f077fe640 1 -- 172.21.15.77:0/2091697240 <== mds.0 v2:172.21.15.77:6835/3491727415 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ff9947dbf0 con 0x55ff995329a0 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f077fe640 10 client.6672 handle_command_reply: tid=0 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 client.6672 shutdown 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 -- 172.21.15.77:0/2091697240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff995296c0 msgr2=0x55ff9952eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff995296c0 0x55ff9952eaf0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0f20007e10 tx=0x7f0f2000f520 comp rx=0 tx=0).stop 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 -- 172.21.15.77:0/2091697240 shutdown_connections 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] conn(0x55ff995329a0 0x55ff99570710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff995296c0 0x55ff9952eaf0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f077fe640 0 client.6672 ms_handle_reset on v2:172.21.15.77:6835/3491727415 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff9952e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 --2- 172.21.15.77:0/2091697240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff99536600 0x55ff99537e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 -- 172.21.15.77:0/2091697240 >> 172.21.15.77:0/2091697240 conn(0x55ff99506870 msgr2=0x55ff995345c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 -- 172.21.15.77:0/2091697240 shutdown_connections 2024-03-06T07:25:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.411+0000 7f0f31a36740 1 -- 172.21.15.77:0/2091697240 wait complete. 2024-03-06T07:25:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.412+0000 7f0f31a36740 20 client.6672 trim_cache size 0 max 16384 2024-03-06T07:25:08.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.428+0000 7f0f31a36740 1 Processor -- start 2024-03-06T07:25:08.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.428+0000 7f0f31a36740 1 -- start start 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff99532f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995354e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99536600 0x55ff99535a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f31a36740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280c9620 con 0x55ff99536600 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f31a36740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280c8a20 con 0x55ff99528ce0 2024-03-06T07:25:08.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f31a36740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280c9020 con 0x55ff995296c0 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99536600 0x55ff99535a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99536600 0x55ff99535a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44684/0 (socket says 172.21.15.77:44684) 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 -- 172.21.15.77:0/3024963767 learned_addr learned my addr 172.21.15.77:0/3024963767 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff99532f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f27fff640 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 -- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 msgr2=0x55ff995354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995354e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 -- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 msgr2=0x55ff99532f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff99532f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 -- 172.21.15.77:0/3024963767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff9924f6b0 con 0x55ff99536600 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff99532f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f27fff640 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995354e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99536600 0x55ff99535a20 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0f200095c0 tx=0x7f0f20014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6986a9826c0304 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f11ffb640 1 -- 172.21.15.77:0/3024963767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000fc20 con 0x55ff99536600 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f11ffb640 1 -- 172.21.15.77:0/3024963767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2001b070 con 0x55ff99536600 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.429+0000 7f0f11ffb640 1 -- 172.21.15.77:0/3024963767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20016ce0 con 0x55ff99536600 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 -- 172.21.15.77:0/3024963767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99536600 msgr2=0x55ff99535a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99536600 0x55ff99535a20 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0f200095c0 tx=0x7f0f20014000 comp rx=0 tx=0).stop 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 -- 172.21.15.77:0/3024963767 shutdown_connections 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99536600 0x55ff99535a20 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995354e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 --2- 172.21.15.77:0/3024963767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99528ce0 0x55ff99532f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 -- 172.21.15.77:0/3024963767 >> 172.21.15.77:0/3024963767 conn(0x55ff99506870 msgr2=0x55ff9950af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 -- 172.21.15.77:0/3024963767 shutdown_connections 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 -- 172.21.15.77:0/3024963767 wait complete. 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.430+0000 7f0f31a36740 1 -- start start 2024-03-06T07:25:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f31a36740 1 Processor -- start 2024-03-06T07:25:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99528ce0 0x55ff995c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995c4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f31a36740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 0x55ff995c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f31a36740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff99538470 con 0x55ff99528ce0 2024-03-06T07:25:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f31a36740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9952de10 con 0x55ff99536600 2024-03-06T07:25:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f31a36740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff9952ce10 con 0x55ff995296c0 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f31a36740 10 client.0 _ll_register_callbacks cb 0x7f0f200076d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f2cbea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99528ce0 0x55ff995c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f2cbea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99528ce0 0x55ff995c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44700/0 (socket says 172.21.15.77:44700) 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.431+0000 7f0f2cbea640 1 -- 172.21.15.77:0/1128546590 learned_addr learned my addr 172.21.15.77:0/1128546590 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 0x55ff995c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f27fff640 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f2cbea640 1 -- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 msgr2=0x55ff995c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995c4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f2cbea640 1 -- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 msgr2=0x55ff995c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 0x55ff995c4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f2cbea640 1 -- 172.21.15.77:0/1128546590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff99570ba0 con 0x55ff99528ce0 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f2d3eb640 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 0x55ff995c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f2cbea640 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99528ce0 0x55ff995c3f80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x55ff9953acf0 tx=0x7f0f1c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5143d66434527edc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f077fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:25:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f077fe640 1 -- 172.21.15.77:0/1128546590 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c036060 con 0x55ff99528ce0 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f077fe640 1 -- 172.21.15.77:0/1128546590 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f1c007560 con 0x55ff99528ce0 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 1 -- 172.21.15.77:0/1128546590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f2001b070 con 0x55ff99528ce0 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f077fe640 1 -- 172.21.15.77:0/1128546590 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f1c03c050 con 0x55ff99528ce0 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 10 client.6678 fetch_fsmap learned FSMap version 33 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 1 -- 172.21.15.77:0/1128546590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55ff99570db0 con 0x55ff99528ce0 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f077fe640 1 -- 172.21.15.77:0/1128546590 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2001b070 con 0x55ff99528ce0 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f077fe640 1 -- 172.21.15.77:0/1128546590 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0f1c03f020 con 0x55ff99528ce0 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 10 client.6678 fetch_fsmap finished waiting for FSMap version 33 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 10 client.6678 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 20 client.6678 populate_metadata read hostname 'smithi077' 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] conn(0x55ff99505b10 0x55ff99505ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 4 client.6678 mds_command: new command op to 6507 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.432+0000 7f0f31a36740 1 -- 172.21.15.77:0/1128546590 --> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ff99507630 con 0x55ff99505b10 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f27fff640 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] conn(0x55ff99505b10 0x55ff99505ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f27fff640 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] conn(0x55ff99505b10 0x55ff99505ef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ca66f18b2f6a9bd server_cookie=923550c98bc2d843 in_seq=0 out_seq=0 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f077fe640 10 client.6678 ms_handle_connect on v2:172.21.15.77:6835/3491727415 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f077fe640 1 -- 172.21.15.77:0/1128546590 <== mds.0 v2:172.21.15.77:6835/3491727415 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55ff99507630 con 0x55ff99505b10 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f077fe640 10 client.6678 handle_command_reply: tid=0 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 client.6678 shutdown 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 -- 172.21.15.77:0/1128546590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99528ce0 msgr2=0x55ff995c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99528ce0 0x55ff995c3f80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x55ff9953acf0 tx=0x7f0f1c000bc0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 -- 172.21.15.77:0/1128546590 shutdown_connections 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.77:6835/3491727415,v1:172.21.15.77:6837/3491727415] conn(0x55ff99505b10 0x55ff99505ef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55ff99536600 0x55ff995c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f077fe640 0 client.6678 ms_handle_reset on v2:172.21.15.77:6835/3491727415 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55ff995296c0 0x55ff995c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 --2- 172.21.15.77:0/1128546590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff99528ce0 0x55ff995c3f80 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 -- 172.21.15.77:0/1128546590 >> 172.21.15.77:0/1128546590 conn(0x55ff99506870 msgr2=0x55ff9950af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 -- 172.21.15.77:0/1128546590 shutdown_connections 2024-03-06T07:25:08.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 1 -- 172.21.15.77:0/1128546590 wait complete. 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.433+0000 7f0f31a36740 20 client.6678 trim_cache size 0 max 16384 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 6507, 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 9.1583841130000003, 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 33, 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 69, 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 69, 2024-03-06T07:25:08.442 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 14.187300976 2024-03-06T07:25:08.443 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:25:08.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0c06b4d0 msgr2=0x7f0f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0f280cbfc0 tx=0x7f0f1c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:08.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f28161cf0 msgr2=0x7f0f281f3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f28161cf0 0x7f0f281f3db0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0f140098c0 tx=0x7f0f140046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/3417269735 shutdown_connections 2024-03-06T07:25:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f28161cf0 0x7f0f281f3db0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f28162970 0x7f0f280cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 --2- 172.21.15.77:0/3417269735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f280b4460 0x7f0f280ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/3417269735 >> 172.21.15.77:0/3417269735 conn(0x7f0f280d3ce0 msgr2=0x7f0f2815f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/3417269735 shutdown_connections 2024-03-06T07:25:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.434+0000 7f0f2ee4e640 1 -- 172.21.15.77:0/3417269735 wait complete. 2024-03-06T07:25:08.447 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:08.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.555+0000 7fee7564e640 1 Processor -- start 2024-03-06T07:25:08.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.555+0000 7fee7564e640 1 -- start start 2024-03-06T07:25:08.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee7564e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fee700b4170 0x7fee700b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee7564e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fee700d0a80 0x7fee700cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee7564e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee700d25d0 con 0x7fee700d4760 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee7564e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700d4bf0 con 0x7fee700d0a80 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee7564e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700ce9e0 con 0x7fee700b4170 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee6f7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fee700d0a80 0x7fee700cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee67fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fee700d4760 0x7fee700d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:36568/0 (socket says 172.21.15.77:36568) 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee67fff640 1 -- 172.21.15.77:0/2883694374 learned_addr learned my addr 172.21.15.77:0/2883694374 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.556+0000 7fee6effd640 1 --2- 172.21.15.77:0/2883694374 >> v2:172.21.15.158:3300/0 conn(0x7fee700b4170 0x7fee700b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.557+0000 7fee6f7fe640 1 -- 172.21.15.77:0/2883694374 >> v2:172.21.15.158:3300/0 conn(0x7fee700b4170 msgr2=0x7fee700b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.557+0000 7fee6f7fe640 1 --2- 172.21.15.77:0/2883694374 >> v2:172.21.15.158:3300/0 conn(0x7fee700b4170 0x7fee700b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.557+0000 7fee6f7fe640 1 -- 172.21.15.77:0/2883694374 >> v1:172.21.15.77:6789/0 conn(0x7fee700d4760 legacy=0x7fee700d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.557+0000 7fee6e7fc640 1 -- 172.21.15.77:0/2883694374 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545050406 0 0) 0x7fee700d25d0 con 0x7fee700d4760 2024-03-06T07:25:08.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.557+0000 7fee6f7fe640 1 -- 172.21.15.77:0/2883694374 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700d2870 con 0x7fee700d0a80 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.558+0000 7fee6f7fe640 1 --2- 172.21.15.77:0/2883694374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700cee70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fee7006e030 tx=0x7fee6802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a007ed65d0c8412e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.558+0000 7fee6e7fc640 1 -- 172.21.15.77:0/2883694374 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68039070 con 0x7fee700d0a80 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.558+0000 7fee6e7fc640 1 -- 172.21.15.77:0/2883694374 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6802cc80 con 0x7fee700d0a80 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.558+0000 7fee6e7fc640 1 -- 172.21.15.77:0/2883694374 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68034660 con 0x7fee700d0a80 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.558+0000 7fee7564e640 1 -- 172.21.15.77:0/2883694374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 msgr2=0x7fee700cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.558+0000 7fee7564e640 1 --2- 172.21.15.77:0/2883694374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700cee70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fee7006e030 tx=0x7fee6802bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.559+0000 7fee7564e640 1 -- 172.21.15.77:0/2883694374 shutdown_connections 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.559+0000 7fee7564e640 1 --2- 172.21.15.77:0/2883694374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700cee70 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.559+0000 7fee7564e640 1 --2- 172.21.15.77:0/2883694374 >> v2:172.21.15.158:3300/0 conn(0x7fee700b4170 0x7fee700b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.559+0000 7fee7564e640 1 -- 172.21.15.77:0/2883694374 >> 172.21.15.77:0/2883694374 conn(0x7fee700d3c40 msgr2=0x7fee700d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.559+0000 7fee7564e640 1 -- 172.21.15.77:0/2883694374 shutdown_connections 2024-03-06T07:25:08.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.559+0000 7fee7564e640 1 -- 172.21.15.77:0/2883694374 wait complete. 2024-03-06T07:25:08.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.560+0000 7fee7564e640 1 Processor -- start 2024-03-06T07:25:08.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.560+0000 7fee7564e640 1 -- start start 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee7564e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fee700b4170 0x7fee701500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee7564e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee7564e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee700b30d0 0x7fee700b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee7564e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700b4dd0 con 0x7fee700d0a80 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee7564e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700b4550 con 0x7fee700b30d0 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee7564e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700b4f50 con 0x7fee700b4170 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700b2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56882/0 (socket says 172.21.15.77:56882) 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 -- 172.21.15.77:0/2243353161 learned_addr learned my addr 172.21.15.77:0/2243353161 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee6effd640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fee700b4170 0x7fee701500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee6f7fe640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee700b30d0 0x7fee700b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 -- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fee700b4170 msgr2=0x7fee701500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fee700b4170 0x7fee701500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 -- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee700b30d0 msgr2=0x7fee700b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee700b30d0 0x7fee700b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 -- 172.21.15.77:0/2243353161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700b14b0 con 0x7fee700d0a80 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee6effd640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fee700b4170 0x7fee701500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee6f7fe640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee700b30d0 0x7fee700b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee67fff640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700b2b90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c460 tx=0x7fee5800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee677fe640 1 -- 172.21.15.77:0/2243353161 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58018070 con 0x7fee700d0a80 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee677fe640 1 -- 172.21.15.77:0/2243353161 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58007500 con 0x7fee700d0a80 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee677fe640 1 -- 172.21.15.77:0/2243353161 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58004a20 con 0x7fee700d0a80 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee700b1790 con 0x7fee700d0a80 2024-03-06T07:25:08.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.562+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee700b1d50 con 0x7fee700d0a80 2024-03-06T07:25:08.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.563+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee700b34e0 con 0x7fee700d0a80 2024-03-06T07:25:08.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.564+0000 7fee677fe640 1 -- 172.21.15.77:0/2243353161 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee5801b070 con 0x7fee700d0a80 2024-03-06T07:25:08.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.564+0000 7fee677fe640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fee5006b4d0 0x7fee5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.564+0000 7fee677fe640 1 -- 172.21.15.77:0/2243353161 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee580894c0 con 0x7fee700d0a80 2024-03-06T07:25:08.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.564+0000 7fee6effd640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fee5006b4d0 0x7fee5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.565+0000 7fee6effd640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fee5006b4d0 0x7fee5006d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fee60006fd0 tx=0x7fee60008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.568+0000 7fee677fe640 1 -- 172.21.15.77:0/2243353161 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee700b34e0 con 0x7fee700d0a80 2024-03-06T07:25:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.784+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fee7004f330 con 0x7fee700d0a80 2024-03-06T07:25:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.784+0000 7fee677fe640 1 -- 172.21.15.77:0/2243353161 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7fee58058340 con 0x7fee700d0a80 2024-03-06T07:25:08.786 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fee5006b4d0 msgr2=0x7fee5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fee5006b4d0 0x7fee5006d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fee60006fd0 tx=0x7fee60008040 comp rx=0 tx=0).stop 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 msgr2=0x7fee700b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700b2b90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c460 tx=0x7fee5800c920 comp rx=0 tx=0).stop 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 shutdown_connections 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fee5006b4d0 0x7fee5006d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee700d0a80 0x7fee700b2b90 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fee700b30d0 0x7fee700b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 --2- 172.21.15.77:0/2243353161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fee700b4170 0x7fee701500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.786+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 >> 172.21.15.77:0/2243353161 conn(0x7fee700d3c40 msgr2=0x7fee700b7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.787+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 shutdown_connections 2024-03-06T07:25:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:08.787+0000 7fee7564e640 1 -- 172.21.15.77:0/2243353161 wait complete. 2024-03-06T07:25:08.799 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:08.799 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:08.799 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:25:08.799 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:08.799 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:08.799 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:08.799 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:08.799 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:08.800 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:08.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:08.800 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:25:08.815 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:08.815 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:08.815 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:08.815 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:08.815 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:25:08.816 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42488sec preferred_lft 42488sec 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:08.817 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:08.818 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:25:08.818 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:08.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:08.818 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:08.880 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:25:08.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:08.883 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:25:08.938 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:25:08.938 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:08.938 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:25:08.938 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:25:08.938 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:08.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:08.938 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:25:08.938 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:25:08.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:25:08.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:25:08.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:25:08.939 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:25:09.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:09.033 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:25:09.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:09.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:09.047 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:09.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:09.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:25:09.114 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:09.114 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:09.114 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:09.169 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:09.169 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:09.169 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:09.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:25:09.290 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:09.347 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:09.348 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:09.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:09.349 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:25:09.403 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:09.403 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:09.445 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:25:09.471 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:09.527 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:25:09.527 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:25:09.527 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:25:09.527 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:25:09.527 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:25:09.527 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:25:09.527 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:25:09.527 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:09.528 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:09.529 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:09.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:09.530 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:25:09.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:25:09.533+0000 7fb3cf441480 -1 init, newargv = 0x55b14b48d800 newargc=15 2024-03-06T07:25:09.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[62453]: starting ceph client 2024-03-06T07:25:09.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[62453]: starting fuse 2024-03-06T07:25:09.550 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:25:09.550 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:09.551 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:09.552 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:25:09.671 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.62453.asok 2024-03-06T07:25:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:09.683 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62453.asok 2024-03-06T07:25:09.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:09.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62453.asok status 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "62453", 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 6693, 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 6693 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3626417177 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3626417177 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.6693 192.168.0.1:0/3626417177", 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/3626417177", 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 33, 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 69, 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:25:09.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:09.828 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:25:09.828 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:09.843 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:25:09.843 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:09.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:09.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:09.915 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:25:10.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.060+0000 7f452fbc7640 1 Processor -- start 2024-03-06T07:25:10.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.060+0000 7f452fbc7640 1 -- start start 2024-03-06T07:25:10.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452fbc7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45281614d0 0x7f45280d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452fbc7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45280d1910 0x7f45280d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452fbc7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f45280d22b0 0x7f45280d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452fbc7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45280d2740 con 0x7f45281614d0 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452fbc7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45281618b0 con 0x7f45280d22b0 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452fbc7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45280cf400 con 0x7f45280d1910 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452d963640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45281614d0 0x7f45280d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452d963640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45281614d0 0x7f45280d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56890/0 (socket says 172.21.15.77:56890) 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.061+0000 7f452d963640 1 -- 172.21.15.77:0/3991432938 learned_addr learned my addr 172.21.15.77:0/3991432938 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452d162640 1 --2- 172.21.15.77:0/3991432938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45280d1910 0x7f45280d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452e164640 1 --2- 172.21.15.77:0/3991432938 >> v2:172.21.15.158:3300/0 conn(0x7f45280d22b0 0x7f45280d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452d963640 1 -- 172.21.15.77:0/3991432938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45280d1910 msgr2=0x7f45280d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452d963640 1 --2- 172.21.15.77:0/3991432938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45280d1910 0x7f45280d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452d963640 1 -- 172.21.15.77:0/3991432938 >> v2:172.21.15.158:3300/0 conn(0x7f45280d22b0 msgr2=0x7f45280d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:10.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452d963640 1 --2- 172.21.15.77:0/3991432938 >> v2:172.21.15.158:3300/0 conn(0x7f45280d22b0 0x7f45280d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452d963640 1 -- 172.21.15.77:0/3991432938 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45280cf6a0 con 0x7f45281614d0 2024-03-06T07:25:10.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452e164640 1 --2- 172.21.15.77:0/3991432938 >> v2:172.21.15.158:3300/0 conn(0x7f45280d22b0 0x7f45280d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:10.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.062+0000 7f452d963640 1 --2- 172.21.15.77:0/3991432938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45281614d0 0x7f45280d4860 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f45280b57d0 tx=0x7f451802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da403df5f64ccc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452c961640 1 -- 172.21.15.77:0/3991432938 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518035020 con 0x7f45281614d0 2024-03-06T07:25:10.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452c961640 1 -- 172.21.15.77:0/3991432938 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4518031870 con 0x7f45281614d0 2024-03-06T07:25:10.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452c961640 1 -- 172.21.15.77:0/3991432938 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518031c40 con 0x7f45281614d0 2024-03-06T07:25:10.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 -- 172.21.15.77:0/3991432938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45281614d0 msgr2=0x7f45280d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3991432938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45281614d0 0x7f45280d4860 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f45280b57d0 tx=0x7f451802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 -- 172.21.15.77:0/3991432938 shutdown_connections 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3991432938 >> v2:172.21.15.158:3300/0 conn(0x7f45280d22b0 0x7f45280d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3991432938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45280d1910 0x7f45280d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3991432938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45281614d0 0x7f45280d4860 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 -- 172.21.15.77:0/3991432938 >> 172.21.15.77:0/3991432938 conn(0x7f45280d3d40 msgr2=0x7f45280d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 -- 172.21.15.77:0/3991432938 shutdown_connections 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.063+0000 7f452fbc7640 1 -- 172.21.15.77:0/3991432938 wait complete. 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.064+0000 7f452fbc7640 1 Processor -- start 2024-03-06T07:25:10.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.064+0000 7f452fbc7640 1 -- start start 2024-03-06T07:25:10.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452fbc7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45280d1910 0x7f45280b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452fbc7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45280d22b0 0x7f452815b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452fbc7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45281614d0 0x7f452815ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452fbc7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45280b0910 con 0x7f45280d22b0 2024-03-06T07:25:10.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452fbc7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45280cfc60 con 0x7f45280d1910 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452fbc7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45280d0be0 con 0x7f45281614d0 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452d162640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45280d22b0 0x7f452815b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452d162640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45280d22b0 0x7f452815b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56902/0 (socket says 172.21.15.77:56902) 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452d162640 1 -- 172.21.15.77:0/3872510556 learned_addr learned my addr 172.21.15.77:0/3872510556 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452d963640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45280d1910 0x7f45280b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452d162640 1 -- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45281614d0 msgr2=0x7f452815ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.065+0000 7f452e164640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45281614d0 0x7f452815ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f452d162640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45281614d0 0x7f452815ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f452d162640 1 -- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45280d1910 msgr2=0x7f45280b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f452d162640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45280d1910 0x7f45280b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f452d162640 1 -- 172.21.15.77:0/3872510556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45280b6f10 con 0x7f45280d22b0 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f452d963640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45280d1910 0x7f45280b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f452d162640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45280d22b0 0x7f452815b120 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f452400c7f0 tx=0x7f452400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f451e7fc640 1 -- 172.21.15.77:0/3872510556 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524015070 con 0x7f45280d22b0 2024-03-06T07:25:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f451e7fc640 1 -- 172.21.15.77:0/3872510556 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4524004240 con 0x7f45280d22b0 2024-03-06T07:25:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45280b71f0 con 0x7f45280d22b0 2024-03-06T07:25:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f451e7fc640 1 -- 172.21.15.77:0/3872510556 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524007620 con 0x7f45280d22b0 2024-03-06T07:25:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.066+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45280b77b0 con 0x7f45280d22b0 2024-03-06T07:25:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.067+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4528068ff0 con 0x7f45280d22b0 2024-03-06T07:25:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.068+0000 7f451e7fc640 1 -- 172.21.15.77:0/3872510556 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45240079d0 con 0x7f45280d22b0 2024-03-06T07:25:10.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.068+0000 7f451e7fc640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f450c06b4d0 0x7f450c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:10.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.068+0000 7f451e7fc640 1 -- 172.21.15.77:0/3872510556 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f452408a370 con 0x7f45280d22b0 2024-03-06T07:25:10.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.068+0000 7f452d963640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f450c06b4d0 0x7f450c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:10.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.069+0000 7f452d963640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f450c06b4d0 0x7f450c06d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4518036000 tx=0x7f4518008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:10.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.077+0000 7f451e7fc640 1 -- 172.21.15.77:0/3872510556 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4524059150 con 0x7f45280d22b0 2024-03-06T07:25:10.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.254+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f45280d4290 con 0x7f45280d22b0 2024-03-06T07:25:10.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.258+0000 7f451e7fc640 1 -- 172.21.15.77:0/3872510556 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4524058a50 con 0x7f45280d22b0 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f450c06b4d0 msgr2=0x7f450c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f450c06b4d0 0x7f450c06d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4518036000 tx=0x7f4518008030 comp rx=0 tx=0).stop 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45280d22b0 msgr2=0x7f452815b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45280d22b0 0x7f452815b120 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f452400c7f0 tx=0x7f452400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 shutdown_connections 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f450c06b4d0 0x7f450c06d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45280d22b0 0x7f452815b120 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45281614d0 0x7f452815ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 --2- 172.21.15.77:0/3872510556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45280d1910 0x7f45280b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 >> 172.21.15.77:0/3872510556 conn(0x7f45280d3d40 msgr2=0x7f45280cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 shutdown_connections 2024-03-06T07:25:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:10.260+0000 7f452fbc7640 1 -- 172.21.15.77:0/3872510556 wait complete. 2024-03-06T07:25:20.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:20.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.397+0000 7f69d9e9a640 1 Processor -- start 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.397+0000 7f69d9e9a640 1 -- start start 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.397+0000 7f69d9e9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d9e9a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69d415b430 0x7f69d4071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d9e9a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69d415b8b0 con 0x7f69d40cd7a0 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d9e9a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d4070a70 con 0x7f69d415b430 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d9e9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d4070d50 con 0x7f69d41515e0 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d3fff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f69d40cd7a0 0x7f69d40cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:37860/0 (socket says 172.21.15.77:37860) 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d2ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69d415b430 0x7f69d4071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d3fff640 1 -- 172.21.15.77:0/2062720467 learned_addr learned my addr 172.21.15.77:0/2062720467 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d2ffd640 1 -- 172.21.15.77:0/2062720467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 msgr2=0x7f69d40cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/2062720467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d40cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d2ffd640 1 -- 172.21.15.77:0/2062720467 >> v1:172.21.15.158:6789/0 conn(0x7f69d40cd7a0 legacy=0x7f69d40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d2ffd640 1 -- 172.21.15.77:0/2062720467 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d4071030 con 0x7f69d415b430 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.398+0000 7f69d37fe640 1 --2- 172.21.15.77:0/2062720467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d40cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:20.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.399+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/2062720467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 0x7f69d4071830 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f69c80097a0 tx=0x7f69c802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e04d7a23fbe930f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.399+0000 7f69d27fc640 1 -- 172.21.15.77:0/2062720467 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8035020 con 0x7f69d415b430 2024-03-06T07:25:20.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.399+0000 7f69d27fc640 1 -- 172.21.15.77:0/2062720467 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c8031830 con 0x7f69d415b430 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.399+0000 7f69d27fc640 1 -- 172.21.15.77:0/2062720467 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8031c00 con 0x7f69d415b430 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.400+0000 7f69d9e9a640 1 -- 172.21.15.77:0/2062720467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 msgr2=0x7f69d4071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.400+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/2062720467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 0x7f69d4071830 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f69c80097a0 tx=0x7f69c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.400+0000 7f69d9e9a640 1 -- 172.21.15.77:0/2062720467 shutdown_connections 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.400+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/2062720467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 0x7f69d4071830 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.400+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/2062720467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d40cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.400+0000 7f69d9e9a640 1 -- 172.21.15.77:0/2062720467 >> 172.21.15.77:0/2062720467 conn(0x7f69d40cf8a0 msgr2=0x7f69d40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.400+0000 7f69d9e9a640 1 -- 172.21.15.77:0/2062720467 shutdown_connections 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.400+0000 7f69d9e9a640 1 -- 172.21.15.77:0/2062720467 wait complete. 2024-03-06T07:25:20.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.401+0000 7f69d9e9a640 1 Processor -- start 2024-03-06T07:25:20.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.401+0000 7f69d9e9a640 1 -- start start 2024-03-06T07:25:20.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d9e9a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d40cd7a0 0x7f69d4152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d9e9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d414d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d9e9a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 0x7f69d4166110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d9e9a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d4071ef0 con 0x7f69d40cd7a0 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d9e9a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d4071d70 con 0x7f69d415b430 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d9e9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d40cc3a0 con 0x7f69d41515e0 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d40cd7a0 0x7f69d4152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d40cd7a0 0x7f69d4152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46030/0 (socket says 172.21.15.77:46030) 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d37fe640 1 -- 172.21.15.77:0/3338507636 learned_addr learned my addr 172.21.15.77:0/3338507636 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d414d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d3fff640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 0x7f69d4166110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d37fe640 1 -- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 msgr2=0x7f69d414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d37fe640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d414d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d37fe640 1 -- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 msgr2=0x7f69d4166110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d37fe640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 0x7f69d4166110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d37fe640 1 -- 172.21.15.77:0/3338507636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d414dce0 con 0x7f69d40cd7a0 2024-03-06T07:25:20.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.402+0000 7f69d3fff640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 0x7f69d4166110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:20.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.403+0000 7f69d37fe640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d40cd7a0 0x7f69d4152a70 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f69c400ec10 tx=0x7f69c400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.403+0000 7f69bbfff640 1 -- 172.21.15.77:0/3338507636 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c4017070 con 0x7f69d40cd7a0 2024-03-06T07:25:20.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.403+0000 7f69bbfff640 1 -- 172.21.15.77:0/3338507636 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c400cc30 con 0x7f69d40cd7a0 2024-03-06T07:25:20.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.403+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d4153170 con 0x7f69d40cd7a0 2024-03-06T07:25:20.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.403+0000 7f69bbfff640 1 -- 172.21.15.77:0/3338507636 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c4013470 con 0x7f69d40cd7a0 2024-03-06T07:25:20.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.403+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d4153730 con 0x7f69d40cd7a0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.404+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d4068840 con 0x7f69d40cd7a0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.404+0000 7f69bbfff640 1 -- 172.21.15.77:0/3338507636 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69c401a070 con 0x7f69d40cd7a0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.405+0000 7f69bbfff640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.405+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.405+0000 7f69bbfff640 1 -- 172.21.15.77:0/3338507636 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69c408a180 con 0x7f69d40cd7a0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.405+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c006b4d0 0x7f69c006d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.409+0000 7f69bbfff640 1 -- 172.21.15.77:0/3338507636 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69c4059000 con 0x7f69d40cd7a0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.709+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69d404f330 con 0x7f69d40cd7a0 2024-03-06T07:25:20.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.710+0000 7f69bbfff640 1 -- 172.21.15.77:0/3338507636 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f69c4058900 con 0x7f69d40cd7a0 2024-03-06T07:25:20.712 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:20.712 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:20.712 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.711+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c006b4d0 msgr2=0x7f69c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.711+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c006b4d0 0x7f69c006d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.711+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d40cd7a0 msgr2=0x7f69d4152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.711+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d40cd7a0 0x7f69d4152a70 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f69c400ec10 tx=0x7f69c400c520 comp rx=0 tx=0).stop 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.711+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 shutdown_connections 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.711+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69d415b430 0x7f69d4166110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.711+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69d41515e0 0x7f69d414d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.711+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69c006b4d0 0x7f69c006d990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.712+0000 7f69d9e9a640 1 --2- 172.21.15.77:0/3338507636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d40cd7a0 0x7f69d4152a70 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.712+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 >> 172.21.15.77:0/3338507636 conn(0x7f69d40cf8a0 msgr2=0x7f69d4072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.712+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 shutdown_connections 2024-03-06T07:25:20.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:20.712+0000 7f69d9e9a640 1 -- 172.21.15.77:0/3338507636 wait complete. 2024-03-06T07:25:20.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:20.839 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:25:20.839 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:25:20.839 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:20.839 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> import uuid 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> n = 100 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = False 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> if 0 > 0: 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = True 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> dpath = os.path.dirname(path) 2024-03-06T07:25:20.855 DEBUG:teuthology.orchestra.run.smithi077:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> try: 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> for i in range(n): 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> f.write(f"{i}") 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> f.flush() 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(f.fileno()) 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> if True: 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> os.unlink(fpath) 2024-03-06T07:25:20.856 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> if create_hard_links: 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> for j in range(0): 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> if True: 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> finally: 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> os.close(dirfd) 2024-03-06T07:25:20.857 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:25:21.006 INFO:teuthology.orchestra.run.smithi077.stderr:Traceback (most recent call last): 2024-03-06T07:25:21.006 INFO:teuthology.orchestra.run.smithi077.stderr: File "", line 26, in 2024-03-06T07:25:21.006 INFO:teuthology.orchestra.run.smithi077.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:25:21.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:25:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:21.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:21.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.159+0000 7f055ecb1640 1 Processor -- start 2024-03-06T07:25:21.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.159+0000 7f055ecb1640 1 -- start start 2024-03-06T07:25:21.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f055ecb1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05580cf850 0x7f05580cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f055ecb1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f055806cf20 0x7f055806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f055ecb1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055815e380 con 0x7f05580d00c0 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f055ecb1640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055806ca90 con 0x7f055806cf20 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f055ecb1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05580cdc10 con 0x7f05580cf850 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f055d24e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f055806cf20 0x7f055806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f055d24e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f055806cf20 0x7f055806d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46040/0 (socket says 172.21.15.77:46040) 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f055d24e640 1 -- 172.21.15.77:0/3740656153 learned_addr learned my addr 172.21.15.77:0/3740656153 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.160+0000 7f0557fff640 1 --2- 172.21.15.77:0/3740656153 >> v2:172.21.15.158:3300/0 conn(0x7f05580cf850 0x7f05580cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.161+0000 7f055d24e640 1 -- 172.21.15.77:0/3740656153 >> v2:172.21.15.158:3300/0 conn(0x7f05580cf850 msgr2=0x7f05580cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.161+0000 7f055d24e640 1 --2- 172.21.15.77:0/3740656153 >> v2:172.21.15.158:3300/0 conn(0x7f05580cf850 0x7f05580cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.161+0000 7f055d24e640 1 -- 172.21.15.77:0/3740656153 >> v1:172.21.15.158:6789/0 conn(0x7f05580d00c0 legacy=0x7f05580d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.161+0000 7f055d24e640 1 -- 172.21.15.77:0/3740656153 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055806c160 con 0x7f055806cf20 2024-03-06T07:25:21.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.161+0000 7f055d24e640 1 --2- 172.21.15.77:0/3740656153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055806cf20 0x7f055806d3b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0558071de0 tx=0x7f055002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0d7014c63f91b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f05577fe640 1 -- 172.21.15.77:0/3740656153 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055002cb20 con 0x7f055806cf20 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f05577fe640 1 -- 172.21.15.77:0/3740656153 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055002ccc0 con 0x7f055806cf20 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f05577fe640 1 -- 172.21.15.77:0/3740656153 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550034610 con 0x7f055806cf20 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f055ecb1640 1 -- 172.21.15.77:0/3740656153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055806cf20 msgr2=0x7f055806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f055ecb1640 1 --2- 172.21.15.77:0/3740656153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055806cf20 0x7f055806d3b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0558071de0 tx=0x7f055002bf80 comp rx=0 tx=0).stop 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f055ecb1640 1 -- 172.21.15.77:0/3740656153 shutdown_connections 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f055ecb1640 1 --2- 172.21.15.77:0/3740656153 >> v2:172.21.15.158:3300/0 conn(0x7f05580cf850 0x7f05580cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f055ecb1640 1 --2- 172.21.15.77:0/3740656153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055806cf20 0x7f055806d3b0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f055ecb1640 1 -- 172.21.15.77:0/3740656153 >> 172.21.15.77:0/3740656153 conn(0x7f05580b35a0 msgr2=0x7f05580b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f055ecb1640 1 -- 172.21.15.77:0/3740656153 shutdown_connections 2024-03-06T07:25:21.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.162+0000 7f055ecb1640 1 -- 172.21.15.77:0/3740656153 wait complete. 2024-03-06T07:25:21.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.164+0000 7f055ecb1640 1 Processor -- start 2024-03-06T07:25:21.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.164+0000 7f055ecb1640 1 -- start start 2024-03-06T07:25:21.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055ecb1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055806cf20 0x7f05580afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055ecb1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf850 0x7f05580c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055ecb1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d00c0 0x7f05580c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055ecb1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05580cd900 con 0x7f05580d00c0 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055ecb1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055806c720 con 0x7f05580cf850 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055ecb1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05580b1030 con 0x7f055806cf20 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055d24e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d00c0 0x7f05580c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055d24e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d00c0 0x7f05580c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46044/0 (socket says 172.21.15.77:46044) 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055d24e640 1 -- 172.21.15.77:0/1328676812 learned_addr learned my addr 172.21.15.77:0/1328676812 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f055ca4d640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055806cf20 0x7f05580afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.165+0000 7f0557fff640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf850 0x7f05580c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f055d24e640 1 -- 172.21.15.77:0/1328676812 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055806cf20 msgr2=0x7f05580afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f055d24e640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055806cf20 0x7f05580afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f055d24e640 1 -- 172.21.15.77:0/1328676812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf850 msgr2=0x7f05580c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f055d24e640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf850 0x7f05580c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f055d24e640 1 -- 172.21.15.77:0/1328676812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580c6220 con 0x7f05580d00c0 2024-03-06T07:25:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f055d24e640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d00c0 0x7f05580c5ae0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f0550036040 tx=0x7f0550004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f05557fa640 1 -- 172.21.15.77:0/1328676812 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550043020 con 0x7f05580d00c0 2024-03-06T07:25:21.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f05557fa640 1 -- 172.21.15.77:0/1328676812 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055003e450 con 0x7f05580d00c0 2024-03-06T07:25:21.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05580c6500 con 0x7f05580d00c0 2024-03-06T07:25:21.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f05557fa640 1 -- 172.21.15.77:0/1328676812 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550009510 con 0x7f05580d00c0 2024-03-06T07:25:21.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.166+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05580c6df0 con 0x7f05580d00c0 2024-03-06T07:25:21.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.167+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055804f330 con 0x7f05580d00c0 2024-03-06T07:25:21.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.168+0000 7f05557fa640 1 -- 172.21.15.77:0/1328676812 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0550005d10 con 0x7f05580d00c0 2024-03-06T07:25:21.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.168+0000 7f05557fa640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b4d0 0x7f053806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.168+0000 7f05557fa640 1 -- 172.21.15.77:0/1328676812 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05500acae0 con 0x7f05580d00c0 2024-03-06T07:25:21.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.168+0000 7f055ca4d640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b4d0 0x7f053806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.169+0000 7f055ca4d640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b4d0 0x7f053806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f05480016e0 tx=0x7f0548009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.172+0000 7f05557fa640 1 -- 172.21.15.77:0/1328676812 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f055007b3a0 con 0x7f05580d00c0 2024-03-06T07:25:21.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.389+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05580ce960 con 0x7f05580d00c0 2024-03-06T07:25:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.389+0000 7f05557fa640 1 -- 172.21.15.77:0/1328676812 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f055007f440 con 0x7f05580d00c0 2024-03-06T07:25:21.391 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:21.391 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:21.391 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b4d0 msgr2=0x7f053806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b4d0 0x7f053806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f05480016e0 tx=0x7f0548009040 comp rx=0 tx=0).stop 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d00c0 msgr2=0x7f05580c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d00c0 0x7f05580c5ae0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f0550036040 tx=0x7f0550004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 shutdown_connections 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b4d0 0x7f053806d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d00c0 0x7f05580c5ae0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf850 0x7f05580c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 --2- 172.21.15.77:0/1328676812 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055806cf20 0x7f05580afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 >> 172.21.15.77:0/1328676812 conn(0x7f05580b35a0 msgr2=0x7f055806fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 shutdown_connections 2024-03-06T07:25:21.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.391+0000 7f055ecb1640 1 -- 172.21.15.77:0/1328676812 wait complete. 2024-03-06T07:25:21.404 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:21.540 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:25:21.540 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:25:21.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:21.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.639+0000 7fb948c2a640 1 Processor -- start 2024-03-06T07:25:21.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.639+0000 7fb948c2a640 1 -- start start 2024-03-06T07:25:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.640+0000 7fb948c2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 0x7fb9440d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.640+0000 7fb948c2a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9440b3430 con 0x7fb94415dfe0 2024-03-06T07:25:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.640+0000 7fb948c2a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9440b3710 con 0x7fb9440b41f0 2024-03-06T07:25:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.640+0000 7fb948c2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440d2c50 con 0x7fb9440d22b0 2024-03-06T07:25:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.640+0000 7fb94259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb94415dfe0 0x7fb9440cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39990/0 (socket says 172.21.15.77:39990) 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.640+0000 7fb94259c640 1 -- 172.21.15.77:0/3424599039 learned_addr learned my addr 172.21.15.77:0/3424599039 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.640+0000 7fb941d9b640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 0x7fb9440d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359921856 0 0) 0x7fb9440b3430 con 0x7fb94415dfe0 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb92c003520 con 0x7fb94415dfe0 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576255280 0 0) 0x7fb9440b3710 con 0x7fb9440b41f0 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9440b3430 con 0x7fb9440b41f0 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2889708573 0 0) 0x7fb92c003520 con 0x7fb94415dfe0 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 msgr2=0x7fb9440d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 0x7fb9440d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 >> v1:172.21.15.158:6789/0 conn(0x7fb9440b41f0 legacy=0x7fb9440b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9440b39f0 con 0x7fb94415dfe0 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fb934001670 con 0x7fb94415dfe0 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb941d9b640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 0x7fb9440d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 >> v1:172.21.15.77:6789/0 conn(0x7fb94415dfe0 legacy=0x7fb9440cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb92c003b70 0x7fb92c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 0x7fb92c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb92c004d40 0x7fb92c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c00ba70 con 0x7fb92c003b70 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c00bbf0 con 0x7fb92c004d40 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c00bd70 con 0x7fb9440d22b0 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94259c640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb92c003b70 0x7fb92c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb941d9b640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 0x7fb92c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb942d9d640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb92c004d40 0x7fb92c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94259c640 1 -- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 msgr2=0x7fb92c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94259c640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9440d22b0 0x7fb92c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94259c640 1 -- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb92c004d40 msgr2=0x7fb92c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94259c640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb92c004d40 0x7fb92c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.641+0000 7fb94259c640 1 -- 172.21.15.77:0/3424599039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c003520 con 0x7fb92c003b70 2024-03-06T07:25:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.642+0000 7fb94259c640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb92c003b70 0x7fb92c003f70 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb93400c1c0 tx=0x7fb92c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c011b384940369f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.642+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934007cc0 con 0x7fb92c003b70 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.642+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb934030070 con 0x7fb92c003b70 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.642+0000 7fb94159a640 1 -- 172.21.15.77:0/3424599039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934005d30 con 0x7fb92c003b70 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.643+0000 7fb948c2a640 1 -- 172.21.15.77:0/3424599039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb92c003b70 msgr2=0x7fb92c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.643+0000 7fb948c2a640 1 --2- 172.21.15.77:0/3424599039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb92c003b70 0x7fb92c003f70 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb93400c1c0 tx=0x7fb92c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.643+0000 7fb942d9d640 1 -- 172.21.15.77:0/3424599039 reap_dead start 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.643+0000 7fb948c2a640 1 -- 172.21.15.77:0/3424599039 shutdown_connections 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.643+0000 7fb948c2a640 1 -- 172.21.15.77:0/3424599039 >> 172.21.15.77:0/3424599039 conn(0x7fb9440d3cd0 msgr2=0x7fb9440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.643+0000 7fb948c2a640 1 -- 172.21.15.77:0/3424599039 shutdown_connections 2024-03-06T07:25:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.643+0000 7fb948c2a640 1 -- 172.21.15.77:0/3424599039 wait complete. 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.645+0000 7fb948c2a640 1 Processor -- start 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.645+0000 7fb948c2a640 1 -- start start 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb948c2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb92c003b70 0x7fb944143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb948c2a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9440b73f0 0x7fb944160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb948c2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb944160e50 0x7fb94415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb948c2a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440b3d60 con 0x7fb944160e50 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb948c2a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440b31e0 con 0x7fb9440b73f0 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb948c2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440d1c70 con 0x7fb92c003b70 2024-03-06T07:25:21.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb942d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb944160e50 0x7fb94415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb942d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb944160e50 0x7fb94415e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46056/0 (socket says 172.21.15.77:46056) 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb942d9d640 1 -- 172.21.15.77:0/401289720 learned_addr learned my addr 172.21.15.77:0/401289720 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb94259c640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb92c003b70 0x7fb944143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.646+0000 7fb941d9b640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9440b73f0 0x7fb944160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb942d9d640 1 -- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb92c003b70 msgr2=0x7fb944143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb942d9d640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb92c003b70 0x7fb944143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb942d9d640 1 -- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9440b73f0 msgr2=0x7fb944160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb942d9d640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9440b73f0 0x7fb944160910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb942d9d640 1 -- 172.21.15.77:0/401289720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944143650 con 0x7fb944160e50 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb941d9b640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9440b73f0 0x7fb944160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb942d9d640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb944160e50 0x7fb94415e3c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb9440cff40 tx=0x7fb93c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb932ffd640 1 -- 172.21.15.77:0/401289720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c003970 con 0x7fb944160e50 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb932ffd640 1 -- 172.21.15.77:0/401289720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c018070 con 0x7fb944160e50 2024-03-06T07:25:21.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb944161460 con 0x7fb944160e50 2024-03-06T07:25:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb932ffd640 1 -- 172.21.15.77:0/401289720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c003d50 con 0x7fb944160e50 2024-03-06T07:25:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.647+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9440cc320 con 0x7fb944160e50 2024-03-06T07:25:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.648+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94404f330 con 0x7fb944160e50 2024-03-06T07:25:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.648+0000 7fb932ffd640 1 -- 172.21.15.77:0/401289720 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb93c0050d0 con 0x7fb944160e50 2024-03-06T07:25:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.649+0000 7fb932ffd640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb92406b500 0x7fb92406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.649+0000 7fb94259c640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb92406b500 0x7fb92406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.649+0000 7fb932ffd640 1 -- 172.21.15.77:0/401289720 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb93c089f10 con 0x7fb944160e50 2024-03-06T07:25:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.650+0000 7fb94259c640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb92406b500 0x7fb92406d9c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb934034000 tx=0x7fb934007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.653+0000 7fb932ffd640 1 -- 172.21.15.77:0/401289720 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb93c058d90 con 0x7fb944160e50 2024-03-06T07:25:21.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.937+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb94406a1b0 con 0x7fb944160e50 2024-03-06T07:25:21.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.938+0000 7fb932ffd640 1 -- 172.21.15.77:0/401289720 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fb93c058690 con 0x7fb944160e50 2024-03-06T07:25:21.939 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:21.940 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:21.940 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:21.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb92406b500 msgr2=0x7fb92406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb92406b500 0x7fb92406d9c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb934034000 tx=0x7fb934007000 comp rx=0 tx=0).stop 2024-03-06T07:25:21.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb944160e50 msgr2=0x7fb94415e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb944160e50 0x7fb94415e3c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb9440cff40 tx=0x7fb93c00eb20 comp rx=0 tx=0).stop 2024-03-06T07:25:21.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 shutdown_connections 2024-03-06T07:25:21.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb92406b500 0x7fb92406d9c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb944160e50 0x7fb94415e3c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9440b73f0 0x7fb944160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 --2- 172.21.15.77:0/401289720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb92c003b70 0x7fb944143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 >> 172.21.15.77:0/401289720 conn(0x7fb9440d3cd0 msgr2=0x7fb94406ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 shutdown_connections 2024-03-06T07:25:21.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:21.939+0000 7fb948c2a640 1 -- 172.21.15.77:0/401289720 wait complete. 2024-03-06T07:25:21.952 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:22.071 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:25:22.071 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:25:22.072 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:25:27.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:27.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.172+0000 7f2290239640 1 Processor -- start 2024-03-06T07:25:27.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.172+0000 7f2290239640 1 -- start start 2024-03-06T07:25:27.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f2290239640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22880d22b0 0x7f22880d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f2290239640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22880b41f0 0x7f22880b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f2290239640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22880b3430 con 0x7f228815dfe0 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f2290239640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22880d2c50 con 0x7f22880b41f0 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f2290239640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22880b31e0 con 0x7f22880d22b0 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f228d7d4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22880d22b0 0x7f22880d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f228e7d6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22880b41f0 0x7f22880b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f228dfd5640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f228815dfe0 0x7f22880cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:37876/0 (socket says 172.21.15.77:37876) 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.174+0000 7f228dfd5640 1 -- 172.21.15.77:0/3671446492 learned_addr learned my addr 172.21.15.77:0/3671446492 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.175+0000 7f228d7d4640 1 -- 172.21.15.77:0/3671446492 >> v2:172.21.15.158:3300/0 conn(0x7f22880b41f0 msgr2=0x7f22880b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.175+0000 7f228d7d4640 1 --2- 172.21.15.77:0/3671446492 >> v2:172.21.15.158:3300/0 conn(0x7f22880b41f0 0x7f22880b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.175+0000 7f228d7d4640 1 -- 172.21.15.77:0/3671446492 >> v1:172.21.15.158:6789/0 conn(0x7f228815dfe0 legacy=0x7f22880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.175+0000 7f228d7d4640 1 -- 172.21.15.77:0/3671446492 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880b3710 con 0x7f22880d22b0 2024-03-06T07:25:27.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.175+0000 7f228cfd3640 1 -- 172.21.15.77:0/3671446492 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007891505 0 0) 0x7f22880b3430 con 0x7f228815dfe0 2024-03-06T07:25:27.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.175+0000 7f228e7d6640 1 --2- 172.21.15.77:0/3671446492 >> v2:172.21.15.158:3300/0 conn(0x7f22880b41f0 0x7f22880b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:27.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.175+0000 7f228d7d4640 1 --2- 172.21.15.77:0/3671446492 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22880d22b0 0x7f22880d2710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f22780097a0 tx=0x7f227802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96de553761e26ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f228cfd3640 1 -- 172.21.15.77:0/3671446492 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2278035020 con 0x7f22880d22b0 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f228cfd3640 1 -- 172.21.15.77:0/3671446492 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2278031830 con 0x7f22880d22b0 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f228cfd3640 1 -- 172.21.15.77:0/3671446492 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2278031c00 con 0x7f22880d22b0 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f2290239640 1 -- 172.21.15.77:0/3671446492 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22880d22b0 msgr2=0x7f22880d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f2290239640 1 --2- 172.21.15.77:0/3671446492 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22880d22b0 0x7f22880d2710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f22780097a0 tx=0x7f227802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f2290239640 1 -- 172.21.15.77:0/3671446492 shutdown_connections 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f2290239640 1 --2- 172.21.15.77:0/3671446492 >> v2:172.21.15.158:3300/0 conn(0x7f22880b41f0 0x7f22880b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f2290239640 1 --2- 172.21.15.77:0/3671446492 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22880d22b0 0x7f22880d2710 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f2290239640 1 -- 172.21.15.77:0/3671446492 >> 172.21.15.77:0/3671446492 conn(0x7f22880d3cd0 msgr2=0x7f22880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f2290239640 1 -- 172.21.15.77:0/3671446492 shutdown_connections 2024-03-06T07:25:27.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.176+0000 7f2290239640 1 -- 172.21.15.77:0/3671446492 wait complete. 2024-03-06T07:25:27.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.177+0000 7f2290239640 1 Processor -- start 2024-03-06T07:25:27.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f2290239640 1 -- start start 2024-03-06T07:25:27.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f2290239640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22880b41f0 0x7f22880b86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f2290239640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22880d22b0 0x7f2288160e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f2290239640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f228815dfe0 0x7f22881f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f2290239640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22880cfa00 con 0x7f22880b41f0 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f2290239640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22880b3d60 con 0x7f22880d22b0 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f2290239640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22880d1c70 con 0x7f228815dfe0 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f228dfd5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22880b41f0 0x7f22880b86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f228dfd5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22880b41f0 0x7f22880b86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46068/0 (socket says 172.21.15.77:46068) 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f228dfd5640 1 -- 172.21.15.77:0/424827968 learned_addr learned my addr 172.21.15.77:0/424827968 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.178+0000 7f228d7d4640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22880d22b0 0x7f2288160e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f228e7d6640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f228815dfe0 0x7f22881f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f228dfd5640 1 -- 172.21.15.77:0/424827968 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f228815dfe0 msgr2=0x7f22881f7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f228dfd5640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f228815dfe0 0x7f22881f7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f228dfd5640 1 -- 172.21.15.77:0/424827968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22880d22b0 msgr2=0x7f2288160e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f228dfd5640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22880d22b0 0x7f2288160e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f228dfd5640 1 -- 172.21.15.77:0/424827968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880b8dd0 con 0x7f22880b41f0 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f228dfd5640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22880b41f0 0x7f22880b86a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f228400c430 tx=0x7f228400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f227e7fc640 1 -- 172.21.15.77:0/424827968 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22840074e0 con 0x7f22880b41f0 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f227e7fc640 1 -- 172.21.15.77:0/424827968 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22840047e0 con 0x7f22880b41f0 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22880b90b0 con 0x7f22880b41f0 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f227e7fc640 1 -- 172.21.15.77:0/424827968 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284004bd0 con 0x7f22880b41f0 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.179+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22880b9670 con 0x7f22880b41f0 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.180+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228804f330 con 0x7f22880b41f0 2024-03-06T07:25:27.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.181+0000 7f227e7fc640 1 -- 172.21.15.77:0/424827968 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2284011070 con 0x7f22880b41f0 2024-03-06T07:25:27.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.181+0000 7f227e7fc640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f226c06b4d0 0x7f226c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.181+0000 7f227e7fc640 1 -- 172.21.15.77:0/424827968 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2284089c60 con 0x7f22880b41f0 2024-03-06T07:25:27.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.181+0000 7f228d7d4640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f226c06b4d0 0x7f226c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.181+0000 7f228d7d4640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f226c06b4d0 0x7f226c06d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2278036000 tx=0x7f227803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.185+0000 7f227e7fc640 1 -- 172.21.15.77:0/424827968 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2284058b70 con 0x7f22880b41f0 2024-03-06T07:25:27.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.395+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22880cea30 con 0x7f22880b41f0 2024-03-06T07:25:27.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.396+0000 7f227e7fc640 1 -- 172.21.15.77:0/424827968 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f2284058470 con 0x7f22880b41f0 2024-03-06T07:25:27.398 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:27.398 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:27.398 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f226c06b4d0 msgr2=0x7f226c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f226c06b4d0 0x7f226c06d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2278036000 tx=0x7f227803d040 comp rx=0 tx=0).stop 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22880b41f0 msgr2=0x7f22880b86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22880b41f0 0x7f22880b86a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f228400c430 tx=0x7f228400c830 comp rx=0 tx=0).stop 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 shutdown_connections 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f228815dfe0 0x7f22881f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22880d22b0 0x7f2288160e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f226c06b4d0 0x7f226c06d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 --2- 172.21.15.77:0/424827968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22880b41f0 0x7f22880b86a0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 >> 172.21.15.77:0/424827968 conn(0x7f22880d3cd0 msgr2=0x7f22880cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 shutdown_connections 2024-03-06T07:25:27.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.399+0000 7f2290239640 1 -- 172.21.15.77:0/424827968 wait complete. 2024-03-06T07:25:27.411 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:27.527 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:25:27.527 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:25:27.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:27.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.626+0000 7fd32811e640 1 Processor -- start 2024-03-06T07:25:27.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.626+0000 7fd32811e640 1 -- start start 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.627+0000 7fd32811e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3201515e0 0x7fd3200cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.627+0000 7fd32811e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.627+0000 7fd32811e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32015b8b0 con 0x7fd3200cd7a0 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.627+0000 7fd32811e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd320070a70 con 0x7fd3201515e0 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.627+0000 7fd32811e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd320070d50 con 0x7fd32015b430 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.627+0000 7fd325eba640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3201515e0 0x7fd3200cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.627+0000 7fd325eba640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3201515e0 0x7fd3200cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46080/0 (socket says 172.21.15.77:46080) 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.627+0000 7fd325eba640 1 -- 172.21.15.77:0/1924995290 learned_addr learned my addr 172.21.15.77:0/1924995290 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.628+0000 7fd3256b9640 1 --2- 172.21.15.77:0/1924995290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.628+0000 7fd325eba640 1 -- 172.21.15.77:0/1924995290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 msgr2=0x7fd320071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.628+0000 7fd325eba640 1 --2- 172.21.15.77:0/1924995290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.628+0000 7fd325eba640 1 -- 172.21.15.77:0/1924995290 >> v1:172.21.15.158:6789/0 conn(0x7fd3200cd7a0 legacy=0x7fd3200cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.628+0000 7fd325eba640 1 -- 172.21.15.77:0/1924995290 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320071030 con 0x7fd3201515e0 2024-03-06T07:25:27.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd325eba640 1 --2- 172.21.15.77:0/1924995290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd3200cce40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009870 tx=0x7fd31c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2d478c2eb602ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd324eb8640 1 -- 172.21.15.77:0/1924995290 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c034070 con 0x7fd3201515e0 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd324eb8640 1 -- 172.21.15.77:0/1924995290 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31c030840 con 0x7fd3201515e0 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd324eb8640 1 -- 172.21.15.77:0/1924995290 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c030c10 con 0x7fd3201515e0 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd32811e640 1 -- 172.21.15.77:0/1924995290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 msgr2=0x7fd3200cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd32811e640 1 --2- 172.21.15.77:0/1924995290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd3200cce40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009870 tx=0x7fd31c02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd32811e640 1 -- 172.21.15.77:0/1924995290 shutdown_connections 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd32811e640 1 --2- 172.21.15.77:0/1924995290 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.629+0000 7fd32811e640 1 --2- 172.21.15.77:0/1924995290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd3200cce40 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.630+0000 7fd32811e640 1 -- 172.21.15.77:0/1924995290 >> 172.21.15.77:0/1924995290 conn(0x7fd3200cf8a0 msgr2=0x7fd3200cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.630+0000 7fd32811e640 1 -- 172.21.15.77:0/1924995290 shutdown_connections 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.630+0000 7fd32811e640 1 -- 172.21.15.77:0/1924995290 wait complete. 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.631+0000 7fd32811e640 1 Processor -- start 2024-03-06T07:25:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.631+0000 7fd32811e640 1 -- start start 2024-03-06T07:25:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.631+0000 7fd32811e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3200cd7a0 0x7fd320143b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.631+0000 7fd32811e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd32015d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.631+0000 7fd32811e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.631+0000 7fd32811e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd320071ef0 con 0x7fd3201515e0 2024-03-06T07:25:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.631+0000 7fd32811e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd320071d70 con 0x7fd3200cd7a0 2024-03-06T07:25:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.631+0000 7fd32811e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200cc3a0 con 0x7fd32015b430 2024-03-06T07:25:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd32015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd32015d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46094/0 (socket says 172.21.15.77:46094) 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 -- 172.21.15.77:0/123599404 learned_addr learned my addr 172.21.15.77:0/123599404 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd325eba640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3200cd7a0 0x7fd320143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3266bb640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 -- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 msgr2=0x7fd320153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 -- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3200cd7a0 msgr2=0x7fd320143b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3200cd7a0 0x7fd320143b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 -- 172.21.15.77:0/123599404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320145080 con 0x7fd3201515e0 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3266bb640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3256b9640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd32015d940 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd31000e870 tx=0x7fd31000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3167fc640 1 -- 172.21.15.77:0/123599404 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310017070 con 0x7fd3201515e0 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3167fc640 1 -- 172.21.15.77:0/123599404 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31000c940 con 0x7fd3201515e0 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd320145360 con 0x7fd3201515e0 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd3167fc640 1 -- 172.21.15.77:0/123599404 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31000cd30 con 0x7fd3201515e0 2024-03-06T07:25:27.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.632+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3201461c0 con 0x7fd3201515e0 2024-03-06T07:25:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.633+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd320068840 con 0x7fd3201515e0 2024-03-06T07:25:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.634+0000 7fd3167fc640 1 -- 172.21.15.77:0/123599404 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd31001a070 con 0x7fd3201515e0 2024-03-06T07:25:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.634+0000 7fd3167fc640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30406b4d0 0x7fd30406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.634+0000 7fd3167fc640 1 -- 172.21.15.77:0/123599404 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd310089ff0 con 0x7fd3201515e0 2024-03-06T07:25:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.634+0000 7fd325eba640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30406b4d0 0x7fd30406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.635+0000 7fd325eba640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30406b4d0 0x7fd30406d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd31c036000 tx=0x7fd31c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.638+0000 7fd3167fc640 1 -- 172.21.15.77:0/123599404 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd310058f00 con 0x7fd3201515e0 2024-03-06T07:25:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.854+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd32004f330 con 0x7fd3201515e0 2024-03-06T07:25:27.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.854+0000 7fd3167fc640 1 -- 172.21.15.77:0/123599404 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fd310058800 con 0x7fd3201515e0 2024-03-06T07:25:27.856 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:27.856 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:27.857 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:27.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30406b4d0 msgr2=0x7fd30406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30406b4d0 0x7fd30406d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd31c036000 tx=0x7fd31c03b040 comp rx=0 tx=0).stop 2024-03-06T07:25:27.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 msgr2=0x7fd32015d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd32015d940 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd31000e870 tx=0x7fd31000ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 shutdown_connections 2024-03-06T07:25:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30406b4d0 0x7fd30406d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3201515e0 0x7fd32015d940 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd32015b430 0x7fd320153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 --2- 172.21.15.77:0/123599404 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3200cd7a0 0x7fd320143b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 >> 172.21.15.77:0/123599404 conn(0x7fd3200cf8a0 msgr2=0x7fd32014fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 shutdown_connections 2024-03-06T07:25:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:27.856+0000 7fd32811e640 1 -- 172.21.15.77:0/123599404 wait complete. 2024-03-06T07:25:27.869 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:25:27.983 INFO:teuthology.orchestra.run.smithi077.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:25:27.983 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:25:27.984 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:27.984 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> import uuid 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> n = 10 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = False 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> if 0 > 0: 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> create_hard_links = True 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:28.000 DEBUG:teuthology.orchestra.run.smithi077:> dpath = os.path.dirname(path) 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> try: 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> for i in range(n): 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> f.write(f"{i}") 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> f.flush() 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(f.fileno()) 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> if True: 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> os.unlink(fpath) 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> if True: 2024-03-06T07:25:28.001 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:25:28.002 DEBUG:teuthology.orchestra.run.smithi077:> if create_hard_links: 2024-03-06T07:25:28.002 DEBUG:teuthology.orchestra.run.smithi077:> for j in range(0): 2024-03-06T07:25:28.002 DEBUG:teuthology.orchestra.run.smithi077:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:28.002 DEBUG:teuthology.orchestra.run.smithi077:> if False: 2024-03-06T07:25:28.002 DEBUG:teuthology.orchestra.run.smithi077:> os.fsync(dirfd) 2024-03-06T07:25:28.002 DEBUG:teuthology.orchestra.run.smithi077:> finally: 2024-03-06T07:25:28.002 DEBUG:teuthology.orchestra.run.smithi077:> os.close(dirfd) 2024-03-06T07:25:28.002 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:25:28.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:28.177 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:28.177 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:28.254 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:28.308 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:28.308 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:28.308 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:28.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:28.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.463+0000 7f80c695e640 1 Processor -- start 2024-03-06T07:25:28.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.463+0000 7f80c695e640 1 -- start start 2024-03-06T07:25:28.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.463+0000 7f80c695e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 0x7f80c00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.463+0000 7f80c695e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f80c00cfff0 0x7f80c00d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.463+0000 7f80c695e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c0161980 con 0x7f80c00ce4b0 2024-03-06T07:25:28.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.463+0000 7f80c695e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c0161c60 con 0x7f80c00cfff0 2024-03-06T07:25:28.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.463+0000 7f80c695e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c0161f40 con 0x7f80c015bb20 2024-03-06T07:25:28.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80c4efb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f80c00ce4b0 0x7f80c00ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46068/0 (socket says 172.21.15.77:46068) 2024-03-06T07:25:28.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80c4efb640 1 -- 172.21.15.77:0/2094853878 learned_addr learned my addr 172.21.15.77:0/2094853878 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:28.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80bf7fe640 1 --2- 172.21.15.77:0/2094853878 >> v2:172.21.15.158:3300/0 conn(0x7f80c00cfff0 0x7f80c00d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80bffff640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 0x7f80c00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423594724 0 0) 0x7f80c0161980 con 0x7f80c00ce4b0 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a8003520 con 0x7f80c00ce4b0 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1118991242 0 0) 0x7f80a8003520 con 0x7f80c00ce4b0 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 msgr2=0x7f80c00d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 0x7f80c00d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 >> v2:172.21.15.158:3300/0 conn(0x7f80c00cfff0 msgr2=0x7f80c00d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 --2- 172.21.15.77:0/2094853878 >> v2:172.21.15.158:3300/0 conn(0x7f80c00cfff0 0x7f80c00d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c0162220 con 0x7f80c00ce4b0 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f80b80019f0 con 0x7f80c00ce4b0 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 >> v1:172.21.15.77:6789/0 conn(0x7f80c00ce4b0 legacy=0x7f80c00ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80a80039b0 0x7f80a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 0x7f80a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a8004d40 0x7f80a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c0161c60 con 0x7f80a80039b0 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c0161f40 con 0x7f80a8004d40 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a800b9c0 con 0x7f80c015bb20 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80bffff640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80a80039b0 0x7f80a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.464+0000 7f80c4efb640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a8004d40 0x7f80a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80bf7fe640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 0x7f80a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80bffff640 1 -- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 msgr2=0x7f80a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80bffff640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 0x7f80a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80bffff640 1 -- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a8004d40 msgr2=0x7f80a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80bffff640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a8004d40 0x7f80a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80bffff640 1 -- 172.21.15.77:0/2094853878 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8003520 con 0x7f80a80039b0 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80bf7fe640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c015bb20 0x7f80a8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80bffff640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80a80039b0 0x7f80a8003dd0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009ac0 tx=0x7f80a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fa23ce90a15c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.465+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0039070 con 0x7f80a80039b0 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b002c6b0 con 0x7f80a80039b0 2024-03-06T07:25:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80beffd640 1 -- 172.21.15.77:0/2094853878 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0009690 con 0x7f80a80039b0 2024-03-06T07:25:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80c695e640 1 -- 172.21.15.77:0/2094853878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80a80039b0 msgr2=0x7f80a8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80c695e640 1 --2- 172.21.15.77:0/2094853878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80a80039b0 0x7f80a8003dd0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009ac0 tx=0x7f80a8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80c4efb640 1 -- 172.21.15.77:0/2094853878 reap_dead start 2024-03-06T07:25:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80c695e640 1 -- 172.21.15.77:0/2094853878 shutdown_connections 2024-03-06T07:25:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80c695e640 1 -- 172.21.15.77:0/2094853878 >> 172.21.15.77:0/2094853878 conn(0x7f80c009a9f0 msgr2=0x7f80c009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80c695e640 1 -- 172.21.15.77:0/2094853878 shutdown_connections 2024-03-06T07:25:28.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.466+0000 7f80c695e640 1 -- 172.21.15.77:0/2094853878 wait complete. 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.467+0000 7f80c695e640 1 Processor -- start 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.467+0000 7f80c695e640 1 -- start start 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80c695e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a80039b0 0x7f80c015b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80c695e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c006ceb0 0x7f80c006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80c695e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c01f3880 0x7f80c01f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80c695e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00d1b00 con 0x7f80c006ceb0 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80c695e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00d1980 con 0x7f80a80039b0 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80c695e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00cf050 con 0x7f80c01f3880 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c006ceb0 0x7f80c006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c006ceb0 0x7f80c006d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48020/0 (socket says 172.21.15.77:48020) 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bf7fe640 1 -- 172.21.15.77:0/4092004435 learned_addr learned my addr 172.21.15.77:0/4092004435 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bffff640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a80039b0 0x7f80c015b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80c4efb640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c01f3880 0x7f80c01f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bf7fe640 1 -- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c01f3880 msgr2=0x7f80c01f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bf7fe640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c01f3880 0x7f80c01f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bf7fe640 1 -- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a80039b0 msgr2=0x7f80c015b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bf7fe640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a80039b0 0x7f80c015b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bf7fe640 1 -- 172.21.15.77:0/4092004435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c01f4240 con 0x7f80c006ceb0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.468+0000 7f80bffff640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a80039b0 0x7f80c015b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.469+0000 7f80bf7fe640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c006ceb0 0x7f80c006d330 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f80b4011700 tx=0x7f80b4011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.469+0000 7f80bcff9640 1 -- 172.21.15.77:0/4092004435 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b400f9c0 con 0x7f80c006ceb0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.469+0000 7f80bcff9640 1 -- 172.21.15.77:0/4092004435 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b400fb60 con 0x7f80c006ceb0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.469+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c01f4500 con 0x7f80c006ceb0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.469+0000 7f80bcff9640 1 -- 172.21.15.77:0/4092004435 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b4015460 con 0x7f80c006ceb0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.469+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c00b5860 con 0x7f80c006ceb0 2024-03-06T07:25:28.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.470+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c004f330 con 0x7f80c006ceb0 2024-03-06T07:25:28.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.470+0000 7f80bcff9640 1 -- 172.21.15.77:0/4092004435 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80b4015600 con 0x7f80c006ceb0 2024-03-06T07:25:28.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.471+0000 7f80bcff9640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f80ac06b500 0x7f80ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.471+0000 7f80bffff640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f80ac06b500 0x7f80ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.471+0000 7f80bcff9640 1 -- 172.21.15.77:0/4092004435 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80b408aa30 con 0x7f80c006ceb0 2024-03-06T07:25:28.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.471+0000 7f80bffff640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f80ac06b500 0x7f80ac06d9c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f80b0036000 tx=0x7f80b0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.475+0000 7f80bcff9640 1 -- 172.21.15.77:0/4092004435 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80b4019030 con 0x7f80c006ceb0 2024-03-06T07:25:28.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.653+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80c00b1fe0 con 0x7f80c006ceb0 2024-03-06T07:25:28.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.654+0000 7f80bcff9640 1 -- 172.21.15.77:0/4092004435 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16656 (secure 0 0 0) 0x7f80b4059940 con 0x7f80c006ceb0 2024-03-06T07:25:28.656 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:28.656 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":69,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:00.275386+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:56.868266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:57.595004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:28.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f80ac06b500 msgr2=0x7f80ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f80ac06b500 0x7f80ac06d9c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f80b0036000 tx=0x7f80b0030000 comp rx=0 tx=0).stop 2024-03-06T07:25:28.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c006ceb0 msgr2=0x7f80c006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c006ceb0 0x7f80c006d330 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f80b4011700 tx=0x7f80b4011bc0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 shutdown_connections 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f80ac06b500 0x7f80ac06d9c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c006ceb0 0x7f80c006d330 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f80c01f3880 0x7f80c01f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 --2- 172.21.15.77:0/4092004435 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f80a80039b0 0x7f80c015b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 >> 172.21.15.77:0/4092004435 conn(0x7f80c009a9f0 msgr2=0x7f80c006f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 shutdown_connections 2024-03-06T07:25:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.656+0000 7f80c695e640 1 -- 172.21.15.77:0/4092004435 wait complete. 2024-03-06T07:25:28.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:56.868266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:57.595004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:28.669 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.771+0000 7fb78f55c640 1 Processor -- start 2024-03-06T07:25:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78f55c640 1 -- start start 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78f55c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb7880d4860 0x7fb7880d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78f55c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 0x7fb7880b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78f55c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7881618b0 con 0x7fb7881614d0 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78f55c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7880d4cf0 con 0x7fb7880d4860 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78f55c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7880cf400 con 0x7fb7880b0e30 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78caf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb7880d4860 0x7fb7880d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78d2f8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb7881614d0 0x7fb7880d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46074/0 (socket says 172.21.15.77:46074) 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78caf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb7880d4860 0x7fb7880d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48032/0 (socket says 172.21.15.77:48032) 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.772+0000 7fb78d2f8640 1 -- 172.21.15.77:0/3062442490 learned_addr learned my addr 172.21.15.77:0/3062442490 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.773+0000 7fb78caf7640 1 -- 172.21.15.77:0/3062442490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 msgr2=0x7fb7880b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.773+0000 7fb78daf9640 1 --2- 172.21.15.77:0/3062442490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 0x7fb7880b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.773+0000 7fb78caf7640 1 --2- 172.21.15.77:0/3062442490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 0x7fb7880b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.773+0000 7fb78caf7640 1 -- 172.21.15.77:0/3062442490 >> v1:172.21.15.77:6789/0 conn(0x7fb7881614d0 legacy=0x7fb7880d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.773+0000 7fb77ffff640 1 -- 172.21.15.77:0/3062442490 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302583385 0 0) 0x7fb7881618b0 con 0x7fb7881614d0 2024-03-06T07:25:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.773+0000 7fb78caf7640 1 -- 172.21.15.77:0/3062442490 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7880cf6a0 con 0x7fb7880d4860 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.773+0000 7fb78caf7640 1 --2- 172.21.15.77:0/3062442490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880d2020 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb7780097a0 tx=0x7fb77802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09f6bd6624755fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb77ffff640 1 -- 172.21.15.77:0/3062442490 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778035020 con 0x7fb7880d4860 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb77ffff640 1 -- 172.21.15.77:0/3062442490 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb778031830 con 0x7fb7880d4860 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb77ffff640 1 -- 172.21.15.77:0/3062442490 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778031c00 con 0x7fb7880d4860 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb78f55c640 1 -- 172.21.15.77:0/3062442490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 msgr2=0x7fb7880d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb78f55c640 1 --2- 172.21.15.77:0/3062442490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880d2020 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb7780097a0 tx=0x7fb77802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb78f55c640 1 -- 172.21.15.77:0/3062442490 shutdown_connections 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb78f55c640 1 --2- 172.21.15.77:0/3062442490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 0x7fb7880b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb78f55c640 1 --2- 172.21.15.77:0/3062442490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880d2020 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb78f55c640 1 -- 172.21.15.77:0/3062442490 >> 172.21.15.77:0/3062442490 conn(0x7fb7880d3d40 msgr2=0x7fb7880d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb78f55c640 1 -- 172.21.15.77:0/3062442490 shutdown_connections 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.774+0000 7fb78f55c640 1 -- 172.21.15.77:0/3062442490 wait complete. 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.775+0000 7fb78f55c640 1 Processor -- start 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.775+0000 7fb78f55c640 1 -- start start 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78f55c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 0x7fb7880cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78f55c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78f55c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7881614d0 0x7fb7880c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78f55c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7880b0910 con 0x7fb7880d4860 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78f55c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7880cfc60 con 0x7fb7881614d0 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78f55c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7880d2560 con 0x7fb7880b0e30 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78caf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78caf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48036/0 (socket says 172.21.15.77:48036) 2024-03-06T07:25:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78caf7640 1 -- 172.21.15.77:0/525335360 learned_addr learned my addr 172.21.15.77:0/525335360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78d2f8640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 0x7fb7880cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78daf9640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7881614d0 0x7fb7880c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78caf7640 1 -- 172.21.15.77:0/525335360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 msgr2=0x7fb7880cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78caf7640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 0x7fb7880cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78caf7640 1 -- 172.21.15.77:0/525335360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7881614d0 msgr2=0x7fb7880c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.776+0000 7fb78caf7640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7881614d0 0x7fb7880c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.777+0000 7fb78caf7640 1 -- 172.21.15.77:0/525335360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7880c9690 con 0x7fb7880d4860 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.777+0000 7fb78caf7640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880c8ab0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fb77802bee0 tx=0x7fb7780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.777+0000 7fb77dffb640 1 -- 172.21.15.77:0/525335360 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778041070 con 0x7fb7880d4860 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.777+0000 7fb77dffb640 1 -- 172.21.15.77:0/525335360 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77803a950 con 0x7fb7880d4860 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.777+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7880c9970 con 0x7fb7880d4860 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.777+0000 7fb77dffb640 1 -- 172.21.15.77:0/525335360 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778035040 con 0x7fb7880d4860 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.777+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7880c9f30 con 0x7fb7880d4860 2024-03-06T07:25:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.778+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78806b9b0 con 0x7fb7880d4860 2024-03-06T07:25:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.779+0000 7fb77dffb640 1 -- 172.21.15.77:0/525335360 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb77804c060 con 0x7fb7880d4860 2024-03-06T07:25:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.779+0000 7fb77dffb640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb76c06b4d0 0x7fb76c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.779+0000 7fb77dffb640 1 -- 172.21.15.77:0/525335360 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7780abbd0 con 0x7fb7880d4860 2024-03-06T07:25:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.779+0000 7fb78d2f8640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb76c06b4d0 0x7fb76c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.779+0000 7fb78d2f8640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb76c06b4d0 0x7fb76c06d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb780007ed0 tx=0x7fb780007e60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.783+0000 7fb77dffb640 1 -- 172.21.15.77:0/525335360 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb78806b9b0 con 0x7fb7880d4860 2024-03-06T07:25:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.956+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7880cde40 con 0x7fb7880d4860 2024-03-06T07:25:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.956+0000 7fb77dffb640 1 -- 172.21.15.77:0/525335360 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb77807aae0 con 0x7fb7880d4860 2024-03-06T07:25:28.958 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:25:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb76c06b4d0 msgr2=0x7fb76c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb76c06b4d0 0x7fb76c06d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb780007ed0 tx=0x7fb780007e60 comp rx=0 tx=0).stop 2024-03-06T07:25:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 msgr2=0x7fb7880c8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880c8ab0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fb77802bee0 tx=0x7fb7780046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 shutdown_connections 2024-03-06T07:25:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb76c06b4d0 0x7fb76c06d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7880d4860 0x7fb7880c8ab0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7881614d0 0x7fb7880c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 --2- 172.21.15.77:0/525335360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7880b0e30 0x7fb7880cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 >> 172.21.15.77:0/525335360 conn(0x7fb7880d3d40 msgr2=0x7fb78806bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 shutdown_connections 2024-03-06T07:25:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:28.958+0000 7fb78f55c640 1 -- 172.21.15.77:0/525335360 wait complete. 2024-03-06T07:25:28.971 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:29.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.072+0000 7f8a53a0e640 1 Processor -- start 2024-03-06T07:25:29.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.073+0000 7f8a53a0e640 1 -- start start 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a53a0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c1614d0 0x7f8a4c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a53a0e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a4c0d1910 0x7f8a4c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a53a0e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c0d2740 con 0x7f8a4c0d22b0 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a53a0e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4c1618b0 con 0x7f8a4c0d1910 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a53a0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0cf400 con 0x7f8a4c1614d0 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a50fa9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a4c0d1910 0x7f8a4c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a50fa9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a4c0d1910 0x7f8a4c0d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48044/0 (socket says 172.21.15.77:48044) 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a50fa9640 1 -- 172.21.15.77:0/4006536009 learned_addr learned my addr 172.21.15.77:0/4006536009 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a517aa640 1 --2- 172.21.15.77:0/4006536009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c1614d0 0x7f8a4c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a50fa9640 1 -- 172.21.15.77:0/4006536009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c1614d0 msgr2=0x7f8a4c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a50fa9640 1 --2- 172.21.15.77:0/4006536009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c1614d0 0x7f8a4c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a50fa9640 1 -- 172.21.15.77:0/4006536009 >> v1:172.21.15.158:6789/0 conn(0x7f8a4c0d22b0 legacy=0x7f8a4c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.074+0000 7f8a50fa9640 1 -- 172.21.15.77:0/4006536009 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0cf6a0 con 0x7f8a4c0d1910 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.075+0000 7f8a50fa9640 1 --2- 172.21.15.77:0/4006536009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c0d1d70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009870 tx=0x7f8a3c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10deb1de556498c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.075+0000 7f8a43fff640 1 -- 172.21.15.77:0/4006536009 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c035020 con 0x7f8a4c0d1910 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.075+0000 7f8a43fff640 1 -- 172.21.15.77:0/4006536009 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c031640 con 0x7f8a4c0d1910 2024-03-06T07:25:29.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.075+0000 7f8a43fff640 1 -- 172.21.15.77:0/4006536009 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c02d040 con 0x7f8a4c0d1910 2024-03-06T07:25:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.076+0000 7f8a53a0e640 1 -- 172.21.15.77:0/4006536009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 msgr2=0x7f8a4c0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.076+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/4006536009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c0d1d70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009870 tx=0x7f8a3c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.076+0000 7f8a53a0e640 1 -- 172.21.15.77:0/4006536009 shutdown_connections 2024-03-06T07:25:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.076+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/4006536009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c0d1d70 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.076+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/4006536009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c1614d0 0x7f8a4c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.076+0000 7f8a53a0e640 1 -- 172.21.15.77:0/4006536009 >> 172.21.15.77:0/4006536009 conn(0x7f8a4c0d3d40 msgr2=0x7f8a4c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.076+0000 7f8a53a0e640 1 -- 172.21.15.77:0/4006536009 shutdown_connections 2024-03-06T07:25:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.076+0000 7f8a53a0e640 1 -- 172.21.15.77:0/4006536009 wait complete. 2024-03-06T07:25:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.078+0000 7f8a53a0e640 1 Processor -- start 2024-03-06T07:25:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.078+0000 7f8a53a0e640 1 -- start start 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a53a0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a53a0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c0d22b0 0x7f8a4c0b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a53a0e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a4c1614d0 0x7f8a4c0b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a53a0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0b0910 con 0x7f8a4c0d1910 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a53a0e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0cfc60 con 0x7f8a4c1614d0 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a53a0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0b1e40 con 0x7f8a4c0d22b0 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48058/0 (socket says 172.21.15.77:48058) 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 -- 172.21.15.77:0/2027280956 learned_addr learned my addr 172.21.15.77:0/2027280956 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 -- 172.21.15.77:0/2027280956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c0d22b0 msgr2=0x7f8a4c0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c0d22b0 0x7f8a4c0b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 -- 172.21.15.77:0/2027280956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a4c1614d0 msgr2=0x7f8a4c0b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a51fab640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a4c1614d0 0x7f8a4c0b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a4c1614d0 0x7f8a4c0b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 -- 172.21.15.77:0/2027280956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c1f76d0 con 0x7f8a4c0d1910 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.079+0000 7f8a517aa640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c150110 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800c7f0 tx=0x7f8a4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.080+0000 7f8a41ffb640 1 -- 172.21.15.77:0/2027280956 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48015070 con 0x7f8a4c0d1910 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.080+0000 7f8a41ffb640 1 -- 172.21.15.77:0/2027280956 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a48004220 con 0x7f8a4c0d1910 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.080+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c1f7970 con 0x7f8a4c0d1910 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.080+0000 7f8a41ffb640 1 -- 172.21.15.77:0/2027280956 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48007620 con 0x7f8a4c0d1910 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.080+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c1f7f30 con 0x7f8a4c0d1910 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.081+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c15f890 con 0x7f8a4c0d1910 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.081+0000 7f8a41ffb640 1 -- 172.21.15.77:0/2027280956 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a480079d0 con 0x7f8a4c0d1910 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.082+0000 7f8a41ffb640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a3006b500 0x7f8a3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.082+0000 7f8a41ffb640 1 -- 172.21.15.77:0/2027280956 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a4808a1f0 con 0x7f8a4c0d1910 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.082+0000 7f8a50fa9640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a3006b500 0x7f8a3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.082+0000 7f8a50fa9640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a3006b500 0x7f8a3006d9c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c036000 tx=0x7f8a3c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.086+0000 7f8a41ffb640 1 -- 172.21.15.77:0/2027280956 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a48059070 con 0x7f8a4c0d1910 2024-03-06T07:25:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.264+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a4c0c75d0 con 0x7f8a4c0d1910 2024-03-06T07:25:29.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.264+0000 7f8a41ffb640 1 -- 172.21.15.77:0/2027280956 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a48058970 con 0x7f8a4c0d1910 2024-03-06T07:25:29.266 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:25:29.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a3006b500 msgr2=0x7f8a3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a3006b500 0x7f8a3006d9c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c036000 tx=0x7f8a3c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 msgr2=0x7f8a4c150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c150110 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800c7f0 tx=0x7f8a4800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 shutdown_connections 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a4c1614d0 0x7f8a4c0b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a4c0d22b0 0x7f8a4c0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a3006b500 0x7f8a3006d9c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 --2- 172.21.15.77:0/2027280956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a4c0d1910 0x7f8a4c150110 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 >> 172.21.15.77:0/2027280956 conn(0x7f8a4c0d3d40 msgr2=0x7f8a4c15fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 shutdown_connections 2024-03-06T07:25:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.266+0000 7f8a53a0e640 1 -- 172.21.15.77:0/2027280956 wait complete. 2024-03-06T07:25:29.279 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:29.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.380+0000 7f5dc041c640 1 Processor -- start 2024-03-06T07:25:29.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.380+0000 7f5dc041c640 1 -- start start 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dc041c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5db81614d0 0x7f5db80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dc041c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dc041c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db81618b0 con 0x7f5db80d1910 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dc041c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db80cf400 con 0x7f5db81614d0 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dc041c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db80cf6a0 con 0x7f5db80d24b0 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dbe9b9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dbe1b8640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5db81614d0 0x7f5db80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dbd9b7640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5db80d1910 0x7f5db80d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:48090/0 (socket says 172.21.15.77:48090) 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.381+0000 7f5dbd9b7640 1 -- 172.21.15.77:0/23062771 learned_addr learned my addr 172.21.15.77:0/23062771 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:29.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.382+0000 7f5dbd1b6640 1 -- 172.21.15.77:0/23062771 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487258539 0 0) 0x7f5db81618b0 con 0x7f5db80d1910 2024-03-06T07:25:29.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.382+0000 7f5dbd1b6640 1 -- 172.21.15.77:0/23062771 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da4003520 con 0x7f5db80d1910 2024-03-06T07:25:29.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.382+0000 7f5dbe9b9640 1 -- 172.21.15.77:0/23062771 >> v2:172.21.15.158:3300/0 conn(0x7f5db81614d0 msgr2=0x7f5db80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.382+0000 7f5dbe9b9640 1 --2- 172.21.15.77:0/23062771 >> v2:172.21.15.158:3300/0 conn(0x7f5db81614d0 0x7f5db80d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.382+0000 7f5dbe9b9640 1 -- 172.21.15.77:0/23062771 >> v1:172.21.15.158:6789/0 conn(0x7f5db80d1910 legacy=0x7f5db80d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.382+0000 7f5dbe9b9640 1 -- 172.21.15.77:0/23062771 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db80cf980 con 0x7f5db80d24b0 2024-03-06T07:25:29.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.382+0000 7f5dbe1b8640 1 --2- 172.21.15.77:0/23062771 >> v2:172.21.15.158:3300/0 conn(0x7f5db81614d0 0x7f5db80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:29.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.383+0000 7f5dbe9b9640 1 --2- 172.21.15.77:0/23062771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b0e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5db0009d90 tx=0x7f5db002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=849a07a0266f67cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.383+0000 7f5dbd1b6640 1 -- 172.21.15.77:0/23062771 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0039070 con 0x7f5db80d24b0 2024-03-06T07:25:29.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.383+0000 7f5dbd1b6640 1 -- 172.21.15.77:0/23062771 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db002cc50 con 0x7f5db80d24b0 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.383+0000 7f5dbd1b6640 1 -- 172.21.15.77:0/23062771 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0034440 con 0x7f5db80d24b0 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.384+0000 7f5dc041c640 1 -- 172.21.15.77:0/23062771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 msgr2=0x7f5db80b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.384+0000 7f5dc041c640 1 --2- 172.21.15.77:0/23062771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b0e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5db0009d90 tx=0x7f5db002bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.384+0000 7f5dc041c640 1 -- 172.21.15.77:0/23062771 shutdown_connections 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.384+0000 7f5dc041c640 1 --2- 172.21.15.77:0/23062771 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b0e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.384+0000 7f5dc041c640 1 --2- 172.21.15.77:0/23062771 >> v2:172.21.15.158:3300/0 conn(0x7f5db81614d0 0x7f5db80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.384+0000 7f5dc041c640 1 -- 172.21.15.77:0/23062771 >> 172.21.15.77:0/23062771 conn(0x7f5db80d3d40 msgr2=0x7f5db80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.384+0000 7f5dc041c640 1 -- 172.21.15.77:0/23062771 shutdown_connections 2024-03-06T07:25:29.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.384+0000 7f5dc041c640 1 -- 172.21.15.77:0/23062771 wait complete. 2024-03-06T07:25:29.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.385+0000 7f5dc041c640 1 Processor -- start 2024-03-06T07:25:29.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.385+0000 7f5dc041c640 1 -- start start 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dc041c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db80d1910 0x7f5db81532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dc041c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dc041c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db81614d0 0x7f5db80b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dc041c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db80b1370 con 0x7f5db81614d0 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dc041c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db80b0910 con 0x7f5db80d1910 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dc041c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db80b14f0 con 0x7f5db80d24b0 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db81614d0 0x7f5db80b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db81614d0 0x7f5db80b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48068/0 (socket says 172.21.15.77:48068) 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 -- 172.21.15.77:0/3675885065 learned_addr learned my addr 172.21.15.77:0/3675885065 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe1b8640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db80d1910 0x7f5db81532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbd9b7640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 -- 172.21.15.77:0/3675885065 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 msgr2=0x7f5db80b8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 -- 172.21.15.77:0/3675885065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db80d1910 msgr2=0x7f5db81532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db80d1910 0x7f5db81532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 -- 172.21.15.77:0/3675885065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db80cbb20 con 0x7f5db81614d0 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.386+0000 7f5dbe9b9640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db81614d0 0x7f5db80b9110 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5db0009d90 tx=0x7f5db0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.387+0000 7f5daaffd640 1 -- 172.21.15.77:0/3675885065 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0039050 con 0x7f5db81614d0 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.387+0000 7f5daaffd640 1 -- 172.21.15.77:0/3675885065 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db0045070 con 0x7f5db81614d0 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.387+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db80cbe00 con 0x7f5db81614d0 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.387+0000 7f5daaffd640 1 -- 172.21.15.77:0/3675885065 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0009450 con 0x7f5db81614d0 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.387+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db8153810 con 0x7f5db81614d0 2024-03-06T07:25:29.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.388+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db80b55b0 con 0x7f5db81614d0 2024-03-06T07:25:29.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.389+0000 7f5daaffd640 1 -- 172.21.15.77:0/3675885065 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5db003e610 con 0x7f5db81614d0 2024-03-06T07:25:29.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.389+0000 7f5daaffd640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9c06b4d0 0x7f5d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.389+0000 7f5daaffd640 1 -- 172.21.15.77:0/3675885065 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5db00acaa0 con 0x7f5db81614d0 2024-03-06T07:25:29.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.389+0000 7f5dbe1b8640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9c06b4d0 0x7f5d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.390+0000 7f5dbe1b8640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9c06b4d0 0x7f5d9c06d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009560 tx=0x7f5db4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.393+0000 7f5daaffd640 1 -- 172.21.15.77:0/3675885065 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5db007b2d0 con 0x7f5db81614d0 2024-03-06T07:25:29.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.570+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5db8068ff0 con 0x7f5db81614d0 2024-03-06T07:25:29.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.571+0000 7f5daaffd640 1 -- 172.21.15.77:0/3675885065 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5db007f370 con 0x7f5db81614d0 2024-03-06T07:25:29.572 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9c06b4d0 msgr2=0x7f5d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9c06b4d0 0x7f5d9c06d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009560 tx=0x7f5db4009190 comp rx=0 tx=0).stop 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db81614d0 msgr2=0x7f5db80b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db81614d0 0x7f5db80b9110 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5db0009d90 tx=0x7f5db0004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 shutdown_connections 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9c06b4d0 0x7f5d9c06d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db81614d0 0x7f5db80b9110 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db80d24b0 0x7f5db80b8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 --2- 172.21.15.77:0/3675885065 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db80d1910 0x7f5db81532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 >> 172.21.15.77:0/3675885065 conn(0x7f5db80d3d40 msgr2=0x7f5db806b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 shutdown_connections 2024-03-06T07:25:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.573+0000 7f5dc041c640 1 -- 172.21.15.77:0/3675885065 wait complete. 2024-03-06T07:25:29.589 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:29.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.690+0000 7f7f1ad13640 1 Processor -- start 2024-03-06T07:25:29.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.690+0000 7f7f1ad13640 1 -- start start 2024-03-06T07:25:29.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f1ad13640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7f141613f0 0x7f7f140cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f1ad13640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140b5ef0 con 0x7f7f140b5320 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f1ad13640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140d0b50 con 0x7f7f141617d0 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f1ad13640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f1406c0a0 con 0x7f7f141613f0 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f192b0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7f140b5320 0x7f7f1406d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46080/0 (socket says 172.21.15.77:46080) 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f192b0640 1 -- 172.21.15.77:0/3479905062 learned_addr learned my addr 172.21.15.77:0/3479905062 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f18aaf640 1 --2- 172.21.15.77:0/3479905062 >> v2:172.21.15.158:3300/0 conn(0x7f7f141613f0 0x7f7f140cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f137fe640 1 -- 172.21.15.77:0/3479905062 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257595222 0 0) 0x7f7f140b5ef0 con 0x7f7f140b5320 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f137fe640 1 -- 172.21.15.77:0/3479905062 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f04003520 con 0x7f7f140b5320 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f137fe640 1 -- 172.21.15.77:0/3479905062 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687383749 0 0) 0x7f7f140d0b50 con 0x7f7f141617d0 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f137fe640 1 -- 172.21.15.77:0/3479905062 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f140b5ef0 con 0x7f7f141617d0 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f137fe640 1 -- 172.21.15.77:0/3479905062 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522632963 0 0) 0x7f7f04003520 con 0x7f7f140b5320 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f18aaf640 1 -- 172.21.15.77:0/3479905062 >> v1:172.21.15.158:6789/0 conn(0x7f7f141617d0 legacy=0x7f7f140ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f18aaf640 1 -- 172.21.15.77:0/3479905062 >> v1:172.21.15.77:6789/0 conn(0x7f7f140b5320 legacy=0x7f7f1406d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.691+0000 7f7f18aaf640 1 -- 172.21.15.77:0/3479905062 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140b5bf0 con 0x7f7f141613f0 2024-03-06T07:25:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.692+0000 7f7f18aaf640 1 --2- 172.21.15.77:0/3479905062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141613f0 0x7f7f140cca20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7f080097a0 tx=0x7f7f0802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ddcfed6584a1fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f137fe640 1 -- 172.21.15.77:0/3479905062 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08038070 con 0x7f7f141613f0 2024-03-06T07:25:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f137fe640 1 -- 172.21.15.77:0/3479905062 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f08031500 con 0x7f7f141613f0 2024-03-06T07:25:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f137fe640 1 -- 172.21.15.77:0/3479905062 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c0019f0 con 0x7f7f141613f0 2024-03-06T07:25:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3479905062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141613f0 msgr2=0x7f7f140cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f1ad13640 1 --2- 172.21.15.77:0/3479905062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141613f0 0x7f7f140cca20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7f080097a0 tx=0x7f7f0802b820 comp rx=0 tx=0).stop 2024-03-06T07:25:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3479905062 shutdown_connections 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f1ad13640 1 --2- 172.21.15.77:0/3479905062 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141613f0 0x7f7f140cca20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3479905062 >> 172.21.15.77:0/3479905062 conn(0x7f7f14072520 msgr2=0x7f7f14072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3479905062 shutdown_connections 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.693+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3479905062 wait complete. 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.694+0000 7f7f1ad13640 1 Processor -- start 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.694+0000 7f7f1ad13640 1 -- start start 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f1ad13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f140b5320 0x7f7f140c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f1ad13640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f141613f0 0x7f7f140b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f1ad13640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141617d0 0x7f7f140b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f1ad13640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140cd7f0 con 0x7f7f140b5320 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f1ad13640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140cd670 con 0x7f7f141617d0 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f1ad13640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1406dff0 con 0x7f7f141613f0 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f18aaf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f140b5320 0x7f7f140c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f18aaf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f140b5320 0x7f7f140c5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48082/0 (socket says 172.21.15.77:48082) 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.695+0000 7f7f18aaf640 1 -- 172.21.15.77:0/3346242938 learned_addr learned my addr 172.21.15.77:0/3346242938 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f13fff640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f141613f0 0x7f7f140b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f192b0640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141617d0 0x7f7f140b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f18aaf640 1 -- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f141613f0 msgr2=0x7f7f140b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f18aaf640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f141613f0 0x7f7f140b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f18aaf640 1 -- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141617d0 msgr2=0x7f7f140b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f18aaf640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141617d0 0x7f7f140b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f18aaf640 1 -- 172.21.15.77:0/3346242938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140b18d0 con 0x7f7f140b5320 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f13fff640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f141613f0 0x7f7f140b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f18aaf640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f140b5320 0x7f7f140c5630 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7f08036040 tx=0x7f7f080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f117fa640 1 -- 172.21.15.77:0/3346242938 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08042070 con 0x7f7f140b5320 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f117fa640 1 -- 172.21.15.77:0/3346242938 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0803d870 con 0x7f7f140b5320 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f140b1bb0 con 0x7f7f140b5320 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f117fa640 1 -- 172.21.15.77:0/3346242938 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08038040 con 0x7f7f140b5320 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.696+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f140b2170 con 0x7f7f140b5320 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.697+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1404f330 con 0x7f7f140b5320 2024-03-06T07:25:29.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.698+0000 7f7f117fa640 1 -- 172.21.15.77:0/3346242938 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f08045070 con 0x7f7f140b5320 2024-03-06T07:25:29.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.698+0000 7f7f117fa640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ef806b4d0 0x7f7ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.699+0000 7f7f13fff640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ef806b4d0 0x7f7ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.699+0000 7f7f117fa640 1 -- 172.21.15.77:0/3346242938 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f080abc80 con 0x7f7f140b5320 2024-03-06T07:25:29.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.699+0000 7f7f13fff640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ef806b4d0 0x7f7ef806d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7f1415d880 tx=0x7f7f00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.703+0000 7f7f117fa640 1 -- 172.21.15.77:0/3346242938 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f08030070 con 0x7f7f140b5320 2024-03-06T07:25:29.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.891+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7f1415ebf0 con 0x7f7f140b5320 2024-03-06T07:25:29.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.892+0000 7f7f117fa640 1 -- 172.21.15.77:0/3346242938 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7f0807aa80 con 0x7f7f140b5320 2024-03-06T07:25:29.894 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:25:29.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.893+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ef806b4d0 msgr2=0x7f7ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.893+0000 7f7f1ad13640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ef806b4d0 0x7f7ef806d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7f1415d880 tx=0x7f7f00009040 comp rx=0 tx=0).stop 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f140b5320 msgr2=0x7f7f140c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f140b5320 0x7f7f140c5630 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7f08036040 tx=0x7f7f080046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 shutdown_connections 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f141617d0 0x7f7f140b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f141613f0 0x7f7f140b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7ef806b4d0 0x7f7ef806d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 --2- 172.21.15.77:0/3346242938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f140b5320 0x7f7f140c5630 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 >> 172.21.15.77:0/3346242938 conn(0x7f7f14072520 msgr2=0x7f7f14072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 shutdown_connections 2024-03-06T07:25:29.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:29.894+0000 7f7f1ad13640 1 -- 172.21.15.77:0/3346242938 wait complete. 2024-03-06T07:25:29.906 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:25:29.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:29.906 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:29.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[62453]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:29.933 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:25:29.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:29.933 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:30.098 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:25:30.098 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:30.116 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:30.116 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:25:30.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.117 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:30.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:30.172 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:30.173 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:30.173 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:30.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:30.176 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:30.188 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:30.188 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:25:30.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.189 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:30.274 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:30.275 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:25:30.275 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:30.292 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:30.292 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:25:30.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:30.292 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:30.348 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:30.349 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.448+0000 7fc4f2b1e640 1 Processor -- start 2024-03-06T07:25:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.448+0000 7fc4f2b1e640 1 -- start start 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f2b1e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4ec15dfe0 0x7fc4ec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f2b1e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4ec0cf830 0x7fc4ec0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f2b1e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec0b3430 con 0x7fc4ec0d27f0 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f2b1e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec0b31e0 con 0x7fc4ec0cf830 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f2b1e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec0b3d60 con 0x7fc4ec15dfe0 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4ebfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4ec0cf830 0x7fc4ec0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f10bb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc4ec0d27f0 0x7fc4ec0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46092/0 (socket says 172.21.15.77:46092) 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f10bb640 1 -- 172.21.15.77:0/372514202 learned_addr learned my addr 172.21.15.77:0/372514202 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4ebfff640 1 -- 172.21.15.77:0/372514202 >> v2:172.21.15.158:3300/0 conn(0x7fc4ec15dfe0 msgr2=0x7fc4ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f08ba640 1 --2- 172.21.15.77:0/372514202 >> v2:172.21.15.158:3300/0 conn(0x7fc4ec15dfe0 0x7fc4ec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4ebfff640 1 --2- 172.21.15.77:0/372514202 >> v2:172.21.15.158:3300/0 conn(0x7fc4ec15dfe0 0x7fc4ec0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4ebfff640 1 -- 172.21.15.77:0/372514202 >> v1:172.21.15.77:6789/0 conn(0x7fc4ec0d27f0 legacy=0x7fc4ec0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4eb7fe640 1 -- 172.21.15.77:0/372514202 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161419027 0 0) 0x7fc4ec0b3430 con 0x7fc4ec0d27f0 2024-03-06T07:25:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4ebfff640 1 -- 172.21.15.77:0/372514202 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0b3710 con 0x7fc4ec0cf830 2024-03-06T07:25:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.449+0000 7fc4f08ba640 1 --2- 172.21.15.77:0/372514202 >> v2:172.21.15.158:3300/0 conn(0x7fc4ec15dfe0 0x7fc4ec0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.450+0000 7fc4ebfff640 1 --2- 172.21.15.77:0/372514202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec0d22b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc4e00097a0 tx=0x7fc4e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67732973d581c83e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.450+0000 7fc4eb7fe640 1 -- 172.21.15.77:0/372514202 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0035020 con 0x7fc4ec0cf830 2024-03-06T07:25:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.450+0000 7fc4eb7fe640 1 -- 172.21.15.77:0/372514202 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e0031830 con 0x7fc4ec0cf830 2024-03-06T07:25:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.450+0000 7fc4eb7fe640 1 -- 172.21.15.77:0/372514202 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0031c00 con 0x7fc4ec0cf830 2024-03-06T07:25:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.451+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/372514202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 msgr2=0x7fc4ec0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.451+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/372514202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec0d22b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc4e00097a0 tx=0x7fc4e002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.451+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/372514202 shutdown_connections 2024-03-06T07:25:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.451+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/372514202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec0d22b0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.451+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/372514202 >> v2:172.21.15.158:3300/0 conn(0x7fc4ec15dfe0 0x7fc4ec0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.451+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/372514202 >> 172.21.15.77:0/372514202 conn(0x7fc4ec0d3cd0 msgr2=0x7fc4ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.451+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/372514202 shutdown_connections 2024-03-06T07:25:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.451+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/372514202 wait complete. 2024-03-06T07:25:30.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.452+0000 7fc4f2b1e640 1 Processor -- start 2024-03-06T07:25:30.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.453+0000 7fc4f2b1e640 1 -- start start 2024-03-06T07:25:30.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f2b1e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec15be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f2b1e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4ec0d27f0 0x7fc4ec15e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f2b1e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4ec15dfe0 0x7fc4ec160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f2b1e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0d1df0 con 0x7fc4ec0cf830 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f2b1e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0d1c70 con 0x7fc4ec15dfe0 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f2b1e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15e3c0 con 0x7fc4ec0d27f0 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f08ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f08ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec15be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48098/0 (socket says 172.21.15.77:48098) 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f08ba640 1 -- 172.21.15.77:0/1721350959 learned_addr learned my addr 172.21.15.77:0/1721350959 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4ebfff640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4ec0d27f0 0x7fc4ec15e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f10bb640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4ec15dfe0 0x7fc4ec160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f08ba640 1 -- 172.21.15.77:0/1721350959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4ec0d27f0 msgr2=0x7fc4ec15e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f08ba640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4ec0d27f0 0x7fc4ec15e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f08ba640 1 -- 172.21.15.77:0/1721350959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4ec15dfe0 msgr2=0x7fc4ec160c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f08ba640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4ec15dfe0 0x7fc4ec160c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.454+0000 7fc4f08ba640 1 -- 172.21.15.77:0/1721350959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec15f020 con 0x7fc4ec0cf830 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.455+0000 7fc4f08ba640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec15be10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc4e400cf60 tx=0x7fc4e400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.455+0000 7fc4e97fa640 1 -- 172.21.15.77:0/1721350959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e401a070 con 0x7fc4ec0cf830 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.455+0000 7fc4e97fa640 1 -- 172.21.15.77:0/1721350959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e400d840 con 0x7fc4ec0cf830 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.455+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ec15f300 con 0x7fc4ec0cf830 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.455+0000 7fc4e97fa640 1 -- 172.21.15.77:0/1721350959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400dc10 con 0x7fc4ec0cf830 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.455+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec15f8c0 con 0x7fc4ec0cf830 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.456+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ec04f330 con 0x7fc4ec0cf830 2024-03-06T07:25:30.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.456+0000 7fc4e97fa640 1 -- 172.21.15.77:0/1721350959 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4e401d070 con 0x7fc4ec0cf830 2024-03-06T07:25:30.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.457+0000 7fc4e97fa640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4d806b4d0 0x7fc4d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.457+0000 7fc4e97fa640 1 -- 172.21.15.77:0/1721350959 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4e408aa10 con 0x7fc4ec0cf830 2024-03-06T07:25:30.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.457+0000 7fc4ebfff640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4d806b4d0 0x7fc4d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.457+0000 7fc4ebfff640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4d806b4d0 0x7fc4d806d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0036000 tx=0x7fc4e003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.461+0000 7fc4e97fa640 1 -- 172.21.15.77:0/1721350959 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4e4059240 con 0x7fc4ec0cf830 2024-03-06T07:25:30.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.676+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4ec0a1af0 con 0x7fc4ec0cf830 2024-03-06T07:25:30.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.676+0000 7fc4e97fa640 1 -- 172.21.15.77:0/1721350959 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fc4e405d2e0 con 0x7fc4ec0cf830 2024-03-06T07:25:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:30.678 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:30.679 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4d806b4d0 msgr2=0x7fc4d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4d806b4d0 0x7fc4d806d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0036000 tx=0x7fc4e003c040 comp rx=0 tx=0).stop 2024-03-06T07:25:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 msgr2=0x7fc4ec15be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec15be10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc4e400cf60 tx=0x7fc4e400cf90 comp rx=0 tx=0).stop 2024-03-06T07:25:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 shutdown_connections 2024-03-06T07:25:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4ec15dfe0 0x7fc4ec160c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4ec0d27f0 0x7fc4ec15e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4d806b4d0 0x7fc4d806d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 --2- 172.21.15.77:0/1721350959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ec0cf830 0x7fc4ec15be10 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 >> 172.21.15.77:0/1721350959 conn(0x7fc4ec0d3cd0 msgr2=0x7fc4ec0ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 shutdown_connections 2024-03-06T07:25:30.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.678+0000 7fc4f2b1e640 1 -- 172.21.15.77:0/1721350959 wait complete. 2024-03-06T07:25:30.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.812+0000 7feac4242640 1 Processor -- start 2024-03-06T07:25:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.812+0000 7feac4242640 1 -- start start 2024-03-06T07:25:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac4242640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feabc15dfe0 0x7feabc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac4242640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc0b3430 con 0x7feabc0cf830 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac4242640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc0b3710 con 0x7feabc0d29c0 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac4242640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feabc0b31e0 con 0x7feabc15dfe0 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac1fde640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feabc15dfe0 0x7feabc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac1fde640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feabc15dfe0 0x7feabc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48110/0 (socket says 172.21.15.77:48110) 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac1fde640 1 -- 172.21.15.77:0/1934699457 learned_addr learned my addr 172.21.15.77:0/1934699457 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac17dd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7feabc0cf830 0x7feabc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46096/0 (socket says 172.21.15.77:46096) 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.813+0000 7feac1fde640 1 -- 172.21.15.77:0/1934699457 >> v1:172.21.15.158:6789/0 conn(0x7feabc0d29c0 legacy=0x7feabc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.814+0000 7feac1fde640 1 -- 172.21.15.77:0/1934699457 >> v1:172.21.15.77:6789/0 conn(0x7feabc0cf830 legacy=0x7feabc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.814+0000 7feac1fde640 1 -- 172.21.15.77:0/1934699457 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc0b39f0 con 0x7feabc15dfe0 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.814+0000 7feac0fdc640 1 -- 172.21.15.77:0/1934699457 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898612247 0 0) 0x7feabc0b3430 con 0x7feabc0cf830 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.814+0000 7feac1fde640 1 --2- 172.21.15.77:0/1934699457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc0cf2f0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7feabc0b3360 tx=0x7feab402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1afcdf5b36e751 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.814+0000 7feac0fdc640 1 -- 172.21.15.77:0/1934699457 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4034070 con 0x7feabc15dfe0 2024-03-06T07:25:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.814+0000 7feac0fdc640 1 -- 172.21.15.77:0/1934699457 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab4030840 con 0x7feabc15dfe0 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.814+0000 7feac0fdc640 1 -- 172.21.15.77:0/1934699457 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4030c10 con 0x7feabc15dfe0 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.815+0000 7feac4242640 1 -- 172.21.15.77:0/1934699457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 msgr2=0x7feabc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.815+0000 7feac4242640 1 --2- 172.21.15.77:0/1934699457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc0cf2f0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7feabc0b3360 tx=0x7feab402b980 comp rx=0 tx=0).stop 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.815+0000 7feac4242640 1 -- 172.21.15.77:0/1934699457 shutdown_connections 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.815+0000 7feac4242640 1 --2- 172.21.15.77:0/1934699457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc0cf2f0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.815+0000 7feac4242640 1 -- 172.21.15.77:0/1934699457 >> 172.21.15.77:0/1934699457 conn(0x7feabc0d3cd0 msgr2=0x7feabc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.815+0000 7feac4242640 1 -- 172.21.15.77:0/1934699457 shutdown_connections 2024-03-06T07:25:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.815+0000 7feac4242640 1 -- 172.21.15.77:0/1934699457 wait complete. 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.816+0000 7feac4242640 1 Processor -- start 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.816+0000 7feac4242640 1 -- start start 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac4242640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feabc0cf830 0x7feabc1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac4242640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feabc0d29c0 0x7feabc0cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac4242640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc06c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac4242640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc15e3c0 con 0x7feabc15dfe0 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac4242640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0b3d60 con 0x7feabc0cf830 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac4242640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc15e540 con 0x7feabc0d29c0 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac27df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc06c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac27df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc06c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48124/0 (socket says 172.21.15.77:48124) 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac27df640 1 -- 172.21.15.77:0/2149604539 learned_addr learned my addr 172.21.15.77:0/2149604539 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.817+0000 7feac17dd640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feabc0d29c0 0x7feabc0cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac27df640 1 -- 172.21.15.77:0/2149604539 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feabc0d29c0 msgr2=0x7feabc0cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac27df640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feabc0d29c0 0x7feabc0cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac1fde640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feabc0cf830 0x7feabc1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac27df640 1 -- 172.21.15.77:0/2149604539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feabc0cf830 msgr2=0x7feabc1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac27df640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feabc0cf830 0x7feabc1524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac27df640 1 -- 172.21.15.77:0/2149604539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc152d30 con 0x7feabc15dfe0 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac27df640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc06c180 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7feab800c7b0 tx=0x7feab800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feab27fc640 1 -- 172.21.15.77:0/2149604539 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8017070 con 0x7feabc15dfe0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feab27fc640 1 -- 172.21.15.77:0/2149604539 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab8003da0 con 0x7feabc15dfe0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc153010 con 0x7feabc15dfe0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feab27fc640 1 -- 172.21.15.77:0/2149604539 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab800ce60 con 0x7feabc15dfe0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.818+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc0cbca0 con 0x7feabc15dfe0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.819+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc04f330 con 0x7feabc15dfe0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.820+0000 7feab27fc640 1 -- 172.21.15.77:0/2149604539 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feab801a070 con 0x7feabc15dfe0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.820+0000 7feab27fc640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feaa006b4d0 0x7feaa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.820+0000 7feab27fc640 1 -- 172.21.15.77:0/2149604539 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feab8089830 con 0x7feabc15dfe0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.820+0000 7feac1fde640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feaa006b4d0 0x7feaa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.820+0000 7feac1fde640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feaa006b4d0 0x7feaa006d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7feab4036000 tx=0x7feab4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.824+0000 7feab27fc640 1 -- 172.21.15.77:0/2149604539 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feab80586b0 con 0x7feabc15dfe0 2024-03-06T07:25:30.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.997+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feabc0d2570 con 0x7feabc15dfe0 2024-03-06T07:25:31.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:30.998+0000 7feab27fc640 1 -- 172.21.15.77:0/2149604539 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16656 (secure 0 0 0) 0x7feab8057fb0 con 0x7feabc15dfe0 2024-03-06T07:25:31.000 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:31.001 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":69,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:00.275386+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:56.868266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:57.595004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:31.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feaa006b4d0 msgr2=0x7feaa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feaa006b4d0 0x7feaa006d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7feab4036000 tx=0x7feab4008030 comp rx=0 tx=0).stop 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 msgr2=0x7feabc06c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc06c180 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7feab800c7b0 tx=0x7feab800cc70 comp rx=0 tx=0).stop 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 shutdown_connections 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feaa006b4d0 0x7feaa006d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feabc15dfe0 0x7feabc06c180 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feabc0d29c0 0x7feabc0cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 --2- 172.21.15.77:0/2149604539 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feabc0cf830 0x7feabc1524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 >> 172.21.15.77:0/2149604539 conn(0x7feabc0d3cd0 msgr2=0x7feabc0b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 shutdown_connections 2024-03-06T07:25:31.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.000+0000 7feac4242640 1 -- 172.21.15.77:0/2149604539 wait complete. 2024-03-06T07:25:31.017 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:56.868266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:57.595004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:31.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.119+0000 7ff930dc7640 1 Processor -- start 2024-03-06T07:25:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.119+0000 7ff930dc7640 1 -- start start 2024-03-06T07:25:31.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff930dc7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff92c1614d0 0x7ff92c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff930dc7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff92c0d1910 0x7ff92c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff930dc7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c1618b0 con 0x7ff92c0d22e0 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff930dc7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c0cf400 con 0x7ff92c0d1910 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff930dc7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c0d2710 con 0x7ff92c1614d0 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff929d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff92c0d1910 0x7ff92c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff92ad9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff92c0d22e0 0x7ff92c0d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46098/0 (socket says 172.21.15.77:46098) 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff929d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff92c0d1910 0x7ff92c0d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48140/0 (socket says 172.21.15.77:48140) 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff92ad9d640 1 -- 172.21.15.77:0/1945233380 learned_addr learned my addr 172.21.15.77:0/1945233380 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff92a59c640 1 --2- 172.21.15.77:0/1945233380 >> v2:172.21.15.158:3300/0 conn(0x7ff92c1614d0 0x7ff92c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff929d9b640 1 -- 172.21.15.77:0/1945233380 >> v2:172.21.15.158:3300/0 conn(0x7ff92c1614d0 msgr2=0x7ff92c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff929d9b640 1 --2- 172.21.15.77:0/1945233380 >> v2:172.21.15.158:3300/0 conn(0x7ff92c1614d0 0x7ff92c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff929d9b640 1 -- 172.21.15.77:0/1945233380 >> v1:172.21.15.77:6789/0 conn(0x7ff92c0d22e0 legacy=0x7ff92c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff929d9b640 1 -- 172.21.15.77:0/1945233380 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0cf6a0 con 0x7ff92c0d1910 2024-03-06T07:25:31.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff92959a640 1 -- 172.21.15.77:0/1945233380 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340431732 0 0) 0x7ff92c1618b0 con 0x7ff92c0d22e0 2024-03-06T07:25:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.121+0000 7ff92a59c640 1 --2- 172.21.15.77:0/1945233380 >> v2:172.21.15.158:3300/0 conn(0x7ff92c1614d0 0x7ff92c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.122+0000 7ff929d9b640 1 --2- 172.21.15.77:0/1945233380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c0d1da0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff92c08c470 tx=0x7ff92002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ae7bd02fcd9c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.122+0000 7ff92959a640 1 -- 172.21.15.77:0/1945233380 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920035020 con 0x7ff92c0d1910 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.122+0000 7ff92959a640 1 -- 172.21.15.77:0/1945233380 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff920031830 con 0x7ff92c0d1910 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.122+0000 7ff92959a640 1 -- 172.21.15.77:0/1945233380 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920031c00 con 0x7ff92c0d1910 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.123+0000 7ff930dc7640 1 -- 172.21.15.77:0/1945233380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 msgr2=0x7ff92c0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.123+0000 7ff930dc7640 1 --2- 172.21.15.77:0/1945233380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c0d1da0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff92c08c470 tx=0x7ff92002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.123+0000 7ff930dc7640 1 -- 172.21.15.77:0/1945233380 shutdown_connections 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.123+0000 7ff930dc7640 1 --2- 172.21.15.77:0/1945233380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c0d1da0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.123+0000 7ff930dc7640 1 --2- 172.21.15.77:0/1945233380 >> v2:172.21.15.158:3300/0 conn(0x7ff92c1614d0 0x7ff92c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.123+0000 7ff930dc7640 1 -- 172.21.15.77:0/1945233380 >> 172.21.15.77:0/1945233380 conn(0x7ff92c0d3d40 msgr2=0x7ff92c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.123+0000 7ff930dc7640 1 -- 172.21.15.77:0/1945233380 shutdown_connections 2024-03-06T07:25:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.123+0000 7ff930dc7640 1 -- 172.21.15.77:0/1945233380 wait complete. 2024-03-06T07:25:31.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.125+0000 7ff930dc7640 1 Processor -- start 2024-03-06T07:25:31.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.125+0000 7ff930dc7640 1 -- start start 2024-03-06T07:25:31.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff930dc7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c1596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff930dc7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff92c0d22e0 0x7ff92c1f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff930dc7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff92c1614d0 0x7ff92c0c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff930dc7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b0910 con 0x7ff92c0d1910 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff930dc7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0cfc60 con 0x7ff92c1614d0 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff930dc7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b1e40 con 0x7ff92c0d22e0 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff92a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c1596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff92a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c1596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48154/0 (socket says 172.21.15.77:48154) 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff92a59c640 1 -- 172.21.15.77:0/704853229 learned_addr learned my addr 172.21.15.77:0/704853229 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff92ad9d640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff92c1614d0 0x7ff92c0c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.126+0000 7ff929d9b640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff92c0d22e0 0x7ff92c1f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff92a59c640 1 -- 172.21.15.77:0/704853229 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff92c0d22e0 msgr2=0x7ff92c1f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff92a59c640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff92c0d22e0 0x7ff92c1f4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff92a59c640 1 -- 172.21.15.77:0/704853229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff92c1614d0 msgr2=0x7ff92c0c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff92a59c640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff92c1614d0 0x7ff92c0c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff92a59c640 1 -- 172.21.15.77:0/704853229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0c7870 con 0x7ff92c0d1910 2024-03-06T07:25:31.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff92a59c640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c1596f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff92400c7f0 tx=0x7ff92400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff91affd640 1 -- 172.21.15.77:0/704853229 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924015070 con 0x7ff92c0d1910 2024-03-06T07:25:31.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff91affd640 1 -- 172.21.15.77:0/704853229 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff924007820 con 0x7ff92c0d1910 2024-03-06T07:25:31.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c1f4870 con 0x7ff92c0d1910 2024-03-06T07:25:31.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff91affd640 1 -- 172.21.15.77:0/704853229 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924007bd0 con 0x7ff92c0d1910 2024-03-06T07:25:31.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.127+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c0c7df0 con 0x7ff92c0d1910 2024-03-06T07:25:31.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.128+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c068ff0 con 0x7ff92c0d1910 2024-03-06T07:25:31.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.128+0000 7ff91affd640 1 -- 172.21.15.77:0/704853229 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff924018070 con 0x7ff92c0d1910 2024-03-06T07:25:31.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.129+0000 7ff91affd640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff90c06b4d0 0x7ff90c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.129+0000 7ff929d9b640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff90c06b4d0 0x7ff90c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.129+0000 7ff91affd640 1 -- 172.21.15.77:0/704853229 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff92408a330 con 0x7ff92c0d1910 2024-03-06T07:25:31.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.129+0000 7ff929d9b640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff90c06b4d0 0x7ff90c06d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff920036000 tx=0x7ff920008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.133+0000 7ff91affd640 1 -- 172.21.15.77:0/704853229 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9240591b0 con 0x7ff92c0d1910 2024-03-06T07:25:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.310+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff92c0a1c10 con 0x7ff92c0d1910 2024-03-06T07:25:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.310+0000 7ff91affd640 1 -- 172.21.15.77:0/704853229 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff924058ab0 con 0x7ff92c0d1910 2024-03-06T07:25:31.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:25:31.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff90c06b4d0 msgr2=0x7ff90c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff90c06b4d0 0x7ff90c06d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff920036000 tx=0x7ff920008030 comp rx=0 tx=0).stop 2024-03-06T07:25:31.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 msgr2=0x7ff92c1596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c1596f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff92400c7f0 tx=0x7ff92400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 shutdown_connections 2024-03-06T07:25:31.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff92c1614d0 0x7ff92c0c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff92c0d22e0 0x7ff92c1f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff90c06b4d0 0x7ff90c06d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 --2- 172.21.15.77:0/704853229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff92c0d1910 0x7ff92c1596f0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 >> 172.21.15.77:0/704853229 conn(0x7ff92c0d3d40 msgr2=0x7ff92c06c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 shutdown_connections 2024-03-06T07:25:31.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.312+0000 7ff930dc7640 1 -- 172.21.15.77:0/704853229 wait complete. 2024-03-06T07:25:31.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:31.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.431+0000 7fe8012c9640 1 Processor -- start 2024-03-06T07:25:31.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.431+0000 7fe8012c9640 1 -- start start 2024-03-06T07:25:31.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.432+0000 7fe8012c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7fc0d06e0 0x7fe7fc15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.432+0000 7fe8012c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.432+0000 7fe8012c9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc0d47f0 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.432+0000 7fe8012c9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc0ced70 con 0x7fe7fc0d06e0 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.432+0000 7fe8012c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0d1ab0 con 0x7fe7fc0d0ac0 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.432+0000 7fe7faffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7fc0d06e0 0x7fe7fc15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.432+0000 7fe7fb7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe7fc0b4d10 0x7fe7fc0cf200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46106/0 (socket says 172.21.15.77:46106) 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.432+0000 7fe7fb7fe640 1 -- 172.21.15.77:0/4067501844 learned_addr learned my addr 172.21.15.77:0/4067501844 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.433+0000 7fe7f3fff640 1 --2- 172.21.15.77:0/4067501844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.433+0000 7fe7faffd640 1 -- 172.21.15.77:0/4067501844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 msgr2=0x7fe7fc0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.433+0000 7fe7faffd640 1 --2- 172.21.15.77:0/4067501844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.433+0000 7fe7faffd640 1 -- 172.21.15.77:0/4067501844 >> v1:172.21.15.77:6789/0 conn(0x7fe7fc0b4d10 legacy=0x7fe7fc0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.433+0000 7fe7faffd640 1 -- 172.21.15.77:0/4067501844 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0d1d90 con 0x7fe7fc0d06e0 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.433+0000 7fe7fa7fc640 1 -- 172.21.15.77:0/4067501844 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388437598 0 0) 0x7fe7fc0d47f0 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.433+0000 7fe7f3fff640 1 --2- 172.21.15.77:0/4067501844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.433+0000 7fe7faffd640 1 --2- 172.21.15.77:0/4067501844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0d06e0 0x7fe7fc15fc30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec0097a0 tx=0x7fe7ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f08dcf7e1a41f5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe7fa7fc640 1 -- 172.21.15.77:0/4067501844 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec035020 con 0x7fe7fc0d06e0 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe7fa7fc640 1 -- 172.21.15.77:0/4067501844 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec031830 con 0x7fe7fc0d06e0 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe7fa7fc640 1 -- 172.21.15.77:0/4067501844 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec031c00 con 0x7fe7fc0d06e0 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe8012c9640 1 -- 172.21.15.77:0/4067501844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0d06e0 msgr2=0x7fe7fc15fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe8012c9640 1 --2- 172.21.15.77:0/4067501844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0d06e0 0x7fe7fc15fc30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec0097a0 tx=0x7fe7ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe8012c9640 1 -- 172.21.15.77:0/4067501844 shutdown_connections 2024-03-06T07:25:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe8012c9640 1 --2- 172.21.15.77:0/4067501844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe8012c9640 1 --2- 172.21.15.77:0/4067501844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0d06e0 0x7fe7fc15fc30 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe8012c9640 1 -- 172.21.15.77:0/4067501844 >> 172.21.15.77:0/4067501844 conn(0x7fe7fc0d3cd0 msgr2=0x7fe7fc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe8012c9640 1 -- 172.21.15.77:0/4067501844 shutdown_connections 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.434+0000 7fe8012c9640 1 -- 172.21.15.77:0/4067501844 wait complete. 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.435+0000 7fe8012c9640 1 Processor -- start 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.435+0000 7fe8012c9640 1 -- start start 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe8012c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0b4d10 0x7fe7fc1520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe8012c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7fc0d06e0 0x7fe7fc0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe8012c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe8012c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0b44d0 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe8012c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0ce990 con 0x7fe7fc0d06e0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe8012c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0d0f00 con 0x7fe7fc0d0ac0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7faffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0b4d10 0x7fe7fc1520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7faffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0b4d10 0x7fe7fc1520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48168/0 (socket says 172.21.15.77:48168) 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7faffd640 1 -- 172.21.15.77:0/2584662461 learned_addr learned my addr 172.21.15.77:0/2584662461 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7fb7fe640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7f3fff640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7fc0d06e0 0x7fe7fc0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7faffd640 1 -- 172.21.15.77:0/2584662461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 msgr2=0x7fe7fc0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7faffd640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7faffd640 1 -- 172.21.15.77:0/2584662461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7fc0d06e0 msgr2=0x7fe7fc0b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.436+0000 7fe7faffd640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7fc0d06e0 0x7fe7fc0b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.437+0000 7fe7faffd640 1 -- 172.21.15.77:0/2584662461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc1f7b50 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.437+0000 7fe7faffd640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0b4d10 0x7fe7fc1520c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec0097a0 tx=0x7fe7ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.437+0000 7fe7f37fe640 1 -- 172.21.15.77:0/2584662461 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec041070 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.437+0000 7fe7f37fe640 1 -- 172.21.15.77:0/2584662461 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec004250 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.437+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc1f7e30 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.437+0000 7fe7f37fe640 1 -- 172.21.15.77:0/2584662461 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec035040 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.437+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc1f83f0 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.438+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc0a1c80 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.438+0000 7fe7f37fe640 1 -- 172.21.15.77:0/2584662461 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7ec038020 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.439+0000 7fe7f37fe640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.439+0000 7fe7f3fff640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.439+0000 7fe7f37fe640 1 -- 172.21.15.77:0/2584662461 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7ec0abd30 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.439+0000 7fe7f3fff640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0c7560 tx=0x7fe7e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.443+0000 7fe7f37fe640 1 -- 172.21.15.77:0/2584662461 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7ec07abb0 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.620+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe7fc068ff0 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.620+0000 7fe7f37fe640 1 -- 172.21.15.77:0/2584662461 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe7ec07a4b0 con 0x7fe7fc0b4d10 2024-03-06T07:25:31.622 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7dc06b4d0 msgr2=0x7fe7dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0c7560 tx=0x7fe7e4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0b4d10 msgr2=0x7fe7fc1520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0b4d10 0x7fe7fc1520c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec0097a0 tx=0x7fe7ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 shutdown_connections 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7fc0d0ac0 0x7fe7fc0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7fc0d06e0 0x7fe7fc0b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe7dc06b4d0 0x7fe7dc06d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 --2- 172.21.15.77:0/2584662461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7fc0b4d10 0x7fe7fc1520c0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 >> 172.21.15.77:0/2584662461 conn(0x7fe7fc0d3cd0 msgr2=0x7fe7fc1606e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 shutdown_connections 2024-03-06T07:25:31.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.622+0000 7fe8012c9640 1 -- 172.21.15.77:0/2584662461 wait complete. 2024-03-06T07:25:31.635 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:31.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.740+0000 7fb5d00b8640 1 Processor -- start 2024-03-06T07:25:31.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.741+0000 7fb5d00b8640 1 -- start start 2024-03-06T07:25:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.741+0000 7fb5d00b8640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c80d4860 0x7fb5c80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.741+0000 7fb5d00b8640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c81618b0 con 0x7fb5c81614d0 2024-03-06T07:25:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.741+0000 7fb5d00b8640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c80cf400 con 0x7fb5c80b0e30 2024-03-06T07:25:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.741+0000 7fb5d00b8640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80d4cc0 con 0x7fb5c80d4860 2024-03-06T07:25:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.741+0000 7fb5cde54640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb5c81614d0 0x7fb5c80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46116/0 (socket says 172.21.15.77:46116) 2024-03-06T07:25:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.741+0000 7fb5cde54640 1 -- 172.21.15.77:0/976539484 learned_addr learned my addr 172.21.15.77:0/976539484 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636588157 0 0) 0x7fb5c81618b0 con 0x7fb5c81614d0 2024-03-06T07:25:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cd653640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c80d4860 0x7fb5c80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b4003520 con 0x7fb5c81614d0 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 255255181 0 0) 0x7fb5b4003520 con 0x7fb5c81614d0 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c80d4860 msgr2=0x7fb5c80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c80d4860 0x7fb5c80d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 >> v1:172.21.15.158:6789/0 conn(0x7fb5c80b0e30 legacy=0x7fb5c80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c80cf6a0 con 0x7fb5c81614d0 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fb5c4001670 con 0x7fb5c81614d0 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 >> v1:172.21.15.77:6789/0 conn(0x7fb5c81614d0 legacy=0x7fb5c80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c80d4860 0x7fb5b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5b4004d40 0x7fb5b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80d4cc0 con 0x7fb5b40039e0 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b400baa0 con 0x7fb5b4004d40 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b400bca0 con 0x7fb5c80d4860 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cde54640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cd653640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c80d4860 0x7fb5b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5ce655640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5b4004d40 0x7fb5b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cde54640 1 -- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c80d4860 msgr2=0x7fb5b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cde54640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c80d4860 0x7fb5b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.742+0000 7fb5cde54640 1 -- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5b4004d40 msgr2=0x7fb5b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.743+0000 7fb5cde54640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5b4004d40 0x7fb5b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.743+0000 7fb5cde54640 1 -- 172.21.15.77:0/976539484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b4003520 con 0x7fb5b40039e0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.743+0000 7fb5ce655640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5b4004d40 0x7fb5b40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.743+0000 7fb5cde54640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5b4003e00 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4001eb0 tx=0x7fb5b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de767618ecb062e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.743+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4007d30 con 0x7fb5b40039e0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.743+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c4030070 con 0x7fb5b40039e0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.743+0000 7fb5cce52640 1 -- 172.21.15.77:0/976539484 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4005d30 con 0x7fb5b40039e0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.744+0000 7fb5d00b8640 1 -- 172.21.15.77:0/976539484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 msgr2=0x7fb5b4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.744+0000 7fb5d00b8640 1 --2- 172.21.15.77:0/976539484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5b4003e00 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4001eb0 tx=0x7fb5b4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.744+0000 7fb5ce655640 1 -- 172.21.15.77:0/976539484 reap_dead start 2024-03-06T07:25:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.744+0000 7fb5d00b8640 1 -- 172.21.15.77:0/976539484 shutdown_connections 2024-03-06T07:25:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.744+0000 7fb5d00b8640 1 -- 172.21.15.77:0/976539484 >> 172.21.15.77:0/976539484 conn(0x7fb5c80d3d40 msgr2=0x7fb5c80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.744+0000 7fb5d00b8640 1 -- 172.21.15.77:0/976539484 shutdown_connections 2024-03-06T07:25:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.744+0000 7fb5d00b8640 1 -- 172.21.15.77:0/976539484 wait complete. 2024-03-06T07:25:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.745+0000 7fb5d00b8640 1 Processor -- start 2024-03-06T07:25:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.745+0000 7fb5d00b8640 1 -- start start 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5d00b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5c80c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5d00b8640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5c80cc4a0 0x7fb5c80cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5d00b8640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c8151b90 0x7fb5c8152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5d00b8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80b0910 con 0x7fb5b40039e0 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5d00b8640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80cfc60 con 0x7fb5c80cc4a0 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5d00b8640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80d2020 con 0x7fb5c8151b90 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5c80c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5c80c8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48186/0 (socket says 172.21.15.77:48186) 2024-03-06T07:25:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 -- 172.21.15.77:0/1430148704 learned_addr learned my addr 172.21.15.77:0/1430148704 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cd653640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5c80cc4a0 0x7fb5c80cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5ce655640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c8151b90 0x7fb5c8152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 -- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c8151b90 msgr2=0x7fb5c8152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c8151b90 0x7fb5c8152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 -- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5c80cc4a0 msgr2=0x7fb5c80cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5c80cc4a0 0x7fb5c80cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 -- 172.21.15.77:0/1430148704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c80cce40 con 0x7fb5b40039e0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5ce655640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c8151b90 0x7fb5c8152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.746+0000 7fb5cde54640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5c80c8490 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fb5c400c300 tx=0x7fb5c4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.747+0000 7fb5be7fc640 1 -- 172.21.15.77:0/1430148704 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4045070 con 0x7fb5b40039e0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.747+0000 7fb5be7fc640 1 -- 172.21.15.77:0/1430148704 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c4000ed0 con 0x7fb5b40039e0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.747+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c8152550 con 0x7fb5b40039e0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.747+0000 7fb5be7fc640 1 -- 172.21.15.77:0/1430148704 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4004df0 con 0x7fb5b40039e0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.747+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c80b6db0 con 0x7fb5b40039e0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.748+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c80ce070 con 0x7fb5b40039e0 2024-03-06T07:25:31.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.748+0000 7fb5be7fc640 1 -- 172.21.15.77:0/1430148704 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5c4007070 con 0x7fb5b40039e0 2024-03-06T07:25:31.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.748+0000 7fb5be7fc640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb5ac06b4d0 0x7fb5ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.749+0000 7fb5be7fc640 1 -- 172.21.15.77:0/1430148704 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5c40ab130 con 0x7fb5b40039e0 2024-03-06T07:25:31.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.749+0000 7fb5cd653640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb5ac06b4d0 0x7fb5ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.749+0000 7fb5cd653640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb5ac06b4d0 0x7fb5ac06d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80cd200 tx=0x7fb5b8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.753+0000 7fb5be7fc640 1 -- 172.21.15.77:0/1430148704 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5c4079b00 con 0x7fb5b40039e0 2024-03-06T07:25:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.932+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb5c8068ff0 con 0x7fb5b40039e0 2024-03-06T07:25:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.932+0000 7fb5be7fc640 1 -- 172.21.15.77:0/1430148704 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb5c407dba0 con 0x7fb5b40039e0 2024-03-06T07:25:31.934 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:25:31.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb5ac06b4d0 msgr2=0x7fb5ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb5ac06b4d0 0x7fb5ac06d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80cd200 tx=0x7fb5b8009190 comp rx=0 tx=0).stop 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 msgr2=0x7fb5c80c8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5c80c8490 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fb5c400c300 tx=0x7fb5c4001a40 comp rx=0 tx=0).stop 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 shutdown_connections 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5c8151b90 0x7fb5c8152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5c80cc4a0 0x7fb5c80cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb5ac06b4d0 0x7fb5ac06d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 --2- 172.21.15.77:0/1430148704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5b40039e0 0x7fb5c80c8490 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 >> 172.21.15.77:0/1430148704 conn(0x7fb5c80d3d40 msgr2=0x7fb5c80d23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 shutdown_connections 2024-03-06T07:25:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:31.934+0000 7fb5d00b8640 1 -- 172.21.15.77:0/1430148704 wait complete. 2024-03-06T07:25:31.947 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.045+0000 7f055ccd4640 1 Processor -- start 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.045+0000 7f055ccd4640 1 -- start start 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.046+0000 7f055ccd4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f055815dfe0 0x7f05580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.046+0000 7f055ccd4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05580b3430 con 0x7f05580d29c0 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.046+0000 7f055ccd4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05580b3710 con 0x7f05580cf830 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.046+0000 7f055ccd4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05580b31e0 con 0x7f055815dfe0 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.046+0000 7f055659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f055815dfe0 0x7f05580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.046+0000 7f0556d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f05580d29c0 0x7f05580b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46124/0 (socket says 172.21.15.77:46124) 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.046+0000 7f0556d9d640 1 -- 172.21.15.77:0/685374336 learned_addr learned my addr 172.21.15.77:0/685374336 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.047+0000 7f055659c640 1 -- 172.21.15.77:0/685374336 >> v1:172.21.15.158:6789/0 conn(0x7f05580cf830 legacy=0x7f05580d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.047+0000 7f055659c640 1 -- 172.21.15.77:0/685374336 >> v1:172.21.15.77:6789/0 conn(0x7f05580d29c0 legacy=0x7f05580b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.047+0000 7f055659c640 1 -- 172.21.15.77:0/685374336 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580b39f0 con 0x7f055815dfe0 2024-03-06T07:25:32.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.048+0000 7f055659c640 1 --2- 172.21.15.77:0/685374336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055815dfe0 0x7f05580cf2f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f05580b3360 tx=0x7f054002b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4742ca00c6adf556 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.048+0000 7f055559a640 1 -- 172.21.15.77:0/685374336 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540037070 con 0x7f055815dfe0 2024-03-06T07:25:32.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.048+0000 7f055559a640 1 -- 172.21.15.77:0/685374336 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0540030710 con 0x7f055815dfe0 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.048+0000 7f055559a640 1 -- 172.21.15.77:0/685374336 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540030ae0 con 0x7f055815dfe0 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.048+0000 7f055ccd4640 1 -- 172.21.15.77:0/685374336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055815dfe0 msgr2=0x7f05580cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.048+0000 7f055ccd4640 1 --2- 172.21.15.77:0/685374336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055815dfe0 0x7f05580cf2f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f05580b3360 tx=0x7f054002b6f0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.049+0000 7f055ccd4640 1 -- 172.21.15.77:0/685374336 shutdown_connections 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.049+0000 7f055ccd4640 1 --2- 172.21.15.77:0/685374336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055815dfe0 0x7f05580cf2f0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.049+0000 7f055ccd4640 1 -- 172.21.15.77:0/685374336 >> 172.21.15.77:0/685374336 conn(0x7f05580d3cd0 msgr2=0x7f05580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.049+0000 7f055ccd4640 1 -- 172.21.15.77:0/685374336 shutdown_connections 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.049+0000 7f055ccd4640 1 -- 172.21.15.77:0/685374336 wait complete. 2024-03-06T07:25:32.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.051+0000 7f055ccd4640 1 Processor -- start 2024-03-06T07:25:32.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.051+0000 7f055ccd4640 1 -- start start 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f055ccd4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf830 0x7f05581f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f055ccd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d29c0 0x7f05580b92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f055ccd4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055815dfe0 0x7f05580b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f055ccd4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055815e3c0 con 0x7f05580d29c0 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f055ccd4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05580b3d60 con 0x7f05580cf830 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f055ccd4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055815e540 con 0x7f055815dfe0 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d29c0 0x7f05580b92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d29c0 0x7f05580b92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48190/0 (socket says 172.21.15.77:48190) 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 -- 172.21.15.77:0/856872713 learned_addr learned my addr 172.21.15.77:0/856872713 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 -- 172.21.15.77:0/856872713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055815dfe0 msgr2=0x7f05580b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055815dfe0 0x7f05580b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 -- 172.21.15.77:0/856872713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf830 msgr2=0x7f05581f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf830 0x7f05581f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 -- 172.21.15.77:0/856872713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05581f3c20 con 0x7f05580d29c0 2024-03-06T07:25:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f0555d9b640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d29c0 0x7f05580b92d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f054800c400 tx=0x7f054800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.052+0000 7f054effd640 1 -- 172.21.15.77:0/856872713 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548011790 con 0x7f05580d29c0 2024-03-06T07:25:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.053+0000 7f054effd640 1 -- 172.21.15.77:0/856872713 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0548017070 con 0x7f05580d29c0 2024-03-06T07:25:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.053+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05581f3f00 con 0x7f05580d29c0 2024-03-06T07:25:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.053+0000 7f054effd640 1 -- 172.21.15.77:0/856872713 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548011bb0 con 0x7f05580d29c0 2024-03-06T07:25:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.053+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05581f44c0 con 0x7f05580d29c0 2024-03-06T07:25:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.054+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05580b6be0 con 0x7f05580d29c0 2024-03-06T07:25:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.054+0000 7f054effd640 1 -- 172.21.15.77:0/856872713 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f054801c460 con 0x7f05580d29c0 2024-03-06T07:25:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.054+0000 7f054effd640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.055+0000 7f055659c640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.055+0000 7f054effd640 1 -- 172.21.15.77:0/856872713 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f054808a020 con 0x7f05580d29c0 2024-03-06T07:25:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.055+0000 7f055659c640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0540034000 tx=0x7f054002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.059+0000 7f054effd640 1 -- 172.21.15.77:0/856872713 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0548058ea0 con 0x7f05580d29c0 2024-03-06T07:25:32.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.295+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f055804f330 con 0x7f05580d29c0 2024-03-06T07:25:32.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.297+0000 7f054effd640 1 -- 172.21.15.77:0/856872713 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f05480587a0 con 0x7f05580d29c0 2024-03-06T07:25:32.299 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:32.299 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:32.300 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:32.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 msgr2=0x7f053806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0540034000 tx=0x7f054002f000 comp rx=0 tx=0).stop 2024-03-06T07:25:32.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d29c0 msgr2=0x7f05580b92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d29c0 0x7f05580b92d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f054800c400 tx=0x7f054800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 shutdown_connections 2024-03-06T07:25:32.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05580d29c0 0x7f05580b92d0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055815dfe0 0x7f05580b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 --2- 172.21.15.77:0/856872713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05580cf830 0x7f05581f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 >> 172.21.15.77:0/856872713 conn(0x7f05580d3cd0 msgr2=0x7f05580ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 shutdown_connections 2024-03-06T07:25:32.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.300+0000 7f055ccd4640 1 -- 172.21.15.77:0/856872713 wait complete. 2024-03-06T07:25:32.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.412+0000 7fb1da266640 1 Processor -- start 2024-03-06T07:25:32.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.412+0000 7fb1da266640 1 -- start start 2024-03-06T07:25:32.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1da266640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1d415bf30 0x7fb1d415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1da266640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d415f580 con 0x7fb1d40b3060 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1da266640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d415b450 con 0x7fb1d413dc10 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1da266640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d40cf3f0 con 0x7fb1d415bf30 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb1d40b3060 0x7fb1d415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46128/0 (socket says 172.21.15.77:46128) 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d37fe640 1 -- 172.21.15.77:0/3365514336 learned_addr learned my addr 172.21.15.77:0/3365514336 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d8803640 1 --2- 172.21.15.77:0/3365514336 >> v2:172.21.15.158:3300/0 conn(0x7fb1d415bf30 0x7fb1d415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071649176 0 0) 0x7fb1d415f580 con 0x7fb1d40b3060 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1bc003520 con 0x7fb1d40b3060 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 964185937 0 0) 0x7fb1bc003520 con 0x7fb1d40b3060 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 >> v2:172.21.15.158:3300/0 conn(0x7fb1d415bf30 msgr2=0x7fb1d415ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d2ffd640 1 --2- 172.21.15.77:0/3365514336 >> v2:172.21.15.158:3300/0 conn(0x7fb1d415bf30 0x7fb1d415ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 >> v1:172.21.15.158:6789/0 conn(0x7fb1d413dc10 legacy=0x7fb1d40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d415b730 con 0x7fb1d40b3060 2024-03-06T07:25:32.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.413+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fb1c4001670 con 0x7fb1d40b3060 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 >> v1:172.21.15.77:6789/0 conn(0x7fb1d40b3060 legacy=0x7fb1d415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d2ffd640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0039b0 0x7fb1bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d2ffd640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc004800 0x7fb1bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d2ffd640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0051c0 0x7fb1bc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d40cf3f0 con 0x7fb1bc0051c0 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc00e370 con 0x7fb1bc0039b0 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc00e570 con 0x7fb1bc004800 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d8803640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0051c0 0x7fb1bc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d37fe640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc004800 0x7fb1bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d3fff640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0039b0 0x7fb1bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d8803640 1 -- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc004800 msgr2=0x7fb1bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d8803640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc004800 0x7fb1bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d8803640 1 -- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0039b0 msgr2=0x7fb1bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d8803640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0039b0 0x7fb1bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d8803640 1 -- 172.21.15.77:0/3365514336 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc003520 con 0x7fb1bc0051c0 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d37fe640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc004800 0x7fb1bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.414+0000 7fb1d3fff640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0039b0 0x7fb1bc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.415+0000 7fb1d8803640 1 -- 172.21.15.77:0/3365514336 reap_dead start 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.415+0000 7fb1d8803640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0051c0 0x7fb1bc00dbd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009fc0 tx=0x7fb1bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bd905209a87550 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.415+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc009c10 con 0x7fb1bc0051c0 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.415+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc02cc60 con 0x7fb1bc0051c0 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.415+0000 7fb1d2ffd640 1 -- 172.21.15.77:0/3365514336 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc003bd0 con 0x7fb1bc0051c0 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.416+0000 7fb1da266640 1 -- 172.21.15.77:0/3365514336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0051c0 msgr2=0x7fb1bc00dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.416+0000 7fb1da266640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0051c0 0x7fb1bc00dbd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009fc0 tx=0x7fb1bc005830 comp rx=0 tx=0).stop 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.416+0000 7fb1da266640 1 -- 172.21.15.77:0/3365514336 shutdown_connections 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.416+0000 7fb1da266640 1 --2- 172.21.15.77:0/3365514336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0051c0 0x7fb1bc00dbd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.416+0000 7fb1da266640 1 -- 172.21.15.77:0/3365514336 >> 172.21.15.77:0/3365514336 conn(0x7fb1d40cf8a0 msgr2=0x7fb1d40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.416+0000 7fb1da266640 1 -- 172.21.15.77:0/3365514336 shutdown_connections 2024-03-06T07:25:32.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.416+0000 7fb1da266640 1 -- 172.21.15.77:0/3365514336 wait complete. 2024-03-06T07:25:32.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.417+0000 7fb1da266640 1 Processor -- start 2024-03-06T07:25:32.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.417+0000 7fb1da266640 1 -- start start 2024-03-06T07:25:32.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.418+0000 7fb1da266640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0051c0 0x7fb1d41552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.418+0000 7fb1da266640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d413e850 0x7fb1d4139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.418+0000 7fb1da266640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d413ecb0 0x7fb1d413c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.418+0000 7fb1da266640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d40b2950 con 0x7fb1d413ecb0 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.418+0000 7fb1da266640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d415b210 con 0x7fb1d413e850 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.418+0000 7fb1da266640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d40b2ad0 con 0x7fb1bc0051c0 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d413ecb0 0x7fb1d413c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d413ecb0 0x7fb1d413c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48208/0 (socket says 172.21.15.77:48208) 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 -- 172.21.15.77:0/1870895230 learned_addr learned my addr 172.21.15.77:0/1870895230 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d37fe640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d413e850 0x7fb1d4139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d3fff640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0051c0 0x7fb1d41552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 -- 172.21.15.77:0/1870895230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0051c0 msgr2=0x7fb1d41552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0051c0 0x7fb1d41552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 -- 172.21.15.77:0/1870895230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d413e850 msgr2=0x7fb1d4139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d413e850 0x7fb1d4139ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 -- 172.21.15.77:0/1870895230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4140750 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d8803640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d413ecb0 0x7fb1d413c340 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00e200 tx=0x7fb1cc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d0ff9640 1 -- 172.21.15.77:0/1870895230 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc003990 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d0ff9640 1 -- 172.21.15.77:0/1870895230 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc044070 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d4140a30 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1d0ff9640 1 -- 172.21.15.77:0/1870895230 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc003d50 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.419+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d413e330 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.421+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d404f330 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.421+0000 7fb1d0ff9640 1 -- 172.21.15.77:0/1870895230 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1cc005d10 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.421+0000 7fb1d0ff9640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.421+0000 7fb1d0ff9640 1 -- 172.21.15.77:0/1870895230 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1cc0ac7e0 con 0x7fb1d413ecb0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.422+0000 7fb1d3fff640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.422+0000 7fb1d3fff640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c006b4d0 0x7fb1c006d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8000c00 tx=0x7fb1c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.426+0000 7fb1d0ff9640 1 -- 172.21.15.77:0/1870895230 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1cc07b010 con 0x7fb1d413ecb0 2024-03-06T07:25:32.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.661+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1d406eb00 con 0x7fb1d413ecb0 2024-03-06T07:25:32.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.662+0000 7fb1d0ff9640 1 -- 172.21.15.77:0/1870895230 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16656 (secure 0 0 0) 0x7fb1cc07f0b0 con 0x7fb1d413ecb0 2024-03-06T07:25:32.664 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":69,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:00.275386+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:56.868266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:57.595004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:32.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c006b4d0 msgr2=0x7fb1c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c006b4d0 0x7fb1c006d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8000c00 tx=0x7fb1c8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:32.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d413ecb0 msgr2=0x7fb1d413c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d413ecb0 0x7fb1d413c340 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00e200 tx=0x7fb1cc009750 comp rx=0 tx=0).stop 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 shutdown_connections 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d413ecb0 0x7fb1d413c340 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1d413e850 0x7fb1d4139ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 --2- 172.21.15.77:0/1870895230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0051c0 0x7fb1d41552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 >> 172.21.15.77:0/1870895230 conn(0x7fb1d40cf8a0 msgr2=0x7fb1d40cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.664+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 shutdown_connections 2024-03-06T07:25:32.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.665+0000 7fb1da266640 1 -- 172.21.15.77:0/1870895230 wait complete. 2024-03-06T07:25:32.681 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:32.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:32.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.781+0000 7f36fd837640 1 Processor -- start 2024-03-06T07:25:32.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.781+0000 7f36fd837640 1 -- start start 2024-03-06T07:25:32.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.782+0000 7f36fd837640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36f80d22b0 0x7f36f80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.782+0000 7f36fd837640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36f80b41f0 0x7f36f80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.782+0000 7f36fd837640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80b3430 con 0x7f36f815dfe0 2024-03-06T07:25:32.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.782+0000 7f36fd837640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f80d2c50 con 0x7f36f80d22b0 2024-03-06T07:25:32.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.782+0000 7f36fd837640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80b31e0 con 0x7f36f80b41f0 2024-03-06T07:25:32.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.782+0000 7f36f6ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f36f815dfe0 0x7f36f80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46138/0 (socket says 172.21.15.77:46138) 2024-03-06T07:25:32.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.782+0000 7f36f6ffd640 1 -- 172.21.15.77:0/2399649140 learned_addr learned my addr 172.21.15.77:0/2399649140 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:32.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711485378 0 0) 0x7f36f80b3430 con 0x7f36f815dfe0 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36effff640 1 --2- 172.21.15.77:0/2399649140 >> v2:172.21.15.158:3300/0 conn(0x7f36f80d22b0 0x7f36f80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f77fe640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36f80b41f0 0x7f36f80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4003520 con 0x7f36f815dfe0 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179629478 0 0) 0x7f36e4003520 con 0x7f36f815dfe0 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36f80b41f0 msgr2=0x7f36f80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36f80b41f0 0x7f36f80b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 >> v2:172.21.15.158:3300/0 conn(0x7f36f80d22b0 msgr2=0x7f36f80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 --2- 172.21.15.77:0/2399649140 >> v2:172.21.15.158:3300/0 conn(0x7f36f80d22b0 0x7f36f80d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80b3710 con 0x7f36f815dfe0 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f36e8001670 con 0x7f36f815dfe0 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 >> v1:172.21.15.77:6789/0 conn(0x7f36f815dfe0 legacy=0x7f36f80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36e40039b0 0x7f36e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36f80b41f0 0x7f36e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e4004d40 0x7f36e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80b31e0 con 0x7f36e4004d40 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e400b9c0 con 0x7f36e40039b0 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e400bb40 con 0x7f36f80b41f0 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f77fe640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e4004d40 0x7f36e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f6ffd640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36e40039b0 0x7f36e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36effff640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36f80b41f0 0x7f36e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f77fe640 1 -- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36f80b41f0 msgr2=0x7f36e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f77fe640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36f80b41f0 0x7f36e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f77fe640 1 -- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36e40039b0 msgr2=0x7f36e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f77fe640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36e40039b0 0x7f36e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.783+0000 7f36f77fe640 1 -- 172.21.15.77:0/2399649140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e4003520 con 0x7f36e4004d40 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.784+0000 7f36f77fe640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e4004d40 0x7f36e40051c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e170 tx=0x7f36e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0187c6f6ec6177 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.784+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0039070 con 0x7f36e4004d40 2024-03-06T07:25:32.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.784+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f002cbf0 con 0x7f36e4004d40 2024-03-06T07:25:32.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.784+0000 7f36f67fc640 1 -- 172.21.15.77:0/2399649140 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0009c10 con 0x7f36e4004d40 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.785+0000 7f36fd837640 1 -- 172.21.15.77:0/2399649140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e4004d40 msgr2=0x7f36e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.785+0000 7f36fd837640 1 --2- 172.21.15.77:0/2399649140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36e4004d40 0x7f36e40051c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e170 tx=0x7f36e4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.785+0000 7f36f77fe640 1 -- 172.21.15.77:0/2399649140 reap_dead start 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.785+0000 7f36fd837640 1 -- 172.21.15.77:0/2399649140 shutdown_connections 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.785+0000 7f36fd837640 1 -- 172.21.15.77:0/2399649140 >> 172.21.15.77:0/2399649140 conn(0x7f36f80d3cd0 msgr2=0x7f36f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.785+0000 7f36fd837640 1 -- 172.21.15.77:0/2399649140 shutdown_connections 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.785+0000 7f36fd837640 1 -- 172.21.15.77:0/2399649140 wait complete. 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.786+0000 7f36fd837640 1 Processor -- start 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.786+0000 7f36fd837640 1 -- start start 2024-03-06T07:25:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36fd837640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e4004d40 0x7f36f80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36fd837640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36f809f9e0 0x7f36f8150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36fd837640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36f809fe60 0x7f36f8151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36fd837640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80cfa00 con 0x7f36f809f9e0 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36fd837640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80b3d60 con 0x7f36f809fe60 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36fd837640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80d1c70 con 0x7f36e4004d40 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36f809f9e0 0x7f36f8150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36f809f9e0 0x7f36f8150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48224/0 (socket says 172.21.15.77:48224) 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36effff640 1 -- 172.21.15.77:0/2830208282 learned_addr learned my addr 172.21.15.77:0/2830208282 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36f6ffd640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e4004d40 0x7f36f80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36effff640 1 -- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e4004d40 msgr2=0x7f36f80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36f77fe640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36f809fe60 0x7f36f8151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36effff640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e4004d40 0x7f36f80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36effff640 1 -- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36f809fe60 msgr2=0x7f36f8151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36effff640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36f809fe60 0x7f36f8151150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36effff640 1 -- 172.21.15.77:0/2830208282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80cb2e0 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.787+0000 7f36f6ffd640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e4004d40 0x7f36f80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.788+0000 7f36effff640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36f809f9e0 0x7f36f8150c10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f36e0011a70 tx=0x7f36e0011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.788+0000 7f36f4ff9640 1 -- 172.21.15.77:0/2830208282 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e000fb50 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.788+0000 7f36f4ff9640 1 -- 172.21.15.77:0/2830208282 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e000fcf0 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.788+0000 7f36f4ff9640 1 -- 172.21.15.77:0/2830208282 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e000fea0 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.788+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f80cb560 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.788+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f80cbb20 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.789+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f804f330 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.789+0000 7f36f4ff9640 1 -- 172.21.15.77:0/2830208282 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36e0012040 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.790+0000 7f36f4ff9640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36d806b4d0 0x7f36d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.790+0000 7f36f4ff9640 1 -- 172.21.15.77:0/2830208282 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36e008a9c0 con 0x7f36f809f9e0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.790+0000 7f36f6ffd640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36d806b4d0 0x7f36d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.790+0000 7f36f6ffd640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36d806b4d0 0x7f36d806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f36e8001050 tx=0x7f36e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:32.794+0000 7f36f4ff9640 1 -- 172.21.15.77:0/2830208282 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36f80cb2e0 con 0x7f36f809f9e0 2024-03-06T07:25:33.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.008+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36f806ca80 con 0x7f36f809f9e0 2024-03-06T07:25:33.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.009+0000 7f36f4ff9640 1 -- 172.21.15.77:0/2830208282 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f36e0019030 con 0x7f36f809f9e0 2024-03-06T07:25:33.011 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:33.011 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:33.011 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36d806b4d0 msgr2=0x7f36d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36d806b4d0 0x7f36d806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f36e8001050 tx=0x7f36e8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36f809f9e0 msgr2=0x7f36f8150c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36f809f9e0 0x7f36f8150c10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f36e0011a70 tx=0x7f36e0011f30 comp rx=0 tx=0).stop 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 shutdown_connections 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36d806b4d0 0x7f36d806d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36f809f9e0 0x7f36f8150c10 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36f809fe60 0x7f36f8151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 --2- 172.21.15.77:0/2830208282 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36e4004d40 0x7f36f80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 >> 172.21.15.77:0/2830208282 conn(0x7f36f80d3cd0 msgr2=0x7f36f80ceac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 shutdown_connections 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.011+0000 7f36fd837640 1 -- 172.21.15.77:0/2830208282 wait complete. 2024-03-06T07:25:33.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:33.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.123+0000 7f69ab5f6640 1 Processor -- start 2024-03-06T07:25:33.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.123+0000 7f69ab5f6640 1 -- start start 2024-03-06T07:25:33.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.124+0000 7f69ab5f6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69a415dfe0 0x7f69a40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.124+0000 7f69ab5f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.124+0000 7f69ab5f6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a40b3430 con 0x7f69a40d27f0 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.124+0000 7f69ab5f6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a40d2c50 con 0x7f69a415dfe0 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.124+0000 7f69ab5f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a40b31e0 con 0x7f69a40cf830 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a8b91640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a9392640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69a415dfe0 0x7f69a40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a8b91640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:34440/0 (socket says 172.21.15.77:34440) 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a9b93640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f69a40d27f0 0x7f69a40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:48144/0 (socket says 172.21.15.77:48144) 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a8b91640 1 -- 172.21.15.77:0/3700573680 learned_addr learned my addr 172.21.15.77:0/3700573680 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a8b91640 1 -- 172.21.15.77:0/3700573680 >> v2:172.21.15.158:3300/0 conn(0x7f69a415dfe0 msgr2=0x7f69a40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a8b91640 1 --2- 172.21.15.77:0/3700573680 >> v2:172.21.15.158:3300/0 conn(0x7f69a415dfe0 0x7f69a40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f699bfff640 1 -- 172.21.15.77:0/3700573680 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137914150 0 0) 0x7f69a40b3430 con 0x7f69a40d27f0 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a8b91640 1 -- 172.21.15.77:0/3700573680 >> v1:172.21.15.158:6789/0 conn(0x7f69a40d27f0 legacy=0x7f69a40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.125+0000 7f69a8b91640 1 -- 172.21.15.77:0/3700573680 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a40b3710 con 0x7f69a40cf830 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.126+0000 7f69a8b91640 1 --2- 172.21.15.77:0/3700573680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a40d22b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6994009870 tx=0x7f699402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f249e9be5bfdd73b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.126+0000 7f699bfff640 1 -- 172.21.15.77:0/3700573680 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994034070 con 0x7f69a40cf830 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.126+0000 7f699bfff640 1 -- 172.21.15.77:0/3700573680 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6994030840 con 0x7f69a40cf830 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.126+0000 7f699bfff640 1 -- 172.21.15.77:0/3700573680 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994030c10 con 0x7f69a40cf830 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.127+0000 7f69ab5f6640 1 -- 172.21.15.77:0/3700573680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 msgr2=0x7f69a40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.127+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/3700573680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a40d22b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6994009870 tx=0x7f699402b980 comp rx=0 tx=0).stop 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.127+0000 7f69ab5f6640 1 -- 172.21.15.77:0/3700573680 shutdown_connections 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.127+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/3700573680 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a40d22b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.127+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/3700573680 >> v2:172.21.15.158:3300/0 conn(0x7f69a415dfe0 0x7f69a40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.127+0000 7f69ab5f6640 1 -- 172.21.15.77:0/3700573680 >> 172.21.15.77:0/3700573680 conn(0x7f69a40d3cd0 msgr2=0x7f69a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.127+0000 7f69ab5f6640 1 -- 172.21.15.77:0/3700573680 shutdown_connections 2024-03-06T07:25:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.127+0000 7f69ab5f6640 1 -- 172.21.15.77:0/3700573680 wait complete. 2024-03-06T07:25:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.129+0000 7f69ab5f6640 1 Processor -- start 2024-03-06T07:25:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.129+0000 7f69ab5f6640 1 -- start start 2024-03-06T07:25:33.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69ab5f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69ab5f6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a40d27f0 0x7f69a406ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69ab5f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a415dfe0 0x7f69a406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69ab5f6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a40d1c70 con 0x7f69a415dfe0 2024-03-06T07:25:33.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69ab5f6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a40b3d60 con 0x7f69a40d27f0 2024-03-06T07:25:33.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69ab5f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a40d1df0 con 0x7f69a40cf830 2024-03-06T07:25:33.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9b93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a415dfe0 0x7f69a406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9b93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a415dfe0 0x7f69a406bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48232/0 (socket says 172.21.15.77:48232) 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9b93640 1 -- 172.21.15.77:0/2682696403 learned_addr learned my addr 172.21.15.77:0/2682696403 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9392640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a8b91640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a40d27f0 0x7f69a406ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9b93640 1 -- 172.21.15.77:0/2682696403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 msgr2=0x7f69a414e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9b93640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a414e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9b93640 1 -- 172.21.15.77:0/2682696403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a40d27f0 msgr2=0x7f69a406ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9b93640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a40d27f0 0x7f69a406ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.130+0000 7f69a9b93640 1 -- 172.21.15.77:0/2682696403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a40b9840 con 0x7f69a415dfe0 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.131+0000 7f69a9b93640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a415dfe0 0x7f69a406bf40 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f69a000c7a0 tx=0x7f69a000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.131+0000 7f6999ffb640 1 -- 172.21.15.77:0/2682696403 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0018070 con 0x7f69a415dfe0 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.131+0000 7f6999ffb640 1 -- 172.21.15.77:0/2682696403 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a000a640 con 0x7f69a415dfe0 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.131+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a40b9b20 con 0x7f69a415dfe0 2024-03-06T07:25:33.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.131+0000 7f6999ffb640 1 -- 172.21.15.77:0/2682696403 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a000aab0 con 0x7f69a415dfe0 2024-03-06T07:25:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.131+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a415ea50 con 0x7f69a415dfe0 2024-03-06T07:25:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.132+0000 7f6999ffb640 1 -- 172.21.15.77:0/2682696403 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69a001b070 con 0x7f69a415dfe0 2024-03-06T07:25:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.132+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a404f330 con 0x7f69a415dfe0 2024-03-06T07:25:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.132+0000 7f6999ffb640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f698806b4d0 0x7f698806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.133+0000 7f69a9392640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f698806b4d0 0x7f698806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.133+0000 7f6999ffb640 1 -- 172.21.15.77:0/2682696403 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69a00897b0 con 0x7f69a415dfe0 2024-03-06T07:25:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.133+0000 7f69a9392640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f698806b4d0 0x7f698806d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f699c009a30 tx=0x7f699c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.137+0000 7f6999ffb640 1 -- 172.21.15.77:0/2682696403 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69a00586c0 con 0x7f69a415dfe0 2024-03-06T07:25:33.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.349+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69a40d0ed0 con 0x7f69a415dfe0 2024-03-06T07:25:33.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.350+0000 7f6999ffb640 1 -- 172.21.15.77:0/2682696403 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f69a0057fc0 con 0x7f69a415dfe0 2024-03-06T07:25:33.352 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:33.352 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:58.627086+0000","modified":"2024-03-06T07:25:00.680393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6507":{"gid":6507,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3491727415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":3491727415},{"type":"v1","addr":"172.21.15.77:6837","nonce":3491727415}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6507,"qdb_cluster":[6507]},"id":5}]} 2024-03-06T07:25:33.353 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.352+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f698806b4d0 msgr2=0x7f698806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f698806b4d0 0x7f698806d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f699c009a30 tx=0x7f699c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a415dfe0 msgr2=0x7f69a406bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a415dfe0 0x7f69a406bf40 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f69a000c7a0 tx=0x7f69a000cc60 comp rx=0 tx=0).stop 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 shutdown_connections 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f698806b4d0 0x7f698806d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a415dfe0 0x7f69a406bf40 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69a40d27f0 0x7f69a406ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 --2- 172.21.15.77:0/2682696403 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69a40cf830 0x7f69a414e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 >> 172.21.15.77:0/2682696403 conn(0x7f69a40d3cd0 msgr2=0x7f69a40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 shutdown_connections 2024-03-06T07:25:33.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.353+0000 7f69ab5f6640 1 -- 172.21.15.77:0/2682696403 wait complete. 2024-03-06T07:25:33.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:33.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.465+0000 7f378202a640 1 Processor -- start 2024-03-06T07:25:33.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.465+0000 7f378202a640 1 -- start start 2024-03-06T07:25:33.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f378202a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c15dfe0 0x7f377c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f378202a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f377c0cf830 0x7f377c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f378202a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f377c0d27f0 0x7f377c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f378202a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c0d2c50 con 0x7f377c0cf830 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f378202a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c0b31e0 con 0x7f377c0d27f0 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f378202a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0b3d60 con 0x7f377c15dfe0 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f3773fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f377c0cf830 0x7f377c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f3773fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f377c0cf830 0x7f377c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48248/0 (socket says 172.21.15.77:48248) 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f3773fff640 1 -- 172.21.15.77:0/2760387809 learned_addr learned my addr 172.21.15.77:0/2760387809 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f3773fff640 1 -- 172.21.15.77:0/2760387809 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c15dfe0 msgr2=0x7f377c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f3773fff640 1 --2- 172.21.15.77:0/2760387809 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c15dfe0 0x7f377c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f3773fff640 1 -- 172.21.15.77:0/2760387809 >> v2:172.21.15.158:3300/0 conn(0x7f377c0d27f0 msgr2=0x7f377c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f3773fff640 1 --2- 172.21.15.77:0/2760387809 >> v2:172.21.15.158:3300/0 conn(0x7f377c0d27f0 0x7f377c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.466+0000 7f3773fff640 1 -- 172.21.15.77:0/2760387809 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0b3430 con 0x7f377c0cf830 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.467+0000 7f3773fff640 1 --2- 172.21.15.77:0/2760387809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c0cf830 0x7f377c0d22b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f377c067ad0 tx=0x7f376402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edefcb34fc50a285 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.467+0000 7f377affd640 1 -- 172.21.15.77:0/2760387809 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764035020 con 0x7f377c0cf830 2024-03-06T07:25:33.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.467+0000 7f377affd640 1 -- 172.21.15.77:0/2760387809 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3764031870 con 0x7f377c0cf830 2024-03-06T07:25:33.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.467+0000 7f377affd640 1 -- 172.21.15.77:0/2760387809 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764031c40 con 0x7f377c0cf830 2024-03-06T07:25:33.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 -- 172.21.15.77:0/2760387809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c0cf830 msgr2=0x7f377c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 --2- 172.21.15.77:0/2760387809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c0cf830 0x7f377c0d22b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f377c067ad0 tx=0x7f376402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:33.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 -- 172.21.15.77:0/2760387809 shutdown_connections 2024-03-06T07:25:33.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 --2- 172.21.15.77:0/2760387809 >> v2:172.21.15.158:3300/0 conn(0x7f377c0d27f0 0x7f377c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 --2- 172.21.15.77:0/2760387809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c0cf830 0x7f377c0d22b0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 --2- 172.21.15.77:0/2760387809 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c15dfe0 0x7f377c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 -- 172.21.15.77:0/2760387809 >> 172.21.15.77:0/2760387809 conn(0x7f377c0d3cd0 msgr2=0x7f377c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 -- 172.21.15.77:0/2760387809 shutdown_connections 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.468+0000 7f378202a640 1 -- 172.21.15.77:0/2760387809 wait complete. 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.469+0000 7f378202a640 1 Processor -- start 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.469+0000 7f378202a640 1 -- start start 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f378202a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c0cf830 0x7f377c157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f378202a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377c0d27f0 0x7f377c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f378202a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c15dfe0 0x7f377c0b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f378202a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0d1c70 con 0x7f377c15dfe0 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f378202a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0b39f0 con 0x7f377c0d27f0 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f378202a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0d1df0 con 0x7f377c0cf830 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c15dfe0 0x7f377c0b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c15dfe0 0x7f377c0b26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48250/0 (socket says 172.21.15.77:48250) 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377bfff640 1 -- 172.21.15.77:0/3250108237 learned_addr learned my addr 172.21.15.77:0/3250108237 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377b7fe640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c0cf830 0x7f377c157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f3773fff640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377c0d27f0 0x7f377c0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377bfff640 1 -- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c0cf830 msgr2=0x7f377c157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377bfff640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c0cf830 0x7f377c157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377bfff640 1 -- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377c0d27f0 msgr2=0x7f377c0b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377bfff640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377c0d27f0 0x7f377c0b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377bfff640 1 -- 172.21.15.77:0/3250108237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0b68d0 con 0x7f377c15dfe0 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f377b7fe640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c0cf830 0x7f377c157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.470+0000 7f3773fff640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377c0d27f0 0x7f377c0b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.471+0000 7f377bfff640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c15dfe0 0x7f377c0b26c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f377400b970 tx=0x7f377400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.471+0000 7f3778ff9640 1 -- 172.21.15.77:0/3250108237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774019070 con 0x7f377c15dfe0 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.471+0000 7f3778ff9640 1 -- 172.21.15.77:0/3250108237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377400cb60 con 0x7f377c15dfe0 2024-03-06T07:25:33.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.471+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c0b6bb0 con 0x7f377c15dfe0 2024-03-06T07:25:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.471+0000 7f3778ff9640 1 -- 172.21.15.77:0/3250108237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774014440 con 0x7f377c15dfe0 2024-03-06T07:25:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.471+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c0b7170 con 0x7f377c15dfe0 2024-03-06T07:25:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.472+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c04f330 con 0x7f377c15dfe0 2024-03-06T07:25:33.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.473+0000 7f3778ff9640 1 -- 172.21.15.77:0/3250108237 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f377401c070 con 0x7f377c15dfe0 2024-03-06T07:25:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.473+0000 7f3778ff9640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f375c06b4d0 0x7f375c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.473+0000 7f3778ff9640 1 -- 172.21.15.77:0/3250108237 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3774089f40 con 0x7f377c15dfe0 2024-03-06T07:25:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.473+0000 7f377b7fe640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f375c06b4d0 0x7f375c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.473+0000 7f377b7fe640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f375c06b4d0 0x7f375c06d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f376c00a870 tx=0x7f376c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.477+0000 7f3778ff9640 1 -- 172.21.15.77:0/3250108237 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3774058dc0 con 0x7f377c15dfe0 2024-03-06T07:25:33.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.656+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f377c0ceac0 con 0x7f377c15dfe0 2024-03-06T07:25:33.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.657+0000 7f3778ff9640 1 -- 172.21.15.77:0/3250108237 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16656 (secure 0 0 0) 0x7f37740586c0 con 0x7f377c15dfe0 2024-03-06T07:25:33.659 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:33.659 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":69,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:00.275386+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:56.868266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:57.595004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:33.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f375c06b4d0 msgr2=0x7f375c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f375c06b4d0 0x7f375c06d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f376c00a870 tx=0x7f376c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:33.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c15dfe0 msgr2=0x7f377c0b26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c15dfe0 0x7f377c0b26c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f377400b970 tx=0x7f377400be30 comp rx=0 tx=0).stop 2024-03-06T07:25:33.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 shutdown_connections 2024-03-06T07:25:33.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f375c06b4d0 0x7f375c06d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377c15dfe0 0x7f377c0b26c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f377c0d27f0 0x7f377c0b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 --2- 172.21.15.77:0/3250108237 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f377c0cf830 0x7f377c157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 >> 172.21.15.77:0/3250108237 conn(0x7f377c0d3cd0 msgr2=0x7f377c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 shutdown_connections 2024-03-06T07:25:33.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.660+0000 7f378202a640 1 -- 172.21.15.77:0/3250108237 wait complete. 2024-03-06T07:25:33.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:33.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.773+0000 7ff2d304e640 1 Processor -- start 2024-03-06T07:25:33.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.773+0000 7ff2d304e640 1 -- start start 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2d304e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff2cc162400 0x7ff2cc0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2d304e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc0d22c0 con 0x7ff2cc162970 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2d304e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc0d25a0 con 0x7ff2cc0b4460 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2d304e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2cc161b00 con 0x7ff2cc162400 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cbfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff2cc162970 0x7ff2cc161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46154/0 (socket says 172.21.15.77:46154) 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cbfff640 1 -- 172.21.15.77:0/2899450630 learned_addr learned my addr 172.21.15.77:0/2899450630 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2d15eb640 1 --2- 172.21.15.77:0/2899450630 >> v2:172.21.15.158:3300/0 conn(0x7ff2cc162400 0x7ff2cc0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820612985 0 0) 0x7ff2cc0d22c0 con 0x7ff2cc162970 2024-03-06T07:25:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b8003520 con 0x7ff2cc162970 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 394250533 0 0) 0x7ff2b8003520 con 0x7ff2cc162970 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 >> v2:172.21.15.158:3300/0 conn(0x7ff2cc162400 msgr2=0x7ff2cc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 --2- 172.21.15.77:0/2899450630 >> v2:172.21.15.158:3300/0 conn(0x7ff2cc162400 0x7ff2cc0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 >> v1:172.21.15.158:6789/0 conn(0x7ff2cc0b4460 legacy=0x7ff2cc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc0d2880 con 0x7ff2cc162970 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ff2b4001670 con 0x7ff2cc162970 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 >> v1:172.21.15.77:6789/0 conn(0x7ff2cc162970 legacy=0x7ff2cc161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2b80039b0 0x7ff2b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b80043b0 0x7ff2b8004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b8005080 0x7ff2b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc161b00 con 0x7ff2b8005080 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b800bfe0 con 0x7ff2b80043b0 2024-03-06T07:25:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.774+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b800c1e0 con 0x7ff2b80039b0 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2d15eb640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b8005080 0x7ff2b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2d0dea640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2b80039b0 0x7ff2b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2cbfff640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b80043b0 0x7ff2b8004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2d15eb640 1 -- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2b80039b0 msgr2=0x7ff2b8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2d15eb640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2b80039b0 0x7ff2b8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2d15eb640 1 -- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b80043b0 msgr2=0x7ff2b8004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2d15eb640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b80043b0 0x7ff2b8004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2d15eb640 1 -- 172.21.15.77:0/2899450630 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b8003520 con 0x7ff2b8005080 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2cbfff640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b80043b0 0x7ff2b8004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.775+0000 7ff2d15eb640 1 -- 172.21.15.77:0/2899450630 reap_dead start 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2d15eb640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b8005080 0x7ff2b800b9c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009a90 tx=0x7ff2b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d149b5db3a8c6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc009690 con 0x7ff2b8005080 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2bc004020 con 0x7ff2b8005080 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2cb7fe640 1 -- 172.21.15.77:0/2899450630 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc004410 con 0x7ff2b8005080 2024-03-06T07:25:33.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2d304e640 1 -- 172.21.15.77:0/2899450630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b8005080 msgr2=0x7ff2b800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2d304e640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b8005080 0x7ff2b800b9c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009a90 tx=0x7ff2b8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:33.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2d304e640 1 -- 172.21.15.77:0/2899450630 shutdown_connections 2024-03-06T07:25:33.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2d304e640 1 --2- 172.21.15.77:0/2899450630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b8005080 0x7ff2b800b9c0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2d304e640 1 -- 172.21.15.77:0/2899450630 >> 172.21.15.77:0/2899450630 conn(0x7ff2cc0d3ce0 msgr2=0x7ff2cc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2d304e640 1 -- 172.21.15.77:0/2899450630 shutdown_connections 2024-03-06T07:25:33.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.776+0000 7ff2d304e640 1 -- 172.21.15.77:0/2899450630 wait complete. 2024-03-06T07:25:33.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.777+0000 7ff2d304e640 1 Processor -- start 2024-03-06T07:25:33.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2d304e640 1 -- start start 2024-03-06T07:25:33.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2d304e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b8005080 0x7ff2cc0ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2d304e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2cc156430 0x7ff2cc0c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2d304e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2cc15fe30 0x7ff2cc1602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2d304e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc0ce950 con 0x7ff2cc156430 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2d304e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc0cecc0 con 0x7ff2b8005080 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2d304e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2cc0b4840 con 0x7ff2cc15fe30 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2cbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2cc156430 0x7ff2cc0c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.778+0000 7ff2cbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2cc156430 0x7ff2cc0c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48272/0 (socket says 172.21.15.77:48272) 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2cbfff640 1 -- 172.21.15.77:0/2438816096 learned_addr learned my addr 172.21.15.77:0/2438816096 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2d0dea640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b8005080 0x7ff2cc0ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2d15eb640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2cc15fe30 0x7ff2cc1602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2cbfff640 1 -- 172.21.15.77:0/2438816096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2cc15fe30 msgr2=0x7ff2cc1602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2cbfff640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2cc15fe30 0x7ff2cc1602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2cbfff640 1 -- 172.21.15.77:0/2438816096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b8005080 msgr2=0x7ff2cc0ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2cbfff640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b8005080 0x7ff2cc0ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2cbfff640 1 -- 172.21.15.77:0/2438816096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc0c8a70 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2cbfff640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2cc156430 0x7ff2cc0c8220 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400e7b0 tx=0x7ff2b400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2c97fa640 1 -- 172.21.15.77:0/2438816096 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b400c870 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2c97fa640 1 -- 172.21.15.77:0/2438816096 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b4018070 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2cc0c8d50 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2c97fa640 1 -- 172.21.15.77:0/2438816096 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b400cc50 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.779+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2cc0c9310 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.780+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2cc1607f0 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.781+0000 7ff2c97fa640 1 -- 172.21.15.77:0/2438816096 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2b401d460 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.781+0000 7ff2c97fa640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.781+0000 7ff2c97fa640 1 -- 172.21.15.77:0/2438816096 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2b4089e10 con 0x7ff2cc156430 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.781+0000 7ff2d0dea640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.782+0000 7ff2d0dea640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4001f90 tx=0x7ff2c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:33.786+0000 7ff2c97fa640 1 -- 172.21.15.77:0/2438816096 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2b40586d0 con 0x7ff2cc156430 2024-03-06T07:25:34.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.000+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff2cc0b65a0 con 0x7ff2cc156430 2024-03-06T07:25:34.002 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:25:34.000+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.002+0000 7ff2c97fa640 1 -- 172.21.15.77:0/2438816096 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff2cc0b65a0 con 0x7ff2cc156430 2024-03-06T07:25:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2ac06b4d0 msgr2=0x7ff2ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4001f90 tx=0x7ff2c4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2cc156430 msgr2=0x7ff2cc0c8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2cc156430 0x7ff2cc0c8220 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400e7b0 tx=0x7ff2b400ec70 comp rx=0 tx=0).stop 2024-03-06T07:25:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 shutdown_connections 2024-03-06T07:25:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2cc156430 0x7ff2cc0c8220 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2cc15fe30 0x7ff2cc1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 --2- 172.21.15.77:0/2438816096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2b8005080 0x7ff2cc0ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 >> 172.21.15.77:0/2438816096 conn(0x7ff2cc0d3ce0 msgr2=0x7ff2cc0d03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 shutdown_connections 2024-03-06T07:25:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.005+0000 7ff2d304e640 1 -- 172.21.15.77:0/2438816096 wait complete. 2024-03-06T07:25:34.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:34.043 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:25:34.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.122+0000 7f14aeedf640 1 Processor -- start 2024-03-06T07:25:34.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.122+0000 7f14aeedf640 1 -- start start 2024-03-06T07:25:34.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14aeedf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14a815bb40 0x7f14a80b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14aeedf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a815f7d0 0x7f14a806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14aeedf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a815ff90 con 0x7f14a806d5c0 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14aeedf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a806c910 con 0x7f14a815bb40 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14aeedf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a8160270 con 0x7f14a815f7d0 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14ad47c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f14a806d5c0 0x7f14a80cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46158/0 (socket says 172.21.15.77:46158) 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14ad47c640 1 -- 172.21.15.77:0/593265923 learned_addr learned my addr 172.21.15.77:0/593265923 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088123177 0 0) 0x7f14a815ff90 con 0x7f14a806d5c0 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1490003520 con 0x7f14a806d5c0 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149ffff640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a815f7d0 0x7f14a806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14acc7b640 1 --2- 172.21.15.77:0/593265923 >> v2:172.21.15.158:3300/0 conn(0x7f14a815bb40 0x7f14a80b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237781617 0 0) 0x7f1490003520 con 0x7f14a806d5c0 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a815f7d0 msgr2=0x7f14a806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a815f7d0 0x7f14a806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 >> v2:172.21.15.158:3300/0 conn(0x7f14a815bb40 msgr2=0x7f14a80b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 --2- 172.21.15.77:0/593265923 >> v2:172.21.15.158:3300/0 conn(0x7f14a815bb40 0x7f14a80b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a8160550 con 0x7f14a806d5c0 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f14a00019f0 con 0x7f14a806d5c0 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14acc7b640 1 --2- 172.21.15.77:0/593265923 >> v2:172.21.15.158:3300/0 conn(0x7f14a815bb40 0x7f14a80b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 >> v1:172.21.15.77:6789/0 conn(0x7f14a806d5c0 legacy=0x7f14a80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a815f7d0 0x7f14900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1490003fe0 0x7f1490004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a806c910 con 0x7f1490004d40 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a8160270 con 0x7f1490003fe0 2024-03-06T07:25:34.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149000bac0 con 0x7f14a815f7d0 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14ad47c640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f14acc7b640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a815f7d0 0x7f14900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.123+0000 7f149ffff640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1490003fe0 0x7f1490004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.124+0000 7f14ad47c640 1 -- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a815f7d0 msgr2=0x7f14900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.124+0000 7f14ad47c640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a815f7d0 0x7f14900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.124+0000 7f14ad47c640 1 -- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1490003fe0 msgr2=0x7f1490004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.124+0000 7f14ad47c640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1490003fe0 0x7f1490004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.124+0000 7f14ad47c640 1 -- 172.21.15.77:0/593265923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490003520 con 0x7f1490004d40 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.124+0000 7f14ad47c640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14900051c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f14a00013d0 tx=0x7f1490005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba537e1f95af5feb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.124+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a000a4e0 con 0x7f1490004d40 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a0034650 con 0x7f1490004d40 2024-03-06T07:25:34.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f149f7fe640 1 -- 172.21.15.77:0/593265923 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0034a40 con 0x7f1490004d40 2024-03-06T07:25:34.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f14aeedf640 1 -- 172.21.15.77:0/593265923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 msgr2=0x7f14900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f14aeedf640 1 --2- 172.21.15.77:0/593265923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14900051c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f14a00013d0 tx=0x7f1490005830 comp rx=0 tx=0).stop 2024-03-06T07:25:34.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f14ad47c640 1 -- 172.21.15.77:0/593265923 reap_dead start 2024-03-06T07:25:34.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f14aeedf640 1 -- 172.21.15.77:0/593265923 shutdown_connections 2024-03-06T07:25:34.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f14aeedf640 1 -- 172.21.15.77:0/593265923 >> 172.21.15.77:0/593265923 conn(0x7f14a806b190 msgr2=0x7f14a806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f14aeedf640 1 -- 172.21.15.77:0/593265923 shutdown_connections 2024-03-06T07:25:34.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.125+0000 7f14aeedf640 1 -- 172.21.15.77:0/593265923 wait complete. 2024-03-06T07:25:34.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.126+0000 7f14aeedf640 1 Processor -- start 2024-03-06T07:25:34.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.126+0000 7f14aeedf640 1 -- start start 2024-03-06T07:25:34.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14aeedf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14a8138e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14aeedf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a814a840 0x7f14a814e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14aeedf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14a814eb80 0x7f14a815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14aeedf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a815bf20 con 0x7f1490004d40 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14aeedf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a815fc30 con 0x7f14a814eb80 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14aeedf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a815c0a0 con 0x7f14a814a840 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14acc7b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14a8138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14acc7b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14a8138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48310/0 (socket says 172.21.15.77:48310) 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14acc7b640 1 -- 172.21.15.77:0/156507727 learned_addr learned my addr 172.21.15.77:0/156507727 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14acc7b640 1 -- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a814a840 msgr2=0x7f14a814e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14ad47c640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14a814eb80 0x7f14a815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f149ffff640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a814a840 0x7f14a814e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.127+0000 7f14acc7b640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a814a840 0x7f14a814e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f14acc7b640 1 -- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14a814eb80 msgr2=0x7f14a815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f14acc7b640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14a814eb80 0x7f14a815a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f14acc7b640 1 -- 172.21.15.77:0/156507727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a813b740 con 0x7f1490004d40 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f14ad47c640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14a814eb80 0x7f14a815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f14acc7b640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14a8138e60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f14a4011700 tx=0x7f14a4011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f149d7fa640 1 -- 172.21.15.77:0/156507727 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4019070 con 0x7f1490004d40 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f149d7fa640 1 -- 172.21.15.77:0/156507727 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4004220 con 0x7f1490004d40 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a813ba20 con 0x7f1490004d40 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f149d7fa640 1 -- 172.21.15.77:0/156507727 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a400f040 con 0x7f1490004d40 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.128+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a813bfe0 con 0x7f1490004d40 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.129+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a80732b0 con 0x7f1490004d40 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.129+0000 7f149d7fa640 1 -- 172.21.15.77:0/156507727 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14a400f760 con 0x7f1490004d40 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.130+0000 7f149d7fa640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f149406b500 0x7f149406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.130+0000 7f149ffff640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f149406b500 0x7f149406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.130+0000 7f149d7fa640 1 -- 172.21.15.77:0/156507727 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f14a408a570 con 0x7f1490004d40 2024-03-06T07:25:34.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.130+0000 7f149ffff640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f149406b500 0x7f149406d9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f14a8153960 tx=0x7f1498008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.134+0000 7f149d7fa640 1 -- 172.21.15.77:0/156507727 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14a4059390 con 0x7f1490004d40 2024-03-06T07:25:34.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:34.343+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f14a8068ff0 con 0x7f1490004d40 2024-03-06T07:25:35.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.004+0000 7f149d7fa640 1 -- 172.21.15.77:0/156507727 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f14a4058c90 con 0x7f1490004d40 2024-03-06T07:25:35.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f149406b500 msgr2=0x7f149406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f149406b500 0x7f149406d9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f14a8153960 tx=0x7f1498008040 comp rx=0 tx=0).stop 2024-03-06T07:25:35.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 msgr2=0x7f14a8138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14a8138e60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f14a4011700 tx=0x7f14a4011bc0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 shutdown_connections 2024-03-06T07:25:35.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14a814eb80 0x7f14a815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14a814a840 0x7f14a814e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f149406b500 0x7f149406d9c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 --2- 172.21.15.77:0/156507727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1490004d40 0x7f14a8138e60 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 >> 172.21.15.77:0/156507727 conn(0x7f14a806b190 msgr2=0x7f14a80ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 shutdown_connections 2024-03-06T07:25:35.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.006+0000 7f14aeedf640 1 -- 172.21.15.77:0/156507727 wait complete. 2024-03-06T07:25:35.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:35.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.168+0000 7fdcb7163640 1 Processor -- start 2024-03-06T07:25:35.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.168+0000 7fdcb7163640 1 -- start start 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb7163640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdcb015ac00 0x7fdcb006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb7163640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb00708b0 0x7fdcb0070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb7163640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb015afe0 con 0x7fdcb015e7a0 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb7163640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb0071250 con 0x7fdcb015ac00 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb7163640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00cdc10 con 0x7fdcb00708b0 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb4eff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdcb015ac00 0x7fdcb006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb5700640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdcb015e7a0 0x7fdcb015ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46174/0 (socket says 172.21.15.77:46174) 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb5700640 1 -- 172.21.15.77:0/4220933604 learned_addr learned my addr 172.21.15.77:0/4220933604 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb4eff640 1 -- 172.21.15.77:0/4220933604 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb00708b0 msgr2=0x7fdcb0070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb4eff640 1 --2- 172.21.15.77:0/4220933604 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb00708b0 0x7fdcb0070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb4eff640 1 -- 172.21.15.77:0/4220933604 >> v1:172.21.15.77:6789/0 conn(0x7fdcb015e7a0 legacy=0x7fdcb015ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcb4eff640 1 -- 172.21.15.77:0/4220933604 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb00cd000 con 0x7fdcb015ac00 2024-03-06T07:25:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.169+0000 7fdcaf7fe640 1 -- 172.21.15.77:0/4220933604 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653192122 0 0) 0x7fdcb015afe0 con 0x7fdcb015e7a0 2024-03-06T07:25:35.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb4eff640 1 --2- 172.21.15.77:0/4220933604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb015ac00 0x7fdcb006c910 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdca400ba40 tx=0x7fdca402dff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67634a5c31c244e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcaf7fe640 1 -- 172.21.15.77:0/4220933604 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca403b070 con 0x7fdcb015ac00 2024-03-06T07:25:35.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcaf7fe640 1 -- 172.21.15.77:0/4220933604 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca402eda0 con 0x7fdcb015ac00 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcaf7fe640 1 -- 172.21.15.77:0/4220933604 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca4036660 con 0x7fdcb015ac00 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb7163640 1 -- 172.21.15.77:0/4220933604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb015ac00 msgr2=0x7fdcb006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb7163640 1 --2- 172.21.15.77:0/4220933604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb015ac00 0x7fdcb006c910 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdca400ba40 tx=0x7fdca402dff0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb7163640 1 -- 172.21.15.77:0/4220933604 shutdown_connections 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb7163640 1 --2- 172.21.15.77:0/4220933604 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb00708b0 0x7fdcb0070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb7163640 1 --2- 172.21.15.77:0/4220933604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb015ac00 0x7fdcb006c910 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb7163640 1 -- 172.21.15.77:0/4220933604 >> 172.21.15.77:0/4220933604 conn(0x7fdcb00b1650 msgr2=0x7fdcb00b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb7163640 1 -- 172.21.15.77:0/4220933604 shutdown_connections 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.170+0000 7fdcb7163640 1 -- 172.21.15.77:0/4220933604 wait complete. 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.172+0000 7fdcb7163640 1 Processor -- start 2024-03-06T07:25:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.172+0000 7fdcb7163640 1 -- start start 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.172+0000 7fdcb7163640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb00708b0 0x7fdcb00b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.172+0000 7fdcb7163640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdcb015ac00 0x7fdcb01410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.172+0000 7fdcb7163640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb015e7a0 0x7fdcb00cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.172+0000 7fdcb7163640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00cd8a0 con 0x7fdcb00708b0 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.172+0000 7fdcb7163640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00cd5c0 con 0x7fdcb015ac00 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.172+0000 7fdcb7163640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb006ce50 con 0x7fdcb015e7a0 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb00708b0 0x7fdcb00b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb00708b0 0x7fdcb00b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48324/0 (socket says 172.21.15.77:48324) 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 -- 172.21.15.77:0/1652317336 learned_addr learned my addr 172.21.15.77:0/1652317336 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb5700640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb015e7a0 0x7fdcb00cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcaffff640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdcb015ac00 0x7fdcb01410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 -- 172.21.15.77:0/1652317336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb015e7a0 msgr2=0x7fdcb00cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb015e7a0 0x7fdcb00cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 -- 172.21.15.77:0/1652317336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdcb015ac00 msgr2=0x7fdcb01410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdcb015ac00 0x7fdcb01410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 -- 172.21.15.77:0/1652317336 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb0141820 con 0x7fdcb00708b0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb4eff640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb00708b0 0x7fdcb00b2740 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fdca400bb60 tx=0x7fdca40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcad7fa640 1 -- 172.21.15.77:0/1652317336 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca4046020 con 0x7fdcb00708b0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcad7fa640 1 -- 172.21.15.77:0/1652317336 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca4041690 con 0x7fdcb00708b0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb00b2de0 con 0x7fdcb00708b0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcad7fa640 1 -- 172.21.15.77:0/1652317336 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca403b040 con 0x7fdcb00708b0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.173+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb00b33a0 con 0x7fdcb00708b0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.174+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb004f330 con 0x7fdcb00708b0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.175+0000 7fdcad7fa640 1 -- 172.21.15.77:0/1652317336 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdca4049020 con 0x7fdcb00708b0 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.175+0000 7fdcad7fa640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc9406b4d0 0x7fdc9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.175+0000 7fdcaffff640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc9406b4d0 0x7fdc9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.175+0000 7fdcad7fa640 1 -- 172.21.15.77:0/1652317336 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdca40aed40 con 0x7fdcb00708b0 2024-03-06T07:25:35.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.176+0000 7fdcaffff640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc9406b4d0 0x7fdc9406d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0072b60 tx=0x7fdc9c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.180+0000 7fdcad7fa640 1 -- 172.21.15.77:0/1652317336 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdca4050030 con 0x7fdcb00708b0 2024-03-06T07:25:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:35.354+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdcb0074880 con 0x7fdcb00708b0 2024-03-06T07:25:36.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.314+0000 7fdcad7fa640 1 -- 172.21.15.77:0/1652317336 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v71) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdca407dc80 con 0x7fdcb00708b0 2024-03-06T07:25:36.316 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc9406b4d0 msgr2=0x7fdc9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc9406b4d0 0x7fdc9406d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0072b60 tx=0x7fdc9c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb00708b0 msgr2=0x7fdcb00b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb00708b0 0x7fdcb00b2740 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fdca400bb60 tx=0x7fdca40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 shutdown_connections 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdcb015e7a0 0x7fdcb00cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdcb015ac00 0x7fdcb01410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc9406b4d0 0x7fdc9406d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 --2- 172.21.15.77:0/1652317336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb00708b0 0x7fdcb00b2740 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 >> 172.21.15.77:0/1652317336 conn(0x7fdcb00b1650 msgr2=0x7fdcb00afc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.318+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 shutdown_connections 2024-03-06T07:25:36.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.319+0000 7fdcb7163640 1 -- 172.21.15.77:0/1652317336 wait complete. 2024-03-06T07:25:36.334 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:36.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.442+0000 7f382d393640 1 Processor -- start 2024-03-06T07:25:36.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.442+0000 7f382d393640 1 -- start start 2024-03-06T07:25:36.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f382d393640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f382815dfa0 0x7f38280d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f382d393640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38280d4630 0x7f38280d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f382d393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 0x7f38280d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f382d393640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38280d2100 con 0x7f38280d4630 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f382d393640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f382815e380 con 0x7f382815dfa0 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f382d393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38280d1e30 con 0x7f38280d2820 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38267fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38280d4630 0x7f38280d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38267fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38280d4630 0x7f38280d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48326/0 (socket says 172.21.15.77:48326) 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38267fc640 1 -- 172.21.15.77:0/1044575123 learned_addr learned my addr 172.21.15.77:0/1044575123 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38277fe640 1 --2- 172.21.15.77:0/1044575123 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 0x7f38280d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38267fc640 1 -- 172.21.15.77:0/1044575123 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 msgr2=0x7f38280d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38267fc640 1 --2- 172.21.15.77:0/1044575123 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 0x7f38280d05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38267fc640 1 -- 172.21.15.77:0/1044575123 >> v2:172.21.15.158:3300/0 conn(0x7f382815dfa0 msgr2=0x7f38280d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38267fc640 1 --2- 172.21.15.77:0/1044575123 >> v2:172.21.15.158:3300/0 conn(0x7f382815dfa0 0x7f38280d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.443+0000 7f38267fc640 1 -- 172.21.15.77:0/1044575123 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280cf470 con 0x7f38280d4630 2024-03-06T07:25:36.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.444+0000 7f38267fc640 1 --2- 172.21.15.77:0/1044575123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38280d4630 0x7f38280d22e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f382808d560 tx=0x7f381c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2918b1db8429350 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.444+0000 7f3825ffb640 1 -- 172.21.15.77:0/1044575123 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c02e070 con 0x7f38280d4630 2024-03-06T07:25:36.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.444+0000 7f3825ffb640 1 -- 172.21.15.77:0/1044575123 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381c037070 con 0x7f38280d4630 2024-03-06T07:25:36.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.444+0000 7f3825ffb640 1 -- 172.21.15.77:0/1044575123 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c031a00 con 0x7f38280d4630 2024-03-06T07:25:36.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 -- 172.21.15.77:0/1044575123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38280d4630 msgr2=0x7f38280d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 --2- 172.21.15.77:0/1044575123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38280d4630 0x7f38280d22e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f382808d560 tx=0x7f381c02b880 comp rx=0 tx=0).stop 2024-03-06T07:25:36.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 -- 172.21.15.77:0/1044575123 shutdown_connections 2024-03-06T07:25:36.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 --2- 172.21.15.77:0/1044575123 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 0x7f38280d05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 --2- 172.21.15.77:0/1044575123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38280d4630 0x7f38280d22e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 --2- 172.21.15.77:0/1044575123 >> v2:172.21.15.158:3300/0 conn(0x7f382815dfa0 0x7f38280d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 -- 172.21.15.77:0/1044575123 >> 172.21.15.77:0/1044575123 conn(0x7f382809d000 msgr2=0x7f38280d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 -- 172.21.15.77:0/1044575123 shutdown_connections 2024-03-06T07:25:36.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.445+0000 7f382d393640 1 -- 172.21.15.77:0/1044575123 wait complete. 2024-03-06T07:25:36.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.446+0000 7f382d393640 1 Processor -- start 2024-03-06T07:25:36.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.446+0000 7f382d393640 1 -- start start 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f382d393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 0x7f382815bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f382d393640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38280d4630 0x7f38280cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f382d393640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f382815dfa0 0x7f38281f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f382d393640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280d0b30 con 0x7f382815dfa0 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f382d393640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280cfa30 con 0x7f38280d4630 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f382d393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38280d0cb0 con 0x7f38280d2820 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f382815dfa0 0x7f38281f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f382815dfa0 0x7f38281f6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48330/0 (socket says 172.21.15.77:48330) 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 -- 172.21.15.77:0/3421596123 learned_addr learned my addr 172.21.15.77:0/3421596123 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38267fc640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38280d4630 0x7f38280cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f3826ffd640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 0x7f382815bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 -- 172.21.15.77:0/3421596123 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 msgr2=0x7f382815bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 0x7f382815bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 -- 172.21.15.77:0/3421596123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38280d4630 msgr2=0x7f38280cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38280d4630 0x7f38280cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 -- 172.21.15.77:0/3421596123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280c8990 con 0x7f382815dfa0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.447+0000 7f38277fe640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f382815dfa0 0x7f38281f6d30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f382000cd70 tx=0x7f382000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.448+0000 7f380f7fe640 1 -- 172.21.15.77:0/3421596123 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820003b90 con 0x7f382815dfa0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.448+0000 7f380f7fe640 1 -- 172.21.15.77:0/3421596123 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3820003d30 con 0x7f382815dfa0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.448+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38280c8c30 con 0x7f382815dfa0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.448+0000 7f380f7fe640 1 -- 172.21.15.77:0/3421596123 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820004240 con 0x7f382815dfa0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.448+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38281e6710 con 0x7f382815dfa0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.449+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382804f330 con 0x7f382815dfa0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.449+0000 7f380f7fe640 1 -- 172.21.15.77:0/3421596123 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38200043e0 con 0x7f382815dfa0 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.449+0000 7f380f7fe640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.450+0000 7f3826ffd640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.450+0000 7f380f7fe640 1 -- 172.21.15.77:0/3421596123 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f382008ab30 con 0x7f382815dfa0 2024-03-06T07:25:36.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.450+0000 7f3826ffd640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f38280cbd50 tx=0x7f3818008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.454+0000 7f380f7fe640 1 -- 172.21.15.77:0/3421596123 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3820059ba0 con 0x7f382815dfa0 2024-03-06T07:25:36.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:36.625+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f38280747a0 con 0x7f382815dfa0 2024-03-06T07:25:37.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.315+0000 7f380f7fe640 1 -- 172.21.15.77:0/3421596123 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v72) v1 ==== 169+0+0 (secure 0 0 0) 0x7f38200594a0 con 0x7f382815dfa0 2024-03-06T07:25:37.317 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:37.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 msgr2=0x7f381406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f38280cbd50 tx=0x7f3818008040 comp rx=0 tx=0).stop 2024-03-06T07:25:37.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f382815dfa0 msgr2=0x7f38281f6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f382815dfa0 0x7f38281f6d30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f382000cd70 tx=0x7f382000a660 comp rx=0 tx=0).stop 2024-03-06T07:25:37.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 shutdown_connections 2024-03-06T07:25:37.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f382815dfa0 0x7f38281f6d30 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38280d4630 0x7f38280cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 --2- 172.21.15.77:0/3421596123 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f38280d2820 0x7f382815bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 >> 172.21.15.77:0/3421596123 conn(0x7f382809d000 msgr2=0x7f38281605a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 shutdown_connections 2024-03-06T07:25:37.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.318+0000 7f382d393640 1 -- 172.21.15.77:0/3421596123 wait complete. 2024-03-06T07:25:37.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:25:37.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.439+0000 7f5b33e66640 1 Processor -- start 2024-03-06T07:25:37.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.439+0000 7f5b33e66640 1 -- start start 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b33e66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 0x7f5b2c15e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b33e66640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0ad900 con 0x7f5b2c15ea20 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b33e66640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c15bd20 con 0x7f5b2c15b940 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b33e66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0adb70 con 0x7f5b2c0d01a0 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b32403640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5b2c15ea20 0x7f5b2c070b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46176/0 (socket says 172.21.15.77:46176) 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b32403640 1 -- 172.21.15.77:0/2439214606 learned_addr learned my addr 172.21.15.77:0/2439214606 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b31401640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 0x7f5b2c15e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651601132 0 0) 0x7f5b2c0ad900 con 0x7f5b2c15ea20 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.440+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b18003520 con 0x7f5b2c15ea20 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3611684097 0 0) 0x7f5b18003520 con 0x7f5b2c15ea20 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 msgr2=0x7f5b2c15e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 0x7f5b2c15e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 >> v1:172.21.15.158:6789/0 conn(0x7f5b2c15b940 legacy=0x7f5b2c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c0ade50 con 0x7f5b2c15ea20 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f5b240019f0 con 0x7f5b2c15ea20 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 >> v1:172.21.15.77:6789/0 conn(0x7f5b2c15ea20 legacy=0x7f5b2c070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 0x7f5b180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b18003f20 0x7f5b18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b18004d40 0x7f5b180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0adb70 con 0x7f5b18003f20 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1800ba20 con 0x7f5b18004d40 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1800bc20 con 0x7f5b2c0d01a0 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b31401640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b18003f20 0x7f5b18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b31c02640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 0x7f5b180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b32403640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b18004d40 0x7f5b180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b31401640 1 -- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 msgr2=0x7f5b180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b31401640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 0x7f5b180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b31401640 1 -- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b18004d40 msgr2=0x7f5b180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b31401640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b18004d40 0x7f5b180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b31401640 1 -- 172.21.15.77:0/2439214606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18003520 con 0x7f5b18003f20 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b31c02640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b2c0d01a0 0x7f5b180039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.441+0000 7f5b32403640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b18004d40 0x7f5b180051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.442+0000 7f5b31401640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b18003f20 0x7f5b18004800 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009ac0 tx=0x7f5b18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2485bb7c7d999040 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.442+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28039070 con 0x7f5b18003f20 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.442+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2802c600 con 0x7f5b18003f20 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.442+0000 7f5b30c00640 1 -- 172.21.15.77:0/2439214606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28009690 con 0x7f5b18003f20 2024-03-06T07:25:37.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.443+0000 7f5b33e66640 1 -- 172.21.15.77:0/2439214606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b18003f20 msgr2=0x7f5b18004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.443+0000 7f5b33e66640 1 --2- 172.21.15.77:0/2439214606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b18003f20 0x7f5b18004800 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009ac0 tx=0x7f5b18005830 comp rx=0 tx=0).stop 2024-03-06T07:25:37.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.443+0000 7f5b32403640 1 -- 172.21.15.77:0/2439214606 reap_dead start 2024-03-06T07:25:37.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.443+0000 7f5b33e66640 1 -- 172.21.15.77:0/2439214606 shutdown_connections 2024-03-06T07:25:37.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.443+0000 7f5b33e66640 1 -- 172.21.15.77:0/2439214606 >> 172.21.15.77:0/2439214606 conn(0x7f5b2c06b190 msgr2=0x7f5b2c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.443+0000 7f5b33e66640 1 -- 172.21.15.77:0/2439214606 shutdown_connections 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.443+0000 7f5b33e66640 1 -- 172.21.15.77:0/2439214606 wait complete. 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.444+0000 7f5b33e66640 1 Processor -- start 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.444+0000 7f5b33e66640 1 -- start start 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b33e66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b18003f20 0x7f5b2c14a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b33e66640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b2c15ac50 0x7f5b2c0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b33e66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c15cb10 0x7f5b2c15cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b33e66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c15f210 con 0x7f5b2c15cb10 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b33e66640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c15eed0 con 0x7f5b2c15ac50 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b33e66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0ae650 con 0x7f5b18003f20 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c15cb10 0x7f5b2c15cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c15cb10 0x7f5b2c15cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48350/0 (socket says 172.21.15.77:48350) 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 -- 172.21.15.77:0/5680229 learned_addr learned my addr 172.21.15.77:0/5680229 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 -- 172.21.15.77:0/5680229 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b18003f20 msgr2=0x7f5b2c14a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b31401640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b2c15ac50 0x7f5b2c0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b31c02640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b18003f20 0x7f5b2c14a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b18003f20 0x7f5b2c14a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 -- 172.21.15.77:0/5680229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b2c15ac50 msgr2=0x7f5b2c0b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b2c15ac50 0x7f5b2c0b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 -- 172.21.15.77:0/5680229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c1598d0 con 0x7f5b2c15cb10 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b32403640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c15cb10 0x7f5b2c15cfc0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c1629b0 tx=0x7f5b2400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b227fc640 1 -- 172.21.15.77:0/5680229 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24004d60 con 0x7f5b2c15cb10 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b227fc640 1 -- 172.21.15.77:0/5680229 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2400bb20 con 0x7f5b2c15cb10 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2c159bb0 con 0x7f5b2c15cb10 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b227fc640 1 -- 172.21.15.77:0/5680229 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24007110 con 0x7f5b2c15cb10 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.445+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c15a170 con 0x7f5b2c15cb10 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.446+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2c0732b0 con 0x7f5b2c15cb10 2024-03-06T07:25:37.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.447+0000 7f5b227fc640 1 -- 172.21.15.77:0/5680229 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b2401b020 con 0x7f5b2c15cb10 2024-03-06T07:25:37.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.447+0000 7f5b227fc640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b1006b500 0x7f5b1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.447+0000 7f5b227fc640 1 -- 172.21.15.77:0/5680229 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5b24089fe0 con 0x7f5b2c15cb10 2024-03-06T07:25:37.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.447+0000 7f5b31c02640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b1006b500 0x7f5b1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.448+0000 7f5b31c02640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b1006b500 0x7f5b1006d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c007ac0 tx=0x7f5b1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.451+0000 7f5b227fc640 1 -- 172.21.15.77:0/5680229 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b240591f0 con 0x7f5b2c15cb10 2024-03-06T07:25:37.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.627+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f5b2c068ff0 con 0x7f5b2c15cb10 2024-03-06T07:25:37.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.631+0000 7f5b227fc640 1 -- 172.21.15.77:0/5680229 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v24) v1 ==== 108+0+0 (secure 0 0 0) 0x7f5b24058af0 con 0x7f5b2c15cb10 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b1006b500 msgr2=0x7f5b1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b1006b500 0x7f5b1006d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c007ac0 tx=0x7f5b1c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c15cb10 msgr2=0x7f5b2c15cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c15cb10 0x7f5b2c15cfc0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c1629b0 tx=0x7f5b2400b4a0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 shutdown_connections 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b1006b500 0x7f5b1006d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c15cb10 0x7f5b2c15cfc0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b2c15ac50 0x7f5b2c0b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 --2- 172.21.15.77:0/5680229 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b18003f20 0x7f5b2c14a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 >> 172.21.15.77:0/5680229 conn(0x7f5b2c06b190 msgr2=0x7f5b2c070e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 shutdown_connections 2024-03-06T07:25:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.633+0000 7f5b33e66640 1 -- 172.21.15.77:0/5680229 wait complete. 2024-03-06T07:25:37.646 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:37.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.750+0000 7f6ea0d13640 1 Processor -- start 2024-03-06T07:25:37.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.750+0000 7f6ea0d13640 1 -- start start 2024-03-06T07:25:37.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6ea0d13640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c15a220 0x7f6e9c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6ea0d13640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6e9c0712d0 0x7f6e9c071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6ea0d13640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e9c15de60 0x7f6e9c15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6ea0d13640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c0cf610 con 0x7f6e9c0712d0 2024-03-06T07:25:37.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6ea0d13640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c071c70 con 0x7f6e9c15de60 2024-03-06T07:25:37.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6ea0d13640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0b3320 con 0x7f6e9c15a220 2024-03-06T07:25:37.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6e99d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6e9c0712d0 0x7f6e9c071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6e99d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6e9c0712d0 0x7f6e9c071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48364/0 (socket says 172.21.15.77:48364) 2024-03-06T07:25:37.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6e99d9b640 1 -- 172.21.15.77:0/3801945505 learned_addr learned my addr 172.21.15.77:0/3801945505 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:37.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6e99d9b640 1 -- 172.21.15.77:0/3801945505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c15a220 msgr2=0x7f6e9c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6e99d9b640 1 --2- 172.21.15.77:0/3801945505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c15a220 0x7f6e9c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6e99d9b640 1 -- 172.21.15.77:0/3801945505 >> v2:172.21.15.158:3300/0 conn(0x7f6e9c15de60 msgr2=0x7f6e9c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6e99d9b640 1 --2- 172.21.15.77:0/3801945505 >> v2:172.21.15.158:3300/0 conn(0x7f6e9c15de60 0x7f6e9c15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.751+0000 7f6e99d9b640 1 -- 172.21.15.77:0/3801945505 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c0728d0 con 0x7f6e9c0712d0 2024-03-06T07:25:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.752+0000 7f6e99d9b640 1 --2- 172.21.15.77:0/3801945505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c0712d0 0x7f6e9c071730 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c15ddc0 tx=0x7f6e9002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1674f10655d236 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.752+0000 7f6e9959a640 1 -- 172.21.15.77:0/3801945505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9002e070 con 0x7f6e9c0712d0 2024-03-06T07:25:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.752+0000 7f6e9959a640 1 -- 172.21.15.77:0/3801945505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90037070 con 0x7f6e9c0712d0 2024-03-06T07:25:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.752+0000 7f6e9959a640 1 -- 172.21.15.77:0/3801945505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90031a00 con 0x7f6e9c0712d0 2024-03-06T07:25:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 -- 172.21.15.77:0/3801945505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c0712d0 msgr2=0x7f6e9c071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/3801945505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c0712d0 0x7f6e9c071730 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c15ddc0 tx=0x7f6e9002b880 comp rx=0 tx=0).stop 2024-03-06T07:25:37.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 -- 172.21.15.77:0/3801945505 shutdown_connections 2024-03-06T07:25:37.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/3801945505 >> v2:172.21.15.158:3300/0 conn(0x7f6e9c15de60 0x7f6e9c15e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/3801945505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c0712d0 0x7f6e9c071730 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/3801945505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c15a220 0x7f6e9c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 -- 172.21.15.77:0/3801945505 >> 172.21.15.77:0/3801945505 conn(0x7f6e9c06b190 msgr2=0x7f6e9c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 -- 172.21.15.77:0/3801945505 shutdown_connections 2024-03-06T07:25:37.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.753+0000 7f6ea0d13640 1 -- 172.21.15.77:0/3801945505 wait complete. 2024-03-06T07:25:37.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.754+0000 7f6ea0d13640 1 Processor -- start 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.754+0000 7f6ea0d13640 1 -- start start 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6ea0d13640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c0712d0 0x7f6e9c1f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6ea0d13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c15a220 0x7f6e9c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6ea0d13640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e9c15de60 0x7f6e9c06ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6ea0d13640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c072030 con 0x7f6e9c15a220 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6ea0d13640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c072bb0 con 0x7f6e9c15de60 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6ea0d13640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c072530 con 0x7f6e9c0712d0 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e99d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c15a220 0x7f6e9c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e99d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c15a220 0x7f6e9c06d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48370/0 (socket says 172.21.15.77:48370) 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e99d9b640 1 -- 172.21.15.77:0/394703536 learned_addr learned my addr 172.21.15.77:0/394703536 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e9a59c640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c0712d0 0x7f6e9c1f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e9ad9d640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e9c15de60 0x7f6e9c06ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e99d9b640 1 -- 172.21.15.77:0/394703536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c0712d0 msgr2=0x7f6e9c1f13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e99d9b640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c0712d0 0x7f6e9c1f13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e99d9b640 1 -- 172.21.15.77:0/394703536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e9c15de60 msgr2=0x7f6e9c06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e99d9b640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e9c15de60 0x7f6e9c06ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.755+0000 7f6e99d9b640 1 -- 172.21.15.77:0/394703536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c0adf80 con 0x7f6e9c15a220 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.756+0000 7f6e99d9b640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c15a220 0x7f6e9c06d880 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6e90000c00 tx=0x7f6e900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.756+0000 7f6e8affd640 1 -- 172.21.15.77:0/394703536 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90032880 con 0x7f6e9c15a220 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.756+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c1f4ba0 con 0x7f6e9c15a220 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.756+0000 7f6e8affd640 1 -- 172.21.15.77:0/394703536 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90032a20 con 0x7f6e9c15a220 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.756+0000 7f6e8affd640 1 -- 172.21.15.77:0/394703536 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90032e10 con 0x7f6e9c15a220 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.756+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c1f7950 con 0x7f6e9c15a220 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.757+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c0732b0 con 0x7f6e9c15a220 2024-03-06T07:25:37.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.757+0000 7f6e8affd640 1 -- 172.21.15.77:0/394703536 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e9004c060 con 0x7f6e9c15a220 2024-03-06T07:25:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.757+0000 7f6e8affd640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7c06b500 0x7f6e7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.758+0000 7f6e8affd640 1 -- 172.21.15.77:0/394703536 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6e900ab7e0 con 0x7f6e9c15a220 2024-03-06T07:25:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.758+0000 7f6e9a59c640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7c06b500 0x7f6e7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.758+0000 7f6e9a59c640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7c06b500 0x7f6e7c06d9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c009f20 tx=0x7f6e8c0094f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.762+0000 7f6e8affd640 1 -- 172.21.15.77:0/394703536 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e9c0adf80 con 0x7f6e9c15a220 2024-03-06T07:25:37.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.933+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6e9c0aef20 con 0x7f6e9c15a220 2024-03-06T07:25:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.937+0000 7f6e8affd640 1 -- 172.21.15.77:0/394703536 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6e90040070 con 0x7f6e9c15a220 2024-03-06T07:25:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.938+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7c06b500 msgr2=0x7f6e7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.938+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7c06b500 0x7f6e7c06d9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c009f20 tx=0x7f6e8c0094f0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.938+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c15a220 msgr2=0x7f6e9c06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.938+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c15a220 0x7f6e9c06d880 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6e90000c00 tx=0x7f6e900046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.939+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 shutdown_connections 2024-03-06T07:25:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.939+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7c06b500 0x7f6e7c06d9c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.939+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e9c15a220 0x7f6e9c06d880 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.939+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e9c15de60 0x7f6e9c06ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.939+0000 7f6ea0d13640 1 --2- 172.21.15.77:0/394703536 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e9c0712d0 0x7f6e9c1f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.939+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 >> 172.21.15.77:0/394703536 conn(0x7f6e9c06b190 msgr2=0x7f6e9c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.939+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 shutdown_connections 2024-03-06T07:25:37.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:37.939+0000 7f6ea0d13640 1 -- 172.21.15.77:0/394703536 wait complete. 2024-03-06T07:25:37.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:25:38.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.051+0000 7f4ba85c7640 1 Processor -- start 2024-03-06T07:25:38.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.051+0000 7f4ba85c7640 1 -- start start 2024-03-06T07:25:38.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba85c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba85c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4ba006b600 0x7f4ba0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba85c7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba015ea30 con 0x7f4ba006b220 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba85c7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba0161b70 con 0x7f4ba006b600 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba85c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba015ed10 con 0x7f4ba00ce920 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba6b64640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4ba006b600 0x7f4ba0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba6b64640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4ba006b600 0x7f4ba0162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48372/0 (socket says 172.21.15.77:48372) 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba6b64640 1 -- 172.21.15.77:0/3020075654 learned_addr learned my addr 172.21.15.77:0/3020075654 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba5b62640 1 --2- 172.21.15.77:0/3020075654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba6b64640 1 -- 172.21.15.77:0/3020075654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 msgr2=0x7f4ba015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba6b64640 1 --2- 172.21.15.77:0/3020075654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba6b64640 1 -- 172.21.15.77:0/3020075654 >> v1:172.21.15.158:6789/0 conn(0x7f4ba006b220 legacy=0x7f4ba0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba6b64640 1 -- 172.21.15.77:0/3020075654 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba015eff0 con 0x7f4ba006b600 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.052+0000 7f4ba5b62640 1 --2- 172.21.15.77:0/3020075654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba6b64640 1 --2- 172.21.15.77:0/3020075654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba0162000 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009e50 tx=0x7f4b9802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae5e17f88db3070 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba5361640 1 -- 172.21.15.77:0/3020075654 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9802ca70 con 0x7f4ba006b600 2024-03-06T07:25:38.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba5361640 1 -- 172.21.15.77:0/3020075654 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9802cc10 con 0x7f4ba006b600 2024-03-06T07:25:38.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba5361640 1 -- 172.21.15.77:0/3020075654 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b980343f0 con 0x7f4ba006b600 2024-03-06T07:25:38.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3020075654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 msgr2=0x7f4ba0162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3020075654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba0162000 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009e50 tx=0x7f4b9802bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3020075654 shutdown_connections 2024-03-06T07:25:38.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3020075654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.053+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3020075654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba0162000 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.054+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3020075654 >> 172.21.15.77:0/3020075654 conn(0x7f4ba006c0c0 msgr2=0x7f4ba006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.054+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3020075654 shutdown_connections 2024-03-06T07:25:38.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.054+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3020075654 wait complete. 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.055+0000 7f4ba85c7640 1 Processor -- start 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.055+0000 7f4ba85c7640 1 -- start start 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.055+0000 7f4ba85c7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ba006b220 0x7f4ba014a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.055+0000 7f4ba85c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba00ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.055+0000 7f4ba85c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.055+0000 7f4ba85c7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba0072090 con 0x7f4ba006b600 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.055+0000 7f4ba85c7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00ce200 con 0x7f4ba006b220 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.055+0000 7f4ba85c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba01f7990 con 0x7f4ba00ce920 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba00ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba00ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48388/0 (socket says 172.21.15.77:48388) 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 -- 172.21.15.77:0/3278739009 learned_addr learned my addr 172.21.15.77:0/3278739009 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba6363640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ba006b220 0x7f4ba014a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 -- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 msgr2=0x7f4ba015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba6b64640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 -- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ba006b220 msgr2=0x7f4ba014a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ba006b220 0x7f4ba014a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 -- 172.21.15.77:0/3278739009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba01f7c70 con 0x7f4ba006b600 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba6363640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ba006b220 0x7f4ba014a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba5b62640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba00ad860 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c7f0 tx=0x7f4b9000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4b96ffd640 1 -- 172.21.15.77:0/3278739009 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90015070 con 0x7f4ba006b600 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4b96ffd640 1 -- 172.21.15.77:0/3278739009 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b90007820 con 0x7f4ba006b600 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba01f7f50 con 0x7f4ba006b600 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4b96ffd640 1 -- 172.21.15.77:0/3278739009 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90007bd0 con 0x7f4ba006b600 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.056+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba00b0220 con 0x7f4ba006b600 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.057+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba00d0460 con 0x7f4ba006b600 2024-03-06T07:25:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.058+0000 7f4b96ffd640 1 -- 172.21.15.77:0/3278739009 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b90018070 con 0x7f4ba006b600 2024-03-06T07:25:38.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.058+0000 7f4b96ffd640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b8406b4d0 0x7f4b8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.058+0000 7f4b96ffd640 1 -- 172.21.15.77:0/3278739009 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4b90089fe0 con 0x7f4ba006b600 2024-03-06T07:25:38.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.058+0000 7f4ba6363640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b8406b4d0 0x7f4b8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.058+0000 7f4ba6363640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b8406b4d0 0x7f4b8406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00a830 tx=0x7f4b9c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.062+0000 7f4b96ffd640 1 -- 172.21.15.77:0/3278739009 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ba00d0460 con 0x7f4ba006b600 2024-03-06T07:25:38.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.302+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f4ba004f330 con 0x7f4ba006b600 2024-03-06T07:25:38.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.327+0000 7f4b96ffd640 1 -- 172.21.15.77:0/3278739009 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v205) v1 ==== 140+0+0 (secure 0 0 0) 0x7f4b900591f0 con 0x7f4ba006b600 2024-03-06T07:25:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b8406b4d0 msgr2=0x7f4b8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b8406b4d0 0x7f4b8406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00a830 tx=0x7f4b9c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 msgr2=0x7f4ba00ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba00ad860 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c7f0 tx=0x7f4b9000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 shutdown_connections 2024-03-06T07:25:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4b8406b4d0 0x7f4b8406d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ba006b600 0x7f4ba00ad860 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4ba00ce920 0x7f4ba015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 --2- 172.21.15.77:0/3278739009 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4ba006b220 0x7f4ba014a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 >> 172.21.15.77:0/3278739009 conn(0x7f4ba006c0c0 msgr2=0x7f4ba0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 shutdown_connections 2024-03-06T07:25:38.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.330+0000 7f4ba85c7640 1 -- 172.21.15.77:0/3278739009 wait complete. 2024-03-06T07:25:38.343 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:25:38.343 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:25:38.343 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:25:38.343 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:38.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.441+0000 7f2628c88640 1 Processor -- start 2024-03-06T07:25:38.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.441+0000 7f2628c88640 1 -- start start 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f2628c88640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f262406da40 0x7f26240cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f2628c88640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240cc980 con 0x7f26240cd7a0 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f2628c88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240ccc60 con 0x7f26241424f0 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f2628c88640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26240cd220 con 0x7f262406da40 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f2622d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f26240cd7a0 0x7f2624071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59344/0 (socket says 172.21.15.77:59344) 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f2622d9d640 1 -- 172.21.15.77:0/1867844903 learned_addr learned my addr 172.21.15.77:0/1867844903 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f2621d9b640 1 --2- 172.21.15.77:0/1867844903 >> v2:172.21.15.158:3300/0 conn(0x7f262406da40 0x7f26240cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664425838 0 0) 0x7f26240cc980 con 0x7f26240cd7a0 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f260c003520 con 0x7f26240cd7a0 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1396813809 0 0) 0x7f260c003520 con 0x7f26240cd7a0 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 >> v2:172.21.15.158:3300/0 conn(0x7f262406da40 msgr2=0x7f26240cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 --2- 172.21.15.77:0/1867844903 >> v2:172.21.15.158:3300/0 conn(0x7f262406da40 0x7f26240cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 >> v1:172.21.15.158:6789/0 conn(0x7f26241424f0 legacy=0x7f262406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26240ccf40 con 0x7f26240cd7a0 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f261c0019f0 con 0x7f26240cd7a0 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 >> v1:172.21.15.77:6789/0 conn(0x7f26240cd7a0 legacy=0x7f2624071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.442+0000 7f262159a640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f260c0039b0 0x7f260c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f262159a640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c004800 0x7f260c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f262159a640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f260c0051c0 0x7f260c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26240cd220 con 0x7f260c0051c0 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c00c060 con 0x7f260c0039b0 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c00c260 con 0x7f260c004800 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f2622d9d640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f260c0051c0 0x7f260c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f262259c640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f260c0039b0 0x7f260c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f2621d9b640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c004800 0x7f260c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f2622d9d640 1 -- 172.21.15.77:0/1867844903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c004800 msgr2=0x7f260c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f2622d9d640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c004800 0x7f260c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f2622d9d640 1 -- 172.21.15.77:0/1867844903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f260c0039b0 msgr2=0x7f260c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f2622d9d640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f260c0039b0 0x7f260c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.443+0000 7f2622d9d640 1 -- 172.21.15.77:0/1867844903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c003520 con 0x7f260c0051c0 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2622d9d640 1 -- 172.21.15.77:0/1867844903 reap_dead start 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2622d9d640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f260c0051c0 0x7f260c00b9c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f261c00a860 tx=0x7f260c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9737d2454749bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c00a480 con 0x7f260c0051c0 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c034630 con 0x7f260c0051c0 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f262159a640 1 -- 172.21.15.77:0/1867844903 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c034a20 con 0x7f260c0051c0 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2628c88640 1 -- 172.21.15.77:0/1867844903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f260c0051c0 msgr2=0x7f260c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2628c88640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f260c0051c0 0x7f260c00b9c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f261c00a860 tx=0x7f260c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2628c88640 1 -- 172.21.15.77:0/1867844903 shutdown_connections 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2628c88640 1 --2- 172.21.15.77:0/1867844903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f260c0051c0 0x7f260c00b9c0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2628c88640 1 -- 172.21.15.77:0/1867844903 >> 172.21.15.77:0/1867844903 conn(0x7f26240cf8a0 msgr2=0x7f26240cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2628c88640 1 -- 172.21.15.77:0/1867844903 shutdown_connections 2024-03-06T07:25:38.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.444+0000 7f2628c88640 1 -- 172.21.15.77:0/1867844903 wait complete. 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.446+0000 7f2628c88640 1 Processor -- start 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.446+0000 7f2628c88640 1 -- start start 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.446+0000 7f2628c88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c0051c0 0x7f26240b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.446+0000 7f2628c88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26240b3460 0x7f262413d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2628c88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2624140a50 0x7f26240ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2628c88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26240cc470 con 0x7f26240b3460 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2628c88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26240cc2f0 con 0x7f2624140a50 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2628c88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26240cc5f0 con 0x7f260c0051c0 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26240b3460 0x7f262413d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26240b3460 0x7f262413d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34034/0 (socket says 172.21.15.77:34034) 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 -- 172.21.15.77:0/2546083173 learned_addr learned my addr 172.21.15.77:0/2546083173 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f262259c640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c0051c0 0x7f26240b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2622d9d640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2624140a50 0x7f26240ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 -- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c0051c0 msgr2=0x7f26240b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c0051c0 0x7f26240b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 -- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2624140a50 msgr2=0x7f26240ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2624140a50 0x7f26240ae900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 -- 172.21.15.77:0/2546083173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26240b3a00 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f262259c640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c0051c0 0x7f26240b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2621d9b640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26240b3460 0x7f262413d320 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f26180118e0 tx=0x7f2618011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2612ffd640 1 -- 172.21.15.77:0/2546083173 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261800faf0 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2612ffd640 1 -- 172.21.15.77:0/2546083173 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261800fc90 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2612ffd640 1 -- 172.21.15.77:0/2546083173 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2618014440 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262413c840 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.447+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262413ce00 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.448+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262404f330 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.449+0000 7f2612ffd640 1 -- 172.21.15.77:0/2546083173 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2618012040 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.449+0000 7f2612ffd640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f260406b4d0 0x7f260406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.449+0000 7f2612ffd640 1 -- 172.21.15.77:0/2546083173 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f261808a5a0 con 0x7f26240b3460 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.449+0000 7f262259c640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f260406b4d0 0x7f260406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.449+0000 7f262259c640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f260406b4d0 0x7f260406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2614005580 tx=0x7f2614009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.453+0000 7f2612ffd640 1 -- 172.21.15.77:0/2546083173 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2618019030 con 0x7f26240b3460 2024-03-06T07:25:38.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.624+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26240d0620 con 0x7f26240b3460 2024-03-06T07:25:38.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.624+0000 7f2612ffd640 1 -- 172.21.15.77:0/2546083173 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+13009 (secure 0 0 0) 0x7f26180597b0 con 0x7f26240b3460 2024-03-06T07:25:38.626 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:38.626 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":72,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:37.315229+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/3491727415":"2024-03-07T07:25:34.001420+0000","172.21.15.77:6835/3491727415":"2024-03-07T07:25:34.001420+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:38.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f260406b4d0 msgr2=0x7f260406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f260406b4d0 0x7f260406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2614005580 tx=0x7f2614009040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26240b3460 msgr2=0x7f262413d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26240b3460 0x7f262413d320 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f26180118e0 tx=0x7f2618011da0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 shutdown_connections 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f260406b4d0 0x7f260406d990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26240b3460 0x7f262413d320 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2624140a50 0x7f26240ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 --2- 172.21.15.77:0/2546083173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f260c0051c0 0x7f26240b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.626+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 >> 172.21.15.77:0/2546083173 conn(0x7f26240cf8a0 msgr2=0x7f262415f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.627+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 shutdown_connections 2024-03-06T07:25:38.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.627+0000 7f2628c88640 1 -- 172.21.15.77:0/2546083173 wait complete. 2024-03-06T07:25:38.639 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:38.640 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:38.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.740+0000 7f9a24381640 1 Processor -- start 2024-03-06T07:25:38.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.740+0000 7f9a24381640 1 -- start start 2024-03-06T07:25:38.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.741+0000 7f9a24381640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c0d4860 0x7f9a1c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.741+0000 7f9a24381640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1c1618b0 con 0x7f9a1c1614d0 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.741+0000 7f9a24381640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1c0cf400 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.741+0000 7f9a24381640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0d4cc0 con 0x7f9a1c0d4860 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.741+0000 7f9a2211d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9a1c1614d0 0x7f9a1c0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59354/0 (socket says 172.21.15.77:59354) 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.741+0000 7f9a2211d640 1 -- 172.21.15.77:0/1670773258 learned_addr learned my addr 172.21.15.77:0/1670773258 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.742+0000 7f9a2191c640 1 --2- 172.21.15.77:0/1670773258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c0d4860 0x7f9a1c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.742+0000 7f9a2111b640 1 -- 172.21.15.77:0/1670773258 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884581170 0 0) 0x7f9a1c1618b0 con 0x7f9a1c1614d0 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.742+0000 7f9a2111b640 1 -- 172.21.15.77:0/1670773258 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a08003520 con 0x7f9a1c1614d0 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.742+0000 7f9a2111b640 1 -- 172.21.15.77:0/1670773258 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379685571 0 0) 0x7f9a08003520 con 0x7f9a1c1614d0 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.742+0000 7f9a2191c640 1 -- 172.21.15.77:0/1670773258 >> v1:172.21.15.158:6789/0 conn(0x7f9a1c0b0e30 legacy=0x7f9a1c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.742+0000 7f9a2191c640 1 -- 172.21.15.77:0/1670773258 >> v1:172.21.15.77:6789/0 conn(0x7f9a1c1614d0 legacy=0x7f9a1c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.742+0000 7f9a2191c640 1 -- 172.21.15.77:0/1670773258 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c0cf6a0 con 0x7f9a1c0d4860 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.743+0000 7f9a2191c640 1 --2- 172.21.15.77:0/1670773258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c0d4860 0x7f9a1c0d2020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0097d0 tx=0x7f9a0c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d1a71d011e307b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.743+0000 7f9a2111b640 1 -- 172.21.15.77:0/1670773258 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c037070 con 0x7f9a1c0d4860 2024-03-06T07:25:38.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.743+0000 7f9a2111b640 1 -- 172.21.15.77:0/1670773258 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0c030510 con 0x7f9a1c0d4860 2024-03-06T07:25:38.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.743+0000 7f9a2111b640 1 -- 172.21.15.77:0/1670773258 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18001670 con 0x7f9a1c0d4860 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.743+0000 7f9a24381640 1 -- 172.21.15.77:0/1670773258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c0d4860 msgr2=0x7f9a1c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.743+0000 7f9a24381640 1 --2- 172.21.15.77:0/1670773258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c0d4860 0x7f9a1c0d2020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0097d0 tx=0x7f9a0c02b800 comp rx=0 tx=0).stop 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.744+0000 7f9a24381640 1 -- 172.21.15.77:0/1670773258 shutdown_connections 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.744+0000 7f9a24381640 1 --2- 172.21.15.77:0/1670773258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c0d4860 0x7f9a1c0d2020 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.744+0000 7f9a24381640 1 -- 172.21.15.77:0/1670773258 >> 172.21.15.77:0/1670773258 conn(0x7f9a1c0d3d40 msgr2=0x7f9a1c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.744+0000 7f9a24381640 1 -- 172.21.15.77:0/1670773258 shutdown_connections 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.744+0000 7f9a24381640 1 -- 172.21.15.77:0/1670773258 wait complete. 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.745+0000 7f9a24381640 1 Processor -- start 2024-03-06T07:25:38.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.745+0000 7f9a24381640 1 -- start start 2024-03-06T07:25:38.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.745+0000 7f9a24381640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a1c0b0e30 0x7f9a1c156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.745+0000 7f9a24381640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1c0d4860 0x7f9a1c0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.745+0000 7f9a24381640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c1614d0 0x7f9a1c06caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.745+0000 7f9a24381640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0b0910 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.745+0000 7f9a24381640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0cfc60 con 0x7f9a1c0d4860 2024-03-06T07:25:38.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.745+0000 7f9a24381640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0d2560 con 0x7f9a1c1614d0 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a1c0b0e30 0x7f9a1c156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a1c0b0e30 0x7f9a1c156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34046/0 (socket says 172.21.15.77:34046) 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 -- 172.21.15.77:0/3514782460 learned_addr learned my addr 172.21.15.77:0/3514782460 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2191c640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1c0d4860 0x7f9a1c0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2291e640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c1614d0 0x7f9a1c06caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 -- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c1614d0 msgr2=0x7f9a1c06caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c1614d0 0x7f9a1c06caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 -- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1c0d4860 msgr2=0x7f9a1c0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1c0d4860 0x7f9a1c0c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 -- 172.21.15.77:0/3514782460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c156ca0 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2291e640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c1614d0 0x7f9a1c06caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2191c640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1c0d4860 0x7f9a1c0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a2211d640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a1c0b0e30 0x7f9a1c156310 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800c550 tx=0x7f9a1800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a12ffd640 1 -- 172.21.15.77:0/3514782460 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18017070 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a12ffd640 1 -- 172.21.15.77:0/3514782460 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a18005ce0 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1c0c84c0 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a12ffd640 1 -- 172.21.15.77:0/3514782460 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a180072b0 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.746+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1c0c8a80 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.747+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1c068ff0 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.748+0000 7f9a12ffd640 1 -- 172.21.15.77:0/3514782460 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a1801a070 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.748+0000 7f9a12ffd640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a0006b4d0 0x7f9a0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.748+0000 7f9a12ffd640 1 -- 172.21.15.77:0/3514782460 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9a180894d0 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.748+0000 7f9a2191c640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a0006b4d0 0x7f9a0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.748+0000 7f9a2191c640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a0006b4d0 0x7f9a0006d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c034000 tx=0x7f9a0c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.752+0000 7f9a12ffd640 1 -- 172.21.15.77:0/3514782460 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a18058540 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.924+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a1c0d3770 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.924+0000 7f9a12ffd640 1 -- 172.21.15.77:0/3514782460 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a18057e40 con 0x7f9a1c0b0e30 2024-03-06T07:25:38.926 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:25:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a0006b4d0 msgr2=0x7f9a0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a0006b4d0 0x7f9a0006d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c034000 tx=0x7f9a0c008030 comp rx=0 tx=0).stop 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a1c0b0e30 msgr2=0x7f9a1c156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a1c0b0e30 0x7f9a1c156310 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800c550 tx=0x7f9a1800ca10 comp rx=0 tx=0).stop 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 shutdown_connections 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a1c1614d0 0x7f9a1c06caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a1c0d4860 0x7f9a1c0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a0006b4d0 0x7f9a0006d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 --2- 172.21.15.77:0/3514782460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a1c0b0e30 0x7f9a1c156310 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.926+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 >> 172.21.15.77:0/3514782460 conn(0x7f9a1c0d3d40 msgr2=0x7f9a1c0b3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.927+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 shutdown_connections 2024-03-06T07:25:38.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:38.927+0000 7f9a24381640 1 -- 172.21.15.77:0/3514782460 wait complete. 2024-03-06T07:25:38.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:25:39.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.038+0000 7ff574720640 1 Processor -- start 2024-03-06T07:25:39.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.038+0000 7ff574720640 1 -- start start 2024-03-06T07:25:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff574720640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff56c145350 0x7ff56c0b4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff574720640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff56c0cf2a0 0x7ff56c0ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff574720640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cd2d0 0x7ff56c15f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff574720640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c0cd740 con 0x7ff56c0cf2a0 2024-03-06T07:25:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff574720640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c0b4830 con 0x7ff56c145350 2024-03-06T07:25:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff574720640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0b4b10 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff571cbb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff56c0cf2a0 0x7ff56c0ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff571cbb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff56c0cf2a0 0x7ff56c0ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34058/0 (socket says 172.21.15.77:34058) 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff571cbb640 1 -- 172.21.15.77:0/677192855 learned_addr learned my addr 172.21.15.77:0/677192855 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff572cbd640 1 --2- 172.21.15.77:0/677192855 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cd2d0 0x7ff56c15f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff5724bc640 1 --2- 172.21.15.77:0/677192855 >> v2:172.21.15.158:3300/0 conn(0x7ff56c145350 0x7ff56c0b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff571cbb640 1 -- 172.21.15.77:0/677192855 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cd2d0 msgr2=0x7ff56c15f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff571cbb640 1 --2- 172.21.15.77:0/677192855 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cd2d0 0x7ff56c15f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff571cbb640 1 -- 172.21.15.77:0/677192855 >> v2:172.21.15.158:3300/0 conn(0x7ff56c145350 msgr2=0x7ff56c0b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff571cbb640 1 --2- 172.21.15.77:0/677192855 >> v2:172.21.15.158:3300/0 conn(0x7ff56c145350 0x7ff56c0b4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff571cbb640 1 -- 172.21.15.77:0/677192855 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c0b4df0 con 0x7ff56c0cf2a0 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff572cbd640 1 --2- 172.21.15.77:0/677192855 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cd2d0 0x7ff56c15f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.039+0000 7ff5724bc640 1 --2- 172.21.15.77:0/677192855 >> v2:172.21.15.158:3300/0 conn(0x7ff56c145350 0x7ff56c0b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.040+0000 7ff571cbb640 1 --2- 172.21.15.77:0/677192855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cf2a0 0x7ff56c0ccd90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0b4a40 tx=0x7ff55c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3911474cd67bc118 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.040+0000 7ff5714ba640 1 -- 172.21.15.77:0/677192855 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c035020 con 0x7ff56c0cf2a0 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.040+0000 7ff5714ba640 1 -- 172.21.15.77:0/677192855 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55c031640 con 0x7ff56c0cf2a0 2024-03-06T07:25:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.040+0000 7ff5714ba640 1 -- 172.21.15.77:0/677192855 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c02d040 con 0x7ff56c0cf2a0 2024-03-06T07:25:39.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 -- 172.21.15.77:0/677192855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cf2a0 msgr2=0x7ff56c0ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 --2- 172.21.15.77:0/677192855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cf2a0 0x7ff56c0ccd90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0b4a40 tx=0x7ff55c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 -- 172.21.15.77:0/677192855 shutdown_connections 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 --2- 172.21.15.77:0/677192855 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cd2d0 0x7ff56c15f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 --2- 172.21.15.77:0/677192855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cf2a0 0x7ff56c0ccd90 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 --2- 172.21.15.77:0/677192855 >> v2:172.21.15.158:3300/0 conn(0x7ff56c145350 0x7ff56c0b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 -- 172.21.15.77:0/677192855 >> 172.21.15.77:0/677192855 conn(0x7ff56c06b3d0 msgr2=0x7ff56c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 -- 172.21.15.77:0/677192855 shutdown_connections 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.041+0000 7ff574720640 1 -- 172.21.15.77:0/677192855 wait complete. 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.042+0000 7ff574720640 1 Processor -- start 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.042+0000 7ff574720640 1 -- start start 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff574720640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cd2d0 0x7ff56c13c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff574720640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cf2a0 0x7ff56c157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff574720640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56c145350 0x7ff56c15dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff574720640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0cf730 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff574720640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c145730 con 0x7ff56c145350 2024-03-06T07:25:39.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff574720640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c15fc80 con 0x7ff56c0cf2a0 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cd2d0 0x7ff56c13c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cd2d0 0x7ff56c13c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34068/0 (socket says 172.21.15.77:34068) 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 -- 172.21.15.77:0/2364835121 learned_addr learned my addr 172.21.15.77:0/2364835121 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 -- 172.21.15.77:0/2364835121 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cf2a0 msgr2=0x7ff56c157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cf2a0 0x7ff56c157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff572cbd640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56c145350 0x7ff56c15dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 -- 172.21.15.77:0/2364835121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56c145350 msgr2=0x7ff56c15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56c145350 0x7ff56c15dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 -- 172.21.15.77:0/2364835121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c159990 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff572cbd640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56c145350 0x7ff56c15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff5724bc640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cd2d0 0x7ff56c13c9e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff56800ce00 tx=0x7ff56800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.043+0000 7ff562ffd640 1 -- 172.21.15.77:0/2364835121 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56800f890 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.044+0000 7ff562ffd640 1 -- 172.21.15.77:0/2364835121 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56800fa30 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.044+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c159c70 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.044+0000 7ff562ffd640 1 -- 172.21.15.77:0/2364835121 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56800fe40 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.044+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c15a230 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.045+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c04f330 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.045+0000 7ff562ffd640 1 -- 172.21.15.77:0/2364835121 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff568018400 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.045+0000 7ff562ffd640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff55006b4d0 0x7ff55006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.045+0000 7ff562ffd640 1 -- 172.21.15.77:0/2364835121 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff56808ad50 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.045+0000 7ff571cbb640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff55006b4d0 0x7ff55006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.046+0000 7ff571cbb640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff55006b4d0 0x7ff55006d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff55c036000 tx=0x7ff55c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.050+0000 7ff562ffd640 1 -- 172.21.15.77:0/2364835121 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff56808e030 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.286+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7ff56c0b33d0 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.329+0000 7ff562ffd640 1 -- 172.21.15.77:0/2364835121 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v206) v1 ==== 135+0+0 (secure 0 0 0) 0x7ff568014030 con 0x7ff56c0cd2d0 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff55006b4d0 msgr2=0x7ff55006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff55006b4d0 0x7ff55006d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff55c036000 tx=0x7ff55c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cd2d0 msgr2=0x7ff56c13c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cd2d0 0x7ff56c13c9e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff56800ce00 tx=0x7ff56800a650 comp rx=0 tx=0).stop 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 shutdown_connections 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff56c145350 0x7ff56c15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff56c0cf2a0 0x7ff56c157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff55006b4d0 0x7ff55006d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 --2- 172.21.15.77:0/2364835121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff56c0cd2d0 0x7ff56c13c9e0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 >> 172.21.15.77:0/2364835121 conn(0x7ff56c06b3d0 msgr2=0x7ff56c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 shutdown_connections 2024-03-06T07:25:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.331+0000 7ff574720640 1 -- 172.21.15.77:0/2364835121 wait complete. 2024-03-06T07:25:39.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:39.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.444+0000 7fcd4e1c9640 1 Processor -- start 2024-03-06T07:25:39.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.445+0000 7fcd4e1c9640 1 -- start start 2024-03-06T07:25:39.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.445+0000 7fcd4e1c9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcd481614d0 0x7fcd480d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.445+0000 7fcd4e1c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480d1910 0x7fcd480d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.445+0000 7fcd4e1c9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd480d2740 con 0x7fcd480d22b0 2024-03-06T07:25:39.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.445+0000 7fcd4e1c9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd481618b0 con 0x7fcd481614d0 2024-03-06T07:25:39.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.445+0000 7fcd4e1c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480cf400 con 0x7fcd480d1910 2024-03-06T07:25:39.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.445+0000 7fcd47fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcd480d22b0 0x7fcd480d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59368/0 (socket says 172.21.15.77:59368) 2024-03-06T07:25:39.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.445+0000 7fcd47fff640 1 -- 172.21.15.77:0/3403489838 learned_addr learned my addr 172.21.15.77:0/3403489838 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd477fe640 1 --2- 172.21.15.77:0/3403489838 >> v2:172.21.15.158:3300/0 conn(0x7fcd481614d0 0x7fcd480d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd3ffff640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480d1910 0x7fcd480d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183927938 0 0) 0x7fcd40001490 con 0x7fcd480d22b0 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd34003520 con 0x7fcd480d22b0 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3147974491 0 0) 0x7fcd34003520 con 0x7fcd480d22b0 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480d1910 msgr2=0x7fcd480d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480d1910 0x7fcd480d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 >> v2:172.21.15.158:3300/0 conn(0x7fcd481614d0 msgr2=0x7fcd480d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 --2- 172.21.15.77:0/3403489838 >> v2:172.21.15.158:3300/0 conn(0x7fcd481614d0 0x7fcd480d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd480cf6a0 con 0x7fcd480d22b0 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fcd40001bc0 con 0x7fcd480d22b0 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 >> v1:172.21.15.77:6789/0 conn(0x7fcd480d22b0 legacy=0x7fcd480d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340039b0 0x7fcd34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004800 0x7fcd34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480d1910 0x7fcd340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481618b0 con 0x7fcd340039b0 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480cf400 con 0x7fcd34004800 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3400b9c0 con 0x7fcd480d1910 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd477fe640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340039b0 0x7fcd34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd3ffff640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004800 0x7fcd34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd47fff640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480d1910 0x7fcd340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd477fe640 1 -- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480d1910 msgr2=0x7fcd340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd477fe640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480d1910 0x7fcd340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd477fe640 1 -- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004800 msgr2=0x7fcd34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd477fe640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd34004800 0x7fcd34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.446+0000 7fcd477fe640 1 -- 172.21.15.77:0/3403489838 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd34003520 con 0x7fcd340039b0 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.447+0000 7fcd477fe640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340039b0 0x7fcd34003dd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009ac0 tx=0x7fcd34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef18c029c780d188 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.447+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38039070 con 0x7fcd340039b0 2024-03-06T07:25:39.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.447+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3802c710 con 0x7fcd340039b0 2024-03-06T07:25:39.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.447+0000 7fcd46ffd640 1 -- 172.21.15.77:0/3403489838 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38009690 con 0x7fcd340039b0 2024-03-06T07:25:39.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.448+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/3403489838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340039b0 msgr2=0x7fcd34003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.448+0000 7fcd4e1c9640 1 --2- 172.21.15.77:0/3403489838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd340039b0 0x7fcd34003dd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009ac0 tx=0x7fcd34005830 comp rx=0 tx=0).stop 2024-03-06T07:25:39.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.448+0000 7fcd47fff640 1 -- 172.21.15.77:0/3403489838 reap_dead start 2024-03-06T07:25:39.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.448+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/3403489838 shutdown_connections 2024-03-06T07:25:39.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.448+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/3403489838 >> 172.21.15.77:0/3403489838 conn(0x7fcd480d3d40 msgr2=0x7fcd480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.448+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/3403489838 shutdown_connections 2024-03-06T07:25:39.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.448+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/3403489838 wait complete. 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.449+0000 7fcd4e1c9640 1 Processor -- start 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.449+0000 7fcd4e1c9640 1 -- start start 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd4e1c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340039b0 0x7fcd480ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd4e1c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480c8b60 0x7fcd4815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd4e1c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4815aa80 0x7fcd4815af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd4e1c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480b0910 con 0x7fcd4815aa80 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd4e1c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480cfc60 con 0x7fcd340039b0 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd4e1c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480b1e40 con 0x7fcd480c8b60 2024-03-06T07:25:39.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4815aa80 0x7fcd4815af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4815aa80 0x7fcd4815af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34090/0 (socket says 172.21.15.77:34090) 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 -- 172.21.15.77:0/2878216903 learned_addr learned my addr 172.21.15.77:0/2878216903 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd3ffff640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480c8b60 0x7fcd4815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd477fe640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340039b0 0x7fcd480ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 -- 172.21.15.77:0/2878216903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480c8b60 msgr2=0x7fcd4815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480c8b60 0x7fcd4815d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 -- 172.21.15.77:0/2878216903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340039b0 msgr2=0x7fcd480ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340039b0 0x7fcd480ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 -- 172.21.15.77:0/2878216903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd480c9100 con 0x7fcd4815aa80 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd47fff640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4815aa80 0x7fcd4815af00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fcd40002230 tx=0x7fcd4000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd44ff9640 1 -- 172.21.15.77:0/2878216903 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40019070 con 0x7fcd4815aa80 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd44ff9640 1 -- 172.21.15.77:0/2878216903 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4000ec10 con 0x7fcd4815aa80 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd480c93e0 con 0x7fcd4815aa80 2024-03-06T07:25:39.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd44ff9640 1 -- 172.21.15.77:0/2878216903 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40014480 con 0x7fcd4815aa80 2024-03-06T07:25:39.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.450+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd480c99a0 con 0x7fcd4815aa80 2024-03-06T07:25:39.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.451+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd480d1d70 con 0x7fcd4815aa80 2024-03-06T07:25:39.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.452+0000 7fcd44ff9640 1 -- 172.21.15.77:0/2878216903 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd4001c070 con 0x7fcd4815aa80 2024-03-06T07:25:39.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.452+0000 7fcd44ff9640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2806b4d0 0x7fcd2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.452+0000 7fcd477fe640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2806b4d0 0x7fcd2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.452+0000 7fcd44ff9640 1 -- 172.21.15.77:0/2878216903 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcd40089ba0 con 0x7fcd4815aa80 2024-03-06T07:25:39.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.453+0000 7fcd477fe640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2806b4d0 0x7fcd2806d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcd38036000 tx=0x7fcd38030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.457+0000 7fcd44ff9640 1 -- 172.21.15.77:0/2878216903 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd480d1d70 con 0x7fcd4815aa80 2024-03-06T07:25:39.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.628+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcd480d1150 con 0x7fcd4815aa80 2024-03-06T07:25:39.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.631+0000 7fcd44ff9640 1 -- 172.21.15.77:0/2878216903 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcd480d1150 con 0x7fcd4815aa80 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2806b4d0 msgr2=0x7fcd2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2806b4d0 0x7fcd2806d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcd38036000 tx=0x7fcd38030000 comp rx=0 tx=0).stop 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4815aa80 msgr2=0x7fcd4815af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4815aa80 0x7fcd4815af00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fcd40002230 tx=0x7fcd4000e530 comp rx=0 tx=0).stop 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 shutdown_connections 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd2806b4d0 0x7fcd2806d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd4815aa80 0x7fcd4815af00 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd480c8b60 0x7fcd4815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 --2- 172.21.15.77:0/2878216903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd340039b0 0x7fcd480ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 >> 172.21.15.77:0/2878216903 conn(0x7fcd480d3d40 msgr2=0x7fcd480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 shutdown_connections 2024-03-06T07:25:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.634+0000 7fcd4e1c9640 1 -- 172.21.15.77:0/2878216903 wait complete. 2024-03-06T07:25:39.646 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:39.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:39.698 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:39.741 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:39.741 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:39.741 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:39.756 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:39.757 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:25:39.757 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:39.799 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:39.799 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:39.799 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:39.814 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:39.814 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:25:39.815 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:39.856 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:39.856 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:25:39.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:39.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.954+0000 7f7cfad93640 1 Processor -- start 2024-03-06T07:25:39.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.954+0000 7f7cfad93640 1 -- start start 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.954+0000 7f7cfad93640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf40cce40 0x7f7cf415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cfad93640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf4071f40 con 0x7f7cf40cd7a0 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cfad93640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf40cd2c0 con 0x7f7cf41515e0 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cfad93640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf4070a70 con 0x7f7cf40cce40 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf9330640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7cf40cd7a0 0x7f7cf40cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59380/0 (socket says 172.21.15.77:59380) 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf9330640 1 -- 172.21.15.77:0/1921084109 learned_addr learned my addr 172.21.15.77:0/1921084109 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809654974 0 0) 0x7f7cf4071f40 con 0x7f7cf40cd7a0 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf40cce40 0x7f7cf415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce4003520 con 0x7f7cf40cd7a0 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3592606373 0 0) 0x7f7ce4003520 con 0x7f7cf40cd7a0 2024-03-06T07:25:39.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf40cce40 msgr2=0x7f7cf415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf40cce40 0x7f7cf415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 >> v1:172.21.15.158:6789/0 conn(0x7f7cf41515e0 legacy=0x7f7cf4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4070d50 con 0x7f7cf40cd7a0 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f7cec0019f0 con 0x7f7cf40cd7a0 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 >> v1:172.21.15.77:6789/0 conn(0x7f7cf40cd7a0 legacy=0x7f7cf40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf40cce40 0x7f7ce40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ce4004010 0x7f7ce4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce4004d40 0x7f7ce40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf4070a70 con 0x7f7ce4004d40 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce400bb80 con 0x7f7ce4004010 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce400bd80 con 0x7f7cf40cce40 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf9330640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce4004d40 0x7f7ce40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.955+0000 7f7cf8b2f640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf40cce40 0x7f7ce40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.956+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ce4004010 0x7f7ce4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.956+0000 7f7cf9330640 1 -- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf40cce40 msgr2=0x7f7ce40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.956+0000 7f7cf9330640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf40cce40 0x7f7ce40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.956+0000 7f7cf9330640 1 -- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ce4004010 msgr2=0x7f7ce4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.956+0000 7f7cf9330640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7ce4004010 0x7f7ce4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.956+0000 7f7cf9330640 1 -- 172.21.15.77:0/1921084109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce4003520 con 0x7f7ce4004d40 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.956+0000 7f7cf9330640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce4004d40 0x7f7ce40051c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00a860 tx=0x7f7ce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5b9a66287f2df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec00a480 con 0x7f7ce4004d40 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec034630 con 0x7f7ce4004d40 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cf37fe640 1 -- 172.21.15.77:0/1921084109 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec034aa0 con 0x7f7ce4004d40 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cfad93640 1 -- 172.21.15.77:0/1921084109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce4004d40 msgr2=0x7f7ce40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cfad93640 1 --2- 172.21.15.77:0/1921084109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce4004d40 0x7f7ce40051c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00a860 tx=0x7f7ce4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cf9330640 1 -- 172.21.15.77:0/1921084109 reap_dead start 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cfad93640 1 -- 172.21.15.77:0/1921084109 shutdown_connections 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cfad93640 1 -- 172.21.15.77:0/1921084109 >> 172.21.15.77:0/1921084109 conn(0x7f7cf40cf8a0 msgr2=0x7f7cf40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cfad93640 1 -- 172.21.15.77:0/1921084109 shutdown_connections 2024-03-06T07:25:39.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.957+0000 7f7cfad93640 1 -- 172.21.15.77:0/1921084109 wait complete. 2024-03-06T07:25:39.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.958+0000 7f7cfad93640 1 Processor -- start 2024-03-06T07:25:39.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.958+0000 7f7cfad93640 1 -- start start 2024-03-06T07:25:39.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cfad93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4141c50 0x7f7cf4154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cfad93640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cf4151a20 0x7f7cf4145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cfad93640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf4145550 0x7f7cf40715f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cfad93640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf415b430 con 0x7f7cf4141c50 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cfad93640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf4071310 con 0x7f7cf4151a20 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cfad93640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf415b5b0 con 0x7f7cf4145550 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cf8b2f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4141c50 0x7f7cf4154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cf8b2f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4141c50 0x7f7cf4154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34102/0 (socket says 172.21.15.77:34102) 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.959+0000 7f7cf8b2f640 1 -- 172.21.15.77:0/1788950068 learned_addr learned my addr 172.21.15.77:0/1788950068 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf8b2f640 1 -- 172.21.15.77:0/1788950068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf4145550 msgr2=0x7f7cf40715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf8b2f640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf4145550 0x7f7cf40715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf8b2f640 1 -- 172.21.15.77:0/1788950068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cf4151a20 msgr2=0x7f7cf4145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf8b2f640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cf4151a20 0x7f7cf4145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf8b2f640 1 -- 172.21.15.77:0/1788950068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4152200 con 0x7f7cf4141c50 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf8b2f640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4141c50 0x7f7cf4154740 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800f7a0 tx=0x7f7ce800fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1788950068 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80050d0 con 0x7f7cf4141c50 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1788950068 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce8018070 con 0x7f7cf4141c50 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf41524e0 con 0x7f7cf4141c50 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1788950068 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800d730 con 0x7f7cf4141c50 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.960+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf4142740 con 0x7f7cf4141c50 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.961+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf4145970 con 0x7f7cf4141c50 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.961+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1788950068 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ce800d8d0 con 0x7f7cf4141c50 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.962+0000 7f7cf17fa640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7cd806b500 0x7f7cd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.962+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1788950068 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7ce8089cf0 con 0x7f7cf4141c50 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.962+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7cd806b500 0x7f7cd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.962+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7cd806b500 0x7f7cd806d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0006a90 tx=0x7f7ce0006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:39.966+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1788950068 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ce808e070 con 0x7f7cf4141c50 2024-03-06T07:25:40.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.175+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf4068840 con 0x7f7cf4141c50 2024-03-06T07:25:40.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.175+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1788950068 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f7cf40cd2c0 con 0x7f7cf4141c50 2024-03-06T07:25:40.177 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:40.177 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/4068642965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":4068642965},{"type":"v1","addr":"172.21.15.158:6834","nonce":4068642965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2657051841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":2657051841},{"type":"v1","addr":"172.21.15.158:6835","nonce":2657051841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/1662185270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1662185270},{"type":"v1","addr":"172.21.15.77:6836","nonce":1662185270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6837,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1751019631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":1751019631},{"type":"v1","addr":"172.21.15.77:6837","nonce":1751019631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[]} 2024-03-06T07:25:40.178 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 35 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7cd806b500 msgr2=0x7f7cd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7cd806b500 0x7f7cd806d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0006a90 tx=0x7f7ce0006a20 comp rx=0 tx=0).stop 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4141c50 msgr2=0x7f7cf4154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4141c50 0x7f7cf4154740 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800f7a0 tx=0x7f7ce800fc60 comp rx=0 tx=0).stop 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 shutdown_connections 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7cf4145550 0x7f7cf40715f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7cf4151a20 0x7f7cf4145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7cd806b500 0x7f7cd806d9c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 --2- 172.21.15.77:0/1788950068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4141c50 0x7f7cf4154740 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 >> 172.21.15.77:0/1788950068 conn(0x7f7cf40cf8a0 msgr2=0x7f7cf40cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 shutdown_connections 2024-03-06T07:25:40.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.178+0000 7f7cfad93640 1 -- 172.21.15.77:0/1788950068 wait complete. 2024-03-06T07:25:40.191 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:40.191 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:40.191 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:40.191 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:40.192 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:40.192 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:40.192 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:40.192 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:40.192 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:40.192 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:40.192 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:40.192 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:40.193 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:40.193 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:40.193 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:40.193 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:40.200 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:40.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:40.202 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:40.202 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:40.204 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:40.204 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:40.205 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:40.205 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:40.243 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:40.243 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:40.244 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:40.245 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:40.247 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:40.247 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:40.247 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:40.248 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:40.248 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:40.320 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:25:40.321 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:25:40.330 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:25:40.330 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:25:40.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.353+0000 7f8b56dac640 1 Processor -- start 2024-03-06T07:25:40.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.353+0000 7f8b56dac640 1 -- start start 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.354+0000 7f8b56dac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b50162970 0x7f8b500d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.354+0000 7f8b56dac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b50161cf0 0x7f8b50162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.354+0000 7f8b56dac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b500d22c0 con 0x7f8b500b4460 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.354+0000 7f8b56dac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b50161b00 con 0x7f8b50161cf0 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.354+0000 7f8b56dac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500cecc0 con 0x7f8b50162970 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b4ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b50162970 0x7f8b500d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b4ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b50162970 0x7f8b500d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:54112/0 (socket says 172.21.15.77:54112) 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b54b48640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8b500b4460 0x7f8b500b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:33938/0 (socket says 172.21.15.77:33938) 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b4ffff640 1 -- 172.21.15.77:0/4199231634 learned_addr learned my addr 172.21.15.77:0/4199231634 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b55349640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8b50161cf0 0x7f8b50162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b4ffff640 1 -- 172.21.15.77:0/4199231634 >> v2:172.21.15.158:3300/0 conn(0x7f8b50161cf0 msgr2=0x7f8b50162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b4ffff640 1 --2- 172.21.15.77:0/4199231634 >> v2:172.21.15.158:3300/0 conn(0x7f8b50161cf0 0x7f8b50162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b4ffff640 1 -- 172.21.15.77:0/4199231634 >> v1:172.21.15.158:6789/0 conn(0x7f8b500b4460 legacy=0x7f8b500b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.355+0000 7f8b4ffff640 1 -- 172.21.15.77:0/4199231634 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b500d25a0 con 0x7f8b50162970 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b4ffff640 1 --2- 172.21.15.77:0/4199231634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b50162970 0x7f8b500d4800 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c009870 tx=0x7f8b3c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=547e59b91470f7be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b4f7fe640 1 -- 172.21.15.77:0/4199231634 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c034070 con 0x7f8b50162970 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b4f7fe640 1 -- 172.21.15.77:0/4199231634 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c030840 con 0x7f8b50162970 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b4f7fe640 1 -- 172.21.15.77:0/4199231634 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c030c10 con 0x7f8b50162970 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b56dac640 1 -- 172.21.15.77:0/4199231634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b50162970 msgr2=0x7f8b500d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b56dac640 1 --2- 172.21.15.77:0/4199231634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b50162970 0x7f8b500d4800 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c009870 tx=0x7f8b3c02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b56dac640 1 -- 172.21.15.77:0/4199231634 shutdown_connections 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b56dac640 1 --2- 172.21.15.77:0/4199231634 >> v2:172.21.15.158:3300/0 conn(0x7f8b50161cf0 0x7f8b50162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b56dac640 1 --2- 172.21.15.77:0/4199231634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b50162970 0x7f8b500d4800 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b56dac640 1 -- 172.21.15.77:0/4199231634 >> 172.21.15.77:0/4199231634 conn(0x7f8b500d3ce0 msgr2=0x7f8b500d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b56dac640 1 -- 172.21.15.77:0/4199231634 shutdown_connections 2024-03-06T07:25:40.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.356+0000 7f8b56dac640 1 -- 172.21.15.77:0/4199231634 wait complete. 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.358+0000 7f8b56dac640 1 Processor -- start 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.358+0000 7f8b56dac640 1 -- start start 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b56dac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b500b4460 0x7f8b500c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b56dac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50161cf0 0x7f8b50155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b56dac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b50162970 0x7f8b500b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b56dac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500ce950 con 0x7f8b50161cf0 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b56dac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500d2b60 con 0x7f8b50162970 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b56dac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500b4840 con 0x7f8b500b4460 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50161cf0 0x7f8b50155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50161cf0 0x7f8b50155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34144/0 (socket says 172.21.15.77:34144) 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 -- 172.21.15.77:0/3678386042 learned_addr learned my addr 172.21.15.77:0/3678386042 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b54b48640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b500b4460 0x7f8b500c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b55349640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b50162970 0x7f8b500b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 -- 172.21.15.77:0/3678386042 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b500b4460 msgr2=0x7f8b500c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b500b4460 0x7f8b500c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 -- 172.21.15.77:0/3678386042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b50162970 msgr2=0x7f8b500b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b50162970 0x7f8b500b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 -- 172.21.15.77:0/3678386042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b500b7e70 con 0x7f8b50161cf0 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4ffff640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50161cf0 0x7f8b50155fc0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c02b9c0 tx=0x7f8b3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4d7fa640 1 -- 172.21.15.77:0/3678386042 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c034040 con 0x7f8b50161cf0 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.359+0000 7f8b4d7fa640 1 -- 172.21.15.77:0/3678386042 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c030640 con 0x7f8b50161cf0 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.360+0000 7f8b4d7fa640 1 -- 172.21.15.77:0/3678386042 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c0423f0 con 0x7f8b50161cf0 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.360+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b500b8150 con 0x7f8b50161cf0 2024-03-06T07:25:40.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.360+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b500b8710 con 0x7f8b50161cf0 2024-03-06T07:25:40.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.360+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b500d20d0 con 0x7f8b50161cf0 2024-03-06T07:25:40.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.361+0000 7f8b4d7fa640 1 -- 172.21.15.77:0/3678386042 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b3c037070 con 0x7f8b50161cf0 2024-03-06T07:25:40.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.361+0000 7f8b4d7fa640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8b3406b4d0 0x7f8b3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.361+0000 7f8b54b48640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8b3406b4d0 0x7f8b3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.361+0000 7f8b4d7fa640 1 -- 172.21.15.77:0/3678386042 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8b3c0abfc0 con 0x7f8b50161cf0 2024-03-06T07:25:40.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.362+0000 7f8b54b48640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8b3406b4d0 0x7f8b3406d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8b44000f60 tx=0x7f8b44000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.366+0000 7f8b4d7fa640 1 -- 172.21.15.77:0/3678386042 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b3c07ac10 con 0x7f8b50161cf0 2024-03-06T07:25:40.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:40.549+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8b5004f330 con 0x7f8b50161cf0 2024-03-06T07:25:41.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.342+0000 7f8b4d7fa640 1 -- 172.21.15.77:0/3678386042 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v74) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8b3c07ecb0 con 0x7f8b50161cf0 2024-03-06T07:25:41.344 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:25:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8b3406b4d0 msgr2=0x7f8b3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8b3406b4d0 0x7f8b3406d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8b44000f60 tx=0x7f8b44000ed0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50161cf0 msgr2=0x7f8b50155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50161cf0 0x7f8b50155fc0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c02b9c0 tx=0x7f8b3c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 shutdown_connections 2024-03-06T07:25:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8b3406b4d0 0x7f8b3406d990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50161cf0 0x7f8b50155fc0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8b50162970 0x7f8b500b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 --2- 172.21.15.77:0/3678386042 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8b500b4460 0x7f8b500c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 >> 172.21.15.77:0/3678386042 conn(0x7f8b500d3ce0 msgr2=0x7f8b500d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 shutdown_connections 2024-03-06T07:25:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.346+0000 7f8b56dac640 1 -- 172.21.15.77:0/3678386042 wait complete. 2024-03-06T07:25:41.364 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.467+0000 7f3d1d356640 1 Processor -- start 2024-03-06T07:25:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.467+0000 7f3d1d356640 1 -- start start 2024-03-06T07:25:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.468+0000 7f3d1d356640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3d18072730 0x7f3d1815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.468+0000 7f3d1d356640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d18162030 0x7f3d18162480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.468+0000 7f3d1d356640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1815eb40 con 0x7f3d18158600 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.468+0000 7f3d1d356640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d18161ba0 con 0x7f3d18162030 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.468+0000 7f3d1d356640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d18072be0 con 0x7f3d18072730 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.468+0000 7f3d177fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d18162030 0x7f3d18162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.468+0000 7f3d16ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3d18158600 0x7f3d1806d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59402/0 (socket says 172.21.15.77:59402) 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.468+0000 7f3d16ffd640 1 -- 172.21.15.77:0/1184367166 learned_addr learned my addr 172.21.15.77:0/1184367166 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.469+0000 7f3d177fe640 1 -- 172.21.15.77:0/1184367166 >> v2:172.21.15.158:3300/0 conn(0x7f3d18072730 msgr2=0x7f3d1815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.469+0000 7f3d167fc640 1 --2- 172.21.15.77:0/1184367166 >> v2:172.21.15.158:3300/0 conn(0x7f3d18072730 0x7f3d1815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.469+0000 7f3d177fe640 1 --2- 172.21.15.77:0/1184367166 >> v2:172.21.15.158:3300/0 conn(0x7f3d18072730 0x7f3d1815e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.469+0000 7f3d15ffb640 1 -- 172.21.15.77:0/1184367166 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493278747 0 0) 0x7f3d1815eb40 con 0x7f3d18158600 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.469+0000 7f3d177fe640 1 -- 172.21.15.77:0/1184367166 >> v1:172.21.15.77:6789/0 conn(0x7f3d18158600 legacy=0x7f3d1806d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.469+0000 7f3d177fe640 1 -- 172.21.15.77:0/1184367166 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1815ee20 con 0x7f3d18162030 2024-03-06T07:25:41.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.469+0000 7f3d177fe640 1 --2- 172.21.15.77:0/1184367166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d18162480 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f3d18161d20 tx=0x7f3d0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f8147fa833503ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d15ffb640 1 -- 172.21.15.77:0/1184367166 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d080315a0 con 0x7f3d18162030 2024-03-06T07:25:41.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d15ffb640 1 -- 172.21.15.77:0/1184367166 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08037070 con 0x7f3d18162030 2024-03-06T07:25:41.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d15ffb640 1 -- 172.21.15.77:0/1184367166 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08031960 con 0x7f3d18162030 2024-03-06T07:25:41.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d1d356640 1 -- 172.21.15.77:0/1184367166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 msgr2=0x7f3d18162480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d1d356640 1 --2- 172.21.15.77:0/1184367166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d18162480 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f3d18161d20 tx=0x7f3d0802b840 comp rx=0 tx=0).stop 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d1d356640 1 -- 172.21.15.77:0/1184367166 shutdown_connections 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d1d356640 1 --2- 172.21.15.77:0/1184367166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d18162480 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d1d356640 1 --2- 172.21.15.77:0/1184367166 >> v2:172.21.15.158:3300/0 conn(0x7f3d18072730 0x7f3d1815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d1d356640 1 -- 172.21.15.77:0/1184367166 >> 172.21.15.77:0/1184367166 conn(0x7f3d1806c0c0 msgr2=0x7f3d1806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d1d356640 1 -- 172.21.15.77:0/1184367166 shutdown_connections 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.470+0000 7f3d1d356640 1 -- 172.21.15.77:0/1184367166 wait complete. 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.471+0000 7f3d1d356640 1 Processor -- start 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.471+0000 7f3d1d356640 1 -- start start 2024-03-06T07:25:41.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d1d356640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d18072730 0x7f3d1815d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d1d356640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d18158600 0x7f3d18145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d1d356640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d180b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d1d356640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1806e080 con 0x7f3d18162030 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d1d356640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1815e8e0 con 0x7f3d18072730 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d1d356640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1815d8e0 con 0x7f3d18158600 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d177fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d180b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d177fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d180b3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34154/0 (socket says 172.21.15.77:34154) 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d177fe640 1 -- 172.21.15.77:0/3856520804 learned_addr learned my addr 172.21.15.77:0/3856520804 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.472+0000 7f3d16ffd640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d18072730 0x7f3d1815d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d167fc640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d18158600 0x7f3d18145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d177fe640 1 -- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d18158600 msgr2=0x7f3d18145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d177fe640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d18158600 0x7f3d18145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d177fe640 1 -- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d18072730 msgr2=0x7f3d1815d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d177fe640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d18072730 0x7f3d1815d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d177fe640 1 -- 172.21.15.77:0/3856520804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d18146900 con 0x7f3d18162030 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d16ffd640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d18072730 0x7f3d1815d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d167fc640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d18158600 0x7f3d18145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d177fe640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d180b3000 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f3d08034040 tx=0x7f3d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3cff7fe640 1 -- 172.21.15.77:0/3856520804 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08032630 con 0x7f3d18162030 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3cff7fe640 1 -- 172.21.15.77:0/3856520804 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d080327d0 con 0x7f3d18162030 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d18146be0 con 0x7f3d18162030 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3cff7fe640 1 -- 172.21.15.77:0/3856520804 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08032bc0 con 0x7f3d18162030 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.473+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1815d5d0 con 0x7f3d18162030 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.474+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1804f330 con 0x7f3d18162030 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.474+0000 7f3cff7fe640 1 -- 172.21.15.77:0/3856520804 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d0804c020 con 0x7f3d18162030 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.475+0000 7f3cff7fe640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d0406b4d0 0x7f3d0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.475+0000 7f3cff7fe640 1 -- 172.21.15.77:0/3856520804 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3d0802f020 con 0x7f3d18162030 2024-03-06T07:25:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.475+0000 7f3d16ffd640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d0406b4d0 0x7f3d0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.475+0000 7f3d16ffd640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d0406b4d0 0x7f3d0406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3d100012e0 tx=0x7f3d10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.479+0000 7f3cff7fe640 1 -- 172.21.15.77:0/3856520804 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d08040070 con 0x7f3d18162030 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.715+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3d18068490 con 0x7f3d18162030 2024-03-06T07:25:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.716+0000 7f3cff7fe640 1 -- 172.21.15.77:0/3856520804 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3d08039310 con 0x7f3d18162030 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.718 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.719 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.720 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:41.721 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.723 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:41.724 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:25:41.725 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:25:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:25:41.727 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.728 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:25:41.729 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:41.730 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:25:41.730 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.730 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:41.730 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:25:41.730 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:25:41.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.717+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d0406b4d0 msgr2=0x7f3d0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.717+0000 7f3d1d356640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d0406b4d0 0x7f3d0406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3d100012e0 tx=0x7f3d10009040 comp rx=0 tx=0).stop 2024-03-06T07:25:41.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 msgr2=0x7f3d180b3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d180b3000 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f3d08034040 tx=0x7f3d080046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 shutdown_connections 2024-03-06T07:25:41.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d0406b4d0 0x7f3d0406d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d18162030 0x7f3d180b3000 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d18158600 0x7f3d18145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 --2- 172.21.15.77:0/3856520804 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d18072730 0x7f3d1815d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 >> 172.21.15.77:0/3856520804 conn(0x7f3d1806c0c0 msgr2=0x7f3d180b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 shutdown_connections 2024-03-06T07:25:41.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.718+0000 7f3d1d356640 1 -- 172.21.15.77:0/3856520804 wait complete. 2024-03-06T07:25:41.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:41.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.831+0000 7f133c925640 1 Processor -- start 2024-03-06T07:25:41.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.831+0000 7f133c925640 1 -- start start 2024-03-06T07:25:41.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133c925640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13381515e0 0x7f13380cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133c925640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13380cd7a0 0x7f13380cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133c925640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1338071f40 con 0x7f133815b430 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133c925640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133815b8e0 con 0x7f13381515e0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133c925640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1338070a70 con 0x7f13380cd7a0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13381515e0 0x7f13380cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13381515e0 0x7f13380cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34164/0 (socket says 172.21.15.77:34164) 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133659c640 1 -- 172.21.15.77:0/827797098 learned_addr learned my addr 172.21.15.77:0/827797098 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f1336d9d640 1 --2- 172.21.15.77:0/827797098 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13380cd7a0 0x7f13380cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133659c640 1 -- 172.21.15.77:0/827797098 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13380cd7a0 msgr2=0x7f13380cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133659c640 1 --2- 172.21.15.77:0/827797098 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13380cd7a0 0x7f13380cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133659c640 1 -- 172.21.15.77:0/827797098 >> v1:172.21.15.158:6789/0 conn(0x7f133815b430 legacy=0x7f1338071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.832+0000 7f133659c640 1 -- 172.21.15.77:0/827797098 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338070d50 con 0x7f13381515e0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.833+0000 7f133659c640 1 --2- 172.21.15.77:0/827797098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13381515e0 0x7f13380cce40 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f132800aa10 tx=0x7f132802ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f6d80995e7a560 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.833+0000 7f1335d9b640 1 -- 172.21.15.77:0/827797098 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132803a070 con 0x7f13381515e0 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.833+0000 7f1335d9b640 1 -- 172.21.15.77:0/827797098 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132802dc10 con 0x7f13381515e0 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.833+0000 7f1335d9b640 1 -- 172.21.15.77:0/827797098 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800b040 con 0x7f13381515e0 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.834+0000 7f133c925640 1 -- 172.21.15.77:0/827797098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13381515e0 msgr2=0x7f13380cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.834+0000 7f133c925640 1 --2- 172.21.15.77:0/827797098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13381515e0 0x7f13380cce40 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f132800aa10 tx=0x7f132802ce60 comp rx=0 tx=0).stop 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.834+0000 7f133c925640 1 -- 172.21.15.77:0/827797098 shutdown_connections 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.834+0000 7f133c925640 1 --2- 172.21.15.77:0/827797098 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13380cd7a0 0x7f13380cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.834+0000 7f133c925640 1 --2- 172.21.15.77:0/827797098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13381515e0 0x7f13380cce40 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.834+0000 7f133c925640 1 -- 172.21.15.77:0/827797098 >> 172.21.15.77:0/827797098 conn(0x7f13380cf8a0 msgr2=0x7f13380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.834+0000 7f133c925640 1 -- 172.21.15.77:0/827797098 shutdown_connections 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.834+0000 7f133c925640 1 -- 172.21.15.77:0/827797098 wait complete. 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.835+0000 7f133c925640 1 Processor -- start 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.835+0000 7f133c925640 1 -- start start 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f133c925640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380cd7a0 0x7f133814c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f133c925640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13381515e0 0x7f133815d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f133c925640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133815b430 0x7f13381f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f133c925640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13380ce0f0 con 0x7f133815b430 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f133c925640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1338071310 con 0x7f13380cd7a0 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f133c925640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13380cc3a0 con 0x7f13381515e0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133815b430 0x7f13381f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133815b430 0x7f13381f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34170/0 (socket says 172.21.15.77:34170) 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 -- 172.21.15.77:0/790121481 learned_addr learned my addr 172.21.15.77:0/790121481 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f133659c640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380cd7a0 0x7f133814c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f132ffff640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13381515e0 0x7f133815d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 -- 172.21.15.77:0/790121481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13381515e0 msgr2=0x7f133815d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13381515e0 0x7f133815d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 -- 172.21.15.77:0/790121481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380cd7a0 msgr2=0x7f133814c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380cd7a0 0x7f133814c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 -- 172.21.15.77:0/790121481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133815c9a0 con 0x7f133815b430 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f1336d9d640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133815b430 0x7f13381f7580 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1330007e40 tx=0x7f133000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.836+0000 7f132effd640 1 -- 172.21.15.77:0/790121481 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330014620 con 0x7f133815b430 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.837+0000 7f132effd640 1 -- 172.21.15.77:0/790121481 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1330019070 con 0x7f133815b430 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.837+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133815cc80 con 0x7f133815b430 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.837+0000 7f132effd640 1 -- 172.21.15.77:0/790121481 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330014a20 con 0x7f133815b430 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.837+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13381f8270 con 0x7f133815b430 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.838+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1338068840 con 0x7f133815b430 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.838+0000 7f132effd640 1 -- 172.21.15.77:0/790121481 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1330014bc0 con 0x7f133815b430 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.838+0000 7f132effd640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.839+0000 7f133659c640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.839+0000 7f132effd640 1 -- 172.21.15.77:0/790121481 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f133008a590 con 0x7f133815b430 2024-03-06T07:25:41.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.839+0000 7f133659c640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f131806b500 0x7f131806d9c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1328037000 tx=0x7f1328031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:41.843+0000 7f132effd640 1 -- 172.21.15.77:0/790121481 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13300591a0 con 0x7f133815b430 2024-03-06T07:25:42.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.015+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f133804f330 con 0x7f133815b430 2024-03-06T07:25:42.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.016+0000 7f132effd640 1 -- 172.21.15.77:0/790121481 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+12880 (secure 0 0 0) 0x7f133005d240 con 0x7f133815b430 2024-03-06T07:25:42.018 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:42.018 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":76,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:41.351569+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f131806b500 msgr2=0x7f131806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f131806b500 0x7f131806d9c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1328037000 tx=0x7f1328031000 comp rx=0 tx=0).stop 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133815b430 msgr2=0x7f13381f7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133815b430 0x7f13381f7580 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1330007e40 tx=0x7f133000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 shutdown_connections 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133815b430 0x7f13381f7580 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13381515e0 0x7f133815d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 --2- 172.21.15.77:0/790121481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13380cd7a0 0x7f133814c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 >> 172.21.15.77:0/790121481 conn(0x7f13380cf8a0 msgr2=0x7f1338072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 shutdown_connections 2024-03-06T07:25:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.018+0000 7f133c925640 1 -- 172.21.15.77:0/790121481 wait complete. 2024-03-06T07:25:42.031 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:42.032 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.132+0000 7feb454fb640 1 Processor -- start 2024-03-06T07:25:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.132+0000 7feb454fb640 1 -- start start 2024-03-06T07:25:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb454fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 0x7feb400d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb454fb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feb400d1910 0x7feb400d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb454fb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400d2740 con 0x7feb400d22b0 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb454fb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb401618b0 con 0x7feb400d1910 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb454fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb400cf400 con 0x7feb401614d0 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb3f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7feb400d22b0 0x7feb400d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59404/0 (socket says 172.21.15.77:59404) 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb3f7fe640 1 -- 172.21.15.77:0/364186494 learned_addr learned my addr 172.21.15.77:0/364186494 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb3effd640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 0x7feb400d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb367fc640 1 --2- 172.21.15.77:0/364186494 >> v2:172.21.15.158:3300/0 conn(0x7feb400d1910 0x7feb400d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846752365 0 0) 0x7feb38001490 con 0x7feb400d22b0 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.133+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb2c003520 con 0x7feb400d22b0 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784175071 0 0) 0x7feb2c003520 con 0x7feb400d22b0 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 msgr2=0x7feb400d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 0x7feb400d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 >> v2:172.21.15.158:3300/0 conn(0x7feb400d1910 msgr2=0x7feb400d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 --2- 172.21.15.77:0/364186494 >> v2:172.21.15.158:3300/0 conn(0x7feb400d1910 0x7feb400d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb400cf6a0 con 0x7feb400d22b0 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7feb38001bc0 con 0x7feb400d22b0 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 >> v1:172.21.15.77:6789/0 conn(0x7feb400d22b0 legacy=0x7feb400d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2c0039b0 0x7feb2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c004800 0x7feb2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 0x7feb2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb401618b0 con 0x7feb2c004800 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400cf400 con 0x7feb2c0039b0 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c00b9c0 con 0x7feb401614d0 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb367fc640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c004800 0x7feb2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3effd640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2c0039b0 0x7feb2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3f7fe640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 0x7feb2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb367fc640 1 -- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 msgr2=0x7feb2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb367fc640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 0x7feb2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb367fc640 1 -- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2c0039b0 msgr2=0x7feb2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb367fc640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb2c0039b0 0x7feb2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb367fc640 1 -- 172.21.15.77:0/364186494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c003520 con 0x7feb2c004800 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.134+0000 7feb3f7fe640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb401614d0 0x7feb2c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.135+0000 7feb367fc640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c004800 0x7feb2c004c80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7feb28009ac0 tx=0x7feb2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bc00a688ae0b95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.135+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28039070 con 0x7feb2c004800 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.135+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2802c710 con 0x7feb2c004800 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.135+0000 7feb3e7fc640 1 -- 172.21.15.77:0/364186494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28009690 con 0x7feb2c004800 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.136+0000 7feb454fb640 1 -- 172.21.15.77:0/364186494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c004800 msgr2=0x7feb2c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.136+0000 7feb454fb640 1 --2- 172.21.15.77:0/364186494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c004800 0x7feb2c004c80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7feb28009ac0 tx=0x7feb2c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.136+0000 7feb3f7fe640 1 -- 172.21.15.77:0/364186494 reap_dead start 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.136+0000 7feb454fb640 1 -- 172.21.15.77:0/364186494 shutdown_connections 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.136+0000 7feb454fb640 1 -- 172.21.15.77:0/364186494 >> 172.21.15.77:0/364186494 conn(0x7feb400d3d40 msgr2=0x7feb400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.136+0000 7feb454fb640 1 -- 172.21.15.77:0/364186494 shutdown_connections 2024-03-06T07:25:42.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.136+0000 7feb454fb640 1 -- 172.21.15.77:0/364186494 wait complete. 2024-03-06T07:25:42.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.137+0000 7feb454fb640 1 Processor -- start 2024-03-06T07:25:42.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.137+0000 7feb454fb640 1 -- start start 2024-03-06T07:25:42.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb454fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb2c004800 0x7feb4014bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb454fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb400c7870 0x7feb401f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb454fb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006bde0 0x7feb4006c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb454fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400b0910 con 0x7feb400c7870 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb454fb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400cfc60 con 0x7feb4006bde0 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb454fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb400b1e40 con 0x7feb2c004800 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb400c7870 0x7feb401f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb400c7870 0x7feb401f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34190/0 (socket says 172.21.15.77:34190) 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 -- 172.21.15.77:0/1345911522 learned_addr learned my addr 172.21.15.77:0/1345911522 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb3f7fe640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006bde0 0x7feb4006c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb3effd640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb2c004800 0x7feb4014bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 -- 172.21.15.77:0/1345911522 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb2c004800 msgr2=0x7feb4014bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb2c004800 0x7feb4014bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 -- 172.21.15.77:0/1345911522 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006bde0 msgr2=0x7feb4006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006bde0 0x7feb4006c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 -- 172.21.15.77:0/1345911522 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb400ca7a0 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb367fc640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb400c7870 0x7feb401f7f80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7feb28037040 tx=0x7feb28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb37fff640 1 -- 172.21.15.77:0/1345911522 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28039050 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb37fff640 1 -- 172.21.15.77:0/1345911522 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb28045070 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb400ca4c0 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb37fff640 1 -- 172.21.15.77:0/1345911522 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28003a50 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.138+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb40157720 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.140+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb40068ff0 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.140+0000 7feb37fff640 1 -- 172.21.15.77:0/1345911522 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb28003bf0 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.140+0000 7feb37fff640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2006b4d0 0x7feb2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.140+0000 7feb37fff640 1 -- 172.21.15.77:0/1345911522 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb280abac0 con 0x7feb400c7870 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.140+0000 7feb3effd640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2006b4d0 0x7feb2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.141+0000 7feb3effd640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2006b4d0 0x7feb2006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb30009a20 tx=0x7feb30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.144+0000 7feb37fff640 1 -- 172.21.15.77:0/1345911522 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb2804e020 con 0x7feb400c7870 2024-03-06T07:25:42.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.315+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb40157210 con 0x7feb400c7870 2024-03-06T07:25:42.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.315+0000 7feb37fff640 1 -- 172.21.15.77:0/1345911522 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb2807adb0 con 0x7feb400c7870 2024-03-06T07:25:42.317 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:25:42.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2006b4d0 msgr2=0x7feb2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2006b4d0 0x7feb2006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb30009a20 tx=0x7feb30008040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb400c7870 msgr2=0x7feb401f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb400c7870 0x7feb401f7f80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7feb28037040 tx=0x7feb28004730 comp rx=0 tx=0).stop 2024-03-06T07:25:42.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 shutdown_connections 2024-03-06T07:25:42.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb2006b4d0 0x7feb2006d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb400c7870 0x7feb401f7f80 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4006bde0 0x7feb4006c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 --2- 172.21.15.77:0/1345911522 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb2c004800 0x7feb4014bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 >> 172.21.15.77:0/1345911522 conn(0x7feb400d3d40 msgr2=0x7feb400b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 shutdown_connections 2024-03-06T07:25:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.317+0000 7feb454fb640 1 -- 172.21.15.77:0/1345911522 wait complete. 2024-03-06T07:25:42.330 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:42.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.430+0000 7f9f1cb72640 1 Processor -- start 2024-03-06T07:25:42.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.430+0000 7f9f1cb72640 1 -- start start 2024-03-06T07:25:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1cb72640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f1806da40 0x7f9f180cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1cb72640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180cc980 con 0x7f9f181424f0 2024-03-06T07:25:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1cb72640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180ccc60 con 0x7f9f180cd7a0 2024-03-06T07:25:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1cb72640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180cd220 con 0x7f9f1806da40 2024-03-06T07:25:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9f181424f0 0x7f9f1806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59418/0 (socket says 172.21.15.77:59418) 2024-03-06T07:25:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1659c640 1 -- 172.21.15.77:0/2313339578 learned_addr learned my addr 172.21.15.77:0/2313339578 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391580161 0 0) 0x7f9f180cc980 con 0x7f9f181424f0 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f00003520 con 0x7f9f181424f0 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f15d9b640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f1806da40 0x7f9f180cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659664233 0 0) 0x7f9f00003520 con 0x7f9f181424f0 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f1806da40 msgr2=0x7f9f180cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1559a640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f1806da40 0x7f9f180cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.431+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 >> v1:172.21.15.158:6789/0 conn(0x7f9f180cd7a0 legacy=0x7f9f18071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180ccf40 con 0x7f9f181424f0 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f9f08001670 con 0x7f9f181424f0 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 >> v1:172.21.15.77:6789/0 conn(0x7f9f181424f0 legacy=0x7f9f1806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f000039e0 0x7f9f00003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f1806da40 0x7f9f000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180cd220 con 0x7f9f00004800 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0000ba70 con 0x7f9f000039e0 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0000bc70 con 0x7f9f1806da40 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f15d9b640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f1659c640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f000039e0 0x7f9f00003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f16d9d640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f1806da40 0x7f9f000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f15d9b640 1 -- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f1806da40 msgr2=0x7f9f000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f15d9b640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f1806da40 0x7f9f000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f15d9b640 1 -- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f000039e0 msgr2=0x7f9f00003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f15d9b640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f000039e0 0x7f9f00003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.432+0000 7f9f15d9b640 1 -- 172.21.15.77:0/2313339578 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f00003520 con 0x7f9f00004800 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f15d9b640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f00004c80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009a90 tx=0x7f9f00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd002e9639e74a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c039070 con 0x7f9f00004800 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c02c680 con 0x7f9f00004800 2024-03-06T07:25:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1559a640 1 -- 172.21.15.77:0/2313339578 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c009690 con 0x7f9f00004800 2024-03-06T07:25:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1cb72640 1 -- 172.21.15.77:0/2313339578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 msgr2=0x7f9f00004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1cb72640 1 --2- 172.21.15.77:0/2313339578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f00004c80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009a90 tx=0x7f9f00005830 comp rx=0 tx=0).stop 2024-03-06T07:25:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f16d9d640 1 -- 172.21.15.77:0/2313339578 reap_dead start 2024-03-06T07:25:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1cb72640 1 -- 172.21.15.77:0/2313339578 shutdown_connections 2024-03-06T07:25:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1cb72640 1 -- 172.21.15.77:0/2313339578 >> 172.21.15.77:0/2313339578 conn(0x7f9f180cf8a0 msgr2=0x7f9f180cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1cb72640 1 -- 172.21.15.77:0/2313339578 shutdown_connections 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.433+0000 7f9f1cb72640 1 -- 172.21.15.77:0/2313339578 wait complete. 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.435+0000 7f9f1cb72640 1 Processor -- start 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.435+0000 7f9f1cb72640 1 -- start start 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.435+0000 7f9f1cb72640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f1813f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.435+0000 7f9f1cb72640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f18156d10 0x7f9f18159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.435+0000 7f9f1cb72640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f18159b70 0x7f9f180ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.435+0000 7f9f1cb72640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180cc470 con 0x7f9f00004800 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.435+0000 7f9f1cb72640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180cc2f0 con 0x7f9f18159b70 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.435+0000 7f9f1cb72640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180cc5f0 con 0x7f9f18156d10 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f1813f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f1813f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34216/0 (socket says 172.21.15.77:34216) 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 -- 172.21.15.77:0/1521603906 learned_addr learned my addr 172.21.15.77:0/1521603906 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f16d9d640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f18159b70 0x7f9f180ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f15d9b640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f18156d10 0x7f9f18159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 -- 172.21.15.77:0/1521603906 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f18156d10 msgr2=0x7f9f18159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f18156d10 0x7f9f18159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 -- 172.21.15.77:0/1521603906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f18159b70 msgr2=0x7f9f180ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f18159b70 0x7f9f180ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 -- 172.21.15.77:0/1521603906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f181576a0 con 0x7f9f00004800 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1659c640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f1813f230 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9f080072c0 tx=0x7f9f08007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f06ffd640 1 -- 172.21.15.77:0/1521603906 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0801a070 con 0x7f9f00004800 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f06ffd640 1 -- 172.21.15.77:0/1521603906 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0800f960 con 0x7f9f00004800 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f18157940 con 0x7f9f00004800 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f06ffd640 1 -- 172.21.15.77:0/1521603906 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0800fd10 con 0x7f9f00004800 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.436+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f18157f00 con 0x7f9f00004800 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.437+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f180b2a90 con 0x7f9f00004800 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.438+0000 7f9f06ffd640 1 -- 172.21.15.77:0/1521603906 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f0801d070 con 0x7f9f00004800 2024-03-06T07:25:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.438+0000 7f9f06ffd640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef806b4d0 0x7f9ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.438+0000 7f9f15d9b640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef806b4d0 0x7f9ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.438+0000 7f9f06ffd640 1 -- 172.21.15.77:0/1521603906 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9f08089a10 con 0x7f9f00004800 2024-03-06T07:25:42.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.438+0000 7f9f15d9b640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef806b4d0 0x7f9ef806d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c036000 tx=0x7f9f0c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.442+0000 7f9f06ffd640 1 -- 172.21.15.77:0/1521603906 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f08058be0 con 0x7f9f00004800 2024-03-06T07:25:42.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.616+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9f1804f330 con 0x7f9f00004800 2024-03-06T07:25:42.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.617+0000 7f9f06ffd640 1 -- 172.21.15.77:0/1521603906 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9f080584e0 con 0x7f9f00004800 2024-03-06T07:25:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:42.619 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:25:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:25:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:25:42.619 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:42.619 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:25:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef806b4d0 msgr2=0x7f9ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef806b4d0 0x7f9ef806d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c036000 tx=0x7f9f0c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 msgr2=0x7f9f1813f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f1813f230 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9f080072c0 tx=0x7f9f08007780 comp rx=0 tx=0).stop 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 shutdown_connections 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f18159b70 0x7f9f180ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f18156d10 0x7f9f18159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ef806b4d0 0x7f9ef806d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 --2- 172.21.15.77:0/1521603906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f00004800 0x7f9f1813f230 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.618+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 >> 172.21.15.77:0/1521603906 conn(0x7f9f180cf8a0 msgr2=0x7f9f1806cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.619+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 shutdown_connections 2024-03-06T07:25:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.619+0000 7f9f1cb72640 1 -- 172.21.15.77:0/1521603906 wait complete. 2024-03-06T07:25:42.631 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:42.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.734+0000 7f14fb41c640 1 Processor -- start 2024-03-06T07:25:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.734+0000 7f14fb41c640 1 -- start start 2024-03-06T07:25:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.735+0000 7f14fb41c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14f40d4630 0x7f14f40d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.735+0000 7f14fb41c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f14f40d2820 0x7f14f40cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.735+0000 7f14fb41c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f415e380 con 0x7f14f415dfa0 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.735+0000 7f14fb41c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f40d2100 con 0x7f14f40d2820 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.735+0000 7f14fb41c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cf470 con 0x7f14f40d4630 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.735+0000 7f14f99b9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f14f40d2820 0x7f14f40cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.735+0000 7f14f91b8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f14f415dfa0 0x7f14f40d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59424/0 (socket says 172.21.15.77:59424) 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.735+0000 7f14f91b8640 1 -- 172.21.15.77:0/2765491202 learned_addr learned my addr 172.21.15.77:0/2765491202 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.736+0000 7f14f99b9640 1 -- 172.21.15.77:0/2765491202 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14f40d4630 msgr2=0x7f14f40d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.736+0000 7f14f99b9640 1 --2- 172.21.15.77:0/2765491202 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14f40d4630 0x7f14f40d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.736+0000 7f14f99b9640 1 -- 172.21.15.77:0/2765491202 >> v1:172.21.15.77:6789/0 conn(0x7f14f415dfa0 legacy=0x7f14f40d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.736+0000 7f14f99b9640 1 -- 172.21.15.77:0/2765491202 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40cf750 con 0x7f14f40d2820 2024-03-06T07:25:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.736+0000 7f14ebfff640 1 -- 172.21.15.77:0/2765491202 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230464124 0 0) 0x7f14f415e380 con 0x7f14f415dfa0 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.736+0000 7f14f99b9640 1 --2- 172.21.15.77:0/2765491202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f40d2820 0x7f14f40cb190 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097a0 tx=0x7f14ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25f5fa4bd868d63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.736+0000 7f14ebfff640 1 -- 172.21.15.77:0/2765491202 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0315a0 con 0x7f14f40d2820 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14ebfff640 1 -- 172.21.15.77:0/2765491202 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec037070 con 0x7f14f40d2820 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14ebfff640 1 -- 172.21.15.77:0/2765491202 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec031960 con 0x7f14f40d2820 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14fb41c640 1 -- 172.21.15.77:0/2765491202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f40d2820 msgr2=0x7f14f40cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2765491202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f40d2820 0x7f14f40cb190 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097a0 tx=0x7f14ec02b840 comp rx=0 tx=0).stop 2024-03-06T07:25:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14fb41c640 1 -- 172.21.15.77:0/2765491202 shutdown_connections 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2765491202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f40d2820 0x7f14f40cb190 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2765491202 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14f40d4630 0x7f14f40d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14fb41c640 1 -- 172.21.15.77:0/2765491202 >> 172.21.15.77:0/2765491202 conn(0x7f14f409d000 msgr2=0x7f14f40d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14fb41c640 1 -- 172.21.15.77:0/2765491202 shutdown_connections 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.737+0000 7f14fb41c640 1 -- 172.21.15.77:0/2765491202 wait complete. 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.738+0000 7f14fb41c640 1 Processor -- start 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.738+0000 7f14fb41c640 1 -- start start 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14fb41c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14f40d2820 0x7f14f40b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14fb41c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14f40d4630 0x7f14f409e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14fb41c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f415dfa0 0x7f14f41f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14fb41c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cb850 con 0x7f14f415dfa0 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14fb41c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cb6d0 con 0x7f14f40d4630 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14fb41c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cb9d0 con 0x7f14f40d2820 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14f99b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f415dfa0 0x7f14f41f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14f99b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f415dfa0 0x7f14f41f7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34228/0 (socket says 172.21.15.77:34228) 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14f99b9640 1 -- 172.21.15.77:0/2584559010 learned_addr learned my addr 172.21.15.77:0/2584559010 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14f99b9640 1 -- 172.21.15.77:0/2584559010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14f40d2820 msgr2=0x7f14f40b8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14f99b9640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14f40d2820 0x7f14f40b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14f99b9640 1 -- 172.21.15.77:0/2584559010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14f40d4630 msgr2=0x7f14f409e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14f99b9640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14f40d4630 0x7f14f409e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.739+0000 7f14f99b9640 1 -- 172.21.15.77:0/2584559010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40a0c10 con 0x7f14f415dfa0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.740+0000 7f14f99b9640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f415dfa0 0x7f14f41f7180 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097a0 tx=0x7f14ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.740+0000 7f14e9ffb640 1 -- 172.21.15.77:0/2584559010 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec004230 con 0x7f14f415dfa0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.740+0000 7f14e9ffb640 1 -- 172.21.15.77:0/2584559010 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec0043d0 con 0x7f14f415dfa0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.740+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f40a0ef0 con 0x7f14f415dfa0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.740+0000 7f14e9ffb640 1 -- 172.21.15.77:0/2584559010 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec032860 con 0x7f14f415dfa0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.740+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f40a14b0 con 0x7f14f415dfa0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.741+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f404f330 con 0x7f14f415dfa0 2024-03-06T07:25:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.741+0000 7f14e9ffb640 1 -- 172.21.15.77:0/2584559010 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14ec032a00 con 0x7f14f415dfa0 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.742+0000 7f14e9ffb640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14d806b4d0 0x7f14d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.742+0000 7f14e9ffb640 1 -- 172.21.15.77:0/2584559010 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f14ec02f030 con 0x7f14f415dfa0 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.742+0000 7f14f91b8640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14d806b4d0 0x7f14d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.742+0000 7f14f91b8640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14d806b4d0 0x7f14d806d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f14f0007fd0 tx=0x7f14f0007ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.746+0000 7f14e9ffb640 1 -- 172.21.15.77:0/2584559010 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14ec041070 con 0x7f14f415dfa0 2024-03-06T07:25:42.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:42.918+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f14f406c350 con 0x7f14f415dfa0 2024-03-06T07:25:43.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.354+0000 7f14e9ffb640 1 -- 172.21.15.77:0/2584559010 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f14ec07b130 con 0x7f14f415dfa0 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14d806b4d0 msgr2=0x7f14d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14d806b4d0 0x7f14d806d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f14f0007fd0 tx=0x7f14f0007ce0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f415dfa0 msgr2=0x7f14f41f7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f415dfa0 0x7f14f41f7180 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097a0 tx=0x7f14ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 shutdown_connections 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f14d806b4d0 0x7f14d806d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14f415dfa0 0x7f14f41f7180 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14f40d4630 0x7f14f409e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 --2- 172.21.15.77:0/2584559010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14f40d2820 0x7f14f40b8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 >> 172.21.15.77:0/2584559010 conn(0x7f14f409d000 msgr2=0x7f14f4160540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.357+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 shutdown_connections 2024-03-06T07:25:43.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.358+0000 7f14fb41c640 1 -- 172.21.15.77:0/2584559010 wait complete. 2024-03-06T07:25:43.374 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:43.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.482+0000 7f8d702df640 1 Processor -- start 2024-03-06T07:25:43.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.482+0000 7f8d702df640 1 -- start start 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d702df640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8d6815f4f0 0x7f8d68066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d702df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d680b0db0 0x7f8d680ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d702df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6806b050 con 0x7f8d680ce580 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d702df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6806b2c0 con 0x7f8d680b0db0 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d702df640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d680b0890 con 0x7f8d6815f4f0 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d6d87a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d680b0db0 0x7f8d680ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d6d87a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d680b0db0 0x7f8d680ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34232/0 (socket says 172.21.15.77:34232) 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d6d87a640 1 -- 172.21.15.77:0/3243678606 learned_addr learned my addr 172.21.15.77:0/3243678606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d6e07b640 1 --2- 172.21.15.77:0/3243678606 >> v2:172.21.15.158:3300/0 conn(0x7f8d6815f4f0 0x7f8d68066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d6d87a640 1 -- 172.21.15.77:0/3243678606 >> v2:172.21.15.158:3300/0 conn(0x7f8d6815f4f0 msgr2=0x7f8d68066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d6d87a640 1 --2- 172.21.15.77:0/3243678606 >> v2:172.21.15.158:3300/0 conn(0x7f8d6815f4f0 0x7f8d68066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d6d87a640 1 -- 172.21.15.77:0/3243678606 >> v1:172.21.15.158:6789/0 conn(0x7f8d680ce580 legacy=0x7f8d68072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.483+0000 7f8d6d87a640 1 -- 172.21.15.77:0/3243678606 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d680672e0 con 0x7f8d680b0db0 2024-03-06T07:25:43.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.484+0000 7f8d6d87a640 1 --2- 172.21.15.77:0/3243678606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d680b0db0 0x7f8d680ce040 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f8d680707a0 tx=0x7f8d5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d633a5ec58ee2da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.484+0000 7f8d6d079640 1 -- 172.21.15.77:0/3243678606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58034070 con 0x7f8d680b0db0 2024-03-06T07:25:43.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.484+0000 7f8d6d079640 1 -- 172.21.15.77:0/3243678606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d58030860 con 0x7f8d680b0db0 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.484+0000 7f8d6d079640 1 -- 172.21.15.77:0/3243678606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58030c30 con 0x7f8d680b0db0 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.485+0000 7f8d702df640 1 -- 172.21.15.77:0/3243678606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d680b0db0 msgr2=0x7f8d680ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.485+0000 7f8d702df640 1 --2- 172.21.15.77:0/3243678606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d680b0db0 0x7f8d680ce040 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f8d680707a0 tx=0x7f8d5802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.485+0000 7f8d702df640 1 -- 172.21.15.77:0/3243678606 shutdown_connections 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.485+0000 7f8d702df640 1 --2- 172.21.15.77:0/3243678606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d680b0db0 0x7f8d680ce040 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.485+0000 7f8d702df640 1 --2- 172.21.15.77:0/3243678606 >> v2:172.21.15.158:3300/0 conn(0x7f8d6815f4f0 0x7f8d68066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.485+0000 7f8d702df640 1 -- 172.21.15.77:0/3243678606 >> 172.21.15.77:0/3243678606 conn(0x7f8d680b1d70 msgr2=0x7f8d680b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.485+0000 7f8d702df640 1 -- 172.21.15.77:0/3243678606 shutdown_connections 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.485+0000 7f8d702df640 1 -- 172.21.15.77:0/3243678606 wait complete. 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.486+0000 7f8d702df640 1 Processor -- start 2024-03-06T07:25:43.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.486+0000 7f8d702df640 1 -- start start 2024-03-06T07:25:43.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.486+0000 7f8d702df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d680b0db0 0x7f8d681579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.486+0000 7f8d702df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d680ce580 0x7f8d680c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.486+0000 7f8d702df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6815f4f0 0x7f8d681f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.486+0000 7f8d702df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6815f8d0 con 0x7f8d6815f4f0 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.486+0000 7f8d702df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6806b5a0 con 0x7f8d680b0db0 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.486+0000 7f8d702df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d681f53b0 con 0x7f8d680ce580 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6815f4f0 0x7f8d681f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6815f4f0 0x7f8d681f4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34236/0 (socket says 172.21.15.77:34236) 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 -- 172.21.15.77:0/3972171316 learned_addr learned my addr 172.21.15.77:0/3972171316 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e07b640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d680b0db0 0x7f8d681579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6d87a640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d680ce580 0x7f8d680c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 -- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d680ce580 msgr2=0x7f8d680c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d680ce580 0x7f8d680c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 -- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d680b0db0 msgr2=0x7f8d681579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d680b0db0 0x7f8d681579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 -- 172.21.15.77:0/3972171316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d68153bf0 con 0x7f8d6815f4f0 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6d87a640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d680ce580 0x7f8d680c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d6e87c640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6815f4f0 0x7f8d681f4e70 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c7a0 tx=0x7f8d6000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d5effd640 1 -- 172.21.15.77:0/3972171316 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60003b70 con 0x7f8d6815f4f0 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d5effd640 1 -- 172.21.15.77:0/3972171316 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d60003d30 con 0x7f8d6815f4f0 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d5effd640 1 -- 172.21.15.77:0/3972171316 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000aa90 con 0x7f8d6815f4f0 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d68157fa0 con 0x7f8d6815f4f0 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.487+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d68158560 con 0x7f8d6815f4f0 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.488+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6809e3d0 con 0x7f8d6815f4f0 2024-03-06T07:25:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.489+0000 7f8d5effd640 1 -- 172.21.15.77:0/3972171316 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d6000ac30 con 0x7f8d6815f4f0 2024-03-06T07:25:43.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.489+0000 7f8d5effd640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d4c06b4d0 0x7f8d4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.489+0000 7f8d6e07b640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d4c06b4d0 0x7f8d4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.489+0000 7f8d5effd640 1 -- 172.21.15.77:0/3972171316 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8d60089e40 con 0x7f8d6815f4f0 2024-03-06T07:25:43.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.490+0000 7f8d6e07b640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d4c06b4d0 0x7f8d4c06d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8d640079a0 tx=0x7f8d64008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.494+0000 7f8d5effd640 1 -- 172.21.15.77:0/3972171316 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d60058f30 con 0x7f8d6815f4f0 2024-03-06T07:25:43.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:43.665+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8d6815c590 con 0x7f8d6815f4f0 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.360+0000 7f8d5effd640 1 -- 172.21.15.77:0/3972171316 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v78) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8d60058830 con 0x7f8d6815f4f0 2024-03-06T07:25:44.362 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d4c06b4d0 msgr2=0x7f8d4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d4c06b4d0 0x7f8d4c06d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8d640079a0 tx=0x7f8d64008040 comp rx=0 tx=0).stop 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6815f4f0 msgr2=0x7f8d681f4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6815f4f0 0x7f8d681f4e70 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c7a0 tx=0x7f8d6000cc60 comp rx=0 tx=0).stop 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 shutdown_connections 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8d4c06b4d0 0x7f8d4c06d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d6815f4f0 0x7f8d681f4e70 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d680ce580 0x7f8d680c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 --2- 172.21.15.77:0/3972171316 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d680b0db0 0x7f8d681579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 >> 172.21.15.77:0/3972171316 conn(0x7f8d680b1d70 msgr2=0x7f8d68072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 shutdown_connections 2024-03-06T07:25:44.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.363+0000 7f8d702df640 1 -- 172.21.15.77:0/3972171316 wait complete. 2024-03-06T07:25:44.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.493+0000 7fefc9639640 1 Processor -- start 2024-03-06T07:25:44.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.493+0000 7fefc9639640 1 -- start start 2024-03-06T07:25:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc9639640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fefc415e0b0 0x7fefc40b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc9639640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc40d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc9639640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc40b4290 con 0x7fefc40d28c0 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc9639640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc40ce990 con 0x7fefc415e0b0 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc9639640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc40b4570 con 0x7fefc40b3df0 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc2ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fefc415e0b0 0x7fefc40b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fefc40d28c0 0x7fefc40d0790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59434/0 (socket says 172.21.15.77:59434) 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc37fe640 1 -- 172.21.15.77:0/3146118556 learned_addr learned my addr 172.21.15.77:0/3146118556 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc27fc640 1 --2- 172.21.15.77:0/3146118556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc40d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc2ffd640 1 -- 172.21.15.77:0/3146118556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 msgr2=0x7fefc40d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc2ffd640 1 --2- 172.21.15.77:0/3146118556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc40d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc2ffd640 1 -- 172.21.15.77:0/3146118556 >> v1:172.21.15.77:6789/0 conn(0x7fefc40d28c0 legacy=0x7fefc40d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc2ffd640 1 -- 172.21.15.77:0/3146118556 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc40b4850 con 0x7fefc415e0b0 2024-03-06T07:25:44.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc1ffb640 1 -- 172.21.15.77:0/3146118556 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154579885 0 0) 0x7fefc40b4290 con 0x7fefc40d28c0 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.494+0000 7fefc27fc640 1 --2- 172.21.15.77:0/3146118556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc40d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.495+0000 7fefc2ffd640 1 --2- 172.21.15.77:0/3146118556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc40b38b0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009d90 tx=0x7fefb402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be1e25f062360e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.495+0000 7fefc1ffb640 1 -- 172.21.15.77:0/3146118556 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4039070 con 0x7fefc415e0b0 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.495+0000 7fefc1ffb640 1 -- 172.21.15.77:0/3146118556 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb402cc50 con 0x7fefc415e0b0 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.495+0000 7fefc1ffb640 1 -- 172.21.15.77:0/3146118556 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4034440 con 0x7fefc415e0b0 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.496+0000 7fefc9639640 1 -- 172.21.15.77:0/3146118556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 msgr2=0x7fefc40b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.496+0000 7fefc9639640 1 --2- 172.21.15.77:0/3146118556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc40b38b0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009d90 tx=0x7fefb402bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.496+0000 7fefc9639640 1 -- 172.21.15.77:0/3146118556 shutdown_connections 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.496+0000 7fefc9639640 1 --2- 172.21.15.77:0/3146118556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc40d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.496+0000 7fefc9639640 1 --2- 172.21.15.77:0/3146118556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc40b38b0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.496+0000 7fefc9639640 1 -- 172.21.15.77:0/3146118556 >> 172.21.15.77:0/3146118556 conn(0x7fefc408d6f0 msgr2=0x7fefc409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.496+0000 7fefc9639640 1 -- 172.21.15.77:0/3146118556 shutdown_connections 2024-03-06T07:25:44.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.496+0000 7fefc9639640 1 -- 172.21.15.77:0/3146118556 wait complete. 2024-03-06T07:25:44.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.497+0000 7fefc9639640 1 Processor -- start 2024-03-06T07:25:44.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.497+0000 7fefc9639640 1 -- start start 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc9639640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc9639640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefc40d28c0 0x7fefc41611a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc9639640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc415f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc9639640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc40d1cd0 con 0x7fefc415e0b0 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc9639640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc415e490 con 0x7fefc40d28c0 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc9639640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc40d1e50 con 0x7fefc40b3df0 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc415f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc415f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34256/0 (socket says 172.21.15.77:34256) 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 -- 172.21.15.77:0/172802715 learned_addr learned my addr 172.21.15.77:0/172802715 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc2ffd640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 -- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 msgr2=0x7fefc415a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc27fc640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefc40d28c0 0x7fefc41611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc415a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 -- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefc40d28c0 msgr2=0x7fefc41611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefc40d28c0 0x7fefc41611a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 -- 172.21.15.77:0/172802715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc415f560 con 0x7fefc415e0b0 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc2ffd640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc415a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc37fe640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc415f020 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00c4b0 tx=0x7fefbc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefab7fe640 1 -- 172.21.15.77:0/172802715 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc018070 con 0x7fefc415e0b0 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefab7fe640 1 -- 172.21.15.77:0/172802715 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc0074e0 con 0x7fefc415e0b0 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc40b8140 con 0x7fefc415e0b0 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefab7fe640 1 -- 172.21.15.77:0/172802715 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc004a20 con 0x7fefc415e0b0 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.498+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc415fac0 con 0x7fefc415e0b0 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.499+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc404f330 con 0x7fefc415e0b0 2024-03-06T07:25:44.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.500+0000 7fefab7fe640 1 -- 172.21.15.77:0/172802715 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefbc01b070 con 0x7fefc415e0b0 2024-03-06T07:25:44.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.500+0000 7fefab7fe640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.500+0000 7fefab7fe640 1 -- 172.21.15.77:0/172802715 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fefbc0893a0 con 0x7fefc415e0b0 2024-03-06T07:25:44.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.500+0000 7fefc2ffd640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.501+0000 7fefc2ffd640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fefb4036000 tx=0x7fefb4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.505+0000 7fefab7fe640 1 -- 172.21.15.77:0/172802715 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefc415f560 con 0x7fefc415e0b0 2024-03-06T07:25:44.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:44.711+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fefc40d2160 con 0x7fefc415e0b0 2024-03-06T07:25:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.362+0000 7fefab7fe640 1 -- 172.21.15.77:0/172802715 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7fefbc058280 con 0x7fefc415e0b0 2024-03-06T07:25:45.364 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.365+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 msgr2=0x7fefb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.365+0000 7fefc9639640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fefb4036000 tx=0x7fefb4030000 comp rx=0 tx=0).stop 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.365+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 msgr2=0x7fefc415f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.365+0000 7fefc9639640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc415f020 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fefbc00c4b0 tx=0x7fefbc00c970 comp rx=0 tx=0).stop 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.366+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 shutdown_connections 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.366+0000 7fefc9639640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.366+0000 7fefc9639640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc415e0b0 0x7fefc415f020 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.366+0000 7fefc9639640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefc40d28c0 0x7fefc41611a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.366+0000 7fefc9639640 1 --2- 172.21.15.77:0/172802715 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefc40b3df0 0x7fefc415a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.366+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 >> 172.21.15.77:0/172802715 conn(0x7fefc408d6f0 msgr2=0x7fefc40b2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.366+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 shutdown_connections 2024-03-06T07:25:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.366+0000 7fefc9639640 1 -- 172.21.15.77:0/172802715 wait complete. 2024-03-06T07:25:45.392 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:45.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.491+0000 7f613caf4640 1 Processor -- start 2024-03-06T07:25:45.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.491+0000 7f613caf4640 1 -- start start 2024-03-06T07:25:45.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613caf4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61381424f0 0x7f61380cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613caf4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61380cc980 con 0x7f613806da40 2024-03-06T07:25:45.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613caf4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61380ccc60 con 0x7f61380cd7a0 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613caf4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61380cd220 con 0x7f61381424f0 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f6136d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f613806da40 0x7f6138071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59436/0 (socket says 172.21.15.77:59436) 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f6136d9d640 1 -- 172.21.15.77:0/758025222 learned_addr learned my addr 172.21.15.77:0/758025222 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613659c640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61381424f0 0x7f61380cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933794432 0 0) 0x7f61380cc980 con 0x7f613806da40 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6120003520 con 0x7f613806da40 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1206172289 0 0) 0x7f6120003520 con 0x7f613806da40 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61381424f0 msgr2=0x7f61380cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.492+0000 7f613559a640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61381424f0 0x7f61380cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 >> v1:172.21.15.158:6789/0 conn(0x7f61380cd7a0 legacy=0x7f613806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380ccf40 con 0x7f613806da40 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f6128001670 con 0x7f613806da40 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 >> v1:172.21.15.77:6789/0 conn(0x7f613806da40 legacy=0x7f6138071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61381424f0 0x7f61200039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f6120004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6120004d40 0x7f61200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61380cd220 con 0x7f6120003f60 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612000baa0 con 0x7f6120004d40 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612000bca0 con 0x7f61381424f0 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f6135d9b640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f6120004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f613659c640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61381424f0 0x7f61200039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f6136d9d640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6120004d40 0x7f61200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f6135d9b640 1 -- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61381424f0 msgr2=0x7f61200039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f6135d9b640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61381424f0 0x7f61200039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f6135d9b640 1 -- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6120004d40 msgr2=0x7f61200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f6135d9b640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6120004d40 0x7f61200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.493+0000 7f6135d9b640 1 -- 172.21.15.77:0/758025222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6120003520 con 0x7f6120003f60 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f6135d9b640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f6120004800 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f612c001e50 tx=0x7f6120005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49700b4590292484 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c001a70 con 0x7f6120003f60 2024-03-06T07:25:45.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612c003b70 con 0x7f6120003f60 2024-03-06T07:25:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f613559a640 1 -- 172.21.15.77:0/758025222 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c0050d0 con 0x7f6120003f60 2024-03-06T07:25:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f613caf4640 1 -- 172.21.15.77:0/758025222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 msgr2=0x7f6120004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f613caf4640 1 --2- 172.21.15.77:0/758025222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f6120004800 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f612c001e50 tx=0x7f6120005830 comp rx=0 tx=0).stop 2024-03-06T07:25:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f6136d9d640 1 -- 172.21.15.77:0/758025222 reap_dead start 2024-03-06T07:25:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f613caf4640 1 -- 172.21.15.77:0/758025222 shutdown_connections 2024-03-06T07:25:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f613caf4640 1 -- 172.21.15.77:0/758025222 >> 172.21.15.77:0/758025222 conn(0x7f61380cf8a0 msgr2=0x7f61380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.494+0000 7f613caf4640 1 -- 172.21.15.77:0/758025222 shutdown_connections 2024-03-06T07:25:45.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.495+0000 7f613caf4640 1 -- 172.21.15.77:0/758025222 wait complete. 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.496+0000 7f613caf4640 1 Processor -- start 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.496+0000 7f613caf4640 1 -- start start 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.496+0000 7f613caf4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f613813e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.496+0000 7f613caf4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f613815ba80 0x7f613813c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.496+0000 7f613caf4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6138144680 0x7f6138144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.496+0000 7f613caf4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61380cc470 con 0x7f6120003f60 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.496+0000 7f613caf4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61380cc2f0 con 0x7f613815ba80 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.496+0000 7f613caf4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61380cc5f0 con 0x7f6138144680 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f613813e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f613813e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34270/0 (socket says 172.21.15.77:34270) 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 -- 172.21.15.77:0/1267146244 learned_addr learned my addr 172.21.15.77:0/1267146244 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:45.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f6136d9d640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6138144680 0x7f6138144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f6135d9b640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f613815ba80 0x7f613813c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 -- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6138144680 msgr2=0x7f6138144b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6138144680 0x7f6138144b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 -- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f613815ba80 msgr2=0x7f613813c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f613815ba80 0x7f613813c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 -- 172.21.15.77:0/1267146244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613813b9f0 con 0x7f6120003f60 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f6136d9d640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6138144680 0x7f6138144b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f6135d9b640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f613815ba80 0x7f613813c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613659c640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f613813e5a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f61300119d0 tx=0x7f6130011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f6126ffd640 1 -- 172.21.15.77:0/1267146244 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613000fa90 con 0x7f6120003f60 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f6126ffd640 1 -- 172.21.15.77:0/1267146244 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613000fc30 con 0x7f6120003f60 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61381392a0 con 0x7f6120003f60 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f6126ffd640 1 -- 172.21.15.77:0/1267146244 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130014440 con 0x7f6120003f60 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.497+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613813bc90 con 0x7f6120003f60 2024-03-06T07:25:45.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.498+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613804f330 con 0x7f6120003f60 2024-03-06T07:25:45.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.499+0000 7f6126ffd640 1 -- 172.21.15.77:0/1267146244 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6130012040 con 0x7f6120003f60 2024-03-06T07:25:45.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.499+0000 7f6126ffd640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611806b500 0x7f611806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.499+0000 7f6126ffd640 1 -- 172.21.15.77:0/1267146244 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f613008a840 con 0x7f6120003f60 2024-03-06T07:25:45.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.499+0000 7f6135d9b640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611806b500 0x7f611806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.499+0000 7f6135d9b640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611806b500 0x7f611806d9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f612c034000 tx=0x7f612c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.503+0000 7f6126ffd640 1 -- 172.21.15.77:0/1267146244 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6130019030 con 0x7f6120003f60 2024-03-06T07:25:45.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.675+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61380d0620 con 0x7f6120003f60 2024-03-06T07:25:45.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.676+0000 7f6126ffd640 1 -- 172.21.15.77:0/1267146244 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f6130059070 con 0x7f6120003f60 2024-03-06T07:25:45.679 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:45.679 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":79,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:44.712250+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:42.937014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:43.684499+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.678+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611806b500 msgr2=0x7f611806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.678+0000 7f613caf4640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611806b500 0x7f611806d9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f612c034000 tx=0x7f612c009030 comp rx=0 tx=0).stop 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.678+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 msgr2=0x7f613813e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.678+0000 7f613caf4640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f613813e5a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f61300119d0 tx=0x7f6130011e90 comp rx=0 tx=0).stop 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.679+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 shutdown_connections 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.679+0000 7f613caf4640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6138144680 0x7f6138144b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.679+0000 7f613caf4640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f613815ba80 0x7f613813c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.679+0000 7f613caf4640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611806b500 0x7f611806d9c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.679+0000 7f613caf4640 1 --2- 172.21.15.77:0/1267146244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6120003f60 0x7f613813e5a0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.679+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 >> 172.21.15.77:0/1267146244 conn(0x7f61380cf8a0 msgr2=0x7f613806c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.679+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 shutdown_connections 2024-03-06T07:25:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.679+0000 7f613caf4640 1 -- 172.21.15.77:0/1267146244 wait complete. 2024-03-06T07:25:45.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:45.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.790+0000 7f11ca165640 1 Processor -- start 2024-03-06T07:25:45.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.790+0000 7f11ca165640 1 -- start start 2024-03-06T07:25:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11ca165640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11c41515e0 0x7f11c40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11ca165640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f11c415b430 0x7f11c4071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11ca165640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c415b8e0 con 0x7f11c40cd7a0 2024-03-06T07:25:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11ca165640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c4070a70 con 0x7f11c41515e0 2024-03-06T07:25:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11ca165640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c4070d50 con 0x7f11c415b430 2024-03-06T07:25:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11c37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11c41515e0 0x7f11c40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11c37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11c41515e0 0x7f11c40cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34278/0 (socket says 172.21.15.77:34278) 2024-03-06T07:25:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11c37fe640 1 -- 172.21.15.77:0/3158927848 learned_addr learned my addr 172.21.15.77:0/3158927848 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:45.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11c2ffd640 1 --2- 172.21.15.77:0/3158927848 >> v2:172.21.15.158:3300/0 conn(0x7f11c415b430 0x7f11c4071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11c37fe640 1 -- 172.21.15.77:0/3158927848 >> v2:172.21.15.158:3300/0 conn(0x7f11c415b430 msgr2=0x7f11c4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11c37fe640 1 --2- 172.21.15.77:0/3158927848 >> v2:172.21.15.158:3300/0 conn(0x7f11c415b430 0x7f11c4071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11c37fe640 1 -- 172.21.15.77:0/3158927848 >> v1:172.21.15.158:6789/0 conn(0x7f11c40cd7a0 legacy=0x7f11c40cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.791+0000 7f11c37fe640 1 -- 172.21.15.77:0/3158927848 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c4071030 con 0x7f11c41515e0 2024-03-06T07:25:45.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11c37fe640 1 --2- 172.21.15.77:0/3158927848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c41515e0 0x7f11c40cce40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f11c40cc110 tx=0x7f11b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f9ff25c5d5a543 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11c27fc640 1 -- 172.21.15.77:0/3158927848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8034070 con 0x7f11c41515e0 2024-03-06T07:25:45.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11c27fc640 1 -- 172.21.15.77:0/3158927848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b8030860 con 0x7f11c41515e0 2024-03-06T07:25:45.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11c27fc640 1 -- 172.21.15.77:0/3158927848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8030c30 con 0x7f11c41515e0 2024-03-06T07:25:45.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11ca165640 1 -- 172.21.15.77:0/3158927848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c41515e0 msgr2=0x7f11c40cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11ca165640 1 --2- 172.21.15.77:0/3158927848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c41515e0 0x7f11c40cce40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f11c40cc110 tx=0x7f11b802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11ca165640 1 -- 172.21.15.77:0/3158927848 shutdown_connections 2024-03-06T07:25:45.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11ca165640 1 --2- 172.21.15.77:0/3158927848 >> v2:172.21.15.158:3300/0 conn(0x7f11c415b430 0x7f11c4071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11ca165640 1 --2- 172.21.15.77:0/3158927848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c41515e0 0x7f11c40cce40 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.792+0000 7f11ca165640 1 -- 172.21.15.77:0/3158927848 >> 172.21.15.77:0/3158927848 conn(0x7f11c40cf8a0 msgr2=0x7f11c40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.793+0000 7f11ca165640 1 -- 172.21.15.77:0/3158927848 shutdown_connections 2024-03-06T07:25:45.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.793+0000 7f11ca165640 1 -- 172.21.15.77:0/3158927848 wait complete. 2024-03-06T07:25:45.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.794+0000 7f11ca165640 1 Processor -- start 2024-03-06T07:25:45.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.794+0000 7f11ca165640 1 -- start start 2024-03-06T07:25:45.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.794+0000 7f11ca165640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40cd7a0 0x7f11c40cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.794+0000 7f11ca165640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c41515e0 0x7f11c4143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.794+0000 7f11ca165640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c415b430 0x7f11c4143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.794+0000 7f11ca165640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4071ef0 con 0x7f11c40cd7a0 2024-03-06T07:25:45.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.794+0000 7f11ca165640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4071d70 con 0x7f11c415b430 2024-03-06T07:25:45.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.794+0000 7f11ca165640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40cc3a0 con 0x7f11c41515e0 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40cd7a0 0x7f11c40cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40cd7a0 0x7f11c40cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34286/0 (socket says 172.21.15.77:34286) 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 -- 172.21.15.77:0/2816723488 learned_addr learned my addr 172.21.15.77:0/2816723488 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c2ffd640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c41515e0 0x7f11c4143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c3fff640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c415b430 0x7f11c4143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 -- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c41515e0 msgr2=0x7f11c4143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c41515e0 0x7f11c4143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 -- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c415b430 msgr2=0x7f11c4143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c415b430 0x7f11c4143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 -- 172.21.15.77:0/2816723488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c41444a0 con 0x7f11c40cd7a0 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c2ffd640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c41515e0 0x7f11c4143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c3fff640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c415b430 0x7f11c4143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:45.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11c37fe640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40cd7a0 0x7f11c40cde60 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f11b802b9e0 tx=0x7f11b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11abfff640 1 -- 172.21.15.77:0/2816723488 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8048070 con 0x7f11c40cd7a0 2024-03-06T07:25:45.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11abfff640 1 -- 172.21.15.77:0/2816723488 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b8030660 con 0x7f11c40cd7a0 2024-03-06T07:25:45.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c4144780 con 0x7f11c40cd7a0 2024-03-06T07:25:45.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11abfff640 1 -- 172.21.15.77:0/2816723488 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8034040 con 0x7f11c40cd7a0 2024-03-06T07:25:45.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.795+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c4144d40 con 0x7f11c40cd7a0 2024-03-06T07:25:45.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.796+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c4068840 con 0x7f11c40cd7a0 2024-03-06T07:25:45.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.797+0000 7f11abfff640 1 -- 172.21.15.77:0/2816723488 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11b804b070 con 0x7f11c40cd7a0 2024-03-06T07:25:45.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.797+0000 7f11abfff640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11b006b500 0x7f11b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.797+0000 7f11abfff640 1 -- 172.21.15.77:0/2816723488 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11b80ac200 con 0x7f11c40cd7a0 2024-03-06T07:25:45.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.797+0000 7f11c2ffd640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11b006b500 0x7f11b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.797+0000 7f11c2ffd640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11b006b500 0x7f11b006d9c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f11b4006fd0 tx=0x7f11b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.801+0000 7f11abfff640 1 -- 172.21.15.77:0/2816723488 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11b807b080 con 0x7f11c40cd7a0 2024-03-06T07:25:45.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.973+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11c404f330 con 0x7f11c40cd7a0 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.974+0000 7f11abfff640 1 -- 172.21.15.77:0/2816723488 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f11b8030830 con 0x7f11c40cd7a0 2024-03-06T07:25:45.976 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:45.976 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":79,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:44.712250+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:42.937014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:43.684499+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:45.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11b006b500 msgr2=0x7f11b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11b006b500 0x7f11b006d9c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f11b4006fd0 tx=0x7f11b4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:45.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40cd7a0 msgr2=0x7f11c40cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40cd7a0 0x7f11c40cde60 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f11b802b9e0 tx=0x7f11b8004730 comp rx=0 tx=0).stop 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 shutdown_connections 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c415b430 0x7f11c4143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c41515e0 0x7f11c4143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11b006b500 0x7f11b006d9c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 --2- 172.21.15.77:0/2816723488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40cd7a0 0x7f11c40cde60 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 >> 172.21.15.77:0/2816723488 conn(0x7f11c40cf8a0 msgr2=0x7f11c414fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 shutdown_connections 2024-03-06T07:25:45.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:45.976+0000 7f11ca165640 1 -- 172.21.15.77:0/2816723488 wait complete. 2024-03-06T07:25:45.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:46.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.091+0000 7f9381481640 1 Processor -- start 2024-03-06T07:25:46.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.091+0000 7f9381481640 1 -- start start 2024-03-06T07:25:46.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f9381481640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f937c0cf0d0 0x7f937c15a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f9381481640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0b3320 con 0x7f937c0cfa90 2024-03-06T07:25:46.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f9381481640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c15a760 con 0x7f937c0719e0 2024-03-06T07:25:46.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f9381481640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937c0728d0 con 0x7f937c0cf0d0 2024-03-06T07:25:46.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f937affd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f937c0cf0d0 0x7f937c15a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f9373fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f937c0cfa90 0x7f937c0712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59438/0 (socket says 172.21.15.77:59438) 2024-03-06T07:25:46.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f9373fff640 1 -- 172.21.15.77:0/671062601 learned_addr learned my addr 172.21.15.77:0/671062601 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:46.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f937affd640 1 -- 172.21.15.77:0/671062601 >> v1:172.21.15.158:6789/0 conn(0x7f937c0719e0 legacy=0x7f937c1651f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f937affd640 1 -- 172.21.15.77:0/671062601 >> v1:172.21.15.77:6789/0 conn(0x7f937c0cfa90 legacy=0x7f937c0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.092+0000 7f937affd640 1 -- 172.21.15.77:0/671062601 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c0b3590 con 0x7f937c0cf0d0 2024-03-06T07:25:46.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.093+0000 7f937affd640 1 --2- 172.21.15.77:0/671062601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cf0d0 0x7f937c15a220 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f936c0097a0 tx=0x7f936c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873091436e61de3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.093+0000 7f937a7fc640 1 -- 172.21.15.77:0/671062601 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c035020 con 0x7f937c0cf0d0 2024-03-06T07:25:46.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.093+0000 7f937a7fc640 1 -- 172.21.15.77:0/671062601 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c031830 con 0x7f937c0cf0d0 2024-03-06T07:25:46.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.093+0000 7f937a7fc640 1 -- 172.21.15.77:0/671062601 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c031c00 con 0x7f937c0cf0d0 2024-03-06T07:25:46.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.094+0000 7f9381481640 1 -- 172.21.15.77:0/671062601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cf0d0 msgr2=0x7f937c15a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.094+0000 7f9381481640 1 --2- 172.21.15.77:0/671062601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cf0d0 0x7f937c15a220 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f936c0097a0 tx=0x7f936c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.094+0000 7f9381481640 1 -- 172.21.15.77:0/671062601 shutdown_connections 2024-03-06T07:25:46.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.094+0000 7f9381481640 1 --2- 172.21.15.77:0/671062601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cf0d0 0x7f937c15a220 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.094+0000 7f9381481640 1 -- 172.21.15.77:0/671062601 >> 172.21.15.77:0/671062601 conn(0x7f937c06b190 msgr2=0x7f937c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.094+0000 7f9381481640 1 -- 172.21.15.77:0/671062601 shutdown_connections 2024-03-06T07:25:46.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.094+0000 7f9381481640 1 -- 172.21.15.77:0/671062601 wait complete. 2024-03-06T07:25:46.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.095+0000 7f9381481640 1 Processor -- start 2024-03-06T07:25:46.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.095+0000 7f9381481640 1 -- start start 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9381481640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f937c0719e0 0x7f937c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9381481640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937c0cf0d0 0x7f937c0cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9381481640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cfa90 0x7f937c0ab110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9381481640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c072030 con 0x7f937c0cfa90 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9381481640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0cf4b0 con 0x7f937c0cf0d0 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9381481640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c072530 con 0x7f937c0719e0 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cfa90 0x7f937c0ab110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cfa90 0x7f937c0ab110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34306/0 (socket says 172.21.15.77:34306) 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 -- 172.21.15.77:0/2808930297 learned_addr learned my addr 172.21.15.77:0/2808930297 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937affd640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f937c0719e0 0x7f937c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9373fff640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937c0cf0d0 0x7f937c0cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 -- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f937c0719e0 msgr2=0x7f937c0a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f937c0719e0 0x7f937c0a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 -- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937c0cf0d0 msgr2=0x7f937c0cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937c0cf0d0 0x7f937c0cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 -- 172.21.15.77:0/2808930297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c1481a0 con 0x7f937c0cfa90 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937affd640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f937c0719e0 0x7f937c0a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f937b7fe640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cfa90 0x7f937c0ab110 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f937400c760 tx=0x7f937400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f93737fe640 1 -- 172.21.15.77:0/2808930297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374016020 con 0x7f937c0cfa90 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f93737fe640 1 -- 172.21.15.77:0/2808930297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9374006d50 con 0x7f937c0cfa90 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c148480 con 0x7f937c0cfa90 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f93737fe640 1 -- 172.21.15.77:0/2808930297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374003bb0 con 0x7f937c0cfa90 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.096+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c148a40 con 0x7f937c0cfa90 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.097+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c0732b0 con 0x7f937c0cfa90 2024-03-06T07:25:46.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.098+0000 7f93737fe640 1 -- 172.21.15.77:0/2808930297 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f937400a850 con 0x7f937c0cfa90 2024-03-06T07:25:46.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.098+0000 7f93737fe640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.098+0000 7f93737fe640 1 -- 172.21.15.77:0/2808930297 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9374089d90 con 0x7f937c0cfa90 2024-03-06T07:25:46.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.098+0000 7f937affd640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.099+0000 7f937affd640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f936c036000 tx=0x7f936c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.103+0000 7f93737fe640 1 -- 172.21.15.77:0/2808930297 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9374020020 con 0x7f937c0cfa90 2024-03-06T07:25:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:46.312+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f937c070ba0 con 0x7f937c0cfa90 2024-03-06T07:25:47.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.029+0000 7f93737fe640 1 -- 172.21.15.77:0/2808930297 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v40) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9374058ca0 con 0x7f937c0cfa90 2024-03-06T07:25:47.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.031+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 msgr2=0x7f935c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.031+0000 7f9381481640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f936c036000 tx=0x7f936c008030 comp rx=0 tx=0).stop 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.031+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cfa90 msgr2=0x7f937c0ab110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.031+0000 7f9381481640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cfa90 0x7f937c0ab110 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f937400c760 tx=0x7f937400cc20 comp rx=0 tx=0).stop 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.032+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 shutdown_connections 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.032+0000 7f9381481640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.032+0000 7f9381481640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f937c0cfa90 0x7f937c0ab110 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.032+0000 7f9381481640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f937c0cf0d0 0x7f937c0cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.032+0000 7f9381481640 1 --2- 172.21.15.77:0/2808930297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f937c0719e0 0x7f937c0a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.032+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 >> 172.21.15.77:0/2808930297 conn(0x7f937c06b190 msgr2=0x7f937c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.032+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 shutdown_connections 2024-03-06T07:25:47.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.032+0000 7f9381481640 1 -- 172.21.15.77:0/2808930297 wait complete. 2024-03-06T07:25:47.046 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:47.046 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.149+0000 7f22a7ceb640 1 Processor -- start 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.150+0000 7f22a7ceb640 1 -- start start 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.150+0000 7f22a7ceb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22a00719e0 0x7f22a00cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.150+0000 7f22a7ceb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22a006c9f0 0x7f22a006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.150+0000 7f22a7ceb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a00b3320 con 0x7f22a015a220 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.150+0000 7f22a7ceb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a00cf610 con 0x7f22a00719e0 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.150+0000 7f22a7ceb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a00728d0 con 0x7f22a006c9f0 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.150+0000 7f22a5286640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22a00719e0 0x7f22a00cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.150+0000 7f22a5286640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22a00719e0 0x7f22a00cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34312/0 (socket says 172.21.15.77:34312) 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.151+0000 7f22a5286640 1 -- 172.21.15.77:0/457535818 learned_addr learned my addr 172.21.15.77:0/457535818 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.151+0000 7f22a6288640 1 --2- 172.21.15.77:0/457535818 >> v2:172.21.15.158:3300/0 conn(0x7f22a006c9f0 0x7f22a006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.151+0000 7f22a5286640 1 -- 172.21.15.77:0/457535818 >> v2:172.21.15.158:3300/0 conn(0x7f22a006c9f0 msgr2=0x7f22a006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.151+0000 7f22a5286640 1 --2- 172.21.15.77:0/457535818 >> v2:172.21.15.158:3300/0 conn(0x7f22a006c9f0 0x7f22a006ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.151+0000 7f22a5286640 1 -- 172.21.15.77:0/457535818 >> v1:172.21.15.158:6789/0 conn(0x7f22a015a220 legacy=0x7f22a00712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.151+0000 7f22a5286640 1 -- 172.21.15.77:0/457535818 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a00b3590 con 0x7f22a00719e0 2024-03-06T07:25:47.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.151+0000 7f22a5286640 1 --2- 172.21.15.77:0/457535818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a00cf0d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f22a00aeed0 tx=0x7f229c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9effce055c60ccba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a4a85640 1 -- 172.21.15.77:0/457535818 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c035020 con 0x7f22a00719e0 2024-03-06T07:25:47.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a4a85640 1 -- 172.21.15.77:0/457535818 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229c031870 con 0x7f22a00719e0 2024-03-06T07:25:47.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a4a85640 1 -- 172.21.15.77:0/457535818 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c031c40 con 0x7f22a00719e0 2024-03-06T07:25:47.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a7ceb640 1 -- 172.21.15.77:0/457535818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 msgr2=0x7f22a00cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/457535818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a00cf0d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f22a00aeed0 tx=0x7f229c02b910 comp rx=0 tx=0).stop 2024-03-06T07:25:47.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a7ceb640 1 -- 172.21.15.77:0/457535818 shutdown_connections 2024-03-06T07:25:47.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/457535818 >> v2:172.21.15.158:3300/0 conn(0x7f22a006c9f0 0x7f22a006ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/457535818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a00cf0d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a7ceb640 1 -- 172.21.15.77:0/457535818 >> 172.21.15.77:0/457535818 conn(0x7f22a006b190 msgr2=0x7f22a006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a7ceb640 1 -- 172.21.15.77:0/457535818 shutdown_connections 2024-03-06T07:25:47.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.152+0000 7f22a7ceb640 1 -- 172.21.15.77:0/457535818 wait complete. 2024-03-06T07:25:47.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.153+0000 7f22a7ceb640 1 Processor -- start 2024-03-06T07:25:47.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.153+0000 7f22a7ceb640 1 -- start start 2024-03-06T07:25:47.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a7ceb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a006c9f0 0x7f22a0145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a7ceb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a7ceb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22a015a220 0x7f22a006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a7ceb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0072530 con 0x7f22a00719e0 2024-03-06T07:25:47.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a7ceb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0072030 con 0x7f22a006c9f0 2024-03-06T07:25:47.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a7ceb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00726b0 con 0x7f22a015a220 2024-03-06T07:25:47.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a5286640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a5286640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a006ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34326/0 (socket says 172.21.15.77:34326) 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a5286640 1 -- 172.21.15.77:0/853907090 learned_addr learned my addr 172.21.15.77:0/853907090 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a5a87640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a006c9f0 0x7f22a0145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.154+0000 7f22a6288640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22a015a220 0x7f22a006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22a5286640 1 -- 172.21.15.77:0/853907090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22a015a220 msgr2=0x7f22a006d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22a5286640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22a015a220 0x7f22a006d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22a5286640 1 -- 172.21.15.77:0/853907090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a006c9f0 msgr2=0x7f22a0145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22a5286640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a006c9f0 0x7f22a0145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22a5286640 1 -- 172.21.15.77:0/853907090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a0153580 con 0x7f22a00719e0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22a5286640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a006ce40 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f229c000c00 tx=0x7f229c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22967fc640 1 -- 172.21.15.77:0/853907090 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c035040 con 0x7f22a00719e0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22967fc640 1 -- 172.21.15.77:0/853907090 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229c030d00 con 0x7f22a00719e0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22967fc640 1 -- 172.21.15.77:0/853907090 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c0456a0 con 0x7f22a00719e0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a00aab30 con 0x7f22a00719e0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.155+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a00ab0f0 con 0x7f22a00719e0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.156+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a00732b0 con 0x7f22a00719e0 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.156+0000 7f22967fc640 1 -- 172.21.15.77:0/853907090 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f229c038020 con 0x7f22a00719e0 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.157+0000 7f22967fc640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f228406b4d0 0x7f228406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.157+0000 7f22967fc640 1 -- 172.21.15.77:0/853907090 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f229c0ac420 con 0x7f22a00719e0 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.157+0000 7f22a5a87640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f228406b4d0 0x7f228406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.157+0000 7f22a5a87640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f228406b4d0 0x7f228406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f22900016e0 tx=0x7f2290009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.161+0000 7f22967fc640 1 -- 172.21.15.77:0/853907090 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f229c07ac50 con 0x7f22a00719e0 2024-03-06T07:25:47.348 INFO:tasks.ceph.mds.d.smithi158.stderr:2024-03-06T07:25:47.347+0000 7f1be5756640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:47.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:47.368+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f22a015d400 con 0x7f22a00719e0 2024-03-06T07:25:48.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.030+0000 7f22967fc640 1 -- 172.21.15.77:0/853907090 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7f229c07ecf0 con 0x7f22a00719e0 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f228406b4d0 msgr2=0x7f228406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f228406b4d0 0x7f228406d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f22900016e0 tx=0x7f2290009040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 msgr2=0x7f22a006ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a006ce40 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f229c000c00 tx=0x7f229c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 shutdown_connections 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f228406b4d0 0x7f228406d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a00719e0 0x7f22a006ce40 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22a015a220 0x7f22a006d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 --2- 172.21.15.77:0/853907090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22a006c9f0 0x7f22a0145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 >> 172.21.15.77:0/853907090 conn(0x7f22a006b190 msgr2=0x7f22a015c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 shutdown_connections 2024-03-06T07:25:48.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.033+0000 7f22a7ceb640 1 -- 172.21.15.77:0/853907090 wait complete. 2024-03-06T07:25:48.046 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:48.046 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.146+0000 7fd0f635c640 1 Processor -- start 2024-03-06T07:25:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.146+0000 7fd0f635c640 1 -- start start 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.146+0000 7fd0f635c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd0f01515e0 0x7fd0f00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.146+0000 7fd0f635c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0f00cd7a0 0x7fd0f00cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.146+0000 7fd0f635c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f0071f40 con 0x7fd0f015b430 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.146+0000 7fd0f635c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f015b8e0 con 0x7fd0f00cd7a0 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.146+0000 7fd0f635c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f0070a70 con 0x7fd0f01515e0 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0f48f9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0f00cd7a0 0x7fd0f00cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0ef7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd0f015b430 0x7fd0f0071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59450/0 (socket says 172.21.15.77:59450) 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0ef7fe640 1 -- 172.21.15.77:0/1451531766 learned_addr learned my addr 172.21.15.77:0/1451531766 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0effff640 1 --2- 172.21.15.77:0/1451531766 >> v2:172.21.15.158:3300/0 conn(0x7fd0f01515e0 0x7fd0f00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0f48f9640 1 -- 172.21.15.77:0/1451531766 >> v2:172.21.15.158:3300/0 conn(0x7fd0f01515e0 msgr2=0x7fd0f00cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0f48f9640 1 --2- 172.21.15.77:0/1451531766 >> v2:172.21.15.158:3300/0 conn(0x7fd0f01515e0 0x7fd0f00cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0f48f9640 1 -- 172.21.15.77:0/1451531766 >> v1:172.21.15.77:6789/0 conn(0x7fd0f015b430 legacy=0x7fd0f0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0f48f9640 1 -- 172.21.15.77:0/1451531766 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f0070d50 con 0x7fd0f00cd7a0 2024-03-06T07:25:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.147+0000 7fd0eeffd640 1 -- 172.21.15.77:0/1451531766 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560500413 0 0) 0x7fd0f0071f40 con 0x7fd0f015b430 2024-03-06T07:25:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f48f9640 1 --2- 172.21.15.77:0/1451531766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f00cd7a0 0x7fd0f00cdb80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd0f006eb50 tx=0x7fd0e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34700f38002ec746 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0eeffd640 1 -- 172.21.15.77:0/1451531766 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8039070 con 0x7fd0f00cd7a0 2024-03-06T07:25:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0eeffd640 1 -- 172.21.15.77:0/1451531766 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e802cc80 con 0x7fd0f00cd7a0 2024-03-06T07:25:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0eeffd640 1 -- 172.21.15.77:0/1451531766 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8034660 con 0x7fd0f00cd7a0 2024-03-06T07:25:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f635c640 1 -- 172.21.15.77:0/1451531766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f00cd7a0 msgr2=0x7fd0f00cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f635c640 1 --2- 172.21.15.77:0/1451531766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f00cd7a0 0x7fd0f00cdb80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd0f006eb50 tx=0x7fd0e802bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f635c640 1 -- 172.21.15.77:0/1451531766 shutdown_connections 2024-03-06T07:25:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f635c640 1 --2- 172.21.15.77:0/1451531766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f00cd7a0 0x7fd0f00cdb80 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f635c640 1 --2- 172.21.15.77:0/1451531766 >> v2:172.21.15.158:3300/0 conn(0x7fd0f01515e0 0x7fd0f00cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f635c640 1 -- 172.21.15.77:0/1451531766 >> 172.21.15.77:0/1451531766 conn(0x7fd0f00cf8a0 msgr2=0x7fd0f00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f635c640 1 -- 172.21.15.77:0/1451531766 shutdown_connections 2024-03-06T07:25:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.148+0000 7fd0f635c640 1 -- 172.21.15.77:0/1451531766 wait complete. 2024-03-06T07:25:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.150+0000 7fd0f635c640 1 Processor -- start 2024-03-06T07:25:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.150+0000 7fd0f635c640 1 -- start start 2024-03-06T07:25:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.150+0000 7fd0f635c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0f00cd7a0 0x7fd0f013ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.150+0000 7fd0f635c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f01515e0 0x7fd0f013d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.150+0000 7fd0f635c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0f015b430 0x7fd0f015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0f635c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00ce0c0 con 0x7fd0f01515e0 2024-03-06T07:25:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0f635c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f0071310 con 0x7fd0f015b430 2024-03-06T07:25:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0f635c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00cc3a0 con 0x7fd0f00cd7a0 2024-03-06T07:25:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f01515e0 0x7fd0f013d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f01515e0 0x7fd0f013d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34346/0 (socket says 172.21.15.77:34346) 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 -- 172.21.15.77:0/2033084987 learned_addr learned my addr 172.21.15.77:0/2033084987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0f48f9640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0f015b430 0x7fd0f015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0effff640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0f00cd7a0 0x7fd0f013ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 -- 172.21.15.77:0/2033084987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0f00cd7a0 msgr2=0x7fd0f013ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0f00cd7a0 0x7fd0f013ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 -- 172.21.15.77:0/2033084987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0f015b430 msgr2=0x7fd0f015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0f015b430 0x7fd0f015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 -- 172.21.15.77:0/2033084987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f013fd60 con 0x7fd0f01515e0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ef7fe640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f01515e0 0x7fd0f013d7d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400c430 tx=0x7fd0e400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ecff9640 1 -- 172.21.15.77:0/2033084987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4017070 con 0x7fd0f01515e0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ecff9640 1 -- 172.21.15.77:0/2033084987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e4005ce0 con 0x7fd0f01515e0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f0140040 con 0x7fd0f01515e0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0ecff9640 1 -- 172.21.15.77:0/2033084987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4007310 con 0x7fd0f01515e0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.151+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f0140600 con 0x7fd0f01515e0 2024-03-06T07:25:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.152+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f0068840 con 0x7fd0f01515e0 2024-03-06T07:25:48.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.153+0000 7fd0ecff9640 1 -- 172.21.15.77:0/2033084987 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0e401a070 con 0x7fd0f01515e0 2024-03-06T07:25:48.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.153+0000 7fd0ecff9640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.153+0000 7fd0ecff9640 1 -- 172.21.15.77:0/2033084987 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0e4089540 con 0x7fd0f01515e0 2024-03-06T07:25:48.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.153+0000 7fd0effff640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.154+0000 7fd0effff640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd0f015e6f0 tx=0x7fd0e0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.157+0000 7fd0ecff9640 1 -- 172.21.15.77:0/2033084987 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0e4057d70 con 0x7fd0f01515e0 2024-03-06T07:25:48.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.369+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0f014f730 con 0x7fd0f01515e0 2024-03-06T07:25:48.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.369+0000 7fd0ecff9640 1 -- 172.21.15.77:0/2033084987 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd0f014f730 con 0x7fd0f01515e0 2024-03-06T07:25:48.371 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:48.371 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3408486989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3408486989},{"type":"v1","addr":"172.21.15.158:6835","nonce":3408486989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/324866984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":324866984},{"type":"v1","addr":"172.21.15.77:6836","nonce":324866984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2426038221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2426038221},{"type":"v1","addr":"172.21.15.77:6837","nonce":2426038221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:44.712557+0000","modified":"2024-03-06T07:25:47.370110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6930},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6930":{"gid":6930,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/3775835533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3775835533},{"type":"v1","addr":"172.21.15.158:6834","nonce":3775835533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6930,"qdb_cluster":[6930]},"id":6}]} 2024-03-06T07:25:48.372 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2024-03-06T07:25:48.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd0dc06b4d0 msgr2=0x7fd0dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd0f015e6f0 tx=0x7fd0e0009190 comp rx=0 tx=0).stop 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f01515e0 msgr2=0x7fd0f013d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f01515e0 0x7fd0f013d7d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400c430 tx=0x7fd0e400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 shutdown_connections 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f01515e0 0x7fd0f013d7d0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd0f015b430 0x7fd0f015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 --2- 172.21.15.77:0/2033084987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0f00cd7a0 0x7fd0f013ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.371+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 >> 172.21.15.77:0/2033084987 conn(0x7fd0f00cf8a0 msgr2=0x7fd0f014f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.372+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 shutdown_connections 2024-03-06T07:25:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.372+0000 7fd0f635c640 1 -- 172.21.15.77:0/2033084987 wait complete. 2024-03-06T07:25:48.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.482+0000 7fc619bac640 1 Processor -- start 2024-03-06T07:25:48.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.482+0000 7fc619bac640 1 -- start start 2024-03-06T07:25:48.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc619bac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc61415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc619bac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc61415bf30 0x7fc61415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc619bac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61415f2d0 con 0x7fc61413dc10 2024-03-06T07:25:48.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc619bac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6140cf3f0 con 0x7fc61415bf30 2024-03-06T07:25:48.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc619bac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61415b210 con 0x7fc6140b3060 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc6137fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc61413dc10 0x7fc6140b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58062/0 (socket says 172.21.15.77:58062) 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc6137fe640 1 -- 172.21.15.77:0/4038375266 learned_addr learned my addr 172.21.15.77:0/4038375266 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc613fff640 1 --2- 172.21.15.77:0/4038375266 >> v2:172.21.15.158:3300/0 conn(0x7fc61415bf30 0x7fc61415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc612ffd640 1 --2- 172.21.15.77:0/4038375266 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc61415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc6127fc640 1 -- 172.21.15.77:0/4038375266 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517244519 0 0) 0x7fc61415f2d0 con 0x7fc61413dc10 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc6127fc640 1 -- 172.21.15.77:0/4038375266 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5fc003520 con 0x7fc61413dc10 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc612ffd640 1 -- 172.21.15.77:0/4038375266 >> v2:172.21.15.158:3300/0 conn(0x7fc61415bf30 msgr2=0x7fc61415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc612ffd640 1 --2- 172.21.15.77:0/4038375266 >> v2:172.21.15.158:3300/0 conn(0x7fc61415bf30 0x7fc61415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc612ffd640 1 -- 172.21.15.77:0/4038375266 >> v1:172.21.15.77:6789/0 conn(0x7fc61413dc10 legacy=0x7fc6140b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc612ffd640 1 -- 172.21.15.77:0/4038375266 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61415f580 con 0x7fc6140b3060 2024-03-06T07:25:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.483+0000 7fc613fff640 1 --2- 172.21.15.77:0/4038375266 >> v2:172.21.15.158:3300/0 conn(0x7fc61415bf30 0x7fc61415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.484+0000 7fc612ffd640 1 --2- 172.21.15.77:0/4038375266 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc61415e850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc6040097a0 tx=0x7fc60402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9db5754df17d3fdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.484+0000 7fc6127fc640 1 -- 172.21.15.77:0/4038375266 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6040315a0 con 0x7fc6140b3060 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.484+0000 7fc6127fc640 1 -- 172.21.15.77:0/4038375266 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc604037070 con 0x7fc6140b3060 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.484+0000 7fc6127fc640 1 -- 172.21.15.77:0/4038375266 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604031960 con 0x7fc6140b3060 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.485+0000 7fc619bac640 1 -- 172.21.15.77:0/4038375266 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 msgr2=0x7fc61415e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.485+0000 7fc619bac640 1 --2- 172.21.15.77:0/4038375266 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc61415e850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc6040097a0 tx=0x7fc60402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.485+0000 7fc619bac640 1 -- 172.21.15.77:0/4038375266 shutdown_connections 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.485+0000 7fc619bac640 1 --2- 172.21.15.77:0/4038375266 >> v2:172.21.15.158:3300/0 conn(0x7fc61415bf30 0x7fc61415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.485+0000 7fc619bac640 1 --2- 172.21.15.77:0/4038375266 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc61415e850 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.485+0000 7fc619bac640 1 -- 172.21.15.77:0/4038375266 >> 172.21.15.77:0/4038375266 conn(0x7fc6140cf8a0 msgr2=0x7fc6140cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.485+0000 7fc619bac640 1 -- 172.21.15.77:0/4038375266 shutdown_connections 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.485+0000 7fc619bac640 1 -- 172.21.15.77:0/4038375266 wait complete. 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.486+0000 7fc619bac640 1 Processor -- start 2024-03-06T07:25:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.486+0000 7fc619bac640 1 -- start start 2024-03-06T07:25:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc619bac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc614141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc619bac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61413dc10 0x7fc6141549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc619bac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc61415bf30 0x7fc61413cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc619bac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61413dff0 con 0x7fc61413dc10 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc619bac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61415b730 con 0x7fc61415bf30 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc619bac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140708d0 con 0x7fc6140b3060 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61413dc10 0x7fc6141549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61413dc10 0x7fc6141549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52058/0 (socket says 172.21.15.77:52058) 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 -- 172.21.15.77:0/4124667248 learned_addr learned my addr 172.21.15.77:0/4124667248 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc6137fe640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc614141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc613fff640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc61415bf30 0x7fc61413cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 -- 172.21.15.77:0/4124667248 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 msgr2=0x7fc614141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc614141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 -- 172.21.15.77:0/4124667248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc61415bf30 msgr2=0x7fc61413cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc61415bf30 0x7fc61413cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 -- 172.21.15.77:0/4124667248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61413f7c0 con 0x7fc61413dc10 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc612ffd640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61413dc10 0x7fc6141549d0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc604034040 tx=0x7fc6040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc5fbfff640 1 -- 172.21.15.77:0/4124667248 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604032630 con 0x7fc61413dc10 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc5fbfff640 1 -- 172.21.15.77:0/4124667248 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6040327f0 con 0x7fc61413dc10 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.487+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61413faa0 con 0x7fc61413dc10 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.488+0000 7fc5fbfff640 1 -- 172.21.15.77:0/4124667248 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604032be0 con 0x7fc61413dc10 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.488+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc614140060 con 0x7fc61413dc10 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.488+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61404f330 con 0x7fc61413dc10 2024-03-06T07:25:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.489+0000 7fc5fbfff640 1 -- 172.21.15.77:0/4124667248 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc60404e430 con 0x7fc61413dc10 2024-03-06T07:25:48.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.489+0000 7fc5fbfff640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc60006b4d0 0x7fc60006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.489+0000 7fc5fbfff640 1 -- 172.21.15.77:0/4124667248 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc60402f030 con 0x7fc61413dc10 2024-03-06T07:25:48.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.489+0000 7fc6137fe640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc60006b4d0 0x7fc60006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.490+0000 7fc6137fe640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc60006b4d0 0x7fc60006d990 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc608005880 tx=0x7fc608009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.494+0000 7fc5fbfff640 1 -- 172.21.15.77:0/4124667248 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc604040070 con 0x7fc61413dc10 2024-03-06T07:25:48.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.665+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc614071df0 con 0x7fc61413dc10 2024-03-06T07:25:48.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.666+0000 7fc5fbfff640 1 -- 172.21.15.77:0/4124667248 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16648 (secure 0 0 0) 0x7fc60407acb0 con 0x7fc61413dc10 2024-03-06T07:25:48.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:48.669 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":80,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:46.362962+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:42.937014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:43.684499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:48.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc60006b4d0 msgr2=0x7fc60006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc60006b4d0 0x7fc60006d990 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc608005880 tx=0x7fc608009040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61413dc10 msgr2=0x7fc6141549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61413dc10 0x7fc6141549d0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc604034040 tx=0x7fc6040046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 shutdown_connections 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc60006b4d0 0x7fc60006d990 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc61413dc10 0x7fc6141549d0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc61415bf30 0x7fc61413cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 --2- 172.21.15.77:0/4124667248 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc6140b3060 0x7fc614141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 >> 172.21.15.77:0/4124667248 conn(0x7fc6140cf8a0 msgr2=0x7fc6140af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.668+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 shutdown_connections 2024-03-06T07:25:48.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.669+0000 7fc619bac640 1 -- 172.21.15.77:0/4124667248 wait complete. 2024-03-06T07:25:48.681 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:48.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.780+0000 7fb1fd239640 1 Processor -- start 2024-03-06T07:25:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.780+0000 7fb1fd239640 1 -- start start 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.780+0000 7fb1fd239640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1f81515e0 0x7fb1f80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.780+0000 7fb1fd239640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 0x7fb1f80cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.780+0000 7fb1fd239640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f8071f40 con 0x7fb1f815b430 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.780+0000 7fb1fd239640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f815b8e0 con 0x7fb1f81515e0 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.780+0000 7fb1fd239640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8070a70 con 0x7fb1f80cd7a0 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f6d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1f81515e0 0x7fb1f80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb1f815b430 0x7fb1f8071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58076/0 (socket says 172.21.15.77:58076) 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f659c640 1 -- 172.21.15.77:0/2549361414 learned_addr learned my addr 172.21.15.77:0/2549361414 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f759e640 1 --2- 172.21.15.77:0/2549361414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 0x7fb1f80cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f6d9d640 1 -- 172.21.15.77:0/2549361414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 msgr2=0x7fb1f80cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f6d9d640 1 --2- 172.21.15.77:0/2549361414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 0x7fb1f80cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f6d9d640 1 -- 172.21.15.77:0/2549361414 >> v1:172.21.15.77:6789/0 conn(0x7fb1f815b430 legacy=0x7fb1f8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f6d9d640 1 -- 172.21.15.77:0/2549361414 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8070d50 con 0x7fb1f81515e0 2024-03-06T07:25:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.781+0000 7fb1f5d9b640 1 -- 172.21.15.77:0/2549361414 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859136206 0 0) 0x7fb1f8071f40 con 0x7fb1f815b430 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1f6d9d640 1 --2- 172.21.15.77:0/2549361414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f81515e0 0x7fb1f80cce40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80097a0 tx=0x7fb1e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef341b34cf33d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1f5d9b640 1 -- 172.21.15.77:0/2549361414 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8035020 con 0x7fb1f81515e0 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1f5d9b640 1 -- 172.21.15.77:0/2549361414 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e8031830 con 0x7fb1f81515e0 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1f5d9b640 1 -- 172.21.15.77:0/2549361414 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8031c00 con 0x7fb1f81515e0 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1fd239640 1 -- 172.21.15.77:0/2549361414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f81515e0 msgr2=0x7fb1f80cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1fd239640 1 --2- 172.21.15.77:0/2549361414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f81515e0 0x7fb1f80cce40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80097a0 tx=0x7fb1e802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1fd239640 1 -- 172.21.15.77:0/2549361414 shutdown_connections 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1fd239640 1 --2- 172.21.15.77:0/2549361414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 0x7fb1f80cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1fd239640 1 --2- 172.21.15.77:0/2549361414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f81515e0 0x7fb1f80cce40 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1fd239640 1 -- 172.21.15.77:0/2549361414 >> 172.21.15.77:0/2549361414 conn(0x7fb1f80cf8a0 msgr2=0x7fb1f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1fd239640 1 -- 172.21.15.77:0/2549361414 shutdown_connections 2024-03-06T07:25:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.782+0000 7fb1fd239640 1 -- 172.21.15.77:0/2549361414 wait complete. 2024-03-06T07:25:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.784+0000 7fb1fd239640 1 Processor -- start 2024-03-06T07:25:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.784+0000 7fb1fd239640 1 -- start start 2024-03-06T07:25:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.784+0000 7fb1fd239640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 0x7fb1f8143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1fd239640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1f81515e0 0x7fb1f815dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1fd239640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f815b430 0x7fb1f8145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1fd239640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f80ce0f0 con 0x7fb1f815b430 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1fd239640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8071310 con 0x7fb1f81515e0 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1fd239640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f80cc3a0 con 0x7fb1f80cd7a0 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f815b430 0x7fb1f8145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f815b430 0x7fb1f8145d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52066/0 (socket says 172.21.15.77:52066) 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 -- 172.21.15.77:0/3424450884 learned_addr learned my addr 172.21.15.77:0/3424450884 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f6d9d640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 0x7fb1f8143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f659c640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1f81515e0 0x7fb1f815dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 -- 172.21.15.77:0/3424450884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 msgr2=0x7fb1f8143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 0x7fb1f8143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 -- 172.21.15.77:0/3424450884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1f81515e0 msgr2=0x7fb1f815dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1f81515e0 0x7fb1f815dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 -- 172.21.15.77:0/3424450884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f815eac0 con 0x7fb1f815b430 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1f759e640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f815b430 0x7fb1f8145d60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009dd0 tx=0x7fb1f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1df7fe640 1 -- 172.21.15.77:0/3424450884 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0018020 con 0x7fb1f815b430 2024-03-06T07:25:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1df7fe640 1 -- 172.21.15.77:0/3424450884 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f0014480 con 0x7fb1f815b430 2024-03-06T07:25:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f81f2a80 con 0x7fb1f815b430 2024-03-06T07:25:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1df7fe640 1 -- 172.21.15.77:0/3424450884 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0014870 con 0x7fb1f815b430 2024-03-06T07:25:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.785+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f8147490 con 0x7fb1f815b430 2024-03-06T07:25:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.786+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f8068840 con 0x7fb1f815b430 2024-03-06T07:25:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.787+0000 7fb1df7fe640 1 -- 172.21.15.77:0/3424450884 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1f0005d10 con 0x7fb1f815b430 2024-03-06T07:25:48.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.787+0000 7fb1df7fe640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1e406b4d0 0x7fb1e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.787+0000 7fb1df7fe640 1 -- 172.21.15.77:0/3424450884 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1f008abd0 con 0x7fb1f815b430 2024-03-06T07:25:48.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.787+0000 7fb1f6d9d640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1e406b4d0 0x7fb1e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.788+0000 7fb1f6d9d640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1e406b4d0 0x7fb1e406d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8036000 tx=0x7fb1e803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.791+0000 7fb1df7fe640 1 -- 172.21.15.77:0/3424450884 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1f0022070 con 0x7fb1f815b430 2024-03-06T07:25:48.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.962+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1f81476d0 con 0x7fb1e406b4d0 2024-03-06T07:25:48.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.968+0000 7fb1df7fe640 1 -- 172.21.15.77:0/3424450884 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252074 (secure 0 0 0) 0x7fb1e80888c0 con 0x7fb1e406b4d0 2024-03-06T07:25:48.972 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:48.974 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1e406b4d0 msgr2=0x7fb1e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1e406b4d0 0x7fb1e406d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8036000 tx=0x7fb1e803d040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f815b430 msgr2=0x7fb1f8145d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f815b430 0x7fb1f8145d60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009dd0 tx=0x7fb1f000c490 comp rx=0 tx=0).stop 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 shutdown_connections 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1e406b4d0 0x7fb1e406d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f815b430 0x7fb1f8145d60 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1f81515e0 0x7fb1f815dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 --2- 172.21.15.77:0/3424450884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1f80cd7a0 0x7fb1f8143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 >> 172.21.15.77:0/3424450884 conn(0x7fb1f80cf8a0 msgr2=0x7fb1f8071710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 shutdown_connections 2024-03-06T07:25:48.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:48.971+0000 7fb1fd239640 1 -- 172.21.15.77:0/3424450884 wait complete. 2024-03-06T07:25:48.984 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-03-06T07:25:47.424713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":275016,"kb_used_data":8904,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754699704,"statfs":{"total":773094113280,"available":772812496896,"internally_reserved":0,"allocated":9117696,"data_stored":4509460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.068210"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366721+0000","last_change":"2024-03-06T07:25:45.377105+0000","last_active":"2024-03-06T07:25:46.366721+0000","last_peered":"2024-03-06T07:25:46.366721+0000","last_clean":"2024-03-06T07:25:46.366721+0000","last_became_active":"2024-03-06T07:25:45.376811+0000","last_became_peered":"2024-03-06T07:25:45.376811+0000","last_unstale":"2024-03-06T07:25:46.366721+0000","last_undegraded":"2024-03-06T07:25:46.366721+0000","last_fullsized":"2024-03-06T07:25:46.366721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:04.143346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366321+0000","last_change":"2024-03-06T07:25:45.378496+0000","last_active":"2024-03-06T07:25:46.366321+0000","last_peered":"2024-03-06T07:25:46.366321+0000","last_clean":"2024-03-06T07:25:46.366321+0000","last_became_active":"2024-03-06T07:25:45.378247+0000","last_became_peered":"2024-03-06T07:25:45.378247+0000","last_unstale":"2024-03-06T07:25:46.366321+0000","last_undegraded":"2024-03-06T07:25:46.366321+0000","last_fullsized":"2024-03-06T07:25:46.366321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:17.336349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430124+0000","last_change":"2024-03-06T07:25:45.381750+0000","last_active":"2024-03-06T07:25:46.430124+0000","last_peered":"2024-03-06T07:25:46.430124+0000","last_clean":"2024-03-06T07:25:46.430124+0000","last_became_active":"2024-03-06T07:25:45.381494+0000","last_became_peered":"2024-03-06T07:25:45.381494+0000","last_unstale":"2024-03-06T07:25:46.430124+0000","last_undegraded":"2024-03-06T07:25:46.430124+0000","last_fullsized":"2024-03-06T07:25:46.430124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:55.190667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430453+0000","last_change":"2024-03-06T07:25:45.390449+0000","last_active":"2024-03-06T07:25:46.430453+0000","last_peered":"2024-03-06T07:25:46.430453+0000","last_clean":"2024-03-06T07:25:46.430453+0000","last_became_active":"2024-03-06T07:25:45.390161+0000","last_became_peered":"2024-03-06T07:25:45.390161+0000","last_unstale":"2024-03-06T07:25:46.430453+0000","last_undegraded":"2024-03-06T07:25:46.430453+0000","last_fullsized":"2024-03-06T07:25:46.430453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:47.519263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366634+0000","last_change":"2024-03-06T07:25:45.381238+0000","last_active":"2024-03-06T07:25:46.366634+0000","last_peered":"2024-03-06T07:25:46.366634+0000","last_clean":"2024-03-06T07:25:46.366634+0000","last_became_active":"2024-03-06T07:25:45.380981+0000","last_became_peered":"2024-03-06T07:25:45.380981+0000","last_unstale":"2024-03-06T07:25:46.366634+0000","last_undegraded":"2024-03-06T07:25:46.366634+0000","last_fullsized":"2024-03-06T07:25:46.366634+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:13.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429935+0000","last_change":"2024-03-06T07:25:45.382092+0000","last_active":"2024-03-06T07:25:46.429935+0000","last_peered":"2024-03-06T07:25:46.429935+0000","last_clean":"2024-03-06T07:25:46.429935+0000","last_became_active":"2024-03-06T07:25:45.381760+0000","last_became_peered":"2024-03-06T07:25:45.381760+0000","last_unstale":"2024-03-06T07:25:46.429935+0000","last_undegraded":"2024-03-06T07:25:46.429935+0000","last_fullsized":"2024-03-06T07:25:46.429935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:32.549564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429994+0000","last_change":"2024-03-06T07:25:45.378408+0000","last_active":"2024-03-06T07:25:46.429994+0000","last_peered":"2024-03-06T07:25:46.429994+0000","last_clean":"2024-03-06T07:25:46.429994+0000","last_became_active":"2024-03-06T07:25:45.378171+0000","last_became_peered":"2024-03-06T07:25:45.378171+0000","last_unstale":"2024-03-06T07:25:46.429994+0000","last_undegraded":"2024-03-06T07:25:46.429994+0000","last_fullsized":"2024-03-06T07:25:46.429994+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:39.546940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430311+0000","last_change":"2024-03-06T07:25:45.387654+0000","last_active":"2024-03-06T07:25:46.430311+0000","last_peered":"2024-03-06T07:25:46.430311+0000","last_clean":"2024-03-06T07:25:46.430311+0000","last_became_active":"2024-03-06T07:25:45.387359+0000","last_became_peered":"2024-03-06T07:25:45.387359+0000","last_unstale":"2024-03-06T07:25:46.430311+0000","last_undegraded":"2024-03-06T07:25:46.430311+0000","last_fullsized":"2024-03-06T07:25:46.430311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:17.123482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430450+0000","last_change":"2024-03-06T07:25:45.383579+0000","last_active":"2024-03-06T07:25:46.430450+0000","last_peered":"2024-03-06T07:25:46.430450+0000","last_clean":"2024-03-06T07:25:46.430450+0000","last_became_active":"2024-03-06T07:25:45.383307+0000","last_became_peered":"2024-03-06T07:25:45.383307+0000","last_unstale":"2024-03-06T07:25:46.430450+0000","last_undegraded":"2024-03-06T07:25:46.430450+0000","last_fullsized":"2024-03-06T07:25:46.430450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:41.188810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366507+0000","last_change":"2024-03-06T07:25:45.388827+0000","last_active":"2024-03-06T07:25:46.366507+0000","last_peered":"2024-03-06T07:25:46.366507+0000","last_clean":"2024-03-06T07:25:46.366507+0000","last_became_active":"2024-03-06T07:25:45.386617+0000","last_became_peered":"2024-03-06T07:25:45.386617+0000","last_unstale":"2024-03-06T07:25:46.366507+0000","last_undegraded":"2024-03-06T07:25:46.366507+0000","last_fullsized":"2024-03-06T07:25:46.366507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:17.123482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366158+0000","last_change":"2024-03-06T07:25:45.382537+0000","last_active":"2024-03-06T07:25:46.366158+0000","last_peered":"2024-03-06T07:25:46.366158+0000","last_clean":"2024-03-06T07:25:46.366158+0000","last_became_active":"2024-03-06T07:25:45.381404+0000","last_became_peered":"2024-03-06T07:25:45.381404+0000","last_unstale":"2024-03-06T07:25:46.366158+0000","last_undegraded":"2024-03-06T07:25:46.366158+0000","last_fullsized":"2024-03-06T07:25:46.366158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:35.891388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366240+0000","last_change":"2024-03-06T07:25:45.378553+0000","last_active":"2024-03-06T07:25:46.366240+0000","last_peered":"2024-03-06T07:25:46.366240+0000","last_clean":"2024-03-06T07:25:46.366240+0000","last_became_active":"2024-03-06T07:25:45.378284+0000","last_became_peered":"2024-03-06T07:25:45.378284+0000","last_unstale":"2024-03-06T07:25:46.366240+0000","last_undegraded":"2024-03-06T07:25:46.366240+0000","last_fullsized":"2024-03-06T07:25:46.366240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:32.549564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429865+0000","last_change":"2024-03-06T07:25:45.378156+0000","last_active":"2024-03-06T07:25:46.429865+0000","last_peered":"2024-03-06T07:25:46.429865+0000","last_clean":"2024-03-06T07:25:46.429865+0000","last_became_active":"2024-03-06T07:25:45.377881+0000","last_became_peered":"2024-03-06T07:25:45.377881+0000","last_unstale":"2024-03-06T07:25:46.429865+0000","last_undegraded":"2024-03-06T07:25:46.429865+0000","last_fullsized":"2024-03-06T07:25:46.429865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:38.059664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430203+0000","last_change":"2024-03-06T07:25:45.386810+0000","last_active":"2024-03-06T07:25:46.430203+0000","last_peered":"2024-03-06T07:25:46.430203+0000","last_clean":"2024-03-06T07:25:46.430203+0000","last_became_active":"2024-03-06T07:25:45.386493+0000","last_became_peered":"2024-03-06T07:25:45.386493+0000","last_unstale":"2024-03-06T07:25:46.430203+0000","last_undegraded":"2024-03-06T07:25:46.430203+0000","last_fullsized":"2024-03-06T07:25:46.430203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:17.787221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429782+0000","last_change":"2024-03-06T07:25:45.381569+0000","last_active":"2024-03-06T07:25:46.429782+0000","last_peered":"2024-03-06T07:25:46.429782+0000","last_clean":"2024-03-06T07:25:46.429782+0000","last_became_active":"2024-03-06T07:25:45.380998+0000","last_became_peered":"2024-03-06T07:25:45.380998+0000","last_unstale":"2024-03-06T07:25:46.429782+0000","last_undegraded":"2024-03-06T07:25:46.429782+0000","last_fullsized":"2024-03-06T07:25:46.429782+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:56.002153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366298+0000","last_change":"2024-03-06T07:25:45.382381+0000","last_active":"2024-03-06T07:25:46.366298+0000","last_peered":"2024-03-06T07:25:46.366298+0000","last_clean":"2024-03-06T07:25:46.366298+0000","last_became_active":"2024-03-06T07:25:45.382104+0000","last_became_peered":"2024-03-06T07:25:45.382104+0000","last_unstale":"2024-03-06T07:25:46.366298+0000","last_undegraded":"2024-03-06T07:25:46.366298+0000","last_fullsized":"2024-03-06T07:25:46.366298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:24.383441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366515+0000","last_change":"2024-03-06T07:25:45.386120+0000","last_active":"2024-03-06T07:25:46.366515+0000","last_peered":"2024-03-06T07:25:46.366515+0000","last_clean":"2024-03-06T07:25:46.366515+0000","last_became_active":"2024-03-06T07:25:45.385846+0000","last_became_peered":"2024-03-06T07:25:45.385846+0000","last_unstale":"2024-03-06T07:25:46.366515+0000","last_undegraded":"2024-03-06T07:25:46.366515+0000","last_fullsized":"2024-03-06T07:25:46.366515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:55.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429905+0000","last_change":"2024-03-06T07:25:45.389075+0000","last_active":"2024-03-06T07:25:46.429905+0000","last_peered":"2024-03-06T07:25:46.429905+0000","last_clean":"2024-03-06T07:25:46.429905+0000","last_became_active":"2024-03-06T07:25:45.388852+0000","last_became_peered":"2024-03-06T07:25:45.388852+0000","last_unstale":"2024-03-06T07:25:46.429905+0000","last_undegraded":"2024-03-06T07:25:46.429905+0000","last_fullsized":"2024-03-06T07:25:46.429905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:04.795323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.372071+0000","last_change":"2024-03-06T07:25:45.381601+0000","last_active":"2024-03-06T07:25:46.372071+0000","last_peered":"2024-03-06T07:25:46.372071+0000","last_clean":"2024-03-06T07:25:46.372071+0000","last_became_active":"2024-03-06T07:25:45.381326+0000","last_became_peered":"2024-03-06T07:25:45.381326+0000","last_unstale":"2024-03-06T07:25:46.372071+0000","last_undegraded":"2024-03-06T07:25:46.372071+0000","last_fullsized":"2024-03-06T07:25:46.372071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:05.816904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430659+0000","last_change":"2024-03-06T07:25:45.386697+0000","last_active":"2024-03-06T07:25:46.430659+0000","last_peered":"2024-03-06T07:25:46.430659+0000","last_clean":"2024-03-06T07:25:46.430659+0000","last_became_active":"2024-03-06T07:25:45.386422+0000","last_became_peered":"2024-03-06T07:25:45.386422+0000","last_unstale":"2024-03-06T07:25:46.430659+0000","last_undegraded":"2024-03-06T07:25:46.430659+0000","last_fullsized":"2024-03-06T07:25:46.430659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:46.432954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430108+0000","last_change":"2024-03-06T07:25:45.384961+0000","last_active":"2024-03-06T07:25:46.430108+0000","last_peered":"2024-03-06T07:25:46.430108+0000","last_clean":"2024-03-06T07:25:46.430108+0000","last_became_active":"2024-03-06T07:25:45.384717+0000","last_became_peered":"2024-03-06T07:25:45.384717+0000","last_unstale":"2024-03-06T07:25:46.430108+0000","last_undegraded":"2024-03-06T07:25:46.430108+0000","last_fullsized":"2024-03-06T07:25:46.430108+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:41.419541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365866+0000","last_change":"2024-03-06T07:25:45.382921+0000","last_active":"2024-03-06T07:25:46.365866+0000","last_peered":"2024-03-06T07:25:46.365866+0000","last_clean":"2024-03-06T07:25:46.365866+0000","last_became_active":"2024-03-06T07:25:45.382556+0000","last_became_peered":"2024-03-06T07:25:45.382556+0000","last_unstale":"2024-03-06T07:25:46.365866+0000","last_undegraded":"2024-03-06T07:25:46.365866+0000","last_fullsized":"2024-03-06T07:25:46.365866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.283759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366944+0000","last_change":"2024-03-06T07:25:45.379716+0000","last_active":"2024-03-06T07:25:46.366944+0000","last_peered":"2024-03-06T07:25:46.366944+0000","last_clean":"2024-03-06T07:25:46.366944+0000","last_became_active":"2024-03-06T07:25:45.379018+0000","last_became_peered":"2024-03-06T07:25:45.379018+0000","last_unstale":"2024-03-06T07:25:46.366944+0000","last_undegraded":"2024-03-06T07:25:46.366944+0000","last_fullsized":"2024-03-06T07:25:46.366944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:23.547706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429843+0000","last_change":"2024-03-06T07:25:45.385565+0000","last_active":"2024-03-06T07:25:46.429843+0000","last_peered":"2024-03-06T07:25:46.429843+0000","last_clean":"2024-03-06T07:25:46.429843+0000","last_became_active":"2024-03-06T07:25:45.385323+0000","last_became_peered":"2024-03-06T07:25:45.385323+0000","last_unstale":"2024-03-06T07:25:46.429843+0000","last_undegraded":"2024-03-06T07:25:46.429843+0000","last_fullsized":"2024-03-06T07:25:46.429843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:21.203793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366040+0000","last_change":"2024-03-06T07:25:45.378472+0000","last_active":"2024-03-06T07:25:46.366040+0000","last_peered":"2024-03-06T07:25:46.366040+0000","last_clean":"2024-03-06T07:25:46.366040+0000","last_became_active":"2024-03-06T07:25:45.377641+0000","last_became_peered":"2024-03-06T07:25:45.377641+0000","last_unstale":"2024-03-06T07:25:46.366040+0000","last_undegraded":"2024-03-06T07:25:46.366040+0000","last_fullsized":"2024-03-06T07:25:46.366040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:10.793581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430204+0000","last_change":"2024-03-06T07:25:45.390307+0000","last_active":"2024-03-06T07:25:46.430204+0000","last_peered":"2024-03-06T07:25:46.430204+0000","last_clean":"2024-03-06T07:25:46.430204+0000","last_became_active":"2024-03-06T07:25:45.390045+0000","last_became_peered":"2024-03-06T07:25:45.390045+0000","last_unstale":"2024-03-06T07:25:46.430204+0000","last_undegraded":"2024-03-06T07:25:46.430204+0000","last_fullsized":"2024-03-06T07:25:46.430204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:37.643558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366621+0000","last_change":"2024-03-06T07:25:45.379268+0000","last_active":"2024-03-06T07:25:46.366621+0000","last_peered":"2024-03-06T07:25:46.366621+0000","last_clean":"2024-03-06T07:25:46.366621+0000","last_became_active":"2024-03-06T07:25:45.378972+0000","last_became_peered":"2024-03-06T07:25:45.378972+0000","last_unstale":"2024-03-06T07:25:46.366621+0000","last_undegraded":"2024-03-06T07:25:46.366621+0000","last_fullsized":"2024-03-06T07:25:46.366621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:32.549564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366399+0000","last_change":"2024-03-06T07:25:45.378452+0000","last_active":"2024-03-06T07:25:46.366399+0000","last_peered":"2024-03-06T07:25:46.366399+0000","last_clean":"2024-03-06T07:25:46.366399+0000","last_became_active":"2024-03-06T07:25:45.378147+0000","last_became_peered":"2024-03-06T07:25:45.378147+0000","last_unstale":"2024-03-06T07:25:46.366399+0000","last_undegraded":"2024-03-06T07:25:46.366399+0000","last_fullsized":"2024-03-06T07:25:46.366399+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:24.364522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365985+0000","last_change":"2024-03-06T07:25:45.375684+0000","last_active":"2024-03-06T07:25:46.365985+0000","last_peered":"2024-03-06T07:25:46.365985+0000","last_clean":"2024-03-06T07:25:46.365985+0000","last_became_active":"2024-03-06T07:25:45.375405+0000","last_became_peered":"2024-03-06T07:25:45.375405+0000","last_unstale":"2024-03-06T07:25:46.365985+0000","last_undegraded":"2024-03-06T07:25:46.365985+0000","last_fullsized":"2024-03-06T07:25:46.365985+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:26.625115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365938+0000","last_change":"2024-03-06T07:25:45.379285+0000","last_active":"2024-03-06T07:25:46.365938+0000","last_peered":"2024-03-06T07:25:46.365938+0000","last_clean":"2024-03-06T07:25:46.365938+0000","last_became_active":"2024-03-06T07:25:45.378923+0000","last_became_peered":"2024-03-06T07:25:45.378923+0000","last_unstale":"2024-03-06T07:25:46.365938+0000","last_undegraded":"2024-03-06T07:25:46.365938+0000","last_fullsized":"2024-03-06T07:25:46.365938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:18.896513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366464+0000","last_change":"2024-03-06T07:25:45.383967+0000","last_active":"2024-03-06T07:25:46.366464+0000","last_peered":"2024-03-06T07:25:46.366464+0000","last_clean":"2024-03-06T07:25:46.366464+0000","last_became_active":"2024-03-06T07:25:45.383673+0000","last_became_peered":"2024-03-06T07:25:45.383673+0000","last_unstale":"2024-03-06T07:25:46.366464+0000","last_undegraded":"2024-03-06T07:25:46.366464+0000","last_fullsized":"2024-03-06T07:25:46.366464+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:35.368146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366728+0000","last_change":"2024-03-06T07:25:45.374749+0000","last_active":"2024-03-06T07:25:46.366728+0000","last_peered":"2024-03-06T07:25:46.366728+0000","last_clean":"2024-03-06T07:25:46.366728+0000","last_became_active":"2024-03-06T07:25:45.374436+0000","last_became_peered":"2024-03-06T07:25:45.374436+0000","last_unstale":"2024-03-06T07:25:46.366728+0000","last_undegraded":"2024-03-06T07:25:46.366728+0000","last_fullsized":"2024-03-06T07:25:46.366728+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:13.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367344+0000","last_change":"2024-03-06T07:25:45.382541+0000","last_active":"2024-03-06T07:25:46.367344+0000","last_peered":"2024-03-06T07:25:46.367344+0000","last_clean":"2024-03-06T07:25:46.367344+0000","last_became_active":"2024-03-06T07:25:45.382228+0000","last_became_peered":"2024-03-06T07:25:45.382228+0000","last_unstale":"2024-03-06T07:25:46.367344+0000","last_undegraded":"2024-03-06T07:25:46.367344+0000","last_fullsized":"2024-03-06T07:25:46.367344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:33.664581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366983+0000","last_change":"2024-03-06T07:25:44.380973+0000","last_active":"2024-03-06T07:25:46.366983+0000","last_peered":"2024-03-06T07:25:46.366983+0000","last_clean":"2024-03-06T07:25:46.366983+0000","last_became_active":"2024-03-06T07:25:44.380687+0000","last_became_peered":"2024-03-06T07:25:44.380687+0000","last_unstale":"2024-03-06T07:25:46.366983+0000","last_undegraded":"2024-03-06T07:25:46.366983+0000","last_fullsized":"2024-03-06T07:25:46.366983+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:01.662584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367370+0000","last_change":"2024-03-06T07:25:45.389284+0000","last_active":"2024-03-06T07:25:46.367370+0000","last_peered":"2024-03-06T07:25:46.367370+0000","last_clean":"2024-03-06T07:25:46.367370+0000","last_became_active":"2024-03-06T07:25:45.385958+0000","last_became_peered":"2024-03-06T07:25:45.385958+0000","last_unstale":"2024-03-06T07:25:46.367370+0000","last_undegraded":"2024-03-06T07:25:46.367370+0000","last_fullsized":"2024-03-06T07:25:46.367370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:08.205197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367427+0000","last_change":"2024-03-06T07:25:44.373988+0000","last_active":"2024-03-06T07:25:46.367427+0000","last_peered":"2024-03-06T07:25:46.367427+0000","last_clean":"2024-03-06T07:25:46.367427+0000","last_became_active":"2024-03-06T07:25:44.373714+0000","last_became_peered":"2024-03-06T07:25:44.373714+0000","last_unstale":"2024-03-06T07:25:46.367427+0000","last_undegraded":"2024-03-06T07:25:46.367427+0000","last_fullsized":"2024-03-06T07:25:46.367427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:16.334519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366544+0000","last_change":"2024-03-06T07:25:45.380776+0000","last_active":"2024-03-06T07:25:46.366544+0000","last_peered":"2024-03-06T07:25:46.366544+0000","last_clean":"2024-03-06T07:25:46.366544+0000","last_became_active":"2024-03-06T07:25:45.380405+0000","last_became_peered":"2024-03-06T07:25:45.380405+0000","last_unstale":"2024-03-06T07:25:46.366544+0000","last_undegraded":"2024-03-06T07:25:46.366544+0000","last_fullsized":"2024-03-06T07:25:46.366544+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:41.188810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371865+0000","last_change":"2024-03-06T07:25:44.379776+0000","last_active":"2024-03-06T07:25:46.371865+0000","last_peered":"2024-03-06T07:25:46.371865+0000","last_clean":"2024-03-06T07:25:46.371865+0000","last_became_active":"2024-03-06T07:25:44.379526+0000","last_became_peered":"2024-03-06T07:25:44.379526+0000","last_unstale":"2024-03-06T07:25:46.371865+0000","last_undegraded":"2024-03-06T07:25:46.371865+0000","last_fullsized":"2024-03-06T07:25:46.371865+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:31.547735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430275+0000","last_change":"2024-03-06T07:25:45.379300+0000","last_active":"2024-03-06T07:25:46.430275+0000","last_peered":"2024-03-06T07:25:46.430275+0000","last_clean":"2024-03-06T07:25:46.430275+0000","last_became_active":"2024-03-06T07:25:45.379037+0000","last_became_peered":"2024-03-06T07:25:45.379037+0000","last_unstale":"2024-03-06T07:25:46.430275+0000","last_undegraded":"2024-03-06T07:25:46.430275+0000","last_fullsized":"2024-03-06T07:25:46.430275+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:18.149300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371837+0000","last_change":"2024-03-06T07:25:44.380909+0000","last_active":"2024-03-06T07:25:46.371837+0000","last_peered":"2024-03-06T07:25:46.371837+0000","last_clean":"2024-03-06T07:25:46.371837+0000","last_became_active":"2024-03-06T07:25:44.379555+0000","last_became_peered":"2024-03-06T07:25:44.379555+0000","last_unstale":"2024-03-06T07:25:46.371837+0000","last_undegraded":"2024-03-06T07:25:46.371837+0000","last_fullsized":"2024-03-06T07:25:46.371837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:23.362693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366010+0000","last_change":"2024-03-06T07:25:45.379221+0000","last_active":"2024-03-06T07:25:46.366010+0000","last_peered":"2024-03-06T07:25:46.366010+0000","last_clean":"2024-03-06T07:25:46.366010+0000","last_became_active":"2024-03-06T07:25:45.378903+0000","last_became_peered":"2024-03-06T07:25:45.378903+0000","last_unstale":"2024-03-06T07:25:46.366010+0000","last_undegraded":"2024-03-06T07:25:46.366010+0000","last_fullsized":"2024-03-06T07:25:46.366010+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:02.664414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366949+0000","last_change":"2024-03-06T07:25:44.375570+0000","last_active":"2024-03-06T07:25:46.366949+0000","last_peered":"2024-03-06T07:25:46.366949+0000","last_clean":"2024-03-06T07:25:46.366949+0000","last_became_active":"2024-03-06T07:25:44.375311+0000","last_became_peered":"2024-03-06T07:25:44.375311+0000","last_unstale":"2024-03-06T07:25:46.366949+0000","last_undegraded":"2024-03-06T07:25:46.366949+0000","last_fullsized":"2024-03-06T07:25:46.366949+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:34.889559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366780+0000","last_change":"2024-03-06T07:25:45.386188+0000","last_active":"2024-03-06T07:25:46.366780+0000","last_peered":"2024-03-06T07:25:46.366780+0000","last_clean":"2024-03-06T07:25:46.366780+0000","last_became_active":"2024-03-06T07:25:45.385427+0000","last_became_peered":"2024-03-06T07:25:45.385427+0000","last_unstale":"2024-03-06T07:25:46.366780+0000","last_undegraded":"2024-03-06T07:25:46.366780+0000","last_fullsized":"2024-03-06T07:25:46.366780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:17.123482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367201+0000","last_change":"2024-03-06T07:25:44.375642+0000","last_active":"2024-03-06T07:25:46.367201+0000","last_peered":"2024-03-06T07:25:46.367201+0000","last_clean":"2024-03-06T07:25:46.367201+0000","last_became_active":"2024-03-06T07:25:44.375053+0000","last_became_peered":"2024-03-06T07:25:44.375053+0000","last_unstale":"2024-03-06T07:25:46.367201+0000","last_undegraded":"2024-03-06T07:25:46.367201+0000","last_fullsized":"2024-03-06T07:25:46.367201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:01.662584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429718+0000","last_change":"2024-03-06T07:25:45.388512+0000","last_active":"2024-03-06T07:25:46.429718+0000","last_peered":"2024-03-06T07:25:46.429718+0000","last_clean":"2024-03-06T07:25:46.429718+0000","last_became_active":"2024-03-06T07:25:45.387203+0000","last_became_peered":"2024-03-06T07:25:45.387203+0000","last_unstale":"2024-03-06T07:25:46.429718+0000","last_undegraded":"2024-03-06T07:25:46.429718+0000","last_fullsized":"2024-03-06T07:25:46.429718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:07.623293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371611+0000","last_change":"2024-03-06T07:25:44.377006+0000","last_active":"2024-03-06T07:25:46.371611+0000","last_peered":"2024-03-06T07:25:46.371611+0000","last_clean":"2024-03-06T07:25:46.371611+0000","last_became_active":"2024-03-06T07:25:44.376579+0000","last_became_peered":"2024-03-06T07:25:44.376579+0000","last_unstale":"2024-03-06T07:25:46.371611+0000","last_undegraded":"2024-03-06T07:25:46.371611+0000","last_fullsized":"2024-03-06T07:25:46.371611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:54.552220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429766+0000","last_change":"2024-03-06T07:25:45.381781+0000","last_active":"2024-03-06T07:25:46.429766+0000","last_peered":"2024-03-06T07:25:46.429766+0000","last_clean":"2024-03-06T07:25:46.429766+0000","last_became_active":"2024-03-06T07:25:45.381315+0000","last_became_peered":"2024-03-06T07:25:45.381315+0000","last_unstale":"2024-03-06T07:25:46.429766+0000","last_undegraded":"2024-03-06T07:25:46.429766+0000","last_fullsized":"2024-03-06T07:25:46.429766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:56.099808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365815+0000","last_change":"2024-03-06T07:25:44.374971+0000","last_active":"2024-03-06T07:25:46.365815+0000","last_peered":"2024-03-06T07:25:46.365815+0000","last_clean":"2024-03-06T07:25:46.365815+0000","last_became_active":"2024-03-06T07:25:44.374681+0000","last_became_peered":"2024-03-06T07:25:44.374681+0000","last_unstale":"2024-03-06T07:25:46.365815+0000","last_undegraded":"2024-03-06T07:25:46.365815+0000","last_fullsized":"2024-03-06T07:25:46.365815+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:40.417711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430787+0000","last_change":"2024-03-06T07:25:45.379222+0000","last_active":"2024-03-06T07:25:46.430787+0000","last_peered":"2024-03-06T07:25:46.430787+0000","last_clean":"2024-03-06T07:25:46.430787+0000","last_became_active":"2024-03-06T07:25:45.378895+0000","last_became_peered":"2024-03-06T07:25:45.378895+0000","last_unstale":"2024-03-06T07:25:46.430787+0000","last_undegraded":"2024-03-06T07:25:46.430787+0000","last_fullsized":"2024-03-06T07:25:46.430787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:34.101979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367072+0000","last_change":"2024-03-06T07:25:44.373583+0000","last_active":"2024-03-06T07:25:46.367072+0000","last_peered":"2024-03-06T07:25:46.367072+0000","last_clean":"2024-03-06T07:25:46.367072+0000","last_became_active":"2024-03-06T07:25:44.373202+0000","last_became_peered":"2024-03-06T07:25:44.373202+0000","last_unstale":"2024-03-06T07:25:46.367072+0000","last_undegraded":"2024-03-06T07:25:46.367072+0000","last_fullsized":"2024-03-06T07:25:46.367072+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:09.791751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430494+0000","last_change":"2024-03-06T07:25:45.381537+0000","last_active":"2024-03-06T07:25:46.430494+0000","last_peered":"2024-03-06T07:25:46.430494+0000","last_clean":"2024-03-06T07:25:46.430494+0000","last_became_active":"2024-03-06T07:25:45.381229+0000","last_became_peered":"2024-03-06T07:25:45.381229+0000","last_unstale":"2024-03-06T07:25:46.430494+0000","last_undegraded":"2024-03-06T07:25:46.430494+0000","last_fullsized":"2024-03-06T07:25:46.430494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:05.951684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366584+0000","last_change":"2024-03-06T07:25:44.377068+0000","last_active":"2024-03-06T07:25:46.366584+0000","last_peered":"2024-03-06T07:25:46.366584+0000","last_clean":"2024-03-06T07:25:46.366584+0000","last_became_active":"2024-03-06T07:25:44.376696+0000","last_became_peered":"2024-03-06T07:25:44.376696+0000","last_unstale":"2024-03-06T07:25:46.366584+0000","last_undegraded":"2024-03-06T07:25:46.366584+0000","last_fullsized":"2024-03-06T07:25:46.366584+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:12.563436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371785+0000","last_change":"2024-03-06T07:25:45.382750+0000","last_active":"2024-03-06T07:25:46.371785+0000","last_peered":"2024-03-06T07:25:46.371785+0000","last_clean":"2024-03-06T07:25:46.371785+0000","last_became_active":"2024-03-06T07:25:45.382451+0000","last_became_peered":"2024-03-06T07:25:45.382451+0000","last_unstale":"2024-03-06T07:25:46.371785+0000","last_undegraded":"2024-03-06T07:25:46.371785+0000","last_fullsized":"2024-03-06T07:25:46.371785+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:08.205197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366307+0000","last_change":"2024-03-06T07:25:44.378522+0000","last_active":"2024-03-06T07:25:46.366307+0000","last_peered":"2024-03-06T07:25:46.366307+0000","last_clean":"2024-03-06T07:25:46.366307+0000","last_became_active":"2024-03-06T07:25:44.378182+0000","last_became_peered":"2024-03-06T07:25:44.378182+0000","last_unstale":"2024-03-06T07:25:46.366307+0000","last_undegraded":"2024-03-06T07:25:46.366307+0000","last_fullsized":"2024-03-06T07:25:46.366307+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.012317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366762+0000","last_change":"2024-03-06T07:25:45.384841+0000","last_active":"2024-03-06T07:25:46.366762+0000","last_peered":"2024-03-06T07:25:46.366762+0000","last_clean":"2024-03-06T07:25:46.366762+0000","last_became_active":"2024-03-06T07:25:45.384116+0000","last_became_peered":"2024-03-06T07:25:45.384116+0000","last_unstale":"2024-03-06T07:25:46.366762+0000","last_undegraded":"2024-03-06T07:25:46.366762+0000","last_fullsized":"2024-03-06T07:25:46.366762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:24.364522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429959+0000","last_change":"2024-03-06T07:25:44.375900+0000","last_active":"2024-03-06T07:25:46.429959+0000","last_peered":"2024-03-06T07:25:46.429959+0000","last_clean":"2024-03-06T07:25:46.429959+0000","last_became_active":"2024-03-06T07:25:44.375629+0000","last_became_peered":"2024-03-06T07:25:44.375629+0000","last_unstale":"2024-03-06T07:25:46.429959+0000","last_undegraded":"2024-03-06T07:25:46.429959+0000","last_fullsized":"2024-03-06T07:25:46.429959+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:42.937961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371661+0000","last_change":"2024-03-06T07:25:45.378796+0000","last_active":"2024-03-06T07:25:46.371661+0000","last_peered":"2024-03-06T07:25:46.371661+0000","last_clean":"2024-03-06T07:25:46.371661+0000","last_became_active":"2024-03-06T07:25:45.378498+0000","last_became_peered":"2024-03-06T07:25:45.378498+0000","last_unstale":"2024-03-06T07:25:46.371661+0000","last_undegraded":"2024-03-06T07:25:46.371661+0000","last_fullsized":"2024-03-06T07:25:46.371661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:23.547706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365992+0000","last_change":"2024-03-06T07:25:44.379715+0000","last_active":"2024-03-06T07:25:46.365992+0000","last_peered":"2024-03-06T07:25:46.365992+0000","last_clean":"2024-03-06T07:25:46.365992+0000","last_became_active":"2024-03-06T07:25:44.379439+0000","last_became_peered":"2024-03-06T07:25:44.379439+0000","last_unstale":"2024-03-06T07:25:46.365992+0000","last_undegraded":"2024-03-06T07:25:46.365992+0000","last_fullsized":"2024-03-06T07:25:46.365992+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:46.517433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430038+0000","last_change":"2024-03-06T07:25:45.386851+0000","last_active":"2024-03-06T07:25:46.430038+0000","last_peered":"2024-03-06T07:25:46.430038+0000","last_clean":"2024-03-06T07:25:46.430038+0000","last_became_active":"2024-03-06T07:25:45.386544+0000","last_became_peered":"2024-03-06T07:25:45.386544+0000","last_unstale":"2024-03-06T07:25:46.430038+0000","last_undegraded":"2024-03-06T07:25:46.430038+0000","last_fullsized":"2024-03-06T07:25:46.430038+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:13.630116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430418+0000","last_change":"2024-03-06T07:25:44.373942+0000","last_active":"2024-03-06T07:25:46.430418+0000","last_peered":"2024-03-06T07:25:46.430418+0000","last_clean":"2024-03-06T07:25:46.430418+0000","last_became_active":"2024-03-06T07:25:44.373655+0000","last_became_peered":"2024-03-06T07:25:44.373655+0000","last_unstale":"2024-03-06T07:25:46.430418+0000","last_undegraded":"2024-03-06T07:25:46.430418+0000","last_fullsized":"2024-03-06T07:25:46.430418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:12.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365803+0000","last_change":"2024-03-06T07:25:45.382316+0000","last_active":"2024-03-06T07:25:46.365803+0000","last_peered":"2024-03-06T07:25:46.365803+0000","last_clean":"2024-03-06T07:25:46.365803+0000","last_became_active":"2024-03-06T07:25:45.382093+0000","last_became_peered":"2024-03-06T07:25:45.382093+0000","last_unstale":"2024-03-06T07:25:46.365803+0000","last_undegraded":"2024-03-06T07:25:46.365803+0000","last_fullsized":"2024-03-06T07:25:46.365803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:01.951919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430035+0000","last_change":"2024-03-06T07:25:44.376438+0000","last_active":"2024-03-06T07:25:46.430035+0000","last_peered":"2024-03-06T07:25:46.430035+0000","last_clean":"2024-03-06T07:25:46.430035+0000","last_became_active":"2024-03-06T07:25:44.376148+0000","last_became_peered":"2024-03-06T07:25:44.376148+0000","last_unstale":"2024-03-06T07:25:46.430035+0000","last_undegraded":"2024-03-06T07:25:46.430035+0000","last_fullsized":"2024-03-06T07:25:46.430035+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:47.873251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365768+0000","last_change":"2024-03-06T07:25:45.377048+0000","last_active":"2024-03-06T07:25:46.365768+0000","last_peered":"2024-03-06T07:25:46.365768+0000","last_clean":"2024-03-06T07:25:46.365768+0000","last_became_active":"2024-03-06T07:25:45.376731+0000","last_became_peered":"2024-03-06T07:25:45.376731+0000","last_unstale":"2024-03-06T07:25:46.365768+0000","last_undegraded":"2024-03-06T07:25:46.365768+0000","last_fullsized":"2024-03-06T07:25:46.365768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:36.831668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371567+0000","last_change":"2024-03-06T07:25:44.380239+0000","last_active":"2024-03-06T07:25:46.371567+0000","last_peered":"2024-03-06T07:25:46.371567+0000","last_clean":"2024-03-06T07:25:46.371567+0000","last_became_active":"2024-03-06T07:25:44.379998+0000","last_became_peered":"2024-03-06T07:25:44.379998+0000","last_unstale":"2024-03-06T07:25:46.371567+0000","last_undegraded":"2024-03-06T07:25:46.371567+0000","last_fullsized":"2024-03-06T07:25:46.371567+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:34.366316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430324+0000","last_change":"2024-03-06T07:25:45.377122+0000","last_active":"2024-03-06T07:25:46.430324+0000","last_peered":"2024-03-06T07:25:46.430324+0000","last_clean":"2024-03-06T07:25:46.430324+0000","last_became_active":"2024-03-06T07:25:45.376836+0000","last_became_peered":"2024-03-06T07:25:45.376836+0000","last_unstale":"2024-03-06T07:25:46.430324+0000","last_undegraded":"2024-03-06T07:25:46.430324+0000","last_fullsized":"2024-03-06T07:25:46.430324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:48.887856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367240+0000","last_change":"2024-03-06T07:25:44.379838+0000","last_active":"2024-03-06T07:25:46.367240+0000","last_peered":"2024-03-06T07:25:46.367240+0000","last_clean":"2024-03-06T07:25:46.367240+0000","last_became_active":"2024-03-06T07:25:44.379308+0000","last_became_peered":"2024-03-06T07:25:44.379308+0000","last_unstale":"2024-03-06T07:25:46.367240+0000","last_undegraded":"2024-03-06T07:25:46.367240+0000","last_fullsized":"2024-03-06T07:25:46.367240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:34.889559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366884+0000","last_change":"2024-03-06T07:25:45.385579+0000","last_active":"2024-03-06T07:25:46.366884+0000","last_peered":"2024-03-06T07:25:46.366884+0000","last_clean":"2024-03-06T07:25:46.366884+0000","last_became_active":"2024-03-06T07:25:45.380570+0000","last_became_peered":"2024-03-06T07:25:45.380570+0000","last_unstale":"2024-03-06T07:25:46.366884+0000","last_undegraded":"2024-03-06T07:25:46.366884+0000","last_fullsized":"2024-03-06T07:25:46.366884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:33.664581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371944+0000","last_change":"2024-03-06T07:25:44.380777+0000","last_active":"2024-03-06T07:25:46.371944+0000","last_peered":"2024-03-06T07:25:46.371944+0000","last_clean":"2024-03-06T07:25:46.371944+0000","last_became_active":"2024-03-06T07:25:44.380528+0000","last_became_peered":"2024-03-06T07:25:44.380528+0000","last_unstale":"2024-03-06T07:25:46.371944+0000","last_undegraded":"2024-03-06T07:25:46.371944+0000","last_fullsized":"2024-03-06T07:25:46.371944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:23.381612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366891+0000","last_change":"2024-03-06T07:25:45.379971+0000","last_active":"2024-03-06T07:25:46.366891+0000","last_peered":"2024-03-06T07:25:46.366891+0000","last_clean":"2024-03-06T07:25:46.366891+0000","last_became_active":"2024-03-06T07:25:45.379445+0000","last_became_peered":"2024-03-06T07:25:45.379445+0000","last_unstale":"2024-03-06T07:25:46.366891+0000","last_undegraded":"2024-03-06T07:25:46.366891+0000","last_fullsized":"2024-03-06T07:25:46.366891+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:41.188810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371866+0000","last_change":"2024-03-06T07:25:44.377950+0000","last_active":"2024-03-06T07:25:46.371866+0000","last_peered":"2024-03-06T07:25:46.371866+0000","last_clean":"2024-03-06T07:25:46.371866+0000","last_became_active":"2024-03-06T07:25:44.377692+0000","last_became_peered":"2024-03-06T07:25:44.377692+0000","last_unstale":"2024-03-06T07:25:46.371866+0000","last_undegraded":"2024-03-06T07:25:46.371866+0000","last_fullsized":"2024-03-06T07:25:46.371866+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:16.121652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430307+0000","last_change":"2024-03-06T07:25:45.386164+0000","last_active":"2024-03-06T07:25:46.430307+0000","last_peered":"2024-03-06T07:25:46.430307+0000","last_clean":"2024-03-06T07:25:46.430307+0000","last_became_active":"2024-03-06T07:25:45.385917+0000","last_became_peered":"2024-03-06T07:25:45.385917+0000","last_unstale":"2024-03-06T07:25:46.430307+0000","last_undegraded":"2024-03-06T07:25:46.430307+0000","last_fullsized":"2024-03-06T07:25:46.430307+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:57.773005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430515+0000","last_change":"2024-03-06T07:25:44.371606+0000","last_active":"2024-03-06T07:25:46.430515+0000","last_peered":"2024-03-06T07:25:46.430515+0000","last_clean":"2024-03-06T07:25:46.430515+0000","last_became_active":"2024-03-06T07:25:44.371097+0000","last_became_peered":"2024-03-06T07:25:44.371097+0000","last_unstale":"2024-03-06T07:25:46.430515+0000","last_undegraded":"2024-03-06T07:25:46.430515+0000","last_fullsized":"2024-03-06T07:25:46.430515+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:54.188838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429890+0000","last_change":"2024-03-06T07:25:45.382784+0000","last_active":"2024-03-06T07:25:46.429890+0000","last_peered":"2024-03-06T07:25:46.429890+0000","last_clean":"2024-03-06T07:25:46.429890+0000","last_became_active":"2024-03-06T07:25:45.382492+0000","last_became_peered":"2024-03-06T07:25:45.382492+0000","last_unstale":"2024-03-06T07:25:46.429890+0000","last_undegraded":"2024-03-06T07:25:46.429890+0000","last_fullsized":"2024-03-06T07:25:46.429890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:55.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430282+0000","last_change":"2024-03-06T07:25:44.376345+0000","last_active":"2024-03-06T07:25:46.430282+0000","last_peered":"2024-03-06T07:25:46.430282+0000","last_clean":"2024-03-06T07:25:46.430282+0000","last_became_active":"2024-03-06T07:25:44.376085+0000","last_became_peered":"2024-03-06T07:25:44.376085+0000","last_unstale":"2024-03-06T07:25:46.430282+0000","last_undegraded":"2024-03-06T07:25:46.430282+0000","last_fullsized":"2024-03-06T07:25:46.430282+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:41.262045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429950+0000","last_change":"2024-03-06T07:25:45.377034+0000","last_active":"2024-03-06T07:25:46.429950+0000","last_peered":"2024-03-06T07:25:46.429950+0000","last_clean":"2024-03-06T07:25:46.429950+0000","last_became_active":"2024-03-06T07:25:45.376751+0000","last_became_peered":"2024-03-06T07:25:45.376751+0000","last_unstale":"2024-03-06T07:25:46.429950+0000","last_undegraded":"2024-03-06T07:25:46.429950+0000","last_fullsized":"2024-03-06T07:25:46.429950+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:42.868277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430059+0000","last_change":"2024-03-06T07:25:44.377832+0000","last_active":"2024-03-06T07:25:46.430059+0000","last_peered":"2024-03-06T07:25:46.430059+0000","last_clean":"2024-03-06T07:25:46.430059+0000","last_became_active":"2024-03-06T07:25:44.377535+0000","last_became_peered":"2024-03-06T07:25:44.377535+0000","last_unstale":"2024-03-06T07:25:46.430059+0000","last_undegraded":"2024-03-06T07:25:46.430059+0000","last_fullsized":"2024-03-06T07:25:46.430059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:19.362948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429961+0000","last_change":"2024-03-06T07:25:45.384259+0000","last_active":"2024-03-06T07:25:46.429961+0000","last_peered":"2024-03-06T07:25:46.429961+0000","last_clean":"2024-03-06T07:25:46.429961+0000","last_became_active":"2024-03-06T07:25:45.384008+0000","last_became_peered":"2024-03-06T07:25:45.384008+0000","last_unstale":"2024-03-06T07:25:46.429961+0000","last_undegraded":"2024-03-06T07:25:46.429961+0000","last_fullsized":"2024-03-06T07:25:46.429961+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:13.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429716+0000","last_change":"2024-03-06T07:25:44.371371+0000","last_active":"2024-03-06T07:25:46.429716+0000","last_peered":"2024-03-06T07:25:46.429716+0000","last_clean":"2024-03-06T07:25:46.429716+0000","last_became_active":"2024-03-06T07:25:44.371076+0000","last_became_peered":"2024-03-06T07:25:44.371076+0000","last_unstale":"2024-03-06T07:25:46.429716+0000","last_undegraded":"2024-03-06T07:25:46.429716+0000","last_fullsized":"2024-03-06T07:25:46.429716+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:37.057834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367332+0000","last_change":"2024-03-06T07:25:45.375740+0000","last_active":"2024-03-06T07:25:46.367332+0000","last_peered":"2024-03-06T07:25:46.367332+0000","last_clean":"2024-03-06T07:25:46.367332+0000","last_became_active":"2024-03-06T07:25:45.375476+0000","last_became_peered":"2024-03-06T07:25:45.375476+0000","last_unstale":"2024-03-06T07:25:46.367332+0000","last_undegraded":"2024-03-06T07:25:46.367332+0000","last_fullsized":"2024-03-06T07:25:46.367332+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:04.143346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366140+0000","last_change":"2024-03-06T07:25:44.378787+0000","last_active":"2024-03-06T07:25:46.366140+0000","last_peered":"2024-03-06T07:25:46.366140+0000","last_clean":"2024-03-06T07:25:46.366140+0000","last_became_active":"2024-03-06T07:25:44.378481+0000","last_became_peered":"2024-03-06T07:25:44.378481+0000","last_unstale":"2024-03-06T07:25:46.366140+0000","last_undegraded":"2024-03-06T07:25:46.366140+0000","last_fullsized":"2024-03-06T07:25:46.366140+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:16.785392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430351+0000","last_change":"2024-03-06T07:25:45.383330+0000","last_active":"2024-03-06T07:25:46.430351+0000","last_peered":"2024-03-06T07:25:46.430351+0000","last_clean":"2024-03-06T07:25:46.430351+0000","last_became_active":"2024-03-06T07:25:45.382932+0000","last_became_peered":"2024-03-06T07:25:45.382932+0000","last_unstale":"2024-03-06T07:25:46.430351+0000","last_undegraded":"2024-03-06T07:25:46.430351+0000","last_fullsized":"2024-03-06T07:25:46.430351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:34.291075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.369006+0000","last_change":"2024-03-06T07:25:44.378760+0000","last_active":"2024-03-06T07:25:46.369006+0000","last_peered":"2024-03-06T07:25:46.369006+0000","last_clean":"2024-03-06T07:25:46.369006+0000","last_became_active":"2024-03-06T07:25:44.377480+0000","last_became_peered":"2024-03-06T07:25:44.377480+0000","last_unstale":"2024-03-06T07:25:46.369006+0000","last_undegraded":"2024-03-06T07:25:46.369006+0000","last_fullsized":"2024-03-06T07:25:46.369006+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:56.771175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366697+0000","last_change":"2024-03-06T07:25:45.380118+0000","last_active":"2024-03-06T07:25:46.366697+0000","last_peered":"2024-03-06T07:25:46.366697+0000","last_clean":"2024-03-06T07:25:46.366697+0000","last_became_active":"2024-03-06T07:25:45.379416+0000","last_became_peered":"2024-03-06T07:25:45.379416+0000","last_unstale":"2024-03-06T07:25:46.366697+0000","last_undegraded":"2024-03-06T07:25:46.366697+0000","last_fullsized":"2024-03-06T07:25:46.366697+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:05.951684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366479+0000","last_change":"2024-03-06T07:25:44.376028+0000","last_active":"2024-03-06T07:25:46.366479+0000","last_peered":"2024-03-06T07:25:46.366479+0000","last_clean":"2024-03-06T07:25:46.366479+0000","last_became_active":"2024-03-06T07:25:44.375743+0000","last_became_peered":"2024-03-06T07:25:44.375743+0000","last_unstale":"2024-03-06T07:25:46.366479+0000","last_undegraded":"2024-03-06T07:25:46.366479+0000","last_fullsized":"2024-03-06T07:25:46.366479+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:22.498536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367018+0000","last_change":"2024-03-06T07:25:45.384459+0000","last_active":"2024-03-06T07:25:46.367018+0000","last_peered":"2024-03-06T07:25:46.367018+0000","last_clean":"2024-03-06T07:25:46.367018+0000","last_became_active":"2024-03-06T07:25:45.383764+0000","last_became_peered":"2024-03-06T07:25:45.383764+0000","last_unstale":"2024-03-06T07:25:46.367018+0000","last_undegraded":"2024-03-06T07:25:46.367018+0000","last_fullsized":"2024-03-06T07:25:46.367018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:05.951684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366757+0000","last_change":"2024-03-06T07:25:44.373902+0000","last_active":"2024-03-06T07:25:46.366757+0000","last_peered":"2024-03-06T07:25:46.366757+0000","last_clean":"2024-03-06T07:25:46.366757+0000","last_became_active":"2024-03-06T07:25:44.372777+0000","last_became_peered":"2024-03-06T07:25:44.372777+0000","last_unstale":"2024-03-06T07:25:46.366757+0000","last_undegraded":"2024-03-06T07:25:46.366757+0000","last_fullsized":"2024-03-06T07:25:46.366757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:16.334519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430076+0000","last_change":"2024-03-06T07:25:45.383160+0000","last_active":"2024-03-06T07:25:46.430076+0000","last_peered":"2024-03-06T07:25:46.430076+0000","last_clean":"2024-03-06T07:25:46.430076+0000","last_became_active":"2024-03-06T07:25:45.382380+0000","last_became_peered":"2024-03-06T07:25:45.382380+0000","last_unstale":"2024-03-06T07:25:46.430076+0000","last_undegraded":"2024-03-06T07:25:46.430076+0000","last_fullsized":"2024-03-06T07:25:46.430076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:24.364522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371809+0000","last_change":"2024-03-06T07:25:44.380512+0000","last_active":"2024-03-06T07:25:46.371809+0000","last_peered":"2024-03-06T07:25:46.371809+0000","last_clean":"2024-03-06T07:25:46.371809+0000","last_became_active":"2024-03-06T07:25:44.380118+0000","last_became_peered":"2024-03-06T07:25:44.380118+0000","last_unstale":"2024-03-06T07:25:46.371809+0000","last_undegraded":"2024-03-06T07:25:46.371809+0000","last_fullsized":"2024-03-06T07:25:46.371809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:12.963628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371764+0000","last_change":"2024-03-06T07:25:45.378630+0000","last_active":"2024-03-06T07:25:46.371764+0000","last_peered":"2024-03-06T07:25:46.371764+0000","last_clean":"2024-03-06T07:25:46.371764+0000","last_became_active":"2024-03-06T07:25:45.378372+0000","last_became_peered":"2024-03-06T07:25:45.378372+0000","last_unstale":"2024-03-06T07:25:46.371764+0000","last_undegraded":"2024-03-06T07:25:46.371764+0000","last_fullsized":"2024-03-06T07:25:46.371764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:49.071208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367294+0000","last_change":"2024-03-06T07:25:44.374667+0000","last_active":"2024-03-06T07:25:46.367294+0000","last_peered":"2024-03-06T07:25:46.367294+0000","last_clean":"2024-03-06T07:25:46.367294+0000","last_became_active":"2024-03-06T07:25:44.374401+0000","last_became_peered":"2024-03-06T07:25:44.374401+0000","last_unstale":"2024-03-06T07:25:46.367294+0000","last_undegraded":"2024-03-06T07:25:46.367294+0000","last_fullsized":"2024-03-06T07:25:46.367294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:09.791751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429772+0000","last_change":"2024-03-06T07:25:45.390230+0000","last_active":"2024-03-06T07:25:46.429772+0000","last_peered":"2024-03-06T07:25:46.429772+0000","last_clean":"2024-03-06T07:25:46.429772+0000","last_became_active":"2024-03-06T07:25:45.389955+0000","last_became_peered":"2024-03-06T07:25:45.389955+0000","last_unstale":"2024-03-06T07:25:46.429772+0000","last_undegraded":"2024-03-06T07:25:46.429772+0000","last_fullsized":"2024-03-06T07:25:46.429772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:56.684522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367229+0000","last_change":"2024-03-06T07:25:44.373098+0000","last_active":"2024-03-06T07:25:46.367229+0000","last_peered":"2024-03-06T07:25:46.367229+0000","last_clean":"2024-03-06T07:25:46.367229+0000","last_became_active":"2024-03-06T07:25:44.371542+0000","last_became_peered":"2024-03-06T07:25:44.371542+0000","last_unstale":"2024-03-06T07:25:46.367229+0000","last_undegraded":"2024-03-06T07:25:46.367229+0000","last_fullsized":"2024-03-06T07:25:46.367229+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:49.360890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367079+0000","last_change":"2024-03-06T07:25:45.379807+0000","last_active":"2024-03-06T07:25:46.367079+0000","last_peered":"2024-03-06T07:25:46.367079+0000","last_clean":"2024-03-06T07:25:46.367079+0000","last_became_active":"2024-03-06T07:25:45.379382+0000","last_became_peered":"2024-03-06T07:25:45.379382+0000","last_unstale":"2024-03-06T07:25:46.367079+0000","last_undegraded":"2024-03-06T07:25:46.367079+0000","last_fullsized":"2024-03-06T07:25:46.367079+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:49.071208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366397+0000","last_change":"2024-03-06T07:25:44.377688+0000","last_active":"2024-03-06T07:25:46.366397+0000","last_peered":"2024-03-06T07:25:46.366397+0000","last_clean":"2024-03-06T07:25:46.366397+0000","last_became_active":"2024-03-06T07:25:44.377411+0000","last_became_peered":"2024-03-06T07:25:44.377411+0000","last_unstale":"2024-03-06T07:25:46.366397+0000","last_undegraded":"2024-03-06T07:25:46.366397+0000","last_fullsized":"2024-03-06T07:25:46.366397+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:12.563436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366107+0000","last_change":"2024-03-06T07:25:45.376362+0000","last_active":"2024-03-06T07:25:46.366107+0000","last_peered":"2024-03-06T07:25:46.366107+0000","last_clean":"2024-03-06T07:25:46.366107+0000","last_became_active":"2024-03-06T07:25:45.376117+0000","last_became_peered":"2024-03-06T07:25:45.376117+0000","last_unstale":"2024-03-06T07:25:46.366107+0000","last_undegraded":"2024-03-06T07:25:46.366107+0000","last_fullsized":"2024-03-06T07:25:46.366107+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:13.565265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430228+0000","last_change":"2024-03-06T07:25:44.382115+0000","last_active":"2024-03-06T07:25:46.430228+0000","last_peered":"2024-03-06T07:25:46.430228+0000","last_clean":"2024-03-06T07:25:46.430228+0000","last_became_active":"2024-03-06T07:25:44.381866+0000","last_became_peered":"2024-03-06T07:25:44.381866+0000","last_unstale":"2024-03-06T07:25:46.430228+0000","last_undegraded":"2024-03-06T07:25:46.430228+0000","last_fullsized":"2024-03-06T07:25:46.430228+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:03.141516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":276,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429776+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:25:46.429776+0000","last_peered":"2024-03-06T07:25:46.429776+0000","last_clean":"2024-03-06T07:25:46.429776+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:25:46.429776+0000","last_undegraded":"2024-03-06T07:25:46.429776+0000","last_fullsized":"2024-03-06T07:25:46.429776+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":1005926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335524,"statfs":{"total":96636764160,"available":96599576576,"internally_reserved":0,"allocated":995328,"data_stored":416182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335520,"statfs":{"total":96636764160,"available":96599572480,"internally_reserved":0,"allocated":999424,"data_stored":417122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":983040,"data_stored":415488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":417100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":987136,"data_stored":416070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":415682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1581056,"data_stored":1005890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:48.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:49.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.124+0000 7fc98b3f4640 1 Processor -- start 2024-03-06T07:25:49.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.124+0000 7fc98b3f4640 1 -- start start 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc98b3f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc98413dc10 0x7fc9840b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc98b3f4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc98415bf30 0x7fc9840b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc98b3f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415e850 0x7fc98415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc98b3f4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9840cf3f0 con 0x7fc98413dc10 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc98b3f4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98415b210 con 0x7fc98415bf30 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc98b3f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98415f580 con 0x7fc98415e850 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989190640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc98413dc10 0x7fc9840b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989190640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc98413dc10 0x7fc9840b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52076/0 (socket says 172.21.15.77:52076) 2024-03-06T07:25:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989190640 1 -- 172.21.15.77:0/1556707425 learned_addr learned my addr 172.21.15.77:0/1556707425 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc98898f640 1 --2- 172.21.15.77:0/1556707425 >> v2:172.21.15.158:3300/0 conn(0x7fc98415bf30 0x7fc9840b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989991640 1 --2- 172.21.15.77:0/1556707425 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415e850 0x7fc98415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989190640 1 -- 172.21.15.77:0/1556707425 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415e850 msgr2=0x7fc98415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989190640 1 --2- 172.21.15.77:0/1556707425 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415e850 0x7fc98415eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989190640 1 -- 172.21.15.77:0/1556707425 >> v2:172.21.15.158:3300/0 conn(0x7fc98415bf30 msgr2=0x7fc9840b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989190640 1 --2- 172.21.15.77:0/1556707425 >> v2:172.21.15.158:3300/0 conn(0x7fc98415bf30 0x7fc9840b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.125+0000 7fc989190640 1 -- 172.21.15.77:0/1556707425 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98415b450 con 0x7fc98413dc10 2024-03-06T07:25:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc989190640 1 --2- 172.21.15.77:0/1556707425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc9840b2950 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc98406c790 tx=0x7fc98002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4026d1766869369 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc97bfff640 1 -- 172.21.15.77:0/1556707425 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98002e070 con 0x7fc98413dc10 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc97bfff640 1 -- 172.21.15.77:0/1556707425 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980037070 con 0x7fc98413dc10 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc97bfff640 1 -- 172.21.15.77:0/1556707425 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980031a00 con 0x7fc98413dc10 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc98b3f4640 1 -- 172.21.15.77:0/1556707425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 msgr2=0x7fc9840b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/1556707425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc9840b2950 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc98406c790 tx=0x7fc98002b880 comp rx=0 tx=0).stop 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc98b3f4640 1 -- 172.21.15.77:0/1556707425 shutdown_connections 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/1556707425 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415e850 0x7fc98415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/1556707425 >> v2:172.21.15.158:3300/0 conn(0x7fc98415bf30 0x7fc9840b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/1556707425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc9840b2950 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.126+0000 7fc98b3f4640 1 -- 172.21.15.77:0/1556707425 >> 172.21.15.77:0/1556707425 conn(0x7fc9840cf8a0 msgr2=0x7fc9840cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.127+0000 7fc98b3f4640 1 -- 172.21.15.77:0/1556707425 shutdown_connections 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.127+0000 7fc98b3f4640 1 -- 172.21.15.77:0/1556707425 wait complete. 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc98b3f4640 1 Processor -- start 2024-03-06T07:25:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc98b3f4640 1 -- start start 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc98b3f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc984156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc98b3f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415bf30 0x7fc9841586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc98b3f4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98415e850 0x7fc98413c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc98b3f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98415f1e0 con 0x7fc98413dc10 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc98b3f4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98413dff0 con 0x7fc98415e850 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc98b3f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98415f360 con 0x7fc98415bf30 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc989190640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc984156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc989190640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc984156e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52080/0 (socket says 172.21.15.77:52080) 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.128+0000 7fc989190640 1 -- 172.21.15.77:0/207115552 learned_addr learned my addr 172.21.15.77:0/207115552 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc989991640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98415e850 0x7fc98413c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc98898f640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415bf30 0x7fc9841586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc989190640 1 -- 172.21.15.77:0/207115552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415bf30 msgr2=0x7fc9841586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc989190640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415bf30 0x7fc9841586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc989190640 1 -- 172.21.15.77:0/207115552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98415e850 msgr2=0x7fc98413c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc989190640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98415e850 0x7fc98413c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc989190640 1 -- 172.21.15.77:0/207115552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98415a030 con 0x7fc98413dc10 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc989190640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc984156e80 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc98002b8c0 tx=0x7fc9800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc979ffb640 1 -- 172.21.15.77:0/207115552 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980032670 con 0x7fc98413dc10 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc979ffb640 1 -- 172.21.15.77:0/207115552 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980032810 con 0x7fc98413dc10 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc984158f90 con 0x7fc98413dc10 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc979ffb640 1 -- 172.21.15.77:0/207115552 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980032bc0 con 0x7fc98413dc10 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.129+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc984159830 con 0x7fc98413dc10 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.130+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9841551a0 con 0x7fc98413dc10 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.131+0000 7fc979ffb640 1 -- 172.21.15.77:0/207115552 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc98004c060 con 0x7fc98413dc10 2024-03-06T07:25:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.131+0000 7fc979ffb640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc96806b4d0 0x7fc96806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.131+0000 7fc979ffb640 1 -- 172.21.15.77:0/207115552 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9800ab9a0 con 0x7fc98413dc10 2024-03-06T07:25:49.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.131+0000 7fc98898f640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc96806b4d0 0x7fc96806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.131+0000 7fc98898f640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc96806b4d0 0x7fc96806d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc98413f710 tx=0x7fc974009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.135+0000 7fc979ffb640 1 -- 172.21.15.77:0/207115552 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9841551a0 con 0x7fc98413dc10 2024-03-06T07:25:49.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.303+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc98404f330 con 0x7fc96806b4d0 2024-03-06T07:25:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.309+0000 7fc979ffb640 1 -- 172.21.15.77:0/207115552 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252074 (secure 0 0 0) 0x7fc98404f330 con 0x7fc96806b4d0 2024-03-06T07:25:49.313 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:49.314 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc96806b4d0 msgr2=0x7fc96806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc96806b4d0 0x7fc96806d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc98413f710 tx=0x7fc974009190 comp rx=0 tx=0).stop 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 msgr2=0x7fc984156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc984156e80 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc98002b8c0 tx=0x7fc9800046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 shutdown_connections 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc98415e850 0x7fc98413c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc98415bf30 0x7fc9841586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc96806b4d0 0x7fc96806d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 --2- 172.21.15.77:0/207115552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc98413dc10 0x7fc984156e80 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 >> 172.21.15.77:0/207115552 conn(0x7fc9840cf8a0 msgr2=0x7fc98415de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 shutdown_connections 2024-03-06T07:25:49.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.312+0000 7fc98b3f4640 1 -- 172.21.15.77:0/207115552 wait complete. 2024-03-06T07:25:49.324 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-03-06T07:25:47.424713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":275016,"kb_used_data":8904,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754699704,"statfs":{"total":773094113280,"available":772812496896,"internally_reserved":0,"allocated":9117696,"data_stored":4509460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.068210"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366721+0000","last_change":"2024-03-06T07:25:45.377105+0000","last_active":"2024-03-06T07:25:46.366721+0000","last_peered":"2024-03-06T07:25:46.366721+0000","last_clean":"2024-03-06T07:25:46.366721+0000","last_became_active":"2024-03-06T07:25:45.376811+0000","last_became_peered":"2024-03-06T07:25:45.376811+0000","last_unstale":"2024-03-06T07:25:46.366721+0000","last_undegraded":"2024-03-06T07:25:46.366721+0000","last_fullsized":"2024-03-06T07:25:46.366721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:04.143346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366321+0000","last_change":"2024-03-06T07:25:45.378496+0000","last_active":"2024-03-06T07:25:46.366321+0000","last_peered":"2024-03-06T07:25:46.366321+0000","last_clean":"2024-03-06T07:25:46.366321+0000","last_became_active":"2024-03-06T07:25:45.378247+0000","last_became_peered":"2024-03-06T07:25:45.378247+0000","last_unstale":"2024-03-06T07:25:46.366321+0000","last_undegraded":"2024-03-06T07:25:46.366321+0000","last_fullsized":"2024-03-06T07:25:46.366321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:17.336349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430124+0000","last_change":"2024-03-06T07:25:45.381750+0000","last_active":"2024-03-06T07:25:46.430124+0000","last_peered":"2024-03-06T07:25:46.430124+0000","last_clean":"2024-03-06T07:25:46.430124+0000","last_became_active":"2024-03-06T07:25:45.381494+0000","last_became_peered":"2024-03-06T07:25:45.381494+0000","last_unstale":"2024-03-06T07:25:46.430124+0000","last_undegraded":"2024-03-06T07:25:46.430124+0000","last_fullsized":"2024-03-06T07:25:46.430124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:55.190667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430453+0000","last_change":"2024-03-06T07:25:45.390449+0000","last_active":"2024-03-06T07:25:46.430453+0000","last_peered":"2024-03-06T07:25:46.430453+0000","last_clean":"2024-03-06T07:25:46.430453+0000","last_became_active":"2024-03-06T07:25:45.390161+0000","last_became_peered":"2024-03-06T07:25:45.390161+0000","last_unstale":"2024-03-06T07:25:46.430453+0000","last_undegraded":"2024-03-06T07:25:46.430453+0000","last_fullsized":"2024-03-06T07:25:46.430453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:47.519263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366634+0000","last_change":"2024-03-06T07:25:45.381238+0000","last_active":"2024-03-06T07:25:46.366634+0000","last_peered":"2024-03-06T07:25:46.366634+0000","last_clean":"2024-03-06T07:25:46.366634+0000","last_became_active":"2024-03-06T07:25:45.380981+0000","last_became_peered":"2024-03-06T07:25:45.380981+0000","last_unstale":"2024-03-06T07:25:46.366634+0000","last_undegraded":"2024-03-06T07:25:46.366634+0000","last_fullsized":"2024-03-06T07:25:46.366634+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:13.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429935+0000","last_change":"2024-03-06T07:25:45.382092+0000","last_active":"2024-03-06T07:25:46.429935+0000","last_peered":"2024-03-06T07:25:46.429935+0000","last_clean":"2024-03-06T07:25:46.429935+0000","last_became_active":"2024-03-06T07:25:45.381760+0000","last_became_peered":"2024-03-06T07:25:45.381760+0000","last_unstale":"2024-03-06T07:25:46.429935+0000","last_undegraded":"2024-03-06T07:25:46.429935+0000","last_fullsized":"2024-03-06T07:25:46.429935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:32.549564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429994+0000","last_change":"2024-03-06T07:25:45.378408+0000","last_active":"2024-03-06T07:25:46.429994+0000","last_peered":"2024-03-06T07:25:46.429994+0000","last_clean":"2024-03-06T07:25:46.429994+0000","last_became_active":"2024-03-06T07:25:45.378171+0000","last_became_peered":"2024-03-06T07:25:45.378171+0000","last_unstale":"2024-03-06T07:25:46.429994+0000","last_undegraded":"2024-03-06T07:25:46.429994+0000","last_fullsized":"2024-03-06T07:25:46.429994+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:39.546940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430311+0000","last_change":"2024-03-06T07:25:45.387654+0000","last_active":"2024-03-06T07:25:46.430311+0000","last_peered":"2024-03-06T07:25:46.430311+0000","last_clean":"2024-03-06T07:25:46.430311+0000","last_became_active":"2024-03-06T07:25:45.387359+0000","last_became_peered":"2024-03-06T07:25:45.387359+0000","last_unstale":"2024-03-06T07:25:46.430311+0000","last_undegraded":"2024-03-06T07:25:46.430311+0000","last_fullsized":"2024-03-06T07:25:46.430311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:17.123482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430450+0000","last_change":"2024-03-06T07:25:45.383579+0000","last_active":"2024-03-06T07:25:46.430450+0000","last_peered":"2024-03-06T07:25:46.430450+0000","last_clean":"2024-03-06T07:25:46.430450+0000","last_became_active":"2024-03-06T07:25:45.383307+0000","last_became_peered":"2024-03-06T07:25:45.383307+0000","last_unstale":"2024-03-06T07:25:46.430450+0000","last_undegraded":"2024-03-06T07:25:46.430450+0000","last_fullsized":"2024-03-06T07:25:46.430450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:41.188810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366507+0000","last_change":"2024-03-06T07:25:45.388827+0000","last_active":"2024-03-06T07:25:46.366507+0000","last_peered":"2024-03-06T07:25:46.366507+0000","last_clean":"2024-03-06T07:25:46.366507+0000","last_became_active":"2024-03-06T07:25:45.386617+0000","last_became_peered":"2024-03-06T07:25:45.386617+0000","last_unstale":"2024-03-06T07:25:46.366507+0000","last_undegraded":"2024-03-06T07:25:46.366507+0000","last_fullsized":"2024-03-06T07:25:46.366507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:17.123482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366158+0000","last_change":"2024-03-06T07:25:45.382537+0000","last_active":"2024-03-06T07:25:46.366158+0000","last_peered":"2024-03-06T07:25:46.366158+0000","last_clean":"2024-03-06T07:25:46.366158+0000","last_became_active":"2024-03-06T07:25:45.381404+0000","last_became_peered":"2024-03-06T07:25:45.381404+0000","last_unstale":"2024-03-06T07:25:46.366158+0000","last_undegraded":"2024-03-06T07:25:46.366158+0000","last_fullsized":"2024-03-06T07:25:46.366158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:35.891388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366240+0000","last_change":"2024-03-06T07:25:45.378553+0000","last_active":"2024-03-06T07:25:46.366240+0000","last_peered":"2024-03-06T07:25:46.366240+0000","last_clean":"2024-03-06T07:25:46.366240+0000","last_became_active":"2024-03-06T07:25:45.378284+0000","last_became_peered":"2024-03-06T07:25:45.378284+0000","last_unstale":"2024-03-06T07:25:46.366240+0000","last_undegraded":"2024-03-06T07:25:46.366240+0000","last_fullsized":"2024-03-06T07:25:46.366240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:32.549564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429865+0000","last_change":"2024-03-06T07:25:45.378156+0000","last_active":"2024-03-06T07:25:46.429865+0000","last_peered":"2024-03-06T07:25:46.429865+0000","last_clean":"2024-03-06T07:25:46.429865+0000","last_became_active":"2024-03-06T07:25:45.377881+0000","last_became_peered":"2024-03-06T07:25:45.377881+0000","last_unstale":"2024-03-06T07:25:46.429865+0000","last_undegraded":"2024-03-06T07:25:46.429865+0000","last_fullsized":"2024-03-06T07:25:46.429865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:38.059664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430203+0000","last_change":"2024-03-06T07:25:45.386810+0000","last_active":"2024-03-06T07:25:46.430203+0000","last_peered":"2024-03-06T07:25:46.430203+0000","last_clean":"2024-03-06T07:25:46.430203+0000","last_became_active":"2024-03-06T07:25:45.386493+0000","last_became_peered":"2024-03-06T07:25:45.386493+0000","last_unstale":"2024-03-06T07:25:46.430203+0000","last_undegraded":"2024-03-06T07:25:46.430203+0000","last_fullsized":"2024-03-06T07:25:46.430203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:17.787221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429782+0000","last_change":"2024-03-06T07:25:45.381569+0000","last_active":"2024-03-06T07:25:46.429782+0000","last_peered":"2024-03-06T07:25:46.429782+0000","last_clean":"2024-03-06T07:25:46.429782+0000","last_became_active":"2024-03-06T07:25:45.380998+0000","last_became_peered":"2024-03-06T07:25:45.380998+0000","last_unstale":"2024-03-06T07:25:46.429782+0000","last_undegraded":"2024-03-06T07:25:46.429782+0000","last_fullsized":"2024-03-06T07:25:46.429782+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:56.002153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366298+0000","last_change":"2024-03-06T07:25:45.382381+0000","last_active":"2024-03-06T07:25:46.366298+0000","last_peered":"2024-03-06T07:25:46.366298+0000","last_clean":"2024-03-06T07:25:46.366298+0000","last_became_active":"2024-03-06T07:25:45.382104+0000","last_became_peered":"2024-03-06T07:25:45.382104+0000","last_unstale":"2024-03-06T07:25:46.366298+0000","last_undegraded":"2024-03-06T07:25:46.366298+0000","last_fullsized":"2024-03-06T07:25:46.366298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:24.383441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366515+0000","last_change":"2024-03-06T07:25:45.386120+0000","last_active":"2024-03-06T07:25:46.366515+0000","last_peered":"2024-03-06T07:25:46.366515+0000","last_clean":"2024-03-06T07:25:46.366515+0000","last_became_active":"2024-03-06T07:25:45.385846+0000","last_became_peered":"2024-03-06T07:25:45.385846+0000","last_unstale":"2024-03-06T07:25:46.366515+0000","last_undegraded":"2024-03-06T07:25:46.366515+0000","last_fullsized":"2024-03-06T07:25:46.366515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:55.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429905+0000","last_change":"2024-03-06T07:25:45.389075+0000","last_active":"2024-03-06T07:25:46.429905+0000","last_peered":"2024-03-06T07:25:46.429905+0000","last_clean":"2024-03-06T07:25:46.429905+0000","last_became_active":"2024-03-06T07:25:45.388852+0000","last_became_peered":"2024-03-06T07:25:45.388852+0000","last_unstale":"2024-03-06T07:25:46.429905+0000","last_undegraded":"2024-03-06T07:25:46.429905+0000","last_fullsized":"2024-03-06T07:25:46.429905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:04.795323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.372071+0000","last_change":"2024-03-06T07:25:45.381601+0000","last_active":"2024-03-06T07:25:46.372071+0000","last_peered":"2024-03-06T07:25:46.372071+0000","last_clean":"2024-03-06T07:25:46.372071+0000","last_became_active":"2024-03-06T07:25:45.381326+0000","last_became_peered":"2024-03-06T07:25:45.381326+0000","last_unstale":"2024-03-06T07:25:46.372071+0000","last_undegraded":"2024-03-06T07:25:46.372071+0000","last_fullsized":"2024-03-06T07:25:46.372071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:05.816904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430659+0000","last_change":"2024-03-06T07:25:45.386697+0000","last_active":"2024-03-06T07:25:46.430659+0000","last_peered":"2024-03-06T07:25:46.430659+0000","last_clean":"2024-03-06T07:25:46.430659+0000","last_became_active":"2024-03-06T07:25:45.386422+0000","last_became_peered":"2024-03-06T07:25:45.386422+0000","last_unstale":"2024-03-06T07:25:46.430659+0000","last_undegraded":"2024-03-06T07:25:46.430659+0000","last_fullsized":"2024-03-06T07:25:46.430659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:46.432954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430108+0000","last_change":"2024-03-06T07:25:45.384961+0000","last_active":"2024-03-06T07:25:46.430108+0000","last_peered":"2024-03-06T07:25:46.430108+0000","last_clean":"2024-03-06T07:25:46.430108+0000","last_became_active":"2024-03-06T07:25:45.384717+0000","last_became_peered":"2024-03-06T07:25:45.384717+0000","last_unstale":"2024-03-06T07:25:46.430108+0000","last_undegraded":"2024-03-06T07:25:46.430108+0000","last_fullsized":"2024-03-06T07:25:46.430108+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:41.419541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365866+0000","last_change":"2024-03-06T07:25:45.382921+0000","last_active":"2024-03-06T07:25:46.365866+0000","last_peered":"2024-03-06T07:25:46.365866+0000","last_clean":"2024-03-06T07:25:46.365866+0000","last_became_active":"2024-03-06T07:25:45.382556+0000","last_became_peered":"2024-03-06T07:25:45.382556+0000","last_unstale":"2024-03-06T07:25:46.365866+0000","last_undegraded":"2024-03-06T07:25:46.365866+0000","last_fullsized":"2024-03-06T07:25:46.365866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.283759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366944+0000","last_change":"2024-03-06T07:25:45.379716+0000","last_active":"2024-03-06T07:25:46.366944+0000","last_peered":"2024-03-06T07:25:46.366944+0000","last_clean":"2024-03-06T07:25:46.366944+0000","last_became_active":"2024-03-06T07:25:45.379018+0000","last_became_peered":"2024-03-06T07:25:45.379018+0000","last_unstale":"2024-03-06T07:25:46.366944+0000","last_undegraded":"2024-03-06T07:25:46.366944+0000","last_fullsized":"2024-03-06T07:25:46.366944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:23.547706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429843+0000","last_change":"2024-03-06T07:25:45.385565+0000","last_active":"2024-03-06T07:25:46.429843+0000","last_peered":"2024-03-06T07:25:46.429843+0000","last_clean":"2024-03-06T07:25:46.429843+0000","last_became_active":"2024-03-06T07:25:45.385323+0000","last_became_peered":"2024-03-06T07:25:45.385323+0000","last_unstale":"2024-03-06T07:25:46.429843+0000","last_undegraded":"2024-03-06T07:25:46.429843+0000","last_fullsized":"2024-03-06T07:25:46.429843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:21.203793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366040+0000","last_change":"2024-03-06T07:25:45.378472+0000","last_active":"2024-03-06T07:25:46.366040+0000","last_peered":"2024-03-06T07:25:46.366040+0000","last_clean":"2024-03-06T07:25:46.366040+0000","last_became_active":"2024-03-06T07:25:45.377641+0000","last_became_peered":"2024-03-06T07:25:45.377641+0000","last_unstale":"2024-03-06T07:25:46.366040+0000","last_undegraded":"2024-03-06T07:25:46.366040+0000","last_fullsized":"2024-03-06T07:25:46.366040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:10.793581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430204+0000","last_change":"2024-03-06T07:25:45.390307+0000","last_active":"2024-03-06T07:25:46.430204+0000","last_peered":"2024-03-06T07:25:46.430204+0000","last_clean":"2024-03-06T07:25:46.430204+0000","last_became_active":"2024-03-06T07:25:45.390045+0000","last_became_peered":"2024-03-06T07:25:45.390045+0000","last_unstale":"2024-03-06T07:25:46.430204+0000","last_undegraded":"2024-03-06T07:25:46.430204+0000","last_fullsized":"2024-03-06T07:25:46.430204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:37.643558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366621+0000","last_change":"2024-03-06T07:25:45.379268+0000","last_active":"2024-03-06T07:25:46.366621+0000","last_peered":"2024-03-06T07:25:46.366621+0000","last_clean":"2024-03-06T07:25:46.366621+0000","last_became_active":"2024-03-06T07:25:45.378972+0000","last_became_peered":"2024-03-06T07:25:45.378972+0000","last_unstale":"2024-03-06T07:25:46.366621+0000","last_undegraded":"2024-03-06T07:25:46.366621+0000","last_fullsized":"2024-03-06T07:25:46.366621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:32.549564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366399+0000","last_change":"2024-03-06T07:25:45.378452+0000","last_active":"2024-03-06T07:25:46.366399+0000","last_peered":"2024-03-06T07:25:46.366399+0000","last_clean":"2024-03-06T07:25:46.366399+0000","last_became_active":"2024-03-06T07:25:45.378147+0000","last_became_peered":"2024-03-06T07:25:45.378147+0000","last_unstale":"2024-03-06T07:25:46.366399+0000","last_undegraded":"2024-03-06T07:25:46.366399+0000","last_fullsized":"2024-03-06T07:25:46.366399+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:24.364522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365985+0000","last_change":"2024-03-06T07:25:45.375684+0000","last_active":"2024-03-06T07:25:46.365985+0000","last_peered":"2024-03-06T07:25:46.365985+0000","last_clean":"2024-03-06T07:25:46.365985+0000","last_became_active":"2024-03-06T07:25:45.375405+0000","last_became_peered":"2024-03-06T07:25:45.375405+0000","last_unstale":"2024-03-06T07:25:46.365985+0000","last_undegraded":"2024-03-06T07:25:46.365985+0000","last_fullsized":"2024-03-06T07:25:46.365985+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:26.625115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365938+0000","last_change":"2024-03-06T07:25:45.379285+0000","last_active":"2024-03-06T07:25:46.365938+0000","last_peered":"2024-03-06T07:25:46.365938+0000","last_clean":"2024-03-06T07:25:46.365938+0000","last_became_active":"2024-03-06T07:25:45.378923+0000","last_became_peered":"2024-03-06T07:25:45.378923+0000","last_unstale":"2024-03-06T07:25:46.365938+0000","last_undegraded":"2024-03-06T07:25:46.365938+0000","last_fullsized":"2024-03-06T07:25:46.365938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:18.896513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366464+0000","last_change":"2024-03-06T07:25:45.383967+0000","last_active":"2024-03-06T07:25:46.366464+0000","last_peered":"2024-03-06T07:25:46.366464+0000","last_clean":"2024-03-06T07:25:46.366464+0000","last_became_active":"2024-03-06T07:25:45.383673+0000","last_became_peered":"2024-03-06T07:25:45.383673+0000","last_unstale":"2024-03-06T07:25:46.366464+0000","last_undegraded":"2024-03-06T07:25:46.366464+0000","last_fullsized":"2024-03-06T07:25:46.366464+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:35.368146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366728+0000","last_change":"2024-03-06T07:25:45.374749+0000","last_active":"2024-03-06T07:25:46.366728+0000","last_peered":"2024-03-06T07:25:46.366728+0000","last_clean":"2024-03-06T07:25:46.366728+0000","last_became_active":"2024-03-06T07:25:45.374436+0000","last_became_peered":"2024-03-06T07:25:45.374436+0000","last_unstale":"2024-03-06T07:25:46.366728+0000","last_undegraded":"2024-03-06T07:25:46.366728+0000","last_fullsized":"2024-03-06T07:25:46.366728+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:13.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367344+0000","last_change":"2024-03-06T07:25:45.382541+0000","last_active":"2024-03-06T07:25:46.367344+0000","last_peered":"2024-03-06T07:25:46.367344+0000","last_clean":"2024-03-06T07:25:46.367344+0000","last_became_active":"2024-03-06T07:25:45.382228+0000","last_became_peered":"2024-03-06T07:25:45.382228+0000","last_unstale":"2024-03-06T07:25:46.367344+0000","last_undegraded":"2024-03-06T07:25:46.367344+0000","last_fullsized":"2024-03-06T07:25:46.367344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:33.664581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366983+0000","last_change":"2024-03-06T07:25:44.380973+0000","last_active":"2024-03-06T07:25:46.366983+0000","last_peered":"2024-03-06T07:25:46.366983+0000","last_clean":"2024-03-06T07:25:46.366983+0000","last_became_active":"2024-03-06T07:25:44.380687+0000","last_became_peered":"2024-03-06T07:25:44.380687+0000","last_unstale":"2024-03-06T07:25:46.366983+0000","last_undegraded":"2024-03-06T07:25:46.366983+0000","last_fullsized":"2024-03-06T07:25:46.366983+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:01.662584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367370+0000","last_change":"2024-03-06T07:25:45.389284+0000","last_active":"2024-03-06T07:25:46.367370+0000","last_peered":"2024-03-06T07:25:46.367370+0000","last_clean":"2024-03-06T07:25:46.367370+0000","last_became_active":"2024-03-06T07:25:45.385958+0000","last_became_peered":"2024-03-06T07:25:45.385958+0000","last_unstale":"2024-03-06T07:25:46.367370+0000","last_undegraded":"2024-03-06T07:25:46.367370+0000","last_fullsized":"2024-03-06T07:25:46.367370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:08.205197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367427+0000","last_change":"2024-03-06T07:25:44.373988+0000","last_active":"2024-03-06T07:25:46.367427+0000","last_peered":"2024-03-06T07:25:46.367427+0000","last_clean":"2024-03-06T07:25:46.367427+0000","last_became_active":"2024-03-06T07:25:44.373714+0000","last_became_peered":"2024-03-06T07:25:44.373714+0000","last_unstale":"2024-03-06T07:25:46.367427+0000","last_undegraded":"2024-03-06T07:25:46.367427+0000","last_fullsized":"2024-03-06T07:25:46.367427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:16.334519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366544+0000","last_change":"2024-03-06T07:25:45.380776+0000","last_active":"2024-03-06T07:25:46.366544+0000","last_peered":"2024-03-06T07:25:46.366544+0000","last_clean":"2024-03-06T07:25:46.366544+0000","last_became_active":"2024-03-06T07:25:45.380405+0000","last_became_peered":"2024-03-06T07:25:45.380405+0000","last_unstale":"2024-03-06T07:25:46.366544+0000","last_undegraded":"2024-03-06T07:25:46.366544+0000","last_fullsized":"2024-03-06T07:25:46.366544+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:41.188810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371865+0000","last_change":"2024-03-06T07:25:44.379776+0000","last_active":"2024-03-06T07:25:46.371865+0000","last_peered":"2024-03-06T07:25:46.371865+0000","last_clean":"2024-03-06T07:25:46.371865+0000","last_became_active":"2024-03-06T07:25:44.379526+0000","last_became_peered":"2024-03-06T07:25:44.379526+0000","last_unstale":"2024-03-06T07:25:46.371865+0000","last_undegraded":"2024-03-06T07:25:46.371865+0000","last_fullsized":"2024-03-06T07:25:46.371865+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:31.547735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430275+0000","last_change":"2024-03-06T07:25:45.379300+0000","last_active":"2024-03-06T07:25:46.430275+0000","last_peered":"2024-03-06T07:25:46.430275+0000","last_clean":"2024-03-06T07:25:46.430275+0000","last_became_active":"2024-03-06T07:25:45.379037+0000","last_became_peered":"2024-03-06T07:25:45.379037+0000","last_unstale":"2024-03-06T07:25:46.430275+0000","last_undegraded":"2024-03-06T07:25:46.430275+0000","last_fullsized":"2024-03-06T07:25:46.430275+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:18.149300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371837+0000","last_change":"2024-03-06T07:25:44.380909+0000","last_active":"2024-03-06T07:25:46.371837+0000","last_peered":"2024-03-06T07:25:46.371837+0000","last_clean":"2024-03-06T07:25:46.371837+0000","last_became_active":"2024-03-06T07:25:44.379555+0000","last_became_peered":"2024-03-06T07:25:44.379555+0000","last_unstale":"2024-03-06T07:25:46.371837+0000","last_undegraded":"2024-03-06T07:25:46.371837+0000","last_fullsized":"2024-03-06T07:25:46.371837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:23.362693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366010+0000","last_change":"2024-03-06T07:25:45.379221+0000","last_active":"2024-03-06T07:25:46.366010+0000","last_peered":"2024-03-06T07:25:46.366010+0000","last_clean":"2024-03-06T07:25:46.366010+0000","last_became_active":"2024-03-06T07:25:45.378903+0000","last_became_peered":"2024-03-06T07:25:45.378903+0000","last_unstale":"2024-03-06T07:25:46.366010+0000","last_undegraded":"2024-03-06T07:25:46.366010+0000","last_fullsized":"2024-03-06T07:25:46.366010+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:02.664414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366949+0000","last_change":"2024-03-06T07:25:44.375570+0000","last_active":"2024-03-06T07:25:46.366949+0000","last_peered":"2024-03-06T07:25:46.366949+0000","last_clean":"2024-03-06T07:25:46.366949+0000","last_became_active":"2024-03-06T07:25:44.375311+0000","last_became_peered":"2024-03-06T07:25:44.375311+0000","last_unstale":"2024-03-06T07:25:46.366949+0000","last_undegraded":"2024-03-06T07:25:46.366949+0000","last_fullsized":"2024-03-06T07:25:46.366949+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:34.889559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366780+0000","last_change":"2024-03-06T07:25:45.386188+0000","last_active":"2024-03-06T07:25:46.366780+0000","last_peered":"2024-03-06T07:25:46.366780+0000","last_clean":"2024-03-06T07:25:46.366780+0000","last_became_active":"2024-03-06T07:25:45.385427+0000","last_became_peered":"2024-03-06T07:25:45.385427+0000","last_unstale":"2024-03-06T07:25:46.366780+0000","last_undegraded":"2024-03-06T07:25:46.366780+0000","last_fullsized":"2024-03-06T07:25:46.366780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:17.123482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367201+0000","last_change":"2024-03-06T07:25:44.375642+0000","last_active":"2024-03-06T07:25:46.367201+0000","last_peered":"2024-03-06T07:25:46.367201+0000","last_clean":"2024-03-06T07:25:46.367201+0000","last_became_active":"2024-03-06T07:25:44.375053+0000","last_became_peered":"2024-03-06T07:25:44.375053+0000","last_unstale":"2024-03-06T07:25:46.367201+0000","last_undegraded":"2024-03-06T07:25:46.367201+0000","last_fullsized":"2024-03-06T07:25:46.367201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:01.662584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429718+0000","last_change":"2024-03-06T07:25:45.388512+0000","last_active":"2024-03-06T07:25:46.429718+0000","last_peered":"2024-03-06T07:25:46.429718+0000","last_clean":"2024-03-06T07:25:46.429718+0000","last_became_active":"2024-03-06T07:25:45.387203+0000","last_became_peered":"2024-03-06T07:25:45.387203+0000","last_unstale":"2024-03-06T07:25:46.429718+0000","last_undegraded":"2024-03-06T07:25:46.429718+0000","last_fullsized":"2024-03-06T07:25:46.429718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:07.623293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371611+0000","last_change":"2024-03-06T07:25:44.377006+0000","last_active":"2024-03-06T07:25:46.371611+0000","last_peered":"2024-03-06T07:25:46.371611+0000","last_clean":"2024-03-06T07:25:46.371611+0000","last_became_active":"2024-03-06T07:25:44.376579+0000","last_became_peered":"2024-03-06T07:25:44.376579+0000","last_unstale":"2024-03-06T07:25:46.371611+0000","last_undegraded":"2024-03-06T07:25:46.371611+0000","last_fullsized":"2024-03-06T07:25:46.371611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:54.552220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429766+0000","last_change":"2024-03-06T07:25:45.381781+0000","last_active":"2024-03-06T07:25:46.429766+0000","last_peered":"2024-03-06T07:25:46.429766+0000","last_clean":"2024-03-06T07:25:46.429766+0000","last_became_active":"2024-03-06T07:25:45.381315+0000","last_became_peered":"2024-03-06T07:25:45.381315+0000","last_unstale":"2024-03-06T07:25:46.429766+0000","last_undegraded":"2024-03-06T07:25:46.429766+0000","last_fullsized":"2024-03-06T07:25:46.429766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:56.099808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365815+0000","last_change":"2024-03-06T07:25:44.374971+0000","last_active":"2024-03-06T07:25:46.365815+0000","last_peered":"2024-03-06T07:25:46.365815+0000","last_clean":"2024-03-06T07:25:46.365815+0000","last_became_active":"2024-03-06T07:25:44.374681+0000","last_became_peered":"2024-03-06T07:25:44.374681+0000","last_unstale":"2024-03-06T07:25:46.365815+0000","last_undegraded":"2024-03-06T07:25:46.365815+0000","last_fullsized":"2024-03-06T07:25:46.365815+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:40.417711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430787+0000","last_change":"2024-03-06T07:25:45.379222+0000","last_active":"2024-03-06T07:25:46.430787+0000","last_peered":"2024-03-06T07:25:46.430787+0000","last_clean":"2024-03-06T07:25:46.430787+0000","last_became_active":"2024-03-06T07:25:45.378895+0000","last_became_peered":"2024-03-06T07:25:45.378895+0000","last_unstale":"2024-03-06T07:25:46.430787+0000","last_undegraded":"2024-03-06T07:25:46.430787+0000","last_fullsized":"2024-03-06T07:25:46.430787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:34.101979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367072+0000","last_change":"2024-03-06T07:25:44.373583+0000","last_active":"2024-03-06T07:25:46.367072+0000","last_peered":"2024-03-06T07:25:46.367072+0000","last_clean":"2024-03-06T07:25:46.367072+0000","last_became_active":"2024-03-06T07:25:44.373202+0000","last_became_peered":"2024-03-06T07:25:44.373202+0000","last_unstale":"2024-03-06T07:25:46.367072+0000","last_undegraded":"2024-03-06T07:25:46.367072+0000","last_fullsized":"2024-03-06T07:25:46.367072+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:09.791751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430494+0000","last_change":"2024-03-06T07:25:45.381537+0000","last_active":"2024-03-06T07:25:46.430494+0000","last_peered":"2024-03-06T07:25:46.430494+0000","last_clean":"2024-03-06T07:25:46.430494+0000","last_became_active":"2024-03-06T07:25:45.381229+0000","last_became_peered":"2024-03-06T07:25:45.381229+0000","last_unstale":"2024-03-06T07:25:46.430494+0000","last_undegraded":"2024-03-06T07:25:46.430494+0000","last_fullsized":"2024-03-06T07:25:46.430494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:05.951684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366584+0000","last_change":"2024-03-06T07:25:44.377068+0000","last_active":"2024-03-06T07:25:46.366584+0000","last_peered":"2024-03-06T07:25:46.366584+0000","last_clean":"2024-03-06T07:25:46.366584+0000","last_became_active":"2024-03-06T07:25:44.376696+0000","last_became_peered":"2024-03-06T07:25:44.376696+0000","last_unstale":"2024-03-06T07:25:46.366584+0000","last_undegraded":"2024-03-06T07:25:46.366584+0000","last_fullsized":"2024-03-06T07:25:46.366584+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:12.563436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371785+0000","last_change":"2024-03-06T07:25:45.382750+0000","last_active":"2024-03-06T07:25:46.371785+0000","last_peered":"2024-03-06T07:25:46.371785+0000","last_clean":"2024-03-06T07:25:46.371785+0000","last_became_active":"2024-03-06T07:25:45.382451+0000","last_became_peered":"2024-03-06T07:25:45.382451+0000","last_unstale":"2024-03-06T07:25:46.371785+0000","last_undegraded":"2024-03-06T07:25:46.371785+0000","last_fullsized":"2024-03-06T07:25:46.371785+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:08.205197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366307+0000","last_change":"2024-03-06T07:25:44.378522+0000","last_active":"2024-03-06T07:25:46.366307+0000","last_peered":"2024-03-06T07:25:46.366307+0000","last_clean":"2024-03-06T07:25:46.366307+0000","last_became_active":"2024-03-06T07:25:44.378182+0000","last_became_peered":"2024-03-06T07:25:44.378182+0000","last_unstale":"2024-03-06T07:25:46.366307+0000","last_undegraded":"2024-03-06T07:25:46.366307+0000","last_fullsized":"2024-03-06T07:25:46.366307+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.012317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366762+0000","last_change":"2024-03-06T07:25:45.384841+0000","last_active":"2024-03-06T07:25:46.366762+0000","last_peered":"2024-03-06T07:25:46.366762+0000","last_clean":"2024-03-06T07:25:46.366762+0000","last_became_active":"2024-03-06T07:25:45.384116+0000","last_became_peered":"2024-03-06T07:25:45.384116+0000","last_unstale":"2024-03-06T07:25:46.366762+0000","last_undegraded":"2024-03-06T07:25:46.366762+0000","last_fullsized":"2024-03-06T07:25:46.366762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:24.364522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429959+0000","last_change":"2024-03-06T07:25:44.375900+0000","last_active":"2024-03-06T07:25:46.429959+0000","last_peered":"2024-03-06T07:25:46.429959+0000","last_clean":"2024-03-06T07:25:46.429959+0000","last_became_active":"2024-03-06T07:25:44.375629+0000","last_became_peered":"2024-03-06T07:25:44.375629+0000","last_unstale":"2024-03-06T07:25:46.429959+0000","last_undegraded":"2024-03-06T07:25:46.429959+0000","last_fullsized":"2024-03-06T07:25:46.429959+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:42.937961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371661+0000","last_change":"2024-03-06T07:25:45.378796+0000","last_active":"2024-03-06T07:25:46.371661+0000","last_peered":"2024-03-06T07:25:46.371661+0000","last_clean":"2024-03-06T07:25:46.371661+0000","last_became_active":"2024-03-06T07:25:45.378498+0000","last_became_peered":"2024-03-06T07:25:45.378498+0000","last_unstale":"2024-03-06T07:25:46.371661+0000","last_undegraded":"2024-03-06T07:25:46.371661+0000","last_fullsized":"2024-03-06T07:25:46.371661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:23.547706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365992+0000","last_change":"2024-03-06T07:25:44.379715+0000","last_active":"2024-03-06T07:25:46.365992+0000","last_peered":"2024-03-06T07:25:46.365992+0000","last_clean":"2024-03-06T07:25:46.365992+0000","last_became_active":"2024-03-06T07:25:44.379439+0000","last_became_peered":"2024-03-06T07:25:44.379439+0000","last_unstale":"2024-03-06T07:25:46.365992+0000","last_undegraded":"2024-03-06T07:25:46.365992+0000","last_fullsized":"2024-03-06T07:25:46.365992+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:46.517433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430038+0000","last_change":"2024-03-06T07:25:45.386851+0000","last_active":"2024-03-06T07:25:46.430038+0000","last_peered":"2024-03-06T07:25:46.430038+0000","last_clean":"2024-03-06T07:25:46.430038+0000","last_became_active":"2024-03-06T07:25:45.386544+0000","last_became_peered":"2024-03-06T07:25:45.386544+0000","last_unstale":"2024-03-06T07:25:46.430038+0000","last_undegraded":"2024-03-06T07:25:46.430038+0000","last_fullsized":"2024-03-06T07:25:46.430038+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:13.630116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430418+0000","last_change":"2024-03-06T07:25:44.373942+0000","last_active":"2024-03-06T07:25:46.430418+0000","last_peered":"2024-03-06T07:25:46.430418+0000","last_clean":"2024-03-06T07:25:46.430418+0000","last_became_active":"2024-03-06T07:25:44.373655+0000","last_became_peered":"2024-03-06T07:25:44.373655+0000","last_unstale":"2024-03-06T07:25:46.430418+0000","last_undegraded":"2024-03-06T07:25:46.430418+0000","last_fullsized":"2024-03-06T07:25:46.430418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:12.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365803+0000","last_change":"2024-03-06T07:25:45.382316+0000","last_active":"2024-03-06T07:25:46.365803+0000","last_peered":"2024-03-06T07:25:46.365803+0000","last_clean":"2024-03-06T07:25:46.365803+0000","last_became_active":"2024-03-06T07:25:45.382093+0000","last_became_peered":"2024-03-06T07:25:45.382093+0000","last_unstale":"2024-03-06T07:25:46.365803+0000","last_undegraded":"2024-03-06T07:25:46.365803+0000","last_fullsized":"2024-03-06T07:25:46.365803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:01.951919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430035+0000","last_change":"2024-03-06T07:25:44.376438+0000","last_active":"2024-03-06T07:25:46.430035+0000","last_peered":"2024-03-06T07:25:46.430035+0000","last_clean":"2024-03-06T07:25:46.430035+0000","last_became_active":"2024-03-06T07:25:44.376148+0000","last_became_peered":"2024-03-06T07:25:44.376148+0000","last_unstale":"2024-03-06T07:25:46.430035+0000","last_undegraded":"2024-03-06T07:25:46.430035+0000","last_fullsized":"2024-03-06T07:25:46.430035+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:47.873251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.365768+0000","last_change":"2024-03-06T07:25:45.377048+0000","last_active":"2024-03-06T07:25:46.365768+0000","last_peered":"2024-03-06T07:25:46.365768+0000","last_clean":"2024-03-06T07:25:46.365768+0000","last_became_active":"2024-03-06T07:25:45.376731+0000","last_became_peered":"2024-03-06T07:25:45.376731+0000","last_unstale":"2024-03-06T07:25:46.365768+0000","last_undegraded":"2024-03-06T07:25:46.365768+0000","last_fullsized":"2024-03-06T07:25:46.365768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:36.831668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371567+0000","last_change":"2024-03-06T07:25:44.380239+0000","last_active":"2024-03-06T07:25:46.371567+0000","last_peered":"2024-03-06T07:25:46.371567+0000","last_clean":"2024-03-06T07:25:46.371567+0000","last_became_active":"2024-03-06T07:25:44.379998+0000","last_became_peered":"2024-03-06T07:25:44.379998+0000","last_unstale":"2024-03-06T07:25:46.371567+0000","last_undegraded":"2024-03-06T07:25:46.371567+0000","last_fullsized":"2024-03-06T07:25:46.371567+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:34.366316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430324+0000","last_change":"2024-03-06T07:25:45.377122+0000","last_active":"2024-03-06T07:25:46.430324+0000","last_peered":"2024-03-06T07:25:46.430324+0000","last_clean":"2024-03-06T07:25:46.430324+0000","last_became_active":"2024-03-06T07:25:45.376836+0000","last_became_peered":"2024-03-06T07:25:45.376836+0000","last_unstale":"2024-03-06T07:25:46.430324+0000","last_undegraded":"2024-03-06T07:25:46.430324+0000","last_fullsized":"2024-03-06T07:25:46.430324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:48.887856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367240+0000","last_change":"2024-03-06T07:25:44.379838+0000","last_active":"2024-03-06T07:25:46.367240+0000","last_peered":"2024-03-06T07:25:46.367240+0000","last_clean":"2024-03-06T07:25:46.367240+0000","last_became_active":"2024-03-06T07:25:44.379308+0000","last_became_peered":"2024-03-06T07:25:44.379308+0000","last_unstale":"2024-03-06T07:25:46.367240+0000","last_undegraded":"2024-03-06T07:25:46.367240+0000","last_fullsized":"2024-03-06T07:25:46.367240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:34.889559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366884+0000","last_change":"2024-03-06T07:25:45.385579+0000","last_active":"2024-03-06T07:25:46.366884+0000","last_peered":"2024-03-06T07:25:46.366884+0000","last_clean":"2024-03-06T07:25:46.366884+0000","last_became_active":"2024-03-06T07:25:45.380570+0000","last_became_peered":"2024-03-06T07:25:45.380570+0000","last_unstale":"2024-03-06T07:25:46.366884+0000","last_undegraded":"2024-03-06T07:25:46.366884+0000","last_fullsized":"2024-03-06T07:25:46.366884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:33.664581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371944+0000","last_change":"2024-03-06T07:25:44.380777+0000","last_active":"2024-03-06T07:25:46.371944+0000","last_peered":"2024-03-06T07:25:46.371944+0000","last_clean":"2024-03-06T07:25:46.371944+0000","last_became_active":"2024-03-06T07:25:44.380528+0000","last_became_peered":"2024-03-06T07:25:44.380528+0000","last_unstale":"2024-03-06T07:25:46.371944+0000","last_undegraded":"2024-03-06T07:25:46.371944+0000","last_fullsized":"2024-03-06T07:25:46.371944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:23.381612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366891+0000","last_change":"2024-03-06T07:25:45.379971+0000","last_active":"2024-03-06T07:25:46.366891+0000","last_peered":"2024-03-06T07:25:46.366891+0000","last_clean":"2024-03-06T07:25:46.366891+0000","last_became_active":"2024-03-06T07:25:45.379445+0000","last_became_peered":"2024-03-06T07:25:45.379445+0000","last_unstale":"2024-03-06T07:25:46.366891+0000","last_undegraded":"2024-03-06T07:25:46.366891+0000","last_fullsized":"2024-03-06T07:25:46.366891+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:41.188810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371866+0000","last_change":"2024-03-06T07:25:44.377950+0000","last_active":"2024-03-06T07:25:46.371866+0000","last_peered":"2024-03-06T07:25:46.371866+0000","last_clean":"2024-03-06T07:25:46.371866+0000","last_became_active":"2024-03-06T07:25:44.377692+0000","last_became_peered":"2024-03-06T07:25:44.377692+0000","last_unstale":"2024-03-06T07:25:46.371866+0000","last_undegraded":"2024-03-06T07:25:46.371866+0000","last_fullsized":"2024-03-06T07:25:46.371866+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:16.121652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430307+0000","last_change":"2024-03-06T07:25:45.386164+0000","last_active":"2024-03-06T07:25:46.430307+0000","last_peered":"2024-03-06T07:25:46.430307+0000","last_clean":"2024-03-06T07:25:46.430307+0000","last_became_active":"2024-03-06T07:25:45.385917+0000","last_became_peered":"2024-03-06T07:25:45.385917+0000","last_unstale":"2024-03-06T07:25:46.430307+0000","last_undegraded":"2024-03-06T07:25:46.430307+0000","last_fullsized":"2024-03-06T07:25:46.430307+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:57.773005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430515+0000","last_change":"2024-03-06T07:25:44.371606+0000","last_active":"2024-03-06T07:25:46.430515+0000","last_peered":"2024-03-06T07:25:46.430515+0000","last_clean":"2024-03-06T07:25:46.430515+0000","last_became_active":"2024-03-06T07:25:44.371097+0000","last_became_peered":"2024-03-06T07:25:44.371097+0000","last_unstale":"2024-03-06T07:25:46.430515+0000","last_undegraded":"2024-03-06T07:25:46.430515+0000","last_fullsized":"2024-03-06T07:25:46.430515+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:54.188838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429890+0000","last_change":"2024-03-06T07:25:45.382784+0000","last_active":"2024-03-06T07:25:46.429890+0000","last_peered":"2024-03-06T07:25:46.429890+0000","last_clean":"2024-03-06T07:25:46.429890+0000","last_became_active":"2024-03-06T07:25:45.382492+0000","last_became_peered":"2024-03-06T07:25:45.382492+0000","last_unstale":"2024-03-06T07:25:46.429890+0000","last_undegraded":"2024-03-06T07:25:46.429890+0000","last_fullsized":"2024-03-06T07:25:46.429890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:55.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430282+0000","last_change":"2024-03-06T07:25:44.376345+0000","last_active":"2024-03-06T07:25:46.430282+0000","last_peered":"2024-03-06T07:25:46.430282+0000","last_clean":"2024-03-06T07:25:46.430282+0000","last_became_active":"2024-03-06T07:25:44.376085+0000","last_became_peered":"2024-03-06T07:25:44.376085+0000","last_unstale":"2024-03-06T07:25:46.430282+0000","last_undegraded":"2024-03-06T07:25:46.430282+0000","last_fullsized":"2024-03-06T07:25:46.430282+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:41.262045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429950+0000","last_change":"2024-03-06T07:25:45.377034+0000","last_active":"2024-03-06T07:25:46.429950+0000","last_peered":"2024-03-06T07:25:46.429950+0000","last_clean":"2024-03-06T07:25:46.429950+0000","last_became_active":"2024-03-06T07:25:45.376751+0000","last_became_peered":"2024-03-06T07:25:45.376751+0000","last_unstale":"2024-03-06T07:25:46.429950+0000","last_undegraded":"2024-03-06T07:25:46.429950+0000","last_fullsized":"2024-03-06T07:25:46.429950+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:42.868277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430059+0000","last_change":"2024-03-06T07:25:44.377832+0000","last_active":"2024-03-06T07:25:46.430059+0000","last_peered":"2024-03-06T07:25:46.430059+0000","last_clean":"2024-03-06T07:25:46.430059+0000","last_became_active":"2024-03-06T07:25:44.377535+0000","last_became_peered":"2024-03-06T07:25:44.377535+0000","last_unstale":"2024-03-06T07:25:46.430059+0000","last_undegraded":"2024-03-06T07:25:46.430059+0000","last_fullsized":"2024-03-06T07:25:46.430059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:19.362948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429961+0000","last_change":"2024-03-06T07:25:45.384259+0000","last_active":"2024-03-06T07:25:46.429961+0000","last_peered":"2024-03-06T07:25:46.429961+0000","last_clean":"2024-03-06T07:25:46.429961+0000","last_became_active":"2024-03-06T07:25:45.384008+0000","last_became_peered":"2024-03-06T07:25:45.384008+0000","last_unstale":"2024-03-06T07:25:46.429961+0000","last_undegraded":"2024-03-06T07:25:46.429961+0000","last_fullsized":"2024-03-06T07:25:46.429961+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:13.965458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429716+0000","last_change":"2024-03-06T07:25:44.371371+0000","last_active":"2024-03-06T07:25:46.429716+0000","last_peered":"2024-03-06T07:25:46.429716+0000","last_clean":"2024-03-06T07:25:46.429716+0000","last_became_active":"2024-03-06T07:25:44.371076+0000","last_became_peered":"2024-03-06T07:25:44.371076+0000","last_unstale":"2024-03-06T07:25:46.429716+0000","last_undegraded":"2024-03-06T07:25:46.429716+0000","last_fullsized":"2024-03-06T07:25:46.429716+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:37.057834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367332+0000","last_change":"2024-03-06T07:25:45.375740+0000","last_active":"2024-03-06T07:25:46.367332+0000","last_peered":"2024-03-06T07:25:46.367332+0000","last_clean":"2024-03-06T07:25:46.367332+0000","last_became_active":"2024-03-06T07:25:45.375476+0000","last_became_peered":"2024-03-06T07:25:45.375476+0000","last_unstale":"2024-03-06T07:25:46.367332+0000","last_undegraded":"2024-03-06T07:25:46.367332+0000","last_fullsized":"2024-03-06T07:25:46.367332+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:04.143346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366140+0000","last_change":"2024-03-06T07:25:44.378787+0000","last_active":"2024-03-06T07:25:46.366140+0000","last_peered":"2024-03-06T07:25:46.366140+0000","last_clean":"2024-03-06T07:25:46.366140+0000","last_became_active":"2024-03-06T07:25:44.378481+0000","last_became_peered":"2024-03-06T07:25:44.378481+0000","last_unstale":"2024-03-06T07:25:46.366140+0000","last_undegraded":"2024-03-06T07:25:46.366140+0000","last_fullsized":"2024-03-06T07:25:46.366140+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:16.785392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430351+0000","last_change":"2024-03-06T07:25:45.383330+0000","last_active":"2024-03-06T07:25:46.430351+0000","last_peered":"2024-03-06T07:25:46.430351+0000","last_clean":"2024-03-06T07:25:46.430351+0000","last_became_active":"2024-03-06T07:25:45.382932+0000","last_became_peered":"2024-03-06T07:25:45.382932+0000","last_unstale":"2024-03-06T07:25:46.430351+0000","last_undegraded":"2024-03-06T07:25:46.430351+0000","last_fullsized":"2024-03-06T07:25:46.430351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:34.291075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.369006+0000","last_change":"2024-03-06T07:25:44.378760+0000","last_active":"2024-03-06T07:25:46.369006+0000","last_peered":"2024-03-06T07:25:46.369006+0000","last_clean":"2024-03-06T07:25:46.369006+0000","last_became_active":"2024-03-06T07:25:44.377480+0000","last_became_peered":"2024-03-06T07:25:44.377480+0000","last_unstale":"2024-03-06T07:25:46.369006+0000","last_undegraded":"2024-03-06T07:25:46.369006+0000","last_fullsized":"2024-03-06T07:25:46.369006+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:56.771175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366697+0000","last_change":"2024-03-06T07:25:45.380118+0000","last_active":"2024-03-06T07:25:46.366697+0000","last_peered":"2024-03-06T07:25:46.366697+0000","last_clean":"2024-03-06T07:25:46.366697+0000","last_became_active":"2024-03-06T07:25:45.379416+0000","last_became_peered":"2024-03-06T07:25:45.379416+0000","last_unstale":"2024-03-06T07:25:46.366697+0000","last_undegraded":"2024-03-06T07:25:46.366697+0000","last_fullsized":"2024-03-06T07:25:46.366697+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:05.951684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366479+0000","last_change":"2024-03-06T07:25:44.376028+0000","last_active":"2024-03-06T07:25:46.366479+0000","last_peered":"2024-03-06T07:25:46.366479+0000","last_clean":"2024-03-06T07:25:46.366479+0000","last_became_active":"2024-03-06T07:25:44.375743+0000","last_became_peered":"2024-03-06T07:25:44.375743+0000","last_unstale":"2024-03-06T07:25:46.366479+0000","last_undegraded":"2024-03-06T07:25:46.366479+0000","last_fullsized":"2024-03-06T07:25:46.366479+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:22.498536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367018+0000","last_change":"2024-03-06T07:25:45.384459+0000","last_active":"2024-03-06T07:25:46.367018+0000","last_peered":"2024-03-06T07:25:46.367018+0000","last_clean":"2024-03-06T07:25:46.367018+0000","last_became_active":"2024-03-06T07:25:45.383764+0000","last_became_peered":"2024-03-06T07:25:45.383764+0000","last_unstale":"2024-03-06T07:25:46.367018+0000","last_undegraded":"2024-03-06T07:25:46.367018+0000","last_fullsized":"2024-03-06T07:25:46.367018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:05.951684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366757+0000","last_change":"2024-03-06T07:25:44.373902+0000","last_active":"2024-03-06T07:25:46.366757+0000","last_peered":"2024-03-06T07:25:46.366757+0000","last_clean":"2024-03-06T07:25:46.366757+0000","last_became_active":"2024-03-06T07:25:44.372777+0000","last_became_peered":"2024-03-06T07:25:44.372777+0000","last_unstale":"2024-03-06T07:25:46.366757+0000","last_undegraded":"2024-03-06T07:25:46.366757+0000","last_fullsized":"2024-03-06T07:25:46.366757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:16.334519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430076+0000","last_change":"2024-03-06T07:25:45.383160+0000","last_active":"2024-03-06T07:25:46.430076+0000","last_peered":"2024-03-06T07:25:46.430076+0000","last_clean":"2024-03-06T07:25:46.430076+0000","last_became_active":"2024-03-06T07:25:45.382380+0000","last_became_peered":"2024-03-06T07:25:45.382380+0000","last_unstale":"2024-03-06T07:25:46.430076+0000","last_undegraded":"2024-03-06T07:25:46.430076+0000","last_fullsized":"2024-03-06T07:25:46.430076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:24.364522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371809+0000","last_change":"2024-03-06T07:25:44.380512+0000","last_active":"2024-03-06T07:25:46.371809+0000","last_peered":"2024-03-06T07:25:46.371809+0000","last_clean":"2024-03-06T07:25:46.371809+0000","last_became_active":"2024-03-06T07:25:44.380118+0000","last_became_peered":"2024-03-06T07:25:44.380118+0000","last_unstale":"2024-03-06T07:25:46.371809+0000","last_undegraded":"2024-03-06T07:25:46.371809+0000","last_fullsized":"2024-03-06T07:25:46.371809+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:12.963628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.371764+0000","last_change":"2024-03-06T07:25:45.378630+0000","last_active":"2024-03-06T07:25:46.371764+0000","last_peered":"2024-03-06T07:25:46.371764+0000","last_clean":"2024-03-06T07:25:46.371764+0000","last_became_active":"2024-03-06T07:25:45.378372+0000","last_became_peered":"2024-03-06T07:25:45.378372+0000","last_unstale":"2024-03-06T07:25:46.371764+0000","last_undegraded":"2024-03-06T07:25:46.371764+0000","last_fullsized":"2024-03-06T07:25:46.371764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:49.071208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367294+0000","last_change":"2024-03-06T07:25:44.374667+0000","last_active":"2024-03-06T07:25:46.367294+0000","last_peered":"2024-03-06T07:25:46.367294+0000","last_clean":"2024-03-06T07:25:46.367294+0000","last_became_active":"2024-03-06T07:25:44.374401+0000","last_became_peered":"2024-03-06T07:25:44.374401+0000","last_unstale":"2024-03-06T07:25:46.367294+0000","last_undegraded":"2024-03-06T07:25:46.367294+0000","last_fullsized":"2024-03-06T07:25:46.367294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:09.791751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429772+0000","last_change":"2024-03-06T07:25:45.390230+0000","last_active":"2024-03-06T07:25:46.429772+0000","last_peered":"2024-03-06T07:25:46.429772+0000","last_clean":"2024-03-06T07:25:46.429772+0000","last_became_active":"2024-03-06T07:25:45.389955+0000","last_became_peered":"2024-03-06T07:25:45.389955+0000","last_unstale":"2024-03-06T07:25:46.429772+0000","last_undegraded":"2024-03-06T07:25:46.429772+0000","last_fullsized":"2024-03-06T07:25:46.429772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:56.684522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367229+0000","last_change":"2024-03-06T07:25:44.373098+0000","last_active":"2024-03-06T07:25:46.367229+0000","last_peered":"2024-03-06T07:25:46.367229+0000","last_clean":"2024-03-06T07:25:46.367229+0000","last_became_active":"2024-03-06T07:25:44.371542+0000","last_became_peered":"2024-03-06T07:25:44.371542+0000","last_unstale":"2024-03-06T07:25:46.367229+0000","last_undegraded":"2024-03-06T07:25:46.367229+0000","last_fullsized":"2024-03-06T07:25:46.367229+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:49.360890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.367079+0000","last_change":"2024-03-06T07:25:45.379807+0000","last_active":"2024-03-06T07:25:46.367079+0000","last_peered":"2024-03-06T07:25:46.367079+0000","last_clean":"2024-03-06T07:25:46.367079+0000","last_became_active":"2024-03-06T07:25:45.379382+0000","last_became_peered":"2024-03-06T07:25:45.379382+0000","last_unstale":"2024-03-06T07:25:46.367079+0000","last_undegraded":"2024-03-06T07:25:46.367079+0000","last_fullsized":"2024-03-06T07:25:46.367079+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:49.071208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366397+0000","last_change":"2024-03-06T07:25:44.377688+0000","last_active":"2024-03-06T07:25:46.366397+0000","last_peered":"2024-03-06T07:25:46.366397+0000","last_clean":"2024-03-06T07:25:46.366397+0000","last_became_active":"2024-03-06T07:25:44.377411+0000","last_became_peered":"2024-03-06T07:25:44.377411+0000","last_unstale":"2024-03-06T07:25:46.366397+0000","last_undegraded":"2024-03-06T07:25:46.366397+0000","last_fullsized":"2024-03-06T07:25:46.366397+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:12.563436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.366107+0000","last_change":"2024-03-06T07:25:45.376362+0000","last_active":"2024-03-06T07:25:46.366107+0000","last_peered":"2024-03-06T07:25:46.366107+0000","last_clean":"2024-03-06T07:25:46.366107+0000","last_became_active":"2024-03-06T07:25:45.376117+0000","last_became_peered":"2024-03-06T07:25:45.376117+0000","last_unstale":"2024-03-06T07:25:46.366107+0000","last_undegraded":"2024-03-06T07:25:46.366107+0000","last_fullsized":"2024-03-06T07:25:46.366107+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:44.355897+0000","last_clean_scrub_stamp":"2024-03-06T07:25:44.355897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:13.565265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.430228+0000","last_change":"2024-03-06T07:25:44.382115+0000","last_active":"2024-03-06T07:25:46.430228+0000","last_peered":"2024-03-06T07:25:46.430228+0000","last_clean":"2024-03-06T07:25:46.430228+0000","last_became_active":"2024-03-06T07:25:44.381866+0000","last_became_peered":"2024-03-06T07:25:44.381866+0000","last_unstale":"2024-03-06T07:25:46.430228+0000","last_undegraded":"2024-03-06T07:25:46.430228+0000","last_fullsized":"2024-03-06T07:25:46.430228+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:43.354067+0000","last_clean_scrub_stamp":"2024-03-06T07:25:43.354067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:03.141516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":276,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:25:46.429776+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:25:46.429776+0000","last_peered":"2024-03-06T07:25:46.429776+0000","last_clean":"2024-03-06T07:25:46.429776+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:25:46.429776+0000","last_undegraded":"2024-03-06T07:25:46.429776+0000","last_fullsized":"2024-03-06T07:25:46.429776+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":1005926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335524,"statfs":{"total":96636764160,"available":96599576576,"internally_reserved":0,"allocated":995328,"data_stored":416182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335520,"statfs":{"total":96636764160,"available":96599572480,"internally_reserved":0,"allocated":999424,"data_stored":417122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":983040,"data_stored":415488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":417100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":987136,"data_stored":416070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":415682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1581056,"data_stored":1005890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:49.328 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:49.328 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:49.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.470+0000 7f46d0b3e640 1 Processor -- start 2024-03-06T07:25:49.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.470+0000 7f46d0b3e640 1 -- start start 2024-03-06T07:25:49.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46d0b3e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f46cc0b4970 0x7f46cc06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46d0b3e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f46cc06d5c0 0x7f46cc0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46d0b3e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc15ff90 con 0x7f46cc15bb40 2024-03-06T07:25:49.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46d0b3e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc0b4dd0 con 0x7f46cc06d5c0 2024-03-06T07:25:49.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46d0b3e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc06c910 con 0x7f46cc0b4970 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46cad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f46cc06d5c0 0x7f46cc0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46ca59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f46cc15bb40 0x7f46cc15f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58082/0 (socket says 172.21.15.77:58082) 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46ca59c640 1 -- 172.21.15.77:0/4093684523 learned_addr learned my addr 172.21.15.77:0/4093684523 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46c9d9b640 1 --2- 172.21.15.77:0/4093684523 >> v2:172.21.15.158:3300/0 conn(0x7f46cc0b4970 0x7f46cc06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46cad9d640 1 -- 172.21.15.77:0/4093684523 >> v2:172.21.15.158:3300/0 conn(0x7f46cc0b4970 msgr2=0x7f46cc06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46cad9d640 1 --2- 172.21.15.77:0/4093684523 >> v2:172.21.15.158:3300/0 conn(0x7f46cc0b4970 0x7f46cc06d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46cad9d640 1 -- 172.21.15.77:0/4093684523 >> v1:172.21.15.77:6789/0 conn(0x7f46cc15bb40 legacy=0x7f46cc15f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46cad9d640 1 -- 172.21.15.77:0/4093684523 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc160270 con 0x7f46cc06d5c0 2024-03-06T07:25:49.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.471+0000 7f46c959a640 1 -- 172.21.15.77:0/4093684523 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682908165 0 0) 0x7f46cc15ff90 con 0x7f46cc15bb40 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.472+0000 7f46cad9d640 1 --2- 172.21.15.77:0/4093684523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc06d5c0 0x7f46cc0cfa90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f46cc06bfe0 tx=0x7f46b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0201f63195c2197 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.472+0000 7f46c959a640 1 -- 172.21.15.77:0/4093684523 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4039070 con 0x7f46cc06d5c0 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.472+0000 7f46c959a640 1 -- 172.21.15.77:0/4093684523 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b402cc80 con 0x7f46cc06d5c0 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.472+0000 7f46c959a640 1 -- 172.21.15.77:0/4093684523 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4034660 con 0x7f46cc06d5c0 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.473+0000 7f46d0b3e640 1 -- 172.21.15.77:0/4093684523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc06d5c0 msgr2=0x7f46cc0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.473+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/4093684523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc06d5c0 0x7f46cc0cfa90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f46cc06bfe0 tx=0x7f46b402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.473+0000 7f46d0b3e640 1 -- 172.21.15.77:0/4093684523 shutdown_connections 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.473+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/4093684523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc06d5c0 0x7f46cc0cfa90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.473+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/4093684523 >> v2:172.21.15.158:3300/0 conn(0x7f46cc0b4970 0x7f46cc06d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.473+0000 7f46d0b3e640 1 -- 172.21.15.77:0/4093684523 >> 172.21.15.77:0/4093684523 conn(0x7f46cc06b190 msgr2=0x7f46cc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.473+0000 7f46d0b3e640 1 -- 172.21.15.77:0/4093684523 shutdown_connections 2024-03-06T07:25:49.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.473+0000 7f46d0b3e640 1 -- 172.21.15.77:0/4093684523 wait complete. 2024-03-06T07:25:49.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.474+0000 7f46d0b3e640 1 Processor -- start 2024-03-06T07:25:49.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.474+0000 7f46d0b3e640 1 -- start start 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46d0b3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc06d5c0 0x7f46cc14a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46d0b3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc0b4970 0x7f46cc15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46d0b3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46cc15bb40 0x7f46cc0b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46d0b3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc15bf20 con 0x7f46cc0b4970 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46d0b3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc06cb60 con 0x7f46cc06d5c0 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46d0b3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc15c0a0 con 0x7f46cc15bb40 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc0b4970 0x7f46cc15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc0b4970 0x7f46cc15c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52094/0 (socket says 172.21.15.77:52094) 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 -- 172.21.15.77:0/1357544843 learned_addr learned my addr 172.21.15.77:0/1357544843 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46ca59c640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc06d5c0 0x7f46cc14a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46cad9d640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46cc15bb40 0x7f46cc0b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 -- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46cc15bb40 msgr2=0x7f46cc0b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46cc15bb40 0x7f46cc0b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 -- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc06d5c0 msgr2=0x7f46cc14a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc06d5c0 0x7f46cc14a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 -- 172.21.15.77:0/1357544843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc14a970 con 0x7f46cc0b4970 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46ca59c640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc06d5c0 0x7f46cc14a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46cad9d640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46cc15bb40 0x7f46cc0b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c9d9b640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc0b4970 0x7f46cc15c2c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f46c400e9b0 tx=0x7f46c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c2ffd640 1 -- 172.21.15.77:0/1357544843 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4017070 con 0x7f46cc0b4970 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46c2ffd640 1 -- 172.21.15.77:0/1357544843 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c400c9a0 con 0x7f46cc0b4970 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.475+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc0af9c0 con 0x7f46cc0b4970 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.476+0000 7f46c2ffd640 1 -- 172.21.15.77:0/1357544843 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400cd90 con 0x7f46cc0b4970 2024-03-06T07:25:49.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.476+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc0aff80 con 0x7f46cc0b4970 2024-03-06T07:25:49.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.477+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc0732b0 con 0x7f46cc0b4970 2024-03-06T07:25:49.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.477+0000 7f46c2ffd640 1 -- 172.21.15.77:0/1357544843 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46c401a070 con 0x7f46cc0b4970 2024-03-06T07:25:49.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.477+0000 7f46c2ffd640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f46b006b4d0 0x7f46b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.477+0000 7f46c2ffd640 1 -- 172.21.15.77:0/1357544843 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46c408a1b0 con 0x7f46cc0b4970 2024-03-06T07:25:49.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.477+0000 7f46ca59c640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f46b006b4d0 0x7f46b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.478+0000 7f46ca59c640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f46b006b4d0 0x7f46b006d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f46bc000f60 tx=0x7f46bc007ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.482+0000 7f46c2ffd640 1 -- 172.21.15.77:0/1357544843 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46c40590c0 con 0x7f46cc0b4970 2024-03-06T07:25:49.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.721+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f46cc068ff0 con 0x7f46cc0b4970 2024-03-06T07:25:49.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.722+0000 7f46c2ffd640 1 -- 172.21.15.77:0/1357544843 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f46c40589c0 con 0x7f46cc0b4970 2024-03-06T07:25:49.724 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:25:49.724 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:25:49.724 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:25:49.725 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:25:49.725 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:49.725 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:25:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f46b006b4d0 msgr2=0x7f46b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f46b006b4d0 0x7f46b006d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f46bc000f60 tx=0x7f46bc007ca0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc0b4970 msgr2=0x7f46cc15c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc0b4970 0x7f46cc15c2c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f46c400e9b0 tx=0x7f46c400ee70 comp rx=0 tx=0).stop 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 shutdown_connections 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f46b006b4d0 0x7f46b006d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46cc0b4970 0x7f46cc15c2c0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f46cc15bb40 0x7f46cc0b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 --2- 172.21.15.77:0/1357544843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f46cc06d5c0 0x7f46cc14a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 >> 172.21.15.77:0/1357544843 conn(0x7f46cc06b190 msgr2=0x7f46cc15a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 shutdown_connections 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.725+0000 7f46d0b3e640 1 -- 172.21.15.77:0/1357544843 wait complete. 2024-03-06T07:25:49.738 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.837+0000 7fd926e81640 1 Processor -- start 2024-03-06T07:25:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.837+0000 7fd926e81640 1 -- start start 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd926e81640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd9200d2920 0x7fd9200d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd926e81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9200d0650 0x7fd9200d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd926e81640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92015e470 con 0x7fd92015e090 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd926e81640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9200d18f0 con 0x7fd9200d2920 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd926e81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200b3cf0 con 0x7fd9200d0650 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd924c1d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd92015e090 0x7fd9200b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58092/0 (socket says 172.21.15.77:58092) 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd924c1d640 1 -- 172.21.15.77:0/2829413861 learned_addr learned my addr 172.21.15.77:0/2829413861 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd91ffff640 1 --2- 172.21.15.77:0/2829413861 >> v2:172.21.15.158:3300/0 conn(0x7fd9200d2920 0x7fd9200d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd92541e640 1 --2- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9200d0650 0x7fd9200d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd91f7fe640 1 -- 172.21.15.77:0/2829413861 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761048351 0 0) 0x7fd92015e470 con 0x7fd92015e090 2024-03-06T07:25:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd91f7fe640 1 -- 172.21.15.77:0/2829413861 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd910003520 con 0x7fd92015e090 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd91ffff640 1 -- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9200d0650 msgr2=0x7fd9200d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd91ffff640 1 --2- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9200d0650 0x7fd9200d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd91ffff640 1 -- 172.21.15.77:0/2829413861 >> v1:172.21.15.77:6789/0 conn(0x7fd92015e090 legacy=0x7fd9200b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd91ffff640 1 -- 172.21.15.77:0/2829413861 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200d1b90 con 0x7fd9200d2920 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.838+0000 7fd92541e640 1 --2- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9200d0650 0x7fd9200d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.839+0000 7fd91ffff640 1 --2- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 0x7fd9200d47a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0097a0 tx=0x7fd90c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5e254eb44e48ade server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.839+0000 7fd91f7fe640 1 -- 172.21.15.77:0/2829413861 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c035020 con 0x7fd9200d2920 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.839+0000 7fd91f7fe640 1 -- 172.21.15.77:0/2829413861 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c031830 con 0x7fd9200d2920 2024-03-06T07:25:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.839+0000 7fd91f7fe640 1 -- 172.21.15.77:0/2829413861 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c031be0 con 0x7fd9200d2920 2024-03-06T07:25:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.840+0000 7fd926e81640 1 -- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 msgr2=0x7fd9200d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.840+0000 7fd926e81640 1 --2- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 0x7fd9200d47a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0097a0 tx=0x7fd90c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.840+0000 7fd926e81640 1 -- 172.21.15.77:0/2829413861 shutdown_connections 2024-03-06T07:25:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.840+0000 7fd926e81640 1 --2- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9200d0650 0x7fd9200d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.840+0000 7fd926e81640 1 --2- 172.21.15.77:0/2829413861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 0x7fd9200d47a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.840+0000 7fd926e81640 1 -- 172.21.15.77:0/2829413861 >> 172.21.15.77:0/2829413861 conn(0x7fd9200d3c80 msgr2=0x7fd9200d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.840+0000 7fd926e81640 1 -- 172.21.15.77:0/2829413861 shutdown_connections 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.840+0000 7fd926e81640 1 -- 172.21.15.77:0/2829413861 wait complete. 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.841+0000 7fd926e81640 1 Processor -- start 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.841+0000 7fd926e81640 1 -- start start 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd926e81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9200d0650 0x7fd92014f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd926e81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 0x7fd9200cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd926e81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd92015e090 0x7fd9200b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd926e81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200b2df0 con 0x7fd9200d0650 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd926e81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200cee30 con 0x7fd9200d2920 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd926e81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200ce800 con 0x7fd92015e090 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9200d0650 0x7fd92014f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9200d0650 0x7fd92014f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52098/0 (socket says 172.21.15.77:52098) 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 -- 172.21.15.77:0/1090803020 learned_addr learned my addr 172.21.15.77:0/1090803020 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd92541e640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd92015e090 0x7fd9200b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd91ffff640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 0x7fd9200cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 -- 172.21.15.77:0/1090803020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd92015e090 msgr2=0x7fd9200b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd92015e090 0x7fd9200b85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 -- 172.21.15.77:0/1090803020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 msgr2=0x7fd9200cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 0x7fd9200cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 -- 172.21.15.77:0/1090803020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200cb290 con 0x7fd9200d0650 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd924c1d640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9200d0650 0x7fd92014f1f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c390 tx=0x7fd91400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd91d7fa640 1 -- 172.21.15.77:0/1090803020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914017070 con 0x7fd9200d0650 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd91d7fa640 1 -- 172.21.15.77:0/1090803020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd914005ce0 con 0x7fd9200d0650 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.842+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9200cded0 con 0x7fd9200d0650 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.843+0000 7fd91d7fa640 1 -- 172.21.15.77:0/1090803020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9140072f0 con 0x7fd9200d0650 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.843+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9200b8090 con 0x7fd9200d0650 2024-03-06T07:25:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.844+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92004f330 con 0x7fd9200d0650 2024-03-06T07:25:49.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.844+0000 7fd91d7fa640 1 -- 172.21.15.77:0/1090803020 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd91401a070 con 0x7fd9200d0650 2024-03-06T07:25:49.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.844+0000 7fd91d7fa640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.845+0000 7fd91ffff640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.845+0000 7fd91d7fa640 1 -- 172.21.15.77:0/1090803020 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd914089400 con 0x7fd9200d0650 2024-03-06T07:25:49.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.845+0000 7fd91ffff640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd90406b4d0 0x7fd90406d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd90c036000 tx=0x7fd90c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:49.849+0000 7fd91d7fa640 1 -- 172.21.15.77:0/1090803020 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd914057cc0 con 0x7fd9200d0650 2024-03-06T07:25:50.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.056+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9200b68a0 con 0x7fd9200d0650 2024-03-06T07:25:50.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.056+0000 7fd91d7fa640 1 -- 172.21.15.77:0/1090803020 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd91405bd60 con 0x7fd9200d0650 2024-03-06T07:25:50.058 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:50.058 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3408486989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3408486989},{"type":"v1","addr":"172.21.15.158:6835","nonce":3408486989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/324866984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":324866984},{"type":"v1","addr":"172.21.15.77:6836","nonce":324866984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2426038221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2426038221},{"type":"v1","addr":"172.21.15.77:6837","nonce":2426038221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:44.712557+0000","modified":"2024-03-06T07:25:47.370110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6930},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6930":{"gid":6930,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/3775835533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3775835533},{"type":"v1","addr":"172.21.15.158:6834","nonce":3775835533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6930,"qdb_cluster":[6930]},"id":6}]} 2024-03-06T07:25:50.059 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.058+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd90406b4d0 msgr2=0x7fd90406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.058+0000 7fd926e81640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd90406b4d0 0x7fd90406d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd90c036000 tx=0x7fd90c03c040 comp rx=0 tx=0).stop 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.058+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9200d0650 msgr2=0x7fd92014f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.058+0000 7fd926e81640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9200d0650 0x7fd92014f1f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c390 tx=0x7fd91400c790 comp rx=0 tx=0).stop 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.059+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 shutdown_connections 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.059+0000 7fd926e81640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd92015e090 0x7fd9200b85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.059+0000 7fd926e81640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9200d2920 0x7fd9200cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.059+0000 7fd926e81640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.059+0000 7fd926e81640 1 --2- 172.21.15.77:0/1090803020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9200d0650 0x7fd92014f1f0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.059+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 >> 172.21.15.77:0/1090803020 conn(0x7fd9200d3c80 msgr2=0x7fd9201618a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.059+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 shutdown_connections 2024-03-06T07:25:50.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.059+0000 7fd926e81640 1 -- 172.21.15.77:0/1090803020 wait complete. 2024-03-06T07:25:50.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:44.712557+0000', 'modified': '2024-03-06T07:25:47.370110+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6930}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6930': {'gid': 6930, 'name': 'd', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6834/3775835533', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6832', 'nonce': 3775835533}, {'type': 'v1', 'addr': '172.21.15.158:6834', 'nonce': 3775835533}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6930, 'qdb_cluster': [6930]} 2024-03-06T07:25:50.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:50.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.576+0000 7f36b8fa0640 1 Processor -- start 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.576+0000 7f36b8fa0640 1 -- start start 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b8fa0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36b4146ac0 0x7f36b40b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b8fa0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b8fa0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40b5830 con 0x7f36b406afb0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b8fa0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b40b5b10 con 0x7f36b4146ac0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b8fa0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40724b0 con 0x7f36b406f500 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b3fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b37fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36b4146ac0 0x7f36b40b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b37fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36b4146ac0 0x7f36b40b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:46988/0 (socket says 172.21.15.77:46988) 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b2ffd640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f36b406afb0 0x7f36b40d0580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44690/0 (socket says 172.21.15.77:44690) 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b37fe640 1 -- 172.21.15.77:0/1965664453 learned_addr learned my addr 172.21.15.77:0/1965664453 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b3fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b406b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:37682/0 (socket says 172.21.15.77:37682) 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b3fff640 1 -- 172.21.15.77:0/1965664453 >> v2:172.21.15.158:3300/0 conn(0x7f36b4146ac0 msgr2=0x7f36b40b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b3fff640 1 --2- 172.21.15.77:0/1965664453 >> v2:172.21.15.158:3300/0 conn(0x7f36b4146ac0 0x7f36b40b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b3fff640 1 -- 172.21.15.77:0/1965664453 >> v1:172.21.15.158:6789/0 conn(0x7f36b406afb0 legacy=0x7f36b40d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b3fff640 1 -- 172.21.15.77:0/1965664453 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4072790 con 0x7f36b406f500 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.577+0000 7f36b37fe640 1 --2- 172.21.15.77:0/1965664453 >> v2:172.21.15.158:3300/0 conn(0x7f36b4146ac0 0x7f36b40b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.578+0000 7f36b3fff640 1 --2- 172.21.15.77:0/1965664453 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b406b460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009d90 tx=0x7f36ac02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fabb8ea6743a42f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.578+0000 7f36b27fc640 1 -- 172.21.15.77:0/1965664453 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac039070 con 0x7f36b406f500 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.578+0000 7f36b27fc640 1 -- 172.21.15.77:0/1965664453 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac02cc50 con 0x7f36b406f500 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.578+0000 7f36b27fc640 1 -- 172.21.15.77:0/1965664453 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac034440 con 0x7f36b406f500 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.579+0000 7f36b8fa0640 1 -- 172.21.15.77:0/1965664453 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 msgr2=0x7f36b406b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.579+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/1965664453 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b406b460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009d90 tx=0x7f36ac02bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.579+0000 7f36b8fa0640 1 -- 172.21.15.77:0/1965664453 shutdown_connections 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.579+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/1965664453 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b406b460 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.579+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/1965664453 >> v2:172.21.15.158:3300/0 conn(0x7f36b4146ac0 0x7f36b40b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.579+0000 7f36b8fa0640 1 -- 172.21.15.77:0/1965664453 >> 172.21.15.77:0/1965664453 conn(0x7f36b406c0c0 msgr2=0x7f36b406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.579+0000 7f36b8fa0640 1 -- 172.21.15.77:0/1965664453 shutdown_connections 2024-03-06T07:25:50.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.579+0000 7f36b8fa0640 1 -- 172.21.15.77:0/1965664453 wait complete. 2024-03-06T07:25:50.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.580+0000 7f36b8fa0640 1 Processor -- start 2024-03-06T07:25:50.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.580+0000 7f36b8fa0640 1 -- start start 2024-03-06T07:25:50.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b8fa0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b406afb0 0x7f36b40b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b8fa0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b4140360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b8fa0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36b4146ac0 0x7f36b41f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b8fa0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40720a0 con 0x7f36b406afb0 2024-03-06T07:25:50.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b8fa0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b4071f20 con 0x7f36b4146ac0 2024-03-06T07:25:50.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b8fa0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b40b3e30 con 0x7f36b406f500 2024-03-06T07:25:50.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b406afb0 0x7f36b40b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b406afb0 0x7f36b40b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52104/0 (socket says 172.21.15.77:52104) 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b37fe640 1 -- 172.21.15.77:0/2200591369 learned_addr learned my addr 172.21.15.77:0/2200591369 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b3fff640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36b4146ac0 0x7f36b41f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b4140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b37fe640 1 -- 172.21.15.77:0/2200591369 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 msgr2=0x7f36b4140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b4140360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b37fe640 1 -- 172.21.15.77:0/2200591369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36b4146ac0 msgr2=0x7f36b41f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36b4146ac0 0x7f36b41f35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.581+0000 7f36b37fe640 1 -- 172.21.15.77:0/2200591369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b41e75a0 con 0x7f36b406afb0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.582+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b406afb0 0x7f36b40b22d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f36a400e9e0 tx=0x7f36a400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.582+0000 7f369bfff640 1 -- 172.21.15.77:0/2200591369 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4017070 con 0x7f36b406afb0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.582+0000 7f369bfff640 1 -- 172.21.15.77:0/2200591369 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a400c9d0 con 0x7f36b406afb0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.582+0000 7f369bfff640 1 -- 172.21.15.77:0/2200591369 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a400cd80 con 0x7f36b406afb0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.582+0000 7f36b8fa0640 1 -- 172.21.15.77:0/2200591369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b40b17f0 con 0x7f36b406afb0 2024-03-06T07:25:50.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.582+0000 7f36b8fa0640 1 -- 172.21.15.77:0/2200591369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b40b1db0 con 0x7f36b406afb0 2024-03-06T07:25:50.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.583+0000 7f369bfff640 1 -- 172.21.15.77:0/2200591369 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36a401a070 con 0x7f36b406afb0 2024-03-06T07:25:50.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.583+0000 7f369bfff640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.584+0000 7f369bfff640 1 -- 172.21.15.77:0/2200591369 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36a408a300 con 0x7f36b406afb0 2024-03-06T07:25:50.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.584+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.584+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f36b40b2b80 tx=0x7f36a8007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.585+0000 7f36bcdeb740 1 Processor -- start 2024-03-06T07:25:50.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.585+0000 7f36bcdeb740 1 -- start start 2024-03-06T07:25:50.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.585+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4798d2a20 0x55b4798d2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.585+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4798d6a70 0x55b4798d8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 0x55b479902c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b47983c6b0 con 0x55b4798d2a20 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4797e5a60 con 0x55b4798d6a70 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b479838750 con 0x55b4799027b0 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4798d2a20 0x55b4798d2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4798d2a20 0x55b4798d2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52118/0 (socket says 172.21.15.77:52118) 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 -- 172.21.15.77:0/2566956761 learned_addr learned my addr 172.21.15.77:0/2566956761 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4798d6a70 0x55b4798d8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b3fff640 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 0x55b479902c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 -- 172.21.15.77:0/2566956761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 msgr2=0x55b479902c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 0x55b479902c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 -- 172.21.15.77:0/2566956761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4798d6a70 msgr2=0x55b4798d8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4798d6a70 0x55b4798d8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 -- 172.21.15.77:0/2566956761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b479803f30 con 0x55b4798d2a20 2024-03-06T07:25:50.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4798d2a20 0x55b4798d2e00 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009fd0 tx=0x7f36a4058520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d11aa87d1799c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36997fa640 1 -- 172.21.15.77:0/2566956761 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4021050 con 0x55b4798d2a20 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36997fa640 1 -- 172.21.15.77:0/2566956761 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a4061220 con 0x55b4798d2a20 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.586+0000 7f36997fa640 1 -- 172.21.15.77:0/2566956761 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4056ec0 con 0x55b4798d2a20 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2566956761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4798d2a20 msgr2=0x55b4798d2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4798d2a20 0x55b4798d2e00 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009fd0 tx=0x7f36a4058520 comp rx=0 tx=0).stop 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2566956761 shutdown_connections 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 0x55b479902c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4798d6a70 0x55b4798d8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2566956761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4798d2a20 0x55b4798d2e00 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2566956761 >> 172.21.15.77:0/2566956761 conn(0x55b4798d4aa0 msgr2=0x55b4798d6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2566956761 shutdown_connections 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2566956761 wait complete. 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.587+0000 7f36bcdeb740 1 -- start start 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36bcdeb740 1 Processor -- start 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4799027b0 0x55b4799048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b4798fd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b47991c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4798d6f00 con 0x55b47992e630 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4797f8ea0 con 0x55b4799027b0 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4798d71e0 con 0x55b47991d8f0 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36bcdeb740 10 client.0 _ll_register_callbacks cb 0x55b4797f8870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b4798fd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b4798fd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52132/0 (socket says 172.21.15.77:52132) 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36b2ffd640 1 -- 172.21.15.77:0/2029185781 learned_addr learned my addr 172.21.15.77:0/2029185781 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4799027b0 0x55b4799048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.588+0000 7f36b3fff640 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b47991c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36b2ffd640 1 -- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 msgr2=0x55b47991c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b47991c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36b2ffd640 1 -- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4799027b0 msgr2=0x55b4799048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4799027b0 0x55b4799048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36b2ffd640 1 -- 172.21.15.77:0/2029185781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4797f9270 con 0x55b47992e630 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b4798fd5a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f36a8000f60 tx=0x7f36a8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf368572cc60db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36927fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36927fc640 1 -- 172.21.15.77:0/2029185781 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8020580 con 0x55b47992e630 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36927fc640 1 -- 172.21.15.77:0/2029185781 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36a8020720 con 0x55b47992e630 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2029185781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36a4061220 con 0x55b47992e630 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36927fc640 1 -- 172.21.15.77:0/2029185781 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36a802e050 con 0x55b47992e630 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 10 client.7065 fetch_fsmap learned FSMap version 41 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2029185781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x55b4795512d0 con 0x55b47992e630 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36927fc640 1 -- 172.21.15.77:0/2029185781 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a4061220 con 0x55b47992e630 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36927fc640 1 -- 172.21.15.77:0/2029185781 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f36a8031020 con 0x55b47992e630 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 10 client.7065 fetch_fsmap finished waiting for FSMap version 41 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 10 client.7065 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 20 client.7065 populate_metadata read hostname 'smithi077' 2024-03-06T07:25:50.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] conn(0x55b47992dd60 0x55b479930f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 4 client.7065 mds_command: new command op to 6930 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.589+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2029185781 --> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b479849bf0 con 0x55b47992dd60 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.590+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] conn(0x55b47992dd60 0x55b479930f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.590+0000 7f36927fc640 10 client.7065 ms_handle_connect on v2:172.21.15.158:6832/3775835533 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.590+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] conn(0x55b47992dd60 0x55b479930f60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d4760b17d1f34ae server_cookie=d135ed52920b386f in_seq=0 out_seq=0 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.590+0000 7f36927fc640 1 -- 172.21.15.77:0/2029185781 <== mds.0 v2:172.21.15.158:6832/3775835533 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b479849bf0 con 0x55b47992dd60 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.590+0000 7f36927fc640 10 client.7065 handle_command_reply: tid=0 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 client.7065 shutdown 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2029185781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 msgr2=0x55b4798fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b4798fd5a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f36a8000f60 tx=0x7f36a8000bc0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2029185781 shutdown_connections 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] conn(0x55b47992dd60 0x55b479930f60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b4798fd5a0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36927fc640 0 client.7065 ms_handle_reset on v2:172.21.15.158:6832/3775835533 2024-03-06T07:25:50.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b47991c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2029185781 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b4799027b0 0x55b4799048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2029185781 >> 172.21.15.77:0/2029185781 conn(0x55b4798d2a20 msgr2=0x55b479900770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2029185781 shutdown_connections 2024-03-06T07:25:50.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2029185781 wait complete. 2024-03-06T07:25:50.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.591+0000 7f36bcdeb740 20 client.7065 trim_cache size 0 max 16384 2024-03-06T07:25:50.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.607+0000 7f36bcdeb740 1 Processor -- start 2024-03-06T07:25:50.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.607+0000 7f36bcdeb740 1 -- start start 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4799027b0 0x55b4798d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b4798d7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47992e630 0x55b4798d7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b41ff310 con 0x55b4799027b0 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b41e6ec0 con 0x55b47992e630 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b41feda0 con 0x55b47991d8f0 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4799027b0 0x55b4798d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4799027b0 0x55b4798d1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52140/0 (socket says 172.21.15.77:52140) 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b37fe640 1 -- 172.21.15.77:0/622838801 learned_addr learned my addr 172.21.15.77:0/622838801 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b4798d7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b3fff640 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47992e630 0x55b4798d7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b37fe640 1 -- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 msgr2=0x55b4798d7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b37fe640 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b4798d7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b37fe640 1 -- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47992e630 msgr2=0x55b4798d7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.608+0000 7f36b37fe640 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47992e630 0x55b4798d7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36b37fe640 1 -- 172.21.15.77:0/622838801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b47961b6b0 con 0x55b4799027b0 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b4798d7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36b37fe640 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4799027b0 0x55b4798d1a30 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f36a40588c0 tx=0x7f36a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f0cf169194b829 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f3698ff9640 1 -- 172.21.15.77:0/622838801 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4021040 con 0x55b4799027b0 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f3698ff9640 1 -- 172.21.15.77:0/622838801 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a4056ec0 con 0x55b4799027b0 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f3698ff9640 1 -- 172.21.15.77:0/622838801 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a405db80 con 0x55b4799027b0 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 -- 172.21.15.77:0/622838801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4799027b0 msgr2=0x55b4798d1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4799027b0 0x55b4798d1a30 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f36a40588c0 tx=0x7f36a4014000 comp rx=0 tx=0).stop 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 -- 172.21.15.77:0/622838801 shutdown_connections 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47992e630 0x55b4798d7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b47991d8f0 0x55b4798d7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/622838801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4799027b0 0x55b4798d1a30 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 -- 172.21.15.77:0/622838801 >> 172.21.15.77:0/622838801 conn(0x55b4798d2a20 msgr2=0x55b4798cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 -- 172.21.15.77:0/622838801 shutdown_connections 2024-03-06T07:25:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.609+0000 7f36bcdeb740 1 -- 172.21.15.77:0/622838801 wait complete. 2024-03-06T07:25:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.610+0000 7f36bcdeb740 1 -- start start 2024-03-06T07:25:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.610+0000 7f36bcdeb740 1 Processor -- start 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 0x55b47991a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47991d8f0 0x55b47992a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36bcdeb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b47992a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b47991d2c0 con 0x55b47992e630 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b479903e90 con 0x55b47991d8f0 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36bcdeb740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4799191a0 con 0x55b4799027b0 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36bcdeb740 10 client.0 _ll_register_callbacks cb 0x7f36a80075d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b47992a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b47992a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52150/0 (socket says 172.21.15.77:52150) 2024-03-06T07:25:50.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 -- 172.21.15.77:0/2849063714 learned_addr learned my addr 172.21.15.77:0/2849063714 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b2ffd640 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47991d8f0 0x55b47992a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 0x55b47991a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 -- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 msgr2=0x55b47991a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 0x55b47991a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 -- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47991d8f0 msgr2=0x55b47992a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47991d8f0 0x55b47992a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 -- 172.21.15.77:0/2849063714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b479990a40 con 0x55b47992e630 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36b3fff640 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b47992a810 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009d90 tx=0x7f36ac02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b42c08d29d47f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36927fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36927fc640 1 -- 172.21.15.77:0/2849063714 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac050060 con 0x55b47992e630 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.611+0000 7f36927fc640 1 -- 172.21.15.77:0/2849063714 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36ac034cb0 con 0x55b47992e630 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2849063714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36a4056ec0 con 0x55b47992e630 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36927fc640 1 -- 172.21.15.77:0/2849063714 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36ac056050 con 0x55b47992e630 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 10 client.7071 fetch_fsmap learned FSMap version 41 2024-03-06T07:25:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2849063714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x55b47992ae10 con 0x55b47992e630 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36927fc640 1 -- 172.21.15.77:0/2849063714 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a4056ec0 con 0x55b47992e630 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36927fc640 1 -- 172.21.15.77:0/2849063714 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f36ac059020 con 0x55b47992e630 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 10 client.7071 fetch_fsmap finished waiting for FSMap version 41 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 10 client.7071 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 20 client.7071 populate_metadata read hostname 'smithi077' 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] conn(0x55b4799287e0 0x55b4798d7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 4 client.7071 mds_command: new command op to 6930 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2849063714 --> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b479843f40 con 0x55b4799287e0 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] conn(0x55b4799287e0 0x55b4798d7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36927fc640 10 client.7071 ms_handle_connect on v2:172.21.15.158:6832/3775835533 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.612+0000 7f36b37fe640 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] conn(0x55b4799287e0 0x55b4798d7110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57d33a0c79c48dcc server_cookie=a9ce20c5905d44f5 in_seq=0 out_seq=0 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36927fc640 1 -- 172.21.15.77:0/2849063714 <== mds.0 v2:172.21.15.158:6832/3775835533 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55b479843f40 con 0x55b4799287e0 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36927fc640 10 client.7071 handle_command_reply: tid=0 2024-03-06T07:25:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 client.7071 shutdown 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2849063714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 msgr2=0x55b47992a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b47992a810 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009d90 tx=0x7f36ac02e070 comp rx=0 tx=0).stop 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2849063714 shutdown_connections 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:6832/3775835533,v1:172.21.15.158:6834/3775835533] conn(0x55b4799287e0 0x55b4798d7110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b47992e630 0x55b47992a810 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36927fc640 0 client.7071 ms_handle_reset on v2:172.21.15.158:6832/3775835533 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b47991d8f0 0x55b47992a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 --2- 172.21.15.77:0/2849063714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b4799027b0 0x55b47991a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2849063714 >> 172.21.15.77:0/2849063714 conn(0x55b4798d2a20 msgr2=0x55b479902050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2849063714 shutdown_connections 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 1 -- 172.21.15.77:0/2849063714 wait complete. 2024-03-06T07:25:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36bcdeb740 20 client.7071 trim_cache size 0 max 16384 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 6930, 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 5.2503165469999997, 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 41, 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 80, 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 80, 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 10.282713478 2024-03-06T07:25:50.622 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:25:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 -- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 msgr2=0x7f36a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f36b40b2b80 tx=0x7f36a8007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 -- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b406afb0 msgr2=0x7f36b40b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b406afb0 0x7f36b40b22d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f36a400e9e0 tx=0x7f36a400eea0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 -- 172.21.15.77:0/2200591369 shutdown_connections 2024-03-06T07:25:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36b4146ac0 0x7f36b41f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36b406f500 0x7f36b4140360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 --2- 172.21.15.77:0/2200591369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b406afb0 0x7f36b40b22d0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.613+0000 7f36b8fa0640 1 -- 172.21.15.77:0/2200591369 >> 172.21.15.77:0/2200591369 conn(0x7f36b406c0c0 msgr2=0x7f36b4143430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.614+0000 7f36b8fa0640 1 -- 172.21.15.77:0/2200591369 shutdown_connections 2024-03-06T07:25:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.614+0000 7f36b8fa0640 1 -- 172.21.15.77:0/2200591369 wait complete. 2024-03-06T07:25:50.626 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:50.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.724+0000 7f0d55cd4640 1 Processor -- start 2024-03-06T07:25:50.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.724+0000 7f0d55cd4640 1 -- start start 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d55cd4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0d500b3060 0x7f0d5015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d55cd4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 0x7f0d5015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d55cd4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5015f2d0 con 0x7f0d5013dc10 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d55cd4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d500cf3f0 con 0x7f0d500b3060 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d55cd4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5015b210 con 0x7f0d5015bf30 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d4effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0d500b3060 0x7f0d5015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d4effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0d500b3060 0x7f0d5015e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52164/0 (socket says 172.21.15.77:52164) 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d4effd640 1 -- 172.21.15.77:0/3254723269 learned_addr learned my addr 172.21.15.77:0/3254723269 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:50.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d4ffff640 1 --2- 172.21.15.77:0/3254723269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 0x7f0d5015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d4effd640 1 -- 172.21.15.77:0/3254723269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 msgr2=0x7f0d5015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d4effd640 1 --2- 172.21.15.77:0/3254723269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 0x7f0d5015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d4effd640 1 -- 172.21.15.77:0/3254723269 >> v1:172.21.15.158:6789/0 conn(0x7f0d5013dc10 legacy=0x7f0d500b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.725+0000 7f0d4effd640 1 -- 172.21.15.77:0/3254723269 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5015f580 con 0x7f0d500b3060 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.726+0000 7f0d4effd640 1 --2- 172.21.15.77:0/3254723269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d500b3060 0x7f0d5015e850 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009870 tx=0x7f0d4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49b66643a54d398 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.726+0000 7f0d4e7fc640 1 -- 172.21.15.77:0/3254723269 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44035020 con 0x7f0d500b3060 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.726+0000 7f0d4e7fc640 1 -- 172.21.15.77:0/3254723269 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44031640 con 0x7f0d500b3060 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.726+0000 7f0d4e7fc640 1 -- 172.21.15.77:0/3254723269 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4402d040 con 0x7f0d500b3060 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.727+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3254723269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d500b3060 msgr2=0x7f0d5015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.727+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3254723269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d500b3060 0x7f0d5015e850 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009870 tx=0x7f0d4402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.727+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3254723269 shutdown_connections 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.727+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3254723269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 0x7f0d5015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.727+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3254723269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d500b3060 0x7f0d5015e850 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.727+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3254723269 >> 172.21.15.77:0/3254723269 conn(0x7f0d500cf8a0 msgr2=0x7f0d500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.727+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3254723269 shutdown_connections 2024-03-06T07:25:50.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.727+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3254723269 wait complete. 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.728+0000 7f0d55cd4640 1 Processor -- start 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.728+0000 7f0d55cd4640 1 -- start start 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d55cd4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d500b3060 0x7f0d50140640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d55cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d5013dc10 0x7f0d50140c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d55cd4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 0x7f0d501438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d55cd4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5013dff0 con 0x7f0d5013dc10 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d55cd4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5015b730 con 0x7f0d500b3060 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d55cd4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d500708d0 con 0x7f0d5015bf30 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d5013dc10 0x7f0d50140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d5013dc10 0x7f0d50140c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52180/0 (socket says 172.21.15.77:52180) 2024-03-06T07:25:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 -- 172.21.15.77:0/3757972458 learned_addr learned my addr 172.21.15.77:0/3757972458 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4f7fe640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d500b3060 0x7f0d50140640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4ffff640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 0x7f0d501438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 -- 172.21.15.77:0/3757972458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 msgr2=0x7f0d501438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 0x7f0d501438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 -- 172.21.15.77:0/3757972458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d500b3060 msgr2=0x7f0d50140640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d500b3060 0x7f0d50140640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 -- 172.21.15.77:0/3757972458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50142480 con 0x7f0d5013dc10 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d4effd640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d5013dc10 0x7f0d50140c30 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0d4402b930 tx=0x7f0d440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.729+0000 7f0d37fff640 1 -- 172.21.15.77:0/3757972458 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44042070 con 0x7f0d5013dc10 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.730+0000 7f0d37fff640 1 -- 172.21.15.77:0/3757972458 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44040070 con 0x7f0d5013dc10 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.730+0000 7f0d37fff640 1 -- 172.21.15.77:0/3757972458 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44035040 con 0x7f0d5013dc10 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.730+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d50141bc0 con 0x7f0d5013dc10 2024-03-06T07:25:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.730+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d501f1790 con 0x7f0d5013dc10 2024-03-06T07:25:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.731+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5004f330 con 0x7f0d5013dc10 2024-03-06T07:25:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.731+0000 7f0d37fff640 1 -- 172.21.15.77:0/3757972458 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d44038020 con 0x7f0d5013dc10 2024-03-06T07:25:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.731+0000 7f0d37fff640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.731+0000 7f0d37fff640 1 -- 172.21.15.77:0/3757972458 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d440adbc0 con 0x7f0d5013dc10 2024-03-06T07:25:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.731+0000 7f0d4f7fe640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.732+0000 7f0d4f7fe640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0d40001780 tx=0x7f0d40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.735+0000 7f0d37fff640 1 -- 172.21.15.77:0/3757972458 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d50142480 con 0x7f0d5013dc10 2024-03-06T07:25:50.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.944+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0d501f19d0 con 0x7f0d5013dc10 2024-03-06T07:25:50.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.944+0000 7f0d37fff640 1 -- 172.21.15.77:0/3757972458 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0d4407c480 con 0x7f0d5013dc10 2024-03-06T07:25:50.946 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:50.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d3c06b4d0 msgr2=0x7f0d3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0d40001780 tx=0x7f0d40009040 comp rx=0 tx=0).stop 2024-03-06T07:25:50.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d5013dc10 msgr2=0x7f0d50140c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d5013dc10 0x7f0d50140c30 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0d4402b930 tx=0x7f0d440046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 shutdown_connections 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d5013dc10 0x7f0d50140c30 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d5015bf30 0x7f0d501438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 --2- 172.21.15.77:0/3757972458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d500b3060 0x7f0d50140640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 >> 172.21.15.77:0/3757972458 conn(0x7f0d500cf8a0 msgr2=0x7f0d500b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 shutdown_connections 2024-03-06T07:25:50.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:50.946+0000 7f0d55cd4640 1 -- 172.21.15.77:0/3757972458 wait complete. 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:50.959 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:50.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:50.960 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:25:50.975 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:50.975 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:50.975 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:50.975 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:50.975 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:50.975 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:50.975 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42446sec preferred_lft 42446sec 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:50.976 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:25:50.977 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:50.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:50.978 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:51.041 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:25:51.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.042 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:25:51.097 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:25:51.098 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:51.098 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:25:51.098 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:25:51.098 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:51.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.098 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:25:51.098 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:25:51.098 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:25:51.098 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:25:51.098 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:25:51.098 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:25:51.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.192 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:25:51.206 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:51.206 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:51.206 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:51.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.206 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:25:51.273 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:51.273 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:51.274 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:51.328 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:51.329 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:51.329 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:51.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:25:51.451 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:51.506 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:25:51.506 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:25:51.506 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:25:51.506 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:25:51.506 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:25:51.506 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:51.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.509 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:25:51.561 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:51.561 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:51.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:25:51.630 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:51.685 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:25:51.685 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:25:51.685 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:25:51.685 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:25:51.685 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:25:51.685 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:25:51.685 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:51.686 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:51.687 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:51.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.688 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:25:51.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:25:51.687+0000 7f88ad80f480 -1 init, newargv = 0x564fb1661640 newargc=15 2024-03-06T07:25:51.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[65522]: starting ceph client 2024-03-06T07:25:51.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[65522]: starting fuse 2024-03-06T07:25:51.703 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:25:51.703 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:25:51.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:51.703 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:25:51.703 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:25:51.703 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:25:51.703 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:25:51.703 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:51.703 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:25:51.703 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:25:51.704 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:51.705 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:25:51.705 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:51.705 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:25:51.823 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.65522.asok 2024-03-06T07:25:51.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:51.834 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.65522.asok 2024-03-06T07:25:51.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.834 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.65522.asok status 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "65522", 2024-03-06T07:25:51.969 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 7083, 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 7083 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:51.970 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1196714732 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1196714732 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.7083 192.168.0.1:0/1196714732", 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/1196714732", 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 41, 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 80, 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:25:51.971 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:25:51.972 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:25:51.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.977 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:25:51.977 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:51.992 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:25:51.992 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:51.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:52.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:52.061 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:25:52.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:52.121 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:25:52.195 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:52.195 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:52.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:52.251 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:25:52.465 INFO:teuthology.orchestra.run.smithi077.stderr:35+0 records in 2024-03-06T07:25:52.465 INFO:teuthology.orchestra.run.smithi077.stderr:35+0 records out 2024-03-06T07:25:52.465 INFO:teuthology.orchestra.run.smithi077.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.156569 s, 234 MB/s 2024-03-06T07:25:52.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:52.466 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:25:52.484 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:52.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.623+0000 7f193d08a640 1 Processor -- start 2024-03-06T07:25:52.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.623+0000 7f193d08a640 1 -- start start 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193d08a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f19380d22b0 0x7f19380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193d08a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380b41f0 0x7f19380b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193d08a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19380b3430 con 0x7f193815dfe0 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193d08a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19380d2c50 con 0x7f19380d22b0 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193d08a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19380b31e0 con 0x7f19380b41f0 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f19380d22b0 0x7f19380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f19380d22b0 0x7f19380d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52186/0 (socket says 172.21.15.77:52186) 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193659c640 1 -- 172.21.15.77:0/2975990568 learned_addr learned my addr 172.21.15.77:0/2975990568 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.624+0000 7f193759e640 1 --2- 172.21.15.77:0/2975990568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380b41f0 0x7f19380b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.625+0000 7f193659c640 1 -- 172.21.15.77:0/2975990568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380b41f0 msgr2=0x7f19380b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.625+0000 7f193659c640 1 --2- 172.21.15.77:0/2975990568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380b41f0 0x7f19380b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.625+0000 7f193659c640 1 -- 172.21.15.77:0/2975990568 >> v1:172.21.15.158:6789/0 conn(0x7f193815dfe0 legacy=0x7f19380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.625+0000 7f193659c640 1 -- 172.21.15.77:0/2975990568 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19380b3710 con 0x7f19380d22b0 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.625+0000 7f193659c640 1 --2- 172.21.15.77:0/2975990568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19380d22b0 0x7f19380d2710 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f192c009870 tx=0x7f192c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53756f0137ea809 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.625+0000 7f1935d9b640 1 -- 172.21.15.77:0/2975990568 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c034070 con 0x7f19380d22b0 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.625+0000 7f1935d9b640 1 -- 172.21.15.77:0/2975990568 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c030840 con 0x7f19380d22b0 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.625+0000 7f1935d9b640 1 -- 172.21.15.77:0/2975990568 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c030c10 con 0x7f19380d22b0 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.626+0000 7f193d08a640 1 -- 172.21.15.77:0/2975990568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19380d22b0 msgr2=0x7f19380d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.626+0000 7f193d08a640 1 --2- 172.21.15.77:0/2975990568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19380d22b0 0x7f19380d2710 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f192c009870 tx=0x7f192c02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.626+0000 7f193d08a640 1 -- 172.21.15.77:0/2975990568 shutdown_connections 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.626+0000 7f193d08a640 1 --2- 172.21.15.77:0/2975990568 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380b41f0 0x7f19380b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.626+0000 7f193d08a640 1 --2- 172.21.15.77:0/2975990568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19380d22b0 0x7f19380d2710 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.626+0000 7f193d08a640 1 -- 172.21.15.77:0/2975990568 >> 172.21.15.77:0/2975990568 conn(0x7f19380d3cd0 msgr2=0x7f19380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.626+0000 7f193d08a640 1 -- 172.21.15.77:0/2975990568 shutdown_connections 2024-03-06T07:25:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.626+0000 7f193d08a640 1 -- 172.21.15.77:0/2975990568 wait complete. 2024-03-06T07:25:52.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.627+0000 7f193d08a640 1 Processor -- start 2024-03-06T07:25:52.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.627+0000 7f193d08a640 1 -- start start 2024-03-06T07:25:52.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193d08a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19380b41f0 0x7f19380b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193d08a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380d22b0 0x7f19381f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193d08a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f193815dfe0 0x7f19381f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193d08a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19380cfa00 con 0x7f193815dfe0 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193d08a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19380b3d60 con 0x7f19380b41f0 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193d08a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19380d1c70 con 0x7f19380d22b0 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f193815dfe0 0x7f19381f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f193815dfe0 0x7f19381f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52190/0 (socket says 172.21.15.77:52190) 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193759e640 1 -- 172.21.15.77:0/1489886972 learned_addr learned my addr 172.21.15.77:0/1489886972 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f1936d9d640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19380b41f0 0x7f19380b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.628+0000 7f193659c640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380d22b0 0x7f19381f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f193759e640 1 -- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380d22b0 msgr2=0x7f19381f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f193759e640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380d22b0 0x7f19381f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f193759e640 1 -- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19380b41f0 msgr2=0x7f19380b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f193759e640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19380b41f0 0x7f19380b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f193759e640 1 -- 172.21.15.77:0/1489886972 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19380b9490 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f1936d9d640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19380b41f0 0x7f19380b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f193759e640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f193815dfe0 0x7f19381f3e40 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f193000d830 tx=0x7f193000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f191f7fe640 1 -- 172.21.15.77:0/1489886972 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193000b6c0 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f191f7fe640 1 -- 172.21.15.77:0/1489886972 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1930019070 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19380b9770 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f191f7fe640 1 -- 172.21.15.77:0/1489886972 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193000bac0 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.629+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19380b1130 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.630+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193804f330 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.630+0000 7f191f7fe640 1 -- 172.21.15.77:0/1489886972 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f193001e460 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.631+0000 7f191f7fe640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f192406b4d0 0x7f192406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.631+0000 7f191f7fe640 1 -- 172.21.15.77:0/1489886972 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f193008a720 con 0x7f193815dfe0 2024-03-06T07:25:52.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.631+0000 7f1936d9d640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f192406b4d0 0x7f192406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.631+0000 7f1936d9d640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f192406b4d0 0x7f192406d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f193815e9e0 tx=0x7f1928009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.635+0000 7f191f7fe640 1 -- 172.21.15.77:0/1489886972 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1930058f60 con 0x7f193815dfe0 2024-03-06T07:25:52.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.842+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19380b2f40 con 0x7f193815dfe0 2024-03-06T07:25:52.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.843+0000 7f191f7fe640 1 -- 172.21.15.77:0/1489886972 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f193005d000 con 0x7f193815dfe0 2024-03-06T07:25:52.845 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:52.845 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3408486989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3408486989},{"type":"v1","addr":"172.21.15.158:6835","nonce":3408486989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/324866984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":324866984},{"type":"v1","addr":"172.21.15.77:6836","nonce":324866984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2426038221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2426038221},{"type":"v1","addr":"172.21.15.77:6837","nonce":2426038221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:44.712557+0000","modified":"2024-03-06T07:25:47.370110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6930},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6930":{"gid":6930,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/3775835533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3775835533},{"type":"v1","addr":"172.21.15.158:6834","nonce":3775835533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6930,"qdb_cluster":[6930]},"id":6}]} 2024-03-06T07:25:52.845 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f192406b4d0 msgr2=0x7f192406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f192406b4d0 0x7f192406d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f193815e9e0 tx=0x7f1928009040 comp rx=0 tx=0).stop 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f193815dfe0 msgr2=0x7f19381f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f193815dfe0 0x7f19381f3e40 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f193000d830 tx=0x7f193000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 shutdown_connections 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f192406b4d0 0x7f192406d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f193815dfe0 0x7f19381f3e40 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f19380d22b0 0x7f19381f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 --2- 172.21.15.77:0/1489886972 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19380b41f0 0x7f19380b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 >> 172.21.15.77:0/1489886972 conn(0x7f19380d3cd0 msgr2=0x7f193806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 shutdown_connections 2024-03-06T07:25:52.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:52.845+0000 7f193d08a640 1 -- 172.21.15.77:0/1489886972 wait complete. 2024-03-06T07:25:52.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-06T07:25:52.981 INFO:teuthology.orchestra.run.smithi158.stdout:{"message":"","return_code":0} 2024-03-06T07:25:52.982 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:25:52.982 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.082+0000 7f6801ab5640 1 Processor -- start 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.082+0000 7f6801ab5640 1 -- start start 2024-03-06T07:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f6801ab5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f67fc1444c0 0x7f67fc06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f6801ab5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc070fe0 0x7f67fc0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f6801ab5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc06d370 con 0x7f67fc15b2d0 2024-03-06T07:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f6801ab5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67fc1448a0 con 0x7f67fc1444c0 2024-03-06T07:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f6801ab5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc06d650 con 0x7f67fc070fe0 2024-03-06T07:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67faffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f67fc15b2d0 0x7f67fc0708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58108/0 (socket says 172.21.15.77:58108) 2024-03-06T07:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67faffd640 1 -- 172.21.15.77:0/2002869989 learned_addr learned my addr 172.21.15.77:0/2002869989 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67fb7fe640 1 --2- 172.21.15.77:0/2002869989 >> v2:172.21.15.158:3300/0 conn(0x7f67fc1444c0 0x7f67fc06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67fbfff640 1 --2- 172.21.15.77:0/2002869989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc070fe0 0x7f67fc0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67fa7fc640 1 -- 172.21.15.77:0/2002869989 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398831747 0 0) 0x7f67fc06d370 con 0x7f67fc15b2d0 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67fa7fc640 1 -- 172.21.15.77:0/2002869989 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e4003520 con 0x7f67fc15b2d0 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67fb7fe640 1 -- 172.21.15.77:0/2002869989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc070fe0 msgr2=0x7f67fc0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67fb7fe640 1 --2- 172.21.15.77:0/2002869989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc070fe0 0x7f67fc0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67fb7fe640 1 -- 172.21.15.77:0/2002869989 >> v1:172.21.15.77:6789/0 conn(0x7f67fc15b2d0 legacy=0x7f67fc0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.083+0000 7f67fb7fe640 1 -- 172.21.15.77:0/2002869989 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc06d930 con 0x7f67fc1444c0 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.084+0000 7f67fb7fe640 1 --2- 172.21.15.77:0/2002869989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 0x7f67fc06ca00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0097a0 tx=0x7f67ec02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac51587f6fd1e03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.084+0000 7f67fa7fc640 1 -- 172.21.15.77:0/2002869989 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec035020 con 0x7f67fc1444c0 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.084+0000 7f67fa7fc640 1 -- 172.21.15.77:0/2002869989 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec031830 con 0x7f67fc1444c0 2024-03-06T07:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.084+0000 7f67fa7fc640 1 -- 172.21.15.77:0/2002869989 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec031c00 con 0x7f67fc1444c0 2024-03-06T07:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.085+0000 7f6801ab5640 1 -- 172.21.15.77:0/2002869989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 msgr2=0x7f67fc06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.085+0000 7f6801ab5640 1 --2- 172.21.15.77:0/2002869989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 0x7f67fc06ca00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0097a0 tx=0x7f67ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.085+0000 7f6801ab5640 1 -- 172.21.15.77:0/2002869989 shutdown_connections 2024-03-06T07:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.085+0000 7f6801ab5640 1 --2- 172.21.15.77:0/2002869989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc070fe0 0x7f67fc0cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.085+0000 7f6801ab5640 1 --2- 172.21.15.77:0/2002869989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 0x7f67fc06ca00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.085+0000 7f6801ab5640 1 -- 172.21.15.77:0/2002869989 >> 172.21.15.77:0/2002869989 conn(0x7f67fc0b1c80 msgr2=0x7f67fc0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.085+0000 7f6801ab5640 1 -- 172.21.15.77:0/2002869989 shutdown_connections 2024-03-06T07:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.085+0000 7f6801ab5640 1 -- 172.21.15.77:0/2002869989 wait complete. 2024-03-06T07:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.086+0000 7f6801ab5640 1 Processor -- start 2024-03-06T07:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.086+0000 7f6801ab5640 1 -- start start 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f6801ab5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc070fe0 0x7f67fc13bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f6801ab5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 0x7f67fc0cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f6801ab5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc15b2d0 0x7f67fc13b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f6801ab5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc06cf40 con 0x7f67fc070fe0 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f6801ab5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc06def0 con 0x7f67fc1444c0 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f6801ab5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc06d0c0 con 0x7f67fc15b2d0 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f67fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc070fe0 0x7f67fc13bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f67fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc070fe0 0x7f67fc13bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52192/0 (socket says 172.21.15.77:52192) 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f67fb7fe640 1 -- 172.21.15.77:0/1711661364 learned_addr learned my addr 172.21.15.77:0/1711661364 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f67faffd640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 0x7f67fc0cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f67fbfff640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc15b2d0 0x7f67fc13b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.087+0000 7f67fb7fe640 1 -- 172.21.15.77:0/1711661364 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc15b2d0 msgr2=0x7f67fc13b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f67fb7fe640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc15b2d0 0x7f67fc13b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f67fb7fe640 1 -- 172.21.15.77:0/1711661364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 msgr2=0x7f67fc0cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f67fb7fe640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 0x7f67fc0cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f67fb7fe640 1 -- 172.21.15.77:0/1711661364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc13daa0 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f67fb7fe640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc070fe0 0x7f67fc13bf80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f67ec02b910 tx=0x7f67ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f67e3fff640 1 -- 172.21.15.77:0/1711661364 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec040070 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f67e3fff640 1 -- 172.21.15.77:0/1711661364 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec03a930 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f67e3fff640 1 -- 172.21.15.77:0/1711661364 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec035040 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67fc13dda0 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.088+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67fc13e5e0 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.089+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67fc04f330 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.089+0000 7f67e3fff640 1 -- 172.21.15.77:0/1711661364 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67ec04b060 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.090+0000 7f67e3fff640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f67e806b4d0 0x7f67e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.090+0000 7f67e3fff640 1 -- 172.21.15.77:0/1711661364 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67ec0ab800 con 0x7f67fc070fe0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.090+0000 7f67faffd640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f67e806b4d0 0x7f67e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.090+0000 7f67faffd640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f67e806b4d0 0x7f67e806d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0cc150 tx=0x7f67f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.094+0000 7f67e3fff640 1 -- 172.21.15.77:0/1711661364 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67fc13dda0 con 0x7f67fc070fe0 2024-03-06T07:25:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.333+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f67fc0b45b0 con 0x7f67fc070fe0 2024-03-06T07:25:53.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.334+0000 7f67e3fff640 1 -- 172.21.15.77:0/1711661364 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1651 (secure 0 0 0) 0x7f67ec07a030 con 0x7f67fc070fe0 2024-03-06T07:25:53.335 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "stats": { 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "total_bytes": 773094113280, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "total_avail_bytes": 772812496896, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_bytes": 281616384, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_raw_bytes": 281616384, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_raw_ratio": 0.00036427180748432875, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "num_osds": 8, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "num_per_pool_osds": 8, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "stats_by_class": { 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "ssd": { 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "total_bytes": 773094113280, 2024-03-06T07:25:53.336 INFO:teuthology.orchestra.run.smithi077.stdout: "total_avail_bytes": 772812496896, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_bytes": 281616384, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_raw_bytes": 281616384, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "total_used_raw_ratio": 0.00036427180748432875 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "pools": [ 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "name": ".mgr", 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 1, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "stats": { 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "stored": 590368, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "objects": 2, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "kb_used": 1160, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_used": 1187840, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "percent_used": 1.61800551268243e-06, 2024-03-06T07:25:53.337 INFO:teuthology.orchestra.run.smithi077.stdout: "max_avail": 367068610560 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephfs_metadata", 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 12, 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "stats": { 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "stored": 2410, 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "objects": 22, 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "kb_used": 64, 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_used": 65536, 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "percent_used": 8.9269406089442782e-08, 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: "max_avail": 367068610560 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:25:53.338 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephfs_data", 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 13, 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "stats": { 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "stored": 0, 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "objects": 0, 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "kb_used": 0, 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "bytes_used": 0, 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "percent_used": 0, 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: "max_avail": 367068610560 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:25:53.339 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:25:53.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f67e806b4d0 msgr2=0x7f67e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f67e806b4d0 0x7f67e806d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f67fc0cc150 tx=0x7f67f0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc070fe0 msgr2=0x7f67fc13bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc070fe0 0x7f67fc13bf80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f67ec02b910 tx=0x7f67ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 shutdown_connections 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67fc15b2d0 0x7f67fc13b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f67fc1444c0 0x7f67fc0cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f67e806b4d0 0x7f67e806d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 --2- 172.21.15.77:0/1711661364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67fc070fe0 0x7f67fc13bf80 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.335+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 >> 172.21.15.77:0/1711661364 conn(0x7f67fc0b1c80 msgr2=0x7f67fc0ce0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.336+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 shutdown_connections 2024-03-06T07:25:53.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.336+0000 7f6801ab5640 1 -- 172.21.15.77:0/1711661364 wait complete. 2024-03-06T07:25:53.348 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:25:53.348 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:53.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:53.395 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:53.440 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:53.440 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:53.440 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:53.455 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:53.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.594+0000 7f633591e640 1 Processor -- start 2024-03-06T07:25:53.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.594+0000 7f633591e640 1 -- start start 2024-03-06T07:25:53.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f633591e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f633015ddf0 0x7f63300cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f633591e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015e1d0 0x7f63300cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f633591e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63300b7b60 con 0x7f63300d06d0 2024-03-06T07:25:53.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f633591e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63300d27c0 con 0x7f633015ddf0 2024-03-06T07:25:53.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f633591e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63300d18d0 con 0x7f633015e1d0 2024-03-06T07:25:53.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f633015ddf0 0x7f63300cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f63300d06d0 0x7f6330161ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58110/0 (socket says 172.21.15.77:58110) 2024-03-06T07:25:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632f7fe640 1 -- 172.21.15.77:0/3527602544 learned_addr learned my addr 172.21.15.77:0/3527602544 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632e7fc640 1 --2- 172.21.15.77:0/3527602544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015e1d0 0x7f63300cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632effd640 1 -- 172.21.15.77:0/3527602544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015e1d0 msgr2=0x7f63300cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632effd640 1 --2- 172.21.15.77:0/3527602544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015e1d0 0x7f63300cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632effd640 1 -- 172.21.15.77:0/3527602544 >> v1:172.21.15.77:6789/0 conn(0x7f63300d06d0 legacy=0x7f6330161ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632dffb640 1 -- 172.21.15.77:0/3527602544 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365207883 0 0) 0x7f63300b7b60 con 0x7f63300d06d0 2024-03-06T07:25:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.595+0000 7f632effd640 1 -- 172.21.15.77:0/3527602544 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63300d1bb0 con 0x7f633015ddf0 2024-03-06T07:25:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.596+0000 7f632effd640 1 --2- 172.21.15.77:0/3527602544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f633015ddf0 0x7f63300cf440 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f63300d2940 tx=0x7f632002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7974804d395683bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.596+0000 7f632dffb640 1 -- 172.21.15.77:0/3527602544 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320035020 con 0x7f633015ddf0 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.596+0000 7f632dffb640 1 -- 172.21.15.77:0/3527602544 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6320031800 con 0x7f633015ddf0 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.596+0000 7f632dffb640 1 -- 172.21.15.77:0/3527602544 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320031bd0 con 0x7f633015ddf0 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.596+0000 7f633591e640 1 -- 172.21.15.77:0/3527602544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f633015ddf0 msgr2=0x7f63300cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.596+0000 7f633591e640 1 --2- 172.21.15.77:0/3527602544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f633015ddf0 0x7f63300cf440 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f63300d2940 tx=0x7f632002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.597+0000 7f633591e640 1 -- 172.21.15.77:0/3527602544 shutdown_connections 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.597+0000 7f633591e640 1 --2- 172.21.15.77:0/3527602544 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015e1d0 0x7f63300cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.597+0000 7f633591e640 1 --2- 172.21.15.77:0/3527602544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f633015ddf0 0x7f63300cf440 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.597+0000 7f633591e640 1 -- 172.21.15.77:0/3527602544 >> 172.21.15.77:0/3527602544 conn(0x7f633009cff0 msgr2=0x7f63300d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.597+0000 7f633591e640 1 -- 172.21.15.77:0/3527602544 shutdown_connections 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.597+0000 7f633591e640 1 -- 172.21.15.77:0/3527602544 wait complete. 2024-03-06T07:25:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.598+0000 7f633591e640 1 Processor -- start 2024-03-06T07:25:53.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.598+0000 7f633591e640 1 -- start start 2024-03-06T07:25:53.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f633591e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63300d06d0 0x7f63300c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f633591e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015ddf0 0x7f63300ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f633591e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f633015e1d0 0x7f63300cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f633591e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63300d2450 con 0x7f63300d06d0 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f633591e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63300d2170 con 0x7f633015e1d0 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f633591e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63300d2c50 con 0x7f633015ddf0 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63300d06d0 0x7f63300c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63300d06d0 0x7f63300c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52214/0 (socket says 172.21.15.77:52214) 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 -- 172.21.15.77:0/695630587 learned_addr learned my addr 172.21.15.77:0/695630587 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632f7fe640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f633015e1d0 0x7f63300cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632e7fc640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015ddf0 0x7f63300ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 -- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015ddf0 msgr2=0x7f63300ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015ddf0 0x7f63300ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 -- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f633015e1d0 msgr2=0x7f63300cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f633015e1d0 0x7f63300cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 -- 172.21.15.77:0/695630587 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63300c8a70 con 0x7f63300d06d0 2024-03-06T07:25:53.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632e7fc640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015ddf0 0x7f63300ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:53.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f632effd640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63300d06d0 0x7f63300c90a0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f632002beb0 tx=0x7f63200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f6317fff640 1 -- 172.21.15.77:0/695630587 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320035050 con 0x7f63300d06d0 2024-03-06T07:25:53.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f6317fff640 1 -- 172.21.15.77:0/695630587 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632003bd70 con 0x7f63300d06d0 2024-03-06T07:25:53.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f6317fff640 1 -- 172.21.15.77:0/695630587 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320006e10 con 0x7f63300d06d0 2024-03-06T07:25:53.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63300c79d0 con 0x7f63300d06d0 2024-03-06T07:25:53.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.599+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63300c7f90 con 0x7f63300d06d0 2024-03-06T07:25:53.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.601+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63300680d0 con 0x7f63300d06d0 2024-03-06T07:25:53.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.601+0000 7f6317fff640 1 -- 172.21.15.77:0/695630587 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6320038020 con 0x7f63300d06d0 2024-03-06T07:25:53.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.601+0000 7f6317fff640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f631c06b500 0x7f631c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.601+0000 7f632e7fc640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f631c06b500 0x7f631c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.602+0000 7f6317fff640 1 -- 172.21.15.77:0/695630587 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63200abf80 con 0x7f63300d06d0 2024-03-06T07:25:53.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.602+0000 7f632e7fc640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f631c06b500 0x7f631c06d9c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6330157510 tx=0x7f6324008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.606+0000 7f6317fff640 1 -- 172.21.15.77:0/695630587 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63300680d0 con 0x7f63300d06d0 2024-03-06T07:25:53.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.779+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63300ce010 con 0x7f63300d06d0 2024-03-06T07:25:53.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.780+0000 7f6317fff640 1 -- 172.21.15.77:0/695630587 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f6320040070 con 0x7f63300d06d0 2024-03-06T07:25:53.782 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:53.782 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":80,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:46.362962+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:42.937014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:43.684499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.782+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f631c06b500 msgr2=0x7f631c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.782+0000 7f633591e640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f631c06b500 0x7f631c06d9c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6330157510 tx=0x7f6324008040 comp rx=0 tx=0).stop 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.782+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63300d06d0 msgr2=0x7f63300c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.782+0000 7f633591e640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63300d06d0 0x7f63300c90a0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f632002beb0 tx=0x7f63200046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.783+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 shutdown_connections 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.783+0000 7f633591e640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f633015e1d0 0x7f63300cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.783+0000 7f633591e640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f633015ddf0 0x7f63300ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.783+0000 7f633591e640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f631c06b500 0x7f631c06d9c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.783+0000 7f633591e640 1 --2- 172.21.15.77:0/695630587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63300d06d0 0x7f63300c90a0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.783+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 >> 172.21.15.77:0/695630587 conn(0x7f633009cff0 msgr2=0x7f633015d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.783+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 shutdown_connections 2024-03-06T07:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.783+0000 7f633591e640 1 -- 172.21.15.77:0/695630587 wait complete. 2024-03-06T07:25:53.796 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:42.937014+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:43.684499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:53.796 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:53.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.897+0000 7f0a1f50d640 1 Processor -- start 2024-03-06T07:25:53.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.897+0000 7f0a1f50d640 1 -- start start 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1f50d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0a180cff40 0x7f0a180b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1f50d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a18158ac0 con 0x7f0a181586e0 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1f50d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1815b2a0 con 0x7f0a180cfa90 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1f50d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a180d0390 con 0x7f0a180cff40 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1d2a9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0a181586e0 0x7f0a180aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58114/0 (socket says 172.21.15.77:58114) 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1d2a9640 1 -- 172.21.15.77:0/811480556 learned_addr learned my addr 172.21.15.77:0/811480556 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1daaa640 1 --2- 172.21.15.77:0/811480556 >> v2:172.21.15.158:3300/0 conn(0x7f0a180cff40 0x7f0a180b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a0ffff640 1 -- 172.21.15.77:0/811480556 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747020486 0 0) 0x7f0a18158ac0 con 0x7f0a181586e0 2024-03-06T07:25:53.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a0ffff640 1 -- 172.21.15.77:0/811480556 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a04003520 con 0x7f0a181586e0 2024-03-06T07:25:53.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1daaa640 1 -- 172.21.15.77:0/811480556 >> v1:172.21.15.158:6789/0 conn(0x7f0a180cfa90 legacy=0x7f0a180b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1daaa640 1 -- 172.21.15.77:0/811480556 >> v1:172.21.15.77:6789/0 conn(0x7f0a181586e0 legacy=0x7f0a180aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.898+0000 7f0a1daaa640 1 -- 172.21.15.77:0/811480556 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1815b540 con 0x7f0a180cff40 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.899+0000 7f0a1daaa640 1 --2- 172.21.15.77:0/811480556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 0x7f0a180b3ab0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009d90 tx=0x7f0a1002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7602e3fe9782e45c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.899+0000 7f0a0ffff640 1 -- 172.21.15.77:0/811480556 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10039070 con 0x7f0a180cff40 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.899+0000 7f0a0ffff640 1 -- 172.21.15.77:0/811480556 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1002cd00 con 0x7f0a180cff40 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.899+0000 7f0a0ffff640 1 -- 172.21.15.77:0/811480556 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10034640 con 0x7f0a180cff40 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.900+0000 7f0a1f50d640 1 -- 172.21.15.77:0/811480556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 msgr2=0x7f0a180b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.900+0000 7f0a1f50d640 1 --2- 172.21.15.77:0/811480556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 0x7f0a180b3ab0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009d90 tx=0x7f0a1002bf50 comp rx=0 tx=0).stop 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.900+0000 7f0a1f50d640 1 -- 172.21.15.77:0/811480556 shutdown_connections 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.900+0000 7f0a1f50d640 1 --2- 172.21.15.77:0/811480556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 0x7f0a180b3ab0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.900+0000 7f0a1f50d640 1 -- 172.21.15.77:0/811480556 >> 172.21.15.77:0/811480556 conn(0x7f0a1806b190 msgr2=0x7f0a1806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.900+0000 7f0a1f50d640 1 -- 172.21.15.77:0/811480556 shutdown_connections 2024-03-06T07:25:53.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.900+0000 7f0a1f50d640 1 -- 172.21.15.77:0/811480556 wait complete. 2024-03-06T07:25:53.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.901+0000 7f0a1f50d640 1 Processor -- start 2024-03-06T07:25:53.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.901+0000 7f0a1f50d640 1 -- start start 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1f50d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a180cfa90 0x7f0a181454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1f50d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 0x7f0a180cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1f50d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a181586e0 0x7f0a1806c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1f50d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1815bde0 con 0x7f0a180cfa90 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1f50d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1815bb00 con 0x7f0a180cff40 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1f50d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a180af150 con 0x7f0a181586e0 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a180cfa90 0x7f0a181454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a180cfa90 0x7f0a181454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52222/0 (socket says 172.21.15.77:52222) 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 -- 172.21.15.77:0/463382695 learned_addr learned my addr 172.21.15.77:0/463382695 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1caa8640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 0x7f0a180cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1daaa640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a181586e0 0x7f0a1806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 -- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a181586e0 msgr2=0x7f0a1806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a181586e0 0x7f0a1806c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 -- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 msgr2=0x7f0a180cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 0x7f0a180cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 -- 172.21.15.77:0/463382695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a18145a20 con 0x7f0a180cfa90 2024-03-06T07:25:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1caa8640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 0x7f0a180cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1d2a9640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a180cfa90 0x7f0a181454e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400c410 tx=0x7f0a1400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a0dffb640 1 -- 172.21.15.77:0/463382695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14017070 con 0x7f0a180cfa90 2024-03-06T07:25:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a0dffb640 1 -- 172.21.15.77:0/463382695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a14005ce0 con 0x7f0a180cfa90 2024-03-06T07:25:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a180ce390 con 0x7f0a180cfa90 2024-03-06T07:25:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a0dffb640 1 -- 172.21.15.77:0/463382695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a140072f0 con 0x7f0a180cfa90 2024-03-06T07:25:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.902+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a180ce950 con 0x7f0a180cfa90 2024-03-06T07:25:53.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.903+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a180732b0 con 0x7f0a180cfa90 2024-03-06T07:25:53.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.904+0000 7f0a0dffb640 1 -- 172.21.15.77:0/463382695 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a1401a070 con 0x7f0a180cfa90 2024-03-06T07:25:53.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.904+0000 7f0a0dffb640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09fc06b4d0 0x7f09fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.904+0000 7f0a0dffb640 1 -- 172.21.15.77:0/463382695 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a140895d0 con 0x7f0a180cfa90 2024-03-06T07:25:53.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.904+0000 7f0a1caa8640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09fc06b4d0 0x7f09fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.905+0000 7f0a1caa8640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09fc06b4d0 0x7f09fc06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0a1815aa10 tx=0x7f0a08009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:53.909+0000 7f0a0dffb640 1 -- 172.21.15.77:0/463382695 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a18145a20 con 0x7f0a180cfa90 2024-03-06T07:25:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.082+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a1806e5f0 con 0x7f0a180cfa90 2024-03-06T07:25:54.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.082+0000 7f0a0dffb640 1 -- 172.21.15.77:0/463382695 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a14057ec0 con 0x7f0a180cfa90 2024-03-06T07:25:54.084 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:25:54.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09fc06b4d0 msgr2=0x7f09fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09fc06b4d0 0x7f09fc06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0a1815aa10 tx=0x7f0a08009040 comp rx=0 tx=0).stop 2024-03-06T07:25:54.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a180cfa90 msgr2=0x7f0a181454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a180cfa90 0x7f0a181454e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400c410 tx=0x7f0a1400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 shutdown_connections 2024-03-06T07:25:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a181586e0 0x7f0a1806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a180cff40 0x7f0a180cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f09fc06b4d0 0x7f09fc06d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 --2- 172.21.15.77:0/463382695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a180cfa90 0x7f0a181454e0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 >> 172.21.15.77:0/463382695 conn(0x7f0a1806b190 msgr2=0x7f0a180ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 shutdown_connections 2024-03-06T07:25:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.084+0000 7f0a1f50d640 1 -- 172.21.15.77:0/463382695 wait complete. 2024-03-06T07:25:54.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:54.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.197+0000 7f7fb100e640 1 Processor -- start 2024-03-06T07:25:54.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.198+0000 7f7fb100e640 1 -- start start 2024-03-06T07:25:54.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.198+0000 7f7fb100e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac1586e0 0x7f7fac0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.198+0000 7f7fb100e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7fac0aef80 0x7f7fac0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.198+0000 7f7fb100e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fac0cffd0 0x7f7fac06da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.198+0000 7f7fb100e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fac0d03f0 con 0x7f7fac0cffd0 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.198+0000 7f7fb100e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fac158ac0 con 0x7f7fac0aef80 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.198+0000 7f7fb100e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac15b2a0 con 0x7f7fac1586e0 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7fab59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fac0cffd0 0x7f7fac06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7fab59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fac0cffd0 0x7f7fac06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52230/0 (socket says 172.21.15.77:52230) 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7fab59e640 1 -- 172.21.15.77:0/1950778846 learned_addr learned my addr 172.21.15.77:0/1950778846 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7faad9d640 1 --2- 172.21.15.77:0/1950778846 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac1586e0 0x7f7fac0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7faa59c640 1 --2- 172.21.15.77:0/1950778846 >> v2:172.21.15.158:3300/0 conn(0x7f7fac0aef80 0x7f7fac0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7fab59e640 1 -- 172.21.15.77:0/1950778846 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac1586e0 msgr2=0x7f7fac0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7fab59e640 1 --2- 172.21.15.77:0/1950778846 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac1586e0 0x7f7fac0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7fab59e640 1 -- 172.21.15.77:0/1950778846 >> v2:172.21.15.158:3300/0 conn(0x7f7fac0aef80 msgr2=0x7f7fac0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7fab59e640 1 --2- 172.21.15.77:0/1950778846 >> v2:172.21.15.158:3300/0 conn(0x7f7fac0aef80 0x7f7fac0cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.199+0000 7f7fab59e640 1 -- 172.21.15.77:0/1950778846 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac15b540 con 0x7f7fac0cffd0 2024-03-06T07:25:54.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fab59e640 1 --2- 172.21.15.77:0/1950778846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac0cffd0 0x7f7fac06da20 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0d0610 tx=0x7f7fa402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55585d74635eff51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fa9d9b640 1 -- 172.21.15.77:0/1950778846 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa402ca70 con 0x7f7fac0cffd0 2024-03-06T07:25:54.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fa9d9b640 1 -- 172.21.15.77:0/1950778846 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa402cc10 con 0x7f7fac0cffd0 2024-03-06T07:25:54.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fa9d9b640 1 -- 172.21.15.77:0/1950778846 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40343f0 con 0x7f7fac0cffd0 2024-03-06T07:25:54.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 -- 172.21.15.77:0/1950778846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac0cffd0 msgr2=0x7f7fac06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 --2- 172.21.15.77:0/1950778846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac0cffd0 0x7f7fac06da20 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0d0610 tx=0x7f7fa402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 -- 172.21.15.77:0/1950778846 shutdown_connections 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 --2- 172.21.15.77:0/1950778846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac0cffd0 0x7f7fac06da20 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 --2- 172.21.15.77:0/1950778846 >> v2:172.21.15.158:3300/0 conn(0x7f7fac0aef80 0x7f7fac0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 --2- 172.21.15.77:0/1950778846 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac1586e0 0x7f7fac0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 -- 172.21.15.77:0/1950778846 >> 172.21.15.77:0/1950778846 conn(0x7f7fac06b190 msgr2=0x7f7fac06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 -- 172.21.15.77:0/1950778846 shutdown_connections 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.200+0000 7f7fb100e640 1 -- 172.21.15.77:0/1950778846 wait complete. 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.202+0000 7f7fb100e640 1 Processor -- start 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.202+0000 7f7fb100e640 1 -- start start 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.202+0000 7f7fb100e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7fac0aef80 0x7f7fac147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.202+0000 7f7fb100e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac0cffd0 0x7f7fac151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.202+0000 7f7fb100e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac1586e0 0x7f7fac156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.202+0000 7f7fb100e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac15bde0 con 0x7f7fac1586e0 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.202+0000 7f7fb100e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac15bb00 con 0x7f7fac0aef80 2024-03-06T07:25:54.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.202+0000 7f7fb100e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac071350 con 0x7f7fac0cffd0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac1586e0 0x7f7fac156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac1586e0 0x7f7fac156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52244/0 (socket says 172.21.15.77:52244) 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 -- 172.21.15.77:0/3158483530 learned_addr learned my addr 172.21.15.77:0/3158483530 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7faa59c640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac0cffd0 0x7f7fac151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7faad9d640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7fac0aef80 0x7f7fac147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 -- 172.21.15.77:0/3158483530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac0cffd0 msgr2=0x7f7fac151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac0cffd0 0x7f7fac151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 -- 172.21.15.77:0/3158483530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7fac0aef80 msgr2=0x7f7fac147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7fac0aef80 0x7f7fac147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 -- 172.21.15.77:0/3158483530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac152920 con 0x7f7fac1586e0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fab59e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac1586e0 0x7f7fac156500 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4036040 tx=0x7f7fa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fa37fe640 1 -- 172.21.15.77:0/3158483530 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4009510 con 0x7f7fac1586e0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fa37fe640 1 -- 172.21.15.77:0/3158483530 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4047680 con 0x7f7fac1586e0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac152c00 con 0x7f7fac1586e0 2024-03-06T07:25:54.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fa37fe640 1 -- 172.21.15.77:0/3158483530 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4047a70 con 0x7f7fac1586e0 2024-03-06T07:25:54.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.203+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac153700 con 0x7f7fac1586e0 2024-03-06T07:25:54.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.204+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac153120 con 0x7f7fac1586e0 2024-03-06T07:25:54.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.205+0000 7f7fa37fe640 1 -- 172.21.15.77:0/3158483530 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fa402ca80 con 0x7f7fac1586e0 2024-03-06T07:25:54.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.205+0000 7f7fa37fe640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f8c06b4d0 0x7f7f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.205+0000 7f7fa37fe640 1 -- 172.21.15.77:0/3158483530 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fa40aee10 con 0x7f7fac1586e0 2024-03-06T07:25:54.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.205+0000 7f7faad9d640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f8c06b4d0 0x7f7f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.206+0000 7f7faad9d640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f8c06b4d0 0x7f7f8c06d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009aa0 tx=0x7f7f94009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.209+0000 7f7fa37fe640 1 -- 172.21.15.77:0/3158483530 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fac153120 con 0x7f7fac1586e0 2024-03-06T07:25:54.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.379+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7fac0732b0 con 0x7f7fac1586e0 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.380+0000 7f7fa37fe640 1 -- 172.21.15.77:0/3158483530 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7fa407d640 con 0x7f7fac1586e0 2024-03-06T07:25:54.382 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f8c06b4d0 msgr2=0x7f7f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f8c06b4d0 0x7f7f8c06d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009aa0 tx=0x7f7f94009190 comp rx=0 tx=0).stop 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac1586e0 msgr2=0x7f7fac156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac1586e0 0x7f7fac156500 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4036040 tx=0x7f7fa4004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 shutdown_connections 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f8c06b4d0 0x7f7f8c06d990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fac1586e0 0x7f7fac156500 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7fac0cffd0 0x7f7fac151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 --2- 172.21.15.77:0/3158483530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7fac0aef80 0x7f7fac147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 >> 172.21.15.77:0/3158483530 conn(0x7f7fac06b190 msgr2=0x7f7fac0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 shutdown_connections 2024-03-06T07:25:54.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.382+0000 7f7fb100e640 1 -- 172.21.15.77:0/3158483530 wait complete. 2024-03-06T07:25:54.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:54.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.495+0000 7f9c7722a640 1 Processor -- start 2024-03-06T07:25:54.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c7722a640 1 -- start start 2024-03-06T07:25:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c7722a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c701614d0 0x7f9c700d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c7722a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c700d24b0 0x7f9c700b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c7722a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c701618b0 con 0x7f9c700d1910 2024-03-06T07:25:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c7722a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c700cf400 con 0x7f9c700d24b0 2024-03-06T07:25:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c7722a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c700cf6a0 con 0x7f9c701614d0 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c757c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c700d24b0 0x7f9c700b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c74fc6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9c701614d0 0x7f9c700d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c757c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c700d24b0 0x7f9c700b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52248/0 (socket says 172.21.15.77:52248) 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.496+0000 7f9c757c7640 1 -- 172.21.15.77:0/1498582454 learned_addr learned my addr 172.21.15.77:0/1498582454 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.497+0000 7f9c757c7640 1 -- 172.21.15.77:0/1498582454 >> v2:172.21.15.158:3300/0 conn(0x7f9c701614d0 msgr2=0x7f9c700d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.497+0000 7f9c757c7640 1 --2- 172.21.15.77:0/1498582454 >> v2:172.21.15.158:3300/0 conn(0x7f9c701614d0 0x7f9c700d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.497+0000 7f9c757c7640 1 -- 172.21.15.77:0/1498582454 >> v1:172.21.15.158:6789/0 conn(0x7f9c700d1910 legacy=0x7f9c700d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.497+0000 7f9c757c7640 1 -- 172.21.15.77:0/1498582454 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c700cf980 con 0x7f9c700d24b0 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.497+0000 7f9c757c7640 1 --2- 172.21.15.77:0/1498582454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c700d24b0 0x7f9c700b0e30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9c7008c470 tx=0x7f9c6802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2f20da2d72d271 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c6f7fe640 1 -- 172.21.15.77:0/1498582454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6802cb20 con 0x7f9c700d24b0 2024-03-06T07:25:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c6f7fe640 1 -- 172.21.15.77:0/1498582454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6802ccc0 con 0x7f9c700d24b0 2024-03-06T07:25:54.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c6f7fe640 1 -- 172.21.15.77:0/1498582454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68034610 con 0x7f9c700d24b0 2024-03-06T07:25:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c7722a640 1 -- 172.21.15.77:0/1498582454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c700d24b0 msgr2=0x7f9c700b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c7722a640 1 --2- 172.21.15.77:0/1498582454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c700d24b0 0x7f9c700b0e30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9c7008c470 tx=0x7f9c6802bf80 comp rx=0 tx=0).stop 2024-03-06T07:25:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c7722a640 1 -- 172.21.15.77:0/1498582454 shutdown_connections 2024-03-06T07:25:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c7722a640 1 --2- 172.21.15.77:0/1498582454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c700d24b0 0x7f9c700b0e30 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c7722a640 1 --2- 172.21.15.77:0/1498582454 >> v2:172.21.15.158:3300/0 conn(0x7f9c701614d0 0x7f9c700d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c7722a640 1 -- 172.21.15.77:0/1498582454 >> 172.21.15.77:0/1498582454 conn(0x7f9c700d3d40 msgr2=0x7f9c700d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c7722a640 1 -- 172.21.15.77:0/1498582454 shutdown_connections 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.498+0000 7f9c7722a640 1 -- 172.21.15.77:0/1498582454 wait complete. 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.499+0000 7f9c7722a640 1 Processor -- start 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.499+0000 7f9c7722a640 1 -- start start 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c7722a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c700d1910 0x7f9c700b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c7722a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c700d24b0 0x7f9c700b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c7722a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c701614d0 0x7f9c700ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c7722a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700b1370 con 0x7f9c701614d0 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c7722a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700b0910 con 0x7f9c700d24b0 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c7722a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700b14f0 con 0x7f9c700d1910 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c757c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c701614d0 0x7f9c700ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c757c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c701614d0 0x7f9c700ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52260/0 (socket says 172.21.15.77:52260) 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c757c7640 1 -- 172.21.15.77:0/932953524 learned_addr learned my addr 172.21.15.77:0/932953524 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c6ffff640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c700d24b0 0x7f9c700b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.500+0000 7f9c74fc6640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c700d1910 0x7f9c700b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c757c7640 1 -- 172.21.15.77:0/932953524 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c700d1910 msgr2=0x7f9c700b98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c757c7640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c700d1910 0x7f9c700b98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c757c7640 1 -- 172.21.15.77:0/932953524 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c700d24b0 msgr2=0x7f9c700b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c757c7640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c700d24b0 0x7f9c700b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c757c7640 1 -- 172.21.15.77:0/932953524 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c700cd7d0 con 0x7f9c701614d0 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c757c7640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c701614d0 0x7f9c700ca060 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9c68009f40 tx=0x7f9c68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c6d7fa640 1 -- 172.21.15.77:0/932953524 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68043020 con 0x7f9c701614d0 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c6d7fa640 1 -- 172.21.15.77:0/932953524 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6803e450 con 0x7f9c701614d0 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c6d7fa640 1 -- 172.21.15.77:0/932953524 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68009510 con 0x7f9c701614d0 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c700b7160 con 0x7f9c701614d0 2024-03-06T07:25:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.501+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c700ccfd0 con 0x7f9c701614d0 2024-03-06T07:25:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.502+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c700ccb20 con 0x7f9c701614d0 2024-03-06T07:25:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.502+0000 7f9c6d7fa640 1 -- 172.21.15.77:0/932953524 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c68046020 con 0x7f9c701614d0 2024-03-06T07:25:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.503+0000 7f9c6d7fa640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5c06b4d0 0x7f9c5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.503+0000 7f9c6d7fa640 1 -- 172.21.15.77:0/932953524 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c680ac810 con 0x7f9c701614d0 2024-03-06T07:25:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.503+0000 7f9c74fc6640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5c06b4d0 0x7f9c5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.503+0000 7f9c74fc6640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5c06b4d0 0x7f9c5c06d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009aa0 tx=0x7f9c60009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.507+0000 7f9c6d7fa640 1 -- 172.21.15.77:0/932953524 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c6804d030 con 0x7f9c701614d0 2024-03-06T07:25:54.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.676+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9c70068ff0 con 0x7f9c701614d0 2024-03-06T07:25:54.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.676+0000 7f9c6d7fa640 1 -- 172.21.15.77:0/932953524 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9c6807b040 con 0x7f9c701614d0 2024-03-06T07:25:54.678 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:25:54.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5c06b4d0 msgr2=0x7f9c5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5c06b4d0 0x7f9c5c06d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009aa0 tx=0x7f9c60009190 comp rx=0 tx=0).stop 2024-03-06T07:25:54.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c701614d0 msgr2=0x7f9c700ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c701614d0 0x7f9c700ca060 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9c68009f40 tx=0x7f9c68004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 shutdown_connections 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9c5c06b4d0 0x7f9c5c06d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c701614d0 0x7f9c700ca060 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9c700d24b0 0x7f9c700b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 --2- 172.21.15.77:0/932953524 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9c700d1910 0x7f9c700b98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 >> 172.21.15.77:0/932953524 conn(0x7f9c700d3d40 msgr2=0x7f9c700d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 shutdown_connections 2024-03-06T07:25:54.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.678+0000 7f9c7722a640 1 -- 172.21.15.77:0/932953524 wait complete. 2024-03-06T07:25:54.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:54.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.790+0000 7fb1eb913640 1 Processor -- start 2024-03-06T07:25:54.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.790+0000 7fb1eb913640 1 -- start start 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1eb913640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1e41444c0 0x7fb1e406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1eb913640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e415b2d0 0x7fb1e40708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1eb913640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1e4070e10 0x7fb1e415e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1eb913640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e41448a0 con 0x7fb1e41444c0 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1eb913640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e4071260 con 0x7fb1e4070e10 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1eb913640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e406d370 con 0x7fb1e415b2d0 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e96af640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1e41444c0 0x7fb1e406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e96af640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1e41444c0 0x7fb1e406ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52262/0 (socket says 172.21.15.77:52262) 2024-03-06T07:25:54.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e96af640 1 -- 172.21.15.77:0/366757515 learned_addr learned my addr 172.21.15.77:0/366757515 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e9eb0640 1 --2- 172.21.15.77:0/366757515 >> v2:172.21.15.158:3300/0 conn(0x7fb1e4070e10 0x7fb1e415e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e8eae640 1 --2- 172.21.15.77:0/366757515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e415b2d0 0x7fb1e40708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e96af640 1 -- 172.21.15.77:0/366757515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e415b2d0 msgr2=0x7fb1e40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e96af640 1 --2- 172.21.15.77:0/366757515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e415b2d0 0x7fb1e40708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e96af640 1 -- 172.21.15.77:0/366757515 >> v2:172.21.15.158:3300/0 conn(0x7fb1e4070e10 msgr2=0x7fb1e415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e96af640 1 --2- 172.21.15.77:0/366757515 >> v2:172.21.15.158:3300/0 conn(0x7fb1e4070e10 0x7fb1e415e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.791+0000 7fb1e96af640 1 -- 172.21.15.77:0/366757515 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e406d650 con 0x7fb1e41444c0 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.792+0000 7fb1e96af640 1 --2- 172.21.15.77:0/366757515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41444c0 0x7fb1e406ca00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40cf510 tx=0x7fb1e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40f852bfda5cf33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.792+0000 7fb1dbfff640 1 -- 172.21.15.77:0/366757515 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e002ca70 con 0x7fb1e41444c0 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.792+0000 7fb1dbfff640 1 -- 172.21.15.77:0/366757515 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e002cc10 con 0x7fb1e41444c0 2024-03-06T07:25:54.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.792+0000 7fb1dbfff640 1 -- 172.21.15.77:0/366757515 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e00343f0 con 0x7fb1e41444c0 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 -- 172.21.15.77:0/366757515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41444c0 msgr2=0x7fb1e406ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 --2- 172.21.15.77:0/366757515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41444c0 0x7fb1e406ca00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40cf510 tx=0x7fb1e002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 -- 172.21.15.77:0/366757515 shutdown_connections 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 --2- 172.21.15.77:0/366757515 >> v2:172.21.15.158:3300/0 conn(0x7fb1e4070e10 0x7fb1e415e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 --2- 172.21.15.77:0/366757515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e415b2d0 0x7fb1e40708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 --2- 172.21.15.77:0/366757515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41444c0 0x7fb1e406ca00 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 -- 172.21.15.77:0/366757515 >> 172.21.15.77:0/366757515 conn(0x7fb1e40b1c80 msgr2=0x7fb1e40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 -- 172.21.15.77:0/366757515 shutdown_connections 2024-03-06T07:25:54.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.793+0000 7fb1eb913640 1 -- 172.21.15.77:0/366757515 wait complete. 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.794+0000 7fb1eb913640 1 Processor -- start 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.794+0000 7fb1eb913640 1 -- start start 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1eb913640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1e4070e10 0x7fb1e41380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1eb913640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e41444c0 0x7fb1e40c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1eb913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e415b2d0 0x7fb1e40cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1eb913640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e406def0 con 0x7fb1e415b2d0 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1eb913640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e406dc10 con 0x7fb1e4070e10 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1eb913640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e406cf40 con 0x7fb1e41444c0 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e415b2d0 0x7fb1e40cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e415b2d0 0x7fb1e40cc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52270/0 (socket says 172.21.15.77:52270) 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 -- 172.21.15.77:0/1097360114 learned_addr learned my addr 172.21.15.77:0/1097360114 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e8eae640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e41444c0 0x7fb1e40c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e96af640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1e4070e10 0x7fb1e41380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 -- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e41444c0 msgr2=0x7fb1e40c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e41444c0 0x7fb1e40c48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 -- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1e4070e10 msgr2=0x7fb1e41380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1e4070e10 0x7fb1e41380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 -- 172.21.15.77:0/1097360114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e40c51c0 con 0x7fb1e415b2d0 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e96af640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1e4070e10 0x7fb1e41380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e8eae640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e41444c0 0x7fb1e40c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1e9eb0640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e415b2d0 0x7fb1e40cc3a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00e7f0 tx=0x7fb1dc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1d9ffb640 1 -- 172.21.15.77:0/1097360114 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc017070 con 0x7fb1e415b2d0 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1d9ffb640 1 -- 172.21.15.77:0/1097360114 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc00c730 con 0x7fb1e415b2d0 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.795+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e40c54a0 con 0x7fb1e415b2d0 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.796+0000 7fb1d9ffb640 1 -- 172.21.15.77:0/1097360114 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc00cb00 con 0x7fb1e415b2d0 2024-03-06T07:25:54.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.796+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e40c5a60 con 0x7fb1e415b2d0 2024-03-06T07:25:54.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.797+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e404f330 con 0x7fb1e415b2d0 2024-03-06T07:25:54.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.797+0000 7fb1d9ffb640 1 -- 172.21.15.77:0/1097360114 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1dc01a070 con 0x7fb1e415b2d0 2024-03-06T07:25:54.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.797+0000 7fb1d9ffb640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c806b500 0x7fb1c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.797+0000 7fb1d9ffb640 1 -- 172.21.15.77:0/1097360114 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1dc08a2d0 con 0x7fb1e415b2d0 2024-03-06T07:25:54.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.797+0000 7fb1e96af640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c806b500 0x7fb1c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.798+0000 7fb1e96af640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c806b500 0x7fb1c806d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0035000 tx=0x7fb1e0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.802+0000 7fb1d9ffb640 1 -- 172.21.15.77:0/1097360114 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1dc0591e0 con 0x7fb1e415b2d0 2024-03-06T07:25:54.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.985+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb1e4141940 con 0x7fb1e415b2d0 2024-03-06T07:25:54.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.986+0000 7fb1d9ffb640 1 -- 172.21.15.77:0/1097360114 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v80) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb1dc058ae0 con 0x7fb1e415b2d0 2024-03-06T07:25:54.987 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c806b500 msgr2=0x7fb1c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c806b500 0x7fb1c806d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0035000 tx=0x7fb1e0030000 comp rx=0 tx=0).stop 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e415b2d0 msgr2=0x7fb1e40cc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e415b2d0 0x7fb1e40cc3a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00e7f0 tx=0x7fb1dc00ecb0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 shutdown_connections 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1c806b500 0x7fb1c806d9c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e415b2d0 0x7fb1e40cc3a0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1e41444c0 0x7fb1e40c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 --2- 172.21.15.77:0/1097360114 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1e4070e10 0x7fb1e41380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 >> 172.21.15.77:0/1097360114 conn(0x7fb1e40b1c80 msgr2=0x7fb1e413ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 shutdown_connections 2024-03-06T07:25:54.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:54.988+0000 7fb1eb913640 1 -- 172.21.15.77:0/1097360114 wait complete. 2024-03-06T07:25:55.000 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:25:55.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:55.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:55.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[65522]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:55.027 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:25:55.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:55.027 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:55.082 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:25:55.082 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:55.098 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:55.098 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:25:55.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:55.099 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:55.153 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:55.153 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:25:55.153 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:55.168 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:55.169 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:25:55.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:55.169 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:55.223 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:55.223 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:25:55.224 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:55.238 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:55.238 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:25:55.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:55.239 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:55.292 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:55.293 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.394+0000 7f84c5e0f640 1 Processor -- start 2024-03-06T07:25:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.394+0000 7f84c5e0f640 1 -- start start 2024-03-06T07:25:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84c5e0f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f84c015bf30 0x7f84c015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84c5e0f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c015f580 con 0x7f84c00b3060 2024-03-06T07:25:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84c5e0f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c015b450 con 0x7f84c013dc10 2024-03-06T07:25:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84c5e0f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c00cf3f0 con 0x7f84c015bf30 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84beffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f84c00b3060 0x7f84c015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58122/0 (socket says 172.21.15.77:58122) 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84beffd640 1 -- 172.21.15.77:0/3352837457 learned_addr learned my addr 172.21.15.77:0/3352837457 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84bffff640 1 --2- 172.21.15.77:0/3352837457 >> v2:172.21.15.158:3300/0 conn(0x7f84c015bf30 0x7f84c015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84be7fc640 1 -- 172.21.15.77:0/3352837457 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234412283 0 0) 0x7f84c015f580 con 0x7f84c00b3060 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84be7fc640 1 -- 172.21.15.77:0/3352837457 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a8003520 con 0x7f84c00b3060 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84bffff640 1 -- 172.21.15.77:0/3352837457 >> v1:172.21.15.158:6789/0 conn(0x7f84c013dc10 legacy=0x7f84c00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84bffff640 1 -- 172.21.15.77:0/3352837457 >> v1:172.21.15.77:6789/0 conn(0x7f84c00b3060 legacy=0x7f84c015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.395+0000 7f84be7fc640 1 -- 172.21.15.77:0/3352837457 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404935625 0 0) 0x7f84c015b450 con 0x7f84c013dc10 2024-03-06T07:25:55.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.396+0000 7f84bffff640 1 -- 172.21.15.77:0/3352837457 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c015b730 con 0x7f84c015bf30 2024-03-06T07:25:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.396+0000 7f84bffff640 1 --2- 172.21.15.77:0/3352837457 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 0x7f84c015ef60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009d90 tx=0x7f84b802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec35a1a1b93a219c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.396+0000 7f84be7fc640 1 -- 172.21.15.77:0/3352837457 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8039070 con 0x7f84c015bf30 2024-03-06T07:25:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.396+0000 7f84be7fc640 1 -- 172.21.15.77:0/3352837457 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b802cc60 con 0x7f84c015bf30 2024-03-06T07:25:55.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.396+0000 7f84be7fc640 1 -- 172.21.15.77:0/3352837457 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8034640 con 0x7f84c015bf30 2024-03-06T07:25:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.397+0000 7f84c5e0f640 1 -- 172.21.15.77:0/3352837457 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 msgr2=0x7f84c015ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.397+0000 7f84c5e0f640 1 --2- 172.21.15.77:0/3352837457 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 0x7f84c015ef60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009d90 tx=0x7f84b802beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.397+0000 7f84c5e0f640 1 -- 172.21.15.77:0/3352837457 shutdown_connections 2024-03-06T07:25:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.397+0000 7f84c5e0f640 1 --2- 172.21.15.77:0/3352837457 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 0x7f84c015ef60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.397+0000 7f84c5e0f640 1 -- 172.21.15.77:0/3352837457 >> 172.21.15.77:0/3352837457 conn(0x7f84c00cf8a0 msgr2=0x7f84c00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.397+0000 7f84c5e0f640 1 -- 172.21.15.77:0/3352837457 shutdown_connections 2024-03-06T07:25:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.397+0000 7f84c5e0f640 1 -- 172.21.15.77:0/3352837457 wait complete. 2024-03-06T07:25:55.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.398+0000 7f84c5e0f640 1 Processor -- start 2024-03-06T07:25:55.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.398+0000 7f84c5e0f640 1 -- start start 2024-03-06T07:25:55.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84c5e0f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f84c00b3060 0x7f84c016a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84c5e0f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84c013dc10 0x7f84c01340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84c5e0f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 0x7f84c014faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84c5e0f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c00718e0 con 0x7f84c013dc10 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84c5e0f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c015b210 con 0x7f84c015bf30 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84c5e0f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c0071a60 con 0x7f84c00b3060 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84c013dc10 0x7f84c01340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84c013dc10 0x7f84c01340f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52280/0 (socket says 172.21.15.77:52280) 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84beffd640 1 -- 172.21.15.77:0/1974006251 learned_addr learned my addr 172.21.15.77:0/1974006251 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84bffff640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 0x7f84c014faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84bf7fe640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f84c00b3060 0x7f84c016a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84beffd640 1 -- 172.21.15.77:0/1974006251 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f84c00b3060 msgr2=0x7f84c016a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84beffd640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f84c00b3060 0x7f84c016a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84beffd640 1 -- 172.21.15.77:0/1974006251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 msgr2=0x7f84c014faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84beffd640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 0x7f84c014faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.399+0000 7f84beffd640 1 -- 172.21.15.77:0/1974006251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0168f10 con 0x7f84c013dc10 2024-03-06T07:25:55.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.400+0000 7f84beffd640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84c013dc10 0x7f84c01340f0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f84b400c430 tx=0x7f84b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.400+0000 7f84a7fff640 1 -- 172.21.15.77:0/1974006251 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4017070 con 0x7f84c013dc10 2024-03-06T07:25:55.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.400+0000 7f84a7fff640 1 -- 172.21.15.77:0/1974006251 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b4005ce0 con 0x7f84c013dc10 2024-03-06T07:25:55.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.400+0000 7f84a7fff640 1 -- 172.21.15.77:0/1974006251 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b40072f0 con 0x7f84c013dc10 2024-03-06T07:25:55.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.400+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c0135050 con 0x7f84c013dc10 2024-03-06T07:25:55.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.400+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c0134d70 con 0x7f84c013dc10 2024-03-06T07:25:55.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.401+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c004f330 con 0x7f84c013dc10 2024-03-06T07:25:55.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.401+0000 7f84a7fff640 1 -- 172.21.15.77:0/1974006251 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84b401a070 con 0x7f84c013dc10 2024-03-06T07:25:55.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.401+0000 7f84a7fff640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f84ac06b4d0 0x7f84ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.402+0000 7f84bf7fe640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f84ac06b4d0 0x7f84ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.402+0000 7f84a7fff640 1 -- 172.21.15.77:0/1974006251 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84b40896c0 con 0x7f84c013dc10 2024-03-06T07:25:55.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.402+0000 7f84bf7fe640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f84ac06b4d0 0x7f84ac06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000f60 tx=0x7f84b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.406+0000 7f84a7fff640 1 -- 172.21.15.77:0/1974006251 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84b4057e60 con 0x7f84c013dc10 2024-03-06T07:25:55.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.615+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84c006eb00 con 0x7f84c013dc10 2024-03-06T07:25:55.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.616+0000 7f84a7fff640 1 -- 172.21.15.77:0/1974006251 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f84b405bf00 con 0x7f84c013dc10 2024-03-06T07:25:55.618 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:55.618 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3408486989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3408486989},{"type":"v1","addr":"172.21.15.158:6835","nonce":3408486989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/324866984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":324866984},{"type":"v1","addr":"172.21.15.77:6836","nonce":324866984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2426038221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2426038221},{"type":"v1","addr":"172.21.15.77:6837","nonce":2426038221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:44.712557+0000","modified":"2024-03-06T07:25:47.370110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6930},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6930":{"gid":6930,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/3775835533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3775835533},{"type":"v1","addr":"172.21.15.158:6834","nonce":3775835533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6930,"qdb_cluster":[6930]},"id":6}]} 2024-03-06T07:25:55.618 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2024-03-06T07:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f84ac06b4d0 msgr2=0x7f84ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f84ac06b4d0 0x7f84ac06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000f60 tx=0x7f84b0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84c013dc10 msgr2=0x7f84c01340f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84c013dc10 0x7f84c01340f0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f84b400c430 tx=0x7f84b400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 shutdown_connections 2024-03-06T07:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f84ac06b4d0 0x7f84ac06d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84c013dc10 0x7f84c01340f0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f84c015bf30 0x7f84c014faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 --2- 172.21.15.77:0/1974006251 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f84c00b3060 0x7f84c016a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 >> 172.21.15.77:0/1974006251 conn(0x7f84c00cf8a0 msgr2=0x7f84c00afbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 shutdown_connections 2024-03-06T07:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.618+0000 7f84c5e0f640 1 -- 172.21.15.77:0/1974006251 wait complete. 2024-03-06T07:25:55.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:55.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.729+0000 7ffb7452c640 1 Processor -- start 2024-03-06T07:25:55.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.729+0000 7ffb7452c640 1 -- start start 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb7452c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb7452c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c15f580 con 0x7ffb6c13dc10 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb7452c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c15b450 con 0x7ffb6c15bf30 2024-03-06T07:25:55.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb7452c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0cf3f0 con 0x7ffb6c0b3060 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb722c8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffb6c13dc10 0x7ffb6c0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58138/0 (socket says 172.21.15.77:58138) 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb722c8640 1 -- 172.21.15.77:0/2190662650 learned_addr learned my addr 172.21.15.77:0/2190662650 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb71ac7640 1 --2- 172.21.15.77:0/2190662650 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb712c6640 1 -- 172.21.15.77:0/2190662650 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570207926 0 0) 0x7ffb6c15f580 con 0x7ffb6c13dc10 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb712c6640 1 -- 172.21.15.77:0/2190662650 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb58003520 con 0x7ffb6c13dc10 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb712c6640 1 -- 172.21.15.77:0/2190662650 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936057710 0 0) 0x7ffb6c15b450 con 0x7ffb6c15bf30 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb712c6640 1 -- 172.21.15.77:0/2190662650 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c15f580 con 0x7ffb6c15bf30 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb71ac7640 1 -- 172.21.15.77:0/2190662650 >> v1:172.21.15.158:6789/0 conn(0x7ffb6c15bf30 legacy=0x7ffb6c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb71ac7640 1 -- 172.21.15.77:0/2190662650 >> v1:172.21.15.77:6789/0 conn(0x7ffb6c13dc10 legacy=0x7ffb6c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.730+0000 7ffb71ac7640 1 -- 172.21.15.77:0/2190662650 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c15b730 con 0x7ffb6c0b3060 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.731+0000 7ffb71ac7640 1 --2- 172.21.15.77:0/2190662650 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c15e850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0097d0 tx=0x7ffb5c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5edb2e0793eace36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.731+0000 7ffb712c6640 1 -- 172.21.15.77:0/2190662650 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c035020 con 0x7ffb6c0b3060 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.731+0000 7ffb712c6640 1 -- 172.21.15.77:0/2190662650 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c031800 con 0x7ffb6c0b3060 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.731+0000 7ffb712c6640 1 -- 172.21.15.77:0/2190662650 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c031bb0 con 0x7ffb6c0b3060 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.732+0000 7ffb7452c640 1 -- 172.21.15.77:0/2190662650 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 msgr2=0x7ffb6c15e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.732+0000 7ffb7452c640 1 --2- 172.21.15.77:0/2190662650 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c15e850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0097d0 tx=0x7ffb5c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.732+0000 7ffb7452c640 1 -- 172.21.15.77:0/2190662650 shutdown_connections 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.732+0000 7ffb7452c640 1 --2- 172.21.15.77:0/2190662650 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c15e850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.732+0000 7ffb7452c640 1 -- 172.21.15.77:0/2190662650 >> 172.21.15.77:0/2190662650 conn(0x7ffb6c0cf8a0 msgr2=0x7ffb6c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.732+0000 7ffb7452c640 1 -- 172.21.15.77:0/2190662650 shutdown_connections 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.732+0000 7ffb7452c640 1 -- 172.21.15.77:0/2190662650 wait complete. 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.733+0000 7ffb7452c640 1 Processor -- start 2024-03-06T07:25:55.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.733+0000 7ffb7452c640 1 -- start start 2024-03-06T07:25:55.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb7452c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb7452c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb6c13dc10 0x7ffb6c13b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb7452c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb6c15bf30 0x7ffb6c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb7452c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0718e0 con 0x7ffb6c15bf30 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb7452c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c15b210 con 0x7ffb6c13dc10 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb7452c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c071a60 con 0x7ffb6c0b3060 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb6c15bf30 0x7ffb6c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb6c15bf30 0x7ffb6c15d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52288/0 (socket says 172.21.15.77:52288) 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 -- 172.21.15.77:0/1922858572 learned_addr learned my addr 172.21.15.77:0/1922858572 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb71ac7640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb6c13dc10 0x7ffb6c13b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb722c8640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 -- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 msgr2=0x7ffb6c158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 -- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb6c13dc10 msgr2=0x7ffb6c13b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb6c13dc10 0x7ffb6c13b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 -- 172.21.15.77:0/1922858572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c140d80 con 0x7ffb6c15bf30 2024-03-06T07:25:55.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb71ac7640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb6c13dc10 0x7ffb6c13b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:55.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb72ac9640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb6c15bf30 0x7ffb6c15d3b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400c710 tx=0x7ffb6400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb62ffd640 1 -- 172.21.15.77:0/1922858572 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64003b70 con 0x7ffb6c15bf30 2024-03-06T07:25:55.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb62ffd640 1 -- 172.21.15.77:0/1922858572 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64003d10 con 0x7ffb6c15bf30 2024-03-06T07:25:55.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c141060 con 0x7ffb6c15bf30 2024-03-06T07:25:55.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb62ffd640 1 -- 172.21.15.77:0/1922858572 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6400aa50 con 0x7ffb6c15bf30 2024-03-06T07:25:55.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.734+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c06cd30 con 0x7ffb6c15bf30 2024-03-06T07:25:55.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.735+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6c04f330 con 0x7ffb6c15bf30 2024-03-06T07:25:55.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.736+0000 7ffb62ffd640 1 -- 172.21.15.77:0/1922858572 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb6400abf0 con 0x7ffb6c15bf30 2024-03-06T07:25:55.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.736+0000 7ffb62ffd640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb5006b4d0 0x7ffb5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.736+0000 7ffb62ffd640 1 -- 172.21.15.77:0/1922858572 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb64089f30 con 0x7ffb6c15bf30 2024-03-06T07:25:55.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.736+0000 7ffb722c8640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb5006b4d0 0x7ffb5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.737+0000 7ffb722c8640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb5006b4d0 0x7ffb5006d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ffb680011e0 tx=0x7ffb68007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.741+0000 7ffb62ffd640 1 -- 172.21.15.77:0/1922858572 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb6400ce20 con 0x7ffb6c15bf30 2024-03-06T07:25:55.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.910+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb6c13c390 con 0x7ffb6c15bf30 2024-03-06T07:25:55.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.911+0000 7ffb62ffd640 1 -- 172.21.15.77:0/1922858572 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16648 (secure 0 0 0) 0x7ffb640587f0 con 0x7ffb6c15bf30 2024-03-06T07:25:55.913 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:55.914 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":80,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:46.362962+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:42.937014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:43.684499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:55.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb5006b4d0 msgr2=0x7ffb5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb5006b4d0 0x7ffb5006d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ffb680011e0 tx=0x7ffb68007d30 comp rx=0 tx=0).stop 2024-03-06T07:25:55.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb6c15bf30 msgr2=0x7ffb6c15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb6c15bf30 0x7ffb6c15d3b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400c710 tx=0x7ffb6400cbd0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 shutdown_connections 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb5006b4d0 0x7ffb5006d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb6c15bf30 0x7ffb6c15d3b0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb6c13dc10 0x7ffb6c13b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 --2- 172.21.15.77:0/1922858572 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb6c0b3060 0x7ffb6c158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 >> 172.21.15.77:0/1922858572 conn(0x7ffb6c0cf8a0 msgr2=0x7ffb6c0cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 shutdown_connections 2024-03-06T07:25:55.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:55.913+0000 7ffb7452c640 1 -- 172.21.15.77:0/1922858572 wait complete. 2024-03-06T07:25:55.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:42.937014+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:43.684499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:55.927 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.028+0000 7f58c31f7640 1 Processor -- start 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.028+0000 7f58c31f7640 1 -- start start 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c31f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc0cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c31f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f58bc0b33a0 0x7f58bc0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c31f7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc158ac0 con 0x7f58bc1586e0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c31f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc15b2a0 con 0x7f58bc0b33a0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c31f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc15b540 con 0x7f58bc0cfa90 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c1794640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f58bc0b33a0 0x7f58bc0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c0f93640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f58bc1586e0 0x7f58bc0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58144/0 (socket says 172.21.15.77:58144) 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c0f93640 1 -- 172.21.15.77:0/3951746312 learned_addr learned my addr 172.21.15.77:0/3951746312 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58bbfff640 1 --2- 172.21.15.77:0/3951746312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc0cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58bb7fe640 1 -- 172.21.15.77:0/3951746312 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510663021 0 0) 0x7f58bc158ac0 con 0x7f58bc1586e0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58bb7fe640 1 -- 172.21.15.77:0/3951746312 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ac005730 con 0x7f58bc1586e0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c1794640 1 -- 172.21.15.77:0/3951746312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 msgr2=0x7f58bc0cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c1794640 1 --2- 172.21.15.77:0/3951746312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc0cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c1794640 1 -- 172.21.15.77:0/3951746312 >> v1:172.21.15.77:6789/0 conn(0x7f58bc1586e0 legacy=0x7f58bc0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.029+0000 7f58c1794640 1 -- 172.21.15.77:0/3951746312 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc15b820 con 0x7f58bc0b33a0 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.030+0000 7f58c1794640 1 --2- 172.21.15.77:0/3951746312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0b33a0 0x7f58bc0b37f0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009d90 tx=0x7f58b402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0afb5b4fcc8648 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.030+0000 7f58bb7fe640 1 -- 172.21.15.77:0/3951746312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4039070 con 0x7f58bc0b33a0 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.030+0000 7f58bb7fe640 1 -- 172.21.15.77:0/3951746312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b402cc50 con 0x7f58bc0b33a0 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.030+0000 7f58bb7fe640 1 -- 172.21.15.77:0/3951746312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4034440 con 0x7f58bc0b33a0 2024-03-06T07:25:56.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.030+0000 7f58c31f7640 1 -- 172.21.15.77:0/3951746312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0b33a0 msgr2=0x7f58bc0b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.030+0000 7f58c31f7640 1 --2- 172.21.15.77:0/3951746312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0b33a0 0x7f58bc0b37f0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009d90 tx=0x7f58b402bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.031+0000 7f58c31f7640 1 -- 172.21.15.77:0/3951746312 shutdown_connections 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.031+0000 7f58c31f7640 1 --2- 172.21.15.77:0/3951746312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc0b33a0 0x7f58bc0b37f0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.031+0000 7f58c31f7640 1 --2- 172.21.15.77:0/3951746312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc0cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.031+0000 7f58c31f7640 1 -- 172.21.15.77:0/3951746312 >> 172.21.15.77:0/3951746312 conn(0x7f58bc06b190 msgr2=0x7f58bc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.031+0000 7f58c31f7640 1 -- 172.21.15.77:0/3951746312 shutdown_connections 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.031+0000 7f58c31f7640 1 -- 172.21.15.77:0/3951746312 wait complete. 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.032+0000 7f58c31f7640 1 Processor -- start 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.032+0000 7f58c31f7640 1 -- start start 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c31f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b33a0 0x7f58bc15a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c31f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc1ef5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c31f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc1586e0 0x7f58bc0ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c31f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0af150 con 0x7f58bc1586e0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c31f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc15bde0 con 0x7f58bc0b33a0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c31f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc071350 con 0x7f58bc0cfa90 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc1586e0 0x7f58bc0ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc1586e0 0x7f58bc0ce020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52300/0 (socket says 172.21.15.77:52300) 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 -- 172.21.15.77:0/2720758118 learned_addr learned my addr 172.21.15.77:0/2720758118 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c0f93640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b33a0 0x7f58bc15a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58bbfff640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc1ef5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 -- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 msgr2=0x7f58bc1ef5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc1ef5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 -- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b33a0 msgr2=0x7f58bc15a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b33a0 0x7f58bc15a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 -- 172.21.15.77:0/2720758118 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc15aa10 con 0x7f58bc1586e0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58bbfff640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc1ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c1794640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc1586e0 0x7f58bc0ce020 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009d90 tx=0x7f58b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58b97fa640 1 -- 172.21.15.77:0/2720758118 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4039050 con 0x7f58bc1586e0 2024-03-06T07:25:56.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58b97fa640 1 -- 172.21.15.77:0/2720758118 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b4045070 con 0x7f58bc1586e0 2024-03-06T07:25:56.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc150260 con 0x7f58bc1586e0 2024-03-06T07:25:56.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58b97fa640 1 -- 172.21.15.77:0/2720758118 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4009450 con 0x7f58bc1586e0 2024-03-06T07:25:56.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.033+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc0cd260 con 0x7f58bc1586e0 2024-03-06T07:25:56.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.034+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc0cdcf0 con 0x7f58bc1586e0 2024-03-06T07:25:56.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.035+0000 7f58b97fa640 1 -- 172.21.15.77:0/2720758118 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58b403e610 con 0x7f58bc1586e0 2024-03-06T07:25:56.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.035+0000 7f58b97fa640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58a006b4d0 0x7f58a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.035+0000 7f58b97fa640 1 -- 172.21.15.77:0/2720758118 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58b40acb70 con 0x7f58bc1586e0 2024-03-06T07:25:56.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.035+0000 7f58c0f93640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58a006b4d0 0x7f58a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.035+0000 7f58c0f93640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58a006b4d0 0x7f58a006d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f58b0005880 tx=0x7f58b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.039+0000 7f58b97fa640 1 -- 172.21.15.77:0/2720758118 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58b407b9f0 con 0x7f58bc1586e0 2024-03-06T07:25:56.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.213+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58bc068ff0 con 0x7f58bc1586e0 2024-03-06T07:25:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.213+0000 7f58b97fa640 1 -- 172.21.15.77:0/2720758118 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58b407b2f0 con 0x7f58bc1586e0 2024-03-06T07:25:56.215 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58a006b4d0 msgr2=0x7f58a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58a006b4d0 0x7f58a006d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f58b0005880 tx=0x7f58b0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc1586e0 msgr2=0x7f58bc0ce020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc1586e0 0x7f58bc0ce020 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009d90 tx=0x7f58b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 shutdown_connections 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58a006b4d0 0x7f58a006d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58bc1586e0 0x7f58bc0ce020 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58bc0cfa90 0x7f58bc1ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 --2- 172.21.15.77:0/2720758118 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58bc0b33a0 0x7f58bc15a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 >> 172.21.15.77:0/2720758118 conn(0x7f58bc06b190 msgr2=0x7f58bc153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 shutdown_connections 2024-03-06T07:25:56.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.215+0000 7f58c31f7640 1 -- 172.21.15.77:0/2720758118 wait complete. 2024-03-06T07:25:56.228 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.329+0000 7f2c48c88640 1 Processor -- start 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.329+0000 7f2c48c88640 1 -- start start 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.330+0000 7f2c48c88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c441586e0 0x7f2c440aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.330+0000 7f2c48c88640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c44158ac0 con 0x7f2c440d01a0 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.330+0000 7f2c48c88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4415b2a0 con 0x7f2c440aef80 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.330+0000 7f2c48c88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4415b540 con 0x7f2c441586e0 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.330+0000 7f2c4259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c441586e0 0x7f2c440aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.330+0000 7f2c42d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2c440d01a0 0x7f2c440b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58152/0 (socket says 172.21.15.77:58152) 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.330+0000 7f2c42d9d640 1 -- 172.21.15.77:0/367285838 learned_addr learned my addr 172.21.15.77:0/367285838 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.331+0000 7f2c4259c640 1 -- 172.21.15.77:0/367285838 >> v1:172.21.15.158:6789/0 conn(0x7f2c440aef80 legacy=0x7f2c440cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.331+0000 7f2c4259c640 1 -- 172.21.15.77:0/367285838 >> v1:172.21.15.77:6789/0 conn(0x7f2c440d01a0 legacy=0x7f2c440b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.331+0000 7f2c4159a640 1 -- 172.21.15.77:0/367285838 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971115256 0 0) 0x7f2c44158ac0 con 0x7f2c440d01a0 2024-03-06T07:25:56.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.331+0000 7f2c4259c640 1 -- 172.21.15.77:0/367285838 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4415b820 con 0x7f2c441586e0 2024-03-06T07:25:56.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.331+0000 7f2c4259c640 1 --2- 172.21.15.77:0/367285838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c440aea40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009d90 tx=0x7f2c3402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df3c877080e906d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c4159a640 1 -- 172.21.15.77:0/367285838 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34039070 con 0x7f2c441586e0 2024-03-06T07:25:56.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c4159a640 1 -- 172.21.15.77:0/367285838 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3402cc60 con 0x7f2c441586e0 2024-03-06T07:25:56.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c4159a640 1 -- 172.21.15.77:0/367285838 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34034660 con 0x7f2c441586e0 2024-03-06T07:25:56.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c48c88640 1 -- 172.21.15.77:0/367285838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 msgr2=0x7f2c440aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c48c88640 1 --2- 172.21.15.77:0/367285838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c440aea40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009d90 tx=0x7f2c3402beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c48c88640 1 -- 172.21.15.77:0/367285838 shutdown_connections 2024-03-06T07:25:56.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c48c88640 1 --2- 172.21.15.77:0/367285838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c440aea40 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c48c88640 1 -- 172.21.15.77:0/367285838 >> 172.21.15.77:0/367285838 conn(0x7f2c4406b190 msgr2=0x7f2c4406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c48c88640 1 -- 172.21.15.77:0/367285838 shutdown_connections 2024-03-06T07:25:56.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.332+0000 7f2c48c88640 1 -- 172.21.15.77:0/367285838 wait complete. 2024-03-06T07:25:56.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.333+0000 7f2c48c88640 1 Processor -- start 2024-03-06T07:25:56.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c48c88640 1 -- start start 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c48c88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c440aef80 0x7f2c441e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c48c88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c440d01a0 0x7f2c4406e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c48c88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c441595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c48c88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c44071350 con 0x7f2c441586e0 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c48c88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4415bde0 con 0x7f2c440aef80 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c48c88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c440714d0 con 0x7f2c440d01a0 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c42d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c441595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c42d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c441595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52320/0 (socket says 172.21.15.77:52320) 2024-03-06T07:25:56.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c42d9d640 1 -- 172.21.15.77:0/293762003 learned_addr learned my addr 172.21.15.77:0/293762003 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c41d9b640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c440d01a0 0x7f2c4406e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.334+0000 7f2c4259c640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c440aef80 0x7f2c441e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c42d9d640 1 -- 172.21.15.77:0/293762003 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c440d01a0 msgr2=0x7f2c4406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c42d9d640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c440d01a0 0x7f2c4406e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c42d9d640 1 -- 172.21.15.77:0/293762003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c440aef80 msgr2=0x7f2c441e5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c42d9d640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c440aef80 0x7f2c441e5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c42d9d640 1 -- 172.21.15.77:0/293762003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c440a96a0 con 0x7f2c441586e0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c42d9d640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c441595b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00c450 tx=0x7f2c3c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c32ffd640 1 -- 172.21.15.77:0/293762003 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c018070 con 0x7f2c441586e0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c32ffd640 1 -- 172.21.15.77:0/293762003 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c007500 con 0x7f2c441586e0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c440a9980 con 0x7f2c441586e0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c32ffd640 1 -- 172.21.15.77:0/293762003 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c004a20 con 0x7f2c441586e0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.335+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c440a9f40 con 0x7f2c441586e0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.336+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c440732b0 con 0x7f2c441586e0 2024-03-06T07:25:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.336+0000 7f2c32ffd640 1 -- 172.21.15.77:0/293762003 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c3c01b070 con 0x7f2c441586e0 2024-03-06T07:25:56.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.337+0000 7f2c32ffd640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.337+0000 7f2c32ffd640 1 -- 172.21.15.77:0/293762003 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c3c0893c0 con 0x7f2c441586e0 2024-03-06T07:25:56.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.337+0000 7f2c4259c640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.337+0000 7f2c4259c640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2c34036000 tx=0x7f2c34030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.341+0000 7f2c32ffd640 1 -- 172.21.15.77:0/293762003 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c3c057bf0 con 0x7f2c441586e0 2024-03-06T07:25:56.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.511+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c440aa110 con 0x7f2c441586e0 2024-03-06T07:25:56.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.512+0000 7f2c32ffd640 1 -- 172.21.15.77:0/293762003 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2c3c05bc90 con 0x7f2c441586e0 2024-03-06T07:25:56.513 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:25:56.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.513+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 msgr2=0x7f2c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.513+0000 7f2c48c88640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2c34036000 tx=0x7f2c34030000 comp rx=0 tx=0).stop 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.513+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 msgr2=0x7f2c441595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.513+0000 7f2c48c88640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c441595b0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00c450 tx=0x7f2c3c00c910 comp rx=0 tx=0).stop 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.513+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 shutdown_connections 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.513+0000 7f2c48c88640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.513+0000 7f2c48c88640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c441586e0 0x7f2c441595b0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.514+0000 7f2c48c88640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c440d01a0 0x7f2c4406e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.514+0000 7f2c48c88640 1 --2- 172.21.15.77:0/293762003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c440aef80 0x7f2c441e5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.514+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 >> 172.21.15.77:0/293762003 conn(0x7f2c4406b190 msgr2=0x7f2c4406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.514+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 shutdown_connections 2024-03-06T07:25:56.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.514+0000 7f2c48c88640 1 -- 172.21.15.77:0/293762003 wait complete. 2024-03-06T07:25:56.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:56.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.627+0000 7f8f7e4e1640 1 Processor -- start 2024-03-06T07:25:56.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.627+0000 7f8f7e4e1640 1 -- start start 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f7e4e1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8f781614d0 0x7f8f780d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f7e4e1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f780d1910 0x7f8f780d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f7e4e1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f780d22e0 0x7f8f780d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f7e4e1640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f780d2750 con 0x7f8f780d1910 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f7e4e1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f781618b0 con 0x7f8f781614d0 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f7e4e1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780cf400 con 0x7f8f780d22e0 2024-03-06T07:25:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f777fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f780d1910 0x7f8f780d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f777fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f780d1910 0x7f8f780d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52336/0 (socket says 172.21.15.77:52336) 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f777fe640 1 -- 172.21.15.77:0/2932119385 learned_addr learned my addr 172.21.15.77:0/2932119385 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f77fff640 1 --2- 172.21.15.77:0/2932119385 >> v2:172.21.15.158:3300/0 conn(0x7f8f781614d0 0x7f8f780d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.628+0000 7f8f7ca7e640 1 --2- 172.21.15.77:0/2932119385 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f780d22e0 0x7f8f780d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f777fe640 1 -- 172.21.15.77:0/2932119385 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f780d22e0 msgr2=0x7f8f780d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f777fe640 1 --2- 172.21.15.77:0/2932119385 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f780d22e0 0x7f8f780d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f777fe640 1 -- 172.21.15.77:0/2932119385 >> v2:172.21.15.158:3300/0 conn(0x7f8f781614d0 msgr2=0x7f8f780d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f777fe640 1 --2- 172.21.15.77:0/2932119385 >> v2:172.21.15.158:3300/0 conn(0x7f8f781614d0 0x7f8f780d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f777fe640 1 -- 172.21.15.77:0/2932119385 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f780cf6a0 con 0x7f8f780d1910 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f777fe640 1 --2- 172.21.15.77:0/2932119385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f780d1da0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f8f7808c470 tx=0x7f8f6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4b559e6699a799 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f76ffd640 1 -- 172.21.15.77:0/2932119385 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c035020 con 0x7f8f780d1910 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f76ffd640 1 -- 172.21.15.77:0/2932119385 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c031870 con 0x7f8f780d1910 2024-03-06T07:25:56.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.629+0000 7f8f76ffd640 1 -- 172.21.15.77:0/2932119385 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c031c40 con 0x7f8f780d1910 2024-03-06T07:25:56.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2932119385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 msgr2=0x7f8f780d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2932119385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f780d1da0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f8f7808c470 tx=0x7f8f6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:25:56.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2932119385 shutdown_connections 2024-03-06T07:25:56.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2932119385 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f780d22e0 0x7f8f780d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2932119385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f780d1da0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2932119385 >> v2:172.21.15.158:3300/0 conn(0x7f8f781614d0 0x7f8f780d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2932119385 >> 172.21.15.77:0/2932119385 conn(0x7f8f780d3d40 msgr2=0x7f8f780d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2932119385 shutdown_connections 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.630+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2932119385 wait complete. 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.631+0000 7f8f7e4e1640 1 Processor -- start 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.631+0000 7f8f7e4e1640 1 -- start start 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f7e4e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f78150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f7e4e1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f780d22e0 0x7f8f781f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f7e4e1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f781614d0 0x7f8f78160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f7e4e1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780b0910 con 0x7f8f780d1910 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f7e4e1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780cfc60 con 0x7f8f780d22e0 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f7e4e1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780d0be0 con 0x7f8f781614d0 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f77fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f78150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f77fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f78150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52342/0 (socket says 172.21.15.77:52342) 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f77fff640 1 -- 172.21.15.77:0/2899987078 learned_addr learned my addr 172.21.15.77:0/2899987078 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f777fe640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f780d22e0 0x7f8f781f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f7ca7e640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f781614d0 0x7f8f78160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f77fff640 1 -- 172.21.15.77:0/2899987078 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f781614d0 msgr2=0x7f8f78160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f77fff640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f781614d0 0x7f8f78160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f77fff640 1 -- 172.21.15.77:0/2899987078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f780d22e0 msgr2=0x7f8f781f0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f77fff640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f780d22e0 0x7f8f781f0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.632+0000 7f8f77fff640 1 -- 172.21.15.77:0/2899987078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f781510a0 con 0x7f8f780d1910 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.633+0000 7f8f77fff640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f78150b60 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f8f6800e870 tx=0x7f8f6800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.633+0000 7f8f74ff9640 1 -- 172.21.15.77:0/2899987078 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68017070 con 0x7f8f780d1910 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.633+0000 7f8f74ff9640 1 -- 172.21.15.77:0/2899987078 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6800c940 con 0x7f8f780d1910 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.633+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f781f3df0 con 0x7f8f780d1910 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.633+0000 7f8f74ff9640 1 -- 172.21.15.77:0/2899987078 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6800cd30 con 0x7f8f780d1910 2024-03-06T07:25:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.633+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f781f43b0 con 0x7f8f780d1910 2024-03-06T07:25:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.634+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78068ff0 con 0x7f8f780d1910 2024-03-06T07:25:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.634+0000 7f8f74ff9640 1 -- 172.21.15.77:0/2899987078 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f6801a070 con 0x7f8f780d1910 2024-03-06T07:25:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.634+0000 7f8f74ff9640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f6406b4d0 0x7f8f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.635+0000 7f8f777fe640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f6406b4d0 0x7f8f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.635+0000 7f8f74ff9640 1 -- 172.21.15.77:0/2899987078 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f68089f90 con 0x7f8f780d1910 2024-03-06T07:25:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.635+0000 7f8f777fe640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f6406b4d0 0x7f8f6406d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c036000 tx=0x7f8f6c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.639+0000 7f8f74ff9640 1 -- 172.21.15.77:0/2899987078 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f781510a0 con 0x7f8f780d1910 2024-03-06T07:25:56.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.812+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f781f1280 con 0x7f8f780d1910 2024-03-06T07:25:56.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.812+0000 7f8f74ff9640 1 -- 172.21.15.77:0/2899987078 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8f680587c0 con 0x7f8f780d1910 2024-03-06T07:25:56.814 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:25:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f6406b4d0 msgr2=0x7f8f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f6406b4d0 0x7f8f6406d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c036000 tx=0x7f8f6c03c040 comp rx=0 tx=0).stop 2024-03-06T07:25:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 msgr2=0x7f8f78150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f78150b60 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f8f6800e870 tx=0x7f8f6800ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 shutdown_connections 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8f781614d0 0x7f8f78160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8f780d22e0 0x7f8f781f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8f6406b4d0 0x7f8f6406d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 --2- 172.21.15.77:0/2899987078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f780d1910 0x7f8f78150b60 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 >> 172.21.15.77:0/2899987078 conn(0x7f8f780d3d40 msgr2=0x7f8f780d4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 shutdown_connections 2024-03-06T07:25:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.814+0000 7f8f7e4e1640 1 -- 172.21.15.77:0/2899987078 wait complete. 2024-03-06T07:25:56.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.925+0000 7f668ca0e640 1 Processor -- start 2024-03-06T07:25:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.925+0000 7f668ca0e640 1 -- start start 2024-03-06T07:25:56.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668ca0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66881424f0 0x7f66880cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668ca0e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66880cc980 con 0x7f668806da40 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668ca0e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66880ccc60 con 0x7f66880cd7a0 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668ca0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66880cd220 con 0x7f66881424f0 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f6686d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f668806da40 0x7f6688071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58158/0 (socket says 172.21.15.77:58158) 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f6686d9d640 1 -- 172.21.15.77:0/1539018246 learned_addr learned my addr 172.21.15.77:0/1539018246 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668659c640 1 --2- 172.21.15.77:0/1539018246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66881424f0 0x7f66880cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668559a640 1 -- 172.21.15.77:0/1539018246 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240975552 0 0) 0x7f66880cc980 con 0x7f668806da40 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668559a640 1 -- 172.21.15.77:0/1539018246 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6670003520 con 0x7f668806da40 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668559a640 1 -- 172.21.15.77:0/1539018246 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916386401 0 0) 0x7f66880ccc60 con 0x7f66880cd7a0 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668659c640 1 -- 172.21.15.77:0/1539018246 >> v1:172.21.15.158:6789/0 conn(0x7f66880cd7a0 legacy=0x7f668806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668659c640 1 -- 172.21.15.77:0/1539018246 >> v1:172.21.15.77:6789/0 conn(0x7f668806da40 legacy=0x7f6688071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.926+0000 7f668659c640 1 -- 172.21.15.77:0/1539018246 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66880ccf40 con 0x7f66881424f0 2024-03-06T07:25:56.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.927+0000 7f668659c640 1 --2- 172.21.15.77:0/1539018246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66881424f0 0x7f66880cc2f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6678009dc0 tx=0x7f667802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7331417a00c019ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.927+0000 7f668559a640 1 -- 172.21.15.77:0/1539018246 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678039070 con 0x7f66881424f0 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.927+0000 7f668559a640 1 -- 172.21.15.77:0/1539018246 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667802ccd0 con 0x7f66881424f0 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.927+0000 7f668559a640 1 -- 172.21.15.77:0/1539018246 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678034660 con 0x7f66881424f0 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.928+0000 7f668ca0e640 1 -- 172.21.15.77:0/1539018246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66881424f0 msgr2=0x7f66880cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.928+0000 7f668ca0e640 1 --2- 172.21.15.77:0/1539018246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66881424f0 0x7f66880cc2f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6678009dc0 tx=0x7f667802bf20 comp rx=0 tx=0).stop 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.928+0000 7f668ca0e640 1 -- 172.21.15.77:0/1539018246 shutdown_connections 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.928+0000 7f668ca0e640 1 --2- 172.21.15.77:0/1539018246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66881424f0 0x7f66880cc2f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.928+0000 7f668ca0e640 1 -- 172.21.15.77:0/1539018246 >> 172.21.15.77:0/1539018246 conn(0x7f66880cf8a0 msgr2=0x7f66880cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.928+0000 7f668ca0e640 1 -- 172.21.15.77:0/1539018246 shutdown_connections 2024-03-06T07:25:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.928+0000 7f668ca0e640 1 -- 172.21.15.77:0/1539018246 wait complete. 2024-03-06T07:25:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.929+0000 7f668ca0e640 1 Processor -- start 2024-03-06T07:25:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.929+0000 7f668ca0e640 1 -- start start 2024-03-06T07:25:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f668ca0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f668806da40 0x7f668815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f668ca0e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66880cd7a0 0x7f66880b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f668ca0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66881424f0 0x7f668815f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f668ca0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668815d070 con 0x7f66881424f0 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f668ca0e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668815cef0 con 0x7f66880cd7a0 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f668ca0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668815d1f0 con 0x7f668806da40 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f6686d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66881424f0 0x7f668815f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f6686d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66881424f0 0x7f668815f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52356/0 (socket says 172.21.15.77:52356) 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f6686d9d640 1 -- 172.21.15.77:0/392576719 learned_addr learned my addr 172.21.15.77:0/392576719 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f668659c640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f668806da40 0x7f668815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.930+0000 7f6685d9b640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66880cd7a0 0x7f66880b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6686d9d640 1 -- 172.21.15.77:0/392576719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f668806da40 msgr2=0x7f668815b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6686d9d640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f668806da40 0x7f668815b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6686d9d640 1 -- 172.21.15.77:0/392576719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66880cd7a0 msgr2=0x7f66880b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6686d9d640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66880cd7a0 0x7f66880b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6686d9d640 1 -- 172.21.15.77:0/392576719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668815bce0 con 0x7f66881424f0 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6686d9d640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66881424f0 0x7f668815f9f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f668000c430 tx=0x7f668000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6676ffd640 1 -- 172.21.15.77:0/392576719 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680017070 con 0x7f66881424f0 2024-03-06T07:25:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6676ffd640 1 -- 172.21.15.77:0/392576719 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6680005ce0 con 0x7f66881424f0 2024-03-06T07:25:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f6676ffd640 1 -- 172.21.15.77:0/392576719 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680007310 con 0x7f66881424f0 2024-03-06T07:25:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66880af740 con 0x7f66881424f0 2024-03-06T07:25:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.931+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66880afd00 con 0x7f66881424f0 2024-03-06T07:25:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.932+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668804f330 con 0x7f66881424f0 2024-03-06T07:25:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.932+0000 7f6676ffd640 1 -- 172.21.15.77:0/392576719 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f668001a070 con 0x7f66881424f0 2024-03-06T07:25:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.933+0000 7f6676ffd640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.933+0000 7f668659c640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.933+0000 7f6676ffd640 1 -- 172.21.15.77:0/392576719 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6680089580 con 0x7f66881424f0 2024-03-06T07:25:56.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.933+0000 7f668659c640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f666806b4d0 0x7f666806d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6678036000 tx=0x7f6678030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:56.937+0000 7f6676ffd640 1 -- 172.21.15.77:0/392576719 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6680057e40 con 0x7f66881424f0 2024-03-06T07:25:57.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.146+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66880d0620 con 0x7f66881424f0 2024-03-06T07:25:57.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.146+0000 7f6676ffd640 1 -- 172.21.15.77:0/392576719 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f668005bee0 con 0x7f66881424f0 2024-03-06T07:25:57.148 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:57.148 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3408486989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3408486989},{"type":"v1","addr":"172.21.15.158:6835","nonce":3408486989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/324866984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":324866984},{"type":"v1","addr":"172.21.15.77:6836","nonce":324866984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2426038221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2426038221},{"type":"v1","addr":"172.21.15.77:6837","nonce":2426038221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:44.712557+0000","modified":"2024-03-06T07:25:47.370110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6930},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6930":{"gid":6930,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/3775835533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3775835533},{"type":"v1","addr":"172.21.15.158:6834","nonce":3775835533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6930,"qdb_cluster":[6930]},"id":6}]} 2024-03-06T07:25:57.148 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2024-03-06T07:25:57.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f666806b4d0 msgr2=0x7f666806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f666806b4d0 0x7f666806d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6678036000 tx=0x7f6678030000 comp rx=0 tx=0).stop 2024-03-06T07:25:57.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66881424f0 msgr2=0x7f668815f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66881424f0 0x7f668815f9f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f668000c430 tx=0x7f668000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 shutdown_connections 2024-03-06T07:25:57.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66881424f0 0x7f668815f9f0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66880cd7a0 0x7f66880b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 --2- 172.21.15.77:0/392576719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f668806da40 0x7f668815b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 >> 172.21.15.77:0/392576719 conn(0x7f66880cf8a0 msgr2=0x7f66880cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 shutdown_connections 2024-03-06T07:25:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.148+0000 7f668ca0e640 1 -- 172.21.15.77:0/392576719 wait complete. 2024-03-06T07:25:57.161 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:57.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.260+0000 7fec9670c640 1 Processor -- start 2024-03-06T07:25:57.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.260+0000 7fec9670c640 1 -- start start 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.260+0000 7fec9670c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fec900d4760 0x7fec900b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.260+0000 7fec9670c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec900d25d0 con 0x7fec900b4170 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.260+0000 7fec9670c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec900d2870 con 0x7fec900d0a80 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.260+0000 7fec9670c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec900d4bf0 con 0x7fec900d4760 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fec900b4170 0x7fec900d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58162/0 (socket says 172.21.15.77:58162) 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8ffff640 1 -- 172.21.15.77:0/257110091 learned_addr learned my addr 172.21.15.77:0/257110091 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8f7fe640 1 --2- 172.21.15.77:0/257110091 >> v2:172.21.15.158:3300/0 conn(0x7fec900d4760 0x7fec900b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8effd640 1 -- 172.21.15.77:0/257110091 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507625986 0 0) 0x7fec900d25d0 con 0x7fec900b4170 2024-03-06T07:25:57.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8effd640 1 -- 172.21.15.77:0/257110091 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec78003520 con 0x7fec900b4170 2024-03-06T07:25:57.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8f7fe640 1 -- 172.21.15.77:0/257110091 >> v1:172.21.15.158:6789/0 conn(0x7fec900d0a80 legacy=0x7fec900cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8effd640 1 -- 172.21.15.77:0/257110091 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822029585 0 0) 0x7fec900d2870 con 0x7fec900d0a80 2024-03-06T07:25:57.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8f7fe640 1 -- 172.21.15.77:0/257110091 >> v1:172.21.15.77:6789/0 conn(0x7fec900b4170 legacy=0x7fec900d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.261+0000 7fec8f7fe640 1 -- 172.21.15.77:0/257110091 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec900d2b50 con 0x7fec900d4760 2024-03-06T07:25:57.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec8f7fe640 1 --2- 172.21.15.77:0/257110091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900d4760 0x7fec900b6a90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fec840097a0 tx=0x7fec8402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4222aa9b5b53ec3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec8effd640 1 -- 172.21.15.77:0/257110091 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84034070 con 0x7fec900d4760 2024-03-06T07:25:57.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec8effd640 1 -- 172.21.15.77:0/257110091 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec84030610 con 0x7fec900d4760 2024-03-06T07:25:57.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec8effd640 1 -- 172.21.15.77:0/257110091 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec840309c0 con 0x7fec900d4760 2024-03-06T07:25:57.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec9670c640 1 -- 172.21.15.77:0/257110091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900d4760 msgr2=0x7fec900b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec9670c640 1 --2- 172.21.15.77:0/257110091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900d4760 0x7fec900b6a90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fec840097a0 tx=0x7fec8402b960 comp rx=0 tx=0).stop 2024-03-06T07:25:57.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec9670c640 1 -- 172.21.15.77:0/257110091 shutdown_connections 2024-03-06T07:25:57.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec9670c640 1 --2- 172.21.15.77:0/257110091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900d4760 0x7fec900b6a90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.262+0000 7fec9670c640 1 -- 172.21.15.77:0/257110091 >> 172.21.15.77:0/257110091 conn(0x7fec900d3c40 msgr2=0x7fec900d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.263+0000 7fec9670c640 1 -- 172.21.15.77:0/257110091 shutdown_connections 2024-03-06T07:25:57.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.263+0000 7fec9670c640 1 -- 172.21.15.77:0/257110091 wait complete. 2024-03-06T07:25:57.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.264+0000 7fec9670c640 1 Processor -- start 2024-03-06T07:25:57.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.264+0000 7fec9670c640 1 -- start start 2024-03-06T07:25:57.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.264+0000 7fec9670c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900b4170 0x7fec9015a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.264+0000 7fec9670c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900d0a80 0x7fec900c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.264+0000 7fec9670c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec900c9150 0x7fec900b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.264+0000 7fec9670c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec900b4dd0 con 0x7fec900d0a80 2024-03-06T07:25:57.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.264+0000 7fec9670c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec900ce9e0 con 0x7fec900b4170 2024-03-06T07:25:57.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.264+0000 7fec9670c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec900b4f50 con 0x7fec900c9150 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900d0a80 0x7fec900c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900d0a80 0x7fec900c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52364/0 (socket says 172.21.15.77:52364) 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 -- 172.21.15.77:0/1226588549 learned_addr learned my addr 172.21.15.77:0/1226588549 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8ffff640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900b4170 0x7fec9015a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec94ca9640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec900c9150 0x7fec900b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 -- 172.21.15.77:0/1226588549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec900c9150 msgr2=0x7fec900b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec900c9150 0x7fec900b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 -- 172.21.15.77:0/1226588549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900b4170 msgr2=0x7fec9015a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900b4170 0x7fec9015a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 -- 172.21.15.77:0/1226588549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec900c7b70 con 0x7fec900d0a80 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8f7fe640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900d0a80 0x7fec900c8c10 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fec8402b9a0 tx=0x7fec840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8cff9640 1 -- 172.21.15.77:0/1226588549 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84034040 con 0x7fec900d0a80 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8cff9640 1 -- 172.21.15.77:0/1226588549 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8403aaa0 con 0x7fec900d0a80 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec900c7e50 con 0x7fec900d0a80 2024-03-06T07:25:57.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec8cff9640 1 -- 172.21.15.77:0/1226588549 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec840443f0 con 0x7fec900d0a80 2024-03-06T07:25:57.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.265+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec900c8410 con 0x7fec900d0a80 2024-03-06T07:25:57.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.266+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9004f330 con 0x7fec900d0a80 2024-03-06T07:25:57.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.267+0000 7fec8cff9640 1 -- 172.21.15.77:0/1226588549 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec8404b060 con 0x7fec900d0a80 2024-03-06T07:25:57.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.267+0000 7fec8cff9640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7c06b4d0 0x7fec7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.267+0000 7fec8cff9640 1 -- 172.21.15.77:0/1226588549 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec840abe60 con 0x7fec900d0a80 2024-03-06T07:25:57.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.267+0000 7fec8ffff640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7c06b4d0 0x7fec7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.267+0000 7fec8ffff640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7c06b4d0 0x7fec7c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fec80001200 tx=0x7fec80000bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.271+0000 7fec8cff9640 1 -- 172.21.15.77:0/1226588549 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec8403f070 con 0x7fec900d0a80 2024-03-06T07:25:57.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.447+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec900b9b00 con 0x7fec900d0a80 2024-03-06T07:25:57.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.449+0000 7fec8cff9640 1 -- 172.21.15.77:0/1226588549 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16648 (secure 0 0 0) 0x7fec8407a690 con 0x7fec900d0a80 2024-03-06T07:25:57.451 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:57.451 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":80,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:46.362962+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:42.937014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:43.684499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7c06b4d0 msgr2=0x7fec7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7c06b4d0 0x7fec7c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fec80001200 tx=0x7fec80000bc0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900d0a80 msgr2=0x7fec900c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900d0a80 0x7fec900c8c10 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fec8402b9a0 tx=0x7fec840046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 shutdown_connections 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fec7c06b4d0 0x7fec7c06d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900d0a80 0x7fec900c8c10 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fec900c9150 0x7fec900b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 --2- 172.21.15.77:0/1226588549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fec900b4170 0x7fec9015a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 >> 172.21.15.77:0/1226588549 conn(0x7fec900d3c40 msgr2=0x7fec900b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 shutdown_connections 2024-03-06T07:25:57.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.451+0000 7fec9670c640 1 -- 172.21.15.77:0/1226588549 wait complete. 2024-03-06T07:25:57.464 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:57.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:57.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.562+0000 7ff212c66640 1 Processor -- start 2024-03-06T07:25:57.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.562+0000 7ff212c66640 1 -- start start 2024-03-06T07:25:57.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff212c66640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff20c0cf100 0x7ff20c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff212c66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0cc810 0x7ff20c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff212c66640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20c0ccc70 con 0x7ff20c06e0a0 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff212c66640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c15e440 con 0x7ff20c0cf100 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff212c66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c15e700 con 0x7ff20c0cc810 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff20bfff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0cc810 0x7ff20c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff20bfff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0cc810 0x7ff20c06db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:37982/0 (socket says 172.21.15.77:37982) 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff20bfff640 1 -- 172.21.15.77:0/1963472226 learned_addr learned my addr 172.21.15.77:0/1963472226 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff210a02640 1 --2- 172.21.15.77:0/1963472226 >> v2:172.21.15.158:3300/0 conn(0x7ff20c0cf100 0x7ff20c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff20bfff640 1 -- 172.21.15.77:0/1963472226 >> v2:172.21.15.158:3300/0 conn(0x7ff20c0cf100 msgr2=0x7ff20c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff20bfff640 1 --2- 172.21.15.77:0/1963472226 >> v2:172.21.15.158:3300/0 conn(0x7ff20c0cf100 0x7ff20c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff20bfff640 1 -- 172.21.15.77:0/1963472226 >> v1:172.21.15.77:6789/0 conn(0x7ff20c06e0a0 legacy=0x7ff20c161a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff20b7fe640 1 -- 172.21.15.77:0/1963472226 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684409736 0 0) 0x7ff20c0ccc70 con 0x7ff20c06e0a0 2024-03-06T07:25:57.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.563+0000 7ff20bfff640 1 -- 172.21.15.77:0/1963472226 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c15e9e0 con 0x7ff20c0cc810 2024-03-06T07:25:57.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.564+0000 7ff20bfff640 1 --2- 172.21.15.77:0/1963472226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0cc810 0x7ff20c06db60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009870 tx=0x7ff1f802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=788671a6a3071326 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.564+0000 7ff20b7fe640 1 -- 172.21.15.77:0/1963472226 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8035020 con 0x7ff20c0cc810 2024-03-06T07:25:57.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.564+0000 7ff20b7fe640 1 -- 172.21.15.77:0/1963472226 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f8031640 con 0x7ff20c0cc810 2024-03-06T07:25:57.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.564+0000 7ff20b7fe640 1 -- 172.21.15.77:0/1963472226 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f802d040 con 0x7ff20c0cc810 2024-03-06T07:25:57.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.565+0000 7ff212c66640 1 -- 172.21.15.77:0/1963472226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0cc810 msgr2=0x7ff20c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.565+0000 7ff212c66640 1 --2- 172.21.15.77:0/1963472226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0cc810 0x7ff20c06db60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009870 tx=0x7ff1f802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.565+0000 7ff212c66640 1 -- 172.21.15.77:0/1963472226 shutdown_connections 2024-03-06T07:25:57.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.565+0000 7ff212c66640 1 --2- 172.21.15.77:0/1963472226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0cc810 0x7ff20c06db60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.565+0000 7ff212c66640 1 --2- 172.21.15.77:0/1963472226 >> v2:172.21.15.158:3300/0 conn(0x7ff20c0cf100 0x7ff20c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.565+0000 7ff212c66640 1 -- 172.21.15.77:0/1963472226 >> 172.21.15.77:0/1963472226 conn(0x7ff20c0cf8a0 msgr2=0x7ff20c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.565+0000 7ff212c66640 1 -- 172.21.15.77:0/1963472226 shutdown_connections 2024-03-06T07:25:57.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.565+0000 7ff212c66640 1 -- 172.21.15.77:0/1963472226 wait complete. 2024-03-06T07:25:57.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.566+0000 7ff212c66640 1 Processor -- start 2024-03-06T07:25:57.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.566+0000 7ff212c66640 1 -- start start 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff212c66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c06e0a0 0x7ff20c1493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff212c66640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0cc810 0x7ff20c159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff212c66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0cf100 0x7ff20c15a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff212c66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0cf4e0 con 0x7ff20c0cf100 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff212c66640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c15efa0 con 0x7ff20c0cc810 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff212c66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c06e510 con 0x7ff20c06e0a0 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff211203640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0cf100 0x7ff20c15a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff211203640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0cf100 0x7ff20c15a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52366/0 (socket says 172.21.15.77:52366) 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff211203640 1 -- 172.21.15.77:0/1108118541 learned_addr learned my addr 172.21.15.77:0/1108118541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff210a02640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c06e0a0 0x7ff20c1493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff20bfff640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0cc810 0x7ff20c159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff211203640 1 -- 172.21.15.77:0/1108118541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c06e0a0 msgr2=0x7ff20c1493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff211203640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c06e0a0 0x7ff20c1493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff211203640 1 -- 172.21.15.77:0/1108118541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0cc810 msgr2=0x7ff20c159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff211203640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0cc810 0x7ff20c159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.567+0000 7ff211203640 1 -- 172.21.15.77:0/1108118541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c157670 con 0x7ff20c0cf100 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.568+0000 7ff211203640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0cf100 0x7ff20c15a4d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7ff20400c7b0 tx=0x7ff20400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.568+0000 7ff2097fa640 1 -- 172.21.15.77:0/1108118541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204003ba0 con 0x7ff20c0cf100 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.568+0000 7ff2097fa640 1 -- 172.21.15.77:0/1108118541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204003d40 con 0x7ff20c0cf100 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.568+0000 7ff2097fa640 1 -- 172.21.15.77:0/1108118541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204005270 con 0x7ff20c0cf100 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.568+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c157ba0 con 0x7ff20c0cf100 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.568+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c158160 con 0x7ff20c0cf100 2024-03-06T07:25:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.569+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c04f330 con 0x7ff20c0cf100 2024-03-06T07:25:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.569+0000 7ff2097fa640 1 -- 172.21.15.77:0/1108118541 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff204005410 con 0x7ff20c0cf100 2024-03-06T07:25:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.570+0000 7ff2097fa640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1f006b4d0 0x7ff1f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.570+0000 7ff2097fa640 1 -- 172.21.15.77:0/1108118541 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff204089f70 con 0x7ff20c0cf100 2024-03-06T07:25:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.570+0000 7ff210a02640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1f006b4d0 0x7ff1f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.570+0000 7ff210a02640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1f006b4d0 0x7ff1f006d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0709d0 tx=0x7ff200009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.574+0000 7ff2097fa640 1 -- 172.21.15.77:0/1108118541 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2040587a0 con 0x7ff20c0cf100 2024-03-06T07:25:57.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.782+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff20c0cd210 con 0x7ff20c0cf100 2024-03-06T07:25:57.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.783+0000 7ff2097fa640 1 -- 172.21.15.77:0/1108118541 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7ff20405c840 con 0x7ff20c0cf100 2024-03-06T07:25:57.785 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:57.785 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3408486989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3408486989},{"type":"v1","addr":"172.21.15.158:6835","nonce":3408486989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/324866984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":324866984},{"type":"v1","addr":"172.21.15.77:6836","nonce":324866984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2426038221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2426038221},{"type":"v1","addr":"172.21.15.77:6837","nonce":2426038221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:44.712557+0000","modified":"2024-03-06T07:25:47.370110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6930},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6930":{"gid":6930,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/3775835533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3775835533},{"type":"v1","addr":"172.21.15.158:6834","nonce":3775835533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6930,"qdb_cluster":[6930]},"id":6}]} 2024-03-06T07:25:57.785 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2024-03-06T07:25:57.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.784+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1f006b4d0 msgr2=0x7ff1f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.784+0000 7ff212c66640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1f006b4d0 0x7ff1f006d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0709d0 tx=0x7ff200009190 comp rx=0 tx=0).stop 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.784+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0cf100 msgr2=0x7ff20c15a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0cf100 0x7ff20c15a4d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7ff20400c7b0 tx=0x7ff20400cbb0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 shutdown_connections 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1f006b4d0 0x7ff1f006d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0cf100 0x7ff20c15a4d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0cc810 0x7ff20c159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 --2- 172.21.15.77:0/1108118541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c06e0a0 0x7ff20c1493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 >> 172.21.15.77:0/1108118541 conn(0x7ff20c0cf8a0 msgr2=0x7ff20c0b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 shutdown_connections 2024-03-06T07:25:57.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.785+0000 7ff212c66640 1 -- 172.21.15.77:0/1108118541 wait complete. 2024-03-06T07:25:57.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:57.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.895+0000 7f3918015640 1 Processor -- start 2024-03-06T07:25:57.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.895+0000 7f3918015640 1 -- start start 2024-03-06T07:25:57.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f3918015640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f391013dc10 0x7f39100b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f3918015640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f391015ef60 0x7f39100b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f3918015640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f391015f580 con 0x7f391015bf30 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f3918015640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39100cf3f0 con 0x7f391015ef60 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f3918015640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391015b210 con 0x7f391013dc10 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f39165b2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f391015ef60 0x7f39100b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f39155b0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f391015bf30 0x7f391015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58178/0 (socket says 172.21.15.77:58178) 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f39155b0640 1 -- 172.21.15.77:0/2398673920 learned_addr learned my addr 172.21.15.77:0/2398673920 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.896+0000 7f3915db1640 1 --2- 172.21.15.77:0/2398673920 >> v2:172.21.15.158:3300/0 conn(0x7f391013dc10 0x7f39100b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.897+0000 7f39165b2640 1 -- 172.21.15.77:0/2398673920 >> v2:172.21.15.158:3300/0 conn(0x7f391013dc10 msgr2=0x7f39100b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.897+0000 7f39165b2640 1 --2- 172.21.15.77:0/2398673920 >> v2:172.21.15.158:3300/0 conn(0x7f391013dc10 0x7f39100b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.897+0000 7f39165b2640 1 -- 172.21.15.77:0/2398673920 >> v1:172.21.15.77:6789/0 conn(0x7f391015bf30 legacy=0x7f391015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.897+0000 7f39165b2640 1 -- 172.21.15.77:0/2398673920 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391015b450 con 0x7f391015ef60 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.897+0000 7f39165b2640 1 --2- 172.21.15.77:0/2398673920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391015ef60 0x7f39100b2e90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f39100731a0 tx=0x7f390c02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4310abb36aaf8621 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.897+0000 7f3914daf640 1 -- 172.21.15.77:0/2398673920 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c03b070 con 0x7f391015ef60 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3914daf640 1 -- 172.21.15.77:0/2398673920 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390c02ee00 con 0x7f391015ef60 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3914daf640 1 -- 172.21.15.77:0/2398673920 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c036660 con 0x7f391015ef60 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3918015640 1 -- 172.21.15.77:0/2398673920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391015ef60 msgr2=0x7f39100b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3918015640 1 --2- 172.21.15.77:0/2398673920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391015ef60 0x7f39100b2e90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f39100731a0 tx=0x7f390c02e050 comp rx=0 tx=0).stop 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3918015640 1 -- 172.21.15.77:0/2398673920 shutdown_connections 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3918015640 1 --2- 172.21.15.77:0/2398673920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391015ef60 0x7f39100b2e90 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3918015640 1 --2- 172.21.15.77:0/2398673920 >> v2:172.21.15.158:3300/0 conn(0x7f391013dc10 0x7f39100b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3918015640 1 -- 172.21.15.77:0/2398673920 >> 172.21.15.77:0/2398673920 conn(0x7f39100cf8a0 msgr2=0x7f39100cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3918015640 1 -- 172.21.15.77:0/2398673920 shutdown_connections 2024-03-06T07:25:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.898+0000 7f3918015640 1 -- 172.21.15.77:0/2398673920 wait complete. 2024-03-06T07:25:57.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.899+0000 7f3918015640 1 Processor -- start 2024-03-06T07:25:57.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.899+0000 7f3918015640 1 -- start start 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3918015640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391013dc10 0x7f39101f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3918015640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f391015bf30 0x7f391006dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3918015640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f391015ef60 0x7f391006e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3918015640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39100cd7a0 con 0x7f391013dc10 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3918015640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391013dff0 con 0x7f391015ef60 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3918015640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39100cd920 con 0x7f391015bf30 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391013dc10 0x7f39101f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391013dc10 0x7f39101f4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52384/0 (socket says 172.21.15.77:52384) 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 -- 172.21.15.77:0/1778854180 learned_addr learned my addr 172.21.15.77:0/1778854180 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f39155b0640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f391015bf30 0x7f391006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f39165b2640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f391015ef60 0x7f391006e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 -- 172.21.15.77:0/1778854180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f391015bf30 msgr2=0x7f391006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f391015bf30 0x7f391006dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 -- 172.21.15.77:0/1778854180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f391015ef60 msgr2=0x7f391006e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f391015ef60 0x7f391006e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 -- 172.21.15.77:0/1778854180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391015e590 con 0x7f391013dc10 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.900+0000 7f3915db1640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391013dc10 0x7f39101f4b90 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f390800e9e0 tx=0x7f390800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.901+0000 7f39067fc640 1 -- 172.21.15.77:0/1778854180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908017070 con 0x7f391013dc10 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.901+0000 7f39067fc640 1 -- 172.21.15.77:0/1778854180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390800c9d0 con 0x7f391013dc10 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.901+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391013e310 con 0x7f391013dc10 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.901+0000 7f39067fc640 1 -- 172.21.15.77:0/1778854180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390800cdc0 con 0x7f391013dc10 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.901+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391013e8d0 con 0x7f391013dc10 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.902+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391004f330 con 0x7f391013dc10 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.902+0000 7f39067fc640 1 -- 172.21.15.77:0/1778854180 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f390801a070 con 0x7f391013dc10 2024-03-06T07:25:57.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.902+0000 7f39067fc640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f38f406b4d0 0x7f38f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.903+0000 7f39155b0640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f38f406b4d0 0x7f38f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.903+0000 7f39067fc640 1 -- 172.21.15.77:0/1778854180 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f390808a190 con 0x7f391013dc10 2024-03-06T07:25:57.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.903+0000 7f39155b0640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f38f406b4d0 0x7f38f406d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f39100d0440 tx=0x7f3900007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:57.907+0000 7f39067fc640 1 -- 172.21.15.77:0/1778854180 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39080589c0 con 0x7f391013dc10 2024-03-06T07:25:58.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.117+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f391006eb00 con 0x7f391013dc10 2024-03-06T07:25:58.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.118+0000 7f39067fc640 1 -- 172.21.15.77:0/1778854180 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f390805ca60 con 0x7f391013dc10 2024-03-06T07:25:58.120 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:58.120 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3408486989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3408486989},{"type":"v1","addr":"172.21.15.158:6835","nonce":3408486989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/324866984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":324866984},{"type":"v1","addr":"172.21.15.77:6836","nonce":324866984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2426038221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2426038221},{"type":"v1","addr":"172.21.15.77:6837","nonce":2426038221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:44.712557+0000","modified":"2024-03-06T07:25:47.370110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6930},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6930":{"gid":6930,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6834/3775835533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3775835533},{"type":"v1","addr":"172.21.15.158:6834","nonce":3775835533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6930,"qdb_cluster":[6930]},"id":6}]} 2024-03-06T07:25:58.120 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2024-03-06T07:25:58.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f38f406b4d0 msgr2=0x7f38f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f38f406b4d0 0x7f38f406d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f39100d0440 tx=0x7f3900007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:58.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391013dc10 msgr2=0x7f39101f4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391013dc10 0x7f39101f4b90 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f390800e9e0 tx=0x7f390800eea0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 shutdown_connections 2024-03-06T07:25:58.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f391015ef60 0x7f391006e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f391015bf30 0x7f391006dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f38f406b4d0 0x7f38f406d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 --2- 172.21.15.77:0/1778854180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f391013dc10 0x7f39101f4b90 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 >> 172.21.15.77:0/1778854180 conn(0x7f39100cf8a0 msgr2=0x7f39100b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 shutdown_connections 2024-03-06T07:25:58.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.120+0000 7f3918015640 1 -- 172.21.15.77:0/1778854180 wait complete. 2024-03-06T07:25:58.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:58.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.230+0000 7f41241f6640 1 Processor -- start 2024-03-06T07:25:58.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.230+0000 7f41241f6640 1 -- start start 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f41241f6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f411c1515e0 0x7f411c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f41241f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c15b430 0x7f411c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f41241f6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c15b8b0 con 0x7f411c0cd7a0 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f41241f6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411c070a70 con 0x7f411c1515e0 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f41241f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411c070d50 con 0x7f411c15b430 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4122793640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f411c0cd7a0 0x7f411c0cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58194/0 (socket says 172.21.15.77:58194) 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4122793640 1 -- 172.21.15.77:0/216909592 learned_addr learned my addr 172.21.15.77:0/216909592 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4121f92640 1 --2- 172.21.15.77:0/216909592 >> v2:172.21.15.158:3300/0 conn(0x7f411c1515e0 0x7f411c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4121791640 1 --2- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c15b430 0x7f411c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4120f90640 1 -- 172.21.15.77:0/216909592 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386703998 0 0) 0x7f411c15b8b0 con 0x7f411c0cd7a0 2024-03-06T07:25:58.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4120f90640 1 -- 172.21.15.77:0/216909592 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4108003520 con 0x7f411c0cd7a0 2024-03-06T07:25:58.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4121f92640 1 -- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c15b430 msgr2=0x7f411c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4121f92640 1 --2- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c15b430 0x7f411c071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4121f92640 1 -- 172.21.15.77:0/216909592 >> v1:172.21.15.77:6789/0 conn(0x7f411c0cd7a0 legacy=0x7f411c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4121f92640 1 -- 172.21.15.77:0/216909592 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c071030 con 0x7f411c1515e0 2024-03-06T07:25:58.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.231+0000 7f4121791640 1 --2- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c15b430 0x7f411c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:58.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.232+0000 7f4121f92640 1 --2- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c1515e0 0x7f411c0cce40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f41180097a0 tx=0x7f411802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6b93f959d19b0ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.232+0000 7f4120f90640 1 -- 172.21.15.77:0/216909592 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118035020 con 0x7f411c1515e0 2024-03-06T07:25:58.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.232+0000 7f4120f90640 1 -- 172.21.15.77:0/216909592 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118031830 con 0x7f411c1515e0 2024-03-06T07:25:58.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.232+0000 7f4120f90640 1 -- 172.21.15.77:0/216909592 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118031be0 con 0x7f411c1515e0 2024-03-06T07:25:58.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.233+0000 7f41241f6640 1 -- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c1515e0 msgr2=0x7f411c0cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.233+0000 7f41241f6640 1 --2- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c1515e0 0x7f411c0cce40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f41180097a0 tx=0x7f411802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.233+0000 7f41241f6640 1 -- 172.21.15.77:0/216909592 shutdown_connections 2024-03-06T07:25:58.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.233+0000 7f41241f6640 1 --2- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c15b430 0x7f411c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.233+0000 7f41241f6640 1 --2- 172.21.15.77:0/216909592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c1515e0 0x7f411c0cce40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.233+0000 7f41241f6640 1 -- 172.21.15.77:0/216909592 >> 172.21.15.77:0/216909592 conn(0x7f411c0cf8a0 msgr2=0x7f411c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.233+0000 7f41241f6640 1 -- 172.21.15.77:0/216909592 shutdown_connections 2024-03-06T07:25:58.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.233+0000 7f41241f6640 1 -- 172.21.15.77:0/216909592 wait complete. 2024-03-06T07:25:58.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.234+0000 7f41241f6640 1 Processor -- start 2024-03-06T07:25:58.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.234+0000 7f41241f6640 1 -- start start 2024-03-06T07:25:58.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41241f6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c0cd7a0 0x7f411c154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41241f6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c1515e0 0x7f411c1419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41241f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f411c15b430 0x7f411c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41241f6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c071ef0 con 0x7f411c15b430 2024-03-06T07:25:58.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41241f6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c071d70 con 0x7f411c0cd7a0 2024-03-06T07:25:58.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41241f6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411c0cc3a0 con 0x7f411c1515e0 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f411c15b430 0x7f411c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f411c15b430 0x7f411c15bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52394/0 (socket says 172.21.15.77:52394) 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 -- 172.21.15.77:0/3471882502 learned_addr learned my addr 172.21.15.77:0/3471882502 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4121791640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c1515e0 0x7f411c1419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4121f92640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c0cd7a0 0x7f411c154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 -- 172.21.15.77:0/3471882502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c1515e0 msgr2=0x7f411c1419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c1515e0 0x7f411c1419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 -- 172.21.15.77:0/3471882502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c0cd7a0 msgr2=0x7f411c154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c0cd7a0 0x7f411c154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 -- 172.21.15.77:0/3471882502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c15c530 con 0x7f411c15b430 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f4122793640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f411c15b430 0x7f411c15bae0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f411400cb40 tx=0x7f411400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41127fc640 1 -- 172.21.15.77:0/3471882502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4114017070 con 0x7f411c15b430 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41127fc640 1 -- 172.21.15.77:0/3471882502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411400aca0 con 0x7f411c15b430 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41127fc640 1 -- 172.21.15.77:0/3471882502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4114012620 con 0x7f411c15b430 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.235+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c15c810 con 0x7f411c15b430 2024-03-06T07:25:58.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.236+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411c15da80 con 0x7f411c15b430 2024-03-06T07:25:58.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.236+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411c068840 con 0x7f411c15b430 2024-03-06T07:25:58.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.237+0000 7f41127fc640 1 -- 172.21.15.77:0/3471882502 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f411401a070 con 0x7f411c15b430 2024-03-06T07:25:58.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.237+0000 7f41127fc640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.237+0000 7f41127fc640 1 -- 172.21.15.77:0/3471882502 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4114089fe0 con 0x7f411c15b430 2024-03-06T07:25:58.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.237+0000 7f4121f92640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.238+0000 7f4121f92640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f410006b4d0 0x7f410006d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4118036000 tx=0x7f4118008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.241+0000 7f41127fc640 1 -- 172.21.15.77:0/3471882502 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4114058810 con 0x7f411c15b430 2024-03-06T07:25:58.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.418+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f411c04f330 con 0x7f411c15b430 2024-03-06T07:25:58.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.419+0000 7f41127fc640 1 -- 172.21.15.77:0/3471882502 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f411405c8b0 con 0x7f411c15b430 2024-03-06T07:25:58.421 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:25:58.421 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":80,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:25:46.362962+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:42.937014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:43.684499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f410006b4d0 msgr2=0x7f410006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f410006b4d0 0x7f410006d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4118036000 tx=0x7f4118008030 comp rx=0 tx=0).stop 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f411c15b430 msgr2=0x7f411c15bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f411c15b430 0x7f411c15bae0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f411400cb40 tx=0x7f411400a060 comp rx=0 tx=0).stop 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 shutdown_connections 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f411c15b430 0x7f411c15bae0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f411c1515e0 0x7f411c1419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 --2- 172.21.15.77:0/3471882502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f411c0cd7a0 0x7f411c154e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 >> 172.21.15.77:0/3471882502 conn(0x7f411c0cf8a0 msgr2=0x7f411c072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 shutdown_connections 2024-03-06T07:25:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.421+0000 7f41241f6640 1 -- 172.21.15.77:0/3471882502 wait complete. 2024-03-06T07:25:58.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:58.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.533+0000 7f2af0be9640 1 Processor -- start 2024-03-06T07:25:58.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.533+0000 7f2af0be9640 1 -- start start 2024-03-06T07:25:58.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2af0be9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec070fe0 0x7f2aec1658d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2af0be9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec06d370 con 0x7f2aec1444c0 2024-03-06T07:25:58.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2af0be9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec06d650 con 0x7f2aec15b2d0 2024-03-06T07:25:58.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2af0be9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec1448a0 con 0x7f2aec070fe0 2024-03-06T07:25:58.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2aea59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2aec1444c0 0x7f2aec06ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56324/0 (socket says 172.21.15.77:56324) 2024-03-06T07:25:58.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2aea59c640 1 -- 172.21.15.77:0/1065031755 learned_addr learned my addr 172.21.15.77:0/1065031755 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:58.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2aead9d640 1 --2- 172.21.15.77:0/1065031755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec070fe0 0x7f2aec1658d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2ae9d9b640 1 -- 172.21.15.77:0/1065031755 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633225731 0 0) 0x7f2aec06d370 con 0x7f2aec1444c0 2024-03-06T07:25:58.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2ae9d9b640 1 -- 172.21.15.77:0/1065031755 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad8003520 con 0x7f2aec1444c0 2024-03-06T07:25:58.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2ae9d9b640 1 -- 172.21.15.77:0/1065031755 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618348039 0 0) 0x7f2aec06d650 con 0x7f2aec15b2d0 2024-03-06T07:25:58.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2ae9d9b640 1 -- 172.21.15.77:0/1065031755 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aec06d370 con 0x7f2aec15b2d0 2024-03-06T07:25:58.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2aead9d640 1 -- 172.21.15.77:0/1065031755 >> v1:172.21.15.158:6789/0 conn(0x7f2aec15b2d0 legacy=0x7f2aec0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2aead9d640 1 -- 172.21.15.77:0/1065031755 >> v1:172.21.15.77:6789/0 conn(0x7f2aec1444c0 legacy=0x7f2aec06ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.534+0000 7f2aead9d640 1 -- 172.21.15.77:0/1065031755 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec06d930 con 0x7f2aec070fe0 2024-03-06T07:25:58.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.535+0000 7f2aead9d640 1 --2- 172.21.15.77:0/1065031755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec070fe0 0x7f2aec1658d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009dc0 tx=0x7f2ae402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ba54d6ed468efa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.535+0000 7f2ae9d9b640 1 -- 172.21.15.77:0/1065031755 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4039070 con 0x7f2aec070fe0 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.535+0000 7f2ae9d9b640 1 -- 172.21.15.77:0/1065031755 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae402ccd0 con 0x7f2aec070fe0 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.535+0000 7f2ae9d9b640 1 -- 172.21.15.77:0/1065031755 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4034640 con 0x7f2aec070fe0 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.536+0000 7f2af0be9640 1 -- 172.21.15.77:0/1065031755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec070fe0 msgr2=0x7f2aec1658d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.536+0000 7f2af0be9640 1 --2- 172.21.15.77:0/1065031755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec070fe0 0x7f2aec1658d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009dc0 tx=0x7f2ae402bf20 comp rx=0 tx=0).stop 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.536+0000 7f2af0be9640 1 -- 172.21.15.77:0/1065031755 shutdown_connections 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.536+0000 7f2af0be9640 1 --2- 172.21.15.77:0/1065031755 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec070fe0 0x7f2aec1658d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.536+0000 7f2af0be9640 1 -- 172.21.15.77:0/1065031755 >> 172.21.15.77:0/1065031755 conn(0x7f2aec0b1c80 msgr2=0x7f2aec0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.536+0000 7f2af0be9640 1 -- 172.21.15.77:0/1065031755 shutdown_connections 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.536+0000 7f2af0be9640 1 -- 172.21.15.77:0/1065031755 wait complete. 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.537+0000 7f2af0be9640 1 Processor -- start 2024-03-06T07:25:58.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.537+0000 7f2af0be9640 1 -- start start 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2af0be9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec070fe0 0x7f2aec0b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2af0be9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aec1444c0 0x7f2aec137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2af0be9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec15b2d0 0x7f2aec169030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2af0be9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec06def0 con 0x7f2aec070fe0 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2af0be9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec06d110 con 0x7f2aec1444c0 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2af0be9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0cea70 con 0x7f2aec15b2d0 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec070fe0 0x7f2aec0b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec070fe0 0x7f2aec0b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40044/0 (socket says 172.21.15.77:40044) 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 -- 172.21.15.77:0/3878875554 learned_addr learned my addr 172.21.15.77:0/3878875554 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aead9d640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec15b2d0 0x7f2aec169030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2ae3fff640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aec1444c0 0x7f2aec137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 -- 172.21.15.77:0/3878875554 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec15b2d0 msgr2=0x7f2aec169030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec15b2d0 0x7f2aec169030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 -- 172.21.15.77:0/3878875554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aec1444c0 msgr2=0x7f2aec137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aec1444c0 0x7f2aec137740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 -- 172.21.15.77:0/3878875554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec0b3280 con 0x7f2aec070fe0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2aea59c640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec070fe0 0x7f2aec0b42e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00c500 tx=0x7f2adc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2ae2ffd640 1 -- 172.21.15.77:0/3878875554 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc017070 con 0x7f2aec070fe0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2ae2ffd640 1 -- 172.21.15.77:0/3878875554 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc005ce0 con 0x7f2aec070fe0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec0b3520 con 0x7f2aec070fe0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2ae2ffd640 1 -- 172.21.15.77:0/3878875554 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc0072f0 con 0x7f2aec070fe0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.538+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec0b3ae0 con 0x7f2aec070fe0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.539+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aec04f330 con 0x7f2aec070fe0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.540+0000 7f2ae2ffd640 1 -- 172.21.15.77:0/3878875554 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2adc01a070 con 0x7f2aec070fe0 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.540+0000 7f2ae2ffd640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2acc06b4d0 0x7f2acc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.540+0000 7f2ae2ffd640 1 -- 172.21.15.77:0/3878875554 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2adc089510 con 0x7f2aec070fe0 2024-03-06T07:25:58.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.540+0000 7f2ae3fff640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2acc06b4d0 0x7f2acc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.541+0000 7f2ae3fff640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2acc06b4d0 0x7f2acc06d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4007fd0 tx=0x7f2ad4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.544+0000 7f2ae2ffd640 1 -- 172.21.15.77:0/3878875554 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2adc057dd0 con 0x7f2aec070fe0 2024-03-06T07:25:58.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.751+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2aec15e3d0 con 0x7f2aec070fe0 2024-03-06T07:25:58.753 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:25:58.752+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:58.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.753+0000 7f2ae2ffd640 1 -- 172.21.15.77:0/3878875554 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2adc05be70 con 0x7f2aec070fe0 2024-03-06T07:25:58.755 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2acc06b4d0 msgr2=0x7f2acc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2acc06b4d0 0x7f2acc06d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4007fd0 tx=0x7f2ad4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec070fe0 msgr2=0x7f2aec0b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec070fe0 0x7f2aec0b42e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00c500 tx=0x7f2adc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 shutdown_connections 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2aec15b2d0 0x7f2aec169030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2aec1444c0 0x7f2aec137740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2acc06b4d0 0x7f2acc06d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 --2- 172.21.15.77:0/3878875554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aec070fe0 0x7f2aec0b42e0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.755+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 >> 172.21.15.77:0/3878875554 conn(0x7f2aec0b1c80 msgr2=0x7f2aec0708d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.756+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 shutdown_connections 2024-03-06T07:25:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.756+0000 7f2af0be9640 1 -- 172.21.15.77:0/3878875554 wait complete. 2024-03-06T07:25:58.760 INFO:tasks.ceph.mds.d.smithi158.stderr: -2778> 2024-03-06T07:25:47.347+0000 7f1be5756640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:58.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:58.786 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:25:58.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.870+0000 7f6ce3c83640 1 Processor -- start 2024-03-06T07:25:58.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.870+0000 7f6ce3c83640 1 -- start start 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce3c83640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6cdc0cf200 0x7f6cdc0cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce3c83640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc1600a0 con 0x7f6cdc0d06e0 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce3c83640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0d47f0 con 0x7f6cdc15fc30 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce3c83640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc0ced70 con 0x7f6cdc0cf200 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce1a1f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6cdc0d06e0 0x7f6cdc0d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56334/0 (socket says 172.21.15.77:56334) 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce1a1f640 1 -- 172.21.15.77:0/1615726036 learned_addr learned my addr 172.21.15.77:0/1615726036 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815185757 0 0) 0x7f6cdc1600a0 con 0x7f6cdc0d06e0 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc4003520 con 0x7f6cdc0d06e0 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce2220640 1 --2- 172.21.15.77:0/1615726036 >> v2:172.21.15.158:3300/0 conn(0x7f6cdc0cf200 0x7f6cdc0cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154977636 0 0) 0x7f6cc4003520 con 0x7f6cdc0d06e0 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 >> v2:172.21.15.158:3300/0 conn(0x7f6cdc0cf200 msgr2=0x7f6cdc0cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 --2- 172.21.15.77:0/1615726036 >> v2:172.21.15.158:3300/0 conn(0x7f6cdc0cf200 0x7f6cdc0cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 >> v1:172.21.15.158:6789/0 conn(0x7f6cdc15fc30 legacy=0x7f6cdc0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0d1ab0 con 0x7f6cdc0d06e0 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f6ccc0019f0 con 0x7f6cdc0d06e0 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 >> v1:172.21.15.77:6789/0 conn(0x7f6cdc0d06e0 legacy=0x7f6cdc0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cc4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cc4004800 0x7f6cc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cc40051c0 0x7f6cc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0ced70 con 0x7f6cc40039b0 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc400c060 con 0x7f6cc40051c0 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc400c260 con 0x7f6cc4004800 2024-03-06T07:25:58.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce1a1f640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cc4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.871+0000 7f6ce121e640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cc4004800 0x7f6cc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.872+0000 7f6ce2220640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cc40051c0 0x7f6cc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.872+0000 7f6ce1a1f640 1 -- 172.21.15.77:0/1615726036 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cc4004800 msgr2=0x7f6cc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.872+0000 7f6ce1a1f640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cc4004800 0x7f6cc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.872+0000 7f6ce1a1f640 1 -- 172.21.15.77:0/1615726036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cc40051c0 msgr2=0x7f6cc400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.872+0000 7f6ce1a1f640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cc40051c0 0x7f6cc400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.872+0000 7f6ce1a1f640 1 -- 172.21.15.77:0/1615726036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4003520 con 0x7f6cc40039b0 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.872+0000 7f6ce2220640 1 -- 172.21.15.77:0/1615726036 reap_dead start 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.872+0000 7f6ce1a1f640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cc4003dd0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00a850 tx=0x7f6cc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffe660b4dc3c112 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc00a4a0 con 0x7f6cc40039b0 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc034650 con 0x7f6cc40039b0 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce0a1d640 1 -- 172.21.15.77:0/1615726036 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc034a80 con 0x7f6cc40039b0 2024-03-06T07:25:58.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce3c83640 1 -- 172.21.15.77:0/1615726036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 msgr2=0x7f6cc4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce3c83640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cc4003dd0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00a850 tx=0x7f6cc4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:58.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce3c83640 1 -- 172.21.15.77:0/1615726036 shutdown_connections 2024-03-06T07:25:58.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce3c83640 1 --2- 172.21.15.77:0/1615726036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cc4003dd0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce3c83640 1 -- 172.21.15.77:0/1615726036 >> 172.21.15.77:0/1615726036 conn(0x7f6cdc0d3cd0 msgr2=0x7f6cdc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce3c83640 1 -- 172.21.15.77:0/1615726036 shutdown_connections 2024-03-06T07:25:58.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.873+0000 7f6ce3c83640 1 -- 172.21.15.77:0/1615726036 wait complete. 2024-03-06T07:25:58.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.874+0000 7f6ce3c83640 1 Processor -- start 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.874+0000 7f6ce3c83640 1 -- start start 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce3c83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cdc14aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce3c83640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cdc14b190 0x7f6cdc1f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce3c83640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cdc14b610 0x7f6cdc0b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce3c83640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0ce990 con 0x7f6cc40039b0 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce3c83640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0d4ab0 con 0x7f6cdc14b610 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce3c83640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0b44d0 con 0x7f6cdc14b190 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce1a1f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cdc14aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce1a1f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cdc14aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40062/0 (socket says 172.21.15.77:40062) 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce1a1f640 1 -- 172.21.15.77:0/334779288 learned_addr learned my addr 172.21.15.77:0/334779288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce121e640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cdc14b190 0x7f6cdc1f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce2220640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cdc14b610 0x7f6cdc0b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce1a1f640 1 -- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cdc14b190 msgr2=0x7f6cdc1f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce1a1f640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cdc14b190 0x7f6cdc1f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce1a1f640 1 -- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cdc14b610 msgr2=0x7f6cdc0b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce1a1f640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cdc14b610 0x7f6cdc0b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce1a1f640 1 -- 172.21.15.77:0/334779288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0ccb30 con 0x7f6cc40039b0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.875+0000 7f6ce2220640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cdc14b610 0x7f6cdc0b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.876+0000 7f6ce1a1f640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cdc14aa60 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00a850 tx=0x7f6ccc001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.876+0000 7f6cda7fc640 1 -- 172.21.15.77:0/334779288 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc039040 con 0x7f6cc40039b0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.876+0000 7f6cda7fc640 1 -- 172.21.15.77:0/334779288 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc043070 con 0x7f6cc40039b0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.876+0000 7f6cda7fc640 1 -- 172.21.15.77:0/334779288 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0019f0 con 0x7f6cc40039b0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.876+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc1f4cd0 con 0x7f6cc40039b0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.876+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc0ccdd0 con 0x7f6cc40039b0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.877+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc1f51f0 con 0x7f6cc40039b0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.877+0000 7f6cda7fc640 1 -- 172.21.15.77:0/334779288 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ccc034650 con 0x7f6cc40039b0 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.878+0000 7f6cda7fc640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6cc806b4d0 0x7f6cc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.878+0000 7f6cda7fc640 1 -- 172.21.15.77:0/334779288 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6ccc0ac2e0 con 0x7f6cc40039b0 2024-03-06T07:25:58.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.878+0000 7f6ce121e640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6cc806b4d0 0x7f6cc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.878+0000 7f6ce121e640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6cc806b4d0 0x7f6cc806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0000c00 tx=0x7f6cd0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:58.882+0000 7f6cda7fc640 1 -- 172.21.15.77:0/334779288 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6cdc1f51f0 con 0x7f6cc40039b0 2024-03-06T07:25:59.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.093+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6cdc0a1c80 con 0x7f6cc40039b0 2024-03-06T07:25:59.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.754+0000 7f6cda7fc640 1 -- 172.21.15.77:0/334779288 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ccc07b100 con 0x7f6cc40039b0 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6cc806b4d0 msgr2=0x7f6cc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6cc806b4d0 0x7f6cc806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0000c00 tx=0x7f6cd0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 msgr2=0x7f6cdc14aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cdc14aa60 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00a850 tx=0x7f6ccc001dc0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 shutdown_connections 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6cdc14b610 0x7f6cdc0b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6cdc14b190 0x7f6cdc1f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6cc806b4d0 0x7f6cc806d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 --2- 172.21.15.77:0/334779288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cc40039b0 0x7f6cdc14aa60 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 >> 172.21.15.77:0/334779288 conn(0x7f6cdc0d3cd0 msgr2=0x7f6cdc06c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 shutdown_connections 2024-03-06T07:25:59.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.757+0000 7f6ce3c83640 1 -- 172.21.15.77:0/334779288 wait complete. 2024-03-06T07:25:59.771 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:59.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.875+0000 7f97fecf4640 1 Processor -- start 2024-03-06T07:25:59.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.875+0000 7f97fecf4640 1 -- start start 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.875+0000 7f97fecf4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f97f815ac00 0x7f97f806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.875+0000 7f97fecf4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f97f80708b0 0x7f97f8070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.875+0000 7f97fecf4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f815afe0 con 0x7f97f815e7a0 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.875+0000 7f97fecf4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f8071250 con 0x7f97f80708b0 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.875+0000 7f97fecf4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f80cdc10 con 0x7f97f815ac00 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97f7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f97f80708b0 0x7f97f8070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97f7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f97f80708b0 0x7f97f8070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40064/0 (socket says 172.21.15.77:40064) 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97f7fff640 1 -- 172.21.15.77:0/3761858886 learned_addr learned my addr 172.21.15.77:0/3761858886 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:59.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97fca90640 1 --2- 172.21.15.77:0/3761858886 >> v2:172.21.15.158:3300/0 conn(0x7f97f815ac00 0x7f97f806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97f7fff640 1 -- 172.21.15.77:0/3761858886 >> v2:172.21.15.158:3300/0 conn(0x7f97f815ac00 msgr2=0x7f97f806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97f7fff640 1 --2- 172.21.15.77:0/3761858886 >> v2:172.21.15.158:3300/0 conn(0x7f97f815ac00 0x7f97f806c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97f7fff640 1 -- 172.21.15.77:0/3761858886 >> v1:172.21.15.158:6789/0 conn(0x7f97f815e7a0 legacy=0x7f97f815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97f7fff640 1 -- 172.21.15.77:0/3761858886 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f80cd000 con 0x7f97f80708b0 2024-03-06T07:25:59.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.876+0000 7f97fca90640 1 --2- 172.21.15.77:0/3761858886 >> v2:172.21.15.158:3300/0 conn(0x7f97f815ac00 0x7f97f806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:59.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97f7fff640 1 --2- 172.21.15.77:0/3761858886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f8070d10 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f97f80730c0 tx=0x7f97e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bddf28221434ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97f77fe640 1 -- 172.21.15.77:0/3761858886 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e402e070 con 0x7f97f80708b0 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97f77fe640 1 -- 172.21.15.77:0/3761858886 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e4037070 con 0x7f97f80708b0 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97f77fe640 1 -- 172.21.15.77:0/3761858886 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4031a00 con 0x7f97f80708b0 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97fecf4640 1 -- 172.21.15.77:0/3761858886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 msgr2=0x7f97f8070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3761858886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f8070d10 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f97f80730c0 tx=0x7f97e402b880 comp rx=0 tx=0).stop 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97fecf4640 1 -- 172.21.15.77:0/3761858886 shutdown_connections 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3761858886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f8070d10 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3761858886 >> v2:172.21.15.158:3300/0 conn(0x7f97f815ac00 0x7f97f806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97fecf4640 1 -- 172.21.15.77:0/3761858886 >> 172.21.15.77:0/3761858886 conn(0x7f97f80b1650 msgr2=0x7f97f80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97fecf4640 1 -- 172.21.15.77:0/3761858886 shutdown_connections 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.877+0000 7f97fecf4640 1 -- 172.21.15.77:0/3761858886 wait complete. 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.878+0000 7f97fecf4640 1 Processor -- start 2024-03-06T07:25:59.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.878+0000 7f97fecf4640 1 -- start start 2024-03-06T07:25:59.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fecf4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f80a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fecf4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f97f815ac00 0x7f97f8159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fecf4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97f815e7a0 0x7f97f80ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fecf4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f80cd8a0 con 0x7f97f80708b0 2024-03-06T07:25:59.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fecf4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f80cd5c0 con 0x7f97f815e7a0 2024-03-06T07:25:59.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fecf4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f806ce50 con 0x7f97f815ac00 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fca90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f80a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fca90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f80a9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40068/0 (socket says 172.21.15.77:40068) 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.879+0000 7f97fca90640 1 -- 172.21.15.77:0/3941563601 learned_addr learned my addr 172.21.15.77:0/3941563601 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fd291640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97f815e7a0 0x7f97f80ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97f7fff640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f97f815ac00 0x7f97f8159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fca90640 1 -- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f97f815ac00 msgr2=0x7f97f8159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fca90640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f97f815ac00 0x7f97f8159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fca90640 1 -- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97f815e7a0 msgr2=0x7f97f80ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fca90640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97f815e7a0 0x7f97f80ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fca90640 1 -- 172.21.15.77:0/3941563601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f80ad8d0 con 0x7f97f80708b0 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97f7fff640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f97f815ac00 0x7f97f8159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fca90640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f80a9e60 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00c7f0 tx=0x7f97ec00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97f57fa640 1 -- 172.21.15.77:0/3941563601 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec015070 con 0x7f97f80708b0 2024-03-06T07:25:59.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97f57fa640 1 -- 172.21.15.77:0/3941563601 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec007a30 con 0x7f97f80708b0 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97f57fa640 1 -- 172.21.15.77:0/3941563601 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec007e00 con 0x7f97f80708b0 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f80aa560 con 0x7f97f80708b0 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.880+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f8157b10 con 0x7f97f80708b0 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.881+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f80687d0 con 0x7f97f80708b0 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.881+0000 7f97f57fa640 1 -- 172.21.15.77:0/3941563601 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97ec018070 con 0x7f97f80708b0 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.882+0000 7f97f57fa640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f97dc06b500 0x7f97dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.882+0000 7f97f57fa640 1 -- 172.21.15.77:0/3941563601 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f97ec08a070 con 0x7f97f80708b0 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.882+0000 7f97f7fff640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f97dc06b500 0x7f97dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.882+0000 7f97f7fff640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f97dc06b500 0x7f97dc06d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f97e4036000 tx=0x7f97e403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:25:59.886+0000 7f97f57fa640 1 -- 172.21.15.77:0/3941563601 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97ec0588d0 con 0x7f97f80708b0 2024-03-06T07:26:00.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.058+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f97f804f330 con 0x7f97f80708b0 2024-03-06T07:26:00.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:00.347 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.756+0000 7f97f57fa640 1 -- 172.21.15.77:0/3941563601 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) v1 ==== 181+0+0 (secure 0 0 0) 0x7f97ec05c970 con 0x7f97f80708b0 2024-03-06T07:26:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f97dc06b500 msgr2=0x7f97dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f97dc06b500 0x7f97dc06d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f97e4036000 tx=0x7f97e403d040 comp rx=0 tx=0).stop 2024-03-06T07:26:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 msgr2=0x7f97f80a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f80a9e60 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00c7f0 tx=0x7f97ec00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 shutdown_connections 2024-03-06T07:26:00.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f97f815e7a0 0x7f97f80ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f97f815ac00 0x7f97f8159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f97dc06b500 0x7f97dc06d9c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 --2- 172.21.15.77:0/3941563601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97f80708b0 0x7f97f80a9e60 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 >> 172.21.15.77:0/3941563601 conn(0x7f97f80b1650 msgr2=0x7f97f80b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 shutdown_connections 2024-03-06T07:26:00.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.758+0000 7f97fecf4640 1 -- 172.21.15.77:0/3941563601 wait complete. 2024-03-06T07:26:00.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:00.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.876+0000 7f41c0a66640 1 Processor -- start 2024-03-06T07:26:00.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.876+0000 7f41c0a66640 1 -- start start 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41c0a66640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f41bc15a490 0x7f41bc06df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41c0a66640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41bc06e4d0 con 0x7f41bc0b3060 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41c0a66640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41bc0b4490 con 0x7f41bc0b3fd0 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41c0a66640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41bc0cf4a0 con 0x7f41bc15a490 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41bad9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f41bc0b3060 0x7f41bc15dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56342/0 (socket says 172.21.15.77:56342) 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41bad9d640 1 -- 172.21.15.77:0/2439241029 learned_addr learned my addr 172.21.15.77:0/2439241029 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41ba59c640 1 --2- 172.21.15.77:0/2439241029 >> v2:172.21.15.158:3300/0 conn(0x7f41bc15a490 0x7f41bc06df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075823202 0 0) 0x7f41b4001490 con 0x7f41bc0b3060 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.877+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a4004630 con 0x7f41bc0b3060 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496983723 0 0) 0x7f41a4004630 con 0x7f41bc0b3060 2024-03-06T07:26:00.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 >> v2:172.21.15.158:3300/0 conn(0x7f41bc15a490 msgr2=0x7f41bc06df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 --2- 172.21.15.77:0/2439241029 >> v2:172.21.15.158:3300/0 conn(0x7f41bc15a490 0x7f41bc06df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 >> v1:172.21.15.158:6789/0 conn(0x7f41bc0b3fd0 legacy=0x7f41bc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc0b4750 con 0x7f41bc0b3060 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f41b4001bc0 con 0x7f41bc0b3060 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 >> v1:172.21.15.77:6789/0 conn(0x7f41bc0b3060 legacy=0x7f41bc15dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41a4004ac0 0x7f41a4004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41a4005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41a40062d0 0x7f41a400cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0cf4a0 con 0x7f41a4005910 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a400d170 con 0x7f41a40062d0 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a400d370 con 0x7f41a4004ac0 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b9d9b640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41a4005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41ba59c640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41a4004ac0 0x7f41a4004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41bad9d640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41a40062d0 0x7f41a400cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b9d9b640 1 -- 172.21.15.77:0/2439241029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41a4004ac0 msgr2=0x7f41a4004ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b9d9b640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41a4004ac0 0x7f41a4004ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b9d9b640 1 -- 172.21.15.77:0/2439241029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41a40062d0 msgr2=0x7f41a400cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b9d9b640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41a40062d0 0x7f41a400cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41bad9d640 1 -- 172.21.15.77:0/2439241029 reap_dead start 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.878+0000 7f41b9d9b640 1 -- 172.21.15.77:0/2439241029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a4004630 con 0x7f41a4005910 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.879+0000 7f41b9d9b640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41a4005d90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f41b0001750 tx=0x7f41a4006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224f26aeebad206e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.879+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0001370 con 0x7f41a4005910 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.879+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b00052d0 con 0x7f41a4005910 2024-03-06T07:26:00.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.879+0000 7f41b959a640 1 -- 172.21.15.77:0/2439241029 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0003830 con 0x7f41a4005910 2024-03-06T07:26:00.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.880+0000 7f41c0a66640 1 -- 172.21.15.77:0/2439241029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 msgr2=0x7f41a4005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.880+0000 7f41c0a66640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41a4005d90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f41b0001750 tx=0x7f41a4006940 comp rx=0 tx=0).stop 2024-03-06T07:26:00.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.880+0000 7f41c0a66640 1 -- 172.21.15.77:0/2439241029 shutdown_connections 2024-03-06T07:26:00.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.880+0000 7f41c0a66640 1 --2- 172.21.15.77:0/2439241029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41a4005d90 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.880+0000 7f41c0a66640 1 -- 172.21.15.77:0/2439241029 >> 172.21.15.77:0/2439241029 conn(0x7f41bc0b1650 msgr2=0x7f41bc0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.880+0000 7f41c0a66640 1 -- 172.21.15.77:0/2439241029 shutdown_connections 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.880+0000 7f41c0a66640 1 -- 172.21.15.77:0/2439241029 wait complete. 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.881+0000 7f41c0a66640 1 Processor -- start 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.881+0000 7f41c0a66640 1 -- start start 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41c0a66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41bc159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41c0a66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41bc1f0ce0 0x7f41bc151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41c0a66640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41bc1f1140 0x7f41bc151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41c0a66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0cf0e0 con 0x7f41a4005910 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41c0a66640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0b3490 con 0x7f41bc1f1140 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41c0a66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0cf260 con 0x7f41bc1f0ce0 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41bc159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41bc159730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40090/0 (socket says 172.21.15.77:40090) 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 -- 172.21.15.77:0/3421987418 learned_addr learned my addr 172.21.15.77:0/3421987418 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:00.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 -- 172.21.15.77:0/3421987418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41bc1f0ce0 msgr2=0x7f41bc151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41bc1f0ce0 0x7f41bc151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 -- 172.21.15.77:0/3421987418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41bc1f1140 msgr2=0x7f41bc151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41bad9d640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41bc1f1140 0x7f41bc151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41bc1f1140 0x7f41bc151ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 -- 172.21.15.77:0/3421987418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc152860 con 0x7f41a4005910 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41ba59c640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41bc159730 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f41ac011700 tx=0x7f41ac011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41aaffd640 1 -- 172.21.15.77:0/3421987418 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00f9c0 con 0x7f41a4005910 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41aaffd640 1 -- 172.21.15.77:0/3421987418 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac00fb60 con 0x7f41a4005910 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc15a910 con 0x7f41a4005910 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41aaffd640 1 -- 172.21.15.77:0/3421987418 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac015460 con 0x7f41a4005910 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.882+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc15aed0 con 0x7f41a4005910 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.883+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc0687d0 con 0x7f41a4005910 2024-03-06T07:26:00.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.884+0000 7f41aaffd640 1 -- 172.21.15.77:0/3421987418 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41ac015600 con 0x7f41a4005910 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.884+0000 7f41aaffd640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f419c06b4d0 0x7f419c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.884+0000 7f41aaffd640 1 -- 172.21.15.77:0/3421987418 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f41ac08aa70 con 0x7f41a4005910 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.884+0000 7f41b9d9b640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f419c06b4d0 0x7f419c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.885+0000 7f41b9d9b640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f419c06b4d0 0x7f419c06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f41b0033000 tx=0x7f41b002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:00.889+0000 7f41aaffd640 1 -- 172.21.15.77:0/3421987418 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41ac019030 con 0x7f41a4005910 2024-03-06T07:26:01.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.063+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f41bc0cbbd0 con 0x7f41a4005910 2024-03-06T07:26:01.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.756+0000 7f41aaffd640 1 -- 172.21.15.77:0/3421987418 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) v1 ==== 169+0+0 (secure 0 0 0) 0x7f41ac059a50 con 0x7f41a4005910 2024-03-06T07:26:01.760 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.759+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f419c06b4d0 msgr2=0x7f419c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.759+0000 7f41c0a66640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f419c06b4d0 0x7f419c06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f41b0033000 tx=0x7f41b002f000 comp rx=0 tx=0).stop 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.759+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 msgr2=0x7f41bc159730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.759+0000 7f41c0a66640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41bc159730 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f41ac011700 tx=0x7f41ac011bc0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.760+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 shutdown_connections 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.760+0000 7f41c0a66640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41bc1f1140 0x7f41bc151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.760+0000 7f41c0a66640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f41bc1f0ce0 0x7f41bc151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.760+0000 7f41c0a66640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f419c06b4d0 0x7f419c06d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.760+0000 7f41c0a66640 1 --2- 172.21.15.77:0/3421987418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41a4005910 0x7f41bc159730 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.760+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 >> 172.21.15.77:0/3421987418 conn(0x7f41bc0b1650 msgr2=0x7f41bc0b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.760+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 shutdown_connections 2024-03-06T07:26:01.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.760+0000 7f41c0a66640 1 -- 172.21.15.77:0/3421987418 wait complete. 2024-03-06T07:26:01.775 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.875+0000 7f159c20f640 1 Processor -- start 2024-03-06T07:26:01.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.875+0000 7f159c20f640 1 -- start start 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159c20f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15940cf0d0 0x7f159406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159c20f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15940b3320 con 0x7f15940719e0 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159c20f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15940728d0 con 0x7f159415a220 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159c20f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15940b3590 con 0x7f15940cf0d0 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159a7ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15940cf0d0 0x7f159406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159a7ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15940cf0d0 0x7f159406d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40098/0 (socket says 172.21.15.77:40098) 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f15997aa640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f15940719e0 0x7f159406c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56350/0 (socket says 172.21.15.77:56350) 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159a7ac640 1 -- 172.21.15.77:0/640715416 learned_addr learned my addr 172.21.15.77:0/640715416 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159a7ac640 1 -- 172.21.15.77:0/640715416 >> v1:172.21.15.158:6789/0 conn(0x7f159415a220 legacy=0x7f15940712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159a7ac640 1 -- 172.21.15.77:0/640715416 >> v1:172.21.15.77:6789/0 conn(0x7f15940719e0 legacy=0x7f159406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f1598fa9640 1 -- 172.21.15.77:0/640715416 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050685142 0 0) 0x7f15940b3320 con 0x7f15940719e0 2024-03-06T07:26:01.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.876+0000 7f159a7ac640 1 -- 172.21.15.77:0/640715416 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940cf520 con 0x7f15940cf0d0 2024-03-06T07:26:01.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.877+0000 7f159a7ac640 1 --2- 172.21.15.77:0/640715416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f159406d100 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f158c00af60 tx=0x7f158c02cfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2829c5fcfc645ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.877+0000 7f1598fa9640 1 -- 172.21.15.77:0/640715416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c02d980 con 0x7f15940cf0d0 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.877+0000 7f1598fa9640 1 -- 172.21.15.77:0/640715416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c02db20 con 0x7f15940cf0d0 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.877+0000 7f1598fa9640 1 -- 172.21.15.77:0/640715416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c008500 con 0x7f15940cf0d0 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.878+0000 7f159c20f640 1 -- 172.21.15.77:0/640715416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 msgr2=0x7f159406d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.878+0000 7f159c20f640 1 --2- 172.21.15.77:0/640715416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f159406d100 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f158c00af60 tx=0x7f158c02cfe0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.878+0000 7f159c20f640 1 -- 172.21.15.77:0/640715416 shutdown_connections 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.878+0000 7f159c20f640 1 --2- 172.21.15.77:0/640715416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f159406d100 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.878+0000 7f159c20f640 1 -- 172.21.15.77:0/640715416 >> 172.21.15.77:0/640715416 conn(0x7f159406b190 msgr2=0x7f159406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.878+0000 7f159c20f640 1 -- 172.21.15.77:0/640715416 shutdown_connections 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.878+0000 7f159c20f640 1 -- 172.21.15.77:0/640715416 wait complete. 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.879+0000 7f159c20f640 1 Processor -- start 2024-03-06T07:26:01.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.879+0000 7f159c20f640 1 -- start start 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.879+0000 7f159c20f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15940719e0 0x7f15940aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.879+0000 7f159c20f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f1594146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.879+0000 7f159c20f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f159415a220 0x7f1594149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f159c20f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594072530 con 0x7f15940cf0d0 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f159c20f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594072030 con 0x7f15940719e0 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f159c20f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15940726b0 con 0x7f159415a220 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f1594146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f1594146c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40108/0 (socket says 172.21.15.77:40108) 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 -- 172.21.15.77:0/3406766829 learned_addr learned my addr 172.21.15.77:0/3406766829 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f1599fab640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15940719e0 0x7f15940aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 -- 172.21.15.77:0/3406766829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f159415a220 msgr2=0x7f1594149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f159415a220 0x7f1594149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 -- 172.21.15.77:0/3406766829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15940719e0 msgr2=0x7f15940aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15940719e0 0x7f15940aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 -- 172.21.15.77:0/3406766829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159414a330 con 0x7f15940cf0d0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f15997aa640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f1594146c10 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f158400e960 tx=0x7f158400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f158a7fc640 1 -- 172.21.15.77:0/3406766829 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840072e0 con 0x7f15940cf0d0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f158a7fc640 1 -- 172.21.15.77:0/3406766829 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1584016020 con 0x7f15940cf0d0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15940cc5a0 con 0x7f15940cf0d0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f158a7fc640 1 -- 172.21.15.77:0/3406766829 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158400c850 con 0x7f15940cf0d0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.880+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15940ccb60 con 0x7f15940cf0d0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.881+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15940732b0 con 0x7f15940cf0d0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.882+0000 7f158a7fc640 1 -- 172.21.15.77:0/3406766829 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f158400c9f0 con 0x7f15940cf0d0 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.882+0000 7f158a7fc640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157806b4d0 0x7f157806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.882+0000 7f158a7fc640 1 -- 172.21.15.77:0/3406766829 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f158408c290 con 0x7f15940cf0d0 2024-03-06T07:26:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.882+0000 7f1599fab640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157806b4d0 0x7f157806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.883+0000 7f1599fab640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157806b4d0 0x7f157806d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1590006fd0 tx=0x7f1590008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:01.886+0000 7f158a7fc640 1 -- 172.21.15.77:0/3406766829 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1584020030 con 0x7f15940cf0d0 2024-03-06T07:26:02.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.059+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1594068ff0 con 0x7f15940cf0d0 2024-03-06T07:26:02.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.062+0000 7f158a7fc640 1 -- 172.21.15.77:0/3406766829 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f158405b410 con 0x7f15940cf0d0 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157806b4d0 msgr2=0x7f157806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157806b4d0 0x7f157806d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1590006fd0 tx=0x7f1590008040 comp rx=0 tx=0).stop 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 msgr2=0x7f1594146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f1594146c10 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f158400e960 tx=0x7f158400ee20 comp rx=0 tx=0).stop 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 shutdown_connections 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f157806b4d0 0x7f157806d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15940cf0d0 0x7f1594146c10 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f159415a220 0x7f1594149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 --2- 172.21.15.77:0/3406766829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f15940719e0 0x7f15940aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 >> 172.21.15.77:0/3406766829 conn(0x7f159406b190 msgr2=0x7f15941592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 shutdown_connections 2024-03-06T07:26:02.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.064+0000 7f159c20f640 1 -- 172.21.15.77:0/3406766829 wait complete. 2024-03-06T07:26:02.078 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:26:02.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.176+0000 7f2cef239640 1 Processor -- start 2024-03-06T07:26:02.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.176+0000 7f2cef239640 1 -- start start 2024-03-06T07:26:02.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cef239640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cef239640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2ce80cd7a0 0x7f2ce80cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cef239640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce8071f40 con 0x7f2ce815b430 2024-03-06T07:26:02.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cef239640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce815b8b0 con 0x7f2ce80cd7a0 2024-03-06T07:26:02.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cef239640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce8070a70 con 0x7f2ce81515e0 2024-03-06T07:26:02.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cecfd5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2ced7d6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2ce80cd7a0 0x7f2ce80cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cecfd5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce80cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:59386/0 (socket says 172.21.15.77:59386) 2024-03-06T07:26:02.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cecfd5640 1 -- 172.21.15.77:0/4142130162 learned_addr learned my addr 172.21.15.77:0/4142130162 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:02.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cecfd5640 1 -- 172.21.15.77:0/4142130162 >> v2:172.21.15.158:3300/0 conn(0x7f2ce80cd7a0 msgr2=0x7f2ce80cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cecfd5640 1 --2- 172.21.15.77:0/4142130162 >> v2:172.21.15.158:3300/0 conn(0x7f2ce80cd7a0 0x7f2ce80cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cecfd5640 1 -- 172.21.15.77:0/4142130162 >> v1:172.21.15.158:6789/0 conn(0x7f2ce815b430 legacy=0x7f2ce8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.177+0000 7f2cecfd5640 1 -- 172.21.15.77:0/4142130162 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce8070d50 con 0x7f2ce81515e0 2024-03-06T07:26:02.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.178+0000 7f2cecfd5640 1 --2- 172.21.15.77:0/4142130162 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce80cce40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009870 tx=0x7f2cdc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5c6c6f8434a7b93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.178+0000 7f2ce77fe640 1 -- 172.21.15.77:0/4142130162 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc034070 con 0x7f2ce81515e0 2024-03-06T07:26:02.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.178+0000 7f2ce77fe640 1 -- 172.21.15.77:0/4142130162 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc030840 con 0x7f2ce81515e0 2024-03-06T07:26:02.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.178+0000 7f2ce77fe640 1 -- 172.21.15.77:0/4142130162 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc030c10 con 0x7f2ce81515e0 2024-03-06T07:26:02.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.179+0000 7f2cef239640 1 -- 172.21.15.77:0/4142130162 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 msgr2=0x7f2ce80cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.179+0000 7f2cef239640 1 --2- 172.21.15.77:0/4142130162 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce80cce40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009870 tx=0x7f2cdc02b980 comp rx=0 tx=0).stop 2024-03-06T07:26:02.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.179+0000 7f2cef239640 1 -- 172.21.15.77:0/4142130162 shutdown_connections 2024-03-06T07:26:02.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.179+0000 7f2cef239640 1 --2- 172.21.15.77:0/4142130162 >> v2:172.21.15.158:3300/0 conn(0x7f2ce80cd7a0 0x7f2ce80cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.179+0000 7f2cef239640 1 --2- 172.21.15.77:0/4142130162 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce80cce40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.179+0000 7f2cef239640 1 -- 172.21.15.77:0/4142130162 >> 172.21.15.77:0/4142130162 conn(0x7f2ce80cf8a0 msgr2=0x7f2ce80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.179+0000 7f2cef239640 1 -- 172.21.15.77:0/4142130162 shutdown_connections 2024-03-06T07:26:02.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.179+0000 7f2cef239640 1 -- 172.21.15.77:0/4142130162 wait complete. 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.180+0000 7f2cef239640 1 Processor -- start 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.180+0000 7f2cef239640 1 -- start start 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2cef239640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ce80cd7a0 0x7f2ce80c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2cef239640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce8072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2cef239640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ce815b430 0x7f2ce8144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2cef239640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80ce110 con 0x7f2ce815b430 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2cef239640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce8071310 con 0x7f2ce80cd7a0 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2cef239640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80b3f90 con 0x7f2ce81515e0 2024-03-06T07:26:02.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ced7d6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ce815b430 0x7f2ce8144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ced7d6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ce815b430 0x7f2ce8144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40124/0 (socket says 172.21.15.77:40124) 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ced7d6640 1 -- 172.21.15.77:0/2834928884 learned_addr learned my addr 172.21.15.77:0/2834928884 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ce7fff640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce8072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2cecfd5640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ce80cd7a0 0x7f2ce80c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ced7d6640 1 -- 172.21.15.77:0/2834928884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 msgr2=0x7f2ce8072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ced7d6640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce8072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ced7d6640 1 -- 172.21.15.77:0/2834928884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ce80cd7a0 msgr2=0x7f2ce80c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ced7d6640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ce80cd7a0 0x7f2ce80c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.181+0000 7f2ced7d6640 1 -- 172.21.15.77:0/2834928884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80a9450 con 0x7f2ce815b430 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.182+0000 7f2ced7d6640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ce815b430 0x7f2ce8144df0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000efc0 tx=0x7f2ce000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.182+0000 7f2ce57fa640 1 -- 172.21.15.77:0/2834928884 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0018020 con 0x7f2ce815b430 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.182+0000 7f2ce57fa640 1 -- 172.21.15.77:0/2834928884 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce0014480 con 0x7f2ce815b430 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.182+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce80a9730 con 0x7f2ce815b430 2024-03-06T07:26:02.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.182+0000 7f2ce57fa640 1 -- 172.21.15.77:0/2834928884 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0014870 con 0x7f2ce815b430 2024-03-06T07:26:02.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.182+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce81f7e50 con 0x7f2ce815b430 2024-03-06T07:26:02.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.183+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce8068840 con 0x7f2ce815b430 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.183+0000 7f2ce57fa640 1 -- 172.21.15.77:0/2834928884 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ce0005d10 con 0x7f2ce815b430 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.183+0000 7f2ce57fa640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.184+0000 7f2ce57fa640 1 -- 172.21.15.77:0/2834928884 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2ce008a900 con 0x7f2ce815b430 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.184+0000 7f2cecfd5640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.184+0000 7f2cecfd5640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc036000 tx=0x7f2cdc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.188+0000 7f2ce57fa640 1 -- 172.21.15.77:0/2834928884 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ce0022070 con 0x7f2ce815b430 2024-03-06T07:26:02.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.430+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f2ce804f330 con 0x7f2ce815b430 2024-03-06T07:26:02.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.758+0000 7f2ce57fa640 1 -- 172.21.15.77:0/2834928884 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v227) v1 ==== 132+0+0 (secure 0 0 0) 0x7f2ce0010040 con 0x7f2ce815b430 2024-03-06T07:26:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2ccc06b4d0 msgr2=0x7f2ccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc036000 tx=0x7f2cdc008030 comp rx=0 tx=0).stop 2024-03-06T07:26:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ce815b430 msgr2=0x7f2ce8144df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ce815b430 0x7f2ce8144df0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000efc0 tx=0x7f2ce000c490 comp rx=0 tx=0).stop 2024-03-06T07:26:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 shutdown_connections 2024-03-06T07:26:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ce815b430 0x7f2ce8144df0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ce81515e0 0x7f2ce8072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 --2- 172.21.15.77:0/2834928884 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ce80cd7a0 0x7f2ce80c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 >> 172.21.15.77:0/2834928884 conn(0x7f2ce80cf8a0 msgr2=0x7f2ce814f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 shutdown_connections 2024-03-06T07:26:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.761+0000 7f2cef239640 1 -- 172.21.15.77:0/2834928884 wait complete. 2024-03-06T07:26:02.775 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:26:02.776 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:26:02.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:02.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.874+0000 7ff4cc33e640 1 Processor -- start 2024-03-06T07:26:02.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.874+0000 7ff4cc33e640 1 -- start start 2024-03-06T07:26:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4cc33e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff4c413dc10 0x7ff4c40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4cc33e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff4c415ef60 0x7ff4c40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4cc33e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c415f580 con 0x7ff4c415bf30 2024-03-06T07:26:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4cc33e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c40cf3f0 con 0x7ff4c413dc10 2024-03-06T07:26:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4cc33e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c415b210 con 0x7ff4c415ef60 2024-03-06T07:26:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4ca0da640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff4c413dc10 0x7ff4c40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4ca0da640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff4c413dc10 0x7ff4c40b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40126/0 (socket says 172.21.15.77:40126) 2024-03-06T07:26:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4ca0da640 1 -- 172.21.15.77:0/1871505301 learned_addr learned my addr 172.21.15.77:0/1871505301 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4ca0da640 1 -- 172.21.15.77:0/1871505301 >> v2:172.21.15.158:3300/0 conn(0x7ff4c415ef60 msgr2=0x7ff4c40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4ca0da640 1 --2- 172.21.15.77:0/1871505301 >> v2:172.21.15.158:3300/0 conn(0x7ff4c415ef60 0x7ff4c40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4ca0da640 1 -- 172.21.15.77:0/1871505301 >> v1:172.21.15.158:6789/0 conn(0x7ff4c415bf30 legacy=0x7ff4c415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.875+0000 7ff4ca0da640 1 -- 172.21.15.77:0/1871505301 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c415b450 con 0x7ff4c413dc10 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.876+0000 7ff4ca0da640 1 --2- 172.21.15.77:0/1871505301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c40b2950 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4068700 tx=0x7ff4c002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00627e8e836340a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.876+0000 7ff4c90d8640 1 -- 172.21.15.77:0/1871505301 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c002c9f0 con 0x7ff4c413dc10 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.876+0000 7ff4c90d8640 1 -- 172.21.15.77:0/1871505301 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c002cb90 con 0x7ff4c413dc10 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.876+0000 7ff4c90d8640 1 -- 172.21.15.77:0/1871505301 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c00343f0 con 0x7ff4c413dc10 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.877+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1871505301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 msgr2=0x7ff4c40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.877+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1871505301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c40b2950 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4068700 tx=0x7ff4c002be50 comp rx=0 tx=0).stop 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.877+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1871505301 shutdown_connections 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.877+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1871505301 >> v2:172.21.15.158:3300/0 conn(0x7ff4c415ef60 0x7ff4c40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.877+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1871505301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c40b2950 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.877+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1871505301 >> 172.21.15.77:0/1871505301 conn(0x7ff4c40cf8a0 msgr2=0x7ff4c40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.877+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1871505301 shutdown_connections 2024-03-06T07:26:02.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.877+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1871505301 wait complete. 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.878+0000 7ff4cc33e640 1 Processor -- start 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.878+0000 7ff4cc33e640 1 -- start start 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4cc33e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c413a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4cc33e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff4c415bf30 0x7ff4c41e4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4cc33e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4c415ef60 0x7ff4c413c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4cc33e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c40708d0 con 0x7ff4c413dc10 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4cc33e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c413dff0 con 0x7ff4c415ef60 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4cc33e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4070a50 con 0x7ff4c415bf30 2024-03-06T07:26:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c413a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c413a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40132/0 (socket says 172.21.15.77:40132) 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 -- 172.21.15.77:0/1752944788 learned_addr learned my addr 172.21.15.77:0/1752944788 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 -- 172.21.15.77:0/1752944788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff4c415bf30 msgr2=0x7ff4c41e4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff4c415bf30 0x7ff4c41e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 -- 172.21.15.77:0/1752944788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4c415ef60 msgr2=0x7ff4c413c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4c415ef60 0x7ff4c413c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 -- 172.21.15.77:0/1752944788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c413e6c0 con 0x7ff4c413dc10 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4ca0da640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c413a8a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00093e0 tx=0x7ff4c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4baffd640 1 -- 172.21.15.77:0/1752944788 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0043020 con 0x7ff4c413dc10 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.879+0000 7ff4baffd640 1 -- 172.21.15.77:0/1752944788 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c003e450 con 0x7ff4c413dc10 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.880+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c41e4e90 con 0x7ff4c413dc10 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.880+0000 7ff4baffd640 1 -- 172.21.15.77:0/1752944788 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c003e840 con 0x7ff4c413dc10 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.880+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c41e5450 con 0x7ff4c413dc10 2024-03-06T07:26:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.880+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c406a090 con 0x7ff4c413dc10 2024-03-06T07:26:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.881+0000 7ff4baffd640 1 -- 172.21.15.77:0/1752944788 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4c0046020 con 0x7ff4c413dc10 2024-03-06T07:26:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.881+0000 7ff4baffd640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.881+0000 7ff4baffd640 1 -- 172.21.15.77:0/1752944788 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff4c00ac880 con 0x7ff4c413dc10 2024-03-06T07:26:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.881+0000 7ff4c98d9640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.882+0000 7ff4c98d9640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff4a806b4d0 0x7ff4a806d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4072840 tx=0x7ff4b4007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:02.885+0000 7ff4baffd640 1 -- 172.21.15.77:0/1752944788 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4c406a090 con 0x7ff4c413dc10 2024-03-06T07:26:03.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.057+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4c404f330 con 0x7ff4c413dc10 2024-03-06T07:26:03.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.058+0000 7ff4baffd640 1 -- 172.21.15.77:0/1752944788 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+13011 (secure 0 0 0) 0x7ff4c004d030 con 0x7ff4c413dc10 2024-03-06T07:26:03.061 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:03.061 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":83,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:01.757193+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6834/3775835533":"2024-03-07T07:25:58.753041+0000","172.21.15.158:6832/3775835533":"2024-03-07T07:25:58.753041+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff4a806b4d0 msgr2=0x7ff4a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff4a806b4d0 0x7ff4a806d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4072840 tx=0x7ff4b4007990 comp rx=0 tx=0).stop 2024-03-06T07:26:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 msgr2=0x7ff4c413a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c413a8a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00093e0 tx=0x7ff4c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 shutdown_connections 2024-03-06T07:26:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff4c415ef60 0x7ff4c413c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff4c415bf30 0x7ff4c41e4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 --2- 172.21.15.77:0/1752944788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4c413dc10 0x7ff4c413a8a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 >> 172.21.15.77:0/1752944788 conn(0x7ff4c40cf8a0 msgr2=0x7ff4c40b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 shutdown_connections 2024-03-06T07:26:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.060+0000 7ff4cc33e640 1 -- 172.21.15.77:0/1752944788 wait complete. 2024-03-06T07:26:03.074 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:03.074 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.174+0000 7f890baed640 1 Processor -- start 2024-03-06T07:26:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.174+0000 7f890baed640 1 -- start start 2024-03-06T07:26:03.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890baed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89041614d0 0x7f89040d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890baed640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f89040d1910 0x7f89040d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890baed640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f89040d22b0 0x7f89040d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890baed640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89040d2740 con 0x7f89040d22b0 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890baed640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89041618b0 con 0x7f89040d1910 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890baed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89040cf400 con 0x7f89041614d0 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890a08a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f89040d22b0 0x7f89040d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890a08a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f89040d22b0 0x7f89040d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40136/0 (socket says 172.21.15.77:40136) 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.175+0000 7f890a08a640 1 -- 172.21.15.77:0/1468372106 learned_addr learned my addr 172.21.15.77:0/1468372106 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f8909889640 1 --2- 172.21.15.77:0/1468372106 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89041614d0 0x7f89040d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f8909088640 1 --2- 172.21.15.77:0/1468372106 >> v2:172.21.15.158:3300/0 conn(0x7f89040d1910 0x7f89040d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f890a08a640 1 -- 172.21.15.77:0/1468372106 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89041614d0 msgr2=0x7f89040d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f890a08a640 1 --2- 172.21.15.77:0/1468372106 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89041614d0 0x7f89040d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f890a08a640 1 -- 172.21.15.77:0/1468372106 >> v2:172.21.15.158:3300/0 conn(0x7f89040d1910 msgr2=0x7f89040d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f890a08a640 1 --2- 172.21.15.77:0/1468372106 >> v2:172.21.15.158:3300/0 conn(0x7f89040d1910 0x7f89040d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f890a08a640 1 -- 172.21.15.77:0/1468372106 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89040cf6a0 con 0x7f89040d22b0 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f890a08a640 1 --2- 172.21.15.77:0/1468372106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f89040d06a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f890408c4f0 tx=0x7f88fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11358a395f35b46d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.176+0000 7f8908887640 1 -- 172.21.15.77:0/1468372106 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc035020 con 0x7f89040d22b0 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f8908887640 1 -- 172.21.15.77:0/1468372106 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc031870 con 0x7f89040d22b0 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f8908887640 1 -- 172.21.15.77:0/1468372106 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc031c40 con 0x7f89040d22b0 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 -- 172.21.15.77:0/1468372106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 msgr2=0x7f89040d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 --2- 172.21.15.77:0/1468372106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f89040d06a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f890408c4f0 tx=0x7f88fc02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 -- 172.21.15.77:0/1468372106 shutdown_connections 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 --2- 172.21.15.77:0/1468372106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f89040d06a0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 --2- 172.21.15.77:0/1468372106 >> v2:172.21.15.158:3300/0 conn(0x7f89040d1910 0x7f89040d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 --2- 172.21.15.77:0/1468372106 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89041614d0 0x7f89040d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 -- 172.21.15.77:0/1468372106 >> 172.21.15.77:0/1468372106 conn(0x7f89040d3d40 msgr2=0x7f89040d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 -- 172.21.15.77:0/1468372106 shutdown_connections 2024-03-06T07:26:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.177+0000 7f890baed640 1 -- 172.21.15.77:0/1468372106 wait complete. 2024-03-06T07:26:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.178+0000 7f890baed640 1 Processor -- start 2024-03-06T07:26:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.178+0000 7f890baed640 1 -- start start 2024-03-06T07:26:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f890baed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89040d1910 0x7f89040cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f890baed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f890415f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f890baed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89041614d0 0x7f89040cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f890baed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89040b0910 con 0x7f89040d22b0 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f890baed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89040cfc60 con 0x7f89041614d0 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f890baed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89040d0be0 con 0x7f89040d1910 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909088640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f890415f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909088640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f890415f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40142/0 (socket says 172.21.15.77:40142) 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909088640 1 -- 172.21.15.77:0/3078332551 learned_addr learned my addr 172.21.15.77:0/3078332551 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909889640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89040d1910 0x7f89040cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f890a08a640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89041614d0 0x7f89040cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909088640 1 -- 172.21.15.77:0/3078332551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89040d1910 msgr2=0x7f89040cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909088640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89040d1910 0x7f89040cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909088640 1 -- 172.21.15.77:0/3078332551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89041614d0 msgr2=0x7f89040cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909088640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89041614d0 0x7f89040cb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.179+0000 7f8909088640 1 -- 172.21.15.77:0/3078332551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89040cdbd0 con 0x7f89040d22b0 2024-03-06T07:26:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.180+0000 7f8909088640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f890415f9b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f88f400d800 tx=0x7f88f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.180+0000 7f88fa7fc640 1 -- 172.21.15.77:0/3078332551 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f400b6c0 con 0x7f89040d22b0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.180+0000 7f88fa7fc640 1 -- 172.21.15.77:0/3078332551 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f4019070 con 0x7f89040d22b0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.180+0000 7f88fa7fc640 1 -- 172.21.15.77:0/3078332551 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f400ba80 con 0x7f89040d22b0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.180+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89040cc5f0 con 0x7f89040d22b0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.180+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89040ccbb0 con 0x7f89040d22b0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.181+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89040cd2d0 con 0x7f89040d22b0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.181+0000 7f88fa7fc640 1 -- 172.21.15.77:0/3078332551 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88f400bc20 con 0x7f89040d22b0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.181+0000 7f88fa7fc640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f88e806b4d0 0x7f88e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.182+0000 7f88fa7fc640 1 -- 172.21.15.77:0/3078332551 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f88f408a320 con 0x7f89040d22b0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.182+0000 7f8909889640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f88e806b4d0 0x7f88e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.182+0000 7f8909889640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f88e806b4d0 0x7f88e806d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8900009940 tx=0x7f8900003cd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.186+0000 7f88fa7fc640 1 -- 172.21.15.77:0/3078332551 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88f4059530 con 0x7f89040d22b0 2024-03-06T07:26:03.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.360+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89040b4e60 con 0x7f89040d22b0 2024-03-06T07:26:03.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.360+0000 7f88fa7fc640 1 -- 172.21.15.77:0/3078332551 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88f4058e30 con 0x7f89040d22b0 2024-03-06T07:26:03.363 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:26:03.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f88e806b4d0 msgr2=0x7f88e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f88e806b4d0 0x7f88e806d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8900009940 tx=0x7f8900003cd0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 msgr2=0x7f890415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f890415f9b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f88f400d800 tx=0x7f88f400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 shutdown_connections 2024-03-06T07:26:03.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f88e806b4d0 0x7f88e806d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89040d22b0 0x7f890415f9b0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89041614d0 0x7f89040cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 --2- 172.21.15.77:0/3078332551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89040d1910 0x7f89040cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 >> 172.21.15.77:0/3078332551 conn(0x7f89040d3d40 msgr2=0x7f89040cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 shutdown_connections 2024-03-06T07:26:03.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.362+0000 7f890baed640 1 -- 172.21.15.77:0/3078332551 wait complete. 2024-03-06T07:26:03.376 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:26:03.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.474+0000 7fcc34c0e640 1 Processor -- start 2024-03-06T07:26:03.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.474+0000 7fcc34c0e640 1 -- start start 2024-03-06T07:26:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.474+0000 7fcc34c0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc300b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.474+0000 7fcc34c0e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc30070930 0x7fcc300b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc34c0e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc30070da0 con 0x7fcc301443d0 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc34c0e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc300b5060 con 0x7fcc30070930 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc34c0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300ce610 con 0x7fcc3015ee80 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2ed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc30070930 0x7fcc300b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2ed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc30070930 0x7fcc300b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40156/0 (socket says 172.21.15.77:40156) 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2ed9d640 1 -- 172.21.15.77:0/2861040674 learned_addr learned my addr 172.21.15.77:0/2861040674 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2dd9b640 1 --2- 172.21.15.77:0/2861040674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc300b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2ed9d640 1 -- 172.21.15.77:0/2861040674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 msgr2=0x7fcc300b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2ed9d640 1 --2- 172.21.15.77:0/2861040674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc300b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2ed9d640 1 -- 172.21.15.77:0/2861040674 >> v1:172.21.15.77:6789/0 conn(0x7fcc301443d0 legacy=0x7fcc3015e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2ed9d640 1 -- 172.21.15.77:0/2861040674 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc300b3bd0 con 0x7fcc30070930 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.475+0000 7fcc2d59a640 1 -- 172.21.15.77:0/2861040674 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188740970 0 0) 0x7fcc30070da0 con 0x7fcc301443d0 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc2ed9d640 1 --2- 172.21.15.77:0/2861040674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc30070930 0x7fcc300b4b20 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800d150 tx=0x7fcc2802e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831e3a88deb316fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc2d59a640 1 -- 172.21.15.77:0/2861040674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800b0d0 con 0x7fcc30070930 2024-03-06T07:26:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc2d59a640 1 -- 172.21.15.77:0/2861040674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2800b270 con 0x7fcc30070930 2024-03-06T07:26:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc2d59a640 1 -- 172.21.15.77:0/2861040674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28009600 con 0x7fcc30070930 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2861040674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc30070930 msgr2=0x7fcc300b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2861040674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc30070930 0x7fcc300b4b20 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800d150 tx=0x7fcc2802e160 comp rx=0 tx=0).stop 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2861040674 shutdown_connections 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2861040674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc30070930 0x7fcc300b4b20 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2861040674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc300b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2861040674 >> 172.21.15.77:0/2861040674 conn(0x7fcc3006b3d0 msgr2=0x7fcc3006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2861040674 shutdown_connections 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.476+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2861040674 wait complete. 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc34c0e640 1 Processor -- start 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc34c0e640 1 -- start start 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc34c0e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc30070930 0x7fcc3015a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc34c0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc301443d0 0x7fcc3013d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc34c0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc3013df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc34c0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc301447b0 con 0x7fcc301443d0 2024-03-06T07:26:03.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc34c0e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300ce2a0 con 0x7fcc30070930 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc34c0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300cdee0 con 0x7fcc3015ee80 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc2dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc301443d0 0x7fcc3013d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.478+0000 7fcc2dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc301443d0 0x7fcc3013d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40160/0 (socket says 172.21.15.77:40160) 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2dd9b640 1 -- 172.21.15.77:0/2199128615 learned_addr learned my addr 172.21.15.77:0/2199128615 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2ed9d640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc3013df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2e59c640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc30070930 0x7fcc3015a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2dd9b640 1 -- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 msgr2=0x7fcc3013df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2dd9b640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc3013df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2dd9b640 1 -- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc30070930 msgr2=0x7fcc3015a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2dd9b640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc30070930 0x7fcc3015a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2dd9b640 1 -- 172.21.15.77:0/2199128615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc300ae240 con 0x7fcc301443d0 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2ed9d640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc3013df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc2dd9b640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc301443d0 0x7fcc3013d9e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000d5f0 tx=0x7fcc2000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc1effd640 1 -- 172.21.15.77:0/2199128615 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc200125f0 con 0x7fcc301443d0 2024-03-06T07:26:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc1effd640 1 -- 172.21.15.77:0/2199128615 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc20019070 con 0x7fcc301443d0 2024-03-06T07:26:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc300ae4c0 con 0x7fcc301443d0 2024-03-06T07:26:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc1effd640 1 -- 172.21.15.77:0/2199128615 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc200129d0 con 0x7fcc301443d0 2024-03-06T07:26:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.479+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc300aea80 con 0x7fcc301443d0 2024-03-06T07:26:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.480+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3004f330 con 0x7fcc301443d0 2024-03-06T07:26:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.481+0000 7fcc1effd640 1 -- 172.21.15.77:0/2199128615 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc20012b70 con 0x7fcc301443d0 2024-03-06T07:26:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.481+0000 7fcc1effd640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc1006b4d0 0x7fcc1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.481+0000 7fcc1effd640 1 -- 172.21.15.77:0/2199128615 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcc2008a260 con 0x7fcc301443d0 2024-03-06T07:26:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.481+0000 7fcc2e59c640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc1006b4d0 0x7fcc1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.481+0000 7fcc2e59c640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc1006b4d0 0x7fcc1006d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcc24000f60 tx=0x7fcc24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.485+0000 7fcc1effd640 1 -- 172.21.15.77:0/2199128615 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc300ae240 con 0x7fcc301443d0 2024-03-06T07:26:03.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.720+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fcc30072650 con 0x7fcc301443d0 2024-03-06T07:26:03.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.762+0000 7fcc1effd640 1 -- 172.21.15.77:0/2199128615 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v228) v1 ==== 137+0+0 (secure 0 0 0) 0x7fcc20059470 con 0x7fcc301443d0 2024-03-06T07:26:03.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc1006b4d0 msgr2=0x7fcc1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc1006b4d0 0x7fcc1006d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcc24000f60 tx=0x7fcc24009040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc301443d0 msgr2=0x7fcc3013d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc301443d0 0x7fcc3013d9e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000d5f0 tx=0x7fcc2000dab0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 shutdown_connections 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc1006b4d0 0x7fcc1006d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc301443d0 0x7fcc3013d9e0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc3015ee80 0x7fcc3013df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.764+0000 7fcc34c0e640 1 --2- 172.21.15.77:0/2199128615 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc30070930 0x7fcc3015a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.765+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 >> 172.21.15.77:0/2199128615 conn(0x7fcc3006b3d0 msgr2=0x7fcc3006e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.765+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 shutdown_connections 2024-03-06T07:26:03.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.765+0000 7fcc34c0e640 1 -- 172.21.15.77:0/2199128615 wait complete. 2024-03-06T07:26:03.778 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:03.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.878+0000 7f7028b72640 1 Processor -- start 2024-03-06T07:26:03.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.878+0000 7f7028b72640 1 -- start start 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7028b72640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f702415bb40 0x7f70240b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7028b72640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70240cfa90 0x7f70240cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7028b72640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702415ff90 con 0x7f702415f7d0 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7028b72640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702406c910 con 0x7f702415bb40 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7028b72640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7024160270 con 0x7f70240cfa90 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f701bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f702415f7d0 0x7f702406d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56366/0 (socket says 172.21.15.77:56366) 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f701bfff640 1 -- 172.21.15.77:0/2623087903 learned_addr learned my addr 172.21.15.77:0/2623087903 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383136386 0 0) 0x7f702415ff90 con 0x7f702415f7d0 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7010003520 con 0x7f702415f7d0 2024-03-06T07:26:03.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7022d9d640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70240cfa90 0x7f70240cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f702259c640 1 --2- 172.21.15.77:0/2623087903 >> v2:172.21.15.158:3300/0 conn(0x7f702415bb40 0x7f70240b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 88755085 0 0) 0x7f7010003520 con 0x7f702415f7d0 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70240cfa90 msgr2=0x7f70240cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70240cfa90 0x7f70240cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 >> v2:172.21.15.158:3300/0 conn(0x7f702415bb40 msgr2=0x7f70240b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 --2- 172.21.15.77:0/2623087903 >> v2:172.21.15.158:3300/0 conn(0x7f702415bb40 0x7f70240b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7024160550 con 0x7f702415f7d0 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f700c001670 con 0x7f702415f7d0 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 >> v1:172.21.15.77:6789/0 conn(0x7f702415f7d0 legacy=0x7f702406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70240cfa90 0x7f70100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7010004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.879+0000 7f7021d9b640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7010004d40 0x7f70100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702406c910 con 0x7f7010003f30 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7024160270 con 0x7f7010004d40 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701000b9c0 con 0x7f70240cfa90 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f701bfff640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7010004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f702259c640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70240cfa90 0x7f70100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f7022d9d640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7010004d40 0x7f70100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f701bfff640 1 -- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70240cfa90 msgr2=0x7f70100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f701bfff640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70240cfa90 0x7f70100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f701bfff640 1 -- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7010004d40 msgr2=0x7f70100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f701bfff640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7010004d40 0x7f70100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.880+0000 7f701bfff640 1 -- 172.21.15.77:0/2623087903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010003520 con 0x7f7010003f30 2024-03-06T07:26:03.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f701bfff640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7010004800 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f700c001050 tx=0x7f7010005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6e7bf315603f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c007d70 con 0x7f7010003f30 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c030070 con 0x7f7010003f30 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7021d9b640 1 -- 172.21.15.77:0/2623087903 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c02cd70 con 0x7f7010003f30 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7028b72640 1 -- 172.21.15.77:0/2623087903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 msgr2=0x7f7010004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7028b72640 1 --2- 172.21.15.77:0/2623087903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7010004800 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f700c001050 tx=0x7f7010005830 comp rx=0 tx=0).stop 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7022d9d640 1 -- 172.21.15.77:0/2623087903 reap_dead start 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7028b72640 1 -- 172.21.15.77:0/2623087903 shutdown_connections 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7028b72640 1 -- 172.21.15.77:0/2623087903 >> 172.21.15.77:0/2623087903 conn(0x7f702406b190 msgr2=0x7f702406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7028b72640 1 -- 172.21.15.77:0/2623087903 shutdown_connections 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.881+0000 7f7028b72640 1 -- 172.21.15.77:0/2623087903 wait complete. 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f7028b72640 1 Processor -- start 2024-03-06T07:26:03.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f7028b72640 1 -- start start 2024-03-06T07:26:03.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f7028b72640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7024140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f7028b72640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7024140be0 0x7f70241f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f7028b72640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7024141060 0x7f70241f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f7028b72640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702406d790 con 0x7f7010003f30 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f7028b72640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702415fc30 con 0x7f7024141060 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f7028b72640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702415bf20 con 0x7f7024140be0 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.883+0000 7f702259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7024140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f702259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7024140240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40178/0 (socket says 172.21.15.77:40178) 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f702259c640 1 -- 172.21.15.77:0/1247684613 learned_addr learned my addr 172.21.15.77:0/1247684613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f701bfff640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7024140be0 0x7f70241f4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f7022d9d640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7024141060 0x7f70241f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f702259c640 1 -- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7024140be0 msgr2=0x7f70241f4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f702259c640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7024140be0 0x7f70241f4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f702259c640 1 -- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7024141060 msgr2=0x7f70241f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f702259c640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7024141060 0x7f70241f53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f702259c640 1 -- 172.21.15.77:0/1247684613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70241f59d0 con 0x7f7010003f30 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f701bfff640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7024140be0 0x7f70241f4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f7022d9d640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7024141060 0x7f70241f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f702259c640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7024140240 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7014011960 tx=0x7f7014011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f701affd640 1 -- 172.21.15.77:0/1247684613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701400faf0 con 0x7f7010003f30 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f701affd640 1 -- 172.21.15.77:0/1247684613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701400fc90 con 0x7f7010003f30 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7024150c00 con 0x7f7010003f30 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f701affd640 1 -- 172.21.15.77:0/1247684613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7014014440 con 0x7f7010003f30 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.884+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70241511c0 con 0x7f7010003f30 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.885+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7024068ff0 con 0x7f7010003f30 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.886+0000 7f701affd640 1 -- 172.21.15.77:0/1247684613 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7014012040 con 0x7f7010003f30 2024-03-06T07:26:03.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.886+0000 7f701affd640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f700406b4d0 0x7f700406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.886+0000 7f701affd640 1 -- 172.21.15.77:0/1247684613 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f701408a600 con 0x7f7010003f30 2024-03-06T07:26:03.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.886+0000 7f701bfff640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f700406b4d0 0x7f700406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.886+0000 7f701bfff640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f700406b4d0 0x7f700406d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f700c034000 tx=0x7f700c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:03.890+0000 7f701affd640 1 -- 172.21.15.77:0/1247684613 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7014019030 con 0x7f7010003f30 2024-03-06T07:26:04.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.062+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f702406e130 con 0x7f7010003f30 2024-03-06T07:26:04.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.065+0000 7f701affd640 1 -- 172.21.15.77:0/1247684613 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7014059780 con 0x7f7010003f30 2024-03-06T07:26:04.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f700406b4d0 msgr2=0x7f700406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f700406b4d0 0x7f700406d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f700c034000 tx=0x7f700c007000 comp rx=0 tx=0).stop 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 msgr2=0x7f7024140240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7024140240 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7014011960 tx=0x7f7014011e20 comp rx=0 tx=0).stop 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 shutdown_connections 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7024141060 0x7f70241f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7024140be0 0x7f70241f4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f700406b4d0 0x7f700406d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 --2- 172.21.15.77:0/1247684613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7010003f30 0x7f7024140240 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 >> 172.21.15.77:0/1247684613 conn(0x7f702406b190 msgr2=0x7f70240b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 shutdown_connections 2024-03-06T07:26:04.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.067+0000 7f7028b72640 1 -- 172.21.15.77:0/1247684613 wait complete. 2024-03-06T07:26:04.081 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:04.081 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:04.121 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:04.171 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:04.171 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:26:04.171 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:04.186 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:04.187 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:26:04.187 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:04.204 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:04.204 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:26:04.204 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:04.241 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:04.241 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:26:04.242 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:04.261 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:04.261 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:26:04.262 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:04.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.379+0000 7fe9f04c3640 1 Processor -- start 2024-03-06T07:26:04.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.379+0000 7fe9f04c3640 1 -- start start 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.379+0000 7fe9f04c3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe9e80d0320 0x7fe9e80b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.379+0000 7fe9f04c3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e815c270 con 0x7fe9e80b54b0 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.379+0000 7fe9f04c3640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80cd900 con 0x7fe9e80d0700 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.379+0000 7fe9f04c3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e80b04e0 con 0x7fe9e80d0320 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.380+0000 7fe9ee25f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe9e80d0320 0x7fe9e80b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.380+0000 7fe9eea60640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe9e80b54b0 0x7fe9e80cd1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56380/0 (socket says 172.21.15.77:56380) 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.380+0000 7fe9eea60640 1 -- 172.21.15.77:0/1093218716 learned_addr learned my addr 172.21.15.77:0/1093218716 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.380+0000 7fe9ee25f640 1 -- 172.21.15.77:0/1093218716 >> v1:172.21.15.158:6789/0 conn(0x7fe9e80d0700 legacy=0x7fe9e80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.380+0000 7fe9ee25f640 1 -- 172.21.15.77:0/1093218716 >> v1:172.21.15.77:6789/0 conn(0x7fe9e80b54b0 legacy=0x7fe9e80cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.380+0000 7fe9ed25d640 1 -- 172.21.15.77:0/1093218716 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803198784 0 0) 0x7fe9e815c270 con 0x7fe9e80b54b0 2024-03-06T07:26:04.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.380+0000 7fe9ee25f640 1 -- 172.21.15.77:0/1093218716 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80d0e90 con 0x7fe9e80d0320 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9ee25f640 1 --2- 172.21.15.77:0/1093218716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e80b4f70 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40097a0 tx=0x7fe9e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196f83c36aab8a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9ed25d640 1 -- 172.21.15.77:0/1093218716 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4035020 con 0x7fe9e80d0320 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9ed25d640 1 -- 172.21.15.77:0/1093218716 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e4031830 con 0x7fe9e80d0320 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9ed25d640 1 -- 172.21.15.77:0/1093218716 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4031c00 con 0x7fe9e80d0320 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9f04c3640 1 -- 172.21.15.77:0/1093218716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 msgr2=0x7fe9e80b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9f04c3640 1 --2- 172.21.15.77:0/1093218716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e80b4f70 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40097a0 tx=0x7fe9e402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9f04c3640 1 -- 172.21.15.77:0/1093218716 shutdown_connections 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9f04c3640 1 --2- 172.21.15.77:0/1093218716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e80b4f70 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9f04c3640 1 -- 172.21.15.77:0/1093218716 >> 172.21.15.77:0/1093218716 conn(0x7fe9e80724d0 msgr2=0x7fe9e80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9f04c3640 1 -- 172.21.15.77:0/1093218716 shutdown_connections 2024-03-06T07:26:04.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.381+0000 7fe9f04c3640 1 -- 172.21.15.77:0/1093218716 wait complete. 2024-03-06T07:26:04.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9f04c3640 1 Processor -- start 2024-03-06T07:26:04.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9f04c3640 1 -- start start 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9f04c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe9e80b54b0 0x7fe9e80ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9f04c3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e809a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9f04c3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9e80d0700 0x7fe9e8148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9f04c3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80b0170 con 0x7fe9e80d0320 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9f04c3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80b5dc0 con 0x7fe9e80d0700 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9f04c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80b60a0 con 0x7fe9e80b54b0 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9eda5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e809a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9eda5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e809a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40204/0 (socket says 172.21.15.77:40204) 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.383+0000 7fe9eda5e640 1 -- 172.21.15.77:0/665615443 learned_addr learned my addr 172.21.15.77:0/665615443 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9eea60640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9e80d0700 0x7fe9e8148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9ee25f640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe9e80b54b0 0x7fe9e80ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9eda5e640 1 -- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe9e80b54b0 msgr2=0x7fe9e80ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9eda5e640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe9e80b54b0 0x7fe9e80ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9eda5e640 1 -- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9e80d0700 msgr2=0x7fe9e8148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9eda5e640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9e80d0700 0x7fe9e8148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9eda5e640 1 -- 172.21.15.77:0/665615443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e81557f0 con 0x7fe9e80d0320 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9eea60640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9e80d0700 0x7fe9e8148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9ee25f640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe9e80b54b0 0x7fe9e80ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9eda5e640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e809a270 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00b3c0 tx=0x7fe9dc00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9daffd640 1 -- 172.21.15.77:0/665615443 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc011830 con 0x7fe9e80d0320 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9daffd640 1 -- 172.21.15.77:0/665615443 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc016070 con 0x7fe9e80d0320 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e8155ad0 con 0x7fe9e80d0320 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9daffd640 1 -- 172.21.15.77:0/665615443 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc011c30 con 0x7fe9e80d0320 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.384+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e8156090 con 0x7fe9e80d0320 2024-03-06T07:26:04.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.385+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e804f330 con 0x7fe9e80d0320 2024-03-06T07:26:04.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.386+0000 7fe9daffd640 1 -- 172.21.15.77:0/665615443 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9dc01b460 con 0x7fe9e80d0320 2024-03-06T07:26:04.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.386+0000 7fe9daffd640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9cc06b4d0 0x7fe9cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.386+0000 7fe9daffd640 1 -- 172.21.15.77:0/665615443 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe9dc089c50 con 0x7fe9e80d0320 2024-03-06T07:26:04.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.386+0000 7fe9ee25f640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9cc06b4d0 0x7fe9cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.386+0000 7fe9ee25f640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9cc06b4d0 0x7fe9cc06d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4036000 tx=0x7fe9e403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.391+0000 7fe9daffd640 1 -- 172.21.15.77:0/665615443 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9dc08f020 con 0x7fe9e80d0320 2024-03-06T07:26:04.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.596+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9e80caf40 con 0x7fe9e80d0320 2024-03-06T07:26:04.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.597+0000 7fe9daffd640 1 -- 172.21.15.77:0/665615443 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fe9dc058dd0 con 0x7fe9e80d0320 2024-03-06T07:26:04.600 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:04.600 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3408486989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":3408486989},{"type":"v1","addr":"172.21.15.158:6835","nonce":3408486989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6053,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/1109227022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1109227022},{"type":"v1","addr":"172.21.15.158:6834","nonce":1109227022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/324866984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":324866984},{"type":"v1","addr":"172.21.15.77:6836","nonce":324866984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2426038221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2426038221},{"type":"v1","addr":"172.21.15.77:6837","nonce":2426038221}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[]} 2024-03-06T07:26:04.600 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2024-03-06T07:26:04.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9cc06b4d0 msgr2=0x7fe9cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9cc06b4d0 0x7fe9cc06d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4036000 tx=0x7fe9e403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 msgr2=0x7fe9e809a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e809a270 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00b3c0 tx=0x7fe9dc00b880 comp rx=0 tx=0).stop 2024-03-06T07:26:04.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 shutdown_connections 2024-03-06T07:26:04.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe9cc06b4d0 0x7fe9cc06d990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9e80d0320 0x7fe9e809a270 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe9e80d0700 0x7fe9e8148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 --2- 172.21.15.77:0/665615443 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe9e80b54b0 0x7fe9e80ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 >> 172.21.15.77:0/665615443 conn(0x7fe9e80724d0 msgr2=0x7fe9e80cbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 shutdown_connections 2024-03-06T07:26:04.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.599+0000 7fe9f04c3640 1 -- 172.21.15.77:0/665615443 wait complete. 2024-03-06T07:26:04.613 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:04.613 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:04.613 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:04.613 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:04.613 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:04.614 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:04.614 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:04.614 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:04.614 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:04.614 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:04.614 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:04.614 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:04.615 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:04.615 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:04.615 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:04.615 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:04.623 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:04.623 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:04.624 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:04.624 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:04.626 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:04.626 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:04.627 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:04.627 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:04.666 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:04.666 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:04.667 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:04.667 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:04.669 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:04.669 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:04.670 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:04.670 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:04.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:04.742 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:26:04.742 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:26:04.752 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:26:04.753 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:26:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.774+0000 7fc871f72640 1 Processor -- start 2024-03-06T07:26:04.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.774+0000 7fc871f72640 1 -- start start 2024-03-06T07:26:04.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc871f72640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 0x7fc86c0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc871f72640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc86c0cf7d0 0x7fc86c0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc871f72640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0d25e0 con 0x7fc86c0d4810 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc871f72640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86c0c7c50 con 0x7fc86c0cf7d0 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc871f72640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0d2880 con 0x7fc86c15e000 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc86bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc86c0cf7d0 0x7fc86c0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc86bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc86c0cf7d0 0x7fc86c0d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40254/0 (socket says 172.21.15.77:40254) 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc86bfff640 1 -- 172.21.15.77:0/1874492334 learned_addr learned my addr 172.21.15.77:0/1874492334 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.775+0000 7fc86b7fe640 1 --2- 172.21.15.77:0/1874492334 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 0x7fc86c0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.776+0000 7fc86bfff640 1 -- 172.21.15.77:0/1874492334 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 msgr2=0x7fc86c0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.776+0000 7fc86bfff640 1 --2- 172.21.15.77:0/1874492334 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 0x7fc86c0d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.776+0000 7fc86bfff640 1 -- 172.21.15.77:0/1874492334 >> v1:172.21.15.158:6789/0 conn(0x7fc86c0d4810 legacy=0x7fc86c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.776+0000 7fc86bfff640 1 -- 172.21.15.77:0/1874492334 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0d2b40 con 0x7fc86c0cf7d0 2024-03-06T07:26:04.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.777+0000 7fc86bfff640 1 --2- 172.21.15.77:0/1874492334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0cf7d0 0x7fc86c0d0bc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fc864009e50 tx=0x7fc86402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b78a39c0cafc29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.777+0000 7fc86a7fc640 1 -- 172.21.15.77:0/1874492334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86402cb90 con 0x7fc86c0cf7d0 2024-03-06T07:26:04.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.777+0000 7fc86a7fc640 1 -- 172.21.15.77:0/1874492334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86402cd30 con 0x7fc86c0cf7d0 2024-03-06T07:26:04.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.777+0000 7fc86a7fc640 1 -- 172.21.15.77:0/1874492334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864034610 con 0x7fc86c0cf7d0 2024-03-06T07:26:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.778+0000 7fc871f72640 1 -- 172.21.15.77:0/1874492334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0cf7d0 msgr2=0x7fc86c0d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.778+0000 7fc871f72640 1 --2- 172.21.15.77:0/1874492334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0cf7d0 0x7fc86c0d0bc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fc864009e50 tx=0x7fc86402bff0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.778+0000 7fc871f72640 1 -- 172.21.15.77:0/1874492334 shutdown_connections 2024-03-06T07:26:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.778+0000 7fc871f72640 1 --2- 172.21.15.77:0/1874492334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0cf7d0 0x7fc86c0d0bc0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.778+0000 7fc871f72640 1 --2- 172.21.15.77:0/1874492334 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 0x7fc86c0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.778+0000 7fc871f72640 1 -- 172.21.15.77:0/1874492334 >> 172.21.15.77:0/1874492334 conn(0x7fc86c0d3cf0 msgr2=0x7fc86c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.778+0000 7fc871f72640 1 -- 172.21.15.77:0/1874492334 shutdown_connections 2024-03-06T07:26:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.778+0000 7fc871f72640 1 -- 172.21.15.77:0/1874492334 wait complete. 2024-03-06T07:26:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.780+0000 7fc871f72640 1 Processor -- start 2024-03-06T07:26:04.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.780+0000 7fc871f72640 1 -- start start 2024-03-06T07:26:04.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc871f72640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc86c0cf7d0 0x7fc86c1542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc871f72640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0d4810 0x7fc86c0c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc871f72640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 0x7fc86c0c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc871f72640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c15e3e0 con 0x7fc86c0d4810 2024-03-06T07:26:04.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc871f72640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0d22d0 con 0x7fc86c0cf7d0 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0d4810 0x7fc86c0c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0d4810 0x7fc86c0c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40258/0 (socket says 172.21.15.77:40258) 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86affd640 1 -- 172.21.15.77:0/1540548658 learned_addr learned my addr 172.21.15.77:0/1540548658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0c87f0 con 0x7fc86c15e000 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86b7fe640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc86c0cf7d0 0x7fc86c1542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86bfff640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 0x7fc86c0c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86affd640 1 -- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 msgr2=0x7fc86c0c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86affd640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 0x7fc86c0c9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86affd640 1 -- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc86c0cf7d0 msgr2=0x7fc86c1542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86affd640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc86c0cf7d0 0x7fc86c1542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86affd640 1 -- 172.21.15.77:0/1540548658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0a1630 con 0x7fc86c0d4810 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.781+0000 7fc86b7fe640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc86c0cf7d0 0x7fc86c1542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.782+0000 7fc86affd640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0d4810 0x7fc86c0c98f0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00c430 tx=0x7fc85c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.782+0000 7fc853fff640 1 -- 172.21.15.77:0/1540548658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0052d0 con 0x7fc86c0d4810 2024-03-06T07:26:04.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.782+0000 7fc853fff640 1 -- 172.21.15.77:0/1540548658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c005490 con 0x7fc86c0d4810 2024-03-06T07:26:04.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.782+0000 7fc853fff640 1 -- 172.21.15.77:0/1540548658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c007260 con 0x7fc86c0d4810 2024-03-06T07:26:04.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.782+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c0a1910 con 0x7fc86c0d4810 2024-03-06T07:26:04.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.782+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c0a08b0 con 0x7fc86c0d4810 2024-03-06T07:26:04.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.783+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c04f330 con 0x7fc86c0d4810 2024-03-06T07:26:04.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.785+0000 7fc853fff640 1 -- 172.21.15.77:0/1540548658 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc85c007400 con 0x7fc86c0d4810 2024-03-06T07:26:04.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.786+0000 7fc853fff640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc85806b4d0 0x7fc85806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.786+0000 7fc853fff640 1 -- 172.21.15.77:0/1540548658 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc85c0898c0 con 0x7fc86c0d4810 2024-03-06T07:26:04.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.786+0000 7fc86b7fe640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc85806b4d0 0x7fc85806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.786+0000 7fc86b7fe640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc85806b4d0 0x7fc85806d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc860009e90 tx=0x7fc860009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.790+0000 7fc853fff640 1 -- 172.21.15.77:0/1540548658 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc85c0583f0 con 0x7fc86c0d4810 2024-03-06T07:26:04.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:04.975+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc86c1617c0 con 0x7fc86c0d4810 2024-03-06T07:26:05.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.785+0000 7fc853fff640 1 -- 172.21.15.77:0/1540548658 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v88) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc85c05c490 con 0x7fc86c0d4810 2024-03-06T07:26:05.787 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:26:05.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc85806b4d0 msgr2=0x7fc85806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc85806b4d0 0x7fc85806d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc860009e90 tx=0x7fc860009190 comp rx=0 tx=0).stop 2024-03-06T07:26:05.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0d4810 msgr2=0x7fc86c0c98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0d4810 0x7fc86c0c98f0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00c430 tx=0x7fc85c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 shutdown_connections 2024-03-06T07:26:05.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc85806b4d0 0x7fc85806d990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc86c0d4810 0x7fc86c0c98f0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc86c15e000 0x7fc86c0c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 --2- 172.21.15.77:0/1540548658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc86c0cf7d0 0x7fc86c1542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 >> 172.21.15.77:0/1540548658 conn(0x7fc86c0d3cf0 msgr2=0x7fc86c0ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 shutdown_connections 2024-03-06T07:26:05.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.788+0000 7fc871f72640 1 -- 172.21.15.77:0/1540548658 wait complete. 2024-03-06T07:26:05.802 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:05.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.902+0000 7f055b53e640 1 Processor -- start 2024-03-06T07:26:05.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.902+0000 7f055b53e640 1 -- start start 2024-03-06T07:26:05.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.903+0000 7f055b53e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055415dfe0 0x7f05540cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.903+0000 7f055b53e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f05540cf830 0x7f05540d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.903+0000 7f055b53e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05540b3430 con 0x7f05540d27f0 2024-03-06T07:26:05.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.903+0000 7f055b53e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05540d2c50 con 0x7f05540cf830 2024-03-06T07:26:05.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.903+0000 7f055b53e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05540b31e0 con 0x7f055415dfe0 2024-03-06T07:26:05.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.903+0000 7f0559adb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f05540d27f0 0x7f05540d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56384/0 (socket says 172.21.15.77:56384) 2024-03-06T07:26:05.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.903+0000 7f0559adb640 1 -- 172.21.15.77:0/1796990908 learned_addr learned my addr 172.21.15.77:0/1796990908 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:05.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f05592da640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055415dfe0 0x7f05540cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267036174 0 0) 0x7f05540b3430 con 0x7f05540d27f0 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0540003520 con 0x7f05540d27f0 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f0558ad9640 1 --2- 172.21.15.77:0/1796990908 >> v2:172.21.15.158:3300/0 conn(0x7f05540cf830 0x7f05540d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3645547493 0 0) 0x7f0540003520 con 0x7f05540d27f0 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055415dfe0 msgr2=0x7f05540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055415dfe0 0x7f05540cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 >> v2:172.21.15.158:3300/0 conn(0x7f05540cf830 msgr2=0x7f05540d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 --2- 172.21.15.77:0/1796990908 >> v2:172.21.15.158:3300/0 conn(0x7f05540cf830 0x7f05540d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05540b3710 con 0x7f05540d27f0 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f054c001670 con 0x7f05540d27f0 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 >> v1:172.21.15.77:6789/0 conn(0x7f05540d27f0 legacy=0x7f05540d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400039b0 0x7f0540003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055415dfe0 0x7f0540004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0540004d40 0x7f05400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05540b31e0 con 0x7f05400039b0 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054000b9c0 con 0x7f0540004d40 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054000bb40 con 0x7f055415dfe0 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f05592da640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400039b0 0x7f0540003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f0558ad9640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055415dfe0 0x7f0540004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f0559adb640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0540004d40 0x7f05400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f05592da640 1 -- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055415dfe0 msgr2=0x7f0540004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f05592da640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f055415dfe0 0x7f0540004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f05592da640 1 -- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0540004d40 msgr2=0x7f05400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f05592da640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0540004d40 0x7f05400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f05592da640 1 -- 172.21.15.77:0/1796990908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0540003520 con 0x7f05400039b0 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.904+0000 7f0559adb640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0540004d40 0x7f05400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.905+0000 7f05592da640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400039b0 0x7f0540003dd0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0550009fc0 tx=0x7f0540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a653695cc319a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.905+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550039070 con 0x7f05400039b0 2024-03-06T07:26:05.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.905+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055002cc20 con 0x7f05400039b0 2024-03-06T07:26:05.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.905+0000 7f054bfff640 1 -- 172.21.15.77:0/1796990908 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550009c10 con 0x7f05400039b0 2024-03-06T07:26:05.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.906+0000 7f055b53e640 1 -- 172.21.15.77:0/1796990908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400039b0 msgr2=0x7f0540003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.906+0000 7f055b53e640 1 --2- 172.21.15.77:0/1796990908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05400039b0 0x7f0540003dd0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0550009fc0 tx=0x7f0540005830 comp rx=0 tx=0).stop 2024-03-06T07:26:05.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.906+0000 7f0559adb640 1 -- 172.21.15.77:0/1796990908 reap_dead start 2024-03-06T07:26:05.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.906+0000 7f055b53e640 1 -- 172.21.15.77:0/1796990908 shutdown_connections 2024-03-06T07:26:05.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.906+0000 7f055b53e640 1 -- 172.21.15.77:0/1796990908 >> 172.21.15.77:0/1796990908 conn(0x7f05540d3cd0 msgr2=0x7f05540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.906+0000 7f055b53e640 1 -- 172.21.15.77:0/1796990908 shutdown_connections 2024-03-06T07:26:05.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.906+0000 7f055b53e640 1 -- 172.21.15.77:0/1796990908 wait complete. 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.907+0000 7f055b53e640 1 Processor -- start 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.907+0000 7f055b53e640 1 -- start start 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f055b53e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f05400039b0 0x7f05540b7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f055b53e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055415c280 0x7f05540b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f055b53e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05540b8390 0x7f0554160740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f055b53e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05540d1c70 con 0x7f055415c280 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f055b53e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05540b3d60 con 0x7f05540b8390 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f055b53e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05540d1df0 con 0x7f05400039b0 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055415c280 0x7f05540b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055415c280 0x7f05540b7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40276/0 (socket says 172.21.15.77:40276) 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 -- 172.21.15.77:0/1094518025 learned_addr learned my addr 172.21.15.77:0/1094518025 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 -- 172.21.15.77:0/1094518025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f05400039b0 msgr2=0x7f05540b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f05400039b0 0x7f05540b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 -- 172.21.15.77:0/1094518025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05540b8390 msgr2=0x7f0554160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05540b8390 0x7f0554160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 -- 172.21.15.77:0/1094518025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05541614a0 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0558ad9640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055415c280 0x7f05540b7e50 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f05440115e0 tx=0x7f0544011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0549ffb640 1 -- 172.21.15.77:0/1094518025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544004220 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0549ffb640 1 -- 172.21.15.77:0/1094518025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05440043c0 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05540b6970 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.908+0000 7f0549ffb640 1 -- 172.21.15.77:0/1094518025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05440158b0 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.909+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05540b6f30 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.909+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05540b87b0 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.910+0000 7f0549ffb640 1 -- 172.21.15.77:0/1094518025 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0544015a50 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.910+0000 7f0549ffb640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.910+0000 7f0549ffb640 1 -- 172.21.15.77:0/1094518025 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f054408a340 con 0x7f055415c280 2024-03-06T07:26:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.911+0000 7f05592da640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.911+0000 7f05592da640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0550036000 tx=0x7f055002e070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:05.915+0000 7f0549ffb640 1 -- 172.21.15.77:0/1094518025 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0544019030 con 0x7f055415c280 2024-03-06T07:26:06.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.151+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f055404f330 con 0x7f055415c280 2024-03-06T07:26:06.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.151+0000 7f0549ffb640 1 -- 172.21.15.77:0/1094518025 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f05440595a0 con 0x7f055415c280 2024-03-06T07:26:06.153 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:06.153 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:26:06.153 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:26:06.153 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.153 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:26:06.154 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:26:06.156 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.158 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.159 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.160 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.161 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:26:06.162 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:26:06.163 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:26:06.164 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.165 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:06.166 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:26:06.166 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 msgr2=0x7f053806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0550036000 tx=0x7f055002e070 comp rx=0 tx=0).stop 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055415c280 msgr2=0x7f05540b7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055415c280 0x7f05540b7e50 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f05440115e0 tx=0x7f0544011aa0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 shutdown_connections 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f053806b500 0x7f053806d9c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055415c280 0x7f05540b7e50 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f05540b8390 0x7f0554160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 --2- 172.21.15.77:0/1094518025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f05400039b0 0x7f05540b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 >> 172.21.15.77:0/1094518025 conn(0x7f05540d3cd0 msgr2=0x7f05540b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 shutdown_connections 2024-03-06T07:26:06.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.153+0000 7f055b53e640 1 -- 172.21.15.77:0/1094518025 wait complete. 2024-03-06T07:26:06.168 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:06.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.266+0000 7f0ccf49a640 1 Processor -- start 2024-03-06T07:26:06.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.266+0000 7f0ccf49a640 1 -- start start 2024-03-06T07:26:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccf49a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0cc80b3060 0x7f0cc815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccf49a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0cc815bf30 0x7f0cc815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccf49a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc815f2d0 con 0x7f0cc813dc10 2024-03-06T07:26:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccf49a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc80cf3f0 con 0x7f0cc80b3060 2024-03-06T07:26:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccf49a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc815b210 con 0x7f0cc815bf30 2024-03-06T07:26:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccca35640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0cc80b3060 0x7f0cc815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccca35640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0cc80b3060 0x7f0cc815e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40286/0 (socket says 172.21.15.77:40286) 2024-03-06T07:26:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccca35640 1 -- 172.21.15.77:0/4273451099 learned_addr learned my addr 172.21.15.77:0/4273451099 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccd236640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0cc813dc10 0x7f0cc80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56390/0 (socket says 172.21.15.77:56390) 2024-03-06T07:26:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccca35640 1 -- 172.21.15.77:0/4273451099 >> v2:172.21.15.158:3300/0 conn(0x7f0cc815bf30 msgr2=0x7f0cc815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccca35640 1 --2- 172.21.15.77:0/4273451099 >> v2:172.21.15.158:3300/0 conn(0x7f0cc815bf30 0x7f0cc815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccca35640 1 -- 172.21.15.77:0/4273451099 >> v1:172.21.15.77:6789/0 conn(0x7f0cc813dc10 legacy=0x7f0cc80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0cbffff640 1 -- 172.21.15.77:0/4273451099 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071618436 0 0) 0x7f0cc815f2d0 con 0x7f0cc813dc10 2024-03-06T07:26:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.267+0000 7f0ccca35640 1 -- 172.21.15.77:0/4273451099 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc815f580 con 0x7f0cc80b3060 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.268+0000 7f0ccca35640 1 --2- 172.21.15.77:0/4273451099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80b3060 0x7f0cc815e850 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80b4e00 tx=0x7f0cb802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26271c800ae4676f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.268+0000 7f0cbffff640 1 -- 172.21.15.77:0/4273451099 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8035020 con 0x7f0cc80b3060 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.268+0000 7f0cbffff640 1 -- 172.21.15.77:0/4273451099 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb8031870 con 0x7f0cc80b3060 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.268+0000 7f0cbffff640 1 -- 172.21.15.77:0/4273451099 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8031c40 con 0x7f0cc80b3060 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.268+0000 7f0ccf49a640 1 -- 172.21.15.77:0/4273451099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80b3060 msgr2=0x7f0cc815e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.268+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/4273451099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80b3060 0x7f0cc815e850 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80b4e00 tx=0x7f0cb802b910 comp rx=0 tx=0).stop 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.269+0000 7f0ccf49a640 1 -- 172.21.15.77:0/4273451099 shutdown_connections 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.269+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/4273451099 >> v2:172.21.15.158:3300/0 conn(0x7f0cc815bf30 0x7f0cc815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.269+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/4273451099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80b3060 0x7f0cc815e850 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.269+0000 7f0ccf49a640 1 -- 172.21.15.77:0/4273451099 >> 172.21.15.77:0/4273451099 conn(0x7f0cc80cf8a0 msgr2=0x7f0cc80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.269+0000 7f0ccf49a640 1 -- 172.21.15.77:0/4273451099 shutdown_connections 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.269+0000 7f0ccf49a640 1 -- 172.21.15.77:0/4273451099 wait complete. 2024-03-06T07:26:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.270+0000 7f0ccf49a640 1 Processor -- start 2024-03-06T07:26:06.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.270+0000 7f0ccf49a640 1 -- start start 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.270+0000 7f0ccf49a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cc80b3060 0x7f0cc8154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.270+0000 7f0ccf49a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0cc813dc10 0x7f0cc8140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.270+0000 7f0ccf49a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc815bf30 0x7f0cc8140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.270+0000 7f0ccf49a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc813dff0 con 0x7f0cc815bf30 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.270+0000 7f0ccf49a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc815b730 con 0x7f0cc80b3060 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.270+0000 7f0ccf49a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80cd7a0 con 0x7f0cc813dc10 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc815bf30 0x7f0cc8140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc815bf30 0x7f0cc8140880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40296/0 (socket says 172.21.15.77:40296) 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 -- 172.21.15.77:0/2695900695 learned_addr learned my addr 172.21.15.77:0/2695900695 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccca35640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0cc813dc10 0x7f0cc8140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccd236640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cc80b3060 0x7f0cc8154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 -- 172.21.15.77:0/2695900695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0cc813dc10 msgr2=0x7f0cc8140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0cc813dc10 0x7f0cc8140340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 -- 172.21.15.77:0/2695900695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cc80b3060 msgr2=0x7f0cc8154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cc80b3060 0x7f0cc8154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 -- 172.21.15.77:0/2695900695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8153d70 con 0x7f0cc815bf30 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccda37640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc815bf30 0x7f0cc8140880 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000b9a0 tx=0x7f0cc000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0cbdffb640 1 -- 172.21.15.77:0/2695900695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0019070 con 0x7f0cc815bf30 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0cbdffb640 1 -- 172.21.15.77:0/2695900695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc000cbc0 con 0x7f0cc815bf30 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc8154050 con 0x7f0cc815bf30 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0cbdffb640 1 -- 172.21.15.77:0/2695900695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0014440 con 0x7f0cc815bf30 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.271+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc8154610 con 0x7f0cc815bf30 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.272+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc804f330 con 0x7f0cc815bf30 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.273+0000 7f0cbdffb640 1 -- 172.21.15.77:0/2695900695 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cc001c070 con 0x7f0cc815bf30 2024-03-06T07:26:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.273+0000 7f0cbdffb640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cac06b4d0 0x7f0cac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.273+0000 7f0ccd236640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cac06b4d0 0x7f0cac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.273+0000 7f0cbdffb640 1 -- 172.21.15.77:0/2695900695 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0cc0089b00 con 0x7f0cc815bf30 2024-03-06T07:26:06.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.274+0000 7f0ccd236640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cac06b4d0 0x7f0cac06d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4000f60 tx=0x7f0cc4007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.278+0000 7f0cbdffb640 1 -- 172.21.15.77:0/2695900695 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cc0058710 con 0x7f0cc815bf30 2024-03-06T07:26:06.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.451+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cc80b1180 con 0x7f0cc815bf30 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.452+0000 7f0cbdffb640 1 -- 172.21.15.77:0/2695900695 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+12880 (secure 0 0 0) 0x7f0cc005c7b0 con 0x7f0cc815bf30 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:06.454 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:05.784619+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:06.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cac06b4d0 msgr2=0x7f0cac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cac06b4d0 0x7f0cac06d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4000f60 tx=0x7f0cc4007d50 comp rx=0 tx=0).stop 2024-03-06T07:26:06.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc815bf30 msgr2=0x7f0cc8140880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc815bf30 0x7f0cc8140880 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000b9a0 tx=0x7f0cc000be60 comp rx=0 tx=0).stop 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 shutdown_connections 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0cac06b4d0 0x7f0cac06d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc815bf30 0x7f0cc8140880 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0cc813dc10 0x7f0cc8140340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 --2- 172.21.15.77:0/2695900695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0cc80b3060 0x7f0cc8154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 >> 172.21.15.77:0/2695900695 conn(0x7f0cc80cf8a0 msgr2=0x7f0cc80715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 shutdown_connections 2024-03-06T07:26:06.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.454+0000 7f0ccf49a640 1 -- 172.21.15.77:0/2695900695 wait complete. 2024-03-06T07:26:06.467 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:06.467 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.569+0000 7f64774fa640 1 Processor -- start 2024-03-06T07:26:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.569+0000 7f64774fa640 1 -- start start 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f64774fa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f647015a220 0x7f64700cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f64774fa640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f64700712d0 0x7f6470071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f64774fa640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64700b3320 con 0x7f647015de60 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f64774fa640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64700cf610 con 0x7f647015a220 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f64774fa640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6470071c70 con 0x7f64700712d0 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6475296640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f647015a220 0x7f64700cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6475296640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f647015a220 0x7f64700cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40312/0 (socket says 172.21.15.77:40312) 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6475a97640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f647015de60 0x7f647015e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56402/0 (socket says 172.21.15.77:56402) 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6475296640 1 -- 172.21.15.77:0/3956431140 learned_addr learned my addr 172.21.15.77:0/3956431140 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6475296640 1 -- 172.21.15.77:0/3956431140 >> v2:172.21.15.158:3300/0 conn(0x7f64700712d0 msgr2=0x7f6470071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6475296640 1 --2- 172.21.15.77:0/3956431140 >> v2:172.21.15.158:3300/0 conn(0x7f64700712d0 0x7f6470071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6475296640 1 -- 172.21.15.77:0/3956431140 >> v1:172.21.15.77:6789/0 conn(0x7f647015de60 legacy=0x7f647015e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6467fff640 1 -- 172.21.15.77:0/3956431140 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696976555 0 0) 0x7f64700b3320 con 0x7f647015de60 2024-03-06T07:26:06.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.570+0000 7f6475296640 1 -- 172.21.15.77:0/3956431140 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700728d0 con 0x7f647015a220 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f6475296640 1 --2- 172.21.15.77:0/3956431140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015a220 0x7f64700cf0d0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f64700d0980 tx=0x7f646c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3b28efb90f1233 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f6467fff640 1 -- 172.21.15.77:0/3956431140 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c035020 con 0x7f647015a220 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f6467fff640 1 -- 172.21.15.77:0/3956431140 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c031870 con 0x7f647015a220 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f6467fff640 1 -- 172.21.15.77:0/3956431140 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c031c40 con 0x7f647015a220 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f64774fa640 1 -- 172.21.15.77:0/3956431140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015a220 msgr2=0x7f64700cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f64774fa640 1 --2- 172.21.15.77:0/3956431140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015a220 0x7f64700cf0d0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f64700d0980 tx=0x7f646c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f64774fa640 1 -- 172.21.15.77:0/3956431140 shutdown_connections 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f64774fa640 1 --2- 172.21.15.77:0/3956431140 >> v2:172.21.15.158:3300/0 conn(0x7f64700712d0 0x7f6470071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f64774fa640 1 --2- 172.21.15.77:0/3956431140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015a220 0x7f64700cf0d0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f64774fa640 1 -- 172.21.15.77:0/3956431140 >> 172.21.15.77:0/3956431140 conn(0x7f647006b190 msgr2=0x7f647006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f64774fa640 1 -- 172.21.15.77:0/3956431140 shutdown_connections 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.571+0000 7f64774fa640 1 -- 172.21.15.77:0/3956431140 wait complete. 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f64774fa640 1 Processor -- start 2024-03-06T07:26:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f64774fa640 1 -- start start 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f64774fa640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700712d0 0x7f64701482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f64774fa640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647015a220 0x7f6470133730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f64774fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015de60 0x7f647015cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f64774fa640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6470072030 con 0x7f647015de60 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f64774fa640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6470072bb0 con 0x7f647015a220 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f64774fa640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6470072530 con 0x7f64700712d0 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f6475a97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015de60 0x7f647015cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f6475a97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015de60 0x7f647015cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40316/0 (socket says 172.21.15.77:40316) 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.573+0000 7f6475a97640 1 -- 172.21.15.77:0/2084141632 learned_addr learned my addr 172.21.15.77:0/2084141632 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6475a97640 1 -- 172.21.15.77:0/2084141632 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700712d0 msgr2=0x7f64701482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6474a95640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647015a220 0x7f6470133730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6475a97640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700712d0 0x7f64701482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6475a97640 1 -- 172.21.15.77:0/2084141632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647015a220 msgr2=0x7f6470133730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6475a97640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647015a220 0x7f6470133730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6475a97640 1 -- 172.21.15.77:0/2084141632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470133c70 con 0x7f647015de60 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6475a97640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015de60 0x7f647015cd60 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f646800c700 tx=0x7f646800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6465ffb640 1 -- 172.21.15.77:0/2084141632 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468018070 con 0x7f647015de60 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6465ffb640 1 -- 172.21.15.77:0/2084141632 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646800ce10 con 0x7f647015de60 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6470148900 con 0x7f647015de60 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f6465ffb640 1 -- 172.21.15.77:0/2084141632 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646800a860 con 0x7f647015de60 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.574+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6470148ec0 con 0x7f647015de60 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.575+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64700732b0 con 0x7f647015de60 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.576+0000 7f6465ffb640 1 -- 172.21.15.77:0/2084141632 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f646801b070 con 0x7f647015de60 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.576+0000 7f6465ffb640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645406b500 0x7f645406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.576+0000 7f6475296640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645406b500 0x7f645406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.576+0000 7f6465ffb640 1 -- 172.21.15.77:0/2084141632 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f64680893e0 con 0x7f647015de60 2024-03-06T07:26:06.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.577+0000 7f6475296640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645406b500 0x7f645406d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f646c036000 tx=0x7f646c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.580+0000 7f6465ffb640 1 -- 172.21.15.77:0/2084141632 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6470133c70 con 0x7f647015de60 2024-03-06T07:26:06.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.753+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6470068ff0 con 0x7f647015de60 2024-03-06T07:26:06.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.753+0000 7f6465ffb640 1 -- 172.21.15.77:0/2084141632 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64680585b0 con 0x7f647015de60 2024-03-06T07:26:06.755 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:26:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645406b500 msgr2=0x7f645406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645406b500 0x7f645406d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f646c036000 tx=0x7f646c030000 comp rx=0 tx=0).stop 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015de60 msgr2=0x7f647015cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015de60 0x7f647015cd60 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f646800c700 tx=0x7f646800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 shutdown_connections 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645406b500 0x7f645406d9c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f647015de60 0x7f647015cd60 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f647015a220 0x7f6470133730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 --2- 172.21.15.77:0/2084141632 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700712d0 0x7f64701482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 >> 172.21.15.77:0/2084141632 conn(0x7f647006b190 msgr2=0x7f64700cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 shutdown_connections 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.755+0000 7f64774fa640 1 -- 172.21.15.77:0/2084141632 wait complete. 2024-03-06T07:26:06.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:06.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.867+0000 7ffa7c0b5640 1 Processor -- start 2024-03-06T07:26:06.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.867+0000 7ffa7c0b5640 1 -- start start 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa7c0b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 0x7ffa7415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa7c0b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa74071f40 con 0x7ffa740cce40 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa7c0b5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa74070a70 con 0x7ffa741515e0 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa7c0b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740cd2f0 con 0x7ffa740cdeb0 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa79650640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffa740cce40 0x7ffa740cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56404/0 (socket says 172.21.15.77:56404) 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa79650640 1 -- 172.21.15.77:0/2811914580 learned_addr learned my addr 172.21.15.77:0/2811914580 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947271923 0 0) 0x7ffa74071f40 con 0x7ffa740cce40 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa60003520 con 0x7ffa740cce40 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.868+0000 7ffa7a652640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 0x7ffa7415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3140901007 0 0) 0x7ffa60003520 con 0x7ffa740cce40 2024-03-06T07:26:06.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 msgr2=0x7ffa7415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 0x7ffa7415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 >> v1:172.21.15.158:6789/0 conn(0x7ffa741515e0 legacy=0x7ffa74071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74070d50 con 0x7ffa740cce40 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ffa64001690 con 0x7ffa740cce40 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 >> v1:172.21.15.77:6789/0 conn(0x7ffa740cce40 legacy=0x7ffa740cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa600039e0 0x7ffa60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa60004800 0x7ffa60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 0x7ffa600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740cd2f0 con 0x7ffa60004800 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6000cb80 con 0x7ffa600039e0 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6000cd80 con 0x7ffa740cdeb0 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa79650640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa60004800 0x7ffa60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa79e51640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa600039e0 0x7ffa60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa7a652640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 0x7ffa600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa79650640 1 -- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 msgr2=0x7ffa600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa79650640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 0x7ffa600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa79650640 1 -- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa600039e0 msgr2=0x7ffa60003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa79650640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa600039e0 0x7ffa60003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa79650640 1 -- 172.21.15.77:0/2811914580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60003520 con 0x7ffa60004800 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.869+0000 7ffa7a652640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa740cdeb0 0x7ffa600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.870+0000 7ffa79650640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa60004800 0x7ffa60004c80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7ffa740cc370 tx=0x7ffa60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb6cb5196e27d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.870+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64007d30 con 0x7ffa60004800 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.870+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa64032070 con 0x7ffa60004800 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.870+0000 7ffa78e4f640 1 -- 172.21.15.77:0/2811914580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa640058d0 con 0x7ffa60004800 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.871+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2811914580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa60004800 msgr2=0x7ffa60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.871+0000 7ffa7c0b5640 1 --2- 172.21.15.77:0/2811914580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa60004800 0x7ffa60004c80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7ffa740cc370 tx=0x7ffa60005830 comp rx=0 tx=0).stop 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.871+0000 7ffa7a652640 1 -- 172.21.15.77:0/2811914580 reap_dead start 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.871+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2811914580 shutdown_connections 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.871+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2811914580 >> 172.21.15.77:0/2811914580 conn(0x7ffa740cf8a0 msgr2=0x7ffa740cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.871+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2811914580 shutdown_connections 2024-03-06T07:26:06.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.871+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2811914580 wait complete. 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.872+0000 7ffa7c0b5640 1 Processor -- start 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.872+0000 7ffa7c0b5640 1 -- start start 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa7c0b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa60004800 0x7ffa74153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa7c0b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa74139160 0x7ffa7406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa7c0b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa741396d0 0x7ffa74139b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa7c0b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7415b430 con 0x7ffa74139160 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa7c0b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74071310 con 0x7ffa741396d0 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa7c0b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7415b5b0 con 0x7ffa60004800 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa74139160 0x7ffa7406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa74139160 0x7ffa7406d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40338/0 (socket says 172.21.15.77:40338) 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 -- 172.21.15.77:0/2541311204 learned_addr learned my addr 172.21.15.77:0/2541311204 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 -- 172.21.15.77:0/2541311204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa60004800 msgr2=0x7ffa74153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa60004800 0x7ffa74153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 -- 172.21.15.77:0/2541311204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa741396d0 msgr2=0x7ffa74139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa741396d0 0x7ffa74139b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 -- 172.21.15.77:0/2541311204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74154590 con 0x7ffa74139160 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa79650640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa74139160 0x7ffa7406d0c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400c300 tx=0x7ffa74154f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa6a7fc640 1 -- 172.21.15.77:0/2541311204 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64007d30 con 0x7ffa74139160 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa6a7fc640 1 -- 172.21.15.77:0/2541311204 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa64004c10 con 0x7ffa74139160 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa74154870 con 0x7ffa74139160 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa6a7fc640 1 -- 172.21.15.77:0/2541311204 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64007070 con 0x7ffa74139160 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.873+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa74143e30 con 0x7ffa74139160 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.874+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa74068840 con 0x7ffa74139160 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.875+0000 7ffa6a7fc640 1 -- 172.21.15.77:0/2541311204 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa64049020 con 0x7ffa74139160 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.875+0000 7ffa6a7fc640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa5806b4d0 0x7ffa5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.875+0000 7ffa6a7fc640 1 -- 172.21.15.77:0/2541311204 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffa640ab310 con 0x7ffa74139160 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.875+0000 7ffa79e51640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa5806b4d0 0x7ffa5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.875+0000 7ffa79e51640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa5806b4d0 0x7ffa5806d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffa70001e50 tx=0x7ffa70009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:06.879+0000 7ffa6a7fc640 1 -- 172.21.15.77:0/2541311204 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa6407a570 con 0x7ffa74139160 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.053+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ffa74154d90 con 0x7ffa74139160 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.053+0000 7ffa6a7fc640 1 -- 172.21.15.77:0/2541311204 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v88) v1 ==== 84+0+65 (secure 0 0 0) 0x7ffa74154d90 con 0x7ffa74139160 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:07.055 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:26:07.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa5806b4d0 msgr2=0x7ffa5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa5806b4d0 0x7ffa5806d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffa70001e50 tx=0x7ffa70009040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa74139160 msgr2=0x7ffa7406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa74139160 0x7ffa7406d0c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400c300 tx=0x7ffa74154f70 comp rx=0 tx=0).stop 2024-03-06T07:26:07.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 shutdown_connections 2024-03-06T07:26:07.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa5806b4d0 0x7ffa5806d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa74139160 0x7ffa7406d0c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa741396d0 0x7ffa74139b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 --2- 172.21.15.77:0/2541311204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa60004800 0x7ffa74153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 >> 172.21.15.77:0/2541311204 conn(0x7ffa740cf8a0 msgr2=0x7ffa7415acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 shutdown_connections 2024-03-06T07:26:07.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.055+0000 7ffa7c0b5640 1 -- 172.21.15.77:0/2541311204 wait complete. 2024-03-06T07:26:07.068 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:26:07.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:26:07.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.170+0000 7f5f845d0640 1 Processor -- start 2024-03-06T07:26:07.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.170+0000 7f5f845d0640 1 -- start start 2024-03-06T07:26:07.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f845d0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5f7c15dfa0 0x7f5f7c0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f845d0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f7c0d4630 0x7f5f7c0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f845d0640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7c15e380 con 0x7f5f7c0d2820 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f845d0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c0d2c70 con 0x7f5f7c0d4630 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f845d0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c0d2100 con 0x7f5f7c15dfa0 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f81b6b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f7c0d4630 0x7f5f7c0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f81b6b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f7c0d4630 0x7f5f7c0d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40352/0 (socket says 172.21.15.77:40352) 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f81b6b640 1 -- 172.21.15.77:0/254363879 learned_addr learned my addr 172.21.15.77:0/254363879 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.171+0000 7f5f82b6d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5f7c0d2820 0x7f5f7c0d05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56414/0 (socket says 172.21.15.77:56414) 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.172+0000 7f5f8236c640 1 --2- 172.21.15.77:0/254363879 >> v2:172.21.15.158:3300/0 conn(0x7f5f7c15dfa0 0x7f5f7c0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.172+0000 7f5f81b6b640 1 -- 172.21.15.77:0/254363879 >> v2:172.21.15.158:3300/0 conn(0x7f5f7c15dfa0 msgr2=0x7f5f7c0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.172+0000 7f5f81b6b640 1 --2- 172.21.15.77:0/254363879 >> v2:172.21.15.158:3300/0 conn(0x7f5f7c15dfa0 0x7f5f7c0d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.172+0000 7f5f81b6b640 1 -- 172.21.15.77:0/254363879 >> v1:172.21.15.77:6789/0 conn(0x7f5f7c0d2820 legacy=0x7f5f7c0d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.172+0000 7f5f81b6b640 1 -- 172.21.15.77:0/254363879 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0d1e30 con 0x7f5f7c0d4630 2024-03-06T07:26:07.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.172+0000 7f5f8136a640 1 -- 172.21.15.77:0/254363879 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942562751 0 0) 0x7f5f7c15e380 con 0x7f5f7c0d2820 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.172+0000 7f5f81b6b640 1 --2- 172.21.15.77:0/254363879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c0d22e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c08d560 tx=0x7f5f6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3098fd8a56222eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f8136a640 1 -- 172.21.15.77:0/254363879 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c035020 con 0x7f5f7c0d4630 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f8136a640 1 -- 172.21.15.77:0/254363879 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c031870 con 0x7f5f7c0d4630 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f8136a640 1 -- 172.21.15.77:0/254363879 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c031c40 con 0x7f5f7c0d4630 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f845d0640 1 -- 172.21.15.77:0/254363879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 msgr2=0x7f5f7c0d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f845d0640 1 --2- 172.21.15.77:0/254363879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c0d22e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c08d560 tx=0x7f5f6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f845d0640 1 -- 172.21.15.77:0/254363879 shutdown_connections 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f845d0640 1 --2- 172.21.15.77:0/254363879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c0d22e0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f845d0640 1 --2- 172.21.15.77:0/254363879 >> v2:172.21.15.158:3300/0 conn(0x7f5f7c15dfa0 0x7f5f7c0d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f845d0640 1 -- 172.21.15.77:0/254363879 >> 172.21.15.77:0/254363879 conn(0x7f5f7c09d000 msgr2=0x7f5f7c0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f845d0640 1 -- 172.21.15.77:0/254363879 shutdown_connections 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.173+0000 7f5f845d0640 1 -- 172.21.15.77:0/254363879 wait complete. 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.174+0000 7f5f845d0640 1 Processor -- start 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.174+0000 7f5f845d0640 1 -- start start 2024-03-06T07:26:07.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f845d0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f7c0d2820 0x7f5f7c0a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f845d0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c09f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f845d0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f7c15dfa0 0x7f5f7c0ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f845d0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0cfa30 con 0x7f5f7c0d4630 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f845d0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0cf750 con 0x7f5f7c0d2820 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f845d0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0cc1a0 con 0x7f5f7c15dfa0 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f81b6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c09f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f81b6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c09f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40360/0 (socket says 172.21.15.77:40360) 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f81b6b640 1 -- 172.21.15.77:0/2508473372 learned_addr learned my addr 172.21.15.77:0/2508473372 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f81b6b640 1 -- 172.21.15.77:0/2508473372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f7c15dfa0 msgr2=0x7f5f7c0ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.175+0000 7f5f82b6d640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f7c15dfa0 0x7f5f7c0ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f8236c640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f7c0d2820 0x7f5f7c0a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f81b6b640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f7c15dfa0 0x7f5f7c0ce7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f81b6b640 1 -- 172.21.15.77:0/2508473372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f7c0d2820 msgr2=0x7f5f7c0a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f81b6b640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f7c0d2820 0x7f5f7c0a1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f81b6b640 1 -- 172.21.15.77:0/2508473372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0a16a0 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f81b6b640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c09f7c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c000c00 tx=0x7f5f6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f72ffd640 1 -- 172.21.15.77:0/2508473372 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c041070 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f72ffd640 1 -- 172.21.15.77:0/2508473372 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c03ad00 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7c0a00c0 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f72ffd640 1 -- 172.21.15.77:0/2508473372 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c035040 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.176+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7c0a0680 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.177+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7c04f330 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.177+0000 7f5f72ffd640 1 -- 172.21.15.77:0/2508473372 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f6c038020 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.178+0000 7f5f72ffd640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f6006b4d0 0x7f5f6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.178+0000 7f5f72ffd640 1 -- 172.21.15.77:0/2508473372 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5f6c0abd50 con 0x7f5f7c0d4630 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.178+0000 7f5f8236c640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f6006b4d0 0x7f5f6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.178+0000 7f5f8236c640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f6006b4d0 0x7f5f6006d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009e90 tx=0x7f5f7800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.182+0000 7f5f72ffd640 1 -- 172.21.15.77:0/2508473372 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f6c0af030 con 0x7f5f7c0d4630 2024-03-06T07:26:07.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.353+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5f7c0c9cd0 con 0x7f5f7c0d4630 2024-03-06T07:26:07.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.791+0000 7f5f72ffd640 1 -- 172.21.15.77:0/2508473372 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5f6c043350 con 0x7f5f7c0d4630 2024-03-06T07:26:07.793 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:26:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f6006b4d0 msgr2=0x7f5f6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f6006b4d0 0x7f5f6006d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009e90 tx=0x7f5f7800b040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 msgr2=0x7f5f7c09f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c09f7c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c000c00 tx=0x7f5f6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 shutdown_connections 2024-03-06T07:26:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f6006b4d0 0x7f5f6006d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f7c0d4630 0x7f5f7c09f7c0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5f7c15dfa0 0x7f5f7c0ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 --2- 172.21.15.77:0/2508473372 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f7c0d2820 0x7f5f7c0a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 >> 172.21.15.77:0/2508473372 conn(0x7f5f7c09d000 msgr2=0x7f5f7c0cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 shutdown_connections 2024-03-06T07:26:07.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.793+0000 7f5f845d0640 1 -- 172.21.15.77:0/2508473372 wait complete. 2024-03-06T07:26:07.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:07.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.917+0000 7f451cd32640 1 Processor -- start 2024-03-06T07:26:07.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.917+0000 7f451cd32640 1 -- start start 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f451cd32640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45180d4860 0x7f45180d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f451cd32640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180b0e30 0x7f45180b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f451cd32640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45181618b0 con 0x7f45181614d0 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f451cd32640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45180d4cf0 con 0x7f45180d4860 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f451cd32640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45180cf400 con 0x7f45180b0e30 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f450ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45180d4860 0x7f45180d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f450ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45180d4860 0x7f45180d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40362/0 (socket says 172.21.15.77:40362) 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f450ffff640 1 -- 172.21.15.77:0/3817188750 learned_addr learned my addr 172.21.15.77:0/3817188750 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f4516d9d640 1 --2- 172.21.15.77:0/3817188750 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180b0e30 0x7f45180b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f450ffff640 1 -- 172.21.15.77:0/3817188750 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180b0e30 msgr2=0x7f45180b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f450ffff640 1 --2- 172.21.15.77:0/3817188750 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180b0e30 0x7f45180b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f450ffff640 1 -- 172.21.15.77:0/3817188750 >> v1:172.21.15.158:6789/0 conn(0x7f45181614d0 legacy=0x7f45180d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.918+0000 7f450ffff640 1 -- 172.21.15.77:0/3817188750 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45180cf6a0 con 0x7f45180d4860 2024-03-06T07:26:07.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.919+0000 7f450ffff640 1 --2- 172.21.15.77:0/3817188750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45180d4860 0x7f45180d2020 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f4500009870 tx=0x7f450002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451d98ba1fb6a142 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.919+0000 7f4515d9b640 1 -- 172.21.15.77:0/3817188750 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500035020 con 0x7f45180d4860 2024-03-06T07:26:07.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.919+0000 7f4515d9b640 1 -- 172.21.15.77:0/3817188750 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4500031640 con 0x7f45180d4860 2024-03-06T07:26:07.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.919+0000 7f4515d9b640 1 -- 172.21.15.77:0/3817188750 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450002d040 con 0x7f45180d4860 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.920+0000 7f451cd32640 1 -- 172.21.15.77:0/3817188750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45180d4860 msgr2=0x7f45180d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.920+0000 7f451cd32640 1 --2- 172.21.15.77:0/3817188750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45180d4860 0x7f45180d2020 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f4500009870 tx=0x7f450002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.920+0000 7f451cd32640 1 -- 172.21.15.77:0/3817188750 shutdown_connections 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.920+0000 7f451cd32640 1 --2- 172.21.15.77:0/3817188750 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180b0e30 0x7f45180b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.920+0000 7f451cd32640 1 --2- 172.21.15.77:0/3817188750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45180d4860 0x7f45180d2020 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.920+0000 7f451cd32640 1 -- 172.21.15.77:0/3817188750 >> 172.21.15.77:0/3817188750 conn(0x7f45180d3d40 msgr2=0x7f45180d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.920+0000 7f451cd32640 1 -- 172.21.15.77:0/3817188750 shutdown_connections 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.920+0000 7f451cd32640 1 -- 172.21.15.77:0/3817188750 wait complete. 2024-03-06T07:26:07.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.921+0000 7f451cd32640 1 Processor -- start 2024-03-06T07:26:07.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.921+0000 7f451cd32640 1 -- start start 2024-03-06T07:26:07.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451cd32640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45180b0e30 0x7f4518151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451cd32640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180d4860 0x7f45180cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451cd32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45181614d0 0x7f45180cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451cd32640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45180b0910 con 0x7f45181614d0 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451cd32640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45180cfc60 con 0x7f45180b0e30 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451cd32640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45180d2560 con 0x7f45180d4860 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45181614d0 0x7f45180cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45181614d0 0x7f45180cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40366/0 (socket says 172.21.15.77:40366) 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 -- 172.21.15.77:0/3771354015 learned_addr learned my addr 172.21.15.77:0/3771354015 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451659c640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45180b0e30 0x7f4518151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 -- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180d4860 msgr2=0x7f45180cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f450ffff640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180d4860 0x7f45180cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180d4860 0x7f45180cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 -- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45180b0e30 msgr2=0x7f4518151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45180b0e30 0x7f4518151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 -- 172.21.15.77:0/3771354015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4518151780 con 0x7f45181614d0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451659c640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45180b0e30 0x7f4518151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f4516d9d640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45181614d0 0x7f45180cd870 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f4510007e40 tx=0x7f451000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f450effd640 1 -- 172.21.15.77:0/3771354015 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510014620 con 0x7f45181614d0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f450effd640 1 -- 172.21.15.77:0/3771354015 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4510019070 con 0x7f45181614d0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f450effd640 1 -- 172.21.15.77:0/3771354015 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510014a20 con 0x7f45181614d0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45180b9600 con 0x7f45181614d0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.922+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45180c8080 con 0x7f45181614d0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.923+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4518068ff0 con 0x7f45181614d0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.924+0000 7f450effd640 1 -- 172.21.15.77:0/3771354015 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4510014bc0 con 0x7f45181614d0 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.924+0000 7f450effd640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44f806b500 0x7f44f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.924+0000 7f450effd640 1 -- 172.21.15.77:0/3771354015 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f451008a8f0 con 0x7f45181614d0 2024-03-06T07:26:07.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.924+0000 7f451659c640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44f806b500 0x7f44f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.925+0000 7f451659c640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44f806b500 0x7f44f806d9c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f450800a830 tx=0x7f4508008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:07.929+0000 7f450effd640 1 -- 172.21.15.77:0/3771354015 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4510059950 con 0x7f45181614d0 2024-03-06T07:26:08.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.104+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f451815a730 con 0x7f45181614d0 2024-03-06T07:26:08.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.796+0000 7f450effd640 1 -- 172.21.15.77:0/3771354015 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v90) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4510059250 con 0x7f45181614d0 2024-03-06T07:26:08.798 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:26:08.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.799+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44f806b500 msgr2=0x7f44f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.799+0000 7f451cd32640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44f806b500 0x7f44f806d9c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f450800a830 tx=0x7f4508008040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.799+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45181614d0 msgr2=0x7f45180cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.799+0000 7f451cd32640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45181614d0 0x7f45180cd870 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f4510007e40 tx=0x7f451000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.800+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 shutdown_connections 2024-03-06T07:26:08.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.800+0000 7f451cd32640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44f806b500 0x7f44f806d9c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.800+0000 7f451cd32640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45181614d0 0x7f45180cd870 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.800+0000 7f451cd32640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f45180d4860 0x7f45180cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.800+0000 7f451cd32640 1 --2- 172.21.15.77:0/3771354015 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f45180b0e30 0x7f4518151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.800+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 >> 172.21.15.77:0/3771354015 conn(0x7f45180d3d40 msgr2=0x7f451806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.801+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 shutdown_connections 2024-03-06T07:26:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.805+0000 7f451cd32640 1 -- 172.21.15.77:0/3771354015 wait complete. 2024-03-06T07:26:08.829 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:26:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.931+0000 7fcc82f8a640 1 Processor -- start 2024-03-06T07:26:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.931+0000 7fcc82f8a640 1 -- start start 2024-03-06T07:26:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.932+0000 7fcc82f8a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc7c147590 0x7fcc7c162220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.932+0000 7fcc82f8a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c06afb0 0x7fcc7c06b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.932+0000 7fcc82f8a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c15f2f0 con 0x7fcc7c072630 2024-03-06T07:26:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.932+0000 7fcc82f8a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c147970 con 0x7fcc7c147590 2024-03-06T07:26:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.932+0000 7fcc82f8a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c15ed30 con 0x7fcc7c06afb0 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.932+0000 7fcc80d26640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc7c147590 0x7fcc7c162220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.932+0000 7fcc7bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcc7c072630 0x7fcc7c15e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43934/0 (socket says 172.21.15.77:43934) 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.932+0000 7fcc7bfff640 1 -- 172.21.15.77:0/478902109 learned_addr learned my addr 172.21.15.77:0/478902109 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc80d26640 1 -- 172.21.15.77:0/478902109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c06afb0 msgr2=0x7fcc7c06b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc81527640 1 --2- 172.21.15.77:0/478902109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c06afb0 0x7fcc7c06b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc80d26640 1 --2- 172.21.15.77:0/478902109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c06afb0 0x7fcc7c06b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc7b7fe640 1 -- 172.21.15.77:0/478902109 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083990125 0 0) 0x7fcc7c15f2f0 con 0x7fcc7c072630 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc80d26640 1 -- 172.21.15.77:0/478902109 >> v1:172.21.15.77:6789/0 conn(0x7fcc7c072630 legacy=0x7fcc7c15e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc80d26640 1 -- 172.21.15.77:0/478902109 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c15f010 con 0x7fcc7c147590 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc81527640 1 --2- 172.21.15.77:0/478902109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c06afb0 0x7fcc7c06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc80d26640 1 --2- 172.21.15.77:0/478902109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c147590 0x7fcc7c162220 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0097a0 tx=0x7fcc6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbf3da50bd59c9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.933+0000 7fcc7b7fe640 1 -- 172.21.15.77:0/478902109 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c035020 con 0x7fcc7c147590 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc7b7fe640 1 -- 172.21.15.77:0/478902109 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c031830 con 0x7fcc7c147590 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc7b7fe640 1 -- 172.21.15.77:0/478902109 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c031c00 con 0x7fcc7c147590 2024-03-06T07:26:08.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc82f8a640 1 -- 172.21.15.77:0/478902109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c147590 msgr2=0x7fcc7c162220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/478902109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c147590 0x7fcc7c162220 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0097a0 tx=0x7fcc6c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc82f8a640 1 -- 172.21.15.77:0/478902109 shutdown_connections 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/478902109 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c06afb0 0x7fcc7c06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/478902109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c147590 0x7fcc7c162220 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc82f8a640 1 -- 172.21.15.77:0/478902109 >> 172.21.15.77:0/478902109 conn(0x7fcc7c06c970 msgr2=0x7fcc7c0b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc82f8a640 1 -- 172.21.15.77:0/478902109 shutdown_connections 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.934+0000 7fcc82f8a640 1 -- 172.21.15.77:0/478902109 wait complete. 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.935+0000 7fcc82f8a640 1 Processor -- start 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.935+0000 7fcc82f8a640 1 -- start start 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc82f8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c06afb0 0x7fcc7c136b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc82f8a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc7c072630 0x7fcc7c0cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc82f8a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c147590 0x7fcc7c0b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc82f8a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c15ea30 con 0x7fcc7c06afb0 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc82f8a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c161d00 con 0x7fcc7c072630 2024-03-06T07:26:08.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc82f8a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c162760 con 0x7fcc7c147590 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc80d26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c06afb0 0x7fcc7c136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc80d26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c06afb0 0x7fcc7c136b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60554/0 (socket says 172.21.15.77:60554) 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc80d26640 1 -- 172.21.15.77:0/3420510489 learned_addr learned my addr 172.21.15.77:0/3420510489 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc80d26640 1 -- 172.21.15.77:0/3420510489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c147590 msgr2=0x7fcc7c0b01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc80d26640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c147590 0x7fcc7c0b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc80d26640 1 -- 172.21.15.77:0/3420510489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc7c072630 msgr2=0x7fcc7c0cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc80d26640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc7c072630 0x7fcc7c0cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.936+0000 7fcc80d26640 1 -- 172.21.15.77:0/3420510489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c0d00f0 con 0x7fcc7c06afb0 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.937+0000 7fcc80d26640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c06afb0 0x7fcc7c136b10 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0098c0 tx=0x7fcc6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.937+0000 7fcc797fa640 1 -- 172.21.15.77:0/3420510489 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c040070 con 0x7fcc7c06afb0 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.937+0000 7fcc797fa640 1 -- 172.21.15.77:0/3420510489 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c039a00 con 0x7fcc7c06afb0 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.937+0000 7fcc797fa640 1 -- 172.21.15.77:0/3420510489 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c035040 con 0x7fcc7c06afb0 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.937+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c0b0830 con 0x7fcc7c06afb0 2024-03-06T07:26:08.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.937+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c0b0df0 con 0x7fcc7c06afb0 2024-03-06T07:26:08.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.938+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c04f330 con 0x7fcc7c06afb0 2024-03-06T07:26:08.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.938+0000 7fcc797fa640 1 -- 172.21.15.77:0/3420510489 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc6c038020 con 0x7fcc7c06afb0 2024-03-06T07:26:08.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.939+0000 7fcc797fa640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc6806b500 0x7fcc6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.939+0000 7fcc797fa640 1 -- 172.21.15.77:0/3420510489 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fcc6c0abb20 con 0x7fcc7c06afb0 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.939+0000 7fcc7bfff640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc6806b500 0x7fcc6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.939+0000 7fcc7bfff640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc6806b500 0x7fcc6806d9c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0d1980 tx=0x7fcc70007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:08.943+0000 7fcc797fa640 1 -- 172.21.15.77:0/3420510489 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc6c04b340 con 0x7fcc7c06afb0 2024-03-06T07:26:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.151+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcc7c06b7c0 con 0x7fcc7c06afb0 2024-03-06T07:26:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:26:09.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.810+0000 7fcc797fa640 1 -- 172.21.15.77:0/3420510489 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v45) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcc6c07aa90 con 0x7fcc7c06afb0 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.815+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc6806b500 msgr2=0x7fcc6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.815+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc6806b500 0x7fcc6806d9c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0d1980 tx=0x7fcc70007920 comp rx=0 tx=0).stop 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.815+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c06afb0 msgr2=0x7fcc7c136b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.815+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c06afb0 0x7fcc7c136b10 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0098c0 tx=0x7fcc6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.821+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 shutdown_connections 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.821+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcc7c147590 0x7fcc7c0b01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.821+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcc7c072630 0x7fcc7c0cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.821+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcc6806b500 0x7fcc6806d9c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.821+0000 7fcc82f8a640 1 --2- 172.21.15.77:0/3420510489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc7c06afb0 0x7fcc7c136b10 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.821+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 >> 172.21.15.77:0/3420510489 conn(0x7fcc7c06c970 msgr2=0x7fcc7c0b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.822+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 shutdown_connections 2024-03-06T07:26:09.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.822+0000 7fcc82f8a640 1 -- 172.21.15.77:0/3420510489 wait complete. 2024-03-06T07:26:09.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:09.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.937+0000 7f4336c5c640 1 Processor -- start 2024-03-06T07:26:09.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.938+0000 7f4336c5c640 1 -- start start 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.938+0000 7f4336c5c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f433013dc10 0x7f43300b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.938+0000 7f4336c5c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f433015ef60 0x7f43300b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.938+0000 7f4336c5c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433015f580 con 0x7f433015bf30 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.938+0000 7f4336c5c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43300cf3f0 con 0x7f433013dc10 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.938+0000 7f4336c5c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433015b210 con 0x7f433015ef60 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.938+0000 7f43349f8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f433013dc10 0x7f43300b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.939+0000 7f43349f8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f433013dc10 0x7f43300b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60560/0 (socket says 172.21.15.77:60560) 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.939+0000 7f43349f8640 1 -- 172.21.15.77:0/1059904452 learned_addr learned my addr 172.21.15.77:0/1059904452 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.939+0000 7f432ffff640 1 --1- 172.21.15.77:0/1059904452 >> v1:172.21.15.77:6789/0 conn(0x7f433015bf30 0x7f433015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43948/0 (socket says 172.21.15.77:43948) 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.939+0000 7f43349f8640 1 -- 172.21.15.77:0/1059904452 >> v2:172.21.15.158:3300/0 conn(0x7f433015ef60 msgr2=0x7f43300b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.939+0000 7f43349f8640 1 --2- 172.21.15.77:0/1059904452 >> v2:172.21.15.158:3300/0 conn(0x7f433015ef60 0x7f43300b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.939+0000 7f43349f8640 1 -- 172.21.15.77:0/1059904452 >> v1:172.21.15.77:6789/0 conn(0x7f433015bf30 legacy=0x7f433015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.939+0000 7f43349f8640 1 -- 172.21.15.77:0/1059904452 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433015b450 con 0x7f433013dc10 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.939+0000 7f432f7fe640 1 -- 172.21.15.77:0/1059904452 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743444156 0 0) 0x7f433015f580 con 0x7f433015bf30 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f43349f8640 1 --2- 172.21.15.77:0/1059904452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43300b2950 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4330068700 tx=0x7f432402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33f006c0d702845 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f432f7fe640 1 -- 172.21.15.77:0/1059904452 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324035020 con 0x7f433013dc10 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f432f7fe640 1 -- 172.21.15.77:0/1059904452 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324031870 con 0x7f433013dc10 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f432f7fe640 1 -- 172.21.15.77:0/1059904452 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324031c40 con 0x7f433013dc10 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f4336c5c640 1 -- 172.21.15.77:0/1059904452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 msgr2=0x7f43300b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f4336c5c640 1 --2- 172.21.15.77:0/1059904452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43300b2950 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4330068700 tx=0x7f432402b910 comp rx=0 tx=0).stop 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f4336c5c640 1 -- 172.21.15.77:0/1059904452 shutdown_connections 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f4336c5c640 1 --2- 172.21.15.77:0/1059904452 >> v2:172.21.15.158:3300/0 conn(0x7f433015ef60 0x7f43300b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f4336c5c640 1 --2- 172.21.15.77:0/1059904452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43300b2950 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f4336c5c640 1 -- 172.21.15.77:0/1059904452 >> 172.21.15.77:0/1059904452 conn(0x7f43300cf8a0 msgr2=0x7f43300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f4336c5c640 1 -- 172.21.15.77:0/1059904452 shutdown_connections 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.940+0000 7f4336c5c640 1 -- 172.21.15.77:0/1059904452 wait complete. 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.941+0000 7f4336c5c640 1 Processor -- start 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.941+0000 7f4336c5c640 1 -- start start 2024-03-06T07:26:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f4336c5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43301f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f4336c5c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f433015bf30 0x7f43301e6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f4336c5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f433015ef60 0x7f4330153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f4336c5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43300708d0 con 0x7f433013dc10 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f4336c5c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433013dff0 con 0x7f433015bf30 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f4336c5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4330070a50 con 0x7f433015ef60 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f43349f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43301f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f43349f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43301f5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60566/0 (socket says 172.21.15.77:60566) 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.942+0000 7f43349f8640 1 -- 172.21.15.77:0/601578634 learned_addr learned my addr 172.21.15.77:0/601578634 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f432ffff640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f433015bf30 0x7f43301e6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f43349f8640 1 -- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f433015ef60 msgr2=0x7f4330153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f43351f9640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f433015ef60 0x7f4330153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f43349f8640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f433015ef60 0x7f4330153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f43349f8640 1 -- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f433015bf30 msgr2=0x7f43301e6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f43349f8640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f433015bf30 0x7f43301e6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f43349f8640 1 -- 172.21.15.77:0/601578634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43301f21a0 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f432ffff640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f433015bf30 0x7f43301e6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f43349f8640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43301f5d90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4324000c00 tx=0x7f43240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f432d7fa640 1 -- 172.21.15.77:0/601578634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432403ad60 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f432d7fa640 1 -- 172.21.15.77:0/601578634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43240443f0 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43301f2480 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f432d7fa640 1 -- 172.21.15.77:0/601578634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43240447e0 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.943+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43301f2a40 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.944+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433004f330 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.944+0000 7f432d7fa640 1 -- 172.21.15.77:0/601578634 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f432404b060 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.945+0000 7f432d7fa640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f431406b500 0x7f431406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.945+0000 7f432ffff640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f431406b500 0x7f431406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.945+0000 7f432d7fa640 1 -- 172.21.15.77:0/601578634 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43240ac3d0 con 0x7f433013dc10 2024-03-06T07:26:09.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.945+0000 7f432ffff640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f431406b500 0x7f431406d9c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f43300ce840 tx=0x7f431c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:09.949+0000 7f432d7fa640 1 -- 172.21.15.77:0/601578634 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f432403f070 con 0x7f433013dc10 2024-03-06T07:26:10.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.122+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4330144790 con 0x7f433013dc10 2024-03-06T07:26:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.123+0000 7f432d7fa640 1 -- 172.21.15.77:0/601578634 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f432402f320 con 0x7f433013dc10 2024-03-06T07:26:10.125 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:10.125 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":91,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:09.152283+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:07.372376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:08.123063+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:10.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f431406b500 msgr2=0x7f431406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f431406b500 0x7f431406d9c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f43300ce840 tx=0x7f431c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 msgr2=0x7f43301f5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43301f5d90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4324000c00 tx=0x7f43240046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 shutdown_connections 2024-03-06T07:26:10.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f433015ef60 0x7f4330153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f433015bf30 0x7f43301e6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f431406b500 0x7f431406d9c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 --2- 172.21.15.77:0/601578634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f433013dc10 0x7f43301f5d90 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 >> 172.21.15.77:0/601578634 conn(0x7f43300cf8a0 msgr2=0x7f43300b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 shutdown_connections 2024-03-06T07:26:10.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.125+0000 7f4336c5c640 1 -- 172.21.15.77:0/601578634 wait complete. 2024-03-06T07:26:10.138 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:10.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.236+0000 7f5dc229d640 1 Processor -- start 2024-03-06T07:26:10.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.236+0000 7f5dc229d640 1 -- start start 2024-03-06T07:26:10.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dc229d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5dbc0b4f70 0x7f5dbc0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dc229d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5dbc06c6d0 0x7f5dbc15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dc229d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc066d90 0x7f5dbc0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dc229d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dbc15e9f0 con 0x7f5dbc0b4f70 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dc229d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dbc0cda10 con 0x7f5dbc06c6d0 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dc229d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0b5890 con 0x7f5dbc066d90 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dbbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5dbc0b4f70 0x7f5dbc0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dbbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5dbc0b4f70 0x7f5dbc0b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60580/0 (socket says 172.21.15.77:60580) 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dbbfff640 1 -- 172.21.15.77:0/2911373704 learned_addr learned my addr 172.21.15.77:0/2911373704 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dbb7fe640 1 --2- 172.21.15.77:0/2911373704 >> v2:172.21.15.158:3300/0 conn(0x7f5dbc06c6d0 0x7f5dbc15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.237+0000 7f5dc083a640 1 --2- 172.21.15.77:0/2911373704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc066d90 0x7f5dbc0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbbfff640 1 -- 172.21.15.77:0/2911373704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc066d90 msgr2=0x7f5dbc0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbbfff640 1 --2- 172.21.15.77:0/2911373704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc066d90 0x7f5dbc0ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbbfff640 1 -- 172.21.15.77:0/2911373704 >> v2:172.21.15.158:3300/0 conn(0x7f5dbc06c6d0 msgr2=0x7f5dbc15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbbfff640 1 --2- 172.21.15.77:0/2911373704 >> v2:172.21.15.158:3300/0 conn(0x7f5dbc06c6d0 0x7f5dbc15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbbfff640 1 -- 172.21.15.77:0/2911373704 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc0cd450 con 0x7f5dbc0b4f70 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbb7fe640 1 --2- 172.21.15.77:0/2911373704 >> v2:172.21.15.158:3300/0 conn(0x7f5dbc06c6d0 0x7f5dbc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbbfff640 1 --2- 172.21.15.77:0/2911373704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0b4f70 0x7f5dbc0b5350 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0ccb00 tx=0x7f5dac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a3c65ea7dd7cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbaffd640 1 -- 172.21.15.77:0/2911373704 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac035020 con 0x7f5dbc0b4f70 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbaffd640 1 -- 172.21.15.77:0/2911373704 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac031870 con 0x7f5dbc0b4f70 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.238+0000 7f5dbaffd640 1 -- 172.21.15.77:0/2911373704 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac031c40 con 0x7f5dbc0b4f70 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 -- 172.21.15.77:0/2911373704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0b4f70 msgr2=0x7f5dbc0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 --2- 172.21.15.77:0/2911373704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0b4f70 0x7f5dbc0b5350 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0ccb00 tx=0x7f5dac02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 -- 172.21.15.77:0/2911373704 shutdown_connections 2024-03-06T07:26:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 --2- 172.21.15.77:0/2911373704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc066d90 0x7f5dbc0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 --2- 172.21.15.77:0/2911373704 >> v2:172.21.15.158:3300/0 conn(0x7f5dbc06c6d0 0x7f5dbc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 --2- 172.21.15.77:0/2911373704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0b4f70 0x7f5dbc0b5350 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 -- 172.21.15.77:0/2911373704 >> 172.21.15.77:0/2911373704 conn(0x7f5dbc0724d0 msgr2=0x7f5dbc0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 -- 172.21.15.77:0/2911373704 shutdown_connections 2024-03-06T07:26:10.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.239+0000 7f5dc229d640 1 -- 172.21.15.77:0/2911373704 wait complete. 2024-03-06T07:26:10.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.240+0000 7f5dc229d640 1 Processor -- start 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.240+0000 7f5dc229d640 1 -- start start 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc229d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dbc066d90 0x7f5dbc160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc229d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc0b4f70 0x7f5dbc0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc229d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0cec40 0x7f5dbc14c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc229d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc15f3c0 con 0x7f5dbc0cec40 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc229d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc15e680 con 0x7f5dbc066d90 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc229d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0cd090 con 0x7f5dbc0b4f70 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc083a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0cec40 0x7f5dbc14c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc083a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0cec40 0x7f5dbc14c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60592/0 (socket says 172.21.15.77:60592) 2024-03-06T07:26:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc083a640 1 -- 172.21.15.77:0/803698621 learned_addr learned my addr 172.21.15.77:0/803698621 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dbbfff640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dbc066d90 0x7f5dbc160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dbb7fe640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc0b4f70 0x7f5dbc0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc083a640 1 -- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc0b4f70 msgr2=0x7f5dbc0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc083a640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc0b4f70 0x7f5dbc0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc083a640 1 -- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dbc066d90 msgr2=0x7f5dbc160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc083a640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dbc066d90 0x7f5dbc160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dc083a640 1 -- 172.21.15.77:0/803698621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc15b960 con 0x7f5dbc0cec40 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dbbfff640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dbc066d90 0x7f5dbc160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.241+0000 7f5dbb7fe640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc0b4f70 0x7f5dbc0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.242+0000 7f5dc083a640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0cec40 0x7f5dbc14c050 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f5db4007d40 tx=0x7f5db400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.242+0000 7f5db8ff9640 1 -- 172.21.15.77:0/803698621 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4014420 con 0x7f5dbc0cec40 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.242+0000 7f5db8ff9640 1 -- 172.21.15.77:0/803698621 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db4019070 con 0x7f5dbc0cec40 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.242+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dbc0cf2a0 con 0x7f5dbc0cec40 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.242+0000 7f5db8ff9640 1 -- 172.21.15.77:0/803698621 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4014800 con 0x7f5dbc0cec40 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.242+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dbc14a3b0 con 0x7f5dbc0cec40 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.243+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dbc0728b0 con 0x7f5dbc0cec40 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.243+0000 7f5db8ff9640 1 -- 172.21.15.77:0/803698621 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5db40149a0 con 0x7f5dbc0cec40 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.244+0000 7f5db8ff9640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5da806b4d0 0x7f5da806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.244+0000 7f5dbbfff640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5da806b4d0 0x7f5da806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.244+0000 7f5db8ff9640 1 -- 172.21.15.77:0/803698621 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5db408a970 con 0x7f5dbc0cec40 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.244+0000 7f5dbbfff640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5da806b4d0 0x7f5da806d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5dac036000 tx=0x7f5dac03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.248+0000 7f5db8ff9640 1 -- 172.21.15.77:0/803698621 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5dbc0728b0 con 0x7f5dbc0cec40 2024-03-06T07:26:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.421+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dbc04f330 con 0x7f5dbc0cec40 2024-03-06T07:26:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.422+0000 7f5db8ff9640 1 -- 172.21.15.77:0/803698621 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f5dbc15b960 con 0x7f5dbc0cec40 2024-03-06T07:26:10.424 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:10.424 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":91,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:09.152283+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:07.372376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:08.123063+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:10.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5da806b4d0 msgr2=0x7f5da806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5da806b4d0 0x7f5da806d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5dac036000 tx=0x7f5dac03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0cec40 msgr2=0x7f5dbc14c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0cec40 0x7f5dbc14c050 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f5db4007d40 tx=0x7f5db400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 shutdown_connections 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5da806b4d0 0x7f5da806d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dbc0cec40 0x7f5dbc14c050 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5dbc0b4f70 0x7f5dbc0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 --2- 172.21.15.77:0/803698621 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5dbc066d90 0x7f5dbc160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 >> 172.21.15.77:0/803698621 conn(0x7f5dbc0724d0 msgr2=0x7f5dbc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 shutdown_connections 2024-03-06T07:26:10.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.424+0000 7f5dc229d640 1 -- 172.21.15.77:0/803698621 wait complete. 2024-03-06T07:26:10.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.538+0000 7f35ec73e640 1 Processor -- start 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.538+0000 7f35ec73e640 1 -- start start 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.539+0000 7f35ec73e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40cf0d0 0x7f35e406d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.539+0000 7f35ec73e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e40b3320 con 0x7f35e415a220 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.539+0000 7f35ec73e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e40cf560 con 0x7f35e40719e0 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.539+0000 7f35ec73e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e40728d0 con 0x7f35e40cf0d0 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.539+0000 7f35ea4da640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f35e415a220 0x7f35e40712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43962/0 (socket says 172.21.15.77:43962) 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.539+0000 7f35ea4da640 1 -- 172.21.15.77:0/1353871263 learned_addr learned my addr 172.21.15.77:0/1353871263 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754681854 0 0) 0x7f35e40b3320 con 0x7f35e415a220 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35eacdb640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40cf0d0 0x7f35e406d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc003520 con 0x7f35e415a220 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045515030 0 0) 0x7f35cc003520 con 0x7f35e415a220 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40cf0d0 msgr2=0x7f35e406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40cf0d0 0x7f35e406d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 >> v1:172.21.15.158:6789/0 conn(0x7f35e40719e0 legacy=0x7f35e406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e40b3590 con 0x7f35e415a220 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f35d40019f0 con 0x7f35e415a220 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 >> v1:172.21.15.77:6789/0 conn(0x7f35e415a220 legacy=0x7f35e40712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc003bb0 0x7f35cc003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc008870 0x7f35cc00ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc00b5a0 0x7f35cc00da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e40728d0 con 0x7f35cc008870 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc00e310 con 0x7f35cc00b5a0 2024-03-06T07:26:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc00e510 con 0x7f35cc003bb0 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35eacdb640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40cf0d0 0x7f35e406d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e9cd9640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc008870 0x7f35cc00ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35ea4da640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc003bb0 0x7f35cc003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35eacdb640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc00b5a0 0x7f35cc00da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e9cd9640 1 -- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc003bb0 msgr2=0x7f35cc003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e9cd9640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc003bb0 0x7f35cc003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e9cd9640 1 -- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc00b5a0 msgr2=0x7f35cc00da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e9cd9640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc00b5a0 0x7f35cc00da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35e9cd9640 1 -- 172.21.15.77:0/1353871263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc003520 con 0x7f35cc008870 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.540+0000 7f35ea4da640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35cc003bb0 0x7f35cc003fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.541+0000 7f35e9cd9640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc008870 0x7f35cc00ad00 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f35d8001750 tx=0x7f35cc003810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c940f4be6c279 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.541+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8001370 con 0x7f35cc008870 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.541+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d80052d0 con 0x7f35cc008870 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.541+0000 7f35e94d8640 1 -- 172.21.15.77:0/1353871263 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8003830 con 0x7f35cc008870 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.542+0000 7f35ec73e640 1 -- 172.21.15.77:0/1353871263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc008870 msgr2=0x7f35cc00ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.542+0000 7f35ec73e640 1 --2- 172.21.15.77:0/1353871263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35cc008870 0x7f35cc00ad00 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f35d8001750 tx=0x7f35cc003810 comp rx=0 tx=0).stop 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.542+0000 7f35eacdb640 1 -- 172.21.15.77:0/1353871263 reap_dead start 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.542+0000 7f35ec73e640 1 -- 172.21.15.77:0/1353871263 shutdown_connections 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.542+0000 7f35ec73e640 1 -- 172.21.15.77:0/1353871263 >> 172.21.15.77:0/1353871263 conn(0x7f35e406b190 msgr2=0x7f35e406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.542+0000 7f35ec73e640 1 -- 172.21.15.77:0/1353871263 shutdown_connections 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.542+0000 7f35ec73e640 1 -- 172.21.15.77:0/1353871263 wait complete. 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.543+0000 7f35ec73e640 1 Processor -- start 2024-03-06T07:26:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.543+0000 7f35ec73e640 1 -- start start 2024-03-06T07:26:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ec73e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc008870 0x7f35e41f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ec73e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e40abe60 0x7f35e40cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ec73e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40aa4a0 0x7f35e414cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ec73e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e4072530 con 0x7f35e40abe60 2024-03-06T07:26:10.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ec73e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e4072030 con 0x7f35cc008870 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ec73e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e40726b0 con 0x7f35e40aa4a0 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e40abe60 0x7f35e40cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e40abe60 0x7f35e40cfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60610/0 (socket says 172.21.15.77:60610) 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 -- 172.21.15.77:0/2435474619 learned_addr learned my addr 172.21.15.77:0/2435474619 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ea4da640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc008870 0x7f35e41f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 -- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40aa4a0 msgr2=0x7f35e414cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35eacdb640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40aa4a0 0x7f35e414cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40aa4a0 0x7f35e414cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 -- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc008870 msgr2=0x7f35e41f0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc008870 0x7f35e41f0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 -- 172.21.15.77:0/2435474619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e40aae10 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ea4da640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc008870 0x7f35e41f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e9cd9640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e40abe60 0x7f35e40cfcd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f35d8005040 tx=0x7f35d8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e2ffd640 1 -- 172.21.15.77:0/2435474619 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8041070 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e2ffd640 1 -- 172.21.15.77:0/2435474619 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d8005480 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e40ab0f0 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35e2ffd640 1 -- 172.21.15.77:0/2435474619 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8001370 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.544+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e40ac680 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.545+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e4071500 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.546+0000 7f35e2ffd640 1 -- 172.21.15.77:0/2435474619 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35d8044070 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.546+0000 7f35e2ffd640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f35d006b500 0x7f35d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.546+0000 7f35ea4da640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f35d006b500 0x7f35d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.546+0000 7f35e2ffd640 1 -- 172.21.15.77:0/2435474619 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35d80abcb0 con 0x7f35e40abe60 2024-03-06T07:26:10.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.547+0000 7f35ea4da640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f35d006b500 0x7f35d006d9c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f35d4001330 tx=0x7f35d4002190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.551+0000 7f35e2ffd640 1 -- 172.21.15.77:0/2435474619 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35e4071500 con 0x7f35e40abe60 2024-03-06T07:26:10.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.757+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f35e40732b0 con 0x7f35e40abe60 2024-03-06T07:26:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.812+0000 7f35e2ffd640 1 -- 172.21.15.77:0/2435474619 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v46) v1 ==== 118+0+0 (secure 0 0 0) 0x7f35d807ab30 con 0x7f35e40abe60 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f35d006b500 msgr2=0x7f35d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f35d006b500 0x7f35d006d9c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f35d4001330 tx=0x7f35d4002190 comp rx=0 tx=0).stop 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e40abe60 msgr2=0x7f35e40cfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e40abe60 0x7f35e40cfcd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f35d8005040 tx=0x7f35d8001120 comp rx=0 tx=0).stop 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 shutdown_connections 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f35d006b500 0x7f35d006d9c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35e40abe60 0x7f35e40cfcd0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35e40aa4a0 0x7f35e414cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 --2- 172.21.15.77:0/2435474619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35cc008870 0x7f35e41f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 >> 172.21.15.77:0/2435474619 conn(0x7f35e406b190 msgr2=0x7f35e415c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 shutdown_connections 2024-03-06T07:26:10.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.814+0000 7f35ec73e640 1 -- 172.21.15.77:0/2435474619 wait complete. 2024-03-06T07:26:10.830 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:10.831 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:26:10.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.932+0000 7f8d0c53a640 1 Processor -- start 2024-03-06T07:26:10.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.932+0000 7f8d0c53a640 1 -- start start 2024-03-06T07:26:10.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d0c53a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8d041614d0 0x7f8d040d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d0c53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d24b0 0x7f8d040b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d0c53a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d041618b0 con 0x7f8d040d1910 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d0c53a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d040cf400 con 0x7f8d041614d0 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d0c53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040cf6a0 con 0x7f8d040d24b0 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d0aad7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d24b0 0x7f8d040b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d0a2d6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8d041614d0 0x7f8d040d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d09ad5640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8d040d1910 0x7f8d040d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:51676/0 (socket says 172.21.15.77:51676) 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.933+0000 7f8d09ad5640 1 -- 172.21.15.77:0/1248439475 learned_addr learned my addr 172.21.15.77:0/1248439475 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.934+0000 7f8d0a2d6640 1 -- 172.21.15.77:0/1248439475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d24b0 msgr2=0x7f8d040b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.934+0000 7f8d0a2d6640 1 --2- 172.21.15.77:0/1248439475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d24b0 0x7f8d040b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.934+0000 7f8d0a2d6640 1 -- 172.21.15.77:0/1248439475 >> v1:172.21.15.158:6789/0 conn(0x7f8d040d1910 legacy=0x7f8d040d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.934+0000 7f8d0a2d6640 1 -- 172.21.15.77:0/1248439475 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d040cf980 con 0x7f8d041614d0 2024-03-06T07:26:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.934+0000 7f8d0aad7640 1 --2- 172.21.15.77:0/1248439475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d24b0 0x7f8d040b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0a2d6640 1 --2- 172.21.15.77:0/1248439475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d041614d0 0x7f8d040d4860 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40097a0 tx=0x7f8cf402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39f2dd4207a24d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d092d4640 1 -- 172.21.15.77:0/1248439475 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4035020 con 0x7f8d041614d0 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d092d4640 1 -- 172.21.15.77:0/1248439475 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf4031830 con 0x7f8d041614d0 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d092d4640 1 -- 172.21.15.77:0/1248439475 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4031c00 con 0x7f8d041614d0 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0c53a640 1 -- 172.21.15.77:0/1248439475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d041614d0 msgr2=0x7f8d040d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/1248439475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d041614d0 0x7f8d040d4860 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40097a0 tx=0x7f8cf402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0c53a640 1 -- 172.21.15.77:0/1248439475 shutdown_connections 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/1248439475 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d24b0 0x7f8d040b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/1248439475 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d041614d0 0x7f8d040d4860 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0c53a640 1 -- 172.21.15.77:0/1248439475 >> 172.21.15.77:0/1248439475 conn(0x7f8d040d3d40 msgr2=0x7f8d040d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0c53a640 1 -- 172.21.15.77:0/1248439475 shutdown_connections 2024-03-06T07:26:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.935+0000 7f8d0c53a640 1 -- 172.21.15.77:0/1248439475 wait complete. 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0c53a640 1 Processor -- start 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0c53a640 1 -- start start 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0c53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d1910 0x7f8d040b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0c53a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d040d24b0 0x7f8d040b90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0c53a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d041614d0 0x7f8d041f7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0c53a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040b1370 con 0x7f8d041614d0 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0c53a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040b0910 con 0x7f8d040d24b0 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0c53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040b14f0 con 0x7f8d040d1910 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0aad7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d041614d0 0x7f8d041f7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0aad7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d041614d0 0x7f8d041f7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60624/0 (socket says 172.21.15.77:60624) 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0aad7640 1 -- 172.21.15.77:0/669175863 learned_addr learned my addr 172.21.15.77:0/669175863 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d09ad5640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d040d24b0 0x7f8d040b90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.937+0000 7f8d0a2d6640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d1910 0x7f8d040b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d0aad7640 1 -- 172.21.15.77:0/669175863 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d1910 msgr2=0x7f8d040b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d0aad7640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d1910 0x7f8d040b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d0aad7640 1 -- 172.21.15.77:0/669175863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d040d24b0 msgr2=0x7f8d040b90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d0aad7640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d040d24b0 0x7f8d040b90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d0aad7640 1 -- 172.21.15.77:0/669175863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d040ca040 con 0x7f8d041614d0 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d0aad7640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d041614d0 0x7f8d041f7880 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00efc0 tx=0x7f8cfc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d02ffd640 1 -- 172.21.15.77:0/669175863 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc018030 con 0x7f8d041614d0 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d02ffd640 1 -- 172.21.15.77:0/669175863 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc00cde0 con 0x7f8d041614d0 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0415ac90 con 0x7f8d041614d0 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d02ffd640 1 -- 172.21.15.77:0/669175863 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc003bb0 con 0x7f8d041614d0 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.938+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d040b9600 con 0x7f8d041614d0 2024-03-06T07:26:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.939+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d04068ff0 con 0x7f8d041614d0 2024-03-06T07:26:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.940+0000 7f8d02ffd640 1 -- 172.21.15.77:0/669175863 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cfc005d10 con 0x7f8d041614d0 2024-03-06T07:26:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.940+0000 7f8d02ffd640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cf006b500 0x7f8cf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.940+0000 7f8d0a2d6640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cf006b500 0x7f8cf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.940+0000 7f8d02ffd640 1 -- 172.21.15.77:0/669175863 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cfc08aae0 con 0x7f8d041614d0 2024-03-06T07:26:10.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.941+0000 7f8d0a2d6640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cf006b500 0x7f8cf006d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4036000 tx=0x7f8cf403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:10.944+0000 7f8d02ffd640 1 -- 172.21.15.77:0/669175863 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8cfc022070 con 0x7f8d041614d0 2024-03-06T07:26:11.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.266+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8d040d0b90 con 0x7f8d041614d0 2024-03-06T07:26:11.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.824+0000 7f8d02ffd640 1 -- 172.21.15.77:0/669175863 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v47) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8cfc010040 con 0x7f8d041614d0 2024-03-06T07:26:11.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cf006b500 msgr2=0x7f8cf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cf006b500 0x7f8cf006d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4036000 tx=0x7f8cf403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d041614d0 msgr2=0x7f8d041f7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d041614d0 0x7f8d041f7880 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00efc0 tx=0x7f8cfc009f90 comp rx=0 tx=0).stop 2024-03-06T07:26:11.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 shutdown_connections 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8cf006b500 0x7f8cf006d9c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d041614d0 0x7f8d041f7880 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8d040d24b0 0x7f8d040b90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 --2- 172.21.15.77:0/669175863 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8d040d1910 0x7f8d040b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 >> 172.21.15.77:0/669175863 conn(0x7f8d040d3d40 msgr2=0x7f8d040d2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 shutdown_connections 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.826+0000 7f8d0c53a640 1 -- 172.21.15.77:0/669175863 wait complete. 2024-03-06T07:26:11.840 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:26:11.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.958+0000 7fd5367b6640 1 Processor -- start 2024-03-06T07:26:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.958+0000 7fd5367b6640 1 -- start start 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.958+0000 7fd5367b6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd53015dfe0 0x7fd5300cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.958+0000 7fd5367b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.958+0000 7fd5367b6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5300d27f0 0x7fd5300d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.958+0000 7fd5367b6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5300d2c50 con 0x7fd53015dfe0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.958+0000 7fd5367b6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5300b31e0 con 0x7fd5300d27f0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.958+0000 7fd5367b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300b3d60 con 0x7fd5300cf830 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd53015dfe0 0x7fd5300cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd53015dfe0 0x7fd5300cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60630/0 (socket says 172.21.15.77:60630) 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52ffff640 1 -- 172.21.15.77:0/4020770862 learned_addr learned my addr 172.21.15.77:0/4020770862 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52f7fe640 1 --2- 172.21.15.77:0/4020770862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd534d53640 1 --2- 172.21.15.77:0/4020770862 >> v2:172.21.15.158:3300/0 conn(0x7fd5300d27f0 0x7fd5300d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52ffff640 1 -- 172.21.15.77:0/4020770862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 msgr2=0x7fd5300d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52ffff640 1 --2- 172.21.15.77:0/4020770862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52ffff640 1 -- 172.21.15.77:0/4020770862 >> v2:172.21.15.158:3300/0 conn(0x7fd5300d27f0 msgr2=0x7fd5300d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52ffff640 1 --2- 172.21.15.77:0/4020770862 >> v2:172.21.15.158:3300/0 conn(0x7fd5300d27f0 0x7fd5300d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd52ffff640 1 -- 172.21.15.77:0/4020770862 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5300b3430 con 0x7fd53015dfe0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.959+0000 7fd534d53640 1 --2- 172.21.15.77:0/4020770862 >> v2:172.21.15.158:3300/0 conn(0x7fd5300d27f0 0x7fd5300d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd52ffff640 1 --2- 172.21.15.77:0/4020770862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cf2f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fd530067ad0 tx=0x7fd52402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9561eece4bba6ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd52effd640 1 -- 172.21.15.77:0/4020770862 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd524035020 con 0x7fd53015dfe0 2024-03-06T07:26:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd52effd640 1 -- 172.21.15.77:0/4020770862 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd524031870 con 0x7fd53015dfe0 2024-03-06T07:26:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd52effd640 1 -- 172.21.15.77:0/4020770862 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd524031c40 con 0x7fd53015dfe0 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 -- 172.21.15.77:0/4020770862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 msgr2=0x7fd5300cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 --2- 172.21.15.77:0/4020770862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cf2f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fd530067ad0 tx=0x7fd52402b910 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 -- 172.21.15.77:0/4020770862 shutdown_connections 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 --2- 172.21.15.77:0/4020770862 >> v2:172.21.15.158:3300/0 conn(0x7fd5300d27f0 0x7fd5300d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 --2- 172.21.15.77:0/4020770862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 --2- 172.21.15.77:0/4020770862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cf2f0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 -- 172.21.15.77:0/4020770862 >> 172.21.15.77:0/4020770862 conn(0x7fd5300d3cd0 msgr2=0x7fd5300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 -- 172.21.15.77:0/4020770862 shutdown_connections 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.960+0000 7fd5367b6640 1 -- 172.21.15.77:0/4020770862 wait complete. 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.961+0000 7fd5367b6640 1 Processor -- start 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd5367b6640 1 -- start start 2024-03-06T07:26:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd5367b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd5367b6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5300d27f0 0x7fd5300cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd5367b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd5367b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300d1c70 con 0x7fd53015dfe0 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd5367b6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300b39f0 con 0x7fd5300d27f0 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd5367b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300d1df0 con 0x7fd5300cf830 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd534d53640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd534d53640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60634/0 (socket says 172.21.15.77:60634) 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.962+0000 7fd534d53640 1 -- 172.21.15.77:0/609922588 learned_addr learned my addr 172.21.15.77:0/609922588 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd52ffff640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd52f7fe640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5300d27f0 0x7fd5300cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd534d53640 1 -- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 msgr2=0x7fd5300cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd534d53640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd534d53640 1 -- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5300d27f0 msgr2=0x7fd5300cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd534d53640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5300d27f0 0x7fd5300cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd534d53640 1 -- 172.21.15.77:0/609922588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5300cd5c0 con 0x7fd53015dfe0 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd52ffff640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd52f7fe640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5300d27f0 0x7fd5300cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd534d53640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cb2e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd528007ef0 tx=0x7fd52800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd52cff9640 1 -- 172.21.15.77:0/609922588 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52801a070 con 0x7fd53015dfe0 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd52cff9640 1 -- 172.21.15.77:0/609922588 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd528003b90 con 0x7fd53015dfe0 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5300cd8a0 con 0x7fd53015dfe0 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd52cff9640 1 -- 172.21.15.77:0/609922588 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528004030 con 0x7fd53015dfe0 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.963+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5300cde60 con 0x7fd53015dfe0 2024-03-06T07:26:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.964+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5300b5200 con 0x7fd53015dfe0 2024-03-06T07:26:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.964+0000 7fd52cff9640 1 -- 172.21.15.77:0/609922588 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd52801d070 con 0x7fd53015dfe0 2024-03-06T07:26:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.965+0000 7fd52cff9640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd51c06b4d0 0x7fd51c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.965+0000 7fd52cff9640 1 -- 172.21.15.77:0/609922588 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd52808a440 con 0x7fd53015dfe0 2024-03-06T07:26:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.965+0000 7fd52ffff640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd51c06b4d0 0x7fd51c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.966+0000 7fd52ffff640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd51c06b4d0 0x7fd51c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd524036000 tx=0x7fd52403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:11.969+0000 7fd52cff9640 1 -- 172.21.15.77:0/609922588 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5300b5200 con 0x7fd53015dfe0 2024-03-06T07:26:12.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.182+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5300d0b60 con 0x7fd53015dfe0 2024-03-06T07:26:12.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.182+0000 7fd52cff9640 1 -- 172.21.15.77:0/609922588 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd528010070 con 0x7fd53015dfe0 2024-03-06T07:26:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:12.184 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/2526815605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":2526815605},{"type":"v1","addr":"172.21.15.158:6834","nonce":2526815605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6850,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/497683896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":497683896},{"type":"v1","addr":"172.21.15.158:6835","nonce":497683896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":7284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1877423196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1877423196},{"type":"v1","addr":"172.21.15.77:6835","nonce":1877423196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:11.267910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7287":{"gid":7287,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3425851566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3425851566},{"type":"v1","addr":"172.21.15.77:6837","nonce":3425851566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7287,"qdb_cluster":[7287]},"id":7}]} 2024-03-06T07:26:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 47 2024-03-06T07:26:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd51c06b4d0 msgr2=0x7fd51c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd51c06b4d0 0x7fd51c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd524036000 tx=0x7fd52403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 msgr2=0x7fd5300cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cb2e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd528007ef0 tx=0x7fd52800d4a0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 shutdown_connections 2024-03-06T07:26:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd51c06b4d0 0x7fd51c06d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd53015dfe0 0x7fd5300cb2e0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5300d27f0 0x7fd5300cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 --2- 172.21.15.77:0/609922588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5300cf830 0x7fd5300cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 >> 172.21.15.77:0/609922588 conn(0x7fd5300d3cd0 msgr2=0x7fd53006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 shutdown_connections 2024-03-06T07:26:12.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.184+0000 7fd5367b6640 1 -- 172.21.15.77:0/609922588 wait complete. 2024-03-06T07:26:12.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:12.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.297+0000 7fc841a4e640 1 Processor -- start 2024-03-06T07:26:12.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.297+0000 7fc841a4e640 1 -- start start 2024-03-06T07:26:12.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc841a4e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc83c0d4760 0x7fc83c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc841a4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc83c0d0a80 0x7fc83c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc841a4e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0d25d0 con 0x7fc83c0b4170 2024-03-06T07:26:12.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc841a4e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c0d4bf0 con 0x7fc83c0d4760 2024-03-06T07:26:12.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc841a4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c0ce9e0 con 0x7fc83c0d0a80 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83a7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc83c0d4760 0x7fc83c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83a7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc83c0d4760 0x7fc83c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60648/0 (socket says 172.21.15.77:60648) 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83affd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc83c0b4170 0x7fc83c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43974/0 (socket says 172.21.15.77:43974) 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83a7fc640 1 -- 172.21.15.77:0/2865298004 learned_addr learned my addr 172.21.15.77:0/2865298004 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83b7fe640 1 --2- 172.21.15.77:0/2865298004 >> v2:172.21.15.158:3300/0 conn(0x7fc83c0d0a80 0x7fc83c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83a7fc640 1 -- 172.21.15.77:0/2865298004 >> v2:172.21.15.158:3300/0 conn(0x7fc83c0d0a80 msgr2=0x7fc83c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83a7fc640 1 --2- 172.21.15.77:0/2865298004 >> v2:172.21.15.158:3300/0 conn(0x7fc83c0d0a80 0x7fc83c0cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83a7fc640 1 -- 172.21.15.77:0/2865298004 >> v1:172.21.15.77:6789/0 conn(0x7fc83c0b4170 legacy=0x7fc83c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc839ffb640 1 -- 172.21.15.77:0/2865298004 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461704324 0 0) 0x7fc83c0d25d0 con 0x7fc83c0b4170 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83a7fc640 1 -- 172.21.15.77:0/2865298004 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c0d2870 con 0x7fc83c0d4760 2024-03-06T07:26:12.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.298+0000 7fc83b7fe640 1 --2- 172.21.15.77:0/2865298004 >> v2:172.21.15.158:3300/0 conn(0x7fc83c0d0a80 0x7fc83c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:12.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.299+0000 7fc83a7fc640 1 --2- 172.21.15.77:0/2865298004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d4760 0x7fc83c0b6a90 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fc83c08df30 tx=0x7fc83002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754cf229482afd10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc839ffb640 1 -- 172.21.15.77:0/2865298004 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830034070 con 0x7fc83c0d4760 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc839ffb640 1 -- 172.21.15.77:0/2865298004 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc830030860 con 0x7fc83c0d4760 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc839ffb640 1 -- 172.21.15.77:0/2865298004 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830030c30 con 0x7fc83c0d4760 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc841a4e640 1 -- 172.21.15.77:0/2865298004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d4760 msgr2=0x7fc83c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc841a4e640 1 --2- 172.21.15.77:0/2865298004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d4760 0x7fc83c0b6a90 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fc83c08df30 tx=0x7fc83002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc841a4e640 1 -- 172.21.15.77:0/2865298004 shutdown_connections 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc841a4e640 1 --2- 172.21.15.77:0/2865298004 >> v2:172.21.15.158:3300/0 conn(0x7fc83c0d0a80 0x7fc83c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc841a4e640 1 --2- 172.21.15.77:0/2865298004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d4760 0x7fc83c0b6a90 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc841a4e640 1 -- 172.21.15.77:0/2865298004 >> 172.21.15.77:0/2865298004 conn(0x7fc83c0d3c40 msgr2=0x7fc83c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc841a4e640 1 -- 172.21.15.77:0/2865298004 shutdown_connections 2024-03-06T07:26:12.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.300+0000 7fc841a4e640 1 -- 172.21.15.77:0/2865298004 wait complete. 2024-03-06T07:26:12.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.301+0000 7fc841a4e640 1 Processor -- start 2024-03-06T07:26:12.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc841a4e640 1 -- start start 2024-03-06T07:26:12.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc841a4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc83c0b4170 0x7fc83c0b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc841a4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d0a80 0x7fc83c15e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc841a4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc83c15e910 0x7fc83c1f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc841a4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0b4dd0 con 0x7fc83c0d0a80 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc841a4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0b4550 con 0x7fc83c0b4170 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc841a4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0b4f50 con 0x7fc83c15e910 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc83a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d0a80 0x7fc83c15e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc83a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d0a80 0x7fc83c15e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60660/0 (socket says 172.21.15.77:60660) 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.302+0000 7fc83a7fc640 1 -- 172.21.15.77:0/4120443658 learned_addr learned my addr 172.21.15.77:0/4120443658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc83a7fc640 1 -- 172.21.15.77:0/4120443658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc83c15e910 msgr2=0x7fc83c1f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc83b7fe640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc83c15e910 0x7fc83c1f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc83a7fc640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc83c15e910 0x7fc83c1f3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc83affd640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc83c0b4170 0x7fc83c0b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc83a7fc640 1 -- 172.21.15.77:0/4120443658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc83c0b4170 msgr2=0x7fc83c0b8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc83a7fc640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc83c0b4170 0x7fc83c0b8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc83a7fc640 1 -- 172.21.15.77:0/4120443658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c0cae90 con 0x7fc83c0d0a80 2024-03-06T07:26:12.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc83a7fc640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d0a80 0x7fc83c15e3d0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc83002b9e0 tx=0x7fc8300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc823fff640 1 -- 172.21.15.77:0/4120443658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830034040 con 0x7fc83c0d0a80 2024-03-06T07:26:12.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc823fff640 1 -- 172.21.15.77:0/4120443658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8300433f0 con 0x7fc83c0d0a80 2024-03-06T07:26:12.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c0cb170 con 0x7fc83c0d0a80 2024-03-06T07:26:12.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc823fff640 1 -- 172.21.15.77:0/4120443658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8300437e0 con 0x7fc83c0d0a80 2024-03-06T07:26:12.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.303+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c0cb730 con 0x7fc83c0d0a80 2024-03-06T07:26:12.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.304+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c15ecf0 con 0x7fc83c0d0a80 2024-03-06T07:26:12.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.304+0000 7fc823fff640 1 -- 172.21.15.77:0/4120443658 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc83004a060 con 0x7fc83c0d0a80 2024-03-06T07:26:12.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.305+0000 7fc823fff640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82806b4d0 0x7fc82806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.305+0000 7fc83affd640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82806b4d0 0x7fc82806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.305+0000 7fc823fff640 1 -- 172.21.15.77:0/4120443658 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8300ac490 con 0x7fc83c0d0a80 2024-03-06T07:26:12.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.305+0000 7fc83affd640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82806b4d0 0x7fc82806d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc8340012e0 tx=0x7fc834009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.309+0000 7fc823fff640 1 -- 172.21.15.77:0/4120443658 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc83003e070 con 0x7fc83c0d0a80 2024-03-06T07:26:12.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.489+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc83c04f330 con 0x7fc83c0d0a80 2024-03-06T07:26:12.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.490+0000 7fc823fff640 1 -- 172.21.15.77:0/4120443658 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16596 (secure 0 0 0) 0x7fc83004be90 con 0x7fc83c0d0a80 2024-03-06T07:26:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:12.492 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":92,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:10.798689+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:07.372376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:08.123063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:12.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82806b4d0 msgr2=0x7fc82806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82806b4d0 0x7fc82806d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc8340012e0 tx=0x7fc834009040 comp rx=0 tx=0).stop 2024-03-06T07:26:12.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d0a80 msgr2=0x7fc83c15e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d0a80 0x7fc83c15e3d0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc83002b9e0 tx=0x7fc8300046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 shutdown_connections 2024-03-06T07:26:12.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82806b4d0 0x7fc82806d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0d0a80 0x7fc83c15e3d0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc83c15e910 0x7fc83c1f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 --2- 172.21.15.77:0/4120443658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc83c0b4170 0x7fc83c0b8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 >> 172.21.15.77:0/4120443658 conn(0x7fc83c0d3c40 msgr2=0x7fc83c0b7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 shutdown_connections 2024-03-06T07:26:12.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.492+0000 7fc841a4e640 1 -- 172.21.15.77:0/4120443658 wait complete. 2024-03-06T07:26:12.509 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:12.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:12.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.644+0000 7f01fd5e9640 1 Processor -- start 2024-03-06T07:26:12.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.644+0000 7f01fd5e9640 1 -- start start 2024-03-06T07:26:12.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01fd5e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f01f815e090 0x7f01f80d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01fd5e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01f80d2920 0x7f01f80b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01fd5e9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f815e470 con 0x7f01f80b3b20 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01fd5e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f80d18f0 con 0x7f01f80d2920 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01fd5e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f80d1b90 con 0x7f01f815e090 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f67fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01f80d2920 0x7f01f80b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f77fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f01f80b3b20 0x7f01f80d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43990/0 (socket says 172.21.15.77:43990) 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f77fe640 1 -- 172.21.15.77:0/3265862250 learned_addr learned my addr 172.21.15.77:0/3265862250 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f6ffd640 1 --2- 172.21.15.77:0/3265862250 >> v2:172.21.15.158:3300/0 conn(0x7f01f815e090 0x7f01f80d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f67fc640 1 -- 172.21.15.77:0/3265862250 >> v2:172.21.15.158:3300/0 conn(0x7f01f815e090 msgr2=0x7f01f80d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f67fc640 1 --2- 172.21.15.77:0/3265862250 >> v2:172.21.15.158:3300/0 conn(0x7f01f815e090 0x7f01f80d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f67fc640 1 -- 172.21.15.77:0/3265862250 >> v1:172.21.15.77:6789/0 conn(0x7f01f80b3b20 legacy=0x7f01f80d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f67fc640 1 -- 172.21.15.77:0/3265862250 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f80ceb50 con 0x7f01f80d2920 2024-03-06T07:26:12.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.645+0000 7f01f5ffb640 1 -- 172.21.15.77:0/3265862250 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319839326 0 0) 0x7f01f815e470 con 0x7f01f80b3b20 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.646+0000 7f01f67fc640 1 --2- 172.21.15.77:0/3265862250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80b35e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f01f80b6780 tx=0x7f01ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbc9a089b5781c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.646+0000 7f01f5ffb640 1 -- 172.21.15.77:0/3265862250 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec035020 con 0x7f01f80d2920 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.646+0000 7f01f5ffb640 1 -- 172.21.15.77:0/3265862250 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ec031830 con 0x7f01f80d2920 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.646+0000 7f01f5ffb640 1 -- 172.21.15.77:0/3265862250 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec031c00 con 0x7f01f80d2920 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.647+0000 7f01fd5e9640 1 -- 172.21.15.77:0/3265862250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 msgr2=0x7f01f80b35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.647+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/3265862250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80b35e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f01f80b6780 tx=0x7f01ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.647+0000 7f01fd5e9640 1 -- 172.21.15.77:0/3265862250 shutdown_connections 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.647+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/3265862250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80b35e0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.647+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/3265862250 >> v2:172.21.15.158:3300/0 conn(0x7f01f815e090 0x7f01f80d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.647+0000 7f01fd5e9640 1 -- 172.21.15.77:0/3265862250 >> 172.21.15.77:0/3265862250 conn(0x7f01f80d3c80 msgr2=0x7f01f80d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.647+0000 7f01fd5e9640 1 -- 172.21.15.77:0/3265862250 shutdown_connections 2024-03-06T07:26:12.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.647+0000 7f01fd5e9640 1 -- 172.21.15.77:0/3265862250 wait complete. 2024-03-06T07:26:12.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.649+0000 7f01fd5e9640 1 Processor -- start 2024-03-06T07:26:12.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.649+0000 7f01fd5e9640 1 -- start start 2024-03-06T07:26:12.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01fd5e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01fd5e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01f815e090 0x7f01f8151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01fd5e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01f80ca350 0x7f01f80b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01fd5e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80ce800 con 0x7f01f80d2920 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01fd5e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80b2df0 con 0x7f01f815e090 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01fd5e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80d0d60 con 0x7f01f80ca350 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60686/0 (socket says 172.21.15.77:60686) 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01f6ffd640 1 -- 172.21.15.77:0/2982919623 learned_addr learned my addr 172.21.15.77:0/2982919623 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01f77fe640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01f80ca350 0x7f01f80b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.650+0000 7f01f67fc640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01f815e090 0x7f01f8151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01f6ffd640 1 -- 172.21.15.77:0/2982919623 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01f80ca350 msgr2=0x7f01f80b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01f6ffd640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01f80ca350 0x7f01f80b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01f6ffd640 1 -- 172.21.15.77:0/2982919623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01f815e090 msgr2=0x7f01f8151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01f6ffd640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01f815e090 0x7f01f8151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01f6ffd640 1 -- 172.21.15.77:0/2982919623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f80b7720 con 0x7f01f80d2920 2024-03-06T07:26:12.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01f6ffd640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80cad50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f01e800d600 tx=0x7f01e800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01df7fe640 1 -- 172.21.15.77:0/2982919623 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e80125f0 con 0x7f01f80d2920 2024-03-06T07:26:12.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01df7fe640 1 -- 172.21.15.77:0/2982919623 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e8019070 con 0x7f01f80d2920 2024-03-06T07:26:12.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f80b9160 con 0x7f01f80d2920 2024-03-06T07:26:12.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01df7fe640 1 -- 172.21.15.77:0/2982919623 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e80129f0 con 0x7f01f80d2920 2024-03-06T07:26:12.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.651+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f80b8e10 con 0x7f01f80d2920 2024-03-06T07:26:12.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.652+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f804f330 con 0x7f01f80d2920 2024-03-06T07:26:12.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.652+0000 7f01df7fe640 1 -- 172.21.15.77:0/2982919623 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01e8012b90 con 0x7f01f80d2920 2024-03-06T07:26:12.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.653+0000 7f01df7fe640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01e406b4d0 0x7f01e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.653+0000 7f01f67fc640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01e406b4d0 0x7f01e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.653+0000 7f01df7fe640 1 -- 172.21.15.77:0/2982919623 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01e808a670 con 0x7f01f80d2920 2024-03-06T07:26:12.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.653+0000 7f01f67fc640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01e406b4d0 0x7f01e406d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f01ec036000 tx=0x7f01ec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.657+0000 7f01df7fe640 1 -- 172.21.15.77:0/2982919623 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01e80594f0 con 0x7f01f80d2920 2024-03-06T07:26:12.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.832+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01f80d1230 con 0x7f01e406b4d0 2024-03-06T07:26:12.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.837+0000 7f01df7fe640 1 -- 172.21.15.77:0/2982919623 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237274 (secure 0 0 0) 0x7f01ec082ef0 con 0x7f01e406b4d0 2024-03-06T07:26:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:12.845 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01e406b4d0 msgr2=0x7f01e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01e406b4d0 0x7f01e406d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f01ec036000 tx=0x7f01ec008030 comp rx=0 tx=0).stop 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 msgr2=0x7f01f80cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80cad50 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f01e800d600 tx=0x7f01e800dac0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 shutdown_connections 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f01f80ca350 0x7f01f80b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f01f815e090 0x7f01f8151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f01e406b4d0 0x7f01e406d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 --2- 172.21.15.77:0/2982919623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f80d2920 0x7f01f80cad50 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 >> 172.21.15.77:0/2982919623 conn(0x7f01f80d3c80 msgr2=0x7f01f80d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 shutdown_connections 2024-03-06T07:26:12.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.841+0000 7f01fd5e9640 1 -- 172.21.15.77:0/2982919623 wait complete. 2024-03-06T07:26:12.853 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-03-06T07:26:11.429155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":275788,"kb_used_data":9672,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698932,"statfs":{"total":773094113280,"available":772811706368,"internally_reserved":0,"allocated":9904128,"data_stored":4851424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000811"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":290,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.402964+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:26:06.402964+0000","last_peered":"2024-03-06T07:26:06.402964+0000","last_clean":"2024-03-06T07:26:06.402964+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:26:06.402964+0000","last_undegraded":"2024-03-06T07:26:06.402964+0000","last_fullsized":"2024-03-06T07:26:06.402964+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1683456,"data_stored":1049204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36416,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36408,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335432,"statfs":{"total":96636764160,"available":96599482368,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335436,"statfs":{"total":96636764160,"available":96599486464,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32252,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1683456,"data_stored":1049204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:12.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.993+0000 7fe893ee1640 1 Processor -- start 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.993+0000 7fe893ee1640 1 -- start start 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.994+0000 7fe893ee1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe88c0b4f70 0x7fe88c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.994+0000 7fe893ee1640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c0b5890 con 0x7fe88c066d90 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.994+0000 7fe893ee1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c0cd450 con 0x7fe88c06c6d0 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.994+0000 7fe893ee1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c15e9f0 con 0x7fe88c0b4f70 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.994+0000 7fe89247e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe88c066d90 0x7fe88c0ccb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43992/0 (socket says 172.21.15.77:43992) 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.994+0000 7fe89247e640 1 -- 172.21.15.77:0/1566124995 learned_addr learned my addr 172.21.15.77:0/1566124995 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.994+0000 7fe891c7d640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe88c0b4f70 0x7fe88c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591707376 0 0) 0x7fe88c0b5890 con 0x7fe88c066d90 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe878003520 con 0x7fe88c066d90 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113980399 0 0) 0x7fe88c0cd450 con 0x7fe88c06c6d0 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe88c0b5890 con 0x7fe88c06c6d0 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094830991 0 0) 0x7fe878003520 con 0x7fe88c066d90 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe88c0b4f70 msgr2=0x7fe88c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe88c0b4f70 0x7fe88c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 >> v1:172.21.15.158:6789/0 conn(0x7fe88c06c6d0 legacy=0x7fe88c15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c0cd730 con 0x7fe88c066d90 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fe8840019f0 con 0x7fe88c066d90 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 >> v1:172.21.15.77:6789/0 conn(0x7fe88c066d90 legacy=0x7fe88c0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe878003b70 0x7fe878003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe88c0b4f70 0x7fe878004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878004d40 0x7fe8780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c15e9f0 con 0x7fe878004d40 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87800ba70 con 0x7fe878003b70 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87800bc20 con 0x7fe88c0b4f70 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe89247e640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878004d40 0x7fe8780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe891c7d640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe878003b70 0x7fe878003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe89147c640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe88c0b4f70 0x7fe878004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe89247e640 1 -- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe88c0b4f70 msgr2=0x7fe878004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe89247e640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe88c0b4f70 0x7fe878004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe89247e640 1 -- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe878003b70 msgr2=0x7fe878003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe89247e640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe878003b70 0x7fe878003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.995+0000 7fe89247e640 1 -- 172.21.15.77:0/1566124995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe878003520 con 0x7fe878004d40 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.996+0000 7fe89247e640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878004d40 0x7fe8780051c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe88400a7d0 tx=0x7fe878005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50933827a04bcb2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.996+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88400a430 con 0x7fe878004d40 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.996+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe884034650 con 0x7fe878004d40 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.996+0000 7fe890c7b640 1 -- 172.21.15.77:0/1566124995 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884034a40 con 0x7fe878004d40 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.997+0000 7fe893ee1640 1 -- 172.21.15.77:0/1566124995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878004d40 msgr2=0x7fe8780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.997+0000 7fe893ee1640 1 --2- 172.21.15.77:0/1566124995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe878004d40 0x7fe8780051c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe88400a7d0 tx=0x7fe878005830 comp rx=0 tx=0).stop 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.997+0000 7fe89247e640 1 -- 172.21.15.77:0/1566124995 reap_dead start 2024-03-06T07:26:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.997+0000 7fe893ee1640 1 -- 172.21.15.77:0/1566124995 shutdown_connections 2024-03-06T07:26:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.997+0000 7fe893ee1640 1 -- 172.21.15.77:0/1566124995 >> 172.21.15.77:0/1566124995 conn(0x7fe88c0724d0 msgr2=0x7fe88c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.997+0000 7fe893ee1640 1 -- 172.21.15.77:0/1566124995 shutdown_connections 2024-03-06T07:26:13.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.997+0000 7fe893ee1640 1 -- 172.21.15.77:0/1566124995 wait complete. 2024-03-06T07:26:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.998+0000 7fe893ee1640 1 Processor -- start 2024-03-06T07:26:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.998+0000 7fe893ee1640 1 -- start start 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe893ee1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe878004d40 0x7fe88c0ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe893ee1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe88c09c570 0x7fe88c09c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe893ee1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe88c153310 0x7fe88c153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe893ee1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c0b5350 con 0x7fe88c153310 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe893ee1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c0cda10 con 0x7fe88c09c570 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe893ee1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c0b54d0 con 0x7fe878004d40 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe89247e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe88c153310 0x7fe88c153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe89247e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe88c153310 0x7fe88c153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60696/0 (socket says 172.21.15.77:60696) 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe89247e640 1 -- 172.21.15.77:0/4103922580 learned_addr learned my addr 172.21.15.77:0/4103922580 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe891c7d640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe878004d40 0x7fe88c0ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe89147c640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe88c09c570 0x7fe88c09c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe89247e640 1 -- 172.21.15.77:0/4103922580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe878004d40 msgr2=0x7fe88c0ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe89247e640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe878004d40 0x7fe88c0ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe89247e640 1 -- 172.21.15.77:0/4103922580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe88c09c570 msgr2=0x7fe88c09c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:12.999+0000 7fe89247e640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe88c09c570 0x7fe88c09c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.000+0000 7fe89247e640 1 -- 172.21.15.77:0/4103922580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c09e170 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.000+0000 7fe89247e640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe88c153310 0x7fe88c153790 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fe884001c70 tx=0x7fe884001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.000+0000 7fe8827fc640 1 -- 172.21.15.77:0/4103922580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884001f90 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.000+0000 7fe8827fc640 1 -- 172.21.15.77:0/4103922580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe884044070 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.000+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88c153cd0 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.000+0000 7fe8827fc640 1 -- 172.21.15.77:0/4103922580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884034c00 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.000+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88c09d3b0 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.001+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88c04f330 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.001+0000 7fe8827fc640 1 -- 172.21.15.77:0/4103922580 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe884049460 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.002+0000 7fe8827fc640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.002+0000 7fe8827fc640 1 -- 172.21.15.77:0/4103922580 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8840ac720 con 0x7fe88c153310 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.002+0000 7fe891c7d640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.002+0000 7fe891c7d640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009b00 tx=0x7fe87c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.006+0000 7fe8827fc640 1 -- 172.21.15.77:0/4103922580 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe88407b5a0 con 0x7fe88c153310 2024-03-06T07:26:13.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.180+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe88c0b1860 con 0x7fe87006b4d0 2024-03-06T07:26:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.186+0000 7fe8827fc640 1 -- 172.21.15.77:0/4103922580 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237274 (secure 0 0 0) 0x7fe88c0b1860 con 0x7fe87006b4d0 2024-03-06T07:26:13.190 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:13.191 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:26:13.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 msgr2=0x7fe87006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009b00 tx=0x7fe87c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe88c153310 msgr2=0x7fe88c153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe88c153310 0x7fe88c153790 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fe884001c70 tx=0x7fe884001d40 comp rx=0 tx=0).stop 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 shutdown_connections 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe88c153310 0x7fe88c153790 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe88c09c570 0x7fe88c09c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 --2- 172.21.15.77:0/4103922580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe878004d40 0x7fe88c0ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 >> 172.21.15.77:0/4103922580 conn(0x7fe88c0724d0 msgr2=0x7fe88c06d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 shutdown_connections 2024-03-06T07:26:13.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.189+0000 7fe893ee1640 1 -- 172.21.15.77:0/4103922580 wait complete. 2024-03-06T07:26:13.202 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-03-06T07:26:11.429155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":275788,"kb_used_data":9672,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698932,"statfs":{"total":773094113280,"available":772811706368,"internally_reserved":0,"allocated":9904128,"data_stored":4851424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000811"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:08.792366+0000","last_change":"2024-03-06T07:26:08.792366+0000","last_active":"2024-03-06T07:26:08.792366+0000","last_peered":"2024-03-06T07:26:08.792366+0000","last_clean":"2024-03-06T07:26:08.792366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:08.792366+0000","last_undegraded":"2024-03-06T07:26:08.792366+0000","last_fullsized":"2024-03-06T07:26:08.792366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:07.787945+0000","last_change":"2024-03-06T07:26:07.787945+0000","last_active":"2024-03-06T07:26:07.787945+0000","last_peered":"2024-03-06T07:26:07.787945+0000","last_clean":"2024-03-06T07:26:07.787945+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:07.787945+0000","last_undegraded":"2024-03-06T07:26:07.787945+0000","last_fullsized":"2024-03-06T07:26:07.787945+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":290,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.402964+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:26:06.402964+0000","last_peered":"2024-03-06T07:26:06.402964+0000","last_clean":"2024-03-06T07:26:06.402964+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:26:06.402964+0000","last_undegraded":"2024-03-06T07:26:06.402964+0000","last_fullsized":"2024-03-06T07:26:06.402964+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1683456,"data_stored":1049204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36416,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36408,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335432,"statfs":{"total":96636764160,"available":96599482368,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335436,"statfs":{"total":96636764160,"available":96599486464,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32252,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":458836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1683456,"data_stored":1049204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:13.206 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.348+0000 7f179d913640 1 Processor -- start 2024-03-06T07:26:13.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.348+0000 7f179d913640 1 -- start start 2024-03-06T07:26:13.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.348+0000 7f179d913640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f179815e090 0x7f17980d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.348+0000 7f179d913640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f17980d2920 0x7f17980b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.348+0000 7f179d913640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179815e470 con 0x7f17980b3b20 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.348+0000 7f179d913640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17980d18f0 con 0x7f17980d2920 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.348+0000 7f179d913640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17980d1b90 con 0x7f179815e090 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f17977fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f17980b3b20 0x7f17980d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43994/0 (socket says 172.21.15.77:43994) 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f17977fe640 1 -- 172.21.15.77:0/1629676513 learned_addr learned my addr 172.21.15.77:0/1629676513 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f17967fc640 1 --2- 172.21.15.77:0/1629676513 >> v2:172.21.15.158:3300/0 conn(0x7f17980d2920 0x7f17980b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1796ffd640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f179815e090 0x7f17980d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518258344 0 0) 0x7f179815e470 con 0x7f17980b3b20 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1780003520 con 0x7f17980b3b20 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 826636225 0 0) 0x7f1780003520 con 0x7f17980b3b20 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f179815e090 msgr2=0x7f17980d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f179815e090 0x7f17980d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 >> v2:172.21.15.158:3300/0 conn(0x7f17980d2920 msgr2=0x7f17980b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 --2- 172.21.15.77:0/1629676513 >> v2:172.21.15.158:3300/0 conn(0x7f17980d2920 0x7f17980b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17980ceb50 con 0x7f17980b3b20 2024-03-06T07:26:13.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f17900019f0 con 0x7f17980b3b20 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 >> v1:172.21.15.77:6789/0 conn(0x7f17980b3b20 legacy=0x7f17980d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f179815e090 0x7f17800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f1780004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1780004d40 0x7f17800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17980d18f0 con 0x7f1780003f30 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17980d1b90 con 0x7f1780004d40 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178000b9c0 con 0x7f179815e090 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f17967fc640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f1780004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f1796ffd640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f179815e090 0x7f17800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.349+0000 7f17977fe640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1780004d40 0x7f17800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f17967fc640 1 -- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f179815e090 msgr2=0x7f17800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f17967fc640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f179815e090 0x7f17800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f17967fc640 1 -- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1780004d40 msgr2=0x7f17800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f17967fc640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1780004d40 0x7f17800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f17967fc640 1 -- 172.21.15.77:0/1629676513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1780003520 con 0x7f1780003f30 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f17967fc640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f1780004800 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f1788009ac0 tx=0x7f1780005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dac570624661a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788039070 con 0x7f1780003f30 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178802c6d0 con 0x7f1780003f30 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.350+0000 7f1795ffb640 1 -- 172.21.15.77:0/1629676513 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788009690 con 0x7f1780003f30 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.351+0000 7f179d913640 1 -- 172.21.15.77:0/1629676513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 msgr2=0x7f1780004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.351+0000 7f179d913640 1 --2- 172.21.15.77:0/1629676513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f1780004800 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f1788009ac0 tx=0x7f1780005830 comp rx=0 tx=0).stop 2024-03-06T07:26:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.351+0000 7f17977fe640 1 -- 172.21.15.77:0/1629676513 reap_dead start 2024-03-06T07:26:13.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.351+0000 7f179d913640 1 -- 172.21.15.77:0/1629676513 shutdown_connections 2024-03-06T07:26:13.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.351+0000 7f179d913640 1 -- 172.21.15.77:0/1629676513 >> 172.21.15.77:0/1629676513 conn(0x7f17980d3c80 msgr2=0x7f17980d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.351+0000 7f179d913640 1 -- 172.21.15.77:0/1629676513 shutdown_connections 2024-03-06T07:26:13.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.351+0000 7f179d913640 1 -- 172.21.15.77:0/1629676513 wait complete. 2024-03-06T07:26:13.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.352+0000 7f179d913640 1 Processor -- start 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.352+0000 7f179d913640 1 -- start start 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f179d913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f17980b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f179d913640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17980cdc80 0x7f17980ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f179d913640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17980cad50 0x7f17980cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f179d913640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17980ce800 con 0x7f1780003f30 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f179d913640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17980b2df0 con 0x7f17980cdc80 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f179d913640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17980d0d60 con 0x7f17980cad50 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f1796ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f17980b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f1796ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f17980b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60708/0 (socket says 172.21.15.77:60708) 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f1796ffd640 1 -- 172.21.15.77:0/2507901446 learned_addr learned my addr 172.21.15.77:0/2507901446 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f17977fe640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17980cad50 0x7f17980cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f17967fc640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17980cdc80 0x7f17980ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f1796ffd640 1 -- 172.21.15.77:0/2507901446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17980cad50 msgr2=0x7f17980cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f1796ffd640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17980cad50 0x7f17980cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f1796ffd640 1 -- 172.21.15.77:0/2507901446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17980cdc80 msgr2=0x7f17980ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f1796ffd640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17980cdc80 0x7f17980ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.353+0000 7f1796ffd640 1 -- 172.21.15.77:0/2507901446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17980b8bb0 con 0x7f1780003f30 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.354+0000 7f1796ffd640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f17980b7230 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f178c0119d0 tx=0x7f178c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.354+0000 7f177ffff640 1 -- 172.21.15.77:0/2507901446 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c00fb50 con 0x7f1780003f30 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.354+0000 7f177ffff640 1 -- 172.21.15.77:0/2507901446 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c00fcf0 con 0x7f1780003f30 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.354+0000 7f177ffff640 1 -- 172.21.15.77:0/2507901446 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c00fea0 con 0x7f1780003f30 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.354+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17980b7b10 con 0x7f1780003f30 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.354+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17980b80d0 con 0x7f1780003f30 2024-03-06T07:26:13.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.355+0000 7f177ffff640 1 -- 172.21.15.77:0/2507901446 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f178c012040 con 0x7f1780003f30 2024-03-06T07:26:13.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.355+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f179804f330 con 0x7f1780003f30 2024-03-06T07:26:13.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.356+0000 7f177ffff640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f178406b4d0 0x7f178406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.356+0000 7f17967fc640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f178406b4d0 0x7f178406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.356+0000 7f177ffff640 1 -- 172.21.15.77:0/2507901446 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f178c08a7e0 con 0x7f1780003f30 2024-03-06T07:26:13.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.356+0000 7f17967fc640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f178406b4d0 0x7f178406d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1788036000 tx=0x7f1788030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.360+0000 7f177ffff640 1 -- 172.21.15.77:0/2507901446 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f178c019030 con 0x7f1780003f30 2024-03-06T07:26:13.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.538+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17980d1230 con 0x7f178406b4d0 2024-03-06T07:26:13.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.544+0000 7f177ffff640 1 -- 172.21.15.77:0/2507901446 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252084 (secure 0 0 0) 0x7f17980d1230 con 0x7f178406b4d0 2024-03-06T07:26:13.548 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:13.549 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f178406b4d0 msgr2=0x7f178406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f178406b4d0 0x7f178406d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1788036000 tx=0x7f1788030000 comp rx=0 tx=0).stop 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 msgr2=0x7f17980b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f17980b7230 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f178c0119d0 tx=0x7f178c011e90 comp rx=0 tx=0).stop 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 shutdown_connections 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17980cad50 0x7f17980cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17980cdc80 0x7f17980ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f178406b4d0 0x7f178406d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 --2- 172.21.15.77:0/2507901446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1780003f30 0x7f17980b7230 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 >> 172.21.15.77:0/2507901446 conn(0x7f17980d3c80 msgr2=0x7f179815d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 shutdown_connections 2024-03-06T07:26:13.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.547+0000 7f179d913640 1 -- 172.21.15.77:0/2507901446 wait complete. 2024-03-06T07:26:13.563 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-03-06T07:26:13.429418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276232,"kb_used_data":10120,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698488,"statfs":{"total":773094113280,"available":772811251712,"internally_reserved":0,"allocated":10362880,"data_stored":5057580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.637017"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307447+0000","last_change":"2024-03-06T07:26:09.813853+0000","last_active":"2024-03-06T07:26:11.307447+0000","last_peered":"2024-03-06T07:26:11.307447+0000","last_clean":"2024-03-06T07:26:11.307447+0000","last_became_active":"2024-03-06T07:26:09.813610+0000","last_became_peered":"2024-03-06T07:26:09.813610+0000","last_unstale":"2024-03-06T07:26:11.307447+0000","last_undegraded":"2024-03-06T07:26:11.307447+0000","last_fullsized":"2024-03-06T07:26:11.307447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:31.816194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819608+0000","last_change":"2024-03-06T07:26:09.819767+0000","last_active":"2024-03-06T07:26:10.819608+0000","last_peered":"2024-03-06T07:26:10.819608+0000","last_clean":"2024-03-06T07:26:10.819608+0000","last_became_active":"2024-03-06T07:26:09.819504+0000","last_became_peered":"2024-03-06T07:26:09.819504+0000","last_unstale":"2024-03-06T07:26:10.819608+0000","last_undegraded":"2024-03-06T07:26:10.819608+0000","last_fullsized":"2024-03-06T07:26:10.819608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307400+0000","last_change":"2024-03-06T07:26:09.811069+0000","last_active":"2024-03-06T07:26:11.307400+0000","last_peered":"2024-03-06T07:26:11.307400+0000","last_clean":"2024-03-06T07:26:11.307400+0000","last_became_active":"2024-03-06T07:26:09.810813+0000","last_became_peered":"2024-03-06T07:26:09.810813+0000","last_unstale":"2024-03-06T07:26:11.307400+0000","last_undegraded":"2024-03-06T07:26:11.307400+0000","last_fullsized":"2024-03-06T07:26:11.307400+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:30.253373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804808+0000","last_change":"2024-03-06T07:26:09.816208+0000","last_active":"2024-03-06T07:26:10.804808+0000","last_peered":"2024-03-06T07:26:10.804808+0000","last_clean":"2024-03-06T07:26:10.804808+0000","last_became_active":"2024-03-06T07:26:09.815932+0000","last_became_peered":"2024-03-06T07:26:09.815932+0000","last_unstale":"2024-03-06T07:26:10.804808+0000","last_undegraded":"2024-03-06T07:26:10.804808+0000","last_fullsized":"2024-03-06T07:26:10.804808+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:19.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307777+0000","last_change":"2024-03-06T07:26:09.815170+0000","last_active":"2024-03-06T07:26:11.307777+0000","last_peered":"2024-03-06T07:26:11.307777+0000","last_clean":"2024-03-06T07:26:11.307777+0000","last_became_active":"2024-03-06T07:26:09.814899+0000","last_became_peered":"2024-03-06T07:26:09.814899+0000","last_unstale":"2024-03-06T07:26:11.307777+0000","last_undegraded":"2024-03-06T07:26:11.307777+0000","last_fullsized":"2024-03-06T07:26:11.307777+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:46.027813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802948+0000","last_change":"2024-03-06T07:26:09.819179+0000","last_active":"2024-03-06T07:26:10.802948+0000","last_peered":"2024-03-06T07:26:10.802948+0000","last_clean":"2024-03-06T07:26:10.802948+0000","last_became_active":"2024-03-06T07:26:09.818840+0000","last_became_peered":"2024-03-06T07:26:09.818840+0000","last_unstale":"2024-03-06T07:26:10.802948+0000","last_undegraded":"2024-03-06T07:26:10.802948+0000","last_fullsized":"2024-03-06T07:26:10.802948+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:17.314686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802910+0000","last_change":"2024-03-06T07:26:09.818359+0000","last_active":"2024-03-06T07:26:10.802910+0000","last_peered":"2024-03-06T07:26:10.802910+0000","last_clean":"2024-03-06T07:26:10.802910+0000","last_became_active":"2024-03-06T07:26:09.818091+0000","last_became_peered":"2024-03-06T07:26:09.818091+0000","last_unstale":"2024-03-06T07:26:10.802910+0000","last_undegraded":"2024-03-06T07:26:10.802910+0000","last_fullsized":"2024-03-06T07:26:10.802910+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:14.992810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819210+0000","last_change":"2024-03-06T07:26:09.814692+0000","last_active":"2024-03-06T07:26:10.819210+0000","last_peered":"2024-03-06T07:26:10.819210+0000","last_clean":"2024-03-06T07:26:10.819210+0000","last_became_active":"2024-03-06T07:26:09.814375+0000","last_became_peered":"2024-03-06T07:26:09.814375+0000","last_unstale":"2024-03-06T07:26:10.819210+0000","last_undegraded":"2024-03-06T07:26:10.819210+0000","last_fullsized":"2024-03-06T07:26:10.819210+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803819+0000","last_change":"2024-03-06T07:26:09.818486+0000","last_active":"2024-03-06T07:26:10.803819+0000","last_peered":"2024-03-06T07:26:10.803819+0000","last_clean":"2024-03-06T07:26:10.803819+0000","last_became_active":"2024-03-06T07:26:09.818183+0000","last_became_peered":"2024-03-06T07:26:09.818183+0000","last_unstale":"2024-03-06T07:26:10.803819+0000","last_undegraded":"2024-03-06T07:26:10.803819+0000","last_fullsized":"2024-03-06T07:26:10.803819+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:21.525054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307565+0000","last_change":"2024-03-06T07:26:09.814301+0000","last_active":"2024-03-06T07:26:11.307565+0000","last_peered":"2024-03-06T07:26:11.307565+0000","last_clean":"2024-03-06T07:26:11.307565+0000","last_became_active":"2024-03-06T07:26:09.810981+0000","last_became_peered":"2024-03-06T07:26:09.810981+0000","last_unstale":"2024-03-06T07:26:11.307565+0000","last_undegraded":"2024-03-06T07:26:11.307565+0000","last_fullsized":"2024-03-06T07:26:11.307565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:41.312503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803579+0000","last_change":"2024-03-06T07:26:09.816331+0000","last_active":"2024-03-06T07:26:10.803579+0000","last_peered":"2024-03-06T07:26:10.803579+0000","last_clean":"2024-03-06T07:26:10.803579+0000","last_became_active":"2024-03-06T07:26:09.816061+0000","last_became_peered":"2024-03-06T07:26:09.816061+0000","last_unstale":"2024-03-06T07:26:10.803579+0000","last_undegraded":"2024-03-06T07:26:10.803579+0000","last_fullsized":"2024-03-06T07:26:10.803579+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803958+0000","last_change":"2024-03-06T07:26:09.810779+0000","last_active":"2024-03-06T07:26:10.803958+0000","last_peered":"2024-03-06T07:26:10.803958+0000","last_clean":"2024-03-06T07:26:10.803958+0000","last_became_active":"2024-03-06T07:26:09.810512+0000","last_became_peered":"2024-03-06T07:26:09.810512+0000","last_unstale":"2024-03-06T07:26:10.803958+0000","last_undegraded":"2024-03-06T07:26:10.803958+0000","last_fullsized":"2024-03-06T07:26:10.803958+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:41.312503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804655+0000","last_change":"2024-03-06T07:26:09.808372+0000","last_active":"2024-03-06T07:26:10.804655+0000","last_peered":"2024-03-06T07:26:10.804655+0000","last_clean":"2024-03-06T07:26:10.804655+0000","last_became_active":"2024-03-06T07:26:09.808072+0000","last_became_peered":"2024-03-06T07:26:09.808072+0000","last_unstale":"2024-03-06T07:26:10.804655+0000","last_undegraded":"2024-03-06T07:26:10.804655+0000","last_fullsized":"2024-03-06T07:26:10.804655+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824446+0000","last_change":"2024-03-06T07:26:09.806321+0000","last_active":"2024-03-06T07:26:10.824446+0000","last_peered":"2024-03-06T07:26:10.824446+0000","last_clean":"2024-03-06T07:26:10.824446+0000","last_became_active":"2024-03-06T07:26:09.806070+0000","last_became_peered":"2024-03-06T07:26:09.806070+0000","last_unstale":"2024-03-06T07:26:10.824446+0000","last_undegraded":"2024-03-06T07:26:10.824446+0000","last_fullsized":"2024-03-06T07:26:10.824446+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:05.439904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804472+0000","last_change":"2024-03-06T07:26:09.819527+0000","last_active":"2024-03-06T07:26:10.804472+0000","last_peered":"2024-03-06T07:26:10.804472+0000","last_clean":"2024-03-06T07:26:10.804472+0000","last_became_active":"2024-03-06T07:26:09.819257+0000","last_became_peered":"2024-03-06T07:26:09.819257+0000","last_unstale":"2024-03-06T07:26:10.804472+0000","last_undegraded":"2024-03-06T07:26:10.804472+0000","last_fullsized":"2024-03-06T07:26:10.804472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:17.314686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803146+0000","last_change":"2024-03-06T07:26:09.817278+0000","last_active":"2024-03-06T07:26:10.803146+0000","last_peered":"2024-03-06T07:26:10.803146+0000","last_clean":"2024-03-06T07:26:10.803146+0000","last_became_active":"2024-03-06T07:26:09.817015+0000","last_became_peered":"2024-03-06T07:26:09.817015+0000","last_unstale":"2024-03-06T07:26:10.803146+0000","last_undegraded":"2024-03-06T07:26:10.803146+0000","last_fullsized":"2024-03-06T07:26:10.803146+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:33.879869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819707+0000","last_change":"2024-03-06T07:26:09.824514+0000","last_active":"2024-03-06T07:26:10.819707+0000","last_peered":"2024-03-06T07:26:10.819707+0000","last_clean":"2024-03-06T07:26:10.819707+0000","last_became_active":"2024-03-06T07:26:09.823591+0000","last_became_peered":"2024-03-06T07:26:09.823591+0000","last_unstale":"2024-03-06T07:26:10.819707+0000","last_undegraded":"2024-03-06T07:26:10.819707+0000","last_fullsized":"2024-03-06T07:26:10.819707+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.005755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804704+0000","last_change":"2024-03-06T07:26:09.811053+0000","last_active":"2024-03-06T07:26:10.804704+0000","last_peered":"2024-03-06T07:26:10.804704+0000","last_clean":"2024-03-06T07:26:10.804704+0000","last_became_active":"2024-03-06T07:26:09.810805+0000","last_became_peered":"2024-03-06T07:26:09.810805+0000","last_unstale":"2024-03-06T07:26:10.804704+0000","last_undegraded":"2024-03-06T07:26:10.804704+0000","last_fullsized":"2024-03-06T07:26:10.804704+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824872+0000","last_change":"2024-03-06T07:26:09.809874+0000","last_active":"2024-03-06T07:26:10.824872+0000","last_peered":"2024-03-06T07:26:10.824872+0000","last_clean":"2024-03-06T07:26:10.824872+0000","last_became_active":"2024-03-06T07:26:09.809601+0000","last_became_peered":"2024-03-06T07:26:09.809601+0000","last_unstale":"2024-03-06T07:26:10.824872+0000","last_undegraded":"2024-03-06T07:26:10.824872+0000","last_fullsized":"2024-03-06T07:26:10.824872+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:56.986034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803542+0000","last_change":"2024-03-06T07:26:09.819152+0000","last_active":"2024-03-06T07:26:10.803542+0000","last_peered":"2024-03-06T07:26:10.803542+0000","last_clean":"2024-03-06T07:26:10.803542+0000","last_became_active":"2024-03-06T07:26:09.818889+0000","last_became_peered":"2024-03-06T07:26:09.818889+0000","last_unstale":"2024-03-06T07:26:10.803542+0000","last_undegraded":"2024-03-06T07:26:10.803542+0000","last_fullsized":"2024-03-06T07:26:10.803542+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:49.745227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824557+0000","last_change":"2024-03-06T07:26:09.815573+0000","last_active":"2024-03-06T07:26:10.824557+0000","last_peered":"2024-03-06T07:26:10.824557+0000","last_clean":"2024-03-06T07:26:10.824557+0000","last_became_active":"2024-03-06T07:26:09.815163+0000","last_became_peered":"2024-03-06T07:26:09.815163+0000","last_unstale":"2024-03-06T07:26:10.824557+0000","last_undegraded":"2024-03-06T07:26:10.824557+0000","last_fullsized":"2024-03-06T07:26:10.824557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:08.603270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819256+0000","last_change":"2024-03-06T07:26:09.820441+0000","last_active":"2024-03-06T07:26:10.819256+0000","last_peered":"2024-03-06T07:26:10.819256+0000","last_clean":"2024-03-06T07:26:10.819256+0000","last_became_active":"2024-03-06T07:26:09.820143+0000","last_became_peered":"2024-03-06T07:26:09.820143+0000","last_unstale":"2024-03-06T07:26:10.819256+0000","last_undegraded":"2024-03-06T07:26:10.819256+0000","last_fullsized":"2024-03-06T07:26:10.819256+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819199+0000","last_change":"2024-03-06T07:26:09.819137+0000","last_active":"2024-03-06T07:26:10.819199+0000","last_peered":"2024-03-06T07:26:10.819199+0000","last_clean":"2024-03-06T07:26:10.819199+0000","last_became_active":"2024-03-06T07:26:09.818884+0000","last_became_peered":"2024-03-06T07:26:09.818884+0000","last_unstale":"2024-03-06T07:26:10.819199+0000","last_undegraded":"2024-03-06T07:26:10.819199+0000","last_fullsized":"2024-03-06T07:26:10.819199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803186+0000","last_change":"2024-03-06T07:26:09.815241+0000","last_active":"2024-03-06T07:26:10.803186+0000","last_peered":"2024-03-06T07:26:10.803186+0000","last_clean":"2024-03-06T07:26:10.803186+0000","last_became_active":"2024-03-06T07:26:09.814964+0000","last_became_peered":"2024-03-06T07:26:09.814964+0000","last_unstale":"2024-03-06T07:26:10.803186+0000","last_undegraded":"2024-03-06T07:26:10.803186+0000","last_fullsized":"2024-03-06T07:26:10.803186+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307953+0000","last_change":"2024-03-06T07:26:09.816181+0000","last_active":"2024-03-06T07:26:11.307953+0000","last_peered":"2024-03-06T07:26:11.307953+0000","last_clean":"2024-03-06T07:26:11.307953+0000","last_became_active":"2024-03-06T07:26:09.815917+0000","last_became_peered":"2024-03-06T07:26:09.815917+0000","last_unstale":"2024-03-06T07:26:11.307953+0000","last_undegraded":"2024-03-06T07:26:11.307953+0000","last_fullsized":"2024-03-06T07:26:11.307953+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:41.338253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819765+0000","last_change":"2024-03-06T07:26:09.821323+0000","last_active":"2024-03-06T07:26:10.819765+0000","last_peered":"2024-03-06T07:26:10.819765+0000","last_clean":"2024-03-06T07:26:10.819765+0000","last_became_active":"2024-03-06T07:26:09.821041+0000","last_became_peered":"2024-03-06T07:26:09.821041+0000","last_unstale":"2024-03-06T07:26:10.819765+0000","last_undegraded":"2024-03-06T07:26:10.819765+0000","last_fullsized":"2024-03-06T07:26:10.819765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:11.188598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803635+0000","last_change":"2024-03-06T07:26:09.812299+0000","last_active":"2024-03-06T07:26:10.803635+0000","last_peered":"2024-03-06T07:26:10.803635+0000","last_clean":"2024-03-06T07:26:10.803635+0000","last_became_active":"2024-03-06T07:26:09.812009+0000","last_became_peered":"2024-03-06T07:26:09.812009+0000","last_unstale":"2024-03-06T07:26:10.803635+0000","last_undegraded":"2024-03-06T07:26:10.803635+0000","last_fullsized":"2024-03-06T07:26:10.803635+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307574+0000","last_change":"2024-03-06T07:26:09.821191+0000","last_active":"2024-03-06T07:26:11.307574+0000","last_peered":"2024-03-06T07:26:11.307574+0000","last_clean":"2024-03-06T07:26:11.307574+0000","last_became_active":"2024-03-06T07:26:09.820905+0000","last_became_peered":"2024-03-06T07:26:09.820905+0000","last_unstale":"2024-03-06T07:26:11.307574+0000","last_undegraded":"2024-03-06T07:26:11.307574+0000","last_fullsized":"2024-03-06T07:26:11.307574+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:27.201988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819336+0000","last_change":"2024-03-06T07:26:09.816456+0000","last_active":"2024-03-06T07:26:10.819336+0000","last_peered":"2024-03-06T07:26:10.819336+0000","last_clean":"2024-03-06T07:26:10.819336+0000","last_became_active":"2024-03-06T07:26:09.816206+0000","last_became_peered":"2024-03-06T07:26:09.816206+0000","last_unstale":"2024-03-06T07:26:10.819336+0000","last_undegraded":"2024-03-06T07:26:10.819336+0000","last_fullsized":"2024-03-06T07:26:10.819336+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824471+0000","last_change":"2024-03-06T07:26:09.808861+0000","last_active":"2024-03-06T07:26:10.824471+0000","last_peered":"2024-03-06T07:26:10.824471+0000","last_clean":"2024-03-06T07:26:10.824471+0000","last_became_active":"2024-03-06T07:26:09.808345+0000","last_became_peered":"2024-03-06T07:26:09.808345+0000","last_unstale":"2024-03-06T07:26:10.824471+0000","last_undegraded":"2024-03-06T07:26:10.824471+0000","last_fullsized":"2024-03-06T07:26:10.824471+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:15.906643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804539+0000","last_change":"2024-03-06T07:26:09.821038+0000","last_active":"2024-03-06T07:26:10.804539+0000","last_peered":"2024-03-06T07:26:10.804539+0000","last_clean":"2024-03-06T07:26:10.804539+0000","last_became_active":"2024-03-06T07:26:09.820749+0000","last_became_peered":"2024-03-06T07:26:09.820749+0000","last_unstale":"2024-03-06T07:26:10.804539+0000","last_undegraded":"2024-03-06T07:26:10.804539+0000","last_fullsized":"2024-03-06T07:26:10.804539+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307065+0000","last_change":"2024-03-06T07:26:09.815360+0000","last_active":"2024-03-06T07:26:11.307065+0000","last_peered":"2024-03-06T07:26:11.307065+0000","last_clean":"2024-03-06T07:26:11.307065+0000","last_became_active":"2024-03-06T07:26:09.815125+0000","last_became_peered":"2024-03-06T07:26:09.815125+0000","last_unstale":"2024-03-06T07:26:11.307065+0000","last_undegraded":"2024-03-06T07:26:11.307065+0000","last_fullsized":"2024-03-06T07:26:11.307065+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:58.101050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824887+0000","last_change":"2024-03-06T07:26:09.809611+0000","last_active":"2024-03-06T07:26:10.824887+0000","last_peered":"2024-03-06T07:26:10.824887+0000","last_clean":"2024-03-06T07:26:10.824887+0000","last_became_active":"2024-03-06T07:26:09.809343+0000","last_became_peered":"2024-03-06T07:26:09.809343+0000","last_unstale":"2024-03-06T07:26:10.824887+0000","last_undegraded":"2024-03-06T07:26:10.824887+0000","last_fullsized":"2024-03-06T07:26:10.824887+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:48.800992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819712+0000","last_change":"2024-03-06T07:26:08.809604+0000","last_active":"2024-03-06T07:26:10.819712+0000","last_peered":"2024-03-06T07:26:10.819712+0000","last_clean":"2024-03-06T07:26:10.819712+0000","last_became_active":"2024-03-06T07:26:08.809360+0000","last_became_peered":"2024-03-06T07:26:08.809360+0000","last_unstale":"2024-03-06T07:26:10.819712+0000","last_undegraded":"2024-03-06T07:26:10.819712+0000","last_fullsized":"2024-03-06T07:26:10.819712+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:10.336592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804780+0000","last_change":"2024-03-06T07:26:09.822258+0000","last_active":"2024-03-06T07:26:10.804780+0000","last_peered":"2024-03-06T07:26:10.804780+0000","last_clean":"2024-03-06T07:26:10.804780+0000","last_became_active":"2024-03-06T07:26:09.822015+0000","last_became_peered":"2024-03-06T07:26:09.822015+0000","last_unstale":"2024-03-06T07:26:10.804780+0000","last_undegraded":"2024-03-06T07:26:10.804780+0000","last_fullsized":"2024-03-06T07:26:10.804780+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803187+0000","last_change":"2024-03-06T07:26:08.812691+0000","last_active":"2024-03-06T07:26:10.803187+0000","last_peered":"2024-03-06T07:26:10.803187+0000","last_clean":"2024-03-06T07:26:10.803187+0000","last_became_active":"2024-03-06T07:26:08.812418+0000","last_became_peered":"2024-03-06T07:26:08.812418+0000","last_unstale":"2024-03-06T07:26:10.803187+0000","last_undegraded":"2024-03-06T07:26:10.803187+0000","last_fullsized":"2024-03-06T07:26:10.803187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:20.520632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307299+0000","last_change":"2024-03-06T07:26:09.807945+0000","last_active":"2024-03-06T07:26:11.307299+0000","last_peered":"2024-03-06T07:26:11.307299+0000","last_clean":"2024-03-06T07:26:11.307299+0000","last_became_active":"2024-03-06T07:26:09.807684+0000","last_became_peered":"2024-03-06T07:26:09.807684+0000","last_unstale":"2024-03-06T07:26:11.307299+0000","last_undegraded":"2024-03-06T07:26:11.307299+0000","last_fullsized":"2024-03-06T07:26:11.307299+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:13.507677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819496+0000","last_change":"2024-03-06T07:26:08.816145+0000","last_active":"2024-03-06T07:26:10.819496+0000","last_peered":"2024-03-06T07:26:10.819496+0000","last_clean":"2024-03-06T07:26:10.819496+0000","last_became_active":"2024-03-06T07:26:08.815639+0000","last_became_peered":"2024-03-06T07:26:08.815639+0000","last_unstale":"2024-03-06T07:26:10.819496+0000","last_undegraded":"2024-03-06T07:26:10.819496+0000","last_fullsized":"2024-03-06T07:26:10.819496+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:47.753067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803830+0000","last_change":"2024-03-06T07:26:09.814294+0000","last_active":"2024-03-06T07:26:10.803830+0000","last_peered":"2024-03-06T07:26:10.803830+0000","last_clean":"2024-03-06T07:26:10.803830+0000","last_became_active":"2024-03-06T07:26:09.814010+0000","last_became_peered":"2024-03-06T07:26:09.814010+0000","last_unstale":"2024-03-06T07:26:10.803830+0000","last_undegraded":"2024-03-06T07:26:10.803830+0000","last_fullsized":"2024-03-06T07:26:10.803830+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:37.893304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819440+0000","last_change":"2024-03-06T07:26:08.814364+0000","last_active":"2024-03-06T07:26:10.819440+0000","last_peered":"2024-03-06T07:26:10.819440+0000","last_clean":"2024-03-06T07:26:10.819440+0000","last_became_active":"2024-03-06T07:26:08.813668+0000","last_became_peered":"2024-03-06T07:26:08.813668+0000","last_unstale":"2024-03-06T07:26:10.819440+0000","last_undegraded":"2024-03-06T07:26:10.819440+0000","last_fullsized":"2024-03-06T07:26:10.819440+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:48.740805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307795+0000","last_change":"2024-03-06T07:26:09.812757+0000","last_active":"2024-03-06T07:26:11.307795+0000","last_peered":"2024-03-06T07:26:11.307795+0000","last_clean":"2024-03-06T07:26:11.307795+0000","last_became_active":"2024-03-06T07:26:09.812460+0000","last_became_peered":"2024-03-06T07:26:09.812460+0000","last_unstale":"2024-03-06T07:26:11.307795+0000","last_undegraded":"2024-03-06T07:26:11.307795+0000","last_fullsized":"2024-03-06T07:26:11.307795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:58.727545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307231+0000","last_change":"2024-03-06T07:26:08.809284+0000","last_active":"2024-03-06T07:26:11.307231+0000","last_peered":"2024-03-06T07:26:11.307231+0000","last_clean":"2024-03-06T07:26:11.307231+0000","last_became_active":"2024-03-06T07:26:08.809034+0000","last_became_peered":"2024-03-06T07:26:08.809034+0000","last_unstale":"2024-03-06T07:26:11.307231+0000","last_undegraded":"2024-03-06T07:26:11.307231+0000","last_fullsized":"2024-03-06T07:26:11.307231+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:07.598849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802744+0000","last_change":"2024-03-06T07:26:09.813345+0000","last_active":"2024-03-06T07:26:10.802744+0000","last_peered":"2024-03-06T07:26:10.802744+0000","last_clean":"2024-03-06T07:26:10.802744+0000","last_became_active":"2024-03-06T07:26:09.813115+0000","last_became_peered":"2024-03-06T07:26:09.813115+0000","last_unstale":"2024-03-06T07:26:10.802744+0000","last_undegraded":"2024-03-06T07:26:10.802744+0000","last_fullsized":"2024-03-06T07:26:10.802744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:04.605670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819254+0000","last_change":"2024-03-06T07:26:08.810941+0000","last_active":"2024-03-06T07:26:10.819254+0000","last_peered":"2024-03-06T07:26:10.819254+0000","last_clean":"2024-03-06T07:26:10.819254+0000","last_became_active":"2024-03-06T07:26:08.810678+0000","last_became_peered":"2024-03-06T07:26:08.810678+0000","last_unstale":"2024-03-06T07:26:10.819254+0000","last_undegraded":"2024-03-06T07:26:10.819254+0000","last_fullsized":"2024-03-06T07:26:10.819254+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:49.528401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804522+0000","last_change":"2024-03-06T07:26:09.820600+0000","last_active":"2024-03-06T07:26:10.804522+0000","last_peered":"2024-03-06T07:26:10.804522+0000","last_clean":"2024-03-06T07:26:10.804522+0000","last_became_active":"2024-03-06T07:26:09.820397+0000","last_became_peered":"2024-03-06T07:26:09.820397+0000","last_unstale":"2024-03-06T07:26:10.804522+0000","last_undegraded":"2024-03-06T07:26:10.804522+0000","last_fullsized":"2024-03-06T07:26:10.804522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:33.879869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308023+0000","last_change":"2024-03-06T07:26:08.810527+0000","last_active":"2024-03-06T07:26:11.308023+0000","last_peered":"2024-03-06T07:26:11.308023+0000","last_clean":"2024-03-06T07:26:11.308023+0000","last_became_active":"2024-03-06T07:26:08.810064+0000","last_became_peered":"2024-03-06T07:26:08.810064+0000","last_unstale":"2024-03-06T07:26:11.308023+0000","last_undegraded":"2024-03-06T07:26:11.308023+0000","last_fullsized":"2024-03-06T07:26:11.308023+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:57.723123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802744+0000","last_change":"2024-03-06T07:26:09.817490+0000","last_active":"2024-03-06T07:26:10.802744+0000","last_peered":"2024-03-06T07:26:10.802744+0000","last_clean":"2024-03-06T07:26:10.802744+0000","last_became_active":"2024-03-06T07:26:09.817201+0000","last_became_peered":"2024-03-06T07:26:09.817201+0000","last_unstale":"2024-03-06T07:26:10.802744+0000","last_undegraded":"2024-03-06T07:26:10.802744+0000","last_fullsized":"2024-03-06T07:26:10.802744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:09.736601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307007+0000","last_change":"2024-03-06T07:26:08.813560+0000","last_active":"2024-03-06T07:26:11.307007+0000","last_peered":"2024-03-06T07:26:11.307007+0000","last_clean":"2024-03-06T07:26:11.307007+0000","last_became_active":"2024-03-06T07:26:08.813280+0000","last_became_peered":"2024-03-06T07:26:08.813280+0000","last_unstale":"2024-03-06T07:26:11.307007+0000","last_undegraded":"2024-03-06T07:26:11.307007+0000","last_fullsized":"2024-03-06T07:26:11.307007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:18.986097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819607+0000","last_change":"2024-03-06T07:26:09.815672+0000","last_active":"2024-03-06T07:26:10.819607+0000","last_peered":"2024-03-06T07:26:10.819607+0000","last_clean":"2024-03-06T07:26:10.819607+0000","last_became_active":"2024-03-06T07:26:09.815424+0000","last_became_peered":"2024-03-06T07:26:09.815424+0000","last_unstale":"2024-03-06T07:26:10.819607+0000","last_undegraded":"2024-03-06T07:26:10.819607+0000","last_fullsized":"2024-03-06T07:26:10.819607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:19.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307510+0000","last_change":"2024-03-06T07:26:08.810875+0000","last_active":"2024-03-06T07:26:11.307510+0000","last_peered":"2024-03-06T07:26:11.307510+0000","last_clean":"2024-03-06T07:26:11.307510+0000","last_became_active":"2024-03-06T07:26:08.810627+0000","last_became_peered":"2024-03-06T07:26:08.810627+0000","last_unstale":"2024-03-06T07:26:11.307510+0000","last_undegraded":"2024-03-06T07:26:11.307510+0000","last_fullsized":"2024-03-06T07:26:11.307510+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:47.815489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803882+0000","last_change":"2024-03-06T07:26:09.813845+0000","last_active":"2024-03-06T07:26:10.803882+0000","last_peered":"2024-03-06T07:26:10.803882+0000","last_clean":"2024-03-06T07:26:10.803882+0000","last_became_active":"2024-03-06T07:26:09.813558+0000","last_became_peered":"2024-03-06T07:26:09.813558+0000","last_unstale":"2024-03-06T07:26:10.803882+0000","last_undegraded":"2024-03-06T07:26:10.803882+0000","last_fullsized":"2024-03-06T07:26:10.803882+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:11.341014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804612+0000","last_change":"2024-03-06T07:26:08.811996+0000","last_active":"2024-03-06T07:26:10.804612+0000","last_peered":"2024-03-06T07:26:10.804612+0000","last_clean":"2024-03-06T07:26:10.804612+0000","last_became_active":"2024-03-06T07:26:08.811309+0000","last_became_peered":"2024-03-06T07:26:08.811309+0000","last_unstale":"2024-03-06T07:26:10.804612+0000","last_undegraded":"2024-03-06T07:26:10.804612+0000","last_fullsized":"2024-03-06T07:26:10.804612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:49.528401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307703+0000","last_change":"2024-03-06T07:26:09.817032+0000","last_active":"2024-03-06T07:26:11.307703+0000","last_peered":"2024-03-06T07:26:11.307703+0000","last_clean":"2024-03-06T07:26:11.307703+0000","last_became_active":"2024-03-06T07:26:09.816744+0000","last_became_peered":"2024-03-06T07:26:09.816744+0000","last_unstale":"2024-03-06T07:26:11.307703+0000","last_undegraded":"2024-03-06T07:26:11.307703+0000","last_fullsized":"2024-03-06T07:26:11.307703+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:50.532823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803867+0000","last_change":"2024-03-06T07:26:08.816283+0000","last_active":"2024-03-06T07:26:10.803867+0000","last_peered":"2024-03-06T07:26:10.803867+0000","last_clean":"2024-03-06T07:26:10.803867+0000","last_became_active":"2024-03-06T07:26:08.815958+0000","last_became_peered":"2024-03-06T07:26:08.815958+0000","last_unstale":"2024-03-06T07:26:10.803867+0000","last_undegraded":"2024-03-06T07:26:10.803867+0000","last_fullsized":"2024-03-06T07:26:10.803867+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:10.336592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803747+0000","last_change":"2024-03-06T07:26:09.817122+0000","last_active":"2024-03-06T07:26:10.803747+0000","last_peered":"2024-03-06T07:26:10.803747+0000","last_clean":"2024-03-06T07:26:10.803747+0000","last_became_active":"2024-03-06T07:26:09.816870+0000","last_became_peered":"2024-03-06T07:26:09.816870+0000","last_unstale":"2024-03-06T07:26:10.803747+0000","last_undegraded":"2024-03-06T07:26:10.803747+0000","last_fullsized":"2024-03-06T07:26:10.803747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803651+0000","last_change":"2024-03-06T07:26:08.813358+0000","last_active":"2024-03-06T07:26:10.803651+0000","last_peered":"2024-03-06T07:26:10.803651+0000","last_clean":"2024-03-06T07:26:10.803651+0000","last_became_active":"2024-03-06T07:26:08.813065+0000","last_became_peered":"2024-03-06T07:26:08.813065+0000","last_unstale":"2024-03-06T07:26:10.803651+0000","last_undegraded":"2024-03-06T07:26:10.803651+0000","last_fullsized":"2024-03-06T07:26:10.803651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:01.075606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804603+0000","last_change":"2024-03-06T07:26:09.818846+0000","last_active":"2024-03-06T07:26:10.804603+0000","last_peered":"2024-03-06T07:26:10.804603+0000","last_clean":"2024-03-06T07:26:10.804603+0000","last_became_active":"2024-03-06T07:26:09.818585+0000","last_became_peered":"2024-03-06T07:26:09.818585+0000","last_unstale":"2024-03-06T07:26:10.804603+0000","last_undegraded":"2024-03-06T07:26:10.804603+0000","last_fullsized":"2024-03-06T07:26:10.804603+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824696+0000","last_change":"2024-03-06T07:26:08.808880+0000","last_active":"2024-03-06T07:26:10.824696+0000","last_peered":"2024-03-06T07:26:10.824696+0000","last_clean":"2024-03-06T07:26:10.824696+0000","last_became_active":"2024-03-06T07:26:08.806760+0000","last_became_peered":"2024-03-06T07:26:08.806760+0000","last_unstale":"2024-03-06T07:26:10.824696+0000","last_undegraded":"2024-03-06T07:26:10.824696+0000","last_fullsized":"2024-03-06T07:26:10.824696+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:25.383967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307252+0000","last_change":"2024-03-06T07:26:09.811502+0000","last_active":"2024-03-06T07:26:11.307252+0000","last_peered":"2024-03-06T07:26:11.307252+0000","last_clean":"2024-03-06T07:26:11.307252+0000","last_became_active":"2024-03-06T07:26:09.811267+0000","last_became_peered":"2024-03-06T07:26:09.811267+0000","last_unstale":"2024-03-06T07:26:11.307252+0000","last_undegraded":"2024-03-06T07:26:11.307252+0000","last_fullsized":"2024-03-06T07:26:11.307252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:47.984176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824775+0000","last_change":"2024-03-06T07:26:08.810238+0000","last_active":"2024-03-06T07:26:10.824775+0000","last_peered":"2024-03-06T07:26:10.824775+0000","last_clean":"2024-03-06T07:26:10.824775+0000","last_became_active":"2024-03-06T07:26:08.809953+0000","last_became_peered":"2024-03-06T07:26:08.809953+0000","last_unstale":"2024-03-06T07:26:10.824775+0000","last_undegraded":"2024-03-06T07:26:10.824775+0000","last_fullsized":"2024-03-06T07:26:10.824775+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:52.715807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802768+0000","last_change":"2024-03-06T07:26:09.818405+0000","last_active":"2024-03-06T07:26:10.802768+0000","last_peered":"2024-03-06T07:26:10.802768+0000","last_clean":"2024-03-06T07:26:10.802768+0000","last_became_active":"2024-03-06T07:26:09.818117+0000","last_became_peered":"2024-03-06T07:26:09.818117+0000","last_unstale":"2024-03-06T07:26:10.802768+0000","last_undegraded":"2024-03-06T07:26:10.802768+0000","last_fullsized":"2024-03-06T07:26:10.802768+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:48.757489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803133+0000","last_change":"2024-03-06T07:26:08.806983+0000","last_active":"2024-03-06T07:26:10.803133+0000","last_peered":"2024-03-06T07:26:10.803133+0000","last_clean":"2024-03-06T07:26:10.803133+0000","last_became_active":"2024-03-06T07:26:08.806690+0000","last_became_peered":"2024-03-06T07:26:08.806690+0000","last_unstale":"2024-03-06T07:26:10.803133+0000","last_undegraded":"2024-03-06T07:26:10.803133+0000","last_fullsized":"2024-03-06T07:26:10.803133+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:18.622715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824443+0000","last_change":"2024-03-06T07:26:09.807025+0000","last_active":"2024-03-06T07:26:10.824443+0000","last_peered":"2024-03-06T07:26:10.824443+0000","last_clean":"2024-03-06T07:26:10.824443+0000","last_became_active":"2024-03-06T07:26:09.806758+0000","last_became_peered":"2024-03-06T07:26:09.806758+0000","last_unstale":"2024-03-06T07:26:10.824443+0000","last_undegraded":"2024-03-06T07:26:10.824443+0000","last_fullsized":"2024-03-06T07:26:10.824443+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:02.489555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307111+0000","last_change":"2024-03-06T07:26:08.809055+0000","last_active":"2024-03-06T07:26:11.307111+0000","last_peered":"2024-03-06T07:26:11.307111+0000","last_clean":"2024-03-06T07:26:11.307111+0000","last_became_active":"2024-03-06T07:26:08.808747+0000","last_became_peered":"2024-03-06T07:26:08.808747+0000","last_unstale":"2024-03-06T07:26:11.307111+0000","last_undegraded":"2024-03-06T07:26:11.307111+0000","last_fullsized":"2024-03-06T07:26:11.307111+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:58.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307875+0000","last_change":"2024-03-06T07:26:09.809108+0000","last_active":"2024-03-06T07:26:11.307875+0000","last_peered":"2024-03-06T07:26:11.307875+0000","last_clean":"2024-03-06T07:26:11.307875+0000","last_became_active":"2024-03-06T07:26:09.808845+0000","last_became_peered":"2024-03-06T07:26:09.808845+0000","last_unstale":"2024-03-06T07:26:11.307875+0000","last_undegraded":"2024-03-06T07:26:11.307875+0000","last_fullsized":"2024-03-06T07:26:11.307875+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:11.993809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803566+0000","last_change":"2024-03-06T07:26:08.814165+0000","last_active":"2024-03-06T07:26:10.803566+0000","last_peered":"2024-03-06T07:26:10.803566+0000","last_clean":"2024-03-06T07:26:10.803566+0000","last_became_active":"2024-03-06T07:26:08.813542+0000","last_became_peered":"2024-03-06T07:26:08.813542+0000","last_unstale":"2024-03-06T07:26:10.803566+0000","last_undegraded":"2024-03-06T07:26:10.803566+0000","last_fullsized":"2024-03-06T07:26:10.803566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:37.062164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307836+0000","last_change":"2024-03-06T07:26:09.816662+0000","last_active":"2024-03-06T07:26:11.307836+0000","last_peered":"2024-03-06T07:26:11.307836+0000","last_clean":"2024-03-06T07:26:11.307836+0000","last_became_active":"2024-03-06T07:26:09.816408+0000","last_became_peered":"2024-03-06T07:26:09.816408+0000","last_unstale":"2024-03-06T07:26:11.307836+0000","last_undegraded":"2024-03-06T07:26:11.307836+0000","last_fullsized":"2024-03-06T07:26:11.307836+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:20.536277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307890+0000","last_change":"2024-03-06T07:26:08.810040+0000","last_active":"2024-03-06T07:26:11.307890+0000","last_peered":"2024-03-06T07:26:11.307890+0000","last_clean":"2024-03-06T07:26:11.307890+0000","last_became_active":"2024-03-06T07:26:08.809759+0000","last_became_peered":"2024-03-06T07:26:08.809759+0000","last_unstale":"2024-03-06T07:26:11.307890+0000","last_undegraded":"2024-03-06T07:26:11.307890+0000","last_fullsized":"2024-03-06T07:26:11.307890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:57.534027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803805+0000","last_change":"2024-03-06T07:26:09.817961+0000","last_active":"2024-03-06T07:26:10.803805+0000","last_peered":"2024-03-06T07:26:10.803805+0000","last_clean":"2024-03-06T07:26:10.803805+0000","last_became_active":"2024-03-06T07:26:09.817654+0000","last_became_peered":"2024-03-06T07:26:09.817654+0000","last_unstale":"2024-03-06T07:26:10.803805+0000","last_undegraded":"2024-03-06T07:26:10.803805+0000","last_fullsized":"2024-03-06T07:26:10.803805+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819490+0000","last_change":"2024-03-06T07:26:08.813115+0000","last_active":"2024-03-06T07:26:10.819490+0000","last_peered":"2024-03-06T07:26:10.819490+0000","last_clean":"2024-03-06T07:26:10.819490+0000","last_became_active":"2024-03-06T07:26:08.812857+0000","last_became_peered":"2024-03-06T07:26:08.812857+0000","last_unstale":"2024-03-06T07:26:10.819490+0000","last_undegraded":"2024-03-06T07:26:10.819490+0000","last_fullsized":"2024-03-06T07:26:10.819490+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:20.520632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824601+0000","last_change":"2024-03-06T07:26:09.811923+0000","last_active":"2024-03-06T07:26:10.824601+0000","last_peered":"2024-03-06T07:26:10.824601+0000","last_clean":"2024-03-06T07:26:10.824601+0000","last_became_active":"2024-03-06T07:26:09.811651+0000","last_became_peered":"2024-03-06T07:26:09.811651+0000","last_unstale":"2024-03-06T07:26:10.824601+0000","last_undegraded":"2024-03-06T07:26:10.824601+0000","last_fullsized":"2024-03-06T07:26:10.824601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:48.819911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819358+0000","last_change":"2024-03-06T07:26:08.816961+0000","last_active":"2024-03-06T07:26:10.819358+0000","last_peered":"2024-03-06T07:26:10.819358+0000","last_clean":"2024-03-06T07:26:10.819358+0000","last_became_active":"2024-03-06T07:26:08.816694+0000","last_became_peered":"2024-03-06T07:26:08.816694+0000","last_unstale":"2024-03-06T07:26:10.819358+0000","last_undegraded":"2024-03-06T07:26:10.819358+0000","last_fullsized":"2024-03-06T07:26:10.819358+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:40.308081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824737+0000","last_change":"2024-03-06T07:26:09.817656+0000","last_active":"2024-03-06T07:26:10.824737+0000","last_peered":"2024-03-06T07:26:10.824737+0000","last_clean":"2024-03-06T07:26:10.824737+0000","last_became_active":"2024-03-06T07:26:09.817006+0000","last_became_peered":"2024-03-06T07:26:09.817006+0000","last_unstale":"2024-03-06T07:26:10.824737+0000","last_undegraded":"2024-03-06T07:26:10.824737+0000","last_fullsized":"2024-03-06T07:26:10.824737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:19.990519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307731+0000","last_change":"2024-03-06T07:26:08.814964+0000","last_active":"2024-03-06T07:26:11.307731+0000","last_peered":"2024-03-06T07:26:11.307731+0000","last_clean":"2024-03-06T07:26:11.307731+0000","last_became_active":"2024-03-06T07:26:08.814713+0000","last_became_peered":"2024-03-06T07:26:08.814713+0000","last_unstale":"2024-03-06T07:26:11.307731+0000","last_undegraded":"2024-03-06T07:26:11.307731+0000","last_fullsized":"2024-03-06T07:26:11.307731+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:41.581348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824464+0000","last_change":"2024-03-06T07:26:09.811939+0000","last_active":"2024-03-06T07:26:10.824464+0000","last_peered":"2024-03-06T07:26:10.824464+0000","last_clean":"2024-03-06T07:26:10.824464+0000","last_became_active":"2024-03-06T07:26:09.811667+0000","last_became_peered":"2024-03-06T07:26:09.811667+0000","last_unstale":"2024-03-06T07:26:10.824464+0000","last_undegraded":"2024-03-06T07:26:10.824464+0000","last_fullsized":"2024-03-06T07:26:10.824464+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:46.742874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307630+0000","last_change":"2024-03-06T07:26:08.813568+0000","last_active":"2024-03-06T07:26:11.307630+0000","last_peered":"2024-03-06T07:26:11.307630+0000","last_clean":"2024-03-06T07:26:11.307630+0000","last_became_active":"2024-03-06T07:26:08.813267+0000","last_became_peered":"2024-03-06T07:26:08.813267+0000","last_unstale":"2024-03-06T07:26:11.307630+0000","last_undegraded":"2024-03-06T07:26:11.307630+0000","last_fullsized":"2024-03-06T07:26:11.307630+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:18.622715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308066+0000","last_change":"2024-03-06T07:26:09.806942+0000","last_active":"2024-03-06T07:26:11.308066+0000","last_peered":"2024-03-06T07:26:11.308066+0000","last_clean":"2024-03-06T07:26:11.308066+0000","last_became_active":"2024-03-06T07:26:09.806694+0000","last_became_peered":"2024-03-06T07:26:09.806694+0000","last_unstale":"2024-03-06T07:26:11.308066+0000","last_undegraded":"2024-03-06T07:26:11.308066+0000","last_fullsized":"2024-03-06T07:26:11.308066+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:02.664185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819198+0000","last_change":"2024-03-06T07:26:08.810885+0000","last_active":"2024-03-06T07:26:10.819198+0000","last_peered":"2024-03-06T07:26:10.819198+0000","last_clean":"2024-03-06T07:26:10.819198+0000","last_became_active":"2024-03-06T07:26:08.810631+0000","last_became_peered":"2024-03-06T07:26:08.810631+0000","last_unstale":"2024-03-06T07:26:10.819198+0000","last_undegraded":"2024-03-06T07:26:10.819198+0000","last_fullsized":"2024-03-06T07:26:10.819198+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:08.732180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819747+0000","last_change":"2024-03-06T07:26:09.819782+0000","last_active":"2024-03-06T07:26:10.819747+0000","last_peered":"2024-03-06T07:26:10.819747+0000","last_clean":"2024-03-06T07:26:10.819747+0000","last_became_active":"2024-03-06T07:26:09.819538+0000","last_became_peered":"2024-03-06T07:26:09.819538+0000","last_unstale":"2024-03-06T07:26:10.819747+0000","last_undegraded":"2024-03-06T07:26:10.819747+0000","last_fullsized":"2024-03-06T07:26:10.819747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:50.283608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308188+0000","last_change":"2024-03-06T07:26:08.813738+0000","last_active":"2024-03-06T07:26:11.308188+0000","last_peered":"2024-03-06T07:26:11.308188+0000","last_clean":"2024-03-06T07:26:11.308188+0000","last_became_active":"2024-03-06T07:26:08.813416+0000","last_became_peered":"2024-03-06T07:26:08.813416+0000","last_unstale":"2024-03-06T07:26:11.308188+0000","last_undegraded":"2024-03-06T07:26:11.308188+0000","last_fullsized":"2024-03-06T07:26:11.308188+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:01.075606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307449+0000","last_change":"2024-03-06T07:26:09.813456+0000","last_active":"2024-03-06T07:26:11.307449+0000","last_peered":"2024-03-06T07:26:11.307449+0000","last_clean":"2024-03-06T07:26:11.307449+0000","last_became_active":"2024-03-06T07:26:09.813219+0000","last_became_peered":"2024-03-06T07:26:09.813219+0000","last_unstale":"2024-03-06T07:26:11.307449+0000","last_undegraded":"2024-03-06T07:26:11.307449+0000","last_fullsized":"2024-03-06T07:26:11.307449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:59.079331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.821601+0000","last_change":"2024-03-06T07:26:08.815514+0000","last_active":"2024-03-06T07:26:10.821601+0000","last_peered":"2024-03-06T07:26:10.821601+0000","last_clean":"2024-03-06T07:26:10.821601+0000","last_became_active":"2024-03-06T07:26:08.815287+0000","last_became_peered":"2024-03-06T07:26:08.815287+0000","last_unstale":"2024-03-06T07:26:10.821601+0000","last_undegraded":"2024-03-06T07:26:10.821601+0000","last_fullsized":"2024-03-06T07:26:10.821601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.888883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307334+0000","last_change":"2024-03-06T07:26:09.809198+0000","last_active":"2024-03-06T07:26:11.307334+0000","last_peered":"2024-03-06T07:26:11.307334+0000","last_clean":"2024-03-06T07:26:11.307334+0000","last_became_active":"2024-03-06T07:26:09.808232+0000","last_became_peered":"2024-03-06T07:26:09.808232+0000","last_unstale":"2024-03-06T07:26:11.307334+0000","last_undegraded":"2024-03-06T07:26:11.307334+0000","last_fullsized":"2024-03-06T07:26:11.307334+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:32.641667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819502+0000","last_change":"2024-03-06T07:26:08.813109+0000","last_active":"2024-03-06T07:26:10.819502+0000","last_peered":"2024-03-06T07:26:10.819502+0000","last_clean":"2024-03-06T07:26:10.819502+0000","last_became_active":"2024-03-06T07:26:08.812841+0000","last_became_peered":"2024-03-06T07:26:08.812841+0000","last_unstale":"2024-03-06T07:26:10.819502+0000","last_undegraded":"2024-03-06T07:26:10.819502+0000","last_fullsized":"2024-03-06T07:26:10.819502+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:40.055990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307839+0000","last_change":"2024-03-06T07:26:09.819549+0000","last_active":"2024-03-06T07:26:11.307839+0000","last_peered":"2024-03-06T07:26:11.307839+0000","last_clean":"2024-03-06T07:26:11.307839+0000","last_became_active":"2024-03-06T07:26:09.819262+0000","last_became_peered":"2024-03-06T07:26:09.819262+0000","last_unstale":"2024-03-06T07:26:11.307839+0000","last_undegraded":"2024-03-06T07:26:11.307839+0000","last_fullsized":"2024-03-06T07:26:11.307839+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:51.051289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307663+0000","last_change":"2024-03-06T07:26:08.807992+0000","last_active":"2024-03-06T07:26:11.307663+0000","last_peered":"2024-03-06T07:26:11.307663+0000","last_clean":"2024-03-06T07:26:11.307663+0000","last_became_active":"2024-03-06T07:26:08.807709+0000","last_became_peered":"2024-03-06T07:26:08.807709+0000","last_unstale":"2024-03-06T07:26:11.307663+0000","last_undegraded":"2024-03-06T07:26:11.307663+0000","last_fullsized":"2024-03-06T07:26:11.307663+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:19.434200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824605+0000","last_change":"2024-03-06T07:26:09.808602+0000","last_active":"2024-03-06T07:26:10.824605+0000","last_peered":"2024-03-06T07:26:10.824605+0000","last_clean":"2024-03-06T07:26:10.824605+0000","last_became_active":"2024-03-06T07:26:09.807997+0000","last_became_peered":"2024-03-06T07:26:09.807997+0000","last_unstale":"2024-03-06T07:26:10.824605+0000","last_undegraded":"2024-03-06T07:26:10.824605+0000","last_fullsized":"2024-03-06T07:26:10.824605+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:59.804615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819434+0000","last_change":"2024-03-06T07:26:08.814261+0000","last_active":"2024-03-06T07:26:10.819434+0000","last_peered":"2024-03-06T07:26:10.819434+0000","last_clean":"2024-03-06T07:26:10.819434+0000","last_became_active":"2024-03-06T07:26:08.813964+0000","last_became_peered":"2024-03-06T07:26:08.813964+0000","last_unstale":"2024-03-06T07:26:10.819434+0000","last_undegraded":"2024-03-06T07:26:10.819434+0000","last_fullsized":"2024-03-06T07:26:10.819434+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:26.197567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803268+0000","last_change":"2024-03-06T07:26:09.806728+0000","last_active":"2024-03-06T07:26:10.803268+0000","last_peered":"2024-03-06T07:26:10.803268+0000","last_clean":"2024-03-06T07:26:10.803268+0000","last_became_active":"2024-03-06T07:26:09.805929+0000","last_became_peered":"2024-03-06T07:26:09.805929+0000","last_unstale":"2024-03-06T07:26:10.803268+0000","last_undegraded":"2024-03-06T07:26:10.803268+0000","last_fullsized":"2024-03-06T07:26:10.803268+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:41.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307729+0000","last_change":"2024-03-06T07:26:08.814662+0000","last_active":"2024-03-06T07:26:11.307729+0000","last_peered":"2024-03-06T07:26:11.307729+0000","last_clean":"2024-03-06T07:26:11.307729+0000","last_became_active":"2024-03-06T07:26:08.814315+0000","last_became_peered":"2024-03-06T07:26:08.814315+0000","last_unstale":"2024-03-06T07:26:11.307729+0000","last_undegraded":"2024-03-06T07:26:11.307729+0000","last_fullsized":"2024-03-06T07:26:11.307729+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:19.531856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824485+0000","last_change":"2024-03-06T07:26:09.810380+0000","last_active":"2024-03-06T07:26:10.824485+0000","last_peered":"2024-03-06T07:26:10.824485+0000","last_clean":"2024-03-06T07:26:10.824485+0000","last_became_active":"2024-03-06T07:26:09.810132+0000","last_became_peered":"2024-03-06T07:26:09.810132+0000","last_unstale":"2024-03-06T07:26:10.824485+0000","last_undegraded":"2024-03-06T07:26:10.824485+0000","last_fullsized":"2024-03-06T07:26:10.824485+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:13.311550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804568+0000","last_change":"2024-03-06T07:26:08.812623+0000","last_active":"2024-03-06T07:26:10.804568+0000","last_peered":"2024-03-06T07:26:10.804568+0000","last_clean":"2024-03-06T07:26:10.804568+0000","last_became_active":"2024-03-06T07:26:08.812312+0000","last_became_peered":"2024-03-06T07:26:08.812312+0000","last_unstale":"2024-03-06T07:26:10.804568+0000","last_undegraded":"2024-03-06T07:26:10.804568+0000","last_fullsized":"2024-03-06T07:26:10.804568+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:45.023392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307906+0000","last_change":"2024-03-06T07:26:09.814986+0000","last_active":"2024-03-06T07:26:11.307906+0000","last_peered":"2024-03-06T07:26:11.307906+0000","last_clean":"2024-03-06T07:26:11.307906+0000","last_became_active":"2024-03-06T07:26:09.814709+0000","last_became_peered":"2024-03-06T07:26:09.814709+0000","last_unstale":"2024-03-06T07:26:11.307906+0000","last_undegraded":"2024-03-06T07:26:11.307906+0000","last_fullsized":"2024-03-06T07:26:11.307906+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:09.051621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824566+0000","last_change":"2024-03-06T07:26:08.814790+0000","last_active":"2024-03-06T07:26:10.824566+0000","last_peered":"2024-03-06T07:26:10.824566+0000","last_clean":"2024-03-06T07:26:10.824566+0000","last_became_active":"2024-03-06T07:26:08.814512+0000","last_became_peered":"2024-03-06T07:26:08.814512+0000","last_unstale":"2024-03-06T07:26:10.824566+0000","last_undegraded":"2024-03-06T07:26:10.824566+0000","last_fullsized":"2024-03-06T07:26:10.824566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:00.263716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824878+0000","last_change":"2024-03-06T07:26:09.812801+0000","last_active":"2024-03-06T07:26:10.824878+0000","last_peered":"2024-03-06T07:26:10.824878+0000","last_clean":"2024-03-06T07:26:10.824878+0000","last_became_active":"2024-03-06T07:26:09.812553+0000","last_became_peered":"2024-03-06T07:26:09.812553+0000","last_unstale":"2024-03-06T07:26:10.824878+0000","last_undegraded":"2024-03-06T07:26:10.824878+0000","last_fullsized":"2024-03-06T07:26:10.824878+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:38.401927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802869+0000","last_change":"2024-03-06T07:26:08.813594+0000","last_active":"2024-03-06T07:26:10.802869+0000","last_peered":"2024-03-06T07:26:10.802869+0000","last_clean":"2024-03-06T07:26:10.802869+0000","last_became_active":"2024-03-06T07:26:08.813342+0000","last_became_peered":"2024-03-06T07:26:08.813342+0000","last_unstale":"2024-03-06T07:26:10.802869+0000","last_undegraded":"2024-03-06T07:26:10.802869+0000","last_fullsized":"2024-03-06T07:26:10.802869+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.888883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":298,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803211+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:26:10.803211+0000","last_peered":"2024-03-06T07:26:10.803211+0000","last_clean":"2024-03-06T07:26:10.803211+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:26:10.803211+0000","last_undegraded":"2024-03-06T07:26:10.803211+0000","last_fullsized":"2024-03-06T07:26:10.803211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":1740800,"data_stored":1074483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36464,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335376,"statfs":{"total":96636764160,"available":96599425024,"internally_reserved":0,"allocated":1146880,"data_stored":484655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36460,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335380,"statfs":{"total":96636764160,"available":96599429120,"internally_reserved":0,"allocated":1142784,"data_stored":484073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1159168,"data_stored":485739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1155072,"data_stored":485637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1142784,"data_stored":484037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":484003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1736704,"data_stored":1074953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:13.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:26:13.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.702+0000 7fa7e1e88640 1 Processor -- start 2024-03-06T07:26:13.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.702+0000 7fa7e1e88640 1 -- start start 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7e1e88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc0b4460 0x7fa7dc0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7e1e88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7dc162970 0x7fa7dc0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7e1e88640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa7dc161cf0 0x7fa7dc162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7e1e88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc161b00 con 0x7fa7dc162970 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7e1e88640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc0cecc0 con 0x7fa7dc161cf0 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7e1e88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0d22c0 con 0x7fa7dc0b4460 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7daffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7dc162970 0x7fa7dc0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7daffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7dc162970 0x7fa7dc0b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60714/0 (socket says 172.21.15.77:60714) 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7daffd640 1 -- 172.21.15.77:0/560501838 learned_addr learned my addr 172.21.15.77:0/560501838 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7db7fe640 1 --2- 172.21.15.77:0/560501838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc0b4460 0x7fa7dc0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7dbfff640 1 --2- 172.21.15.77:0/560501838 >> v2:172.21.15.158:3300/0 conn(0x7fa7dc161cf0 0x7fa7dc162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7daffd640 1 -- 172.21.15.77:0/560501838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc0b4460 msgr2=0x7fa7dc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7daffd640 1 --2- 172.21.15.77:0/560501838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc0b4460 0x7fa7dc0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7daffd640 1 -- 172.21.15.77:0/560501838 >> v2:172.21.15.158:3300/0 conn(0x7fa7dc161cf0 msgr2=0x7fa7dc162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.703+0000 7fa7daffd640 1 --2- 172.21.15.77:0/560501838 >> v2:172.21.15.158:3300/0 conn(0x7fa7dc161cf0 0x7fa7dc162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.704+0000 7fa7daffd640 1 -- 172.21.15.77:0/560501838 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc0d25a0 con 0x7fa7dc162970 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7daffd640 1 --2- 172.21.15.77:0/560501838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0b3580 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0ce2c0 tx=0x7fa7d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebb94fd1f799a9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7da7fc640 1 -- 172.21.15.77:0/560501838 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0035020 con 0x7fa7dc162970 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7da7fc640 1 -- 172.21.15.77:0/560501838 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d0031870 con 0x7fa7dc162970 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7da7fc640 1 -- 172.21.15.77:0/560501838 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0031c40 con 0x7fa7dc162970 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 -- 172.21.15.77:0/560501838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 msgr2=0x7fa7dc0b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/560501838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0b3580 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0ce2c0 tx=0x7fa7d002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 -- 172.21.15.77:0/560501838 shutdown_connections 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/560501838 >> v2:172.21.15.158:3300/0 conn(0x7fa7dc161cf0 0x7fa7dc162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/560501838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0b3580 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/560501838 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc0b4460 0x7fa7dc0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 -- 172.21.15.77:0/560501838 >> 172.21.15.77:0/560501838 conn(0x7fa7dc0d3ce0 msgr2=0x7fa7dc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 -- 172.21.15.77:0/560501838 shutdown_connections 2024-03-06T07:26:13.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.705+0000 7fa7e1e88640 1 -- 172.21.15.77:0/560501838 wait complete. 2024-03-06T07:26:13.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.707+0000 7fa7e1e88640 1 Processor -- start 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.707+0000 7fa7e1e88640 1 -- start start 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.707+0000 7fa7e1e88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7dc0b4460 0x7fa7dc159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.707+0000 7fa7e1e88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc161cf0 0x7fa7dc0b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.707+0000 7fa7e1e88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.707+0000 7fa7e1e88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0ce950 con 0x7fa7dc162970 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.707+0000 7fa7e1e88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0d2b60 con 0x7fa7dc0b4460 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.707+0000 7fa7e1e88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0b3ac0 con 0x7fa7dc161cf0 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60716/0 (socket says 172.21.15.77:60716) 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 -- 172.21.15.77:0/2822703640 learned_addr learned my addr 172.21.15.77:0/2822703640 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7db7fe640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7dc0b4460 0x7fa7dc159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7daffd640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc161cf0 0x7fa7dc0b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 -- 172.21.15.77:0/2822703640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc161cf0 msgr2=0x7fa7dc0b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc161cf0 0x7fa7dc0b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 -- 172.21.15.77:0/2822703640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7dc0b4460 msgr2=0x7fa7dc159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7dc0b4460 0x7fa7dc159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 -- 172.21.15.77:0/2822703640 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc1f7470 con 0x7fa7dc162970 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7dbfff640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0cca90 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4007e60 tx=0x7fa7d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7c3fff640 1 -- 172.21.15.77:0/2822703640 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4014420 con 0x7fa7dc162970 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7c3fff640 1 -- 172.21.15.77:0/2822703640 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d4019070 con 0x7fa7dc162970 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7dc1f7750 con 0x7fa7dc162970 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7c3fff640 1 -- 172.21.15.77:0/2822703640 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4014820 con 0x7fa7dc162970 2024-03-06T07:26:13.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.708+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7dc1f7d10 con 0x7fa7dc162970 2024-03-06T07:26:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.709+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7dc04f330 con 0x7fa7dc162970 2024-03-06T07:26:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.710+0000 7fa7c3fff640 1 -- 172.21.15.77:0/2822703640 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7d40149c0 con 0x7fa7dc162970 2024-03-06T07:26:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.710+0000 7fa7c3fff640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7c806b4d0 0x7fa7c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.710+0000 7fa7c3fff640 1 -- 172.21.15.77:0/2822703640 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7d408a950 con 0x7fa7dc162970 2024-03-06T07:26:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.710+0000 7fa7db7fe640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7c806b4d0 0x7fa7c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.710+0000 7fa7db7fe640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7c806b4d0 0x7fa7c806d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc006fd0 tx=0x7fa7cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.714+0000 7fa7c3fff640 1 -- 172.21.15.77:0/2822703640 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7d4059860 con 0x7fa7dc162970 2024-03-06T07:26:13.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.961+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa7dc06b180 con 0x7fa7dc162970 2024-03-06T07:26:13.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.961+0000 7fa7c3fff640 1 -- 172.21.15.77:0/2822703640 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1432 (secure 0 0 0) 0x7fa7d4059160 con 0x7fa7dc162970 2024-03-06T07:26:13.963 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:13.963 INFO:teuthology.orchestra.run.smithi077.stdout:{"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":272,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":92,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709711,"num_in_osds":8,"osd_in_since":1709709703,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":282861568,"bytes_avail":772811251712,"bytes_total":773094113280,"write_bytes_sec":2543,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":47,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"c","status":"up:active","gid":7287}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-06T07:25:59.427031+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:26:13.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7c806b4d0 msgr2=0x7fa7c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7c806b4d0 0x7fa7c806d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc006fd0 tx=0x7fa7cc008040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 msgr2=0x7fa7dc0cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0cca90 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4007e60 tx=0x7fa7d400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 shutdown_connections 2024-03-06T07:26:13.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa7c806b4d0 0x7fa7c806d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7dc162970 0x7fa7dc0cca90 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7dc161cf0 0x7fa7dc0b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 --2- 172.21.15.77:0/2822703640 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa7dc0b4460 0x7fa7dc159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 >> 172.21.15.77:0/2822703640 conn(0x7fa7dc0d3ce0 msgr2=0x7fa7dc15fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 shutdown_connections 2024-03-06T07:26:13.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:13.963+0000 7fa7e1e88640 1 -- 172.21.15.77:0/2822703640 wait complete. 2024-03-06T07:26:13.976 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:26:13.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:14.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.076+0000 7fb0d839a640 1 Processor -- start 2024-03-06T07:26:14.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.076+0000 7fb0d839a640 1 -- start start 2024-03-06T07:26:14.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d839a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0d00d22b0 0x7fb0d00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d839a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d00b3430 con 0x7fb0d015dfe0 2024-03-06T07:26:14.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d839a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d00b3710 con 0x7fb0d00b41f0 2024-03-06T07:26:14.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d839a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d00d2c50 con 0x7fb0d00d22b0 2024-03-06T07:26:14.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d5935640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0d00d22b0 0x7fb0d00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d6136640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb0d015dfe0 0x7fb0d00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44010/0 (socket says 172.21.15.77:44010) 2024-03-06T07:26:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d6136640 1 -- 172.21.15.77:0/1528215183 learned_addr learned my addr 172.21.15.77:0/1528215183 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d5935640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0d00d22b0 0x7fb0d00d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60720/0 (socket says 172.21.15.77:60720) 2024-03-06T07:26:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d5935640 1 -- 172.21.15.77:0/1528215183 >> v1:172.21.15.158:6789/0 conn(0x7fb0d00b41f0 legacy=0x7fb0d00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d5935640 1 -- 172.21.15.77:0/1528215183 >> v1:172.21.15.77:6789/0 conn(0x7fb0d015dfe0 legacy=0x7fb0d00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d5935640 1 -- 172.21.15.77:0/1528215183 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d00b39f0 con 0x7fb0d00d22b0 2024-03-06T07:26:14.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.077+0000 7fb0d5134640 1 -- 172.21.15.77:0/1528215183 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425742640 0 0) 0x7fb0d00b3430 con 0x7fb0d015dfe0 2024-03-06T07:26:14.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.078+0000 7fb0d5935640 1 --2- 172.21.15.77:0/1528215183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00d2710 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00097d0 tx=0x7fb0c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9facce9996a566 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d5134640 1 -- 172.21.15.77:0/1528215183 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0035020 con 0x7fb0d00d22b0 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d5134640 1 -- 172.21.15.77:0/1528215183 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0031830 con 0x7fb0d00d22b0 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d5134640 1 -- 172.21.15.77:0/1528215183 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0031c00 con 0x7fb0d00d22b0 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d839a640 1 -- 172.21.15.77:0/1528215183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 msgr2=0x7fb0d00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d839a640 1 --2- 172.21.15.77:0/1528215183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00d2710 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00097d0 tx=0x7fb0c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d839a640 1 -- 172.21.15.77:0/1528215183 shutdown_connections 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d839a640 1 --2- 172.21.15.77:0/1528215183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00d2710 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d839a640 1 -- 172.21.15.77:0/1528215183 >> 172.21.15.77:0/1528215183 conn(0x7fb0d00d3cd0 msgr2=0x7fb0d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d839a640 1 -- 172.21.15.77:0/1528215183 shutdown_connections 2024-03-06T07:26:14.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.079+0000 7fb0d839a640 1 -- 172.21.15.77:0/1528215183 wait complete. 2024-03-06T07:26:14.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.080+0000 7fb0d839a640 1 Processor -- start 2024-03-06T07:26:14.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.080+0000 7fb0d839a640 1 -- start start 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d839a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d00b41f0 0x7fb0d0156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d839a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d839a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0d015dfe0 0x7fb0d00b84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d839a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d00b3d60 con 0x7fb0d00d22b0 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d839a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d00b31e0 con 0x7fb0d00b41f0 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d839a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d00d1c70 con 0x7fb0d015dfe0 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d5935640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d5935640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60722/0 (socket says 172.21.15.77:60722) 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d5935640 1 -- 172.21.15.77:0/2481593794 learned_addr learned my addr 172.21.15.77:0/2481593794 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d6136640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d00b41f0 0x7fb0d0156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d6937640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0d015dfe0 0x7fb0d00b84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d5935640 1 -- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0d015dfe0 msgr2=0x7fb0d00b84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d5935640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0d015dfe0 0x7fb0d00b84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d5935640 1 -- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d00b41f0 msgr2=0x7fb0d0156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d5935640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d00b41f0 0x7fb0d0156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.081+0000 7fb0d5935640 1 -- 172.21.15.77:0/2481593794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0157800 con 0x7fb0d00d22b0 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.082+0000 7fb0d6136640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d00b41f0 0x7fb0d0156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.082+0000 7fb0d6937640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0d015dfe0 0x7fb0d00b84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.082+0000 7fb0d5935640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00b68a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0000c00 tx=0x7fb0c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.082+0000 7fb0c6ffd640 1 -- 172.21.15.77:0/2481593794 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0004230 con 0x7fb0d00d22b0 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.082+0000 7fb0c6ffd640 1 -- 172.21.15.77:0/2481593794 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c00043f0 con 0x7fb0d00d22b0 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.082+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d00b93b0 con 0x7fb0d00d22b0 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.082+0000 7fb0c6ffd640 1 -- 172.21.15.77:0/2481593794 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c003c9d0 con 0x7fb0d00d22b0 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.082+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d00b9970 con 0x7fb0d00d22b0 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.083+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d004f330 con 0x7fb0d00d22b0 2024-03-06T07:26:14.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.083+0000 7fb0c6ffd640 1 -- 172.21.15.77:0/2481593794 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0c0038020 con 0x7fb0d00d22b0 2024-03-06T07:26:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.084+0000 7fb0c6ffd640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0b406b4d0 0x7fb0b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.084+0000 7fb0c6ffd640 1 -- 172.21.15.77:0/2481593794 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0c00ac4c0 con 0x7fb0d00d22b0 2024-03-06T07:26:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.084+0000 7fb0d6136640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0b406b4d0 0x7fb0b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.084+0000 7fb0d6136640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0b406b4d0 0x7fb0b406d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8000f60 tx=0x7fb0c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.092+0000 7fb0c6ffd640 1 -- 172.21.15.77:0/2481593794 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0d0157800 con 0x7fb0d00d22b0 2024-03-06T07:26:14.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.260+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0d0157300 con 0x7fb0b406b4d0 2024-03-06T07:26:14.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.266+0000 7fb0c6ffd640 1 -- 172.21.15.77:0/2481593794 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252084 (secure 0 0 0) 0x7fb0d0157300 con 0x7fb0b406b4d0 2024-03-06T07:26:14.270 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:14.271 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.268+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0b406b4d0 msgr2=0x7fb0b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.268+0000 7fb0d839a640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0b406b4d0 0x7fb0b406d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8000f60 tx=0x7fb0c8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.268+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 msgr2=0x7fb0d00b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.268+0000 7fb0d839a640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00b68a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0000c00 tx=0x7fb0c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.269+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 shutdown_connections 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.269+0000 7fb0d839a640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb0b406b4d0 0x7fb0b406d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.269+0000 7fb0d839a640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d00d22b0 0x7fb0d00b68a0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.269+0000 7fb0d839a640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb0d015dfe0 0x7fb0d00b84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.269+0000 7fb0d839a640 1 --2- 172.21.15.77:0/2481593794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb0d00b41f0 0x7fb0d0156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.269+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 >> 172.21.15.77:0/2481593794 conn(0x7fb0d00d3cd0 msgr2=0x7fb0d006ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.269+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 shutdown_connections 2024-03-06T07:26:14.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:14.269+0000 7fb0d839a640 1 -- 172.21.15.77:0/2481593794 wait complete. 2024-03-06T07:26:14.282 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-03-06T07:26:13.429418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276232,"kb_used_data":10120,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698488,"statfs":{"total":773094113280,"available":772811251712,"internally_reserved":0,"allocated":10362880,"data_stored":5057580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.637017"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307447+0000","last_change":"2024-03-06T07:26:09.813853+0000","last_active":"2024-03-06T07:26:11.307447+0000","last_peered":"2024-03-06T07:26:11.307447+0000","last_clean":"2024-03-06T07:26:11.307447+0000","last_became_active":"2024-03-06T07:26:09.813610+0000","last_became_peered":"2024-03-06T07:26:09.813610+0000","last_unstale":"2024-03-06T07:26:11.307447+0000","last_undegraded":"2024-03-06T07:26:11.307447+0000","last_fullsized":"2024-03-06T07:26:11.307447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:31.816194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819608+0000","last_change":"2024-03-06T07:26:09.819767+0000","last_active":"2024-03-06T07:26:10.819608+0000","last_peered":"2024-03-06T07:26:10.819608+0000","last_clean":"2024-03-06T07:26:10.819608+0000","last_became_active":"2024-03-06T07:26:09.819504+0000","last_became_peered":"2024-03-06T07:26:09.819504+0000","last_unstale":"2024-03-06T07:26:10.819608+0000","last_undegraded":"2024-03-06T07:26:10.819608+0000","last_fullsized":"2024-03-06T07:26:10.819608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307400+0000","last_change":"2024-03-06T07:26:09.811069+0000","last_active":"2024-03-06T07:26:11.307400+0000","last_peered":"2024-03-06T07:26:11.307400+0000","last_clean":"2024-03-06T07:26:11.307400+0000","last_became_active":"2024-03-06T07:26:09.810813+0000","last_became_peered":"2024-03-06T07:26:09.810813+0000","last_unstale":"2024-03-06T07:26:11.307400+0000","last_undegraded":"2024-03-06T07:26:11.307400+0000","last_fullsized":"2024-03-06T07:26:11.307400+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:30.253373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804808+0000","last_change":"2024-03-06T07:26:09.816208+0000","last_active":"2024-03-06T07:26:10.804808+0000","last_peered":"2024-03-06T07:26:10.804808+0000","last_clean":"2024-03-06T07:26:10.804808+0000","last_became_active":"2024-03-06T07:26:09.815932+0000","last_became_peered":"2024-03-06T07:26:09.815932+0000","last_unstale":"2024-03-06T07:26:10.804808+0000","last_undegraded":"2024-03-06T07:26:10.804808+0000","last_fullsized":"2024-03-06T07:26:10.804808+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:19.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307777+0000","last_change":"2024-03-06T07:26:09.815170+0000","last_active":"2024-03-06T07:26:11.307777+0000","last_peered":"2024-03-06T07:26:11.307777+0000","last_clean":"2024-03-06T07:26:11.307777+0000","last_became_active":"2024-03-06T07:26:09.814899+0000","last_became_peered":"2024-03-06T07:26:09.814899+0000","last_unstale":"2024-03-06T07:26:11.307777+0000","last_undegraded":"2024-03-06T07:26:11.307777+0000","last_fullsized":"2024-03-06T07:26:11.307777+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:46.027813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802948+0000","last_change":"2024-03-06T07:26:09.819179+0000","last_active":"2024-03-06T07:26:10.802948+0000","last_peered":"2024-03-06T07:26:10.802948+0000","last_clean":"2024-03-06T07:26:10.802948+0000","last_became_active":"2024-03-06T07:26:09.818840+0000","last_became_peered":"2024-03-06T07:26:09.818840+0000","last_unstale":"2024-03-06T07:26:10.802948+0000","last_undegraded":"2024-03-06T07:26:10.802948+0000","last_fullsized":"2024-03-06T07:26:10.802948+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:17.314686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802910+0000","last_change":"2024-03-06T07:26:09.818359+0000","last_active":"2024-03-06T07:26:10.802910+0000","last_peered":"2024-03-06T07:26:10.802910+0000","last_clean":"2024-03-06T07:26:10.802910+0000","last_became_active":"2024-03-06T07:26:09.818091+0000","last_became_peered":"2024-03-06T07:26:09.818091+0000","last_unstale":"2024-03-06T07:26:10.802910+0000","last_undegraded":"2024-03-06T07:26:10.802910+0000","last_fullsized":"2024-03-06T07:26:10.802910+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:14.992810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819210+0000","last_change":"2024-03-06T07:26:09.814692+0000","last_active":"2024-03-06T07:26:10.819210+0000","last_peered":"2024-03-06T07:26:10.819210+0000","last_clean":"2024-03-06T07:26:10.819210+0000","last_became_active":"2024-03-06T07:26:09.814375+0000","last_became_peered":"2024-03-06T07:26:09.814375+0000","last_unstale":"2024-03-06T07:26:10.819210+0000","last_undegraded":"2024-03-06T07:26:10.819210+0000","last_fullsized":"2024-03-06T07:26:10.819210+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803819+0000","last_change":"2024-03-06T07:26:09.818486+0000","last_active":"2024-03-06T07:26:10.803819+0000","last_peered":"2024-03-06T07:26:10.803819+0000","last_clean":"2024-03-06T07:26:10.803819+0000","last_became_active":"2024-03-06T07:26:09.818183+0000","last_became_peered":"2024-03-06T07:26:09.818183+0000","last_unstale":"2024-03-06T07:26:10.803819+0000","last_undegraded":"2024-03-06T07:26:10.803819+0000","last_fullsized":"2024-03-06T07:26:10.803819+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:21.525054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307565+0000","last_change":"2024-03-06T07:26:09.814301+0000","last_active":"2024-03-06T07:26:11.307565+0000","last_peered":"2024-03-06T07:26:11.307565+0000","last_clean":"2024-03-06T07:26:11.307565+0000","last_became_active":"2024-03-06T07:26:09.810981+0000","last_became_peered":"2024-03-06T07:26:09.810981+0000","last_unstale":"2024-03-06T07:26:11.307565+0000","last_undegraded":"2024-03-06T07:26:11.307565+0000","last_fullsized":"2024-03-06T07:26:11.307565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:41.312503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803579+0000","last_change":"2024-03-06T07:26:09.816331+0000","last_active":"2024-03-06T07:26:10.803579+0000","last_peered":"2024-03-06T07:26:10.803579+0000","last_clean":"2024-03-06T07:26:10.803579+0000","last_became_active":"2024-03-06T07:26:09.816061+0000","last_became_peered":"2024-03-06T07:26:09.816061+0000","last_unstale":"2024-03-06T07:26:10.803579+0000","last_undegraded":"2024-03-06T07:26:10.803579+0000","last_fullsized":"2024-03-06T07:26:10.803579+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803958+0000","last_change":"2024-03-06T07:26:09.810779+0000","last_active":"2024-03-06T07:26:10.803958+0000","last_peered":"2024-03-06T07:26:10.803958+0000","last_clean":"2024-03-06T07:26:10.803958+0000","last_became_active":"2024-03-06T07:26:09.810512+0000","last_became_peered":"2024-03-06T07:26:09.810512+0000","last_unstale":"2024-03-06T07:26:10.803958+0000","last_undegraded":"2024-03-06T07:26:10.803958+0000","last_fullsized":"2024-03-06T07:26:10.803958+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:41.312503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804655+0000","last_change":"2024-03-06T07:26:09.808372+0000","last_active":"2024-03-06T07:26:10.804655+0000","last_peered":"2024-03-06T07:26:10.804655+0000","last_clean":"2024-03-06T07:26:10.804655+0000","last_became_active":"2024-03-06T07:26:09.808072+0000","last_became_peered":"2024-03-06T07:26:09.808072+0000","last_unstale":"2024-03-06T07:26:10.804655+0000","last_undegraded":"2024-03-06T07:26:10.804655+0000","last_fullsized":"2024-03-06T07:26:10.804655+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824446+0000","last_change":"2024-03-06T07:26:09.806321+0000","last_active":"2024-03-06T07:26:10.824446+0000","last_peered":"2024-03-06T07:26:10.824446+0000","last_clean":"2024-03-06T07:26:10.824446+0000","last_became_active":"2024-03-06T07:26:09.806070+0000","last_became_peered":"2024-03-06T07:26:09.806070+0000","last_unstale":"2024-03-06T07:26:10.824446+0000","last_undegraded":"2024-03-06T07:26:10.824446+0000","last_fullsized":"2024-03-06T07:26:10.824446+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:05.439904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804472+0000","last_change":"2024-03-06T07:26:09.819527+0000","last_active":"2024-03-06T07:26:10.804472+0000","last_peered":"2024-03-06T07:26:10.804472+0000","last_clean":"2024-03-06T07:26:10.804472+0000","last_became_active":"2024-03-06T07:26:09.819257+0000","last_became_peered":"2024-03-06T07:26:09.819257+0000","last_unstale":"2024-03-06T07:26:10.804472+0000","last_undegraded":"2024-03-06T07:26:10.804472+0000","last_fullsized":"2024-03-06T07:26:10.804472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:17.314686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803146+0000","last_change":"2024-03-06T07:26:09.817278+0000","last_active":"2024-03-06T07:26:10.803146+0000","last_peered":"2024-03-06T07:26:10.803146+0000","last_clean":"2024-03-06T07:26:10.803146+0000","last_became_active":"2024-03-06T07:26:09.817015+0000","last_became_peered":"2024-03-06T07:26:09.817015+0000","last_unstale":"2024-03-06T07:26:10.803146+0000","last_undegraded":"2024-03-06T07:26:10.803146+0000","last_fullsized":"2024-03-06T07:26:10.803146+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:33.879869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819707+0000","last_change":"2024-03-06T07:26:09.824514+0000","last_active":"2024-03-06T07:26:10.819707+0000","last_peered":"2024-03-06T07:26:10.819707+0000","last_clean":"2024-03-06T07:26:10.819707+0000","last_became_active":"2024-03-06T07:26:09.823591+0000","last_became_peered":"2024-03-06T07:26:09.823591+0000","last_unstale":"2024-03-06T07:26:10.819707+0000","last_undegraded":"2024-03-06T07:26:10.819707+0000","last_fullsized":"2024-03-06T07:26:10.819707+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.005755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804704+0000","last_change":"2024-03-06T07:26:09.811053+0000","last_active":"2024-03-06T07:26:10.804704+0000","last_peered":"2024-03-06T07:26:10.804704+0000","last_clean":"2024-03-06T07:26:10.804704+0000","last_became_active":"2024-03-06T07:26:09.810805+0000","last_became_peered":"2024-03-06T07:26:09.810805+0000","last_unstale":"2024-03-06T07:26:10.804704+0000","last_undegraded":"2024-03-06T07:26:10.804704+0000","last_fullsized":"2024-03-06T07:26:10.804704+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824872+0000","last_change":"2024-03-06T07:26:09.809874+0000","last_active":"2024-03-06T07:26:10.824872+0000","last_peered":"2024-03-06T07:26:10.824872+0000","last_clean":"2024-03-06T07:26:10.824872+0000","last_became_active":"2024-03-06T07:26:09.809601+0000","last_became_peered":"2024-03-06T07:26:09.809601+0000","last_unstale":"2024-03-06T07:26:10.824872+0000","last_undegraded":"2024-03-06T07:26:10.824872+0000","last_fullsized":"2024-03-06T07:26:10.824872+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:56.986034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803542+0000","last_change":"2024-03-06T07:26:09.819152+0000","last_active":"2024-03-06T07:26:10.803542+0000","last_peered":"2024-03-06T07:26:10.803542+0000","last_clean":"2024-03-06T07:26:10.803542+0000","last_became_active":"2024-03-06T07:26:09.818889+0000","last_became_peered":"2024-03-06T07:26:09.818889+0000","last_unstale":"2024-03-06T07:26:10.803542+0000","last_undegraded":"2024-03-06T07:26:10.803542+0000","last_fullsized":"2024-03-06T07:26:10.803542+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:49.745227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824557+0000","last_change":"2024-03-06T07:26:09.815573+0000","last_active":"2024-03-06T07:26:10.824557+0000","last_peered":"2024-03-06T07:26:10.824557+0000","last_clean":"2024-03-06T07:26:10.824557+0000","last_became_active":"2024-03-06T07:26:09.815163+0000","last_became_peered":"2024-03-06T07:26:09.815163+0000","last_unstale":"2024-03-06T07:26:10.824557+0000","last_undegraded":"2024-03-06T07:26:10.824557+0000","last_fullsized":"2024-03-06T07:26:10.824557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:08.603270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819256+0000","last_change":"2024-03-06T07:26:09.820441+0000","last_active":"2024-03-06T07:26:10.819256+0000","last_peered":"2024-03-06T07:26:10.819256+0000","last_clean":"2024-03-06T07:26:10.819256+0000","last_became_active":"2024-03-06T07:26:09.820143+0000","last_became_peered":"2024-03-06T07:26:09.820143+0000","last_unstale":"2024-03-06T07:26:10.819256+0000","last_undegraded":"2024-03-06T07:26:10.819256+0000","last_fullsized":"2024-03-06T07:26:10.819256+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819199+0000","last_change":"2024-03-06T07:26:09.819137+0000","last_active":"2024-03-06T07:26:10.819199+0000","last_peered":"2024-03-06T07:26:10.819199+0000","last_clean":"2024-03-06T07:26:10.819199+0000","last_became_active":"2024-03-06T07:26:09.818884+0000","last_became_peered":"2024-03-06T07:26:09.818884+0000","last_unstale":"2024-03-06T07:26:10.819199+0000","last_undegraded":"2024-03-06T07:26:10.819199+0000","last_fullsized":"2024-03-06T07:26:10.819199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803186+0000","last_change":"2024-03-06T07:26:09.815241+0000","last_active":"2024-03-06T07:26:10.803186+0000","last_peered":"2024-03-06T07:26:10.803186+0000","last_clean":"2024-03-06T07:26:10.803186+0000","last_became_active":"2024-03-06T07:26:09.814964+0000","last_became_peered":"2024-03-06T07:26:09.814964+0000","last_unstale":"2024-03-06T07:26:10.803186+0000","last_undegraded":"2024-03-06T07:26:10.803186+0000","last_fullsized":"2024-03-06T07:26:10.803186+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307953+0000","last_change":"2024-03-06T07:26:09.816181+0000","last_active":"2024-03-06T07:26:11.307953+0000","last_peered":"2024-03-06T07:26:11.307953+0000","last_clean":"2024-03-06T07:26:11.307953+0000","last_became_active":"2024-03-06T07:26:09.815917+0000","last_became_peered":"2024-03-06T07:26:09.815917+0000","last_unstale":"2024-03-06T07:26:11.307953+0000","last_undegraded":"2024-03-06T07:26:11.307953+0000","last_fullsized":"2024-03-06T07:26:11.307953+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:41.338253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819765+0000","last_change":"2024-03-06T07:26:09.821323+0000","last_active":"2024-03-06T07:26:10.819765+0000","last_peered":"2024-03-06T07:26:10.819765+0000","last_clean":"2024-03-06T07:26:10.819765+0000","last_became_active":"2024-03-06T07:26:09.821041+0000","last_became_peered":"2024-03-06T07:26:09.821041+0000","last_unstale":"2024-03-06T07:26:10.819765+0000","last_undegraded":"2024-03-06T07:26:10.819765+0000","last_fullsized":"2024-03-06T07:26:10.819765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:11.188598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803635+0000","last_change":"2024-03-06T07:26:09.812299+0000","last_active":"2024-03-06T07:26:10.803635+0000","last_peered":"2024-03-06T07:26:10.803635+0000","last_clean":"2024-03-06T07:26:10.803635+0000","last_became_active":"2024-03-06T07:26:09.812009+0000","last_became_peered":"2024-03-06T07:26:09.812009+0000","last_unstale":"2024-03-06T07:26:10.803635+0000","last_undegraded":"2024-03-06T07:26:10.803635+0000","last_fullsized":"2024-03-06T07:26:10.803635+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307574+0000","last_change":"2024-03-06T07:26:09.821191+0000","last_active":"2024-03-06T07:26:11.307574+0000","last_peered":"2024-03-06T07:26:11.307574+0000","last_clean":"2024-03-06T07:26:11.307574+0000","last_became_active":"2024-03-06T07:26:09.820905+0000","last_became_peered":"2024-03-06T07:26:09.820905+0000","last_unstale":"2024-03-06T07:26:11.307574+0000","last_undegraded":"2024-03-06T07:26:11.307574+0000","last_fullsized":"2024-03-06T07:26:11.307574+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:27.201988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819336+0000","last_change":"2024-03-06T07:26:09.816456+0000","last_active":"2024-03-06T07:26:10.819336+0000","last_peered":"2024-03-06T07:26:10.819336+0000","last_clean":"2024-03-06T07:26:10.819336+0000","last_became_active":"2024-03-06T07:26:09.816206+0000","last_became_peered":"2024-03-06T07:26:09.816206+0000","last_unstale":"2024-03-06T07:26:10.819336+0000","last_undegraded":"2024-03-06T07:26:10.819336+0000","last_fullsized":"2024-03-06T07:26:10.819336+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824471+0000","last_change":"2024-03-06T07:26:09.808861+0000","last_active":"2024-03-06T07:26:10.824471+0000","last_peered":"2024-03-06T07:26:10.824471+0000","last_clean":"2024-03-06T07:26:10.824471+0000","last_became_active":"2024-03-06T07:26:09.808345+0000","last_became_peered":"2024-03-06T07:26:09.808345+0000","last_unstale":"2024-03-06T07:26:10.824471+0000","last_undegraded":"2024-03-06T07:26:10.824471+0000","last_fullsized":"2024-03-06T07:26:10.824471+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:15.906643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804539+0000","last_change":"2024-03-06T07:26:09.821038+0000","last_active":"2024-03-06T07:26:10.804539+0000","last_peered":"2024-03-06T07:26:10.804539+0000","last_clean":"2024-03-06T07:26:10.804539+0000","last_became_active":"2024-03-06T07:26:09.820749+0000","last_became_peered":"2024-03-06T07:26:09.820749+0000","last_unstale":"2024-03-06T07:26:10.804539+0000","last_undegraded":"2024-03-06T07:26:10.804539+0000","last_fullsized":"2024-03-06T07:26:10.804539+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307065+0000","last_change":"2024-03-06T07:26:09.815360+0000","last_active":"2024-03-06T07:26:11.307065+0000","last_peered":"2024-03-06T07:26:11.307065+0000","last_clean":"2024-03-06T07:26:11.307065+0000","last_became_active":"2024-03-06T07:26:09.815125+0000","last_became_peered":"2024-03-06T07:26:09.815125+0000","last_unstale":"2024-03-06T07:26:11.307065+0000","last_undegraded":"2024-03-06T07:26:11.307065+0000","last_fullsized":"2024-03-06T07:26:11.307065+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:58.101050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824887+0000","last_change":"2024-03-06T07:26:09.809611+0000","last_active":"2024-03-06T07:26:10.824887+0000","last_peered":"2024-03-06T07:26:10.824887+0000","last_clean":"2024-03-06T07:26:10.824887+0000","last_became_active":"2024-03-06T07:26:09.809343+0000","last_became_peered":"2024-03-06T07:26:09.809343+0000","last_unstale":"2024-03-06T07:26:10.824887+0000","last_undegraded":"2024-03-06T07:26:10.824887+0000","last_fullsized":"2024-03-06T07:26:10.824887+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:48.800992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819712+0000","last_change":"2024-03-06T07:26:08.809604+0000","last_active":"2024-03-06T07:26:10.819712+0000","last_peered":"2024-03-06T07:26:10.819712+0000","last_clean":"2024-03-06T07:26:10.819712+0000","last_became_active":"2024-03-06T07:26:08.809360+0000","last_became_peered":"2024-03-06T07:26:08.809360+0000","last_unstale":"2024-03-06T07:26:10.819712+0000","last_undegraded":"2024-03-06T07:26:10.819712+0000","last_fullsized":"2024-03-06T07:26:10.819712+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:10.336592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804780+0000","last_change":"2024-03-06T07:26:09.822258+0000","last_active":"2024-03-06T07:26:10.804780+0000","last_peered":"2024-03-06T07:26:10.804780+0000","last_clean":"2024-03-06T07:26:10.804780+0000","last_became_active":"2024-03-06T07:26:09.822015+0000","last_became_peered":"2024-03-06T07:26:09.822015+0000","last_unstale":"2024-03-06T07:26:10.804780+0000","last_undegraded":"2024-03-06T07:26:10.804780+0000","last_fullsized":"2024-03-06T07:26:10.804780+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803187+0000","last_change":"2024-03-06T07:26:08.812691+0000","last_active":"2024-03-06T07:26:10.803187+0000","last_peered":"2024-03-06T07:26:10.803187+0000","last_clean":"2024-03-06T07:26:10.803187+0000","last_became_active":"2024-03-06T07:26:08.812418+0000","last_became_peered":"2024-03-06T07:26:08.812418+0000","last_unstale":"2024-03-06T07:26:10.803187+0000","last_undegraded":"2024-03-06T07:26:10.803187+0000","last_fullsized":"2024-03-06T07:26:10.803187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:20.520632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307299+0000","last_change":"2024-03-06T07:26:09.807945+0000","last_active":"2024-03-06T07:26:11.307299+0000","last_peered":"2024-03-06T07:26:11.307299+0000","last_clean":"2024-03-06T07:26:11.307299+0000","last_became_active":"2024-03-06T07:26:09.807684+0000","last_became_peered":"2024-03-06T07:26:09.807684+0000","last_unstale":"2024-03-06T07:26:11.307299+0000","last_undegraded":"2024-03-06T07:26:11.307299+0000","last_fullsized":"2024-03-06T07:26:11.307299+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:13.507677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819496+0000","last_change":"2024-03-06T07:26:08.816145+0000","last_active":"2024-03-06T07:26:10.819496+0000","last_peered":"2024-03-06T07:26:10.819496+0000","last_clean":"2024-03-06T07:26:10.819496+0000","last_became_active":"2024-03-06T07:26:08.815639+0000","last_became_peered":"2024-03-06T07:26:08.815639+0000","last_unstale":"2024-03-06T07:26:10.819496+0000","last_undegraded":"2024-03-06T07:26:10.819496+0000","last_fullsized":"2024-03-06T07:26:10.819496+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:47.753067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803830+0000","last_change":"2024-03-06T07:26:09.814294+0000","last_active":"2024-03-06T07:26:10.803830+0000","last_peered":"2024-03-06T07:26:10.803830+0000","last_clean":"2024-03-06T07:26:10.803830+0000","last_became_active":"2024-03-06T07:26:09.814010+0000","last_became_peered":"2024-03-06T07:26:09.814010+0000","last_unstale":"2024-03-06T07:26:10.803830+0000","last_undegraded":"2024-03-06T07:26:10.803830+0000","last_fullsized":"2024-03-06T07:26:10.803830+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:37.893304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819440+0000","last_change":"2024-03-06T07:26:08.814364+0000","last_active":"2024-03-06T07:26:10.819440+0000","last_peered":"2024-03-06T07:26:10.819440+0000","last_clean":"2024-03-06T07:26:10.819440+0000","last_became_active":"2024-03-06T07:26:08.813668+0000","last_became_peered":"2024-03-06T07:26:08.813668+0000","last_unstale":"2024-03-06T07:26:10.819440+0000","last_undegraded":"2024-03-06T07:26:10.819440+0000","last_fullsized":"2024-03-06T07:26:10.819440+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:48.740805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307795+0000","last_change":"2024-03-06T07:26:09.812757+0000","last_active":"2024-03-06T07:26:11.307795+0000","last_peered":"2024-03-06T07:26:11.307795+0000","last_clean":"2024-03-06T07:26:11.307795+0000","last_became_active":"2024-03-06T07:26:09.812460+0000","last_became_peered":"2024-03-06T07:26:09.812460+0000","last_unstale":"2024-03-06T07:26:11.307795+0000","last_undegraded":"2024-03-06T07:26:11.307795+0000","last_fullsized":"2024-03-06T07:26:11.307795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:58.727545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307231+0000","last_change":"2024-03-06T07:26:08.809284+0000","last_active":"2024-03-06T07:26:11.307231+0000","last_peered":"2024-03-06T07:26:11.307231+0000","last_clean":"2024-03-06T07:26:11.307231+0000","last_became_active":"2024-03-06T07:26:08.809034+0000","last_became_peered":"2024-03-06T07:26:08.809034+0000","last_unstale":"2024-03-06T07:26:11.307231+0000","last_undegraded":"2024-03-06T07:26:11.307231+0000","last_fullsized":"2024-03-06T07:26:11.307231+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:07.598849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802744+0000","last_change":"2024-03-06T07:26:09.813345+0000","last_active":"2024-03-06T07:26:10.802744+0000","last_peered":"2024-03-06T07:26:10.802744+0000","last_clean":"2024-03-06T07:26:10.802744+0000","last_became_active":"2024-03-06T07:26:09.813115+0000","last_became_peered":"2024-03-06T07:26:09.813115+0000","last_unstale":"2024-03-06T07:26:10.802744+0000","last_undegraded":"2024-03-06T07:26:10.802744+0000","last_fullsized":"2024-03-06T07:26:10.802744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:04.605670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819254+0000","last_change":"2024-03-06T07:26:08.810941+0000","last_active":"2024-03-06T07:26:10.819254+0000","last_peered":"2024-03-06T07:26:10.819254+0000","last_clean":"2024-03-06T07:26:10.819254+0000","last_became_active":"2024-03-06T07:26:08.810678+0000","last_became_peered":"2024-03-06T07:26:08.810678+0000","last_unstale":"2024-03-06T07:26:10.819254+0000","last_undegraded":"2024-03-06T07:26:10.819254+0000","last_fullsized":"2024-03-06T07:26:10.819254+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:49.528401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804522+0000","last_change":"2024-03-06T07:26:09.820600+0000","last_active":"2024-03-06T07:26:10.804522+0000","last_peered":"2024-03-06T07:26:10.804522+0000","last_clean":"2024-03-06T07:26:10.804522+0000","last_became_active":"2024-03-06T07:26:09.820397+0000","last_became_peered":"2024-03-06T07:26:09.820397+0000","last_unstale":"2024-03-06T07:26:10.804522+0000","last_undegraded":"2024-03-06T07:26:10.804522+0000","last_fullsized":"2024-03-06T07:26:10.804522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:33.879869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308023+0000","last_change":"2024-03-06T07:26:08.810527+0000","last_active":"2024-03-06T07:26:11.308023+0000","last_peered":"2024-03-06T07:26:11.308023+0000","last_clean":"2024-03-06T07:26:11.308023+0000","last_became_active":"2024-03-06T07:26:08.810064+0000","last_became_peered":"2024-03-06T07:26:08.810064+0000","last_unstale":"2024-03-06T07:26:11.308023+0000","last_undegraded":"2024-03-06T07:26:11.308023+0000","last_fullsized":"2024-03-06T07:26:11.308023+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:57.723123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802744+0000","last_change":"2024-03-06T07:26:09.817490+0000","last_active":"2024-03-06T07:26:10.802744+0000","last_peered":"2024-03-06T07:26:10.802744+0000","last_clean":"2024-03-06T07:26:10.802744+0000","last_became_active":"2024-03-06T07:26:09.817201+0000","last_became_peered":"2024-03-06T07:26:09.817201+0000","last_unstale":"2024-03-06T07:26:10.802744+0000","last_undegraded":"2024-03-06T07:26:10.802744+0000","last_fullsized":"2024-03-06T07:26:10.802744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:09.736601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307007+0000","last_change":"2024-03-06T07:26:08.813560+0000","last_active":"2024-03-06T07:26:11.307007+0000","last_peered":"2024-03-06T07:26:11.307007+0000","last_clean":"2024-03-06T07:26:11.307007+0000","last_became_active":"2024-03-06T07:26:08.813280+0000","last_became_peered":"2024-03-06T07:26:08.813280+0000","last_unstale":"2024-03-06T07:26:11.307007+0000","last_undegraded":"2024-03-06T07:26:11.307007+0000","last_fullsized":"2024-03-06T07:26:11.307007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:18.986097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819607+0000","last_change":"2024-03-06T07:26:09.815672+0000","last_active":"2024-03-06T07:26:10.819607+0000","last_peered":"2024-03-06T07:26:10.819607+0000","last_clean":"2024-03-06T07:26:10.819607+0000","last_became_active":"2024-03-06T07:26:09.815424+0000","last_became_peered":"2024-03-06T07:26:09.815424+0000","last_unstale":"2024-03-06T07:26:10.819607+0000","last_undegraded":"2024-03-06T07:26:10.819607+0000","last_fullsized":"2024-03-06T07:26:10.819607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:19.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307510+0000","last_change":"2024-03-06T07:26:08.810875+0000","last_active":"2024-03-06T07:26:11.307510+0000","last_peered":"2024-03-06T07:26:11.307510+0000","last_clean":"2024-03-06T07:26:11.307510+0000","last_became_active":"2024-03-06T07:26:08.810627+0000","last_became_peered":"2024-03-06T07:26:08.810627+0000","last_unstale":"2024-03-06T07:26:11.307510+0000","last_undegraded":"2024-03-06T07:26:11.307510+0000","last_fullsized":"2024-03-06T07:26:11.307510+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:47.815489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803882+0000","last_change":"2024-03-06T07:26:09.813845+0000","last_active":"2024-03-06T07:26:10.803882+0000","last_peered":"2024-03-06T07:26:10.803882+0000","last_clean":"2024-03-06T07:26:10.803882+0000","last_became_active":"2024-03-06T07:26:09.813558+0000","last_became_peered":"2024-03-06T07:26:09.813558+0000","last_unstale":"2024-03-06T07:26:10.803882+0000","last_undegraded":"2024-03-06T07:26:10.803882+0000","last_fullsized":"2024-03-06T07:26:10.803882+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:11.341014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804612+0000","last_change":"2024-03-06T07:26:08.811996+0000","last_active":"2024-03-06T07:26:10.804612+0000","last_peered":"2024-03-06T07:26:10.804612+0000","last_clean":"2024-03-06T07:26:10.804612+0000","last_became_active":"2024-03-06T07:26:08.811309+0000","last_became_peered":"2024-03-06T07:26:08.811309+0000","last_unstale":"2024-03-06T07:26:10.804612+0000","last_undegraded":"2024-03-06T07:26:10.804612+0000","last_fullsized":"2024-03-06T07:26:10.804612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:49.528401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307703+0000","last_change":"2024-03-06T07:26:09.817032+0000","last_active":"2024-03-06T07:26:11.307703+0000","last_peered":"2024-03-06T07:26:11.307703+0000","last_clean":"2024-03-06T07:26:11.307703+0000","last_became_active":"2024-03-06T07:26:09.816744+0000","last_became_peered":"2024-03-06T07:26:09.816744+0000","last_unstale":"2024-03-06T07:26:11.307703+0000","last_undegraded":"2024-03-06T07:26:11.307703+0000","last_fullsized":"2024-03-06T07:26:11.307703+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:50.532823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803867+0000","last_change":"2024-03-06T07:26:08.816283+0000","last_active":"2024-03-06T07:26:10.803867+0000","last_peered":"2024-03-06T07:26:10.803867+0000","last_clean":"2024-03-06T07:26:10.803867+0000","last_became_active":"2024-03-06T07:26:08.815958+0000","last_became_peered":"2024-03-06T07:26:08.815958+0000","last_unstale":"2024-03-06T07:26:10.803867+0000","last_undegraded":"2024-03-06T07:26:10.803867+0000","last_fullsized":"2024-03-06T07:26:10.803867+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:10.336592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803747+0000","last_change":"2024-03-06T07:26:09.817122+0000","last_active":"2024-03-06T07:26:10.803747+0000","last_peered":"2024-03-06T07:26:10.803747+0000","last_clean":"2024-03-06T07:26:10.803747+0000","last_became_active":"2024-03-06T07:26:09.816870+0000","last_became_peered":"2024-03-06T07:26:09.816870+0000","last_unstale":"2024-03-06T07:26:10.803747+0000","last_undegraded":"2024-03-06T07:26:10.803747+0000","last_fullsized":"2024-03-06T07:26:10.803747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803651+0000","last_change":"2024-03-06T07:26:08.813358+0000","last_active":"2024-03-06T07:26:10.803651+0000","last_peered":"2024-03-06T07:26:10.803651+0000","last_clean":"2024-03-06T07:26:10.803651+0000","last_became_active":"2024-03-06T07:26:08.813065+0000","last_became_peered":"2024-03-06T07:26:08.813065+0000","last_unstale":"2024-03-06T07:26:10.803651+0000","last_undegraded":"2024-03-06T07:26:10.803651+0000","last_fullsized":"2024-03-06T07:26:10.803651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:01.075606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804603+0000","last_change":"2024-03-06T07:26:09.818846+0000","last_active":"2024-03-06T07:26:10.804603+0000","last_peered":"2024-03-06T07:26:10.804603+0000","last_clean":"2024-03-06T07:26:10.804603+0000","last_became_active":"2024-03-06T07:26:09.818585+0000","last_became_peered":"2024-03-06T07:26:09.818585+0000","last_unstale":"2024-03-06T07:26:10.804603+0000","last_undegraded":"2024-03-06T07:26:10.804603+0000","last_fullsized":"2024-03-06T07:26:10.804603+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824696+0000","last_change":"2024-03-06T07:26:08.808880+0000","last_active":"2024-03-06T07:26:10.824696+0000","last_peered":"2024-03-06T07:26:10.824696+0000","last_clean":"2024-03-06T07:26:10.824696+0000","last_became_active":"2024-03-06T07:26:08.806760+0000","last_became_peered":"2024-03-06T07:26:08.806760+0000","last_unstale":"2024-03-06T07:26:10.824696+0000","last_undegraded":"2024-03-06T07:26:10.824696+0000","last_fullsized":"2024-03-06T07:26:10.824696+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:25.383967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307252+0000","last_change":"2024-03-06T07:26:09.811502+0000","last_active":"2024-03-06T07:26:11.307252+0000","last_peered":"2024-03-06T07:26:11.307252+0000","last_clean":"2024-03-06T07:26:11.307252+0000","last_became_active":"2024-03-06T07:26:09.811267+0000","last_became_peered":"2024-03-06T07:26:09.811267+0000","last_unstale":"2024-03-06T07:26:11.307252+0000","last_undegraded":"2024-03-06T07:26:11.307252+0000","last_fullsized":"2024-03-06T07:26:11.307252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:47.984176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824775+0000","last_change":"2024-03-06T07:26:08.810238+0000","last_active":"2024-03-06T07:26:10.824775+0000","last_peered":"2024-03-06T07:26:10.824775+0000","last_clean":"2024-03-06T07:26:10.824775+0000","last_became_active":"2024-03-06T07:26:08.809953+0000","last_became_peered":"2024-03-06T07:26:08.809953+0000","last_unstale":"2024-03-06T07:26:10.824775+0000","last_undegraded":"2024-03-06T07:26:10.824775+0000","last_fullsized":"2024-03-06T07:26:10.824775+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:52.715807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802768+0000","last_change":"2024-03-06T07:26:09.818405+0000","last_active":"2024-03-06T07:26:10.802768+0000","last_peered":"2024-03-06T07:26:10.802768+0000","last_clean":"2024-03-06T07:26:10.802768+0000","last_became_active":"2024-03-06T07:26:09.818117+0000","last_became_peered":"2024-03-06T07:26:09.818117+0000","last_unstale":"2024-03-06T07:26:10.802768+0000","last_undegraded":"2024-03-06T07:26:10.802768+0000","last_fullsized":"2024-03-06T07:26:10.802768+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:48.757489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803133+0000","last_change":"2024-03-06T07:26:08.806983+0000","last_active":"2024-03-06T07:26:10.803133+0000","last_peered":"2024-03-06T07:26:10.803133+0000","last_clean":"2024-03-06T07:26:10.803133+0000","last_became_active":"2024-03-06T07:26:08.806690+0000","last_became_peered":"2024-03-06T07:26:08.806690+0000","last_unstale":"2024-03-06T07:26:10.803133+0000","last_undegraded":"2024-03-06T07:26:10.803133+0000","last_fullsized":"2024-03-06T07:26:10.803133+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:18.622715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824443+0000","last_change":"2024-03-06T07:26:09.807025+0000","last_active":"2024-03-06T07:26:10.824443+0000","last_peered":"2024-03-06T07:26:10.824443+0000","last_clean":"2024-03-06T07:26:10.824443+0000","last_became_active":"2024-03-06T07:26:09.806758+0000","last_became_peered":"2024-03-06T07:26:09.806758+0000","last_unstale":"2024-03-06T07:26:10.824443+0000","last_undegraded":"2024-03-06T07:26:10.824443+0000","last_fullsized":"2024-03-06T07:26:10.824443+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:02.489555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307111+0000","last_change":"2024-03-06T07:26:08.809055+0000","last_active":"2024-03-06T07:26:11.307111+0000","last_peered":"2024-03-06T07:26:11.307111+0000","last_clean":"2024-03-06T07:26:11.307111+0000","last_became_active":"2024-03-06T07:26:08.808747+0000","last_became_peered":"2024-03-06T07:26:08.808747+0000","last_unstale":"2024-03-06T07:26:11.307111+0000","last_undegraded":"2024-03-06T07:26:11.307111+0000","last_fullsized":"2024-03-06T07:26:11.307111+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:58.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307875+0000","last_change":"2024-03-06T07:26:09.809108+0000","last_active":"2024-03-06T07:26:11.307875+0000","last_peered":"2024-03-06T07:26:11.307875+0000","last_clean":"2024-03-06T07:26:11.307875+0000","last_became_active":"2024-03-06T07:26:09.808845+0000","last_became_peered":"2024-03-06T07:26:09.808845+0000","last_unstale":"2024-03-06T07:26:11.307875+0000","last_undegraded":"2024-03-06T07:26:11.307875+0000","last_fullsized":"2024-03-06T07:26:11.307875+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:11.993809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803566+0000","last_change":"2024-03-06T07:26:08.814165+0000","last_active":"2024-03-06T07:26:10.803566+0000","last_peered":"2024-03-06T07:26:10.803566+0000","last_clean":"2024-03-06T07:26:10.803566+0000","last_became_active":"2024-03-06T07:26:08.813542+0000","last_became_peered":"2024-03-06T07:26:08.813542+0000","last_unstale":"2024-03-06T07:26:10.803566+0000","last_undegraded":"2024-03-06T07:26:10.803566+0000","last_fullsized":"2024-03-06T07:26:10.803566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:37.062164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307836+0000","last_change":"2024-03-06T07:26:09.816662+0000","last_active":"2024-03-06T07:26:11.307836+0000","last_peered":"2024-03-06T07:26:11.307836+0000","last_clean":"2024-03-06T07:26:11.307836+0000","last_became_active":"2024-03-06T07:26:09.816408+0000","last_became_peered":"2024-03-06T07:26:09.816408+0000","last_unstale":"2024-03-06T07:26:11.307836+0000","last_undegraded":"2024-03-06T07:26:11.307836+0000","last_fullsized":"2024-03-06T07:26:11.307836+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:20.536277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307890+0000","last_change":"2024-03-06T07:26:08.810040+0000","last_active":"2024-03-06T07:26:11.307890+0000","last_peered":"2024-03-06T07:26:11.307890+0000","last_clean":"2024-03-06T07:26:11.307890+0000","last_became_active":"2024-03-06T07:26:08.809759+0000","last_became_peered":"2024-03-06T07:26:08.809759+0000","last_unstale":"2024-03-06T07:26:11.307890+0000","last_undegraded":"2024-03-06T07:26:11.307890+0000","last_fullsized":"2024-03-06T07:26:11.307890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:57.534027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803805+0000","last_change":"2024-03-06T07:26:09.817961+0000","last_active":"2024-03-06T07:26:10.803805+0000","last_peered":"2024-03-06T07:26:10.803805+0000","last_clean":"2024-03-06T07:26:10.803805+0000","last_became_active":"2024-03-06T07:26:09.817654+0000","last_became_peered":"2024-03-06T07:26:09.817654+0000","last_unstale":"2024-03-06T07:26:10.803805+0000","last_undegraded":"2024-03-06T07:26:10.803805+0000","last_fullsized":"2024-03-06T07:26:10.803805+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819490+0000","last_change":"2024-03-06T07:26:08.813115+0000","last_active":"2024-03-06T07:26:10.819490+0000","last_peered":"2024-03-06T07:26:10.819490+0000","last_clean":"2024-03-06T07:26:10.819490+0000","last_became_active":"2024-03-06T07:26:08.812857+0000","last_became_peered":"2024-03-06T07:26:08.812857+0000","last_unstale":"2024-03-06T07:26:10.819490+0000","last_undegraded":"2024-03-06T07:26:10.819490+0000","last_fullsized":"2024-03-06T07:26:10.819490+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:20.520632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824601+0000","last_change":"2024-03-06T07:26:09.811923+0000","last_active":"2024-03-06T07:26:10.824601+0000","last_peered":"2024-03-06T07:26:10.824601+0000","last_clean":"2024-03-06T07:26:10.824601+0000","last_became_active":"2024-03-06T07:26:09.811651+0000","last_became_peered":"2024-03-06T07:26:09.811651+0000","last_unstale":"2024-03-06T07:26:10.824601+0000","last_undegraded":"2024-03-06T07:26:10.824601+0000","last_fullsized":"2024-03-06T07:26:10.824601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:48.819911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819358+0000","last_change":"2024-03-06T07:26:08.816961+0000","last_active":"2024-03-06T07:26:10.819358+0000","last_peered":"2024-03-06T07:26:10.819358+0000","last_clean":"2024-03-06T07:26:10.819358+0000","last_became_active":"2024-03-06T07:26:08.816694+0000","last_became_peered":"2024-03-06T07:26:08.816694+0000","last_unstale":"2024-03-06T07:26:10.819358+0000","last_undegraded":"2024-03-06T07:26:10.819358+0000","last_fullsized":"2024-03-06T07:26:10.819358+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:40.308081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824737+0000","last_change":"2024-03-06T07:26:09.817656+0000","last_active":"2024-03-06T07:26:10.824737+0000","last_peered":"2024-03-06T07:26:10.824737+0000","last_clean":"2024-03-06T07:26:10.824737+0000","last_became_active":"2024-03-06T07:26:09.817006+0000","last_became_peered":"2024-03-06T07:26:09.817006+0000","last_unstale":"2024-03-06T07:26:10.824737+0000","last_undegraded":"2024-03-06T07:26:10.824737+0000","last_fullsized":"2024-03-06T07:26:10.824737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:19.990519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307731+0000","last_change":"2024-03-06T07:26:08.814964+0000","last_active":"2024-03-06T07:26:11.307731+0000","last_peered":"2024-03-06T07:26:11.307731+0000","last_clean":"2024-03-06T07:26:11.307731+0000","last_became_active":"2024-03-06T07:26:08.814713+0000","last_became_peered":"2024-03-06T07:26:08.814713+0000","last_unstale":"2024-03-06T07:26:11.307731+0000","last_undegraded":"2024-03-06T07:26:11.307731+0000","last_fullsized":"2024-03-06T07:26:11.307731+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:41.581348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824464+0000","last_change":"2024-03-06T07:26:09.811939+0000","last_active":"2024-03-06T07:26:10.824464+0000","last_peered":"2024-03-06T07:26:10.824464+0000","last_clean":"2024-03-06T07:26:10.824464+0000","last_became_active":"2024-03-06T07:26:09.811667+0000","last_became_peered":"2024-03-06T07:26:09.811667+0000","last_unstale":"2024-03-06T07:26:10.824464+0000","last_undegraded":"2024-03-06T07:26:10.824464+0000","last_fullsized":"2024-03-06T07:26:10.824464+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:46.742874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307630+0000","last_change":"2024-03-06T07:26:08.813568+0000","last_active":"2024-03-06T07:26:11.307630+0000","last_peered":"2024-03-06T07:26:11.307630+0000","last_clean":"2024-03-06T07:26:11.307630+0000","last_became_active":"2024-03-06T07:26:08.813267+0000","last_became_peered":"2024-03-06T07:26:08.813267+0000","last_unstale":"2024-03-06T07:26:11.307630+0000","last_undegraded":"2024-03-06T07:26:11.307630+0000","last_fullsized":"2024-03-06T07:26:11.307630+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:18.622715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308066+0000","last_change":"2024-03-06T07:26:09.806942+0000","last_active":"2024-03-06T07:26:11.308066+0000","last_peered":"2024-03-06T07:26:11.308066+0000","last_clean":"2024-03-06T07:26:11.308066+0000","last_became_active":"2024-03-06T07:26:09.806694+0000","last_became_peered":"2024-03-06T07:26:09.806694+0000","last_unstale":"2024-03-06T07:26:11.308066+0000","last_undegraded":"2024-03-06T07:26:11.308066+0000","last_fullsized":"2024-03-06T07:26:11.308066+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:02.664185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819198+0000","last_change":"2024-03-06T07:26:08.810885+0000","last_active":"2024-03-06T07:26:10.819198+0000","last_peered":"2024-03-06T07:26:10.819198+0000","last_clean":"2024-03-06T07:26:10.819198+0000","last_became_active":"2024-03-06T07:26:08.810631+0000","last_became_peered":"2024-03-06T07:26:08.810631+0000","last_unstale":"2024-03-06T07:26:10.819198+0000","last_undegraded":"2024-03-06T07:26:10.819198+0000","last_fullsized":"2024-03-06T07:26:10.819198+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:08.732180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819747+0000","last_change":"2024-03-06T07:26:09.819782+0000","last_active":"2024-03-06T07:26:10.819747+0000","last_peered":"2024-03-06T07:26:10.819747+0000","last_clean":"2024-03-06T07:26:10.819747+0000","last_became_active":"2024-03-06T07:26:09.819538+0000","last_became_peered":"2024-03-06T07:26:09.819538+0000","last_unstale":"2024-03-06T07:26:10.819747+0000","last_undegraded":"2024-03-06T07:26:10.819747+0000","last_fullsized":"2024-03-06T07:26:10.819747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:50.283608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308188+0000","last_change":"2024-03-06T07:26:08.813738+0000","last_active":"2024-03-06T07:26:11.308188+0000","last_peered":"2024-03-06T07:26:11.308188+0000","last_clean":"2024-03-06T07:26:11.308188+0000","last_became_active":"2024-03-06T07:26:08.813416+0000","last_became_peered":"2024-03-06T07:26:08.813416+0000","last_unstale":"2024-03-06T07:26:11.308188+0000","last_undegraded":"2024-03-06T07:26:11.308188+0000","last_fullsized":"2024-03-06T07:26:11.308188+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:01.075606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307449+0000","last_change":"2024-03-06T07:26:09.813456+0000","last_active":"2024-03-06T07:26:11.307449+0000","last_peered":"2024-03-06T07:26:11.307449+0000","last_clean":"2024-03-06T07:26:11.307449+0000","last_became_active":"2024-03-06T07:26:09.813219+0000","last_became_peered":"2024-03-06T07:26:09.813219+0000","last_unstale":"2024-03-06T07:26:11.307449+0000","last_undegraded":"2024-03-06T07:26:11.307449+0000","last_fullsized":"2024-03-06T07:26:11.307449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:59.079331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.821601+0000","last_change":"2024-03-06T07:26:08.815514+0000","last_active":"2024-03-06T07:26:10.821601+0000","last_peered":"2024-03-06T07:26:10.821601+0000","last_clean":"2024-03-06T07:26:10.821601+0000","last_became_active":"2024-03-06T07:26:08.815287+0000","last_became_peered":"2024-03-06T07:26:08.815287+0000","last_unstale":"2024-03-06T07:26:10.821601+0000","last_undegraded":"2024-03-06T07:26:10.821601+0000","last_fullsized":"2024-03-06T07:26:10.821601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.888883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307334+0000","last_change":"2024-03-06T07:26:09.809198+0000","last_active":"2024-03-06T07:26:11.307334+0000","last_peered":"2024-03-06T07:26:11.307334+0000","last_clean":"2024-03-06T07:26:11.307334+0000","last_became_active":"2024-03-06T07:26:09.808232+0000","last_became_peered":"2024-03-06T07:26:09.808232+0000","last_unstale":"2024-03-06T07:26:11.307334+0000","last_undegraded":"2024-03-06T07:26:11.307334+0000","last_fullsized":"2024-03-06T07:26:11.307334+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:32.641667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819502+0000","last_change":"2024-03-06T07:26:08.813109+0000","last_active":"2024-03-06T07:26:10.819502+0000","last_peered":"2024-03-06T07:26:10.819502+0000","last_clean":"2024-03-06T07:26:10.819502+0000","last_became_active":"2024-03-06T07:26:08.812841+0000","last_became_peered":"2024-03-06T07:26:08.812841+0000","last_unstale":"2024-03-06T07:26:10.819502+0000","last_undegraded":"2024-03-06T07:26:10.819502+0000","last_fullsized":"2024-03-06T07:26:10.819502+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:40.055990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307839+0000","last_change":"2024-03-06T07:26:09.819549+0000","last_active":"2024-03-06T07:26:11.307839+0000","last_peered":"2024-03-06T07:26:11.307839+0000","last_clean":"2024-03-06T07:26:11.307839+0000","last_became_active":"2024-03-06T07:26:09.819262+0000","last_became_peered":"2024-03-06T07:26:09.819262+0000","last_unstale":"2024-03-06T07:26:11.307839+0000","last_undegraded":"2024-03-06T07:26:11.307839+0000","last_fullsized":"2024-03-06T07:26:11.307839+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:51.051289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307663+0000","last_change":"2024-03-06T07:26:08.807992+0000","last_active":"2024-03-06T07:26:11.307663+0000","last_peered":"2024-03-06T07:26:11.307663+0000","last_clean":"2024-03-06T07:26:11.307663+0000","last_became_active":"2024-03-06T07:26:08.807709+0000","last_became_peered":"2024-03-06T07:26:08.807709+0000","last_unstale":"2024-03-06T07:26:11.307663+0000","last_undegraded":"2024-03-06T07:26:11.307663+0000","last_fullsized":"2024-03-06T07:26:11.307663+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:19.434200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824605+0000","last_change":"2024-03-06T07:26:09.808602+0000","last_active":"2024-03-06T07:26:10.824605+0000","last_peered":"2024-03-06T07:26:10.824605+0000","last_clean":"2024-03-06T07:26:10.824605+0000","last_became_active":"2024-03-06T07:26:09.807997+0000","last_became_peered":"2024-03-06T07:26:09.807997+0000","last_unstale":"2024-03-06T07:26:10.824605+0000","last_undegraded":"2024-03-06T07:26:10.824605+0000","last_fullsized":"2024-03-06T07:26:10.824605+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:59.804615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819434+0000","last_change":"2024-03-06T07:26:08.814261+0000","last_active":"2024-03-06T07:26:10.819434+0000","last_peered":"2024-03-06T07:26:10.819434+0000","last_clean":"2024-03-06T07:26:10.819434+0000","last_became_active":"2024-03-06T07:26:08.813964+0000","last_became_peered":"2024-03-06T07:26:08.813964+0000","last_unstale":"2024-03-06T07:26:10.819434+0000","last_undegraded":"2024-03-06T07:26:10.819434+0000","last_fullsized":"2024-03-06T07:26:10.819434+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:26.197567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803268+0000","last_change":"2024-03-06T07:26:09.806728+0000","last_active":"2024-03-06T07:26:10.803268+0000","last_peered":"2024-03-06T07:26:10.803268+0000","last_clean":"2024-03-06T07:26:10.803268+0000","last_became_active":"2024-03-06T07:26:09.805929+0000","last_became_peered":"2024-03-06T07:26:09.805929+0000","last_unstale":"2024-03-06T07:26:10.803268+0000","last_undegraded":"2024-03-06T07:26:10.803268+0000","last_fullsized":"2024-03-06T07:26:10.803268+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:41.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307729+0000","last_change":"2024-03-06T07:26:08.814662+0000","last_active":"2024-03-06T07:26:11.307729+0000","last_peered":"2024-03-06T07:26:11.307729+0000","last_clean":"2024-03-06T07:26:11.307729+0000","last_became_active":"2024-03-06T07:26:08.814315+0000","last_became_peered":"2024-03-06T07:26:08.814315+0000","last_unstale":"2024-03-06T07:26:11.307729+0000","last_undegraded":"2024-03-06T07:26:11.307729+0000","last_fullsized":"2024-03-06T07:26:11.307729+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:19.531856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824485+0000","last_change":"2024-03-06T07:26:09.810380+0000","last_active":"2024-03-06T07:26:10.824485+0000","last_peered":"2024-03-06T07:26:10.824485+0000","last_clean":"2024-03-06T07:26:10.824485+0000","last_became_active":"2024-03-06T07:26:09.810132+0000","last_became_peered":"2024-03-06T07:26:09.810132+0000","last_unstale":"2024-03-06T07:26:10.824485+0000","last_undegraded":"2024-03-06T07:26:10.824485+0000","last_fullsized":"2024-03-06T07:26:10.824485+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:13.311550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804568+0000","last_change":"2024-03-06T07:26:08.812623+0000","last_active":"2024-03-06T07:26:10.804568+0000","last_peered":"2024-03-06T07:26:10.804568+0000","last_clean":"2024-03-06T07:26:10.804568+0000","last_became_active":"2024-03-06T07:26:08.812312+0000","last_became_peered":"2024-03-06T07:26:08.812312+0000","last_unstale":"2024-03-06T07:26:10.804568+0000","last_undegraded":"2024-03-06T07:26:10.804568+0000","last_fullsized":"2024-03-06T07:26:10.804568+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:45.023392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307906+0000","last_change":"2024-03-06T07:26:09.814986+0000","last_active":"2024-03-06T07:26:11.307906+0000","last_peered":"2024-03-06T07:26:11.307906+0000","last_clean":"2024-03-06T07:26:11.307906+0000","last_became_active":"2024-03-06T07:26:09.814709+0000","last_became_peered":"2024-03-06T07:26:09.814709+0000","last_unstale":"2024-03-06T07:26:11.307906+0000","last_undegraded":"2024-03-06T07:26:11.307906+0000","last_fullsized":"2024-03-06T07:26:11.307906+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:09.051621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824566+0000","last_change":"2024-03-06T07:26:08.814790+0000","last_active":"2024-03-06T07:26:10.824566+0000","last_peered":"2024-03-06T07:26:10.824566+0000","last_clean":"2024-03-06T07:26:10.824566+0000","last_became_active":"2024-03-06T07:26:08.814512+0000","last_became_peered":"2024-03-06T07:26:08.814512+0000","last_unstale":"2024-03-06T07:26:10.824566+0000","last_undegraded":"2024-03-06T07:26:10.824566+0000","last_fullsized":"2024-03-06T07:26:10.824566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:00.263716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824878+0000","last_change":"2024-03-06T07:26:09.812801+0000","last_active":"2024-03-06T07:26:10.824878+0000","last_peered":"2024-03-06T07:26:10.824878+0000","last_clean":"2024-03-06T07:26:10.824878+0000","last_became_active":"2024-03-06T07:26:09.812553+0000","last_became_peered":"2024-03-06T07:26:09.812553+0000","last_unstale":"2024-03-06T07:26:10.824878+0000","last_undegraded":"2024-03-06T07:26:10.824878+0000","last_fullsized":"2024-03-06T07:26:10.824878+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:38.401927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802869+0000","last_change":"2024-03-06T07:26:08.813594+0000","last_active":"2024-03-06T07:26:10.802869+0000","last_peered":"2024-03-06T07:26:10.802869+0000","last_clean":"2024-03-06T07:26:10.802869+0000","last_became_active":"2024-03-06T07:26:08.813342+0000","last_became_peered":"2024-03-06T07:26:08.813342+0000","last_unstale":"2024-03-06T07:26:10.802869+0000","last_undegraded":"2024-03-06T07:26:10.802869+0000","last_fullsized":"2024-03-06T07:26:10.802869+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.888883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":298,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803211+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:26:10.803211+0000","last_peered":"2024-03-06T07:26:10.803211+0000","last_clean":"2024-03-06T07:26:10.803211+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:26:10.803211+0000","last_undegraded":"2024-03-06T07:26:10.803211+0000","last_fullsized":"2024-03-06T07:26:10.803211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":1740800,"data_stored":1074483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36464,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335376,"statfs":{"total":96636764160,"available":96599425024,"internally_reserved":0,"allocated":1146880,"data_stored":484655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36460,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335380,"statfs":{"total":96636764160,"available":96599429120,"internally_reserved":0,"allocated":1142784,"data_stored":484073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1159168,"data_stored":485739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1155072,"data_stored":485637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1142784,"data_stored":484037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":484003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1736704,"data_stored":1074953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:15.073 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-03-06T07:26:15.072+0000 7fee3dd24640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:17.286 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:17.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.385+0000 7f9d518b5640 1 Processor -- start 2024-03-06T07:26:17.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.385+0000 7f9d518b5640 1 -- start start 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.386+0000 7f9d518b5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d4c15dfe0 0x7f9d4c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.386+0000 7f9d518b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c0cf830 0x7f9d4c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.386+0000 7f9d518b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c0b3430 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.386+0000 7f9d518b5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c0d2c50 con 0x7f9d4c15dfe0 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.386+0000 7f9d518b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0b31e0 con 0x7f9d4c0cf830 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.386+0000 7f9d4b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9d4c0d27f0 0x7f9d4c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44022/0 (socket says 172.21.15.77:44022) 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.386+0000 7f9d4b7fe640 1 -- 172.21.15.77:0/3324469084 learned_addr learned my addr 172.21.15.77:0/3324469084 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d4a7fc640 1 --2- 172.21.15.77:0/3324469084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c0cf830 0x7f9d4c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d4affd640 1 --2- 172.21.15.77:0/3324469084 >> v2:172.21.15.158:3300/0 conn(0x7f9d4c15dfe0 0x7f9d4c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d49ffb640 1 -- 172.21.15.77:0/3324469084 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723943163 0 0) 0x7f9d4c0b3430 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d49ffb640 1 -- 172.21.15.77:0/3324469084 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34003520 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d49ffb640 1 -- 172.21.15.77:0/3324469084 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 336326387 0 0) 0x7f9d34003520 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d4a7fc640 1 -- 172.21.15.77:0/3324469084 >> v2:172.21.15.158:3300/0 conn(0x7f9d4c15dfe0 msgr2=0x7f9d4c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d4a7fc640 1 --2- 172.21.15.77:0/3324469084 >> v2:172.21.15.158:3300/0 conn(0x7f9d4c15dfe0 0x7f9d4c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d4a7fc640 1 -- 172.21.15.77:0/3324469084 >> v1:172.21.15.77:6789/0 conn(0x7f9d4c0d27f0 legacy=0x7f9d4c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d4a7fc640 1 -- 172.21.15.77:0/3324469084 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c0b3710 con 0x7f9d4c0cf830 2024-03-06T07:26:17.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.387+0000 7f9d4affd640 1 --2- 172.21.15.77:0/3324469084 >> v2:172.21.15.158:3300/0 conn(0x7f9d4c15dfe0 0x7f9d4c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:17.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d4a7fc640 1 --2- 172.21.15.77:0/3324469084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c0cf830 0x7f9d4c0d22b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9d400097a0 tx=0x7f9d4002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=815177f491f11a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d49ffb640 1 -- 172.21.15.77:0/3324469084 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d40037070 con 0x7f9d4c0cf830 2024-03-06T07:26:17.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d49ffb640 1 -- 172.21.15.77:0/3324469084 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d40030510 con 0x7f9d4c0cf830 2024-03-06T07:26:17.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d49ffb640 1 -- 172.21.15.77:0/3324469084 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c001670 con 0x7f9d4c0cf830 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d518b5640 1 -- 172.21.15.77:0/3324469084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c0cf830 msgr2=0x7f9d4c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d518b5640 1 --2- 172.21.15.77:0/3324469084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c0cf830 0x7f9d4c0d22b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9d400097a0 tx=0x7f9d4002b790 comp rx=0 tx=0).stop 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d518b5640 1 -- 172.21.15.77:0/3324469084 shutdown_connections 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d518b5640 1 --2- 172.21.15.77:0/3324469084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c0cf830 0x7f9d4c0d22b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.388+0000 7f9d518b5640 1 --2- 172.21.15.77:0/3324469084 >> v2:172.21.15.158:3300/0 conn(0x7f9d4c15dfe0 0x7f9d4c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.389+0000 7f9d518b5640 1 -- 172.21.15.77:0/3324469084 >> 172.21.15.77:0/3324469084 conn(0x7f9d4c0d3cd0 msgr2=0x7f9d4c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.389+0000 7f9d518b5640 1 -- 172.21.15.77:0/3324469084 shutdown_connections 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.389+0000 7f9d518b5640 1 -- 172.21.15.77:0/3324469084 wait complete. 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.390+0000 7f9d518b5640 1 Processor -- start 2024-03-06T07:26:17.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.390+0000 7f9d518b5640 1 -- start start 2024-03-06T07:26:17.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.390+0000 7f9d518b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d4c0cf830 0x7f9d4c0b9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.390+0000 7f9d518b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d4c0d27f0 0x7f9d4c160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d518b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c15dfe0 0x7f9d4c160e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d518b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0d1c70 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d518b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0b3d60 con 0x7f9d4c0cf830 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d518b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0d1df0 con 0x7f9d4c15dfe0 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d4c0d27f0 0x7f9d4c160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d4c0d27f0 0x7f9d4c160900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60730/0 (socket says 172.21.15.77:60730) 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 -- 172.21.15.77:0/640818593 learned_addr learned my addr 172.21.15.77:0/640818593 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4affd640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d4c0cf830 0x7f9d4c0b9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4b7fe640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c15dfe0 0x7f9d4c160e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 -- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c15dfe0 msgr2=0x7f9d4c160e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c15dfe0 0x7f9d4c160e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 -- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d4c0cf830 msgr2=0x7f9d4c0b9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d4c0cf830 0x7f9d4c0b9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 -- 172.21.15.77:0/640818593 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c0b8b80 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4b7fe640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c15dfe0 0x7f9d4c160e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:17.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d4a7fc640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d4c0d27f0 0x7f9d4c160900 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f9d4002b7d0 tx=0x7f9d400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d33fff640 1 -- 172.21.15.77:0/640818593 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d40041020 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d33fff640 1 -- 172.21.15.77:0/640818593 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4003c870 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d33fff640 1 -- 172.21.15.77:0/640818593 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d40037040 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c15f860 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.391+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c0b9100 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.392+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c04f330 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.393+0000 7f9d33fff640 1 -- 172.21.15.77:0/640818593 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d40044020 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.393+0000 7f9d33fff640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d3806b4d0 0x7f9d3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.393+0000 7f9d33fff640 1 -- 172.21.15.77:0/640818593 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d400abcc0 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.393+0000 7f9d4affd640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d3806b4d0 0x7f9d3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.393+0000 7f9d4affd640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d3806b4d0 0x7f9d3806d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0ccb40 tx=0x7f9d4400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.398+0000 7f9d33fff640 1 -- 172.21.15.77:0/640818593 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d4004b030 con 0x7f9d4c0d27f0 2024-03-06T07:26:17.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.574+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d4c0b92d0 con 0x7f9d3806b4d0 2024-03-06T07:26:17.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.580+0000 7f9d33fff640 1 -- 172.21.15.77:0/640818593 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251974 (secure 0 0 0) 0x7f9d4c0b92d0 con 0x7f9d3806b4d0 2024-03-06T07:26:17.584 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:17.585 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.582+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d3806b4d0 msgr2=0x7f9d3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.582+0000 7f9d518b5640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d3806b4d0 0x7f9d3806d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0ccb40 tx=0x7f9d4400b040 comp rx=0 tx=0).stop 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.582+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d4c0d27f0 msgr2=0x7f9d4c160900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d4c0d27f0 0x7f9d4c160900 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f9d4002b7d0 tx=0x7f9d400046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 shutdown_connections 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9d3806b4d0 0x7f9d3806d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d4c0d27f0 0x7f9d4c160900 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d4c15dfe0 0x7f9d4c160e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 --2- 172.21.15.77:0/640818593 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d4c0cf830 0x7f9d4c0b9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 >> 172.21.15.77:0/640818593 conn(0x7f9d4c0d3cd0 msgr2=0x7f9d4c0b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 shutdown_connections 2024-03-06T07:26:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.583+0000 7f9d518b5640 1 -- 172.21.15.77:0/640818593 wait complete. 2024-03-06T07:26:17.599 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-03-06T07:26:17.430216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276232,"kb_used_data":10120,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698488,"statfs":{"total":773094113280,"available":772811251712,"internally_reserved":0,"allocated":10362880,"data_stored":5057580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001425"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307447+0000","last_change":"2024-03-06T07:26:09.813853+0000","last_active":"2024-03-06T07:26:11.307447+0000","last_peered":"2024-03-06T07:26:11.307447+0000","last_clean":"2024-03-06T07:26:11.307447+0000","last_became_active":"2024-03-06T07:26:09.813610+0000","last_became_peered":"2024-03-06T07:26:09.813610+0000","last_unstale":"2024-03-06T07:26:11.307447+0000","last_undegraded":"2024-03-06T07:26:11.307447+0000","last_fullsized":"2024-03-06T07:26:11.307447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:31.816194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819608+0000","last_change":"2024-03-06T07:26:09.819767+0000","last_active":"2024-03-06T07:26:10.819608+0000","last_peered":"2024-03-06T07:26:10.819608+0000","last_clean":"2024-03-06T07:26:10.819608+0000","last_became_active":"2024-03-06T07:26:09.819504+0000","last_became_peered":"2024-03-06T07:26:09.819504+0000","last_unstale":"2024-03-06T07:26:10.819608+0000","last_undegraded":"2024-03-06T07:26:10.819608+0000","last_fullsized":"2024-03-06T07:26:10.819608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307400+0000","last_change":"2024-03-06T07:26:09.811069+0000","last_active":"2024-03-06T07:26:11.307400+0000","last_peered":"2024-03-06T07:26:11.307400+0000","last_clean":"2024-03-06T07:26:11.307400+0000","last_became_active":"2024-03-06T07:26:09.810813+0000","last_became_peered":"2024-03-06T07:26:09.810813+0000","last_unstale":"2024-03-06T07:26:11.307400+0000","last_undegraded":"2024-03-06T07:26:11.307400+0000","last_fullsized":"2024-03-06T07:26:11.307400+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:30.253373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804808+0000","last_change":"2024-03-06T07:26:09.816208+0000","last_active":"2024-03-06T07:26:10.804808+0000","last_peered":"2024-03-06T07:26:10.804808+0000","last_clean":"2024-03-06T07:26:10.804808+0000","last_became_active":"2024-03-06T07:26:09.815932+0000","last_became_peered":"2024-03-06T07:26:09.815932+0000","last_unstale":"2024-03-06T07:26:10.804808+0000","last_undegraded":"2024-03-06T07:26:10.804808+0000","last_fullsized":"2024-03-06T07:26:10.804808+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:19.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307777+0000","last_change":"2024-03-06T07:26:09.815170+0000","last_active":"2024-03-06T07:26:11.307777+0000","last_peered":"2024-03-06T07:26:11.307777+0000","last_clean":"2024-03-06T07:26:11.307777+0000","last_became_active":"2024-03-06T07:26:09.814899+0000","last_became_peered":"2024-03-06T07:26:09.814899+0000","last_unstale":"2024-03-06T07:26:11.307777+0000","last_undegraded":"2024-03-06T07:26:11.307777+0000","last_fullsized":"2024-03-06T07:26:11.307777+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:46.027813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802948+0000","last_change":"2024-03-06T07:26:09.819179+0000","last_active":"2024-03-06T07:26:10.802948+0000","last_peered":"2024-03-06T07:26:10.802948+0000","last_clean":"2024-03-06T07:26:10.802948+0000","last_became_active":"2024-03-06T07:26:09.818840+0000","last_became_peered":"2024-03-06T07:26:09.818840+0000","last_unstale":"2024-03-06T07:26:10.802948+0000","last_undegraded":"2024-03-06T07:26:10.802948+0000","last_fullsized":"2024-03-06T07:26:10.802948+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:17.314686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802910+0000","last_change":"2024-03-06T07:26:09.818359+0000","last_active":"2024-03-06T07:26:10.802910+0000","last_peered":"2024-03-06T07:26:10.802910+0000","last_clean":"2024-03-06T07:26:10.802910+0000","last_became_active":"2024-03-06T07:26:09.818091+0000","last_became_peered":"2024-03-06T07:26:09.818091+0000","last_unstale":"2024-03-06T07:26:10.802910+0000","last_undegraded":"2024-03-06T07:26:10.802910+0000","last_fullsized":"2024-03-06T07:26:10.802910+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:14.992810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819210+0000","last_change":"2024-03-06T07:26:09.814692+0000","last_active":"2024-03-06T07:26:10.819210+0000","last_peered":"2024-03-06T07:26:10.819210+0000","last_clean":"2024-03-06T07:26:10.819210+0000","last_became_active":"2024-03-06T07:26:09.814375+0000","last_became_peered":"2024-03-06T07:26:09.814375+0000","last_unstale":"2024-03-06T07:26:10.819210+0000","last_undegraded":"2024-03-06T07:26:10.819210+0000","last_fullsized":"2024-03-06T07:26:10.819210+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803819+0000","last_change":"2024-03-06T07:26:09.818486+0000","last_active":"2024-03-06T07:26:10.803819+0000","last_peered":"2024-03-06T07:26:10.803819+0000","last_clean":"2024-03-06T07:26:10.803819+0000","last_became_active":"2024-03-06T07:26:09.818183+0000","last_became_peered":"2024-03-06T07:26:09.818183+0000","last_unstale":"2024-03-06T07:26:10.803819+0000","last_undegraded":"2024-03-06T07:26:10.803819+0000","last_fullsized":"2024-03-06T07:26:10.803819+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:21.525054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307565+0000","last_change":"2024-03-06T07:26:09.814301+0000","last_active":"2024-03-06T07:26:11.307565+0000","last_peered":"2024-03-06T07:26:11.307565+0000","last_clean":"2024-03-06T07:26:11.307565+0000","last_became_active":"2024-03-06T07:26:09.810981+0000","last_became_peered":"2024-03-06T07:26:09.810981+0000","last_unstale":"2024-03-06T07:26:11.307565+0000","last_undegraded":"2024-03-06T07:26:11.307565+0000","last_fullsized":"2024-03-06T07:26:11.307565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:41.312503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803579+0000","last_change":"2024-03-06T07:26:09.816331+0000","last_active":"2024-03-06T07:26:10.803579+0000","last_peered":"2024-03-06T07:26:10.803579+0000","last_clean":"2024-03-06T07:26:10.803579+0000","last_became_active":"2024-03-06T07:26:09.816061+0000","last_became_peered":"2024-03-06T07:26:09.816061+0000","last_unstale":"2024-03-06T07:26:10.803579+0000","last_undegraded":"2024-03-06T07:26:10.803579+0000","last_fullsized":"2024-03-06T07:26:10.803579+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803958+0000","last_change":"2024-03-06T07:26:09.810779+0000","last_active":"2024-03-06T07:26:10.803958+0000","last_peered":"2024-03-06T07:26:10.803958+0000","last_clean":"2024-03-06T07:26:10.803958+0000","last_became_active":"2024-03-06T07:26:09.810512+0000","last_became_peered":"2024-03-06T07:26:09.810512+0000","last_unstale":"2024-03-06T07:26:10.803958+0000","last_undegraded":"2024-03-06T07:26:10.803958+0000","last_fullsized":"2024-03-06T07:26:10.803958+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:41.312503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804655+0000","last_change":"2024-03-06T07:26:09.808372+0000","last_active":"2024-03-06T07:26:10.804655+0000","last_peered":"2024-03-06T07:26:10.804655+0000","last_clean":"2024-03-06T07:26:10.804655+0000","last_became_active":"2024-03-06T07:26:09.808072+0000","last_became_peered":"2024-03-06T07:26:09.808072+0000","last_unstale":"2024-03-06T07:26:10.804655+0000","last_undegraded":"2024-03-06T07:26:10.804655+0000","last_fullsized":"2024-03-06T07:26:10.804655+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824446+0000","last_change":"2024-03-06T07:26:09.806321+0000","last_active":"2024-03-06T07:26:10.824446+0000","last_peered":"2024-03-06T07:26:10.824446+0000","last_clean":"2024-03-06T07:26:10.824446+0000","last_became_active":"2024-03-06T07:26:09.806070+0000","last_became_peered":"2024-03-06T07:26:09.806070+0000","last_unstale":"2024-03-06T07:26:10.824446+0000","last_undegraded":"2024-03-06T07:26:10.824446+0000","last_fullsized":"2024-03-06T07:26:10.824446+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:05.439904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804472+0000","last_change":"2024-03-06T07:26:09.819527+0000","last_active":"2024-03-06T07:26:10.804472+0000","last_peered":"2024-03-06T07:26:10.804472+0000","last_clean":"2024-03-06T07:26:10.804472+0000","last_became_active":"2024-03-06T07:26:09.819257+0000","last_became_peered":"2024-03-06T07:26:09.819257+0000","last_unstale":"2024-03-06T07:26:10.804472+0000","last_undegraded":"2024-03-06T07:26:10.804472+0000","last_fullsized":"2024-03-06T07:26:10.804472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:17.314686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803146+0000","last_change":"2024-03-06T07:26:09.817278+0000","last_active":"2024-03-06T07:26:10.803146+0000","last_peered":"2024-03-06T07:26:10.803146+0000","last_clean":"2024-03-06T07:26:10.803146+0000","last_became_active":"2024-03-06T07:26:09.817015+0000","last_became_peered":"2024-03-06T07:26:09.817015+0000","last_unstale":"2024-03-06T07:26:10.803146+0000","last_undegraded":"2024-03-06T07:26:10.803146+0000","last_fullsized":"2024-03-06T07:26:10.803146+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:33.879869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819707+0000","last_change":"2024-03-06T07:26:09.824514+0000","last_active":"2024-03-06T07:26:10.819707+0000","last_peered":"2024-03-06T07:26:10.819707+0000","last_clean":"2024-03-06T07:26:10.819707+0000","last_became_active":"2024-03-06T07:26:09.823591+0000","last_became_peered":"2024-03-06T07:26:09.823591+0000","last_unstale":"2024-03-06T07:26:10.819707+0000","last_undegraded":"2024-03-06T07:26:10.819707+0000","last_fullsized":"2024-03-06T07:26:10.819707+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.005755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804704+0000","last_change":"2024-03-06T07:26:09.811053+0000","last_active":"2024-03-06T07:26:10.804704+0000","last_peered":"2024-03-06T07:26:10.804704+0000","last_clean":"2024-03-06T07:26:10.804704+0000","last_became_active":"2024-03-06T07:26:09.810805+0000","last_became_peered":"2024-03-06T07:26:09.810805+0000","last_unstale":"2024-03-06T07:26:10.804704+0000","last_undegraded":"2024-03-06T07:26:10.804704+0000","last_fullsized":"2024-03-06T07:26:10.804704+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824872+0000","last_change":"2024-03-06T07:26:09.809874+0000","last_active":"2024-03-06T07:26:10.824872+0000","last_peered":"2024-03-06T07:26:10.824872+0000","last_clean":"2024-03-06T07:26:10.824872+0000","last_became_active":"2024-03-06T07:26:09.809601+0000","last_became_peered":"2024-03-06T07:26:09.809601+0000","last_unstale":"2024-03-06T07:26:10.824872+0000","last_undegraded":"2024-03-06T07:26:10.824872+0000","last_fullsized":"2024-03-06T07:26:10.824872+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:56.986034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803542+0000","last_change":"2024-03-06T07:26:09.819152+0000","last_active":"2024-03-06T07:26:10.803542+0000","last_peered":"2024-03-06T07:26:10.803542+0000","last_clean":"2024-03-06T07:26:10.803542+0000","last_became_active":"2024-03-06T07:26:09.818889+0000","last_became_peered":"2024-03-06T07:26:09.818889+0000","last_unstale":"2024-03-06T07:26:10.803542+0000","last_undegraded":"2024-03-06T07:26:10.803542+0000","last_fullsized":"2024-03-06T07:26:10.803542+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:49.745227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824557+0000","last_change":"2024-03-06T07:26:09.815573+0000","last_active":"2024-03-06T07:26:10.824557+0000","last_peered":"2024-03-06T07:26:10.824557+0000","last_clean":"2024-03-06T07:26:10.824557+0000","last_became_active":"2024-03-06T07:26:09.815163+0000","last_became_peered":"2024-03-06T07:26:09.815163+0000","last_unstale":"2024-03-06T07:26:10.824557+0000","last_undegraded":"2024-03-06T07:26:10.824557+0000","last_fullsized":"2024-03-06T07:26:10.824557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:08.603270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819256+0000","last_change":"2024-03-06T07:26:09.820441+0000","last_active":"2024-03-06T07:26:10.819256+0000","last_peered":"2024-03-06T07:26:10.819256+0000","last_clean":"2024-03-06T07:26:10.819256+0000","last_became_active":"2024-03-06T07:26:09.820143+0000","last_became_peered":"2024-03-06T07:26:09.820143+0000","last_unstale":"2024-03-06T07:26:10.819256+0000","last_undegraded":"2024-03-06T07:26:10.819256+0000","last_fullsized":"2024-03-06T07:26:10.819256+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819199+0000","last_change":"2024-03-06T07:26:09.819137+0000","last_active":"2024-03-06T07:26:10.819199+0000","last_peered":"2024-03-06T07:26:10.819199+0000","last_clean":"2024-03-06T07:26:10.819199+0000","last_became_active":"2024-03-06T07:26:09.818884+0000","last_became_peered":"2024-03-06T07:26:09.818884+0000","last_unstale":"2024-03-06T07:26:10.819199+0000","last_undegraded":"2024-03-06T07:26:10.819199+0000","last_fullsized":"2024-03-06T07:26:10.819199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803186+0000","last_change":"2024-03-06T07:26:09.815241+0000","last_active":"2024-03-06T07:26:10.803186+0000","last_peered":"2024-03-06T07:26:10.803186+0000","last_clean":"2024-03-06T07:26:10.803186+0000","last_became_active":"2024-03-06T07:26:09.814964+0000","last_became_peered":"2024-03-06T07:26:09.814964+0000","last_unstale":"2024-03-06T07:26:10.803186+0000","last_undegraded":"2024-03-06T07:26:10.803186+0000","last_fullsized":"2024-03-06T07:26:10.803186+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307953+0000","last_change":"2024-03-06T07:26:09.816181+0000","last_active":"2024-03-06T07:26:11.307953+0000","last_peered":"2024-03-06T07:26:11.307953+0000","last_clean":"2024-03-06T07:26:11.307953+0000","last_became_active":"2024-03-06T07:26:09.815917+0000","last_became_peered":"2024-03-06T07:26:09.815917+0000","last_unstale":"2024-03-06T07:26:11.307953+0000","last_undegraded":"2024-03-06T07:26:11.307953+0000","last_fullsized":"2024-03-06T07:26:11.307953+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:41.338253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819765+0000","last_change":"2024-03-06T07:26:09.821323+0000","last_active":"2024-03-06T07:26:10.819765+0000","last_peered":"2024-03-06T07:26:10.819765+0000","last_clean":"2024-03-06T07:26:10.819765+0000","last_became_active":"2024-03-06T07:26:09.821041+0000","last_became_peered":"2024-03-06T07:26:09.821041+0000","last_unstale":"2024-03-06T07:26:10.819765+0000","last_undegraded":"2024-03-06T07:26:10.819765+0000","last_fullsized":"2024-03-06T07:26:10.819765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:11.188598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803635+0000","last_change":"2024-03-06T07:26:09.812299+0000","last_active":"2024-03-06T07:26:10.803635+0000","last_peered":"2024-03-06T07:26:10.803635+0000","last_clean":"2024-03-06T07:26:10.803635+0000","last_became_active":"2024-03-06T07:26:09.812009+0000","last_became_peered":"2024-03-06T07:26:09.812009+0000","last_unstale":"2024-03-06T07:26:10.803635+0000","last_undegraded":"2024-03-06T07:26:10.803635+0000","last_fullsized":"2024-03-06T07:26:10.803635+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:07.673272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307574+0000","last_change":"2024-03-06T07:26:09.821191+0000","last_active":"2024-03-06T07:26:11.307574+0000","last_peered":"2024-03-06T07:26:11.307574+0000","last_clean":"2024-03-06T07:26:11.307574+0000","last_became_active":"2024-03-06T07:26:09.820905+0000","last_became_peered":"2024-03-06T07:26:09.820905+0000","last_unstale":"2024-03-06T07:26:11.307574+0000","last_undegraded":"2024-03-06T07:26:11.307574+0000","last_fullsized":"2024-03-06T07:26:11.307574+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:27.201988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819336+0000","last_change":"2024-03-06T07:26:09.816456+0000","last_active":"2024-03-06T07:26:10.819336+0000","last_peered":"2024-03-06T07:26:10.819336+0000","last_clean":"2024-03-06T07:26:10.819336+0000","last_became_active":"2024-03-06T07:26:09.816206+0000","last_became_peered":"2024-03-06T07:26:09.816206+0000","last_unstale":"2024-03-06T07:26:10.819336+0000","last_undegraded":"2024-03-06T07:26:10.819336+0000","last_fullsized":"2024-03-06T07:26:10.819336+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824471+0000","last_change":"2024-03-06T07:26:09.808861+0000","last_active":"2024-03-06T07:26:10.824471+0000","last_peered":"2024-03-06T07:26:10.824471+0000","last_clean":"2024-03-06T07:26:10.824471+0000","last_became_active":"2024-03-06T07:26:09.808345+0000","last_became_peered":"2024-03-06T07:26:09.808345+0000","last_unstale":"2024-03-06T07:26:10.824471+0000","last_undegraded":"2024-03-06T07:26:10.824471+0000","last_fullsized":"2024-03-06T07:26:10.824471+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:15.906643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804539+0000","last_change":"2024-03-06T07:26:09.821038+0000","last_active":"2024-03-06T07:26:10.804539+0000","last_peered":"2024-03-06T07:26:10.804539+0000","last_clean":"2024-03-06T07:26:10.804539+0000","last_became_active":"2024-03-06T07:26:09.820749+0000","last_became_peered":"2024-03-06T07:26:09.820749+0000","last_unstale":"2024-03-06T07:26:10.804539+0000","last_undegraded":"2024-03-06T07:26:10.804539+0000","last_fullsized":"2024-03-06T07:26:10.804539+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:25.827560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307065+0000","last_change":"2024-03-06T07:26:09.815360+0000","last_active":"2024-03-06T07:26:11.307065+0000","last_peered":"2024-03-06T07:26:11.307065+0000","last_clean":"2024-03-06T07:26:11.307065+0000","last_became_active":"2024-03-06T07:26:09.815125+0000","last_became_peered":"2024-03-06T07:26:09.815125+0000","last_unstale":"2024-03-06T07:26:11.307065+0000","last_undegraded":"2024-03-06T07:26:11.307065+0000","last_fullsized":"2024-03-06T07:26:11.307065+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:58.101050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824887+0000","last_change":"2024-03-06T07:26:09.809611+0000","last_active":"2024-03-06T07:26:10.824887+0000","last_peered":"2024-03-06T07:26:10.824887+0000","last_clean":"2024-03-06T07:26:10.824887+0000","last_became_active":"2024-03-06T07:26:09.809343+0000","last_became_peered":"2024-03-06T07:26:09.809343+0000","last_unstale":"2024-03-06T07:26:10.824887+0000","last_undegraded":"2024-03-06T07:26:10.824887+0000","last_fullsized":"2024-03-06T07:26:10.824887+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:48.800992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819712+0000","last_change":"2024-03-06T07:26:08.809604+0000","last_active":"2024-03-06T07:26:10.819712+0000","last_peered":"2024-03-06T07:26:10.819712+0000","last_clean":"2024-03-06T07:26:10.819712+0000","last_became_active":"2024-03-06T07:26:08.809360+0000","last_became_peered":"2024-03-06T07:26:08.809360+0000","last_unstale":"2024-03-06T07:26:10.819712+0000","last_undegraded":"2024-03-06T07:26:10.819712+0000","last_fullsized":"2024-03-06T07:26:10.819712+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:10.336592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804780+0000","last_change":"2024-03-06T07:26:09.822258+0000","last_active":"2024-03-06T07:26:10.804780+0000","last_peered":"2024-03-06T07:26:10.804780+0000","last_clean":"2024-03-06T07:26:10.804780+0000","last_became_active":"2024-03-06T07:26:09.822015+0000","last_became_peered":"2024-03-06T07:26:09.822015+0000","last_unstale":"2024-03-06T07:26:10.804780+0000","last_undegraded":"2024-03-06T07:26:10.804780+0000","last_fullsized":"2024-03-06T07:26:10.804780+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803187+0000","last_change":"2024-03-06T07:26:08.812691+0000","last_active":"2024-03-06T07:26:10.803187+0000","last_peered":"2024-03-06T07:26:10.803187+0000","last_clean":"2024-03-06T07:26:10.803187+0000","last_became_active":"2024-03-06T07:26:08.812418+0000","last_became_peered":"2024-03-06T07:26:08.812418+0000","last_unstale":"2024-03-06T07:26:10.803187+0000","last_undegraded":"2024-03-06T07:26:10.803187+0000","last_fullsized":"2024-03-06T07:26:10.803187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:20.520632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307299+0000","last_change":"2024-03-06T07:26:09.807945+0000","last_active":"2024-03-06T07:26:11.307299+0000","last_peered":"2024-03-06T07:26:11.307299+0000","last_clean":"2024-03-06T07:26:11.307299+0000","last_became_active":"2024-03-06T07:26:09.807684+0000","last_became_peered":"2024-03-06T07:26:09.807684+0000","last_unstale":"2024-03-06T07:26:11.307299+0000","last_undegraded":"2024-03-06T07:26:11.307299+0000","last_fullsized":"2024-03-06T07:26:11.307299+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:13.507677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819496+0000","last_change":"2024-03-06T07:26:08.816145+0000","last_active":"2024-03-06T07:26:10.819496+0000","last_peered":"2024-03-06T07:26:10.819496+0000","last_clean":"2024-03-06T07:26:10.819496+0000","last_became_active":"2024-03-06T07:26:08.815639+0000","last_became_peered":"2024-03-06T07:26:08.815639+0000","last_unstale":"2024-03-06T07:26:10.819496+0000","last_undegraded":"2024-03-06T07:26:10.819496+0000","last_fullsized":"2024-03-06T07:26:10.819496+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:47.753067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803830+0000","last_change":"2024-03-06T07:26:09.814294+0000","last_active":"2024-03-06T07:26:10.803830+0000","last_peered":"2024-03-06T07:26:10.803830+0000","last_clean":"2024-03-06T07:26:10.803830+0000","last_became_active":"2024-03-06T07:26:09.814010+0000","last_became_peered":"2024-03-06T07:26:09.814010+0000","last_unstale":"2024-03-06T07:26:10.803830+0000","last_undegraded":"2024-03-06T07:26:10.803830+0000","last_fullsized":"2024-03-06T07:26:10.803830+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:37.893304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819440+0000","last_change":"2024-03-06T07:26:08.814364+0000","last_active":"2024-03-06T07:26:10.819440+0000","last_peered":"2024-03-06T07:26:10.819440+0000","last_clean":"2024-03-06T07:26:10.819440+0000","last_became_active":"2024-03-06T07:26:08.813668+0000","last_became_peered":"2024-03-06T07:26:08.813668+0000","last_unstale":"2024-03-06T07:26:10.819440+0000","last_undegraded":"2024-03-06T07:26:10.819440+0000","last_fullsized":"2024-03-06T07:26:10.819440+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:48.740805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307795+0000","last_change":"2024-03-06T07:26:09.812757+0000","last_active":"2024-03-06T07:26:11.307795+0000","last_peered":"2024-03-06T07:26:11.307795+0000","last_clean":"2024-03-06T07:26:11.307795+0000","last_became_active":"2024-03-06T07:26:09.812460+0000","last_became_peered":"2024-03-06T07:26:09.812460+0000","last_unstale":"2024-03-06T07:26:11.307795+0000","last_undegraded":"2024-03-06T07:26:11.307795+0000","last_fullsized":"2024-03-06T07:26:11.307795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:58.727545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307231+0000","last_change":"2024-03-06T07:26:08.809284+0000","last_active":"2024-03-06T07:26:11.307231+0000","last_peered":"2024-03-06T07:26:11.307231+0000","last_clean":"2024-03-06T07:26:11.307231+0000","last_became_active":"2024-03-06T07:26:08.809034+0000","last_became_peered":"2024-03-06T07:26:08.809034+0000","last_unstale":"2024-03-06T07:26:11.307231+0000","last_undegraded":"2024-03-06T07:26:11.307231+0000","last_fullsized":"2024-03-06T07:26:11.307231+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:07.598849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802744+0000","last_change":"2024-03-06T07:26:09.813345+0000","last_active":"2024-03-06T07:26:10.802744+0000","last_peered":"2024-03-06T07:26:10.802744+0000","last_clean":"2024-03-06T07:26:10.802744+0000","last_became_active":"2024-03-06T07:26:09.813115+0000","last_became_peered":"2024-03-06T07:26:09.813115+0000","last_unstale":"2024-03-06T07:26:10.802744+0000","last_undegraded":"2024-03-06T07:26:10.802744+0000","last_fullsized":"2024-03-06T07:26:10.802744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:04.605670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819254+0000","last_change":"2024-03-06T07:26:08.810941+0000","last_active":"2024-03-06T07:26:10.819254+0000","last_peered":"2024-03-06T07:26:10.819254+0000","last_clean":"2024-03-06T07:26:10.819254+0000","last_became_active":"2024-03-06T07:26:08.810678+0000","last_became_peered":"2024-03-06T07:26:08.810678+0000","last_unstale":"2024-03-06T07:26:10.819254+0000","last_undegraded":"2024-03-06T07:26:10.819254+0000","last_fullsized":"2024-03-06T07:26:10.819254+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:49.528401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804522+0000","last_change":"2024-03-06T07:26:09.820600+0000","last_active":"2024-03-06T07:26:10.804522+0000","last_peered":"2024-03-06T07:26:10.804522+0000","last_clean":"2024-03-06T07:26:10.804522+0000","last_became_active":"2024-03-06T07:26:09.820397+0000","last_became_peered":"2024-03-06T07:26:09.820397+0000","last_unstale":"2024-03-06T07:26:10.804522+0000","last_undegraded":"2024-03-06T07:26:10.804522+0000","last_fullsized":"2024-03-06T07:26:10.804522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:33.879869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308023+0000","last_change":"2024-03-06T07:26:08.810527+0000","last_active":"2024-03-06T07:26:11.308023+0000","last_peered":"2024-03-06T07:26:11.308023+0000","last_clean":"2024-03-06T07:26:11.308023+0000","last_became_active":"2024-03-06T07:26:08.810064+0000","last_became_peered":"2024-03-06T07:26:08.810064+0000","last_unstale":"2024-03-06T07:26:11.308023+0000","last_undegraded":"2024-03-06T07:26:11.308023+0000","last_fullsized":"2024-03-06T07:26:11.308023+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:57.723123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802744+0000","last_change":"2024-03-06T07:26:09.817490+0000","last_active":"2024-03-06T07:26:10.802744+0000","last_peered":"2024-03-06T07:26:10.802744+0000","last_clean":"2024-03-06T07:26:10.802744+0000","last_became_active":"2024-03-06T07:26:09.817201+0000","last_became_peered":"2024-03-06T07:26:09.817201+0000","last_unstale":"2024-03-06T07:26:10.802744+0000","last_undegraded":"2024-03-06T07:26:10.802744+0000","last_fullsized":"2024-03-06T07:26:10.802744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:09.736601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307007+0000","last_change":"2024-03-06T07:26:08.813560+0000","last_active":"2024-03-06T07:26:11.307007+0000","last_peered":"2024-03-06T07:26:11.307007+0000","last_clean":"2024-03-06T07:26:11.307007+0000","last_became_active":"2024-03-06T07:26:08.813280+0000","last_became_peered":"2024-03-06T07:26:08.813280+0000","last_unstale":"2024-03-06T07:26:11.307007+0000","last_undegraded":"2024-03-06T07:26:11.307007+0000","last_fullsized":"2024-03-06T07:26:11.307007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:18.986097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819607+0000","last_change":"2024-03-06T07:26:09.815672+0000","last_active":"2024-03-06T07:26:10.819607+0000","last_peered":"2024-03-06T07:26:10.819607+0000","last_clean":"2024-03-06T07:26:10.819607+0000","last_became_active":"2024-03-06T07:26:09.815424+0000","last_became_peered":"2024-03-06T07:26:09.815424+0000","last_unstale":"2024-03-06T07:26:10.819607+0000","last_undegraded":"2024-03-06T07:26:10.819607+0000","last_fullsized":"2024-03-06T07:26:10.819607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:19.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307510+0000","last_change":"2024-03-06T07:26:08.810875+0000","last_active":"2024-03-06T07:26:11.307510+0000","last_peered":"2024-03-06T07:26:11.307510+0000","last_clean":"2024-03-06T07:26:11.307510+0000","last_became_active":"2024-03-06T07:26:08.810627+0000","last_became_peered":"2024-03-06T07:26:08.810627+0000","last_unstale":"2024-03-06T07:26:11.307510+0000","last_undegraded":"2024-03-06T07:26:11.307510+0000","last_fullsized":"2024-03-06T07:26:11.307510+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:47.815489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803882+0000","last_change":"2024-03-06T07:26:09.813845+0000","last_active":"2024-03-06T07:26:10.803882+0000","last_peered":"2024-03-06T07:26:10.803882+0000","last_clean":"2024-03-06T07:26:10.803882+0000","last_became_active":"2024-03-06T07:26:09.813558+0000","last_became_peered":"2024-03-06T07:26:09.813558+0000","last_unstale":"2024-03-06T07:26:10.803882+0000","last_undegraded":"2024-03-06T07:26:10.803882+0000","last_fullsized":"2024-03-06T07:26:10.803882+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:11.341014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804612+0000","last_change":"2024-03-06T07:26:08.811996+0000","last_active":"2024-03-06T07:26:10.804612+0000","last_peered":"2024-03-06T07:26:10.804612+0000","last_clean":"2024-03-06T07:26:10.804612+0000","last_became_active":"2024-03-06T07:26:08.811309+0000","last_became_peered":"2024-03-06T07:26:08.811309+0000","last_unstale":"2024-03-06T07:26:10.804612+0000","last_undegraded":"2024-03-06T07:26:10.804612+0000","last_fullsized":"2024-03-06T07:26:10.804612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:49.528401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307703+0000","last_change":"2024-03-06T07:26:09.817032+0000","last_active":"2024-03-06T07:26:11.307703+0000","last_peered":"2024-03-06T07:26:11.307703+0000","last_clean":"2024-03-06T07:26:11.307703+0000","last_became_active":"2024-03-06T07:26:09.816744+0000","last_became_peered":"2024-03-06T07:26:09.816744+0000","last_unstale":"2024-03-06T07:26:11.307703+0000","last_undegraded":"2024-03-06T07:26:11.307703+0000","last_fullsized":"2024-03-06T07:26:11.307703+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:50.532823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803867+0000","last_change":"2024-03-06T07:26:08.816283+0000","last_active":"2024-03-06T07:26:10.803867+0000","last_peered":"2024-03-06T07:26:10.803867+0000","last_clean":"2024-03-06T07:26:10.803867+0000","last_became_active":"2024-03-06T07:26:08.815958+0000","last_became_peered":"2024-03-06T07:26:08.815958+0000","last_unstale":"2024-03-06T07:26:10.803867+0000","last_undegraded":"2024-03-06T07:26:10.803867+0000","last_fullsized":"2024-03-06T07:26:10.803867+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:10.336592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803747+0000","last_change":"2024-03-06T07:26:09.817122+0000","last_active":"2024-03-06T07:26:10.803747+0000","last_peered":"2024-03-06T07:26:10.803747+0000","last_clean":"2024-03-06T07:26:10.803747+0000","last_became_active":"2024-03-06T07:26:09.816870+0000","last_became_peered":"2024-03-06T07:26:09.816870+0000","last_unstale":"2024-03-06T07:26:10.803747+0000","last_undegraded":"2024-03-06T07:26:10.803747+0000","last_fullsized":"2024-03-06T07:26:10.803747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:30.359519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803651+0000","last_change":"2024-03-06T07:26:08.813358+0000","last_active":"2024-03-06T07:26:10.803651+0000","last_peered":"2024-03-06T07:26:10.803651+0000","last_clean":"2024-03-06T07:26:10.803651+0000","last_became_active":"2024-03-06T07:26:08.813065+0000","last_became_peered":"2024-03-06T07:26:08.813065+0000","last_unstale":"2024-03-06T07:26:10.803651+0000","last_undegraded":"2024-03-06T07:26:10.803651+0000","last_fullsized":"2024-03-06T07:26:10.803651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:01.075606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804603+0000","last_change":"2024-03-06T07:26:09.818846+0000","last_active":"2024-03-06T07:26:10.804603+0000","last_peered":"2024-03-06T07:26:10.804603+0000","last_clean":"2024-03-06T07:26:10.804603+0000","last_became_active":"2024-03-06T07:26:09.818585+0000","last_became_peered":"2024-03-06T07:26:09.818585+0000","last_unstale":"2024-03-06T07:26:10.804603+0000","last_undegraded":"2024-03-06T07:26:10.804603+0000","last_fullsized":"2024-03-06T07:26:10.804603+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:10.602347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824696+0000","last_change":"2024-03-06T07:26:08.808880+0000","last_active":"2024-03-06T07:26:10.824696+0000","last_peered":"2024-03-06T07:26:10.824696+0000","last_clean":"2024-03-06T07:26:10.824696+0000","last_became_active":"2024-03-06T07:26:08.806760+0000","last_became_peered":"2024-03-06T07:26:08.806760+0000","last_unstale":"2024-03-06T07:26:10.824696+0000","last_undegraded":"2024-03-06T07:26:10.824696+0000","last_fullsized":"2024-03-06T07:26:10.824696+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:25.383967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307252+0000","last_change":"2024-03-06T07:26:09.811502+0000","last_active":"2024-03-06T07:26:11.307252+0000","last_peered":"2024-03-06T07:26:11.307252+0000","last_clean":"2024-03-06T07:26:11.307252+0000","last_became_active":"2024-03-06T07:26:09.811267+0000","last_became_peered":"2024-03-06T07:26:09.811267+0000","last_unstale":"2024-03-06T07:26:11.307252+0000","last_undegraded":"2024-03-06T07:26:11.307252+0000","last_fullsized":"2024-03-06T07:26:11.307252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:47.984176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824775+0000","last_change":"2024-03-06T07:26:08.810238+0000","last_active":"2024-03-06T07:26:10.824775+0000","last_peered":"2024-03-06T07:26:10.824775+0000","last_clean":"2024-03-06T07:26:10.824775+0000","last_became_active":"2024-03-06T07:26:08.809953+0000","last_became_peered":"2024-03-06T07:26:08.809953+0000","last_unstale":"2024-03-06T07:26:10.824775+0000","last_undegraded":"2024-03-06T07:26:10.824775+0000","last_fullsized":"2024-03-06T07:26:10.824775+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:52.715807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802768+0000","last_change":"2024-03-06T07:26:09.818405+0000","last_active":"2024-03-06T07:26:10.802768+0000","last_peered":"2024-03-06T07:26:10.802768+0000","last_clean":"2024-03-06T07:26:10.802768+0000","last_became_active":"2024-03-06T07:26:09.818117+0000","last_became_peered":"2024-03-06T07:26:09.818117+0000","last_unstale":"2024-03-06T07:26:10.802768+0000","last_undegraded":"2024-03-06T07:26:10.802768+0000","last_fullsized":"2024-03-06T07:26:10.802768+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:48.757489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803133+0000","last_change":"2024-03-06T07:26:08.806983+0000","last_active":"2024-03-06T07:26:10.803133+0000","last_peered":"2024-03-06T07:26:10.803133+0000","last_clean":"2024-03-06T07:26:10.803133+0000","last_became_active":"2024-03-06T07:26:08.806690+0000","last_became_peered":"2024-03-06T07:26:08.806690+0000","last_unstale":"2024-03-06T07:26:10.803133+0000","last_undegraded":"2024-03-06T07:26:10.803133+0000","last_fullsized":"2024-03-06T07:26:10.803133+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:18.622715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824443+0000","last_change":"2024-03-06T07:26:09.807025+0000","last_active":"2024-03-06T07:26:10.824443+0000","last_peered":"2024-03-06T07:26:10.824443+0000","last_clean":"2024-03-06T07:26:10.824443+0000","last_became_active":"2024-03-06T07:26:09.806758+0000","last_became_peered":"2024-03-06T07:26:09.806758+0000","last_unstale":"2024-03-06T07:26:10.824443+0000","last_undegraded":"2024-03-06T07:26:10.824443+0000","last_fullsized":"2024-03-06T07:26:10.824443+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:02.489555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307111+0000","last_change":"2024-03-06T07:26:08.809055+0000","last_active":"2024-03-06T07:26:11.307111+0000","last_peered":"2024-03-06T07:26:11.307111+0000","last_clean":"2024-03-06T07:26:11.307111+0000","last_became_active":"2024-03-06T07:26:08.808747+0000","last_became_peered":"2024-03-06T07:26:08.808747+0000","last_unstale":"2024-03-06T07:26:11.307111+0000","last_undegraded":"2024-03-06T07:26:11.307111+0000","last_fullsized":"2024-03-06T07:26:11.307111+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:58.800194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307875+0000","last_change":"2024-03-06T07:26:09.809108+0000","last_active":"2024-03-06T07:26:11.307875+0000","last_peered":"2024-03-06T07:26:11.307875+0000","last_clean":"2024-03-06T07:26:11.307875+0000","last_became_active":"2024-03-06T07:26:09.808845+0000","last_became_peered":"2024-03-06T07:26:09.808845+0000","last_unstale":"2024-03-06T07:26:11.307875+0000","last_undegraded":"2024-03-06T07:26:11.307875+0000","last_fullsized":"2024-03-06T07:26:11.307875+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:11.993809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803566+0000","last_change":"2024-03-06T07:26:08.814165+0000","last_active":"2024-03-06T07:26:10.803566+0000","last_peered":"2024-03-06T07:26:10.803566+0000","last_clean":"2024-03-06T07:26:10.803566+0000","last_became_active":"2024-03-06T07:26:08.813542+0000","last_became_peered":"2024-03-06T07:26:08.813542+0000","last_unstale":"2024-03-06T07:26:10.803566+0000","last_undegraded":"2024-03-06T07:26:10.803566+0000","last_fullsized":"2024-03-06T07:26:10.803566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:37.062164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307836+0000","last_change":"2024-03-06T07:26:09.816662+0000","last_active":"2024-03-06T07:26:11.307836+0000","last_peered":"2024-03-06T07:26:11.307836+0000","last_clean":"2024-03-06T07:26:11.307836+0000","last_became_active":"2024-03-06T07:26:09.816408+0000","last_became_peered":"2024-03-06T07:26:09.816408+0000","last_unstale":"2024-03-06T07:26:11.307836+0000","last_undegraded":"2024-03-06T07:26:11.307836+0000","last_fullsized":"2024-03-06T07:26:11.307836+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:20.536277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307890+0000","last_change":"2024-03-06T07:26:08.810040+0000","last_active":"2024-03-06T07:26:11.307890+0000","last_peered":"2024-03-06T07:26:11.307890+0000","last_clean":"2024-03-06T07:26:11.307890+0000","last_became_active":"2024-03-06T07:26:08.809759+0000","last_became_peered":"2024-03-06T07:26:08.809759+0000","last_unstale":"2024-03-06T07:26:11.307890+0000","last_undegraded":"2024-03-06T07:26:11.307890+0000","last_fullsized":"2024-03-06T07:26:11.307890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:57.534027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803805+0000","last_change":"2024-03-06T07:26:09.817961+0000","last_active":"2024-03-06T07:26:10.803805+0000","last_peered":"2024-03-06T07:26:10.803805+0000","last_clean":"2024-03-06T07:26:10.803805+0000","last_became_active":"2024-03-06T07:26:09.817654+0000","last_became_peered":"2024-03-06T07:26:09.817654+0000","last_unstale":"2024-03-06T07:26:10.803805+0000","last_undegraded":"2024-03-06T07:26:10.803805+0000","last_fullsized":"2024-03-06T07:26:10.803805+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:00.537526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819490+0000","last_change":"2024-03-06T07:26:08.813115+0000","last_active":"2024-03-06T07:26:10.819490+0000","last_peered":"2024-03-06T07:26:10.819490+0000","last_clean":"2024-03-06T07:26:10.819490+0000","last_became_active":"2024-03-06T07:26:08.812857+0000","last_became_peered":"2024-03-06T07:26:08.812857+0000","last_unstale":"2024-03-06T07:26:10.819490+0000","last_undegraded":"2024-03-06T07:26:10.819490+0000","last_fullsized":"2024-03-06T07:26:10.819490+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:20.520632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824601+0000","last_change":"2024-03-06T07:26:09.811923+0000","last_active":"2024-03-06T07:26:10.824601+0000","last_peered":"2024-03-06T07:26:10.824601+0000","last_clean":"2024-03-06T07:26:10.824601+0000","last_became_active":"2024-03-06T07:26:09.811651+0000","last_became_peered":"2024-03-06T07:26:09.811651+0000","last_unstale":"2024-03-06T07:26:10.824601+0000","last_undegraded":"2024-03-06T07:26:10.824601+0000","last_fullsized":"2024-03-06T07:26:10.824601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:48.819911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819358+0000","last_change":"2024-03-06T07:26:08.816961+0000","last_active":"2024-03-06T07:26:10.819358+0000","last_peered":"2024-03-06T07:26:10.819358+0000","last_clean":"2024-03-06T07:26:10.819358+0000","last_became_active":"2024-03-06T07:26:08.816694+0000","last_became_peered":"2024-03-06T07:26:08.816694+0000","last_unstale":"2024-03-06T07:26:10.819358+0000","last_undegraded":"2024-03-06T07:26:10.819358+0000","last_fullsized":"2024-03-06T07:26:10.819358+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:40.308081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824737+0000","last_change":"2024-03-06T07:26:09.817656+0000","last_active":"2024-03-06T07:26:10.824737+0000","last_peered":"2024-03-06T07:26:10.824737+0000","last_clean":"2024-03-06T07:26:10.824737+0000","last_became_active":"2024-03-06T07:26:09.817006+0000","last_became_peered":"2024-03-06T07:26:09.817006+0000","last_unstale":"2024-03-06T07:26:10.824737+0000","last_undegraded":"2024-03-06T07:26:10.824737+0000","last_fullsized":"2024-03-06T07:26:10.824737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:19.990519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307731+0000","last_change":"2024-03-06T07:26:08.814964+0000","last_active":"2024-03-06T07:26:11.307731+0000","last_peered":"2024-03-06T07:26:11.307731+0000","last_clean":"2024-03-06T07:26:11.307731+0000","last_became_active":"2024-03-06T07:26:08.814713+0000","last_became_peered":"2024-03-06T07:26:08.814713+0000","last_unstale":"2024-03-06T07:26:11.307731+0000","last_undegraded":"2024-03-06T07:26:11.307731+0000","last_fullsized":"2024-03-06T07:26:11.307731+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:41.581348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824464+0000","last_change":"2024-03-06T07:26:09.811939+0000","last_active":"2024-03-06T07:26:10.824464+0000","last_peered":"2024-03-06T07:26:10.824464+0000","last_clean":"2024-03-06T07:26:10.824464+0000","last_became_active":"2024-03-06T07:26:09.811667+0000","last_became_peered":"2024-03-06T07:26:09.811667+0000","last_unstale":"2024-03-06T07:26:10.824464+0000","last_undegraded":"2024-03-06T07:26:10.824464+0000","last_fullsized":"2024-03-06T07:26:10.824464+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:46.742874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307630+0000","last_change":"2024-03-06T07:26:08.813568+0000","last_active":"2024-03-06T07:26:11.307630+0000","last_peered":"2024-03-06T07:26:11.307630+0000","last_clean":"2024-03-06T07:26:11.307630+0000","last_became_active":"2024-03-06T07:26:08.813267+0000","last_became_peered":"2024-03-06T07:26:08.813267+0000","last_unstale":"2024-03-06T07:26:11.307630+0000","last_undegraded":"2024-03-06T07:26:11.307630+0000","last_fullsized":"2024-03-06T07:26:11.307630+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:18.622715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308066+0000","last_change":"2024-03-06T07:26:09.806942+0000","last_active":"2024-03-06T07:26:11.308066+0000","last_peered":"2024-03-06T07:26:11.308066+0000","last_clean":"2024-03-06T07:26:11.308066+0000","last_became_active":"2024-03-06T07:26:09.806694+0000","last_became_peered":"2024-03-06T07:26:09.806694+0000","last_unstale":"2024-03-06T07:26:11.308066+0000","last_undegraded":"2024-03-06T07:26:11.308066+0000","last_fullsized":"2024-03-06T07:26:11.308066+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:02.664185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819198+0000","last_change":"2024-03-06T07:26:08.810885+0000","last_active":"2024-03-06T07:26:10.819198+0000","last_peered":"2024-03-06T07:26:10.819198+0000","last_clean":"2024-03-06T07:26:10.819198+0000","last_became_active":"2024-03-06T07:26:08.810631+0000","last_became_peered":"2024-03-06T07:26:08.810631+0000","last_unstale":"2024-03-06T07:26:10.819198+0000","last_undegraded":"2024-03-06T07:26:10.819198+0000","last_fullsized":"2024-03-06T07:26:10.819198+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:08.732180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819747+0000","last_change":"2024-03-06T07:26:09.819782+0000","last_active":"2024-03-06T07:26:10.819747+0000","last_peered":"2024-03-06T07:26:10.819747+0000","last_clean":"2024-03-06T07:26:10.819747+0000","last_became_active":"2024-03-06T07:26:09.819538+0000","last_became_peered":"2024-03-06T07:26:09.819538+0000","last_unstale":"2024-03-06T07:26:10.819747+0000","last_undegraded":"2024-03-06T07:26:10.819747+0000","last_fullsized":"2024-03-06T07:26:10.819747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:50.283608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.308188+0000","last_change":"2024-03-06T07:26:08.813738+0000","last_active":"2024-03-06T07:26:11.308188+0000","last_peered":"2024-03-06T07:26:11.308188+0000","last_clean":"2024-03-06T07:26:11.308188+0000","last_became_active":"2024-03-06T07:26:08.813416+0000","last_became_peered":"2024-03-06T07:26:08.813416+0000","last_unstale":"2024-03-06T07:26:11.308188+0000","last_undegraded":"2024-03-06T07:26:11.308188+0000","last_fullsized":"2024-03-06T07:26:11.308188+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:01.075606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307449+0000","last_change":"2024-03-06T07:26:09.813456+0000","last_active":"2024-03-06T07:26:11.307449+0000","last_peered":"2024-03-06T07:26:11.307449+0000","last_clean":"2024-03-06T07:26:11.307449+0000","last_became_active":"2024-03-06T07:26:09.813219+0000","last_became_peered":"2024-03-06T07:26:09.813219+0000","last_unstale":"2024-03-06T07:26:11.307449+0000","last_undegraded":"2024-03-06T07:26:11.307449+0000","last_fullsized":"2024-03-06T07:26:11.307449+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:59.079331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.821601+0000","last_change":"2024-03-06T07:26:08.815514+0000","last_active":"2024-03-06T07:26:10.821601+0000","last_peered":"2024-03-06T07:26:10.821601+0000","last_clean":"2024-03-06T07:26:10.821601+0000","last_became_active":"2024-03-06T07:26:08.815287+0000","last_became_peered":"2024-03-06T07:26:08.815287+0000","last_unstale":"2024-03-06T07:26:10.821601+0000","last_undegraded":"2024-03-06T07:26:10.821601+0000","last_fullsized":"2024-03-06T07:26:10.821601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.888883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307334+0000","last_change":"2024-03-06T07:26:09.809198+0000","last_active":"2024-03-06T07:26:11.307334+0000","last_peered":"2024-03-06T07:26:11.307334+0000","last_clean":"2024-03-06T07:26:11.307334+0000","last_became_active":"2024-03-06T07:26:09.808232+0000","last_became_peered":"2024-03-06T07:26:09.808232+0000","last_unstale":"2024-03-06T07:26:11.307334+0000","last_undegraded":"2024-03-06T07:26:11.307334+0000","last_fullsized":"2024-03-06T07:26:11.307334+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:32.641667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819502+0000","last_change":"2024-03-06T07:26:08.813109+0000","last_active":"2024-03-06T07:26:10.819502+0000","last_peered":"2024-03-06T07:26:10.819502+0000","last_clean":"2024-03-06T07:26:10.819502+0000","last_became_active":"2024-03-06T07:26:08.812841+0000","last_became_peered":"2024-03-06T07:26:08.812841+0000","last_unstale":"2024-03-06T07:26:10.819502+0000","last_undegraded":"2024-03-06T07:26:10.819502+0000","last_fullsized":"2024-03-06T07:26:10.819502+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:40.055990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307839+0000","last_change":"2024-03-06T07:26:09.819549+0000","last_active":"2024-03-06T07:26:11.307839+0000","last_peered":"2024-03-06T07:26:11.307839+0000","last_clean":"2024-03-06T07:26:11.307839+0000","last_became_active":"2024-03-06T07:26:09.819262+0000","last_became_peered":"2024-03-06T07:26:09.819262+0000","last_unstale":"2024-03-06T07:26:11.307839+0000","last_undegraded":"2024-03-06T07:26:11.307839+0000","last_fullsized":"2024-03-06T07:26:11.307839+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:51.051289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307663+0000","last_change":"2024-03-06T07:26:08.807992+0000","last_active":"2024-03-06T07:26:11.307663+0000","last_peered":"2024-03-06T07:26:11.307663+0000","last_clean":"2024-03-06T07:26:11.307663+0000","last_became_active":"2024-03-06T07:26:08.807709+0000","last_became_peered":"2024-03-06T07:26:08.807709+0000","last_unstale":"2024-03-06T07:26:11.307663+0000","last_undegraded":"2024-03-06T07:26:11.307663+0000","last_fullsized":"2024-03-06T07:26:11.307663+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:19.434200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824605+0000","last_change":"2024-03-06T07:26:09.808602+0000","last_active":"2024-03-06T07:26:10.824605+0000","last_peered":"2024-03-06T07:26:10.824605+0000","last_clean":"2024-03-06T07:26:10.824605+0000","last_became_active":"2024-03-06T07:26:09.807997+0000","last_became_peered":"2024-03-06T07:26:09.807997+0000","last_unstale":"2024-03-06T07:26:10.824605+0000","last_undegraded":"2024-03-06T07:26:10.824605+0000","last_fullsized":"2024-03-06T07:26:10.824605+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:59.804615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.819434+0000","last_change":"2024-03-06T07:26:08.814261+0000","last_active":"2024-03-06T07:26:10.819434+0000","last_peered":"2024-03-06T07:26:10.819434+0000","last_clean":"2024-03-06T07:26:10.819434+0000","last_became_active":"2024-03-06T07:26:08.813964+0000","last_became_peered":"2024-03-06T07:26:08.813964+0000","last_unstale":"2024-03-06T07:26:10.819434+0000","last_undegraded":"2024-03-06T07:26:10.819434+0000","last_fullsized":"2024-03-06T07:26:10.819434+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:26.197567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803268+0000","last_change":"2024-03-06T07:26:09.806728+0000","last_active":"2024-03-06T07:26:10.803268+0000","last_peered":"2024-03-06T07:26:10.803268+0000","last_clean":"2024-03-06T07:26:10.803268+0000","last_became_active":"2024-03-06T07:26:09.805929+0000","last_became_peered":"2024-03-06T07:26:09.805929+0000","last_unstale":"2024-03-06T07:26:10.803268+0000","last_undegraded":"2024-03-06T07:26:10.803268+0000","last_fullsized":"2024-03-06T07:26:10.803268+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:41.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307729+0000","last_change":"2024-03-06T07:26:08.814662+0000","last_active":"2024-03-06T07:26:11.307729+0000","last_peered":"2024-03-06T07:26:11.307729+0000","last_clean":"2024-03-06T07:26:11.307729+0000","last_became_active":"2024-03-06T07:26:08.814315+0000","last_became_peered":"2024-03-06T07:26:08.814315+0000","last_unstale":"2024-03-06T07:26:11.307729+0000","last_undegraded":"2024-03-06T07:26:11.307729+0000","last_fullsized":"2024-03-06T07:26:11.307729+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:19.531856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824485+0000","last_change":"2024-03-06T07:26:09.810380+0000","last_active":"2024-03-06T07:26:10.824485+0000","last_peered":"2024-03-06T07:26:10.824485+0000","last_clean":"2024-03-06T07:26:10.824485+0000","last_became_active":"2024-03-06T07:26:09.810132+0000","last_became_peered":"2024-03-06T07:26:09.810132+0000","last_unstale":"2024-03-06T07:26:10.824485+0000","last_undegraded":"2024-03-06T07:26:10.824485+0000","last_fullsized":"2024-03-06T07:26:10.824485+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:13.311550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.804568+0000","last_change":"2024-03-06T07:26:08.812623+0000","last_active":"2024-03-06T07:26:10.804568+0000","last_peered":"2024-03-06T07:26:10.804568+0000","last_clean":"2024-03-06T07:26:10.804568+0000","last_became_active":"2024-03-06T07:26:08.812312+0000","last_became_peered":"2024-03-06T07:26:08.812312+0000","last_unstale":"2024-03-06T07:26:10.804568+0000","last_undegraded":"2024-03-06T07:26:10.804568+0000","last_fullsized":"2024-03-06T07:26:10.804568+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:45.023392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:11.307906+0000","last_change":"2024-03-06T07:26:09.814986+0000","last_active":"2024-03-06T07:26:11.307906+0000","last_peered":"2024-03-06T07:26:11.307906+0000","last_clean":"2024-03-06T07:26:11.307906+0000","last_became_active":"2024-03-06T07:26:09.814709+0000","last_became_peered":"2024-03-06T07:26:09.814709+0000","last_unstale":"2024-03-06T07:26:11.307906+0000","last_undegraded":"2024-03-06T07:26:11.307906+0000","last_fullsized":"2024-03-06T07:26:11.307906+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:09.051621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824566+0000","last_change":"2024-03-06T07:26:08.814790+0000","last_active":"2024-03-06T07:26:10.824566+0000","last_peered":"2024-03-06T07:26:10.824566+0000","last_clean":"2024-03-06T07:26:10.824566+0000","last_became_active":"2024-03-06T07:26:08.814512+0000","last_became_peered":"2024-03-06T07:26:08.814512+0000","last_unstale":"2024-03-06T07:26:10.824566+0000","last_undegraded":"2024-03-06T07:26:10.824566+0000","last_fullsized":"2024-03-06T07:26:10.824566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:00.263716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.824878+0000","last_change":"2024-03-06T07:26:09.812801+0000","last_active":"2024-03-06T07:26:10.824878+0000","last_peered":"2024-03-06T07:26:10.824878+0000","last_clean":"2024-03-06T07:26:10.824878+0000","last_became_active":"2024-03-06T07:26:09.812553+0000","last_became_peered":"2024-03-06T07:26:09.812553+0000","last_unstale":"2024-03-06T07:26:10.824878+0000","last_undegraded":"2024-03-06T07:26:10.824878+0000","last_fullsized":"2024-03-06T07:26:10.824878+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:08.792366+0000","last_clean_scrub_stamp":"2024-03-06T07:26:08.792366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:38.401927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.802869+0000","last_change":"2024-03-06T07:26:08.813594+0000","last_active":"2024-03-06T07:26:10.802869+0000","last_peered":"2024-03-06T07:26:10.802869+0000","last_clean":"2024-03-06T07:26:10.802869+0000","last_became_active":"2024-03-06T07:26:08.813342+0000","last_became_peered":"2024-03-06T07:26:08.813342+0000","last_unstale":"2024-03-06T07:26:10.802869+0000","last_undegraded":"2024-03-06T07:26:10.802869+0000","last_fullsized":"2024-03-06T07:26:10.802869+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:07.787945+0000","last_clean_scrub_stamp":"2024-03-06T07:26:07.787945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.888883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":298,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:10.803211+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:26:10.803211+0000","last_peered":"2024-03-06T07:26:10.803211+0000","last_clean":"2024-03-06T07:26:10.803211+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:26:10.803211+0000","last_undegraded":"2024-03-06T07:26:10.803211+0000","last_fullsized":"2024-03-06T07:26:10.803211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":1740800,"data_stored":1074483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36464,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335376,"statfs":{"total":96636764160,"available":96599425024,"internally_reserved":0,"allocated":1146880,"data_stored":484655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36460,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335380,"statfs":{"total":96636764160,"available":96599429120,"internally_reserved":0,"allocated":1142784,"data_stored":484073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1159168,"data_stored":485739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1155072,"data_stored":485637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1142784,"data_stored":484037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":484003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1736704,"data_stored":1074953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:17.637 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:17.638 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:26:17.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.746+0000 7f9013393640 1 Processor -- start 2024-03-06T07:26:17.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.746+0000 7f9013393640 1 -- start start 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.747+0000 7f9013393640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f900c0719e0 0x7f900c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f9013393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 0x7f900c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f9013393640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c0b3320 con 0x7f900c15a220 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f9013393640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900c0cf610 con 0x7f900c0719e0 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f9013393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900c0728d0 con 0x7f900c06c9f0 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f901092e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f900c0719e0 0x7f900c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f901092e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f900c0719e0 0x7f900c0cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60738/0 (socket says 172.21.15.77:60738) 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f901092e640 1 -- 172.21.15.77:0/4255308723 learned_addr learned my addr 172.21.15.77:0/4255308723 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f901092e640 1 -- 172.21.15.77:0/4255308723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 msgr2=0x7f900c06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f901092e640 1 --2- 172.21.15.77:0/4255308723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 0x7f900c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f901092e640 1 -- 172.21.15.77:0/4255308723 >> v1:172.21.15.158:6789/0 conn(0x7f900c15a220 legacy=0x7f900c0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.748+0000 7f901092e640 1 -- 172.21.15.77:0/4255308723 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c0b3590 con 0x7f900c0719e0 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.749+0000 7f901092e640 1 --2- 172.21.15.77:0/4255308723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c0cf0d0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00aa10 tx=0x7f8ffc02cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c959d3432b3273 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.749+0000 7f9003fff640 1 -- 172.21.15.77:0/4255308723 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc03a070 con 0x7f900c0719e0 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.749+0000 7f9003fff640 1 -- 172.21.15.77:0/4255308723 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc02dae0 con 0x7f900c0719e0 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.749+0000 7f9003fff640 1 -- 172.21.15.77:0/4255308723 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00b040 con 0x7f900c0719e0 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.750+0000 7f9013393640 1 -- 172.21.15.77:0/4255308723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 msgr2=0x7f900c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.750+0000 7f9013393640 1 --2- 172.21.15.77:0/4255308723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c0cf0d0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00aa10 tx=0x7f8ffc02cd30 comp rx=0 tx=0).stop 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.750+0000 7f9013393640 1 -- 172.21.15.77:0/4255308723 shutdown_connections 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.750+0000 7f9013393640 1 --2- 172.21.15.77:0/4255308723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 0x7f900c06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.750+0000 7f9013393640 1 --2- 172.21.15.77:0/4255308723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c0cf0d0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.750+0000 7f9013393640 1 -- 172.21.15.77:0/4255308723 >> 172.21.15.77:0/4255308723 conn(0x7f900c06b190 msgr2=0x7f900c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.750+0000 7f9013393640 1 -- 172.21.15.77:0/4255308723 shutdown_connections 2024-03-06T07:26:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.750+0000 7f9013393640 1 -- 172.21.15.77:0/4255308723 wait complete. 2024-03-06T07:26:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.752+0000 7f9013393640 1 Processor -- start 2024-03-06T07:26:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.752+0000 7f9013393640 1 -- start start 2024-03-06T07:26:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9013393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 0x7f900c145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9013393640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9013393640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f900c15a220 0x7f900c0a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9013393640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c072530 con 0x7f900c0719e0 2024-03-06T07:26:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9013393640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c072030 con 0x7f900c15a220 2024-03-06T07:26:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9013393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900c0726b0 con 0x7f900c06c9f0 2024-03-06T07:26:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c149130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60744/0 (socket says 172.21.15.77:60744) 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 -- 172.21.15.77:0/2990399215 learned_addr learned my addr 172.21.15.77:0/2990399215 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901112f640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 0x7f900c145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9011930640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f900c15a220 0x7f900c0a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 -- 172.21.15.77:0/2990399215 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 msgr2=0x7f900c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 0x7f900c145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 -- 172.21.15.77:0/2990399215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f900c15a220 msgr2=0x7f900c0a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f900c15a220 0x7f900c0a9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 -- 172.21.15.77:0/2990399215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c149670 con 0x7f900c0719e0 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f901092e640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c149130 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc000c00 tx=0x7f8ffc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9001ffb640 1 -- 172.21.15.77:0/2990399215 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc045070 con 0x7f900c0719e0 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.753+0000 7f9001ffb640 1 -- 172.21.15.77:0/2990399215 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc033550 con 0x7f900c0719e0 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.754+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c0ac730 con 0x7f900c0719e0 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.754+0000 7f9001ffb640 1 -- 172.21.15.77:0/2990399215 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc03a070 con 0x7f900c0719e0 2024-03-06T07:26:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.754+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c146e10 con 0x7f900c0719e0 2024-03-06T07:26:17.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.755+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900c0732b0 con 0x7f900c0719e0 2024-03-06T07:26:17.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.755+0000 7f9001ffb640 1 -- 172.21.15.77:0/2990399215 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ffc048070 con 0x7f900c0719e0 2024-03-06T07:26:17.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.756+0000 7f9001ffb640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ff006b4d0 0x7f8ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.756+0000 7f901112f640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ff006b4d0 0x7f8ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.756+0000 7f9001ffb640 1 -- 172.21.15.77:0/2990399215 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ffc0abeb0 con 0x7f900c0719e0 2024-03-06T07:26:17.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.756+0000 7f901112f640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ff006b4d0 0x7f8ff006d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9008009f60 tx=0x7f90080094f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:17.760+0000 7f9001ffb640 1 -- 172.21.15.77:0/2990399215 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ffc07ad30 con 0x7f900c0719e0 2024-03-06T07:26:18.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.186+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f900c0b4840 con 0x7f900c0719e0 2024-03-06T07:26:18.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.190+0000 7f9001ffb640 1 -- 172.21.15.77:0/2990399215 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f8ffc07a630 con 0x7f900c0719e0 2024-03-06T07:26:18.192 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:26:18.192 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:26:18.192 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:26:18.192 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:26:18.192 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:18.192 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:26:18.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ff006b4d0 msgr2=0x7f8ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ff006b4d0 0x7f8ff006d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9008009f60 tx=0x7f90080094f0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 msgr2=0x7f900c149130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c149130 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc000c00 tx=0x7f8ffc004730 comp rx=0 tx=0).stop 2024-03-06T07:26:18.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 shutdown_connections 2024-03-06T07:26:18.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ff006b4d0 0x7f8ff006d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c0719e0 0x7f900c149130 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f900c15a220 0x7f900c0a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 --2- 172.21.15.77:0/2990399215 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f900c06c9f0 0x7f900c145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 >> 172.21.15.77:0/2990399215 conn(0x7f900c06b190 msgr2=0x7f900c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 shutdown_connections 2024-03-06T07:26:18.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.192+0000 7f9013393640 1 -- 172.21.15.77:0/2990399215 wait complete. 2024-03-06T07:26:18.208 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:18.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.321+0000 7fa8cf8c2640 1 Processor -- start 2024-03-06T07:26:18.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.321+0000 7fa8cf8c2640 1 -- start start 2024-03-06T07:26:18.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cf8c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa8c815dfe0 0x7fa8c80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cf8c2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa8c80d29c0 0x7fa8c80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cf8c2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c80b3430 con 0x7fa8c80cf830 2024-03-06T07:26:18.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cf8c2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c80b31e0 con 0x7fa8c815dfe0 2024-03-06T07:26:18.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cf8c2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c80b3d60 con 0x7fa8c80d29c0 2024-03-06T07:26:18.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cd65e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa8c815dfe0 0x7fa8c80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cd65e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa8c815dfe0 0x7fa8c80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60750/0 (socket says 172.21.15.77:60750) 2024-03-06T07:26:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cd65e640 1 -- 172.21.15.77:0/1320440709 learned_addr learned my addr 172.21.15.77:0/1320440709 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cde5f640 1 --2- 172.21.15.77:0/1320440709 >> v2:172.21.15.158:3300/0 conn(0x7fa8c80d29c0 0x7fa8c80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cd65e640 1 -- 172.21.15.77:0/1320440709 >> v2:172.21.15.158:3300/0 conn(0x7fa8c80d29c0 msgr2=0x7fa8c80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cd65e640 1 --2- 172.21.15.77:0/1320440709 >> v2:172.21.15.158:3300/0 conn(0x7fa8c80d29c0 0x7fa8c80b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cd65e640 1 -- 172.21.15.77:0/1320440709 >> v1:172.21.15.158:6789/0 conn(0x7fa8c80cf830 legacy=0x7fa8c80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cd65e640 1 -- 172.21.15.77:0/1320440709 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c80b3710 con 0x7fa8c815dfe0 2024-03-06T07:26:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.322+0000 7fa8cde5f640 1 --2- 172.21.15.77:0/1320440709 >> v2:172.21.15.158:3300/0 conn(0x7fa8c80d29c0 0x7fa8c80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8cd65e640 1 --2- 172.21.15.77:0/1320440709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c815dfe0 0x7fa8c80cf2f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fa8c80b3cd0 tx=0x7fa8b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3a1466373ddae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8c7fff640 1 -- 172.21.15.77:0/1320440709 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b802ca80 con 0x7fa8c815dfe0 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8c7fff640 1 -- 172.21.15.77:0/1320440709 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b802cc20 con 0x7fa8c815dfe0 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8c7fff640 1 -- 172.21.15.77:0/1320440709 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b80343f0 con 0x7fa8c815dfe0 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/1320440709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c815dfe0 msgr2=0x7fa8c80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/1320440709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c815dfe0 0x7fa8c80cf2f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fa8c80b3cd0 tx=0x7fa8b802bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/1320440709 shutdown_connections 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/1320440709 >> v2:172.21.15.158:3300/0 conn(0x7fa8c80d29c0 0x7fa8c80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/1320440709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c815dfe0 0x7fa8c80cf2f0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.323+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/1320440709 >> 172.21.15.77:0/1320440709 conn(0x7fa8c80d3cd0 msgr2=0x7fa8c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.324+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/1320440709 shutdown_connections 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.324+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/1320440709 wait complete. 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cf8c2640 1 Processor -- start 2024-03-06T07:26:18.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cf8c2640 1 -- start start 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cf8c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8c80cf830 0x7fa8c80b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cf8c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c80d29c0 0x7fa8c81611c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cf8c2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8c815dfe0 0x7fa8c81f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cf8c2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80d1df0 con 0x7fa8c80d29c0 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cf8c2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80d1c70 con 0x7fa8c815dfe0 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cf8c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c815e3c0 con 0x7fa8c80cf830 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.325+0000 7fa8cce5d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c80d29c0 0x7fa8c81611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cce5d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c80d29c0 0x7fa8c81611c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60766/0 (socket says 172.21.15.77:60766) 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cce5d640 1 -- 172.21.15.77:0/3833062036 learned_addr learned my addr 172.21.15.77:0/3833062036 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cd65e640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8c80cf830 0x7fa8c80b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cde5f640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8c815dfe0 0x7fa8c81f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cce5d640 1 -- 172.21.15.77:0/3833062036 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8c80cf830 msgr2=0x7fa8c80b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cce5d640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8c80cf830 0x7fa8c80b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cce5d640 1 -- 172.21.15.77:0/3833062036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8c815dfe0 msgr2=0x7fa8c81f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cce5d640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8c815dfe0 0x7fa8c81f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cce5d640 1 -- 172.21.15.77:0/3833062036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c80b7a30 con 0x7fa8c80d29c0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cce5d640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c80d29c0 0x7fa8c81611c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c430 tx=0x7fa8bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8c5ffb640 1 -- 172.21.15.77:0/3833062036 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc011790 con 0x7fa8c80d29c0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8c5ffb640 1 -- 172.21.15.77:0/3833062036 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc017070 con 0x7fa8c80d29c0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c815b900 con 0x7fa8c80d29c0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8c5ffb640 1 -- 172.21.15.77:0/3833062036 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc011b70 con 0x7fa8c80d29c0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.326+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c80b7f80 con 0x7fa8c80d29c0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.327+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c804f330 con 0x7fa8c80d29c0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.328+0000 7fa8c5ffb640 1 -- 172.21.15.77:0/3833062036 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8bc011d10 con 0x7fa8c80d29c0 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.328+0000 7fa8c5ffb640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa8b406b4d0 0x7fa8b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.328+0000 7fa8c5ffb640 1 -- 172.21.15.77:0/3833062036 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8bc08a070 con 0x7fa8c80d29c0 2024-03-06T07:26:18.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.328+0000 7fa8cd65e640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa8b406b4d0 0x7fa8b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.329+0000 7fa8cd65e640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa8b406b4d0 0x7fa8b406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8035000 tx=0x7fa8b8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.333+0000 7fa8c5ffb640 1 -- 172.21.15.77:0/3833062036 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8bc058ef0 con 0x7fa8c80d29c0 2024-03-06T07:26:18.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.545+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8c80d2ea0 con 0x7fa8c80d29c0 2024-03-06T07:26:18.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.546+0000 7fa8c5ffb640 1 -- 172.21.15.77:0/3833062036 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fa8bc0587f0 con 0x7fa8c80d29c0 2024-03-06T07:26:18.548 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:18.548 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6101,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/2526815605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":2526815605},{"type":"v1","addr":"172.21.15.158:6834","nonce":2526815605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6850,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/497683896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":497683896},{"type":"v1","addr":"172.21.15.158:6835","nonce":497683896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":7284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1877423196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1877423196},{"type":"v1","addr":"172.21.15.77:6835","nonce":1877423196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:11.267910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7287":{"gid":7287,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3425851566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3425851566},{"type":"v1","addr":"172.21.15.77:6837","nonce":3425851566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7287,"qdb_cluster":[7287]},"id":7}]} 2024-03-06T07:26:18.548 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 47 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa8b406b4d0 msgr2=0x7fa8b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa8b406b4d0 0x7fa8b406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8035000 tx=0x7fa8b8030000 comp rx=0 tx=0).stop 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c80d29c0 msgr2=0x7fa8c81611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c80d29c0 0x7fa8c81611c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c430 tx=0x7fa8bc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 shutdown_connections 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa8b406b4d0 0x7fa8b406d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8c80d29c0 0x7fa8c81611c0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8c815dfe0 0x7fa8c81f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 --2- 172.21.15.77:0/3833062036 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa8c80cf830 0x7fa8c80b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 >> 172.21.15.77:0/3833062036 conn(0x7fa8c80d3cd0 msgr2=0x7fa8c806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 shutdown_connections 2024-03-06T07:26:18.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:18.548+0000 7fa8cf8c2640 1 -- 172.21.15.77:0/3833062036 wait complete. 2024-03-06T07:26:18.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:09.152591+0000', 'modified': '2024-03-06T07:26:11.267910+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7287': {'gid': 7287, 'name': 'c', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/3425851566', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 3425851566}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 3425851566}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7287, 'qdb_cluster': [7287]} 2024-03-06T07:26:18.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:18.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:26:19.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.066+0000 7fc2a7a42640 1 Processor -- start 2024-03-06T07:26:19.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.066+0000 7fc2a7a42640 1 -- start start 2024-03-06T07:26:19.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc2a7a42640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2a00b2fb0 0x7fc2a01669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc2a7a42640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a015d510 con 0x7fc2a00b4fe0 2024-03-06T07:26:19.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc2a7a42640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a015d780 con 0x7fc2a00b4c00 2024-03-06T07:26:19.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc2a7a42640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a015e0b0 con 0x7fc2a00b2fb0 2024-03-06T07:26:19.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc2a4fdd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc2a00b4fe0 0x7fc2a015ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58936/0 (socket says 172.21.15.77:58936) 2024-03-06T07:26:19.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc2a4fdd640 1 -- 172.21.15.77:0/1056088982 learned_addr learned my addr 172.21.15.77:0/1056088982 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:19.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465351891 0 0) 0x7fc2a015d510 con 0x7fc2a00b4fe0 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28c003520 con 0x7fc2a00b4fe0 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc2a5fdf640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2a00b2fb0 0x7fc2a01669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588772717 0 0) 0x7fc28c003520 con 0x7fc2a00b4fe0 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2a00b2fb0 msgr2=0x7fc2a01669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2a00b2fb0 0x7fc2a01669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 >> v1:172.21.15.158:6789/0 conn(0x7fc2a00b4c00 legacy=0x7fc2a00b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a015da60 con 0x7fc2a00b4fe0 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fc290001670 con 0x7fc2a00b4fe0 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 >> v1:172.21.15.77:6789/0 conn(0x7fc2a00b4fe0 legacy=0x7fc2a015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc28c0039e0 0x7fc28c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2a00b2fb0 0x7fc28c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc28c004d40 0x7fc28c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a015e0b0 con 0x7fc28c0039e0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.067+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c005990 con 0x7fc28c004d40 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c005b90 con 0x7fc2a00b2fb0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc2a57de640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc28c0039e0 0x7fc28c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2a00b2fb0 0x7fc28c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc2a5fdf640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc28c004d40 0x7fc28c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc2a57de640 1 -- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2a00b2fb0 msgr2=0x7fc28c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc2a57de640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc2a00b2fb0 0x7fc28c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc2a57de640 1 -- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc28c004d40 msgr2=0x7fc28c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc2a57de640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc28c004d40 0x7fc28c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.068+0000 7fc2a57de640 1 -- 172.21.15.77:0/1056088982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c003520 con 0x7fc28c0039e0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc2a57de640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc28c0039e0 0x7fc28c003e00 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc29c001750 tx=0x7fc28c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0610b6564e5a213 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c001370 con 0x7fc28c0039e0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c0052d0 con 0x7fc28c0039e0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc297fff640 1 -- 172.21.15.77:0/1056088982 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c003830 con 0x7fc28c0039e0 2024-03-06T07:26:19.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc2a7a42640 1 -- 172.21.15.77:0/1056088982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc28c0039e0 msgr2=0x7fc28c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc2a7a42640 1 --2- 172.21.15.77:0/1056088982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc28c0039e0 0x7fc28c003e00 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc29c001750 tx=0x7fc28c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:19.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc2a5fdf640 1 -- 172.21.15.77:0/1056088982 reap_dead start 2024-03-06T07:26:19.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc2a7a42640 1 -- 172.21.15.77:0/1056088982 shutdown_connections 2024-03-06T07:26:19.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc2a7a42640 1 -- 172.21.15.77:0/1056088982 >> 172.21.15.77:0/1056088982 conn(0x7fc2a009a9f0 msgr2=0x7fc2a009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc2a7a42640 1 -- 172.21.15.77:0/1056088982 shutdown_connections 2024-03-06T07:26:19.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.069+0000 7fc2a7a42640 1 -- 172.21.15.77:0/1056088982 wait complete. 2024-03-06T07:26:19.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.070+0000 7fc2a7a42640 1 Processor -- start 2024-03-06T07:26:19.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a7a42640 1 -- start start 2024-03-06T07:26:19.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a7a42640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc28c0039e0 0x7fc2a015af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a7a42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2a00b1360 0x7fc2a014af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a7a42640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2a01f3700 0x7fc2a0151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a7a42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a015d170 con 0x7fc2a00b1360 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a7a42640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00b33f0 con 0x7fc2a01f3700 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a7a42640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a015d2f0 con 0x7fc28c0039e0 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a4fdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2a00b1360 0x7fc2a014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a4fdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2a00b1360 0x7fc2a014af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37744/0 (socket says 172.21.15.77:37744) 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a4fdd640 1 -- 172.21.15.77:0/2002013410 learned_addr learned my addr 172.21.15.77:0/2002013410 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a5fdf640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2a01f3700 0x7fc2a0151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.071+0000 7fc2a57de640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc28c0039e0 0x7fc2a015af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc2a4fdd640 1 -- 172.21.15.77:0/2002013410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc28c0039e0 msgr2=0x7fc2a015af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc28c0039e0 0x7fc2a015af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc2a4fdd640 1 -- 172.21.15.77:0/2002013410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2a01f3700 msgr2=0x7fc2a0151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2a01f3700 0x7fc2a0151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc2a4fdd640 1 -- 172.21.15.77:0/2002013410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00b18d0 con 0x7fc2a00b1360 2024-03-06T07:26:19.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2a00b1360 0x7fc2a014af10 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e8a0 tx=0x7fc29000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc295ffb640 1 -- 172.21.15.77:0/2002013410 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000c850 con 0x7fc2a00b1360 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc295ffb640 1 -- 172.21.15.77:0/2002013410 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc290018070 con 0x7fc2a00b1360 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc2a7a42640 1 -- 172.21.15.77:0/2002013410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a00b1bb0 con 0x7fc2a00b1360 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc295ffb640 1 -- 172.21.15.77:0/2002013410 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000cc50 con 0x7fc2a00b1360 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.072+0000 7fc2a7a42640 1 -- 172.21.15.77:0/2002013410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a01f31e0 con 0x7fc2a00b1360 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.073+0000 7fc295ffb640 1 -- 172.21.15.77:0/2002013410 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc29001d460 con 0x7fc2a00b1360 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.074+0000 7fc295ffb640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc28406b4d0 0x7fc28406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.074+0000 7fc295ffb640 1 -- 172.21.15.77:0/2002013410 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc29008a1a0 con 0x7fc2a00b1360 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.074+0000 7fc2a57de640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc28406b4d0 0x7fc28406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.074+0000 7fc2a57de640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc28406b4d0 0x7fc28406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc29c033000 tx=0x7fc29c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.075+0000 7fc2aa61f740 1 Processor -- start 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.075+0000 7fc2aa61f740 1 -- start start 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560031074870 0x560031074c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5600310788c0 0x56003107a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5600310a4600 0x5600310a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560030fde6b0 con 0x560031074870 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x560030f87a60 con 0x5600310788c0 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x560030fda750 con 0x5600310a4600 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560031074870 0x560031074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560031074870 0x560031074c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37750/0 (socket says 172.21.15.77:37750) 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 -- 172.21.15.77:0/177645194 learned_addr learned my addr 172.21.15.77:0/177645194 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 -- 172.21.15.77:0/177645194 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5600310a4600 msgr2=0x5600310a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 --2- 172.21.15.77:0/177645194 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5600310a4600 0x5600310a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 -- 172.21.15.77:0/177645194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5600310788c0 msgr2=0x56003107a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 --2- 172.21.15.77:0/177645194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5600310788c0 0x56003107a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 -- 172.21.15.77:0/177645194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560030fa5f30 con 0x560031074870 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc2a57de640 1 --2- 172.21.15.77:0/177645194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560031074870 0x560031074c50 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc29c001840 tx=0x7fc29c02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46aa3b2b89bb797 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc28affd640 1 -- 172.21.15.77:0/177645194 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c042030 con 0x560031074870 2024-03-06T07:26:19.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc28affd640 1 -- 172.21.15.77:0/177645194 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c04e070 con 0x560031074870 2024-03-06T07:26:19.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.076+0000 7fc28affd640 1 -- 172.21.15.77:0/177645194 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c0075c0 con 0x560031074870 2024-03-06T07:26:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 -- 172.21.15.77:0/177645194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560031074870 msgr2=0x560031074c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/177645194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560031074870 0x560031074c50 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc29c001840 tx=0x7fc29c02f000 comp rx=0 tx=0).stop 2024-03-06T07:26:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 -- 172.21.15.77:0/177645194 shutdown_connections 2024-03-06T07:26:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/177645194 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5600310a4600 0x5600310a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/177645194 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5600310788c0 0x56003107a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/177645194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560031074870 0x560031074c50 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 -- 172.21.15.77:0/177645194 >> 172.21.15.77:0/177645194 conn(0x5600310768f0 msgr2=0x5600310781b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 -- 172.21.15.77:0/177645194 shutdown_connections 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.077+0000 7fc2aa61f740 1 -- 172.21.15.77:0/177645194 wait complete. 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.078+0000 7fc2aa61f740 1 -- start start 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.078+0000 7fc2aa61f740 1 Processor -- start 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x56003108d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109efe0 0x5600310a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109f410 0x5600310a7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560031078d50 con 0x5600310a4600 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x560030f9aea0 con 0x56003109f410 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x560031079030 con 0x56003109efe0 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 10 client.0 _ll_register_callbacks cb 0x560030f9a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x56003108d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x56003108d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37758/0 (socket says 172.21.15.77:37758) 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 -- 172.21.15.77:0/970713555 learned_addr learned my addr 172.21.15.77:0/970713555 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a5fdf640 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109f410 0x5600310a7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109efe0 0x5600310a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 -- 172.21.15.77:0/970713555 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109efe0 msgr2=0x5600310a78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109efe0 0x5600310a78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 -- 172.21.15.77:0/970713555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109f410 msgr2=0x5600310a7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109f410 0x5600310a7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 -- 172.21.15.77:0/970713555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560030f9b270 con 0x5600310a4600 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2a57de640 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x56003108d9a0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc29c04b040 tx=0x7fc29c001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6876d66d024c151 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc27ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc27ffff640 1 -- 172.21.15.77:0/970713555 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c042040 con 0x5600310a4600 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc27ffff640 1 -- 172.21.15.77:0/970713555 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc29c0035f0 con 0x5600310a4600 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 1 -- 172.21.15.77:0/970713555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc29c04e070 con 0x5600310a4600 2024-03-06T07:26:19.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc27ffff640 1 -- 172.21.15.77:0/970713555 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc29c063050 con 0x5600310a4600 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 10 client.7464 fetch_fsmap learned FSMap version 47 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.079+0000 7fc2aa61f740 1 -- 172.21.15.77:0/970713555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x560030cf32d0 con 0x5600310a4600 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc27ffff640 1 -- 172.21.15.77:0/970713555 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c04e070 con 0x5600310a4600 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc27ffff640 1 -- 172.21.15.77:0/970713555 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc29c066020 con 0x5600310a4600 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc2aa61f740 10 client.7464 fetch_fsmap finished waiting for FSMap version 47 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc2aa61f740 10 client.7464 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc2aa61f740 20 client.7464 populate_metadata read hostname 'smithi077' 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] conn(0x5600310a6a00 0x5600310a6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc2aa61f740 4 client.7464 mds_command: new command op to 7287 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc2aa61f740 1 -- 172.21.15.77:0/970713555 --> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560030febbf0 con 0x5600310a6a00 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] conn(0x5600310a6a00 0x5600310a6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] conn(0x5600310a6a00 0x5600310a6de0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d6d45240c1a774 server_cookie=f73a9d2a7561fd9 in_seq=0 out_seq=0 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.080+0000 7fc27ffff640 10 client.7464 ms_handle_connect on v2:172.21.15.77:6836/3425851566 2024-03-06T07:26:19.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc27ffff640 1 -- 172.21.15.77:0/970713555 <== mds.0 v2:172.21.15.77:6836/3425851566 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560030febbf0 con 0x5600310a6a00 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc27ffff640 10 client.7464 handle_command_reply: tid=0 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 client.7464 shutdown 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 -- 172.21.15.77:0/970713555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 msgr2=0x56003108d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x56003108d9a0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc29c04b040 tx=0x7fc29c001190 comp rx=0 tx=0).stop 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 -- 172.21.15.77:0/970713555 shutdown_connections 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] conn(0x5600310a6a00 0x5600310a6de0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109f410 0x5600310a7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109efe0 0x5600310a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc27ffff640 0 client.7464 ms_handle_reset on v2:172.21.15.77:6836/3425851566 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/970713555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x56003108d9a0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 -- 172.21.15.77:0/970713555 >> 172.21.15.77:0/970713555 conn(0x560031074870 msgr2=0x560031073510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 -- 172.21.15.77:0/970713555 shutdown_connections 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 1 -- 172.21.15.77:0/970713555 wait complete. 2024-03-06T07:26:19.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.081+0000 7fc2aa61f740 20 client.7464 trim_cache size 0 max 16384 2024-03-06T07:26:19.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.113+0000 7fc2aa61f740 1 Processor -- start 2024-03-06T07:26:19.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.113+0000 7fc2aa61f740 1 -- start start 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109efe0 0x5600310bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109f410 0x5600310b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x5600310b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a01fb5b0 con 0x5600310a4600 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a01f0370 con 0x56003109efe0 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0159af0 con 0x56003109f410 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2a5fdf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x5600310b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2a5fdf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x5600310b94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37774/0 (socket says 172.21.15.77:37774) 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.114+0000 7fc2a5fdf640 1 -- 172.21.15.77:0/2615732027 learned_addr learned my addr 172.21.15.77:0/2615732027 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc2a5fdf640 1 -- 172.21.15.77:0/2615732027 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109f410 msgr2=0x5600310b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc2a5fdf640 1 --2- 172.21.15.77:0/2615732027 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109f410 0x5600310b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc2a5fdf640 1 -- 172.21.15.77:0/2615732027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109efe0 msgr2=0x5600310bb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc2a57de640 1 --2- 172.21.15.77:0/2615732027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109efe0 0x5600310bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc2a5fdf640 1 --2- 172.21.15.77:0/2615732027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109efe0 0x5600310bb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc2a5fdf640 1 -- 172.21.15.77:0/2615732027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560030dbd6b0 con 0x5600310a4600 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc2a5fdf640 1 --2- 172.21.15.77:0/2615732027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x5600310b94f0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc298009d00 tx=0x7fc298013620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474cf999fffb4440 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc28a7fc640 1 -- 172.21.15.77:0/2615732027 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29801b070 con 0x5600310a4600 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc28a7fc640 1 -- 172.21.15.77:0/2615732027 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298017430 con 0x5600310a4600 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.115+0000 7fc28a7fc640 1 -- 172.21.15.77:0/2615732027 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298017800 con 0x5600310a4600 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 -- 172.21.15.77:0/2615732027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 msgr2=0x5600310b94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/2615732027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x5600310b94f0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc298009d00 tx=0x7fc298013620 comp rx=0 tx=0).stop 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 -- 172.21.15.77:0/2615732027 shutdown_connections 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/2615732027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600310a4600 0x5600310b94f0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/2615732027 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109f410 0x5600310b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/2615732027 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x56003109efe0 0x5600310bb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 -- 172.21.15.77:0/2615732027 >> 172.21.15.77:0/2615732027 conn(0x560031074870 msgr2=0x56003108d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 -- 172.21.15.77:0/2615732027 shutdown_connections 2024-03-06T07:26:19.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 -- 172.21.15.77:0/2615732027 wait complete. 2024-03-06T07:26:19.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 -- start start 2024-03-06T07:26:19.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.116+0000 7fc2aa61f740 1 Processor -- start 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56003109efe0 0x5600310bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109f410 0x56003112ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2aa61f740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5600310a4600 0x5600310a1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560031078d50 con 0x56003109efe0 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x560030f9aea0 con 0x5600310a4600 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2aa61f740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600310a5770 con 0x56003109f410 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2aa61f740 10 client.0 _ll_register_callbacks cb 0x7fc29c0015f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56003109efe0 0x5600310bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56003109efe0 0x5600310bb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37788/0 (socket says 172.21.15.77:37788) 2024-03-06T07:26:19.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 -- 172.21.15.77:0/3228892894 learned_addr learned my addr 172.21.15.77:0/3228892894 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 -- 172.21.15.77:0/3228892894 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109f410 msgr2=0x56003112ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109f410 0x56003112ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 -- 172.21.15.77:0/3228892894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5600310a4600 msgr2=0x5600310a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5600310a4600 0x5600310a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 -- 172.21.15.77:0/3228892894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56003112f850 con 0x56003109efe0 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2a57de640 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56003109efe0 0x5600310bb4e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fc29c007a80 tx=0x7fc29c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587639b411b52b5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc27ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc27ffff640 1 -- 172.21.15.77:0/3228892894 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c06b070 con 0x56003109efe0 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc27ffff640 1 -- 172.21.15.77:0/3228892894 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc29c003a00 con 0x56003109efe0 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.118+0000 7fc2aa61f740 1 -- 172.21.15.77:0/3228892894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc298017430 con 0x56003109efe0 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc27ffff640 1 -- 172.21.15.77:0/3228892894 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc29c076050 con 0x56003109efe0 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 10 client.7470 fetch_fsmap learned FSMap version 47 2024-03-06T07:26:19.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 1 -- 172.21.15.77:0/3228892894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x56003112f4c0 con 0x56003109efe0 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc27ffff640 1 -- 172.21.15.77:0/3228892894 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298017430 con 0x56003109efe0 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc27ffff640 1 -- 172.21.15.77:0/3228892894 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc29c079020 con 0x56003109efe0 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 10 client.7470 fetch_fsmap finished waiting for FSMap version 47 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 10 client.7470 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 20 client.7470 populate_metadata read hostname 'smithi077' 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] conn(0x560031079b50 0x5600310a3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 4 client.7470 mds_command: new command op to 7287 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 1 -- 172.21.15.77:0/3228892894 --> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560030fe5f40 con 0x560031079b50 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] conn(0x560031079b50 0x5600310a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc27ffff640 10 client.7470 ms_handle_connect on v2:172.21.15.77:6836/3425851566 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2a4fdd640 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] conn(0x560031079b50 0x5600310a3390 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3abecd8a9e564abd server_cookie=e0a6a36cb5cf7b6f in_seq=0 out_seq=0 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc27ffff640 1 -- 172.21.15.77:0/3228892894 <== mds.0 v2:172.21.15.77:6836/3425851566 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x560030fe5f40 con 0x560031079b50 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc27ffff640 10 client.7470 handle_command_reply: tid=0 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.119+0000 7fc2aa61f740 1 client.7470 shutdown 2024-03-06T07:26:19.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 -- 172.21.15.77:0/3228892894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56003109efe0 msgr2=0x5600310bb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56003109efe0 0x5600310bb4e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fc29c007a80 tx=0x7fc29c030000 comp rx=0 tx=0).stop 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 -- 172.21.15.77:0/3228892894 shutdown_connections 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.77:6836/3425851566,v1:172.21.15.77:6837/3425851566] conn(0x560031079b50 0x5600310a3390 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5600310a4600 0x5600310a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x56003109f410 0x56003112ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc27ffff640 0 client.7470 ms_handle_reset on v2:172.21.15.77:6836/3425851566 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 --2- 172.21.15.77:0/3228892894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56003109efe0 0x5600310bb4e0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 -- 172.21.15.77:0/3228892894 >> 172.21.15.77:0/3228892894 conn(0x560031074870 msgr2=0x56003108d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 -- 172.21.15.77:0/3228892894 shutdown_connections 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 1 -- 172.21.15.77:0/3228892894 wait complete. 2024-03-06T07:26:19.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2aa61f740 20 client.7470 trim_cache size 0 max 16384 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 7287, 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 9.3099709809999993, 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 47, 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 92, 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 92, 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 14.377247950999999 2024-03-06T07:26:19.128 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:26:19.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 -- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc28406b4d0 msgr2=0x7fc28406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc28406b4d0 0x7fc28406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc29c033000 tx=0x7fc29c009030 comp rx=0 tx=0).stop 2024-03-06T07:26:19.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 -- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2a00b1360 msgr2=0x7fc2a014af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2a00b1360 0x7fc2a014af10 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e8a0 tx=0x7fc29000ed60 comp rx=0 tx=0).stop 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 -- 172.21.15.77:0/2002013410 shutdown_connections 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc28406b4d0 0x7fc28406d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2a00b1360 0x7fc2a014af10 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc2a01f3700 0x7fc2a0151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 --2- 172.21.15.77:0/2002013410 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc28c0039e0 0x7fc2a015af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 -- 172.21.15.77:0/2002013410 >> 172.21.15.77:0/2002013410 conn(0x7fc2a009a9f0 msgr2=0x7fc2a006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.120+0000 7fc2a7a42640 1 -- 172.21.15.77:0/2002013410 shutdown_connections 2024-03-06T07:26:19.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.121+0000 7fc2a7a42640 1 -- 172.21.15.77:0/2002013410 wait complete. 2024-03-06T07:26:19.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:26:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.232+0000 7fd74e8af640 1 Processor -- start 2024-03-06T07:26:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.232+0000 7fd74e8af640 1 -- start start 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd74e8af640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd74815dfe0 0x7fd7480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd74e8af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd7480d29c0 0x7fd7480b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd74e8af640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7480b3430 con 0x7fd7480cf830 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd74e8af640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7480b31e0 con 0x7fd74815dfe0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd74e8af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7480b3d60 con 0x7fd7480d29c0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd747fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd74815dfe0 0x7fd7480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd747fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd74815dfe0 0x7fd7480cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37804/0 (socket says 172.21.15.77:37804) 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd747fff640 1 -- 172.21.15.77:0/177272261 learned_addr learned my addr 172.21.15.77:0/177272261 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd74ce4c640 1 --2- 172.21.15.77:0/177272261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd7480d29c0 0x7fd7480b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd747fff640 1 -- 172.21.15.77:0/177272261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd7480d29c0 msgr2=0x7fd7480b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd747fff640 1 --2- 172.21.15.77:0/177272261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd7480d29c0 0x7fd7480b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd747fff640 1 -- 172.21.15.77:0/177272261 >> v1:172.21.15.158:6789/0 conn(0x7fd7480cf830 legacy=0x7fd7480d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd747fff640 1 -- 172.21.15.77:0/177272261 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7480b3710 con 0x7fd74815dfe0 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.233+0000 7fd74ce4c640 1 --2- 172.21.15.77:0/177272261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd7480d29c0 0x7fd7480b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.234+0000 7fd747fff640 1 --2- 172.21.15.77:0/177272261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd74815dfe0 0x7fd7480cf2f0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fd738009870 tx=0x7fd73802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b665b012c29253a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.234+0000 7fd746ffd640 1 -- 172.21.15.77:0/177272261 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738035020 con 0x7fd74815dfe0 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.234+0000 7fd746ffd640 1 -- 172.21.15.77:0/177272261 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd738031640 con 0x7fd74815dfe0 2024-03-06T07:26:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.234+0000 7fd746ffd640 1 -- 172.21.15.77:0/177272261 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73802d040 con 0x7fd74815dfe0 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.235+0000 7fd74e8af640 1 -- 172.21.15.77:0/177272261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd74815dfe0 msgr2=0x7fd7480cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.235+0000 7fd74e8af640 1 --2- 172.21.15.77:0/177272261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd74815dfe0 0x7fd7480cf2f0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fd738009870 tx=0x7fd73802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.235+0000 7fd74e8af640 1 -- 172.21.15.77:0/177272261 shutdown_connections 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.235+0000 7fd74e8af640 1 --2- 172.21.15.77:0/177272261 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd7480d29c0 0x7fd7480b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.235+0000 7fd74e8af640 1 --2- 172.21.15.77:0/177272261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd74815dfe0 0x7fd7480cf2f0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.235+0000 7fd74e8af640 1 -- 172.21.15.77:0/177272261 >> 172.21.15.77:0/177272261 conn(0x7fd7480d3cd0 msgr2=0x7fd7480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.235+0000 7fd74e8af640 1 -- 172.21.15.77:0/177272261 shutdown_connections 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.235+0000 7fd74e8af640 1 -- 172.21.15.77:0/177272261 wait complete. 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.236+0000 7fd74e8af640 1 Processor -- start 2024-03-06T07:26:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.236+0000 7fd74e8af640 1 -- start start 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd74e8af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7480cf830 0x7fd7481547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd74e8af640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7480d29c0 0x7fd74815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd74e8af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd74815dfe0 0x7fd7480b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd74e8af640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7480d1df0 con 0x7fd7480cf830 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd74e8af640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7480d1c70 con 0x7fd7480d29c0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd74e8af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74815e3c0 con 0x7fd74815dfe0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7480cf830 0x7fd7481547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7480cf830 0x7fd7481547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37806/0 (socket says 172.21.15.77:37806) 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 -- 172.21.15.77:0/3339481371 learned_addr learned my addr 172.21.15.77:0/3339481371 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 -- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd74815dfe0 msgr2=0x7fd7480b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd74ce4c640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd74815dfe0 0x7fd7480b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd7477fe640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7480d29c0 0x7fd74815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd74815dfe0 0x7fd7480b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 -- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7480d29c0 msgr2=0x7fd74815fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7480d29c0 0x7fd74815fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 -- 172.21.15.77:0/3339481371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7480b9b60 con 0x7fd7480cf830 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd74ce4c640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd74815dfe0 0x7fd7480b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd7477fe640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7480d29c0 0x7fd74815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd747fff640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7480cf830 0x7fd7481547c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fd73802b930 tx=0x7fd7380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.237+0000 7fd744ff9640 1 -- 172.21.15.77:0/3339481371 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738040070 con 0x7fd7480cf830 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.238+0000 7fd744ff9640 1 -- 172.21.15.77:0/3339481371 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd738039c10 con 0x7fd7480cf830 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.238+0000 7fd744ff9640 1 -- 172.21.15.77:0/3339481371 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738035040 con 0x7fd7480cf830 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.238+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7480b92c0 con 0x7fd7480cf830 2024-03-06T07:26:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.238+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7480b9880 con 0x7fd7480cf830 2024-03-06T07:26:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.239+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74804f330 con 0x7fd7480cf830 2024-03-06T07:26:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.239+0000 7fd744ff9640 1 -- 172.21.15.77:0/3339481371 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd73804b060 con 0x7fd7480cf830 2024-03-06T07:26:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.239+0000 7fd744ff9640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd73406b4d0 0x7fd73406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.240+0000 7fd7477fe640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd73406b4d0 0x7fd73406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.240+0000 7fd744ff9640 1 -- 172.21.15.77:0/3339481371 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7380abd20 con 0x7fd7480cf830 2024-03-06T07:26:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.240+0000 7fd7477fe640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd73406b4d0 0x7fd73406d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd7480ca100 tx=0x7fd73c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.244+0000 7fd744ff9640 1 -- 172.21.15.77:0/3339481371 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd73802f300 con 0x7fd7480cf830 2024-03-06T07:26:19.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.453+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd748154d00 con 0x7fd7480cf830 2024-03-06T07:26:19.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.454+0000 7fd744ff9640 1 -- 172.21.15.77:0/3339481371 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd73807aba0 con 0x7fd7480cf830 2024-03-06T07:26:19.456 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.455+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd73406b4d0 msgr2=0x7fd73406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.455+0000 7fd74e8af640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd73406b4d0 0x7fd73406d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd7480ca100 tx=0x7fd73c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.455+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7480cf830 msgr2=0x7fd7481547c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.455+0000 7fd74e8af640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7480cf830 0x7fd7481547c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fd73802b930 tx=0x7fd7380046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.456+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 shutdown_connections 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.456+0000 7fd74e8af640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd74815dfe0 0x7fd7480b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.456+0000 7fd74e8af640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd7480d29c0 0x7fd74815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.456+0000 7fd74e8af640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd73406b4d0 0x7fd73406d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.456+0000 7fd74e8af640 1 --2- 172.21.15.77:0/3339481371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7480cf830 0x7fd7481547c0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.456+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 >> 172.21.15.77:0/3339481371 conn(0x7fd7480d3cd0 msgr2=0x7fd74806c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.456+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 shutdown_connections 2024-03-06T07:26:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:19.456+0000 7fd74e8af640 1 -- 172.21.15.77:0/3339481371 wait complete. 2024-03-06T07:26:19.468 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:26:19.468 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:26:19.469 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:26:19.469 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:26:19.469 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:19.469 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:26:19.469 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:26:19.469 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:26:19.469 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:19.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:19.469 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:26:19.485 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:26:19.485 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:19.486 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42418sec preferred_lft 42418sec 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:19.487 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:26:19.488 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:19.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:19.488 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:26:19.550 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:26:19.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:19.552 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:26:19.607 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:26:19.608 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:26:19.608 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:26:19.608 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:26:19.608 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:26:19.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:19.608 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:26:19.608 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:26:19.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:26:19.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:26:19.609 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:26:19.609 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:26:19.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:19.702 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:26:19.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:26:19.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:26:19.717 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:19.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:19.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:26:19.785 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:26:19.785 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:19.785 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:19.840 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:19.840 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:19.840 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:19.895 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:26:19.962 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:26:20.018 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:20.019 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:20.020 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:20.020 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:20.020 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:26:20.020 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:20.020 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:20.020 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:20.020 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:20.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:20.021 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:26:20.071 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:26:20.071 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:26:20.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:26:20.140 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:20.196 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:26:20.197 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:20.198 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:20.199 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:20.199 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:20.199 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:26:20.199 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:20.199 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:20.199 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:20.199 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:20.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:20.199 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:26:20.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:26:20.199+0000 7fcbcea0f480 -1 init, newargv = 0x5620d34804f0 newargc=15 2024-03-06T07:26:20.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[68461]: starting ceph client 2024-03-06T07:26:20.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[68461]: starting fuse 2024-03-06T07:26:20.216 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:26:20.216 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:26:20.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:26:20.217 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:26:20.218 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:26:20.337 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.68461.asok 2024-03-06T07:26:20.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:20.347 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68461.asok 2024-03-06T07:26:20.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:20.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68461.asok status 2024-03-06T07:26:20.483 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:26:20.483 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:26:20.483 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:26:20.483 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:26:20.483 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:26:20.483 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:26:20.483 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "68461", 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 7485, 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 7485 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:20.484 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 785002939 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 785002939 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.7485 192.168.0.1:0/785002939", 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/785002939", 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 47, 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 92, 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:26:20.485 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:26:20.486 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:26:20.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:20.493 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:26:20.493 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:20.508 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:26:20.508 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:20.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:20.508 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:20.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:20.618 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:20.667 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.668 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:20.669 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.670 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-03-06T07:26:20.671 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-03-06T07:26:20.672 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-03-06T07:26:20.673 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:20.674 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:20.676 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:20.677 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-03-06T07:26:20.678 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-03-06T07:26:20.679 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_files = 6 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:20.681 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:20.682 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.683 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:20.684 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug trim objects = true 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.685 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = bitmap 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:20.686 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2024-03-06T07:26:20.687 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:20.688 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:20.689 INFO:teuthology.orchestra.run.smithi158.stdout: mon op complaint time = 120 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: client mount timeout = 600 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = "" 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:20.690 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.b] 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.c] 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout:[mds] 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds = 20 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds balancer = 20 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug frag = True 2024-03-06T07:26:20.691 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug scatterstat = True 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi158.stdout: mds op complaint time = 180 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi158.stdout: mds verify scatter = True 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:20.692 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_files = 6 2024-03-06T07:26:20.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:20.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:20.786 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-06T07:26:20.787 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-03-06T07:26:20.787 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:20.787 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-03-06T07:26:20.787 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.787 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-03-06T07:26:20.787 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.787 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:20.787 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.788 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-03-06T07:26:20.789 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-03-06T07:26:20.790 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.792 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:20.793 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.794 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-03-06T07:26:20.795 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-03-06T07:26:20.796 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:20.797 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:20.798 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-03-06T07:26:20.799 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_files = 6 2024-03-06T07:26:20.800 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:20.801 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2024-03-06T07:26:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2024-03-06T07:26:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2024-03-06T07:26:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2024-03-06T07:26:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.805 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug trim objects = true 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:20.807 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = bitmap 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2024-03-06T07:26:20.808 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:20.809 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2024-03-06T07:26:20.810 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:20.811 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:20.812 INFO:teuthology.orchestra.run.smithi158.stdout: mon op complaint time = 120 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: client mount timeout = 600 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = "" 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:20.813 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.b] 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.c] 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout:[mds] 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds = 20 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds balancer = 20 2024-03-06T07:26:20.814 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug frag = True 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug scatterstat = True 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: mds op complaint time = 180 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: mds verify scatter = True 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_files = 6 2024-03-06T07:26:20.815 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:20.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:20.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.958+0000 7f6a1913a640 1 Processor -- start 2024-03-06T07:26:20.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.958+0000 7f6a1913a640 1 -- start start 2024-03-06T07:26:20.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a1913a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140d4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a1913a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6a140b2e40 0x7f6a140b1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a1913a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a140d10f0 con 0x7f6a141616f0 2024-03-06T07:26:20.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a1913a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a140d0ea0 con 0x7f6a140b2e40 2024-03-06T07:26:20.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a1913a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140d13d0 con 0x7f6a140d2b50 2024-03-06T07:26:20.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a12d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6a141616f0 0x7f6a140d2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58942/0 (socket says 172.21.15.77:58942) 2024-03-06T07:26:20.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a12d9d640 1 -- 172.21.15.77:0/315893241 learned_addr learned my addr 172.21.15.77:0/315893241 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:20.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a1359e640 1 --2- 172.21.15.77:0/315893241 >> v2:172.21.15.158:3300/0 conn(0x7f6a140b2e40 0x7f6a140b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.959+0000 7f6a0bfff640 1 --2- 172.21.15.77:0/315893241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140d4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.960+0000 7f6a1259c640 1 -- 172.21.15.77:0/315893241 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283681928 0 0) 0x7f6a140d10f0 con 0x7f6a141616f0 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.960+0000 7f6a1259c640 1 -- 172.21.15.77:0/315893241 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a00003520 con 0x7f6a141616f0 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.960+0000 7f6a0bfff640 1 -- 172.21.15.77:0/315893241 >> v2:172.21.15.158:3300/0 conn(0x7f6a140b2e40 msgr2=0x7f6a140b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.960+0000 7f6a0bfff640 1 --2- 172.21.15.77:0/315893241 >> v2:172.21.15.158:3300/0 conn(0x7f6a140b2e40 0x7f6a140b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.960+0000 7f6a0bfff640 1 -- 172.21.15.77:0/315893241 >> v1:172.21.15.77:6789/0 conn(0x7f6a141616f0 legacy=0x7f6a140d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.960+0000 7f6a0bfff640 1 -- 172.21.15.77:0/315893241 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140d16b0 con 0x7f6a140d2b50 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.960+0000 7f6a1359e640 1 --2- 172.21.15.77:0/315893241 >> v2:172.21.15.158:3300/0 conn(0x7f6a140b2e40 0x7f6a140b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a0bfff640 1 --2- 172.21.15.77:0/315893241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140d4980 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0097a0 tx=0x7f69fc02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17da6841287c018e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1259c640 1 -- 172.21.15.77:0/315893241 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc038070 con 0x7f6a140d2b50 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1259c640 1 -- 172.21.15.77:0/315893241 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc031700 con 0x7f6a140d2b50 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1259c640 1 -- 172.21.15.77:0/315893241 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc031ab0 con 0x7f6a140d2b50 2024-03-06T07:26:20.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1913a640 1 -- 172.21.15.77:0/315893241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 msgr2=0x7f6a140d4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1913a640 1 --2- 172.21.15.77:0/315893241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140d4980 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0097a0 tx=0x7f69fc02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1913a640 1 -- 172.21.15.77:0/315893241 shutdown_connections 2024-03-06T07:26:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1913a640 1 --2- 172.21.15.77:0/315893241 >> v2:172.21.15.158:3300/0 conn(0x7f6a140b2e40 0x7f6a140b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1913a640 1 --2- 172.21.15.77:0/315893241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140d4980 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1913a640 1 -- 172.21.15.77:0/315893241 >> 172.21.15.77:0/315893241 conn(0x7f6a140d3e60 msgr2=0x7f6a140d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1913a640 1 -- 172.21.15.77:0/315893241 shutdown_connections 2024-03-06T07:26:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.961+0000 7f6a1913a640 1 -- 172.21.15.77:0/315893241 wait complete. 2024-03-06T07:26:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1913a640 1 Processor -- start 2024-03-06T07:26:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1913a640 1 -- start start 2024-03-06T07:26:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1913a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a140b2e40 0x7f6a1409e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1913a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1913a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a141616f0 0x7f6a140cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1913a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140b18b0 con 0x7f6a141616f0 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1913a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a14161ad0 con 0x7f6a140b2e40 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1913a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140b1a30 con 0x7f6a140d2b50 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a141616f0 0x7f6a140cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a141616f0 0x7f6a140cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37814/0 (socket says 172.21.15.77:37814) 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a1359e640 1 -- 172.21.15.77:0/4033451716 learned_addr learned my addr 172.21.15.77:0/4033451716 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a12d9d640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a140b2e40 0x7f6a1409e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.963+0000 7f6a0bfff640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a1359e640 1 -- 172.21.15.77:0/4033451716 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 msgr2=0x7f6a140cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a1359e640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a1359e640 1 -- 172.21.15.77:0/4033451716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a140b2e40 msgr2=0x7f6a1409e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a1359e640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a140b2e40 0x7f6a1409e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a1359e640 1 -- 172.21.15.77:0/4033451716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140a09e0 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a1359e640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a141616f0 0x7f6a140cd640 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009f00 tx=0x7f6a0c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a0b7fe640 1 -- 172.21.15.77:0/4033451716 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c018030 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a0b7fe640 1 -- 172.21.15.77:0/4033451716 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c00ce50 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a140a0cc0 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a0b7fe640 1 -- 172.21.15.77:0/4033451716 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c003bb0 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.964+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a140a1280 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.965+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a14068fb0 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.966+0000 7f6a0b7fe640 1 -- 172.21.15.77:0/4033451716 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a0c01b070 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.966+0000 7f6a0b7fe640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69f406b4d0 0x7f69f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.966+0000 7f6a12d9d640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69f406b4d0 0x7f69f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.966+0000 7f6a0b7fe640 1 -- 172.21.15.77:0/4033451716 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a0c08ab50 con 0x7f6a141616f0 2024-03-06T07:26:20.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.967+0000 7f6a12d9d640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69f406b4d0 0x7f69f406d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6a04005880 tx=0x7f6a04009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:20.970+0000 7f6a0b7fe640 1 -- 172.21.15.77:0/4033451716 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a0c010040 con 0x7f6a141616f0 2024-03-06T07:26:21.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.147+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6a14160830 con 0x7f6a141616f0 2024-03-06T07:26:21.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.148+0000 7f6a0b7fe640 1 -- 172.21.15.77:0/4033451716 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6a0c059380 con 0x7f6a141616f0 2024-03-06T07:26:21.150 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:26:21.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69f406b4d0 msgr2=0x7f69f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69f406b4d0 0x7f69f406d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6a04005880 tx=0x7f6a04009040 comp rx=0 tx=0).stop 2024-03-06T07:26:21.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a141616f0 msgr2=0x7f6a140cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a141616f0 0x7f6a140cd640 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009f00 tx=0x7f6a0c00c490 comp rx=0 tx=0).stop 2024-03-06T07:26:21.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 shutdown_connections 2024-03-06T07:26:21.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69f406b4d0 0x7f69f406d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a141616f0 0x7f6a140cd640 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a140d2b50 0x7f6a140cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 --2- 172.21.15.77:0/4033451716 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a140b2e40 0x7f6a1409e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.149+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 >> 172.21.15.77:0/4033451716 conn(0x7f6a140d3e60 msgr2=0x7f6a1415fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.150+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 shutdown_connections 2024-03-06T07:26:21.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.150+0000 7f6a1913a640 1 -- 172.21.15.77:0/4033451716 wait complete. 2024-03-06T07:26:21.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:21.165 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:21.214 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-03-06T07:26:21.215 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:21.216 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-03-06T07:26:21.217 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.218 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.219 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-03-06T07:26:21.220 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:21.221 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:21.222 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:21.223 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-03-06T07:26:21.224 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:21.225 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_files = 6 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:21.226 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2024-03-06T07:26:21.227 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:21.228 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2024-03-06T07:26:21.229 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.230 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug trim objects = true 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2024-03-06T07:26:21.231 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = bitmap 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:21.232 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:21.233 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:21.234 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.235 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: mon op complaint time = 120 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: client mount timeout = 600 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = "" 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:21.236 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.b] 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.c] 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout:[mds] 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds = 20 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds balancer = 20 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug frag = True 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug scatterstat = True 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout: mds op complaint time = 180 2024-03-06T07:26:21.237 INFO:teuthology.orchestra.run.smithi158.stdout: mds verify scatter = True 2024-03-06T07:26:21.238 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:21.238 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:21.238 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:21.238 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_files = 6 2024-03-06T07:26:21.238 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:21.238 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:21.239 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:21.240 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:21.240 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:21.240 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:21.240 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:21.240 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:21.241 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:21.241 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:21.290 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:21.290 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:26:21.291 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:21.291 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:26:21.292 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:21.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:26:21.293 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:21.293 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:26:21.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.436+0000 7f82f3278640 1 Processor -- start 2024-03-06T07:26:21.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.436+0000 7ff8c36c0640 1 Processor -- start 2024-03-06T07:26:21.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.436+0000 7ff259525640 1 Processor -- start 2024-03-06T07:26:21.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.436+0000 7f82f3278640 1 -- start start 2024-03-06T07:26:21.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.436+0000 7ff259525640 1 -- start start 2024-03-06T07:26:21.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.436+0000 7ff8c36c0640 1 -- start start 2024-03-06T07:26:21.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7fe4916c3640 1 Processor -- start 2024-03-06T07:26:21.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7fe4916c3640 1 -- start start 2024-03-06T07:26:21.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff259525640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff25415e110 0x7ff25406da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff259525640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff25406f4c0 0x7ff25406af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7f82f3278640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82ec0b65e0 0x7f82ec1624a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff8c36c0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc0d0950 0x7ff8bc0d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff8c36c0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff8bc0b9060 0x7ff8bc0b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7f82f3278640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec0cf2f0 con 0x7f82ec15e310 2024-03-06T07:26:21.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7f82f3278640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec0cf5d0 con 0x7f82ec0d12d0 2024-03-06T07:26:21.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7f82f3278640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ec0b5140 con 0x7f82ec0b65e0 2024-03-06T07:26:21.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7f82f1014640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f82ec15e310 0x7f82ec161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58956/0 (socket says 172.21.15.77:58956) 2024-03-06T07:26:21.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7f82f1014640 1 -- 172.21.15.77:0/678998694 learned_addr learned my addr 172.21.15.77:0/678998694 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:21.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff259525640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25406f950 0x7ff25406b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff259525640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25415e4f0 con 0x7ff25406f4c0 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff259525640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2540b0520 con 0x7ff25415e110 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff259525640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540d0b40 con 0x7ff25406f950 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff2527fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff25406f4c0 0x7ff25406af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff8c36c0640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8bc0d1ea0 con 0x7ff8bc15e150 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff8c36c0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8bc0d12f0 con 0x7ff8bc0b9060 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff8c36c0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0d2ab0 con 0x7ff8bc0d0950 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff8c145c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff8bc15e150 0x7ff8bc0cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58962/0 (socket says 172.21.15.77:58962) 2024-03-06T07:26:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.437+0000 7ff8c145c640 1 -- 172.21.15.77:0/244187892 learned_addr learned my addr 172.21.15.77:0/244187892 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:21.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff8c0c5b640 1 --2- 172.21.15.77:0/244187892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc0d0950 0x7ff8bc0d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff2527fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff25406f4c0 0x7ff25406af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37820/0 (socket says 172.21.15.77:37820) 2024-03-06T07:26:21.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff2527fc640 1 -- 172.21.15.77:0/3384858129 learned_addr learned my addr 172.21.15.77:0/3384858129 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:21.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82f0813640 1 --2- 172.21.15.77:0/678998694 >> v2:172.21.15.158:3300/0 conn(0x7f82ec0b65e0 0x7f82ec1624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff252ffd640 1 --2- 172.21.15.77:0/3384858129 >> v2:172.21.15.158:3300/0 conn(0x7ff25415e110 0x7ff25406da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff2537fe640 1 --2- 172.21.15.77:0/3384858129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25406f950 0x7ff25406b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff8c1c5d640 1 --2- 172.21.15.77:0/244187892 >> v2:172.21.15.158:3300/0 conn(0x7ff8bc0b9060 0x7ff8bc0b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe4916c3640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe48c15bf90 0x7fe48c0aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305651092 0 0) 0x7f82ec0cf2f0 con 0x7f82ec15e310 2024-03-06T07:26:21.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d8003520 con 0x7f82ec15e310 2024-03-06T07:26:21.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe4916c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c0d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff8b3fff640 1 -- 172.21.15.77:0/244187892 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5933360 0 0) 0x7ff8bc0d1ea0 con 0x7ff8bc15e150 2024-03-06T07:26:21.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff8b3fff640 1 -- 172.21.15.77:0/244187892 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a8003520 con 0x7ff8bc15e150 2024-03-06T07:26:21.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe48affd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe48c15bf90 0x7fe48c0aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2329580017 0 0) 0x7f82d8003520 con 0x7f82ec15e310 2024-03-06T07:26:21.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 >> v2:172.21.15.158:3300/0 conn(0x7f82ec0b65e0 msgr2=0x7f82ec1624a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 --2- 172.21.15.77:0/678998694 >> v2:172.21.15.158:3300/0 conn(0x7f82ec0b65e0 0x7f82ec1624a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 >> v1:172.21.15.158:6789/0 conn(0x7f82ec0d12d0 legacy=0x7f82ec0d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0b4dd0 con 0x7f82ec15e310 2024-03-06T07:26:21.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff2527fc640 1 -- 172.21.15.77:0/3384858129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25406f950 msgr2=0x7ff25406b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff2527fc640 1 --2- 172.21.15.77:0/3384858129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25406f950 0x7ff25406b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff2527fc640 1 -- 172.21.15.77:0/3384858129 >> v2:172.21.15.158:3300/0 conn(0x7ff25415e110 msgr2=0x7ff25406da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff2527fc640 1 --2- 172.21.15.77:0/3384858129 >> v2:172.21.15.158:3300/0 conn(0x7ff25415e110 0x7ff25406da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff2527fc640 1 -- 172.21.15.77:0/3384858129 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2540afc80 con 0x7ff25406f4c0 2024-03-06T07:26:21.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe4916c3640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c0af2d0 con 0x7fe48c0d0630 2024-03-06T07:26:21.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe4916c3640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c0af5b0 con 0x7fe48c15bf90 2024-03-06T07:26:21.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe4916c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0af890 con 0x7fe48c0b32b0 2024-03-06T07:26:21.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f82e8001650 con 0x7f82ec15e310 2024-03-06T07:26:21.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe48affd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe48c15bf90 0x7fe48c0aea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:48942/0 (socket says 172.21.15.77:48942) 2024-03-06T07:26:21.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe48affd640 1 -- 172.21.15.77:0/107801157 learned_addr learned my addr 172.21.15.77:0/107801157 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:21.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff8c1c5d640 1 -- 172.21.15.77:0/244187892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc0d0950 msgr2=0x7ff8bc0d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff8c1c5d640 1 --2- 172.21.15.77:0/244187892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc0d0950 0x7ff8bc0d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7fe48a7fc640 1 --2- 172.21.15.77:0/107801157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c0d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff8c1c5d640 1 -- 172.21.15.77:0/244187892 >> v1:172.21.15.77:6789/0 conn(0x7ff8bc15e150 legacy=0x7ff8bc0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7ff8c1c5d640 1 -- 172.21.15.77:0/244187892 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc0d2180 con 0x7ff8bc0b9060 2024-03-06T07:26:21.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 >> v1:172.21.15.77:6789/0 conn(0x7f82ec15e310 legacy=0x7f82ec161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82d80039b0 0x7f82d8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d8004800 0x7f82d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82d8005210 0x7f82d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0b5140 con 0x7f82d80039b0 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d800c140 con 0x7f82d8005210 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d800c340 con 0x7f82d8004800 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.438+0000 7f82f1014640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82d80039b0 0x7f82d8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7f82f1815640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82d8005210 0x7f82d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7f82f1014640 1 -- 172.21.15.77:0/678998694 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d8004800 msgr2=0x7f82d8004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7f82f1014640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d8004800 0x7f82d8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7f82f1014640 1 -- 172.21.15.77:0/678998694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82d8005210 msgr2=0x7f82d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7fe48affd640 1 -- 172.21.15.77:0/107801157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 msgr2=0x7fe48c0d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7fe48affd640 1 --2- 172.21.15.77:0/107801157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c0d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7fe48affd640 1 -- 172.21.15.77:0/107801157 >> v1:172.21.15.158:6789/0 conn(0x7fe48c0d0630 legacy=0x7fe48c0b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7fe48affd640 1 -- 172.21.15.77:0/107801157 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0afb70 con 0x7fe48c15bf90 2024-03-06T07:26:21.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7f82f1014640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82d8005210 0x7f82d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7f82f1014640 1 -- 172.21.15.77:0/678998694 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8003520 con 0x7f82d80039b0 2024-03-06T07:26:21.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7f82f1815640 1 -- 172.21.15.77:0/678998694 reap_dead start 2024-03-06T07:26:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7ff2527fc640 1 --2- 172.21.15.77:0/3384858129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff25406f4c0 0x7ff25406af80 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff25409efb0 tx=0x7ff24802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815b18e68215765c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7ff251ffb640 1 -- 172.21.15.77:0/3384858129 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff248038070 con 0x7ff25406f4c0 2024-03-06T07:26:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7ff251ffb640 1 -- 172.21.15.77:0/3384858129 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff248031740 con 0x7ff25406f4c0 2024-03-06T07:26:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7ff251ffb640 1 -- 172.21.15.77:0/3384858129 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24802f050 con 0x7ff25406f4c0 2024-03-06T07:26:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.439+0000 7ff8c1c5d640 1 --2- 172.21.15.77:0/244187892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 0x7ff8bc0b94f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aa9d6bb9dfcb089 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7fe48affd640 1 --2- 172.21.15.77:0/107801157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48c15bf90 0x7fe48c0aea30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009870 tx=0x7fe47c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec2063ba546907c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8b3fff640 1 -- 172.21.15.77:0/244187892 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4039070 con 0x7ff8bc0b9060 2024-03-06T07:26:21.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8b3fff640 1 -- 172.21.15.77:0/244187892 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b402cbc0 con 0x7ff8bc0b9060 2024-03-06T07:26:21.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8b3fff640 1 -- 172.21.15.77:0/244187892 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4034440 con 0x7ff8bc0b9060 2024-03-06T07:26:21.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7f82f1014640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82d80039b0 0x7f82d8003df0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f82e800c1c0 tx=0x7f82d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5f5a54deccfdb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7fe489ffb640 1 -- 172.21.15.77:0/107801157 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c034070 con 0x7fe48c15bf90 2024-03-06T07:26:21.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7fe489ffb640 1 -- 172.21.15.77:0/107801157 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c030840 con 0x7fe48c15bf90 2024-03-06T07:26:21.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7fe489ffb640 1 -- 172.21.15.77:0/107801157 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c030c10 con 0x7fe48c15bf90 2024-03-06T07:26:21.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8007cc0 con 0x7f82d80039b0 2024-03-06T07:26:21.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8030030 con 0x7f82d80039b0 2024-03-06T07:26:21.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7f82e3fff640 1 -- 172.21.15.77:0/678998694 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8004ce0 con 0x7f82d80039b0 2024-03-06T07:26:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 -- 172.21.15.77:0/3384858129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff25406f4c0 msgr2=0x7ff25406af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 --2- 172.21.15.77:0/3384858129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff25406f4c0 0x7ff25406af80 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff25409efb0 tx=0x7ff24802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 -- 172.21.15.77:0/3384858129 shutdown_connections 2024-03-06T07:26:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 --2- 172.21.15.77:0/3384858129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25406f950 0x7ff25406b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 --2- 172.21.15.77:0/3384858129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff25406f4c0 0x7ff25406af80 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 --2- 172.21.15.77:0/3384858129 >> v2:172.21.15.158:3300/0 conn(0x7ff25415e110 0x7ff25406da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 -- 172.21.15.77:0/3384858129 >> 172.21.15.77:0/3384858129 conn(0x7ff25406c090 msgr2=0x7ff25406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8c36c0640 1 -- 172.21.15.77:0/244187892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 msgr2=0x7ff8bc0b94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 -- 172.21.15.77:0/3384858129 shutdown_connections 2024-03-06T07:26:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff259525640 1 -- 172.21.15.77:0/3384858129 wait complete. 2024-03-06T07:26:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/244187892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 0x7ff8bc0b94f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009d90 tx=0x7ff8b402be10 comp rx=0 tx=0).stop 2024-03-06T07:26:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7fe4916c3640 1 -- 172.21.15.77:0/107801157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48c15bf90 msgr2=0x7fe48c0aea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7fe4916c3640 1 --2- 172.21.15.77:0/107801157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48c15bf90 0x7fe48c0aea30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009870 tx=0x7fe47c02b980 comp rx=0 tx=0).stop 2024-03-06T07:26:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8c36c0640 1 -- 172.21.15.77:0/244187892 shutdown_connections 2024-03-06T07:26:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/244187892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 0x7ff8bc0b94f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/244187892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc0d0950 0x7ff8bc0d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.440+0000 7ff8c36c0640 1 -- 172.21.15.77:0/244187892 >> 172.21.15.77:0/244187892 conn(0x7ff8bc09d230 msgr2=0x7ff8bc0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7ff8c36c0640 1 -- 172.21.15.77:0/244187892 shutdown_connections 2024-03-06T07:26:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7ff8c36c0640 1 -- 172.21.15.77:0/244187892 wait complete. 2024-03-06T07:26:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7f82f3278640 1 -- 172.21.15.77:0/678998694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82d80039b0 msgr2=0x7f82d8003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7f82f3278640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82d80039b0 0x7f82d8003df0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f82e800c1c0 tx=0x7f82d8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:21.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7fe4916c3640 1 -- 172.21.15.77:0/107801157 shutdown_connections 2024-03-06T07:26:21.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7fe4916c3640 1 --2- 172.21.15.77:0/107801157 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7fe4916c3640 1 --2- 172.21.15.77:0/107801157 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48c15bf90 0x7fe48c0aea30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7fe4916c3640 1 -- 172.21.15.77:0/107801157 >> 172.21.15.77:0/107801157 conn(0x7fe48c09aa40 msgr2=0x7fe48c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7fe4916c3640 1 -- 172.21.15.77:0/107801157 shutdown_connections 2024-03-06T07:26:21.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7fe4916c3640 1 -- 172.21.15.77:0/107801157 wait complete. 2024-03-06T07:26:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7f82f3278640 1 -- 172.21.15.77:0/678998694 shutdown_connections 2024-03-06T07:26:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7f82f3278640 1 --2- 172.21.15.77:0/678998694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82d80039b0 0x7f82d8003df0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7f82f3278640 1 -- 172.21.15.77:0/678998694 >> 172.21.15.77:0/678998694 conn(0x7f82ec09aa40 msgr2=0x7f82ec09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7f82f3278640 1 -- 172.21.15.77:0/678998694 shutdown_connections 2024-03-06T07:26:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.441+0000 7f82f3278640 1 -- 172.21.15.77:0/678998694 wait complete. 2024-03-06T07:26:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.442+0000 7ff259525640 1 Processor -- start 2024-03-06T07:26:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.442+0000 7ff259525640 1 -- start start 2024-03-06T07:26:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.442+0000 7ff8c36c0640 1 Processor -- start 2024-03-06T07:26:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.442+0000 7fe4916c3640 1 Processor -- start 2024-03-06T07:26:21.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.442+0000 7ff8c36c0640 1 -- start start 2024-03-06T07:26:21.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.442+0000 7fe4916c3640 1 -- start start 2024-03-06T07:26:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.442+0000 7ff259525640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff25406f950 0x7ff254159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff259525640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25415e110 0x7ff25414db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff259525640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff254158180 0x7ff254152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff259525640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25406dfc0 con 0x7ff254158180 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff259525640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540b0240 con 0x7ff25406f950 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff259525640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540b3df0 con 0x7ff25415e110 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff254158180 0x7ff254152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff254158180 0x7ff254152840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37836/0 (socket says 172.21.15.77:37836) 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 -- 172.21.15.77:0/3990636843 learned_addr learned my addr 172.21.15.77:0/3990636843 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff252ffd640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff25406f950 0x7ff254159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2527fc640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25415e110 0x7ff25414db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 -- 172.21.15.77:0/3990636843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25415e110 msgr2=0x7ff25414db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25415e110 0x7ff25414db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 -- 172.21.15.77:0/3990636843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff25406f950 msgr2=0x7ff254159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff25406f950 0x7ff254159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 -- 172.21.15.77:0/3990636843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25414e090 con 0x7ff254158180 2024-03-06T07:26:21.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7fe4916c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c06dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7fe4916c3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48c0d0630 0x7fe48c06c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff2537fe640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff254158180 0x7ff254152840 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00ef00 tx=0x7ff24c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7f82f3278640 1 Processor -- start 2024-03-06T07:26:21.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7fe4916c3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe48c15bf90 0x7fe48c1f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7fe4916c3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0aef70 con 0x7fe48c15bf90 2024-03-06T07:26:21.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7fe4916c3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c15c370 con 0x7fe48c0d0630 2024-03-06T07:26:21.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7f82f3278640 1 -- start start 2024-03-06T07:26:21.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7fe4916c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0b1380 con 0x7fe48c0b32b0 2024-03-06T07:26:21.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7ff8c36c0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 0x7ff8bc151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c36c0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8bc0d0950 0x7ff8bc1f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.443+0000 7fe48b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe48c15bf90 0x7fe48c1f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff23b7fe640 1 -- 172.21.15.77:0/3990636843 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c018020 con 0x7ff254158180 2024-03-06T07:26:21.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe48c15bf90 0x7fe48c1f6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37852/0 (socket says 172.21.15.77:37852) 2024-03-06T07:26:21.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48b7fe640 1 -- 172.21.15.77:0/3080244104 learned_addr learned my addr 172.21.15.77:0/3080244104 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:21.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c36c0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc15e150 0x7ff8bc0ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c36c0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0d2f40 con 0x7ff8bc0d0950 2024-03-06T07:26:21.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c0c5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8bc0d0950 0x7ff8bc1f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48affd640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c06dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c0c5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8bc0d0950 0x7ff8bc1f40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37864/0 (socket says 172.21.15.77:37864) 2024-03-06T07:26:21.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c0c5b640 1 -- 172.21.15.77:0/547034431 learned_addr learned my addr 172.21.15.77:0/547034431 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:21.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48b7fe640 1 -- 172.21.15.77:0/3080244104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 msgr2=0x7fe48c06dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48b7fe640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c06dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff23b7fe640 1 -- 172.21.15.77:0/3990636843 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c00bd20 con 0x7ff254158180 2024-03-06T07:26:21.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48b7fe640 1 -- 172.21.15.77:0/3080244104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48c0d0630 msgr2=0x7fe48c06c090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:26:21.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48b7fe640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48c0d0630 0x7fe48c06c090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48b7fe640 1 -- 172.21.15.77:0/3080244104 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c06c5d0 con 0x7fe48c15bf90 2024-03-06T07:26:21.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff23b7fe640 1 -- 172.21.15.77:0/3990636843 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c014640 con 0x7ff254158180 2024-03-06T07:26:21.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c1c5d640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc15e150 0x7ff8bc0ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff254158680 con 0x7ff254158180 2024-03-06T07:26:21.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c36c0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0d2740 con 0x7ff8bc0b9060 2024-03-06T07:26:21.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc0d1b30 con 0x7ff8bc15e150 2024-03-06T07:26:21.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48affd640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c06dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:21.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7fe48b7fe640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe48c15bf90 0x7fe48c1f6a90 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe48400c710 tx=0x7fe48400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c145c640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 0x7ff8bc151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c0c5b640 1 -- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc15e150 msgr2=0x7ff8bc0ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c0c5b640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc15e150 0x7ff8bc0ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c0c5b640 1 -- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 msgr2=0x7ff8bc151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c0c5b640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 0x7ff8bc151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff8c0c5b640 1 -- 172.21.15.77:0/547034431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc0ca720 con 0x7ff8bc0d0950 2024-03-06T07:26:21.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7ff8c1c5d640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc15e150 0x7ff8bc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.444+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff254158c40 con 0x7ff254158180 2024-03-06T07:26:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7fe4737fe640 1 -- 172.21.15.77:0/3080244104 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484016020 con 0x7fe48c15bf90 2024-03-06T07:26:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f3278640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d80039b0 0x7f82ec1438f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c06c850 con 0x7fe48c15bf90 2024-03-06T07:26:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f3278640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ec0b0aa0 0x7f82ec151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c06ce10 con 0x7fe48c15bf90 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f3278640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ec0af440 0x7f82ec153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f3278640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec161d90 con 0x7f82ec0af440 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f3278640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0cf8b0 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f3278640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec161f10 con 0x7f82d80039b0 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f1014640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d80039b0 0x7f82ec1438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f1014640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d80039b0 0x7f82ec1438f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:38166/0 (socket says 172.21.15.77:38166) 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f0813640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ec0b0aa0 0x7f82ec151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f1014640 1 -- 172.21.15.77:0/2945063950 learned_addr learned my addr 172.21.15.77:0/2945063950 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f0813640 1 -- 172.21.15.77:0/2945063950 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d80039b0 msgr2=0x7f82ec1438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7ff8c0c5b640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8bc0d0950 0x7ff8bc1f40c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00e9b0 tx=0x7ff8ac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f0813640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d80039b0 0x7f82ec1438f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f0813640 1 -- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ec0af440 msgr2=0x7f82ec153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f0813640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ec0af440 0x7f82ec153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.445+0000 7f82f0813640 1 -- 172.21.15.77:0/2945063950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0afa00 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7fe4737fe640 1 -- 172.21.15.77:0/3080244104 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48400ce20 con 0x7fe48c15bf90 2024-03-06T07:26:21.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7fe4737fe640 1 -- 172.21.15.77:0/3080244104 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484003970 con 0x7fe48c15bf90 2024-03-06T07:26:21.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7f82f0813640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ec0b0aa0 0x7f82ec151be0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00e7f0 tx=0x7f82dc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff254068050 con 0x7ff254158180 2024-03-06T07:26:21.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7f82e1ffb640 1 -- 172.21.15.77:0/2945063950 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc017070 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7f82e1ffb640 1 -- 172.21.15.77:0/2945063950 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc00c860 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7f82e1ffb640 1 -- 172.21.15.77:0/2945063950 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc00cc30 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7ff8b1ffb640 1 -- 172.21.15.77:0/547034431 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac017070 con 0x7ff8bc0d0950 2024-03-06T07:26:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7ff23b7fe640 1 -- 172.21.15.77:0/3990636843 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff24c005d10 con 0x7ff254158180 2024-03-06T07:26:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7ff8b1ffb640 1 -- 172.21.15.77:0/547034431 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac00c9a0 con 0x7ff8bc0d0950 2024-03-06T07:26:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8bc1f4860 con 0x7ff8bc0d0950 2024-03-06T07:26:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7ff8b1ffb640 1 -- 172.21.15.77:0/547034431 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00cd90 con 0x7ff8bc0d0950 2024-03-06T07:26:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec0afce0 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec0b02a0 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.447+0000 7ff23b7fe640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24006b4d0 0x7ff24006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.447+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46c005290 con 0x7fe48c15bf90 2024-03-06T07:26:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.447+0000 7ff23b7fe640 1 -- 172.21.15.77:0/3990636843 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff24c08a7b0 con 0x7ff254158180 2024-03-06T07:26:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.447+0000 7ff252ffd640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24006b4d0 0x7ff24006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.446+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc1f4e20 con 0x7ff8bc0d0950 2024-03-06T07:26:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.448+0000 7fe4737fe640 1 -- 172.21.15.77:0/3080244104 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe484003b10 con 0x7fe48c15bf90 2024-03-06T07:26:21.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.448+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8bc1f5060 con 0x7ff8bc0d0950 2024-03-06T07:26:21.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.448+0000 7ff252ffd640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24006b4d0 0x7ff24006d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff244009b40 tx=0x7ff244009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.448+0000 7fe4737fe640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe47806b4d0 0x7fe47806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.448+0000 7fe4737fe640 1 -- 172.21.15.77:0/3080244104 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe484089ec0 con 0x7fe48c15bf90 2024-03-06T07:26:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.448+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec06b440 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.448+0000 7f82e1ffb640 1 -- 172.21.15.77:0/2945063950 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82dc01a070 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.448+0000 7fe48affd640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe47806b4d0 0x7fe47806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.449+0000 7f82e1ffb640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82d006b500 0x7f82d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.449+0000 7fe48affd640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe47806b4d0 0x7fe47806d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe47c036000 tx=0x7fe47c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.449+0000 7f82e1ffb640 1 -- 172.21.15.77:0/2945063950 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82dc089e20 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.449+0000 7f82f1014640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82d006b500 0x7f82d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.449+0000 7f82f1014640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82d006b500 0x7f82d006d9c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f82e8000c00 tx=0x7f82e8009070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.452+0000 7ff23b7fe640 1 -- 172.21.15.77:0/3990636843 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff24c022070 con 0x7ff254158180 2024-03-06T07:26:21.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.453+0000 7ff8b1ffb640 1 -- 172.21.15.77:0/547034431 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8ac01a070 con 0x7ff8bc0d0950 2024-03-06T07:26:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.453+0000 7f82e1ffb640 1 -- 172.21.15.77:0/2945063950 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82ec06b440 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.454+0000 7ff8b1ffb640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8a006b520 0x7ff8a006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.456+0000 7ff8c145c640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8a006b520 0x7ff8a006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.456+0000 7ff8c145c640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8a006b520 0x7ff8a006d9e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8000c00 tx=0x7ff8b8007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.457+0000 7ff8b1ffb640 1 -- 172.21.15.77:0/547034431 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8ac057230 con 0x7ff8bc0d0950 2024-03-06T07:26:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.457+0000 7fe4737fe640 1 -- 172.21.15.77:0/3080244104 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe484020020 con 0x7fe48c15bf90 2024-03-06T07:26:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.462+0000 7ff8b1ffb640 1 -- 172.21.15.77:0/547034431 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8bc1f5060 con 0x7ff8bc0d0950 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.713+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f82ec0b4a80 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.716 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:26:21.714+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:21.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.719+0000 7f82e1ffb640 1 -- 172.21.15.77:0/2945063950 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7287 v48) v1 ==== 95+0+0 (secure 0 0 0) 0x7f82ec0b4a80 con 0x7f82ec0b0aa0 2024-03-06T07:26:21.720 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 7287 2024-03-06T07:26:21.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.719+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7ff8bc069ee0 con 0x7ff8bc0d0950 2024-03-06T07:26:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82d006b500 msgr2=0x7f82d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82d006b500 0x7f82d006d9c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f82e8000c00 tx=0x7f82e8009070 comp rx=0 tx=0).stop 2024-03-06T07:26:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ec0b0aa0 msgr2=0x7f82ec151be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ec0b0aa0 0x7f82ec151be0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00e7f0 tx=0x7f82dc00ecb0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 shutdown_connections 2024-03-06T07:26:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82d006b500 0x7f82d006d9c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ec0af440 0x7f82ec153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ec0b0aa0 0x7f82ec151be0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 --2- 172.21.15.77:0/2945063950 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82d80039b0 0x7f82ec1438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 >> 172.21.15.77:0/2945063950 conn(0x7f82ec09aa40 msgr2=0x7f82ec06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 shutdown_connections 2024-03-06T07:26:21.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.721+0000 7f82f3278640 1 -- 172.21.15.77:0/2945063950 wait complete. 2024-03-06T07:26:21.735 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:21.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:21.738 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:21.738 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:21.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.742+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7ff2540724e0 con 0x7ff254158180 2024-03-06T07:26:21.744 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:26:21.743+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.744+0000 7ff8b1ffb640 1 -- 172.21.15.77:0/547034431 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6101 v50) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff8ac059190 con 0x7ff8bc0d0950 2024-03-06T07:26:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.744+0000 7ff23b7fe640 1 -- 172.21.15.77:0/3990636843 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7284 v50) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff24c058f40 con 0x7ff254158180 2024-03-06T07:26:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 6101 2024-03-06T07:26:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 7284 2024-03-06T07:26:21.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.746+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8a006b520 msgr2=0x7ff8a006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.746+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8a006b520 0x7ff8a006d9e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8000c00 tx=0x7ff8b8007d30 comp rx=0 tx=0).stop 2024-03-06T07:26:21.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.746+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8bc0d0950 msgr2=0x7ff8bc1f40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.746+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8bc0d0950 0x7ff8bc1f40c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00e9b0 tx=0x7ff8ac00ee70 comp rx=0 tx=0).stop 2024-03-06T07:26:21.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24006b4d0 msgr2=0x7ff24006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24006b4d0 0x7ff24006d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff244009b40 tx=0x7ff244009190 comp rx=0 tx=0).stop 2024-03-06T07:26:21.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff254158180 msgr2=0x7ff254152840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff254158180 0x7ff254152840 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00ef00 tx=0x7ff24c00b480 comp rx=0 tx=0).stop 2024-03-06T07:26:21.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 shutdown_connections 2024-03-06T07:26:21.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8a006b520 0x7ff8a006d9e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8bc0d0950 0x7ff8bc1f40c0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8bc15e150 0x7ff8bc0ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff8c36c0640 1 --2- 172.21.15.77:0/547034431 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8bc0b9060 0x7ff8bc151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 >> 172.21.15.77:0/547034431 conn(0x7ff8bc09d230 msgr2=0x7ff8bc06c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 shutdown_connections 2024-03-06T07:26:21.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24006b4d0 0x7ff24006d990 secure :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff244009b40 tx=0x7ff244009190 comp rx=0 tx=0).stop 2024-03-06T07:26:21.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff254158180 0x7ff254152840 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff25415e110 0x7ff25414db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 --2- 172.21.15.77:0/3990636843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff25406f950 0x7ff254159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 >> 172.21.15.77:0/3990636843 conn(0x7ff25406c090 msgr2=0x7ff25406e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 shutdown_connections 2024-03-06T07:26:21.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 shutdown_connections 2024-03-06T07:26:21.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff8c36c0640 1 -- 172.21.15.77:0/547034431 wait complete. 2024-03-06T07:26:21.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.747+0000 7ff259525640 1 -- 172.21.15.77:0/3990636843 wait complete. 2024-03-06T07:26:21.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.748+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fe46c005580 con 0x7fe48c15bf90 2024-03-06T07:26:21.754 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:26:21.752+0000 7f63c7eac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:21.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.755+0000 7fe4737fe640 1 -- 172.21.15.77:0/3080244104 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6850 v52) v1 ==== 95+0+0 (secure 0 0 0) 0x7fe48c06c5d0 con 0x7fe48c15bf90 2024-03-06T07:26:21.757 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 6850 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe47806b4d0 msgr2=0x7fe47806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe47806b4d0 0x7fe47806d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe47c036000 tx=0x7fe47c03b040 comp rx=0 tx=0).stop 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe48c15bf90 msgr2=0x7fe48c1f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe48c15bf90 0x7fe48c1f6a90 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe48400c710 tx=0x7fe48400cbd0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 shutdown_connections 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe47806b4d0 0x7fe47806d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe48c15bf90 0x7fe48c1f6a90 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe48c0d0630 0x7fe48c06c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 --2- 172.21.15.77:0/3080244104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe48c0b32b0 0x7fe48c06dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 >> 172.21.15.77:0/3080244104 conn(0x7fe48c09aa40 msgr2=0x7fe48c0b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 shutdown_connections 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.758+0000 7fe4916c3640 1 -- 172.21.15.77:0/3080244104 wait complete. 2024-03-06T07:26:21.765 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:21.765 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:21.766 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:21.766 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:21.768 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:21.768 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:21.773 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:21.773 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:21.775 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:21.775 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:21.814 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:21.814 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:21.814 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:21.817 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:26:21.844 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:26:21.855 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:26:21.891 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:26:21.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.916+0000 7f69f615c640 1 Processor -- start 2024-03-06T07:26:21.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.916+0000 7f69f615c640 1 -- start start 2024-03-06T07:26:21.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69f615c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f00b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69f615c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69f00b16b0 0x7f69f00ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69f615c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f00cd4d0 con 0x7f69f00d2870 2024-03-06T07:26:21.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69f615c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f00cf920 con 0x7f69f00b16b0 2024-03-06T07:26:21.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69f615c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00cd7b0 con 0x7f69f015f5e0 2024-03-06T07:26:21.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69ef7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f00b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69eeffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69f00b16b0 0x7f69f00ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69ef7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f00b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:38186/0 (socket says 172.21.15.77:38186) 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69ef7fe640 1 -- 172.21.15.77:0/2310318614 learned_addr learned my addr 172.21.15.77:0/2310318614 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69effff640 1 --1- 172.21.15.77:0/2310318614 >> v1:172.21.15.158:6789/0 conn(0x7f69f00d2870 0x7f69f015fcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49438/0 (socket says 172.21.15.77:49438) 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69eeffd640 1 -- 172.21.15.77:0/2310318614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 msgr2=0x7f69f00b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69eeffd640 1 --2- 172.21.15.77:0/2310318614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f00b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69eeffd640 1 -- 172.21.15.77:0/2310318614 >> v1:172.21.15.158:6789/0 conn(0x7f69f00d2870 legacy=0x7f69f015fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69eeffd640 1 -- 172.21.15.77:0/2310318614 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f00cda90 con 0x7f69f00b16b0 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69ee7fc640 1 -- 172.21.15.77:0/2310318614 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391207718 0 0) 0x7f69f00cd4d0 con 0x7f69f00d2870 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.917+0000 7f69ef7fe640 1 --2- 172.21.15.77:0/2310318614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f00b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.918+0000 7f69eeffd640 1 --2- 172.21.15.77:0/2310318614 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 0x7f69f00ce550 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f69d80097a0 tx=0x7f69d802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6df3d1098c740246 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.918+0000 7f69ee7fc640 1 -- 172.21.15.77:0/2310318614 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d80315a0 con 0x7f69f00b16b0 2024-03-06T07:26:21.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.918+0000 7f69ee7fc640 1 -- 172.21.15.77:0/2310318614 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d8037070 con 0x7f69f00b16b0 2024-03-06T07:26:21.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.918+0000 7f69ee7fc640 1 -- 172.21.15.77:0/2310318614 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8031960 con 0x7f69f00b16b0 2024-03-06T07:26:21.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.919+0000 7f69f615c640 1 -- 172.21.15.77:0/2310318614 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 msgr2=0x7f69f00ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.919+0000 7f69f615c640 1 --2- 172.21.15.77:0/2310318614 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 0x7f69f00ce550 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f69d80097a0 tx=0x7f69d802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.919+0000 7f69f615c640 1 -- 172.21.15.77:0/2310318614 shutdown_connections 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.919+0000 7f69f615c640 1 --2- 172.21.15.77:0/2310318614 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 0x7f69f00ce550 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.919+0000 7f69f615c640 1 --2- 172.21.15.77:0/2310318614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f00b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.919+0000 7f69f615c640 1 -- 172.21.15.77:0/2310318614 >> 172.21.15.77:0/2310318614 conn(0x7f69f00d1d50 msgr2=0x7f69f00d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.919+0000 7f69f615c640 1 -- 172.21.15.77:0/2310318614 shutdown_connections 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.919+0000 7f69f615c640 1 -- 172.21.15.77:0/2310318614 wait complete. 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.920+0000 7f69f615c640 1 Processor -- start 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.920+0000 7f69f615c640 1 -- start start 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69f615c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 0x7f69f01492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69f615c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69f00d2870 0x7f69f00b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69f615c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f0149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69f615c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00cf270 con 0x7f69f00d2870 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69f615c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00cf0f0 con 0x7f69f00b16b0 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69f615c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00cea90 con 0x7f69f015f5e0 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69f00d2870 0x7f69f00b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69f00d2870 0x7f69f00b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37906/0 (socket says 172.21.15.77:37906) 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 -- 172.21.15.77:0/270965392 learned_addr learned my addr 172.21.15.77:0/270965392 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69ef7fe640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 0x7f69f01492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69effff640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f0149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 -- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 msgr2=0x7f69f0149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f0149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 -- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 msgr2=0x7f69f01492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 0x7f69f01492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 -- 172.21.15.77:0/270965392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f00b5660 con 0x7f69f00d2870 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69ef7fe640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 0x7f69f01492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.921+0000 7f69eeffd640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69f00d2870 0x7f69f00b4ba0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f69d80098c0 tx=0x7f69d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.922+0000 7f69e7fff640 1 -- 172.21.15.77:0/270965392 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8032630 con 0x7f69f00d2870 2024-03-06T07:26:21.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.922+0000 7f69e7fff640 1 -- 172.21.15.77:0/270965392 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d80327f0 con 0x7f69f00d2870 2024-03-06T07:26:21.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.922+0000 7f69e7fff640 1 -- 172.21.15.77:0/270965392 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8032be0 con 0x7f69f00d2870 2024-03-06T07:26:21.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.922+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f00b5940 con 0x7f69f00d2870 2024-03-06T07:26:21.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.922+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f015aec0 con 0x7f69f00d2870 2024-03-06T07:26:21.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.923+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f0068050 con 0x7f69f00d2870 2024-03-06T07:26:21.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.923+0000 7f69e7fff640 1 -- 172.21.15.77:0/270965392 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69d804c020 con 0x7f69f00d2870 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.924+0000 7f69e7fff640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d006b4d0 0x7f69d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.924+0000 7f69ef7fe640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d006b4d0 0x7f69d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.924+0000 7f69e7fff640 1 -- 172.21.15.77:0/270965392 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f69d802f020 con 0x7f69f00d2870 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.924+0000 7f69ef7fe640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d006b4d0 0x7f69d006d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009d20 tx=0x7f69e000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:21.928+0000 7f69e7fff640 1 -- 172.21.15.77:0/270965392 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69d8040070 con 0x7f69f00d2870 2024-03-06T07:26:22.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.136+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69f00d1b00 con 0x7f69f00d2870 2024-03-06T07:26:22.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.136+0000 7f69e7fff640 1 -- 172.21.15.77:0/270965392 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1801 (secure 0 0 0) 0x7f69d807ab90 con 0x7f69f00d2870 2024-03-06T07:26:22.138 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:22.138 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":52,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:21.753205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:22.139 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 52 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d006b4d0 msgr2=0x7f69d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d006b4d0 0x7f69d006d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009d20 tx=0x7f69e000b040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69f00d2870 msgr2=0x7f69f00b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69f00d2870 0x7f69f00b4ba0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f69d80098c0 tx=0x7f69d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 shutdown_connections 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d006b4d0 0x7f69d006d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69f00d2870 0x7f69f00b4ba0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f015f5e0 0x7f69f0149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 --2- 172.21.15.77:0/270965392 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f00b16b0 0x7f69f01492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 >> 172.21.15.77:0/270965392 conn(0x7f69f00d1d50 msgr2=0x7f69f00b23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 shutdown_connections 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:22.138+0000 7f69f615c640 1 -- 172.21.15.77:0/270965392 wait complete. 2024-03-06T07:26:22.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:09.152591+0000', 'modified': '2024-03-06T07:26:21.753205+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:22.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:26:23.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.665+0000 7f90cb615640 1 Processor -- start 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.666+0000 7f90cb615640 1 -- start start 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.666+0000 7f90cb615640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 0x7f90c40d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.666+0000 7f90cb615640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c40d1ea0 con 0x7f90c415e150 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.666+0000 7f90cb615640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c40d2180 con 0x7f90c40b9060 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.666+0000 7f90cb615640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c40d12f0 con 0x7f90c40d0950 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90c93b1640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f90c415e150 0x7f90c40cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58984/0 (socket says 172.21.15.77:58984) 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90c93b1640 1 -- 172.21.15.77:0/2270384022 learned_addr learned my addr 172.21.15.77:0/2270384022 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:23.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058173058 0 0) 0x7f90c40d1ea0 con 0x7f90c415e150 2024-03-06T07:26:23.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b0003520 con 0x7f90c415e150 2024-03-06T07:26:23.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90c8bb0640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 0x7f90c40d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1024712568 0 0) 0x7f90b0003520 con 0x7f90c415e150 2024-03-06T07:26:23.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 msgr2=0x7f90c40d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 0x7f90c40d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 >> v1:172.21.15.158:6789/0 conn(0x7f90c40b9060 legacy=0x7f90c40b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c40d2460 con 0x7f90c415e150 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f90c0001670 con 0x7f90c415e150 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90c8bb0640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 0x7f90c40d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 >> v1:172.21.15.77:6789/0 conn(0x7f90c415e150 legacy=0x7f90c40cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 0x7f90b00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90b0003fd0 0x7f90b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90b0004d40 0x7f90b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b000bb80 con 0x7f90b0004d40 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b000bd30 con 0x7f90b0003fd0 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b000bf30 con 0x7f90c40d0950 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90c9bb2640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90b0004d40 0x7f90b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90c8bb0640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90b0003fd0 0x7f90b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.667+0000 7f90c93b1640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 0x7f90b00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90c9bb2640 1 -- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 msgr2=0x7f90b00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90c9bb2640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40d0950 0x7f90b00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90c9bb2640 1 -- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90b0003fd0 msgr2=0x7f90b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90c9bb2640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90b0003fd0 0x7f90b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90c9bb2640 1 -- 172.21.15.77:0/2270384022 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b0003520 con 0x7f90b0004d40 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90c9bb2640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90b0004d40 0x7f90b00051c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f90bc001e50 tx=0x7f90b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0dd18f6c3e66db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc001a70 con 0x7f90b0004d40 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc003b70 con 0x7f90b0004d40 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.668+0000 7f90bbfff640 1 -- 172.21.15.77:0/2270384022 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0050d0 con 0x7f90b0004d40 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.669+0000 7f90cb615640 1 -- 172.21.15.77:0/2270384022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90b0004d40 msgr2=0x7f90b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.669+0000 7f90cb615640 1 --2- 172.21.15.77:0/2270384022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90b0004d40 0x7f90b00051c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f90bc001e50 tx=0x7f90b0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.669+0000 7f90c9bb2640 1 -- 172.21.15.77:0/2270384022 reap_dead start 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.669+0000 7f90cb615640 1 -- 172.21.15.77:0/2270384022 shutdown_connections 2024-03-06T07:26:23.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.669+0000 7f90cb615640 1 -- 172.21.15.77:0/2270384022 >> 172.21.15.77:0/2270384022 conn(0x7f90c409d230 msgr2=0x7f90c40d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.669+0000 7f90cb615640 1 -- 172.21.15.77:0/2270384022 shutdown_connections 2024-03-06T07:26:23.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.669+0000 7f90cb615640 1 -- 172.21.15.77:0/2270384022 wait complete. 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.670+0000 7f90cb615640 1 Processor -- start 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.670+0000 7f90cb615640 1 -- start start 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90cb615640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c414cd10 0x7f90c414d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90cb615640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c414ff50 0x7f90c414eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90cb615640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40cd4a0 0x7f90c40cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90cb615640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c40d2f40 con 0x7f90c414ff50 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90cb615640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c40d2ab0 con 0x7f90c414cd10 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90cb615640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c40b7dc0 con 0x7f90c40cd4a0 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c8bb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c414ff50 0x7f90c414eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c8bb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c414ff50 0x7f90c414eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37910/0 (socket says 172.21.15.77:37910) 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c8bb0640 1 -- 172.21.15.77:0/296582238 learned_addr learned my addr 172.21.15.77:0/296582238 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c93b1640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c414cd10 0x7f90c414d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c9bb2640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40cd4a0 0x7f90c40cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c8bb0640 1 -- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40cd4a0 msgr2=0x7f90c40cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c8bb0640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40cd4a0 0x7f90c40cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c8bb0640 1 -- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c414cd10 msgr2=0x7f90c414d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c8bb0640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c414cd10 0x7f90c414d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c8bb0640 1 -- 172.21.15.77:0/296582238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c41503d0 con 0x7f90c414ff50 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c93b1640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c414cd10 0x7f90c414d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.671+0000 7f90c9bb2640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40cd4a0 0x7f90c40cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.672+0000 7f90c8bb0640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c414ff50 0x7f90c414eb70 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f90b4011620 tx=0x7f90b4011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.672+0000 7f90b9ffb640 1 -- 172.21.15.77:0/296582238 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4019070 con 0x7f90c414ff50 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.672+0000 7f90b9ffb640 1 -- 172.21.15.77:0/296582238 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b4004220 con 0x7f90c414ff50 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.672+0000 7f90b9ffb640 1 -- 172.21.15.77:0/296582238 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400f040 con 0x7f90c414ff50 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.672+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c414dad0 con 0x7f90c414ff50 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.672+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c414e090 con 0x7f90c414ff50 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.673+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c4069ee0 con 0x7f90c414ff50 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.673+0000 7f90b9ffb640 1 -- 172.21.15.77:0/296582238 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90b400f720 con 0x7f90c414ff50 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.674+0000 7f90b9ffb640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90a806b4d0 0x7f90a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.674+0000 7f90b9ffb640 1 -- 172.21.15.77:0/296582238 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f90b408a600 con 0x7f90c414ff50 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.674+0000 7f90c93b1640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90a806b4d0 0x7f90a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.674+0000 7f90c93b1640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90a806b4d0 0x7f90a806d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f90c0004840 tx=0x7f90c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.678+0000 7f90b9ffb640 1 -- 172.21.15.77:0/296582238 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90b40592f0 con 0x7f90c414ff50 2024-03-06T07:26:23.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.888+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90c40d4670 con 0x7f90c414ff50 2024-03-06T07:26:23.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.888+0000 7f90b9ffb640 1 -- 172.21.15.77:0/296582238 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4970 (secure 0 0 0) 0x7f90b4058bf0 con 0x7f90c414ff50 2024-03-06T07:26:23.890 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:23.890 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:22.771500+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:23.891 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 55 2024-03-06T07:26:23.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90a806b4d0 msgr2=0x7f90a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90a806b4d0 0x7f90a806d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f90c0004840 tx=0x7f90c0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:23.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c414ff50 msgr2=0x7f90c414eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c414ff50 0x7f90c414eb70 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f90b4011620 tx=0x7f90b4011ae0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 shutdown_connections 2024-03-06T07:26:23.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90a806b4d0 0x7f90a806d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c414ff50 0x7f90c414eb70 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c40cd4a0 0x7f90c40cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 --2- 172.21.15.77:0/296582238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c414cd10 0x7f90c414d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 >> 172.21.15.77:0/296582238 conn(0x7f90c409d230 msgr2=0x7f90c40cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 shutdown_connections 2024-03-06T07:26:23.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:23.890+0000 7f90cb615640 1 -- 172.21.15.77:0/296582238 wait complete. 2024-03-06T07:26:24.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:09.152591+0000', 'modified': '2024-03-06T07:26:22.771500+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6985}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6985': {'gid': 6985, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.77:6837/82690693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 82690693}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 82690693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:24.307 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6985:up:reconnect 2024-03-06T07:26:25.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:25.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.418+0000 7f835fa1c640 1 Processor -- start 2024-03-06T07:26:25.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.418+0000 7f835fa1c640 1 -- start start 2024-03-06T07:26:25.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835fa1c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f83580b3cc0 0x7f83580afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835fa1c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83580affa0 0x7f835815e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835fa1c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835806b190 con 0x7f8358158570 2024-03-06T07:26:25.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835fa1c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f835806b470 con 0x7f83580b3cc0 2024-03-06T07:26:25.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835fa1c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835806b750 con 0x7f83580affa0 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835dfb9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83580affa0 0x7f835815e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835cfb7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f83580b3cc0 0x7f83580afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835dfb9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83580affa0 0x7f835815e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:38220/0 (socket says 172.21.15.77:38220) 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835d7b8640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f8358158570 0x7f835806f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49442/0 (socket says 172.21.15.77:49442) 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.419+0000 7f835dfb9640 1 -- 172.21.15.77:0/1454551418 learned_addr learned my addr 172.21.15.77:0/1454551418 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.420+0000 7f835dfb9640 1 -- 172.21.15.77:0/1454551418 >> v2:172.21.15.158:3300/0 conn(0x7f83580b3cc0 msgr2=0x7f83580afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.420+0000 7f835dfb9640 1 --2- 172.21.15.77:0/1454551418 >> v2:172.21.15.158:3300/0 conn(0x7f83580b3cc0 0x7f83580afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.420+0000 7f835dfb9640 1 -- 172.21.15.77:0/1454551418 >> v1:172.21.15.158:6789/0 conn(0x7f8358158570 legacy=0x7f835806f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.420+0000 7f835dfb9640 1 -- 172.21.15.77:0/1454551418 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835806ba30 con 0x7f83580affa0 2024-03-06T07:26:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.420+0000 7f835cfb7640 1 --2- 172.21.15.77:0/1454551418 >> v2:172.21.15.158:3300/0 conn(0x7f83580b3cc0 0x7f83580afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.420+0000 7f835dfb9640 1 --2- 172.21.15.77:0/1454551418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83580affa0 0x7f835815e620 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8350009e50 tx=0x7f835002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adec14b294d812b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f834ffff640 1 -- 172.21.15.77:0/1454551418 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835002cb00 con 0x7f83580affa0 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f834ffff640 1 -- 172.21.15.77:0/1454551418 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835002cca0 con 0x7f83580affa0 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f834ffff640 1 -- 172.21.15.77:0/1454551418 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350034610 con 0x7f83580affa0 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f835fa1c640 1 -- 172.21.15.77:0/1454551418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83580affa0 msgr2=0x7f835815e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f835fa1c640 1 --2- 172.21.15.77:0/1454551418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83580affa0 0x7f835815e620 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8350009e50 tx=0x7f835002bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f835fa1c640 1 -- 172.21.15.77:0/1454551418 shutdown_connections 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f835fa1c640 1 --2- 172.21.15.77:0/1454551418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83580affa0 0x7f835815e620 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f835fa1c640 1 --2- 172.21.15.77:0/1454551418 >> v2:172.21.15.158:3300/0 conn(0x7f83580b3cc0 0x7f83580afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f835fa1c640 1 -- 172.21.15.77:0/1454551418 >> 172.21.15.77:0/1454551418 conn(0x7f83580b29c0 msgr2=0x7f83580b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f835fa1c640 1 -- 172.21.15.77:0/1454551418 shutdown_connections 2024-03-06T07:26:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.421+0000 7f835fa1c640 1 -- 172.21.15.77:0/1454551418 wait complete. 2024-03-06T07:26:25.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.422+0000 7f835fa1c640 1 Processor -- start 2024-03-06T07:26:25.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.422+0000 7f835fa1c640 1 -- start start 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835fa1c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580affa0 0x7f83580d1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835fa1c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580b3cc0 0x7f83581442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835fa1c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8358158570 0x7f8358144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835fa1c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835815ece0 con 0x7f83580affa0 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835fa1c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835815eb60 con 0x7f83580b3cc0 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835fa1c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835815ee60 con 0x7f8358158570 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835d7b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580affa0 0x7f83580d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835d7b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580affa0 0x7f83580d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37924/0 (socket says 172.21.15.77:37924) 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835d7b8640 1 -- 172.21.15.77:0/2295887328 learned_addr learned my addr 172.21.15.77:0/2295887328 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835cfb7640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580b3cc0 0x7f83581442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.423+0000 7f835dfb9640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8358158570 0x7f8358144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f835d7b8640 1 -- 172.21.15.77:0/2295887328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8358158570 msgr2=0x7f8358144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f835d7b8640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8358158570 0x7f8358144820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f835d7b8640 1 -- 172.21.15.77:0/2295887328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580b3cc0 msgr2=0x7f83581442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f835d7b8640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580b3cc0 0x7f83581442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f835d7b8640 1 -- 172.21.15.77:0/2295887328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8358145160 con 0x7f83580affa0 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f835d7b8640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580affa0 0x7f83580d1250 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f835400c3f0 tx=0x7f835400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f834dffb640 1 -- 172.21.15.77:0/2295887328 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83540074e0 con 0x7f83580affa0 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f834dffb640 1 -- 172.21.15.77:0/2295887328 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8354007680 con 0x7f83580affa0 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83581e9490 con 0x7f83580affa0 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f834dffb640 1 -- 172.21.15.77:0/2295887328 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8354004a00 con 0x7f83580affa0 2024-03-06T07:26:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.424+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83581e9a50 con 0x7f83580affa0 2024-03-06T07:26:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.425+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8358069ee0 con 0x7f83580affa0 2024-03-06T07:26:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.425+0000 7f834dffb640 1 -- 172.21.15.77:0/2295887328 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f835401a020 con 0x7f83580affa0 2024-03-06T07:26:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.426+0000 7f834dffb640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f833c06b4d0 0x7f833c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.426+0000 7f835cfb7640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f833c06b4d0 0x7f833c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.426+0000 7f834dffb640 1 -- 172.21.15.77:0/2295887328 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f8354089910 con 0x7f83580affa0 2024-03-06T07:26:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.426+0000 7f835cfb7640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f833c06b4d0 0x7f833c06d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8348009560 tx=0x7f8348009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.430+0000 7f834dffb640 1 -- 172.21.15.77:0/2295887328 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8354058040 con 0x7f83580affa0 2024-03-06T07:26:25.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.636+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f835806de00 con 0x7f83580affa0 2024-03-06T07:26:25.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.636+0000 7f834dffb640 1 -- 172.21.15.77:0/2295887328 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4967 (secure 0 0 0) 0x7f835405c0e0 con 0x7f83580affa0 2024-03-06T07:26:25.638 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:25.638 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:23.767957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:25.639 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 56 2024-03-06T07:26:25.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.638+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f833c06b4d0 msgr2=0x7f833c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.638+0000 7f835fa1c640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f833c06b4d0 0x7f833c06d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8348009560 tx=0x7f8348009190 comp rx=0 tx=0).stop 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.638+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580affa0 msgr2=0x7f83580d1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.638+0000 7f835fa1c640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580affa0 0x7f83580d1250 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f835400c3f0 tx=0x7f835400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.639+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 shutdown_connections 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.639+0000 7f835fa1c640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8358158570 0x7f8358144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.639+0000 7f835fa1c640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580b3cc0 0x7f83581442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.639+0000 7f835fa1c640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f833c06b4d0 0x7f833c06d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.639+0000 7f835fa1c640 1 --2- 172.21.15.77:0/2295887328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580affa0 0x7f83580d1250 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.639+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 >> 172.21.15.77:0/2295887328 conn(0x7f83580b29c0 msgr2=0x7f835806ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.639+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 shutdown_connections 2024-03-06T07:26:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:25.639+0000 7f835fa1c640 1 -- 172.21.15.77:0/2295887328 wait complete. 2024-03-06T07:26:26.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:09.152591+0000', 'modified': '2024-03-06T07:26:23.767957+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6985}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6985': {'gid': 6985, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.77:6837/82690693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 82690693}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 82690693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:26.054 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6985:up:rejoin 2024-03-06T07:26:27.054 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:27.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.153+0000 7f72ed57a640 1 Processor -- start 2024-03-06T07:26:27.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.153+0000 7f72ed57a640 1 -- start start 2024-03-06T07:26:27.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72ed57a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e815f5e0 0x7f72e80b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72ed57a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72e80b16b0 0x7f72e80ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72ed57a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72e80d2870 0x7f72e80cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72ed57a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e80cf920 con 0x7f72e80d2870 2024-03-06T07:26:27.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72ed57a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e80cd4d0 con 0x7f72e80b16b0 2024-03-06T07:26:27.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72ed57a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80cd7b0 con 0x7f72e815f5e0 2024-03-06T07:26:27.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72e77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72e80d2870 0x7f72e80cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72e77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72e80d2870 0x7f72e80cea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37926/0 (socket says 172.21.15.77:37926) 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72e77fe640 1 -- 172.21.15.77:0/1197483616 learned_addr learned my addr 172.21.15.77:0/1197483616 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.154+0000 7f72de7fc640 1 --2- 172.21.15.77:0/1197483616 >> v2:172.21.15.158:3300/0 conn(0x7f72e80b16b0 0x7f72e80ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e77fe640 1 -- 172.21.15.77:0/1197483616 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e815f5e0 msgr2=0x7f72e80b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e77fe640 1 --2- 172.21.15.77:0/1197483616 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e815f5e0 0x7f72e80b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e77fe640 1 -- 172.21.15.77:0/1197483616 >> v2:172.21.15.158:3300/0 conn(0x7f72e80b16b0 msgr2=0x7f72e80ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e77fe640 1 --2- 172.21.15.77:0/1197483616 >> v2:172.21.15.158:3300/0 conn(0x7f72e80b16b0 0x7f72e80ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e77fe640 1 -- 172.21.15.77:0/1197483616 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80cda90 con 0x7f72e80d2870 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72de7fc640 1 --2- 172.21.15.77:0/1197483616 >> v2:172.21.15.158:3300/0 conn(0x7f72e80b16b0 0x7f72e80ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e77fe640 1 --2- 172.21.15.77:0/1197483616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e80d2870 0x7f72e80cea90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f72e8071c80 tx=0x7f72e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=204f90a18e0317a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e67fc640 1 -- 172.21.15.77:0/1197483616 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e002c9e0 con 0x7f72e80d2870 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e67fc640 1 -- 172.21.15.77:0/1197483616 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e002cb80 con 0x7f72e80d2870 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.155+0000 7f72e67fc640 1 -- 172.21.15.77:0/1197483616 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e00343f0 con 0x7f72e80d2870 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 -- 172.21.15.77:0/1197483616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e80d2870 msgr2=0x7f72e80cea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 --2- 172.21.15.77:0/1197483616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e80d2870 0x7f72e80cea90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f72e8071c80 tx=0x7f72e002be40 comp rx=0 tx=0).stop 2024-03-06T07:26:27.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 -- 172.21.15.77:0/1197483616 shutdown_connections 2024-03-06T07:26:27.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 --2- 172.21.15.77:0/1197483616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e80d2870 0x7f72e80cea90 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 --2- 172.21.15.77:0/1197483616 >> v2:172.21.15.158:3300/0 conn(0x7f72e80b16b0 0x7f72e80ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 --2- 172.21.15.77:0/1197483616 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e815f5e0 0x7f72e80b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 -- 172.21.15.77:0/1197483616 >> 172.21.15.77:0/1197483616 conn(0x7f72e80d1d50 msgr2=0x7f72e80d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 -- 172.21.15.77:0/1197483616 shutdown_connections 2024-03-06T07:26:27.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.156+0000 7f72ed57a640 1 -- 172.21.15.77:0/1197483616 wait complete. 2024-03-06T07:26:27.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.157+0000 7f72ed57a640 1 Processor -- start 2024-03-06T07:26:27.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.157+0000 7f72ed57a640 1 -- start start 2024-03-06T07:26:27.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72ed57a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e80b16b0 0x7f72e815a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72ed57a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e80d2870 0x7f72e81575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72ed57a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e815f5e0 0x7f72e80cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72ed57a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80cf270 con 0x7f72e815f5e0 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72ed57a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80cf0f0 con 0x7f72e80b16b0 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72ed57a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e815fcd0 con 0x7f72e80d2870 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e815f5e0 0x7f72e80cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e815f5e0 0x7f72e80cb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37940/0 (socket says 172.21.15.77:37940) 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72e77fe640 1 -- 172.21.15.77:0/2103981654 learned_addr learned my addr 172.21.15.77:0/2103981654 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72de7fc640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e80d2870 0x7f72e81575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72e6ffd640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e80b16b0 0x7f72e815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72e77fe640 1 -- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e80d2870 msgr2=0x7f72e81575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72e77fe640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e80d2870 0x7f72e81575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72e77fe640 1 -- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e80b16b0 msgr2=0x7f72e815a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.158+0000 7f72e77fe640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e80b16b0 0x7f72e815a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.159+0000 7f72e77fe640 1 -- 172.21.15.77:0/2103981654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e815ae60 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.159+0000 7f72e6ffd640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e80b16b0 0x7f72e815a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.159+0000 7f72e77fe640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e815f5e0 0x7f72e80cb3b0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009f40 tx=0x7f72e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.159+0000 7f72dffff640 1 -- 172.21.15.77:0/2103981654 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0043020 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.159+0000 7f72dffff640 1 -- 172.21.15.77:0/2103981654 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e003e450 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.159+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e80b6810 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.159+0000 7f72dffff640 1 -- 172.21.15.77:0/2103981654 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e003e860 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.159+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e80b6d90 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.160+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e8068050 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.160+0000 7f72dffff640 1 -- 172.21.15.77:0/2103981654 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72e0005d10 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.161+0000 7f72dffff640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.161+0000 7f72dffff640 1 -- 172.21.15.77:0/2103981654 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f72e00acf00 con 0x7f72e815f5e0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.161+0000 7f72e6ffd640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.161+0000 7f72e6ffd640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f72d800a870 tx=0x7f72d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.165+0000 7f72dffff640 1 -- 172.21.15.77:0/2103981654 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72e815ae60 con 0x7f72e815f5e0 2024-03-06T07:26:27.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.432+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72e80d00c0 con 0x7f72e815f5e0 2024-03-06T07:26:27.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.433+0000 7f72dffff640 1 -- 172.21.15.77:0/2103981654 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f72e007bc80 con 0x7f72e815f5e0 2024-03-06T07:26:27.435 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:27.435 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:27.435 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 msgr2=0x7f72c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f72d800a870 tx=0x7f72d8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e815f5e0 msgr2=0x7f72e80cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e815f5e0 0x7f72e80cb3b0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009f40 tx=0x7f72e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 shutdown_connections 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e815f5e0 0x7f72e80cb3b0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e80d2870 0x7f72e81575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 --2- 172.21.15.77:0/2103981654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e80b16b0 0x7f72e815a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 >> 172.21.15.77:0/2103981654 conn(0x7f72e80d1d50 msgr2=0x7f72e80ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 shutdown_connections 2024-03-06T07:26:27.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.435+0000 7f72ed57a640 1 -- 172.21.15.77:0/2103981654 wait complete. 2024-03-06T07:26:27.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2024-03-06T07:26:27.764+0000 7fbdc6b76640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:27.850 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:09.152591+0000', 'modified': '2024-03-06T07:26:26.774080+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6985}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6985': {'gid': 6985, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.77:6837/82690693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 82690693}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 82690693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6985, 'qdb_cluster': [6985]} 2024-03-06T07:26:27.850 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:27.851 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:27.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.951+0000 7fdc9569a640 1 Processor -- start 2024-03-06T07:26:27.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.951+0000 7fdc9569a640 1 -- start start 2024-03-06T07:26:27.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc9569a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc900d0950 0x7fdc900d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc9569a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc900d1ea0 con 0x7fdc9015e150 2024-03-06T07:26:27.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc9569a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc900d2180 con 0x7fdc900b9060 2024-03-06T07:26:27.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc9569a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc900d12f0 con 0x7fdc900d0950 2024-03-06T07:26:27.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8effd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdc9015e150 0x7fdc900cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58998/0 (socket says 172.21.15.77:58998) 2024-03-06T07:26:27.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8effd640 1 -- 172.21.15.77:0/3512477135 learned_addr learned my addr 172.21.15.77:0/3512477135 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:27.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172331925 0 0) 0x7fdc900d1ea0 con 0x7fdc9015e150 2024-03-06T07:26:27.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc7c003520 con 0x7fdc9015e150 2024-03-06T07:26:27.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc87fff640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc900d0950 0x7fdc900d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4088900788 0 0) 0x7fdc7c003520 con 0x7fdc9015e150 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc900d0950 msgr2=0x7fdc900d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc900d0950 0x7fdc900d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 >> v1:172.21.15.158:6789/0 conn(0x7fdc900b9060 legacy=0x7fdc900b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc900d2460 con 0x7fdc9015e150 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fdc80001670 con 0x7fdc9015e150 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 >> v1:172.21.15.77:6789/0 conn(0x7fdc9015e150 legacy=0x7fdc900cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c0039e0 0x7fdc7c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.952+0000 7fdc8e7fc640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c004800 0x7fdc7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8e7fc640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc7c005210 0x7fdc7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c00c0e0 con 0x7fdc7c005210 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c00c290 con 0x7fdc7c004800 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c00c490 con 0x7fdc7c0039e0 2024-03-06T07:26:27.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc87fff640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc900d0950 0x7fdc900d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc7c005210 0x7fdc7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8effd640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c0039e0 0x7fdc7c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc87fff640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c004800 0x7fdc7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c0039e0 msgr2=0x7fdc7c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c0039e0 0x7fdc7c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c004800 msgr2=0x7fdc7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c004800 0x7fdc7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3512477135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c003520 con 0x7fdc7c005210 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc87fff640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc7c004800 0x7fdc7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.953+0000 7fdc8effd640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c0039e0 0x7fdc7c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.954+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc7c005210 0x7fdc7c00b9c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fdc88001e50 tx=0x7fdc7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71011a033aa2ca06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.954+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88001a70 con 0x7fdc7c005210 2024-03-06T07:26:27.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.954+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88003b70 con 0x7fdc7c005210 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.954+0000 7fdc8e7fc640 1 -- 172.21.15.77:0/3512477135 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc880050d0 con 0x7fdc7c005210 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.955+0000 7fdc9569a640 1 -- 172.21.15.77:0/3512477135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc7c005210 msgr2=0x7fdc7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.955+0000 7fdc9569a640 1 --2- 172.21.15.77:0/3512477135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc7c005210 0x7fdc7c00b9c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fdc88001e50 tx=0x7fdc7c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.955+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3512477135 reap_dead start 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.955+0000 7fdc9569a640 1 -- 172.21.15.77:0/3512477135 shutdown_connections 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.955+0000 7fdc9569a640 1 -- 172.21.15.77:0/3512477135 >> 172.21.15.77:0/3512477135 conn(0x7fdc9009d230 msgr2=0x7fdc900d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.955+0000 7fdc9569a640 1 -- 172.21.15.77:0/3512477135 shutdown_connections 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.955+0000 7fdc9569a640 1 -- 172.21.15.77:0/3512477135 wait complete. 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.956+0000 7fdc9569a640 1 Processor -- start 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.956+0000 7fdc9569a640 1 -- start start 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc9569a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c005210 0x7fdc90153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc9569a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc901600a0 0x7fdc9015e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc9569a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9015c520 0x7fdc90160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc9569a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc900d2f40 con 0x7fdc9015c520 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc9569a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc900d2ab0 con 0x7fdc901600a0 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc9569a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc900b7dc0 con 0x7fdc7c005210 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9015c520 0x7fdc90160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9015c520 0x7fdc90160b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37968/0 (socket says 172.21.15.77:37968) 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/2158489429 learned_addr learned my addr 172.21.15.77:0/2158489429 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8effd640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c005210 0x7fdc90153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc87fff640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc901600a0 0x7fdc9015e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c005210 msgr2=0x7fdc90153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c005210 0x7fdc90153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc901600a0 msgr2=0x7fdc9015e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc901600a0 0x7fdc9015e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/2158489429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc901539a0 con 0x7fdc9015c520 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8effd640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c005210 0x7fdc90153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9015c520 0x7fdc90160b00 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fdc8802c060 tx=0x7fdc880014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc877fe640 1 -- 172.21.15.77:0/2158489429 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88007db0 con 0x7fdc9015c520 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc877fe640 1 -- 172.21.15.77:0/2158489429 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88003970 con 0x7fdc9015c520 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc9569a640 1 -- 172.21.15.77:0/2158489429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9015f000 con 0x7fdc9015c520 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc877fe640 1 -- 172.21.15.77:0/2158489429 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88006d90 con 0x7fdc9015c520 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.957+0000 7fdc9569a640 1 -- 172.21.15.77:0/2158489429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9015f5c0 con 0x7fdc9015c520 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.959+0000 7fdc877fe640 1 -- 172.21.15.77:0/2158489429 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc88007030 con 0x7fdc9015c520 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.959+0000 7fdc877fe640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7006b500 0x7fdc7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.959+0000 7fdc877fe640 1 -- 172.21.15.77:0/2158489429 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdc880ab410 con 0x7fdc9015c520 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.959+0000 7fdc8effd640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7006b500 0x7fdc7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.960+0000 7fdc8effd640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7006b500 0x7fdc7006d9c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdc80000fd0 tx=0x7fdc80009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.960+0000 7fdc98281740 1 Processor -- start 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.960+0000 7fdc98281740 1 -- start start 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc98281740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13c9870 0x55e6f13c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc98281740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13cd8c0 0x55e6f13cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc98281740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc98281740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6f13336b0 con 0x55e6f13f9600 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc98281740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6f12dca60 con 0x55e6f13c9870 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc98281740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e6f132f750 con 0x55e6f13cd8c0 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37970/0 (socket says 172.21.15.77:37970) 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3215919794 learned_addr learned my addr 172.21.15.77:0/3215919794 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8effd640 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13c9870 0x55e6f13c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc87fff640 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13cd8c0 0x55e6f13cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3215919794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13cd8c0 msgr2=0x55e6f13cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13cd8c0 0x55e6f13cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3215919794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13c9870 msgr2=0x55e6f13c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13c9870 0x55e6f13c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.961+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3215919794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6f12faf30 con 0x55e6f13f9600 2024-03-06T07:26:27.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f9a80 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fdc880a9920 tx=0x7fdc88075fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600cc13aca92752d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc84ff9640 1 -- 172.21.15.77:0/3215919794 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88048030 con 0x55e6f13f9600 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc84ff9640 1 -- 172.21.15.77:0/3215919794 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88079c00 con 0x55e6f13f9600 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc84ff9640 1 -- 172.21.15.77:0/3215919794 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8807dd50 con 0x55e6f13f9600 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 -- 172.21.15.77:0/3215919794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 msgr2=0x55e6f13f9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f9a80 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fdc880a9920 tx=0x7fdc88075fd0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 -- 172.21.15.77:0/3215919794 shutdown_connections 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f9a80 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13cd8c0 0x55e6f13cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 --2- 172.21.15.77:0/3215919794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13c9870 0x55e6f13c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 -- 172.21.15.77:0/3215919794 >> 172.21.15.77:0/3215919794 conn(0x55e6f13cb8f0 msgr2=0x55e6f13cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 -- 172.21.15.77:0/3215919794 shutdown_connections 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.962+0000 7fdc98281740 1 -- 172.21.15.77:0/3215919794 wait complete. 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.963+0000 7fdc98281740 1 -- start start 2024-03-06T07:26:27.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.963+0000 7fdc98281740 1 Processor -- start 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc98281740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 0x55e6f13f5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc98281740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc600 0x55e6f14115b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc98281740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc98281740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6f12dca60 con 0x55e6f13fc9e0 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc98281740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6f12efea0 con 0x55e6f13fc600 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc98281740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e6f13336b0 con 0x55e6f13f9600 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc98281740 10 client.0 _ll_register_callbacks cb 0x55e6f12ef870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37978/0 (socket says 172.21.15.77:37978) 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3885909523 learned_addr learned my addr 172.21.15.77:0/3885909523 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc87fff640 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc600 0x55e6f14115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8effd640 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 0x55e6f13f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 msgr2=0x55e6f13f5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 0x55e6f13f5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc600 msgr2=0x55e6f14115b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc600 0x55e6f14115b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 -- 172.21.15.77:0/3885909523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6f12f0270 con 0x55e6f13fc9e0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc87fff640 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc600 0x55e6f14115b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411af0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fdc88077eb0 tx=0x7fdc88045070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b6028b33ed8b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc6dffb640 1 -- 172.21.15.77:0/3885909523 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88048040 con 0x55e6f13fc9e0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc6dffb640 1 -- 172.21.15.77:0/3885909523 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdc8808dbc0 con 0x55e6f13fc9e0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.964+0000 7fdc98281740 1 -- 172.21.15.77:0/3885909523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc88079c00 con 0x55e6f13fc9e0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc6dffb640 1 -- 172.21.15.77:0/3885909523 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc8808ddc0 con 0x55e6f13fc9e0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc98281740 10 client.7569 fetch_fsmap learned FSMap version 58 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc98281740 1 -- 172.21.15.77:0/3885909523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x55e6f10482d0 con 0x55e6f13fc9e0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc6dffb640 1 -- 172.21.15.77:0/3885909523 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88079c00 con 0x55e6f13fc9e0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc6dffb640 1 -- 172.21.15.77:0/3885909523 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 58) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdc88079050 con 0x55e6f13fc9e0 2024-03-06T07:26:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc98281740 10 client.7569 fetch_fsmap finished waiting for FSMap version 58 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc98281740 10 client.7569 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc98281740 20 client.7569 populate_metadata read hostname 'smithi077' 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc98281740 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] conn(0x55e6f14173c0 0x55e6f14177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc98281740 4 client.7569 mds_command: new command op to 6985 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc98281740 1 -- 172.21.15.77:0/3885909523 --> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6f1340bf0 con 0x55e6f14173c0 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc8effd640 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] conn(0x55e6f14173c0 0x55e6f14177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc6dffb640 10 client.7569 ms_handle_connect on v2:172.21.15.77:6836/82690693 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.965+0000 7fdc8effd640 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] conn(0x55e6f14173c0 0x55e6f14177a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac12e37767f15d2 server_cookie=a99747a56f67217d in_seq=0 out_seq=0 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc6dffb640 1 -- 172.21.15.77:0/3885909523 <== mds.0 v2:172.21.15.77:6836/82690693 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e6f1340bf0 con 0x55e6f14173c0 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc6dffb640 10 client.7569 handle_command_reply: tid=0 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 client.7569 shutdown 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 -- 172.21.15.77:0/3885909523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc9e0 msgr2=0x55e6f1411af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411af0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fdc88077eb0 tx=0x7fdc88045070 comp rx=0 tx=0).stop 2024-03-06T07:26:27.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 -- 172.21.15.77:0/3885909523 shutdown_connections 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] conn(0x55e6f14173c0 0x55e6f14177a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411af0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc6dffb640 0 client.7569 ms_handle_reset on v2:172.21.15.77:6836/82690693 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc600 0x55e6f14115b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 --2- 172.21.15.77:0/3885909523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 0x55e6f13f5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.966+0000 7fdc98281740 1 -- 172.21.15.77:0/3885909523 >> 172.21.15.77:0/3885909523 conn(0x55e6f13c9870 msgr2=0x55e6f13f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.967+0000 7fdc98281740 1 -- 172.21.15.77:0/3885909523 shutdown_connections 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.967+0000 7fdc98281740 1 -- 172.21.15.77:0/3885909523 wait complete. 2024-03-06T07:26:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.967+0000 7fdc98281740 20 client.7569 trim_cache size 0 max 16384 2024-03-06T07:26:27.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.983+0000 7fdc98281740 1 Processor -- start 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.983+0000 7fdc98281740 1 -- start start 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc98281740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc98281740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13fc600 0x55e6f14141c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc98281740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc98281740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901ffec0 con 0x55e6f13f9600 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc98281740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc900b2b50 con 0x55e6f13fc9e0 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc98281740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901ff8c0 con 0x55e6f13fc600 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37982/0 (socket says 172.21.15.77:37982) 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 -- 172.21.15.77:0/1842381266 learned_addr learned my addr 172.21.15.77:0/1842381266 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 -- 172.21.15.77:0/1842381266 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13fc600 msgr2=0x55e6f14141c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13fc600 0x55e6f14141c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 -- 172.21.15.77:0/1842381266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 msgr2=0x55e6f1411570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 -- 172.21.15.77:0/1842381266 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6f11126b0 con 0x55e6f13f9600 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc8effd640 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f79e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fdc80001200 tx=0x7fdc8000c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580056da1c8c09e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.984+0000 7fdc6ffff640 1 -- 172.21.15.77:0/1842381266 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8002e070 con 0x55e6f13f9600 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc6ffff640 1 -- 172.21.15.77:0/1842381266 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc80010e70 con 0x55e6f13f9600 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc6ffff640 1 -- 172.21.15.77:0/1842381266 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80024960 con 0x55e6f13f9600 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 -- 172.21.15.77:0/1842381266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 msgr2=0x55e6f13f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f79e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fdc80001200 tx=0x7fdc8000c000 comp rx=0 tx=0).stop 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 -- 172.21.15.77:0/1842381266 shutdown_connections 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1411570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13fc600 0x55e6f14141c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 --2- 172.21.15.77:0/1842381266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13f9600 0x55e6f13f79e0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 -- 172.21.15.77:0/1842381266 >> 172.21.15.77:0/1842381266 conn(0x55e6f13c9870 msgr2=0x55e6f13f3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 -- 172.21.15.77:0/1842381266 shutdown_connections 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.985+0000 7fdc98281740 1 -- 172.21.15.77:0/1842381266 wait complete. 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 1 -- start start 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 1 Processor -- start 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 0x55e6f14145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc600 0x55e6f1482cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1483230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6f12dca60 con 0x55e6f13fc600 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6f13336b0 con 0x55e6f13fc9e0 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e6f12efea0 con 0x55e6f13f9600 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.986+0000 7fdc98281740 10 client.0 _ll_register_callbacks cb 0x7fdc88082350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc600 0x55e6f1482cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc600 0x55e6f1482cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37996/0 (socket says 172.21.15.77:37996) 2024-03-06T07:26:27.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 -- 172.21.15.77:0/1649145867 learned_addr learned my addr 172.21.15.77:0/1649145867 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc8effd640 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 0x55e6f14145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1483230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 -- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 msgr2=0x55e6f14145c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 0x55e6f14145c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 -- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 msgr2=0x55e6f1483230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1483230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 -- 172.21.15.77:0/1649145867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6f1414b00 con 0x55e6f13fc600 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc8f7fe640 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1483230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc87fff640 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc600 0x55e6f1482cf0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fdc9006b870 tx=0x7fdc78015730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1c3ab366349bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc6dffb640 1 -- 172.21.15.77:0/1649145867 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7801b070 con 0x55e6f13fc600 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc6dffb640 1 -- 172.21.15.77:0/1649145867 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdc78020bb0 con 0x55e6f13fc600 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc98281740 1 -- 172.21.15.77:0/1649145867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc80010e70 con 0x55e6f13fc600 2024-03-06T07:26:27.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc6dffb640 1 -- 172.21.15.77:0/1649145867 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc78015e80 con 0x55e6f13fc600 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc98281740 10 client.7575 fetch_fsmap learned FSMap version 58 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc98281740 1 -- 172.21.15.77:0/1649145867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x55e6f1418700 con 0x55e6f13fc600 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.987+0000 7fdc6dffb640 1 -- 172.21.15.77:0/1649145867 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc80010e70 con 0x55e6f13fc600 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc6dffb640 1 -- 172.21.15.77:0/1649145867 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 58) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdc78004030 con 0x55e6f13fc600 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 10 client.7575 fetch_fsmap finished waiting for FSMap version 58 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 10 client.7575 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 20 client.7575 populate_metadata read hostname 'smithi077' 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] conn(0x55e6f14180d0 0x55e6f1437050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 4 client.7575 mds_command: new command op to 6985 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 -- 172.21.15.77:0/1649145867 --> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e6f133af40 con 0x55e6f14180d0 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc8effd640 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] conn(0x55e6f14180d0 0x55e6f1437050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc8effd640 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] conn(0x55e6f14180d0 0x55e6f1437050 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6a7fd46c32e5aab server_cookie=2fb0f1b754d779c3 in_seq=0 out_seq=0 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc6dffb640 10 client.7575 ms_handle_connect on v2:172.21.15.77:6836/82690693 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc6dffb640 1 -- 172.21.15.77:0/1649145867 <== mds.0 v2:172.21.15.77:6836/82690693 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55e6f133af40 con 0x55e6f14180d0 2024-03-06T07:26:27.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc6dffb640 10 client.7575 handle_command_reply: tid=0 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 client.7575 shutdown 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 -- 172.21.15.77:0/1649145867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc600 msgr2=0x55e6f1482cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc600 0x55e6f1482cf0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fdc9006b870 tx=0x7fdc78015730 comp rx=0 tx=0).stop 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 -- 172.21.15.77:0/1649145867 shutdown_connections 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.77:6836/82690693,v1:172.21.15.77:6837/82690693] conn(0x55e6f14180d0 0x55e6f1437050 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e6f13fc600 0x55e6f1482cf0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55e6f13fc9e0 0x55e6f1483230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc6dffb640 0 client.7575 ms_handle_reset on v2:172.21.15.77:6836/82690693 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 --2- 172.21.15.77:0/1649145867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55e6f13f9600 0x55e6f14145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 -- 172.21.15.77:0/1649145867 >> 172.21.15.77:0/1649145867 conn(0x55e6f13c9870 msgr2=0x55e6f13ecfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 -- 172.21.15.77:0/1649145867 shutdown_connections 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.988+0000 7fdc98281740 1 -- 172.21.15.77:0/1649145867 wait complete. 2024-03-06T07:26:27.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc98281740 20 client.7575 trim_cache size 0 max 16384 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 6985, 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 5.2248764980000004, 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 58, 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 96, 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 96, 2024-03-06T07:26:28.000 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 6.0970747100000002 2024-03-06T07:26:28.001 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:26:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 -- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7006b500 msgr2=0x7fdc7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7006b500 0x7fdc7006d9c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdc80000fd0 tx=0x7fdc80009040 comp rx=0 tx=0).stop 2024-03-06T07:26:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 -- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9015c520 msgr2=0x7fdc90160b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9015c520 0x7fdc90160b00 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fdc8802c060 tx=0x7fdc880014a0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 -- 172.21.15.77:0/2158489429 shutdown_connections 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7006b500 0x7fdc7006d9c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9015c520 0x7fdc90160b00 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc901600a0 0x7fdc9015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 --2- 172.21.15.77:0/2158489429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc7c005210 0x7fdc90153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 -- 172.21.15.77:0/2158489429 >> 172.21.15.77:0/2158489429 conn(0x7fdc9009d230 msgr2=0x7fdc9006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 -- 172.21.15.77:0/2158489429 shutdown_connections 2024-03-06T07:26:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:27.989+0000 7fdc9569a640 1 -- 172.21.15.77:0/2158489429 wait complete. 2024-03-06T07:26:28.003 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:26:28.003 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> subdir = "delete_me" 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> size_unit = 1024 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> file_multiplier = 200 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> for i in range(0, file_multiplier): 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> f.write(size * '"'"'x'"'"') 2024-03-06T07:26:28.004 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:26:30.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:30.477 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:34.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:34.698 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_ops 2024-03-06T07:26:34.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:34.699 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:26:34.840 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:26:34.841 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:26:34.841 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_files 2024-03-06T07:26:34.977 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:26:34.977 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:26:34.977 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.077+0000 7f1c42be9640 1 Processor -- start 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.077+0000 7f1c42be9640 1 -- start start 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c42be9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c3c15e150 0x7f1c3c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c42be9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c0cfba0 0x7f1c3c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c42be9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c3c0d0e90 0x7f1c3c0b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c42be9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3c0d2ab0 con 0x7f1c3c15e150 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c42be9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3c0d1ea0 con 0x7f1c3c0d0e90 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c42be9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0d2180 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c40985640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c3c15e150 0x7f1c3c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c40985640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c3c15e150 0x7f1c3c0cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59124/0 (socket says 172.21.15.77:59124) 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c40985640 1 -- 172.21.15.77:0/2121000043 learned_addr learned my addr 172.21.15.77:0/2121000043 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c41186640 1 --2- 172.21.15.77:0/2121000043 >> v2:172.21.15.158:3300/0 conn(0x7f1c3c0d0e90 0x7f1c3c0b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c3bfff640 1 --2- 172.21.15.77:0/2121000043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c0cfba0 0x7f1c3c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c40985640 1 -- 172.21.15.77:0/2121000043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c0cfba0 msgr2=0x7f1c3c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c40985640 1 --2- 172.21.15.77:0/2121000043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c0cfba0 0x7f1c3c0d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c40985640 1 -- 172.21.15.77:0/2121000043 >> v2:172.21.15.158:3300/0 conn(0x7f1c3c0d0e90 msgr2=0x7f1c3c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c40985640 1 --2- 172.21.15.77:0/2121000043 >> v2:172.21.15.158:3300/0 conn(0x7f1c3c0d0e90 0x7f1c3c0b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.078+0000 7f1c40985640 1 -- 172.21.15.77:0/2121000043 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c0d2460 con 0x7f1c3c15e150 2024-03-06T07:26:35.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.079+0000 7f1c40985640 1 --2- 172.21.15.77:0/2121000043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c15e150 0x7f1c3c0cf660 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c08dfc0 tx=0x7f1c3002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2bf1fa97892a25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.079+0000 7f1c3b7fe640 1 -- 172.21.15.77:0/2121000043 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3002e070 con 0x7f1c3c15e150 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.079+0000 7f1c3b7fe640 1 -- 172.21.15.77:0/2121000043 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c30037070 con 0x7f1c3c15e150 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.079+0000 7f1c3b7fe640 1 -- 172.21.15.77:0/2121000043 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30031a00 con 0x7f1c3c15e150 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 -- 172.21.15.77:0/2121000043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c15e150 msgr2=0x7f1c3c0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2121000043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c15e150 0x7f1c3c0cf660 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c08dfc0 tx=0x7f1c3002b880 comp rx=0 tx=0).stop 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 -- 172.21.15.77:0/2121000043 shutdown_connections 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2121000043 >> v2:172.21.15.158:3300/0 conn(0x7f1c3c0d0e90 0x7f1c3c0b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2121000043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c0cfba0 0x7f1c3c0d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2121000043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c15e150 0x7f1c3c0cf660 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 -- 172.21.15.77:0/2121000043 >> 172.21.15.77:0/2121000043 conn(0x7f1c3c09d230 msgr2=0x7f1c3c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 -- 172.21.15.77:0/2121000043 shutdown_connections 2024-03-06T07:26:35.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.080+0000 7f1c42be9640 1 -- 172.21.15.77:0/2121000043 wait complete. 2024-03-06T07:26:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.082+0000 7f1c42be9640 1 Processor -- start 2024-03-06T07:26:35.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.082+0000 7f1c42be9640 1 -- start start 2024-03-06T07:26:35.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c42be9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c0cfba0 0x7f1c3c1e9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c42be9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c3c0d0e90 0x7f1c3c0c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c42be9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c15e150 0x7f1c3c0ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c42be9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0d1b30 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c42be9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0d2f40 con 0x7f1c3c0d0e90 2024-03-06T07:26:35.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c42be9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c15e530 con 0x7f1c3c15e150 2024-03-06T07:26:35.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c40985640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c0cfba0 0x7f1c3c1e9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c40985640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c0cfba0 0x7f1c3c1e9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59138/0 (socket says 172.21.15.77:59138) 2024-03-06T07:26:35.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c40985640 1 -- 172.21.15.77:0/2862729711 learned_addr learned my addr 172.21.15.77:0/2862729711 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:35.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c3bfff640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c3c0d0e90 0x7f1c3c0c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c41186640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c15e150 0x7f1c3c0ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c40985640 1 -- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c15e150 msgr2=0x7f1c3c0ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c40985640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c15e150 0x7f1c3c0ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c40985640 1 -- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c3c0d0e90 msgr2=0x7f1c3c0c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c40985640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c3c0d0e90 0x7f1c3c0c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c40985640 1 -- 172.21.15.77:0/2862729711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c0c8490 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.083+0000 7f1c41186640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c15e150 0x7f1c3c0ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.084+0000 7f1c40985640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c0cfba0 0x7f1c3c1e9cc0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f1c30000c00 tx=0x7f1c300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.084+0000 7f1c397fa640 1 -- 172.21.15.77:0/2862729711 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30032670 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.084+0000 7f1c397fa640 1 -- 172.21.15.77:0/2862729711 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c30032810 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.084+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3c0c8770 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.084+0000 7f1c397fa640 1 -- 172.21.15.77:0/2862729711 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30032c00 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.084+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3c0c8d30 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.085+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3c069ee0 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.087+0000 7f1c397fa640 1 -- 172.21.15.77:0/2862729711 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c3004c060 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.087+0000 7f1c397fa640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c2006b4d0 0x7f1c2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.087+0000 7f1c397fa640 1 -- 172.21.15.77:0/2862729711 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1c300aba20 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.087+0000 7f1c3bfff640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c2006b4d0 0x7f1c2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.088+0000 7f1c3bfff640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c2006b4d0 0x7f1c2006d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c074df0 tx=0x7f1c28009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.091+0000 7f1c397fa640 1 -- 172.21.15.77:0/2862729711 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c30040070 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.366+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c3c0d4670 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.366+0000 7f1c397fa640 1 -- 172.21.15.77:0/2862729711 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f1c3007a910 con 0x7f1c3c0cfba0 2024-03-06T07:26:35.368 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:35.368 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:35.369 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c2006b4d0 msgr2=0x7f1c2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c2006b4d0 0x7f1c2006d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c074df0 tx=0x7f1c28009190 comp rx=0 tx=0).stop 2024-03-06T07:26:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c0cfba0 msgr2=0x7f1c3c1e9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c0cfba0 0x7f1c3c1e9cc0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f1c30000c00 tx=0x7f1c300046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 shutdown_connections 2024-03-06T07:26:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c3c15e150 0x7f1c3c0ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c3c0d0e90 0x7f1c3c0c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c2006b4d0 0x7f1c2006d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 --2- 172.21.15.77:0/2862729711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c3c0cfba0 0x7f1c3c1e9cc0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 >> 172.21.15.77:0/2862729711 conn(0x7f1c3c09d230 msgr2=0x7f1c3c15fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 shutdown_connections 2024-03-06T07:26:35.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:35.368+0000 7f1c42be9640 1 -- 172.21.15.77:0/2862729711 wait complete. 2024-03-06T07:26:35.381 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:35.506 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":37,"msgr_send_messages":32,"msgr_recv_bytes":114615,"msgr_send_bytes":4228018,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.013748375,"msgr_running_send_time":0.007246534,"msgr_running_recv_time":0.003574034,"msgr_running_fast_dispatch_time":0.002538245,"msgr_send_messages_queue_lat":{"avgcount":31,"sum":0.003158501,"avgtime":0.000101887},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":107664,"msgr_send_encrypted_bytes":6656},"AsyncMessenger::Worker-1":{"msgr_recv_messages":4581,"msgr_send_messages":10503,"msgr_recv_bytes":1584077,"msgr_send_bytes":6967902,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.440428456,"msgr_running_send_time":0.233328572,"msgr_running_recv_time":0.245331680,"msgr_running_fast_dispatch_time":0.002501643,"msgr_send_messages_queue_lat":{"avgcount":10502,"sum":0.108552196,"avgtime":0.000010336},"msgr_handle_ack_lat":{"avgcount":14976,"sum":0.002113147,"avgtime":0.000000141},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92672},"AsyncMessenger::Worker-2":{"msgr_recv_messages":28,"msgr_send_messages":30,"msgr_recv_bytes":18057,"msgr_send_bytes":3312187,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.008815075,"msgr_running_send_time":0.004963308,"msgr_running_recv_time":0.001754220,"msgr_running_fast_dispatch_time":0.001896122,"msgr_send_messages_queue_lat":{"avgcount":29,"sum":0.004896508,"avgtime":0.000168845},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":373,"complete_latency":{"avgcount":74,"sum":0.946830481,"avgtime":0.012795006}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000089119,"avgtime":0.000022279}},"mds":{"request":3294,"reply":3292,"reply_latency":{"avgcount":3292,"sum":3.671102424,"avgtime":0.001115158},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3293,"inodes_top":0,"inodes_bottom":3293,"inodes_pin_tail":0,"inodes_pinned":3293,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3300,"traverse_hit":3299,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3199,"root_rbytes":5008384,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":69,"ceph_cap_op_grant":3249,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1010,"handle_client_caps_dirty":940,"handle_client_cap_release":0,"process_request_cap_release":3365},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":82,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":82,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4296,"evex":0,"evtrm":0,"ev":4299,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":12698929,"rdpos":4195244,"jlat":{"avgcount":11,"sum":0.053436891,"avgtime":0.004857899},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3293,"dn+":3293,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":82336,"heap":248164},"mds_server":{"dispatch_client_request":3297,"dispatch_server_request":0,"handle_client_request":3294,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":82,"sum":0.114569956,"avgtime":0.001397194},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3141,"average_load":3141,"avg_session_uptime":806,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":378293,"buffer_anon_items":595,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":6686272,"mds_co_items":32875,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":64,"op_send_bytes":8746894,"op_resend":0,"op_reply":64,"op_latency":{"avgcount":64,"sum":0.130997131,"avgtime":0.002046830},"op_inflight":0,"oplen_avg":{"avgcount":64,"sum":119},"op":64,"op_r":36,"op_w":28,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":7,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":62697,"max":104857600,"get_started":0,"get":4642,"get_sum":1296304,"get_or_fail_fail":0,"get_or_fail_success":4642,"take":0,"take_sum":0,"put":4401,"put_sum":1233607,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":8748011,"put":64,"put_sum":8748011,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":64,"put":64,"put_sum":64,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":116017,"max":3758096384,"get_started":0,"get":4296,"get_sum":8503244,"get_or_fail_fail":0,"get_or_fail_success":4296,"take":0,"take_sum":0,"put":11,"put_sum":8387227,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:35.508 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 114615, "msgr_recv_encrypted_bytes": 107664, "msgr_recv_messages": 37, "msgr_running_fast_dispatch_time": 0.002538245, "msgr_running_recv_time": 0.003574034, "msgr_running_send_time": 0.007246534, "msgr_running_total_time": 0.013748375, "msgr_send_bytes": 4228018, "msgr_send_encrypted_bytes": 6656, "msgr_send_messages": 32, "msgr_send_messages_queue_lat": { "avgcount": 31, "avgtime": 0.000101887, "sum": 0.003158501 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 14976, "avgtime": 1.41e-07, "sum": 0.002113147 }, "msgr_recv_bytes": 1584077, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 4581, "msgr_running_fast_dispatch_time": 0.002501643, "msgr_running_recv_time": 0.24533168, "msgr_running_send_time": 0.233328572, "msgr_running_total_time": 0.440428456, "msgr_send_bytes": 6967902, "msgr_send_encrypted_bytes": 92672, "msgr_send_messages": 10503, "msgr_send_messages_queue_lat": { "avgcount": 10502, "avgtime": 1.0336e-05, "sum": 0.108552196 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 18057, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 28, "msgr_running_fast_dispatch_time": 0.001896122, "msgr_running_recv_time": 0.00175422, "msgr_running_send_time": 0.004963308, "msgr_running_total_time": 0.008815075, "msgr_send_bytes": 3312187, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 30, "msgr_send_messages_queue_lat": { "avgcount": 29, "avgtime": 0.000168845, "sum": 0.004896508 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.012795006, "sum": 0.946830481 }, "queue_len": 373 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.2279e-05, "sum": 8.9119e-05 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3249, "ceph_cap_op_revoke": 69, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 1010, "handle_client_caps_dirty": 940, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3293, "inodes_bottom": 3293, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3293, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3365, "q": 0, "reply": 3292, "reply_latency": { "avgcount": 3292, "avgtime": 0.001115158, "sum": 3.671102424 }, "request": 3294, "root_rbytes": 5008384, "root_rfiles": 3199, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3300, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3299, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 82, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 82, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 4299, "evadd": 4296, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.004857899, "sum": 0.053436891 }, "rdpos": 4195244, "replayed": 3, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 12698929 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3293, "dn+": 3293, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 82336 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3297, "dispatch_server_request": 0, "handle_client_request": 3294, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 82, "avgtime": 0.001397194, "sum": 0.114569956 } }, "mds_sessions": { "average_load": 3141, "avg_session_uptime": 806, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3141 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 378293, "buffer_anon_items": 595, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 6686272, "mds_co_items": 32875, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 15, "op": 64, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 64, "avgtime": 0.00204683, "sum": 0.130997131 }, "op_pg": 0, "op_r": 36, "op_reply": 64, "op_resend": 0, "op_rmw": 0, "op_send": 64, "op_send_bytes": 8746894, "op_w": 28, "oplen_avg": { "avgcount": 64, "sum": 119 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 7, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 4642, "get_or_fail_fail": 0, "get_or_fail_success": 4642, "get_started": 0, "get_sum": 1296304, "max": 104857600, "put": 4401, "put_sum": 1233607, "take": 0, "take_sum": 0, "val": 62697, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 64, "put_sum": 8748011, "take": 64, "take_sum": 8748011, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 64, "put_sum": 64, "take": 64, "take_sum": 64, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 4296, "get_or_fail_fail": 0, "get_or_fail_success": 4296, "get_started": 0, "get_sum": 8503244, "max": 3758096384, "put": 11, "put_sum": 8387227, "take": 0, "take_sum": 0, "val": 116017, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:35.508 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/82, 0/82 2024-03-06T07:26:36.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.610+0000 7efce19eb640 1 Processor -- start 2024-03-06T07:26:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.610+0000 7efce19eb640 1 -- start start 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.611+0000 7efce19eb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efcdc15e150 0x7efcdc0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.611+0000 7efce19eb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc0d1ea0 con 0x7efcdc0cfba0 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.611+0000 7efce19eb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc0d2180 con 0x7efcdc0d1060 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.611+0000 7efce19eb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc0d2ab0 con 0x7efcdc15e150 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.611+0000 7efcdaffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efcdc15e150 0x7efcdc0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.611+0000 7efcda7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efcdc0cfba0 0x7efcdc0d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46924/0 (socket says 172.21.15.77:46924) 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.611+0000 7efcdaffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efcdc15e150 0x7efcdc0cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59144/0 (socket says 172.21.15.77:59144) 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.611+0000 7efcda7fc640 1 -- 172.21.15.77:0/2524149023 learned_addr learned my addr 172.21.15.77:0/2524149023 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.612+0000 7efcdaffd640 1 -- 172.21.15.77:0/2524149023 >> v1:172.21.15.158:6789/0 conn(0x7efcdc0d1060 legacy=0x7efcdc0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.612+0000 7efcdaffd640 1 -- 172.21.15.77:0/2524149023 >> v1:172.21.15.77:6789/0 conn(0x7efcdc0cfba0 legacy=0x7efcdc0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.612+0000 7efcd9ffb640 1 -- 172.21.15.77:0/2524149023 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144400364 0 0) 0x7efcdc0d1ea0 con 0x7efcdc0cfba0 2024-03-06T07:26:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.612+0000 7efcdaffd640 1 -- 172.21.15.77:0/2524149023 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0d2460 con 0x7efcdc15e150 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.613+0000 7efcdaffd640 1 --2- 172.21.15.77:0/2524149023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc15e150 0x7efcdc0cf660 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7efccc0097a0 tx=0x7efccc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9695e8da3db1c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.613+0000 7efcd9ffb640 1 -- 172.21.15.77:0/2524149023 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc0315a0 con 0x7efcdc15e150 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.613+0000 7efcd9ffb640 1 -- 172.21.15.77:0/2524149023 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc037070 con 0x7efcdc15e150 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.613+0000 7efcd9ffb640 1 -- 172.21.15.77:0/2524149023 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc031960 con 0x7efcdc15e150 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.614+0000 7efce19eb640 1 -- 172.21.15.77:0/2524149023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc15e150 msgr2=0x7efcdc0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.614+0000 7efce19eb640 1 --2- 172.21.15.77:0/2524149023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc15e150 0x7efcdc0cf660 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7efccc0097a0 tx=0x7efccc02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.614+0000 7efce19eb640 1 -- 172.21.15.77:0/2524149023 shutdown_connections 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.614+0000 7efce19eb640 1 --2- 172.21.15.77:0/2524149023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc15e150 0x7efcdc0cf660 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.614+0000 7efce19eb640 1 -- 172.21.15.77:0/2524149023 >> 172.21.15.77:0/2524149023 conn(0x7efcdc09d230 msgr2=0x7efcdc0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.614+0000 7efce19eb640 1 -- 172.21.15.77:0/2524149023 shutdown_connections 2024-03-06T07:26:36.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.614+0000 7efce19eb640 1 -- 172.21.15.77:0/2524149023 wait complete. 2024-03-06T07:26:36.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.615+0000 7efce19eb640 1 Processor -- start 2024-03-06T07:26:36.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.616+0000 7efce19eb640 1 -- start start 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efce19eb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcdc0cfba0 0x7efcdc157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efce19eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc0d1060 0x7efcdc0ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efce19eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcdc15e150 0x7efcdc0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efce19eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0b7dc0 con 0x7efcdc0d1060 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efce19eb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0d2f40 con 0x7efcdc0cfba0 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efce19eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0d1b30 con 0x7efcdc15e150 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcda7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc0d1060 0x7efcdc0ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcda7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc0d1060 0x7efcdc0ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59160/0 (socket says 172.21.15.77:59160) 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcda7fc640 1 -- 172.21.15.77:0/1296505810 learned_addr learned my addr 172.21.15.77:0/1296505810 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcdaffd640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcdc0cfba0 0x7efcdc157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcdb7fe640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcdc15e150 0x7efcdc0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcda7fc640 1 -- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcdc15e150 msgr2=0x7efcdc0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcda7fc640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcdc15e150 0x7efcdc0c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcda7fc640 1 -- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcdc0cfba0 msgr2=0x7efcdc157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcda7fc640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcdc0cfba0 0x7efcdc157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcda7fc640 1 -- 172.21.15.77:0/1296505810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0cd460 con 0x7efcdc0d1060 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.617+0000 7efcdaffd640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcdc0cfba0 0x7efcdc157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.618+0000 7efcda7fc640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc0d1060 0x7efcdc0ccf20 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7efcd000c500 tx=0x7efcd000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.618+0000 7efcc3fff640 1 -- 172.21.15.77:0/1296505810 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0018070 con 0x7efcdc0d1060 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.618+0000 7efcc3fff640 1 -- 172.21.15.77:0/1296505810 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd0007580 con 0x7efcdc0d1060 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.618+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc1608c0 con 0x7efcdc0d1060 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.618+0000 7efcc3fff640 1 -- 172.21.15.77:0/1296505810 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0004a20 con 0x7efcdc0d1060 2024-03-06T07:26:36.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.618+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc160e80 con 0x7efcdc0d1060 2024-03-06T07:26:36.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.619+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc069ee0 con 0x7efcdc0d1060 2024-03-06T07:26:36.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.620+0000 7efcc3fff640 1 -- 172.21.15.77:0/1296505810 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcd001b070 con 0x7efcdc0d1060 2024-03-06T07:26:36.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.620+0000 7efcc3fff640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcc806b4d0 0x7efcc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.620+0000 7efcc3fff640 1 -- 172.21.15.77:0/1296505810 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7efcd00895c0 con 0x7efcdc0d1060 2024-03-06T07:26:36.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.620+0000 7efcdaffd640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcc806b4d0 0x7efcc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.621+0000 7efcdaffd640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcc806b4d0 0x7efcc806d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efccc036000 tx=0x7efccc03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.627+0000 7efcc3fff640 1 -- 172.21.15.77:0/1296505810 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcd00583d0 con 0x7efcdc0d1060 2024-03-06T07:26:36.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.961+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcdc0d4670 con 0x7efcdc0d1060 2024-03-06T07:26:36.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.962+0000 7efcc3fff640 1 -- 172.21.15.77:0/1296505810 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7efcd0057cd0 con 0x7efcdc0d1060 2024-03-06T07:26:36.964 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:36.964 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:36.964 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:36.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.964+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcc806b4d0 msgr2=0x7efcc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.964+0000 7efce19eb640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcc806b4d0 0x7efcc806d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efccc036000 tx=0x7efccc03d040 comp rx=0 tx=0).stop 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.964+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc0d1060 msgr2=0x7efcdc0ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.964+0000 7efce19eb640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc0d1060 0x7efcdc0ccf20 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7efcd000c500 tx=0x7efcd000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.965+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 shutdown_connections 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.965+0000 7efce19eb640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcc806b4d0 0x7efcc806d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.965+0000 7efce19eb640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcdc0d1060 0x7efcdc0ccf20 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.965+0000 7efce19eb640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcdc15e150 0x7efcdc0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.965+0000 7efce19eb640 1 --2- 172.21.15.77:0/1296505810 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcdc0cfba0 0x7efcdc157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.965+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 >> 172.21.15.77:0/1296505810 conn(0x7efcdc09d230 msgr2=0x7efcdc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.965+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 shutdown_connections 2024-03-06T07:26:36.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:36.965+0000 7efce19eb640 1 -- 172.21.15.77:0/1296505810 wait complete. 2024-03-06T07:26:36.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:37.129 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":37,"msgr_send_messages":32,"msgr_recv_bytes":114615,"msgr_send_bytes":4228018,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.013748375,"msgr_running_send_time":0.007246534,"msgr_running_recv_time":0.003574034,"msgr_running_fast_dispatch_time":0.002538245,"msgr_send_messages_queue_lat":{"avgcount":31,"sum":0.003158501,"avgtime":0.000101887},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":107664,"msgr_send_encrypted_bytes":6656},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6015,"msgr_send_messages":12109,"msgr_recv_bytes":2108762,"msgr_send_bytes":11424040,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.534446277,"msgr_running_send_time":0.283907227,"msgr_running_recv_time":0.300564701,"msgr_running_fast_dispatch_time":0.004189769,"msgr_send_messages_queue_lat":{"avgcount":12108,"sum":0.135477310,"avgtime":0.000011189},"msgr_handle_ack_lat":{"avgcount":17945,"sum":0.002504257,"avgtime":0.000000139},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92672},"AsyncMessenger::Worker-2":{"msgr_recv_messages":28,"msgr_send_messages":30,"msgr_recv_bytes":18057,"msgr_send_bytes":3312187,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.008815075,"msgr_running_send_time":0.004963308,"msgr_running_recv_time":0.001754220,"msgr_running_fast_dispatch_time":0.001896122,"msgr_send_messages_queue_lat":{"avgcount":29,"sum":0.004896508,"avgtime":0.000168845},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1593,"complete_latency":{"avgcount":78,"sum":1.279445854,"avgtime":0.016403151}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000089119,"avgtime":0.000022279}},"mds":{"request":3917,"reply":3915,"reply_latency":{"avgcount":3915,"sum":5.128998325,"avgtime":0.001310088},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3816,"inodes_top":0,"inodes_bottom":3816,"inodes_pin_tail":0,"inodes_pinned":3816,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3923,"traverse_hit":3922,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3199,"root_rbytes":5008384,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":422,"ceph_cap_op_grant":3344,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2057,"handle_client_caps_dirty":1928,"handle_client_cap_release":0,"process_request_cap_release":4611},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":705,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":705,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":5968,"evex":0,"evtrm":0,"ev":5971,"evexg":0,"evexd":0,"segadd":7,"segex":0,"segtrm":0,"seg":9,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":16728455,"rdpos":4195244,"jlat":{"avgcount":12,"sum":0.071739575,"avgtime":0.005978297},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3916,"dn+":3916,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":99232,"heap":248164},"mds_server":{"dispatch_client_request":3920,"dispatch_server_request":0,"handle_client_request":3917,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":705,"sum":1.572465857,"avgtime":0.002230448},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3692,"average_load":3692,"avg_session_uptime":811,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":23696541,"buffer_anon_items":6420,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14704495,"mds_co_items":106170,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":67,"op_send_bytes":12517793,"op_resend":0,"op_reply":67,"op_latency":{"avgcount":67,"sum":0.153996628,"avgtime":0.002298457},"op_inflight":0,"oplen_avg":{"avgcount":67,"sum":125},"op":67,"op_r":36,"op_w":31,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":12,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":7,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":18,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":297,"max":104857600,"get_started":0,"get":6077,"get_sum":1692226,"get_or_fail_fail":0,"get_or_fail_success":6077,"take":0,"take_sum":0,"put":6076,"put_sum":1691929,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":67,"take_sum":12518910,"put":67,"put_sum":12518910,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":67,"take_sum":67,"put":67,"put_sum":67,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":400119,"max":3758096384,"get_started":0,"get":5968,"get_sum":12532770,"get_or_fail_fail":0,"get_or_fail_success":5968,"take":0,"take_sum":0,"put":12,"put_sum":12132651,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:37.131 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 114615, "msgr_recv_encrypted_bytes": 107664, "msgr_recv_messages": 37, "msgr_running_fast_dispatch_time": 0.002538245, "msgr_running_recv_time": 0.003574034, "msgr_running_send_time": 0.007246534, "msgr_running_total_time": 0.013748375, "msgr_send_bytes": 4228018, "msgr_send_encrypted_bytes": 6656, "msgr_send_messages": 32, "msgr_send_messages_queue_lat": { "avgcount": 31, "avgtime": 0.000101887, "sum": 0.003158501 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 17945, "avgtime": 1.39e-07, "sum": 0.002504257 }, "msgr_recv_bytes": 2108762, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 6015, "msgr_running_fast_dispatch_time": 0.004189769, "msgr_running_recv_time": 0.300564701, "msgr_running_send_time": 0.283907227, "msgr_running_total_time": 0.534446277, "msgr_send_bytes": 11424040, "msgr_send_encrypted_bytes": 92672, "msgr_send_messages": 12109, "msgr_send_messages_queue_lat": { "avgcount": 12108, "avgtime": 1.1189e-05, "sum": 0.13547731 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 18057, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 28, "msgr_running_fast_dispatch_time": 0.001896122, "msgr_running_recv_time": 0.00175422, "msgr_running_send_time": 0.004963308, "msgr_running_total_time": 0.008815075, "msgr_send_bytes": 3312187, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 30, "msgr_send_messages_queue_lat": { "avgcount": 29, "avgtime": 0.000168845, "sum": 0.004896508 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 78, "avgtime": 0.016403151, "sum": 1.279445854 }, "queue_len": 1593 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.2279e-05, "sum": 8.9119e-05 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3344, "ceph_cap_op_revoke": 422, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2057, "handle_client_caps_dirty": 1928, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3816, "inodes_bottom": 3816, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3816, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 4611, "q": 0, "reply": 3915, "reply_latency": { "avgcount": 3915, "avgtime": 0.001310088, "sum": 5.128998325 }, "request": 3917, "root_rbytes": 5008384, "root_rfiles": 3199, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3923, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3922, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 705, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 705, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 5971, "evadd": 5968, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.005978297, "sum": 0.071739575 }, "rdpos": 4195244, "replayed": 3, "seg": 9, "segadd": 7, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 16728455 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3916, "dn+": 3916, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 99232 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3920, "dispatch_server_request": 0, "handle_client_request": 3917, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 705, "avgtime": 0.002230448, "sum": 1.572465857 } }, "mds_sessions": { "average_load": 3692, "avg_session_uptime": 811, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3692 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 23696541, "buffer_anon_items": 6420, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 14704495, "mds_co_items": 106170, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 18, "op": 67, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 67, "avgtime": 0.002298457, "sum": 0.153996628 }, "op_pg": 0, "op_r": 36, "op_reply": 67, "op_resend": 0, "op_rmw": 0, "op_send": 67, "op_send_bytes": 12517793, "op_w": 31, "oplen_avg": { "avgcount": 67, "sum": 125 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 7, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 12, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6077, "get_or_fail_fail": 0, "get_or_fail_success": 6077, "get_started": 0, "get_sum": 1692226, "max": 104857600, "put": 6076, "put_sum": 1691929, "take": 0, "take_sum": 0, "val": 297, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 67, "put_sum": 12518910, "take": 67, "take_sum": 12518910, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 67, "put_sum": 67, "take": 67, "take_sum": 67, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 5968, "get_or_fail_fail": 0, "get_or_fail_success": 5968, "get_started": 0, "get_sum": 12532770, "max": 3758096384, "put": 12, "put_sum": 12132651, "take": 0, "take_sum": 0, "val": 400119, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:37.131 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/705, 0/705 2024-03-06T07:26:38.132 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:38.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.253+0000 7f1a6428f640 1 Processor -- start 2024-03-06T07:26:38.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.253+0000 7f1a6428f640 1 -- start start 2024-03-06T07:26:38.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a6428f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1a5c0d0db0 0x7f1a5c0b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a6428f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0d1ea0 con 0x7f1a5c0d0950 2024-03-06T07:26:38.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a6428f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0d2180 con 0x7f1a5c15e150 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a6428f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d2ab0 con 0x7f1a5c0d0db0 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a6182a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1a5c0d0950 0x7f1a5c0b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46940/0 (socket says 172.21.15.77:46940) 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a6182a640 1 -- 172.21.15.77:0/2371710592 learned_addr learned my addr 172.21.15.77:0/2371710592 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240099363 0 0) 0x7f1a5c0d1ea0 con 0x7f1a5c0d0950 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a6282c640 1 --2- 172.21.15.77:0/2371710592 >> v2:172.21.15.158:3300/0 conn(0x7f1a5c0d0db0 0x7f1a5c0b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a48003520 con 0x7f1a5c0d0950 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3163393243 0 0) 0x7f1a48003520 con 0x7f1a5c0d0950 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 >> v2:172.21.15.158:3300/0 conn(0x7f1a5c0d0db0 msgr2=0x7f1a5c0b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:38.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 --2- 172.21.15.77:0/2371710592 >> v2:172.21.15.158:3300/0 conn(0x7f1a5c0d0db0 0x7f1a5c0b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 >> v1:172.21.15.158:6789/0 conn(0x7f1a5c15e150 legacy=0x7f1a5c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0d2460 con 0x7f1a5c0d0950 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f1a4c001670 con 0x7f1a5c0d0950 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 >> v1:172.21.15.77:6789/0 conn(0x7f1a5c0d0950 legacy=0x7f1a5c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a48003a00 0x7f1a48003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.254+0000 7f1a61029640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a61029640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a48005210 0x7f1a4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d2ab0 con 0x7f1a48003a00 2024-03-06T07:26:38.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4800c160 con 0x7f1a48005210 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4800c360 con 0x7f1a48004800 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6202b640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a48003a00 0x7f1a48003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6282c640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a48005210 0x7f1a4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6182a640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6282c640 1 -- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48004800 msgr2=0x7f1a48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6282c640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6282c640 1 -- 172.21.15.77:0/2371710592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a48003a00 msgr2=0x7f1a48003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6282c640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a48003a00 0x7f1a48003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6282c640 1 -- 172.21.15.77:0/2371710592 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a48003520 con 0x7f1a48005210 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.255+0000 7f1a6202b640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a48003a00 0x7f1a48003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.256+0000 7f1a6282c640 1 -- 172.21.15.77:0/2371710592 reap_dead start 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.256+0000 7f1a6282c640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a48005210 0x7f1a4800b9c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009f90 tx=0x7f1a48005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abc23b2b0a6ed8cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.256+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a58001370 con 0x7f1a48005210 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.256+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5402cbf0 con 0x7f1a48005210 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.256+0000 7f1a61029640 1 -- 172.21.15.77:0/2371710592 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54009c10 con 0x7f1a48005210 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.257+0000 7f1a6428f640 1 -- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a48005210 msgr2=0x7f1a4800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.257+0000 7f1a6428f640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a48005210 0x7f1a4800b9c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009f90 tx=0x7f1a48005830 comp rx=0 tx=0).stop 2024-03-06T07:26:38.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.257+0000 7f1a6428f640 1 -- 172.21.15.77:0/2371710592 shutdown_connections 2024-03-06T07:26:38.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.257+0000 7f1a6428f640 1 --2- 172.21.15.77:0/2371710592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a48005210 0x7f1a4800b9c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.257+0000 7f1a6428f640 1 -- 172.21.15.77:0/2371710592 >> 172.21.15.77:0/2371710592 conn(0x7f1a5c09d230 msgr2=0x7f1a5c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:38.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.257+0000 7f1a6428f640 1 -- 172.21.15.77:0/2371710592 shutdown_connections 2024-03-06T07:26:38.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.257+0000 7f1a6428f640 1 -- 172.21.15.77:0/2371710592 wait complete. 2024-03-06T07:26:38.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.258+0000 7f1a6428f640 1 Processor -- start 2024-03-06T07:26:38.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.258+0000 7f1a6428f640 1 -- start start 2024-03-06T07:26:38.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6428f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48005210 0x7f1a5c0b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6428f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a5c152d70 0x7f1a5c0b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6428f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a5c0b3c10 0x7f1a5c0b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6428f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d2f40 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6428f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d1220 con 0x7f1a5c152d70 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6428f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d1b30 con 0x7f1a48005210 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6282c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a5c0b3c10 0x7f1a5c0b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6282c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a5c0b3c10 0x7f1a5c0b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59188/0 (socket says 172.21.15.77:59188) 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6282c640 1 -- 172.21.15.77:0/3671927428 learned_addr learned my addr 172.21.15.77:0/3671927428 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6182a640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a5c152d70 0x7f1a5c0b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.259+0000 7f1a6202b640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48005210 0x7f1a5c0b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6282c640 1 -- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48005210 msgr2=0x7f1a5c0b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6282c640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48005210 0x7f1a5c0b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6282c640 1 -- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a5c152d70 msgr2=0x7f1a5c0b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6282c640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a5c152d70 0x7f1a5c0b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6282c640 1 -- 172.21.15.77:0/3671927428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0b3420 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6202b640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48005210 0x7f1a5c0b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6182a640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a5c152d70 0x7f1a5c0b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6282c640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a5c0b3c10 0x7f1a5c0b4090 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402c660 tx=0x7f1a54009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a52ffd640 1 -- 172.21.15.77:0/3671927428 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54004830 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a52ffd640 1 -- 172.21.15.77:0/3671927428 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a54044070 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c0b1e40 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a52ffd640 1 -- 172.21.15.77:0/3671927428 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54003e10 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.260+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c0b2400 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.261+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c069ee0 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.262+0000 7f1a52ffd640 1 -- 172.21.15.77:0/3671927428 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a54005d10 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.263+0000 7f1a52ffd640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1a4006b4d0 0x7f1a4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:38.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.263+0000 7f1a52ffd640 1 -- 172.21.15.77:0/3671927428 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1a540ac9b0 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.263+0000 7f1a6202b640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1a4006b4d0 0x7f1a4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:38.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.263+0000 7f1a6202b640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1a4006b4d0 0x7f1a4006d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1a58001b50 tx=0x7f1a58009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:38.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.270+0000 7f1a52ffd640 1 -- 172.21.15.77:0/3671927428 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a5407b040 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.632+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5c0d0200 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.632+0000 7f1a52ffd640 1 -- 172.21.15.77:0/3671927428 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f1a5407f0e0 con 0x7f1a5c0b3c10 2024-03-06T07:26:38.634 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:38.634 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:38.635 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:38.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1a4006b4d0 msgr2=0x7f1a4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1a4006b4d0 0x7f1a4006d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1a58001b50 tx=0x7f1a58009040 comp rx=0 tx=0).stop 2024-03-06T07:26:38.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a5c0b3c10 msgr2=0x7f1a5c0b4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:38.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a5c0b3c10 0x7f1a5c0b4090 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402c660 tx=0x7f1a54009750 comp rx=0 tx=0).stop 2024-03-06T07:26:38.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 shutdown_connections 2024-03-06T07:26:38.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1a4006b4d0 0x7f1a4006d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a5c0b3c10 0x7f1a5c0b4090 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1a5c152d70 0x7f1a5c0b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 --2- 172.21.15.77:0/3671927428 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1a48005210 0x7f1a5c0b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:38.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 >> 172.21.15.77:0/3671927428 conn(0x7f1a5c09d230 msgr2=0x7f1a5c0cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:38.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 shutdown_connections 2024-03-06T07:26:38.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:38.635+0000 7f1a6428f640 1 -- 172.21.15.77:0/3671927428 wait complete. 2024-03-06T07:26:38.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:38.797 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":65,"msgr_send_messages":60,"msgr_recv_bytes":121737,"msgr_send_bytes":4237436,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.016644107,"msgr_running_send_time":0.008943918,"msgr_running_recv_time":0.004121505,"msgr_running_fast_dispatch_time":0.003085818,"msgr_send_messages_queue_lat":{"avgcount":59,"sum":0.004064259,"avgtime":0.000068885},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":107936,"msgr_send_encrypted_bytes":7280},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7452,"msgr_send_messages":15755,"msgr_recv_bytes":2640290,"msgr_send_bytes":15551287,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.672627145,"msgr_running_send_time":0.366624004,"msgr_running_recv_time":0.358912712,"msgr_running_fast_dispatch_time":0.006916794,"msgr_send_messages_queue_lat":{"avgcount":15754,"sum":0.192669545,"avgtime":0.000012229},"msgr_handle_ack_lat":{"avgcount":22141,"sum":0.003243741,"avgtime":0.000000146},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94464},"AsyncMessenger::Worker-2":{"msgr_recv_messages":71,"msgr_send_messages":73,"msgr_recv_bytes":28947,"msgr_send_bytes":3326353,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.011747563,"msgr_running_send_time":0.005976782,"msgr_running_recv_time":0.002427887,"msgr_running_fast_dispatch_time":0.002854955,"msgr_send_messages_queue_lat":{"avgcount":72,"sum":0.006931643,"avgtime":0.000096272},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1195,"complete_latency":{"avgcount":81,"sum":2.687661514,"avgtime":0.033181006}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":131,"sum":0.068795484,"avgtime":0.000525156}},"mds":{"request":4439,"reply":4437,"reply_latency":{"avgcount":4437,"sum":6.465792739,"avgtime":0.001457244},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4382,"inodes_top":0,"inodes_bottom":4382,"inodes_pin_tail":0,"inodes_pinned":4382,"inodes_expired":0,"inodes_with_caps":3021,"caps":3021,"subtrees":2,"traverse":4445,"traverse_hit":4444,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2866,"root_rbytes":8290304,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":582,"ceph_cap_op_grant":4477,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2939,"handle_client_caps_dirty":2749,"handle_client_cap_release":2,"process_request_cap_release":5655},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1146,"num_strays_delayed":100,"num_strays_enqueuing":0,"strays_created":1227,"strays_enqueued":81,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7456,"evex":0,"evtrm":0,"ev":7459,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":20154462,"rdpos":4195244,"jlat":{"avgcount":14,"sum":1.259774324,"avgtime":0.089983880},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4438,"dn+":4438,"dn-":0,"cap":3021,"cap+":3202,"cap-":181,"rss":103072,"heap":248164},"mds_server":{"dispatch_client_request":4442,"dispatch_server_request":0,"handle_client_request":4439,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1227,"sum":2.909260271,"avgtime":0.002371035},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4139,"average_load":4139,"avg_session_uptime":811,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":23696541,"buffer_anon_items":6420,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14704495,"mds_co_items":106170,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":166,"op_send_bytes":15426649,"op_resend":0,"op_reply":166,"op_latency":{"avgcount":166,"sum":0.331992726,"avgtime":0.001999956},"op_inflight":0,"oplen_avg":{"avgcount":166,"sum":226},"op":166,"op_r":39,"op_w":127,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":11,"osdop_write":17,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":95,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":20,"omap_rd":48,"omap_del":5},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":81,"pq_executed":81,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":296,"max":104857600,"get_started":0,"get":7585,"get_sum":2105993,"get_or_fail_fail":0,"get_or_fail_success":7585,"take":0,"take_sum":0,"put":7584,"put_sum":2105697,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":6829,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":167,"take_sum":15442776,"put":166,"put_sum":15435947,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":1,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":167,"take_sum":167,"put":166,"put_sum":166,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":81,"get_sum":8181,"get_or_fail_fail":0,"get_or_fail_success":81,"take":0,"take_sum":0,"put":3,"put_sum":8181,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":933372,"max":3758096384,"get_started":0,"get":7456,"get_sum":15958777,"get_or_fail_fail":0,"get_or_fail_success":7456,"take":0,"take_sum":0,"put":14,"put_sum":15025405,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:38.799 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 121737, "msgr_recv_encrypted_bytes": 107936, "msgr_recv_messages": 65, "msgr_running_fast_dispatch_time": 0.003085818, "msgr_running_recv_time": 0.004121505, "msgr_running_send_time": 0.008943918, "msgr_running_total_time": 0.016644107, "msgr_send_bytes": 4237436, "msgr_send_encrypted_bytes": 7280, "msgr_send_messages": 60, "msgr_send_messages_queue_lat": { "avgcount": 59, "avgtime": 6.8885e-05, "sum": 0.004064259 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 22141, "avgtime": 1.46e-07, "sum": 0.003243741 }, "msgr_recv_bytes": 2640290, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 7452, "msgr_running_fast_dispatch_time": 0.006916794, "msgr_running_recv_time": 0.358912712, "msgr_running_send_time": 0.366624004, "msgr_running_total_time": 0.672627145, "msgr_send_bytes": 15551287, "msgr_send_encrypted_bytes": 94464, "msgr_send_messages": 15755, "msgr_send_messages_queue_lat": { "avgcount": 15754, "avgtime": 1.2229e-05, "sum": 0.192669545 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 28947, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 71, "msgr_running_fast_dispatch_time": 0.002854955, "msgr_running_recv_time": 0.002427887, "msgr_running_send_time": 0.005976782, "msgr_running_total_time": 0.011747563, "msgr_send_bytes": 3326353, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 73, "msgr_send_messages_queue_lat": { "avgcount": 72, "avgtime": 9.6272e-05, "sum": 0.006931643 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 81, "avgtime": 0.033181006, "sum": 2.687661514 }, "queue_len": 1195 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 131, "avgtime": 0.000525156, "sum": 0.068795484 }, "queue_len": 0 }, "mds": { "caps": 3021, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4477, "ceph_cap_op_revoke": 582, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 2, "handle_client_caps": 2939, "handle_client_caps_dirty": 2749, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4382, "inodes_bottom": 4382, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4382, "inodes_top": 0, "inodes_with_caps": 3021, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 5655, "q": 0, "reply": 4437, "reply_latency": { "avgcount": 4437, "avgtime": 0.001457244, "sum": 6.465792739 }, "request": 4439, "root_rbytes": 8290304, "root_rfiles": 2866, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4445, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4444, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1146, "num_strays_delayed": 100, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1227, "strays_enqueued": 81, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7459, "evadd": 7456, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 14, "avgtime": 0.08998388, "sum": 1.259774324 }, "rdpos": 4195244, "replayed": 3, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 20154462 }, "mds_mem": { "cap": 3021, "cap+": 3202, "cap-": 181, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4438, "dn+": 4438, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 103072 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4442, "dispatch_server_request": 0, "handle_client_request": 4439, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1227, "avgtime": 0.002371035, "sum": 2.909260271 } }, "mds_sessions": { "average_load": 4139, "avg_session_uptime": 811, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4139 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 23696541, "buffer_anon_items": 6420, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 14704495, "mds_co_items": 106170, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 5, "omap_rd": 48, "omap_wr": 20, "op": 166, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 166, "avgtime": 0.001999956, "sum": 0.331992726 }, "op_pg": 0, "op_r": 39, "op_reply": 166, "op_resend": 0, "op_rmw": 0, "op_send": 166, "op_send_bytes": 15426649, "op_w": 127, "oplen_avg": { "avgcount": 166, "sum": 226 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 95, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 11, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 17, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 81, "pq_executed_ops": 81, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7585, "get_or_fail_fail": 0, "get_or_fail_success": 7585, "get_started": 0, "get_sum": 2105993, "max": 104857600, "put": 7584, "put_sum": 2105697, "take": 0, "take_sum": 0, "val": 296, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 166, "put_sum": 15435947, "take": 167, "take_sum": 15442776, "val": 6829, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 166, "put_sum": 166, "take": 167, "take_sum": 167, "val": 1, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 81, "get_or_fail_fail": 0, "get_or_fail_success": 81, "get_started": 0, "get_sum": 8181, "max": 3758096384, "put": 3, "put_sum": 8181, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 7456, "get_or_fail_fail": 0, "get_or_fail_success": 7456, "get_started": 0, "get_sum": 15958777, "max": 3758096384, "put": 14, "put_sum": 15025405, "take": 0, "take_sum": 0, "val": 933372, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:38.799 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1146, 81/1227 2024-03-06T07:26:39.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.923+0000 7f6486829640 1 Processor -- start 2024-03-06T07:26:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.923+0000 7f6486829640 1 -- start start 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.924+0000 7f6486829640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64800d0db0 0x7f64800b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.924+0000 7f6486829640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64800d1ea0 con 0x7f648015e150 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.924+0000 7f6486829640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64800d2180 con 0x7f64800d0950 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.924+0000 7f6486829640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64800d2ab0 con 0x7f64800d0db0 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.924+0000 7f647ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f648015e150 0x7f64800cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51454/0 (socket says 172.21.15.77:51454) 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.924+0000 7f647ffff640 1 -- 172.21.15.77:0/2681340719 learned_addr learned my addr 172.21.15.77:0/2681340719 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052791344 0 0) 0x7f64800d1ea0 con 0x7f648015e150 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6468003520 con 0x7f648015e150 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f6484dc6640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64800d0db0 0x7f64800b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774274427 0 0) 0x7f6468003520 con 0x7f648015e150 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64800d0db0 msgr2=0x7f64800b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:39.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64800d0db0 0x7f64800b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 >> v1:172.21.15.158:6789/0 conn(0x7f64800d0950 legacy=0x7f64800b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64800d2460 con 0x7f648015e150 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f6470001670 con 0x7f648015e150 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 >> v1:172.21.15.77:6789/0 conn(0x7f648015e150 legacy=0x7f64800cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64800d0db0 0x7f64680039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6468003f60 0x7f6468004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6468004d40 0x7f64680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64800d2ab0 con 0x7f6468004d40 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646800bb50 con 0x7f6468003f60 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646800bd50 con 0x7f64800d0db0 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f6484dc6640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6468004d40 0x7f64680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647ffff640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64800d0db0 0x7f64680039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.925+0000 7f647f7fe640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6468003f60 0x7f6468004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.926+0000 7f6484dc6640 1 -- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64800d0db0 msgr2=0x7f64680039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:39.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.926+0000 7f6484dc6640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64800d0db0 0x7f64680039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.926+0000 7f6484dc6640 1 -- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6468003f60 msgr2=0x7f6468004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:39.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.926+0000 7f6484dc6640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6468003f60 0x7f6468004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.926+0000 7f6484dc6640 1 -- 172.21.15.77:0/2681340719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6468003520 con 0x7f6468004d40 2024-03-06T07:26:39.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.926+0000 7f647f7fe640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6468003f60 0x7f6468004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:39.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.927+0000 7f6484dc6640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6468004d40 0x7f64680051c0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6478009fc0 tx=0x7f6468005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e960320db2bde6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:39.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.927+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478039070 con 0x7f6468004d40 2024-03-06T07:26:39.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.927+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647802cc20 con 0x7f6468004d40 2024-03-06T07:26:39.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.927+0000 7f647effd640 1 -- 172.21.15.77:0/2681340719 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478009c10 con 0x7f6468004d40 2024-03-06T07:26:39.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.928+0000 7f6486829640 1 -- 172.21.15.77:0/2681340719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6468004d40 msgr2=0x7f64680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.928+0000 7f6486829640 1 --2- 172.21.15.77:0/2681340719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6468004d40 0x7f64680051c0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6478009fc0 tx=0x7f6468005830 comp rx=0 tx=0).stop 2024-03-06T07:26:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.928+0000 7f6484dc6640 1 -- 172.21.15.77:0/2681340719 reap_dead start 2024-03-06T07:26:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.928+0000 7f6486829640 1 -- 172.21.15.77:0/2681340719 shutdown_connections 2024-03-06T07:26:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.928+0000 7f6486829640 1 -- 172.21.15.77:0/2681340719 >> 172.21.15.77:0/2681340719 conn(0x7f648009d230 msgr2=0x7f64800d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.928+0000 7f6486829640 1 -- 172.21.15.77:0/2681340719 shutdown_connections 2024-03-06T07:26:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.928+0000 7f6486829640 1 -- 172.21.15.77:0/2681340719 wait complete. 2024-03-06T07:26:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.929+0000 7f6486829640 1 Processor -- start 2024-03-06T07:26:39.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.929+0000 7f6486829640 1 -- start start 2024-03-06T07:26:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6486829640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6480154400 0x7f64801602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6486829640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800cd560 0x7f64800cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6486829640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f648006c840 0x7f648006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6486829640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64800d2f40 con 0x7f648006c840 2024-03-06T07:26:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6486829640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64800d1240 con 0x7f64800cd560 2024-03-06T07:26:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6486829640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64800b7dc0 con 0x7f6480154400 2024-03-06T07:26:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6484dc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f648006c840 0x7f648006ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6484dc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f648006c840 0x7f648006ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33228/0 (socket says 172.21.15.77:33228) 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6484dc6640 1 -- 172.21.15.77:0/1728663989 learned_addr learned my addr 172.21.15.77:0/1728663989 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f647f7fe640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800cd560 0x7f64800cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f647ffff640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6480154400 0x7f64801602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6484dc6640 1 -- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6480154400 msgr2=0x7f64801602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6484dc6640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6480154400 0x7f64801602c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6484dc6640 1 -- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800cd560 msgr2=0x7f64800cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6484dc6640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800cd560 0x7f64800cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f6484dc6640 1 -- 172.21.15.77:0/1728663989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648015f220 con 0x7f648006c840 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f647f7fe640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800cd560 0x7f64800cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.930+0000 7f647ffff640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6480154400 0x7f64801602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.931+0000 7f6484dc6640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f648006c840 0x7f648006ccc0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6478002230 tx=0x7f6478009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.931+0000 7f647cff9640 1 -- 172.21.15.77:0/1728663989 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478039040 con 0x7f648006c840 2024-03-06T07:26:39.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.931+0000 7f647cff9640 1 -- 172.21.15.77:0/1728663989 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6478046070 con 0x7f648006c840 2024-03-06T07:26:39.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.931+0000 7f647cff9640 1 -- 172.21.15.77:0/1728663989 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478009c10 con 0x7f648006c840 2024-03-06T07:26:39.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.931+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648015f500 con 0x7f648006c840 2024-03-06T07:26:39.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.931+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648015fac0 con 0x7f648006c840 2024-03-06T07:26:39.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.932+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6480069ee0 con 0x7f648006c840 2024-03-06T07:26:39.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.933+0000 7f647cff9640 1 -- 172.21.15.77:0/1728663989 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f647803f470 con 0x7f648006c840 2024-03-06T07:26:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.934+0000 7f647cff9640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f646c06b500 0x7f646c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.934+0000 7f647cff9640 1 -- 172.21.15.77:0/1728663989 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f64780acb20 con 0x7f648006c840 2024-03-06T07:26:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.934+0000 7f647ffff640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f646c06b500 0x7f646c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.934+0000 7f647ffff640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f646c06b500 0x7f646c06d9c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6470004840 tx=0x7f6470009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:39.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:39.939+0000 7f647cff9640 1 -- 172.21.15.77:0/1728663989 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f647807b9c0 con 0x7f648006c840 2024-03-06T07:26:40.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.254+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64800b5510 con 0x7f648006c840 2024-03-06T07:26:40.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.254+0000 7f647cff9640 1 -- 172.21.15.77:0/1728663989 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f647807b2c0 con 0x7f648006c840 2024-03-06T07:26:40.256 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:40.257 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:40.257 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:40.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f646c06b500 msgr2=0x7f646c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f646c06b500 0x7f646c06d9c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6470004840 tx=0x7f6470009040 comp rx=0 tx=0).stop 2024-03-06T07:26:40.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f648006c840 msgr2=0x7f648006ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:40.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f648006c840 0x7f648006ccc0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6478002230 tx=0x7f6478009750 comp rx=0 tx=0).stop 2024-03-06T07:26:40.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 shutdown_connections 2024-03-06T07:26:40.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f646c06b500 0x7f646c06d9c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f648006c840 0x7f648006ccc0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64800cd560 0x7f64800cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 --2- 172.21.15.77:0/1728663989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6480154400 0x7f64801602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:40.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 >> 172.21.15.77:0/1728663989 conn(0x7f648009d230 msgr2=0x7f64800b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:40.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 shutdown_connections 2024-03-06T07:26:40.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:40.257+0000 7f6486829640 1 -- 172.21.15.77:0/1728663989 wait complete. 2024-03-06T07:26:40.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:40.454 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":206,"msgr_send_messages":201,"msgr_recv_bytes":157551,"msgr_send_bytes":4283402,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.034034579,"msgr_running_send_time":0.017253420,"msgr_running_recv_time":0.007622063,"msgr_running_fast_dispatch_time":0.006936179,"msgr_send_messages_queue_lat":{"avgcount":200,"sum":0.013205392,"avgtime":0.000066026},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":107936,"msgr_send_encrypted_bytes":7280},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8616,"msgr_send_messages":18901,"msgr_recv_bytes":3137366,"msgr_send_bytes":18812651,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.812461648,"msgr_running_send_time":0.451241874,"msgr_running_recv_time":0.415814644,"msgr_running_fast_dispatch_time":0.013056144,"msgr_send_messages_queue_lat":{"avgcount":18900,"sum":0.274316231,"avgtime":0.000014514},"msgr_handle_ack_lat":{"avgcount":25342,"sum":0.003847864,"avgtime":0.000000151},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94464},"AsyncMessenger::Worker-2":{"msgr_recv_messages":334,"msgr_send_messages":338,"msgr_recv_bytes":95733,"msgr_send_bytes":3412817,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.041530461,"msgr_running_send_time":0.018705122,"msgr_running_recv_time":0.008223894,"msgr_running_fast_dispatch_time":0.011006977,"msgr_send_messages_queue_lat":{"avgcount":337,"sum":0.025229822,"avgtime":0.000074865},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":86,"sum":4.649127949,"avgtime":0.054059627}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":954,"sum":0.694323744,"avgtime":0.000727802}},"mds":{"request":5180,"reply":5178,"reply_latency":{"avgcount":5178,"sum":7.665040514,"avgtime":0.001480309},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4937,"inodes_top":0,"inodes_bottom":4937,"inodes_pin_tail":0,"inodes_pinned":4937,"inodes_expired":0,"inodes_with_caps":2579,"caps":2579,"subtrees":2,"traverse":5186,"traverse_hit":5185,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2048,"root_rbytes":11075584,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":624,"ceph_cap_op_grant":5335,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3190,"handle_client_caps_dirty":2986,"handle_client_cap_release":4,"process_request_cap_release":7137},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1345,"num_strays_delayed":499,"num_strays_enqueuing":0,"strays_created":1968,"strays_enqueued":623,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":8992,"evex":0,"evtrm":0,"ev":8995,"evexg":0,"evexd":0,"segadd":11,"segex":0,"segtrm":0,"seg":13,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":23327477,"rdpos":4195244,"jlat":{"avgcount":16,"sum":1.478778254,"avgtime":0.092423640},"replayed":3},"mds_mem":{"ino":3033,"ino+":3214,"ino-":181,"dir":13,"dir+":13,"dir-":0,"dn":4998,"dn+":5179,"dn-":181,"cap":2080,"cap+":3202,"cap-":1122,"rss":103200,"heap":248164},"mds_server":{"dispatch_client_request":5183,"dispatch_server_request":0,"handle_client_request":5180,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1968,"sum":4.108508046,"avgtime":0.002087656},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4796,"average_load":4796,"avg_session_uptime":811,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":23696541,"buffer_anon_items":6420,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14704495,"mds_co_items":106170,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":2,"op_laggy":0,"op_send":726,"op_send_bytes":17595120,"op_resend":0,"op_reply":724,"op_latency":{"avgcount":724,"sum":1.285971834,"avgtime":0.001776204},"op_inflight":2,"oplen_avg":{"avgcount":726,"sum":790},"op":726,"op_r":45,"op_w":681,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":17,"osdop_write":25,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":638,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":24,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":1,"pq_executing_ops_high_water":6,"pq_executing":1,"pq_executing_high_water":6,"pq_executed_ops":622,"pq_executed":622,"pq_item_in_journal":2},"throttle-msgr_dispatch_throttler-mds":{"val":4196,"max":104857600,"get_started":0,"get":9155,"get_sum":2564745,"get_or_fail_fail":0,"get_or_fail_success":9155,"take":0,"take_sum":0,"put":9139,"put_sum":2560549,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":727,"take_sum":17659250,"put":726,"put_sum":17659160,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":1,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":727,"take_sum":727,"put":726,"put_sum":726,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":623,"get_sum":62923,"get_or_fail_fail":0,"get_or_fail_success":623,"take":0,"take_sum":0,"put":9,"put_sum":62923,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":2004840,"max":3758096384,"get_started":0,"get":8993,"get_sum":19133725,"get_or_fail_fail":0,"get_or_fail_success":8993,"take":0,"take_sum":0,"put":16,"put_sum":17128885,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:40.456 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 157551, "msgr_recv_encrypted_bytes": 107936, "msgr_recv_messages": 206, "msgr_running_fast_dispatch_time": 0.006936179, "msgr_running_recv_time": 0.007622063, "msgr_running_send_time": 0.01725342, "msgr_running_total_time": 0.034034579, "msgr_send_bytes": 4283402, "msgr_send_encrypted_bytes": 7280, "msgr_send_messages": 201, "msgr_send_messages_queue_lat": { "avgcount": 200, "avgtime": 6.6026e-05, "sum": 0.013205392 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 25342, "avgtime": 1.51e-07, "sum": 0.003847864 }, "msgr_recv_bytes": 3137366, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 8616, "msgr_running_fast_dispatch_time": 0.013056144, "msgr_running_recv_time": 0.415814644, "msgr_running_send_time": 0.451241874, "msgr_running_total_time": 0.812461648, "msgr_send_bytes": 18812651, "msgr_send_encrypted_bytes": 94464, "msgr_send_messages": 18901, "msgr_send_messages_queue_lat": { "avgcount": 18900, "avgtime": 1.4514e-05, "sum": 0.274316231 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 95733, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 334, "msgr_running_fast_dispatch_time": 0.011006977, "msgr_running_recv_time": 0.008223894, "msgr_running_send_time": 0.018705122, "msgr_running_total_time": 0.041530461, "msgr_send_bytes": 3412817, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 338, "msgr_send_messages_queue_lat": { "avgcount": 337, "avgtime": 7.4865e-05, "sum": 0.025229822 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 86, "avgtime": 0.054059627, "sum": 4.649127949 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 954, "avgtime": 0.000727802, "sum": 0.694323744 }, "queue_len": 0 }, "mds": { "caps": 2579, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5335, "ceph_cap_op_revoke": 624, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 4, "handle_client_caps": 3190, "handle_client_caps_dirty": 2986, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4937, "inodes_bottom": 4937, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4937, "inodes_top": 0, "inodes_with_caps": 2579, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 7137, "q": 0, "reply": 5178, "reply_latency": { "avgcount": 5178, "avgtime": 0.001480309, "sum": 7.665040514 }, "request": 5180, "root_rbytes": 11075584, "root_rfiles": 2048, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5186, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5185, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1345, "num_strays_delayed": 499, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1968, "strays_enqueued": 623, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8995, "evadd": 8992, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 16, "avgtime": 0.09242364, "sum": 1.478778254 }, "rdpos": 4195244, "replayed": 3, "seg": 13, "segadd": 11, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 23327477 }, "mds_mem": { "cap": 2080, "cap+": 3202, "cap-": 1122, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4998, "dn+": 5179, "dn-": 181, "heap": 248164, "ino": 3033, "ino+": 3214, "ino-": 181, "rss": 103200 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5183, "dispatch_server_request": 0, "handle_client_request": 5180, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1968, "avgtime": 0.002087656, "sum": 4.108508046 } }, "mds_sessions": { "average_load": 4796, "avg_session_uptime": 811, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4796 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 23696541, "buffer_anon_items": 6420, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 14704495, "mds_co_items": 106170, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 24, "op": 726, "op_Infinitylight": 2, "op_active": 2, "op_laggy": 0, "op_latency": { "avgcount": 724, "avgtime": 0.001776204, "sum": 1.285971834 }, "op_pg": 0, "op_r": 45, "op_reply": 724, "op_resend": 0, "op_rmw": 0, "op_send": 726, "op_send_bytes": 17595120, "op_w": 681, "oplen_avg": { "avgcount": 726, "sum": 790 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 638, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 25, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 622, "pq_executed_ops": 622, "pq_executing": 1, "pq_executing_high_water": 6, "pq_executing_ops": 1, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 2 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9155, "get_or_fail_fail": 0, "get_or_fail_success": 9155, "get_started": 0, "get_sum": 2564745, "max": 104857600, "put": 9139, "put_sum": 2560549, "take": 0, "take_sum": 0, "val": 4196, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 726, "put_sum": 17659160, "take": 727, "take_sum": 17659250, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 726, "put_sum": 726, "take": 727, "take_sum": 727, "val": 1, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 623, "get_or_fail_fail": 0, "get_or_fail_success": 623, "get_started": 0, "get_sum": 62923, "max": 3758096384, "put": 9, "put_sum": 62923, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 8993, "get_or_fail_fail": 0, "get_or_fail_success": 8993, "get_started": 0, "get_sum": 19133725, "max": 3758096384, "put": 16, "put_sum": 17128885, "take": 0, "take_sum": 0, "val": 2004840, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:40.456 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 1/1345, 622/1968 2024-03-06T07:26:41.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:41.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.582+0000 7f9e29aeb640 1 Processor -- start 2024-03-06T07:26:41.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.582+0000 7f9e29aeb640 1 -- start start 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e29aeb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9e24161a80 0x7f9e240d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e29aeb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e240b5010 0x7f9e240b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e29aeb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e240d25a0 0x7f9e240d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e29aeb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240b3e30 con 0x7f9e240b5010 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e29aeb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e24161e60 con 0x7f9e24161a80 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e29aeb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240d1470 con 0x7f9e240d25a0 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e22ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e240b5010 0x7f9e240b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e22ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e240b5010 0x7f9e240b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33236/0 (socket says 172.21.15.77:33236) 2024-03-06T07:26:41.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e22ffd640 1 -- 172.21.15.77:0/1367111525 learned_addr learned my addr 172.21.15.77:0/1367111525 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e237fe640 1 --2- 172.21.15.77:0/1367111525 >> v2:172.21.15.158:3300/0 conn(0x7f9e24161a80 0x7f9e240d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.584+0000 7f9e22ffd640 1 -- 172.21.15.77:0/1367111525 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e240d25a0 msgr2=0x7f9e240d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.583+0000 7f9e23fff640 1 --2- 172.21.15.77:0/1367111525 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e240d25a0 0x7f9e240d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.584+0000 7f9e22ffd640 1 --2- 172.21.15.77:0/1367111525 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e240d25a0 0x7f9e240d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.584+0000 7f9e22ffd640 1 -- 172.21.15.77:0/1367111525 >> v2:172.21.15.158:3300/0 conn(0x7f9e24161a80 msgr2=0x7f9e240d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.584+0000 7f9e22ffd640 1 --2- 172.21.15.77:0/1367111525 >> v2:172.21.15.158:3300/0 conn(0x7f9e24161a80 0x7f9e240d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.584+0000 7f9e22ffd640 1 -- 172.21.15.77:0/1367111525 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240d1750 con 0x7f9e240b5010 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.584+0000 7f9e23fff640 1 --2- 172.21.15.77:0/1367111525 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e240d25a0 0x7f9e240d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:41.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e22ffd640 1 --2- 172.21.15.77:0/1367111525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e240b0b40 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f9e240b6bb0 tx=0x7f9e1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db2d9de7b312c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e227fc640 1 -- 172.21.15.77:0/1367111525 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18035020 con 0x7f9e240b5010 2024-03-06T07:26:41.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e227fc640 1 -- 172.21.15.77:0/1367111525 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e18031870 con 0x7f9e240b5010 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e227fc640 1 -- 172.21.15.77:0/1367111525 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18031c40 con 0x7f9e240b5010 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 -- 172.21.15.77:0/1367111525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 msgr2=0x7f9e240b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/1367111525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e240b0b40 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f9e240b6bb0 tx=0x7f9e1802b910 comp rx=0 tx=0).stop 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 -- 172.21.15.77:0/1367111525 shutdown_connections 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/1367111525 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e240d25a0 0x7f9e240d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/1367111525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e240b0b40 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/1367111525 >> v2:172.21.15.158:3300/0 conn(0x7f9e24161a80 0x7f9e240d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 -- 172.21.15.77:0/1367111525 >> 172.21.15.77:0/1367111525 conn(0x7f9e240d3fc0 msgr2=0x7f9e240d43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 -- 172.21.15.77:0/1367111525 shutdown_connections 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.585+0000 7f9e29aeb640 1 -- 172.21.15.77:0/1367111525 wait complete. 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.587+0000 7f9e29aeb640 1 Processor -- start 2024-03-06T07:26:41.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.587+0000 7f9e29aeb640 1 -- start start 2024-03-06T07:26:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e29aeb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e2415c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e29aeb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e240d25a0 0x7f9e2415c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e29aeb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e24161a80 0x7f9e241f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e29aeb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240d1210 con 0x7f9e240b5010 2024-03-06T07:26:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e29aeb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240d1090 con 0x7f9e240d25a0 2024-03-06T07:26:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e29aeb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240b1080 con 0x7f9e24161a80 2024-03-06T07:26:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e2415c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e2415c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33250/0 (socket says 172.21.15.77:33250) 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e237fe640 1 -- 172.21.15.77:0/2174347992 learned_addr learned my addr 172.21.15.77:0/2174347992 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e22ffd640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e240d25a0 0x7f9e2415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e23fff640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e24161a80 0x7f9e241f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e237fe640 1 -- 172.21.15.77:0/2174347992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e24161a80 msgr2=0x7f9e241f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e237fe640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e24161a80 0x7f9e241f75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e237fe640 1 -- 172.21.15.77:0/2174347992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e240d25a0 msgr2=0x7f9e2415c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e237fe640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e240d25a0 0x7f9e2415c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.588+0000 7f9e237fe640 1 -- 172.21.15.77:0/2174347992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240c7f80 con 0x7f9e240b5010 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.589+0000 7f9e237fe640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e2415c2e0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400e870 tx=0x7f9e1400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.589+0000 7f9e0bfff640 1 -- 172.21.15.77:0/2174347992 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14017070 con 0x7f9e240b5010 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.589+0000 7f9e0bfff640 1 -- 172.21.15.77:0/2174347992 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1400c940 con 0x7f9e240b5010 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.589+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e240c8260 con 0x7f9e240b5010 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.589+0000 7f9e0bfff640 1 -- 172.21.15.77:0/2174347992 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400cd30 con 0x7f9e240b5010 2024-03-06T07:26:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.589+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e240c8820 con 0x7f9e240b5010 2024-03-06T07:26:41.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.590+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e24068050 con 0x7f9e240b5010 2024-03-06T07:26:41.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.591+0000 7f9e0bfff640 1 -- 172.21.15.77:0/2174347992 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e1401a070 con 0x7f9e240b5010 2024-03-06T07:26:41.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.591+0000 7f9e0bfff640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e1006b4d0 0x7f9e1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.592+0000 7f9e0bfff640 1 -- 172.21.15.77:0/2174347992 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9e1408a580 con 0x7f9e240b5010 2024-03-06T07:26:41.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.592+0000 7f9e22ffd640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e1006b4d0 0x7f9e1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.592+0000 7f9e22ffd640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e1006b4d0 0x7f9e1006d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9e18036000 tx=0x7f9e1803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.596+0000 7f9e0bfff640 1 -- 172.21.15.77:0/2174347992 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e14059300 con 0x7f9e240b5010 2024-03-06T07:26:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.942+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e240d2380 con 0x7f9e240b5010 2024-03-06T07:26:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.943+0000 7f9e0bfff640 1 -- 172.21.15.77:0/2174347992 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f9e14058c00 con 0x7f9e240b5010 2024-03-06T07:26:41.945 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:41.945 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.945+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e1006b4d0 msgr2=0x7f9e1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.945+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e1006b4d0 0x7f9e1006d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9e18036000 tx=0x7f9e1803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.945+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 msgr2=0x7f9e2415c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.945+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e2415c2e0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400e870 tx=0x7f9e1400ed30 comp rx=0 tx=0).stop 2024-03-06T07:26:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.946+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 shutdown_connections 2024-03-06T07:26:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.946+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9e24161a80 0x7f9e241f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.946+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9e240d25a0 0x7f9e2415c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.946+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9e1006b4d0 0x7f9e1006d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.946+0000 7f9e29aeb640 1 --2- 172.21.15.77:0/2174347992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e240b5010 0x7f9e2415c2e0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.946+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 >> 172.21.15.77:0/2174347992 conn(0x7f9e240d3fc0 msgr2=0x7f9e2406c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.946+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 shutdown_connections 2024-03-06T07:26:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:41.946+0000 7f9e29aeb640 1 -- 172.21.15.77:0/2174347992 wait complete. 2024-03-06T07:26:41.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:42.111 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":331,"msgr_send_messages":326,"msgr_recv_bytes":189311,"msgr_send_bytes":4324442,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.048016728,"msgr_running_send_time":0.024603954,"msgr_running_recv_time":0.010513650,"msgr_running_fast_dispatch_time":0.009900312,"msgr_send_messages_queue_lat":{"avgcount":325,"sum":0.018616589,"avgtime":0.000057281},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":108208,"msgr_send_encrypted_bytes":7904},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9681,"msgr_send_messages":20910,"msgr_recv_bytes":3631194,"msgr_send_bytes":23553123,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.922006476,"msgr_running_send_time":0.512088731,"msgr_running_recv_time":0.467104269,"msgr_running_fast_dispatch_time":0.020164814,"msgr_send_messages_queue_lat":{"avgcount":20909,"sum":0.306767249,"avgtime":0.000014671},"msgr_handle_ack_lat":{"avgcount":27706,"sum":0.004245939,"avgtime":0.000000153},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94464},"AsyncMessenger::Worker-2":{"msgr_recv_messages":589,"msgr_send_messages":591,"msgr_recv_bytes":160463,"msgr_send_bytes":3631282,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.066630232,"msgr_running_send_time":0.030990717,"msgr_running_recv_time":0.012603690,"msgr_running_fast_dispatch_time":0.018076913,"msgr_send_messages_queue_lat":{"avgcount":590,"sum":0.038957378,"avgtime":0.000066029},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":2052,"complete_latency":{"avgcount":89,"sum":4.649545566,"avgtime":0.052242085}},"finisher-PurgeQueue":{"queue_len":382,"complete_latency":{"avgcount":1763,"sum":0.846786946,"avgtime":0.000480310}},"mds":{"request":6098,"reply":6095,"reply_latency":{"avgcount":6095,"sum":8.863272043,"avgtime":0.001454187},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5488,"inodes_top":0,"inodes_bottom":5488,"inodes_pin_tail":0,"inodes_pinned":5488,"inodes_expired":0,"inodes_with_caps":1670,"caps":1670,"subtrees":2,"traverse":6104,"traverse_hit":6103,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1275,"root_rbytes":9475072,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":624,"ceph_cap_op_grant":5743,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3204,"handle_client_caps_dirty":3000,"handle_client_cap_release":5,"process_request_cap_release":8973},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1440,"num_strays_delayed":0,"num_strays_enqueuing":87,"strays_created":2885,"strays_enqueued":1445,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10749,"evex":0,"evtrm":0,"ev":10752,"evexg":0,"evexd":0,"segadd":13,"segex":0,"segtrm":0,"seg":15,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26776493,"rdpos":4195244,"jlat":{"avgcount":17,"sum":1.502771351,"avgtime":0.088398314},"replayed":3},"mds_mem":{"ino":2607,"ino+":3214,"ino-":607,"dir":13,"dir+":13,"dir-":0,"dn":5489,"dn+":6096,"dn-":607,"cap":1670,"cap+":3202,"cap-":1532,"rss":111344,"heap":248164},"mds_server":{"dispatch_client_request":6101,"dispatch_server_request":0,"handle_client_request":6098,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2885,"sum":5.306739575,"avgtime":0.001839424},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5611,"average_load":5611,"avg_session_uptime":816,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":7738685,"buffer_anon_items":2814,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14140592,"mds_co_items":162408,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1247,"op_send_bytes":22659562,"op_resend":0,"op_reply":1247,"op_latency":{"avgcount":1247,"sum":2.121953529,"avgtime":0.001701646},"op_inflight":0,"oplen_avg":{"avgcount":1247,"sum":1314},"op":1247,"op_r":51,"op_w":1196,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":23,"osdop_write":34,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1139,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":27,"omap_rd":48,"omap_del":9},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":1,"pq_executing_ops_high_water":6,"pq_executing":1,"pq_executing_high_water":6,"pq_executed_ops":1122,"pq_executed":1122,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":298,"max":104857600,"get_started":0,"get":10597,"get_sum":3024469,"get_or_fail_fail":0,"get_or_fail_success":10597,"take":0,"take_sum":0,"put":10596,"put_sum":3024171,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1247,"take_sum":22815411,"put":1247,"put_sum":22815411,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1247,"take_sum":1247,"put":1247,"put_sum":1247,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1532,"get_sum":154732,"get_or_fail_fail":0,"get_or_fail_success":1532,"take":0,"take_sum":0,"put":15,"put_sum":154732,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":480513,"max":3758096384,"get_started":0,"get":10749,"get_sum":22580808,"get_or_fail_fail":0,"get_or_fail_success":10749,"take":0,"take_sum":0,"put":19,"put_sum":22100295,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:42.113 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 189311, "msgr_recv_encrypted_bytes": 108208, "msgr_recv_messages": 331, "msgr_running_fast_dispatch_time": 0.009900312, "msgr_running_recv_time": 0.01051365, "msgr_running_send_time": 0.024603954, "msgr_running_total_time": 0.048016728, "msgr_send_bytes": 4324442, "msgr_send_encrypted_bytes": 7904, "msgr_send_messages": 326, "msgr_send_messages_queue_lat": { "avgcount": 325, "avgtime": 5.7281e-05, "sum": 0.018616589 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27706, "avgtime": 1.53e-07, "sum": 0.004245939 }, "msgr_recv_bytes": 3631194, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 9681, "msgr_running_fast_dispatch_time": 0.020164814, "msgr_running_recv_time": 0.467104269, "msgr_running_send_time": 0.512088731, "msgr_running_total_time": 0.922006476, "msgr_send_bytes": 23553123, "msgr_send_encrypted_bytes": 94464, "msgr_send_messages": 20910, "msgr_send_messages_queue_lat": { "avgcount": 20909, "avgtime": 1.4671e-05, "sum": 0.306767249 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 160463, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 589, "msgr_running_fast_dispatch_time": 0.018076913, "msgr_running_recv_time": 0.01260369, "msgr_running_send_time": 0.030990717, "msgr_running_total_time": 0.066630232, "msgr_send_bytes": 3631282, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 591, "msgr_send_messages_queue_lat": { "avgcount": 590, "avgtime": 6.6029e-05, "sum": 0.038957378 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 89, "avgtime": 0.052242085, "sum": 4.649545566 }, "queue_len": 2052 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1763, "avgtime": 0.00048031, "sum": 0.846786946 }, "queue_len": 382 }, "mds": { "caps": 1670, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5743, "ceph_cap_op_revoke": 624, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 5, "handle_client_caps": 3204, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5488, "inodes_bottom": 5488, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5488, "inodes_top": 0, "inodes_with_caps": 1670, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 8973, "q": 0, "reply": 6095, "reply_latency": { "avgcount": 6095, "avgtime": 0.001454187, "sum": 8.863272043 }, "request": 6098, "root_rbytes": 9475072, "root_rfiles": 1275, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6104, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6103, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1440, "num_strays_delayed": 0, "num_strays_enqueuing": 87, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2885, "strays_enqueued": 1445, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10752, "evadd": 10749, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 17, "avgtime": 0.088398314, "sum": 1.502771351 }, "rdpos": 4195244, "replayed": 3, "seg": 15, "segadd": 13, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26776493 }, "mds_mem": { "cap": 1670, "cap+": 3202, "cap-": 1532, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5489, "dn+": 6096, "dn-": 607, "heap": 248164, "ino": 2607, "ino+": 3214, "ino-": 607, "rss": 111344 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6101, "dispatch_server_request": 0, "handle_client_request": 6098, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2885, "avgtime": 0.001839424, "sum": 5.306739575 } }, "mds_sessions": { "average_load": 5611, "avg_session_uptime": 816, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5611 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 7738685, "buffer_anon_items": 2814, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 14140592, "mds_co_items": 162408, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 9, "omap_rd": 48, "omap_wr": 27, "op": 1247, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1247, "avgtime": 0.001701646, "sum": 2.121953529 }, "op_pg": 0, "op_r": 51, "op_reply": 1247, "op_resend": 0, "op_rmw": 0, "op_send": 1247, "op_send_bytes": 22659562, "op_w": 1196, "oplen_avg": { "avgcount": 1247, "sum": 1314 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1139, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 23, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 34, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1122, "pq_executed_ops": 1122, "pq_executing": 1, "pq_executing_high_water": 6, "pq_executing_ops": 1, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10597, "get_or_fail_fail": 0, "get_or_fail_success": 10597, "get_started": 0, "get_sum": 3024469, "max": 104857600, "put": 10596, "put_sum": 3024171, "take": 0, "take_sum": 0, "val": 298, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1247, "put_sum": 22815411, "take": 1247, "take_sum": 22815411, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1247, "put_sum": 1247, "take": 1247, "take_sum": 1247, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1532, "get_or_fail_fail": 0, "get_or_fail_success": 1532, "get_started": 0, "get_sum": 154732, "max": 3758096384, "put": 15, "put_sum": 154732, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 10749, "get_or_fail_fail": 0, "get_or_fail_success": 10749, "get_started": 0, "get_sum": 22580808, "max": 3758096384, "put": 19, "put_sum": 22100295, "take": 0, "take_sum": 0, "val": 480513, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:42.113 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 1/1440, 1122/2885 2024-03-06T07:26:43.114 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:43.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.232+0000 7f000b888640 1 Processor -- start 2024-03-06T07:26:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.232+0000 7f000b888640 1 -- start start 2024-03-06T07:26:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f000b888640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f000415e150 0x7f00040cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f000b888640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00040b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f000b888640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040d1ea0 con 0x7f00040cfba0 2024-03-06T07:26:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f000b888640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00040d2ab0 con 0x7f000415e150 2024-03-06T07:26:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f000b888640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00040d2180 con 0x7f00040d1060 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0009624640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f000415e150 0x7f00040cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0008e23640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f00040cfba0 0x7f00040d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51462/0 (socket says 172.21.15.77:51462) 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0009624640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f000415e150 0x7f00040cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33260/0 (socket says 172.21.15.77:33260) 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0009624640 1 -- 172.21.15.77:0/4025968505 learned_addr learned my addr 172.21.15.77:0/4025968505 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0009e25640 1 --2- 172.21.15.77:0/4025968505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00040b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0009624640 1 -- 172.21.15.77:0/4025968505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 msgr2=0x7f00040b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0009624640 1 --2- 172.21.15.77:0/4025968505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00040b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0009624640 1 -- 172.21.15.77:0/4025968505 >> v1:172.21.15.77:6789/0 conn(0x7f00040cfba0 legacy=0x7f00040d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.234+0000 7f0009624640 1 -- 172.21.15.77:0/4025968505 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040d2460 con 0x7f000415e150 2024-03-06T07:26:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.235+0000 7f0009624640 1 --2- 172.21.15.77:0/4025968505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040cf660 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f0000009870 tx=0x7f000002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102efc8038cacf40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.235+0000 7efffbfff640 1 -- 172.21.15.77:0/4025968505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000035020 con 0x7f000415e150 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.235+0000 7efffbfff640 1 -- 172.21.15.77:0/4025968505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0000031640 con 0x7f000415e150 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.235+0000 7efffbfff640 1 -- 172.21.15.77:0/4025968505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000002d040 con 0x7f000415e150 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.236+0000 7f000b888640 1 -- 172.21.15.77:0/4025968505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 msgr2=0x7f00040cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.236+0000 7f000b888640 1 --2- 172.21.15.77:0/4025968505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040cf660 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f0000009870 tx=0x7f000002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.236+0000 7f000b888640 1 -- 172.21.15.77:0/4025968505 shutdown_connections 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.236+0000 7f000b888640 1 --2- 172.21.15.77:0/4025968505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00040b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.236+0000 7f000b888640 1 --2- 172.21.15.77:0/4025968505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040cf660 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.236+0000 7f000b888640 1 -- 172.21.15.77:0/4025968505 >> 172.21.15.77:0/4025968505 conn(0x7f000409d230 msgr2=0x7f00040d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.236+0000 7f000b888640 1 -- 172.21.15.77:0/4025968505 shutdown_connections 2024-03-06T07:26:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.236+0000 7f000b888640 1 -- 172.21.15.77:0/4025968505 wait complete. 2024-03-06T07:26:43.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.237+0000 7f000b888640 1 Processor -- start 2024-03-06T07:26:43.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.237+0000 7f000b888640 1 -- start start 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f000b888640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00040cfba0 0x7f000415fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f000b888640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00041f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f000b888640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f000b888640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00040d1b30 con 0x7f000415e150 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f000b888640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00040d2f40 con 0x7f00040cfba0 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f000b888640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000415e530 con 0x7f00040d1060 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009e25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009e25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33272/0 (socket says 172.21.15.77:33272) 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009e25640 1 -- 172.21.15.77:0/71943729 learned_addr learned my addr 172.21.15.77:0/71943729 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009624640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00040cfba0 0x7f000415fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0008e23640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00041f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009e25640 1 -- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 msgr2=0x7f00041f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009e25640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00041f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009e25640 1 -- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00040cfba0 msgr2=0x7f000415fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009e25640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00040cfba0 0x7f000415fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009e25640 1 -- 172.21.15.77:0/71943729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000409f6e0 con 0x7f000415e150 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0009624640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00040cfba0 0x7f000415fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.238+0000 7f0008e23640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00041f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.239+0000 7f0009e25640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040c9fc0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7efffc007e30 tx=0x7efffc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.239+0000 7efff9ffb640 1 -- 172.21.15.77:0/71943729 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc014620 con 0x7f000415e150 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.239+0000 7efff9ffb640 1 -- 172.21.15.77:0/71943729 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc019070 con 0x7f000415e150 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.239+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00040c9220 con 0x7f000415e150 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.239+0000 7efff9ffb640 1 -- 172.21.15.77:0/71943729 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc014a00 con 0x7f000415e150 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.239+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00040c97e0 con 0x7f000415e150 2024-03-06T07:26:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.240+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0004069ee0 con 0x7f000415e150 2024-03-06T07:26:43.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.240+0000 7efff9ffb640 1 -- 172.21.15.77:0/71943729 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efffc014ba0 con 0x7f000415e150 2024-03-06T07:26:43.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.241+0000 7efff9ffb640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7effe806b4d0 0x7effe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.241+0000 7efff9ffb640 1 -- 172.21.15.77:0/71943729 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7efffc08ab60 con 0x7f000415e150 2024-03-06T07:26:43.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.241+0000 7f0009624640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7effe806b4d0 0x7effe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.241+0000 7f0009624640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7effe806b4d0 0x7effe806d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0000036000 tx=0x7f0000008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.249+0000 7efff9ffb640 1 -- 172.21.15.77:0/71943729 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f000409f6e0 con 0x7f000415e150 2024-03-06T07:26:43.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.297+0000 7f095a3a0640 1 Processor -- start 2024-03-06T07:26:43.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.297+0000 7f095a3a0640 1 -- start start 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f095a3a0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095415e220 0x7f09540ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f095a3a0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0954162aa0 0x7f095406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f095a3a0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f095406ead0 0x7f0954071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f095a3a0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09540cf2e0 con 0x7f0954162aa0 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f095a3a0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09540cf6e0 con 0x7f095406ead0 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f095a3a0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09540cf9c0 con 0x7f095415e220 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0958b9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0954162aa0 0x7f095406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0958b9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0954162aa0 0x7f095406e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33288/0 (socket says 172.21.15.77:33288) 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0958b9d640 1 -- 172.21.15.77:0/2294914010 learned_addr learned my addr 172.21.15.77:0/2294914010 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f095939e640 1 --2- 172.21.15.77:0/2294914010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095415e220 0x7f09540ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0959b9f640 1 --2- 172.21.15.77:0/2294914010 >> v2:172.21.15.158:3300/0 conn(0x7f095406ead0 0x7f0954071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0958b9d640 1 -- 172.21.15.77:0/2294914010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095415e220 msgr2=0x7f09540ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0958b9d640 1 --2- 172.21.15.77:0/2294914010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095415e220 0x7f09540ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0958b9d640 1 -- 172.21.15.77:0/2294914010 >> v2:172.21.15.158:3300/0 conn(0x7f095406ead0 msgr2=0x7f0954071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0958b9d640 1 --2- 172.21.15.77:0/2294914010 >> v2:172.21.15.158:3300/0 conn(0x7f095406ead0 0x7f0954071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0958b9d640 1 -- 172.21.15.77:0/2294914010 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09540cfca0 con 0x7f0954162aa0 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.298+0000 7f0959b9f640 1 --2- 172.21.15.77:0/2294914010 >> v2:172.21.15.158:3300/0 conn(0x7f095406ead0 0x7f0954071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.299+0000 7f0958b9d640 1 --2- 172.21.15.77:0/2294914010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f095406e590 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f09540b0840 tx=0x7f094402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22823ede9ff9eac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.299+0000 7f094bfff640 1 -- 172.21.15.77:0/2294914010 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094402e070 con 0x7f0954162aa0 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.299+0000 7f094bfff640 1 -- 172.21.15.77:0/2294914010 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0944037070 con 0x7f0954162aa0 2024-03-06T07:26:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.299+0000 7f094bfff640 1 -- 172.21.15.77:0/2294914010 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944031a00 con 0x7f0954162aa0 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 -- 172.21.15.77:0/2294914010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 msgr2=0x7f095406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 --2- 172.21.15.77:0/2294914010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f095406e590 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f09540b0840 tx=0x7f094402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 -- 172.21.15.77:0/2294914010 shutdown_connections 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 --2- 172.21.15.77:0/2294914010 >> v2:172.21.15.158:3300/0 conn(0x7f095406ead0 0x7f0954071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 --2- 172.21.15.77:0/2294914010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f095406e590 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 --2- 172.21.15.77:0/2294914010 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095415e220 0x7f09540ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 -- 172.21.15.77:0/2294914010 >> 172.21.15.77:0/2294914010 conn(0x7f09540b2930 msgr2=0x7f09540b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 -- 172.21.15.77:0/2294914010 shutdown_connections 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.300+0000 7f095a3a0640 1 -- 172.21.15.77:0/2294914010 wait complete. 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.301+0000 7f095a3a0640 1 Processor -- start 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.301+0000 7f095a3a0640 1 -- start start 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f095a3a0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095406ead0 0x7f09541f1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f095a3a0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f095415e220 0x7f095415c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f095a3a0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f0954159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f095a3a0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095415e600 con 0x7f0954162aa0 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f095a3a0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540d0260 con 0x7f095415e220 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f095a3a0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09540722b0 con 0x7f095406ead0 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f0954159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f0954159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33304/0 (socket says 172.21.15.77:33304) 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 -- 172.21.15.77:0/1292497125 learned_addr learned my addr 172.21.15.77:0/1292497125 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0958b9d640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f095415e220 0x7f095415c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 -- 172.21.15.77:0/1292497125 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095406ead0 msgr2=0x7f09541f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095406ead0 0x7f09541f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 -- 172.21.15.77:0/1292497125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f095415e220 msgr2=0x7f095415c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f095415e220 0x7f095415c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 -- 172.21.15.77:0/1292497125 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095415efe0 con 0x7f0954162aa0 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0958b9d640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f095415e220 0x7f095415c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.302+0000 7f0959b9f640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f0954159c90 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f094c009fd0 tx=0x7f094c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.303+0000 7f0949ffb640 1 -- 172.21.15.77:0/1292497125 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c01a070 con 0x7f0954162aa0 2024-03-06T07:26:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.303+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09541517c0 con 0x7f0954162aa0 2024-03-06T07:26:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.303+0000 7f0949ffb640 1 -- 172.21.15.77:0/1292497125 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c003bb0 con 0x7f0954162aa0 2024-03-06T07:26:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.303+0000 7f0949ffb640 1 -- 172.21.15.77:0/1292497125 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c004030 con 0x7f0954162aa0 2024-03-06T07:26:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.303+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0954151d80 con 0x7f0954162aa0 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.304+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0954067ca0 con 0x7f0954162aa0 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.305+0000 7f0949ffb640 1 -- 172.21.15.77:0/1292497125 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f094c01d070 con 0x7f0954162aa0 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.305+0000 7f0949ffb640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f093806b500 0x7f093806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.305+0000 7f095939e640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f093806b500 0x7f093806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.305+0000 7f0949ffb640 1 -- 172.21.15.77:0/1292497125 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f094c08a570 con 0x7f0954162aa0 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.306+0000 7f095939e640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f093806b500 0x7f093806d9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0950009a40 tx=0x7f0950009500 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.314+0000 7f0949ffb640 1 -- 172.21.15.77:0/1292497125 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f094c010070 con 0x7f0954162aa0 2024-03-06T07:26:43.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.489+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00040d0be0 con 0x7f000415e150 2024-03-06T07:26:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.490+0000 7efff9ffb640 1 -- 172.21.15.77:0/71943729 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7efffc059970 con 0x7f000415e150 2024-03-06T07:26:43.492 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:43.492 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:43.492 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7effe806b4d0 msgr2=0x7effe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7effe806b4d0 0x7effe806d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0000036000 tx=0x7f0000008030 comp rx=0 tx=0).stop 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 msgr2=0x7f00040c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040c9fc0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7efffc007e30 tx=0x7efffc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 shutdown_connections 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7effe806b4d0 0x7effe806d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f000415e150 0x7f00040c9fc0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f00040d1060 0x7f00041f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 --2- 172.21.15.77:0/71943729 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f00040cfba0 0x7f000415fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 >> 172.21.15.77:0/71943729 conn(0x7f000409d230 msgr2=0x7f000415f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 shutdown_connections 2024-03-06T07:26:43.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.492+0000 7f000b888640 1 -- 172.21.15.77:0/71943729 wait complete. 2024-03-06T07:26:43.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:43.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.573+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09541593e0 con 0x7f0954162aa0 2024-03-06T07:26:43.576 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:43.576 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:43.576 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.574+0000 7f0949ffb640 1 -- 172.21.15.77:0/1292497125 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f094c059300 con 0x7f0954162aa0 2024-03-06T07:26:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f093806b500 msgr2=0x7f093806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f093806b500 0x7f093806d9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0950009a40 tx=0x7f0950009500 comp rx=0 tx=0).stop 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 msgr2=0x7f0954159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f0954159c90 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f094c009fd0 tx=0x7f094c00d560 comp rx=0 tx=0).stop 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 shutdown_connections 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f093806b500 0x7f093806d9c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0954162aa0 0x7f0954159c90 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f095415e220 0x7f095415c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 --2- 172.21.15.77:0/1292497125 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f095406ead0 0x7f09541f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 >> 172.21.15.77:0/1292497125 conn(0x7f09540b2930 msgr2=0x7f09540b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 shutdown_connections 2024-03-06T07:26:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:43.577+0000 7f095a3a0640 1 -- 172.21.15.77:0/1292497125 wait complete. 2024-03-06T07:26:43.617 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:26:43.662 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":440,"msgr_send_messages":435,"msgr_recv_bytes":216997,"msgr_send_bytes":4359976,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.058543841,"msgr_running_send_time":0.029587026,"msgr_running_recv_time":0.012591586,"msgr_running_fast_dispatch_time":0.012520767,"msgr_send_messages_queue_lat":{"avgcount":434,"sum":0.023669791,"avgtime":0.000054538},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":108208,"msgr_send_encrypted_bytes":7904},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10152,"msgr_send_messages":23968,"msgr_recv_bytes":3815119,"msgr_send_bytes":24402065,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.018834829,"msgr_running_send_time":0.574815670,"msgr_running_recv_time":0.494925944,"msgr_running_fast_dispatch_time":0.024124246,"msgr_send_messages_queue_lat":{"avgcount":23967,"sum":0.370529526,"avgtime":0.000015459},"msgr_handle_ack_lat":{"avgcount":29653,"sum":0.004600653,"avgtime":0.000000155},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":825,"msgr_send_messages":827,"msgr_recv_bytes":220383,"msgr_send_bytes":5245115,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.092997230,"msgr_running_send_time":0.044753648,"msgr_running_recv_time":0.017226690,"msgr_running_fast_dispatch_time":0.024323650,"msgr_send_messages_queue_lat":{"avgcount":826,"sum":0.051891709,"avgtime":0.000062822},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":93,"sum":6.599858111,"avgtime":0.070966216}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2552,"sum":1.211220089,"avgtime":0.000474616}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.776799583,"avgtime":0.001525003},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4880,"inodes_top":0,"inodes_bottom":4880,"inodes_pin_tail":0,"inodes_pinned":4880,"inodes_expired":0,"inodes_with_caps":1594,"caps":1594,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":232,"root_rbytes":1807360,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":626,"ceph_cap_op_grant":7019,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":7,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1593,"num_strays_delayed":531,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":1608,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11229,"evex":0,"evtrm":0,"ev":11232,"evexg":0,"evexd":0,"segadd":14,"segex":0,"segtrm":0,"seg":16,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":27881743,"rdpos":4195244,"jlat":{"avgcount":20,"sum":3.316910039,"avgtime":0.165845501},"replayed":3},"mds_mem":{"ino":1682,"ino+":3214,"ino-":1532,"dir":13,"dir+":13,"dir-":0,"dn":4880,"dn+":6412,"dn-":1532,"cap":1063,"cap+":3202,"cap-":2139,"rss":111344,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.480993773,"avgtime":0.480993773},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.739273342,"avgtime":0.001793522},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5854,"average_load":5854,"avg_session_uptime":816,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":7738685,"buffer_anon_items":2814,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14140592,"mds_co_items":162408,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1742,"op_send_bytes":24204373,"op_resend":0,"op_reply":1742,"op_latency":{"avgcount":1742,"sum":2.932935784,"avgtime":0.001683660},"op_inflight":0,"oplen_avg":{"avgcount":1742,"sum":1810},"op":1742,"op_r":54,"op_w":1688,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":26,"osdop_write":38,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1624,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":29,"omap_rd":48,"omap_del":9},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1608,"pq_executed":1608,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":11413,"get_sum":3222560,"get_or_fail_fail":0,"get_or_fail_success":11413,"take":0,"take_sum":0,"put":11413,"put_sum":3222560,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1742,"take_sum":24367898,"put":1742,"put_sum":24367898,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1742,"take_sum":1742,"put":1742,"put_sum":1742,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1608,"get_sum":162408,"get_or_fail_fail":0,"get_or_fail_success":1608,"take":0,"take_sum":0,"put":18,"put_sum":162408,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":49022,"max":3758096384,"get_started":0,"get":11229,"get_sum":23686058,"get_or_fail_fail":0,"get_or_fail_success":11229,"take":0,"take_sum":0,"put":20,"put_sum":23637036,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:43.663 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 216997, "msgr_recv_encrypted_bytes": 108208, "msgr_recv_messages": 440, "msgr_running_fast_dispatch_time": 0.012520767, "msgr_running_recv_time": 0.012591586, "msgr_running_send_time": 0.029587026, "msgr_running_total_time": 0.058543841, "msgr_send_bytes": 4359976, "msgr_send_encrypted_bytes": 7904, "msgr_send_messages": 435, "msgr_send_messages_queue_lat": { "avgcount": 434, "avgtime": 5.4538e-05, "sum": 0.023669791 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29653, "avgtime": 1.55e-07, "sum": 0.004600653 }, "msgr_recv_bytes": 3815119, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 10152, "msgr_running_fast_dispatch_time": 0.024124246, "msgr_running_recv_time": 0.494925944, "msgr_running_send_time": 0.57481567, "msgr_running_total_time": 1.018834829, "msgr_send_bytes": 24402065, "msgr_send_encrypted_bytes": 96256, "msgr_send_messages": 23968, "msgr_send_messages_queue_lat": { "avgcount": 23967, "avgtime": 1.5459e-05, "sum": 0.370529526 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 220383, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 825, "msgr_running_fast_dispatch_time": 0.02432365, "msgr_running_recv_time": 0.01722669, "msgr_running_send_time": 0.044753648, "msgr_running_total_time": 0.09299723, "msgr_send_bytes": 5245115, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 827, "msgr_send_messages_queue_lat": { "avgcount": 826, "avgtime": 6.2822e-05, "sum": 0.051891709 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 93, "avgtime": 0.070966216, "sum": 6.599858111 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2552, "avgtime": 0.000474616, "sum": 1.211220089 }, "queue_len": 0 }, "mds": { "caps": 1594, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7019, "ceph_cap_op_revoke": 626, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 7, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4880, "inodes_bottom": 4880, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4880, "inodes_top": 0, "inodes_with_caps": 1594, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001525003, "sum": 9.776799583 }, "request": 6413, "root_rbytes": 1807360, "root_rfiles": 232, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1593, "num_strays_delayed": 531, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 1608, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11232, "evadd": 11229, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 20, "avgtime": 0.165845501, "sum": 3.316910039 }, "rdpos": 4195244, "replayed": 3, "seg": 16, "segadd": 14, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 27881743 }, "mds_mem": { "cap": 1063, "cap+": 3202, "cap-": 2139, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4880, "dn+": 6412, "dn-": 1532, "heap": 248164, "ino": 1682, "ino+": 3214, "ino-": 1532, "rss": 111344 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.480993773, "sum": 0.480993773 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001793522, "sum": 5.739273342 } }, "mds_sessions": { "average_load": 5854, "avg_session_uptime": 816, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5854 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 7738685, "buffer_anon_items": 2814, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 14140592, "mds_co_items": 162408, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 9, "omap_rd": 48, "omap_wr": 29, "op": 1742, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1742, "avgtime": 0.00168366, "sum": 2.932935784 }, "op_pg": 0, "op_r": 54, "op_reply": 1742, "op_resend": 0, "op_rmw": 0, "op_send": 1742, "op_send_bytes": 24204373, "op_w": 1688, "oplen_avg": { "avgcount": 1742, "sum": 1810 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1624, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 26, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 38, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1608, "pq_executed_ops": 1608, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11413, "get_or_fail_fail": 0, "get_or_fail_success": 11413, "get_started": 0, "get_sum": 3222560, "max": 104857600, "put": 11413, "put_sum": 3222560, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1742, "put_sum": 24367898, "take": 1742, "take_sum": 24367898, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1742, "put_sum": 1742, "take": 1742, "take_sum": 1742, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1608, "get_or_fail_fail": 0, "get_or_fail_success": 1608, "get_started": 0, "get_sum": 162408, "max": 3758096384, "put": 18, "put_sum": 162408, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 11229, "get_or_fail_fail": 0, "get_or_fail_success": 11229, "get_started": 0, "get_sum": 23686058, "max": 3758096384, "put": 20, "put_sum": 23637036, "take": 0, "take_sum": 0, "val": 49022, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:43.663 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1593, 1608/3201 2024-03-06T07:26:44.031 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:26:44.032 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:26:44.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.768+0000 7f2c669e4640 1 Processor -- start 2024-03-06T07:26:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.768+0000 7f2c669e4640 1 -- start start 2024-03-06T07:26:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c669e4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c60154af0 0x7f2c600cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c669e4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c6006d9e0 0x7f2c6006de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c669e4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c600ce190 con 0x7f2c6015abc0 2024-03-06T07:26:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c669e4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c600cf9a0 con 0x7f2c60154af0 2024-03-06T07:26:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c669e4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c60070d70 con 0x7f2c6006d9e0 2024-03-06T07:26:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c5ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c60154af0 0x7f2c600cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c5f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2c6015abc0 0x7f2c60072aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51468/0 (socket says 172.21.15.77:51468) 2024-03-06T07:26:44.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c5f7fe640 1 -- 172.21.15.77:0/1484391940 learned_addr learned my addr 172.21.15.77:0/1484391940 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:44.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c5ffff640 1 -- 172.21.15.77:0/1484391940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c6006d9e0 msgr2=0x7f2c6006de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:44.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c5ffff640 1 --2- 172.21.15.77:0/1484391940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c6006d9e0 0x7f2c6006de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c5ffff640 1 -- 172.21.15.77:0/1484391940 >> v1:172.21.15.77:6789/0 conn(0x7f2c6015abc0 legacy=0x7f2c60072aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c5ffff640 1 -- 172.21.15.77:0/1484391940 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6015b0c0 con 0x7f2c60154af0 2024-03-06T07:26:44.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.769+0000 7f2c5effd640 1 -- 172.21.15.77:0/1484391940 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048455491 0 0) 0x7f2c600ce190 con 0x7f2c6015abc0 2024-03-06T07:26:44.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c5ffff640 1 --2- 172.21.15.77:0/1484391940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60154af0 0x7f2c600cdc50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f2c500097a0 tx=0x7f2c5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9817e5b8351da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c5effd640 1 -- 172.21.15.77:0/1484391940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50035020 con 0x7f2c60154af0 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c5effd640 1 -- 172.21.15.77:0/1484391940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c50031830 con 0x7f2c60154af0 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c5effd640 1 -- 172.21.15.77:0/1484391940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50031c00 con 0x7f2c60154af0 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c669e4640 1 -- 172.21.15.77:0/1484391940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60154af0 msgr2=0x7f2c600cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1484391940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60154af0 0x7f2c600cdc50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f2c500097a0 tx=0x7f2c5002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c669e4640 1 -- 172.21.15.77:0/1484391940 shutdown_connections 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1484391940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c6006d9e0 0x7f2c6006de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1484391940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60154af0 0x7f2c600cdc50 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.770+0000 7f2c669e4640 1 -- 172.21.15.77:0/1484391940 >> 172.21.15.77:0/1484391940 conn(0x7f2c6006b160 msgr2=0x7f2c6006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.771+0000 7f2c669e4640 1 -- 172.21.15.77:0/1484391940 shutdown_connections 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.771+0000 7f2c669e4640 1 -- 172.21.15.77:0/1484391940 wait complete. 2024-03-06T07:26:44.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.772+0000 7f2c669e4640 1 Processor -- start 2024-03-06T07:26:44.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.772+0000 7f2c669e4640 1 -- start start 2024-03-06T07:26:44.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.773+0000 7f2c669e4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c6006d9e0 0x7f2c6014e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.773+0000 7f2c669e4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60154af0 0x7f2c60152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.773+0000 7f2c669e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6015abc0 0x7f2c60156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.773+0000 7f2c669e4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6015b680 con 0x7f2c6015abc0 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.773+0000 7f2c669e4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6006e360 con 0x7f2c6006d9e0 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.773+0000 7f2c669e4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6015b960 con 0x7f2c60154af0 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.773+0000 7f2c64f81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6015abc0 0x7f2c60156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c64f81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6015abc0 0x7f2c60156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33316/0 (socket says 172.21.15.77:33316) 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c64f81640 1 -- 172.21.15.77:0/1092931570 learned_addr learned my addr 172.21.15.77:0/1092931570 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c5f7fe640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60154af0 0x7f2c60152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c5ffff640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c6006d9e0 0x7f2c6014e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c64f81640 1 -- 172.21.15.77:0/1092931570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60154af0 msgr2=0x7f2c60152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c64f81640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60154af0 0x7f2c60152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c64f81640 1 -- 172.21.15.77:0/1092931570 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c6006d9e0 msgr2=0x7f2c6014e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c64f81640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c6006d9e0 0x7f2c6014e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c64f81640 1 -- 172.21.15.77:0/1092931570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c601554d0 con 0x7f2c6015abc0 2024-03-06T07:26:44.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c64f81640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6015abc0 0x7f2c60156570 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800db90 tx=0x7f2c5800dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c5cff9640 1 -- 172.21.15.77:0/1092931570 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5800e920 con 0x7f2c6015abc0 2024-03-06T07:26:44.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c5cff9640 1 -- 172.21.15.77:0/1092931570 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5800eae0 con 0x7f2c6015abc0 2024-03-06T07:26:44.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c5cff9640 1 -- 172.21.15.77:0/1092931570 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58009580 con 0x7f2c6015abc0 2024-03-06T07:26:44.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c601557b0 con 0x7f2c6015abc0 2024-03-06T07:26:44.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.774+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c60155d70 con 0x7f2c6015abc0 2024-03-06T07:26:44.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.775+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c60069ee0 con 0x7f2c6015abc0 2024-03-06T07:26:44.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.776+0000 7f2c5cff9640 1 -- 172.21.15.77:0/1092931570 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c58009720 con 0x7f2c6015abc0 2024-03-06T07:26:44.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.776+0000 7f2c5cff9640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c4c06b4d0 0x7f2c4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.776+0000 7f2c5cff9640 1 -- 172.21.15.77:0/1092931570 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2c5808be90 con 0x7f2c6015abc0 2024-03-06T07:26:44.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.776+0000 7f2c5ffff640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c4c06b4d0 0x7f2c4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.776+0000 7f2c5ffff640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c4c06b4d0 0x7f2c4c06d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2c50036000 tx=0x7f2c5003e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:44.784+0000 7f2c5cff9640 1 -- 172.21.15.77:0/1092931570 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c5805a5c0 con 0x7f2c6015abc0 2024-03-06T07:26:45.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.086+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c60072370 con 0x7f2c6015abc0 2024-03-06T07:26:45.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.087+0000 7f2c5cff9640 1 -- 172.21.15.77:0/1092931570 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f2c5805e660 con 0x7f2c6015abc0 2024-03-06T07:26:45.088 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:45.089 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:45.089 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c4c06b4d0 msgr2=0x7f2c4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c4c06b4d0 0x7f2c4c06d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2c50036000 tx=0x7f2c5003e040 comp rx=0 tx=0).stop 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6015abc0 msgr2=0x7f2c60156570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6015abc0 0x7f2c60156570 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800db90 tx=0x7f2c5800dc80 comp rx=0 tx=0).stop 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 shutdown_connections 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c4c06b4d0 0x7f2c4c06d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6015abc0 0x7f2c60156570 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c60154af0 0x7f2c60152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 --2- 172.21.15.77:0/1092931570 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c6006d9e0 0x7f2c6014e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.090+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 >> 172.21.15.77:0/1092931570 conn(0x7f2c6006b160 msgr2=0x7f2c600b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.091+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 shutdown_connections 2024-03-06T07:26:45.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:45.091+0000 7f2c669e4640 1 -- 172.21.15.77:0/1092931570 wait complete. 2024-03-06T07:26:45.108 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:45.295 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1169,"msgr_send_messages":1165,"msgr_recv_bytes":436027,"msgr_send_bytes":5474491,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.116072076,"msgr_running_send_time":0.061102251,"msgr_running_recv_time":0.020383331,"msgr_running_fast_dispatch_time":0.028352860,"msgr_send_messages_queue_lat":{"avgcount":1164,"sum":1.255916464,"avgtime":0.001078966},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":108208,"msgr_send_encrypted_bytes":7904},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10981,"msgr_send_messages":24797,"msgr_recv_bytes":4250332,"msgr_send_bytes":24950130,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.076386471,"msgr_running_send_time":0.600736167,"msgr_running_recv_time":0.504108613,"msgr_running_fast_dispatch_time":0.039372636,"msgr_send_messages_queue_lat":{"avgcount":24796,"sum":0.533959862,"avgtime":0.000021534},"msgr_handle_ack_lat":{"avgcount":29656,"sum":0.004601009,"avgtime":0.000000155},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2182,"msgr_send_messages":2188,"msgr_recv_bytes":628078,"msgr_send_bytes":5988098,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.166725417,"msgr_running_send_time":0.069841689,"msgr_running_recv_time":0.031847916,"msgr_running_fast_dispatch_time":0.054280764,"msgr_send_messages_queue_lat":{"avgcount":2187,"sum":0.340613766,"avgtime":0.000155744},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":122,"sum":6.798218769,"avgtime":0.055723104}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4697,"sum":1.346658263,"avgtime":0.000286706}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.776799583,"avgtime":0.001525003},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4804,"inodes_top":0,"inodes_bottom":4804,"inodes_pin_tail":0,"inodes_pinned":4804,"inodes_expired":0,"inodes_with_caps":1063,"caps":1063,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":626,"ceph_cap_op_grant":7019,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3200,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12825,"evex":11232,"evtrm":11232,"ev":1596,"evexg":0,"evexd":11231,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":27881743,"wrpos":28842476,"rdpos":4195244,"jlat":{"avgcount":23,"sum":3.384360348,"avgtime":0.147146102},"replayed":3},"mds_mem":{"ino":1606,"ino+":3214,"ino-":1608,"dir":13,"dir+":13,"dir-":0,"dn":1604,"dn+":6412,"dn-":4808,"cap":2,"cap+":3202,"cap-":3200,"rss":116976,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.480993773,"avgtime":0.480993773},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.739273342,"avgtime":0.001793522},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5854,"average_load":5854,"avg_session_uptime":816,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":7738685,"buffer_anon_items":2814,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14140592,"mds_co_items":162408,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":4661,"op_send_bytes":25535877,"op_resend":0,"op_reply":4655,"op_latency":{"avgcount":4655,"sum":165.895367771,"avgtime":0.035638102},"op_inflight":5,"oplen_avg":{"avgcount":4661,"sum":7958},"op":4661,"op_r":57,"op_w":4604,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1604,"osdop_read":29,"osdop_write":44,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2909,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3208,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":52,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":2882,"pq_executed":2882,"pq_item_in_journal":318},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14328,"get_sum":4022312,"get_or_fail_fail":0,"get_or_fail_success":14329,"take":0,"take_sum":0,"put":14328,"put_sum":4022148,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4661,"take_sum":25860194,"put":4656,"put_sum":25860194,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":5,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4661,"take_sum":4661,"put":4656,"put_sum":4656,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3200,"get_sum":323200,"get_or_fail_fail":0,"get_or_fail_success":3200,"take":0,"take_sum":0,"put":21,"put_sum":323200,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":957052,"max":3758096384,"get_started":0,"get":12825,"get_sum":24646791,"get_or_fail_fail":0,"get_or_fail_success":12825,"take":0,"take_sum":0,"put":23,"put_sum":23689739,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:45.296 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 436027, "msgr_recv_encrypted_bytes": 108208, "msgr_recv_messages": 1169, "msgr_running_fast_dispatch_time": 0.02835286, "msgr_running_recv_time": 0.020383331, "msgr_running_send_time": 0.061102251, "msgr_running_total_time": 0.116072076, "msgr_send_bytes": 5474491, "msgr_send_encrypted_bytes": 7904, "msgr_send_messages": 1165, "msgr_send_messages_queue_lat": { "avgcount": 1164, "avgtime": 0.001078966, "sum": 1.255916464 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29656, "avgtime": 1.55e-07, "sum": 0.004601009 }, "msgr_recv_bytes": 4250332, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 10981, "msgr_running_fast_dispatch_time": 0.039372636, "msgr_running_recv_time": 0.504108613, "msgr_running_send_time": 0.600736167, "msgr_running_total_time": 1.076386471, "msgr_send_bytes": 24950130, "msgr_send_encrypted_bytes": 96256, "msgr_send_messages": 24797, "msgr_send_messages_queue_lat": { "avgcount": 24796, "avgtime": 2.1534e-05, "sum": 0.533959862 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 628078, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2182, "msgr_running_fast_dispatch_time": 0.054280764, "msgr_running_recv_time": 0.031847916, "msgr_running_send_time": 0.069841689, "msgr_running_total_time": 0.166725417, "msgr_send_bytes": 5988098, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2188, "msgr_send_messages_queue_lat": { "avgcount": 2187, "avgtime": 0.000155744, "sum": 0.340613766 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 122, "avgtime": 0.055723104, "sum": 6.798218769 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4697, "avgtime": 0.000286706, "sum": 1.346658263 }, "queue_len": 0 }, "mds": { "caps": 1063, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7019, "ceph_cap_op_revoke": 626, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4804, "inodes_bottom": 4804, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4804, "inodes_top": 0, "inodes_with_caps": 1063, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001525003, "sum": 9.776799583 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3200, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1596, "evadd": 12825, "evex": 11232, "evexd": 11231, "evexg": 0, "evlrg": 0, "evtrm": 11232, "expos": 27881743, "jlat": { "avgcount": 23, "avgtime": 0.147146102, "sum": 3.384360348 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28842476 }, "mds_mem": { "cap": 2, "cap+": 3202, "cap-": 3200, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1604, "dn+": 6412, "dn-": 4808, "heap": 248164, "ino": 1606, "ino+": 3214, "ino-": 1608, "rss": 116976 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.480993773, "sum": 0.480993773 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001793522, "sum": 5.739273342 } }, "mds_sessions": { "average_load": 5854, "avg_session_uptime": 816, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5854 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 7738685, "buffer_anon_items": 2814, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 14140592, "mds_co_items": 162408, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 52, "op": 4661, "op_Infinitylight": 5, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 4655, "avgtime": 0.035638102, "sum": 165.895367771 }, "op_pg": 0, "op_r": 57, "op_reply": 4655, "op_resend": 0, "op_rmw": 0, "op_send": 4661, "op_send_bytes": 25535877, "op_w": 4604, "oplen_avg": { "avgcount": 4661, "sum": 7958 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1604, "osdop_delete": 2909, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 29, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3208, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 44, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2882, "pq_executed_ops": 2882, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 318 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14328, "get_or_fail_fail": 0, "get_or_fail_success": 14329, "get_started": 0, "get_sum": 4022312, "max": 104857600, "put": 14328, "put_sum": 4022148, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4656, "put_sum": 25860194, "take": 4661, "take_sum": 25860194, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4656, "put_sum": 4656, "take": 4661, "take_sum": 4661, "val": 5, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3200, "get_or_fail_fail": 0, "get_or_fail_success": 3200, "get_started": 0, "get_sum": 323200, "max": 3758096384, "put": 21, "put_sum": 323200, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 12825, "get_or_fail_fail": 0, "get_or_fail_success": 12825, "get_started": 0, "get_sum": 24646791, "max": 3758096384, "put": 23, "put_sum": 23689739, "take": 0, "take_sum": 0, "val": 957052, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:45.296 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/1, 2882/3201 2024-03-06T07:26:46.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:46.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.396+0000 7fed3ffe9640 1 Processor -- start 2024-03-06T07:26:46.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3ffe9640 1 -- start start 2024-03-06T07:26:46.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3ffe9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fed3815e150 0x7fed380cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3ffe9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed380d1060 0x7fed380b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3ffe9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed380d1ea0 con 0x7fed380cfba0 2024-03-06T07:26:46.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3ffe9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed380d2ab0 con 0x7fed380d1060 2024-03-06T07:26:46.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3ffe9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed380d2180 con 0x7fed3815e150 2024-03-06T07:26:46.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3e586640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed380d1060 0x7fed380b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3e586640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed380d1060 0x7fed380b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33326/0 (socket says 172.21.15.77:33326) 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.397+0000 7fed3e586640 1 -- 172.21.15.77:0/3686586412 learned_addr learned my addr 172.21.15.77:0/3686586412 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.398+0000 7fed3dd85640 1 --2- 172.21.15.77:0/3686586412 >> v2:172.21.15.158:3300/0 conn(0x7fed3815e150 0x7fed380cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.398+0000 7fed3e586640 1 -- 172.21.15.77:0/3686586412 >> v2:172.21.15.158:3300/0 conn(0x7fed3815e150 msgr2=0x7fed380cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.398+0000 7fed3e586640 1 --2- 172.21.15.77:0/3686586412 >> v2:172.21.15.158:3300/0 conn(0x7fed3815e150 0x7fed380cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.398+0000 7fed3e586640 1 -- 172.21.15.77:0/3686586412 >> v1:172.21.15.158:6789/0 conn(0x7fed380cfba0 legacy=0x7fed380d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.398+0000 7fed3e586640 1 -- 172.21.15.77:0/3686586412 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed380d2460 con 0x7fed380d1060 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.398+0000 7fed3dd85640 1 --2- 172.21.15.77:0/3686586412 >> v2:172.21.15.158:3300/0 conn(0x7fed3815e150 0x7fed380cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.398+0000 7fed3e586640 1 --2- 172.21.15.77:0/3686586412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed380b9060 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fed380b6af0 tx=0x7fed3002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6779bed735dae188 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3cd83640 1 -- 172.21.15.77:0/3686586412 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3002c9f0 con 0x7fed380d1060 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3cd83640 1 -- 172.21.15.77:0/3686586412 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3002cb90 con 0x7fed380d1060 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3cd83640 1 -- 172.21.15.77:0/3686586412 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed300343f0 con 0x7fed380d1060 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3686586412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 msgr2=0x7fed380b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3686586412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed380b9060 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fed380b6af0 tx=0x7fed3002be50 comp rx=0 tx=0).stop 2024-03-06T07:26:46.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3686586412 shutdown_connections 2024-03-06T07:26:46.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3686586412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed380b9060 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3686586412 >> v2:172.21.15.158:3300/0 conn(0x7fed3815e150 0x7fed380cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3686586412 >> 172.21.15.77:0/3686586412 conn(0x7fed3809d230 msgr2=0x7fed380d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3686586412 shutdown_connections 2024-03-06T07:26:46.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.399+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3686586412 wait complete. 2024-03-06T07:26:46.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.400+0000 7fed3ffe9640 1 Processor -- start 2024-03-06T07:26:46.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.400+0000 7fed3ffe9640 1 -- start start 2024-03-06T07:26:46.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3ffe9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed380cfba0 0x7fed3814f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3ffe9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed3815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3ffe9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed3815e150 0x7fed3815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3ffe9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed380d1b30 con 0x7fed380d1060 2024-03-06T07:26:46.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3ffe9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed380d2f40 con 0x7fed380cfba0 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3ffe9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3815e530 con 0x7fed3815e150 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3d584640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed3815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3d584640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed3815a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33332/0 (socket says 172.21.15.77:33332) 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3d584640 1 -- 172.21.15.77:0/3327324823 learned_addr learned my addr 172.21.15.77:0/3327324823 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3dd85640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed380cfba0 0x7fed3814f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.401+0000 7fed3e586640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed3815e150 0x7fed3815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed3d584640 1 -- 172.21.15.77:0/3327324823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed3815e150 msgr2=0x7fed3815a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed3d584640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed3815e150 0x7fed3815a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed3d584640 1 -- 172.21.15.77:0/3327324823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed380cfba0 msgr2=0x7fed3814f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed3d584640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed380cfba0 0x7fed3814f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed3d584640 1 -- 172.21.15.77:0/3327324823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed380ce2b0 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed3d584640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed3815a360 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fed2800c430 tx=0x7fed2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed2e7fc640 1 -- 172.21.15.77:0/3327324823 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed28011790 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed2e7fc640 1 -- 172.21.15.77:0/3327324823 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed28017070 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3815af00 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed2e7fc640 1 -- 172.21.15.77:0/3327324823 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed28011bb0 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.402+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3815b4c0 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.403+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed38069ee0 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.404+0000 7fed2e7fc640 1 -- 172.21.15.77:0/3327324823 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed2801c460 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.404+0000 7fed2e7fc640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed1c06b4d0 0x7fed1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.404+0000 7fed2e7fc640 1 -- 172.21.15.77:0/3327324823 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fed28089ff0 con 0x7fed380d1060 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.404+0000 7fed3dd85640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed1c06b4d0 0x7fed1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.404+0000 7fed3dd85640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed1c06b4d0 0x7fed1c06d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fed3400aac0 tx=0x7fed34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.408+0000 7fed2e7fc640 1 -- 172.21.15.77:0/3327324823 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed280587b0 con 0x7fed380d1060 2024-03-06T07:26:46.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.627+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed380d4670 con 0x7fed380d1060 2024-03-06T07:26:46.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.628+0000 7fed2e7fc640 1 -- 172.21.15.77:0/3327324823 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fed2805c850 con 0x7fed380d1060 2024-03-06T07:26:46.629 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:46.630 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:46.630 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.629+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed1c06b4d0 msgr2=0x7fed1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.629+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed1c06b4d0 0x7fed1c06d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fed3400aac0 tx=0x7fed34008040 comp rx=0 tx=0).stop 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.629+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 msgr2=0x7fed3815a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.629+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed3815a360 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fed2800c430 tx=0x7fed2800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.629+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 shutdown_connections 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.629+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fed1c06b4d0 0x7fed1c06d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.630+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed380d1060 0x7fed3815a360 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.630+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fed3815e150 0x7fed3815a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.630+0000 7fed3ffe9640 1 --2- 172.21.15.77:0/3327324823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fed380cfba0 0x7fed3814f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.630+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 >> 172.21.15.77:0/3327324823 conn(0x7fed3809d230 msgr2=0x7fed3806d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.630+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 shutdown_connections 2024-03-06T07:26:46.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.630+0000 7fed3ffe9640 1 -- 172.21.15.77:0/3327324823 wait complete. 2024-03-06T07:26:46.642 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:46.759 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1242,"msgr_send_messages":1237,"msgr_recv_bytes":454587,"msgr_send_bytes":5498261,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.120469138,"msgr_running_send_time":0.062622518,"msgr_running_recv_time":0.021501707,"msgr_running_fast_dispatch_time":0.029748516,"msgr_send_messages_queue_lat":{"avgcount":1236,"sum":1.258069421,"avgtime":0.001017855},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":108480,"msgr_send_encrypted_bytes":8528},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11084,"msgr_send_messages":24896,"msgr_recv_bytes":4276813,"msgr_send_bytes":24982598,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.083068452,"msgr_running_send_time":0.603282730,"msgr_running_recv_time":0.505744759,"msgr_running_fast_dispatch_time":0.041330739,"msgr_send_messages_queue_lat":{"avgcount":24895,"sum":0.536286274,"avgtime":0.000021541},"msgr_handle_ack_lat":{"avgcount":29659,"sum":0.004601353,"avgtime":0.000000155},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2334,"msgr_send_messages":2336,"msgr_recv_bytes":666670,"msgr_send_bytes":6036510,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.175220743,"msgr_running_send_time":0.072703809,"msgr_running_recv_time":0.034078798,"msgr_running_fast_dispatch_time":0.057125389,"msgr_send_messages_queue_lat":{"avgcount":2335,"sum":0.344324847,"avgtime":0.000147462},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":123,"sum":6.798229662,"avgtime":0.055270159}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5248,"sum":1.362058933,"avgtime":0.000259538}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.776799583,"avgtime":0.001525003},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4804,"inodes_top":0,"inodes_bottom":4804,"inodes_pin_tail":0,"inodes_pinned":4804,"inodes_expired":0,"inodes_with_caps":1063,"caps":1063,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":626,"ceph_cap_op_grant":7019,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":9,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12826,"evex":11232,"evtrm":11232,"ev":1597,"evexg":0,"evexd":11231,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":27881743,"wrpos":28843075,"rdpos":4195244,"jlat":{"avgcount":23,"sum":3.384360348,"avgtime":0.147146102},"replayed":3},"mds_mem":{"ino":1606,"ino+":3214,"ino-":1608,"dir":13,"dir+":13,"dir-":0,"dn":1604,"dn+":6412,"dn-":4808,"cap":1,"cap+":3202,"cap-":3201,"rss":116976,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.480993773,"avgtime":0.480993773},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.739273342,"avgtime":0.001793522},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5854,"average_load":5854,"avg_session_uptime":816,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":7738685,"buffer_anon_items":2814,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14140592,"mds_co_items":162408,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4979,"op_send_bytes":25536233,"op_resend":0,"op_reply":4979,"op_latency":{"avgcount":4979,"sum":166.368357418,"avgtime":0.033414010},"op_inflight":0,"oplen_avg":{"avgcount":4979,"sum":8277},"op":4979,"op_r":58,"op_w":4921,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1604,"osdop_read":30,"osdop_write":45,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3208,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":53,"omap_rd":48,"omap_del":13},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14656,"get_sum":4076209,"get_or_fail_fail":0,"get_or_fail_success":14656,"take":0,"take_sum":0,"put":14656,"put_sum":4076209,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4979,"take_sum":25860651,"put":4979,"put_sum":25860651,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4979,"take_sum":4979,"put":4979,"put_sum":4979,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":22,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":957651,"max":3758096384,"get_started":0,"get":12826,"get_sum":24647390,"get_or_fail_fail":0,"get_or_fail_success":12826,"take":0,"take_sum":0,"put":23,"put_sum":23689739,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:46.760 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 454587, "msgr_recv_encrypted_bytes": 108480, "msgr_recv_messages": 1242, "msgr_running_fast_dispatch_time": 0.029748516, "msgr_running_recv_time": 0.021501707, "msgr_running_send_time": 0.062622518, "msgr_running_total_time": 0.120469138, "msgr_send_bytes": 5498261, "msgr_send_encrypted_bytes": 8528, "msgr_send_messages": 1237, "msgr_send_messages_queue_lat": { "avgcount": 1236, "avgtime": 0.001017855, "sum": 1.258069421 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29659, "avgtime": 1.55e-07, "sum": 0.004601353 }, "msgr_recv_bytes": 4276813, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 11084, "msgr_running_fast_dispatch_time": 0.041330739, "msgr_running_recv_time": 0.505744759, "msgr_running_send_time": 0.60328273, "msgr_running_total_time": 1.083068452, "msgr_send_bytes": 24982598, "msgr_send_encrypted_bytes": 96256, "msgr_send_messages": 24896, "msgr_send_messages_queue_lat": { "avgcount": 24895, "avgtime": 2.1541e-05, "sum": 0.536286274 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 666670, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2334, "msgr_running_fast_dispatch_time": 0.057125389, "msgr_running_recv_time": 0.034078798, "msgr_running_send_time": 0.072703809, "msgr_running_total_time": 0.175220743, "msgr_send_bytes": 6036510, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2336, "msgr_send_messages_queue_lat": { "avgcount": 2335, "avgtime": 0.000147462, "sum": 0.344324847 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 123, "avgtime": 0.055270159, "sum": 6.798229662 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5248, "avgtime": 0.000259538, "sum": 1.362058933 }, "queue_len": 0 }, "mds": { "caps": 1063, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7019, "ceph_cap_op_revoke": 626, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 9, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4804, "inodes_bottom": 4804, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4804, "inodes_top": 0, "inodes_with_caps": 1063, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001525003, "sum": 9.776799583 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1597, "evadd": 12826, "evex": 11232, "evexd": 11231, "evexg": 0, "evlrg": 0, "evtrm": 11232, "expos": 27881743, "jlat": { "avgcount": 23, "avgtime": 0.147146102, "sum": 3.384360348 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28843075 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1604, "dn+": 6412, "dn-": 4808, "heap": 248164, "ino": 1606, "ino+": 3214, "ino-": 1608, "rss": 116976 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.480993773, "sum": 0.480993773 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001793522, "sum": 5.739273342 } }, "mds_sessions": { "average_load": 5854, "avg_session_uptime": 816, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5854 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 7738685, "buffer_anon_items": 2814, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 14140592, "mds_co_items": 162408, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 13, "omap_rd": 48, "omap_wr": 53, "op": 4979, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4979, "avgtime": 0.03341401, "sum": 166.368357418 }, "op_pg": 0, "op_r": 58, "op_reply": 4979, "op_resend": 0, "op_rmw": 0, "op_send": 4979, "op_send_bytes": 25536233, "op_w": 4921, "oplen_avg": { "avgcount": 4979, "sum": 8277 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1604, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3208, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 45, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14656, "get_or_fail_fail": 0, "get_or_fail_success": 14656, "get_started": 0, "get_sum": 4076209, "max": 104857600, "put": 14656, "put_sum": 4076209, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4979, "put_sum": 25860651, "take": 4979, "take_sum": 25860651, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4979, "put_sum": 4979, "take": 4979, "take_sum": 4979, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 22, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 12826, "get_or_fail_fail": 0, "get_or_fail_success": 12826, "get_started": 0, "get_sum": 24647390, "max": 3758096384, "put": 23, "put_sum": 23689739, "take": 0, "take_sum": 0, "val": 957651, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:46.760 INFO:tasks.cephfs.test_strays:Complete purge in 7 seconds 2024-03-06T07:26:46.761 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:46.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.859+0000 7f134ca39640 1 Processor -- start 2024-03-06T07:26:46.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.859+0000 7f134ca39640 1 -- start start 2024-03-06T07:26:46.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f134ca39640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f134815e150 0x7f13480cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f134ca39640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13480cfba0 0x7f13480d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f134ca39640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13480d1ea0 con 0x7f13480d0e90 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f134ca39640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13480d2ab0 con 0x7f134815e150 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f134ca39640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13480d2180 con 0x7f13480cfba0 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f1346d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f13480d0e90 0x7f13480b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51480/0 (socket says 172.21.15.77:51480) 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f1346d9d640 1 -- 172.21.15.77:0/1092102 learned_addr learned my addr 172.21.15.77:0/1092102 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f134659c640 1 --2- 172.21.15.77:0/1092102 >> v2:172.21.15.158:3300/0 conn(0x7f134815e150 0x7f13480cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f133ffff640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13480cfba0 0x7f13480d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896869921 0 0) 0x7f13480d1ea0 con 0x7f13480d0e90 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.860+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1334003520 con 0x7f13480d0e90 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811194986 0 0) 0x7f1334003520 con 0x7f13480d0e90 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13480cfba0 msgr2=0x7f13480d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13480cfba0 0x7f13480d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 >> v2:172.21.15.158:3300/0 conn(0x7f134815e150 msgr2=0x7f13480cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 --2- 172.21.15.77:0/1092102 >> v2:172.21.15.158:3300/0 conn(0x7f134815e150 0x7f13480cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13480d2460 con 0x7f13480d0e90 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f13400019f0 con 0x7f13480d0e90 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 >> v1:172.21.15.77:6789/0 conn(0x7f13480d0e90 legacy=0x7f13480b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13340039b0 0x7f1334003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13480cfba0 0x7f1334004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1334004d40 0x7f13340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480d2ab0 con 0x7f1334004d40 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480d2180 con 0x7f13340039b0 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133400b9c0 con 0x7f13480cfba0 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1346d9d640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1334004d40 0x7f13340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f134659c640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13340039b0 0x7f1334003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f133ffff640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13480cfba0 0x7f1334004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1346d9d640 1 -- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13480cfba0 msgr2=0x7f1334004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1346d9d640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13480cfba0 0x7f1334004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1346d9d640 1 -- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13340039b0 msgr2=0x7f1334003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1346d9d640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13340039b0 0x7f1334003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.861+0000 7f1346d9d640 1 -- 172.21.15.77:0/1092102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1334003520 con 0x7f1334004d40 2024-03-06T07:26:46.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.862+0000 7f1346d9d640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1334004d40 0x7f13340051c0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f134000a870 tx=0x7f1334005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10dcd3bbe05ca26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.862+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134000a4c0 con 0x7f1334004d40 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.862+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1340034630 con 0x7f1334004d40 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.862+0000 7f1345d9b640 1 -- 172.21.15.77:0/1092102 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1340034a20 con 0x7f1334004d40 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.863+0000 7f134ca39640 1 -- 172.21.15.77:0/1092102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1334004d40 msgr2=0x7f13340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.863+0000 7f134ca39640 1 --2- 172.21.15.77:0/1092102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1334004d40 0x7f13340051c0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f134000a870 tx=0x7f1334005830 comp rx=0 tx=0).stop 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.863+0000 7f1346d9d640 1 -- 172.21.15.77:0/1092102 reap_dead start 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.863+0000 7f134ca39640 1 -- 172.21.15.77:0/1092102 shutdown_connections 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.863+0000 7f134ca39640 1 -- 172.21.15.77:0/1092102 >> 172.21.15.77:0/1092102 conn(0x7f134809d230 msgr2=0x7f13480d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.863+0000 7f134ca39640 1 -- 172.21.15.77:0/1092102 shutdown_connections 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.863+0000 7f134ca39640 1 -- 172.21.15.77:0/1092102 wait complete. 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.864+0000 7f134ca39640 1 Processor -- start 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.864+0000 7f134ca39640 1 -- start start 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f134ca39640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1348151210 0x7f13480b92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f134ca39640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13481515f0 0x7f13481f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f134ca39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13481f4410 0x7f13480c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f134ca39640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480d1b30 con 0x7f13481f4410 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f134ca39640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480d2f40 con 0x7f1348151210 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f134ca39640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134815e530 con 0x7f13481515f0 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f1346d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13481f4410 0x7f13480c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f1346d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13481f4410 0x7f13480c9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33338/0 (socket says 172.21.15.77:33338) 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f1346d9d640 1 -- 172.21.15.77:0/1358809687 learned_addr learned my addr 172.21.15.77:0/1358809687 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f134659c640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1348151210 0x7f13480b92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f1346d9d640 1 -- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13481515f0 msgr2=0x7f13481f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f133ffff640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13481515f0 0x7f13481f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f1346d9d640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13481515f0 0x7f13481f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f1346d9d640 1 -- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1348151210 msgr2=0x7f13480b92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f1346d9d640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1348151210 0x7f13480b92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f1346d9d640 1 -- 172.21.15.77:0/1358809687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13480b7e20 con 0x7f13481f4410 2024-03-06T07:26:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.865+0000 7f134659c640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1348151210 0x7f13480b92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.866+0000 7f1346d9d640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13481f4410 0x7f13480c9920 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f134000a0b0 tx=0x7f13480d2a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.866+0000 7f133effd640 1 -- 172.21.15.77:0/1358809687 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1340039040 con 0x7f13481f4410 2024-03-06T07:26:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.866+0000 7f133effd640 1 -- 172.21.15.77:0/1358809687 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1340045070 con 0x7f13481f4410 2024-03-06T07:26:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.866+0000 7f133effd640 1 -- 172.21.15.77:0/1358809687 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134000a4c0 con 0x7f13481f4410 2024-03-06T07:26:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.866+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13480c77e0 con 0x7f13481f4410 2024-03-06T07:26:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.866+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13480c7da0 con 0x7f13481f4410 2024-03-06T07:26:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.867+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1348069ee0 con 0x7f13481f4410 2024-03-06T07:26:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.868+0000 7f133effd640 1 -- 172.21.15.77:0/1358809687 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1340034a00 con 0x7f13481f4410 2024-03-06T07:26:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.868+0000 7f133effd640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f132806b4d0 0x7f132806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.868+0000 7f134659c640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f132806b4d0 0x7f132806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.868+0000 7f133effd640 1 -- 172.21.15.77:0/1358809687 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f13400ac6f0 con 0x7f13481f4410 2024-03-06T07:26:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.869+0000 7f134659c640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f132806b4d0 0x7f132806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1338009ab0 tx=0x7f1338008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:46.873+0000 7f133effd640 1 -- 172.21.15.77:0/1358809687 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f134007b500 con 0x7f13481f4410 2024-03-06T07:26:47.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.225+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13480d4670 con 0x7f13481f4410 2024-03-06T07:26:47.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.226+0000 7f133effd640 1 -- 172.21.15.77:0/1358809687 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f134007ae00 con 0x7f13481f4410 2024-03-06T07:26:47.229 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:47.229 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:47.229 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:47.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f132806b4d0 msgr2=0x7f132806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f132806b4d0 0x7f132806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1338009ab0 tx=0x7f1338008040 comp rx=0 tx=0).stop 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13481f4410 msgr2=0x7f13480c9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13481f4410 0x7f13480c9920 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f134000a0b0 tx=0x7f13480d2a20 comp rx=0 tx=0).stop 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 shutdown_connections 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f132806b4d0 0x7f132806d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13481f4410 0x7f13480c9920 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13481515f0 0x7f13481f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 --2- 172.21.15.77:0/1358809687 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1348151210 0x7f13480b92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.228+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 >> 172.21.15.77:0/1358809687 conn(0x7f134809d230 msgr2=0x7f134815ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.229+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 shutdown_connections 2024-03-06T07:26:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.229+0000 7f134ca39640 1 -- 172.21.15.77:0/1358809687 wait complete. 2024-03-06T07:26:47.242 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:47.382 INFO:teuthology.orchestra.run.smithi077.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1242,"msgr_send_messages":1237,"msgr_recv_bytes":454587,"msgr_send_bytes":5498261,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.120469138,"msgr_running_send_time":0.062622518,"msgr_running_recv_time":0.021501707,"msgr_running_fast_dispatch_time":0.029748516,"msgr_send_messages_queue_lat":{"avgcount":1236,"sum":1.258069421,"avgtime":0.001017855},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":108480,"msgr_send_encrypted_bytes":8528},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11084,"msgr_send_messages":24896,"msgr_recv_bytes":4276813,"msgr_send_bytes":24982598,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.083068452,"msgr_running_send_time":0.603282730,"msgr_running_recv_time":0.505744759,"msgr_running_fast_dispatch_time":0.041330739,"msgr_send_messages_queue_lat":{"avgcount":24895,"sum":0.536286274,"avgtime":0.000021541},"msgr_handle_ack_lat":{"avgcount":29659,"sum":0.004601353,"avgtime":0.000000155},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2335,"msgr_send_messages":2337,"msgr_recv_bytes":666916,"msgr_send_bytes":6994479,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.178883938,"msgr_running_send_time":0.075544440,"msgr_running_recv_time":0.034101176,"msgr_running_fast_dispatch_time":0.057919279,"msgr_send_messages_queue_lat":{"avgcount":2336,"sum":0.344646835,"avgtime":0.000147537},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":125,"sum":6.871560096,"avgtime":0.054972480}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5248,"sum":1.362058933,"avgtime":0.000259538}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.776799583,"avgtime":0.001525003},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":1604,"inodes_top":0,"inodes_bottom":1604,"inodes_pin_tail":0,"inodes_pinned":1604,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":626,"ceph_cap_op_grant":7019,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3206,"handle_client_caps_dirty":3000,"handle_client_cap_release":9,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12826,"evex":11232,"evtrm":11232,"ev":1597,"evexg":0,"evexd":11231,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":27881743,"wrpos":28843075,"rdpos":4195244,"jlat":{"avgcount":24,"sum":3.392946531,"avgtime":0.141372772},"replayed":3},"mds_mem":{"ino":1606,"ino+":3214,"ino-":1608,"dir":12,"dir+":13,"dir-":1,"dn":1604,"dn+":6412,"dn-":4808,"cap":1,"cap+":3202,"cap-":3201,"rss":116976,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.308638696,"avgtime":0.001033949},"req_getattr_latency":{"avgcount":6,"sum":0.001667035,"avgtime":0.000277839},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002687298,"avgtime":0.002687298},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.243539439,"avgtime":0.081179813},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.480993773,"avgtime":0.480993773},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.739273342,"avgtime":0.001793522},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5854,"average_load":5854,"avg_session_uptime":821,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13074073,"buffer_anon_items":3287,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":5098059,"mds_co_items":84559,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4980,"op_send_bytes":26493884,"op_resend":0,"op_reply":4980,"op_latency":{"avgcount":4980,"sum":166.376357243,"avgtime":0.033408907},"op_inflight":0,"oplen_avg":{"avgcount":4980,"sum":8278},"op":4980,"op_r":58,"op_w":4922,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1604,"osdop_read":30,"osdop_write":46,"osdop_writefull":17,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3208,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":53,"omap_rd":48,"omap_del":13},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14657,"get_sum":4076365,"get_or_fail_fail":0,"get_or_fail_success":14657,"take":0,"take_sum":0,"put":14657,"put_sum":4076365,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4980,"take_sum":26818302,"put":4980,"put_sum":26818302,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4980,"take_sum":4980,"put":4980,"put_sum":4980,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":22,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5638b7270a00":{"val":0,"max":3758096384,"get_started":0,"get":12826,"get_sum":24647390,"get_or_fail_fail":0,"get_or_fail_success":12826,"take":0,"take_sum":0,"put":24,"put_sum":24647390,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:47.384 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 454587, "msgr_recv_encrypted_bytes": 108480, "msgr_recv_messages": 1242, "msgr_running_fast_dispatch_time": 0.029748516, "msgr_running_recv_time": 0.021501707, "msgr_running_send_time": 0.062622518, "msgr_running_total_time": 0.120469138, "msgr_send_bytes": 5498261, "msgr_send_encrypted_bytes": 8528, "msgr_send_messages": 1237, "msgr_send_messages_queue_lat": { "avgcount": 1236, "avgtime": 0.001017855, "sum": 1.258069421 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 29659, "avgtime": 1.55e-07, "sum": 0.004601353 }, "msgr_recv_bytes": 4276813, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 11084, "msgr_running_fast_dispatch_time": 0.041330739, "msgr_running_recv_time": 0.505744759, "msgr_running_send_time": 0.60328273, "msgr_running_total_time": 1.083068452, "msgr_send_bytes": 24982598, "msgr_send_encrypted_bytes": 96256, "msgr_send_messages": 24896, "msgr_send_messages_queue_lat": { "avgcount": 24895, "avgtime": 2.1541e-05, "sum": 0.536286274 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 666916, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2335, "msgr_running_fast_dispatch_time": 0.057919279, "msgr_running_recv_time": 0.034101176, "msgr_running_send_time": 0.07554444, "msgr_running_total_time": 0.178883938, "msgr_send_bytes": 6994479, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2337, "msgr_send_messages_queue_lat": { "avgcount": 2336, "avgtime": 0.000147537, "sum": 0.344646835 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 125, "avgtime": 0.05497248, "sum": 6.871560096 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5248, "avgtime": 0.000259538, "sum": 1.362058933 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7019, "ceph_cap_op_revoke": 626, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 9, "handle_client_caps": 3206, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1604, "inodes_bottom": 1604, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 1604, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001525003, "sum": 9.776799583 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1597, "evadd": 12826, "evex": 11232, "evexd": 11231, "evexg": 0, "evlrg": 0, "evtrm": 11232, "expos": 27881743, "jlat": { "avgcount": 24, "avgtime": 0.141372772, "sum": 3.392946531 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28843075 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 1604, "dn+": 6412, "dn-": 4808, "heap": 248164, "ino": 1606, "ino+": 3214, "ino-": 1608, "rss": 116976 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001033949, "sum": 3.308638696 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000277839, "sum": 0.001667035 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002687298, "sum": 0.002687298 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.081179813, "sum": 0.243539439 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.480993773, "sum": 0.480993773 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001793522, "sum": 5.739273342 } }, "mds_sessions": { "average_load": 5854, "avg_session_uptime": 821, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5854 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13074073, "buffer_anon_items": 3287, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 5098059, "mds_co_items": 84559, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 13, "omap_rd": 48, "omap_wr": 53, "op": 4980, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4980, "avgtime": 0.033408907, "sum": 166.376357243 }, "op_pg": 0, "op_r": 58, "op_reply": 4980, "op_resend": 0, "op_rmw": 0, "op_send": 4980, "op_send_bytes": 26493884, "op_w": 4922, "oplen_avg": { "avgcount": 4980, "sum": 8278 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1604, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3208, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 46, "osdop_writefull": 17, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14657, "get_or_fail_fail": 0, "get_or_fail_success": 14657, "get_started": 0, "get_sum": 4076365, "max": 104857600, "put": 14657, "put_sum": 4076365, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4980, "put_sum": 26818302, "take": 4980, "take_sum": 26818302, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4980, "put_sum": 4980, "take": 4980, "take_sum": 4980, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 22, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5638b7270a00": { "get": 12826, "get_or_fail_fail": 0, "get_or_fail_success": 12826, "get_started": 0, "get_sum": 24647390, "max": 3758096384, "put": 24, "put_sum": 24647390, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:47.385 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:47.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:47.424 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:47.485 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:47.485 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:26:47.486 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:47.501 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:47.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.650+0000 7fa0b7102640 1 Processor -- start 2024-03-06T07:26:47.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.650+0000 7fa0b7102640 1 -- start start 2024-03-06T07:26:47.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0b7102640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa0b00d0db0 0x7fa0b00b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0b7102640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00d1ea0 con 0x7fa0b00d0950 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0b7102640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00d2180 con 0x7fa0b015e150 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0b7102640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b00d2ab0 con 0x7fa0b00d0db0 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0affff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa0b00d0950 0x7fa0b00b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51492/0 (socket says 172.21.15.77:51492) 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0affff640 1 -- 172.21.15.77:0/372089964 learned_addr learned my addr 172.21.15.77:0/372089964 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612776304 0 0) 0x7fa0b00d1ea0 con 0x7fa0b00d0950 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa09c003520 con 0x7fa0b00d0950 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0b569f640 1 --2- 172.21.15.77:0/372089964 >> v2:172.21.15.158:3300/0 conn(0x7fa0b00d0db0 0x7fa0b00b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1099484583 0 0) 0x7fa09c003520 con 0x7fa0b00d0950 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 >> v2:172.21.15.158:3300/0 conn(0x7fa0b00d0db0 msgr2=0x7fa0b00b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/372089964 >> v2:172.21.15.158:3300/0 conn(0x7fa0b00d0db0 0x7fa0b00b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 >> v1:172.21.15.158:6789/0 conn(0x7fa0b015e150 legacy=0x7fa0b00cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.651+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00d2460 con 0x7fa0b00d0950 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fa098001670 con 0x7fa0b00d0950 2024-03-06T07:26:47.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 >> v1:172.21.15.77:6789/0 conn(0x7fa0b00d0950 legacy=0x7fa0b00b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0039b0 0x7fa09c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c004800 0x7fa09c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09c0051c0 0x7fa09c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00d2ab0 con 0x7fa09c0051c0 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c00c160 con 0x7fa09c004800 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c00c360 con 0x7fa09c0039b0 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0b569f640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09c0051c0 0x7fa09c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0b4e9e640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0039b0 0x7fa09c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0affff640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c004800 0x7fa09c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0b569f640 1 -- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0039b0 msgr2=0x7fa09c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0b569f640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0039b0 0x7fa09c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0b569f640 1 -- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c004800 msgr2=0x7fa09c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0b569f640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa09c004800 0x7fa09c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0b569f640 1 -- 172.21.15.77:0/372089964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c003520 con 0x7fa09c0051c0 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.652+0000 7fa0b4e9e640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0039b0 0x7fa09c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:47.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b569f640 1 -- 172.21.15.77:0/372089964 reap_dead start 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b569f640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09c0051c0 0x7fa09c00b9c0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009ac0 tx=0x7fa09c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c34081b67311b17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0009690 con 0x7fa09c0051c0 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a0004020 con 0x7fa09c0051c0 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0af7fe640 1 -- 172.21.15.77:0/372089964 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0004410 con 0x7fa09c0051c0 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b7102640 1 -- 172.21.15.77:0/372089964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09c0051c0 msgr2=0x7fa09c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b7102640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09c0051c0 0x7fa09c00b9c0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009ac0 tx=0x7fa09c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b7102640 1 -- 172.21.15.77:0/372089964 shutdown_connections 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b7102640 1 --2- 172.21.15.77:0/372089964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa09c0051c0 0x7fa09c00b9c0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b7102640 1 -- 172.21.15.77:0/372089964 >> 172.21.15.77:0/372089964 conn(0x7fa0b009d230 msgr2=0x7fa0b00d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b7102640 1 -- 172.21.15.77:0/372089964 shutdown_connections 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.653+0000 7fa0b7102640 1 -- 172.21.15.77:0/372089964 wait complete. 2024-03-06T07:26:47.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.655+0000 7fa0b7102640 1 Processor -- start 2024-03-06T07:26:47.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.655+0000 7fa0b7102640 1 -- start start 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.655+0000 7fa0b7102640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0051c0 0x7fa0b01f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.655+0000 7fa0b7102640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c8020 0x7fa0b00c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.655+0000 7fa0b7102640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00c8480 0x7fa0b00ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.655+0000 7fa0b7102640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00d2f40 con 0x7fa0b00c8480 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.655+0000 7fa0b7102640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00d1220 con 0x7fa0b00c8020 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.655+0000 7fa0b7102640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00d1b30 con 0x7fa09c0051c0 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00c8480 0x7fa0b00ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00c8480 0x7fa0b00ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33350/0 (socket says 172.21.15.77:33350) 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 -- 172.21.15.77:0/3912929468 learned_addr learned my addr 172.21.15.77:0/3912929468 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b4e9e640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0051c0 0x7fa0b01f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0affff640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c8020 0x7fa0b00c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 -- 172.21.15.77:0/3912929468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0051c0 msgr2=0x7fa0b01f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0051c0 0x7fa0b01f71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 -- 172.21.15.77:0/3912929468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c8020 msgr2=0x7fa0b00c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c8020 0x7fa0b00c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 -- 172.21.15.77:0/3912929468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b01f76e0 con 0x7fa0b00c8480 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b569f640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00c8480 0x7fa0b00ca0b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fa0a002c380 tx=0x7fa0a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0ad7fa640 1 -- 172.21.15.77:0/3912929468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a002cbd0 con 0x7fa0b00c8480 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0ad7fa640 1 -- 172.21.15.77:0/3912929468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a002cd90 con 0x7fa0b00c8480 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b00c8a00 con 0x7fa0b00c8480 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0ad7fa640 1 -- 172.21.15.77:0/3912929468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0003850 con 0x7fa0b00c8480 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.656+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b00c8fc0 con 0x7fa0b00c8480 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.657+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b0069ee0 con 0x7fa0b00c8480 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.658+0000 7fa0ad7fa640 1 -- 172.21.15.77:0/3912929468 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0a00039f0 con 0x7fa0b00c8480 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.658+0000 7fa0ad7fa640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09006b4d0 0x7fa09006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.658+0000 7fa0ad7fa640 1 -- 172.21.15.77:0/3912929468 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa0a00ab010 con 0x7fa0b00c8480 2024-03-06T07:26:47.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.658+0000 7fa0b4e9e640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09006b4d0 0x7fa09006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.659+0000 7fa0b4e9e640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09006b4d0 0x7fa09006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8000f80 tx=0x7fa0a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.662+0000 7fa0ad7fa640 1 -- 172.21.15.77:0/3912929468 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0a0079740 con 0x7fa0b00c8480 2024-03-06T07:26:47.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.844+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0b00b4de0 con 0x7fa0b00c8480 2024-03-06T07:26:47.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.845+0000 7fa0ad7fa640 1 -- 172.21.15.77:0/3912929468 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16985 (secure 0 0 0) 0x7fa0a007d7e0 con 0x7fa0b00c8480 2024-03-06T07:26:47.848 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":96,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:21.753221+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:07.372376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:08.123063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6835/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.158:6835/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6834/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.77:6837/3425851566":"2024-03-07T07:26:21.715277+0000","172.21.15.77:6836/3425851566":"2024-03-07T07:26:21.715277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:47.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09006b4d0 msgr2=0x7fa09006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09006b4d0 0x7fa09006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8000f80 tx=0x7fa0a8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:47.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00c8480 msgr2=0x7fa0b00ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00c8480 0x7fa0b00ca0b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fa0a002c380 tx=0x7fa0a0004730 comp rx=0 tx=0).stop 2024-03-06T07:26:47.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 shutdown_connections 2024-03-06T07:26:47.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09006b4d0 0x7fa09006d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00c8480 0x7fa0b00ca0b0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c8020 0x7fa0b00c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 --2- 172.21.15.77:0/3912929468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa09c0051c0 0x7fa0b01f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 >> 172.21.15.77:0/3912929468 conn(0x7fa0b009d230 msgr2=0x7fa0b0160d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 shutdown_connections 2024-03-06T07:26:47.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.847+0000 7fa0b7102640 1 -- 172.21.15.77:0/3912929468 wait complete. 2024-03-06T07:26:47.861 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:07.372376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:08.123063+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:47.861 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.962+0000 7fb4722bb640 1 Processor -- start 2024-03-06T07:26:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.962+0000 7fb4722bb640 1 -- start start 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.963+0000 7fb4722bb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0b3b80 0x7fb46c071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.963+0000 7fb4722bb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46c06edc0 con 0x7fb46c16ae90 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.963+0000 7fb4722bb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46c0b30a0 con 0x7fb46c16aab0 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.963+0000 7fb4722bb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c08b5d0 con 0x7fb46c0b3b80 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.963+0000 7fb470858640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb46c16ae90 0x7fb46c0cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51494/0 (socket says 172.21.15.77:51494) 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.963+0000 7fb470858640 1 -- 172.21.15.77:0/1086269667 learned_addr learned my addr 172.21.15.77:0/1086269667 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.963+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068381499 0 0) 0x7fb46c06edc0 con 0x7fb46c16ae90 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:47.963+0000 7fb46b7fe640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0b3b80 0x7fb46c071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb454003970 con 0x7fb46c16ae90 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797759968 0 0) 0x7fb46c0b30a0 con 0x7fb46c16aab0 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46c06edc0 con 0x7fb46c16aab0 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1634334978 0 0) 0x7fb454003970 con 0x7fb46c16ae90 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0b3b80 msgr2=0x7fb46c071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0b3b80 0x7fb46c071d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 >> v1:172.21.15.158:6789/0 conn(0x7fb46c16aab0 legacy=0x7fb46c06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c0b3380 con 0x7fb46c16ae90 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fb4640019f0 con 0x7fb46c16ae90 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 >> v1:172.21.15.77:6789/0 conn(0x7fb46c16ae90 legacy=0x7fb46c0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb454003e30 0x7fb454004230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0b3b80 0x7fb454006c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb454004800 0x7fb454007220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb454007a70 con 0x7fb454004800 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb454007c20 con 0x7fb454003e30 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4540103a0 con 0x7fb46c0b3b80 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb470858640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb454004800 0x7fb454007220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46bfff640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb454003e30 0x7fb454004230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.037+0000 7fb46b7fe640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0b3b80 0x7fb454006c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.038+0000 7fb470858640 1 -- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0b3b80 msgr2=0x7fb454006c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.038+0000 7fb470858640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0b3b80 0x7fb454006c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.038+0000 7fb470858640 1 -- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb454003e30 msgr2=0x7fb454004230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.038+0000 7fb470858640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb454003e30 0x7fb454004230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.038+0000 7fb470858640 1 -- 172.21.15.77:0/1086269667 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb454003970 con 0x7fb454004800 2024-03-06T07:26:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.038+0000 7fb470858640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb454004800 0x7fb454007220 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb46400a7c0 tx=0x7fb454003540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2ffc7a0a14260d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46400a410 con 0x7fb454004800 2024-03-06T07:26:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb464034650 con 0x7fb454004800 2024-03-06T07:26:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb46affd640 1 -- 172.21.15.77:0/1086269667 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464034a40 con 0x7fb454004800 2024-03-06T07:26:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb4722bb640 1 -- 172.21.15.77:0/1086269667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb454004800 msgr2=0x7fb454007220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb4722bb640 1 --2- 172.21.15.77:0/1086269667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb454004800 0x7fb454007220 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb46400a7c0 tx=0x7fb454003540 comp rx=0 tx=0).stop 2024-03-06T07:26:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb470858640 1 -- 172.21.15.77:0/1086269667 reap_dead start 2024-03-06T07:26:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb4722bb640 1 -- 172.21.15.77:0/1086269667 shutdown_connections 2024-03-06T07:26:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb4722bb640 1 -- 172.21.15.77:0/1086269667 >> 172.21.15.77:0/1086269667 conn(0x7fb46c0b1b50 msgr2=0x7fb46c0b1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb4722bb640 1 -- 172.21.15.77:0/1086269667 shutdown_connections 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.039+0000 7fb4722bb640 1 -- 172.21.15.77:0/1086269667 wait complete. 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.040+0000 7fb4722bb640 1 Processor -- start 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb4722bb640 1 -- start start 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb4722bb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb46c0c9710 0x7fb46c0cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb4722bb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46c0c9af0 0x7fb46c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb4722bb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0cb5b0 0x7fb46c0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb4722bb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c06e960 con 0x7fb46c0c9710 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb4722bb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0b2e50 con 0x7fb46c0c9af0 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb4722bb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c06eae0 con 0x7fb46c0cb5b0 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb46bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb46c0c9710 0x7fb46c0cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb46bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb46c0c9710 0x7fb46c0cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33366/0 (socket says 172.21.15.77:33366) 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.041+0000 7fb46bfff640 1 -- 172.21.15.77:0/525177076 learned_addr learned my addr 172.21.15.77:0/525177076 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb470858640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0cb5b0 0x7fb46c0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb46b7fe640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46c0c9af0 0x7fb46c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb46bfff640 1 -- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0cb5b0 msgr2=0x7fb46c0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb46bfff640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0cb5b0 0x7fb46c0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb46bfff640 1 -- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46c0c9af0 msgr2=0x7fb46c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb46bfff640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46c0c9af0 0x7fb46c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb46bfff640 1 -- 172.21.15.77:0/525177076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c158300 con 0x7fb46c0c9710 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb470858640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0cb5b0 0x7fb46c0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb46b7fe640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46c0c9af0 0x7fb46c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb46bfff640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb46c0c9710 0x7fb46c0cedb0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb46000f7d0 tx=0x7fb46000fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb468ff9640 1 -- 172.21.15.77:0/525177076 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46000fe80 con 0x7fb46c0c9710 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb468ff9640 1 -- 172.21.15.77:0/525177076 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb460018070 con 0x7fb46c0c9710 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb468ff9640 1 -- 172.21.15.77:0/525177076 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46000d6d0 con 0x7fb46c0c9710 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c1585e0 con 0x7fb46c0c9710 2024-03-06T07:26:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.042+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c158ba0 con 0x7fb46c0c9710 2024-03-06T07:26:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.043+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c069ee0 con 0x7fb46c0c9710 2024-03-06T07:26:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.044+0000 7fb468ff9640 1 -- 172.21.15.77:0/525177076 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb46000d870 con 0x7fb46c0c9710 2024-03-06T07:26:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.044+0000 7fb468ff9640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45806b4d0 0x7fb45806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.044+0000 7fb468ff9640 1 -- 172.21.15.77:0/525177076 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb46008a2a0 con 0x7fb46c0c9710 2024-03-06T07:26:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.044+0000 7fb46b7fe640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45806b4d0 0x7fb45806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.045+0000 7fb46b7fe640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45806b4d0 0x7fb45806d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb46c203d70 tx=0x7fb45c00f040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.049+0000 7fb468ff9640 1 -- 172.21.15.77:0/525177076 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb460059020 con 0x7fb46c0c9710 2024-03-06T07:26:48.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.228+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb46c158d70 con 0x7fb46c0c9710 2024-03-06T07:26:48.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.228+0000 7fb468ff9640 1 -- 172.21.15.77:0/525177076 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb46008a0c0 con 0x7fb46c0c9710 2024-03-06T07:26:48.231 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:26:48.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45806b4d0 msgr2=0x7fb45806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45806b4d0 0x7fb45806d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb46c203d70 tx=0x7fb45c00f040 comp rx=0 tx=0).stop 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb46c0c9710 msgr2=0x7fb46c0cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb46c0c9710 0x7fb46c0cedb0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb46000f7d0 tx=0x7fb46000fc90 comp rx=0 tx=0).stop 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 shutdown_connections 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb46c0cb5b0 0x7fb46c0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46c0c9af0 0x7fb46c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb45806b4d0 0x7fb45806d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 --2- 172.21.15.77:0/525177076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb46c0c9710 0x7fb46c0cedb0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.230+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 >> 172.21.15.77:0/525177076 conn(0x7fb46c0b1b50 msgr2=0x7fb46c071600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.231+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 shutdown_connections 2024-03-06T07:26:48.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.231+0000 7fb4722bb640 1 -- 172.21.15.77:0/525177076 wait complete. 2024-03-06T07:26:48.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea5dcfb640 1 Processor -- start 2024-03-06T07:26:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea5dcfb640 1 -- start start 2024-03-06T07:26:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea5dcfb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea58161a80 0x7fea580d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea5dcfb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea580d25a0 0x7fea580d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea5dcfb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea58161e60 con 0x7fea580b5010 2024-03-06T07:26:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea5dcfb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea580b1250 con 0x7fea58161a80 2024-03-06T07:26:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea5dcfb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea580b3e30 con 0x7fea580d25a0 2024-03-06T07:26:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea577fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea58161a80 0x7fea580d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea577fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea58161a80 0x7fea580d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33374/0 (socket says 172.21.15.77:33374) 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.347+0000 7fea577fe640 1 -- 172.21.15.77:0/773297597 learned_addr learned my addr 172.21.15.77:0/773297597 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.348+0000 7fea577fe640 1 -- 172.21.15.77:0/773297597 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea580d25a0 msgr2=0x7fea580d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.348+0000 7fea577fe640 1 --2- 172.21.15.77:0/773297597 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea580d25a0 0x7fea580d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.348+0000 7fea577fe640 1 -- 172.21.15.77:0/773297597 >> v1:172.21.15.158:6789/0 conn(0x7fea580b5010 legacy=0x7fea580b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.348+0000 7fea577fe640 1 -- 172.21.15.77:0/773297597 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580d1470 con 0x7fea58161a80 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.348+0000 7fea577fe640 1 --2- 172.21.15.77:0/773297597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea58161a80 0x7fea580d1da0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fea50009e50 tx=0x7fea5002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d3eee33a84f065 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea567fc640 1 -- 172.21.15.77:0/773297597 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5002c9e0 con 0x7fea58161a80 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea567fc640 1 -- 172.21.15.77:0/773297597 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5002cb80 con 0x7fea58161a80 2024-03-06T07:26:48.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea567fc640 1 -- 172.21.15.77:0/773297597 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea500343f0 con 0x7fea58161a80 2024-03-06T07:26:48.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea5dcfb640 1 -- 172.21.15.77:0/773297597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea58161a80 msgr2=0x7fea580d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/773297597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea58161a80 0x7fea580d1da0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fea50009e50 tx=0x7fea5002be40 comp rx=0 tx=0).stop 2024-03-06T07:26:48.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea5dcfb640 1 -- 172.21.15.77:0/773297597 shutdown_connections 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/773297597 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea580d25a0 0x7fea580d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/773297597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea58161a80 0x7fea580d1da0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea5dcfb640 1 -- 172.21.15.77:0/773297597 >> 172.21.15.77:0/773297597 conn(0x7fea580d3fc0 msgr2=0x7fea580d43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea5dcfb640 1 -- 172.21.15.77:0/773297597 shutdown_connections 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.349+0000 7fea5dcfb640 1 -- 172.21.15.77:0/773297597 wait complete. 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.350+0000 7fea5dcfb640 1 Processor -- start 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.350+0000 7fea5dcfb640 1 -- start start 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea5dcfb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea580b5010 0x7fea580a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea5dcfb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea580d25a0 0x7fea581514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea5dcfb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea58161a80 0x7fea580b8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea5dcfb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea580d1090 con 0x7fea580b5010 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea5dcfb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea580b3ac0 con 0x7fea580d25a0 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea5dcfb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea580d1210 con 0x7fea58161a80 2024-03-06T07:26:48.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea580b5010 0x7fea580a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea580b5010 0x7fea580a1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33382/0 (socket says 172.21.15.77:33382) 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea577fe640 1 -- 172.21.15.77:0/2433919455 learned_addr learned my addr 172.21.15.77:0/2433919455 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.351+0000 7fea56ffd640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea580d25a0 0x7fea581514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea57fff640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea58161a80 0x7fea580b8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea577fe640 1 -- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea58161a80 msgr2=0x7fea580b8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea577fe640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea58161a80 0x7fea580b8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea577fe640 1 -- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea580d25a0 msgr2=0x7fea581514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea577fe640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea580d25a0 0x7fea581514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea577fe640 1 -- 172.21.15.77:0/2433919455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580a03d0 con 0x7fea580b5010 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea56ffd640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea580d25a0 0x7fea581514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea577fe640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea580b5010 0x7fea580a1470 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fea50036040 tx=0x7fea50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea4ffff640 1 -- 172.21.15.77:0/2433919455 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50043020 con 0x7fea580b5010 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea580a06b0 con 0x7fea580b5010 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea4ffff640 1 -- 172.21.15.77:0/2433919455 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5003e450 con 0x7fea580b5010 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea4ffff640 1 -- 172.21.15.77:0/2433919455 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5003e840 con 0x7fea580b5010 2024-03-06T07:26:48.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.352+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea580a0c70 con 0x7fea580b5010 2024-03-06T07:26:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.353+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea580d34c0 con 0x7fea580b5010 2024-03-06T07:26:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.354+0000 7fea4ffff640 1 -- 172.21.15.77:0/2433919455 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea50046020 con 0x7fea580b5010 2024-03-06T07:26:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.354+0000 7fea4ffff640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea3806b4d0 0x7fea3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.354+0000 7fea4ffff640 1 -- 172.21.15.77:0/2433919455 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fea500acc90 con 0x7fea580b5010 2024-03-06T07:26:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.354+0000 7fea56ffd640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea3806b4d0 0x7fea3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.354+0000 7fea56ffd640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea3806b4d0 0x7fea3806d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fea40007fd0 tx=0x7fea40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.359+0000 7fea4ffff640 1 -- 172.21.15.77:0/2433919455 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea580d34c0 con 0x7fea580b5010 2024-03-06T07:26:48.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.620+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fea58068050 con 0x7fea580b5010 2024-03-06T07:26:48.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.621+0000 7fea4ffff640 1 -- 172.21.15.77:0/2433919455 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7fea5004d030 con 0x7fea580b5010 2024-03-06T07:26:48.624 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea3806b4d0 msgr2=0x7fea3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea3806b4d0 0x7fea3806d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fea40007fd0 tx=0x7fea40009040 comp rx=0 tx=0).stop 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea580b5010 msgr2=0x7fea580a1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea580b5010 0x7fea580a1470 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fea50036040 tx=0x7fea50004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 shutdown_connections 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea58161a80 0x7fea580b8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea580d25a0 0x7fea581514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea3806b4d0 0x7fea3806d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 --2- 172.21.15.77:0/2433919455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea580b5010 0x7fea580a1470 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 >> 172.21.15.77:0/2433919455 conn(0x7fea580d3fc0 msgr2=0x7fea580b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 shutdown_connections 2024-03-06T07:26:48.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.623+0000 7fea5dcfb640 1 -- 172.21.15.77:0/2433919455 wait complete. 2024-03-06T07:26:48.637 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:48.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.738+0000 7f5ed9d7a640 1 Processor -- start 2024-03-06T07:26:48.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.738+0000 7f5ed9d7a640 1 -- start start 2024-03-06T07:26:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.739+0000 7f5ed9d7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.739+0000 7f5ed9d7a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ed40b9060 0x7f5ed40b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.739+0000 7f5ed9d7a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed40d1ea0 con 0x7f5ed415e150 2024-03-06T07:26:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.739+0000 7f5ed9d7a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed40d12f0 con 0x7f5ed40b9060 2024-03-06T07:26:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.739+0000 7f5ed9d7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40d2ab0 con 0x7f5ed40d0950 2024-03-06T07:26:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed2ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed3fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ed40b9060 0x7f5ed40b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed3fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5ed40b9060 0x7f5ed40b94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:39718/0 (socket says 172.21.15.77:39718) 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed37fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f5ed415e150 0x7f5ed40cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:59088/0 (socket says 172.21.15.77:59088) 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed2ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:55662/0 (socket says 172.21.15.77:55662) 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed3fff640 1 -- 172.21.15.77:0/1892431718 learned_addr learned my addr 172.21.15.77:0/1892431718 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed2ffd640 1 -- 172.21.15.77:0/1892431718 >> v2:172.21.15.158:3300/0 conn(0x7f5ed40b9060 msgr2=0x7f5ed40b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed2ffd640 1 --2- 172.21.15.77:0/1892431718 >> v2:172.21.15.158:3300/0 conn(0x7f5ed40b9060 0x7f5ed40b94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed2ffd640 1 -- 172.21.15.77:0/1892431718 >> v1:172.21.15.158:6789/0 conn(0x7f5ed415e150 legacy=0x7f5ed40cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed2ffd640 1 -- 172.21.15.77:0/1892431718 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed40d2180 con 0x7f5ed40d0950 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.740+0000 7f5ed3fff640 1 --2- 172.21.15.77:0/1892431718 >> v2:172.21.15.158:3300/0 conn(0x7f5ed40b9060 0x7f5ed40b94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.741+0000 7f5ed2ffd640 1 --2- 172.21.15.77:0/1892431718 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40d0db0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80097a0 tx=0x7f5ec802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4b44d4fdfb0bd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.741+0000 7f5ed27fc640 1 -- 172.21.15.77:0/1892431718 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec80315a0 con 0x7f5ed40d0950 2024-03-06T07:26:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.741+0000 7f5ed27fc640 1 -- 172.21.15.77:0/1892431718 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec8037070 con 0x7f5ed40d0950 2024-03-06T07:26:48.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.741+0000 7f5ed27fc640 1 -- 172.21.15.77:0/1892431718 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8031960 con 0x7f5ed40d0950 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.741+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/1892431718 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 msgr2=0x7f5ed40d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.741+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/1892431718 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40d0db0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80097a0 tx=0x7f5ec802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.742+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/1892431718 shutdown_connections 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.742+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/1892431718 >> v2:172.21.15.158:3300/0 conn(0x7f5ed40b9060 0x7f5ed40b94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.742+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/1892431718 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40d0db0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.742+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/1892431718 >> 172.21.15.77:0/1892431718 conn(0x7f5ed409d230 msgr2=0x7f5ed40d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.742+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/1892431718 shutdown_connections 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.742+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/1892431718 wait complete. 2024-03-06T07:26:48.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.743+0000 7f5ed9d7a640 1 Processor -- start 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.743+0000 7f5ed9d7a640 1 -- start start 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed9d7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ed40b9060 0x7f5ed4155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed9d7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed9d7a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ed415e150 0x7f5ed40c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed9d7a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40d2f40 con 0x7f5ed40b9060 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed9d7a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40d2740 con 0x7f5ed415e150 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed9d7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40d1b30 con 0x7f5ed40d0950 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ed40b9060 0x7f5ed4155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ed40b9060 0x7f5ed4155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36030/0 (socket says 172.21.15.77:36030) 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 -- 172.21.15.77:0/3284760205 learned_addr learned my addr 172.21.15.77:0/3284760205 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:48.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed3fff640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ed415e150 0x7f5ed40c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed2ffd640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 -- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 msgr2=0x7f5ed40cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 -- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ed415e150 msgr2=0x7f5ed40c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ed415e150 0x7f5ed40c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 -- 172.21.15.77:0/3284760205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed40c9ca0 con 0x7f5ed40b9060 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed2ffd640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed37fe640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ed40b9060 0x7f5ed4155480 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400c3c0 tx=0x7f5ec400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ebbfff640 1 -- 172.21.15.77:0/3284760205 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec40074e0 con 0x7f5ed40b9060 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ebbfff640 1 -- 172.21.15.77:0/3284760205 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec4007680 con 0x7f5ed40b9060 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed40cbe90 con 0x7f5ed40b9060 2024-03-06T07:26:48.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ebbfff640 1 -- 172.21.15.77:0/3284760205 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4004a00 con 0x7f5ed40b9060 2024-03-06T07:26:48.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.744+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed40cc450 con 0x7f5ed40b9060 2024-03-06T07:26:48.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.745+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed406b730 con 0x7f5ed40b9060 2024-03-06T07:26:48.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.746+0000 7f5ebbfff640 1 -- 172.21.15.77:0/3284760205 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ec401a020 con 0x7f5ed40b9060 2024-03-06T07:26:48.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.746+0000 7f5ebbfff640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ec006b4d0 0x7f5ec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.746+0000 7f5ebbfff640 1 -- 172.21.15.77:0/3284760205 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5ec40899d0 con 0x7f5ed40b9060 2024-03-06T07:26:48.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.746+0000 7f5ed2ffd640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ec006b4d0 0x7f5ec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.747+0000 7f5ed2ffd640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ec006b4d0 0x7f5ec006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8036000 tx=0x7f5ec803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.751+0000 7f5ebbfff640 1 -- 172.21.15.77:0/3284760205 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ed406b730 con 0x7f5ed40b9060 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.928+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ed4069ee0 con 0x7f5ed40b9060 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.928+0000 7f5ebbfff640 1 -- 172.21.15.77:0/3284760205 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ec4058750 con 0x7f5ed40b9060 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ec006b4d0 msgr2=0x7f5ec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ec006b4d0 0x7f5ec006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8036000 tx=0x7f5ec803d040 comp rx=0 tx=0).stop 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ed40b9060 msgr2=0x7f5ed4155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ed40b9060 0x7f5ed4155480 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400c3c0 tx=0x7f5ec400c880 comp rx=0 tx=0).stop 2024-03-06T07:26:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 shutdown_connections 2024-03-06T07:26:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ed415e150 0x7f5ed40c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5ed40d0950 0x7f5ed40cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ec006b4d0 0x7f5ec006d990 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 --2- 172.21.15.77:0/3284760205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ed40b9060 0x7f5ed4155480 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 >> 172.21.15.77:0/3284760205 conn(0x7f5ed409d230 msgr2=0x7f5ed40b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 shutdown_connections 2024-03-06T07:26:48.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:48.930+0000 7f5ed9d7a640 1 -- 172.21.15.77:0/3284760205 wait complete. 2024-03-06T07:26:48.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.043+0000 7f4a95935640 1 Processor -- start 2024-03-06T07:26:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.043+0000 7f4a95935640 1 -- start start 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a95935640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4a9015e150 0x7f4a900cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a95935640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a900d1ea0 con 0x7f4a900cfba0 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a95935640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a900d2180 con 0x7f4a900d1060 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a95935640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a900d2ab0 con 0x7f4a9015e150 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a87fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4a900cfba0 0x7f4a900d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57530/0 (socket says 172.21.15.77:57530) 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a87fff640 1 -- 172.21.15.77:0/1581276424 learned_addr learned my addr 172.21.15.77:0/1581276424 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714875900 0 0) 0x7f4a900d1ea0 con 0x7f4a900cfba0 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8effd640 1 --2- 172.21.15.77:0/1581276424 >> v2:172.21.15.158:3300/0 conn(0x7f4a9015e150 0x7f4a900cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c003520 con 0x7f4a900cfba0 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98689757 0 0) 0x7f4a7c003520 con 0x7f4a900cfba0 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 >> v2:172.21.15.158:3300/0 conn(0x7f4a9015e150 msgr2=0x7f4a900cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 --2- 172.21.15.77:0/1581276424 >> v2:172.21.15.158:3300/0 conn(0x7f4a9015e150 0x7f4a900cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 >> v1:172.21.15.158:6789/0 conn(0x7f4a900d1060 legacy=0x7f4a900b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a900d2460 con 0x7f4a900cfba0 2024-03-06T07:26:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f4a78001670 con 0x7f4a900cfba0 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 >> v1:172.21.15.77:6789/0 conn(0x7f4a900cfba0 legacy=0x7f4a900d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a7c0039b0 0x7f4a7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7c004800 0x7f4a7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a900d2ab0 con 0x7f4a7c0051c0 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.044+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c00c160 con 0x7f4a7c0039b0 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c00c360 con 0x7f4a7c004800 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8effd640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a7c0039b0 0x7f4a7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8f7fe640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a87fff640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7c004800 0x7f4a7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8f7fe640 1 -- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7c004800 msgr2=0x7f4a7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8f7fe640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7c004800 0x7f4a7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8f7fe640 1 -- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a7c0039b0 msgr2=0x7f4a7c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8f7fe640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a7c0039b0 0x7f4a7c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8f7fe640 1 -- 172.21.15.77:0/1581276424 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c003520 con 0x7f4a7c0051c0 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.045+0000 7f4a8effd640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a7c0039b0 0x7f4a7c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a8f7fe640 1 -- 172.21.15.77:0/1581276424 reap_dead start 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a8f7fe640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a7c00b9c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f4a9008dfe0 tx=0x7f4a7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e646cf07cc6c6d02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80009690 con 0x7f4a7c0051c0 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88003b70 con 0x7f4a7c0051c0 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a8e7fc640 1 -- 172.21.15.77:0/1581276424 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88001a70 con 0x7f4a7c0051c0 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a95935640 1 -- 172.21.15.77:0/1581276424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 msgr2=0x7f4a7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a95935640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a7c00b9c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f4a9008dfe0 tx=0x7f4a7c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:49.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a95935640 1 -- 172.21.15.77:0/1581276424 shutdown_connections 2024-03-06T07:26:49.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a95935640 1 --2- 172.21.15.77:0/1581276424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a7c00b9c0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a95935640 1 -- 172.21.15.77:0/1581276424 >> 172.21.15.77:0/1581276424 conn(0x7f4a9009d230 msgr2=0x7f4a900d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.046+0000 7f4a95935640 1 -- 172.21.15.77:0/1581276424 shutdown_connections 2024-03-06T07:26:49.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.047+0000 7f4a95935640 1 -- 172.21.15.77:0/1581276424 wait complete. 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a95935640 1 Processor -- start 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a95935640 1 -- start start 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a95935640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a900ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a95935640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a9015a3f0 0x7f4a9009e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a95935640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a9009e5c0 0x7f4a901f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a95935640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a900d1b30 con 0x7f4a7c0051c0 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a95935640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a900d2f40 con 0x7f4a9009e5c0 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a95935640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9015e530 con 0x7f4a9015a3f0 2024-03-06T07:26:49.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a8effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a900ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a8effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a900ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36048/0 (socket says 172.21.15.77:36048) 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.048+0000 7f4a8effd640 1 -- 172.21.15.77:0/1125864329 learned_addr learned my addr 172.21.15.77:0/1125864329 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a87fff640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a9015a3f0 0x7f4a9009e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8f7fe640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a9009e5c0 0x7f4a901f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8effd640 1 -- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a9015a3f0 msgr2=0x7f4a9009e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8effd640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a9015a3f0 0x7f4a9009e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8effd640 1 -- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a9009e5c0 msgr2=0x7f4a901f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8effd640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a9009e5c0 0x7f4a901f3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8effd640 1 -- 172.21.15.77:0/1125864329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a900c8460 con 0x7f4a7c0051c0 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a87fff640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a9015a3f0 0x7f4a9009e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8effd640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a900ca5a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f4a80011930 tx=0x7f4a80011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8cff9640 1 -- 172.21.15.77:0/1125864329 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8000f9d0 con 0x7f4a7c0051c0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8cff9640 1 -- 172.21.15.77:0/1125864329 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8000fb70 con 0x7f4a7c0051c0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a8cff9640 1 -- 172.21.15.77:0/1125864329 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80014440 con 0x7f4a7c0051c0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a900c8740 con 0x7f4a7c0051c0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.049+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a900c8d00 con 0x7f4a7c0051c0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.050+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a900745b0 con 0x7f4a7c0051c0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.051+0000 7f4a8cff9640 1 -- 172.21.15.77:0/1125864329 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a80012040 con 0x7f4a7c0051c0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.051+0000 7f4a8cff9640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a7006b4d0 0x7f4a7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.051+0000 7f4a8cff9640 1 -- 172.21.15.77:0/1125864329 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4a8008a870 con 0x7f4a7c0051c0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.051+0000 7f4a87fff640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a7006b4d0 0x7f4a7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.051+0000 7f4a87fff640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a7006b4d0 0x7f4a7006d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4a9015f380 tx=0x7f4a78009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.055+0000 7f4a8cff9640 1 -- 172.21.15.77:0/1125864329 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a900745b0 con 0x7f4a7c0051c0 2024-03-06T07:26:49.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.246+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4a90069ee0 con 0x7f4a7c0051c0 2024-03-06T07:26:49.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.247+0000 7f4a8cff9640 1 -- 172.21.15.77:0/1125864329 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v96) v1 ==== 80+0+366 (secure 0 0 0) 0x7f4a80019030 con 0x7f4a7c0051c0 2024-03-06T07:26:49.250 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.158:6833/497683896 2024-03-07T07:26:21.753195+0000 2024-03-06T07:26:49.250 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6835/1877423196 2024-03-07T07:26:21.743752+0000 2024-03-06T07:26:49.250 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.158:6835/497683896 2024-03-07T07:26:21.753195+0000 2024-03-06T07:26:49.250 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6834/1877423196 2024-03-07T07:26:21.743752+0000 2024-03-06T07:26:49.250 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6837/3425851566 2024-03-07T07:26:21.715277+0000 2024-03-06T07:26:49.251 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6836/3425851566 2024-03-07T07:26:21.715277+0000 2024-03-06T07:26:49.251 INFO:teuthology.orchestra.run.smithi077.stderr:listed 6 entries 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.248+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a7006b4d0 msgr2=0x7f4a7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.248+0000 7f4a95935640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a7006b4d0 0x7f4a7006d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4a9015f380 tx=0x7f4a78009040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.248+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 msgr2=0x7f4a900ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.248+0000 7f4a95935640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a900ca5a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f4a80011930 tx=0x7f4a80011df0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.249+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 shutdown_connections 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.249+0000 7f4a95935640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a9009e5c0 0x7f4a901f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.249+0000 7f4a95935640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a9015a3f0 0x7f4a9009e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.249+0000 7f4a95935640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a7006b4d0 0x7f4a7006d990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.249+0000 7f4a95935640 1 --2- 172.21.15.77:0/1125864329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7c0051c0 0x7f4a900ca5a0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.249+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 >> 172.21.15.77:0/1125864329 conn(0x7f4a9009d230 msgr2=0x7f4a90160e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.249+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 shutdown_connections 2024-03-06T07:26:49.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.249+0000 7f4a95935640 1 -- 172.21.15.77:0/1125864329 wait complete. 2024-03-06T07:26:49.263 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:26:49.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:49.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:49.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[68461]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:49.290 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:26:49.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:49.290 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:49.420 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:26:49.420 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:49.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:49.438 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:26:49.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:49.438 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:49.494 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:49.495 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:26:49.495 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:49.511 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:49.511 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:26:49.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:49.511 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:49.566 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:49.567 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:26:49.567 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:49.583 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:49.584 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:26:49.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:49.584 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:49.639 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:49.640 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:49.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.737+0000 7f182a247640 1 Processor -- start 2024-03-06T07:26:49.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.737+0000 7f182a247640 1 -- start start 2024-03-06T07:26:49.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f182a247640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f182415e150 0x7f18240cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f182a247640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240d1ea0 con 0x7f18240d1060 2024-03-06T07:26:49.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f182a247640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240d2180 con 0x7f18240cfba0 2024-03-06T07:26:49.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f182a247640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240d2ab0 con 0x7f182415e150 2024-03-06T07:26:49.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f1823fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f18240d1060 0x7f18240b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57540/0 (socket says 172.21.15.77:57540) 2024-03-06T07:26:49.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f1823fff640 1 -- 172.21.15.77:0/2340576220 learned_addr learned my addr 172.21.15.77:0/2340576220 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:49.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f18237fe640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f182415e150 0x7f18240cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709742068 0 0) 0x7f18240d1ea0 con 0x7f18240d1060 2024-03-06T07:26:49.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f180c003520 con 0x7f18240d1060 2024-03-06T07:26:49.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944837451 0 0) 0x7f180c003520 con 0x7f18240d1060 2024-03-06T07:26:49.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f182415e150 msgr2=0x7f18240cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f18227fc640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f182415e150 0x7f18240cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.738+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 >> v1:172.21.15.158:6789/0 conn(0x7f18240cfba0 legacy=0x7f18240d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240d2460 con 0x7f18240d1060 2024-03-06T07:26:49.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f181c0019f0 con 0x7f18240d1060 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 >> v1:172.21.15.77:6789/0 conn(0x7f18240d1060 legacy=0x7f18240b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f180c0039e0 0x7f180c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f182415e150 0x7f180c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f180c004d40 0x7f180c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240d2ab0 con 0x7f180c004d40 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c00baf0 con 0x7f180c0039e0 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180c00bcf0 con 0x7f182415e150 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f1823fff640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f180c004d40 0x7f180c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f18237fe640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f180c0039e0 0x7f180c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f1822ffd640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f182415e150 0x7f180c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f1823fff640 1 -- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f182415e150 msgr2=0x7f180c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f1823fff640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f182415e150 0x7f180c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f1823fff640 1 -- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f180c0039e0 msgr2=0x7f180c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f1823fff640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f180c0039e0 0x7f180c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.739+0000 7f1823fff640 1 -- 172.21.15.77:0/2340576220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c003520 con 0x7f180c004d40 2024-03-06T07:26:49.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.740+0000 7f1823fff640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f180c004d40 0x7f180c0051c0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f181c00a860 tx=0x7f180c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a0f3e3fd5579ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.740+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c00a480 con 0x7f180c004d40 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.740+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c034630 con 0x7f180c004d40 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.740+0000 7f18227fc640 1 -- 172.21.15.77:0/2340576220 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c034a60 con 0x7f180c004d40 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.741+0000 7f182a247640 1 -- 172.21.15.77:0/2340576220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f180c004d40 msgr2=0x7f180c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.741+0000 7f182a247640 1 --2- 172.21.15.77:0/2340576220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f180c004d40 0x7f180c0051c0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f181c00a860 tx=0x7f180c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.741+0000 7f1823fff640 1 -- 172.21.15.77:0/2340576220 reap_dead start 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.741+0000 7f182a247640 1 -- 172.21.15.77:0/2340576220 shutdown_connections 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.741+0000 7f182a247640 1 -- 172.21.15.77:0/2340576220 >> 172.21.15.77:0/2340576220 conn(0x7f182409d230 msgr2=0x7f18240d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.741+0000 7f182a247640 1 -- 172.21.15.77:0/2340576220 shutdown_connections 2024-03-06T07:26:49.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.741+0000 7f182a247640 1 -- 172.21.15.77:0/2340576220 wait complete. 2024-03-06T07:26:49.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.742+0000 7f182a247640 1 Processor -- start 2024-03-06T07:26:49.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f182a247640 1 -- start start 2024-03-06T07:26:49.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f182a247640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f18241539a0 0x7f18240cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f182a247640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18240cd740 0x7f182409fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f182a247640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18240a0310 0x7f18241f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f182a247640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240b7dc0 con 0x7f18240a0310 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f182a247640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240d2f40 con 0x7f18240cd740 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f182a247640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240d1b30 con 0x7f18241539a0 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f1823fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18240a0310 0x7f18241f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f1823fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18240a0310 0x7f18241f77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36068/0 (socket says 172.21.15.77:36068) 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f1823fff640 1 -- 172.21.15.77:0/599319213 learned_addr learned my addr 172.21.15.77:0/599319213 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f1822ffd640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18240cd740 0x7f182409fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.743+0000 7f18237fe640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f18241539a0 0x7f18240cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f1823fff640 1 -- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f18241539a0 msgr2=0x7f18240cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f1823fff640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f18241539a0 0x7f18240cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f1823fff640 1 -- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18240cd740 msgr2=0x7f182409fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f1823fff640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18240cd740 0x7f182409fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f1823fff640 1 -- 172.21.15.77:0/599319213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240a1be0 con 0x7f18240a0310 2024-03-06T07:26:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f1822ffd640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18240cd740 0x7f182409fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f18237fe640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f18241539a0 0x7f18240cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f1823fff640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18240a0310 0x7f18241f77b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f181c02cc70 tx=0x7f18240d2a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f180bfff640 1 -- 172.21.15.77:0/599319213 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c045070 con 0x7f18240a0310 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f180bfff640 1 -- 172.21.15.77:0/599319213 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c03a040 con 0x7f18240a0310 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18240a0b40 con 0x7f18240a0310 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f180bfff640 1 -- 172.21.15.77:0/599319213 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c00a480 con 0x7f18240a0310 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.744+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18240a1100 con 0x7f18240a0310 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.745+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1824069ee0 con 0x7f18240a0310 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.745+0000 7f180bfff640 1 -- 172.21.15.77:0/599319213 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f181c048070 con 0x7f18240a0310 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.746+0000 7f180bfff640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f181006b4d0 0x7f181006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.746+0000 7f180bfff640 1 -- 172.21.15.77:0/599319213 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f181c0abc50 con 0x7f18240a0310 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.746+0000 7f18237fe640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f181006b4d0 0x7f181006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.746+0000 7f18237fe640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f181006b4d0 0x7f181006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1814009bc0 tx=0x7f1814008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.750+0000 7f180bfff640 1 -- 172.21.15.77:0/599319213 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f181c0b1020 con 0x7f18240a0310 2024-03-06T07:26:50.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.998+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18240d4670 con 0x7f18240a0310 2024-03-06T07:26:50.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:49.999+0000 7f180bfff640 1 -- 172.21.15.77:0/599319213 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f181c07a380 con 0x7f18240a0310 2024-03-06T07:26:50.002 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:50.002 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:50.002 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:50.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f181006b4d0 msgr2=0x7f181006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f181006b4d0 0x7f181006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1814009bc0 tx=0x7f1814008040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18240a0310 msgr2=0x7f18241f77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18240a0310 0x7f18241f77b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f181c02cc70 tx=0x7f18240d2a20 comp rx=0 tx=0).stop 2024-03-06T07:26:50.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 shutdown_connections 2024-03-06T07:26:50.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f181006b4d0 0x7f181006d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18240a0310 0x7f18241f77b0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f18240cd740 0x7f182409fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 --2- 172.21.15.77:0/599319213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f18241539a0 0x7f18240cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 >> 172.21.15.77:0/599319213 conn(0x7f182409d230 msgr2=0x7f18240b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 shutdown_connections 2024-03-06T07:26:50.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.001+0000 7f182a247640 1 -- 172.21.15.77:0/599319213 wait complete. 2024-03-06T07:26:50.019 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.118+0000 7fa178ac2640 1 Processor -- start 2024-03-06T07:26:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.118+0000 7fa178ac2640 1 -- start start 2024-03-06T07:26:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa178ac2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa174154af0 0x7fa1740cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa178ac2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1740ce190 con 0x7fa17415abc0 2024-03-06T07:26:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa178ac2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17415b0c0 con 0x7fa17406d9e0 2024-03-06T07:26:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa178ac2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1740cf9a0 con 0x7fa174154af0 2024-03-06T07:26:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa17259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa174154af0 0x7fa1740cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa171d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa17415abc0 0x7fa174072aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57554/0 (socket says 172.21.15.77:57554) 2024-03-06T07:26:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa171d9b640 1 -- 172.21.15.77:0/374461934 learned_addr learned my addr 172.21.15.77:0/374461934 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa17259c640 1 -- 172.21.15.77:0/374461934 >> v1:172.21.15.158:6789/0 conn(0x7fa17406d9e0 legacy=0x7fa174071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa17259c640 1 -- 172.21.15.77:0/374461934 >> v1:172.21.15.77:6789/0 conn(0x7fa17415abc0 legacy=0x7fa174072aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa17159a640 1 -- 172.21.15.77:0/374461934 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669000018 0 0) 0x7fa1740ce190 con 0x7fa17415abc0 2024-03-06T07:26:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.119+0000 7fa17259c640 1 -- 172.21.15.77:0/374461934 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17415b3a0 con 0x7fa174154af0 2024-03-06T07:26:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.120+0000 7fa17259c640 1 --2- 172.21.15.77:0/374461934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa174154af0 0x7fa1740cdc50 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa1680097a0 tx=0x7fa16802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d729a3068092c71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.120+0000 7fa17159a640 1 -- 172.21.15.77:0/374461934 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168034070 con 0x7fa174154af0 2024-03-06T07:26:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.120+0000 7fa17159a640 1 -- 172.21.15.77:0/374461934 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168030610 con 0x7fa174154af0 2024-03-06T07:26:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.120+0000 7fa17159a640 1 -- 172.21.15.77:0/374461934 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1680309e0 con 0x7fa174154af0 2024-03-06T07:26:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.121+0000 7fa178ac2640 1 -- 172.21.15.77:0/374461934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa174154af0 msgr2=0x7fa1740cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.121+0000 7fa178ac2640 1 --2- 172.21.15.77:0/374461934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa174154af0 0x7fa1740cdc50 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa1680097a0 tx=0x7fa16802b960 comp rx=0 tx=0).stop 2024-03-06T07:26:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.121+0000 7fa178ac2640 1 -- 172.21.15.77:0/374461934 shutdown_connections 2024-03-06T07:26:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.121+0000 7fa178ac2640 1 --2- 172.21.15.77:0/374461934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa174154af0 0x7fa1740cdc50 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.121+0000 7fa178ac2640 1 -- 172.21.15.77:0/374461934 >> 172.21.15.77:0/374461934 conn(0x7fa17406b160 msgr2=0x7fa17406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.121+0000 7fa178ac2640 1 -- 172.21.15.77:0/374461934 shutdown_connections 2024-03-06T07:26:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.121+0000 7fa178ac2640 1 -- 172.21.15.77:0/374461934 wait complete. 2024-03-06T07:26:50.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.122+0000 7fa178ac2640 1 Processor -- start 2024-03-06T07:26:50.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.122+0000 7fa178ac2640 1 -- start start 2024-03-06T07:26:50.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa178ac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa17406d9e0 0x7fa17415e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa178ac2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa174154af0 0x7fa1740b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa178ac2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa17415abc0 0x7fa1741404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa178ac2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17415b960 con 0x7fa17406d9e0 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa178ac2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa174070d70 con 0x7fa174154af0 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa178ac2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa174154ed0 con 0x7fa17415abc0 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa17406d9e0 0x7fa17415e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa17406d9e0 0x7fa17415e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36096/0 (socket says 172.21.15.77:36096) 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 -- 172.21.15.77:0/3771439418 learned_addr learned my addr 172.21.15.77:0/3771439418 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa171d9b640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa174154af0 0x7fa1740b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa172d9d640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa17415abc0 0x7fa1741404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 -- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa17415abc0 msgr2=0x7fa1741404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa17415abc0 0x7fa1741404b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 -- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa174154af0 msgr2=0x7fa1740b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa174154af0 0x7fa1740b2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 -- 172.21.15.77:0/3771439418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17414a0c0 con 0x7fa17406d9e0 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa171d9b640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa174154af0 0x7fa1740b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa172d9d640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa17415abc0 0x7fa1741404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.123+0000 7fa17259c640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa17406d9e0 0x7fa17415e510 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fa168000c00 tx=0x7fa1680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.124+0000 7fa162ffd640 1 -- 172.21.15.77:0/3771439418 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168034040 con 0x7fa17406d9e0 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.124+0000 7fa162ffd640 1 -- 172.21.15.77:0/3771439418 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168004030 con 0x7fa17406d9e0 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.124+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17415c3b0 con 0x7fa17406d9e0 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.124+0000 7fa162ffd640 1 -- 172.21.15.77:0/3771439418 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168004420 con 0x7fa17406d9e0 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.124+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17415c970 con 0x7fa17406d9e0 2024-03-06T07:26:50.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.125+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa174069ee0 con 0x7fa17406d9e0 2024-03-06T07:26:50.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.125+0000 7fa162ffd640 1 -- 172.21.15.77:0/3771439418 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa16804c060 con 0x7fa17406d9e0 2024-03-06T07:26:50.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.125+0000 7fa162ffd640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa15406b4d0 0x7fa15406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.126+0000 7fa162ffd640 1 -- 172.21.15.77:0/3771439418 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa1680abf50 con 0x7fa17406d9e0 2024-03-06T07:26:50.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.126+0000 7fa171d9b640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa15406b4d0 0x7fa15406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.126+0000 7fa171d9b640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa15406b4d0 0x7fa15406d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa17406b9a0 tx=0x7fa164009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.130+0000 7fa162ffd640 1 -- 172.21.15.77:0/3771439418 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa16807acd0 con 0x7fa17406d9e0 2024-03-06T07:26:50.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.303+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa174073f70 con 0x7fa17406d9e0 2024-03-06T07:26:50.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.304+0000 7fa162ffd640 1 -- 172.21.15.77:0/3771439418 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16985 (secure 0 0 0) 0x7fa16807a5d0 con 0x7fa17406d9e0 2024-03-06T07:26:50.307 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:50.307 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":96,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:21.753221+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:07.372376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:08.123063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6835/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.158:6835/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6834/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.77:6837/3425851566":"2024-03-07T07:26:21.715277+0000","172.21.15.77:6836/3425851566":"2024-03-07T07:26:21.715277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:50.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.305+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa15406b4d0 msgr2=0x7fa15406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.305+0000 7fa178ac2640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa15406b4d0 0x7fa15406d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa17406b9a0 tx=0x7fa164009040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.305+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa17406d9e0 msgr2=0x7fa17415e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.305+0000 7fa178ac2640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa17406d9e0 0x7fa17415e510 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fa168000c00 tx=0x7fa1680046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.306+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 shutdown_connections 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.306+0000 7fa178ac2640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa17415abc0 0x7fa1741404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.306+0000 7fa178ac2640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa174154af0 0x7fa1740b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.306+0000 7fa178ac2640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa15406b4d0 0x7fa15406d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.306+0000 7fa178ac2640 1 --2- 172.21.15.77:0/3771439418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa17406d9e0 0x7fa17415e510 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.306+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 >> 172.21.15.77:0/3771439418 conn(0x7fa17406b160 msgr2=0x7fa1740b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.306+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 shutdown_connections 2024-03-06T07:26:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.306+0000 7fa178ac2640 1 -- 172.21.15.77:0/3771439418 wait complete. 2024-03-06T07:26:50.320 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:07.372376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:08.123063+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:50.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.424+0000 7fda3206d640 1 Processor -- start 2024-03-06T07:26:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.424+0000 7fda3206d640 1 -- start start 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.425+0000 7fda3206d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fda2c15e150 0x7fda2c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.425+0000 7fda3206d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c0d1ea0 con 0x7fda2c0d1060 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.425+0000 7fda3206d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c0d2180 con 0x7fda2c0cfba0 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.425+0000 7fda3206d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2c0d2ab0 con 0x7fda2c15e150 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.425+0000 7fda2bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fda2c0d1060 0x7fda2c0b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57570/0 (socket says 172.21.15.77:57570) 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.425+0000 7fda2bfff640 1 -- 172.21.15.77:0/648635351 learned_addr learned my addr 172.21.15.77:0/648635351 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.425+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/648635351 >> v2:172.21.15.158:3300/0 conn(0x7fda2c15e150 0x7fda2c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336898957 0 0) 0x7fda2c0d1ea0 con 0x7fda2c0d1060 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda14003520 con 0x7fda2c0d1060 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1112760230 0 0) 0x7fda14003520 con 0x7fda2c0d1060 2024-03-06T07:26:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 >> v2:172.21.15.158:3300/0 conn(0x7fda2c15e150 msgr2=0x7fda2c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 --2- 172.21.15.77:0/648635351 >> v2:172.21.15.158:3300/0 conn(0x7fda2c15e150 0x7fda2c0cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 >> v1:172.21.15.158:6789/0 conn(0x7fda2c0cfba0 legacy=0x7fda2c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0d2460 con 0x7fda2c0d1060 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fda240019f0 con 0x7fda2c0d1060 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 >> v1:172.21.15.77:6789/0 conn(0x7fda2c0d1060 legacy=0x7fda2c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda14004800 0x7fda14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda140051c0 0x7fda1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0d2ab0 con 0x7fda140039b0 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1400c160 con 0x7fda140051c0 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1400c360 con 0x7fda14004800 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/648635351 >> v2:172.21.15.158:3300/0 conn(0x7fda2c15e150 0x7fda2c0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2bfff640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda140051c0 0x7fda1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2affd640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda14004800 0x7fda14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2b7fe640 1 -- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda14004800 msgr2=0x7fda14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda14004800 0x7fda14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2b7fe640 1 -- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda140051c0 msgr2=0x7fda1400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda140051c0 0x7fda1400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2bfff640 1 -- 172.21.15.77:0/648635351 reap_dead start 2024-03-06T07:26:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2b7fe640 1 -- 172.21.15.77:0/648635351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda14003520 con 0x7fda140039b0 2024-03-06T07:26:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.426+0000 7fda2affd640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda14004800 0x7fda14004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:50.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.427+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda14003dd0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fda20009c70 tx=0x7fda14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cb0b06275691f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20009840 con 0x7fda140039b0 2024-03-06T07:26:50.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda20004020 con 0x7fda140039b0 2024-03-06T07:26:50.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda2a7fc640 1 -- 172.21.15.77:0/648635351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20004450 con 0x7fda140039b0 2024-03-06T07:26:50.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda3206d640 1 -- 172.21.15.77:0/648635351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 msgr2=0x7fda14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda3206d640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda14003dd0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fda20009c70 tx=0x7fda14005830 comp rx=0 tx=0).stop 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda3206d640 1 -- 172.21.15.77:0/648635351 shutdown_connections 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda3206d640 1 --2- 172.21.15.77:0/648635351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda14003dd0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda3206d640 1 -- 172.21.15.77:0/648635351 >> 172.21.15.77:0/648635351 conn(0x7fda2c09d230 msgr2=0x7fda2c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda3206d640 1 -- 172.21.15.77:0/648635351 shutdown_connections 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.428+0000 7fda3206d640 1 -- 172.21.15.77:0/648635351 wait complete. 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.429+0000 7fda3206d640 1 Processor -- start 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda3206d640 1 -- start start 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda3206d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda2c15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda3206d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda2c0ceb10 0x7fda2c0cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda3206d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda2c0b7940 0x7fda2c0b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda3206d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0d1b30 con 0x7fda140039b0 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda3206d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0d2f40 con 0x7fda2c0ceb10 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda3206d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c15e530 con 0x7fda2c0b7940 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda2b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda2c15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda2b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda2c15ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36112/0 (socket says 172.21.15.77:36112) 2024-03-06T07:26:50.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.430+0000 7fda2b7fe640 1 -- 172.21.15.77:0/2244901321 learned_addr learned my addr 172.21.15.77:0/2244901321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda2affd640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda2c0ceb10 0x7fda2c0cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda2bfff640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda2c0b7940 0x7fda2c0b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda2b7fe640 1 -- 172.21.15.77:0/2244901321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda2c0b7940 msgr2=0x7fda2c0b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda2c0b7940 0x7fda2c0b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda2b7fe640 1 -- 172.21.15.77:0/2244901321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda2c0ceb10 msgr2=0x7fda2c0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda2c0ceb10 0x7fda2c0cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda2b7fe640 1 -- 172.21.15.77:0/2244901321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0b8670 con 0x7fda140039b0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda2b7fe640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda2c15ca60 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fda20009c70 tx=0x7fda20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda13fff640 1 -- 172.21.15.77:0/2244901321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20043020 con 0x7fda140039b0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda13fff640 1 -- 172.21.15.77:0/2244901321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2002c8f0 con 0x7fda140039b0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda13fff640 1 -- 172.21.15.77:0/2244901321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2002cca0 con 0x7fda140039b0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c0b8950 con 0x7fda140039b0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.431+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c0b8f10 con 0x7fda140039b0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.432+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c069ee0 con 0x7fda140039b0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.432+0000 7fda13fff640 1 -- 172.21.15.77:0/2244901321 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda20046020 con 0x7fda140039b0 2024-03-06T07:26:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.433+0000 7fda13fff640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b500 0x7fda1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.433+0000 7fda13fff640 1 -- 172.21.15.77:0/2244901321 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fda200ac1c0 con 0x7fda140039b0 2024-03-06T07:26:50.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.433+0000 7fda2affd640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b500 0x7fda1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.433+0000 7fda2affd640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b500 0x7fda1806d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000c00 tx=0x7fda1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.437+0000 7fda13fff640 1 -- 172.21.15.77:0/2244901321 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda2004d030 con 0x7fda140039b0 2024-03-06T07:26:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.608+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda2c0d4670 con 0x7fda140039b0 2024-03-06T07:26:50.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.609+0000 7fda13fff640 1 -- 172.21.15.77:0/2244901321 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda2007afd0 con 0x7fda140039b0 2024-03-06T07:26:50.612 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b500 msgr2=0x7fda1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b500 0x7fda1806d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000c00 tx=0x7fda1c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 msgr2=0x7fda2c15ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda2c15ca60 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fda20009c70 tx=0x7fda20004730 comp rx=0 tx=0).stop 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 shutdown_connections 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fda2c0b7940 0x7fda2c0b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fda2c0ceb10 0x7fda2c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda1806b500 0x7fda1806d9c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 --2- 172.21.15.77:0/2244901321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda140039b0 0x7fda2c15ca60 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 >> 172.21.15.77:0/2244901321 conn(0x7fda2c09d230 msgr2=0x7fda2c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 shutdown_connections 2024-03-06T07:26:50.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.611+0000 7fda3206d640 1 -- 172.21.15.77:0/2244901321 wait complete. 2024-03-06T07:26:50.625 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.727+0000 7fba0480e640 1 Processor -- start 2024-03-06T07:26:50.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.727+0000 7fba0480e640 1 -- start start 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fba0480e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fba000cec60 0x7fba000b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fba0480e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba000b16b0 0x7fba000cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fba0480e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba000cd4d0 con 0x7fba0015f5e0 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fba0480e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba000cf920 con 0x7fba000b16b0 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fba0480e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba000cd7b0 con 0x7fba000cec60 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba000b16b0 0x7fba000cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fe59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fba0015f5e0 0x7fba000ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57586/0 (socket says 172.21.15.77:57586) 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fe59c640 1 -- 172.21.15.77:0/1554809538 learned_addr learned my addr 172.21.15.77:0/1554809538 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fdd9b640 1 --2- 172.21.15.77:0/1554809538 >> v2:172.21.15.158:3300/0 conn(0x7fba000cec60 0x7fba000b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fed9d640 1 -- 172.21.15.77:0/1554809538 >> v2:172.21.15.158:3300/0 conn(0x7fba000cec60 msgr2=0x7fba000b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fed9d640 1 --2- 172.21.15.77:0/1554809538 >> v2:172.21.15.158:3300/0 conn(0x7fba000cec60 0x7fba000b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fed9d640 1 -- 172.21.15.77:0/1554809538 >> v1:172.21.15.77:6789/0 conn(0x7fba0015f5e0 legacy=0x7fba000ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fed9d640 1 -- 172.21.15.77:0/1554809538 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba000cda90 con 0x7fba000b16b0 2024-03-06T07:26:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.728+0000 7fb9fd59a640 1 -- 172.21.15.77:0/1554809538 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156954633 0 0) 0x7fba000cd4d0 con 0x7fba0015f5e0 2024-03-06T07:26:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fb9fed9d640 1 --2- 172.21.15.77:0/1554809538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba000b16b0 0x7fba000cfdb0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fba00071dc0 tx=0x7fb9f802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0213fcf0d46b467 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fb9fd59a640 1 -- 172.21.15.77:0/1554809538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f80315a0 con 0x7fba000b16b0 2024-03-06T07:26:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fb9fd59a640 1 -- 172.21.15.77:0/1554809538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f8037070 con 0x7fba000b16b0 2024-03-06T07:26:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fb9fd59a640 1 -- 172.21.15.77:0/1554809538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8031960 con 0x7fba000b16b0 2024-03-06T07:26:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fba0480e640 1 -- 172.21.15.77:0/1554809538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba000b16b0 msgr2=0x7fba000cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fba0480e640 1 --2- 172.21.15.77:0/1554809538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba000b16b0 0x7fba000cfdb0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fba00071dc0 tx=0x7fb9f802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fba0480e640 1 -- 172.21.15.77:0/1554809538 shutdown_connections 2024-03-06T07:26:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fba0480e640 1 --2- 172.21.15.77:0/1554809538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba000b16b0 0x7fba000cfdb0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fba0480e640 1 --2- 172.21.15.77:0/1554809538 >> v2:172.21.15.158:3300/0 conn(0x7fba000cec60 0x7fba000b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.729+0000 7fba0480e640 1 -- 172.21.15.77:0/1554809538 >> 172.21.15.77:0/1554809538 conn(0x7fba000d1d50 msgr2=0x7fba000d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.730+0000 7fba0480e640 1 -- 172.21.15.77:0/1554809538 shutdown_connections 2024-03-06T07:26:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.730+0000 7fba0480e640 1 -- 172.21.15.77:0/1554809538 wait complete. 2024-03-06T07:26:50.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.731+0000 7fba0480e640 1 Processor -- start 2024-03-06T07:26:50.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.731+0000 7fba0480e640 1 -- start start 2024-03-06T07:26:50.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fba0480e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba000b16b0 0x7fba00159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fba0480e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba000cec60 0x7fba000c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fba0480e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0015f5e0 0x7fba000c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fba0480e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba000cf270 con 0x7fba0015f5e0 2024-03-06T07:26:50.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fba0480e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba000cf0f0 con 0x7fba000cec60 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fba0480e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba000d2870 con 0x7fba000b16b0 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0015f5e0 0x7fba000c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0015f5e0 0x7fba000c5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36130/0 (socket says 172.21.15.77:36130) 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fed9d640 1 -- 172.21.15.77:0/3485717090 learned_addr learned my addr 172.21.15.77:0/3485717090 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fdd9b640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba000cec60 0x7fba000c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fe59c640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba000b16b0 0x7fba00159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fed9d640 1 -- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba000b16b0 msgr2=0x7fba00159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fed9d640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba000b16b0 0x7fba00159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fed9d640 1 -- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba000cec60 msgr2=0x7fba000c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fed9d640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba000cec60 0x7fba000c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fed9d640 1 -- 172.21.15.77:0/3485717090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00159c60 con 0x7fba0015f5e0 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.732+0000 7fb9fdd9b640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba000cec60 0x7fba000c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.733+0000 7fb9fed9d640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0015f5e0 0x7fba000c5990 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009890 tx=0x7fb9f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.733+0000 7fb9f6ffd640 1 -- 172.21.15.77:0/3485717090 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8032630 con 0x7fba0015f5e0 2024-03-06T07:26:50.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.733+0000 7fb9f6ffd640 1 -- 172.21.15.77:0/3485717090 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f80327d0 con 0x7fba0015f5e0 2024-03-06T07:26:50.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.733+0000 7fb9f6ffd640 1 -- 172.21.15.77:0/3485717090 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8032ba0 con 0x7fba0015f5e0 2024-03-06T07:26:50.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.733+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba000b4d40 con 0x7fba0015f5e0 2024-03-06T07:26:50.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.733+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba000b5300 con 0x7fba0015f5e0 2024-03-06T07:26:50.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.734+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba00068050 con 0x7fba0015f5e0 2024-03-06T07:26:50.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.734+0000 7fb9f6ffd640 1 -- 172.21.15.77:0/3485717090 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9f804e430 con 0x7fba0015f5e0 2024-03-06T07:26:50.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.735+0000 7fb9f6ffd640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9e006b4d0 0x7fb9e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.735+0000 7fb9fe59c640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9e006b4d0 0x7fb9e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.735+0000 7fb9f6ffd640 1 -- 172.21.15.77:0/3485717090 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb9f802f030 con 0x7fba0015f5e0 2024-03-06T07:26:50.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.735+0000 7fb9fe59c640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9e006b4d0 0x7fb9e006d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00012e0 tx=0x7fb9f0005eb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:50.739+0000 7fb9f6ffd640 1 -- 172.21.15.77:0/3485717090 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba00159c60 con 0x7fba0015f5e0 2024-03-06T07:26:51.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.052+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba000b7cd0 con 0x7fba0015f5e0 2024-03-06T07:26:51.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.053+0000 7fb9f6ffd640 1 -- 172.21.15.77:0/3485717090 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb9f8040070 con 0x7fba0015f5e0 2024-03-06T07:26:51.055 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9e006b4d0 msgr2=0x7fb9e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9e006b4d0 0x7fb9e006d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00012e0 tx=0x7fb9f0005eb0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0015f5e0 msgr2=0x7fba000c5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0015f5e0 0x7fba000c5990 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009890 tx=0x7fb9f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 shutdown_connections 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9e006b4d0 0x7fb9e006d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba0015f5e0 0x7fba000c5990 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba000cec60 0x7fba000c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 --2- 172.21.15.77:0/3485717090 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba000b16b0 0x7fba00159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 >> 172.21.15.77:0/3485717090 conn(0x7fba000d1d50 msgr2=0x7fba0006edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 shutdown_connections 2024-03-06T07:26:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.055+0000 7fba0480e640 1 -- 172.21.15.77:0/3485717090 wait complete. 2024-03-06T07:26:51.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.171+0000 7fc4d3247640 1 Processor -- start 2024-03-06T07:26:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.171+0000 7fc4d3247640 1 -- start start 2024-03-06T07:26:51.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4d3247640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4cc161670 0x7fc4cc161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4d3247640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc0cd690 0x7fc4cc0cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4d3247640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4cc162c70 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4d3247640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4cc0b3590 con 0x7fc4cc161670 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4d3247640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc06cd80 con 0x7fc4cc0cd690 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4d0fe3640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4cc161670 0x7fc4cc161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4d17e4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc0cd690 0x7fc4cc0cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4cbfff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fc4cc0b2da0 0x7fc4cc1623a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:59132/0 (socket says 172.21.15.77:59132) 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4d0fe3640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4cc161670 0x7fc4cc161e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:39850/0 (socket says 172.21.15.77:39850) 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.172+0000 7fc4cbfff640 1 -- 172.21.15.77:0/1663099560 learned_addr learned my addr 172.21.15.77:0/1663099560 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.173+0000 7fc4cb7fe640 1 -- 172.21.15.77:0/1663099560 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406766128 0 0) 0x7fc4cc162c70 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.173+0000 7fc4cb7fe640 1 -- 172.21.15.77:0/1663099560 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc003520 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.173+0000 7fc4d0fe3640 1 -- 172.21.15.77:0/1663099560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc0cd690 msgr2=0x7fc4cc0cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.173+0000 7fc4d0fe3640 1 --2- 172.21.15.77:0/1663099560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc0cd690 0x7fc4cc0cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.173+0000 7fc4d0fe3640 1 -- 172.21.15.77:0/1663099560 >> v1:172.21.15.158:6789/0 conn(0x7fc4cc0b2da0 legacy=0x7fc4cc1623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.173+0000 7fc4d0fe3640 1 -- 172.21.15.77:0/1663099560 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc06d060 con 0x7fc4cc161670 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.173+0000 7fc4d0fe3640 1 --2- 172.21.15.77:0/1663099560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc161670 0x7fc4cc161e60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00097d0 tx=0x7fc4c002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68b12bceeca34a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4cb7fe640 1 -- 172.21.15.77:0/1663099560 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0035020 con 0x7fc4cc161670 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4cb7fe640 1 -- 172.21.15.77:0/1663099560 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c0031800 con 0x7fc4cc161670 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4cb7fe640 1 -- 172.21.15.77:0/1663099560 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0031bb0 con 0x7fc4cc161670 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4d3247640 1 -- 172.21.15.77:0/1663099560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc161670 msgr2=0x7fc4cc161e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1663099560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc161670 0x7fc4cc161e60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00097d0 tx=0x7fc4c002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4d3247640 1 -- 172.21.15.77:0/1663099560 shutdown_connections 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1663099560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc0cd690 0x7fc4cc0cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1663099560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc161670 0x7fc4cc161e60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4d3247640 1 -- 172.21.15.77:0/1663099560 >> 172.21.15.77:0/1663099560 conn(0x7fc4cc0d1d50 msgr2=0x7fc4cc0d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4d3247640 1 -- 172.21.15.77:0/1663099560 shutdown_connections 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.174+0000 7fc4d3247640 1 -- 172.21.15.77:0/1663099560 wait complete. 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.175+0000 7fc4d3247640 1 Processor -- start 2024-03-06T07:26:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.175+0000 7fc4d3247640 1 -- start start 2024-03-06T07:26:51.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d3247640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4cc0b2da0 0x7fc4cc152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d3247640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc0cd690 0x7fc4cc0b7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d3247640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc161670 0x7fc4cc0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d3247640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc06d4c0 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d3247640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc06d340 con 0x7fc4cc0cd690 2024-03-06T07:26:51.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d3247640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc161a50 con 0x7fc4cc161670 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d0fe3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4cc0b2da0 0x7fc4cc152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d0fe3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4cc0b2da0 0x7fc4cc152420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36140/0 (socket says 172.21.15.77:36140) 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d0fe3640 1 -- 172.21.15.77:0/1308680895 learned_addr learned my addr 172.21.15.77:0/1308680895 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4cbfff640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc0cd690 0x7fc4cc0b7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.176+0000 7fc4d17e4640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc161670 0x7fc4cc0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d0fe3640 1 -- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc161670 msgr2=0x7fc4cc0cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d0fe3640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc161670 0x7fc4cc0cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d0fe3640 1 -- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc0cd690 msgr2=0x7fc4cc0b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d0fe3640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc0cd690 0x7fc4cc0b7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d0fe3640 1 -- 172.21.15.77:0/1308680895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc0cb560 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d17e4640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc161670 0x7fc4cc0cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d0fe3640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4cc0b2da0 0x7fc4cc152420 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fc4c002b8e0 tx=0x7fc4c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4c97fa640 1 -- 172.21.15.77:0/1308680895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0040070 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4c97fa640 1 -- 172.21.15.77:0/1308680895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c0031e50 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4c97fa640 1 -- 172.21.15.77:0/1308680895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0035040 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4cc0b51a0 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.177+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4cc0b5760 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.178+0000 7fc4c97fa640 1 -- 172.21.15.77:0/1308680895 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4c004b060 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.178+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4cc068050 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.179+0000 7fc4c97fa640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4b006b500 0x7fc4b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.179+0000 7fc4c97fa640 1 -- 172.21.15.77:0/1308680895 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc4c00abb50 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.179+0000 7fc4cbfff640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4b006b500 0x7fc4b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.179+0000 7fc4cbfff640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4b006b500 0x7fc4b006d9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0cc090 tx=0x7fc4b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.187+0000 7fc4c97fa640 1 -- 172.21.15.77:0/1308680895 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4c007a960 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.364+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc4cc0ce480 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.364+0000 7fc4c97fa640 1 -- 172.21.15.77:0/1308680895 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc4c007a260 con 0x7fc4cc0b2da0 2024-03-06T07:26:51.368 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4b006b500 msgr2=0x7fc4b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4b006b500 0x7fc4b006d9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0cc090 tx=0x7fc4b8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4cc0b2da0 msgr2=0x7fc4cc152420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4cc0b2da0 0x7fc4cc152420 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fc4c002b8e0 tx=0x7fc4c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 shutdown_connections 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4cc161670 0x7fc4cc0cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4cc0cd690 0x7fc4cc0b7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc4b006b500 0x7fc4b006d9c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 --2- 172.21.15.77:0/1308680895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4cc0b2da0 0x7fc4cc152420 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 >> 172.21.15.77:0/1308680895 conn(0x7fc4cc0d1d50 msgr2=0x7fc4cc161e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 shutdown_connections 2024-03-06T07:26:51.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.366+0000 7fc4d3247640 1 -- 172.21.15.77:0/1308680895 wait complete. 2024-03-06T07:26:51.380 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:51.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.480+0000 7fce4278c640 1 Processor -- start 2024-03-06T07:26:51.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.480+0000 7fce4278c640 1 -- start start 2024-03-06T07:26:51.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce4278c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce3c15e150 0x7fce3c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce4278c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce3c0d1060 0x7fce3c0b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce4278c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3c0d1ea0 con 0x7fce3c0cfba0 2024-03-06T07:26:51.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce4278c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3c0d2ab0 con 0x7fce3c0d1060 2024-03-06T07:26:51.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce4278c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3c0d2180 con 0x7fce3c15e150 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce40d29640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce3c0d1060 0x7fce3c0b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce40d29640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce3c0d1060 0x7fce3c0b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36144/0 (socket says 172.21.15.77:36144) 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce3b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fce3c0cfba0 0x7fce3c0d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57592/0 (socket says 172.21.15.77:57592) 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.481+0000 7fce40d29640 1 -- 172.21.15.77:0/1123779700 learned_addr learned my addr 172.21.15.77:0/1123779700 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce3bfff640 1 --2- 172.21.15.77:0/1123779700 >> v2:172.21.15.158:3300/0 conn(0x7fce3c15e150 0x7fce3c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce40d29640 1 -- 172.21.15.77:0/1123779700 >> v2:172.21.15.158:3300/0 conn(0x7fce3c15e150 msgr2=0x7fce3c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce40d29640 1 --2- 172.21.15.77:0/1123779700 >> v2:172.21.15.158:3300/0 conn(0x7fce3c15e150 0x7fce3c0cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce40d29640 1 -- 172.21.15.77:0/1123779700 >> v1:172.21.15.77:6789/0 conn(0x7fce3c0cfba0 legacy=0x7fce3c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce40d29640 1 -- 172.21.15.77:0/1123779700 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c0d2460 con 0x7fce3c0d1060 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce3affd640 1 -- 172.21.15.77:0/1123779700 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141967021 0 0) 0x7fce3c0d1ea0 con 0x7fce3c0cfba0 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce3bfff640 1 --2- 172.21.15.77:0/1123779700 >> v2:172.21.15.158:3300/0 conn(0x7fce3c15e150 0x7fce3c0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce40d29640 1 --2- 172.21.15.77:0/1123779700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c0b9060 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0b6af0 tx=0x7fce3402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4129d7c6bf843c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce3affd640 1 -- 172.21.15.77:0/1123779700 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3402ca80 con 0x7fce3c0d1060 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.482+0000 7fce3affd640 1 -- 172.21.15.77:0/1123779700 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3402cc20 con 0x7fce3c0d1060 2024-03-06T07:26:51.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce3affd640 1 -- 172.21.15.77:0/1123779700 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340343f0 con 0x7fce3c0d1060 2024-03-06T07:26:51.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce4278c640 1 -- 172.21.15.77:0/1123779700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 msgr2=0x7fce3c0b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce4278c640 1 --2- 172.21.15.77:0/1123779700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c0b9060 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0b6af0 tx=0x7fce3402bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce4278c640 1 -- 172.21.15.77:0/1123779700 shutdown_connections 2024-03-06T07:26:51.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce4278c640 1 --2- 172.21.15.77:0/1123779700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c0b9060 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce4278c640 1 --2- 172.21.15.77:0/1123779700 >> v2:172.21.15.158:3300/0 conn(0x7fce3c15e150 0x7fce3c0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce4278c640 1 -- 172.21.15.77:0/1123779700 >> 172.21.15.77:0/1123779700 conn(0x7fce3c09d230 msgr2=0x7fce3c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce4278c640 1 -- 172.21.15.77:0/1123779700 shutdown_connections 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.483+0000 7fce4278c640 1 -- 172.21.15.77:0/1123779700 wait complete. 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.484+0000 7fce4278c640 1 Processor -- start 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.484+0000 7fce4278c640 1 -- start start 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce4278c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce3c0cfba0 0x7fce3c0cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce4278c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c1f8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce4278c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce3c15e150 0x7fce3c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce4278c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0d1b30 con 0x7fce3c0d1060 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce4278c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0d2f40 con 0x7fce3c0cfba0 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce4278c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c15e530 con 0x7fce3c15e150 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c1f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c1f8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36158/0 (socket says 172.21.15.77:36158) 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3b7fe640 1 -- 172.21.15.77:0/2624014422 learned_addr learned my addr 172.21.15.77:0/2624014422 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce40d29640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce3c15e150 0x7fce3c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3bfff640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce3c0cfba0 0x7fce3c0cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3b7fe640 1 -- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce3c15e150 msgr2=0x7fce3c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3b7fe640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce3c15e150 0x7fce3c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3b7fe640 1 -- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce3c0cfba0 msgr2=0x7fce3c0cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3b7fe640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce3c0cfba0 0x7fce3c0cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3b7fe640 1 -- 172.21.15.77:0/2624014422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c0cd4f0 con 0x7fce3c0d1060 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce40d29640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce3c15e150 0x7fce3c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.485+0000 7fce3bfff640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce3c0cfba0 0x7fce3c0cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.486+0000 7fce3b7fe640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c1f8140 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fce3000c500 tx=0x7fce3000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.486+0000 7fce38ff9640 1 -- 172.21.15.77:0/2624014422 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30018070 con 0x7fce3c0d1060 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.486+0000 7fce38ff9640 1 -- 172.21.15.77:0/2624014422 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce30007580 con 0x7fce3c0d1060 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.486+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c0b9720 con 0x7fce3c0d1060 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.486+0000 7fce38ff9640 1 -- 172.21.15.77:0/2624014422 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30004a20 con 0x7fce3c0d1060 2024-03-06T07:26:51.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.486+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c0b9ca0 con 0x7fce3c0d1060 2024-03-06T07:26:51.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.487+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c069ee0 con 0x7fce3c0d1060 2024-03-06T07:26:51.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.487+0000 7fce38ff9640 1 -- 172.21.15.77:0/2624014422 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce3001b070 con 0x7fce3c0d1060 2024-03-06T07:26:51.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.488+0000 7fce38ff9640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2806b4d0 0x7fce2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.488+0000 7fce38ff9640 1 -- 172.21.15.77:0/2624014422 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fce300895a0 con 0x7fce3c0d1060 2024-03-06T07:26:51.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.488+0000 7fce3bfff640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2806b4d0 0x7fce2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.488+0000 7fce3bfff640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2806b4d0 0x7fce2806d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fce2c006fd0 tx=0x7fce2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.492+0000 7fce38ff9640 1 -- 172.21.15.77:0/2624014422 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce300583b0 con 0x7fce3c0d1060 2024-03-06T07:26:51.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.746+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce3c06be20 con 0x7fce3c0d1060 2024-03-06T07:26:51.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.747+0000 7fce38ff9640 1 -- 172.21.15.77:0/2624014422 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fce30057cb0 con 0x7fce3c0d1060 2024-03-06T07:26:51.749 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:51.749 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:51.749 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2806b4d0 msgr2=0x7fce2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2806b4d0 0x7fce2806d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fce2c006fd0 tx=0x7fce2c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 msgr2=0x7fce3c1f8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c1f8140 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fce3000c500 tx=0x7fce3000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 shutdown_connections 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2806b4d0 0x7fce2806d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce3c0d1060 0x7fce3c1f8140 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce3c15e150 0x7fce3c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 --2- 172.21.15.77:0/2624014422 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce3c0cfba0 0x7fce3c0cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 >> 172.21.15.77:0/2624014422 conn(0x7fce3c09d230 msgr2=0x7fce3c15f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 shutdown_connections 2024-03-06T07:26:51.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.749+0000 7fce4278c640 1 -- 172.21.15.77:0/2624014422 wait complete. 2024-03-06T07:26:51.762 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:51.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.866+0000 7fe0999a0640 1 Processor -- start 2024-03-06T07:26:51.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.866+0000 7fe0999a0640 1 -- start start 2024-03-06T07:26:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe0999a0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe09406eca0 0x7fe094168f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe0999a0640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0940cf6e0 con 0x7fe094162aa0 2024-03-06T07:26:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe0999a0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0940cf9c0 con 0x7fe09415e220 2024-03-06T07:26:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe0999a0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0940cfca0 con 0x7fe09406eca0 2024-03-06T07:26:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe09919f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe09406eca0 0x7fe094168f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe09919f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe09406eca0 0x7fe094168f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36168/0 (socket says 172.21.15.77:36168) 2024-03-06T07:26:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe093fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe094162aa0 0x7fe09406e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57594/0 (socket says 172.21.15.77:57594) 2024-03-06T07:26:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe09919f640 1 -- 172.21.15.77:0/2966857775 learned_addr learned my addr 172.21.15.77:0/2966857775 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:51.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe09919f640 1 -- 172.21.15.77:0/2966857775 >> v1:172.21.15.158:6789/0 conn(0x7fe09415e220 legacy=0x7fe0940ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe09919f640 1 -- 172.21.15.77:0/2966857775 >> v1:172.21.15.77:6789/0 conn(0x7fe094162aa0 legacy=0x7fe09406e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe0937fe640 1 -- 172.21.15.77:0/2966857775 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030819899 0 0) 0x7fe0940cf6e0 con 0x7fe094162aa0 2024-03-06T07:26:51.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.867+0000 7fe09919f640 1 -- 172.21.15.77:0/2966857775 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940cff80 con 0x7fe09406eca0 2024-03-06T07:26:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.868+0000 7fe09919f640 1 --2- 172.21.15.77:0/2966857775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09406eca0 0x7fe094168f70 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009e50 tx=0x7fe08c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d1d34d8bcb2bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.868+0000 7fe0937fe640 1 -- 172.21.15.77:0/2966857775 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c02cb20 con 0x7fe09406eca0 2024-03-06T07:26:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.868+0000 7fe0937fe640 1 -- 172.21.15.77:0/2966857775 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c02ccc0 con 0x7fe09406eca0 2024-03-06T07:26:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.868+0000 7fe0937fe640 1 -- 172.21.15.77:0/2966857775 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c034610 con 0x7fe09406eca0 2024-03-06T07:26:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.869+0000 7fe0999a0640 1 -- 172.21.15.77:0/2966857775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09406eca0 msgr2=0x7fe094168f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.869+0000 7fe0999a0640 1 --2- 172.21.15.77:0/2966857775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09406eca0 0x7fe094168f70 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009e50 tx=0x7fe08c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.869+0000 7fe0999a0640 1 -- 172.21.15.77:0/2966857775 shutdown_connections 2024-03-06T07:26:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.869+0000 7fe0999a0640 1 --2- 172.21.15.77:0/2966857775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09406eca0 0x7fe094168f70 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.869+0000 7fe0999a0640 1 -- 172.21.15.77:0/2966857775 >> 172.21.15.77:0/2966857775 conn(0x7fe0940b2930 msgr2=0x7fe0940b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.869+0000 7fe0999a0640 1 -- 172.21.15.77:0/2966857775 shutdown_connections 2024-03-06T07:26:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.869+0000 7fe0999a0640 1 -- 172.21.15.77:0/2966857775 wait complete. 2024-03-06T07:26:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.870+0000 7fe0999a0640 1 Processor -- start 2024-03-06T07:26:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.870+0000 7fe0999a0640 1 -- start start 2024-03-06T07:26:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe0999a0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe09406eca0 0x7fe0940add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe0999a0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09415e220 0x7fe094155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe0999a0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe094162aa0 0x7fe09415f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe0999a0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09406ca10 con 0x7fe09415e220 2024-03-06T07:26:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe0999a0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09415e600 con 0x7fe094162aa0 2024-03-06T07:26:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe0999a0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09406cb90 con 0x7fe09406eca0 2024-03-06T07:26:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09415e220 0x7fe094155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09415e220 0x7fe094155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36184/0 (socket says 172.21.15.77:36184) 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 -- 172.21.15.77:0/2833448420 learned_addr learned my addr 172.21.15.77:0/2833448420 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe09919f640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe094162aa0 0x7fe09415f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe09899e640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe09406eca0 0x7fe0940add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 -- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe09406eca0 msgr2=0x7fe0940add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe09406eca0 0x7fe0940add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 -- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe094162aa0 msgr2=0x7fe09415f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe094162aa0 0x7fe09415f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 -- 172.21.15.77:0/2833448420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940ae2c0 con 0x7fe09415e220 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe09899e640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe09406eca0 0x7fe0940add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.872+0000 7fe093fff640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09415e220 0x7fe094155880 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe08000c3f0 tx=0x7fe08000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.873+0000 7fe0917fa640 1 -- 172.21.15.77:0/2833448420 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0800074e0 con 0x7fe09415e220 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.873+0000 7fe0917fa640 1 -- 172.21.15.77:0/2833448420 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0800047e0 con 0x7fe09415e220 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.873+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe094155dc0 con 0x7fe09415e220 2024-03-06T07:26:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.873+0000 7fe0917fa640 1 -- 172.21.15.77:0/2833448420 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080004c10 con 0x7fe09415e220 2024-03-06T07:26:51.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.873+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0940ae820 con 0x7fe09415e220 2024-03-06T07:26:51.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.874+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe094067ca0 con 0x7fe09415e220 2024-03-06T07:26:51.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.874+0000 7fe0917fa640 1 -- 172.21.15.77:0/2833448420 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe080011070 con 0x7fe09415e220 2024-03-06T07:26:51.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.875+0000 7fe0917fa640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe07806b4d0 0x7fe07806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.875+0000 7fe09899e640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe07806b4d0 0x7fe07806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.875+0000 7fe0917fa640 1 -- 172.21.15.77:0/2833448420 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fe080089e10 con 0x7fe09415e220 2024-03-06T07:26:51.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.875+0000 7fe09899e640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe07806b4d0 0x7fe07806d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe088000f60 tx=0x7fe088009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:51.879+0000 7fe0917fa640 1 -- 172.21.15.77:0/2833448420 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe094155dc0 con 0x7fe09415e220 2024-03-06T07:26:52.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.052+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe09404f330 con 0x7fe09415e220 2024-03-06T07:26:52.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.053+0000 7fe0917fa640 1 -- 172.21.15.77:0/2833448420 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16985 (secure 0 0 0) 0x7fe0940ae2c0 con 0x7fe09415e220 2024-03-06T07:26:52.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:52.055 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":96,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:21.753221+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:07.372376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:08.123063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6835/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.158:6835/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6834/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.77:6837/3425851566":"2024-03-07T07:26:21.715277+0000","172.21.15.77:6836/3425851566":"2024-03-07T07:26:21.715277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:52.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe07806b4d0 msgr2=0x7fe07806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe07806b4d0 0x7fe07806d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe088000f60 tx=0x7fe088009040 comp rx=0 tx=0).stop 2024-03-06T07:26:52.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09415e220 msgr2=0x7fe094155880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09415e220 0x7fe094155880 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe08000c3f0 tx=0x7fe08000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 shutdown_connections 2024-03-06T07:26:52.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe07806b4d0 0x7fe07806d990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe09415e220 0x7fe094155880 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe094162aa0 0x7fe09415f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 --2- 172.21.15.77:0/2833448420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe09406eca0 0x7fe0940add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 >> 172.21.15.77:0/2833448420 conn(0x7fe0940b2930 msgr2=0x7fe09406e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 shutdown_connections 2024-03-06T07:26:52.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.055+0000 7fe0999a0640 1 -- 172.21.15.77:0/2833448420 wait complete. 2024-03-06T07:26:52.071 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:52.072 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.169+0000 7fd5f36ec640 1 Processor -- start 2024-03-06T07:26:52.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f36ec640 1 -- start start 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f36ec640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd5ec15e150 0x7fd5ec0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f36ec640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd5ec0cfba0 0x7fd5ec0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f36ec640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec0d0e90 0x7fd5ec0b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f36ec640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ec0d2ab0 con 0x7fd5ec0cfba0 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f36ec640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ec0d1ea0 con 0x7fd5ec15e150 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f36ec640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0d2180 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f0c87640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd5ec0cfba0 0x7fd5ec0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f0c87640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd5ec0cfba0 0x7fd5ec0d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36186/0 (socket says 172.21.15.77:36186) 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.170+0000 7fd5f0c87640 1 -- 172.21.15.77:0/975707173 learned_addr learned my addr 172.21.15.77:0/975707173 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.171+0000 7fd5f1488640 1 --2- 172.21.15.77:0/975707173 >> v2:172.21.15.158:3300/0 conn(0x7fd5ec15e150 0x7fd5ec0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.171+0000 7fd5f1c89640 1 --2- 172.21.15.77:0/975707173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec0d0e90 0x7fd5ec0b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.171+0000 7fd5f0c87640 1 -- 172.21.15.77:0/975707173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec0d0e90 msgr2=0x7fd5ec0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.171+0000 7fd5f0c87640 1 --2- 172.21.15.77:0/975707173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec0d0e90 0x7fd5ec0b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.171+0000 7fd5f0c87640 1 -- 172.21.15.77:0/975707173 >> v2:172.21.15.158:3300/0 conn(0x7fd5ec15e150 msgr2=0x7fd5ec0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.171+0000 7fd5f0c87640 1 --2- 172.21.15.77:0/975707173 >> v2:172.21.15.158:3300/0 conn(0x7fd5ec15e150 0x7fd5ec0cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.171+0000 7fd5f0c87640 1 -- 172.21.15.77:0/975707173 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0d2460 con 0x7fd5ec0cfba0 2024-03-06T07:26:52.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.171+0000 7fd5f0c87640 1 --2- 172.21.15.77:0/975707173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0cfba0 0x7fd5ec0d0950 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0b6af0 tx=0x7fd5dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580f9c44717b9a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5e3fff640 1 -- 172.21.15.77:0/975707173 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc02e070 con 0x7fd5ec0cfba0 2024-03-06T07:26:52.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5e3fff640 1 -- 172.21.15.77:0/975707173 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc037070 con 0x7fd5ec0cfba0 2024-03-06T07:26:52.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5e3fff640 1 -- 172.21.15.77:0/975707173 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc031a00 con 0x7fd5ec0cfba0 2024-03-06T07:26:52.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 -- 172.21.15.77:0/975707173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0cfba0 msgr2=0x7fd5ec0d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/975707173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0cfba0 0x7fd5ec0d0950 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0b6af0 tx=0x7fd5dc02b880 comp rx=0 tx=0).stop 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 -- 172.21.15.77:0/975707173 shutdown_connections 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/975707173 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec0d0e90 0x7fd5ec0b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/975707173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0cfba0 0x7fd5ec0d0950 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/975707173 >> v2:172.21.15.158:3300/0 conn(0x7fd5ec15e150 0x7fd5ec0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 -- 172.21.15.77:0/975707173 >> 172.21.15.77:0/975707173 conn(0x7fd5ec09d230 msgr2=0x7fd5ec0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 -- 172.21.15.77:0/975707173 shutdown_connections 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.172+0000 7fd5f36ec640 1 -- 172.21.15.77:0/975707173 wait complete. 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.173+0000 7fd5f36ec640 1 Processor -- start 2024-03-06T07:26:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f36ec640 1 -- start start 2024-03-06T07:26:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f36ec640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5ec0cfba0 0x7fd5ec1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f36ec640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0d0e90 0x7fd5ec0ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f36ec640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec15e150 0x7fd5ec0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f36ec640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0d1b30 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f36ec640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0d2f40 con 0x7fd5ec0cfba0 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f36ec640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec15e530 con 0x7fd5ec15e150 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f0c87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0d0e90 0x7fd5ec0ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f0c87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0d0e90 0x7fd5ec0ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36190/0 (socket says 172.21.15.77:36190) 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f0c87640 1 -- 172.21.15.77:0/2816337581 learned_addr learned my addr 172.21.15.77:0/2816337581 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f1488640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5ec0cfba0 0x7fd5ec1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.174+0000 7fd5f1c89640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec15e150 0x7fd5ec0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f0c87640 1 -- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec15e150 msgr2=0x7fd5ec0c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f0c87640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec15e150 0x7fd5ec0c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f0c87640 1 -- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5ec0cfba0 msgr2=0x7fd5ec1585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f0c87640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5ec0cfba0 0x7fd5ec1585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f0c87640 1 -- 172.21.15.77:0/2816337581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0c84c0 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f1488640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5ec0cfba0 0x7fd5ec1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f0c87640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0d0e90 0x7fd5ec0ca060 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc000c00 tx=0x7fd5dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5e1ffb640 1 -- 172.21.15.77:0/2816337581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc032670 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5e1ffb640 1 -- 172.21.15.77:0/2816337581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc032830 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec0c87a0 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5e1ffb640 1 -- 172.21.15.77:0/2816337581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc032c20 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.175+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec0c8d60 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.176+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec069ee0 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.177+0000 7fd5e1ffb640 1 -- 172.21.15.77:0/2816337581 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5dc0453f0 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.177+0000 7fd5e1ffb640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd5d006b4d0 0x7fd5d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.177+0000 7fd5e1ffb640 1 -- 172.21.15.77:0/2816337581 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd5dc0ac0f0 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.177+0000 7fd5f1488640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd5d006b4d0 0x7fd5d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.178+0000 7fd5f1488640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd5d006b4d0 0x7fd5d006d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009fd0 tx=0x7fd5e400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.182+0000 7fd5e1ffb640 1 -- 172.21.15.77:0/2816337581 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5dc040070 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.396+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5ec15f7b0 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.397+0000 7fd5e1ffb640 1 -- 172.21.15.77:0/2816337581 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fd5dc07af90 con 0x7fd5ec0d0e90 2024-03-06T07:26:52.399 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:52.399 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:52.399 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd5d006b4d0 msgr2=0x7fd5d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd5d006b4d0 0x7fd5d006d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009fd0 tx=0x7fd5e400b040 comp rx=0 tx=0).stop 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0d0e90 msgr2=0x7fd5ec0ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0d0e90 0x7fd5ec0ca060 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc000c00 tx=0x7fd5dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 shutdown_connections 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd5d006b4d0 0x7fd5d006d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec0d0e90 0x7fd5ec0ca060 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd5ec15e150 0x7fd5ec0c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 --2- 172.21.15.77:0/2816337581 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5ec0cfba0 0x7fd5ec1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 >> 172.21.15.77:0/2816337581 conn(0x7fd5ec09d230 msgr2=0x7fd5ec160b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 shutdown_connections 2024-03-06T07:26:52.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.399+0000 7fd5f36ec640 1 -- 172.21.15.77:0/2816337581 wait complete. 2024-03-06T07:26:52.415 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:52.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.517+0000 7f5917c37640 1 Processor -- start 2024-03-06T07:26:52.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.517+0000 7f5917c37640 1 -- start start 2024-03-06T07:26:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f5917c37640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f59100b2da0 0x7f5910162570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f5917c37640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5910162c70 con 0x7f5910161670 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f5917c37640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591006cd80 con 0x7f59100cd690 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f5917c37640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59100b3590 con 0x7f59100b2da0 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59159d3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5910161670 0x7f5910161e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57596/0 (socket says 172.21.15.77:57596) 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59159d3640 1 -- 172.21.15.77:0/2327944132 learned_addr learned my addr 172.21.15.77:0/2327944132 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59161d4640 1 --2- 172.21.15.77:0/2327944132 >> v2:172.21.15.158:3300/0 conn(0x7f59100b2da0 0x7f5910162570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356760928 0 0) 0x7f5910162c70 con 0x7f5910161670 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58fc003520 con 0x7f5910161670 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3737219956 0 0) 0x7f58fc003520 con 0x7f5910161670 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 >> v2:172.21.15.158:3300/0 conn(0x7f59100b2da0 msgr2=0x7f5910162570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 --2- 172.21.15.77:0/2327944132 >> v2:172.21.15.158:3300/0 conn(0x7f59100b2da0 0x7f5910162570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 >> v1:172.21.15.158:6789/0 conn(0x7f59100cd690 legacy=0x7f59100cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591006d060 con 0x7f5910161670 2024-03-06T07:26:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f590c001670 con 0x7f5910161670 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 >> v1:172.21.15.77:6789/0 conn(0x7f5910161670 legacy=0x7f5910161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58fc0039b0 0x7f58fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc004800 0x7f58fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58fc0051c0 0x7f58fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc00c030 con 0x7f58fc0051c0 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.518+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc00c1e0 con 0x7f58fc0039b0 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc00c3e0 con 0x7f58fc004800 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59161d4640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58fc0051c0 0x7f58fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59159d3640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58fc0039b0 0x7f58fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59151d2640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc004800 0x7f58fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59161d4640 1 -- 172.21.15.77:0/2327944132 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc004800 msgr2=0x7f58fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59161d4640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc004800 0x7f58fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59161d4640 1 -- 172.21.15.77:0/2327944132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58fc0039b0 msgr2=0x7f58fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59161d4640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f58fc0039b0 0x7f58fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59161d4640 1 -- 172.21.15.77:0/2327944132 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc003520 con 0x7f58fc0051c0 2024-03-06T07:26:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.519+0000 7f59161d4640 1 -- 172.21.15.77:0/2327944132 reap_dead start 2024-03-06T07:26:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f59161d4640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58fc0051c0 0x7f58fc00b9c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f5908009fc0 tx=0x7f58fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b896bbc1fda9569c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908009c10 con 0x7f58fc0051c0 2024-03-06T07:26:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590802cca0 con 0x7f58fc0051c0 2024-03-06T07:26:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f59149d1640 1 -- 172.21.15.77:0/2327944132 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908003bd0 con 0x7f58fc0051c0 2024-03-06T07:26:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f5917c37640 1 -- 172.21.15.77:0/2327944132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58fc0051c0 msgr2=0x7f58fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f5917c37640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58fc0051c0 0x7f58fc00b9c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f5908009fc0 tx=0x7f58fc005830 comp rx=0 tx=0).stop 2024-03-06T07:26:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f5917c37640 1 -- 172.21.15.77:0/2327944132 shutdown_connections 2024-03-06T07:26:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f5917c37640 1 --2- 172.21.15.77:0/2327944132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58fc0051c0 0x7f58fc00b9c0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f5917c37640 1 -- 172.21.15.77:0/2327944132 >> 172.21.15.77:0/2327944132 conn(0x7f59100d1d50 msgr2=0x7f59100d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f5917c37640 1 -- 172.21.15.77:0/2327944132 shutdown_connections 2024-03-06T07:26:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.520+0000 7f5917c37640 1 -- 172.21.15.77:0/2327944132 wait complete. 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f5917c37640 1 Processor -- start 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f5917c37640 1 -- start start 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f5917c37640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc0051c0 0x7f59100b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f5917c37640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f591015e1f0 0x7f5910152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f5917c37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5910161e60 0x7f59101622e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f5917c37640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591006d4c0 con 0x7f5910161e60 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f5917c37640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591006d340 con 0x7f591015e1f0 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f5917c37640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5910161a50 con 0x7f58fc0051c0 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f59161d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5910161e60 0x7f59101622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f59161d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5910161e60 0x7f59101622e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36208/0 (socket says 172.21.15.77:36208) 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.522+0000 7f59161d4640 1 -- 172.21.15.77:0/3007950965 learned_addr learned my addr 172.21.15.77:0/3007950965 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59151d2640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f591015e1f0 0x7f5910152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59159d3640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc0051c0 0x7f59100b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59161d4640 1 -- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc0051c0 msgr2=0x7f59100b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59161d4640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc0051c0 0x7f59100b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59161d4640 1 -- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f591015e1f0 msgr2=0x7f5910152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59161d4640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f591015e1f0 0x7f5910152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59161d4640 1 -- 172.21.15.77:0/3007950965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591015f820 con 0x7f5910161e60 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59159d3640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc0051c0 0x7f59100b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59161d4640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5910161e60 0x7f59101622e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f590800e200 tx=0x7f5908009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59067fc640 1 -- 172.21.15.77:0/3007950965 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908003990 con 0x7f5910161e60 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59067fc640 1 -- 172.21.15.77:0/3007950965 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5908044070 con 0x7f5910161e60 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591015e780 con 0x7f5910161e60 2024-03-06T07:26:52.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f59067fc640 1 -- 172.21.15.77:0/3007950965 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908003d90 con 0x7f5910161e60 2024-03-06T07:26:52.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.523+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591015ed40 con 0x7f5910161e60 2024-03-06T07:26:52.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.524+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5910162820 con 0x7f5910161e60 2024-03-06T07:26:52.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.525+0000 7f59067fc640 1 -- 172.21.15.77:0/3007950965 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5908005d10 con 0x7f5910161e60 2024-03-06T07:26:52.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.525+0000 7f59067fc640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58f406b500 0x7f58f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.525+0000 7f59159d3640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58f406b500 0x7f58f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.525+0000 7f59067fc640 1 -- 172.21.15.77:0/3007950965 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f59080ac8c0 con 0x7f5910161e60 2024-03-06T07:26:52.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.525+0000 7f59159d3640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58f406b500 0x7f58f406d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f590c001eb0 tx=0x7f590c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.529+0000 7f59067fc640 1 -- 172.21.15.77:0/3007950965 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f590807b760 con 0x7f5910161e60 2024-03-06T07:26:52.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.736+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5910068050 con 0x7f5910161e60 2024-03-06T07:26:52.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.737+0000 7f59067fc640 1 -- 172.21.15.77:0/3007950965 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f590807b060 con 0x7f5910161e60 2024-03-06T07:26:52.739 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:52.739 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:09.152591+0000","modified":"2024-03-06T07:26:26.774080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6985},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6985":{"gid":6985,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6837/82690693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":82690693},{"type":"v1","addr":"172.21.15.77:6837","nonce":82690693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6985,"qdb_cluster":[6985]},"id":7}]} 2024-03-06T07:26:52.739 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 58 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.739+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58f406b500 msgr2=0x7f58f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.739+0000 7f5917c37640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58f406b500 0x7f58f406d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f590c001eb0 tx=0x7f590c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.739+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5910161e60 msgr2=0x7f59101622e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.739+0000 7f5917c37640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5910161e60 0x7f59101622e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f590800e200 tx=0x7f5908009750 comp rx=0 tx=0).stop 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.740+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 shutdown_connections 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.740+0000 7f5917c37640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f58f406b500 0x7f58f406d9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.740+0000 7f5917c37640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5910161e60 0x7f59101622e0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.740+0000 7f5917c37640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f591015e1f0 0x7f5910152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.740+0000 7f5917c37640 1 --2- 172.21.15.77:0/3007950965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58fc0051c0 0x7f59100b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.740+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 >> 172.21.15.77:0/3007950965 conn(0x7f59100d1d50 msgr2=0x7f591006f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.740+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 shutdown_connections 2024-03-06T07:26:52.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.740+0000 7f5917c37640 1 -- 172.21.15.77:0/3007950965 wait complete. 2024-03-06T07:26:52.753 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:52.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.869+0000 7fb2c6102640 1 Processor -- start 2024-03-06T07:26:52.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.869+0000 7fb2c6102640 1 -- start start 2024-03-06T07:26:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2c6102640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2c0154af0 0x7fb2c00cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2c6102640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2c6102640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb2c006d9e0 0x7fb2c006de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2c6102640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c006e390 con 0x7fb2c0154af0 2024-03-06T07:26:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2c6102640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c00cf9a0 con 0x7fb2c006d9e0 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2c6102640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0070d70 con 0x7fb2c015abc0 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2c0154af0 0x7fb2c00cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2c0154af0 0x7fb2c00cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36218/0 (socket says 172.21.15.77:36218) 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bf7fe640 1 -- 172.21.15.77:0/3300109442 learned_addr learned my addr 172.21.15.77:0/3300109442 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bffff640 1 --2- 172.21.15.77:0/3300109442 >> v2:172.21.15.158:3300/0 conn(0x7fb2c006d9e0 0x7fb2c006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2beffd640 1 --2- 172.21.15.77:0/3300109442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bf7fe640 1 -- 172.21.15.77:0/3300109442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 msgr2=0x7fb2c0072aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bf7fe640 1 --2- 172.21.15.77:0/3300109442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0072aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bf7fe640 1 -- 172.21.15.77:0/3300109442 >> v2:172.21.15.158:3300/0 conn(0x7fb2c006d9e0 msgr2=0x7fb2c006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bf7fe640 1 --2- 172.21.15.77:0/3300109442 >> v2:172.21.15.158:3300/0 conn(0x7fb2c006d9e0 0x7fb2c006de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.870+0000 7fb2bf7fe640 1 -- 172.21.15.77:0/3300109442 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c00ce190 con 0x7fb2c0154af0 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2bf7fe640 1 --2- 172.21.15.77:0/3300109442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c0154af0 0x7fb2c00cdc50 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00ce360 tx=0x7fb2b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37ebc981d2700d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2be7fc640 1 -- 172.21.15.77:0/3300109442 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4035020 con 0x7fb2c0154af0 2024-03-06T07:26:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2be7fc640 1 -- 172.21.15.77:0/3300109442 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b4031640 con 0x7fb2c0154af0 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2be7fc640 1 -- 172.21.15.77:0/3300109442 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b402d040 con 0x7fb2c0154af0 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 -- 172.21.15.77:0/3300109442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c0154af0 msgr2=0x7fb2c00cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 --2- 172.21.15.77:0/3300109442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c0154af0 0x7fb2c00cdc50 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00ce360 tx=0x7fb2b402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 -- 172.21.15.77:0/3300109442 shutdown_connections 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 --2- 172.21.15.77:0/3300109442 >> v2:172.21.15.158:3300/0 conn(0x7fb2c006d9e0 0x7fb2c006de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 --2- 172.21.15.77:0/3300109442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0072aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 --2- 172.21.15.77:0/3300109442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c0154af0 0x7fb2c00cdc50 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 -- 172.21.15.77:0/3300109442 >> 172.21.15.77:0/3300109442 conn(0x7fb2c006b160 msgr2=0x7fb2c006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 -- 172.21.15.77:0/3300109442 shutdown_connections 2024-03-06T07:26:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.871+0000 7fb2c6102640 1 -- 172.21.15.77:0/3300109442 wait complete. 2024-03-06T07:26:52.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.873+0000 7fb2c6102640 1 Processor -- start 2024-03-06T07:26:52.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.873+0000 7fb2c6102640 1 -- start start 2024-03-06T07:26:52.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2c6102640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c006d9e0 0x7fb2c01648f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2c6102640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2c0154af0 0x7fb2c01574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2c6102640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2c6102640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c015b3a0 con 0x7fb2c006d9e0 2024-03-06T07:26:52.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2c6102640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c015b0c0 con 0x7fb2c0154af0 2024-03-06T07:26:52.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2c6102640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c015b680 con 0x7fb2c015abc0 2024-03-06T07:26:52.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c006d9e0 0x7fb2c01648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c006d9e0 0x7fb2c01648f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36232/0 (socket says 172.21.15.77:36232) 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 -- 172.21.15.77:0/2006749072 learned_addr learned my addr 172.21.15.77:0/2006749072 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2beffd640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2c0154af0 0x7fb2c01574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bffff640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 -- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 msgr2=0x7fb2c0159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 -- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2c0154af0 msgr2=0x7fb2c01574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2c0154af0 0x7fb2c01574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 -- 172.21.15.77:0/2006749072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c0154fe0 con 0x7fb2c006d9e0 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bffff640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2beffd640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2c0154af0 0x7fb2c01574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2bf7fe640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c006d9e0 0x7fb2c01648f0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4000c00 tx=0x7fb2b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2a7fff640 1 -- 172.21.15.77:0/2006749072 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b403aa60 con 0x7fb2c006d9e0 2024-03-06T07:26:52.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2a7fff640 1 -- 172.21.15.77:0/2006749072 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b403ac20 con 0x7fb2c006d9e0 2024-03-06T07:26:52.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c01552c0 con 0x7fb2c006d9e0 2024-03-06T07:26:52.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2a7fff640 1 -- 172.21.15.77:0/2006749072 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40443f0 con 0x7fb2c006d9e0 2024-03-06T07:26:52.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.875+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c0155880 con 0x7fb2c006d9e0 2024-03-06T07:26:52.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.876+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c0069ee0 con 0x7fb2c006d9e0 2024-03-06T07:26:52.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.877+0000 7fb2a7fff640 1 -- 172.21.15.77:0/2006749072 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2b404b060 con 0x7fb2c006d9e0 2024-03-06T07:26:52.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.877+0000 7fb2a7fff640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2ac06b4d0 0x7fb2ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.877+0000 7fb2a7fff640 1 -- 172.21.15.77:0/2006749072 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb2b40abe40 con 0x7fb2c006d9e0 2024-03-06T07:26:52.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.877+0000 7fb2beffd640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2ac06b4d0 0x7fb2ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.878+0000 7fb2beffd640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2ac06b4d0 0x7fb2ac06d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb2c01f5080 tx=0x7fb2b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:52.882+0000 7fb2a7fff640 1 -- 172.21.15.77:0/2006749072 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2b403f070 con 0x7fb2c006d9e0 2024-03-06T07:26:53.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.070+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2c0072370 con 0x7fb2c006d9e0 2024-03-06T07:26:53.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.071+0000 7fb2a7fff640 1 -- 172.21.15.77:0/2006749072 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16985 (secure 0 0 0) 0x7fb2b402f300 con 0x7fb2c006d9e0 2024-03-06T07:26:53.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:53.074 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":96,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:21.753221+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:07.372376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:08.123063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6833/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6835/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.158:6835/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6834/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.77:6837/3425851566":"2024-03-07T07:26:21.715277+0000","172.21.15.77:6836/3425851566":"2024-03-07T07:26:21.715277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2ac06b4d0 msgr2=0x7fb2ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2ac06b4d0 0x7fb2ac06d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb2c01f5080 tx=0x7fb2b0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c006d9e0 msgr2=0x7fb2c01648f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c006d9e0 0x7fb2c01648f0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4000c00 tx=0x7fb2b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 shutdown_connections 2024-03-06T07:26:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2c015abc0 0x7fb2c0159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2c0154af0 0x7fb2c01574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2ac06b4d0 0x7fb2ac06d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 --2- 172.21.15.77:0/2006749072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2c006d9e0 0x7fb2c01648f0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.073+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 >> 172.21.15.77:0/2006749072 conn(0x7fb2c006b160 msgr2=0x7fb2c00b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.074+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 shutdown_connections 2024-03-06T07:26:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.074+0000 7fb2c6102640 1 -- 172.21.15.77:0/2006749072 wait complete. 2024-03-06T07:26:53.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:53.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.190+0000 7f40c150c640 1 Processor -- start 2024-03-06T07:26:53.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.190+0000 7f40c150c640 1 -- start start 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.191+0000 7f40c150c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f40bc15bd30 0x7f40bc0b3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.191+0000 7f40c150c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f40bc0721b0 0x7f40bc06bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.191+0000 7f40c150c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc072610 0x7f40bc06c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.191+0000 7f40c150c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc0b3ab0 con 0x7f40bc0721b0 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.191+0000 7f40c150c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc0cd2b0 con 0x7f40bc15bd30 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.191+0000 7f40c150c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0cd590 con 0x7f40bc072610 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.191+0000 7f40ba7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f40bc0721b0 0x7f40bc06bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40ba7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f40bc0721b0 0x7f40bc06bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36238/0 (socket says 172.21.15.77:36238) 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2913722422 learned_addr learned my addr 172.21.15.77:0/2913722422 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40baffd640 1 --2- 172.21.15.77:0/2913722422 >> v2:172.21.15.158:3300/0 conn(0x7f40bc15bd30 0x7f40bc0b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40bb7fe640 1 --2- 172.21.15.77:0/2913722422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc072610 0x7f40bc06c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2913722422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc072610 msgr2=0x7f40bc06c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40ba7fc640 1 --2- 172.21.15.77:0/2913722422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc072610 0x7f40bc06c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2913722422 >> v2:172.21.15.158:3300/0 conn(0x7f40bc15bd30 msgr2=0x7f40bc0b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40ba7fc640 1 --2- 172.21.15.77:0/2913722422 >> v2:172.21.15.158:3300/0 conn(0x7f40bc15bd30 0x7f40bc0b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2913722422 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc0cefd0 con 0x7f40bc0721b0 2024-03-06T07:26:53.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.192+0000 7f40ba7fc640 1 --2- 172.21.15.77:0/2913722422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc0721b0 0x7f40bc06bf90 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f40bc04f3a0 tx=0x7f40b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39c00383f38c5da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40b9ffb640 1 -- 172.21.15.77:0/2913722422 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b002e070 con 0x7f40bc0721b0 2024-03-06T07:26:53.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40b9ffb640 1 -- 172.21.15.77:0/2913722422 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b0037070 con 0x7f40bc0721b0 2024-03-06T07:26:53.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40b9ffb640 1 -- 172.21.15.77:0/2913722422 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0031a00 con 0x7f40bc0721b0 2024-03-06T07:26:53.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 -- 172.21.15.77:0/2913722422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc0721b0 msgr2=0x7f40bc06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 --2- 172.21.15.77:0/2913722422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc0721b0 0x7f40bc06bf90 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f40bc04f3a0 tx=0x7f40b002b880 comp rx=0 tx=0).stop 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 -- 172.21.15.77:0/2913722422 shutdown_connections 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 --2- 172.21.15.77:0/2913722422 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc072610 0x7f40bc06c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 --2- 172.21.15.77:0/2913722422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc0721b0 0x7f40bc06bf90 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 --2- 172.21.15.77:0/2913722422 >> v2:172.21.15.158:3300/0 conn(0x7f40bc15bd30 0x7f40bc0b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 -- 172.21.15.77:0/2913722422 >> 172.21.15.77:0/2913722422 conn(0x7f40bc071690 msgr2=0x7f40bc071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 -- 172.21.15.77:0/2913722422 shutdown_connections 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.193+0000 7f40c150c640 1 -- 172.21.15.77:0/2913722422 wait complete. 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40c150c640 1 Processor -- start 2024-03-06T07:26:53.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40c150c640 1 -- start start 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40c150c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc0721b0 0x7f40bc146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40c150c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40bc072610 0x7f40bc156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40c150c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc15bd30 0x7f40bc15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40c150c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0cec80 con 0x7f40bc15bd30 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40c150c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc15c110 con 0x7f40bc072610 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40c150c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0ccf10 con 0x7f40bc0721b0 2024-03-06T07:26:53.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc15bd30 0x7f40bc15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc15bd30 0x7f40bc15c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36240/0 (socket says 172.21.15.77:36240) 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.195+0000 7f40bb7fe640 1 -- 172.21.15.77:0/3772125904 learned_addr learned my addr 172.21.15.77:0/3772125904 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40ba7fc640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40bc072610 0x7f40bc156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40baffd640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc0721b0 0x7f40bc146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40bb7fe640 1 -- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc0721b0 msgr2=0x7f40bc146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40bb7fe640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc0721b0 0x7f40bc146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40bb7fe640 1 -- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40bc072610 msgr2=0x7f40bc156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40bb7fe640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40bc072610 0x7f40bc156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40bb7fe640 1 -- 172.21.15.77:0/3772125904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc157c70 con 0x7f40bc15bd30 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40baffd640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc0721b0 0x7f40bc146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40bb7fe640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc15bd30 0x7f40bc15c450 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f40b400d5d0 tx=0x7f40b400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40a37fe640 1 -- 172.21.15.77:0/3772125904 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b40125f0 con 0x7f40bc15bd30 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40a37fe640 1 -- 172.21.15.77:0/3772125904 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b4019070 con 0x7f40bc15bd30 2024-03-06T07:26:53.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40a37fe640 1 -- 172.21.15.77:0/3772125904 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b40129f0 con 0x7f40bc15bd30 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc1576b0 con 0x7f40bc15bd30 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.196+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc150860 con 0x7f40bc15bd30 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.197+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc068050 con 0x7f40bc15bd30 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.197+0000 7f40a37fe640 1 -- 172.21.15.77:0/3772125904 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40b4012b90 con 0x7f40bc15bd30 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.198+0000 7f40a37fe640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f40a806b4d0 0x7f40a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.198+0000 7f40a37fe640 1 -- 172.21.15.77:0/3772125904 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f40b408a6a0 con 0x7f40bc15bd30 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.198+0000 7f40baffd640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f40a806b4d0 0x7f40a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.198+0000 7f40baffd640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f40a806b4d0 0x7f40a806d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009ed0 tx=0x7f40ac00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.206+0000 7f40a37fe640 1 -- 172.21.15.77:0/3772125904 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40b4059430 con 0x7f40bc15bd30 2024-03-06T07:26:53.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.449+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f40bc06ffa0 con 0x7f40bc15bd30 2024-03-06T07:26:53.451 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:26:53.450+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:53.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.451+0000 7f40a37fe640 1 -- 172.21.15.77:0/3772125904 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v59) v1 ==== 161+0+0 (secure 0 0 0) 0x7f40b4058d30 con 0x7f40bc15bd30 2024-03-06T07:26:53.453 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:53.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.454+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f40a806b4d0 msgr2=0x7f40a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.454+0000 7f40c150c640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f40a806b4d0 0x7f40a806d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009ed0 tx=0x7f40ac00b040 comp rx=0 tx=0).stop 2024-03-06T07:26:53.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.454+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc15bd30 msgr2=0x7f40bc15c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.454+0000 7f40c150c640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc15bd30 0x7f40bc15c450 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f40b400d5d0 tx=0x7f40b400da90 comp rx=0 tx=0).stop 2024-03-06T07:26:53.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.454+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 shutdown_connections 2024-03-06T07:26:53.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.454+0000 7f40c150c640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f40a806b4d0 0x7f40a806d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.454+0000 7f40c150c640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40bc15bd30 0x7f40bc15c450 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.455+0000 7f40c150c640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40bc072610 0x7f40bc156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.455+0000 7f40c150c640 1 --2- 172.21.15.77:0/3772125904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40bc0721b0 0x7f40bc146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.455+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 >> 172.21.15.77:0/3772125904 conn(0x7f40bc071690 msgr2=0x7f40bc0b4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.455+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 shutdown_connections 2024-03-06T07:26:53.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.455+0000 7f40c150c640 1 -- 172.21.15.77:0/3772125904 wait complete. 2024-03-06T07:26:53.468 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:53.496 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.569+0000 7f11ccaf4640 1 Processor -- start 2024-03-06T07:26:53.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.569+0000 7f11ccaf4640 1 -- start start 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11ccaf4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11c815e150 0x7f11c80cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11ccaf4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f11c80cfba0 0x7f11c80d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11ccaf4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80d0e90 0x7f11c80b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11ccaf4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c80d2ab0 con 0x7f11c815e150 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11ccaf4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c80d1ea0 con 0x7f11c80cfba0 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11ccaf4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80d2180 con 0x7f11c80d0e90 2024-03-06T07:26:53.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11c659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11c815e150 0x7f11c80cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11c659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11c815e150 0x7f11c80cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36274/0 (socket says 172.21.15.77:36274) 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.570+0000 7f11c659c640 1 -- 172.21.15.77:0/3840519332 learned_addr learned my addr 172.21.15.77:0/3840519332 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.571+0000 7f11c659c640 1 -- 172.21.15.77:0/3840519332 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80d0e90 msgr2=0x7f11c80b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.571+0000 7f11c659c640 1 --2- 172.21.15.77:0/3840519332 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80d0e90 0x7f11c80b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.571+0000 7f11c659c640 1 -- 172.21.15.77:0/3840519332 >> v2:172.21.15.158:3300/0 conn(0x7f11c80cfba0 msgr2=0x7f11c80d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.571+0000 7f11c659c640 1 --2- 172.21.15.77:0/3840519332 >> v2:172.21.15.158:3300/0 conn(0x7f11c80cfba0 0x7f11c80d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.571+0000 7f11c659c640 1 -- 172.21.15.77:0/3840519332 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80d2460 con 0x7f11c815e150 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.571+0000 7f11c659c640 1 --2- 172.21.15.77:0/3840519332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c80cf660 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f11c80b6b50 tx=0x7f11b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6f2b91476fd264 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.571+0000 7f11c5d9b640 1 -- 172.21.15.77:0/3840519332 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b802e070 con 0x7f11c815e150 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.571+0000 7f11c5d9b640 1 -- 172.21.15.77:0/3840519332 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b8037070 con 0x7f11c815e150 2024-03-06T07:26:53.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11c5d9b640 1 -- 172.21.15.77:0/3840519332 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8031a00 con 0x7f11c815e150 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 -- 172.21.15.77:0/3840519332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 msgr2=0x7f11c80cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/3840519332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c80cf660 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f11c80b6b50 tx=0x7f11b802b880 comp rx=0 tx=0).stop 2024-03-06T07:26:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 -- 172.21.15.77:0/3840519332 shutdown_connections 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/3840519332 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80d0e90 0x7f11c80b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/3840519332 >> v2:172.21.15.158:3300/0 conn(0x7f11c80cfba0 0x7f11c80d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/3840519332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c80cf660 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 -- 172.21.15.77:0/3840519332 >> 172.21.15.77:0/3840519332 conn(0x7f11c809d230 msgr2=0x7f11c80d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 -- 172.21.15.77:0/3840519332 shutdown_connections 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.572+0000 7f11ccaf4640 1 -- 172.21.15.77:0/3840519332 wait complete. 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.573+0000 7f11ccaf4640 1 Processor -- start 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.573+0000 7f11ccaf4640 1 -- start start 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11ccaf4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80cfba0 0x7f11c8154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11ccaf4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c80d0e90 0x7f11c809e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11ccaf4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c8160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11ccaf4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80d1b30 con 0x7f11c815e150 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11ccaf4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80d2f40 con 0x7f11c80d0e90 2024-03-06T07:26:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11ccaf4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c815e530 con 0x7f11c80cfba0 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c8160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c8160bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36278/0 (socket says 172.21.15.77:36278) 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c6d9d640 1 -- 172.21.15.77:0/1971462899 learned_addr learned my addr 172.21.15.77:0/1971462899 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c659c640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80cfba0 0x7f11c8154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11bffff640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c80d0e90 0x7f11c809e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c6d9d640 1 -- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80cfba0 msgr2=0x7f11c8154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c6d9d640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80cfba0 0x7f11c8154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c6d9d640 1 -- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c80d0e90 msgr2=0x7f11c809e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c6d9d640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c80d0e90 0x7f11c809e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c6d9d640 1 -- 172.21.15.77:0/1971462899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80a12a0 con 0x7f11c815e150 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11bffff640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c80d0e90 0x7f11c809e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.574+0000 7f11c659c640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80cfba0 0x7f11c8154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.575+0000 7f11c6d9d640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c8160bf0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f11c000b970 tx=0x7f11c000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.575+0000 7f11beffd640 1 -- 172.21.15.77:0/1971462899 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0019070 con 0x7f11c815e150 2024-03-06T07:26:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.575+0000 7f11beffd640 1 -- 172.21.15.77:0/1971462899 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c000cb60 con 0x7f11c815e150 2024-03-06T07:26:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.575+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c80a1580 con 0x7f11c815e150 2024-03-06T07:26:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.575+0000 7f11beffd640 1 -- 172.21.15.77:0/1971462899 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0014440 con 0x7f11c815e150 2024-03-06T07:26:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.575+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c80a1b40 con 0x7f11c815e150 2024-03-06T07:26:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.576+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c8069ee0 con 0x7f11c815e150 2024-03-06T07:26:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.576+0000 7f11beffd640 1 -- 172.21.15.77:0/1971462899 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11c001c070 con 0x7f11c815e150 2024-03-06T07:26:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.577+0000 7f11beffd640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11a806b4d0 0x7f11a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.577+0000 7f11c659c640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11a806b4d0 0x7f11a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.577+0000 7f11beffd640 1 -- 172.21.15.77:0/1971462899 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f11c008a090 con 0x7f11c815e150 2024-03-06T07:26:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.577+0000 7f11c659c640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11a806b4d0 0x7f11a806d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f11b8036000 tx=0x7f11b8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.581+0000 7f11beffd640 1 -- 172.21.15.77:0/1971462899 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11c0058dc0 con 0x7f11c815e150 2024-03-06T07:26:53.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:53.790+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f11c815f280 con 0x7f11c815e150 2024-03-06T07:26:54.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.452+0000 7f11beffd640 1 -- 172.21.15.77:0/1971462899 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7f11c00586c0 con 0x7f11c815e150 2024-03-06T07:26:54.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11a806b4d0 msgr2=0x7f11a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11a806b4d0 0x7f11a806d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f11b8036000 tx=0x7f11b8008030 comp rx=0 tx=0).stop 2024-03-06T07:26:54.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 msgr2=0x7f11c8160bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c8160bf0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f11c000b970 tx=0x7f11c000be30 comp rx=0 tx=0).stop 2024-03-06T07:26:54.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 shutdown_connections 2024-03-06T07:26:54.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11a806b4d0 0x7f11a806d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c815e150 0x7f11c8160bf0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11c80d0e90 0x7f11c809e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 --2- 172.21.15.77:0/1971462899 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11c80cfba0 0x7f11c8154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 >> 172.21.15.77:0/1971462899 conn(0x7f11c809d230 msgr2=0x7f11c806ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 shutdown_connections 2024-03-06T07:26:54.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.455+0000 7f11ccaf4640 1 -- 172.21.15.77:0/1971462899 wait complete. 2024-03-06T07:26:54.470 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.617+0000 7f20a21e9640 1 Processor -- start 2024-03-06T07:26:54.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.617+0000 7f20a21e9640 1 -- start start 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.617+0000 7f20a21e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f209c0cdb30 0x7f209c15c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.617+0000 7f20a21e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f209c15c970 0x7f209c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.617+0000 7f20a21e9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c0cdf90 con 0x7f209c145900 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.617+0000 7f20a21e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209c06d150 con 0x7f209c15c970 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.617+0000 7f20a21e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209c06d3a0 con 0x7f209c0cdb30 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f209c145900 0x7f209c0cf610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57598/0 (socket says 172.21.15.77:57598) 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209b7fe640 1 -- 172.21.15.77:0/1709477577 learned_addr learned my addr 172.21.15.77:0/1709477577 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209affd640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f209c0cdb30 0x7f209c15c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209bfff640 1 --2- 172.21.15.77:0/1709477577 >> v2:172.21.15.158:3300/0 conn(0x7f209c15c970 0x7f209c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618325264 0 0) 0x7f209c0cdf90 con 0x7f209c145900 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2084003520 con 0x7f209c145900 2024-03-06T07:26:54.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3717393458 0 0) 0x7f2084003520 con 0x7f209c145900 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f209c0cdb30 msgr2=0x7f209c15c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f209c0cdb30 0x7f209c15c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 >> v2:172.21.15.158:3300/0 conn(0x7f209c15c970 msgr2=0x7f209c06e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 --2- 172.21.15.77:0/1709477577 >> v2:172.21.15.158:3300/0 conn(0x7f209c15c970 0x7f209c06e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c06d680 con 0x7f209c145900 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f208c001670 con 0x7f209c145900 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 >> v1:172.21.15.77:6789/0 conn(0x7f209c145900 legacy=0x7f209c0cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f209c0cdb30 0x7f20840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084003f30 0x7f2084004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2084004d40 0x7f20840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c06d150 con 0x7f2084003f30 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c06d3a0 con 0x7f2084004d40 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208400baa0 con 0x7f209c0cdb30 2024-03-06T07:26:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209affd640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084003f30 0x7f2084004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209bfff640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2084004d40 0x7f20840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.618+0000 7f209b7fe640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f209c0cdb30 0x7f20840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209affd640 1 -- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f209c0cdb30 msgr2=0x7f20840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209affd640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f209c0cdb30 0x7f20840039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209affd640 1 -- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2084004d40 msgr2=0x7f20840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209affd640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2084004d40 0x7f20840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209affd640 1 -- 172.21.15.77:0/1709477577 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084003520 con 0x7f2084003f30 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209affd640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084003f30 0x7f2084004800 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2090009ac0 tx=0x7f2084005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88223250d9baa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090039070 con 0x7f2084003f30 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209002c6b0 con 0x7f2084003f30 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.619+0000 7f209a7fc640 1 -- 172.21.15.77:0/1709477577 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090009690 con 0x7f2084003f30 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.620+0000 7f20a21e9640 1 -- 172.21.15.77:0/1709477577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084003f30 msgr2=0x7f2084004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.620+0000 7f20a21e9640 1 --2- 172.21.15.77:0/1709477577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084003f30 0x7f2084004800 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2090009ac0 tx=0x7f2084005830 comp rx=0 tx=0).stop 2024-03-06T07:26:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.620+0000 7f209bfff640 1 -- 172.21.15.77:0/1709477577 reap_dead start 2024-03-06T07:26:54.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.620+0000 7f20a21e9640 1 -- 172.21.15.77:0/1709477577 shutdown_connections 2024-03-06T07:26:54.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.620+0000 7f20a21e9640 1 -- 172.21.15.77:0/1709477577 >> 172.21.15.77:0/1709477577 conn(0x7f209c06b160 msgr2=0x7f209c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.620+0000 7f20a21e9640 1 -- 172.21.15.77:0/1709477577 shutdown_connections 2024-03-06T07:26:54.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.620+0000 7f20a21e9640 1 -- 172.21.15.77:0/1709477577 wait complete. 2024-03-06T07:26:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.621+0000 7f20a21e9640 1 Processor -- start 2024-03-06T07:26:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.621+0000 7f20a21e9640 1 -- start start 2024-03-06T07:26:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f20a21e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2084003f30 0x7f209c13d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f20a21e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f209c071080 0x7f209c15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f20a21e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f209c072820 0x7f209c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f20a21e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c15c430 con 0x7f209c072820 2024-03-06T07:26:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f20a21e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c06dc40 con 0x7f209c071080 2024-03-06T07:26:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f20a21e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209c15c5b0 con 0x7f2084003f30 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f209bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f209c072820 0x7f209c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f209bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f209c072820 0x7f209c0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36284/0 (socket says 172.21.15.77:36284) 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f209bfff640 1 -- 172.21.15.77:0/951647559 learned_addr learned my addr 172.21.15.77:0/951647559 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f209bfff640 1 -- 172.21.15.77:0/951647559 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2084003f30 msgr2=0x7f209c13d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f209bfff640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2084003f30 0x7f209c13d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f209bfff640 1 -- 172.21.15.77:0/951647559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f209c071080 msgr2=0x7f209c15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f209bfff640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f209c071080 0x7f209c15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.622+0000 7f209bfff640 1 -- 172.21.15.77:0/951647559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c13a630 con 0x7f209c072820 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.623+0000 7f209bfff640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f209c072820 0x7f209c0b0b90 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f209c070780 tx=0x7f20940115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.623+0000 7f2083fff640 1 -- 172.21.15.77:0/951647559 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2094011e10 con 0x7f209c072820 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.623+0000 7f2083fff640 1 -- 172.21.15.77:0/951647559 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209401b070 con 0x7f209c072820 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.623+0000 7f2083fff640 1 -- 172.21.15.77:0/951647559 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2094003990 con 0x7f209c072820 2024-03-06T07:26:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.623+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209c071500 con 0x7f209c072820 2024-03-06T07:26:54.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.623+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209c071a60 con 0x7f209c072820 2024-03-06T07:26:54.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.624+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209c069ee0 con 0x7f209c072820 2024-03-06T07:26:54.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.624+0000 7f2083fff640 1 -- 172.21.15.77:0/951647559 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2094005d10 con 0x7f209c072820 2024-03-06T07:26:54.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.624+0000 7f2083fff640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f208806b4d0 0x7f208806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.625+0000 7f2083fff640 1 -- 172.21.15.77:0/951647559 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f209408a880 con 0x7f209c072820 2024-03-06T07:26:54.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.625+0000 7f209b7fe640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f208806b4d0 0x7f208806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.625+0000 7f209b7fe640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f208806b4d0 0x7f208806d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f208c001eb0 tx=0x7f208c000ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.629+0000 7f2083fff640 1 -- 172.21.15.77:0/951647559 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f209c071500 con 0x7f209c072820 2024-03-06T07:26:54.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:54.800+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f209c0b39d0 con 0x7f209c072820 2024-03-06T07:26:55.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.454+0000 7f2083fff640 1 -- 172.21.15.77:0/951647559 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) v1 ==== 181+0+0 (secure 0 0 0) 0x7f209c13a630 con 0x7f209c072820 2024-03-06T07:26:55.456 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.457+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f208806b4d0 msgr2=0x7f208806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.457+0000 7f20a21e9640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f208806b4d0 0x7f208806d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f208c001eb0 tx=0x7f208c000ef0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.457+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f209c072820 msgr2=0x7f209c0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.457+0000 7f20a21e9640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f209c072820 0x7f209c0b0b90 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f209c070780 tx=0x7f20940115a0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.458+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 shutdown_connections 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.458+0000 7f20a21e9640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f208806b4d0 0x7f208806d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.458+0000 7f20a21e9640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f209c072820 0x7f209c0b0b90 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.458+0000 7f20a21e9640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f209c071080 0x7f209c15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.458+0000 7f20a21e9640 1 --2- 172.21.15.77:0/951647559 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2084003f30 0x7f209c13d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.458+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 >> 172.21.15.77:0/951647559 conn(0x7f209c06b160 msgr2=0x7f209c1437c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.458+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 shutdown_connections 2024-03-06T07:26:55.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.458+0000 7f20a21e9640 1 -- 172.21.15.77:0/951647559 wait complete. 2024-03-06T07:26:55.472 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.576+0000 7f86e1563640 1 Processor -- start 2024-03-06T07:26:55.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.576+0000 7f86e1563640 1 -- start start 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86e1563640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc161a80 0x7f86dc0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86e1563640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86dc0d25a0 0x7f86dc0d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86e1563640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc161e60 con 0x7f86dc0b5010 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86e1563640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc0b1250 con 0x7f86dc0d25a0 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86e1563640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0b3e30 con 0x7f86dc161a80 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86da7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f86dc0b5010 0x7f86dc0b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57606/0 (socket says 172.21.15.77:57606) 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86da7fc640 1 -- 172.21.15.77:0/2357034467 learned_addr learned my addr 172.21.15.77:0/2357034467 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263775296 0 0) 0x7f86d0001110 con 0x7f86dc0b5010 2024-03-06T07:26:55.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.577+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c4003520 con 0x7f86dc0b5010 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86daffd640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc161a80 0x7f86dc0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86db7fe640 1 --2- 172.21.15.77:0/2357034467 >> v2:172.21.15.158:3300/0 conn(0x7f86dc0d25a0 0x7f86dc0d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3422112819 0 0) 0x7f86c4003520 con 0x7f86dc0b5010 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc161a80 msgr2=0x7f86dc0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc161a80 0x7f86dc0d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 >> v2:172.21.15.158:3300/0 conn(0x7f86dc0d25a0 msgr2=0x7f86dc0d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 --2- 172.21.15.77:0/2357034467 >> v2:172.21.15.158:3300/0 conn(0x7f86dc0d25a0 0x7f86dc0d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0d1470 con 0x7f86dc0b5010 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f86d0001840 con 0x7f86dc0b5010 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86db7fe640 1 --2- 172.21.15.77:0/2357034467 >> v2:172.21.15.158:3300/0 conn(0x7f86dc0d25a0 0x7f86dc0d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 >> v1:172.21.15.77:6789/0 conn(0x7f86dc0b5010 legacy=0x7f86dc0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc161a80 0x7f86c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c4004d40 0x7f86c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0b1250 con 0x7f86c40039b0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0b3e30 con 0x7f86c4004d40 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c400ba70 con 0x7f86dc161a80 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86daffd640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86db7fe640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c4004d40 0x7f86c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86da7fc640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc161a80 0x7f86c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86daffd640 1 -- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc161a80 msgr2=0x7f86c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86daffd640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc161a80 0x7f86c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86daffd640 1 -- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c4004d40 msgr2=0x7f86c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86daffd640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c4004d40 0x7f86c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86daffd640 1 -- 172.21.15.77:0/2357034467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4003520 con 0x7f86c40039b0 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.578+0000 7f86db7fe640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c4004d40 0x7f86c40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.579+0000 7f86daffd640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f86cc009fc0 tx=0x7f86c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd8a07e0ebf8263 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.579+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc039070 con 0x7f86c40039b0 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.579+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc02cc20 con 0x7f86c40039b0 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.579+0000 7f86d9ffb640 1 -- 172.21.15.77:0/2357034467 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc009c10 con 0x7f86c40039b0 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.579+0000 7f86e1563640 1 -- 172.21.15.77:0/2357034467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 msgr2=0x7f86c4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.579+0000 7f86e1563640 1 --2- 172.21.15.77:0/2357034467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f86cc009fc0 tx=0x7f86c4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.580+0000 7f86db7fe640 1 -- 172.21.15.77:0/2357034467 reap_dead start 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.580+0000 7f86e1563640 1 -- 172.21.15.77:0/2357034467 shutdown_connections 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.580+0000 7f86e1563640 1 -- 172.21.15.77:0/2357034467 >> 172.21.15.77:0/2357034467 conn(0x7f86dc0d3fc0 msgr2=0x7f86dc0d43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.580+0000 7f86e1563640 1 -- 172.21.15.77:0/2357034467 shutdown_connections 2024-03-06T07:26:55.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.580+0000 7f86e1563640 1 -- 172.21.15.77:0/2357034467 wait complete. 2024-03-06T07:26:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.581+0000 7f86e1563640 1 Processor -- start 2024-03-06T07:26:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.581+0000 7f86e1563640 1 -- start start 2024-03-06T07:26:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86e1563640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86dc0cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86e1563640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86dc0b8d00 0x7f86dc06cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86e1563640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc0b6be0 0x7f86dc0b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86e1563640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0d1090 con 0x7f86c40039b0 2024-03-06T07:26:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86e1563640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0b3ac0 con 0x7f86dc0b8d00 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86e1563640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0d1210 con 0x7f86dc0b6be0 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86dc0cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86dc0cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36308/0 (socket says 172.21.15.77:36308) 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 -- 172.21.15.77:0/3806733148 learned_addr learned my addr 172.21.15.77:0/3806733148 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86db7fe640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc0b6be0 0x7f86dc0b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86da7fc640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86dc0b8d00 0x7f86dc06cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 -- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc0b6be0 msgr2=0x7f86dc0b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc0b6be0 0x7f86dc0b7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 -- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86dc0b8d00 msgr2=0x7f86dc06cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86dc0b8d00 0x7f86dc06cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 -- 172.21.15.77:0/3806733148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0cc2f0 con 0x7f86c40039b0 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86da7fc640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86dc0b8d00 0x7f86dc06cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86daffd640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86dc0cbaa0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00e230 tx=0x7f86cc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86c37fe640 1 -- 172.21.15.77:0/3806733148 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc039050 con 0x7f86c40039b0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86c37fe640 1 -- 172.21.15.77:0/3806733148 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc03a040 con 0x7f86c40039b0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86c37fe640 1 -- 172.21.15.77:0/3806733148 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc003990 con 0x7f86c40039b0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc0cc5d0 con 0x7f86c40039b0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.582+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc0ccb90 con 0x7f86c40039b0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.583+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc068050 con 0x7f86c40039b0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.584+0000 7f86c37fe640 1 -- 172.21.15.77:0/3806733148 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86cc047030 con 0x7f86c40039b0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.584+0000 7f86c37fe640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86c806b500 0x7f86c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.584+0000 7f86c37fe640 1 -- 172.21.15.77:0/3806733148 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5615+0+0 (secure 0 0 0) 0x7f86cc0ac7f0 con 0x7f86c40039b0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.584+0000 7f86da7fc640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86c806b500 0x7f86c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.585+0000 7f86da7fc640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86c806b500 0x7f86c806d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f86d0001a30 tx=0x7f86d0000ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.588+0000 7f86c37fe640 1 -- 172.21.15.77:0/3806733148 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86cc04e020 con 0x7f86c40039b0 2024-03-06T07:26:55.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:55.759+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86dc0d2380 con 0x7f86c40039b0 2024-03-06T07:26:56.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.459+0000 7f86c37fe640 1 -- 172.21.15.77:0/3806733148 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) v1 ==== 169+0+0 (secure 0 0 0) 0x7f86cc07b760 con 0x7f86c40039b0 2024-03-06T07:26:56.461 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86c806b500 msgr2=0x7f86c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86c806b500 0x7f86c806d9c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f86d0001a30 tx=0x7f86d0000ef0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 msgr2=0x7f86dc0cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86dc0cbaa0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00e230 tx=0x7f86cc009750 comp rx=0 tx=0).stop 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 shutdown_connections 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86dc0b6be0 0x7f86dc0b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86dc0b8d00 0x7f86dc06cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86c806b500 0x7f86c806d9c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 --2- 172.21.15.77:0/3806733148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40039b0 0x7f86dc0cbaa0 secure :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00e230 tx=0x7f86cc009750 comp rx=0 tx=0).stop 2024-03-06T07:26:56.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 >> 172.21.15.77:0/3806733148 conn(0x7f86dc0d3fc0 msgr2=0x7f86dc06c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 shutdown_connections 2024-03-06T07:26:56.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.463+0000 7f86e1563640 1 -- 172.21.15.77:0/3806733148 wait complete. 2024-03-06T07:26:56.480 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:56.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:56.531 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-06T07:26:56.531 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-03-06T07:26:56.531 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:56.531 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:56.532 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.538 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.543 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:56.544 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2024-03-06T07:26:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug trim objects = true 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = bitmap 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: mon op complaint time = 120 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: client mount timeout = 600 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2024-03-06T07:26:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = "" 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.b] 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.c] 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2024-03-06T07:26:56.555 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout:[mds] 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds = 20 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds balancer = 20 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug frag = True 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug scatterstat = True 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: mds op complaint time = 180 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: mds verify scatter = True 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:56.556 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:56.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:56.601 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:56.648 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-03-06T07:26:56.649 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:56.650 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-03-06T07:26:56.651 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-03-06T07:26:56.652 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.653 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:56.654 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-03-06T07:26:56.655 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.656 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:56.657 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-03-06T07:26:56.658 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi077.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:56.659 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.660 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2024-03-06T07:26:56.661 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:56.662 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug trim objects = true 2024-03-06T07:26:56.663 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2024-03-06T07:26:56.664 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = bitmap 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2024-03-06T07:26:56.665 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2024-03-06T07:26:56.666 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:56.667 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: mon op complaint time = 120 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: client mount timeout = 600 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2024-03-06T07:26:56.668 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = "" 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.b] 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.c] 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout:[mds] 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds = 20 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds balancer = 20 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.669 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug frag = True 2024-03-06T07:26:56.670 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug scatterstat = True 2024-03-06T07:26:56.670 INFO:teuthology.orchestra.run.smithi158.stdout: mds op complaint time = 180 2024-03-06T07:26:56.670 INFO:teuthology.orchestra.run.smithi158.stdout: mds verify scatter = True 2024-03-06T07:26:56.670 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:56.670 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.670 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.670 INFO:teuthology.orchestra.run.smithi158.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:56.672 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:56.714 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:56.762 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-06T07:26:56.762 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-03-06T07:26:56.762 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:56.762 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-03-06T07:26:56.762 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.762 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-03-06T07:26:56.762 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:56.763 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-03-06T07:26:56.764 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-03-06T07:26:56.765 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-03-06T07:26:56.766 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-03-06T07:26:56.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-03-06T07:26:56.768 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:56.769 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-03-06T07:26:56.770 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.771 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-03-06T07:26:56.772 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-03-06T07:26:56.773 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.774 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.775 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2024-03-06T07:26:56.775 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2024-03-06T07:26:56.775 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:56.775 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2024-03-06T07:26:56.775 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.775 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2024-03-06T07:26:56.775 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.775 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:56.776 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:56.777 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = "172.21.15.77,172.21.15.158,[v2:172.21.15.158:3301,v1:172.21.15.158:6790]" 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = eee1dc2e-9e65-49b7-aabb-947580759696 2024-03-06T07:26:56.778 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:56.779 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug trim objects = true 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:56.780 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: bdev async discard = True 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: bdev enable discard = True 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore allocator = bitmap 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore block size = 96636764160 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: bluestore fsck on mount = True 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluefs = 1/20 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: debug bluestore = 1/20 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: debug rocksdb = 4/10 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:56.781 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: osd objectstore = bluestore 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.782 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.783 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:56.784 INFO:teuthology.orchestra.run.smithi158.stdout: mon op complaint time = 120 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: client mount timeout = 600 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: debug client = 20 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = "" 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.785 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.b] 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.c] 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.a] 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.c] 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.b] 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout:[mds.d] 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout:[mds] 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds = 20 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout: debug mds balancer = 20 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug frag = True 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout: mds debug scatterstat = True 2024-03-06T07:26:56.786 INFO:teuthology.orchestra.run.smithi158.stdout: mds op complaint time = 180 2024-03-06T07:26:56.787 INFO:teuthology.orchestra.run.smithi158.stdout: mds verify scatter = True 2024-03-06T07:26:56.787 INFO:teuthology.orchestra.run.smithi158.stdout: osd op complaint time = 180 2024-03-06T07:26:56.787 INFO:teuthology.orchestra.run.smithi158.stdout: rados mon op timeout = 900 2024-03-06T07:26:56.787 INFO:teuthology.orchestra.run.smithi158.stdout: rados osd op timeout = 900 2024-03-06T07:26:56.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.932+0000 7f322a0bb640 1 Processor -- start 2024-03-06T07:26:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.932+0000 7f322a0bb640 1 -- start start 2024-03-06T07:26:56.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.932+0000 7f322a0bb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32240cff40 0x7f32240b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.932+0000 7f322a0bb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3224158ac0 con 0x7f32241586e0 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.932+0000 7f322a0bb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322415b2a0 con 0x7f32240cfa90 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.932+0000 7f322a0bb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322415b540 con 0x7f32240cff40 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32237fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f32241586e0 0x7f32240aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57610/0 (socket says 172.21.15.77:57610) 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32237fe640 1 -- 172.21.15.77:0/2134237381 learned_addr learned my addr 172.21.15.77:0/2134237381 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210206990 0 0) 0x7f3224158ac0 con 0x7f32241586e0 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f3223fff640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32240cff40 0x7f32240b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f320c003520 con 0x7f32241586e0 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2685283144 0 0) 0x7f320c003520 con 0x7f32241586e0 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32240cff40 msgr2=0x7f32240b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32240cff40 0x7f32240b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 >> v1:172.21.15.158:6789/0 conn(0x7f32240cfa90 legacy=0x7f32240b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322415b820 con 0x7f32241586e0 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3214001670 con 0x7f32241586e0 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 >> v1:172.21.15.77:6789/0 conn(0x7f32241586e0 legacy=0x7f32240aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f320c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f320c004800 0x7f320c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32240cff40 0x7f320c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322415b540 con 0x7f320c0039e0 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c00ba70 con 0x7f320c004800 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c00bc70 con 0x7f32240cff40 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f32237fe640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f320c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f3223fff640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32240cff40 0x7f320c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.933+0000 7f3222ffd640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f320c004800 0x7f320c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32237fe640 1 -- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32240cff40 msgr2=0x7f320c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32237fe640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32240cff40 0x7f320c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32237fe640 1 -- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f320c004800 msgr2=0x7f320c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32237fe640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f320c004800 0x7f320c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32237fe640 1 -- 172.21.15.77:0/2134237381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c003520 con 0x7f320c0039e0 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32237fe640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f320c003e00 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f321400c210 tx=0x7f320c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e871011f068e7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214007cf0 con 0x7f320c0039e0 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3214030070 con 0x7f320c0039e0 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.934+0000 7f32227fc640 1 -- 172.21.15.77:0/2134237381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214005d10 con 0x7f320c0039e0 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.935+0000 7f322a0bb640 1 -- 172.21.15.77:0/2134237381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 msgr2=0x7f320c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.935+0000 7f322a0bb640 1 --2- 172.21.15.77:0/2134237381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f320c003e00 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f321400c210 tx=0x7f320c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.935+0000 7f3223fff640 1 -- 172.21.15.77:0/2134237381 reap_dead start 2024-03-06T07:26:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.935+0000 7f322a0bb640 1 -- 172.21.15.77:0/2134237381 shutdown_connections 2024-03-06T07:26:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.935+0000 7f322a0bb640 1 -- 172.21.15.77:0/2134237381 >> 172.21.15.77:0/2134237381 conn(0x7f322406b190 msgr2=0x7f322406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.935+0000 7f322a0bb640 1 -- 172.21.15.77:0/2134237381 shutdown_connections 2024-03-06T07:26:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.935+0000 7f322a0bb640 1 -- 172.21.15.77:0/2134237381 wait complete. 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.936+0000 7f322a0bb640 1 Processor -- start 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.936+0000 7f322a0bb640 1 -- start start 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f322a0bb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f32241433a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f322a0bb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32241f2f50 0x7f32241f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f322a0bb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32241f22d0 0x7f3224153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f322a0bb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32240d0360 con 0x7f320c0039e0 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f322a0bb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322415bde0 con 0x7f32241f22d0 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f322a0bb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32240af150 con 0x7f32241f2f50 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f32237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f32241433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f32237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f32241433a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36332/0 (socket says 172.21.15.77:36332) 2024-03-06T07:26:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f32237fe640 1 -- 172.21.15.77:0/762648040 learned_addr learned my addr 172.21.15.77:0/762648040 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f3222ffd640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32241f2f50 0x7f32241f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f3223fff640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32241f22d0 0x7f3224153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f32237fe640 1 -- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32241f2f50 msgr2=0x7f32241f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f32237fe640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32241f2f50 0x7f32241f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f32237fe640 1 -- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32241f22d0 msgr2=0x7f3224153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f32237fe640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32241f22d0 0x7f3224153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f32237fe640 1 -- 172.21.15.77:0/762648040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32241f27e0 con 0x7f320c0039e0 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.937+0000 7f3223fff640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32241f22d0 0x7f3224153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.938+0000 7f32237fe640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f32241433a0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f321400c210 tx=0x7f322415b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.938+0000 7f320bfff640 1 -- 172.21.15.77:0/762648040 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214001c30 con 0x7f320c0039e0 2024-03-06T07:26:56.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.938+0000 7f320bfff640 1 -- 172.21.15.77:0/762648040 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3214004bf0 con 0x7f320c0039e0 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.938+0000 7f320bfff640 1 -- 172.21.15.77:0/762648040 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214007070 con 0x7f320c0039e0 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.938+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32241f2ac0 con 0x7f320c0039e0 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.938+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32241438e0 con 0x7f320c0039e0 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.939+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32240732b0 con 0x7f320c0039e0 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.939+0000 7f320bfff640 1 -- 172.21.15.77:0/762648040 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3214007210 con 0x7f320c0039e0 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.940+0000 7f320bfff640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f321006b4d0 0x7f321006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.940+0000 7f320bfff640 1 -- 172.21.15.77:0/762648040 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f32140aae50 con 0x7f320c0039e0 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.940+0000 7f3222ffd640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f321006b4d0 0x7f321006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.940+0000 7f3222ffd640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f321006b4d0 0x7f321006d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f32241f3330 tx=0x7f3218009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:56.944+0000 7f320bfff640 1 -- 172.21.15.77:0/762648040 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3214079f50 con 0x7f320c0039e0 2024-03-06T07:26:57.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.115+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3224068ff0 con 0x7f320c0039e0 2024-03-06T07:26:57.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.118+0000 7f320bfff640 1 -- 172.21.15.77:0/762648040 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3214079850 con 0x7f320c0039e0 2024-03-06T07:26:57.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f321006b4d0 msgr2=0x7f321006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f321006b4d0 0x7f321006d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f32241f3330 tx=0x7f3218009040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 msgr2=0x7f32241433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f32241433a0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f321400c210 tx=0x7f322415b9f0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 shutdown_connections 2024-03-06T07:26:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f32241f22d0 0x7f3224153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f32241f2f50 0x7f32241f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f321006b4d0 0x7f321006d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 --2- 172.21.15.77:0/762648040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320c0039e0 0x7f32241433a0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 >> 172.21.15.77:0/762648040 conn(0x7f322406b190 msgr2=0x7f3224158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 shutdown_connections 2024-03-06T07:26:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.120+0000 7f322a0bb640 1 -- 172.21.15.77:0/762648040 wait complete. 2024-03-06T07:26:57.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:26:57.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.232+0000 7fa3abf25640 1 Processor -- start 2024-03-06T07:26:57.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.232+0000 7fa3abf25640 1 -- start start 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3abf25640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa3a406bf10 0x7fa3a4066d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3abf25640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa3a406d2a0 0x7fa3a40d0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3abf25640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a415f560 con 0x7fa3a415e570 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3abf25640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a415ea00 con 0x7fa3a406d2a0 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3abf25640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a415ecc0 con 0x7fa3a406bf10 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3aa4c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa3a406d2a0 0x7fa3a40d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3a94c0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa3a415e570 0x7fa3a406c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57612/0 (socket says 172.21.15.77:57612) 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3a94c0640 1 -- 172.21.15.77:0/1053177476 learned_addr learned my addr 172.21.15.77:0/1053177476 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:26:57.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3aa4c2640 1 -- 172.21.15.77:0/1053177476 >> v2:172.21.15.158:3300/0 conn(0x7fa3a406bf10 msgr2=0x7fa3a4066d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3aa4c2640 1 --2- 172.21.15.77:0/1053177476 >> v2:172.21.15.158:3300/0 conn(0x7fa3a406bf10 0x7fa3a4066d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3aa4c2640 1 -- 172.21.15.77:0/1053177476 >> v1:172.21.15.77:6789/0 conn(0x7fa3a415e570 legacy=0x7fa3a406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3aa4c2640 1 -- 172.21.15.77:0/1053177476 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a415efa0 con 0x7fa3a406d2a0 2024-03-06T07:26:57.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.233+0000 7fa3a8cbf640 1 -- 172.21.15.77:0/1053177476 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292716877 0 0) 0x7fa3a415f560 con 0x7fa3a415e570 2024-03-06T07:26:57.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.234+0000 7fa3aa4c2640 1 --2- 172.21.15.77:0/1053177476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a406d2a0 0x7fa3a40d0600 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40cdca0 tx=0x7fa39c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf2e9748bb17be4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.234+0000 7fa3a8cbf640 1 -- 172.21.15.77:0/1053177476 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c039070 con 0x7fa3a406d2a0 2024-03-06T07:26:57.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.234+0000 7fa3a8cbf640 1 -- 172.21.15.77:0/1053177476 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c02cb60 con 0x7fa3a406d2a0 2024-03-06T07:26:57.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.234+0000 7fa3a8cbf640 1 -- 172.21.15.77:0/1053177476 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c034440 con 0x7fa3a406d2a0 2024-03-06T07:26:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.235+0000 7fa3abf25640 1 -- 172.21.15.77:0/1053177476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a406d2a0 msgr2=0x7fa3a40d0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.235+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1053177476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a406d2a0 0x7fa3a40d0600 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40cdca0 tx=0x7fa39c02bdb0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.235+0000 7fa3abf25640 1 -- 172.21.15.77:0/1053177476 shutdown_connections 2024-03-06T07:26:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.235+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1053177476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a406d2a0 0x7fa3a40d0600 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.235+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1053177476 >> v2:172.21.15.158:3300/0 conn(0x7fa3a406bf10 0x7fa3a4066d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.235+0000 7fa3abf25640 1 -- 172.21.15.77:0/1053177476 >> 172.21.15.77:0/1053177476 conn(0x7fa3a40725f0 msgr2=0x7fa3a4072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.235+0000 7fa3abf25640 1 -- 172.21.15.77:0/1053177476 shutdown_connections 2024-03-06T07:26:57.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.235+0000 7fa3abf25640 1 -- 172.21.15.77:0/1053177476 wait complete. 2024-03-06T07:26:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.236+0000 7fa3abf25640 1 Processor -- start 2024-03-06T07:26:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.236+0000 7fa3abf25640 1 -- start start 2024-03-06T07:26:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3abf25640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a406bf10 0x7fa3a40afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3abf25640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3a406d2a0 0x7fa3a40b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3abf25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a415e570 0x7fa3a4149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3abf25640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a415be70 con 0x7fa3a415e570 2024-03-06T07:26:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3abf25640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a40d00e0 con 0x7fa3a406bf10 2024-03-06T07:26:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3abf25640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a415bff0 con 0x7fa3a406d2a0 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a415e570 0x7fa3a4149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a415e570 0x7fa3a4149e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36360/0 (socket says 172.21.15.77:36360) 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 -- 172.21.15.77:0/1003448383 learned_addr learned my addr 172.21.15.77:0/1003448383 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3a9cc1640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a406bf10 0x7fa3a40afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3a94c0640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3a406d2a0 0x7fa3a40b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 -- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3a406d2a0 msgr2=0x7fa3a40b27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3a406d2a0 0x7fa3a40b27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 -- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a406bf10 msgr2=0x7fa3a40afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a406bf10 0x7fa3a40afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 -- 172.21.15.77:0/1003448383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a40b2ce0 con 0x7fa3a415e570 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3a9cc1640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a406bf10 0x7fa3a40afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3a94c0640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3a406d2a0 0x7fa3a40b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa3aa4c2640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a415e570 0x7fa3a4149e00 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fa39c037040 tx=0x7fa39c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.237+0000 7fa39a7fc640 1 -- 172.21.15.77:0/1003448383 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c044020 con 0x7fa3a415e570 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.238+0000 7fa39a7fc640 1 -- 172.21.15.77:0/1003448383 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c03f670 con 0x7fa3a415e570 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.238+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a40add90 con 0x7fa3a415e570 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.238+0000 7fa39a7fc640 1 -- 172.21.15.77:0/1003448383 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c039040 con 0x7fa3a415e570 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.238+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a40ae350 con 0x7fa3a415e570 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.238+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a404f330 con 0x7fa3a415e570 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.239+0000 7fa39a7fc640 1 -- 172.21.15.77:0/1003448383 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa39c047020 con 0x7fa3a415e570 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.239+0000 7fa39a7fc640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa38806b4d0 0x7fa38806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.239+0000 7fa39a7fc640 1 -- 172.21.15.77:0/1003448383 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fa39c0acc80 con 0x7fa3a415e570 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.239+0000 7fa3a9cc1640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa38806b4d0 0x7fa38806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.240+0000 7fa3a9cc1640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa38806b4d0 0x7fa38806d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009930 tx=0x7fa3a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.244+0000 7fa39a7fc640 1 -- 172.21.15.77:0/1003448383 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa39c04e030 con 0x7fa3a415e570 2024-03-06T07:26:57.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:57.492+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fa3a40729d0 con 0x7fa3a415e570 2024-03-06T07:26:58.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.462+0000 7fa39a7fc640 1 -- 172.21.15.77:0/1003448383 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v274) v1 ==== 134+0+0 (secure 0 0 0) 0x7fa3a40b2ce0 con 0x7fa3a415e570 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa38806b4d0 msgr2=0x7fa38806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa38806b4d0 0x7fa38806d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009930 tx=0x7fa3a0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a415e570 msgr2=0x7fa3a4149e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a415e570 0x7fa3a4149e00 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fa39c037040 tx=0x7fa39c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 shutdown_connections 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa38806b4d0 0x7fa38806d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a415e570 0x7fa3a4149e00 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3a406d2a0 0x7fa3a40b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 --2- 172.21.15.77:0/1003448383 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3a406bf10 0x7fa3a40afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 >> 172.21.15.77:0/1003448383 conn(0x7fa3a40725f0 msgr2=0x7fa3a40b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 shutdown_connections 2024-03-06T07:26:58.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.465+0000 7fa3abf25640 1 -- 172.21.15.77:0/1003448383 wait complete. 2024-03-06T07:26:58.480 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:26:58.481 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:26:58.481 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:26:58.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:58.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.584+0000 7f6ea359e640 1 Processor -- start 2024-03-06T07:26:58.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.584+0000 7f6ea359e640 1 -- start start 2024-03-06T07:26:58.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ea415bbf0 0x7f6ea40b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea359e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6ea40cead0 0x7f6ea40d1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea359e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea40d2150 con 0x7f6ea40d1d40 2024-03-06T07:26:58.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea359e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea406f020 con 0x7f6ea415bbf0 2024-03-06T07:26:58.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea359e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea40d0450 con 0x7f6ea40cead0 2024-03-06T07:26:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ea415bbf0 0x7f6ea40b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ea415bbf0 0x7f6ea40b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58484/0 (socket says 172.21.15.77:58484) 2024-03-06T07:26:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea259c640 1 -- 172.21.15.77:0/2544118308 learned_addr learned my addr 172.21.15.77:0/2544118308 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea259c640 1 -- 172.21.15.77:0/2544118308 >> v2:172.21.15.158:3300/0 conn(0x7f6ea40cead0 msgr2=0x7f6ea40d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea259c640 1 --2- 172.21.15.77:0/2544118308 >> v2:172.21.15.158:3300/0 conn(0x7f6ea40cead0 0x7f6ea40d1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea259c640 1 -- 172.21.15.77:0/2544118308 >> v1:172.21.15.158:6789/0 conn(0x7f6ea40d1d40 legacy=0x7f6ea415f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.585+0000 7f6ea259c640 1 -- 172.21.15.77:0/2544118308 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea406f270 con 0x7f6ea415bbf0 2024-03-06T07:26:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.586+0000 7f6ea259c640 1 --2- 172.21.15.77:0/2544118308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea415bbf0 0x7f6ea40b0d10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4072580 tx=0x7f6e9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6340dff4635c4695 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.586+0000 7f6ea1d9b640 1 -- 172.21.15.77:0/2544118308 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e94035020 con 0x7f6ea415bbf0 2024-03-06T07:26:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.586+0000 7f6ea1d9b640 1 -- 172.21.15.77:0/2544118308 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e94031870 con 0x7f6ea415bbf0 2024-03-06T07:26:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.586+0000 7f6ea1d9b640 1 -- 172.21.15.77:0/2544118308 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e94031c40 con 0x7f6ea415bbf0 2024-03-06T07:26:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.587+0000 7f6ea359e640 1 -- 172.21.15.77:0/2544118308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea415bbf0 msgr2=0x7f6ea40b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.587+0000 7f6ea359e640 1 --2- 172.21.15.77:0/2544118308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea415bbf0 0x7f6ea40b0d10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4072580 tx=0x7f6e9402b910 comp rx=0 tx=0).stop 2024-03-06T07:26:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.587+0000 7f6ea359e640 1 -- 172.21.15.77:0/2544118308 shutdown_connections 2024-03-06T07:26:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.587+0000 7f6ea359e640 1 --2- 172.21.15.77:0/2544118308 >> v2:172.21.15.158:3300/0 conn(0x7f6ea40cead0 0x7f6ea40d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.587+0000 7f6ea359e640 1 --2- 172.21.15.77:0/2544118308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea415bbf0 0x7f6ea40b0d10 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.587+0000 7f6ea359e640 1 -- 172.21.15.77:0/2544118308 >> 172.21.15.77:0/2544118308 conn(0x7f6ea409a9f0 msgr2=0x7f6ea409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.587+0000 7f6ea359e640 1 -- 172.21.15.77:0/2544118308 shutdown_connections 2024-03-06T07:26:58.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.587+0000 7f6ea359e640 1 -- 172.21.15.77:0/2544118308 wait complete. 2024-03-06T07:26:58.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.588+0000 7f6ea359e640 1 Processor -- start 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.588+0000 7f6ea359e640 1 -- start start 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40cead0 0x7f6ea415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea359e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ea40d1d40 0x7f6ea415e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea359e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ea415bbf0 0x7f6ea415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea359e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea406fb10 con 0x7f6ea40cead0 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea359e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea406f830 con 0x7f6ea40d1d40 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea359e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea40cef60 con 0x7f6ea415bbf0 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40cead0 0x7f6ea415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40cead0 0x7f6ea415af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58492/0 (socket says 172.21.15.77:58492) 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 -- 172.21.15.77:0/4023335519 learned_addr learned my addr 172.21.15.77:0/4023335519 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:58.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 -- 172.21.15.77:0/4023335519 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ea415bbf0 msgr2=0x7f6ea415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ea415bbf0 0x7f6ea415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 -- 172.21.15.77:0/4023335519 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ea40d1d40 msgr2=0x7f6ea415e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ea40d1d40 0x7f6ea415e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 -- 172.21.15.77:0/4023335519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea406cad0 con 0x7f6ea40cead0 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6ea259c640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40cead0 0x7f6ea415af30 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f6e94000c00 tx=0x7f6e940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.589+0000 7f6e9affd640 1 -- 172.21.15.77:0/4023335519 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9403ad60 con 0x7f6ea40cead0 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.590+0000 7f6e9affd640 1 -- 172.21.15.77:0/4023335519 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e940443f0 con 0x7f6ea40cead0 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.590+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea406cdb0 con 0x7f6ea40cead0 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.590+0000 7f6e9affd640 1 -- 172.21.15.77:0/4023335519 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e940447e0 con 0x7f6ea40cead0 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.590+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea406d370 con 0x7f6ea40cead0 2024-03-06T07:26:58.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.590+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea4068f80 con 0x7f6ea40cead0 2024-03-06T07:26:58.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.591+0000 7f6e9affd640 1 -- 172.21.15.77:0/4023335519 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e9404b060 con 0x7f6ea40cead0 2024-03-06T07:26:58.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.591+0000 7f6e9affd640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e8406b4d0 0x7f6e8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.591+0000 7f6e9bfff640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e8406b4d0 0x7f6e8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.592+0000 7f6e9affd640 1 -- 172.21.15.77:0/4023335519 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f6e940ac1c0 con 0x7f6ea40cead0 2024-03-06T07:26:58.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.592+0000 7f6e9bfff640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e8406b4d0 0x7f6e8406d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40b01c0 tx=0x7f6e8c006bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.596+0000 7f6e9affd640 1 -- 172.21.15.77:0/4023335519 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e9403f070 con 0x7f6ea40cead0 2024-03-06T07:26:58.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.769+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ea40b77f0 con 0x7f6ea40cead0 2024-03-06T07:26:58.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.770+0000 7f6e9affd640 1 -- 172.21.15.77:0/4023335519 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+13395 (secure 0 0 0) 0x7f6ea40d2150 con 0x7f6ea40cead0 2024-03-06T07:26:58.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:26:58.772 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":99,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:26:56.455632+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6837/82690693":"2024-03-07T07:26:53.450693+0000","172.21.15.158:6833/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6835/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.77:6836/82690693":"2024-03-07T07:26:53.450693+0000","172.21.15.158:6835/497683896":"2024-03-07T07:26:21.753195+0000","172.21.15.77:6834/1877423196":"2024-03-07T07:26:21.743752+0000","172.21.15.77:6837/3425851566":"2024-03-07T07:26:21.715277+0000","172.21.15.77:6836/3425851566":"2024-03-07T07:26:21.715277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:58.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e8406b4d0 msgr2=0x7f6e8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e8406b4d0 0x7f6e8406d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40b01c0 tx=0x7f6e8c006bf0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40cead0 msgr2=0x7f6ea415af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40cead0 0x7f6ea415af30 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f6e94000c00 tx=0x7f6e940046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 shutdown_connections 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ea415bbf0 0x7f6ea415eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ea40d1d40 0x7f6ea415e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e8406b4d0 0x7f6e8406d990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 --2- 172.21.15.77:0/4023335519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40cead0 0x7f6ea415af30 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 >> 172.21.15.77:0/4023335519 conn(0x7f6ea409a9f0 msgr2=0x7f6ea40ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 shutdown_connections 2024-03-06T07:26:58.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.772+0000 7f6ea359e640 1 -- 172.21.15.77:0/4023335519 wait complete. 2024-03-06T07:26:58.786 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:58.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:58.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.886+0000 7f3ec5565640 1 Processor -- start 2024-03-06T07:26:58.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.886+0000 7f3ec5565640 1 -- start start 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ec5565640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3ec01614d0 0x7f3ec00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ec5565640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3ec00d24b0 0x7f3ec00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ec5565640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec01618b0 con 0x7f3ec00d1910 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ec5565640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec00cf400 con 0x7f3ec01614d0 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ec5565640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec00cf6a0 con 0x7f3ec00d24b0 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ebeffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3ec01614d0 0x7f3ec00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ebeffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3ec01614d0 0x7f3ec00d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58496/0 (socket says 172.21.15.77:58496) 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ebeffd640 1 -- 172.21.15.77:0/3010869366 learned_addr learned my addr 172.21.15.77:0/3010869366 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ebf7fe640 1 --2- 172.21.15.77:0/3010869366 >> v2:172.21.15.158:3300/0 conn(0x7f3ec00d24b0 0x7f3ec00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ebeffd640 1 -- 172.21.15.77:0/3010869366 >> v2:172.21.15.158:3300/0 conn(0x7f3ec00d24b0 msgr2=0x7f3ec00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ebeffd640 1 --2- 172.21.15.77:0/3010869366 >> v2:172.21.15.158:3300/0 conn(0x7f3ec00d24b0 0x7f3ec00b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ebeffd640 1 -- 172.21.15.77:0/3010869366 >> v1:172.21.15.158:6789/0 conn(0x7f3ec00d1910 legacy=0x7f3ec00d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.887+0000 7f3ebeffd640 1 -- 172.21.15.77:0/3010869366 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec00cf980 con 0x7f3ec01614d0 2024-03-06T07:26:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.888+0000 7f3ebeffd640 1 --2- 172.21.15.77:0/3010869366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec01614d0 0x7f3ec00d4860 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00b2a70 tx=0x7f3eb002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d663ddcd13658b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.888+0000 7f3ebdffb640 1 -- 172.21.15.77:0/3010869366 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb002ca80 con 0x7f3ec01614d0 2024-03-06T07:26:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.888+0000 7f3ebdffb640 1 -- 172.21.15.77:0/3010869366 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb002cc20 con 0x7f3ec01614d0 2024-03-06T07:26:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.888+0000 7f3ebdffb640 1 -- 172.21.15.77:0/3010869366 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00343f0 con 0x7f3ec01614d0 2024-03-06T07:26:58.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.889+0000 7f3ec5565640 1 -- 172.21.15.77:0/3010869366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec01614d0 msgr2=0x7f3ec00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.889+0000 7f3ec5565640 1 --2- 172.21.15.77:0/3010869366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec01614d0 0x7f3ec00d4860 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00b2a70 tx=0x7f3eb002bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.889+0000 7f3ec5565640 1 -- 172.21.15.77:0/3010869366 shutdown_connections 2024-03-06T07:26:58.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.889+0000 7f3ec5565640 1 --2- 172.21.15.77:0/3010869366 >> v2:172.21.15.158:3300/0 conn(0x7f3ec00d24b0 0x7f3ec00b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.889+0000 7f3ec5565640 1 --2- 172.21.15.77:0/3010869366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec01614d0 0x7f3ec00d4860 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.889+0000 7f3ec5565640 1 -- 172.21.15.77:0/3010869366 >> 172.21.15.77:0/3010869366 conn(0x7f3ec00d3d40 msgr2=0x7f3ec00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.889+0000 7f3ec5565640 1 -- 172.21.15.77:0/3010869366 shutdown_connections 2024-03-06T07:26:58.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.889+0000 7f3ec5565640 1 -- 172.21.15.77:0/3010869366 wait complete. 2024-03-06T07:26:58.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.890+0000 7f3ec5565640 1 Processor -- start 2024-03-06T07:26:58.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.890+0000 7f3ec5565640 1 -- start start 2024-03-06T07:26:58.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ec5565640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec00d1910 0x7f3ec0154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ec5565640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3ec00d24b0 0x7f3ec015eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ec5565640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ec01614d0 0x7f3ec00b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ec5565640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00b1370 con 0x7f3ec00d1910 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ec5565640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00b0910 con 0x7f3ec01614d0 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ec5565640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00b14f0 con 0x7f3ec00d24b0 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec00d1910 0x7f3ec0154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec00d1910 0x7f3ec0154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58512/0 (socket says 172.21.15.77:58512) 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 -- 172.21.15.77:0/2299120739 learned_addr learned my addr 172.21.15.77:0/2299120739 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 -- 172.21.15.77:0/2299120739 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3ec00d24b0 msgr2=0x7f3ec015eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3ec00d24b0 0x7f3ec015eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 -- 172.21.15.77:0/2299120739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ec01614d0 msgr2=0x7f3ec00b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ec01614d0 0x7f3ec00b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 -- 172.21.15.77:0/2299120739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec015e110 con 0x7f3ec00d1910 2024-03-06T07:26:58.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.891+0000 7f3ebeffd640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec00d1910 0x7f3ec0154fc0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009f40 tx=0x7f3eb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.892+0000 7f3ea77fe640 1 -- 172.21.15.77:0/2299120739 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0043020 con 0x7f3ec00d1910 2024-03-06T07:26:58.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.892+0000 7f3ea77fe640 1 -- 172.21.15.77:0/2299120739 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb003e450 con 0x7f3ec00d1910 2024-03-06T07:26:58.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.892+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec015e3f0 con 0x7f3ec00d1910 2024-03-06T07:26:58.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.892+0000 7f3ea77fe640 1 -- 172.21.15.77:0/2299120739 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0009510 con 0x7f3ec00d1910 2024-03-06T07:26:58.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.892+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec015e9b0 con 0x7f3ec00d1910 2024-03-06T07:26:58.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.893+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec0068ff0 con 0x7f3ec00d1910 2024-03-06T07:26:58.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.893+0000 7f3ea77fe640 1 -- 172.21.15.77:0/2299120739 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3eb0046020 con 0x7f3ec00d1910 2024-03-06T07:26:58.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.893+0000 7f3ea77fe640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3eac06b4d0 0x7f3eac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.894+0000 7f3ea77fe640 1 -- 172.21.15.77:0/2299120739 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f3eb00ac8b0 con 0x7f3ec00d1910 2024-03-06T07:26:58.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.894+0000 7f3ebe7fc640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3eac06b4d0 0x7f3eac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.894+0000 7f3ebe7fc640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3eac06b4d0 0x7f3eac06d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3ec01f74f0 tx=0x7f3eb40079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:58.898+0000 7f3ea77fe640 1 -- 172.21.15.77:0/2299120739 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3eb004d030 con 0x7f3ec00d1910 2024-03-06T07:26:59.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.068+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3ec00d0cc0 con 0x7f3ec00d1910 2024-03-06T07:26:59.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.068+0000 7f3ea77fe640 1 -- 172.21.15.77:0/2299120739 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3eb007b920 con 0x7f3ec00d1910 2024-03-06T07:26:59.071 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:26:59.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3eac06b4d0 msgr2=0x7f3eac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3eac06b4d0 0x7f3eac06d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3ec01f74f0 tx=0x7f3eb40079b0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec00d1910 msgr2=0x7f3ec0154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec00d1910 0x7f3ec0154fc0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009f40 tx=0x7f3eb0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 shutdown_connections 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3ec01614d0 0x7f3ec00b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3ec00d24b0 0x7f3ec015eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3eac06b4d0 0x7f3eac06d990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 --2- 172.21.15.77:0/2299120739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec00d1910 0x7f3ec0154fc0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 >> 172.21.15.77:0/2299120739 conn(0x7f3ec00d3d40 msgr2=0x7f3ec00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 shutdown_connections 2024-03-06T07:26:59.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.070+0000 7f3ec5565640 1 -- 172.21.15.77:0/2299120739 wait complete. 2024-03-06T07:26:59.083 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:26:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.183+0000 7f826653e640 1 Processor -- start 2024-03-06T07:26:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.183+0000 7f826653e640 1 -- start start 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.183+0000 7f826653e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82600d4760 0x7f82600b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.183+0000 7f826653e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600d0a80 0x7f82600cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.183+0000 7f826653e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82600d25d0 con 0x7f82600b4170 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.183+0000 7f826653e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82600d4bf0 con 0x7f82600d4760 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.183+0000 7f826653e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82600ce9e0 con 0x7f82600d0a80 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8257fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82600d4760 0x7f82600b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8257fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82600d4760 0x7f82600b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58516/0 (socket says 172.21.15.77:58516) 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8257fff640 1 -- 172.21.15.77:0/571266653 learned_addr learned my addr 172.21.15.77:0/571266653 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8264adb640 1 --2- 172.21.15.77:0/571266653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600d0a80 0x7f82600cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8257fff640 1 -- 172.21.15.77:0/571266653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600d0a80 msgr2=0x7f82600cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8257fff640 1 --2- 172.21.15.77:0/571266653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600d0a80 0x7f82600cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8257fff640 1 -- 172.21.15.77:0/571266653 >> v1:172.21.15.158:6789/0 conn(0x7f82600b4170 legacy=0x7f82600d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8257fff640 1 -- 172.21.15.77:0/571266653 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82600d2870 con 0x7f82600d4760 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.184+0000 7f8264adb640 1 --2- 172.21.15.77:0/571266653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600d0a80 0x7f82600cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f8257fff640 1 --2- 172.21.15.77:0/571266653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82600d4760 0x7f82600b6a90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f8260067920 tx=0x7f824802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe8c3ecf58eb1bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f825f7fe640 1 -- 172.21.15.77:0/571266653 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248035020 con 0x7f82600d4760 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f825f7fe640 1 -- 172.21.15.77:0/571266653 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8248031830 con 0x7f82600d4760 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f825f7fe640 1 -- 172.21.15.77:0/571266653 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248031c00 con 0x7f82600d4760 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f826653e640 1 -- 172.21.15.77:0/571266653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82600d4760 msgr2=0x7f82600b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f826653e640 1 --2- 172.21.15.77:0/571266653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82600d4760 0x7f82600b6a90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f8260067920 tx=0x7f824802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f826653e640 1 -- 172.21.15.77:0/571266653 shutdown_connections 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f826653e640 1 --2- 172.21.15.77:0/571266653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600d0a80 0x7f82600cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f826653e640 1 --2- 172.21.15.77:0/571266653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82600d4760 0x7f82600b6a90 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f826653e640 1 -- 172.21.15.77:0/571266653 >> 172.21.15.77:0/571266653 conn(0x7f82600d3c40 msgr2=0x7f82600d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f826653e640 1 -- 172.21.15.77:0/571266653 shutdown_connections 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.185+0000 7f826653e640 1 -- 172.21.15.77:0/571266653 wait complete. 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f826653e640 1 Processor -- start 2024-03-06T07:26:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f826653e640 1 -- start start 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f826653e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600b4170 0x7f82600ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f826653e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82600d0a80 0x7f82600cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f826653e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159550 0x7f82600ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f826653e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82600b4dd0 con 0x7f8260159550 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f826653e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82600b4550 con 0x7f82600d0a80 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f826653e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82600b4f50 con 0x7f82600b4170 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f8264adb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159550 0x7f82600ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f8264adb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159550 0x7f82600ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58524/0 (socket says 172.21.15.77:58524) 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.187+0000 7f8264adb640 1 -- 172.21.15.77:0/4006584540 learned_addr learned my addr 172.21.15.77:0/4006584540 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f825ffff640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600b4170 0x7f82600ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f8257fff640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82600d0a80 0x7f82600cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f8264adb640 1 -- 172.21.15.77:0/4006584540 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600b4170 msgr2=0x7f82600ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f8264adb640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600b4170 0x7f82600ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f8264adb640 1 -- 172.21.15.77:0/4006584540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82600d0a80 msgr2=0x7f82600cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f8264adb640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82600d0a80 0x7f82600cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f8264adb640 1 -- 172.21.15.77:0/4006584540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82600cb250 con 0x7f8260159550 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f8264adb640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159550 0x7f82600ccd60 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f825800cf20 tx=0x7f825800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f825d7fa640 1 -- 172.21.15.77:0/4006584540 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825801a070 con 0x7f8260159550 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f825d7fa640 1 -- 172.21.15.77:0/4006584540 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825800d840 con 0x7f8260159550 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f825d7fa640 1 -- 172.21.15.77:0/4006584540 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825800dc30 con 0x7f8260159550 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82600cc570 con 0x7f8260159550 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.188+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82600cb7b0 con 0x7f8260159550 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.189+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826004f330 con 0x7f8260159550 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.189+0000 7f825d7fa640 1 -- 172.21.15.77:0/4006584540 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f825801d070 con 0x7f8260159550 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.190+0000 7f825d7fa640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f824006b4d0 0x7f824006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.190+0000 7f825ffff640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f824006b4d0 0x7f824006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.190+0000 7f825d7fa640 1 -- 172.21.15.77:0/4006584540 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f825808a840 con 0x7f8260159550 2024-03-06T07:26:59.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.190+0000 7f825ffff640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f824006b4d0 0x7f824006d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8250006fd0 tx=0x7f8250008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.194+0000 7f825d7fa640 1 -- 172.21.15.77:0/4006584540 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82600cb250 con 0x7f8260159550 2024-03-06T07:26:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.432+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f82600733b0 con 0x7f8260159550 2024-03-06T07:26:59.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.466+0000 7f825d7fa640 1 -- 172.21.15.77:0/4006584540 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v275) v1 ==== 145+0+0 (secure 0 0 0) 0x7f8258059940 con 0x7f8260159550 2024-03-06T07:26:59.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f824006b4d0 msgr2=0x7f824006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f824006b4d0 0x7f824006d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8250006fd0 tx=0x7f8250008040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159550 msgr2=0x7f82600ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159550 0x7f82600ccd60 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f825800cf20 tx=0x7f825800a7d0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 shutdown_connections 2024-03-06T07:26:59.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f824006b4d0 0x7f824006d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159550 0x7f82600ccd60 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82600d0a80 0x7f82600cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 --2- 172.21.15.77:0/4006584540 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82600b4170 0x7f82600ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 >> 172.21.15.77:0/4006584540 conn(0x7f82600d3c40 msgr2=0x7f82600d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 shutdown_connections 2024-03-06T07:26:59.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.468+0000 7f826653e640 1 -- 172.21.15.77:0/4006584540 wait complete. 2024-03-06T07:26:59.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:59.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.582+0000 7f6deff7a640 1 Processor -- start 2024-03-06T07:26:59.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.582+0000 7f6deff7a640 1 -- start start 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.583+0000 7f6deff7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80d4860 0x7f6de80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.583+0000 7f6deff7a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6de80b0e30 0x7f6de80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.583+0000 7f6deff7a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de81618b0 con 0x7f6de81614d0 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.583+0000 7f6deff7a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de80d4cc0 con 0x7f6de80b0e30 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.583+0000 7f6deff7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de80cf400 con 0x7f6de80d4860 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.583+0000 7f6dee517640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6de80b0e30 0x7f6de80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.583+0000 7f6dee517640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6de80b0e30 0x7f6de80b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58538/0 (socket says 172.21.15.77:58538) 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.583+0000 7f6dee517640 1 -- 172.21.15.77:0/873473441 learned_addr learned my addr 172.21.15.77:0/873473441 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.584+0000 7f6dee517640 1 -- 172.21.15.77:0/873473441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80d4860 msgr2=0x7f6de80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.584+0000 7f6ded515640 1 --2- 172.21.15.77:0/873473441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80d4860 0x7f6de80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.584+0000 7f6dee517640 1 --2- 172.21.15.77:0/873473441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80d4860 0x7f6de80d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.584+0000 7f6dee517640 1 -- 172.21.15.77:0/873473441 >> v1:172.21.15.158:6789/0 conn(0x7f6de81614d0 legacy=0x7f6de80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.584+0000 7f6dee517640 1 -- 172.21.15.77:0/873473441 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de80cf6a0 con 0x7f6de80b0e30 2024-03-06T07:26:59.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.584+0000 7f6dee517640 1 --2- 172.21.15.77:0/873473441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80b0e30 0x7f6de80b12c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009870 tx=0x7f6de002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ed848f838128d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6decd14640 1 -- 172.21.15.77:0/873473441 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de0035020 con 0x7f6de80b0e30 2024-03-06T07:26:59.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6decd14640 1 -- 172.21.15.77:0/873473441 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de0031640 con 0x7f6de80b0e30 2024-03-06T07:26:59.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6decd14640 1 -- 172.21.15.77:0/873473441 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de002d040 con 0x7f6de80b0e30 2024-03-06T07:26:59.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6deff7a640 1 -- 172.21.15.77:0/873473441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80b0e30 msgr2=0x7f6de80b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6deff7a640 1 --2- 172.21.15.77:0/873473441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80b0e30 0x7f6de80b12c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009870 tx=0x7f6de002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6deff7a640 1 -- 172.21.15.77:0/873473441 shutdown_connections 2024-03-06T07:26:59.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6deff7a640 1 --2- 172.21.15.77:0/873473441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80d4860 0x7f6de80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6deff7a640 1 --2- 172.21.15.77:0/873473441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80b0e30 0x7f6de80b12c0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6deff7a640 1 -- 172.21.15.77:0/873473441 >> 172.21.15.77:0/873473441 conn(0x7f6de80d3d40 msgr2=0x7f6de80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6deff7a640 1 -- 172.21.15.77:0/873473441 shutdown_connections 2024-03-06T07:26:59.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.585+0000 7f6deff7a640 1 -- 172.21.15.77:0/873473441 wait complete. 2024-03-06T07:26:59.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.586+0000 7f6deff7a640 1 Processor -- start 2024-03-06T07:26:59.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.586+0000 7f6deff7a640 1 -- start start 2024-03-06T07:26:59.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6deff7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80b0e30 0x7f6de815d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6deff7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80d4860 0x7f6de80a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6deff7a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6de81614d0 0x7f6de80cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6deff7a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de80b0910 con 0x7f6de80d4860 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6deff7a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de80cfc60 con 0x7f6de81614d0 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6deff7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de80d2560 con 0x7f6de80b0e30 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6ded515640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80d4860 0x7f6de80a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6ded515640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80d4860 0x7f6de80a1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58548/0 (socket says 172.21.15.77:58548) 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6ded515640 1 -- 172.21.15.77:0/3181560413 learned_addr learned my addr 172.21.15.77:0/3181560413 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6dedd16640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80b0e30 0x7f6de815d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6dee517640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6de81614d0 0x7f6de80cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6ded515640 1 -- 172.21.15.77:0/3181560413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80b0e30 msgr2=0x7f6de815d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6ded515640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80b0e30 0x7f6de815d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6ded515640 1 -- 172.21.15.77:0/3181560413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6de81614d0 msgr2=0x7f6de80cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6ded515640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6de81614d0 0x7f6de80cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.587+0000 7f6ded515640 1 -- 172.21.15.77:0/3181560413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de815dbd0 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.588+0000 7f6ded515640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80d4860 0x7f6de80a1400 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800c820 tx=0x7f6dd800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.588+0000 7f6dde7fc640 1 -- 172.21.15.77:0/3181560413 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8015070 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.588+0000 7f6dde7fc640 1 -- 172.21.15.77:0/3181560413 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd8004240 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.588+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de80c8340 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.588+0000 7f6dde7fc640 1 -- 172.21.15.77:0/3181560413 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd800ce10 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.588+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de80b6b80 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.589+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de8068ff0 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.589+0000 7f6dde7fc640 1 -- 172.21.15.77:0/3181560413 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6dd8007830 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.590+0000 7f6dde7fc640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6dcc06b4d0 0x7f6dcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.590+0000 7f6dde7fc640 1 -- 172.21.15.77:0/3181560413 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f6dd8089e20 con 0x7f6de80d4860 2024-03-06T07:26:59.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.590+0000 7f6dedd16640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6dcc06b4d0 0x7f6dcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.590+0000 7f6dedd16640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6dcc06b4d0 0x7f6dcc06d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6de80c7b00 tx=0x7f6de400d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.594+0000 7f6dde7fc640 1 -- 172.21.15.77:0/3181560413 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6dd80588d0 con 0x7f6de80d4860 2024-03-06T07:26:59.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.765+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6de80670d0 con 0x7f6de80d4860 2024-03-06T07:26:59.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.768+0000 7f6dde7fc640 1 -- 172.21.15.77:0/3181560413 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6dd805c970 con 0x7f6de80d4860 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6dcc06b4d0 msgr2=0x7f6dcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6dcc06b4d0 0x7f6dcc06d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6de80c7b00 tx=0x7f6de400d040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80d4860 msgr2=0x7f6de80a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80d4860 0x7f6de80a1400 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800c820 tx=0x7f6dd800cce0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 shutdown_connections 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6dcc06b4d0 0x7f6dcc06d990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6de80d4860 0x7f6de80a1400 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6de81614d0 0x7f6de80cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 --2- 172.21.15.77:0/3181560413 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6de80b0e30 0x7f6de815d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 >> 172.21.15.77:0/3181560413 conn(0x7f6de80d3d40 msgr2=0x7f6de80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 shutdown_connections 2024-03-06T07:26:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:26:59.771+0000 7f6deff7a640 1 -- 172.21.15.77:0/3181560413 wait complete. 2024-03-06T07:26:59.784 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:59.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:59.834 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:59.875 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:59.875 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:26:59.875 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:59.890 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:59.891 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:26:59.891 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:59.932 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:59.933 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:26:59.933 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:59.948 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:59.948 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:26:59.948 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:59.989 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:59.989 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:26:59.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:00.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.087+0000 7fd6c7646640 1 Processor -- start 2024-03-06T07:27:00.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.087+0000 7fd6c7646640 1 -- start start 2024-03-06T07:27:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.088+0000 7fd6c7646640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c013dc10 0x7fd6c00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.088+0000 7fd6c7646640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6c015bf30 0x7fd6c00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.088+0000 7fd6c7646640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c015f580 con 0x7fd6c015e850 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.088+0000 7fd6c7646640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c00cf3f0 con 0x7fd6c015bf30 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.088+0000 7fd6c7646640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c015b210 con 0x7fd6c013dc10 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.088+0000 7fd6c4be1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6c015bf30 0x7fd6c00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.088+0000 7fd6c4be1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6c015bf30 0x7fd6c00b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58550/0 (socket says 172.21.15.77:58550) 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.088+0000 7fd6c4be1640 1 -- 172.21.15.77:0/3387412585 learned_addr learned my addr 172.21.15.77:0/3387412585 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.089+0000 7fd6c53e2640 1 --2- 172.21.15.77:0/3387412585 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c013dc10 0x7fd6c00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.089+0000 7fd6c4be1640 1 -- 172.21.15.77:0/3387412585 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c013dc10 msgr2=0x7fd6c00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.089+0000 7fd6c4be1640 1 --2- 172.21.15.77:0/3387412585 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c013dc10 0x7fd6c00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.089+0000 7fd6c4be1640 1 -- 172.21.15.77:0/3387412585 >> v1:172.21.15.158:6789/0 conn(0x7fd6c015e850 legacy=0x7fd6c015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.089+0000 7fd6c4be1640 1 -- 172.21.15.77:0/3387412585 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c015b450 con 0x7fd6c015bf30 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.089+0000 7fd6c4be1640 1 --2- 172.21.15.77:0/3387412585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c015bf30 0x7fd6c00b2e90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009870 tx=0x7fd6b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8610becfd27dab8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6b7fff640 1 -- 172.21.15.77:0/3387412585 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0035020 con 0x7fd6c015bf30 2024-03-06T07:27:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6b7fff640 1 -- 172.21.15.77:0/3387412585 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b0031640 con 0x7fd6c015bf30 2024-03-06T07:27:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6b7fff640 1 -- 172.21.15.77:0/3387412585 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b002d040 con 0x7fd6c015bf30 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6c7646640 1 -- 172.21.15.77:0/3387412585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c015bf30 msgr2=0x7fd6c00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3387412585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c015bf30 0x7fd6c00b2e90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009870 tx=0x7fd6b002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6c7646640 1 -- 172.21.15.77:0/3387412585 shutdown_connections 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3387412585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c015bf30 0x7fd6c00b2e90 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3387412585 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c013dc10 0x7fd6c00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6c7646640 1 -- 172.21.15.77:0/3387412585 >> 172.21.15.77:0/3387412585 conn(0x7fd6c00cf8a0 msgr2=0x7fd6c00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6c7646640 1 -- 172.21.15.77:0/3387412585 shutdown_connections 2024-03-06T07:27:00.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.090+0000 7fd6c7646640 1 -- 172.21.15.77:0/3387412585 wait complete. 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.091+0000 7fd6c7646640 1 Processor -- start 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.091+0000 7fd6c7646640 1 -- start start 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c7646640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c013dc10 0x7fd6c01514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c7646640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6c015bf30 0x7fd6c01f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c7646640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c015e850 0x7fd6c0071b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c7646640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c00cd7a0 con 0x7fd6c013dc10 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c7646640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c013dff0 con 0x7fd6c015bf30 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c7646640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c00cd920 con 0x7fd6c015e850 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c53e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c013dc10 0x7fd6c01514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c53e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c013dc10 0x7fd6c01514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58558/0 (socket says 172.21.15.77:58558) 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c53e2640 1 -- 172.21.15.77:0/3070084322 learned_addr learned my addr 172.21.15.77:0/3070084322 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:00.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c5be3640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c015e850 0x7fd6c0071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c4be1640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6c015bf30 0x7fd6c01f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c53e2640 1 -- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c015e850 msgr2=0x7fd6c0071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.092+0000 7fd6c53e2640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c015e850 0x7fd6c0071b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6c53e2640 1 -- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6c015bf30 msgr2=0x7fd6c01f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6c53e2640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6c015bf30 0x7fd6c01f3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6c53e2640 1 -- 172.21.15.77:0/3070084322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c01f3d70 con 0x7fd6c013dc10 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6c4be1640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6c015bf30 0x7fd6c01f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6c53e2640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c013dc10 0x7fd6c01514c0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009f20 tx=0x7fd6bc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6b5ffb640 1 -- 172.21.15.77:0/3070084322 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc01c070 con 0x7fd6c013dc10 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6b5ffb640 1 -- 172.21.15.77:0/3070084322 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc017600 con 0x7fd6c013dc10 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c00707d0 con 0x7fd6c013dc10 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6b5ffb640 1 -- 172.21.15.77:0/3070084322 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0179d0 con 0x7fd6c013dc10 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.093+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c0070d90 con 0x7fd6c013dc10 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.094+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c004f330 con 0x7fd6c013dc10 2024-03-06T07:27:00.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.094+0000 7fd6b5ffb640 1 -- 172.21.15.77:0/3070084322 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6bc01f070 con 0x7fd6c013dc10 2024-03-06T07:27:00.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.095+0000 7fd6b5ffb640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a406b4d0 0x7fd6a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.095+0000 7fd6c4be1640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a406b4d0 0x7fd6a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.095+0000 7fd6b5ffb640 1 -- 172.21.15.77:0/3070084322 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fd6bc08c0d0 con 0x7fd6c013dc10 2024-03-06T07:27:00.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.095+0000 7fd6c4be1640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a406b4d0 0x7fd6a406d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0036000 tx=0x7fd6b003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.099+0000 7fd6b5ffb640 1 -- 172.21.15.77:0/3070084322 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6bc012070 con 0x7fd6c013dc10 2024-03-06T07:27:00.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.310+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6c013b930 con 0x7fd6c013dc10 2024-03-06T07:27:00.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.310+0000 7fd6b5ffb640 1 -- 172.21.15.77:0/3070084322 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3693 (secure 0 0 0) 0x7fd6bc05abb0 con 0x7fd6c013dc10 2024-03-06T07:27:00.312 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:00.312 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6212,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/83327672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":83327672},{"type":"v1","addr":"172.21.15.158:6833","nonce":83327672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3338368134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3338368134},{"type":"v1","addr":"172.21.15.158:6835","nonce":3338368134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6967,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3524166228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3524166228},{"type":"v1","addr":"172.21.15.77:6835","nonce":3524166228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":7752,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/69093282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":69093282},{"type":"v1","addr":"172.21.15.77:6837","nonce":69093282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[]} 2024-03-06T07:27:00.312 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 60 2024-03-06T07:27:00.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a406b4d0 msgr2=0x7fd6a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a406b4d0 0x7fd6a406d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0036000 tx=0x7fd6b003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:00.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c013dc10 msgr2=0x7fd6c01514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c013dc10 0x7fd6c01514c0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009f20 tx=0x7fd6bc00f4a0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 shutdown_connections 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6c015e850 0x7fd6c0071b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6c015bf30 0x7fd6c01f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd6a406b4d0 0x7fd6a406d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 --2- 172.21.15.77:0/3070084322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c013dc10 0x7fd6c01514c0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 >> 172.21.15.77:0/3070084322 conn(0x7fd6c00cf8a0 msgr2=0x7fd6c013ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 shutdown_connections 2024-03-06T07:27:00.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.312+0000 7fd6c7646640 1 -- 172.21.15.77:0/3070084322 wait complete. 2024-03-06T07:27:00.325 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:00.326 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:00.326 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:00.326 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:00.326 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:00.326 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:00.326 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:00.326 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:00.327 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:00.327 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:00.327 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:00.327 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:00.327 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:00.327 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:00.327 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:00.327 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:00.335 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:00.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:00.336 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:00.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:00.342 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:00.342 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:00.343 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:00.343 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:00.377 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:00.377 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:00.377 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:00.378 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:00.385 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:00.385 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:00.386 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:00.386 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:00.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:00.453 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:27:00.454 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:27:00.469 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:27:00.469 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:27:00.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.492+0000 7fbaa580e640 1 Processor -- start 2024-03-06T07:27:00.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.492+0000 7fbaa580e640 1 -- start start 2024-03-06T07:27:00.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.493+0000 7fbaa580e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa0162970 0x7fbaa00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.493+0000 7fbaa580e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00d22c0 con 0x7fbaa00b4460 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.493+0000 7fbaa580e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00d25a0 con 0x7fbaa0161cf0 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.493+0000 7fba9effd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbaa00b4460 0x7fbaa00b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56706/0 (socket says 172.21.15.77:56706) 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.493+0000 7fba9effd640 1 -- 172.21.15.77:0/372765723 learned_addr learned my addr 172.21.15.77:0/372765723 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.493+0000 7fbaa580e640 1 -- 172.21.15.77:0/372765723 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa0161b00 con 0x7fbaa0162970 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.493+0000 7fba97fff640 1 --2- 172.21.15.77:0/372765723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa0162970 0x7fbaa00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.493+0000 7fba9e7fc640 1 -- 172.21.15.77:0/372765723 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8713002 0 0) 0x7fbaa00d22c0 con 0x7fbaa00b4460 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.494+0000 7fba9e7fc640 1 -- 172.21.15.77:0/372765723 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba8c003520 con 0x7fbaa00b4460 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.494+0000 7fba9e7fc640 1 -- 172.21.15.77:0/372765723 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949118554 0 0) 0x7fbaa00d25a0 con 0x7fbaa0161cf0 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.494+0000 7fba9e7fc640 1 -- 172.21.15.77:0/372765723 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa00d22c0 con 0x7fbaa0161cf0 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.494+0000 7fba97fff640 1 -- 172.21.15.77:0/372765723 >> v1:172.21.15.158:6789/0 conn(0x7fbaa0161cf0 legacy=0x7fbaa00cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.494+0000 7fba97fff640 1 -- 172.21.15.77:0/372765723 >> v1:172.21.15.77:6789/0 conn(0x7fbaa00b4460 legacy=0x7fbaa00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.494+0000 7fba9e7fc640 1 -- 172.21.15.77:0/372765723 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2861754732 0 0) 0x7fba8c003520 con 0x7fbaa00b4460 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.494+0000 7fba97fff640 1 -- 172.21.15.77:0/372765723 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00d2880 con 0x7fbaa0162970 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fba97fff640 1 --2- 172.21.15.77:0/372765723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa0162970 0x7fbaa00d4800 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba880097d0 tx=0x7fba8802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1ef5eb04ca8222b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fba9e7fc640 1 -- 172.21.15.77:0/372765723 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88037070 con 0x7fbaa0162970 2024-03-06T07:27:00.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fba9e7fc640 1 -- 172.21.15.77:0/372765723 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba88030510 con 0x7fbaa0162970 2024-03-06T07:27:00.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fba9e7fc640 1 -- 172.21.15.77:0/372765723 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90001670 con 0x7fbaa0162970 2024-03-06T07:27:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fbaa580e640 1 -- 172.21.15.77:0/372765723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa0162970 msgr2=0x7fbaa00d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fbaa580e640 1 --2- 172.21.15.77:0/372765723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa0162970 0x7fbaa00d4800 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba880097d0 tx=0x7fba8802b800 comp rx=0 tx=0).stop 2024-03-06T07:27:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fbaa580e640 1 -- 172.21.15.77:0/372765723 shutdown_connections 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fbaa580e640 1 --2- 172.21.15.77:0/372765723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa0162970 0x7fbaa00d4800 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fbaa580e640 1 -- 172.21.15.77:0/372765723 >> 172.21.15.77:0/372765723 conn(0x7fbaa00d3ce0 msgr2=0x7fbaa00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fbaa580e640 1 -- 172.21.15.77:0/372765723 shutdown_connections 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.495+0000 7fbaa580e640 1 -- 172.21.15.77:0/372765723 wait complete. 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.496+0000 7fbaa580e640 1 Processor -- start 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.496+0000 7fbaa580e640 1 -- start start 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fbaa580e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa00b4460 0x7fbaa01563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fbaa580e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa0161cf0 0x7fbaa00cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fbaa580e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa0162970 0x7fbaa00cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fbaa580e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00ce950 con 0x7fbaa0161cf0 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fbaa580e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00cecc0 con 0x7fbaa0162970 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fbaa580e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00b4840 con 0x7fbaa00b4460 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fba97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa0161cf0 0x7fbaa00cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fba97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa0161cf0 0x7fbaa00cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58582/0 (socket says 172.21.15.77:58582) 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fba97fff640 1 -- 172.21.15.77:0/3548083228 learned_addr learned my addr 172.21.15.77:0/3548083228 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fba9f7fe640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa0162970 0x7fbaa00cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.497+0000 7fba9effd640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa00b4460 0x7fbaa01563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba97fff640 1 -- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa00b4460 msgr2=0x7fbaa01563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba97fff640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa00b4460 0x7fbaa01563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba97fff640 1 -- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa0162970 msgr2=0x7fbaa00cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba97fff640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa0162970 0x7fbaa00cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba97fff640 1 -- 172.21.15.77:0/3548083228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00cba10 con 0x7fbaa0161cf0 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba9effd640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa00b4460 0x7fbaa01563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba97fff640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa0161cf0 0x7fbaa00cd1e0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fba880097d0 tx=0x7fba880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba9cff9640 1 -- 172.21.15.77:0/3548083228 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88042070 con 0x7fbaa0161cf0 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba9cff9640 1 -- 172.21.15.77:0/3548083228 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba88004220 con 0x7fbaa0161cf0 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa00cbcf0 con 0x7fbaa0161cf0 2024-03-06T07:27:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fba9cff9640 1 -- 172.21.15.77:0/3548083228 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88037040 con 0x7fbaa0161cf0 2024-03-06T07:27:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.498+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa00cc2b0 con 0x7fbaa0161cf0 2024-03-06T07:27:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.499+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa004f330 con 0x7fbaa0161cf0 2024-03-06T07:27:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.499+0000 7fba9cff9640 1 -- 172.21.15.77:0/3548083228 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba88003610 con 0x7fbaa0161cf0 2024-03-06T07:27:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.500+0000 7fba9cff9640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b500 0x7fba8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.500+0000 7fba9cff9640 1 -- 172.21.15.77:0/3548083228 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fba880abc60 con 0x7fbaa0161cf0 2024-03-06T07:27:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.500+0000 7fba9effd640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b500 0x7fba8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.500+0000 7fba9effd640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b500 0x7fba8006d9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fba900010e0 tx=0x7fba90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.504+0000 7fba9cff9640 1 -- 172.21.15.77:0/3548083228 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba8807ac40 con 0x7fbaa0161cf0 2024-03-06T07:27:00.612 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:00.614 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:00.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:00.691+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbaa006b180 con 0x7fbaa0161cf0 2024-03-06T07:27:01.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.477+0000 7fba9cff9640 1 -- 172.21.15.77:0/3548083228 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v102) v1 ==== 97+0+0 (secure 0 0 0) 0x7fba8807a540 con 0x7fbaa0161cf0 2024-03-06T07:27:01.479 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:27:01.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b500 msgr2=0x7fba8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b500 0x7fba8006d9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fba900010e0 tx=0x7fba90009040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa0161cf0 msgr2=0x7fbaa00cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa0161cf0 0x7fbaa00cd1e0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fba880097d0 tx=0x7fba880046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 shutdown_connections 2024-03-06T07:27:01.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b500 0x7fba8006d9c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa0161cf0 0x7fbaa00cd1e0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa0162970 0x7fbaa00cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 --2- 172.21.15.77:0/3548083228 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa00b4460 0x7fbaa01563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.479+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 >> 172.21.15.77:0/3548083228 conn(0x7fbaa00d3ce0 msgr2=0x7fbaa00cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.480+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 shutdown_connections 2024-03-06T07:27:01.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.480+0000 7fbaa580e640 1 -- 172.21.15.77:0/3548083228 wait complete. 2024-03-06T07:27:01.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:01.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.638+0000 7f11442f4640 1 Processor -- start 2024-03-06T07:27:01.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.638+0000 7f11442f4640 1 -- start start 2024-03-06T07:27:01.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f11442f4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f113c06da40 0x7f113c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f11442f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c0cd7a0 0x7f113c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f11442f4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0cc980 con 0x7f113c1424f0 2024-03-06T07:27:01.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f11442f4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113c0cd220 con 0x7f113c06da40 2024-03-06T07:27:01.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f11442f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0ccc60 con 0x7f113c0cd7a0 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f114188f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f113c06da40 0x7f113c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f1142891640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c0cd7a0 0x7f113c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f1142090640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f113c1424f0 0x7f113c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:53382/0 (socket says 172.21.15.77:53382) 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f114188f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f113c06da40 0x7f113c0cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:37136/0 (socket says 172.21.15.77:37136) 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f1142891640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c0cd7a0 0x7f113c15cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:58576/0 (socket says 172.21.15.77:58576) 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f1142090640 1 -- 172.21.15.77:0/2889284776 learned_addr learned my addr 172.21.15.77:0/2889284776 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f114188f640 1 -- 172.21.15.77:0/2889284776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c0cd7a0 msgr2=0x7f113c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f114188f640 1 --2- 172.21.15.77:0/2889284776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c0cd7a0 0x7f113c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f114188f640 1 -- 172.21.15.77:0/2889284776 >> v1:172.21.15.158:6789/0 conn(0x7f113c1424f0 legacy=0x7f113c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f114108e640 1 -- 172.21.15.77:0/2889284776 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540135639 0 0) 0x7f113c0cc980 con 0x7f113c1424f0 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f114188f640 1 -- 172.21.15.77:0/2889284776 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0ccf40 con 0x7f113c06da40 2024-03-06T07:27:01.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.639+0000 7f1142891640 1 --2- 172.21.15.77:0/2889284776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c0cd7a0 0x7f113c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.640+0000 7f114188f640 1 --2- 172.21.15.77:0/2889284776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c06da40 0x7f113c0cc2f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f112c0097a0 tx=0x7f112c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2123877ac2c2c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.640+0000 7f114108e640 1 -- 172.21.15.77:0/2889284776 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c035020 con 0x7f113c06da40 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.640+0000 7f114108e640 1 -- 172.21.15.77:0/2889284776 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c031810 con 0x7f113c06da40 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.640+0000 7f114108e640 1 -- 172.21.15.77:0/2889284776 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c031be0 con 0x7f113c06da40 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.641+0000 7f11442f4640 1 -- 172.21.15.77:0/2889284776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c06da40 msgr2=0x7f113c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.641+0000 7f11442f4640 1 --2- 172.21.15.77:0/2889284776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c06da40 0x7f113c0cc2f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f112c0097a0 tx=0x7f112c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.641+0000 7f11442f4640 1 -- 172.21.15.77:0/2889284776 shutdown_connections 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.641+0000 7f11442f4640 1 --2- 172.21.15.77:0/2889284776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c0cd7a0 0x7f113c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.641+0000 7f11442f4640 1 --2- 172.21.15.77:0/2889284776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c06da40 0x7f113c0cc2f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.641+0000 7f11442f4640 1 -- 172.21.15.77:0/2889284776 >> 172.21.15.77:0/2889284776 conn(0x7f113c0cf8a0 msgr2=0x7f113c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.641+0000 7f11442f4640 1 -- 172.21.15.77:0/2889284776 shutdown_connections 2024-03-06T07:27:01.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.641+0000 7f11442f4640 1 -- 172.21.15.77:0/2889284776 wait complete. 2024-03-06T07:27:01.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.642+0000 7f11442f4640 1 Processor -- start 2024-03-06T07:27:01.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.642+0000 7f11442f4640 1 -- start start 2024-03-06T07:27:01.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f11442f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c06da40 0x7f113c154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f11442f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c0cd7a0 0x7f113c140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f11442f4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c1424f0 0x7f113c0b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f11442f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c071ce0 con 0x7f113c0cd7a0 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f11442f4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c071b60 con 0x7f113c1424f0 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f11442f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c071e60 con 0x7f113c06da40 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c0cd7a0 0x7f113c140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c0cd7a0 0x7f113c140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58596/0 (socket says 172.21.15.77:58596) 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 -- 172.21.15.77:0/1130931866 learned_addr learned my addr 172.21.15.77:0/1130931866 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f1142090640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c06da40 0x7f113c154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f1142891640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c1424f0 0x7f113c0b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 -- 172.21.15.77:0/1130931866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c06da40 msgr2=0x7f113c154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c06da40 0x7f113c154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 -- 172.21.15.77:0/1130931866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c1424f0 msgr2=0x7f113c0b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c1424f0 0x7f113c0b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 -- 172.21.15.77:0/1130931866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c13f3d0 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.643+0000 7f114188f640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c0cd7a0 0x7f113c140c10 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f112c02bec0 tx=0x7f112c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.644+0000 7f1132ffd640 1 -- 172.21.15.77:0/1130931866 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c041070 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.644+0000 7f1132ffd640 1 -- 172.21.15.77:0/1130931866 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c03ad00 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.644+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c1e5a20 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.644+0000 7f1132ffd640 1 -- 172.21.15.77:0/1130931866 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c035040 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.644+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c1e5fe0 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.645+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c04f330 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.645+0000 7f1132ffd640 1 -- 172.21.15.77:0/1130931866 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f112c038020 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.645+0000 7f1132ffd640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f112006b4d0 0x7f112006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.646+0000 7f1142090640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f112006b4d0 0x7f112006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.646+0000 7f1132ffd640 1 -- 172.21.15.77:0/1130931866 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f112c0abad0 con 0x7f113c0cd7a0 2024-03-06T07:27:01.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.646+0000 7f1142090640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f112006b4d0 0x7f112006d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1138000f60 tx=0x7f1138007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.650+0000 7f1132ffd640 1 -- 172.21.15.77:0/1130931866 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f112c07aca0 con 0x7f113c0cd7a0 2024-03-06T07:27:01.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.889+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f113c0d0620 con 0x7f113c0cd7a0 2024-03-06T07:27:01.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.890+0000 7f1132ffd640 1 -- 172.21.15.77:0/1130931866 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f112c07a5a0 con 0x7f113c0cd7a0 2024-03-06T07:27:01.892 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:01.892 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:27:01.892 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:27:01.892 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.892 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.893 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:27:01.895 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.896 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.897 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.898 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.899 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.900 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:27:01.901 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:27:01.902 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.904 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:01.905 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:27:01.906 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.906 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:01.906 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:27:01.906 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:27:01.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f112006b4d0 msgr2=0x7f112006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f112006b4d0 0x7f112006d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1138000f60 tx=0x7f1138007c00 comp rx=0 tx=0).stop 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c0cd7a0 msgr2=0x7f113c140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c0cd7a0 0x7f113c140c10 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f112c02bec0 tx=0x7f112c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 shutdown_connections 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f112006b4d0 0x7f112006d990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c0cd7a0 0x7f113c140c10 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f113c1424f0 0x7f113c0b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 --2- 172.21.15.77:0/1130931866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f113c06da40 0x7f113c154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 >> 172.21.15.77:0/1130931866 conn(0x7f113c0cf8a0 msgr2=0x7f113c072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 shutdown_connections 2024-03-06T07:27:01.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:01.892+0000 7f11442f4640 1 -- 172.21.15.77:0/1130931866 wait complete. 2024-03-06T07:27:01.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:02.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.008+0000 7ff60189a640 1 Processor -- start 2024-03-06T07:27:02.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.008+0000 7ff60189a640 1 -- start start 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.008+0000 7ff60189a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 0x7ff5fc0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.008+0000 7ff60189a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff5fc06da40 0x7ff5fc15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.008+0000 7ff60189a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0cc980 con 0x7ff5fc0cd7a0 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.008+0000 7ff60189a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc0cd220 con 0x7ff5fc06da40 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.008+0000 7ff60189a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0ccc60 con 0x7ff5fc1424f0 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5fa7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff5fc0cd7a0 0x7ff5fc06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56722/0 (socket says 172.21.15.77:56722) 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5fa7fc640 1 -- 172.21.15.77:0/3195241940 learned_addr learned my addr 172.21.15.77:0/3195241940 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420941266 0 0) 0x7ff5fc0cc980 con 0x7ff5fc0cd7a0 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e8003520 con 0x7ff5fc0cd7a0 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5faffd640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 0x7ff5fc0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5fb7fe640 1 --2- 172.21.15.77:0/3195241940 >> v2:172.21.15.158:3300/0 conn(0x7ff5fc06da40 0x7ff5fc15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2478371216 0 0) 0x7ff5e8003520 con 0x7ff5fc0cd7a0 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 msgr2=0x7ff5fc0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 0x7ff5fc0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 >> v2:172.21.15.158:3300/0 conn(0x7ff5fc06da40 msgr2=0x7ff5fc15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 --2- 172.21.15.77:0/3195241940 >> v2:172.21.15.158:3300/0 conn(0x7ff5fc06da40 0x7ff5fc15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0ccf40 con 0x7ff5fc0cd7a0 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ff5e4001670 con 0x7ff5fc0cd7a0 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 >> v1:172.21.15.77:6789/0 conn(0x7ff5fc0cd7a0 legacy=0x7ff5fc06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e80039b0 0x7ff5e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 0x7ff5e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5e8004d40 0x7ff5e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0cd220 con 0x7ff5e8004d40 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0ccc60 con 0x7ff5e80039b0 2024-03-06T07:27:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e800b9c0 con 0x7ff5fc1424f0 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5fb7fe640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5e8004d40 0x7ff5e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5faffd640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e80039b0 0x7ff5e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.009+0000 7ff5fa7fc640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 0x7ff5e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5fb7fe640 1 -- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 msgr2=0x7ff5e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5fb7fe640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 0x7ff5e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5fb7fe640 1 -- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e80039b0 msgr2=0x7ff5e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5fb7fe640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e80039b0 0x7ff5e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5fb7fe640 1 -- 172.21.15.77:0/3195241940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e8003520 con 0x7ff5e8004d40 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5fa7fc640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1424f0 0x7ff5e8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5fb7fe640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5e8004d40 0x7ff5e80051c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009fc0 tx=0x7ff5e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76baf9e5ffeeb084 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4039070 con 0x7ff5e8004d40 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f402cc60 con 0x7ff5e8004d40 2024-03-06T07:27:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.010+0000 7ff5f9ffb640 1 -- 172.21.15.77:0/3195241940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4009c10 con 0x7ff5e8004d40 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.011+0000 7ff60189a640 1 -- 172.21.15.77:0/3195241940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5e8004d40 msgr2=0x7ff5e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.011+0000 7ff60189a640 1 --2- 172.21.15.77:0/3195241940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5e8004d40 0x7ff5e80051c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009fc0 tx=0x7ff5e8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.011+0000 7ff5fb7fe640 1 -- 172.21.15.77:0/3195241940 reap_dead start 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.011+0000 7ff60189a640 1 -- 172.21.15.77:0/3195241940 shutdown_connections 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.011+0000 7ff60189a640 1 -- 172.21.15.77:0/3195241940 >> 172.21.15.77:0/3195241940 conn(0x7ff5fc0cf8a0 msgr2=0x7ff5fc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.011+0000 7ff60189a640 1 -- 172.21.15.77:0/3195241940 shutdown_connections 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.011+0000 7ff60189a640 1 -- 172.21.15.77:0/3195241940 wait complete. 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.012+0000 7ff60189a640 1 Processor -- start 2024-03-06T07:27:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.012+0000 7ff60189a640 1 -- start start 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff60189a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e8004d40 0x7ff5fc13e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff60189a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1f2e30 0x7ff5fc1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff60189a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc1559e0 0x7ff5fc15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff60189a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0cc470 con 0x7ff5fc1559e0 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff60189a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0cc2f0 con 0x7ff5e8004d40 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff60189a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0cc5f0 con 0x7ff5fc1f2e30 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc1559e0 0x7ff5fc15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc1559e0 0x7ff5fc15f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58608/0 (socket says 172.21.15.77:58608) 2024-03-06T07:27:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 -- 172.21.15.77:0/543580702 learned_addr learned my addr 172.21.15.77:0/543580702 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5faffd640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e8004d40 0x7ff5fc13e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fa7fc640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1f2e30 0x7ff5fc1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 -- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1f2e30 msgr2=0x7ff5fc1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1f2e30 0x7ff5fc1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 -- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e8004d40 msgr2=0x7ff5fc13e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e8004d40 0x7ff5fc13e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 -- 172.21.15.77:0/543580702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc154dd0 con 0x7ff5fc1559e0 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5faffd640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e8004d40 0x7ff5fc13e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fa7fc640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1f2e30 0x7ff5fc1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.013+0000 7ff5fb7fe640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc1559e0 0x7ff5fc15f9f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400e230 tx=0x7ff5f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.014+0000 7ff5f3fff640 1 -- 172.21.15.77:0/543580702 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4039030 con 0x7ff5fc1559e0 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.014+0000 7ff5f3fff640 1 -- 172.21.15.77:0/543580702 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f403a040 con 0x7ff5fc1559e0 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.014+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc13ed90 con 0x7ff5fc1559e0 2024-03-06T07:27:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.014+0000 7ff5f3fff640 1 -- 172.21.15.77:0/543580702 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4003990 con 0x7ff5fc1559e0 2024-03-06T07:27:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.014+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc13f350 con 0x7ff5fc1559e0 2024-03-06T07:27:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.015+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc141c10 con 0x7ff5fc1559e0 2024-03-06T07:27:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.015+0000 7ff5f3fff640 1 -- 172.21.15.77:0/543580702 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5f4045030 con 0x7ff5fc1559e0 2024-03-06T07:27:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.016+0000 7ff5f3fff640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff5dc06b500 0x7ff5dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.016+0000 7ff5faffd640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff5dc06b500 0x7ff5dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.016+0000 7ff5f3fff640 1 -- 172.21.15.77:0/543580702 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff5f40abe20 con 0x7ff5fc1559e0 2024-03-06T07:27:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.016+0000 7ff5faffd640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff5dc06b500 0x7ff5dc06d9c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009a40 tx=0x7ff5ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.020+0000 7ff5f3fff640 1 -- 172.21.15.77:0/543580702 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5fc141c10 con 0x7ff5fc1559e0 2024-03-06T07:27:02.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.193+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5fc04f330 con 0x7ff5fc1559e0 2024-03-06T07:27:02.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.193+0000 7ff5f3fff640 1 -- 172.21.15.77:0/543580702 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+12881 (secure 0 0 0) 0x7ff5f404c020 con 0x7ff5fc1559e0 2024-03-06T07:27:02.195 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:02.196 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":103,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:01.483046+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.195+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff5dc06b500 msgr2=0x7ff5dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.195+0000 7ff60189a640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff5dc06b500 0x7ff5dc06d9c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009a40 tx=0x7ff5ec008040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.195+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc1559e0 msgr2=0x7ff5fc15f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.195+0000 7ff60189a640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc1559e0 0x7ff5fc15f9f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400e230 tx=0x7ff5f4009750 comp rx=0 tx=0).stop 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.196+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 shutdown_connections 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.196+0000 7ff60189a640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff5dc06b500 0x7ff5dc06d9c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.196+0000 7ff60189a640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc1559e0 0x7ff5fc15f9f0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.196+0000 7ff60189a640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5fc1f2e30 0x7ff5fc1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.196+0000 7ff60189a640 1 --2- 172.21.15.77:0/543580702 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5e8004d40 0x7ff5fc13e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.196+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 >> 172.21.15.77:0/543580702 conn(0x7ff5fc0cf8a0 msgr2=0x7ff5fc0cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.196+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 shutdown_connections 2024-03-06T07:27:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.196+0000 7ff60189a640 1 -- 172.21.15.77:0/543580702 wait complete. 2024-03-06T07:27:02.209 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:02.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.310+0000 7ffaf0ff6640 1 Processor -- start 2024-03-06T07:27:02.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.310+0000 7ffaf0ff6640 1 -- start start 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.311+0000 7ffaf0ff6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffaec0d0af0 0x7ffaec0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.311+0000 7ffaf0ff6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0cd4b0 con 0x7ffaec06fd10 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.311+0000 7ffaf0ff6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0cd790 con 0x7ffaec15c050 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.311+0000 7ffaf0ff6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec0cda70 con 0x7ffaec0d0af0 2024-03-06T07:27:02.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.311+0000 7ffae9d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffaec06fd10 0x7ffaec0cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56730/0 (socket says 172.21.15.77:56730) 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.311+0000 7ffae9d9b640 1 -- 172.21.15.77:0/1276523961 learned_addr learned my addr 172.21.15.77:0/1276523961 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135173688 0 0) 0x7ffaec0cd4b0 con 0x7ffaec06fd10 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffaead9d640 1 --2- 172.21.15.77:0/1276523961 >> v2:172.21.15.158:3300/0 conn(0x7ffaec0d0af0 0x7ffaec0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad4003520 con 0x7ffaec06fd10 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 337374932 0 0) 0x7ffad4003520 con 0x7ffaec06fd10 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 >> v2:172.21.15.158:3300/0 conn(0x7ffaec0d0af0 msgr2=0x7ffaec0cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 --2- 172.21.15.77:0/1276523961 >> v2:172.21.15.158:3300/0 conn(0x7ffaec0d0af0 0x7ffaec0cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 >> v1:172.21.15.158:6789/0 conn(0x7ffaec15c050 legacy=0x7ffaec06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0cdd50 con 0x7ffaec06fd10 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ffae0001670 con 0x7ffaec06fd10 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 >> v1:172.21.15.77:6789/0 conn(0x7ffaec06fd10 legacy=0x7ffaec0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffad40039b0 0x7ffad4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffad4004800 0x7ffad4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffad40051c0 0x7ffad400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0cda70 con 0x7ffad40051c0 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad400c160 con 0x7ffad40039b0 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad400c360 con 0x7ffad4004800 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffaead9d640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffad40051c0 0x7ffad400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffaea59c640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffad40039b0 0x7ffad4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae9d9b640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffad4004800 0x7ffad4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffaead9d640 1 -- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffad4004800 msgr2=0x7ffad4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffaead9d640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffad4004800 0x7ffad4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffaead9d640 1 -- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffad40039b0 msgr2=0x7ffad4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffaead9d640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffad40039b0 0x7ffad4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffaead9d640 1 -- 172.21.15.77:0/1276523961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad4003520 con 0x7ffad40051c0 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.312+0000 7ffae9d9b640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffad4004800 0x7ffad4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.313+0000 7ffaead9d640 1 -- 172.21.15.77:0/1276523961 reap_dead start 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.313+0000 7ffaead9d640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffad40051c0 0x7ffad400b9c0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009a90 tx=0x7ffad4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43d299ac5fbd269 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.313+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4039070 con 0x7ffad40051c0 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.313+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae402c6b0 con 0x7ffad40051c0 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.313+0000 7ffae959a640 1 -- 172.21.15.77:0/1276523961 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4009690 con 0x7ffad40051c0 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.314+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/1276523961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffad40051c0 msgr2=0x7ffad400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.314+0000 7ffaf0ff6640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffad40051c0 0x7ffad400b9c0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009a90 tx=0x7ffad4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.314+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/1276523961 shutdown_connections 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.314+0000 7ffaf0ff6640 1 --2- 172.21.15.77:0/1276523961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffad40051c0 0x7ffad400b9c0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.314+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/1276523961 >> 172.21.15.77:0/1276523961 conn(0x7ffaec0b3800 msgr2=0x7ffaec0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.314+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/1276523961 shutdown_connections 2024-03-06T07:27:02.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.314+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/1276523961 wait complete. 2024-03-06T07:27:02.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.315+0000 7ffaf0ff6640 1 Processor -- start 2024-03-06T07:27:02.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.315+0000 7ffaf0ff6640 1 -- start start 2024-03-06T07:27:02.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaf0ff6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffad40051c0 0x7ffaec14b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaf0ff6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffaec09f5f0 0x7ffaec09d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaf0ff6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaec09db00 0x7ffaec1f5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaf0ff6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec06f990 con 0x7ffaec09db00 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaf0ff6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec06f810 con 0x7ffad40051c0 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaf0ff6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec06fb10 con 0x7ffaec09f5f0 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaec09db00 0x7ffaec1f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaec09db00 0x7ffaec1f5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58628/0 (socket says 172.21.15.77:58628) 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 -- 172.21.15.77:0/2855883502 learned_addr learned my addr 172.21.15.77:0/2855883502 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 -- 172.21.15.77:0/2855883502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffaec09f5f0 msgr2=0x7ffaec09d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffaec09f5f0 0x7ffaec09d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 -- 172.21.15.77:0/2855883502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffad40051c0 msgr2=0x7ffaec14b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffad40051c0 0x7ffaec14b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 -- 172.21.15.77:0/2855883502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec09be60 con 0x7ffaec09db00 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaead9d640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaec09db00 0x7ffaec1f5510 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009bb0 tx=0x7ffae4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffadaffd640 1 -- 172.21.15.77:0/2855883502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4046070 con 0x7ffaec09db00 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffadaffd640 1 -- 172.21.15.77:0/2855883502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae403e890 con 0x7ffaec09db00 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffadaffd640 1 -- 172.21.15.77:0/2855883502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4039040 con 0x7ffaec09db00 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.316+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec09c140 con 0x7ffaec09db00 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.317+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec09c700 con 0x7ffaec09db00 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.318+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec09fe60 con 0x7ffaec09db00 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.318+0000 7ffadaffd640 1 -- 172.21.15.77:0/2855883502 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffae4049070 con 0x7ffaec09db00 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.318+0000 7ffadaffd640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffacc06b500 0x7ffacc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.318+0000 7ffadaffd640 1 -- 172.21.15.77:0/2855883502 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffae40aac90 con 0x7ffaec09db00 2024-03-06T07:27:02.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.318+0000 7ffaea59c640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffacc06b500 0x7ffacc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.319+0000 7ffaea59c640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffacc06b500 0x7ffacc06d9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000f80 tx=0x7ffadc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.323+0000 7ffadaffd640 1 -- 172.21.15.77:0/2855883502 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffae4079810 con 0x7ffaec09db00 2024-03-06T07:27:02.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.497+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffaec068ff0 con 0x7ffaec09db00 2024-03-06T07:27:02.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.497+0000 7ffadaffd640 1 -- 172.21.15.77:0/2855883502 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffae407d8b0 con 0x7ffaec09db00 2024-03-06T07:27:02.499 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:27:02.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffacc06b500 msgr2=0x7ffacc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffacc06b500 0x7ffacc06d9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000f80 tx=0x7ffadc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaec09db00 msgr2=0x7ffaec1f5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaec09db00 0x7ffaec1f5510 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009bb0 tx=0x7ffae4004730 comp rx=0 tx=0).stop 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 shutdown_connections 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffacc06b500 0x7ffacc06d9c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaec09db00 0x7ffaec1f5510 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffaec09f5f0 0x7ffaec09d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 --2- 172.21.15.77:0/2855883502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffad40051c0 0x7ffaec14b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 >> 172.21.15.77:0/2855883502 conn(0x7ffaec0b3800 msgr2=0x7ffaec0aeae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 shutdown_connections 2024-03-06T07:27:02.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.499+0000 7ffaf0ff6640 1 -- 172.21.15.77:0/2855883502 wait complete. 2024-03-06T07:27:02.512 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.611+0000 7f64eaf72640 1 Processor -- start 2024-03-06T07:27:02.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.611+0000 7f64eaf72640 1 -- start start 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.611+0000 7f64eaf72640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64e41515e0 0x7f64e40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.611+0000 7f64eaf72640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f64e415b430 0x7f64e4071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.611+0000 7f64eaf72640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e415b8e0 con 0x7f64e40cd7a0 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.611+0000 7f64eaf72640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e4070a70 con 0x7f64e41515e0 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.611+0000 7f64eaf72640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e4070d50 con 0x7f64e415b430 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.612+0000 7f64e8d0e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64e41515e0 0x7f64e40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.612+0000 7f64e8d0e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64e41515e0 0x7f64e40cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58638/0 (socket says 172.21.15.77:58638) 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.612+0000 7f64e8d0e640 1 -- 172.21.15.77:0/3830572760 learned_addr learned my addr 172.21.15.77:0/3830572760 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.612+0000 7f64e8d0e640 1 -- 172.21.15.77:0/3830572760 >> v2:172.21.15.158:3300/0 conn(0x7f64e415b430 msgr2=0x7f64e4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.612+0000 7f64e8d0e640 1 --2- 172.21.15.77:0/3830572760 >> v2:172.21.15.158:3300/0 conn(0x7f64e415b430 0x7f64e4071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.612+0000 7f64e8d0e640 1 -- 172.21.15.77:0/3830572760 >> v1:172.21.15.158:6789/0 conn(0x7f64e40cd7a0 legacy=0x7f64e40cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.612+0000 7f64e8d0e640 1 -- 172.21.15.77:0/3830572760 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e4071030 con 0x7f64e41515e0 2024-03-06T07:27:02.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64e8d0e640 1 --2- 172.21.15.77:0/3830572760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e41515e0 0x7f64e40cce40 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f64e406eb50 tx=0x7f64dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b009b6a7defb46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64e37fe640 1 -- 172.21.15.77:0/3830572760 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc02ca70 con 0x7f64e41515e0 2024-03-06T07:27:02.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64e37fe640 1 -- 172.21.15.77:0/3830572760 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc02cc10 con 0x7f64e41515e0 2024-03-06T07:27:02.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64e37fe640 1 -- 172.21.15.77:0/3830572760 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc0343f0 con 0x7f64e41515e0 2024-03-06T07:27:02.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64eaf72640 1 -- 172.21.15.77:0/3830572760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e41515e0 msgr2=0x7f64e40cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64eaf72640 1 --2- 172.21.15.77:0/3830572760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e41515e0 0x7f64e40cce40 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f64e406eb50 tx=0x7f64dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64eaf72640 1 -- 172.21.15.77:0/3830572760 shutdown_connections 2024-03-06T07:27:02.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64eaf72640 1 --2- 172.21.15.77:0/3830572760 >> v2:172.21.15.158:3300/0 conn(0x7f64e415b430 0x7f64e4071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64eaf72640 1 --2- 172.21.15.77:0/3830572760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e41515e0 0x7f64e40cce40 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64eaf72640 1 -- 172.21.15.77:0/3830572760 >> 172.21.15.77:0/3830572760 conn(0x7f64e40cf8a0 msgr2=0x7f64e40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64eaf72640 1 -- 172.21.15.77:0/3830572760 shutdown_connections 2024-03-06T07:27:02.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.613+0000 7f64eaf72640 1 -- 172.21.15.77:0/3830572760 wait complete. 2024-03-06T07:27:02.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.614+0000 7f64eaf72640 1 Processor -- start 2024-03-06T07:27:02.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.614+0000 7f64eaf72640 1 -- start start 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64eaf72640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e40cd7a0 0x7f64e414b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64eaf72640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64e41515e0 0x7f64e4140640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64eaf72640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64e415b430 0x7f64e41f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64eaf72640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e4071ef0 con 0x7f64e40cd7a0 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64eaf72640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e4071d70 con 0x7f64e41515e0 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64eaf72640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40cc3a0 con 0x7f64e415b430 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64e8d0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e40cd7a0 0x7f64e414b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64e8d0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e40cd7a0 0x7f64e414b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58646/0 (socket says 172.21.15.77:58646) 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.615+0000 7f64e8d0e640 1 -- 172.21.15.77:0/525063449 learned_addr learned my addr 172.21.15.77:0/525063449 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e950f640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64e415b430 0x7f64e41f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e3fff640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64e41515e0 0x7f64e4140640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e8d0e640 1 -- 172.21.15.77:0/525063449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64e415b430 msgr2=0x7f64e41f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e8d0e640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64e415b430 0x7f64e41f4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e8d0e640 1 -- 172.21.15.77:0/525063449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64e41515e0 msgr2=0x7f64e4140640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e8d0e640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64e41515e0 0x7f64e4140640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e8d0e640 1 -- 172.21.15.77:0/525063449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e414e6b0 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e8d0e640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e40cd7a0 0x7f64e414b960 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0093e0 tx=0x7f64dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e17fa640 1 -- 172.21.15.77:0/525063449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc043020 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e17fa640 1 -- 172.21.15.77:0/525063449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc03e450 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64e17fa640 1 -- 172.21.15.77:0/525063449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc03e840 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e414e990 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.616+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e414e360 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.617+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e4068840 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.617+0000 7f64e17fa640 1 -- 172.21.15.77:0/525063449 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64dc046020 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.618+0000 7f64e17fa640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.618+0000 7f64e17fa640 1 -- 172.21.15.77:0/525063449 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f64dc0ac890 con 0x7f64e40cd7a0 2024-03-06T07:27:02.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.618+0000 7f64e3fff640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.618+0000 7f64e3fff640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64c406b4d0 0x7f64c406d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f64e414a800 tx=0x7f64cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.622+0000 7f64e17fa640 1 -- 172.21.15.77:0/525063449 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64dc04d030 con 0x7f64e40cd7a0 2024-03-06T07:27:02.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.794+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f64e404f330 con 0x7f64e40cd7a0 2024-03-06T07:27:02.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.795+0000 7f64e17fa640 1 -- 172.21.15.77:0/525063449 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v103) v1 ==== 84+0+65 (secure 0 0 0) 0x7f64dc07bb20 con 0x7f64e40cd7a0 2024-03-06T07:27:02.797 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:02.797 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:27:02.797 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:02.797 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:27:02.797 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:27:02.797 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:02.797 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64c406b4d0 msgr2=0x7f64c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64c406b4d0 0x7f64c406d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f64e414a800 tx=0x7f64cc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e40cd7a0 msgr2=0x7f64e414b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e40cd7a0 0x7f64e414b960 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0093e0 tx=0x7f64dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 shutdown_connections 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64e415b430 0x7f64e41f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64e41515e0 0x7f64e4140640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 --2- 172.21.15.77:0/525063449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64e40cd7a0 0x7f64e414b960 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 >> 172.21.15.77:0/525063449 conn(0x7f64e40cf8a0 msgr2=0x7f64e4165910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 shutdown_connections 2024-03-06T07:27:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.797+0000 7f64eaf72640 1 -- 172.21.15.77:0/525063449 wait complete. 2024-03-06T07:27:02.810 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:02.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:02.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.912+0000 7f8be1156640 1 Processor -- start 2024-03-06T07:27:02.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.912+0000 7f8be1156640 1 -- start start 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.912+0000 7f8be1156640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8bdc15dfa0 0x7f8bdc0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.912+0000 7f8be1156640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc0d29f0 0x7f8bdc0cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.912+0000 7f8be1156640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bdc15e380 con 0x7f8bdc0d4630 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.912+0000 7f8be1156640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bdc0d2100 con 0x7f8bdc15dfa0 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.912+0000 7f8be1156640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0d1e30 con 0x7f8bdc0d29f0 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bd3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8bdc0d4630 0x7f8bdc0d22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56746/0 (socket says 172.21.15.77:56746) 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bd3fff640 1 -- 172.21.15.77:0/2827989502 learned_addr learned my addr 172.21.15.77:0/2827989502 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832698729 0 0) 0x7f8bdc15e380 con 0x7f8bdc0d4630 2024-03-06T07:27:02.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bdb59e640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc0d29f0 0x7f8bdc0cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc8003520 con 0x7f8bdc0d4630 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bdad9d640 1 --2- 172.21.15.77:0/2827989502 >> v2:172.21.15.158:3300/0 conn(0x7f8bdc15dfa0 0x7f8bdc0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4101232626 0 0) 0x7f8bc8003520 con 0x7f8bdc0d4630 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc0d29f0 msgr2=0x7f8bdc0cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc0d29f0 0x7f8bdc0cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 >> v2:172.21.15.158:3300/0 conn(0x7f8bdc15dfa0 msgr2=0x7f8bdc0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 --2- 172.21.15.77:0/2827989502 >> v2:172.21.15.158:3300/0 conn(0x7f8bdc15dfa0 0x7f8bdc0d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc0cf470 con 0x7f8bdc0d4630 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f8bc4001670 con 0x7f8bdc0d4630 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 >> v1:172.21.15.77:6789/0 conn(0x7f8bdc0d4630 legacy=0x7f8bdc0d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bc80039b0 0x7f8bc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc0d29f0 0x7f8bc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0d2100 con 0x7f8bc8004800 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0d1e30 con 0x7f8bc80039b0 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc800b9f0 con 0x7f8bdc0d29f0 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bd3fff640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.913+0000 7f8bdad9d640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bc80039b0 0x7f8bc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bdb59e640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc0d29f0 0x7f8bc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bd3fff640 1 -- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc0d29f0 msgr2=0x7f8bc80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bd3fff640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc0d29f0 0x7f8bc80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bd3fff640 1 -- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bc80039b0 msgr2=0x7f8bc8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bd3fff640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bc80039b0 0x7f8bc8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bd3fff640 1 -- 172.21.15.77:0/2827989502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8003520 con 0x7f8bc8004800 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bdad9d640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bc80039b0 0x7f8bc8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:02.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bd3fff640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bc8004c80 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0d4aa0 tx=0x7f8bc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de26471dcd7aed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4007d50 con 0x7f8bc8004800 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc4030070 con 0x7f8bc8004800 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.914+0000 7f8bda59c640 1 -- 172.21.15.77:0/2827989502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4005d10 con 0x7f8bc8004800 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.915+0000 7f8be1156640 1 -- 172.21.15.77:0/2827989502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 msgr2=0x7f8bc8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.915+0000 7f8be1156640 1 --2- 172.21.15.77:0/2827989502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bc8004c80 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0d4aa0 tx=0x7f8bc8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.915+0000 7f8bdb59e640 1 -- 172.21.15.77:0/2827989502 reap_dead start 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.915+0000 7f8be1156640 1 -- 172.21.15.77:0/2827989502 shutdown_connections 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.915+0000 7f8be1156640 1 -- 172.21.15.77:0/2827989502 >> 172.21.15.77:0/2827989502 conn(0x7f8bdc09d000 msgr2=0x7f8bdc0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.915+0000 7f8be1156640 1 -- 172.21.15.77:0/2827989502 shutdown_connections 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.915+0000 7f8be1156640 1 -- 172.21.15.77:0/2827989502 wait complete. 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.916+0000 7f8be1156640 1 Processor -- start 2024-03-06T07:27:02.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.916+0000 7f8be1156640 1 -- start start 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8be1156640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bdc0b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8be1156640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bdc09da70 0x7f8bdc1f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8be1156640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc09def0 0x7f8bdc1f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8be1156640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0cb190 con 0x7f8bc8004800 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8be1156640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0cfa30 con 0x7f8bdc09da70 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8be1156640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0cb310 con 0x7f8bdc09def0 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bdc0b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bdc0b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58674/0 (socket says 172.21.15.77:58674) 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdad9d640 1 -- 172.21.15.77:0/621158860 learned_addr learned my addr 172.21.15.77:0/621158860 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdb59e640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc09def0 0x7f8bdc1f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdad9d640 1 -- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc09def0 msgr2=0x7f8bdc1f7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bd3fff640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bdc09da70 0x7f8bdc1f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdad9d640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc09def0 0x7f8bdc1f7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdad9d640 1 -- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bdc09da70 msgr2=0x7f8bdc1f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdad9d640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bdc09da70 0x7f8bdc1f6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdad9d640 1 -- 172.21.15.77:0/621158860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc0b7880 con 0x7f8bc8004800 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.917+0000 7f8bdb59e640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc09def0 0x7f8bdc1f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.918+0000 7f8bdad9d640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bdc0b4ba0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc0115d0 tx=0x7f8bcc011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.918+0000 7f8bd37fe640 1 -- 172.21.15.77:0/621158860 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc00f8f0 con 0x7f8bc8004800 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.918+0000 7f8bd37fe640 1 -- 172.21.15.77:0/621158860 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc00fa90 con 0x7f8bc8004800 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.918+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bdc0b7b60 con 0x7f8bc8004800 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.918+0000 7f8bd37fe640 1 -- 172.21.15.77:0/621158860 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc00fe60 con 0x7f8bc8004800 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.918+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bdc0b8120 con 0x7f8bc8004800 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.919+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bdc04f330 con 0x7f8bc8004800 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.919+0000 7f8bd37fe640 1 -- 172.21.15.77:0/621158860 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bcc015460 con 0x7f8bc8004800 2024-03-06T07:27:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.920+0000 7f8bd37fe640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bbc06b500 0x7f8bbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.920+0000 7f8bd3fff640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bbc06b500 0x7f8bbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.920+0000 7f8bd37fe640 1 -- 172.21.15.77:0/621158860 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8bcc08a820 con 0x7f8bc8004800 2024-03-06T07:27:02.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.920+0000 7f8bd3fff640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bbc06b500 0x7f8bbc06d9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4034000 tx=0x7f8bc4007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:02.924+0000 7f8bd37fe640 1 -- 172.21.15.77:0/621158860 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bcc019030 con 0x7f8bc8004800 2024-03-06T07:27:03.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.097+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8bdc0cffc0 con 0x7f8bc8004800 2024-03-06T07:27:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.485+0000 7f8bd37fe640 1 -- 172.21.15.77:0/621158860 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v104) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8bcc0599f0 con 0x7f8bc8004800 2024-03-06T07:27:03.487 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bbc06b500 msgr2=0x7f8bbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bbc06b500 0x7f8bbc06d9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4034000 tx=0x7f8bc4007000 comp rx=0 tx=0).stop 2024-03-06T07:27:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 msgr2=0x7f8bdc0b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bdc0b4ba0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc0115d0 tx=0x7f8bcc011a90 comp rx=0 tx=0).stop 2024-03-06T07:27:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 shutdown_connections 2024-03-06T07:27:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8bdc09def0 0x7f8bdc1f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8bdc09da70 0x7f8bdc1f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bbc06b500 0x7f8bbc06d9c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 --2- 172.21.15.77:0/621158860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bc8004800 0x7f8bdc0b4ba0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 >> 172.21.15.77:0/621158860 conn(0x7f8bdc09d000 msgr2=0x7f8bdc0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 shutdown_connections 2024-03-06T07:27:03.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.487+0000 7f8be1156640 1 -- 172.21.15.77:0/621158860 wait complete. 2024-03-06T07:27:03.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:03.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.610+0000 7f2c3d7e9640 1 Processor -- start 2024-03-06T07:27:03.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.610+0000 7f2c3d7e9640 1 -- start start 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c3d7e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2c3815def0 0x7f2c380b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c3d7e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c3806cab0 0x7f2c380ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c3d7e9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3806d030 con 0x7f2c380ce880 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c3d7e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c380cf1c0 con 0x7f2c3806cab0 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c3d7e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c380cc760 con 0x7f2c3815def0 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c367fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c3806cab0 0x7f2c380ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c367fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c3806cab0 0x7f2c380ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58680/0 (socket says 172.21.15.77:58680) 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c367fc640 1 -- 172.21.15.77:0/2249249411 learned_addr learned my addr 172.21.15.77:0/2249249411 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c367fc640 1 -- 172.21.15.77:0/2249249411 >> v2:172.21.15.158:3300/0 conn(0x7f2c3815def0 msgr2=0x7f2c380b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c367fc640 1 --2- 172.21.15.77:0/2249249411 >> v2:172.21.15.158:3300/0 conn(0x7f2c3815def0 0x7f2c380b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c367fc640 1 -- 172.21.15.77:0/2249249411 >> v1:172.21.15.158:6789/0 conn(0x7f2c380ce880 legacy=0x7f2c380cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.611+0000 7f2c367fc640 1 -- 172.21.15.77:0/2249249411 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3806d310 con 0x7f2c3806cab0 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.612+0000 7f2c367fc640 1 --2- 172.21.15.77:0/2249249411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c3806cab0 0x7f2c380ce340 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f2c38071f60 tx=0x7f2c2c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b75ac0c670fe7ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.612+0000 7f2c35ffb640 1 -- 172.21.15.77:0/2249249411 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c034070 con 0x7f2c3806cab0 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.612+0000 7f2c35ffb640 1 -- 172.21.15.77:0/2249249411 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c030860 con 0x7f2c3806cab0 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.612+0000 7f2c35ffb640 1 -- 172.21.15.77:0/2249249411 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c030c30 con 0x7f2c3806cab0 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.613+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/2249249411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c3806cab0 msgr2=0x7f2c380ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.613+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/2249249411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c3806cab0 0x7f2c380ce340 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f2c38071f60 tx=0x7f2c2c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.613+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/2249249411 shutdown_connections 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.613+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/2249249411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c3806cab0 0x7f2c380ce340 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.613+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/2249249411 >> v2:172.21.15.158:3300/0 conn(0x7f2c3815def0 0x7f2c380b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.613+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/2249249411 >> 172.21.15.77:0/2249249411 conn(0x7f2c380b3690 msgr2=0x7f2c380b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.613+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/2249249411 shutdown_connections 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.613+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/2249249411 wait complete. 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.614+0000 7f2c3d7e9640 1 Processor -- start 2024-03-06T07:27:03.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.614+0000 7f2c3d7e9640 1 -- start start 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c3d7e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c3806cab0 0x7f2c38148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c3d7e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c380ce880 0x7f2c380c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c3d7e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c3815def0 0x7f2c380c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c3d7e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c380cc3f0 con 0x7f2c380ce880 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c3d7e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c380cecb0 con 0x7f2c3806cab0 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c3d7e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3815e2d0 con 0x7f2c3815def0 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c380ce880 0x7f2c380c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c380ce880 0x7f2c380c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58694/0 (socket says 172.21.15.77:58694) 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 -- 172.21.15.77:0/3353432908 learned_addr learned my addr 172.21.15.77:0/3353432908 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 -- 172.21.15.77:0/3353432908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c3815def0 msgr2=0x7f2c380c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c3815def0 0x7f2c380c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 -- 172.21.15.77:0/3353432908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c3806cab0 msgr2=0x7f2c38148270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c3806cab0 0x7f2c38148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 -- 172.21.15.77:0/3353432908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38148910 con 0x7f2c380ce880 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.615+0000 7f2c367fc640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c380ce880 0x7f2c380c7ed0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c000c00 tx=0x7f2c2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.616+0000 7f2c1f7fe640 1 -- 172.21.15.77:0/3353432908 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c048070 con 0x7f2c380ce880 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.616+0000 7f2c1f7fe640 1 -- 172.21.15.77:0/3353432908 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c030660 con 0x7f2c380ce880 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.616+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c38148bf0 con 0x7f2c380ce880 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.616+0000 7f2c1f7fe640 1 -- 172.21.15.77:0/3353432908 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c034040 con 0x7f2c380ce880 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.616+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c381491b0 con 0x7f2c380ce880 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.617+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3809fcf0 con 0x7f2c380ce880 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.617+0000 7f2c1f7fe640 1 -- 172.21.15.77:0/3353432908 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c2c04b070 con 0x7f2c380ce880 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.617+0000 7f2c1f7fe640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.617+0000 7f2c1f7fe640 1 -- 172.21.15.77:0/3353432908 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2c2c0ac1b0 con 0x7f2c380ce880 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.618+0000 7f2c36ffd640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.618+0000 7f2c36ffd640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800ab90 tx=0x7f2c28009250 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.622+0000 7f2c1f7fe640 1 -- 172.21.15.77:0/3353432908 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c2c07b210 con 0x7f2c380ce880 2024-03-06T07:27:03.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:03.793+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2c380d2230 con 0x7f2c380ce880 2024-03-06T07:27:04.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.487+0000 7f2c1f7fe640 1 -- 172.21.15.77:0/3353432908 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v105) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2c2c07ab10 con 0x7f2c380ce880 2024-03-06T07:27:04.489 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.490+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 msgr2=0x7f2c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800ab90 tx=0x7f2c28009250 comp rx=0 tx=0).stop 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c380ce880 msgr2=0x7f2c380c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c380ce880 0x7f2c380c7ed0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c000c00 tx=0x7f2c2c004730 comp rx=0 tx=0).stop 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 shutdown_connections 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c2406b4d0 0x7f2c2406d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c380ce880 0x7f2c380c7ed0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c3815def0 0x7f2c380c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 --2- 172.21.15.77:0/3353432908 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c3806cab0 0x7f2c38148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 >> 172.21.15.77:0/3353432908 conn(0x7f2c380b3690 msgr2=0x7f2c380b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 shutdown_connections 2024-03-06T07:27:04.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.491+0000 7f2c3d7e9640 1 -- 172.21.15.77:0/3353432908 wait complete. 2024-03-06T07:27:04.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:04.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.626+0000 7f1697d3e640 1 Processor -- start 2024-03-06T07:27:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.626+0000 7f1697d3e640 1 -- start start 2024-03-06T07:27:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1697d3e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1690158420 0x7f16900b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1697d3e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1690158800 0x7f16900b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1697d3e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169015e8a0 con 0x7f169015b370 2024-03-06T07:27:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1697d3e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169015e650 con 0x7f1690158420 2024-03-06T07:27:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1697d3e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169015eb80 con 0x7f1690158800 2024-03-06T07:27:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1695ada640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1690158420 0x7f16900b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1695ada640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1690158420 0x7f16900b35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58698/0 (socket says 172.21.15.77:58698) 2024-03-06T07:27:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1695ada640 1 -- 172.21.15.77:0/3382436081 learned_addr learned my addr 172.21.15.77:0/3382436081 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:04.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f16952d9640 1 --2- 172.21.15.77:0/3382436081 >> v2:172.21.15.158:3300/0 conn(0x7f1690158800 0x7f16900b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1695ada640 1 -- 172.21.15.77:0/3382436081 >> v2:172.21.15.158:3300/0 conn(0x7f1690158800 msgr2=0x7f16900b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1695ada640 1 --2- 172.21.15.77:0/3382436081 >> v2:172.21.15.158:3300/0 conn(0x7f1690158800 0x7f16900b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1695ada640 1 -- 172.21.15.77:0/3382436081 >> v1:172.21.15.158:6789/0 conn(0x7f169015b370 legacy=0x7f169015b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.627+0000 7f1695ada640 1 -- 172.21.15.77:0/3382436081 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169015ee60 con 0x7f1690158420 2024-03-06T07:27:04.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.628+0000 7f1695ada640 1 --2- 172.21.15.77:0/3382436081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1690158420 0x7f16900b35c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f16900cc690 tx=0x7f168c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774ef193e2725368 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.628+0000 7f1694ad8640 1 -- 172.21.15.77:0/3382436081 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c035020 con 0x7f1690158420 2024-03-06T07:27:04.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.628+0000 7f1694ad8640 1 -- 172.21.15.77:0/3382436081 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c031870 con 0x7f1690158420 2024-03-06T07:27:04.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.628+0000 7f1694ad8640 1 -- 172.21.15.77:0/3382436081 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c031c40 con 0x7f1690158420 2024-03-06T07:27:04.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.629+0000 7f1697d3e640 1 -- 172.21.15.77:0/3382436081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1690158420 msgr2=0x7f16900b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.629+0000 7f1697d3e640 1 --2- 172.21.15.77:0/3382436081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1690158420 0x7f16900b35c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f16900cc690 tx=0x7f168c02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:04.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.629+0000 7f1697d3e640 1 -- 172.21.15.77:0/3382436081 shutdown_connections 2024-03-06T07:27:04.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.629+0000 7f1697d3e640 1 --2- 172.21.15.77:0/3382436081 >> v2:172.21.15.158:3300/0 conn(0x7f1690158800 0x7f16900b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.629+0000 7f1697d3e640 1 --2- 172.21.15.77:0/3382436081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1690158420 0x7f16900b35c0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.629+0000 7f1697d3e640 1 -- 172.21.15.77:0/3382436081 >> 172.21.15.77:0/3382436081 conn(0x7f169006afb0 msgr2=0x7f169006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.629+0000 7f1697d3e640 1 -- 172.21.15.77:0/3382436081 shutdown_connections 2024-03-06T07:27:04.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.629+0000 7f1697d3e640 1 -- 172.21.15.77:0/3382436081 wait complete. 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.630+0000 7f1697d3e640 1 Processor -- start 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.630+0000 7f1697d3e640 1 -- start start 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.630+0000 7f1697d3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1690158420 0x7f169006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.630+0000 7f1697d3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169015b370 0x7f16900a9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.630+0000 7f1697d3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16901471f0 0x7f16900b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f1697d3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1690071ba0 con 0x7f169015b370 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f1697d3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1690071a20 con 0x7f16901471f0 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f1697d3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1690071d20 con 0x7f1690158420 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169015b370 0x7f16900a9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169015b370 0x7f16900a9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58710/0 (socket says 172.21.15.77:58710) 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 -- 172.21.15.77:0/2645822886 learned_addr learned my addr 172.21.15.77:0/2645822886 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f1695ada640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1690158420 0x7f169006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16962db640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16901471f0 0x7f16900b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 -- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1690158420 msgr2=0x7f169006d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1690158420 0x7f169006d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 -- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16901471f0 msgr2=0x7f16900b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16901471f0 0x7f16900b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 -- 172.21.15.77:0/2645822886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16901476c0 con 0x7f169015b370 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f1695ada640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1690158420 0x7f169006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16962db640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16901471f0 0x7f16900b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16952d9640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169015b370 0x7f16900a9a10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f168000d800 tx=0x7f168000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16867fc640 1 -- 172.21.15.77:0/2645822886 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168000b660 con 0x7f169015b370 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16867fc640 1 -- 172.21.15.77:0/2645822886 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1680019070 con 0x7f169015b370 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16901479a0 con 0x7f169015b370 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f16867fc640 1 -- 172.21.15.77:0/2645822886 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168000ba20 con 0x7f169015b370 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.631+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1690147f60 con 0x7f169015b370 2024-03-06T07:27:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.632+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169004f330 con 0x7f169015b370 2024-03-06T07:27:04.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.633+0000 7f16867fc640 1 -- 172.21.15.77:0/2645822886 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f168000bbc0 con 0x7f169015b370 2024-03-06T07:27:04.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.633+0000 7f16867fc640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f167406b4d0 0x7f167406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.633+0000 7f16867fc640 1 -- 172.21.15.77:0/2645822886 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f168008a5d0 con 0x7f169015b370 2024-03-06T07:27:04.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.633+0000 7f1695ada640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f167406b4d0 0x7f167406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.634+0000 7f1695ada640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f167406b4d0 0x7f167406d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f168c036000 tx=0x7f168c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.638+0000 7f16867fc640 1 -- 172.21.15.77:0/2645822886 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16800594b0 con 0x7f169015b370 2024-03-06T07:27:04.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:04.847+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f16900b2a30 con 0x7f169015b370 2024-03-06T07:27:05.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.505+0000 7f16867fc640 1 -- 172.21.15.77:0/2645822886 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v63) v1 ==== 174+0+0 (secure 0 0 0) 0x7f168005de20 con 0x7f169015b370 2024-03-06T07:27:05.507 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:27:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.518+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f167406b4d0 msgr2=0x7f167406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.518+0000 7f1697d3e640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f167406b4d0 0x7f167406d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f168c036000 tx=0x7f168c03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.518+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169015b370 msgr2=0x7f16900a9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.518+0000 7f1697d3e640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169015b370 0x7f16900a9a10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f168000d800 tx=0x7f168000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.520+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 shutdown_connections 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.520+0000 7f1697d3e640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f167406b4d0 0x7f167406d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.520+0000 7f1697d3e640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f169015b370 0x7f16900a9a10 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.520+0000 7f1697d3e640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16901471f0 0x7f16900b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.520+0000 7f1697d3e640 1 --2- 172.21.15.77:0/2645822886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1690158420 0x7f169006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.520+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 >> 172.21.15.77:0/2645822886 conn(0x7f169006afb0 msgr2=0x7f1690159590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.523+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 shutdown_connections 2024-03-06T07:27:05.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.523+0000 7f1697d3e640 1 -- 172.21.15.77:0/2645822886 wait complete. 2024-03-06T07:27:05.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:05.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.638+0000 7fe526b3a640 1 Processor -- start 2024-03-06T07:27:05.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.639+0000 7fe526b3a640 1 -- start start 2024-03-06T07:27:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.639+0000 7fe526b3a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe52015dfe0 0x7fe5200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.639+0000 7fe526b3a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe5200cf830 0x7fe5200d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.639+0000 7fe526b3a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5200d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.639+0000 7fe526b3a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5200b31e0 con 0x7fe52015dfe0 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.639+0000 7fe526b3a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5200b3d60 con 0x7fe5200cf830 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.639+0000 7fe526b3a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200b3430 con 0x7fe5200d27f0 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.639+0000 7fe5248d6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe52015dfe0 0x7fe5200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5248d6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe52015dfe0 0x7fe5200cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58712/0 (socket says 172.21.15.77:58712) 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5248d6640 1 -- 172.21.15.77:0/1784346143 learned_addr learned my addr 172.21.15.77:0/1784346143 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5248d6640 1 -- 172.21.15.77:0/1784346143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 msgr2=0x7fe5200d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5250d7640 1 --2- 172.21.15.77:0/1784346143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5200d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe51ffff640 1 --2- 172.21.15.77:0/1784346143 >> v2:172.21.15.158:3300/0 conn(0x7fe5200cf830 0x7fe5200d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5248d6640 1 --2- 172.21.15.77:0/1784346143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5200d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5248d6640 1 -- 172.21.15.77:0/1784346143 >> v2:172.21.15.158:3300/0 conn(0x7fe5200cf830 msgr2=0x7fe5200d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5248d6640 1 --2- 172.21.15.77:0/1784346143 >> v2:172.21.15.158:3300/0 conn(0x7fe5200cf830 0x7fe5200d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5248d6640 1 -- 172.21.15.77:0/1784346143 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5200b3710 con 0x7fe52015dfe0 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5250d7640 1 --2- 172.21.15.77:0/1784346143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5200d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:05.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe51ffff640 1 --2- 172.21.15.77:0/1784346143 >> v2:172.21.15.158:3300/0 conn(0x7fe5200cf830 0x7fe5200d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.640+0000 7fe5248d6640 1 --2- 172.21.15.77:0/1784346143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200cf2f0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fe5200d16b0 tx=0x7fe51002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba1d05fea691266 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe51f7fe640 1 -- 172.21.15.77:0/1784346143 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510035020 con 0x7fe52015dfe0 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe51f7fe640 1 -- 172.21.15.77:0/1784346143 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe510031870 con 0x7fe52015dfe0 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe51f7fe640 1 -- 172.21.15.77:0/1784346143 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510031c40 con 0x7fe52015dfe0 2024-03-06T07:27:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 -- 172.21.15.77:0/1784346143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 msgr2=0x7fe5200cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 --2- 172.21.15.77:0/1784346143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200cf2f0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fe5200d16b0 tx=0x7fe51002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 -- 172.21.15.77:0/1784346143 shutdown_connections 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 --2- 172.21.15.77:0/1784346143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5200d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 --2- 172.21.15.77:0/1784346143 >> v2:172.21.15.158:3300/0 conn(0x7fe5200cf830 0x7fe5200d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 --2- 172.21.15.77:0/1784346143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200cf2f0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 -- 172.21.15.77:0/1784346143 >> 172.21.15.77:0/1784346143 conn(0x7fe5200d3cd0 msgr2=0x7fe5200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 -- 172.21.15.77:0/1784346143 shutdown_connections 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.641+0000 7fe526b3a640 1 -- 172.21.15.77:0/1784346143 wait complete. 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.642+0000 7fe526b3a640 1 Processor -- start 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.642+0000 7fe526b3a640 1 -- start start 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe526b3a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5200cf830 0x7fe52015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe526b3a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5201f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe526b3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200b8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe526b3a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200d1df0 con 0x7fe52015dfe0 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe526b3a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5200d1c70 con 0x7fe5200cf830 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe526b3a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52015e3c0 con 0x7fe5200d27f0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe5250d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe5250d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200b8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58728/0 (socket says 172.21.15.77:58728) 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.643+0000 7fe5250d7640 1 -- 172.21.15.77:0/4294414495 learned_addr learned my addr 172.21.15.77:0/4294414495 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe5248d6640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5200cf830 0x7fe52015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe51ffff640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5201f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe5250d7640 1 -- 172.21.15.77:0/4294414495 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 msgr2=0x7fe5201f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe5250d7640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5201f6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe5250d7640 1 -- 172.21.15.77:0/4294414495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5200cf830 msgr2=0x7fe52015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe5250d7640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5200cf830 0x7fe52015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe5250d7640 1 -- 172.21.15.77:0/4294414495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5201f7400 con 0x7fe52015dfe0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe5250d7640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200b8300 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fe51800cf20 tx=0x7fe51800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe51d7fa640 1 -- 172.21.15.77:0/4294414495 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51801a070 con 0x7fe52015dfe0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe51d7fa640 1 -- 172.21.15.77:0/4294414495 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51800d840 con 0x7fe52015dfe0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5201f76e0 con 0x7fe52015dfe0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe51d7fa640 1 -- 172.21.15.77:0/4294414495 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51800dc30 con 0x7fe52015dfe0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.644+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5200cb220 con 0x7fe52015dfe0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.645+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5200cb420 con 0x7fe52015dfe0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.645+0000 7fe51d7fa640 1 -- 172.21.15.77:0/4294414495 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe51801d070 con 0x7fe52015dfe0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.646+0000 7fe51d7fa640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe50c06b4d0 0x7fe50c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.646+0000 7fe51d7fa640 1 -- 172.21.15.77:0/4294414495 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe51808aae0 con 0x7fe52015dfe0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.646+0000 7fe5248d6640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe50c06b4d0 0x7fe50c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.646+0000 7fe5248d6640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe50c06b4d0 0x7fe50c06d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe510036000 tx=0x7fe51003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.650+0000 7fe51d7fa640 1 -- 172.21.15.77:0/4294414495 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5200cb420 con 0x7fe52015dfe0 2024-03-06T07:27:05.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.823+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5200747a0 con 0x7fe52015dfe0 2024-03-06T07:27:05.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.824+0000 7fe51d7fa640 1 -- 172.21.15.77:0/4294414495 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16628 (secure 0 0 0) 0x7fe518059940 con 0x7fe52015dfe0 2024-03-06T07:27:05.826 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:05.826 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:04.848840+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:05.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe50c06b4d0 msgr2=0x7fe50c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe50c06b4d0 0x7fe50c06d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe510036000 tx=0x7fe51003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 msgr2=0x7fe5200b8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200b8300 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fe51800cf20 tx=0x7fe51800a7d0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 shutdown_connections 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe50c06b4d0 0x7fe50c06d990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe52015dfe0 0x7fe5200b8300 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe5200d27f0 0x7fe5201f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 --2- 172.21.15.77:0/4294414495 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5200cf830 0x7fe52015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 >> 172.21.15.77:0/4294414495 conn(0x7fe5200d3cd0 msgr2=0x7fe52006cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 shutdown_connections 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.826+0000 7fe526b3a640 1 -- 172.21.15.77:0/4294414495 wait complete. 2024-03-06T07:27:05.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.938+0000 7fe0ab12c640 1 Processor -- start 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.938+0000 7fe0ab12c640 1 -- start start 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0ab12c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe0a41424f0 0x7fe0a40cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0ab12c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe0a406da40 0x7fe0a415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0ab12c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a40cc980 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0ab12c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40cd220 con 0x7fe0a41424f0 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0ab12c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40ccc60 con 0x7fe0a406da40 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0a8ec8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe0a41424f0 0x7fe0a40cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0a8ec8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe0a41424f0 0x7fe0a40cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58730/0 (socket says 172.21.15.77:58730) 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0a8ec8640 1 -- 172.21.15.77:0/4293130908 learned_addr learned my addr 172.21.15.77:0/4293130908 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0a8ec8640 1 -- 172.21.15.77:0/4293130908 >> v2:172.21.15.158:3300/0 conn(0x7fe0a406da40 msgr2=0x7fe0a415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0a8ec8640 1 --2- 172.21.15.77:0/4293130908 >> v2:172.21.15.158:3300/0 conn(0x7fe0a406da40 0x7fe0a415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0a8ec8640 1 -- 172.21.15.77:0/4293130908 >> v1:172.21.15.158:6789/0 conn(0x7fe0a40cd7a0 legacy=0x7fe0a406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.939+0000 7fe0a8ec8640 1 -- 172.21.15.77:0/4293130908 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a40ccf40 con 0x7fe0a41424f0 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.940+0000 7fe0a8ec8640 1 --2- 172.21.15.77:0/4293130908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a41424f0 0x7fe0a40cc2f0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4068700 tx=0x7fe09402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e53a2d7b0820b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.940+0000 7fe0a37fe640 1 -- 172.21.15.77:0/4293130908 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094035020 con 0x7fe0a41424f0 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.940+0000 7fe0a37fe640 1 -- 172.21.15.77:0/4293130908 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe094031890 con 0x7fe0a41424f0 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.940+0000 7fe0a37fe640 1 -- 172.21.15.77:0/4293130908 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094031c60 con 0x7fe0a41424f0 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.941+0000 7fe0ab12c640 1 -- 172.21.15.77:0/4293130908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a41424f0 msgr2=0x7fe0a40cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.941+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/4293130908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a41424f0 0x7fe0a40cc2f0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4068700 tx=0x7fe09402b930 comp rx=0 tx=0).stop 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.941+0000 7fe0ab12c640 1 -- 172.21.15.77:0/4293130908 shutdown_connections 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.941+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/4293130908 >> v2:172.21.15.158:3300/0 conn(0x7fe0a406da40 0x7fe0a415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.941+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/4293130908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a41424f0 0x7fe0a40cc2f0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.941+0000 7fe0ab12c640 1 -- 172.21.15.77:0/4293130908 >> 172.21.15.77:0/4293130908 conn(0x7fe0a40cf8a0 msgr2=0x7fe0a40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.941+0000 7fe0ab12c640 1 -- 172.21.15.77:0/4293130908 shutdown_connections 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.941+0000 7fe0ab12c640 1 -- 172.21.15.77:0/4293130908 wait complete. 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.942+0000 7fe0ab12c640 1 Processor -- start 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.942+0000 7fe0ab12c640 1 -- start start 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0ab12c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe0a406da40 0x7fe0a41414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0ab12c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a40cd7a0 0x7fe0a41f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0ab12c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0a41424f0 0x7fe0a4151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0ab12c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a4071ce0 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0ab12c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a4071b60 con 0x7fe0a41424f0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0ab12c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a4071e60 con 0x7fe0a406da40 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a40cd7a0 0x7fe0a41f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a40cd7a0 0x7fe0a41f2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58746/0 (socket says 172.21.15.77:58746) 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 -- 172.21.15.77:0/3501935346 learned_addr learned my addr 172.21.15.77:0/3501935346 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a8ec8640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe0a406da40 0x7fe0a41414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a96c9640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0a41424f0 0x7fe0a4151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 -- 172.21.15.77:0/3501935346 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe0a406da40 msgr2=0x7fe0a41414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe0a406da40 0x7fe0a41414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 -- 172.21.15.77:0/3501935346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0a41424f0 msgr2=0x7fe0a4151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0a41424f0 0x7fe0a4151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 -- 172.21.15.77:0/3501935346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a41f2c10 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a3fff640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a40cd7a0 0x7fe0a41f2350 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c410 tx=0x7fe09800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.943+0000 7fe0a17fa640 1 -- 172.21.15.77:0/3501935346 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098017070 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.944+0000 7fe0a17fa640 1 -- 172.21.15.77:0/3501935346 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0980035f0 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.944+0000 7fe0a17fa640 1 -- 172.21.15.77:0/3501935346 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0980039e0 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.944+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a41f2ef0 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.944+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a41f2890 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.944+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a404f330 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.945+0000 7fe0a17fa640 1 -- 172.21.15.77:0/3501935346 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe09801a070 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.945+0000 7fe0a17fa640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe09006b4d0 0x7fe09006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.946+0000 7fe0a8ec8640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe09006b4d0 0x7fe09006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.946+0000 7fe0a17fa640 1 -- 172.21.15.77:0/3501935346 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe098089540 con 0x7fe0a40cd7a0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.946+0000 7fe0a8ec8640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe09006b4d0 0x7fe09006d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe094036000 tx=0x7fe09403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:05.950+0000 7fe0a17fa640 1 -- 172.21.15.77:0/3501935346 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0980010c0 con 0x7fe0a40cd7a0 2024-03-06T07:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.126+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0a4141ca0 con 0x7fe0a40cd7a0 2024-03-06T07:27:06.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.127+0000 7fe0a17fa640 1 -- 172.21.15.77:0/3501935346 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16628 (secure 0 0 0) 0x7fe098057e00 con 0x7fe0a40cd7a0 2024-03-06T07:27:06.129 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:06.129 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:04.848840+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.129+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe09006b4d0 msgr2=0x7fe09006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.129+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe09006b4d0 0x7fe09006d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe094036000 tx=0x7fe09403c040 comp rx=0 tx=0).stop 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.129+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a40cd7a0 msgr2=0x7fe0a41f2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.129+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a40cd7a0 0x7fe0a41f2350 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c410 tx=0x7fe09800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.129+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 shutdown_connections 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.129+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe09006b4d0 0x7fe09006d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.129+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0a40cd7a0 0x7fe0a41f2350 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.129+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe0a41424f0 0x7fe0a4151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.130+0000 7fe0ab12c640 1 --2- 172.21.15.77:0/3501935346 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe0a406da40 0x7fe0a41414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.130+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 >> 172.21.15.77:0/3501935346 conn(0x7fe0a40cf8a0 msgr2=0x7fe0a415cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.130+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 shutdown_connections 2024-03-06T07:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.130+0000 7fe0ab12c640 1 -- 172.21.15.77:0/3501935346 wait complete. 2024-03-06T07:27:06.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:06.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.245+0000 7f28173a1640 1 Processor -- start 2024-03-06T07:27:06.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.245+0000 7f28173a1640 1 -- start start 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f28173a1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 0x7f2810070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f28173a1640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28100cf430 con 0x7f281015e6d0 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f28173a1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281006e240 con 0x7f28100701a0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f28173a1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2810070ac0 con 0x7f28100b42f0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f281593e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f281015e6d0 0x7f281015eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56750/0 (socket says 172.21.15.77:56750) 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f281593e640 1 -- 172.21.15.77:0/3386360803 learned_addr learned my addr 172.21.15.77:0/3386360803 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f281493c640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 0x7f2810070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553898306 0 0) 0x7f28100cf430 con 0x7f281015e6d0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27fc003520 con 0x7f281015e6d0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434814956 0 0) 0x7f281006e240 con 0x7f28100701a0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28100cf430 con 0x7f28100701a0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266649313 0 0) 0x7f27fc003520 con 0x7f281015e6d0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 msgr2=0x7f2810070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 0x7f2810070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 >> v1:172.21.15.158:6789/0 conn(0x7f28100701a0 legacy=0x7f281006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28100cae80 con 0x7f281015e6d0 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f2808001670 con 0x7f281015e6d0 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f281493c640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 0x7f2810070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 >> v1:172.21.15.77:6789/0 conn(0x7f281015e6d0 legacy=0x7f281015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.246+0000 7f2807fff640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27fc003b70 0x7f27fc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f2807fff640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 0x7f27fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f2807fff640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27fc004d40 0x7f27fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2810070ac0 con 0x7f27fc003b70 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc00ba70 con 0x7f27fc004d40 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc00bbf0 con 0x7f28100b42f0 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f281513d640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27fc003b70 0x7f27fc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f281493c640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 0x7f27fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f281593e640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27fc004d40 0x7f27fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f281513d640 1 -- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 msgr2=0x7f27fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f281513d640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f28100b42f0 0x7f27fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f281513d640 1 -- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27fc004d40 msgr2=0x7f27fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f281513d640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f27fc004d40 0x7f27fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.247+0000 7f281513d640 1 -- 172.21.15.77:0/3386360803 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc003520 con 0x7f27fc003b70 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f281513d640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27fc003b70 0x7f27fc003f70 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f280c002000 tx=0x7f27fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d32ccc86f7f4c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c001bf0 con 0x7f27fc003b70 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c003b70 con 0x7f27fc003b70 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f2807fff640 1 -- 172.21.15.77:0/3386360803 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c0050d0 con 0x7f27fc003b70 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f28173a1640 1 -- 172.21.15.77:0/3386360803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27fc003b70 msgr2=0x7f27fc003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f28173a1640 1 --2- 172.21.15.77:0/3386360803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27fc003b70 0x7f27fc003f70 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f280c002000 tx=0x7f27fc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f281593e640 1 -- 172.21.15.77:0/3386360803 reap_dead start 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f28173a1640 1 -- 172.21.15.77:0/3386360803 shutdown_connections 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f28173a1640 1 -- 172.21.15.77:0/3386360803 >> 172.21.15.77:0/3386360803 conn(0x7f28100b3800 msgr2=0x7f28100b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f28173a1640 1 -- 172.21.15.77:0/3386360803 shutdown_connections 2024-03-06T07:27:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.248+0000 7f28173a1640 1 -- 172.21.15.77:0/3386360803 wait complete. 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f28173a1640 1 Processor -- start 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f28173a1640 1 -- start start 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f28173a1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27fc003b70 0x7f28100b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f28173a1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28100b51e0 0x7f2810157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f28173a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810153240 0x7f28101536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f28173a1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100cec30 con 0x7f2810153240 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f28173a1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100ce950 con 0x7f28100b51e0 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f28173a1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28100cef10 con 0x7f27fc003b70 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f281593e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810153240 0x7f28101536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f281593e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810153240 0x7f28101536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58762/0 (socket says 172.21.15.77:58762) 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.250+0000 7f281593e640 1 -- 172.21.15.77:0/2856602482 learned_addr learned my addr 172.21.15.77:0/2856602482 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f281513d640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27fc003b70 0x7f28100b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f281493c640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28100b51e0 0x7f2810157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f281593e640 1 -- 172.21.15.77:0/2856602482 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27fc003b70 msgr2=0x7f28100b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f281593e640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27fc003b70 0x7f28100b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f281593e640 1 -- 172.21.15.77:0/2856602482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28100b51e0 msgr2=0x7f2810157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f281593e640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28100b51e0 0x7f2810157080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f281593e640 1 -- 172.21.15.77:0/2856602482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28100b5660 con 0x7f2810153240 2024-03-06T07:27:06.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f281593e640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810153240 0x7f28101536d0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2808000c00 tx=0x7f2808007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f2805ffb640 1 -- 172.21.15.77:0/2856602482 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280800f6a0 con 0x7f2810153240 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f2805ffb640 1 -- 172.21.15.77:0/2856602482 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808018070 con 0x7f2810153240 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28100b5e50 con 0x7f2810153240 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f2805ffb640 1 -- 172.21.15.77:0/2856602482 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280800faa0 con 0x7f2810153240 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.251+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28100b6410 con 0x7f2810153240 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.252+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281009fe60 con 0x7f2810153240 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.253+0000 7f2805ffb640 1 -- 172.21.15.77:0/2856602482 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f280800fc40 con 0x7f2810153240 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.253+0000 7f2805ffb640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f27f406b4d0 0x7f27f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.253+0000 7f2805ffb640 1 -- 172.21.15.77:0/2856602482 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2808089ec0 con 0x7f2810153240 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.253+0000 7f281513d640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f27f406b4d0 0x7f27f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.253+0000 7f281513d640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f27f406b4d0 0x7f27f406d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f280c034000 tx=0x7f280c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.257+0000 7f2805ffb640 1 -- 172.21.15.77:0/2856602482 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28080586f0 con 0x7f2810153240 2024-03-06T07:27:06.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:06.477+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2810068ff0 con 0x7f2810153240 2024-03-06T07:27:07.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.039+0000 7f2805ffb640 1 -- 172.21.15.77:0/2856602482 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7f280805c790 con 0x7f2810153240 2024-03-06T07:27:07.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f27f406b4d0 msgr2=0x7f27f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f27f406b4d0 0x7f27f406d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f280c034000 tx=0x7f280c009030 comp rx=0 tx=0).stop 2024-03-06T07:27:07.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810153240 msgr2=0x7f28101536d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810153240 0x7f28101536d0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2808000c00 tx=0x7f2808007430 comp rx=0 tx=0).stop 2024-03-06T07:27:07.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 shutdown_connections 2024-03-06T07:27:07.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f27f406b4d0 0x7f27f406d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810153240 0x7f28101536d0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f28100b51e0 0x7f2810157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 --2- 172.21.15.77:0/2856602482 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27fc003b70 0x7f28100b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 >> 172.21.15.77:0/2856602482 conn(0x7f28100b3800 msgr2=0x7f281006d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 shutdown_connections 2024-03-06T07:27:07.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.041+0000 7f28173a1640 1 -- 172.21.15.77:0/2856602482 wait complete. 2024-03-06T07:27:07.057 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:07.117 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:07.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.220+0000 7fe88a0fb640 1 Processor -- start 2024-03-06T07:27:07.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.220+0000 7fe88a0fb640 1 -- start start 2024-03-06T07:27:07.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe88a0fb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe8841614d0 0x7fe8840d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe88a0fb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8840d1910 0x7fe8840d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe88a0fb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8841618b0 con 0x7fe8840d22e0 2024-03-06T07:27:07.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe88a0fb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840cf400 con 0x7fe8840d1910 2024-03-06T07:27:07.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe88a0fb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840d2710 con 0x7fe8841614d0 2024-03-06T07:27:07.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe882ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8840d1910 0x7fe8840d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe882ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8840d1910 0x7fe8840d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58764/0 (socket says 172.21.15.77:58764) 2024-03-06T07:27:07.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe882ffd640 1 -- 172.21.15.77:0/222250502 learned_addr learned my addr 172.21.15.77:0/222250502 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:07.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe882ffd640 1 -- 172.21.15.77:0/222250502 >> v2:172.21.15.158:3300/0 conn(0x7fe8841614d0 msgr2=0x7fe8840d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe882ffd640 1 --2- 172.21.15.77:0/222250502 >> v2:172.21.15.158:3300/0 conn(0x7fe8841614d0 0x7fe8840d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe882ffd640 1 -- 172.21.15.77:0/222250502 >> v1:172.21.15.158:6789/0 conn(0x7fe8840d22e0 legacy=0x7fe8840d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.221+0000 7fe882ffd640 1 -- 172.21.15.77:0/222250502 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840cf6a0 con 0x7fe8840d1910 2024-03-06T07:27:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe882ffd640 1 --2- 172.21.15.77:0/222250502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d1910 0x7fe8840d1da0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe8840ce550 tx=0x7fe87802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984c8109ac27d2a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe8827fc640 1 -- 172.21.15.77:0/222250502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878035020 con 0x7fe8840d1910 2024-03-06T07:27:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe8827fc640 1 -- 172.21.15.77:0/222250502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe878031870 con 0x7fe8840d1910 2024-03-06T07:27:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe8827fc640 1 -- 172.21.15.77:0/222250502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878031c40 con 0x7fe8840d1910 2024-03-06T07:27:07.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe88a0fb640 1 -- 172.21.15.77:0/222250502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d1910 msgr2=0x7fe8840d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/222250502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d1910 0x7fe8840d1da0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe8840ce550 tx=0x7fe87802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe88a0fb640 1 -- 172.21.15.77:0/222250502 shutdown_connections 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/222250502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d1910 0x7fe8840d1da0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/222250502 >> v2:172.21.15.158:3300/0 conn(0x7fe8841614d0 0x7fe8840d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.222+0000 7fe88a0fb640 1 -- 172.21.15.77:0/222250502 >> 172.21.15.77:0/222250502 conn(0x7fe8840d3d40 msgr2=0x7fe8840d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.223+0000 7fe88a0fb640 1 -- 172.21.15.77:0/222250502 shutdown_connections 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.223+0000 7fe88a0fb640 1 -- 172.21.15.77:0/222250502 wait complete. 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.224+0000 7fe88a0fb640 1 Processor -- start 2024-03-06T07:27:07.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.224+0000 7fe88a0fb640 1 -- start start 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.224+0000 7fe88a0fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8840d1910 0x7fe8840b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.224+0000 7fe88a0fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d22e0 0x7fe88415b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.224+0000 7fe88a0fb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8841614d0 0x7fe8840cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.224+0000 7fe88a0fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840b0910 con 0x7fe8840d22e0 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.224+0000 7fe88a0fb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840cfc60 con 0x7fe8841614d0 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.224+0000 7fe88a0fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840b1e40 con 0x7fe8840d1910 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d22e0 0x7fe88415b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d22e0 0x7fe88415b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58778/0 (socket says 172.21.15.77:58778) 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 -- 172.21.15.77:0/2229160406 learned_addr learned my addr 172.21.15.77:0/2229160406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:07.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe8837fe640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8840d1910 0x7fe8840b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe883fff640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8841614d0 0x7fe8840cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 -- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8840d1910 msgr2=0x7fe8840b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8840d1910 0x7fe8840b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 -- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8841614d0 msgr2=0x7fe8840cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8841614d0 0x7fe8840cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 -- 172.21.15.77:0/2229160406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840cb820 con 0x7fe8840d22e0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe8837fe640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8840d1910 0x7fe8840b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe883fff640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8841614d0 0x7fe8840cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe882ffd640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d22e0 0x7fe88415b330 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe878000c00 tx=0x7fe8780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe86bfff640 1 -- 172.21.15.77:0/2229160406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87803ad60 con 0x7fe8840d22e0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe86bfff640 1 -- 172.21.15.77:0/2229160406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8780443f0 con 0x7fe8840d22e0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8840cbac0 con 0x7fe8840d22e0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe86bfff640 1 -- 172.21.15.77:0/2229160406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8780447e0 con 0x7fe8840d22e0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.225+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8840cc080 con 0x7fe8840d22e0 2024-03-06T07:27:07.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.227+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8840b9b90 con 0x7fe8840d22e0 2024-03-06T07:27:07.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.227+0000 7fe86bfff640 1 -- 172.21.15.77:0/2229160406 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe87804b060 con 0x7fe8840d22e0 2024-03-06T07:27:07.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.227+0000 7fe86bfff640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.227+0000 7fe8837fe640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.227+0000 7fe86bfff640 1 -- 172.21.15.77:0/2229160406 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8780ac4a0 con 0x7fe8840d22e0 2024-03-06T07:27:07.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.227+0000 7fe8837fe640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe874009890 tx=0x7fe874008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.231+0000 7fe86bfff640 1 -- 172.21.15.77:0/2229160406 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe87803f070 con 0x7fe8840d22e0 2024-03-06T07:27:07.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:07.449+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe884068ff0 con 0x7fe8840d22e0 2024-03-06T07:27:08.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.043+0000 7fe86bfff640 1 -- 172.21.15.77:0/2229160406 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe87802f320 con 0x7fe8840d22e0 2024-03-06T07:27:08.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 msgr2=0x7fe87006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe874009890 tx=0x7fe874008040 comp rx=0 tx=0).stop 2024-03-06T07:27:08.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d22e0 msgr2=0x7fe88415b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d22e0 0x7fe88415b330 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe878000c00 tx=0x7fe8780046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 shutdown_connections 2024-03-06T07:27:08.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840d22e0 0x7fe88415b330 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8841614d0 0x7fe8840cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 --2- 172.21.15.77:0/2229160406 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe8840d1910 0x7fe8840b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 >> 172.21.15.77:0/2229160406 conn(0x7fe8840d3d40 msgr2=0x7fe88406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 shutdown_connections 2024-03-06T07:27:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.045+0000 7fe88a0fb640 1 -- 172.21.15.77:0/2229160406 wait complete. 2024-03-06T07:27:08.058 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:08.058 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.158+0000 7f51053e9640 1 Processor -- start 2024-03-06T07:27:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.158+0000 7f51053e9640 1 -- start start 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.159+0000 7f51053e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51000b3060 0x7f510015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.159+0000 7f51053e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f510015bf30 0x7f510015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.159+0000 7f51053e9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510015f2d0 con 0x7f510013dc10 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.159+0000 7f51053e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51000cf3f0 con 0x7f510015bf30 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.159+0000 7f51053e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510015b210 con 0x7f51000b3060 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50feffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f510013dc10 0x7f51000b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56766/0 (socket says 172.21.15.77:56766) 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50feffd640 1 -- 172.21.15.77:0/3341248560 learned_addr learned my addr 172.21.15.77:0/3341248560 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50ff7fe640 1 --2- 172.21.15.77:0/3341248560 >> v2:172.21.15.158:3300/0 conn(0x7f510015bf30 0x7f510015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50f7fff640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51000b3060 0x7f510015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19381570 0 0) 0x7f510015f2d0 con 0x7f510013dc10 2024-03-06T07:27:08.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec003520 con 0x7f510013dc10 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567235920 0 0) 0x7f50ec003520 con 0x7f510013dc10 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51000b3060 msgr2=0x7f510015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51000b3060 0x7f510015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 >> v2:172.21.15.158:3300/0 conn(0x7f510015bf30 msgr2=0x7f510015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 --2- 172.21.15.77:0/3341248560 >> v2:172.21.15.158:3300/0 conn(0x7f510015bf30 0x7f510015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510015f580 con 0x7f510013dc10 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f50f0001670 con 0x7f510013dc10 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50ff7fe640 1 --2- 172.21.15.77:0/3341248560 >> v2:172.21.15.158:3300/0 conn(0x7f510015bf30 0x7f510015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 >> v1:172.21.15.77:6789/0 conn(0x7f510013dc10 legacy=0x7f51000b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51000b3060 0x7f50ec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f50ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ec004d40 0x7f50ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51000cf3f0 con 0x7f50ec003f30 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510015b210 con 0x7f50ec004d40 2024-03-06T07:27:08.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec00dc20 con 0x7f51000b3060 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50f7fff640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f50ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50feffd640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51000b3060 0x7f50ec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.160+0000 7f50ff7fe640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ec004d40 0x7f50ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50f7fff640 1 -- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51000b3060 msgr2=0x7f50ec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50f7fff640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f51000b3060 0x7f50ec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50f7fff640 1 -- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ec004d40 msgr2=0x7f50ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50f7fff640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ec004d40 0x7f50ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50f7fff640 1 -- 172.21.15.77:0/3341248560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec003520 con 0x7f50ec003f30 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50ff7fe640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ec004d40 0x7f50ec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50f7fff640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f50ec004800 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f50e8009af0 tx=0x7f50ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb88b31c695a240 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8039070 con 0x7f50ec003f30 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.161+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e802c6b0 con 0x7f50ec003f30 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.162+0000 7f50fe7fc640 1 -- 172.21.15.77:0/3341248560 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8009690 con 0x7f50ec003f30 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.162+0000 7f51053e9640 1 -- 172.21.15.77:0/3341248560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 msgr2=0x7f50ec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.162+0000 7f51053e9640 1 --2- 172.21.15.77:0/3341248560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f50ec004800 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f50e8009af0 tx=0x7f50ec005830 comp rx=0 tx=0).stop 2024-03-06T07:27:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.162+0000 7f50ff7fe640 1 -- 172.21.15.77:0/3341248560 reap_dead start 2024-03-06T07:27:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.162+0000 7f51053e9640 1 -- 172.21.15.77:0/3341248560 shutdown_connections 2024-03-06T07:27:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.162+0000 7f51053e9640 1 -- 172.21.15.77:0/3341248560 >> 172.21.15.77:0/3341248560 conn(0x7f51000cf8a0 msgr2=0x7f51000cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.162+0000 7f51053e9640 1 -- 172.21.15.77:0/3341248560 shutdown_connections 2024-03-06T07:27:08.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.162+0000 7f51053e9640 1 -- 172.21.15.77:0/3341248560 wait complete. 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.163+0000 7f51053e9640 1 Processor -- start 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.163+0000 7f51053e9640 1 -- start start 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f51053e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f5100143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f51053e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f510013d0c0 0x7f510014fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f51053e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5100152f50 0x7f5100150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f51053e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510013dff0 con 0x7f50ec003f30 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f51053e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510015b730 con 0x7f5100152f50 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f51053e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510015e850 con 0x7f510013d0c0 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50feffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f5100143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50feffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f5100143ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58800/0 (socket says 172.21.15.77:58800) 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50feffd640 1 -- 172.21.15.77:0/3056413736 learned_addr learned my addr 172.21.15.77:0/3056413736 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50f7fff640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f510013d0c0 0x7f510014fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50ff7fe640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5100152f50 0x7f5100150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50feffd640 1 -- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f510013d0c0 msgr2=0x7f510014fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50feffd640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f510013d0c0 0x7f510014fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50feffd640 1 -- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5100152f50 msgr2=0x7f5100150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50feffd640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5100152f50 0x7f5100150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50feffd640 1 -- 172.21.15.77:0/3056413736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510015c590 con 0x7f50ec003f30 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.164+0000 7f50ff7fe640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5100152f50 0x7f5100150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.165+0000 7f50feffd640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f5100143ec0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f50f00073a0 tx=0x7f50f0007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.165+0000 7f50f77fe640 1 -- 172.21.15.77:0/3056413736 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f001a070 con 0x7f50ec003f30 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.165+0000 7f50f77fe640 1 -- 172.21.15.77:0/3056413736 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f000f990 con 0x7f50ec003f30 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.165+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510015c870 con 0x7f50ec003f30 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.165+0000 7f50f77fe640 1 -- 172.21.15.77:0/3056413736 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f000fd40 con 0x7f50ec003f30 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.165+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51001501f0 con 0x7f50ec003f30 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.166+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510004f330 con 0x7f50ec003f30 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.167+0000 7f50f77fe640 1 -- 172.21.15.77:0/3056413736 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50f001d070 con 0x7f50ec003f30 2024-03-06T07:27:08.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.167+0000 7f50f77fe640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f50e006b4d0 0x7f50e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.167+0000 7f50f77fe640 1 -- 172.21.15.77:0/3056413736 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50f0089d70 con 0x7f50ec003f30 2024-03-06T07:27:08.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.167+0000 7f50f7fff640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f50e006b4d0 0x7f50e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.167+0000 7f50f7fff640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f50e006b4d0 0x7f50e006d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f50e8036000 tx=0x7f50e8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.171+0000 7f50f77fe640 1 -- 172.21.15.77:0/3056413736 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50f0058bf0 con 0x7f50ec003f30 2024-03-06T07:27:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.488+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f510006eb00 con 0x7f50ec003f30 2024-03-06T07:27:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.488+0000 7f50f77fe640 1 -- 172.21.15.77:0/3056413736 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f50f00584f0 con 0x7f50ec003f30 2024-03-06T07:27:08.490 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:08.490 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/582596329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":582596329},{"type":"v1","addr":"172.21.15.158:6834","nonce":582596329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3505002958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3505002958},{"type":"v1","addr":"172.21.15.158:6835","nonce":3505002958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1169947364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1169947364},{"type":"v1","addr":"172.21.15.77:6835","nonce":1169947364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:07.450568+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7827":{"gid":7827,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3247585937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3247585937},{"type":"v1","addr":"172.21.15.77:6837","nonce":3247585937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7827,"qdb_cluster":[7827]},"id":8}]} 2024-03-06T07:27:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 66 2024-03-06T07:27:08.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f50e006b4d0 msgr2=0x7f50e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f50e006b4d0 0x7f50e006d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f50e8036000 tx=0x7f50e8030000 comp rx=0 tx=0).stop 2024-03-06T07:27:08.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 msgr2=0x7f5100143ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f5100143ec0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f50f00073a0 tx=0x7f50f0007860 comp rx=0 tx=0).stop 2024-03-06T07:27:08.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 shutdown_connections 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5100152f50 0x7f5100150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f510013d0c0 0x7f510014fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f50e006b4d0 0x7f50e006d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 --2- 172.21.15.77:0/3056413736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ec003f30 0x7f5100143ec0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 >> 172.21.15.77:0/3056413736 conn(0x7f51000cf8a0 msgr2=0x7f51000b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 shutdown_connections 2024-03-06T07:27:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.491+0000 7f51053e9640 1 -- 172.21.15.77:0/3056413736 wait complete. 2024-03-06T07:27:08.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a3165640 1 Processor -- start 2024-03-06T07:27:08.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a3165640 1 -- start start 2024-03-06T07:27:08.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a3165640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f449c15bdd0 0x7f449c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a3165640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f449c0d16e0 0x7f449c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a3165640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0cebe0 con 0x7f449c15fa10 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a3165640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449c0ce980 con 0x7f449c0d16e0 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a3165640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449c0ceec0 con 0x7f449c15bdd0 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a1702640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f449c0d16e0 0x7f449c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.603+0000 7f44a1702640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f449c0d16e0 0x7f449c0d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36310/0 (socket says 172.21.15.77:36310) 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.604+0000 7f44a1702640 1 -- 172.21.15.77:0/283053270 learned_addr learned my addr 172.21.15.77:0/283053270 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.604+0000 7f44a1702640 1 -- 172.21.15.77:0/283053270 >> v2:172.21.15.158:3300/0 conn(0x7f449c15bdd0 msgr2=0x7f449c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.604+0000 7f44a1702640 1 --2- 172.21.15.77:0/283053270 >> v2:172.21.15.158:3300/0 conn(0x7f449c15bdd0 0x7f449c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.604+0000 7f44a1702640 1 -- 172.21.15.77:0/283053270 >> v1:172.21.15.158:6789/0 conn(0x7f449c15fa10 legacy=0x7f449c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.604+0000 7f44a1702640 1 -- 172.21.15.77:0/283053270 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c0cf1a0 con 0x7f449c0d16e0 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.604+0000 7f44a1702640 1 --2- 172.21.15.77:0/283053270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c0d1af0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f449c0c5520 tx=0x7f449402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca15cfff93c9314 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f449b7fe640 1 -- 172.21.15.77:0/283053270 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449402c9e0 con 0x7f449c0d16e0 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f449b7fe640 1 -- 172.21.15.77:0/283053270 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449402cb80 con 0x7f449c0d16e0 2024-03-06T07:27:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f449b7fe640 1 -- 172.21.15.77:0/283053270 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44940343f0 con 0x7f449c0d16e0 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f44a3165640 1 -- 172.21.15.77:0/283053270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 msgr2=0x7f449c0d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f44a3165640 1 --2- 172.21.15.77:0/283053270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c0d1af0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f449c0c5520 tx=0x7f449402be40 comp rx=0 tx=0).stop 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f44a3165640 1 -- 172.21.15.77:0/283053270 shutdown_connections 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f44a3165640 1 --2- 172.21.15.77:0/283053270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c0d1af0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f44a3165640 1 --2- 172.21.15.77:0/283053270 >> v2:172.21.15.158:3300/0 conn(0x7f449c15bdd0 0x7f449c0c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f44a3165640 1 -- 172.21.15.77:0/283053270 >> 172.21.15.77:0/283053270 conn(0x7f449c09a9f0 msgr2=0x7f449c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f44a3165640 1 -- 172.21.15.77:0/283053270 shutdown_connections 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.605+0000 7f44a3165640 1 -- 172.21.15.77:0/283053270 wait complete. 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.607+0000 7f44a3165640 1 Processor -- start 2024-03-06T07:27:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.607+0000 7f44a3165640 1 -- start start 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.607+0000 7f44a3165640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c1f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.607+0000 7f44a3165640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f449c15bdd0 0x7f449c0c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.607+0000 7f44a3165640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f449c15fa10 0x7f449c0cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.607+0000 7f44a3165640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0cc5d0 con 0x7f449c0d16e0 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.607+0000 7f44a3165640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0c5b50 con 0x7f449c15bdd0 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.607+0000 7f44a3165640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449c15c1b0 con 0x7f449c15fa10 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c1f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c1f2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36312/0 (socket says 172.21.15.77:36312) 2024-03-06T07:27:08.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 -- 172.21.15.77:0/796090360 learned_addr learned my addr 172.21.15.77:0/796090360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f449bfff640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f449c15bdd0 0x7f449c0c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a1702640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f449c15fa10 0x7f449c0cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 -- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f449c15fa10 msgr2=0x7f449c0cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f449c15fa10 0x7f449c0cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 -- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f449c15bdd0 msgr2=0x7f449c0c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f449c15bdd0 0x7f449c0c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 -- 172.21.15.77:0/796090360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c1f2b90 con 0x7f449c0d16e0 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a1702640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f449c15fa10 0x7f449c0cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a0f01640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c1f2290 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f449000b410 tx=0x7f449000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44997fa640 1 -- 172.21.15.77:0/796090360 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490015020 con 0x7f449c0d16e0 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44997fa640 1 -- 172.21.15.77:0/796090360 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4490011850 con 0x7f449c0d16e0 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449c1f2e70 con 0x7f449c0d16e0 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44997fa640 1 -- 172.21.15.77:0/796090360 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490011c40 con 0x7f449c0d16e0 2024-03-06T07:27:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.608+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c1f3710 con 0x7f449c0d16e0 2024-03-06T07:27:08.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.609+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c1f3910 con 0x7f449c0d16e0 2024-03-06T07:27:08.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.610+0000 7f44997fa640 1 -- 172.21.15.77:0/796090360 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4490018020 con 0x7f449c0d16e0 2024-03-06T07:27:08.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.610+0000 7f44997fa640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f448006b4d0 0x7f448006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.610+0000 7f44997fa640 1 -- 172.21.15.77:0/796090360 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f449008a230 con 0x7f449c0d16e0 2024-03-06T07:27:08.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.610+0000 7f449bfff640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f448006b4d0 0x7f448006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.611+0000 7f449bfff640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f448006b4d0 0x7f448006d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4488006fd0 tx=0x7f4488008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.615+0000 7f44997fa640 1 -- 172.21.15.77:0/796090360 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f449001f020 con 0x7f449c0d16e0 2024-03-06T07:27:08.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.834+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f449c0cbed0 con 0x7f449c0d16e0 2024-03-06T07:27:08.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.836+0000 7f44997fa640 1 -- 172.21.15.77:0/796090360 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f44900172c0 con 0x7f449c0d16e0 2024-03-06T07:27:08.838 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:08.838 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:06.505066+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:08.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f448006b4d0 msgr2=0x7f448006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f448006b4d0 0x7f448006d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4488006fd0 tx=0x7f4488008040 comp rx=0 tx=0).stop 2024-03-06T07:27:08.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 msgr2=0x7f449c1f2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c1f2290 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f449000b410 tx=0x7f449000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 shutdown_connections 2024-03-06T07:27:08.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f449c15fa10 0x7f449c0cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f449c15bdd0 0x7f449c0c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f448006b4d0 0x7f448006d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 --2- 172.21.15.77:0/796090360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f449c0d16e0 0x7f449c1f2290 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 >> 172.21.15.77:0/796090360 conn(0x7f449c09a9f0 msgr2=0x7f449c06fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 shutdown_connections 2024-03-06T07:27:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.838+0000 7f44a3165640 1 -- 172.21.15.77:0/796090360 wait complete. 2024-03-06T07:27:08.851 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:08.851 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.950+0000 7fb0130f4640 1 Processor -- start 2024-03-06T07:27:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.950+0000 7fb0130f4640 1 -- start start 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.951+0000 7fb0130f4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb00c0d22b0 0x7fb00c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.951+0000 7fb0130f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb00c0b41f0 0x7fb00c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.951+0000 7fb0130f4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c0b3430 con 0x7fb00c15dfe0 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.951+0000 7fb0130f4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00c0d2c50 con 0x7fb00c0b41f0 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.951+0000 7fb0130f4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00c0b31e0 con 0x7fb00c0d22b0 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.951+0000 7fb011691640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb00c0b41f0 0x7fb00c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.952+0000 7fb011691640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb00c0b41f0 0x7fb00c0b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36328/0 (socket says 172.21.15.77:36328) 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.952+0000 7fb011691640 1 -- 172.21.15.77:0/4282683449 learned_addr learned my addr 172.21.15.77:0/4282683449 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.952+0000 7fb00bfff640 1 --2- 172.21.15.77:0/4282683449 >> v2:172.21.15.158:3300/0 conn(0x7fb00c0d22b0 0x7fb00c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.952+0000 7fb011691640 1 -- 172.21.15.77:0/4282683449 >> v2:172.21.15.158:3300/0 conn(0x7fb00c0d22b0 msgr2=0x7fb00c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.952+0000 7fb011691640 1 --2- 172.21.15.77:0/4282683449 >> v2:172.21.15.158:3300/0 conn(0x7fb00c0d22b0 0x7fb00c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.952+0000 7fb011691640 1 -- 172.21.15.77:0/4282683449 >> v1:172.21.15.158:6789/0 conn(0x7fb00c15dfe0 legacy=0x7fb00c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.952+0000 7fb011691640 1 -- 172.21.15.77:0/4282683449 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c0b3710 con 0x7fb00c0b41f0 2024-03-06T07:27:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.952+0000 7fb00bfff640 1 --2- 172.21.15.77:0/4282683449 >> v2:172.21.15.158:3300/0 conn(0x7fb00c0d22b0 0x7fb00c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:08.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.953+0000 7fb011691640 1 --2- 172.21.15.77:0/4282683449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0b41f0 0x7fb00c0b4680 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7faffc009870 tx=0x7faffc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d857292d234b76d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.953+0000 7fb00b7fe640 1 -- 172.21.15.77:0/4282683449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc035020 con 0x7fb00c0b41f0 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.953+0000 7fb00b7fe640 1 -- 172.21.15.77:0/4282683449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc031870 con 0x7fb00c0b41f0 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.953+0000 7fb00b7fe640 1 -- 172.21.15.77:0/4282683449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc031c40 con 0x7fb00c0b41f0 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.953+0000 7fb0130f4640 1 -- 172.21.15.77:0/4282683449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0b41f0 msgr2=0x7fb00c0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.953+0000 7fb0130f4640 1 --2- 172.21.15.77:0/4282683449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0b41f0 0x7fb00c0b4680 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7faffc009870 tx=0x7faffc02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.953+0000 7fb0130f4640 1 -- 172.21.15.77:0/4282683449 shutdown_connections 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.953+0000 7fb0130f4640 1 --2- 172.21.15.77:0/4282683449 >> v2:172.21.15.158:3300/0 conn(0x7fb00c0d22b0 0x7fb00c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.954+0000 7fb0130f4640 1 --2- 172.21.15.77:0/4282683449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0b41f0 0x7fb00c0b4680 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.954+0000 7fb0130f4640 1 -- 172.21.15.77:0/4282683449 >> 172.21.15.77:0/4282683449 conn(0x7fb00c0d3cd0 msgr2=0x7fb00c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.954+0000 7fb0130f4640 1 -- 172.21.15.77:0/4282683449 shutdown_connections 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.954+0000 7fb0130f4640 1 -- 172.21.15.77:0/4282683449 wait complete. 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.955+0000 7fb0130f4640 1 Processor -- start 2024-03-06T07:27:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.955+0000 7fb0130f4640 1 -- start start 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.955+0000 7fb0130f4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb00c0b41f0 0x7fb00c161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.955+0000 7fb0130f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0d22b0 0x7fb00c0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0130f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb00c15dfe0 0x7fb00c0b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0130f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0cfa00 con 0x7fb00c0d22b0 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0130f4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0b3d60 con 0x7fb00c0b41f0 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0130f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0d1c70 con 0x7fb00c15dfe0 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0d22b0 0x7fb00c0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0d22b0 0x7fb00c0cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36340/0 (socket says 172.21.15.77:36340) 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 -- 172.21.15.77:0/2091887665 learned_addr learned my addr 172.21.15.77:0/2091887665 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb010e90640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb00c0b41f0 0x7fb00c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb011691640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb00c15dfe0 0x7fb00c0b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 -- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb00c15dfe0 msgr2=0x7fb00c0b9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb00c15dfe0 0x7fb00c0b9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 -- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb00c0b41f0 msgr2=0x7fb00c161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb00c0b41f0 0x7fb00c161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 -- 172.21.15.77:0/2091887665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c0cc250 con 0x7fb00c0d22b0 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb011691640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb00c15dfe0 0x7fb00c0b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb00bfff640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0d22b0 0x7fb00c0cd2f0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c7f0 tx=0x7fb00000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0097fa640 1 -- 172.21.15.77:0/2091887665 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000015070 con 0x7fb00c0d22b0 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0097fa640 1 -- 172.21.15.77:0/2091887665 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb000007a30 con 0x7fb00c0d22b0 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0097fa640 1 -- 172.21.15.77:0/2091887665 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000007e00 con 0x7fb00c0d22b0 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c0cc530 con 0x7fb00c0d22b0 2024-03-06T07:27:08.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.956+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c0ccaf0 con 0x7fb00c0d22b0 2024-03-06T07:27:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.957+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c04f330 con 0x7fb00c0d22b0 2024-03-06T07:27:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.958+0000 7fb0097fa640 1 -- 172.21.15.77:0/2091887665 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb000018070 con 0x7fb00c0d22b0 2024-03-06T07:27:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.958+0000 7fb0097fa640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faff806b4d0 0x7faff806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.958+0000 7fb010e90640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faff806b4d0 0x7faff806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.958+0000 7fb0097fa640 1 -- 172.21.15.77:0/2091887665 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb00008a110 con 0x7fb00c0d22b0 2024-03-06T07:27:08.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.959+0000 7fb010e90640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faff806b4d0 0x7faff806d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb00400b450 tx=0x7fb00400d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:08.963+0000 7fb0097fa640 1 -- 172.21.15.77:0/2091887665 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb000058930 con 0x7fb00c0d22b0 2024-03-06T07:27:09.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.138+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb00c15d500 con 0x7faff806b4d0 2024-03-06T07:27:09.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.144+0000 7fb0097fa640 1 -- 172.21.15.77:0/2091887665 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252480 (secure 0 0 0) 0x7fb00c15d500 con 0x7faff806b4d0 2024-03-06T07:27:09.149 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:09.150 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:27:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faff806b4d0 msgr2=0x7faff806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faff806b4d0 0x7faff806d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb00400b450 tx=0x7fb00400d040 comp rx=0 tx=0).stop 2024-03-06T07:27:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0d22b0 msgr2=0x7fb00c0cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0d22b0 0x7fb00c0cd2f0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c7f0 tx=0x7fb00000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 shutdown_connections 2024-03-06T07:27:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faff806b4d0 0x7faff806d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb00c0d22b0 0x7fb00c0cd2f0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb00c15dfe0 0x7fb00c0b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 --2- 172.21.15.77:0/2091887665 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb00c0b41f0 0x7fb00c161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 >> 172.21.15.77:0/2091887665 conn(0x7fb00c0d3cd0 msgr2=0x7fb00c0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 shutdown_connections 2024-03-06T07:27:09.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.147+0000 7fb0130f4640 1 -- 172.21.15.77:0/2091887665 wait complete. 2024-03-06T07:27:09.161 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-03-06T07:27:07.440308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302676,"kb_used_data":11624,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754672044,"statfs":{"total":773094113280,"available":772784173056,"internally_reserved":0,"allocated":11902976,"data_stored":5771836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000877"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508739+0000","last_change":"2024-03-06T07:27:05.517498+0000","last_active":"2024-03-06T07:27:06.508739+0000","last_peered":"2024-03-06T07:27:06.508739+0000","last_clean":"2024-03-06T07:27:06.508739+0000","last_became_active":"2024-03-06T07:27:05.516214+0000","last_became_peered":"2024-03-06T07:27:05.516214+0000","last_unstale":"2024-03-06T07:27:06.508739+0000","last_undegraded":"2024-03-06T07:27:06.508739+0000","last_fullsized":"2024-03-06T07:27:06.508739+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:10.145209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509087+0000","last_change":"2024-03-06T07:27:05.515799+0000","last_active":"2024-03-06T07:27:06.509087+0000","last_peered":"2024-03-06T07:27:06.509087+0000","last_clean":"2024-03-06T07:27:06.509087+0000","last_became_active":"2024-03-06T07:27:05.514927+0000","last_became_peered":"2024-03-06T07:27:05.514927+0000","last_unstale":"2024-03-06T07:27:06.509087+0000","last_undegraded":"2024-03-06T07:27:06.509087+0000","last_fullsized":"2024-03-06T07:27:06.509087+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:09.335581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708276+0000","last_change":"2024-03-06T07:27:05.512196+0000","last_active":"2024-03-06T07:27:06.708276+0000","last_peered":"2024-03-06T07:27:06.708276+0000","last_clean":"2024-03-06T07:27:06.708276+0000","last_became_active":"2024-03-06T07:27:05.510165+0000","last_became_peered":"2024-03-06T07:27:05.510165+0000","last_unstale":"2024-03-06T07:27:06.708276+0000","last_undegraded":"2024-03-06T07:27:06.708276+0000","last_fullsized":"2024-03-06T07:27:06.708276+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:45.347711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508812+0000","last_change":"2024-03-06T07:27:05.509649+0000","last_active":"2024-03-06T07:27:06.508812+0000","last_peered":"2024-03-06T07:27:06.508812+0000","last_clean":"2024-03-06T07:27:06.508812+0000","last_became_active":"2024-03-06T07:27:05.509372+0000","last_became_peered":"2024-03-06T07:27:05.509372+0000","last_unstale":"2024-03-06T07:27:06.508812+0000","last_undegraded":"2024-03-06T07:27:06.508812+0000","last_fullsized":"2024-03-06T07:27:06.508812+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:21.408950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708725+0000","last_change":"2024-03-06T07:27:05.518091+0000","last_active":"2024-03-06T07:27:06.708725+0000","last_peered":"2024-03-06T07:27:06.708725+0000","last_clean":"2024-03-06T07:27:06.708725+0000","last_became_active":"2024-03-06T07:27:05.517801+0000","last_became_peered":"2024-03-06T07:27:05.517801+0000","last_unstale":"2024-03-06T07:27:06.708725+0000","last_undegraded":"2024-03-06T07:27:06.708725+0000","last_fullsized":"2024-03-06T07:27:06.708725+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:46.744879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708005+0000","last_change":"2024-03-06T07:27:05.521549+0000","last_active":"2024-03-06T07:27:06.708005+0000","last_peered":"2024-03-06T07:27:06.708005+0000","last_clean":"2024-03-06T07:27:06.708005+0000","last_became_active":"2024-03-06T07:27:05.521273+0000","last_became_peered":"2024-03-06T07:27:05.521273+0000","last_unstale":"2024-03-06T07:27:06.708005+0000","last_undegraded":"2024-03-06T07:27:06.708005+0000","last_fullsized":"2024-03-06T07:27:06.708005+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.687400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515081+0000","last_change":"2024-03-06T07:27:05.519605+0000","last_active":"2024-03-06T07:27:06.515081+0000","last_peered":"2024-03-06T07:27:06.515081+0000","last_clean":"2024-03-06T07:27:06.515081+0000","last_became_active":"2024-03-06T07:27:05.518706+0000","last_became_peered":"2024-03-06T07:27:05.518706+0000","last_unstale":"2024-03-06T07:27:06.515081+0000","last_undegraded":"2024-03-06T07:27:06.515081+0000","last_fullsized":"2024-03-06T07:27:06.515081+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:37.699345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708419+0000","last_change":"2024-03-06T07:27:05.512689+0000","last_active":"2024-03-06T07:27:06.708419+0000","last_peered":"2024-03-06T07:27:06.708419+0000","last_clean":"2024-03-06T07:27:06.708419+0000","last_became_active":"2024-03-06T07:27:05.511991+0000","last_became_peered":"2024-03-06T07:27:05.511991+0000","last_unstale":"2024-03-06T07:27:06.708419+0000","last_undegraded":"2024-03-06T07:27:06.708419+0000","last_fullsized":"2024-03-06T07:27:06.708419+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:47.743962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508899+0000","last_change":"2024-03-06T07:27:05.527111+0000","last_active":"2024-03-06T07:27:06.508899+0000","last_peered":"2024-03-06T07:27:06.508899+0000","last_clean":"2024-03-06T07:27:06.508899+0000","last_became_active":"2024-03-06T07:27:05.526568+0000","last_became_peered":"2024-03-06T07:27:05.526568+0000","last_unstale":"2024-03-06T07:27:06.508899+0000","last_undegraded":"2024-03-06T07:27:06.508899+0000","last_fullsized":"2024-03-06T07:27:06.508899+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:40.382167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508654+0000","last_change":"2024-03-06T07:27:05.514806+0000","last_active":"2024-03-06T07:27:06.508654+0000","last_peered":"2024-03-06T07:27:06.508654+0000","last_clean":"2024-03-06T07:27:06.508654+0000","last_became_active":"2024-03-06T07:27:05.514487+0000","last_became_peered":"2024-03-06T07:27:05.514487+0000","last_unstale":"2024-03-06T07:27:06.508654+0000","last_undegraded":"2024-03-06T07:27:06.508654+0000","last_fullsized":"2024-03-06T07:27:06.508654+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:14.000752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509167+0000","last_change":"2024-03-06T07:27:05.518246+0000","last_active":"2024-03-06T07:27:06.509167+0000","last_peered":"2024-03-06T07:27:06.509167+0000","last_clean":"2024-03-06T07:27:06.509167+0000","last_became_active":"2024-03-06T07:27:05.517970+0000","last_became_peered":"2024-03-06T07:27:05.517970+0000","last_unstale":"2024-03-06T07:27:06.509167+0000","last_undegraded":"2024-03-06T07:27:06.509167+0000","last_fullsized":"2024-03-06T07:27:06.509167+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:42.395357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508665+0000","last_change":"2024-03-06T07:27:05.512849+0000","last_active":"2024-03-06T07:27:06.508665+0000","last_peered":"2024-03-06T07:27:06.508665+0000","last_clean":"2024-03-06T07:27:06.508665+0000","last_became_active":"2024-03-06T07:27:05.512560+0000","last_became_peered":"2024-03-06T07:27:05.512560+0000","last_unstale":"2024-03-06T07:27:06.508665+0000","last_undegraded":"2024-03-06T07:27:06.508665+0000","last_fullsized":"2024-03-06T07:27:06.508665+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:51.149239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708417+0000","last_change":"2024-03-06T07:27:05.518981+0000","last_active":"2024-03-06T07:27:06.708417+0000","last_peered":"2024-03-06T07:27:06.708417+0000","last_clean":"2024-03-06T07:27:06.708417+0000","last_became_active":"2024-03-06T07:27:05.518688+0000","last_became_peered":"2024-03-06T07:27:05.518688+0000","last_unstale":"2024-03-06T07:27:06.708417+0000","last_undegraded":"2024-03-06T07:27:06.708417+0000","last_fullsized":"2024-03-06T07:27:06.708417+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:07.982069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708108+0000","last_change":"2024-03-06T07:27:05.512368+0000","last_active":"2024-03-06T07:27:06.708108+0000","last_peered":"2024-03-06T07:27:06.708108+0000","last_clean":"2024-03-06T07:27:06.708108+0000","last_became_active":"2024-03-06T07:27:05.512045+0000","last_became_peered":"2024-03-06T07:27:05.512045+0000","last_unstale":"2024-03-06T07:27:06.708108+0000","last_undegraded":"2024-03-06T07:27:06.708108+0000","last_fullsized":"2024-03-06T07:27:06.708108+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:34.722915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708625+0000","last_change":"2024-03-06T07:27:05.507962+0000","last_active":"2024-03-06T07:27:06.708625+0000","last_peered":"2024-03-06T07:27:06.708625+0000","last_clean":"2024-03-06T07:27:06.708625+0000","last_became_active":"2024-03-06T07:27:05.507665+0000","last_became_peered":"2024-03-06T07:27:05.507665+0000","last_unstale":"2024-03-06T07:27:06.708625+0000","last_undegraded":"2024-03-06T07:27:06.708625+0000","last_fullsized":"2024-03-06T07:27:06.708625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:58.357775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508307+0000","last_change":"2024-03-06T07:27:05.515381+0000","last_active":"2024-03-06T07:27:06.508307+0000","last_peered":"2024-03-06T07:27:06.508307+0000","last_clean":"2024-03-06T07:27:06.508307+0000","last_became_active":"2024-03-06T07:27:05.515062+0000","last_became_peered":"2024-03-06T07:27:05.515062+0000","last_unstale":"2024-03-06T07:27:06.508307+0000","last_undegraded":"2024-03-06T07:27:06.508307+0000","last_fullsized":"2024-03-06T07:27:06.508307+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:58.299008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509805+0000","last_change":"2024-03-06T07:27:05.520807+0000","last_active":"2024-03-06T07:27:06.509805+0000","last_peered":"2024-03-06T07:27:06.509805+0000","last_clean":"2024-03-06T07:27:06.509805+0000","last_became_active":"2024-03-06T07:27:05.520537+0000","last_became_peered":"2024-03-06T07:27:05.520537+0000","last_unstale":"2024-03-06T07:27:06.509805+0000","last_undegraded":"2024-03-06T07:27:06.509805+0000","last_fullsized":"2024-03-06T07:27:06.509805+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:09.236618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510002+0000","last_change":"2024-03-06T07:27:05.510433+0000","last_active":"2024-03-06T07:27:06.510002+0000","last_peered":"2024-03-06T07:27:06.510002+0000","last_clean":"2024-03-06T07:27:06.510002+0000","last_became_active":"2024-03-06T07:27:05.510117+0000","last_became_peered":"2024-03-06T07:27:05.510117+0000","last_unstale":"2024-03-06T07:27:06.510002+0000","last_undegraded":"2024-03-06T07:27:06.510002+0000","last_fullsized":"2024-03-06T07:27:06.510002+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:09.594835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708476+0000","last_change":"2024-03-06T07:27:05.512658+0000","last_active":"2024-03-06T07:27:06.708476+0000","last_peered":"2024-03-06T07:27:06.708476+0000","last_clean":"2024-03-06T07:27:06.708476+0000","last_became_active":"2024-03-06T07:27:05.511478+0000","last_became_peered":"2024-03-06T07:27:05.511478+0000","last_unstale":"2024-03-06T07:27:06.708476+0000","last_undegraded":"2024-03-06T07:27:06.708476+0000","last_fullsized":"2024-03-06T07:27:06.708476+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:03.124647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509646+0000","last_change":"2024-03-06T07:27:05.519916+0000","last_active":"2024-03-06T07:27:06.509646+0000","last_peered":"2024-03-06T07:27:06.509646+0000","last_clean":"2024-03-06T07:27:06.509646+0000","last_became_active":"2024-03-06T07:27:05.517121+0000","last_became_peered":"2024-03-06T07:27:05.517121+0000","last_unstale":"2024-03-06T07:27:06.509646+0000","last_undegraded":"2024-03-06T07:27:06.509646+0000","last_fullsized":"2024-03-06T07:27:06.509646+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:14.941244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510058+0000","last_change":"2024-03-06T07:27:05.517655+0000","last_active":"2024-03-06T07:27:06.510058+0000","last_peered":"2024-03-06T07:27:06.510058+0000","last_clean":"2024-03-06T07:27:06.510058+0000","last_became_active":"2024-03-06T07:27:05.517390+0000","last_became_peered":"2024-03-06T07:27:05.517390+0000","last_unstale":"2024-03-06T07:27:06.510058+0000","last_undegraded":"2024-03-06T07:27:06.510058+0000","last_fullsized":"2024-03-06T07:27:06.510058+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:10.754527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708442+0000","last_change":"2024-03-06T07:27:05.520445+0000","last_active":"2024-03-06T07:27:06.708442+0000","last_peered":"2024-03-06T07:27:06.708442+0000","last_clean":"2024-03-06T07:27:06.708442+0000","last_became_active":"2024-03-06T07:27:05.519628+0000","last_became_peered":"2024-03-06T07:27:05.519628+0000","last_unstale":"2024-03-06T07:27:06.708442+0000","last_undegraded":"2024-03-06T07:27:06.708442+0000","last_fullsized":"2024-03-06T07:27:06.708442+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:11.264585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509047+0000","last_change":"2024-03-06T07:27:05.514630+0000","last_active":"2024-03-06T07:27:06.509047+0000","last_peered":"2024-03-06T07:27:06.509047+0000","last_clean":"2024-03-06T07:27:06.509047+0000","last_became_active":"2024-03-06T07:27:05.514326+0000","last_became_peered":"2024-03-06T07:27:05.514326+0000","last_unstale":"2024-03-06T07:27:06.509047+0000","last_undegraded":"2024-03-06T07:27:06.509047+0000","last_fullsized":"2024-03-06T07:27:06.509047+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:12.163357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708088+0000","last_change":"2024-03-06T07:27:05.520274+0000","last_active":"2024-03-06T07:27:06.708088+0000","last_peered":"2024-03-06T07:27:06.708088+0000","last_clean":"2024-03-06T07:27:06.708088+0000","last_became_active":"2024-03-06T07:27:05.519795+0000","last_became_peered":"2024-03-06T07:27:05.519795+0000","last_unstale":"2024-03-06T07:27:06.708088+0000","last_undegraded":"2024-03-06T07:27:06.708088+0000","last_fullsized":"2024-03-06T07:27:06.708088+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:50.417505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510111+0000","last_change":"2024-03-06T07:27:05.521312+0000","last_active":"2024-03-06T07:27:06.510111+0000","last_peered":"2024-03-06T07:27:06.510111+0000","last_clean":"2024-03-06T07:27:06.510111+0000","last_became_active":"2024-03-06T07:27:05.517145+0000","last_became_peered":"2024-03-06T07:27:05.517145+0000","last_unstale":"2024-03-06T07:27:06.510111+0000","last_undegraded":"2024-03-06T07:27:06.510111+0000","last_fullsized":"2024-03-06T07:27:06.510111+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:15.437041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508794+0000","last_change":"2024-03-06T07:27:05.525463+0000","last_active":"2024-03-06T07:27:06.508794+0000","last_peered":"2024-03-06T07:27:06.508794+0000","last_clean":"2024-03-06T07:27:06.508794+0000","last_became_active":"2024-03-06T07:27:05.522514+0000","last_became_peered":"2024-03-06T07:27:05.522514+0000","last_unstale":"2024-03-06T07:27:06.508794+0000","last_undegraded":"2024-03-06T07:27:06.508794+0000","last_fullsized":"2024-03-06T07:27:06.508794+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:41.987030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708263+0000","last_change":"2024-03-06T07:27:05.516401+0000","last_active":"2024-03-06T07:27:06.708263+0000","last_peered":"2024-03-06T07:27:06.708263+0000","last_clean":"2024-03-06T07:27:06.708263+0000","last_became_active":"2024-03-06T07:27:05.516119+0000","last_became_peered":"2024-03-06T07:27:05.516119+0000","last_unstale":"2024-03-06T07:27:06.708263+0000","last_undegraded":"2024-03-06T07:27:06.708263+0000","last_fullsized":"2024-03-06T07:27:06.708263+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:04.745211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708528+0000","last_change":"2024-03-06T07:27:05.518745+0000","last_active":"2024-03-06T07:27:06.708528+0000","last_peered":"2024-03-06T07:27:06.708528+0000","last_clean":"2024-03-06T07:27:06.708528+0000","last_became_active":"2024-03-06T07:27:05.517763+0000","last_became_peered":"2024-03-06T07:27:05.517763+0000","last_unstale":"2024-03-06T07:27:06.708528+0000","last_undegraded":"2024-03-06T07:27:06.708528+0000","last_fullsized":"2024-03-06T07:27:06.708528+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:25.199337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509377+0000","last_change":"2024-03-06T07:27:05.517165+0000","last_active":"2024-03-06T07:27:06.509377+0000","last_peered":"2024-03-06T07:27:06.509377+0000","last_clean":"2024-03-06T07:27:06.509377+0000","last_became_active":"2024-03-06T07:27:05.516858+0000","last_became_peered":"2024-03-06T07:27:05.516858+0000","last_unstale":"2024-03-06T07:27:06.509377+0000","last_undegraded":"2024-03-06T07:27:06.509377+0000","last_fullsized":"2024-03-06T07:27:06.509377+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:15.364800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515097+0000","last_change":"2024-03-06T07:27:05.518068+0000","last_active":"2024-03-06T07:27:06.515097+0000","last_peered":"2024-03-06T07:27:06.515097+0000","last_clean":"2024-03-06T07:27:06.515097+0000","last_became_active":"2024-03-06T07:27:05.517784+0000","last_became_peered":"2024-03-06T07:27:05.517784+0000","last_unstale":"2024-03-06T07:27:06.515097+0000","last_undegraded":"2024-03-06T07:27:06.515097+0000","last_fullsized":"2024-03-06T07:27:06.515097+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:45.977198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508442+0000","last_change":"2024-03-06T07:27:05.513283+0000","last_active":"2024-03-06T07:27:06.508442+0000","last_peered":"2024-03-06T07:27:06.508442+0000","last_clean":"2024-03-06T07:27:06.508442+0000","last_became_active":"2024-03-06T07:27:05.512982+0000","last_became_peered":"2024-03-06T07:27:05.512982+0000","last_unstale":"2024-03-06T07:27:06.508442+0000","last_undegraded":"2024-03-06T07:27:06.508442+0000","last_fullsized":"2024-03-06T07:27:06.508442+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:03.049668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708787+0000","last_change":"2024-03-06T07:27:05.509224+0000","last_active":"2024-03-06T07:27:06.708787+0000","last_peered":"2024-03-06T07:27:06.708787+0000","last_clean":"2024-03-06T07:27:06.708787+0000","last_became_active":"2024-03-06T07:27:05.508451+0000","last_became_peered":"2024-03-06T07:27:05.508451+0000","last_unstale":"2024-03-06T07:27:06.708787+0000","last_undegraded":"2024-03-06T07:27:06.708787+0000","last_fullsized":"2024-03-06T07:27:06.708787+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:19.304510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708403+0000","last_change":"2024-03-06T07:27:05.508024+0000","last_active":"2024-03-06T07:27:06.708403+0000","last_peered":"2024-03-06T07:27:06.708403+0000","last_clean":"2024-03-06T07:27:06.708403+0000","last_became_active":"2024-03-06T07:27:05.507735+0000","last_became_peered":"2024-03-06T07:27:05.507735+0000","last_unstale":"2024-03-06T07:27:06.708403+0000","last_undegraded":"2024-03-06T07:27:06.708403+0000","last_fullsized":"2024-03-06T07:27:06.708403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:37.031844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"106'4","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515183+0000","last_change":"2024-03-06T07:27:04.502116+0000","last_active":"2024-03-06T07:27:06.515183+0000","last_peered":"2024-03-06T07:27:06.515183+0000","last_clean":"2024-03-06T07:27:06.515183+0000","last_became_active":"2024-03-06T07:27:04.501823+0000","last_became_peered":"2024-03-06T07:27:04.501823+0000","last_unstale":"2024-03-06T07:27:06.515183+0000","last_undegraded":"2024-03-06T07:27:06.515183+0000","last_fullsized":"2024-03-06T07:27:06.515183+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:11.999443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510383+0000","last_change":"2024-03-06T07:27:05.511261+0000","last_active":"2024-03-06T07:27:06.510383+0000","last_peered":"2024-03-06T07:27:06.510383+0000","last_clean":"2024-03-06T07:27:06.510383+0000","last_became_active":"2024-03-06T07:27:05.510998+0000","last_became_peered":"2024-03-06T07:27:05.510998+0000","last_unstale":"2024-03-06T07:27:06.510383+0000","last_undegraded":"2024-03-06T07:27:06.510383+0000","last_fullsized":"2024-03-06T07:27:06.510383+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:14.941244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708228+0000","last_change":"2024-03-06T07:27:04.502792+0000","last_active":"2024-03-06T07:27:06.708228+0000","last_peered":"2024-03-06T07:27:06.708228+0000","last_clean":"2024-03-06T07:27:06.708228+0000","last_became_active":"2024-03-06T07:27:04.502507+0000","last_became_peered":"2024-03-06T07:27:04.502507+0000","last_unstale":"2024-03-06T07:27:06.708228+0000","last_undegraded":"2024-03-06T07:27:06.708228+0000","last_fullsized":"2024-03-06T07:27:06.708228+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:41.386524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.709062+0000","last_change":"2024-03-06T07:27:05.522334+0000","last_active":"2024-03-06T07:27:06.709062+0000","last_peered":"2024-03-06T07:27:06.709062+0000","last_clean":"2024-03-06T07:27:06.709062+0000","last_became_active":"2024-03-06T07:27:05.522086+0000","last_became_peered":"2024-03-06T07:27:05.522086+0000","last_unstale":"2024-03-06T07:27:06.709062+0000","last_undegraded":"2024-03-06T07:27:06.709062+0000","last_fullsized":"2024-03-06T07:27:06.709062+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:10.686400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708367+0000","last_change":"2024-03-06T07:27:04.507345+0000","last_active":"2024-03-06T07:27:06.708367+0000","last_peered":"2024-03-06T07:27:06.708367+0000","last_clean":"2024-03-06T07:27:06.708367+0000","last_became_active":"2024-03-06T07:27:04.507045+0000","last_became_peered":"2024-03-06T07:27:04.507045+0000","last_unstale":"2024-03-06T07:27:06.708367+0000","last_undegraded":"2024-03-06T07:27:06.708367+0000","last_fullsized":"2024-03-06T07:27:06.708367+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:16.209811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708579+0000","last_change":"2024-03-06T07:27:05.513084+0000","last_active":"2024-03-06T07:27:06.708579+0000","last_peered":"2024-03-06T07:27:06.708579+0000","last_clean":"2024-03-06T07:27:06.708579+0000","last_became_active":"2024-03-06T07:27:05.511910+0000","last_became_peered":"2024-03-06T07:27:05.511910+0000","last_unstale":"2024-03-06T07:27:06.708579+0000","last_undegraded":"2024-03-06T07:27:06.708579+0000","last_fullsized":"2024-03-06T07:27:06.708579+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:30.105296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707990+0000","last_change":"2024-03-06T07:27:04.505744+0000","last_active":"2024-03-06T07:27:06.707990+0000","last_peered":"2024-03-06T07:27:06.707990+0000","last_clean":"2024-03-06T07:27:06.707990+0000","last_became_active":"2024-03-06T07:27:04.505407+0000","last_became_peered":"2024-03-06T07:27:04.505407+0000","last_unstale":"2024-03-06T07:27:06.707990+0000","last_undegraded":"2024-03-06T07:27:06.707990+0000","last_fullsized":"2024-03-06T07:27:06.707990+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:35.997259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508397+0000","last_change":"2024-03-06T07:27:05.524970+0000","last_active":"2024-03-06T07:27:06.508397+0000","last_peered":"2024-03-06T07:27:06.508397+0000","last_clean":"2024-03-06T07:27:06.508397+0000","last_became_active":"2024-03-06T07:27:05.524697+0000","last_became_peered":"2024-03-06T07:27:05.524697+0000","last_unstale":"2024-03-06T07:27:06.508397+0000","last_undegraded":"2024-03-06T07:27:06.508397+0000","last_fullsized":"2024-03-06T07:27:06.508397+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:32.520587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515006+0000","last_change":"2024-03-06T07:27:04.504039+0000","last_active":"2024-03-06T07:27:06.515006+0000","last_peered":"2024-03-06T07:27:06.515006+0000","last_clean":"2024-03-06T07:27:06.515006+0000","last_became_active":"2024-03-06T07:27:04.503760+0000","last_became_peered":"2024-03-06T07:27:04.503760+0000","last_unstale":"2024-03-06T07:27:06.515006+0000","last_undegraded":"2024-03-06T07:27:06.515006+0000","last_fullsized":"2024-03-06T07:27:06.515006+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:09.677567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509596+0000","last_change":"2024-03-06T07:27:05.516797+0000","last_active":"2024-03-06T07:27:06.509596+0000","last_peered":"2024-03-06T07:27:06.509596+0000","last_clean":"2024-03-06T07:27:06.509596+0000","last_became_active":"2024-03-06T07:27:05.513702+0000","last_became_peered":"2024-03-06T07:27:05.513702+0000","last_unstale":"2024-03-06T07:27:06.509596+0000","last_undegraded":"2024-03-06T07:27:06.509596+0000","last_fullsized":"2024-03-06T07:27:06.509596+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:09.594835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708529+0000","last_change":"2024-03-06T07:27:04.508176+0000","last_active":"2024-03-06T07:27:06.708529+0000","last_peered":"2024-03-06T07:27:06.708529+0000","last_clean":"2024-03-06T07:27:06.708529+0000","last_became_active":"2024-03-06T07:27:04.507909+0000","last_became_peered":"2024-03-06T07:27:04.507909+0000","last_unstale":"2024-03-06T07:27:06.708529+0000","last_undegraded":"2024-03-06T07:27:06.708529+0000","last_fullsized":"2024-03-06T07:27:06.708529+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:09.745694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508382+0000","last_change":"2024-03-06T07:27:05.516805+0000","last_active":"2024-03-06T07:27:06.508382+0000","last_peered":"2024-03-06T07:27:06.508382+0000","last_clean":"2024-03-06T07:27:06.508382+0000","last_became_active":"2024-03-06T07:27:05.515620+0000","last_became_peered":"2024-03-06T07:27:05.515620+0000","last_unstale":"2024-03-06T07:27:06.508382+0000","last_undegraded":"2024-03-06T07:27:06.508382+0000","last_fullsized":"2024-03-06T07:27:06.508382+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:02.772037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508888+0000","last_change":"2024-03-06T07:27:04.511197+0000","last_active":"2024-03-06T07:27:06.508888+0000","last_peered":"2024-03-06T07:27:06.508888+0000","last_clean":"2024-03-06T07:27:06.508888+0000","last_became_active":"2024-03-06T07:27:04.510955+0000","last_became_peered":"2024-03-06T07:27:04.510955+0000","last_unstale":"2024-03-06T07:27:06.508888+0000","last_undegraded":"2024-03-06T07:27:06.508888+0000","last_fullsized":"2024-03-06T07:27:06.508888+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:25.044276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515166+0000","last_change":"2024-03-06T07:27:05.518317+0000","last_active":"2024-03-06T07:27:06.515166+0000","last_peered":"2024-03-06T07:27:06.515166+0000","last_clean":"2024-03-06T07:27:06.515166+0000","last_became_active":"2024-03-06T07:27:05.518067+0000","last_became_peered":"2024-03-06T07:27:05.518067+0000","last_unstale":"2024-03-06T07:27:06.515166+0000","last_undegraded":"2024-03-06T07:27:06.515166+0000","last_fullsized":"2024-03-06T07:27:06.515166+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:06.882188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508406+0000","last_change":"2024-03-06T07:27:04.510589+0000","last_active":"2024-03-06T07:27:06.508406+0000","last_peered":"2024-03-06T07:27:06.508406+0000","last_clean":"2024-03-06T07:27:06.508406+0000","last_became_active":"2024-03-06T07:27:04.510334+0000","last_became_peered":"2024-03-06T07:27:04.510334+0000","last_unstale":"2024-03-06T07:27:06.508406+0000","last_undegraded":"2024-03-06T07:27:06.508406+0000","last_fullsized":"2024-03-06T07:27:06.508406+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:33.736855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515251+0000","last_change":"2024-03-06T07:27:05.513812+0000","last_active":"2024-03-06T07:27:06.515251+0000","last_peered":"2024-03-06T07:27:06.515251+0000","last_clean":"2024-03-06T07:27:06.515251+0000","last_became_active":"2024-03-06T07:27:05.513549+0000","last_became_peered":"2024-03-06T07:27:05.513549+0000","last_unstale":"2024-03-06T07:27:06.515251+0000","last_undegraded":"2024-03-06T07:27:06.515251+0000","last_fullsized":"2024-03-06T07:27:06.515251+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:46.921413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708089+0000","last_change":"2024-03-06T07:27:04.507007+0000","last_active":"2024-03-06T07:27:06.708089+0000","last_peered":"2024-03-06T07:27:06.708089+0000","last_clean":"2024-03-06T07:27:06.708089+0000","last_became_active":"2024-03-06T07:27:04.506737+0000","last_became_peered":"2024-03-06T07:27:04.506737+0000","last_unstale":"2024-03-06T07:27:06.708089+0000","last_undegraded":"2024-03-06T07:27:06.708089+0000","last_fullsized":"2024-03-06T07:27:06.708089+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:06.025771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510396+0000","last_change":"2024-03-06T07:27:05.514386+0000","last_active":"2024-03-06T07:27:06.510396+0000","last_peered":"2024-03-06T07:27:06.510396+0000","last_clean":"2024-03-06T07:27:06.510396+0000","last_became_active":"2024-03-06T07:27:05.514000+0000","last_became_peered":"2024-03-06T07:27:05.514000+0000","last_unstale":"2024-03-06T07:27:06.510396+0000","last_undegraded":"2024-03-06T07:27:06.510396+0000","last_fullsized":"2024-03-06T07:27:06.510396+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:09.236618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708837+0000","last_change":"2024-03-06T07:27:04.509676+0000","last_active":"2024-03-06T07:27:06.708837+0000","last_peered":"2024-03-06T07:27:06.708837+0000","last_clean":"2024-03-06T07:27:06.708837+0000","last_became_active":"2024-03-06T07:27:04.509404+0000","last_became_peered":"2024-03-06T07:27:04.509404+0000","last_unstale":"2024-03-06T07:27:06.708837+0000","last_undegraded":"2024-03-06T07:27:06.708837+0000","last_fullsized":"2024-03-06T07:27:06.708837+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:08.227784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708654+0000","last_change":"2024-03-06T07:27:05.516853+0000","last_active":"2024-03-06T07:27:06.708654+0000","last_peered":"2024-03-06T07:27:06.708654+0000","last_clean":"2024-03-06T07:27:06.708654+0000","last_became_active":"2024-03-06T07:27:05.516605+0000","last_became_peered":"2024-03-06T07:27:05.516605+0000","last_unstale":"2024-03-06T07:27:06.708654+0000","last_undegraded":"2024-03-06T07:27:06.708654+0000","last_fullsized":"2024-03-06T07:27:06.708654+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:48.505441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"106'3","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508596+0000","last_change":"2024-03-06T07:27:04.506613+0000","last_active":"2024-03-06T07:27:06.508596+0000","last_peered":"2024-03-06T07:27:06.508596+0000","last_clean":"2024-03-06T07:27:06.508596+0000","last_became_active":"2024-03-06T07:27:04.506356+0000","last_became_peered":"2024-03-06T07:27:04.506356+0000","last_unstale":"2024-03-06T07:27:06.508596+0000","last_undegraded":"2024-03-06T07:27:06.508596+0000","last_fullsized":"2024-03-06T07:27:06.508596+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:44.338877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515202+0000","last_change":"2024-03-06T07:27:05.514902+0000","last_active":"2024-03-06T07:27:06.515202+0000","last_peered":"2024-03-06T07:27:06.515202+0000","last_clean":"2024-03-06T07:27:06.515202+0000","last_became_active":"2024-03-06T07:27:05.514635+0000","last_became_peered":"2024-03-06T07:27:05.514635+0000","last_unstale":"2024-03-06T07:27:06.515202+0000","last_undegraded":"2024-03-06T07:27:06.515202+0000","last_fullsized":"2024-03-06T07:27:06.515202+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:40.357849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"106'6","reported_seq":20,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508934+0000","last_change":"2024-03-06T07:27:04.510935+0000","last_active":"2024-03-06T07:27:06.508934+0000","last_peered":"2024-03-06T07:27:06.508934+0000","last_clean":"2024-03-06T07:27:06.508934+0000","last_became_active":"2024-03-06T07:27:04.510640+0000","last_became_peered":"2024-03-06T07:27:04.510640+0000","last_unstale":"2024-03-06T07:27:06.508934+0000","last_undegraded":"2024-03-06T07:27:06.508934+0000","last_fullsized":"2024-03-06T07:27:06.508934+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:06.973235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708478+0000","last_change":"2024-03-06T07:27:05.512490+0000","last_active":"2024-03-06T07:27:06.708478+0000","last_peered":"2024-03-06T07:27:06.708478+0000","last_clean":"2024-03-06T07:27:06.708478+0000","last_became_active":"2024-03-06T07:27:05.512232+0000","last_became_peered":"2024-03-06T07:27:05.512232+0000","last_unstale":"2024-03-06T07:27:06.708478+0000","last_undegraded":"2024-03-06T07:27:06.708478+0000","last_fullsized":"2024-03-06T07:27:06.708478+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:13.805842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708936+0000","last_change":"2024-03-06T07:27:04.504658+0000","last_active":"2024-03-06T07:27:06.708936+0000","last_peered":"2024-03-06T07:27:06.708936+0000","last_clean":"2024-03-06T07:27:06.708936+0000","last_became_active":"2024-03-06T07:27:04.503982+0000","last_became_peered":"2024-03-06T07:27:04.503982+0000","last_unstale":"2024-03-06T07:27:06.708936+0000","last_undegraded":"2024-03-06T07:27:06.708936+0000","last_fullsized":"2024-03-06T07:27:06.708936+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.429984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708110+0000","last_change":"2024-03-06T07:27:05.519929+0000","last_active":"2024-03-06T07:27:06.708110+0000","last_peered":"2024-03-06T07:27:06.708110+0000","last_clean":"2024-03-06T07:27:06.708110+0000","last_became_active":"2024-03-06T07:27:05.519561+0000","last_became_peered":"2024-03-06T07:27:05.519561+0000","last_unstale":"2024-03-06T07:27:06.708110+0000","last_undegraded":"2024-03-06T07:27:06.708110+0000","last_fullsized":"2024-03-06T07:27:06.708110+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:30.105296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509432+0000","last_change":"2024-03-06T07:27:04.495580+0000","last_active":"2024-03-06T07:27:06.509432+0000","last_peered":"2024-03-06T07:27:06.509432+0000","last_clean":"2024-03-06T07:27:06.509432+0000","last_became_active":"2024-03-06T07:27:04.495226+0000","last_became_peered":"2024-03-06T07:27:04.495226+0000","last_unstale":"2024-03-06T07:27:06.509432+0000","last_undegraded":"2024-03-06T07:27:06.509432+0000","last_fullsized":"2024-03-06T07:27:06.509432+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:26.865440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707870+0000","last_change":"2024-03-06T07:27:05.512090+0000","last_active":"2024-03-06T07:27:06.707870+0000","last_peered":"2024-03-06T07:27:06.707870+0000","last_clean":"2024-03-06T07:27:06.707870+0000","last_became_active":"2024-03-06T07:27:05.511808+0000","last_became_peered":"2024-03-06T07:27:05.511808+0000","last_unstale":"2024-03-06T07:27:06.707870+0000","last_undegraded":"2024-03-06T07:27:06.707870+0000","last_fullsized":"2024-03-06T07:27:06.707870+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:07.982069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"106'8","reported_seq":22,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707735+0000","last_change":"2024-03-06T07:27:04.509819+0000","last_active":"2024-03-06T07:27:06.707735+0000","last_peered":"2024-03-06T07:27:06.707735+0000","last_clean":"2024-03-06T07:27:06.707735+0000","last_became_active":"2024-03-06T07:27:04.509471+0000","last_became_peered":"2024-03-06T07:27:04.509471+0000","last_unstale":"2024-03-06T07:27:06.707735+0000","last_undegraded":"2024-03-06T07:27:06.707735+0000","last_fullsized":"2024-03-06T07:27:06.707735+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:43.442246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708093+0000","last_change":"2024-03-06T07:27:05.515923+0000","last_active":"2024-03-06T07:27:06.708093+0000","last_peered":"2024-03-06T07:27:06.708093+0000","last_clean":"2024-03-06T07:27:06.708093+0000","last_became_active":"2024-03-06T07:27:05.515620+0000","last_became_peered":"2024-03-06T07:27:05.515620+0000","last_unstale":"2024-03-06T07:27:06.708093+0000","last_undegraded":"2024-03-06T07:27:06.708093+0000","last_fullsized":"2024-03-06T07:27:06.708093+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:41.721404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509107+0000","last_change":"2024-03-06T07:27:04.500475+0000","last_active":"2024-03-06T07:27:06.509107+0000","last_peered":"2024-03-06T07:27:06.509107+0000","last_clean":"2024-03-06T07:27:06.509107+0000","last_became_active":"2024-03-06T07:27:04.500179+0000","last_became_peered":"2024-03-06T07:27:04.500179+0000","last_unstale":"2024-03-06T07:27:06.509107+0000","last_undegraded":"2024-03-06T07:27:06.509107+0000","last_fullsized":"2024-03-06T07:27:06.509107+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:25.482014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708991+0000","last_change":"2024-03-06T07:27:05.524995+0000","last_active":"2024-03-06T07:27:06.708991+0000","last_peered":"2024-03-06T07:27:06.708991+0000","last_clean":"2024-03-06T07:27:06.708991+0000","last_became_active":"2024-03-06T07:27:05.524744+0000","last_became_peered":"2024-03-06T07:27:05.524744+0000","last_unstale":"2024-03-06T07:27:06.708991+0000","last_undegraded":"2024-03-06T07:27:06.708991+0000","last_fullsized":"2024-03-06T07:27:06.708991+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:00.299260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708189+0000","last_change":"2024-03-06T07:27:04.507777+0000","last_active":"2024-03-06T07:27:06.708189+0000","last_peered":"2024-03-06T07:27:06.708189+0000","last_clean":"2024-03-06T07:27:06.708189+0000","last_became_active":"2024-03-06T07:27:04.507525+0000","last_became_peered":"2024-03-06T07:27:04.507525+0000","last_unstale":"2024-03-06T07:27:06.708189+0000","last_undegraded":"2024-03-06T07:27:06.708189+0000","last_fullsized":"2024-03-06T07:27:06.708189+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.429984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708295+0000","last_change":"2024-03-06T07:27:05.516417+0000","last_active":"2024-03-06T07:27:06.708295+0000","last_peered":"2024-03-06T07:27:06.708295+0000","last_clean":"2024-03-06T07:27:06.708295+0000","last_became_active":"2024-03-06T07:27:05.516118+0000","last_became_peered":"2024-03-06T07:27:05.516118+0000","last_unstale":"2024-03-06T07:27:06.708295+0000","last_undegraded":"2024-03-06T07:27:06.708295+0000","last_fullsized":"2024-03-06T07:27:06.708295+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:27.252775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509768+0000","last_change":"2024-03-06T07:27:04.511779+0000","last_active":"2024-03-06T07:27:06.509768+0000","last_peered":"2024-03-06T07:27:06.509768+0000","last_clean":"2024-03-06T07:27:06.509768+0000","last_became_active":"2024-03-06T07:27:04.511485+0000","last_became_peered":"2024-03-06T07:27:04.511485+0000","last_unstale":"2024-03-06T07:27:06.509768+0000","last_undegraded":"2024-03-06T07:27:06.509768+0000","last_fullsized":"2024-03-06T07:27:06.509768+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:58.571185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708629+0000","last_change":"2024-03-06T07:27:05.516589+0000","last_active":"2024-03-06T07:27:06.708629+0000","last_peered":"2024-03-06T07:27:06.708629+0000","last_clean":"2024-03-06T07:27:06.708629+0000","last_became_active":"2024-03-06T07:27:05.516012+0000","last_became_peered":"2024-03-06T07:27:05.516012+0000","last_unstale":"2024-03-06T07:27:06.708629+0000","last_undegraded":"2024-03-06T07:27:06.708629+0000","last_fullsized":"2024-03-06T07:27:06.708629+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:11.264585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515038+0000","last_change":"2024-03-06T07:27:04.504834+0000","last_active":"2024-03-06T07:27:06.515038+0000","last_peered":"2024-03-06T07:27:06.515038+0000","last_clean":"2024-03-06T07:27:06.515038+0000","last_became_active":"2024-03-06T07:27:04.504370+0000","last_became_peered":"2024-03-06T07:27:04.504370+0000","last_unstale":"2024-03-06T07:27:06.515038+0000","last_undegraded":"2024-03-06T07:27:06.515038+0000","last_fullsized":"2024-03-06T07:27:06.515038+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:59.290427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708133+0000","last_change":"2024-03-06T07:27:05.510415+0000","last_active":"2024-03-06T07:27:06.708133+0000","last_peered":"2024-03-06T07:27:06.708133+0000","last_clean":"2024-03-06T07:27:06.708133+0000","last_became_active":"2024-03-06T07:27:05.510112+0000","last_became_peered":"2024-03-06T07:27:05.510112+0000","last_unstale":"2024-03-06T07:27:06.708133+0000","last_undegraded":"2024-03-06T07:27:06.708133+0000","last_fullsized":"2024-03-06T07:27:06.708133+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:54.039387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708991+0000","last_change":"2024-03-06T07:27:04.508545+0000","last_active":"2024-03-06T07:27:06.708991+0000","last_peered":"2024-03-06T07:27:06.708991+0000","last_clean":"2024-03-06T07:27:06.708991+0000","last_became_active":"2024-03-06T07:27:04.508279+0000","last_became_peered":"2024-03-06T07:27:04.508279+0000","last_unstale":"2024-03-06T07:27:06.708991+0000","last_undegraded":"2024-03-06T07:27:06.708991+0000","last_fullsized":"2024-03-06T07:27:06.708991+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:16.209811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508510+0000","last_change":"2024-03-06T07:27:05.513279+0000","last_active":"2024-03-06T07:27:06.508510+0000","last_peered":"2024-03-06T07:27:06.508510+0000","last_clean":"2024-03-06T07:27:06.508510+0000","last_became_active":"2024-03-06T07:27:05.513013+0000","last_became_peered":"2024-03-06T07:27:05.513013+0000","last_unstale":"2024-03-06T07:27:06.508510+0000","last_undegraded":"2024-03-06T07:27:06.508510+0000","last_fullsized":"2024-03-06T07:27:06.508510+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:22.699468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708582+0000","last_change":"2024-03-06T07:27:04.507769+0000","last_active":"2024-03-06T07:27:06.708582+0000","last_peered":"2024-03-06T07:27:06.708582+0000","last_clean":"2024-03-06T07:27:06.708582+0000","last_became_active":"2024-03-06T07:27:04.507455+0000","last_became_peered":"2024-03-06T07:27:04.507455+0000","last_unstale":"2024-03-06T07:27:06.708582+0000","last_undegraded":"2024-03-06T07:27:06.708582+0000","last_fullsized":"2024-03-06T07:27:06.708582+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:32.578061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708259+0000","last_change":"2024-03-06T07:27:05.512930+0000","last_active":"2024-03-06T07:27:06.708259+0000","last_peered":"2024-03-06T07:27:06.708259+0000","last_clean":"2024-03-06T07:27:06.708259+0000","last_became_active":"2024-03-06T07:27:05.512332+0000","last_became_peered":"2024-03-06T07:27:05.512332+0000","last_unstale":"2024-03-06T07:27:06.708259+0000","last_undegraded":"2024-03-06T07:27:06.708259+0000","last_fullsized":"2024-03-06T07:27:06.708259+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:26.490847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708777+0000","last_change":"2024-03-06T07:27:04.508385+0000","last_active":"2024-03-06T07:27:06.708777+0000","last_peered":"2024-03-06T07:27:06.708777+0000","last_clean":"2024-03-06T07:27:06.708777+0000","last_became_active":"2024-03-06T07:27:04.508006+0000","last_became_peered":"2024-03-06T07:27:04.508006+0000","last_unstale":"2024-03-06T07:27:06.708777+0000","last_undegraded":"2024-03-06T07:27:06.708777+0000","last_fullsized":"2024-03-06T07:27:06.708777+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:35.745169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707993+0000","last_change":"2024-03-06T07:27:05.518929+0000","last_active":"2024-03-06T07:27:06.707993+0000","last_peered":"2024-03-06T07:27:06.707993+0000","last_clean":"2024-03-06T07:27:06.707993+0000","last_became_active":"2024-03-06T07:27:05.517876+0000","last_became_peered":"2024-03-06T07:27:05.517876+0000","last_unstale":"2024-03-06T07:27:06.707993+0000","last_undegraded":"2024-03-06T07:27:06.707993+0000","last_fullsized":"2024-03-06T07:27:06.707993+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:03.124647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508552+0000","last_change":"2024-03-06T07:27:04.495315+0000","last_active":"2024-03-06T07:27:06.508552+0000","last_peered":"2024-03-06T07:27:06.508552+0000","last_clean":"2024-03-06T07:27:06.508552+0000","last_became_active":"2024-03-06T07:27:04.494327+0000","last_became_peered":"2024-03-06T07:27:04.494327+0000","last_unstale":"2024-03-06T07:27:06.508552+0000","last_undegraded":"2024-03-06T07:27:06.508552+0000","last_fullsized":"2024-03-06T07:27:06.508552+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:33.714081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509961+0000","last_change":"2024-03-06T07:27:05.515820+0000","last_active":"2024-03-06T07:27:06.509961+0000","last_peered":"2024-03-06T07:27:06.509961+0000","last_clean":"2024-03-06T07:27:06.509961+0000","last_became_active":"2024-03-06T07:27:05.515336+0000","last_became_peered":"2024-03-06T07:27:05.515336+0000","last_unstale":"2024-03-06T07:27:06.509961+0000","last_undegraded":"2024-03-06T07:27:06.509961+0000","last_fullsized":"2024-03-06T07:27:06.509961+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:15.364800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708295+0000","last_change":"2024-03-06T07:27:04.513054+0000","last_active":"2024-03-06T07:27:06.708295+0000","last_peered":"2024-03-06T07:27:06.708295+0000","last_clean":"2024-03-06T07:27:06.708295+0000","last_became_active":"2024-03-06T07:27:04.512612+0000","last_became_peered":"2024-03-06T07:27:04.512612+0000","last_unstale":"2024-03-06T07:27:06.708295+0000","last_undegraded":"2024-03-06T07:27:06.708295+0000","last_fullsized":"2024-03-06T07:27:06.708295+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:23.404575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509042+0000","last_change":"2024-03-06T07:27:05.511737+0000","last_active":"2024-03-06T07:27:06.509042+0000","last_peered":"2024-03-06T07:27:06.509042+0000","last_clean":"2024-03-06T07:27:06.509042+0000","last_became_active":"2024-03-06T07:27:05.511252+0000","last_became_peered":"2024-03-06T07:27:05.511252+0000","last_unstale":"2024-03-06T07:27:06.509042+0000","last_undegraded":"2024-03-06T07:27:06.509042+0000","last_fullsized":"2024-03-06T07:27:06.509042+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:43.413821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.514293+0000","last_change":"2024-03-06T07:27:04.499630+0000","last_active":"2024-03-06T07:27:06.514293+0000","last_peered":"2024-03-06T07:27:06.514293+0000","last_clean":"2024-03-06T07:27:06.514293+0000","last_became_active":"2024-03-06T07:27:04.499350+0000","last_became_peered":"2024-03-06T07:27:04.499350+0000","last_unstale":"2024-03-06T07:27:06.514293+0000","last_undegraded":"2024-03-06T07:27:06.514293+0000","last_fullsized":"2024-03-06T07:27:06.514293+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:40.717999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508915+0000","last_change":"2024-03-06T07:27:05.516038+0000","last_active":"2024-03-06T07:27:06.508915+0000","last_peered":"2024-03-06T07:27:06.508915+0000","last_clean":"2024-03-06T07:27:06.508915+0000","last_became_active":"2024-03-06T07:27:05.515697+0000","last_became_peered":"2024-03-06T07:27:05.515697+0000","last_unstale":"2024-03-06T07:27:06.508915+0000","last_undegraded":"2024-03-06T07:27:06.508915+0000","last_fullsized":"2024-03-06T07:27:06.508915+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:31.540460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508688+0000","last_change":"2024-03-06T07:27:04.494918+0000","last_active":"2024-03-06T07:27:06.508688+0000","last_peered":"2024-03-06T07:27:06.508688+0000","last_clean":"2024-03-06T07:27:06.508688+0000","last_became_active":"2024-03-06T07:27:04.494284+0000","last_became_peered":"2024-03-06T07:27:04.494284+0000","last_unstale":"2024-03-06T07:27:06.508688+0000","last_undegraded":"2024-03-06T07:27:06.508688+0000","last_fullsized":"2024-03-06T07:27:06.508688+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:26.243942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509639+0000","last_change":"2024-03-06T07:27:05.516906+0000","last_active":"2024-03-06T07:27:06.509639+0000","last_peered":"2024-03-06T07:27:06.509639+0000","last_clean":"2024-03-06T07:27:06.509639+0000","last_became_active":"2024-03-06T07:27:05.515925+0000","last_became_peered":"2024-03-06T07:27:05.515925+0000","last_unstale":"2024-03-06T07:27:06.509639+0000","last_undegraded":"2024-03-06T07:27:06.509639+0000","last_fullsized":"2024-03-06T07:27:06.509639+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:42.395357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708797+0000","last_change":"2024-03-06T07:27:04.509753+0000","last_active":"2024-03-06T07:27:06.708797+0000","last_peered":"2024-03-06T07:27:06.708797+0000","last_clean":"2024-03-06T07:27:06.708797+0000","last_became_active":"2024-03-06T07:27:04.509420+0000","last_became_peered":"2024-03-06T07:27:04.509420+0000","last_unstale":"2024-03-06T07:27:06.708797+0000","last_undegraded":"2024-03-06T07:27:06.708797+0000","last_fullsized":"2024-03-06T07:27:06.708797+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:14.428207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508910+0000","last_change":"2024-03-06T07:27:05.514877+0000","last_active":"2024-03-06T07:27:06.508910+0000","last_peered":"2024-03-06T07:27:06.508910+0000","last_clean":"2024-03-06T07:27:06.508910+0000","last_became_active":"2024-03-06T07:27:05.514589+0000","last_became_peered":"2024-03-06T07:27:05.514589+0000","last_unstale":"2024-03-06T07:27:06.508910+0000","last_undegraded":"2024-03-06T07:27:06.508910+0000","last_fullsized":"2024-03-06T07:27:06.508910+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:00.136435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515018+0000","last_change":"2024-03-06T07:27:04.503742+0000","last_active":"2024-03-06T07:27:06.515018+0000","last_peered":"2024-03-06T07:27:06.515018+0000","last_clean":"2024-03-06T07:27:06.515018+0000","last_became_active":"2024-03-06T07:27:04.503451+0000","last_became_peered":"2024-03-06T07:27:04.503451+0000","last_unstale":"2024-03-06T07:27:06.515018+0000","last_undegraded":"2024-03-06T07:27:06.515018+0000","last_fullsized":"2024-03-06T07:27:06.515018+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:28.564626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509895+0000","last_change":"2024-03-06T07:27:05.516508+0000","last_active":"2024-03-06T07:27:06.509895+0000","last_peered":"2024-03-06T07:27:06.509895+0000","last_clean":"2024-03-06T07:27:06.509895+0000","last_became_active":"2024-03-06T07:27:05.514001+0000","last_became_peered":"2024-03-06T07:27:05.514001+0000","last_unstale":"2024-03-06T07:27:06.509895+0000","last_undegraded":"2024-03-06T07:27:06.509895+0000","last_fullsized":"2024-03-06T07:27:06.509895+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:24.413408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508979+0000","last_change":"2024-03-06T07:27:04.503200+0000","last_active":"2024-03-06T07:27:06.508979+0000","last_peered":"2024-03-06T07:27:06.508979+0000","last_clean":"2024-03-06T07:27:06.508979+0000","last_became_active":"2024-03-06T07:27:04.502934+0000","last_became_peered":"2024-03-06T07:27:04.502934+0000","last_unstale":"2024-03-06T07:27:06.508979+0000","last_undegraded":"2024-03-06T07:27:06.508979+0000","last_fullsized":"2024-03-06T07:27:06.508979+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:55.222283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708499+0000","last_change":"2024-03-06T07:27:05.519614+0000","last_active":"2024-03-06T07:27:06.708499+0000","last_peered":"2024-03-06T07:27:06.708499+0000","last_clean":"2024-03-06T07:27:06.708499+0000","last_became_active":"2024-03-06T07:27:05.519341+0000","last_became_peered":"2024-03-06T07:27:05.519341+0000","last_unstale":"2024-03-06T07:27:06.708499+0000","last_undegraded":"2024-03-06T07:27:06.708499+0000","last_fullsized":"2024-03-06T07:27:06.708499+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:50.417505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707798+0000","last_change":"2024-03-06T07:27:04.506431+0000","last_active":"2024-03-06T07:27:06.707798+0000","last_peered":"2024-03-06T07:27:06.707798+0000","last_clean":"2024-03-06T07:27:06.707798+0000","last_became_active":"2024-03-06T07:27:04.506072+0000","last_became_peered":"2024-03-06T07:27:04.506072+0000","last_unstale":"2024-03-06T07:27:06.707798+0000","last_undegraded":"2024-03-06T07:27:06.707798+0000","last_fullsized":"2024-03-06T07:27:06.707798+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:35.745169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508571+0000","last_change":"2024-03-06T07:27:05.518288+0000","last_active":"2024-03-06T07:27:06.508571+0000","last_peered":"2024-03-06T07:27:06.508571+0000","last_clean":"2024-03-06T07:27:06.508571+0000","last_became_active":"2024-03-06T07:27:05.518038+0000","last_became_peered":"2024-03-06T07:27:05.518038+0000","last_unstale":"2024-03-06T07:27:06.508571+0000","last_undegraded":"2024-03-06T07:27:06.508571+0000","last_fullsized":"2024-03-06T07:27:06.508571+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:38.789157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508730+0000","last_change":"2024-03-06T07:27:04.495246+0000","last_active":"2024-03-06T07:27:06.508730+0000","last_peered":"2024-03-06T07:27:06.508730+0000","last_clean":"2024-03-06T07:27:06.508730+0000","last_became_active":"2024-03-06T07:27:04.494755+0000","last_became_peered":"2024-03-06T07:27:04.494755+0000","last_unstale":"2024-03-06T07:27:06.508730+0000","last_undegraded":"2024-03-06T07:27:06.508730+0000","last_fullsized":"2024-03-06T07:27:06.508730+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:53.030553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707902+0000","last_change":"2024-03-06T07:27:05.520066+0000","last_active":"2024-03-06T07:27:06.707902+0000","last_peered":"2024-03-06T07:27:06.707902+0000","last_clean":"2024-03-06T07:27:06.707902+0000","last_became_active":"2024-03-06T07:27:05.519462+0000","last_became_peered":"2024-03-06T07:27:05.519462+0000","last_unstale":"2024-03-06T07:27:06.707902+0000","last_undegraded":"2024-03-06T07:27:06.707902+0000","last_fullsized":"2024-03-06T07:27:06.707902+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:47.743962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515205+0000","last_change":"2024-03-06T07:27:04.504761+0000","last_active":"2024-03-06T07:27:06.515205+0000","last_peered":"2024-03-06T07:27:06.515205+0000","last_clean":"2024-03-06T07:27:06.515205+0000","last_became_active":"2024-03-06T07:27:04.504443+0000","last_became_peered":"2024-03-06T07:27:04.504443+0000","last_unstale":"2024-03-06T07:27:06.515205+0000","last_undegraded":"2024-03-06T07:27:06.515205+0000","last_fullsized":"2024-03-06T07:27:06.515205+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:20.512317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":322,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708542+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:27:06.708542+0000","last_peered":"2024-03-06T07:27:06.708542+0000","last_clean":"2024-03-06T07:27:06.708542+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:27:06.708542+0000","last_undegraded":"2024-03-06T07:27:06.708542+0000","last_fullsized":"2024-03-06T07:27:06.708542+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1933312,"data_stored":1164269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28324,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343516,"statfs":{"total":96636764160,"available":96607760384,"internally_reserved":0,"allocated":1335296,"data_stored":573355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1339392,"data_stored":573889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28324,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343516,"statfs":{"total":96636764160,"available":96607760384,"internally_reserved":0,"allocated":1331200,"data_stored":573285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1347584,"data_stored":574967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":573285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1339392,"data_stored":573409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1945600,"data_stored":1165377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:09.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:09.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.304+0000 7f3c5cf20640 1 Processor -- start 2024-03-06T07:27:09.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.304+0000 7f3c5cf20640 1 -- start start 2024-03-06T07:27:09.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c5cf20640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3c580d22b0 0x7f3c580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c5cf20640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c580b3430 con 0x7f3c580b41f0 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c5cf20640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c580b3710 con 0x7f3c5815dfe0 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c5cf20640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c580d2c50 con 0x7f3c580d22b0 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c56d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3c580b41f0 0x7f3c580b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38192/0 (socket says 172.21.15.77:38192) 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c56d9d640 1 -- 172.21.15.77:0/4285124258 learned_addr learned my addr 172.21.15.77:0/4285124258 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c4ffff640 1 --2- 172.21.15.77:0/4285124258 >> v2:172.21.15.158:3300/0 conn(0x7f3c580d22b0 0x7f3c580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121957094 0 0) 0x7f3c580b3430 con 0x7f3c580b41f0 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c44003520 con 0x7f3c580b41f0 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013534376 0 0) 0x7f3c44003520 con 0x7f3c580b41f0 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 >> v2:172.21.15.158:3300/0 conn(0x7f3c580d22b0 msgr2=0x7f3c580d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 --2- 172.21.15.77:0/4285124258 >> v2:172.21.15.158:3300/0 conn(0x7f3c580d22b0 0x7f3c580d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 >> v1:172.21.15.158:6789/0 conn(0x7f3c5815dfe0 legacy=0x7f3c580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c580b39f0 con 0x7f3c580b41f0 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3c500019f0 con 0x7f3c580b41f0 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.305+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 >> v1:172.21.15.77:6789/0 conn(0x7f3c580b41f0 legacy=0x7f3c580b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c55d9b640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c55d9b640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c44004800 0x7f3c44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c55d9b640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440051c0 0x7f3c4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4400c030 con 0x7f3c440039b0 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4400c1e0 con 0x7f3c44004800 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4400c3e0 con 0x7f3c440051c0 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c5659c640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c4ffff640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c44004800 0x7f3c44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c56d9d640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440051c0 0x7f3c4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c5659c640 1 -- 172.21.15.77:0/4285124258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440051c0 msgr2=0x7f3c4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c5659c640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440051c0 0x7f3c4400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c5659c640 1 -- 172.21.15.77:0/4285124258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c44004800 msgr2=0x7f3c44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c5659c640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c44004800 0x7f3c44004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c5659c640 1 -- 172.21.15.77:0/4285124258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c44003520 con 0x7f3c440039b0 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.306+0000 7f3c56d9d640 1 -- 172.21.15.77:0/4285124258 reap_dead start 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c5659c640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c44003dd0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f3c48001750 tx=0x7f3c44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82f3855678028d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48001370 con 0x7f3c440039b0 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c480052d0 con 0x7f3c440039b0 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c55d9b640 1 -- 172.21.15.77:0/4285124258 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48003830 con 0x7f3c440039b0 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c5cf20640 1 -- 172.21.15.77:0/4285124258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 msgr2=0x7f3c44003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c5cf20640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c44003dd0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f3c48001750 tx=0x7f3c44005830 comp rx=0 tx=0).stop 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c5cf20640 1 -- 172.21.15.77:0/4285124258 shutdown_connections 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c5cf20640 1 --2- 172.21.15.77:0/4285124258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c44003dd0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c5cf20640 1 -- 172.21.15.77:0/4285124258 >> 172.21.15.77:0/4285124258 conn(0x7f3c580d3cd0 msgr2=0x7f3c580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c5cf20640 1 -- 172.21.15.77:0/4285124258 shutdown_connections 2024-03-06T07:27:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.307+0000 7f3c5cf20640 1 -- 172.21.15.77:0/4285124258 wait complete. 2024-03-06T07:27:09.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.309+0000 7f3c5cf20640 1 Processor -- start 2024-03-06T07:27:09.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.309+0000 7f3c5cf20640 1 -- start start 2024-03-06T07:27:09.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.309+0000 7f3c5cf20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c580ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.309+0000 7f3c5cf20640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c580cf5e0 0x7f3c5806bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.309+0000 7f3c5cf20640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c5815efb0 0x7f3c5815f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.309+0000 7f3c5cf20640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580b3d60 con 0x7f3c440039b0 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.309+0000 7f3c5cf20640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580b31e0 con 0x7f3c580cf5e0 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.309+0000 7f3c5cf20640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5815e3c0 con 0x7f3c5815efb0 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c580ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c580ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36362/0 (socket says 172.21.15.77:36362) 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 -- 172.21.15.77:0/3413256480 learned_addr learned my addr 172.21.15.77:0/3413256480 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c4ffff640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c580cf5e0 0x7f3c5806bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c56d9d640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c5815efb0 0x7f3c5815f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 -- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c5815efb0 msgr2=0x7f3c5815f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c5815efb0 0x7f3c5815f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 -- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c580cf5e0 msgr2=0x7f3c5806bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c580cf5e0 0x7f3c5806bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 -- 172.21.15.77:0/3413256480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5815e7f0 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c56d9d640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c5815efb0 0x7f3c5815f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c4ffff640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c580cf5e0 0x7f3c5806bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5659c640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c580ccdb0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f3c48000c00 tx=0x7f3c48001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c4effd640 1 -- 172.21.15.77:0/3413256480 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48041070 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c4effd640 1 -- 172.21.15.77:0/3413256480 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c48005480 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c580cd690 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c4effd640 1 -- 172.21.15.77:0/3413256480 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48001370 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.310+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5815ea90 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.311+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5804f330 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.312+0000 7f3c4effd640 1 -- 172.21.15.77:0/3413256480 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c48044070 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.312+0000 7f3c4effd640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3806b500 0x7f3c3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.312+0000 7f3c4effd640 1 -- 172.21.15.77:0/3413256480 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c480abc10 con 0x7f3c440039b0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.312+0000 7f3c4ffff640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3806b500 0x7f3c3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.313+0000 7f3c4ffff640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3806b500 0x7f3c3806d9c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3c580a1d10 tx=0x7f3c40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.316+0000 7f3c4effd640 1 -- 172.21.15.77:0/3413256480 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c4807aa90 con 0x7f3c440039b0 2024-03-06T07:27:09.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.492+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3c580d2a90 con 0x7f3c3806b500 2024-03-06T07:27:09.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.498+0000 7f3c4effd640 1 -- 172.21.15.77:0/3413256480 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252480 (secure 0 0 0) 0x7f3c580d2a90 con 0x7f3c3806b500 2024-03-06T07:27:09.503 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:09.504 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3806b500 msgr2=0x7f3c3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3806b500 0x7f3c3806d9c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3c580a1d10 tx=0x7f3c40008040 comp rx=0 tx=0).stop 2024-03-06T07:27:09.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 msgr2=0x7f3c580ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c580ccdb0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f3c48000c00 tx=0x7f3c48001120 comp rx=0 tx=0).stop 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 shutdown_connections 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c5815efb0 0x7f3c5815f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c580cf5e0 0x7f3c5806bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3806b500 0x7f3c3806d9c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 --2- 172.21.15.77:0/3413256480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440039b0 0x7f3c580ccdb0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 >> 172.21.15.77:0/3413256480 conn(0x7f3c580d3cd0 msgr2=0x7f3c5806cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 shutdown_connections 2024-03-06T07:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.501+0000 7f3c5cf20640 1 -- 172.21.15.77:0/3413256480 wait complete. 2024-03-06T07:27:09.514 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-03-06T07:27:09.440796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302676,"kb_used_data":11624,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754672044,"statfs":{"total":773094113280,"available":772784173056,"internally_reserved":0,"allocated":11902976,"data_stored":5771836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.954388"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508739+0000","last_change":"2024-03-06T07:27:05.517498+0000","last_active":"2024-03-06T07:27:06.508739+0000","last_peered":"2024-03-06T07:27:06.508739+0000","last_clean":"2024-03-06T07:27:06.508739+0000","last_became_active":"2024-03-06T07:27:05.516214+0000","last_became_peered":"2024-03-06T07:27:05.516214+0000","last_unstale":"2024-03-06T07:27:06.508739+0000","last_undegraded":"2024-03-06T07:27:06.508739+0000","last_fullsized":"2024-03-06T07:27:06.508739+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:10.145209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509087+0000","last_change":"2024-03-06T07:27:05.515799+0000","last_active":"2024-03-06T07:27:06.509087+0000","last_peered":"2024-03-06T07:27:06.509087+0000","last_clean":"2024-03-06T07:27:06.509087+0000","last_became_active":"2024-03-06T07:27:05.514927+0000","last_became_peered":"2024-03-06T07:27:05.514927+0000","last_unstale":"2024-03-06T07:27:06.509087+0000","last_undegraded":"2024-03-06T07:27:06.509087+0000","last_fullsized":"2024-03-06T07:27:06.509087+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:09.335581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708276+0000","last_change":"2024-03-06T07:27:05.512196+0000","last_active":"2024-03-06T07:27:06.708276+0000","last_peered":"2024-03-06T07:27:06.708276+0000","last_clean":"2024-03-06T07:27:06.708276+0000","last_became_active":"2024-03-06T07:27:05.510165+0000","last_became_peered":"2024-03-06T07:27:05.510165+0000","last_unstale":"2024-03-06T07:27:06.708276+0000","last_undegraded":"2024-03-06T07:27:06.708276+0000","last_fullsized":"2024-03-06T07:27:06.708276+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:45.347711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508812+0000","last_change":"2024-03-06T07:27:05.509649+0000","last_active":"2024-03-06T07:27:06.508812+0000","last_peered":"2024-03-06T07:27:06.508812+0000","last_clean":"2024-03-06T07:27:06.508812+0000","last_became_active":"2024-03-06T07:27:05.509372+0000","last_became_peered":"2024-03-06T07:27:05.509372+0000","last_unstale":"2024-03-06T07:27:06.508812+0000","last_undegraded":"2024-03-06T07:27:06.508812+0000","last_fullsized":"2024-03-06T07:27:06.508812+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:21.408950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708725+0000","last_change":"2024-03-06T07:27:05.518091+0000","last_active":"2024-03-06T07:27:06.708725+0000","last_peered":"2024-03-06T07:27:06.708725+0000","last_clean":"2024-03-06T07:27:06.708725+0000","last_became_active":"2024-03-06T07:27:05.517801+0000","last_became_peered":"2024-03-06T07:27:05.517801+0000","last_unstale":"2024-03-06T07:27:06.708725+0000","last_undegraded":"2024-03-06T07:27:06.708725+0000","last_fullsized":"2024-03-06T07:27:06.708725+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:46.744879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708005+0000","last_change":"2024-03-06T07:27:05.521549+0000","last_active":"2024-03-06T07:27:06.708005+0000","last_peered":"2024-03-06T07:27:06.708005+0000","last_clean":"2024-03-06T07:27:06.708005+0000","last_became_active":"2024-03-06T07:27:05.521273+0000","last_became_peered":"2024-03-06T07:27:05.521273+0000","last_unstale":"2024-03-06T07:27:06.708005+0000","last_undegraded":"2024-03-06T07:27:06.708005+0000","last_fullsized":"2024-03-06T07:27:06.708005+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.687400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515081+0000","last_change":"2024-03-06T07:27:05.519605+0000","last_active":"2024-03-06T07:27:06.515081+0000","last_peered":"2024-03-06T07:27:06.515081+0000","last_clean":"2024-03-06T07:27:06.515081+0000","last_became_active":"2024-03-06T07:27:05.518706+0000","last_became_peered":"2024-03-06T07:27:05.518706+0000","last_unstale":"2024-03-06T07:27:06.515081+0000","last_undegraded":"2024-03-06T07:27:06.515081+0000","last_fullsized":"2024-03-06T07:27:06.515081+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:37.699345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708419+0000","last_change":"2024-03-06T07:27:05.512689+0000","last_active":"2024-03-06T07:27:06.708419+0000","last_peered":"2024-03-06T07:27:06.708419+0000","last_clean":"2024-03-06T07:27:06.708419+0000","last_became_active":"2024-03-06T07:27:05.511991+0000","last_became_peered":"2024-03-06T07:27:05.511991+0000","last_unstale":"2024-03-06T07:27:06.708419+0000","last_undegraded":"2024-03-06T07:27:06.708419+0000","last_fullsized":"2024-03-06T07:27:06.708419+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:47.743962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508899+0000","last_change":"2024-03-06T07:27:05.527111+0000","last_active":"2024-03-06T07:27:06.508899+0000","last_peered":"2024-03-06T07:27:06.508899+0000","last_clean":"2024-03-06T07:27:06.508899+0000","last_became_active":"2024-03-06T07:27:05.526568+0000","last_became_peered":"2024-03-06T07:27:05.526568+0000","last_unstale":"2024-03-06T07:27:06.508899+0000","last_undegraded":"2024-03-06T07:27:06.508899+0000","last_fullsized":"2024-03-06T07:27:06.508899+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:40.382167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508654+0000","last_change":"2024-03-06T07:27:05.514806+0000","last_active":"2024-03-06T07:27:06.508654+0000","last_peered":"2024-03-06T07:27:06.508654+0000","last_clean":"2024-03-06T07:27:06.508654+0000","last_became_active":"2024-03-06T07:27:05.514487+0000","last_became_peered":"2024-03-06T07:27:05.514487+0000","last_unstale":"2024-03-06T07:27:06.508654+0000","last_undegraded":"2024-03-06T07:27:06.508654+0000","last_fullsized":"2024-03-06T07:27:06.508654+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:14.000752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509167+0000","last_change":"2024-03-06T07:27:05.518246+0000","last_active":"2024-03-06T07:27:06.509167+0000","last_peered":"2024-03-06T07:27:06.509167+0000","last_clean":"2024-03-06T07:27:06.509167+0000","last_became_active":"2024-03-06T07:27:05.517970+0000","last_became_peered":"2024-03-06T07:27:05.517970+0000","last_unstale":"2024-03-06T07:27:06.509167+0000","last_undegraded":"2024-03-06T07:27:06.509167+0000","last_fullsized":"2024-03-06T07:27:06.509167+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:42.395357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508665+0000","last_change":"2024-03-06T07:27:05.512849+0000","last_active":"2024-03-06T07:27:06.508665+0000","last_peered":"2024-03-06T07:27:06.508665+0000","last_clean":"2024-03-06T07:27:06.508665+0000","last_became_active":"2024-03-06T07:27:05.512560+0000","last_became_peered":"2024-03-06T07:27:05.512560+0000","last_unstale":"2024-03-06T07:27:06.508665+0000","last_undegraded":"2024-03-06T07:27:06.508665+0000","last_fullsized":"2024-03-06T07:27:06.508665+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:51.149239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708417+0000","last_change":"2024-03-06T07:27:05.518981+0000","last_active":"2024-03-06T07:27:06.708417+0000","last_peered":"2024-03-06T07:27:06.708417+0000","last_clean":"2024-03-06T07:27:06.708417+0000","last_became_active":"2024-03-06T07:27:05.518688+0000","last_became_peered":"2024-03-06T07:27:05.518688+0000","last_unstale":"2024-03-06T07:27:06.708417+0000","last_undegraded":"2024-03-06T07:27:06.708417+0000","last_fullsized":"2024-03-06T07:27:06.708417+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:07.982069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708108+0000","last_change":"2024-03-06T07:27:05.512368+0000","last_active":"2024-03-06T07:27:06.708108+0000","last_peered":"2024-03-06T07:27:06.708108+0000","last_clean":"2024-03-06T07:27:06.708108+0000","last_became_active":"2024-03-06T07:27:05.512045+0000","last_became_peered":"2024-03-06T07:27:05.512045+0000","last_unstale":"2024-03-06T07:27:06.708108+0000","last_undegraded":"2024-03-06T07:27:06.708108+0000","last_fullsized":"2024-03-06T07:27:06.708108+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:34.722915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708625+0000","last_change":"2024-03-06T07:27:05.507962+0000","last_active":"2024-03-06T07:27:06.708625+0000","last_peered":"2024-03-06T07:27:06.708625+0000","last_clean":"2024-03-06T07:27:06.708625+0000","last_became_active":"2024-03-06T07:27:05.507665+0000","last_became_peered":"2024-03-06T07:27:05.507665+0000","last_unstale":"2024-03-06T07:27:06.708625+0000","last_undegraded":"2024-03-06T07:27:06.708625+0000","last_fullsized":"2024-03-06T07:27:06.708625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:58.357775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508307+0000","last_change":"2024-03-06T07:27:05.515381+0000","last_active":"2024-03-06T07:27:06.508307+0000","last_peered":"2024-03-06T07:27:06.508307+0000","last_clean":"2024-03-06T07:27:06.508307+0000","last_became_active":"2024-03-06T07:27:05.515062+0000","last_became_peered":"2024-03-06T07:27:05.515062+0000","last_unstale":"2024-03-06T07:27:06.508307+0000","last_undegraded":"2024-03-06T07:27:06.508307+0000","last_fullsized":"2024-03-06T07:27:06.508307+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:58.299008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509805+0000","last_change":"2024-03-06T07:27:05.520807+0000","last_active":"2024-03-06T07:27:06.509805+0000","last_peered":"2024-03-06T07:27:06.509805+0000","last_clean":"2024-03-06T07:27:06.509805+0000","last_became_active":"2024-03-06T07:27:05.520537+0000","last_became_peered":"2024-03-06T07:27:05.520537+0000","last_unstale":"2024-03-06T07:27:06.509805+0000","last_undegraded":"2024-03-06T07:27:06.509805+0000","last_fullsized":"2024-03-06T07:27:06.509805+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:09.236618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510002+0000","last_change":"2024-03-06T07:27:05.510433+0000","last_active":"2024-03-06T07:27:06.510002+0000","last_peered":"2024-03-06T07:27:06.510002+0000","last_clean":"2024-03-06T07:27:06.510002+0000","last_became_active":"2024-03-06T07:27:05.510117+0000","last_became_peered":"2024-03-06T07:27:05.510117+0000","last_unstale":"2024-03-06T07:27:06.510002+0000","last_undegraded":"2024-03-06T07:27:06.510002+0000","last_fullsized":"2024-03-06T07:27:06.510002+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:09.594835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708476+0000","last_change":"2024-03-06T07:27:05.512658+0000","last_active":"2024-03-06T07:27:06.708476+0000","last_peered":"2024-03-06T07:27:06.708476+0000","last_clean":"2024-03-06T07:27:06.708476+0000","last_became_active":"2024-03-06T07:27:05.511478+0000","last_became_peered":"2024-03-06T07:27:05.511478+0000","last_unstale":"2024-03-06T07:27:06.708476+0000","last_undegraded":"2024-03-06T07:27:06.708476+0000","last_fullsized":"2024-03-06T07:27:06.708476+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:03.124647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509646+0000","last_change":"2024-03-06T07:27:05.519916+0000","last_active":"2024-03-06T07:27:06.509646+0000","last_peered":"2024-03-06T07:27:06.509646+0000","last_clean":"2024-03-06T07:27:06.509646+0000","last_became_active":"2024-03-06T07:27:05.517121+0000","last_became_peered":"2024-03-06T07:27:05.517121+0000","last_unstale":"2024-03-06T07:27:06.509646+0000","last_undegraded":"2024-03-06T07:27:06.509646+0000","last_fullsized":"2024-03-06T07:27:06.509646+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:14.941244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510058+0000","last_change":"2024-03-06T07:27:05.517655+0000","last_active":"2024-03-06T07:27:06.510058+0000","last_peered":"2024-03-06T07:27:06.510058+0000","last_clean":"2024-03-06T07:27:06.510058+0000","last_became_active":"2024-03-06T07:27:05.517390+0000","last_became_peered":"2024-03-06T07:27:05.517390+0000","last_unstale":"2024-03-06T07:27:06.510058+0000","last_undegraded":"2024-03-06T07:27:06.510058+0000","last_fullsized":"2024-03-06T07:27:06.510058+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:10.754527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708442+0000","last_change":"2024-03-06T07:27:05.520445+0000","last_active":"2024-03-06T07:27:06.708442+0000","last_peered":"2024-03-06T07:27:06.708442+0000","last_clean":"2024-03-06T07:27:06.708442+0000","last_became_active":"2024-03-06T07:27:05.519628+0000","last_became_peered":"2024-03-06T07:27:05.519628+0000","last_unstale":"2024-03-06T07:27:06.708442+0000","last_undegraded":"2024-03-06T07:27:06.708442+0000","last_fullsized":"2024-03-06T07:27:06.708442+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:11.264585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509047+0000","last_change":"2024-03-06T07:27:05.514630+0000","last_active":"2024-03-06T07:27:06.509047+0000","last_peered":"2024-03-06T07:27:06.509047+0000","last_clean":"2024-03-06T07:27:06.509047+0000","last_became_active":"2024-03-06T07:27:05.514326+0000","last_became_peered":"2024-03-06T07:27:05.514326+0000","last_unstale":"2024-03-06T07:27:06.509047+0000","last_undegraded":"2024-03-06T07:27:06.509047+0000","last_fullsized":"2024-03-06T07:27:06.509047+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:12.163357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708088+0000","last_change":"2024-03-06T07:27:05.520274+0000","last_active":"2024-03-06T07:27:06.708088+0000","last_peered":"2024-03-06T07:27:06.708088+0000","last_clean":"2024-03-06T07:27:06.708088+0000","last_became_active":"2024-03-06T07:27:05.519795+0000","last_became_peered":"2024-03-06T07:27:05.519795+0000","last_unstale":"2024-03-06T07:27:06.708088+0000","last_undegraded":"2024-03-06T07:27:06.708088+0000","last_fullsized":"2024-03-06T07:27:06.708088+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:50.417505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510111+0000","last_change":"2024-03-06T07:27:05.521312+0000","last_active":"2024-03-06T07:27:06.510111+0000","last_peered":"2024-03-06T07:27:06.510111+0000","last_clean":"2024-03-06T07:27:06.510111+0000","last_became_active":"2024-03-06T07:27:05.517145+0000","last_became_peered":"2024-03-06T07:27:05.517145+0000","last_unstale":"2024-03-06T07:27:06.510111+0000","last_undegraded":"2024-03-06T07:27:06.510111+0000","last_fullsized":"2024-03-06T07:27:06.510111+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:15.437041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508794+0000","last_change":"2024-03-06T07:27:05.525463+0000","last_active":"2024-03-06T07:27:06.508794+0000","last_peered":"2024-03-06T07:27:06.508794+0000","last_clean":"2024-03-06T07:27:06.508794+0000","last_became_active":"2024-03-06T07:27:05.522514+0000","last_became_peered":"2024-03-06T07:27:05.522514+0000","last_unstale":"2024-03-06T07:27:06.508794+0000","last_undegraded":"2024-03-06T07:27:06.508794+0000","last_fullsized":"2024-03-06T07:27:06.508794+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:41.987030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708263+0000","last_change":"2024-03-06T07:27:05.516401+0000","last_active":"2024-03-06T07:27:06.708263+0000","last_peered":"2024-03-06T07:27:06.708263+0000","last_clean":"2024-03-06T07:27:06.708263+0000","last_became_active":"2024-03-06T07:27:05.516119+0000","last_became_peered":"2024-03-06T07:27:05.516119+0000","last_unstale":"2024-03-06T07:27:06.708263+0000","last_undegraded":"2024-03-06T07:27:06.708263+0000","last_fullsized":"2024-03-06T07:27:06.708263+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:04.745211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708528+0000","last_change":"2024-03-06T07:27:05.518745+0000","last_active":"2024-03-06T07:27:06.708528+0000","last_peered":"2024-03-06T07:27:06.708528+0000","last_clean":"2024-03-06T07:27:06.708528+0000","last_became_active":"2024-03-06T07:27:05.517763+0000","last_became_peered":"2024-03-06T07:27:05.517763+0000","last_unstale":"2024-03-06T07:27:06.708528+0000","last_undegraded":"2024-03-06T07:27:06.708528+0000","last_fullsized":"2024-03-06T07:27:06.708528+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:25.199337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509377+0000","last_change":"2024-03-06T07:27:05.517165+0000","last_active":"2024-03-06T07:27:06.509377+0000","last_peered":"2024-03-06T07:27:06.509377+0000","last_clean":"2024-03-06T07:27:06.509377+0000","last_became_active":"2024-03-06T07:27:05.516858+0000","last_became_peered":"2024-03-06T07:27:05.516858+0000","last_unstale":"2024-03-06T07:27:06.509377+0000","last_undegraded":"2024-03-06T07:27:06.509377+0000","last_fullsized":"2024-03-06T07:27:06.509377+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:15.364800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515097+0000","last_change":"2024-03-06T07:27:05.518068+0000","last_active":"2024-03-06T07:27:06.515097+0000","last_peered":"2024-03-06T07:27:06.515097+0000","last_clean":"2024-03-06T07:27:06.515097+0000","last_became_active":"2024-03-06T07:27:05.517784+0000","last_became_peered":"2024-03-06T07:27:05.517784+0000","last_unstale":"2024-03-06T07:27:06.515097+0000","last_undegraded":"2024-03-06T07:27:06.515097+0000","last_fullsized":"2024-03-06T07:27:06.515097+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:45.977198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508442+0000","last_change":"2024-03-06T07:27:05.513283+0000","last_active":"2024-03-06T07:27:06.508442+0000","last_peered":"2024-03-06T07:27:06.508442+0000","last_clean":"2024-03-06T07:27:06.508442+0000","last_became_active":"2024-03-06T07:27:05.512982+0000","last_became_peered":"2024-03-06T07:27:05.512982+0000","last_unstale":"2024-03-06T07:27:06.508442+0000","last_undegraded":"2024-03-06T07:27:06.508442+0000","last_fullsized":"2024-03-06T07:27:06.508442+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:03.049668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708787+0000","last_change":"2024-03-06T07:27:05.509224+0000","last_active":"2024-03-06T07:27:06.708787+0000","last_peered":"2024-03-06T07:27:06.708787+0000","last_clean":"2024-03-06T07:27:06.708787+0000","last_became_active":"2024-03-06T07:27:05.508451+0000","last_became_peered":"2024-03-06T07:27:05.508451+0000","last_unstale":"2024-03-06T07:27:06.708787+0000","last_undegraded":"2024-03-06T07:27:06.708787+0000","last_fullsized":"2024-03-06T07:27:06.708787+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:19.304510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708403+0000","last_change":"2024-03-06T07:27:05.508024+0000","last_active":"2024-03-06T07:27:06.708403+0000","last_peered":"2024-03-06T07:27:06.708403+0000","last_clean":"2024-03-06T07:27:06.708403+0000","last_became_active":"2024-03-06T07:27:05.507735+0000","last_became_peered":"2024-03-06T07:27:05.507735+0000","last_unstale":"2024-03-06T07:27:06.708403+0000","last_undegraded":"2024-03-06T07:27:06.708403+0000","last_fullsized":"2024-03-06T07:27:06.708403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:37.031844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"106'4","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515183+0000","last_change":"2024-03-06T07:27:04.502116+0000","last_active":"2024-03-06T07:27:06.515183+0000","last_peered":"2024-03-06T07:27:06.515183+0000","last_clean":"2024-03-06T07:27:06.515183+0000","last_became_active":"2024-03-06T07:27:04.501823+0000","last_became_peered":"2024-03-06T07:27:04.501823+0000","last_unstale":"2024-03-06T07:27:06.515183+0000","last_undegraded":"2024-03-06T07:27:06.515183+0000","last_fullsized":"2024-03-06T07:27:06.515183+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:11.999443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510383+0000","last_change":"2024-03-06T07:27:05.511261+0000","last_active":"2024-03-06T07:27:06.510383+0000","last_peered":"2024-03-06T07:27:06.510383+0000","last_clean":"2024-03-06T07:27:06.510383+0000","last_became_active":"2024-03-06T07:27:05.510998+0000","last_became_peered":"2024-03-06T07:27:05.510998+0000","last_unstale":"2024-03-06T07:27:06.510383+0000","last_undegraded":"2024-03-06T07:27:06.510383+0000","last_fullsized":"2024-03-06T07:27:06.510383+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:14.941244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708228+0000","last_change":"2024-03-06T07:27:04.502792+0000","last_active":"2024-03-06T07:27:06.708228+0000","last_peered":"2024-03-06T07:27:06.708228+0000","last_clean":"2024-03-06T07:27:06.708228+0000","last_became_active":"2024-03-06T07:27:04.502507+0000","last_became_peered":"2024-03-06T07:27:04.502507+0000","last_unstale":"2024-03-06T07:27:06.708228+0000","last_undegraded":"2024-03-06T07:27:06.708228+0000","last_fullsized":"2024-03-06T07:27:06.708228+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:41.386524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.709062+0000","last_change":"2024-03-06T07:27:05.522334+0000","last_active":"2024-03-06T07:27:06.709062+0000","last_peered":"2024-03-06T07:27:06.709062+0000","last_clean":"2024-03-06T07:27:06.709062+0000","last_became_active":"2024-03-06T07:27:05.522086+0000","last_became_peered":"2024-03-06T07:27:05.522086+0000","last_unstale":"2024-03-06T07:27:06.709062+0000","last_undegraded":"2024-03-06T07:27:06.709062+0000","last_fullsized":"2024-03-06T07:27:06.709062+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:10.686400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708367+0000","last_change":"2024-03-06T07:27:04.507345+0000","last_active":"2024-03-06T07:27:06.708367+0000","last_peered":"2024-03-06T07:27:06.708367+0000","last_clean":"2024-03-06T07:27:06.708367+0000","last_became_active":"2024-03-06T07:27:04.507045+0000","last_became_peered":"2024-03-06T07:27:04.507045+0000","last_unstale":"2024-03-06T07:27:06.708367+0000","last_undegraded":"2024-03-06T07:27:06.708367+0000","last_fullsized":"2024-03-06T07:27:06.708367+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:16.209811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708579+0000","last_change":"2024-03-06T07:27:05.513084+0000","last_active":"2024-03-06T07:27:06.708579+0000","last_peered":"2024-03-06T07:27:06.708579+0000","last_clean":"2024-03-06T07:27:06.708579+0000","last_became_active":"2024-03-06T07:27:05.511910+0000","last_became_peered":"2024-03-06T07:27:05.511910+0000","last_unstale":"2024-03-06T07:27:06.708579+0000","last_undegraded":"2024-03-06T07:27:06.708579+0000","last_fullsized":"2024-03-06T07:27:06.708579+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:30.105296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707990+0000","last_change":"2024-03-06T07:27:04.505744+0000","last_active":"2024-03-06T07:27:06.707990+0000","last_peered":"2024-03-06T07:27:06.707990+0000","last_clean":"2024-03-06T07:27:06.707990+0000","last_became_active":"2024-03-06T07:27:04.505407+0000","last_became_peered":"2024-03-06T07:27:04.505407+0000","last_unstale":"2024-03-06T07:27:06.707990+0000","last_undegraded":"2024-03-06T07:27:06.707990+0000","last_fullsized":"2024-03-06T07:27:06.707990+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:35.997259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508397+0000","last_change":"2024-03-06T07:27:05.524970+0000","last_active":"2024-03-06T07:27:06.508397+0000","last_peered":"2024-03-06T07:27:06.508397+0000","last_clean":"2024-03-06T07:27:06.508397+0000","last_became_active":"2024-03-06T07:27:05.524697+0000","last_became_peered":"2024-03-06T07:27:05.524697+0000","last_unstale":"2024-03-06T07:27:06.508397+0000","last_undegraded":"2024-03-06T07:27:06.508397+0000","last_fullsized":"2024-03-06T07:27:06.508397+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:32.520587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515006+0000","last_change":"2024-03-06T07:27:04.504039+0000","last_active":"2024-03-06T07:27:06.515006+0000","last_peered":"2024-03-06T07:27:06.515006+0000","last_clean":"2024-03-06T07:27:06.515006+0000","last_became_active":"2024-03-06T07:27:04.503760+0000","last_became_peered":"2024-03-06T07:27:04.503760+0000","last_unstale":"2024-03-06T07:27:06.515006+0000","last_undegraded":"2024-03-06T07:27:06.515006+0000","last_fullsized":"2024-03-06T07:27:06.515006+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:09.677567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509596+0000","last_change":"2024-03-06T07:27:05.516797+0000","last_active":"2024-03-06T07:27:06.509596+0000","last_peered":"2024-03-06T07:27:06.509596+0000","last_clean":"2024-03-06T07:27:06.509596+0000","last_became_active":"2024-03-06T07:27:05.513702+0000","last_became_peered":"2024-03-06T07:27:05.513702+0000","last_unstale":"2024-03-06T07:27:06.509596+0000","last_undegraded":"2024-03-06T07:27:06.509596+0000","last_fullsized":"2024-03-06T07:27:06.509596+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:09.594835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708529+0000","last_change":"2024-03-06T07:27:04.508176+0000","last_active":"2024-03-06T07:27:06.708529+0000","last_peered":"2024-03-06T07:27:06.708529+0000","last_clean":"2024-03-06T07:27:06.708529+0000","last_became_active":"2024-03-06T07:27:04.507909+0000","last_became_peered":"2024-03-06T07:27:04.507909+0000","last_unstale":"2024-03-06T07:27:06.708529+0000","last_undegraded":"2024-03-06T07:27:06.708529+0000","last_fullsized":"2024-03-06T07:27:06.708529+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:09.745694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508382+0000","last_change":"2024-03-06T07:27:05.516805+0000","last_active":"2024-03-06T07:27:06.508382+0000","last_peered":"2024-03-06T07:27:06.508382+0000","last_clean":"2024-03-06T07:27:06.508382+0000","last_became_active":"2024-03-06T07:27:05.515620+0000","last_became_peered":"2024-03-06T07:27:05.515620+0000","last_unstale":"2024-03-06T07:27:06.508382+0000","last_undegraded":"2024-03-06T07:27:06.508382+0000","last_fullsized":"2024-03-06T07:27:06.508382+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:02.772037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508888+0000","last_change":"2024-03-06T07:27:04.511197+0000","last_active":"2024-03-06T07:27:06.508888+0000","last_peered":"2024-03-06T07:27:06.508888+0000","last_clean":"2024-03-06T07:27:06.508888+0000","last_became_active":"2024-03-06T07:27:04.510955+0000","last_became_peered":"2024-03-06T07:27:04.510955+0000","last_unstale":"2024-03-06T07:27:06.508888+0000","last_undegraded":"2024-03-06T07:27:06.508888+0000","last_fullsized":"2024-03-06T07:27:06.508888+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:47:25.044276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515166+0000","last_change":"2024-03-06T07:27:05.518317+0000","last_active":"2024-03-06T07:27:06.515166+0000","last_peered":"2024-03-06T07:27:06.515166+0000","last_clean":"2024-03-06T07:27:06.515166+0000","last_became_active":"2024-03-06T07:27:05.518067+0000","last_became_peered":"2024-03-06T07:27:05.518067+0000","last_unstale":"2024-03-06T07:27:06.515166+0000","last_undegraded":"2024-03-06T07:27:06.515166+0000","last_fullsized":"2024-03-06T07:27:06.515166+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:06.882188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508406+0000","last_change":"2024-03-06T07:27:04.510589+0000","last_active":"2024-03-06T07:27:06.508406+0000","last_peered":"2024-03-06T07:27:06.508406+0000","last_clean":"2024-03-06T07:27:06.508406+0000","last_became_active":"2024-03-06T07:27:04.510334+0000","last_became_peered":"2024-03-06T07:27:04.510334+0000","last_unstale":"2024-03-06T07:27:06.508406+0000","last_undegraded":"2024-03-06T07:27:06.508406+0000","last_fullsized":"2024-03-06T07:27:06.508406+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:33.736855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515251+0000","last_change":"2024-03-06T07:27:05.513812+0000","last_active":"2024-03-06T07:27:06.515251+0000","last_peered":"2024-03-06T07:27:06.515251+0000","last_clean":"2024-03-06T07:27:06.515251+0000","last_became_active":"2024-03-06T07:27:05.513549+0000","last_became_peered":"2024-03-06T07:27:05.513549+0000","last_unstale":"2024-03-06T07:27:06.515251+0000","last_undegraded":"2024-03-06T07:27:06.515251+0000","last_fullsized":"2024-03-06T07:27:06.515251+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:46.921413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708089+0000","last_change":"2024-03-06T07:27:04.507007+0000","last_active":"2024-03-06T07:27:06.708089+0000","last_peered":"2024-03-06T07:27:06.708089+0000","last_clean":"2024-03-06T07:27:06.708089+0000","last_became_active":"2024-03-06T07:27:04.506737+0000","last_became_peered":"2024-03-06T07:27:04.506737+0000","last_unstale":"2024-03-06T07:27:06.708089+0000","last_undegraded":"2024-03-06T07:27:06.708089+0000","last_fullsized":"2024-03-06T07:27:06.708089+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:06.025771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.510396+0000","last_change":"2024-03-06T07:27:05.514386+0000","last_active":"2024-03-06T07:27:06.510396+0000","last_peered":"2024-03-06T07:27:06.510396+0000","last_clean":"2024-03-06T07:27:06.510396+0000","last_became_active":"2024-03-06T07:27:05.514000+0000","last_became_peered":"2024-03-06T07:27:05.514000+0000","last_unstale":"2024-03-06T07:27:06.510396+0000","last_undegraded":"2024-03-06T07:27:06.510396+0000","last_fullsized":"2024-03-06T07:27:06.510396+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:09.236618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708837+0000","last_change":"2024-03-06T07:27:04.509676+0000","last_active":"2024-03-06T07:27:06.708837+0000","last_peered":"2024-03-06T07:27:06.708837+0000","last_clean":"2024-03-06T07:27:06.708837+0000","last_became_active":"2024-03-06T07:27:04.509404+0000","last_became_peered":"2024-03-06T07:27:04.509404+0000","last_unstale":"2024-03-06T07:27:06.708837+0000","last_undegraded":"2024-03-06T07:27:06.708837+0000","last_fullsized":"2024-03-06T07:27:06.708837+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:08.227784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708654+0000","last_change":"2024-03-06T07:27:05.516853+0000","last_active":"2024-03-06T07:27:06.708654+0000","last_peered":"2024-03-06T07:27:06.708654+0000","last_clean":"2024-03-06T07:27:06.708654+0000","last_became_active":"2024-03-06T07:27:05.516605+0000","last_became_peered":"2024-03-06T07:27:05.516605+0000","last_unstale":"2024-03-06T07:27:06.708654+0000","last_undegraded":"2024-03-06T07:27:06.708654+0000","last_fullsized":"2024-03-06T07:27:06.708654+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:48.505441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"106'3","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508596+0000","last_change":"2024-03-06T07:27:04.506613+0000","last_active":"2024-03-06T07:27:06.508596+0000","last_peered":"2024-03-06T07:27:06.508596+0000","last_clean":"2024-03-06T07:27:06.508596+0000","last_became_active":"2024-03-06T07:27:04.506356+0000","last_became_peered":"2024-03-06T07:27:04.506356+0000","last_unstale":"2024-03-06T07:27:06.508596+0000","last_undegraded":"2024-03-06T07:27:06.508596+0000","last_fullsized":"2024-03-06T07:27:06.508596+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:44.338877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515202+0000","last_change":"2024-03-06T07:27:05.514902+0000","last_active":"2024-03-06T07:27:06.515202+0000","last_peered":"2024-03-06T07:27:06.515202+0000","last_clean":"2024-03-06T07:27:06.515202+0000","last_became_active":"2024-03-06T07:27:05.514635+0000","last_became_peered":"2024-03-06T07:27:05.514635+0000","last_unstale":"2024-03-06T07:27:06.515202+0000","last_undegraded":"2024-03-06T07:27:06.515202+0000","last_fullsized":"2024-03-06T07:27:06.515202+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:40.357849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"106'6","reported_seq":20,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508934+0000","last_change":"2024-03-06T07:27:04.510935+0000","last_active":"2024-03-06T07:27:06.508934+0000","last_peered":"2024-03-06T07:27:06.508934+0000","last_clean":"2024-03-06T07:27:06.508934+0000","last_became_active":"2024-03-06T07:27:04.510640+0000","last_became_peered":"2024-03-06T07:27:04.510640+0000","last_unstale":"2024-03-06T07:27:06.508934+0000","last_undegraded":"2024-03-06T07:27:06.508934+0000","last_fullsized":"2024-03-06T07:27:06.508934+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:06.973235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708478+0000","last_change":"2024-03-06T07:27:05.512490+0000","last_active":"2024-03-06T07:27:06.708478+0000","last_peered":"2024-03-06T07:27:06.708478+0000","last_clean":"2024-03-06T07:27:06.708478+0000","last_became_active":"2024-03-06T07:27:05.512232+0000","last_became_peered":"2024-03-06T07:27:05.512232+0000","last_unstale":"2024-03-06T07:27:06.708478+0000","last_undegraded":"2024-03-06T07:27:06.708478+0000","last_fullsized":"2024-03-06T07:27:06.708478+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:13.805842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708936+0000","last_change":"2024-03-06T07:27:04.504658+0000","last_active":"2024-03-06T07:27:06.708936+0000","last_peered":"2024-03-06T07:27:06.708936+0000","last_clean":"2024-03-06T07:27:06.708936+0000","last_became_active":"2024-03-06T07:27:04.503982+0000","last_became_peered":"2024-03-06T07:27:04.503982+0000","last_unstale":"2024-03-06T07:27:06.708936+0000","last_undegraded":"2024-03-06T07:27:06.708936+0000","last_fullsized":"2024-03-06T07:27:06.708936+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.429984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708110+0000","last_change":"2024-03-06T07:27:05.519929+0000","last_active":"2024-03-06T07:27:06.708110+0000","last_peered":"2024-03-06T07:27:06.708110+0000","last_clean":"2024-03-06T07:27:06.708110+0000","last_became_active":"2024-03-06T07:27:05.519561+0000","last_became_peered":"2024-03-06T07:27:05.519561+0000","last_unstale":"2024-03-06T07:27:06.708110+0000","last_undegraded":"2024-03-06T07:27:06.708110+0000","last_fullsized":"2024-03-06T07:27:06.708110+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:30.105296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509432+0000","last_change":"2024-03-06T07:27:04.495580+0000","last_active":"2024-03-06T07:27:06.509432+0000","last_peered":"2024-03-06T07:27:06.509432+0000","last_clean":"2024-03-06T07:27:06.509432+0000","last_became_active":"2024-03-06T07:27:04.495226+0000","last_became_peered":"2024-03-06T07:27:04.495226+0000","last_unstale":"2024-03-06T07:27:06.509432+0000","last_undegraded":"2024-03-06T07:27:06.509432+0000","last_fullsized":"2024-03-06T07:27:06.509432+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:26.865440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707870+0000","last_change":"2024-03-06T07:27:05.512090+0000","last_active":"2024-03-06T07:27:06.707870+0000","last_peered":"2024-03-06T07:27:06.707870+0000","last_clean":"2024-03-06T07:27:06.707870+0000","last_became_active":"2024-03-06T07:27:05.511808+0000","last_became_peered":"2024-03-06T07:27:05.511808+0000","last_unstale":"2024-03-06T07:27:06.707870+0000","last_undegraded":"2024-03-06T07:27:06.707870+0000","last_fullsized":"2024-03-06T07:27:06.707870+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:07.982069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"106'8","reported_seq":22,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707735+0000","last_change":"2024-03-06T07:27:04.509819+0000","last_active":"2024-03-06T07:27:06.707735+0000","last_peered":"2024-03-06T07:27:06.707735+0000","last_clean":"2024-03-06T07:27:06.707735+0000","last_became_active":"2024-03-06T07:27:04.509471+0000","last_became_peered":"2024-03-06T07:27:04.509471+0000","last_unstale":"2024-03-06T07:27:06.707735+0000","last_undegraded":"2024-03-06T07:27:06.707735+0000","last_fullsized":"2024-03-06T07:27:06.707735+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:43.442246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708093+0000","last_change":"2024-03-06T07:27:05.515923+0000","last_active":"2024-03-06T07:27:06.708093+0000","last_peered":"2024-03-06T07:27:06.708093+0000","last_clean":"2024-03-06T07:27:06.708093+0000","last_became_active":"2024-03-06T07:27:05.515620+0000","last_became_peered":"2024-03-06T07:27:05.515620+0000","last_unstale":"2024-03-06T07:27:06.708093+0000","last_undegraded":"2024-03-06T07:27:06.708093+0000","last_fullsized":"2024-03-06T07:27:06.708093+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:41.721404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509107+0000","last_change":"2024-03-06T07:27:04.500475+0000","last_active":"2024-03-06T07:27:06.509107+0000","last_peered":"2024-03-06T07:27:06.509107+0000","last_clean":"2024-03-06T07:27:06.509107+0000","last_became_active":"2024-03-06T07:27:04.500179+0000","last_became_peered":"2024-03-06T07:27:04.500179+0000","last_unstale":"2024-03-06T07:27:06.509107+0000","last_undegraded":"2024-03-06T07:27:06.509107+0000","last_fullsized":"2024-03-06T07:27:06.509107+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:25.482014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708991+0000","last_change":"2024-03-06T07:27:05.524995+0000","last_active":"2024-03-06T07:27:06.708991+0000","last_peered":"2024-03-06T07:27:06.708991+0000","last_clean":"2024-03-06T07:27:06.708991+0000","last_became_active":"2024-03-06T07:27:05.524744+0000","last_became_peered":"2024-03-06T07:27:05.524744+0000","last_unstale":"2024-03-06T07:27:06.708991+0000","last_undegraded":"2024-03-06T07:27:06.708991+0000","last_fullsized":"2024-03-06T07:27:06.708991+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:00.299260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708189+0000","last_change":"2024-03-06T07:27:04.507777+0000","last_active":"2024-03-06T07:27:06.708189+0000","last_peered":"2024-03-06T07:27:06.708189+0000","last_clean":"2024-03-06T07:27:06.708189+0000","last_became_active":"2024-03-06T07:27:04.507525+0000","last_became_peered":"2024-03-06T07:27:04.507525+0000","last_unstale":"2024-03-06T07:27:06.708189+0000","last_undegraded":"2024-03-06T07:27:06.708189+0000","last_fullsized":"2024-03-06T07:27:06.708189+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:44.429984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708295+0000","last_change":"2024-03-06T07:27:05.516417+0000","last_active":"2024-03-06T07:27:06.708295+0000","last_peered":"2024-03-06T07:27:06.708295+0000","last_clean":"2024-03-06T07:27:06.708295+0000","last_became_active":"2024-03-06T07:27:05.516118+0000","last_became_peered":"2024-03-06T07:27:05.516118+0000","last_unstale":"2024-03-06T07:27:06.708295+0000","last_undegraded":"2024-03-06T07:27:06.708295+0000","last_fullsized":"2024-03-06T07:27:06.708295+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:27.252775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509768+0000","last_change":"2024-03-06T07:27:04.511779+0000","last_active":"2024-03-06T07:27:06.509768+0000","last_peered":"2024-03-06T07:27:06.509768+0000","last_clean":"2024-03-06T07:27:06.509768+0000","last_became_active":"2024-03-06T07:27:04.511485+0000","last_became_peered":"2024-03-06T07:27:04.511485+0000","last_unstale":"2024-03-06T07:27:06.509768+0000","last_undegraded":"2024-03-06T07:27:06.509768+0000","last_fullsized":"2024-03-06T07:27:06.509768+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:58.571185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708629+0000","last_change":"2024-03-06T07:27:05.516589+0000","last_active":"2024-03-06T07:27:06.708629+0000","last_peered":"2024-03-06T07:27:06.708629+0000","last_clean":"2024-03-06T07:27:06.708629+0000","last_became_active":"2024-03-06T07:27:05.516012+0000","last_became_peered":"2024-03-06T07:27:05.516012+0000","last_unstale":"2024-03-06T07:27:06.708629+0000","last_undegraded":"2024-03-06T07:27:06.708629+0000","last_fullsized":"2024-03-06T07:27:06.708629+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:11.264585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515038+0000","last_change":"2024-03-06T07:27:04.504834+0000","last_active":"2024-03-06T07:27:06.515038+0000","last_peered":"2024-03-06T07:27:06.515038+0000","last_clean":"2024-03-06T07:27:06.515038+0000","last_became_active":"2024-03-06T07:27:04.504370+0000","last_became_peered":"2024-03-06T07:27:04.504370+0000","last_unstale":"2024-03-06T07:27:06.515038+0000","last_undegraded":"2024-03-06T07:27:06.515038+0000","last_fullsized":"2024-03-06T07:27:06.515038+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:59.290427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708133+0000","last_change":"2024-03-06T07:27:05.510415+0000","last_active":"2024-03-06T07:27:06.708133+0000","last_peered":"2024-03-06T07:27:06.708133+0000","last_clean":"2024-03-06T07:27:06.708133+0000","last_became_active":"2024-03-06T07:27:05.510112+0000","last_became_peered":"2024-03-06T07:27:05.510112+0000","last_unstale":"2024-03-06T07:27:06.708133+0000","last_undegraded":"2024-03-06T07:27:06.708133+0000","last_fullsized":"2024-03-06T07:27:06.708133+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:54.039387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708991+0000","last_change":"2024-03-06T07:27:04.508545+0000","last_active":"2024-03-06T07:27:06.708991+0000","last_peered":"2024-03-06T07:27:06.708991+0000","last_clean":"2024-03-06T07:27:06.708991+0000","last_became_active":"2024-03-06T07:27:04.508279+0000","last_became_peered":"2024-03-06T07:27:04.508279+0000","last_unstale":"2024-03-06T07:27:06.708991+0000","last_undegraded":"2024-03-06T07:27:06.708991+0000","last_fullsized":"2024-03-06T07:27:06.708991+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:16.209811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508510+0000","last_change":"2024-03-06T07:27:05.513279+0000","last_active":"2024-03-06T07:27:06.508510+0000","last_peered":"2024-03-06T07:27:06.508510+0000","last_clean":"2024-03-06T07:27:06.508510+0000","last_became_active":"2024-03-06T07:27:05.513013+0000","last_became_peered":"2024-03-06T07:27:05.513013+0000","last_unstale":"2024-03-06T07:27:06.508510+0000","last_undegraded":"2024-03-06T07:27:06.508510+0000","last_fullsized":"2024-03-06T07:27:06.508510+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:22.699468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708582+0000","last_change":"2024-03-06T07:27:04.507769+0000","last_active":"2024-03-06T07:27:06.708582+0000","last_peered":"2024-03-06T07:27:06.708582+0000","last_clean":"2024-03-06T07:27:06.708582+0000","last_became_active":"2024-03-06T07:27:04.507455+0000","last_became_peered":"2024-03-06T07:27:04.507455+0000","last_unstale":"2024-03-06T07:27:06.708582+0000","last_undegraded":"2024-03-06T07:27:06.708582+0000","last_fullsized":"2024-03-06T07:27:06.708582+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:32.578061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708259+0000","last_change":"2024-03-06T07:27:05.512930+0000","last_active":"2024-03-06T07:27:06.708259+0000","last_peered":"2024-03-06T07:27:06.708259+0000","last_clean":"2024-03-06T07:27:06.708259+0000","last_became_active":"2024-03-06T07:27:05.512332+0000","last_became_peered":"2024-03-06T07:27:05.512332+0000","last_unstale":"2024-03-06T07:27:06.708259+0000","last_undegraded":"2024-03-06T07:27:06.708259+0000","last_fullsized":"2024-03-06T07:27:06.708259+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:26.490847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708777+0000","last_change":"2024-03-06T07:27:04.508385+0000","last_active":"2024-03-06T07:27:06.708777+0000","last_peered":"2024-03-06T07:27:06.708777+0000","last_clean":"2024-03-06T07:27:06.708777+0000","last_became_active":"2024-03-06T07:27:04.508006+0000","last_became_peered":"2024-03-06T07:27:04.508006+0000","last_unstale":"2024-03-06T07:27:06.708777+0000","last_undegraded":"2024-03-06T07:27:06.708777+0000","last_fullsized":"2024-03-06T07:27:06.708777+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:35.745169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707993+0000","last_change":"2024-03-06T07:27:05.518929+0000","last_active":"2024-03-06T07:27:06.707993+0000","last_peered":"2024-03-06T07:27:06.707993+0000","last_clean":"2024-03-06T07:27:06.707993+0000","last_became_active":"2024-03-06T07:27:05.517876+0000","last_became_peered":"2024-03-06T07:27:05.517876+0000","last_unstale":"2024-03-06T07:27:06.707993+0000","last_undegraded":"2024-03-06T07:27:06.707993+0000","last_fullsized":"2024-03-06T07:27:06.707993+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:03.124647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508552+0000","last_change":"2024-03-06T07:27:04.495315+0000","last_active":"2024-03-06T07:27:06.508552+0000","last_peered":"2024-03-06T07:27:06.508552+0000","last_clean":"2024-03-06T07:27:06.508552+0000","last_became_active":"2024-03-06T07:27:04.494327+0000","last_became_peered":"2024-03-06T07:27:04.494327+0000","last_unstale":"2024-03-06T07:27:06.508552+0000","last_undegraded":"2024-03-06T07:27:06.508552+0000","last_fullsized":"2024-03-06T07:27:06.508552+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:33.714081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509961+0000","last_change":"2024-03-06T07:27:05.515820+0000","last_active":"2024-03-06T07:27:06.509961+0000","last_peered":"2024-03-06T07:27:06.509961+0000","last_clean":"2024-03-06T07:27:06.509961+0000","last_became_active":"2024-03-06T07:27:05.515336+0000","last_became_peered":"2024-03-06T07:27:05.515336+0000","last_unstale":"2024-03-06T07:27:06.509961+0000","last_undegraded":"2024-03-06T07:27:06.509961+0000","last_fullsized":"2024-03-06T07:27:06.509961+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:15.364800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708295+0000","last_change":"2024-03-06T07:27:04.513054+0000","last_active":"2024-03-06T07:27:06.708295+0000","last_peered":"2024-03-06T07:27:06.708295+0000","last_clean":"2024-03-06T07:27:06.708295+0000","last_became_active":"2024-03-06T07:27:04.512612+0000","last_became_peered":"2024-03-06T07:27:04.512612+0000","last_unstale":"2024-03-06T07:27:06.708295+0000","last_undegraded":"2024-03-06T07:27:06.708295+0000","last_fullsized":"2024-03-06T07:27:06.708295+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:23.404575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509042+0000","last_change":"2024-03-06T07:27:05.511737+0000","last_active":"2024-03-06T07:27:06.509042+0000","last_peered":"2024-03-06T07:27:06.509042+0000","last_clean":"2024-03-06T07:27:06.509042+0000","last_became_active":"2024-03-06T07:27:05.511252+0000","last_became_peered":"2024-03-06T07:27:05.511252+0000","last_unstale":"2024-03-06T07:27:06.509042+0000","last_undegraded":"2024-03-06T07:27:06.509042+0000","last_fullsized":"2024-03-06T07:27:06.509042+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:43.413821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.514293+0000","last_change":"2024-03-06T07:27:04.499630+0000","last_active":"2024-03-06T07:27:06.514293+0000","last_peered":"2024-03-06T07:27:06.514293+0000","last_clean":"2024-03-06T07:27:06.514293+0000","last_became_active":"2024-03-06T07:27:04.499350+0000","last_became_peered":"2024-03-06T07:27:04.499350+0000","last_unstale":"2024-03-06T07:27:06.514293+0000","last_undegraded":"2024-03-06T07:27:06.514293+0000","last_fullsized":"2024-03-06T07:27:06.514293+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:40.717999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508915+0000","last_change":"2024-03-06T07:27:05.516038+0000","last_active":"2024-03-06T07:27:06.508915+0000","last_peered":"2024-03-06T07:27:06.508915+0000","last_clean":"2024-03-06T07:27:06.508915+0000","last_became_active":"2024-03-06T07:27:05.515697+0000","last_became_peered":"2024-03-06T07:27:05.515697+0000","last_unstale":"2024-03-06T07:27:06.508915+0000","last_undegraded":"2024-03-06T07:27:06.508915+0000","last_fullsized":"2024-03-06T07:27:06.508915+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:31.540460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508688+0000","last_change":"2024-03-06T07:27:04.494918+0000","last_active":"2024-03-06T07:27:06.508688+0000","last_peered":"2024-03-06T07:27:06.508688+0000","last_clean":"2024-03-06T07:27:06.508688+0000","last_became_active":"2024-03-06T07:27:04.494284+0000","last_became_peered":"2024-03-06T07:27:04.494284+0000","last_unstale":"2024-03-06T07:27:06.508688+0000","last_undegraded":"2024-03-06T07:27:06.508688+0000","last_fullsized":"2024-03-06T07:27:06.508688+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:26.243942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509639+0000","last_change":"2024-03-06T07:27:05.516906+0000","last_active":"2024-03-06T07:27:06.509639+0000","last_peered":"2024-03-06T07:27:06.509639+0000","last_clean":"2024-03-06T07:27:06.509639+0000","last_became_active":"2024-03-06T07:27:05.515925+0000","last_became_peered":"2024-03-06T07:27:05.515925+0000","last_unstale":"2024-03-06T07:27:06.509639+0000","last_undegraded":"2024-03-06T07:27:06.509639+0000","last_fullsized":"2024-03-06T07:27:06.509639+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:42.395357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708797+0000","last_change":"2024-03-06T07:27:04.509753+0000","last_active":"2024-03-06T07:27:06.708797+0000","last_peered":"2024-03-06T07:27:06.708797+0000","last_clean":"2024-03-06T07:27:06.708797+0000","last_became_active":"2024-03-06T07:27:04.509420+0000","last_became_peered":"2024-03-06T07:27:04.509420+0000","last_unstale":"2024-03-06T07:27:06.708797+0000","last_undegraded":"2024-03-06T07:27:06.708797+0000","last_fullsized":"2024-03-06T07:27:06.708797+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:14.428207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508910+0000","last_change":"2024-03-06T07:27:05.514877+0000","last_active":"2024-03-06T07:27:06.508910+0000","last_peered":"2024-03-06T07:27:06.508910+0000","last_clean":"2024-03-06T07:27:06.508910+0000","last_became_active":"2024-03-06T07:27:05.514589+0000","last_became_peered":"2024-03-06T07:27:05.514589+0000","last_unstale":"2024-03-06T07:27:06.508910+0000","last_undegraded":"2024-03-06T07:27:06.508910+0000","last_fullsized":"2024-03-06T07:27:06.508910+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:00.136435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515018+0000","last_change":"2024-03-06T07:27:04.503742+0000","last_active":"2024-03-06T07:27:06.515018+0000","last_peered":"2024-03-06T07:27:06.515018+0000","last_clean":"2024-03-06T07:27:06.515018+0000","last_became_active":"2024-03-06T07:27:04.503451+0000","last_became_peered":"2024-03-06T07:27:04.503451+0000","last_unstale":"2024-03-06T07:27:06.515018+0000","last_undegraded":"2024-03-06T07:27:06.515018+0000","last_fullsized":"2024-03-06T07:27:06.515018+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:28.564626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.509895+0000","last_change":"2024-03-06T07:27:05.516508+0000","last_active":"2024-03-06T07:27:06.509895+0000","last_peered":"2024-03-06T07:27:06.509895+0000","last_clean":"2024-03-06T07:27:06.509895+0000","last_became_active":"2024-03-06T07:27:05.514001+0000","last_became_peered":"2024-03-06T07:27:05.514001+0000","last_unstale":"2024-03-06T07:27:06.509895+0000","last_undegraded":"2024-03-06T07:27:06.509895+0000","last_fullsized":"2024-03-06T07:27:06.509895+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:24.413408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508979+0000","last_change":"2024-03-06T07:27:04.503200+0000","last_active":"2024-03-06T07:27:06.508979+0000","last_peered":"2024-03-06T07:27:06.508979+0000","last_clean":"2024-03-06T07:27:06.508979+0000","last_became_active":"2024-03-06T07:27:04.502934+0000","last_became_peered":"2024-03-06T07:27:04.502934+0000","last_unstale":"2024-03-06T07:27:06.508979+0000","last_undegraded":"2024-03-06T07:27:06.508979+0000","last_fullsized":"2024-03-06T07:27:06.508979+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:55.222283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708499+0000","last_change":"2024-03-06T07:27:05.519614+0000","last_active":"2024-03-06T07:27:06.708499+0000","last_peered":"2024-03-06T07:27:06.708499+0000","last_clean":"2024-03-06T07:27:06.708499+0000","last_became_active":"2024-03-06T07:27:05.519341+0000","last_became_peered":"2024-03-06T07:27:05.519341+0000","last_unstale":"2024-03-06T07:27:06.708499+0000","last_undegraded":"2024-03-06T07:27:06.708499+0000","last_fullsized":"2024-03-06T07:27:06.708499+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:50.417505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707798+0000","last_change":"2024-03-06T07:27:04.506431+0000","last_active":"2024-03-06T07:27:06.707798+0000","last_peered":"2024-03-06T07:27:06.707798+0000","last_clean":"2024-03-06T07:27:06.707798+0000","last_became_active":"2024-03-06T07:27:04.506072+0000","last_became_peered":"2024-03-06T07:27:04.506072+0000","last_unstale":"2024-03-06T07:27:06.707798+0000","last_undegraded":"2024-03-06T07:27:06.707798+0000","last_fullsized":"2024-03-06T07:27:06.707798+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:35.745169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508571+0000","last_change":"2024-03-06T07:27:05.518288+0000","last_active":"2024-03-06T07:27:06.508571+0000","last_peered":"2024-03-06T07:27:06.508571+0000","last_clean":"2024-03-06T07:27:06.508571+0000","last_became_active":"2024-03-06T07:27:05.518038+0000","last_became_peered":"2024-03-06T07:27:05.518038+0000","last_unstale":"2024-03-06T07:27:06.508571+0000","last_undegraded":"2024-03-06T07:27:06.508571+0000","last_fullsized":"2024-03-06T07:27:06.508571+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:38.789157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.508730+0000","last_change":"2024-03-06T07:27:04.495246+0000","last_active":"2024-03-06T07:27:06.508730+0000","last_peered":"2024-03-06T07:27:06.508730+0000","last_clean":"2024-03-06T07:27:06.508730+0000","last_became_active":"2024-03-06T07:27:04.494755+0000","last_became_peered":"2024-03-06T07:27:04.494755+0000","last_unstale":"2024-03-06T07:27:06.508730+0000","last_undegraded":"2024-03-06T07:27:06.508730+0000","last_fullsized":"2024-03-06T07:27:06.508730+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:53.030553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.707902+0000","last_change":"2024-03-06T07:27:05.520066+0000","last_active":"2024-03-06T07:27:06.707902+0000","last_peered":"2024-03-06T07:27:06.707902+0000","last_clean":"2024-03-06T07:27:06.707902+0000","last_became_active":"2024-03-06T07:27:05.519462+0000","last_became_peered":"2024-03-06T07:27:05.519462+0000","last_unstale":"2024-03-06T07:27:06.707902+0000","last_undegraded":"2024-03-06T07:27:06.707902+0000","last_fullsized":"2024-03-06T07:27:06.707902+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:04.485957+0000","last_clean_scrub_stamp":"2024-03-06T07:27:04.485957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:47.743962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.515205+0000","last_change":"2024-03-06T07:27:04.504761+0000","last_active":"2024-03-06T07:27:06.515205+0000","last_peered":"2024-03-06T07:27:06.515205+0000","last_clean":"2024-03-06T07:27:06.515205+0000","last_became_active":"2024-03-06T07:27:04.504443+0000","last_became_peered":"2024-03-06T07:27:04.504443+0000","last_unstale":"2024-03-06T07:27:06.515205+0000","last_undegraded":"2024-03-06T07:27:06.515205+0000","last_fullsized":"2024-03-06T07:27:06.515205+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:03.477123+0000","last_clean_scrub_stamp":"2024-03-06T07:27:03.477123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:20.512317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":322,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:06.708542+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:27:06.708542+0000","last_peered":"2024-03-06T07:27:06.708542+0000","last_clean":"2024-03-06T07:27:06.708542+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:27:06.708542+0000","last_undegraded":"2024-03-06T07:27:06.708542+0000","last_fullsized":"2024-03-06T07:27:06.708542+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1933312,"data_stored":1164269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28324,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343516,"statfs":{"total":96636764160,"available":96607760384,"internally_reserved":0,"allocated":1335296,"data_stored":573355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1339392,"data_stored":573889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28324,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343516,"statfs":{"total":96636764160,"available":96607760384,"internally_reserved":0,"allocated":1331200,"data_stored":573285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1347584,"data_stored":574967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":573285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1339392,"data_stored":573409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1945600,"data_stored":1165377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:09.518 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:09.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:09.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.661+0000 7f3556c50640 1 Processor -- start 2024-03-06T07:27:09.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.661+0000 7f3556c50640 1 -- start start 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.662+0000 7f3556c50640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f355015def0 0x7f35500b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f3556c50640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35500cea50 0x7f35500ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f3556c50640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355006d030 con 0x7f355006cab0 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f3556c50640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35500cf1c0 con 0x7f355015def0 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f3556c50640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35500cc760 con 0x7f35500cea50 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f355006cab0 0x7f35500ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38198/0 (socket says 172.21.15.77:38198) 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354ffff640 1 -- 172.21.15.77:0/60751232 learned_addr learned my addr 172.21.15.77:0/60751232 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f35549ec640 1 --2- 172.21.15.77:0/60751232 >> v2:172.21.15.158:3300/0 conn(0x7f355015def0 0x7f35500b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f35551ed640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35500cea50 0x7f35500ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033911094 0 0) 0x7f355006d030 con 0x7f355006cab0 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3540003520 con 0x7f355006cab0 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142378716 0 0) 0x7f3540003520 con 0x7f355006cab0 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35500cea50 msgr2=0x7f35500ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35500cea50 0x7f35500ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 >> v2:172.21.15.158:3300/0 conn(0x7f355015def0 msgr2=0x7f35500b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 --2- 172.21.15.77:0/60751232 >> v2:172.21.15.158:3300/0 conn(0x7f355015def0 0x7f35500b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355006d310 con 0x7f355006cab0 2024-03-06T07:27:09.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f353c001670 con 0x7f355006cab0 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 >> v1:172.21.15.77:6789/0 conn(0x7f355006cab0 legacy=0x7f35500ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35400039b0 0x7f3540003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35500cea50 0x7f3540004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35500cc760 con 0x7f3540004d40 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354000b9c0 con 0x7f35400039b0 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354000bb40 con 0x7f35500cea50 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.663+0000 7f35551ed640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f354ffff640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35500cea50 0x7f3540004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f35549ec640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35400039b0 0x7f3540003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f35551ed640 1 -- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35500cea50 msgr2=0x7f3540004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f35551ed640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35500cea50 0x7f3540004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f35551ed640 1 -- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35400039b0 msgr2=0x7f3540003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f35551ed640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35400039b0 0x7f3540003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f35551ed640 1 -- 172.21.15.77:0/60751232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3540003520 con 0x7f3540004d40 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f35549ec640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35400039b0 0x7f3540003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.664+0000 7f35551ed640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35400051c0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f35500729c0 tx=0x7f3540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c3318caac761ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548039070 con 0x7f3540004d40 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354802cb90 con 0x7f3540004d40 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f354f7fe640 1 -- 172.21.15.77:0/60751232 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548009c10 con 0x7f3540004d40 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f3556c50640 1 -- 172.21.15.77:0/60751232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 msgr2=0x7f35400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f3556c50640 1 --2- 172.21.15.77:0/60751232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35400051c0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f35500729c0 tx=0x7f3540005830 comp rx=0 tx=0).stop 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f35551ed640 1 -- 172.21.15.77:0/60751232 reap_dead start 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f3556c50640 1 -- 172.21.15.77:0/60751232 shutdown_connections 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f3556c50640 1 -- 172.21.15.77:0/60751232 >> 172.21.15.77:0/60751232 conn(0x7f35500b3690 msgr2=0x7f35500b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f3556c50640 1 -- 172.21.15.77:0/60751232 shutdown_connections 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.665+0000 7f3556c50640 1 -- 172.21.15.77:0/60751232 wait complete. 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.666+0000 7f3556c50640 1 Processor -- start 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.666+0000 7f3556c50640 1 -- start start 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f3556c50640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35500c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f3556c50640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3550151e30 0x7f35501f7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f3556c50640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35501522b0 0x7f35501f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f3556c50640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355015e2d0 con 0x7f3540004d40 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f3556c50640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35500cc3f0 con 0x7f35501522b0 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f3556c50640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355015e450 con 0x7f3550151e30 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f35549ec640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35500c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f35549ec640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35500c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36378/0 (socket says 172.21.15.77:36378) 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f35549ec640 1 -- 172.21.15.77:0/1493517497 learned_addr learned my addr 172.21.15.77:0/1493517497 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f354ffff640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3550151e30 0x7f35501f7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.667+0000 7f35551ed640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35501522b0 0x7f35501f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f35549ec640 1 -- 172.21.15.77:0/1493517497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3550151e30 msgr2=0x7f35501f7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f35549ec640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3550151e30 0x7f35501f7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f35549ec640 1 -- 172.21.15.77:0/1493517497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35501522b0 msgr2=0x7f35501f7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f35549ec640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35501522b0 0x7f35501f7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f35549ec640 1 -- 172.21.15.77:0/1493517497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500b49d0 con 0x7f3540004d40 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f35549ec640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35500c7750 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f35440116d0 tx=0x7f3544011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f354d7fa640 1 -- 172.21.15.77:0/1493517497 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354400f9c0 con 0x7f3540004d40 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f354d7fa640 1 -- 172.21.15.77:0/1493517497 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354400fb60 con 0x7f3540004d40 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f354d7fa640 1 -- 172.21.15.77:0/1493517497 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544015460 con 0x7f3540004d40 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35500b4cb0 con 0x7f3540004d40 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.668+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35500b5270 con 0x7f3540004d40 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.669+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355009fcf0 con 0x7f3540004d40 2024-03-06T07:27:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.669+0000 7f354d7fa640 1 -- 172.21.15.77:0/1493517497 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3544015600 con 0x7f3540004d40 2024-03-06T07:27:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.670+0000 7f354d7fa640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f353406b4d0 0x7f353406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.670+0000 7f354d7fa640 1 -- 172.21.15.77:0/1493517497 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f354408ac60 con 0x7f3540004d40 2024-03-06T07:27:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.670+0000 7f354ffff640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f353406b4d0 0x7f353406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.670+0000 7f354ffff640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f353406b4d0 0x7f353406d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f353c000fb0 tx=0x7f353c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.674+0000 7f354d7fa640 1 -- 172.21.15.77:0/1493517497 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3544019030 con 0x7f3540004d40 2024-03-06T07:27:09.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.912+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3550068ff0 con 0x7f3540004d40 2024-03-06T07:27:09.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.913+0000 7f354d7fa640 1 -- 172.21.15.77:0/1493517497 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3544059ae0 con 0x7f3540004d40 2024-03-06T07:27:09.915 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:27:09.915 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:27:09.915 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:27:09.916 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:27:09.916 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:09.916 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:27:09.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f353406b4d0 msgr2=0x7f353406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f353406b4d0 0x7f353406d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f353c000fb0 tx=0x7f353c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 msgr2=0x7f35500c7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35500c7750 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f35440116d0 tx=0x7f3544011b90 comp rx=0 tx=0).stop 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 shutdown_connections 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35501522b0 0x7f35501f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3550151e30 0x7f35501f7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f353406b4d0 0x7f353406d990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 --2- 172.21.15.77:0/1493517497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3540004d40 0x7f35500c7750 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.915+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 >> 172.21.15.77:0/1493517497 conn(0x7f35500b3690 msgr2=0x7f355006fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.916+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 shutdown_connections 2024-03-06T07:27:09.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:09.916+0000 7f3556c50640 1 -- 172.21.15.77:0/1493517497 wait complete. 2024-03-06T07:27:09.928 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.032+0000 7f9ae3913640 1 Processor -- start 2024-03-06T07:27:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.032+0000 7f9ae3913640 1 -- start start 2024-03-06T07:27:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae3913640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9adc15dfe0 0x7f9adc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae3913640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae3913640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc0b3430 con 0x7f9adc0cf830 2024-03-06T07:27:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae3913640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9adc0b31e0 con 0x7f9adc15dfe0 2024-03-06T07:27:10.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae3913640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0b3d60 con 0x7f9adc0d29c0 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae16af640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9adc15dfe0 0x7f9adc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae16af640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9adc15dfe0 0x7f9adc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36392/0 (socket says 172.21.15.77:36392) 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae16af640 1 -- 172.21.15.77:0/2535294302 learned_addr learned my addr 172.21.15.77:0/2535294302 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae1eb0640 1 --2- 172.21.15.77:0/2535294302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae16af640 1 -- 172.21.15.77:0/2535294302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 msgr2=0x7f9adc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae16af640 1 --2- 172.21.15.77:0/2535294302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae16af640 1 -- 172.21.15.77:0/2535294302 >> v1:172.21.15.158:6789/0 conn(0x7f9adc0cf830 legacy=0x7f9adc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae16af640 1 -- 172.21.15.77:0/2535294302 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc0b3710 con 0x7f9adc15dfe0 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.033+0000 7f9ae1eb0640 1 --2- 172.21.15.77:0/2535294302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:10.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ae16af640 1 --2- 172.21.15.77:0/2535294302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0cf2f0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009870 tx=0x7f9ad802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5080309fddf1476c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ad3fff640 1 -- 172.21.15.77:0/2535294302 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8034070 con 0x7f9adc15dfe0 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ad3fff640 1 -- 172.21.15.77:0/2535294302 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad8030840 con 0x7f9adc15dfe0 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ad3fff640 1 -- 172.21.15.77:0/2535294302 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8030c10 con 0x7f9adc15dfe0 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ae3913640 1 -- 172.21.15.77:0/2535294302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 msgr2=0x7f9adc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2535294302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0cf2f0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009870 tx=0x7f9ad802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ae3913640 1 -- 172.21.15.77:0/2535294302 shutdown_connections 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2535294302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2535294302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0cf2f0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.034+0000 7f9ae3913640 1 -- 172.21.15.77:0/2535294302 >> 172.21.15.77:0/2535294302 conn(0x7f9adc0d3cd0 msgr2=0x7f9adc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.035+0000 7f9ae3913640 1 -- 172.21.15.77:0/2535294302 shutdown_connections 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.035+0000 7f9ae3913640 1 -- 172.21.15.77:0/2535294302 wait complete. 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae3913640 1 Processor -- start 2024-03-06T07:27:10.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae3913640 1 -- start start 2024-03-06T07:27:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae3913640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9adc0cf830 0x7f9adc1f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae3913640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae3913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae3913640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0d1df0 con 0x7f9adc15dfe0 2024-03-06T07:27:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae3913640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0d1c70 con 0x7f9adc0cf830 2024-03-06T07:27:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae3913640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc15e3c0 con 0x7f9adc0d29c0 2024-03-06T07:27:10.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.036+0000 7f9ae1eb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae1eb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36406/0 (socket says 172.21.15.77:36406) 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae1eb0640 1 -- 172.21.15.77:0/2933223549 learned_addr learned my addr 172.21.15.77:0/2933223549 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae16af640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9adc0cf830 0x7f9adc1f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae0eae640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae1eb0640 1 -- 172.21.15.77:0/2933223549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 msgr2=0x7f9adc160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae1eb0640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae1eb0640 1 -- 172.21.15.77:0/2933223549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9adc0cf830 msgr2=0x7f9adc1f0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae1eb0640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9adc0cf830 0x7f9adc1f0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae1eb0640 1 -- 172.21.15.77:0/2933223549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc1f1520 con 0x7f9adc15dfe0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae1eb0640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0b88b0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400cef0 tx=0x7f9ad400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ad1ffb640 1 -- 172.21.15.77:0/2933223549 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad401a070 con 0x7f9adc15dfe0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ad1ffb640 1 -- 172.21.15.77:0/2933223549 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad400d860 con 0x7f9adc15dfe0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc1601f0 con 0x7f9adc15dfe0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ad1ffb640 1 -- 172.21.15.77:0/2933223549 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad400dc50 con 0x7f9adc15dfe0 2024-03-06T07:27:10.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.037+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc160790 con 0x7f9adc15dfe0 2024-03-06T07:27:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.038+0000 7f9ad1ffb640 1 -- 172.21.15.77:0/2933223549 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ad401d070 con 0x7f9adc15dfe0 2024-03-06T07:27:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.038+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc04f330 con 0x7f9adc15dfe0 2024-03-06T07:27:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.039+0000 7f9ad1ffb640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ac006b4d0 0x7f9ac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.039+0000 7f9ad1ffb640 1 -- 172.21.15.77:0/2933223549 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ad408a990 con 0x7f9adc15dfe0 2024-03-06T07:27:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.039+0000 7f9ae16af640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ac006b4d0 0x7f9ac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.039+0000 7f9ae16af640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ac006b4d0 0x7f9ac006d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8036000 tx=0x7f9ad803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.043+0000 7f9ad1ffb640 1 -- 172.21.15.77:0/2933223549 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ad4059310 con 0x7f9adc15dfe0 2024-03-06T07:27:10.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.258+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9adc0cab60 con 0x7f9adc15dfe0 2024-03-06T07:27:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.258+0000 7f9ad1ffb640 1 -- 172.21.15.77:0/2933223549 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f9ad405d3b0 con 0x7f9adc15dfe0 2024-03-06T07:27:10.260 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:10.260 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/582596329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":582596329},{"type":"v1","addr":"172.21.15.158:6834","nonce":582596329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3505002958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3505002958},{"type":"v1","addr":"172.21.15.158:6835","nonce":3505002958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1169947364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1169947364},{"type":"v1","addr":"172.21.15.77:6835","nonce":1169947364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:07.450568+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7827":{"gid":7827,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3247585937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3247585937},{"type":"v1","addr":"172.21.15.77:6837","nonce":3247585937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7827,"qdb_cluster":[7827]},"id":8}]} 2024-03-06T07:27:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 66 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ac006b4d0 msgr2=0x7f9ac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ac006b4d0 0x7f9ac006d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8036000 tx=0x7f9ad803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 msgr2=0x7f9adc0b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0b88b0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400cef0 tx=0x7f9ad400a710 comp rx=0 tx=0).stop 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 shutdown_connections 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ac006b4d0 0x7f9ac006d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc15dfe0 0x7f9adc0b88b0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9adc0d29c0 0x7f9adc160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 --2- 172.21.15.77:0/2933223549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9adc0cf830 0x7f9adc1f0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 >> 172.21.15.77:0/2933223549 conn(0x7f9adc0d3cd0 msgr2=0x7f9adc06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 shutdown_connections 2024-03-06T07:27:10.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.260+0000 7f9ae3913640 1 -- 172.21.15.77:0/2933223549 wait complete. 2024-03-06T07:27:10.676 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.849148+0000', 'modified': '2024-03-06T07:27:07.450568+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7827}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7827': {'gid': 7827, 'name': 'c', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/3247585937', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 3247585937}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 3247585937}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7827, 'qdb_cluster': [7827]} 2024-03-06T07:27:10.677 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:10.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:10.781 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-03-06T07:27:10.779+0000 7f8ee7814640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.782+0000 7fb9c6039640 1 Processor -- start 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.782+0000 7fb9c6039640 1 -- start start 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9c6039640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9c015e000 0x7fb9c00d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9c6039640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb9c00cf7d0 0x7fb9c00d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9c6039640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00d25e0 con 0x7fb9c00d4810 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9c6039640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c00c7c50 con 0x7fb9c015e000 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9c6039640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c00d2880 con 0x7fb9c00cf7d0 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9bf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9c015e000 0x7fb9c00d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9bf7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9c015e000 0x7fb9c00d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36412/0 (socket says 172.21.15.77:36412) 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9bf7fe640 1 -- 172.21.15.77:0/3454805302 learned_addr learned my addr 172.21.15.77:0/3454805302 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9bffff640 1 --2- 172.21.15.77:0/3454805302 >> v2:172.21.15.158:3300/0 conn(0x7fb9c00cf7d0 0x7fb9c00d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9bf7fe640 1 -- 172.21.15.77:0/3454805302 >> v2:172.21.15.158:3300/0 conn(0x7fb9c00cf7d0 msgr2=0x7fb9c00d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/3454805302 >> v2:172.21.15.158:3300/0 conn(0x7fb9c00cf7d0 0x7fb9c00d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9bf7fe640 1 -- 172.21.15.77:0/3454805302 >> v1:172.21.15.158:6789/0 conn(0x7fb9c00d4810 legacy=0x7fb9c00c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.783+0000 7fb9bf7fe640 1 -- 172.21.15.77:0/3454805302 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00d2b40 con 0x7fb9c015e000 2024-03-06T07:27:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.784+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/3454805302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c015e000 0x7fb9c00d0680 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fb9c008df20 tx=0x7fb9b002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1cadbe9732f75ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.784+0000 7fb9be7fc640 1 -- 172.21.15.77:0/3454805302 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b002cb20 con 0x7fb9c015e000 2024-03-06T07:27:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.784+0000 7fb9be7fc640 1 -- 172.21.15.77:0/3454805302 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b002ccc0 con 0x7fb9c015e000 2024-03-06T07:27:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.784+0000 7fb9be7fc640 1 -- 172.21.15.77:0/3454805302 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0034610 con 0x7fb9c015e000 2024-03-06T07:27:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.785+0000 7fb9c6039640 1 -- 172.21.15.77:0/3454805302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c015e000 msgr2=0x7fb9c00d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.785+0000 7fb9c6039640 1 --2- 172.21.15.77:0/3454805302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c015e000 0x7fb9c00d0680 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fb9c008df20 tx=0x7fb9b002bf80 comp rx=0 tx=0).stop 2024-03-06T07:27:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.785+0000 7fb9c6039640 1 -- 172.21.15.77:0/3454805302 shutdown_connections 2024-03-06T07:27:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.785+0000 7fb9c6039640 1 --2- 172.21.15.77:0/3454805302 >> v2:172.21.15.158:3300/0 conn(0x7fb9c00cf7d0 0x7fb9c00d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.785+0000 7fb9c6039640 1 --2- 172.21.15.77:0/3454805302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c015e000 0x7fb9c00d0680 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.785+0000 7fb9c6039640 1 -- 172.21.15.77:0/3454805302 >> 172.21.15.77:0/3454805302 conn(0x7fb9c00d3cf0 msgr2=0x7fb9c00d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.785+0000 7fb9c6039640 1 -- 172.21.15.77:0/3454805302 shutdown_connections 2024-03-06T07:27:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.785+0000 7fb9c6039640 1 -- 172.21.15.77:0/3454805302 wait complete. 2024-03-06T07:27:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.787+0000 7fb9c6039640 1 Processor -- start 2024-03-06T07:27:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.787+0000 7fb9c6039640 1 -- start start 2024-03-06T07:27:10.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9c6039640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9c00cf7d0 0x7fb9c015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9c6039640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c00d4810 0x7fb9c00cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9c6039640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9c015e000 0x7fb9c00c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9c6039640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c015e3e0 con 0x7fb9c00d4810 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9c6039640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00d22d0 con 0x7fb9c015e000 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9c6039640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00c87f0 con 0x7fb9c00cf7d0 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c00d4810 0x7fb9c00cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c00d4810 0x7fb9c00cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36420/0 (socket says 172.21.15.77:36420) 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9beffd640 1 -- 172.21.15.77:0/631309714 learned_addr learned my addr 172.21.15.77:0/631309714 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9c00cf7d0 0x7fb9c015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9bffff640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9c015e000 0x7fb9c00c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9beffd640 1 -- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9c00cf7d0 msgr2=0x7fb9c015ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9beffd640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9c00cf7d0 0x7fb9c015ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9beffd640 1 -- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9c015e000 msgr2=0x7fb9c00c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9beffd640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9c015e000 0x7fb9c00c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9beffd640 1 -- 172.21.15.77:0/631309714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c0158f80 con 0x7fb9c00d4810 2024-03-06T07:27:10.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9c00cf7d0 0x7fb9c015ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.788+0000 7fb9bffff640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9c015e000 0x7fb9c00c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.789+0000 7fb9beffd640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c00d4810 0x7fb9c00cd710 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c410 tx=0x7fb9b400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.789+0000 7fb9a7fff640 1 -- 172.21.15.77:0/631309714 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4017070 con 0x7fb9c00d4810 2024-03-06T07:27:10.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.789+0000 7fb9a7fff640 1 -- 172.21.15.77:0/631309714 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b40035f0 con 0x7fb9c00d4810 2024-03-06T07:27:10.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.789+0000 7fb9c6039640 1 -- 172.21.15.77:0/631309714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c0159260 con 0x7fb9c00d4810 2024-03-06T07:27:10.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.789+0000 7fb9a7fff640 1 -- 172.21.15.77:0/631309714 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4003a20 con 0x7fb9c00d4810 2024-03-06T07:27:10.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.789+0000 7fb9c6039640 1 -- 172.21.15.77:0/631309714 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c0159820 con 0x7fb9c00d4810 2024-03-06T07:27:10.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.790+0000 7fb9a7fff640 1 -- 172.21.15.77:0/631309714 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9b401a070 con 0x7fb9c00d4810 2024-03-06T07:27:10.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.791+0000 7fb9a7fff640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.791+0000 7fb9a7fff640 1 -- 172.21.15.77:0/631309714 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9b4089720 con 0x7fb9c00d4810 2024-03-06T07:27:10.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.791+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.791+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0035000 tx=0x7fb9b0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.793+0000 7fb9c8c27740 1 Processor -- start 2024-03-06T07:27:10.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.793+0000 7fb9c8c27740 1 -- start start 2024-03-06T07:27:10.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562222693870 0x562222693c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226978c0 0x562222699360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226c3600 0x5622226c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622225fd6b0 con 0x562222693870 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622225a6a60 con 0x5622226c3600 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5622225f9750 con 0x5622226978c0 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562222693870 0x562222693c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562222693870 0x562222693c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36422/0 (socket says 172.21.15.77:36422) 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9bf7fe640 1 -- 172.21.15.77:0/3409229558 learned_addr learned my addr 172.21.15.77:0/3409229558 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9bf7fe640 1 -- 172.21.15.77:0/3409229558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226978c0 msgr2=0x562222699360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/3409229558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226978c0 0x562222699360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9bf7fe640 1 -- 172.21.15.77:0/3409229558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226c3600 msgr2=0x5622226c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/3409229558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226c3600 0x5622226c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.794+0000 7fb9bf7fe640 1 -- 172.21.15.77:0/3409229558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622225c4f30 con 0x562222693870 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.795+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/3409229558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562222693870 0x562222693c50 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0000f80 tx=0x7fb9b002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14b8781a62fd71b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.795+0000 7fb9a57fa640 1 -- 172.21.15.77:0/3409229558 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b004e070 con 0x562222693870 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.795+0000 7fb9a57fa640 1 -- 172.21.15.77:0/3409229558 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b003ebc0 con 0x562222693870 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.795+0000 7fb9a57fa640 1 -- 172.21.15.77:0/3409229558 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00493f0 con 0x562222693870 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 -- 172.21.15.77:0/3409229558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562222693870 msgr2=0x562222693c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/3409229558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562222693870 0x562222693c50 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0000f80 tx=0x7fb9b002e000 comp rx=0 tx=0).stop 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 -- 172.21.15.77:0/3409229558 shutdown_connections 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/3409229558 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226c3600 0x5622226c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/3409229558 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226978c0 0x562222699360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/3409229558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562222693870 0x562222693c50 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 -- 172.21.15.77:0/3409229558 >> 172.21.15.77:0/3409229558 conn(0x5622226958f0 msgr2=0x5622226971b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 -- 172.21.15.77:0/3409229558 shutdown_connections 2024-03-06T07:27:10.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.796+0000 7fb9c8c27740 1 -- 172.21.15.77:0/3409229558 wait complete. 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.797+0000 7fb9c8c27740 1 -- start start 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.797+0000 7fb9c8c27740 1 Processor -- start 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226c3600 0x5622226bc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226b6d70 0x5622226c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562222697d50 con 0x5622226dbc50 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622225b9ea0 con 0x5622226c3600 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x562222698030 con 0x5622226b6d70 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9c8c27740 10 client.0 _ll_register_callbacks cb 0x5622225b9870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36436/0 (socket says 172.21.15.77:36436) 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9bffff640 1 -- 172.21.15.77:0/1164272978 learned_addr learned my addr 172.21.15.77:0/1164272978 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9bffff640 1 -- 172.21.15.77:0/1164272978 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226b6d70 msgr2=0x5622226c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9bffff640 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226b6d70 0x5622226c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9bffff640 1 -- 172.21.15.77:0/1164272978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226c3600 msgr2=0x5622226bc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9bffff640 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226c3600 0x5622226bc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.798+0000 7fb9bffff640 1 -- 172.21.15.77:0/1164272978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622225ba270 con 0x5622226dbc50 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9bffff640 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c2a10 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800e4d0 tx=0x7fb9b800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29510e926be3db87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb99e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb99e7fc640 1 -- 172.21.15.77:0/1164272978 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8018030 con 0x5622226dbc50 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb99e7fc640 1 -- 172.21.15.77:0/1164272978 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9b800f040 con 0x5622226dbc50 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 1 -- 172.21.15.77:0/1164272978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9b003ebc0 con 0x5622226dbc50 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb99e7fc640 1 -- 172.21.15.77:0/1164272978 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9b8021050 con 0x5622226dbc50 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 10 client.7956 fetch_fsmap learned FSMap version 66 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 1 -- 172.21.15.77:0/1164272978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5622223122d0 con 0x5622226dbc50 2024-03-06T07:27:10.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb99e7fc640 1 -- 172.21.15.77:0/1164272978 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b003ebc0 con 0x5622226dbc50 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb99e7fc640 1 -- 172.21.15.77:0/1164272978 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb9b8024020 con 0x5622226dbc50 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 10 client.7956 fetch_fsmap finished waiting for FSMap version 66 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 10 client.7956 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 20 client.7956 populate_metadata read hostname 'smithi077' 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] conn(0x5622226fbb60 0x5622226fdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 4 client.7956 mds_command: new command op to 7827 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9c8c27740 1 -- 172.21.15.77:0/1164272978 --> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56222260abf0 con 0x5622226fbb60 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] conn(0x5622226fbb60 0x5622226fdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb99e7fc640 10 client.7956 ms_handle_connect on v2:172.21.15.77:6836/3247585937 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.799+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] conn(0x5622226fbb60 0x5622226fdf50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d75eb93dba2fb354 server_cookie=e656d5b60479a704 in_seq=0 out_seq=0 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb99e7fc640 1 -- 172.21.15.77:0/1164272978 <== mds.0 v2:172.21.15.77:6836/3247585937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56222260abf0 con 0x5622226fbb60 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb99e7fc640 10 client.7956 handle_command_reply: tid=0 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 client.7956 shutdown 2024-03-06T07:27:10.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 -- 172.21.15.77:0/1164272978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 msgr2=0x5622226c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c2a10 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800e4d0 tx=0x7fb9b800e990 comp rx=0 tx=0).stop 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 -- 172.21.15.77:0/1164272978 shutdown_connections 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] conn(0x5622226fbb60 0x5622226fdf50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c2a10 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226b6d70 0x5622226c24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb99e7fc640 0 client.7956 ms_handle_reset on v2:172.21.15.77:6836/3247585937 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/1164272978 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226c3600 0x5622226bc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 -- 172.21.15.77:0/1164272978 >> 172.21.15.77:0/1164272978 conn(0x562222693870 msgr2=0x5622226c15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 -- 172.21.15.77:0/1164272978 shutdown_connections 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.800+0000 7fb9c8c27740 1 -- 172.21.15.77:0/1164272978 wait complete. 2024-03-06T07:27:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.801+0000 7fb9c8c27740 20 client.7956 trim_cache size 0 max 16384 2024-03-06T07:27:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.817+0000 7fb9c8c27740 1 Processor -- start 2024-03-06T07:27:10.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.817+0000 7fb9c8c27740 1 -- start start 2024-03-06T07:27:10.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226baf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 0x56222268ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c01fffb0 con 0x5622226dbc50 2024-03-06T07:27:10.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0150c30 con 0x5622226b6d70 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00cbad0 con 0x5622226c3600 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36444/0 (socket says 172.21.15.77:36444) 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bffff640 1 -- 172.21.15.77:0/854365344 learned_addr learned my addr 172.21.15.77:0/854365344 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226baf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9beffd640 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 0x56222268ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bffff640 1 -- 172.21.15.77:0/854365344 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 msgr2=0x56222268ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bffff640 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 0x56222268ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bffff640 1 -- 172.21.15.77:0/854365344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 msgr2=0x5622226baf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bffff640 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226baf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.818+0000 7fb9bffff640 1 -- 172.21.15.77:0/854365344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622223dc6b0 con 0x5622226dbc50 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9bffff640 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c6ac0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8013970 tx=0x7fb9b8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aed89dfb6388c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9a4ff9640 1 -- 172.21.15.77:0/854365344 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8018070 con 0x5622226dbc50 2024-03-06T07:27:10.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9a4ff9640 1 -- 172.21.15.77:0/854365344 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b8013c50 con 0x5622226dbc50 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9a4ff9640 1 -- 172.21.15.77:0/854365344 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8004070 con 0x5622226dbc50 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9c8c27740 1 -- 172.21.15.77:0/854365344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 msgr2=0x5622226c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c6ac0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8013970 tx=0x7fb9b8014000 comp rx=0 tx=0).stop 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9c8c27740 1 -- 172.21.15.77:0/854365344 shutdown_connections 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x5622226c6ac0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 0x56222268ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/854365344 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226baf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9c8c27740 1 -- 172.21.15.77:0/854365344 >> 172.21.15.77:0/854365344 conn(0x562222693870 msgr2=0x5622226ba800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.819+0000 7fb9c8c27740 1 -- 172.21.15.77:0/854365344 shutdown_connections 2024-03-06T07:27:10.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.820+0000 7fb9c8c27740 1 -- 172.21.15.77:0/854365344 wait complete. 2024-03-06T07:27:10.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.820+0000 7fb9c8c27740 1 -- start start 2024-03-06T07:27:10.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.820+0000 7fb9c8c27740 1 Processor -- start 2024-03-06T07:27:10.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 0x5622226c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9c8c27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x562222752de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622226bb9d0 con 0x5622226dbc50 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0150630 con 0x5622226b6d70 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9c8c27740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5622226c5c40 con 0x5622226c3600 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9c8c27740 10 client.0 _ll_register_callbacks cb 0x7fb9b800cce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x562222752de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x562222752de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36456/0 (socket says 172.21.15.77:36456) 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 -- 172.21.15.77:0/579359044 learned_addr learned my addr 172.21.15.77:0/579359044 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9beffd640 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 0x5622226c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 -- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 msgr2=0x5622226c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 0x5622226c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 -- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 msgr2=0x5622226c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 -- 172.21.15.77:0/579359044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622226ba0a0 con 0x5622226dbc50 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb9bffff640 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x562222752de0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8006ec0 tx=0x7fb9b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c98d7dd9aa6762a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.821+0000 7fb99e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb99e7fc640 1 -- 172.21.15.77:0/579359044 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8033070 con 0x5622226dbc50 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb99e7fc640 1 -- 172.21.15.77:0/579359044 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9b8003810 con 0x5622226dbc50 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 1 -- 172.21.15.77:0/579359044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9b8013c50 con 0x5622226dbc50 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb99e7fc640 1 -- 172.21.15.77:0/579359044 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9b803b050 con 0x5622226dbc50 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 10 client.7962 fetch_fsmap learned FSMap version 66 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 1 -- 172.21.15.77:0/579359044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x5622226bb010 con 0x5622226dbc50 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb99e7fc640 1 -- 172.21.15.77:0/579359044 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b8013c50 con 0x5622226dbc50 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb99e7fc640 1 -- 172.21.15.77:0/579359044 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb9b8036030 con 0x5622226dbc50 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 10 client.7962 fetch_fsmap finished waiting for FSMap version 66 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 10 client.7962 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 20 client.7962 populate_metadata read hostname 'smithi077' 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] conn(0x56222274f450 0x56222274f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 4 client.7962 mds_command: new command op to 7827 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9c8c27740 1 -- 172.21.15.77:0/579359044 --> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562222604f40 con 0x56222274f450 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] conn(0x56222274f450 0x56222274f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb9bf7fe640 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] conn(0x56222274f450 0x56222274f830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a56524fa912b96c1 server_cookie=1e494d6d43cfcadf in_seq=0 out_seq=0 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.822+0000 7fb99e7fc640 10 client.7962 ms_handle_connect on v2:172.21.15.77:6836/3247585937 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb99e7fc640 1 -- 172.21.15.77:0/579359044 <== mds.0 v2:172.21.15.77:6836/3247585937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x562222604f40 con 0x56222274f450 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb99e7fc640 10 client.7962 handle_command_reply: tid=0 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 client.7962 shutdown 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 -- 172.21.15.77:0/579359044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 msgr2=0x562222752de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x562222752de0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8006ec0 tx=0x7fb9b8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 -- 172.21.15.77:0/579359044 shutdown_connections 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.77:6836/3247585937,v1:172.21.15.77:6837/3247585937] conn(0x56222274f450 0x56222274f830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622226dbc50 0x562222752de0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5622226c3600 0x5622226c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb99e7fc640 0 client.7962 ms_handle_reset on v2:172.21.15.77:6836/3247585937 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 --2- 172.21.15.77:0/579359044 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5622226b6d70 0x5622226c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 -- 172.21.15.77:0/579359044 >> 172.21.15.77:0/579359044 conn(0x562222693870 msgr2=0x5622226c39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 -- 172.21.15.77:0/579359044 shutdown_connections 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 1 -- 172.21.15.77:0/579359044 wait complete. 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.823+0000 7fb9c8c27740 20 client.7962 trim_cache size 0 max 16384 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:27:10.831 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 7827, 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 5.3179016140000002, 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 66, 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 107, 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 107, 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 10.368621228 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:27:10.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 -- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9ac06b500 msgr2=0x7fb9ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0035000 tx=0x7fb9b0030000 comp rx=0 tx=0).stop 2024-03-06T07:27:10.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 -- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c00d4810 msgr2=0x7fb9c00cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c00d4810 0x7fb9c00cd710 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c410 tx=0x7fb9b400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 -- 172.21.15.77:0/631309714 shutdown_connections 2024-03-06T07:27:10.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9ac06b500 0x7fb9ac06d9c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9c00d4810 0x7fb9c00cd710 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9c015e000 0x7fb9c00c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 --2- 172.21.15.77:0/631309714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9c00cf7d0 0x7fb9c015ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 -- 172.21.15.77:0/631309714 >> 172.21.15.77:0/631309714 conn(0x7fb9c00d3cf0 msgr2=0x7fb9c00b8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 -- 172.21.15.77:0/631309714 shutdown_connections 2024-03-06T07:27:10.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.824+0000 7fb9c6039640 1 -- 172.21.15.77:0/631309714 wait complete. 2024-03-06T07:27:10.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.939+0000 7f356740e640 1 Processor -- start 2024-03-06T07:27:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.939+0000 7f356740e640 1 -- start start 2024-03-06T07:27:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f356740e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f356015dfe0 0x7f35600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f356740e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35600cf830 0x7f35600d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f356740e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35600b3430 con 0x7f35600d27f0 2024-03-06T07:27:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f356740e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35600d2c50 con 0x7f356015dfe0 2024-03-06T07:27:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f356740e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35600b31e0 con 0x7f35600cf830 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35651aa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f356015dfe0 0x7f35600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35659ab640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f35600d27f0 0x7f35600d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38200/0 (socket says 172.21.15.77:38200) 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35659ab640 1 -- 172.21.15.77:0/4033730126 learned_addr learned my addr 172.21.15.77:0/4033730126 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35649a9640 1 --2- 172.21.15.77:0/4033730126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35600cf830 0x7f35600d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f3557fff640 1 -- 172.21.15.77:0/4033730126 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904653405 0 0) 0x7f35600b3430 con 0x7f35600d27f0 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f3557fff640 1 -- 172.21.15.77:0/4033730126 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f354c003520 con 0x7f35600d27f0 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35651aa640 1 -- 172.21.15.77:0/4033730126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35600cf830 msgr2=0x7f35600d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35651aa640 1 --2- 172.21.15.77:0/4033730126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35600cf830 0x7f35600d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35651aa640 1 -- 172.21.15.77:0/4033730126 >> v1:172.21.15.77:6789/0 conn(0x7f35600d27f0 legacy=0x7f35600d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35651aa640 1 -- 172.21.15.77:0/4033730126 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35600b3710 con 0x7f356015dfe0 2024-03-06T07:27:10.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.940+0000 7f35649a9640 1 --2- 172.21.15.77:0/4033730126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35600cf830 0x7f35600d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.941+0000 7f35651aa640 1 --2- 172.21.15.77:0/4033730126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f356015dfe0 0x7f35600cf2f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f355c0097a0 tx=0x7f355c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afcf274bd3158c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.941+0000 7f3557fff640 1 -- 172.21.15.77:0/4033730126 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c035020 con 0x7f356015dfe0 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.941+0000 7f3557fff640 1 -- 172.21.15.77:0/4033730126 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c031830 con 0x7f356015dfe0 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.941+0000 7f3557fff640 1 -- 172.21.15.77:0/4033730126 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c031c00 con 0x7f356015dfe0 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.942+0000 7f356740e640 1 -- 172.21.15.77:0/4033730126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f356015dfe0 msgr2=0x7f35600cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.942+0000 7f356740e640 1 --2- 172.21.15.77:0/4033730126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f356015dfe0 0x7f35600cf2f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f355c0097a0 tx=0x7f355c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.942+0000 7f356740e640 1 -- 172.21.15.77:0/4033730126 shutdown_connections 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.942+0000 7f356740e640 1 --2- 172.21.15.77:0/4033730126 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f35600cf830 0x7f35600d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.942+0000 7f356740e640 1 --2- 172.21.15.77:0/4033730126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f356015dfe0 0x7f35600cf2f0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.942+0000 7f356740e640 1 -- 172.21.15.77:0/4033730126 >> 172.21.15.77:0/4033730126 conn(0x7f35600d3cd0 msgr2=0x7f35600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.942+0000 7f356740e640 1 -- 172.21.15.77:0/4033730126 shutdown_connections 2024-03-06T07:27:10.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.942+0000 7f356740e640 1 -- 172.21.15.77:0/4033730126 wait complete. 2024-03-06T07:27:10.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.943+0000 7f356740e640 1 Processor -- start 2024-03-06T07:27:10.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.943+0000 7f356740e640 1 -- start start 2024-03-06T07:27:10.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f356740e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35600cf830 0x7f35600b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f356740e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35600d27f0 0x7f35600b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f356740e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f356015dfe0 0x7f35600b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f356740e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35600d1c70 con 0x7f35600d27f0 2024-03-06T07:27:10.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f356740e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35600b3d60 con 0x7f35600cf830 2024-03-06T07:27:10.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f356740e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35600d1df0 con 0x7f356015dfe0 2024-03-06T07:27:10.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35649a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35600d27f0 0x7f35600b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35649a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35600d27f0 0x7f35600b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36474/0 (socket says 172.21.15.77:36474) 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35649a9640 1 -- 172.21.15.77:0/937973862 learned_addr learned my addr 172.21.15.77:0/937973862 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35649a9640 1 -- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f356015dfe0 msgr2=0x7f35600b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35659ab640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f356015dfe0 0x7f35600b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35651aa640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35600cf830 0x7f35600b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35649a9640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f356015dfe0 0x7f35600b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35649a9640 1 -- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35600cf830 msgr2=0x7f35600b9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35649a9640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35600cf830 0x7f35600b9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35649a9640 1 -- 172.21.15.77:0/937973862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35600b7040 con 0x7f35600d27f0 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35659ab640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f356015dfe0 0x7f35600b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.944+0000 7f35651aa640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35600cf830 0x7f35600b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.945+0000 7f35649a9640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35600d27f0 0x7f35600b1f30 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f355000e840 tx=0x7f355000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.945+0000 7f3555ffb640 1 -- 172.21.15.77:0/937973862 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355000f040 con 0x7f35600d27f0 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.945+0000 7f3555ffb640 1 -- 172.21.15.77:0/937973862 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3550004030 con 0x7f35600d27f0 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.945+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356015a8a0 con 0x7f35600d27f0 2024-03-06T07:27:10.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.945+0000 7f3555ffb640 1 -- 172.21.15.77:0/937973862 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3550004400 con 0x7f35600d27f0 2024-03-06T07:27:10.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.945+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35600b6d60 con 0x7f35600d27f0 2024-03-06T07:27:10.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.946+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356004f330 con 0x7f35600d27f0 2024-03-06T07:27:10.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.946+0000 7f3555ffb640 1 -- 172.21.15.77:0/937973862 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f355000c6d0 con 0x7f35600d27f0 2024-03-06T07:27:10.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.947+0000 7f3555ffb640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f354406b4d0 0x7f354406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.947+0000 7f3555ffb640 1 -- 172.21.15.77:0/937973862 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3550089ec0 con 0x7f35600d27f0 2024-03-06T07:27:10.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.947+0000 7f35651aa640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f354406b4d0 0x7f354406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.947+0000 7f35651aa640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f354406b4d0 0x7f354406d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f355c036000 tx=0x7f355c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:10.951+0000 7f3555ffb640 1 -- 172.21.15.77:0/937973862 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f356015a8a0 con 0x7f35600d27f0 2024-03-06T07:27:11.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.166+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f35600b09c0 con 0x7f35600d27f0 2024-03-06T07:27:11.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.167+0000 7f3555ffb640 1 -- 172.21.15.77:0/937973862 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f35600b7040 con 0x7f35600d27f0 2024-03-06T07:27:11.169 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f354406b4d0 msgr2=0x7f354406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f354406b4d0 0x7f354406d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f355c036000 tx=0x7f355c008030 comp rx=0 tx=0).stop 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35600d27f0 msgr2=0x7f35600b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35600d27f0 0x7f35600b1f30 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f355000e840 tx=0x7f355000ed00 comp rx=0 tx=0).stop 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 shutdown_connections 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f354406b4d0 0x7f354406d990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35600d27f0 0x7f35600b1f30 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f356015dfe0 0x7f35600b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 --2- 172.21.15.77:0/937973862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f35600cf830 0x7f35600b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 >> 172.21.15.77:0/937973862 conn(0x7f35600d3cd0 msgr2=0x7f35600d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 shutdown_connections 2024-03-06T07:27:11.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:11.169+0000 7f356740e640 1 -- 172.21.15.77:0/937973862 wait complete. 2024-03-06T07:27:11.181 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:11.181 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:11.182 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:27:11.182 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:11.182 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:11.182 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:11.182 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:11.182 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:11.182 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:11.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:11.182 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:27:11.197 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:11.197 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:11.197 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:11.197 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:11.197 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:11.197 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42366sec preferred_lft 42366sec 2024-03-06T07:27:11.198 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:27:11.199 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:11.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:11.200 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:11.262 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:27:11.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:11.264 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:27:11.320 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:27:11.320 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:11.320 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:11.320 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:11.320 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:11.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:11.321 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:27:11.321 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:27:11.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:11.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:11.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:11.321 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:27:11.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:11.417 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:27:11.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:11.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:11.432 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:11.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:11.432 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:27:11.499 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:11.499 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:11.499 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:11.555 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:11.555 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:11.555 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:11.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:27:11.678 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:27:11.733 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:11.735 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:27:11.735 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:11.735 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:11.735 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:11.735 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:11.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:11.735 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:27:11.786 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:11.786 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:11.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:27:11.855 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:27:11.911 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:11.912 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:11.913 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:11.913 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:11.913 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:27:11.913 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:11.913 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:11.913 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:11.913 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:11.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:11.913 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:27:11.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:27:11.923+0000 7fa331a3d480 -1 init, newargv = 0x562ef59e8540 newargc=15 2024-03-06T07:27:11.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[72506]: starting ceph client 2024-03-06T07:27:11.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[72506]: starting fuse 2024-03-06T07:27:11.939 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:27:11.940 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:27:11.940 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:11.941 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:11.942 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:27:12.060 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.72506.asok 2024-03-06T07:27:12.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:12.071 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72506.asok 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72506.asok status 2024-03-06T07:27:12.211 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "72506", 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:27:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 7974, 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 7974 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1836207599 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1836207599 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.7974 192.168.0.1:0/1836207599", 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/1836207599", 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 66, 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 107, 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:27:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:27:12.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.221 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:27:12.221 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:12.235 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:27:12.236 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:12.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:12.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.303 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:27:12.362 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:27:12.362 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.417 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:27:12.533 INFO:teuthology.orchestra.run.smithi077.stderr:8+0 records in 2024-03-06T07:27:12.533 INFO:teuthology.orchestra.run.smithi077.stderr:8+0 records out 2024-03-06T07:27:12.533 INFO:teuthology.orchestra.run.smithi077.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.058876 s, 142 MB/s 2024-03-06T07:27:12.535 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:27:12.535 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:27:12.535 DEBUG:teuthology.orchestra.run.smithi077:> import stat 2024-03-06T07:27:12.535 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:12.535 DEBUG:teuthology.orchestra.run.smithi077:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:27:12.535 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:27:12.619 INFO:teuthology.orchestra.run.smithi077.stdout:1099511627777 2024-03-06T07:27:12.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:12.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.630 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:27:12.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.688 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:27:12.747 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:27:12.747 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:27:12.747 DEBUG:teuthology.orchestra.run.smithi077:> import stat 2024-03-06T07:27:12.747 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:12.747 DEBUG:teuthology.orchestra.run.smithi077:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:27:12.747 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:27:12.853 INFO:teuthology.orchestra.run.smithi077.stdout:1099511627777 2024-03-06T07:27:12.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:12.861 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:13.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.005+0000 7fd3e596b640 1 Processor -- start 2024-03-06T07:27:13.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.005+0000 7fd3e596b640 1 -- start start 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.006+0000 7fd3e596b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd3e00b4170 0x7fd3e00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.006+0000 7fd3e596b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3e00d4760 0x7fd3e00d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.006+0000 7fd3e596b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e00d25d0 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.006+0000 7fd3e596b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e00d4bf0 con 0x7fd3e00d4760 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.006+0000 7fd3e596b640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e00ce9e0 con 0x7fd3e00b4170 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.006+0000 7fd3de7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3e00d4760 0x7fd3e00d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.006+0000 7fd3de7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3e00d4760 0x7fd3e00d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36486/0 (socket says 172.21.15.77:36486) 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.006+0000 7fd3de7fc640 1 -- 172.21.15.77:0/3021255851 learned_addr learned my addr 172.21.15.77:0/3021255851 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.007+0000 7fd3deffd640 1 --2- 172.21.15.77:0/3021255851 >> v2:172.21.15.158:3300/0 conn(0x7fd3e00b4170 0x7fd3e00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.007+0000 7fd3de7fc640 1 -- 172.21.15.77:0/3021255851 >> v2:172.21.15.158:3300/0 conn(0x7fd3e00b4170 msgr2=0x7fd3e00b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.007+0000 7fd3de7fc640 1 --2- 172.21.15.77:0/3021255851 >> v2:172.21.15.158:3300/0 conn(0x7fd3e00b4170 0x7fd3e00b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.007+0000 7fd3de7fc640 1 -- 172.21.15.77:0/3021255851 >> v1:172.21.15.158:6789/0 conn(0x7fd3e00b4dd0 legacy=0x7fd3e00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.007+0000 7fd3de7fc640 1 -- 172.21.15.77:0/3021255851 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00d2870 con 0x7fd3e00d4760 2024-03-06T07:27:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.007+0000 7fd3deffd640 1 --2- 172.21.15.77:0/3021255851 >> v2:172.21.15.158:3300/0 conn(0x7fd3e00b4170 0x7fd3e00b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3de7fc640 1 --2- 172.21.15.77:0/3021255851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00d4760 0x7fd3e00d0370 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd3e006e030 tx=0x7fd3d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86525758f682200d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3ddffb640 1 -- 172.21.15.77:0/3021255851 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4035020 con 0x7fd3e00d4760 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3ddffb640 1 -- 172.21.15.77:0/3021255851 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d4031870 con 0x7fd3e00d4760 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3ddffb640 1 -- 172.21.15.77:0/3021255851 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4031c40 con 0x7fd3e00d4760 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3e596b640 1 -- 172.21.15.77:0/3021255851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00d4760 msgr2=0x7fd3e00d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3e596b640 1 --2- 172.21.15.77:0/3021255851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00d4760 0x7fd3e00d0370 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd3e006e030 tx=0x7fd3d402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3e596b640 1 -- 172.21.15.77:0/3021255851 shutdown_connections 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3e596b640 1 --2- 172.21.15.77:0/3021255851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00d4760 0x7fd3e00d0370 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.008+0000 7fd3e596b640 1 --2- 172.21.15.77:0/3021255851 >> v2:172.21.15.158:3300/0 conn(0x7fd3e00b4170 0x7fd3e00b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.009+0000 7fd3e596b640 1 -- 172.21.15.77:0/3021255851 >> 172.21.15.77:0/3021255851 conn(0x7fd3e00d3c40 msgr2=0x7fd3e00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.009+0000 7fd3e596b640 1 -- 172.21.15.77:0/3021255851 shutdown_connections 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.009+0000 7fd3e596b640 1 -- 172.21.15.77:0/3021255851 wait complete. 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.010+0000 7fd3e596b640 1 Processor -- start 2024-03-06T07:27:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.010+0000 7fd3e596b640 1 -- start start 2024-03-06T07:27:13.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3e596b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e00b4170 0x7fd3e014a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3e596b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00b4dd0 0x7fd3e00c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3e596b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3e014ae70 0x7fd3e00c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3e596b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00b47d0 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3e596b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00b4550 con 0x7fd3e00b4170 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3e596b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00b4950 con 0x7fd3e014ae70 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00b4dd0 0x7fd3e00c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00b4dd0 0x7fd3e00c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36492/0 (socket says 172.21.15.77:36492) 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 -- 172.21.15.77:0/895126498 learned_addr learned my addr 172.21.15.77:0/895126498 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3deffd640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e00b4170 0x7fd3e014a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3df7fe640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3e014ae70 0x7fd3e00c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 -- 172.21.15.77:0/895126498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3e014ae70 msgr2=0x7fd3e00c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3e014ae70 0x7fd3e00c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 -- 172.21.15.77:0/895126498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e00b4170 msgr2=0x7fd3e014a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e00b4170 0x7fd3e014a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 -- 172.21.15.77:0/895126498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00b2bf0 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3de7fc640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00b4dd0 0x7fd3e00c76a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4000c00 tx=0x7fd3d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3c7fff640 1 -- 172.21.15.77:0/895126498 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4040070 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3c7fff640 1 -- 172.21.15.77:0/895126498 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d403a420 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e00b2ed0 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3c7fff640 1 -- 172.21.15.77:0/895126498 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4035040 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.011+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e00b3490 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.012+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e004f330 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.013+0000 7fd3c7fff640 1 -- 172.21.15.77:0/895126498 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3d4038020 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.013+0000 7fd3c7fff640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3cc06b500 0x7fd3cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.013+0000 7fd3c7fff640 1 -- 172.21.15.77:0/895126498 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3d40ac110 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.013+0000 7fd3deffd640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3cc06b500 0x7fd3cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.014+0000 7fd3deffd640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3cc06b500 0x7fd3cc06d9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000a990 tx=0x7fd3d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.018+0000 7fd3c7fff640 1 -- 172.21.15.77:0/895126498 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3d407b020 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.225+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3e00733b0 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.225+0000 7fd3c7fff640 1 -- 172.21.15.77:0/895126498 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd3d407a920 con 0x7fd3e00b4dd0 2024-03-06T07:27:13.227 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:13.227 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/582596329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":582596329},{"type":"v1","addr":"172.21.15.158:6834","nonce":582596329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3505002958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3505002958},{"type":"v1","addr":"172.21.15.158:6835","nonce":3505002958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1169947364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1169947364},{"type":"v1","addr":"172.21.15.77:6835","nonce":1169947364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:07.450568+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7827":{"gid":7827,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3247585937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3247585937},{"type":"v1","addr":"172.21.15.77:6837","nonce":3247585937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7827,"qdb_cluster":[7827]},"id":8}]} 2024-03-06T07:27:13.228 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 66 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3cc06b500 msgr2=0x7fd3cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3cc06b500 0x7fd3cc06d9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000a990 tx=0x7fd3d0008040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00b4dd0 msgr2=0x7fd3e00c76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00b4dd0 0x7fd3e00c76a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4000c00 tx=0x7fd3d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 shutdown_connections 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3cc06b500 0x7fd3cc06d9c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e00b4dd0 0x7fd3e00c76a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3e014ae70 0x7fd3e00c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 --2- 172.21.15.77:0/895126498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3e00b4170 0x7fd3e014a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.227+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 >> 172.21.15.77:0/895126498 conn(0x7fd3e00d3c40 msgr2=0x7fd3e00b6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.228+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 shutdown_connections 2024-03-06T07:27:13.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.228+0000 7fd3e596b640 1 -- 172.21.15.77:0/895126498 wait complete. 2024-03-06T07:27:13.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:27:13.372 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:27:13.373 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:13.373 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:27:13.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.413+0000 7f85db665f40 1 Processor -- start 2024-03-06T07:27:13.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.413+0000 7f85db665f40 1 -- start start 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85db665f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85db665f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d8d2da0 0x55cf4d917eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85db665f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d9a48f0 0x55cf4d99cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85db665f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4d917020 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85db665f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4d876b70 con 0x55cf4d9a48f0 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85db665f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf4d9172c0 con 0x55cf4d8d2da0 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85d9166640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85d9166640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36494/0 (socket says 172.21.15.77:36494) 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85d9166640 1 -- 172.21.15.77:0/3800528889 learned_addr learned my addr 172.21.15.77:0/3800528889 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85d3fff640 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d8d2da0 0x55cf4d917eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85d9967640 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d9a48f0 0x55cf4d99cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85d9166640 1 -- 172.21.15.77:0/3800528889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d8d2da0 msgr2=0x55cf4d917eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.414+0000 7f85d9166640 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d8d2da0 0x55cf4d917eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85d9166640 1 -- 172.21.15.77:0/3800528889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d9a48f0 msgr2=0x55cf4d99cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85d9166640 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d9a48f0 0x55cf4d99cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85d9166640 1 -- 172.21.15.77:0/3800528889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf4d99dcd0 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85d9166640 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a20c0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00b500 tx=0x7f85cc00b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134bd4fbe1e8da4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85d37fe640 1 -- 172.21.15.77:0/3800528889 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00b0d0 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85d37fe640 1 -- 172.21.15.77:0/3800528889 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc00b290 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85d37fe640 1 -- 172.21.15.77:0/3800528889 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc0096a0 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 -- 172.21.15.77:0/3800528889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 msgr2=0x55cf4d9a20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a20c0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00b500 tx=0x7f85cc00b9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 -- 172.21.15.77:0/3800528889 shutdown_connections 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d9a48f0 0x55cf4d99cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d8d2da0 0x55cf4d917eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 --2- 172.21.15.77:0/3800528889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a20c0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 -- 172.21.15.77:0/3800528889 >> 172.21.15.77:0/3800528889 conn(0x55cf4d8fa790 msgr2=0x55cf4d918ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 -- 172.21.15.77:0/3800528889 shutdown_connections 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.415+0000 7f85db665f40 1 -- 172.21.15.77:0/3800528889 wait complete. 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.416+0000 7f85db665f40 1 Processor -- start 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.416+0000 7f85db665f40 1 -- start start 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85db665f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85db665f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d8d2da0 0x55cf4d993450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85db665f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d9a48f0 0x55cf4d9986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85db665f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4d8ff670 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85db665f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4d8ff8d0 con 0x55cf4d8d2da0 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85db665f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf4d9a2600 con 0x55cf4d9a48f0 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36506/0 (socket says 172.21.15.77:36506) 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 -- 172.21.15.77:0/3569022019 learned_addr learned my addr 172.21.15.77:0/3569022019 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d3fff640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d8d2da0 0x55cf4d993450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9967640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d9a48f0 0x55cf4d9986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 -- 172.21.15.77:0/3569022019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d9a48f0 msgr2=0x55cf4d9986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d9a48f0 0x55cf4d9986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 -- 172.21.15.77:0/3569022019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d8d2da0 msgr2=0x55cf4d993450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d8d2da0 0x55cf4d993450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 -- 172.21.15.77:0/3569022019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf4d99d750 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9967640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55cf4d9a48f0 0x55cf4d9986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d3fff640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55cf4d8d2da0 0x55cf4d993450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d9166640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a3260 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00ba00 tx=0x7f85cc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d27fc640 1 -- 172.21.15.77:0/3569022019 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00a780 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d27fc640 1 -- 172.21.15.77:0/3569022019 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc024070 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85d27fc640 1 -- 172.21.15.77:0/3569022019 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00ab60 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55cf4d994d30 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.417+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf4d9952f0 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.419+0000 7f85d27fc640 1 -- 172.21.15.77:0/3569022019 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85cc01e070 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.419+0000 7f85d27fc640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85bc069eb0 0x7f85bc06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.420+0000 7f85d27fc640 1 -- 172.21.15.77:0/3569022019 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85cc08b750 con 0x55cf4d8cf7d0 2024-03-06T07:27:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.420+0000 7f85d3fff640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85bc069eb0 0x7f85bc06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.420+0000 7f85db665f40 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55cf4d995810 0x55cf4d995c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.420+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107) v9 -- 0x55cf4d996180 con 0x55cf4d995810 2024-03-06T07:27:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.420+0000 7f85d9967640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55cf4d995810 0x55cf4d995c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.420+0000 7f85d3fff640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85bc069eb0 0x7f85bc06c370 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f85c4005fd0 tx=0x7f85c400c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.420+0000 7f85d9967640 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55cf4d995810 0x55cf4d995c40 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.420+0000 7f85d9967640 1 -- 172.21.15.77:0/3569022019 <== osd.0 v2:172.21.15.77:6808/3119591583 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f85d4057070 con 0x55cf4d995810 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55cf4d995810 msgr2=0x55cf4d995c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55cf4d995810 0x55cf4d995c40 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85bc069eb0 msgr2=0x7f85bc06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f85bc069eb0 0x7f85bc06c370 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f85c4005fd0 tx=0x7f85c400c040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 msgr2=0x55cf4d9a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 --2- 172.21.15.77:0/3569022019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf4d8cf7d0 0x55cf4d9a3260 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00ba00 tx=0x7f85cc000ed0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85d9967640 1 -- 172.21.15.77:0/3569022019 reap_dead start 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 shutdown_connections 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 >> 172.21.15.77:0/3569022019 conn(0x55cf4d8fa790 msgr2=0x55cf4d8d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 shutdown_connections 2024-03-06T07:27:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.421+0000 7f85db665f40 1 -- 172.21.15.77:0/3569022019 wait complete. 2024-03-06T07:27:13.428 INFO:teuthology.orchestra.run.smithi077.stdout:Wfile_a 2024-03-06T07:27:13.428 INFO:teuthology.orchestra.run.smithi077.stdout:dir_1 2024-03-06T07:27:13.428 DEBUG:teuthology.orchestra.run.smithi077:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:27:13.651 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:27:13.651 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 1099511627777, 2024-03-06T07:27:13.651 INFO:teuthology.orchestra.run.smithi077.stdout: "ancestors": [ 2024-03-06T07:27:13.651 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:13.651 INFO:teuthology.orchestra.run.smithi077.stdout: "dirino": 1099511627776, 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: "dname": "file_a", 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: "version": 10 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: "dirino": 1, 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: "dname": "dir_1", 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: "version": 17 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: "pool": 17, 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: "old_pools": [] 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:27:13.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:13.657 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:27:13.658 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:13.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:13.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.811+0000 7f626bc4e640 1 Processor -- start 2024-03-06T07:27:13.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.812+0000 7f626bc4e640 1 -- start start 2024-03-06T07:27:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.812+0000 7f626bc4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62640d2710 0x7f62640b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.812+0000 7f626bc4e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640b3430 con 0x7f626415dfe0 2024-03-06T07:27:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.812+0000 7f626bc4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640b3710 con 0x7f62640d22b0 2024-03-06T07:27:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.812+0000 7f626bc4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62640b31e0 con 0x7f62640d2710 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.812+0000 7f62699ea640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f626415dfe0 0x7f62640cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38206/0 (socket says 172.21.15.77:38206) 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.812+0000 7f62699ea640 1 -- 172.21.15.77:0/4040742283 learned_addr learned my addr 172.21.15.77:0/4040742283 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930895625 0 0) 0x7f62640b3430 con 0x7f626415dfe0 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624c003520 con 0x7f626415dfe0 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f626a1eb640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62640d2710 0x7f62640b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4193992984 0 0) 0x7f624c003520 con 0x7f626415dfe0 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62640d2710 msgr2=0x7f62640b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62640d2710 0x7f62640b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 >> v1:172.21.15.158:6789/0 conn(0x7f62640d22b0 legacy=0x7f62640b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640b39f0 con 0x7f626415dfe0 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f625c001670 con 0x7f626415dfe0 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 >> v1:172.21.15.77:6789/0 conn(0x7f626415dfe0 legacy=0x7f62640cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f624c0039e0 0x7f624c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f624c004800 0x7f624c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62640d2710 0x7f624c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62640b31e0 con 0x7f624c004800 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c00bb50 con 0x7f624c0039e0 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624c00bd50 con 0x7f62640d2710 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62691e9640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f624c004800 0x7f624c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f62699ea640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f624c0039e0 0x7f624c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.813+0000 7f626a1eb640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62640d2710 0x7f624c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62691e9640 1 -- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62640d2710 msgr2=0x7f624c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62691e9640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f62640d2710 0x7f624c005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62691e9640 1 -- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f624c0039e0 msgr2=0x7f624c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62691e9640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f624c0039e0 0x7f624c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62691e9640 1 -- 172.21.15.77:0/4040742283 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c003520 con 0x7f624c004800 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62691e9640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f624c004800 0x7f624c004c80 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f6258001e50 tx=0x7f624c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8016fe1c4ab01706 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258001a70 con 0x7f624c004800 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6258003b70 con 0x7f624c004800 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.814+0000 7f62689e8640 1 -- 172.21.15.77:0/4040742283 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62580050d0 con 0x7f624c004800 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.815+0000 7f626bc4e640 1 -- 172.21.15.77:0/4040742283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f624c004800 msgr2=0x7f624c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.815+0000 7f626bc4e640 1 --2- 172.21.15.77:0/4040742283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f624c004800 0x7f624c004c80 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f6258001e50 tx=0x7f624c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.815+0000 7f626a1eb640 1 -- 172.21.15.77:0/4040742283 reap_dead start 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.815+0000 7f626bc4e640 1 -- 172.21.15.77:0/4040742283 shutdown_connections 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.815+0000 7f626bc4e640 1 -- 172.21.15.77:0/4040742283 >> 172.21.15.77:0/4040742283 conn(0x7f62640d3cd0 msgr2=0x7f62640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.815+0000 7f626bc4e640 1 -- 172.21.15.77:0/4040742283 shutdown_connections 2024-03-06T07:27:13.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.815+0000 7f626bc4e640 1 -- 172.21.15.77:0/4040742283 wait complete. 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.816+0000 7f626bc4e640 1 Processor -- start 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.816+0000 7f626bc4e640 1 -- start start 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f626bc4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f624c004800 0x7f62641f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f626bc4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62640b6a60 0x7f62640b9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f626bc4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6264159b10 0x7f6264159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f626bc4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62640cfa00 con 0x7f62640b6a60 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f626bc4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62640b3d60 con 0x7f6264159b10 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f626bc4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62640d1c70 con 0x7f624c004800 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62691e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62640b6a60 0x7f62640b9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62691e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62640b6a60 0x7f62640b9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36534/0 (socket says 172.21.15.77:36534) 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62691e9640 1 -- 172.21.15.77:0/2929322058 learned_addr learned my addr 172.21.15.77:0/2929322058 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62699ea640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f624c004800 0x7f62641f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f626a1eb640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6264159b10 0x7f6264159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62691e9640 1 -- 172.21.15.77:0/2929322058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f624c004800 msgr2=0x7f62641f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62691e9640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f624c004800 0x7f62641f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62691e9640 1 -- 172.21.15.77:0/2929322058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6264159b10 msgr2=0x7f6264159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62691e9640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6264159b10 0x7f6264159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.817+0000 7f62691e9640 1 -- 172.21.15.77:0/2929322058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626415a4d0 con 0x7f62640b6a60 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.818+0000 7f62691e9640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62640b6a60 0x7f62640b9470 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f625802c060 tx=0x7f62580014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.818+0000 7f62627fc640 1 -- 172.21.15.77:0/2929322058 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258047070 con 0x7f62640b6a60 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.818+0000 7f62627fc640 1 -- 172.21.15.77:0/2929322058 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6258003970 con 0x7f62640b6a60 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.818+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62640b8270 con 0x7f62640b6a60 2024-03-06T07:27:13.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.818+0000 7f62627fc640 1 -- 172.21.15.77:0/2929322058 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625802f680 con 0x7f62640b6a60 2024-03-06T07:27:13.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.818+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62640b74b0 con 0x7f62640b6a60 2024-03-06T07:27:13.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.819+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62640b5200 con 0x7f62640b6a60 2024-03-06T07:27:13.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.819+0000 7f62627fc640 1 -- 172.21.15.77:0/2929322058 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f625804a070 con 0x7f62640b6a60 2024-03-06T07:27:13.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.820+0000 7f62627fc640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f625006b4d0 0x7f625006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.820+0000 7f62699ea640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f625006b4d0 0x7f625006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.820+0000 7f62627fc640 1 -- 172.21.15.77:0/2929322058 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62580ab890 con 0x7f62640b6a60 2024-03-06T07:27:13.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.820+0000 7f62699ea640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f625006b4d0 0x7f625006d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f625c000fb0 tx=0x7f625c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.824+0000 7f62627fc640 1 -- 172.21.15.77:0/2929322058 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62640b5200 con 0x7f62640b6a60 2024-03-06T07:27:14.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:13.999+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f626404f330 con 0x7f62640b6a60 2024-03-06T07:27:14.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.000+0000 7f62627fc640 1 -- 172.21.15.77:0/2929322058 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f626415a4d0 con 0x7f62640b6a60 2024-03-06T07:27:14.002 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:14.003 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:06.505066+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:14.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f625006b4d0 msgr2=0x7f625006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f625006b4d0 0x7f625006d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f625c000fb0 tx=0x7f625c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62640b6a60 msgr2=0x7f62640b9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62640b6a60 0x7f62640b9470 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f625802c060 tx=0x7f62580014a0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 shutdown_connections 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f625006b4d0 0x7f625006d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62640b6a60 0x7f62640b9470 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6264159b10 0x7f6264159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 --2- 172.21.15.77:0/2929322058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f624c004800 0x7f62641f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 >> 172.21.15.77:0/2929322058 conn(0x7f62640d3cd0 msgr2=0x7f62640cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 shutdown_connections 2024-03-06T07:27:14.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.002+0000 7f626bc4e640 1 -- 172.21.15.77:0/2929322058 wait complete. 2024-03-06T07:27:14.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:03.116010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:03.812681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:14.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:14.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.144+0000 7f94d3c37640 1 Processor -- start 2024-03-06T07:27:14.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.145+0000 7f94d3c37640 1 -- start start 2024-03-06T07:27:14.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.145+0000 7f94d3c37640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f94cc0b1e40 0x7f94cc168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.145+0000 7f94d3c37640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94cc0d2730 con 0x7f94cc1614d0 2024-03-06T07:27:14.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.145+0000 7f94d3c37640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94cc1618b0 con 0x7f94cc0d4860 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.145+0000 7f94d3c37640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94cc0d4cf0 con 0x7f94cc0b1e40 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.145+0000 7f94d19d3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f94cc1614d0 0x7f94cc0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38216/0 (socket says 172.21.15.77:38216) 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.145+0000 7f94d19d3640 1 -- 172.21.15.77:0/629585304 learned_addr learned my addr 172.21.15.77:0/629585304 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d21d4640 1 --2- 172.21.15.77:0/629585304 >> v2:172.21.15.158:3300/0 conn(0x7f94cc0b1e40 0x7f94cc168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d09d1640 1 -- 172.21.15.77:0/629585304 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534708475 0 0) 0x7f94cc0d2730 con 0x7f94cc1614d0 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d09d1640 1 -- 172.21.15.77:0/629585304 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94b8003520 con 0x7f94cc1614d0 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d09d1640 1 -- 172.21.15.77:0/629585304 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228968039 0 0) 0x7f94cc1618b0 con 0x7f94cc0d4860 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d09d1640 1 -- 172.21.15.77:0/629585304 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94cc0d2730 con 0x7f94cc0d4860 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d21d4640 1 -- 172.21.15.77:0/629585304 >> v1:172.21.15.158:6789/0 conn(0x7f94cc0d4860 legacy=0x7f94cc0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d21d4640 1 -- 172.21.15.77:0/629585304 >> v1:172.21.15.77:6789/0 conn(0x7f94cc1614d0 legacy=0x7f94cc0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d09d1640 1 -- 172.21.15.77:0/629585304 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 733551994 0 0) 0x7f94b8003520 con 0x7f94cc1614d0 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.146+0000 7f94d21d4640 1 -- 172.21.15.77:0/629585304 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc0cf400 con 0x7f94cc0b1e40 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.147+0000 7f94d21d4640 1 --2- 172.21.15.77:0/629585304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0b1e40 0x7f94cc168b60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0cf5d0 tx=0x7f94c402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=810b1e5c7dc26c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.147+0000 7f94d09d1640 1 -- 172.21.15.77:0/629585304 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4039070 con 0x7f94cc0b1e40 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.147+0000 7f94d09d1640 1 -- 172.21.15.77:0/629585304 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c402c8a0 con 0x7f94cc0b1e40 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.147+0000 7f94d09d1640 1 -- 172.21.15.77:0/629585304 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc001670 con 0x7f94cc0b1e40 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.147+0000 7f94d3c37640 1 -- 172.21.15.77:0/629585304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0b1e40 msgr2=0x7f94cc168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.147+0000 7f94d3c37640 1 --2- 172.21.15.77:0/629585304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0b1e40 0x7f94cc168b60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0cf5d0 tx=0x7f94c402bd70 comp rx=0 tx=0).stop 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.148+0000 7f94d3c37640 1 -- 172.21.15.77:0/629585304 shutdown_connections 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.148+0000 7f94d3c37640 1 --2- 172.21.15.77:0/629585304 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0b1e40 0x7f94cc168b60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.148+0000 7f94d3c37640 1 -- 172.21.15.77:0/629585304 >> 172.21.15.77:0/629585304 conn(0x7f94cc0d3d40 msgr2=0x7f94cc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.148+0000 7f94d3c37640 1 -- 172.21.15.77:0/629585304 shutdown_connections 2024-03-06T07:27:14.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.148+0000 7f94d3c37640 1 -- 172.21.15.77:0/629585304 wait complete. 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.149+0000 7f94d3c37640 1 Processor -- start 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.149+0000 7f94d3c37640 1 -- start start 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d3c37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc0b1e40 0x7f94cc06c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d3c37640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0d4860 0x7f94cc0c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d3c37640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94cc1614d0 0x7f94cc151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d3c37640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc0cfc60 con 0x7f94cc0b1e40 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d3c37640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc0cf980 con 0x7f94cc0d4860 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d3c37640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc0b0910 con 0x7f94cc1614d0 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc0b1e40 0x7f94cc06c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc0b1e40 0x7f94cc06c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36536/0 (socket says 172.21.15.77:36536) 2024-03-06T07:27:14.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 -- 172.21.15.77:0/4224030113 learned_addr learned my addr 172.21.15.77:0/4224030113 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d21d4640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94cc1614d0 0x7f94cc151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d11d2640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0d4860 0x7f94cc0c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 -- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94cc1614d0 msgr2=0x7f94cc151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94cc1614d0 0x7f94cc151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 -- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0d4860 msgr2=0x7f94cc0c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0d4860 0x7f94cc0c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 -- 172.21.15.77:0/4224030113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc0c93f0 con 0x7f94cc0b1e40 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d21d4640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94cc1614d0 0x7f94cc151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d11d2640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0d4860 0x7f94cc0c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d19d3640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc0b1e40 0x7f94cc06c930 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00c780 tx=0x7f94bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94c27fc640 1 -- 172.21.15.77:0/4224030113 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc017070 con 0x7f94cc0b1e40 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94c27fc640 1 -- 172.21.15.77:0/4224030113 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc005af0 con 0x7f94cc0b1e40 2024-03-06T07:27:14.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94c27fc640 1 -- 172.21.15.77:0/4224030113 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc007070 con 0x7f94cc0b1e40 2024-03-06T07:27:14.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.151+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc0c96d0 con 0x7f94cc0b1e40 2024-03-06T07:27:14.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.152+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94cc0c9c90 con 0x7f94cc0b1e40 2024-03-06T07:27:14.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.153+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94cc068ff0 con 0x7f94cc0b1e40 2024-03-06T07:27:14.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.153+0000 7f94c27fc640 1 -- 172.21.15.77:0/4224030113 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94bc01a070 con 0x7f94cc0b1e40 2024-03-06T07:27:14.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.153+0000 7f94c27fc640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94b006b4d0 0x7f94b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.153+0000 7f94c27fc640 1 -- 172.21.15.77:0/4224030113 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94bc089790 con 0x7f94cc0b1e40 2024-03-06T07:27:14.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.153+0000 7f94d11d2640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94b006b4d0 0x7f94b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.154+0000 7f94d11d2640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94b006b4d0 0x7f94b006d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0cbe90 tx=0x7f94c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.158+0000 7f94c27fc640 1 -- 172.21.15.77:0/4224030113 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94bc058610 con 0x7f94cc0b1e40 2024-03-06T07:27:14.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.351+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94cc0d42a0 con 0x7f94cc0b1e40 2024-03-06T07:27:14.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.352+0000 7f94c27fc640 1 -- 172.21.15.77:0/4224030113 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f94bc057f10 con 0x7f94cc0b1e40 2024-03-06T07:27:14.413 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:27:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94b006b4d0 msgr2=0x7f94b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94b006b4d0 0x7f94b006d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0cbe90 tx=0x7f94c8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc0b1e40 msgr2=0x7f94cc06c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc0b1e40 0x7f94cc06c930 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00c780 tx=0x7f94bc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:27:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 shutdown_connections 2024-03-06T07:27:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f94cc1614d0 0x7f94cc151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f94cc0d4860 0x7f94cc0c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f94b006b4d0 0x7f94b006d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 --2- 172.21.15.77:0/4224030113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc0b1e40 0x7f94cc06c930 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 >> 172.21.15.77:0/4224030113 conn(0x7f94cc0d3d40 msgr2=0x7f94cc06bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 shutdown_connections 2024-03-06T07:27:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.354+0000 7f94d3c37640 1 -- 172.21.15.77:0/4224030113 wait complete. 2024-03-06T07:27:14.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:14.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.519+0000 7fd26f64e640 1 Processor -- start 2024-03-06T07:27:14.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.519+0000 7fd26f64e640 1 -- start start 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26f64e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd2681614d0 0x7fd2680d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26f64e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2681618b0 con 0x7fd2680d1910 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26f64e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2680cf400 con 0x7fd2680d24b0 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26f64e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2680cf6a0 con 0x7fd2681614d0 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26d3ea640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd2681614d0 0x7fd2680d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26cbe9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd2680d1910 0x7fd2680d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38230/0 (socket says 172.21.15.77:38230) 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26d3ea640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd2681614d0 0x7fd2680d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36540/0 (socket says 172.21.15.77:36540) 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26d3ea640 1 -- 172.21.15.77:0/3081630741 learned_addr learned my addr 172.21.15.77:0/3081630741 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26d3ea640 1 -- 172.21.15.77:0/3081630741 >> v1:172.21.15.158:6789/0 conn(0x7fd2680d24b0 legacy=0x7fd2680b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26d3ea640 1 -- 172.21.15.77:0/3081630741 >> v1:172.21.15.77:6789/0 conn(0x7fd2680d1910 legacy=0x7fd2680d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd26d3ea640 1 -- 172.21.15.77:0/3081630741 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2680cf980 con 0x7fd2681614d0 2024-03-06T07:27:14.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.520+0000 7fd25ffff640 1 -- 172.21.15.77:0/3081630741 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076277583 0 0) 0x7fd2681618b0 con 0x7fd2680d1910 2024-03-06T07:27:14.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.521+0000 7fd26d3ea640 1 --2- 172.21.15.77:0/3081630741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2681614d0 0x7fd2680d4860 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fd26808c470 tx=0x7fd26402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58bca3e62e8e6eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.521+0000 7fd25ffff640 1 -- 172.21.15.77:0/3081630741 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264034070 con 0x7fd2681614d0 2024-03-06T07:27:14.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.521+0000 7fd25ffff640 1 -- 172.21.15.77:0/3081630741 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd264030610 con 0x7fd2681614d0 2024-03-06T07:27:14.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.521+0000 7fd25ffff640 1 -- 172.21.15.77:0/3081630741 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640309e0 con 0x7fd2681614d0 2024-03-06T07:27:14.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.522+0000 7fd26f64e640 1 -- 172.21.15.77:0/3081630741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2681614d0 msgr2=0x7fd2680d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.522+0000 7fd26f64e640 1 --2- 172.21.15.77:0/3081630741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2681614d0 0x7fd2680d4860 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fd26808c470 tx=0x7fd26402b960 comp rx=0 tx=0).stop 2024-03-06T07:27:14.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.522+0000 7fd26f64e640 1 -- 172.21.15.77:0/3081630741 shutdown_connections 2024-03-06T07:27:14.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.522+0000 7fd26f64e640 1 --2- 172.21.15.77:0/3081630741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2681614d0 0x7fd2680d4860 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.522+0000 7fd26f64e640 1 -- 172.21.15.77:0/3081630741 >> 172.21.15.77:0/3081630741 conn(0x7fd2680d3d40 msgr2=0x7fd2680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.522+0000 7fd26f64e640 1 -- 172.21.15.77:0/3081630741 shutdown_connections 2024-03-06T07:27:14.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.522+0000 7fd26f64e640 1 -- 172.21.15.77:0/3081630741 wait complete. 2024-03-06T07:27:14.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.524+0000 7fd26f64e640 1 Processor -- start 2024-03-06T07:27:14.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.524+0000 7fd26f64e640 1 -- start start 2024-03-06T07:27:14.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26f64e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2680d1910 0x7fd268158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26f64e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2680d24b0 0x7fd2680cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26f64e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2681614d0 0x7fd2680cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26f64e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680b1540 con 0x7fd2680d24b0 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26f64e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680b0910 con 0x7fd2680d1910 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26f64e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680b16c0 con 0x7fd2681614d0 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26cbe9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2680d24b0 0x7fd2680cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26cbe9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2680d24b0 0x7fd2680cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36550/0 (socket says 172.21.15.77:36550) 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.525+0000 7fd26cbe9640 1 -- 172.21.15.77:0/3543749039 learned_addr learned my addr 172.21.15.77:0/3543749039 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26d3ea640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2680d1910 0x7fd268158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26dbeb640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2681614d0 0x7fd2680cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26cbe9640 1 -- 172.21.15.77:0/3543749039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2681614d0 msgr2=0x7fd2680cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26cbe9640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2681614d0 0x7fd2680cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26cbe9640 1 -- 172.21.15.77:0/3543749039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2680d1910 msgr2=0x7fd268158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26cbe9640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2680d1910 0x7fd268158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26cbe9640 1 -- 172.21.15.77:0/3543749039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2680b8b80 con 0x7fd2680d24b0 2024-03-06T07:27:14.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26cbe9640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2680d24b0 0x7fd2680cd700 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd25800c4d0 tx=0x7fd25800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd25dffb640 1 -- 172.21.15.77:0/3543749039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258018070 con 0x7fd2680d24b0 2024-03-06T07:27:14.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd25dffb640 1 -- 172.21.15.77:0/3543749039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd258007500 con 0x7fd2680d24b0 2024-03-06T07:27:14.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd25dffb640 1 -- 172.21.15.77:0/3543749039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258004a20 con 0x7fd2680d24b0 2024-03-06T07:27:14.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2680b8e60 con 0x7fd2680d24b0 2024-03-06T07:27:14.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.526+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2680b9420 con 0x7fd2680d24b0 2024-03-06T07:27:14.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.527+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd268068ff0 con 0x7fd2680d24b0 2024-03-06T07:27:14.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.528+0000 7fd25dffb640 1 -- 172.21.15.77:0/3543749039 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd25801b070 con 0x7fd2680d24b0 2024-03-06T07:27:14.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.528+0000 7fd25dffb640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd24c06b4d0 0x7fd24c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.528+0000 7fd25dffb640 1 -- 172.21.15.77:0/3543749039 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd258089550 con 0x7fd2680d24b0 2024-03-06T07:27:14.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.528+0000 7fd26d3ea640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd24c06b4d0 0x7fd24c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.528+0000 7fd26d3ea640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd24c06b4d0 0x7fd24c06d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd264036000 tx=0x7fd26403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.534+0000 7fd25dffb640 1 -- 172.21.15.77:0/3543749039 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2580583c0 con 0x7fd2680d24b0 2024-03-06T07:27:14.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.737+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd2680d1060 con 0x7fd2680d24b0 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.738+0000 7fd25dffb640 1 -- 172.21.15.77:0/3543749039 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd258057cc0 con 0x7fd2680d24b0 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd24c06b4d0 msgr2=0x7fd24c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd24c06b4d0 0x7fd24c06d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd264036000 tx=0x7fd26403b040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2680d24b0 msgr2=0x7fd2680cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2680d24b0 0x7fd2680cd700 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd25800c4d0 tx=0x7fd25800c990 comp rx=0 tx=0).stop 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 shutdown_connections 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd24c06b4d0 0x7fd24c06d990 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2680d24b0 0x7fd2680cd700 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2681614d0 0x7fd2680cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 --2- 172.21.15.77:0/3543749039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2680d1910 0x7fd268158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 >> 172.21.15.77:0/3543749039 conn(0x7fd2680d3d40 msgr2=0x7fd2680b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 shutdown_connections 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.740+0000 7fd26f64e640 1 -- 172.21.15.77:0/3543749039 wait complete. 2024-03-06T07:27:14.760 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.864+0000 7f615972b640 1 Processor -- start 2024-03-06T07:27:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.864+0000 7f615972b640 1 -- start start 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.865+0000 7f615972b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f61541455d0 0x7f6154072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.865+0000 7f615972b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f615406b3f0 0x7f615406e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.865+0000 7f615972b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61541459b0 con 0x7f615406afb0 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.865+0000 7f615972b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61540cfdf0 con 0x7f61541455d0 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.865+0000 7f615972b640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61540cf1e0 con 0x7f615406b3f0 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.866+0000 7f6152ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f61541455d0 0x7f6154072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.866+0000 7f6152ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f61541455d0 0x7f6154072480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36564/0 (socket says 172.21.15.77:36564) 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.866+0000 7f6152ffd640 1 -- 172.21.15.77:0/1608849170 learned_addr learned my addr 172.21.15.77:0/1608849170 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.866+0000 7f6152ffd640 1 -- 172.21.15.77:0/1608849170 >> v2:172.21.15.158:3300/0 conn(0x7f615406b3f0 msgr2=0x7f615406e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.866+0000 7f6152ffd640 1 --2- 172.21.15.77:0/1608849170 >> v2:172.21.15.158:3300/0 conn(0x7f615406b3f0 0x7f615406e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.866+0000 7f6152ffd640 1 -- 172.21.15.77:0/1608849170 >> v1:172.21.15.158:6789/0 conn(0x7f615406afb0 legacy=0x7f615406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.866+0000 7f6152ffd640 1 -- 172.21.15.77:0/1608849170 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61540cf4c0 con 0x7f61541455d0 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f6152ffd640 1 --2- 172.21.15.77:0/1608849170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61541455d0 0x7f6154072480 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f61540d09a0 tx=0x7f614402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52401a1293e05bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f6151ffb640 1 -- 172.21.15.77:0/1608849170 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6144035020 con 0x7f61541455d0 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f6151ffb640 1 -- 172.21.15.77:0/1608849170 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6144031870 con 0x7f61541455d0 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f6151ffb640 1 -- 172.21.15.77:0/1608849170 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6144031c40 con 0x7f61541455d0 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f615972b640 1 -- 172.21.15.77:0/1608849170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61541455d0 msgr2=0x7f6154072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f615972b640 1 --2- 172.21.15.77:0/1608849170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61541455d0 0x7f6154072480 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f61540d09a0 tx=0x7f614402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f615972b640 1 -- 172.21.15.77:0/1608849170 shutdown_connections 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f615972b640 1 --2- 172.21.15.77:0/1608849170 >> v2:172.21.15.158:3300/0 conn(0x7f615406b3f0 0x7f615406e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f615972b640 1 --2- 172.21.15.77:0/1608849170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61541455d0 0x7f6154072480 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f615972b640 1 -- 172.21.15.77:0/1608849170 >> 172.21.15.77:0/1608849170 conn(0x7f61540b2660 msgr2=0x7f61540b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f615972b640 1 -- 172.21.15.77:0/1608849170 shutdown_connections 2024-03-06T07:27:14.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.867+0000 7f615972b640 1 -- 172.21.15.77:0/1608849170 wait complete. 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f615972b640 1 Processor -- start 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f615972b640 1 -- start start 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f615972b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f615406afb0 0x7f61540b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f615972b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f615406b3f0 0x7f615413cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f615972b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61541455d0 0x7f615413d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f615972b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540729c0 con 0x7f615406b3f0 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f615972b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540cfa80 con 0x7f615406afb0 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f615972b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61540b3400 con 0x7f61541455d0 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f61527fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f615406b3f0 0x7f615413cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f61527fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f615406b3f0 0x7f615413cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36570/0 (socket says 172.21.15.77:36570) 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.869+0000 7f61527fc640 1 -- 172.21.15.77:0/2516474032 learned_addr learned my addr 172.21.15.77:0/2516474032 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f6152ffd640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f615406afb0 0x7f61540b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f61537fe640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61541455d0 0x7f615413d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f61527fc640 1 -- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61541455d0 msgr2=0x7f615413d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f61527fc640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61541455d0 0x7f615413d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f61527fc640 1 -- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f615406afb0 msgr2=0x7f61540b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f61527fc640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f615406afb0 0x7f61540b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f61527fc640 1 -- 172.21.15.77:0/2516474032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61541f78f0 con 0x7f615406b3f0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f6152ffd640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f615406afb0 0x7f61540b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f61527fc640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f615406b3f0 0x7f615413cc80 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f614800c410 tx=0x7f614800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f613b7fe640 1 -- 172.21.15.77:0/2516474032 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148017070 con 0x7f615406b3f0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f613b7fe640 1 -- 172.21.15.77:0/2516474032 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61480035f0 con 0x7f615406b3f0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61541f7bd0 con 0x7f615406b3f0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f613b7fe640 1 -- 172.21.15.77:0/2516474032 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148003a20 con 0x7f615406b3f0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.870+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61541f8190 con 0x7f615406b3f0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.871+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615415c980 con 0x7f615406b3f0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.872+0000 7f613b7fe640 1 -- 172.21.15.77:0/2516474032 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f614801a070 con 0x7f615406b3f0 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.872+0000 7f613b7fe640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f614006b4d0 0x7f614006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.872+0000 7f613b7fe640 1 -- 172.21.15.77:0/2516474032 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6148089630 con 0x7f615406b3f0 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.872+0000 7f6152ffd640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f614006b4d0 0x7f614006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.872+0000 7f6152ffd640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f614006b4d0 0x7f614006d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6144036000 tx=0x7f614403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:14.876+0000 7f613b7fe640 1 -- 172.21.15.77:0/2516474032 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f615415c980 con 0x7f615406b3f0 2024-03-06T07:27:15.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.052+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f61541f8360 con 0x7f615406b3f0 2024-03-06T07:27:15.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.053+0000 7f613b7fe640 1 -- 172.21.15.77:0/2516474032 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7f614805ce20 con 0x7f615406b3f0 2024-03-06T07:27:15.055 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:27:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f614006b4d0 msgr2=0x7f614006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f614006b4d0 0x7f614006d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6144036000 tx=0x7f614403c040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f615406b3f0 msgr2=0x7f615413cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f615406b3f0 0x7f615413cc80 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f614800c410 tx=0x7f614800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 shutdown_connections 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f614006b4d0 0x7f614006d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f615406b3f0 0x7f615413cc80 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f61541455d0 0x7f615413d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 --2- 172.21.15.77:0/2516474032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f615406afb0 0x7f61540b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 >> 172.21.15.77:0/2516474032 conn(0x7f61540b2660 msgr2=0x7f61540b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 shutdown_connections 2024-03-06T07:27:15.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.055+0000 7f615972b640 1 -- 172.21.15.77:0/2516474032 wait complete. 2024-03-06T07:27:15.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.168+0000 7fde7217a640 1 Processor -- start 2024-03-06T07:27:15.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.168+0000 7fde7217a640 1 -- start start 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde7217a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde6c162400 0x7fde6c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde7217a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c0d22c0 con 0x7fde6c0b4460 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde7217a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c0d25a0 con 0x7fde6c162970 2024-03-06T07:27:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde7217a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c161b00 con 0x7fde6c162400 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde6bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde6c162400 0x7fde6c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde6b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fde6c0b4460 0x7fde6c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38232/0 (socket says 172.21.15.77:38232) 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde6b7fe640 1 -- 172.21.15.77:0/611632698 learned_addr learned my addr 172.21.15.77:0/611632698 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde6bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde6c162400 0x7fde6c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36572/0 (socket says 172.21.15.77:36572) 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde6bfff640 1 -- 172.21.15.77:0/611632698 >> v1:172.21.15.158:6789/0 conn(0x7fde6c162970 legacy=0x7fde6c161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde6bfff640 1 -- 172.21.15.77:0/611632698 >> v1:172.21.15.77:6789/0 conn(0x7fde6c0b4460 legacy=0x7fde6c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde6bfff640 1 -- 172.21.15.77:0/611632698 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0d2880 con 0x7fde6c162400 2024-03-06T07:27:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.169+0000 7fde6a7fc640 1 -- 172.21.15.77:0/611632698 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591439570 0 0) 0x7fde6c0d22c0 con 0x7fde6c0b4460 2024-03-06T07:27:15.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde6bfff640 1 --2- 172.21.15.77:0/611632698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c0d4800 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fde640097a0 tx=0x7fde6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cfcb36c111b441 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde6a7fc640 1 -- 172.21.15.77:0/611632698 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64035020 con 0x7fde6c162400 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde6a7fc640 1 -- 172.21.15.77:0/611632698 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64031830 con 0x7fde6c162400 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde6a7fc640 1 -- 172.21.15.77:0/611632698 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64031c00 con 0x7fde6c162400 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde7217a640 1 -- 172.21.15.77:0/611632698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 msgr2=0x7fde6c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde7217a640 1 --2- 172.21.15.77:0/611632698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c0d4800 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fde640097a0 tx=0x7fde6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde7217a640 1 -- 172.21.15.77:0/611632698 shutdown_connections 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde7217a640 1 --2- 172.21.15.77:0/611632698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c0d4800 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde7217a640 1 -- 172.21.15.77:0/611632698 >> 172.21.15.77:0/611632698 conn(0x7fde6c0d3ce0 msgr2=0x7fde6c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde7217a640 1 -- 172.21.15.77:0/611632698 shutdown_connections 2024-03-06T07:27:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.171+0000 7fde7217a640 1 -- 172.21.15.77:0/611632698 wait complete. 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.172+0000 7fde7217a640 1 Processor -- start 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde7217a640 1 -- start start 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde7217a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde6c0b4460 0x7fde6c0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde7217a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde7217a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde6c162970 0x7fde6c0c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde7217a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0ce950 con 0x7fde6c162400 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde7217a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0cecc0 con 0x7fde6c0b4460 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde7217a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b4840 con 0x7fde6c162970 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde6affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde6affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36584/0 (socket says 172.21.15.77:36584) 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.173+0000 7fde6affd640 1 -- 172.21.15.77:0/652138224 learned_addr learned my addr 172.21.15.77:0/652138224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6b7fe640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde6c0b4460 0x7fde6c0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6bfff640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde6c162970 0x7fde6c0c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6affd640 1 -- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde6c162970 msgr2=0x7fde6c0c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6affd640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde6c162970 0x7fde6c0c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6affd640 1 -- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde6c0b4460 msgr2=0x7fde6c0c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6affd640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde6c0b4460 0x7fde6c0c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6affd640 1 -- 172.21.15.77:0/652138224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c156250 con 0x7fde6c162400 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6bfff640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde6c162970 0x7fde6c0c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde6affd640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c155750 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fde5400b3c0 tx=0x7fde5400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde63fff640 1 -- 172.21.15.77:0/652138224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54011830 con 0x7fde6c162400 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde63fff640 1 -- 172.21.15.77:0/652138224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54016070 con 0x7fde6c162400 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde63fff640 1 -- 172.21.15.77:0/652138224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54011c50 con 0x7fde6c162400 2024-03-06T07:27:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c0c8380 con 0x7fde6c162400 2024-03-06T07:27:15.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.174+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c0c8940 con 0x7fde6c162400 2024-03-06T07:27:15.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.175+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c04f330 con 0x7fde6c162400 2024-03-06T07:27:15.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.176+0000 7fde63fff640 1 -- 172.21.15.77:0/652138224 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde5401b460 con 0x7fde6c162400 2024-03-06T07:27:15.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.176+0000 7fde63fff640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde4c06b4d0 0x7fde4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.176+0000 7fde6b7fe640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde4c06b4d0 0x7fde4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.176+0000 7fde63fff640 1 -- 172.21.15.77:0/652138224 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde54089bd0 con 0x7fde6c162400 2024-03-06T07:27:15.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.177+0000 7fde6b7fe640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde4c06b4d0 0x7fde4c06d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0012e0 tx=0x7fde5c001700 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.184+0000 7fde63fff640 1 -- 172.21.15.77:0/652138224 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde54020020 con 0x7fde6c162400 2024-03-06T07:27:15.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.374+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fde6c06b180 con 0x7fde6c162400 2024-03-06T07:27:15.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.375+0000 7fde63fff640 1 -- 172.21.15.77:0/652138224 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v107) v1 ==== 80+0+0 (secure 0 0 0) 0x7fde54058ae0 con 0x7fde6c162400 2024-03-06T07:27:15.376 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde4c06b4d0 msgr2=0x7fde4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde4c06b4d0 0x7fde4c06d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0012e0 tx=0x7fde5c001700 comp rx=0 tx=0).stop 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 msgr2=0x7fde6c155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c155750 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fde5400b3c0 tx=0x7fde5400b880 comp rx=0 tx=0).stop 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 shutdown_connections 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde4c06b4d0 0x7fde4c06d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde6c162400 0x7fde6c155750 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde6c162970 0x7fde6c0c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 --2- 172.21.15.77:0/652138224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde6c0b4460 0x7fde6c0c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.376+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 >> 172.21.15.77:0/652138224 conn(0x7fde6c0d3ce0 msgr2=0x7fde6c0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.377+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 shutdown_connections 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.377+0000 7fde7217a640 1 -- 172.21.15.77:0/652138224 wait complete. 2024-03-06T07:27:15.389 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:27:15.389 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:15.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:15.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.546+0000 7f40a8b0f640 1 Processor -- start 2024-03-06T07:27:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.547+0000 7f40a8b0f640 1 -- start start 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.547+0000 7f40a8b0f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f40a415dfe0 0x7f40a40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.547+0000 7f40a8b0f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40cf830 0x7f40a40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.547+0000 7f40a8b0f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a40b3430 con 0x7f40a40d27f0 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.547+0000 7f40a8b0f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a40d2c50 con 0x7f40a415dfe0 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.547+0000 7f40a8b0f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40b31e0 con 0x7f40a40cf830 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.548+0000 7f40a259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f40a415dfe0 0x7f40a40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.548+0000 7f40a259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f40a415dfe0 0x7f40a40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36588/0 (socket says 172.21.15.77:36588) 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.548+0000 7f40a259c640 1 -- 172.21.15.77:0/376049509 learned_addr learned my addr 172.21.15.77:0/376049509 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.548+0000 7f40a1d9b640 1 --2- 172.21.15.77:0/376049509 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40cf830 0x7f40a40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.548+0000 7f40a259c640 1 -- 172.21.15.77:0/376049509 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40cf830 msgr2=0x7f40a40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.548+0000 7f40a259c640 1 --2- 172.21.15.77:0/376049509 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40cf830 0x7f40a40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.548+0000 7f40a259c640 1 -- 172.21.15.77:0/376049509 >> v1:172.21.15.158:6789/0 conn(0x7f40a40d27f0 legacy=0x7f40a40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.548+0000 7f40a259c640 1 -- 172.21.15.77:0/376049509 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a40b3710 con 0x7f40a415dfe0 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a259c640 1 --2- 172.21.15.77:0/376049509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a415dfe0 0x7f40a40cf2f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f409c009e50 tx=0x7f409c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16147a0adc618f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a159a640 1 -- 172.21.15.77:0/376049509 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c02ca70 con 0x7f40a415dfe0 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a159a640 1 -- 172.21.15.77:0/376049509 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c02cc10 con 0x7f40a415dfe0 2024-03-06T07:27:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a159a640 1 -- 172.21.15.77:0/376049509 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0343f0 con 0x7f40a415dfe0 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a8b0f640 1 -- 172.21.15.77:0/376049509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a415dfe0 msgr2=0x7f40a40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/376049509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a415dfe0 0x7f40a40cf2f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f409c009e50 tx=0x7f409c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a8b0f640 1 -- 172.21.15.77:0/376049509 shutdown_connections 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/376049509 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40cf830 0x7f40a40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/376049509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a415dfe0 0x7f40a40cf2f0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a8b0f640 1 -- 172.21.15.77:0/376049509 >> 172.21.15.77:0/376049509 conn(0x7f40a40d3cd0 msgr2=0x7f40a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a8b0f640 1 -- 172.21.15.77:0/376049509 shutdown_connections 2024-03-06T07:27:15.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.549+0000 7f40a8b0f640 1 -- 172.21.15.77:0/376049509 wait complete. 2024-03-06T07:27:15.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.551+0000 7f40a8b0f640 1 Processor -- start 2024-03-06T07:27:15.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.551+0000 7f40a8b0f640 1 -- start start 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a8b0f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a40cf830 0x7f40a40b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a8b0f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40d27f0 0x7f40a40cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a8b0f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40a415dfe0 0x7f40a40cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a8b0f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40d1c70 con 0x7f40a40cf830 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a8b0f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40b3d60 con 0x7f40a415dfe0 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a8b0f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40d1df0 con 0x7f40a40d27f0 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a40cf830 0x7f40a40b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a40cf830 0x7f40a40b69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36592/0 (socket says 172.21.15.77:36592) 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 -- 172.21.15.77:0/3979664788 learned_addr learned my addr 172.21.15.77:0/3979664788 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a1d9b640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40d27f0 0x7f40a40cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a2d9d640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40a415dfe0 0x7f40a40cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 -- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40d27f0 msgr2=0x7f40a40cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40d27f0 0x7f40a40cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 -- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40a415dfe0 msgr2=0x7f40a40cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40a415dfe0 0x7f40a40cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 -- 172.21.15.77:0/3979664788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a40ccfb0 con 0x7f40a40cf830 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a2d9d640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40a415dfe0 0x7f40a40cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a1d9b640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40d27f0 0x7f40a40cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a259c640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a40cf830 0x7f40a40b69c0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f409c009e50 tx=0x7f409c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f4092ffd640 1 -- 172.21.15.77:0/3979664788 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c043020 con 0x7f40a40cf830 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f4092ffd640 1 -- 172.21.15.77:0/3979664788 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c03e450 con 0x7f40a40cf830 2024-03-06T07:27:15.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a40cc470 con 0x7f40a40cf830 2024-03-06T07:27:15.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f4092ffd640 1 -- 172.21.15.77:0/3979664788 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c009510 con 0x7f40a40cf830 2024-03-06T07:27:15.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.553+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a40cc9f0 con 0x7f40a40cf830 2024-03-06T07:27:15.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.554+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a40ce3c0 con 0x7f40a40cf830 2024-03-06T07:27:15.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.555+0000 7f4092ffd640 1 -- 172.21.15.77:0/3979664788 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f409c046020 con 0x7f40a40cf830 2024-03-06T07:27:15.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.555+0000 7f4092ffd640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f408406b4d0 0x7f408406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.555+0000 7f4092ffd640 1 -- 172.21.15.77:0/3979664788 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f409c0aca80 con 0x7f40a40cf830 2024-03-06T07:27:15.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.555+0000 7f40a1d9b640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f408406b4d0 0x7f408406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.556+0000 7f40a1d9b640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f408406b4d0 0x7f408406d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f40a4160d00 tx=0x7f4098008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.560+0000 7f4092ffd640 1 -- 172.21.15.77:0/3979664788 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40a40ce3c0 con 0x7f40a40cf830 2024-03-06T07:27:15.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.734+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40a415f000 con 0x7f40a40cf830 2024-03-06T07:27:15.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.735+0000 7f4092ffd640 1 -- 172.21.15.77:0/3979664788 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f409c04d030 con 0x7f40a40cf830 2024-03-06T07:27:15.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:15.738 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:06.505066+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f408406b4d0 msgr2=0x7f408406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f408406b4d0 0x7f408406d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f40a4160d00 tx=0x7f4098008040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a40cf830 msgr2=0x7f40a40b69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a40cf830 0x7f40a40b69c0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f409c009e50 tx=0x7f409c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 shutdown_connections 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f40a415dfe0 0x7f40a40cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f40a40d27f0 0x7f40a40cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f408406b4d0 0x7f408406d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 --2- 172.21.15.77:0/3979664788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40a40cf830 0x7f40a40b69c0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 >> 172.21.15.77:0/3979664788 conn(0x7f40a40d3cd0 msgr2=0x7f40a406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 shutdown_connections 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.737+0000 7f40a8b0f640 1 -- 172.21.15.77:0/3979664788 wait complete. 2024-03-06T07:27:15.750 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:03.116010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:03.812681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:15.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.853+0000 7f473f7f4640 1 Processor -- start 2024-03-06T07:27:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.853+0000 7f473f7f4640 1 -- start start 2024-03-06T07:27:15.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473f7f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f47381614d0 0x7f47380d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473f7f4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f47380d1910 0x7f47380d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473f7f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 0x7f47380d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473f7f4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47380d2750 con 0x7f47381614d0 2024-03-06T07:27:15.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473f7f4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47381618b0 con 0x7f47380d1910 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473f7f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47380cf400 con 0x7f47380d22e0 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473d590640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f47381614d0 0x7f47380d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473d590640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f47381614d0 0x7f47380d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36596/0 (socket says 172.21.15.77:36596) 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473d590640 1 -- 172.21.15.77:0/1834173740 learned_addr learned my addr 172.21.15.77:0/1834173740 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473dd91640 1 --2- 172.21.15.77:0/1834173740 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 0x7f47380d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473cd8f640 1 --2- 172.21.15.77:0/1834173740 >> v2:172.21.15.158:3300/0 conn(0x7f47380d1910 0x7f47380d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473d590640 1 -- 172.21.15.77:0/1834173740 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 msgr2=0x7f47380d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473d590640 1 --2- 172.21.15.77:0/1834173740 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 0x7f47380d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473d590640 1 -- 172.21.15.77:0/1834173740 >> v2:172.21.15.158:3300/0 conn(0x7f47380d1910 msgr2=0x7f47380d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473d590640 1 --2- 172.21.15.77:0/1834173740 >> v2:172.21.15.158:3300/0 conn(0x7f47380d1910 0x7f47380d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473d590640 1 -- 172.21.15.77:0/1834173740 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47380cf6a0 con 0x7f47381614d0 2024-03-06T07:27:15.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.854+0000 7f473cd8f640 1 --2- 172.21.15.77:0/1834173740 >> v2:172.21.15.158:3300/0 conn(0x7f47380d1910 0x7f47380d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.855+0000 7f473d590640 1 --2- 172.21.15.77:0/1834173740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47381614d0 0x7f47380d4860 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f47380ce550 tx=0x7f473402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf955f5bf0fd0a03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.855+0000 7f472ffff640 1 -- 172.21.15.77:0/1834173740 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734035020 con 0x7f47381614d0 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.855+0000 7f472ffff640 1 -- 172.21.15.77:0/1834173740 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734031870 con 0x7f47381614d0 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.855+0000 7f472ffff640 1 -- 172.21.15.77:0/1834173740 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734031c40 con 0x7f47381614d0 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 -- 172.21.15.77:0/1834173740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47381614d0 msgr2=0x7f47380d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1834173740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47381614d0 0x7f47380d4860 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f47380ce550 tx=0x7f473402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 -- 172.21.15.77:0/1834173740 shutdown_connections 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1834173740 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 0x7f47380d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1834173740 >> v2:172.21.15.158:3300/0 conn(0x7f47380d1910 0x7f47380d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1834173740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47381614d0 0x7f47380d4860 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 -- 172.21.15.77:0/1834173740 >> 172.21.15.77:0/1834173740 conn(0x7f47380d3d40 msgr2=0x7f47380d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 -- 172.21.15.77:0/1834173740 shutdown_connections 2024-03-06T07:27:15.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.856+0000 7f473f7f4640 1 -- 172.21.15.77:0/1834173740 wait complete. 2024-03-06T07:27:15.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.857+0000 7f473f7f4640 1 Processor -- start 2024-03-06T07:27:15.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.857+0000 7f473f7f4640 1 -- start start 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.858+0000 7f473f7f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47380d1910 0x7f47380c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.858+0000 7f473f7f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 0x7f4738151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.858+0000 7f473f7f4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47381614d0 0x7f47380c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.858+0000 7f473f7f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47380b0910 con 0x7f47380d1910 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.858+0000 7f473f7f4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47380cfc60 con 0x7f47381614d0 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.858+0000 7f473f7f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47380d0be0 con 0x7f47380d22e0 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47380d1910 0x7f47380c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47380d1910 0x7f47380c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36602/0 (socket says 172.21.15.77:36602) 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 -- 172.21.15.77:0/1041812725 learned_addr learned my addr 172.21.15.77:0/1041812725 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:15.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473cd8f640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 0x7f4738151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473dd91640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47381614d0 0x7f47380c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 -- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 msgr2=0x7f4738151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 0x7f4738151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 -- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47381614d0 msgr2=0x7f47380c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47381614d0 0x7f47380c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 -- 172.21.15.77:0/1041812725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47380b6ef0 con 0x7f47380d1910 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473dd91640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47381614d0 0x7f47380c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473d590640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47380d1910 0x7f47380c8ba0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f4734000c00 tx=0x7f47340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f472dffb640 1 -- 172.21.15.77:0/1041812725 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734035040 con 0x7f47380d1910 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f472dffb640 1 -- 172.21.15.77:0/1041812725 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734030d00 con 0x7f47380d1910 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47380b71d0 con 0x7f47380d1910 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.859+0000 7f472dffb640 1 -- 172.21.15.77:0/1041812725 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47340456a0 con 0x7f47380d1910 2024-03-06T07:27:15.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.860+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47380b7790 con 0x7f47380d1910 2024-03-06T07:27:15.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.861+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47380a1c90 con 0x7f47380d1910 2024-03-06T07:27:15.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.861+0000 7f472dffb640 1 -- 172.21.15.77:0/1041812725 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4734038020 con 0x7f47380d1910 2024-03-06T07:27:15.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.861+0000 7f472dffb640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f471c06b4d0 0x7f471c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.861+0000 7f472dffb640 1 -- 172.21.15.77:0/1041812725 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47340ac500 con 0x7f47380d1910 2024-03-06T07:27:15.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.861+0000 7f473cd8f640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f471c06b4d0 0x7f471c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.862+0000 7f473cd8f640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f471c06b4d0 0x7f471c06d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4738160970 tx=0x7f4728008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:15.870+0000 7f472dffb640 1 -- 172.21.15.77:0/1041812725 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f473407b2e0 con 0x7f47380d1910 2024-03-06T07:27:16.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.047+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4738068ff0 con 0x7f47380d1910 2024-03-06T07:27:16.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.048+0000 7f472dffb640 1 -- 172.21.15.77:0/1041812725 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f473407abe0 con 0x7f47380d1910 2024-03-06T07:27:16.050 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:27:16.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f471c06b4d0 msgr2=0x7f471c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f471c06b4d0 0x7f471c06d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4738160970 tx=0x7f4728008040 comp rx=0 tx=0).stop 2024-03-06T07:27:16.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47380d1910 msgr2=0x7f47380c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47380d1910 0x7f47380c8ba0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f4734000c00 tx=0x7f47340046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 shutdown_connections 2024-03-06T07:27:16.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47381614d0 0x7f47380c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f47380d22e0 0x7f4738151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f471c06b4d0 0x7f471c06d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 --2- 172.21.15.77:0/1041812725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47380d1910 0x7f47380c8ba0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 >> 172.21.15.77:0/1041812725 conn(0x7f47380d3d40 msgr2=0x7f47380ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 shutdown_connections 2024-03-06T07:27:16.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.050+0000 7f473f7f4640 1 -- 172.21.15.77:0/1041812725 wait complete. 2024-03-06T07:27:16.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:16.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.164+0000 7f59b2165640 1 Processor -- start 2024-03-06T07:27:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.164+0000 7f59b2165640 1 -- start start 2024-03-06T07:27:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.165+0000 7f59b2165640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0d4860 0x7f59ac0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.165+0000 7f59b2165640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59ac0b0e30 0x7f59ac0b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.165+0000 7f59b2165640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ac1618b0 con 0x7f59ac1614d0 2024-03-06T07:27:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.165+0000 7f59b2165640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59ac0d4cc0 con 0x7f59ac0b0e30 2024-03-06T07:27:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.165+0000 7f59b2165640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0cf400 con 0x7f59ac0d4860 2024-03-06T07:27:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.166+0000 7f59abfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59ac0b0e30 0x7f59ac0b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.166+0000 7f59abfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59ac0b0e30 0x7f59ac0b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36610/0 (socket says 172.21.15.77:36610) 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.166+0000 7f59abfff640 1 -- 172.21.15.77:0/2005441793 learned_addr learned my addr 172.21.15.77:0/2005441793 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.166+0000 7f59aaffd640 1 --2- 172.21.15.77:0/2005441793 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0d4860 0x7f59ac0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.166+0000 7f59abfff640 1 -- 172.21.15.77:0/2005441793 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0d4860 msgr2=0x7f59ac0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.166+0000 7f59abfff640 1 --2- 172.21.15.77:0/2005441793 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0d4860 0x7f59ac0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.166+0000 7f59abfff640 1 -- 172.21.15.77:0/2005441793 >> v1:172.21.15.158:6789/0 conn(0x7f59ac1614d0 legacy=0x7f59ac0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.166+0000 7f59abfff640 1 -- 172.21.15.77:0/2005441793 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac0cf6a0 con 0x7f59ac0b0e30 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59abfff640 1 --2- 172.21.15.77:0/2005441793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0b0e30 0x7f59ac0b12c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f599c009870 tx=0x7f599c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68fa1d76559294f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59aa7fc640 1 -- 172.21.15.77:0/2005441793 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c034070 con 0x7f59ac0b0e30 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59aa7fc640 1 -- 172.21.15.77:0/2005441793 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c030840 con 0x7f59ac0b0e30 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59aa7fc640 1 -- 172.21.15.77:0/2005441793 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c030c10 con 0x7f59ac0b0e30 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59b2165640 1 -- 172.21.15.77:0/2005441793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0b0e30 msgr2=0x7f59ac0b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59b2165640 1 --2- 172.21.15.77:0/2005441793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0b0e30 0x7f59ac0b12c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f599c009870 tx=0x7f599c02b980 comp rx=0 tx=0).stop 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59b2165640 1 -- 172.21.15.77:0/2005441793 shutdown_connections 2024-03-06T07:27:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59b2165640 1 --2- 172.21.15.77:0/2005441793 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0d4860 0x7f59ac0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59b2165640 1 --2- 172.21.15.77:0/2005441793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0b0e30 0x7f59ac0b12c0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59b2165640 1 -- 172.21.15.77:0/2005441793 >> 172.21.15.77:0/2005441793 conn(0x7f59ac0d3d40 msgr2=0x7f59ac0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59b2165640 1 -- 172.21.15.77:0/2005441793 shutdown_connections 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.167+0000 7f59b2165640 1 -- 172.21.15.77:0/2005441793 wait complete. 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.169+0000 7f59b2165640 1 Processor -- start 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.169+0000 7f59b2165640 1 -- start start 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.169+0000 7f59b2165640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0b0e30 0x7f59ac1f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.169+0000 7f59b2165640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0d4860 0x7f59ac0b8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.169+0000 7f59b2165640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59ac1614d0 0x7f59ac1f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.169+0000 7f59b2165640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0b0910 con 0x7f59ac0d4860 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.169+0000 7f59b2165640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0cfc60 con 0x7f59ac1614d0 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.169+0000 7f59b2165640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0d2560 con 0x7f59ac0b0e30 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0d4860 0x7f59ac0b8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0d4860 0x7f59ac0b8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36624/0 (socket says 172.21.15.77:36624) 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 -- 172.21.15.77:0/2979729792 learned_addr learned my addr 172.21.15.77:0/2979729792 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59abfff640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59ac1614d0 0x7f59ac1f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59ab7fe640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0b0e30 0x7f59ac1f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 -- 172.21.15.77:0/2979729792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0b0e30 msgr2=0x7f59ac1f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0b0e30 0x7f59ac1f3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 -- 172.21.15.77:0/2979729792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59ac1614d0 msgr2=0x7f59ac1f6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59ac1614d0 0x7f59ac1f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 -- 172.21.15.77:0/2979729792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac0c7a10 con 0x7f59ac0d4860 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59aaffd640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0d4860 0x7f59ac0b8950 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f599400e9b0 tx=0x7f599400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59a3fff640 1 -- 172.21.15.77:0/2979729792 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994017070 con 0x7f59ac0d4860 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59a3fff640 1 -- 172.21.15.77:0/2979729792 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599400c9a0 con 0x7f59ac0d4860 2024-03-06T07:27:16.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ac0c7cf0 con 0x7f59ac0d4860 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59a3fff640 1 -- 172.21.15.77:0/2979729792 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599400cd90 con 0x7f59ac0d4860 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.170+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ac0c82b0 con 0x7f59ac0d4860 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.171+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ac0d42f0 con 0x7f59ac0d4860 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.172+0000 7f59a3fff640 1 -- 172.21.15.77:0/2979729792 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f599401a070 con 0x7f59ac0d4860 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.172+0000 7f59a3fff640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f598c06b4d0 0x7f598c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.173+0000 7f59a3fff640 1 -- 172.21.15.77:0/2979729792 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f599408a670 con 0x7f59ac0d4860 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.173+0000 7f59ab7fe640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f598c06b4d0 0x7f598c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.173+0000 7f59ab7fe640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f598c06b4d0 0x7f598c06d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f59a4007fd0 tx=0x7f59a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.177+0000 7f59a3fff640 1 -- 172.21.15.77:0/2979729792 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59ac0d42f0 con 0x7f59ac0d4860 2024-03-06T07:27:16.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.354+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f59ac06d520 con 0x7f59ac0d4860 2024-03-06T07:27:16.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.354+0000 7f59a3fff640 1 -- 172.21.15.77:0/2979729792 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5994059460 con 0x7f59ac0d4860 2024-03-06T07:27:16.356 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:27:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f598c06b4d0 msgr2=0x7f598c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f598c06b4d0 0x7f598c06d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f59a4007fd0 tx=0x7f59a4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0d4860 msgr2=0x7f59ac0b8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0d4860 0x7f59ac0b8950 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f599400e9b0 tx=0x7f599400ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 shutdown_connections 2024-03-06T07:27:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f598c06b4d0 0x7f598c06d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ac0d4860 0x7f59ac0b8950 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f59ac1614d0 0x7f59ac1f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 --2- 172.21.15.77:0/2979729792 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f59ac0b0e30 0x7f59ac1f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 >> 172.21.15.77:0/2979729792 conn(0x7f59ac0d3d40 msgr2=0x7f59ac06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 shutdown_connections 2024-03-06T07:27:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.357+0000 7f59b2165640 1 -- 172.21.15.77:0/2979729792 wait complete. 2024-03-06T07:27:16.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:16.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.488+0000 7f9939888640 1 Processor -- start 2024-03-06T07:27:16.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.488+0000 7f9939888640 1 -- start start 2024-03-06T07:27:16.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f9939888640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f993406f430 0x7f99340cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f9939888640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993406b6c0 con 0x7f9934145590 2024-03-06T07:27:16.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f9939888640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993406b970 con 0x7f993406efd0 2024-03-06T07:27:16.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f9939888640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9934145970 con 0x7f993406f430 2024-03-06T07:27:16.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f9932ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9934145590 0x7f993406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38234/0 (socket says 172.21.15.77:38234) 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f9932ffd640 1 -- 172.21.15.77:0/446941730 learned_addr learned my addr 172.21.15.77:0/446941730 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720060575 0 0) 0x7f993406b6c0 con 0x7f9934145590 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9920003520 con 0x7f9934145590 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99337fe640 1 --2- 172.21.15.77:0/446941730 >> v2:172.21.15.158:3300/0 conn(0x7f993406f430 0x7f99340cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3404570087 0 0) 0x7f9920003520 con 0x7f9934145590 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 >> v2:172.21.15.158:3300/0 conn(0x7f993406f430 msgr2=0x7f99340cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 --2- 172.21.15.77:0/446941730 >> v2:172.21.15.158:3300/0 conn(0x7f993406f430 0x7f99340cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 >> v1:172.21.15.158:6789/0 conn(0x7f993406efd0 legacy=0x7f99340cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993415f7e0 con 0x7f9934145590 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f9924001670 con 0x7f9934145590 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 >> v1:172.21.15.77:6789/0 conn(0x7f9934145590 legacy=0x7f993406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99200039b0 0x7f9920003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f9920004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99200051c0 0x7f992000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9934145970 con 0x7f9920004800 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992000c0f0 con 0x7f99200039b0 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992000c2f0 con 0x7f99200051c0 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.489+0000 7f992bfff640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f9920004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.490+0000 7f9932ffd640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99200039b0 0x7f9920003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.490+0000 7f99337fe640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99200051c0 0x7f992000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.490+0000 7f992bfff640 1 -- 172.21.15.77:0/446941730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99200051c0 msgr2=0x7f992000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.490+0000 7f992bfff640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99200051c0 0x7f992000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.490+0000 7f992bfff640 1 -- 172.21.15.77:0/446941730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99200039b0 msgr2=0x7f9920003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.490+0000 7f992bfff640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99200039b0 0x7f9920003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.490+0000 7f99337fe640 1 -- 172.21.15.77:0/446941730 reap_dead start 2024-03-06T07:27:16.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.490+0000 7f992bfff640 1 -- 172.21.15.77:0/446941730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9920003520 con 0x7f9920004800 2024-03-06T07:27:16.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.491+0000 7f992bfff640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f9920004c80 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f991c003d80 tx=0x7f9920005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e8092025be9f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.491+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c037070 con 0x7f9920004800 2024-03-06T07:27:16.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.491+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991c032600 con 0x7f9920004800 2024-03-06T07:27:16.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.491+0000 7f99327fc640 1 -- 172.21.15.77:0/446941730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c003980 con 0x7f9920004800 2024-03-06T07:27:16.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.492+0000 7f9939888640 1 -- 172.21.15.77:0/446941730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 msgr2=0x7f9920004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.492+0000 7f9939888640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f9920004c80 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f991c003d80 tx=0x7f9920005830 comp rx=0 tx=0).stop 2024-03-06T07:27:16.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.492+0000 7f9939888640 1 -- 172.21.15.77:0/446941730 shutdown_connections 2024-03-06T07:27:16.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.492+0000 7f9939888640 1 --2- 172.21.15.77:0/446941730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f9920004c80 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.492+0000 7f9939888640 1 -- 172.21.15.77:0/446941730 >> 172.21.15.77:0/446941730 conn(0x7f99340b2660 msgr2=0x7f99340b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.492+0000 7f9939888640 1 -- 172.21.15.77:0/446941730 shutdown_connections 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.492+0000 7f9939888640 1 -- 172.21.15.77:0/446941730 wait complete. 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.493+0000 7f9939888640 1 Processor -- start 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.493+0000 7f9939888640 1 -- start start 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9939888640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f99341e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9939888640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99340b54c0 0x7f993415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9939888640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f993415ac70 0x7f993415b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9939888640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993415fda0 con 0x7f9920004800 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9939888640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9934160110 con 0x7f99340b54c0 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9939888640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993406f8c0 con 0x7f993415ac70 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f99341e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f99341e54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36638/0 (socket says 172.21.15.77:36638) 2024-03-06T07:27:16.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 -- 172.21.15.77:0/259907674 learned_addr learned my addr 172.21.15.77:0/259907674 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f992bfff640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99340b54c0 0x7f993415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f99337fe640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f993415ac70 0x7f993415b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 -- 172.21.15.77:0/259907674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f993415ac70 msgr2=0x7f993415b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f993415ac70 0x7f993415b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 -- 172.21.15.77:0/259907674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99340b54c0 msgr2=0x7f993415e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99340b54c0 0x7f993415e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 -- 172.21.15.77:0/259907674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993415b5f0 con 0x7f9920004800 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9932ffd640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f99341e54c0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f99240072c0 tx=0x7f9924007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9930ff9640 1 -- 172.21.15.77:0/259907674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992400f960 con 0x7f9920004800 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9930ff9640 1 -- 172.21.15.77:0/259907674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9924018070 con 0x7f9920004800 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99341e6040 con 0x7f9920004800 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9930ff9640 1 -- 172.21.15.77:0/259907674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992400fd80 con 0x7f9920004800 2024-03-06T07:27:16.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.494+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99341e6600 con 0x7f9920004800 2024-03-06T07:27:16.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.495+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99340733b0 con 0x7f9920004800 2024-03-06T07:27:16.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.496+0000 7f9930ff9640 1 -- 172.21.15.77:0/259907674 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f992401d460 con 0x7f9920004800 2024-03-06T07:27:16.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.496+0000 7f9930ff9640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f991406b4d0 0x7f991406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.496+0000 7f9930ff9640 1 -- 172.21.15.77:0/259907674 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f992408a260 con 0x7f9920004800 2024-03-06T07:27:16.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.496+0000 7f992bfff640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f991406b4d0 0x7f991406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.497+0000 7f992bfff640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f991406b4d0 0x7f991406d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f991c034000 tx=0x7f991c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.500+0000 7f9930ff9640 1 -- 172.21.15.77:0/259907674 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9924059110 con 0x7f9920004800 2024-03-06T07:27:16.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.675+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9934068ff0 con 0x7f9920004800 2024-03-06T07:27:16.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.675+0000 7f9930ff9640 1 -- 172.21.15.77:0/259907674 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7f992408a080 con 0x7f9920004800 2024-03-06T07:27:16.677 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:27:16.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f991406b4d0 msgr2=0x7f991406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f991406b4d0 0x7f991406d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f991c034000 tx=0x7f991c009030 comp rx=0 tx=0).stop 2024-03-06T07:27:16.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 msgr2=0x7f99341e54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f99341e54c0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f99240072c0 tx=0x7f9924007780 comp rx=0 tx=0).stop 2024-03-06T07:27:16.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 shutdown_connections 2024-03-06T07:27:16.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f993415ac70 0x7f993415b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99340b54c0 0x7f993415e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f991406b4d0 0x7f991406d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 --2- 172.21.15.77:0/259907674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9920004800 0x7f99341e54c0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 >> 172.21.15.77:0/259907674 conn(0x7f99340b2660 msgr2=0x7f993415c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 shutdown_connections 2024-03-06T07:27:16.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.677+0000 7f9939888640 1 -- 172.21.15.77:0/259907674 wait complete. 2024-03-06T07:27:16.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:16.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.789+0000 7f2feb80e640 1 Processor -- start 2024-03-06T07:27:16.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.789+0000 7f2feb80e640 1 -- start start 2024-03-06T07:27:16.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.790+0000 7f2feb80e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 0x7f2fe40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.790+0000 7f2feb80e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe40d22c0 con 0x7f2fe40b4460 2024-03-06T07:27:16.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.790+0000 7f2feb80e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe40d25a0 con 0x7f2fe4161cf0 2024-03-06T07:27:16.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.790+0000 7f2feb80e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4161b00 con 0x7f2fe4162970 2024-03-06T07:27:16.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.790+0000 7f2fe95aa640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2fe40b4460 0x7f2fe40b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38246/0 (socket says 172.21.15.77:38246) 2024-03-06T07:27:16.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.790+0000 7f2fe95aa640 1 -- 172.21.15.77:0/1502628396 learned_addr learned my addr 172.21.15.77:0/1502628396 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.790+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422948012 0 0) 0x7f2fe40d22c0 con 0x7f2fe40b4460 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc003520 con 0x7f2fe40b4460 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe8da9640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 0x7f2fe40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3896459405 0 0) 0x7f2fcc003520 con 0x7f2fe40b4460 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 msgr2=0x7f2fe40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 0x7f2fe40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 >> v1:172.21.15.158:6789/0 conn(0x7f2fe4161cf0 legacy=0x7f2fe40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40d2880 con 0x7f2fe40b4460 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f2fd40019f0 con 0x7f2fe40b4460 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 >> v1:172.21.15.77:6789/0 conn(0x7f2fe40b4460 legacy=0x7f2fe40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc0039e0 0x7f2fcc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc004800 0x7f2fcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 0x7f2fcc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4161b00 con 0x7f2fcc004800 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc00baf0 con 0x7f2fcc0039e0 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc00bcf0 con 0x7f2fe4162970 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe8da9640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc004800 0x7f2fcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe95aa640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc0039e0 0x7f2fcc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe9dab640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 0x7f2fcc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe8da9640 1 -- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 msgr2=0x7f2fcc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe8da9640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 0x7f2fcc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe8da9640 1 -- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc0039e0 msgr2=0x7f2fcc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe8da9640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc0039e0 0x7f2fcc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe8da9640 1 -- 172.21.15.77:0/1502628396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc003520 con 0x7f2fcc004800 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.791+0000 7f2fe9dab640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4162970 0x7f2fcc005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.792+0000 7f2fe8da9640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc004800 0x7f2fcc004c80 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800ba00 tx=0x7f2fcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c468ce2867e68b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.792+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8036020 con 0x7f2fcc004800 2024-03-06T07:27:16.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.792+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd8032620 con 0x7f2fcc004800 2024-03-06T07:27:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.792+0000 7f2fe3fff640 1 -- 172.21.15.77:0/1502628396 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd800b600 con 0x7f2fcc004800 2024-03-06T07:27:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.793+0000 7f2feb80e640 1 -- 172.21.15.77:0/1502628396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc004800 msgr2=0x7f2fcc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.793+0000 7f2feb80e640 1 --2- 172.21.15.77:0/1502628396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc004800 0x7f2fcc004c80 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800ba00 tx=0x7f2fcc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.793+0000 7f2fe9dab640 1 -- 172.21.15.77:0/1502628396 reap_dead start 2024-03-06T07:27:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.793+0000 7f2feb80e640 1 -- 172.21.15.77:0/1502628396 shutdown_connections 2024-03-06T07:27:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.793+0000 7f2feb80e640 1 -- 172.21.15.77:0/1502628396 >> 172.21.15.77:0/1502628396 conn(0x7f2fe40d3ce0 msgr2=0x7f2fe40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.793+0000 7f2feb80e640 1 -- 172.21.15.77:0/1502628396 shutdown_connections 2024-03-06T07:27:16.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.793+0000 7f2feb80e640 1 -- 172.21.15.77:0/1502628396 wait complete. 2024-03-06T07:27:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.794+0000 7f2feb80e640 1 Processor -- start 2024-03-06T07:27:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.794+0000 7f2feb80e640 1 -- start start 2024-03-06T07:27:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2feb80e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc004800 0x7f2fe415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2feb80e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4155e90 0x7f2fe40b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2feb80e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe415ecf0 0x7f2fe41552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2feb80e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40ce950 con 0x7f2fe415ecf0 2024-03-06T07:27:16.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2feb80e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40cecc0 con 0x7f2fcc004800 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2feb80e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40b4840 con 0x7f2fe4155e90 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe415ecf0 0x7f2fe41552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe415ecf0 0x7f2fe41552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36654/0 (socket says 172.21.15.77:36654) 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 -- 172.21.15.77:0/20935234 learned_addr learned my addr 172.21.15.77:0/20935234 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe95aa640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc004800 0x7f2fe415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe8da9640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4155e90 0x7f2fe40b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 -- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4155e90 msgr2=0x7f2fe40b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4155e90 0x7f2fe40b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 -- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc004800 msgr2=0x7f2fe415a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc004800 0x7f2fe415a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 -- 172.21.15.77:0/20935234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40c7a60 con 0x7f2fe415ecf0 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe8da9640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4155e90 0x7f2fe40b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:16.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe9dab640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe415ecf0 0x7f2fe41552b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40d17e0 tx=0x7f2fdc00e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe1ffb640 1 -- 172.21.15.77:0/20935234 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc019070 con 0x7f2fe415ecf0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe1ffb640 1 -- 172.21.15.77:0/20935234 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc0052e0 con 0x7f2fe415ecf0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2fe1ffb640 1 -- 172.21.15.77:0/20935234 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc003830 con 0x7f2fe415ecf0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.795+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe40c7d40 con 0x7f2fe415ecf0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.796+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe40b8950 con 0x7f2fe415ecf0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.797+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe404f330 con 0x7f2fe415ecf0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.797+0000 7f2fe1ffb640 1 -- 172.21.15.77:0/20935234 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fdc01c070 con 0x7f2fe415ecf0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.797+0000 7f2fe1ffb640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fd006b4d0 0x7f2fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.797+0000 7f2fe1ffb640 1 -- 172.21.15.77:0/20935234 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fdc089550 con 0x7f2fe415ecf0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.797+0000 7f2fe95aa640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fd006b4d0 0x7f2fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.798+0000 7f2fe95aa640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fd006b4d0 0x7f2fd006d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4000f80 tx=0x7f2fd4006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.802+0000 7f2fe1ffb640 1 -- 172.21.15.77:0/20935234 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fdc057d80 con 0x7f2fe415ecf0 2024-03-06T07:27:16.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.993+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2fe406b180 con 0x7f2fe415ecf0 2024-03-06T07:27:16.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.993+0000 7f2fe1ffb640 1 -- 172.21.15.77:0/20935234 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v107) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2fdc05be20 con 0x7f2fe415ecf0 2024-03-06T07:27:16.995 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:27:16.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fd006b4d0 msgr2=0x7f2fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fd006b4d0 0x7f2fd006d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4000f80 tx=0x7f2fd4006040 comp rx=0 tx=0).stop 2024-03-06T07:27:16.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe415ecf0 msgr2=0x7f2fe41552b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe415ecf0 0x7f2fe41552b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40d17e0 tx=0x7f2fdc00e820 comp rx=0 tx=0).stop 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 shutdown_connections 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2fd006b4d0 0x7f2fd006d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe415ecf0 0x7f2fe41552b0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2fe4155e90 0x7f2fe40b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 --2- 172.21.15.77:0/20935234 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2fcc004800 0x7f2fe415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 >> 172.21.15.77:0/20935234 conn(0x7f2fe40d3ce0 msgr2=0x7f2fe40d0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 shutdown_connections 2024-03-06T07:27:16.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:16.995+0000 7f2feb80e640 1 -- 172.21.15.77:0/20935234 wait complete. 2024-03-06T07:27:17.008 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:27:17.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:17.008 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:17.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[72506]: fuse finished with error 0 and tester_r 0 2024-03-06T07:27:17.036 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:17.112 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:27:17.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:17.113 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:27:17.127 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:17.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.267+0000 7f8c404a4640 1 Processor -- start 2024-03-06T07:27:17.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.267+0000 7f8c404a4640 1 -- start start 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.268+0000 7f8c404a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 0x7f8c380cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.268+0000 7f8c404a4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8c380d29c0 0x7f8c380b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.268+0000 7f8c404a4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c380b3430 con 0x7f8c380cf830 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.268+0000 7f8c404a4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c380b31e0 con 0x7f8c380d29c0 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.268+0000 7f8c404a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c380b3d60 con 0x7f8c3815dfe0 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.268+0000 7f8c3da3f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8c380cf830 0x7f8c380d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38248/0 (socket says 172.21.15.77:38248) 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.268+0000 7f8c3da3f640 1 -- 172.21.15.77:0/1205627208 learned_addr learned my addr 172.21.15.77:0/1205627208 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3e240640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 0x7f8c380cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282714513 0 0) 0x7f8c380b3430 con 0x7f8c380cf830 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3ea41640 1 --2- 172.21.15.77:0/1205627208 >> v2:172.21.15.158:3300/0 conn(0x7f8c380d29c0 0x7f8c380b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c24003520 con 0x7f8c380cf830 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1084669543 0 0) 0x7f8c24003520 con 0x7f8c380cf830 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 msgr2=0x7f8c380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 0x7f8c380cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 >> v2:172.21.15.158:3300/0 conn(0x7f8c380d29c0 msgr2=0x7f8c380b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 --2- 172.21.15.77:0/1205627208 >> v2:172.21.15.158:3300/0 conn(0x7f8c380d29c0 0x7f8c380b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c380b3710 con 0x7f8c380cf830 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f8c28001670 con 0x7f8c380cf830 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 >> v1:172.21.15.77:6789/0 conn(0x7f8c380cf830 legacy=0x7f8c380d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 0x7f8c24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c24004d40 0x7f8c240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c380b31e0 con 0x7f8c240039b0 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c380b3d60 con 0x7f8c24004d40 2024-03-06T07:27:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2400b9c0 con 0x7f8c3815dfe0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3e240640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3da3f640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 0x7f8c24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.269+0000 7f8c3ea41640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c24004d40 0x7f8c240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3e240640 1 -- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 msgr2=0x7f8c24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3e240640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 0x7f8c24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3e240640 1 -- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c24004d40 msgr2=0x7f8c240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3e240640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c24004d40 0x7f8c240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3e240640 1 -- 172.21.15.77:0/1205627208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c24003520 con 0x7f8c240039b0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3da3f640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c3815dfe0 0x7f8c24004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3ea41640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c24004d40 0x7f8c240051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3e240640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c24003dd0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009ac0 tx=0x7f8c24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2f7b8bb1f42219 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34039070 con 0x7f8c240039b0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3402c6b0 con 0x7f8c240039b0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.270+0000 7f8c3d23e640 1 -- 172.21.15.77:0/1205627208 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34009690 con 0x7f8c240039b0 2024-03-06T07:27:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.271+0000 7f8c404a4640 1 -- 172.21.15.77:0/1205627208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 msgr2=0x7f8c24003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.271+0000 7f8c404a4640 1 --2- 172.21.15.77:0/1205627208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c24003dd0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009ac0 tx=0x7f8c24005830 comp rx=0 tx=0).stop 2024-03-06T07:27:17.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.271+0000 7f8c3ea41640 1 -- 172.21.15.77:0/1205627208 reap_dead start 2024-03-06T07:27:17.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.271+0000 7f8c404a4640 1 -- 172.21.15.77:0/1205627208 shutdown_connections 2024-03-06T07:27:17.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.271+0000 7f8c404a4640 1 -- 172.21.15.77:0/1205627208 >> 172.21.15.77:0/1205627208 conn(0x7f8c380d3cd0 msgr2=0x7f8c380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.271+0000 7f8c404a4640 1 -- 172.21.15.77:0/1205627208 shutdown_connections 2024-03-06T07:27:17.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.271+0000 7f8c404a4640 1 -- 172.21.15.77:0/1205627208 wait complete. 2024-03-06T07:27:17.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.272+0000 7f8c404a4640 1 Processor -- start 2024-03-06T07:27:17.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.272+0000 7f8c404a4640 1 -- start start 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c404a4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c3815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c404a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c380b8e70 0x7f8c380b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c404a4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c381f3a00 0x7f8c380cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c404a4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c380d1df0 con 0x7f8c240039b0 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c404a4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c380d1c70 con 0x7f8c381f3a00 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c404a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3815e3c0 con 0x7f8c380b8e70 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c3815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c3815c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36668/0 (socket says 172.21.15.77:36668) 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 -- 172.21.15.77:0/1462699175 learned_addr learned my addr 172.21.15.77:0/1462699175 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 -- 172.21.15.77:0/1462699175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c380b8e70 msgr2=0x7f8c380b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c380b8e70 0x7f8c380b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 -- 172.21.15.77:0/1462699175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c381f3a00 msgr2=0x7f8c380cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c381f3a00 0x7f8c380cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 -- 172.21.15.77:0/1462699175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c380cae40 con 0x7f8c240039b0 2024-03-06T07:27:17.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.273+0000 7f8c3e240640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c3815c260 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f8c34000c00 tx=0x7f8c34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.274+0000 7f8c2effd640 1 -- 172.21.15.77:0/1462699175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34042020 con 0x7f8c240039b0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.274+0000 7f8c2effd640 1 -- 172.21.15.77:0/1462699175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3403a040 con 0x7f8c240039b0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.274+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c380cb120 con 0x7f8c240039b0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.274+0000 7f8c2effd640 1 -- 172.21.15.77:0/1462699175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34039070 con 0x7f8c240039b0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.274+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c380cb6e0 con 0x7f8c240039b0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.275+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3804f330 con 0x7f8c240039b0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.275+0000 7f8c2effd640 1 -- 172.21.15.77:0/1462699175 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c34045020 con 0x7f8c240039b0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.275+0000 7f8c2effd640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c1c06b4d0 0x7f8c1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.276+0000 7f8c2effd640 1 -- 172.21.15.77:0/1462699175 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c340ab600 con 0x7f8c240039b0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.276+0000 7f8c3da3f640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c1c06b4d0 0x7f8c1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.276+0000 7f8c3da3f640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c1c06b4d0 0x7f8c1c06d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8c3806a0b0 tx=0x7f8c28001c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.280+0000 7f8c2effd640 1 -- 172.21.15.77:0/1462699175 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c3404c070 con 0x7f8c240039b0 2024-03-06T07:27:17.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.497+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c3815c7a0 con 0x7f8c240039b0 2024-03-06T07:27:17.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.497+0000 7f8c2effd640 1 -- 172.21.15.77:0/1462699175 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f8c3407a480 con 0x7f8c240039b0 2024-03-06T07:27:17.499 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:17.499 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/582596329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":582596329},{"type":"v1","addr":"172.21.15.158:6834","nonce":582596329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3505002958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3505002958},{"type":"v1","addr":"172.21.15.158:6835","nonce":3505002958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1169947364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1169947364},{"type":"v1","addr":"172.21.15.77:6835","nonce":1169947364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:07.450568+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7827":{"gid":7827,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/3247585937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3247585937},{"type":"v1","addr":"172.21.15.77:6837","nonce":3247585937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7827,"qdb_cluster":[7827]},"id":8}]} 2024-03-06T07:27:17.500 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 66 2024-03-06T07:27:17.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c1c06b4d0 msgr2=0x7f8c1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c1c06b4d0 0x7f8c1c06d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8c3806a0b0 tx=0x7f8c28001c90 comp rx=0 tx=0).stop 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 msgr2=0x7f8c3815c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c3815c260 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f8c34000c00 tx=0x7f8c34004730 comp rx=0 tx=0).stop 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 shutdown_connections 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c381f3a00 0x7f8c380cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c380b8e70 0x7f8c380b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c1c06b4d0 0x7f8c1c06d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 --2- 172.21.15.77:0/1462699175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c240039b0 0x7f8c3815c260 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 >> 172.21.15.77:0/1462699175 conn(0x7f8c380d3cd0 msgr2=0x7f8c380b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 shutdown_connections 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.499+0000 7f8c404a4640 1 -- 172.21.15.77:0/1462699175 wait complete. 2024-03-06T07:27:17.512 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:27:17.637 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:27:17.637 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:17.638 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:17.638 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:17.638 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:17.639 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:17.639 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:17.639 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:17.639 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:17.639 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:17.647 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:17.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:27:17.648 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:17.648 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:27:17.651 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:17.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:27:17.652 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:17.652 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:27:17.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.792+0000 7ff0b45e9640 1 Processor -- start 2024-03-06T07:27:17.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.792+0000 7ff0b45e9640 1 -- start start 2024-03-06T07:27:17.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.792+0000 7f339c4df640 1 Processor -- start 2024-03-06T07:27:17.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.792+0000 7f339c4df640 1 -- start start 2024-03-06T07:27:17.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.792+0000 7f2f8666b640 1 Processor -- start 2024-03-06T07:27:17.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.792+0000 7f2f8666b640 1 -- start start 2024-03-06T07:27:17.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7ff00270f640 1 Processor -- start 2024-03-06T07:27:17.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7ff00270f640 1 -- start start 2024-03-06T07:27:17.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f2f8666b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f80162970 0x7f2f800d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f2f8666b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f80161cf0 0x7f2f80162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f339c4df640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f339415b380 0x7f33940708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f339c4df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3394070e10 0x7f33940cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f339c4df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339406cd60 con 0x7f3394158880 2024-03-06T07:27:17.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f339c4df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3394071280 con 0x7f3394070e10 2024-03-06T07:27:17.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f339c4df640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339406d030 con 0x7f339415b380 2024-03-06T07:27:17.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f2f8666b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f800d22c0 con 0x7f2f800b4460 2024-03-06T07:27:17.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f2f8666b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f80161b00 con 0x7f2f80161cf0 2024-03-06T07:27:17.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f2f8666b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f800cecc0 con 0x7f2f80162970 2024-03-06T07:27:17.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7ff0b45e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0ac15b5b0 0x7ff0ac0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7ff0b45e9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac0cea50 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7f339aa7c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3394070e10 0x7f33940cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f339aa7c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3394070e10 0x7f33940cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36692/0 (socket says 172.21.15.77:36692) 2024-03-06T07:27:17.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f339aa7c640 1 -- 172.21.15.77:0/3740265321 learned_addr learned my addr 172.21.15.77:0/3740265321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:17.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff00270f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc072520 0x7feffc066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff00270f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7feffc0b5c30 0x7feffc072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f7f7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f80162970 0x7f2f800d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f84c08640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f80161cf0 0x7f2f80162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f7ffff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2f800b4460 0x7f2f800b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:60672/0 (socket says 172.21.15.77:60672) 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f7f7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f80162970 0x7f2f800d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:48712/0 (socket says 172.21.15.77:48712) 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f84c08640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f80161cf0 0x7f2f80162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36702/0 (socket says 172.21.15.77:36702) 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff00270f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc072e40 con 0x7feffc15d4f0 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff00270f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc067240 con 0x7feffc0b5c30 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff00270f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc06c460 con 0x7feffc072520 2024-03-06T07:27:17.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f3399a7a640 1 --2- 172.21.15.77:0/3740265321 >> v2:172.21.15.158:3300/0 conn(0x7f339415b380 0x7f33940708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f7ffff640 1 -- 172.21.15.77:0/1981995191 learned_addr learned my addr 172.21.15.77:0/1981995191 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:17.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7feffbfff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc072520 0x7feffc066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7feffbfff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc072520 0x7feffc066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:48724/0 (socket says 172.21.15.77:48724) 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7ff0b45e9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac0cca00 con 0x7ff0ac0b2950 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7ff0b45e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ac15ba40 con 0x7ff0ac15b5b0 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.793+0000 7ff0b1b84640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0ac15b5b0 0x7ff0ac0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff0b1b84640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0ac15b5b0 0x7ff0ac0cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36680/0 (socket says 172.21.15.77:36680) 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff0b1b84640 1 -- 172.21.15.77:0/1246752911 learned_addr learned my addr 172.21.15.77:0/1246752911 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7feffbfff640 1 -- 172.21.15.77:0/1639574660 learned_addr learned my addr 172.21.15.77:0/1639574660 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7feff3fff640 1 --2- 172.21.15.77:0/1639574660 >> v2:172.21.15.158:3300/0 conn(0x7feffc0b5c30 0x7feffc072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f339aa7c640 1 -- 172.21.15.77:0/3740265321 >> v2:172.21.15.158:3300/0 conn(0x7f339415b380 msgr2=0x7f33940708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff0b1b84640 1 -- 172.21.15.77:0/1246752911 >> v1:172.21.15.158:6789/0 conn(0x7ff0ac0b2950 legacy=0x7ff0ac0b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff0b1b84640 1 -- 172.21.15.77:0/1246752911 >> v1:172.21.15.77:6789/0 conn(0x7ff0ac0b3c90 legacy=0x7ff0ac0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7ff0b1b84640 1 -- 172.21.15.77:0/1246752911 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac0ccce0 con 0x7ff0ac15b5b0 2024-03-06T07:27:17.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f339aa7c640 1 --2- 172.21.15.77:0/3740265321 >> v2:172.21.15.158:3300/0 conn(0x7f339415b380 0x7f33940708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f339aa7c640 1 -- 172.21.15.77:0/3740265321 >> v1:172.21.15.158:6789/0 conn(0x7f3394158880 legacy=0x7f339406c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f3399279640 1 -- 172.21.15.77:0/3740265321 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688976793 0 0) 0x7f339406cd60 con 0x7f3394158880 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f7f7fe640 1 -- 172.21.15.77:0/1981995191 >> v2:172.21.15.77:3300/0 conn(0x7f2f80161cf0 msgr2=0x7f2f80162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f7f7fe640 1 --2- 172.21.15.77:0/1981995191 >> v2:172.21.15.77:3300/0 conn(0x7f2f80161cf0 0x7f2f80162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f7f7fe640 1 -- 172.21.15.77:0/1981995191 >> v1:172.21.15.158:6789/0 conn(0x7f2f800b4460 legacy=0x7f2f800b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7f2f7f7fe640 1 -- 172.21.15.77:0/1981995191 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f800d25a0 con 0x7f2f80162970 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7feffbfff640 1 -- 172.21.15.77:0/1639574660 >> v2:172.21.15.158:3300/0 conn(0x7feffc0b5c30 msgr2=0x7feffc072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7feffbfff640 1 --2- 172.21.15.77:0/1639574660 >> v2:172.21.15.158:3300/0 conn(0x7feffc0b5c30 0x7feffc072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7feffbfff640 1 -- 172.21.15.77:0/1639574660 >> v1:172.21.15.158:6789/0 conn(0x7feffc15d4f0 legacy=0x7feffc15d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.794+0000 7feffbfff640 1 -- 172.21.15.77:0/1639574660 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc06c740 con 0x7feffc072520 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.795+0000 7f339aa7c640 1 -- 172.21.15.77:0/3740265321 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339406d310 con 0x7f3394070e10 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.795+0000 7ff0b1b84640 1 --2- 172.21.15.77:0/1246752911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac15b5b0 0x7ff0ac0cc2d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009870 tx=0x7ff0a802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0542e2c0bf9a55f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.795+0000 7ff0b1383640 1 -- 172.21.15.77:0/1246752911 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8034070 con 0x7ff0ac15b5b0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.795+0000 7ff0b1383640 1 -- 172.21.15.77:0/1246752911 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a8030860 con 0x7ff0ac15b5b0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.795+0000 7ff0b1383640 1 -- 172.21.15.77:0/1246752911 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8030c30 con 0x7ff0ac15b5b0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.795+0000 7f2f7f7fe640 1 --2- 172.21.15.77:0/1981995191 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f80162970 0x7f2f800d4800 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2f740097a0 tx=0x7f2f7402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc57ea59ee59e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.795+0000 7feffbfff640 1 --2- 172.21.15.77:0/1639574660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc072520 0x7feffc066d00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fefec009870 tx=0x7fefec02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e4268aebafacf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f2f7effd640 1 -- 172.21.15.77:0/1981995191 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74035020 con 0x7f2f80162970 2024-03-06T07:27:17.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f2f7effd640 1 -- 172.21.15.77:0/1981995191 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f74031830 con 0x7f2f80162970 2024-03-06T07:27:17.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f2f7effd640 1 -- 172.21.15.77:0/1981995191 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74031c00 con 0x7f2f80162970 2024-03-06T07:27:17.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7feffb7fe640 1 -- 172.21.15.77:0/1639574660 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec035020 con 0x7feffc072520 2024-03-06T07:27:17.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f339aa7c640 1 --2- 172.21.15.77:0/3740265321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3394070e10 0x7f33940cd7c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f3394072d80 tx=0x7f338c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26e23532fed176 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7feffb7fe640 1 -- 172.21.15.77:0/1639574660 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec031640 con 0x7feffc072520 2024-03-06T07:27:17.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7feffb7fe640 1 -- 172.21.15.77:0/1639574660 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec02d040 con 0x7feffc072520 2024-03-06T07:27:17.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f3399279640 1 -- 172.21.15.77:0/3740265321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c02cb20 con 0x7f3394070e10 2024-03-06T07:27:17.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7ff0b45e9640 1 -- 172.21.15.77:0/1246752911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac15b5b0 msgr2=0x7ff0ac0cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7ff0b45e9640 1 --2- 172.21.15.77:0/1246752911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac15b5b0 0x7ff0ac0cc2d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009870 tx=0x7ff0a802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f3399279640 1 -- 172.21.15.77:0/3740265321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c02ccc0 con 0x7f3394070e10 2024-03-06T07:27:17.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f3399279640 1 -- 172.21.15.77:0/3740265321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c034610 con 0x7f3394070e10 2024-03-06T07:27:17.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7ff0b45e9640 1 -- 172.21.15.77:0/1246752911 shutdown_connections 2024-03-06T07:27:17.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7ff0b45e9640 1 --2- 172.21.15.77:0/1246752911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac15b5b0 0x7ff0ac0cc2d0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7ff0b45e9640 1 -- 172.21.15.77:0/1246752911 >> 172.21.15.77:0/1246752911 conn(0x7ff0ac0b1c80 msgr2=0x7ff0ac0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7ff0b45e9640 1 -- 172.21.15.77:0/1246752911 shutdown_connections 2024-03-06T07:27:17.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7ff0b45e9640 1 -- 172.21.15.77:0/1246752911 wait complete. 2024-03-06T07:27:17.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f2f8666b640 1 -- 172.21.15.77:0/1981995191 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f80162970 msgr2=0x7f2f800d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.796+0000 7f2f8666b640 1 --2- 172.21.15.77:0/1981995191 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f80162970 0x7f2f800d4800 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2f740097a0 tx=0x7f2f7402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7ff00270f640 1 -- 172.21.15.77:0/1639574660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc072520 msgr2=0x7feffc066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7ff00270f640 1 --2- 172.21.15.77:0/1639574660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc072520 0x7feffc066d00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fefec009870 tx=0x7fefec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f2f8666b640 1 -- 172.21.15.77:0/1981995191 shutdown_connections 2024-03-06T07:27:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f2f8666b640 1 --2- 172.21.15.77:0/1981995191 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f80162970 0x7f2f800d4800 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f2f8666b640 1 --2- 172.21.15.77:0/1981995191 >> v2:172.21.15.77:3300/0 conn(0x7f2f80161cf0 0x7f2f80162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f2f8666b640 1 -- 172.21.15.77:0/1981995191 >> 172.21.15.77:0/1981995191 conn(0x7f2f800d3ce0 msgr2=0x7f2f800d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7ff00270f640 1 -- 172.21.15.77:0/1639574660 shutdown_connections 2024-03-06T07:27:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7ff00270f640 1 --2- 172.21.15.77:0/1639574660 >> v2:172.21.15.158:3300/0 conn(0x7feffc0b5c30 0x7feffc072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f2f8666b640 1 -- 172.21.15.77:0/1981995191 shutdown_connections 2024-03-06T07:27:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7ff00270f640 1 --2- 172.21.15.77:0/1639574660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc072520 0x7feffc066d00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f2f8666b640 1 -- 172.21.15.77:0/1981995191 wait complete. 2024-03-06T07:27:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7ff00270f640 1 -- 172.21.15.77:0/1639574660 >> 172.21.15.77:0/1639574660 conn(0x7feffc0b4e60 msgr2=0x7feffc0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7ff00270f640 1 -- 172.21.15.77:0/1639574660 shutdown_connections 2024-03-06T07:27:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7ff00270f640 1 -- 172.21.15.77:0/1639574660 wait complete. 2024-03-06T07:27:17.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f339c4df640 1 -- 172.21.15.77:0/3740265321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3394070e10 msgr2=0x7f33940cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f339c4df640 1 --2- 172.21.15.77:0/3740265321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3394070e10 0x7f33940cd7c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f3394072d80 tx=0x7f338c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:27:17.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f339c4df640 1 -- 172.21.15.77:0/3740265321 shutdown_connections 2024-03-06T07:27:17.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f339c4df640 1 --2- 172.21.15.77:0/3740265321 >> v2:172.21.15.158:3300/0 conn(0x7f339415b380 0x7f33940708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f339c4df640 1 --2- 172.21.15.77:0/3740265321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3394070e10 0x7f33940cd7c0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f339c4df640 1 -- 172.21.15.77:0/3740265321 >> 172.21.15.77:0/3740265321 conn(0x7f33940b1c80 msgr2=0x7f33940cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f339c4df640 1 -- 172.21.15.77:0/3740265321 shutdown_connections 2024-03-06T07:27:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.797+0000 7f339c4df640 1 -- 172.21.15.77:0/3740265321 wait complete. 2024-03-06T07:27:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.798+0000 7ff0b45e9640 1 Processor -- start 2024-03-06T07:27:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.798+0000 7ff0b45e9640 1 -- start start 2024-03-06T07:27:17.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.798+0000 7ff00270f640 1 Processor -- start 2024-03-06T07:27:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.798+0000 7ff00270f640 1 -- start start 2024-03-06T07:27:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.798+0000 7f2f8666b640 1 Processor -- start 2024-03-06T07:27:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7f2f8666b640 1 -- start start 2024-03-06T07:27:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b45e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0ac0b2950 0x7ff0ac148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b45e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac0b3c90 0x7ff0ac168a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7f339c4df640 1 Processor -- start 2024-03-06T07:27:17.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7f339c4df640 1 -- start start 2024-03-06T07:27:17.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b45e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0ac15b5b0 0x7ff0ac15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b45e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac0cde20 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b45e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac0cdb40 con 0x7ff0ac0b2950 2024-03-06T07:27:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b45e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac0b4070 con 0x7ff0ac15b5b0 2024-03-06T07:27:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b1b84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac0b3c90 0x7ff0ac168a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b1b84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac0b3c90 0x7ff0ac168a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36710/0 (socket says 172.21.15.77:36710) 2024-03-06T07:27:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b1b84640 1 -- 172.21.15.77:0/2557398337 learned_addr learned my addr 172.21.15.77:0/2557398337 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b2b86640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0ac15b5b0 0x7ff0ac15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b2385640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0ac0b2950 0x7ff0ac148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff00270f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc072520 0x7feffc155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff00270f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feffc0b5c30 0x7feffc159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff00270f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc15d4f0 0x7feffc0b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff00270f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc15e050 con 0x7feffc072520 2024-03-06T07:27:17.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b1b84640 1 -- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0ac15b5b0 msgr2=0x7ff0ac15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff00270f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc15fe30 con 0x7feffc0b5c30 2024-03-06T07:27:17.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff00270f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc15d040 con 0x7feffc15d4f0 2024-03-06T07:27:17.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7feffbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc072520 0x7feffc155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7feffbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc072520 0x7feffc155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36716/0 (socket says 172.21.15.77:36716) 2024-03-06T07:27:17.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7feffbfff640 1 -- 172.21.15.77:0/4172291873 learned_addr learned my addr 172.21.15.77:0/4172291873 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:17.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b1b84640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0ac15b5b0 0x7ff0ac15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b1b84640 1 -- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0ac0b2950 msgr2=0x7ff0ac148270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b1b84640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0ac0b2950 0x7ff0ac148270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b1b84640 1 -- 172.21.15.77:0/2557398337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac148ae0 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff0b2385640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0ac0b2950 0x7ff0ac148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7feff3fff640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feffc0b5c30 0x7feffc159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.799+0000 7ff000cac640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc15d4f0 0x7feffc0b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f8666b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f800b4460 0x7f2f80153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7feffbfff640 1 -- 172.21.15.77:0/4172291873 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc15d4f0 msgr2=0x7feffc0b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7feffbfff640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc15d4f0 0x7feffc0b0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7feffbfff640 1 -- 172.21.15.77:0/4172291873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feffc0b5c30 msgr2=0x7feffc159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7feffbfff640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feffc0b5c30 0x7feffc159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7feffbfff640 1 -- 172.21.15.77:0/4172291873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc0afd30 con 0x7feffc072520 2024-03-06T07:27:17.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7ff0b1b84640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac0b3c90 0x7ff0ac168a40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80090b0 tx=0x7ff0a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f8666b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f80161cf0 0x7f2f800ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f8666b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f80162970 0x7f2f800b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f8666b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f800ce950 con 0x7f2f80161cf0 2024-03-06T07:27:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f8666b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f800d2b60 con 0x7f2f80162970 2024-03-06T07:27:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f8666b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f800b4840 con 0x7f2f800b4460 2024-03-06T07:27:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f7ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f800b4460 0x7f2f80153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f7ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f800b4460 0x7f2f80153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:48760/0 (socket says 172.21.15.77:48760) 2024-03-06T07:27:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f7ffff640 1 -- 172.21.15.77:0/2852949634 learned_addr learned my addr 172.21.15.77:0/2852949634 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339c4df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3394070e10 0x7f3394151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7ff0a2ffd640 1 -- 172.21.15.77:0/2557398337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8034040 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339c4df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3394158880 0x7f3394159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7feffbfff640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc072520 0x7feffc155b60 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fefec02b930 tx=0x7fefec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339c4df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f339415b380 0x7f3394159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f84c08640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f80162970 0x7f2f800b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ac15bd00 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7ff0a2ffd640 1 -- 172.21.15.77:0/2557398337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a8030660 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f2f7f7fe640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f80161cf0 0x7f2f800ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339c4df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339406dbb0 con 0x7f3394158880 2024-03-06T07:27:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339a27b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3394070e10 0x7f3394151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339a27b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3394070e10 0x7f3394151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:48776/0 (socket says 172.21.15.77:48776) 2024-03-06T07:27:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339a27b640 1 -- 172.21.15.77:0/4180491587 learned_addr learned my addr 172.21.15.77:0/4180491587 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339406d8d0 con 0x7f339415b380 2024-03-06T07:27:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33940cdd00 con 0x7f3394070e10 2024-03-06T07:27:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7f339aa7c640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f339415b380 0x7f3394159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f7f7fe640 1 -- 172.21.15.77:0/2852949634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f800b4460 msgr2=0x7f2f80153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f7f7fe640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f800b4460 0x7f2f80153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f7f7fe640 1 -- 172.21.15.77:0/2852949634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f80162970 msgr2=0x7f2f800b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f7f7fe640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f80162970 0x7f2f800b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f7f7fe640 1 -- 172.21.15.77:0/2852949634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f800b6e50 con 0x7f2f80161cf0 2024-03-06T07:27:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f84c08640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f80162970 0x7f2f800b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f339aa7c640 1 -- 172.21.15.77:0/4180491587 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3394070e10 msgr2=0x7f3394151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f339aa7c640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3394070e10 0x7f3394151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f339aa7c640 1 -- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3394158880 msgr2=0x7f3394159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f339aa7c640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3394158880 0x7f3394159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f339aa7c640 1 -- 172.21.15.77:0/4180491587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33941526d0 con 0x7f339415b380 2024-03-06T07:27:17.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7feff97fa640 1 -- 172.21.15.77:0/4172291873 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec040070 con 0x7feffc072520 2024-03-06T07:27:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f7f7fe640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f80161cf0 0x7f2f800ccbf0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f2f74033000 tx=0x7f2f740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7feff97fa640 1 -- 172.21.15.77:0/4172291873 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec03e070 con 0x7feffc072520 2024-03-06T07:27:17.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f339aa7c640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f339415b380 0x7f3394159b40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f338c032000 tx=0x7f338c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7feff97fa640 1 -- 172.21.15.77:0/4172291873 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec035040 con 0x7feffc072520 2024-03-06T07:27:17.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ac15c2c0 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.800+0000 7ff0a2ffd640 1 -- 172.21.15.77:0/2557398337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a802f5c0 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f7cff9640 1 -- 172.21.15.77:0/2852949634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74041070 con 0x7f2f80161cf0 2024-03-06T07:27:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f80153640 con 0x7f2f80161cf0 2024-03-06T07:27:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f7cff9640 1 -- 172.21.15.77:0/2852949634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7403b900 con 0x7f2f80161cf0 2024-03-06T07:27:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f800cb090 con 0x7f2f80161cf0 2024-03-06T07:27:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.802+0000 7f2f7cff9640 1 -- 172.21.15.77:0/2852949634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74041070 con 0x7f2f80161cf0 2024-03-06T07:27:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc0b0010 con 0x7feffc072520 2024-03-06T07:27:17.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc152cf0 con 0x7feffc072520 2024-03-06T07:27:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.802+0000 7ff0a2ffd640 1 -- 172.21.15.77:0/2557398337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0a8037070 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f338affd640 1 -- 172.21.15.77:0/4180491587 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c009510 con 0x7f339415b380 2024-03-06T07:27:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f338affd640 1 -- 172.21.15.77:0/4180491587 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c003990 con 0x7f339415b380 2024-03-06T07:27:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f338affd640 1 -- 172.21.15.77:0/4180491587 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c003dc0 con 0x7f339415b380 2024-03-06T07:27:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.801+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33941523f0 con 0x7f339415b380 2024-03-06T07:27:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.802+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3394150820 con 0x7f339415b380 2024-03-06T07:27:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.803+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8004f330 con 0x7f2f80161cf0 2024-03-06T07:27:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.802+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ac04f330 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.803+0000 7feff97fa640 1 -- 172.21.15.77:0/4172291873 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefec038020 con 0x7feffc072520 2024-03-06T07:27:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.803+0000 7feff97fa640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefdc06b520 0x7fefdc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.803+0000 7feff97fa640 1 -- 172.21.15.77:0/4172291873 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefec0aba70 con 0x7feffc072520 2024-03-06T07:27:17.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.803+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc04f330 con 0x7feffc072520 2024-03-06T07:27:17.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.803+0000 7f2f7cff9640 1 -- 172.21.15.77:0/2852949634 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f7404c060 con 0x7f2f80161cf0 2024-03-06T07:27:17.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7feff3fff640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefdc06b520 0x7fefdc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7ff0a2ffd640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff09006b5a0 0x7ff09006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339404f330 con 0x7f339415b380 2024-03-06T07:27:17.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7f338affd640 1 -- 172.21.15.77:0/4180491587 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f338c02cb30 con 0x7f339415b380 2024-03-06T07:27:17.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7ff0a2ffd640 1 -- 172.21.15.77:0/2557398337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0a80ac6a0 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7f2f7cff9640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f6c06b6d0 0x7f2f6c06db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7ff0b2385640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff09006b5a0 0x7ff09006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7feff3fff640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefdc06b520 0x7fefdc06d9e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7feffc068380 tx=0x7fefe4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7f338affd640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337806b550 0x7f337806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7f2f7cff9640 1 -- 172.21.15.77:0/2852949634 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f740ab710 con 0x7f2f80161cf0 2024-03-06T07:27:17.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7f338affd640 1 -- 172.21.15.77:0/4180491587 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f338c0ab820 con 0x7f339415b380 2024-03-06T07:27:17.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.804+0000 7f2f7ffff640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f6c06b6d0 0x7f2f6c06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.805+0000 7f339a27b640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337806b550 0x7f337806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.805+0000 7ff0b2385640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff09006b5a0 0x7ff09006da60 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff09c003eb0 tx=0x7ff09c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.805+0000 7f2f7ffff640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f6c06b6d0 0x7f2f6c06db90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2f70005880 tx=0x7f2f70009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.805+0000 7f339a27b640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337806b550 0x7f337806da10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3390000f60 tx=0x7f3390009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.808+0000 7ff0a2ffd640 1 -- 172.21.15.77:0/2557398337 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0a807b590 con 0x7ff0ac0b3c90 2024-03-06T07:27:17.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.809+0000 7f338affd640 1 -- 172.21.15.77:0/4180491587 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f338c07a130 con 0x7f339415b380 2024-03-06T07:27:17.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.817+0000 7feff97fa640 1 -- 172.21.15.77:0/4172291873 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefec07a280 con 0x7feffc072520 2024-03-06T07:27:17.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:17.825+0000 7f2f7cff9640 1 -- 172.21.15.77:0/2852949634 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f800b6e50 con 0x7f2f80161cf0 2024-03-06T07:27:18.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.058+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f33940d0300 con 0x7f339415b380 2024-03-06T07:27:18.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.068+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7feffc0cd090 con 0x7feffc072520 2024-03-06T07:27:18.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.073+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7ff0ac071620 con 0x7ff0ac0b3c90 2024-03-06T07:27:18.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.080+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f2f8006b180 con 0x7f2f80161cf0 2024-03-06T07:27:18.551 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:27:18.550+0000 7f63c7eac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:18.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.550+0000 7f338affd640 1 -- 172.21.15.77:0/4180491587 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6383 v67) v1 ==== 95+0+0 (secure 0 0 0) 0x7f338c07e1d0 con 0x7f339415b380 2024-03-06T07:27:18.552 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 6383 2024-03-06T07:27:18.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.551+0000 7feff97fa640 1 -- 172.21.15.77:0/4172291873 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7827 v68) v1 ==== 95+0+0 (secure 0 0 0) 0x7fefec07e320 con 0x7feffc072520 2024-03-06T07:27:18.553 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 7827 2024-03-06T07:27:18.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337806b550 msgr2=0x7f337806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337806b550 0x7f337806da10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3390000f60 tx=0x7f3390009040 comp rx=0 tx=0).stop 2024-03-06T07:27:18.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f339415b380 msgr2=0x7f3394159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f339415b380 0x7f3394159b40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f338c032000 tx=0x7f338c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 shutdown_connections 2024-03-06T07:27:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f337806b550 0x7f337806da10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3394158880 0x7f3394159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f339415b380 0x7f3394159b40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 --2- 172.21.15.77:0/4180491587 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3394070e10 0x7f3394151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 >> 172.21.15.77:0/4180491587 conn(0x7f33940b1c80 msgr2=0x7f33940ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 shutdown_connections 2024-03-06T07:27:18.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.553+0000 7f339c4df640 1 -- 172.21.15.77:0/4180491587 wait complete. 2024-03-06T07:27:18.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefdc06b520 msgr2=0x7fefdc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefdc06b520 0x7fefdc06d9e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7feffc068380 tx=0x7fefe4009190 comp rx=0 tx=0).stop 2024-03-06T07:27:18.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc072520 msgr2=0x7feffc155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc072520 0x7feffc155b60 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fefec02b930 tx=0x7fefec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 shutdown_connections 2024-03-06T07:27:18.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feffc15d4f0 0x7feffc0b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feffc0b5c30 0x7feffc159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefdc06b520 0x7fefdc06d9e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 --2- 172.21.15.77:0/4172291873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc072520 0x7feffc155b60 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 >> 172.21.15.77:0/4172291873 conn(0x7feffc0b4e60 msgr2=0x7feffc15b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 shutdown_connections 2024-03-06T07:27:18.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.554+0000 7ff00270f640 1 -- 172.21.15.77:0/4172291873 wait complete. 2024-03-06T07:27:18.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.556+0000 7ff0a2ffd640 1 -- 172.21.15.77:0/2557398337 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6380 v69) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff0a807ae90 con 0x7ff0ac0b3c90 2024-03-06T07:27:18.558 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 6380 2024-03-06T07:27:18.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff09006b5a0 msgr2=0x7ff09006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff09006b5a0 0x7ff09006da60 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff09c003eb0 tx=0x7ff09c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac0b3c90 msgr2=0x7ff0ac168a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac0b3c90 0x7ff0ac168a40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80090b0 tx=0x7ff0a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 shutdown_connections 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff09006b5a0 0x7ff09006da60 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0ac0b3c90 0x7ff0ac168a40 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0ac15b5b0 0x7ff0ac15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 --2- 172.21.15.77:0/2557398337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0ac0b2950 0x7ff0ac148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.558+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 >> 172.21.15.77:0/2557398337 conn(0x7ff0ac0b1c80 msgr2=0x7ff0ac072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.559+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 shutdown_connections 2024-03-06T07:27:18.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:18.559+0000 7ff0b45e9640 1 -- 172.21.15.77:0/2557398337 wait complete. 2024-03-06T07:27:18.573 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:18.573 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:18.574 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:18.574 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:18.575 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:18.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:18.578 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:18.578 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:18.579 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:18.579 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:18.617 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:18.617 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:18.660 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:27:18.661 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:27:18.691 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:27:19.550 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:27:19.549+0000 7f63c7eac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:19.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.550+0000 7f2f7cff9640 1 -- 172.21.15.77:0/2852949634 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7824 v70) v1 ==== 95+0+0 (secure 0 0 0) 0x7f2f800b6e50 con 0x7f2f80161cf0 2024-03-06T07:27:19.552 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 7824 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f6c06b6d0 msgr2=0x7f2f6c06db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f6c06b6d0 0x7f2f6c06db90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2f70005880 tx=0x7f2f70009040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f80161cf0 msgr2=0x7f2f800ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f80161cf0 0x7f2f800ccbf0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f2f74033000 tx=0x7f2f740046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 shutdown_connections 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f6c06b6d0 0x7f2f6c06db90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f80161cf0 0x7f2f800ccbf0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f80162970 0x7f2f800b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 --2- 172.21.15.77:0/2852949634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f800b4460 0x7f2f80153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 >> 172.21.15.77:0/2852949634 conn(0x7f2f800d3ce0 msgr2=0x7f2f800b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 shutdown_connections 2024-03-06T07:27:19.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.553+0000 7f2f8666b640 1 -- 172.21.15.77:0/2852949634 wait complete. 2024-03-06T07:27:19.573 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:19.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:19.615 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:19.615 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:19.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:19.694 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:27:19.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.720+0000 7f36c4113640 1 Processor -- start 2024-03-06T07:27:19.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.720+0000 7f36c4113640 1 -- start start 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c4113640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c4113640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36bc0d16e0 0x7f36bc0d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c4113640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0cebe0 con 0x7f36bc15bdd0 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c4113640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36bc0ceec0 con 0x7f36bc0d16e0 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c4113640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0cf1a0 con 0x7f36bc15fa10 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c26b0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36bc0d16e0 0x7f36bc0d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c16ae640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c1eaf640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f36bc15bdd0 0x7f36bc0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:46608/0 (socket says 172.21.15.77:46608) 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c26b0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36bc0d16e0 0x7f36bc0d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:49620/0 (socket says 172.21.15.77:49620) 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c16ae640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:43314/0 (socket says 172.21.15.77:43314) 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.721+0000 7f36c1eaf640 1 -- 172.21.15.77:0/1038129372 learned_addr learned my addr 172.21.15.77:0/1038129372 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:19.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.722+0000 7f36c16ae640 1 -- 172.21.15.77:0/1038129372 >> v2:172.21.15.158:3300/0 conn(0x7f36bc0d16e0 msgr2=0x7f36bc0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.722+0000 7f36c16ae640 1 --2- 172.21.15.77:0/1038129372 >> v2:172.21.15.158:3300/0 conn(0x7f36bc0d16e0 0x7f36bc0d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.722+0000 7f36c16ae640 1 -- 172.21.15.77:0/1038129372 >> v1:172.21.15.158:6789/0 conn(0x7f36bc15bdd0 legacy=0x7f36bc0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.722+0000 7f36c16ae640 1 -- 172.21.15.77:0/1038129372 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc0cc8e0 con 0x7f36bc15fa10 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.722+0000 7f36c0ead640 1 -- 172.21.15.77:0/1038129372 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612480412 0 0) 0x7f36bc0cebe0 con 0x7f36bc15bdd0 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.722+0000 7f36c26b0640 1 --2- 172.21.15.77:0/1038129372 >> v2:172.21.15.158:3300/0 conn(0x7f36bc0d16e0 0x7f36bc0d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.722+0000 7f36c16ae640 1 --2- 172.21.15.77:0/1038129372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ce270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f36b80097a0 tx=0x7f36b802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50c74080cf9c01bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c0ead640 1 -- 172.21.15.77:0/1038129372 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80315a0 con 0x7f36bc15fa10 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c0ead640 1 -- 172.21.15.77:0/1038129372 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b8037070 con 0x7f36bc15fa10 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c0ead640 1 -- 172.21.15.77:0/1038129372 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8031960 con 0x7f36bc15fa10 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c4113640 1 -- 172.21.15.77:0/1038129372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 msgr2=0x7f36bc0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c4113640 1 --2- 172.21.15.77:0/1038129372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ce270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f36b80097a0 tx=0x7f36b802b840 comp rx=0 tx=0).stop 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c4113640 1 -- 172.21.15.77:0/1038129372 shutdown_connections 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c4113640 1 --2- 172.21.15.77:0/1038129372 >> v2:172.21.15.158:3300/0 conn(0x7f36bc0d16e0 0x7f36bc0d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c4113640 1 --2- 172.21.15.77:0/1038129372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ce270 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c4113640 1 -- 172.21.15.77:0/1038129372 >> 172.21.15.77:0/1038129372 conn(0x7f36bc09a9f0 msgr2=0x7f36bc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c4113640 1 -- 172.21.15.77:0/1038129372 shutdown_connections 2024-03-06T07:27:19.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.723+0000 7f36c4113640 1 -- 172.21.15.77:0/1038129372 wait complete. 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.724+0000 7f36c4113640 1 Processor -- start 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.724+0000 7f36c4113640 1 -- start start 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c4113640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36bc0d16e0 0x7f36bc0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c4113640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc15bdd0 0x7f36bc0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c4113640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c4113640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc15c1b0 con 0x7f36bc15bdd0 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c4113640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0cc5d0 con 0x7f36bc0d16e0 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c4113640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0ce7b0 con 0x7f36bc15fa10 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c16ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc15bdd0 0x7f36bc0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c16ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc15bdd0 0x7f36bc0ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56132/0 (socket says 172.21.15.77:56132) 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c16ae640 1 -- 172.21.15.77:0/2360034848 learned_addr learned my addr 172.21.15.77:0/2360034848 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c1eaf640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36bc0d16e0 0x7f36bc0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c26b0640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c16ae640 1 -- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 msgr2=0x7f36bc0ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c16ae640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c16ae640 1 -- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36bc0d16e0 msgr2=0x7f36bc0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.725+0000 7f36c16ae640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36bc0d16e0 0x7f36bc0b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36c16ae640 1 -- 172.21.15.77:0/2360034848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc1f16a0 con 0x7f36bc15bdd0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36c26b0640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36c1eaf640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36bc0d16e0 0x7f36bc0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36c16ae640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc15bdd0 0x7f36bc0ca180 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f36b80097a0 tx=0x7f36b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36b27fc640 1 -- 172.21.15.77:0/2360034848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8004230 con 0x7f36bc15bdd0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36b27fc640 1 -- 172.21.15.77:0/2360034848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b80043f0 con 0x7f36bc15bdd0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc1f1920 con 0x7f36bc15bdd0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36b27fc640 1 -- 172.21.15.77:0/2360034848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8003850 con 0x7f36bc15bdd0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.726+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc1f1ee0 con 0x7f36bc15bdd0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.727+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36bc04f330 con 0x7f36bc15bdd0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.727+0000 7f36b27fc640 1 -- 172.21.15.77:0/2360034848 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36b80039f0 con 0x7f36bc15bdd0 2024-03-06T07:27:19.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.728+0000 7f36b27fc640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.728+0000 7f36b27fc640 1 -- 172.21.15.77:0/2360034848 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f36b802f030 con 0x7f36bc15bdd0 2024-03-06T07:27:19.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.728+0000 7f36c1eaf640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.728+0000 7f36c1eaf640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f36ac000f60 tx=0x7f36ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.732+0000 7f36b27fc640 1 -- 172.21.15.77:0/2360034848 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36bc1f16a0 con 0x7f36bc15bdd0 2024-03-06T07:27:19.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.941+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36bc0b7a40 con 0x7f36bc15bdd0 2024-03-06T07:27:19.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.941+0000 7f36b27fc640 1 -- 172.21.15.77:0/2360034848 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4186 (secure 0 0 0) 0x7f36b8041070 con 0x7f36bc15bdd0 2024-03-06T07:27:19.943 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:19.943 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":71,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:19.556261+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:19.943 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 71 2024-03-06T07:27:19.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 msgr2=0x7f36a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f36ac000f60 tx=0x7f36ac009040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc15bdd0 msgr2=0x7f36bc0ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc15bdd0 0x7f36bc0ca180 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f36b80097a0 tx=0x7f36b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 shutdown_connections 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f36a006b4d0 0x7f36a006d990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc15bdd0 0x7f36bc0ca180 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36bc15fa10 0x7f36bc0ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 --2- 172.21.15.77:0/2360034848 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36bc0d16e0 0x7f36bc0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 >> 172.21.15.77:0/2360034848 conn(0x7f36bc09a9f0 msgr2=0x7f36bc09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 shutdown_connections 2024-03-06T07:27:19.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:19.943+0000 7f36c4113640 1 -- 172.21.15.77:0/2360034848 wait complete. 2024-03-06T07:27:20.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.849148+0000', 'modified': '2024-03-06T07:27:19.556261+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8097}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8097': {'gid': 8097, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.77:6835/493819683', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 493819683}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 493819683}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:20.359 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8097:up:replay 2024-03-06T07:27:21.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.470+0000 7f50b3cb5640 1 Processor -- start 2024-03-06T07:27:21.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.470+0000 7f50b3cb5640 1 -- start start 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.471+0000 7f50b3cb5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f50ac15e3a0 0x7f50ac06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.471+0000 7f50b3cb5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ac15eb40 con 0x7f50ac072730 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.471+0000 7f50b3cb5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ac15ee20 con 0x7f50ac158600 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.471+0000 7f50b3cb5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ac161ba0 con 0x7f50ac15e3a0 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.471+0000 7f50b1250640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f50ac072730 0x7f50ac162030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59844/0 (socket says 172.21.15.77:59844) 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.471+0000 7f50b1250640 1 -- 172.21.15.77:0/1656540455 learned_addr learned my addr 172.21.15.77:0/1656540455 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.471+0000 7f50b2252640 1 --2- 172.21.15.77:0/1656540455 >> v2:172.21.15.158:3300/0 conn(0x7f50ac15e3a0 0x7f50ac06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.471+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916379676 0 0) 0x7f50ac15eb40 con 0x7f50ac072730 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5098003520 con 0x7f50ac072730 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1177027129 0 0) 0x7f5098003520 con 0x7f50ac072730 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 >> v2:172.21.15.158:3300/0 conn(0x7f50ac15e3a0 msgr2=0x7f50ac06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 --2- 172.21.15.77:0/1656540455 >> v2:172.21.15.158:3300/0 conn(0x7f50ac15e3a0 0x7f50ac06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 >> v1:172.21.15.158:6789/0 conn(0x7f50ac158600 legacy=0x7f50ac06d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac15f100 con 0x7f50ac072730 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f509c001670 con 0x7f50ac072730 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 >> v1:172.21.15.77:6789/0 conn(0x7f50ac072730 legacy=0x7f50ac162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980039b0 0x7f5098003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50980043b0 0x7f5098004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5098005120 0x7f509800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac161ba0 con 0x7f50980043b0 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509800c060 con 0x7f5098005120 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509800c260 con 0x7f50980039b0 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b1250640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50980043b0 0x7f5098004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b2252640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5098005120 0x7f509800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b1a51640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980039b0 0x7f5098003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b1250640 1 -- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980039b0 msgr2=0x7f5098003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b1250640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980039b0 0x7f5098003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b1250640 1 -- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5098005120 msgr2=0x7f509800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b1250640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5098005120 0x7f509800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b1250640 1 -- 172.21.15.77:0/1656540455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5098003520 con 0x7f50980043b0 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b2252640 1 -- 172.21.15.77:0/1656540455 reap_dead start 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.472+0000 7f50b1a51640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980039b0 0x7f5098003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.473+0000 7f50b1250640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50980043b0 0x7f5098004880 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f509c00c240 tx=0x7f5098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5438c363649d680 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.473+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c007d10 con 0x7f50980043b0 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.473+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c030070 con 0x7f50980043b0 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.473+0000 7f50b0a4f640 1 -- 172.21.15.77:0/1656540455 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c005d70 con 0x7f50980043b0 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.474+0000 7f50b3cb5640 1 -- 172.21.15.77:0/1656540455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50980043b0 msgr2=0x7f5098004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.474+0000 7f50b3cb5640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50980043b0 0x7f5098004880 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f509c00c240 tx=0x7f5098005830 comp rx=0 tx=0).stop 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.474+0000 7f50b3cb5640 1 -- 172.21.15.77:0/1656540455 shutdown_connections 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.474+0000 7f50b3cb5640 1 --2- 172.21.15.77:0/1656540455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50980043b0 0x7f5098004880 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.474+0000 7f50b3cb5640 1 -- 172.21.15.77:0/1656540455 >> 172.21.15.77:0/1656540455 conn(0x7f50ac06c0c0 msgr2=0x7f50ac06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.474+0000 7f50b3cb5640 1 -- 172.21.15.77:0/1656540455 shutdown_connections 2024-03-06T07:27:21.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.474+0000 7f50b3cb5640 1 -- 172.21.15.77:0/1656540455 wait complete. 2024-03-06T07:27:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.475+0000 7f50b3cb5640 1 Processor -- start 2024-03-06T07:27:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.475+0000 7f50b3cb5640 1 -- start start 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b3cb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980043b0 0x7f50ac1f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b3cb5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac1e7e00 0x7f50ac1f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b3cb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0b3960 0x7f50ac14fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b3cb5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac06d970 con 0x7f50ac0b3960 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b3cb5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac072be0 con 0x7f50ac1e7e00 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b3cb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac06daf0 con 0x7f50980043b0 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0b3960 0x7f50ac14fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0b3960 0x7f50ac14fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56156/0 (socket says 172.21.15.77:56156) 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 -- 172.21.15.77:0/3116761844 learned_addr learned my addr 172.21.15.77:0/3116761844 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b1250640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac1e7e00 0x7f50ac1f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b1a51640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980043b0 0x7f50ac1f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 -- 172.21.15.77:0/3116761844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980043b0 msgr2=0x7f50ac1f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980043b0 0x7f50ac1f7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 -- 172.21.15.77:0/3116761844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac1e7e00 msgr2=0x7f50ac1f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac1e7e00 0x7f50ac1f46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 -- 172.21.15.77:0/3116761844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac1f7290 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b2252640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0b3960 0x7f50ac14fbb0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0732b0 tx=0x7f50a4011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50a27fc640 1 -- 172.21.15.77:0/3116761844 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4019020 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50a27fc640 1 -- 172.21.15.77:0/3116761844 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a4004030 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ac1e8290 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.476+0000 7f50a27fc640 1 -- 172.21.15.77:0/3116761844 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4004400 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.477+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ac1e8570 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.477+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac04f330 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.478+0000 7f50a27fc640 1 -- 172.21.15.77:0/3116761844 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50a401c020 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.478+0000 7f50a27fc640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.478+0000 7f50a27fc640 1 -- 172.21.15.77:0/3116761844 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f50a408a650 con 0x7f50ac0b3960 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.478+0000 7f50b1a51640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.479+0000 7f50b1a51640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509006b4d0 0x7f509006d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0686a0 tx=0x7f50a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.483+0000 7f50a27fc640 1 -- 172.21.15.77:0/3116761844 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50a4023020 con 0x7f50ac0b3960 2024-03-06T07:27:21.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.698+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50ac0d06f0 con 0x7f50ac0b3960 2024-03-06T07:27:21.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.698+0000 7f50a27fc640 1 -- 172.21.15.77:0/3116761844 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f50a4059420 con 0x7f50ac0b3960 2024-03-06T07:27:21.700 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:21.701 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:20.570155+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:21.701 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 73 2024-03-06T07:27:21.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509006b4d0 msgr2=0x7f509006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509006b4d0 0x7f509006d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0686a0 tx=0x7f50a8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0b3960 msgr2=0x7f50ac14fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0b3960 0x7f50ac14fbb0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0732b0 tx=0x7f50a4011cf0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 shutdown_connections 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0b3960 0x7f50ac14fbb0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac1e7e00 0x7f50ac1f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 --2- 172.21.15.77:0/3116761844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50980043b0 0x7f50ac1f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 >> 172.21.15.77:0/3116761844 conn(0x7f50ac06c0c0 msgr2=0x7f50ac06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 shutdown_connections 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:21.700+0000 7f50b3cb5640 1 -- 172.21.15.77:0/3116761844 wait complete. 2024-03-06T07:27:22.115 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.849148+0000', 'modified': '2024-03-06T07:27:20.570155+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8097}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8097': {'gid': 8097, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.77:6835/493819683', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 493819683}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 493819683}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:22.116 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8097:up:rejoin 2024-03-06T07:27:23.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:23.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.216+0000 7f348238a640 1 Processor -- start 2024-03-06T07:27:23.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.216+0000 7f348238a640 1 -- start start 2024-03-06T07:27:23.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f348238a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f347c15dfe0 0x7f347c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f348238a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0b3430 con 0x7f347c0d29c0 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f348238a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0b3710 con 0x7f347c0cf830 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f348238a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c0b31e0 con 0x7f347c15dfe0 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f347bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f347c15dfe0 0x7f347c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f3480927640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f347c0d29c0 0x7f347c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59852/0 (socket says 172.21.15.77:59852) 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f3480927640 1 -- 172.21.15.77:0/514282362 learned_addr learned my addr 172.21.15.77:0/514282362 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f347bfff640 1 -- 172.21.15.77:0/514282362 >> v1:172.21.15.158:6789/0 conn(0x7f347c0cf830 legacy=0x7f347c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f347bfff640 1 -- 172.21.15.77:0/514282362 >> v1:172.21.15.77:6789/0 conn(0x7f347c0d29c0 legacy=0x7f347c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f347bfff640 1 -- 172.21.15.77:0/514282362 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c0b39f0 con 0x7f347c15dfe0 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.217+0000 7f347affd640 1 -- 172.21.15.77:0/514282362 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707095355 0 0) 0x7f347c0b3430 con 0x7f347c0d29c0 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.218+0000 7f347bfff640 1 --2- 172.21.15.77:0/514282362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c15dfe0 0x7f347c0cf2f0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f347c0b3360 tx=0x7f346c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d562e8af4e42df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.218+0000 7f347affd640 1 -- 172.21.15.77:0/514282362 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c035020 con 0x7f347c15dfe0 2024-03-06T07:27:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.218+0000 7f347affd640 1 -- 172.21.15.77:0/514282362 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c031830 con 0x7f347c15dfe0 2024-03-06T07:27:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.218+0000 7f347affd640 1 -- 172.21.15.77:0/514282362 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c031c00 con 0x7f347c15dfe0 2024-03-06T07:27:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.219+0000 7f348238a640 1 -- 172.21.15.77:0/514282362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c15dfe0 msgr2=0x7f347c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.219+0000 7f348238a640 1 --2- 172.21.15.77:0/514282362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c15dfe0 0x7f347c0cf2f0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f347c0b3360 tx=0x7f346c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.219+0000 7f348238a640 1 -- 172.21.15.77:0/514282362 shutdown_connections 2024-03-06T07:27:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.219+0000 7f348238a640 1 --2- 172.21.15.77:0/514282362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c15dfe0 0x7f347c0cf2f0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.219+0000 7f348238a640 1 -- 172.21.15.77:0/514282362 >> 172.21.15.77:0/514282362 conn(0x7f347c0d3cd0 msgr2=0x7f347c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.219+0000 7f348238a640 1 -- 172.21.15.77:0/514282362 shutdown_connections 2024-03-06T07:27:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.219+0000 7f348238a640 1 -- 172.21.15.77:0/514282362 wait complete. 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.220+0000 7f348238a640 1 Processor -- start 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.220+0000 7f348238a640 1 -- start start 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f348238a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347c0cf830 0x7f347c154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f348238a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c0d29c0 0x7f347c1f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f348238a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f347c15dfe0 0x7f347c1f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f348238a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c15e3c0 con 0x7f347c0d29c0 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f348238a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0b3d60 con 0x7f347c0cf830 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f348238a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c15e540 con 0x7f347c15dfe0 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c0d29c0 0x7f347c1f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c0d29c0 0x7f347c1f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56180/0 (socket says 172.21.15.77:56180) 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 -- 172.21.15.77:0/32992927 learned_addr learned my addr 172.21.15.77:0/32992927 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:23.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347bfff640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347c0cf830 0x7f347c154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f3480927640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f347c15dfe0 0x7f347c1f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 -- 172.21.15.77:0/32992927 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f347c15dfe0 msgr2=0x7f347c1f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f347c15dfe0 0x7f347c1f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 -- 172.21.15.77:0/32992927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347c0cf830 msgr2=0x7f347c154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347c0cf830 0x7f347c154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 -- 172.21.15.77:0/32992927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c1f5950 con 0x7f347c0d29c0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f347b7fe640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c0d29c0 0x7f347c1f4f60 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f347000c430 tx=0x7f347000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.221+0000 7f3478ff9640 1 -- 172.21.15.77:0/32992927 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470017070 con 0x7f347c0d29c0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.222+0000 7f3478ff9640 1 -- 172.21.15.77:0/32992927 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34700035f0 con 0x7f347c0d29c0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.222+0000 7f3478ff9640 1 -- 172.21.15.77:0/32992927 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700039e0 con 0x7f347c0d29c0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.222+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c1f5c30 con 0x7f347c0d29c0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.222+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c1f61f0 con 0x7f347c0d29c0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.223+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c04f330 con 0x7f347c0d29c0 2024-03-06T07:27:23.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.223+0000 7f3478ff9640 1 -- 172.21.15.77:0/32992927 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f347001a070 con 0x7f347c0d29c0 2024-03-06T07:27:23.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.224+0000 7f3478ff9640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f346806b500 0x7f346806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.224+0000 7f3478ff9640 1 -- 172.21.15.77:0/32992927 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f3470089590 con 0x7f347c0d29c0 2024-03-06T07:27:23.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.224+0000 7f347bfff640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f346806b500 0x7f346806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.224+0000 7f347bfff640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f346806b500 0x7f346806d9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f346c036000 tx=0x7f346c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.228+0000 7f3478ff9640 1 -- 172.21.15.77:0/32992927 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34700010c0 con 0x7f347c0d29c0 2024-03-06T07:27:23.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.439+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f347c0cf5b0 con 0x7f347c0d29c0 2024-03-06T07:27:23.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.439+0000 7f3478ff9640 1 -- 172.21.15.77:0/32992927 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f3470057d10 con 0x7f347c0d29c0 2024-03-06T07:27:23.441 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:23.441 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:23.441 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:23.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f346806b500 msgr2=0x7f346806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f346806b500 0x7f346806d9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f346c036000 tx=0x7f346c03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:23.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c0d29c0 msgr2=0x7f347c1f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c0d29c0 0x7f347c1f4f60 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f347000c430 tx=0x7f347000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 shutdown_connections 2024-03-06T07:27:23.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f346806b500 0x7f346806d9c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c0d29c0 0x7f347c1f4f60 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f347c15dfe0 0x7f347c1f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 --2- 172.21.15.77:0/32992927 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f347c0cf830 0x7f347c154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 >> 172.21.15.77:0/32992927 conn(0x7f347c0d3cd0 msgr2=0x7f347c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 shutdown_connections 2024-03-06T07:27:23.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.441+0000 7f348238a640 1 -- 172.21.15.77:0/32992927 wait complete. 2024-03-06T07:27:23.857 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:04.849148+0000', 'modified': '2024-03-06T07:27:22.573254+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8097}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8097': {'gid': 8097, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.77:6835/493819683', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 493819683}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 493819683}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8097, 'qdb_cluster': [8097]} 2024-03-06T07:27:23.857 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:23.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:23.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.959+0000 7fa75d563640 1 Processor -- start 2024-03-06T07:27:23.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.959+0000 7fa75d563640 1 -- start start 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.959+0000 7fa75d563640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa758162970 0x7fa7580d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.959+0000 7fa75d563640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa758161cf0 0x7fa758162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.959+0000 7fa75d563640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7580d22c0 con 0x7fa7580b4460 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.959+0000 7fa75d563640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa758161b00 con 0x7fa758161cf0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.959+0000 7fa75d563640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7580cecc0 con 0x7fa758162970 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.959+0000 7fa7577fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa758161cf0 0x7fa758162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.960+0000 7fa7577fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa758161cf0 0x7fa758162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56184/0 (socket says 172.21.15.77:56184) 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.960+0000 7fa7577fe640 1 -- 172.21.15.77:0/1412662877 learned_addr learned my addr 172.21.15.77:0/1412662877 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.960+0000 7fa7577fe640 1 -- 172.21.15.77:0/1412662877 >> v2:172.21.15.158:3300/0 conn(0x7fa758162970 msgr2=0x7fa7580d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.960+0000 7fa7567fc640 1 --2- 172.21.15.77:0/1412662877 >> v2:172.21.15.158:3300/0 conn(0x7fa758162970 0x7fa7580d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.960+0000 7fa7577fe640 1 --2- 172.21.15.77:0/1412662877 >> v2:172.21.15.158:3300/0 conn(0x7fa758162970 0x7fa7580d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.960+0000 7fa7577fe640 1 -- 172.21.15.77:0/1412662877 >> v1:172.21.15.158:6789/0 conn(0x7fa7580b4460 legacy=0x7fa7580b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.960+0000 7fa7577fe640 1 -- 172.21.15.77:0/1412662877 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580d25a0 con 0x7fa758161cf0 2024-03-06T07:27:23.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.960+0000 7fa7577fe640 1 --2- 172.21.15.77:0/1412662877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758161cf0 0x7fa758162180 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa7580d17e0 tx=0x7fa74802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c229ea90932916ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa755ffb640 1 -- 172.21.15.77:0/1412662877 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa748035020 con 0x7fa758161cf0 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa755ffb640 1 -- 172.21.15.77:0/1412662877 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa748031870 con 0x7fa758161cf0 2024-03-06T07:27:23.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa755ffb640 1 -- 172.21.15.77:0/1412662877 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa748031c40 con 0x7fa758161cf0 2024-03-06T07:27:23.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa75d563640 1 -- 172.21.15.77:0/1412662877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758161cf0 msgr2=0x7fa758162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa75d563640 1 --2- 172.21.15.77:0/1412662877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758161cf0 0x7fa758162180 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa7580d17e0 tx=0x7fa74802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa75d563640 1 -- 172.21.15.77:0/1412662877 shutdown_connections 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa75d563640 1 --2- 172.21.15.77:0/1412662877 >> v2:172.21.15.158:3300/0 conn(0x7fa758162970 0x7fa7580d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa75d563640 1 --2- 172.21.15.77:0/1412662877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758161cf0 0x7fa758162180 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa75d563640 1 -- 172.21.15.77:0/1412662877 >> 172.21.15.77:0/1412662877 conn(0x7fa7580d3ce0 msgr2=0x7fa7580d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa75d563640 1 -- 172.21.15.77:0/1412662877 shutdown_connections 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.961+0000 7fa75d563640 1 -- 172.21.15.77:0/1412662877 wait complete. 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa75d563640 1 Processor -- start 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa75d563640 1 -- start start 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa75d563640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7580b4460 0x7fa758153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa75d563640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa758161cf0 0x7fa7580b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa75d563640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758162970 0x7fa7580cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa75d563640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580ce950 con 0x7fa758162970 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa75d563640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580d2b60 con 0x7fa758161cf0 2024-03-06T07:27:23.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa75d563640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580b4840 con 0x7fa7580b4460 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758162970 0x7fa7580cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758162970 0x7fa7580cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56196/0 (socket says 172.21.15.77:56196) 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.963+0000 7fa7577fe640 1 -- 172.21.15.77:0/3758253343 learned_addr learned my addr 172.21.15.77:0/3758253343 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa7567fc640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa758161cf0 0x7fa7580b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7580b4460 0x7fa758153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa7577fe640 1 -- 172.21.15.77:0/3758253343 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7580b4460 msgr2=0x7fa758153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa7577fe640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7580b4460 0x7fa758153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa7577fe640 1 -- 172.21.15.77:0/3758253343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa758161cf0 msgr2=0x7fa7580b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa7577fe640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa758161cf0 0x7fa7580b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa7577fe640 1 -- 172.21.15.77:0/3758253343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580b8290 con 0x7fa758162970 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa7577fe640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758162970 0x7fa7580cadb0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fa74802b950 tx=0x7fa7480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa74f7fe640 1 -- 172.21.15.77:0/3758253343 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7480443f0 con 0x7fa758162970 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa74f7fe640 1 -- 172.21.15.77:0/3758253343 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7480445b0 con 0x7fa758162970 2024-03-06T07:27:23.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa75d563640 1 -- 172.21.15.77:0/3758253343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7580b8530 con 0x7fa758162970 2024-03-06T07:27:23.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa74f7fe640 1 -- 172.21.15.77:0/3758253343 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7480449a0 con 0x7fa758162970 2024-03-06T07:27:23.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.964+0000 7fa75d563640 1 -- 172.21.15.77:0/3758253343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7580b87d0 con 0x7fa758162970 2024-03-06T07:27:23.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.965+0000 7fa74f7fe640 1 -- 172.21.15.77:0/3758253343 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa748038020 con 0x7fa758162970 2024-03-06T07:27:23.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.966+0000 7fa74f7fe640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa73c06b4d0 0x7fa73c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.966+0000 7fa74f7fe640 1 -- 172.21.15.77:0/3758253343 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fa7480ac570 con 0x7fa758162970 2024-03-06T07:27:23.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.966+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa73c06b4d0 0x7fa73c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.966+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa73c06b4d0 0x7fa73c06d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa750009fd0 tx=0x7fa75000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.968+0000 7fa76021f740 1 Processor -- start 2024-03-06T07:27:23.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.968+0000 7fa76021f740 1 -- start start 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa76021f740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168a7870 0x5608168a7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa76021f740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168ab8c0 0x5608168ad360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa76021f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168d7600 0x5608168d7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa76021f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608168116b0 con 0x5608168d7600 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa76021f740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608167baa60 con 0x5608168ab8c0 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa76021f740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x56081680d750 con 0x5608168a7870 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168d7600 0x5608168d7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168d7600 0x5608168d7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56204/0 (socket says 172.21.15.77:56204) 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa7577fe640 1 -- 172.21.15.77:0/573421964 learned_addr learned my addr 172.21.15.77:0/573421964 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa7567fc640 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168ab8c0 0x5608168ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.969+0000 7fa756ffd640 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168a7870 0x5608168a7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa7577fe640 1 -- 172.21.15.77:0/573421964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168a7870 msgr2=0x5608168a7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa7577fe640 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168a7870 0x5608168a7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa7577fe640 1 -- 172.21.15.77:0/573421964 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168ab8c0 msgr2=0x5608168ad360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa7577fe640 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168ab8c0 0x5608168ad360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa7577fe640 1 -- 172.21.15.77:0/573421964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608167d8f30 con 0x5608168d7600 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa7577fe640 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168d7600 0x5608168d7a80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fa7480aa960 tx=0x7fa74807abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfb5f3cc266807c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa74cff9640 1 -- 172.21.15.77:0/573421964 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa748083350 con 0x5608168d7600 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa74cff9640 1 -- 172.21.15.77:0/573421964 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa748078ff0 con 0x5608168d7600 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.970+0000 7fa74cff9640 1 -- 172.21.15.77:0/573421964 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa748077360 con 0x5608168d7600 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 -- 172.21.15.77:0/573421964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168d7600 msgr2=0x5608168d7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168d7600 0x5608168d7a80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fa7480aa960 tx=0x7fa74807abf0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 -- 172.21.15.77:0/573421964 shutdown_connections 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168d7600 0x5608168d7a80 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168ab8c0 0x5608168ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 --2- 172.21.15.77:0/573421964 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168a7870 0x5608168a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 -- 172.21.15.77:0/573421964 >> 172.21.15.77:0/573421964 conn(0x5608168a98f0 msgr2=0x5608168ab1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 -- 172.21.15.77:0/573421964 shutdown_connections 2024-03-06T07:27:23.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.971+0000 7fa76021f740 1 -- 172.21.15.77:0/573421964 wait complete. 2024-03-06T07:27:23.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.972+0000 7fa76021f740 1 -- start start 2024-03-06T07:27:23.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.972+0000 7fa76021f740 1 Processor -- start 2024-03-06T07:27:23.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa76021f740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d7600 0x5608168cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa76021f740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168cded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa76021f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168b9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa76021f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608167baa60 con 0x5608168b9a90 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa76021f740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608167cdea0 con 0x5608168d7600 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa76021f740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608168116b0 con 0x5608168cdaf0 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa76021f740 10 client.0 _ll_register_callbacks cb 0x5608167cd870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168b9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168b9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56214/0 (socket says 172.21.15.77:56214) 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7577fe640 1 -- 172.21.15.77:0/4273736850 learned_addr learned my addr 172.21.15.77:0/4273736850 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7567fc640 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa756ffd640 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d7600 0x5608168cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7577fe640 1 -- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 msgr2=0x5608168cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7577fe640 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7577fe640 1 -- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d7600 msgr2=0x5608168cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7577fe640 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d7600 0x5608168cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7577fe640 1 -- 172.21.15.77:0/4273736850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5608167ce270 con 0x5608168b9a90 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.973+0000 7fa7567fc640 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa7577fe640 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168b9f10 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa74802bf20 tx=0x7fa748009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d0744fe5c6e51a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa745ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa745ffb640 1 -- 172.21.15.77:0/4273736850 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7480830c0 con 0x5608168b9a90 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa745ffb640 1 -- 172.21.15.77:0/4273736850 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fa748044b40 con 0x5608168b9a90 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 1 -- 172.21.15.77:0/4273736850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa748078ff0 con 0x5608168b9a90 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa745ffb640 1 -- 172.21.15.77:0/4273736850 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa748083350 con 0x5608168b9a90 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 10 client.8136 fetch_fsmap learned FSMap version 74 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 1 -- 172.21.15.77:0/4273736850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x5608165262d0 con 0x5608168b9a90 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa745ffb640 1 -- 172.21.15.77:0/4273736850 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa748078ff0 con 0x5608168b9a90 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa745ffb640 1 -- 172.21.15.77:0/4273736850 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 74) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa74807a140 con 0x5608168b9a90 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 10 client.8136 fetch_fsmap finished waiting for FSMap version 74 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 10 client.8136 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 20 client.8136 populate_metadata read hostname 'smithi077' 2024-03-06T07:27:23.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] conn(0x5608168d3a40 0x5608168d3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 4 client.8136 mds_command: new command op to 8097 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa76021f740 1 -- 172.21.15.77:0/4273736850 --> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56081681ebf0 con 0x5608168d3a40 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.974+0000 7fa756ffd640 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] conn(0x5608168d3a40 0x5608168d3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa756ffd640 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] conn(0x5608168d3a40 0x5608168d3e20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6d8887b88726520 server_cookie=24a22e311485160a in_seq=0 out_seq=0 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa745ffb640 10 client.8136 ms_handle_connect on v2:172.21.15.77:6834/493819683 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa745ffb640 1 -- 172.21.15.77:0/4273736850 <== mds.0 v2:172.21.15.77:6834/493819683 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56081681ebf0 con 0x5608168d3a40 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa745ffb640 10 client.8136 handle_command_reply: tid=0 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 client.8136 shutdown 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 -- 172.21.15.77:0/4273736850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 msgr2=0x5608168b9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168b9f10 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa74802bf20 tx=0x7fa748009240 comp rx=0 tx=0).stop 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 -- 172.21.15.77:0/4273736850 shutdown_connections 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168b9f10 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] conn(0x5608168d3a40 0x5608168d3e20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa745ffb640 0 client.8136 ms_handle_reset on v2:172.21.15.77:6834/493819683 2024-03-06T07:27:23.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 --2- 172.21.15.77:0/4273736850 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d7600 0x5608168cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 -- 172.21.15.77:0/4273736850 >> 172.21.15.77:0/4273736850 conn(0x5608168a7870 msgr2=0x5608168d55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 -- 172.21.15.77:0/4273736850 shutdown_connections 2024-03-06T07:27:23.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.975+0000 7fa76021f740 1 -- 172.21.15.77:0/4273736850 wait complete. 2024-03-06T07:27:23.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.976+0000 7fa76021f740 20 client.8136 trim_cache size 0 max 16384 2024-03-06T07:27:23.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.992+0000 7fa76021f740 1 Processor -- start 2024-03-06T07:27:23.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.992+0000 7fa76021f740 1 -- start start 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa76021f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168d6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa76021f740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168d4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa76021f740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa76021f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7581ff490 con 0x5608168b9a90 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa76021f740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580b98f0 con 0x5608168d3a40 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa76021f740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7581fee90 con 0x5608168cdaf0 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168d6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168d6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56224/0 (socket says 172.21.15.77:56224) 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 -- 172.21.15.77:0/3452894698 learned_addr learned my addr 172.21.15.77:0/3452894698 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa7567fc640 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168d4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa7577fe640 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 -- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 msgr2=0x5608168d4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168d4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 -- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 msgr2=0x5608168d46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 -- 172.21.15.77:0/3452894698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608165f06b0 con 0x5608168b9a90 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa7567fc640 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168d4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa7577fe640 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168d6730 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa750016990 tx=0x7fa750015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befa3956bfaee93c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa747fff640 1 -- 172.21.15.77:0/3452894698 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7500260d0 con 0x5608168b9a90 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa747fff640 1 -- 172.21.15.77:0/3452894698 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75002c070 con 0x5608168b9a90 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.993+0000 7fa747fff640 1 -- 172.21.15.77:0/3452894698 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750024420 con 0x5608168b9a90 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 -- 172.21.15.77:0/3452894698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 msgr2=0x5608168d6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168d6730 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa750016990 tx=0x7fa750015000 comp rx=0 tx=0).stop 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 -- 172.21.15.77:0/3452894698 shutdown_connections 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168cdaf0 0x5608168d4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 --2- 172.21.15.77:0/3452894698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168b9a90 0x5608168d6730 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 -- 172.21.15.77:0/3452894698 >> 172.21.15.77:0/3452894698 conn(0x5608168a7870 msgr2=0x5608168b9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 -- 172.21.15.77:0/3452894698 shutdown_connections 2024-03-06T07:27:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.994+0000 7fa76021f740 1 -- 172.21.15.77:0/3452894698 wait complete. 2024-03-06T07:27:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.995+0000 7fa76021f740 1 -- start start 2024-03-06T07:27:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.995+0000 7fa76021f740 1 Processor -- start 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa76021f740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168b9a90 0x5608169649d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa76021f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168cdaf0 0x5608168d6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa76021f740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa76021f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608167baa60 con 0x5608168cdaf0 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa76021f740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608168116b0 con 0x5608168d3a40 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa76021f740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608168cc900 con 0x5608168b9a90 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa76021f740 10 client.0 _ll_register_callbacks cb 0x7fa7480094d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7567fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168cdaf0 0x5608168d6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7567fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168cdaf0 0x5608168d6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56226/0 (socket says 172.21.15.77:56226) 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7567fc640 1 -- 172.21.15.77:0/3218164658 learned_addr learned my addr 172.21.15.77:0/3218164658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168b9a90 0x5608169649d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7577fe640 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7567fc640 1 -- 172.21.15.77:0/3218164658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168b9a90 msgr2=0x5608169649d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7567fc640 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168b9a90 0x5608169649d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7567fc640 1 -- 172.21.15.77:0/3218164658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 msgr2=0x5608168d6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7567fc640 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.996+0000 7fa7567fc640 1 -- 172.21.15.77:0/3218164658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5608168f2180 con 0x5608168cdaf0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa7567fc640 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168cdaf0 0x5608168d6730 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa7580672f0 tx=0x7fa7400157f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a49cb1edc47fdc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa745ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa745ffb640 1 -- 172.21.15.77:0/3218164658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740017850 con 0x5608168cdaf0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa745ffb640 1 -- 172.21.15.77:0/3218164658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fa7400179f0 con 0x5608168cdaf0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 1 -- 172.21.15.77:0/3218164658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa75002c070 con 0x5608168cdaf0 2024-03-06T07:27:24.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa745ffb640 1 -- 172.21.15.77:0/3218164658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa740028050 con 0x5608168cdaf0 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 10 client.8142 fetch_fsmap learned FSMap version 74 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 1 -- 172.21.15.77:0/3218164658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x5608168ad510 con 0x5608168cdaf0 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa745ffb640 1 -- 172.21.15.77:0/3218164658 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75002c070 con 0x5608168cdaf0 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa745ffb640 1 -- 172.21.15.77:0/3218164658 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 74) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa74002b020 con 0x5608168cdaf0 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 10 client.8142 fetch_fsmap finished waiting for FSMap version 74 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 10 client.8142 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 20 client.8142 populate_metadata read hostname 'smithi077' 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] conn(0x5608168d7600 0x5608168d5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 4 client.8142 mds_command: new command op to 8097 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa76021f740 1 -- 172.21.15.77:0/3218164658 --> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560816818f40 con 0x5608168d7600 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.997+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] conn(0x5608168d7600 0x5608168d5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa745ffb640 10 client.8142 ms_handle_connect on v2:172.21.15.77:6834/493819683 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa756ffd640 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] conn(0x5608168d7600 0x5608168d5bd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=663b4941e3a1628a server_cookie=fc779681ac99dcaf in_seq=0 out_seq=0 2024-03-06T07:27:24.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa745ffb640 1 -- 172.21.15.77:0/3218164658 <== mds.0 v2:172.21.15.77:6834/493819683 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x560816818f40 con 0x5608168d7600 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa745ffb640 10 client.8142 handle_command_reply: tid=0 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 client.8142 shutdown 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 -- 172.21.15.77:0/3218164658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168cdaf0 msgr2=0x5608168d6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168cdaf0 0x5608168d6730 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa7580672f0 tx=0x7fa7400157f0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 -- 172.21.15.77:0/3218164658 shutdown_connections 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608168cdaf0 0x5608168d6730 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.77:6834/493819683,v1:172.21.15.77:6835/493819683] conn(0x5608168d7600 0x5608168d5bd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5608168d3a40 0x5608168d6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 --2- 172.21.15.77:0/3218164658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5608168b9a90 0x5608169649d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa745ffb640 0 client.8142 ms_handle_reset on v2:172.21.15.77:6834/493819683 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 -- 172.21.15.77:0/3218164658 >> 172.21.15.77:0/3218164658 conn(0x5608168a7870 msgr2=0x5608168ab080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 -- 172.21.15.77:0/3218164658 shutdown_connections 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 1 -- 172.21.15.77:0/3218164658 wait complete. 2024-03-06T07:27:24.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.998+0000 7fa76021f740 20 client.8142 trim_cache size 0 max 16384 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 8097, 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 4.4363425169999999, 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 74, 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 111, 2024-03-06T07:27:24.007 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 111, 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 5.3070361669999997 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:27:24.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 -- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa73c06b4d0 msgr2=0x7fa73c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa73c06b4d0 0x7fa73c06d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa750009fd0 tx=0x7fa75000d040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 -- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758162970 msgr2=0x7fa7580cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758162970 0x7fa7580cadb0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fa74802b950 tx=0x7fa7480046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 -- 172.21.15.77:0/3758253343 shutdown_connections 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa73c06b4d0 0x7fa73c06d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa758162970 0x7fa7580cadb0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa758161cf0 0x7fa7580b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 --2- 172.21.15.77:0/3758253343 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa7580b4460 0x7fa758153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 -- 172.21.15.77:0/3758253343 >> 172.21.15.77:0/3758253343 conn(0x7fa7580d3ce0 msgr2=0x7fa7580cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 -- 172.21.15.77:0/3758253343 shutdown_connections 2024-03-06T07:27:24.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:23.999+0000 7fa75d563640 1 -- 172.21.15.77:0/3758253343 wait complete. 2024-03-06T07:27:24.012 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.110+0000 7f3f4ff3e640 1 Processor -- start 2024-03-06T07:27:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.111+0000 7f3f4ff3e640 1 -- start start 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.111+0000 7f3f4ff3e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f480d00c0 0x7f3f480d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.111+0000 7f3f4ff3e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3f4806cf20 0x7f3f4806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.111+0000 7f3f4ff3e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f480d09e0 con 0x7f3f480cf850 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.111+0000 7f3f4ff3e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4806ca90 con 0x7f3f480d00c0 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.111+0000 7f3f4ff3e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4815e380 con 0x7f3f4806cf20 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.112+0000 7f3f4dcda640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f480d00c0 0x7f3f480d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.112+0000 7f3f4dcda640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f480d00c0 0x7f3f480d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56236/0 (socket says 172.21.15.77:56236) 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.112+0000 7f3f4dcda640 1 -- 172.21.15.77:0/3035120755 learned_addr learned my addr 172.21.15.77:0/3035120755 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.112+0000 7f3f4e4db640 1 --2- 172.21.15.77:0/3035120755 >> v2:172.21.15.158:3300/0 conn(0x7f3f4806cf20 0x7f3f4806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.112+0000 7f3f4dcda640 1 -- 172.21.15.77:0/3035120755 >> v2:172.21.15.158:3300/0 conn(0x7f3f4806cf20 msgr2=0x7f3f4806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.112+0000 7f3f4dcda640 1 --2- 172.21.15.77:0/3035120755 >> v2:172.21.15.158:3300/0 conn(0x7f3f4806cf20 0x7f3f4806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.112+0000 7f3f4dcda640 1 -- 172.21.15.77:0/3035120755 >> v1:172.21.15.158:6789/0 conn(0x7f3f480cf850 legacy=0x7f3f480cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.112+0000 7f3f4dcda640 1 -- 172.21.15.77:0/3035120755 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f480cdc10 con 0x7f3f480d00c0 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.113+0000 7f3f4dcda640 1 --2- 172.21.15.77:0/3035120755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f480d04a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f3f480cfda0 tx=0x7f3f4402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766dc8dc6986a6f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.113+0000 7f3f4ccd8640 1 -- 172.21.15.77:0/3035120755 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4402cb20 con 0x7f3f480d00c0 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.113+0000 7f3f4ccd8640 1 -- 172.21.15.77:0/3035120755 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4402ccc0 con 0x7f3f480d00c0 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.113+0000 7f3f4ccd8640 1 -- 172.21.15.77:0/3035120755 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44034610 con 0x7f3f480d00c0 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.114+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/3035120755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 msgr2=0x7f3f480d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.114+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/3035120755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f480d04a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f3f480cfda0 tx=0x7f3f4402bf80 comp rx=0 tx=0).stop 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.114+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/3035120755 shutdown_connections 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.114+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/3035120755 >> v2:172.21.15.158:3300/0 conn(0x7f3f4806cf20 0x7f3f4806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.114+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/3035120755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f480d04a0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.114+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/3035120755 >> 172.21.15.77:0/3035120755 conn(0x7f3f480b35a0 msgr2=0x7f3f480b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.114+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/3035120755 shutdown_connections 2024-03-06T07:27:24.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.114+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/3035120755 wait complete. 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.115+0000 7f3f4ff3e640 1 Processor -- start 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.115+0000 7f3f4ff3e640 1 -- start start 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.115+0000 7f3f4ff3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f4806cf20 0x7f3f480c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4ff3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f480cf850 0x7f3f480cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4ff3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f48155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4ff3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4806c720 con 0x7f3f480d00c0 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4ff3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4806c440 con 0x7f3f4806cf20 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4ff3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480b1030 con 0x7f3f480cf850 2024-03-06T07:27:24.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f48155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f48155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56238/0 (socket says 172.21.15.77:56238) 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 -- 172.21.15.77:0/1440851933 learned_addr learned my addr 172.21.15.77:0/1440851933 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4d4d9640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f480cf850 0x7f3f480cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4dcda640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f4806cf20 0x7f3f480c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 -- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f480cf850 msgr2=0x7f3f480cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f480cf850 0x7f3f480cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 -- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f4806cf20 msgr2=0x7f3f480c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f4806cf20 0x7f3f480c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 -- 172.21.15.77:0/1440851933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f480cb110 con 0x7f3f480d00c0 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4dcda640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f4806cf20 0x7f3f480c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4e4db640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f48155fe0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000e7f0 tx=0x7f3f4000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f3e7fc640 1 -- 172.21.15.77:0/1440851933 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40017070 con 0x7f3f480d00c0 2024-03-06T07:27:24.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f3e7fc640 1 -- 172.21.15.77:0/1440851933 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4000ee40 con 0x7f3f480d00c0 2024-03-06T07:27:24.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f48156520 con 0x7f3f480d00c0 2024-03-06T07:27:24.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f3e7fc640 1 -- 172.21.15.77:0/1440851933 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4000cb80 con 0x7f3f480d00c0 2024-03-06T07:27:24.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.116+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f480c9e10 con 0x7f3f480d00c0 2024-03-06T07:27:24.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.117+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4804f330 con 0x7f3f480d00c0 2024-03-06T07:27:24.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.118+0000 7f3f3e7fc640 1 -- 172.21.15.77:0/1440851933 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f4001a070 con 0x7f3f480d00c0 2024-03-06T07:27:24.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.118+0000 7f3f3e7fc640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f2c06b500 0x7f3f2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.118+0000 7f3f3e7fc640 1 -- 172.21.15.77:0/1440851933 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f3f4008a0c0 con 0x7f3f480d00c0 2024-03-06T07:27:24.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.118+0000 7f3f4dcda640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f2c06b500 0x7f3f2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.119+0000 7f3f4dcda640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f2c06b500 0x7f3f2c06d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3f44035000 tx=0x7f3f44030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.126+0000 7f3f3e7fc640 1 -- 172.21.15.77:0/1440851933 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f40058830 con 0x7f3f480d00c0 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.340+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3f480b1300 con 0x7f3f480d00c0 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.340+0000 7f3f3e7fc640 1 -- 172.21.15.77:0/1440851933 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3f4005c8d0 con 0x7f3f480d00c0 2024-03-06T07:27:24.342 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:24.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f2c06b500 msgr2=0x7f3f2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f2c06b500 0x7f3f2c06d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3f44035000 tx=0x7f3f44030000 comp rx=0 tx=0).stop 2024-03-06T07:27:24.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 msgr2=0x7f3f48155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f48155fe0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000e7f0 tx=0x7f3f4000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 shutdown_connections 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f2c06b500 0x7f3f2c06d9c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f480d00c0 0x7f3f48155fe0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3f480cf850 0x7f3f480cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 --2- 172.21.15.77:0/1440851933 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3f4806cf20 0x7f3f480c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 >> 172.21.15.77:0/1440851933 conn(0x7f3f480b35a0 msgr2=0x7f3f480b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 shutdown_connections 2024-03-06T07:27:24.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:24.342+0000 7f3f4ff3e640 1 -- 172.21.15.77:0/1440851933 wait complete. 2024-03-06T07:27:24.355 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:24.355 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:24.355 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:27:24.355 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:24.355 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:24.355 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:24.355 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:24.356 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:24.356 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:24.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:24.356 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:24.371 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42353sec preferred_lft 42353sec 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:24.372 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:24.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:24.374 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:24.437 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:27:24.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:24.439 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:27:24.494 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:27:24.494 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:24.494 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:24.494 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:24.494 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:24.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:24.495 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:27:24.495 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:27:24.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:24.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:24.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:24.495 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:27:24.564 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-03-06T07:27:24.563+0000 7f0ee3a43640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:24.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:24.589 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:24.645 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:24.645 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:27:24.715 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:24.715 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:24.715 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:24.771 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:24.771 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:24.771 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:24.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:27:24.896 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:24.952 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:27:24.952 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:27:24.952 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:27:24.952 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:27:24.952 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:27:24.952 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:24.953 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:24.954 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:24.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:24.955 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:27:25.007 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:25.007 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:25.049 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:27:25.075 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:25.132 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.133 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:27:25.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:27:25.137+0000 7f227e839480 -1 init, newargv = 0x5558ab0b8840 newargc=15 2024-03-06T07:27:25.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[74158]: starting ceph client 2024-03-06T07:27:25.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[74158]: starting fuse 2024-03-06T07:27:25.154 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:27:25.154 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:27:25.154 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:25.154 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:27:25.154 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:27:25.155 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:25.156 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:27:25.276 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.74158.asok 2024-03-06T07:27:25.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:25.286 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.74158.asok 2024-03-06T07:27:25.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.286 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.74158.asok status 2024-03-06T07:27:25.424 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "74158", 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:27:25.425 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 8154, 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 8154 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 220836789 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:27:25.426 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 220836789 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.8154 192.168.0.1:0/220836789", 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/220836789", 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 74, 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 111, 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:27:25.427 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:27:25.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.434 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:27:25.434 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:25.448 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:27:25.448 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:25.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.449 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:25.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.516 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:27:25.574 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:25.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.714+0000 7fa4e3c84640 1 Processor -- start 2024-03-06T07:27:25.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.714+0000 7fa4e3c84640 1 -- start start 2024-03-06T07:27:25.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.715+0000 7fa4e3c84640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa4dc15dfe0 0x7fa4dc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.715+0000 7fa4e3c84640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc0b3430 con 0x7fa4dc0cf830 2024-03-06T07:27:25.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.715+0000 7fa4e3c84640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc0b3710 con 0x7fa4dc0d29c0 2024-03-06T07:27:25.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.715+0000 7fa4e3c84640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4dc0b31e0 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.715+0000 7fa4e1a20640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa4dc15dfe0 0x7fa4dc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.715+0000 7fa4e121f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa4dc0cf830 0x7fa4dc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59860/0 (socket says 172.21.15.77:59860) 2024-03-06T07:27:25.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.715+0000 7fa4e121f640 1 -- 172.21.15.77:0/666350377 learned_addr learned my addr 172.21.15.77:0/666350377 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:25.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.716+0000 7fa4e1a20640 1 -- 172.21.15.77:0/666350377 >> v1:172.21.15.158:6789/0 conn(0x7fa4dc0d29c0 legacy=0x7fa4dc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.716+0000 7fa4e1a20640 1 -- 172.21.15.77:0/666350377 >> v1:172.21.15.77:6789/0 conn(0x7fa4dc0cf830 legacy=0x7fa4dc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.716+0000 7fa4e1a20640 1 -- 172.21.15.77:0/666350377 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc0b39f0 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.716+0000 7fa4e1a20640 1 --2- 172.21.15.77:0/666350377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc0cf2f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0b3360 tx=0x7fa4d402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58159acd847748da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.716+0000 7fa4e0a1e640 1 -- 172.21.15.77:0/666350377 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4039070 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.716+0000 7fa4e0a1e640 1 -- 172.21.15.77:0/666350377 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d402cc50 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.716+0000 7fa4e0a1e640 1 -- 172.21.15.77:0/666350377 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4034440 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.717+0000 7fa4e3c84640 1 -- 172.21.15.77:0/666350377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 msgr2=0x7fa4dc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.717+0000 7fa4e3c84640 1 --2- 172.21.15.77:0/666350377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc0cf2f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0b3360 tx=0x7fa4d402bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.717+0000 7fa4e3c84640 1 -- 172.21.15.77:0/666350377 shutdown_connections 2024-03-06T07:27:25.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.717+0000 7fa4e3c84640 1 --2- 172.21.15.77:0/666350377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc0cf2f0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.717+0000 7fa4e3c84640 1 -- 172.21.15.77:0/666350377 >> 172.21.15.77:0/666350377 conn(0x7fa4dc0d3cd0 msgr2=0x7fa4dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.717+0000 7fa4e3c84640 1 -- 172.21.15.77:0/666350377 shutdown_connections 2024-03-06T07:27:25.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.717+0000 7fa4e3c84640 1 -- 172.21.15.77:0/666350377 wait complete. 2024-03-06T07:27:25.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.718+0000 7fa4e3c84640 1 Processor -- start 2024-03-06T07:27:25.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.718+0000 7fa4e3c84640 1 -- start start 2024-03-06T07:27:25.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.719+0000 7fa4e3c84640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4dc0cf830 0x7fa4dc1610e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.719+0000 7fa4e3c84640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa4dc0d29c0 0x7fa4dc158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.719+0000 7fa4e3c84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.719+0000 7fa4e3c84640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc15e3c0 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.719+0000 7fa4e3c84640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc0b3d60 con 0x7fa4dc0cf830 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.719+0000 7fa4e3c84640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc15e540 con 0x7fa4dc0d29c0 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc158220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56262/0 (socket says 172.21.15.77:56262) 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 -- 172.21.15.77:0/2139341074 learned_addr learned my addr 172.21.15.77:0/2139341074 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e1a20640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4dc0cf830 0x7fa4dc1610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e121f640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa4dc0d29c0 0x7fa4dc158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 -- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa4dc0d29c0 msgr2=0x7fa4dc158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa4dc0d29c0 0x7fa4dc158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 -- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4dc0cf830 msgr2=0x7fa4dc1610e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4dc0cf830 0x7fa4dc1610e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 -- 172.21.15.77:0/2139341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc158fc0 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e121f640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa4dc0d29c0 0x7fa4dc158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e2221640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc158220 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc00c3e0 tx=0x7fa4cc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4da7fc640 1 -- 172.21.15.77:0/2139341074 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc011790 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4da7fc640 1 -- 172.21.15.77:0/2139341074 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc017070 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc159240 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4da7fc640 1 -- 172.21.15.77:0/2139341074 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc011b70 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.720+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc159800 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.721+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc04f330 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.722+0000 7fa4da7fc640 1 -- 172.21.15.77:0/2139341074 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4cc011d10 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.722+0000 7fa4da7fc640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa4c806b4d0 0x7fa4c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.722+0000 7fa4da7fc640 1 -- 172.21.15.77:0/2139341074 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fa4cc089fe0 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.722+0000 7fa4e1a20640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa4c806b4d0 0x7fa4c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.723+0000 7fa4e1a20640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa4c806b4d0 0x7fa4c806d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4036000 tx=0x7fa4d4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.726+0000 7fa4da7fc640 1 -- 172.21.15.77:0/2139341074 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4dc158fc0 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.965+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4dc0b10b0 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.965+0000 7fa4da7fc640 1 -- 172.21.15.77:0/2139341074 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fa4cc058db0 con 0x7fa4dc15dfe0 2024-03-06T07:27:25.967 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:25.967 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:25.968 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:25.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa4c806b4d0 msgr2=0x7fa4c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa4c806b4d0 0x7fa4c806d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4036000 tx=0x7fa4d4030000 comp rx=0 tx=0).stop 2024-03-06T07:27:25.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 msgr2=0x7fa4dc158220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc158220 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc00c3e0 tx=0x7fa4cc00c8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 shutdown_connections 2024-03-06T07:27:25.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa4c806b4d0 0x7fa4c806d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4dc15dfe0 0x7fa4dc158220 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa4dc0d29c0 0x7fa4dc158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 --2- 172.21.15.77:0/2139341074 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa4dc0cf830 0x7fa4dc1610e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 >> 172.21.15.77:0/2139341074 conn(0x7fa4dc0d3cd0 msgr2=0x7fa4dc06ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 shutdown_connections 2024-03-06T07:27:25.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:25.967+0000 7fa4e3c84640 1 -- 172.21.15.77:0/2139341074 wait complete. 2024-03-06T07:27:25.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:27:26.097 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:27:26.097 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:27:26.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:26.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.197+0000 7f6ba2464640 1 Processor -- start 2024-03-06T07:27:26.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.197+0000 7f6ba2464640 1 -- start start 2024-03-06T07:27:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.197+0000 7f6ba2464640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b9c0d22b0 0x7f6b9c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.197+0000 7f6ba2464640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c0b3430 con 0x7f6b9c15dfe0 2024-03-06T07:27:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.197+0000 7f6ba2464640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c0b3710 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.197+0000 7f6ba2464640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c0d2c50 con 0x7f6b9c0d22b0 2024-03-06T07:27:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6b9c15dfe0 0x7f6b9c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59866/0 (socket says 172.21.15.77:59866) 2024-03-06T07:27:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9bfff640 1 -- 172.21.15.77:0/1589722225 learned_addr learned my addr 172.21.15.77:0/1589722225 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9b7fe640 1 --2- 172.21.15.77:0/1589722225 >> v2:172.21.15.158:3300/0 conn(0x7f6b9c0d22b0 0x7f6b9c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9affd640 1 -- 172.21.15.77:0/1589722225 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268544519 0 0) 0x7f6b9c0b3430 con 0x7f6b9c15dfe0 2024-03-06T07:27:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9affd640 1 -- 172.21.15.77:0/1589722225 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b84003520 con 0x7f6b9c15dfe0 2024-03-06T07:27:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9affd640 1 -- 172.21.15.77:0/1589722225 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363817903 0 0) 0x7f6b9c0b3710 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9affd640 1 -- 172.21.15.77:0/1589722225 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b9c0b3430 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9b7fe640 1 -- 172.21.15.77:0/1589722225 >> v1:172.21.15.158:6789/0 conn(0x7f6b9c0b41f0 legacy=0x7f6b9c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9b7fe640 1 -- 172.21.15.77:0/1589722225 >> v1:172.21.15.77:6789/0 conn(0x7f6b9c15dfe0 legacy=0x7f6b9c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.198+0000 7f6b9b7fe640 1 -- 172.21.15.77:0/1589722225 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c0b39f0 con 0x7f6b9c0d22b0 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.199+0000 7f6b9b7fe640 1 --2- 172.21.15.77:0/1589722225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c0d22b0 0x7f6b9c0d2710 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6b900097d0 tx=0x7f6b9002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65f63726f766971b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.199+0000 7f6b9affd640 1 -- 172.21.15.77:0/1589722225 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b90034070 con 0x7f6b9c0d22b0 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.199+0000 7f6b9affd640 1 -- 172.21.15.77:0/1589722225 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b90030610 con 0x7f6b9c0d22b0 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.199+0000 7f6b9affd640 1 -- 172.21.15.77:0/1589722225 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b900309c0 con 0x7f6b9c0d22b0 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.200+0000 7f6ba2464640 1 -- 172.21.15.77:0/1589722225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c0d22b0 msgr2=0x7f6b9c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.200+0000 7f6ba2464640 1 --2- 172.21.15.77:0/1589722225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c0d22b0 0x7f6b9c0d2710 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6b900097d0 tx=0x7f6b9002b960 comp rx=0 tx=0).stop 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.200+0000 7f6ba2464640 1 -- 172.21.15.77:0/1589722225 shutdown_connections 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.200+0000 7f6ba2464640 1 --2- 172.21.15.77:0/1589722225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c0d22b0 0x7f6b9c0d2710 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.200+0000 7f6ba2464640 1 -- 172.21.15.77:0/1589722225 >> 172.21.15.77:0/1589722225 conn(0x7f6b9c0d3cd0 msgr2=0x7f6b9c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.200+0000 7f6ba2464640 1 -- 172.21.15.77:0/1589722225 shutdown_connections 2024-03-06T07:27:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.200+0000 7f6ba2464640 1 -- 172.21.15.77:0/1589722225 wait complete. 2024-03-06T07:27:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.201+0000 7f6ba2464640 1 Processor -- start 2024-03-06T07:27:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.201+0000 7f6ba2464640 1 -- start start 2024-03-06T07:27:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6ba2464640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b9c0b41f0 0x7f6b9c152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6ba2464640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b9c0d22b0 0x7f6b9c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6ba2464640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c15dfe0 0x7f6b9c160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6ba2464640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c0b3d60 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6ba2464640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c0b31e0 con 0x7f6b9c15dfe0 2024-03-06T07:27:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6ba2464640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c0d1c70 con 0x7f6b9c0d22b0 2024-03-06T07:27:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b9c0b41f0 0x7f6b9c152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b9c0b41f0 0x7f6b9c152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56274/0 (socket says 172.21.15.77:56274) 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 -- 172.21.15.77:0/2574534221 learned_addr learned my addr 172.21.15.77:0/2574534221 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6ba0a01640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c15dfe0 0x7f6b9c160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9b7fe640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b9c0d22b0 0x7f6b9c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 -- 172.21.15.77:0/2574534221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b9c0d22b0 msgr2=0x7f6b9c15e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b9c0d22b0 0x7f6b9c15e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 -- 172.21.15.77:0/2574534221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c15dfe0 msgr2=0x7f6b9c160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c15dfe0 0x7f6b9c160840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 -- 172.21.15.77:0/2574534221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c15ee10 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.202+0000 7f6b9bfff640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b9c0b41f0 0x7f6b9c152150 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00c410 tx=0x7f6b8c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.203+0000 7f6b98ff9640 1 -- 172.21.15.77:0/2574534221 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c017070 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.203+0000 7f6b98ff9640 1 -- 172.21.15.77:0/2574534221 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c005ce0 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.203+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c15f0f0 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.203+0000 7f6b98ff9640 1 -- 172.21.15.77:0/2574534221 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c0072d0 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.203+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c160320 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.204+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9c04f330 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.204+0000 7f6b98ff9640 1 -- 172.21.15.77:0/2574534221 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b8c01a070 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.205+0000 7f6b98ff9640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b8806b4d0 0x7f6b8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.205+0000 7f6b9b7fe640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b8806b4d0 0x7f6b8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.205+0000 7f6b98ff9640 1 -- 172.21.15.77:0/2574534221 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6b8c0895b0 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.205+0000 7f6b9b7fe640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b8806b4d0 0x7f6b8806d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6b90036000 tx=0x7f6b90008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.209+0000 7f6b98ff9640 1 -- 172.21.15.77:0/2574534221 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b8c057d30 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.456+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b9c15f2c0 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.456+0000 7f6b98ff9640 1 -- 172.21.15.77:0/2574534221 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f6b8c05bdd0 con 0x7f6b9c0b41f0 2024-03-06T07:27:26.458 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:26.458 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:26.458 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:26.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b8806b4d0 msgr2=0x7f6b8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b8806b4d0 0x7f6b8806d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6b90036000 tx=0x7f6b90008030 comp rx=0 tx=0).stop 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b9c0b41f0 msgr2=0x7f6b9c152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b9c0b41f0 0x7f6b9c152150 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00c410 tx=0x7f6b8c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 shutdown_connections 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b9c15dfe0 0x7f6b9c160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b9c0d22b0 0x7f6b9c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b8806b4d0 0x7f6b8806d990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 --2- 172.21.15.77:0/2574534221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b9c0b41f0 0x7f6b9c152150 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 >> 172.21.15.77:0/2574534221 conn(0x7f6b9c0d3cd0 msgr2=0x7f6b9c0b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 shutdown_connections 2024-03-06T07:27:26.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:26.458+0000 7f6ba2464640 1 -- 172.21.15.77:0/2574534221 wait complete. 2024-03-06T07:27:26.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:27:26.592 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:27:26.592 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:27:30.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:30.756 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:56.594 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:27:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.636+0000 7f101c86cf40 1 Processor -- start 2024-03-06T07:27:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.636+0000 7f101c86cf40 1 -- start start 2024-03-06T07:27:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.636+0000 7f101c86cf40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e836810 0x55f65e875be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.636+0000 7f101c86cf40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e87dda0 0x55f65e8ec310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101c86cf40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e8f3690 0x55f65e8f2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101c86cf40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65e864b60 con 0x55f65e8f3690 2024-03-06T07:27:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101c86cf40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65e7dbb70 con 0x55f65e87dda0 2024-03-06T07:27:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101c86cf40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f65e836bf0 con 0x55f65e836810 2024-03-06T07:27:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e8f3690 0x55f65e8f2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e8f3690 0x55f65e8f2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36990/0 (socket says 172.21.15.77:36990) 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 -- 172.21.15.77:0/1494356685 learned_addr learned my addr 172.21.15.77:0/1494356685 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f1011b6c640 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e87dda0 0x55f65e8ec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101a36d640 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e836810 0x55f65e875be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 -- 172.21.15.77:0/1494356685 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e836810 msgr2=0x55f65e875be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e836810 0x55f65e875be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 -- 172.21.15.77:0/1494356685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e87dda0 msgr2=0x55f65e8ec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e87dda0 0x55f65e8ec310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 -- 172.21.15.77:0/1494356685 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f65e874c10 con 0x55f65e8f3690 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101ab6e640 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e8f3690 0x55f65e8f2190 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f1014001690 tx=0x7f1014001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d34e461c263e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101936b640 1 -- 172.21.15.77:0/1494356685 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1014004520 con 0x55f65e8f3690 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101936b640 1 -- 172.21.15.77:0/1494356685 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10140046c0 con 0x55f65e8f3690 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.637+0000 7f101936b640 1 -- 172.21.15.77:0/1494356685 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1014004a70 con 0x55f65e8f3690 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 -- 172.21.15.77:0/1494356685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e8f3690 msgr2=0x55f65e8f2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e8f3690 0x55f65e8f2190 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f1014001690 tx=0x7f1014001b50 comp rx=0 tx=0).stop 2024-03-06T07:27:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 -- 172.21.15.77:0/1494356685 shutdown_connections 2024-03-06T07:27:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e8f3690 0x55f65e8f2190 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e87dda0 0x55f65e8ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 --2- 172.21.15.77:0/1494356685 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e836810 0x55f65e875be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 -- 172.21.15.77:0/1494356685 >> 172.21.15.77:0/1494356685 conn(0x55f65e860340 msgr2=0x55f65e87c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 -- 172.21.15.77:0/1494356685 shutdown_connections 2024-03-06T07:27:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 -- 172.21.15.77:0/1494356685 wait complete. 2024-03-06T07:27:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 Processor -- start 2024-03-06T07:27:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.638+0000 7f101c86cf40 1 -- start start 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101c86cf40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e836810 0x55f65e8e20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101c86cf40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e87dda0 0x55f65e8f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101c86cf40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e8f3690 0x55f65e9a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101c86cf40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65e8770c0 con 0x55f65e87dda0 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101c86cf40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65e876e20 con 0x55f65e836810 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101c86cf40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f65e8edf70 con 0x55f65e8f3690 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f1011b6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e87dda0 0x55f65e8f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f1011b6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e87dda0 0x55f65e8f0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37000/0 (socket says 172.21.15.77:37000) 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 learned_addr learned my addr 172.21.15.77:0/3625213221 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e8f3690 0x55f65e9a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e836810 0x55f65e8e20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e8f3690 msgr2=0x55f65e9a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e8f3690 0x55f65e9a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e836810 msgr2=0x55f65e8e20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e836810 0x55f65e8e20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f65e8efc70 con 0x55f65e87dda0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55f65e836810 0x55f65e8e20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.639+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55f65e8f3690 0x55f65e9a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.640+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e87dda0 0x55f65e8f0cd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f10040086b0 tx=0x7f10040087a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.640+0000 7f1013fff640 1 -- 172.21.15.77:0/3625213221 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004013760 con 0x55f65e87dda0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.640+0000 7f1013fff640 1 -- 172.21.15.77:0/3625213221 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1004019070 con 0x55f65e87dda0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.640+0000 7f1013fff640 1 -- 172.21.15.77:0/3625213221 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004013b40 con 0x55f65e87dda0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.640+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f65e8eff10 con 0x55f65e87dda0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.640+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f65e8f04d0 con 0x55f65e87dda0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.641+0000 7f1013fff640 1 -- 172.21.15.77:0/3625213221 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1004013ce0 con 0x55f65e87dda0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.641+0000 7f1013fff640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ffc069ee0 0x7f0ffc06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.641+0000 7f1013fff640 1 -- 172.21.15.77:0/3625213221 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1004089090 con 0x55f65e87dda0 2024-03-06T07:27:56.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.642+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ffc069ee0 0x7f0ffc06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.642+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55f65e879870 0x55f65e879c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.642+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.642+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ffc069ee0 0x7f0ffc06c3a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f100c057000 tx=0x7f100c052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.642+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55f65e879870 0x55f65e879c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.642+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55f65e879870 0x55f65e879c50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.643+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 1 ==== osd_op_reply(1 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.643+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f1014013930 0x7f10140123c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.643+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014013930 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.643+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f1014013930 0x7f10140123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.643+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f1014013930 0x7f10140123c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.1 v2:172.21.15.77:6800/1774367160 1 ==== osd_op_reply(2 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f1014013930 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f100c058400 0x7f100c058840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f100c058400 0x7f100c058840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f100c058400 0x7f100c058840 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 1 ==== osd_op_reply(3 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1004057a10 0x7f1004056670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.644+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1004057a10 0x7f1004056670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.645+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1004057a10 0x7f1004056670 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.645+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 1 ==== osd_op_reply(4 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.645+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.645+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 2 ==== osd_op_reply(5 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.645+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.646+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 2 ==== osd_op_reply(6 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.646+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.646+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 2 ==== osd_op_reply(7 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.646+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.646+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 3 ==== osd_op_reply(8 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.646+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.647+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 3 ==== osd_op_reply(9 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.647+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.647+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 3 ==== osd_op_reply(10 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.647+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.647+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 4 ==== osd_op_reply(11 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.647+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f100406f0b0 0x7f10040878f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.647+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100406f0b0 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.647+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f100406f0b0 0x7f10040878f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.648+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f100406f0b0 0x7f10040878f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.648+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.5 v2:172.21.15.158:6820/3476303095 1 ==== osd_op_reply(12 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f100406f0b0 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.648+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.649+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 4 ==== osd_op_reply(13 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.649+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f1014023970 0x7f1014021440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.649+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.649+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f1014023970 0x7f1014021440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.649+0000 7f1011b6c640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f1014023970 0x7f1014021440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.650+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 1 ==== osd_op_reply(14 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.650+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014013930 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.650+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.1 v2:172.21.15.77:6800/1774367160 2 ==== osd_op_reply(15 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f1014013930 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.650+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.650+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 4 ==== osd_op_reply(16 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.650+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.651+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 2 ==== osd_op_reply(17 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.651+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100406f0b0 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.651+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.5 v2:172.21.15.158:6820/3476303095 2 ==== osd_op_reply(18 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f100406f0b0 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.651+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.651+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 5 ==== osd_op_reply(19 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.651+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.652+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 5 ==== osd_op_reply(20 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.652+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.652+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 6 ==== osd_op_reply(21 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.652+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.652+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 6 ==== osd_op_reply(22 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.652+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.653+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 5 ==== osd_op_reply(23 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.653+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014013930 2024-03-06T07:27:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.653+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.1 v2:172.21.15.77:6800/1774367160 3 ==== osd_op_reply(24 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f1014013930 2024-03-06T07:27:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.653+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f100c075040 0x7f100c074410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.653+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c075040 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.653+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f100c075040 0x7f100c074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.654+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f100c075040 0x7f100c074410 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.654+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.6 v2:172.21.15.158:6801/2971141295 1 ==== osd_op_reply(25 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f100c075040 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.654+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.654+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 6 ==== osd_op_reply(26 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.654+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100406f0b0 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.655+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.5 v2:172.21.15.158:6820/3476303095 3 ==== osd_op_reply(27 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f100406f0b0 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.655+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.655+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 7 ==== osd_op_reply(28 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.655+0000 7f101ab6e640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f101403f910 0x7f1014031400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.655+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f101403f910 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.655+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f101403f910 0x7f1014031400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.656+0000 7f101a36d640 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f101403f910 0x7f1014031400 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.656+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.2 v2:172.21.15.77:6824/107698445 1 ==== osd_op_reply(29 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f101403f910 2024-03-06T07:27:56.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.656+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c075040 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.656+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.6 v2:172.21.15.158:6801/2971141295 2 ==== osd_op_reply(30 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f100c075040 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.656+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014013930 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.657+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.1 v2:172.21.15.77:6800/1774367160 4 ==== osd_op_reply(31 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f1014013930 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.657+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.657+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 3 ==== osd_op_reply(32 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.657+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.658+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 8 ==== osd_op_reply(33 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.658+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.658+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 9 ==== osd_op_reply(34 [pgnls start_epoch 113 out=49b] v107'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.658+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.658+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 7 ==== osd_op_reply(35 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.658+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014013930 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.658+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.1 v2:172.21.15.77:6800/1774367160 5 ==== osd_op_reply(36 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f1014013930 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.658+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.659+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 8 ==== osd_op_reply(37 [pgnls start_epoch 113 out=49b] v107'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.659+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.659+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 4 ==== osd_op_reply(38 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.659+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100406f0b0 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.659+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.5 v2:172.21.15.158:6820/3476303095 4 ==== osd_op_reply(39 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f100406f0b0 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.659+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100406f0b0 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.660+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.5 v2:172.21.15.158:6820/3476303095 5 ==== osd_op_reply(40 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f100406f0b0 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.660+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c075040 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.660+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.6 v2:172.21.15.158:6801/2971141295 3 ==== osd_op_reply(41 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f100c075040 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.660+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014013930 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.660+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.1 v2:172.21.15.77:6800/1774367160 6 ==== osd_op_reply(42 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f1014013930 2024-03-06T07:27:56.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.660+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100406f0b0 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.661+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.5 v2:172.21.15.158:6820/3476303095 6 ==== osd_op_reply(43 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f100406f0b0 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.661+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.661+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 5 ==== osd_op_reply(44 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.661+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.661+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 7 ==== osd_op_reply(45 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.661+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.662+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 8 ==== osd_op_reply(46 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.662+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.662+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 9 ==== osd_op_reply(47 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.662+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.662+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 10 ==== osd_op_reply(48 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.662+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.663+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 6 ==== osd_op_reply(49 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.663+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.663+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 7 ==== osd_op_reply(50 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.663+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c058400 2024-03-06T07:27:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.663+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.3 v2:172.21.15.77:6816/461341877 9 ==== osd_op_reply(51 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f100c058400 2024-03-06T07:27:56.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.663+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x55f65e879870 2024-03-06T07:27:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.663+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.7 v2:172.21.15.158:6816/3281717214 11 ==== osd_op_reply(52 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x55f65e879870 2024-03-06T07:27:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.663+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.664+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 10 ==== osd_op_reply(53 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.664+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.664+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 8 ==== osd_op_reply(54 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.664+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.664+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 9 ==== osd_op_reply(55 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.665+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f101403f910 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.665+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.2 v2:172.21.15.77:6824/107698445 2 ==== osd_op_reply(56 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f101403f910 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.665+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.665+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 11 ==== osd_op_reply(57 [pgnls start_epoch 113 out=81b] v107'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.665+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.666+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 10 ==== osd_op_reply(58 [pgnls start_epoch 113 out=49b] v107'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.666+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c075040 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.666+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.6 v2:172.21.15.158:6801/2971141295 4 ==== osd_op_reply(59 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f100c075040 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.666+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.666+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 12 ==== osd_op_reply(60 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.666+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014013930 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.666+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 <== osd.1 v2:172.21.15.77:6800/1774367160 7 ==== osd_op_reply(61 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f100c050cf0 con 0x7f1014013930 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.666+0000 7f101a36d640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1014023970 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.667+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 <== osd.4 v2:172.21.15.158:6800/2421130833 11 ==== osd_op_reply(62 [pgnls start_epoch 113 out=81b] v107'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f1004053e20 con 0x7f1014023970 2024-03-06T07:27:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.667+0000 7f1011b6c640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f100c075040 2024-03-06T07:27:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.667+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.6 v2:172.21.15.158:6801/2971141295 5 ==== osd_op_reply(63 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f100c075040 2024-03-06T07:27:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.667+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55f65e87a190 con 0x7f1004057a10 2024-03-06T07:27:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.667+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 <== osd.0 v2:172.21.15.77:6808/3119591583 13 ==== osd_op_reply(64 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f101400f030 con 0x7f1004057a10 2024-03-06T07:27:56.671 INFO:teuthology.orchestra.run.smithi077.stdout:10000000001.00000000 2024-03-06T07:27:56.671 INFO:teuthology.orchestra.run.smithi077.stdout:10000000001.00000001 2024-03-06T07:27:56.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1004057a10 msgr2=0x7f1004056670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1004057a10 0x7f1004056670 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f1014013930 msgr2=0x7f10140123c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f1014013930 0x7f10140123c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f101403f910 msgr2=0x7f1014031400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f101403f910 0x7f1014031400 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f100c058400 msgr2=0x7f100c058840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f100c058400 0x7f100c058840 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 reap_dead start 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 reap_dead start 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f1014023970 msgr2=0x7f1014021440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f1014023970 0x7f1014021440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f100406f0b0 msgr2=0x7f10040878f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f100406f0b0 0x7f10040878f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f100c075040 msgr2=0x7f100c074410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f100c075040 0x7f100c074410 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55f65e879870 msgr2=0x55f65e879c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55f65e879870 0x55f65e879c50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ffc069ee0 msgr2=0x7f0ffc06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0ffc069ee0 0x7f0ffc06c3a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f100c057000 tx=0x7f100c052000 comp rx=0 tx=0).stop 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e87dda0 msgr2=0x55f65e8f0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101ab6e640 1 -- 172.21.15.77:0/3625213221 reap_dead start 2024-03-06T07:27:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e87dda0 0x55f65e8f0cd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f10040086b0 tx=0x7f10040087a0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 shutdown_connections 2024-03-06T07:27:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 --2- 172.21.15.77:0/3625213221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f65e87dda0 0x55f65e8f0cd0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 >> 172.21.15.77:0/3625213221 conn(0x55f65e860340 msgr2=0x55f65e8619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 shutdown_connections 2024-03-06T07:27:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.668+0000 7f101c86cf40 1 -- 172.21.15.77:0/3625213221 wait complete. 2024-03-06T07:27:56.675 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:27:56.675 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.813+0000 7fb9d654b640 1 Processor -- start 2024-03-06T07:27:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.813+0000 7fb9d654b640 1 -- start start 2024-03-06T07:27:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.814+0000 7fb9d654b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d015dfe0 0x7fb9d00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.814+0000 7fb9d654b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9d00d29c0 0x7fb9d00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.814+0000 7fb9d654b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d00b3430 con 0x7fb9d00cf830 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.814+0000 7fb9d654b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d00b31e0 con 0x7fb9d00d29c0 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.814+0000 7fb9d654b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d00b3d60 con 0x7fb9d015dfe0 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.814+0000 7fb9d4ae8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9d00d29c0 0x7fb9d00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.814+0000 7fb9cf7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb9d00cf830 0x7fb9d00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38406/0 (socket says 172.21.15.77:38406) 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.814+0000 7fb9cf7fe640 1 -- 172.21.15.77:0/2747955956 learned_addr learned my addr 172.21.15.77:0/2747955956 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.815+0000 7fb9cffff640 1 --2- 172.21.15.77:0/2747955956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d015dfe0 0x7fb9d00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.815+0000 7fb9d4ae8640 1 -- 172.21.15.77:0/2747955956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d015dfe0 msgr2=0x7fb9d00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.815+0000 7fb9d4ae8640 1 --2- 172.21.15.77:0/2747955956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d015dfe0 0x7fb9d00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.815+0000 7fb9d4ae8640 1 -- 172.21.15.77:0/2747955956 >> v1:172.21.15.77:6789/0 conn(0x7fb9d00cf830 legacy=0x7fb9d00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.815+0000 7fb9d4ae8640 1 -- 172.21.15.77:0/2747955956 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d00b3710 con 0x7fb9d00d29c0 2024-03-06T07:27:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.815+0000 7fb9ceffd640 1 -- 172.21.15.77:0/2747955956 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122210981 0 0) 0x7fb9d00b3430 con 0x7fb9d00cf830 2024-03-06T07:27:56.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.815+0000 7fb9d4ae8640 1 --2- 172.21.15.77:0/2747955956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00d29c0 0x7fb9d00b41f0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009d90 tx=0x7fb9c802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eef6581b225a16f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9ceffd640 1 -- 172.21.15.77:0/2747955956 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8039070 con 0x7fb9d00d29c0 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9ceffd640 1 -- 172.21.15.77:0/2747955956 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c802cc50 con 0x7fb9d00d29c0 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9ceffd640 1 -- 172.21.15.77:0/2747955956 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8034440 con 0x7fb9d00d29c0 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9d654b640 1 -- 172.21.15.77:0/2747955956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00d29c0 msgr2=0x7fb9d00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9d654b640 1 --2- 172.21.15.77:0/2747955956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00d29c0 0x7fb9d00b41f0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009d90 tx=0x7fb9c802bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9d654b640 1 -- 172.21.15.77:0/2747955956 shutdown_connections 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9d654b640 1 --2- 172.21.15.77:0/2747955956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00d29c0 0x7fb9d00b41f0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9d654b640 1 --2- 172.21.15.77:0/2747955956 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d015dfe0 0x7fb9d00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9d654b640 1 -- 172.21.15.77:0/2747955956 >> 172.21.15.77:0/2747955956 conn(0x7fb9d00d3cd0 msgr2=0x7fb9d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9d654b640 1 -- 172.21.15.77:0/2747955956 shutdown_connections 2024-03-06T07:27:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.816+0000 7fb9d654b640 1 -- 172.21.15.77:0/2747955956 wait complete. 2024-03-06T07:27:56.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.818+0000 7fb9d654b640 1 Processor -- start 2024-03-06T07:27:56.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.818+0000 7fb9d654b640 1 -- start start 2024-03-06T07:27:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9d654b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00cf830 0x7fb9d00b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9d654b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d00d29c0 0x7fb9d014d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9d654b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9d015dfe0 0x7fb9d00cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9d654b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d00d1df0 con 0x7fb9d00cf830 2024-03-06T07:27:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9d654b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d00d1c70 con 0x7fb9d015dfe0 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9d654b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d015e3c0 con 0x7fb9d00d29c0 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00cf830 0x7fb9d00b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00cf830 0x7fb9d00b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37016/0 (socket says 172.21.15.77:37016) 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.819+0000 7fb9cffff640 1 -- 172.21.15.77:0/4198518069 learned_addr learned my addr 172.21.15.77:0/4198518069 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9cf7fe640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d00d29c0 0x7fb9d014d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9d4ae8640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9d015dfe0 0x7fb9d00cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9cffff640 1 -- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d00d29c0 msgr2=0x7fb9d014d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9cffff640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d00d29c0 0x7fb9d014d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9cffff640 1 -- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9d015dfe0 msgr2=0x7fb9d00cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9cffff640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9d015dfe0 0x7fb9d00cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9cffff640 1 -- 172.21.15.77:0/4198518069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d014da50 con 0x7fb9d00cf830 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9cf7fe640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d00d29c0 0x7fb9d014d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9cffff640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00cf830 0x7fb9d00b68d0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000c7f0 tx=0x7fb9c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9ccff9640 1 -- 172.21.15.77:0/4198518069 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0015070 con 0x7fb9d00cf830 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9ccff9640 1 -- 172.21.15.77:0/4198518069 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c0004220 con 0x7fb9d00cf830 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9ccff9640 1 -- 172.21.15.77:0/4198518069 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0007620 con 0x7fb9d00cf830 2024-03-06T07:27:56.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d014dd30 con 0x7fb9d00cf830 2024-03-06T07:27:56.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.820+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d014e2f0 con 0x7fb9d00cf830 2024-03-06T07:27:56.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.821+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d004f330 con 0x7fb9d00cf830 2024-03-06T07:27:56.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.822+0000 7fb9ccff9640 1 -- 172.21.15.77:0/4198518069 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9c00079d0 con 0x7fb9d00cf830 2024-03-06T07:27:56.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.822+0000 7fb9ccff9640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.822+0000 7fb9cf7fe640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.822+0000 7fb9ccff9640 1 -- 172.21.15.77:0/4198518069 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb9c008a620 con 0x7fb9d00cf830 2024-03-06T07:27:56.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.823+0000 7fb9cf7fe640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00cd440 tx=0x7fb9c4007c10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:56.826+0000 7fb9ccff9640 1 -- 172.21.15.77:0/4198518069 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9c0059440 con 0x7fb9d00cf830 2024-03-06T07:27:57.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.042+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9d006c290 con 0x7fb9d00cf830 2024-03-06T07:27:57.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.043+0000 7fb9ccff9640 1 -- 172.21.15.77:0/4198518069 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fb9c0058d40 con 0x7fb9d00cf830 2024-03-06T07:27:57.044 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:57.045 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:57.045 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:57.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.044+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9bc06b4d0 msgr2=0x7fb9bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.044+0000 7fb9d654b640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00cd440 tx=0x7fb9c4007c10 comp rx=0 tx=0).stop 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.044+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00cf830 msgr2=0x7fb9d00b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.044+0000 7fb9d654b640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00cf830 0x7fb9d00b68d0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000c7f0 tx=0x7fb9c000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.044+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 shutdown_connections 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.044+0000 7fb9d654b640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9d015dfe0 0x7fb9d00cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.045+0000 7fb9d654b640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9d00d29c0 0x7fb9d014d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.045+0000 7fb9d654b640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9bc06b4d0 0x7fb9bc06d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.045+0000 7fb9d654b640 1 --2- 172.21.15.77:0/4198518069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9d00cf830 0x7fb9d00b68d0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.045+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 >> 172.21.15.77:0/4198518069 conn(0x7fb9d00d3cd0 msgr2=0x7fb9d00b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.045+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 shutdown_connections 2024-03-06T07:27:57.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.045+0000 7fb9d654b640 1 -- 172.21.15.77:0/4198518069 wait complete. 2024-03-06T07:27:57.058 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:27:57.174 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:27:57.174 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:27:57.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.274+0000 7f7681b93640 1 Processor -- start 2024-03-06T07:27:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.274+0000 7f7681b93640 1 -- start start 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f7681b93640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f767c0d2710 0x7f767c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f7681b93640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c0b3430 con 0x7f767c0d22b0 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f7681b93640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c0b3710 con 0x7f767c15dfe0 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f7681b93640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767c0b31e0 con 0x7f767c0d2710 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f7673fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f767c0d22b0 0x7f767c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38410/0 (socket says 172.21.15.77:38410) 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f7673fff640 1 -- 172.21.15.77:0/772398712 learned_addr learned my addr 172.21.15.77:0/772398712 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f767bfff640 1 --2- 172.21.15.77:0/772398712 >> v2:172.21.15.158:3300/0 conn(0x7f767c0d2710 0x7f767c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041175426 0 0) 0x7f767c0b3430 con 0x7f767c0d22b0 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.275+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7668003520 con 0x7f767c0d22b0 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1268419614 0 0) 0x7f7668003520 con 0x7f767c0d22b0 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 >> v2:172.21.15.158:3300/0 conn(0x7f767c0d2710 msgr2=0x7f767c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 --2- 172.21.15.77:0/772398712 >> v2:172.21.15.158:3300/0 conn(0x7f767c0d2710 0x7f767c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 >> v1:172.21.15.158:6789/0 conn(0x7f767c15dfe0 legacy=0x7f767c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c0b39f0 con 0x7f767c0d22b0 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f7664001670 con 0x7f767c0d22b0 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 >> v1:172.21.15.77:6789/0 conn(0x7f767c0d22b0 legacy=0x7f767c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f76680039e0 0x7f7668003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7668004800 0x7f7668004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680051c0 0x7f766800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0b31e0 con 0x7f7668004800 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766800c010 con 0x7f76680051c0 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766800c210 con 0x7f76680039e0 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f7673fff640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7668004800 0x7f7668004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767bfff640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680051c0 0x7f766800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767b7fe640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f76680039e0 0x7f7668003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f7673fff640 1 -- 172.21.15.77:0/772398712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f76680039e0 msgr2=0x7f7668003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f7673fff640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f76680039e0 0x7f7668003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f7673fff640 1 -- 172.21.15.77:0/772398712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680051c0 msgr2=0x7f766800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f7673fff640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f76680051c0 0x7f766800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f767bfff640 1 -- 172.21.15.77:0/772398712 reap_dead start 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.276+0000 7f7673fff640 1 -- 172.21.15.77:0/772398712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7668003520 con 0x7f7668004800 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.277+0000 7f7673fff640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7668004800 0x7f7668004c80 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f766400c210 tx=0x7f7668005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c656f6d1d69906 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.277+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664007ce0 con 0x7f7668004800 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.277+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7664030070 con 0x7f7668004800 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.277+0000 7f767affd640 1 -- 172.21.15.77:0/772398712 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664005d70 con 0x7f7668004800 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.278+0000 7f7681b93640 1 -- 172.21.15.77:0/772398712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7668004800 msgr2=0x7f7668004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.278+0000 7f7681b93640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7668004800 0x7f7668004c80 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f766400c210 tx=0x7f7668005830 comp rx=0 tx=0).stop 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.278+0000 7f7681b93640 1 -- 172.21.15.77:0/772398712 shutdown_connections 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.278+0000 7f7681b93640 1 --2- 172.21.15.77:0/772398712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7668004800 0x7f7668004c80 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.278+0000 7f7681b93640 1 -- 172.21.15.77:0/772398712 >> 172.21.15.77:0/772398712 conn(0x7f767c0d3cd0 msgr2=0x7f767c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.278+0000 7f7681b93640 1 -- 172.21.15.77:0/772398712 shutdown_connections 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.278+0000 7f7681b93640 1 -- 172.21.15.77:0/772398712 wait complete. 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.279+0000 7f7681b93640 1 Processor -- start 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.279+0000 7f7681b93640 1 -- start start 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.279+0000 7f7681b93640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7668004800 0x7f767c0c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.279+0000 7f7681b93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f767c15f020 0x7f767c15f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.279+0000 7f7681b93640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f767c153aa0 0x7f767c153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.279+0000 7f7681b93640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0d1c70 con 0x7f767c15f020 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.279+0000 7f7681b93640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0b3d60 con 0x7f7668004800 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.279+0000 7f7681b93640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0d1df0 con 0x7f767c153aa0 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f767c15f020 0x7f767c15f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f767c15f020 0x7f767c15f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37038/0 (socket says 172.21.15.77:37038) 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 -- 172.21.15.77:0/713244043 learned_addr learned my addr 172.21.15.77:0/713244043 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:57.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f767b7fe640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7668004800 0x7f767c0c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f767bfff640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f767c153aa0 0x7f767c153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 -- 172.21.15.77:0/713244043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f767c153aa0 msgr2=0x7f767c153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f767c153aa0 0x7f767c153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 -- 172.21.15.77:0/713244043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7668004800 msgr2=0x7f767c0c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7668004800 0x7f767c0c9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 -- 172.21.15.77:0/713244043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c15fb90 con 0x7f767c15f020 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7673fff640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f767c15f020 0x7f767c15f4a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f766400c330 tx=0x7f766402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7678ff9640 1 -- 172.21.15.77:0/713244043 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766402c9e0 con 0x7f767c15f020 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7678ff9640 1 -- 172.21.15.77:0/713244043 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766402cb80 con 0x7f767c15f020 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c15fe70 con 0x7f767c15f020 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7678ff9640 1 -- 172.21.15.77:0/713244043 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664007070 con 0x7f767c15f020 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.280+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c160430 con 0x7f767c15f020 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.281+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c04f330 con 0x7f767c15f020 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.282+0000 7f7678ff9640 1 -- 172.21.15.77:0/713244043 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7664007210 con 0x7f767c15f020 2024-03-06T07:27:57.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.282+0000 7f7678ff9640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f765c06b500 0x7f765c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.282+0000 7f7678ff9640 1 -- 172.21.15.77:0/713244043 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f76640aba70 con 0x7f767c15f020 2024-03-06T07:27:57.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.282+0000 7f767b7fe640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f765c06b500 0x7f765c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.283+0000 7f767b7fe640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f765c06b500 0x7f765c06d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f766c0055a0 tx=0x7f766c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.287+0000 7f7678ff9640 1 -- 172.21.15.77:0/713244043 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f766407a2d0 con 0x7f767c15f020 2024-03-06T07:27:57.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.502+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f767c04f330 con 0x7f767c15f020 2024-03-06T07:27:57.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.503+0000 7f7678ff9640 1 -- 172.21.15.77:0/713244043 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f766407e370 con 0x7f767c15f020 2024-03-06T07:27:57.505 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:57.505 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:57.505 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:57.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f765c06b500 msgr2=0x7f765c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f765c06b500 0x7f765c06d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f766c0055a0 tx=0x7f766c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:57.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f767c15f020 msgr2=0x7f767c15f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f767c15f020 0x7f767c15f4a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f766400c330 tx=0x7f766402e070 comp rx=0 tx=0).stop 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 shutdown_connections 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f765c06b500 0x7f765c06d9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f767c15f020 0x7f767c15f4a0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f767c153aa0 0x7f767c153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 --2- 172.21.15.77:0/713244043 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7668004800 0x7f767c0c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 >> 172.21.15.77:0/713244043 conn(0x7f767c0d3cd0 msgr2=0x7f767c0cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 shutdown_connections 2024-03-06T07:27:57.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.505+0000 7f7681b93640 1 -- 172.21.15.77:0/713244043 wait complete. 2024-03-06T07:27:57.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:27:57.643 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:27:57.643 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:57.644 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:27:57.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.683+0000 7ff9b783cf40 1 Processor -- start 2024-03-06T07:27:57.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.683+0000 7ff9b783cf40 1 -- start start 2024-03-06T07:27:57.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9b783cf40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5619ffc09a80 0x5619ffd1fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9b783cf40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc7f810 0x5619ffbee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9b783cf40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5619ffc978d0 0x5619ffd27f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9b783cf40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619ffc97e50 con 0x5619ffc7f810 2024-03-06T07:27:57.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9b783cf40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619ffb67b70 con 0x5619ffc09a80 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9b783cf40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619ffc980f0 con 0x5619ffc978d0 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc7f810 0x5619ffbee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc7f810 0x5619ffbee7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37058/0 (socket says 172.21.15.77:37058) 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 -- 172.21.15.77:0/1925059961 learned_addr learned my addr 172.21.15.77:0/1925059961 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 -- 172.21.15.77:0/1925059961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5619ffc978d0 msgr2=0x5619ffd27f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 --2- 172.21.15.77:0/1925059961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5619ffc978d0 0x5619ffd27f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 -- 172.21.15.77:0/1925059961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5619ffc09a80 msgr2=0x5619ffd1fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 --2- 172.21.15.77:0/1925059961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5619ffc09a80 0x5619ffd1fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 -- 172.21.15.77:0/1925059961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619ffd22620 con 0x5619ffc7f810 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9affff640 1 --2- 172.21.15.77:0/1925059961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc7f810 0x5619ffbee7f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000a440 tx=0x7ff9a000a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43363b942e63ab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9aeffd640 1 -- 172.21.15.77:0/1925059961 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a000b930 con 0x5619ffc7f810 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9aeffd640 1 -- 172.21.15.77:0/1925059961 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a000f070 con 0x5619ffc7f810 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.685+0000 7ff9aeffd640 1 -- 172.21.15.77:0/1925059961 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a000bcf0 con 0x5619ffc7f810 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 -- 172.21.15.77:0/1925059961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc7f810 msgr2=0x5619ffbee7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 --2- 172.21.15.77:0/1925059961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc7f810 0x5619ffbee7f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000a440 tx=0x7ff9a000a900 comp rx=0 tx=0).stop 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 -- 172.21.15.77:0/1925059961 shutdown_connections 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 --2- 172.21.15.77:0/1925059961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc7f810 0x5619ffbee7f0 secure :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000a440 tx=0x7ff9a000a900 comp rx=0 tx=0).stop 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 --2- 172.21.15.77:0/1925059961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5619ffc978d0 0x5619ffd27f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 --2- 172.21.15.77:0/1925059961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5619ffc09a80 0x5619ffd1fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 -- 172.21.15.77:0/1925059961 >> 172.21.15.77:0/1925059961 conn(0x5619ffbec350 msgr2=0x5619ffbc2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 -- 172.21.15.77:0/1925059961 shutdown_connections 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 -- 172.21.15.77:0/1925059961 wait complete. 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 Processor -- start 2024-03-06T07:27:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.686+0000 7ff9b783cf40 1 -- start start 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b783cf40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc09a80 0x5619ffc76970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b783cf40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5619ffc978d0 0x5619ffc785a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b783cf40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5619ffd1dc00 0x5619ffd1e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b783cf40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619ffd20050 con 0x5619ffc09a80 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b783cf40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619ffd204a0 con 0x5619ffd1dc00 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b783cf40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619ffd22240 con 0x5619ffc978d0 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b4a8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc09a80 0x5619ffc76970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b4a8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc09a80 0x5619ffc76970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37066/0 (socket says 172.21.15.77:37066) 2024-03-06T07:27:57.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b4a8e640 1 -- 172.21.15.77:0/3312560459 learned_addr learned my addr 172.21.15.77:0/3312560459 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b528f640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5619ffd1dc00 0x5619ffd1e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9affff640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5619ffc978d0 0x5619ffc785a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b4a8e640 1 -- 172.21.15.77:0/3312560459 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5619ffc978d0 msgr2=0x5619ffc785a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b4a8e640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5619ffc978d0 0x5619ffc785a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b4a8e640 1 -- 172.21.15.77:0/3312560459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5619ffd1dc00 msgr2=0x5619ffd1e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b4a8e640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5619ffd1dc00 0x5619ffd1e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.687+0000 7ff9b4a8e640 1 -- 172.21.15.77:0/3312560459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619ffd1e5f0 con 0x5619ffc09a80 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.688+0000 7ff9b4a8e640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc09a80 0x5619ffc76970 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800bde0 tx=0x7ff9a8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.688+0000 7ff9adffb640 1 -- 172.21.15.77:0/3312560459 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8008070 con 0x5619ffc09a80 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.688+0000 7ff9adffb640 1 -- 172.21.15.77:0/3312560459 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a800f650 con 0x5619ffc09a80 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.688+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5619ffc7c160 con 0x5619ffc09a80 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.688+0000 7ff9adffb640 1 -- 172.21.15.77:0/3312560459 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a800fa40 con 0x5619ffc09a80 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.688+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619ffd1eb90 con 0x5619ffc09a80 2024-03-06T07:27:57.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.689+0000 7ff9adffb640 1 -- 172.21.15.77:0/3312560459 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9a800fbe0 con 0x5619ffc09a80 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.689+0000 7ff9adffb640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff998069eb0 0x7ff99806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.689+0000 7ff9affff640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff998069eb0 0x7ff99806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.689+0000 7ff9adffb640 1 -- 172.21.15.77:0/3312560459 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff9a8089140 con 0x5619ffc09a80 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b783cf40 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5619ffd20be0 0x5619ffd20fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) v9 -- 0x5619ffc78ae0 con 0x5619ffd20be0 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b528f640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5619ffd20be0 0x5619ffd20fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9affff640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff998069eb0 0x7ff99806c370 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x5619ffc964b0 tx=0x7ff9a0002000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b528f640 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5619ffd20be0 0x5619ffd20fc0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b528f640 1 -- 172.21.15.77:0/3312560459 <== osd.0 v2:172.21.15.77:6808/3119591583 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7ff9b004dac0 con 0x5619ffd20be0 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5619ffd20be0 msgr2=0x5619ffd20fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b783cf40 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5619ffd20be0 0x5619ffd20fc0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff998069eb0 msgr2=0x7ff99806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b783cf40 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff998069eb0 0x7ff99806c370 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x5619ffc964b0 tx=0x7ff9a0002000 comp rx=0 tx=0).stop 2024-03-06T07:27:57.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc09a80 msgr2=0x5619ffc76970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.690+0000 7ff9b783cf40 1 --2- 172.21.15.77:0/3312560459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5619ffc09a80 0x5619ffc76970 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800bde0 tx=0x7ff9a8009560 comp rx=0 tx=0).stop 2024-03-06T07:27:57.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.691+0000 7ff9b528f640 1 -- 172.21.15.77:0/3312560459 reap_dead start 2024-03-06T07:27:57.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.691+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 shutdown_connections 2024-03-06T07:27:57.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.691+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 >> 172.21.15.77:0/3312560459 conn(0x5619ffbec350 msgr2=0x5619ffc7b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.691+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 shutdown_connections 2024-03-06T07:27:57.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.691+0000 7ff9b783cf40 1 -- 172.21.15.77:0/3312560459 wait complete. 2024-03-06T07:27:57.698 INFO:teuthology.orchestra.run.smithi077.stdout:] 10000000001 stray0 2024-03-06T07:27:57.698 DEBUG:teuthology.orchestra.run.smithi077:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 1099511627777, 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: "ancestors": [ 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: "dirino": 1536, 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: "dname": "10000000001", 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: "version": 11 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: "dirino": 256, 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: "dname": "stray0", 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi077.stdout: "version": 5 2024-03-06T07:27:57.784 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:27:57.784 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-06T07:27:57.784 INFO:teuthology.orchestra.run.smithi077.stdout: "pool": 17, 2024-03-06T07:27:57.784 INFO:teuthology.orchestra.run.smithi077.stdout: "old_pools": [] 2024-03-06T07:27:57.784 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:27:57.784 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:57.785 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:57.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.929+0000 7fe8a40ec640 1 Processor -- start 2024-03-06T07:27:57.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.929+0000 7fe8a40ec640 1 -- start start 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.930+0000 7fe8a40ec640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe89c15abf0 0x7fe89c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.930+0000 7fe8a40ec640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c066e30 con 0x7fe89c15afd0 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.930+0000 7fe8a40ec640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c06bba0 con 0x7fe89c0cfd20 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.930+0000 7fe8a40ec640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c06be80 con 0x7fe89c15abf0 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.930+0000 7fe8a2689640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe89c15afd0 0x7fe89c06c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38420/0 (socket says 172.21.15.77:38420) 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.930+0000 7fe8a2689640 1 -- 172.21.15.77:0/1556853737 learned_addr learned my addr 172.21.15.77:0/1556853737 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.930+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377684233 0 0) 0x7fe89c066e30 con 0x7fe89c15afd0 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.930+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/1556853737 >> v2:172.21.15.158:3300/0 conn(0x7fe89c15abf0 0x7fe89c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe888005730 con 0x7fe89c15afd0 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219759093 0 0) 0x7fe888005730 con 0x7fe89c15afd0 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 >> v2:172.21.15.158:3300/0 conn(0x7fe89c15abf0 msgr2=0x7fe89c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 --2- 172.21.15.77:0/1556853737 >> v2:172.21.15.158:3300/0 conn(0x7fe89c15abf0 0x7fe89c06d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 >> v1:172.21.15.158:6789/0 conn(0x7fe89c0cfd20 legacy=0x7fe89c0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c06eae0 con 0x7fe89c15afd0 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fe8940019f0 con 0x7fe89c15afd0 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/1556853737 >> v2:172.21.15.158:3300/0 conn(0x7fe89c15abf0 0x7fe89c06d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 >> v1:172.21.15.77:6789/0 conn(0x7fe89c15afd0 legacy=0x7fe89c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe888005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe888006a10 0x7fe888006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8880075d0 0x7fe88800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c06be80 con 0x7fe888005bc0 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88800e460 con 0x7fe8880075d0 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88800e660 con 0x7fe888006a10 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe888005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1687640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe888006a10 0x7fe888006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a2689640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8880075d0 0x7fe88800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1e88640 1 -- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe888006a10 msgr2=0x7fe888006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe888006a10 0x7fe888006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1e88640 1 -- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8880075d0 msgr2=0x7fe88800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8880075d0 0x7fe88800dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1e88640 1 -- 172.21.15.77:0/1556853737 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe888005730 con 0x7fe888005bc0 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a2689640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe8880075d0 0x7fe88800dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a1687640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe888006a10 0x7fe888006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.931+0000 7fe8a2689640 1 -- 172.21.15.77:0/1556853737 reap_dead start 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.932+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe888005fe0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fe8980098a0 tx=0x7fe888007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100ac01d236ab17d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.932+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89800f040 con 0x7fe888005bc0 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.932+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898004230 con 0x7fe888005bc0 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.932+0000 7fe8a0e86640 1 -- 172.21.15.77:0/1556853737 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898003610 con 0x7fe888005bc0 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.933+0000 7fe8a40ec640 1 -- 172.21.15.77:0/1556853737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 msgr2=0x7fe888005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.933+0000 7fe8a40ec640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe888005fe0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fe8980098a0 tx=0x7fe888007a40 comp rx=0 tx=0).stop 2024-03-06T07:27:57.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.933+0000 7fe8a40ec640 1 -- 172.21.15.77:0/1556853737 shutdown_connections 2024-03-06T07:27:57.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.933+0000 7fe8a40ec640 1 --2- 172.21.15.77:0/1556853737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe888005fe0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.933+0000 7fe8a40ec640 1 -- 172.21.15.77:0/1556853737 >> 172.21.15.77:0/1556853737 conn(0x7fe89c0b1d00 msgr2=0x7fe89c0b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.933+0000 7fe8a40ec640 1 -- 172.21.15.77:0/1556853737 shutdown_connections 2024-03-06T07:27:57.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.933+0000 7fe8a40ec640 1 -- 172.21.15.77:0/1556853737 wait complete. 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.934+0000 7fe8a40ec640 1 Processor -- start 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.934+0000 7fe8a40ec640 1 -- start start 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a40ec640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe89c0ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a40ec640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c0aeb80 0x7fe89c1f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a40ec640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c1f2f00 0x7fe89c15b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a40ec640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c06f180 con 0x7fe888005bc0 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a40ec640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c06f000 con 0x7fe89c1f2f00 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a40ec640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c06d160 con 0x7fe89c0aeb80 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe89c0ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe89c0ce0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37074/0 (socket says 172.21.15.77:37074) 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 -- 172.21.15.77:0/525635244 learned_addr learned my addr 172.21.15.77:0/525635244 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a2689640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c1f2f00 0x7fe89c15b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1687640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c0aeb80 0x7fe89c1f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 -- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c0aeb80 msgr2=0x7fe89c1f29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c0aeb80 0x7fe89c1f29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 -- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c1f2f00 msgr2=0x7fe89c15b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c1f2f00 0x7fe89c15b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 -- 172.21.15.77:0/525635244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0ce910 con 0x7fe888005bc0 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a2689640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c1f2f00 0x7fe89c15b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1687640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c0aeb80 0x7fe89c1f29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a1e88640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe89c0ce0c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe89802bf90 tx=0x7fe898004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8927fc640 1 -- 172.21.15.77:0/525635244 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898048070 con 0x7fe888005bc0 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8927fc640 1 -- 172.21.15.77:0/525635244 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898030430 con 0x7fe888005bc0 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8927fc640 1 -- 172.21.15.77:0/525635244 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980307e0 con 0x7fe888005bc0 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.935+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c0cebf0 con 0x7fe888005bc0 2024-03-06T07:27:57.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.936+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c0cf1b0 con 0x7fe888005bc0 2024-03-06T07:27:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.936+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c04f330 con 0x7fe888005bc0 2024-03-06T07:27:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.937+0000 7fe8927fc640 1 -- 172.21.15.77:0/525635244 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe89804b070 con 0x7fe888005bc0 2024-03-06T07:27:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.938+0000 7fe8927fc640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88006b500 0x7fe88006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.938+0000 7fe8a1687640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88006b500 0x7fe88006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.938+0000 7fe8927fc640 1 -- 172.21.15.77:0/525635244 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe8980abb50 con 0x7fe888005bc0 2024-03-06T07:27:57.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.938+0000 7fe8a1687640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88006b500 0x7fe88006d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000c00 tx=0x7fe88c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:57.946+0000 7fe8927fc640 1 -- 172.21.15.77:0/525635244 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe898034020 con 0x7fe888005bc0 2024-03-06T07:27:58.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.218+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe89c0b6000 con 0x7fe888005bc0 2024-03-06T07:27:58.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.218+0000 7fe8927fc640 1 -- 172.21.15.77:0/525635244 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fe89807a2a0 con 0x7fe888005bc0 2024-03-06T07:27:58.220 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:58.220 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:58.220 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:58.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88006b500 msgr2=0x7fe88006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88006b500 0x7fe88006d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000c00 tx=0x7fe88c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 msgr2=0x7fe89c0ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe89c0ce0c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe89802bf90 tx=0x7fe898004730 comp rx=0 tx=0).stop 2024-03-06T07:27:58.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 shutdown_connections 2024-03-06T07:27:58.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe89c1f2f00 0x7fe89c15b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe89c0aeb80 0x7fe89c1f29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe88006b500 0x7fe88006d9c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 --2- 172.21.15.77:0/525635244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe888005bc0 0x7fe89c0ce0c0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 >> 172.21.15.77:0/525635244 conn(0x7fe89c0b1d00 msgr2=0x7fe89c0cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 shutdown_connections 2024-03-06T07:27:58.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.220+0000 7fe8a40ec640 1 -- 172.21.15.77:0/525635244 wait complete. 2024-03-06T07:27:58.233 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:27:58.358 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:27:58.358 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:27:58.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:58.359 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:27:58.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:58.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.523+0000 7f53c3f79640 1 Processor -- start 2024-03-06T07:27:58.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.523+0000 7f53c3f79640 1 -- start start 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.524+0000 7f53c3f79640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53bc15dfe0 0x7f53bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.524+0000 7f53c3f79640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0d29c0 0x7f53bc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.524+0000 7f53c3f79640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0b3430 con 0x7f53bc0cf830 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.524+0000 7f53c3f79640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0b31e0 con 0x7f53bc15dfe0 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.524+0000 7f53c3f79640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0b3d60 con 0x7f53bc0d29c0 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c2516640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0d29c0 0x7f53bc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c1d15640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53bc15dfe0 0x7f53bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c1514640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f53bc0cf830 0x7f53bc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:59392/0 (socket says 172.21.15.77:59392) 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c1514640 1 -- 172.21.15.77:0/2676946657 learned_addr learned my addr 172.21.15.77:0/2676946657 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c0d13640 1 -- 172.21.15.77:0/2676946657 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226714190 0 0) 0x7f53bc0b3430 con 0x7f53bc0cf830 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c0d13640 1 -- 172.21.15.77:0/2676946657 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a8003520 con 0x7f53bc0cf830 2024-03-06T07:27:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c2516640 1 -- 172.21.15.77:0/2676946657 >> v2:172.21.15.158:3300/0 conn(0x7f53bc15dfe0 msgr2=0x7f53bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c2516640 1 --2- 172.21.15.77:0/2676946657 >> v2:172.21.15.158:3300/0 conn(0x7f53bc15dfe0 0x7f53bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c2516640 1 -- 172.21.15.77:0/2676946657 >> v1:172.21.15.158:6789/0 conn(0x7f53bc0cf830 legacy=0x7f53bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c2516640 1 -- 172.21.15.77:0/2676946657 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0b3710 con 0x7f53bc0d29c0 2024-03-06T07:27:58.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.525+0000 7f53c1d15640 1 --2- 172.21.15.77:0/2676946657 >> v2:172.21.15.158:3300/0 conn(0x7f53bc15dfe0 0x7f53bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.526+0000 7f53c2516640 1 --2- 172.21.15.77:0/2676946657 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0d29c0 0x7f53bc0b41f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009d90 tx=0x7f53b402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96b1b3b97e58463d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.526+0000 7f53c0d13640 1 -- 172.21.15.77:0/2676946657 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4039070 con 0x7f53bc0d29c0 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.526+0000 7f53c0d13640 1 -- 172.21.15.77:0/2676946657 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b402cc50 con 0x7f53bc0d29c0 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.526+0000 7f53c0d13640 1 -- 172.21.15.77:0/2676946657 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4034440 con 0x7f53bc0d29c0 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.526+0000 7f53c3f79640 1 -- 172.21.15.77:0/2676946657 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0d29c0 msgr2=0x7f53bc0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.526+0000 7f53c3f79640 1 --2- 172.21.15.77:0/2676946657 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0d29c0 0x7f53bc0b41f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009d90 tx=0x7f53b402bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.527+0000 7f53c3f79640 1 -- 172.21.15.77:0/2676946657 shutdown_connections 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.527+0000 7f53c3f79640 1 --2- 172.21.15.77:0/2676946657 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0d29c0 0x7f53bc0b41f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.527+0000 7f53c3f79640 1 --2- 172.21.15.77:0/2676946657 >> v2:172.21.15.158:3300/0 conn(0x7f53bc15dfe0 0x7f53bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.527+0000 7f53c3f79640 1 -- 172.21.15.77:0/2676946657 >> 172.21.15.77:0/2676946657 conn(0x7f53bc0d3cd0 msgr2=0x7f53bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.527+0000 7f53c3f79640 1 -- 172.21.15.77:0/2676946657 shutdown_connections 2024-03-06T07:27:58.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.527+0000 7f53c3f79640 1 -- 172.21.15.77:0/2676946657 wait complete. 2024-03-06T07:27:58.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.528+0000 7f53c3f79640 1 Processor -- start 2024-03-06T07:27:58.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.528+0000 7f53c3f79640 1 -- start start 2024-03-06T07:27:58.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.528+0000 7f53c3f79640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0cf830 0x7f53bc0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.528+0000 7f53c3f79640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53bc0d29c0 0x7f53bc14d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.528+0000 7f53c3f79640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53bc15dfe0 0x7f53bc1f7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.528+0000 7f53c3f79640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0d1df0 con 0x7f53bc0d29c0 2024-03-06T07:27:58.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.528+0000 7f53c3f79640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0d1c70 con 0x7f53bc15dfe0 2024-03-06T07:27:58.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.528+0000 7f53c3f79640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc15e3c0 con 0x7f53bc0cf830 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53bc0d29c0 0x7f53bc14d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53bc0d29c0 0x7f53bc14d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43286/0 (socket says 172.21.15.77:43286) 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 -- 172.21.15.77:0/1666241833 learned_addr learned my addr 172.21.15.77:0/1666241833 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1d15640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0cf830 0x7f53bc0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c2516640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53bc15dfe0 0x7f53bc1f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 -- 172.21.15.77:0/1666241833 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0cf830 msgr2=0x7f53bc0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0cf830 0x7f53bc0ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 -- 172.21.15.77:0/1666241833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53bc15dfe0 msgr2=0x7f53bc1f7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53bc15dfe0 0x7f53bc1f7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 -- 172.21.15.77:0/1666241833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc1f79a0 con 0x7f53bc0d29c0 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c1514640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53bc0d29c0 0x7f53bc14d790 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00c400 tx=0x7f53ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53b27fc640 1 -- 172.21.15.77:0/1666241833 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac017070 con 0x7f53bc0d29c0 2024-03-06T07:27:58.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc1f7c80 con 0x7f53bc0d29c0 2024-03-06T07:27:58.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc14d270 con 0x7f53bc0d29c0 2024-03-06T07:27:58.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53b27fc640 1 -- 172.21.15.77:0/1666241833 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac005ce0 con 0x7f53bc0d29c0 2024-03-06T07:27:58.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.529+0000 7f53b27fc640 1 -- 172.21.15.77:0/1666241833 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac021060 con 0x7f53bc0d29c0 2024-03-06T07:27:58.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.530+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a4005290 con 0x7f53bc0d29c0 2024-03-06T07:27:58.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.531+0000 7f53b27fc640 1 -- 172.21.15.77:0/1666241833 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53ac01a070 con 0x7f53bc0d29c0 2024-03-06T07:27:58.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.531+0000 7f53b27fc640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.531+0000 7f53b27fc640 1 -- 172.21.15.77:0/1666241833 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f53ac089620 con 0x7f53bc0d29c0 2024-03-06T07:27:58.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.531+0000 7f53c1d15640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.532+0000 7f53c1d15640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53a006b4d0 0x7f53a006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f53bc06a360 tx=0x7f53b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.539+0000 7f53b27fc640 1 -- 172.21.15.77:0/1666241833 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53ac057ce0 con 0x7f53bc0d29c0 2024-03-06T07:27:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.755+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53a4005850 con 0x7f53bc0d29c0 2024-03-06T07:27:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.756+0000 7f53b27fc640 1 -- 172.21.15.77:0/1666241833 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f53ac05bd80 con 0x7f53bc0d29c0 2024-03-06T07:27:58.757 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:58.758 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:58.758 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:58.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.757+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53a006b4d0 msgr2=0x7f53a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.757+0000 7f53c3f79640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53a006b4d0 0x7f53a006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f53bc06a360 tx=0x7f53b8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53bc0d29c0 msgr2=0x7f53bc14d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53bc0d29c0 0x7f53bc14d790 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00c400 tx=0x7f53ac00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 shutdown_connections 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53bc0d29c0 0x7f53bc14d790 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53bc15dfe0 0x7f53bc1f7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 --2- 172.21.15.77:0/1666241833 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53bc0cf830 0x7f53bc0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 >> 172.21.15.77:0/1666241833 conn(0x7f53bc0d3cd0 msgr2=0x7f53bc0ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 shutdown_connections 2024-03-06T07:27:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.758+0000 7f53c3f79640 1 -- 172.21.15.77:0/1666241833 wait complete. 2024-03-06T07:27:58.770 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:27:58.887 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:27:58.887 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:27:58.888 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.987+0000 7f374ab47640 1 Processor -- start 2024-03-06T07:27:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.987+0000 7f374ab47640 1 -- start start 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f374ab47640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f374415dfe0 0x7f37440cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f374ab47640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37440d29c0 0x7f37440b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f374ab47640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37440b3430 con 0x7f37440cf830 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f374ab47640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37440b31e0 con 0x7f37440d29c0 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f374ab47640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37440b3d60 con 0x7f374415dfe0 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f37490e4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37440d29c0 0x7f37440b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f37490e4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37440d29c0 0x7f37440b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43294/0 (socket says 172.21.15.77:43294) 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f37490e4640 1 -- 172.21.15.77:0/621513246 learned_addr learned my addr 172.21.15.77:0/621513246 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f37490e4640 1 -- 172.21.15.77:0/621513246 >> v2:172.21.15.158:3300/0 conn(0x7f374415dfe0 msgr2=0x7f37440cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f37490e4640 1 --2- 172.21.15.77:0/621513246 >> v2:172.21.15.158:3300/0 conn(0x7f374415dfe0 0x7f37440cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f37490e4640 1 -- 172.21.15.77:0/621513246 >> v1:172.21.15.158:6789/0 conn(0x7f37440cf830 legacy=0x7f37440d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.988+0000 7f37490e4640 1 -- 172.21.15.77:0/621513246 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37440b3710 con 0x7f37440d29c0 2024-03-06T07:27:58.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.989+0000 7f37490e4640 1 --2- 172.21.15.77:0/621513246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440d29c0 0x7f37440b41f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3744067990 tx=0x7f373c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75efe15fd7e73d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.989+0000 7f37437fe640 1 -- 172.21.15.77:0/621513246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c02ca80 con 0x7f37440d29c0 2024-03-06T07:27:58.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.989+0000 7f37437fe640 1 -- 172.21.15.77:0/621513246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c02cc20 con 0x7f37440d29c0 2024-03-06T07:27:58.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.989+0000 7f37437fe640 1 -- 172.21.15.77:0/621513246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c0343f0 con 0x7f37440d29c0 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.990+0000 7f374ab47640 1 -- 172.21.15.77:0/621513246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440d29c0 msgr2=0x7f37440b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.990+0000 7f374ab47640 1 --2- 172.21.15.77:0/621513246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440d29c0 0x7f37440b41f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3744067990 tx=0x7f373c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.990+0000 7f374ab47640 1 -- 172.21.15.77:0/621513246 shutdown_connections 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.990+0000 7f374ab47640 1 --2- 172.21.15.77:0/621513246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440d29c0 0x7f37440b41f0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.990+0000 7f374ab47640 1 --2- 172.21.15.77:0/621513246 >> v2:172.21.15.158:3300/0 conn(0x7f374415dfe0 0x7f37440cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.990+0000 7f374ab47640 1 -- 172.21.15.77:0/621513246 >> 172.21.15.77:0/621513246 conn(0x7f37440d3cd0 msgr2=0x7f37440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.990+0000 7f374ab47640 1 -- 172.21.15.77:0/621513246 shutdown_connections 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.990+0000 7f374ab47640 1 -- 172.21.15.77:0/621513246 wait complete. 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.991+0000 7f374ab47640 1 Processor -- start 2024-03-06T07:27:58.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.991+0000 7f374ab47640 1 -- start start 2024-03-06T07:27:58.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f374ab47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440cf830 0x7f37440cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f374ab47640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37440d29c0 0x7f374415f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f374ab47640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f374415dfe0 0x7f3744160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f374ab47640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37440d1df0 con 0x7f37440cf830 2024-03-06T07:27:58.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f374ab47640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37440d1c70 con 0x7f37440d29c0 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f374ab47640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374415e3c0 con 0x7f374415dfe0 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440cf830 0x7f37440cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440cf830 0x7f37440cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43310/0 (socket says 172.21.15.77:43310) 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 -- 172.21.15.77:0/2583581083 learned_addr learned my addr 172.21.15.77:0/2583581083 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 -- 172.21.15.77:0/2583581083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f374415dfe0 msgr2=0x7f3744160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f374415dfe0 0x7f3744160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 -- 172.21.15.77:0/2583581083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37440d29c0 msgr2=0x7f374415f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37440d29c0 0x7f374415f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 -- 172.21.15.77:0/2583581083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374415fee0 con 0x7f37440cf830 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37488e3640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440cf830 0x7f37440cd7f0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f373800b410 tx=0x7f373800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37417fa640 1 -- 172.21.15.77:0/2583581083 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3738015020 con 0x7f37440cf830 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37417fa640 1 -- 172.21.15.77:0/2583581083 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3738011850 con 0x7f37440cf830 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37440caed0 con 0x7f37440cf830 2024-03-06T07:27:58.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f37417fa640 1 -- 172.21.15.77:0/2583581083 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3738011c40 con 0x7f37440cf830 2024-03-06T07:27:58.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.992+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374415fc00 con 0x7f37440cf830 2024-03-06T07:27:58.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.993+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374404f330 con 0x7f37440cf830 2024-03-06T07:27:58.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.994+0000 7f37417fa640 1 -- 172.21.15.77:0/2583581083 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3738018020 con 0x7f37440cf830 2024-03-06T07:27:58.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.994+0000 7f37417fa640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f372806b500 0x7f372806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.994+0000 7f37417fa640 1 -- 172.21.15.77:0/2583581083 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f373808a1c0 con 0x7f37440cf830 2024-03-06T07:27:58.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.994+0000 7f3743fff640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f372806b500 0x7f372806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.995+0000 7f3743fff640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f372806b500 0x7f372806d9c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f37441f73a0 tx=0x7f3730007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:58.999+0000 7f37417fa640 1 -- 172.21.15.77:0/2583581083 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f373801f020 con 0x7f37440cf830 2024-03-06T07:27:59.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.214+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f374406a1b0 con 0x7f37440cf830 2024-03-06T07:27:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.215+0000 7f37417fa640 1 -- 172.21.15.77:0/2583581083 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f37380172c0 con 0x7f37440cf830 2024-03-06T07:27:59.217 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:59.217 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:59.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f372806b500 msgr2=0x7f372806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f372806b500 0x7f372806d9c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f37441f73a0 tx=0x7f3730007990 comp rx=0 tx=0).stop 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440cf830 msgr2=0x7f37440cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440cf830 0x7f37440cd7f0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f373800b410 tx=0x7f373800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 shutdown_connections 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f374415dfe0 0x7f3744160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f37440d29c0 0x7f374415f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f372806b500 0x7f372806d9c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 --2- 172.21.15.77:0/2583581083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37440cf830 0x7f37440cd7f0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 >> 172.21.15.77:0/2583581083 conn(0x7f37440d3cd0 msgr2=0x7f37440b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 shutdown_connections 2024-03-06T07:27:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.217+0000 7f374ab47640 1 -- 172.21.15.77:0/2583581083 wait complete. 2024-03-06T07:27:59.230 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:27:59.347 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:27:59.348 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:27:59.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:59.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.447+0000 7f89839b6640 1 Processor -- start 2024-03-06T07:27:59.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f89839b6640 1 -- start start 2024-03-06T07:27:59.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f89839b6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f897c0d22b0 0x7f897c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f89839b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c0b41f0 0x7f897c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f89839b6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c0b3430 con 0x7f897c15dfe0 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f89839b6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f897c0d2c50 con 0x7f897c0d22b0 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f89839b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0b31e0 con 0x7f897c0b41f0 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f8980f51640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f897c0d22b0 0x7f897c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f8980f51640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f897c0d22b0 0x7f897c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43312/0 (socket says 172.21.15.77:43312) 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f8981752640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f897c15dfe0 0x7f897c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46138/0 (socket says 172.21.15.77:46138) 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.448+0000 7f8980f51640 1 -- 172.21.15.77:0/2808672577 learned_addr learned my addr 172.21.15.77:0/2808672577 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.449+0000 7f8981f53640 1 --2- 172.21.15.77:0/2808672577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c0b41f0 0x7f897c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.449+0000 7f8980f51640 1 -- 172.21.15.77:0/2808672577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c0b41f0 msgr2=0x7f897c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.449+0000 7f8980f51640 1 --2- 172.21.15.77:0/2808672577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c0b41f0 0x7f897c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.449+0000 7f8980f51640 1 -- 172.21.15.77:0/2808672577 >> v1:172.21.15.77:6789/0 conn(0x7f897c15dfe0 legacy=0x7f897c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.449+0000 7f8973fff640 1 -- 172.21.15.77:0/2808672577 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815073043 0 0) 0x7f897c0b3430 con 0x7f897c15dfe0 2024-03-06T07:27:59.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.449+0000 7f8980f51640 1 -- 172.21.15.77:0/2808672577 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c0b3710 con 0x7f897c0d22b0 2024-03-06T07:27:59.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.449+0000 7f8981f53640 1 --2- 172.21.15.77:0/2808672577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c0b41f0 0x7f897c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:59.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f8980f51640 1 --2- 172.21.15.77:0/2808672577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0d22b0 0x7f897c0d2710 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f896c009870 tx=0x7f896c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ef638f6f290048 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f8973fff640 1 -- 172.21.15.77:0/2808672577 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c034070 con 0x7f897c0d22b0 2024-03-06T07:27:59.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f8973fff640 1 -- 172.21.15.77:0/2808672577 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896c030840 con 0x7f897c0d22b0 2024-03-06T07:27:59.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f8973fff640 1 -- 172.21.15.77:0/2808672577 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c030c10 con 0x7f897c0d22b0 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f89839b6640 1 -- 172.21.15.77:0/2808672577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0d22b0 msgr2=0x7f897c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f89839b6640 1 --2- 172.21.15.77:0/2808672577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0d22b0 0x7f897c0d2710 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f896c009870 tx=0x7f896c02b980 comp rx=0 tx=0).stop 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f89839b6640 1 -- 172.21.15.77:0/2808672577 shutdown_connections 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f89839b6640 1 --2- 172.21.15.77:0/2808672577 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c0b41f0 0x7f897c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f89839b6640 1 --2- 172.21.15.77:0/2808672577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0d22b0 0x7f897c0d2710 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f89839b6640 1 -- 172.21.15.77:0/2808672577 >> 172.21.15.77:0/2808672577 conn(0x7f897c0d3cd0 msgr2=0x7f897c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f89839b6640 1 -- 172.21.15.77:0/2808672577 shutdown_connections 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.450+0000 7f89839b6640 1 -- 172.21.15.77:0/2808672577 wait complete. 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.451+0000 7f89839b6640 1 Processor -- start 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f89839b6640 1 -- start start 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f89839b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0b41f0 0x7f897c15b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f89839b6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f897c0d22b0 0x7f897c1603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f89839b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c15dfe0 0x7f897c160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f89839b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0cfa00 con 0x7f897c0b41f0 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f89839b6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0b3d60 con 0x7f897c0d22b0 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f89839b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0d1c70 con 0x7f897c15dfe0 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f8981752640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0b41f0 0x7f897c15b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f8981752640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0b41f0 0x7f897c15b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43318/0 (socket says 172.21.15.77:43318) 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.452+0000 7f8981752640 1 -- 172.21.15.77:0/322894440 learned_addr learned my addr 172.21.15.77:0/322894440 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8980f51640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f897c0d22b0 0x7f897c1603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8981f53640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c15dfe0 0x7f897c160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8981752640 1 -- 172.21.15.77:0/322894440 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c15dfe0 msgr2=0x7f897c160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8981752640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c15dfe0 0x7f897c160910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8981752640 1 -- 172.21.15.77:0/322894440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f897c0d22b0 msgr2=0x7f897c1603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8981752640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f897c0d22b0 0x7f897c1603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8981752640 1 -- 172.21.15.77:0/322894440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c1f3a00 con 0x7f897c0b41f0 2024-03-06T07:27:59.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8981752640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0b41f0 0x7f897c15b350 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f897800c430 tx=0x7f897800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8971ffb640 1 -- 172.21.15.77:0/322894440 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89780074e0 con 0x7f897c0b41f0 2024-03-06T07:27:59.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8971ffb640 1 -- 172.21.15.77:0/322894440 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89780047e0 con 0x7f897c0b41f0 2024-03-06T07:27:59.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897c1f3c90 con 0x7f897c0b41f0 2024-03-06T07:27:59.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f8971ffb640 1 -- 172.21.15.77:0/322894440 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978004bd0 con 0x7f897c0b41f0 2024-03-06T07:27:59.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.453+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f897c1f4250 con 0x7f897c0b41f0 2024-03-06T07:27:59.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.454+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897c04f330 con 0x7f897c0b41f0 2024-03-06T07:27:59.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.454+0000 7f8971ffb640 1 -- 172.21.15.77:0/322894440 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8978011070 con 0x7f897c0b41f0 2024-03-06T07:27:59.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.455+0000 7f8971ffb640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f896006b4d0 0x7f896006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.455+0000 7f8971ffb640 1 -- 172.21.15.77:0/322894440 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f8978089ed0 con 0x7f897c0b41f0 2024-03-06T07:27:59.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.455+0000 7f8980f51640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f896006b4d0 0x7f896006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.455+0000 7f8980f51640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f896006b4d0 0x7f896006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f896c036000 tx=0x7f896c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.459+0000 7f8971ffb640 1 -- 172.21.15.77:0/322894440 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8978058cf0 con 0x7f897c0b41f0 2024-03-06T07:27:59.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.759+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f897c06bc50 con 0x7f897c0b41f0 2024-03-06T07:27:59.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.759+0000 7f8971ffb640 1 -- 172.21.15.77:0/322894440 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f89780585f0 con 0x7f897c0b41f0 2024-03-06T07:27:59.761 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:27:59.761 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:27:59.762 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:27:59.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f896006b4d0 msgr2=0x7f896006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f896006b4d0 0x7f896006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f896c036000 tx=0x7f896c03b040 comp rx=0 tx=0).stop 2024-03-06T07:27:59.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0b41f0 msgr2=0x7f897c15b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0b41f0 0x7f897c15b350 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f897800c430 tx=0x7f897800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 shutdown_connections 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f897c15dfe0 0x7f897c160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f897c0d22b0 0x7f897c1603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f896006b4d0 0x7f896006d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 --2- 172.21.15.77:0/322894440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f897c0b41f0 0x7f897c15b350 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 >> 172.21.15.77:0/322894440 conn(0x7f897c0d3cd0 msgr2=0x7f897c0b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 shutdown_connections 2024-03-06T07:27:59.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.761+0000 7f89839b6640 1 -- 172.21.15.77:0/322894440 wait complete. 2024-03-06T07:27:59.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:27:59.896 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:27:59.896 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:59.897 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:27:59.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.936+0000 7f8cc9c51f40 1 Processor -- start 2024-03-06T07:27:59.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.936+0000 7f8cc9c51f40 1 -- start start 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc9c51f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 0x55c50525efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc9c51f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 0x55c505247100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc9c51f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505247640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc9c51f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c505268dc0 con 0x55c505249fb0 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc9c51f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c505138b70 con 0x55c5051db3a0 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc9c51f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c505268f40 con 0x55c5051db780 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc781b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505247640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc781b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505247640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43338/0 (socket says 172.21.15.77:43338) 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc781b640 1 -- 172.21.15.77:0/2097232446 learned_addr learned my addr 172.21.15.77:0/2097232446 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cbffff640 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 0x55c505247100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc701a640 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 0x55c50525efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc781b640 1 -- 172.21.15.77:0/2097232446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 msgr2=0x55c505247100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc781b640 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 0x55c505247100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc781b640 1 -- 172.21.15.77:0/2097232446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 msgr2=0x55c50525efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc781b640 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 0x55c50525efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.937+0000 7f8cc781b640 1 -- 172.21.15.77:0/2097232446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5051bf5f0 con 0x55c505249fb0 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc781b640 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505247640 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000b600 tx=0x7f8cc000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad56d8be0c069fea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc6018640 1 -- 172.21.15.77:0/2097232446 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0014030 con 0x55c505249fb0 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc6018640 1 -- 172.21.15.77:0/2097232446 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0010620 con 0x55c505249fb0 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc6018640 1 -- 172.21.15.77:0/2097232446 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00109f0 con 0x55c505249fb0 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/2097232446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 msgr2=0x55c505247640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc9c51f40 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505247640 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000b600 tx=0x7f8cc000bac0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/2097232446 shutdown_connections 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc9c51f40 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505247640 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc9c51f40 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 0x55c505247100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc9c51f40 1 --2- 172.21.15.77:0/2097232446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 0x55c50525efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.938+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/2097232446 >> 172.21.15.77:0/2097232446 conn(0x55c5051bc790 msgr2=0x55c5051c1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.939+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/2097232446 shutdown_connections 2024-03-06T07:27:59.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.939+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/2097232446 wait complete. 2024-03-06T07:27:59.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.939+0000 7f8cc9c51f40 1 Processor -- start 2024-03-06T07:27:59.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.939+0000 7f8cc9c51f40 1 -- start start 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc9c51f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 0x55c505301170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc9c51f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 0x55c5053016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc9c51f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505241740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc9c51f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5051bf350 con 0x55c505249fb0 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc9c51f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5051c0d00 con 0x55c5051db3a0 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc9c51f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5051c0f60 con 0x55c5051db780 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505241740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505241740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43340/0 (socket says 172.21.15.77:43340) 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 -- 172.21.15.77:0/1741376580 learned_addr learned my addr 172.21.15.77:0/1741376580 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc701a640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 0x55c505301170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cbffff640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 0x55c5053016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 -- 172.21.15.77:0/1741376580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 msgr2=0x55c5053016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55c5051db780 0x55c5053016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 -- 172.21.15.77:0/1741376580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 msgr2=0x55c505301170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55c5051db3a0 0x55c505301170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 -- 172.21.15.77:0/1741376580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c50525e720 con 0x55c505249fb0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.940+0000 7f8cc781b640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505241740 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x55c5051c0bf0 tx=0x7f8cc00033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.941+0000 7f8cc5016640 1 -- 172.21.15.77:0/1741376580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0005940 con 0x55c505249fb0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.941+0000 7f8cc5016640 1 -- 172.21.15.77:0/1741376580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc001abb0 con 0x55c505249fb0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.941+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c5051c0100 con 0x55c505249fb0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.941+0000 7f8cc5016640 1 -- 172.21.15.77:0/1741376580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0023440 con 0x55c505249fb0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.941+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c505241220 con 0x55c505249fb0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.942+0000 7f8cc5016640 1 -- 172.21.15.77:0/1741376580 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cc0017070 con 0x55c505249fb0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.942+0000 7f8cc5016640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ca8069ee0 0x7f8ca806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.943+0000 7f8cc5016640 1 -- 172.21.15.77:0/1741376580 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f8cc008a880 con 0x55c505249fb0 2024-03-06T07:27:59.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.943+0000 7f8cc701a640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ca8069ee0 0x7f8ca806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.943+0000 7f8cc9c51f40 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55c50525ab70 0x55c505241c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.943+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) v9 -- 0x55c505301bf0 con 0x55c50525ab70 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.943+0000 7f8cbffff640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55c50525ab70 0x55c505241c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.943+0000 7f8cc701a640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ca8069ee0 0x7f8ca806c3a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x55c5051bbcb0 tx=0x7f8cb80018b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.943+0000 7f8cbffff640 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55c50525ab70 0x55c505241c80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cbffff640 1 -- 172.21.15.77:0/1741376580 <== osd.0 v2:172.21.15.77:6808/3119591583 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f8cb004eb30 con 0x55c50525ab70 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55c50525ab70 msgr2=0x55c505241c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x55c50525ab70 0x55c505241c80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ca8069ee0 msgr2=0x7f8ca806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ca8069ee0 0x7f8ca806c3a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x55c5051bbcb0 tx=0x7f8cb80018b0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 msgr2=0x55c505241740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 --2- 172.21.15.77:0/1741376580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c505249fb0 0x55c505241740 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x55c5051c0bf0 tx=0x7f8cc00033a0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc781b640 1 -- 172.21.15.77:0/1741376580 reap_dead start 2024-03-06T07:27:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 shutdown_connections 2024-03-06T07:27:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 >> 172.21.15.77:0/1741376580 conn(0x55c5051bc790 msgr2=0x55c5051bf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 shutdown_connections 2024-03-06T07:27:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:27:59.944+0000 7f8cc9c51f40 1 -- 172.21.15.77:0/1741376580 wait complete. 2024-03-06T07:27:59.950 INFO:teuthology.orchestra.run.smithi077.stdout:W�file_cdir_2 2024-03-06T07:27:59.951 DEBUG:teuthology.orchestra.run.smithi077:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:00.036 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:28:00.036 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 1099511627777, 2024-03-06T07:28:00.036 INFO:teuthology.orchestra.run.smithi077.stdout: "ancestors": [ 2024-03-06T07:28:00.036 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:00.036 INFO:teuthology.orchestra.run.smithi077.stdout: "dirino": 1099511628278, 2024-03-06T07:28:00.036 INFO:teuthology.orchestra.run.smithi077.stdout: "dname": "file_c", 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: "version": 15 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: "dirino": 1, 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: "dname": "dir_2", 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: "version": 31 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: "pool": 17, 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: "old_pools": [] 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:28:00.037 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:00.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.039 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:28:00.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.097 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:28:00.155 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:00.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.293+0000 7fba9993a640 1 Processor -- start 2024-03-06T07:28:00.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.293+0000 7fba9993a640 1 -- start start 2024-03-06T07:28:00.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.294+0000 7fba9993a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fba940d04a0 0x7fba940ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.294+0000 7fba9993a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba94161980 con 0x7fba940cfff0 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.294+0000 7fba9993a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba94161c60 con 0x7fba9415bb20 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.294+0000 7fba9993a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba940d2500 con 0x7fba940d04a0 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.294+0000 7fba927fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fba940cfff0 0x7fba940cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46142/0 (socket says 172.21.15.77:46142) 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.294+0000 7fba927fc640 1 -- 172.21.15.77:0/4124030117 learned_addr learned my addr 172.21.15.77:0/4124030117 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.294+0000 7fba937fe640 1 --2- 172.21.15.77:0/4124030117 >> v2:172.21.15.158:3300/0 conn(0x7fba940d04a0 0x7fba940ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.295+0000 7fba91ffb640 1 -- 172.21.15.77:0/4124030117 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072067543 0 0) 0x7fba94161980 con 0x7fba940cfff0 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.295+0000 7fba91ffb640 1 -- 172.21.15.77:0/4124030117 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c003520 con 0x7fba940cfff0 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.295+0000 7fba91ffb640 1 -- 172.21.15.77:0/4124030117 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580529596 0 0) 0x7fba94161c60 con 0x7fba9415bb20 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.295+0000 7fba91ffb640 1 -- 172.21.15.77:0/4124030117 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba94161980 con 0x7fba9415bb20 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.295+0000 7fba91ffb640 1 -- 172.21.15.77:0/4124030117 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3214081849 0 0) 0x7fba7c003520 con 0x7fba940cfff0 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.295+0000 7fba937fe640 1 -- 172.21.15.77:0/4124030117 >> v1:172.21.15.158:6789/0 conn(0x7fba9415bb20 legacy=0x7fba940d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.295+0000 7fba937fe640 1 -- 172.21.15.77:0/4124030117 >> v1:172.21.15.77:6789/0 conn(0x7fba940cfff0 legacy=0x7fba940cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.295+0000 7fba937fe640 1 -- 172.21.15.77:0/4124030117 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba94161f40 con 0x7fba940d04a0 2024-03-06T07:28:00.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba937fe640 1 --2- 172.21.15.77:0/4124030117 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba940d04a0 0x7fba940ccfc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009d90 tx=0x7fba8c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=788a11e17f1360d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba91ffb640 1 -- 172.21.15.77:0/4124030117 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c039070 con 0x7fba940d04a0 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba91ffb640 1 -- 172.21.15.77:0/4124030117 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c02c940 con 0x7fba940d04a0 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba91ffb640 1 -- 172.21.15.77:0/4124030117 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88001670 con 0x7fba940d04a0 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba9993a640 1 -- 172.21.15.77:0/4124030117 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba940d04a0 msgr2=0x7fba940ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba9993a640 1 --2- 172.21.15.77:0/4124030117 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba940d04a0 0x7fba940ccfc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009d90 tx=0x7fba8c02be10 comp rx=0 tx=0).stop 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba9993a640 1 -- 172.21.15.77:0/4124030117 shutdown_connections 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba9993a640 1 --2- 172.21.15.77:0/4124030117 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba940d04a0 0x7fba940ccfc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba9993a640 1 -- 172.21.15.77:0/4124030117 >> 172.21.15.77:0/4124030117 conn(0x7fba9409a9f0 msgr2=0x7fba9409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba9993a640 1 -- 172.21.15.77:0/4124030117 shutdown_connections 2024-03-06T07:28:00.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.296+0000 7fba9993a640 1 -- 172.21.15.77:0/4124030117 wait complete. 2024-03-06T07:28:00.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.298+0000 7fba9993a640 1 Processor -- start 2024-03-06T07:28:00.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.298+0000 7fba9993a640 1 -- start start 2024-03-06T07:28:00.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.298+0000 7fba9993a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba940cfff0 0x7fba940ce720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.298+0000 7fba9993a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba940d04a0 0x7fba940c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.298+0000 7fba9993a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9415bb20 0x7fba94157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.298+0000 7fba9993a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba940d08d0 con 0x7fba940cfff0 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.298+0000 7fba9993a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba94162500 con 0x7fba9415bb20 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.298+0000 7fba9993a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba940d1980 con 0x7fba940d04a0 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba940cfff0 0x7fba940ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba940cfff0 0x7fba940ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43346/0 (socket says 172.21.15.77:43346) 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 -- 172.21.15.77:0/960282641 learned_addr learned my addr 172.21.15.77:0/960282641 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba927fc640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba940d04a0 0x7fba940c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba937fe640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9415bb20 0x7fba94157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 -- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba940d04a0 msgr2=0x7fba940c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba940d04a0 0x7fba940c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 -- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9415bb20 msgr2=0x7fba94157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9415bb20 0x7fba94157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 -- 172.21.15.77:0/960282641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba940c5f60 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba937fe640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9415bb20 0x7fba94157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba92ffd640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba940cfff0 0x7fba940ce720 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fba8400c490 tx=0x7fba8400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba7bfff640 1 -- 172.21.15.77:0/960282641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba840052d0 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba7bfff640 1 -- 172.21.15.77:0/960282641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba84005470 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba940c6240 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba7bfff640 1 -- 172.21.15.77:0/960282641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84012870 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.299+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba940c6800 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.300+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9404f330 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.301+0000 7fba7bfff640 1 -- 172.21.15.77:0/960282641 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba84012a10 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.301+0000 7fba7bfff640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b4d0 0x7fba8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.301+0000 7fba7bfff640 1 -- 172.21.15.77:0/960282641 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fba8408a020 con 0x7fba940cfff0 2024-03-06T07:28:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.301+0000 7fba927fc640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b4d0 0x7fba8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.302+0000 7fba927fc640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b4d0 0x7fba8006d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fba88005880 tx=0x7fba88009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.310+0000 7fba7bfff640 1 -- 172.21.15.77:0/960282641 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba84058d30 con 0x7fba940cfff0 2024-03-06T07:28:00.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.528+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba940cce80 con 0x7fba940cfff0 2024-03-06T07:28:00.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.528+0000 7fba7bfff640 1 -- 172.21.15.77:0/960282641 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fba84058630 con 0x7fba940cfff0 2024-03-06T07:28:00.530 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:00.530 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:00.531 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:00.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b4d0 msgr2=0x7fba8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b4d0 0x7fba8006d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fba88005880 tx=0x7fba88009040 comp rx=0 tx=0).stop 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba940cfff0 msgr2=0x7fba940ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba940cfff0 0x7fba940ce720 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fba8400c490 tx=0x7fba8400c950 comp rx=0 tx=0).stop 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 shutdown_connections 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba9415bb20 0x7fba94157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba940d04a0 0x7fba940c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8006b4d0 0x7fba8006d990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 --2- 172.21.15.77:0/960282641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba940cfff0 0x7fba940ce720 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.531+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 >> 172.21.15.77:0/960282641 conn(0x7fba9409a9f0 msgr2=0x7fba9409add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.532+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 shutdown_connections 2024-03-06T07:28:00.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:00.532+0000 7fba9993a640 1 -- 172.21.15.77:0/960282641 wait complete. 2024-03-06T07:28:00.545 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:00.689 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:28:00.690 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:28:00.690 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:28:00.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:00.845 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:05.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:05.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.799+0000 7f82b2e2a640 1 Processor -- start 2024-03-06T07:28:05.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.799+0000 7f82b2e2a640 1 -- start start 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.800+0000 7f82b2e2a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82ac1617e0 0x7f82ac0cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.800+0000 7f82b2e2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0cf9a0 0x7f82ac0d2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.800+0000 7f82b2e2a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac0b3840 con 0x7f82ac0d27b0 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.800+0000 7f82b2e2a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac0b35f0 con 0x7f82ac1617e0 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.800+0000 7f82b2e2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0b4170 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.800+0000 7f82b0bc6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82ac1617e0 0x7f82ac0cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.800+0000 7f82b0bc6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82ac1617e0 0x7f82ac0cf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43352/0 (socket says 172.21.15.77:43352) 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.800+0000 7f82b0bc6640 1 -- 172.21.15.77:0/1714766634 learned_addr learned my addr 172.21.15.77:0/1714766634 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.801+0000 7f82b0bc6640 1 -- 172.21.15.77:0/1714766634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0cf9a0 msgr2=0x7f82ac0d2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.801+0000 7f82b0bc6640 1 --2- 172.21.15.77:0/1714766634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0cf9a0 0x7f82ac0d2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.801+0000 7f82b0bc6640 1 -- 172.21.15.77:0/1714766634 >> v1:172.21.15.158:6789/0 conn(0x7f82ac0d27b0 legacy=0x7f82ac0d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.801+0000 7f82b0bc6640 1 -- 172.21.15.77:0/1714766634 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac0b3b20 con 0x7f82ac1617e0 2024-03-06T07:28:05.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.801+0000 7f82b0bc6640 1 --2- 172.21.15.77:0/1714766634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac1617e0 0x7f82ac0cf460 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009870 tx=0x7f82a802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781ab9dd64d7d5f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.802+0000 7f82a37fe640 1 -- 172.21.15.77:0/1714766634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a80317c0 con 0x7f82ac1617e0 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.802+0000 7f82a37fe640 1 -- 172.21.15.77:0/1714766634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a8037070 con 0x7f82ac1617e0 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.802+0000 7f82a37fe640 1 -- 172.21.15.77:0/1714766634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8031b80 con 0x7f82ac1617e0 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.802+0000 7f82b2e2a640 1 -- 172.21.15.77:0/1714766634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac1617e0 msgr2=0x7f82ac0cf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.802+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/1714766634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac1617e0 0x7f82ac0cf460 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009870 tx=0x7f82a802b860 comp rx=0 tx=0).stop 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.802+0000 7f82b2e2a640 1 -- 172.21.15.77:0/1714766634 shutdown_connections 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.802+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/1714766634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0cf9a0 0x7f82ac0d2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.802+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/1714766634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac1617e0 0x7f82ac0cf460 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.803+0000 7f82b2e2a640 1 -- 172.21.15.77:0/1714766634 >> 172.21.15.77:0/1714766634 conn(0x7f82ac0d3c90 msgr2=0x7f82ac0d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.803+0000 7f82b2e2a640 1 -- 172.21.15.77:0/1714766634 shutdown_connections 2024-03-06T07:28:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.803+0000 7f82b2e2a640 1 -- 172.21.15.77:0/1714766634 wait complete. 2024-03-06T07:28:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.805+0000 7f82b2e2a640 1 Processor -- start 2024-03-06T07:28:05.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.805+0000 7f82b2e2a640 1 -- start start 2024-03-06T07:28:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b2e2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac0cf9a0 0x7f82ac0b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b2e2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0d27b0 0x7f82ac151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b2e2a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ac1617e0 0x7f82ac0b8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b2e2a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0b0860 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b2e2a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0d4990 con 0x7f82ac1617e0 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b2e2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0b09e0 con 0x7f82ac0d27b0 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b0bc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac0cf9a0 0x7f82ac0b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b0bc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac0cf9a0 0x7f82ac0b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43362/0 (socket says 172.21.15.77:43362) 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b0bc6640 1 -- 172.21.15.77:0/3042575024 learned_addr learned my addr 172.21.15.77:0/3042575024 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82b13c7640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ac1617e0 0x7f82ac0b8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.806+0000 7f82a3fff640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0d27b0 0x7f82ac151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82b0bc6640 1 -- 172.21.15.77:0/3042575024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0d27b0 msgr2=0x7f82ac151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82b0bc6640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0d27b0 0x7f82ac151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82b0bc6640 1 -- 172.21.15.77:0/3042575024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ac1617e0 msgr2=0x7f82ac0b8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82b0bc6640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ac1617e0 0x7f82ac0b8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82b0bc6640 1 -- 172.21.15.77:0/3042575024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac0cbef0 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82b0bc6640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac0cf9a0 0x7f82ac0b1be0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f82a8000c00 tx=0x7f82a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82a17fa640 1 -- 172.21.15.77:0/3042575024 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8032860 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82a17fa640 1 -- 172.21.15.77:0/3042575024 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a8032a00 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac0b2120 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82a17fa640 1 -- 172.21.15.77:0/3042575024 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8032df0 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.807+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac0cc1b0 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.808+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac08cce0 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.809+0000 7f82a17fa640 1 -- 172.21.15.77:0/3042575024 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82a804c060 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.809+0000 7f82a17fa640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f829806b4d0 0x7f829806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.809+0000 7f82a17fa640 1 -- 172.21.15.77:0/3042575024 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f82a80abd00 con 0x7f82ac0cf9a0 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.809+0000 7f82a3fff640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f829806b4d0 0x7f829806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.810+0000 7f82a3fff640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f829806b4d0 0x7f829806d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0d0fb0 tx=0x7f829c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:05.818+0000 7f82a17fa640 1 -- 172.21.15.77:0/3042575024 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82ac08cce0 con 0x7f82ac0cf9a0 2024-03-06T07:28:06.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.033+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82ac04f330 con 0x7f82ac0cf9a0 2024-03-06T07:28:06.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.034+0000 7f82a17fa640 1 -- 172.21.15.77:0/3042575024 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f82a8040070 con 0x7f82ac0cf9a0 2024-03-06T07:28:06.036 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:06.036 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:06.036 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f829806b4d0 msgr2=0x7f829806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f829806b4d0 0x7f829806d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0d0fb0 tx=0x7f829c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac0cf9a0 msgr2=0x7f82ac0b1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac0cf9a0 0x7f82ac0b1be0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f82a8000c00 tx=0x7f82a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 shutdown_connections 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82ac1617e0 0x7f82ac0b8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82ac0d27b0 0x7f82ac151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f829806b4d0 0x7f829806d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 --2- 172.21.15.77:0/3042575024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82ac0cf9a0 0x7f82ac0b1be0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 >> 172.21.15.77:0/3042575024 conn(0x7f82ac0d3c90 msgr2=0x7f82ac0b5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 shutdown_connections 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.036+0000 7f82b2e2a640 1 -- 172.21.15.77:0/3042575024 wait complete. 2024-03-06T07:28:06.049 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:06.166 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:06.166 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:06.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:06.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.265+0000 7fe74ac66640 1 Processor -- start 2024-03-06T07:28:06.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.265+0000 7fe74ac66640 1 -- start start 2024-03-06T07:28:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.266+0000 7fe74ac66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440d22b0 0x7fe7440d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.266+0000 7fe74ac66640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7440b41f0 0x7fe7440b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.266+0000 7fe74ac66640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7440b3430 con 0x7fe74415dfe0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.266+0000 7fe74ac66640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7440d2c50 con 0x7fe7440b41f0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.266+0000 7fe74ac66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440b31e0 con 0x7fe7440d22b0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.266+0000 7fe748a02640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe74415dfe0 0x7fe7440cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46150/0 (socket says 172.21.15.77:46150) 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe748a02640 1 -- 172.21.15.77:0/926438468 learned_addr learned my addr 172.21.15.77:0/926438468 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe749203640 1 --2- 172.21.15.77:0/926438468 >> v2:172.21.15.77:3300/0 conn(0x7fe7440b41f0 0x7fe7440b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe743fff640 1 --2- 172.21.15.77:0/926438468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440d22b0 0x7fe7440d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe7437fe640 1 -- 172.21.15.77:0/926438468 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350936849 0 0) 0x7fe7440b3430 con 0x7fe74415dfe0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe7437fe640 1 -- 172.21.15.77:0/926438468 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe734003520 con 0x7fe74415dfe0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe749203640 1 -- 172.21.15.77:0/926438468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440d22b0 msgr2=0x7fe7440d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe749203640 1 --2- 172.21.15.77:0/926438468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440d22b0 0x7fe7440d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe749203640 1 -- 172.21.15.77:0/926438468 >> v1:172.21.15.77:6789/0 conn(0x7fe74415dfe0 legacy=0x7fe7440cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe749203640 1 -- 172.21.15.77:0/926438468 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7440b3710 con 0x7fe7440b41f0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.267+0000 7fe743fff640 1 --2- 172.21.15.77:0/926438468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440d22b0 0x7fe7440d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe749203640 1 --2- 172.21.15.77:0/926438468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7440b41f0 0x7fe7440b4650 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009d90 tx=0x7fe73c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2843fdae6e784c8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe7437fe640 1 -- 172.21.15.77:0/926438468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c039070 con 0x7fe7440b41f0 2024-03-06T07:28:06.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe7437fe640 1 -- 172.21.15.77:0/926438468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c02cc50 con 0x7fe7440b41f0 2024-03-06T07:28:06.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe7437fe640 1 -- 172.21.15.77:0/926438468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c034440 con 0x7fe7440b41f0 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe74ac66640 1 -- 172.21.15.77:0/926438468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7440b41f0 msgr2=0x7fe7440b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe74ac66640 1 --2- 172.21.15.77:0/926438468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7440b41f0 0x7fe7440b4650 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009d90 tx=0x7fe73c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe74ac66640 1 -- 172.21.15.77:0/926438468 shutdown_connections 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe74ac66640 1 --2- 172.21.15.77:0/926438468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7440b41f0 0x7fe7440b4650 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe74ac66640 1 --2- 172.21.15.77:0/926438468 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440d22b0 0x7fe7440d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe74ac66640 1 -- 172.21.15.77:0/926438468 >> 172.21.15.77:0/926438468 conn(0x7fe7440d3cd0 msgr2=0x7fe7440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.268+0000 7fe74ac66640 1 -- 172.21.15.77:0/926438468 shutdown_connections 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.269+0000 7fe74ac66640 1 -- 172.21.15.77:0/926438468 wait complete. 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.270+0000 7fe74ac66640 1 Processor -- start 2024-03-06T07:28:06.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.270+0000 7fe74ac66640 1 -- start start 2024-03-06T07:28:06.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.270+0000 7fe74ac66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440b41f0 0x7fe74415c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.270+0000 7fe74ac66640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7440d22b0 0x7fe74406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.270+0000 7fe74ac66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe74415dfe0 0x7fe74406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.270+0000 7fe74ac66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440cfa00 con 0x7fe74415dfe0 2024-03-06T07:28:06.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.270+0000 7fe74ac66640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440b3d60 con 0x7fe7440d22b0 2024-03-06T07:28:06.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.270+0000 7fe74ac66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440d1c70 con 0x7fe7440b41f0 2024-03-06T07:28:06.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe74415dfe0 0x7fe74406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe74415dfe0 0x7fe74406cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43374/0 (socket says 172.21.15.77:43374) 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 -- 172.21.15.77:0/3378810084 learned_addr learned my addr 172.21.15.77:0/3378810084 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe748a02640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440b41f0 0x7fe74415c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe743fff640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7440d22b0 0x7fe74406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 -- 172.21.15.77:0/3378810084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440b41f0 msgr2=0x7fe74415c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440b41f0 0x7fe74415c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 -- 172.21.15.77:0/3378810084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7440d22b0 msgr2=0x7fe74406ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7440d22b0 0x7fe74406ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 -- 172.21.15.77:0/3378810084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7440b7760 con 0x7fe74415dfe0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe749203640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe74415dfe0 0x7fe74406cf90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009eb0 tx=0x7fe73c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe7417fa640 1 -- 172.21.15.77:0/3378810084 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c039050 con 0x7fe74415dfe0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe7417fa640 1 -- 172.21.15.77:0/3378810084 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c045070 con 0x7fe74415dfe0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe7417fa640 1 -- 172.21.15.77:0/3378810084 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c03e610 con 0x7fe74415dfe0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7440b7a40 con 0x7fe74415dfe0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.271+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7440b8000 con 0x7fe74415dfe0 2024-03-06T07:28:06.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.272+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74404f330 con 0x7fe74415dfe0 2024-03-06T07:28:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.273+0000 7fe7417fa640 1 -- 172.21.15.77:0/3378810084 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe73c03e7b0 con 0x7fe74415dfe0 2024-03-06T07:28:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.273+0000 7fe7417fa640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe72806b4d0 0x7fe72806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.273+0000 7fe7417fa640 1 -- 172.21.15.77:0/3378810084 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe73c0ace30 con 0x7fe74415dfe0 2024-03-06T07:28:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.273+0000 7fe748a02640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe72806b4d0 0x7fe72806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.273+0000 7fe748a02640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe72806b4d0 0x7fe72806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe7440cc810 tx=0x7fe738009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.282+0000 7fe7417fa640 1 -- 172.21.15.77:0/3378810084 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe73c07bd70 con 0x7fe74415dfe0 2024-03-06T07:28:06.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.491+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe74415d5c0 con 0x7fe74415dfe0 2024-03-06T07:28:06.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.491+0000 7fe7417fa640 1 -- 172.21.15.77:0/3378810084 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fe73c07b670 con 0x7fe74415dfe0 2024-03-06T07:28:06.493 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:06.493 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:06.494 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:06.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe72806b4d0 msgr2=0x7fe72806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe72806b4d0 0x7fe72806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe7440cc810 tx=0x7fe738009040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe74415dfe0 msgr2=0x7fe74406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe74415dfe0 0x7fe74406cf90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009eb0 tx=0x7fe73c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 shutdown_connections 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe72806b4d0 0x7fe72806d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe74415dfe0 0x7fe74406cf90 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7440d22b0 0x7fe74406ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 --2- 172.21.15.77:0/3378810084 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7440b41f0 0x7fe74415c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 >> 172.21.15.77:0/3378810084 conn(0x7fe7440d3cd0 msgr2=0x7fe7440cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 shutdown_connections 2024-03-06T07:28:06.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.493+0000 7fe74ac66640 1 -- 172.21.15.77:0/3378810084 wait complete. 2024-03-06T07:28:06.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:06.622 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:28:06.622 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:28:06.623 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:06.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.722+0000 7f4ccbb32640 1 Processor -- start 2024-03-06T07:28:06.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4ccbb32640 1 -- start start 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4ccbb32640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4ccbb32640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4cc40cf830 0x7f4cc40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4ccbb32640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc40b3430 con 0x7f4cc40d27f0 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4ccbb32640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc40d2c50 con 0x7f4cc40cf830 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4ccbb32640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40b31e0 con 0x7f4cc415dfe0 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4cc90cd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4cc40cf830 0x7f4cc40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4cca0cf640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4cc40d27f0 0x7f4cc40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46156/0 (socket says 172.21.15.77:46156) 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.723+0000 7f4cca0cf640 1 -- 172.21.15.77:0/979318322 learned_addr learned my addr 172.21.15.77:0/979318322 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.724+0000 7f4cc90cd640 1 -- 172.21.15.77:0/979318322 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc415dfe0 msgr2=0x7f4cc40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.724+0000 7f4cc90cd640 1 --2- 172.21.15.77:0/979318322 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.724+0000 7f4cc90cd640 1 -- 172.21.15.77:0/979318322 >> v1:172.21.15.77:6789/0 conn(0x7f4cc40d27f0 legacy=0x7f4cc40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.724+0000 7f4cc90cd640 1 -- 172.21.15.77:0/979318322 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40b3710 con 0x7f4cc40cf830 2024-03-06T07:28:06.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.724+0000 7f4cc88cc640 1 -- 172.21.15.77:0/979318322 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432024489 0 0) 0x7f4cc40b3430 con 0x7f4cc40d27f0 2024-03-06T07:28:06.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.725+0000 7f4cc90cd640 1 --2- 172.21.15.77:0/979318322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40cf830 0x7f4cc40d22b0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80097a0 tx=0x7f4cb802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e813041ad0a7c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.725+0000 7f4cc88cc640 1 -- 172.21.15.77:0/979318322 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8035020 con 0x7f4cc40cf830 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.725+0000 7f4cc88cc640 1 -- 172.21.15.77:0/979318322 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb8031830 con 0x7f4cc40cf830 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.725+0000 7f4cc88cc640 1 -- 172.21.15.77:0/979318322 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8031c00 con 0x7f4cc40cf830 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.725+0000 7f4ccbb32640 1 -- 172.21.15.77:0/979318322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40cf830 msgr2=0x7f4cc40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.725+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/979318322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40cf830 0x7f4cc40d22b0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80097a0 tx=0x7f4cb802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.726+0000 7f4ccbb32640 1 -- 172.21.15.77:0/979318322 shutdown_connections 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.726+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/979318322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40cf830 0x7f4cc40d22b0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.726+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/979318322 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc415dfe0 0x7f4cc40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.726+0000 7f4ccbb32640 1 -- 172.21.15.77:0/979318322 >> 172.21.15.77:0/979318322 conn(0x7f4cc40d3cd0 msgr2=0x7f4cc40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.726+0000 7f4ccbb32640 1 -- 172.21.15.77:0/979318322 shutdown_connections 2024-03-06T07:28:06.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.726+0000 7f4ccbb32640 1 -- 172.21.15.77:0/979318322 wait complete. 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.727+0000 7f4ccbb32640 1 Processor -- start 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.727+0000 7f4ccbb32640 1 -- start start 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.727+0000 7f4ccbb32640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc40cf830 0x7f4cc41f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.727+0000 7f4ccbb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40d27f0 0x7f4cc40cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.727+0000 7f4ccbb32640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cc415dfe0 0x7f4cc40ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.727+0000 7f4ccbb32640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40d1c70 con 0x7f4cc40d27f0 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.727+0000 7f4ccbb32640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40b3d60 con 0x7f4cc415dfe0 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.727+0000 7f4ccbb32640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40d1df0 con 0x7f4cc40cf830 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40d27f0 0x7f4cc40cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40d27f0 0x7f4cc40cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43404/0 (socket says 172.21.15.77:43404) 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 -- 172.21.15.77:0/3634996961 learned_addr learned my addr 172.21.15.77:0/3634996961 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc98ce640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc40cf830 0x7f4cc41f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cca0cf640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cc415dfe0 0x7f4cc40ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 -- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc40cf830 msgr2=0x7f4cc41f2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc40cf830 0x7f4cc41f2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 -- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cc415dfe0 msgr2=0x7f4cc40ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cc415dfe0 0x7f4cc40ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 -- 172.21.15.77:0/3634996961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40cdb70 con 0x7f4cc40d27f0 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc98ce640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc40cf830 0x7f4cc41f2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc90cd640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40d27f0 0x7f4cc40cca40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80098c0 tx=0x7f4cb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc27fc640 1 -- 172.21.15.77:0/3634996961 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8004230 con 0x7f4cc40d27f0 2024-03-06T07:28:06.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc27fc640 1 -- 172.21.15.77:0/3634996961 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb80043d0 con 0x7f4cc40d27f0 2024-03-06T07:28:06.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc41f7020 con 0x7f4cc40d27f0 2024-03-06T07:28:06.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4cc27fc640 1 -- 172.21.15.77:0/3634996961 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb803cad0 con 0x7f4cc40d27f0 2024-03-06T07:28:06.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.728+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc40cd890 con 0x7f4cc40d27f0 2024-03-06T07:28:06.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.729+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc404f330 con 0x7f4cc40d27f0 2024-03-06T07:28:06.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.730+0000 7f4cc27fc640 1 -- 172.21.15.77:0/3634996961 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cb8038020 con 0x7f4cc40d27f0 2024-03-06T07:28:06.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.730+0000 7f4cc27fc640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb006b4d0 0x7f4cb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.730+0000 7f4cc98ce640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb006b4d0 0x7f4cb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.730+0000 7f4cc27fc640 1 -- 172.21.15.77:0/3634996961 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f4cb80ac520 con 0x7f4cc40d27f0 2024-03-06T07:28:06.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.731+0000 7f4cc98ce640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb006b4d0 0x7f4cb006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400a870 tx=0x7f4cb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.735+0000 7f4cc27fc640 1 -- 172.21.15.77:0/3634996961 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cb8041070 con 0x7f4cc40d27f0 2024-03-06T07:28:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.946+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cc40747a0 con 0x7f4cc40d27f0 2024-03-06T07:28:06.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.947+0000 7f4cc27fc640 1 -- 172.21.15.77:0/3634996961 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f4cb807b2b0 con 0x7f4cc40d27f0 2024-03-06T07:28:06.949 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:06.949 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:06.949 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb006b4d0 msgr2=0x7f4cb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb006b4d0 0x7f4cb006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400a870 tx=0x7f4cb4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40d27f0 msgr2=0x7f4cc40cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40d27f0 0x7f4cc40cca40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80098c0 tx=0x7f4cb80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 shutdown_connections 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4cb006b4d0 0x7f4cb006d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cc40d27f0 0x7f4cc40cca40 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cc415dfe0 0x7f4cc40ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 --2- 172.21.15.77:0/3634996961 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cc40cf830 0x7f4cc41f2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 >> 172.21.15.77:0/3634996961 conn(0x7f4cc40d3cd0 msgr2=0x7f4cc40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 shutdown_connections 2024-03-06T07:28:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:06.949+0000 7f4ccbb32640 1 -- 172.21.15.77:0/3634996961 wait complete. 2024-03-06T07:28:06.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:07.089 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:28:07.089 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:07.089 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:07.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.128+0000 7fa1976b6f40 1 Processor -- start 2024-03-06T07:28:07.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.128+0000 7fa1976b6f40 1 -- start start 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1976b6f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5626f2e4e550 0x5626f2eda900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1976b6f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 0x5626f2e50660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1976b6f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2ed98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1976b6f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f2e573c0 con 0x5626f2ee6060 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1976b6f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f2db4b70 con 0x5626f2e563b0 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1976b6f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f2e57660 con 0x5626f2e4e550 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1959b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2ed98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1959b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2ed98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43418/0 (socket says 172.21.15.77:43418) 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1959b8640 1 -- 172.21.15.77:0/502060715 learned_addr learned my addr 172.21.15.77:0/502060715 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:07.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1959b8640 1 -- 172.21.15.77:0/502060715 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5626f2e4e550 msgr2=0x5626f2eda900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1959b8640 1 --2- 172.21.15.77:0/502060715 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5626f2e4e550 0x5626f2eda900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1959b8640 1 -- 172.21.15.77:0/502060715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 msgr2=0x5626f2e50660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1959b8640 1 --2- 172.21.15.77:0/502060715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 0x5626f2e50660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.129+0000 7fa1959b8640 1 -- 172.21.15.77:0/502060715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f2e519e0 con 0x5626f2ee6060 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1959b8640 1 --2- 172.21.15.77:0/502060715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2ed98e0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fa190047010 tx=0x7fa19004f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f470493e1df7084 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1877fe640 1 -- 172.21.15.77:0/502060715 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa190066070 con 0x5626f2ee6060 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1877fe640 1 -- 172.21.15.77:0/502060715 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19004dcc0 con 0x5626f2ee6060 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1877fe640 1 -- 172.21.15.77:0/502060715 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19004d770 con 0x5626f2ee6060 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 -- 172.21.15.77:0/502060715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 msgr2=0x5626f2ed98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 --2- 172.21.15.77:0/502060715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2ed98e0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fa190047010 tx=0x7fa19004f000 comp rx=0 tx=0).stop 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 -- 172.21.15.77:0/502060715 shutdown_connections 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 --2- 172.21.15.77:0/502060715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2ed98e0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 --2- 172.21.15.77:0/502060715 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 0x5626f2e50660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 --2- 172.21.15.77:0/502060715 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5626f2e4e550 0x5626f2eda900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 -- 172.21.15.77:0/502060715 >> 172.21.15.77:0/502060715 conn(0x5626f2e38790 msgr2=0x5626f2e3d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 -- 172.21.15.77:0/502060715 shutdown_connections 2024-03-06T07:28:07.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.130+0000 7fa1976b6f40 1 -- 172.21.15.77:0/502060715 wait complete. 2024-03-06T07:28:07.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.131+0000 7fa1976b6f40 1 Processor -- start 2024-03-06T07:28:07.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.131+0000 7fa1976b6f40 1 -- start start 2024-03-06T07:28:07.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1976b6f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5626f2e4e550 0x5626f2ed7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1976b6f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 0x5626f2ed7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1976b6f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2e3b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1976b6f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f2e51f60 con 0x5626f2ee6060 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1976b6f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f2e51c80 con 0x5626f2e563b0 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1976b6f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f2edf2b0 con 0x5626f2e4e550 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1959b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2e3b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1959b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2e3b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43420/0 (socket says 172.21.15.77:43420) 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1959b8640 1 -- 172.21.15.77:0/749092329 learned_addr learned my addr 172.21.15.77:0/749092329 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1951b7640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5626f2e4e550 0x5626f2ed7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1949b6640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 0x5626f2ed7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1959b8640 1 -- 172.21.15.77:0/749092329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5626f2e4e550 msgr2=0x5626f2ed7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1959b8640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5626f2e4e550 0x5626f2ed7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1959b8640 1 -- 172.21.15.77:0/749092329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 msgr2=0x5626f2ed7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.132+0000 7fa1959b8640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 0x5626f2ed7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.133+0000 7fa1959b8640 1 -- 172.21.15.77:0/749092329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f2ecfe50 con 0x5626f2ee6060 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.133+0000 7fa1949b6640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5626f2e563b0 0x5626f2ed7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.133+0000 7fa1959b8640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2e3b2b0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fa190061a20 tx=0x7fa190003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.133+0000 7fa1867fc640 1 -- 172.21.15.77:0/749092329 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa190066040 con 0x5626f2ee6060 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.133+0000 7fa1867fc640 1 -- 172.21.15.77:0/749092329 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa190055430 con 0x5626f2ee6060 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.133+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5626f2ed7db0 con 0x5626f2ee6060 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.133+0000 7fa1867fc640 1 -- 172.21.15.77:0/749092329 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19004d770 con 0x5626f2ee6060 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.133+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626f2ecfb70 con 0x5626f2ee6060 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.134+0000 7fa1867fc640 1 -- 172.21.15.77:0/749092329 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa190055820 con 0x5626f2ee6060 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.135+0000 7fa1867fc640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa178069eb0 0x7fa17806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.135+0000 7fa1867fc640 1 -- 172.21.15.77:0/749092329 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa1900ceb40 con 0x5626f2ee6060 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.135+0000 7fa1951b7640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa178069eb0 0x7fa17806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.135+0000 7fa1976b6f40 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5626f2e3b7f0 0x5626f2e3bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.135+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) v9 -- 0x5626f2ed01b0 con 0x5626f2e3b7f0 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.135+0000 7fa1949b6640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5626f2e3b7f0 0x5626f2e3bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.135+0000 7fa1951b7640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa178069eb0 0x7fa17806c370 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa188000f60 tx=0x7fa1880016f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.135+0000 7fa1949b6640 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5626f2e3b7f0 0x5626f2e3bc00 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1949b6640 1 -- 172.21.15.77:0/749092329 <== osd.0 v2:172.21.15.77:6808/3119591583 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fa18c00c730 con 0x5626f2e3b7f0 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5626f2e3b7f0 msgr2=0x5626f2e3bc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x5626f2e3b7f0 0x5626f2e3bc00 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa178069eb0 msgr2=0x7fa17806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa178069eb0 0x7fa17806c370 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa188000f60 tx=0x7fa1880016f0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 msgr2=0x5626f2e3b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 --2- 172.21.15.77:0/749092329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626f2ee6060 0x5626f2e3b2b0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fa190061a20 tx=0x7fa190003330 comp rx=0 tx=0).stop 2024-03-06T07:28:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1959b8640 1 -- 172.21.15.77:0/749092329 reap_dead start 2024-03-06T07:28:07.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 shutdown_connections 2024-03-06T07:28:07.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 >> 172.21.15.77:0/749092329 conn(0x5626f2e38790 msgr2=0x5626f2ed96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 shutdown_connections 2024-03-06T07:28:07.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.136+0000 7fa1976b6f40 1 -- 172.21.15.77:0/749092329 wait complete. 2024-03-06T07:28:07.142 INFO:teuthology.orchestra.run.smithi077.stdout:W�file_ddir_2* 2024-03-06T07:28:07.143 DEBUG:teuthology.orchestra.run.smithi077:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:07.241 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:28:07.241 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 1099511627777, 2024-03-06T07:28:07.241 INFO:teuthology.orchestra.run.smithi077.stdout: "ancestors": [ 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: "dirino": 1099511628278, 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: "dname": "file_d", 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: "version": 25 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: "dirino": 1, 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: "dname": "dir_2", 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: "version": 42 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: "pool": 17, 2024-03-06T07:28:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: "old_pools": [] 2024-03-06T07:28:07.243 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:28:07.243 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:07.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:07.245 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:28:07.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:07.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.442+0000 7fbfbc40e640 1 Processor -- start 2024-03-06T07:28:07.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.442+0000 7fbfbc40e640 1 -- start start 2024-03-06T07:28:07.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfbc40e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbfb40d22b0 0x7fbfb40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfbc40e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbfb40b41f0 0x7fbfb40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfbc40e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40b3430 con 0x7fbfb415dfe0 2024-03-06T07:28:07.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfbc40e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb40d2c50 con 0x7fbfb40d22b0 2024-03-06T07:28:07.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfbc40e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb40b31e0 con 0x7fbfb40b41f0 2024-03-06T07:28:07.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfb99a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbfb40d22b0 0x7fbfb40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfba1aa640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbfb415dfe0 0x7fbfb40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46166/0 (socket says 172.21.15.77:46166) 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfb99a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbfb40d22b0 0x7fbfb40d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43422/0 (socket says 172.21.15.77:43422) 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.443+0000 7fbfb99a9640 1 -- 172.21.15.77:0/2211451131 learned_addr learned my addr 172.21.15.77:0/2211451131 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.444+0000 7fbfba9ab640 1 --2- 172.21.15.77:0/2211451131 >> v2:172.21.15.158:3300/0 conn(0x7fbfb40b41f0 0x7fbfb40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.444+0000 7fbfb99a9640 1 -- 172.21.15.77:0/2211451131 >> v2:172.21.15.158:3300/0 conn(0x7fbfb40b41f0 msgr2=0x7fbfb40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.444+0000 7fbfb99a9640 1 --2- 172.21.15.77:0/2211451131 >> v2:172.21.15.158:3300/0 conn(0x7fbfb40b41f0 0x7fbfb40b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.444+0000 7fbfb99a9640 1 -- 172.21.15.77:0/2211451131 >> v1:172.21.15.77:6789/0 conn(0x7fbfb415dfe0 legacy=0x7fbfb40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.444+0000 7fbfb99a9640 1 -- 172.21.15.77:0/2211451131 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb40b3710 con 0x7fbfb40d22b0 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.444+0000 7fbfb91a8640 1 -- 172.21.15.77:0/2211451131 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508161193 0 0) 0x7fbfb40b3430 con 0x7fbfb415dfe0 2024-03-06T07:28:07.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.444+0000 7fbfba9ab640 1 --2- 172.21.15.77:0/2211451131 >> v2:172.21.15.158:3300/0 conn(0x7fbfb40b41f0 0x7fbfb40b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:07.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.445+0000 7fbfb99a9640 1 --2- 172.21.15.77:0/2211451131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb40d22b0 0x7fbfb40d2710 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009870 tx=0x7fbfa402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79791bf2167a7b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.445+0000 7fbfb91a8640 1 -- 172.21.15.77:0/2211451131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4034070 con 0x7fbfb40d22b0 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.445+0000 7fbfb91a8640 1 -- 172.21.15.77:0/2211451131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa4030860 con 0x7fbfb40d22b0 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.445+0000 7fbfb91a8640 1 -- 172.21.15.77:0/2211451131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4030c30 con 0x7fbfb40d22b0 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.445+0000 7fbfbc40e640 1 -- 172.21.15.77:0/2211451131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb40d22b0 msgr2=0x7fbfb40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.445+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/2211451131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb40d22b0 0x7fbfb40d2710 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009870 tx=0x7fbfa402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.445+0000 7fbfbc40e640 1 -- 172.21.15.77:0/2211451131 shutdown_connections 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.445+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/2211451131 >> v2:172.21.15.158:3300/0 conn(0x7fbfb40b41f0 0x7fbfb40b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.446+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/2211451131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb40d22b0 0x7fbfb40d2710 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.446+0000 7fbfbc40e640 1 -- 172.21.15.77:0/2211451131 >> 172.21.15.77:0/2211451131 conn(0x7fbfb40d3cd0 msgr2=0x7fbfb40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.446+0000 7fbfbc40e640 1 -- 172.21.15.77:0/2211451131 shutdown_connections 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.446+0000 7fbfbc40e640 1 -- 172.21.15.77:0/2211451131 wait complete. 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.447+0000 7fbfbc40e640 1 Processor -- start 2024-03-06T07:28:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.447+0000 7fbfbc40e640 1 -- start start 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.447+0000 7fbfbc40e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfb40b41f0 0x7fbfb415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.447+0000 7fbfbc40e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfb40d22b0 0x7fbfb40b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.447+0000 7fbfbc40e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb415dfe0 0x7fbfb4151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.447+0000 7fbfbc40e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40cfa00 con 0x7fbfb415dfe0 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.447+0000 7fbfbc40e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40b3d60 con 0x7fbfb40d22b0 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.447+0000 7fbfbc40e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40d1c70 con 0x7fbfb40b41f0 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb415dfe0 0x7fbfb4151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb415dfe0 0x7fbfb4151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43436/0 (socket says 172.21.15.77:43436) 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 -- 172.21.15.77:0/1769942507 learned_addr learned my addr 172.21.15.77:0/1769942507 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba1aa640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfb40b41f0 0x7fbfb415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfb99a9640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfb40d22b0 0x7fbfb40b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 -- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfb40b41f0 msgr2=0x7fbfb415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfb40b41f0 0x7fbfb415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 -- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfb40d22b0 msgr2=0x7fbfb40b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfb40d22b0 0x7fbfb40b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 -- 172.21.15.77:0/1769942507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb415ec10 con 0x7fbfb415dfe0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfb99a9640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfb40d22b0 0x7fbfb40b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfba9ab640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb415dfe0 0x7fbfb4151cb0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fbfac007ef0 tx=0x7fbfac00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfaaffd640 1 -- 172.21.15.77:0/1769942507 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac01a070 con 0x7fbfb415dfe0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfaaffd640 1 -- 172.21.15.77:0/1769942507 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac003b90 con 0x7fbfb415dfe0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb415eef0 con 0x7fbfb415dfe0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfaaffd640 1 -- 172.21.15.77:0/1769942507 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac004030 con 0x7fbfb415dfe0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.448+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb415f7d0 con 0x7fbfb415dfe0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.449+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb404f330 con 0x7fbfb415dfe0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.450+0000 7fbfaaffd640 1 -- 172.21.15.77:0/1769942507 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfac01d070 con 0x7fbfb415dfe0 2024-03-06T07:28:07.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.450+0000 7fbfaaffd640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf9806b4d0 0x7fbf9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.450+0000 7fbfaaffd640 1 -- 172.21.15.77:0/1769942507 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fbfac08a4e0 con 0x7fbfb415dfe0 2024-03-06T07:28:07.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.450+0000 7fbfba1aa640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf9806b4d0 0x7fbf9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.451+0000 7fbfba1aa640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf9806b4d0 0x7fbf9806d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0007fd0 tx=0x7fbfb0007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.455+0000 7fbfaaffd640 1 -- 172.21.15.77:0/1769942507 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfac010070 con 0x7fbfb415dfe0 2024-03-06T07:28:07.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.668+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfb40b1ed0 con 0x7fbfb415dfe0 2024-03-06T07:28:07.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.669+0000 7fbfaaffd640 1 -- 172.21.15.77:0/1769942507 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fbfac058c20 con 0x7fbfb415dfe0 2024-03-06T07:28:07.671 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:07.671 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:07.671 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf9806b4d0 msgr2=0x7fbf9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf9806b4d0 0x7fbf9806d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0007fd0 tx=0x7fbfb0007de0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb415dfe0 msgr2=0x7fbfb4151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb415dfe0 0x7fbfb4151cb0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fbfac007ef0 tx=0x7fbfac00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 shutdown_connections 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf9806b4d0 0x7fbf9806d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb415dfe0 0x7fbfb4151cb0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfb40d22b0 0x7fbfb40b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.670+0000 7fbfbc40e640 1 --2- 172.21.15.77:0/1769942507 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfb40b41f0 0x7fbfb415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.671+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 >> 172.21.15.77:0/1769942507 conn(0x7fbfb40d3cd0 msgr2=0x7fbfb415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.671+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 shutdown_connections 2024-03-06T07:28:07.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:07.671+0000 7fbfbc40e640 1 -- 172.21.15.77:0/1769942507 wait complete. 2024-03-06T07:28:07.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:07.799 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:07.799 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:07.799 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:28:12.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:12.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.936+0000 7f6059a88640 1 Processor -- start 2024-03-06T07:28:12.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.936+0000 7f6059a88640 1 -- start start 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6059a88640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f605415dfe0 0x7f60540cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6059a88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60540d29c0 0x7f60540b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6059a88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60540b3430 con 0x7f60540cf830 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6059a88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60540b31e0 con 0x7f60540d29c0 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6059a88640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60540b3d60 con 0x7f605415dfe0 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6053fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60540d29c0 0x7f60540b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6053fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60540d29c0 0x7f60540b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54814/0 (socket says 172.21.15.77:54814) 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6053fff640 1 -- 172.21.15.77:0/3581409359 learned_addr learned my addr 172.21.15.77:0/3581409359 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6053fff640 1 -- 172.21.15.77:0/3581409359 >> v2:172.21.15.158:3300/0 conn(0x7f605415dfe0 msgr2=0x7f60540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f60537fe640 1 --2- 172.21.15.77:0/3581409359 >> v2:172.21.15.158:3300/0 conn(0x7f605415dfe0 0x7f60540cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6053fff640 1 --2- 172.21.15.77:0/3581409359 >> v2:172.21.15.158:3300/0 conn(0x7f605415dfe0 0x7f60540cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6053fff640 1 -- 172.21.15.77:0/3581409359 >> v1:172.21.15.158:6789/0 conn(0x7f60540cf830 legacy=0x7f60540d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.937+0000 7f6053fff640 1 -- 172.21.15.77:0/3581409359 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60540b3710 con 0x7f60540d29c0 2024-03-06T07:28:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.938+0000 7f6053fff640 1 --2- 172.21.15.77:0/3581409359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60540d29c0 0x7f60540b41f0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f6054067990 tx=0x7f604c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e5fbd9125e3e7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.938+0000 7f6052ffd640 1 -- 172.21.15.77:0/3581409359 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c02ca80 con 0x7f60540d29c0 2024-03-06T07:28:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.938+0000 7f6052ffd640 1 -- 172.21.15.77:0/3581409359 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c02cc20 con 0x7f60540d29c0 2024-03-06T07:28:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.938+0000 7f6052ffd640 1 -- 172.21.15.77:0/3581409359 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c0343f0 con 0x7f60540d29c0 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.939+0000 7f6059a88640 1 -- 172.21.15.77:0/3581409359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60540d29c0 msgr2=0x7f60540b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.939+0000 7f6059a88640 1 --2- 172.21.15.77:0/3581409359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60540d29c0 0x7f60540b41f0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f6054067990 tx=0x7f604c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.939+0000 7f6059a88640 1 -- 172.21.15.77:0/3581409359 shutdown_connections 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.939+0000 7f6059a88640 1 --2- 172.21.15.77:0/3581409359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60540d29c0 0x7f60540b41f0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.939+0000 7f6059a88640 1 --2- 172.21.15.77:0/3581409359 >> v2:172.21.15.158:3300/0 conn(0x7f605415dfe0 0x7f60540cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.939+0000 7f6059a88640 1 -- 172.21.15.77:0/3581409359 >> 172.21.15.77:0/3581409359 conn(0x7f60540d3cd0 msgr2=0x7f60540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.939+0000 7f6059a88640 1 -- 172.21.15.77:0/3581409359 shutdown_connections 2024-03-06T07:28:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.939+0000 7f6059a88640 1 -- 172.21.15.77:0/3581409359 wait complete. 2024-03-06T07:28:12.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.940+0000 7f6059a88640 1 Processor -- start 2024-03-06T07:28:12.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.940+0000 7f6059a88640 1 -- start start 2024-03-06T07:28:12.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6059a88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540cf830 0x7f60540cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6059a88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60540d29c0 0x7f60540cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6059a88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605415dfe0 0x7f605406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6059a88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540d1df0 con 0x7f605415dfe0 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6059a88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540d1c70 con 0x7f60540cf830 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6059a88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605415e3c0 con 0x7f60540d29c0 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605415dfe0 0x7f605406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605415dfe0 0x7f605406ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54830/0 (socket says 172.21.15.77:54830) 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 -- 172.21.15.77:0/1234072631 learned_addr learned my addr 172.21.15.77:0/1234072631 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f60537fe640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540cf830 0x7f60540cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f604affd640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60540d29c0 0x7f60540cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 -- 172.21.15.77:0/1234072631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60540d29c0 msgr2=0x7f60540cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60540d29c0 0x7f60540cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 -- 172.21.15.77:0/1234072631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540cf830 msgr2=0x7f60540cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540cf830 0x7f60540cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 -- 172.21.15.77:0/1234072631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6054159370 con 0x7f605415dfe0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6053fff640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605415dfe0 0x7f605406ccb0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f604c009f40 tx=0x7f604c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6050ff9640 1 -- 172.21.15.77:0/1234072631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c043020 con 0x7f605415dfe0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.941+0000 7f6050ff9640 1 -- 172.21.15.77:0/1234072631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c03e450 con 0x7f605415dfe0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.942+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60540cbd80 con 0x7f605415dfe0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.942+0000 7f6050ff9640 1 -- 172.21.15.77:0/1234072631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c03e860 con 0x7f605415dfe0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.942+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60540cdd60 con 0x7f605415dfe0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.943+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605404f330 con 0x7f605415dfe0 2024-03-06T07:28:12.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.943+0000 7f6050ff9640 1 -- 172.21.15.77:0/1234072631 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f604c046020 con 0x7f605415dfe0 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.943+0000 7f6050ff9640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.944+0000 7f60537fe640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.944+0000 7f6050ff9640 1 -- 172.21.15.77:0/1234072631 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f604c0acc70 con 0x7f605415dfe0 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.944+0000 7f60537fe640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f603406b4d0 0x7f603406d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f604400aa20 tx=0x7f6044008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:12.948+0000 7f6050ff9640 1 -- 172.21.15.77:0/1234072631 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f604c04d030 con 0x7f605415dfe0 2024-03-06T07:28:13.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.182+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60540d2630 con 0x7f605415dfe0 2024-03-06T07:28:13.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.182+0000 7f6050ff9640 1 -- 172.21.15.77:0/1234072631 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f604c07ba90 con 0x7f605415dfe0 2024-03-06T07:28:13.184 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:13.185 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:13.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f603406b4d0 msgr2=0x7f603406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f603406b4d0 0x7f603406d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f604400aa20 tx=0x7f6044008040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605415dfe0 msgr2=0x7f605406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605415dfe0 0x7f605406ccb0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f604c009f40 tx=0x7f604c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 shutdown_connections 2024-03-06T07:28:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605415dfe0 0x7f605406ccb0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60540d29c0 0x7f60540cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 --2- 172.21.15.77:0/1234072631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540cf830 0x7f60540cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 >> 172.21.15.77:0/1234072631 conn(0x7f60540d3cd0 msgr2=0x7f60540b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.184+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 shutdown_connections 2024-03-06T07:28:13.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.185+0000 7f6059a88640 1 -- 172.21.15.77:0/1234072631 wait complete. 2024-03-06T07:28:13.197 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:13.314 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:13.315 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:13.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.415+0000 7fc80db0c640 1 Processor -- start 2024-03-06T07:28:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.415+0000 7fc80db0c640 1 -- start start 2024-03-06T07:28:13.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.415+0000 7fc80db0c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc80815dfe0 0x7fc8080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.415+0000 7fc80db0c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8080cf830 0x7fc8080d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc80db0c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8080b3430 con 0x7fc8080d27f0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc80db0c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8080b31e0 con 0x7fc8080cf830 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc80db0c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8080b3d60 con 0x7fc80815dfe0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc806ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8080cf830 0x7fc8080d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc807fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc8080d27f0 0x7fc8080d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53408/0 (socket says 172.21.15.77:53408) 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc807fff640 1 -- 172.21.15.77:0/3634663140 learned_addr learned my addr 172.21.15.77:0/3634663140 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc8077fe640 1 --2- 172.21.15.77:0/3634663140 >> v2:172.21.15.158:3300/0 conn(0x7fc80815dfe0 0x7fc8080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc806ffd640 1 -- 172.21.15.77:0/3634663140 >> v2:172.21.15.158:3300/0 conn(0x7fc80815dfe0 msgr2=0x7fc8080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc806ffd640 1 --2- 172.21.15.77:0/3634663140 >> v2:172.21.15.158:3300/0 conn(0x7fc80815dfe0 0x7fc8080cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc806ffd640 1 -- 172.21.15.77:0/3634663140 >> v1:172.21.15.77:6789/0 conn(0x7fc8080d27f0 legacy=0x7fc8080d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc8067fc640 1 -- 172.21.15.77:0/3634663140 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6779939 0 0) 0x7fc8080b3430 con 0x7fc8080d27f0 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.416+0000 7fc806ffd640 1 -- 172.21.15.77:0/3634663140 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8080b3710 con 0x7fc8080cf830 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc806ffd640 1 --2- 172.21.15.77:0/3634663140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080cf830 0x7fc8080d22b0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0097a0 tx=0x7fc7fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647334357d06273f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc8067fc640 1 -- 172.21.15.77:0/3634663140 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc035020 con 0x7fc8080cf830 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc8067fc640 1 -- 172.21.15.77:0/3634663140 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc031830 con 0x7fc8080cf830 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc8067fc640 1 -- 172.21.15.77:0/3634663140 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc031c00 con 0x7fc8080cf830 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc80db0c640 1 -- 172.21.15.77:0/3634663140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080cf830 msgr2=0x7fc8080d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3634663140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080cf830 0x7fc8080d22b0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0097a0 tx=0x7fc7fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc80db0c640 1 -- 172.21.15.77:0/3634663140 shutdown_connections 2024-03-06T07:28:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3634663140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080cf830 0x7fc8080d22b0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.417+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3634663140 >> v2:172.21.15.158:3300/0 conn(0x7fc80815dfe0 0x7fc8080cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.418+0000 7fc80db0c640 1 -- 172.21.15.77:0/3634663140 >> 172.21.15.77:0/3634663140 conn(0x7fc8080d3cd0 msgr2=0x7fc8080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.418+0000 7fc80db0c640 1 -- 172.21.15.77:0/3634663140 shutdown_connections 2024-03-06T07:28:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.418+0000 7fc80db0c640 1 -- 172.21.15.77:0/3634663140 wait complete. 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.419+0000 7fc80db0c640 1 Processor -- start 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.419+0000 7fc80db0c640 1 -- start start 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.419+0000 7fc80db0c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8080cf830 0x7fc808161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.419+0000 7fc80db0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080d27f0 0x7fc8080b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.419+0000 7fc80db0c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc80815dfe0 0x7fc8080b9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc80db0c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8080d1df0 con 0x7fc8080d27f0 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc80db0c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8080d1c70 con 0x7fc8080cf830 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc80db0c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80815e3c0 con 0x7fc80815dfe0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080d27f0 0x7fc8080b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080d27f0 0x7fc8080b8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54850/0 (socket says 172.21.15.77:54850) 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 -- 172.21.15.77:0/3609649235 learned_addr learned my addr 172.21.15.77:0/3609649235 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc8077fe640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8080cf830 0x7fc808161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 -- 172.21.15.77:0/3609649235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc80815dfe0 msgr2=0x7fc8080b9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc80815dfe0 0x7fc8080b9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 -- 172.21.15.77:0/3609649235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8080cf830 msgr2=0x7fc808161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8080cf830 0x7fc808161000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 -- 172.21.15.77:0/3609649235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80809f550 con 0x7fc8080d27f0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc8077fe640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8080cf830 0x7fc808161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc806ffd640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080d27f0 0x7fc8080b8b10 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0097a0 tx=0x7fc7fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc7effff640 1 -- 172.21.15.77:0/3609649235 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc041070 con 0x7fc8080d27f0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc7effff640 1 -- 172.21.15.77:0/3609649235 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc004250 con 0x7fc8080d27f0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc80809f830 con 0x7fc8080d27f0 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc7effff640 1 -- 172.21.15.77:0/3609649235 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc035040 con 0x7fc8080d27f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.420+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80809fdf0 con 0x7fc8080d27f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.421+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8080cf5f0 con 0x7fc8080d27f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.422+0000 7fc7effff640 1 -- 172.21.15.77:0/3609649235 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7fc038020 con 0x7fc8080d27f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.422+0000 7fc7effff640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7f406b4d0 0x7fc7f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.422+0000 7fc7effff640 1 -- 172.21.15.77:0/3609649235 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc7fc0abe30 con 0x7fc8080d27f0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.422+0000 7fc8077fe640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7f406b4d0 0x7fc7f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.423+0000 7fc8077fe640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7f406b4d0 0x7fc7f406d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8008890 tx=0x7fc7f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.427+0000 7fc7effff640 1 -- 172.21.15.77:0/3609649235 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8080cf5f0 con 0x7fc8080d27f0 2024-03-06T07:28:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.708+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc80804f330 con 0x7fc8080d27f0 2024-03-06T07:28:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.709+0000 7fc7effff640 1 -- 172.21.15.77:0/3609649235 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fc7fc07ab30 con 0x7fc8080d27f0 2024-03-06T07:28:13.711 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:13.711 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:13.711 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7f406b4d0 msgr2=0x7fc7f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7f406b4d0 0x7fc7f406d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8008890 tx=0x7fc7f8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080d27f0 msgr2=0x7fc8080b8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080d27f0 0x7fc8080b8b10 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0097a0 tx=0x7fc7fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 shutdown_connections 2024-03-06T07:28:13.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7f406b4d0 0x7fc7f406d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8080d27f0 0x7fc8080b8b10 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc80815dfe0 0x7fc8080b9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 --2- 172.21.15.77:0/3609649235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8080cf830 0x7fc808161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 >> 172.21.15.77:0/3609649235 conn(0x7fc8080d3cd0 msgr2=0x7fc80806c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 shutdown_connections 2024-03-06T07:28:13.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.711+0000 7fc80db0c640 1 -- 172.21.15.77:0/3609649235 wait complete. 2024-03-06T07:28:13.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:28:13.839 INFO:teuthology.orchestra.run.smithi077.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:28:13.839 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:28:13.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:13.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.938+0000 7f3a96bb5640 1 Processor -- start 2024-03-06T07:28:13.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.938+0000 7f3a96bb5640 1 -- start start 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.939+0000 7f3a96bb5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3a9015dfe0 0x7f3a900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.939+0000 7f3a96bb5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a900cf830 0x7f3a900d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.939+0000 7f3a96bb5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a900b3430 con 0x7f3a900d27f0 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.939+0000 7f3a96bb5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a900b31e0 con 0x7f3a9015dfe0 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.939+0000 7f3a96bb5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a900b3d60 con 0x7f3a900cf830 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.939+0000 7f3a94951640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3a9015dfe0 0x7f3a900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.939+0000 7f3a95152640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3a900d27f0 0x7f3a900d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53410/0 (socket says 172.21.15.77:53410) 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.939+0000 7f3a95152640 1 -- 172.21.15.77:0/1493028633 learned_addr learned my addr 172.21.15.77:0/1493028633 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.940+0000 7f3a8ffff640 1 --2- 172.21.15.77:0/1493028633 >> v2:172.21.15.158:3300/0 conn(0x7f3a900cf830 0x7f3a900d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.940+0000 7f3a94951640 1 -- 172.21.15.77:0/1493028633 >> v2:172.21.15.158:3300/0 conn(0x7f3a900cf830 msgr2=0x7f3a900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.940+0000 7f3a94951640 1 --2- 172.21.15.77:0/1493028633 >> v2:172.21.15.158:3300/0 conn(0x7f3a900cf830 0x7f3a900d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.940+0000 7f3a94951640 1 -- 172.21.15.77:0/1493028633 >> v1:172.21.15.77:6789/0 conn(0x7f3a900d27f0 legacy=0x7f3a900d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.940+0000 7f3a94951640 1 -- 172.21.15.77:0/1493028633 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a900b3710 con 0x7f3a9015dfe0 2024-03-06T07:28:13.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.940+0000 7f3a8ffff640 1 --2- 172.21.15.77:0/1493028633 >> v2:172.21.15.158:3300/0 conn(0x7f3a900cf830 0x7f3a900d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.940+0000 7f3a94951640 1 --2- 172.21.15.77:0/1493028633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a9015dfe0 0x7f3a900cf2f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f3a840097a0 tx=0x7f3a8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a651354da1d78e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.940+0000 7f3a8f7fe640 1 -- 172.21.15.77:0/1493028633 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84035020 con 0x7f3a9015dfe0 2024-03-06T07:28:13.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a8f7fe640 1 -- 172.21.15.77:0/1493028633 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a84031830 con 0x7f3a9015dfe0 2024-03-06T07:28:13.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a8f7fe640 1 -- 172.21.15.77:0/1493028633 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84031c00 con 0x7f3a9015dfe0 2024-03-06T07:28:13.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a96bb5640 1 -- 172.21.15.77:0/1493028633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a9015dfe0 msgr2=0x7f3a900cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/1493028633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a9015dfe0 0x7f3a900cf2f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f3a840097a0 tx=0x7f3a8402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a96bb5640 1 -- 172.21.15.77:0/1493028633 shutdown_connections 2024-03-06T07:28:13.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/1493028633 >> v2:172.21.15.158:3300/0 conn(0x7f3a900cf830 0x7f3a900d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/1493028633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a9015dfe0 0x7f3a900cf2f0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a96bb5640 1 -- 172.21.15.77:0/1493028633 >> 172.21.15.77:0/1493028633 conn(0x7f3a900d3cd0 msgr2=0x7f3a900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a96bb5640 1 -- 172.21.15.77:0/1493028633 shutdown_connections 2024-03-06T07:28:13.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.941+0000 7f3a96bb5640 1 -- 172.21.15.77:0/1493028633 wait complete. 2024-03-06T07:28:13.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.942+0000 7f3a96bb5640 1 Processor -- start 2024-03-06T07:28:13.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.942+0000 7f3a96bb5640 1 -- start start 2024-03-06T07:28:13.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a96bb5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a900cf830 0x7f3a9015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a96bb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a900d27f0 0x7f3a9006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a96bb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a9015dfe0 0x7f3a9006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a96bb5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a900d1df0 con 0x7f3a900d27f0 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a96bb5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a900d1c70 con 0x7f3a900cf830 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a96bb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9015e3c0 con 0x7f3a9015dfe0 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a8ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a900d27f0 0x7f3a9006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a8ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a900d27f0 0x7f3a9006c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54880/0 (socket says 172.21.15.77:54880) 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a8ffff640 1 -- 172.21.15.77:0/2858418029 learned_addr learned my addr 172.21.15.77:0/2858418029 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a94951640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a900cf830 0x7f3a9015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a95152640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a9015dfe0 0x7f3a9006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a8ffff640 1 -- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a9015dfe0 msgr2=0x7f3a9006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a8ffff640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a9015dfe0 0x7f3a9006ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a8ffff640 1 -- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a900cf830 msgr2=0x7f3a9015c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.943+0000 7f3a8ffff640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a900cf830 0x7f3a9015c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.944+0000 7f3a8ffff640 1 -- 172.21.15.77:0/2858418029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a900b89e0 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.944+0000 7f3a95152640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a9015dfe0 0x7f3a9006ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.944+0000 7f3a8ffff640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a900d27f0 0x7f3a9006c8f0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00c7f0 tx=0x7f3a7c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.944+0000 7f3a8d7fa640 1 -- 172.21.15.77:0/2858418029 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c015070 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.944+0000 7f3a8d7fa640 1 -- 172.21.15.77:0/2858418029 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7c007820 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.944+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a900b8c50 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.944+0000 7f3a8d7fa640 1 -- 172.21.15.77:0/2858418029 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c007bf0 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.944+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a901f7d30 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.945+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9004f330 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.945+0000 7f3a8d7fa640 1 -- 172.21.15.77:0/2858418029 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a7c018070 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.946+0000 7f3a8d7fa640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a7406b500 0x7f3a7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.946+0000 7f3a8d7fa640 1 -- 172.21.15.77:0/2858418029 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f3a7c08a410 con 0x7f3a900d27f0 2024-03-06T07:28:13.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.946+0000 7f3a94951640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a7406b500 0x7f3a7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.946+0000 7f3a94951640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a7406b500 0x7f3a7406d9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3a84036000 tx=0x7f3a84008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:13.954+0000 7f3a8d7fa640 1 -- 172.21.15.77:0/2858418029 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a7c059110 con 0x7f3a900d27f0 2024-03-06T07:28:14.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.170+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a9015d3e0 con 0x7f3a900d27f0 2024-03-06T07:28:14.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.170+0000 7f3a8d7fa640 1 -- 172.21.15.77:0/2858418029 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f3a7c058a10 con 0x7f3a900d27f0 2024-03-06T07:28:14.172 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:14.172 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:14.173 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:14.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a7406b500 msgr2=0x7f3a7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a7406b500 0x7f3a7406d9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3a84036000 tx=0x7f3a84008030 comp rx=0 tx=0).stop 2024-03-06T07:28:14.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a900d27f0 msgr2=0x7f3a9006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a900d27f0 0x7f3a9006c8f0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00c7f0 tx=0x7f3a7c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 shutdown_connections 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a7406b500 0x7f3a7406d9c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a900d27f0 0x7f3a9006c8f0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a9015dfe0 0x7f3a9006ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 --2- 172.21.15.77:0/2858418029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a900cf830 0x7f3a9015c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 >> 172.21.15.77:0/2858418029 conn(0x7f3a900d3cd0 msgr2=0x7f3a900b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 shutdown_connections 2024-03-06T07:28:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.172+0000 7f3a96bb5640 1 -- 172.21.15.77:0/2858418029 wait complete. 2024-03-06T07:28:14.186 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache 2024-03-06T07:28:14.300 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:28:14.301 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:28:14.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:14.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.400+0000 7f835de42640 1 Processor -- start 2024-03-06T07:28:14.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.400+0000 7f835de42640 1 -- start start 2024-03-06T07:28:14.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.401+0000 7f835de42640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f83580d0d60 0x7f83580d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.401+0000 7f835de42640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835815e470 con 0x7f835815e090 2024-03-06T07:28:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.401+0000 7f835de42640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83580d18f0 con 0x7f83580d2920 2024-03-06T07:28:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.401+0000 7f835de42640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83580b3cf0 con 0x7f83580d0d60 2024-03-06T07:28:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.401+0000 7f8357fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f83580d0d60 0x7f83580d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.401+0000 7f83577fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f835815e090 0x7f83580b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53420/0 (socket says 172.21.15.77:53420) 2024-03-06T07:28:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.401+0000 7f83577fe640 1 -- 172.21.15.77:0/2244839823 learned_addr learned my addr 172.21.15.77:0/2244839823 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.402+0000 7f8357fff640 1 -- 172.21.15.77:0/2244839823 >> v1:172.21.15.158:6789/0 conn(0x7f83580d2920 legacy=0x7f83580d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.402+0000 7f8357fff640 1 -- 172.21.15.77:0/2244839823 >> v1:172.21.15.77:6789/0 conn(0x7f835815e090 legacy=0x7f83580b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.402+0000 7f8357fff640 1 -- 172.21.15.77:0/2244839823 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83580d1b90 con 0x7f83580d0d60 2024-03-06T07:28:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.402+0000 7f8357fff640 1 --2- 172.21.15.77:0/2244839823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580d47a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f8350009d90 tx=0x7f835002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e79285f0eb05415 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.402+0000 7f83567fc640 1 -- 172.21.15.77:0/2244839823 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350039070 con 0x7f83580d0d60 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f83567fc640 1 -- 172.21.15.77:0/2244839823 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835002cc60 con 0x7f83580d0d60 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f83567fc640 1 -- 172.21.15.77:0/2244839823 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350034660 con 0x7f83580d0d60 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f835de42640 1 -- 172.21.15.77:0/2244839823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 msgr2=0x7f83580d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f835de42640 1 --2- 172.21.15.77:0/2244839823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580d47a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f8350009d90 tx=0x7f835002beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f835de42640 1 -- 172.21.15.77:0/2244839823 shutdown_connections 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f835de42640 1 --2- 172.21.15.77:0/2244839823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580d47a0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f835de42640 1 -- 172.21.15.77:0/2244839823 >> 172.21.15.77:0/2244839823 conn(0x7f83580d3c80 msgr2=0x7f83580d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f835de42640 1 -- 172.21.15.77:0/2244839823 shutdown_connections 2024-03-06T07:28:14.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.403+0000 7f835de42640 1 -- 172.21.15.77:0/2244839823 wait complete. 2024-03-06T07:28:14.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.405+0000 7f835de42640 1 Processor -- start 2024-03-06T07:28:14.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.405+0000 7f835de42640 1 -- start start 2024-03-06T07:28:14.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f835de42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580b9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f835de42640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580d2920 0x7f83581f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f835de42640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f835815e090 0x7f83581f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f835de42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83580b2df0 con 0x7f83580d0d60 2024-03-06T07:28:14.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f835de42640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83580cee30 con 0x7f83580d2920 2024-03-06T07:28:14.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f835de42640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83580ce800 con 0x7f835815e090 2024-03-06T07:28:14.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f83577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580b9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f83577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580b9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54904/0 (socket says 172.21.15.77:54904) 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.406+0000 7f83577fe640 1 -- 172.21.15.77:0/907307163 learned_addr learned my addr 172.21.15.77:0/907307163 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f8356ffd640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580d2920 0x7f83581f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f8357fff640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f835815e090 0x7f83581f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f83577fe640 1 -- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f835815e090 msgr2=0x7f83581f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f83577fe640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f835815e090 0x7f83581f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f83577fe640 1 -- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580d2920 msgr2=0x7f83581f52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f83577fe640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580d2920 0x7f83581f52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f83577fe640 1 -- 172.21.15.77:0/907307163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83581f4ba0 con 0x7f83580d0d60 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f8357fff640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f835815e090 0x7f83581f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f83577fe640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580b9260 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f834c00c3c0 tx=0x7f834c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f833ffff640 1 -- 172.21.15.77:0/907307163 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c017070 con 0x7f83580d0d60 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f833ffff640 1 -- 172.21.15.77:0/907307163 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c005ce0 con 0x7f83580d0d60 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83581f4e80 con 0x7f83580d0d60 2024-03-06T07:28:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f833ffff640 1 -- 172.21.15.77:0/907307163 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c0072f0 con 0x7f83580d0d60 2024-03-06T07:28:14.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.407+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83580b6d20 con 0x7f83580d0d60 2024-03-06T07:28:14.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.408+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83580b6f20 con 0x7f83580d0d60 2024-03-06T07:28:14.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.409+0000 7f833ffff640 1 -- 172.21.15.77:0/907307163 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f834c01a070 con 0x7f83580d0d60 2024-03-06T07:28:14.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.409+0000 7f833ffff640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.409+0000 7f8356ffd640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.409+0000 7f833ffff640 1 -- 172.21.15.77:0/907307163 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f834c089590 con 0x7f83580d0d60 2024-03-06T07:28:14.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.410+0000 7f8356ffd640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f834406b4d0 0x7f834406d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f83580ca2d0 tx=0x7f8348009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.414+0000 7f833ffff640 1 -- 172.21.15.77:0/907307163 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83580b6f20 con 0x7f83580d0d60 2024-03-06T07:28:14.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.630+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f835804f330 con 0x7f83580d0d60 2024-03-06T07:28:14.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.631+0000 7f833ffff640 1 -- 172.21.15.77:0/907307163 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f834c057d60 con 0x7f83580d0d60 2024-03-06T07:28:14.633 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:14.633 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:14.633 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.632+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f834406b4d0 msgr2=0x7f834406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.632+0000 7f835de42640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f834406b4d0 0x7f834406d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f83580ca2d0 tx=0x7f8348009060 comp rx=0 tx=0).stop 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.632+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 msgr2=0x7f83580b9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.632+0000 7f835de42640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580b9260 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f834c00c3c0 tx=0x7f834c00c880 comp rx=0 tx=0).stop 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.633+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 shutdown_connections 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.633+0000 7f835de42640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f835815e090 0x7f83581f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.633+0000 7f835de42640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f83580d2920 0x7f83581f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.633+0000 7f835de42640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.633+0000 7f835de42640 1 --2- 172.21.15.77:0/907307163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83580d0d60 0x7f83580b9260 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.633+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 >> 172.21.15.77:0/907307163 conn(0x7f83580d3c80 msgr2=0x7f83580cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.633+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 shutdown_connections 2024-03-06T07:28:14.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.633+0000 7f835de42640 1 -- 172.21.15.77:0/907307163 wait complete. 2024-03-06T07:28:14.645 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue 2024-03-06T07:28:14.800 INFO:teuthology.orchestra.run.smithi077.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:28:14.801 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:28:14.801 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:14.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.839+0000 7f7122c37f40 1 Processor -- start 2024-03-06T07:28:14.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.839+0000 7f7122c37f40 1 -- start start 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f7122c37f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71080022a0 0x55b5726bdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f7122c37f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b572779680 0x55b5726ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f7122c37f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b572779b10 0x55b5728293c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f7122c37f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b572666b70 con 0x55b572779680 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f7122c37f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57266aad0 con 0x55b572779b10 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f7122c37f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b572707ce0 con 0x7f71080022a0 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f711f6e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b572779680 0x55b5726ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f711f6e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b572779680 0x55b5726ee2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54930/0 (socket says 172.21.15.77:54930) 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f711f6e1640 1 -- 172.21.15.77:0/4213701001 learned_addr learned my addr 172.21.15.77:0/4213701001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f711f6e1640 1 -- 172.21.15.77:0/4213701001 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71080022a0 msgr2=0x55b5726bdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f711f6e1640 1 --2- 172.21.15.77:0/4213701001 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71080022a0 0x55b5726bdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f711f6e1640 1 -- 172.21.15.77:0/4213701001 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b572779b10 msgr2=0x55b5728293c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f711f6e1640 1 --2- 172.21.15.77:0/4213701001 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b572779b10 0x55b5728293c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.840+0000 7f711f6e1640 1 -- 172.21.15.77:0/4213701001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b572779170 con 0x55b572779680 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f711f6e1640 1 --2- 172.21.15.77:0/4213701001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b572779680 0x55b5726ee2b0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f711400f890 tx=0x7f711400fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3202a278cde79893 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f711e6df640 1 -- 172.21.15.77:0/4213701001 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114005070 con 0x55b572779680 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f711e6df640 1 -- 172.21.15.77:0/4213701001 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711400db70 con 0x55b572779680 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f711e6df640 1 -- 172.21.15.77:0/4213701001 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711401a410 con 0x55b572779680 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 -- 172.21.15.77:0/4213701001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b572779680 msgr2=0x55b5726ee2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 --2- 172.21.15.77:0/4213701001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b572779680 0x55b5726ee2b0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f711400f890 tx=0x7f711400fd50 comp rx=0 tx=0).stop 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 -- 172.21.15.77:0/4213701001 shutdown_connections 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 --2- 172.21.15.77:0/4213701001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b572779680 0x55b5726ee2b0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 --2- 172.21.15.77:0/4213701001 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b572779b10 0x55b5728293c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 --2- 172.21.15.77:0/4213701001 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71080022a0 0x55b5726bdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 -- 172.21.15.77:0/4213701001 >> 172.21.15.77:0/4213701001 conn(0x55b5726eb340 msgr2=0x55b572799c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 -- 172.21.15.77:0/4213701001 shutdown_connections 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.841+0000 7f7122c37f40 1 -- 172.21.15.77:0/4213701001 wait complete. 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.842+0000 7f7122c37f40 1 Processor -- start 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.842+0000 7f7122c37f40 1 -- start start 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.842+0000 7f7122c37f40 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b572779680 0x55b5726be530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.842+0000 7f7122c37f40 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b572779b10 0x55b572778580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.842+0000 7f7122c37f40 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71080022a0 0x55b572796aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.842+0000 7f7122c37f40 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b57277a370 con 0x7f71080022a0 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f7122c37f40 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5726eec00 con 0x55b572779680 2024-03-06T07:28:14.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f7122c37f40 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5726bf1d0 con 0x55b572779b10 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71080022a0 0x55b572796aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71080022a0 0x55b572796aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54938/0 (socket says 172.21.15.77:54938) 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 learned_addr learned my addr 172.21.15.77:0/2587506861 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b572779680 0x55b5726be530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b572779b10 0x55b572778580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b572779b10 msgr2=0x55b572778580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b572779b10 0x55b572778580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b572779680 msgr2=0x55b5726be530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55b572779680 0x55b5726be530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5726bec10 con 0x7f71080022a0 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55b572779b10 0x55b572778580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f71206e3640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71080022a0 0x55b572796aa0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x55b572709a90 tx=0x7f711800d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f711d6dd640 1 -- 172.21.15.77:0/2587506861 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118018070 con 0x7f71080022a0 2024-03-06T07:28:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f711d6dd640 1 -- 172.21.15.77:0/2587506861 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71180040c0 con 0x7f71080022a0 2024-03-06T07:28:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b5726beef0 con 0x7f71080022a0 2024-03-06T07:28:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f711d6dd640 1 -- 172.21.15.77:0/2587506861 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71180030d0 con 0x7f71080022a0 2024-03-06T07:28:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.843+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b57277d7d0 con 0x7f71080022a0 2024-03-06T07:28:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.845+0000 7f711d6dd640 1 -- 172.21.15.77:0/2587506861 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f711801b070 con 0x7f71080022a0 2024-03-06T07:28:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.845+0000 7f711d6dd640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f710c069ee0 0x7f710c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.845+0000 7f711d6dd640 1 -- 172.21.15.77:0/2587506861 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7118087fe0 con 0x7f71080022a0 2024-03-06T07:28:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.845+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f710c069ee0 0x7f710c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.845+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55b57277dcf0 0x55b57277e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.845+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.846+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55b57277dcf0 0x55b57277e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.846+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f710c069ee0 0x7f710c06c3a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7110009810 tx=0x7f7110002fb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.846+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55b57277dcf0 0x55b57277e0d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.846+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 1 ==== osd_op_reply(1 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711401a410 con 0x55b57277dcf0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.846+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f711401ab50 0x7f711400e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.846+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711401ab50 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.847+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f711401ab50 0x7f711400e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.847+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f711401ab50 0x7f711400e8f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.847+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.1 v2:172.21.15.77:6800/1774367160 1 ==== osd_op_reply(2 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711401ab50 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.847+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f711005e970 0x7f7110006440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.847+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.847+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f711005e970 0x7f7110006440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.847+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f711005e970 0x7f7110006440 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.848+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 1 ==== osd_op_reply(3 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711401a410 con 0x7f711005e970 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.848+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f71140193c0 0x7f7114019820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.848+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.848+0000 7f71206e3640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f71140193c0 0x7f7114019820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.848+0000 7f71206e3640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f71140193c0 0x7f7114019820 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.848+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 1 ==== osd_op_reply(4 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.848+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.849+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 2 ==== osd_op_reply(5 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.849+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.849+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 2 ==== osd_op_reply(6 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711005e970 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.849+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.849+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 2 ==== osd_op_reply(7 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.849+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 3 ==== osd_op_reply(8 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 3 ==== osd_op_reply(9 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 3 ==== osd_op_reply(10 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711005e970 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 4 ==== osd_op_reply(11 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711005e970 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f711402b870 0x7f711402d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.850+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711402b870 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.851+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f711402b870 0x7f711402d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.851+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f711402b870 0x7f711402d380 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.851+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.5 v2:172.21.15.158:6820/3476303095 1 ==== osd_op_reply(12 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711402b870 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.851+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.852+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 4 ==== osd_op_reply(13 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.852+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f711400b350 0x7f711400b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.852+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.852+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f711400b350 0x7f711400b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.852+0000 7f711f6e1640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f711400b350 0x7f711400b7d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.853+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 1 ==== osd_op_reply(14 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.853+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711401ab50 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.853+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.1 v2:172.21.15.77:6800/1774367160 2 ==== osd_op_reply(15 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711401ab50 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.853+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.853+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 4 ==== osd_op_reply(16 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.853+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.854+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 2 ==== osd_op_reply(17 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.854+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711402b870 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.854+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.5 v2:172.21.15.158:6820/3476303095 2 ==== osd_op_reply(18 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711402b870 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.854+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.854+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 5 ==== osd_op_reply(19 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711005e970 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.854+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.855+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 5 ==== osd_op_reply(20 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.855+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.855+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 6 ==== osd_op_reply(21 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.855+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.855+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 6 ==== osd_op_reply(22 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711005e970 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.855+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.855+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 5 ==== osd_op_reply(23 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.856+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711401ab50 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.856+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.1 v2:172.21.15.77:6800/1774367160 3 ==== osd_op_reply(24 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711401ab50 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.856+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f711006d960 0x7f7110079440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.856+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711006d960 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.856+0000 7f71206e3640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f711006d960 0x7f7110079440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.857+0000 7f71206e3640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f711006d960 0x7f7110079440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.857+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.6 v2:172.21.15.158:6801/2971141295 1 ==== osd_op_reply(25 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f711006d960 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.857+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.857+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 6 ==== osd_op_reply(26 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.857+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711402b870 2024-03-06T07:28:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.5 v2:172.21.15.158:6820/3476303095 3 ==== osd_op_reply(27 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711402b870 2024-03-06T07:28:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 7 ==== osd_op_reply(28 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f71206e3640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f711801d060 0x7f7118055670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711801d060 2024-03-06T07:28:14.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f711801d060 0x7f7118055670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f711fee2640 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f711801d060 0x7f7118055670 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.2 v2:172.21.15.77:6824/107698445 1 ==== osd_op_reply(29 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711801d060 2024-03-06T07:28:14.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.858+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711006d960 2024-03-06T07:28:14.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.859+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.6 v2:172.21.15.158:6801/2971141295 2 ==== osd_op_reply(30 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f711006d960 2024-03-06T07:28:14.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.859+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711401ab50 2024-03-06T07:28:14.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.859+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.1 v2:172.21.15.77:6800/1774367160 4 ==== osd_op_reply(31 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711401ab50 2024-03-06T07:28:14.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.859+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.860+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 3 ==== osd_op_reply(32 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.860+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.860+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 8 ==== osd_op_reply(33 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.860+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.860+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 9 ==== osd_op_reply(34 [pgnls start_epoch 113 out=49b] v107'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.860+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.860+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 7 ==== osd_op_reply(35 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711005e970 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.860+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711401ab50 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.861+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.1 v2:172.21.15.77:6800/1774367160 5 ==== osd_op_reply(36 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711401ab50 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.861+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.861+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 8 ==== osd_op_reply(37 [pgnls start_epoch 113 out=49b] v107'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711005e970 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.861+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.861+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 4 ==== osd_op_reply(38 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.861+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711402b870 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.862+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.5 v2:172.21.15.158:6820/3476303095 4 ==== osd_op_reply(39 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711402b870 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.862+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711402b870 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.862+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.5 v2:172.21.15.158:6820/3476303095 5 ==== osd_op_reply(40 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711402b870 2024-03-06T07:28:14.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.862+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711006d960 2024-03-06T07:28:14.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.862+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.6 v2:172.21.15.158:6801/2971141295 3 ==== osd_op_reply(41 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f711006d960 2024-03-06T07:28:14.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.862+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711401ab50 2024-03-06T07:28:14.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.863+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.1 v2:172.21.15.77:6800/1774367160 6 ==== osd_op_reply(42 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711401ab50 2024-03-06T07:28:14.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.863+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711402b870 2024-03-06T07:28:14.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.863+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.5 v2:172.21.15.158:6820/3476303095 6 ==== osd_op_reply(43 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711402b870 2024-03-06T07:28:14.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.863+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.863+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 5 ==== osd_op_reply(44 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.863+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.864+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 7 ==== osd_op_reply(45 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.864+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.864+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 8 ==== osd_op_reply(46 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.864+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.864+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 9 ==== osd_op_reply(47 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.864+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.865+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 10 ==== osd_op_reply(48 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.865+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.865+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 6 ==== osd_op_reply(49 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.865+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.865+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 7 ==== osd_op_reply(50 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.865+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711005e970 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.866+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.3 v2:172.21.15.77:6816/461341877 9 ==== osd_op_reply(51 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711005e970 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.866+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x55b57277dcf0 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.866+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.7 v2:172.21.15.158:6816/3281717214 11 ==== osd_op_reply(52 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x55b57277dcf0 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.866+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.866+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 10 ==== osd_op_reply(53 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.866+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.867+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 8 ==== osd_op_reply(54 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.867+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.867+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 9 ==== osd_op_reply(55 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.867+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711801d060 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.867+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.2 v2:172.21.15.77:6824/107698445 2 ==== osd_op_reply(56 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711801d060 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.867+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.867+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 11 ==== osd_op_reply(57 [pgnls start_epoch 113 out=49b] v113'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.867+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.868+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 10 ==== osd_op_reply(58 [pgnls start_epoch 113 out=49b] v107'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.868+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711006d960 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.868+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.6 v2:172.21.15.158:6801/2971141295 4 ==== osd_op_reply(59 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f711006d960 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.868+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.868+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 12 ==== osd_op_reply(60 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.868+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711401ab50 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.869+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 <== osd.1 v2:172.21.15.77:6800/1774367160 7 ==== osd_op_reply(61 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711005a040 con 0x7f711401ab50 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.869+0000 7f711fee2640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711400b350 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.869+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 <== osd.4 v2:172.21.15.158:6800/2421130833 11 ==== osd_op_reply(62 [pgnls start_epoch 113 out=49b] v113'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7114016040 con 0x7f711400b350 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.869+0000 7f711f6e1640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f711006d960 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.869+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.6 v2:172.21.15.158:6801/2971141295 5 ==== osd_op_reply(63 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f711006d960 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.869+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) v9 -- 0x55b57277e610 con 0x7f71140193c0 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 <== osd.0 v2:172.21.15.77:6808/3119591583 13 ==== osd_op_reply(64 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f711801a030 con 0x7f71140193c0 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f71140193c0 msgr2=0x7f7114019820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f71140193c0 0x7f7114019820 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f711401ab50 msgr2=0x7f711400e8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7f711401ab50 0x7f711400e8f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f711801d060 msgr2=0x7f7118055670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7f711801d060 0x7f7118055670 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f711005e970 msgr2=0x7f7110006440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 reap_dead start 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f711005e970 0x7f7110006440 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f711400b350 msgr2=0x7f711400b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f711400b350 0x7f711400b7d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f711402b870 msgr2=0x7f711402d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f711402b870 0x7f711402d380 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f711006d960 msgr2=0x7f7110079440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7f711006d960 0x7f7110079440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55b57277dcf0 msgr2=0x55b57277e0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x55b57277dcf0 0x55b57277e0d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f71206e3640 1 -- 172.21.15.77:0/2587506861 reap_dead start 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f710c069ee0 msgr2=0x7f710c06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f710c069ee0 0x7f710c06c3a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7110009810 tx=0x7f7110002fb0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71080022a0 msgr2=0x55b572796aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71080022a0 0x55b572796aa0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x55b572709a90 tx=0x7f711800d530 comp rx=0 tx=0).stop 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 shutdown_connections 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f710c069ee0 0x7f710c06c3a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 --2- 172.21.15.77:0/2587506861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71080022a0 0x55b572796aa0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 >> 172.21.15.77:0/2587506861 conn(0x55b5726eb340 msgr2=0x55b5726ee2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 shutdown_connections 2024-03-06T07:28:14.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:14.870+0000 7f7122c37f40 1 -- 172.21.15.77:0/2587506861 wait complete. 2024-03-06T07:28:14.881 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:28:14.882 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:15.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.053+0000 7fd474640640 1 Processor -- start 2024-03-06T07:28:15.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.053+0000 7fd474640640 1 -- start start 2024-03-06T07:28:15.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd474640640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd46c0d22b0 0x7fd46c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd474640640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0b41f0 0x7fd46c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd474640640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c0b3430 con 0x7fd46c15dfe0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd474640640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46c0d2c50 con 0x7fd46c0d22b0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd474640640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0b31e0 con 0x7fd46c0b41f0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd471bdb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd46c0d22b0 0x7fd46c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd471bdb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd46c0d22b0 0x7fd46c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54948/0 (socket says 172.21.15.77:54948) 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd471bdb640 1 -- 172.21.15.77:0/2278700538 learned_addr learned my addr 172.21.15.77:0/2278700538 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd472bdd640 1 --2- 172.21.15.77:0/2278700538 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0b41f0 0x7fd46c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd471bdb640 1 -- 172.21.15.77:0/2278700538 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0b41f0 msgr2=0x7fd46c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd471bdb640 1 --2- 172.21.15.77:0/2278700538 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0b41f0 0x7fd46c0b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd471bdb640 1 -- 172.21.15.77:0/2278700538 >> v1:172.21.15.158:6789/0 conn(0x7fd46c15dfe0 legacy=0x7fd46c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.054+0000 7fd471bdb640 1 -- 172.21.15.77:0/2278700538 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0b3710 con 0x7fd46c0d22b0 2024-03-06T07:28:15.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.055+0000 7fd471bdb640 1 --2- 172.21.15.77:0/2278700538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c0d22b0 0x7fd46c0d2710 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009870 tx=0x7fd45c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccc79bdb56d8f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.055+0000 7fd4713da640 1 -- 172.21.15.77:0/2278700538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c035020 con 0x7fd46c0d22b0 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.055+0000 7fd4713da640 1 -- 172.21.15.77:0/2278700538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c031640 con 0x7fd46c0d22b0 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.055+0000 7fd4713da640 1 -- 172.21.15.77:0/2278700538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c02d040 con 0x7fd46c0d22b0 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.056+0000 7fd474640640 1 -- 172.21.15.77:0/2278700538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c0d22b0 msgr2=0x7fd46c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.056+0000 7fd474640640 1 --2- 172.21.15.77:0/2278700538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c0d22b0 0x7fd46c0d2710 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009870 tx=0x7fd45c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.056+0000 7fd474640640 1 -- 172.21.15.77:0/2278700538 shutdown_connections 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.056+0000 7fd474640640 1 --2- 172.21.15.77:0/2278700538 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0b41f0 0x7fd46c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.056+0000 7fd474640640 1 --2- 172.21.15.77:0/2278700538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c0d22b0 0x7fd46c0d2710 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.056+0000 7fd474640640 1 -- 172.21.15.77:0/2278700538 >> 172.21.15.77:0/2278700538 conn(0x7fd46c0d3cd0 msgr2=0x7fd46c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.056+0000 7fd474640640 1 -- 172.21.15.77:0/2278700538 shutdown_connections 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.056+0000 7fd474640640 1 -- 172.21.15.77:0/2278700538 wait complete. 2024-03-06T07:28:15.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.057+0000 7fd474640640 1 Processor -- start 2024-03-06T07:28:15.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.057+0000 7fd474640640 1 -- start start 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd474640640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd46c0b41f0 0x7fd46c152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd474640640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0d22b0 0x7fd46c0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd474640640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c15dfe0 0x7fd46c160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd474640640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0cfa00 con 0x7fd46c15dfe0 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd474640640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0b3d60 con 0x7fd46c0b41f0 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd474640640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0d1c70 con 0x7fd46c0d22b0 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c15dfe0 0x7fd46c160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c15dfe0 0x7fd46c160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54958/0 (socket says 172.21.15.77:54958) 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 -- 172.21.15.77:0/3533539032 learned_addr learned my addr 172.21.15.77:0/3533539032 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd471bdb640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0d22b0 0x7fd46c0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd4723dc640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd46c0b41f0 0x7fd46c152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 -- 172.21.15.77:0/3533539032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0d22b0 msgr2=0x7fd46c0b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0d22b0 0x7fd46c0b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 -- 172.21.15.77:0/3533539032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd46c0b41f0 msgr2=0x7fd46c152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd46c0b41f0 0x7fd46c152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 -- 172.21.15.77:0/3533539032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0b7210 con 0x7fd46c15dfe0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd472bdd640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c15dfe0 0x7fd46c160cb0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fd46400cd70 tx=0x7fd46400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd462ffd640 1 -- 172.21.15.77:0/3533539032 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464003b90 con 0x7fd46c15dfe0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd462ffd640 1 -- 172.21.15.77:0/3533539032 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464003d30 con 0x7fd46c15dfe0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.058+0000 7fd462ffd640 1 -- 172.21.15.77:0/3533539032 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464004240 con 0x7fd46c15dfe0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.059+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46c0b74f0 con 0x7fd46c15dfe0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.059+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46c0b7ab0 con 0x7fd46c15dfe0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.059+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46c04f330 con 0x7fd46c15dfe0 2024-03-06T07:28:15.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.060+0000 7fd462ffd640 1 -- 172.21.15.77:0/3533539032 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4640043e0 con 0x7fd46c15dfe0 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.060+0000 7fd462ffd640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45006b4d0 0x7fd45006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.060+0000 7fd462ffd640 1 -- 172.21.15.77:0/3533539032 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd46408af10 con 0x7fd46c15dfe0 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.060+0000 7fd4723dc640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45006b4d0 0x7fd45006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.061+0000 7fd4723dc640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45006b4d0 0x7fd45006d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd46800a750 tx=0x7fd468008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.065+0000 7fd462ffd640 1 -- 172.21.15.77:0/3533539032 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd46401d340 con 0x7fd46c15dfe0 2024-03-06T07:28:15.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.280+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd46c0ceb20 con 0x7fd46c15dfe0 2024-03-06T07:28:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.281+0000 7fd462ffd640 1 -- 172.21.15.77:0/3533539032 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fd464059c00 con 0x7fd46c15dfe0 2024-03-06T07:28:15.283 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:15.283 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45006b4d0 msgr2=0x7fd45006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45006b4d0 0x7fd45006d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd46800a750 tx=0x7fd468008040 comp rx=0 tx=0).stop 2024-03-06T07:28:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c15dfe0 msgr2=0x7fd46c160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c15dfe0 0x7fd46c160cb0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fd46400cd70 tx=0x7fd46400a660 comp rx=0 tx=0).stop 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 shutdown_connections 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd45006b4d0 0x7fd45006d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd46c15dfe0 0x7fd46c160cb0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd46c0d22b0 0x7fd46c0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 --2- 172.21.15.77:0/3533539032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd46c0b41f0 0x7fd46c152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 >> 172.21.15.77:0/3533539032 conn(0x7fd46c0d3cd0 msgr2=0x7fd46c06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 shutdown_connections 2024-03-06T07:28:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.283+0000 7fd474640640 1 -- 172.21.15.77:0/3533539032 wait complete. 2024-03-06T07:28:15.299 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:28:15.414 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:28:15.416 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:28:15.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:15.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.519+0000 7f20fd666640 1 Processor -- start 2024-03-06T07:28:15.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.519+0000 7f20fd666640 1 -- start start 2024-03-06T07:28:15.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.519+0000 7f20fd666640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20f80b4170 0x7f20f80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20fd666640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80d4760 0x7f20f80d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20fd666640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f20f80b4dd0 0x7f20f80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20fd666640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20f80d4bf0 con 0x7f20f80b4170 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20fd666640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20f80ce9e0 con 0x7f20f80b4dd0 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20fd666640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80d25d0 con 0x7f20f80d4760 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f6ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20f80b4170 0x7f20f80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f6ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20f80b4170 0x7f20f80b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54960/0 (socket says 172.21.15.77:54960) 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f6ffd640 1 -- 172.21.15.77:0/2411424 learned_addr learned my addr 172.21.15.77:0/2411424 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20effff640 1 --2- 172.21.15.77:0/2411424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80d4760 0x7f20f80d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f77fe640 1 --2- 172.21.15.77:0/2411424 >> v2:172.21.15.158:3300/0 conn(0x7f20f80b4dd0 0x7f20f80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f6ffd640 1 -- 172.21.15.77:0/2411424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80d4760 msgr2=0x7f20f80d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f6ffd640 1 --2- 172.21.15.77:0/2411424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80d4760 0x7f20f80d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f6ffd640 1 -- 172.21.15.77:0/2411424 >> v2:172.21.15.158:3300/0 conn(0x7f20f80b4dd0 msgr2=0x7f20f80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f6ffd640 1 --2- 172.21.15.77:0/2411424 >> v2:172.21.15.158:3300/0 conn(0x7f20f80b4dd0 0x7f20f80d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f6ffd640 1 -- 172.21.15.77:0/2411424 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f80d2870 con 0x7f20f80b4170 2024-03-06T07:28:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20effff640 1 --2- 172.21.15.77:0/2411424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80d4760 0x7f20f80d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.520+0000 7f20f77fe640 1 --2- 172.21.15.77:0/2411424 >> v2:172.21.15.158:3300/0 conn(0x7f20f80b4dd0 0x7f20f80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.521+0000 7f20f6ffd640 1 --2- 172.21.15.77:0/2411424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f80b6a90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f20f80cffb0 tx=0x7f20e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd68bb29582c204 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.521+0000 7f20f67fc640 1 -- 172.21.15.77:0/2411424 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8035020 con 0x7f20f80b4170 2024-03-06T07:28:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.521+0000 7f20f67fc640 1 -- 172.21.15.77:0/2411424 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e8031870 con 0x7f20f80b4170 2024-03-06T07:28:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.521+0000 7f20f67fc640 1 -- 172.21.15.77:0/2411424 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8031c40 con 0x7f20f80b4170 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.521+0000 7f20fd666640 1 -- 172.21.15.77:0/2411424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 msgr2=0x7f20f80b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.521+0000 7f20fd666640 1 --2- 172.21.15.77:0/2411424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f80b6a90 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f20f80cffb0 tx=0x7f20e802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.522+0000 7f20fd666640 1 -- 172.21.15.77:0/2411424 shutdown_connections 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.522+0000 7f20fd666640 1 --2- 172.21.15.77:0/2411424 >> v2:172.21.15.158:3300/0 conn(0x7f20f80b4dd0 0x7f20f80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.522+0000 7f20fd666640 1 --2- 172.21.15.77:0/2411424 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80d4760 0x7f20f80d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.522+0000 7f20fd666640 1 --2- 172.21.15.77:0/2411424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f80b6a90 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.522+0000 7f20fd666640 1 -- 172.21.15.77:0/2411424 >> 172.21.15.77:0/2411424 conn(0x7f20f80d3c40 msgr2=0x7f20f80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.522+0000 7f20fd666640 1 -- 172.21.15.77:0/2411424 shutdown_connections 2024-03-06T07:28:15.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.522+0000 7f20fd666640 1 -- 172.21.15.77:0/2411424 wait complete. 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.523+0000 7f20fd666640 1 Processor -- start 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.523+0000 7f20fd666640 1 -- start start 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.523+0000 7f20fd666640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f814e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.523+0000 7f20fd666640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80b4dd0 0x7f20f80ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.523+0000 7f20fd666640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20f80ca540 0x7f20f815d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.523+0000 7f20fd666640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80b47d0 con 0x7f20f80b4170 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.523+0000 7f20fd666640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80b4550 con 0x7f20f80ca540 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.523+0000 7f20fd666640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80b4950 con 0x7f20f80b4dd0 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f814e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f814e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54974/0 (socket says 172.21.15.77:54974) 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 -- 172.21.15.77:0/1008384858 learned_addr learned my addr 172.21.15.77:0/1008384858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20effff640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80b4dd0 0x7f20f80ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f77fe640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20f80ca540 0x7f20f815d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 -- 172.21.15.77:0/1008384858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80b4dd0 msgr2=0x7f20f80ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80b4dd0 0x7f20f80ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 -- 172.21.15.77:0/1008384858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20f80ca540 msgr2=0x7f20f815d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20f80ca540 0x7f20f815d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 -- 172.21.15.77:0/1008384858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f815a8f0 con 0x7f20f80b4170 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20f6ffd640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f814e010 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f20e8000c00 tx=0x7f20e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20ef7fe640 1 -- 172.21.15.77:0/1008384858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8035040 con 0x7f20f80b4170 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20ef7fe640 1 -- 172.21.15.77:0/1008384858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e8030d00 con 0x7f20f80b4170 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f815e040 con 0x7f20f80b4170 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20ef7fe640 1 -- 172.21.15.77:0/1008384858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80456a0 con 0x7f20f80b4170 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.524+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f815e600 con 0x7f20f80b4170 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.525+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f80ca950 con 0x7f20f80b4170 2024-03-06T07:28:15.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.525+0000 7f20ef7fe640 1 -- 172.21.15.77:0/1008384858 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20e8038020 con 0x7f20f80b4170 2024-03-06T07:28:15.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.526+0000 7f20ef7fe640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f20d806b4d0 0x7f20d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.526+0000 7f20ef7fe640 1 -- 172.21.15.77:0/1008384858 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f20e80ac5f0 con 0x7f20f80b4170 2024-03-06T07:28:15.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.526+0000 7f20effff640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f20d806b4d0 0x7f20d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.526+0000 7f20effff640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f20d806b4d0 0x7f20d806d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f20f815a580 tx=0x7f20e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.530+0000 7f20ef7fe640 1 -- 172.21.15.77:0/1008384858 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20f80ca950 con 0x7f20f80b4170 2024-03-06T07:28:15.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.745+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20f815e800 con 0x7f20f80b4170 2024-03-06T07:28:15.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.745+0000 7f20ef7fe640 1 -- 172.21.15.77:0/1008384858 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f20e802f320 con 0x7f20f80b4170 2024-03-06T07:28:15.747 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:15.747 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:15.748 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:15.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f20d806b4d0 msgr2=0x7f20d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f20d806b4d0 0x7f20d806d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f20f815a580 tx=0x7f20e0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 msgr2=0x7f20f814e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f814e010 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f20e8000c00 tx=0x7f20e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 shutdown_connections 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f20f80ca540 0x7f20f815d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f20f80b4dd0 0x7f20f80ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f20d806b4d0 0x7f20d806d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 --2- 172.21.15.77:0/1008384858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20f80b4170 0x7f20f814e010 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 >> 172.21.15.77:0/1008384858 conn(0x7f20f80d3c40 msgr2=0x7f20f80b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 shutdown_connections 2024-03-06T07:28:15.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:15.747+0000 7f20fd666640 1 -- 172.21.15.77:0/1008384858 wait complete. 2024-03-06T07:28:15.760 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:15.878 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:15.878 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:15.878 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:15.878 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:15.915 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:15.978 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:15.978 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:28:15.978 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:15.993 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:16.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.131+0000 7f5df989d640 1 Processor -- start 2024-03-06T07:28:16.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.131+0000 7f5df989d640 1 -- start start 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.132+0000 7f5df989d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5df40d22b0 0x7f5df40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.132+0000 7f5df989d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df40b3430 con 0x7f5df40b41f0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.132+0000 7f5df989d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df40b3710 con 0x7f5df415dfe0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.132+0000 7f5df989d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df40d2c50 con 0x7f5df40d22b0 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.132+0000 7f5df37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5df40b41f0 0x7f5df40b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53440/0 (socket says 172.21.15.77:53440) 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.132+0000 7f5df37fe640 1 -- 172.21.15.77:0/825693240 learned_addr learned my addr 172.21.15.77:0/825693240 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:16.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5debfff640 1 --2- 172.21.15.77:0/825693240 >> v2:172.21.15.158:3300/0 conn(0x7f5df40d22b0 0x7f5df40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070618957 0 0) 0x7f5df40b3430 con 0x7f5df40b41f0 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de0003520 con 0x7f5df40b41f0 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045863552 0 0) 0x7f5de0003520 con 0x7f5df40b41f0 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 >> v2:172.21.15.158:3300/0 conn(0x7f5df40d22b0 msgr2=0x7f5df40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 --2- 172.21.15.77:0/825693240 >> v2:172.21.15.158:3300/0 conn(0x7f5df40d22b0 0x7f5df40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 >> v1:172.21.15.158:6789/0 conn(0x7f5df415dfe0 legacy=0x7f5df40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df40b39f0 con 0x7f5df40b41f0 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f5dec0019f0 con 0x7f5df40b41f0 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 >> v1:172.21.15.77:6789/0 conn(0x7f5df40b41f0 legacy=0x7f5df40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5de00039b0 0x7f5de0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5de0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5de00051c0 0x7f5de000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de000c030 con 0x7f5de0004800 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de000c1e0 con 0x7f5de00051c0 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de000c3e0 con 0x7f5de00039b0 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5debfff640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5de0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df37fe640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5de00051c0 0x7f5de000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df2ffd640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5de00039b0 0x7f5de0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5debfff640 1 -- 172.21.15.77:0/825693240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5de00039b0 msgr2=0x7f5de0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5debfff640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5de00039b0 0x7f5de0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5debfff640 1 -- 172.21.15.77:0/825693240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5de00051c0 msgr2=0x7f5de000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5debfff640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5de00051c0 0x7f5de000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5debfff640 1 -- 172.21.15.77:0/825693240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de0003520 con 0x7f5de0004800 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.133+0000 7f5df37fe640 1 -- 172.21.15.77:0/825693240 reap_dead start 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.134+0000 7f5debfff640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5de0004c80 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009ac0 tx=0x7f5de0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ddae4eb84a9d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.134+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc039070 con 0x7f5de0004800 2024-03-06T07:28:16.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.134+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc02c6b0 con 0x7f5de0004800 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.134+0000 7f5df27fc640 1 -- 172.21.15.77:0/825693240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc009690 con 0x7f5de0004800 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.135+0000 7f5df989d640 1 -- 172.21.15.77:0/825693240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 msgr2=0x7f5de0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.135+0000 7f5df989d640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5de0004c80 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009ac0 tx=0x7f5de0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.135+0000 7f5df989d640 1 -- 172.21.15.77:0/825693240 shutdown_connections 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.135+0000 7f5df989d640 1 --2- 172.21.15.77:0/825693240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5de0004c80 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.135+0000 7f5df989d640 1 -- 172.21.15.77:0/825693240 >> 172.21.15.77:0/825693240 conn(0x7f5df40d3cd0 msgr2=0x7f5df40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.135+0000 7f5df989d640 1 -- 172.21.15.77:0/825693240 shutdown_connections 2024-03-06T07:28:16.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.135+0000 7f5df989d640 1 -- 172.21.15.77:0/825693240 wait complete. 2024-03-06T07:28:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.136+0000 7f5df989d640 1 Processor -- start 2024-03-06T07:28:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.136+0000 7f5df989d640 1 -- start start 2024-03-06T07:28:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df989d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5df414ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df989d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5df414d830 0x7f5df414b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df989d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5df414bce0 0x7f5df40b9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df989d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df40b3d60 con 0x7f5de0004800 2024-03-06T07:28:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df989d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df40b31e0 con 0x7f5df414bce0 2024-03-06T07:28:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df989d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df415e3c0 con 0x7f5df414d830 2024-03-06T07:28:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5df414ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5df414ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54990/0 (socket says 172.21.15.77:54990) 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1908802822 learned_addr learned my addr 172.21.15.77:0/1908802822 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1908802822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5df414d830 msgr2=0x7f5df414b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5df414d830 0x7f5df414b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1908802822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5df414bce0 msgr2=0x7f5df40b9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5df414bce0 0x7f5df40b9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 -- 172.21.15.77:0/1908802822 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df40b8880 con 0x7f5de0004800 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.137+0000 7f5df2ffd640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5df414ae90 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f5de400f540 tx=0x7f5de400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.138+0000 7f5df0ff9640 1 -- 172.21.15.77:0/1908802822 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de40052e0 con 0x7f5de0004800 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.138+0000 7f5df0ff9640 1 -- 172.21.15.77:0/1908802822 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de4018070 con 0x7f5de0004800 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.138+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df414e280 con 0x7f5de0004800 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.138+0000 7f5df0ff9640 1 -- 172.21.15.77:0/1908802822 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4003830 con 0x7f5de0004800 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.138+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df40b8b00 con 0x7f5de0004800 2024-03-06T07:28:16.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.139+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df404f330 con 0x7f5de0004800 2024-03-06T07:28:16.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.139+0000 7f5df0ff9640 1 -- 172.21.15.77:0/1908802822 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5de40039d0 con 0x7f5de0004800 2024-03-06T07:28:16.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.140+0000 7f5df0ff9640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dd406b4d0 0x7f5dd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.140+0000 7f5df0ff9640 1 -- 172.21.15.77:0/1908802822 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f5de4089fb0 con 0x7f5de0004800 2024-03-06T07:28:16.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.140+0000 7f5debfff640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dd406b4d0 0x7f5dd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.140+0000 7f5debfff640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dd406b4d0 0x7f5dd406d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc036000 tx=0x7f5ddc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.144+0000 7f5df0ff9640 1 -- 172.21.15.77:0/1908802822 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5df40b8880 con 0x7f5de0004800 2024-03-06T07:28:16.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.322+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5df414b3d0 con 0x7f5de0004800 2024-03-06T07:28:16.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.323+0000 7f5df0ff9640 1 -- 172.21.15.77:0/1908802822 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+17027 (secure 0 0 0) 0x7f5de4058d40 con 0x7f5de0004800 2024-03-06T07:28:16.325 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:16.325 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":113,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:48.616890+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.34","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6835/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6834/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6837/3247585937":"2024-03-07T07:27:18.550938+0000","172.21.15.77:6836/3247585937":"2024-03-07T07:27:18.550938+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:16.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dd406b4d0 msgr2=0x7f5dd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dd406b4d0 0x7f5dd406d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc036000 tx=0x7f5ddc02e000 comp rx=0 tx=0).stop 2024-03-06T07:28:16.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 msgr2=0x7f5df414ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5df414ae90 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f5de400f540 tx=0x7f5de400fa00 comp rx=0 tx=0).stop 2024-03-06T07:28:16.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 shutdown_connections 2024-03-06T07:28:16.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5df414bce0 0x7f5df40b9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5df414d830 0x7f5df414b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5dd406b4d0 0x7f5dd406d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 --2- 172.21.15.77:0/1908802822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5de0004800 0x7f5df414ae90 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 >> 172.21.15.77:0/1908802822 conn(0x7f5df40d3cd0 msgr2=0x7f5df406c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 shutdown_connections 2024-03-06T07:28:16.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.326+0000 7f5df989d640 1 -- 172.21.15.77:0/1908802822 wait complete. 2024-03-06T07:28:16.339 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:03.116010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:03.812681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:16.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:16.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.441+0000 7fa0b5cc3640 1 Processor -- start 2024-03-06T07:28:16.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.441+0000 7fa0b5cc3640 1 -- start start 2024-03-06T07:28:16.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0b5cc3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0b00cf0d0 0x7fa0b006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0b5cc3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00b3320 con 0x7fa0b015a220 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0b5cc3640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00cf560 con 0x7fa0b00719e0 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0b5cc3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00728d0 con 0x7fa0b00cf0d0 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0af7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa0b015a220 0x7fa0b00712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53448/0 (socket says 172.21.15.77:53448) 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0af7fe640 1 -- 172.21.15.77:0/3557983639 learned_addr learned my addr 172.21.15.77:0/3557983639 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301490038 0 0) 0x7fa0b00b3320 con 0x7fa0b015a220 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0affff640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0b00cf0d0 0x7fa0b006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa098004630 con 0x7fa0b015a220 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257952651 0 0) 0x7fa098004630 con 0x7fa0b015a220 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0b00cf0d0 msgr2=0x7fa0b006d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0b00cf0d0 0x7fa0b006d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 >> v1:172.21.15.158:6789/0 conn(0x7fa0b00719e0 legacy=0x7fa0b006c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00b3590 con 0x7fa0b015a220 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fa0a0001670 con 0x7fa0b015a220 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 >> v1:172.21.15.77:6789/0 conn(0x7fa0b015a220 legacy=0x7fa0b00712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa098004af0 0x7fa098004f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa098005910 0x7fa098005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0b00cf0d0 0x7fa098006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00728d0 con 0x7fa098004af0 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09800cc00 con 0x7fa098005910 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.442+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09800ce00 con 0x7fa0b00cf0d0 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.443+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa098004af0 0x7fa098004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.443+0000 7fa0aeffd640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa098005910 0x7fa098005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.443+0000 7fa0affff640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0b00cf0d0 0x7fa098006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.443+0000 7fa0af7fe640 1 -- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0b00cf0d0 msgr2=0x7fa098006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.443+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa0b00cf0d0 0x7fa098006310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.443+0000 7fa0af7fe640 1 -- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa098005910 msgr2=0x7fa098005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.443+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa098005910 0x7fa098005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.443+0000 7fa0af7fe640 1 -- 172.21.15.77:0/3557983639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa098004630 con 0x7fa098004af0 2024-03-06T07:28:16.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa098004af0 0x7fa098004f10 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000c240 tx=0x7fa098006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3dde8668c2389 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0007d10 con 0x7fa098004af0 2024-03-06T07:28:16.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a0030070 con 0x7fa098004af0 2024-03-06T07:28:16.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0ae7fc640 1 -- 172.21.15.77:0/3557983639 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a002cd80 con 0x7fa098004af0 2024-03-06T07:28:16.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/3557983639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa098004af0 msgr2=0x7fa098004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0b5cc3640 1 --2- 172.21.15.77:0/3557983639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa098004af0 0x7fa098004f10 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000c240 tx=0x7fa098006940 comp rx=0 tx=0).stop 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0affff640 1 -- 172.21.15.77:0/3557983639 reap_dead start 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/3557983639 shutdown_connections 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/3557983639 >> 172.21.15.77:0/3557983639 conn(0x7fa0b006b190 msgr2=0x7fa0b006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.444+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/3557983639 shutdown_connections 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.445+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/3557983639 wait complete. 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.446+0000 7fa0b5cc3640 1 Processor -- start 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.446+0000 7fa0b5cc3640 1 -- start start 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.446+0000 7fa0b5cc3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa098004af0 0x7fa0b014f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.446+0000 7fa0b5cc3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c47e0 0x7fa0b00c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.446+0000 7fa0b5cc3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00af670 0x7fa0b00afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0b5cc3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0072530 con 0x7fa0b00af670 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0b5cc3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0072030 con 0x7fa0b00c47e0 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0b5cc3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00726b0 con 0x7fa098004af0 2024-03-06T07:28:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00af670 0x7fa0b00afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00af670 0x7fa0b00afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55016/0 (socket says 172.21.15.77:55016) 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 -- 172.21.15.77:0/1239467340 learned_addr learned my addr 172.21.15.77:0/1239467340 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa098004af0 0x7fa0b014f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0aeffd640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c47e0 0x7fa0b00c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 -- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa098004af0 msgr2=0x7fa0b014f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa098004af0 0x7fa0b014f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 -- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c47e0 msgr2=0x7fa0b00c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c47e0 0x7fa0b00c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 -- 172.21.15.77:0/1239467340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00c5330 con 0x7fa0b00af670 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0aeffd640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c47e0 0x7fa0b00c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0affff640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00af670 0x7fa0b00afaf0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fa0b006e9f0 tx=0x7fa0a800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa097fff640 1 -- 172.21.15.77:0/1239467340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8015430 con 0x7fa0b00af670 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa097fff640 1 -- 172.21.15.77:0/1239467340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a801d070 con 0x7fa0b00af670 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b00c5610 con 0x7fa0b00af670 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa097fff640 1 -- 172.21.15.77:0/1239467340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8015830 con 0x7fa0b00af670 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.447+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b00c5bd0 con 0x7fa0b00af670 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.448+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b00732b0 con 0x7fa0b00af670 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.449+0000 7fa097fff640 1 -- 172.21.15.77:0/1239467340 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0a80159d0 con 0x7fa0b00af670 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.449+0000 7fa097fff640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09c06b500 0x7fa09c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.449+0000 7fa097fff640 1 -- 172.21.15.77:0/1239467340 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa0a808abd0 con 0x7fa0b00af670 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.449+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09c06b500 0x7fa09c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.450+0000 7fa0af7fe640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09c06b500 0x7fa09c06d9c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0034000 tx=0x7fa0a0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.453+0000 7fa097fff640 1 -- 172.21.15.77:0/1239467340 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0a8010040 con 0x7fa0b00af670 2024-03-06T07:28:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.631+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0b0068ff0 con 0x7fa0b00af670 2024-03-06T07:28:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.632+0000 7fa097fff640 1 -- 172.21.15.77:0/1239467340 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0a8059960 con 0x7fa0b00af670 2024-03-06T07:28:16.634 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09c06b500 msgr2=0x7fa09c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09c06b500 0x7fa09c06d9c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0034000 tx=0x7fa0a0009030 comp rx=0 tx=0).stop 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00af670 msgr2=0x7fa0b00afaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00af670 0x7fa0b00afaf0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fa0b006e9f0 tx=0x7fa0a800f580 comp rx=0 tx=0).stop 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 shutdown_connections 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa09c06b500 0x7fa09c06d9c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0b00af670 0x7fa0b00afaf0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0b00c47e0 0x7fa0b00c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 --2- 172.21.15.77:0/1239467340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa098004af0 0x7fa0b014f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 >> 172.21.15.77:0/1239467340 conn(0x7fa0b006b190 msgr2=0x7fa0b006db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 shutdown_connections 2024-03-06T07:28:16.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.634+0000 7fa0b5cc3640 1 -- 172.21.15.77:0/1239467340 wait complete. 2024-03-06T07:28:16.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.750+0000 7f796e28f640 1 Processor -- start 2024-03-06T07:28:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.750+0000 7f796e28f640 1 -- start start 2024-03-06T07:28:16.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f796e28f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79681614d0 0x7f79680d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f796e28f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 0x7f79680d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f796e28f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79680d2740 con 0x7f79680d22b0 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f796e28f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79681618b0 con 0x7f79681614d0 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f796e28f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79680cf400 con 0x7f79680d1910 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f7967fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79681614d0 0x7f79680d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f7967fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79681614d0 0x7f79680d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55030/0 (socket says 172.21.15.77:55030) 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f796c82c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f79680d22b0 0x7f79680d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53454/0 (socket says 172.21.15.77:53454) 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.751+0000 7f7967fff640 1 -- 172.21.15.77:0/3920223797 learned_addr learned my addr 172.21.15.77:0/3920223797 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.752+0000 7f79677fe640 1 --2- 172.21.15.77:0/3920223797 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 0x7f79680d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.752+0000 7f7967fff640 1 -- 172.21.15.77:0/3920223797 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 msgr2=0x7f79680d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.752+0000 7f7967fff640 1 --2- 172.21.15.77:0/3920223797 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 0x7f79680d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.752+0000 7f7967fff640 1 -- 172.21.15.77:0/3920223797 >> v1:172.21.15.77:6789/0 conn(0x7f79680d22b0 legacy=0x7f79680d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.752+0000 7f7966ffd640 1 -- 172.21.15.77:0/3920223797 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598849774 0 0) 0x7f7960001490 con 0x7f79680d22b0 2024-03-06T07:28:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.752+0000 7f7967fff640 1 -- 172.21.15.77:0/3920223797 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680cf6a0 con 0x7f79681614d0 2024-03-06T07:28:16.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.752+0000 7f7967fff640 1 --2- 172.21.15.77:0/3920223797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680d4860 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f7958009870 tx=0x7f795802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee08634e9941a67d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f7966ffd640 1 -- 172.21.15.77:0/3920223797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958034070 con 0x7f79681614d0 2024-03-06T07:28:16.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f7966ffd640 1 -- 172.21.15.77:0/3920223797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7958030840 con 0x7f79681614d0 2024-03-06T07:28:16.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f7966ffd640 1 -- 172.21.15.77:0/3920223797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958030c10 con 0x7f79681614d0 2024-03-06T07:28:16.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f796e28f640 1 -- 172.21.15.77:0/3920223797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 msgr2=0x7f79680d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f796e28f640 1 --2- 172.21.15.77:0/3920223797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680d4860 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f7958009870 tx=0x7f795802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:16.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f796e28f640 1 -- 172.21.15.77:0/3920223797 shutdown_connections 2024-03-06T07:28:16.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f796e28f640 1 --2- 172.21.15.77:0/3920223797 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 0x7f79680d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f796e28f640 1 --2- 172.21.15.77:0/3920223797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680d4860 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f796e28f640 1 -- 172.21.15.77:0/3920223797 >> 172.21.15.77:0/3920223797 conn(0x7f79680d3d40 msgr2=0x7f79680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f796e28f640 1 -- 172.21.15.77:0/3920223797 shutdown_connections 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.753+0000 7f796e28f640 1 -- 172.21.15.77:0/3920223797 wait complete. 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.754+0000 7f796e28f640 1 Processor -- start 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796e28f640 1 -- start start 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796e28f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 0x7f79680ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796e28f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79680d22b0 0x7f7968151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796e28f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796e28f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79680b0910 con 0x7f79681614d0 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796e28f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79680cfc60 con 0x7f79680d22b0 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796e28f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79680b1e40 con 0x7f79680d1910 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796c82c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796c82c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680cbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55032/0 (socket says 172.21.15.77:55032) 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.755+0000 7f796c82c640 1 -- 172.21.15.77:0/1974792212 learned_addr learned my addr 172.21.15.77:0/1974792212 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f7967fff640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 0x7f79680ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f79677fe640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79680d22b0 0x7f7968151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f796c82c640 1 -- 172.21.15.77:0/1974792212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 msgr2=0x7f79680ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f796c82c640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 0x7f79680ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f796c82c640 1 -- 172.21.15.77:0/1974792212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79680d22b0 msgr2=0x7f7968151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f796c82c640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79680d22b0 0x7f7968151190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f796c82c640 1 -- 172.21.15.77:0/1974792212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79681516d0 con 0x7f79681614d0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f796c82c640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680cbe60 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f796000c7f0 tx=0x7f796000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f7964ff9640 1 -- 172.21.15.77:0/1974792212 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960018070 con 0x7f79681614d0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f7964ff9640 1 -- 172.21.15.77:0/1974792212 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796000a5e0 con 0x7f79681614d0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79680b9750 con 0x7f79681614d0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f7964ff9640 1 -- 172.21.15.77:0/1974792212 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796000aa10 con 0x7f79681614d0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.756+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79680c7eb0 con 0x7f79681614d0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.757+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7968068ff0 con 0x7f79681614d0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.757+0000 7f7964ff9640 1 -- 172.21.15.77:0/1974792212 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f796001b070 con 0x7f79681614d0 2024-03-06T07:28:16.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.758+0000 7f7964ff9640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f795406b4d0 0x7f795406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.758+0000 7f7964ff9640 1 -- 172.21.15.77:0/1974792212 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7960089830 con 0x7f79681614d0 2024-03-06T07:28:16.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.758+0000 7f7967fff640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f795406b4d0 0x7f795406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.758+0000 7f7967fff640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f795406b4d0 0x7f795406d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7958036000 tx=0x7f795803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.766+0000 7f7964ff9640 1 -- 172.21.15.77:0/1974792212 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79680d2740 con 0x7f79681614d0 2024-03-06T07:28:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.938+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f796806bc50 con 0x7f79681614d0 2024-03-06T07:28:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.939+0000 7f7964ff9640 1 -- 172.21.15.77:0/1974792212 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7960058650 con 0x7f79681614d0 2024-03-06T07:28:16.941 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:28:16.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f795406b4d0 msgr2=0x7f795406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f795406b4d0 0x7f795406d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7958036000 tx=0x7f795803b040 comp rx=0 tx=0).stop 2024-03-06T07:28:16.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 msgr2=0x7f79680cbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680cbe60 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f796000c7f0 tx=0x7f796000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 shutdown_connections 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f795406b4d0 0x7f795406d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79681614d0 0x7f79680cbe60 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f79680d22b0 0x7f7968151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 --2- 172.21.15.77:0/1974792212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f79680d1910 0x7f79680ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 >> 172.21.15.77:0/1974792212 conn(0x7f79680d3d40 msgr2=0x7f79680d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 shutdown_connections 2024-03-06T07:28:16.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:16.940+0000 7f796e28f640 1 -- 172.21.15.77:0/1974792212 wait complete. 2024-03-06T07:28:16.957 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:17.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.058+0000 7f72950eb640 1 Processor -- start 2024-03-06T07:28:17.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.058+0000 7f72950eb640 1 -- start start 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f72950eb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72901614d0 0x7f72900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f72950eb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72900d24b0 0x7f72900b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f72950eb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72901618b0 con 0x7f72900d1910 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f72950eb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72900cf400 con 0x7f72900d24b0 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f72950eb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72900cf6a0 con 0x7f72901614d0 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728f59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72900d24b0 0x7f72900b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728e59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f72900d1910 0x7f72900d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53458/0 (socket says 172.21.15.77:53458) 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728e59c640 1 -- 172.21.15.77:0/123122553 learned_addr learned my addr 172.21.15.77:0/123122553 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728f59e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72900d24b0 0x7f72900b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55040/0 (socket says 172.21.15.77:55040) 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728f59e640 1 -- 172.21.15.77:0/123122553 >> v2:172.21.15.158:3300/0 conn(0x7f72901614d0 msgr2=0x7f72900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728f59e640 1 --2- 172.21.15.77:0/123122553 >> v2:172.21.15.158:3300/0 conn(0x7f72901614d0 0x7f72900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728f59e640 1 -- 172.21.15.77:0/123122553 >> v1:172.21.15.77:6789/0 conn(0x7f72900d1910 legacy=0x7f72900d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728dd9b640 1 -- 172.21.15.77:0/123122553 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639589598 0 0) 0x7f72901618b0 con 0x7f72900d1910 2024-03-06T07:28:17.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.059+0000 7f728f59e640 1 -- 172.21.15.77:0/123122553 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900cf980 con 0x7f72900d24b0 2024-03-06T07:28:17.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.060+0000 7f728f59e640 1 --2- 172.21.15.77:0/123122553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72900b0e30 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f729008c470 tx=0x7f728802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9479fa9b581c32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.060+0000 7f728dd9b640 1 -- 172.21.15.77:0/123122553 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288035020 con 0x7f72900d24b0 2024-03-06T07:28:17.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.060+0000 7f728dd9b640 1 -- 172.21.15.77:0/123122553 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7288031830 con 0x7f72900d24b0 2024-03-06T07:28:17.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.060+0000 7f728dd9b640 1 -- 172.21.15.77:0/123122553 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288031c00 con 0x7f72900d24b0 2024-03-06T07:28:17.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.061+0000 7f72950eb640 1 -- 172.21.15.77:0/123122553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 msgr2=0x7f72900b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.061+0000 7f72950eb640 1 --2- 172.21.15.77:0/123122553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72900b0e30 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f729008c470 tx=0x7f728802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.061+0000 7f72950eb640 1 -- 172.21.15.77:0/123122553 shutdown_connections 2024-03-06T07:28:17.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.061+0000 7f72950eb640 1 --2- 172.21.15.77:0/123122553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72900b0e30 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.061+0000 7f72950eb640 1 --2- 172.21.15.77:0/123122553 >> v2:172.21.15.158:3300/0 conn(0x7f72901614d0 0x7f72900d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.061+0000 7f72950eb640 1 -- 172.21.15.77:0/123122553 >> 172.21.15.77:0/123122553 conn(0x7f72900d3d40 msgr2=0x7f72900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.061+0000 7f72950eb640 1 -- 172.21.15.77:0/123122553 shutdown_connections 2024-03-06T07:28:17.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.061+0000 7f72950eb640 1 -- 172.21.15.77:0/123122553 wait complete. 2024-03-06T07:28:17.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.062+0000 7f72950eb640 1 Processor -- start 2024-03-06T07:28:17.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.062+0000 7f72950eb640 1 -- start start 2024-03-06T07:28:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f72950eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72900d1910 0x7f7290155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f72950eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72901f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f72950eb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72901614d0 0x7f72900ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f72950eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900b1370 con 0x7f72900d24b0 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f72950eb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900b0910 con 0x7f72901614d0 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f72950eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72900b14f0 con 0x7f72900d1910 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f728e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72901f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f728e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72901f7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55052/0 (socket says 172.21.15.77:55052) 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f728e59c640 1 -- 172.21.15.77:0/133625225 learned_addr learned my addr 172.21.15.77:0/133625225 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f728ed9d640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72900d1910 0x7f7290155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.063+0000 7f728f59e640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72901614d0 0x7f72900ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f728e59c640 1 -- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72900d1910 msgr2=0x7f7290155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f728e59c640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72900d1910 0x7f7290155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f728e59c640 1 -- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72901614d0 msgr2=0x7f72900ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f728e59c640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72901614d0 0x7f72900ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f728e59c640 1 -- 172.21.15.77:0/133625225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290155b50 con 0x7f72900d24b0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f728ed9d640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72900d1910 0x7f7290155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f728e59c640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72901f7870 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f727800c890 tx=0x7f727800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f72877fe640 1 -- 172.21.15.77:0/133625225 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7278018070 con 0x7f72900d24b0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f72877fe640 1 -- 172.21.15.77:0/133625225 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727800a640 con 0x7f72900d24b0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72900b5070 con 0x7f72900d24b0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f72877fe640 1 -- 172.21.15.77:0/133625225 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727800aa30 con 0x7f72900d24b0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.064+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72900b55f0 con 0x7f72900d24b0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.065+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72900d3770 con 0x7f72900d24b0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.065+0000 7f72877fe640 1 -- 172.21.15.77:0/133625225 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f727801b070 con 0x7f72900d24b0 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.066+0000 7f72877fe640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f727006b4d0 0x7f727006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.066+0000 7f72877fe640 1 -- 172.21.15.77:0/133625225 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7278089990 con 0x7f72900d24b0 2024-03-06T07:28:17.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.066+0000 7f728ed9d640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f727006b4d0 0x7f727006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.066+0000 7f728ed9d640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f727006b4d0 0x7f727006d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7280009890 tx=0x7f7280008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.070+0000 7f72877fe640 1 -- 172.21.15.77:0/133625225 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7278058720 con 0x7f72900d24b0 2024-03-06T07:28:17.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.247+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7290068ff0 con 0x7f72900d24b0 2024-03-06T07:28:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.248+0000 7f72877fe640 1 -- 172.21.15.77:0/133625225 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7278058020 con 0x7f72900d24b0 2024-03-06T07:28:17.250 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:28:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f727006b4d0 msgr2=0x7f727006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f727006b4d0 0x7f727006d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7280009890 tx=0x7f7280008040 comp rx=0 tx=0).stop 2024-03-06T07:28:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 msgr2=0x7f72901f7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72901f7870 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f727800c890 tx=0x7f727800cd50 comp rx=0 tx=0).stop 2024-03-06T07:28:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 shutdown_connections 2024-03-06T07:28:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f727006b4d0 0x7f727006d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72900d24b0 0x7f72901f7870 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72901614d0 0x7f72900ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 --2- 172.21.15.77:0/133625225 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72900d1910 0x7f7290155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 >> 172.21.15.77:0/133625225 conn(0x7f72900d3d40 msgr2=0x7f729006cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 shutdown_connections 2024-03-06T07:28:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.250+0000 7f72950eb640 1 -- 172.21.15.77:0/133625225 wait complete. 2024-03-06T07:28:17.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:17.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.364+0000 7f4e550fb640 1 Processor -- start 2024-03-06T07:28:17.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.364+0000 7f4e550fb640 1 -- start start 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e550fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e50162970 0x7f4e500d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e550fb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500d22c0 con 0x7f4e500b4460 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e550fb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500d25a0 con 0x7f4e50161cf0 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e550fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e50161b00 con 0x7f4e50162970 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4ed9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4e500b4460 0x7f4e500b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53468/0 (socket says 172.21.15.77:53468) 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4ed9d640 1 -- 172.21.15.77:0/2021310130 learned_addr learned my addr 172.21.15.77:0/2021310130 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e47fff640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e50162970 0x7f4e500d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037871516 0 0) 0x7f4e500d22c0 con 0x7f4e500b4460 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e3c003520 con 0x7f4e500b4460 2024-03-06T07:28:17.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3753473022 0 0) 0x7f4e3c003520 con 0x7f4e500b4460 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e50162970 msgr2=0x7f4e500d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e50162970 0x7f4e500d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 >> v1:172.21.15.158:6789/0 conn(0x7f4e50161cf0 legacy=0x7f4e500cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e500d2880 con 0x7f4e500b4460 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f4e40001670 con 0x7f4e500b4460 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 >> v1:172.21.15.77:6789/0 conn(0x7f4e500b4460 legacy=0x7f4e500b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e50162970 0x7f4e3c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e3c003f60 0x7f4e3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e3c004d40 0x7f4e3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.365+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e50161b00 con 0x7f4e3c004d40 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c00baa0 con 0x7f4e3c003f60 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c00bca0 con 0x7f4e50162970 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4f59e640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e3c004d40 0x7f4e3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4ed9d640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e50162970 0x7f4e3c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e47fff640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e3c003f60 0x7f4e3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4f59e640 1 -- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e50162970 msgr2=0x7f4e3c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4f59e640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e50162970 0x7f4e3c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4f59e640 1 -- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e3c003f60 msgr2=0x7f4e3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4f59e640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e3c003f60 0x7f4e3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.366+0000 7f4e4f59e640 1 -- 172.21.15.77:0/2021310130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c003520 con 0x7f4e3c004d40 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e4f59e640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e3c004d40 0x7f4e3c0051c0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4e48001e50 tx=0x7f4e3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1bedde88cbce238 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48001a70 con 0x7f4e3c004d40 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e48003b70 con 0x7f4e3c004d40 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e4e59c640 1 -- 172.21.15.77:0/2021310130 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e480050d0 con 0x7f4e3c004d40 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e550fb640 1 -- 172.21.15.77:0/2021310130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e3c004d40 msgr2=0x7f4e3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e550fb640 1 --2- 172.21.15.77:0/2021310130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e3c004d40 0x7f4e3c0051c0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4e48001e50 tx=0x7f4e3c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e4f59e640 1 -- 172.21.15.77:0/2021310130 reap_dead start 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e550fb640 1 -- 172.21.15.77:0/2021310130 shutdown_connections 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e550fb640 1 -- 172.21.15.77:0/2021310130 >> 172.21.15.77:0/2021310130 conn(0x7f4e500d3ce0 msgr2=0x7f4e500d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e550fb640 1 -- 172.21.15.77:0/2021310130 shutdown_connections 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.367+0000 7f4e550fb640 1 -- 172.21.15.77:0/2021310130 wait complete. 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.369+0000 7f4e550fb640 1 Processor -- start 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.369+0000 7f4e550fb640 1 -- start start 2024-03-06T07:28:17.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.369+0000 7f4e550fb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e501566f0 0x7f4e500b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.369+0000 7f4e550fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e500ccd00 0x7f4e500b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.369+0000 7f4e550fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e500b2520 0x7f4e500cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.369+0000 7f4e550fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500ce950 con 0x7f4e500b2520 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.369+0000 7f4e550fb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500cecc0 con 0x7f4e501566f0 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.369+0000 7f4e550fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500b4840 con 0x7f4e500ccd00 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e500b2520 0x7f4e500cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e500b2520 0x7f4e500cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55076/0 (socket says 172.21.15.77:55076) 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 -- 172.21.15.77:0/3271742567 learned_addr learned my addr 172.21.15.77:0/3271742567 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e47fff640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e500ccd00 0x7f4e500b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4ed9d640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e501566f0 0x7f4e500b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 -- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e500ccd00 msgr2=0x7f4e500b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e500ccd00 0x7f4e500b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 -- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e501566f0 msgr2=0x7f4e500b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e501566f0 0x7f4e500b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 -- 172.21.15.77:0/3271742567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e50157360 con 0x7f4e500b2520 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e47fff640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e500ccd00 0x7f4e500b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e4f59e640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e500b2520 0x7f4e500cadb0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f4e4802c060 tx=0x7f4e480014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e477fe640 1 -- 172.21.15.77:0/3271742567 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48001a70 con 0x7f4e500b2520 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e477fe640 1 -- 172.21.15.77:0/3271742567 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e48003970 con 0x7f4e500b2520 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e477fe640 1 -- 172.21.15.77:0/3271742567 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48007db0 con 0x7f4e500b2520 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e500b6f70 con 0x7f4e500b2520 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.370+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e500b7530 con 0x7f4e500b2520 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.371+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e500b7730 con 0x7f4e500b2520 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.372+0000 7f4e477fe640 1 -- 172.21.15.77:0/3271742567 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e48007030 con 0x7f4e500b2520 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.372+0000 7f4e477fe640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4e3006b4d0 0x7f4e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.372+0000 7f4e477fe640 1 -- 172.21.15.77:0/3271742567 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f4e480ab410 con 0x7f4e500b2520 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.372+0000 7f4e4ed9d640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4e3006b4d0 0x7f4e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.372+0000 7f4e4ed9d640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4e3006b4d0 0x7f4e3006d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4e40000fb0 tx=0x7f4e40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.376+0000 7f4e477fe640 1 -- 172.21.15.77:0/3271742567 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e500b7730 con 0x7f4e500b2520 2024-03-06T07:28:17.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.569+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4e5004f330 con 0x7f4e500b2520 2024-03-06T07:28:17.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.569+0000 7f4e477fe640 1 -- 172.21.15.77:0/3271742567 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v113) v1 ==== 80+0+244 (secure 0 0 0) 0x7f4e48033020 con 0x7f4e500b2520 2024-03-06T07:28:17.571 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6835/1169947364 2024-03-07T07:27:19.549687+0000 2024-03-06T07:28:17.571 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6834/1169947364 2024-03-07T07:27:19.549687+0000 2024-03-06T07:28:17.571 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6837/3247585937 2024-03-07T07:27:18.550938+0000 2024-03-06T07:28:17.571 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6836/3247585937 2024-03-07T07:27:18.550938+0000 2024-03-06T07:28:17.572 INFO:teuthology.orchestra.run.smithi077.stderr:listed 4 entries 2024-03-06T07:28:17.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4e3006b4d0 msgr2=0x7f4e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4e3006b4d0 0x7f4e3006d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4e40000fb0 tx=0x7f4e40009040 comp rx=0 tx=0).stop 2024-03-06T07:28:17.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e500b2520 msgr2=0x7f4e500cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e500b2520 0x7f4e500cadb0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f4e4802c060 tx=0x7f4e480014a0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 shutdown_connections 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4e3006b4d0 0x7f4e3006d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e500b2520 0x7f4e500cadb0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4e500ccd00 0x7f4e500b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 --2- 172.21.15.77:0/3271742567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4e501566f0 0x7f4e500b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 >> 172.21.15.77:0/3271742567 conn(0x7f4e500d3ce0 msgr2=0x7f4e500d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 shutdown_connections 2024-03-06T07:28:17.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:17.571+0000 7f4e550fb640 1 -- 172.21.15.77:0/3271742567 wait complete. 2024-03-06T07:28:17.584 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:28:17.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:17.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[74158]: fuse finished with error 0 and tester_r 0 2024-03-06T07:28:17.611 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:28:17.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.611 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:17.777 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:28:17.777 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:17.795 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:17.795 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:28:17.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.796 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:28:17.853 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:28:17.853 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:17.868 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:17.868 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:28:17.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.868 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:17.921 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:28:17.922 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:28:17.922 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:17.939 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:17.939 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:28:17.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.939 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:17.996 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:28:17.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:18.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.094+0000 7f145bf5c640 1 Processor -- start 2024-03-06T07:28:18.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.094+0000 7f145bf5c640 1 -- start start 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.095+0000 7f145bf5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 0x7f14540d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.095+0000 7f145bf5c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1454161980 con 0x7f14540cd8c0 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.095+0000 7f145bf5c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1454161c60 con 0x7f14540cfff0 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.095+0000 7f145bf5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1454161f40 con 0x7f145415bb20 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f145a4f9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f14540cd8c0 0x7f14540cebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53478/0 (socket says 172.21.15.77:53478) 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f145a4f9640 1 -- 172.21.15.77:0/3967365161 learned_addr learned my addr 172.21.15.77:0/3967365161 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396361269 0 0) 0x7f1454161980 con 0x7f14540cd8c0 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1440003520 con 0x7f14540cd8c0 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1459cf8640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 0x7f14540d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3279917534 0 0) 0x7f1440003520 con 0x7f14540cd8c0 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 msgr2=0x7f14540d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 0x7f14540d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 >> v1:172.21.15.158:6789/0 conn(0x7f14540cfff0 legacy=0x7f14540ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1454162220 con 0x7f14540cd8c0 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f144c0019f0 con 0x7f14540cd8c0 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 >> v1:172.21.15.77:6789/0 conn(0x7f14540cd8c0 legacy=0x7f14540cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14400039e0 0x7f1440003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1440004800 0x7f1440004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 0x7f1440005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1454161f40 con 0x7f14400039e0 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144000baf0 con 0x7f1440004800 2024-03-06T07:28:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144000bcf0 con 0x7f145415bb20 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f1459cf8640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14400039e0 0x7f1440003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f14594f7640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1440004800 0x7f1440004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.096+0000 7f145a4f9640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 0x7f1440005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1459cf8640 1 -- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 msgr2=0x7f1440005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1459cf8640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 0x7f1440005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1459cf8640 1 -- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1440004800 msgr2=0x7f1440004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1459cf8640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1440004800 0x7f1440004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1459cf8640 1 -- 172.21.15.77:0/3967365161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1440003520 con 0x7f14400039e0 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f145a4f9640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f145415bb20 0x7f1440005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f14594f7640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1440004800 0x7f1440004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1459cf8640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14400039e0 0x7f1440003e00 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f1450009ac0 tx=0x7f1440005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f35854128e4457 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450039070 con 0x7f14400039e0 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145002c6b0 con 0x7f14400039e0 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.097+0000 7f1458cf6640 1 -- 172.21.15.77:0/3967365161 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450009690 con 0x7f14400039e0 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.098+0000 7f145bf5c640 1 -- 172.21.15.77:0/3967365161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14400039e0 msgr2=0x7f1440003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.098+0000 7f145bf5c640 1 --2- 172.21.15.77:0/3967365161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14400039e0 0x7f1440003e00 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f1450009ac0 tx=0x7f1440005830 comp rx=0 tx=0).stop 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.098+0000 7f145a4f9640 1 -- 172.21.15.77:0/3967365161 reap_dead start 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.098+0000 7f145bf5c640 1 -- 172.21.15.77:0/3967365161 shutdown_connections 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.098+0000 7f145bf5c640 1 -- 172.21.15.77:0/3967365161 >> 172.21.15.77:0/3967365161 conn(0x7f145409a9f0 msgr2=0x7f145409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.098+0000 7f145bf5c640 1 -- 172.21.15.77:0/3967365161 shutdown_connections 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.098+0000 7f145bf5c640 1 -- 172.21.15.77:0/3967365161 wait complete. 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.100+0000 7f145bf5c640 1 Processor -- start 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.100+0000 7f145bf5c640 1 -- start start 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145bf5c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14400039e0 0x7f14541f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145bf5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14540b46c0 0x7f145414bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145bf5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14540af940 0x7f14540afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145bf5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14540d1980 con 0x7f14540af940 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145bf5c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14540cf2d0 con 0x7f14400039e0 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145bf5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14540d1b00 con 0x7f14540b46c0 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145a4f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14540af940 0x7f14540afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145a4f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14540af940 0x7f14540afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55096/0 (socket says 172.21.15.77:55096) 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145a4f9640 1 -- 172.21.15.77:0/1655315513 learned_addr learned my addr 172.21.15.77:0/1655315513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f14594f7640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14540b46c0 0x7f145414bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f1459cf8640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14400039e0 0x7f14541f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.101+0000 7f145a4f9640 1 -- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14540b46c0 msgr2=0x7f145414bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f145a4f9640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14540b46c0 0x7f145414bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f145a4f9640 1 -- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14400039e0 msgr2=0x7f14541f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f145a4f9640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14400039e0 0x7f14541f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f145a4f9640 1 -- 172.21.15.77:0/1655315513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14541f80c0 con 0x7f14540af940 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f1459cf8640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14400039e0 0x7f14541f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f145a4f9640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14540af940 0x7f14540afdc0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f1440004780 tx=0x7f144c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f144a7fc640 1 -- 172.21.15.77:0/1655315513 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c00eda0 con 0x7f14540af940 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f144a7fc640 1 -- 172.21.15.77:0/1655315513 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c014430 con 0x7f14540af940 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14541f7820 con 0x7f14540af940 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f144a7fc640 1 -- 172.21.15.77:0/1655315513 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c014820 con 0x7f14540af940 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.102+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14541f7de0 con 0x7f14540af940 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.103+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145404f330 con 0x7f14540af940 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.103+0000 7f144a7fc640 1 -- 172.21.15.77:0/1655315513 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f144c0149c0 con 0x7f14540af940 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.104+0000 7f144a7fc640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.104+0000 7f144a7fc640 1 -- 172.21.15.77:0/1655315513 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f144c08a0f0 con 0x7f14540af940 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.104+0000 7f1459cf8640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.104+0000 7f1459cf8640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f143806b4d0 0x7f143806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1450036000 tx=0x7f1450030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.108+0000 7f144a7fc640 1 -- 172.21.15.77:0/1655315513 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f144c0588c0 con 0x7f14540af940 2024-03-06T07:28:18.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.404+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14540b1820 con 0x7f14540af940 2024-03-06T07:28:18.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.404+0000 7f144a7fc640 1 -- 172.21.15.77:0/1655315513 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f144c05c960 con 0x7f14540af940 2024-03-06T07:28:18.407 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:18.408 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:18.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f143806b4d0 msgr2=0x7f143806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f143806b4d0 0x7f143806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1450036000 tx=0x7f1450030000 comp rx=0 tx=0).stop 2024-03-06T07:28:18.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14540af940 msgr2=0x7f14540afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14540af940 0x7f14540afdc0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f1440004780 tx=0x7f144c00e530 comp rx=0 tx=0).stop 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 shutdown_connections 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f143806b4d0 0x7f143806d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14540af940 0x7f14540afdc0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14540b46c0 0x7f145414bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 --2- 172.21.15.77:0/1655315513 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14400039e0 0x7f14541f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 >> 172.21.15.77:0/1655315513 conn(0x7f145409a9f0 msgr2=0x7f14540b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 shutdown_connections 2024-03-06T07:28:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.406+0000 7f145bf5c640 1 -- 172.21.15.77:0/1655315513 wait complete. 2024-03-06T07:28:18.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:18.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.522+0000 7f268524e640 1 Processor -- start 2024-03-06T07:28:18.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.522+0000 7f268524e640 1 -- start start 2024-03-06T07:28:18.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f268524e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f268015dfe0 0x7f26800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f268524e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26800d29c0 0x7f26800b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f268524e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26800b3430 con 0x7f26800cf830 2024-03-06T07:28:18.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f268524e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26800b31e0 con 0x7f268015dfe0 2024-03-06T07:28:18.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f268524e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26800b3d60 con 0x7f26800d29c0 2024-03-06T07:28:18.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f267ed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f268015dfe0 0x7f26800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f2677fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f26800cf830 0x7f26800d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38748/0 (socket says 172.21.15.77:38748) 2024-03-06T07:28:18.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f2677fff640 1 -- 172.21.15.77:0/81172614 learned_addr learned my addr 172.21.15.77:0/81172614 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f267f59e640 1 --2- 172.21.15.77:0/81172614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26800d29c0 0x7f26800b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f267ed9d640 1 -- 172.21.15.77:0/81172614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26800d29c0 msgr2=0x7f26800b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f267ed9d640 1 --2- 172.21.15.77:0/81172614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26800d29c0 0x7f26800b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f267ed9d640 1 -- 172.21.15.77:0/81172614 >> v1:172.21.15.77:6789/0 conn(0x7f26800cf830 legacy=0x7f26800d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f267ed9d640 1 -- 172.21.15.77:0/81172614 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26800b3710 con 0x7f268015dfe0 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f267e59c640 1 -- 172.21.15.77:0/81172614 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531581629 0 0) 0x7f26800b3430 con 0x7f26800cf830 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.523+0000 7f267f59e640 1 --2- 172.21.15.77:0/81172614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26800d29c0 0x7f26800b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.524+0000 7f267ed9d640 1 --2- 172.21.15.77:0/81172614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f268015dfe0 0x7f26800cf2f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f26700097a0 tx=0x7f267002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79bdb03d4242b47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.524+0000 7f267e59c640 1 -- 172.21.15.77:0/81172614 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2670035020 con 0x7f268015dfe0 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.524+0000 7f267e59c640 1 -- 172.21.15.77:0/81172614 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2670031830 con 0x7f268015dfe0 2024-03-06T07:28:18.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.524+0000 7f267e59c640 1 -- 172.21.15.77:0/81172614 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2670031c00 con 0x7f268015dfe0 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.525+0000 7f268524e640 1 -- 172.21.15.77:0/81172614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f268015dfe0 msgr2=0x7f26800cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.525+0000 7f268524e640 1 --2- 172.21.15.77:0/81172614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f268015dfe0 0x7f26800cf2f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f26700097a0 tx=0x7f267002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.525+0000 7f268524e640 1 -- 172.21.15.77:0/81172614 shutdown_connections 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.525+0000 7f268524e640 1 --2- 172.21.15.77:0/81172614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26800d29c0 0x7f26800b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.525+0000 7f268524e640 1 --2- 172.21.15.77:0/81172614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f268015dfe0 0x7f26800cf2f0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.525+0000 7f268524e640 1 -- 172.21.15.77:0/81172614 >> 172.21.15.77:0/81172614 conn(0x7f26800d3cd0 msgr2=0x7f26800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.525+0000 7f268524e640 1 -- 172.21.15.77:0/81172614 shutdown_connections 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.525+0000 7f268524e640 1 -- 172.21.15.77:0/81172614 wait complete. 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.526+0000 7f268524e640 1 Processor -- start 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.526+0000 7f268524e640 1 -- start start 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f268524e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26800cf830 0x7f26800c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f268524e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26800d29c0 0x7f268014d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f268524e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f268015dfe0 0x7f268014d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f268524e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26800d1df0 con 0x7f26800cf830 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f268524e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26800d1c70 con 0x7f26800d29c0 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f268524e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268015e3c0 con 0x7f268015dfe0 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26800cf830 0x7f26800c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26800cf830 0x7f26800c9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59132/0 (socket says 172.21.15.77:59132) 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 -- 172.21.15.77:0/809293087 learned_addr learned my addr 172.21.15.77:0/809293087 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f2677fff640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26800d29c0 0x7f268014d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267f59e640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f268015dfe0 0x7f268014d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 -- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f268015dfe0 msgr2=0x7f268014d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f268015dfe0 0x7f268014d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 -- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26800d29c0 msgr2=0x7f268014d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26800d29c0 0x7f268014d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 -- 172.21.15.77:0/809293087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680161120 con 0x7f26800cf830 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267f59e640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f268015dfe0 0x7f268014d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.527+0000 7f267ed9d640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26800cf830 0x7f26800c9f50 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f26700097a0 tx=0x7f26700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.528+0000 7f26777fe640 1 -- 172.21.15.77:0/809293087 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2670035040 con 0x7f26800cf830 2024-03-06T07:28:18.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.528+0000 7f26777fe640 1 -- 172.21.15.77:0/809293087 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2670004230 con 0x7f26800cf830 2024-03-06T07:28:18.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.528+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268014b150 con 0x7f26800cf830 2024-03-06T07:28:18.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.528+0000 7f26777fe640 1 -- 172.21.15.77:0/809293087 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267003a700 con 0x7f26800cf830 2024-03-06T07:28:18.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.528+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268014b710 con 0x7f26800cf830 2024-03-06T07:28:18.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.529+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268004f330 con 0x7f26800cf830 2024-03-06T07:28:18.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.529+0000 7f26777fe640 1 -- 172.21.15.77:0/809293087 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2670038020 con 0x7f26800cf830 2024-03-06T07:28:18.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.529+0000 7f26777fe640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f266006b4d0 0x7f266006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.529+0000 7f26777fe640 1 -- 172.21.15.77:0/809293087 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f26700abfe0 con 0x7f26800cf830 2024-03-06T07:28:18.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.530+0000 7f2677fff640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f266006b4d0 0x7f266006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.530+0000 7f2677fff640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f266006b4d0 0x7f266006d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f2668007fd0 tx=0x7f2668009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.534+0000 7f26777fe640 1 -- 172.21.15.77:0/809293087 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f267007ae00 con 0x7f26800cf830 2024-03-06T07:28:18.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.713+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26800b51a0 con 0x7f26800cf830 2024-03-06T07:28:18.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.714+0000 7f26777fe640 1 -- 172.21.15.77:0/809293087 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+17027 (secure 0 0 0) 0x7f267007a700 con 0x7f26800cf830 2024-03-06T07:28:18.717 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:18.717 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":113,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:48.616890+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.34","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6835/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6834/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6837/3247585937":"2024-03-07T07:27:18.550938+0000","172.21.15.77:6836/3247585937":"2024-03-07T07:27:18.550938+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:18.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f266006b4d0 msgr2=0x7f266006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f266006b4d0 0x7f266006d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f2668007fd0 tx=0x7f2668009040 comp rx=0 tx=0).stop 2024-03-06T07:28:18.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26800cf830 msgr2=0x7f26800c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26800cf830 0x7f26800c9f50 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f26700097a0 tx=0x7f26700046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 shutdown_connections 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f268015dfe0 0x7f268014d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26800d29c0 0x7f268014d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f266006b4d0 0x7f266006d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 --2- 172.21.15.77:0/809293087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26800cf830 0x7f26800c9f50 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 >> 172.21.15.77:0/809293087 conn(0x7f26800d3cd0 msgr2=0x7f268006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 shutdown_connections 2024-03-06T07:28:18.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.716+0000 7f268524e640 1 -- 172.21.15.77:0/809293087 wait complete. 2024-03-06T07:28:18.734 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:03.116010+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:03.812681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:18.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:18.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.835+0000 7fc8e313a640 1 Processor -- start 2024-03-06T07:28:18.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.835+0000 7fc8e313a640 1 -- start start 2024-03-06T07:28:18.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8e313a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8dc0d4860 0x7fc8dc0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8e313a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc8dc0b0e30 0x7fc8dc0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8e313a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc1618b0 con 0x7fc8dc1614d0 2024-03-06T07:28:18.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8e313a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc0d4cf0 con 0x7fc8dc0d4860 2024-03-06T07:28:18.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8e313a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc0cf400 con 0x7fc8dc0b0e30 2024-03-06T07:28:18.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8dbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8dc0d4860 0x7fc8dc0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8dbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8dc0d4860 0x7fc8dc0d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59138/0 (socket says 172.21.15.77:59138) 2024-03-06T07:28:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8dbfff640 1 -- 172.21.15.77:0/330053544 learned_addr learned my addr 172.21.15.77:0/330053544 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8e16d7640 1 --2- 172.21.15.77:0/330053544 >> v2:172.21.15.158:3300/0 conn(0x7fc8dc0b0e30 0x7fc8dc0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8dbfff640 1 -- 172.21.15.77:0/330053544 >> v2:172.21.15.158:3300/0 conn(0x7fc8dc0b0e30 msgr2=0x7fc8dc0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8dbfff640 1 --2- 172.21.15.77:0/330053544 >> v2:172.21.15.158:3300/0 conn(0x7fc8dc0b0e30 0x7fc8dc0b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8dbfff640 1 -- 172.21.15.77:0/330053544 >> v1:172.21.15.158:6789/0 conn(0x7fc8dc1614d0 legacy=0x7fc8dc0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.836+0000 7fc8dbfff640 1 -- 172.21.15.77:0/330053544 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc0cf6a0 con 0x7fc8dc0d4860 2024-03-06T07:28:18.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.837+0000 7fc8dbfff640 1 --2- 172.21.15.77:0/330053544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc0d4860 0x7fc8dc0d2020 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0b2a70 tx=0x7fc8c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5702411f4dfea6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.837+0000 7fc8db7fe640 1 -- 172.21.15.77:0/330053544 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8034070 con 0x7fc8dc0d4860 2024-03-06T07:28:18.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.837+0000 7fc8db7fe640 1 -- 172.21.15.77:0/330053544 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c8030860 con 0x7fc8dc0d4860 2024-03-06T07:28:18.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.837+0000 7fc8db7fe640 1 -- 172.21.15.77:0/330053544 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8030c30 con 0x7fc8dc0d4860 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.838+0000 7fc8e313a640 1 -- 172.21.15.77:0/330053544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc0d4860 msgr2=0x7fc8dc0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.838+0000 7fc8e313a640 1 --2- 172.21.15.77:0/330053544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc0d4860 0x7fc8dc0d2020 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0b2a70 tx=0x7fc8c802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.838+0000 7fc8e313a640 1 -- 172.21.15.77:0/330053544 shutdown_connections 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.838+0000 7fc8e313a640 1 --2- 172.21.15.77:0/330053544 >> v2:172.21.15.158:3300/0 conn(0x7fc8dc0b0e30 0x7fc8dc0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.838+0000 7fc8e313a640 1 --2- 172.21.15.77:0/330053544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc0d4860 0x7fc8dc0d2020 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.838+0000 7fc8e313a640 1 -- 172.21.15.77:0/330053544 >> 172.21.15.77:0/330053544 conn(0x7fc8dc0d3d40 msgr2=0x7fc8dc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.838+0000 7fc8e313a640 1 -- 172.21.15.77:0/330053544 shutdown_connections 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.838+0000 7fc8e313a640 1 -- 172.21.15.77:0/330053544 wait complete. 2024-03-06T07:28:18.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.840+0000 7fc8e313a640 1 Processor -- start 2024-03-06T07:28:18.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.840+0000 7fc8e313a640 1 -- start start 2024-03-06T07:28:18.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e313a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8dc0b0e30 0x7fc8dc0b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e313a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8dc0d4860 0x7fc8dc159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e313a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc1614d0 0x7fc8dc1e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e313a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0b0910 con 0x7fc8dc1614d0 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e313a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0cfc60 con 0x7fc8dc0b0e30 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e313a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0d2560 con 0x7fc8dc0d4860 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e16d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc1614d0 0x7fc8dc1e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e16d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc1614d0 0x7fc8dc1e9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59146/0 (socket says 172.21.15.77:59146) 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e16d7640 1 -- 172.21.15.77:0/3650995490 learned_addr learned my addr 172.21.15.77:0/3650995490 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e0ed6640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8dc0b0e30 0x7fc8dc0b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8dbfff640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8dc0d4860 0x7fc8dc159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e16d7640 1 -- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8dc0d4860 msgr2=0x7fc8dc159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e16d7640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8dc0d4860 0x7fc8dc159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e16d7640 1 -- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8dc0b0e30 msgr2=0x7fc8dc0b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e16d7640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8dc0b0e30 0x7fc8dc0b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8e16d7640 1 -- 172.21.15.77:0/3650995490 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc0cb310 con 0x7fc8dc1614d0 2024-03-06T07:28:18.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.841+0000 7fc8dbfff640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8dc0d4860 0x7fc8dc159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.842+0000 7fc8e16d7640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc1614d0 0x7fc8dc1e9930 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400efc0 tx=0x7fc8d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.842+0000 7fc8d97fa640 1 -- 172.21.15.77:0/3650995490 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4018020 con 0x7fc8dc1614d0 2024-03-06T07:28:18.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.842+0000 7fc8d97fa640 1 -- 172.21.15.77:0/3650995490 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d4014480 con 0x7fc8dc1614d0 2024-03-06T07:28:18.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.842+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc0cb5f0 con 0x7fc8dc1614d0 2024-03-06T07:28:18.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.842+0000 7fc8d97fa640 1 -- 172.21.15.77:0/3650995490 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4014870 con 0x7fc8dc1614d0 2024-03-06T07:28:18.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.842+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc0cbbb0 con 0x7fc8dc1614d0 2024-03-06T07:28:18.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.843+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc068ff0 con 0x7fc8dc1614d0 2024-03-06T07:28:18.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.843+0000 7fc8d97fa640 1 -- 172.21.15.77:0/3650995490 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8d4005d10 con 0x7fc8dc1614d0 2024-03-06T07:28:18.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.844+0000 7fc8d97fa640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc8c006b4d0 0x7fc8c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.844+0000 7fc8e0ed6640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc8c006b4d0 0x7fc8c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.844+0000 7fc8d97fa640 1 -- 172.21.15.77:0/3650995490 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc8d408ad80 con 0x7fc8dc1614d0 2024-03-06T07:28:18.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.844+0000 7fc8e0ed6640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc8c006b4d0 0x7fc8c006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc15e160 tx=0x7fc8d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:18.848+0000 7fc8d97fa640 1 -- 172.21.15.77:0/3650995490 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8d4022070 con 0x7fc8dc1614d0 2024-03-06T07:28:19.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.025+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8dc0cbdf0 con 0x7fc8dc1614d0 2024-03-06T07:28:19.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.025+0000 7fc8d97fa640 1 -- 172.21.15.77:0/3650995490 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8d4010040 con 0x7fc8dc1614d0 2024-03-06T07:28:19.028 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:28:19.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc8c006b4d0 msgr2=0x7fc8c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc8c006b4d0 0x7fc8c006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc15e160 tx=0x7fc8d0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc1614d0 msgr2=0x7fc8dc1e9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc1614d0 0x7fc8dc1e9930 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400efc0 tx=0x7fc8d400c490 comp rx=0 tx=0).stop 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 shutdown_connections 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc8c006b4d0 0x7fc8c006d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8dc1614d0 0x7fc8dc1e9930 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8dc0d4860 0x7fc8dc159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 --2- 172.21.15.77:0/3650995490 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8dc0b0e30 0x7fc8dc0b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.027+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 >> 172.21.15.77:0/3650995490 conn(0x7fc8dc0d3d40 msgr2=0x7fc8dc0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.028+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 shutdown_connections 2024-03-06T07:28:19.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.028+0000 7fc8e313a640 1 -- 172.21.15.77:0/3650995490 wait complete. 2024-03-06T07:28:19.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:19.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.141+0000 7fbaa97c9640 1 Processor -- start 2024-03-06T07:28:19.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.141+0000 7fbaa97c9640 1 -- start start 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.142+0000 7fbaa97c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa415c050 0x7fbaa406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.142+0000 7fbaa97c9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbaa40d0af0 0x7fbaa40cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.142+0000 7fbaa97c9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa40cd4b0 con 0x7fbaa406fb40 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.142+0000 7fbaa97c9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa406ff80 con 0x7fbaa40d0af0 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.142+0000 7fbaa97c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa40cd790 con 0x7fbaa415c050 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fbaa37fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbaa40d0af0 0x7fbaa40cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fbaa2ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa415c050 0x7fbaa406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fba9a7fc640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbaa406fb40 0x7fbaa40cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44056/0 (socket says 172.21.15.77:44056) 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fbaa2ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa415c050 0x7fbaa406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:40662/0 (socket says 172.21.15.77:40662) 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fbaa37fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbaa40d0af0 0x7fbaa40cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:42992/0 (socket says 172.21.15.77:42992) 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fba9a7fc640 1 -- 172.21.15.77:0/958462627 learned_addr learned my addr 172.21.15.77:0/958462627 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fbaa2ffd640 1 -- 172.21.15.77:0/958462627 >> v2:172.21.15.158:3300/0 conn(0x7fbaa40d0af0 msgr2=0x7fbaa40cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fbaa2ffd640 1 --2- 172.21.15.77:0/958462627 >> v2:172.21.15.158:3300/0 conn(0x7fbaa40d0af0 0x7fbaa40cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fbaa2ffd640 1 -- 172.21.15.77:0/958462627 >> v1:172.21.15.158:6789/0 conn(0x7fbaa406fb40 legacy=0x7fbaa40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.143+0000 7fbaa2ffd640 1 -- 172.21.15.77:0/958462627 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa40cda70 con 0x7fbaa415c050 2024-03-06T07:28:19.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.144+0000 7fbaa2ffd640 1 --2- 172.21.15.77:0/958462627 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa415c050 0x7fbaa406f600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fba940097a0 tx=0x7fba9402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a9e94a597e8a96a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.144+0000 7fbaa27fc640 1 -- 172.21.15.77:0/958462627 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba940315a0 con 0x7fbaa415c050 2024-03-06T07:28:19.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.144+0000 7fbaa27fc640 1 -- 172.21.15.77:0/958462627 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba94037070 con 0x7fbaa415c050 2024-03-06T07:28:19.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.144+0000 7fbaa27fc640 1 -- 172.21.15.77:0/958462627 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba94031960 con 0x7fbaa415c050 2024-03-06T07:28:19.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.145+0000 7fbaa97c9640 1 -- 172.21.15.77:0/958462627 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa415c050 msgr2=0x7fbaa406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.145+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/958462627 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa415c050 0x7fbaa406f600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fba940097a0 tx=0x7fba9402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:19.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.145+0000 7fbaa97c9640 1 -- 172.21.15.77:0/958462627 shutdown_connections 2024-03-06T07:28:19.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.145+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/958462627 >> v2:172.21.15.158:3300/0 conn(0x7fbaa40d0af0 0x7fbaa40cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.145+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/958462627 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa415c050 0x7fbaa406f600 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.145+0000 7fbaa97c9640 1 -- 172.21.15.77:0/958462627 >> 172.21.15.77:0/958462627 conn(0x7fbaa40b3800 msgr2=0x7fbaa40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.145+0000 7fbaa97c9640 1 -- 172.21.15.77:0/958462627 shutdown_connections 2024-03-06T07:28:19.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.145+0000 7fbaa97c9640 1 -- 172.21.15.77:0/958462627 wait complete. 2024-03-06T07:28:19.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.147+0000 7fbaa97c9640 1 Processor -- start 2024-03-06T07:28:19.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.147+0000 7fbaa97c9640 1 -- start start 2024-03-06T07:28:19.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fbaa97c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa406fb40 0x7fbaa41f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fbaa97c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa40d0af0 0x7fbaa40b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fbaa97c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa415c050 0x7fbaa409bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fbaa97c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4165c30 con 0x7fbaa40d0af0 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fbaa97c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4165ab0 con 0x7fbaa415c050 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fbaa97c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4165db0 con 0x7fbaa406fb40 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fba9a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa40d0af0 0x7fbaa40b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fba9a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa40d0af0 0x7fbaa40b0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59150/0 (socket says 172.21.15.77:59150) 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fba9a7fc640 1 -- 172.21.15.77:0/3810869531 learned_addr learned my addr 172.21.15.77:0/3810869531 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fbaa2ffd640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa406fb40 0x7fbaa41f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fbaa37fe640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa415c050 0x7fbaa409bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.148+0000 7fba9a7fc640 1 -- 172.21.15.77:0/3810869531 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa406fb40 msgr2=0x7fbaa41f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fba9a7fc640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa406fb40 0x7fbaa41f33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fba9a7fc640 1 -- 172.21.15.77:0/3810869531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa415c050 msgr2=0x7fbaa409bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fba9a7fc640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa415c050 0x7fbaa409bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fba9a7fc640 1 -- 172.21.15.77:0/3810869531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa409d9d0 con 0x7fbaa40d0af0 2024-03-06T07:28:19.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fba9a7fc640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa40d0af0 0x7fbaa40b0070 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00c3c0 tx=0x7fba8c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fba9bfff640 1 -- 172.21.15.77:0/3810869531 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c017070 con 0x7fbaa40d0af0 2024-03-06T07:28:19.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fba9bfff640 1 -- 172.21.15.77:0/3810869531 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c005ce0 con 0x7fbaa40d0af0 2024-03-06T07:28:19.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa409dcd0 con 0x7fbaa40d0af0 2024-03-06T07:28:19.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fba9bfff640 1 -- 172.21.15.77:0/3810869531 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c0072f0 con 0x7fbaa40d0af0 2024-03-06T07:28:19.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.149+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa409e510 con 0x7fbaa40d0af0 2024-03-06T07:28:19.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.150+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa409e6e0 con 0x7fbaa40d0af0 2024-03-06T07:28:19.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.151+0000 7fba9bfff640 1 -- 172.21.15.77:0/3810869531 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba8c01a070 con 0x7fbaa40d0af0 2024-03-06T07:28:19.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.151+0000 7fba9bfff640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8406b4d0 0x7fba8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.151+0000 7fba9bfff640 1 -- 172.21.15.77:0/3810869531 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fba8c089690 con 0x7fbaa40d0af0 2024-03-06T07:28:19.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.151+0000 7fbaa2ffd640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8406b4d0 0x7fba8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.152+0000 7fbaa2ffd640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8406b4d0 0x7fba8406d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fba94036000 tx=0x7fba9403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.156+0000 7fba9bfff640 1 -- 172.21.15.77:0/3810869531 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaa409dcd0 con 0x7fbaa40d0af0 2024-03-06T07:28:19.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.332+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbaa4068ff0 con 0x7fbaa40d0af0 2024-03-06T07:28:19.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.332+0000 7fba9bfff640 1 -- 172.21.15.77:0/3810869531 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba8c057d50 con 0x7fbaa40d0af0 2024-03-06T07:28:19.335 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:28:19.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8406b4d0 msgr2=0x7fba8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8406b4d0 0x7fba8406d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fba94036000 tx=0x7fba9403d040 comp rx=0 tx=0).stop 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa40d0af0 msgr2=0x7fbaa40b0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa40d0af0 0x7fbaa40b0070 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00c3c0 tx=0x7fba8c00c880 comp rx=0 tx=0).stop 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 shutdown_connections 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fba8406b4d0 0x7fba8406d990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaa40d0af0 0x7fbaa40b0070 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbaa415c050 0x7fbaa409bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 --2- 172.21.15.77:0/3810869531 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbaa406fb40 0x7fbaa41f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 >> 172.21.15.77:0/3810869531 conn(0x7fbaa40b3800 msgr2=0x7fbaa40aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 shutdown_connections 2024-03-06T07:28:19.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.334+0000 7fbaa97c9640 1 -- 172.21.15.77:0/3810869531 wait complete. 2024-03-06T07:28:19.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:19.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.454+0000 7f1082cc9640 1 Processor -- start 2024-03-06T07:28:19.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.454+0000 7f1082cc9640 1 -- start start 2024-03-06T07:28:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.454+0000 7f1082cc9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f107c15c050 0x7f107c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1082cc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c0d0af0 0x7f107c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1082cc9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0cd4b0 con 0x7f107c06fb40 2024-03-06T07:28:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1082cc9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c06ffb0 con 0x7f107c15c050 2024-03-06T07:28:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1082cc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c0cd790 con 0x7f107c0d0af0 2024-03-06T07:28:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1081266640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c0d0af0 0x7f107c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1080a65640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f107c15c050 0x7f107c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f107bfff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f107c06fb40 0x7f107c0cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44070/0 (socket says 172.21.15.77:44070) 2024-03-06T07:28:19.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1081266640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c0d0af0 0x7f107c0cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:40672/0 (socket says 172.21.15.77:40672) 2024-03-06T07:28:19.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1080a65640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f107c15c050 0x7f107c06f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:43014/0 (socket says 172.21.15.77:43014) 2024-03-06T07:28:19.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f107bfff640 1 -- 172.21.15.77:0/3308556891 learned_addr learned my addr 172.21.15.77:0/3308556891 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:19.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f107b7fe640 1 -- 172.21.15.77:0/3308556891 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830834323 0 0) 0x7f107c0cd4b0 con 0x7f107c06fb40 2024-03-06T07:28:19.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1081266640 1 -- 172.21.15.77:0/3308556891 >> v2:172.21.15.158:3300/0 conn(0x7f107c15c050 msgr2=0x7f107c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1081266640 1 --2- 172.21.15.77:0/3308556891 >> v2:172.21.15.158:3300/0 conn(0x7f107c15c050 0x7f107c06f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1081266640 1 -- 172.21.15.77:0/3308556891 >> v1:172.21.15.158:6789/0 conn(0x7f107c06fb40 legacy=0x7f107c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.455+0000 7f1081266640 1 -- 172.21.15.77:0/3308556891 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c0cda70 con 0x7f107c0d0af0 2024-03-06T07:28:19.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.456+0000 7f1081266640 1 --2- 172.21.15.77:0/3308556891 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c0d0af0 0x7f107c0cce70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f107402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a1694f7f8c34a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.456+0000 7f107b7fe640 1 -- 172.21.15.77:0/3308556891 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074039070 con 0x7f107c0d0af0 2024-03-06T07:28:19.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.456+0000 7f107b7fe640 1 -- 172.21.15.77:0/3308556891 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107402cbc0 con 0x7f107c0d0af0 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.456+0000 7f107b7fe640 1 -- 172.21.15.77:0/3308556891 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074034440 con 0x7f107c0d0af0 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.457+0000 7f1082cc9640 1 -- 172.21.15.77:0/3308556891 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c0d0af0 msgr2=0x7f107c0cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.457+0000 7f1082cc9640 1 --2- 172.21.15.77:0/3308556891 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c0d0af0 0x7f107c0cce70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f107402be10 comp rx=0 tx=0).stop 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.457+0000 7f1082cc9640 1 -- 172.21.15.77:0/3308556891 shutdown_connections 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.457+0000 7f1082cc9640 1 --2- 172.21.15.77:0/3308556891 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c0d0af0 0x7f107c0cce70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.457+0000 7f1082cc9640 1 --2- 172.21.15.77:0/3308556891 >> v2:172.21.15.158:3300/0 conn(0x7f107c15c050 0x7f107c06f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.457+0000 7f1082cc9640 1 -- 172.21.15.77:0/3308556891 >> 172.21.15.77:0/3308556891 conn(0x7f107c0b3800 msgr2=0x7f107c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.457+0000 7f1082cc9640 1 -- 172.21.15.77:0/3308556891 shutdown_connections 2024-03-06T07:28:19.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.457+0000 7f1082cc9640 1 -- 172.21.15.77:0/3308556891 wait complete. 2024-03-06T07:28:19.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.459+0000 7f1082cc9640 1 Processor -- start 2024-03-06T07:28:19.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.459+0000 7f1082cc9640 1 -- start start 2024-03-06T07:28:19.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1082cc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c06fb40 0x7f107c0d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1082cc9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f107c0d0af0 0x7f107c09d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1082cc9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f107c15c050 0x7f107c13ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1082cc9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c165c30 con 0x7f107c15c050 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1082cc9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c165ab0 con 0x7f107c0d0af0 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1082cc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c165db0 con 0x7f107c06fb40 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1081266640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f107c15c050 0x7f107c13ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1081266640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f107c15c050 0x7f107c13ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59156/0 (socket says 172.21.15.77:59156) 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1081266640 1 -- 172.21.15.77:0/99166328 learned_addr learned my addr 172.21.15.77:0/99166328 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f107bfff640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f107c0d0af0 0x7f107c09d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.460+0000 7f1080a65640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c06fb40 0x7f107c0d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f1081266640 1 -- 172.21.15.77:0/99166328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c06fb40 msgr2=0x7f107c0d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f1081266640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c06fb40 0x7f107c0d1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f1081266640 1 -- 172.21.15.77:0/99166328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f107c0d0af0 msgr2=0x7f107c09d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f1081266640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f107c0d0af0 0x7f107c09d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f1081266640 1 -- 172.21.15.77:0/99166328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c09e160 con 0x7f107c15c050 2024-03-06T07:28:19.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f1081266640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f107c15c050 0x7f107c13ed50 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1074037040 tx=0x7f1074004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f10797fa640 1 -- 172.21.15.77:0/99166328 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074044020 con 0x7f107c15c050 2024-03-06T07:28:19.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f10797fa640 1 -- 172.21.15.77:0/99166328 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107403a040 con 0x7f107c15c050 2024-03-06T07:28:19.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f10797fa640 1 -- 172.21.15.77:0/99166328 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074039050 con 0x7f107c15c050 2024-03-06T07:28:19.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c09e440 con 0x7f107c15c050 2024-03-06T07:28:19.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.461+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c09ece0 con 0x7f107c15c050 2024-03-06T07:28:19.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.462+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c0707f0 con 0x7f107c15c050 2024-03-06T07:28:19.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.463+0000 7f10797fa640 1 -- 172.21.15.77:0/99166328 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1074003bb0 con 0x7f107c15c050 2024-03-06T07:28:19.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.463+0000 7f10797fa640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f106006b4d0 0x7f106006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.463+0000 7f1080a65640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f106006b4d0 0x7f106006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.463+0000 7f10797fa640 1 -- 172.21.15.77:0/99166328 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f10740ac750 con 0x7f107c15c050 2024-03-06T07:28:19.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.464+0000 7f1080a65640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f106006b4d0 0x7f106006d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f107c159510 tx=0x7f1070009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.467+0000 7f10797fa640 1 -- 172.21.15.77:0/99166328 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f107c0707f0 con 0x7f107c15c050 2024-03-06T07:28:19.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.643+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f107c09fe60 con 0x7f107c15c050 2024-03-06T07:28:19.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.644+0000 7f10797fa640 1 -- 172.21.15.77:0/99166328 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7f107404e060 con 0x7f107c15c050 2024-03-06T07:28:19.647 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.645+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f106006b4d0 msgr2=0x7f106006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.645+0000 7f1082cc9640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f106006b4d0 0x7f106006d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f107c159510 tx=0x7f1070009190 comp rx=0 tx=0).stop 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.645+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f107c15c050 msgr2=0x7f107c13ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.645+0000 7f1082cc9640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f107c15c050 0x7f107c13ed50 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f1074037040 tx=0x7f1074004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.646+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 shutdown_connections 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.646+0000 7f1082cc9640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f106006b4d0 0x7f106006d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.646+0000 7f1082cc9640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f107c15c050 0x7f107c13ed50 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.646+0000 7f1082cc9640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f107c0d0af0 0x7f107c09d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.646+0000 7f1082cc9640 1 --2- 172.21.15.77:0/99166328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f107c06fb40 0x7f107c0d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.646+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 >> 172.21.15.77:0/99166328 conn(0x7f107c0b3800 msgr2=0x7f107c15b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.646+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 shutdown_connections 2024-03-06T07:28:19.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.646+0000 7f1082cc9640 1 -- 172.21.15.77:0/99166328 wait complete. 2024-03-06T07:28:19.663 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:19.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.762+0000 7fe022dd7640 1 Processor -- start 2024-03-06T07:28:19.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.762+0000 7fe022dd7640 1 -- start start 2024-03-06T07:28:19.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe022dd7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe01c0d22b0 0x7fe01c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe022dd7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c0b3430 con 0x7fe01c0b41f0 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe022dd7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c0b3710 con 0x7fe01c15dfe0 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe022dd7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01c0d2c50 con 0x7fe01c0d22b0 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe021374640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe01c0b41f0 0x7fe01c0b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38762/0 (socket says 172.21.15.77:38762) 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe021374640 1 -- 172.21.15.77:0/3984547865 learned_addr learned my addr 172.21.15.77:0/3984547865 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe01bfff640 1 --2- 172.21.15.77:0/3984547865 >> v2:172.21.15.158:3300/0 conn(0x7fe01c0d22b0 0x7fe01c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23928481 0 0) 0x7fe01c0b3430 con 0x7fe01c0b41f0 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.763+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe00c003520 con 0x7fe01c0b41f0 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2821063085 0 0) 0x7fe00c003520 con 0x7fe01c0b41f0 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 >> v2:172.21.15.158:3300/0 conn(0x7fe01c0d22b0 msgr2=0x7fe01c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 --2- 172.21.15.77:0/3984547865 >> v2:172.21.15.158:3300/0 conn(0x7fe01c0d22b0 0x7fe01c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 >> v1:172.21.15.158:6789/0 conn(0x7fe01c15dfe0 legacy=0x7fe01c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c0b39f0 con 0x7fe01c0b41f0 2024-03-06T07:28:19.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fe0140019f0 con 0x7fe01c0b41f0 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 >> v1:172.21.15.77:6789/0 conn(0x7fe01c0b41f0 legacy=0x7fe01c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe00c0039b0 0x7fe00c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe00c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe00c0051c0 0x7fe00c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c00c030 con 0x7fe00c004800 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c00c1e0 con 0x7fe00c0051c0 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c00c3e0 con 0x7fe00c0039b0 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01bfff640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe00c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe020b73640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe00c0039b0 0x7fe00c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe021374640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe00c0051c0 0x7fe00c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01bfff640 1 -- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe00c0039b0 msgr2=0x7fe00c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01bfff640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe00c0039b0 0x7fe00c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01bfff640 1 -- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe00c0051c0 msgr2=0x7fe00c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01bfff640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe00c0051c0 0x7fe00c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe021374640 1 -- 172.21.15.77:0/3984547865 reap_dead start 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe01bfff640 1 -- 172.21.15.77:0/3984547865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c003520 con 0x7fe00c004800 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.764+0000 7fe020b73640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe00c0039b0 0x7fe00c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.765+0000 7fe01bfff640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe00c004c80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fe008009ac0 tx=0x7fe00c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7d5bc4ac32b5d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.765+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008039070 con 0x7fe00c004800 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.765+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00802c6b0 con 0x7fe00c004800 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.765+0000 7fe01b7fe640 1 -- 172.21.15.77:0/3984547865 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008009690 con 0x7fe00c004800 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.766+0000 7fe022dd7640 1 -- 172.21.15.77:0/3984547865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 msgr2=0x7fe00c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.766+0000 7fe022dd7640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe00c004c80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fe008009ac0 tx=0x7fe00c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.766+0000 7fe022dd7640 1 -- 172.21.15.77:0/3984547865 shutdown_connections 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.766+0000 7fe022dd7640 1 --2- 172.21.15.77:0/3984547865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe00c004c80 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.766+0000 7fe022dd7640 1 -- 172.21.15.77:0/3984547865 >> 172.21.15.77:0/3984547865 conn(0x7fe01c0d3cd0 msgr2=0x7fe01c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.766+0000 7fe022dd7640 1 -- 172.21.15.77:0/3984547865 shutdown_connections 2024-03-06T07:28:19.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.766+0000 7fe022dd7640 1 -- 172.21.15.77:0/3984547865 wait complete. 2024-03-06T07:28:19.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.767+0000 7fe022dd7640 1 Processor -- start 2024-03-06T07:28:19.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.767+0000 7fe022dd7640 1 -- start start 2024-03-06T07:28:19.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe022dd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe01c0a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe022dd7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe01c09dd40 0x7fe01c09f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe022dd7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe01c09e1c0 0x7fe01c09ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe022dd7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0b3d60 con 0x7fe00c004800 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe022dd7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0b31e0 con 0x7fe01c09e1c0 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe022dd7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c15e3c0 con 0x7fe01c09dd40 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe01c0a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe01c0a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59168/0 (socket says 172.21.15.77:59168) 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 -- 172.21.15.77:0/1668450 learned_addr learned my addr 172.21.15.77:0/1668450 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe021374640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe01c09e1c0 0x7fe01c09ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe01bfff640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe01c09dd40 0x7fe01c09f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 -- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe01c09dd40 msgr2=0x7fe01c09f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe01c09dd40 0x7fe01c09f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 -- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe01c09e1c0 msgr2=0x7fe01c09ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe01c09e1c0 0x7fe01c09ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 -- 172.21.15.77:0/1668450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c0a05a0 con 0x7fe00c004800 2024-03-06T07:28:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe021374640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe01c09e1c0 0x7fe01c09ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe020b73640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe01c0a1640 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fe01000e4b0 tx=0x7fe01000e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe0197fa640 1 -- 172.21.15.77:0/1668450 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100052e0 con 0x7fe00c004800 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe0197fa640 1 -- 172.21.15.77:0/1668450 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe010007020 con 0x7fe00c004800 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c0a0880 con 0x7fe00c004800 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe0197fa640 1 -- 172.21.15.77:0/1668450 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010007210 con 0x7fe00c004800 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.768+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c0a0e40 con 0x7fe00c004800 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.769+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c04f330 con 0x7fe00c004800 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.770+0000 7fe0197fa640 1 -- 172.21.15.77:0/1668450 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe01001a030 con 0x7fe00c004800 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.770+0000 7fe0197fa640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe00006b500 0x7fe00006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.770+0000 7fe0197fa640 1 -- 172.21.15.77:0/1668450 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe010089820 con 0x7fe00c004800 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.770+0000 7fe01bfff640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe00006b500 0x7fe00006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.771+0000 7fe01bfff640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe00006b500 0x7fe00006d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe008036000 tx=0x7fe008030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.775+0000 7fe0197fa640 1 -- 172.21.15.77:0/1668450 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe010021020 con 0x7fe00c004800 2024-03-06T07:28:19.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.989+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe01c0d0bc0 con 0x7fe00c004800 2024-03-06T07:28:19.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.990+0000 7fe0197fa640 1 -- 172.21.15.77:0/1668450 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fe010058640 con 0x7fe00c004800 2024-03-06T07:28:19.993 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:19.993 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:19.993 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:19.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe00006b500 msgr2=0x7fe00006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe00006b500 0x7fe00006d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe008036000 tx=0x7fe008030000 comp rx=0 tx=0).stop 2024-03-06T07:28:19.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 msgr2=0x7fe01c0a1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe01c0a1640 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fe01000e4b0 tx=0x7fe01000e970 comp rx=0 tx=0).stop 2024-03-06T07:28:19.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 shutdown_connections 2024-03-06T07:28:19.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe01c09e1c0 0x7fe01c09ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe01c09dd40 0x7fe01c09f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe00006b500 0x7fe00006d9c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 --2- 172.21.15.77:0/1668450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe00c004800 0x7fe01c0a1640 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 >> 172.21.15.77:0/1668450 conn(0x7fe01c0d3cd0 msgr2=0x7fe01c06c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 shutdown_connections 2024-03-06T07:28:19.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:19.992+0000 7fe022dd7640 1 -- 172.21.15.77:0/1668450 wait complete. 2024-03-06T07:28:20.006 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:20.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.104+0000 7fb8bf49a640 1 Processor -- start 2024-03-06T07:28:20.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.104+0000 7fb8bf49a640 1 -- start start 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bf49a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb8b815bdd0 0x7fb8b80c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bf49a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b80d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bf49a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80cebe0 con 0x7fb8b815fa10 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bf49a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b80ce980 con 0x7fb8b815bdd0 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bf49a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80ceec0 con 0x7fb8b80d16e0 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bd236640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb8b815bdd0 0x7fb8b80c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bda37640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b80d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bca35640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb8b815fa10 0x7fb8b80ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44090/0 (socket says 172.21.15.77:44090) 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.105+0000 7fb8bca35640 1 -- 172.21.15.77:0/1257777246 learned_addr learned my addr 172.21.15.77:0/1257777246 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.106+0000 7fb8affff640 1 -- 172.21.15.77:0/1257777246 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437729054 0 0) 0x7fb8b80cebe0 con 0x7fb8b815fa10 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.106+0000 7fb8affff640 1 -- 172.21.15.77:0/1257777246 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a4003520 con 0x7fb8b815fa10 2024-03-06T07:28:20.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.106+0000 7fb8bd236640 1 -- 172.21.15.77:0/1257777246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 msgr2=0x7fb8b80d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.106+0000 7fb8bd236640 1 --2- 172.21.15.77:0/1257777246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b80d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.106+0000 7fb8bd236640 1 -- 172.21.15.77:0/1257777246 >> v1:172.21.15.158:6789/0 conn(0x7fb8b815fa10 legacy=0x7fb8b80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.106+0000 7fb8bd236640 1 -- 172.21.15.77:0/1257777246 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80cf1a0 con 0x7fb8b815bdd0 2024-03-06T07:28:20.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.106+0000 7fb8bda37640 1 --2- 172.21.15.77:0/1257777246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b80d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bd236640 1 --2- 172.21.15.77:0/1257777246 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815bdd0 0x7fb8b80c5610 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009d90 tx=0x7fb8b402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7d918febc8e807c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8affff640 1 -- 172.21.15.77:0/1257777246 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4039070 con 0x7fb8b815bdd0 2024-03-06T07:28:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8affff640 1 -- 172.21.15.77:0/1257777246 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b402cbc0 con 0x7fb8b815bdd0 2024-03-06T07:28:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8affff640 1 -- 172.21.15.77:0/1257777246 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4034440 con 0x7fb8b815bdd0 2024-03-06T07:28:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1257777246 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815bdd0 msgr2=0x7fb8b80c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1257777246 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815bdd0 0x7fb8b80c5610 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009d90 tx=0x7fb8b402be10 comp rx=0 tx=0).stop 2024-03-06T07:28:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1257777246 shutdown_connections 2024-03-06T07:28:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1257777246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b80d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1257777246 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815bdd0 0x7fb8b80c5610 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1257777246 >> 172.21.15.77:0/1257777246 conn(0x7fb8b809a9f0 msgr2=0x7fb8b809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1257777246 shutdown_connections 2024-03-06T07:28:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.107+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1257777246 wait complete. 2024-03-06T07:28:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.108+0000 7fb8bf49a640 1 Processor -- start 2024-03-06T07:28:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.108+0000 7fb8bf49a640 1 -- start start 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bf49a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b809cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bf49a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8b815bdd0 0x7fb8b815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bf49a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815fa10 0x7fb8b80b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bf49a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80d2050 con 0x7fb8b815bdd0 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bf49a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80c5b50 con 0x7fb8b815fa10 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bf49a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80cc5d0 con 0x7fb8b80d16e0 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bca35640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8b815bdd0 0x7fb8b815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bca35640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8b815bdd0 0x7fb8b815cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59184/0 (socket says 172.21.15.77:59184) 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bca35640 1 -- 172.21.15.77:0/1507965014 learned_addr learned my addr 172.21.15.77:0/1507965014 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bd236640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b809cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.109+0000 7fb8bda37640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815fa10 0x7fb8b80b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8bca35640 1 -- 172.21.15.77:0/1507965014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 msgr2=0x7fb8b809cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8bca35640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b809cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8bca35640 1 -- 172.21.15.77:0/1507965014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815fa10 msgr2=0x7fb8b80b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8bca35640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815fa10 0x7fb8b80b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8bca35640 1 -- 172.21.15.77:0/1507965014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b815a950 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8bca35640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8b815bdd0 0x7fb8b815cbc0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c410 tx=0x7fb8a800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8adffb640 1 -- 172.21.15.77:0/1507965014 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8017070 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8adffb640 1 -- 172.21.15.77:0/1507965014 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a8005ce0 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b809d0f0 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8adffb640 1 -- 172.21.15.77:0/1507965014 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80072d0 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.110+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b80b49d0 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.111+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b804f330 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.112+0000 7fb8adffb640 1 -- 172.21.15.77:0/1507965014 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8a801a070 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.112+0000 7fb8adffb640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb89c06b4d0 0x7fb89c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.112+0000 7fb8adffb640 1 -- 172.21.15.77:0/1507965014 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb8a8089590 con 0x7fb8b815bdd0 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.112+0000 7fb8bd236640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb89c06b4d0 0x7fb89c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.112+0000 7fb8bd236640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb89c06b4d0 0x7fb89c06d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4036000 tx=0x7fb8b4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.116+0000 7fb8adffb640 1 -- 172.21.15.77:0/1507965014 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8b815a950 con 0x7fb8b815bdd0 2024-03-06T07:28:20.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.290+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8b80b7a40 con 0x7fb8b815bdd0 2024-03-06T07:28:20.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.292+0000 7fb8adffb640 1 -- 172.21.15.77:0/1507965014 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+17027 (secure 0 0 0) 0x7fb8a8057c50 con 0x7fb8b815bdd0 2024-03-06T07:28:20.295 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:20.295 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":113,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:48.616890+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.34","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6835/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6834/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6837/3247585937":"2024-03-07T07:27:18.550938+0000","172.21.15.77:6836/3247585937":"2024-03-07T07:27:18.550938+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb89c06b4d0 msgr2=0x7fb89c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb89c06b4d0 0x7fb89c06d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4036000 tx=0x7fb8b4030000 comp rx=0 tx=0).stop 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8b815bdd0 msgr2=0x7fb8b815cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8b815bdd0 0x7fb8b815cbc0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c410 tx=0x7fb8a800c810 comp rx=0 tx=0).stop 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 shutdown_connections 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb89c06b4d0 0x7fb89c06d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8b815bdd0 0x7fb8b815cbc0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8b815fa10 0x7fb8b80b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 --2- 172.21.15.77:0/1507965014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb8b80d16e0 0x7fb8b809cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 >> 172.21.15.77:0/1507965014 conn(0x7fb8b809a9f0 msgr2=0x7fb8b80c7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 shutdown_connections 2024-03-06T07:28:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.294+0000 7fb8bf49a640 1 -- 172.21.15.77:0/1507965014 wait complete. 2024-03-06T07:28:20.308 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:20.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.421+0000 7fd3220cf640 1 Processor -- start 2024-03-06T07:28:20.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.421+0000 7fd3220cf640 1 -- start start 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.421+0000 7fd3220cf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd31c15dfe0 0x7fd31c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.421+0000 7fd3220cf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0cf830 0x7fd31c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.421+0000 7fd3220cf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31c0b3430 con 0x7fd31c0d27f0 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.421+0000 7fd3220cf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31c0d2c50 con 0x7fd31c15dfe0 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.421+0000 7fd3220cf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0b31e0 con 0x7fd31c0cf830 2024-03-06T07:28:20.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.421+0000 7fd31b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd31c15dfe0 0x7fd31c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd31c15dfe0 0x7fd31c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59198/0 (socket says 172.21.15.77:59198) 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31b7fe640 1 -- 172.21.15.77:0/1513368986 learned_addr learned my addr 172.21.15.77:0/1513368986 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31bfff640 1 --1- 172.21.15.77:0/1513368986 >> v1:172.21.15.77:6789/0 conn(0x7fd31c0d27f0 0x7fd31c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38766/0 (socket says 172.21.15.77:38766) 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31affd640 1 --2- 172.21.15.77:0/1513368986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0cf830 0x7fd31c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31b7fe640 1 -- 172.21.15.77:0/1513368986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0cf830 msgr2=0x7fd31c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31b7fe640 1 --2- 172.21.15.77:0/1513368986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0cf830 0x7fd31c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31b7fe640 1 -- 172.21.15.77:0/1513368986 >> v1:172.21.15.77:6789/0 conn(0x7fd31c0d27f0 legacy=0x7fd31c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31a7fc640 1 -- 172.21.15.77:0/1513368986 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006293162 0 0) 0x7fd31c0b3430 con 0x7fd31c0d27f0 2024-03-06T07:28:20.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.422+0000 7fd31b7fe640 1 -- 172.21.15.77:0/1513368986 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c0b3710 con 0x7fd31c15dfe0 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.423+0000 7fd31b7fe640 1 --2- 172.21.15.77:0/1513368986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c15dfe0 0x7fd31c0cf2f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009870 tx=0x7fd30c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d5711606c7fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.423+0000 7fd31a7fc640 1 -- 172.21.15.77:0/1513368986 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c034070 con 0x7fd31c15dfe0 2024-03-06T07:28:20.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.423+0000 7fd31a7fc640 1 -- 172.21.15.77:0/1513368986 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c030840 con 0x7fd31c15dfe0 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.423+0000 7fd31a7fc640 1 -- 172.21.15.77:0/1513368986 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c030c10 con 0x7fd31c15dfe0 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.424+0000 7fd3220cf640 1 -- 172.21.15.77:0/1513368986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c15dfe0 msgr2=0x7fd31c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.424+0000 7fd3220cf640 1 --2- 172.21.15.77:0/1513368986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c15dfe0 0x7fd31c0cf2f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009870 tx=0x7fd30c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.424+0000 7fd3220cf640 1 -- 172.21.15.77:0/1513368986 shutdown_connections 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.424+0000 7fd3220cf640 1 --2- 172.21.15.77:0/1513368986 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0cf830 0x7fd31c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.424+0000 7fd3220cf640 1 --2- 172.21.15.77:0/1513368986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c15dfe0 0x7fd31c0cf2f0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.424+0000 7fd3220cf640 1 -- 172.21.15.77:0/1513368986 >> 172.21.15.77:0/1513368986 conn(0x7fd31c0d3cd0 msgr2=0x7fd31c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.424+0000 7fd3220cf640 1 -- 172.21.15.77:0/1513368986 shutdown_connections 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.424+0000 7fd3220cf640 1 -- 172.21.15.77:0/1513368986 wait complete. 2024-03-06T07:28:20.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.425+0000 7fd3220cf640 1 Processor -- start 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.425+0000 7fd3220cf640 1 -- start start 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd3220cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c0cf830 0x7fd31c1f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd3220cf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0d27f0 0x7fd31c160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd3220cf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31c15dfe0 0x7fd31c0b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd3220cf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0d1c70 con 0x7fd31c0cf830 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd3220cf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0b3d60 con 0x7fd31c15dfe0 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd3220cf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0d1df0 con 0x7fd31c0d27f0 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c0cf830 0x7fd31c1f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c0cf830 0x7fd31c1f6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59208/0 (socket says 172.21.15.77:59208) 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 -- 172.21.15.77:0/3625008710 learned_addr learned my addr 172.21.15.77:0/3625008710 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31affd640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0d27f0 0x7fd31c160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31bfff640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31c15dfe0 0x7fd31c0b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 -- 172.21.15.77:0/3625008710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0d27f0 msgr2=0x7fd31c160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0d27f0 0x7fd31c160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 -- 172.21.15.77:0/3625008710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31c15dfe0 msgr2=0x7fd31c0b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31c15dfe0 0x7fd31c0b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 -- 172.21.15.77:0/3625008710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c15f4a0 con 0x7fd31c0cf830 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.426+0000 7fd31b7fe640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c0cf830 0x7fd31c1f6d10 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009870 tx=0x7fd30c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.427+0000 7fd303fff640 1 -- 172.21.15.77:0/3625008710 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c034040 con 0x7fd31c0cf830 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.427+0000 7fd303fff640 1 -- 172.21.15.77:0/3625008710 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c03ab90 con 0x7fd31c0cf830 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.427+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c15f780 con 0x7fd31c0cf830 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.427+0000 7fd303fff640 1 -- 172.21.15.77:0/3625008710 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c004030 con 0x7fd31c0cf830 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.427+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c15fd40 con 0x7fd31c0cf830 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.428+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c04f330 con 0x7fd31c0cf830 2024-03-06T07:28:20.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.428+0000 7fd303fff640 1 -- 172.21.15.77:0/3625008710 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd30c04b060 con 0x7fd31c0cf830 2024-03-06T07:28:20.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.429+0000 7fd303fff640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30806b500 0x7fd30806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.429+0000 7fd303fff640 1 -- 172.21.15.77:0/3625008710 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd30c0abf30 con 0x7fd31c0cf830 2024-03-06T07:28:20.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.429+0000 7fd31affd640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30806b500 0x7fd30806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.429+0000 7fd31affd640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30806b500 0x7fd30806d9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd31c1611e0 tx=0x7fd310009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.437+0000 7fd303fff640 1 -- 172.21.15.77:0/3625008710 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd30c03f070 con 0x7fd31c0cf830 2024-03-06T07:28:20.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.654+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd31c0b09c0 con 0x7fd31c0cf830 2024-03-06T07:28:20.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.654+0000 7fd303fff640 1 -- 172.21.15.77:0/3625008710 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fd30c07a670 con 0x7fd31c0cf830 2024-03-06T07:28:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:20.657 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:20.658 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:20.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30806b500 msgr2=0x7fd30806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30806b500 0x7fd30806d9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd31c1611e0 tx=0x7fd310009190 comp rx=0 tx=0).stop 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c0cf830 msgr2=0x7fd31c1f6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c0cf830 0x7fd31c1f6d10 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009870 tx=0x7fd30c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 shutdown_connections 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd31c15dfe0 0x7fd31c0b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd31c0d27f0 0x7fd31c160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd30806b500 0x7fd30806d9c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.656+0000 7fd3220cf640 1 --2- 172.21.15.77:0/3625008710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31c0cf830 0x7fd31c1f6d10 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.657+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 >> 172.21.15.77:0/3625008710 conn(0x7fd31c0d3cd0 msgr2=0x7fd31c0d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.657+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 shutdown_connections 2024-03-06T07:28:20.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.657+0000 7fd3220cf640 1 -- 172.21.15.77:0/3625008710 wait complete. 2024-03-06T07:28:20.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:20.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.772+0000 7f5fb63e2640 1 Processor -- start 2024-03-06T07:28:20.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.772+0000 7f5fb63e2640 1 -- start start 2024-03-06T07:28:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5fb63e2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb00d2710 0x7f5fb00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5fb63e2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00b3430 con 0x7f5fb00d22b0 2024-03-06T07:28:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5fb63e2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00b3710 con 0x7f5fb015dfe0 2024-03-06T07:28:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5fb63e2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b31e0 con 0x7f5fb00d2710 2024-03-06T07:28:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faf7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5fb00d22b0 0x7f5fb00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38772/0 (socket says 172.21.15.77:38772) 2024-03-06T07:28:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faf7fe640 1 -- 172.21.15.77:0/2743445474 learned_addr learned my addr 172.21.15.77:0/2743445474 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984674244 0 0) 0x7f5fb00b3430 con 0x7f5fb00d22b0 2024-03-06T07:28:20.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5fb497f640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb00d2710 0x7f5fb00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f98003520 con 0x7f5fb00d22b0 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2256127367 0 0) 0x7f5f98003520 con 0x7f5fb00d22b0 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb00d2710 msgr2=0x7f5fb00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb00d2710 0x7f5fb00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 >> v1:172.21.15.158:6789/0 conn(0x7f5fb015dfe0 legacy=0x7f5fb00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00b39f0 con 0x7f5fb00d22b0 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f5fa0001670 con 0x7f5fb00d22b0 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 >> v1:172.21.15.77:6789/0 conn(0x7f5fb00d22b0 legacy=0x7f5fb00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f980039e0 0x7f5f98003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98004800 0x7f5f98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb00d2710 0x7f5f980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b31e0 con 0x7f5f980039e0 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9800baa0 con 0x7f5f98004800 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9800bca0 con 0x7f5fb00d2710 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.773+0000 7f5faffff640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f980039e0 0x7f5f98003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.774+0000 7f5faf7fe640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98004800 0x7f5f98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.774+0000 7f5fb497f640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb00d2710 0x7f5f980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.774+0000 7f5faffff640 1 -- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb00d2710 msgr2=0x7f5f980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.774+0000 7f5faffff640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb00d2710 0x7f5f980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.774+0000 7f5faffff640 1 -- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98004800 msgr2=0x7f5f98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.774+0000 7f5faffff640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f98004800 0x7f5f98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.774+0000 7f5faffff640 1 -- 172.21.15.77:0/2743445474 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98003520 con 0x7f5f980039e0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5faffff640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f980039e0 0x7f5f98003e20 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4001720 tx=0x7f5f98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7d824969f5a373 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4001370 con 0x7f5f980039e0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa40052d0 con 0x7f5f980039e0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5faeffd640 1 -- 172.21.15.77:0/2743445474 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4003830 con 0x7f5f980039e0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5fb63e2640 1 -- 172.21.15.77:0/2743445474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f980039e0 msgr2=0x7f5f98003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5fb63e2640 1 --2- 172.21.15.77:0/2743445474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f980039e0 0x7f5f98003e20 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4001720 tx=0x7f5f98005830 comp rx=0 tx=0).stop 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5fb497f640 1 -- 172.21.15.77:0/2743445474 reap_dead start 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5fb63e2640 1 -- 172.21.15.77:0/2743445474 shutdown_connections 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5fb63e2640 1 -- 172.21.15.77:0/2743445474 >> 172.21.15.77:0/2743445474 conn(0x7f5fb00d3cd0 msgr2=0x7f5fb00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5fb63e2640 1 -- 172.21.15.77:0/2743445474 shutdown_connections 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.775+0000 7f5fb63e2640 1 -- 172.21.15.77:0/2743445474 wait complete. 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.776+0000 7f5fb63e2640 1 Processor -- start 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb63e2640 1 -- start start 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb63e2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980039e0 0x7f5fb01f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb63e2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb015bda0 0x7f5fb00b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb63e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb00b6710 0x7f5fb00b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb63e2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00cfa00 con 0x7f5fb00b6710 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb63e2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b3d60 con 0x7f5f980039e0 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb63e2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00d1c70 con 0x7f5fb015bda0 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb497f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb00b6710 0x7f5fb00b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb497f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb00b6710 0x7f5fb00b7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59234/0 (socket says 172.21.15.77:59234) 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.777+0000 7f5fb497f640 1 -- 172.21.15.77:0/1611090416 learned_addr learned my addr 172.21.15.77:0/1611090416 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5faf7fe640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb015bda0 0x7f5fb00b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5faffff640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980039e0 0x7f5fb01f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5fb497f640 1 -- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb015bda0 msgr2=0x7f5fb00b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5fb497f640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb015bda0 0x7f5fb00b61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5fb497f640 1 -- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980039e0 msgr2=0x7f5fb01f3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5fb497f640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980039e0 0x7f5fb01f3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5fb497f640 1 -- 172.21.15.77:0/1611090416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00b6c90 con 0x7f5fb00b6710 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5faf7fe640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb015bda0 0x7f5fb00b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5faffff640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980039e0 0x7f5fb01f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5fb497f640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb00b6710 0x7f5fb00b7b20 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8011c80 tx=0x7f5fa800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5facff9640 1 -- 172.21.15.77:0/1611090416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8015430 con 0x7f5fb00b6710 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5facff9640 1 -- 172.21.15.77:0/1611090416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa80155f0 con 0x7f5fb00b6710 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb00cb140 con 0x7f5fb00b6710 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5facff9640 1 -- 172.21.15.77:0/1611090416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80159e0 con 0x7f5fb00b6710 2024-03-06T07:28:20.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.778+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb00cb700 con 0x7f5fb00b6710 2024-03-06T07:28:20.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.779+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb0160970 con 0x7f5fb00b6710 2024-03-06T07:28:20.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.780+0000 7f5facff9640 1 -- 172.21.15.77:0/1611090416 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fa8015b80 con 0x7f5fb00b6710 2024-03-06T07:28:20.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.780+0000 7f5facff9640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.780+0000 7f5facff9640 1 -- 172.21.15.77:0/1611090416 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f5fa808ad20 con 0x7f5fb00b6710 2024-03-06T07:28:20.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.780+0000 7f5faffff640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.780+0000 7f5faffff640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4033000 tx=0x7f5fa4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:20.788+0000 7f5facff9640 1 -- 172.21.15.77:0/1611090416 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fb0160970 con 0x7f5fb00b6710 2024-03-06T07:28:21.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.002+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fb004f330 con 0x7f5fb00b6710 2024-03-06T07:28:21.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.003+0000 7f5facff9640 1 -- 172.21.15.77:0/1611090416 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f5fa8024030 con 0x7f5fb00b6710 2024-03-06T07:28:21.006 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:21.006 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:04.849148+0000","modified":"2024-03-06T07:27:22.573254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8097},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8097":{"gid":8097,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6835/493819683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":493819683},{"type":"v1","addr":"172.21.15.77:6835","nonce":493819683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8097,"qdb_cluster":[8097]},"id":8}]} 2024-03-06T07:28:21.006 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 74 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f9c06b4d0 msgr2=0x7f5f9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4033000 tx=0x7f5fa4009030 comp rx=0 tx=0).stop 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb00b6710 msgr2=0x7f5fb00b7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb00b6710 0x7f5fb00b7b20 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8011c80 tx=0x7f5fa800f580 comp rx=0 tx=0).stop 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 shutdown_connections 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5f9c06b4d0 0x7f5f9c06d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb00b6710 0x7f5fb00b7b20 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5fb015bda0 0x7f5fb00b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 --2- 172.21.15.77:0/1611090416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f980039e0 0x7f5fb01f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 >> 172.21.15.77:0/1611090416 conn(0x7f5fb00d3cd0 msgr2=0x7f5fb00b2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 shutdown_connections 2024-03-06T07:28:21.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.005+0000 7f5fb63e2640 1 -- 172.21.15.77:0/1611090416 wait complete. 2024-03-06T07:28:21.019 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:21.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.117+0000 7f73b5df7640 1 Processor -- start 2024-03-06T07:28:21.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.118+0000 7f73b5df7640 1 -- start start 2024-03-06T07:28:21.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.118+0000 7f73b5df7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73b00b4170 0x7f73b00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.118+0000 7f73b5df7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b00cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.118+0000 7f73b5df7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b00d25d0 con 0x7f73b00d4760 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.118+0000 7f73b5df7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b00d4bf0 con 0x7f73b00b4170 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.118+0000 7f73b5df7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00ce9e0 con 0x7f73b00d0a80 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.118+0000 7f73af7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73b00b4170 0x7f73b00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.118+0000 7f73a7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f73b00d4760 0x7f73b00d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38786/0 (socket says 172.21.15.77:38786) 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.119+0000 7f73a7fff640 1 -- 172.21.15.77:0/3953477896 learned_addr learned my addr 172.21.15.77:0/3953477896 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.119+0000 7f73af7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73b00b4170 0x7f73b00b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59248/0 (socket says 172.21.15.77:59248) 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.119+0000 7f73affff640 1 --2- 172.21.15.77:0/3953477896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b00cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.119+0000 7f73af7fe640 1 -- 172.21.15.77:0/3953477896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 msgr2=0x7f73b00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.119+0000 7f73af7fe640 1 --2- 172.21.15.77:0/3953477896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b00cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.119+0000 7f73af7fe640 1 -- 172.21.15.77:0/3953477896 >> v1:172.21.15.77:6789/0 conn(0x7f73b00d4760 legacy=0x7f73b00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.119+0000 7f73af7fe640 1 -- 172.21.15.77:0/3953477896 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00d2870 con 0x7f73b00b4170 2024-03-06T07:28:21.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.119+0000 7f73aeffd640 1 -- 172.21.15.77:0/3953477896 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098583762 0 0) 0x7f73b00d25d0 con 0x7f73b00d4760 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73af7fe640 1 --2- 172.21.15.77:0/3953477896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b4170 0x7f73b00b6a90 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f73a00097a0 tx=0x7f73a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d4c60b9edb7184 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73aeffd640 1 -- 172.21.15.77:0/3953477896 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0035020 con 0x7f73b00b4170 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73aeffd640 1 -- 172.21.15.77:0/3953477896 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0031830 con 0x7f73b00b4170 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73aeffd640 1 -- 172.21.15.77:0/3953477896 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0031c00 con 0x7f73b00b4170 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73b5df7640 1 -- 172.21.15.77:0/3953477896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b4170 msgr2=0x7f73b00b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73b5df7640 1 --2- 172.21.15.77:0/3953477896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b4170 0x7f73b00b6a90 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f73a00097a0 tx=0x7f73a002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73b5df7640 1 -- 172.21.15.77:0/3953477896 shutdown_connections 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73b5df7640 1 --2- 172.21.15.77:0/3953477896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b00cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73b5df7640 1 --2- 172.21.15.77:0/3953477896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b4170 0x7f73b00b6a90 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73b5df7640 1 -- 172.21.15.77:0/3953477896 >> 172.21.15.77:0/3953477896 conn(0x7f73b00d3c40 msgr2=0x7f73b00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73b5df7640 1 -- 172.21.15.77:0/3953477896 shutdown_connections 2024-03-06T07:28:21.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.120+0000 7f73b5df7640 1 -- 172.21.15.77:0/3953477896 wait complete. 2024-03-06T07:28:21.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.122+0000 7f73b5df7640 1 Processor -- start 2024-03-06T07:28:21.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.122+0000 7f73b5df7640 1 -- start start 2024-03-06T07:28:21.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.123+0000 7f73b5df7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73b00b4170 0x7f73b00ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.123+0000 7f73b5df7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.123+0000 7f73b5df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b07f0 0x7f73b00b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.123+0000 7f73b5df7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b4dd0 con 0x7f73b00b07f0 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.123+0000 7f73b5df7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b4550 con 0x7f73b00b4170 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.123+0000 7f73b5df7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b4f50 con 0x7f73b00d0a80 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b07f0 0x7f73b00b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b07f0 0x7f73b00b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59256/0 (socket says 172.21.15.77:59256) 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 -- 172.21.15.77:0/2284798551 learned_addr learned my addr 172.21.15.77:0/2284798551 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73af7fe640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73b00b4170 0x7f73b00ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73a7fff640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 -- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 msgr2=0x7f73b015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 -- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73b00b4170 msgr2=0x7f73b00ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73b00b4170 0x7f73b00ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 -- 172.21.15.77:0/2284798551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00ca690 con 0x7f73b00b07f0 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73a7fff640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73affff640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b07f0 0x7f73b00b0ca0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f73a8007e30 tx=0x7f73a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73acff9640 1 -- 172.21.15.77:0/2284798551 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8014620 con 0x7f73b00b07f0 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73acff9640 1 -- 172.21.15.77:0/2284798551 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a8019070 con 0x7f73b00b07f0 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73acff9640 1 -- 172.21.15.77:0/2284798551 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8014a00 con 0x7f73b00b07f0 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b015cbf0 con 0x7f73b00b07f0 2024-03-06T07:28:21.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.124+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b015d1b0 con 0x7f73b00b07f0 2024-03-06T07:28:21.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.125+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b00d4370 con 0x7f73b00b07f0 2024-03-06T07:28:21.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.126+0000 7f73acff9640 1 -- 172.21.15.77:0/2284798551 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73a8014ba0 con 0x7f73b00b07f0 2024-03-06T07:28:21.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.126+0000 7f73acff9640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f739006b4d0 0x7f739006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.126+0000 7f73acff9640 1 -- 172.21.15.77:0/2284798551 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f73a808aaf0 con 0x7f73b00b07f0 2024-03-06T07:28:21.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.126+0000 7f73af7fe640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f739006b4d0 0x7f739006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.127+0000 7f73af7fe640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f739006b4d0 0x7f739006d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f73a0036000 tx=0x7f73a0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.131+0000 7f73acff9640 1 -- 172.21.15.77:0/2284798551 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73b00d4370 con 0x7f73b00b07f0 2024-03-06T07:28:21.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.361+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73b004f330 con 0x7f73b00b07f0 2024-03-06T07:28:21.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.362+0000 7f73acff9640 1 -- 172.21.15.77:0/2284798551 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+17027 (secure 0 0 0) 0x7f73a8059880 con 0x7f73b00b07f0 2024-03-06T07:28:21.366 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:21.366 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":113,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:27:48.616890+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:03.116010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:03.812681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.34","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6835/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6834/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6837/3247585937":"2024-03-07T07:27:18.550938+0000","172.21.15.77:6836/3247585937":"2024-03-07T07:27:18.550938+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.364+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f739006b4d0 msgr2=0x7f739006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.364+0000 7f73b5df7640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f739006b4d0 0x7f739006d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f73a0036000 tx=0x7f73a0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.364+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b07f0 msgr2=0x7f73b00b0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.364+0000 7f73b5df7640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b07f0 0x7f73b00b0ca0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f73a8007e30 tx=0x7f73a800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.365+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 shutdown_connections 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.365+0000 7f73b5df7640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f739006b4d0 0x7f739006d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.365+0000 7f73b5df7640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b00b07f0 0x7f73b00b0ca0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.365+0000 7f73b5df7640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73b00d0a80 0x7f73b015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.365+0000 7f73b5df7640 1 --2- 172.21.15.77:0/2284798551 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73b00b4170 0x7f73b00ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.365+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 >> 172.21.15.77:0/2284798551 conn(0x7f73b00d3c40 msgr2=0x7f73b006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.365+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 shutdown_connections 2024-03-06T07:28:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.365+0000 7f73b5df7640 1 -- 172.21.15.77:0/2284798551 wait complete. 2024-03-06T07:28:21.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.482+0000 7f96eecc9640 1 Processor -- start 2024-03-06T07:28:21.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.482+0000 7f96eecc9640 1 -- start start 2024-03-06T07:28:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.483+0000 7f96eecc9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f96e8162970 0x7f96e80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.483+0000 7f96eecc9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e80d22c0 con 0x7f96e80b4460 2024-03-06T07:28:21.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.483+0000 7f96eecc9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e80d25a0 con 0x7f96e8161cf0 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.483+0000 7f96eecc9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e8161b00 con 0x7f96e8162970 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.483+0000 7f96eca65640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f96e80b4460 0x7f96e80b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38790/0 (socket says 172.21.15.77:38790) 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.483+0000 7f96eca65640 1 -- 172.21.15.77:0/2591912494 learned_addr learned my addr 172.21.15.77:0/2591912494 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.483+0000 7f96e7fff640 1 --2- 172.21.15.77:0/2591912494 >> v2:172.21.15.77:3300/0 conn(0x7f96e8162970 0x7f96e80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.484+0000 7f96e77fe640 1 -- 172.21.15.77:0/2591912494 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165329843 0 0) 0x7f96e80d22c0 con 0x7f96e80b4460 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.484+0000 7f96e77fe640 1 -- 172.21.15.77:0/2591912494 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d8003520 con 0x7f96e80b4460 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.484+0000 7f96e7fff640 1 -- 172.21.15.77:0/2591912494 >> v1:172.21.15.158:6789/0 conn(0x7f96e8161cf0 legacy=0x7f96e80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.484+0000 7f96e7fff640 1 -- 172.21.15.77:0/2591912494 >> v1:172.21.15.77:6789/0 conn(0x7f96e80b4460 legacy=0x7f96e80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.484+0000 7f96e7fff640 1 -- 172.21.15.77:0/2591912494 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e80d2880 con 0x7f96e8162970 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96e7fff640 1 --2- 172.21.15.77:0/2591912494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e8162970 0x7f96e80d4800 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f96d40097a0 tx=0x7f96d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efddf6b3eb6b3436 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96e77fe640 1 -- 172.21.15.77:0/2591912494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4035020 con 0x7f96e8162970 2024-03-06T07:28:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96e77fe640 1 -- 172.21.15.77:0/2591912494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d4031830 con 0x7f96e8162970 2024-03-06T07:28:21.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96e77fe640 1 -- 172.21.15.77:0/2591912494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4031be0 con 0x7f96e8162970 2024-03-06T07:28:21.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96eecc9640 1 -- 172.21.15.77:0/2591912494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e8162970 msgr2=0x7f96e80d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96eecc9640 1 --2- 172.21.15.77:0/2591912494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e8162970 0x7f96e80d4800 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f96d40097a0 tx=0x7f96d402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96eecc9640 1 -- 172.21.15.77:0/2591912494 shutdown_connections 2024-03-06T07:28:21.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96eecc9640 1 --2- 172.21.15.77:0/2591912494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e8162970 0x7f96e80d4800 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96eecc9640 1 -- 172.21.15.77:0/2591912494 >> 172.21.15.77:0/2591912494 conn(0x7f96e80d3ce0 msgr2=0x7f96e80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96eecc9640 1 -- 172.21.15.77:0/2591912494 shutdown_connections 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.485+0000 7f96eecc9640 1 -- 172.21.15.77:0/2591912494 wait complete. 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.486+0000 7f96eecc9640 1 Processor -- start 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.486+0000 7f96eecc9640 1 -- start start 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.487+0000 7f96eecc9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e80b4460 0x7f96e80cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.487+0000 7f96eecc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96e8161cf0 0x7f96e80b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.487+0000 7f96eecc9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e8162970 0x7f96e81f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.487+0000 7f96eecc9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e80ce950 con 0x7f96e80b4460 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.487+0000 7f96eecc9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e80cecc0 con 0x7f96e8162970 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.487+0000 7f96eecc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e80b4840 con 0x7f96e8161cf0 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.487+0000 7f96eca65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e80b4460 0x7f96e80cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.487+0000 7f96eca65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e80b4460 0x7f96e80cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59272/0 (socket says 172.21.15.77:59272) 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eca65640 1 -- 172.21.15.77:0/2378941245 learned_addr learned my addr 172.21.15.77:0/2378941245 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96e7fff640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96e8161cf0 0x7f96e80b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96ed266640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e8162970 0x7f96e81f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eca65640 1 -- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96e8161cf0 msgr2=0x7f96e80b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eca65640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96e8161cf0 0x7f96e80b7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eca65640 1 -- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e8162970 msgr2=0x7f96e81f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eca65640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e8162970 0x7f96e81f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eca65640 1 -- 172.21.15.77:0/2378941245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e80b8240 con 0x7f96e80b4460 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96ed266640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e8162970 0x7f96e81f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96e7fff640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96e8161cf0 0x7f96e80b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eca65640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e80b4460 0x7f96e80cdba0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c390 tx=0x7f96dc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96e57fa640 1 -- 172.21.15.77:0/2378941245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc017070 con 0x7f96e80b4460 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96e57fa640 1 -- 172.21.15.77:0/2378941245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96dc005ce0 con 0x7f96e80b4460 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e80b8520 con 0x7f96e80b4460 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96e57fa640 1 -- 172.21.15.77:0/2378941245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc0072f0 con 0x7f96e80b4460 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.488+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e80b8ae0 con 0x7f96e80b4460 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.489+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e804f330 con 0x7f96e80b4460 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.490+0000 7f96e57fa640 1 -- 172.21.15.77:0/2378941245 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96dc01a070 con 0x7f96e80b4460 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.490+0000 7f96e57fa640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f96cc06b4d0 0x7f96cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.490+0000 7f96e57fa640 1 -- 172.21.15.77:0/2378941245 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f96dc089790 con 0x7f96e80b4460 2024-03-06T07:28:21.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.490+0000 7f96e7fff640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f96cc06b4d0 0x7f96cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.491+0000 7f96e7fff640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f96cc06b4d0 0x7f96cc06d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f96d4036000 tx=0x7f96d4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.494+0000 7f96e57fa640 1 -- 172.21.15.77:0/2378941245 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96dc058490 con 0x7f96e80b4460 2024-03-06T07:28:21.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.710+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f96e806b180 con 0x7f96e80b4460 2024-03-06T07:28:21.713 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:28:21.711+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:21.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.712+0000 7f96e57fa640 1 -- 172.21.15.77:0/2378941245 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v75) v1 ==== 161+0+0 (secure 0 0 0) 0x7f96dc057d90 con 0x7f96e80b4460 2024-03-06T07:28:21.715 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.714+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f96cc06b4d0 msgr2=0x7f96cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.714+0000 7f96eecc9640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f96cc06b4d0 0x7f96cc06d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f96d4036000 tx=0x7f96d4008030 comp rx=0 tx=0).stop 2024-03-06T07:28:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.714+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e80b4460 msgr2=0x7f96e80cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.714+0000 7f96eecc9640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e80b4460 0x7f96e80cdba0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c390 tx=0x7f96dc00c850 comp rx=0 tx=0).stop 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.715+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 shutdown_connections 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.715+0000 7f96eecc9640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f96e8162970 0x7f96e81f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.715+0000 7f96eecc9640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96e8161cf0 0x7f96e80b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.715+0000 7f96eecc9640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f96cc06b4d0 0x7f96cc06d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.715+0000 7f96eecc9640 1 --2- 172.21.15.77:0/2378941245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96e80b4460 0x7f96e80cdba0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.715+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 >> 172.21.15.77:0/2378941245 conn(0x7f96e80d3ce0 msgr2=0x7f96e80d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.715+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 shutdown_connections 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.715+0000 7f96eecc9640 1 -- 172.21.15.77:0/2378941245 wait complete. 2024-03-06T07:28:21.721 INFO:tasks.ceph.mds.c.smithi077.stderr: -8790> 2024-03-06T07:27:24.563+0000 7f0ee3a43640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:21.730 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:21.754 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:28:21.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17dca0e640 1 Processor -- start 2024-03-06T07:28:21.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17dca0e640 1 -- start start 2024-03-06T07:28:21.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17dca0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17dca0e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f17d80d1910 0x7f17d80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17dca0e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d80d2740 con 0x7f17d80d22b0 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17dca0e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d81618b0 con 0x7f17d80d1910 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17dca0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80cf400 con 0x7f17d81614d0 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17d5d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f17d80d1910 0x7f17d80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17d5d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f17d80d1910 0x7f17d80d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59314/0 (socket says 172.21.15.77:59314) 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.830+0000 7f17d5d9b640 1 -- 172.21.15.77:0/347253511 learned_addr learned my addr 172.21.15.77:0/347253511 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.831+0000 7f17d659c640 1 --2- 172.21.15.77:0/347253511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.831+0000 7f17d5d9b640 1 -- 172.21.15.77:0/347253511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 msgr2=0x7f17d80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.831+0000 7f17d5d9b640 1 --2- 172.21.15.77:0/347253511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.831+0000 7f17d5d9b640 1 -- 172.21.15.77:0/347253511 >> v1:172.21.15.158:6789/0 conn(0x7f17d80d22b0 legacy=0x7f17d80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.831+0000 7f17d5d9b640 1 -- 172.21.15.77:0/347253511 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d80cf6a0 con 0x7f17d80d1910 2024-03-06T07:28:21.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.831+0000 7f17d5d9b640 1 --2- 172.21.15.77:0/347253511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d1910 0x7f17d80d1d70 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009870 tx=0x7f17cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da47bfbacfdd8d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17d559a640 1 -- 172.21.15.77:0/347253511 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc034070 con 0x7f17d80d1910 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17d559a640 1 -- 172.21.15.77:0/347253511 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc030840 con 0x7f17d80d1910 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17d559a640 1 -- 172.21.15.77:0/347253511 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc030c10 con 0x7f17d80d1910 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17dca0e640 1 -- 172.21.15.77:0/347253511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d1910 msgr2=0x7f17d80d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17dca0e640 1 --2- 172.21.15.77:0/347253511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d1910 0x7f17d80d1d70 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009870 tx=0x7f17cc02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17dca0e640 1 -- 172.21.15.77:0/347253511 shutdown_connections 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17dca0e640 1 --2- 172.21.15.77:0/347253511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d1910 0x7f17d80d1d70 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17dca0e640 1 --2- 172.21.15.77:0/347253511 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17dca0e640 1 -- 172.21.15.77:0/347253511 >> 172.21.15.77:0/347253511 conn(0x7f17d80d3d40 msgr2=0x7f17d80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17dca0e640 1 -- 172.21.15.77:0/347253511 shutdown_connections 2024-03-06T07:28:21.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.832+0000 7f17dca0e640 1 -- 172.21.15.77:0/347253511 wait complete. 2024-03-06T07:28:21.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.833+0000 7f17dca0e640 1 Processor -- start 2024-03-06T07:28:21.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17dca0e640 1 -- start start 2024-03-06T07:28:21.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17dca0e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d80d1910 0x7f17d8153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17dca0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d22b0 0x7f17d81f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17dca0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80b83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17dca0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80b0910 con 0x7f17d80d22b0 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17dca0e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80cfc60 con 0x7f17d80d1910 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17dca0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80b1e40 con 0x7f17d81614d0 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17d5d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d22b0 0x7f17d81f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17d5d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d22b0 0x7f17d81f4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59320/0 (socket says 172.21.15.77:59320) 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.834+0000 7f17d5d9b640 1 -- 172.21.15.77:0/1135760254 learned_addr learned my addr 172.21.15.77:0/1135760254 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d6d9d640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80b83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d659c640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d80d1910 0x7f17d8153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d5d9b640 1 -- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 msgr2=0x7f17d80b83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d5d9b640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80b83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d5d9b640 1 -- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d80d1910 msgr2=0x7f17d8153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d5d9b640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d80d1910 0x7f17d8153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d5d9b640 1 -- 172.21.15.77:0/1135760254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d81f51a0 con 0x7f17d80d22b0 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d6d9d640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80b83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d659c640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d80d1910 0x7f17d8153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:21.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17d5d9b640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d22b0 0x7f17d81f4c60 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f17cc02b9c0 tx=0x7f17cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17c6ffd640 1 -- 172.21.15.77:0/1135760254 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc048070 con 0x7f17d80d22b0 2024-03-06T07:28:21.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d80c7a10 con 0x7f17d80d22b0 2024-03-06T07:28:21.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17c6ffd640 1 -- 172.21.15.77:0/1135760254 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc030640 con 0x7f17d80d22b0 2024-03-06T07:28:21.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17c6ffd640 1 -- 172.21.15.77:0/1135760254 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc034040 con 0x7f17d80d22b0 2024-03-06T07:28:21.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.835+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d80c7fd0 con 0x7f17d80d22b0 2024-03-06T07:28:21.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.836+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d8068ff0 con 0x7f17d80d22b0 2024-03-06T07:28:21.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.836+0000 7f17c6ffd640 1 -- 172.21.15.77:0/1135760254 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17cc037070 con 0x7f17d80d22b0 2024-03-06T07:28:21.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.837+0000 7f17c6ffd640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17b806b4d0 0x7f17b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.837+0000 7f17c6ffd640 1 -- 172.21.15.77:0/1135760254 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f17cc0abfd0 con 0x7f17d80d22b0 2024-03-06T07:28:21.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.837+0000 7f17d659c640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17b806b4d0 0x7f17b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.837+0000 7f17d659c640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17b806b4d0 0x7f17b806d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f17d80b4de0 tx=0x7f17c80039f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:21.841+0000 7f17c6ffd640 1 -- 172.21.15.77:0/1135760254 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17cc07ae40 con 0x7f17d80d22b0 2024-03-06T07:28:22.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.049+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f17d80ce9b0 con 0x7f17d80d22b0 2024-03-06T07:28:22.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.714+0000 7f17c6ffd640 1 -- 172.21.15.77:0/1135760254 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v76) v1 ==== 104+0+0 (secure 0 0 0) 0x7f17cc07a740 con 0x7f17d80d22b0 2024-03-06T07:28:22.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17b806b4d0 msgr2=0x7f17b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17b806b4d0 0x7f17b806d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f17d80b4de0 tx=0x7f17c80039f0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d22b0 msgr2=0x7f17d81f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d22b0 0x7f17d81f4c60 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f17cc02b9c0 tx=0x7f17cc004730 comp rx=0 tx=0).stop 2024-03-06T07:28:22.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 shutdown_connections 2024-03-06T07:28:22.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17b806b4d0 0x7f17b806d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80d22b0 0x7f17d81f4c60 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f17d81614d0 0x7f17d80b83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 --2- 172.21.15.77:0/1135760254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d80d1910 0x7f17d8153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 >> 172.21.15.77:0/1135760254 conn(0x7f17d80d3d40 msgr2=0x7f17d806bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 shutdown_connections 2024-03-06T07:28:22.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.717+0000 7f17dca0e640 1 -- 172.21.15.77:0/1135760254 wait complete. 2024-03-06T07:28:22.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:22.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.879+0000 7f8df9b4b640 1 Processor -- start 2024-03-06T07:28:22.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.879+0000 7f8df9b4b640 1 -- start start 2024-03-06T07:28:22.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df9b4b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8df415ac00 0x7f8df406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df9b4b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8df4070d10 0x7f8df415eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df9b4b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df40711c0 con 0x7f8df40708b0 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df9b4b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df40cdc10 con 0x7f8df415ac00 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df9b4b640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df415afe0 con 0x7f8df4070d10 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8df415ac00 0x7f8df406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8df415ac00 0x7f8df406c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59336/0 (socket says 172.21.15.77:59336) 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df37fe640 1 -- 172.21.15.77:0/1964625384 learned_addr learned my addr 172.21.15.77:0/1964625384 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df37fe640 1 -- 172.21.15.77:0/1964625384 >> v2:172.21.15.158:3300/0 conn(0x7f8df4070d10 msgr2=0x7f8df415eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df3fff640 1 --2- 172.21.15.77:0/1964625384 >> v2:172.21.15.158:3300/0 conn(0x7f8df4070d10 0x7f8df415eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df37fe640 1 --2- 172.21.15.77:0/1964625384 >> v2:172.21.15.158:3300/0 conn(0x7f8df4070d10 0x7f8df415eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df37fe640 1 -- 172.21.15.77:0/1964625384 >> v1:172.21.15.158:6789/0 conn(0x7f8df40708b0 legacy=0x7f8df415e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.880+0000 7f8df37fe640 1 -- 172.21.15.77:0/1964625384 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df40cd000 con 0x7f8df415ac00 2024-03-06T07:28:22.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.881+0000 7f8df3fff640 1 --2- 172.21.15.77:0/1964625384 >> v2:172.21.15.158:3300/0 conn(0x7f8df4070d10 0x7f8df415eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.881+0000 7f8df37fe640 1 --2- 172.21.15.77:0/1964625384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df406c910 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f8df40cc0a0 tx=0x7f8de402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44b84c0a2c69d74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.881+0000 7f8df27fc640 1 -- 172.21.15.77:0/1964625384 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de403b070 con 0x7f8df415ac00 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.881+0000 7f8df27fc640 1 -- 172.21.15.77:0/1964625384 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de402ee00 con 0x7f8df415ac00 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.881+0000 7f8df27fc640 1 -- 172.21.15.77:0/1964625384 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4036660 con 0x7f8df415ac00 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.882+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1964625384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 msgr2=0x7f8df406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.882+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1964625384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df406c910 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f8df40cc0a0 tx=0x7f8de402e050 comp rx=0 tx=0).stop 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.882+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1964625384 shutdown_connections 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.882+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1964625384 >> v2:172.21.15.158:3300/0 conn(0x7f8df4070d10 0x7f8df415eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.882+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1964625384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df406c910 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.882+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1964625384 >> 172.21.15.77:0/1964625384 conn(0x7f8df40b1650 msgr2=0x7f8df40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.882+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1964625384 shutdown_connections 2024-03-06T07:28:22.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.882+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1964625384 wait complete. 2024-03-06T07:28:22.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.883+0000 7f8df9b4b640 1 Processor -- start 2024-03-06T07:28:22.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.883+0000 7f8df9b4b640 1 -- start start 2024-03-06T07:28:22.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df9b4b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df40708b0 0x7f8df41f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df9b4b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8df4070d10 0x7f8df4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df9b4b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df415c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df9b4b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40cd8a0 con 0x7f8df415ac00 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df9b4b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40cd5c0 con 0x7f8df40708b0 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df9b4b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df406ce50 con 0x7f8df4070d10 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df415c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59342/0 (socket says 172.21.15.77:59342) 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df3fff640 1 -- 172.21.15.77:0/1581039628 learned_addr learned my addr 172.21.15.77:0/1581039628 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df3fff640 1 -- 172.21.15.77:0/1581039628 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8df4070d10 msgr2=0x7f8df4159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df3fff640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8df4070d10 0x7f8df4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df3fff640 1 -- 172.21.15.77:0/1581039628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df40708b0 msgr2=0x7f8df41f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df3fff640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df40708b0 0x7f8df41f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.884+0000 7f8df3fff640 1 -- 172.21.15.77:0/1581039628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df40b3e30 con 0x7f8df415ac00 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.885+0000 7f8df3fff640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df415c610 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00cef0 tx=0x7f8dec00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.885+0000 7f8ddbfff640 1 -- 172.21.15.77:0/1581039628 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dec01a070 con 0x7f8df415ac00 2024-03-06T07:28:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.885+0000 7f8ddbfff640 1 -- 172.21.15.77:0/1581039628 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dec00d860 con 0x7f8df415ac00 2024-03-06T07:28:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.885+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df415b350 con 0x7f8df415ac00 2024-03-06T07:28:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.885+0000 7f8ddbfff640 1 -- 172.21.15.77:0/1581039628 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dec00dc50 con 0x7f8df415ac00 2024-03-06T07:28:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.885+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df415b910 con 0x7f8df415ac00 2024-03-06T07:28:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.886+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df40687d0 con 0x7f8df415ac00 2024-03-06T07:28:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.886+0000 7f8ddbfff640 1 -- 172.21.15.77:0/1581039628 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dec01d070 con 0x7f8df415ac00 2024-03-06T07:28:22.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.887+0000 7f8ddbfff640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8de006b4d0 0x7f8de006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.887+0000 7f8ddbfff640 1 -- 172.21.15.77:0/1581039628 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f8dec08ab80 con 0x7f8df415ac00 2024-03-06T07:28:22.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.887+0000 7f8df37fe640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8de006b4d0 0x7f8de006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.887+0000 7f8df37fe640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8de006b4d0 0x7f8de006d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8de4038000 tx=0x7f8de4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:22.891+0000 7f8ddbfff640 1 -- 172.21.15.77:0/1581039628 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8df40b3e30 con 0x7f8df415ac00 2024-03-06T07:28:23.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.061+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8df415a3c0 con 0x7f8df415ac00 2024-03-06T07:28:23.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.719+0000 7f8ddbfff640 1 -- 172.21.15.77:0/1581039628 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8dec059a10 con 0x7f8df415ac00 2024-03-06T07:28:23.721 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.722+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8de006b4d0 msgr2=0x7f8de006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.722+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8de006b4d0 0x7f8de006d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8de4038000 tx=0x7f8de4030000 comp rx=0 tx=0).stop 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.722+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 msgr2=0x7f8df415c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.722+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df415c610 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00cef0 tx=0x7f8dec00a710 comp rx=0 tx=0).stop 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.723+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 shutdown_connections 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.723+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8de006b4d0 0x7f8de006d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.723+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df415ac00 0x7f8df415c610 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.723+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8df4070d10 0x7f8df4159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.723+0000 7f8df9b4b640 1 --2- 172.21.15.77:0/1581039628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df40708b0 0x7f8df41f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.723+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 >> 172.21.15.77:0/1581039628 conn(0x7f8df40b1650 msgr2=0x7f8df40afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.723+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 shutdown_connections 2024-03-06T07:28:23.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.723+0000 7f8df9b4b640 1 -- 172.21.15.77:0/1581039628 wait complete. 2024-03-06T07:28:23.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.847+0000 7f4c9dac2640 1 Processor -- start 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.848+0000 7f4c9dac2640 1 -- start start 2024-03-06T07:28:23.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.848+0000 7f4c9dac2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4c980d29f0 0x7f4c98168ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.848+0000 7f4c9dac2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9815e380 con 0x7f4c9815dfa0 2024-03-06T07:28:23.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.848+0000 7f4c9dac2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c980cf470 con 0x7f4c980d4630 2024-03-06T07:28:23.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.848+0000 7f4c9dac2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c980d2100 con 0x7f4c980d29f0 2024-03-06T07:28:23.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.848+0000 7f4c977fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4c9815dfa0 0x7f4c980d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38808/0 (socket says 172.21.15.77:38808) 2024-03-06T07:28:23.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.848+0000 7f4c977fe640 1 -- 172.21.15.77:0/3942100260 learned_addr learned my addr 172.21.15.77:0/3942100260 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:23.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229212933 0 0) 0x7f4c9815e380 con 0x7f4c9815dfa0 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c80003520 con 0x7f4c9815dfa0 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c97fff640 1 --2- 172.21.15.77:0/3942100260 >> v2:172.21.15.158:3300/0 conn(0x7f4c980d29f0 0x7f4c98168ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3970775468 0 0) 0x7f4c80003520 con 0x7f4c9815dfa0 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 >> v2:172.21.15.158:3300/0 conn(0x7f4c980d29f0 msgr2=0x7f4c98168ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 --2- 172.21.15.77:0/3942100260 >> v2:172.21.15.158:3300/0 conn(0x7f4c980d29f0 0x7f4c98168ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 >> v1:172.21.15.158:6789/0 conn(0x7f4c980d4630 legacy=0x7f4c980d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c980cf750 con 0x7f4c9815dfa0 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f4c88001670 con 0x7f4c9815dfa0 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 >> v1:172.21.15.77:6789/0 conn(0x7f4c9815dfa0 legacy=0x7f4c980d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c800039b0 0x7f4c80003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c80004800 0x7f4c80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c800051c0 0x7f4c8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d2100 con 0x7f4c800051c0 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8000c160 con 0x7f4c800039b0 2024-03-06T07:28:23.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8000c360 con 0x7f4c80004800 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c97fff640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c800051c0 0x7f4c8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c977fe640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c800039b0 0x7f4c80003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c96ffd640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c80004800 0x7f4c80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c97fff640 1 -- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c80004800 msgr2=0x7f4c80004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c97fff640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c80004800 0x7f4c80004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c97fff640 1 -- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c800039b0 msgr2=0x7f4c80003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c97fff640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c800039b0 0x7f4c80003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c97fff640 1 -- 172.21.15.77:0/3942100260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c80003520 con 0x7f4c800051c0 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.849+0000 7f4c96ffd640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c80004800 0x7f4c80004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.850+0000 7f4c97fff640 1 -- 172.21.15.77:0/3942100260 reap_dead start 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.850+0000 7f4c97fff640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c800051c0 0x7f4c8000b9c0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009f90 tx=0x7f4c80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9013c0bbc44766 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.850+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90009c10 con 0x7f4c800051c0 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.850+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9002cc20 con 0x7f4c800051c0 2024-03-06T07:28:23.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.850+0000 7f4c967fc640 1 -- 172.21.15.77:0/3942100260 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90003990 con 0x7f4c800051c0 2024-03-06T07:28:23.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.851+0000 7f4c9dac2640 1 -- 172.21.15.77:0/3942100260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c800051c0 msgr2=0x7f4c8000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.851+0000 7f4c9dac2640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c800051c0 0x7f4c8000b9c0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009f90 tx=0x7f4c80005830 comp rx=0 tx=0).stop 2024-03-06T07:28:23.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.851+0000 7f4c9dac2640 1 -- 172.21.15.77:0/3942100260 shutdown_connections 2024-03-06T07:28:23.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.851+0000 7f4c9dac2640 1 --2- 172.21.15.77:0/3942100260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c800051c0 0x7f4c8000b9c0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.851+0000 7f4c9dac2640 1 -- 172.21.15.77:0/3942100260 >> 172.21.15.77:0/3942100260 conn(0x7f4c9809d000 msgr2=0x7f4c980d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.851+0000 7f4c9dac2640 1 -- 172.21.15.77:0/3942100260 shutdown_connections 2024-03-06T07:28:23.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.851+0000 7f4c9dac2640 1 -- 172.21.15.77:0/3942100260 wait complete. 2024-03-06T07:28:23.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.852+0000 7f4c9dac2640 1 Processor -- start 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.852+0000 7f4c9dac2640 1 -- start start 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c9dac2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c800051c0 0x7f4c9809dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c9dac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c980c9a70 0x7f4c980b7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c9dac2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c980c9ef0 0x7f4c980b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c9dac2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d2460 con 0x7f4c980c9a70 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c9dac2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d22e0 con 0x7f4c980c9ef0 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c9dac2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980d25e0 con 0x7f4c800051c0 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c980c9a70 0x7f4c980b7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c980c9a70 0x7f4c980b7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59356/0 (socket says 172.21.15.77:59356) 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 -- 172.21.15.77:0/1607057419 learned_addr learned my addr 172.21.15.77:0/1607057419 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 -- 172.21.15.77:0/1607057419 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c800051c0 msgr2=0x7f4c9809dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c800051c0 0x7f4c9809dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 -- 172.21.15.77:0/1607057419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c980c9ef0 msgr2=0x7f4c980b7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c980c9ef0 0x7f4c980b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 -- 172.21.15.77:0/1607057419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c980c89d0 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c96ffd640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c980c9a70 0x7f4c980b7560 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c007810 tx=0x7f4c8c00e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c7ffff640 1 -- 172.21.15.77:0/1607057419 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c019070 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c7ffff640 1 -- 172.21.15.77:0/1607057419 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8c0052e0 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.853+0000 7f4c7ffff640 1 -- 172.21.15.77:0/1607057419 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c005490 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.854+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c980c8cb0 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.854+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c980c9270 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.855+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9804f330 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.855+0000 7f4c7ffff640 1 -- 172.21.15.77:0/1607057419 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c8c01c070 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.855+0000 7f4c7ffff640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4c8406b500 0x7f4c8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.856+0000 7f4c7ffff640 1 -- 172.21.15.77:0/1607057419 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5587+0+0 (secure 0 0 0) 0x7f4c8c0896b0 con 0x7f4c980c9a70 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.856+0000 7f4c977fe640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4c8406b500 0x7f4c8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.856+0000 7f4c977fe640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4c8406b500 0x7f4c8406d9c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4c88000ff0 tx=0x7f4c88001ad0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:23.860+0000 7f4c7ffff640 1 -- 172.21.15.77:0/1607057419 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c8c057fe0 con 0x7f4c980c9a70 2024-03-06T07:28:24.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.031+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4c9815d550 con 0x7f4c980c9a70 2024-03-06T07:28:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.721+0000 7f4c7ffff640 1 -- 172.21.15.77:0/1607057419 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4c8c05c080 con 0x7f4c980c9a70 2024-03-06T07:28:24.723 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.724+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4c8406b500 msgr2=0x7f4c8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.724+0000 7f4c9dac2640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4c8406b500 0x7f4c8406d9c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4c88000ff0 tx=0x7f4c88001ad0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.724+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c980c9a70 msgr2=0x7f4c980b7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.724+0000 7f4c9dac2640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c980c9a70 0x7f4c980b7560 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c007810 tx=0x7f4c8c00e910 comp rx=0 tx=0).stop 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.725+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 shutdown_connections 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.725+0000 7f4c9dac2640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4c8406b500 0x7f4c8406d9c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.725+0000 7f4c9dac2640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c980c9a70 0x7f4c980b7560 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.725+0000 7f4c9dac2640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4c980c9ef0 0x7f4c980b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.725+0000 7f4c9dac2640 1 --2- 172.21.15.77:0/1607057419 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4c800051c0 0x7f4c9809dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.725+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 >> 172.21.15.77:0/1607057419 conn(0x7f4c9809d000 msgr2=0x7f4c981605a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.725+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 shutdown_connections 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.726+0000 7f4c9dac2640 1 -- 172.21.15.77:0/1607057419 wait complete. 2024-03-06T07:28:24.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.848+0000 7f1d45c34640 1 Processor -- start 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.848+0000 7f1d45c34640 1 -- start start 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.848+0000 7f1d45c34640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1d400d4860 0x7f1d400d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.848+0000 7f1d45c34640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.848+0000 7f1d45c34640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d401618b0 con 0x7f1d401614d0 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.848+0000 7f1d45c34640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d400d4cf0 con 0x7f1d400d4860 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.848+0000 7f1d45c34640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400cf400 con 0x7f1d400b0e30 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d36ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1d400d4860 0x7f1d400d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d3ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d36ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1d400d4860 0x7f1d400d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:43186/0 (socket says 172.21.15.77:43186) 2024-03-06T07:28:24.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d36ffd640 1 -- 172.21.15.77:0/4253556749 learned_addr learned my addr 172.21.15.77:0/4253556749 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d36ffd640 1 -- 172.21.15.77:0/4253556749 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 msgr2=0x7f1d400b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d36ffd640 1 --2- 172.21.15.77:0/4253556749 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d36ffd640 1 -- 172.21.15.77:0/4253556749 >> v1:172.21.15.158:6789/0 conn(0x7f1d401614d0 legacy=0x7f1d400d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d36ffd640 1 -- 172.21.15.77:0/4253556749 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d400cf6a0 con 0x7f1d400d4860 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.849+0000 7f1d3ffff640 1 --2- 172.21.15.77:0/4253556749 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.850+0000 7f1d36ffd640 1 --2- 172.21.15.77:0/4253556749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d400d4860 0x7f1d400d2020 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009870 tx=0x7f1d2802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2e4212992ed2ae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.850+0000 7f1d3effd640 1 -- 172.21.15.77:0/4253556749 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28034070 con 0x7f1d400d4860 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.850+0000 7f1d3effd640 1 -- 172.21.15.77:0/4253556749 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d28030840 con 0x7f1d400d4860 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.850+0000 7f1d3effd640 1 -- 172.21.15.77:0/4253556749 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28030c10 con 0x7f1d400d4860 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.851+0000 7f1d45c34640 1 -- 172.21.15.77:0/4253556749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d400d4860 msgr2=0x7f1d400d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.851+0000 7f1d45c34640 1 --2- 172.21.15.77:0/4253556749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d400d4860 0x7f1d400d2020 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009870 tx=0x7f1d2802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.851+0000 7f1d45c34640 1 -- 172.21.15.77:0/4253556749 shutdown_connections 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.851+0000 7f1d45c34640 1 --2- 172.21.15.77:0/4253556749 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.851+0000 7f1d45c34640 1 --2- 172.21.15.77:0/4253556749 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d400d4860 0x7f1d400d2020 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.851+0000 7f1d45c34640 1 -- 172.21.15.77:0/4253556749 >> 172.21.15.77:0/4253556749 conn(0x7f1d400d3d40 msgr2=0x7f1d400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.851+0000 7f1d45c34640 1 -- 172.21.15.77:0/4253556749 shutdown_connections 2024-03-06T07:28:24.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.851+0000 7f1d45c34640 1 -- 172.21.15.77:0/4253556749 wait complete. 2024-03-06T07:28:24.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.852+0000 7f1d45c34640 1 Processor -- start 2024-03-06T07:28:24.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.852+0000 7f1d45c34640 1 -- start start 2024-03-06T07:28:24.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d45c34640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d45c34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d400d4860 0x7f1d4014f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d45c34640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d401614d0 0x7f1d400c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d45c34640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400b0910 con 0x7f1d400d4860 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d45c34640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400cfc60 con 0x7f1d401614d0 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d45c34640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400d2560 con 0x7f1d400b0e30 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d400d4860 0x7f1d4014f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d400d4860 0x7f1d4014f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59366/0 (socket says 172.21.15.77:59366) 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 -- 172.21.15.77:0/3823770281 learned_addr learned my addr 172.21.15.77:0/3823770281 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d3f7fe640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d3ffff640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d401614d0 0x7f1d400c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 -- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 msgr2=0x7f1d400b9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 -- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d401614d0 msgr2=0x7f1d400c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d401614d0 0x7f1d400c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 -- 172.21.15.77:0/3823770281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d400b7ee0 con 0x7f1d400d4860 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d3f7fe640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d3ffff640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d401614d0 0x7f1d400c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.853+0000 7f1d36ffd640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d400d4860 0x7f1d4014f190 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1d28000c00 tx=0x7f1d28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.854+0000 7f1d3cff9640 1 -- 172.21.15.77:0/3823770281 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28048070 con 0x7f1d400d4860 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.854+0000 7f1d3cff9640 1 -- 172.21.15.77:0/3823770281 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d28030640 con 0x7f1d400d4860 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.854+0000 7f1d3cff9640 1 -- 172.21.15.77:0/3823770281 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28034040 con 0x7f1d400d4860 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.854+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d400b8180 con 0x7f1d400d4860 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.854+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d400b8740 con 0x7f1d400d4860 2024-03-06T07:28:24.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.855+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d40068ff0 con 0x7f1d400d4860 2024-03-06T07:28:24.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.855+0000 7f1d3cff9640 1 -- 172.21.15.77:0/3823770281 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d2804b070 con 0x7f1d400d4860 2024-03-06T07:28:24.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.856+0000 7f1d3cff9640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d2006b500 0x7f1d2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.856+0000 7f1d3cff9640 1 -- 172.21.15.77:0/3823770281 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f1d280abe60 con 0x7f1d400d4860 2024-03-06T07:28:24.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.856+0000 7f1d3f7fe640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d2006b500 0x7f1d2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.856+0000 7f1d3f7fe640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d2006b500 0x7f1d2006d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1d300015e0 tx=0x7f1d30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:24.860+0000 7f1d3cff9640 1 -- 172.21.15.77:0/3823770281 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d2807b050 con 0x7f1d400d4860 2024-03-06T07:28:25.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.033+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1d400b48c0 con 0x7f1d400d4860 2024-03-06T07:28:25.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.036+0000 7f1d3cff9640 1 -- 172.21.15.77:0/3823770281 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1d28030810 con 0x7f1d400d4860 2024-03-06T07:28:25.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d2006b500 msgr2=0x7f1d2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d2006b500 0x7f1d2006d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1d300015e0 tx=0x7f1d30009040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d400d4860 msgr2=0x7f1d4014f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d400d4860 0x7f1d4014f190 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1d28000c00 tx=0x7f1d28004730 comp rx=0 tx=0).stop 2024-03-06T07:28:25.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 shutdown_connections 2024-03-06T07:28:25.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d2006b500 0x7f1d2006d9c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d400d4860 0x7f1d4014f190 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d401614d0 0x7f1d400c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 --2- 172.21.15.77:0/3823770281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d400b0e30 0x7f1d400b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 >> 172.21.15.77:0/3823770281 conn(0x7f1d400d3d40 msgr2=0x7f1d400cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 shutdown_connections 2024-03-06T07:28:25.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.038+0000 7f1d45c34640 1 -- 172.21.15.77:0/3823770281 wait complete. 2024-03-06T07:28:25.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:28:25.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.149+0000 7f21de7ac640 1 Processor -- start 2024-03-06T07:28:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.149+0000 7f21de7ac640 1 -- start start 2024-03-06T07:28:25.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.149+0000 7f21de7ac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f21d815dfe0 0x7f21d80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.149+0000 7f21de7ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80cf830 0x7f21d80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.149+0000 7f21de7ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f21d80d27f0 0x7f21d80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.149+0000 7f21de7ac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80d2c50 con 0x7f21d80d27f0 2024-03-06T07:28:25.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.149+0000 7f21de7ac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80b31e0 con 0x7f21d815dfe0 2024-03-06T07:28:25.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.149+0000 7f21de7ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80b3d60 con 0x7f21d80cf830 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21dcd49640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f21d80d27f0 0x7f21d80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21dcd49640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f21d80d27f0 0x7f21d80d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59368/0 (socket says 172.21.15.77:59368) 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21dcd49640 1 -- 172.21.15.77:0/3938441932 learned_addr learned my addr 172.21.15.77:0/3938441932 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21d77fe640 1 --2- 172.21.15.77:0/3938441932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80cf830 0x7f21d80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21d7fff640 1 --2- 172.21.15.77:0/3938441932 >> v2:172.21.15.158:3300/0 conn(0x7f21d815dfe0 0x7f21d80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21dcd49640 1 -- 172.21.15.77:0/3938441932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80cf830 msgr2=0x7f21d80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21dcd49640 1 --2- 172.21.15.77:0/3938441932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80cf830 0x7f21d80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21dcd49640 1 -- 172.21.15.77:0/3938441932 >> v2:172.21.15.158:3300/0 conn(0x7f21d815dfe0 msgr2=0x7f21d80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21dcd49640 1 --2- 172.21.15.77:0/3938441932 >> v2:172.21.15.158:3300/0 conn(0x7f21d815dfe0 0x7f21d80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21dcd49640 1 -- 172.21.15.77:0/3938441932 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80b3430 con 0x7f21d80d27f0 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21d77fe640 1 --2- 172.21.15.77:0/3938441932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80cf830 0x7f21d80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.150+0000 7f21d7fff640 1 --2- 172.21.15.77:0/3938441932 >> v2:172.21.15.158:3300/0 conn(0x7f21d815dfe0 0x7f21d80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21dcd49640 1 --2- 172.21.15.77:0/3938441932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d80d27f0 0x7f21d80d0300 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f21d8067990 tx=0x7f21d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1ba4c68f52e9d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21d6ffd640 1 -- 172.21.15.77:0/3938441932 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d002ca70 con 0x7f21d80d27f0 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21d6ffd640 1 -- 172.21.15.77:0/3938441932 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d002cc10 con 0x7f21d80d27f0 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21d6ffd640 1 -- 172.21.15.77:0/3938441932 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d00343f0 con 0x7f21d80d27f0 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 -- 172.21.15.77:0/3938441932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d80d27f0 msgr2=0x7f21d80d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3938441932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d80d27f0 0x7f21d80d0300 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f21d8067990 tx=0x7f21d002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 -- 172.21.15.77:0/3938441932 shutdown_connections 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3938441932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d80d27f0 0x7f21d80d0300 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3938441932 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80cf830 0x7f21d80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3938441932 >> v2:172.21.15.158:3300/0 conn(0x7f21d815dfe0 0x7f21d80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 -- 172.21.15.77:0/3938441932 >> 172.21.15.77:0/3938441932 conn(0x7f21d80d3cd0 msgr2=0x7f21d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 -- 172.21.15.77:0/3938441932 shutdown_connections 2024-03-06T07:28:25.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.151+0000 7f21de7ac640 1 -- 172.21.15.77:0/3938441932 wait complete. 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.152+0000 7f21de7ac640 1 Processor -- start 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21de7ac640 1 -- start start 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21de7ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d80cf830 0x7f21d814f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21de7ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80d27f0 0x7f21d815e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21de7ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815dfe0 0x7f21d815bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21de7ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80d1c70 con 0x7f21d815dfe0 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21de7ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80b39f0 con 0x7f21d80cf830 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21de7ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80d1df0 con 0x7f21d80d27f0 2024-03-06T07:28:25.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21dcd49640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815dfe0 0x7f21d815bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21dcd49640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815dfe0 0x7f21d815bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59382/0 (socket says 172.21.15.77:59382) 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.153+0000 7f21dcd49640 1 -- 172.21.15.77:0/3572619523 learned_addr learned my addr 172.21.15.77:0/3572619523 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21d77fe640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80d27f0 0x7f21d815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21d7fff640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d80cf830 0x7f21d814f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21dcd49640 1 -- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80d27f0 msgr2=0x7f21d815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21dcd49640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80d27f0 0x7f21d815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21dcd49640 1 -- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d80cf830 msgr2=0x7f21d814f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21dcd49640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d80cf830 0x7f21d814f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21dcd49640 1 -- 172.21.15.77:0/3572619523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80cada0 con 0x7f21d815dfe0 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21d77fe640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80d27f0 0x7f21d815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21d7fff640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d80cf830 0x7f21d814f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21dcd49640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815dfe0 0x7f21d815bf70 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009f40 tx=0x7f21d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21d4ff9640 1 -- 172.21.15.77:0/3572619523 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0043020 con 0x7f21d815dfe0 2024-03-06T07:28:25.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21d4ff9640 1 -- 172.21.15.77:0/3572619523 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d003e450 con 0x7f21d815dfe0 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21d4ff9640 1 -- 172.21.15.77:0/3572619523 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0009510 con 0x7f21d815dfe0 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d80cb080 con 0x7f21d815dfe0 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.154+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d80cb640 con 0x7f21d815dfe0 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.155+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d804f330 con 0x7f21d815dfe0 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.156+0000 7f21d4ff9640 1 -- 172.21.15.77:0/3572619523 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21d0046020 con 0x7f21d815dfe0 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.156+0000 7f21d4ff9640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c406b4d0 0x7f21c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.156+0000 7f21d4ff9640 1 -- 172.21.15.77:0/3572619523 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f21d00ac7c0 con 0x7f21d815dfe0 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.156+0000 7f21d7fff640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c406b4d0 0x7f21c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.156+0000 7f21d7fff640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c406b4d0 0x7f21c406d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f21c8006fd0 tx=0x7f21c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.160+0000 7f21d4ff9640 1 -- 172.21.15.77:0/3572619523 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21d004d030 con 0x7f21d815dfe0 2024-03-06T07:28:25.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.405+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f21d80b9b90 con 0x7f21d815dfe0 2024-03-06T07:28:25.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.727+0000 7f21d4ff9640 1 -- 172.21.15.77:0/3572619523 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v342) v1 ==== 142+0+0 (secure 0 0 0) 0x7f21d007b890 con 0x7f21d815dfe0 2024-03-06T07:28:25.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.729+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c406b4d0 msgr2=0x7f21c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.729+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c406b4d0 0x7f21c406d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f21c8006fd0 tx=0x7f21c8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.729+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815dfe0 msgr2=0x7f21d815bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.729+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815dfe0 0x7f21d815bf70 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009f40 tx=0x7f21d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.730+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 shutdown_connections 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.730+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f21c406b4d0 0x7f21c406d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.730+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815dfe0 0x7f21d815bf70 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.730+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f21d80d27f0 0x7f21d815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.730+0000 7f21de7ac640 1 --2- 172.21.15.77:0/3572619523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f21d80cf830 0x7f21d814f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.730+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 >> 172.21.15.77:0/3572619523 conn(0x7f21d80d3cd0 msgr2=0x7f21d80cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.730+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 shutdown_connections 2024-03-06T07:28:25.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.730+0000 7f21de7ac640 1 -- 172.21.15.77:0/3572619523 wait complete. 2024-03-06T07:28:25.743 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:28:25.743 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:28:25.744 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:25.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.843+0000 7feb4d12b640 1 Processor -- start 2024-03-06T07:28:25.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.843+0000 7feb4d12b640 1 -- start start 2024-03-06T07:28:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4d12b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4d12b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb4815bf30 0x7feb4815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4d12b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4815f2d0 con 0x7feb4813dc10 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4d12b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb480cf3f0 con 0x7feb4815bf30 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4d12b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4815b210 con 0x7feb480b3060 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb4815bf30 0x7feb4815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb46d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7feb4813dc10 0x7feb480b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38810/0 (socket says 172.21.15.77:38810) 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb4815bf30 0x7feb4815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59392/0 (socket says 172.21.15.77:59392) 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb46d9d640 1 -- 172.21.15.77:0/3582315842 learned_addr learned my addr 172.21.15.77:0/3582315842 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4659c640 1 --2- 172.21.15.77:0/3582315842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4759e640 1 -- 172.21.15.77:0/3582315842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 msgr2=0x7feb4815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4759e640 1 --2- 172.21.15.77:0/3582315842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4759e640 1 -- 172.21.15.77:0/3582315842 >> v1:172.21.15.77:6789/0 conn(0x7feb4813dc10 legacy=0x7feb480b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.844+0000 7feb4759e640 1 -- 172.21.15.77:0/3582315842 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4815f580 con 0x7feb4815bf30 2024-03-06T07:28:25.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.845+0000 7feb4759e640 1 --2- 172.21.15.77:0/3582315842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815ed90 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7feb400097a0 tx=0x7feb4002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a4a6da3afb12f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.845+0000 7feb45d9b640 1 -- 172.21.15.77:0/3582315842 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400315a0 con 0x7feb4815bf30 2024-03-06T07:28:25.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.845+0000 7feb45d9b640 1 -- 172.21.15.77:0/3582315842 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb40037070 con 0x7feb4815bf30 2024-03-06T07:28:25.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.845+0000 7feb45d9b640 1 -- 172.21.15.77:0/3582315842 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40031960 con 0x7feb4815bf30 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.846+0000 7feb4d12b640 1 -- 172.21.15.77:0/3582315842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 msgr2=0x7feb4815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.846+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3582315842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815ed90 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7feb400097a0 tx=0x7feb4002b840 comp rx=0 tx=0).stop 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.846+0000 7feb4d12b640 1 -- 172.21.15.77:0/3582315842 shutdown_connections 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.846+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3582315842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815ed90 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.846+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3582315842 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.846+0000 7feb4d12b640 1 -- 172.21.15.77:0/3582315842 >> 172.21.15.77:0/3582315842 conn(0x7feb480cf8a0 msgr2=0x7feb480cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.846+0000 7feb4d12b640 1 -- 172.21.15.77:0/3582315842 shutdown_connections 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.846+0000 7feb4d12b640 1 -- 172.21.15.77:0/3582315842 wait complete. 2024-03-06T07:28:25.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.847+0000 7feb4d12b640 1 Processor -- start 2024-03-06T07:28:25.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.847+0000 7feb4d12b640 1 -- start start 2024-03-06T07:28:25.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4d12b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4d12b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4813dc10 0x7feb481382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4d12b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4d12b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4813dff0 con 0x7feb4815bf30 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4d12b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4815b730 con 0x7feb4813dc10 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4d12b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480cd7a0 con 0x7feb480b3060 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59404/0 (socket says 172.21.15.77:59404) 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4759e640 1 -- 172.21.15.77:0/3098235914 learned_addr learned my addr 172.21.15.77:0/3098235914 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb46d9d640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4659c640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4813dc10 0x7feb481382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4759e640 1 -- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 msgr2=0x7feb4815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4759e640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4759e640 1 -- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4813dc10 msgr2=0x7feb481382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4759e640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4813dc10 0x7feb481382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4759e640 1 -- 172.21.15.77:0/3098235914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4813f8c0 con 0x7feb4815bf30 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb4659c640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4813dc10 0x7feb481382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.848+0000 7feb46d9d640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:25.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.849+0000 7feb4759e640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815d9b0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7feb400098c0 tx=0x7feb400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.849+0000 7feb2f7fe640 1 -- 172.21.15.77:0/3098235914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40032630 con 0x7feb4815bf30 2024-03-06T07:28:25.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.849+0000 7feb2f7fe640 1 -- 172.21.15.77:0/3098235914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb400327f0 con 0x7feb4815bf30 2024-03-06T07:28:25.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.849+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4813fba0 con 0x7feb4815bf30 2024-03-06T07:28:25.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.849+0000 7feb2f7fe640 1 -- 172.21.15.77:0/3098235914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40032be0 con 0x7feb4815bf30 2024-03-06T07:28:25.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.849+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb48140160 con 0x7feb4815bf30 2024-03-06T07:28:25.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.850+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4804f330 con 0x7feb4815bf30 2024-03-06T07:28:25.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.850+0000 7feb2f7fe640 1 -- 172.21.15.77:0/3098235914 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb4004c020 con 0x7feb4815bf30 2024-03-06T07:28:25.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.850+0000 7feb2f7fe640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.851+0000 7feb2f7fe640 1 -- 172.21.15.77:0/3098235914 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5115+0+0 (secure 0 0 0) 0x7feb4002f020 con 0x7feb4815bf30 2024-03-06T07:28:25.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.851+0000 7feb46d9d640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.851+0000 7feb46d9d640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb3406b4d0 0x7feb3406d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7feb380012e0 tx=0x7feb38009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:25.855+0000 7feb2f7fe640 1 -- 172.21.15.77:0/3098235914 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb40040070 con 0x7feb4815bf30 2024-03-06T07:28:26.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.031+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb4806eb00 con 0x7feb4815bf30 2024-03-06T07:28:26.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.032+0000 7feb2f7fe640 1 -- 172.21.15.77:0/3098235914 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+13276 (secure 0 0 0) 0x7feb400302f0 con 0x7feb4815bf30 2024-03-06T07:28:26.034 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:26.034 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":116,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:24.720169+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6835/493819683":"2024-03-07T07:28:21.712791+0000","172.21.15.77:6834/493819683":"2024-03-07T07:28:21.712791+0000","172.21.15.77:6835/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6834/1169947364":"2024-03-07T07:27:19.549687+0000","172.21.15.77:6837/3247585937":"2024-03-07T07:27:18.550938+0000","172.21.15.77:6836/3247585937":"2024-03-07T07:27:18.550938+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb3406b4d0 msgr2=0x7feb3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb3406b4d0 0x7feb3406d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7feb380012e0 tx=0x7feb38009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 msgr2=0x7feb4815d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815d9b0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7feb400098c0 tx=0x7feb400046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 shutdown_connections 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb4815bf30 0x7feb4815d9b0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7feb4813dc10 0x7feb481382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 --2- 172.21.15.77:0/3098235914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7feb480b3060 0x7feb4815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 >> 172.21.15.77:0/3098235914 conn(0x7feb480cf8a0 msgr2=0x7feb4806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 shutdown_connections 2024-03-06T07:28:26.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.034+0000 7feb4d12b640 1 -- 172.21.15.77:0/3098235914 wait complete. 2024-03-06T07:28:26.047 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:26.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:26.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.148+0000 7fc674516640 1 Processor -- start 2024-03-06T07:28:26.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.148+0000 7fc674516640 1 -- start start 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc674516640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc66c1586e0 0x7fc66c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc674516640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c158ac0 con 0x7fc66c0d01a0 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc674516640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c15b2a0 con 0x7fc66c0aef80 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc674516640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c15b540 con 0x7fc66c1586e0 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc672ab3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc66c0d01a0 0x7fc66c1658c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38812/0 (socket says 172.21.15.77:38812) 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc672ab3640 1 -- 172.21.15.77:0/2869522438 learned_addr learned my addr 172.21.15.77:0/2869522438 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6722b2640 1 --2- 172.21.15.77:0/2869522438 >> v2:172.21.15.158:3300/0 conn(0x7fc66c1586e0 0x7fc66c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573655146 0 0) 0x7fc66c158ac0 con 0x7fc66c0d01a0 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc658003520 con 0x7fc66c0d01a0 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 92407116 0 0) 0x7fc658003520 con 0x7fc66c0d01a0 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 >> v2:172.21.15.158:3300/0 conn(0x7fc66c1586e0 msgr2=0x7fc66c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6712b0640 1 --2- 172.21.15.77:0/2869522438 >> v2:172.21.15.158:3300/0 conn(0x7fc66c1586e0 0x7fc66c0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 >> v1:172.21.15.158:6789/0 conn(0x7fc66c0aef80 legacy=0x7fc66c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c15b820 con 0x7fc66c0d01a0 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.149+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fc6640019f0 con 0x7fc66c0d01a0 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 >> v1:172.21.15.77:6789/0 conn(0x7fc66c0d01a0 legacy=0x7fc66c1658c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6712b0640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc658003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6712b0640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc658004800 0x7fc658004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6712b0640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6580051c0 0x7fc65800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c15b540 con 0x7fc6580039b0 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65800c160 con 0x7fc6580051c0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65800c360 con 0x7fc658004800 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6722b2640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc658003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc672ab3640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6580051c0 0x7fc65800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc671ab1640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc658004800 0x7fc658004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6722b2640 1 -- 172.21.15.77:0/2869522438 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc658004800 msgr2=0x7fc658004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6722b2640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc658004800 0x7fc658004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6722b2640 1 -- 172.21.15.77:0/2869522438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6580051c0 msgr2=0x7fc65800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6722b2640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6580051c0 0x7fc65800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc672ab3640 1 -- 172.21.15.77:0/2869522438 reap_dead start 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.150+0000 7fc6722b2640 1 -- 172.21.15.77:0/2869522438 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658003520 con 0x7fc6580039b0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc6722b2640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc658003dd0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fc668009ac0 tx=0x7fc658005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732fe0d5cbe63e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668039070 con 0x7fc6580039b0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66802c6b0 con 0x7fc6580039b0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc6712b0640 1 -- 172.21.15.77:0/2869522438 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668009690 con 0x7fc6580039b0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc674516640 1 -- 172.21.15.77:0/2869522438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 msgr2=0x7fc658003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc674516640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc658003dd0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fc668009ac0 tx=0x7fc658005830 comp rx=0 tx=0).stop 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc674516640 1 -- 172.21.15.77:0/2869522438 shutdown_connections 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc674516640 1 --2- 172.21.15.77:0/2869522438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc658003dd0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc674516640 1 -- 172.21.15.77:0/2869522438 >> 172.21.15.77:0/2869522438 conn(0x7fc66c06b190 msgr2=0x7fc66c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc674516640 1 -- 172.21.15.77:0/2869522438 shutdown_connections 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.151+0000 7fc674516640 1 -- 172.21.15.77:0/2869522438 wait complete. 2024-03-06T07:28:26.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.152+0000 7fc674516640 1 Processor -- start 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.152+0000 7fc674516640 1 -- start start 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc674516640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc66c156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc674516640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc66c1f5800 0x7fc66c1549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc674516640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc66c154f00 0x7fc66c150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc674516640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c071350 con 0x7fc6580039b0 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc674516640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c15bde0 con 0x7fc66c154f00 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc674516640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0714d0 con 0x7fc66c1f5800 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc6722b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc66c156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc6722b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc66c156c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59412/0 (socket says 172.21.15.77:59412) 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc6722b2640 1 -- 172.21.15.77:0/550017757 learned_addr learned my addr 172.21.15.77:0/550017757 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc671ab1640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc66c1f5800 0x7fc66c1549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc672ab3640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc66c154f00 0x7fc66c150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc6722b2640 1 -- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc66c1f5800 msgr2=0x7fc66c1549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc6722b2640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc66c1f5800 0x7fc66c1549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc6722b2640 1 -- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc66c154f00 msgr2=0x7fc66c150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.153+0000 7fc6722b2640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc66c154f00 0x7fc66c150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc6722b2640 1 -- 172.21.15.77:0/550017757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c152d70 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc671ab1640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc66c1f5800 0x7fc66c1549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc672ab3640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc66c154f00 0x7fc66c150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc6722b2640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc66c156c00 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fc668037040 tx=0x7fc668004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc662ffd640 1 -- 172.21.15.77:0/550017757 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668039050 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc662ffd640 1 -- 172.21.15.77:0/550017757 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc668045070 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc662ffd640 1 -- 172.21.15.77:0/550017757 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680044e0 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66c153050 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.154+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66c1f52e0 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.155+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66c153570 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.155+0000 7fc662ffd640 1 -- 172.21.15.77:0/550017757 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc668003a50 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.156+0000 7fc662ffd640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc65006b500 0x7fc65006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.156+0000 7fc662ffd640 1 -- 172.21.15.77:0/550017757 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc6680aba60 con 0x7fc6580039b0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.156+0000 7fc671ab1640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc65006b500 0x7fc65006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.156+0000 7fc671ab1640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc65006b500 0x7fc65006d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc66c1f4ff0 tx=0x7fc65c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.160+0000 7fc662ffd640 1 -- 172.21.15.77:0/550017757 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc66804e020 con 0x7fc6580039b0 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.332+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc66c068ff0 con 0x7fc6580039b0 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.332+0000 7fc662ffd640 1 -- 172.21.15.77:0/550017757 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc66807ac50 con 0x7fc6580039b0 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc65006b500 msgr2=0x7fc65006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc65006b500 0x7fc65006d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc66c1f4ff0 tx=0x7fc65c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 msgr2=0x7fc66c156c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc66c156c00 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fc668037040 tx=0x7fc668004730 comp rx=0 tx=0).stop 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 shutdown_connections 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc66c154f00 0x7fc66c150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc66c1f5800 0x7fc66c1549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc65006b500 0x7fc65006d9c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 --2- 172.21.15.77:0/550017757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6580039b0 0x7fc66c156c00 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 >> 172.21.15.77:0/550017757 conn(0x7fc66c06b190 msgr2=0x7fc66c15a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 shutdown_connections 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.335+0000 7fc674516640 1 -- 172.21.15.77:0/550017757 wait complete. 2024-03-06T07:28:26.347 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:28:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.446+0000 7fb14ce4e640 1 Processor -- start 2024-03-06T07:28:26.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.446+0000 7fb14ce4e640 1 -- start start 2024-03-06T07:28:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb14ce4e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1480b4420 0x7fb1480cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb14ce4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1480ce0d0 0x7fb14815c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb14ce4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1480d0a70 con 0x7fb14806afb0 2024-03-06T07:28:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb14ce4e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1480d0850 con 0x7fb1480b4420 2024-03-06T07:28:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb14ce4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1480d0d10 con 0x7fb1480ce0d0 2024-03-06T07:28:26.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb145d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1480b4420 0x7fb1480cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb145d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1480b4420 0x7fb1480cdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59424/0 (socket says 172.21.15.77:59424) 2024-03-06T07:28:26.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb145d9b640 1 -- 172.21.15.77:0/1288565466 learned_addr learned my addr 172.21.15.77:0/1288565466 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:26.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb145d9b640 1 -- 172.21.15.77:0/1288565466 >> v2:172.21.15.158:3300/0 conn(0x7fb1480ce0d0 msgr2=0x7fb14815c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb145d9b640 1 --2- 172.21.15.77:0/1288565466 >> v2:172.21.15.158:3300/0 conn(0x7fb1480ce0d0 0x7fb14815c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb145d9b640 1 -- 172.21.15.77:0/1288565466 >> v1:172.21.15.158:6789/0 conn(0x7fb14806afb0 legacy=0x7fb14806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.447+0000 7fb145d9b640 1 -- 172.21.15.77:0/1288565466 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1480d0ff0 con 0x7fb1480b4420 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.448+0000 7fb145d9b640 1 --2- 172.21.15.77:0/1288565466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1480b4420 0x7fb1480cdb90 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fb1481569f0 tx=0x7fb13c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d3cd2e0f3a419a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.448+0000 7fb14559a640 1 -- 172.21.15.77:0/1288565466 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c02e070 con 0x7fb1480b4420 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.448+0000 7fb14559a640 1 -- 172.21.15.77:0/1288565466 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c037070 con 0x7fb1480b4420 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.448+0000 7fb14559a640 1 -- 172.21.15.77:0/1288565466 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c031a00 con 0x7fb1480b4420 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.449+0000 7fb14ce4e640 1 -- 172.21.15.77:0/1288565466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1480b4420 msgr2=0x7fb1480cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.449+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/1288565466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1480b4420 0x7fb1480cdb90 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fb1481569f0 tx=0x7fb13c02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.449+0000 7fb14ce4e640 1 -- 172.21.15.77:0/1288565466 shutdown_connections 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.449+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/1288565466 >> v2:172.21.15.158:3300/0 conn(0x7fb1480ce0d0 0x7fb14815c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.449+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/1288565466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1480b4420 0x7fb1480cdb90 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.449+0000 7fb14ce4e640 1 -- 172.21.15.77:0/1288565466 >> 172.21.15.77:0/1288565466 conn(0x7fb14806c0c0 msgr2=0x7fb14806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.449+0000 7fb14ce4e640 1 -- 172.21.15.77:0/1288565466 shutdown_connections 2024-03-06T07:28:26.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.449+0000 7fb14ce4e640 1 -- 172.21.15.77:0/1288565466 wait complete. 2024-03-06T07:28:26.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.450+0000 7fb14ce4e640 1 Processor -- start 2024-03-06T07:28:26.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.450+0000 7fb14ce4e640 1 -- start start 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14ce4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb14806afb0 0x7fb14815b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14ce4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1480b4420 0x7fb1480ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14ce4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1480ce0d0 0x7fb1481f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14ce4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1480ceb90 con 0x7fb14806afb0 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14ce4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb148145840 con 0x7fb1480ce0d0 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14ce4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815c9d0 con 0x7fb1480b4420 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb14806afb0 0x7fb14815b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb14806afb0 0x7fb14815b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59430/0 (socket says 172.21.15.77:59430) 2024-03-06T07:28:26.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 -- 172.21.15.77:0/3281029948 learned_addr learned my addr 172.21.15.77:0/3281029948 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 -- 172.21.15.77:0/3281029948 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1480b4420 msgr2=0x7fb1480ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1480b4420 0x7fb1480ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 -- 172.21.15.77:0/3281029948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1480ce0d0 msgr2=0x7fb1481f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1480ce0d0 0x7fb1481f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 -- 172.21.15.77:0/3281029948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14806e100 con 0x7fb14806afb0 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.451+0000 7fb14659c640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb14806afb0 0x7fb14815b3a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb13800c7a0 tx=0x7fb13800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.452+0000 7fb136ffd640 1 -- 172.21.15.77:0/3281029948 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138017070 con 0x7fb14806afb0 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.452+0000 7fb136ffd640 1 -- 172.21.15.77:0/3281029948 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb138003da0 con 0x7fb14806afb0 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.452+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1480af0f0 con 0x7fb14806afb0 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.452+0000 7fb136ffd640 1 -- 172.21.15.77:0/3281029948 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1380052b0 con 0x7fb14806afb0 2024-03-06T07:28:26.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.452+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14806e660 con 0x7fb14806afb0 2024-03-06T07:28:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.453+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14804f330 con 0x7fb14806afb0 2024-03-06T07:28:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.453+0000 7fb136ffd640 1 -- 172.21.15.77:0/3281029948 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb13801a070 con 0x7fb14806afb0 2024-03-06T07:28:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.453+0000 7fb136ffd640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb12806b4d0 0x7fb12806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.454+0000 7fb136ffd640 1 -- 172.21.15.77:0/3281029948 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fb138089660 con 0x7fb14806afb0 2024-03-06T07:28:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.454+0000 7fb145d9b640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb12806b4d0 0x7fb12806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.454+0000 7fb145d9b640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb12806b4d0 0x7fb12806d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb13c036000 tx=0x7fb13c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.458+0000 7fb136ffd640 1 -- 172.21.15.77:0/3281029948 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb14806e100 con 0x7fb14806afb0 2024-03-06T07:28:26.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.697+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7fb1480b5cb0 con 0x7fb14806afb0 2024-03-06T07:28:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.731+0000 7fb136ffd640 1 -- 172.21.15.77:0/3281029948 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v343) v1 ==== 143+0+0 (secure 0 0 0) 0x7fb1380587c0 con 0x7fb14806afb0 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb12806b4d0 msgr2=0x7fb12806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb12806b4d0 0x7fb12806d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb13c036000 tx=0x7fb13c030000 comp rx=0 tx=0).stop 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb14806afb0 msgr2=0x7fb14815b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb14806afb0 0x7fb14815b3a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb13800c7a0 tx=0x7fb13800cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 shutdown_connections 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1480ce0d0 0x7fb1481f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1480b4420 0x7fb1480ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb12806b4d0 0x7fb12806d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 --2- 172.21.15.77:0/3281029948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb14806afb0 0x7fb14815b3a0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 >> 172.21.15.77:0/3281029948 conn(0x7fb14806c0c0 msgr2=0x7fb1480b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 shutdown_connections 2024-03-06T07:28:26.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.733+0000 7fb14ce4e640 1 -- 172.21.15.77:0/3281029948 wait complete. 2024-03-06T07:28:26.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.847+0000 7f364ef5e640 1 Processor -- start 2024-03-06T07:28:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.847+0000 7f364ef5e640 1 -- start start 2024-03-06T07:28:26.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f364ef5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f364806e9d0 0x7f36481625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f364ef5e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3648072870 con 0x7f364815bb50 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f364ef5e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3648072b50 con 0x7f364815f140 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f364ef5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3648162160 con 0x7f364806e9d0 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f364ccfa640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f364815bb50 0x7f3648071fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38826/0 (socket says 172.21.15.77:38826) 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f364ccfa640 1 -- 172.21.15.77:0/4077589204 learned_addr learned my addr 172.21.15.77:0/4077589204 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f364d4fb640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f364806e9d0 0x7f36481625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034474960 0 0) 0x7f3648072870 con 0x7f364815bb50 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.848+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630003520 con 0x7f364815bb50 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212769308 0 0) 0x7f3648072b50 con 0x7f364815f140 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3648072870 con 0x7f364815f140 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2321227587 0 0) 0x7f3630003520 con 0x7f364815bb50 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f364806e9d0 msgr2=0x7f36481625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f364806e9d0 0x7f36481625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 >> v1:172.21.15.158:6789/0 conn(0x7f364815f140 legacy=0x7f364806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648161b10 con 0x7f364815bb50 2024-03-06T07:28:26.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3644001610 con 0x7f364815bb50 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 >> v1:172.21.15.77:6789/0 conn(0x7f364815bb50 legacy=0x7f3648071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3630003b70 0x7f3630003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f3630004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f364806e9d0 0x7f3630005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3648162160 con 0x7f3630004800 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363000bb50 con 0x7f3630003b70 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363000bd50 con 0x7f364806e9d0 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363ffff640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f3630004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f364ccfa640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3630003b70 0x7f3630003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f364d4fb640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f364806e9d0 0x7f3630005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363ffff640 1 -- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f364806e9d0 msgr2=0x7f3630005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363ffff640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f364806e9d0 0x7f3630005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363ffff640 1 -- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3630003b70 msgr2=0x7f3630003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363ffff640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3630003b70 0x7f3630003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.849+0000 7f363ffff640 1 -- 172.21.15.77:0/4077589204 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630003520 con 0x7f3630004800 2024-03-06T07:28:26.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.850+0000 7f363ffff640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f3630004c80 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f36380018d0 tx=0x7f3630005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe75d8bddc07351 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.850+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638037070 con 0x7f3630004800 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.850+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36380052d0 con 0x7f3630004800 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.850+0000 7f363f7fe640 1 -- 172.21.15.77:0/4077589204 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36380014f0 con 0x7f3630004800 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.850+0000 7f364ef5e640 1 -- 172.21.15.77:0/4077589204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 msgr2=0x7f3630004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.850+0000 7f364ef5e640 1 --2- 172.21.15.77:0/4077589204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f3630004c80 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f36380018d0 tx=0x7f3630005830 comp rx=0 tx=0).stop 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.850+0000 7f364d4fb640 1 -- 172.21.15.77:0/4077589204 reap_dead start 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.851+0000 7f364ef5e640 1 -- 172.21.15.77:0/4077589204 shutdown_connections 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.851+0000 7f364ef5e640 1 -- 172.21.15.77:0/4077589204 >> 172.21.15.77:0/4077589204 conn(0x7f364806b830 msgr2=0x7f36480b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.851+0000 7f364ef5e640 1 -- 172.21.15.77:0/4077589204 shutdown_connections 2024-03-06T07:28:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.851+0000 7f364ef5e640 1 -- 172.21.15.77:0/4077589204 wait complete. 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.852+0000 7f364ef5e640 1 Processor -- start 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.852+0000 7f364ef5e640 1 -- start start 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.852+0000 7f364ef5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f36480cf790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.852+0000 7f364ef5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36480cc8c0 0x7f3648144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ef5e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36481f75d0 0x7f36481f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ef5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3648162770 con 0x7f3630004800 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ef5e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36481625f0 con 0x7f36481f75d0 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ef5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36481628f0 con 0x7f36480cc8c0 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f36480cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f36480cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59444/0 (socket says 172.21.15.77:59444) 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 -- 172.21.15.77:0/3723258059 learned_addr learned my addr 172.21.15.77:0/3723258059 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f363ffff640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36480cc8c0 0x7f3648144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364d4fb640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36481f75d0 0x7f36481f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 -- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36480cc8c0 msgr2=0x7f3648144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36480cc8c0 0x7f3648144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 -- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36481f75d0 msgr2=0x7f36481f7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36481f75d0 0x7f36481f7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 -- 172.21.15.77:0/3723258059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648159320 con 0x7f3630004800 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364d4fb640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36481f75d0 0x7f36481f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ccfa640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f36480cf790 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f3644004df0 tx=0x7f3644007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f363d7fa640 1 -- 172.21.15.77:0/3723258059 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364400f7b0 con 0x7f3630004800 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f363d7fa640 1 -- 172.21.15.77:0/3723258059 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3644018070 con 0x7f3630004800 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3648159600 con 0x7f3630004800 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f363d7fa640 1 -- 172.21.15.77:0/3723258059 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364400fb90 con 0x7f3630004800 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.853+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3648159bc0 con 0x7f3630004800 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.854+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364806e0b0 con 0x7f3630004800 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.855+0000 7f363d7fa640 1 -- 172.21.15.77:0/3723258059 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f364400fd30 con 0x7f3630004800 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.855+0000 7f363d7fa640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f363406b500 0x7f363406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.855+0000 7f363d7fa640 1 -- 172.21.15.77:0/3723258059 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3644089e70 con 0x7f3630004800 2024-03-06T07:28:26.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.855+0000 7f363ffff640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f363406b500 0x7f363406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.856+0000 7f363ffff640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f363406b500 0x7f363406d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f36380018d0 tx=0x7f3638031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:26.859+0000 7f363d7fa640 1 -- 172.21.15.77:0/3723258059 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3644058f40 con 0x7f3630004800 2024-03-06T07:28:27.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.030+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3648068ff0 con 0x7f3630004800 2024-03-06T07:28:27.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.035+0000 7f363d7fa640 1 -- 172.21.15.77:0/3723258059 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3644058840 con 0x7f3630004800 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f363406b500 msgr2=0x7f363406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f363406b500 0x7f363406d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f36380018d0 tx=0x7f3638031000 comp rx=0 tx=0).stop 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 msgr2=0x7f36480cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f36480cf790 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f3644004df0 tx=0x7f3644007430 comp rx=0 tx=0).stop 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 shutdown_connections 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36481f75d0 0x7f36481f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f36480cc8c0 0x7f3648144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f363406b500 0x7f363406d9c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 --2- 172.21.15.77:0/3723258059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3630004800 0x7f36480cf790 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 >> 172.21.15.77:0/3723258059 conn(0x7f364806b830 msgr2=0x7f36480ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 shutdown_connections 2024-03-06T07:28:27.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.038+0000 7f364ef5e640 1 -- 172.21.15.77:0/3723258059 wait complete. 2024-03-06T07:28:27.051 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:27.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:27.090 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:27.141 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:27.141 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:28:27.141 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.156 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:27.156 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:28:27.157 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:27.173 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:28:27.173 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.211 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:27.211 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:28:27.211 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.230 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:27.230 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:27.230 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:27.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.350+0000 7f8aa0bf7640 1 Processor -- start 2024-03-06T07:28:27.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.350+0000 7f8aa0bf7640 1 -- start start 2024-03-06T07:28:27.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.350+0000 7f8aa0bf7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8a9c15dfe0 0x7f8a9c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.350+0000 7f8aa0bf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a9c0cf830 0x7f8a9c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.350+0000 7f8aa0bf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c0d27f0 0x7f8a9c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.350+0000 7f8aa0bf7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9c0b31e0 con 0x7f8a9c0cf830 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.350+0000 7f8aa0bf7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9c0b3d60 con 0x7f8a9c15dfe0 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.350+0000 7f8aa0bf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0b3430 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a93fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a9c0cf830 0x7f8a9c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a93fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a9c0cf830 0x7f8a9c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59446/0 (socket says 172.21.15.77:59446) 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a93fff640 1 -- 172.21.15.77:0/281200783 learned_addr learned my addr 172.21.15.77:0/281200783 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a93fff640 1 -- 172.21.15.77:0/281200783 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c0d27f0 msgr2=0x7f8a9c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a9a59c640 1 --2- 172.21.15.77:0/281200783 >> v2:172.21.15.158:3300/0 conn(0x7f8a9c15dfe0 0x7f8a9c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a9ad9d640 1 --2- 172.21.15.77:0/281200783 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c0d27f0 0x7f8a9c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a93fff640 1 --2- 172.21.15.77:0/281200783 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c0d27f0 0x7f8a9c0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a93fff640 1 -- 172.21.15.77:0/281200783 >> v2:172.21.15.158:3300/0 conn(0x7f8a9c15dfe0 msgr2=0x7f8a9c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a93fff640 1 --2- 172.21.15.77:0/281200783 >> v2:172.21.15.158:3300/0 conn(0x7f8a9c15dfe0 0x7f8a9c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.351+0000 7f8a93fff640 1 -- 172.21.15.77:0/281200783 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c0b3710 con 0x7f8a9c0cf830 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8a93fff640 1 --2- 172.21.15.77:0/281200783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0cf830 0x7f8a9c0d22b0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c067990 tx=0x7f8a8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c739b0e01d9bac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8a99d9b640 1 -- 172.21.15.77:0/281200783 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84035020 con 0x7f8a9c0cf830 2024-03-06T07:28:27.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8a99d9b640 1 -- 172.21.15.77:0/281200783 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a84031870 con 0x7f8a9c0cf830 2024-03-06T07:28:27.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8a99d9b640 1 -- 172.21.15.77:0/281200783 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84031c40 con 0x7f8a9c0cf830 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/281200783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0cf830 msgr2=0x7f8a9c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/281200783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0cf830 0x7f8a9c0d22b0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c067990 tx=0x7f8a8402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/281200783 shutdown_connections 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/281200783 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c0d27f0 0x7f8a9c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/281200783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0cf830 0x7f8a9c0d22b0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/281200783 >> v2:172.21.15.158:3300/0 conn(0x7f8a9c15dfe0 0x7f8a9c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/281200783 >> 172.21.15.77:0/281200783 conn(0x7f8a9c0d3cd0 msgr2=0x7f8a9c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/281200783 shutdown_connections 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.352+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/281200783 wait complete. 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8aa0bf7640 1 Processor -- start 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8aa0bf7640 1 -- start start 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8aa0bf7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9c0cf830 0x7f8a9c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8aa0bf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27f0 0x7f8a9c0b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8aa0bf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c15dfe0 0x7f8a9c0b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8aa0bf7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0d1df0 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8aa0bf7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0d1c70 con 0x7f8a9c0cf830 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8aa0bf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c15e3c0 con 0x7f8a9c15dfe0 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8a93fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27f0 0x7f8a9c0b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8a93fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27f0 0x7f8a9c0b87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59462/0 (socket says 172.21.15.77:59462) 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8a93fff640 1 -- 172.21.15.77:0/1565515537 learned_addr learned my addr 172.21.15.77:0/1565515537 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8a9a59c640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9c0cf830 0x7f8a9c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.354+0000 7f8a9ad9d640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c15dfe0 0x7f8a9c0b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a93fff640 1 -- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c15dfe0 msgr2=0x7f8a9c0b8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a93fff640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c15dfe0 0x7f8a9c0b8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a93fff640 1 -- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9c0cf830 msgr2=0x7f8a9c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a93fff640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9c0cf830 0x7f8a9c15bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a93fff640 1 -- 172.21.15.77:0/1565515537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c0cbbb0 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a9a59c640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9c0cf830 0x7f8a9c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a93fff640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27f0 0x7f8a9c0b87d0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f8a8402b950 tx=0x7f8a840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a92ffd640 1 -- 172.21.15.77:0/1565515537 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84040070 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a92ffd640 1 -- 172.21.15.77:0/1565515537 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8403a420 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c0cbe90 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8a92ffd640 1 -- 172.21.15.77:0/1565515537 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84035040 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.355+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c0cc450 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.356+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c04f330 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.356+0000 7f8a92ffd640 1 -- 172.21.15.77:0/1565515537 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a8404b060 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.357+0000 7f8a92ffd640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.357+0000 7f8a92ffd640 1 -- 172.21.15.77:0/1565515537 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f8a840abe90 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.357+0000 7f8a9a59c640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.357+0000 7f8a9a59c640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c006fd0 tx=0x7f8a8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.361+0000 7f8a92ffd640 1 -- 172.21.15.77:0/1565515537 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a8407aff0 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.569+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a9c0ce9b0 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.569+0000 7f8a92ffd640 1 -- 172.21.15.77:0/1565515537 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f8a8407a8f0 con 0x7f8a9c0d27f0 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":76,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/1914794023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1914794023},{"type":"v1","addr":"172.21.15.158:6835","nonce":1914794023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7414,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/3049978903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3049978903},{"type":"v1","addr":"172.21.15.158:6834","nonce":3049978903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1383747794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1383747794},{"type":"v1","addr":"172.21.15.77:6837","nonce":1383747794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":8457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3971396503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3971396503},{"type":"v1","addr":"172.21.15.77:6835","nonce":3971396503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[]} 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 76 2024-03-06T07:28:27.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a7c06b4d0 msgr2=0x7f8a7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c006fd0 tx=0x7f8a8c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27f0 msgr2=0x7f8a9c0b87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27f0 0x7f8a9c0b87d0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f8a8402b950 tx=0x7f8a840046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 shutdown_connections 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a7c06b4d0 0x7f8a7c06d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0d27f0 0x7f8a9c0b87d0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9c15dfe0 0x7f8a9c0b8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 --2- 172.21.15.77:0/1565515537 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9c0cf830 0x7f8a9c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 >> 172.21.15.77:0/1565515537 conn(0x7f8a9c0d3cd0 msgr2=0x7f8a9c06bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 shutdown_connections 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.571+0000 7f8aa0bf7640 1 -- 172.21.15.77:0/1565515537 wait complete. 2024-03-06T07:28:27.585 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:27.585 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:27.585 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:27.585 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:27.585 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:27.585 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:27.585 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:27.586 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:27.586 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:27.586 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:27.586 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:27.586 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:27.586 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:27.586 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:27.587 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:27.587 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:27.596 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:27.596 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:27.597 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:27.597 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:27.598 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:27.598 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:27.599 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:27.599 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:27.640 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:27.640 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:27.641 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:27.641 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:27.641 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:27.641 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:27.642 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:27.642 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:27.642 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:27.716 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:28:27.716 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:28:27.723 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:28:27.723 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:28:27.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.746+0000 7f9380442640 1 Processor -- start 2024-03-06T07:28:27.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.746+0000 7f9380442640 1 -- start start 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f9380442640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9378162970 0x7f93780d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f9380442640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9378161cf0 0x7f9378162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f9380442640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93780d22c0 con 0x7f93780b4460 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f9380442640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9378161b00 con 0x7f9378162970 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f9380442640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93780cecc0 con 0x7f9378161cf0 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937e1de640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f93780b4460 0x7f93780b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38854/0 (socket says 172.21.15.77:38854) 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937e1de640 1 -- 172.21.15.77:0/710484144 learned_addr learned my addr 172.21.15.77:0/710484144 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937d9dd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9378162970 0x7f93780d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937e9df640 1 --2- 172.21.15.77:0/710484144 >> v2:172.21.15.158:3300/0 conn(0x7f9378161cf0 0x7f9378162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937d1dc640 1 -- 172.21.15.77:0/710484144 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921619813 0 0) 0x7f93780d22c0 con 0x7f93780b4460 2024-03-06T07:28:27.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937d9dd640 1 -- 172.21.15.77:0/710484144 >> v2:172.21.15.158:3300/0 conn(0x7f9378161cf0 msgr2=0x7f9378162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937d9dd640 1 --2- 172.21.15.77:0/710484144 >> v2:172.21.15.158:3300/0 conn(0x7f9378161cf0 0x7f9378162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937d9dd640 1 -- 172.21.15.77:0/710484144 >> v1:172.21.15.77:6789/0 conn(0x7f93780b4460 legacy=0x7f93780b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.748+0000 7f937d9dd640 1 -- 172.21.15.77:0/710484144 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93780d25a0 con 0x7f9378162970 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.749+0000 7f937d9dd640 1 --2- 172.21.15.77:0/710484144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378162970 0x7f93780d4800 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f937808c900 tx=0x7f936802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59659e0fd74342ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.749+0000 7f937d1dc640 1 -- 172.21.15.77:0/710484144 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368035020 con 0x7f9378162970 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.749+0000 7f937d1dc640 1 -- 172.21.15.77:0/710484144 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9368031830 con 0x7f9378162970 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.749+0000 7f937d1dc640 1 -- 172.21.15.77:0/710484144 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368031c00 con 0x7f9378162970 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.750+0000 7f9380442640 1 -- 172.21.15.77:0/710484144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378162970 msgr2=0x7f93780d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.750+0000 7f9380442640 1 --2- 172.21.15.77:0/710484144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378162970 0x7f93780d4800 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f937808c900 tx=0x7f936802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.750+0000 7f9380442640 1 -- 172.21.15.77:0/710484144 shutdown_connections 2024-03-06T07:28:27.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.750+0000 7f9380442640 1 --2- 172.21.15.77:0/710484144 >> v2:172.21.15.158:3300/0 conn(0x7f9378161cf0 0x7f9378162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.750+0000 7f9380442640 1 --2- 172.21.15.77:0/710484144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378162970 0x7f93780d4800 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.750+0000 7f9380442640 1 -- 172.21.15.77:0/710484144 >> 172.21.15.77:0/710484144 conn(0x7f93780d3ce0 msgr2=0x7f93780d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.750+0000 7f9380442640 1 -- 172.21.15.77:0/710484144 shutdown_connections 2024-03-06T07:28:27.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.750+0000 7f9380442640 1 -- 172.21.15.77:0/710484144 wait complete. 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.751+0000 7f9380442640 1 Processor -- start 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.751+0000 7f9380442640 1 -- start start 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f9380442640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f93780b4460 0x7f937814f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f9380442640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378161cf0 0x7f93780b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f9380442640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9378162970 0x7f93780cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f9380442640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780ce950 con 0x7f9378161cf0 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f9380442640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780d2b60 con 0x7f9378162970 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f9380442640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93780b4840 con 0x7f93780b4460 2024-03-06T07:28:27.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937d9dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378161cf0 0x7f93780b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e1de640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f93780b4460 0x7f937814f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937d9dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378161cf0 0x7f93780b6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59512/0 (socket says 172.21.15.77:59512) 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e1de640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f93780b4460 0x7f937814f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:41024/0 (socket says 172.21.15.77:41024) 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e1de640 1 -- 172.21.15.77:0/4215742070 learned_addr learned my addr 172.21.15.77:0/4215742070 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e9df640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9378162970 0x7f93780cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e1de640 1 -- 172.21.15.77:0/4215742070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9378162970 msgr2=0x7f93780cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e1de640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9378162970 0x7f93780cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e1de640 1 -- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378161cf0 msgr2=0x7f93780b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e1de640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378161cf0 0x7f93780b6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.752+0000 7f937e1de640 1 -- 172.21.15.77:0/4215742070 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93780cb520 con 0x7f93780b4460 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.753+0000 7f937e1de640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f93780b4460 0x7f937814f250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f937400c430 tx=0x7f937400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.753+0000 7f936effd640 1 -- 172.21.15.77:0/4215742070 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93740074e0 con 0x7f93780b4460 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.753+0000 7f936effd640 1 -- 172.21.15.77:0/4215742070 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93740047e0 con 0x7f93780b4460 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.753+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93780cb800 con 0x7f93780b4460 2024-03-06T07:28:27.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.753+0000 7f936effd640 1 -- 172.21.15.77:0/4215742070 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374004bd0 con 0x7f93780b4460 2024-03-06T07:28:27.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.753+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93780d1890 con 0x7f93780b4460 2024-03-06T07:28:27.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.754+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937804f330 con 0x7f93780b4460 2024-03-06T07:28:27.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.755+0000 7f936effd640 1 -- 172.21.15.77:0/4215742070 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9374011070 con 0x7f93780b4460 2024-03-06T07:28:27.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.755+0000 7f936effd640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.755+0000 7f936effd640 1 -- 172.21.15.77:0/4215742070 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f9374089ad0 con 0x7f93780b4460 2024-03-06T07:28:27.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.755+0000 7f937d9dd640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.756+0000 7f937d9dd640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9368036000 tx=0x7f9368008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.761+0000 7f936effd640 1 -- 172.21.15.77:0/4215742070 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9374058560 con 0x7f93780b4460 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:27.946+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f937806b180 con 0x7f93780b4460 2024-03-06T07:28:28.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.757+0000 7f936effd640 1 -- 172.21.15.77:0/4215742070 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v121) v1 ==== 97+0+0 (secure 0 0 0) 0x7f937405c600 con 0x7f93780b4460 2024-03-06T07:28:28.759 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 msgr2=0x7f935c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9368036000 tx=0x7f9368008030 comp rx=0 tx=0).stop 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f93780b4460 msgr2=0x7f937814f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f93780b4460 0x7f937814f250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f937400c430 tx=0x7f937400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 shutdown_connections 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f935c06b4d0 0x7f935c06d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9378161cf0 0x7f93780b6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9378162970 0x7f93780cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 --2- 172.21.15.77:0/4215742070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f93780b4460 0x7f937814f250 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 >> 172.21.15.77:0/4215742070 conn(0x7f93780d3ce0 msgr2=0x7f93781610d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 shutdown_connections 2024-03-06T07:28:28.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.759+0000 7f9380442640 1 -- 172.21.15.77:0/4215742070 wait complete. 2024-03-06T07:28:28.772 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:28.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.871+0000 7fabae61c640 1 Processor -- start 2024-03-06T07:28:28.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.871+0000 7fabae61c640 1 -- start start 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.871+0000 7fabae61c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faba81424f0 0x7faba80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.871+0000 7fabae61c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faba80cd7a0 0x7faba815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.871+0000 7fabae61c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba80cc980 con 0x7faba806d330 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.871+0000 7fabae61c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba80cd220 con 0x7faba81424f0 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.871+0000 7fabae61c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba80ccc60 con 0x7faba80cd7a0 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.872+0000 7faba7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faba81424f0 0x7faba80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.872+0000 7faba7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faba81424f0 0x7faba80cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40374/0 (socket says 172.21.15.77:40374) 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.872+0000 7faba7fff640 1 -- 172.21.15.77:0/2204838482 learned_addr learned my addr 172.21.15.77:0/2204838482 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.872+0000 7faba7fff640 1 -- 172.21.15.77:0/2204838482 >> v2:172.21.15.158:3300/0 conn(0x7faba80cd7a0 msgr2=0x7faba815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.872+0000 7faba7fff640 1 --2- 172.21.15.77:0/2204838482 >> v2:172.21.15.158:3300/0 conn(0x7faba80cd7a0 0x7faba815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.872+0000 7faba7fff640 1 -- 172.21.15.77:0/2204838482 >> v1:172.21.15.158:6789/0 conn(0x7faba806d330 legacy=0x7faba8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.872+0000 7faba7fff640 1 -- 172.21.15.77:0/2204838482 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba80ccf40 con 0x7faba81424f0 2024-03-06T07:28:28.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7faba7fff640 1 --2- 172.21.15.77:0/2204838482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba81424f0 0x7faba80cc2f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7faba80b2a00 tx=0x7fab9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37857bef5b021dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7faba6ffd640 1 -- 172.21.15.77:0/2204838482 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98035020 con 0x7faba81424f0 2024-03-06T07:28:28.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7faba6ffd640 1 -- 172.21.15.77:0/2204838482 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab98031870 con 0x7faba81424f0 2024-03-06T07:28:28.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7faba6ffd640 1 -- 172.21.15.77:0/2204838482 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98031c40 con 0x7faba81424f0 2024-03-06T07:28:28.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7fabae61c640 1 -- 172.21.15.77:0/2204838482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba81424f0 msgr2=0x7faba80cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7fabae61c640 1 --2- 172.21.15.77:0/2204838482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba81424f0 0x7faba80cc2f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7faba80b2a00 tx=0x7fab9802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7fabae61c640 1 -- 172.21.15.77:0/2204838482 shutdown_connections 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7fabae61c640 1 --2- 172.21.15.77:0/2204838482 >> v2:172.21.15.158:3300/0 conn(0x7faba80cd7a0 0x7faba815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7fabae61c640 1 --2- 172.21.15.77:0/2204838482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba81424f0 0x7faba80cc2f0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7fabae61c640 1 -- 172.21.15.77:0/2204838482 >> 172.21.15.77:0/2204838482 conn(0x7faba80cf8a0 msgr2=0x7faba80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7fabae61c640 1 -- 172.21.15.77:0/2204838482 shutdown_connections 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.873+0000 7fabae61c640 1 -- 172.21.15.77:0/2204838482 wait complete. 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.874+0000 7fabae61c640 1 Processor -- start 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7fabae61c640 1 -- start start 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7fabae61c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba806d330 0x7faba81f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7fabae61c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba80cd7a0 0x7faba80b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7fabae61c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba81424f0 0x7faba80b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7fabae61c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba806d920 con 0x7faba80cd7a0 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7fabae61c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba806d7a0 con 0x7faba806d330 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7fabae61c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba806daa0 con 0x7faba81424f0 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7faba77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba80cd7a0 0x7faba80b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.875+0000 7faba77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba80cd7a0 0x7faba80b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40388/0 (socket says 172.21.15.77:40388) 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba77fe640 1 -- 172.21.15.77:0/1996400268 learned_addr learned my addr 172.21.15.77:0/1996400268 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba7fff640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba806d330 0x7faba81f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7fabacbb9640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba81424f0 0x7faba80b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba77fe640 1 -- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba81424f0 msgr2=0x7faba80b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba77fe640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba81424f0 0x7faba80b0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba77fe640 1 -- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba806d330 msgr2=0x7faba81f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba77fe640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba806d330 0x7faba81f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba77fe640 1 -- 172.21.15.77:0/1996400268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba80ae8e0 con 0x7faba80cd7a0 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba7fff640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba806d330 0x7faba81f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba77fe640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba80cd7a0 0x7faba80b0610 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fab9000b4a0 tx=0x7fab9000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba4ff9640 1 -- 172.21.15.77:0/1996400268 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90014070 con 0x7faba80cd7a0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba4ff9640 1 -- 172.21.15.77:0/1996400268 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab90010610 con 0x7faba80cd7a0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba80ae600 con 0x7faba80cd7a0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7faba4ff9640 1 -- 172.21.15.77:0/1996400268 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90010a00 con 0x7faba80cd7a0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.876+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba80b2dd0 con 0x7faba80cd7a0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.877+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba804f330 con 0x7faba80cd7a0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.878+0000 7faba4ff9640 1 -- 172.21.15.77:0/1996400268 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab90017070 con 0x7faba80cd7a0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.878+0000 7faba4ff9640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8806b4d0 0x7fab8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.878+0000 7faba4ff9640 1 -- 172.21.15.77:0/1996400268 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fab90089860 con 0x7faba80cd7a0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.878+0000 7faba7fff640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8806b4d0 0x7fab8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.878+0000 7faba7fff640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8806b4d0 0x7fab8806d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fab98036000 tx=0x7fab9803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:28.882+0000 7faba4ff9640 1 -- 172.21.15.77:0/1996400268 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab90058ac0 con 0x7faba80cd7a0 2024-03-06T07:28:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.120+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7faba80cc6a0 con 0x7faba80cd7a0 2024-03-06T07:28:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.121+0000 7faba4ff9640 1 -- 172.21.15.77:0/1996400268 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fab900583c0 con 0x7faba80cd7a0 2024-03-06T07:28:29.122 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:28:29.123 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.124 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.125 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:28:29.126 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.127 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:29.128 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.129 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-03-06T07:28:29.130 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:28:29.131 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:28:29.132 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.133 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:29.134 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:28:29.135 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.135 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:29.135 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:28:29.135 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:28:29.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.122+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8806b4d0 msgr2=0x7fab8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.122+0000 7fabae61c640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8806b4d0 0x7fab8806d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fab98036000 tx=0x7fab9803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.122+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba80cd7a0 msgr2=0x7faba80b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba80cd7a0 0x7faba80b0610 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fab9000b4a0 tx=0x7fab9000b960 comp rx=0 tx=0).stop 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 shutdown_connections 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8806b4d0 0x7fab8806d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faba80cd7a0 0x7faba80b0610 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faba81424f0 0x7faba80b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 --2- 172.21.15.77:0/1996400268 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faba806d330 0x7faba81f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 >> 172.21.15.77:0/1996400268 conn(0x7faba80cf8a0 msgr2=0x7faba80ce140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 shutdown_connections 2024-03-06T07:28:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.123+0000 7fabae61c640 1 -- 172.21.15.77:0/1996400268 wait complete. 2024-03-06T07:28:29.138 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.236+0000 7f6d3748f640 1 Processor -- start 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.236+0000 7f6d3748f640 1 -- start start 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3748f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d301515e0 0x7f6d300cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3748f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6d3015b430 0x7f6d30071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3748f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3015b8e0 con 0x7f6d300cc340 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3748f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d30070a70 con 0x7f6d301515e0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3748f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d30070d50 con 0x7f6d3015b430 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3522b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d301515e0 0x7f6d300cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d35a2c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6d300cc340 0x7f6d300cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53992/0 (socket says 172.21.15.77:53992) 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3522b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d301515e0 0x7f6d300cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40404/0 (socket says 172.21.15.77:40404) 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d35a2c640 1 -- 172.21.15.77:0/437144477 learned_addr learned my addr 172.21.15.77:0/437144477 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3522b640 1 -- 172.21.15.77:0/437144477 >> v2:172.21.15.158:3300/0 conn(0x7f6d3015b430 msgr2=0x7f6d30071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3522b640 1 --2- 172.21.15.77:0/437144477 >> v2:172.21.15.158:3300/0 conn(0x7f6d3015b430 0x7f6d30071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3522b640 1 -- 172.21.15.77:0/437144477 >> v1:172.21.15.77:6789/0 conn(0x7f6d300cc340 legacy=0x7f6d300cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.237+0000 7f6d3522b640 1 -- 172.21.15.77:0/437144477 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d30071030 con 0x7f6d301515e0 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d3522b640 1 --2- 172.21.15.77:0/437144477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d301515e0 0x7f6d300cce40 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f6d300cc110 tx=0x7f6d2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b24ff1dc73e762b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d27fff640 1 -- 172.21.15.77:0/437144477 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c035020 con 0x7f6d301515e0 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d27fff640 1 -- 172.21.15.77:0/437144477 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2c031830 con 0x7f6d301515e0 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d27fff640 1 -- 172.21.15.77:0/437144477 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c031c00 con 0x7f6d301515e0 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d3748f640 1 -- 172.21.15.77:0/437144477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d301515e0 msgr2=0x7f6d300cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d3748f640 1 --2- 172.21.15.77:0/437144477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d301515e0 0x7f6d300cce40 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f6d300cc110 tx=0x7f6d2c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d3748f640 1 -- 172.21.15.77:0/437144477 shutdown_connections 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d3748f640 1 --2- 172.21.15.77:0/437144477 >> v2:172.21.15.158:3300/0 conn(0x7f6d3015b430 0x7f6d30071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.238+0000 7f6d3748f640 1 --2- 172.21.15.77:0/437144477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d301515e0 0x7f6d300cce40 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.239+0000 7f6d3748f640 1 -- 172.21.15.77:0/437144477 >> 172.21.15.77:0/437144477 conn(0x7f6d300cf8a0 msgr2=0x7f6d300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.239+0000 7f6d3748f640 1 -- 172.21.15.77:0/437144477 shutdown_connections 2024-03-06T07:28:29.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.239+0000 7f6d3748f640 1 -- 172.21.15.77:0/437144477 wait complete. 2024-03-06T07:28:29.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.240+0000 7f6d3748f640 1 Processor -- start 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.240+0000 7f6d3748f640 1 -- start start 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.240+0000 7f6d3748f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d300cc340 0x7f6d30146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.240+0000 7f6d3748f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d301515e0 0x7f6d30072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.240+0000 7f6d3748f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d3015b430 0x7f6d300725b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.240+0000 7f6d3748f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d30071ef0 con 0x7f6d3015b430 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.240+0000 7f6d3748f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d30071d70 con 0x7f6d300cc340 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.240+0000 7f6d3748f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d300cdeb0 con 0x7f6d301515e0 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d3015b430 0x7f6d300725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d3015b430 0x7f6d300725b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40414/0 (socket says 172.21.15.77:40414) 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 -- 172.21.15.77:0/1779212348 learned_addr learned my addr 172.21.15.77:0/1779212348 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d34a2a640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d301515e0 0x7f6d30072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d3522b640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d300cc340 0x7f6d30146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 -- 172.21.15.77:0/1779212348 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d301515e0 msgr2=0x7f6d30072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d301515e0 0x7f6d30072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 -- 172.21.15.77:0/1779212348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d300cc340 msgr2=0x7f6d30146ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d300cc340 0x7f6d30146ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 -- 172.21.15.77:0/1779212348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3015e000 con 0x7f6d3015b430 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d35a2c640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d3015b430 0x7f6d300725b0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800cae0 tx=0x7f6d2800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d25ffb640 1 -- 172.21.15.77:0/1779212348 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28017070 con 0x7f6d3015b430 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d25ffb640 1 -- 172.21.15.77:0/1779212348 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2800ac40 con 0x7f6d3015b430 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d301505e0 con 0x7f6d3015b430 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d25ffb640 1 -- 172.21.15.77:0/1779212348 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28012400 con 0x7f6d3015b430 2024-03-06T07:28:29.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.241+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3015f450 con 0x7f6d3015b430 2024-03-06T07:28:29.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.242+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d30068840 con 0x7f6d3015b430 2024-03-06T07:28:29.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.243+0000 7f6d25ffb640 1 -- 172.21.15.77:0/1779212348 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d2801a070 con 0x7f6d3015b430 2024-03-06T07:28:29.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.243+0000 7f6d25ffb640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d1406b4d0 0x7f6d1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.243+0000 7f6d3522b640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d1406b4d0 0x7f6d1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.243+0000 7f6d25ffb640 1 -- 172.21.15.77:0/1779212348 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6d28089b70 con 0x7f6d3015b430 2024-03-06T07:28:29.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.244+0000 7f6d3522b640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d1406b4d0 0x7f6d1406d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c036000 tx=0x7f6d2c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.248+0000 7f6d25ffb640 1 -- 172.21.15.77:0/1779212348 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d280586f0 con 0x7f6d3015b430 2024-03-06T07:28:29.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.422+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d3004f330 con 0x7f6d3015b430 2024-03-06T07:28:29.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.423+0000 7f6d25ffb640 1 -- 172.21.15.77:0/1779212348 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+12889 (secure 0 0 0) 0x7f6d2805c790 con 0x7f6d3015b430 2024-03-06T07:28:29.425 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:29.425 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":121,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:28.754460+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d1406b4d0 msgr2=0x7f6d1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d1406b4d0 0x7f6d1406d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c036000 tx=0x7f6d2c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d3015b430 msgr2=0x7f6d300725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d3015b430 0x7f6d300725b0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800cae0 tx=0x7f6d2800a060 comp rx=0 tx=0).stop 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 shutdown_connections 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d1406b4d0 0x7f6d1406d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d3015b430 0x7f6d300725b0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d301515e0 0x7f6d30072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 --2- 172.21.15.77:0/1779212348 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d300cc340 0x7f6d30146ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 >> 172.21.15.77:0/1779212348 conn(0x7f6d300cf8a0 msgr2=0x7f6d300cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 shutdown_connections 2024-03-06T07:28:29.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.425+0000 7f6d3748f640 1 -- 172.21.15.77:0/1779212348 wait complete. 2024-03-06T07:28:29.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:29.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:29.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.539+0000 7f7d5b04e640 1 Processor -- start 2024-03-06T07:28:29.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.539+0000 7f7d5b04e640 1 -- start start 2024-03-06T07:28:29.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d5b04e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 0x7f7d540cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d5b04e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5415ff90 con 0x7f7d5415bb40 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d5b04e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d54160270 con 0x7f7d540b4970 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d5b04e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d540d0440 con 0x7f7d540cfa90 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d58dea640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7d5415bb40 0x7f7d5415f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53998/0 (socket says 172.21.15.77:53998) 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d58dea640 1 -- 172.21.15.77:0/1163555688 learned_addr learned my addr 172.21.15.77:0/1163555688 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d595eb640 1 --2- 172.21.15.77:0/1163555688 >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 0x7f7d540cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290197984 0 0) 0x7f7d5415ff90 con 0x7f7d5415bb40 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.540+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d3c003520 con 0x7f7d5415bb40 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 574091262 0 0) 0x7f7d3c003520 con 0x7f7d5415bb40 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 msgr2=0x7f7d540cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 --2- 172.21.15.77:0/1163555688 >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 0x7f7d540cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 >> v1:172.21.15.158:6789/0 conn(0x7f7d540b4970 legacy=0x7f7d5406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d54160550 con 0x7f7d5415bb40 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f7d44001670 con 0x7f7d5415bb40 2024-03-06T07:28:29.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 >> v1:172.21.15.77:6789/0 conn(0x7f7d5415bb40 legacy=0x7f7d5415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c0039b0 0x7f7d3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c004800 0x7f7d3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d3c0051c0 0x7f7d3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540d0440 con 0x7f7d3c0039b0 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c00c060 con 0x7f7d3c0051c0 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c00c260 con 0x7f7d3c004800 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d58dea640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c0039b0 0x7f7d3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d53fff640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c004800 0x7f7d3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d595eb640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d3c0051c0 0x7f7d3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d58dea640 1 -- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c004800 msgr2=0x7f7d3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d58dea640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c004800 0x7f7d3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d58dea640 1 -- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d3c0051c0 msgr2=0x7f7d3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d58dea640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d3c0051c0 0x7f7d3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d58dea640 1 -- 172.21.15.77:0/1163555688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c003520 con 0x7f7d3c0039b0 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d53fff640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c004800 0x7f7d3c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d595eb640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d3c0051c0 0x7f7d3c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.541+0000 7f7d595eb640 1 -- 172.21.15.77:0/1163555688 reap_dead start 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.542+0000 7f7d58dea640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c0039b0 0x7f7d3c003dd0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400c240 tx=0x7f7d3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae116e40bc779fe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.542+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44007d10 con 0x7f7d3c0039b0 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.542+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d44030070 con 0x7f7d3c0039b0 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.542+0000 7f7d537fe640 1 -- 172.21.15.77:0/1163555688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44005d70 con 0x7f7d3c0039b0 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.543+0000 7f7d5b04e640 1 -- 172.21.15.77:0/1163555688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c0039b0 msgr2=0x7f7d3c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.543+0000 7f7d5b04e640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c0039b0 0x7f7d3c003dd0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400c240 tx=0x7f7d3c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.543+0000 7f7d5b04e640 1 -- 172.21.15.77:0/1163555688 shutdown_connections 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.543+0000 7f7d5b04e640 1 --2- 172.21.15.77:0/1163555688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d3c0039b0 0x7f7d3c003dd0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.543+0000 7f7d5b04e640 1 -- 172.21.15.77:0/1163555688 >> 172.21.15.77:0/1163555688 conn(0x7f7d5406b190 msgr2=0x7f7d5406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.543+0000 7f7d5b04e640 1 -- 172.21.15.77:0/1163555688 shutdown_connections 2024-03-06T07:28:29.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.543+0000 7f7d5b04e640 1 -- 172.21.15.77:0/1163555688 wait complete. 2024-03-06T07:28:29.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.544+0000 7f7d5b04e640 1 Processor -- start 2024-03-06T07:28:29.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.544+0000 7f7d5b04e640 1 -- start start 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d5b04e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0039b0 0x7f7d54150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d5b04e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d54071e40 0x7f7d540ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d5b04e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d540722a0 0x7f7d540adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d5b04e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5406c910 con 0x7f7d540722a0 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d5b04e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5415bf20 con 0x7f7d54071e40 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d5b04e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5415c0a0 con 0x7f7d3c0039b0 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d540722a0 0x7f7d540adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d540722a0 0x7f7d540adda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40440/0 (socket says 172.21.15.77:40440) 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 -- 172.21.15.77:0/2173023489 learned_addr learned my addr 172.21.15.77:0/2173023489 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d58dea640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0039b0 0x7f7d54150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d53fff640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d54071e40 0x7f7d540ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 -- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0039b0 msgr2=0x7f7d54150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0039b0 0x7f7d54150440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 -- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d54071e40 msgr2=0x7f7d540ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d54071e40 0x7f7d540ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 -- 172.21.15.77:0/2173023489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5415ee90 con 0x7f7d540722a0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d58dea640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0039b0 0x7f7d54150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d53fff640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d54071e40 0x7f7d540ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d595eb640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d540722a0 0x7f7d540adda0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f7d5406ee70 tx=0x7f7d4c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d517fa640 1 -- 172.21.15.77:0/2173023489 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c010e30 con 0x7f7d540722a0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d517fa640 1 -- 172.21.15.77:0/2173023489 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c003990 con 0x7f7d540722a0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d517fa640 1 -- 172.21.15.77:0/2173023489 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c003d80 con 0x7f7d540722a0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d54151f10 con 0x7f7d540722a0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.545+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d54150c00 con 0x7f7d540722a0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.546+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d540732b0 con 0x7f7d540722a0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.547+0000 7f7d517fa640 1 -- 172.21.15.77:0/2173023489 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d4c01e020 con 0x7f7d540722a0 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.547+0000 7f7d517fa640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.547+0000 7f7d517fa640 1 -- 172.21.15.77:0/2173023489 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7d4c08a5c0 con 0x7f7d540722a0 2024-03-06T07:28:29.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.547+0000 7f7d58dea640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.548+0000 7f7d58dea640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7d44034000 tx=0x7f7d44039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.551+0000 7f7d517fa640 1 -- 172.21.15.77:0/2173023489 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d4c059700 con 0x7f7d540722a0 2024-03-06T07:28:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.721+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d54068ff0 con 0x7f7d540722a0 2024-03-06T07:28:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.721+0000 7f7d517fa640 1 -- 172.21.15.77:0/2173023489 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d4c059000 con 0x7f7d540722a0 2024-03-06T07:28:29.723 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 msgr2=0x7f7d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7d44034000 tx=0x7f7d44039040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d540722a0 msgr2=0x7f7d540adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d540722a0 0x7f7d540adda0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f7d5406ee70 tx=0x7f7d4c010590 comp rx=0 tx=0).stop 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 shutdown_connections 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d540722a0 0x7f7d540adda0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d54071e40 0x7f7d540ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 --2- 172.21.15.77:0/2173023489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d3c0039b0 0x7f7d54150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 >> 172.21.15.77:0/2173023489 conn(0x7f7d5406b190 msgr2=0x7f7d540b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 shutdown_connections 2024-03-06T07:28:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.723+0000 7f7d5b04e640 1 -- 172.21.15.77:0/2173023489 wait complete. 2024-03-06T07:28:29.736 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:29.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.836+0000 7f52f5c40640 1 Processor -- start 2024-03-06T07:28:29.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.836+0000 7f52f5c40640 1 -- start start 2024-03-06T07:28:29.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.836+0000 7f52f5c40640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f01424f0 0x7f52f00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.836+0000 7f52f5c40640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f52f006da40 0x7f52f015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.836+0000 7f52f5c40640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00cc980 con 0x7f52f00cd7a0 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.836+0000 7f52f5c40640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f00cd220 con 0x7f52f006da40 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.836+0000 7f52f5c40640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00ccc60 con 0x7f52f01424f0 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52effff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f52f006da40 0x7f52f015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52ef7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f01424f0 0x7f52f00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52eeffd640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f52f00cd7a0 0x7f52f006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49782/0 (socket says 172.21.15.77:49782) 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52effff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f52f006da40 0x7f52f015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:46806/0 (socket says 172.21.15.77:46806) 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52ef7fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f01424f0 0x7f52f00cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:36294/0 (socket says 172.21.15.77:36294) 2024-03-06T07:28:29.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52eeffd640 1 -- 172.21.15.77:0/2082666463 learned_addr learned my addr 172.21.15.77:0/2082666463 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52effff640 1 -- 172.21.15.77:0/2082666463 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f01424f0 msgr2=0x7f52f00cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52effff640 1 --2- 172.21.15.77:0/2082666463 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f01424f0 0x7f52f00cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52effff640 1 -- 172.21.15.77:0/2082666463 >> v1:172.21.15.158:6789/0 conn(0x7f52f00cd7a0 legacy=0x7f52f006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52effff640 1 -- 172.21.15.77:0/2082666463 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00ccf40 con 0x7f52f006da40 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.837+0000 7f52ef7fe640 1 --2- 172.21.15.77:0/2082666463 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f01424f0 0x7f52f00cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.838+0000 7f52effff640 1 --2- 172.21.15.77:0/2082666463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 0x7f52f015cef0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009d90 tx=0x7f52e802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a9c761ec1ac5ee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.838+0000 7f52ee7fc640 1 -- 172.21.15.77:0/2082666463 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8039070 con 0x7f52f006da40 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.838+0000 7f52ee7fc640 1 -- 172.21.15.77:0/2082666463 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e802cc50 con 0x7f52f006da40 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.838+0000 7f52ee7fc640 1 -- 172.21.15.77:0/2082666463 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8034440 con 0x7f52f006da40 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.839+0000 7f52f5c40640 1 -- 172.21.15.77:0/2082666463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 msgr2=0x7f52f015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.839+0000 7f52f5c40640 1 --2- 172.21.15.77:0/2082666463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 0x7f52f015cef0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009d90 tx=0x7f52e802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.839+0000 7f52f5c40640 1 -- 172.21.15.77:0/2082666463 shutdown_connections 2024-03-06T07:28:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.839+0000 7f52f5c40640 1 --2- 172.21.15.77:0/2082666463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 0x7f52f015cef0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.839+0000 7f52f5c40640 1 --2- 172.21.15.77:0/2082666463 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f01424f0 0x7f52f00cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.839+0000 7f52f5c40640 1 -- 172.21.15.77:0/2082666463 >> 172.21.15.77:0/2082666463 conn(0x7f52f00cf8a0 msgr2=0x7f52f00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.839+0000 7f52f5c40640 1 -- 172.21.15.77:0/2082666463 shutdown_connections 2024-03-06T07:28:29.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.839+0000 7f52f5c40640 1 -- 172.21.15.77:0/2082666463 wait complete. 2024-03-06T07:28:29.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.840+0000 7f52f5c40640 1 Processor -- start 2024-03-06T07:28:29.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.840+0000 7f52f5c40640 1 -- start start 2024-03-06T07:28:29.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52f5c40640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 0x7f52f015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52f5c40640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f00cd7a0 0x7f52f0144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52f5c40640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52f01424f0 0x7f52f0140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52f5c40640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0071ce0 con 0x7f52f01424f0 2024-03-06T07:28:29.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52f5c40640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0071b60 con 0x7f52f006da40 2024-03-06T07:28:29.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52f5c40640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0071e60 con 0x7f52f00cd7a0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52f01424f0 0x7f52f0140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52f01424f0 0x7f52f0140d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40452/0 (socket says 172.21.15.77:40452) 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 -- 172.21.15.77:0/3292284163 learned_addr learned my addr 172.21.15.77:0/3292284163 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52eeffd640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f00cd7a0 0x7f52f0144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52ef7fe640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 0x7f52f015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 -- 172.21.15.77:0/3292284163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f00cd7a0 msgr2=0x7f52f0144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f00cd7a0 0x7f52f0144420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 -- 172.21.15.77:0/3292284163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 msgr2=0x7f52f015b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 0x7f52f015b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 -- 172.21.15.77:0/3292284163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f0142dd0 con 0x7f52f01424f0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.841+0000 7f52effff640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52f01424f0 0x7f52f0140d90 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009d90 tx=0x7f52e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.842+0000 7f52d7fff640 1 -- 172.21.15.77:0/3292284163 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8039050 con 0x7f52f01424f0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.842+0000 7f52d7fff640 1 -- 172.21.15.77:0/3292284163 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e8045070 con 0x7f52f01424f0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.842+0000 7f52d7fff640 1 -- 172.21.15.77:0/3292284163 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8009450 con 0x7f52f01424f0 2024-03-06T07:28:29.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.842+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f01430b0 con 0x7f52f01424f0 2024-03-06T07:28:29.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.842+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f0143670 con 0x7f52f01424f0 2024-03-06T07:28:29.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.843+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f004f330 con 0x7f52f01424f0 2024-03-06T07:28:29.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.843+0000 7f52d7fff640 1 -- 172.21.15.77:0/3292284163 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52e803e610 con 0x7f52f01424f0 2024-03-06T07:28:29.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.843+0000 7f52d7fff640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f52dc06b4d0 0x7f52dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.844+0000 7f52d7fff640 1 -- 172.21.15.77:0/3292284163 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f52e80ac8f0 con 0x7f52f01424f0 2024-03-06T07:28:29.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.844+0000 7f52ef7fe640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f52dc06b4d0 0x7f52dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.844+0000 7f52ef7fe640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f52dc06b4d0 0x7f52dc06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f52f013e4a0 tx=0x7f52e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:29.848+0000 7f52d7fff640 1 -- 172.21.15.77:0/3292284163 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52e807bc70 con 0x7f52f01424f0 2024-03-06T07:28:30.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.022+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f52f0145b30 con 0x7f52f01424f0 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.023+0000 7f52d7fff640 1 -- 172.21.15.77:0/3292284163 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) v1 ==== 84+0+65 (secure 0 0 0) 0x7f52e807b570 con 0x7f52f01424f0 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f52dc06b4d0 msgr2=0x7f52dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f52dc06b4d0 0x7f52dc06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f52f013e4a0 tx=0x7f52e0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52f01424f0 msgr2=0x7f52f0140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52f01424f0 0x7f52f0140d90 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009d90 tx=0x7f52e8004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 shutdown_connections 2024-03-06T07:28:30.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f52dc06b4d0 0x7f52dc06d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52f01424f0 0x7f52f0140d90 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f52f00cd7a0 0x7f52f0144420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 --2- 172.21.15.77:0/3292284163 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f52f006da40 0x7f52f015b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 >> 172.21.15.77:0/3292284163 conn(0x7f52f00cf8a0 msgr2=0x7f52f00cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 shutdown_connections 2024-03-06T07:28:30.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.025+0000 7f52f5c40640 1 -- 172.21.15.77:0/3292284163 wait complete. 2024-03-06T07:28:30.038 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:30.038 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:30.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.140+0000 7f3068c88640 1 Processor -- start 2024-03-06T07:28:30.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.140+0000 7f3068c88640 1 -- start start 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f3068c88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30640cc6e0 0x7f306406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f3068c88640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306406f270 con 0x7f306415bd60 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f3068c88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306406f550 con 0x7f30640b57b0 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f3068c88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306415f750 con 0x7f30640cc6e0 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f3062d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30640cc6e0 0x7f306406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f306259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f306415bd60 0x7f30640b50a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54006/0 (socket says 172.21.15.77:54006) 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f306259c640 1 -- 172.21.15.77:0/611640523 learned_addr learned my addr 172.21.15.77:0/611640523 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f3062d9d640 1 -- 172.21.15.77:0/611640523 >> v1:172.21.15.158:6789/0 conn(0x7f30640b57b0 legacy=0x7f30640ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f3062d9d640 1 -- 172.21.15.77:0/611640523 >> v1:172.21.15.77:6789/0 conn(0x7f306415bd60 legacy=0x7f30640b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.141+0000 7f3062d9d640 1 -- 172.21.15.77:0/611640523 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306415f9a0 con 0x7f30640cc6e0 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f3062d9d640 1 --2- 172.21.15.77:0/611640523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640cc6e0 0x7f306406ded0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f3054009d90 tx=0x7f305402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae54ff19e324f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f306159a640 1 -- 172.21.15.77:0/611640523 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054039070 con 0x7f30640cc6e0 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f306159a640 1 -- 172.21.15.77:0/611640523 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305402cb40 con 0x7f30640cc6e0 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f306159a640 1 -- 172.21.15.77:0/611640523 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054034440 con 0x7f30640cc6e0 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f3068c88640 1 -- 172.21.15.77:0/611640523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640cc6e0 msgr2=0x7f306406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f3068c88640 1 --2- 172.21.15.77:0/611640523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640cc6e0 0x7f306406ded0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f3054009d90 tx=0x7f305402bd90 comp rx=0 tx=0).stop 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f3068c88640 1 -- 172.21.15.77:0/611640523 shutdown_connections 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f3068c88640 1 --2- 172.21.15.77:0/611640523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640cc6e0 0x7f306406ded0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.142+0000 7f3068c88640 1 -- 172.21.15.77:0/611640523 >> 172.21.15.77:0/611640523 conn(0x7f30640b1a90 msgr2=0x7f30640b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.143+0000 7f3068c88640 1 -- 172.21.15.77:0/611640523 shutdown_connections 2024-03-06T07:28:30.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.143+0000 7f3068c88640 1 -- 172.21.15.77:0/611640523 wait complete. 2024-03-06T07:28:30.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.144+0000 7f3068c88640 1 Processor -- start 2024-03-06T07:28:30.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.144+0000 7f3068c88640 1 -- start start 2024-03-06T07:28:30.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.144+0000 7f3068c88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640b57b0 0x7f306415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.144+0000 7f3068c88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30640cc6e0 0x7f306415c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f3068c88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306415bd60 0x7f306415e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f3068c88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30641605b0 con 0x7f30640b57b0 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f3068c88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3064160430 con 0x7f306415bd60 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f3068c88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306415fd10 con 0x7f30640cc6e0 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640b57b0 0x7f306415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640b57b0 0x7f306415d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40480/0 (socket says 172.21.15.77:40480) 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 -- 172.21.15.77:0/3636437818 learned_addr learned my addr 172.21.15.77:0/3636437818 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f3061d9b640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30640cc6e0 0x7f306415c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f3062d9d640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306415bd60 0x7f306415e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 -- 172.21.15.77:0/3636437818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30640cc6e0 msgr2=0x7f306415c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30640cc6e0 0x7f306415c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 -- 172.21.15.77:0/3636437818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306415bd60 msgr2=0x7f306415e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306415bd60 0x7f306415e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 -- 172.21.15.77:0/3636437818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306415c8d0 con 0x7f30640b57b0 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f306259c640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640b57b0 0x7f306415d970 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c3f0 tx=0x7f305c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f305affd640 1 -- 172.21.15.77:0/3636437818 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c017070 con 0x7f30640b57b0 2024-03-06T07:28:30.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f305affd640 1 -- 172.21.15.77:0/3636437818 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305c005ce0 con 0x7f30640b57b0 2024-03-06T07:28:30.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306415cbb0 con 0x7f30640b57b0 2024-03-06T07:28:30.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f305affd640 1 -- 172.21.15.77:0/3636437818 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c0072f0 con 0x7f30640b57b0 2024-03-06T07:28:30.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.145+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306415d170 con 0x7f30640b57b0 2024-03-06T07:28:30.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.146+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306404f330 con 0x7f30640b57b0 2024-03-06T07:28:30.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.147+0000 7f305affd640 1 -- 172.21.15.77:0/3636437818 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f305c01a070 con 0x7f30640b57b0 2024-03-06T07:28:30.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.147+0000 7f305affd640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f304406b4d0 0x7f304406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.147+0000 7f3061d9b640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f304406b4d0 0x7f304406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.147+0000 7f305affd640 1 -- 172.21.15.77:0/3636437818 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f305c0891c0 con 0x7f30640b57b0 2024-03-06T07:28:30.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.148+0000 7f3061d9b640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f304406b4d0 0x7f304406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f304c007fd0 tx=0x7f304c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.151+0000 7f305affd640 1 -- 172.21.15.77:0/3636437818 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f305c058420 con 0x7f30640b57b0 2024-03-06T07:28:30.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.324+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f306406eeb0 con 0x7f30640b57b0 2024-03-06T07:28:30.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.760+0000 7f305affd640 1 -- 172.21.15.77:0/3636437818 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v122) v1 ==== 138+0+0 (secure 0 0 0) 0x7f305c057d20 con 0x7f30640b57b0 2024-03-06T07:28:30.762 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f304406b4d0 msgr2=0x7f304406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f304406b4d0 0x7f304406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f304c007fd0 tx=0x7f304c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640b57b0 msgr2=0x7f306415d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640b57b0 0x7f306415d970 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c3f0 tx=0x7f305c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 shutdown_connections 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f306415bd60 0x7f306415e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30640cc6e0 0x7f306415c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f304406b4d0 0x7f304406d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 --2- 172.21.15.77:0/3636437818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30640b57b0 0x7f306415d970 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 >> 172.21.15.77:0/3636437818 conn(0x7f30640b1a90 msgr2=0x7f3064071570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 shutdown_connections 2024-03-06T07:28:30.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.763+0000 7f3068c88640 1 -- 172.21.15.77:0/3636437818 wait complete. 2024-03-06T07:28:30.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.887+0000 7f604954b640 1 Processor -- start 2024-03-06T07:28:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.887+0000 7f604954b640 1 -- start start 2024-03-06T07:28:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f604954b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60440d4860 0x7f60440d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f604954b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60440b0e30 0x7f60440b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f604954b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60441618b0 con 0x7f60441614d0 2024-03-06T07:28:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f604954b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60440d4cc0 con 0x7f60440b0e30 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f604954b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60440cf400 con 0x7f60440d4860 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f60437fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60440b0e30 0x7f60440b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f60437fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f60440b0e30 0x7f60440b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40490/0 (socket says 172.21.15.77:40490) 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f60437fe640 1 -- 172.21.15.77:0/2708075333 learned_addr learned my addr 172.21.15.77:0/2708075333 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f603bfff640 1 --2- 172.21.15.77:0/2708075333 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60440d4860 0x7f60440d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f60437fe640 1 -- 172.21.15.77:0/2708075333 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60440d4860 msgr2=0x7f60440d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f60437fe640 1 --2- 172.21.15.77:0/2708075333 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60440d4860 0x7f60440d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f60437fe640 1 -- 172.21.15.77:0/2708075333 >> v1:172.21.15.158:6789/0 conn(0x7f60441614d0 legacy=0x7f60440d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.888+0000 7f60437fe640 1 -- 172.21.15.77:0/2708075333 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440cf6a0 con 0x7f60440b0e30 2024-03-06T07:28:30.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.889+0000 7f60437fe640 1 --2- 172.21.15.77:0/2708075333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f60440b12c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f603c009e50 tx=0x7f603c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b7b0ce68931c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.889+0000 7f60427fc640 1 -- 172.21.15.77:0/2708075333 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c02ca70 con 0x7f60440b0e30 2024-03-06T07:28:30.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.889+0000 7f60427fc640 1 -- 172.21.15.77:0/2708075333 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c02cc10 con 0x7f60440b0e30 2024-03-06T07:28:30.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.889+0000 7f60427fc640 1 -- 172.21.15.77:0/2708075333 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c0343f0 con 0x7f60440b0e30 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.890+0000 7f604954b640 1 -- 172.21.15.77:0/2708075333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 msgr2=0x7f60440b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.890+0000 7f604954b640 1 --2- 172.21.15.77:0/2708075333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f60440b12c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f603c009e50 tx=0x7f603c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.890+0000 7f604954b640 1 -- 172.21.15.77:0/2708075333 shutdown_connections 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.890+0000 7f604954b640 1 --2- 172.21.15.77:0/2708075333 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60440d4860 0x7f60440d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.890+0000 7f604954b640 1 --2- 172.21.15.77:0/2708075333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f60440b12c0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.890+0000 7f604954b640 1 -- 172.21.15.77:0/2708075333 >> 172.21.15.77:0/2708075333 conn(0x7f60440d3d40 msgr2=0x7f60440d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.890+0000 7f604954b640 1 -- 172.21.15.77:0/2708075333 shutdown_connections 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.890+0000 7f604954b640 1 -- 172.21.15.77:0/2708075333 wait complete. 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.891+0000 7f604954b640 1 Processor -- start 2024-03-06T07:28:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.891+0000 7f604954b640 1 -- start start 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.891+0000 7f604954b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f6044159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.891+0000 7f604954b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60440d4860 0x7f60440ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.891+0000 7f604954b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60441614d0 0x7f60440c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f604954b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440b0910 con 0x7f60440b0e30 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f604954b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440cfc60 con 0x7f60440d4860 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f604954b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60440d2560 con 0x7f60441614d0 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f6044159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f6044159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40500/0 (socket says 172.21.15.77:40500) 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 -- 172.21.15.77:0/1157016616 learned_addr learned my addr 172.21.15.77:0/1157016616 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f603bfff640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60440d4860 0x7f60440ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f60437fe640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60441614d0 0x7f60440c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 -- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60441614d0 msgr2=0x7f60440c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60441614d0 0x7f60440c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 -- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60440d4860 msgr2=0x7f60440ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60440d4860 0x7f60440ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 -- 172.21.15.77:0/1157016616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604415a0e0 con 0x7f60440b0e30 2024-03-06T07:28:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f603bfff640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60440d4860 0x7f60440ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f6042ffd640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f6044159ba0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f603400c3c0 tx=0x7f603400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f603b7fe640 1 -- 172.21.15.77:0/1157016616 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034011790 con 0x7f60440b0e30 2024-03-06T07:28:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f603b7fe640 1 -- 172.21.15.77:0/1157016616 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034017070 con 0x7f60440b0e30 2024-03-06T07:28:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60440b38c0 con 0x7f60440b0e30 2024-03-06T07:28:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f603b7fe640 1 -- 172.21.15.77:0/1157016616 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034011b70 con 0x7f60440b0e30 2024-03-06T07:28:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.892+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60440b3e60 con 0x7f60440b0e30 2024-03-06T07:28:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.893+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60440ce170 con 0x7f60440b0e30 2024-03-06T07:28:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.894+0000 7f603b7fe640 1 -- 172.21.15.77:0/1157016616 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6034011d10 con 0x7f60440b0e30 2024-03-06T07:28:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.894+0000 7f603b7fe640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f602406b4d0 0x7f602406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.894+0000 7f603b7fe640 1 -- 172.21.15.77:0/1157016616 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6034089d60 con 0x7f60440b0e30 2024-03-06T07:28:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.894+0000 7f603bfff640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f602406b4d0 0x7f602406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.895+0000 7f603bfff640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f602406b4d0 0x7f602406d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f60440cf180 tx=0x7f602c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:30.899+0000 7f603b7fe640 1 -- 172.21.15.77:0/1157016616 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6034058d30 con 0x7f60440b0e30 2024-03-06T07:28:31.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:31.007 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.075+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6044068ff0 con 0x7f60440b0e30 2024-03-06T07:28:31.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.763+0000 7f603b7fe640 1 -- 172.21.15.77:0/1157016616 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v123) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6034058630 con 0x7f60440b0e30 2024-03-06T07:28:31.765 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f602406b4d0 msgr2=0x7f602406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f602406b4d0 0x7f602406d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f60440cf180 tx=0x7f602c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 msgr2=0x7f6044159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f6044159ba0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f603400c3c0 tx=0x7f603400c880 comp rx=0 tx=0).stop 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 shutdown_connections 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f60441614d0 0x7f60440c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60440d4860 0x7f60440ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f602406b4d0 0x7f602406d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 --2- 172.21.15.77:0/1157016616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60440b0e30 0x7f6044159ba0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 >> 172.21.15.77:0/1157016616 conn(0x7f60440d3d40 msgr2=0x7f60440d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 shutdown_connections 2024-03-06T07:28:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.765+0000 7f604954b640 1 -- 172.21.15.77:0/1157016616 wait complete. 2024-03-06T07:28:31.790 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:31.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.893+0000 7fbf93365640 1 Processor -- start 2024-03-06T07:28:31.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf93365640 1 -- start start 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf93365640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf8c158420 0x7fbf8c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf93365640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbf8c15ba80 0x7fbf8c0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf93365640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c15e8a0 con 0x7fbf8c158800 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf93365640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c15e650 con 0x7fbf8c15ba80 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf93365640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c15eb80 con 0x7fbf8c158420 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf91902640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbf8c15ba80 0x7fbf8c0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf90900640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbf8c158800 0x7fbf8c15b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54010/0 (socket says 172.21.15.77:54010) 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.894+0000 7fbf90900640 1 -- 172.21.15.77:0/1502573834 learned_addr learned my addr 172.21.15.77:0/1502573834 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.895+0000 7fbf91902640 1 -- 172.21.15.77:0/1502573834 >> v2:172.21.15.158:3300/0 conn(0x7fbf8c158420 msgr2=0x7fbf8c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.895+0000 7fbf91101640 1 --2- 172.21.15.77:0/1502573834 >> v2:172.21.15.158:3300/0 conn(0x7fbf8c158420 0x7fbf8c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.895+0000 7fbf83fff640 1 -- 172.21.15.77:0/1502573834 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410972253 0 0) 0x7fbf8c15e8a0 con 0x7fbf8c158800 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.895+0000 7fbf91902640 1 --2- 172.21.15.77:0/1502573834 >> v2:172.21.15.158:3300/0 conn(0x7fbf8c158420 0x7fbf8c0b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.895+0000 7fbf91902640 1 -- 172.21.15.77:0/1502573834 >> v1:172.21.15.77:6789/0 conn(0x7fbf8c158800 legacy=0x7fbf8c15b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.895+0000 7fbf91902640 1 -- 172.21.15.77:0/1502573834 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c15ee60 con 0x7fbf8c15ba80 2024-03-06T07:28:31.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.895+0000 7fbf91101640 1 --2- 172.21.15.77:0/1502573834 >> v2:172.21.15.158:3300/0 conn(0x7fbf8c158420 0x7fbf8c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:31.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf91902640 1 --2- 172.21.15.77:0/1502573834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c15ba80 0x7fbf8c0b3b00 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0d0470 tx=0x7fbf8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527d52c87997e2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf83fff640 1 -- 172.21.15.77:0/1502573834 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf840315a0 con 0x7fbf8c15ba80 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf83fff640 1 -- 172.21.15.77:0/1502573834 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf84037070 con 0x7fbf8c15ba80 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf83fff640 1 -- 172.21.15.77:0/1502573834 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84031960 con 0x7fbf8c15ba80 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf93365640 1 -- 172.21.15.77:0/1502573834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c15ba80 msgr2=0x7fbf8c0b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf93365640 1 --2- 172.21.15.77:0/1502573834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c15ba80 0x7fbf8c0b3b00 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0d0470 tx=0x7fbf8402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf93365640 1 -- 172.21.15.77:0/1502573834 shutdown_connections 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf93365640 1 --2- 172.21.15.77:0/1502573834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c15ba80 0x7fbf8c0b3b00 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf93365640 1 --2- 172.21.15.77:0/1502573834 >> v2:172.21.15.158:3300/0 conn(0x7fbf8c158420 0x7fbf8c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf93365640 1 -- 172.21.15.77:0/1502573834 >> 172.21.15.77:0/1502573834 conn(0x7fbf8c06afb0 msgr2=0x7fbf8c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf93365640 1 -- 172.21.15.77:0/1502573834 shutdown_connections 2024-03-06T07:28:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.896+0000 7fbf93365640 1 -- 172.21.15.77:0/1502573834 wait complete. 2024-03-06T07:28:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.897+0000 7fbf93365640 1 Processor -- start 2024-03-06T07:28:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.897+0000 7fbf93365640 1 -- start start 2024-03-06T07:28:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf93365640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf8c158420 0x7fbf8c1f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf93365640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf8c15ba80 0x7fbf8c155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf93365640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c0ae010 0x7fbf8c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf93365640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c071ba0 con 0x7fbf8c0ae010 2024-03-06T07:28:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf93365640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c071a20 con 0x7fbf8c158420 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf93365640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c071d20 con 0x7fbf8c15ba80 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf91902640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c0ae010 0x7fbf8c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf91902640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c0ae010 0x7fbf8c06d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40514/0 (socket says 172.21.15.77:40514) 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf91902640 1 -- 172.21.15.77:0/2969802089 learned_addr learned my addr 172.21.15.77:0/2969802089 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf90900640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf8c15ba80 0x7fbf8c155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf91902640 1 -- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf8c15ba80 msgr2=0x7fbf8c155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf91902640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf8c15ba80 0x7fbf8c155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf91902640 1 -- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf8c158420 msgr2=0x7fbf8c1f7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.898+0000 7fbf91101640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf8c158420 0x7fbf8c1f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf91902640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf8c158420 0x7fbf8c1f7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf91902640 1 -- 172.21.15.77:0/2969802089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c06dd60 con 0x7fbf8c0ae010 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf91101640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf8c158420 0x7fbf8c1f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf91902640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c0ae010 0x7fbf8c06d3f0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbf84034040 tx=0x7fbf840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf81ffb640 1 -- 172.21.15.77:0/2969802089 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84032840 con 0x7fbf8c0ae010 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf81ffb640 1 -- 172.21.15.77:0/2969802089 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf840329e0 con 0x7fbf8c0ae010 2024-03-06T07:28:31.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c06e040 con 0x7fbf8c0ae010 2024-03-06T07:28:31.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf81ffb640 1 -- 172.21.15.77:0/2969802089 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84032dd0 con 0x7fbf8c0ae010 2024-03-06T07:28:31.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.899+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8c1f7280 con 0x7fbf8c0ae010 2024-03-06T07:28:31.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.900+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8c04f330 con 0x7fbf8c0ae010 2024-03-06T07:28:31.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.900+0000 7fbf81ffb640 1 -- 172.21.15.77:0/2969802089 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf8404c020 con 0x7fbf8c0ae010 2024-03-06T07:28:31.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.901+0000 7fbf81ffb640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf7006b4d0 0x7fbf7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.901+0000 7fbf81ffb640 1 -- 172.21.15.77:0/2969802089 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fbf8402f020 con 0x7fbf8c0ae010 2024-03-06T07:28:31.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.901+0000 7fbf91101640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf7006b4d0 0x7fbf7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.901+0000 7fbf91101640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf7006b4d0 0x7fbf7006d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c159eb0 tx=0x7fbf88007550 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:31.905+0000 7fbf81ffb640 1 -- 172.21.15.77:0/2969802089 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf84040070 con 0x7fbf8c0ae010 2024-03-06T07:28:32.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.112+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbf8c06b740 con 0x7fbf8c0ae010 2024-03-06T07:28:32.799 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:28:32.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.797+0000 7fbf81ffb640 1 -- 172.21.15.77:0/2969802089 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v78) v1 ==== 174+0+0 (secure 0 0 0) 0x7fbf8407aee0 con 0x7fbf8c0ae010 2024-03-06T07:28:32.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.803+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf7006b4d0 msgr2=0x7fbf7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.803+0000 7fbf93365640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf7006b4d0 0x7fbf7006d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c159eb0 tx=0x7fbf88007550 comp rx=0 tx=0).stop 2024-03-06T07:28:32.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.803+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c0ae010 msgr2=0x7fbf8c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.803+0000 7fbf93365640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c0ae010 0x7fbf8c06d3f0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbf84034040 tx=0x7fbf840046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.804+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 shutdown_connections 2024-03-06T07:28:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.804+0000 7fbf93365640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf7006b4d0 0x7fbf7006d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.804+0000 7fbf93365640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c0ae010 0x7fbf8c06d3f0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.804+0000 7fbf93365640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf8c15ba80 0x7fbf8c155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.804+0000 7fbf93365640 1 --2- 172.21.15.77:0/2969802089 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf8c158420 0x7fbf8c1f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.804+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 >> 172.21.15.77:0/2969802089 conn(0x7fbf8c06afb0 msgr2=0x7fbf8c15c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.805+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 shutdown_connections 2024-03-06T07:28:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.806+0000 7fbf93365640 1 -- 172.21.15.77:0/2969802089 wait complete. 2024-03-06T07:28:32.823 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:32.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.921+0000 7f399e7e9640 1 Processor -- start 2024-03-06T07:28:32.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.921+0000 7f399e7e9640 1 -- start start 2024-03-06T07:28:32.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f399e7e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f399806b220 0x7f399815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f399e7e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f3998162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f399e7e9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399815ea30 con 0x7f39980ce920 2024-03-06T07:28:32.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f399e7e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3998161b70 con 0x7f399806b220 2024-03-06T07:28:32.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f399e7e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399815ed10 con 0x7f399806b600 2024-03-06T07:28:32.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f399cd86640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f3998162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f3997fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f399806b220 0x7f399815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f3997fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f399806b220 0x7f399815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:46864/0 (socket says 172.21.15.77:46864) 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f399cd86640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f3998162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:36368/0 (socket says 172.21.15.77:36368) 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f3997fff640 1 -- 172.21.15.77:0/1799952614 learned_addr learned my addr 172.21.15.77:0/1799952614 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.922+0000 7f39977fe640 1 --1- 172.21.15.77:0/1799952614 >> v1:172.21.15.158:6789/0 conn(0x7f39980ce920 0x7f3998071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49814/0 (socket says 172.21.15.77:49814) 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.923+0000 7f399cd86640 1 -- 172.21.15.77:0/1799952614 >> v2:172.21.15.158:3300/0 conn(0x7f399806b220 msgr2=0x7f399815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.923+0000 7f399cd86640 1 --2- 172.21.15.77:0/1799952614 >> v2:172.21.15.158:3300/0 conn(0x7f399806b220 0x7f399815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.923+0000 7f399cd86640 1 -- 172.21.15.77:0/1799952614 >> v1:172.21.15.158:6789/0 conn(0x7f39980ce920 legacy=0x7f3998071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.923+0000 7f399cd86640 1 -- 172.21.15.77:0/1799952614 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399815eff0 con 0x7f399806b600 2024-03-06T07:28:32.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.923+0000 7f399cd86640 1 --2- 172.21.15.77:0/1799952614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f3998162000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3990009d90 tx=0x7f399002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1687a5006ca54c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f3996ffd640 1 -- 172.21.15.77:0/1799952614 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990039070 con 0x7f399806b600 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f3996ffd640 1 -- 172.21.15.77:0/1799952614 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399002cbc0 con 0x7f399806b600 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f3996ffd640 1 -- 172.21.15.77:0/1799952614 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990034440 con 0x7f399806b600 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f399e7e9640 1 -- 172.21.15.77:0/1799952614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 msgr2=0x7f3998162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f399e7e9640 1 --2- 172.21.15.77:0/1799952614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f3998162000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3990009d90 tx=0x7f399002be10 comp rx=0 tx=0).stop 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f399e7e9640 1 -- 172.21.15.77:0/1799952614 shutdown_connections 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f399e7e9640 1 --2- 172.21.15.77:0/1799952614 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f3998162000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f399e7e9640 1 --2- 172.21.15.77:0/1799952614 >> v2:172.21.15.158:3300/0 conn(0x7f399806b220 0x7f399815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f399e7e9640 1 -- 172.21.15.77:0/1799952614 >> 172.21.15.77:0/1799952614 conn(0x7f399806c0c0 msgr2=0x7f399806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f399e7e9640 1 -- 172.21.15.77:0/1799952614 shutdown_connections 2024-03-06T07:28:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.924+0000 7f399e7e9640 1 -- 172.21.15.77:0/1799952614 wait complete. 2024-03-06T07:28:32.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.925+0000 7f399e7e9640 1 Processor -- start 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.925+0000 7f399e7e9640 1 -- start start 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399e7e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399806b220 0x7f3998146510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399e7e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f39981541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399e7e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39980ce920 0x7f399806e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399e7e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3998072090 con 0x7f39980ce920 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399e7e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980ce200 con 0x7f399806b220 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399e7e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3998154c40 con 0x7f399806b600 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399cd86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39980ce920 0x7f399806e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399cd86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39980ce920 0x7f399806e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40522/0 (socket says 172.21.15.77:40522) 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f399cd86640 1 -- 172.21.15.77:0/555742502 learned_addr learned my addr 172.21.15.77:0/555742502 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f39977fe640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f39981541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.926+0000 7f3997fff640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399806b220 0x7f3998146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f399cd86640 1 -- 172.21.15.77:0/555742502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 msgr2=0x7f39981541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f399cd86640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f39981541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f399cd86640 1 -- 172.21.15.77:0/555742502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399806b220 msgr2=0x7f3998146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f399cd86640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399806b220 0x7f3998146510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f399cd86640 1 -- 172.21.15.77:0/555742502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998154f20 con 0x7f39980ce920 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f399cd86640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39980ce920 0x7f399806e1d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f3990009d90 tx=0x7f3990004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f3994ff9640 1 -- 172.21.15.77:0/555742502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990039050 con 0x7f39980ce920 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f3994ff9640 1 -- 172.21.15.77:0/555742502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3990045070 con 0x7f39980ce920 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3998155200 con 0x7f39980ce920 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f3994ff9640 1 -- 172.21.15.77:0/555742502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990009450 con 0x7f39980ce920 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.927+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39981557c0 con 0x7f39980ce920 2024-03-06T07:28:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.928+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399804f330 con 0x7f39980ce920 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.928+0000 7f3994ff9640 1 -- 172.21.15.77:0/555742502 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f399003e850 con 0x7f39980ce920 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.929+0000 7f3994ff9640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398406b4d0 0x7f398406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.929+0000 7f3994ff9640 1 -- 172.21.15.77:0/555742502 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39900ad0d0 con 0x7f39980ce920 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.929+0000 7f3997fff640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398406b4d0 0x7f398406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.929+0000 7f3997fff640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398406b4d0 0x7f398406d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3998162a00 tx=0x7f3988009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:32.933+0000 7f3994ff9640 1 -- 172.21.15.77:0/555742502 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f399007ba20 con 0x7f39980ce920 2024-03-06T07:28:33.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.105+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39980cdf00 con 0x7f39980ce920 2024-03-06T07:28:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.106+0000 7f3994ff9640 1 -- 172.21.15.77:0/555742502 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f399007fac0 con 0x7f39980ce920 2024-03-06T07:28:33.108 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:33.108 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":124,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:32.113683+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:30.343353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:31.094099+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:33.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398406b4d0 msgr2=0x7f398406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398406b4d0 0x7f398406d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3998162a00 tx=0x7f3988009240 comp rx=0 tx=0).stop 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39980ce920 msgr2=0x7f399806e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39980ce920 0x7f399806e1d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f3990009d90 tx=0x7f3990004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 shutdown_connections 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398406b4d0 0x7f398406d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39980ce920 0x7f399806e1d0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399806b600 0x7f39981541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 --2- 172.21.15.77:0/555742502 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399806b220 0x7f3998146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.108+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 >> 172.21.15.77:0/555742502 conn(0x7f399806c0c0 msgr2=0x7f39980d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.109+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 shutdown_connections 2024-03-06T07:28:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.109+0000 7f399e7e9640 1 -- 172.21.15.77:0/555742502 wait complete. 2024-03-06T07:28:33.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:33.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.218+0000 7f0460325640 1 Processor -- start 2024-03-06T07:28:33.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.218+0000 7f0460325640 1 -- start start 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.219+0000 7f0460325640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 0x7f04580cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.219+0000 7f0460325640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f04580cd7a0 0x7f045815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.219+0000 7f0460325640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04580cc980 con 0x7f04581424f0 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.219+0000 7f0460325640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04580cd220 con 0x7f04580cd7a0 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.219+0000 7f0460325640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04580ccc60 con 0x7f045806da40 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.219+0000 7f045e8c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f04580cd7a0 0x7f045815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.219+0000 7f045e8c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f04580cd7a0 0x7f045815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40538/0 (socket says 172.21.15.77:40538) 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.219+0000 7f045e8c2640 1 -- 172.21.15.77:0/2995518541 learned_addr learned my addr 172.21.15.77:0/2995518541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.220+0000 7f045d8c0640 1 --2- 172.21.15.77:0/2995518541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 0x7f04580cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.220+0000 7f045e8c2640 1 -- 172.21.15.77:0/2995518541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 msgr2=0x7f04580cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.220+0000 7f045e8c2640 1 --2- 172.21.15.77:0/2995518541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 0x7f04580cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.220+0000 7f045e8c2640 1 -- 172.21.15.77:0/2995518541 >> v1:172.21.15.158:6789/0 conn(0x7f04581424f0 legacy=0x7f045806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.220+0000 7f045e8c2640 1 -- 172.21.15.77:0/2995518541 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580ccf40 con 0x7f04580cd7a0 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.220+0000 7f045e8c2640 1 --2- 172.21.15.77:0/2995518541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04580cd7a0 0x7f045815cef0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f0450009e50 tx=0x7f045002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5ec62d08ceffcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f045d0bf640 1 -- 172.21.15.77:0/2995518541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045002ca70 con 0x7f04580cd7a0 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f045d0bf640 1 -- 172.21.15.77:0/2995518541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045002cc10 con 0x7f04580cd7a0 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f045d0bf640 1 -- 172.21.15.77:0/2995518541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04500343f0 con 0x7f04580cd7a0 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f0460325640 1 -- 172.21.15.77:0/2995518541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04580cd7a0 msgr2=0x7f045815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f0460325640 1 --2- 172.21.15.77:0/2995518541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04580cd7a0 0x7f045815cef0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f0450009e50 tx=0x7f045002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f0460325640 1 -- 172.21.15.77:0/2995518541 shutdown_connections 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f0460325640 1 --2- 172.21.15.77:0/2995518541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 0x7f04580cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f0460325640 1 --2- 172.21.15.77:0/2995518541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04580cd7a0 0x7f045815cef0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f0460325640 1 -- 172.21.15.77:0/2995518541 >> 172.21.15.77:0/2995518541 conn(0x7f04580cf8a0 msgr2=0x7f04580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f0460325640 1 -- 172.21.15.77:0/2995518541 shutdown_connections 2024-03-06T07:28:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.221+0000 7f0460325640 1 -- 172.21.15.77:0/2995518541 wait complete. 2024-03-06T07:28:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.222+0000 7f0460325640 1 Processor -- start 2024-03-06T07:28:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.222+0000 7f0460325640 1 -- start start 2024-03-06T07:28:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f0460325640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 0x7f0458143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f0460325640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04580cd7a0 0x7f045815b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f0460325640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04581424f0 0x7f04581e68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f0460325640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0458071ce0 con 0x7f04581424f0 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f0460325640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0458071b60 con 0x7f04580cd7a0 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f0460325640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0458071e60 con 0x7f045806da40 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f045e8c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04581424f0 0x7f04581e68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f045e8c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04581424f0 0x7f04581e68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40548/0 (socket says 172.21.15.77:40548) 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f045e8c2640 1 -- 172.21.15.77:0/52105675 learned_addr learned my addr 172.21.15.77:0/52105675 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f045e8c2640 1 -- 172.21.15.77:0/52105675 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 msgr2=0x7f0458143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f045e8c2640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 0x7f0458143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f045e8c2640 1 -- 172.21.15.77:0/52105675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04580cd7a0 msgr2=0x7f045815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f045d8c0640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04580cd7a0 0x7f045815b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.223+0000 7f045e8c2640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04580cd7a0 0x7f045815b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.224+0000 7f045e8c2640 1 -- 172.21.15.77:0/52105675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0458143650 con 0x7f04581424f0 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.224+0000 7f045e8c2640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04581424f0 0x7f04581e68d0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f0450036040 tx=0x7f0450004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.224+0000 7f044effd640 1 -- 172.21.15.77:0/52105675 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450043020 con 0x7f04581424f0 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.224+0000 7f044effd640 1 -- 172.21.15.77:0/52105675 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045003e450 con 0x7f04581424f0 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.224+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0458143930 con 0x7f04581424f0 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.224+0000 7f044effd640 1 -- 172.21.15.77:0/52105675 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045003e820 con 0x7f04581424f0 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.224+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0458143ef0 con 0x7f04581424f0 2024-03-06T07:28:33.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.225+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045804f330 con 0x7f04581424f0 2024-03-06T07:28:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.225+0000 7f044effd640 1 -- 172.21.15.77:0/52105675 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0450046020 con 0x7f04581424f0 2024-03-06T07:28:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.226+0000 7f044effd640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f043c06b4d0 0x7f043c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.226+0000 7f044effd640 1 -- 172.21.15.77:0/52105675 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04500acb40 con 0x7f04581424f0 2024-03-06T07:28:33.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.226+0000 7f045e0c1640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f043c06b4d0 0x7f043c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.226+0000 7f045e0c1640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f043c06b4d0 0x7f043c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f04580af1e0 tx=0x7f0448006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.230+0000 7f044effd640 1 -- 172.21.15.77:0/52105675 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f045004d030 con 0x7f04581424f0 2024-03-06T07:28:33.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.402+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04580b0390 con 0x7f04581424f0 2024-03-06T07:28:33.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.403+0000 7f044effd640 1 -- 172.21.15.77:0/52105675 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f045007b9c0 con 0x7f04581424f0 2024-03-06T07:28:33.405 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:33.405 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":124,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:32.113683+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:30.343353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:31.094099+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:33.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f043c06b4d0 msgr2=0x7f043c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f043c06b4d0 0x7f043c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f04580af1e0 tx=0x7f0448006ba0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04581424f0 msgr2=0x7f04581e68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04581424f0 0x7f04581e68d0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f0450036040 tx=0x7f0450004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 shutdown_connections 2024-03-06T07:28:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f043c06b4d0 0x7f043c06d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04581424f0 0x7f04581e68d0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04580cd7a0 0x7f045815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 --2- 172.21.15.77:0/52105675 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f045806da40 0x7f0458143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 >> 172.21.15.77:0/52105675 conn(0x7f04580cf8a0 msgr2=0x7f045815ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 shutdown_connections 2024-03-06T07:28:33.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.405+0000 7f0460325640 1 -- 172.21.15.77:0/52105675 wait complete. 2024-03-06T07:28:33.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:33.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.519+0000 7f199c15e640 1 Processor -- start 2024-03-06T07:28:33.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.520+0000 7f199c15e640 1 -- start start 2024-03-06T07:28:33.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.520+0000 7f199c15e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415bb40 0x7f19940b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.520+0000 7f199c15e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199415ff90 con 0x7f199415f7d0 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.520+0000 7f199c15e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1994160270 con 0x7f19940cfa90 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.520+0000 7f199c15e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199406c910 con 0x7f199415bb40 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.520+0000 7f19996f9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f199415f7d0 0x7f199406d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54012/0 (socket says 172.21.15.77:54012) 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.521+0000 7f19996f9640 1 -- 172.21.15.77:0/2902949164 learned_addr learned my addr 172.21.15.77:0/2902949164 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.521+0000 7f1999efa640 1 --2- 172.21.15.77:0/2902949164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415bb40 0x7f19940b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.521+0000 7f1998ef8640 1 -- 172.21.15.77:0/2902949164 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360463182 0 0) 0x7f199415ff90 con 0x7f199415f7d0 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.521+0000 7f1998ef8640 1 -- 172.21.15.77:0/2902949164 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1980003520 con 0x7f199415f7d0 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.521+0000 7f1999efa640 1 -- 172.21.15.77:0/2902949164 >> v1:172.21.15.158:6789/0 conn(0x7f19940cfa90 legacy=0x7f19940cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.521+0000 7f1999efa640 1 -- 172.21.15.77:0/2902949164 >> v1:172.21.15.77:6789/0 conn(0x7f199415f7d0 legacy=0x7f199406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.521+0000 7f1999efa640 1 -- 172.21.15.77:0/2902949164 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1994160550 con 0x7f199415bb40 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f1999efa640 1 --2- 172.21.15.77:0/2902949164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415bb40 0x7f19940b4970 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f198c0097d0 tx=0x7f198c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b64d1f1a293f41e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f1998ef8640 1 -- 172.21.15.77:0/2902949164 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c035020 con 0x7f199415bb40 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f1998ef8640 1 -- 172.21.15.77:0/2902949164 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c031890 con 0x7f199415bb40 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f1998ef8640 1 -- 172.21.15.77:0/2902949164 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c031c60 con 0x7f199415bb40 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f199c15e640 1 -- 172.21.15.77:0/2902949164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415bb40 msgr2=0x7f19940b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f199c15e640 1 --2- 172.21.15.77:0/2902949164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415bb40 0x7f19940b4970 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f198c0097d0 tx=0x7f198c02b930 comp rx=0 tx=0).stop 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f199c15e640 1 -- 172.21.15.77:0/2902949164 shutdown_connections 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f199c15e640 1 --2- 172.21.15.77:0/2902949164 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415bb40 0x7f19940b4970 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.522+0000 7f199c15e640 1 -- 172.21.15.77:0/2902949164 >> 172.21.15.77:0/2902949164 conn(0x7f199406b190 msgr2=0x7f199406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.523+0000 7f199c15e640 1 -- 172.21.15.77:0/2902949164 shutdown_connections 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.523+0000 7f199c15e640 1 -- 172.21.15.77:0/2902949164 wait complete. 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.524+0000 7f199c15e640 1 Processor -- start 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.524+0000 7f199c15e640 1 -- start start 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.524+0000 7f199c15e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19940cfa90 0x7f1994151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.524+0000 7f199c15e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199415bb40 0x7f1994150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.524+0000 7f199c15e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415f7d0 0x7f199415df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.524+0000 7f199c15e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199406d790 con 0x7f199415bb40 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.524+0000 7f199c15e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199415fc30 con 0x7f19940cfa90 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.524+0000 7f199c15e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199415bf20 con 0x7f199415f7d0 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199415bb40 0x7f1994150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199415bb40 0x7f1994150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40562/0 (socket says 172.21.15.77:40562) 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 -- 172.21.15.77:0/1865758940 learned_addr learned my addr 172.21.15.77:0/1865758940 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f1999efa640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19940cfa90 0x7f1994151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f199a6fb640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415f7d0 0x7f199415df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 -- 172.21.15.77:0/1865758940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415f7d0 msgr2=0x7f199415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415f7d0 0x7f199415df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 -- 172.21.15.77:0/1865758940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19940cfa90 msgr2=0x7f1994151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19940cfa90 0x7f1994151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 -- 172.21.15.77:0/1865758940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1994152ad0 con 0x7f199415bb40 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f19996f9640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199415bb40 0x7f1994150000 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f198400c410 tx=0x7f198400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f198a7fc640 1 -- 172.21.15.77:0/1865758940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984017070 con 0x7f199415bb40 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f198a7fc640 1 -- 172.21.15.77:0/1865758940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1984005ce0 con 0x7f199415bb40 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1994151f50 con 0x7f199415bb40 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f198a7fc640 1 -- 172.21.15.77:0/1865758940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19840072f0 con 0x7f199415bb40 2024-03-06T07:28:33.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.525+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1994152510 con 0x7f199415bb40 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.526+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19940732b0 con 0x7f199415bb40 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.527+0000 7f198a7fc640 1 -- 172.21.15.77:0/1865758940 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f198401a070 con 0x7f199415bb40 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.527+0000 7f198a7fc640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f197806b4d0 0x7f197806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.527+0000 7f198a7fc640 1 -- 172.21.15.77:0/1865758940 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1984089580 con 0x7f199415bb40 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.527+0000 7f1999efa640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f197806b4d0 0x7f197806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.528+0000 7f1999efa640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f197806b4d0 0x7f197806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f198c036000 tx=0x7f198c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.531+0000 7f198a7fc640 1 -- 172.21.15.77:0/1865758940 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1984057f00 con 0x7f199415bb40 2024-03-06T07:28:33.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.739+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1994068ff0 con 0x7f199415bb40 2024-03-06T07:28:33.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.799+0000 7f198a7fc640 1 -- 172.21.15.77:0/1865758940 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v79) v1 ==== 118+0+0 (secure 0 0 0) 0x7f198405bfa0 con 0x7f199415bb40 2024-03-06T07:28:33.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f197806b4d0 msgr2=0x7f197806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f197806b4d0 0x7f197806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f198c036000 tx=0x7f198c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:33.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199415bb40 msgr2=0x7f1994150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199415bb40 0x7f1994150000 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f198400c410 tx=0x7f198400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 shutdown_connections 2024-03-06T07:28:33.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f197806b4d0 0x7f197806d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199415bb40 0x7f1994150000 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f199415f7d0 0x7f199415df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 --2- 172.21.15.77:0/1865758940 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f19940cfa90 0x7f1994151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 >> 172.21.15.77:0/1865758940 conn(0x7f199406b190 msgr2=0x7f19940b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 shutdown_connections 2024-03-06T07:28:33.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.802+0000 7f199c15e640 1 -- 172.21.15.77:0/1865758940 wait complete. 2024-03-06T07:28:33.816 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:33.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.918+0000 7fd36dcfb640 1 Processor -- start 2024-03-06T07:28:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd36dcfb640 1 -- start start 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd36dcfb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3680cfa90 0x7fd3680cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd36dcfb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd3680b33a0 0x7fd3680b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd36dcfb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd368158ac0 con 0x7fd3681586e0 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd36dcfb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36815b2a0 con 0x7fd3680cfa90 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd36dcfb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36815b540 con 0x7fd3680b33a0 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd35effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3680cfa90 0x7fd3680cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd35effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3680cfa90 0x7fd3680cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40576/0 (socket says 172.21.15.77:40576) 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.919+0000 7fd35effd640 1 -- 172.21.15.77:0/2061258729 learned_addr learned my addr 172.21.15.77:0/2061258729 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.920+0000 7fd367fff640 1 --2- 172.21.15.77:0/2061258729 >> v2:172.21.15.158:3300/0 conn(0x7fd3680b33a0 0x7fd3680b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.920+0000 7fd35effd640 1 -- 172.21.15.77:0/2061258729 >> v2:172.21.15.158:3300/0 conn(0x7fd3680b33a0 msgr2=0x7fd3680b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.920+0000 7fd35effd640 1 --2- 172.21.15.77:0/2061258729 >> v2:172.21.15.158:3300/0 conn(0x7fd3680b33a0 0x7fd3680b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.920+0000 7fd35effd640 1 -- 172.21.15.77:0/2061258729 >> v1:172.21.15.158:6789/0 conn(0x7fd3681586e0 legacy=0x7fd3680aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.920+0000 7fd35effd640 1 -- 172.21.15.77:0/2061258729 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36815b820 con 0x7fd3680cfa90 2024-03-06T07:28:33.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.920+0000 7fd35effd640 1 --2- 172.21.15.77:0/2061258729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3680cfa90 0x7fd3680cff40 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd3680b2ae0 tx=0x7fd35002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e0689c1468746 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd366ffd640 1 -- 172.21.15.77:0/2061258729 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350034070 con 0x7fd3680cfa90 2024-03-06T07:28:33.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd366ffd640 1 -- 172.21.15.77:0/2061258729 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd350030860 con 0x7fd3680cfa90 2024-03-06T07:28:33.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd366ffd640 1 -- 172.21.15.77:0/2061258729 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350030c30 con 0x7fd3680cfa90 2024-03-06T07:28:33.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd36dcfb640 1 -- 172.21.15.77:0/2061258729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3680cfa90 msgr2=0x7fd3680cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/2061258729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3680cfa90 0x7fd3680cff40 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd3680b2ae0 tx=0x7fd35002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd36dcfb640 1 -- 172.21.15.77:0/2061258729 shutdown_connections 2024-03-06T07:28:33.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/2061258729 >> v2:172.21.15.158:3300/0 conn(0x7fd3680b33a0 0x7fd3680b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/2061258729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3680cfa90 0x7fd3680cff40 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd36dcfb640 1 -- 172.21.15.77:0/2061258729 >> 172.21.15.77:0/2061258729 conn(0x7fd36806b190 msgr2=0x7fd36806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd36dcfb640 1 -- 172.21.15.77:0/2061258729 shutdown_connections 2024-03-06T07:28:33.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.921+0000 7fd36dcfb640 1 -- 172.21.15.77:0/2061258729 wait complete. 2024-03-06T07:28:33.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.922+0000 7fd36dcfb640 1 Processor -- start 2024-03-06T07:28:33.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd36dcfb640 1 -- start start 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd36dcfb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3680b33a0 0x7fd368139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd36dcfb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3680cfa90 0x7fd36814b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd36dcfb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3681586e0 0x7fd3680ced00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd36dcfb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680af150 con 0x7fd3681586e0 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd36dcfb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36815bde0 con 0x7fd3680cfa90 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd36dcfb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd368071350 con 0x7fd3680b33a0 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd367fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3681586e0 0x7fd3680ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd367fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3681586e0 0x7fd3680ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40592/0 (socket says 172.21.15.77:40592) 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd367fff640 1 -- 172.21.15.77:0/3311761321 learned_addr learned my addr 172.21.15.77:0/3311761321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:33.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd35effd640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3680cfa90 0x7fd36814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.923+0000 7fd3677fe640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3680b33a0 0x7fd368139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd367fff640 1 -- 172.21.15.77:0/3311761321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3680b33a0 msgr2=0x7fd368139650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd367fff640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3680b33a0 0x7fd368139650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd367fff640 1 -- 172.21.15.77:0/3311761321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3680cfa90 msgr2=0x7fd36814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd367fff640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3680cfa90 0x7fd36814b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd367fff640 1 -- 172.21.15.77:0/3311761321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36814d900 con 0x7fd3681586e0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd367fff640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3681586e0 0x7fd3680ced00 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fd36000efc0 tx=0x7fd36000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd364ff9640 1 -- 172.21.15.77:0/3311761321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360018020 con 0x7fd3681586e0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd364ff9640 1 -- 172.21.15.77:0/3311761321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360014480 con 0x7fd3681586e0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36814dbe0 con 0x7fd3681586e0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd364ff9640 1 -- 172.21.15.77:0/3311761321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360014870 con 0x7fd3681586e0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.924+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36814e1a0 con 0x7fd3681586e0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.925+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3680732b0 con 0x7fd3681586e0 2024-03-06T07:28:33.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.926+0000 7fd364ff9640 1 -- 172.21.15.77:0/3311761321 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd360005d10 con 0x7fd3681586e0 2024-03-06T07:28:33.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.926+0000 7fd364ff9640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd34806b4d0 0x7fd34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.926+0000 7fd364ff9640 1 -- 172.21.15.77:0/3311761321 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd36008abb0 con 0x7fd3681586e0 2024-03-06T07:28:33.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.926+0000 7fd3677fe640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd34806b4d0 0x7fd34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.926+0000 7fd3677fe640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd34806b4d0 0x7fd34806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd358001780 tx=0x7fd358009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:33.930+0000 7fd364ff9640 1 -- 172.21.15.77:0/3311761321 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd360022070 con 0x7fd3681586e0 2024-03-06T07:28:34.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.141+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd368068ff0 con 0x7fd3681586e0 2024-03-06T07:28:34.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.804+0000 7fd364ff9640 1 -- 172.21.15.77:0/3311761321 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v80) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd360010040 con 0x7fd3681586e0 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd34806b4d0 msgr2=0x7fd34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd34806b4d0 0x7fd34806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd358001780 tx=0x7fd358009040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3681586e0 msgr2=0x7fd3680ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3681586e0 0x7fd3680ced00 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fd36000efc0 tx=0x7fd36000c490 comp rx=0 tx=0).stop 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 shutdown_connections 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd34806b4d0 0x7fd34806d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3681586e0 0x7fd3680ced00 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3680cfa90 0x7fd36814b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 --2- 172.21.15.77:0/3311761321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3680b33a0 0x7fd368139650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 >> 172.21.15.77:0/3311761321 conn(0x7fd36806b190 msgr2=0x7fd3680b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 shutdown_connections 2024-03-06T07:28:34.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.807+0000 7fd36dcfb640 1 -- 172.21.15.77:0/3311761321 wait complete. 2024-03-06T07:28:34.820 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:34.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:34.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.918+0000 7f4a68e4e640 1 Processor -- start 2024-03-06T07:28:34.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.918+0000 7f4a68e4e640 1 -- start start 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.919+0000 7f4a68e4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a6415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.919+0000 7f4a68e4e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a64071f40 con 0x7f4a640cce40 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.919+0000 7f4a68e4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a64070a70 con 0x7f4a641515e0 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.919+0000 7f4a68e4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640cd2f0 con 0x7f4a640cdeb0 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.919+0000 7f4a61d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4a640cce40 0x7f4a640cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54028/0 (socket says 172.21.15.77:54028) 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.919+0000 7f4a61d9b640 1 -- 172.21.15.77:0/2785506269 learned_addr learned my addr 172.21.15.77:0/2785506269 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.920+0000 7f4a62d9d640 1 --2- 172.21.15.77:0/2785506269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a6415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.920+0000 7f4a6159a640 1 -- 172.21.15.77:0/2785506269 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491596990 0 0) 0x7f4a64071f40 con 0x7f4a640cce40 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.920+0000 7f4a6159a640 1 -- 172.21.15.77:0/2785506269 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a50003520 con 0x7f4a640cce40 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.920+0000 7f4a6159a640 1 -- 172.21.15.77:0/2785506269 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787642463 0 0) 0x7f4a64070a70 con 0x7f4a641515e0 2024-03-06T07:28:34.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.920+0000 7f4a6159a640 1 -- 172.21.15.77:0/2785506269 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a64071f40 con 0x7f4a641515e0 2024-03-06T07:28:34.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.920+0000 7f4a62d9d640 1 -- 172.21.15.77:0/2785506269 >> v1:172.21.15.158:6789/0 conn(0x7f4a641515e0 legacy=0x7f4a64071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.920+0000 7f4a62d9d640 1 -- 172.21.15.77:0/2785506269 >> v1:172.21.15.77:6789/0 conn(0x7f4a640cce40 legacy=0x7f4a640cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.920+0000 7f4a62d9d640 1 -- 172.21.15.77:0/2785506269 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64070d50 con 0x7f4a640cdeb0 2024-03-06T07:28:34.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a62d9d640 1 --2- 172.21.15.77:0/2785506269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a6415b430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4a54009dc0 tx=0x7f4a5402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=552da78b7bf02ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a6159a640 1 -- 172.21.15.77:0/2785506269 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54039070 con 0x7f4a640cdeb0 2024-03-06T07:28:34.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a6159a640 1 -- 172.21.15.77:0/2785506269 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5402ccd0 con 0x7f4a640cdeb0 2024-03-06T07:28:34.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a6159a640 1 -- 172.21.15.77:0/2785506269 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54034640 con 0x7f4a640cdeb0 2024-03-06T07:28:34.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a68e4e640 1 -- 172.21.15.77:0/2785506269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 msgr2=0x7f4a6415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a68e4e640 1 --2- 172.21.15.77:0/2785506269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a6415b430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4a54009dc0 tx=0x7f4a5402bf20 comp rx=0 tx=0).stop 2024-03-06T07:28:34.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a68e4e640 1 -- 172.21.15.77:0/2785506269 shutdown_connections 2024-03-06T07:28:34.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a68e4e640 1 --2- 172.21.15.77:0/2785506269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a6415b430 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.921+0000 7f4a68e4e640 1 -- 172.21.15.77:0/2785506269 >> 172.21.15.77:0/2785506269 conn(0x7f4a640cf8a0 msgr2=0x7f4a640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.922+0000 7f4a68e4e640 1 -- 172.21.15.77:0/2785506269 shutdown_connections 2024-03-06T07:28:34.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.922+0000 7f4a68e4e640 1 -- 172.21.15.77:0/2785506269 wait complete. 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.923+0000 7f4a68e4e640 1 Processor -- start 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.923+0000 7f4a68e4e640 1 -- start start 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a68e4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a640cce40 0x7f4a64145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a68e4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a64143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a68e4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a641515e0 0x7f4a6414ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a68e4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640cc340 con 0x7f4a640cce40 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a68e4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64071310 con 0x7f4a641515e0 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a68e4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640cc4c0 con 0x7f4a640cdeb0 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a640cce40 0x7f4a64145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a640cce40 0x7f4a64145ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40600/0 (socket says 172.21.15.77:40600) 2024-03-06T07:28:34.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 -- 172.21.15.77:0/3381841455 learned_addr learned my addr 172.21.15.77:0/3381841455 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a61d9b640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a64143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a62d9d640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a641515e0 0x7f4a6414ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 -- 172.21.15.77:0/3381841455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 msgr2=0x7f4a64143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a64143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 -- 172.21.15.77:0/3381841455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a641515e0 msgr2=0x7f4a6414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a641515e0 0x7f4a6414ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 -- 172.21.15.77:0/3381841455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64147d70 con 0x7f4a640cce40 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a6259c640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a640cce40 0x7f4a64145ee0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00c430 tx=0x7f4a5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a5affd640 1 -- 172.21.15.77:0/3381841455 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c017070 con 0x7f4a640cce40 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a5affd640 1 -- 172.21.15.77:0/3381841455 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5c0052f0 con 0x7f4a640cce40 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a641439e0 con 0x7f4a640cce40 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a5affd640 1 -- 172.21.15.77:0/3381841455 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c003830 con 0x7f4a640cce40 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.924+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a641482f0 con 0x7f4a640cce40 2024-03-06T07:28:34.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.925+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a64068840 con 0x7f4a640cce40 2024-03-06T07:28:34.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.926+0000 7f4a5affd640 1 -- 172.21.15.77:0/3381841455 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a5c01a070 con 0x7f4a640cce40 2024-03-06T07:28:34.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.926+0000 7f4a5affd640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a4406b500 0x7f4a4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.926+0000 7f4a5affd640 1 -- 172.21.15.77:0/3381841455 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a5c089430 con 0x7f4a640cce40 2024-03-06T07:28:34.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.926+0000 7f4a61d9b640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a4406b500 0x7f4a4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.927+0000 7f4a61d9b640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a4406b500 0x7f4a4406d9c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4a64150c30 tx=0x7f4a4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:34.931+0000 7f4a5affd640 1 -- 172.21.15.77:0/3381841455 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a5c057cf0 con 0x7f4a640cce40 2024-03-06T07:28:35.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.136+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a6404f330 con 0x7f4a640cce40 2024-03-06T07:28:35.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.137+0000 7f4a5affd640 1 -- 172.21.15.77:0/3381841455 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f4a5c05bd90 con 0x7f4a640cce40 2024-03-06T07:28:35.139 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:35.139 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":80,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6731,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/2348087487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":2348087487},{"type":"v1","addr":"172.21.15.158:6834","nonce":2348087487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/238243004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":238243004},{"type":"v1","addr":"172.21.15.77:6835","nonce":238243004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:34.142346+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:28:35.139 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 80 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a4406b500 msgr2=0x7f4a4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a4406b500 0x7f4a4406d9c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4a64150c30 tx=0x7f4a4c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a640cce40 msgr2=0x7f4a64145ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a640cce40 0x7f4a64145ee0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00c430 tx=0x7f4a5c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 shutdown_connections 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a641515e0 0x7f4a6414ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a640cdeb0 0x7f4a64143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a4406b500 0x7f4a4406d9c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 --2- 172.21.15.77:0/3381841455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a640cce40 0x7f4a64145ee0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 >> 172.21.15.77:0/3381841455 conn(0x7f4a640cf8a0 msgr2=0x7f4a6415de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 shutdown_connections 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.139+0000 7f4a68e4e640 1 -- 172.21.15.77:0/3381841455 wait complete. 2024-03-06T07:28:35.151 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:35.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.249+0000 7fc1ee77a640 1 Processor -- start 2024-03-06T07:28:35.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.249+0000 7fc1ee77a640 1 -- start start 2024-03-06T07:28:35.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1ee77a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc1e813dc10 0x7fc1e80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1ee77a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc1e815bf30 0x7fc1e80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1ee77a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e815f580 con 0x7fc1e815e850 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1ee77a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e80cf3f0 con 0x7fc1e815bf30 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1ee77a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e815b210 con 0x7fc1e813dc10 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1e77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc1e815bf30 0x7fc1e80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1ecd17640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc1e815e850 0x7fc1e815ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54038/0 (socket says 172.21.15.77:54038) 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1ecd17640 1 -- 172.21.15.77:0/263907306 learned_addr learned my addr 172.21.15.77:0/263907306 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1e7fff640 1 --2- 172.21.15.77:0/263907306 >> v2:172.21.15.158:3300/0 conn(0x7fc1e813dc10 0x7fc1e80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1e77fe640 1 -- 172.21.15.77:0/263907306 >> v2:172.21.15.158:3300/0 conn(0x7fc1e813dc10 msgr2=0x7fc1e80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1e77fe640 1 --2- 172.21.15.77:0/263907306 >> v2:172.21.15.158:3300/0 conn(0x7fc1e813dc10 0x7fc1e80b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1e77fe640 1 -- 172.21.15.77:0/263907306 >> v1:172.21.15.77:6789/0 conn(0x7fc1e815e850 legacy=0x7fc1e815ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1e77fe640 1 -- 172.21.15.77:0/263907306 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e815b450 con 0x7fc1e815bf30 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1e6ffd640 1 -- 172.21.15.77:0/263907306 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392350710 0 0) 0x7fc1e815f580 con 0x7fc1e815e850 2024-03-06T07:28:35.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.250+0000 7fc1e7fff640 1 --2- 172.21.15.77:0/263907306 >> v2:172.21.15.158:3300/0 conn(0x7fc1e813dc10 0x7fc1e80b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:35.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.251+0000 7fc1e77fe640 1 --2- 172.21.15.77:0/263907306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815bf30 0x7fc1e80b2e90 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80731a0 tx=0x7fc1dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf4c16a93bc77eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.251+0000 7fc1e6ffd640 1 -- 172.21.15.77:0/263907306 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc0315a0 con 0x7fc1e815bf30 2024-03-06T07:28:35.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.251+0000 7fc1e6ffd640 1 -- 172.21.15.77:0/263907306 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc037070 con 0x7fc1e815bf30 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.251+0000 7fc1e6ffd640 1 -- 172.21.15.77:0/263907306 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc031960 con 0x7fc1e815bf30 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.252+0000 7fc1ee77a640 1 -- 172.21.15.77:0/263907306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815bf30 msgr2=0x7fc1e80b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.252+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/263907306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815bf30 0x7fc1e80b2e90 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80731a0 tx=0x7fc1dc02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.252+0000 7fc1ee77a640 1 -- 172.21.15.77:0/263907306 shutdown_connections 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.252+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/263907306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815bf30 0x7fc1e80b2e90 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.252+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/263907306 >> v2:172.21.15.158:3300/0 conn(0x7fc1e813dc10 0x7fc1e80b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.252+0000 7fc1ee77a640 1 -- 172.21.15.77:0/263907306 >> 172.21.15.77:0/263907306 conn(0x7fc1e80cf8a0 msgr2=0x7fc1e80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.252+0000 7fc1ee77a640 1 -- 172.21.15.77:0/263907306 shutdown_connections 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.252+0000 7fc1ee77a640 1 -- 172.21.15.77:0/263907306 wait complete. 2024-03-06T07:28:35.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.253+0000 7fc1ee77a640 1 Processor -- start 2024-03-06T07:28:35.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.253+0000 7fc1ee77a640 1 -- start start 2024-03-06T07:28:35.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ee77a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1e813dc10 0x7fc1e81edb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ee77a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1e815bf30 0x7fc1e81f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ee77a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815e850 0x7fc1e806dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ee77a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80708d0 con 0x7fc1e815e850 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ee77a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e813dff0 con 0x7fc1e813dc10 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ee77a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e8070a50 con 0x7fc1e815bf30 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815e850 0x7fc1e806dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815e850 0x7fc1e806dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40610/0 (socket says 172.21.15.77:40610) 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 -- 172.21.15.77:0/481353959 learned_addr learned my addr 172.21.15.77:0/481353959 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1e7fff640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1e813dc10 0x7fc1e81edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1e77fe640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1e815bf30 0x7fc1e81f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 -- 172.21.15.77:0/481353959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1e815bf30 msgr2=0x7fc1e81f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1e815bf30 0x7fc1e81f4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 -- 172.21.15.77:0/481353959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1e813dc10 msgr2=0x7fc1e81edb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1e813dc10 0x7fc1e81edb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 -- 172.21.15.77:0/481353959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e81f4960 con 0x7fc1e815e850 2024-03-06T07:28:35.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.254+0000 7fc1ecd17640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815e850 0x7fc1e806dac0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000c780 tx=0x7fc1e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.255+0000 7fc1e4ff9640 1 -- 172.21.15.77:0/481353959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0018070 con 0x7fc1e815e850 2024-03-06T07:28:35.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.255+0000 7fc1e4ff9640 1 -- 172.21.15.77:0/481353959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e000a640 con 0x7fc1e815e850 2024-03-06T07:28:35.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.255+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e81f4c40 con 0x7fc1e815e850 2024-03-06T07:28:35.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.255+0000 7fc1e4ff9640 1 -- 172.21.15.77:0/481353959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000aa30 con 0x7fc1e815e850 2024-03-06T07:28:35.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.255+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e813e4f0 con 0x7fc1e815e850 2024-03-06T07:28:35.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.256+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e804f330 con 0x7fc1e815e850 2024-03-06T07:28:35.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.256+0000 7fc1e4ff9640 1 -- 172.21.15.77:0/481353959 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1e001b070 con 0x7fc1e815e850 2024-03-06T07:28:35.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.256+0000 7fc1e4ff9640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1d406b4d0 0x7fc1d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.257+0000 7fc1e4ff9640 1 -- 172.21.15.77:0/481353959 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1e0089750 con 0x7fc1e815e850 2024-03-06T07:28:35.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.257+0000 7fc1e7fff640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1d406b4d0 0x7fc1d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.257+0000 7fc1e7fff640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1d406b4d0 0x7fc1d406d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fc1e81e5180 tx=0x7fc1d8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.261+0000 7fc1e4ff9640 1 -- 172.21.15.77:0/481353959 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1e0057f80 con 0x7fc1e815e850 2024-03-06T07:28:35.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.432+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1e80b3db0 con 0x7fc1e815e850 2024-03-06T07:28:35.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.433+0000 7fc1e4ff9640 1 -- 172.21.15.77:0/481353959 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16667 (secure 0 0 0) 0x7fc1e005c020 con 0x7fc1e815e850 2024-03-06T07:28:35.435 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":125,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:33.787595+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:30.343353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:31.094099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1d406b4d0 msgr2=0x7fc1d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1d406b4d0 0x7fc1d406d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fc1e81e5180 tx=0x7fc1d8009190 comp rx=0 tx=0).stop 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815e850 msgr2=0x7fc1e806dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815e850 0x7fc1e806dac0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000c780 tx=0x7fc1e000cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 shutdown_connections 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc1d406b4d0 0x7fc1d406d990 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1e815e850 0x7fc1e806dac0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc1e815bf30 0x7fc1e81f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 --2- 172.21.15.77:0/481353959 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc1e813dc10 0x7fc1e81edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.435+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 >> 172.21.15.77:0/481353959 conn(0x7fc1e80cf8a0 msgr2=0x7fc1e806c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.436+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 shutdown_connections 2024-03-06T07:28:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.436+0000 7fc1ee77a640 1 -- 172.21.15.77:0/481353959 wait complete. 2024-03-06T07:28:35.449 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:35.449 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:35.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.546+0000 7fbf58b7a640 1 Processor -- start 2024-03-06T07:28:35.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.546+0000 7fbf58b7a640 1 -- start start 2024-03-06T07:28:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf58b7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5413dc10 0x7fbf540b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf58b7a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf5415ef60 0x7fbf540b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf58b7a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5415f580 con 0x7fbf5415bf30 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf58b7a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf540cf3f0 con 0x7fbf5415ef60 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf58b7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5415b210 con 0x7fbf5413dc10 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5413dc10 0x7fbf540b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf52d9d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf5415ef60 0x7fbf540b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5413dc10 0x7fbf540b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:36468/0 (socket says 172.21.15.77:36468) 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf52d9d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbf5415ef60 0x7fbf540b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:46954/0 (socket says 172.21.15.77:46954) 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf51d9b640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fbf5415bf30 0x7fbf5415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49864/0 (socket says 172.21.15.77:49864) 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.547+0000 7fbf52d9d640 1 -- 172.21.15.77:0/2537070822 learned_addr learned my addr 172.21.15.77:0/2537070822 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.548+0000 7fbf52d9d640 1 -- 172.21.15.77:0/2537070822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5413dc10 msgr2=0x7fbf540b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.548+0000 7fbf52d9d640 1 --2- 172.21.15.77:0/2537070822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5413dc10 0x7fbf540b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.548+0000 7fbf52d9d640 1 -- 172.21.15.77:0/2537070822 >> v1:172.21.15.158:6789/0 conn(0x7fbf5415bf30 legacy=0x7fbf5415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.548+0000 7fbf5159a640 1 -- 172.21.15.77:0/2537070822 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844567419 0 0) 0x7fbf5415f580 con 0x7fbf5415bf30 2024-03-06T07:28:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.548+0000 7fbf52d9d640 1 -- 172.21.15.77:0/2537070822 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5415b450 con 0x7fbf5415ef60 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.548+0000 7fbf52d9d640 1 --2- 172.21.15.77:0/2537070822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5415ef60 0x7fbf540b2e90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009870 tx=0x7fbf4c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b9065373b89bb01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf5159a640 1 -- 172.21.15.77:0/2537070822 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c035020 con 0x7fbf5415ef60 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf5159a640 1 -- 172.21.15.77:0/2537070822 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c031640 con 0x7fbf5415ef60 2024-03-06T07:28:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf5159a640 1 -- 172.21.15.77:0/2537070822 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c02d040 con 0x7fbf5415ef60 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf58b7a640 1 -- 172.21.15.77:0/2537070822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5415ef60 msgr2=0x7fbf540b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/2537070822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5415ef60 0x7fbf540b2e90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009870 tx=0x7fbf4c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf58b7a640 1 -- 172.21.15.77:0/2537070822 shutdown_connections 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/2537070822 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5415ef60 0x7fbf540b2e90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/2537070822 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5413dc10 0x7fbf540b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf58b7a640 1 -- 172.21.15.77:0/2537070822 >> 172.21.15.77:0/2537070822 conn(0x7fbf540cf8a0 msgr2=0x7fbf540cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf58b7a640 1 -- 172.21.15.77:0/2537070822 shutdown_connections 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.549+0000 7fbf58b7a640 1 -- 172.21.15.77:0/2537070822 wait complete. 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.551+0000 7fbf58b7a640 1 Processor -- start 2024-03-06T07:28:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.551+0000 7fbf58b7a640 1 -- start start 2024-03-06T07:28:35.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.551+0000 7fbf58b7a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5413dc10 0x7fbf541f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.551+0000 7fbf58b7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf5415bf30 0x7fbf54159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.551+0000 7fbf58b7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5415ef60 0x7fbf5415a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.551+0000 7fbf58b7a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540cd7a0 con 0x7fbf5415bf30 2024-03-06T07:28:35.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.551+0000 7fbf58b7a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5413dff0 con 0x7fbf5413dc10 2024-03-06T07:28:35.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.551+0000 7fbf58b7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540cd920 con 0x7fbf5415ef60 2024-03-06T07:28:35.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf5415bf30 0x7fbf54159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5413dc10 0x7fbf541f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf5415bf30 0x7fbf54159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40612/0 (socket says 172.21.15.77:40612) 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 -- 172.21.15.77:0/1287825627 learned_addr learned my addr 172.21.15.77:0/1287825627 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf52d9d640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5415ef60 0x7fbf5415a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 -- 172.21.15.77:0/1287825627 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5415ef60 msgr2=0x7fbf5415a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5415ef60 0x7fbf5415a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 -- 172.21.15.77:0/1287825627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5413dc10 msgr2=0x7fbf541f6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5413dc10 0x7fbf541f6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 -- 172.21.15.77:0/1287825627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf54135190 con 0x7fbf5415bf30 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf5259c640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5413dc10 0x7fbf541f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf51d9b640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf5415bf30 0x7fbf54159d30 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c4d0 tx=0x7fbf4800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf42ffd640 1 -- 172.21.15.77:0/1287825627 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48018070 con 0x7fbf5415bf30 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf42ffd640 1 -- 172.21.15.77:0/1287825627 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf48007500 con 0x7fbf5415bf30 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf54135470 con 0x7fbf5415bf30 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf42ffd640 1 -- 172.21.15.77:0/1287825627 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48004a20 con 0x7fbf5415bf30 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.552+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf54135a30 con 0x7fbf5415bf30 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.553+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5404f330 con 0x7fbf5415bf30 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.554+0000 7fbf42ffd640 1 -- 172.21.15.77:0/1287825627 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf48004bc0 con 0x7fbf5415bf30 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.554+0000 7fbf42ffd640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf3406b500 0x7fbf3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.554+0000 7fbf42ffd640 1 -- 172.21.15.77:0/1287825627 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf48089470 con 0x7fbf5415bf30 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.554+0000 7fbf5259c640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf3406b500 0x7fbf3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.555+0000 7fbf5259c640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf3406b500 0x7fbf3406d9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400e740 tx=0x7fbf4400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.558+0000 7fbf42ffd640 1 -- 172.21.15.77:0/1287825627 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf48057c20 con 0x7fbf5415bf30 2024-03-06T07:28:35.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.727+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbf5406eb00 con 0x7fbf3406b500 2024-03-06T07:28:35.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.733+0000 7fbf42ffd640 1 -- 172.21.15.77:0/1287825627 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247089 (secure 0 0 0) 0x7fbf5406eb00 con 0x7fbf3406b500 2024-03-06T07:28:35.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:35.740 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:28:35.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf3406b500 msgr2=0x7fbf3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf3406b500 0x7fbf3406d9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400e740 tx=0x7fbf4400b040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf5415bf30 msgr2=0x7fbf54159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf5415bf30 0x7fbf54159d30 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c4d0 tx=0x7fbf4800c990 comp rx=0 tx=0).stop 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 shutdown_connections 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbf3406b500 0x7fbf3406d9c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf5415bf30 0x7fbf54159d30 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbf5415ef60 0x7fbf5415a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 --2- 172.21.15.77:0/1287825627 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbf5413dc10 0x7fbf541f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 >> 172.21.15.77:0/1287825627 conn(0x7fbf540cf8a0 msgr2=0x7fbf540b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 shutdown_connections 2024-03-06T07:28:35.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.736+0000 7fbf58b7a640 1 -- 172.21.15.77:0/1287825627 wait complete. 2024-03-06T07:28:35.749 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-03-06T07:28:35.455923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304196,"kb_used_data":13136,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754670524,"statfs":{"total":773094113280,"available":772782616576,"internally_reserved":0,"allocated":13451264,"data_stored":6501970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.693890"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770447+0000","last_change":"2024-03-06T07:28:31.770447+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770447+0000","last_undegraded":"2024-03-06T07:28:31.770447+0000","last_fullsized":"2024-03-06T07:28:31.770447+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775687+0000","last_change":"2024-03-06T07:28:31.775687+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775687+0000","last_undegraded":"2024-03-06T07:28:31.775687+0000","last_fullsized":"2024-03-06T07:28:31.775687+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768247+0000","last_change":"2024-03-06T07:28:31.768247+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768247+0000","last_undegraded":"2024-03-06T07:28:31.768247+0000","last_fullsized":"2024-03-06T07:28:31.768247+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770523+0000","last_change":"2024-03-06T07:28:31.770523+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770523+0000","last_undegraded":"2024-03-06T07:28:31.770523+0000","last_fullsized":"2024-03-06T07:28:31.770523+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772284+0000","last_change":"2024-03-06T07:28:31.772284+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772284+0000","last_undegraded":"2024-03-06T07:28:31.772284+0000","last_fullsized":"2024-03-06T07:28:31.772284+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767124+0000","last_change":"2024-03-06T07:28:31.767124+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767124+0000","last_undegraded":"2024-03-06T07:28:31.767124+0000","last_fullsized":"2024-03-06T07:28:31.767124+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770134+0000","last_change":"2024-03-06T07:28:31.770134+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770134+0000","last_undegraded":"2024-03-06T07:28:31.770134+0000","last_fullsized":"2024-03-06T07:28:31.770134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768838+0000","last_change":"2024-03-06T07:28:31.768838+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768838+0000","last_undegraded":"2024-03-06T07:28:31.768838+0000","last_fullsized":"2024-03-06T07:28:31.768838+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767826+0000","last_change":"2024-03-06T07:28:31.767826+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767826+0000","last_undegraded":"2024-03-06T07:28:31.767826+0000","last_fullsized":"2024-03-06T07:28:31.767826+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775301+0000","last_change":"2024-03-06T07:28:31.775301+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775301+0000","last_undegraded":"2024-03-06T07:28:31.775301+0000","last_fullsized":"2024-03-06T07:28:31.775301+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769962+0000","last_change":"2024-03-06T07:28:31.769962+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769962+0000","last_undegraded":"2024-03-06T07:28:31.769962+0000","last_fullsized":"2024-03-06T07:28:31.769962+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768517+0000","last_change":"2024-03-06T07:28:31.768517+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768517+0000","last_undegraded":"2024-03-06T07:28:31.768517+0000","last_fullsized":"2024-03-06T07:28:31.768517+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772112+0000","last_change":"2024-03-06T07:28:31.772112+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772112+0000","last_undegraded":"2024-03-06T07:28:31.772112+0000","last_fullsized":"2024-03-06T07:28:31.772112+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775114+0000","last_change":"2024-03-06T07:28:31.775114+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775114+0000","last_undegraded":"2024-03-06T07:28:31.775114+0000","last_fullsized":"2024-03-06T07:28:31.775114+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769966+0000","last_change":"2024-03-06T07:28:31.769966+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769966+0000","last_undegraded":"2024-03-06T07:28:31.769966+0000","last_fullsized":"2024-03-06T07:28:31.769966+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773730+0000","last_change":"2024-03-06T07:28:31.773730+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773730+0000","last_undegraded":"2024-03-06T07:28:31.773730+0000","last_fullsized":"2024-03-06T07:28:31.773730+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767225+0000","last_change":"2024-03-06T07:28:31.767225+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767225+0000","last_undegraded":"2024-03-06T07:28:31.767225+0000","last_fullsized":"2024-03-06T07:28:31.767225+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.777806+0000","last_change":"2024-03-06T07:28:31.777806+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.777806+0000","last_undegraded":"2024-03-06T07:28:31.777806+0000","last_fullsized":"2024-03-06T07:28:31.777806+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769133+0000","last_change":"2024-03-06T07:28:31.769133+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769133+0000","last_undegraded":"2024-03-06T07:28:31.769133+0000","last_fullsized":"2024-03-06T07:28:31.769133+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774231+0000","last_change":"2024-03-06T07:28:31.774231+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774231+0000","last_undegraded":"2024-03-06T07:28:31.774231+0000","last_fullsized":"2024-03-06T07:28:31.774231+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774487+0000","last_change":"2024-03-06T07:28:31.774487+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774487+0000","last_undegraded":"2024-03-06T07:28:31.774487+0000","last_fullsized":"2024-03-06T07:28:31.774487+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769129+0000","last_change":"2024-03-06T07:28:31.769129+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769129+0000","last_undegraded":"2024-03-06T07:28:31.769129+0000","last_fullsized":"2024-03-06T07:28:31.769129+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768187+0000","last_change":"2024-03-06T07:28:31.768187+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768187+0000","last_undegraded":"2024-03-06T07:28:31.768187+0000","last_fullsized":"2024-03-06T07:28:31.768187+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768712+0000","last_change":"2024-03-06T07:28:31.768712+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768712+0000","last_undegraded":"2024-03-06T07:28:31.768712+0000","last_fullsized":"2024-03-06T07:28:31.768712+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776050+0000","last_change":"2024-03-06T07:28:31.776050+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776050+0000","last_undegraded":"2024-03-06T07:28:31.776050+0000","last_fullsized":"2024-03-06T07:28:31.776050+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.777368+0000","last_change":"2024-03-06T07:28:31.777368+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.777368+0000","last_undegraded":"2024-03-06T07:28:31.777368+0000","last_fullsized":"2024-03-06T07:28:31.777368+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770197+0000","last_change":"2024-03-06T07:28:31.770197+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770197+0000","last_undegraded":"2024-03-06T07:28:31.770197+0000","last_fullsized":"2024-03-06T07:28:31.770197+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771951+0000","last_change":"2024-03-06T07:28:31.771951+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771951+0000","last_undegraded":"2024-03-06T07:28:31.771951+0000","last_fullsized":"2024-03-06T07:28:31.771951+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771671+0000","last_change":"2024-03-06T07:28:31.771671+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771671+0000","last_undegraded":"2024-03-06T07:28:31.771671+0000","last_fullsized":"2024-03-06T07:28:31.771671+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770827+0000","last_change":"2024-03-06T07:28:31.770827+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770827+0000","last_undegraded":"2024-03-06T07:28:31.770827+0000","last_fullsized":"2024-03-06T07:28:31.770827+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771788+0000","last_change":"2024-03-06T07:28:31.771788+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771788+0000","last_undegraded":"2024-03-06T07:28:31.771788+0000","last_fullsized":"2024-03-06T07:28:31.771788+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768199+0000","last_change":"2024-03-06T07:28:31.768199+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768199+0000","last_undegraded":"2024-03-06T07:28:31.768199+0000","last_fullsized":"2024-03-06T07:28:31.768199+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776999+0000","last_change":"2024-03-06T07:28:31.776999+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776999+0000","last_undegraded":"2024-03-06T07:28:31.776999+0000","last_fullsized":"2024-03-06T07:28:31.776999+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783927+0000","last_change":"2024-03-06T07:28:31.782852+0000","last_active":"2024-03-06T07:28:31.783927+0000","last_peered":"2024-03-06T07:28:31.783927+0000","last_clean":"2024-03-06T07:28:31.783927+0000","last_became_active":"2024-03-06T07:28:31.782575+0000","last_became_peered":"2024-03-06T07:28:31.782575+0000","last_unstale":"2024-03-06T07:28:31.783927+0000","last_undegraded":"2024-03-06T07:28:31.783927+0000","last_fullsized":"2024-03-06T07:28:31.783927+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:30.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767260+0000","last_change":"2024-03-06T07:28:31.767260+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767260+0000","last_undegraded":"2024-03-06T07:28:31.767260+0000","last_fullsized":"2024-03-06T07:28:31.767260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783865+0000","last_change":"2024-03-06T07:28:31.782726+0000","last_active":"2024-03-06T07:28:31.783865+0000","last_peered":"2024-03-06T07:28:31.783865+0000","last_clean":"2024-03-06T07:28:31.783865+0000","last_became_active":"2024-03-06T07:28:31.782447+0000","last_became_peered":"2024-03-06T07:28:31.782447+0000","last_unstale":"2024-03-06T07:28:31.783865+0000","last_undegraded":"2024-03-06T07:28:31.783865+0000","last_fullsized":"2024-03-06T07:28:31.783865+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:40.079443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774697+0000","last_change":"2024-03-06T07:28:31.774697+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774697+0000","last_undegraded":"2024-03-06T07:28:31.774697+0000","last_fullsized":"2024-03-06T07:28:31.774697+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.785061+0000","last_change":"2024-03-06T07:28:31.783586+0000","last_active":"2024-03-06T07:28:31.785061+0000","last_peered":"2024-03-06T07:28:31.785061+0000","last_clean":"2024-03-06T07:28:31.785061+0000","last_became_active":"2024-03-06T07:28:31.783329+0000","last_became_peered":"2024-03-06T07:28:31.783329+0000","last_unstale":"2024-03-06T07:28:31.785061+0000","last_undegraded":"2024-03-06T07:28:31.785061+0000","last_fullsized":"2024-03-06T07:28:31.785061+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:07.918185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773437+0000","last_change":"2024-03-06T07:28:31.773437+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773437+0000","last_undegraded":"2024-03-06T07:28:31.773437+0000","last_fullsized":"2024-03-06T07:28:31.773437+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782314+0000","last_change":"2024-03-06T07:28:31.780002+0000","last_active":"2024-03-06T07:28:31.782314+0000","last_peered":"2024-03-06T07:28:31.782314+0000","last_clean":"2024-03-06T07:28:31.782314+0000","last_became_active":"2024-03-06T07:28:31.779736+0000","last_became_peered":"2024-03-06T07:28:31.779736+0000","last_unstale":"2024-03-06T07:28:31.782314+0000","last_undegraded":"2024-03-06T07:28:31.782314+0000","last_fullsized":"2024-03-06T07:28:31.782314+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:37.028128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774574+0000","last_change":"2024-03-06T07:28:31.774574+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774574+0000","last_undegraded":"2024-03-06T07:28:31.774574+0000","last_fullsized":"2024-03-06T07:28:31.774574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784714+0000","last_change":"2024-03-06T07:28:31.783109+0000","last_active":"2024-03-06T07:28:31.784714+0000","last_peered":"2024-03-06T07:28:31.784714+0000","last_clean":"2024-03-06T07:28:31.784714+0000","last_became_active":"2024-03-06T07:28:31.782858+0000","last_became_peered":"2024-03-06T07:28:31.782858+0000","last_unstale":"2024-03-06T07:28:31.784714+0000","last_undegraded":"2024-03-06T07:28:31.784714+0000","last_fullsized":"2024-03-06T07:28:31.784714+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:41.710642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768260+0000","last_change":"2024-03-06T07:28:31.768260+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768260+0000","last_undegraded":"2024-03-06T07:28:31.768260+0000","last_fullsized":"2024-03-06T07:28:31.768260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783299+0000","last_change":"2024-03-06T07:28:31.781824+0000","last_active":"2024-03-06T07:28:31.783299+0000","last_peered":"2024-03-06T07:28:31.783299+0000","last_clean":"2024-03-06T07:28:31.783299+0000","last_became_active":"2024-03-06T07:28:31.781484+0000","last_became_peered":"2024-03-06T07:28:31.781484+0000","last_unstale":"2024-03-06T07:28:31.783299+0000","last_undegraded":"2024-03-06T07:28:31.783299+0000","last_fullsized":"2024-03-06T07:28:31.783299+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:59.115102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768166+0000","last_change":"2024-03-06T07:28:31.768166+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768166+0000","last_undegraded":"2024-03-06T07:28:31.768166+0000","last_fullsized":"2024-03-06T07:28:31.768166+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779805+0000","last_change":"2024-03-06T07:28:31.777579+0000","last_active":"2024-03-06T07:28:31.779805+0000","last_peered":"2024-03-06T07:28:31.779805+0000","last_clean":"2024-03-06T07:28:31.779805+0000","last_became_active":"2024-03-06T07:28:31.777290+0000","last_became_peered":"2024-03-06T07:28:31.777290+0000","last_unstale":"2024-03-06T07:28:31.779805+0000","last_undegraded":"2024-03-06T07:28:31.779805+0000","last_fullsized":"2024-03-06T07:28:31.779805+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:48.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775054+0000","last_change":"2024-03-06T07:28:31.775054+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775054+0000","last_undegraded":"2024-03-06T07:28:31.775054+0000","last_fullsized":"2024-03-06T07:28:31.775054+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780903+0000","last_change":"2024-03-06T07:28:31.775485+0000","last_active":"2024-03-06T07:28:31.780903+0000","last_peered":"2024-03-06T07:28:31.780903+0000","last_clean":"2024-03-06T07:28:31.780903+0000","last_became_active":"2024-03-06T07:28:31.775212+0000","last_became_peered":"2024-03-06T07:28:31.775212+0000","last_unstale":"2024-03-06T07:28:31.780903+0000","last_undegraded":"2024-03-06T07:28:31.780903+0000","last_fullsized":"2024-03-06T07:28:31.780903+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:35.868436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770026+0000","last_change":"2024-03-06T07:28:31.770026+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770026+0000","last_undegraded":"2024-03-06T07:28:31.770026+0000","last_fullsized":"2024-03-06T07:28:31.770026+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784013+0000","last_change":"2024-03-06T07:28:31.783404+0000","last_active":"2024-03-06T07:28:31.784013+0000","last_peered":"2024-03-06T07:28:31.784013+0000","last_clean":"2024-03-06T07:28:31.784013+0000","last_became_active":"2024-03-06T07:28:31.782969+0000","last_became_peered":"2024-03-06T07:28:31.782969+0000","last_unstale":"2024-03-06T07:28:31.784013+0000","last_undegraded":"2024-03-06T07:28:31.784013+0000","last_fullsized":"2024-03-06T07:28:31.784013+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:06.016386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776092+0000","last_change":"2024-03-06T07:28:31.776092+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776092+0000","last_undegraded":"2024-03-06T07:28:31.776092+0000","last_fullsized":"2024-03-06T07:28:31.776092+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784990+0000","last_change":"2024-03-06T07:28:31.784102+0000","last_active":"2024-03-06T07:28:31.784990+0000","last_peered":"2024-03-06T07:28:31.784990+0000","last_clean":"2024-03-06T07:28:31.784990+0000","last_became_active":"2024-03-06T07:28:31.783888+0000","last_became_peered":"2024-03-06T07:28:31.783888+0000","last_unstale":"2024-03-06T07:28:31.784990+0000","last_undegraded":"2024-03-06T07:28:31.784990+0000","last_fullsized":"2024-03-06T07:28:31.784990+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:56.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773622+0000","last_change":"2024-03-06T07:28:31.773622+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773622+0000","last_undegraded":"2024-03-06T07:28:31.773622+0000","last_fullsized":"2024-03-06T07:28:31.773622+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783836+0000","last_change":"2024-03-06T07:28:31.782822+0000","last_active":"2024-03-06T07:28:31.783836+0000","last_peered":"2024-03-06T07:28:31.783836+0000","last_clean":"2024-03-06T07:28:31.783836+0000","last_became_active":"2024-03-06T07:28:31.782523+0000","last_became_peered":"2024-03-06T07:28:31.782523+0000","last_unstale":"2024-03-06T07:28:31.783836+0000","last_undegraded":"2024-03-06T07:28:31.783836+0000","last_fullsized":"2024-03-06T07:28:31.783836+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:51.472938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773035+0000","last_change":"2024-03-06T07:28:31.773035+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773035+0000","last_undegraded":"2024-03-06T07:28:31.773035+0000","last_fullsized":"2024-03-06T07:28:31.773035+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779098+0000","last_change":"2024-03-06T07:28:31.776785+0000","last_active":"2024-03-06T07:28:31.779098+0000","last_peered":"2024-03-06T07:28:31.779098+0000","last_clean":"2024-03-06T07:28:31.779098+0000","last_became_active":"2024-03-06T07:28:31.776521+0000","last_became_peered":"2024-03-06T07:28:31.776521+0000","last_unstale":"2024-03-06T07:28:31.779098+0000","last_undegraded":"2024-03-06T07:28:31.779098+0000","last_fullsized":"2024-03-06T07:28:31.779098+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:58.794188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771283+0000","last_change":"2024-03-06T07:28:31.771283+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771283+0000","last_undegraded":"2024-03-06T07:28:31.771283+0000","last_fullsized":"2024-03-06T07:28:31.771283+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778422+0000","last_change":"2024-03-06T07:28:31.775867+0000","last_active":"2024-03-06T07:28:31.778422+0000","last_peered":"2024-03-06T07:28:31.778422+0000","last_clean":"2024-03-06T07:28:31.778422+0000","last_became_active":"2024-03-06T07:28:31.775622+0000","last_became_peered":"2024-03-06T07:28:31.775622+0000","last_unstale":"2024-03-06T07:28:31.778422+0000","last_undegraded":"2024-03-06T07:28:31.778422+0000","last_fullsized":"2024-03-06T07:28:31.778422+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:25.853619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772244+0000","last_change":"2024-03-06T07:28:31.772244+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772244+0000","last_undegraded":"2024-03-06T07:28:31.772244+0000","last_fullsized":"2024-03-06T07:28:31.772244+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780993+0000","last_change":"2024-03-06T07:28:31.777064+0000","last_active":"2024-03-06T07:28:31.780993+0000","last_peered":"2024-03-06T07:28:31.780993+0000","last_clean":"2024-03-06T07:28:31.780993+0000","last_became_active":"2024-03-06T07:28:31.776788+0000","last_became_peered":"2024-03-06T07:28:31.776788+0000","last_unstale":"2024-03-06T07:28:31.780993+0000","last_undegraded":"2024-03-06T07:28:31.780993+0000","last_fullsized":"2024-03-06T07:28:31.780993+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:41.214845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769329+0000","last_change":"2024-03-06T07:28:31.769329+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769329+0000","last_undegraded":"2024-03-06T07:28:31.769329+0000","last_fullsized":"2024-03-06T07:28:31.769329+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782969+0000","last_change":"2024-03-06T07:28:31.781673+0000","last_active":"2024-03-06T07:28:31.782969+0000","last_peered":"2024-03-06T07:28:31.782969+0000","last_clean":"2024-03-06T07:28:31.782969+0000","last_became_active":"2024-03-06T07:28:31.781383+0000","last_became_peered":"2024-03-06T07:28:31.781383+0000","last_unstale":"2024-03-06T07:28:31.782969+0000","last_undegraded":"2024-03-06T07:28:31.782969+0000","last_fullsized":"2024-03-06T07:28:31.782969+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:29.323269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768865+0000","last_change":"2024-03-06T07:28:31.768865+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768865+0000","last_undegraded":"2024-03-06T07:28:31.768865+0000","last_fullsized":"2024-03-06T07:28:31.768865+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782261+0000","last_change":"2024-03-06T07:28:31.780402+0000","last_active":"2024-03-06T07:28:31.782261+0000","last_peered":"2024-03-06T07:28:31.782261+0000","last_clean":"2024-03-06T07:28:31.782261+0000","last_became_active":"2024-03-06T07:28:31.780050+0000","last_became_peered":"2024-03-06T07:28:31.780050+0000","last_unstale":"2024-03-06T07:28:31.782261+0000","last_undegraded":"2024-03-06T07:28:31.782261+0000","last_fullsized":"2024-03-06T07:28:31.782261+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:37.538186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775650+0000","last_change":"2024-03-06T07:28:31.775650+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775650+0000","last_undegraded":"2024-03-06T07:28:31.775650+0000","last_fullsized":"2024-03-06T07:28:31.775650+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779815+0000","last_change":"2024-03-06T07:28:31.773233+0000","last_active":"2024-03-06T07:28:31.779815+0000","last_peered":"2024-03-06T07:28:31.779815+0000","last_clean":"2024-03-06T07:28:31.779815+0000","last_became_active":"2024-03-06T07:28:31.773028+0000","last_became_peered":"2024-03-06T07:28:31.773028+0000","last_unstale":"2024-03-06T07:28:31.779815+0000","last_undegraded":"2024-03-06T07:28:31.779815+0000","last_fullsized":"2024-03-06T07:28:31.779815+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:33.073077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773859+0000","last_change":"2024-03-06T07:28:31.773859+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773859+0000","last_undegraded":"2024-03-06T07:28:31.773859+0000","last_fullsized":"2024-03-06T07:28:31.773859+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783149+0000","last_change":"2024-03-06T07:28:31.782644+0000","last_active":"2024-03-06T07:28:31.783149+0000","last_peered":"2024-03-06T07:28:31.783149+0000","last_clean":"2024-03-06T07:28:31.783149+0000","last_became_active":"2024-03-06T07:28:31.782362+0000","last_became_peered":"2024-03-06T07:28:31.782362+0000","last_unstale":"2024-03-06T07:28:31.783149+0000","last_undegraded":"2024-03-06T07:28:31.783149+0000","last_fullsized":"2024-03-06T07:28:31.783149+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:05.062758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772257+0000","last_change":"2024-03-06T07:28:31.772257+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772257+0000","last_undegraded":"2024-03-06T07:28:31.772257+0000","last_fullsized":"2024-03-06T07:28:31.772257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783884+0000","last_change":"2024-03-06T07:28:31.783136+0000","last_active":"2024-03-06T07:28:31.783884+0000","last_peered":"2024-03-06T07:28:31.783884+0000","last_clean":"2024-03-06T07:28:31.783884+0000","last_became_active":"2024-03-06T07:28:31.782350+0000","last_became_peered":"2024-03-06T07:28:31.782350+0000","last_unstale":"2024-03-06T07:28:31.783884+0000","last_undegraded":"2024-03-06T07:28:31.783884+0000","last_fullsized":"2024-03-06T07:28:31.783884+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:14.779042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770386+0000","last_change":"2024-03-06T07:28:31.770386+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770386+0000","last_undegraded":"2024-03-06T07:28:31.770386+0000","last_fullsized":"2024-03-06T07:28:31.770386+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782410+0000","last_change":"2024-03-06T07:28:31.779718+0000","last_active":"2024-03-06T07:28:31.782410+0000","last_peered":"2024-03-06T07:28:31.782410+0000","last_clean":"2024-03-06T07:28:31.782410+0000","last_became_active":"2024-03-06T07:28:31.779454+0000","last_became_peered":"2024-03-06T07:28:31.779454+0000","last_unstale":"2024-03-06T07:28:31.782410+0000","last_undegraded":"2024-03-06T07:28:31.782410+0000","last_fullsized":"2024-03-06T07:28:31.782410+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:56.378897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767719+0000","last_change":"2024-03-06T07:28:31.767719+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767719+0000","last_undegraded":"2024-03-06T07:28:31.767719+0000","last_fullsized":"2024-03-06T07:28:31.767719+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783278+0000","last_change":"2024-03-06T07:28:31.780216+0000","last_active":"2024-03-06T07:28:31.783278+0000","last_peered":"2024-03-06T07:28:31.783278+0000","last_clean":"2024-03-06T07:28:31.783278+0000","last_became_active":"2024-03-06T07:28:31.779950+0000","last_became_peered":"2024-03-06T07:28:31.779950+0000","last_unstale":"2024-03-06T07:28:31.783278+0000","last_undegraded":"2024-03-06T07:28:31.783278+0000","last_fullsized":"2024-03-06T07:28:31.783278+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:30.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771692+0000","last_change":"2024-03-06T07:28:31.771692+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771692+0000","last_undegraded":"2024-03-06T07:28:31.771692+0000","last_fullsized":"2024-03-06T07:28:31.771692+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.785702+0000","last_change":"2024-03-06T07:28:31.784524+0000","last_active":"2024-03-06T07:28:31.785702+0000","last_peered":"2024-03-06T07:28:31.785702+0000","last_clean":"2024-03-06T07:28:31.785702+0000","last_became_active":"2024-03-06T07:28:31.784259+0000","last_became_peered":"2024-03-06T07:28:31.784259+0000","last_unstale":"2024-03-06T07:28:31.785702+0000","last_undegraded":"2024-03-06T07:28:31.785702+0000","last_fullsized":"2024-03-06T07:28:31.785702+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:25.998178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774033+0000","last_change":"2024-03-06T07:28:31.774033+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774033+0000","last_undegraded":"2024-03-06T07:28:31.774033+0000","last_fullsized":"2024-03-06T07:28:31.774033+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778733+0000","last_change":"2024-03-06T07:28:31.772492+0000","last_active":"2024-03-06T07:28:31.778733+0000","last_peered":"2024-03-06T07:28:31.778733+0000","last_clean":"2024-03-06T07:28:31.778733+0000","last_became_active":"2024-03-06T07:28:31.772186+0000","last_became_peered":"2024-03-06T07:28:31.772186+0000","last_unstale":"2024-03-06T07:28:31.778733+0000","last_undegraded":"2024-03-06T07:28:31.778733+0000","last_fullsized":"2024-03-06T07:28:31.778733+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:18.698871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767571+0000","last_change":"2024-03-06T07:28:31.767571+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767571+0000","last_undegraded":"2024-03-06T07:28:31.767571+0000","last_fullsized":"2024-03-06T07:28:31.767571+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780832+0000","last_change":"2024-03-06T07:28:31.779645+0000","last_active":"2024-03-06T07:28:31.780832+0000","last_peered":"2024-03-06T07:28:31.780832+0000","last_clean":"2024-03-06T07:28:31.780832+0000","last_became_active":"2024-03-06T07:28:31.779391+0000","last_became_peered":"2024-03-06T07:28:31.779391+0000","last_unstale":"2024-03-06T07:28:31.780832+0000","last_undegraded":"2024-03-06T07:28:31.780832+0000","last_fullsized":"2024-03-06T07:28:31.780832+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:08.000434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771368+0000","last_change":"2024-03-06T07:28:31.771368+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771368+0000","last_undegraded":"2024-03-06T07:28:31.771368+0000","last_fullsized":"2024-03-06T07:28:31.771368+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781208+0000","last_change":"2024-03-06T07:28:31.779359+0000","last_active":"2024-03-06T07:28:31.781208+0000","last_peered":"2024-03-06T07:28:31.781208+0000","last_clean":"2024-03-06T07:28:31.781208+0000","last_became_active":"2024-03-06T07:28:31.779096+0000","last_became_peered":"2024-03-06T07:28:31.779096+0000","last_unstale":"2024-03-06T07:28:31.781208+0000","last_undegraded":"2024-03-06T07:28:31.781208+0000","last_fullsized":"2024-03-06T07:28:31.781208+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:47.517522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771504+0000","last_change":"2024-03-06T07:28:31.771504+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771504+0000","last_undegraded":"2024-03-06T07:28:31.771504+0000","last_fullsized":"2024-03-06T07:28:31.771504+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779248+0000","last_change":"2024-03-06T07:28:31.778300+0000","last_active":"2024-03-06T07:28:31.779248+0000","last_peered":"2024-03-06T07:28:31.779248+0000","last_clean":"2024-03-06T07:28:31.779248+0000","last_became_active":"2024-03-06T07:28:31.778036+0000","last_became_peered":"2024-03-06T07:28:31.778036+0000","last_unstale":"2024-03-06T07:28:31.779248+0000","last_undegraded":"2024-03-06T07:28:31.779248+0000","last_fullsized":"2024-03-06T07:28:31.779248+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:50.687009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768950+0000","last_change":"2024-03-06T07:28:31.768950+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768950+0000","last_undegraded":"2024-03-06T07:28:31.768950+0000","last_fullsized":"2024-03-06T07:28:31.768950+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780689+0000","last_change":"2024-03-06T07:28:31.779932+0000","last_active":"2024-03-06T07:28:31.780689+0000","last_peered":"2024-03-06T07:28:31.780689+0000","last_clean":"2024-03-06T07:28:31.780689+0000","last_became_active":"2024-03-06T07:28:31.779668+0000","last_became_peered":"2024-03-06T07:28:31.779668+0000","last_unstale":"2024-03-06T07:28:31.780689+0000","last_undegraded":"2024-03-06T07:28:31.780689+0000","last_fullsized":"2024-03-06T07:28:31.780689+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:57.626735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771353+0000","last_change":"2024-03-06T07:28:31.771353+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771353+0000","last_undegraded":"2024-03-06T07:28:31.771353+0000","last_fullsized":"2024-03-06T07:28:31.771353+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780358+0000","last_change":"2024-03-06T07:28:31.777451+0000","last_active":"2024-03-06T07:28:31.780358+0000","last_peered":"2024-03-06T07:28:31.780358+0000","last_clean":"2024-03-06T07:28:31.780358+0000","last_became_active":"2024-03-06T07:28:31.777201+0000","last_became_peered":"2024-03-06T07:28:31.777201+0000","last_unstale":"2024-03-06T07:28:31.780358+0000","last_undegraded":"2024-03-06T07:28:31.780358+0000","last_fullsized":"2024-03-06T07:28:31.780358+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:35.510219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774264+0000","last_change":"2024-03-06T07:28:31.774264+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774264+0000","last_undegraded":"2024-03-06T07:28:31.774264+0000","last_fullsized":"2024-03-06T07:28:31.774264+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781923+0000","last_change":"2024-03-06T07:28:31.781199+0000","last_active":"2024-03-06T07:28:31.781923+0000","last_peered":"2024-03-06T07:28:31.781923+0000","last_clean":"2024-03-06T07:28:31.781923+0000","last_became_active":"2024-03-06T07:28:31.780846+0000","last_became_peered":"2024-03-06T07:28:31.780846+0000","last_unstale":"2024-03-06T07:28:31.781923+0000","last_undegraded":"2024-03-06T07:28:31.781923+0000","last_fullsized":"2024-03-06T07:28:31.781923+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:13.183534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773280+0000","last_change":"2024-03-06T07:28:31.773280+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773280+0000","last_undegraded":"2024-03-06T07:28:31.773280+0000","last_fullsized":"2024-03-06T07:28:31.773280+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783201+0000","last_change":"2024-03-06T07:28:31.780249+0000","last_active":"2024-03-06T07:28:31.783201+0000","last_peered":"2024-03-06T07:28:31.783201+0000","last_clean":"2024-03-06T07:28:31.783201+0000","last_became_active":"2024-03-06T07:28:31.779962+0000","last_became_peered":"2024-03-06T07:28:31.779962+0000","last_unstale":"2024-03-06T07:28:31.783201+0000","last_undegraded":"2024-03-06T07:28:31.783201+0000","last_fullsized":"2024-03-06T07:28:31.783201+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:19.219570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769436+0000","last_change":"2024-03-06T07:28:31.769436+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769436+0000","last_undegraded":"2024-03-06T07:28:31.769436+0000","last_fullsized":"2024-03-06T07:28:31.769436+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778758+0000","last_change":"2024-03-06T07:28:31.773134+0000","last_active":"2024-03-06T07:28:31.778758+0000","last_peered":"2024-03-06T07:28:31.778758+0000","last_clean":"2024-03-06T07:28:31.778758+0000","last_became_active":"2024-03-06T07:28:31.772882+0000","last_became_peered":"2024-03-06T07:28:31.772882+0000","last_unstale":"2024-03-06T07:28:31.778758+0000","last_undegraded":"2024-03-06T07:28:31.778758+0000","last_fullsized":"2024-03-06T07:28:31.778758+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:51.160241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771848+0000","last_change":"2024-03-06T07:28:31.771848+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771848+0000","last_undegraded":"2024-03-06T07:28:31.771848+0000","last_fullsized":"2024-03-06T07:28:31.771848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781556+0000","last_change":"2024-03-06T07:28:31.780614+0000","last_active":"2024-03-06T07:28:31.781556+0000","last_peered":"2024-03-06T07:28:31.781556+0000","last_clean":"2024-03-06T07:28:31.781556+0000","last_became_active":"2024-03-06T07:28:31.780312+0000","last_became_peered":"2024-03-06T07:28:31.780312+0000","last_unstale":"2024-03-06T07:28:31.781556+0000","last_undegraded":"2024-03-06T07:28:31.781556+0000","last_fullsized":"2024-03-06T07:28:31.781556+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:20.580859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":350,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.775726+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:28:31.775726+0000","last_peered":"2024-03-06T07:28:31.775726+0000","last_clean":"2024-03-06T07:28:31.775726+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:28:31.775726+0000","last_undegraded":"2024-03-06T07:28:31.775726+0000","last_fullsized":"2024-03-06T07:28:31.775726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1257091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28508,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343332,"statfs":{"total":96636764160,"available":96607571968,"internally_reserved":0,"allocated":1523712,"data_stored":660448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40988,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330852,"statfs":{"total":96636764160,"available":96594792448,"internally_reserved":0,"allocated":1523712,"data_stored":660448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41584,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330256,"statfs":{"total":96636764160,"available":96594182144,"internally_reserved":0,"allocated":2129920,"data_stored":1257091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:35.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.888+0000 7f515f215640 1 Processor -- start 2024-03-06T07:28:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.888+0000 7f515f215640 1 -- start start 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f515f215640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f515806da40 0x7f51580cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f515f215640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580cc980 con 0x7f51581424f0 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f515f215640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580ccc60 con 0x7f51580cd7a0 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f515f215640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51580cd220 con 0x7f515806da40 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f515cfb1640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f51581424f0 0x7f515806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54040/0 (socket says 172.21.15.77:54040) 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f515cfb1640 1 -- 172.21.15.77:0/2438626644 learned_addr learned my addr 172.21.15.77:0/2438626644 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f5157fff640 1 --2- 172.21.15.77:0/2438626644 >> v2:172.21.15.158:3300/0 conn(0x7f515806da40 0x7f51580cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f51577fe640 1 -- 172.21.15.77:0/2438626644 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75896039 0 0) 0x7f51580cc980 con 0x7f51581424f0 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f51577fe640 1 -- 172.21.15.77:0/2438626644 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5148003520 con 0x7f51581424f0 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f5157fff640 1 -- 172.21.15.77:0/2438626644 >> v1:172.21.15.158:6789/0 conn(0x7f51580cd7a0 legacy=0x7f5158071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f5157fff640 1 -- 172.21.15.77:0/2438626644 >> v1:172.21.15.77:6789/0 conn(0x7f51581424f0 legacy=0x7f515806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.889+0000 7f5157fff640 1 -- 172.21.15.77:0/2438626644 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580ccf40 con 0x7f515806da40 2024-03-06T07:28:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.890+0000 7f5157fff640 1 --2- 172.21.15.77:0/2438626644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f515806da40 0x7f51580cc2f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f51440097a0 tx=0x7f514402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4595083b604be006 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.890+0000 7f51577fe640 1 -- 172.21.15.77:0/2438626644 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5144034070 con 0x7f515806da40 2024-03-06T07:28:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.890+0000 7f51577fe640 1 -- 172.21.15.77:0/2438626644 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5144030610 con 0x7f515806da40 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.890+0000 7f51577fe640 1 -- 172.21.15.77:0/2438626644 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51440309c0 con 0x7f515806da40 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.891+0000 7f515f215640 1 -- 172.21.15.77:0/2438626644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f515806da40 msgr2=0x7f51580cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.891+0000 7f515f215640 1 --2- 172.21.15.77:0/2438626644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f515806da40 0x7f51580cc2f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f51440097a0 tx=0x7f514402b960 comp rx=0 tx=0).stop 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.891+0000 7f515f215640 1 -- 172.21.15.77:0/2438626644 shutdown_connections 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.891+0000 7f515f215640 1 --2- 172.21.15.77:0/2438626644 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f515806da40 0x7f51580cc2f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.891+0000 7f515f215640 1 -- 172.21.15.77:0/2438626644 >> 172.21.15.77:0/2438626644 conn(0x7f51580cf8a0 msgr2=0x7f51580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.891+0000 7f515f215640 1 -- 172.21.15.77:0/2438626644 shutdown_connections 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.891+0000 7f515f215640 1 -- 172.21.15.77:0/2438626644 wait complete. 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.892+0000 7f515f215640 1 Processor -- start 2024-03-06T07:28:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.892+0000 7f515f215640 1 -- start start 2024-03-06T07:28:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515f215640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f515806da40 0x7f51581498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515f215640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51580cd7a0 0x7f51580b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515f215640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51581424f0 0x7f51580b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515f215640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515815d070 con 0x7f51581424f0 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515f215640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515815cef0 con 0x7f51580cd7a0 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515f215640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515815d1f0 con 0x7f515806da40 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51581424f0 0x7f51580b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51581424f0 0x7f51580b0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40618/0 (socket says 172.21.15.77:40618) 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 -- 172.21.15.77:0/1640195057 learned_addr learned my addr 172.21.15.77:0/1640195057 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515cfb1640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f515806da40 0x7f51581498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f5157fff640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51580cd7a0 0x7f51580b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 -- 172.21.15.77:0/1640195057 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f515806da40 msgr2=0x7f51581498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f515806da40 0x7f51581498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 -- 172.21.15.77:0/1640195057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51580cd7a0 msgr2=0x7f51580b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51580cd7a0 0x7f51580b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 -- 172.21.15.77:0/1640195057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51581f6530 con 0x7f51581424f0 2024-03-06T07:28:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515d7b2640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51581424f0 0x7f51580b0df0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f515000c7d0 tx=0x7f515000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f51557fa640 1 -- 172.21.15.77:0/1640195057 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150003b70 con 0x7f51581424f0 2024-03-06T07:28:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f51557fa640 1 -- 172.21.15.77:0/1640195057 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150003d30 con 0x7f51581424f0 2024-03-06T07:28:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515814a270 con 0x7f51581424f0 2024-03-06T07:28:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f51557fa640 1 -- 172.21.15.77:0/1640195057 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515000aa90 con 0x7f51581424f0 2024-03-06T07:28:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.893+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515814a830 con 0x7f51581424f0 2024-03-06T07:28:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.894+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515804f330 con 0x7f51581424f0 2024-03-06T07:28:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.895+0000 7f51557fa640 1 -- 172.21.15.77:0/1640195057 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f515000ac30 con 0x7f51581424f0 2024-03-06T07:28:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.895+0000 7f51557fa640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513c06b4d0 0x7f513c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.896+0000 7f515cfb1640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513c06b4d0 0x7f513c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.896+0000 7f51557fa640 1 -- 172.21.15.77:0/1640195057 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5150089f40 con 0x7f51581424f0 2024-03-06T07:28:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.896+0000 7f515cfb1640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513c06b4d0 0x7f513c06d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f514c0011e0 tx=0x7f514c007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:35.900+0000 7f51557fa640 1 -- 172.21.15.77:0/1640195057 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5150058800 con 0x7f51581424f0 2024-03-06T07:28:36.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.068+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f515814aa00 con 0x7f513c06b4d0 2024-03-06T07:28:36.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.074+0000 7f51557fa640 1 -- 172.21.15.77:0/1640195057 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247089 (secure 0 0 0) 0x7f515814aa00 con 0x7f513c06b4d0 2024-03-06T07:28:36.078 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:36.079 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:28:36.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513c06b4d0 msgr2=0x7f513c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513c06b4d0 0x7f513c06d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f514c0011e0 tx=0x7f514c007d30 comp rx=0 tx=0).stop 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51581424f0 msgr2=0x7f51580b0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51581424f0 0x7f51580b0df0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f515000c7d0 tx=0x7f515000cc90 comp rx=0 tx=0).stop 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 shutdown_connections 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513c06b4d0 0x7f513c06d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51581424f0 0x7f51580b0df0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f51580cd7a0 0x7f51580b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 --2- 172.21.15.77:0/1640195057 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f515806da40 0x7f51581498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 >> 172.21.15.77:0/1640195057 conn(0x7f51580cf8a0 msgr2=0x7f515806b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 shutdown_connections 2024-03-06T07:28:36.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.077+0000 7f515f215640 1 -- 172.21.15.77:0/1640195057 wait complete. 2024-03-06T07:28:36.090 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-03-06T07:28:35.455923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304196,"kb_used_data":13136,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754670524,"statfs":{"total":773094113280,"available":772782616576,"internally_reserved":0,"allocated":13451264,"data_stored":6501970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.693890"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770447+0000","last_change":"2024-03-06T07:28:31.770447+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770447+0000","last_undegraded":"2024-03-06T07:28:31.770447+0000","last_fullsized":"2024-03-06T07:28:31.770447+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775687+0000","last_change":"2024-03-06T07:28:31.775687+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775687+0000","last_undegraded":"2024-03-06T07:28:31.775687+0000","last_fullsized":"2024-03-06T07:28:31.775687+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768247+0000","last_change":"2024-03-06T07:28:31.768247+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768247+0000","last_undegraded":"2024-03-06T07:28:31.768247+0000","last_fullsized":"2024-03-06T07:28:31.768247+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770523+0000","last_change":"2024-03-06T07:28:31.770523+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770523+0000","last_undegraded":"2024-03-06T07:28:31.770523+0000","last_fullsized":"2024-03-06T07:28:31.770523+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772284+0000","last_change":"2024-03-06T07:28:31.772284+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772284+0000","last_undegraded":"2024-03-06T07:28:31.772284+0000","last_fullsized":"2024-03-06T07:28:31.772284+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767124+0000","last_change":"2024-03-06T07:28:31.767124+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767124+0000","last_undegraded":"2024-03-06T07:28:31.767124+0000","last_fullsized":"2024-03-06T07:28:31.767124+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770134+0000","last_change":"2024-03-06T07:28:31.770134+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770134+0000","last_undegraded":"2024-03-06T07:28:31.770134+0000","last_fullsized":"2024-03-06T07:28:31.770134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768838+0000","last_change":"2024-03-06T07:28:31.768838+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768838+0000","last_undegraded":"2024-03-06T07:28:31.768838+0000","last_fullsized":"2024-03-06T07:28:31.768838+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767826+0000","last_change":"2024-03-06T07:28:31.767826+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767826+0000","last_undegraded":"2024-03-06T07:28:31.767826+0000","last_fullsized":"2024-03-06T07:28:31.767826+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775301+0000","last_change":"2024-03-06T07:28:31.775301+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775301+0000","last_undegraded":"2024-03-06T07:28:31.775301+0000","last_fullsized":"2024-03-06T07:28:31.775301+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769962+0000","last_change":"2024-03-06T07:28:31.769962+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769962+0000","last_undegraded":"2024-03-06T07:28:31.769962+0000","last_fullsized":"2024-03-06T07:28:31.769962+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768517+0000","last_change":"2024-03-06T07:28:31.768517+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768517+0000","last_undegraded":"2024-03-06T07:28:31.768517+0000","last_fullsized":"2024-03-06T07:28:31.768517+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772112+0000","last_change":"2024-03-06T07:28:31.772112+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772112+0000","last_undegraded":"2024-03-06T07:28:31.772112+0000","last_fullsized":"2024-03-06T07:28:31.772112+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775114+0000","last_change":"2024-03-06T07:28:31.775114+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775114+0000","last_undegraded":"2024-03-06T07:28:31.775114+0000","last_fullsized":"2024-03-06T07:28:31.775114+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769966+0000","last_change":"2024-03-06T07:28:31.769966+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769966+0000","last_undegraded":"2024-03-06T07:28:31.769966+0000","last_fullsized":"2024-03-06T07:28:31.769966+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773730+0000","last_change":"2024-03-06T07:28:31.773730+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773730+0000","last_undegraded":"2024-03-06T07:28:31.773730+0000","last_fullsized":"2024-03-06T07:28:31.773730+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767225+0000","last_change":"2024-03-06T07:28:31.767225+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767225+0000","last_undegraded":"2024-03-06T07:28:31.767225+0000","last_fullsized":"2024-03-06T07:28:31.767225+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.777806+0000","last_change":"2024-03-06T07:28:31.777806+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.777806+0000","last_undegraded":"2024-03-06T07:28:31.777806+0000","last_fullsized":"2024-03-06T07:28:31.777806+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769133+0000","last_change":"2024-03-06T07:28:31.769133+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769133+0000","last_undegraded":"2024-03-06T07:28:31.769133+0000","last_fullsized":"2024-03-06T07:28:31.769133+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774231+0000","last_change":"2024-03-06T07:28:31.774231+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774231+0000","last_undegraded":"2024-03-06T07:28:31.774231+0000","last_fullsized":"2024-03-06T07:28:31.774231+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774487+0000","last_change":"2024-03-06T07:28:31.774487+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774487+0000","last_undegraded":"2024-03-06T07:28:31.774487+0000","last_fullsized":"2024-03-06T07:28:31.774487+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769129+0000","last_change":"2024-03-06T07:28:31.769129+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769129+0000","last_undegraded":"2024-03-06T07:28:31.769129+0000","last_fullsized":"2024-03-06T07:28:31.769129+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768187+0000","last_change":"2024-03-06T07:28:31.768187+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768187+0000","last_undegraded":"2024-03-06T07:28:31.768187+0000","last_fullsized":"2024-03-06T07:28:31.768187+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768712+0000","last_change":"2024-03-06T07:28:31.768712+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768712+0000","last_undegraded":"2024-03-06T07:28:31.768712+0000","last_fullsized":"2024-03-06T07:28:31.768712+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776050+0000","last_change":"2024-03-06T07:28:31.776050+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776050+0000","last_undegraded":"2024-03-06T07:28:31.776050+0000","last_fullsized":"2024-03-06T07:28:31.776050+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.777368+0000","last_change":"2024-03-06T07:28:31.777368+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.777368+0000","last_undegraded":"2024-03-06T07:28:31.777368+0000","last_fullsized":"2024-03-06T07:28:31.777368+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770197+0000","last_change":"2024-03-06T07:28:31.770197+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770197+0000","last_undegraded":"2024-03-06T07:28:31.770197+0000","last_fullsized":"2024-03-06T07:28:31.770197+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771951+0000","last_change":"2024-03-06T07:28:31.771951+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771951+0000","last_undegraded":"2024-03-06T07:28:31.771951+0000","last_fullsized":"2024-03-06T07:28:31.771951+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771671+0000","last_change":"2024-03-06T07:28:31.771671+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771671+0000","last_undegraded":"2024-03-06T07:28:31.771671+0000","last_fullsized":"2024-03-06T07:28:31.771671+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770827+0000","last_change":"2024-03-06T07:28:31.770827+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770827+0000","last_undegraded":"2024-03-06T07:28:31.770827+0000","last_fullsized":"2024-03-06T07:28:31.770827+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771788+0000","last_change":"2024-03-06T07:28:31.771788+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771788+0000","last_undegraded":"2024-03-06T07:28:31.771788+0000","last_fullsized":"2024-03-06T07:28:31.771788+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768199+0000","last_change":"2024-03-06T07:28:31.768199+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768199+0000","last_undegraded":"2024-03-06T07:28:31.768199+0000","last_fullsized":"2024-03-06T07:28:31.768199+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776999+0000","last_change":"2024-03-06T07:28:31.776999+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776999+0000","last_undegraded":"2024-03-06T07:28:31.776999+0000","last_fullsized":"2024-03-06T07:28:31.776999+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783927+0000","last_change":"2024-03-06T07:28:31.782852+0000","last_active":"2024-03-06T07:28:31.783927+0000","last_peered":"2024-03-06T07:28:31.783927+0000","last_clean":"2024-03-06T07:28:31.783927+0000","last_became_active":"2024-03-06T07:28:31.782575+0000","last_became_peered":"2024-03-06T07:28:31.782575+0000","last_unstale":"2024-03-06T07:28:31.783927+0000","last_undegraded":"2024-03-06T07:28:31.783927+0000","last_fullsized":"2024-03-06T07:28:31.783927+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:30.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767260+0000","last_change":"2024-03-06T07:28:31.767260+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767260+0000","last_undegraded":"2024-03-06T07:28:31.767260+0000","last_fullsized":"2024-03-06T07:28:31.767260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783865+0000","last_change":"2024-03-06T07:28:31.782726+0000","last_active":"2024-03-06T07:28:31.783865+0000","last_peered":"2024-03-06T07:28:31.783865+0000","last_clean":"2024-03-06T07:28:31.783865+0000","last_became_active":"2024-03-06T07:28:31.782447+0000","last_became_peered":"2024-03-06T07:28:31.782447+0000","last_unstale":"2024-03-06T07:28:31.783865+0000","last_undegraded":"2024-03-06T07:28:31.783865+0000","last_fullsized":"2024-03-06T07:28:31.783865+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:40.079443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774697+0000","last_change":"2024-03-06T07:28:31.774697+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774697+0000","last_undegraded":"2024-03-06T07:28:31.774697+0000","last_fullsized":"2024-03-06T07:28:31.774697+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.785061+0000","last_change":"2024-03-06T07:28:31.783586+0000","last_active":"2024-03-06T07:28:31.785061+0000","last_peered":"2024-03-06T07:28:31.785061+0000","last_clean":"2024-03-06T07:28:31.785061+0000","last_became_active":"2024-03-06T07:28:31.783329+0000","last_became_peered":"2024-03-06T07:28:31.783329+0000","last_unstale":"2024-03-06T07:28:31.785061+0000","last_undegraded":"2024-03-06T07:28:31.785061+0000","last_fullsized":"2024-03-06T07:28:31.785061+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:07.918185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773437+0000","last_change":"2024-03-06T07:28:31.773437+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773437+0000","last_undegraded":"2024-03-06T07:28:31.773437+0000","last_fullsized":"2024-03-06T07:28:31.773437+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782314+0000","last_change":"2024-03-06T07:28:31.780002+0000","last_active":"2024-03-06T07:28:31.782314+0000","last_peered":"2024-03-06T07:28:31.782314+0000","last_clean":"2024-03-06T07:28:31.782314+0000","last_became_active":"2024-03-06T07:28:31.779736+0000","last_became_peered":"2024-03-06T07:28:31.779736+0000","last_unstale":"2024-03-06T07:28:31.782314+0000","last_undegraded":"2024-03-06T07:28:31.782314+0000","last_fullsized":"2024-03-06T07:28:31.782314+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:37.028128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774574+0000","last_change":"2024-03-06T07:28:31.774574+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774574+0000","last_undegraded":"2024-03-06T07:28:31.774574+0000","last_fullsized":"2024-03-06T07:28:31.774574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784714+0000","last_change":"2024-03-06T07:28:31.783109+0000","last_active":"2024-03-06T07:28:31.784714+0000","last_peered":"2024-03-06T07:28:31.784714+0000","last_clean":"2024-03-06T07:28:31.784714+0000","last_became_active":"2024-03-06T07:28:31.782858+0000","last_became_peered":"2024-03-06T07:28:31.782858+0000","last_unstale":"2024-03-06T07:28:31.784714+0000","last_undegraded":"2024-03-06T07:28:31.784714+0000","last_fullsized":"2024-03-06T07:28:31.784714+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:41.710642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768260+0000","last_change":"2024-03-06T07:28:31.768260+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768260+0000","last_undegraded":"2024-03-06T07:28:31.768260+0000","last_fullsized":"2024-03-06T07:28:31.768260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783299+0000","last_change":"2024-03-06T07:28:31.781824+0000","last_active":"2024-03-06T07:28:31.783299+0000","last_peered":"2024-03-06T07:28:31.783299+0000","last_clean":"2024-03-06T07:28:31.783299+0000","last_became_active":"2024-03-06T07:28:31.781484+0000","last_became_peered":"2024-03-06T07:28:31.781484+0000","last_unstale":"2024-03-06T07:28:31.783299+0000","last_undegraded":"2024-03-06T07:28:31.783299+0000","last_fullsized":"2024-03-06T07:28:31.783299+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:59.115102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768166+0000","last_change":"2024-03-06T07:28:31.768166+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768166+0000","last_undegraded":"2024-03-06T07:28:31.768166+0000","last_fullsized":"2024-03-06T07:28:31.768166+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779805+0000","last_change":"2024-03-06T07:28:31.777579+0000","last_active":"2024-03-06T07:28:31.779805+0000","last_peered":"2024-03-06T07:28:31.779805+0000","last_clean":"2024-03-06T07:28:31.779805+0000","last_became_active":"2024-03-06T07:28:31.777290+0000","last_became_peered":"2024-03-06T07:28:31.777290+0000","last_unstale":"2024-03-06T07:28:31.779805+0000","last_undegraded":"2024-03-06T07:28:31.779805+0000","last_fullsized":"2024-03-06T07:28:31.779805+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:48.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775054+0000","last_change":"2024-03-06T07:28:31.775054+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775054+0000","last_undegraded":"2024-03-06T07:28:31.775054+0000","last_fullsized":"2024-03-06T07:28:31.775054+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780903+0000","last_change":"2024-03-06T07:28:31.775485+0000","last_active":"2024-03-06T07:28:31.780903+0000","last_peered":"2024-03-06T07:28:31.780903+0000","last_clean":"2024-03-06T07:28:31.780903+0000","last_became_active":"2024-03-06T07:28:31.775212+0000","last_became_peered":"2024-03-06T07:28:31.775212+0000","last_unstale":"2024-03-06T07:28:31.780903+0000","last_undegraded":"2024-03-06T07:28:31.780903+0000","last_fullsized":"2024-03-06T07:28:31.780903+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:35.868436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770026+0000","last_change":"2024-03-06T07:28:31.770026+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770026+0000","last_undegraded":"2024-03-06T07:28:31.770026+0000","last_fullsized":"2024-03-06T07:28:31.770026+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784013+0000","last_change":"2024-03-06T07:28:31.783404+0000","last_active":"2024-03-06T07:28:31.784013+0000","last_peered":"2024-03-06T07:28:31.784013+0000","last_clean":"2024-03-06T07:28:31.784013+0000","last_became_active":"2024-03-06T07:28:31.782969+0000","last_became_peered":"2024-03-06T07:28:31.782969+0000","last_unstale":"2024-03-06T07:28:31.784013+0000","last_undegraded":"2024-03-06T07:28:31.784013+0000","last_fullsized":"2024-03-06T07:28:31.784013+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:06.016386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776092+0000","last_change":"2024-03-06T07:28:31.776092+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776092+0000","last_undegraded":"2024-03-06T07:28:31.776092+0000","last_fullsized":"2024-03-06T07:28:31.776092+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784990+0000","last_change":"2024-03-06T07:28:31.784102+0000","last_active":"2024-03-06T07:28:31.784990+0000","last_peered":"2024-03-06T07:28:31.784990+0000","last_clean":"2024-03-06T07:28:31.784990+0000","last_became_active":"2024-03-06T07:28:31.783888+0000","last_became_peered":"2024-03-06T07:28:31.783888+0000","last_unstale":"2024-03-06T07:28:31.784990+0000","last_undegraded":"2024-03-06T07:28:31.784990+0000","last_fullsized":"2024-03-06T07:28:31.784990+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:56.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773622+0000","last_change":"2024-03-06T07:28:31.773622+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773622+0000","last_undegraded":"2024-03-06T07:28:31.773622+0000","last_fullsized":"2024-03-06T07:28:31.773622+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783836+0000","last_change":"2024-03-06T07:28:31.782822+0000","last_active":"2024-03-06T07:28:31.783836+0000","last_peered":"2024-03-06T07:28:31.783836+0000","last_clean":"2024-03-06T07:28:31.783836+0000","last_became_active":"2024-03-06T07:28:31.782523+0000","last_became_peered":"2024-03-06T07:28:31.782523+0000","last_unstale":"2024-03-06T07:28:31.783836+0000","last_undegraded":"2024-03-06T07:28:31.783836+0000","last_fullsized":"2024-03-06T07:28:31.783836+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:51.472938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773035+0000","last_change":"2024-03-06T07:28:31.773035+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773035+0000","last_undegraded":"2024-03-06T07:28:31.773035+0000","last_fullsized":"2024-03-06T07:28:31.773035+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779098+0000","last_change":"2024-03-06T07:28:31.776785+0000","last_active":"2024-03-06T07:28:31.779098+0000","last_peered":"2024-03-06T07:28:31.779098+0000","last_clean":"2024-03-06T07:28:31.779098+0000","last_became_active":"2024-03-06T07:28:31.776521+0000","last_became_peered":"2024-03-06T07:28:31.776521+0000","last_unstale":"2024-03-06T07:28:31.779098+0000","last_undegraded":"2024-03-06T07:28:31.779098+0000","last_fullsized":"2024-03-06T07:28:31.779098+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:58.794188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771283+0000","last_change":"2024-03-06T07:28:31.771283+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771283+0000","last_undegraded":"2024-03-06T07:28:31.771283+0000","last_fullsized":"2024-03-06T07:28:31.771283+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778422+0000","last_change":"2024-03-06T07:28:31.775867+0000","last_active":"2024-03-06T07:28:31.778422+0000","last_peered":"2024-03-06T07:28:31.778422+0000","last_clean":"2024-03-06T07:28:31.778422+0000","last_became_active":"2024-03-06T07:28:31.775622+0000","last_became_peered":"2024-03-06T07:28:31.775622+0000","last_unstale":"2024-03-06T07:28:31.778422+0000","last_undegraded":"2024-03-06T07:28:31.778422+0000","last_fullsized":"2024-03-06T07:28:31.778422+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:25.853619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772244+0000","last_change":"2024-03-06T07:28:31.772244+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772244+0000","last_undegraded":"2024-03-06T07:28:31.772244+0000","last_fullsized":"2024-03-06T07:28:31.772244+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780993+0000","last_change":"2024-03-06T07:28:31.777064+0000","last_active":"2024-03-06T07:28:31.780993+0000","last_peered":"2024-03-06T07:28:31.780993+0000","last_clean":"2024-03-06T07:28:31.780993+0000","last_became_active":"2024-03-06T07:28:31.776788+0000","last_became_peered":"2024-03-06T07:28:31.776788+0000","last_unstale":"2024-03-06T07:28:31.780993+0000","last_undegraded":"2024-03-06T07:28:31.780993+0000","last_fullsized":"2024-03-06T07:28:31.780993+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:41.214845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769329+0000","last_change":"2024-03-06T07:28:31.769329+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769329+0000","last_undegraded":"2024-03-06T07:28:31.769329+0000","last_fullsized":"2024-03-06T07:28:31.769329+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782969+0000","last_change":"2024-03-06T07:28:31.781673+0000","last_active":"2024-03-06T07:28:31.782969+0000","last_peered":"2024-03-06T07:28:31.782969+0000","last_clean":"2024-03-06T07:28:31.782969+0000","last_became_active":"2024-03-06T07:28:31.781383+0000","last_became_peered":"2024-03-06T07:28:31.781383+0000","last_unstale":"2024-03-06T07:28:31.782969+0000","last_undegraded":"2024-03-06T07:28:31.782969+0000","last_fullsized":"2024-03-06T07:28:31.782969+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:29.323269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768865+0000","last_change":"2024-03-06T07:28:31.768865+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768865+0000","last_undegraded":"2024-03-06T07:28:31.768865+0000","last_fullsized":"2024-03-06T07:28:31.768865+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782261+0000","last_change":"2024-03-06T07:28:31.780402+0000","last_active":"2024-03-06T07:28:31.782261+0000","last_peered":"2024-03-06T07:28:31.782261+0000","last_clean":"2024-03-06T07:28:31.782261+0000","last_became_active":"2024-03-06T07:28:31.780050+0000","last_became_peered":"2024-03-06T07:28:31.780050+0000","last_unstale":"2024-03-06T07:28:31.782261+0000","last_undegraded":"2024-03-06T07:28:31.782261+0000","last_fullsized":"2024-03-06T07:28:31.782261+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:37.538186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775650+0000","last_change":"2024-03-06T07:28:31.775650+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775650+0000","last_undegraded":"2024-03-06T07:28:31.775650+0000","last_fullsized":"2024-03-06T07:28:31.775650+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779815+0000","last_change":"2024-03-06T07:28:31.773233+0000","last_active":"2024-03-06T07:28:31.779815+0000","last_peered":"2024-03-06T07:28:31.779815+0000","last_clean":"2024-03-06T07:28:31.779815+0000","last_became_active":"2024-03-06T07:28:31.773028+0000","last_became_peered":"2024-03-06T07:28:31.773028+0000","last_unstale":"2024-03-06T07:28:31.779815+0000","last_undegraded":"2024-03-06T07:28:31.779815+0000","last_fullsized":"2024-03-06T07:28:31.779815+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:33.073077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773859+0000","last_change":"2024-03-06T07:28:31.773859+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773859+0000","last_undegraded":"2024-03-06T07:28:31.773859+0000","last_fullsized":"2024-03-06T07:28:31.773859+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783149+0000","last_change":"2024-03-06T07:28:31.782644+0000","last_active":"2024-03-06T07:28:31.783149+0000","last_peered":"2024-03-06T07:28:31.783149+0000","last_clean":"2024-03-06T07:28:31.783149+0000","last_became_active":"2024-03-06T07:28:31.782362+0000","last_became_peered":"2024-03-06T07:28:31.782362+0000","last_unstale":"2024-03-06T07:28:31.783149+0000","last_undegraded":"2024-03-06T07:28:31.783149+0000","last_fullsized":"2024-03-06T07:28:31.783149+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:05.062758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772257+0000","last_change":"2024-03-06T07:28:31.772257+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772257+0000","last_undegraded":"2024-03-06T07:28:31.772257+0000","last_fullsized":"2024-03-06T07:28:31.772257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783884+0000","last_change":"2024-03-06T07:28:31.783136+0000","last_active":"2024-03-06T07:28:31.783884+0000","last_peered":"2024-03-06T07:28:31.783884+0000","last_clean":"2024-03-06T07:28:31.783884+0000","last_became_active":"2024-03-06T07:28:31.782350+0000","last_became_peered":"2024-03-06T07:28:31.782350+0000","last_unstale":"2024-03-06T07:28:31.783884+0000","last_undegraded":"2024-03-06T07:28:31.783884+0000","last_fullsized":"2024-03-06T07:28:31.783884+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:14.779042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770386+0000","last_change":"2024-03-06T07:28:31.770386+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770386+0000","last_undegraded":"2024-03-06T07:28:31.770386+0000","last_fullsized":"2024-03-06T07:28:31.770386+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782410+0000","last_change":"2024-03-06T07:28:31.779718+0000","last_active":"2024-03-06T07:28:31.782410+0000","last_peered":"2024-03-06T07:28:31.782410+0000","last_clean":"2024-03-06T07:28:31.782410+0000","last_became_active":"2024-03-06T07:28:31.779454+0000","last_became_peered":"2024-03-06T07:28:31.779454+0000","last_unstale":"2024-03-06T07:28:31.782410+0000","last_undegraded":"2024-03-06T07:28:31.782410+0000","last_fullsized":"2024-03-06T07:28:31.782410+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:56.378897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767719+0000","last_change":"2024-03-06T07:28:31.767719+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767719+0000","last_undegraded":"2024-03-06T07:28:31.767719+0000","last_fullsized":"2024-03-06T07:28:31.767719+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783278+0000","last_change":"2024-03-06T07:28:31.780216+0000","last_active":"2024-03-06T07:28:31.783278+0000","last_peered":"2024-03-06T07:28:31.783278+0000","last_clean":"2024-03-06T07:28:31.783278+0000","last_became_active":"2024-03-06T07:28:31.779950+0000","last_became_peered":"2024-03-06T07:28:31.779950+0000","last_unstale":"2024-03-06T07:28:31.783278+0000","last_undegraded":"2024-03-06T07:28:31.783278+0000","last_fullsized":"2024-03-06T07:28:31.783278+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:30.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771692+0000","last_change":"2024-03-06T07:28:31.771692+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771692+0000","last_undegraded":"2024-03-06T07:28:31.771692+0000","last_fullsized":"2024-03-06T07:28:31.771692+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.785702+0000","last_change":"2024-03-06T07:28:31.784524+0000","last_active":"2024-03-06T07:28:31.785702+0000","last_peered":"2024-03-06T07:28:31.785702+0000","last_clean":"2024-03-06T07:28:31.785702+0000","last_became_active":"2024-03-06T07:28:31.784259+0000","last_became_peered":"2024-03-06T07:28:31.784259+0000","last_unstale":"2024-03-06T07:28:31.785702+0000","last_undegraded":"2024-03-06T07:28:31.785702+0000","last_fullsized":"2024-03-06T07:28:31.785702+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:25.998178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774033+0000","last_change":"2024-03-06T07:28:31.774033+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774033+0000","last_undegraded":"2024-03-06T07:28:31.774033+0000","last_fullsized":"2024-03-06T07:28:31.774033+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778733+0000","last_change":"2024-03-06T07:28:31.772492+0000","last_active":"2024-03-06T07:28:31.778733+0000","last_peered":"2024-03-06T07:28:31.778733+0000","last_clean":"2024-03-06T07:28:31.778733+0000","last_became_active":"2024-03-06T07:28:31.772186+0000","last_became_peered":"2024-03-06T07:28:31.772186+0000","last_unstale":"2024-03-06T07:28:31.778733+0000","last_undegraded":"2024-03-06T07:28:31.778733+0000","last_fullsized":"2024-03-06T07:28:31.778733+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:18.698871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767571+0000","last_change":"2024-03-06T07:28:31.767571+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767571+0000","last_undegraded":"2024-03-06T07:28:31.767571+0000","last_fullsized":"2024-03-06T07:28:31.767571+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780832+0000","last_change":"2024-03-06T07:28:31.779645+0000","last_active":"2024-03-06T07:28:31.780832+0000","last_peered":"2024-03-06T07:28:31.780832+0000","last_clean":"2024-03-06T07:28:31.780832+0000","last_became_active":"2024-03-06T07:28:31.779391+0000","last_became_peered":"2024-03-06T07:28:31.779391+0000","last_unstale":"2024-03-06T07:28:31.780832+0000","last_undegraded":"2024-03-06T07:28:31.780832+0000","last_fullsized":"2024-03-06T07:28:31.780832+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:08.000434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771368+0000","last_change":"2024-03-06T07:28:31.771368+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771368+0000","last_undegraded":"2024-03-06T07:28:31.771368+0000","last_fullsized":"2024-03-06T07:28:31.771368+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781208+0000","last_change":"2024-03-06T07:28:31.779359+0000","last_active":"2024-03-06T07:28:31.781208+0000","last_peered":"2024-03-06T07:28:31.781208+0000","last_clean":"2024-03-06T07:28:31.781208+0000","last_became_active":"2024-03-06T07:28:31.779096+0000","last_became_peered":"2024-03-06T07:28:31.779096+0000","last_unstale":"2024-03-06T07:28:31.781208+0000","last_undegraded":"2024-03-06T07:28:31.781208+0000","last_fullsized":"2024-03-06T07:28:31.781208+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:47.517522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771504+0000","last_change":"2024-03-06T07:28:31.771504+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771504+0000","last_undegraded":"2024-03-06T07:28:31.771504+0000","last_fullsized":"2024-03-06T07:28:31.771504+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779248+0000","last_change":"2024-03-06T07:28:31.778300+0000","last_active":"2024-03-06T07:28:31.779248+0000","last_peered":"2024-03-06T07:28:31.779248+0000","last_clean":"2024-03-06T07:28:31.779248+0000","last_became_active":"2024-03-06T07:28:31.778036+0000","last_became_peered":"2024-03-06T07:28:31.778036+0000","last_unstale":"2024-03-06T07:28:31.779248+0000","last_undegraded":"2024-03-06T07:28:31.779248+0000","last_fullsized":"2024-03-06T07:28:31.779248+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:50.687009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768950+0000","last_change":"2024-03-06T07:28:31.768950+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768950+0000","last_undegraded":"2024-03-06T07:28:31.768950+0000","last_fullsized":"2024-03-06T07:28:31.768950+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780689+0000","last_change":"2024-03-06T07:28:31.779932+0000","last_active":"2024-03-06T07:28:31.780689+0000","last_peered":"2024-03-06T07:28:31.780689+0000","last_clean":"2024-03-06T07:28:31.780689+0000","last_became_active":"2024-03-06T07:28:31.779668+0000","last_became_peered":"2024-03-06T07:28:31.779668+0000","last_unstale":"2024-03-06T07:28:31.780689+0000","last_undegraded":"2024-03-06T07:28:31.780689+0000","last_fullsized":"2024-03-06T07:28:31.780689+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:57.626735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771353+0000","last_change":"2024-03-06T07:28:31.771353+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771353+0000","last_undegraded":"2024-03-06T07:28:31.771353+0000","last_fullsized":"2024-03-06T07:28:31.771353+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780358+0000","last_change":"2024-03-06T07:28:31.777451+0000","last_active":"2024-03-06T07:28:31.780358+0000","last_peered":"2024-03-06T07:28:31.780358+0000","last_clean":"2024-03-06T07:28:31.780358+0000","last_became_active":"2024-03-06T07:28:31.777201+0000","last_became_peered":"2024-03-06T07:28:31.777201+0000","last_unstale":"2024-03-06T07:28:31.780358+0000","last_undegraded":"2024-03-06T07:28:31.780358+0000","last_fullsized":"2024-03-06T07:28:31.780358+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:35.510219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774264+0000","last_change":"2024-03-06T07:28:31.774264+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774264+0000","last_undegraded":"2024-03-06T07:28:31.774264+0000","last_fullsized":"2024-03-06T07:28:31.774264+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781923+0000","last_change":"2024-03-06T07:28:31.781199+0000","last_active":"2024-03-06T07:28:31.781923+0000","last_peered":"2024-03-06T07:28:31.781923+0000","last_clean":"2024-03-06T07:28:31.781923+0000","last_became_active":"2024-03-06T07:28:31.780846+0000","last_became_peered":"2024-03-06T07:28:31.780846+0000","last_unstale":"2024-03-06T07:28:31.781923+0000","last_undegraded":"2024-03-06T07:28:31.781923+0000","last_fullsized":"2024-03-06T07:28:31.781923+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:13.183534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773280+0000","last_change":"2024-03-06T07:28:31.773280+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773280+0000","last_undegraded":"2024-03-06T07:28:31.773280+0000","last_fullsized":"2024-03-06T07:28:31.773280+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783201+0000","last_change":"2024-03-06T07:28:31.780249+0000","last_active":"2024-03-06T07:28:31.783201+0000","last_peered":"2024-03-06T07:28:31.783201+0000","last_clean":"2024-03-06T07:28:31.783201+0000","last_became_active":"2024-03-06T07:28:31.779962+0000","last_became_peered":"2024-03-06T07:28:31.779962+0000","last_unstale":"2024-03-06T07:28:31.783201+0000","last_undegraded":"2024-03-06T07:28:31.783201+0000","last_fullsized":"2024-03-06T07:28:31.783201+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:19.219570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769436+0000","last_change":"2024-03-06T07:28:31.769436+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769436+0000","last_undegraded":"2024-03-06T07:28:31.769436+0000","last_fullsized":"2024-03-06T07:28:31.769436+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778758+0000","last_change":"2024-03-06T07:28:31.773134+0000","last_active":"2024-03-06T07:28:31.778758+0000","last_peered":"2024-03-06T07:28:31.778758+0000","last_clean":"2024-03-06T07:28:31.778758+0000","last_became_active":"2024-03-06T07:28:31.772882+0000","last_became_peered":"2024-03-06T07:28:31.772882+0000","last_unstale":"2024-03-06T07:28:31.778758+0000","last_undegraded":"2024-03-06T07:28:31.778758+0000","last_fullsized":"2024-03-06T07:28:31.778758+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:51.160241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771848+0000","last_change":"2024-03-06T07:28:31.771848+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771848+0000","last_undegraded":"2024-03-06T07:28:31.771848+0000","last_fullsized":"2024-03-06T07:28:31.771848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781556+0000","last_change":"2024-03-06T07:28:31.780614+0000","last_active":"2024-03-06T07:28:31.781556+0000","last_peered":"2024-03-06T07:28:31.781556+0000","last_clean":"2024-03-06T07:28:31.781556+0000","last_became_active":"2024-03-06T07:28:31.780312+0000","last_became_peered":"2024-03-06T07:28:31.780312+0000","last_unstale":"2024-03-06T07:28:31.781556+0000","last_undegraded":"2024-03-06T07:28:31.781556+0000","last_fullsized":"2024-03-06T07:28:31.781556+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:20.580859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":350,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.775726+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:28:31.775726+0000","last_peered":"2024-03-06T07:28:31.775726+0000","last_clean":"2024-03-06T07:28:31.775726+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:28:31.775726+0000","last_undegraded":"2024-03-06T07:28:31.775726+0000","last_fullsized":"2024-03-06T07:28:31.775726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1257091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28508,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343332,"statfs":{"total":96636764160,"available":96607571968,"internally_reserved":0,"allocated":1523712,"data_stored":660448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40988,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330852,"statfs":{"total":96636764160,"available":96594792448,"internally_reserved":0,"allocated":1523712,"data_stored":660448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41584,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330256,"statfs":{"total":96636764160,"available":96594182144,"internally_reserved":0,"allocated":2129920,"data_stored":1257091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:36.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:36.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.228+0000 7fb15c93a640 1 Processor -- start 2024-03-06T07:28:36.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.228+0000 7fb15c93a640 1 -- start start 2024-03-06T07:28:36.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15c93a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1581424f0 0x7fb1580cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15c93a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1580cd7a0 0x7fb15815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15c93a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580cc980 con 0x7fb15806d330 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15c93a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1580cd220 con 0x7fb1580cd7a0 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15c93a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580ccc60 con 0x7fb1581424f0 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15659c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1581424f0 0x7fb1580cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb155d9b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1580cd7a0 0x7fb15815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb155d9b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb1580cd7a0 0x7fb15815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:46988/0 (socket says 172.21.15.77:46988) 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb155d9b640 1 -- 172.21.15.77:0/1979738312 learned_addr learned my addr 172.21.15.77:0/1979738312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb156d9d640 1 --1- 172.21.15.77:0/1979738312 >> v1:172.21.15.158:6789/0 conn(0x7fb15806d330 0x7fb158071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49876/0 (socket says 172.21.15.77:49876) 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15659c640 1 --2- 172.21.15.77:0/1979738312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1581424f0 0x7fb1580cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:36488/0 (socket says 172.21.15.77:36488) 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15659c640 1 -- 172.21.15.77:0/1979738312 >> v2:172.21.15.158:3300/0 conn(0x7fb1580cd7a0 msgr2=0x7fb15815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15659c640 1 --2- 172.21.15.77:0/1979738312 >> v2:172.21.15.158:3300/0 conn(0x7fb1580cd7a0 0x7fb15815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.229+0000 7fb15659c640 1 -- 172.21.15.77:0/1979738312 >> v1:172.21.15.158:6789/0 conn(0x7fb15806d330 legacy=0x7fb158071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.230+0000 7fb15559a640 1 -- 172.21.15.77:0/1979738312 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562453314 0 0) 0x7fb1580cc980 con 0x7fb15806d330 2024-03-06T07:28:36.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.230+0000 7fb15659c640 1 -- 172.21.15.77:0/1979738312 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1580ccf40 con 0x7fb1581424f0 2024-03-06T07:28:36.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.230+0000 7fb155d9b640 1 --2- 172.21.15.77:0/1979738312 >> v2:172.21.15.158:3300/0 conn(0x7fb1580cd7a0 0x7fb15815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:36.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.230+0000 7fb15659c640 1 --2- 172.21.15.77:0/1979738312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1581424f0 0x7fb1580cc2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb148009d90 tx=0x7fb14802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e85fac4c20e6171 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.230+0000 7fb15559a640 1 -- 172.21.15.77:0/1979738312 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148039070 con 0x7fb1581424f0 2024-03-06T07:28:36.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.230+0000 7fb15559a640 1 -- 172.21.15.77:0/1979738312 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14802cc50 con 0x7fb1581424f0 2024-03-06T07:28:36.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.230+0000 7fb15559a640 1 -- 172.21.15.77:0/1979738312 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148034440 con 0x7fb1581424f0 2024-03-06T07:28:36.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.231+0000 7fb15c93a640 1 -- 172.21.15.77:0/1979738312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1581424f0 msgr2=0x7fb1580cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.231+0000 7fb15c93a640 1 --2- 172.21.15.77:0/1979738312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1581424f0 0x7fb1580cc2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb148009d90 tx=0x7fb14802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.231+0000 7fb15c93a640 1 -- 172.21.15.77:0/1979738312 shutdown_connections 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.231+0000 7fb15c93a640 1 --2- 172.21.15.77:0/1979738312 >> v2:172.21.15.158:3300/0 conn(0x7fb1580cd7a0 0x7fb15815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.231+0000 7fb15c93a640 1 --2- 172.21.15.77:0/1979738312 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1581424f0 0x7fb1580cc2f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.231+0000 7fb15c93a640 1 -- 172.21.15.77:0/1979738312 >> 172.21.15.77:0/1979738312 conn(0x7fb1580cf8a0 msgr2=0x7fb1580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.231+0000 7fb15c93a640 1 -- 172.21.15.77:0/1979738312 shutdown_connections 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.231+0000 7fb15c93a640 1 -- 172.21.15.77:0/1979738312 wait complete. 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.232+0000 7fb15c93a640 1 Processor -- start 2024-03-06T07:28:36.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.232+0000 7fb15c93a640 1 -- start start 2024-03-06T07:28:36.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb15c93a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb15806d330 0x7fb15815ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb15c93a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1580cd7a0 0x7fb15813c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb15c93a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1581424f0 0x7fb15815a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb15c93a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15806d940 con 0x7fb1581424f0 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb15c93a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15806d7c0 con 0x7fb15806d330 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb15c93a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15806dac0 con 0x7fb1580cd7a0 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb156d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1581424f0 0x7fb15815a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb156d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1581424f0 0x7fb15815a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40622/0 (socket says 172.21.15.77:40622) 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb156d9d640 1 -- 172.21.15.77:0/3793597272 learned_addr learned my addr 172.21.15.77:0/3793597272 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb155d9b640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1580cd7a0 0x7fb15813c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.233+0000 7fb15659c640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb15806d330 0x7fb15815ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb156d9d640 1 -- 172.21.15.77:0/3793597272 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1580cd7a0 msgr2=0x7fb15813c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb156d9d640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1580cd7a0 0x7fb15813c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb156d9d640 1 -- 172.21.15.77:0/3793597272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb15806d330 msgr2=0x7fb15815ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb156d9d640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb15806d330 0x7fb15815ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb156d9d640 1 -- 172.21.15.77:0/3793597272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15815f350 con 0x7fb1581424f0 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb156d9d640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1581424f0 0x7fb15815a850 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fb15000c450 tx=0x7fb15000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb146ffd640 1 -- 172.21.15.77:0/3793597272 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500074e0 con 0x7fb1581424f0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb146ffd640 1 -- 172.21.15.77:0/3793597272 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1500047e0 con 0x7fb1581424f0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15815e0c0 con 0x7fb1581424f0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb146ffd640 1 -- 172.21.15.77:0/3793597272 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150004bd0 con 0x7fb1581424f0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.234+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15815e680 con 0x7fb1581424f0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.235+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb158071010 con 0x7fb1581424f0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.235+0000 7fb146ffd640 1 -- 172.21.15.77:0/3793597272 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb150011070 con 0x7fb1581424f0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.236+0000 7fb146ffd640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb13806b4d0 0x7fb13806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.236+0000 7fb15659c640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb13806b4d0 0x7fb13806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.236+0000 7fb146ffd640 1 -- 172.21.15.77:0/3793597272 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb150089b90 con 0x7fb1581424f0 2024-03-06T07:28:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.236+0000 7fb15659c640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb13806b4d0 0x7fb13806d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb148036000 tx=0x7fb148030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.240+0000 7fb146ffd640 1 -- 172.21.15.77:0/3793597272 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb158071010 con 0x7fb1581424f0 2024-03-06T07:28:36.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.408+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb15804f330 con 0x7fb13806b4d0 2024-03-06T07:28:36.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.414+0000 7fb146ffd640 1 -- 172.21.15.77:0/3793597272 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247089 (secure 0 0 0) 0x7fb15804f330 con 0x7fb13806b4d0 2024-03-06T07:28:36.419 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:36.420 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:28:36.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb13806b4d0 msgr2=0x7fb13806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb13806b4d0 0x7fb13806d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb148036000 tx=0x7fb148030000 comp rx=0 tx=0).stop 2024-03-06T07:28:36.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1581424f0 msgr2=0x7fb15815a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1581424f0 0x7fb15815a850 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fb15000c450 tx=0x7fb15000c910 comp rx=0 tx=0).stop 2024-03-06T07:28:36.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 shutdown_connections 2024-03-06T07:28:36.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb13806b4d0 0x7fb13806d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1581424f0 0x7fb15815a850 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1580cd7a0 0x7fb15813c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 --2- 172.21.15.77:0/3793597272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb15806d330 0x7fb15815ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 >> 172.21.15.77:0/3793597272 conn(0x7fb1580cf8a0 msgr2=0x7fb1580b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 shutdown_connections 2024-03-06T07:28:36.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.417+0000 7fb15c93a640 1 -- 172.21.15.77:0/3793597272 wait complete. 2024-03-06T07:28:36.430 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-03-06T07:28:35.455923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304196,"kb_used_data":13136,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754670524,"statfs":{"total":773094113280,"available":772782616576,"internally_reserved":0,"allocated":13451264,"data_stored":6501970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.693890"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770447+0000","last_change":"2024-03-06T07:28:31.770447+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770447+0000","last_undegraded":"2024-03-06T07:28:31.770447+0000","last_fullsized":"2024-03-06T07:28:31.770447+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775687+0000","last_change":"2024-03-06T07:28:31.775687+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775687+0000","last_undegraded":"2024-03-06T07:28:31.775687+0000","last_fullsized":"2024-03-06T07:28:31.775687+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768247+0000","last_change":"2024-03-06T07:28:31.768247+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768247+0000","last_undegraded":"2024-03-06T07:28:31.768247+0000","last_fullsized":"2024-03-06T07:28:31.768247+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770523+0000","last_change":"2024-03-06T07:28:31.770523+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770523+0000","last_undegraded":"2024-03-06T07:28:31.770523+0000","last_fullsized":"2024-03-06T07:28:31.770523+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772284+0000","last_change":"2024-03-06T07:28:31.772284+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772284+0000","last_undegraded":"2024-03-06T07:28:31.772284+0000","last_fullsized":"2024-03-06T07:28:31.772284+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767124+0000","last_change":"2024-03-06T07:28:31.767124+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767124+0000","last_undegraded":"2024-03-06T07:28:31.767124+0000","last_fullsized":"2024-03-06T07:28:31.767124+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770134+0000","last_change":"2024-03-06T07:28:31.770134+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770134+0000","last_undegraded":"2024-03-06T07:28:31.770134+0000","last_fullsized":"2024-03-06T07:28:31.770134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768838+0000","last_change":"2024-03-06T07:28:31.768838+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768838+0000","last_undegraded":"2024-03-06T07:28:31.768838+0000","last_fullsized":"2024-03-06T07:28:31.768838+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767826+0000","last_change":"2024-03-06T07:28:31.767826+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767826+0000","last_undegraded":"2024-03-06T07:28:31.767826+0000","last_fullsized":"2024-03-06T07:28:31.767826+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775301+0000","last_change":"2024-03-06T07:28:31.775301+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775301+0000","last_undegraded":"2024-03-06T07:28:31.775301+0000","last_fullsized":"2024-03-06T07:28:31.775301+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769962+0000","last_change":"2024-03-06T07:28:31.769962+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769962+0000","last_undegraded":"2024-03-06T07:28:31.769962+0000","last_fullsized":"2024-03-06T07:28:31.769962+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768517+0000","last_change":"2024-03-06T07:28:31.768517+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768517+0000","last_undegraded":"2024-03-06T07:28:31.768517+0000","last_fullsized":"2024-03-06T07:28:31.768517+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772112+0000","last_change":"2024-03-06T07:28:31.772112+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772112+0000","last_undegraded":"2024-03-06T07:28:31.772112+0000","last_fullsized":"2024-03-06T07:28:31.772112+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775114+0000","last_change":"2024-03-06T07:28:31.775114+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775114+0000","last_undegraded":"2024-03-06T07:28:31.775114+0000","last_fullsized":"2024-03-06T07:28:31.775114+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769966+0000","last_change":"2024-03-06T07:28:31.769966+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769966+0000","last_undegraded":"2024-03-06T07:28:31.769966+0000","last_fullsized":"2024-03-06T07:28:31.769966+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773730+0000","last_change":"2024-03-06T07:28:31.773730+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773730+0000","last_undegraded":"2024-03-06T07:28:31.773730+0000","last_fullsized":"2024-03-06T07:28:31.773730+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767225+0000","last_change":"2024-03-06T07:28:31.767225+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767225+0000","last_undegraded":"2024-03-06T07:28:31.767225+0000","last_fullsized":"2024-03-06T07:28:31.767225+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.777806+0000","last_change":"2024-03-06T07:28:31.777806+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.777806+0000","last_undegraded":"2024-03-06T07:28:31.777806+0000","last_fullsized":"2024-03-06T07:28:31.777806+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769133+0000","last_change":"2024-03-06T07:28:31.769133+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769133+0000","last_undegraded":"2024-03-06T07:28:31.769133+0000","last_fullsized":"2024-03-06T07:28:31.769133+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774231+0000","last_change":"2024-03-06T07:28:31.774231+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774231+0000","last_undegraded":"2024-03-06T07:28:31.774231+0000","last_fullsized":"2024-03-06T07:28:31.774231+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774487+0000","last_change":"2024-03-06T07:28:31.774487+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774487+0000","last_undegraded":"2024-03-06T07:28:31.774487+0000","last_fullsized":"2024-03-06T07:28:31.774487+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769129+0000","last_change":"2024-03-06T07:28:31.769129+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769129+0000","last_undegraded":"2024-03-06T07:28:31.769129+0000","last_fullsized":"2024-03-06T07:28:31.769129+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768187+0000","last_change":"2024-03-06T07:28:31.768187+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768187+0000","last_undegraded":"2024-03-06T07:28:31.768187+0000","last_fullsized":"2024-03-06T07:28:31.768187+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768712+0000","last_change":"2024-03-06T07:28:31.768712+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768712+0000","last_undegraded":"2024-03-06T07:28:31.768712+0000","last_fullsized":"2024-03-06T07:28:31.768712+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776050+0000","last_change":"2024-03-06T07:28:31.776050+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776050+0000","last_undegraded":"2024-03-06T07:28:31.776050+0000","last_fullsized":"2024-03-06T07:28:31.776050+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.777368+0000","last_change":"2024-03-06T07:28:31.777368+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.777368+0000","last_undegraded":"2024-03-06T07:28:31.777368+0000","last_fullsized":"2024-03-06T07:28:31.777368+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770197+0000","last_change":"2024-03-06T07:28:31.770197+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770197+0000","last_undegraded":"2024-03-06T07:28:31.770197+0000","last_fullsized":"2024-03-06T07:28:31.770197+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771951+0000","last_change":"2024-03-06T07:28:31.771951+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771951+0000","last_undegraded":"2024-03-06T07:28:31.771951+0000","last_fullsized":"2024-03-06T07:28:31.771951+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771671+0000","last_change":"2024-03-06T07:28:31.771671+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771671+0000","last_undegraded":"2024-03-06T07:28:31.771671+0000","last_fullsized":"2024-03-06T07:28:31.771671+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770827+0000","last_change":"2024-03-06T07:28:31.770827+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770827+0000","last_undegraded":"2024-03-06T07:28:31.770827+0000","last_fullsized":"2024-03-06T07:28:31.770827+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771788+0000","last_change":"2024-03-06T07:28:31.771788+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771788+0000","last_undegraded":"2024-03-06T07:28:31.771788+0000","last_fullsized":"2024-03-06T07:28:31.771788+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768199+0000","last_change":"2024-03-06T07:28:31.768199+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768199+0000","last_undegraded":"2024-03-06T07:28:31.768199+0000","last_fullsized":"2024-03-06T07:28:31.768199+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776999+0000","last_change":"2024-03-06T07:28:31.776999+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776999+0000","last_undegraded":"2024-03-06T07:28:31.776999+0000","last_fullsized":"2024-03-06T07:28:31.776999+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783927+0000","last_change":"2024-03-06T07:28:31.782852+0000","last_active":"2024-03-06T07:28:31.783927+0000","last_peered":"2024-03-06T07:28:31.783927+0000","last_clean":"2024-03-06T07:28:31.783927+0000","last_became_active":"2024-03-06T07:28:31.782575+0000","last_became_peered":"2024-03-06T07:28:31.782575+0000","last_unstale":"2024-03-06T07:28:31.783927+0000","last_undegraded":"2024-03-06T07:28:31.783927+0000","last_fullsized":"2024-03-06T07:28:31.783927+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:30.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767260+0000","last_change":"2024-03-06T07:28:31.767260+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767260+0000","last_undegraded":"2024-03-06T07:28:31.767260+0000","last_fullsized":"2024-03-06T07:28:31.767260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783865+0000","last_change":"2024-03-06T07:28:31.782726+0000","last_active":"2024-03-06T07:28:31.783865+0000","last_peered":"2024-03-06T07:28:31.783865+0000","last_clean":"2024-03-06T07:28:31.783865+0000","last_became_active":"2024-03-06T07:28:31.782447+0000","last_became_peered":"2024-03-06T07:28:31.782447+0000","last_unstale":"2024-03-06T07:28:31.783865+0000","last_undegraded":"2024-03-06T07:28:31.783865+0000","last_fullsized":"2024-03-06T07:28:31.783865+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:40.079443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774697+0000","last_change":"2024-03-06T07:28:31.774697+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774697+0000","last_undegraded":"2024-03-06T07:28:31.774697+0000","last_fullsized":"2024-03-06T07:28:31.774697+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.785061+0000","last_change":"2024-03-06T07:28:31.783586+0000","last_active":"2024-03-06T07:28:31.785061+0000","last_peered":"2024-03-06T07:28:31.785061+0000","last_clean":"2024-03-06T07:28:31.785061+0000","last_became_active":"2024-03-06T07:28:31.783329+0000","last_became_peered":"2024-03-06T07:28:31.783329+0000","last_unstale":"2024-03-06T07:28:31.785061+0000","last_undegraded":"2024-03-06T07:28:31.785061+0000","last_fullsized":"2024-03-06T07:28:31.785061+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:07.918185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773437+0000","last_change":"2024-03-06T07:28:31.773437+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773437+0000","last_undegraded":"2024-03-06T07:28:31.773437+0000","last_fullsized":"2024-03-06T07:28:31.773437+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782314+0000","last_change":"2024-03-06T07:28:31.780002+0000","last_active":"2024-03-06T07:28:31.782314+0000","last_peered":"2024-03-06T07:28:31.782314+0000","last_clean":"2024-03-06T07:28:31.782314+0000","last_became_active":"2024-03-06T07:28:31.779736+0000","last_became_peered":"2024-03-06T07:28:31.779736+0000","last_unstale":"2024-03-06T07:28:31.782314+0000","last_undegraded":"2024-03-06T07:28:31.782314+0000","last_fullsized":"2024-03-06T07:28:31.782314+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:37.028128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774574+0000","last_change":"2024-03-06T07:28:31.774574+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774574+0000","last_undegraded":"2024-03-06T07:28:31.774574+0000","last_fullsized":"2024-03-06T07:28:31.774574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784714+0000","last_change":"2024-03-06T07:28:31.783109+0000","last_active":"2024-03-06T07:28:31.784714+0000","last_peered":"2024-03-06T07:28:31.784714+0000","last_clean":"2024-03-06T07:28:31.784714+0000","last_became_active":"2024-03-06T07:28:31.782858+0000","last_became_peered":"2024-03-06T07:28:31.782858+0000","last_unstale":"2024-03-06T07:28:31.784714+0000","last_undegraded":"2024-03-06T07:28:31.784714+0000","last_fullsized":"2024-03-06T07:28:31.784714+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:41.710642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768260+0000","last_change":"2024-03-06T07:28:31.768260+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768260+0000","last_undegraded":"2024-03-06T07:28:31.768260+0000","last_fullsized":"2024-03-06T07:28:31.768260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783299+0000","last_change":"2024-03-06T07:28:31.781824+0000","last_active":"2024-03-06T07:28:31.783299+0000","last_peered":"2024-03-06T07:28:31.783299+0000","last_clean":"2024-03-06T07:28:31.783299+0000","last_became_active":"2024-03-06T07:28:31.781484+0000","last_became_peered":"2024-03-06T07:28:31.781484+0000","last_unstale":"2024-03-06T07:28:31.783299+0000","last_undegraded":"2024-03-06T07:28:31.783299+0000","last_fullsized":"2024-03-06T07:28:31.783299+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:59.115102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768166+0000","last_change":"2024-03-06T07:28:31.768166+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768166+0000","last_undegraded":"2024-03-06T07:28:31.768166+0000","last_fullsized":"2024-03-06T07:28:31.768166+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779805+0000","last_change":"2024-03-06T07:28:31.777579+0000","last_active":"2024-03-06T07:28:31.779805+0000","last_peered":"2024-03-06T07:28:31.779805+0000","last_clean":"2024-03-06T07:28:31.779805+0000","last_became_active":"2024-03-06T07:28:31.777290+0000","last_became_peered":"2024-03-06T07:28:31.777290+0000","last_unstale":"2024-03-06T07:28:31.779805+0000","last_undegraded":"2024-03-06T07:28:31.779805+0000","last_fullsized":"2024-03-06T07:28:31.779805+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:48.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775054+0000","last_change":"2024-03-06T07:28:31.775054+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775054+0000","last_undegraded":"2024-03-06T07:28:31.775054+0000","last_fullsized":"2024-03-06T07:28:31.775054+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780903+0000","last_change":"2024-03-06T07:28:31.775485+0000","last_active":"2024-03-06T07:28:31.780903+0000","last_peered":"2024-03-06T07:28:31.780903+0000","last_clean":"2024-03-06T07:28:31.780903+0000","last_became_active":"2024-03-06T07:28:31.775212+0000","last_became_peered":"2024-03-06T07:28:31.775212+0000","last_unstale":"2024-03-06T07:28:31.780903+0000","last_undegraded":"2024-03-06T07:28:31.780903+0000","last_fullsized":"2024-03-06T07:28:31.780903+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:35.868436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770026+0000","last_change":"2024-03-06T07:28:31.770026+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770026+0000","last_undegraded":"2024-03-06T07:28:31.770026+0000","last_fullsized":"2024-03-06T07:28:31.770026+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784013+0000","last_change":"2024-03-06T07:28:31.783404+0000","last_active":"2024-03-06T07:28:31.784013+0000","last_peered":"2024-03-06T07:28:31.784013+0000","last_clean":"2024-03-06T07:28:31.784013+0000","last_became_active":"2024-03-06T07:28:31.782969+0000","last_became_peered":"2024-03-06T07:28:31.782969+0000","last_unstale":"2024-03-06T07:28:31.784013+0000","last_undegraded":"2024-03-06T07:28:31.784013+0000","last_fullsized":"2024-03-06T07:28:31.784013+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:06.016386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776092+0000","last_change":"2024-03-06T07:28:31.776092+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776092+0000","last_undegraded":"2024-03-06T07:28:31.776092+0000","last_fullsized":"2024-03-06T07:28:31.776092+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784990+0000","last_change":"2024-03-06T07:28:31.784102+0000","last_active":"2024-03-06T07:28:31.784990+0000","last_peered":"2024-03-06T07:28:31.784990+0000","last_clean":"2024-03-06T07:28:31.784990+0000","last_became_active":"2024-03-06T07:28:31.783888+0000","last_became_peered":"2024-03-06T07:28:31.783888+0000","last_unstale":"2024-03-06T07:28:31.784990+0000","last_undegraded":"2024-03-06T07:28:31.784990+0000","last_fullsized":"2024-03-06T07:28:31.784990+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:56.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773622+0000","last_change":"2024-03-06T07:28:31.773622+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773622+0000","last_undegraded":"2024-03-06T07:28:31.773622+0000","last_fullsized":"2024-03-06T07:28:31.773622+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783836+0000","last_change":"2024-03-06T07:28:31.782822+0000","last_active":"2024-03-06T07:28:31.783836+0000","last_peered":"2024-03-06T07:28:31.783836+0000","last_clean":"2024-03-06T07:28:31.783836+0000","last_became_active":"2024-03-06T07:28:31.782523+0000","last_became_peered":"2024-03-06T07:28:31.782523+0000","last_unstale":"2024-03-06T07:28:31.783836+0000","last_undegraded":"2024-03-06T07:28:31.783836+0000","last_fullsized":"2024-03-06T07:28:31.783836+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:51.472938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773035+0000","last_change":"2024-03-06T07:28:31.773035+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773035+0000","last_undegraded":"2024-03-06T07:28:31.773035+0000","last_fullsized":"2024-03-06T07:28:31.773035+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779098+0000","last_change":"2024-03-06T07:28:31.776785+0000","last_active":"2024-03-06T07:28:31.779098+0000","last_peered":"2024-03-06T07:28:31.779098+0000","last_clean":"2024-03-06T07:28:31.779098+0000","last_became_active":"2024-03-06T07:28:31.776521+0000","last_became_peered":"2024-03-06T07:28:31.776521+0000","last_unstale":"2024-03-06T07:28:31.779098+0000","last_undegraded":"2024-03-06T07:28:31.779098+0000","last_fullsized":"2024-03-06T07:28:31.779098+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:58.794188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771283+0000","last_change":"2024-03-06T07:28:31.771283+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771283+0000","last_undegraded":"2024-03-06T07:28:31.771283+0000","last_fullsized":"2024-03-06T07:28:31.771283+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778422+0000","last_change":"2024-03-06T07:28:31.775867+0000","last_active":"2024-03-06T07:28:31.778422+0000","last_peered":"2024-03-06T07:28:31.778422+0000","last_clean":"2024-03-06T07:28:31.778422+0000","last_became_active":"2024-03-06T07:28:31.775622+0000","last_became_peered":"2024-03-06T07:28:31.775622+0000","last_unstale":"2024-03-06T07:28:31.778422+0000","last_undegraded":"2024-03-06T07:28:31.778422+0000","last_fullsized":"2024-03-06T07:28:31.778422+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:25.853619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772244+0000","last_change":"2024-03-06T07:28:31.772244+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772244+0000","last_undegraded":"2024-03-06T07:28:31.772244+0000","last_fullsized":"2024-03-06T07:28:31.772244+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780993+0000","last_change":"2024-03-06T07:28:31.777064+0000","last_active":"2024-03-06T07:28:31.780993+0000","last_peered":"2024-03-06T07:28:31.780993+0000","last_clean":"2024-03-06T07:28:31.780993+0000","last_became_active":"2024-03-06T07:28:31.776788+0000","last_became_peered":"2024-03-06T07:28:31.776788+0000","last_unstale":"2024-03-06T07:28:31.780993+0000","last_undegraded":"2024-03-06T07:28:31.780993+0000","last_fullsized":"2024-03-06T07:28:31.780993+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:41.214845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769329+0000","last_change":"2024-03-06T07:28:31.769329+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769329+0000","last_undegraded":"2024-03-06T07:28:31.769329+0000","last_fullsized":"2024-03-06T07:28:31.769329+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782969+0000","last_change":"2024-03-06T07:28:31.781673+0000","last_active":"2024-03-06T07:28:31.782969+0000","last_peered":"2024-03-06T07:28:31.782969+0000","last_clean":"2024-03-06T07:28:31.782969+0000","last_became_active":"2024-03-06T07:28:31.781383+0000","last_became_peered":"2024-03-06T07:28:31.781383+0000","last_unstale":"2024-03-06T07:28:31.782969+0000","last_undegraded":"2024-03-06T07:28:31.782969+0000","last_fullsized":"2024-03-06T07:28:31.782969+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:29.323269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768865+0000","last_change":"2024-03-06T07:28:31.768865+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768865+0000","last_undegraded":"2024-03-06T07:28:31.768865+0000","last_fullsized":"2024-03-06T07:28:31.768865+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782261+0000","last_change":"2024-03-06T07:28:31.780402+0000","last_active":"2024-03-06T07:28:31.782261+0000","last_peered":"2024-03-06T07:28:31.782261+0000","last_clean":"2024-03-06T07:28:31.782261+0000","last_became_active":"2024-03-06T07:28:31.780050+0000","last_became_peered":"2024-03-06T07:28:31.780050+0000","last_unstale":"2024-03-06T07:28:31.782261+0000","last_undegraded":"2024-03-06T07:28:31.782261+0000","last_fullsized":"2024-03-06T07:28:31.782261+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:37.538186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775650+0000","last_change":"2024-03-06T07:28:31.775650+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775650+0000","last_undegraded":"2024-03-06T07:28:31.775650+0000","last_fullsized":"2024-03-06T07:28:31.775650+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779815+0000","last_change":"2024-03-06T07:28:31.773233+0000","last_active":"2024-03-06T07:28:31.779815+0000","last_peered":"2024-03-06T07:28:31.779815+0000","last_clean":"2024-03-06T07:28:31.779815+0000","last_became_active":"2024-03-06T07:28:31.773028+0000","last_became_peered":"2024-03-06T07:28:31.773028+0000","last_unstale":"2024-03-06T07:28:31.779815+0000","last_undegraded":"2024-03-06T07:28:31.779815+0000","last_fullsized":"2024-03-06T07:28:31.779815+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:33.073077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773859+0000","last_change":"2024-03-06T07:28:31.773859+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773859+0000","last_undegraded":"2024-03-06T07:28:31.773859+0000","last_fullsized":"2024-03-06T07:28:31.773859+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783149+0000","last_change":"2024-03-06T07:28:31.782644+0000","last_active":"2024-03-06T07:28:31.783149+0000","last_peered":"2024-03-06T07:28:31.783149+0000","last_clean":"2024-03-06T07:28:31.783149+0000","last_became_active":"2024-03-06T07:28:31.782362+0000","last_became_peered":"2024-03-06T07:28:31.782362+0000","last_unstale":"2024-03-06T07:28:31.783149+0000","last_undegraded":"2024-03-06T07:28:31.783149+0000","last_fullsized":"2024-03-06T07:28:31.783149+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:05.062758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772257+0000","last_change":"2024-03-06T07:28:31.772257+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772257+0000","last_undegraded":"2024-03-06T07:28:31.772257+0000","last_fullsized":"2024-03-06T07:28:31.772257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783884+0000","last_change":"2024-03-06T07:28:31.783136+0000","last_active":"2024-03-06T07:28:31.783884+0000","last_peered":"2024-03-06T07:28:31.783884+0000","last_clean":"2024-03-06T07:28:31.783884+0000","last_became_active":"2024-03-06T07:28:31.782350+0000","last_became_peered":"2024-03-06T07:28:31.782350+0000","last_unstale":"2024-03-06T07:28:31.783884+0000","last_undegraded":"2024-03-06T07:28:31.783884+0000","last_fullsized":"2024-03-06T07:28:31.783884+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:14.779042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770386+0000","last_change":"2024-03-06T07:28:31.770386+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770386+0000","last_undegraded":"2024-03-06T07:28:31.770386+0000","last_fullsized":"2024-03-06T07:28:31.770386+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782410+0000","last_change":"2024-03-06T07:28:31.779718+0000","last_active":"2024-03-06T07:28:31.782410+0000","last_peered":"2024-03-06T07:28:31.782410+0000","last_clean":"2024-03-06T07:28:31.782410+0000","last_became_active":"2024-03-06T07:28:31.779454+0000","last_became_peered":"2024-03-06T07:28:31.779454+0000","last_unstale":"2024-03-06T07:28:31.782410+0000","last_undegraded":"2024-03-06T07:28:31.782410+0000","last_fullsized":"2024-03-06T07:28:31.782410+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:56.378897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767719+0000","last_change":"2024-03-06T07:28:31.767719+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767719+0000","last_undegraded":"2024-03-06T07:28:31.767719+0000","last_fullsized":"2024-03-06T07:28:31.767719+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783278+0000","last_change":"2024-03-06T07:28:31.780216+0000","last_active":"2024-03-06T07:28:31.783278+0000","last_peered":"2024-03-06T07:28:31.783278+0000","last_clean":"2024-03-06T07:28:31.783278+0000","last_became_active":"2024-03-06T07:28:31.779950+0000","last_became_peered":"2024-03-06T07:28:31.779950+0000","last_unstale":"2024-03-06T07:28:31.783278+0000","last_undegraded":"2024-03-06T07:28:31.783278+0000","last_fullsized":"2024-03-06T07:28:31.783278+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:30.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771692+0000","last_change":"2024-03-06T07:28:31.771692+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771692+0000","last_undegraded":"2024-03-06T07:28:31.771692+0000","last_fullsized":"2024-03-06T07:28:31.771692+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.785702+0000","last_change":"2024-03-06T07:28:31.784524+0000","last_active":"2024-03-06T07:28:31.785702+0000","last_peered":"2024-03-06T07:28:31.785702+0000","last_clean":"2024-03-06T07:28:31.785702+0000","last_became_active":"2024-03-06T07:28:31.784259+0000","last_became_peered":"2024-03-06T07:28:31.784259+0000","last_unstale":"2024-03-06T07:28:31.785702+0000","last_undegraded":"2024-03-06T07:28:31.785702+0000","last_fullsized":"2024-03-06T07:28:31.785702+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:25.998178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774033+0000","last_change":"2024-03-06T07:28:31.774033+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774033+0000","last_undegraded":"2024-03-06T07:28:31.774033+0000","last_fullsized":"2024-03-06T07:28:31.774033+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778733+0000","last_change":"2024-03-06T07:28:31.772492+0000","last_active":"2024-03-06T07:28:31.778733+0000","last_peered":"2024-03-06T07:28:31.778733+0000","last_clean":"2024-03-06T07:28:31.778733+0000","last_became_active":"2024-03-06T07:28:31.772186+0000","last_became_peered":"2024-03-06T07:28:31.772186+0000","last_unstale":"2024-03-06T07:28:31.778733+0000","last_undegraded":"2024-03-06T07:28:31.778733+0000","last_fullsized":"2024-03-06T07:28:31.778733+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:18.698871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767571+0000","last_change":"2024-03-06T07:28:31.767571+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767571+0000","last_undegraded":"2024-03-06T07:28:31.767571+0000","last_fullsized":"2024-03-06T07:28:31.767571+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780832+0000","last_change":"2024-03-06T07:28:31.779645+0000","last_active":"2024-03-06T07:28:31.780832+0000","last_peered":"2024-03-06T07:28:31.780832+0000","last_clean":"2024-03-06T07:28:31.780832+0000","last_became_active":"2024-03-06T07:28:31.779391+0000","last_became_peered":"2024-03-06T07:28:31.779391+0000","last_unstale":"2024-03-06T07:28:31.780832+0000","last_undegraded":"2024-03-06T07:28:31.780832+0000","last_fullsized":"2024-03-06T07:28:31.780832+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:08.000434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771368+0000","last_change":"2024-03-06T07:28:31.771368+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771368+0000","last_undegraded":"2024-03-06T07:28:31.771368+0000","last_fullsized":"2024-03-06T07:28:31.771368+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781208+0000","last_change":"2024-03-06T07:28:31.779359+0000","last_active":"2024-03-06T07:28:31.781208+0000","last_peered":"2024-03-06T07:28:31.781208+0000","last_clean":"2024-03-06T07:28:31.781208+0000","last_became_active":"2024-03-06T07:28:31.779096+0000","last_became_peered":"2024-03-06T07:28:31.779096+0000","last_unstale":"2024-03-06T07:28:31.781208+0000","last_undegraded":"2024-03-06T07:28:31.781208+0000","last_fullsized":"2024-03-06T07:28:31.781208+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:47.517522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771504+0000","last_change":"2024-03-06T07:28:31.771504+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771504+0000","last_undegraded":"2024-03-06T07:28:31.771504+0000","last_fullsized":"2024-03-06T07:28:31.771504+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779248+0000","last_change":"2024-03-06T07:28:31.778300+0000","last_active":"2024-03-06T07:28:31.779248+0000","last_peered":"2024-03-06T07:28:31.779248+0000","last_clean":"2024-03-06T07:28:31.779248+0000","last_became_active":"2024-03-06T07:28:31.778036+0000","last_became_peered":"2024-03-06T07:28:31.778036+0000","last_unstale":"2024-03-06T07:28:31.779248+0000","last_undegraded":"2024-03-06T07:28:31.779248+0000","last_fullsized":"2024-03-06T07:28:31.779248+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:50.687009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768950+0000","last_change":"2024-03-06T07:28:31.768950+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768950+0000","last_undegraded":"2024-03-06T07:28:31.768950+0000","last_fullsized":"2024-03-06T07:28:31.768950+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780689+0000","last_change":"2024-03-06T07:28:31.779932+0000","last_active":"2024-03-06T07:28:31.780689+0000","last_peered":"2024-03-06T07:28:31.780689+0000","last_clean":"2024-03-06T07:28:31.780689+0000","last_became_active":"2024-03-06T07:28:31.779668+0000","last_became_peered":"2024-03-06T07:28:31.779668+0000","last_unstale":"2024-03-06T07:28:31.780689+0000","last_undegraded":"2024-03-06T07:28:31.780689+0000","last_fullsized":"2024-03-06T07:28:31.780689+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:57.626735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771353+0000","last_change":"2024-03-06T07:28:31.771353+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771353+0000","last_undegraded":"2024-03-06T07:28:31.771353+0000","last_fullsized":"2024-03-06T07:28:31.771353+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780358+0000","last_change":"2024-03-06T07:28:31.777451+0000","last_active":"2024-03-06T07:28:31.780358+0000","last_peered":"2024-03-06T07:28:31.780358+0000","last_clean":"2024-03-06T07:28:31.780358+0000","last_became_active":"2024-03-06T07:28:31.777201+0000","last_became_peered":"2024-03-06T07:28:31.777201+0000","last_unstale":"2024-03-06T07:28:31.780358+0000","last_undegraded":"2024-03-06T07:28:31.780358+0000","last_fullsized":"2024-03-06T07:28:31.780358+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:35.510219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774264+0000","last_change":"2024-03-06T07:28:31.774264+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774264+0000","last_undegraded":"2024-03-06T07:28:31.774264+0000","last_fullsized":"2024-03-06T07:28:31.774264+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781923+0000","last_change":"2024-03-06T07:28:31.781199+0000","last_active":"2024-03-06T07:28:31.781923+0000","last_peered":"2024-03-06T07:28:31.781923+0000","last_clean":"2024-03-06T07:28:31.781923+0000","last_became_active":"2024-03-06T07:28:31.780846+0000","last_became_peered":"2024-03-06T07:28:31.780846+0000","last_unstale":"2024-03-06T07:28:31.781923+0000","last_undegraded":"2024-03-06T07:28:31.781923+0000","last_fullsized":"2024-03-06T07:28:31.781923+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:13.183534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773280+0000","last_change":"2024-03-06T07:28:31.773280+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773280+0000","last_undegraded":"2024-03-06T07:28:31.773280+0000","last_fullsized":"2024-03-06T07:28:31.773280+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783201+0000","last_change":"2024-03-06T07:28:31.780249+0000","last_active":"2024-03-06T07:28:31.783201+0000","last_peered":"2024-03-06T07:28:31.783201+0000","last_clean":"2024-03-06T07:28:31.783201+0000","last_became_active":"2024-03-06T07:28:31.779962+0000","last_became_peered":"2024-03-06T07:28:31.779962+0000","last_unstale":"2024-03-06T07:28:31.783201+0000","last_undegraded":"2024-03-06T07:28:31.783201+0000","last_fullsized":"2024-03-06T07:28:31.783201+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:19.219570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769436+0000","last_change":"2024-03-06T07:28:31.769436+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769436+0000","last_undegraded":"2024-03-06T07:28:31.769436+0000","last_fullsized":"2024-03-06T07:28:31.769436+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778758+0000","last_change":"2024-03-06T07:28:31.773134+0000","last_active":"2024-03-06T07:28:31.778758+0000","last_peered":"2024-03-06T07:28:31.778758+0000","last_clean":"2024-03-06T07:28:31.778758+0000","last_became_active":"2024-03-06T07:28:31.772882+0000","last_became_peered":"2024-03-06T07:28:31.772882+0000","last_unstale":"2024-03-06T07:28:31.778758+0000","last_undegraded":"2024-03-06T07:28:31.778758+0000","last_fullsized":"2024-03-06T07:28:31.778758+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:51.160241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771848+0000","last_change":"2024-03-06T07:28:31.771848+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771848+0000","last_undegraded":"2024-03-06T07:28:31.771848+0000","last_fullsized":"2024-03-06T07:28:31.771848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781556+0000","last_change":"2024-03-06T07:28:31.780614+0000","last_active":"2024-03-06T07:28:31.781556+0000","last_peered":"2024-03-06T07:28:31.781556+0000","last_clean":"2024-03-06T07:28:31.781556+0000","last_became_active":"2024-03-06T07:28:31.780312+0000","last_became_peered":"2024-03-06T07:28:31.780312+0000","last_unstale":"2024-03-06T07:28:31.781556+0000","last_undegraded":"2024-03-06T07:28:31.781556+0000","last_fullsized":"2024-03-06T07:28:31.781556+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:20.580859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":350,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.775726+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:28:31.775726+0000","last_peered":"2024-03-06T07:28:31.775726+0000","last_clean":"2024-03-06T07:28:31.775726+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:28:31.775726+0000","last_undegraded":"2024-03-06T07:28:31.775726+0000","last_fullsized":"2024-03-06T07:28:31.775726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1257091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28508,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343332,"statfs":{"total":96636764160,"available":96607571968,"internally_reserved":0,"allocated":1523712,"data_stored":660448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40988,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330852,"statfs":{"total":96636764160,"available":96594792448,"internally_reserved":0,"allocated":1523712,"data_stored":660448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41584,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330256,"statfs":{"total":96636764160,"available":96594182144,"internally_reserved":0,"allocated":2129920,"data_stored":1257091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:36.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3f is not active+clean 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.770447+0000', 'last_change': '2024-03-06T07:28:31.770447+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.770447+0000', 'last_undegraded': '2024-03-06T07:28:31.770447+0000', 'last_fullsized': '2024-03-06T07:28:31.770447+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3e is not active+clean 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.775687+0000', 'last_change': '2024-03-06T07:28:31.775687+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.775687+0000', 'last_undegraded': '2024-03-06T07:28:31.775687+0000', 'last_fullsized': '2024-03-06T07:28:31.775687+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3d is not active+clean 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768247+0000', 'last_change': '2024-03-06T07:28:31.768247+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768247+0000', 'last_undegraded': '2024-03-06T07:28:31.768247+0000', 'last_fullsized': '2024-03-06T07:28:31.768247+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3c is not active+clean 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.770523+0000', 'last_change': '2024-03-06T07:28:31.770523+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.770523+0000', 'last_undegraded': '2024-03-06T07:28:31.770523+0000', 'last_fullsized': '2024-03-06T07:28:31.770523+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3b is not active+clean 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.772284+0000', 'last_change': '2024-03-06T07:28:31.772284+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.772284+0000', 'last_undegraded': '2024-03-06T07:28:31.772284+0000', 'last_fullsized': '2024-03-06T07:28:31.772284+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3a is not active+clean 2024-03-06T07:28:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.767124+0000', 'last_change': '2024-03-06T07:28:31.767124+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.767124+0000', 'last_undegraded': '2024-03-06T07:28:31.767124+0000', 'last_fullsized': '2024-03-06T07:28:31.767124+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.39 is not active+clean 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.770134+0000', 'last_change': '2024-03-06T07:28:31.770134+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.770134+0000', 'last_undegraded': '2024-03-06T07:28:31.770134+0000', 'last_fullsized': '2024-03-06T07:28:31.770134+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.38 is not active+clean 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768838+0000', 'last_change': '2024-03-06T07:28:31.768838+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768838+0000', 'last_undegraded': '2024-03-06T07:28:31.768838+0000', 'last_fullsized': '2024-03-06T07:28:31.768838+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.37 is not active+clean 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.767826+0000', 'last_change': '2024-03-06T07:28:31.767826+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.767826+0000', 'last_undegraded': '2024-03-06T07:28:31.767826+0000', 'last_fullsized': '2024-03-06T07:28:31.767826+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.36 is not active+clean 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.775301+0000', 'last_change': '2024-03-06T07:28:31.775301+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.775301+0000', 'last_undegraded': '2024-03-06T07:28:31.775301+0000', 'last_fullsized': '2024-03-06T07:28:31.775301+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.35 is not active+clean 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.769962+0000', 'last_change': '2024-03-06T07:28:31.769962+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.769962+0000', 'last_undegraded': '2024-03-06T07:28:31.769962+0000', 'last_fullsized': '2024-03-06T07:28:31.769962+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.34 is not active+clean 2024-03-06T07:28:36.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768517+0000', 'last_change': '2024-03-06T07:28:31.768517+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768517+0000', 'last_undegraded': '2024-03-06T07:28:31.768517+0000', 'last_fullsized': '2024-03-06T07:28:31.768517+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.33 is not active+clean 2024-03-06T07:28:36.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.772112+0000', 'last_change': '2024-03-06T07:28:31.772112+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.772112+0000', 'last_undegraded': '2024-03-06T07:28:31.772112+0000', 'last_fullsized': '2024-03-06T07:28:31.772112+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.32 is not active+clean 2024-03-06T07:28:36.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.775114+0000', 'last_change': '2024-03-06T07:28:31.775114+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.775114+0000', 'last_undegraded': '2024-03-06T07:28:31.775114+0000', 'last_fullsized': '2024-03-06T07:28:31.775114+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.31 is not active+clean 2024-03-06T07:28:36.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.769966+0000', 'last_change': '2024-03-06T07:28:31.769966+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.769966+0000', 'last_undegraded': '2024-03-06T07:28:31.769966+0000', 'last_fullsized': '2024-03-06T07:28:31.769966+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.30 is not active+clean 2024-03-06T07:28:36.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.773730+0000', 'last_change': '2024-03-06T07:28:31.773730+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.773730+0000', 'last_undegraded': '2024-03-06T07:28:31.773730+0000', 'last_fullsized': '2024-03-06T07:28:31.773730+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2f is not active+clean 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.767225+0000', 'last_change': '2024-03-06T07:28:31.767225+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.767225+0000', 'last_undegraded': '2024-03-06T07:28:31.767225+0000', 'last_fullsized': '2024-03-06T07:28:31.767225+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2e is not active+clean 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.777806+0000', 'last_change': '2024-03-06T07:28:31.777806+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.777806+0000', 'last_undegraded': '2024-03-06T07:28:31.777806+0000', 'last_fullsized': '2024-03-06T07:28:31.777806+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2d is not active+clean 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.769133+0000', 'last_change': '2024-03-06T07:28:31.769133+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.769133+0000', 'last_undegraded': '2024-03-06T07:28:31.769133+0000', 'last_fullsized': '2024-03-06T07:28:31.769133+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2c is not active+clean 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.774231+0000', 'last_change': '2024-03-06T07:28:31.774231+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.774231+0000', 'last_undegraded': '2024-03-06T07:28:31.774231+0000', 'last_fullsized': '2024-03-06T07:28:31.774231+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2b is not active+clean 2024-03-06T07:28:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.774487+0000', 'last_change': '2024-03-06T07:28:31.774487+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.774487+0000', 'last_undegraded': '2024-03-06T07:28:31.774487+0000', 'last_fullsized': '2024-03-06T07:28:31.774487+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2a is not active+clean 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.769129+0000', 'last_change': '2024-03-06T07:28:31.769129+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.769129+0000', 'last_undegraded': '2024-03-06T07:28:31.769129+0000', 'last_fullsized': '2024-03-06T07:28:31.769129+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.29 is not active+clean 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768187+0000', 'last_change': '2024-03-06T07:28:31.768187+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768187+0000', 'last_undegraded': '2024-03-06T07:28:31.768187+0000', 'last_fullsized': '2024-03-06T07:28:31.768187+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.28 is not active+clean 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768712+0000', 'last_change': '2024-03-06T07:28:31.768712+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768712+0000', 'last_undegraded': '2024-03-06T07:28:31.768712+0000', 'last_fullsized': '2024-03-06T07:28:31.768712+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.27 is not active+clean 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.776050+0000', 'last_change': '2024-03-06T07:28:31.776050+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.776050+0000', 'last_undegraded': '2024-03-06T07:28:31.776050+0000', 'last_fullsized': '2024-03-06T07:28:31.776050+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.438 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.26 is not active+clean 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.777368+0000', 'last_change': '2024-03-06T07:28:31.777368+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.777368+0000', 'last_undegraded': '2024-03-06T07:28:31.777368+0000', 'last_fullsized': '2024-03-06T07:28:31.777368+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.25 is not active+clean 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.770197+0000', 'last_change': '2024-03-06T07:28:31.770197+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.770197+0000', 'last_undegraded': '2024-03-06T07:28:31.770197+0000', 'last_fullsized': '2024-03-06T07:28:31.770197+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.24 is not active+clean 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771951+0000', 'last_change': '2024-03-06T07:28:31.771951+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771951+0000', 'last_undegraded': '2024-03-06T07:28:31.771951+0000', 'last_fullsized': '2024-03-06T07:28:31.771951+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.23 is not active+clean 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771671+0000', 'last_change': '2024-03-06T07:28:31.771671+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771671+0000', 'last_undegraded': '2024-03-06T07:28:31.771671+0000', 'last_fullsized': '2024-03-06T07:28:31.771671+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.22 is not active+clean 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.770827+0000', 'last_change': '2024-03-06T07:28:31.770827+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.770827+0000', 'last_undegraded': '2024-03-06T07:28:31.770827+0000', 'last_fullsized': '2024-03-06T07:28:31.770827+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.21 is not active+clean 2024-03-06T07:28:36.439 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771788+0000', 'last_change': '2024-03-06T07:28:31.771788+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771788+0000', 'last_undegraded': '2024-03-06T07:28:31.771788+0000', 'last_fullsized': '2024-03-06T07:28:31.771788+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.20 is not active+clean 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768199+0000', 'last_change': '2024-03-06T07:28:31.768199+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768199+0000', 'last_undegraded': '2024-03-06T07:28:31.768199+0000', 'last_fullsized': '2024-03-06T07:28:31.768199+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1e is not active+clean 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.776999+0000', 'last_change': '2024-03-06T07:28:31.776999+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.776999+0000', 'last_undegraded': '2024-03-06T07:28:31.776999+0000', 'last_fullsized': '2024-03-06T07:28:31.776999+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1f is not active+clean 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.767260+0000', 'last_change': '2024-03-06T07:28:31.767260+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.767260+0000', 'last_undegraded': '2024-03-06T07:28:31.767260+0000', 'last_fullsized': '2024-03-06T07:28:31.767260+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1c is not active+clean 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.774697+0000', 'last_change': '2024-03-06T07:28:31.774697+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.774697+0000', 'last_undegraded': '2024-03-06T07:28:31.774697+0000', 'last_fullsized': '2024-03-06T07:28:31.774697+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1d is not active+clean 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.773437+0000', 'last_change': '2024-03-06T07:28:31.773437+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.773437+0000', 'last_undegraded': '2024-03-06T07:28:31.773437+0000', 'last_fullsized': '2024-03-06T07:28:31.773437+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.440 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1a is not active+clean 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.774574+0000', 'last_change': '2024-03-06T07:28:31.774574+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.774574+0000', 'last_undegraded': '2024-03-06T07:28:31.774574+0000', 'last_fullsized': '2024-03-06T07:28:31.774574+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1b is not active+clean 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768260+0000', 'last_change': '2024-03-06T07:28:31.768260+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768260+0000', 'last_undegraded': '2024-03-06T07:28:31.768260+0000', 'last_fullsized': '2024-03-06T07:28:31.768260+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.18 is not active+clean 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768166+0000', 'last_change': '2024-03-06T07:28:31.768166+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768166+0000', 'last_undegraded': '2024-03-06T07:28:31.768166+0000', 'last_fullsized': '2024-03-06T07:28:31.768166+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.19 is not active+clean 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.775054+0000', 'last_change': '2024-03-06T07:28:31.775054+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.775054+0000', 'last_undegraded': '2024-03-06T07:28:31.775054+0000', 'last_fullsized': '2024-03-06T07:28:31.775054+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.16 is not active+clean 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.770026+0000', 'last_change': '2024-03-06T07:28:31.770026+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.770026+0000', 'last_undegraded': '2024-03-06T07:28:31.770026+0000', 'last_fullsized': '2024-03-06T07:28:31.770026+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.17 is not active+clean 2024-03-06T07:28:36.441 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.776092+0000', 'last_change': '2024-03-06T07:28:31.776092+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.776092+0000', 'last_undegraded': '2024-03-06T07:28:31.776092+0000', 'last_fullsized': '2024-03-06T07:28:31.776092+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.14 is not active+clean 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.773622+0000', 'last_change': '2024-03-06T07:28:31.773622+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.773622+0000', 'last_undegraded': '2024-03-06T07:28:31.773622+0000', 'last_fullsized': '2024-03-06T07:28:31.773622+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.15 is not active+clean 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.773035+0000', 'last_change': '2024-03-06T07:28:31.773035+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.773035+0000', 'last_undegraded': '2024-03-06T07:28:31.773035+0000', 'last_fullsized': '2024-03-06T07:28:31.773035+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.13 is not active+clean 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771283+0000', 'last_change': '2024-03-06T07:28:31.771283+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771283+0000', 'last_undegraded': '2024-03-06T07:28:31.771283+0000', 'last_fullsized': '2024-03-06T07:28:31.771283+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.10 is not active+clean 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.772244+0000', 'last_change': '2024-03-06T07:28:31.772244+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.772244+0000', 'last_undegraded': '2024-03-06T07:28:31.772244+0000', 'last_fullsized': '2024-03-06T07:28:31.772244+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.11 is not active+clean 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.769329+0000', 'last_change': '2024-03-06T07:28:31.769329+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.769329+0000', 'last_undegraded': '2024-03-06T07:28:31.769329+0000', 'last_fullsized': '2024-03-06T07:28:31.769329+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.e is not active+clean 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768865+0000', 'last_change': '2024-03-06T07:28:31.768865+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768865+0000', 'last_undegraded': '2024-03-06T07:28:31.768865+0000', 'last_fullsized': '2024-03-06T07:28:31.768865+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.f is not active+clean 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.775650+0000', 'last_change': '2024-03-06T07:28:31.775650+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.775650+0000', 'last_undegraded': '2024-03-06T07:28:31.775650+0000', 'last_fullsized': '2024-03-06T07:28:31.775650+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.c is not active+clean 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.773859+0000', 'last_change': '2024-03-06T07:28:31.773859+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.773859+0000', 'last_undegraded': '2024-03-06T07:28:31.773859+0000', 'last_fullsized': '2024-03-06T07:28:31.773859+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.d is not active+clean 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.772257+0000', 'last_change': '2024-03-06T07:28:31.772257+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.772257+0000', 'last_undegraded': '2024-03-06T07:28:31.772257+0000', 'last_fullsized': '2024-03-06T07:28:31.772257+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.a is not active+clean 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.770386+0000', 'last_change': '2024-03-06T07:28:31.770386+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.770386+0000', 'last_undegraded': '2024-03-06T07:28:31.770386+0000', 'last_fullsized': '2024-03-06T07:28:31.770386+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.b is not active+clean 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.767719+0000', 'last_change': '2024-03-06T07:28:31.767719+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.767719+0000', 'last_undegraded': '2024-03-06T07:28:31.767719+0000', 'last_fullsized': '2024-03-06T07:28:31.767719+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:36.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.8 is not active+clean 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771692+0000', 'last_change': '2024-03-06T07:28:31.771692+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771692+0000', 'last_undegraded': '2024-03-06T07:28:31.771692+0000', 'last_fullsized': '2024-03-06T07:28:31.771692+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.9 is not active+clean 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.774033+0000', 'last_change': '2024-03-06T07:28:31.774033+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.774033+0000', 'last_undegraded': '2024-03-06T07:28:31.774033+0000', 'last_fullsized': '2024-03-06T07:28:31.774033+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.6 is not active+clean 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.767571+0000', 'last_change': '2024-03-06T07:28:31.767571+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.767571+0000', 'last_undegraded': '2024-03-06T07:28:31.767571+0000', 'last_fullsized': '2024-03-06T07:28:31.767571+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.7 is not active+clean 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771368+0000', 'last_change': '2024-03-06T07:28:31.771368+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771368+0000', 'last_undegraded': '2024-03-06T07:28:31.771368+0000', 'last_fullsized': '2024-03-06T07:28:31.771368+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.4 is not active+clean 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771504+0000', 'last_change': '2024-03-06T07:28:31.771504+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771504+0000', 'last_undegraded': '2024-03-06T07:28:31.771504+0000', 'last_fullsized': '2024-03-06T07:28:31.771504+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.5 is not active+clean 2024-03-06T07:28:36.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.768950+0000', 'last_change': '2024-03-06T07:28:31.768950+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.768950+0000', 'last_undegraded': '2024-03-06T07:28:31.768950+0000', 'last_fullsized': '2024-03-06T07:28:31.768950+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2 is not active+clean 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771353+0000', 'last_change': '2024-03-06T07:28:31.771353+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771353+0000', 'last_undegraded': '2024-03-06T07:28:31.771353+0000', 'last_fullsized': '2024-03-06T07:28:31.771353+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3 is not active+clean 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.774264+0000', 'last_change': '2024-03-06T07:28:31.774264+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.774264+0000', 'last_undegraded': '2024-03-06T07:28:31.774264+0000', 'last_fullsized': '2024-03-06T07:28:31.774264+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.0 is not active+clean 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.773280+0000', 'last_change': '2024-03-06T07:28:31.773280+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.773280+0000', 'last_undegraded': '2024-03-06T07:28:31.773280+0000', 'last_fullsized': '2024-03-06T07:28:31.773280+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1 is not active+clean 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.769436+0000', 'last_change': '2024-03-06T07:28:31.769436+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.769436+0000', 'last_undegraded': '2024-03-06T07:28:31.769436+0000', 'last_fullsized': '2024-03-06T07:28:31.769436+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.12 is not active+clean 2024-03-06T07:28:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:31.771848+0000', 'last_change': '2024-03-06T07:28:31.771848+0000', 'last_active': '2024-03-06T07:28:31.761490+0000', 'last_peered': '2024-03-06T07:28:31.761490+0000', 'last_clean': '2024-03-06T07:28:31.761490+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:31.771848+0000', 'last_undegraded': '2024-03-06T07:28:31.771848+0000', 'last_fullsized': '2024-03-06T07:28:31.771848+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:31.761490+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:36.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:28:36.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.566+0000 7fadf8bbc640 1 Processor -- start 2024-03-06T07:28:36.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.566+0000 7fadf8bbc640 1 -- start start 2024-03-06T07:28:36.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf8bbc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fadf41444c0 0x7fadf406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf8bbc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fadf415b2d0 0x7fadf40708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf8bbc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf406d370 con 0x7fadf4070e10 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf8bbc640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf41448a0 con 0x7fadf415b2d0 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf8bbc640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf4071260 con 0x7fadf41444c0 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf1d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fadf415b2d0 0x7fadf40708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf2d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fadf4070e10 0x7fadf415e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54042/0 (socket says 172.21.15.77:54042) 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf2d9d640 1 -- 172.21.15.77:0/3142252393 learned_addr learned my addr 172.21.15.77:0/3142252393 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf259c640 1 --2- 172.21.15.77:0/3142252393 >> v2:172.21.15.158:3300/0 conn(0x7fadf41444c0 0x7fadf406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf1d9b640 1 -- 172.21.15.77:0/3142252393 >> v2:172.21.15.158:3300/0 conn(0x7fadf41444c0 msgr2=0x7fadf406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf1d9b640 1 --2- 172.21.15.77:0/3142252393 >> v2:172.21.15.158:3300/0 conn(0x7fadf41444c0 0x7fadf406ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf1d9b640 1 -- 172.21.15.77:0/3142252393 >> v1:172.21.15.77:6789/0 conn(0x7fadf4070e10 legacy=0x7fadf415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf1d9b640 1 -- 172.21.15.77:0/3142252393 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf406d650 con 0x7fadf415b2d0 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.567+0000 7fadf159a640 1 -- 172.21.15.77:0/3142252393 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686927048 0 0) 0x7fadf406d370 con 0x7fadf4070e10 2024-03-06T07:28:36.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.568+0000 7fadf1d9b640 1 --2- 172.21.15.77:0/3142252393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf415b2d0 0x7fadf40708d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fadf406ecb0 tx=0x7fade802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d75d339cf40867 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.568+0000 7fadf159a640 1 -- 172.21.15.77:0/3142252393 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8035020 con 0x7fadf415b2d0 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.568+0000 7fadf159a640 1 -- 172.21.15.77:0/3142252393 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade8031830 con 0x7fadf415b2d0 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.568+0000 7fadf159a640 1 -- 172.21.15.77:0/3142252393 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8031c00 con 0x7fadf415b2d0 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.569+0000 7fadf8bbc640 1 -- 172.21.15.77:0/3142252393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf415b2d0 msgr2=0x7fadf40708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.569+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/3142252393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf415b2d0 0x7fadf40708d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fadf406ecb0 tx=0x7fade802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.569+0000 7fadf8bbc640 1 -- 172.21.15.77:0/3142252393 shutdown_connections 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.569+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/3142252393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf415b2d0 0x7fadf40708d0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.569+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/3142252393 >> v2:172.21.15.158:3300/0 conn(0x7fadf41444c0 0x7fadf406ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.569+0000 7fadf8bbc640 1 -- 172.21.15.77:0/3142252393 >> 172.21.15.77:0/3142252393 conn(0x7fadf40b1c80 msgr2=0x7fadf40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.569+0000 7fadf8bbc640 1 -- 172.21.15.77:0/3142252393 shutdown_connections 2024-03-06T07:28:36.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.569+0000 7fadf8bbc640 1 -- 172.21.15.77:0/3142252393 wait complete. 2024-03-06T07:28:36.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.570+0000 7fadf8bbc640 1 Processor -- start 2024-03-06T07:28:36.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.570+0000 7fadf8bbc640 1 -- start start 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf8bbc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf4070e10 0x7fadf41f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf8bbc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadf41444c0 0x7fadf40b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf8bbc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadf415b2d0 0x7fadf40b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf8bbc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf406def0 con 0x7fadf4070e10 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf8bbc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf406dc10 con 0x7fadf41444c0 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf8bbc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf406cf40 con 0x7fadf415b2d0 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf4070e10 0x7fadf41f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf4070e10 0x7fadf41f36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40648/0 (socket says 172.21.15.77:40648) 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf259c640 1 -- 172.21.15.77:0/1163167161 learned_addr learned my addr 172.21.15.77:0/1163167161 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf1d9b640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadf41444c0 0x7fadf40b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf2d9d640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadf415b2d0 0x7fadf40b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf259c640 1 -- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadf415b2d0 msgr2=0x7fadf40b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf259c640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadf415b2d0 0x7fadf40b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf259c640 1 -- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadf41444c0 msgr2=0x7fadf40b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf259c640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadf41444c0 0x7fadf40b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf259c640 1 -- 172.21.15.77:0/1163167161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf413df90 con 0x7fadf4070e10 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.571+0000 7fadf2d9d640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadf415b2d0 0x7fadf40b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.572+0000 7fadf259c640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf4070e10 0x7fadf41f36a0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fade4010f50 tx=0x7fade400e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.572+0000 7fade2ffd640 1 -- 172.21.15.77:0/1163167161 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4012840 con 0x7fadf4070e10 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.572+0000 7fade2ffd640 1 -- 172.21.15.77:0/1163167161 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade40129e0 con 0x7fadf4070e10 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.572+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf413e270 con 0x7fadf4070e10 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.572+0000 7fade2ffd640 1 -- 172.21.15.77:0/1163167161 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4012db0 con 0x7fadf4070e10 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.572+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf4139840 con 0x7fadf4070e10 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.573+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf404f330 con 0x7fadf4070e10 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.573+0000 7fade2ffd640 1 -- 172.21.15.77:0/1163167161 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fade4021020 con 0x7fadf4070e10 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.574+0000 7fade2ffd640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fadd406b4d0 0x7fadd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.574+0000 7fade2ffd640 1 -- 172.21.15.77:0/1163167161 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fade408d0a0 con 0x7fadf4070e10 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.574+0000 7fadf1d9b640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fadd406b4d0 0x7fadd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.574+0000 7fadf1d9b640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fadd406b4d0 0x7fadd406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fade8036000 tx=0x7fade803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.578+0000 7fade2ffd640 1 -- 172.21.15.77:0/1163167161 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fade4054e10 con 0x7fadf4070e10 2024-03-06T07:28:36.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.814+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fadf406ca90 con 0x7fadf4070e10 2024-03-06T07:28:36.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.814+0000 7fade2ffd640 1 -- 172.21.15.77:0/1163167161 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1196 (secure 0 0 0) 0x7fade405bfb0 con 0x7fadf4070e10 2024-03-06T07:28:36.816 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:36.817 INFO:teuthology.orchestra.run.smithi077.stdout:{"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":415,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":125,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709711,"num_in_osds":8,"osd_in_since":1709709703,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":311496704,"bytes_avail":772782616576,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":80,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"b","status":"up:active","gid":8547}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-06T07:28:21.453757+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:28:36.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fadd406b4d0 msgr2=0x7fadd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fadd406b4d0 0x7fadd406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fade8036000 tx=0x7fade803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf4070e10 msgr2=0x7fadf41f36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf4070e10 0x7fadf41f36a0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fade4010f50 tx=0x7fade400e760 comp rx=0 tx=0).stop 2024-03-06T07:28:36.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 shutdown_connections 2024-03-06T07:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fadf415b2d0 0x7fadf40b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fadf41444c0 0x7fadf40b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fadd406b4d0 0x7fadd406d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 --2- 172.21.15.77:0/1163167161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadf4070e10 0x7fadf41f36a0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 >> 172.21.15.77:0/1163167161 conn(0x7fadf40b1c80 msgr2=0x7fadf40cf5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 shutdown_connections 2024-03-06T07:28:36.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.816+0000 7fadf8bbc640 1 -- 172.21.15.77:0/1163167161 wait complete. 2024-03-06T07:28:36.829 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:28:36.830 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.928+0000 7f7598b7a640 1 Processor -- start 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.928+0000 7f7598b7a640 1 -- start start 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f7598b7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f759406da40 0x7f75940cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f7598b7a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f75940cd7a0 0x7f759415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f7598b7a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75940cc980 con 0x7f75941424f0 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f7598b7a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75940cd220 con 0x7f75940cd7a0 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f7598b7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75940ccc60 con 0x7f759406da40 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f759259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f75941424f0 0x7f759406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54044/0 (socket says 172.21.15.77:54044) 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f759259c640 1 -- 172.21.15.77:0/3141861330 learned_addr learned my addr 172.21.15.77:0/3141861330 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f7591d9b640 1 --2- 172.21.15.77:0/3141861330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f759406da40 0x7f75940cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f7592d9d640 1 --2- 172.21.15.77:0/3141861330 >> v2:172.21.15.158:3300/0 conn(0x7f75940cd7a0 0x7f759415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.929+0000 7f759159a640 1 -- 172.21.15.77:0/3141861330 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579300710 0 0) 0x7f75940cc980 con 0x7f75941424f0 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.930+0000 7f759159a640 1 -- 172.21.15.77:0/3141861330 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757c003520 con 0x7f75941424f0 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.930+0000 7f7591d9b640 1 -- 172.21.15.77:0/3141861330 >> v2:172.21.15.158:3300/0 conn(0x7f75940cd7a0 msgr2=0x7f759415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.930+0000 7f7591d9b640 1 --2- 172.21.15.77:0/3141861330 >> v2:172.21.15.158:3300/0 conn(0x7f75940cd7a0 0x7f759415cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.930+0000 7f7591d9b640 1 -- 172.21.15.77:0/3141861330 >> v1:172.21.15.77:6789/0 conn(0x7f75941424f0 legacy=0x7f759406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.930+0000 7f7591d9b640 1 -- 172.21.15.77:0/3141861330 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940ccf40 con 0x7f759406da40 2024-03-06T07:28:36.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.930+0000 7f7592d9d640 1 --2- 172.21.15.77:0/3141861330 >> v2:172.21.15.158:3300/0 conn(0x7f75940cd7a0 0x7f759415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.930+0000 7f7591d9b640 1 --2- 172.21.15.77:0/3141861330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f759406da40 0x7f75940cc2f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f75880097a0 tx=0x7f758802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e8b402e164037d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f759159a640 1 -- 172.21.15.77:0/3141861330 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588035020 con 0x7f759406da40 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f759159a640 1 -- 172.21.15.77:0/3141861330 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7588031830 con 0x7f759406da40 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f759159a640 1 -- 172.21.15.77:0/3141861330 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588031be0 con 0x7f759406da40 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f7598b7a640 1 -- 172.21.15.77:0/3141861330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f759406da40 msgr2=0x7f75940cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f7598b7a640 1 --2- 172.21.15.77:0/3141861330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f759406da40 0x7f75940cc2f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f75880097a0 tx=0x7f758802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f7598b7a640 1 -- 172.21.15.77:0/3141861330 shutdown_connections 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f7598b7a640 1 --2- 172.21.15.77:0/3141861330 >> v2:172.21.15.158:3300/0 conn(0x7f75940cd7a0 0x7f759415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f7598b7a640 1 --2- 172.21.15.77:0/3141861330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f759406da40 0x7f75940cc2f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f7598b7a640 1 -- 172.21.15.77:0/3141861330 >> 172.21.15.77:0/3141861330 conn(0x7f75940cf8a0 msgr2=0x7f75940cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f7598b7a640 1 -- 172.21.15.77:0/3141861330 shutdown_connections 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.931+0000 7f7598b7a640 1 -- 172.21.15.77:0/3141861330 wait complete. 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7598b7a640 1 Processor -- start 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7598b7a640 1 -- start start 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7598b7a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f759406da40 0x7f759414ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7598b7a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75940cd7a0 0x7f759413e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7598b7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75941424f0 0x7f759413d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7598b7a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594071ce0 con 0x7f75941424f0 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7598b7a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594071b60 con 0x7f759406da40 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7598b7a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7594071e60 con 0x7f75940cd7a0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7592d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75941424f0 0x7f759413d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7592d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75941424f0 0x7f759413d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40654/0 (socket says 172.21.15.77:40654) 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7592d9d640 1 -- 172.21.15.77:0/601187293 learned_addr learned my addr 172.21.15.77:0/601187293 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f759259c640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f759406da40 0x7f759414ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.933+0000 7f7591d9b640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75940cd7a0 0x7f759413e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7592d9d640 1 -- 172.21.15.77:0/601187293 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75940cd7a0 msgr2=0x7f759413e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7592d9d640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75940cd7a0 0x7f759413e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7592d9d640 1 -- 172.21.15.77:0/601187293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f759406da40 msgr2=0x7f759414ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7592d9d640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f759406da40 0x7f759414ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7592d9d640 1 -- 172.21.15.77:0/601187293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759413c460 con 0x7f75941424f0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7592d9d640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75941424f0 0x7f759413d500 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f758c00e930 tx=0x7f758c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7582ffd640 1 -- 172.21.15.77:0/601187293 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c017070 con 0x7f75941424f0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7582ffd640 1 -- 172.21.15.77:0/601187293 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c00ca00 con 0x7f75941424f0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7582ffd640 1 -- 172.21.15.77:0/601187293 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c00cdd0 con 0x7f75941424f0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759413c740 con 0x7f75941424f0 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.934+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759413cd00 con 0x7f75941424f0 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.935+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759404f330 con 0x7f75941424f0 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.935+0000 7f7582ffd640 1 -- 172.21.15.77:0/601187293 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f758c013470 con 0x7f75941424f0 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.936+0000 7f7582ffd640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f757406b4d0 0x7f757406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.936+0000 7f759259c640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f757406b4d0 0x7f757406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.936+0000 7f7582ffd640 1 -- 172.21.15.77:0/601187293 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f758c08a090 con 0x7f75941424f0 2024-03-06T07:28:36.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.936+0000 7f759259c640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f757406b4d0 0x7f757406d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7584005c00 tx=0x7f7584009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:36.940+0000 7f7582ffd640 1 -- 172.21.15.77:0/601187293 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f758c058900 con 0x7f75941424f0 2024-03-06T07:28:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.107+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7594145b30 con 0x7f757406b4d0 2024-03-06T07:28:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.112+0000 7f7582ffd640 1 -- 172.21.15.77:0/601187293 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247089 (secure 0 0 0) 0x7f7594145b30 con 0x7f757406b4d0 2024-03-06T07:28:37.117 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:37.118 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:28:37.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f757406b4d0 msgr2=0x7f757406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f757406b4d0 0x7f757406d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7584005c00 tx=0x7f7584009040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75941424f0 msgr2=0x7f759413d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75941424f0 0x7f759413d500 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f758c00e930 tx=0x7f758c00edf0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 shutdown_connections 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f757406b4d0 0x7f757406d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75941424f0 0x7f759413d500 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f75940cd7a0 0x7f759413e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 --2- 172.21.15.77:0/601187293 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f759406da40 0x7f759414ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.115+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 >> 172.21.15.77:0/601187293 conn(0x7f75940cf8a0 msgr2=0x7f759415d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.116+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 shutdown_connections 2024-03-06T07:28:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:37.116+0000 7f7598b7a640 1 -- 172.21.15.77:0/601187293 wait complete. 2024-03-06T07:28:37.128 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-03-06T07:28:35.455923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304196,"kb_used_data":13136,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754670524,"statfs":{"total":773094113280,"available":772782616576,"internally_reserved":0,"allocated":13451264,"data_stored":6501970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.693890"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770447+0000","last_change":"2024-03-06T07:28:31.770447+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770447+0000","last_undegraded":"2024-03-06T07:28:31.770447+0000","last_fullsized":"2024-03-06T07:28:31.770447+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775687+0000","last_change":"2024-03-06T07:28:31.775687+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775687+0000","last_undegraded":"2024-03-06T07:28:31.775687+0000","last_fullsized":"2024-03-06T07:28:31.775687+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768247+0000","last_change":"2024-03-06T07:28:31.768247+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768247+0000","last_undegraded":"2024-03-06T07:28:31.768247+0000","last_fullsized":"2024-03-06T07:28:31.768247+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770523+0000","last_change":"2024-03-06T07:28:31.770523+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770523+0000","last_undegraded":"2024-03-06T07:28:31.770523+0000","last_fullsized":"2024-03-06T07:28:31.770523+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772284+0000","last_change":"2024-03-06T07:28:31.772284+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772284+0000","last_undegraded":"2024-03-06T07:28:31.772284+0000","last_fullsized":"2024-03-06T07:28:31.772284+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767124+0000","last_change":"2024-03-06T07:28:31.767124+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767124+0000","last_undegraded":"2024-03-06T07:28:31.767124+0000","last_fullsized":"2024-03-06T07:28:31.767124+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770134+0000","last_change":"2024-03-06T07:28:31.770134+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770134+0000","last_undegraded":"2024-03-06T07:28:31.770134+0000","last_fullsized":"2024-03-06T07:28:31.770134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768838+0000","last_change":"2024-03-06T07:28:31.768838+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768838+0000","last_undegraded":"2024-03-06T07:28:31.768838+0000","last_fullsized":"2024-03-06T07:28:31.768838+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767826+0000","last_change":"2024-03-06T07:28:31.767826+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767826+0000","last_undegraded":"2024-03-06T07:28:31.767826+0000","last_fullsized":"2024-03-06T07:28:31.767826+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775301+0000","last_change":"2024-03-06T07:28:31.775301+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775301+0000","last_undegraded":"2024-03-06T07:28:31.775301+0000","last_fullsized":"2024-03-06T07:28:31.775301+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769962+0000","last_change":"2024-03-06T07:28:31.769962+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769962+0000","last_undegraded":"2024-03-06T07:28:31.769962+0000","last_fullsized":"2024-03-06T07:28:31.769962+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768517+0000","last_change":"2024-03-06T07:28:31.768517+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768517+0000","last_undegraded":"2024-03-06T07:28:31.768517+0000","last_fullsized":"2024-03-06T07:28:31.768517+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772112+0000","last_change":"2024-03-06T07:28:31.772112+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772112+0000","last_undegraded":"2024-03-06T07:28:31.772112+0000","last_fullsized":"2024-03-06T07:28:31.772112+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775114+0000","last_change":"2024-03-06T07:28:31.775114+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775114+0000","last_undegraded":"2024-03-06T07:28:31.775114+0000","last_fullsized":"2024-03-06T07:28:31.775114+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769966+0000","last_change":"2024-03-06T07:28:31.769966+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769966+0000","last_undegraded":"2024-03-06T07:28:31.769966+0000","last_fullsized":"2024-03-06T07:28:31.769966+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773730+0000","last_change":"2024-03-06T07:28:31.773730+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773730+0000","last_undegraded":"2024-03-06T07:28:31.773730+0000","last_fullsized":"2024-03-06T07:28:31.773730+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767225+0000","last_change":"2024-03-06T07:28:31.767225+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767225+0000","last_undegraded":"2024-03-06T07:28:31.767225+0000","last_fullsized":"2024-03-06T07:28:31.767225+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.777806+0000","last_change":"2024-03-06T07:28:31.777806+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.777806+0000","last_undegraded":"2024-03-06T07:28:31.777806+0000","last_fullsized":"2024-03-06T07:28:31.777806+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769133+0000","last_change":"2024-03-06T07:28:31.769133+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769133+0000","last_undegraded":"2024-03-06T07:28:31.769133+0000","last_fullsized":"2024-03-06T07:28:31.769133+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774231+0000","last_change":"2024-03-06T07:28:31.774231+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774231+0000","last_undegraded":"2024-03-06T07:28:31.774231+0000","last_fullsized":"2024-03-06T07:28:31.774231+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774487+0000","last_change":"2024-03-06T07:28:31.774487+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774487+0000","last_undegraded":"2024-03-06T07:28:31.774487+0000","last_fullsized":"2024-03-06T07:28:31.774487+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769129+0000","last_change":"2024-03-06T07:28:31.769129+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769129+0000","last_undegraded":"2024-03-06T07:28:31.769129+0000","last_fullsized":"2024-03-06T07:28:31.769129+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768187+0000","last_change":"2024-03-06T07:28:31.768187+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768187+0000","last_undegraded":"2024-03-06T07:28:31.768187+0000","last_fullsized":"2024-03-06T07:28:31.768187+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768712+0000","last_change":"2024-03-06T07:28:31.768712+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768712+0000","last_undegraded":"2024-03-06T07:28:31.768712+0000","last_fullsized":"2024-03-06T07:28:31.768712+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776050+0000","last_change":"2024-03-06T07:28:31.776050+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776050+0000","last_undegraded":"2024-03-06T07:28:31.776050+0000","last_fullsized":"2024-03-06T07:28:31.776050+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.777368+0000","last_change":"2024-03-06T07:28:31.777368+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.777368+0000","last_undegraded":"2024-03-06T07:28:31.777368+0000","last_fullsized":"2024-03-06T07:28:31.777368+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770197+0000","last_change":"2024-03-06T07:28:31.770197+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770197+0000","last_undegraded":"2024-03-06T07:28:31.770197+0000","last_fullsized":"2024-03-06T07:28:31.770197+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771951+0000","last_change":"2024-03-06T07:28:31.771951+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771951+0000","last_undegraded":"2024-03-06T07:28:31.771951+0000","last_fullsized":"2024-03-06T07:28:31.771951+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771671+0000","last_change":"2024-03-06T07:28:31.771671+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771671+0000","last_undegraded":"2024-03-06T07:28:31.771671+0000","last_fullsized":"2024-03-06T07:28:31.771671+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770827+0000","last_change":"2024-03-06T07:28:31.770827+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770827+0000","last_undegraded":"2024-03-06T07:28:31.770827+0000","last_fullsized":"2024-03-06T07:28:31.770827+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771788+0000","last_change":"2024-03-06T07:28:31.771788+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771788+0000","last_undegraded":"2024-03-06T07:28:31.771788+0000","last_fullsized":"2024-03-06T07:28:31.771788+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768199+0000","last_change":"2024-03-06T07:28:31.768199+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768199+0000","last_undegraded":"2024-03-06T07:28:31.768199+0000","last_fullsized":"2024-03-06T07:28:31.768199+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776999+0000","last_change":"2024-03-06T07:28:31.776999+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776999+0000","last_undegraded":"2024-03-06T07:28:31.776999+0000","last_fullsized":"2024-03-06T07:28:31.776999+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783927+0000","last_change":"2024-03-06T07:28:31.782852+0000","last_active":"2024-03-06T07:28:31.783927+0000","last_peered":"2024-03-06T07:28:31.783927+0000","last_clean":"2024-03-06T07:28:31.783927+0000","last_became_active":"2024-03-06T07:28:31.782575+0000","last_became_peered":"2024-03-06T07:28:31.782575+0000","last_unstale":"2024-03-06T07:28:31.783927+0000","last_undegraded":"2024-03-06T07:28:31.783927+0000","last_fullsized":"2024-03-06T07:28:31.783927+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:30.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767260+0000","last_change":"2024-03-06T07:28:31.767260+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767260+0000","last_undegraded":"2024-03-06T07:28:31.767260+0000","last_fullsized":"2024-03-06T07:28:31.767260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783865+0000","last_change":"2024-03-06T07:28:31.782726+0000","last_active":"2024-03-06T07:28:31.783865+0000","last_peered":"2024-03-06T07:28:31.783865+0000","last_clean":"2024-03-06T07:28:31.783865+0000","last_became_active":"2024-03-06T07:28:31.782447+0000","last_became_peered":"2024-03-06T07:28:31.782447+0000","last_unstale":"2024-03-06T07:28:31.783865+0000","last_undegraded":"2024-03-06T07:28:31.783865+0000","last_fullsized":"2024-03-06T07:28:31.783865+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:40.079443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774697+0000","last_change":"2024-03-06T07:28:31.774697+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774697+0000","last_undegraded":"2024-03-06T07:28:31.774697+0000","last_fullsized":"2024-03-06T07:28:31.774697+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.785061+0000","last_change":"2024-03-06T07:28:31.783586+0000","last_active":"2024-03-06T07:28:31.785061+0000","last_peered":"2024-03-06T07:28:31.785061+0000","last_clean":"2024-03-06T07:28:31.785061+0000","last_became_active":"2024-03-06T07:28:31.783329+0000","last_became_peered":"2024-03-06T07:28:31.783329+0000","last_unstale":"2024-03-06T07:28:31.785061+0000","last_undegraded":"2024-03-06T07:28:31.785061+0000","last_fullsized":"2024-03-06T07:28:31.785061+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:07.918185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773437+0000","last_change":"2024-03-06T07:28:31.773437+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773437+0000","last_undegraded":"2024-03-06T07:28:31.773437+0000","last_fullsized":"2024-03-06T07:28:31.773437+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782314+0000","last_change":"2024-03-06T07:28:31.780002+0000","last_active":"2024-03-06T07:28:31.782314+0000","last_peered":"2024-03-06T07:28:31.782314+0000","last_clean":"2024-03-06T07:28:31.782314+0000","last_became_active":"2024-03-06T07:28:31.779736+0000","last_became_peered":"2024-03-06T07:28:31.779736+0000","last_unstale":"2024-03-06T07:28:31.782314+0000","last_undegraded":"2024-03-06T07:28:31.782314+0000","last_fullsized":"2024-03-06T07:28:31.782314+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:37.028128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774574+0000","last_change":"2024-03-06T07:28:31.774574+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774574+0000","last_undegraded":"2024-03-06T07:28:31.774574+0000","last_fullsized":"2024-03-06T07:28:31.774574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784714+0000","last_change":"2024-03-06T07:28:31.783109+0000","last_active":"2024-03-06T07:28:31.784714+0000","last_peered":"2024-03-06T07:28:31.784714+0000","last_clean":"2024-03-06T07:28:31.784714+0000","last_became_active":"2024-03-06T07:28:31.782858+0000","last_became_peered":"2024-03-06T07:28:31.782858+0000","last_unstale":"2024-03-06T07:28:31.784714+0000","last_undegraded":"2024-03-06T07:28:31.784714+0000","last_fullsized":"2024-03-06T07:28:31.784714+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:41.710642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768260+0000","last_change":"2024-03-06T07:28:31.768260+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768260+0000","last_undegraded":"2024-03-06T07:28:31.768260+0000","last_fullsized":"2024-03-06T07:28:31.768260+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783299+0000","last_change":"2024-03-06T07:28:31.781824+0000","last_active":"2024-03-06T07:28:31.783299+0000","last_peered":"2024-03-06T07:28:31.783299+0000","last_clean":"2024-03-06T07:28:31.783299+0000","last_became_active":"2024-03-06T07:28:31.781484+0000","last_became_peered":"2024-03-06T07:28:31.781484+0000","last_unstale":"2024-03-06T07:28:31.783299+0000","last_undegraded":"2024-03-06T07:28:31.783299+0000","last_fullsized":"2024-03-06T07:28:31.783299+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:59.115102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768166+0000","last_change":"2024-03-06T07:28:31.768166+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768166+0000","last_undegraded":"2024-03-06T07:28:31.768166+0000","last_fullsized":"2024-03-06T07:28:31.768166+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779805+0000","last_change":"2024-03-06T07:28:31.777579+0000","last_active":"2024-03-06T07:28:31.779805+0000","last_peered":"2024-03-06T07:28:31.779805+0000","last_clean":"2024-03-06T07:28:31.779805+0000","last_became_active":"2024-03-06T07:28:31.777290+0000","last_became_peered":"2024-03-06T07:28:31.777290+0000","last_unstale":"2024-03-06T07:28:31.779805+0000","last_undegraded":"2024-03-06T07:28:31.779805+0000","last_fullsized":"2024-03-06T07:28:31.779805+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:48.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775054+0000","last_change":"2024-03-06T07:28:31.775054+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775054+0000","last_undegraded":"2024-03-06T07:28:31.775054+0000","last_fullsized":"2024-03-06T07:28:31.775054+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780903+0000","last_change":"2024-03-06T07:28:31.775485+0000","last_active":"2024-03-06T07:28:31.780903+0000","last_peered":"2024-03-06T07:28:31.780903+0000","last_clean":"2024-03-06T07:28:31.780903+0000","last_became_active":"2024-03-06T07:28:31.775212+0000","last_became_peered":"2024-03-06T07:28:31.775212+0000","last_unstale":"2024-03-06T07:28:31.780903+0000","last_undegraded":"2024-03-06T07:28:31.780903+0000","last_fullsized":"2024-03-06T07:28:31.780903+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:35.868436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770026+0000","last_change":"2024-03-06T07:28:31.770026+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770026+0000","last_undegraded":"2024-03-06T07:28:31.770026+0000","last_fullsized":"2024-03-06T07:28:31.770026+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784013+0000","last_change":"2024-03-06T07:28:31.783404+0000","last_active":"2024-03-06T07:28:31.784013+0000","last_peered":"2024-03-06T07:28:31.784013+0000","last_clean":"2024-03-06T07:28:31.784013+0000","last_became_active":"2024-03-06T07:28:31.782969+0000","last_became_peered":"2024-03-06T07:28:31.782969+0000","last_unstale":"2024-03-06T07:28:31.784013+0000","last_undegraded":"2024-03-06T07:28:31.784013+0000","last_fullsized":"2024-03-06T07:28:31.784013+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:06.016386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.776092+0000","last_change":"2024-03-06T07:28:31.776092+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.776092+0000","last_undegraded":"2024-03-06T07:28:31.776092+0000","last_fullsized":"2024-03-06T07:28:31.776092+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.784990+0000","last_change":"2024-03-06T07:28:31.784102+0000","last_active":"2024-03-06T07:28:31.784990+0000","last_peered":"2024-03-06T07:28:31.784990+0000","last_clean":"2024-03-06T07:28:31.784990+0000","last_became_active":"2024-03-06T07:28:31.783888+0000","last_became_peered":"2024-03-06T07:28:31.783888+0000","last_unstale":"2024-03-06T07:28:31.784990+0000","last_undegraded":"2024-03-06T07:28:31.784990+0000","last_fullsized":"2024-03-06T07:28:31.784990+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:56.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773622+0000","last_change":"2024-03-06T07:28:31.773622+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773622+0000","last_undegraded":"2024-03-06T07:28:31.773622+0000","last_fullsized":"2024-03-06T07:28:31.773622+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783836+0000","last_change":"2024-03-06T07:28:31.782822+0000","last_active":"2024-03-06T07:28:31.783836+0000","last_peered":"2024-03-06T07:28:31.783836+0000","last_clean":"2024-03-06T07:28:31.783836+0000","last_became_active":"2024-03-06T07:28:31.782523+0000","last_became_peered":"2024-03-06T07:28:31.782523+0000","last_unstale":"2024-03-06T07:28:31.783836+0000","last_undegraded":"2024-03-06T07:28:31.783836+0000","last_fullsized":"2024-03-06T07:28:31.783836+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:51.472938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773035+0000","last_change":"2024-03-06T07:28:31.773035+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773035+0000","last_undegraded":"2024-03-06T07:28:31.773035+0000","last_fullsized":"2024-03-06T07:28:31.773035+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779098+0000","last_change":"2024-03-06T07:28:31.776785+0000","last_active":"2024-03-06T07:28:31.779098+0000","last_peered":"2024-03-06T07:28:31.779098+0000","last_clean":"2024-03-06T07:28:31.779098+0000","last_became_active":"2024-03-06T07:28:31.776521+0000","last_became_peered":"2024-03-06T07:28:31.776521+0000","last_unstale":"2024-03-06T07:28:31.779098+0000","last_undegraded":"2024-03-06T07:28:31.779098+0000","last_fullsized":"2024-03-06T07:28:31.779098+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:58.794188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771283+0000","last_change":"2024-03-06T07:28:31.771283+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771283+0000","last_undegraded":"2024-03-06T07:28:31.771283+0000","last_fullsized":"2024-03-06T07:28:31.771283+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778422+0000","last_change":"2024-03-06T07:28:31.775867+0000","last_active":"2024-03-06T07:28:31.778422+0000","last_peered":"2024-03-06T07:28:31.778422+0000","last_clean":"2024-03-06T07:28:31.778422+0000","last_became_active":"2024-03-06T07:28:31.775622+0000","last_became_peered":"2024-03-06T07:28:31.775622+0000","last_unstale":"2024-03-06T07:28:31.778422+0000","last_undegraded":"2024-03-06T07:28:31.778422+0000","last_fullsized":"2024-03-06T07:28:31.778422+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:25.853619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772244+0000","last_change":"2024-03-06T07:28:31.772244+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772244+0000","last_undegraded":"2024-03-06T07:28:31.772244+0000","last_fullsized":"2024-03-06T07:28:31.772244+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780993+0000","last_change":"2024-03-06T07:28:31.777064+0000","last_active":"2024-03-06T07:28:31.780993+0000","last_peered":"2024-03-06T07:28:31.780993+0000","last_clean":"2024-03-06T07:28:31.780993+0000","last_became_active":"2024-03-06T07:28:31.776788+0000","last_became_peered":"2024-03-06T07:28:31.776788+0000","last_unstale":"2024-03-06T07:28:31.780993+0000","last_undegraded":"2024-03-06T07:28:31.780993+0000","last_fullsized":"2024-03-06T07:28:31.780993+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:41.214845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769329+0000","last_change":"2024-03-06T07:28:31.769329+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769329+0000","last_undegraded":"2024-03-06T07:28:31.769329+0000","last_fullsized":"2024-03-06T07:28:31.769329+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782969+0000","last_change":"2024-03-06T07:28:31.781673+0000","last_active":"2024-03-06T07:28:31.782969+0000","last_peered":"2024-03-06T07:28:31.782969+0000","last_clean":"2024-03-06T07:28:31.782969+0000","last_became_active":"2024-03-06T07:28:31.781383+0000","last_became_peered":"2024-03-06T07:28:31.781383+0000","last_unstale":"2024-03-06T07:28:31.782969+0000","last_undegraded":"2024-03-06T07:28:31.782969+0000","last_fullsized":"2024-03-06T07:28:31.782969+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:29.323269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768865+0000","last_change":"2024-03-06T07:28:31.768865+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768865+0000","last_undegraded":"2024-03-06T07:28:31.768865+0000","last_fullsized":"2024-03-06T07:28:31.768865+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782261+0000","last_change":"2024-03-06T07:28:31.780402+0000","last_active":"2024-03-06T07:28:31.782261+0000","last_peered":"2024-03-06T07:28:31.782261+0000","last_clean":"2024-03-06T07:28:31.782261+0000","last_became_active":"2024-03-06T07:28:31.780050+0000","last_became_peered":"2024-03-06T07:28:31.780050+0000","last_unstale":"2024-03-06T07:28:31.782261+0000","last_undegraded":"2024-03-06T07:28:31.782261+0000","last_fullsized":"2024-03-06T07:28:31.782261+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:37.538186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.775650+0000","last_change":"2024-03-06T07:28:31.775650+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.775650+0000","last_undegraded":"2024-03-06T07:28:31.775650+0000","last_fullsized":"2024-03-06T07:28:31.775650+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779815+0000","last_change":"2024-03-06T07:28:31.773233+0000","last_active":"2024-03-06T07:28:31.779815+0000","last_peered":"2024-03-06T07:28:31.779815+0000","last_clean":"2024-03-06T07:28:31.779815+0000","last_became_active":"2024-03-06T07:28:31.773028+0000","last_became_peered":"2024-03-06T07:28:31.773028+0000","last_unstale":"2024-03-06T07:28:31.779815+0000","last_undegraded":"2024-03-06T07:28:31.779815+0000","last_fullsized":"2024-03-06T07:28:31.779815+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:33.073077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773859+0000","last_change":"2024-03-06T07:28:31.773859+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773859+0000","last_undegraded":"2024-03-06T07:28:31.773859+0000","last_fullsized":"2024-03-06T07:28:31.773859+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783149+0000","last_change":"2024-03-06T07:28:31.782644+0000","last_active":"2024-03-06T07:28:31.783149+0000","last_peered":"2024-03-06T07:28:31.783149+0000","last_clean":"2024-03-06T07:28:31.783149+0000","last_became_active":"2024-03-06T07:28:31.782362+0000","last_became_peered":"2024-03-06T07:28:31.782362+0000","last_unstale":"2024-03-06T07:28:31.783149+0000","last_undegraded":"2024-03-06T07:28:31.783149+0000","last_fullsized":"2024-03-06T07:28:31.783149+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:05.062758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.772257+0000","last_change":"2024-03-06T07:28:31.772257+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.772257+0000","last_undegraded":"2024-03-06T07:28:31.772257+0000","last_fullsized":"2024-03-06T07:28:31.772257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783884+0000","last_change":"2024-03-06T07:28:31.783136+0000","last_active":"2024-03-06T07:28:31.783884+0000","last_peered":"2024-03-06T07:28:31.783884+0000","last_clean":"2024-03-06T07:28:31.783884+0000","last_became_active":"2024-03-06T07:28:31.782350+0000","last_became_peered":"2024-03-06T07:28:31.782350+0000","last_unstale":"2024-03-06T07:28:31.783884+0000","last_undegraded":"2024-03-06T07:28:31.783884+0000","last_fullsized":"2024-03-06T07:28:31.783884+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:14.779042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.770386+0000","last_change":"2024-03-06T07:28:31.770386+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.770386+0000","last_undegraded":"2024-03-06T07:28:31.770386+0000","last_fullsized":"2024-03-06T07:28:31.770386+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.782410+0000","last_change":"2024-03-06T07:28:31.779718+0000","last_active":"2024-03-06T07:28:31.782410+0000","last_peered":"2024-03-06T07:28:31.782410+0000","last_clean":"2024-03-06T07:28:31.782410+0000","last_became_active":"2024-03-06T07:28:31.779454+0000","last_became_peered":"2024-03-06T07:28:31.779454+0000","last_unstale":"2024-03-06T07:28:31.782410+0000","last_undegraded":"2024-03-06T07:28:31.782410+0000","last_fullsized":"2024-03-06T07:28:31.782410+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:56.378897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767719+0000","last_change":"2024-03-06T07:28:31.767719+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767719+0000","last_undegraded":"2024-03-06T07:28:31.767719+0000","last_fullsized":"2024-03-06T07:28:31.767719+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783278+0000","last_change":"2024-03-06T07:28:31.780216+0000","last_active":"2024-03-06T07:28:31.783278+0000","last_peered":"2024-03-06T07:28:31.783278+0000","last_clean":"2024-03-06T07:28:31.783278+0000","last_became_active":"2024-03-06T07:28:31.779950+0000","last_became_peered":"2024-03-06T07:28:31.779950+0000","last_unstale":"2024-03-06T07:28:31.783278+0000","last_undegraded":"2024-03-06T07:28:31.783278+0000","last_fullsized":"2024-03-06T07:28:31.783278+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:30.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771692+0000","last_change":"2024-03-06T07:28:31.771692+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771692+0000","last_undegraded":"2024-03-06T07:28:31.771692+0000","last_fullsized":"2024-03-06T07:28:31.771692+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.785702+0000","last_change":"2024-03-06T07:28:31.784524+0000","last_active":"2024-03-06T07:28:31.785702+0000","last_peered":"2024-03-06T07:28:31.785702+0000","last_clean":"2024-03-06T07:28:31.785702+0000","last_became_active":"2024-03-06T07:28:31.784259+0000","last_became_peered":"2024-03-06T07:28:31.784259+0000","last_unstale":"2024-03-06T07:28:31.785702+0000","last_undegraded":"2024-03-06T07:28:31.785702+0000","last_fullsized":"2024-03-06T07:28:31.785702+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:25.998178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774033+0000","last_change":"2024-03-06T07:28:31.774033+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774033+0000","last_undegraded":"2024-03-06T07:28:31.774033+0000","last_fullsized":"2024-03-06T07:28:31.774033+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778733+0000","last_change":"2024-03-06T07:28:31.772492+0000","last_active":"2024-03-06T07:28:31.778733+0000","last_peered":"2024-03-06T07:28:31.778733+0000","last_clean":"2024-03-06T07:28:31.778733+0000","last_became_active":"2024-03-06T07:28:31.772186+0000","last_became_peered":"2024-03-06T07:28:31.772186+0000","last_unstale":"2024-03-06T07:28:31.778733+0000","last_undegraded":"2024-03-06T07:28:31.778733+0000","last_fullsized":"2024-03-06T07:28:31.778733+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:18.698871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.767571+0000","last_change":"2024-03-06T07:28:31.767571+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.767571+0000","last_undegraded":"2024-03-06T07:28:31.767571+0000","last_fullsized":"2024-03-06T07:28:31.767571+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780832+0000","last_change":"2024-03-06T07:28:31.779645+0000","last_active":"2024-03-06T07:28:31.780832+0000","last_peered":"2024-03-06T07:28:31.780832+0000","last_clean":"2024-03-06T07:28:31.780832+0000","last_became_active":"2024-03-06T07:28:31.779391+0000","last_became_peered":"2024-03-06T07:28:31.779391+0000","last_unstale":"2024-03-06T07:28:31.780832+0000","last_undegraded":"2024-03-06T07:28:31.780832+0000","last_fullsized":"2024-03-06T07:28:31.780832+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:08.000434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771368+0000","last_change":"2024-03-06T07:28:31.771368+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771368+0000","last_undegraded":"2024-03-06T07:28:31.771368+0000","last_fullsized":"2024-03-06T07:28:31.771368+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781208+0000","last_change":"2024-03-06T07:28:31.779359+0000","last_active":"2024-03-06T07:28:31.781208+0000","last_peered":"2024-03-06T07:28:31.781208+0000","last_clean":"2024-03-06T07:28:31.781208+0000","last_became_active":"2024-03-06T07:28:31.779096+0000","last_became_peered":"2024-03-06T07:28:31.779096+0000","last_unstale":"2024-03-06T07:28:31.781208+0000","last_undegraded":"2024-03-06T07:28:31.781208+0000","last_fullsized":"2024-03-06T07:28:31.781208+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:47.517522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771504+0000","last_change":"2024-03-06T07:28:31.771504+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771504+0000","last_undegraded":"2024-03-06T07:28:31.771504+0000","last_fullsized":"2024-03-06T07:28:31.771504+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.779248+0000","last_change":"2024-03-06T07:28:31.778300+0000","last_active":"2024-03-06T07:28:31.779248+0000","last_peered":"2024-03-06T07:28:31.779248+0000","last_clean":"2024-03-06T07:28:31.779248+0000","last_became_active":"2024-03-06T07:28:31.778036+0000","last_became_peered":"2024-03-06T07:28:31.778036+0000","last_unstale":"2024-03-06T07:28:31.779248+0000","last_undegraded":"2024-03-06T07:28:31.779248+0000","last_fullsized":"2024-03-06T07:28:31.779248+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:50.687009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.768950+0000","last_change":"2024-03-06T07:28:31.768950+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.768950+0000","last_undegraded":"2024-03-06T07:28:31.768950+0000","last_fullsized":"2024-03-06T07:28:31.768950+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780689+0000","last_change":"2024-03-06T07:28:31.779932+0000","last_active":"2024-03-06T07:28:31.780689+0000","last_peered":"2024-03-06T07:28:31.780689+0000","last_clean":"2024-03-06T07:28:31.780689+0000","last_became_active":"2024-03-06T07:28:31.779668+0000","last_became_peered":"2024-03-06T07:28:31.779668+0000","last_unstale":"2024-03-06T07:28:31.780689+0000","last_undegraded":"2024-03-06T07:28:31.780689+0000","last_fullsized":"2024-03-06T07:28:31.780689+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:57.626735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771353+0000","last_change":"2024-03-06T07:28:31.771353+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771353+0000","last_undegraded":"2024-03-06T07:28:31.771353+0000","last_fullsized":"2024-03-06T07:28:31.771353+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.780358+0000","last_change":"2024-03-06T07:28:31.777451+0000","last_active":"2024-03-06T07:28:31.780358+0000","last_peered":"2024-03-06T07:28:31.780358+0000","last_clean":"2024-03-06T07:28:31.780358+0000","last_became_active":"2024-03-06T07:28:31.777201+0000","last_became_peered":"2024-03-06T07:28:31.777201+0000","last_unstale":"2024-03-06T07:28:31.780358+0000","last_undegraded":"2024-03-06T07:28:31.780358+0000","last_fullsized":"2024-03-06T07:28:31.780358+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:35.510219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.774264+0000","last_change":"2024-03-06T07:28:31.774264+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.774264+0000","last_undegraded":"2024-03-06T07:28:31.774264+0000","last_fullsized":"2024-03-06T07:28:31.774264+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781923+0000","last_change":"2024-03-06T07:28:31.781199+0000","last_active":"2024-03-06T07:28:31.781923+0000","last_peered":"2024-03-06T07:28:31.781923+0000","last_clean":"2024-03-06T07:28:31.781923+0000","last_became_active":"2024-03-06T07:28:31.780846+0000","last_became_peered":"2024-03-06T07:28:31.780846+0000","last_unstale":"2024-03-06T07:28:31.781923+0000","last_undegraded":"2024-03-06T07:28:31.781923+0000","last_fullsized":"2024-03-06T07:28:31.781923+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:13.183534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.773280+0000","last_change":"2024-03-06T07:28:31.773280+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.773280+0000","last_undegraded":"2024-03-06T07:28:31.773280+0000","last_fullsized":"2024-03-06T07:28:31.773280+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.783201+0000","last_change":"2024-03-06T07:28:31.780249+0000","last_active":"2024-03-06T07:28:31.783201+0000","last_peered":"2024-03-06T07:28:31.783201+0000","last_clean":"2024-03-06T07:28:31.783201+0000","last_became_active":"2024-03-06T07:28:31.779962+0000","last_became_peered":"2024-03-06T07:28:31.779962+0000","last_unstale":"2024-03-06T07:28:31.783201+0000","last_undegraded":"2024-03-06T07:28:31.783201+0000","last_fullsized":"2024-03-06T07:28:31.783201+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:19.219570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.769436+0000","last_change":"2024-03-06T07:28:31.769436+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.769436+0000","last_undegraded":"2024-03-06T07:28:31.769436+0000","last_fullsized":"2024-03-06T07:28:31.769436+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.778758+0000","last_change":"2024-03-06T07:28:31.773134+0000","last_active":"2024-03-06T07:28:31.778758+0000","last_peered":"2024-03-06T07:28:31.778758+0000","last_clean":"2024-03-06T07:28:31.778758+0000","last_became_active":"2024-03-06T07:28:31.772882+0000","last_became_peered":"2024-03-06T07:28:31.772882+0000","last_unstale":"2024-03-06T07:28:31.778758+0000","last_undegraded":"2024-03-06T07:28:31.778758+0000","last_fullsized":"2024-03-06T07:28:31.778758+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:51.160241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:31.771848+0000","last_change":"2024-03-06T07:28:31.771848+0000","last_active":"2024-03-06T07:28:31.761490+0000","last_peered":"2024-03-06T07:28:31.761490+0000","last_clean":"2024-03-06T07:28:31.761490+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:31.771848+0000","last_undegraded":"2024-03-06T07:28:31.771848+0000","last_fullsized":"2024-03-06T07:28:31.771848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.781556+0000","last_change":"2024-03-06T07:28:31.780614+0000","last_active":"2024-03-06T07:28:31.781556+0000","last_peered":"2024-03-06T07:28:31.781556+0000","last_clean":"2024-03-06T07:28:31.781556+0000","last_became_active":"2024-03-06T07:28:31.780312+0000","last_became_peered":"2024-03-06T07:28:31.780312+0000","last_unstale":"2024-03-06T07:28:31.781556+0000","last_undegraded":"2024-03-06T07:28:31.781556+0000","last_fullsized":"2024-03-06T07:28:31.781556+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:20.580859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":350,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:31.775726+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:28:31.775726+0000","last_peered":"2024-03-06T07:28:31.775726+0000","last_clean":"2024-03-06T07:28:31.775726+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:28:31.775726+0000","last_undegraded":"2024-03-06T07:28:31.775726+0000","last_fullsized":"2024-03-06T07:28:31.775726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1257091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28508,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343332,"statfs":{"total":96636764160,"available":96607571968,"internally_reserved":0,"allocated":1523712,"data_stored":660448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40988,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330852,"statfs":{"total":96636764160,"available":96594792448,"internally_reserved":0,"allocated":1523712,"data_stored":660448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41004,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330836,"statfs":{"total":96636764160,"available":96594776064,"internally_reserved":0,"allocated":1536000,"data_stored":666723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574931,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41584,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330256,"statfs":{"total":96636764160,"available":96594182144,"internally_reserved":0,"allocated":2129920,"data_stored":1257091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:38.047 INFO:tasks.ceph.mds.b.smithi158.stderr:2024-03-06T07:28:38.044+0000 7fdf9d25e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:40.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:40.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.243+0000 7fa5ee5e9640 1 Processor -- start 2024-03-06T07:28:40.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.243+0000 7fa5ee5e9640 1 -- start start 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ee5e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa5e81515e0 0x7fa5e80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ee5e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e815b430 0x7fa5e8071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ee5e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5e80cd7a0 0x7fa5e80cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ee5e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e815b8b0 con 0x7fa5e80cd7a0 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ee5e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e8070a70 con 0x7fa5e81515e0 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ee5e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e8070d50 con 0x7fa5e815b430 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ecb86640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5e80cd7a0 0x7fa5e80cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ecb86640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5e80cd7a0 0x7fa5e80cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42200/0 (socket says 172.21.15.77:42200) 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ecb86640 1 -- 172.21.15.77:0/861890426 learned_addr learned my addr 172.21.15.77:0/861890426 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5e77fe640 1 --2- 172.21.15.77:0/861890426 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e815b430 0x7fa5e8071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5e7fff640 1 --2- 172.21.15.77:0/861890426 >> v2:172.21.15.158:3300/0 conn(0x7fa5e81515e0 0x7fa5e80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ecb86640 1 -- 172.21.15.77:0/861890426 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e815b430 msgr2=0x7fa5e8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ecb86640 1 --2- 172.21.15.77:0/861890426 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e815b430 0x7fa5e8071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ecb86640 1 -- 172.21.15.77:0/861890426 >> v2:172.21.15.158:3300/0 conn(0x7fa5e81515e0 msgr2=0x7fa5e80cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ecb86640 1 --2- 172.21.15.77:0/861890426 >> v2:172.21.15.158:3300/0 conn(0x7fa5e81515e0 0x7fa5e80cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5ecb86640 1 -- 172.21.15.77:0/861890426 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8071030 con 0x7fa5e80cd7a0 2024-03-06T07:28:40.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.244+0000 7fa5e77fe640 1 --2- 172.21.15.77:0/861890426 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e815b430 0x7fa5e8071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.245+0000 7fa5ecb86640 1 --2- 172.21.15.77:0/861890426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e80cd7a0 0x7fa5e80cdbd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fa5e806eb50 tx=0x7fa5e002cfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ec7235c0cc024b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.245+0000 7fa5e6ffd640 1 -- 172.21.15.77:0/861890426 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e002d980 con 0x7fa5e80cd7a0 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.245+0000 7fa5e6ffd640 1 -- 172.21.15.77:0/861890426 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e002db20 con 0x7fa5e80cd7a0 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.245+0000 7fa5e6ffd640 1 -- 172.21.15.77:0/861890426 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0008500 con 0x7fa5e80cd7a0 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/861890426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e80cd7a0 msgr2=0x7fa5e80cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/861890426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e80cd7a0 0x7fa5e80cdbd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fa5e806eb50 tx=0x7fa5e002cfe0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/861890426 shutdown_connections 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/861890426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e80cd7a0 0x7fa5e80cdbd0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/861890426 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e815b430 0x7fa5e8071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/861890426 >> v2:172.21.15.158:3300/0 conn(0x7fa5e81515e0 0x7fa5e80cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/861890426 >> 172.21.15.77:0/861890426 conn(0x7fa5e80cf8a0 msgr2=0x7fa5e80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/861890426 shutdown_connections 2024-03-06T07:28:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.246+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/861890426 wait complete. 2024-03-06T07:28:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.247+0000 7fa5ee5e9640 1 Processor -- start 2024-03-06T07:28:40.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.247+0000 7fa5ee5e9640 1 -- start start 2024-03-06T07:28:40.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ee5e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5e80cd7a0 0x7fa5e814c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ee5e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e81515e0 0x7fa5e8157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ee5e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e815b430 0x7fa5e814fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ee5e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e8071d70 con 0x7fa5e815b430 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ee5e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80ce110 con 0x7fa5e80cd7a0 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ee5e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e8071ef0 con 0x7fa5e81515e0 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e815b430 0x7fa5e814fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e815b430 0x7fa5e814fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42206/0 (socket says 172.21.15.77:42206) 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 -- 172.21.15.77:0/3271334434 learned_addr learned my addr 172.21.15.77:0/3271334434 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5e77fe640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e81515e0 0x7fa5e8157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5e7fff640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5e80cd7a0 0x7fa5e814c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 -- 172.21.15.77:0/3271334434 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e81515e0 msgr2=0x7fa5e8157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e81515e0 0x7fa5e8157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 -- 172.21.15.77:0/3271334434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5e80cd7a0 msgr2=0x7fa5e814c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5e80cd7a0 0x7fa5e814c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 -- 172.21.15.77:0/3271334434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e81584c0 con 0x7fa5e815b430 2024-03-06T07:28:40.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.248+0000 7fa5ecb86640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e815b430 0x7fa5e814fd80 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c140 tx=0x7fa5e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.249+0000 7fa5e4ff9640 1 -- 172.21.15.77:0/3271334434 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0044020 con 0x7fa5e815b430 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.249+0000 7fa5e4ff9640 1 -- 172.21.15.77:0/3271334434 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e0040690 con 0x7fa5e815b430 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.249+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e81550d0 con 0x7fa5e815b430 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.249+0000 7fa5e4ff9640 1 -- 172.21.15.77:0/3271334434 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e002d980 con 0x7fa5e815b430 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.249+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e8155690 con 0x7fa5e815b430 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.250+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e8068840 con 0x7fa5e815b430 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.250+0000 7fa5e4ff9640 1 -- 172.21.15.77:0/3271334434 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5e0047020 con 0x7fa5e815b430 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.251+0000 7fa5e4ff9640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa5d406b4d0 0x7fa5d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.251+0000 7fa5e7fff640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa5d406b4d0 0x7fa5d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.251+0000 7fa5e4ff9640 1 -- 172.21.15.77:0/3271334434 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5e00ace80 con 0x7fa5e815b430 2024-03-06T07:28:40.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.251+0000 7fa5e7fff640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa5d406b4d0 0x7fa5d406d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa5e814eb80 tx=0x7fa5d8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.255+0000 7fa5e4ff9640 1 -- 172.21.15.77:0/3271334434 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5e004e030 con 0x7fa5e815b430 2024-03-06T07:28:40.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.422+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5e814ccd0 con 0x7fa5d406b4d0 2024-03-06T07:28:40.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.428+0000 7fa5e4ff9640 1 -- 172.21.15.77:0/3271334434 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7fa5e814ccd0 con 0x7fa5d406b4d0 2024-03-06T07:28:40.433 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:40.434 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:28:40.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa5d406b4d0 msgr2=0x7fa5d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa5d406b4d0 0x7fa5d406d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa5e814eb80 tx=0x7fa5d8009190 comp rx=0 tx=0).stop 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e815b430 msgr2=0x7fa5e814fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e815b430 0x7fa5e814fd80 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c140 tx=0x7fa5e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 shutdown_connections 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa5d406b4d0 0x7fa5d406d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5e815b430 0x7fa5e814fd80 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa5e81515e0 0x7fa5e8157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 --2- 172.21.15.77:0/3271334434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa5e80cd7a0 0x7fa5e814c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 >> 172.21.15.77:0/3271334434 conn(0x7fa5e80cf8a0 msgr2=0x7fa5e8154240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 shutdown_connections 2024-03-06T07:28:40.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.431+0000 7fa5ee5e9640 1 -- 172.21.15.77:0/3271334434 wait complete. 2024-03-06T07:28:40.444 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-03-06T07:28:39.456779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304468,"kb_used_data":13416,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754670252,"statfs":{"total":773094113280,"available":772782338048,"internally_reserved":0,"allocated":13737984,"data_stored":6623940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.683573"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122370+0000","last_change":"2024-03-06T07:28:32.810379+0000","last_active":"2024-03-06T07:28:34.122370+0000","last_peered":"2024-03-06T07:28:34.122370+0000","last_clean":"2024-03-06T07:28:34.122370+0000","last_became_active":"2024-03-06T07:28:32.810123+0000","last_became_peered":"2024-03-06T07:28:32.810123+0000","last_unstale":"2024-03-06T07:28:34.122370+0000","last_undegraded":"2024-03-06T07:28:34.122370+0000","last_fullsized":"2024-03-06T07:28:34.122370+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:12.623244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122383+0000","last_change":"2024-03-06T07:28:32.795002+0000","last_active":"2024-03-06T07:28:34.122383+0000","last_peered":"2024-03-06T07:28:34.122383+0000","last_clean":"2024-03-06T07:28:34.122383+0000","last_became_active":"2024-03-06T07:28:32.793998+0000","last_became_peered":"2024-03-06T07:28:32.793998+0000","last_unstale":"2024-03-06T07:28:34.122383+0000","last_undegraded":"2024-03-06T07:28:34.122383+0000","last_fullsized":"2024-03-06T07:28:34.122383+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:14.695719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797992+0000","last_change":"2024-03-06T07:28:32.802526+0000","last_active":"2024-03-06T07:28:33.797992+0000","last_peered":"2024-03-06T07:28:33.797992+0000","last_clean":"2024-03-06T07:28:33.797992+0000","last_became_active":"2024-03-06T07:28:32.802035+0000","last_became_peered":"2024-03-06T07:28:32.802035+0000","last_unstale":"2024-03-06T07:28:33.797992+0000","last_undegraded":"2024-03-06T07:28:33.797992+0000","last_fullsized":"2024-03-06T07:28:33.797992+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:11.019596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121473+0000","last_change":"2024-03-06T07:28:32.813212+0000","last_active":"2024-03-06T07:28:34.121473+0000","last_peered":"2024-03-06T07:28:34.121473+0000","last_clean":"2024-03-06T07:28:34.121473+0000","last_became_active":"2024-03-06T07:28:32.812529+0000","last_became_peered":"2024-03-06T07:28:32.812529+0000","last_unstale":"2024-03-06T07:28:34.121473+0000","last_undegraded":"2024-03-06T07:28:34.121473+0000","last_fullsized":"2024-03-06T07:28:34.121473+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:03.117543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804723+0000","last_change":"2024-03-06T07:28:32.812746+0000","last_active":"2024-03-06T07:28:33.804723+0000","last_peered":"2024-03-06T07:28:33.804723+0000","last_clean":"2024-03-06T07:28:33.804723+0000","last_became_active":"2024-03-06T07:28:32.812450+0000","last_became_peered":"2024-03-06T07:28:32.812450+0000","last_unstale":"2024-03-06T07:28:33.804723+0000","last_undegraded":"2024-03-06T07:28:33.804723+0000","last_fullsized":"2024-03-06T07:28:33.804723+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:29.519474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797819+0000","last_change":"2024-03-06T07:28:32.807222+0000","last_active":"2024-03-06T07:28:33.797819+0000","last_peered":"2024-03-06T07:28:33.797819+0000","last_clean":"2024-03-06T07:28:33.797819+0000","last_became_active":"2024-03-06T07:28:32.806960+0000","last_became_peered":"2024-03-06T07:28:32.806960+0000","last_unstale":"2024-03-06T07:28:33.797819+0000","last_undegraded":"2024-03-06T07:28:33.797819+0000","last_fullsized":"2024-03-06T07:28:33.797819+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:18.110159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.796623+0000","last_change":"2024-03-06T07:28:32.801229+0000","last_active":"2024-03-06T07:28:33.796623+0000","last_peered":"2024-03-06T07:28:33.796623+0000","last_clean":"2024-03-06T07:28:33.796623+0000","last_became_active":"2024-03-06T07:28:32.800962+0000","last_became_peered":"2024-03-06T07:28:32.800962+0000","last_unstale":"2024-03-06T07:28:33.796623+0000","last_undegraded":"2024-03-06T07:28:33.796623+0000","last_fullsized":"2024-03-06T07:28:33.796623+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:57.396799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.798071+0000","last_change":"2024-03-06T07:28:32.810529+0000","last_active":"2024-03-06T07:28:33.798071+0000","last_peered":"2024-03-06T07:28:33.798071+0000","last_clean":"2024-03-06T07:28:33.798071+0000","last_became_active":"2024-03-06T07:28:32.810253+0000","last_became_peered":"2024-03-06T07:28:32.810253+0000","last_unstale":"2024-03-06T07:28:33.798071+0000","last_undegraded":"2024-03-06T07:28:33.798071+0000","last_fullsized":"2024-03-06T07:28:33.798071+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:18.110159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.798000+0000","last_change":"2024-03-06T07:28:32.813387+0000","last_active":"2024-03-06T07:28:33.798000+0000","last_peered":"2024-03-06T07:28:33.798000+0000","last_clean":"2024-03-06T07:28:33.798000+0000","last_became_active":"2024-03-06T07:28:32.813106+0000","last_became_peered":"2024-03-06T07:28:32.813106+0000","last_unstale":"2024-03-06T07:28:33.798000+0000","last_undegraded":"2024-03-06T07:28:33.798000+0000","last_fullsized":"2024-03-06T07:28:33.798000+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:27.000111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122528+0000","last_change":"2024-03-06T07:28:32.794864+0000","last_active":"2024-03-06T07:28:34.122528+0000","last_peered":"2024-03-06T07:28:34.122528+0000","last_clean":"2024-03-06T07:28:34.122528+0000","last_became_active":"2024-03-06T07:28:32.794621+0000","last_became_peered":"2024-03-06T07:28:32.794621+0000","last_unstale":"2024-03-06T07:28:34.122528+0000","last_undegraded":"2024-03-06T07:28:34.122528+0000","last_fullsized":"2024-03-06T07:28:34.122528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:49.340853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.795967+0000","last_change":"2024-03-06T07:28:32.802604+0000","last_active":"2024-03-06T07:28:33.795967+0000","last_peered":"2024-03-06T07:28:33.795967+0000","last_clean":"2024-03-06T07:28:33.795967+0000","last_became_active":"2024-03-06T07:28:32.802342+0000","last_became_peered":"2024-03-06T07:28:32.802342+0000","last_unstale":"2024-03-06T07:28:33.795967+0000","last_undegraded":"2024-03-06T07:28:33.795967+0000","last_fullsized":"2024-03-06T07:28:33.795967+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:55.596292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797912+0000","last_change":"2024-03-06T07:28:32.806359+0000","last_active":"2024-03-06T07:28:33.797912+0000","last_peered":"2024-03-06T07:28:33.797912+0000","last_clean":"2024-03-06T07:28:33.797912+0000","last_became_active":"2024-03-06T07:28:32.806061+0000","last_became_peered":"2024-03-06T07:28:32.806061+0000","last_unstale":"2024-03-06T07:28:33.797912+0000","last_undegraded":"2024-03-06T07:28:33.797912+0000","last_fullsized":"2024-03-06T07:28:33.797912+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:00.117035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122116+0000","last_change":"2024-03-06T07:28:32.811024+0000","last_active":"2024-03-06T07:28:34.122116+0000","last_peered":"2024-03-06T07:28:34.122116+0000","last_clean":"2024-03-06T07:28:34.122116+0000","last_became_active":"2024-03-06T07:28:32.810776+0000","last_became_peered":"2024-03-06T07:28:32.810776+0000","last_unstale":"2024-03-06T07:28:34.122116+0000","last_undegraded":"2024-03-06T07:28:34.122116+0000","last_fullsized":"2024-03-06T07:28:34.122116+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:53.766381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.123075+0000","last_change":"2024-03-06T07:28:32.802029+0000","last_active":"2024-03-06T07:28:34.123075+0000","last_peered":"2024-03-06T07:28:34.123075+0000","last_clean":"2024-03-06T07:28:34.123075+0000","last_became_active":"2024-03-06T07:28:32.801777+0000","last_became_peered":"2024-03-06T07:28:32.801777+0000","last_unstale":"2024-03-06T07:28:34.123075+0000","last_undegraded":"2024-03-06T07:28:34.123075+0000","last_fullsized":"2024-03-06T07:28:34.123075+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:59.796120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122009+0000","last_change":"2024-03-06T07:28:32.812408+0000","last_active":"2024-03-06T07:28:34.122009+0000","last_peered":"2024-03-06T07:28:34.122009+0000","last_clean":"2024-03-06T07:28:34.122009+0000","last_became_active":"2024-03-06T07:28:32.812141+0000","last_became_peered":"2024-03-06T07:28:32.812141+0000","last_unstale":"2024-03-06T07:28:34.122009+0000","last_undegraded":"2024-03-06T07:28:34.122009+0000","last_fullsized":"2024-03-06T07:28:34.122009+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:21.314921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122413+0000","last_change":"2024-03-06T07:28:32.805185+0000","last_active":"2024-03-06T07:28:34.122413+0000","last_peered":"2024-03-06T07:28:34.122413+0000","last_clean":"2024-03-06T07:28:34.122413+0000","last_became_active":"2024-03-06T07:28:32.804935+0000","last_became_peered":"2024-03-06T07:28:32.804935+0000","last_unstale":"2024-03-06T07:28:34.122413+0000","last_undegraded":"2024-03-06T07:28:34.122413+0000","last_fullsized":"2024-03-06T07:28:34.122413+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:19.700804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.796990+0000","last_change":"2024-03-06T07:28:32.796914+0000","last_active":"2024-03-06T07:28:33.796990+0000","last_peered":"2024-03-06T07:28:33.796990+0000","last_clean":"2024-03-06T07:28:33.796990+0000","last_became_active":"2024-03-06T07:28:32.796631+0000","last_became_peered":"2024-03-06T07:28:32.796631+0000","last_unstale":"2024-03-06T07:28:33.796990+0000","last_undegraded":"2024-03-06T07:28:33.796990+0000","last_fullsized":"2024-03-06T07:28:33.796990+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:20.221502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121656+0000","last_change":"2024-03-06T07:28:32.810616+0000","last_active":"2024-03-06T07:28:34.121656+0000","last_peered":"2024-03-06T07:28:34.121656+0000","last_clean":"2024-03-06T07:28:34.121656+0000","last_became_active":"2024-03-06T07:28:32.810320+0000","last_became_peered":"2024-03-06T07:28:32.810320+0000","last_unstale":"2024-03-06T07:28:34.121656+0000","last_undegraded":"2024-03-06T07:28:34.121656+0000","last_fullsized":"2024-03-06T07:28:34.121656+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:10.606710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.796028+0000","last_change":"2024-03-06T07:28:32.799858+0000","last_active":"2024-03-06T07:28:33.796028+0000","last_peered":"2024-03-06T07:28:33.796028+0000","last_clean":"2024-03-06T07:28:33.796028+0000","last_became_active":"2024-03-06T07:28:32.799611+0000","last_became_peered":"2024-03-06T07:28:32.799611+0000","last_unstale":"2024-03-06T07:28:33.796028+0000","last_undegraded":"2024-03-06T07:28:33.796028+0000","last_fullsized":"2024-03-06T07:28:33.796028+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:05.781056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121954+0000","last_change":"2024-03-06T07:28:32.798274+0000","last_active":"2024-03-06T07:28:34.121954+0000","last_peered":"2024-03-06T07:28:34.121954+0000","last_clean":"2024-03-06T07:28:34.121954+0000","last_became_active":"2024-03-06T07:28:32.798040+0000","last_became_peered":"2024-03-06T07:28:32.798040+0000","last_unstale":"2024-03-06T07:28:34.121954+0000","last_undegraded":"2024-03-06T07:28:34.121954+0000","last_fullsized":"2024-03-06T07:28:34.121954+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:16.265963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804509+0000","last_change":"2024-03-06T07:28:32.812093+0000","last_active":"2024-03-06T07:28:33.804509+0000","last_peered":"2024-03-06T07:28:33.804509+0000","last_clean":"2024-03-06T07:28:33.804509+0000","last_became_active":"2024-03-06T07:28:32.811836+0000","last_became_peered":"2024-03-06T07:28:32.811836+0000","last_unstale":"2024-03-06T07:28:33.804509+0000","last_undegraded":"2024-03-06T07:28:33.804509+0000","last_fullsized":"2024-03-06T07:28:33.804509+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:16.114529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.795719+0000","last_change":"2024-03-06T07:28:32.798026+0000","last_active":"2024-03-06T07:28:33.795719+0000","last_peered":"2024-03-06T07:28:33.795719+0000","last_clean":"2024-03-06T07:28:33.795719+0000","last_became_active":"2024-03-06T07:28:32.797789+0000","last_became_peered":"2024-03-06T07:28:32.797789+0000","last_unstale":"2024-03-06T07:28:33.795719+0000","last_undegraded":"2024-03-06T07:28:33.795719+0000","last_fullsized":"2024-03-06T07:28:33.795719+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:00.015794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797791+0000","last_change":"2024-03-06T07:28:32.805048+0000","last_active":"2024-03-06T07:28:33.797791+0000","last_peered":"2024-03-06T07:28:33.797791+0000","last_clean":"2024-03-06T07:28:33.797791+0000","last_became_active":"2024-03-06T07:28:32.804704+0000","last_became_peered":"2024-03-06T07:28:32.804704+0000","last_unstale":"2024-03-06T07:28:33.797791+0000","last_undegraded":"2024-03-06T07:28:33.797791+0000","last_fullsized":"2024-03-06T07:28:33.797791+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:09.436899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.795661+0000","last_change":"2024-03-06T07:28:32.810458+0000","last_active":"2024-03-06T07:28:33.795661+0000","last_peered":"2024-03-06T07:28:33.795661+0000","last_clean":"2024-03-06T07:28:33.795661+0000","last_became_active":"2024-03-06T07:28:32.810193+0000","last_became_peered":"2024-03-06T07:28:32.810193+0000","last_unstale":"2024-03-06T07:28:33.795661+0000","last_undegraded":"2024-03-06T07:28:33.795661+0000","last_fullsized":"2024-03-06T07:28:33.795661+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:46.853009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122808+0000","last_change":"2024-03-06T07:28:32.800353+0000","last_active":"2024-03-06T07:28:34.122808+0000","last_peered":"2024-03-06T07:28:34.122808+0000","last_clean":"2024-03-06T07:28:34.122808+0000","last_became_active":"2024-03-06T07:28:32.800081+0000","last_became_peered":"2024-03-06T07:28:32.800081+0000","last_unstale":"2024-03-06T07:28:34.122808+0000","last_undegraded":"2024-03-06T07:28:34.122808+0000","last_fullsized":"2024-03-06T07:28:34.122808+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:25.574541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121804+0000","last_change":"2024-03-06T07:28:32.803172+0000","last_active":"2024-03-06T07:28:34.121804+0000","last_peered":"2024-03-06T07:28:34.121804+0000","last_clean":"2024-03-06T07:28:34.121804+0000","last_became_active":"2024-03-06T07:28:32.802856+0000","last_became_peered":"2024-03-06T07:28:32.802856+0000","last_unstale":"2024-03-06T07:28:34.121804+0000","last_undegraded":"2024-03-06T07:28:34.121804+0000","last_fullsized":"2024-03-06T07:28:34.121804+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:29.519474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121324+0000","last_change":"2024-03-06T07:28:32.808536+0000","last_active":"2024-03-06T07:28:34.121324+0000","last_peered":"2024-03-06T07:28:34.121324+0000","last_clean":"2024-03-06T07:28:34.121324+0000","last_became_active":"2024-03-06T07:28:32.808085+0000","last_became_peered":"2024-03-06T07:28:32.808085+0000","last_unstale":"2024-03-06T07:28:34.121324+0000","last_undegraded":"2024-03-06T07:28:34.121324+0000","last_fullsized":"2024-03-06T07:28:34.121324+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:00.211241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121200+0000","last_change":"2024-03-06T07:28:32.808876+0000","last_active":"2024-03-06T07:28:34.121200+0000","last_peered":"2024-03-06T07:28:34.121200+0000","last_clean":"2024-03-06T07:28:34.121200+0000","last_became_active":"2024-03-06T07:28:32.808317+0000","last_became_peered":"2024-03-06T07:28:32.808317+0000","last_unstale":"2024-03-06T07:28:34.121200+0000","last_undegraded":"2024-03-06T07:28:34.121200+0000","last_fullsized":"2024-03-06T07:28:34.121200+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:41.276286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122270+0000","last_change":"2024-03-06T07:28:32.812531+0000","last_active":"2024-03-06T07:28:34.122270+0000","last_peered":"2024-03-06T07:28:34.122270+0000","last_clean":"2024-03-06T07:28:34.122270+0000","last_became_active":"2024-03-06T07:28:32.812285+0000","last_became_peered":"2024-03-06T07:28:32.812285+0000","last_unstale":"2024-03-06T07:28:34.122270+0000","last_undegraded":"2024-03-06T07:28:34.122270+0000","last_fullsized":"2024-03-06T07:28:34.122270+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:15.019496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121668+0000","last_change":"2024-03-06T07:28:32.812468+0000","last_active":"2024-03-06T07:28:34.121668+0000","last_peered":"2024-03-06T07:28:34.121668+0000","last_clean":"2024-03-06T07:28:34.121668+0000","last_became_active":"2024-03-06T07:28:32.812184+0000","last_became_peered":"2024-03-06T07:28:32.812184+0000","last_unstale":"2024-03-06T07:28:34.121668+0000","last_undegraded":"2024-03-06T07:28:34.121668+0000","last_fullsized":"2024-03-06T07:28:34.121668+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:59.747768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804984+0000","last_change":"2024-03-06T07:28:32.804513+0000","last_active":"2024-03-06T07:28:33.804984+0000","last_peered":"2024-03-06T07:28:33.804984+0000","last_clean":"2024-03-06T07:28:33.804984+0000","last_became_active":"2024-03-06T07:28:32.804257+0000","last_became_peered":"2024-03-06T07:28:32.804257+0000","last_unstale":"2024-03-06T07:28:33.804984+0000","last_undegraded":"2024-03-06T07:28:33.804984+0000","last_fullsized":"2024-03-06T07:28:33.804984+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:52.162173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.796713+0000","last_change":"2024-03-06T07:28:32.808335+0000","last_active":"2024-03-06T07:28:33.796713+0000","last_peered":"2024-03-06T07:28:33.796713+0000","last_clean":"2024-03-06T07:28:33.796713+0000","last_became_active":"2024-03-06T07:28:32.808047+0000","last_became_peered":"2024-03-06T07:28:32.808047+0000","last_unstale":"2024-03-06T07:28:33.796713+0000","last_undegraded":"2024-03-06T07:28:33.796713+0000","last_fullsized":"2024-03-06T07:28:33.796713+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:14.185466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121953+0000","last_change":"2024-03-06T07:28:32.811032+0000","last_active":"2024-03-06T07:28:34.121953+0000","last_peered":"2024-03-06T07:28:34.121953+0000","last_clean":"2024-03-06T07:28:34.121953+0000","last_became_active":"2024-03-06T07:28:32.810715+0000","last_became_peered":"2024-03-06T07:28:32.810715+0000","last_unstale":"2024-03-06T07:28:34.121953+0000","last_undegraded":"2024-03-06T07:28:34.121953+0000","last_fullsized":"2024-03-06T07:28:34.121953+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:02.777791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"124'4","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.805113+0000","last_change":"2024-03-06T07:28:31.782852+0000","last_active":"2024-03-06T07:28:33.805113+0000","last_peered":"2024-03-06T07:28:33.805113+0000","last_clean":"2024-03-06T07:28:33.805113+0000","last_became_active":"2024-03-06T07:28:31.782575+0000","last_became_peered":"2024-03-06T07:28:31.782575+0000","last_unstale":"2024-03-06T07:28:33.805113+0000","last_undegraded":"2024-03-06T07:28:33.805113+0000","last_fullsized":"2024-03-06T07:28:33.805113+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:25.998178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797560+0000","last_change":"2024-03-06T07:28:32.806769+0000","last_active":"2024-03-06T07:28:33.797560+0000","last_peered":"2024-03-06T07:28:33.797560+0000","last_clean":"2024-03-06T07:28:33.797560+0000","last_became_active":"2024-03-06T07:28:32.806103+0000","last_became_peered":"2024-03-06T07:28:32.806103+0000","last_unstale":"2024-03-06T07:28:33.797560+0000","last_undegraded":"2024-03-06T07:28:33.797560+0000","last_fullsized":"2024-03-06T07:28:33.797560+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:31.777440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121269+0000","last_change":"2024-03-06T07:28:31.782726+0000","last_active":"2024-03-06T07:28:34.121269+0000","last_peered":"2024-03-06T07:28:34.121269+0000","last_clean":"2024-03-06T07:28:34.121269+0000","last_became_active":"2024-03-06T07:28:31.782447+0000","last_became_peered":"2024-03-06T07:28:31.782447+0000","last_unstale":"2024-03-06T07:28:34.121269+0000","last_undegraded":"2024-03-06T07:28:34.121269+0000","last_fullsized":"2024-03-06T07:28:34.121269+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:23.936759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804859+0000","last_change":"2024-03-06T07:28:32.805684+0000","last_active":"2024-03-06T07:28:33.804859+0000","last_peered":"2024-03-06T07:28:33.804859+0000","last_clean":"2024-03-06T07:28:33.804859+0000","last_became_active":"2024-03-06T07:28:32.805437+0000","last_became_peered":"2024-03-06T07:28:32.805437+0000","last_unstale":"2024-03-06T07:28:33.804859+0000","last_undegraded":"2024-03-06T07:28:33.804859+0000","last_fullsized":"2024-03-06T07:28:33.804859+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:07.020527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.796030+0000","last_change":"2024-03-06T07:28:31.783586+0000","last_active":"2024-03-06T07:28:33.796030+0000","last_peered":"2024-03-06T07:28:33.796030+0000","last_clean":"2024-03-06T07:28:33.796030+0000","last_became_active":"2024-03-06T07:28:31.783329+0000","last_became_peered":"2024-03-06T07:28:31.783329+0000","last_unstale":"2024-03-06T07:28:33.796030+0000","last_undegraded":"2024-03-06T07:28:33.796030+0000","last_fullsized":"2024-03-06T07:28:33.796030+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:38:11.354370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804703+0000","last_change":"2024-03-06T07:28:32.803227+0000","last_active":"2024-03-06T07:28:33.804703+0000","last_peered":"2024-03-06T07:28:33.804703+0000","last_clean":"2024-03-06T07:28:33.804703+0000","last_became_active":"2024-03-06T07:28:32.802936+0000","last_became_peered":"2024-03-06T07:28:32.802936+0000","last_unstale":"2024-03-06T07:28:33.804703+0000","last_undegraded":"2024-03-06T07:28:33.804703+0000","last_fullsized":"2024-03-06T07:28:33.804703+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:10.606710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122672+0000","last_change":"2024-03-06T07:28:31.780002+0000","last_active":"2024-03-06T07:28:34.122672+0000","last_peered":"2024-03-06T07:28:34.122672+0000","last_clean":"2024-03-06T07:28:34.122672+0000","last_became_active":"2024-03-06T07:28:31.779736+0000","last_became_peered":"2024-03-06T07:28:31.779736+0000","last_unstale":"2024-03-06T07:28:34.122672+0000","last_undegraded":"2024-03-06T07:28:34.122672+0000","last_fullsized":"2024-03-06T07:28:34.122672+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:20.312988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121499+0000","last_change":"2024-03-06T07:28:32.810775+0000","last_active":"2024-03-06T07:28:34.121499+0000","last_peered":"2024-03-06T07:28:34.121499+0000","last_clean":"2024-03-06T07:28:34.121499+0000","last_became_active":"2024-03-06T07:28:32.810507+0000","last_became_peered":"2024-03-06T07:28:32.810507+0000","last_unstale":"2024-03-06T07:28:34.121499+0000","last_undegraded":"2024-03-06T07:28:34.121499+0000","last_fullsized":"2024-03-06T07:28:34.121499+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:16.114529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122573+0000","last_change":"2024-03-06T07:28:31.783109+0000","last_active":"2024-03-06T07:28:34.122573+0000","last_peered":"2024-03-06T07:28:34.122573+0000","last_clean":"2024-03-06T07:28:34.122573+0000","last_became_active":"2024-03-06T07:28:31.782858+0000","last_became_peered":"2024-03-06T07:28:31.782858+0000","last_unstale":"2024-03-06T07:28:34.122573+0000","last_undegraded":"2024-03-06T07:28:34.122573+0000","last_fullsized":"2024-03-06T07:28:34.122573+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:02.115610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797252+0000","last_change":"2024-03-06T07:28:32.806429+0000","last_active":"2024-03-06T07:28:33.797252+0000","last_peered":"2024-03-06T07:28:33.797252+0000","last_clean":"2024-03-06T07:28:33.797252+0000","last_became_active":"2024-03-06T07:28:32.805778+0000","last_became_peered":"2024-03-06T07:28:32.805778+0000","last_unstale":"2024-03-06T07:28:33.797252+0000","last_undegraded":"2024-03-06T07:28:33.797252+0000","last_fullsized":"2024-03-06T07:28:33.797252+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:00.117035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121590+0000","last_change":"2024-03-06T07:28:31.781824+0000","last_active":"2024-03-06T07:28:34.121590+0000","last_peered":"2024-03-06T07:28:34.121590+0000","last_clean":"2024-03-06T07:28:34.121590+0000","last_became_active":"2024-03-06T07:28:31.781484+0000","last_became_peered":"2024-03-06T07:28:31.781484+0000","last_unstale":"2024-03-06T07:28:34.121590+0000","last_undegraded":"2024-03-06T07:28:34.121590+0000","last_fullsized":"2024-03-06T07:28:34.121590+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:43.691775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797543+0000","last_change":"2024-03-06T07:28:32.799756+0000","last_active":"2024-03-06T07:28:33.797543+0000","last_peered":"2024-03-06T07:28:33.797543+0000","last_clean":"2024-03-06T07:28:33.797543+0000","last_became_active":"2024-03-06T07:28:32.799447+0000","last_became_peered":"2024-03-06T07:28:32.799447+0000","last_unstale":"2024-03-06T07:28:33.797543+0000","last_undegraded":"2024-03-06T07:28:33.797543+0000","last_fullsized":"2024-03-06T07:28:33.797543+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:14.185466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797375+0000","last_change":"2024-03-06T07:28:31.777579+0000","last_active":"2024-03-06T07:28:33.797375+0000","last_peered":"2024-03-06T07:28:33.797375+0000","last_clean":"2024-03-06T07:28:33.797375+0000","last_became_active":"2024-03-06T07:28:31.777290+0000","last_became_peered":"2024-03-06T07:28:31.777290+0000","last_unstale":"2024-03-06T07:28:33.797375+0000","last_undegraded":"2024-03-06T07:28:33.797375+0000","last_fullsized":"2024-03-06T07:28:33.797375+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:17.422840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121349+0000","last_change":"2024-03-06T07:28:32.807134+0000","last_active":"2024-03-06T07:28:34.121349+0000","last_peered":"2024-03-06T07:28:34.121349+0000","last_clean":"2024-03-06T07:28:34.121349+0000","last_became_active":"2024-03-06T07:28:32.806840+0000","last_became_peered":"2024-03-06T07:28:32.806840+0000","last_unstale":"2024-03-06T07:28:34.121349+0000","last_undegraded":"2024-03-06T07:28:34.121349+0000","last_fullsized":"2024-03-06T07:28:34.121349+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:15.780975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121475+0000","last_change":"2024-03-06T07:28:31.775485+0000","last_active":"2024-03-06T07:28:34.121475+0000","last_peered":"2024-03-06T07:28:34.121475+0000","last_clean":"2024-03-06T07:28:34.121475+0000","last_became_active":"2024-03-06T07:28:31.775212+0000","last_became_peered":"2024-03-06T07:28:31.775212+0000","last_unstale":"2024-03-06T07:28:34.121475+0000","last_undegraded":"2024-03-06T07:28:34.121475+0000","last_fullsized":"2024-03-06T07:28:34.121475+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:01.019289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121771+0000","last_change":"2024-03-06T07:28:32.801218+0000","last_active":"2024-03-06T07:28:34.121771+0000","last_peered":"2024-03-06T07:28:34.121771+0000","last_clean":"2024-03-06T07:28:34.121771+0000","last_became_active":"2024-03-06T07:28:32.800939+0000","last_became_peered":"2024-03-06T07:28:32.800939+0000","last_unstale":"2024-03-06T07:28:34.121771+0000","last_undegraded":"2024-03-06T07:28:34.121771+0000","last_fullsized":"2024-03-06T07:28:34.121771+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:02.021222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121480+0000","last_change":"2024-03-06T07:28:31.783404+0000","last_active":"2024-03-06T07:28:34.121480+0000","last_peered":"2024-03-06T07:28:34.121480+0000","last_clean":"2024-03-06T07:28:34.121480+0000","last_became_active":"2024-03-06T07:28:31.782969+0000","last_became_peered":"2024-03-06T07:28:31.782969+0000","last_unstale":"2024-03-06T07:28:34.121480+0000","last_undegraded":"2024-03-06T07:28:34.121480+0000","last_fullsized":"2024-03-06T07:28:34.121480+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:48.338921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121334+0000","last_change":"2024-03-06T07:28:32.805343+0000","last_active":"2024-03-06T07:28:34.121334+0000","last_peered":"2024-03-06T07:28:34.121334+0000","last_clean":"2024-03-06T07:28:34.121334+0000","last_became_active":"2024-03-06T07:28:32.805050+0000","last_became_peered":"2024-03-06T07:28:32.805050+0000","last_unstale":"2024-03-06T07:28:34.121334+0000","last_undegraded":"2024-03-06T07:28:34.121334+0000","last_fullsized":"2024-03-06T07:28:34.121334+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:18.424772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122049+0000","last_change":"2024-03-06T07:28:31.784102+0000","last_active":"2024-03-06T07:28:34.122049+0000","last_peered":"2024-03-06T07:28:34.122049+0000","last_clean":"2024-03-06T07:28:34.122049+0000","last_became_active":"2024-03-06T07:28:31.783888+0000","last_became_peered":"2024-03-06T07:28:31.783888+0000","last_unstale":"2024-03-06T07:28:34.122049+0000","last_undegraded":"2024-03-06T07:28:34.122049+0000","last_fullsized":"2024-03-06T07:28:34.122049+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:26.625983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804971+0000","last_change":"2024-03-06T07:28:32.809085+0000","last_active":"2024-03-06T07:28:33.804971+0000","last_peered":"2024-03-06T07:28:33.804971+0000","last_clean":"2024-03-06T07:28:33.804971+0000","last_became_active":"2024-03-06T07:28:32.808804+0000","last_became_peered":"2024-03-06T07:28:32.808804+0000","last_unstale":"2024-03-06T07:28:33.804971+0000","last_undegraded":"2024-03-06T07:28:33.804971+0000","last_fullsized":"2024-03-06T07:28:33.804971+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:19.700804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"124'3","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121142+0000","last_change":"2024-03-06T07:28:31.782822+0000","last_active":"2024-03-06T07:28:34.121142+0000","last_peered":"2024-03-06T07:28:34.121142+0000","last_clean":"2024-03-06T07:28:34.121142+0000","last_became_active":"2024-03-06T07:28:31.782523+0000","last_became_peered":"2024-03-06T07:28:31.782523+0000","last_unstale":"2024-03-06T07:28:34.121142+0000","last_undegraded":"2024-03-06T07:28:34.121142+0000","last_fullsized":"2024-03-06T07:28:34.121142+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:10.429520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804785+0000","last_change":"2024-03-06T07:28:32.810884+0000","last_active":"2024-03-06T07:28:33.804785+0000","last_peered":"2024-03-06T07:28:33.804785+0000","last_clean":"2024-03-06T07:28:33.804785+0000","last_became_active":"2024-03-06T07:28:32.810606+0000","last_became_peered":"2024-03-06T07:28:32.810606+0000","last_unstale":"2024-03-06T07:28:33.804785+0000","last_undegraded":"2024-03-06T07:28:33.804785+0000","last_fullsized":"2024-03-06T07:28:33.804785+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:16.265963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"124'6","reported_seq":20,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797127+0000","last_change":"2024-03-06T07:28:31.776785+0000","last_active":"2024-03-06T07:28:33.797127+0000","last_peered":"2024-03-06T07:28:33.797127+0000","last_clean":"2024-03-06T07:28:33.797127+0000","last_became_active":"2024-03-06T07:28:31.776521+0000","last_became_peered":"2024-03-06T07:28:31.776521+0000","last_unstale":"2024-03-06T07:28:33.797127+0000","last_undegraded":"2024-03-06T07:28:33.797127+0000","last_fullsized":"2024-03-06T07:28:33.797127+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:05.062758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122642+0000","last_change":"2024-03-06T07:28:32.801889+0000","last_active":"2024-03-06T07:28:34.122642+0000","last_peered":"2024-03-06T07:28:34.122642+0000","last_clean":"2024-03-06T07:28:34.122642+0000","last_became_active":"2024-03-06T07:28:32.801606+0000","last_became_peered":"2024-03-06T07:28:32.801606+0000","last_unstale":"2024-03-06T07:28:34.122642+0000","last_undegraded":"2024-03-06T07:28:34.122642+0000","last_fullsized":"2024-03-06T07:28:34.122642+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:35.257603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121888+0000","last_change":"2024-03-06T07:28:31.775867+0000","last_active":"2024-03-06T07:28:34.121888+0000","last_peered":"2024-03-06T07:28:34.121888+0000","last_clean":"2024-03-06T07:28:34.121888+0000","last_became_active":"2024-03-06T07:28:31.775622+0000","last_became_peered":"2024-03-06T07:28:31.775622+0000","last_unstale":"2024-03-06T07:28:34.121888+0000","last_undegraded":"2024-03-06T07:28:34.121888+0000","last_fullsized":"2024-03-06T07:28:34.121888+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:56.696521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121467+0000","last_change":"2024-03-06T07:28:32.794301+0000","last_active":"2024-03-06T07:28:34.121467+0000","last_peered":"2024-03-06T07:28:34.121467+0000","last_clean":"2024-03-06T07:28:34.121467+0000","last_became_active":"2024-03-06T07:28:32.793984+0000","last_became_peered":"2024-03-06T07:28:32.793984+0000","last_unstale":"2024-03-06T07:28:34.121467+0000","last_undegraded":"2024-03-06T07:28:34.121467+0000","last_fullsized":"2024-03-06T07:28:34.121467+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:20.089068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122335+0000","last_change":"2024-03-06T07:28:31.777064+0000","last_active":"2024-03-06T07:28:34.122335+0000","last_peered":"2024-03-06T07:28:34.122335+0000","last_clean":"2024-03-06T07:28:34.122335+0000","last_became_active":"2024-03-06T07:28:31.776788+0000","last_became_peered":"2024-03-06T07:28:31.776788+0000","last_unstale":"2024-03-06T07:28:34.122335+0000","last_undegraded":"2024-03-06T07:28:34.122335+0000","last_fullsized":"2024-03-06T07:28:34.122335+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:00.996516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797991+0000","last_change":"2024-03-06T07:28:32.813904+0000","last_active":"2024-03-06T07:28:33.797991+0000","last_peered":"2024-03-06T07:28:33.797991+0000","last_clean":"2024-03-06T07:28:33.797991+0000","last_became_active":"2024-03-06T07:28:32.813603+0000","last_became_peered":"2024-03-06T07:28:32.813603+0000","last_unstale":"2024-03-06T07:28:33.797991+0000","last_undegraded":"2024-03-06T07:28:33.797991+0000","last_fullsized":"2024-03-06T07:28:33.797991+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:57.396799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"124'8","reported_seq":22,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797790+0000","last_change":"2024-03-06T07:28:31.781673+0000","last_active":"2024-03-06T07:28:33.797790+0000","last_peered":"2024-03-06T07:28:33.797790+0000","last_clean":"2024-03-06T07:28:33.797790+0000","last_became_active":"2024-03-06T07:28:31.781383+0000","last_became_peered":"2024-03-06T07:28:31.781383+0000","last_unstale":"2024-03-06T07:28:33.797790+0000","last_undegraded":"2024-03-06T07:28:33.797790+0000","last_fullsized":"2024-03-06T07:28:33.797790+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:40.274353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797864+0000","last_change":"2024-03-06T07:28:32.802740+0000","last_active":"2024-03-06T07:28:33.797864+0000","last_peered":"2024-03-06T07:28:33.797864+0000","last_clean":"2024-03-06T07:28:33.797864+0000","last_became_active":"2024-03-06T07:28:32.802459+0000","last_became_peered":"2024-03-06T07:28:32.802459+0000","last_unstale":"2024-03-06T07:28:33.797864+0000","last_undegraded":"2024-03-06T07:28:33.797864+0000","last_fullsized":"2024-03-06T07:28:33.797864+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:20.221502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121486+0000","last_change":"2024-03-06T07:28:31.780402+0000","last_active":"2024-03-06T07:28:34.121486+0000","last_peered":"2024-03-06T07:28:34.121486+0000","last_clean":"2024-03-06T07:28:34.121486+0000","last_became_active":"2024-03-06T07:28:31.780050+0000","last_became_peered":"2024-03-06T07:28:31.780050+0000","last_unstale":"2024-03-06T07:28:34.121486+0000","last_undegraded":"2024-03-06T07:28:34.121486+0000","last_fullsized":"2024-03-06T07:28:34.121486+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:24.572609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121395+0000","last_change":"2024-03-06T07:28:32.803853+0000","last_active":"2024-03-06T07:28:34.121395+0000","last_peered":"2024-03-06T07:28:34.121395+0000","last_clean":"2024-03-06T07:28:34.121395+0000","last_became_active":"2024-03-06T07:28:32.803200+0000","last_became_peered":"2024-03-06T07:28:32.803200+0000","last_unstale":"2024-03-06T07:28:34.121395+0000","last_undegraded":"2024-03-06T07:28:34.121395+0000","last_fullsized":"2024-03-06T07:28:34.121395+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:37.420743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.796027+0000","last_change":"2024-03-06T07:28:31.773233+0000","last_active":"2024-03-06T07:28:33.796027+0000","last_peered":"2024-03-06T07:28:33.796027+0000","last_clean":"2024-03-06T07:28:33.796027+0000","last_became_active":"2024-03-06T07:28:31.773028+0000","last_became_peered":"2024-03-06T07:28:31.773028+0000","last_unstale":"2024-03-06T07:28:33.796027+0000","last_undegraded":"2024-03-06T07:28:33.796027+0000","last_fullsized":"2024-03-06T07:28:33.796027+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:22.894378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122131+0000","last_change":"2024-03-06T07:28:32.804070+0000","last_active":"2024-03-06T07:28:34.122131+0000","last_peered":"2024-03-06T07:28:34.122131+0000","last_clean":"2024-03-06T07:28:34.122131+0000","last_became_active":"2024-03-06T07:28:32.803301+0000","last_became_peered":"2024-03-06T07:28:32.803301+0000","last_unstale":"2024-03-06T07:28:34.122131+0000","last_undegraded":"2024-03-06T07:28:34.122131+0000","last_fullsized":"2024-03-06T07:28:34.122131+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:07.020527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.796963+0000","last_change":"2024-03-06T07:28:31.782644+0000","last_active":"2024-03-06T07:28:33.796963+0000","last_peered":"2024-03-06T07:28:33.796963+0000","last_clean":"2024-03-06T07:28:33.796963+0000","last_became_active":"2024-03-06T07:28:31.782362+0000","last_became_peered":"2024-03-06T07:28:31.782362+0000","last_unstale":"2024-03-06T07:28:33.796963+0000","last_undegraded":"2024-03-06T07:28:33.796963+0000","last_fullsized":"2024-03-06T07:28:33.796963+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:40.274353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122985+0000","last_change":"2024-03-06T07:28:32.808107+0000","last_active":"2024-03-06T07:28:34.122985+0000","last_peered":"2024-03-06T07:28:34.122985+0000","last_clean":"2024-03-06T07:28:34.122985+0000","last_became_active":"2024-03-06T07:28:32.807772+0000","last_became_peered":"2024-03-06T07:28:32.807772+0000","last_unstale":"2024-03-06T07:28:34.122985+0000","last_undegraded":"2024-03-06T07:28:34.122985+0000","last_fullsized":"2024-03-06T07:28:34.122985+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:30.047570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121199+0000","last_change":"2024-03-06T07:28:31.783136+0000","last_active":"2024-03-06T07:28:34.121199+0000","last_peered":"2024-03-06T07:28:34.121199+0000","last_clean":"2024-03-06T07:28:34.121199+0000","last_became_active":"2024-03-06T07:28:31.782350+0000","last_became_peered":"2024-03-06T07:28:31.782350+0000","last_unstale":"2024-03-06T07:28:34.121199+0000","last_undegraded":"2024-03-06T07:28:34.121199+0000","last_fullsized":"2024-03-06T07:28:34.121199+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:13.693786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121806+0000","last_change":"2024-03-06T07:28:32.808298+0000","last_active":"2024-03-06T07:28:34.121806+0000","last_peered":"2024-03-06T07:28:34.121806+0000","last_clean":"2024-03-06T07:28:34.121806+0000","last_became_active":"2024-03-06T07:28:32.808051+0000","last_became_peered":"2024-03-06T07:28:32.808051+0000","last_unstale":"2024-03-06T07:28:34.121806+0000","last_undegraded":"2024-03-06T07:28:34.121806+0000","last_fullsized":"2024-03-06T07:28:34.121806+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:01.998448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121118+0000","last_change":"2024-03-06T07:28:31.779718+0000","last_active":"2024-03-06T07:28:34.121118+0000","last_peered":"2024-03-06T07:28:34.121118+0000","last_clean":"2024-03-06T07:28:34.121118+0000","last_became_active":"2024-03-06T07:28:31.779454+0000","last_became_peered":"2024-03-06T07:28:31.779454+0000","last_unstale":"2024-03-06T07:28:34.121118+0000","last_undegraded":"2024-03-06T07:28:34.121118+0000","last_fullsized":"2024-03-06T07:28:34.121118+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:19.087135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797445+0000","last_change":"2024-03-06T07:28:32.800573+0000","last_active":"2024-03-06T07:28:33.797445+0000","last_peered":"2024-03-06T07:28:33.797445+0000","last_clean":"2024-03-06T07:28:33.797445+0000","last_became_active":"2024-03-06T07:28:32.800304+0000","last_became_peered":"2024-03-06T07:28:32.800304+0000","last_unstale":"2024-03-06T07:28:33.797445+0000","last_undegraded":"2024-03-06T07:28:33.797445+0000","last_fullsized":"2024-03-06T07:28:33.797445+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:27.000111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121717+0000","last_change":"2024-03-06T07:28:31.780216+0000","last_active":"2024-03-06T07:28:34.121717+0000","last_peered":"2024-03-06T07:28:34.121717+0000","last_clean":"2024-03-06T07:28:34.121717+0000","last_became_active":"2024-03-06T07:28:31.779950+0000","last_became_peered":"2024-03-06T07:28:31.779950+0000","last_unstale":"2024-03-06T07:28:34.121717+0000","last_undegraded":"2024-03-06T07:28:34.121717+0000","last_fullsized":"2024-03-06T07:28:34.121717+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:05.435804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121383+0000","last_change":"2024-03-06T07:28:32.798788+0000","last_active":"2024-03-06T07:28:34.121383+0000","last_peered":"2024-03-06T07:28:34.121383+0000","last_clean":"2024-03-06T07:28:34.121383+0000","last_became_active":"2024-03-06T07:28:32.798517+0000","last_became_peered":"2024-03-06T07:28:32.798517+0000","last_unstale":"2024-03-06T07:28:34.121383+0000","last_undegraded":"2024-03-06T07:28:34.121383+0000","last_fullsized":"2024-03-06T07:28:34.121383+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:07.018319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121124+0000","last_change":"2024-03-06T07:28:31.784524+0000","last_active":"2024-03-06T07:28:34.121124+0000","last_peered":"2024-03-06T07:28:34.121124+0000","last_clean":"2024-03-06T07:28:34.121124+0000","last_became_active":"2024-03-06T07:28:31.784259+0000","last_became_peered":"2024-03-06T07:28:31.784259+0000","last_unstale":"2024-03-06T07:28:34.121124+0000","last_undegraded":"2024-03-06T07:28:34.121124+0000","last_fullsized":"2024-03-06T07:28:34.121124+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:08.434967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804414+0000","last_change":"2024-03-06T07:28:32.803808+0000","last_active":"2024-03-06T07:28:33.804414+0000","last_peered":"2024-03-06T07:28:33.804414+0000","last_clean":"2024-03-06T07:28:33.804414+0000","last_became_active":"2024-03-06T07:28:32.803265+0000","last_became_peered":"2024-03-06T07:28:32.803265+0000","last_unstale":"2024-03-06T07:28:33.804414+0000","last_undegraded":"2024-03-06T07:28:33.804414+0000","last_fullsized":"2024-03-06T07:28:33.804414+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:36.303510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.795583+0000","last_change":"2024-03-06T07:28:31.772492+0000","last_active":"2024-03-06T07:28:33.795583+0000","last_peered":"2024-03-06T07:28:33.795583+0000","last_clean":"2024-03-06T07:28:33.795583+0000","last_became_active":"2024-03-06T07:28:31.772186+0000","last_became_peered":"2024-03-06T07:28:31.772186+0000","last_unstale":"2024-03-06T07:28:33.795583+0000","last_undegraded":"2024-03-06T07:28:33.795583+0000","last_fullsized":"2024-03-06T07:28:33.795583+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:07.934135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.797650+0000","last_change":"2024-03-06T07:28:32.813099+0000","last_active":"2024-03-06T07:28:33.797650+0000","last_peered":"2024-03-06T07:28:33.797650+0000","last_clean":"2024-03-06T07:28:33.797650+0000","last_became_active":"2024-03-06T07:28:32.812824+0000","last_became_peered":"2024-03-06T07:28:32.812824+0000","last_unstale":"2024-03-06T07:28:33.797650+0000","last_undegraded":"2024-03-06T07:28:33.797650+0000","last_fullsized":"2024-03-06T07:28:33.797650+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:09.436899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122601+0000","last_change":"2024-03-06T07:28:31.779645+0000","last_active":"2024-03-06T07:28:34.122601+0000","last_peered":"2024-03-06T07:28:34.122601+0000","last_clean":"2024-03-06T07:28:34.122601+0000","last_became_active":"2024-03-06T07:28:31.779391+0000","last_became_peered":"2024-03-06T07:28:31.779391+0000","last_unstale":"2024-03-06T07:28:34.122601+0000","last_undegraded":"2024-03-06T07:28:34.122601+0000","last_fullsized":"2024-03-06T07:28:34.122601+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:58.373735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122249+0000","last_change":"2024-03-06T07:28:32.813966+0000","last_active":"2024-03-06T07:28:34.122249+0000","last_peered":"2024-03-06T07:28:34.122249+0000","last_clean":"2024-03-06T07:28:34.122249+0000","last_became_active":"2024-03-06T07:28:32.813579+0000","last_became_peered":"2024-03-06T07:28:32.813579+0000","last_unstale":"2024-03-06T07:28:34.122249+0000","last_undegraded":"2024-03-06T07:28:34.122249+0000","last_fullsized":"2024-03-06T07:28:34.122249+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:54.528309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"125'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804659+0000","last_change":"2024-03-06T07:28:31.779359+0000","last_active":"2024-03-06T07:28:33.804659+0000","last_peered":"2024-03-06T07:28:33.804659+0000","last_clean":"2024-03-06T07:28:33.804659+0000","last_became_active":"2024-03-06T07:28:31.779096+0000","last_became_peered":"2024-03-06T07:28:31.779096+0000","last_unstale":"2024-03-06T07:28:33.804659+0000","last_undegraded":"2024-03-06T07:28:33.804659+0000","last_fullsized":"2024-03-06T07:28:33.804659+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:56.394867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122517+0000","last_change":"2024-03-06T07:28:32.800953+0000","last_active":"2024-03-06T07:28:34.122517+0000","last_peered":"2024-03-06T07:28:34.122517+0000","last_clean":"2024-03-06T07:28:34.122517+0000","last_became_active":"2024-03-06T07:28:32.800688+0000","last_became_peered":"2024-03-06T07:28:32.800688+0000","last_unstale":"2024-03-06T07:28:34.122517+0000","last_undegraded":"2024-03-06T07:28:34.122517+0000","last_fullsized":"2024-03-06T07:28:34.122517+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:24.938692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.123034+0000","last_change":"2024-03-06T07:28:31.778300+0000","last_active":"2024-03-06T07:28:34.123034+0000","last_peered":"2024-03-06T07:28:34.123034+0000","last_clean":"2024-03-06T07:28:34.123034+0000","last_became_active":"2024-03-06T07:28:31.778036+0000","last_became_peered":"2024-03-06T07:28:31.778036+0000","last_unstale":"2024-03-06T07:28:34.123034+0000","last_undegraded":"2024-03-06T07:28:34.123034+0000","last_fullsized":"2024-03-06T07:28:34.123034+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:52.764448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.796850+0000","last_change":"2024-03-06T07:28:32.806621+0000","last_active":"2024-03-06T07:28:33.796850+0000","last_peered":"2024-03-06T07:28:33.796850+0000","last_clean":"2024-03-06T07:28:33.796850+0000","last_became_active":"2024-03-06T07:28:32.806296+0000","last_became_peered":"2024-03-06T07:28:32.806296+0000","last_unstale":"2024-03-06T07:28:33.796850+0000","last_undegraded":"2024-03-06T07:28:33.796850+0000","last_fullsized":"2024-03-06T07:28:33.796850+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:31.777440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.795847+0000","last_change":"2024-03-06T07:28:31.779932+0000","last_active":"2024-03-06T07:28:33.795847+0000","last_peered":"2024-03-06T07:28:33.795847+0000","last_clean":"2024-03-06T07:28:33.795847+0000","last_became_active":"2024-03-06T07:28:31.779668+0000","last_became_peered":"2024-03-06T07:28:31.779668+0000","last_unstale":"2024-03-06T07:28:33.795847+0000","last_undegraded":"2024-03-06T07:28:33.795847+0000","last_fullsized":"2024-03-06T07:28:33.795847+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:08.406235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804507+0000","last_change":"2024-03-06T07:28:32.804110+0000","last_active":"2024-03-06T07:28:33.804507+0000","last_peered":"2024-03-06T07:28:33.804507+0000","last_clean":"2024-03-06T07:28:33.804507+0000","last_became_active":"2024-03-06T07:28:32.803858+0000","last_became_peered":"2024-03-06T07:28:32.803858+0000","last_unstale":"2024-03-06T07:28:33.804507+0000","last_undegraded":"2024-03-06T07:28:33.804507+0000","last_fullsized":"2024-03-06T07:28:33.804507+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:59:20.004678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122683+0000","last_change":"2024-03-06T07:28:31.777451+0000","last_active":"2024-03-06T07:28:34.122683+0000","last_peered":"2024-03-06T07:28:34.122683+0000","last_clean":"2024-03-06T07:28:34.122683+0000","last_became_active":"2024-03-06T07:28:31.777201+0000","last_became_peered":"2024-03-06T07:28:31.777201+0000","last_unstale":"2024-03-06T07:28:34.122683+0000","last_undegraded":"2024-03-06T07:28:34.122683+0000","last_fullsized":"2024-03-06T07:28:34.122683+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:58.745835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121150+0000","last_change":"2024-03-06T07:28:32.803639+0000","last_active":"2024-03-06T07:28:34.121150+0000","last_peered":"2024-03-06T07:28:34.121150+0000","last_clean":"2024-03-06T07:28:34.121150+0000","last_became_active":"2024-03-06T07:28:32.803343+0000","last_became_peered":"2024-03-06T07:28:32.803343+0000","last_unstale":"2024-03-06T07:28:34.121150+0000","last_undegraded":"2024-03-06T07:28:34.121150+0000","last_fullsized":"2024-03-06T07:28:34.121150+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:06.064690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804630+0000","last_change":"2024-03-06T07:28:31.781199+0000","last_active":"2024-03-06T07:28:33.804630+0000","last_peered":"2024-03-06T07:28:33.804630+0000","last_clean":"2024-03-06T07:28:33.804630+0000","last_became_active":"2024-03-06T07:28:31.780846+0000","last_became_peered":"2024-03-06T07:28:31.780846+0000","last_unstale":"2024-03-06T07:28:33.804630+0000","last_undegraded":"2024-03-06T07:28:33.804630+0000","last_fullsized":"2024-03-06T07:28:33.804630+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:19.219570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122457+0000","last_change":"2024-03-06T07:28:32.805077+0000","last_active":"2024-03-06T07:28:34.122457+0000","last_peered":"2024-03-06T07:28:34.122457+0000","last_clean":"2024-03-06T07:28:34.122457+0000","last_became_active":"2024-03-06T07:28:32.804811+0000","last_became_peered":"2024-03-06T07:28:32.804811+0000","last_unstale":"2024-03-06T07:28:34.122457+0000","last_undegraded":"2024-03-06T07:28:34.122457+0000","last_fullsized":"2024-03-06T07:28:34.122457+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:17:52.162173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.121813+0000","last_change":"2024-03-06T07:28:31.780249+0000","last_active":"2024-03-06T07:28:34.121813+0000","last_peered":"2024-03-06T07:28:34.121813+0000","last_clean":"2024-03-06T07:28:34.121813+0000","last_became_active":"2024-03-06T07:28:31.779962+0000","last_became_peered":"2024-03-06T07:28:31.779962+0000","last_unstale":"2024-03-06T07:28:34.121813+0000","last_undegraded":"2024-03-06T07:28:34.121813+0000","last_fullsized":"2024-03-06T07:28:34.121813+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:17.108227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:34.122242+0000","last_change":"2024-03-06T07:28:32.801349+0000","last_active":"2024-03-06T07:28:34.122242+0000","last_peered":"2024-03-06T07:28:34.122242+0000","last_clean":"2024-03-06T07:28:34.122242+0000","last_became_active":"2024-03-06T07:28:32.801102+0000","last_became_peered":"2024-03-06T07:28:32.801102+0000","last_unstale":"2024-03-06T07:28:34.122242+0000","last_undegraded":"2024-03-06T07:28:34.122242+0000","last_fullsized":"2024-03-06T07:28:34.122242+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:11.431453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.795724+0000","last_change":"2024-03-06T07:28:31.773134+0000","last_active":"2024-03-06T07:28:33.795724+0000","last_peered":"2024-03-06T07:28:33.795724+0000","last_clean":"2024-03-06T07:28:33.795724+0000","last_became_active":"2024-03-06T07:28:31.772882+0000","last_became_peered":"2024-03-06T07:28:31.772882+0000","last_unstale":"2024-03-06T07:28:33.795724+0000","last_undegraded":"2024-03-06T07:28:33.795724+0000","last_fullsized":"2024-03-06T07:28:33.795724+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:40.050711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804769+0000","last_change":"2024-03-06T07:28:32.803959+0000","last_active":"2024-03-06T07:28:33.804769+0000","last_peered":"2024-03-06T07:28:33.804769+0000","last_clean":"2024-03-06T07:28:33.804769+0000","last_became_active":"2024-03-06T07:28:32.803668+0000","last_became_peered":"2024-03-06T07:28:32.803668+0000","last_unstale":"2024-03-06T07:28:33.804769+0000","last_undegraded":"2024-03-06T07:28:33.804769+0000","last_fullsized":"2024-03-06T07:28:33.804769+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:31.761490+0000","last_clean_scrub_stamp":"2024-03-06T07:28:31.761490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:02.777791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804640+0000","last_change":"2024-03-06T07:28:31.780614+0000","last_active":"2024-03-06T07:28:33.804640+0000","last_peered":"2024-03-06T07:28:33.804640+0000","last_clean":"2024-03-06T07:28:33.804640+0000","last_became_active":"2024-03-06T07:28:31.780312+0000","last_became_peered":"2024-03-06T07:28:31.780312+0000","last_unstale":"2024-03-06T07:28:33.804640+0000","last_undegraded":"2024-03-06T07:28:33.804640+0000","last_fullsized":"2024-03-06T07:28:33.804640+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:30.759558+0000","last_clean_scrub_stamp":"2024-03-06T07:28:30.759558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:59.115102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":354,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:33.804331+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:28:33.804331+0000","last_peered":"2024-03-06T07:28:33.804331+0000","last_clean":"2024-03-06T07:28:33.804331+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:28:33.804331+0000","last_undegraded":"2024-03-06T07:28:33.804331+0000","last_fullsized":"2024-03-06T07:28:33.804331+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574932,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41636,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330204,"statfs":{"total":96636764160,"available":96594128896,"internally_reserved":0,"allocated":2179072,"data_stored":1272416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28556,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343284,"statfs":{"total":96636764160,"available":96607522816,"internally_reserved":0,"allocated":1572864,"data_stored":681410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41032,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330808,"statfs":{"total":96636764160,"available":96594747392,"internally_reserved":0,"allocated":1568768,"data_stored":679910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574931,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28552,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343288,"statfs":{"total":96636764160,"available":96607526912,"internally_reserved":0,"allocated":1564672,"data_stored":679868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41032,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330808,"statfs":{"total":96636764160,"available":96594747392,"internally_reserved":0,"allocated":1568768,"data_stored":679958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41032,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330808,"statfs":{"total":96636764160,"available":96594747392,"internally_reserved":0,"allocated":1568768,"data_stored":680414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1560576,"data_stored":679798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41604,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330236,"statfs":{"total":96636764160,"available":96594161664,"internally_reserved":0,"allocated":2154496,"data_stored":1270166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:40.448 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:28:40.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:28:40.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.590+0000 7fde9153a640 1 Processor -- start 2024-03-06T07:28:40.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.590+0000 7fde9153a640 1 -- start start 2024-03-06T07:28:40.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde9153a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fde8c1614d0 0x7fde8c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde9153a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde9153a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde8c0d22b0 0x7fde8c0d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde9153a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8c0d2740 con 0x7fde8c0d22b0 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde9153a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8c1618b0 con 0x7fde8c1614d0 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde9153a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0cf400 con 0x7fde8c0d1910 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde8c0d22b0 0x7fde8c0d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde8c0d22b0 0x7fde8c0d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42222/0 (socket says 172.21.15.77:42222) 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8b7fe640 1 -- 172.21.15.77:0/3118680762 learned_addr learned my addr 172.21.15.77:0/3118680762 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde827fc640 1 --2- 172.21.15.77:0/3118680762 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8affd640 1 --2- 172.21.15.77:0/3118680762 >> v2:172.21.15.158:3300/0 conn(0x7fde8c1614d0 0x7fde8c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8b7fe640 1 -- 172.21.15.77:0/3118680762 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 msgr2=0x7fde8c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8b7fe640 1 --2- 172.21.15.77:0/3118680762 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8b7fe640 1 -- 172.21.15.77:0/3118680762 >> v2:172.21.15.158:3300/0 conn(0x7fde8c1614d0 msgr2=0x7fde8c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8b7fe640 1 --2- 172.21.15.77:0/3118680762 >> v2:172.21.15.158:3300/0 conn(0x7fde8c1614d0 0x7fde8c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8b7fe640 1 -- 172.21.15.77:0/3118680762 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0cf6a0 con 0x7fde8c0d22b0 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde8affd640 1 --2- 172.21.15.77:0/3118680762 >> v2:172.21.15.158:3300/0 conn(0x7fde8c1614d0 0x7fde8c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.591+0000 7fde827fc640 1 --2- 172.21.15.77:0/3118680762 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:40.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.592+0000 7fde8b7fe640 1 --2- 172.21.15.77:0/3118680762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c0d06a0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fde8c08c470 tx=0x7fde8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9277b82c99dc6425 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.592+0000 7fde8a7fc640 1 -- 172.21.15.77:0/3118680762 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8402ca70 con 0x7fde8c0d22b0 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.592+0000 7fde8a7fc640 1 -- 172.21.15.77:0/3118680762 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8402cc10 con 0x7fde8c0d22b0 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.592+0000 7fde8a7fc640 1 -- 172.21.15.77:0/3118680762 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde840343f0 con 0x7fde8c0d22b0 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 -- 172.21.15.77:0/3118680762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 msgr2=0x7fde8c0d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 --2- 172.21.15.77:0/3118680762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c0d06a0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fde8c08c470 tx=0x7fde8402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 -- 172.21.15.77:0/3118680762 shutdown_connections 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 --2- 172.21.15.77:0/3118680762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c0d06a0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 --2- 172.21.15.77:0/3118680762 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 --2- 172.21.15.77:0/3118680762 >> v2:172.21.15.158:3300/0 conn(0x7fde8c1614d0 0x7fde8c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 -- 172.21.15.77:0/3118680762 >> 172.21.15.77:0/3118680762 conn(0x7fde8c0d3d40 msgr2=0x7fde8c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 -- 172.21.15.77:0/3118680762 shutdown_connections 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.593+0000 7fde9153a640 1 -- 172.21.15.77:0/3118680762 wait complete. 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.594+0000 7fde9153a640 1 Processor -- start 2024-03-06T07:28:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.594+0000 7fde9153a640 1 -- start start 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde9153a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c1f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde9153a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c1f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde9153a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde8c1614d0 0x7fde8c0b7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde9153a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0b0910 con 0x7fde8c0d22b0 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde9153a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0cfc60 con 0x7fde8c1614d0 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde9153a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0d0be0 con 0x7fde8c0d1910 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c1f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c1f6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42232/0 (socket says 172.21.15.77:42232) 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 -- 172.21.15.77:0/1514458046 learned_addr learned my addr 172.21.15.77:0/1514458046 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde8affd640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c1f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde8b7fe640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde8c1614d0 0x7fde8c0b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 -- 172.21.15.77:0/1514458046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 msgr2=0x7fde8c1f50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c1f50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 -- 172.21.15.77:0/1514458046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde8c1614d0 msgr2=0x7fde8c0b7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde8c1614d0 0x7fde8c0b7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 -- 172.21.15.77:0/1514458046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0b7c40 con 0x7fde8c0d22b0 2024-03-06T07:28:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde827fc640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c1f6e50 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fde7400ebe0 tx=0x7fde7400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde83fff640 1 -- 172.21.15.77:0/1514458046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74017070 con 0x7fde8c0d22b0 2024-03-06T07:28:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde83fff640 1 -- 172.21.15.77:0/1514458046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7400cc30 con 0x7fde8c0d22b0 2024-03-06T07:28:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde83fff640 1 -- 172.21.15.77:0/1514458046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74013470 con 0x7fde8c0d22b0 2024-03-06T07:28:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c0caeb0 con 0x7fde8c0d22b0 2024-03-06T07:28:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.595+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c0cb470 con 0x7fde8c0d22b0 2024-03-06T07:28:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.596+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8c068ff0 con 0x7fde8c0d22b0 2024-03-06T07:28:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.597+0000 7fde83fff640 1 -- 172.21.15.77:0/1514458046 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde7401a070 con 0x7fde8c0d22b0 2024-03-06T07:28:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.597+0000 7fde83fff640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.597+0000 7fde83fff640 1 -- 172.21.15.77:0/1514458046 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde7408a450 con 0x7fde8c0d22b0 2024-03-06T07:28:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.597+0000 7fde8affd640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.598+0000 7fde8affd640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde6c06b4d0 0x7fde6c06d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009560 tx=0x7fde7c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.602+0000 7fde83fff640 1 -- 172.21.15.77:0/1514458046 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde74058c80 con 0x7fde8c0d22b0 2024-03-06T07:28:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.843+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fde8c0b4f70 con 0x7fde8c0d22b0 2024-03-06T07:28:40.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.845+0000 7fde83fff640 1 -- 172.21.15.77:0/1514458046 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7fde7405cd20 con 0x7fde8c0d22b0 2024-03-06T07:28:40.847 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:28:40.847 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:28:40.847 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:28:40.847 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:28:40.847 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:28:40.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde6c06b4d0 msgr2=0x7fde6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde6c06b4d0 0x7fde6c06d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009560 tx=0x7fde7c009190 comp rx=0 tx=0).stop 2024-03-06T07:28:40.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 msgr2=0x7fde8c1f6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c1f6e50 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fde7400ebe0 tx=0x7fde7400c520 comp rx=0 tx=0).stop 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 shutdown_connections 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8c0d22b0 0x7fde8c1f6e50 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fde8c1614d0 0x7fde8c0b7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 --2- 172.21.15.77:0/1514458046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fde8c0d1910 0x7fde8c1f50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 >> 172.21.15.77:0/1514458046 conn(0x7fde8c0d3d40 msgr2=0x7fde8c0d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 shutdown_connections 2024-03-06T07:28:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.847+0000 7fde9153a640 1 -- 172.21.15.77:0/1514458046 wait complete. 2024-03-06T07:28:40.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:40.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.958+0000 7f90da5f7640 1 Processor -- start 2024-03-06T07:28:40.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.959+0000 7f90da5f7640 1 -- start start 2024-03-06T07:28:40.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.959+0000 7f90da5f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d41424f0 0x7f90d40cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.959+0000 7f90da5f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f90d406da40 0x7f90d415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.959+0000 7f90da5f7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d40cc980 con 0x7f90d40cd7a0 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.959+0000 7f90da5f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d40cd220 con 0x7f90d406da40 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.959+0000 7f90da5f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d40ccc60 con 0x7f90d41424f0 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d8b94640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f90d406da40 0x7f90d415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d8b94640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f90d406da40 0x7f90d415cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42246/0 (socket says 172.21.15.77:42246) 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f90d40cd7a0 0x7f90d406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46664/0 (socket says 172.21.15.77:46664) 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d8b94640 1 -- 172.21.15.77:0/4076779423 learned_addr learned my addr 172.21.15.77:0/4076779423 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d3fff640 1 --2- 172.21.15.77:0/4076779423 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d41424f0 0x7f90d40cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d8b94640 1 -- 172.21.15.77:0/4076779423 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d41424f0 msgr2=0x7f90d40cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d8b94640 1 --2- 172.21.15.77:0/4076779423 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d41424f0 0x7f90d40cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d8b94640 1 -- 172.21.15.77:0/4076779423 >> v1:172.21.15.77:6789/0 conn(0x7f90d40cd7a0 legacy=0x7f90d406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d8b94640 1 -- 172.21.15.77:0/4076779423 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d40ccf40 con 0x7f90d406da40 2024-03-06T07:28:40.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.960+0000 7f90d2ffd640 1 -- 172.21.15.77:0/4076779423 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870709068 0 0) 0x7f90d40cc980 con 0x7f90d40cd7a0 2024-03-06T07:28:40.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90d8b94640 1 --2- 172.21.15.77:0/4076779423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d406da40 0x7f90d415cef0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009e50 tx=0x7f90cc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4976e3b31321cba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90d2ffd640 1 -- 172.21.15.77:0/4076779423 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc02cb00 con 0x7f90d406da40 2024-03-06T07:28:40.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90d2ffd640 1 -- 172.21.15.77:0/4076779423 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc02cca0 con 0x7f90d406da40 2024-03-06T07:28:40.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90d2ffd640 1 -- 172.21.15.77:0/4076779423 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc034610 con 0x7f90d406da40 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90da5f7640 1 -- 172.21.15.77:0/4076779423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d406da40 msgr2=0x7f90d415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90da5f7640 1 --2- 172.21.15.77:0/4076779423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d406da40 0x7f90d415cef0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009e50 tx=0x7f90cc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90da5f7640 1 -- 172.21.15.77:0/4076779423 shutdown_connections 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90da5f7640 1 --2- 172.21.15.77:0/4076779423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d406da40 0x7f90d415cef0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90da5f7640 1 --2- 172.21.15.77:0/4076779423 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d41424f0 0x7f90d40cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90da5f7640 1 -- 172.21.15.77:0/4076779423 >> 172.21.15.77:0/4076779423 conn(0x7f90d40cf8a0 msgr2=0x7f90d40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90da5f7640 1 -- 172.21.15.77:0/4076779423 shutdown_connections 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.961+0000 7f90da5f7640 1 -- 172.21.15.77:0/4076779423 wait complete. 2024-03-06T07:28:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.962+0000 7f90da5f7640 1 Processor -- start 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90da5f7640 1 -- start start 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90da5f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d406da40 0x7f90d415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90da5f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90d40cd7a0 0x7f90d4140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90da5f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d41424f0 0x7f90d4141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90da5f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4071ce0 con 0x7f90d41424f0 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90da5f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4071b60 con 0x7f90d40cd7a0 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90da5f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4071e60 con 0x7f90d406da40 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90d8b94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d41424f0 0x7f90d4141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90d8b94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d41424f0 0x7f90d4141070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42254/0 (socket says 172.21.15.77:42254) 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90d8b94640 1 -- 172.21.15.77:0/262719091 learned_addr learned my addr 172.21.15.77:0/262719091 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.963+0000 7f90d3fff640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d406da40 0x7f90d415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d37fe640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90d40cd7a0 0x7f90d4140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d8b94640 1 -- 172.21.15.77:0/262719091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d406da40 msgr2=0x7f90d415b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d8b94640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d406da40 0x7f90d415b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d8b94640 1 -- 172.21.15.77:0/262719091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90d40cd7a0 msgr2=0x7f90d4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d8b94640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90d40cd7a0 0x7f90d4140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d8b94640 1 -- 172.21.15.77:0/262719091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4142e70 con 0x7f90d41424f0 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d8b94640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d41424f0 0x7f90d4141070 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009e50 tx=0x7f90cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d0ff9640 1 -- 172.21.15.77:0/262719091 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc043020 con 0x7f90d41424f0 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d0ff9640 1 -- 172.21.15.77:0/262719091 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc02ce50 con 0x7f90d41424f0 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d4143150 con 0x7f90d41424f0 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90d0ff9640 1 -- 172.21.15.77:0/262719091 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc009510 con 0x7f90d41424f0 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.964+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d4143710 con 0x7f90d41424f0 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.965+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d404f330 con 0x7f90d41424f0 2024-03-06T07:28:40.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.965+0000 7f90d0ff9640 1 -- 172.21.15.77:0/262719091 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90cc046020 con 0x7f90d41424f0 2024-03-06T07:28:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.966+0000 7f90d0ff9640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90c006b4d0 0x7f90c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.966+0000 7f90d0ff9640 1 -- 172.21.15.77:0/262719091 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90cc0ac640 con 0x7f90d41424f0 2024-03-06T07:28:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.966+0000 7f90d3fff640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90c006b4d0 0x7f90c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.966+0000 7f90d3fff640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90c006b4d0 0x7f90c006d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009aa0 tx=0x7f90c8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:40.970+0000 7f90d0ff9640 1 -- 172.21.15.77:0/262719091 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90cc04d030 con 0x7f90d41424f0 2024-03-06T07:28:41.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.178+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90d40d0620 con 0x7f90d41424f0 2024-03-06T07:28:41.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.179+0000 7f90d0ff9640 1 -- 172.21.15.77:0/262719091 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f90cc07ae70 con 0x7f90d41424f0 2024-03-06T07:28:41.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:41.181 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":80,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6731,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/2348087487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":2348087487},{"type":"v1","addr":"172.21.15.158:6834","nonce":2348087487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/238243004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":238243004},{"type":"v1","addr":"172.21.15.77:6835","nonce":238243004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:34.142346+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:28:41.181 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 80 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90c006b4d0 msgr2=0x7f90c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90c006b4d0 0x7f90c006d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009aa0 tx=0x7f90c8009190 comp rx=0 tx=0).stop 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d41424f0 msgr2=0x7f90d4141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d41424f0 0x7f90d4141070 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009e50 tx=0x7f90cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 shutdown_connections 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90c006b4d0 0x7f90c006d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90d41424f0 0x7f90d4141070 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90d40cd7a0 0x7f90d4140b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 --2- 172.21.15.77:0/262719091 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90d406da40 0x7f90d415b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 >> 172.21.15.77:0/262719091 conn(0x7f90d40cf8a0 msgr2=0x7f90d415dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 shutdown_connections 2024-03-06T07:28:41.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.181+0000 7f90da5f7640 1 -- 172.21.15.77:0/262719091 wait complete. 2024-03-06T07:28:41.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:28:34.142346+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8547}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:28:41.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:41.596 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:28:41.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.698+0000 7faa6d5a0640 1 Processor -- start 2024-03-06T07:28:41.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.698+0000 7faa6d5a0640 1 -- start start 2024-03-06T07:28:41.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa6d5a0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faa680cff00 0x7faa6806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa6d5a0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faa6806d640 0x7faa680ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa6d5a0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6815cc30 con 0x7faa68156250 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa6d5a0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6815c9f0 con 0x7faa680cff00 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa6d5a0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6815cf10 con 0x7faa6806d640 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa677fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faa680cff00 0x7faa6806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa677fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faa680cff00 0x7faa6806d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42256/0 (socket says 172.21.15.77:42256) 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa677fe640 1 -- 172.21.15.77:0/3940527556 learned_addr learned my addr 172.21.15.77:0/3940527556 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.699+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/3940527556 >> v2:172.21.15.158:3300/0 conn(0x7faa6806d640 0x7faa680ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.700+0000 7faa677fe640 1 -- 172.21.15.77:0/3940527556 >> v2:172.21.15.158:3300/0 conn(0x7faa6806d640 msgr2=0x7faa680ae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.700+0000 7faa677fe640 1 --2- 172.21.15.77:0/3940527556 >> v2:172.21.15.158:3300/0 conn(0x7faa6806d640 0x7faa680ae7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.700+0000 7faa677fe640 1 -- 172.21.15.77:0/3940527556 >> v1:172.21.15.158:6789/0 conn(0x7faa68156250 legacy=0x7faa6815c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.700+0000 7faa677fe640 1 -- 172.21.15.77:0/3940527556 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6815d1f0 con 0x7faa680cff00 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.700+0000 7faa677fe640 1 --2- 172.21.15.77:0/3940527556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6806d100 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7faa680cd3f0 tx=0x7faa5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7818ffe5826ab15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.700+0000 7faa66ffd640 1 -- 172.21.15.77:0/3940527556 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58034070 con 0x7faa680cff00 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.700+0000 7faa66ffd640 1 -- 172.21.15.77:0/3940527556 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa58030860 con 0x7faa680cff00 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.700+0000 7faa66ffd640 1 -- 172.21.15.77:0/3940527556 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58030c30 con 0x7faa680cff00 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.701+0000 7faa6d5a0640 1 -- 172.21.15.77:0/3940527556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 msgr2=0x7faa6806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.701+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/3940527556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6806d100 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7faa680cd3f0 tx=0x7faa5802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.701+0000 7faa6d5a0640 1 -- 172.21.15.77:0/3940527556 shutdown_connections 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.701+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/3940527556 >> v2:172.21.15.158:3300/0 conn(0x7faa6806d640 0x7faa680ae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.701+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/3940527556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6806d100 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.701+0000 7faa6d5a0640 1 -- 172.21.15.77:0/3940527556 >> 172.21.15.77:0/3940527556 conn(0x7faa6806afb0 msgr2=0x7faa6806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.701+0000 7faa6d5a0640 1 -- 172.21.15.77:0/3940527556 shutdown_connections 2024-03-06T07:28:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.701+0000 7faa6d5a0640 1 -- 172.21.15.77:0/3940527556 wait complete. 2024-03-06T07:28:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.702+0000 7faa6d5a0640 1 Processor -- start 2024-03-06T07:28:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.702+0000 7faa6d5a0640 1 -- start start 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa6d5a0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa6806d640 0x7faa681471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa6d5a0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6815ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa6d5a0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa68156250 0x7faa6815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa6d5a0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa68156630 con 0x7faa680cff00 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa6d5a0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6806cbe0 con 0x7faa68156250 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa6d5a0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa681567b0 con 0x7faa6806d640 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6815ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6815ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42264/0 (socket says 172.21.15.77:42264) 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa677fe640 1 -- 172.21.15.77:0/2917024186 learned_addr learned my addr 172.21.15.77:0/2917024186 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa67fff640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa6806d640 0x7faa681471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa68156250 0x7faa6815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa677fe640 1 -- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa6806d640 msgr2=0x7faa681471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa677fe640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa6806d640 0x7faa681471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa677fe640 1 -- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa68156250 msgr2=0x7faa6815b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa677fe640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa68156250 0x7faa6815b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa677fe640 1 -- 172.21.15.77:0/2917024186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa68147730 con 0x7faa680cff00 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.703+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa68156250 0x7faa6815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.704+0000 7faa677fe640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6815ad40 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7faa5802b9e0 tx=0x7faa58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.704+0000 7faa64ff9640 1 -- 172.21.15.77:0/2917024186 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58048070 con 0x7faa680cff00 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.704+0000 7faa64ff9640 1 -- 172.21.15.77:0/2917024186 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa58030660 con 0x7faa680cff00 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.704+0000 7faa6d5a0640 1 -- 172.21.15.77:0/2917024186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa681f7550 con 0x7faa680cff00 2024-03-06T07:28:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.704+0000 7faa64ff9640 1 -- 172.21.15.77:0/2917024186 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58034040 con 0x7faa680cff00 2024-03-06T07:28:41.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.704+0000 7faa6d5a0640 1 -- 172.21.15.77:0/2917024186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa681f7b10 con 0x7faa680cff00 2024-03-06T07:28:41.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.705+0000 7faa64ff9640 1 -- 172.21.15.77:0/2917024186 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa5804b070 con 0x7faa680cff00 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.706+0000 7faa64ff9640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.706+0000 7faa67fff640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.706+0000 7faa64ff9640 1 -- 172.21.15.77:0/2917024186 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa580ac0a0 con 0x7faa680cff00 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.706+0000 7faa67fff640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa5406b4d0 0x7faa5406d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0016e0 tx=0x7faa5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.707+0000 7faa713e2740 1 Processor -- start 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.707+0000 7faa713e2740 1 -- start start 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.707+0000 7faa713e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d522090a20 0x55d522090e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.707+0000 7faa713e2740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d522094a70 0x55d522096510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.707+0000 7faa713e2740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220c0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.707+0000 7faa713e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d521ffa6b0 con 0x55d522090a20 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.707+0000 7faa713e2740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d521fa3a60 con 0x55d5220c07b0 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.707+0000 7faa713e2740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d521ff6750 con 0x55d522094a70 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d522090a20 0x55d522090e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d522090a20 0x55d522090e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42276/0 (socket says 172.21.15.77:42276) 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 -- 172.21.15.77:0/2088510414 learned_addr learned my addr 172.21.15.77:0/2088510414 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa677fe640 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d522094a70 0x55d522096510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220c0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 -- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d522094a70 msgr2=0x55d522096510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d522094a70 0x55d522096510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 -- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 msgr2=0x55d5220c0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220c0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 -- 172.21.15.77:0/2088510414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d521fc1f30 con 0x55d522090a20 2024-03-06T07:28:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220c0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa67fff640 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d522090a20 0x55d522090e00 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00fb70 tx=0x7faa5c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b97c381c3041cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa4dffb640 1 -- 172.21.15.77:0/2088510414 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c026070 con 0x55d522090a20 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa4dffb640 1 -- 172.21.15.77:0/2088510414 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c0067d0 con 0x55d522090a20 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.708+0000 7faa4dffb640 1 -- 172.21.15.77:0/2088510414 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c006ba0 con 0x55d522090a20 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 -- 172.21.15.77:0/2088510414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d522090a20 msgr2=0x55d522090e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d522090a20 0x55d522090e00 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00fb70 tx=0x7faa5c012000 comp rx=0 tx=0).stop 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 -- 172.21.15.77:0/2088510414 shutdown_connections 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220c0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d522094a70 0x55d522096510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 --2- 172.21.15.77:0/2088510414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d522090a20 0x55d522090e00 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 -- 172.21.15.77:0/2088510414 >> 172.21.15.77:0/2088510414 conn(0x55d522092aa0 msgr2=0x55d522094360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 -- 172.21.15.77:0/2088510414 shutdown_connections 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 -- 172.21.15.77:0/2088510414 wait complete. 2024-03-06T07:28:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.709+0000 7faa713e2740 1 -- start start 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa713e2740 1 Processor -- start 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa713e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220c07b0 0x55d5220dc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa713e2740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220da860 0x55d5220b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa713e2740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 0x55d5220c1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa713e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d522094f00 con 0x55d5220c07b0 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa713e2740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d521fb6ea0 con 0x55d5220da860 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa713e2740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d5220951e0 con 0x55d5220c14c0 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa713e2740 10 client.0 _ll_register_callbacks cb 0x55d521fb6870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220c07b0 0x55d5220dc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220c07b0 0x55d5220dc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42280/0 (socket says 172.21.15.77:42280) 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa67fff640 1 -- 172.21.15.77:0/1598845181 learned_addr learned my addr 172.21.15.77:0/1598845181 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa677fe640 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220da860 0x55d5220b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.710+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 0x55d5220c1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa67fff640 1 -- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 msgr2=0x55d5220c1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa67fff640 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 0x55d5220c1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa67fff640 1 -- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220da860 msgr2=0x55d5220b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa67fff640 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220da860 0x55d5220b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa67fff640 1 -- 172.21.15.77:0/1598845181 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d521fb7270 con 0x55d5220c07b0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa67fff640 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220c07b0 0x55d5220dc000 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7faa5c001510 tx=0x7faa5c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8ff22656860f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa46ffd640 1 -- 172.21.15.77:0/1598845181 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c026070 con 0x55d5220c07b0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa46ffd640 1 -- 172.21.15.77:0/1598845181 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa5c020830 con 0x55d5220c07b0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 1 -- 172.21.15.77:0/1598845181 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa5c0067d0 con 0x55d5220c07b0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa46ffd640 1 -- 172.21.15.77:0/1598845181 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa5c039050 con 0x55d5220c07b0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 10 client.8700 fetch_fsmap learned FSMap version 80 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 1 -- 172.21.15.77:0/1598845181 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) v3 -- 0x55d521d0f2d0 con 0x55d5220c07b0 2024-03-06T07:28:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa46ffd640 1 -- 172.21.15.77:0/1598845181 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c0067d0 con 0x55d5220c07b0 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa46ffd640 1 -- 172.21.15.77:0/1598845181 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 80) v1 ==== 3314+0+0 (secure 0 0 0) 0x7faa5c03c020 con 0x55d5220c07b0 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 10 client.8700 fetch_fsmap finished waiting for FSMap version 80 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 10 client.8700 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 20 client.8700 populate_metadata read hostname 'smithi077' 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55d5220d9b90 0x55d5220fa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 4 client.8700 mds_command: new command op to 8547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.711+0000 7faa713e2740 1 -- 172.21.15.77:0/1598845181 --> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d522007bf0 con 0x55d5220d9b90 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.712+0000 7faa677fe640 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55d5220d9b90 0x55d5220fa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.712+0000 7faa46ffd640 10 client.8700 ms_handle_connect on v2:172.21.15.158:6833/1245424942 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.712+0000 7faa677fe640 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55d5220d9b90 0x55d5220fa4c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b0359258597594b server_cookie=2d4a12d5ff0da59d in_seq=0 out_seq=0 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa46ffd640 1 -- 172.21.15.77:0/1598845181 <== mds.0 v2:172.21.15.158:6833/1245424942 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d522007bf0 con 0x55d5220d9b90 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa46ffd640 10 client.8700 handle_command_reply: tid=0 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 client.8700 shutdown 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 -- 172.21.15.77:0/1598845181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220c07b0 msgr2=0x55d5220dc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220c07b0 0x55d5220dc000 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7faa5c001510 tx=0x7faa5c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 -- 172.21.15.77:0/1598845181 shutdown_connections 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55d5220d9b90 0x55d5220fa4c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 0x55d5220c1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa46ffd640 0 client.8700 ms_handle_reset on v2:172.21.15.158:6833/1245424942 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220da860 0x55d5220b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 --2- 172.21.15.77:0/1598845181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220c07b0 0x55d5220dc000 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 -- 172.21.15.77:0/1598845181 >> 172.21.15.77:0/1598845181 conn(0x55d522090a20 msgr2=0x55d52208f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 -- 172.21.15.77:0/1598845181 shutdown_connections 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 1 -- 172.21.15.77:0/1598845181 wait complete. 2024-03-06T07:28:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.713+0000 7faa713e2740 20 client.8700 trim_cache size 0 max 16384 2024-03-06T07:28:41.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa713e2740 1 Processor -- start 2024-03-06T07:28:41.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa713e2740 1 -- start start 2024-03-06T07:28:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa713e2740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220be800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa713e2740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 0x55d52208fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa713e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d522090240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa713e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa68200040 con 0x55d5220da860 2024-03-06T07:28:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa713e2740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa681ff1c0 con 0x55d5220c07b0 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa713e2740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa681fe5f0 con 0x55d5220c14c0 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa6cd9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d522090240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa6cd9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d522090240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42296/0 (socket says 172.21.15.77:42296) 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa6cd9f640 1 -- 172.21.15.77:0/1231427245 learned_addr learned my addr 172.21.15.77:0/1231427245 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa67fff640 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220be800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.730+0000 7faa677fe640 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 0x55d52208fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa6cd9f640 1 -- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 msgr2=0x55d52208fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 0x55d52208fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa6cd9f640 1 -- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 msgr2=0x55d5220be800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220be800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa6cd9f640 1 -- 172.21.15.77:0/1231427245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d521dd96b0 con 0x55d5220da860 2024-03-06T07:28:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa67fff640 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220be800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d522090240 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7faa600166f0 tx=0x7faa60009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376611f83eb41775 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa4d7fa640 1 -- 172.21.15.77:0/1231427245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6001c070 con 0x55d5220da860 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa4d7fa640 1 -- 172.21.15.77:0/1231427245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa60007890 con 0x55d5220da860 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa4d7fa640 1 -- 172.21.15.77:0/1231427245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60007c40 con 0x55d5220da860 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa713e2740 1 -- 172.21.15.77:0/1231427245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 msgr2=0x55d522090240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa713e2740 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d522090240 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7faa600166f0 tx=0x7faa60009750 comp rx=0 tx=0).stop 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa713e2740 1 -- 172.21.15.77:0/1231427245 shutdown_connections 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa713e2740 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d522090240 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa713e2740 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c14c0 0x55d52208fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa713e2740 1 --2- 172.21.15.77:0/1231427245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c07b0 0x55d5220be800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa713e2740 1 -- 172.21.15.77:0/1231427245 >> 172.21.15.77:0/1231427245 conn(0x55d522090a20 msgr2=0x55d5220bba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.731+0000 7faa713e2740 1 -- 172.21.15.77:0/1231427245 shutdown_connections 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.732+0000 7faa713e2740 1 -- 172.21.15.77:0/1231427245 wait complete. 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.732+0000 7faa713e2740 1 -- start start 2024-03-06T07:28:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.732+0000 7faa713e2740 1 Processor -- start 2024-03-06T07:28:41.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa713e2740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c07b0 0x55d522090290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa713e2740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c14c0 0x55d5220fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa713e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d5220fc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa713e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5220db290 con 0x55d5220da860 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa713e2740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa681febc0 con 0x55d5220c14c0 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa713e2740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d5220dbae0 con 0x55d5220c07b0 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa713e2740 10 client.0 _ll_register_callbacks cb 0x7faa5c00fa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa6cd9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d5220fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa6cd9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d5220fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42312/0 (socket says 172.21.15.77:42312) 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa6cd9f640 1 -- 172.21.15.77:0/3665231634 learned_addr learned my addr 172.21.15.77:0/3665231634 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa67fff640 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c07b0 0x55d522090290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa677fe640 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c14c0 0x55d5220fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa6cd9f640 1 -- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c07b0 msgr2=0x55d522090290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c07b0 0x55d522090290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa6cd9f640 1 -- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c14c0 msgr2=0x55d5220fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c14c0 0x55d5220fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.733+0000 7faa6cd9f640 1 -- 172.21.15.77:0/3665231634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5220b9dc0 con 0x55d5220da860 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa6cd9f640 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d5220fc290 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7faa600226a0 tx=0x7faa60004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f506cd4c4f094b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa46ffd640 1 -- 172.21.15.77:0/3665231634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6002f060 con 0x55d5220da860 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa46ffd640 1 -- 172.21.15.77:0/3665231634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa60022c90 con 0x55d5220da860 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 1 -- 172.21.15.77:0/3665231634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa60007890 con 0x55d5220da860 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa46ffd640 1 -- 172.21.15.77:0/3665231634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa60035050 con 0x55d5220da860 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 10 client.8706 fetch_fsmap learned FSMap version 80 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 1 -- 172.21.15.77:0/3665231634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) v3 -- 0x55d5220fc890 con 0x55d5220da860 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa46ffd640 1 -- 172.21.15.77:0/3665231634 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa60007890 con 0x55d5220da860 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa46ffd640 1 -- 172.21.15.77:0/3665231634 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 80) v1 ==== 3314+0+0 (secure 0 0 0) 0x7faa60038020 con 0x55d5220da860 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 10 client.8706 fetch_fsmap finished waiting for FSMap version 80 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 10 client.8706 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 20 client.8706 populate_metadata read hostname 'smithi077' 2024-03-06T07:28:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55d5220f4250 0x55d5220f4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 4 client.8706 mds_command: new command op to 8547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa713e2740 1 -- 172.21.15.77:0/3665231634 --> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d522001f40 con 0x55d5220f4250 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.734+0000 7faa67fff640 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55d5220f4250 0x55d5220f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa46ffd640 10 client.8706 ms_handle_connect on v2:172.21.15.158:6833/1245424942 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa67fff640 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55d5220f4250 0x55d5220f4630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f340dff7035fee0d server_cookie=a25bf76a7e33b804 in_seq=0 out_seq=0 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa46ffd640 1 -- 172.21.15.77:0/3665231634 <== mds.0 v2:172.21.15.158:6833/1245424942 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55d522001f40 con 0x55d5220f4250 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa46ffd640 10 client.8706 handle_command_reply: tid=0 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 client.8706 shutdown 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 -- 172.21.15.77:0/3665231634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 msgr2=0x55d5220fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d5220fc290 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7faa600226a0 tx=0x7faa60004ab0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 -- 172.21.15.77:0/3665231634 shutdown_connections 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55d5220f4250 0x55d5220f4630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5220da860 0x55d5220fc290 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55d5220c14c0 0x55d5220fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 --2- 172.21.15.77:0/3665231634 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55d5220c07b0 0x55d522090290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 -- 172.21.15.77:0/3665231634 >> 172.21.15.77:0/3665231634 conn(0x55d522090a20 msgr2=0x55d522094230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa46ffd640 0 client.8706 ms_handle_reset on v2:172.21.15.158:6833/1245424942 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 -- 172.21.15.77:0/3665231634 shutdown_connections 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 1 -- 172.21.15.77:0/3665231634 wait complete. 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.735+0000 7faa713e2740 20 client.8706 trim_cache size 0 max 16384 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:28:41.744 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 8547, 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 8.9377718290000008, 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 80, 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 125, 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 125, 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 14.012059089999999 2024-03-06T07:28:41.745 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 -- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa5406b4d0 msgr2=0x7faa5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa5406b4d0 0x7faa5406d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0016e0 tx=0x7faa5c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 -- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 msgr2=0x7faa6815ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6815ad40 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7faa5802b9e0 tx=0x7faa58004730 comp rx=0 tx=0).stop 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 -- 172.21.15.77:0/2917024186 shutdown_connections 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa680cff00 0x7faa6815ad40 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa68156250 0x7faa6815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 --2- 172.21.15.77:0/2917024186 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa6806d640 0x7faa681471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 -- 172.21.15.77:0/2917024186 >> 172.21.15.77:0/2917024186 conn(0x7faa6806afb0 msgr2=0x7faa6806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 -- 172.21.15.77:0/2917024186 shutdown_connections 2024-03-06T07:28:41.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.736+0000 7faa6d5a0640 1 -- 172.21.15.77:0/2917024186 wait complete. 2024-03-06T07:28:41.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.849+0000 7f3d57864640 1 Processor -- start 2024-03-06T07:28:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.849+0000 7f3d57864640 1 -- start start 2024-03-06T07:28:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d57864640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d500d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d57864640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d500b3430 con 0x7f3d500b41f0 2024-03-06T07:28:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d57864640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d500b3710 con 0x7f3d5015dfe0 2024-03-06T07:28:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d57864640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500d2c50 con 0x7f3d500d22b0 2024-03-06T07:28:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d55600640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3d5015dfe0 0x7f3d500cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:38442/0 (socket says 172.21.15.77:38442) 2024-03-06T07:28:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d54dff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d500d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d55600640 1 -- 172.21.15.77:0/106969100 learned_addr learned my addr 172.21.15.77:0/106969100 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d47fff640 1 -- 172.21.15.77:0/106969100 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304135052 0 0) 0x7f3d500b3710 con 0x7f3d5015dfe0 2024-03-06T07:28:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d47fff640 1 -- 172.21.15.77:0/106969100 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c003520 con 0x7f3d5015dfe0 2024-03-06T07:28:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d47fff640 1 -- 172.21.15.77:0/106969100 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074134001 0 0) 0x7f3d500b3430 con 0x7f3d500b41f0 2024-03-06T07:28:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d47fff640 1 -- 172.21.15.77:0/106969100 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d500b3710 con 0x7f3d500b41f0 2024-03-06T07:28:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d54dff640 1 -- 172.21.15.77:0/106969100 >> v1:172.21.15.158:6789/0 conn(0x7f3d5015dfe0 legacy=0x7f3d500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d54dff640 1 -- 172.21.15.77:0/106969100 >> v1:172.21.15.77:6789/0 conn(0x7f3d500b41f0 legacy=0x7f3d500b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.850+0000 7f3d54dff640 1 -- 172.21.15.77:0/106969100 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500b39f0 con 0x7f3d500d22b0 2024-03-06T07:28:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.851+0000 7f3d54dff640 1 --2- 172.21.15.77:0/106969100 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d500d2710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3d400097d0 tx=0x7f3d4002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85b8e954c6dc04a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.851+0000 7f3d47fff640 1 -- 172.21.15.77:0/106969100 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40035020 con 0x7f3d500d22b0 2024-03-06T07:28:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.851+0000 7f3d47fff640 1 -- 172.21.15.77:0/106969100 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d40031890 con 0x7f3d500d22b0 2024-03-06T07:28:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.851+0000 7f3d47fff640 1 -- 172.21.15.77:0/106969100 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40031c40 con 0x7f3d500d22b0 2024-03-06T07:28:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.852+0000 7f3d57864640 1 -- 172.21.15.77:0/106969100 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 msgr2=0x7f3d500d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.852+0000 7f3d57864640 1 --2- 172.21.15.77:0/106969100 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d500d2710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3d400097d0 tx=0x7f3d4002b930 comp rx=0 tx=0).stop 2024-03-06T07:28:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.852+0000 7f3d57864640 1 -- 172.21.15.77:0/106969100 shutdown_connections 2024-03-06T07:28:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.852+0000 7f3d57864640 1 --2- 172.21.15.77:0/106969100 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d500d2710 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.852+0000 7f3d57864640 1 -- 172.21.15.77:0/106969100 >> 172.21.15.77:0/106969100 conn(0x7f3d500d3cd0 msgr2=0x7f3d500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.852+0000 7f3d57864640 1 -- 172.21.15.77:0/106969100 shutdown_connections 2024-03-06T07:28:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.852+0000 7f3d57864640 1 -- 172.21.15.77:0/106969100 wait complete. 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.853+0000 7f3d57864640 1 Processor -- start 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.853+0000 7f3d57864640 1 -- start start 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d57864640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d500b41f0 0x7f3d500b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d57864640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d5015e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d57864640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d5015dfe0 0x7f3d501605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d57864640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500b3d60 con 0x7f3d500b41f0 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d57864640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500b31e0 con 0x7f3d5015dfe0 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d57864640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500d1c70 con 0x7f3d500d22b0 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55600640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d500b41f0 0x7f3d500b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55600640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d500b41f0 0x7f3d500b1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42314/0 (socket says 172.21.15.77:42314) 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55600640 1 -- 172.21.15.77:0/1582224712 learned_addr learned my addr 172.21.15.77:0/1582224712 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d54dff640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d5015e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55e01640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d5015dfe0 0x7f3d501605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55600640 1 -- 172.21.15.77:0/1582224712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 msgr2=0x7f3d5015e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55600640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d5015e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55600640 1 -- 172.21.15.77:0/1582224712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d5015dfe0 msgr2=0x7f3d501605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55600640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d5015dfe0 0x7f3d501605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.854+0000 7f3d55600640 1 -- 172.21.15.77:0/1582224712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500b1a10 con 0x7f3d500b41f0 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.855+0000 7f3d55600640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d500b41f0 0x7f3d500b1130 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c410 tx=0x7f3d4c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.855+0000 7f3d45ffb640 1 -- 172.21.15.77:0/1582224712 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c017070 con 0x7f3d500b41f0 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.855+0000 7f3d45ffb640 1 -- 172.21.15.77:0/1582224712 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c005ce0 con 0x7f3d500b41f0 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.855+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d500b1cf0 con 0x7f3d500b41f0 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.855+0000 7f3d45ffb640 1 -- 172.21.15.77:0/1582224712 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c0072d0 con 0x7f3d500b41f0 2024-03-06T07:28:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.855+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d500b22b0 con 0x7f3d500b41f0 2024-03-06T07:28:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.856+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5004f330 con 0x7f3d500b41f0 2024-03-06T07:28:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.856+0000 7f3d45ffb640 1 -- 172.21.15.77:0/1582224712 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d4c01a070 con 0x7f3d500b41f0 2024-03-06T07:28:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.857+0000 7f3d45ffb640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d3406b4d0 0x7f3d3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.857+0000 7f3d45ffb640 1 -- 172.21.15.77:0/1582224712 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d4c089500 con 0x7f3d500b41f0 2024-03-06T07:28:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.857+0000 7f3d54dff640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d3406b4d0 0x7f3d3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.857+0000 7f3d54dff640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d3406b4d0 0x7f3d3406d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f3d40036000 tx=0x7f3d40008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:41.861+0000 7f3d45ffb640 1 -- 172.21.15.77:0/1582224712 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d4c057d30 con 0x7f3d500b41f0 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.067+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3d5015d470 con 0x7f3d500b41f0 2024-03-06T07:28:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.068+0000 7f3d45ffb640 1 -- 172.21.15.77:0/1582224712 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v80) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3d4c05bdd0 con 0x7f3d500b41f0 2024-03-06T07:28:42.070 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d3406b4d0 msgr2=0x7f3d3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d3406b4d0 0x7f3d3406d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f3d40036000 tx=0x7f3d40008030 comp rx=0 tx=0).stop 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d500b41f0 msgr2=0x7f3d500b1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d500b41f0 0x7f3d500b1130 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c410 tx=0x7f3d4c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 shutdown_connections 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d5015dfe0 0x7f3d501605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3d500d22b0 0x7f3d5015e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3d3406b4d0 0x7f3d3406d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 --2- 172.21.15.77:0/1582224712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d500b41f0 0x7f3d500b1130 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 >> 172.21.15.77:0/1582224712 conn(0x7f3d500d3cd0 msgr2=0x7f3d500cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 shutdown_connections 2024-03-06T07:28:42.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42.070+0000 7f3d57864640 1 -- 172.21.15.77:0/1582224712 wait complete. 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:42.083 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:42.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.084 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:28:42.099 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:42.099 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:42.099 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:42.099 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:42.099 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:42.100 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42275sec preferred_lft 42275sec 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:28:42.101 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:42.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.102 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:42.165 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:28:42.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.167 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:28:42.222 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:28:42.222 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:42.222 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:42.222 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:42.222 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:42.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.223 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:28:42.223 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:28:42.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:42.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:42.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:42.223 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:28:42.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.318 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:28:42.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:28:42.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:42.333 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:42.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.333 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:28:42.399 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:28:42.399 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:42.399 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:42.454 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:42.454 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:42.454 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:42.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:28:42.575 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:42.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.634 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:28:42.689 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:28:42.689 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:28:42.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:28:42.758 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:28:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:28:42.814 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:42.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.815 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:28:42.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:28:42.814+0000 7fef30042480 -1 init, newargv = 0x55d01928c450 newargc=15 2024-03-06T07:28:42.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[78473]: starting ceph client 2024-03-06T07:28:42.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[78473]: starting fuse 2024-03-06T07:28:42.830 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:28:42.831 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:28:42.831 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:28:42.832 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:28:42.951 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.78473.asok 2024-03-06T07:28:42.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:42.961 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78473.asok 2024-03-06T07:28:42.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78473.asok status 2024-03-06T07:28:43.097 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "78473", 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:28:43.098 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 8721, 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 8721 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 583960351 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:28:43.099 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 583960351 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.8721 192.168.0.1:0/583960351", 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/583960351", 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 80, 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 125, 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:28:43.100 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:28:43.101 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:28:43.101 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:28:43.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:43.106 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:28:43.106 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:43.121 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:28:43.122 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:43.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:43.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:43.189 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:28:43.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.330+0000 7f8e68eaf640 1 Processor -- start 2024-03-06T07:28:43.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.330+0000 7f8e68eaf640 1 -- start start 2024-03-06T07:28:43.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.331+0000 7f8e68eaf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8e641614d0 0x7f8e640d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.331+0000 7f8e68eaf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e641618b0 con 0x7f8e640d1910 2024-03-06T07:28:43.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.331+0000 7f8e68eaf640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e640cf400 con 0x7f8e640d24b0 2024-03-06T07:28:43.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.331+0000 7f8e68eaf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e640cf6a0 con 0x7f8e641614d0 2024-03-06T07:28:43.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.331+0000 7f8e6259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8e641614d0 0x7f8e640d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.331+0000 7f8e61d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8e640d1910 0x7f8e640d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46688/0 (socket says 172.21.15.77:46688) 2024-03-06T07:28:43.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.331+0000 7f8e61d9b640 1 -- 172.21.15.77:0/3151965176 learned_addr learned my addr 172.21.15.77:0/3151965176 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:43.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.332+0000 7f8e6259c640 1 -- 172.21.15.77:0/3151965176 >> v1:172.21.15.158:6789/0 conn(0x7f8e640d24b0 legacy=0x7f8e640b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.332+0000 7f8e6259c640 1 -- 172.21.15.77:0/3151965176 >> v1:172.21.15.77:6789/0 conn(0x7f8e640d1910 legacy=0x7f8e640d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.332+0000 7f8e6259c640 1 -- 172.21.15.77:0/3151965176 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e640cf980 con 0x7f8e641614d0 2024-03-06T07:28:43.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.332+0000 7f8e6159a640 1 -- 172.21.15.77:0/3151965176 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076550308 0 0) 0x7f8e641618b0 con 0x7f8e640d1910 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.332+0000 7f8e6259c640 1 --2- 172.21.15.77:0/3151965176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e641614d0 0x7f8e640d4860 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009d90 tx=0x7f8e5402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390d83f968719071 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.332+0000 7f8e6159a640 1 -- 172.21.15.77:0/3151965176 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54039070 con 0x7f8e641614d0 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e6159a640 1 -- 172.21.15.77:0/3151965176 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5402cd00 con 0x7f8e641614d0 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e6159a640 1 -- 172.21.15.77:0/3151965176 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54034660 con 0x7f8e641614d0 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e68eaf640 1 -- 172.21.15.77:0/3151965176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e641614d0 msgr2=0x7f8e640d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e68eaf640 1 --2- 172.21.15.77:0/3151965176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e641614d0 0x7f8e640d4860 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009d90 tx=0x7f8e5402bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e68eaf640 1 -- 172.21.15.77:0/3151965176 shutdown_connections 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e68eaf640 1 --2- 172.21.15.77:0/3151965176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e641614d0 0x7f8e640d4860 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e68eaf640 1 -- 172.21.15.77:0/3151965176 >> 172.21.15.77:0/3151965176 conn(0x7f8e640d3d40 msgr2=0x7f8e640d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e68eaf640 1 -- 172.21.15.77:0/3151965176 shutdown_connections 2024-03-06T07:28:43.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.333+0000 7f8e68eaf640 1 -- 172.21.15.77:0/3151965176 wait complete. 2024-03-06T07:28:43.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.334+0000 7f8e68eaf640 1 Processor -- start 2024-03-06T07:28:43.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.334+0000 7f8e68eaf640 1 -- start start 2024-03-06T07:28:43.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e68eaf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e640d1910 0x7f8e6415e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e68eaf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e640d24b0 0x7f8e6415a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e68eaf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e641614d0 0x7f8e641f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e68eaf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640b1540 con 0x7f8e640d1910 2024-03-06T07:28:43.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e68eaf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640b0910 con 0x7f8e640d24b0 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e68eaf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640b16c0 con 0x7f8e641614d0 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e6259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e640d1910 0x7f8e6415e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e6259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e640d1910 0x7f8e6415e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42338/0 (socket says 172.21.15.77:42338) 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e6259c640 1 -- 172.21.15.77:0/2870290837 learned_addr learned my addr 172.21.15.77:0/2870290837 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e62d9d640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e641614d0 0x7f8e641f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.335+0000 7f8e61d9b640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e640d24b0 0x7f8e6415a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e6259c640 1 -- 172.21.15.77:0/2870290837 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e641614d0 msgr2=0x7f8e641f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e6259c640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e641614d0 0x7f8e641f3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e6259c640 1 -- 172.21.15.77:0/2870290837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e640d24b0 msgr2=0x7f8e6415a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e6259c640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e640d24b0 0x7f8e6415a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e6259c640 1 -- 172.21.15.77:0/2870290837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e640b7dd0 con 0x7f8e640d1910 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e6259c640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e640d1910 0x7f8e6415e710 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f8e54037040 tx=0x7f8e54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e52ffd640 1 -- 172.21.15.77:0/2870290837 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54039050 con 0x7f8e640d1910 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e52ffd640 1 -- 172.21.15.77:0/2870290837 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e54045070 con 0x7f8e640d1910 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e640b80b0 con 0x7f8e640d1910 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e52ffd640 1 -- 172.21.15.77:0/2870290837 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5403ea50 con 0x7f8e640d1910 2024-03-06T07:28:43.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.336+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e640b8670 con 0x7f8e640d1910 2024-03-06T07:28:43.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.337+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e64068ff0 con 0x7f8e640d1910 2024-03-06T07:28:43.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.338+0000 7f8e52ffd640 1 -- 172.21.15.77:0/2870290837 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e5402cbc0 con 0x7f8e640d1910 2024-03-06T07:28:43.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.338+0000 7f8e52ffd640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e4406b4d0 0x7f8e4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.338+0000 7f8e52ffd640 1 -- 172.21.15.77:0/2870290837 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e540acf40 con 0x7f8e640d1910 2024-03-06T07:28:43.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.338+0000 7f8e61d9b640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e4406b4d0 0x7f8e4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.338+0000 7f8e61d9b640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e4406b4d0 0x7f8e4406d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8e58007f70 tx=0x7f8e58007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.342+0000 7f8e52ffd640 1 -- 172.21.15.77:0/2870290837 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e5407b770 con 0x7f8e640d1910 2024-03-06T07:28:43.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.552+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f8e640c7350 con 0x7f8e640d1910 2024-03-06T07:28:43.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.852+0000 7f8e52ffd640 1 -- 172.21.15.77:0/2870290837 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v81) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8e5407f810 con 0x7f8e640d1910 2024-03-06T07:28:43.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.854+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e4406b4d0 msgr2=0x7f8e4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.854+0000 7f8e68eaf640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e4406b4d0 0x7f8e4406d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8e58007f70 tx=0x7f8e58007f00 comp rx=0 tx=0).stop 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.854+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e640d1910 msgr2=0x7f8e6415e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.854+0000 7f8e68eaf640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e640d1910 0x7f8e6415e710 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f8e54037040 tx=0x7f8e54004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.855+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 shutdown_connections 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.855+0000 7f8e68eaf640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e641614d0 0x7f8e641f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.855+0000 7f8e68eaf640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8e640d24b0 0x7f8e6415a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.855+0000 7f8e68eaf640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e4406b4d0 0x7f8e4406d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.855+0000 7f8e68eaf640 1 --2- 172.21.15.77:0/2870290837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e640d1910 0x7f8e6415e710 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.855+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 >> 172.21.15.77:0/2870290837 conn(0x7f8e640d3d40 msgr2=0x7f8e640ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.855+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 shutdown_connections 2024-03-06T07:28:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.855+0000 7f8e68eaf640 1 -- 172.21.15.77:0/2870290837 wait complete. 2024-03-06T07:28:43.871 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:43.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.969+0000 7f2337837640 1 Processor -- start 2024-03-06T07:28:43.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.969+0000 7f2337837640 1 -- start start 2024-03-06T07:28:43.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f2337837640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f23300b4170 0x7f23300b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f2337837640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23300d25d0 con 0x7f23300d0a80 2024-03-06T07:28:43.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f2337837640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23300d2870 con 0x7f23300d4760 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f2337837640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23300d4bf0 con 0x7f23300b4170 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f2335dd4640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f23300d0a80 0x7f23300cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46700/0 (socket says 172.21.15.77:46700) 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f2335dd4640 1 -- 172.21.15.77:0/1580216880 learned_addr learned my addr 172.21.15.77:0/1580216880 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f23355d3640 1 --2- 172.21.15.77:0/1580216880 >> v2:172.21.15.158:3300/0 conn(0x7f23300b4170 0x7f23300b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f2327fff640 1 -- 172.21.15.77:0/1580216880 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389684211 0 0) 0x7f23300d25d0 con 0x7f23300d0a80 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.970+0000 7f2327fff640 1 -- 172.21.15.77:0/1580216880 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231c003520 con 0x7f23300d0a80 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.971+0000 7f23355d3640 1 -- 172.21.15.77:0/1580216880 >> v1:172.21.15.158:6789/0 conn(0x7f23300d4760 legacy=0x7f23300d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.971+0000 7f23355d3640 1 -- 172.21.15.77:0/1580216880 >> v1:172.21.15.77:6789/0 conn(0x7f23300d0a80 legacy=0x7f23300cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.971+0000 7f23355d3640 1 -- 172.21.15.77:0/1580216880 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23300d2b50 con 0x7f23300b4170 2024-03-06T07:28:43.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.971+0000 7f23355d3640 1 --2- 172.21.15.77:0/1580216880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 0x7f23300b6a90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f232c0097a0 tx=0x7f232c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=691235ccb0cce6a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.971+0000 7f2327fff640 1 -- 172.21.15.77:0/1580216880 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c034070 con 0x7f23300b4170 2024-03-06T07:28:43.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2327fff640 1 -- 172.21.15.77:0/1580216880 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c030610 con 0x7f23300b4170 2024-03-06T07:28:43.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2327fff640 1 -- 172.21.15.77:0/1580216880 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c0309c0 con 0x7f23300b4170 2024-03-06T07:28:43.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2337837640 1 -- 172.21.15.77:0/1580216880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 msgr2=0x7f23300b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2337837640 1 --2- 172.21.15.77:0/1580216880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 0x7f23300b6a90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f232c0097a0 tx=0x7f232c02b960 comp rx=0 tx=0).stop 2024-03-06T07:28:43.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2337837640 1 -- 172.21.15.77:0/1580216880 shutdown_connections 2024-03-06T07:28:43.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2337837640 1 --2- 172.21.15.77:0/1580216880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 0x7f23300b6a90 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2337837640 1 -- 172.21.15.77:0/1580216880 >> 172.21.15.77:0/1580216880 conn(0x7f23300d3c40 msgr2=0x7f23300d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2337837640 1 -- 172.21.15.77:0/1580216880 shutdown_connections 2024-03-06T07:28:43.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.972+0000 7f2337837640 1 -- 172.21.15.77:0/1580216880 wait complete. 2024-03-06T07:28:43.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.973+0000 7f2337837640 1 Processor -- start 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.973+0000 7f2337837640 1 -- start start 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2337837640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 0x7f23301599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2337837640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23300d0a80 0x7f23301567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2337837640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2330156ef0 0x7f233015b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2337837640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23300b4dd0 con 0x7f23300d0a80 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2337837640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23300ce9e0 con 0x7f23300b4170 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2337837640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23300b4f50 con 0x7f2330156ef0 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2334dd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23300d0a80 0x7f23301567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2334dd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23300d0a80 0x7f23301567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42350/0 (socket says 172.21.15.77:42350) 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2334dd2640 1 -- 172.21.15.77:0/465570647 learned_addr learned my addr 172.21.15.77:0/465570647 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f23355d3640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 0x7f23301599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2335dd4640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2330156ef0 0x7f233015b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2334dd2640 1 -- 172.21.15.77:0/465570647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2330156ef0 msgr2=0x7f233015b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2334dd2640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2330156ef0 0x7f233015b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2334dd2640 1 -- 172.21.15.77:0/465570647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 msgr2=0x7f23301599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2334dd2640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 0x7f23301599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.974+0000 7f2334dd2640 1 -- 172.21.15.77:0/465570647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233015a110 con 0x7f23300d0a80 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.975+0000 7f2334dd2640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23300d0a80 0x7f23301567e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f232000c3c0 tx=0x7f232000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.975+0000 7f2325ffb640 1 -- 172.21.15.77:0/465570647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23200052d0 con 0x7f23300d0a80 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.975+0000 7f2325ffb640 1 -- 172.21.15.77:0/465570647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320005490 con 0x7f23300d0a80 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.975+0000 7f2325ffb640 1 -- 172.21.15.77:0/465570647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320007260 con 0x7f23300d0a80 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.975+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233015a3f0 con 0x7f23300d0a80 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.975+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233015b8d0 con 0x7f23300d0a80 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.976+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23300cb530 con 0x7f23300d0a80 2024-03-06T07:28:43.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.976+0000 7f2325ffb640 1 -- 172.21.15.77:0/465570647 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2320007400 con 0x7f23300d0a80 2024-03-06T07:28:43.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.976+0000 7f2325ffb640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f231406b500 0x7f231406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.977+0000 7f2325ffb640 1 -- 172.21.15.77:0/465570647 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23200896a0 con 0x7f23300d0a80 2024-03-06T07:28:43.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.977+0000 7f23355d3640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f231406b500 0x7f231406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.977+0000 7f23355d3640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f231406b500 0x7f231406d9c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f232c036000 tx=0x7f232c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:43.981+0000 7f2325ffb640 1 -- 172.21.15.77:0/465570647 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2320057e40 con 0x7f23300d0a80 2024-03-06T07:28:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.190+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f233004f330 con 0x7f23300d0a80 2024-03-06T07:28:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.191+0000 7f2325ffb640 1 -- 172.21.15.77:0/465570647 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4998 (secure 0 0 0) 0x7f232005bee0 con 0x7f23300d0a80 2024-03-06T07:28:44.193 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:44.193 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6731,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/2348087487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":2348087487},{"type":"v1","addr":"172.21.15.158:6834","nonce":2348087487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/238243004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":238243004},{"type":"v1","addr":"172.21.15.77:6835","nonce":238243004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:43.855282+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:creating","state_seq":1,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:28:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 82 2024-03-06T07:28:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f231406b500 msgr2=0x7f231406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f231406b500 0x7f231406d9c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f232c036000 tx=0x7f232c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23300d0a80 msgr2=0x7f23301567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23300d0a80 0x7f23301567e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f232000c3c0 tx=0x7f232000c880 comp rx=0 tx=0).stop 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 shutdown_connections 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f231406b500 0x7f231406d9c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23300d0a80 0x7f23301567e0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2330156ef0 0x7f233015b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 --2- 172.21.15.77:0/465570647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f23300b4170 0x7f23301599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 >> 172.21.15.77:0/465570647 conn(0x7f23300d3c40 msgr2=0x7f233006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.193+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 shutdown_connections 2024-03-06T07:28:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:44.194+0000 7f2337837640 1 -- 172.21.15.77:0/465570647 wait complete. 2024-03-06T07:28:44.206 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:28:48.046 INFO:tasks.ceph.mds.b.smithi158.stderr:2024-03-06T07:28:48.044+0000 7fdf9d25e640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:28:49.207 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:49.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.307+0000 7f2a7de66640 1 Processor -- start 2024-03-06T07:28:49.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.307+0000 7f2a7de66640 1 -- start start 2024-03-06T07:28:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a7de66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a7815dfe0 0x7f2a780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a7de66640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a780cf830 0x7f2a780d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a7de66640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a780b3430 con 0x7f2a780d27f0 2024-03-06T07:28:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a7de66640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a780d2c50 con 0x7f2a780cf830 2024-03-06T07:28:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a7de66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780b31e0 con 0x7f2a7815dfe0 2024-03-06T07:28:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a76ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a780cf830 0x7f2a780d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a76ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a780cf830 0x7f2a780d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57372/0 (socket says 172.21.15.77:57372) 2024-03-06T07:28:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a76ffd640 1 -- 172.21.15.77:0/2221554574 learned_addr learned my addr 172.21.15.77:0/2221554574 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:49.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a777fe640 1 --2- 172.21.15.77:0/2221554574 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a7815dfe0 0x7f2a780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a76ffd640 1 -- 172.21.15.77:0/2221554574 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a7815dfe0 msgr2=0x7f2a780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a76ffd640 1 --2- 172.21.15.77:0/2221554574 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a7815dfe0 0x7f2a780cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a76ffd640 1 -- 172.21.15.77:0/2221554574 >> v1:172.21.15.158:6789/0 conn(0x7f2a780d27f0 legacy=0x7f2a780d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.308+0000 7f2a76ffd640 1 -- 172.21.15.77:0/2221554574 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a780b3710 con 0x7f2a780cf830 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.309+0000 7f2a76ffd640 1 --2- 172.21.15.77:0/2221554574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a780cf830 0x7f2a780d22b0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c009870 tx=0x7f2a6c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac0e743921c19c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.309+0000 7f2a767fc640 1 -- 172.21.15.77:0/2221554574 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c034070 con 0x7f2a780cf830 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.309+0000 7f2a767fc640 1 -- 172.21.15.77:0/2221554574 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6c030840 con 0x7f2a780cf830 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.309+0000 7f2a767fc640 1 -- 172.21.15.77:0/2221554574 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c030c10 con 0x7f2a780cf830 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.310+0000 7f2a7de66640 1 -- 172.21.15.77:0/2221554574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a780cf830 msgr2=0x7f2a780d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.310+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2221554574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a780cf830 0x7f2a780d22b0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c009870 tx=0x7f2a6c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.310+0000 7f2a7de66640 1 -- 172.21.15.77:0/2221554574 shutdown_connections 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.310+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2221554574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a780cf830 0x7f2a780d22b0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.310+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2221554574 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a7815dfe0 0x7f2a780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.310+0000 7f2a7de66640 1 -- 172.21.15.77:0/2221554574 >> 172.21.15.77:0/2221554574 conn(0x7f2a780d3cd0 msgr2=0x7f2a780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.310+0000 7f2a7de66640 1 -- 172.21.15.77:0/2221554574 shutdown_connections 2024-03-06T07:28:49.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.310+0000 7f2a7de66640 1 -- 172.21.15.77:0/2221554574 wait complete. 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.311+0000 7f2a7de66640 1 Processor -- start 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.311+0000 7f2a7de66640 1 -- start start 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a7de66640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a780cf830 0x7f2a78154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a7de66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a780d27f0 0x7f2a780cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a7de66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7815dfe0 0x7f2a780cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a7de66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780d1c70 con 0x7f2a7815dfe0 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a7de66640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780b3d60 con 0x7f2a780cf830 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a7de66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780d1df0 con 0x7f2a780d27f0 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7815dfe0 0x7f2a780cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7815dfe0 0x7f2a780cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57374/0 (socket says 172.21.15.77:57374) 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 -- 172.21.15.77:0/2994244667 learned_addr learned my addr 172.21.15.77:0/2994244667 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a777fe640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a780cf830 0x7f2a78154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a76ffd640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a780d27f0 0x7f2a780cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 -- 172.21.15.77:0/2994244667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a780d27f0 msgr2=0x7f2a780cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a780d27f0 0x7f2a780cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 -- 172.21.15.77:0/2994244667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a780cf830 msgr2=0x7f2a78154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a780cf830 0x7f2a78154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 -- 172.21.15.77:0/2994244667 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a780b7c50 con 0x7f2a7815dfe0 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.312+0000 7f2a77fff640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7815dfe0 0x7f2a780cbb60 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f2a70001eb0 tx=0x7f2a7000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.313+0000 7f2a5ffff640 1 -- 172.21.15.77:0/2994244667 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70017070 con 0x7f2a7815dfe0 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.313+0000 7f2a5ffff640 1 -- 172.21.15.77:0/2994244667 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a700052d0 con 0x7f2a7815dfe0 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.313+0000 7f2a5ffff640 1 -- 172.21.15.77:0/2994244667 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70003810 con 0x7f2a7815dfe0 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.313+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a780b7f30 con 0x7f2a7815dfe0 2024-03-06T07:28:49.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.313+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a780b84f0 con 0x7f2a7815dfe0 2024-03-06T07:28:49.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.314+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7804f330 con 0x7f2a7815dfe0 2024-03-06T07:28:49.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.314+0000 7f2a5ffff640 1 -- 172.21.15.77:0/2994244667 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a7001a070 con 0x7f2a7815dfe0 2024-03-06T07:28:49.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.315+0000 7f2a5ffff640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a6406b4d0 0x7f2a6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.315+0000 7f2a5ffff640 1 -- 172.21.15.77:0/2994244667 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a70089590 con 0x7f2a7815dfe0 2024-03-06T07:28:49.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.315+0000 7f2a777fe640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a6406b4d0 0x7f2a6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.315+0000 7f2a777fe640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a6406b4d0 0x7f2a6406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2a68011340 tx=0x7f2a6800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.319+0000 7f2a5ffff640 1 -- 172.21.15.77:0/2994244667 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a70057d00 con 0x7f2a7815dfe0 2024-03-06T07:28:49.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.529+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a780ceb50 con 0x7f2a7815dfe0 2024-03-06T07:28:49.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.529+0000 7f2a5ffff640 1 -- 172.21.15.77:0/2994244667 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5001 (secure 0 0 0) 0x7f2a7005bda0 con 0x7f2a7815dfe0 2024-03-06T07:28:49.531 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:49.531 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6731,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/2348087487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":2348087487},{"type":"v1","addr":"172.21.15.158:6834","nonce":2348087487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/238243004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":238243004},{"type":"v1","addr":"172.21.15.77:6835","nonce":238243004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:44.856173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:active","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8535,8547]},"id":9}]} 2024-03-06T07:28:49.532 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 83 2024-03-06T07:28:49.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a6406b4d0 msgr2=0x7f2a6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a6406b4d0 0x7f2a6406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2a68011340 tx=0x7f2a6800b040 comp rx=0 tx=0).stop 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7815dfe0 msgr2=0x7f2a780cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7815dfe0 0x7f2a780cbb60 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f2a70001eb0 tx=0x7f2a7000c890 comp rx=0 tx=0).stop 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 shutdown_connections 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a6406b4d0 0x7f2a6406d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7815dfe0 0x7f2a780cbb60 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a780d27f0 0x7f2a780cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 --2- 172.21.15.77:0/2994244667 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a780cf830 0x7f2a78154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.531+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 >> 172.21.15.77:0/2994244667 conn(0x7f2a780d3cd0 msgr2=0x7f2a7806cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.532+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 shutdown_connections 2024-03-06T07:28:49.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.532+0000 7f2a7de66640 1 -- 172.21.15.77:0/2994244667 wait complete. 2024-03-06T07:28:49.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:49.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.681+0000 7fd1ef302640 1 Processor -- start 2024-03-06T07:28:49.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.681+0000 7fd1ef302640 1 -- start start 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ef302640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd1e80cd7a0 0x7fd1e80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ef302640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80cc980 con 0x7fd1e806da40 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ef302640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80ccc60 con 0x7fd1e81424f0 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ef302640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e80cd220 con 0x7fd1e80cd7a0 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ec89d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd1e806da40 0x7fd1e8071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:60452/0 (socket says 172.21.15.77:60452) 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ec89d640 1 -- 172.21.15.77:0/69965578 learned_addr learned my addr 172.21.15.77:0/69965578 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ed89f640 1 --2- 172.21.15.77:0/69965578 >> v2:172.21.15.158:3300/0 conn(0x7fd1e80cd7a0 0x7fd1e80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1dffff640 1 -- 172.21.15.77:0/69965578 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504023334 0 0) 0x7fd1e80cc980 con 0x7fd1e806da40 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1dffff640 1 -- 172.21.15.77:0/69965578 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d4003520 con 0x7fd1e806da40 2024-03-06T07:28:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1dffff640 1 -- 172.21.15.77:0/69965578 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732915814 0 0) 0x7fd1e80ccc60 con 0x7fd1e81424f0 2024-03-06T07:28:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1dffff640 1 -- 172.21.15.77:0/69965578 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e80cc980 con 0x7fd1e81424f0 2024-03-06T07:28:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ed89f640 1 -- 172.21.15.77:0/69965578 >> v1:172.21.15.158:6789/0 conn(0x7fd1e81424f0 legacy=0x7fd1e806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ed89f640 1 -- 172.21.15.77:0/69965578 >> v1:172.21.15.77:6789/0 conn(0x7fd1e806da40 legacy=0x7fd1e8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.682+0000 7fd1ed89f640 1 -- 172.21.15.77:0/69965578 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80ccf40 con 0x7fd1e80cd7a0 2024-03-06T07:28:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.683+0000 7fd1ed89f640 1 --2- 172.21.15.77:0/69965578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e80cd7a0 0x7fd1e80cc2f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009d90 tx=0x7fd1e002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c09a08c37f05e910 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.683+0000 7fd1dffff640 1 -- 172.21.15.77:0/69965578 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0039070 con 0x7fd1e80cd7a0 2024-03-06T07:28:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.683+0000 7fd1dffff640 1 -- 172.21.15.77:0/69965578 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e002cd00 con 0x7fd1e80cd7a0 2024-03-06T07:28:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.683+0000 7fd1dffff640 1 -- 172.21.15.77:0/69965578 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0034640 con 0x7fd1e80cd7a0 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.684+0000 7fd1ef302640 1 -- 172.21.15.77:0/69965578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e80cd7a0 msgr2=0x7fd1e80cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.684+0000 7fd1ef302640 1 --2- 172.21.15.77:0/69965578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e80cd7a0 0x7fd1e80cc2f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009d90 tx=0x7fd1e002bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.684+0000 7fd1ef302640 1 -- 172.21.15.77:0/69965578 shutdown_connections 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.684+0000 7fd1ef302640 1 --2- 172.21.15.77:0/69965578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e80cd7a0 0x7fd1e80cc2f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.684+0000 7fd1ef302640 1 -- 172.21.15.77:0/69965578 >> 172.21.15.77:0/69965578 conn(0x7fd1e80cf8a0 msgr2=0x7fd1e80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.684+0000 7fd1ef302640 1 -- 172.21.15.77:0/69965578 shutdown_connections 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.684+0000 7fd1ef302640 1 -- 172.21.15.77:0/69965578 wait complete. 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.685+0000 7fd1ef302640 1 Processor -- start 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.685+0000 7fd1ef302640 1 -- start start 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ef302640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e806da40 0x7fd1e81562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ef302640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1e80cd7a0 0x7fd1e813b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ef302640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1e81424f0 0x7fd1e81408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ef302640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e815d070 con 0x7fd1e81424f0 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ef302640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e815cef0 con 0x7fd1e806da40 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ef302640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e815d1f0 con 0x7fd1e80cd7a0 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1e81424f0 0x7fd1e81408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1e81424f0 0x7fd1e81408b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57388/0 (socket says 172.21.15.77:57388) 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 -- 172.21.15.77:0/2676332250 learned_addr learned my addr 172.21.15.77:0/2676332250 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ec89d640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1e80cd7a0 0x7fd1e813b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed09e640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e806da40 0x7fd1e81562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 -- 172.21.15.77:0/2676332250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1e80cd7a0 msgr2=0x7fd1e813b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1e80cd7a0 0x7fd1e813b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 -- 172.21.15.77:0/2676332250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e806da40 msgr2=0x7fd1e81562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e806da40 0x7fd1e81562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 -- 172.21.15.77:0/2676332250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e81568b0 con 0x7fd1e81424f0 2024-03-06T07:28:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ed89f640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1e81424f0 0x7fd1e81408b0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009eb0 tx=0x7fd1e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.686+0000 7fd1ddffb640 1 -- 172.21.15.77:0/2676332250 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0039050 con 0x7fd1e81424f0 2024-03-06T07:28:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.687+0000 7fd1ddffb640 1 -- 172.21.15.77:0/2676332250 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e0045070 con 0x7fd1e81424f0 2024-03-06T07:28:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.687+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e8156b90 con 0x7fd1e81424f0 2024-03-06T07:28:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.687+0000 7fd1ddffb640 1 -- 172.21.15.77:0/2676332250 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0009450 con 0x7fd1e81424f0 2024-03-06T07:28:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.687+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e8155070 con 0x7fd1e81424f0 2024-03-06T07:28:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.688+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e804f330 con 0x7fd1e81424f0 2024-03-06T07:28:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.688+0000 7fd1ddffb640 1 -- 172.21.15.77:0/2676332250 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1e003e830 con 0x7fd1e81424f0 2024-03-06T07:28:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.689+0000 7fd1ddffb640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1cc06b4d0 0x7fd1cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.689+0000 7fd1ddffb640 1 -- 172.21.15.77:0/2676332250 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd1e00ad0e0 con 0x7fd1e81424f0 2024-03-06T07:28:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.689+0000 7fd1ed09e640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1cc06b4d0 0x7fd1cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.689+0000 7fd1ed09e640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1cc06b4d0 0x7fd1cc06d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4007fd0 tx=0x7fd1e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.697+0000 7fd1ddffb640 1 -- 172.21.15.77:0/2676332250 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1e007bb40 con 0x7fd1e81424f0 2024-03-06T07:28:49.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.912+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1e80d0620 con 0x7fd1e81424f0 2024-03-06T07:28:49.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.913+0000 7fd1ddffb640 1 -- 172.21.15.77:0/2676332250 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5001 (secure 0 0 0) 0x7fd1e007fbe0 con 0x7fd1e81424f0 2024-03-06T07:28:49.915 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:49.915 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6731,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6834/2348087487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":2348087487},{"type":"v1","addr":"172.21.15.158:6834","nonce":2348087487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/238243004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":238243004},{"type":"v1","addr":"172.21.15.77:6835","nonce":238243004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:44.856173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:active","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8535,8547]},"id":9}]} 2024-03-06T07:28:49.915 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 83 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1cc06b4d0 msgr2=0x7fd1cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1cc06b4d0 0x7fd1cc06d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4007fd0 tx=0x7fd1e4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1e81424f0 msgr2=0x7fd1e81408b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1e81424f0 0x7fd1e81408b0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009eb0 tx=0x7fd1e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 shutdown_connections 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd1cc06b4d0 0x7fd1cc06d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1e81424f0 0x7fd1e81408b0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd1e80cd7a0 0x7fd1e813b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 --2- 172.21.15.77:0/2676332250 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1e806da40 0x7fd1e81562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 >> 172.21.15.77:0/2676332250 conn(0x7fd1e80cf8a0 msgr2=0x7fd1e815a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 shutdown_connections 2024-03-06T07:28:49.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:49.916+0000 7fd1ef302640 1 -- 172.21.15.77:0/2676332250 wait complete. 2024-03-06T07:28:49.929 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are b and c 2024-03-06T07:28:49.929 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:49.929 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:49.940 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:49.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:28:50.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.081+0000 7f743999a640 1 Processor -- start 2024-03-06T07:28:50.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.081+0000 7f743999a640 1 -- start start 2024-03-06T07:28:50.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.081+0000 7f743999a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7434162400 0x7f74340d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.081+0000 7f743999a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74340d22c0 con 0x7f74340b4460 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.081+0000 7f743999a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74340d25a0 con 0x7f7434162970 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.081+0000 7f743999a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7434161b00 con 0x7f7434162400 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.082+0000 7f7432ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f74340b4460 0x7f74340b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:60468/0 (socket says 172.21.15.77:60468) 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.082+0000 7f7432ffd640 1 -- 172.21.15.77:0/3868191217 learned_addr learned my addr 172.21.15.77:0/3868191217 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.082+0000 7f74337fe640 1 --2- 172.21.15.77:0/3868191217 >> v2:172.21.15.158:3300/0 conn(0x7f7434162400 0x7f74340d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.082+0000 7f74327fc640 1 -- 172.21.15.77:0/3868191217 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942010990 0 0) 0x7f74340d22c0 con 0x7f74340b4460 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.082+0000 7f74327fc640 1 -- 172.21.15.77:0/3868191217 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7420003520 con 0x7f74340b4460 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.082+0000 7f74337fe640 1 -- 172.21.15.77:0/3868191217 >> v1:172.21.15.158:6789/0 conn(0x7f7434162970 legacy=0x7f7434161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.082+0000 7f74337fe640 1 -- 172.21.15.77:0/3868191217 >> v1:172.21.15.77:6789/0 conn(0x7f74340b4460 legacy=0x7f74340b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.082+0000 7f74337fe640 1 -- 172.21.15.77:0/3868191217 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340d2880 con 0x7f7434162400 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.083+0000 7f74337fe640 1 --2- 172.21.15.77:0/3868191217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 0x7f74340d4800 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f742c009d90 tx=0x7f742c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7459467350de3e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.083+0000 7f74327fc640 1 -- 172.21.15.77:0/3868191217 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c039070 con 0x7f7434162400 2024-03-06T07:28:50.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.083+0000 7f74327fc640 1 -- 172.21.15.77:0/3868191217 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c02cd00 con 0x7f7434162400 2024-03-06T07:28:50.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.083+0000 7f74327fc640 1 -- 172.21.15.77:0/3868191217 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c034640 con 0x7f7434162400 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.083+0000 7f743999a640 1 -- 172.21.15.77:0/3868191217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 msgr2=0x7f74340d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.083+0000 7f743999a640 1 --2- 172.21.15.77:0/3868191217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 0x7f74340d4800 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f742c009d90 tx=0x7f742c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.084+0000 7f743999a640 1 -- 172.21.15.77:0/3868191217 shutdown_connections 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.084+0000 7f743999a640 1 --2- 172.21.15.77:0/3868191217 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 0x7f74340d4800 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.084+0000 7f743999a640 1 -- 172.21.15.77:0/3868191217 >> 172.21.15.77:0/3868191217 conn(0x7f74340d3ce0 msgr2=0x7f74340d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.084+0000 7f743999a640 1 -- 172.21.15.77:0/3868191217 shutdown_connections 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.084+0000 7f743999a640 1 -- 172.21.15.77:0/3868191217 wait complete. 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.085+0000 7f743999a640 1 Processor -- start 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.085+0000 7f743999a640 1 -- start start 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f743999a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74340b4460 0x7f74340b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f743999a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 0x7f74340b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f743999a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7434162970 0x7f74340b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f743999a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74340ce950 con 0x7f74340b4460 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f743999a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74340cecc0 con 0x7f7434162400 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f743999a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74340b4840 con 0x7f7434162970 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74340b4460 0x7f74340b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74340b4460 0x7f74340b7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57400/0 (socket says 172.21.15.77:57400) 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 -- 172.21.15.77:0/3315812337 learned_addr learned my addr 172.21.15.77:0/3315812337 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f742a7fc640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 0x7f74340b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f74337fe640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7434162970 0x7f74340b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 -- 172.21.15.77:0/3315812337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7434162970 msgr2=0x7f74340b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7434162970 0x7f74340b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 -- 172.21.15.77:0/3315812337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 msgr2=0x7f74340b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 0x7f74340b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 -- 172.21.15.77:0/3315812337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340b8880 con 0x7f74340b4460 2024-03-06T07:28:50.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7432ffd640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74340b4460 0x7f74340b7ea0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f742400c410 tx=0x7f742400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7430ff9640 1 -- 172.21.15.77:0/3315812337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7424017070 con 0x7f74340b4460 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7430ff9640 1 -- 172.21.15.77:0/3315812337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7424005ce0 con 0x7f74340b4460 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74340b8b60 con 0x7f74340b4460 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.086+0000 7f7430ff9640 1 -- 172.21.15.77:0/3315812337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74240072f0 con 0x7f74340b4460 2024-03-06T07:28:50.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.087+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74340b9120 con 0x7f74340b4460 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.087+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743404f330 con 0x7f74340b4460 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.088+0000 7f7430ff9640 1 -- 172.21.15.77:0/3315812337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f742401a070 con 0x7f74340b4460 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.089+0000 7f7430ff9640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f741406b4d0 0x7f741406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.089+0000 7f7430ff9640 1 -- 172.21.15.77:0/3315812337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7424089650 con 0x7f74340b4460 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.089+0000 7f742a7fc640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f741406b4d0 0x7f741406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.089+0000 7f742a7fc640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f741406b4d0 0x7f741406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74340c78c0 tx=0x7f741c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.093+0000 7f7430ff9640 1 -- 172.21.15.77:0/3315812337 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7424057f00 con 0x7f74340b4460 2024-03-06T07:28:50.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.341+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f74340c6fe0 con 0x7f74340b4460 2024-03-06T07:28:50.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.876+0000 7f7430ff9640 1 -- 172.21.15.77:0/3315812337 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 8532 v84) v1 ==== 95+0+0 (secure 0 0 0) 0x7f742405bfa0 con 0x7f74340b4460 2024-03-06T07:28:50.878 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 8532 2024-03-06T07:28:50.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f741406b4d0 msgr2=0x7f741406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f741406b4d0 0x7f741406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74340c78c0 tx=0x7f741c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74340b4460 msgr2=0x7f74340b7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74340b4460 0x7f74340b7ea0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f742400c410 tx=0x7f742400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 shutdown_connections 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7434162970 0x7f74340b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7434162400 0x7f74340b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f741406b4d0 0x7f741406d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 --2- 172.21.15.77:0/3315812337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74340b4460 0x7f74340b7ea0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 >> 172.21.15.77:0/3315812337 conn(0x7f74340d3ce0 msgr2=0x7f74340d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 shutdown_connections 2024-03-06T07:28:50.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:50.879+0000 7f743999a640 1 -- 172.21.15.77:0/3315812337 wait complete. 2024-03-06T07:28:50.892 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:50.892 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:50.900 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:50.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:28:51.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.001+0000 7f4a4df0c640 1 Processor -- start 2024-03-06T07:28:51.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.001+0000 7f4a4df0c640 1 -- start start 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.002+0000 7f4a4df0c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a481444c0 0x7f4a4806ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.002+0000 7f4a4df0c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4a4815b2d0 0x7f4a480708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.002+0000 7f4a4df0c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4806d370 con 0x7f4a48070e10 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.002+0000 7f4a4df0c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a481448a0 con 0x7f4a481444c0 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.002+0000 7f4a4df0c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a48071260 con 0x7f4a4815b2d0 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.002+0000 7f4a477fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a481444c0 0x7f4a4806ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.002+0000 7f4a477fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a481444c0 0x7f4a4806ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57406/0 (socket says 172.21.15.77:57406) 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.003+0000 7f4a477fe640 1 -- 172.21.15.77:0/841188727 learned_addr learned my addr 172.21.15.77:0/841188727 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.003+0000 7f4a3effd640 1 --2- 172.21.15.77:0/841188727 >> v2:172.21.15.158:3300/0 conn(0x7f4a4815b2d0 0x7f4a480708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.003+0000 7f4a477fe640 1 -- 172.21.15.77:0/841188727 >> v2:172.21.15.158:3300/0 conn(0x7f4a4815b2d0 msgr2=0x7f4a480708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.003+0000 7f4a477fe640 1 --2- 172.21.15.77:0/841188727 >> v2:172.21.15.158:3300/0 conn(0x7f4a4815b2d0 0x7f4a480708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.003+0000 7f4a477fe640 1 -- 172.21.15.77:0/841188727 >> v1:172.21.15.158:6789/0 conn(0x7f4a48070e10 legacy=0x7f4a4815e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.003+0000 7f4a477fe640 1 -- 172.21.15.77:0/841188727 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4806d650 con 0x7f4a481444c0 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.004+0000 7f4a477fe640 1 --2- 172.21.15.77:0/841188727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a481444c0 0x7f4a4806ca00 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f4a480cd5a0 tx=0x7f4a3802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9ddd2dc3d0d747 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.004+0000 7f4a46ffd640 1 -- 172.21.15.77:0/841188727 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38034070 con 0x7f4a481444c0 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.004+0000 7f4a46ffd640 1 -- 172.21.15.77:0/841188727 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38030860 con 0x7f4a481444c0 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.004+0000 7f4a46ffd640 1 -- 172.21.15.77:0/841188727 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38030c30 con 0x7f4a481444c0 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.005+0000 7f4a4df0c640 1 -- 172.21.15.77:0/841188727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a481444c0 msgr2=0x7f4a4806ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.005+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/841188727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a481444c0 0x7f4a4806ca00 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f4a480cd5a0 tx=0x7f4a3802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.005+0000 7f4a4df0c640 1 -- 172.21.15.77:0/841188727 shutdown_connections 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.005+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/841188727 >> v2:172.21.15.158:3300/0 conn(0x7f4a4815b2d0 0x7f4a480708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.005+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/841188727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a481444c0 0x7f4a4806ca00 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.005+0000 7f4a4df0c640 1 -- 172.21.15.77:0/841188727 >> 172.21.15.77:0/841188727 conn(0x7f4a480b1c80 msgr2=0x7f4a480cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.005+0000 7f4a4df0c640 1 -- 172.21.15.77:0/841188727 shutdown_connections 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.005+0000 7f4a4df0c640 1 -- 172.21.15.77:0/841188727 wait complete. 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.006+0000 7f4a4df0c640 1 Processor -- start 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.007+0000 7f4a4df0c640 1 -- start start 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a4df0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48070e10 0x7f4a481e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a4df0c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a481444c0 0x7f4a4813da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a4df0c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a4815b2d0 0x7f4a4813df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a4df0c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4806def0 con 0x7f4a48070e10 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a4df0c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4806dc10 con 0x7f4a481444c0 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a4df0c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4806cf40 con 0x7f4a4815b2d0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48070e10 0x7f4a481e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48070e10 0x7f4a481e6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57420/0 (socket says 172.21.15.77:57420) 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 -- 172.21.15.77:0/3822534441 learned_addr learned my addr 172.21.15.77:0/3822534441 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a3effd640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a481444c0 0x7f4a4813da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a47fff640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a4815b2d0 0x7f4a4813df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 -- 172.21.15.77:0/3822534441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a4815b2d0 msgr2=0x7f4a4813df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a4815b2d0 0x7f4a4813df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 -- 172.21.15.77:0/3822534441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a481444c0 msgr2=0x7f4a4813da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a481444c0 0x7f4a4813da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 -- 172.21.15.77:0/3822534441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4813c060 con 0x7f4a48070e10 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.008+0000 7f4a477fe640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48070e10 0x7f4a481e6a30 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f4a3802b9e0 tx=0x7f4a38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.009+0000 7f4a44ff9640 1 -- 172.21.15.77:0/3822534441 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38048070 con 0x7f4a48070e10 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.009+0000 7f4a44ff9640 1 -- 172.21.15.77:0/3822534441 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38030660 con 0x7f4a48070e10 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.009+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4813c340 con 0x7f4a48070e10 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.009+0000 7f4a44ff9640 1 -- 172.21.15.77:0/3822534441 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38034040 con 0x7f4a48070e10 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.009+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4813c900 con 0x7f4a48070e10 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.010+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4804f330 con 0x7f4a48070e10 2024-03-06T07:28:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.010+0000 7f4a44ff9640 1 -- 172.21.15.77:0/3822534441 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a3804b070 con 0x7f4a48070e10 2024-03-06T07:28:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.011+0000 7f4a44ff9640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a2806b4d0 0x7f4a2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.011+0000 7f4a44ff9640 1 -- 172.21.15.77:0/3822534441 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a380ac150 con 0x7f4a48070e10 2024-03-06T07:28:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.011+0000 7f4a3effd640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a2806b4d0 0x7f4a2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.011+0000 7f4a3effd640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a2806b4d0 0x7f4a2806d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4a4813d890 tx=0x7f4a30009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.015+0000 7f4a44ff9640 1 -- 172.21.15.77:0/3822534441 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a3807a940 con 0x7f4a48070e10 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.260+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f4a4815d760 con 0x7f4a48070e10 2024-03-06T07:28:51.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.880+0000 7f4a44ff9640 1 -- 172.21.15.77:0/3822534441 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6731 v85) v1 ==== 95+0+0 (secure 0 0 0) 0x7f4a3807e9e0 con 0x7f4a48070e10 2024-03-06T07:28:51.882 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 6731 2024-03-06T07:28:51.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a2806b4d0 msgr2=0x7f4a2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a2806b4d0 0x7f4a2806d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4a4813d890 tx=0x7f4a30009190 comp rx=0 tx=0).stop 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48070e10 msgr2=0x7f4a481e6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48070e10 0x7f4a481e6a30 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f4a3802b9e0 tx=0x7f4a38004730 comp rx=0 tx=0).stop 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 shutdown_connections 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a4815b2d0 0x7f4a4813df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a481444c0 0x7f4a4813da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a2806b4d0 0x7f4a2806d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 --2- 172.21.15.77:0/3822534441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48070e10 0x7f4a481e6a30 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.882+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 >> 172.21.15.77:0/3822534441 conn(0x7f4a480b1c80 msgr2=0x7f4a4815c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.883+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 shutdown_connections 2024-03-06T07:28:51.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:51.883+0000 7f4a4df0c640 1 -- 172.21.15.77:0/3822534441 wait complete. 2024-03-06T07:28:51.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:51.896 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:28:51.896 DEBUG:teuthology.orchestra.run.smithi077:> mkdir pin 2024-03-06T07:28:51.896 DEBUG:teuthology.orchestra.run.smithi077:> touch pin/placeholder 2024-03-06T07:28:51.896 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:28:51.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:51.922 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:28:51.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:52.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.120+0000 7f29e020e640 1 Processor -- start 2024-03-06T07:28:52.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.120+0000 7f29e020e640 1 -- start start 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29e020e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 0x7f29d80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29e020e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29d80cf830 0x7f29d80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29e020e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f29d80d27f0 0x7f29d80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29e020e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d80d2c50 con 0x7f29d80cf830 2024-03-06T07:28:52.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29e020e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d80b31e0 con 0x7f29d80d27f0 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29e020e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d80b3d60 con 0x7f29d815dfe0 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29dd7a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29d80cf830 0x7f29d80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29dd7a9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29d80cf830 0x7f29d80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57432/0 (socket says 172.21.15.77:57432) 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29dd7a9640 1 -- 172.21.15.77:0/2897149531 learned_addr learned my addr 172.21.15.77:0/2897149531 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29ddfaa640 1 --2- 172.21.15.77:0/2897149531 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 0x7f29d80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29de7ab640 1 --2- 172.21.15.77:0/2897149531 >> v2:172.21.15.158:3300/0 conn(0x7f29d80d27f0 0x7f29d80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29dd7a9640 1 -- 172.21.15.77:0/2897149531 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 msgr2=0x7f29d80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29dd7a9640 1 --2- 172.21.15.77:0/2897149531 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 0x7f29d80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29dd7a9640 1 -- 172.21.15.77:0/2897149531 >> v2:172.21.15.158:3300/0 conn(0x7f29d80d27f0 msgr2=0x7f29d80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29dd7a9640 1 --2- 172.21.15.77:0/2897149531 >> v2:172.21.15.158:3300/0 conn(0x7f29d80d27f0 0x7f29d80d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.121+0000 7f29dd7a9640 1 -- 172.21.15.77:0/2897149531 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d80b3430 con 0x7f29d80cf830 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.122+0000 7f29dd7a9640 1 --2- 172.21.15.77:0/2897149531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d80d22b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f29d8067ad0 tx=0x7f29c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7eea5c049d13b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.122+0000 7f29dcfa8640 1 -- 172.21.15.77:0/2897149531 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8035020 con 0x7f29d80cf830 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.122+0000 7f29dcfa8640 1 -- 172.21.15.77:0/2897149531 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c8031870 con 0x7f29d80cf830 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.122+0000 7f29dcfa8640 1 -- 172.21.15.77:0/2897149531 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8031c40 con 0x7f29d80cf830 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 -- 172.21.15.77:0/2897149531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 msgr2=0x7f29d80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 --2- 172.21.15.77:0/2897149531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d80d22b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f29d8067ad0 tx=0x7f29c802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 -- 172.21.15.77:0/2897149531 shutdown_connections 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 --2- 172.21.15.77:0/2897149531 >> v2:172.21.15.158:3300/0 conn(0x7f29d80d27f0 0x7f29d80d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 --2- 172.21.15.77:0/2897149531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d80d22b0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 --2- 172.21.15.77:0/2897149531 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 0x7f29d80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 -- 172.21.15.77:0/2897149531 >> 172.21.15.77:0/2897149531 conn(0x7f29d80d3cd0 msgr2=0x7f29d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 -- 172.21.15.77:0/2897149531 shutdown_connections 2024-03-06T07:28:52.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.123+0000 7f29e020e640 1 -- 172.21.15.77:0/2897149531 wait complete. 2024-03-06T07:28:52.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.124+0000 7f29e020e640 1 Processor -- start 2024-03-06T07:28:52.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.124+0000 7f29e020e640 1 -- start start 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29e020e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d81f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29e020e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29d80d27f0 0x7f29d81f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29e020e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 0x7f29d8149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29e020e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d80d1c70 con 0x7f29d80cf830 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29e020e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d80b39f0 con 0x7f29d80d27f0 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29e020e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d80d1df0 con 0x7f29d815dfe0 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d81f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d81f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57440/0 (socket says 172.21.15.77:57440) 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 -- 172.21.15.77:0/2662815085 learned_addr learned my addr 172.21.15.77:0/2662815085 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29dd7a9640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29d80d27f0 0x7f29d81f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29de7ab640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 0x7f29d8149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 -- 172.21.15.77:0/2662815085 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 msgr2=0x7f29d8149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 0x7f29d8149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 -- 172.21.15.77:0/2662815085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29d80d27f0 msgr2=0x7f29d81f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29d80d27f0 0x7f29d81f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 -- 172.21.15.77:0/2662815085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d8161450 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.125+0000 7f29ddfaa640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d81f2830 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f29d400e8a0 tx=0x7f29d400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.126+0000 7f29ce7fc640 1 -- 172.21.15.77:0/2662815085 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4017070 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.126+0000 7f29ce7fc640 1 -- 172.21.15.77:0/2662815085 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d400c940 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.126+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d80cc980 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.126+0000 7f29ce7fc640 1 -- 172.21.15.77:0/2662815085 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d400cd30 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.126+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d814dcc0 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.127+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d80cc7a0 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.127+0000 7f29ce7fc640 1 -- 172.21.15.77:0/2662815085 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29d401a070 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.127+0000 7f29ce7fc640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f29bc06b4d0 0x7f29bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.127+0000 7f29ce7fc640 1 -- 172.21.15.77:0/2662815085 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f29d4089f90 con 0x7f29d80cf830 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.128+0000 7f29dd7a9640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f29bc06b4d0 0x7f29bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.128+0000 7f29dd7a9640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f29bc06b4d0 0x7f29bc06d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f29c8036000 tx=0x7f29c803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.132+0000 7f29ce7fc640 1 -- 172.21.15.77:0/2662815085 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29d80cc7a0 con 0x7f29d80cf830 2024-03-06T07:28:52.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.374+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d804f330 con 0x7f29d80cf830 2024-03-06T07:28:52.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.374+0000 7f29ce7fc640 1 -- 172.21.15.77:0/2662815085 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+3420 (secure 0 0 0) 0x7f29d4058810 con 0x7f29d80cf830 2024-03-06T07:28:52.377 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:52.377 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":85,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:44.856173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:active","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8535,8547]},"id":9}]} 2024-03-06T07:28:52.377 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 85 2024-03-06T07:28:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f29bc06b4d0 msgr2=0x7f29bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f29bc06b4d0 0x7f29bc06d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f29c8036000 tx=0x7f29c803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 msgr2=0x7f29d81f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d81f2830 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f29d400e8a0 tx=0x7f29d400ed60 comp rx=0 tx=0).stop 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 shutdown_connections 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f29d815dfe0 0x7f29d8149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f29d80d27f0 0x7f29d81f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f29bc06b4d0 0x7f29bc06d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 --2- 172.21.15.77:0/2662815085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d80cf830 0x7f29d81f2830 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 >> 172.21.15.77:0/2662815085 conn(0x7f29d80d3cd0 msgr2=0x7f29d806ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 shutdown_connections 2024-03-06T07:28:52.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:52.376+0000 7f29e020e640 1 -- 172.21.15.77:0/2662815085 wait complete. 2024-03-06T07:28:52.389 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:28:52.505 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:28:52.506 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:28:52.506 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:28:54.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:54.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.607+0000 7f6b3ef56640 1 Processor -- start 2024-03-06T07:28:54.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b3ef56640 1 -- start start 2024-03-06T07:28:54.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b3ef56640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b3815dfe0 0x7f6b380cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b3ef56640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b380cf830 0x7f6b380d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b3ef56640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b380b3430 con 0x7f6b380d27f0 2024-03-06T07:28:54.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b3ef56640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b380b31e0 con 0x7f6b380cf830 2024-03-06T07:28:54.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b3ef56640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b380b3d60 con 0x7f6b3815dfe0 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b37fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b380cf830 0x7f6b380d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b3d4f3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6b380d27f0 0x7f6b380d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:60470/0 (socket says 172.21.15.77:60470) 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.608+0000 7f6b3d4f3640 1 -- 172.21.15.77:0/3488477523 learned_addr learned my addr 172.21.15.77:0/3488477523 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.609+0000 7f6b3ccf2640 1 --2- 172.21.15.77:0/3488477523 >> v2:172.21.15.158:3300/0 conn(0x7f6b3815dfe0 0x7f6b380cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.609+0000 7f6b37fff640 1 -- 172.21.15.77:0/3488477523 >> v2:172.21.15.158:3300/0 conn(0x7f6b3815dfe0 msgr2=0x7f6b380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.609+0000 7f6b37fff640 1 --2- 172.21.15.77:0/3488477523 >> v2:172.21.15.158:3300/0 conn(0x7f6b3815dfe0 0x7f6b380cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.609+0000 7f6b37fff640 1 -- 172.21.15.77:0/3488477523 >> v1:172.21.15.77:6789/0 conn(0x7f6b380d27f0 legacy=0x7f6b380d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.609+0000 7f6b37fff640 1 -- 172.21.15.77:0/3488477523 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b380b3710 con 0x7f6b380cf830 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.609+0000 7f6b37fff640 1 --2- 172.21.15.77:0/3488477523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cf830 0x7f6b380d22b0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f6b240097a0 tx=0x7f6b2402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc7206161239da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b377fe640 1 -- 172.21.15.77:0/3488477523 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24035020 con 0x7f6b380cf830 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b377fe640 1 -- 172.21.15.77:0/3488477523 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b24031830 con 0x7f6b380cf830 2024-03-06T07:28:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b377fe640 1 -- 172.21.15.77:0/3488477523 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24031c00 con 0x7f6b380cf830 2024-03-06T07:28:54.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b3ef56640 1 -- 172.21.15.77:0/3488477523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cf830 msgr2=0x7f6b380d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/3488477523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cf830 0x7f6b380d22b0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f6b240097a0 tx=0x7f6b2402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b3ef56640 1 -- 172.21.15.77:0/3488477523 shutdown_connections 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/3488477523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cf830 0x7f6b380d22b0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/3488477523 >> v2:172.21.15.158:3300/0 conn(0x7f6b3815dfe0 0x7f6b380cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b3ef56640 1 -- 172.21.15.77:0/3488477523 >> 172.21.15.77:0/3488477523 conn(0x7f6b380d3cd0 msgr2=0x7f6b380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b3ef56640 1 -- 172.21.15.77:0/3488477523 shutdown_connections 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.610+0000 7f6b3ef56640 1 -- 172.21.15.77:0/3488477523 wait complete. 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.611+0000 7f6b3ef56640 1 Processor -- start 2024-03-06T07:28:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.611+0000 7f6b3ef56640 1 -- start start 2024-03-06T07:28:54.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b3ef56640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380cf830 0x7f6b38159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b3ef56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380d27f0 0x7f6b3815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b3ef56640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b3815dfe0 0x7f6b380b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b3ef56640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380d1df0 con 0x7f6b380d27f0 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b3ef56640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380d1c70 con 0x7f6b380cf830 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b3ef56640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3815e3c0 con 0x7f6b3815dfe0 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b37fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380d27f0 0x7f6b3815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b37fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380d27f0 0x7f6b3815a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57462/0 (socket says 172.21.15.77:57462) 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b37fff640 1 -- 172.21.15.77:0/2364942892 learned_addr learned my addr 172.21.15.77:0/2364942892 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.612+0000 7f6b3ccf2640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380cf830 0x7f6b38159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b3d4f3640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b3815dfe0 0x7f6b380b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b37fff640 1 -- 172.21.15.77:0/2364942892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b3815dfe0 msgr2=0x7f6b380b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b37fff640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b3815dfe0 0x7f6b380b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b37fff640 1 -- 172.21.15.77:0/2364942892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380cf830 msgr2=0x7f6b38159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b37fff640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380cf830 0x7f6b38159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b37fff640 1 -- 172.21.15.77:0/2364942892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3815e790 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b37fff640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380d27f0 0x7f6b3815a3f0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f6b24034040 tx=0x7f6b240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b357fa640 1 -- 172.21.15.77:0/2364942892 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24040070 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b357fa640 1 -- 172.21.15.77:0/2364942892 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2403a910 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3815ea70 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b357fa640 1 -- 172.21.15.77:0/2364942892 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24035040 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.613+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3815f310 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.614+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3815ee60 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.615+0000 7f6b357fa640 1 -- 172.21.15.77:0/2364942892 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b24038020 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.615+0000 7f6b357fa640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.615+0000 7f6b357fa640 1 -- 172.21.15.77:0/2364942892 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6b240abc00 con 0x7f6b380d27f0 2024-03-06T07:28:54.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.615+0000 7f6b3ccf2640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.615+0000 7f6b3ccf2640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009b40 tx=0x7f6b2c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.624+0000 7f6b357fa640 1 -- 172.21.15.77:0/2364942892 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b240b2080 con 0x7f6b380d27f0 2024-03-06T07:28:54.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.840+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b3804f330 con 0x7f6b380d27f0 2024-03-06T07:28:54.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.841+0000 7f6b357fa640 1 -- 172.21.15.77:0/2364942892 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+3420 (secure 0 0 0) 0x7f6b24031800 con 0x7f6b380d27f0 2024-03-06T07:28:54.843 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:54.843 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":85,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:44.856173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:active","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8535,8547]},"id":9}]} 2024-03-06T07:28:54.844 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 85 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 msgr2=0x7f6b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009b40 tx=0x7f6b2c009190 comp rx=0 tx=0).stop 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380d27f0 msgr2=0x7f6b3815a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380d27f0 0x7f6b3815a3f0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f6b24034040 tx=0x7f6b240046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 shutdown_connections 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380d27f0 0x7f6b3815a3f0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b3815dfe0 0x7f6b380b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 --2- 172.21.15.77:0/2364942892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380cf830 0x7f6b38159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 >> 172.21.15.77:0/2364942892 conn(0x7f6b380d3cd0 msgr2=0x7f6b3806c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 shutdown_connections 2024-03-06T07:28:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:54.843+0000 7f6b3ef56640 1 -- 172.21.15.77:0/2364942892 wait complete. 2024-03-06T07:28:54.856 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:28:54.971 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:28:54.971 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:28:54.971 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:28:56.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.071+0000 7f8fda284640 1 Processor -- start 2024-03-06T07:28:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.072+0000 7f8fda284640 1 -- start start 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.072+0000 7f8fda284640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.072+0000 7f8fda284640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 0x7f8fd40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.072+0000 7f8fda284640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8fd40d27f0 0x7f8fd40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.072+0000 7f8fda284640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40d2c50 con 0x7f8fd415dfe0 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.072+0000 7f8fda284640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40b31e0 con 0x7f8fd40d27f0 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.072+0000 7f8fda284640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b3d60 con 0x7f8fd40cf830 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.072+0000 7f8fd3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57470/0 (socket says 172.21.15.77:57470) 2024-03-06T07:28:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd3fff640 1 -- 172.21.15.77:0/3022026834 learned_addr learned my addr 172.21.15.77:0/3022026834 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd37fe640 1 --2- 172.21.15.77:0/3022026834 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 0x7f8fd40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd8821640 1 --2- 172.21.15.77:0/3022026834 >> v2:172.21.15.158:3300/0 conn(0x7f8fd40d27f0 0x7f8fd40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd3fff640 1 -- 172.21.15.77:0/3022026834 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 msgr2=0x7f8fd40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd3fff640 1 --2- 172.21.15.77:0/3022026834 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 0x7f8fd40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd3fff640 1 -- 172.21.15.77:0/3022026834 >> v2:172.21.15.158:3300/0 conn(0x7f8fd40d27f0 msgr2=0x7f8fd40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd3fff640 1 --2- 172.21.15.77:0/3022026834 >> v2:172.21.15.158:3300/0 conn(0x7f8fd40d27f0 0x7f8fd40d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.073+0000 7f8fd3fff640 1 -- 172.21.15.77:0/3022026834 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40b3430 con 0x7f8fd415dfe0 2024-03-06T07:28:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fd3fff640 1 --2- 172.21.15.77:0/3022026834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4067ad0 tx=0x7f8fc402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1869af71238e9de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fd2ffd640 1 -- 172.21.15.77:0/3022026834 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4035020 con 0x7f8fd415dfe0 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fd2ffd640 1 -- 172.21.15.77:0/3022026834 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4031870 con 0x7f8fd415dfe0 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fd2ffd640 1 -- 172.21.15.77:0/3022026834 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4031c40 con 0x7f8fd415dfe0 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 -- 172.21.15.77:0/3022026834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 msgr2=0x7f8fd40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 --2- 172.21.15.77:0/3022026834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4067ad0 tx=0x7f8fc402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 -- 172.21.15.77:0/3022026834 shutdown_connections 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 --2- 172.21.15.77:0/3022026834 >> v2:172.21.15.158:3300/0 conn(0x7f8fd40d27f0 0x7f8fd40d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 --2- 172.21.15.77:0/3022026834 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 0x7f8fd40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 --2- 172.21.15.77:0/3022026834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd40cf2f0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 -- 172.21.15.77:0/3022026834 >> 172.21.15.77:0/3022026834 conn(0x7f8fd40d3cd0 msgr2=0x7f8fd40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 -- 172.21.15.77:0/3022026834 shutdown_connections 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.074+0000 7f8fda284640 1 -- 172.21.15.77:0/3022026834 wait complete. 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.076+0000 7f8fda284640 1 Processor -- start 2024-03-06T07:28:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.076+0000 7f8fda284640 1 -- start start 2024-03-06T07:28:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.076+0000 7f8fda284640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 0x7f8fd41f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.076+0000 7f8fda284640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd40d27f0 0x7f8fd4160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.076+0000 7f8fda284640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.076+0000 7f8fda284640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40d1c70 con 0x7f8fd415dfe0 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.076+0000 7f8fda284640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b39f0 con 0x7f8fd40d27f0 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.076+0000 7f8fda284640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40d1df0 con 0x7f8fd40cf830 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd415fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57476/0 (socket says 172.21.15.77:57476) 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 -- 172.21.15.77:0/2442381249 learned_addr learned my addr 172.21.15.77:0/2442381249 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd37fe640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd40d27f0 0x7f8fd4160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd3fff640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 0x7f8fd41f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 -- 172.21.15.77:0/2442381249 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 msgr2=0x7f8fd41f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 0x7f8fd41f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 -- 172.21.15.77:0/2442381249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd40d27f0 msgr2=0x7f8fd4160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd40d27f0 0x7f8fd4160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 -- 172.21.15.77:0/2442381249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40b9740 con 0x7f8fd415dfe0 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd8821640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd415fcb0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00efc0 tx=0x7f8fcc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd0ff9640 1 -- 172.21.15.77:0/2442381249 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc018020 con 0x7f8fd415dfe0 2024-03-06T07:28:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd0ff9640 1 -- 172.21.15.77:0/2442381249 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc014480 con 0x7f8fd415dfe0 2024-03-06T07:28:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd40b9a20 con 0x7f8fd415dfe0 2024-03-06T07:28:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fd0ff9640 1 -- 172.21.15.77:0/2442381249 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc014870 con 0x7f8fd415dfe0 2024-03-06T07:28:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.077+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd40cceb0 con 0x7f8fd415dfe0 2024-03-06T07:28:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.078+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd40b5200 con 0x7f8fd415dfe0 2024-03-06T07:28:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.079+0000 7f8fd0ff9640 1 -- 172.21.15.77:0/2442381249 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fcc005d10 con 0x7f8fd415dfe0 2024-03-06T07:28:57.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.079+0000 7f8fd0ff9640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8fc006b4d0 0x7f8fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.079+0000 7f8fd0ff9640 1 -- 172.21.15.77:0/2442381249 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8fcc08af80 con 0x7f8fd415dfe0 2024-03-06T07:28:57.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.079+0000 7f8fd3fff640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8fc006b4d0 0x7f8fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.080+0000 7f8fd3fff640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8fc006b4d0 0x7f8fc006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4036000 tx=0x7f8fc403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.084+0000 7f8fd0ff9640 1 -- 172.21.15.77:0/2442381249 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fd40b5200 con 0x7f8fd415dfe0 2024-03-06T07:28:57.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.384+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fd40cd0b0 con 0x7f8fd415dfe0 2024-03-06T07:28:57.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.385+0000 7f8fd0ff9640 1 -- 172.21.15.77:0/2442381249 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+3420 (secure 0 0 0) 0x7f8fd40cd0b0 con 0x7f8fd415dfe0 2024-03-06T07:28:57.387 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:57.387 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":85,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:44.856173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:active","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8535,8547]},"id":9}]} 2024-03-06T07:28:57.387 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 85 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.386+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8fc006b4d0 msgr2=0x7f8fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.386+0000 7f8fda284640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8fc006b4d0 0x7f8fc006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4036000 tx=0x7f8fc403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.386+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 msgr2=0x7f8fd415fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.386+0000 7f8fda284640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd415fcb0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00efc0 tx=0x7f8fcc00c490 comp rx=0 tx=0).stop 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.387+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 shutdown_connections 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.387+0000 7f8fda284640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8fc006b4d0 0x7f8fc006d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.387+0000 7f8fda284640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd415dfe0 0x7f8fd415fcb0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.387+0000 7f8fda284640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8fd40d27f0 0x7f8fd4160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.387+0000 7f8fda284640 1 --2- 172.21.15.77:0/2442381249 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8fd40cf830 0x7f8fd41f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.387+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 >> 172.21.15.77:0/2442381249 conn(0x7f8fd40d3cd0 msgr2=0x7f8fd406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.387+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 shutdown_connections 2024-03-06T07:28:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:57.387+0000 7f8fda284640 1 -- 172.21.15.77:0/2442381249 wait complete. 2024-03-06T07:28:57.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:28:57.519 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:28:57.519 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:28:57.520 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:28:59.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:59.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.620+0000 7f830b4c3640 1 Processor -- start 2024-03-06T07:28:59.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.620+0000 7f830b4c3640 1 -- start start 2024-03-06T07:28:59.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f830b4c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040d22b0 0x7f83040d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f830b4c3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83040b3430 con 0x7f83040b41f0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f830b4c3640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83040b3710 con 0x7f830415dfe0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f830b4c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83040d2c50 con 0x7f83040d22b0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f8309a60640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f83040b41f0 0x7f83040b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56162/0 (socket says 172.21.15.77:56162) 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f8309a60640 1 -- 172.21.15.77:0/495212349 learned_addr learned my addr 172.21.15.77:0/495212349 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f8308a5e640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040d22b0 0x7f83040d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079203639 0 0) 0x7f83040b3430 con 0x7f83040b41f0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f0003520 con 0x7f83040b41f0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478268445 0 0) 0x7f83040b3710 con 0x7f830415dfe0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83040b3430 con 0x7f830415dfe0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 835278902 0 0) 0x7f82f0003520 con 0x7f83040b41f0 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040d22b0 msgr2=0x7f83040d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040d22b0 0x7f83040d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 >> v1:172.21.15.158:6789/0 conn(0x7f830415dfe0 legacy=0x7f83040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040b39f0 con 0x7f83040b41f0 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f82fc0019f0 con 0x7f83040b41f0 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.621+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 >> v1:172.21.15.77:6789/0 conn(0x7f83040b41f0 legacy=0x7f83040b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f82fbfff640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0003b70 0x7f82f0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f82fbfff640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040d22b0 0x7f82f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f82fbfff640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f0004d40 0x7f82f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f000ba70 con 0x7f82f0004d40 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f000bbf0 con 0x7f82f0003b70 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f000bda0 con 0x7f83040d22b0 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f8309a60640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f0004d40 0x7f82f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f830925f640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0003b70 0x7f82f0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f8308a5e640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040d22b0 0x7f82f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f8309a60640 1 -- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040d22b0 msgr2=0x7f82f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f8309a60640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040d22b0 0x7f82f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f8309a60640 1 -- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0003b70 msgr2=0x7f82f0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f8309a60640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0003b70 0x7f82f0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f8309a60640 1 -- 172.21.15.77:0/495212349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0003520 con 0x7f82f0004d40 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.622+0000 7f830925f640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0003b70 0x7f82f0003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f8309a60640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f0004d40 0x7f82f00051c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00a7f0 tx=0x7f82f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f7128d11eae38f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00a410 con 0x7f82f0004d40 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc034630 con 0x7f82f0004d40 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f82fbfff640 1 -- 172.21.15.77:0/495212349 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc02ce60 con 0x7f82f0004d40 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f830b4c3640 1 -- 172.21.15.77:0/495212349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f0004d40 msgr2=0x7f82f00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f830b4c3640 1 --2- 172.21.15.77:0/495212349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f0004d40 0x7f82f00051c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00a7f0 tx=0x7f82f0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f8309a60640 1 -- 172.21.15.77:0/495212349 reap_dead start 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f830b4c3640 1 -- 172.21.15.77:0/495212349 shutdown_connections 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f830b4c3640 1 -- 172.21.15.77:0/495212349 >> 172.21.15.77:0/495212349 conn(0x7f83040d3cd0 msgr2=0x7f83040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f830b4c3640 1 -- 172.21.15.77:0/495212349 shutdown_connections 2024-03-06T07:28:59.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.623+0000 7f830b4c3640 1 -- 172.21.15.77:0/495212349 wait complete. 2024-03-06T07:28:59.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.625+0000 7f830b4c3640 1 Processor -- start 2024-03-06T07:28:59.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.625+0000 7f830b4c3640 1 -- start start 2024-03-06T07:28:59.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f830b4c3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0004d40 0x7f830415a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f830b4c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040cbd10 0x7f8304160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f830b4c3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83041611b0 0x7f83041f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f830b4c3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83040b3d60 con 0x7f83041611b0 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f830b4c3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83040b31e0 con 0x7f82f0004d40 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f830b4c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83040d1c70 con 0x7f83040cbd10 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f8309a60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83041611b0 0x7f83041f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f8309a60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83041611b0 0x7f83041f6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46218/0 (socket says 172.21.15.77:46218) 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f8309a60640 1 -- 172.21.15.77:0/93601358 learned_addr learned my addr 172.21.15.77:0/93601358 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f830925f640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0004d40 0x7f830415a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.626+0000 7f8308a5e640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040cbd10 0x7f8304160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f8309a60640 1 -- 172.21.15.77:0/93601358 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040cbd10 msgr2=0x7f8304160c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f8309a60640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040cbd10 0x7f8304160c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f8309a60640 1 -- 172.21.15.77:0/93601358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0004d40 msgr2=0x7f830415a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f8309a60640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0004d40 0x7f830415a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f8309a60640 1 -- 172.21.15.77:0/93601358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040b6030 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f8309a60640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83041611b0 0x7f83041f6ee0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f82fc005c20 tx=0x7f83040cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f82f9ffb640 1 -- 172.21.15.77:0/93601358 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83000014f0 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f82f9ffb640 1 -- 172.21.15.77:0/93601358 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc044070 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83040b6310 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f82f9ffb640 1 -- 172.21.15.77:0/93601358 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00a410 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.627+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8304160470 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.628+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830404f330 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.628+0000 7f82f9ffb640 1 -- 172.21.15.77:0/93601358 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82fc034c10 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.629+0000 7f82f9ffb640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82e806b4d0 0x7f82e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.629+0000 7f82f9ffb640 1 -- 172.21.15.77:0/93601358 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f82fc0ac4b0 con 0x7f83041611b0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.629+0000 7f830925f640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82e806b4d0 0x7f82e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.629+0000 7f830925f640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82e806b4d0 0x7f82e806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f83040d4340 tx=0x7f8300009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.633+0000 7f82f9ffb640 1 -- 172.21.15.77:0/93601358 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82fc07b560 con 0x7f83041611b0 2024-03-06T07:28:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.976+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83040ca6b0 con 0x7f83041611b0 2024-03-06T07:28:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.976+0000 7f82f9ffb640 1 -- 172.21.15.77:0/93601358 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+3421 (secure 0 0 0) 0x7f82fc07ae60 con 0x7f83041611b0 2024-03-06T07:28:59.978 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:28:59.979 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":86,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:57.726800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:active","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547,8535]},"id":9}]} 2024-03-06T07:28:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 86 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.978+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82e806b4d0 msgr2=0x7f82e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.978+0000 7f830b4c3640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82e806b4d0 0x7f82e806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f83040d4340 tx=0x7f8300009040 comp rx=0 tx=0).stop 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83041611b0 msgr2=0x7f83041f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83041611b0 0x7f83041f6ee0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f82fc005c20 tx=0x7f83040cfd40 comp rx=0 tx=0).stop 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 shutdown_connections 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f82e806b4d0 0x7f82e806d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83041611b0 0x7f83041f6ee0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f83040cbd10 0x7f8304160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 --2- 172.21.15.77:0/93601358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82f0004d40 0x7f830415a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 >> 172.21.15.77:0/93601358 conn(0x7f83040d3cd0 msgr2=0x7f83040cffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 shutdown_connections 2024-03-06T07:28:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:28:59.979+0000 7f830b4c3640 1 -- 172.21.15.77:0/93601358 wait complete. 2024-03-06T07:28:59.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:00.107 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:00.107 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:00.108 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:01.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:01.252 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:02.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.208+0000 7fcd9a09d640 1 Processor -- start 2024-03-06T07:29:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.209+0000 7fcd9a09d640 1 -- start start 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.209+0000 7fcd9a09d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd9415dfe0 0x7fcd940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.209+0000 7fcd9a09d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd940d29c0 0x7fcd940b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.209+0000 7fcd9a09d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940b3430 con 0x7fcd940cf830 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.209+0000 7fcd9a09d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd940b31e0 con 0x7fcd940d29c0 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.209+0000 7fcd9a09d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940b3d60 con 0x7fcd9415dfe0 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.210+0000 7fcd93fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd940d29c0 0x7fcd940b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.210+0000 7fcd93fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd940d29c0 0x7fcd940b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46224/0 (socket says 172.21.15.77:46224) 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.210+0000 7fcd93fff640 1 -- 172.21.15.77:0/3542817060 learned_addr learned my addr 172.21.15.77:0/3542817060 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.210+0000 7fcd937fe640 1 --2- 172.21.15.77:0/3542817060 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd9415dfe0 0x7fcd940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.210+0000 7fcd93fff640 1 -- 172.21.15.77:0/3542817060 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd9415dfe0 msgr2=0x7fcd940cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.210+0000 7fcd93fff640 1 --2- 172.21.15.77:0/3542817060 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd9415dfe0 0x7fcd940cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.210+0000 7fcd93fff640 1 -- 172.21.15.77:0/3542817060 >> v1:172.21.15.158:6789/0 conn(0x7fcd940cf830 legacy=0x7fcd940d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.210+0000 7fcd93fff640 1 -- 172.21.15.77:0/3542817060 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940b3710 con 0x7fcd940d29c0 2024-03-06T07:29:02.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd93fff640 1 --2- 172.21.15.77:0/3542817060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd940d29c0 0x7fcd940b41f0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009e50 tx=0x7fcd8c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a772a68a141474 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd927fc640 1 -- 172.21.15.77:0/3542817060 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c02cb00 con 0x7fcd940d29c0 2024-03-06T07:29:02.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd927fc640 1 -- 172.21.15.77:0/3542817060 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c02cca0 con 0x7fcd940d29c0 2024-03-06T07:29:02.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd927fc640 1 -- 172.21.15.77:0/3542817060 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c034610 con 0x7fcd940d29c0 2024-03-06T07:29:02.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3542817060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd940d29c0 msgr2=0x7fcd940b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3542817060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd940d29c0 0x7fcd940b41f0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009e50 tx=0x7fcd8c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3542817060 shutdown_connections 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3542817060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd940d29c0 0x7fcd940b41f0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3542817060 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd9415dfe0 0x7fcd940cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3542817060 >> 172.21.15.77:0/3542817060 conn(0x7fcd940d3cd0 msgr2=0x7fcd940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3542817060 shutdown_connections 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.211+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3542817060 wait complete. 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.212+0000 7fcd9a09d640 1 Processor -- start 2024-03-06T07:29:02.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd9a09d640 1 -- start start 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd9a09d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd940cf830 0x7fcd940a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd9a09d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd940d29c0 0x7fcd940cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd9a09d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9415dfe0 0x7fcd9415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd9a09d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940d1df0 con 0x7fcd9415dfe0 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd9a09d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940d1c70 con 0x7fcd940cf830 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd9a09d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9415e3c0 con 0x7fcd940d29c0 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd93fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9415dfe0 0x7fcd9415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd93fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9415dfe0 0x7fcd9415be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46236/0 (socket says 172.21.15.77:46236) 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd93fff640 1 -- 172.21.15.77:0/3935359131 learned_addr learned my addr 172.21.15.77:0/3935359131 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd937fe640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd940cf830 0x7fcd940a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.213+0000 7fcd92ffd640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd940d29c0 0x7fcd940cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd93fff640 1 -- 172.21.15.77:0/3935359131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd940d29c0 msgr2=0x7fcd940cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd93fff640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd940d29c0 0x7fcd940cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd93fff640 1 -- 172.21.15.77:0/3935359131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd940cf830 msgr2=0x7fcd940a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd93fff640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd940cf830 0x7fcd940a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd93fff640 1 -- 172.21.15.77:0/3935359131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940caf80 con 0x7fcd9415dfe0 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd93fff640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9415dfe0 0x7fcd9415be30 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009e50 tx=0x7fcd8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd7bfff640 1 -- 172.21.15.77:0/3935359131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c043020 con 0x7fcd9415dfe0 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd7bfff640 1 -- 172.21.15.77:0/3935359131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c02ce50 con 0x7fcd9415dfe0 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd940cb260 con 0x7fcd9415dfe0 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd7bfff640 1 -- 172.21.15.77:0/3935359131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c009510 con 0x7fcd9415dfe0 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.214+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd940cd7a0 con 0x7fcd9415dfe0 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.215+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9404f330 con 0x7fcd9415dfe0 2024-03-06T07:29:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.216+0000 7fcd7bfff640 1 -- 172.21.15.77:0/3935359131 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd8c046020 con 0x7fcd9415dfe0 2024-03-06T07:29:02.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.216+0000 7fcd7bfff640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd8006b4d0 0x7fcd8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.216+0000 7fcd7bfff640 1 -- 172.21.15.77:0/3935359131 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcd8c0ac630 con 0x7fcd9415dfe0 2024-03-06T07:29:02.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.216+0000 7fcd937fe640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd8006b4d0 0x7fcd8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.217+0000 7fcd937fe640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd8006b4d0 0x7fcd8006d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009aa0 tx=0x7fcd84009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.221+0000 7fcd7bfff640 1 -- 172.21.15.77:0/3935359131 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd8c04d030 con 0x7fcd9415dfe0 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.438+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd940ca380 con 0x7fcd9415dfe0 2024-03-06T07:29:02.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.438+0000 7fcd7bfff640 1 -- 172.21.15.77:0/3935359131 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+3421 (secure 0 0 0) 0x7fcd8c07ae30 con 0x7fcd9415dfe0 2024-03-06T07:29:02.440 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:02.440 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":86,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:57.726800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:active","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547,8535]},"id":9}]} 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 86 2024-03-06T07:29:02.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd8006b4d0 msgr2=0x7fcd8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd8006b4d0 0x7fcd8006d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009aa0 tx=0x7fcd84009190 comp rx=0 tx=0).stop 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9415dfe0 msgr2=0x7fcd9415be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9415dfe0 0x7fcd9415be30 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009e50 tx=0x7fcd8c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 shutdown_connections 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd8006b4d0 0x7fcd8006d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9415dfe0 0x7fcd9415be30 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd940d29c0 0x7fcd940cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 --2- 172.21.15.77:0/3935359131 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd940cf830 0x7fcd940a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 >> 172.21.15.77:0/3935359131 conn(0x7fcd940d3cd0 msgr2=0x7fcd940b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 shutdown_connections 2024-03-06T07:29:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:02.440+0000 7fcd9a09d640 1 -- 172.21.15.77:0/3935359131 wait complete. 2024-03-06T07:29:02.455 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:02.572 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:02.572 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:02.572 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:04.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:04.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.672+0000 7ff6a3ff7640 1 Processor -- start 2024-03-06T07:29:04.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.672+0000 7ff6a3ff7640 1 -- start start 2024-03-06T07:29:04.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a3ff7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff69c0cce40 0x7ff69c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a3ff7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff69c0cd7a0 0x7ff69c0cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a3ff7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69c071f40 con 0x7ff69c1515e0 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a3ff7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69c0cd2f0 con 0x7ff69c0cce40 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a3ff7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69c070a70 con 0x7ff69c0cd7a0 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a1592640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff69c0cce40 0x7ff69c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a1592640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff69c0cce40 0x7ff69c15b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46240/0 (socket says 172.21.15.77:46240) 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a1592640 1 -- 172.21.15.77:0/4191409297 learned_addr learned my addr 172.21.15.77:0/4191409297 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a2594640 1 --2- 172.21.15.77:0/4191409297 >> v2:172.21.15.158:3300/0 conn(0x7ff69c0cd7a0 0x7ff69c0cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a1592640 1 -- 172.21.15.77:0/4191409297 >> v2:172.21.15.158:3300/0 conn(0x7ff69c0cd7a0 msgr2=0x7ff69c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a1592640 1 --2- 172.21.15.77:0/4191409297 >> v2:172.21.15.158:3300/0 conn(0x7ff69c0cd7a0 0x7ff69c0cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a1592640 1 -- 172.21.15.77:0/4191409297 >> v1:172.21.15.158:6789/0 conn(0x7ff69c1515e0 legacy=0x7ff69c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.673+0000 7ff6a1592640 1 -- 172.21.15.77:0/4191409297 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c070d50 con 0x7ff69c0cce40 2024-03-06T07:29:04.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.674+0000 7ff6a1592640 1 --2- 172.21.15.77:0/4191409297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c15b430 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7ff69c0cc340 tx=0x7ff69802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fac8cccb3ee5edf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.674+0000 7ff6a0d91640 1 -- 172.21.15.77:0/4191409297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698034070 con 0x7ff69c0cce40 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.674+0000 7ff6a0d91640 1 -- 172.21.15.77:0/4191409297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff698030860 con 0x7ff69c0cce40 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.674+0000 7ff6a0d91640 1 -- 172.21.15.77:0/4191409297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698030c30 con 0x7ff69c0cce40 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.675+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4191409297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 msgr2=0x7ff69c15b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.675+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4191409297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c15b430 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7ff69c0cc340 tx=0x7ff69802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.675+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4191409297 shutdown_connections 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.675+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4191409297 >> v2:172.21.15.158:3300/0 conn(0x7ff69c0cd7a0 0x7ff69c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.675+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4191409297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c15b430 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.675+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4191409297 >> 172.21.15.77:0/4191409297 conn(0x7ff69c0cf8a0 msgr2=0x7ff69c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.675+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4191409297 shutdown_connections 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.675+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4191409297 wait complete. 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.676+0000 7ff6a3ff7640 1 Processor -- start 2024-03-06T07:29:04.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.676+0000 7ff6a3ff7640 1 -- start start 2024-03-06T07:29:04.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a3ff7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c13d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a3ff7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff69c0cd7a0 0x7ff69c153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a3ff7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff69c1515e0 0x7ff69c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a3ff7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0ce0c0 con 0x7ff69c0cce40 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a3ff7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c071310 con 0x7ff69c0cd7a0 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a3ff7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0b3f90 con 0x7ff69c1515e0 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c13d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c13d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46248/0 (socket says 172.21.15.77:46248) 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 -- 172.21.15.77:0/4245280721 learned_addr learned my addr 172.21.15.77:0/4245280721 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1592640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff69c0cd7a0 0x7ff69c153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a2594640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff69c1515e0 0x7ff69c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 -- 172.21.15.77:0/4245280721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff69c1515e0 msgr2=0x7ff69c15d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff69c1515e0 0x7ff69c15d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 -- 172.21.15.77:0/4245280721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff69c0cd7a0 msgr2=0x7ff69c153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff69c0cd7a0 0x7ff69c153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 -- 172.21.15.77:0/4245280721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c153f40 con 0x7ff69c0cce40 2024-03-06T07:29:04.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a1d93640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c13d2f0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00c340 tx=0x7ff68c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6927fc640 1 -- 172.21.15.77:0/4245280721 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c017070 con 0x7ff69c0cce40 2024-03-06T07:29:04.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6927fc640 1 -- 172.21.15.77:0/4245280721 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c0052d0 con 0x7ff69c0cce40 2024-03-06T07:29:04.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6927fc640 1 -- 172.21.15.77:0/4245280721 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c003850 con 0x7ff69c0cce40 2024-03-06T07:29:04.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c154220 con 0x7ff69c0cce40 2024-03-06T07:29:04.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.677+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c15b8b0 con 0x7ff69c0cce40 2024-03-06T07:29:04.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.678+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c068840 con 0x7ff69c0cce40 2024-03-06T07:29:04.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.679+0000 7ff6927fc640 1 -- 172.21.15.77:0/4245280721 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff68c01a070 con 0x7ff69c0cce40 2024-03-06T07:29:04.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.679+0000 7ff6927fc640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff68006b4d0 0x7ff68006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.680+0000 7ff6a1592640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff68006b4d0 0x7ff68006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.680+0000 7ff6927fc640 1 -- 172.21.15.77:0/4245280721 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff68c0893c0 con 0x7ff69c0cce40 2024-03-06T07:29:04.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.680+0000 7ff6a1592640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff68006b4d0 0x7ff68006d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff698036000 tx=0x7ff698008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.684+0000 7ff6927fc640 1 -- 172.21.15.77:0/4245280721 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff68c057c40 con 0x7ff69c0cce40 2024-03-06T07:29:04.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.901+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff69c15b3c0 con 0x7ff69c0cce40 2024-03-06T07:29:04.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.902+0000 7ff6927fc640 1 -- 172.21.15.77:0/4245280721 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+3421 (secure 0 0 0) 0x7ff68c05bce0 con 0x7ff69c0cce40 2024-03-06T07:29:04.905 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:04.905 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":86,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:28:57.726800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:active","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547,8535]},"id":9}]} 2024-03-06T07:29:04.905 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 86 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff68006b4d0 msgr2=0x7ff68006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff68006b4d0 0x7ff68006d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff698036000 tx=0x7ff698008030 comp rx=0 tx=0).stop 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 msgr2=0x7ff69c13d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c13d2f0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00c340 tx=0x7ff68c00c800 comp rx=0 tx=0).stop 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 shutdown_connections 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff69c1515e0 0x7ff69c15d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff69c0cd7a0 0x7ff69c153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff68006b4d0 0x7ff68006d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 --2- 172.21.15.77:0/4245280721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff69c0cce40 0x7ff69c13d2f0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 >> 172.21.15.77:0/4245280721 conn(0x7ff69c0cf8a0 msgr2=0x7ff69c150840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 shutdown_connections 2024-03-06T07:29:04.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:04.904+0000 7ff6a3ff7640 1 -- 172.21.15.77:0/4245280721 wait complete. 2024-03-06T07:29:04.922 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:05.039 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:05.039 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:05.039 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:29:05.040 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:29:05.040 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:05.055 DEBUG:teuthology.orchestra.run.smithi077:> stdin-killer -- python3 -c ' 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> import fcntl 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> import sys 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> import time 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> while True: 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> try: 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> if os.read(0, 4096) == b"": 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> break 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> except BlockingIOError: 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> pass 2024-03-06T07:29:05.056 DEBUG:teuthology.orchestra.run.smithi077:> time.sleep(2) 2024-03-06T07:29:05.057 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:29:05.098 DEBUG:teuthology.orchestra.run.smithi077:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:05.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:05.114 INFO:teuthology.orchestra.run.smithi077.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:29:05.170 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:06.115 DEBUG:teuthology.orchestra.run.smithi077:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 2ah/42d Inode: 2199023255552 Links: 2 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-06 07:29:05.163367538 +0000 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-06 07:29:05.163367538 +0000 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-06 07:29:05.163367538 +0000 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-03-06T07:29:06.132 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:06.132 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:29:06.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:06.346 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:29:06.346 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:29:06.347 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:06.464 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:29:06.464 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:29:06.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:06.581 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:29:06.581 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:29:06.581 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:29:06.708 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:29:06.708 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:06.709 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json cache drop 2024-03-06T07:29:06.829 INFO:teuthology.orchestra.run.smithi077.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":330,"bytes":65167}},"duration":0.0073876419999999998} 2024-03-06T07:29:06.829 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65167, "items": 330 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.007387642, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:29:06.829 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:29:06.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.931+0000 7f7b2db79640 1 Processor -- start 2024-03-06T07:29:06.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.931+0000 7f7b2db79640 1 -- start start 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b2db79640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7b281614d0 0x7f7b280d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b2db79640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b281618b0 con 0x7f7b280d1910 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b2db79640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b280cf400 con 0x7f7b280d24b0 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b2db79640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b280cf6a0 con 0x7f7b281614d0 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b26ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7b280d1910 0x7f7b280d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56172/0 (socket says 172.21.15.77:56172) 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b26ffd640 1 -- 172.21.15.77:0/184005327 learned_addr learned my addr 172.21.15.77:0/184005327 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b277fe640 1 --2- 172.21.15.77:0/184005327 >> v2:172.21.15.158:3300/0 conn(0x7f7b281614d0 0x7f7b280d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b267fc640 1 -- 172.21.15.77:0/184005327 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269019142 0 0) 0x7f7b281618b0 con 0x7f7b280d1910 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b267fc640 1 -- 172.21.15.77:0/184005327 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b14003520 con 0x7f7b280d1910 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b267fc640 1 -- 172.21.15.77:0/184005327 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211320296 0 0) 0x7f7b280cf400 con 0x7f7b280d24b0 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b267fc640 1 -- 172.21.15.77:0/184005327 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b281618b0 con 0x7f7b280d24b0 2024-03-06T07:29:06.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b277fe640 1 -- 172.21.15.77:0/184005327 >> v1:172.21.15.158:6789/0 conn(0x7f7b280d24b0 legacy=0x7f7b280b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b277fe640 1 -- 172.21.15.77:0/184005327 >> v1:172.21.15.77:6789/0 conn(0x7f7b280d1910 legacy=0x7f7b280d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b267fc640 1 -- 172.21.15.77:0/184005327 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 533428906 0 0) 0x7f7b14003520 con 0x7f7b280d1910 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.932+0000 7f7b277fe640 1 -- 172.21.15.77:0/184005327 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b280cf980 con 0x7f7b281614d0 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.933+0000 7f7b277fe640 1 --2- 172.21.15.77:0/184005327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b281614d0 0x7f7b280d4860 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7b200097a0 tx=0x7f7b2002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc27ab034ffd9b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.933+0000 7f7b267fc640 1 -- 172.21.15.77:0/184005327 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20038070 con 0x7f7b281614d0 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.933+0000 7f7b267fc640 1 -- 172.21.15.77:0/184005327 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20031500 con 0x7f7b281614d0 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.933+0000 7f7b267fc640 1 -- 172.21.15.77:0/184005327 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b10001670 con 0x7f7b281614d0 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.934+0000 7f7b2db79640 1 -- 172.21.15.77:0/184005327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b281614d0 msgr2=0x7f7b280d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.934+0000 7f7b2db79640 1 --2- 172.21.15.77:0/184005327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b281614d0 0x7f7b280d4860 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7b200097a0 tx=0x7f7b2002b820 comp rx=0 tx=0).stop 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.934+0000 7f7b2db79640 1 -- 172.21.15.77:0/184005327 shutdown_connections 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.934+0000 7f7b2db79640 1 --2- 172.21.15.77:0/184005327 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b281614d0 0x7f7b280d4860 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.934+0000 7f7b2db79640 1 -- 172.21.15.77:0/184005327 >> 172.21.15.77:0/184005327 conn(0x7f7b280d3d40 msgr2=0x7f7b280d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.934+0000 7f7b2db79640 1 -- 172.21.15.77:0/184005327 shutdown_connections 2024-03-06T07:29:06.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.934+0000 7f7b2db79640 1 -- 172.21.15.77:0/184005327 wait complete. 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.935+0000 7f7b2db79640 1 Processor -- start 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.935+0000 7f7b2db79640 1 -- start start 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b2db79640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b280d1910 0x7f7b281f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b2db79640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b280d24b0 0x7f7b281530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b2db79640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b281614d0 0x7f7b280c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b2db79640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280b1540 con 0x7f7b280d1910 2024-03-06T07:29:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b2db79640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280b0910 con 0x7f7b280d24b0 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b2db79640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280b16c0 con 0x7f7b281614d0 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b280d1910 0x7f7b281f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b277fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b280d1910 0x7f7b281f4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46254/0 (socket says 172.21.15.77:46254) 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b277fe640 1 -- 172.21.15.77:0/3469068356 learned_addr learned my addr 172.21.15.77:0/3469068356 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b26ffd640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b280d24b0 0x7f7b281530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b27fff640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b281614d0 0x7f7b280c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b277fe640 1 -- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b281614d0 msgr2=0x7f7b280c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.936+0000 7f7b277fe640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b281614d0 0x7f7b280c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b277fe640 1 -- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b280d24b0 msgr2=0x7f7b281530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b277fe640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b280d24b0 0x7f7b281530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b277fe640 1 -- 172.21.15.77:0/3469068356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b281f4850 con 0x7f7b280d1910 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b26ffd640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b280d24b0 0x7f7b281530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:06.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b277fe640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b280d1910 0x7f7b281f4310 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f7b20036040 tx=0x7f7b200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b1ffff640 1 -- 172.21.15.77:0/3469068356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20042070 con 0x7f7b280d1910 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b1ffff640 1 -- 172.21.15.77:0/3469068356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2003d870 con 0x7f7b280d1910 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b280c7be0 con 0x7f7b280d1910 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b1ffff640 1 -- 172.21.15.77:0/3469068356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20038040 con 0x7f7b280d1910 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.937+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b280c8480 con 0x7f7b280d1910 2024-03-06T07:29:06.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.938+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b28068ff0 con 0x7f7b280d1910 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.939+0000 7f7b1ffff640 1 -- 172.21.15.77:0/3469068356 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b20045070 con 0x7f7b280d1910 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.939+0000 7f7b1ffff640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b0806b4d0 0x7f7b0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.939+0000 7f7b1ffff640 1 -- 172.21.15.77:0/3469068356 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7b200abbc0 con 0x7f7b280d1910 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.939+0000 7f7b26ffd640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b0806b4d0 0x7f7b0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.940+0000 7f7b26ffd640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b0806b4d0 0x7f7b0806d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7b10001eb0 tx=0x7f7b10001150 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:06.944+0000 7f7b1ffff640 1 -- 172.21.15.77:0/3469068356 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b20030070 con 0x7f7b280d1910 2024-03-06T07:29:07.172 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:07.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.187+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f7b280747a0 con 0x7f7b280d1910 2024-03-06T07:29:07.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.897+0000 7f7b1ffff640 1 -- 172.21.15.77:0/3469068356 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v87) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7b2007a3b0 con 0x7f7b280d1910 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b0806b4d0 msgr2=0x7f7b0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b0806b4d0 0x7f7b0806d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7b10001eb0 tx=0x7f7b10001150 comp rx=0 tx=0).stop 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b280d1910 msgr2=0x7f7b281f4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b280d1910 0x7f7b281f4310 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f7b20036040 tx=0x7f7b200046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 shutdown_connections 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b281614d0 0x7f7b280c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b280d24b0 0x7f7b281530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b0806b4d0 0x7f7b0806d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 --2- 172.21.15.77:0/3469068356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b280d1910 0x7f7b281f4310 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 >> 172.21.15.77:0/3469068356 conn(0x7f7b280d3d40 msgr2=0x7f7b280b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 shutdown_connections 2024-03-06T07:29:07.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:07.900+0000 7f7b2db79640 1 -- 172.21.15.77:0/3469068356 wait complete. 2024-03-06T07:29:07.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:08.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.012+0000 7f89bfbac640 1 Processor -- start 2024-03-06T07:29:08.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.012+0000 7f89bfbac640 1 -- start start 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.012+0000 7f89bfbac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f89b81515e0 0x7f89b80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bfbac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8071f40 con 0x7f89b815b430 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bfbac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b815b8e0 con 0x7f89b80cc340 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bfbac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b8070a70 con 0x7f89b81515e0 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bd147640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f89b815b430 0x7f89b8071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56182/0 (socket says 172.21.15.77:56182) 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bd147640 1 -- 172.21.15.77:0/382315815 learned_addr learned my addr 172.21.15.77:0/382315815 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bd948640 1 --2- 172.21.15.77:0/382315815 >> v2:172.21.15.158:3300/0 conn(0x7f89b81515e0 0x7f89b80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bc946640 1 -- 172.21.15.77:0/382315815 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838205290 0 0) 0x7f89b8071f40 con 0x7f89b815b430 2024-03-06T07:29:08.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bc946640 1 -- 172.21.15.77:0/382315815 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a4003520 con 0x7f89b815b430 2024-03-06T07:29:08.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bd948640 1 -- 172.21.15.77:0/382315815 >> v1:172.21.15.158:6789/0 conn(0x7f89b80cc340 legacy=0x7f89b80cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bd948640 1 -- 172.21.15.77:0/382315815 >> v1:172.21.15.77:6789/0 conn(0x7f89b815b430 legacy=0x7f89b8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.013+0000 7f89bd948640 1 -- 172.21.15.77:0/382315815 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b8070d50 con 0x7f89b81515e0 2024-03-06T07:29:08.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.014+0000 7f89bd948640 1 --2- 172.21.15.77:0/382315815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 0x7f89b80cce40 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f89a80097a0 tx=0x7f89a802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e6a8fe39270073b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.014+0000 7f89bc946640 1 -- 172.21.15.77:0/382315815 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8034070 con 0x7f89b81515e0 2024-03-06T07:29:08.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.014+0000 7f89bc946640 1 -- 172.21.15.77:0/382315815 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a8030610 con 0x7f89b81515e0 2024-03-06T07:29:08.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.014+0000 7f89bc946640 1 -- 172.21.15.77:0/382315815 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a80309c0 con 0x7f89b81515e0 2024-03-06T07:29:08.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.014+0000 7f89bfbac640 1 -- 172.21.15.77:0/382315815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 msgr2=0x7f89b80cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.014+0000 7f89bfbac640 1 --2- 172.21.15.77:0/382315815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 0x7f89b80cce40 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f89a80097a0 tx=0x7f89a802b960 comp rx=0 tx=0).stop 2024-03-06T07:29:08.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.015+0000 7f89bfbac640 1 -- 172.21.15.77:0/382315815 shutdown_connections 2024-03-06T07:29:08.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.015+0000 7f89bfbac640 1 --2- 172.21.15.77:0/382315815 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 0x7f89b80cce40 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.015+0000 7f89bfbac640 1 -- 172.21.15.77:0/382315815 >> 172.21.15.77:0/382315815 conn(0x7f89b80cf8a0 msgr2=0x7f89b80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.015+0000 7f89bfbac640 1 -- 172.21.15.77:0/382315815 shutdown_connections 2024-03-06T07:29:08.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.015+0000 7f89bfbac640 1 -- 172.21.15.77:0/382315815 wait complete. 2024-03-06T07:29:08.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.016+0000 7f89bfbac640 1 Processor -- start 2024-03-06T07:29:08.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.016+0000 7f89bfbac640 1 -- start start 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.016+0000 7f89bfbac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89b80cc340 0x7f89b81f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.016+0000 7f89bfbac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 0x7f89b81ef430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.016+0000 7f89bfbac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b815b430 0x7f89b8137320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.016+0000 7f89bfbac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80cdeb0 con 0x7f89b815b430 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.016+0000 7f89bfbac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b8071310 con 0x7f89b81515e0 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.016+0000 7f89bfbac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80ce030 con 0x7f89b80cc340 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b815b430 0x7f89b8137320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b815b430 0x7f89b8137320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46264/0 (socket says 172.21.15.77:46264) 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89bd948640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89b80cc340 0x7f89b81f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 -- 172.21.15.77:0/3544546063 learned_addr learned my addr 172.21.15.77:0/3544546063 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89bd147640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 0x7f89b81ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 -- 172.21.15.77:0/3544546063 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89b80cc340 msgr2=0x7f89b81f5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89b80cc340 0x7f89b81f5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 -- 172.21.15.77:0/3544546063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 msgr2=0x7f89b81ef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 0x7f89b81ef430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 -- 172.21.15.77:0/3544546063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b81f5be0 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89be149640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b815b430 0x7f89b8137320 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c6b0 tx=0x7f89b000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89ae7fc640 1 -- 172.21.15.77:0/3544546063 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0003b70 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89ae7fc640 1 -- 172.21.15.77:0/3544546063 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b0003d30 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b81f5ec0 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89ae7fc640 1 -- 172.21.15.77:0/3544546063 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0005310 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.017+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b8137860 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.018+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b8068840 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.019+0000 7f89ae7fc640 1 -- 172.21.15.77:0/3544546063 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89b00054b0 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.019+0000 7f89ae7fc640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f899c06b4d0 0x7f899c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.019+0000 7f89ae7fc640 1 -- 172.21.15.77:0/3544546063 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f89b0089ed0 con 0x7f89b815b430 2024-03-06T07:29:08.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.019+0000 7f89bd948640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f899c06b4d0 0x7f899c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.019+0000 7f89bd948640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f899c06b4d0 0x7f899c06d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f89a8036000 tx=0x7f89a8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.023+0000 7f89ae7fc640 1 -- 172.21.15.77:0/3544546063 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89b0058750 con 0x7f89b815b430 2024-03-06T07:29:08.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.235+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89b806a1b0 con 0x7f89b815b430 2024-03-06T07:29:08.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.236+0000 7f89ae7fc640 1 -- 172.21.15.77:0/3544546063 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3418 (secure 0 0 0) 0x7f89b806a1b0 con 0x7f89b815b430 2024-03-06T07:29:08.238 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:08.238 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:07.901583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:08.238 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 88 2024-03-06T07:29:08.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f899c06b4d0 msgr2=0x7f899c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f899c06b4d0 0x7f899c06d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f89a8036000 tx=0x7f89a8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:08.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b815b430 msgr2=0x7f89b8137320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b815b430 0x7f89b8137320 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f89b000c6b0 tx=0x7f89b000cb70 comp rx=0 tx=0).stop 2024-03-06T07:29:08.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 shutdown_connections 2024-03-06T07:29:08.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f899c06b4d0 0x7f899c06d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b815b430 0x7f89b8137320 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89b81515e0 0x7f89b81ef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 --2- 172.21.15.77:0/3544546063 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89b80cc340 0x7f89b81f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 >> 172.21.15.77:0/3544546063 conn(0x7f89b80cf8a0 msgr2=0x7f89b80cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 shutdown_connections 2024-03-06T07:29:08.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:08.238+0000 7f89bfbac640 1 -- 172.21.15.77:0/3544546063 wait complete. 2024-03-06T07:29:08.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:07.901583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:08.452 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:09.174 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:09.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:09.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.563+0000 7f2c24981640 1 Processor -- start 2024-03-06T07:29:09.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.563+0000 7f2c24981640 1 -- start start 2024-03-06T07:29:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c24981640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c201424f0 0x7f2c200cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c24981640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2c2006da40 0x7f2c2015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c24981640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c200cc980 con 0x7f2c200cd7a0 2024-03-06T07:29:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c24981640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c200cd220 con 0x7f2c201424f0 2024-03-06T07:29:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c24981640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c200ccc60 con 0x7f2c2006da40 2024-03-06T07:29:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1e59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c201424f0 0x7f2c200cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1e59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c201424f0 0x7f2c200cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47616/0 (socket says 172.21.15.77:47616) 2024-03-06T07:29:09.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1e59c640 1 -- 172.21.15.77:0/2470836151 learned_addr learned my addr 172.21.15.77:0/2470836151 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:09.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1ed9d640 1 --2- 172.21.15.77:0/2470836151 >> v2:172.21.15.158:3300/0 conn(0x7f2c2006da40 0x7f2c2015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1e59c640 1 -- 172.21.15.77:0/2470836151 >> v2:172.21.15.158:3300/0 conn(0x7f2c2006da40 msgr2=0x7f2c2015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1e59c640 1 --2- 172.21.15.77:0/2470836151 >> v2:172.21.15.158:3300/0 conn(0x7f2c2006da40 0x7f2c2015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1e59c640 1 -- 172.21.15.77:0/2470836151 >> v1:172.21.15.158:6789/0 conn(0x7f2c200cd7a0 legacy=0x7f2c2006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1e59c640 1 -- 172.21.15.77:0/2470836151 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c200ccf40 con 0x7f2c201424f0 2024-03-06T07:29:09.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.564+0000 7f2c1ed9d640 1 --2- 172.21.15.77:0/2470836151 >> v2:172.21.15.158:3300/0 conn(0x7f2c2006da40 0x7f2c2015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.565+0000 7f2c1e59c640 1 --2- 172.21.15.77:0/2470836151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c201424f0 0x7f2c200cc2f0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f2c20068700 tx=0x7f2c1002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3eefbc4aded09a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.565+0000 7f2c1dd9b640 1 -- 172.21.15.77:0/2470836151 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10034070 con 0x7f2c201424f0 2024-03-06T07:29:09.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.565+0000 7f2c1dd9b640 1 -- 172.21.15.77:0/2470836151 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c10030860 con 0x7f2c201424f0 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.565+0000 7f2c1dd9b640 1 -- 172.21.15.77:0/2470836151 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10030c30 con 0x7f2c201424f0 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.566+0000 7f2c24981640 1 -- 172.21.15.77:0/2470836151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c201424f0 msgr2=0x7f2c200cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.566+0000 7f2c24981640 1 --2- 172.21.15.77:0/2470836151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c201424f0 0x7f2c200cc2f0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f2c20068700 tx=0x7f2c1002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.566+0000 7f2c24981640 1 -- 172.21.15.77:0/2470836151 shutdown_connections 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.566+0000 7f2c24981640 1 --2- 172.21.15.77:0/2470836151 >> v2:172.21.15.158:3300/0 conn(0x7f2c2006da40 0x7f2c2015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.566+0000 7f2c24981640 1 --2- 172.21.15.77:0/2470836151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c201424f0 0x7f2c200cc2f0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.566+0000 7f2c24981640 1 -- 172.21.15.77:0/2470836151 >> 172.21.15.77:0/2470836151 conn(0x7f2c200cf8a0 msgr2=0x7f2c200cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.566+0000 7f2c24981640 1 -- 172.21.15.77:0/2470836151 shutdown_connections 2024-03-06T07:29:09.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.566+0000 7f2c24981640 1 -- 172.21.15.77:0/2470836151 wait complete. 2024-03-06T07:29:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.567+0000 7f2c24981640 1 Processor -- start 2024-03-06T07:29:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.567+0000 7f2c24981640 1 -- start start 2024-03-06T07:29:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c24981640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c2006da40 0x7f2c20140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c24981640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c200cd7a0 0x7f2c200aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c24981640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c201424f0 0x7f2c200ab990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c24981640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c20071ce0 con 0x7f2c200cd7a0 2024-03-06T07:29:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c24981640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c20071b60 con 0x7f2c201424f0 2024-03-06T07:29:09.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c24981640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c20071e60 con 0x7f2c2006da40 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c200cd7a0 0x7f2c200aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c200cd7a0 0x7f2c200aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47624/0 (socket says 172.21.15.77:47624) 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 -- 172.21.15.77:0/2008288732 learned_addr learned my addr 172.21.15.77:0/2008288732 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c1ed9d640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c201424f0 0x7f2c200ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c1e59c640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c2006da40 0x7f2c20140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 -- 172.21.15.77:0/2008288732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c2006da40 msgr2=0x7f2c20140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c2006da40 0x7f2c20140a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 -- 172.21.15.77:0/2008288732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c201424f0 msgr2=0x7f2c200ab990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c201424f0 0x7f2c200ab990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 -- 172.21.15.77:0/2008288732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c200aa8f0 con 0x7f2c200cd7a0 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.568+0000 7f2c17fff640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c200cd7a0 0x7f2c200aed50 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800c430 tx=0x7f2c0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.569+0000 7f2c16ffd640 1 -- 172.21.15.77:0/2008288732 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c080052d0 con 0x7f2c200cd7a0 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.569+0000 7f2c16ffd640 1 -- 172.21.15.77:0/2008288732 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c08005490 con 0x7f2c200cd7a0 2024-03-06T07:29:09.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.569+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c200aabd0 con 0x7f2c200cd7a0 2024-03-06T07:29:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.569+0000 7f2c16ffd640 1 -- 172.21.15.77:0/2008288732 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08007260 con 0x7f2c200cd7a0 2024-03-06T07:29:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.569+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c200ab190 con 0x7f2c200cd7a0 2024-03-06T07:29:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.570+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2004f330 con 0x7f2c200cd7a0 2024-03-06T07:29:09.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.570+0000 7f2c16ffd640 1 -- 172.21.15.77:0/2008288732 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c08007400 con 0x7f2c200cd7a0 2024-03-06T07:29:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.570+0000 7f2c16ffd640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c0006b4d0 0x7f2c0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.570+0000 7f2c16ffd640 1 -- 172.21.15.77:0/2008288732 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2c080896a0 con 0x7f2c200cd7a0 2024-03-06T07:29:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.571+0000 7f2c1e59c640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c0006b4d0 0x7f2c0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.571+0000 7f2c1e59c640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c0006b4d0 0x7f2c0006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2c10036000 tx=0x7f2c1003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.575+0000 7f2c16ffd640 1 -- 172.21.15.77:0/2008288732 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c08057e90 con 0x7f2c200cd7a0 2024-03-06T07:29:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.784+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c200d0620 con 0x7f2c200cd7a0 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.784+0000 7f2c16ffd640 1 -- 172.21.15.77:0/2008288732 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3418 (secure 0 0 0) 0x7f2c0805bf30 con 0x7f2c200cd7a0 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:07.901583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 88 2024-03-06T07:29:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c0006b4d0 msgr2=0x7f2c0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c0006b4d0 0x7f2c0006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2c10036000 tx=0x7f2c1003b040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c200cd7a0 msgr2=0x7f2c200aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c200cd7a0 0x7f2c200aed50 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800c430 tx=0x7f2c0800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 shutdown_connections 2024-03-06T07:29:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c0006b4d0 0x7f2c0006d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c200cd7a0 0x7f2c200aed50 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c201424f0 0x7f2c200ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 --2- 172.21.15.77:0/2008288732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c2006da40 0x7f2c20140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 >> 172.21.15.77:0/2008288732 conn(0x7f2c200cf8a0 msgr2=0x7f2c2015c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 shutdown_connections 2024-03-06T07:29:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:09.786+0000 7f2c24981640 1 -- 172.21.15.77:0/2008288732 wait complete. 2024-03-06T07:29:10.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:07.901583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:10.001 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:11.001 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:11.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.111+0000 7fdb6ec2a640 1 Processor -- start 2024-03-06T07:29:11.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.111+0000 7fdb6ec2a640 1 -- start start 2024-03-06T07:29:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.112+0000 7fdb6ec2a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdb6806da40 0x7fdb680cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.112+0000 7fdb6ec2a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb680cc980 con 0x7fdb680cd7a0 2024-03-06T07:29:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.112+0000 7fdb6ec2a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb680ccc60 con 0x7fdb681424f0 2024-03-06T07:29:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.112+0000 7fdb6ec2a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb680cd220 con 0x7fdb6806da40 2024-03-06T07:29:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.112+0000 7fdb67fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdb6806da40 0x7fdb680cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.112+0000 7fdb6d1c7640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdb680cd7a0 0x7fdb68071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:60180/0 (socket says 172.21.15.77:60180) 2024-03-06T07:29:11.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.112+0000 7fdb6d1c7640 1 -- 172.21.15.77:0/2986788470 learned_addr learned my addr 172.21.15.77:0/2986788470 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.113+0000 7fdb67fff640 1 -- 172.21.15.77:0/2986788470 >> v1:172.21.15.158:6789/0 conn(0x7fdb681424f0 legacy=0x7fdb6806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.113+0000 7fdb67fff640 1 -- 172.21.15.77:0/2986788470 >> v1:172.21.15.77:6789/0 conn(0x7fdb680cd7a0 legacy=0x7fdb68071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.113+0000 7fdb677fe640 1 -- 172.21.15.77:0/2986788470 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918846894 0 0) 0x7fdb680ccc60 con 0x7fdb681424f0 2024-03-06T07:29:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.113+0000 7fdb67fff640 1 -- 172.21.15.77:0/2986788470 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb680ccf40 con 0x7fdb6806da40 2024-03-06T07:29:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.113+0000 7fdb67fff640 1 --2- 172.21.15.77:0/2986788470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6806da40 0x7fdb680cc2f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fdb540097a0 tx=0x7fdb5402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f8a90776611487 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb677fe640 1 -- 172.21.15.77:0/2986788470 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54037070 con 0x7fdb6806da40 2024-03-06T07:29:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb677fe640 1 -- 172.21.15.77:0/2986788470 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb54030710 con 0x7fdb6806da40 2024-03-06T07:29:11.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb677fe640 1 -- 172.21.15.77:0/2986788470 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54030ae0 con 0x7fdb6806da40 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2986788470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6806da40 msgr2=0x7fdb680cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb6ec2a640 1 --2- 172.21.15.77:0/2986788470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6806da40 0x7fdb680cc2f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fdb540097a0 tx=0x7fdb5402b790 comp rx=0 tx=0).stop 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2986788470 shutdown_connections 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb6ec2a640 1 --2- 172.21.15.77:0/2986788470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6806da40 0x7fdb680cc2f0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2986788470 >> 172.21.15.77:0/2986788470 conn(0x7fdb680cf8a0 msgr2=0x7fdb680cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2986788470 shutdown_connections 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.114+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2986788470 wait complete. 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.115+0000 7fdb6ec2a640 1 Processor -- start 2024-03-06T07:29:11.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.115+0000 7fdb6ec2a640 1 -- start start 2024-03-06T07:29:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb6ec2a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb6806da40 0x7fdb680aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb6ec2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb680cd7a0 0x7fdb6813fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb6ec2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb681424f0 0x7fdb68159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb6ec2a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6815d070 con 0x7fdb680cd7a0 2024-03-06T07:29:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb6ec2a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6815cef0 con 0x7fdb6806da40 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb6ec2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6815d1f0 con 0x7fdb681424f0 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb680cd7a0 0x7fdb6813fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb680cd7a0 0x7fdb6813fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47630/0 (socket says 172.21.15.77:47630) 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.116+0000 7fdb67fff640 1 -- 172.21.15.77:0/2438460106 learned_addr learned my addr 172.21.15.77:0/2438460106 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb6c9c6640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb6806da40 0x7fdb680aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb6d1c7640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb681424f0 0x7fdb68159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb67fff640 1 -- 172.21.15.77:0/2438460106 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb681424f0 msgr2=0x7fdb68159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb67fff640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb681424f0 0x7fdb68159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb67fff640 1 -- 172.21.15.77:0/2438460106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb6806da40 msgr2=0x7fdb680aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb67fff640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb6806da40 0x7fdb680aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb67fff640 1 -- 172.21.15.77:0/2438460106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb681335e0 con 0x7fdb680cd7a0 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb67fff640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb680cd7a0 0x7fdb6813fae0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fdb540097a0 tx=0x7fdb540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb657fa640 1 -- 172.21.15.77:0/2438460106 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54037040 con 0x7fdb680cd7a0 2024-03-06T07:29:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb657fa640 1 -- 172.21.15.77:0/2438460106 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb540329c0 con 0x7fdb680cd7a0 2024-03-06T07:29:11.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb657fa640 1 -- 172.21.15.77:0/2438460106 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54032d90 con 0x7fdb680cd7a0 2024-03-06T07:29:11.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb681338c0 con 0x7fdb680cd7a0 2024-03-06T07:29:11.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.117+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb68133e80 con 0x7fdb680cd7a0 2024-03-06T07:29:11.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.118+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6804f330 con 0x7fdb680cd7a0 2024-03-06T07:29:11.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.118+0000 7fdb657fa640 1 -- 172.21.15.77:0/2438460106 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb54004030 con 0x7fdb680cd7a0 2024-03-06T07:29:11.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.119+0000 7fdb657fa640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb4c06b4d0 0x7fdb4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.119+0000 7fdb657fa640 1 -- 172.21.15.77:0/2438460106 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdb540abf40 con 0x7fdb680cd7a0 2024-03-06T07:29:11.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.119+0000 7fdb6c9c6640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb4c06b4d0 0x7fdb4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.119+0000 7fdb6c9c6640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb4c06b4d0 0x7fdb4c06d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c000f60 tx=0x7fdb5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.123+0000 7fdb657fa640 1 -- 172.21.15.77:0/2438460106 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb54042020 con 0x7fdb680cd7a0 2024-03-06T07:29:11.176 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.329+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb680d0620 con 0x7fdb680cd7a0 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.329+0000 7fdb657fa640 1 -- 172.21.15.77:0/2438460106 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3418 (secure 0 0 0) 0x7fdb5407ad80 con 0x7fdb680cd7a0 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:11.331 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:07.901583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:11.332 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 88 2024-03-06T07:29:11.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb4c06b4d0 msgr2=0x7fdb4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb4c06b4d0 0x7fdb4c06d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c000f60 tx=0x7fdb5c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb680cd7a0 msgr2=0x7fdb6813fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb680cd7a0 0x7fdb6813fae0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fdb540097a0 tx=0x7fdb540046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 shutdown_connections 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdb4c06b4d0 0x7fdb4c06d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb680cd7a0 0x7fdb6813fae0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdb681424f0 0x7fdb68159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 --2- 172.21.15.77:0/2438460106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdb6806da40 0x7fdb680aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 >> 172.21.15.77:0/2438460106 conn(0x7fdb680cf8a0 msgr2=0x7fdb6815f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 shutdown_connections 2024-03-06T07:29:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:11.331+0000 7fdb6ec2a640 1 -- 172.21.15.77:0/2438460106 wait complete. 2024-03-06T07:29:11.548 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:07.901583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:11.549 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:12.550 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:12.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.661+0000 7f54c4df6640 1 Processor -- start 2024-03-06T07:29:12.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.661+0000 7f54c4df6640 1 -- start start 2024-03-06T07:29:12.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.661+0000 7f54c4df6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f54c013dc10 0x7f54c00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54c4df6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c015ef60 0x7f54c00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54c4df6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c015f580 con 0x7f54c015bf30 2024-03-06T07:29:12.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54c4df6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c00cf3f0 con 0x7f54c013dc10 2024-03-06T07:29:12.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54c4df6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c015b210 con 0x7f54c015ef60 2024-03-06T07:29:12.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bdd9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f54c015bf30 0x7f54c015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:60196/0 (socket says 172.21.15.77:60196) 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bdd9b640 1 -- 172.21.15.77:0/4177494338 learned_addr learned my addr 172.21.15.77:0/4177494338 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54be59c640 1 --2- 172.21.15.77:0/4177494338 >> v2:172.21.15.158:3300/0 conn(0x7f54c013dc10 0x7f54c00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bed9d640 1 --2- 172.21.15.77:0/4177494338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c015ef60 0x7f54c00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bd59a640 1 -- 172.21.15.77:0/4177494338 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329430070 0 0) 0x7f54c015f580 con 0x7f54c015bf30 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bd59a640 1 -- 172.21.15.77:0/4177494338 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a8003520 con 0x7f54c015bf30 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bed9d640 1 -- 172.21.15.77:0/4177494338 >> v2:172.21.15.158:3300/0 conn(0x7f54c013dc10 msgr2=0x7f54c00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bed9d640 1 --2- 172.21.15.77:0/4177494338 >> v2:172.21.15.158:3300/0 conn(0x7f54c013dc10 0x7f54c00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bed9d640 1 -- 172.21.15.77:0/4177494338 >> v1:172.21.15.77:6789/0 conn(0x7f54c015bf30 legacy=0x7f54c015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54bed9d640 1 -- 172.21.15.77:0/4177494338 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c015b450 con 0x7f54c015ef60 2024-03-06T07:29:12.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.662+0000 7f54be59c640 1 --2- 172.21.15.77:0/4177494338 >> v2:172.21.15.158:3300/0 conn(0x7f54c013dc10 0x7f54c00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:12.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.663+0000 7f54bed9d640 1 --2- 172.21.15.77:0/4177494338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c015ef60 0x7f54c00b2e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009d90 tx=0x7f54b002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3499573eabf8ecd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.663+0000 7f54bd59a640 1 -- 172.21.15.77:0/4177494338 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0039070 con 0x7f54c015ef60 2024-03-06T07:29:12.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.663+0000 7f54bd59a640 1 -- 172.21.15.77:0/4177494338 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b002cbc0 con 0x7f54c015ef60 2024-03-06T07:29:12.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.663+0000 7f54bd59a640 1 -- 172.21.15.77:0/4177494338 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0034440 con 0x7f54c015ef60 2024-03-06T07:29:12.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.664+0000 7f54c4df6640 1 -- 172.21.15.77:0/4177494338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c015ef60 msgr2=0x7f54c00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.664+0000 7f54c4df6640 1 --2- 172.21.15.77:0/4177494338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c015ef60 0x7f54c00b2e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009d90 tx=0x7f54b002be10 comp rx=0 tx=0).stop 2024-03-06T07:29:12.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.664+0000 7f54c4df6640 1 -- 172.21.15.77:0/4177494338 shutdown_connections 2024-03-06T07:29:12.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.664+0000 7f54c4df6640 1 --2- 172.21.15.77:0/4177494338 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c015ef60 0x7f54c00b2e90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.664+0000 7f54c4df6640 1 --2- 172.21.15.77:0/4177494338 >> v2:172.21.15.158:3300/0 conn(0x7f54c013dc10 0x7f54c00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.664+0000 7f54c4df6640 1 -- 172.21.15.77:0/4177494338 >> 172.21.15.77:0/4177494338 conn(0x7f54c00cf8a0 msgr2=0x7f54c00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.664+0000 7f54c4df6640 1 -- 172.21.15.77:0/4177494338 shutdown_connections 2024-03-06T07:29:12.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.664+0000 7f54c4df6640 1 -- 172.21.15.77:0/4177494338 wait complete. 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.665+0000 7f54c4df6640 1 Processor -- start 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.665+0000 7f54c4df6640 1 -- start start 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54c4df6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c013dc10 0x7f54c0153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54c4df6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54c015bf30 0x7f54c00b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54c4df6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c015ef60 0x7f54c00b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54c4df6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00cd7a0 con 0x7f54c015ef60 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54c4df6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c013dff0 con 0x7f54c015bf30 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54c4df6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00cd920 con 0x7f54c013dc10 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c015ef60 0x7f54c00b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c015ef60 0x7f54c00b2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47634/0 (socket says 172.21.15.77:47634) 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bed9d640 1 -- 172.21.15.77:0/1353279222 learned_addr learned my addr 172.21.15.77:0/1353279222 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bdd9b640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54c015bf30 0x7f54c00b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54be59c640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c013dc10 0x7f54c0153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bed9d640 1 -- 172.21.15.77:0/1353279222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c013dc10 msgr2=0x7f54c0153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bed9d640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c013dc10 0x7f54c0153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bed9d640 1 -- 172.21.15.77:0/1353279222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54c015bf30 msgr2=0x7f54c00b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bed9d640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54c015bf30 0x7f54c00b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.666+0000 7f54bed9d640 1 -- 172.21.15.77:0/1353279222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c0143b70 con 0x7f54c015ef60 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.667+0000 7f54bed9d640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c015ef60 0x7f54c00b2c80 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009eb0 tx=0x7f54b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.667+0000 7f54aeffd640 1 -- 172.21.15.77:0/1353279222 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0039050 con 0x7f54c015ef60 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.667+0000 7f54aeffd640 1 -- 172.21.15.77:0/1353279222 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b0045070 con 0x7f54c015ef60 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.667+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c0143e50 con 0x7f54c015ef60 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.667+0000 7f54aeffd640 1 -- 172.21.15.77:0/1353279222 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0009450 con 0x7f54c015ef60 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.667+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c0144410 con 0x7f54c015ef60 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.668+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c004f330 con 0x7f54c015ef60 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.668+0000 7f54aeffd640 1 -- 172.21.15.77:0/1353279222 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54b003e830 con 0x7f54c015ef60 2024-03-06T07:29:12.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.668+0000 7f54aeffd640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f54a006b4d0 0x7f54a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.669+0000 7f54be59c640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f54a006b4d0 0x7f54a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.669+0000 7f54aeffd640 1 -- 172.21.15.77:0/1353279222 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f54b00ad0e0 con 0x7f54c015ef60 2024-03-06T07:29:12.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.669+0000 7f54be59c640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f54a006b4d0 0x7f54a006d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009560 tx=0x7f54b8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.673+0000 7f54aeffd640 1 -- 172.21.15.77:0/1353279222 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54b007bb40 con 0x7f54c015ef60 2024-03-06T07:29:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.886+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54c00b03e0 con 0x7f54c015ef60 2024-03-06T07:29:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.886+0000 7f54aeffd640 1 -- 172.21.15.77:0/1353279222 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3418 (secure 0 0 0) 0x7f54b007fbe0 con 0x7f54c015ef60 2024-03-06T07:29:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:12.888 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:07.901583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 88 2024-03-06T07:29:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f54a006b4d0 msgr2=0x7f54a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f54a006b4d0 0x7f54a006d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009560 tx=0x7f54b8009190 comp rx=0 tx=0).stop 2024-03-06T07:29:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c015ef60 msgr2=0x7f54c00b2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c015ef60 0x7f54c00b2c80 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009eb0 tx=0x7f54b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 shutdown_connections 2024-03-06T07:29:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f54a006b4d0 0x7f54a006d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c015ef60 0x7f54c00b2c80 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f54c015bf30 0x7f54c00b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 --2- 172.21.15.77:0/1353279222 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f54c013dc10 0x7f54c0153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 >> 172.21.15.77:0/1353279222 conn(0x7f54c00cf8a0 msgr2=0x7f54c015d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 shutdown_connections 2024-03-06T07:29:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:12.888+0000 7f54c4df6640 1 -- 172.21.15.77:0/1353279222 wait complete. 2024-03-06T07:29:13.102 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:07.901583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:13.102 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:13.178 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:14.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:14.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.214+0000 7fa365efb640 1 Processor -- start 2024-03-06T07:29:14.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.214+0000 7fa365efb640 1 -- start start 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.214+0000 7fa365efb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa3600d22b0 0x7fa3600d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.214+0000 7fa365efb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3600b41f0 0x7fa3600b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.214+0000 7fa365efb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3600b3430 con 0x7fa36015dfe0 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.214+0000 7fa365efb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3600d2c50 con 0x7fa3600d22b0 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.214+0000 7fa365efb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600b31e0 con 0x7fa3600b41f0 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa356ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa3600d22b0 0x7fa3600d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa35f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa36015dfe0 0x7fa3600cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:60206/0 (socket says 172.21.15.77:60206) 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa35f7fe640 1 -- 172.21.15.77:0/1708856950 learned_addr learned my addr 172.21.15.77:0/1708856950 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa35ffff640 1 --2- 172.21.15.77:0/1708856950 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3600b41f0 0x7fa3600b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa356ffd640 1 -- 172.21.15.77:0/1708856950 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3600b41f0 msgr2=0x7fa3600b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa356ffd640 1 --2- 172.21.15.77:0/1708856950 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3600b41f0 0x7fa3600b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa356ffd640 1 -- 172.21.15.77:0/1708856950 >> v1:172.21.15.77:6789/0 conn(0x7fa36015dfe0 legacy=0x7fa3600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa356ffd640 1 -- 172.21.15.77:0/1708856950 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3600b3710 con 0x7fa3600d22b0 2024-03-06T07:29:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.215+0000 7fa35effd640 1 -- 172.21.15.77:0/1708856950 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014326155 0 0) 0x7fa3600b3430 con 0x7fa36015dfe0 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa356ffd640 1 --2- 172.21.15.77:0/1708856950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600d22b0 0x7fa3600d2710 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fa3500097a0 tx=0x7fa35002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79991d4f0043aacf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa35effd640 1 -- 172.21.15.77:0/1708856950 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350035020 con 0x7fa3600d22b0 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa35effd640 1 -- 172.21.15.77:0/1708856950 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa350031830 con 0x7fa3600d22b0 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa35effd640 1 -- 172.21.15.77:0/1708856950 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350031c00 con 0x7fa3600d22b0 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa365efb640 1 -- 172.21.15.77:0/1708856950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600d22b0 msgr2=0x7fa3600d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa365efb640 1 --2- 172.21.15.77:0/1708856950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600d22b0 0x7fa3600d2710 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fa3500097a0 tx=0x7fa35002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa365efb640 1 -- 172.21.15.77:0/1708856950 shutdown_connections 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa365efb640 1 --2- 172.21.15.77:0/1708856950 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa3600b41f0 0x7fa3600b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa365efb640 1 --2- 172.21.15.77:0/1708856950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600d22b0 0x7fa3600d2710 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa365efb640 1 -- 172.21.15.77:0/1708856950 >> 172.21.15.77:0/1708856950 conn(0x7fa3600d3cd0 msgr2=0x7fa3600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa365efb640 1 -- 172.21.15.77:0/1708856950 shutdown_connections 2024-03-06T07:29:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.216+0000 7fa365efb640 1 -- 172.21.15.77:0/1708856950 wait complete. 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.218+0000 7fa365efb640 1 Processor -- start 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.218+0000 7fa365efb640 1 -- start start 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.218+0000 7fa365efb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600b41f0 0x7fa3600cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa365efb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3600d22b0 0x7fa3600b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa365efb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa36015dfe0 0x7fa3600b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa365efb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600cfa00 con 0x7fa3600b41f0 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa365efb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600b3d60 con 0x7fa3600d22b0 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa365efb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600d1c70 con 0x7fa36015dfe0 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600b41f0 0x7fa3600cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600b41f0 0x7fa3600cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47646/0 (socket says 172.21.15.77:47646) 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 -- 172.21.15.77:0/2100911695 learned_addr learned my addr 172.21.15.77:0/2100911695 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35ffff640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa36015dfe0 0x7fa3600b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa356ffd640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3600d22b0 0x7fa3600b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 -- 172.21.15.77:0/2100911695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa36015dfe0 msgr2=0x7fa3600b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa36015dfe0 0x7fa3600b7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 -- 172.21.15.77:0/2100911695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3600d22b0 msgr2=0x7fa3600b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3600d22b0 0x7fa3600b78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 -- 172.21.15.77:0/2100911695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36015cf10 con 0x7fa3600b41f0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35f7fe640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600b41f0 0x7fa3600cd870 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fa34800c460 tx=0x7fa34800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35cff9640 1 -- 172.21.15.77:0/2100911695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348018070 con 0x7fa3600b41f0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35cff9640 1 -- 172.21.15.77:0/2100911695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa348007500 con 0x7fa3600b41f0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36009ddd0 con 0x7fa3600b41f0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa35cff9640 1 -- 172.21.15.77:0/2100911695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348004a20 con 0x7fa3600b41f0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.219+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3600cb8d0 con 0x7fa3600b41f0 2024-03-06T07:29:14.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.220+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3600cbad0 con 0x7fa3600b41f0 2024-03-06T07:29:14.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.221+0000 7fa35cff9640 1 -- 172.21.15.77:0/2100911695 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa34801b070 con 0x7fa3600b41f0 2024-03-06T07:29:14.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.221+0000 7fa35cff9640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34006b4d0 0x7fa34006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.221+0000 7fa35cff9640 1 -- 172.21.15.77:0/2100911695 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa3480893d0 con 0x7fa3600b41f0 2024-03-06T07:29:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.221+0000 7fa356ffd640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34006b4d0 0x7fa34006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.222+0000 7fa356ffd640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34006b4d0 0x7fa34006d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa350036000 tx=0x7fa35003d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.226+0000 7fa35cff9640 1 -- 172.21.15.77:0/2100911695 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3600cbad0 con 0x7fa3600b41f0 2024-03-06T07:29:14.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.433+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3600cb270 con 0x7fa3600b41f0 2024-03-06T07:29:14.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.434+0000 7fa35cff9640 1 -- 172.21.15.77:0/2100911695 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3418 (secure 0 0 0) 0x7fa36009ddd0 con 0x7fa3600b41f0 2024-03-06T07:29:14.436 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:07.901583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:14.436 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 88 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34006b4d0 msgr2=0x7fa34006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34006b4d0 0x7fa34006d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa350036000 tx=0x7fa35003d040 comp rx=0 tx=0).stop 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600b41f0 msgr2=0x7fa3600cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600b41f0 0x7fa3600cd870 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fa34800c460 tx=0x7fa34800c920 comp rx=0 tx=0).stop 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 shutdown_connections 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa36015dfe0 0x7fa3600b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa3600d22b0 0x7fa3600b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa34006b4d0 0x7fa34006d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 --2- 172.21.15.77:0/2100911695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600b41f0 0x7fa3600cd870 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 >> 172.21.15.77:0/2100911695 conn(0x7fa3600d3cd0 msgr2=0x7fa3600b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 shutdown_connections 2024-03-06T07:29:14.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:14.436+0000 7fa365efb640 1 -- 172.21.15.77:0/2100911695 wait complete. 2024-03-06T07:29:14.650 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:07.901583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:14.650 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:15.181 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:15.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.761+0000 7f77ec913640 1 Processor -- start 2024-03-06T07:29:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.761+0000 7f77ec913640 1 -- start start 2024-03-06T07:29:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77ec913640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f77e80b1080 0x7f77e81674e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77ec913640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e815c270 con 0x7f77e80d0700 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77ec913640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e80d0e90 con 0x7f77e80d0320 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77ec913640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e80b04e0 con 0x7f77e80b1080 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77dffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f77e80d0700 0x7f77e80b0970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:60216/0 (socket says 172.21.15.77:60216) 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77dffff640 1 -- 172.21.15.77:0/4294053238 learned_addr learned my addr 172.21.15.77:0/4294053238 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77e6d9d640 1 --2- 172.21.15.77:0/4294053238 >> v2:172.21.15.158:3300/0 conn(0x7f77e80b1080 0x7f77e81674e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77e5d9b640 1 -- 172.21.15.77:0/4294053238 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576249129 0 0) 0x7f77e815c270 con 0x7f77e80d0700 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77e5d9b640 1 -- 172.21.15.77:0/4294053238 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d4003520 con 0x7f77e80d0700 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77e5d9b640 1 -- 172.21.15.77:0/4294053238 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114886029 0 0) 0x7f77e80d0e90 con 0x7f77e80d0320 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77e5d9b640 1 -- 172.21.15.77:0/4294053238 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e815c270 con 0x7f77e80d0320 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77e6d9d640 1 -- 172.21.15.77:0/4294053238 >> v1:172.21.15.158:6789/0 conn(0x7f77e80d0320 legacy=0x7f77e80b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77e6d9d640 1 -- 172.21.15.77:0/4294053238 >> v1:172.21.15.77:6789/0 conn(0x7f77e80d0700 legacy=0x7f77e80b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.762+0000 7f77e6d9d640 1 -- 172.21.15.77:0/4294053238 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e80b5ae0 con 0x7f77e80b1080 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.763+0000 7f77e6d9d640 1 --2- 172.21.15.77:0/4294053238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80b1080 0x7f77e81674e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009d90 tx=0x7f77e002bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=574bf7649b8f50b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.763+0000 7f77e5d9b640 1 -- 172.21.15.77:0/4294053238 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0039070 con 0x7f77e80b1080 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.763+0000 7f77e5d9b640 1 -- 172.21.15.77:0/4294053238 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e002cd80 con 0x7f77e80b1080 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.763+0000 7f77e5d9b640 1 -- 172.21.15.77:0/4294053238 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0034660 con 0x7f77e80b1080 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.764+0000 7f77ec913640 1 -- 172.21.15.77:0/4294053238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80b1080 msgr2=0x7f77e81674e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.764+0000 7f77ec913640 1 --2- 172.21.15.77:0/4294053238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80b1080 0x7f77e81674e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009d90 tx=0x7f77e002bfd0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.764+0000 7f77ec913640 1 -- 172.21.15.77:0/4294053238 shutdown_connections 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.764+0000 7f77ec913640 1 --2- 172.21.15.77:0/4294053238 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80b1080 0x7f77e81674e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.764+0000 7f77ec913640 1 -- 172.21.15.77:0/4294053238 >> 172.21.15.77:0/4294053238 conn(0x7f77e80724d0 msgr2=0x7f77e80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.764+0000 7f77ec913640 1 -- 172.21.15.77:0/4294053238 shutdown_connections 2024-03-06T07:29:15.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.764+0000 7f77ec913640 1 -- 172.21.15.77:0/4294053238 wait complete. 2024-03-06T07:29:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.765+0000 7f77ec913640 1 Processor -- start 2024-03-06T07:29:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.765+0000 7f77ec913640 1 -- start start 2024-03-06T07:29:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77ec913640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f77e80b1080 0x7f77e814c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77ec913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77e80d0320 0x7f77e8158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77ec913640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80d0700 0x7f77e809b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77ec913640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80b60a0 con 0x7f77e80d0320 2024-03-06T07:29:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77ec913640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80b0170 con 0x7f77e80d0700 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77ec913640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e80ca790 con 0x7f77e80b1080 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77dffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77e80d0320 0x7f77e8158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77dffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77e80d0320 0x7f77e8158f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47648/0 (socket says 172.21.15.77:47648) 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77dffff640 1 -- 172.21.15.77:0/46699307 learned_addr learned my addr 172.21.15.77:0/46699307 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77e659c640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f77e80b1080 0x7f77e814c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.766+0000 7f77e6d9d640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80d0700 0x7f77e809b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77dffff640 1 -- 172.21.15.77:0/46699307 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f77e80b1080 msgr2=0x7f77e814c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77dffff640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f77e80b1080 0x7f77e814c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77dffff640 1 -- 172.21.15.77:0/46699307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80d0700 msgr2=0x7f77e809b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77dffff640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80d0700 0x7f77e809b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77dffff640 1 -- 172.21.15.77:0/46699307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e8159470 con 0x7f77e80d0320 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77dffff640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77e80d0320 0x7f77e8158f30 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f77d000c4d0 tx=0x7f77d000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77deffd640 1 -- 172.21.15.77:0/46699307 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0017070 con 0x7f77e80d0320 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77deffd640 1 -- 172.21.15.77:0/46699307 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d0005ce0 con 0x7f77e80d0320 2024-03-06T07:29:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e809a700 con 0x7f77e80d0320 2024-03-06T07:29:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77deffd640 1 -- 172.21.15.77:0/46699307 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0007270 con 0x7f77e80d0320 2024-03-06T07:29:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.767+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e809acc0 con 0x7f77e80d0320 2024-03-06T07:29:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.768+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e804f330 con 0x7f77e80d0320 2024-03-06T07:29:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.769+0000 7f77deffd640 1 -- 172.21.15.77:0/46699307 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77d001a070 con 0x7f77e80d0320 2024-03-06T07:29:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.769+0000 7f77deffd640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f77c806b4d0 0x7f77c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.769+0000 7f77e659c640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f77c806b4d0 0x7f77c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.769+0000 7f77deffd640 1 -- 172.21.15.77:0/46699307 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f77d00894e0 con 0x7f77e80d0320 2024-03-06T07:29:15.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.769+0000 7f77e659c640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f77c806b4d0 0x7f77c806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f77d8007fd0 tx=0x7f77d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.773+0000 7f77deffd640 1 -- 172.21.15.77:0/46699307 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77d0057cd0 con 0x7f77e80d0320 2024-03-06T07:29:15.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.981+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77e80ca550 con 0x7f77e80d0320 2024-03-06T07:29:15.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.981+0000 7f77deffd640 1 -- 172.21.15.77:0/46699307 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3418 (secure 0 0 0) 0x7f77d005bd70 con 0x7f77e80d0320 2024-03-06T07:29:15.983 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:15.984 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:07.901583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:15.984 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 88 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f77c806b4d0 msgr2=0x7f77c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f77c806b4d0 0x7f77c806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f77d8007fd0 tx=0x7f77d8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77e80d0320 msgr2=0x7f77e8158f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77e80d0320 0x7f77e8158f30 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f77d000c4d0 tx=0x7f77d000c990 comp rx=0 tx=0).stop 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 shutdown_connections 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f77c806b4d0 0x7f77c806d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77e80d0320 0x7f77e8158f30 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f77e80d0700 0x7f77e809b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 --2- 172.21.15.77:0/46699307 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f77e80b1080 0x7f77e814c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.983+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 >> 172.21.15.77:0/46699307 conn(0x7f77e80724d0 msgr2=0x7f77e806e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.984+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 shutdown_connections 2024-03-06T07:29:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:15.984+0000 7f77ec913640 1 -- 172.21.15.77:0/46699307 wait complete. 2024-03-06T07:29:16.198 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:07.901583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:16.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:17.183 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:17.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:17.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.309+0000 7f63ac1e9640 1 Processor -- start 2024-03-06T07:29:17.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.309+0000 7f63ac1e9640 1 -- start start 2024-03-06T07:29:17.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63ac1e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f63a40b4f70 0x7f63a40b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63ac1e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63a4066d90 0x7f63a40ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63ac1e9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a40b5890 con 0x7f63a406c6d0 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63ac1e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a415e9f0 con 0x7f63a4066d90 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63ac1e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a40cda10 con 0x7f63a40b4f70 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63aa786640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63a4066d90 0x7f63a40ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63aa786640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63a4066d90 0x7f63a40ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47664/0 (socket says 172.21.15.77:47664) 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63aa786640 1 -- 172.21.15.77:0/1320137020 learned_addr learned my addr 172.21.15.77:0/1320137020 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63a9f85640 1 --2- 172.21.15.77:0/1320137020 >> v2:172.21.15.158:3300/0 conn(0x7f63a40b4f70 0x7f63a40b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63aa786640 1 -- 172.21.15.77:0/1320137020 >> v2:172.21.15.158:3300/0 conn(0x7f63a40b4f70 msgr2=0x7f63a40b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63aa786640 1 --2- 172.21.15.77:0/1320137020 >> v2:172.21.15.158:3300/0 conn(0x7f63a40b4f70 0x7f63a40b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63aa786640 1 -- 172.21.15.77:0/1320137020 >> v1:172.21.15.158:6789/0 conn(0x7f63a406c6d0 legacy=0x7f63a415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.310+0000 7f63aa786640 1 -- 172.21.15.77:0/1320137020 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40cd450 con 0x7f63a4066d90 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.311+0000 7f63aa786640 1 --2- 172.21.15.77:0/1320137020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a4066d90 0x7f63a40ccb50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f63a406cac0 tx=0x7f639c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30112dc6af64837b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.311+0000 7f63a8f83640 1 -- 172.21.15.77:0/1320137020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c02cb20 con 0x7f63a4066d90 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.311+0000 7f63a8f83640 1 -- 172.21.15.77:0/1320137020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c02ccc0 con 0x7f63a4066d90 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.311+0000 7f63a8f83640 1 -- 172.21.15.77:0/1320137020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c034610 con 0x7f63a4066d90 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.312+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1320137020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a4066d90 msgr2=0x7f63a40ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.312+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1320137020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a4066d90 0x7f63a40ccb50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f63a406cac0 tx=0x7f639c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.312+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1320137020 shutdown_connections 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.312+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1320137020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a4066d90 0x7f63a40ccb50 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.312+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1320137020 >> v2:172.21.15.158:3300/0 conn(0x7f63a40b4f70 0x7f63a40b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.312+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1320137020 >> 172.21.15.77:0/1320137020 conn(0x7f63a40724d0 msgr2=0x7f63a40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.312+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1320137020 shutdown_connections 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.312+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1320137020 wait complete. 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.313+0000 7f63ac1e9640 1 Processor -- start 2024-03-06T07:29:17.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.313+0000 7f63ac1e9640 1 -- start start 2024-03-06T07:29:17.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63ac1e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a4066d90 0x7f63a40cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63ac1e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a40b4f70 0x7f63a414b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63ac1e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63a41551f0 0x7f63a40c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63ac1e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40cd090 con 0x7f63a40b4f70 2024-03-06T07:29:17.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63ac1e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a415e680 con 0x7f63a4066d90 2024-03-06T07:29:17.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63ac1e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40cd210 con 0x7f63a41551f0 2024-03-06T07:29:17.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63a9784640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a40b4f70 0x7f63a414b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63a9784640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a40b4f70 0x7f63a414b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47678/0 (socket says 172.21.15.77:47678) 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63a9784640 1 -- 172.21.15.77:0/1777000592 learned_addr learned my addr 172.21.15.77:0/1777000592 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63a9f85640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a4066d90 0x7f63a40cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.314+0000 7f63aa786640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63a41551f0 0x7f63a40c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f63a9784640 1 -- 172.21.15.77:0/1777000592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63a41551f0 msgr2=0x7f63a40c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f63a9784640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63a41551f0 0x7f63a40c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f63a9784640 1 -- 172.21.15.77:0/1777000592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a4066d90 msgr2=0x7f63a40cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f63a9784640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a4066d90 0x7f63a40cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f63a9784640 1 -- 172.21.15.77:0/1777000592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40cb7f0 con 0x7f63a40b4f70 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f63a9784640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a40b4f70 0x7f63a414b720 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f639400c430 tx=0x7f639400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f639a7fc640 1 -- 172.21.15.77:0/1777000592 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63940052d0 con 0x7f63a40b4f70 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f639a7fc640 1 -- 172.21.15.77:0/1777000592 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6394005490 con 0x7f63a40b4f70 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a40cbad0 con 0x7f63a40b4f70 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f639a7fc640 1 -- 172.21.15.77:0/1777000592 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394007260 con 0x7f63a40b4f70 2024-03-06T07:29:17.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.315+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a40c98f0 con 0x7f63a40b4f70 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.316+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a404f330 con 0x7f63a40b4f70 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.316+0000 7f639a7fc640 1 -- 172.21.15.77:0/1777000592 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6394007400 con 0x7f63a40b4f70 2024-03-06T07:29:17.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.317+0000 7f639a7fc640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f638806b500 0x7f638806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.317+0000 7f639a7fc640 1 -- 172.21.15.77:0/1777000592 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f63940895b0 con 0x7f63a40b4f70 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.317+0000 7f63a9f85640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f638806b500 0x7f638806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.317+0000 7f63a9f85640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f638806b500 0x7f638806d9c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f63a4155d90 tx=0x7f63a0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.321+0000 7f639a7fc640 1 -- 172.21.15.77:0/1777000592 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6394057e30 con 0x7f63a40b4f70 2024-03-06T07:29:17.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.530+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63a40cbca0 con 0x7f63a40b4f70 2024-03-06T07:29:17.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.531+0000 7f639a7fc640 1 -- 172.21.15.77:0/1777000592 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3418 (secure 0 0 0) 0x7f639405bed0 con 0x7f63a40b4f70 2024-03-06T07:29:17.533 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:17.533 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:07.901583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:17.533 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 88 2024-03-06T07:29:17.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f638806b500 msgr2=0x7f638806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f638806b500 0x7f638806d9c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f63a4155d90 tx=0x7f63a0009190 comp rx=0 tx=0).stop 2024-03-06T07:29:17.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a40b4f70 msgr2=0x7f63a414b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a40b4f70 0x7f63a414b720 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f639400c430 tx=0x7f639400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 shutdown_connections 2024-03-06T07:29:17.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f638806b500 0x7f638806d9c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63a40b4f70 0x7f63a414b720 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63a41551f0 0x7f63a40c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 --2- 172.21.15.77:0/1777000592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63a4066d90 0x7f63a40cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 >> 172.21.15.77:0/1777000592 conn(0x7f63a40724d0 msgr2=0x7f63a40728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 shutdown_connections 2024-03-06T07:29:17.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:17.533+0000 7f63ac1e9640 1 -- 172.21.15.77:0/1777000592 wait complete. 2024-03-06T07:29:17.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:07.901583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:17.747 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:18.748 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:18.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.858+0000 7fdf59a4e640 1 Processor -- start 2024-03-06T07:29:18.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.858+0000 7fdf59a4e640 1 -- start start 2024-03-06T07:29:18.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf59a4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540d22b0 0x7fdf540d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf59a4e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540b3430 con 0x7fdf5415dfe0 2024-03-06T07:29:18.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf59a4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540b3710 con 0x7fdf540b41f0 2024-03-06T07:29:18.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf59a4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d2c50 con 0x7fdf540d22b0 2024-03-06T07:29:18.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf52ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdf5415dfe0 0x7fdf540cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43424/0 (socket says 172.21.15.77:43424) 2024-03-06T07:29:18.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf52ffd640 1 -- 172.21.15.77:0/1635039361 learned_addr learned my addr 172.21.15.77:0/1635039361 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:18.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf527fc640 1 --2- 172.21.15.77:0/1635039361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540d22b0 0x7fdf540d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf51ffb640 1 -- 172.21.15.77:0/1635039361 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068256107 0 0) 0x7fdf540b3430 con 0x7fdf5415dfe0 2024-03-06T07:29:18.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.859+0000 7fdf51ffb640 1 -- 172.21.15.77:0/1635039361 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3c003520 con 0x7fdf5415dfe0 2024-03-06T07:29:18.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.860+0000 7fdf51ffb640 1 -- 172.21.15.77:0/1635039361 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071148611 0 0) 0x7fdf540b3710 con 0x7fdf540b41f0 2024-03-06T07:29:18.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.860+0000 7fdf527fc640 1 -- 172.21.15.77:0/1635039361 >> v1:172.21.15.158:6789/0 conn(0x7fdf540b41f0 legacy=0x7fdf540b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.860+0000 7fdf527fc640 1 -- 172.21.15.77:0/1635039361 >> v1:172.21.15.77:6789/0 conn(0x7fdf5415dfe0 legacy=0x7fdf540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.860+0000 7fdf527fc640 1 -- 172.21.15.77:0/1635039361 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540b39f0 con 0x7fdf540d22b0 2024-03-06T07:29:18.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.860+0000 7fdf527fc640 1 --2- 172.21.15.77:0/1635039361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540d22b0 0x7fdf540d2710 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdf480097d0 tx=0x7fdf4802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51c6367d112f119c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf51ffb640 1 -- 172.21.15.77:0/1635039361 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48035020 con 0x7fdf540d22b0 2024-03-06T07:29:18.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf51ffb640 1 -- 172.21.15.77:0/1635039361 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf48031890 con 0x7fdf540d22b0 2024-03-06T07:29:18.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf51ffb640 1 -- 172.21.15.77:0/1635039361 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48031c40 con 0x7fdf540d22b0 2024-03-06T07:29:18.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1635039361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540d22b0 msgr2=0x7fdf540d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf59a4e640 1 --2- 172.21.15.77:0/1635039361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540d22b0 0x7fdf540d2710 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdf480097d0 tx=0x7fdf4802b930 comp rx=0 tx=0).stop 2024-03-06T07:29:18.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1635039361 shutdown_connections 2024-03-06T07:29:18.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf59a4e640 1 --2- 172.21.15.77:0/1635039361 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540d22b0 0x7fdf540d2710 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1635039361 >> 172.21.15.77:0/1635039361 conn(0x7fdf540d3cd0 msgr2=0x7fdf540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1635039361 shutdown_connections 2024-03-06T07:29:18.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.861+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1635039361 wait complete. 2024-03-06T07:29:18.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.862+0000 7fdf59a4e640 1 Processor -- start 2024-03-06T07:29:18.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.863+0000 7fdf59a4e640 1 -- start start 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.863+0000 7fdf59a4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540b41f0 0x7fdf541f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.863+0000 7fdf59a4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf540d22b0 0x7fdf540b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.863+0000 7fdf59a4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5415dfe0 0x7fdf540b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.863+0000 7fdf59a4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540b3d60 con 0x7fdf540d22b0 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.863+0000 7fdf59a4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540b31e0 con 0x7fdf5415dfe0 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.863+0000 7fdf59a4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d1c70 con 0x7fdf540b41f0 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf52ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540b41f0 0x7fdf541f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf527fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf540d22b0 0x7fdf540b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf52ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540b41f0 0x7fdf541f2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:55364/0 (socket says 172.21.15.77:55364) 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf527fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf540d22b0 0x7fdf540b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40376/0 (socket says 172.21.15.77:40376) 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf52ffd640 1 -- 172.21.15.77:0/1270218829 learned_addr learned my addr 172.21.15.77:0/1270218829 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf537fe640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5415dfe0 0x7fdf540b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf527fc640 1 -- 172.21.15.77:0/1270218829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540b41f0 msgr2=0x7fdf541f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf527fc640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540b41f0 0x7fdf541f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf527fc640 1 -- 172.21.15.77:0/1270218829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5415dfe0 msgr2=0x7fdf540b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf527fc640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5415dfe0 0x7fdf540b92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf527fc640 1 -- 172.21.15.77:0/1270218829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540b7de0 con 0x7fdf540d22b0 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf527fc640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf540d22b0 0x7fdf540b8d70 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fdf48033000 tx=0x7fdf480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf3bfff640 1 -- 172.21.15.77:0/1270218829 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48035040 con 0x7fdf540d22b0 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf3bfff640 1 -- 172.21.15.77:0/1270218829 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf48030d20 con 0x7fdf540d22b0 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5415f4b0 con 0x7fdf540d22b0 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf3bfff640 1 -- 172.21.15.77:0/1270218829 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48006e10 con 0x7fdf540d22b0 2024-03-06T07:29:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.864+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf540b80c0 con 0x7fdf540d22b0 2024-03-06T07:29:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.865+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5404f330 con 0x7fdf540d22b0 2024-03-06T07:29:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.866+0000 7fdf3bfff640 1 -- 172.21.15.77:0/1270218829 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf48038020 con 0x7fdf540d22b0 2024-03-06T07:29:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.866+0000 7fdf3bfff640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.866+0000 7fdf3bfff640 1 -- 172.21.15.77:0/1270218829 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf480ac860 con 0x7fdf540d22b0 2024-03-06T07:29:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.866+0000 7fdf52ffd640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.867+0000 7fdf52ffd640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf4006b4d0 0x7fdf4006d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdf44005880 tx=0x7fdf44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:18.871+0000 7fdf3bfff640 1 -- 172.21.15.77:0/1270218829 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf4802d040 con 0x7fdf540d22b0 2024-03-06T07:29:19.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.079+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf540c9fc0 con 0x7fdf540d22b0 2024-03-06T07:29:19.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.079+0000 7fdf3bfff640 1 -- 172.21.15.77:0/1270218829 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7fdf4807afb0 con 0x7fdf540d22b0 2024-03-06T07:29:19.081 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:19.081 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:19.082 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:19.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf4006b4d0 msgr2=0x7fdf4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf4006b4d0 0x7fdf4006d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdf44005880 tx=0x7fdf44009040 comp rx=0 tx=0).stop 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf540d22b0 msgr2=0x7fdf540b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf540d22b0 0x7fdf540b8d70 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fdf48033000 tx=0x7fdf480046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 shutdown_connections 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf540d22b0 0x7fdf540b8d70 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf5415dfe0 0x7fdf540b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 --2- 172.21.15.77:0/1270218829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf540b41f0 0x7fdf541f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 >> 172.21.15.77:0/1270218829 conn(0x7fdf540d3cd0 msgr2=0x7fdf540d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 shutdown_connections 2024-03-06T07:29:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:19.081+0000 7fdf59a4e640 1 -- 172.21.15.77:0/1270218829 wait complete. 2024-03-06T07:29:19.185 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:19.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:19.295 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:20.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:20.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.406+0000 7f0f1f664640 1 Processor -- start 2024-03-06T07:29:20.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.406+0000 7f0f1f664640 1 -- start start 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1f664640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0f181424f0 0x7f0f180cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1f664640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0f180cd7a0 0x7f0f1815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1f664640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f180cc980 con 0x7f0f1806d330 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1f664640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f180cd220 con 0x7f0f181424f0 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1f664640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f180ccc60 con 0x7f0f180cd7a0 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1d400640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0f181424f0 0x7f0f180cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1d400640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0f181424f0 0x7f0f180cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40380/0 (socket says 172.21.15.77:40380) 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1d400640 1 -- 172.21.15.77:0/1079840993 learned_addr learned my addr 172.21.15.77:0/1079840993 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.407+0000 7f0f1cbff640 1 --2- 172.21.15.77:0/1079840993 >> v2:172.21.15.158:3300/0 conn(0x7f0f180cd7a0 0x7f0f1815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.408+0000 7f0f1d400640 1 -- 172.21.15.77:0/1079840993 >> v2:172.21.15.158:3300/0 conn(0x7f0f180cd7a0 msgr2=0x7f0f1815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.408+0000 7f0f1d400640 1 --2- 172.21.15.77:0/1079840993 >> v2:172.21.15.158:3300/0 conn(0x7f0f180cd7a0 0x7f0f1815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.408+0000 7f0f1d400640 1 -- 172.21.15.77:0/1079840993 >> v1:172.21.15.158:6789/0 conn(0x7f0f1806d330 legacy=0x7f0f18071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.408+0000 7f0f1d400640 1 -- 172.21.15.77:0/1079840993 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180ccf40 con 0x7f0f181424f0 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.408+0000 7f0f1d400640 1 --2- 172.21.15.77:0/1079840993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f181424f0 0x7f0f180cc2f0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f0f180b2a00 tx=0x7f0f1402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cb65a1cdcfa9f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f0ffff640 1 -- 172.21.15.77:0/1079840993 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1402ca80 con 0x7f0f181424f0 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f0ffff640 1 -- 172.21.15.77:0/1079840993 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1402cc20 con 0x7f0f181424f0 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f0ffff640 1 -- 172.21.15.77:0/1079840993 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f140343f0 con 0x7f0f181424f0 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f1f664640 1 -- 172.21.15.77:0/1079840993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f181424f0 msgr2=0x7f0f180cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f1f664640 1 --2- 172.21.15.77:0/1079840993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f181424f0 0x7f0f180cc2f0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f0f180b2a00 tx=0x7f0f1402bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f1f664640 1 -- 172.21.15.77:0/1079840993 shutdown_connections 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f1f664640 1 --2- 172.21.15.77:0/1079840993 >> v2:172.21.15.158:3300/0 conn(0x7f0f180cd7a0 0x7f0f1815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f1f664640 1 --2- 172.21.15.77:0/1079840993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f181424f0 0x7f0f180cc2f0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f1f664640 1 -- 172.21.15.77:0/1079840993 >> 172.21.15.77:0/1079840993 conn(0x7f0f180cf8a0 msgr2=0x7f0f180cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f1f664640 1 -- 172.21.15.77:0/1079840993 shutdown_connections 2024-03-06T07:29:20.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.409+0000 7f0f1f664640 1 -- 172.21.15.77:0/1079840993 wait complete. 2024-03-06T07:29:20.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1f664640 1 Processor -- start 2024-03-06T07:29:20.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1f664640 1 -- start start 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1f664640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1806d330 0x7f0f181f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1f664640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f180cd7a0 0x7f0f180cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1f664640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f181424f0 0x7f0f180b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1f664640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1806d920 con 0x7f0f180cd7a0 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1f664640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1806d7a0 con 0x7f0f181424f0 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1f664640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1806daa0 con 0x7f0f1806d330 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.411+0000 7f0f1cbff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f180cd7a0 0x7f0f180cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1cbff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f180cd7a0 0x7f0f180cbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40396/0 (socket says 172.21.15.77:40396) 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1cbff640 1 -- 172.21.15.77:0/361536727 learned_addr learned my addr 172.21.15.77:0/361536727 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1dc01640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f181424f0 0x7f0f180b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1d400640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1806d330 0x7f0f181f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1cbff640 1 -- 172.21.15.77:0/361536727 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1806d330 msgr2=0x7f0f181f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1cbff640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1806d330 0x7f0f181f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1cbff640 1 -- 172.21.15.77:0/361536727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f181424f0 msgr2=0x7f0f180b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1cbff640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f181424f0 0x7f0f180b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1cbff640 1 -- 172.21.15.77:0/361536727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f181f5ea0 con 0x7f0f180cd7a0 2024-03-06T07:29:20.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1cbff640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f180cd7a0 0x7f0f180cbed0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f0f0800e9b0 tx=0x7f0f0800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f0dffb640 1 -- 172.21.15.77:0/361536727 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08017070 con 0x7f0f180cd7a0 2024-03-06T07:29:20.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f0dffb640 1 -- 172.21.15.77:0/361536727 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0800c9a0 con 0x7f0f180cd7a0 2024-03-06T07:29:20.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f181f6180 con 0x7f0f180cd7a0 2024-03-06T07:29:20.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f0dffb640 1 -- 172.21.15.77:0/361536727 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0800cd90 con 0x7f0f180cd7a0 2024-03-06T07:29:20.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.412+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f181f6740 con 0x7f0f180cd7a0 2024-03-06T07:29:20.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.413+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f180682b0 con 0x7f0f180cd7a0 2024-03-06T07:29:20.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.414+0000 7f0f0dffb640 1 -- 172.21.15.77:0/361536727 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f0801a070 con 0x7f0f180cd7a0 2024-03-06T07:29:20.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.414+0000 7f0f0dffb640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.414+0000 7f0f0dffb640 1 -- 172.21.15.77:0/361536727 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f0808a570 con 0x7f0f180cd7a0 2024-03-06T07:29:20.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.414+0000 7f0f1d400640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.415+0000 7f0f1d400640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0efc06b4d0 0x7f0efc06d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0f14035000 tx=0x7f0f14030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.419+0000 7f0f0dffb640 1 -- 172.21.15.77:0/361536727 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f180682b0 con 0x7f0f180cd7a0 2024-03-06T07:29:20.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.628+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f1806a090 con 0x7f0f180cd7a0 2024-03-06T07:29:20.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.628+0000 7f0f0dffb640 1 -- 172.21.15.77:0/361536727 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f0f1806a090 con 0x7f0f180cd7a0 2024-03-06T07:29:20.630 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:20.630 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:20.631 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:20.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0efc06b4d0 msgr2=0x7f0efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0efc06b4d0 0x7f0efc06d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0f14035000 tx=0x7f0f14030000 comp rx=0 tx=0).stop 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f180cd7a0 msgr2=0x7f0f180cbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f180cd7a0 0x7f0f180cbed0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f0f0800e9b0 tx=0x7f0f0800ee70 comp rx=0 tx=0).stop 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 shutdown_connections 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f180cd7a0 0x7f0f180cbed0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f181424f0 0x7f0f180b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 --2- 172.21.15.77:0/361536727 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1806d330 0x7f0f181f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 >> 172.21.15.77:0/361536727 conn(0x7f0f180cf8a0 msgr2=0x7f0f1815adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 shutdown_connections 2024-03-06T07:29:20.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:20.630+0000 7f0f1f664640 1 -- 172.21.15.77:0/361536727 wait complete. 2024-03-06T07:29:20.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:20.844 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:21.187 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:21.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:21.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.952+0000 7f786759e640 1 Processor -- start 2024-03-06T07:29:21.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.952+0000 7f786759e640 1 -- start start 2024-03-06T07:29:21.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f786759e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f786815e990 0x7f786815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f786759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7868162250 0x7f78680aed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f786759e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f786806bfc0 0x7f786806c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f786759e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78681626b0 con 0x7f7868162250 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f786759e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78680af620 con 0x7f786815e990 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f786759e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78680af900 con 0x7f786806bfc0 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7865d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7868162250 0x7f78680aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7865d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7868162250 0x7f78680aed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40406/0 (socket says 172.21.15.77:40406) 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7865d9b640 1 -- 172.21.15.77:0/2357634658 learned_addr learned my addr 172.21.15.77:0/2357634658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f786659c640 1 --2- 172.21.15.77:0/2357634658 >> v2:172.21.15.158:3300/0 conn(0x7f786815e990 0x7f786815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7866d9d640 1 --2- 172.21.15.77:0/2357634658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f786806bfc0 0x7f786806c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7865d9b640 1 -- 172.21.15.77:0/2357634658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f786806bfc0 msgr2=0x7f786806c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7865d9b640 1 --2- 172.21.15.77:0/2357634658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f786806bfc0 0x7f786806c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7865d9b640 1 -- 172.21.15.77:0/2357634658 >> v2:172.21.15.158:3300/0 conn(0x7f786815e990 msgr2=0x7f786815ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7865d9b640 1 --2- 172.21.15.77:0/2357634658 >> v2:172.21.15.158:3300/0 conn(0x7f786815e990 0x7f786815ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7865d9b640 1 -- 172.21.15.77:0/2357634658 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868161770 con 0x7f7868162250 2024-03-06T07:29:21.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f786659c640 1 --2- 172.21.15.77:0/2357634658 >> v2:172.21.15.158:3300/0 conn(0x7f786815e990 0x7f786815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:21.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.953+0000 7f7866d9d640 1 --2- 172.21.15.77:0/2357634658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f786806bfc0 0x7f786806c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:21.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.954+0000 7f7865d9b640 1 --2- 172.21.15.77:0/2357634658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868162250 0x7f78680aed40 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f78680b2e10 tx=0x7f785002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad424c7c82509066 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.954+0000 7f786559a640 1 -- 172.21.15.77:0/2357634658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850035020 con 0x7f7868162250 2024-03-06T07:29:21.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.954+0000 7f786559a640 1 -- 172.21.15.77:0/2357634658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7850031870 con 0x7f7868162250 2024-03-06T07:29:21.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.954+0000 7f786559a640 1 -- 172.21.15.77:0/2357634658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850031c40 con 0x7f7868162250 2024-03-06T07:29:21.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 -- 172.21.15.77:0/2357634658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868162250 msgr2=0x7f78680aed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 --2- 172.21.15.77:0/2357634658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868162250 0x7f78680aed40 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f78680b2e10 tx=0x7f785002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:21.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 -- 172.21.15.77:0/2357634658 shutdown_connections 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 --2- 172.21.15.77:0/2357634658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f786806bfc0 0x7f786806c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 --2- 172.21.15.77:0/2357634658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868162250 0x7f78680aed40 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 --2- 172.21.15.77:0/2357634658 >> v2:172.21.15.158:3300/0 conn(0x7f786815e990 0x7f786815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 -- 172.21.15.77:0/2357634658 >> 172.21.15.77:0/2357634658 conn(0x7f786806ddf0 msgr2=0x7f786806e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 -- 172.21.15.77:0/2357634658 shutdown_connections 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.955+0000 7f786759e640 1 -- 172.21.15.77:0/2357634658 wait complete. 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.956+0000 7f786759e640 1 Processor -- start 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.956+0000 7f786759e640 1 -- start start 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815e990 0x7f7868154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786759e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7868162250 0x7f78681f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786759e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7868160650 0x7f78681539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786759e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680af400 con 0x7f786815e990 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786759e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680af280 con 0x7f7868160650 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786759e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7868071650 con 0x7f7868162250 2024-03-06T07:29:21.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815e990 0x7f7868154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815e990 0x7f7868154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40416/0 (socket says 172.21.15.77:40416) 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786659c640 1 -- 172.21.15.77:0/2808278297 learned_addr learned my addr 172.21.15.77:0/2808278297 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f7866d9d640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7868160650 0x7f78681539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f7865d9b640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7868162250 0x7f78681f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786659c640 1 -- 172.21.15.77:0/2808278297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7868162250 msgr2=0x7f78681f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786659c640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7868162250 0x7f78681f7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786659c640 1 -- 172.21.15.77:0/2808278297 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7868160650 msgr2=0x7f78681539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786659c640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7868160650 0x7f78681539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.957+0000 7f786659c640 1 -- 172.21.15.77:0/2808278297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868161210 con 0x7f786815e990 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.958+0000 7f786659c640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815e990 0x7f7868154bf0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7858009fc0 tx=0x7f785800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.958+0000 7f785effd640 1 -- 172.21.15.77:0/2808278297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785801a070 con 0x7f786815e990 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.958+0000 7f785effd640 1 -- 172.21.15.77:0/2808278297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7858003bb0 con 0x7f786815e990 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.958+0000 7f785effd640 1 -- 172.21.15.77:0/2808278297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7858004030 con 0x7f786815e990 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.958+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78681f7e30 con 0x7f786815e990 2024-03-06T07:29:21.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.958+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78681f83f0 con 0x7f786815e990 2024-03-06T07:29:21.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.959+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78680aea80 con 0x7f786815e990 2024-03-06T07:29:21.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.959+0000 7f785effd640 1 -- 172.21.15.77:0/2808278297 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f785801d070 con 0x7f786815e990 2024-03-06T07:29:21.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.960+0000 7f785effd640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f784806b500 0x7f784806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.960+0000 7f785effd640 1 -- 172.21.15.77:0/2808278297 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f785808a310 con 0x7f786815e990 2024-03-06T07:29:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.960+0000 7f7865d9b640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f784806b500 0x7f784806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.960+0000 7f7865d9b640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f784806b500 0x7f784806d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7850036000 tx=0x7f785003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:21.964+0000 7f785effd640 1 -- 172.21.15.77:0/2808278297 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78680aea80 con 0x7f786815e990 2024-03-06T07:29:22.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.172+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7868068f80 con 0x7f786815e990 2024-03-06T07:29:22.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.173+0000 7f785effd640 1 -- 172.21.15.77:0/2808278297 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f7858010070 con 0x7f786815e990 2024-03-06T07:29:22.176 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:22.176 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f784806b500 msgr2=0x7f784806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f784806b500 0x7f784806d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7850036000 tx=0x7f785003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815e990 msgr2=0x7f7868154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815e990 0x7f7868154bf0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7858009fc0 tx=0x7f785800d560 comp rx=0 tx=0).stop 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 shutdown_connections 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7868160650 0x7f78681539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7868162250 0x7f78681f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f784806b500 0x7f784806d9c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 --2- 172.21.15.77:0/2808278297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815e990 0x7f7868154bf0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 >> 172.21.15.77:0/2808278297 conn(0x7f786806bfc0 msgr2=0x7f78680b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 shutdown_connections 2024-03-06T07:29:22.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:22.175+0000 7f786759e640 1 -- 172.21.15.77:0/2808278297 wait complete. 2024-03-06T07:29:22.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:22.389 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:23.189 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:23.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:23.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.499+0000 7fe3cd5b6640 1 Processor -- start 2024-03-06T07:29:23.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.499+0000 7fe3cd5b6640 1 -- start start 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3cd5b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80d22b0 0x7fe3c80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3cd5b6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3c80b41f0 0x7fe3c80b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3cd5b6640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c80b3430 con 0x7fe3c815dfe0 2024-03-06T07:29:23.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3cd5b6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c80d2c50 con 0x7fe3c80b41f0 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3cd5b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80b31e0 con 0x7fe3c80d22b0 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3c77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3c80b41f0 0x7fe3c80b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3c6ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe3c815dfe0 0x7fe3c80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43430/0 (socket says 172.21.15.77:43430) 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3c6ffd640 1 -- 172.21.15.77:0/2045112721 learned_addr learned my addr 172.21.15.77:0/2045112721 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3bffff640 1 --2- 172.21.15.77:0/2045112721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80d22b0 0x7fe3c80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3c77fe640 1 -- 172.21.15.77:0/2045112721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80d22b0 msgr2=0x7fe3c80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3c77fe640 1 --2- 172.21.15.77:0/2045112721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80d22b0 0x7fe3c80d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3c77fe640 1 -- 172.21.15.77:0/2045112721 >> v1:172.21.15.77:6789/0 conn(0x7fe3c815dfe0 legacy=0x7fe3c80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3c67fc640 1 -- 172.21.15.77:0/2045112721 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762718087 0 0) 0x7fe3c80b3430 con 0x7fe3c815dfe0 2024-03-06T07:29:23.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.500+0000 7fe3c77fe640 1 -- 172.21.15.77:0/2045112721 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c80b3710 con 0x7fe3c80b41f0 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.501+0000 7fe3c77fe640 1 --2- 172.21.15.77:0/2045112721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80b41f0 0x7fe3c80b4650 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009d90 tx=0x7fe3c002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2e751fe96102ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.501+0000 7fe3c67fc640 1 -- 172.21.15.77:0/2045112721 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0039070 con 0x7fe3c80b41f0 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.501+0000 7fe3c67fc640 1 -- 172.21.15.77:0/2045112721 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c002cc50 con 0x7fe3c80b41f0 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.501+0000 7fe3c67fc640 1 -- 172.21.15.77:0/2045112721 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0034440 con 0x7fe3c80b41f0 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.502+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/2045112721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80b41f0 msgr2=0x7fe3c80b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.502+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/2045112721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80b41f0 0x7fe3c80b4650 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009d90 tx=0x7fe3c002bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.502+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/2045112721 shutdown_connections 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.502+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/2045112721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80b41f0 0x7fe3c80b4650 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.502+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/2045112721 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80d22b0 0x7fe3c80d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.502+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/2045112721 >> 172.21.15.77:0/2045112721 conn(0x7fe3c80d3cd0 msgr2=0x7fe3c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.502+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/2045112721 shutdown_connections 2024-03-06T07:29:23.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.502+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/2045112721 wait complete. 2024-03-06T07:29:23.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.503+0000 7fe3cd5b6640 1 Processor -- start 2024-03-06T07:29:23.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.503+0000 7fe3cd5b6640 1 -- start start 2024-03-06T07:29:23.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3cd5b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80b41f0 0x7fe3c81f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3cd5b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80d22b0 0x7fe3c81522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3cd5b6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3c815dfe0 0x7fe3c81f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3cd5b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80cfa00 con 0x7fe3c80d22b0 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3cd5b6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80b3d60 con 0x7fe3c815dfe0 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3cd5b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80d1c70 con 0x7fe3c80b41f0 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80d22b0 0x7fe3c81522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80d22b0 0x7fe3c81522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40422/0 (socket says 172.21.15.77:40422) 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 -- 172.21.15.77:0/4075691150 learned_addr learned my addr 172.21.15.77:0/4075691150 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3c6ffd640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80b41f0 0x7fe3c81f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3c77fe640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3c815dfe0 0x7fe3c81f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 -- 172.21.15.77:0/4075691150 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80b41f0 msgr2=0x7fe3c81f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80b41f0 0x7fe3c81f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 -- 172.21.15.77:0/4075691150 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3c815dfe0 msgr2=0x7fe3c81f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3c815dfe0 0x7fe3c81f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 -- 172.21.15.77:0/4075691150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c80b7a20 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bffff640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80d22b0 0x7fe3c81522b0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9e0 tx=0x7fe3b000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.504+0000 7fe3bf7fe640 1 -- 172.21.15.77:0/4075691150 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0017070 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.505+0000 7fe3bf7fe640 1 -- 172.21.15.77:0/4075691150 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b000c970 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.505+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c80cbf70 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.505+0000 7fe3bf7fe640 1 -- 172.21.15.77:0/4075691150 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000cd60 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.505+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c80cc530 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.506+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c804f330 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.506+0000 7fe3bf7fe640 1 -- 172.21.15.77:0/4075691150 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3b001a070 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.506+0000 7fe3bf7fe640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe3a806b4d0 0x7fe3a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.507+0000 7fe3c6ffd640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe3a806b4d0 0x7fe3a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.507+0000 7fe3bf7fe640 1 -- 172.21.15.77:0/4075691150 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe3b008a500 con 0x7fe3c80d22b0 2024-03-06T07:29:23.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.507+0000 7fe3c6ffd640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe3a806b4d0 0x7fe3a806d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8007fd0 tx=0x7fe3b8007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.511+0000 7fe3bf7fe640 1 -- 172.21.15.77:0/4075691150 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3b0058cf0 con 0x7fe3c80d22b0 2024-03-06T07:29:23.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.718+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3c80ce910 con 0x7fe3c80d22b0 2024-03-06T07:29:23.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.718+0000 7fe3bf7fe640 1 -- 172.21.15.77:0/4075691150 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7fe3b005cd90 con 0x7fe3c80d22b0 2024-03-06T07:29:23.720 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:23.720 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:23.721 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:23.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe3a806b4d0 msgr2=0x7fe3a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe3a806b4d0 0x7fe3a806d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8007fd0 tx=0x7fe3b8007de0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80d22b0 msgr2=0x7fe3c81522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80d22b0 0x7fe3c81522b0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9e0 tx=0x7fe3b000eea0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 shutdown_connections 2024-03-06T07:29:23.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe3a806b4d0 0x7fe3a806d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c80d22b0 0x7fe3c81522b0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe3c815dfe0 0x7fe3c81f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 --2- 172.21.15.77:0/4075691150 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe3c80b41f0 0x7fe3c81f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 >> 172.21.15.77:0/4075691150 conn(0x7fe3c80d3cd0 msgr2=0x7fe3c806cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 shutdown_connections 2024-03-06T07:29:23.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:23.720+0000 7fe3cd5b6640 1 -- 172.21.15.77:0/4075691150 wait complete. 2024-03-06T07:29:23.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:23.934 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:24.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:25.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.045+0000 7effad7d7640 1 Processor -- start 2024-03-06T07:29:25.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.045+0000 7effad7d7640 1 -- start start 2024-03-06T07:29:25.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effad7d7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80d22b0 0x7effa80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effad7d7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa80b3430 con 0x7effa815dfe0 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effad7d7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa80b3710 con 0x7effa80b41f0 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effad7d7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa80d2c50 con 0x7effa80d22b0 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa6ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7effa815dfe0 0x7effa80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43434/0 (socket says 172.21.15.77:43434) 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa6ffd640 1 -- 172.21.15.77:0/2502091661 learned_addr learned my addr 172.21.15.77:0/2502091661 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa67fc640 1 --2- 172.21.15.77:0/2502091661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80d22b0 0x7effa80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa5ffb640 1 -- 172.21.15.77:0/2502091661 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523164293 0 0) 0x7effa80b3430 con 0x7effa815dfe0 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa5ffb640 1 -- 172.21.15.77:0/2502091661 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff90003520 con 0x7effa815dfe0 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa5ffb640 1 -- 172.21.15.77:0/2502091661 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736928796 0 0) 0x7effa80b3710 con 0x7effa80b41f0 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa5ffb640 1 -- 172.21.15.77:0/2502091661 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa80b3430 con 0x7effa80b41f0 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa67fc640 1 -- 172.21.15.77:0/2502091661 >> v1:172.21.15.158:6789/0 conn(0x7effa80b41f0 legacy=0x7effa80b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa67fc640 1 -- 172.21.15.77:0/2502091661 >> v1:172.21.15.77:6789/0 conn(0x7effa815dfe0 legacy=0x7effa80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.046+0000 7effa67fc640 1 -- 172.21.15.77:0/2502091661 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa80b39f0 con 0x7effa80d22b0 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.047+0000 7effa67fc640 1 --2- 172.21.15.77:0/2502091661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80d22b0 0x7effa80d2710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0097d0 tx=0x7eff9c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be901c6c54368d86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.047+0000 7effa5ffb640 1 -- 172.21.15.77:0/2502091661 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c035020 con 0x7effa80d22b0 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.047+0000 7effa5ffb640 1 -- 172.21.15.77:0/2502091661 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c031890 con 0x7effa80d22b0 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.047+0000 7effa5ffb640 1 -- 172.21.15.77:0/2502091661 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c031c40 con 0x7effa80d22b0 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.048+0000 7effad7d7640 1 -- 172.21.15.77:0/2502091661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80d22b0 msgr2=0x7effa80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.048+0000 7effad7d7640 1 --2- 172.21.15.77:0/2502091661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80d22b0 0x7effa80d2710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0097d0 tx=0x7eff9c02b930 comp rx=0 tx=0).stop 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.048+0000 7effad7d7640 1 -- 172.21.15.77:0/2502091661 shutdown_connections 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.048+0000 7effad7d7640 1 --2- 172.21.15.77:0/2502091661 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80d22b0 0x7effa80d2710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.048+0000 7effad7d7640 1 -- 172.21.15.77:0/2502091661 >> 172.21.15.77:0/2502091661 conn(0x7effa80d3cd0 msgr2=0x7effa80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.048+0000 7effad7d7640 1 -- 172.21.15.77:0/2502091661 shutdown_connections 2024-03-06T07:29:25.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.048+0000 7effad7d7640 1 -- 172.21.15.77:0/2502091661 wait complete. 2024-03-06T07:29:25.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.049+0000 7effad7d7640 1 Processor -- start 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.049+0000 7effad7d7640 1 -- start start 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effad7d7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80b41f0 0x7effa80c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effad7d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80d22b0 0x7effa80b8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effad7d7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa815dfe0 0x7effa80b8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effad7d7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa80b3d60 con 0x7effa80d22b0 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effad7d7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa80b31e0 con 0x7effa815dfe0 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effad7d7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa80d1c70 con 0x7effa80b41f0 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effa67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80d22b0 0x7effa80b8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effa67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80d22b0 0x7effa80b8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40428/0 (socket says 172.21.15.77:40428) 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effa67fc640 1 -- 172.21.15.77:0/868860905 learned_addr learned my addr 172.21.15.77:0/868860905 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effa6ffd640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80b41f0 0x7effa80c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.050+0000 7effa77fe640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa815dfe0 0x7effa80b8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7effa67fc640 1 -- 172.21.15.77:0/868860905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80b41f0 msgr2=0x7effa80c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7effa67fc640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80b41f0 0x7effa80c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7effa67fc640 1 -- 172.21.15.77:0/868860905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa815dfe0 msgr2=0x7effa80b8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7effa67fc640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa815dfe0 0x7effa80b8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7effa67fc640 1 -- 172.21.15.77:0/868860905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa80b9410 con 0x7effa80d22b0 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7effa67fc640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80d22b0 0x7effa80b8890 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0097d0 tx=0x7eff9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7eff8f7fe640 1 -- 172.21.15.77:0/868860905 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c035040 con 0x7effa80d22b0 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7eff8f7fe640 1 -- 172.21.15.77:0/868860905 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c003820 con 0x7effa80d22b0 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7eff8f7fe640 1 -- 172.21.15.77:0/868860905 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c003bf0 con 0x7effa80d22b0 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa80b96f0 con 0x7effa80d22b0 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.051+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa81575f0 con 0x7effa80d22b0 2024-03-06T07:29:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.052+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa804f330 con 0x7effa80d22b0 2024-03-06T07:29:25.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.052+0000 7eff8f7fe640 1 -- 172.21.15.77:0/868860905 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff9c038020 con 0x7effa80d22b0 2024-03-06T07:29:25.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.053+0000 7eff8f7fe640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff9406b4d0 0x7eff9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.053+0000 7eff8f7fe640 1 -- 172.21.15.77:0/868860905 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7eff9c0ac730 con 0x7effa80d22b0 2024-03-06T07:29:25.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.053+0000 7effa6ffd640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff9406b4d0 0x7eff9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.053+0000 7effa6ffd640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff9406b4d0 0x7eff9406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7eff98000f60 tx=0x7eff98007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.057+0000 7eff8f7fe640 1 -- 172.21.15.77:0/868860905 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff9c02d040 con 0x7effa80d22b0 2024-03-06T07:29:25.191 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:25.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.267+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effa80b0c10 con 0x7effa80d22b0 2024-03-06T07:29:25.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.267+0000 7eff8f7fe640 1 -- 172.21.15.77:0/868860905 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7eff9c07af30 con 0x7effa80d22b0 2024-03-06T07:29:25.269 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:25.269 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:25.270 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:25.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff9406b4d0 msgr2=0x7eff9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff9406b4d0 0x7eff9406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7eff98000f60 tx=0x7eff98007d70 comp rx=0 tx=0).stop 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80d22b0 msgr2=0x7effa80b8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80d22b0 0x7effa80b8890 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0097d0 tx=0x7eff9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 shutdown_connections 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff9406b4d0 0x7eff9406d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80d22b0 0x7effa80b8890 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7effa815dfe0 0x7effa80b8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 --2- 172.21.15.77:0/868860905 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7effa80b41f0 0x7effa80c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 >> 172.21.15.77:0/868860905 conn(0x7effa80d3cd0 msgr2=0x7effa80d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 shutdown_connections 2024-03-06T07:29:25.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:25.269+0000 7effad7d7640 1 -- 172.21.15.77:0/868860905 wait complete. 2024-03-06T07:29:25.483 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:25.483 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:26.485 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:26.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.594+0000 7f2e4f8ed640 1 Processor -- start 2024-03-06T07:29:26.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.594+0000 7f2e4f8ed640 1 -- start start 2024-03-06T07:29:26.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.595+0000 7f2e4f8ed640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e480d4920 0x7f2e480cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.595+0000 7f2e4f8ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481625a0 0x7f2e4815dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.595+0000 7f2e4f8ed640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e480ceac0 con 0x7f2e481621c0 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.595+0000 7f2e4f8ed640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e480cf690 con 0x7f2e480d4920 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.595+0000 7f2e4f8ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e480d2dc0 con 0x7f2e481625a0 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4de8a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481625a0 0x7f2e4815dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4ce88640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e480d4920 0x7f2e480cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4d689640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2e481621c0 0x7f2e480d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:48016/0 (socket says 172.21.15.77:48016) 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4d689640 1 -- 172.21.15.77:0/430104713 learned_addr learned my addr 172.21.15.77:0/430104713 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4de8a640 1 -- 172.21.15.77:0/430104713 >> v2:172.21.15.158:3300/0 conn(0x7f2e480d4920 msgr2=0x7f2e480cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4de8a640 1 --2- 172.21.15.77:0/430104713 >> v2:172.21.15.158:3300/0 conn(0x7f2e480d4920 0x7f2e480cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4de8a640 1 -- 172.21.15.77:0/430104713 >> v1:172.21.15.158:6789/0 conn(0x7f2e481621c0 legacy=0x7f2e480d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4de8a640 1 -- 172.21.15.77:0/430104713 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e480ced60 con 0x7f2e481625a0 2024-03-06T07:29:26.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.596+0000 7f2e4ce88640 1 --2- 172.21.15.77:0/430104713 >> v2:172.21.15.158:3300/0 conn(0x7f2e480d4920 0x7f2e480cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e4de8a640 1 --2- 172.21.15.77:0/430104713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481625a0 0x7f2e4815dd40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2e400097a0 tx=0x7f2e4002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ba7c8ae9a325014 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e47fff640 1 -- 172.21.15.77:0/430104713 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40035020 con 0x7f2e481625a0 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e47fff640 1 -- 172.21.15.77:0/430104713 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e40031830 con 0x7f2e481625a0 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e47fff640 1 -- 172.21.15.77:0/430104713 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40031c00 con 0x7f2e481625a0 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/430104713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481625a0 msgr2=0x7f2e4815dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/430104713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481625a0 0x7f2e4815dd40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2e400097a0 tx=0x7f2e4002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/430104713 shutdown_connections 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/430104713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481625a0 0x7f2e4815dd40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/430104713 >> v2:172.21.15.158:3300/0 conn(0x7f2e480d4920 0x7f2e480cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.597+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/430104713 >> 172.21.15.77:0/430104713 conn(0x7f2e480d3d10 msgr2=0x7f2e480d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.598+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/430104713 shutdown_connections 2024-03-06T07:29:26.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.598+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/430104713 wait complete. 2024-03-06T07:29:26.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.599+0000 7f2e4f8ed640 1 Processor -- start 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.599+0000 7f2e4f8ed640 1 -- start start 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.599+0000 7f2e4f8ed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e480d4920 0x7f2e481f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.599+0000 7f2e4f8ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481621c0 0x7f2e481591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.599+0000 7f2e4f8ed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e481625a0 0x7f2e480b8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.599+0000 7f2e4f8ed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4815e280 con 0x7f2e481625a0 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4f8ed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e480cf320 con 0x7f2e480d4920 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4f8ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4815e400 con 0x7f2e481621c0 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e480d4920 0x7f2e481f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4de8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e481625a0 0x7f2e480b8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4ce88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481621c0 0x7f2e481591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e480d4920 0x7f2e481f51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:39198/0 (socket says 172.21.15.77:39198) 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 -- 172.21.15.77:0/3602252275 learned_addr learned my addr 172.21.15.77:0/3602252275 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 -- 172.21.15.77:0/3602252275 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481621c0 msgr2=0x7f2e481591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481621c0 0x7f2e481591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 -- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e481625a0 msgr2=0x7f2e480b8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e481625a0 0x7f2e480b8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 -- 172.21.15.77:0/3602252275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e481f5720 con 0x7f2e480d4920 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4de8a640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e481625a0 0x7f2e480b8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4d689640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e480d4920 0x7f2e481f51e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800cb10 tx=0x7f2e3800a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e45ffb640 1 -- 172.21.15.77:0/3602252275 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38017070 con 0x7f2e480d4920 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e45ffb640 1 -- 172.21.15.77:0/3602252275 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3800ac70 con 0x7f2e480d4920 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e480b6800 con 0x7f2e480d4920 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.600+0000 7f2e45ffb640 1 -- 172.21.15.77:0/3602252275 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38012620 con 0x7f2e480d4920 2024-03-06T07:29:26.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.601+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e480b6dc0 con 0x7f2e480d4920 2024-03-06T07:29:26.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.602+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4804f330 con 0x7f2e480d4920 2024-03-06T07:29:26.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.602+0000 7f2e45ffb640 1 -- 172.21.15.77:0/3602252275 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e3801a070 con 0x7f2e480d4920 2024-03-06T07:29:26.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.602+0000 7f2e45ffb640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e3406b500 0x7f2e3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.602+0000 7f2e45ffb640 1 -- 172.21.15.77:0/3602252275 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2e38089ec0 con 0x7f2e480d4920 2024-03-06T07:29:26.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.602+0000 7f2e4ce88640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e3406b500 0x7f2e3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.603+0000 7f2e4ce88640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e3406b500 0x7f2e3406d9c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009560 tx=0x7f2e3c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.607+0000 7f2e45ffb640 1 -- 172.21.15.77:0/3602252275 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e481f5720 con 0x7f2e480d4920 2024-03-06T07:29:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.815+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e480ce0b0 con 0x7f2e480d4920 2024-03-06T07:29:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.815+0000 7f2e45ffb640 1 -- 172.21.15.77:0/3602252275 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f2e38058720 con 0x7f2e480d4920 2024-03-06T07:29:26.818 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:26.818 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:26.818 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.817+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e3406b500 msgr2=0x7f2e3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e3406b500 0x7f2e3406d9c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009560 tx=0x7f2e3c009190 comp rx=0 tx=0).stop 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e480d4920 msgr2=0x7f2e481f51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e480d4920 0x7f2e481f51e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800cb10 tx=0x7f2e3800a060 comp rx=0 tx=0).stop 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 shutdown_connections 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e3406b500 0x7f2e3406d9c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e481625a0 0x7f2e480b8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e481621c0 0x7f2e481591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 --2- 172.21.15.77:0/3602252275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e480d4920 0x7f2e481f51e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 >> 172.21.15.77:0/3602252275 conn(0x7f2e480d3d10 msgr2=0x7f2e480d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 shutdown_connections 2024-03-06T07:29:26.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:26.818+0000 7f2e4f8ed640 1 -- 172.21.15.77:0/3602252275 wait complete. 2024-03-06T07:29:27.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:27.032 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:27.193 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:28.033 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:28.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.143+0000 7f1cc19ac640 1 Processor -- start 2024-03-06T07:29:28.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.143+0000 7f1cc19ac640 1 -- start start 2024-03-06T07:29:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cc19ac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1cbc15bf30 0x7f1cbc15ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cc19ac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cbc15f580 con 0x7f1cbc13dc10 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cc19ac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cbc15b450 con 0x7f1cbc0b3060 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cc19ac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cbc0cf3f0 con 0x7f1cbc15bf30 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cbaffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1cbc13dc10 0x7f1cbc0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43448/0 (socket says 172.21.15.77:43448) 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cbaffd640 1 -- 172.21.15.77:0/95502311 learned_addr learned my addr 172.21.15.77:0/95502311 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cbb7fe640 1 --2- 172.21.15.77:0/95502311 >> v2:172.21.15.158:3300/0 conn(0x7f1cbc15bf30 0x7f1cbc15ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cb9ffb640 1 -- 172.21.15.77:0/95502311 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074915749 0 0) 0x7f1cbc15f580 con 0x7f1cbc13dc10 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.144+0000 7f1cb9ffb640 1 -- 172.21.15.77:0/95502311 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca4003520 con 0x7f1cbc13dc10 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.145+0000 7f1cb9ffb640 1 -- 172.21.15.77:0/95502311 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493005493 0 0) 0x7f1cbc15b450 con 0x7f1cbc0b3060 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.145+0000 7f1cb9ffb640 1 -- 172.21.15.77:0/95502311 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cbc15f580 con 0x7f1cbc0b3060 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.145+0000 7f1cbb7fe640 1 -- 172.21.15.77:0/95502311 >> v1:172.21.15.158:6789/0 conn(0x7f1cbc0b3060 legacy=0x7f1cbc15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.145+0000 7f1cb9ffb640 1 -- 172.21.15.77:0/95502311 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1549637731 0 0) 0x7f1ca4003520 con 0x7f1cbc13dc10 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.145+0000 7f1cbb7fe640 1 -- 172.21.15.77:0/95502311 >> v1:172.21.15.77:6789/0 conn(0x7f1cbc13dc10 legacy=0x7f1cbc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.145+0000 7f1cbb7fe640 1 -- 172.21.15.77:0/95502311 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc15b730 con 0x7f1cbc15bf30 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.145+0000 7f1cbb7fe640 1 --2- 172.21.15.77:0/95502311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15ef60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40097a0 tx=0x7f1cb402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=391c49e7fbffd2f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cb9ffb640 1 -- 172.21.15.77:0/95502311 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4037070 con 0x7f1cbc15bf30 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cb9ffb640 1 -- 172.21.15.77:0/95502311 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb4030510 con 0x7f1cbc15bf30 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cb9ffb640 1 -- 172.21.15.77:0/95502311 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac0019f0 con 0x7f1cbc15bf30 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cc19ac640 1 -- 172.21.15.77:0/95502311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc15bf30 msgr2=0x7f1cbc15ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cc19ac640 1 --2- 172.21.15.77:0/95502311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15ef60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40097a0 tx=0x7f1cb402b790 comp rx=0 tx=0).stop 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cc19ac640 1 -- 172.21.15.77:0/95502311 shutdown_connections 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cc19ac640 1 --2- 172.21.15.77:0/95502311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15ef60 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cc19ac640 1 -- 172.21.15.77:0/95502311 >> 172.21.15.77:0/95502311 conn(0x7f1cbc0cf8a0 msgr2=0x7f1cbc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cc19ac640 1 -- 172.21.15.77:0/95502311 shutdown_connections 2024-03-06T07:29:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.146+0000 7f1cc19ac640 1 -- 172.21.15.77:0/95502311 wait complete. 2024-03-06T07:29:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.147+0000 7f1cc19ac640 1 Processor -- start 2024-03-06T07:29:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.148+0000 7f1cc19ac640 1 -- start start 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.148+0000 7f1cc19ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1cbc0b3060 0x7f1cbc06d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.148+0000 7f1cc19ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc13dc10 0x7f1cbc138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.148+0000 7f1cc19ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.148+0000 7f1cc19ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc0718e0 con 0x7f1cbc15bf30 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.148+0000 7f1cc19ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc15b210 con 0x7f1cbc13dc10 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.148+0000 7f1cc19ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc071a60 con 0x7f1cbc0b3060 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.148+0000 7f1cbb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40454/0 (socket says 172.21.15.77:40454) 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbb7fe640 1 -- 172.21.15.77:0/3641055647 learned_addr learned my addr 172.21.15.77:0/3641055647 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbaffd640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1cbc0b3060 0x7f1cbc06d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cba7fc640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc13dc10 0x7f1cbc138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbb7fe640 1 -- 172.21.15.77:0/3641055647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1cbc0b3060 msgr2=0x7f1cbc06d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbb7fe640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1cbc0b3060 0x7f1cbc06d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbb7fe640 1 -- 172.21.15.77:0/3641055647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc13dc10 msgr2=0x7f1cbc138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbb7fe640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc13dc10 0x7f1cbc138ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbb7fe640 1 -- 172.21.15.77:0/3641055647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc15e720 con 0x7f1cbc15bf30 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cbb7fe640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15c340 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1cb402b7d0 tx=0x7f1cb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1ca3fff640 1 -- 172.21.15.77:0/3641055647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4037040 con 0x7f1cbc15bf30 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1ca3fff640 1 -- 172.21.15.77:0/3641055647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb4004230 con 0x7f1cbc15bf30 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1ca3fff640 1 -- 172.21.15.77:0/3641055647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4006e10 con 0x7f1cbc15bf30 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cbc15ea00 con 0x7f1cbc15bf30 2024-03-06T07:29:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.149+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cbc15efc0 con 0x7f1cbc15bf30 2024-03-06T07:29:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.150+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cbc0af830 con 0x7f1cbc15bf30 2024-03-06T07:29:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.151+0000 7f1ca3fff640 1 -- 172.21.15.77:0/3641055647 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cb403a070 con 0x7f1cbc15bf30 2024-03-06T07:29:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.151+0000 7f1ca3fff640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ca806b4d0 0x7f1ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.151+0000 7f1cbaffd640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ca806b4d0 0x7f1ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.151+0000 7f1ca3fff640 1 -- 172.21.15.77:0/3641055647 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1cb40abb00 con 0x7f1cbc15bf30 2024-03-06T07:29:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.152+0000 7f1cbaffd640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ca806b4d0 0x7f1ca806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1cac001350 tx=0x7f1cac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.156+0000 7f1ca3fff640 1 -- 172.21.15.77:0/3641055647 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1cbc0af830 con 0x7f1cbc15bf30 2024-03-06T07:29:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.364+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cbc04f330 con 0x7f1cbc15bf30 2024-03-06T07:29:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.365+0000 7f1ca3fff640 1 -- 172.21.15.77:0/3641055647 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f1cb4041020 con 0x7f1cbc15bf30 2024-03-06T07:29:28.367 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:28.367 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ca806b4d0 msgr2=0x7f1ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ca806b4d0 0x7f1ca806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1cac001350 tx=0x7f1cac009040 comp rx=0 tx=0).stop 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc15bf30 msgr2=0x7f1cbc15c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15c340 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1cb402b7d0 tx=0x7f1cb40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 shutdown_connections 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1ca806b4d0 0x7f1ca806d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc15bf30 0x7f1cbc15c340 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1cbc13dc10 0x7f1cbc138ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 --2- 172.21.15.77:0/3641055647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1cbc0b3060 0x7f1cbc06d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 >> 172.21.15.77:0/3641055647 conn(0x7f1cbc0cf8a0 msgr2=0x7f1cbc06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 shutdown_connections 2024-03-06T07:29:28.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:28.367+0000 7f1cc19ac640 1 -- 172.21.15.77:0/3641055647 wait complete. 2024-03-06T07:29:28.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:28.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:29.195 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:29.582 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.693+0000 7f1b75735640 1 Processor -- start 2024-03-06T07:29:29.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.693+0000 7f1b75735640 1 -- start start 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.693+0000 7f1b75735640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b7015dfe0 0x7f1b700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.693+0000 7f1b75735640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1b700d29c0 0x7f1b700b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.693+0000 7f1b75735640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b700b3430 con 0x7f1b700cf830 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.693+0000 7f1b75735640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b700b31e0 con 0x7f1b7015dfe0 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.693+0000 7f1b75735640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b700b3d60 con 0x7f1b700d29c0 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b7015dfe0 0x7f1b700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6e7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1b700cf830 0x7f1b700d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:37110/0 (socket says 172.21.15.77:37110) 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6e7fc640 1 -- 172.21.15.77:0/1836325943 learned_addr learned my addr 172.21.15.77:0/1836325943 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6f7fe640 1 --2- 172.21.15.77:0/1836325943 >> v2:172.21.15.158:3300/0 conn(0x7f1b700d29c0 0x7f1b700b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6effd640 1 -- 172.21.15.77:0/1836325943 >> v2:172.21.15.158:3300/0 conn(0x7f1b700d29c0 msgr2=0x7f1b700b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6effd640 1 --2- 172.21.15.77:0/1836325943 >> v2:172.21.15.158:3300/0 conn(0x7f1b700d29c0 0x7f1b700b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6effd640 1 -- 172.21.15.77:0/1836325943 >> v1:172.21.15.77:6789/0 conn(0x7f1b700cf830 legacy=0x7f1b700d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6effd640 1 -- 172.21.15.77:0/1836325943 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700b3710 con 0x7f1b7015dfe0 2024-03-06T07:29:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.694+0000 7f1b6dffb640 1 -- 172.21.15.77:0/1836325943 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945097947 0 0) 0x7f1b700b3430 con 0x7f1b700cf830 2024-03-06T07:29:29.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b6effd640 1 --2- 172.21.15.77:0/1836325943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b7015dfe0 0x7f1b700cf2f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f1b700b3cd0 tx=0x7f1b6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2500d7b2244927 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b6dffb640 1 -- 172.21.15.77:0/1836325943 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60035020 con 0x7f1b7015dfe0 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b6dffb640 1 -- 172.21.15.77:0/1836325943 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b60031830 con 0x7f1b7015dfe0 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b6dffb640 1 -- 172.21.15.77:0/1836325943 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60031c00 con 0x7f1b7015dfe0 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b75735640 1 -- 172.21.15.77:0/1836325943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b7015dfe0 msgr2=0x7f1b700cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b75735640 1 --2- 172.21.15.77:0/1836325943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b7015dfe0 0x7f1b700cf2f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f1b700b3cd0 tx=0x7f1b6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b75735640 1 -- 172.21.15.77:0/1836325943 shutdown_connections 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b75735640 1 --2- 172.21.15.77:0/1836325943 >> v2:172.21.15.158:3300/0 conn(0x7f1b700d29c0 0x7f1b700b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b75735640 1 --2- 172.21.15.77:0/1836325943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b7015dfe0 0x7f1b700cf2f0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b75735640 1 -- 172.21.15.77:0/1836325943 >> 172.21.15.77:0/1836325943 conn(0x7f1b700d3cd0 msgr2=0x7f1b700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b75735640 1 -- 172.21.15.77:0/1836325943 shutdown_connections 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.695+0000 7f1b75735640 1 -- 172.21.15.77:0/1836325943 wait complete. 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.697+0000 7f1b75735640 1 Processor -- start 2024-03-06T07:29:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.697+0000 7f1b75735640 1 -- start start 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.697+0000 7f1b75735640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b700cf830 0x7f1b7015bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.697+0000 7f1b75735640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b700d29c0 0x7f1b701f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.697+0000 7f1b75735640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b7015dfe0 0x7f1b700b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.697+0000 7f1b75735640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700d1df0 con 0x7f1b700cf830 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.697+0000 7f1b75735640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700d1c70 con 0x7f1b700d29c0 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.697+0000 7f1b75735640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7015e3c0 con 0x7f1b7015dfe0 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b700cf830 0x7f1b7015bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b700cf830 0x7f1b7015bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51386/0 (socket says 172.21.15.77:51386) 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 -- 172.21.15.77:0/2656735161 learned_addr learned my addr 172.21.15.77:0/2656735161 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6e7fc640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b700d29c0 0x7f1b701f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6f7fe640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b7015dfe0 0x7f1b700b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 -- 172.21.15.77:0/2656735161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b7015dfe0 msgr2=0x7f1b700b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b7015dfe0 0x7f1b700b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 -- 172.21.15.77:0/2656735161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b700d29c0 msgr2=0x7f1b701f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b700d29c0 0x7f1b701f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 -- 172.21.15.77:0/2656735161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b701f7280 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b6effd640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b700cf830 0x7f1b7015bb70 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f1b60000c00 tx=0x7f1b600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b577fe640 1 -- 172.21.15.77:0/2656735161 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60040070 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b577fe640 1 -- 172.21.15.77:0/2656735161 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6003a850 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b701f7560 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b577fe640 1 -- 172.21.15.77:0/2656735161 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60035040 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.698+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b701f7b20 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.699+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b701f7d60 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.700+0000 7f1b577fe640 1 -- 172.21.15.77:0/2656735161 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b6004b060 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.700+0000 7f1b577fe640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b5c06b500 0x7f1b5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.700+0000 7f1b577fe640 1 -- 172.21.15.77:0/2656735161 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1b600aba20 con 0x7f1b700cf830 2024-03-06T07:29:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.700+0000 7f1b6e7fc640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b5c06b500 0x7f1b5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.701+0000 7f1b6e7fc640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b5c06b500 0x7f1b5c06d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1b700cce70 tx=0x7f1b64007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.705+0000 7f1b577fe640 1 -- 172.21.15.77:0/2656735161 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b701f7d60 con 0x7f1b700cf830 2024-03-06T07:29:29.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.913+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b7004f330 con 0x7f1b700cf830 2024-03-06T07:29:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.913+0000 7f1b577fe640 1 -- 172.21.15.77:0/2656735161 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f1b60031800 con 0x7f1b700cf830 2024-03-06T07:29:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:29.915 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:29.916 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:29.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b5c06b500 msgr2=0x7f1b5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b5c06b500 0x7f1b5c06d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1b700cce70 tx=0x7f1b64007d50 comp rx=0 tx=0).stop 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b700cf830 msgr2=0x7f1b7015bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b700cf830 0x7f1b7015bb70 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f1b60000c00 tx=0x7f1b600046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 shutdown_connections 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b7015dfe0 0x7f1b700b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b700d29c0 0x7f1b701f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b5c06b500 0x7f1b5c06d9c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 --2- 172.21.15.77:0/2656735161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b700cf830 0x7f1b7015bb70 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 >> 172.21.15.77:0/2656735161 conn(0x7f1b700d3cd0 msgr2=0x7f1b700ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 shutdown_connections 2024-03-06T07:29:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:29.915+0000 7f1b75735640 1 -- 172.21.15.77:0/2656735161 wait complete. 2024-03-06T07:29:30.131 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:30.131 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:31.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:31.197 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:31.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.237+0000 7febdc223640 1 Processor -- start 2024-03-06T07:29:31.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.238+0000 7febdc223640 1 -- start start 2024-03-06T07:29:31.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.238+0000 7febdc223640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7febd406b220 0x7febd415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.238+0000 7febdc223640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd415ea30 con 0x7febd40ce920 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.238+0000 7febdc223640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd415ed10 con 0x7febd406b600 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.238+0000 7febdc223640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd4161b70 con 0x7febd406b220 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.239+0000 7febd9fbf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7febd406b220 0x7febd415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.239+0000 7febd97be640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7febd40ce920 0x7febd4071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:37122/0 (socket says 172.21.15.77:37122) 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.239+0000 7febd9fbf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7febd406b220 0x7febd415e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51390/0 (socket says 172.21.15.77:51390) 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.239+0000 7febd97be640 1 -- 172.21.15.77:0/1825747277 learned_addr learned my addr 172.21.15.77:0/1825747277 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.239+0000 7febd9fbf640 1 -- 172.21.15.77:0/1825747277 >> v1:172.21.15.158:6789/0 conn(0x7febd406b600 legacy=0x7febd4162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.239+0000 7febd9fbf640 1 -- 172.21.15.77:0/1825747277 >> v1:172.21.15.77:6789/0 conn(0x7febd40ce920 legacy=0x7febd4071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.239+0000 7febd9fbf640 1 -- 172.21.15.77:0/1825747277 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd415eff0 con 0x7febd406b220 2024-03-06T07:29:31.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.239+0000 7febd8fbd640 1 -- 172.21.15.77:0/1825747277 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195645898 0 0) 0x7febd415ea30 con 0x7febd40ce920 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febd9fbf640 1 --2- 172.21.15.77:0/1825747277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd415e450 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7febd4161cf0 tx=0x7febd002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2321e1567e1b0dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febd8fbd640 1 -- 172.21.15.77:0/1825747277 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0034070 con 0x7febd406b220 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febd8fbd640 1 -- 172.21.15.77:0/1825747277 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd0030610 con 0x7febd406b220 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febd8fbd640 1 -- 172.21.15.77:0/1825747277 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd00309e0 con 0x7febd406b220 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febdc223640 1 -- 172.21.15.77:0/1825747277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 msgr2=0x7febd415e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febdc223640 1 --2- 172.21.15.77:0/1825747277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd415e450 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7febd4161cf0 tx=0x7febd002b960 comp rx=0 tx=0).stop 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febdc223640 1 -- 172.21.15.77:0/1825747277 shutdown_connections 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febdc223640 1 --2- 172.21.15.77:0/1825747277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd415e450 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febdc223640 1 -- 172.21.15.77:0/1825747277 >> 172.21.15.77:0/1825747277 conn(0x7febd406c0c0 msgr2=0x7febd406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febdc223640 1 -- 172.21.15.77:0/1825747277 shutdown_connections 2024-03-06T07:29:31.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.240+0000 7febdc223640 1 -- 172.21.15.77:0/1825747277 wait complete. 2024-03-06T07:29:31.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febdc223640 1 Processor -- start 2024-03-06T07:29:31.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febdc223640 1 -- start start 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febdc223640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd4157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febdc223640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd406b600 0x7febd41465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febdc223640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febd40ce920 0x7febd4154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febdc223640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd4072090 con 0x7febd406b220 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febdc223640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd40ce200 con 0x7febd406b600 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febdc223640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd4160360 con 0x7febd40ce920 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febd9fbf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd4157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febd9fbf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd4157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51400/0 (socket says 172.21.15.77:51400) 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febd9fbf640 1 -- 172.21.15.77:0/467021029 learned_addr learned my addr 172.21.15.77:0/467021029 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febd97be640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd406b600 0x7febd41465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.242+0000 7febda7c0640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febd40ce920 0x7febd4154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febd9fbf640 1 -- 172.21.15.77:0/467021029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febd40ce920 msgr2=0x7febd4154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febd9fbf640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febd40ce920 0x7febd4154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febd9fbf640 1 -- 172.21.15.77:0/467021029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd406b600 msgr2=0x7febd41465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febd9fbf640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd406b600 0x7febd41465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febd9fbf640 1 -- 172.21.15.77:0/467021029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4160640 con 0x7febd406b220 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febd9fbf640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd4157050 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7febd0000c00 tx=0x7febd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febc67fc640 1 -- 172.21.15.77:0/467021029 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0047070 con 0x7febd406b220 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febc67fc640 1 -- 172.21.15.77:0/467021029 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd0004030 con 0x7febd406b220 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd4160920 con 0x7febd406b220 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febc67fc640 1 -- 172.21.15.77:0/467021029 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0034040 con 0x7febd406b220 2024-03-06T07:29:31.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.243+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd4154590 con 0x7febd406b220 2024-03-06T07:29:31.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.244+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd404f330 con 0x7febd406b220 2024-03-06T07:29:31.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.245+0000 7febc67fc640 1 -- 172.21.15.77:0/467021029 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febd004a070 con 0x7febd406b220 2024-03-06T07:29:31.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.245+0000 7febc67fc640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7febb806b4d0 0x7febb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.245+0000 7febc67fc640 1 -- 172.21.15.77:0/467021029 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7febd00abcc0 con 0x7febd406b220 2024-03-06T07:29:31.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.245+0000 7febd97be640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7febb806b4d0 0x7febb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.245+0000 7febd97be640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7febb806b4d0 0x7febb806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7febd41f74d0 tx=0x7febc8007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.249+0000 7febc67fc640 1 -- 172.21.15.77:0/467021029 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febd007a4b0 con 0x7febd406b220 2024-03-06T07:29:31.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.459+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febd40b1c40 con 0x7febd406b220 2024-03-06T07:29:31.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.459+0000 7febc67fc640 1 -- 172.21.15.77:0/467021029 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7febd007e550 con 0x7febd406b220 2024-03-06T07:29:31.461 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:31.461 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:31.462 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:31.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7febb806b4d0 msgr2=0x7febb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7febb806b4d0 0x7febb806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7febd41f74d0 tx=0x7febc8007f00 comp rx=0 tx=0).stop 2024-03-06T07:29:31.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 msgr2=0x7febd4157050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd4157050 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7febd0000c00 tx=0x7febd00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 shutdown_connections 2024-03-06T07:29:31.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7febd40ce920 0x7febd4154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7febd406b600 0x7febd41465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7febb806b4d0 0x7febb806d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 --2- 172.21.15.77:0/467021029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febd406b220 0x7febd4157050 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 >> 172.21.15.77:0/467021029 conn(0x7febd406c0c0 msgr2=0x7febd40b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 shutdown_connections 2024-03-06T07:29:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:31.461+0000 7febdc223640 1 -- 172.21.15.77:0/467021029 wait complete. 2024-03-06T07:29:31.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:31.481 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:31.675 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:31.676 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:32.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.787+0000 7fe6581dd640 1 Processor -- start 2024-03-06T07:29:32.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.788+0000 7fe6581dd640 1 -- start start 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.788+0000 7fe6581dd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe6500d22b0 0x7fe6500d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.788+0000 7fe6581dd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6500b41f0 0x7fe6500b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.788+0000 7fe6581dd640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6500b3430 con 0x7fe65015dfe0 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.788+0000 7fe6581dd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6500d2c50 con 0x7fe6500b41f0 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.788+0000 7fe6581dd640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6500b31e0 con 0x7fe6500d22b0 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.789+0000 7fe65677a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6500b41f0 0x7fe6500b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.789+0000 7fe65677a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6500b41f0 0x7fe6500b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51412/0 (socket says 172.21.15.77:51412) 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.789+0000 7fe65677a640 1 -- 172.21.15.77:0/3379095496 learned_addr learned my addr 172.21.15.77:0/3379095496 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:32.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.789+0000 7fe655778640 1 --2- 172.21.15.77:0/3379095496 >> v2:172.21.15.158:3300/0 conn(0x7fe6500d22b0 0x7fe6500d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.789+0000 7fe65677a640 1 -- 172.21.15.77:0/3379095496 >> v2:172.21.15.158:3300/0 conn(0x7fe6500d22b0 msgr2=0x7fe6500d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.789+0000 7fe65677a640 1 --2- 172.21.15.77:0/3379095496 >> v2:172.21.15.158:3300/0 conn(0x7fe6500d22b0 0x7fe6500d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.789+0000 7fe65677a640 1 -- 172.21.15.77:0/3379095496 >> v1:172.21.15.158:6789/0 conn(0x7fe65015dfe0 legacy=0x7fe6500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.789+0000 7fe65677a640 1 -- 172.21.15.77:0/3379095496 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6500b3710 con 0x7fe6500b41f0 2024-03-06T07:29:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe65677a640 1 --2- 172.21.15.77:0/3379095496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6500b41f0 0x7fe6500b4680 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fe648009e50 tx=0x7fe64802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59f23654f1fa145 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe654f77640 1 -- 172.21.15.77:0/3379095496 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64802cb20 con 0x7fe6500b41f0 2024-03-06T07:29:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe654f77640 1 -- 172.21.15.77:0/3379095496 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64802ccc0 con 0x7fe6500b41f0 2024-03-06T07:29:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe654f77640 1 -- 172.21.15.77:0/3379095496 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648034610 con 0x7fe6500b41f0 2024-03-06T07:29:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe6581dd640 1 -- 172.21.15.77:0/3379095496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6500b41f0 msgr2=0x7fe6500b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe6581dd640 1 --2- 172.21.15.77:0/3379095496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6500b41f0 0x7fe6500b4680 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fe648009e50 tx=0x7fe64802bf80 comp rx=0 tx=0).stop 2024-03-06T07:29:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe6581dd640 1 -- 172.21.15.77:0/3379095496 shutdown_connections 2024-03-06T07:29:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe6581dd640 1 --2- 172.21.15.77:0/3379095496 >> v2:172.21.15.158:3300/0 conn(0x7fe6500d22b0 0x7fe6500d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe6581dd640 1 --2- 172.21.15.77:0/3379095496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6500b41f0 0x7fe6500b4680 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe6581dd640 1 -- 172.21.15.77:0/3379095496 >> 172.21.15.77:0/3379095496 conn(0x7fe6500d3cd0 msgr2=0x7fe6500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe6581dd640 1 -- 172.21.15.77:0/3379095496 shutdown_connections 2024-03-06T07:29:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.790+0000 7fe6581dd640 1 -- 172.21.15.77:0/3379095496 wait complete. 2024-03-06T07:29:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.792+0000 7fe6581dd640 1 Processor -- start 2024-03-06T07:29:32.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.792+0000 7fe6581dd640 1 -- start start 2024-03-06T07:29:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.792+0000 7fe6581dd640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6500b41f0 0x7fe6501f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.792+0000 7fe6581dd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6500d22b0 0x7fe6500cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.792+0000 7fe6581dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65015dfe0 0x7fe650149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.792+0000 7fe6581dd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6500cfa00 con 0x7fe65015dfe0 2024-03-06T07:29:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.792+0000 7fe6581dd640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6500b3d60 con 0x7fe6500b41f0 2024-03-06T07:29:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.792+0000 7fe6581dd640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6500d1c70 con 0x7fe6500d22b0 2024-03-06T07:29:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65015dfe0 0x7fe650149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65015dfe0 0x7fe650149e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51428/0 (socket says 172.21.15.77:51428) 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 -- 172.21.15.77:0/1441406215 learned_addr learned my addr 172.21.15.77:0/1441406215 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe655f79640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6500b41f0 0x7fe6501f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe655778640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6500d22b0 0x7fe6500cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 -- 172.21.15.77:0/1441406215 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6500d22b0 msgr2=0x7fe6500cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6500d22b0 0x7fe6500cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 -- 172.21.15.77:0/1441406215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6500b41f0 msgr2=0x7fe6501f6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6500b41f0 0x7fe6501f6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 -- 172.21.15.77:0/1441406215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6500b70b0 con 0x7fe65015dfe0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe65677a640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65015dfe0 0x7fe650149e70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fe648036040 tx=0x7fe648004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe6427fc640 1 -- 172.21.15.77:0/1441406215 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648043020 con 0x7fe65015dfe0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe6427fc640 1 -- 172.21.15.77:0/1441406215 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64803e450 con 0x7fe65015dfe0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6500b6af0 con 0x7fe65015dfe0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe6427fc640 1 -- 172.21.15.77:0/1441406215 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648009510 con 0x7fe65015dfe0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.793+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6500cc9a0 con 0x7fe65015dfe0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.794+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65004f330 con 0x7fe65015dfe0 2024-03-06T07:29:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.795+0000 7fe6427fc640 1 -- 172.21.15.77:0/1441406215 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe648046020 con 0x7fe65015dfe0 2024-03-06T07:29:32.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.795+0000 7fe6427fc640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe63406b4d0 0x7fe63406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.795+0000 7fe6427fc640 1 -- 172.21.15.77:0/1441406215 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe6480ac8b0 con 0x7fe65015dfe0 2024-03-06T07:29:32.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.795+0000 7fe655f79640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe63406b4d0 0x7fe63406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.795+0000 7fe655f79640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe63406b4d0 0x7fe63406d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0016e0 tx=0x7fe64c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:32.799+0000 7fe6427fc640 1 -- 172.21.15.77:0/1441406215 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe64804d030 con 0x7fe65015dfe0 2024-03-06T07:29:33.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.009+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6500d0860 con 0x7fe65015dfe0 2024-03-06T07:29:33.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.010+0000 7fe6427fc640 1 -- 172.21.15.77:0/1441406215 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7fe6500b70b0 con 0x7fe65015dfe0 2024-03-06T07:29:33.012 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:33.012 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:33.012 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe63406b4d0 msgr2=0x7fe63406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe63406b4d0 0x7fe63406d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0016e0 tx=0x7fe64c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65015dfe0 msgr2=0x7fe650149e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65015dfe0 0x7fe650149e70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fe648036040 tx=0x7fe648004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 shutdown_connections 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe63406b4d0 0x7fe63406d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe65015dfe0 0x7fe650149e70 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6500d22b0 0x7fe6500cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 --2- 172.21.15.77:0/1441406215 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6500b41f0 0x7fe6501f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 >> 172.21.15.77:0/1441406215 conn(0x7fe6500d3cd0 msgr2=0x7fe6500b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 shutdown_connections 2024-03-06T07:29:33.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:33.012+0000 7fe6581dd640 1 -- 172.21.15.77:0/1441406215 wait complete. 2024-03-06T07:29:33.199 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:33.226 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:33.226 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:34.227 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:34.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.337+0000 7efcfc8df640 1 Processor -- start 2024-03-06T07:29:34.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.337+0000 7efcfc8df640 1 -- start start 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcfc8df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcfc8df640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80cc980 con 0x7efcf806da40 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcfc8df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80ccc60 con 0x7efcf80cd7a0 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcfc8df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80cd220 con 0x7efcf81424f0 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf6d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efcf806da40 0x7efcf8071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:37132/0 (socket says 172.21.15.77:37132) 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf6d9d640 1 -- 172.21.15.77:0/996076578 learned_addr learned my addr 172.21.15.77:0/996076578 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf659c640 1 --2- 172.21.15.77:0/996076578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf5d9b640 1 -- 172.21.15.77:0/996076578 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346771594 0 0) 0x7efcf80cc980 con 0x7efcf806da40 2024-03-06T07:29:34.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf5d9b640 1 -- 172.21.15.77:0/996076578 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce4003520 con 0x7efcf806da40 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf5d9b640 1 -- 172.21.15.77:0/996076578 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353116223 0 0) 0x7efcf80ccc60 con 0x7efcf80cd7a0 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf5d9b640 1 -- 172.21.15.77:0/996076578 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf80cc980 con 0x7efcf80cd7a0 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf659c640 1 -- 172.21.15.77:0/996076578 >> v1:172.21.15.158:6789/0 conn(0x7efcf80cd7a0 legacy=0x7efcf806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf659c640 1 -- 172.21.15.77:0/996076578 >> v1:172.21.15.77:6789/0 conn(0x7efcf806da40 legacy=0x7efcf8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.338+0000 7efcf659c640 1 -- 172.21.15.77:0/996076578 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80ccf40 con 0x7efcf81424f0 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.339+0000 7efcf659c640 1 --2- 172.21.15.77:0/996076578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf80cc2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efce80097d0 tx=0x7efce802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec9d6d0cf452f19a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.339+0000 7efcf5d9b640 1 -- 172.21.15.77:0/996076578 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8035020 con 0x7efcf81424f0 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.339+0000 7efcf5d9b640 1 -- 172.21.15.77:0/996076578 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce8031890 con 0x7efcf81424f0 2024-03-06T07:29:34.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.339+0000 7efcf5d9b640 1 -- 172.21.15.77:0/996076578 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8031c60 con 0x7efcf81424f0 2024-03-06T07:29:34.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.340+0000 7efcfc8df640 1 -- 172.21.15.77:0/996076578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 msgr2=0x7efcf80cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.340+0000 7efcfc8df640 1 --2- 172.21.15.77:0/996076578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf80cc2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efce80097d0 tx=0x7efce802b930 comp rx=0 tx=0).stop 2024-03-06T07:29:34.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.340+0000 7efcfc8df640 1 -- 172.21.15.77:0/996076578 shutdown_connections 2024-03-06T07:29:34.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.340+0000 7efcfc8df640 1 --2- 172.21.15.77:0/996076578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf80cc2f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.340+0000 7efcfc8df640 1 -- 172.21.15.77:0/996076578 >> 172.21.15.77:0/996076578 conn(0x7efcf80cf8a0 msgr2=0x7efcf80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.340+0000 7efcfc8df640 1 -- 172.21.15.77:0/996076578 shutdown_connections 2024-03-06T07:29:34.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.340+0000 7efcfc8df640 1 -- 172.21.15.77:0/996076578 wait complete. 2024-03-06T07:29:34.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.341+0000 7efcfc8df640 1 Processor -- start 2024-03-06T07:29:34.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.341+0000 7efcfc8df640 1 -- start start 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efcfc8df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcf806da40 0x7efcf81f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efcfc8df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcf80cd7a0 0x7efcf81e54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efcfc8df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf81e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efcfc8df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf815d070 con 0x7efcf80cd7a0 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efcfc8df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf815cef0 con 0x7efcf806da40 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efcfc8df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf815d1f0 con 0x7efcf81424f0 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efceffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcf80cd7a0 0x7efcf81e54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efceffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcf80cd7a0 0x7efcf81e54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51434/0 (socket says 172.21.15.77:51434) 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efceffff640 1 -- 172.21.15.77:0/154922991 learned_addr learned my addr 172.21.15.77:0/154922991 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efcf659c640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcf806da40 0x7efcf81f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efcf6d9d640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf81e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efceffff640 1 -- 172.21.15.77:0/154922991 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 msgr2=0x7efcf81e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efceffff640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf81e5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efceffff640 1 -- 172.21.15.77:0/154922991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcf806da40 msgr2=0x7efcf81f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efceffff640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcf806da40 0x7efcf81f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.342+0000 7efceffff640 1 -- 172.21.15.77:0/154922991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf81e6c50 con 0x7efcf80cd7a0 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.343+0000 7efceffff640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcf80cd7a0 0x7efcf81e54d0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7efce000c500 tx=0x7efce000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.343+0000 7efceeffd640 1 -- 172.21.15.77:0/154922991 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce00052d0 con 0x7efcf80cd7a0 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.343+0000 7efceeffd640 1 -- 172.21.15.77:0/154922991 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce0005490 con 0x7efcf80cd7a0 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.343+0000 7efceeffd640 1 -- 172.21.15.77:0/154922991 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0012870 con 0x7efcf80cd7a0 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.343+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf81e6f30 con 0x7efcf80cd7a0 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.343+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf813ef50 con 0x7efcf80cd7a0 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.344+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf804f330 con 0x7efcf80cd7a0 2024-03-06T07:29:34.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.344+0000 7efceeffd640 1 -- 172.21.15.77:0/154922991 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efce0012a10 con 0x7efcf80cd7a0 2024-03-06T07:29:34.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.344+0000 7efceeffd640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcd806b4d0 0x7efcd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.345+0000 7efceeffd640 1 -- 172.21.15.77:0/154922991 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efce0089de0 con 0x7efcf80cd7a0 2024-03-06T07:29:34.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.345+0000 7efcf659c640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcd806b4d0 0x7efcd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.345+0000 7efcf659c640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcd806b4d0 0x7efcd806d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7efce8036000 tx=0x7efce803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.349+0000 7efceeffd640 1 -- 172.21.15.77:0/154922991 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efce00585e0 con 0x7efcf80cd7a0 2024-03-06T07:29:34.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.559+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcf81e5c40 con 0x7efcf80cd7a0 2024-03-06T07:29:34.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.559+0000 7efceeffd640 1 -- 172.21.15.77:0/154922991 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7efce005c680 con 0x7efcf80cd7a0 2024-03-06T07:29:34.561 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:34.561 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:34.561 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:34.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcd806b4d0 msgr2=0x7efcd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcd806b4d0 0x7efcd806d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7efce8036000 tx=0x7efce803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcf80cd7a0 msgr2=0x7efcf81e54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcf80cd7a0 0x7efcf81e54d0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7efce000c500 tx=0x7efce000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 shutdown_connections 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efcd806b4d0 0x7efcd806d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcf80cd7a0 0x7efcf81e54d0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efcf81424f0 0x7efcf81e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 --2- 172.21.15.77:0/154922991 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efcf806da40 0x7efcf81f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 >> 172.21.15.77:0/154922991 conn(0x7efcf80cf8a0 msgr2=0x7efcf815d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 shutdown_connections 2024-03-06T07:29:34.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:34.561+0000 7efcfc8df640 1 -- 172.21.15.77:0/154922991 wait complete. 2024-03-06T07:29:34.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:34.776 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:35.202 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:35.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:35.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.885+0000 7f863dc88640 1 Processor -- start 2024-03-06T07:29:35.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.885+0000 7f863dc88640 1 -- start start 2024-03-06T07:29:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f863dc88640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86381515e0 0x7f86380cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f863dc88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f863dc88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863815b8b0 con 0x7f86380cd7a0 2024-03-06T07:29:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f863dc88640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8638070a70 con 0x7f86381515e0 2024-03-06T07:29:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f863dc88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8638070d50 con 0x7f863815b430 2024-03-06T07:29:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f86377fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86381515e0 0x7f86380cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f8636ffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f86377fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86381515e0 0x7f86380cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:51680/0 (socket says 172.21.15.77:51680) 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f86377fe640 1 -- 172.21.15.77:0/1535785662 learned_addr learned my addr 172.21.15.77:0/1535785662 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.886+0000 7f8637fff640 1 --1- 172.21.15.77:0/1535785662 >> v1:172.21.15.158:6789/0 conn(0x7f86380cd7a0 0x7f86380cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49094/0 (socket says 172.21.15.77:49094) 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.887+0000 7f8636ffd640 1 -- 172.21.15.77:0/1535785662 >> v2:172.21.15.158:3300/0 conn(0x7f86381515e0 msgr2=0x7f86380cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.887+0000 7f8636ffd640 1 --2- 172.21.15.77:0/1535785662 >> v2:172.21.15.158:3300/0 conn(0x7f86381515e0 0x7f86380cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.887+0000 7f8636ffd640 1 -- 172.21.15.77:0/1535785662 >> v1:172.21.15.158:6789/0 conn(0x7f86380cd7a0 legacy=0x7f86380cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.887+0000 7f8636ffd640 1 -- 172.21.15.77:0/1535785662 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638071030 con 0x7f863815b430 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.887+0000 7f86367fc640 1 -- 172.21.15.77:0/1535785662 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178417755 0 0) 0x7f863815b8b0 con 0x7f86380cd7a0 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.887+0000 7f86377fe640 1 --2- 172.21.15.77:0/1535785662 >> v2:172.21.15.158:3300/0 conn(0x7f86381515e0 0x7f86380cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.887+0000 7f8636ffd640 1 --2- 172.21.15.77:0/1535785662 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638071830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f862c0097a0 tx=0x7f862c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4bfbd735d61a47b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.887+0000 7f86367fc640 1 -- 172.21.15.77:0/1535785662 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c035020 con 0x7f863815b430 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f86367fc640 1 -- 172.21.15.77:0/1535785662 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862c031830 con 0x7f863815b430 2024-03-06T07:29:35.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f86367fc640 1 -- 172.21.15.77:0/1535785662 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c031c00 con 0x7f863815b430 2024-03-06T07:29:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f863dc88640 1 -- 172.21.15.77:0/1535785662 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 msgr2=0x7f8638071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f863dc88640 1 --2- 172.21.15.77:0/1535785662 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638071830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f862c0097a0 tx=0x7f862c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f863dc88640 1 -- 172.21.15.77:0/1535785662 shutdown_connections 2024-03-06T07:29:35.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f863dc88640 1 --2- 172.21.15.77:0/1535785662 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638071830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f863dc88640 1 --2- 172.21.15.77:0/1535785662 >> v2:172.21.15.158:3300/0 conn(0x7f86381515e0 0x7f86380cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f863dc88640 1 -- 172.21.15.77:0/1535785662 >> 172.21.15.77:0/1535785662 conn(0x7f86380cf8a0 msgr2=0x7f86380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f863dc88640 1 -- 172.21.15.77:0/1535785662 shutdown_connections 2024-03-06T07:29:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.888+0000 7f863dc88640 1 -- 172.21.15.77:0/1535785662 wait complete. 2024-03-06T07:29:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.889+0000 7f863dc88640 1 Processor -- start 2024-03-06T07:29:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.889+0000 7f863dc88640 1 -- start start 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f863dc88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86380cd7a0 0x7f8638142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f863dc88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86381515e0 0x7f863815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f863dc88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f863dc88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638071ef0 con 0x7f86381515e0 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f863dc88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638071d70 con 0x7f86380cd7a0 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f863dc88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86380cc3a0 con 0x7f863815b430 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f8636ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86381515e0 0x7f863815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f8636ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86381515e0 0x7f863815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51436/0 (socket says 172.21.15.77:51436) 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f8636ffd640 1 -- 172.21.15.77:0/815072895 learned_addr learned my addr 172.21.15.77:0/815072895 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f86377fe640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86380cd7a0 0x7f8638142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.890+0000 7f8637fff640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f8636ffd640 1 -- 172.21.15.77:0/815072895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 msgr2=0x7f8638159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f8636ffd640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f8636ffd640 1 -- 172.21.15.77:0/815072895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86380cd7a0 msgr2=0x7f8638142ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f8636ffd640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86380cd7a0 0x7f8638142ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f8636ffd640 1 -- 172.21.15.77:0/815072895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638143730 con 0x7f86381515e0 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f8636ffd640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86381515e0 0x7f863815bae0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f862c034040 tx=0x7f862c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f861ffff640 1 -- 172.21.15.77:0/815072895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c035040 con 0x7f86381515e0 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f861ffff640 1 -- 172.21.15.77:0/815072895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862c004230 con 0x7f86381515e0 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8638151a20 con 0x7f86381515e0 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f861ffff640 1 -- 172.21.15.77:0/815072895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c03a640 con 0x7f86381515e0 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.891+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8638151fe0 con 0x7f86381515e0 2024-03-06T07:29:35.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.892+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8638068840 con 0x7f86381515e0 2024-03-06T07:29:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.892+0000 7f861ffff640 1 -- 172.21.15.77:0/815072895 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f862c038020 con 0x7f86381515e0 2024-03-06T07:29:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.893+0000 7f861ffff640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f862406b4d0 0x7f862406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.893+0000 7f861ffff640 1 -- 172.21.15.77:0/815072895 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f862c0abe50 con 0x7f86381515e0 2024-03-06T07:29:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.893+0000 7f86377fe640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f862406b4d0 0x7f862406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.893+0000 7f86377fe640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f862406b4d0 0x7f862406d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f862800a7d0 tx=0x7f862800a350 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:35.897+0000 7f861ffff640 1 -- 172.21.15.77:0/815072895 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f862c07a640 con 0x7f86381515e0 2024-03-06T07:29:36.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.107+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86380b4c30 con 0x7f86381515e0 2024-03-06T07:29:36.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.108+0000 7f861ffff640 1 -- 172.21.15.77:0/815072895 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f862c07e6e0 con 0x7f86381515e0 2024-03-06T07:29:36.110 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:36.110 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:36.110 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f862406b4d0 msgr2=0x7f862406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f862406b4d0 0x7f862406d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f862800a7d0 tx=0x7f862800a350 comp rx=0 tx=0).stop 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86381515e0 msgr2=0x7f863815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86381515e0 0x7f863815bae0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f862c034040 tx=0x7f862c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 shutdown_connections 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f862406b4d0 0x7f862406d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86381515e0 0x7f863815bae0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f863815b430 0x7f8638159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 --2- 172.21.15.77:0/815072895 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86380cd7a0 0x7f8638142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 >> 172.21.15.77:0/815072895 conn(0x7f86380cf8a0 msgr2=0x7f863814e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 shutdown_connections 2024-03-06T07:29:36.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:36.110+0000 7f863dc88640 1 -- 172.21.15.77:0/815072895 wait complete. 2024-03-06T07:29:36.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:36.372 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:37.204 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:37.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:37.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.483+0000 7f2dd6bac640 1 Processor -- start 2024-03-06T07:29:37.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.483+0000 7f2dd6bac640 1 -- start start 2024-03-06T07:29:37.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dd6bac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2dd00d0700 0x7f2dd00b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dd6bac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00b0eb0 0x7f2dd00b1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dd6bac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd015c270 con 0x7f2dd00d0320 2024-03-06T07:29:37.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dd6bac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd00b04e0 con 0x7f2dd00d0700 2024-03-06T07:29:37.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dd6bac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00d0e90 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dd5149640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00b0eb0 0x7f2dd00b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dcffff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2dd00d0700 0x7f2dd00b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dd4948640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2dd00d0320 0x7f2dd00b4f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49108/0 (socket says 172.21.15.77:49108) 2024-03-06T07:29:37.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.484+0000 7f2dd4948640 1 -- 172.21.15.77:0/2213983778 learned_addr learned my addr 172.21.15.77:0/2213983778 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:37.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.485+0000 7f2dcffff640 1 -- 172.21.15.77:0/2213983778 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00b0eb0 msgr2=0x7f2dd00b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.485+0000 7f2dcffff640 1 --2- 172.21.15.77:0/2213983778 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00b0eb0 0x7f2dd00b1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.485+0000 7f2dcffff640 1 -- 172.21.15.77:0/2213983778 >> v1:172.21.15.158:6789/0 conn(0x7f2dd00d0320 legacy=0x7f2dd00b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.485+0000 7f2dcffff640 1 -- 172.21.15.77:0/2213983778 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd00b5ae0 con 0x7f2dd00d0700 2024-03-06T07:29:37.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dcffff640 1 --2- 172.21.15.77:0/2213983778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00d0700 0x7f2dd00b0970 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40097a0 tx=0x7f2dc402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed9507956ff9778d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dcf7fe640 1 -- 172.21.15.77:0/2213983778 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4034070 con 0x7f2dd00d0700 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dcf7fe640 1 -- 172.21.15.77:0/2213983778 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc4030610 con 0x7f2dd00d0700 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dcf7fe640 1 -- 172.21.15.77:0/2213983778 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40309e0 con 0x7f2dd00d0700 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dd6bac640 1 -- 172.21.15.77:0/2213983778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00d0700 msgr2=0x7f2dd00b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/2213983778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00d0700 0x7f2dd00b0970 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40097a0 tx=0x7f2dc402b960 comp rx=0 tx=0).stop 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dd6bac640 1 -- 172.21.15.77:0/2213983778 shutdown_connections 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/2213983778 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00b0eb0 0x7f2dd00b1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/2213983778 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00d0700 0x7f2dd00b0970 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dd6bac640 1 -- 172.21.15.77:0/2213983778 >> 172.21.15.77:0/2213983778 conn(0x7f2dd00724d0 msgr2=0x7f2dd00728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dd6bac640 1 -- 172.21.15.77:0/2213983778 shutdown_connections 2024-03-06T07:29:37.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.486+0000 7f2dd6bac640 1 -- 172.21.15.77:0/2213983778 wait complete. 2024-03-06T07:29:37.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.488+0000 7f2dd6bac640 1 Processor -- start 2024-03-06T07:29:37.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.488+0000 7f2dd6bac640 1 -- start start 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.488+0000 7f2dd6bac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00b0eb0 0x7f2dd01f1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.488+0000 7f2dd6bac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00d0320 0x7f2dd0151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.488+0000 7f2dd6bac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2dd00d0700 0x7f2dd01518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.488+0000 7f2dd6bac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00b60a0 con 0x7f2dd00d0700 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.488+0000 7f2dd6bac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00b0170 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd6bac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00ca790 con 0x7f2dd00d0320 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd5149640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2dd00d0700 0x7f2dd01518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd4948640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00b0eb0 0x7f2dd01f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dcffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00d0320 0x7f2dd0151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd5149640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2dd00d0700 0x7f2dd01518b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51448/0 (socket says 172.21.15.77:51448) 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd5149640 1 -- 172.21.15.77:0/3004450481 learned_addr learned my addr 172.21.15.77:0/3004450481 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd4948640 1 -- 172.21.15.77:0/3004450481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00d0320 msgr2=0x7f2dd0151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd4948640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00d0320 0x7f2dd0151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd4948640 1 -- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2dd00d0700 msgr2=0x7f2dd01518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd4948640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2dd00d0700 0x7f2dd01518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd4948640 1 -- 172.21.15.77:0/3004450481 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd0156420 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd4948640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00b0eb0 0x7f2dd01f1660 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c370 tx=0x7f2dc000c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dcd7fa640 1 -- 172.21.15.77:0/3004450481 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0017070 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dcd7fa640 1 -- 172.21.15.77:0/3004450481 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc0005ce0 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd01590b0 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dcd7fa640 1 -- 172.21.15.77:0/3004450481 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0007310 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.489+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd0159670 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.490+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd004f330 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.491+0000 7f2dcd7fa640 1 -- 172.21.15.77:0/3004450481 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dc001a070 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.491+0000 7f2dcd7fa640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.491+0000 7f2dcffff640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.491+0000 7f2dcd7fa640 1 -- 172.21.15.77:0/3004450481 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2dc0089500 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.492+0000 7f2dcffff640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4036000 tx=0x7f2dc403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.496+0000 7f2dcd7fa640 1 -- 172.21.15.77:0/3004450481 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dc0057cf0 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.702+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dd00b5800 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.703+0000 7f2dcd7fa640 1 -- 172.21.15.77:0/3004450481 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f2dc005bd90 con 0x7f2dd00b0eb0 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:37.705 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:37.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dbc06b4d0 msgr2=0x7f2dbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4036000 tx=0x7f2dc403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00b0eb0 msgr2=0x7f2dd01f1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00b0eb0 0x7f2dd01f1660 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c370 tx=0x7f2dc000c830 comp rx=0 tx=0).stop 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 shutdown_connections 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2dbc06b4d0 0x7f2dbc06d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2dd00d0700 0x7f2dd01518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2dd00d0320 0x7f2dd0151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 --2- 172.21.15.77:0/3004450481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2dd00b0eb0 0x7f2dd01f1660 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 >> 172.21.15.77:0/3004450481 conn(0x7f2dd00724d0 msgr2=0x7f2dd00728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 shutdown_connections 2024-03-06T07:29:37.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:37.705+0000 7f2dd6bac640 1 -- 172.21.15.77:0/3004450481 wait complete. 2024-03-06T07:29:37.919 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:37.920 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:38.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.030+0000 7f181bf8a640 1 Processor -- start 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.030+0000 7f181bf8a640 1 -- start start 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f181bf8a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181413dc10 0x7f18140b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f181bf8a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f181415ef60 0x7f18140b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f181bf8a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181415f580 con 0x7f181415bf30 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f181bf8a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18140cf3f0 con 0x7f181415ef60 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f181bf8a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181415b210 con 0x7f181413dc10 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f1819525640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f181415bf30 0x7f181415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45604/0 (socket says 172.21.15.77:45604) 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f1819525640 1 -- 172.21.15.77:0/277701868 learned_addr learned my addr 172.21.15.77:0/277701868 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f1819d26640 1 --2- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181413dc10 0x7f18140b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.031+0000 7f181a527640 1 --2- 172.21.15.77:0/277701868 >> v2:172.21.15.158:3300/0 conn(0x7f181415ef60 0x7f18140b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.032+0000 7f1818d24640 1 -- 172.21.15.77:0/277701868 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945631215 0 0) 0x7f181415f580 con 0x7f181415bf30 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.032+0000 7f1818d24640 1 -- 172.21.15.77:0/277701868 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1800003520 con 0x7f181415bf30 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.032+0000 7f181a527640 1 -- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181413dc10 msgr2=0x7f18140b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.032+0000 7f181a527640 1 --2- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181413dc10 0x7f18140b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.032+0000 7f181a527640 1 -- 172.21.15.77:0/277701868 >> v1:172.21.15.77:6789/0 conn(0x7f181415bf30 legacy=0x7f181415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.032+0000 7f181a527640 1 -- 172.21.15.77:0/277701868 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181415b450 con 0x7f181415ef60 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.032+0000 7f1819d26640 1 --2- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181413dc10 0x7f18140b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.032+0000 7f181a527640 1 --2- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 0x7f18140b2e90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f180c009d90 tx=0x7f180c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7310213d83add28c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f1818d24640 1 -- 172.21.15.77:0/277701868 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c039070 con 0x7f181415ef60 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f1818d24640 1 -- 172.21.15.77:0/277701868 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c02cbc0 con 0x7f181415ef60 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f1818d24640 1 -- 172.21.15.77:0/277701868 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c034440 con 0x7f181415ef60 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f181bf8a640 1 -- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 msgr2=0x7f18140b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f181bf8a640 1 --2- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 0x7f18140b2e90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f180c009d90 tx=0x7f180c02be10 comp rx=0 tx=0).stop 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f181bf8a640 1 -- 172.21.15.77:0/277701868 shutdown_connections 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f181bf8a640 1 --2- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 0x7f18140b2e90 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f181bf8a640 1 --2- 172.21.15.77:0/277701868 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181413dc10 0x7f18140b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.033+0000 7f181bf8a640 1 -- 172.21.15.77:0/277701868 >> 172.21.15.77:0/277701868 conn(0x7f18140cf8a0 msgr2=0x7f18140cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.034+0000 7f181bf8a640 1 -- 172.21.15.77:0/277701868 shutdown_connections 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.034+0000 7f181bf8a640 1 -- 172.21.15.77:0/277701868 wait complete. 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.035+0000 7f181bf8a640 1 Processor -- start 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.035+0000 7f181bf8a640 1 -- start start 2024-03-06T07:29:39.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.035+0000 7f181bf8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f181413dc10 0x7f181413a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.035+0000 7f181bf8a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181415bf30 0x7f181413bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.035+0000 7f181bf8a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 0x7f181413c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.035+0000 7f181bf8a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18140708d0 con 0x7f181413dc10 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.035+0000 7f181bf8a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181413dff0 con 0x7f181415ef60 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.035+0000 7f181bf8a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1814070a50 con 0x7f181415bf30 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f181413dc10 0x7f181413a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f181413dc10 0x7f181413a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34348/0 (socket says 172.21.15.77:34348) 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 -- 172.21.15.77:0/4054984649 learned_addr learned my addr 172.21.15.77:0/4054984649 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819525640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181415bf30 0x7f181413bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f181a527640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 0x7f181413c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 -- 172.21.15.77:0/4054984649 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181415bf30 msgr2=0x7f181413bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181415bf30 0x7f181413bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 -- 172.21.15.77:0/4054984649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 msgr2=0x7f181413c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 0x7f181413c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 -- 172.21.15.77:0/4054984649 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18141f0d80 con 0x7f181413dc10 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f1819d26640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f181413dc10 0x7f181413a710 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f1810010e20 tx=0x7f181000e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f180a7fc640 1 -- 172.21.15.77:0/4054984649 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181001e070 con 0x7f181413dc10 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f180a7fc640 1 -- 172.21.15.77:0/4054984649 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1810011860 con 0x7f181413dc10 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1814157bc0 con 0x7f181413dc10 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f180a7fc640 1 -- 172.21.15.77:0/4054984649 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810011c30 con 0x7f181413dc10 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.036+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1814158180 con 0x7f181413dc10 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.037+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181404f330 con 0x7f181413dc10 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.038+0000 7f180a7fc640 1 -- 172.21.15.77:0/4054984649 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1810021070 con 0x7f181413dc10 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.038+0000 7f180a7fc640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17f806b4d0 0x7f17f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.038+0000 7f1819525640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17f806b4d0 0x7f17f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.038+0000 7f180a7fc640 1 -- 172.21.15.77:0/4054984649 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f181008c440 con 0x7f181413dc10 2024-03-06T07:29:39.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.039+0000 7f1819525640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17f806b4d0 0x7f17f806d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f181415a9b0 tx=0x7f1804009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.043+0000 7f180a7fc640 1 -- 172.21.15.77:0/4054984649 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1810009e20 con 0x7f181413dc10 2024-03-06T07:29:39.206 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:39.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.251+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f181406eb00 con 0x7f181413dc10 2024-03-06T07:29:39.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.251+0000 7f180a7fc640 1 -- 172.21.15.77:0/4054984649 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f181005ac30 con 0x7f181413dc10 2024-03-06T07:29:39.253 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:39.253 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:39.254 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:39.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17f806b4d0 msgr2=0x7f17f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17f806b4d0 0x7f17f806d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f181415a9b0 tx=0x7f1804009040 comp rx=0 tx=0).stop 2024-03-06T07:29:39.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f181413dc10 msgr2=0x7f181413a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f181413dc10 0x7f181413a710 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f1810010e20 tx=0x7f181000e730 comp rx=0 tx=0).stop 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 shutdown_connections 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f181415ef60 0x7f181413c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f181415bf30 0x7f181413bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f17f806b4d0 0x7f17f806d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 --2- 172.21.15.77:0/4054984649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f181413dc10 0x7f181413a710 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 >> 172.21.15.77:0/4054984649 conn(0x7f18140cf8a0 msgr2=0x7f1814070e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 shutdown_connections 2024-03-06T07:29:39.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:39.253+0000 7f181bf8a640 1 -- 172.21.15.77:0/4054984649 wait complete. 2024-03-06T07:29:39.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:39.467 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:40.468 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:40.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.578+0000 7fa03730f640 1 Processor -- start 2024-03-06T07:29:40.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.578+0000 7fa03730f640 1 -- start start 2024-03-06T07:29:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa03730f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa0300b3060 0x7fa03015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa03730f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa03015bf30 0x7fa03015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa03730f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03015f2d0 con 0x7fa03013dc10 2024-03-06T07:29:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa03730f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0300cf3f0 con 0x7fa03015bf30 2024-03-06T07:29:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa03730f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa03015b210 con 0x7fa0300b3060 2024-03-06T07:29:40.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa0358ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa03015bf30 0x7fa03015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa0358ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa03015bf30 0x7fa03015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34364/0 (socket says 172.21.15.77:34364) 2024-03-06T07:29:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa0358ac640 1 -- 172.21.15.77:0/2821790048 learned_addr learned my addr 172.21.15.77:0/2821790048 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa0348aa640 1 --2- 172.21.15.77:0/2821790048 >> v2:172.21.15.158:3300/0 conn(0x7fa0300b3060 0x7fa03015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa0358ac640 1 -- 172.21.15.77:0/2821790048 >> v2:172.21.15.158:3300/0 conn(0x7fa0300b3060 msgr2=0x7fa03015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa0358ac640 1 --2- 172.21.15.77:0/2821790048 >> v2:172.21.15.158:3300/0 conn(0x7fa0300b3060 0x7fa03015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa0358ac640 1 -- 172.21.15.77:0/2821790048 >> v1:172.21.15.158:6789/0 conn(0x7fa03013dc10 legacy=0x7fa0300b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.579+0000 7fa0358ac640 1 -- 172.21.15.77:0/2821790048 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03015f580 con 0x7fa03015bf30 2024-03-06T07:29:40.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.580+0000 7fa0358ac640 1 --2- 172.21.15.77:0/2821790048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03015bf30 0x7fa03015ed90 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fa030068700 tx=0x7fa02c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6dd64ae468780c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.580+0000 7fa027fff640 1 -- 172.21.15.77:0/2821790048 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c035020 con 0x7fa03015bf30 2024-03-06T07:29:40.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.580+0000 7fa027fff640 1 -- 172.21.15.77:0/2821790048 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c031870 con 0x7fa03015bf30 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.580+0000 7fa027fff640 1 -- 172.21.15.77:0/2821790048 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c031c40 con 0x7fa03015bf30 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.580+0000 7fa03730f640 1 -- 172.21.15.77:0/2821790048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03015bf30 msgr2=0x7fa03015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.580+0000 7fa03730f640 1 --2- 172.21.15.77:0/2821790048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03015bf30 0x7fa03015ed90 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fa030068700 tx=0x7fa02c02b910 comp rx=0 tx=0).stop 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.581+0000 7fa03730f640 1 -- 172.21.15.77:0/2821790048 shutdown_connections 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.581+0000 7fa03730f640 1 --2- 172.21.15.77:0/2821790048 >> v2:172.21.15.158:3300/0 conn(0x7fa0300b3060 0x7fa03015e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.581+0000 7fa03730f640 1 --2- 172.21.15.77:0/2821790048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03015bf30 0x7fa03015ed90 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.581+0000 7fa03730f640 1 -- 172.21.15.77:0/2821790048 >> 172.21.15.77:0/2821790048 conn(0x7fa0300cf8a0 msgr2=0x7fa0300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.581+0000 7fa03730f640 1 -- 172.21.15.77:0/2821790048 shutdown_connections 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.581+0000 7fa03730f640 1 -- 172.21.15.77:0/2821790048 wait complete. 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.582+0000 7fa03730f640 1 Processor -- start 2024-03-06T07:29:40.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.582+0000 7fa03730f640 1 -- start start 2024-03-06T07:29:40.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.582+0000 7fa03730f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0300b3060 0x7fa030151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.582+0000 7fa03730f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03013dc10 0x7fa0301373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.582+0000 7fa03730f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa03015bf30 0x7fa030071480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa03730f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03013dff0 con 0x7fa03013dc10 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa03730f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03015b730 con 0x7fa0300b3060 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa03730f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0300708d0 con 0x7fa03015bf30 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03013dc10 0x7fa0301373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03013dc10 0x7fa0301373b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34380/0 (socket says 172.21.15.77:34380) 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 -- 172.21.15.77:0/2927786844 learned_addr learned my addr 172.21.15.77:0/2927786844 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0350ab640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0300b3060 0x7fa030151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0358ac640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa03015bf30 0x7fa030071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 -- 172.21.15.77:0/2927786844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa03015bf30 msgr2=0x7fa030071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa03015bf30 0x7fa030071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 -- 172.21.15.77:0/2927786844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0300b3060 msgr2=0x7fa030151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0300b3060 0x7fa030151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 -- 172.21.15.77:0/2927786844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0300cd840 con 0x7fa03013dc10 2024-03-06T07:29:40.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa0348aa640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03013dc10 0x7fa0301373b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fa02000e9b0 tx=0x7fa02000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa025ffb640 1 -- 172.21.15.77:0/2927786844 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa020017070 con 0x7fa03013dc10 2024-03-06T07:29:40.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa025ffb640 1 -- 172.21.15.77:0/2927786844 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02000c9a0 con 0x7fa03013dc10 2024-03-06T07:29:40.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0300cdab0 con 0x7fa03013dc10 2024-03-06T07:29:40.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.583+0000 7fa025ffb640 1 -- 172.21.15.77:0/2927786844 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02000cd90 con 0x7fa03013dc10 2024-03-06T07:29:40.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.584+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03015a910 con 0x7fa03013dc10 2024-03-06T07:29:40.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.584+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa03004f330 con 0x7fa03013dc10 2024-03-06T07:29:40.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.585+0000 7fa025ffb640 1 -- 172.21.15.77:0/2927786844 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa02001a070 con 0x7fa03013dc10 2024-03-06T07:29:40.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.585+0000 7fa025ffb640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa01406b4d0 0x7fa01406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.585+0000 7fa025ffb640 1 -- 172.21.15.77:0/2927786844 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa02008a1d0 con 0x7fa03013dc10 2024-03-06T07:29:40.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.585+0000 7fa0350ab640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa01406b4d0 0x7fa01406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.586+0000 7fa0350ab640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa01406b4d0 0x7fa01406d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa028001f10 tx=0x7fa028009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.590+0000 7fa025ffb640 1 -- 172.21.15.77:0/2927786844 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0200589c0 con 0x7fa03013dc10 2024-03-06T07:29:40.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.797+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa03006eb00 con 0x7fa03013dc10 2024-03-06T07:29:40.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.797+0000 7fa025ffb640 1 -- 172.21.15.77:0/2927786844 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7fa02005ca60 con 0x7fa03013dc10 2024-03-06T07:29:40.800 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:40.800 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:40.800 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:40.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa01406b4d0 msgr2=0x7fa01406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa01406b4d0 0x7fa01406d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa028001f10 tx=0x7fa028009040 comp rx=0 tx=0).stop 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03013dc10 msgr2=0x7fa0301373b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03013dc10 0x7fa0301373b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fa02000e9b0 tx=0x7fa02000ee70 comp rx=0 tx=0).stop 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 shutdown_connections 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa01406b4d0 0x7fa01406d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03013dc10 0x7fa0301373b0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa03015bf30 0x7fa030071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 --2- 172.21.15.77:0/2927786844 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0300b3060 0x7fa030151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 >> 172.21.15.77:0/2927786844 conn(0x7fa0300cf8a0 msgr2=0x7fa0300afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.799+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 shutdown_connections 2024-03-06T07:29:40.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:40.800+0000 7fa03730f640 1 -- 172.21.15.77:0/2927786844 wait complete. 2024-03-06T07:29:41.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:41.013 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:42.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.125+0000 7f9873a15640 1 Processor -- start 2024-03-06T07:29:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.125+0000 7f9873a15640 1 -- start start 2024-03-06T07:29:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9873a15640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f986c06da40 0x7f986c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9873a15640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0cc980 con 0x7f986c0cd7a0 2024-03-06T07:29:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9873a15640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0ccc60 con 0x7f986c1424f0 2024-03-06T07:29:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9873a15640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c0cd220 con 0x7f986c06da40 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9870fb0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f986c06da40 0x7f986c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9871fb2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f986c0cd7a0 0x7f986c071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45616/0 (socket says 172.21.15.77:45616) 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9871fb2640 1 -- 172.21.15.77:0/3913891905 learned_addr learned my addr 172.21.15.77:0/3913891905 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9863fff640 1 -- 172.21.15.77:0/3913891905 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657954932 0 0) 0x7f986c0cc980 con 0x7f986c0cd7a0 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9863fff640 1 -- 172.21.15.77:0/3913891905 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9858003520 con 0x7f986c0cd7a0 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9870fb0640 1 -- 172.21.15.77:0/3913891905 >> v1:172.21.15.158:6789/0 conn(0x7f986c1424f0 legacy=0x7f986c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9863fff640 1 -- 172.21.15.77:0/3913891905 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533517674 0 0) 0x7f986c0ccc60 con 0x7f986c1424f0 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9870fb0640 1 -- 172.21.15.77:0/3913891905 >> v1:172.21.15.77:6789/0 conn(0x7f986c0cd7a0 legacy=0x7f986c071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.126+0000 7f9870fb0640 1 -- 172.21.15.77:0/3913891905 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0ccf40 con 0x7f986c06da40 2024-03-06T07:29:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.127+0000 7f9870fb0640 1 --2- 172.21.15.77:0/3913891905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 0x7f986c0cc2f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f985c0097a0 tx=0x7f985c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2498e68ed22e3987 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.127+0000 7f9863fff640 1 -- 172.21.15.77:0/3913891905 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c034070 con 0x7f986c06da40 2024-03-06T07:29:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.127+0000 7f9863fff640 1 -- 172.21.15.77:0/3913891905 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c030610 con 0x7f986c06da40 2024-03-06T07:29:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.127+0000 7f9863fff640 1 -- 172.21.15.77:0/3913891905 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c0309e0 con 0x7f986c06da40 2024-03-06T07:29:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.128+0000 7f9873a15640 1 -- 172.21.15.77:0/3913891905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 msgr2=0x7f986c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.128+0000 7f9873a15640 1 --2- 172.21.15.77:0/3913891905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 0x7f986c0cc2f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f985c0097a0 tx=0x7f985c02b960 comp rx=0 tx=0).stop 2024-03-06T07:29:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.128+0000 7f9873a15640 1 -- 172.21.15.77:0/3913891905 shutdown_connections 2024-03-06T07:29:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.128+0000 7f9873a15640 1 --2- 172.21.15.77:0/3913891905 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 0x7f986c0cc2f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.128+0000 7f9873a15640 1 -- 172.21.15.77:0/3913891905 >> 172.21.15.77:0/3913891905 conn(0x7f986c0cf8a0 msgr2=0x7f986c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.128+0000 7f9873a15640 1 -- 172.21.15.77:0/3913891905 shutdown_connections 2024-03-06T07:29:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.128+0000 7f9873a15640 1 -- 172.21.15.77:0/3913891905 wait complete. 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.129+0000 7f9873a15640 1 Processor -- start 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.129+0000 7f9873a15640 1 -- start start 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9873a15640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 0x7f986c06b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9873a15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f986c0cd7a0 0x7f986c1f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9873a15640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f986c1424f0 0x7f986c13e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9873a15640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c15d070 con 0x7f986c0cd7a0 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9873a15640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c15cef0 con 0x7f986c06da40 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9873a15640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c15d1f0 con 0x7f986c1424f0 2024-03-06T07:29:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9870fb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f986c0cd7a0 0x7f986c1f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9870fb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f986c0cd7a0 0x7f986c1f2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34388/0 (socket says 172.21.15.77:34388) 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9870fb0640 1 -- 172.21.15.77:0/3392975951 learned_addr learned my addr 172.21.15.77:0/3392975951 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f98717b1640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 0x7f986c06b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9871fb2640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f986c1424f0 0x7f986c13e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9870fb0640 1 -- 172.21.15.77:0/3392975951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f986c1424f0 msgr2=0x7f986c13e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9870fb0640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f986c1424f0 0x7f986c13e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9870fb0640 1 -- 172.21.15.77:0/3392975951 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 msgr2=0x7f986c06b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9870fb0640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 0x7f986c06b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.130+0000 7f9870fb0640 1 -- 172.21.15.77:0/3392975951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c13d180 con 0x7f986c0cd7a0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.131+0000 7f9870fb0640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f986c0cd7a0 0x7f986c1f2b90 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f985c0098c0 tx=0x7f985c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.131+0000 7f9861ffb640 1 -- 172.21.15.77:0/3392975951 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c049070 con 0x7f986c0cd7a0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.131+0000 7f9861ffb640 1 -- 172.21.15.77:0/3392975951 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c030410 con 0x7f986c0cd7a0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.131+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c13d460 con 0x7f986c0cd7a0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.131+0000 7f9861ffb640 1 -- 172.21.15.77:0/3392975951 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c034040 con 0x7f986c0cd7a0 2024-03-06T07:29:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.131+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c13dd00 con 0x7f986c0cd7a0 2024-03-06T07:29:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.132+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c04f330 con 0x7f986c0cd7a0 2024-03-06T07:29:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.132+0000 7f9861ffb640 1 -- 172.21.15.77:0/3392975951 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f985c04c070 con 0x7f986c0cd7a0 2024-03-06T07:29:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.133+0000 7f9861ffb640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.133+0000 7f9861ffb640 1 -- 172.21.15.77:0/3392975951 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f985c0abd00 con 0x7f986c0cd7a0 2024-03-06T07:29:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.133+0000 7f98717b1640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.133+0000 7f98717b1640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f985006b4d0 0x7f985006d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9868000f60 tx=0x7f9868009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.137+0000 7f9861ffb640 1 -- 172.21.15.77:0/3392975951 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f985c0305e0 con 0x7f986c0cd7a0 2024-03-06T07:29:42.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.346+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f986c06a170 con 0x7f986c0cd7a0 2024-03-06T07:29:42.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.346+0000 7f9861ffb640 1 -- 172.21.15.77:0/3392975951 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f985c07a4f0 con 0x7f986c0cd7a0 2024-03-06T07:29:42.349 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:42.349 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:42.349 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.348+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f985006b4d0 msgr2=0x7f985006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.348+0000 7f9873a15640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f985006b4d0 0x7f985006d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9868000f60 tx=0x7f9868009040 comp rx=0 tx=0).stop 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.348+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f986c0cd7a0 msgr2=0x7f986c1f2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.348+0000 7f9873a15640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f986c0cd7a0 0x7f986c1f2b90 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f985c0098c0 tx=0x7f985c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.349+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 shutdown_connections 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.349+0000 7f9873a15640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.349+0000 7f9873a15640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f986c0cd7a0 0x7f986c1f2b90 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.349+0000 7f9873a15640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f986c1424f0 0x7f986c13e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.349+0000 7f9873a15640 1 --2- 172.21.15.77:0/3392975951 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f986c06da40 0x7f986c06b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.349+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 >> 172.21.15.77:0/3392975951 conn(0x7f986c0cf8a0 msgr2=0x7f986c15b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.349+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 shutdown_connections 2024-03-06T07:29:42.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:42.349+0000 7f9873a15640 1 -- 172.21.15.77:0/3392975951 wait complete. 2024-03-06T07:29:42.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:42.563 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:43.210 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:43.564 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:43.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.674+0000 7f63ff53e640 1 Processor -- start 2024-03-06T07:29:43.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.674+0000 7f63ff53e640 1 -- start start 2024-03-06T07:29:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63ff53e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f81515e0 0x7f63f80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63ff53e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f63f80cd7a0 0x7f63f80cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63ff53e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f8071f40 con 0x7f63f815b430 2024-03-06T07:29:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63ff53e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f815b8b0 con 0x7f63f80cd7a0 2024-03-06T07:29:43.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63ff53e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8070a70 con 0x7f63f81515e0 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63fdadb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f63f80cd7a0 0x7f63f80cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63fcad9640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f63f815b430 0x7f63f8071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:39766/0 (socket says 172.21.15.77:39766) 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63fd2da640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f81515e0 0x7f63f80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.675+0000 7f63fcad9640 1 -- 172.21.15.77:0/1833273879 learned_addr learned my addr 172.21.15.77:0/1833273879 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.676+0000 7f63fdadb640 1 -- 172.21.15.77:0/1833273879 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f81515e0 msgr2=0x7f63f80cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.676+0000 7f63fdadb640 1 --2- 172.21.15.77:0/1833273879 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f81515e0 0x7f63f80cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.676+0000 7f63fdadb640 1 -- 172.21.15.77:0/1833273879 >> v1:172.21.15.158:6789/0 conn(0x7f63f815b430 legacy=0x7f63f8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.676+0000 7f63fdadb640 1 -- 172.21.15.77:0/1833273879 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8070d50 con 0x7f63f80cd7a0 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.676+0000 7f63effff640 1 -- 172.21.15.77:0/1833273879 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020072808 0 0) 0x7f63f8071f40 con 0x7f63f815b430 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.676+0000 7f63fd2da640 1 --2- 172.21.15.77:0/1833273879 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f81515e0 0x7f63f80cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.676+0000 7f63fdadb640 1 --2- 172.21.15.77:0/1833273879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f80cd7a0 0x7f63f80cdbd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009d90 tx=0x7f63f002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=937f59f74ddc0ae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63effff640 1 -- 172.21.15.77:0/1833273879 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0039070 con 0x7f63f80cd7a0 2024-03-06T07:29:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63effff640 1 -- 172.21.15.77:0/1833273879 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002cc50 con 0x7f63f80cd7a0 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63effff640 1 -- 172.21.15.77:0/1833273879 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0034440 con 0x7f63f80cd7a0 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63ff53e640 1 -- 172.21.15.77:0/1833273879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f80cd7a0 msgr2=0x7f63f80cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63ff53e640 1 --2- 172.21.15.77:0/1833273879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f80cd7a0 0x7f63f80cdbd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009d90 tx=0x7f63f002bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63ff53e640 1 -- 172.21.15.77:0/1833273879 shutdown_connections 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63ff53e640 1 --2- 172.21.15.77:0/1833273879 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f80cd7a0 0x7f63f80cdbd0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63ff53e640 1 --2- 172.21.15.77:0/1833273879 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f81515e0 0x7f63f80cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63ff53e640 1 -- 172.21.15.77:0/1833273879 >> 172.21.15.77:0/1833273879 conn(0x7f63f80cf8a0 msgr2=0x7f63f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63ff53e640 1 -- 172.21.15.77:0/1833273879 shutdown_connections 2024-03-06T07:29:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.677+0000 7f63ff53e640 1 -- 172.21.15.77:0/1833273879 wait complete. 2024-03-06T07:29:43.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.678+0000 7f63ff53e640 1 Processor -- start 2024-03-06T07:29:43.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63ff53e640 1 -- start start 2024-03-06T07:29:43.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63ff53e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f80cd7a0 0x7f63f8150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63ff53e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f81515e0 0x7f63f81f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63ff53e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f815b430 0x7f63f813c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63ff53e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80ce110 con 0x7f63f815b430 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63ff53e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8071310 con 0x7f63f81515e0 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63ff53e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80cc3a0 con 0x7f63f80cd7a0 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63fdadb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f815b430 0x7f63f813c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63fdadb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f815b430 0x7f63f813c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34398/0 (socket says 172.21.15.77:34398) 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63fd2da640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f80cd7a0 0x7f63f8150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63fdadb640 1 -- 172.21.15.77:0/457597552 learned_addr learned my addr 172.21.15.77:0/457597552 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.679+0000 7f63fcad9640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f81515e0 0x7f63f81f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63fd2da640 1 -- 172.21.15.77:0/457597552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f81515e0 msgr2=0x7f63f81f5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63fd2da640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f81515e0 0x7f63f81f5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63fd2da640 1 -- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f815b430 msgr2=0x7f63f813c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63fd2da640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f815b430 0x7f63f813c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63fd2da640 1 -- 172.21.15.77:0/457597552 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f813fc00 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63fdadb640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f815b430 0x7f63f813c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63fd2da640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f80cd7a0 0x7f63f8150590 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f63f400e900 tx=0x7f63f400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63edffb640 1 -- 172.21.15.77:0/457597552 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4017070 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63edffb640 1 -- 172.21.15.77:0/457597552 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400ca00 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f8140150 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63edffb640 1 -- 172.21.15.77:0/457597552 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400cdf0 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.680+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f81f51f0 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.681+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f8068840 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.682+0000 7f63edffb640 1 -- 172.21.15.77:0/457597552 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63f401a070 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.682+0000 7f63edffb640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f63dc06b4d0 0x7f63dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.682+0000 7f63edffb640 1 -- 172.21.15.77:0/457597552 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f63f408a040 con 0x7f63f80cd7a0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.682+0000 7f63fcad9640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f63dc06b4d0 0x7f63dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.683+0000 7f63fcad9640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f63dc06b4d0 0x7f63dc06d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f63f80cceb0 tx=0x7f63e8007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.687+0000 7f63edffb640 1 -- 172.21.15.77:0/457597552 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63f4058830 con 0x7f63f80cd7a0 2024-03-06T07:29:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.898+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63f804f330 con 0x7f63f80cd7a0 2024-03-06T07:29:43.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.899+0000 7f63edffb640 1 -- 172.21.15.77:0/457597552 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7f63f405c8d0 con 0x7f63f80cd7a0 2024-03-06T07:29:43.901 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:43.901 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:43.901 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f63dc06b4d0 msgr2=0x7f63dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f63dc06b4d0 0x7f63dc06d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f63f80cceb0 tx=0x7f63e8007de0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f80cd7a0 msgr2=0x7f63f8150590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f80cd7a0 0x7f63f8150590 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f63f400e900 tx=0x7f63f400edc0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 shutdown_connections 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f63dc06b4d0 0x7f63dc06d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f815b430 0x7f63f813c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f63f81515e0 0x7f63f81f5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 --2- 172.21.15.77:0/457597552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63f80cd7a0 0x7f63f8150590 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 >> 172.21.15.77:0/457597552 conn(0x7f63f80cf8a0 msgr2=0x7f63f814d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 shutdown_connections 2024-03-06T07:29:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:43.901+0000 7f63ff53e640 1 -- 172.21.15.77:0/457597552 wait complete. 2024-03-06T07:29:44.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:44.115 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:45.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:45.212 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:45.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.226+0000 7fa22893e640 1 Processor -- start 2024-03-06T07:29:45.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.226+0000 7fa22893e640 1 -- start start 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.227+0000 7fa22893e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa2240cf850 0x7fa2240cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.227+0000 7fa22893e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa22415e380 con 0x7fa2240d00c0 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.227+0000 7fa22893e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240cdc10 con 0x7fa22406cf20 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.227+0000 7fa22893e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa22406ca90 con 0x7fa2240cf850 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.227+0000 7fa22259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa2240d00c0 0x7fa2240d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45630/0 (socket says 172.21.15.77:45630) 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.227+0000 7fa22259c640 1 -- 172.21.15.77:0/4045502552 learned_addr learned my addr 172.21.15.77:0/4045502552 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.227+0000 7fa221d9b640 1 --2- 172.21.15.77:0/4045502552 >> v2:172.21.15.158:3300/0 conn(0x7fa2240cf850 0x7fa2240cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.228+0000 7fa22159a640 1 -- 172.21.15.77:0/4045502552 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086770340 0 0) 0x7fa22415e380 con 0x7fa2240d00c0 2024-03-06T07:29:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.228+0000 7fa22159a640 1 -- 172.21.15.77:0/4045502552 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c003520 con 0x7fa2240d00c0 2024-03-06T07:29:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.228+0000 7fa22159a640 1 -- 172.21.15.77:0/4045502552 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343698049 0 0) 0x7fa2240cdc10 con 0x7fa22406cf20 2024-03-06T07:29:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.228+0000 7fa22159a640 1 -- 172.21.15.77:0/4045502552 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa22415e380 con 0x7fa22406cf20 2024-03-06T07:29:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.228+0000 7fa221d9b640 1 -- 172.21.15.77:0/4045502552 >> v1:172.21.15.158:6789/0 conn(0x7fa22406cf20 legacy=0x7fa22406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.228+0000 7fa221d9b640 1 -- 172.21.15.77:0/4045502552 >> v1:172.21.15.77:6789/0 conn(0x7fa2240d00c0 legacy=0x7fa2240d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.228+0000 7fa221d9b640 1 -- 172.21.15.77:0/4045502552 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22406c160 con 0x7fa2240cf850 2024-03-06T07:29:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.229+0000 7fa221d9b640 1 --2- 172.21.15.77:0/4045502552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2240cf850 0x7fa2240cd3c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa2180097a0 tx=0x7fa21802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75658677e32a075a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.229+0000 7fa22159a640 1 -- 172.21.15.77:0/4045502552 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218034070 con 0x7fa2240cf850 2024-03-06T07:29:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.229+0000 7fa22159a640 1 -- 172.21.15.77:0/4045502552 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa218030610 con 0x7fa2240cf850 2024-03-06T07:29:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.229+0000 7fa22159a640 1 -- 172.21.15.77:0/4045502552 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2180309c0 con 0x7fa2240cf850 2024-03-06T07:29:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.229+0000 7fa22893e640 1 -- 172.21.15.77:0/4045502552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2240cf850 msgr2=0x7fa2240cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.229+0000 7fa22893e640 1 --2- 172.21.15.77:0/4045502552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2240cf850 0x7fa2240cd3c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa2180097a0 tx=0x7fa21802b960 comp rx=0 tx=0).stop 2024-03-06T07:29:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.230+0000 7fa22893e640 1 -- 172.21.15.77:0/4045502552 shutdown_connections 2024-03-06T07:29:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.230+0000 7fa22893e640 1 --2- 172.21.15.77:0/4045502552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa2240cf850 0x7fa2240cd3c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.230+0000 7fa22893e640 1 -- 172.21.15.77:0/4045502552 >> 172.21.15.77:0/4045502552 conn(0x7fa2240b35a0 msgr2=0x7fa2240b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.230+0000 7fa22893e640 1 -- 172.21.15.77:0/4045502552 shutdown_connections 2024-03-06T07:29:45.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.230+0000 7fa22893e640 1 -- 172.21.15.77:0/4045502552 wait complete. 2024-03-06T07:29:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.231+0000 7fa22893e640 1 Processor -- start 2024-03-06T07:29:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.231+0000 7fa22893e640 1 -- start start 2024-03-06T07:29:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa22893e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22406cf20 0x7fa2240cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa22893e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2240cf850 0x7fa2240c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa22893e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2240d00c0 0x7fa224150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa22893e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240cd900 con 0x7fa2240d00c0 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa22893e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22406c720 con 0x7fa22406cf20 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa22893e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240b1030 con 0x7fa2240cf850 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2240d00c0 0x7fa224150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2240d00c0 0x7fa224150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34414/0 (socket says 172.21.15.77:34414) 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 -- 172.21.15.77:0/2100181460 learned_addr learned my addr 172.21.15.77:0/2100181460 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa22259c640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22406cf20 0x7fa2240cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa221d9b640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2240cf850 0x7fa2240c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 -- 172.21.15.77:0/2100181460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2240cf850 msgr2=0x7fa2240c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2240cf850 0x7fa2240c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 -- 172.21.15.77:0/2100181460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22406cf20 msgr2=0x7fa2240cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22406cf20 0x7fa2240cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 -- 172.21.15.77:0/2100181460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa224151420 con 0x7fa2240d00c0 2024-03-06T07:29:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa222d9d640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2240d00c0 0x7fa224150ee0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00c400 tx=0x7fa21c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa212ffd640 1 -- 172.21.15.77:0/2100181460 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c017070 con 0x7fa2240d00c0 2024-03-06T07:29:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.232+0000 7fa212ffd640 1 -- 172.21.15.77:0/2100181460 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c0052d0 con 0x7fa2240d00c0 2024-03-06T07:29:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.233+0000 7fa212ffd640 1 -- 172.21.15.77:0/2100181460 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c0037f0 con 0x7fa2240d00c0 2024-03-06T07:29:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.233+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2240c5ca0 con 0x7fa2240d00c0 2024-03-06T07:29:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.233+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2240c6260 con 0x7fa2240d00c0 2024-03-06T07:29:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.234+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2240cd510 con 0x7fa2240d00c0 2024-03-06T07:29:45.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.234+0000 7fa212ffd640 1 -- 172.21.15.77:0/2100181460 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa21c01a070 con 0x7fa2240d00c0 2024-03-06T07:29:45.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.235+0000 7fa212ffd640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.235+0000 7fa22259c640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.235+0000 7fa212ffd640 1 -- 172.21.15.77:0/2100181460 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa21c089510 con 0x7fa2240d00c0 2024-03-06T07:29:45.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.235+0000 7fa22259c640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa20406b4d0 0x7fa20406d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa214001560 tx=0x7fa2140019f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.239+0000 7fa212ffd640 1 -- 172.21.15.77:0/2100181460 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2240cd510 con 0x7fa2240d00c0 2024-03-06T07:29:45.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.445+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa22404f330 con 0x7fa2240d00c0 2024-03-06T07:29:45.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.445+0000 7fa212ffd640 1 -- 172.21.15.77:0/2100181460 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7fa21c087e20 con 0x7fa2240d00c0 2024-03-06T07:29:45.448 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:45.448 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:45.448 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa20406b4d0 msgr2=0x7fa20406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa20406b4d0 0x7fa20406d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa214001560 tx=0x7fa2140019f0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2240d00c0 msgr2=0x7fa224150ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2240d00c0 0x7fa224150ee0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00c400 tx=0x7fa21c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 shutdown_connections 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2240d00c0 0x7fa224150ee0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa2240cf850 0x7fa2240c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 --2- 172.21.15.77:0/2100181460 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa22406cf20 0x7fa2240cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.447+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 >> 172.21.15.77:0/2100181460 conn(0x7fa2240b35a0 msgr2=0x7fa2240cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.448+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 shutdown_connections 2024-03-06T07:29:45.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:45.448+0000 7fa22893e640 1 -- 172.21.15.77:0/2100181460 wait complete. 2024-03-06T07:29:45.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:45.662 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:46.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:46.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.773+0000 7efc0dcb5640 1 Processor -- start 2024-03-06T07:29:46.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.773+0000 7efc0dcb5640 1 -- start start 2024-03-06T07:29:46.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc0dcb5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7efc080cce40 0x7efc0815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc0dcb5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc08071f40 con 0x7efc081515e0 2024-03-06T07:29:46.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc0dcb5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc08070a70 con 0x7efc080cd7a0 2024-03-06T07:29:46.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc0dcb5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc080cd2f0 con 0x7efc080cce40 2024-03-06T07:29:46.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc077fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efc081515e0 0x7efc08071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:45646/0 (socket says 172.21.15.77:45646) 2024-03-06T07:29:46.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc077fe640 1 -- 172.21.15.77:0/1762730852 learned_addr learned my addr 172.21.15.77:0/1762730852 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:46.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc06ffd640 1 --2- 172.21.15.77:0/1762730852 >> v2:172.21.15.158:3300/0 conn(0x7efc080cce40 0x7efc0815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc067fc640 1 -- 172.21.15.77:0/1762730852 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865617037 0 0) 0x7efc08071f40 con 0x7efc081515e0 2024-03-06T07:29:46.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc067fc640 1 -- 172.21.15.77:0/1762730852 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf0003520 con 0x7efc081515e0 2024-03-06T07:29:46.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc06ffd640 1 -- 172.21.15.77:0/1762730852 >> v1:172.21.15.158:6789/0 conn(0x7efc080cd7a0 legacy=0x7efc080cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc06ffd640 1 -- 172.21.15.77:0/1762730852 >> v1:172.21.15.77:6789/0 conn(0x7efc081515e0 legacy=0x7efc08071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc067fc640 1 -- 172.21.15.77:0/1762730852 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220864641 0 0) 0x7efc08070a70 con 0x7efc080cd7a0 2024-03-06T07:29:46.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.774+0000 7efc06ffd640 1 -- 172.21.15.77:0/1762730852 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc08070d50 con 0x7efc080cce40 2024-03-06T07:29:46.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.775+0000 7efc06ffd640 1 --2- 172.21.15.77:0/1762730852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc080cce40 0x7efc0815b430 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efbf80097a0 tx=0x7efbf802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=273f4570b8d34f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.775+0000 7efc067fc640 1 -- 172.21.15.77:0/1762730852 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8034070 con 0x7efc080cce40 2024-03-06T07:29:46.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.775+0000 7efc067fc640 1 -- 172.21.15.77:0/1762730852 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf8030610 con 0x7efc080cce40 2024-03-06T07:29:46.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.775+0000 7efc067fc640 1 -- 172.21.15.77:0/1762730852 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf80309e0 con 0x7efc080cce40 2024-03-06T07:29:46.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.776+0000 7efc0dcb5640 1 -- 172.21.15.77:0/1762730852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc080cce40 msgr2=0x7efc0815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.776+0000 7efc0dcb5640 1 --2- 172.21.15.77:0/1762730852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc080cce40 0x7efc0815b430 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efbf80097a0 tx=0x7efbf802b960 comp rx=0 tx=0).stop 2024-03-06T07:29:46.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.776+0000 7efc0dcb5640 1 -- 172.21.15.77:0/1762730852 shutdown_connections 2024-03-06T07:29:46.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.776+0000 7efc0dcb5640 1 --2- 172.21.15.77:0/1762730852 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc080cce40 0x7efc0815b430 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.776+0000 7efc0dcb5640 1 -- 172.21.15.77:0/1762730852 >> 172.21.15.77:0/1762730852 conn(0x7efc080cf8a0 msgr2=0x7efc080cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.776+0000 7efc0dcb5640 1 -- 172.21.15.77:0/1762730852 shutdown_connections 2024-03-06T07:29:46.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.776+0000 7efc0dcb5640 1 -- 172.21.15.77:0/1762730852 wait complete. 2024-03-06T07:29:46.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.777+0000 7efc0dcb5640 1 Processor -- start 2024-03-06T07:29:46.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.777+0000 7efc0dcb5640 1 -- start start 2024-03-06T07:29:46.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc0dcb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc080cce40 0x7efc08156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc0dcb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc080cd7a0 0x7efc0806d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc0dcb5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc081515e0 0x7efc08159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc0dcb5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080cc3a0 con 0x7efc080cce40 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc0dcb5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08071310 con 0x7efc081515e0 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc0dcb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080cc520 con 0x7efc080cd7a0 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc077fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc080cce40 0x7efc08156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc077fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc080cce40 0x7efc08156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34426/0 (socket says 172.21.15.77:34426) 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc077fe640 1 -- 172.21.15.77:0/3988681983 learned_addr learned my addr 172.21.15.77:0/3988681983 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc06ffd640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc080cd7a0 0x7efc0806d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc07fff640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc081515e0 0x7efc08159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc077fe640 1 -- 172.21.15.77:0/3988681983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc080cd7a0 msgr2=0x7efc0806d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc077fe640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc080cd7a0 0x7efc0806d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc077fe640 1 -- 172.21.15.77:0/3988681983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc081515e0 msgr2=0x7efc08159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc077fe640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc081515e0 0x7efc08159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.778+0000 7efc077fe640 1 -- 172.21.15.77:0/3988681983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0806c980 con 0x7efc080cce40 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.779+0000 7efc077fe640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc080cce40 0x7efc08156d70 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0010e0 tx=0x7efbfc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.779+0000 7efbeffff640 1 -- 172.21.15.77:0/3988681983 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc017070 con 0x7efc080cce40 2024-03-06T07:29:46.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.779+0000 7efbeffff640 1 -- 172.21.15.77:0/3988681983 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc005d00 con 0x7efc080cce40 2024-03-06T07:29:46.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.779+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0806cc60 con 0x7efc080cce40 2024-03-06T07:29:46.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.779+0000 7efbeffff640 1 -- 172.21.15.77:0/3988681983 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc007310 con 0x7efc080cce40 2024-03-06T07:29:46.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.779+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0806d220 con 0x7efc080cce40 2024-03-06T07:29:46.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.780+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc08068840 con 0x7efc080cce40 2024-03-06T07:29:46.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.780+0000 7efbeffff640 1 -- 172.21.15.77:0/3988681983 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efbfc01a070 con 0x7efc080cce40 2024-03-06T07:29:46.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.781+0000 7efbeffff640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf406b4d0 0x7efbf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.781+0000 7efc06ffd640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf406b4d0 0x7efbf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.781+0000 7efbeffff640 1 -- 172.21.15.77:0/3988681983 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efbfc089480 con 0x7efc080cce40 2024-03-06T07:29:46.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.781+0000 7efc06ffd640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf406b4d0 0x7efbf406d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7efbf8036000 tx=0x7efbf8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.785+0000 7efbeffff640 1 -- 172.21.15.77:0/3988681983 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efbfc057d00 con 0x7efc080cce40 2024-03-06T07:29:46.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.992+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc0804f330 con 0x7efc080cce40 2024-03-06T07:29:46.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.992+0000 7efbeffff640 1 -- 172.21.15.77:0/3988681983 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3419 (secure 0 0 0) 0x7efbfc05bda0 con 0x7efc080cce40 2024-03-06T07:29:46.994 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:46.994 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:17.720486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8547,"mds_1":8535},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8535":{"gid":8535,"name":"c","rank":1,"incarnation":82,"state":"up:stopping","state_seq":6,"addr":"172.21.15.77:6837/1546132565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1546132565},{"type":"v1","addr":"172.21.15.77:6837","nonce":1546132565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:46.994 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 89 2024-03-06T07:29:46.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf406b4d0 msgr2=0x7efbf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf406b4d0 0x7efbf406d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7efbf8036000 tx=0x7efbf8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:46.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc080cce40 msgr2=0x7efc08156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc080cce40 0x7efc08156d70 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0010e0 tx=0x7efbfc00c890 comp rx=0 tx=0).stop 2024-03-06T07:29:46.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 shutdown_connections 2024-03-06T07:29:46.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efc081515e0 0x7efc08159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7efc080cd7a0 0x7efc0806d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7efbf406b4d0 0x7efbf406d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 --2- 172.21.15.77:0/3988681983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc080cce40 0x7efc08156d70 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 >> 172.21.15.77:0/3988681983 conn(0x7efc080cf8a0 msgr2=0x7efc0815e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 shutdown_connections 2024-03-06T07:29:46.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:46.994+0000 7efc0dcb5640 1 -- 172.21.15.77:0/3988681983 wait complete. 2024-03-06T07:29:47.207 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:17.720486+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8547, 'mds_1': 8535}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8535': {'gid': 8535, 'name': 'c', 'rank': 1, 'incarnation': 82, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.77:6837/1546132565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 1546132565}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1546132565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:47.208 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8535:up:stopping 2024-03-06T07:29:47.214 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:48.103 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:29:48.208 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:48.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.316+0000 7f031f44e640 1 Processor -- start 2024-03-06T07:29:48.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.316+0000 7f031f44e640 1 -- start start 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031f44e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f03181515e0 0x7f03180cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031f44e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03180cd7a0 0x7f03180cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031f44e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0318071f40 con 0x7f031815b430 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031f44e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031815b8e0 con 0x7f03180cd7a0 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031f44e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0318070a70 con 0x7f03181515e0 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031d9eb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03180cd7a0 0x7f03180cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031d9eb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03180cd7a0 0x7f03180cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34462/0 (socket says 172.21.15.77:34462) 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031d1ea640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f03181515e0 0x7f03180cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031d9eb640 1 -- 172.21.15.77:0/2949891767 learned_addr learned my addr 172.21.15.77:0/2949891767 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:48.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031d9eb640 1 -- 172.21.15.77:0/2949891767 >> v2:172.21.15.158:3300/0 conn(0x7f03181515e0 msgr2=0x7f03180cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031d9eb640 1 --2- 172.21.15.77:0/2949891767 >> v2:172.21.15.158:3300/0 conn(0x7f03181515e0 0x7f03180cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031d9eb640 1 -- 172.21.15.77:0/2949891767 >> v1:172.21.15.158:6789/0 conn(0x7f031815b430 legacy=0x7f0318071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.317+0000 7f031d9eb640 1 -- 172.21.15.77:0/2949891767 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0318070d50 con 0x7f03180cd7a0 2024-03-06T07:29:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.318+0000 7f031d9eb640 1 --2- 172.21.15.77:0/2949891767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f03180cdb80 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f031806eb50 tx=0x7f031000a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9e5b4a447e437d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.318+0000 7f030ffff640 1 -- 172.21.15.77:0/2949891767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031002c910 con 0x7f03180cd7a0 2024-03-06T07:29:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.318+0000 7f030ffff640 1 -- 172.21.15.77:0/2949891767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031002cab0 con 0x7f03180cd7a0 2024-03-06T07:29:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.318+0000 7f030ffff640 1 -- 172.21.15.77:0/2949891767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031002ce60 con 0x7f03180cd7a0 2024-03-06T07:29:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.319+0000 7f031f44e640 1 -- 172.21.15.77:0/2949891767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 msgr2=0x7f03180cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.319+0000 7f031f44e640 1 --2- 172.21.15.77:0/2949891767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f03180cdb80 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f031806eb50 tx=0x7f031000a870 comp rx=0 tx=0).stop 2024-03-06T07:29:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.319+0000 7f031f44e640 1 -- 172.21.15.77:0/2949891767 shutdown_connections 2024-03-06T07:29:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.319+0000 7f031f44e640 1 --2- 172.21.15.77:0/2949891767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f03180cdb80 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.319+0000 7f031f44e640 1 --2- 172.21.15.77:0/2949891767 >> v2:172.21.15.158:3300/0 conn(0x7f03181515e0 0x7f03180cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.319+0000 7f031f44e640 1 -- 172.21.15.77:0/2949891767 >> 172.21.15.77:0/2949891767 conn(0x7f03180cf8a0 msgr2=0x7f03180cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.319+0000 7f031f44e640 1 -- 172.21.15.77:0/2949891767 shutdown_connections 2024-03-06T07:29:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.319+0000 7f031f44e640 1 -- 172.21.15.77:0/2949891767 wait complete. 2024-03-06T07:29:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.320+0000 7f031f44e640 1 Processor -- start 2024-03-06T07:29:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.320+0000 7f031f44e640 1 -- start start 2024-03-06T07:29:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031f44e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f031814c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031f44e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03181515e0 0x7f03181f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031f44e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f031815b430 0x7f03181f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031f44e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03180ce0c0 con 0x7f03180cd7a0 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031f44e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0318071310 con 0x7f031815b430 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031f44e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03180cc3a0 con 0x7f03181515e0 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f031814c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f031814c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34466/0 (socket says 172.21.15.77:34466) 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 -- 172.21.15.77:0/3769421432 learned_addr learned my addr 172.21.15.77:0/3769421432 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d9eb640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f031815b430 0x7f03181f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031c9e9640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03181515e0 0x7f03181f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 -- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03181515e0 msgr2=0x7f03181f2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03181515e0 0x7f03181f2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 -- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f031815b430 msgr2=0x7f03181f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f031815b430 0x7f03181f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 -- 172.21.15.77:0/3769421432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031814d1c0 con 0x7f03180cd7a0 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031c9e9640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03181515e0 0x7f03181f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031d1ea640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f031814c600 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f0314010dc0 tx=0x7f031400e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f030dffb640 1 -- 172.21.15.77:0/3769421432 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031401e070 con 0x7f03180cd7a0 2024-03-06T07:29:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f030dffb640 1 -- 172.21.15.77:0/3769421432 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0314011860 con 0x7f03180cd7a0 2024-03-06T07:29:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.321+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03181f3710 con 0x7f03180cd7a0 2024-03-06T07:29:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.322+0000 7f030dffb640 1 -- 172.21.15.77:0/3769421432 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314011c50 con 0x7f03180cd7a0 2024-03-06T07:29:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.322+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0318153300 con 0x7f03180cd7a0 2024-03-06T07:29:48.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.323+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0318068840 con 0x7f03180cd7a0 2024-03-06T07:29:48.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.323+0000 7f030dffb640 1 -- 172.21.15.77:0/3769421432 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0314021070 con 0x7f03180cd7a0 2024-03-06T07:29:48.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.324+0000 7f030dffb640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02fc06b500 0x7f02fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.324+0000 7f030dffb640 1 -- 172.21.15.77:0/3769421432 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f031408cdb0 con 0x7f03180cd7a0 2024-03-06T07:29:48.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.324+0000 7f031c9e9640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02fc06b500 0x7f02fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.324+0000 7f031c9e9640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02fc06b500 0x7f02fc06d9c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0308007fd0 tx=0x7f0308009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.328+0000 7f030dffb640 1 -- 172.21.15.77:0/3769421432 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f031405b5a0 con 0x7f03180cd7a0 2024-03-06T07:29:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.533+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f031804f330 con 0x7f03180cd7a0 2024-03-06T07:29:48.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.534+0000 7f030dffb640 1 -- 172.21.15.77:0/3769421432 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2612 (secure 0 0 0) 0x7f031405f640 con 0x7f03180cd7a0 2024-03-06T07:29:48.536 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:48.536 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:47.721340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:48.536 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 90 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02fc06b500 msgr2=0x7f02fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02fc06b500 0x7f02fc06d9c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0308007fd0 tx=0x7f0308009040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 msgr2=0x7f031814c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f031814c600 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f0314010dc0 tx=0x7f031400e730 comp rx=0 tx=0).stop 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 shutdown_connections 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f031815b430 0x7f03181f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f03181515e0 0x7f03181f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f02fc06b500 0x7f02fc06d9c0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 --2- 172.21.15.77:0/3769421432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03180cd7a0 0x7f031814c600 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 >> 172.21.15.77:0/3769421432 conn(0x7f03180cf8a0 msgr2=0x7f031814d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 shutdown_connections 2024-03-06T07:29:48.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.536+0000 7f031f44e640 1 -- 172.21.15.77:0/3769421432 wait complete. 2024-03-06T07:29:48.750 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:32.114016+0000', 'modified': '2024-03-06T07:29:47.721340+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8547}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_8547': {'gid': 8547, 'name': 'b', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6835/1245424942', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6833', 'nonce': 1245424942}, {'type': 'v1', 'addr': '172.21.15.158:6835', 'nonce': 1245424942}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8547, 'qdb_cluster': [8547]} 2024-03-06T07:29:48.750 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:48.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:29:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.852+0000 7fb46c91e640 1 Processor -- start 2024-03-06T07:29:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.852+0000 7fb46c91e640 1 -- start start 2024-03-06T07:29:48.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46c91e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468071e00 0x7fb4680cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46c91e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb4680cd7c0 0x7fb4680cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46c91e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb468072220 con 0x7fb46815bbf0 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46c91e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4680ce170 con 0x7fb4680cd7c0 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46c91e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680cf100 con 0x7fb468071e00 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb46815bbf0 0x7fb46815bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56596/0 (socket says 172.21.15.77:56596) 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46659c640 1 -- 172.21.15.77:0/2315211996 learned_addr learned my addr 172.21.15.77:0/2315211996 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb466d9d640 1 --2- 172.21.15.77:0/2315211996 >> v2:172.21.15.158:3300/0 conn(0x7fb4680cd7c0 0x7fb4680cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb465d9b640 1 --2- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468071e00 0x7fb4680cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46559a640 1 -- 172.21.15.77:0/2315211996 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768570322 0 0) 0x7fb468072220 con 0x7fb46815bbf0 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.853+0000 7fb46559a640 1 -- 172.21.15.77:0/2315211996 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb450003520 con 0x7fb46815bbf0 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.854+0000 7fb466d9d640 1 -- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468071e00 msgr2=0x7fb4680cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.854+0000 7fb466d9d640 1 --2- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468071e00 0x7fb4680cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.854+0000 7fb466d9d640 1 -- 172.21.15.77:0/2315211996 >> v1:172.21.15.77:6789/0 conn(0x7fb46815bbf0 legacy=0x7fb46815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.854+0000 7fb466d9d640 1 -- 172.21.15.77:0/2315211996 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb468072490 con 0x7fb4680cd7c0 2024-03-06T07:29:48.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.854+0000 7fb465d9b640 1 --2- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468071e00 0x7fb4680cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:48.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.854+0000 7fb466d9d640 1 --2- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cd7c0 0x7fb4680cdc30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb4600097d0 tx=0x7fb46002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dde190270ffb5601 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46559a640 1 -- 172.21.15.77:0/2315211996 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460035020 con 0x7fb4680cd7c0 2024-03-06T07:29:48.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46559a640 1 -- 172.21.15.77:0/2315211996 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb460031800 con 0x7fb4680cd7c0 2024-03-06T07:29:48.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46559a640 1 -- 172.21.15.77:0/2315211996 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460031bb0 con 0x7fb4680cd7c0 2024-03-06T07:29:48.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46c91e640 1 -- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cd7c0 msgr2=0x7fb4680cdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cd7c0 0x7fb4680cdc30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb4600097d0 tx=0x7fb46002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46c91e640 1 -- 172.21.15.77:0/2315211996 shutdown_connections 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb4680cd7c0 0x7fb4680cdc30 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2315211996 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb468071e00 0x7fb4680cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46c91e640 1 -- 172.21.15.77:0/2315211996 >> 172.21.15.77:0/2315211996 conn(0x7fb4680b1c80 msgr2=0x7fb4680cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46c91e640 1 -- 172.21.15.77:0/2315211996 shutdown_connections 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.855+0000 7fb46c91e640 1 -- 172.21.15.77:0/2315211996 wait complete. 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.856+0000 7fb46c91e640 1 Processor -- start 2024-03-06T07:29:48.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46c91e640 1 -- start start 2024-03-06T07:29:48.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46c91e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468071e00 0x7fb468151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46c91e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680cd7c0 0x7fb4681401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46c91e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46815bbf0 0x7fb468150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46c91e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680d0650 con 0x7fb468071e00 2024-03-06T07:29:48.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46c91e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680d0370 con 0x7fb46815bbf0 2024-03-06T07:29:48.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46c91e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46815c6e0 con 0x7fb4680cd7c0 2024-03-06T07:29:48.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468071e00 0x7fb468151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468071e00 0x7fb468151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41346/0 (socket says 172.21.15.77:41346) 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb46659c640 1 -- 172.21.15.77:0/2291149676 learned_addr learned my addr 172.21.15.77:0/2291149676 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb465d9b640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680cd7c0 0x7fb4681401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.857+0000 7fb466d9d640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46815bbf0 0x7fb468150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb46659c640 1 -- 172.21.15.77:0/2291149676 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680cd7c0 msgr2=0x7fb4681401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb46659c640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680cd7c0 0x7fb4681401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb46659c640 1 -- 172.21.15.77:0/2291149676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46815bbf0 msgr2=0x7fb468150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb46659c640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46815bbf0 0x7fb468150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb46659c640 1 -- 172.21.15.77:0/2291149676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46814f5d0 con 0x7fb468071e00 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb46659c640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468071e00 0x7fb468151d30 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fb458001460 tx=0x7fb45800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb456ffd640 1 -- 172.21.15.77:0/2291149676 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458017070 con 0x7fb468071e00 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb456ffd640 1 -- 172.21.15.77:0/2291149676 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45800ac00 con 0x7fb468071e00 2024-03-06T07:29:48.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb46c91e640 1 -- 172.21.15.77:0/2291149676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46814f8b0 con 0x7fb468071e00 2024-03-06T07:29:48.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb456ffd640 1 -- 172.21.15.77:0/2291149676 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458012640 con 0x7fb468071e00 2024-03-06T07:29:48.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.858+0000 7fb46c91e640 1 -- 172.21.15.77:0/2291149676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46814fe70 con 0x7fb468071e00 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.860+0000 7fb456ffd640 1 -- 172.21.15.77:0/2291149676 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb45801a070 con 0x7fb468071e00 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.860+0000 7fb456ffd640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb44806b4d0 0x7fb44806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.860+0000 7fb456ffd640 1 -- 172.21.15.77:0/2291149676 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb458089ef0 con 0x7fb468071e00 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.860+0000 7fb465d9b640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb44806b4d0 0x7fb44806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.861+0000 7fb465d9b640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb44806b4d0 0x7fb44806d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009560 tx=0x7fb45c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.861+0000 7fb46f5e3740 1 Processor -- start 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.861+0000 7fb46f5e3740 1 -- start start 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc0870c870 0x55fc0870cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc087108c0 0x55fc08712360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc0873ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc086766b0 con 0x55fc087108c0 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0861fa60 con 0x55fc0873c600 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc08672750 con 0x55fc0870c870 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc087108c0 0x55fc08712360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc087108c0 0x55fc08712360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41348/0 (socket says 172.21.15.77:41348) 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 -- 172.21.15.77:0/66474045 learned_addr learned my addr 172.21.15.77:0/66474045 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb46659c640 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc0870c870 0x55fc0870cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb466d9d640 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc0873ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 -- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc0870c870 msgr2=0x55fc0870cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc0870c870 0x55fc0870cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 -- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 msgr2=0x55fc0873ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc0873ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 -- 172.21.15.77:0/66474045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc0863df30 con 0x55fc087108c0 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb466d9d640 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc0873ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.862+0000 7fb465d9b640 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc087108c0 0x55fc08712360 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fb45c011e90 tx=0x7fb45c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fca6748035cb931 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb447fff640 1 -- 172.21.15.77:0/66474045 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c01b070 con 0x55fc087108c0 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb447fff640 1 -- 172.21.15.77:0/66474045 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c027070 con 0x55fc087108c0 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb447fff640 1 -- 172.21.15.77:0/66474045 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c003a20 con 0x55fc087108c0 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 -- 172.21.15.77:0/66474045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc087108c0 msgr2=0x55fc08712360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc087108c0 0x55fc08712360 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fb45c011e90 tx=0x7fb45c014000 comp rx=0 tx=0).stop 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 -- 172.21.15.77:0/66474045 shutdown_connections 2024-03-06T07:29:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc087108c0 0x55fc08712360 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc0873ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/66474045 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc0870c870 0x55fc0870cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 -- 172.21.15.77:0/66474045 >> 172.21.15.77:0/66474045 conn(0x55fc0870e8f0 msgr2=0x55fc087101b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 -- 172.21.15.77:0/66474045 shutdown_connections 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.863+0000 7fb46f5e3740 1 -- 172.21.15.77:0/66474045 wait complete. 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.864+0000 7fb46f5e3740 1 -- start start 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.864+0000 7fb46f5e3740 1 Processor -- start 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc08740350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08731a80 0x55fc087680a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc08737370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc08710d50 con 0x55fc08731400 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc08632ea0 con 0x55fc0873c600 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc08711030 con 0x55fc08731a80 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46f5e3740 10 client.0 _ll_register_callbacks cb 0x55fc08632870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc08737370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc08737370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41356/0 (socket says 172.21.15.77:41356) 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 -- 172.21.15.77:0/3604812212 learned_addr learned my addr 172.21.15.77:0/3604812212 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46659c640 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc08740350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb465d9b640 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08731a80 0x55fc087680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 -- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08731a80 msgr2=0x55fc087680a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08731a80 0x55fc087680a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 -- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 msgr2=0x55fc08740350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc08740350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 -- 172.21.15.77:0/3604812212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc08633270 con 0x55fc08731400 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb46659c640 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc08740350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb466d9d640 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc08737370 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fb460009720 tx=0x7fb46002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7780263f202727 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.865+0000 7fb4457fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:29:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb4457fa640 1 -- 172.21.15.77:0/3604812212 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46004a060 con 0x55fc08731400 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb4457fa640 1 -- 172.21.15.77:0/3604812212 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb46003b840 con 0x55fc08731400 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 1 -- 172.21.15.77:0/3604812212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb45c027070 con 0x55fc08731400 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb4457fa640 1 -- 172.21.15.77:0/3604812212 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb460050050 con 0x55fc08731400 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 10 client.8991 fetch_fsmap learned FSMap version 90 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 1 -- 172.21.15.77:0/3604812212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55fc0838b2d0 con 0x55fc08731400 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb4457fa640 1 -- 172.21.15.77:0/3604812212 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c027070 con 0x55fc08731400 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb4457fa640 1 -- 172.21.15.77:0/3604812212 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 90) v1 ==== 1597+0+0 (secure 0 0 0) 0x7fb460053020 con 0x55fc08731400 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 10 client.8991 fetch_fsmap finished waiting for FSMap version 90 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 10 client.8991 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 20 client.8991 populate_metadata read hostname 'smithi077' 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55fc08734750 0x55fc08711240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 4 client.8991 mds_command: new command op to 8547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46f5e3740 1 -- 172.21.15.77:0/3604812212 --> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fc08683bf0 con 0x55fc08734750 2024-03-06T07:29:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.866+0000 7fb46659c640 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55fc08734750 0x55fc08711240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.867+0000 7fb4457fa640 10 client.8991 ms_handle_connect on v2:172.21.15.158:6833/1245424942 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.867+0000 7fb46659c640 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55fc08734750 0x55fc08711240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d24a78aee555132 server_cookie=34ac34fe58e64665 in_seq=0 out_seq=0 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.867+0000 7fb4457fa640 1 -- 172.21.15.77:0/3604812212 <== mds.0 v2:172.21.15.158:6833/1245424942 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fc08683bf0 con 0x55fc08734750 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.867+0000 7fb4457fa640 10 client.8991 handle_command_reply: tid=0 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 client.8991 shutdown 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 -- 172.21.15.77:0/3604812212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 msgr2=0x55fc08737370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc08737370 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fb460009720 tx=0x7fb46002e000 comp rx=0 tx=0).stop 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 -- 172.21.15.77:0/3604812212 shutdown_connections 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55fc08734750 0x55fc08711240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc08737370 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08731a80 0x55fc087680a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb4457fa640 0 client.8991 ms_handle_reset on v2:172.21.15.158:6833/1245424942 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/3604812212 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc0873c600 0x55fc08740350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 -- 172.21.15.77:0/3604812212 >> 172.21.15.77:0/3604812212 conn(0x55fc0870c870 msgr2=0x55fc0870b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 -- 172.21.15.77:0/3604812212 shutdown_connections 2024-03-06T07:29:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 1 -- 172.21.15.77:0/3604812212 wait complete. 2024-03-06T07:29:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.868+0000 7fb46f5e3740 20 client.8991 trim_cache size 0 max 16384 2024-03-06T07:29:48.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.884+0000 7fb46f5e3740 1 Processor -- start 2024-03-06T07:29:48.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.884+0000 7fb46f5e3740 1 -- start start 2024-03-06T07:29:48.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc0873c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc08771bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc08735c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb468152f10 con 0x55fc08731400 2024-03-06T07:29:48.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc08733ac0 con 0x55fc08731a80 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc0873d690 con 0x55fc08734750 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc0873c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc0873c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41366/0 (socket says 172.21.15.77:41366) 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb46659c640 1 -- 172.21.15.77:0/788595704 learned_addr learned my addr 172.21.15.77:0/788595704 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb466d9d640 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc08735c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.885+0000 7fb465d9b640 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc08771bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46659c640 1 -- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 msgr2=0x55fc08735c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46659c640 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc08735c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46659c640 1 -- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 msgr2=0x55fc08771bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46659c640 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc08771bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46659c640 1 -- 172.21.15.77:0/788595704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc087132a0 con 0x55fc08731400 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb465d9b640 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc08771bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb466d9d640 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc08735c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46659c640 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc0873c9e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb45800afd0 tx=0x7fb45805b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8402b03008f1d0d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb4477fe640 1 -- 172.21.15.77:0/788595704 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458086070 con 0x55fc08731400 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb4477fe640 1 -- 172.21.15.77:0/788595704 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4580569e0 con 0x55fc08731400 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb4477fe640 1 -- 172.21.15.77:0/788595704 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458060ab0 con 0x55fc08731400 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46f5e3740 1 -- 172.21.15.77:0/788595704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 msgr2=0x55fc0873c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc0873c9e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb45800afd0 tx=0x7fb45805b040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46f5e3740 1 -- 172.21.15.77:0/788595704 shutdown_connections 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc08735c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc08771bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/788595704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc0873c9e0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46f5e3740 1 -- 172.21.15.77:0/788595704 >> 172.21.15.77:0/788595704 conn(0x55fc0870c870 msgr2=0x55fc08739500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.886+0000 7fb46f5e3740 1 -- 172.21.15.77:0/788595704 shutdown_connections 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.887+0000 7fb46f5e3740 1 -- 172.21.15.77:0/788595704 wait complete. 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.887+0000 7fb46f5e3740 1 -- start start 2024-03-06T07:29:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.887+0000 7fb46f5e3740 1 Processor -- start 2024-03-06T07:29:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc087680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc087c6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46f5e3740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc0873b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0877bc40 con 0x55fc08731400 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc0867b930 con 0x55fc08731a80 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46f5e3740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc08708610 con 0x55fc08734750 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46f5e3740 10 client.0 _ll_register_callbacks cb 0x7fb4600319c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc087680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc087680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41378/0 (socket says 172.21.15.77:41378) 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 -- 172.21.15.77:0/1055119728 learned_addr learned my addr 172.21.15.77:0/1055119728 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb466d9d640 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc0873b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb465d9b640 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc087c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 -- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 msgr2=0x55fc0873b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc0873b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 -- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 msgr2=0x55fc087c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc087c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 -- 172.21.15.77:0/1055119728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc084556b0 con 0x55fc08731400 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb465d9b640 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc087c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb46659c640 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc087680f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb458001580 tx=0x7fb4580882b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178c28190708bc08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.888+0000 7fb4457fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb4457fa640 1 -- 172.21.15.77:0/1055119728 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458086070 con 0x55fc08731400 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb4457fa640 1 -- 172.21.15.77:0/1055119728 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb458004b60 con 0x55fc08731400 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 1 -- 172.21.15.77:0/1055119728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4580569e0 con 0x55fc08731400 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb4457fa640 1 -- 172.21.15.77:0/1055119728 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb458004da0 con 0x55fc08731400 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 10 client.8997 fetch_fsmap learned FSMap version 90 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 1 -- 172.21.15.77:0/1055119728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55fc08732040 con 0x55fc08731400 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb4457fa640 1 -- 172.21.15.77:0/1055119728 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4580569e0 con 0x55fc08731400 2024-03-06T07:29:48.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb4457fa640 1 -- 172.21.15.77:0/1055119728 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 90) v1 ==== 1597+0+0 (secure 0 0 0) 0x7fb458057b30 con 0x55fc08731400 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 10 client.8997 fetch_fsmap finished waiting for FSMap version 90 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 10 client.8997 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 20 client.8997 populate_metadata read hostname 'smithi077' 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55fc0873c600 0x55fc0873b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 4 client.8997 mds_command: new command op to 8547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb46f5e3740 1 -- 172.21.15.77:0/1055119728 --> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fc0867df40 con 0x55fc0873c600 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb465d9b640 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55fc0873c600 0x55fc0873b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb4457fa640 10 client.8997 ms_handle_connect on v2:172.21.15.158:6833/1245424942 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.889+0000 7fb465d9b640 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55fc0873c600 0x55fc0873b840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35c5f987d9eb73c5 server_cookie=f405d3a084fe1a56 in_seq=0 out_seq=0 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb4457fa640 1 -- 172.21.15.77:0/1055119728 <== mds.0 v2:172.21.15.158:6833/1245424942 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55fc0867df40 con 0x55fc0873c600 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb4457fa640 10 client.8997 handle_command_reply: tid=0 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 client.8997 shutdown 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 -- 172.21.15.77:0/1055119728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 msgr2=0x55fc087680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc087680f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb458001580 tx=0x7fb4580882b0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 -- 172.21.15.77:0/1055119728 shutdown_connections 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:6833/1245424942,v1:172.21.15.158:6835/1245424942] conn(0x55fc0873c600 0x55fc0873b840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x55fc08734750 0x55fc0873b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb4457fa640 0 client.8997 ms_handle_reset on v2:172.21.15.158:6833/1245424942 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x55fc08731a80 0x55fc087c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 --2- 172.21.15.77:0/1055119728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fc08731400 0x55fc087680f0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 -- 172.21.15.77:0/1055119728 >> 172.21.15.77:0/1055119728 conn(0x55fc0870c870 msgr2=0x55fc0873c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 -- 172.21.15.77:0/1055119728 shutdown_connections 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 1 -- 172.21.15.77:0/1055119728 wait complete. 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46f5e3740 20 client.8997 trim_cache size 0 max 16384 2024-03-06T07:29:48.899 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 8547, 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 76.092567951000007, 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 90, 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 129, 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 129, 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 81.166866326999994 2024-03-06T07:29:48.900 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:29:48.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46c91e640 1 -- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb44806b4d0 msgr2=0x7fb44806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb44806b4d0 0x7fb44806d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009560 tx=0x7fb45c009190 comp rx=0 tx=0).stop 2024-03-06T07:29:48.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46c91e640 1 -- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468071e00 msgr2=0x7fb468151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.890+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468071e00 0x7fb468151d30 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fb458001460 tx=0x7fb45800cf70 comp rx=0 tx=0).stop 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.891+0000 7fb46c91e640 1 -- 172.21.15.77:0/2291149676 shutdown_connections 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.891+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb46815bbf0 0x7fb468150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.891+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb4680cd7c0 0x7fb4681401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.891+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb44806b4d0 0x7fb44806d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.891+0000 7fb46c91e640 1 --2- 172.21.15.77:0/2291149676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb468071e00 0x7fb468151d30 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.891+0000 7fb46c91e640 1 -- 172.21.15.77:0/2291149676 >> 172.21.15.77:0/2291149676 conn(0x7fb4680b1c80 msgr2=0x7fb4681579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.891+0000 7fb46c91e640 1 -- 172.21.15.77:0/2291149676 shutdown_connections 2024-03-06T07:29:48.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48.891+0000 7fb46c91e640 1 -- 172.21.15.77:0/2291149676 wait complete. 2024-03-06T07:29:48.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:48 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:29:49.216 INFO:teuthology.orchestra.run.smithi077.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:49.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:49.226 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:49.226 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:49.308 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:49.352 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:49.353 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:29:49.353 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:49.368 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:49.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.506+0000 7f9d0b565640 1 Processor -- start 2024-03-06T07:29:49.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.506+0000 7f9d0b565640 1 -- start start 2024-03-06T07:29:49.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d0b565640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d0415dfe0 0x7f9d040cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d0b565640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9d040d29c0 0x7f9d040b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d0b565640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d040b3430 con 0x7f9d040cf830 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d0b565640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d040b31e0 con 0x7f9d0415dfe0 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d0b565640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d040b3d60 con 0x7f9d040d29c0 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d09301640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d0415dfe0 0x7f9d040cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d09301640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9d0415dfe0 0x7f9d040cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41386/0 (socket says 172.21.15.77:41386) 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d09301640 1 -- 172.21.15.77:0/3990695593 learned_addr learned my addr 172.21.15.77:0/3990695593 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.507+0000 7f9d09b02640 1 --2- 172.21.15.77:0/3990695593 >> v2:172.21.15.158:3300/0 conn(0x7f9d040d29c0 0x7f9d040b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.508+0000 7f9d09301640 1 -- 172.21.15.77:0/3990695593 >> v2:172.21.15.158:3300/0 conn(0x7f9d040d29c0 msgr2=0x7f9d040b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.508+0000 7f9d09301640 1 --2- 172.21.15.77:0/3990695593 >> v2:172.21.15.158:3300/0 conn(0x7f9d040d29c0 0x7f9d040b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.508+0000 7f9d09301640 1 -- 172.21.15.77:0/3990695593 >> v1:172.21.15.158:6789/0 conn(0x7f9d040cf830 legacy=0x7f9d040d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.508+0000 7f9d09301640 1 -- 172.21.15.77:0/3990695593 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d040b3710 con 0x7f9d0415dfe0 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.508+0000 7f9d09301640 1 --2- 172.21.15.77:0/3990695593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0415dfe0 0x7f9d040cf2f0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f9d040b3cd0 tx=0x7f9d0002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39867ab5f1ab58ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.508+0000 7f9cfbfff640 1 -- 172.21.15.77:0/3990695593 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0002cb20 con 0x7f9d0415dfe0 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.508+0000 7f9cfbfff640 1 -- 172.21.15.77:0/3990695593 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0002ccc0 con 0x7f9d0415dfe0 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.508+0000 7f9cfbfff640 1 -- 172.21.15.77:0/3990695593 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00034610 con 0x7f9d0415dfe0 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.509+0000 7f9d0b565640 1 -- 172.21.15.77:0/3990695593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0415dfe0 msgr2=0x7f9d040cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.509+0000 7f9d0b565640 1 --2- 172.21.15.77:0/3990695593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0415dfe0 0x7f9d040cf2f0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f9d040b3cd0 tx=0x7f9d0002bf80 comp rx=0 tx=0).stop 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.509+0000 7f9d0b565640 1 -- 172.21.15.77:0/3990695593 shutdown_connections 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.509+0000 7f9d0b565640 1 --2- 172.21.15.77:0/3990695593 >> v2:172.21.15.158:3300/0 conn(0x7f9d040d29c0 0x7f9d040b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.509+0000 7f9d0b565640 1 --2- 172.21.15.77:0/3990695593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0415dfe0 0x7f9d040cf2f0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.509+0000 7f9d0b565640 1 -- 172.21.15.77:0/3990695593 >> 172.21.15.77:0/3990695593 conn(0x7f9d040d3cd0 msgr2=0x7f9d040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.509+0000 7f9d0b565640 1 -- 172.21.15.77:0/3990695593 shutdown_connections 2024-03-06T07:29:49.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.509+0000 7f9d0b565640 1 -- 172.21.15.77:0/3990695593 wait complete. 2024-03-06T07:29:49.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.510+0000 7f9d0b565640 1 Processor -- start 2024-03-06T07:29:49.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.510+0000 7f9d0b565640 1 -- start start 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d0b565640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d040cf830 0x7f9d040cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d0b565640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d040d29c0 0x7f9d0409dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d0b565640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d0415dfe0 0x7f9d040cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d0b565640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d1df0 con 0x7f9d040d29c0 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d0b565640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040d1c70 con 0x7f9d040cf830 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d0b565640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0415e3c0 con 0x7f9d0415dfe0 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d08b00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d040d29c0 0x7f9d0409dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d08b00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d040d29c0 0x7f9d0409dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41390/0 (socket says 172.21.15.77:41390) 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d08b00640 1 -- 172.21.15.77:0/2141007602 learned_addr learned my addr 172.21.15.77:0/2141007602 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d09301640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d040cf830 0x7f9d040cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d09b02640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d0415dfe0 0x7f9d040cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d08b00640 1 -- 172.21.15.77:0/2141007602 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d0415dfe0 msgr2=0x7f9d040cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d08b00640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d0415dfe0 0x7f9d040cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d08b00640 1 -- 172.21.15.77:0/2141007602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d040cf830 msgr2=0x7f9d040cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d08b00640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d040cf830 0x7f9d040cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.511+0000 7f9d08b00640 1 -- 172.21.15.77:0/2141007602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0415b740 con 0x7f9d040d29c0 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.512+0000 7f9d08b00640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d040d29c0 0x7f9d0409dad0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400c470 tx=0x7f9cf4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.512+0000 7f9cf9ffb640 1 -- 172.21.15.77:0/2141007602 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf40052d0 con 0x7f9d040d29c0 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.512+0000 7f9cf9ffb640 1 -- 172.21.15.77:0/2141007602 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4005470 con 0x7f9d040d29c0 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.512+0000 7f9cf9ffb640 1 -- 172.21.15.77:0/2141007602 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4007260 con 0x7f9d040d29c0 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.512+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0415fb60 con 0x7f9d040d29c0 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.512+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d04160120 con 0x7f9d040d29c0 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.513+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0404f330 con 0x7f9d040d29c0 2024-03-06T07:29:49.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.513+0000 7f9cf9ffb640 1 -- 172.21.15.77:0/2141007602 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cf4007400 con 0x7f9d040d29c0 2024-03-06T07:29:49.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.513+0000 7f9cf9ffb640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ce806b4d0 0x7f9ce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.514+0000 7f9cf9ffb640 1 -- 172.21.15.77:0/2141007602 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9cf4089720 con 0x7f9d040d29c0 2024-03-06T07:29:49.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.514+0000 7f9d09301640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ce806b4d0 0x7f9ce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.514+0000 7f9d09301640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ce806b4d0 0x7f9ce806d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f9d00035000 tx=0x7f9d00030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.518+0000 7f9cf9ffb640 1 -- 172.21.15.77:0/2141007602 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d0415b740 con 0x7f9d040d29c0 2024-03-06T07:29:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.691+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d0406a1b0 con 0x7f9d040d29c0 2024-03-06T07:29:49.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.692+0000 7f9cf9ffb640 1 -- 172.21.15.77:0/2141007602 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16764 (secure 0 0 0) 0x7f9cf4057e90 con 0x7f9d040d29c0 2024-03-06T07:29:49.694 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:49.694 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":129,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:51.262074+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:30.343353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:31.094099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.9","mappings":[{"from":2,"to":0}]},{"pgid":"19.12","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ce806b4d0 msgr2=0x7f9ce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ce806b4d0 0x7f9ce806d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f9d00035000 tx=0x7f9d00030000 comp rx=0 tx=0).stop 2024-03-06T07:29:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d040d29c0 msgr2=0x7f9d0409dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d040d29c0 0x7f9d0409dad0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400c470 tx=0x7f9cf4001e70 comp rx=0 tx=0).stop 2024-03-06T07:29:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 shutdown_connections 2024-03-06T07:29:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ce806b4d0 0x7f9ce806d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d040d29c0 0x7f9d0409dad0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9d0415dfe0 0x7f9d040cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 --2- 172.21.15.77:0/2141007602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9d040cf830 0x7f9d040cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 >> 172.21.15.77:0/2141007602 conn(0x7f9d040d3cd0 msgr2=0x7f9d040cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 shutdown_connections 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.694+0000 7f9d0b565640 1 -- 172.21.15.77:0/2141007602 wait complete. 2024-03-06T07:29:49.707 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:30.343353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:31.094099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:49.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:49.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.808+0000 7f1c5fa0e640 1 Processor -- start 2024-03-06T07:29:49.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.808+0000 7f1c5fa0e640 1 -- start start 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5fa0e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1c580cfa90 0x7f1c580cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5fa0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c580b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5fa0e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c58158ac0 con 0x7f1c581586e0 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5fa0e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c5815b2a0 con 0x7f1c580cfa90 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5fa0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5815b540 con 0x7f1c580b33a0 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5d7aa640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1c581586e0 0x7f1c580aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56604/0 (socket says 172.21.15.77:56604) 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5d7aa640 1 -- 172.21.15.77:0/3120200404 learned_addr learned my addr 172.21.15.77:0/3120200404 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5dfab640 1 --2- 172.21.15.77:0/3120200404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c580b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5cfa9640 1 --2- 172.21.15.77:0/3120200404 >> v2:172.21.15.158:3300/0 conn(0x7f1c580cfa90 0x7f1c580cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c4ffff640 1 -- 172.21.15.77:0/3120200404 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524631138 0 0) 0x7f1c58158ac0 con 0x7f1c581586e0 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c4ffff640 1 -- 172.21.15.77:0/3120200404 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c44003520 con 0x7f1c581586e0 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5dfab640 1 -- 172.21.15.77:0/3120200404 >> v2:172.21.15.158:3300/0 conn(0x7f1c580cfa90 msgr2=0x7f1c580cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5dfab640 1 --2- 172.21.15.77:0/3120200404 >> v2:172.21.15.158:3300/0 conn(0x7f1c580cfa90 0x7f1c580cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5dfab640 1 -- 172.21.15.77:0/3120200404 >> v1:172.21.15.77:6789/0 conn(0x7f1c581586e0 legacy=0x7f1c580aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5dfab640 1 -- 172.21.15.77:0/3120200404 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5815b820 con 0x7f1c580b33a0 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.809+0000 7f1c5cfa9640 1 --2- 172.21.15.77:0/3120200404 >> v2:172.21.15.158:3300/0 conn(0x7f1c580cfa90 0x7f1c580cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.810+0000 7f1c5dfab640 1 --2- 172.21.15.77:0/3120200404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c580b37c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009d90 tx=0x7f1c5002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41351eb108134c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.810+0000 7f1c4ffff640 1 -- 172.21.15.77:0/3120200404 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50039070 con 0x7f1c580b33a0 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.810+0000 7f1c4ffff640 1 -- 172.21.15.77:0/3120200404 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5002cc50 con 0x7f1c580b33a0 2024-03-06T07:29:49.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.810+0000 7f1c4ffff640 1 -- 172.21.15.77:0/3120200404 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50034440 con 0x7f1c580b33a0 2024-03-06T07:29:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.811+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3120200404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 msgr2=0x7f1c580b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.811+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3120200404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c580b37c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009d90 tx=0x7f1c5002bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.811+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3120200404 shutdown_connections 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.811+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3120200404 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c580b37c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.811+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3120200404 >> v2:172.21.15.158:3300/0 conn(0x7f1c580cfa90 0x7f1c580cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.811+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3120200404 >> 172.21.15.77:0/3120200404 conn(0x7f1c5806b190 msgr2=0x7f1c5806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.811+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3120200404 shutdown_connections 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.811+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3120200404 wait complete. 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.812+0000 7f1c5fa0e640 1 Processor -- start 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.812+0000 7f1c5fa0e640 1 -- start start 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5fa0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c58155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5fa0e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c580cfa90 0x7f1c58153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5fa0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c581586e0 0x7f1c58156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5fa0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c580af150 con 0x7f1c581586e0 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5fa0e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5815bde0 con 0x7f1c580cfa90 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5fa0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c58071350 con 0x7f1c580b33a0 2024-03-06T07:29:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c581586e0 0x7f1c58156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c581586e0 0x7f1c58156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41396/0 (socket says 172.21.15.77:41396) 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 -- 172.21.15.77:0/3330692828 learned_addr learned my addr 172.21.15.77:0/3330692828 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5cfa9640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c580cfa90 0x7f1c58153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5d7aa640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c58155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 -- 172.21.15.77:0/3330692828 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 msgr2=0x7f1c58155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c58155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 -- 172.21.15.77:0/3330692828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c580cfa90 msgr2=0x7f1c58153b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c580cfa90 0x7f1c58153b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 -- 172.21.15.77:0/3330692828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c58154050 con 0x7f1c581586e0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5dfab640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c581586e0 0x7f1c58156e50 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009d90 tx=0x7f1c50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c4dffb640 1 -- 172.21.15.77:0/3330692828 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50039050 con 0x7f1c581586e0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c4dffb640 1 -- 172.21.15.77:0/3330692828 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c50045070 con 0x7f1c581586e0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c4dffb640 1 -- 172.21.15.77:0/3330692828 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50009450 con 0x7f1c581586e0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.813+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c58157390 con 0x7f1c581586e0 2024-03-06T07:29:49.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.814+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c58156370 con 0x7f1c581586e0 2024-03-06T07:29:49.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.815+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c580732b0 con 0x7f1c581586e0 2024-03-06T07:29:49.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.815+0000 7f1c4dffb640 1 -- 172.21.15.77:0/3330692828 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c5003e610 con 0x7f1c581586e0 2024-03-06T07:29:49.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.815+0000 7f1c4dffb640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c3c06b4d0 0x7f1c3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.815+0000 7f1c4dffb640 1 -- 172.21.15.77:0/3330692828 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1c500acab0 con 0x7f1c581586e0 2024-03-06T07:29:49.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.815+0000 7f1c5d7aa640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c3c06b4d0 0x7f1c3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.816+0000 7f1c5d7aa640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c3c06b4d0 0x7f1c3c06d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1c54005880 tx=0x7f1c54009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.820+0000 7f1c4dffb640 1 -- 172.21.15.77:0/3330692828 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c5007b2a0 con 0x7f1c581586e0 2024-03-06T07:29:49.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.989+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c58071de0 con 0x7f1c581586e0 2024-03-06T07:29:49.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.989+0000 7f1c4dffb640 1 -- 172.21.15.77:0/3330692828 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c5007f340 con 0x7f1c581586e0 2024-03-06T07:29:49.992 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:29:49.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c3c06b4d0 msgr2=0x7f1c3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c3c06b4d0 0x7f1c3c06d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1c54005880 tx=0x7f1c54009040 comp rx=0 tx=0).stop 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c581586e0 msgr2=0x7f1c58156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c581586e0 0x7f1c58156e50 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009d90 tx=0x7f1c50004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 shutdown_connections 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1c3c06b4d0 0x7f1c3c06d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c581586e0 0x7f1c58156e50 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1c580cfa90 0x7f1c58153b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 --2- 172.21.15.77:0/3330692828 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1c580b33a0 0x7f1c58155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 >> 172.21.15.77:0/3330692828 conn(0x7f1c5806b190 msgr2=0x7f1c580af430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 shutdown_connections 2024-03-06T07:29:49.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:49.991+0000 7f1c5fa0e640 1 -- 172.21.15.77:0/3330692828 wait complete. 2024-03-06T07:29:50.006 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:50.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.106+0000 7fb1c477a640 1 Processor -- start 2024-03-06T07:29:50.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.106+0000 7fb1c477a640 1 -- start start 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.107+0000 7fb1c477a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0cfa90 0x7fb1bc0cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.107+0000 7fb1c477a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1bc158ac0 con 0x7fb1bc1586e0 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.107+0000 7fb1c477a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1bc15b2a0 con 0x7fb1bc0b33a0 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.107+0000 7fb1c477a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc15b540 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c2516640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb1bc1586e0 0x7fb1bc0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56616/0 (socket says 172.21.15.77:56616) 2024-03-06T07:29:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c2516640 1 -- 172.21.15.77:0/486454541 learned_addr learned my addr 172.21.15.77:0/486454541 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c1d15640 1 --2- 172.21.15.77:0/486454541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0cfa90 0x7fb1bc0cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c1514640 1 -- 172.21.15.77:0/486454541 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955779312 0 0) 0x7fb1bc158ac0 con 0x7fb1bc1586e0 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c1514640 1 -- 172.21.15.77:0/486454541 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a8003520 con 0x7fb1bc1586e0 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c1514640 1 -- 172.21.15.77:0/486454541 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993785547 0 0) 0x7fb1bc15b2a0 con 0x7fb1bc0b33a0 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c1514640 1 -- 172.21.15.77:0/486454541 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1bc158ac0 con 0x7fb1bc0b33a0 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c1d15640 1 -- 172.21.15.77:0/486454541 >> v1:172.21.15.158:6789/0 conn(0x7fb1bc0b33a0 legacy=0x7fb1bc0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c1d15640 1 -- 172.21.15.77:0/486454541 >> v1:172.21.15.77:6789/0 conn(0x7fb1bc1586e0 legacy=0x7fb1bc0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.108+0000 7fb1c1d15640 1 -- 172.21.15.77:0/486454541 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc15b820 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.109+0000 7fb1c1d15640 1 --2- 172.21.15.77:0/486454541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0cfa90 0x7fb1bc0cff10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80097d0 tx=0x7fb1b802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=248fb94d9d5ee1b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.109+0000 7fb1c1514640 1 -- 172.21.15.77:0/486454541 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8035020 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.109+0000 7fb1c1514640 1 -- 172.21.15.77:0/486454541 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b8031890 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.109+0000 7fb1c1514640 1 -- 172.21.15.77:0/486454541 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8031c40 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.109+0000 7fb1c477a640 1 -- 172.21.15.77:0/486454541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0cfa90 msgr2=0x7fb1bc0cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.109+0000 7fb1c477a640 1 --2- 172.21.15.77:0/486454541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0cfa90 0x7fb1bc0cff10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80097d0 tx=0x7fb1b802b930 comp rx=0 tx=0).stop 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.110+0000 7fb1c477a640 1 -- 172.21.15.77:0/486454541 shutdown_connections 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.110+0000 7fb1c477a640 1 --2- 172.21.15.77:0/486454541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc0cfa90 0x7fb1bc0cff10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.110+0000 7fb1c477a640 1 -- 172.21.15.77:0/486454541 >> 172.21.15.77:0/486454541 conn(0x7fb1bc06b190 msgr2=0x7fb1bc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.110+0000 7fb1c477a640 1 -- 172.21.15.77:0/486454541 shutdown_connections 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.110+0000 7fb1c477a640 1 -- 172.21.15.77:0/486454541 wait complete. 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.111+0000 7fb1c477a640 1 Processor -- start 2024-03-06T07:29:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.111+0000 7fb1c477a640 1 -- start start 2024-03-06T07:29:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.111+0000 7fb1c477a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0b33a0 0x7fb1bc1504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.111+0000 7fb1c477a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0cfa90 0x7fb1bc150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.111+0000 7fb1c477a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc1586e0 0x7fb1bc14bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.111+0000 7fb1c477a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc0af150 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.111+0000 7fb1c477a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc15bde0 con 0x7fb1bc0b33a0 2024-03-06T07:29:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.111+0000 7fb1c477a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc071350 con 0x7fb1bc1586e0 2024-03-06T07:29:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0cfa90 0x7fb1bc150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0cfa90 0x7fb1bc150ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41404/0 (socket says 172.21.15.77:41404) 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 -- 172.21.15.77:0/1303667518 learned_addr learned my addr 172.21.15.77:0/1303667518 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c2516640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0b33a0 0x7fb1bc1504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c2d17640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc1586e0 0x7fb1bc14bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 -- 172.21.15.77:0/1303667518 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc1586e0 msgr2=0x7fb1bc14bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc1586e0 0x7fb1bc14bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 -- 172.21.15.77:0/1303667518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0b33a0 msgr2=0x7fb1bc1504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0b33a0 0x7fb1bc1504f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 -- 172.21.15.77:0/1303667518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc1515b0 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c1d15640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0cfa90 0x7fb1bc150ac0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8000c00 tx=0x7fb1b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1aeffd640 1 -- 172.21.15.77:0/1303667518 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8035040 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1aeffd640 1 -- 172.21.15.77:0/1303667518 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b8030d20 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1bc151000 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1aeffd640 1 -- 172.21.15.77:0/1303667518 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8006e10 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.112+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1bc0cd060 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.113+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1bc15aff0 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.114+0000 7fb1aeffd640 1 -- 172.21.15.77:0/1303667518 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1b8038020 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.114+0000 7fb1aeffd640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1a006b4d0 0x7fb1a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.114+0000 7fb1aeffd640 1 -- 172.21.15.77:0/1303667518 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb1b80ac6f0 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.114+0000 7fb1c2516640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1a006b4d0 0x7fb1a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.114+0000 7fb1c2516640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1a006b4d0 0x7fb1a006d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0000f60 tx=0x7fb1b0007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.118+0000 7fb1aeffd640 1 -- 172.21.15.77:0/1303667518 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1bc15aff0 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.290+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1bc0732b0 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.291+0000 7fb1aeffd640 1 -- 172.21.15.77:0/1303667518 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1bc1515b0 con 0x7fb1bc0cfa90 2024-03-06T07:29:50.293 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1a006b4d0 msgr2=0x7fb1a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1a006b4d0 0x7fb1a006d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0000f60 tx=0x7fb1b0007d70 comp rx=0 tx=0).stop 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0cfa90 msgr2=0x7fb1bc150ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0cfa90 0x7fb1bc150ac0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8000c00 tx=0x7fb1b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 shutdown_connections 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb1a006b4d0 0x7fb1a006d990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1bc0cfa90 0x7fb1bc150ac0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb1bc1586e0 0x7fb1bc14bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 --2- 172.21.15.77:0/1303667518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb1bc0b33a0 0x7fb1bc1504f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 >> 172.21.15.77:0/1303667518 conn(0x7fb1bc06b190 msgr2=0x7fb1bc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 shutdown_connections 2024-03-06T07:29:50.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.293+0000 7fb1c477a640 1 -- 172.21.15.77:0/1303667518 wait complete. 2024-03-06T07:29:50.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:50.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.406+0000 7fa65c5ac640 1 Processor -- start 2024-03-06T07:29:50.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.406+0000 7fa65c5ac640 1 -- start start 2024-03-06T07:29:50.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65c5ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa65415f4f0 0x7fa654066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65c5ac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa6540b0db0 0x7fa6540ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65c5ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6540ce580 0x7fa654072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65c5ac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65406b050 con 0x7fa65415f4f0 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65c5ac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65406b2c0 con 0x7fa6540b0db0 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65c5ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6540b0890 con 0x7fa6540ce580 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65a348640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa65415f4f0 0x7fa654066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65a348640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa65415f4f0 0x7fa654066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41414/0 (socket says 172.21.15.77:41414) 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65a348640 1 -- 172.21.15.77:0/607976466 learned_addr learned my addr 172.21.15.77:0/607976466 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa659b47640 1 --2- 172.21.15.77:0/607976466 >> v2:172.21.15.158:3300/0 conn(0x7fa6540b0db0 0x7fa6540ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65ab49640 1 --2- 172.21.15.77:0/607976466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6540ce580 0x7fa654072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65a348640 1 -- 172.21.15.77:0/607976466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6540ce580 msgr2=0x7fa654072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65a348640 1 --2- 172.21.15.77:0/607976466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6540ce580 0x7fa654072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65a348640 1 -- 172.21.15.77:0/607976466 >> v2:172.21.15.158:3300/0 conn(0x7fa6540b0db0 msgr2=0x7fa6540ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65a348640 1 --2- 172.21.15.77:0/607976466 >> v2:172.21.15.158:3300/0 conn(0x7fa6540b0db0 0x7fa6540ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.407+0000 7fa65a348640 1 -- 172.21.15.77:0/607976466 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6540672e0 con 0x7fa65415f4f0 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.408+0000 7fa65a348640 1 --2- 172.21.15.77:0/607976466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa65415f4f0 0x7fa654066d00 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa6540d11d0 tx=0x7fa65002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387e83b791f9b6ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.408+0000 7fa659346640 1 -- 172.21.15.77:0/607976466 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650035020 con 0x7fa65415f4f0 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.408+0000 7fa659346640 1 -- 172.21.15.77:0/607976466 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650031870 con 0x7fa65415f4f0 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.408+0000 7fa659346640 1 -- 172.21.15.77:0/607976466 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650031c40 con 0x7fa65415f4f0 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 -- 172.21.15.77:0/607976466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa65415f4f0 msgr2=0x7fa654066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/607976466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa65415f4f0 0x7fa654066d00 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa6540d11d0 tx=0x7fa65002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 -- 172.21.15.77:0/607976466 shutdown_connections 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/607976466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa6540ce580 0x7fa654072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/607976466 >> v2:172.21.15.158:3300/0 conn(0x7fa6540b0db0 0x7fa6540ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/607976466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa65415f4f0 0x7fa654066d00 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 -- 172.21.15.77:0/607976466 >> 172.21.15.77:0/607976466 conn(0x7fa6540b1d70 msgr2=0x7fa6540b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 -- 172.21.15.77:0/607976466 shutdown_connections 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.409+0000 7fa65c5ac640 1 -- 172.21.15.77:0/607976466 wait complete. 2024-03-06T07:29:50.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.410+0000 7fa65c5ac640 1 Processor -- start 2024-03-06T07:29:50.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.410+0000 7fa65c5ac640 1 -- start start 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65c5ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6540b0db0 0x7fa654148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65c5ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6540ce580 0x7fa65409d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65c5ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa65415f4f0 0x7fa6540c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65c5ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65415f8d0 con 0x7fa6540ce580 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65c5ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65406b5a0 con 0x7fa6540b0db0 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65c5ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6540cc9c0 con 0x7fa65415f4f0 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6540ce580 0x7fa65409d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6540ce580 0x7fa65409d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41420/0 (socket says 172.21.15.77:41420) 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 -- 172.21.15.77:0/2078156485 learned_addr learned my addr 172.21.15.77:0/2078156485 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65a348640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6540b0db0 0x7fa654148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65ab49640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa65415f4f0 0x7fa6540c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 -- 172.21.15.77:0/2078156485 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa65415f4f0 msgr2=0x7fa6540c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa65415f4f0 0x7fa6540c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 -- 172.21.15.77:0/2078156485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6540b0db0 msgr2=0x7fa654148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6540b0db0 0x7fa654148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 -- 172.21.15.77:0/2078156485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6541492c0 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa659b47640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6540ce580 0x7fa65409d950 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fa64400e870 tx=0x7fa64400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa64affd640 1 -- 172.21.15.77:0/2078156485 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644017070 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa64affd640 1 -- 172.21.15.77:0/2078156485 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64400ee60 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa64affd640 1 -- 172.21.15.77:0/2078156485 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400cb80 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6540cbdb0 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.411+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6540cc650 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.412+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65409e3d0 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.413+0000 7fa64affd640 1 -- 172.21.15.77:0/2078156485 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa64401a070 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.413+0000 7fa64affd640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa63806b500 0x7fa63806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.413+0000 7fa64affd640 1 -- 172.21.15.77:0/2078156485 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa644089fd0 con 0x7fa6540ce580 2024-03-06T07:29:50.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.413+0000 7fa65a348640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa63806b500 0x7fa63806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.414+0000 7fa65a348640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa63806b500 0x7fa63806d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa650036000 tx=0x7fa65003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.417+0000 7fa64affd640 1 -- 172.21.15.77:0/2078156485 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6440587c0 con 0x7fa6540ce580 2024-03-06T07:29:50.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.589+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa6540b22c0 con 0x7fa6540ce580 2024-03-06T07:29:50.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.590+0000 7fa64affd640 1 -- 172.21.15.77:0/2078156485 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa64405c860 con 0x7fa6540ce580 2024-03-06T07:29:50.592 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa63806b500 msgr2=0x7fa63806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa63806b500 0x7fa63806d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa650036000 tx=0x7fa65003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6540ce580 msgr2=0x7fa65409d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6540ce580 0x7fa65409d950 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fa64400e870 tx=0x7fa64400ed30 comp rx=0 tx=0).stop 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 shutdown_connections 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa63806b500 0x7fa63806d9c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6540ce580 0x7fa65409d950 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa65415f4f0 0x7fa6540c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 --2- 172.21.15.77:0/2078156485 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6540b0db0 0x7fa654148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 >> 172.21.15.77:0/2078156485 conn(0x7fa6540b1d70 msgr2=0x7fa654072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 shutdown_connections 2024-03-06T07:29:50.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.592+0000 7fa65c5ac640 1 -- 172.21.15.77:0/2078156485 wait complete. 2024-03-06T07:29:50.606 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:50.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.703+0000 7f33ba316640 1 Processor -- start 2024-03-06T07:29:50.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.704+0000 7f33ba316640 1 -- start start 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.704+0000 7f33ba316640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33b40b4460 0x7f33b40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.704+0000 7f33ba316640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4162970 0x7f33b40b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.704+0000 7f33ba316640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b40d22c0 con 0x7f33b4161cf0 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.704+0000 7f33ba316640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b4161b00 con 0x7f33b40b4460 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.704+0000 7f33ba316640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b40cecc0 con 0x7f33b4162970 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.705+0000 7f33b37fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4162970 0x7f33b40b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.705+0000 7f33b3fff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f33b40b4460 0x7f33b40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.705+0000 7f33b88b3640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f33b4161cf0 0x7f33b40cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44294/0 (socket says 172.21.15.77:44294) 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.705+0000 7f33b88b3640 1 -- 172.21.15.77:0/4114978330 learned_addr learned my addr 172.21.15.77:0/4114978330 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.705+0000 7f33b3fff640 1 -- 172.21.15.77:0/4114978330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4162970 msgr2=0x7f33b40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.705+0000 7f33b3fff640 1 --2- 172.21.15.77:0/4114978330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4162970 0x7f33b40b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.705+0000 7f33b3fff640 1 -- 172.21.15.77:0/4114978330 >> v1:172.21.15.158:6789/0 conn(0x7f33b4161cf0 legacy=0x7f33b40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.705+0000 7f33b3fff640 1 -- 172.21.15.77:0/4114978330 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b40d25a0 con 0x7f33b40b4460 2024-03-06T07:29:50.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33b3fff640 1 --2- 172.21.15.77:0/4114978330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b40b4460 0x7f33b40d4800 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f33a40097a0 tx=0x7f33a402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ec6d030020b0f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33b2ffd640 1 -- 172.21.15.77:0/4114978330 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4037070 con 0x7f33b40b4460 2024-03-06T07:29:50.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33b2ffd640 1 -- 172.21.15.77:0/4114978330 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a4030710 con 0x7f33b40b4460 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33b2ffd640 1 -- 172.21.15.77:0/4114978330 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4030ae0 con 0x7f33b40b4460 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33ba316640 1 -- 172.21.15.77:0/4114978330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b40b4460 msgr2=0x7f33b40d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33ba316640 1 --2- 172.21.15.77:0/4114978330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b40b4460 0x7f33b40d4800 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f33a40097a0 tx=0x7f33a402b710 comp rx=0 tx=0).stop 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33ba316640 1 -- 172.21.15.77:0/4114978330 shutdown_connections 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33ba316640 1 --2- 172.21.15.77:0/4114978330 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4162970 0x7f33b40b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33ba316640 1 --2- 172.21.15.77:0/4114978330 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b40b4460 0x7f33b40d4800 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33ba316640 1 -- 172.21.15.77:0/4114978330 >> 172.21.15.77:0/4114978330 conn(0x7f33b40d3ce0 msgr2=0x7f33b40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33ba316640 1 -- 172.21.15.77:0/4114978330 shutdown_connections 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.706+0000 7f33ba316640 1 -- 172.21.15.77:0/4114978330 wait complete. 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.707+0000 7f33ba316640 1 Processor -- start 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.708+0000 7f33ba316640 1 -- start start 2024-03-06T07:29:50.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.708+0000 7f33ba316640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40b4460 0x7f33b415fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.708+0000 7f33ba316640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4161cf0 0x7f33b4160380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.708+0000 7f33ba316640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b4162970 0x7f33b40c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.708+0000 7f33ba316640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b40ce950 con 0x7f33b40b4460 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.708+0000 7f33ba316640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b40d2b60 con 0x7f33b4162970 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.708+0000 7f33ba316640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b40b3ac0 con 0x7f33b4161cf0 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40b4460 0x7f33b415fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b37fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4161cf0 0x7f33b4160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b88b3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b4162970 0x7f33b40c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40b4460 0x7f33b415fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41428/0 (socket says 172.21.15.77:41428) 2024-03-06T07:29:50.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b3fff640 1 -- 172.21.15.77:0/3384418151 learned_addr learned my addr 172.21.15.77:0/3384418151 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:50.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b37fe640 1 -- 172.21.15.77:0/3384418151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b4162970 msgr2=0x7f33b40c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b37fe640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b4162970 0x7f33b40c7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b37fe640 1 -- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40b4460 msgr2=0x7f33b415fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b37fe640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40b4460 0x7f33b415fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b37fe640 1 -- 172.21.15.77:0/3384418151 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b40ca5c0 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b3fff640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40b4460 0x7f33b415fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b37fe640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4161cf0 0x7f33b4160380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f33a800e850 tx=0x7f33a800ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b0ff9640 1 -- 172.21.15.77:0/3384418151 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8017070 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b0ff9640 1 -- 172.21.15.77:0/3384418151 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a800c940 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b40c9520 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33b0ff9640 1 -- 172.21.15.77:0/3384418151 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a800cd10 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.709+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b40c9ae0 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.710+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b404f330 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.711+0000 7f33b0ff9640 1 -- 172.21.15.77:0/3384418151 <== mon.2 v2:172.21.15.158:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33a801a070 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.711+0000 7f33b0ff9640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f33a006b4d0 0x7f33a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.711+0000 7f33b0ff9640 1 -- 172.21.15.77:0/3384418151 <== mon.2 v2:172.21.15.158:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f33a808a160 con 0x7f33b4161cf0 2024-03-06T07:29:50.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.711+0000 7f33b3fff640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f33a006b4d0 0x7f33a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.712+0000 7f33b3fff640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f33a006b4d0 0x7f33a006d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f33a40097a0 tx=0x7f33a4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.721+0000 7f33b0ff9640 1 -- 172.21.15.77:0/3384418151 <== mon.2 v2:172.21.15.158:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33a80588d0 con 0x7f33b4161cf0 2024-03-06T07:29:50.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.904+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f33b406b180 con 0x7f33b4161cf0 2024-03-06T07:29:50.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.905+0000 7f33b0ff9640 1 -- 172.21.15.77:0/3384418151 <== mon.2 v2:172.21.15.158:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v129) v1 ==== 80+0+0 (secure 0 0 0) 0x7f33a805c970 con 0x7f33b4161cf0 2024-03-06T07:29:50.907 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.906+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f33a006b4d0 msgr2=0x7f33a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.906+0000 7f33ba316640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f33a006b4d0 0x7f33a006d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f33a40097a0 tx=0x7f33a4008030 comp rx=0 tx=0).stop 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.906+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4161cf0 msgr2=0x7f33b4160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.906+0000 7f33ba316640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4161cf0 0x7f33b4160380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f33a800e850 tx=0x7f33a800ed10 comp rx=0 tx=0).stop 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.907+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 shutdown_connections 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.907+0000 7f33ba316640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f33b4162970 0x7f33b40c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.907+0000 7f33ba316640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f33b4161cf0 0x7f33b4160380 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.907+0000 7f33ba316640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f33a006b4d0 0x7f33a006d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.907+0000 7f33ba316640 1 --2- 172.21.15.77:0/3384418151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b40b4460 0x7f33b415fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.907+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 >> 172.21.15.77:0/3384418151 conn(0x7f33b40d3ce0 msgr2=0x7f33b406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.907+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 shutdown_connections 2024-03-06T07:29:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:50.907+0000 7f33ba316640 1 -- 172.21.15.77:0/3384418151 wait complete. 2024-03-06T07:29:50.920 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:29:50.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:50.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:50.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[78473]: fuse finished with error 0 and tester_r 0 2024-03-06T07:29:50.947 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:29:50.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:50.948 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:51.090 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:29:51.090 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.106 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:51.106 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:29:51.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:51.106 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:29:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:51.161 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:29:51.161 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:29:51.161 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.177 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:51.177 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:29:51.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:51.177 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:29:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:51.232 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:29:51.233 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:29:51.233 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.247 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:51.247 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:29:51.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:51.248 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:29:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:51.301 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:29:51.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.399+0000 7f7f60a1c640 1 Processor -- start 2024-03-06T07:29:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.399+0000 7f7f60a1c640 1 -- start start 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f60a1c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f5c0d02c0 0x7f7f5c0cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f60a1c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7f5c0b6550 0x7f7f5c0cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f60a1c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c15c620 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f60a1c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c0b50b0 con 0x7f7f5c0d02c0 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f60a1c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c0cd800 con 0x7f7f5c0b6550 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f5a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f5c0d02c0 0x7f7f5c0cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f5a59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f5c0d02c0 0x7f7f5c0cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41440/0 (socket says 172.21.15.77:41440) 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f5a59c640 1 -- 172.21.15.77:0/4132136983 learned_addr learned my addr 172.21.15.77:0/4132136983 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f5ad9d640 1 --2- 172.21.15.77:0/4132136983 >> v2:172.21.15.158:3300/0 conn(0x7f7f5c0b6550 0x7f7f5c0cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f5a59c640 1 -- 172.21.15.77:0/4132136983 >> v2:172.21.15.158:3300/0 conn(0x7f7f5c0b6550 msgr2=0x7f7f5c0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f5a59c640 1 --2- 172.21.15.77:0/4132136983 >> v2:172.21.15.158:3300/0 conn(0x7f7f5c0b6550 0x7f7f5c0cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f5a59c640 1 -- 172.21.15.77:0/4132136983 >> v1:172.21.15.158:6789/0 conn(0x7f7f5c0d06a0 legacy=0x7f7f5c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.400+0000 7f7f5a59c640 1 -- 172.21.15.77:0/4132136983 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c0b4a60 con 0x7f7f5c0d02c0 2024-03-06T07:29:51.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.401+0000 7f7f5a59c640 1 --2- 172.21.15.77:0/4132136983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d02c0 0x7f7f5c0cf850 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0cc050 tx=0x7f7f4c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b3d81f1794aee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.401+0000 7f7f59d9b640 1 -- 172.21.15.77:0/4132136983 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c034070 con 0x7f7f5c0d02c0 2024-03-06T07:29:51.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.401+0000 7f7f59d9b640 1 -- 172.21.15.77:0/4132136983 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c030860 con 0x7f7f5c0d02c0 2024-03-06T07:29:51.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.401+0000 7f7f59d9b640 1 -- 172.21.15.77:0/4132136983 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c030c30 con 0x7f7f5c0d02c0 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.402+0000 7f7f60a1c640 1 -- 172.21.15.77:0/4132136983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d02c0 msgr2=0x7f7f5c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.402+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/4132136983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d02c0 0x7f7f5c0cf850 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0cc050 tx=0x7f7f4c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.402+0000 7f7f60a1c640 1 -- 172.21.15.77:0/4132136983 shutdown_connections 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.402+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/4132136983 >> v2:172.21.15.158:3300/0 conn(0x7f7f5c0b6550 0x7f7f5c0cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.402+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/4132136983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d02c0 0x7f7f5c0cf850 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.402+0000 7f7f60a1c640 1 -- 172.21.15.77:0/4132136983 >> 172.21.15.77:0/4132136983 conn(0x7f7f5c0b35a0 msgr2=0x7f7f5c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.402+0000 7f7f60a1c640 1 -- 172.21.15.77:0/4132136983 shutdown_connections 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.402+0000 7f7f60a1c640 1 -- 172.21.15.77:0/4132136983 wait complete. 2024-03-06T07:29:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.403+0000 7f7f60a1c640 1 Processor -- start 2024-03-06T07:29:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.403+0000 7f7f60a1c640 1 -- start start 2024-03-06T07:29:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f60a1c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f5c0b6550 0x7f7f5c0cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f60a1c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f5c0d02c0 0x7f7f5c06d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f60a1c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d06a0 0x7f7f5c0c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f60a1c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0ce9c0 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f60a1c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0ce840 con 0x7f7f5c0b6550 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f60a1c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0ceb40 con 0x7f7f5c0d02c0 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d06a0 0x7f7f5c0c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d06a0 0x7f7f5c0c8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41450/0 (socket says 172.21.15.77:41450) 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 -- 172.21.15.77:0/1798891904 learned_addr learned my addr 172.21.15.77:0/1798891904 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5a59c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f5c0b6550 0x7f7f5c0cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f53fff640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f5c0d02c0 0x7f7f5c06d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 -- 172.21.15.77:0/1798891904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f5c0d02c0 msgr2=0x7f7f5c06d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f5c0d02c0 0x7f7f5c06d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 -- 172.21.15.77:0/1798891904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f5c0b6550 msgr2=0x7f7f5c0cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f5c0b6550 0x7f7f5c0cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 -- 172.21.15.77:0/1798891904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c14ad50 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f5ad9d640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d06a0 0x7f7f5c0c8c60 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400efc0 tx=0x7f7f5400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f52ffd640 1 -- 172.21.15.77:0/1798891904 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54018020 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f52ffd640 1 -- 172.21.15.77:0/1798891904 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f54014480 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f52ffd640 1 -- 172.21.15.77:0/1798891904 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54014870 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c14b030 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.404+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c14b5f0 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.405+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c04f330 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.406+0000 7f7f52ffd640 1 -- 172.21.15.77:0/1798891904 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f54005d10 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.406+0000 7f7f52ffd640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.406+0000 7f7f52ffd640 1 -- 172.21.15.77:0/1798891904 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f5408b060 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.406+0000 7f7f5a59c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.407+0000 7f7f5a59c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c036000 tx=0x7f7f4c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.411+0000 7f7f52ffd640 1 -- 172.21.15.77:0/1798891904 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f54022030 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.619+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5c0cf3d0 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.619+0000 7f7f52ffd640 1 -- 172.21.15.77:0/1798891904 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+3402 (secure 0 0 0) 0x7f7f54010040 con 0x7f7f5c0d06a0 2024-03-06T07:29:51.622 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:51.622 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8973,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/4200914134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4200914134},{"type":"v1","addr":"172.21.15.77:6835","nonce":4200914134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:47.721340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:51.623 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 91 2024-03-06T07:29:51.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f3c06b4d0 msgr2=0x7f7f3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c036000 tx=0x7f7f4c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d06a0 msgr2=0x7f7f5c0c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d06a0 0x7f7f5c0c8c60 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400efc0 tx=0x7f7f5400c490 comp rx=0 tx=0).stop 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 shutdown_connections 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0d06a0 0x7f7f5c0c8c60 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7f5c0d02c0 0x7f7f5c06d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 --2- 172.21.15.77:0/1798891904 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7f5c0b6550 0x7f7f5c0cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 >> 172.21.15.77:0/1798891904 conn(0x7f7f5c0b35a0 msgr2=0x7f7f5c06f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 shutdown_connections 2024-03-06T07:29:51.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.621+0000 7f7f60a1c640 1 -- 172.21.15.77:0/1798891904 wait complete. 2024-03-06T07:29:51.635 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:51.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.732+0000 7ffb9e3d0640 1 Processor -- start 2024-03-06T07:29:51.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.732+0000 7ffb9e3d0640 1 -- start start 2024-03-06T07:29:51.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb9e3d0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ffb980cf850 0x7ffb980cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb9e3d0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffb9806cf20 0x7ffb9806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb9e3d0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9815e380 con 0x7ffb980d00c0 2024-03-06T07:29:51.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb9e3d0640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9806ca90 con 0x7ffb9806cf20 2024-03-06T07:29:51.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb9e3d0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb980cdc10 con 0x7ffb980cf850 2024-03-06T07:29:51.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb9c96d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffb9806cf20 0x7ffb9806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb97fff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ffb980d00c0 0x7ffb980d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44304/0 (socket says 172.21.15.77:44304) 2024-03-06T07:29:51.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb9c96d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffb9806cf20 0x7ffb9806d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41458/0 (socket says 172.21.15.77:41458) 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb97fff640 1 -- 172.21.15.77:0/11494881 learned_addr learned my addr 172.21.15.77:0/11494881 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb977fe640 1 --2- 172.21.15.77:0/11494881 >> v2:172.21.15.158:3300/0 conn(0x7ffb980cf850 0x7ffb980cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb96ffd640 1 -- 172.21.15.77:0/11494881 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559809187 0 0) 0x7ffb9815e380 con 0x7ffb980d00c0 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.733+0000 7ffb96ffd640 1 -- 172.21.15.77:0/11494881 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb80003520 con 0x7ffb980d00c0 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb9c96d640 1 -- 172.21.15.77:0/11494881 >> v2:172.21.15.158:3300/0 conn(0x7ffb980cf850 msgr2=0x7ffb980cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb9c96d640 1 --2- 172.21.15.77:0/11494881 >> v2:172.21.15.158:3300/0 conn(0x7ffb980cf850 0x7ffb980cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb9c96d640 1 -- 172.21.15.77:0/11494881 >> v1:172.21.15.158:6789/0 conn(0x7ffb980d00c0 legacy=0x7ffb980d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb9c96d640 1 -- 172.21.15.77:0/11494881 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9806c160 con 0x7ffb9806cf20 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb977fe640 1 --2- 172.21.15.77:0/11494881 >> v2:172.21.15.158:3300/0 conn(0x7ffb980cf850 0x7ffb980cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb9c96d640 1 --2- 172.21.15.77:0/11494881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb9806d3b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7ffb98071de0 tx=0x7ffb9002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddac32d4b1cd2471 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb96ffd640 1 -- 172.21.15.77:0/11494881 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90039070 con 0x7ffb9806cf20 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb96ffd640 1 -- 172.21.15.77:0/11494881 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9002cc80 con 0x7ffb9806cf20 2024-03-06T07:29:51.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.734+0000 7ffb96ffd640 1 -- 172.21.15.77:0/11494881 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90034660 con 0x7ffb9806cf20 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.735+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/11494881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 msgr2=0x7ffb9806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.735+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/11494881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb9806d3b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7ffb98071de0 tx=0x7ffb9002bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.735+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/11494881 shutdown_connections 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.735+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/11494881 >> v2:172.21.15.158:3300/0 conn(0x7ffb980cf850 0x7ffb980cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.735+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/11494881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb9806d3b0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.735+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/11494881 >> 172.21.15.77:0/11494881 conn(0x7ffb980b35a0 msgr2=0x7ffb980b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.735+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/11494881 shutdown_connections 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.735+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/11494881 wait complete. 2024-03-06T07:29:51.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.736+0000 7ffb9e3d0640 1 Processor -- start 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.736+0000 7ffb9e3d0640 1 -- start start 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb9e3d0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb980c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb9e3d0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb980cf850 0x7ffb981f7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb9e3d0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb980d00c0 0x7ffb980caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb9e3d0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb980cd900 con 0x7ffb9806cf20 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb9e3d0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9806c720 con 0x7ffb980cf850 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb9e3d0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb980b1030 con 0x7ffb980d00c0 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb980c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb980c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41468/0 (socket says 172.21.15.77:41468) 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb97fff640 1 -- 172.21.15.77:0/3662630738 learned_addr learned my addr 172.21.15.77:0/3662630738 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb9c96d640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb980d00c0 0x7ffb980caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb977fe640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb980cf850 0x7ffb981f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb97fff640 1 -- 172.21.15.77:0/3662630738 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb980d00c0 msgr2=0x7ffb980caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb97fff640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb980d00c0 0x7ffb980caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb97fff640 1 -- 172.21.15.77:0/3662630738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb980cf850 msgr2=0x7ffb981f7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb97fff640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb980cf850 0x7ffb981f7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.737+0000 7ffb97fff640 1 -- 172.21.15.77:0/3662630738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb981f7ba0 con 0x7ffb9806cf20 2024-03-06T07:29:51.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.738+0000 7ffb97fff640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb980c7990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c390 tx=0x7ffb8800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.738+0000 7ffb94ff9640 1 -- 172.21.15.77:0/3662630738 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88017070 con 0x7ffb9806cf20 2024-03-06T07:29:51.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.738+0000 7ffb94ff9640 1 -- 172.21.15.77:0/3662630738 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb88005ce0 con 0x7ffb9806cf20 2024-03-06T07:29:51.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.738+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb981f7e80 con 0x7ffb9806cf20 2024-03-06T07:29:51.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.738+0000 7ffb94ff9640 1 -- 172.21.15.77:0/3662630738 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb880072f0 con 0x7ffb9806cf20 2024-03-06T07:29:51.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.738+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb980caad0 con 0x7ffb9806cf20 2024-03-06T07:29:51.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.739+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb980cacd0 con 0x7ffb9806cf20 2024-03-06T07:29:51.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.739+0000 7ffb94ff9640 1 -- 172.21.15.77:0/3662630738 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb8801a070 con 0x7ffb9806cf20 2024-03-06T07:29:51.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.740+0000 7ffb94ff9640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb8406b4d0 0x7ffb8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.740+0000 7ffb977fe640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb8406b4d0 0x7ffb8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.740+0000 7ffb94ff9640 1 -- 172.21.15.77:0/3662630738 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffb88089540 con 0x7ffb9806cf20 2024-03-06T07:29:51.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.740+0000 7ffb977fe640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb8406b4d0 0x7ffb8406d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009560 tx=0x7ffb8c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.744+0000 7ffb94ff9640 1 -- 172.21.15.77:0/3662630738 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb980cacd0 con 0x7ffb9806cf20 2024-03-06T07:29:51.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.918+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb9804f330 con 0x7ffb9806cf20 2024-03-06T07:29:51.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.920+0000 7ffb94ff9640 1 -- 172.21.15.77:0/3662630738 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16764 (secure 0 0 0) 0x7ffb88057d30 con 0x7ffb9806cf20 2024-03-06T07:29:51.923 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:51.923 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":129,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:51.262074+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:30.343353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:31.094099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.9","mappings":[{"from":2,"to":0}]},{"pgid":"19.12","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.921+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb8406b4d0 msgr2=0x7ffb8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.921+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb8406b4d0 0x7ffb8406d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009560 tx=0x7ffb8c009190 comp rx=0 tx=0).stop 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.921+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 msgr2=0x7ffb980c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.921+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb980c7990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c390 tx=0x7ffb8800c850 comp rx=0 tx=0).stop 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.922+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 shutdown_connections 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.922+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffb980d00c0 0x7ffb980caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.922+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffb980cf850 0x7ffb981f7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.922+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffb8406b4d0 0x7ffb8406d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.922+0000 7ffb9e3d0640 1 --2- 172.21.15.77:0/3662630738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb9806cf20 0x7ffb980c7990 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.922+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 >> 172.21.15.77:0/3662630738 conn(0x7ffb980b35a0 msgr2=0x7ffb980b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.922+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 shutdown_connections 2024-03-06T07:29:51.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:51.922+0000 7ffb9e3d0640 1 -- 172.21.15.77:0/3662630738 wait complete. 2024-03-06T07:29:51.936 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:30.343353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:31.094099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:51.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.036+0000 7f2d637eb640 1 Processor -- start 2024-03-06T07:29:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.036+0000 7f2d637eb640 1 -- start start 2024-03-06T07:29:52.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d637eb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2d5c1614d0 0x7f2d5c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d637eb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d5c0d1910 0x7f2d5c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d637eb640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5c1618b0 con 0x7f2d5c0d22e0 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d637eb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d5c0cf400 con 0x7f2d5c0d1910 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d637eb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d5c0d2710 con 0x7f2d5c1614d0 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d60d86640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d5c0d1910 0x7f2d5c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d61d88640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2d5c0d22e0 0x7f2d5c0d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56632/0 (socket says 172.21.15.77:56632) 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d60d86640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d5c0d1910 0x7f2d5c0d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41478/0 (socket says 172.21.15.77:41478) 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.037+0000 7f2d61d88640 1 -- 172.21.15.77:0/955868610 learned_addr learned my addr 172.21.15.77:0/955868610 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.038+0000 7f2d61587640 1 --2- 172.21.15.77:0/955868610 >> v2:172.21.15.158:3300/0 conn(0x7f2d5c1614d0 0x7f2d5c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.038+0000 7f2d60d86640 1 -- 172.21.15.77:0/955868610 >> v2:172.21.15.158:3300/0 conn(0x7f2d5c1614d0 msgr2=0x7f2d5c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.038+0000 7f2d60d86640 1 --2- 172.21.15.77:0/955868610 >> v2:172.21.15.158:3300/0 conn(0x7f2d5c1614d0 0x7f2d5c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.038+0000 7f2d60d86640 1 -- 172.21.15.77:0/955868610 >> v1:172.21.15.77:6789/0 conn(0x7f2d5c0d22e0 legacy=0x7f2d5c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.038+0000 7f2d53fff640 1 -- 172.21.15.77:0/955868610 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080051440 0 0) 0x7f2d5c1618b0 con 0x7f2d5c0d22e0 2024-03-06T07:29:52.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.038+0000 7f2d60d86640 1 -- 172.21.15.77:0/955868610 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c0cf6a0 con 0x7f2d5c0d1910 2024-03-06T07:29:52.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.038+0000 7f2d60d86640 1 --2- 172.21.15.77:0/955868610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c0d1910 0x7f2d5c0d1da0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c08c470 tx=0x7f2d4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308f617c68559899 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d53fff640 1 -- 172.21.15.77:0/955868610 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c035020 con 0x7f2d5c0d1910 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d53fff640 1 -- 172.21.15.77:0/955868610 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c031830 con 0x7f2d5c0d1910 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d53fff640 1 -- 172.21.15.77:0/955868610 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c031c00 con 0x7f2d5c0d1910 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d637eb640 1 -- 172.21.15.77:0/955868610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c0d1910 msgr2=0x7f2d5c0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d637eb640 1 --2- 172.21.15.77:0/955868610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c0d1910 0x7f2d5c0d1da0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c08c470 tx=0x7f2d4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d637eb640 1 -- 172.21.15.77:0/955868610 shutdown_connections 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d637eb640 1 --2- 172.21.15.77:0/955868610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c0d1910 0x7f2d5c0d1da0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d637eb640 1 --2- 172.21.15.77:0/955868610 >> v2:172.21.15.158:3300/0 conn(0x7f2d5c1614d0 0x7f2d5c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d637eb640 1 -- 172.21.15.77:0/955868610 >> 172.21.15.77:0/955868610 conn(0x7f2d5c0d3d40 msgr2=0x7f2d5c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d637eb640 1 -- 172.21.15.77:0/955868610 shutdown_connections 2024-03-06T07:29:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.039+0000 7f2d637eb640 1 -- 172.21.15.77:0/955868610 wait complete. 2024-03-06T07:29:52.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.040+0000 7f2d637eb640 1 Processor -- start 2024-03-06T07:29:52.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.040+0000 7f2d637eb640 1 -- start start 2024-03-06T07:29:52.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d637eb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d5c0d1910 0x7f2d5c15be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d637eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2d5c0d22e0 0x7f2d5c0b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d637eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c1614d0 0x7f2d5c15dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d637eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c0b0910 con 0x7f2d5c1614d0 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d637eb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c0cfc60 con 0x7f2d5c0d1910 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d637eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c0b1e40 con 0x7f2d5c0d22e0 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61d88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c1614d0 0x7f2d5c15dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61d88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c1614d0 0x7f2d5c15dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41480/0 (socket says 172.21.15.77:41480) 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61d88640 1 -- 172.21.15.77:0/3272674713 learned_addr learned my addr 172.21.15.77:0/3272674713 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61587640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d5c0d1910 0x7f2d5c15be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d60d86640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2d5c0d22e0 0x7f2d5c0b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61d88640 1 -- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2d5c0d22e0 msgr2=0x7f2d5c0b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61d88640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2d5c0d22e0 0x7f2d5c0b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61d88640 1 -- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d5c0d1910 msgr2=0x7f2d5c15be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61d88640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d5c0d1910 0x7f2d5c15be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.041+0000 7f2d61d88640 1 -- 172.21.15.77:0/3272674713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c0cb1d0 con 0x7f2d5c1614d0 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.042+0000 7f2d60d86640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2d5c0d22e0 0x7f2d5c0b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.042+0000 7f2d61d88640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c1614d0 0x7f2d5c15dfc0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400c4b0 tx=0x7f2d5400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.042+0000 7f2d51ffb640 1 -- 172.21.15.77:0/3272674713 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d54005ce0 con 0x7f2d5c1614d0 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.042+0000 7f2d51ffb640 1 -- 172.21.15.77:0/3272674713 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d54007070 con 0x7f2d5c1614d0 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.042+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c0cb4b0 con 0x7f2d5c1614d0 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.042+0000 7f2d51ffb640 1 -- 172.21.15.77:0/3272674713 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d540074a0 con 0x7f2d5c1614d0 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.042+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c0cba70 con 0x7f2d5c1614d0 2024-03-06T07:29:52.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.043+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c0d1da0 con 0x7f2d5c1614d0 2024-03-06T07:29:52.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.043+0000 7f2d51ffb640 1 -- 172.21.15.77:0/3272674713 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d5401b400 con 0x7f2d5c1614d0 2024-03-06T07:29:52.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.044+0000 7f2d51ffb640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2d4006b4d0 0x7f2d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.044+0000 7f2d51ffb640 1 -- 172.21.15.77:0/3272674713 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2d54089d10 con 0x7f2d5c1614d0 2024-03-06T07:29:52.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.044+0000 7f2d61587640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2d4006b4d0 0x7f2d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.044+0000 7f2d61587640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2d4006b4d0 0x7f2d4006d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0ca490 tx=0x7f2d5800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.048+0000 7f2d51ffb640 1 -- 172.21.15.77:0/3272674713 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d54058be0 con 0x7f2d5c1614d0 2024-03-06T07:29:52.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.219+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d5c068ff0 con 0x7f2d5c1614d0 2024-03-06T07:29:52.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.219+0000 7f2d51ffb640 1 -- 172.21.15.77:0/3272674713 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d540584e0 con 0x7f2d5c1614d0 2024-03-06T07:29:52.222 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2d4006b4d0 msgr2=0x7f2d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2d4006b4d0 0x7f2d4006d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0ca490 tx=0x7f2d5800b040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c1614d0 msgr2=0x7f2d5c15dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c1614d0 0x7f2d5c15dfc0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400c4b0 tx=0x7f2d5400c970 comp rx=0 tx=0).stop 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 shutdown_connections 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2d4006b4d0 0x7f2d4006d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d5c1614d0 0x7f2d5c15dfc0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2d5c0d22e0 0x7f2d5c0b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 --2- 172.21.15.77:0/3272674713 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2d5c0d1910 0x7f2d5c15be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 >> 172.21.15.77:0/3272674713 conn(0x7f2d5c0d3d40 msgr2=0x7f2d5c06b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 shutdown_connections 2024-03-06T07:29:52.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.221+0000 7f2d637eb640 1 -- 172.21.15.77:0/3272674713 wait complete. 2024-03-06T07:29:52.235 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:52.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.337+0000 7f7d5bb13640 1 Processor -- start 2024-03-06T07:29:52.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.337+0000 7f7d5bb13640 1 -- start start 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5bb13640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 0x7f7d540cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5bb13640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d540b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5bb13640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d54158ac0 con 0x7f7d541586e0 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5bb13640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d5415b2a0 con 0x7f7d540cfa90 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5bb13640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5415b540 con 0x7f7d540b33a0 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d598af640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7d541586e0 0x7f7d540aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56638/0 (socket says 172.21.15.77:56638) 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d598af640 1 -- 172.21.15.77:0/1216105043 learned_addr learned my addr 172.21.15.77:0/1216105043 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5a0b0640 1 --2- 172.21.15.77:0/1216105043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d540b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d590ae640 1 --2- 172.21.15.77:0/1216105043 >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 0x7f7d540cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d588ad640 1 -- 172.21.15.77:0/1216105043 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694877719 0 0) 0x7f7d54158ac0 con 0x7f7d541586e0 2024-03-06T07:29:52.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d588ad640 1 -- 172.21.15.77:0/1216105043 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d3c003520 con 0x7f7d541586e0 2024-03-06T07:29:52.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5a0b0640 1 -- 172.21.15.77:0/1216105043 >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 msgr2=0x7f7d540cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5a0b0640 1 --2- 172.21.15.77:0/1216105043 >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 0x7f7d540cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5a0b0640 1 -- 172.21.15.77:0/1216105043 >> v1:172.21.15.77:6789/0 conn(0x7f7d541586e0 legacy=0x7f7d540aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d5a0b0640 1 -- 172.21.15.77:0/1216105043 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5415b820 con 0x7f7d540b33a0 2024-03-06T07:29:52.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.338+0000 7f7d590ae640 1 --2- 172.21.15.77:0/1216105043 >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 0x7f7d540cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:52.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.339+0000 7f7d5a0b0640 1 --2- 172.21.15.77:0/1216105043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d540b37c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009d90 tx=0x7f7d4402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80a386fe5d5b4eea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.339+0000 7f7d588ad640 1 -- 172.21.15.77:0/1216105043 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44039070 con 0x7f7d540b33a0 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.339+0000 7f7d588ad640 1 -- 172.21.15.77:0/1216105043 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4402cc50 con 0x7f7d540b33a0 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.339+0000 7f7d588ad640 1 -- 172.21.15.77:0/1216105043 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44034440 con 0x7f7d540b33a0 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.339+0000 7f7d5bb13640 1 -- 172.21.15.77:0/1216105043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 msgr2=0x7f7d540b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.339+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/1216105043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d540b37c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009d90 tx=0x7f7d4402bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.340+0000 7f7d5bb13640 1 -- 172.21.15.77:0/1216105043 shutdown_connections 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.340+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/1216105043 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d540b37c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.340+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/1216105043 >> v2:172.21.15.158:3300/0 conn(0x7f7d540cfa90 0x7f7d540cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.340+0000 7f7d5bb13640 1 -- 172.21.15.77:0/1216105043 >> 172.21.15.77:0/1216105043 conn(0x7f7d5406b190 msgr2=0x7f7d5406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.340+0000 7f7d5bb13640 1 -- 172.21.15.77:0/1216105043 shutdown_connections 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.340+0000 7f7d5bb13640 1 -- 172.21.15.77:0/1216105043 wait complete. 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.341+0000 7f7d5bb13640 1 Processor -- start 2024-03-06T07:29:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.341+0000 7f7d5bb13640 1 -- start start 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.341+0000 7f7d5bb13640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d54141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.341+0000 7f7d5bb13640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d540cfa90 0x7f7d540ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.341+0000 7f7d5bb13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d541586e0 0x7f7d54147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.341+0000 7f7d5bb13640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540af150 con 0x7f7d541586e0 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.341+0000 7f7d5bb13640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5415bde0 con 0x7f7d540cfa90 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.341+0000 7f7d5bb13640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54071350 con 0x7f7d540b33a0 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d541586e0 0x7f7d54147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d541586e0 0x7f7d54147140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41490/0 (socket says 172.21.15.77:41490) 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 -- 172.21.15.77:0/130981742 learned_addr learned my addr 172.21.15.77:0/130981742 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d590ae640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d540cfa90 0x7f7d540ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d598af640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d54141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 -- 172.21.15.77:0/130981742 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 msgr2=0x7f7d54141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d54141780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 -- 172.21.15.77:0/130981742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d540cfa90 msgr2=0x7f7d540ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d540cfa90 0x7f7d540ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 -- 172.21.15.77:0/130981742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d54141cc0 con 0x7f7d541586e0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5a0b0640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d541586e0 0x7f7d54147140 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009d90 tx=0x7f7d44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d527fc640 1 -- 172.21.15.77:0/130981742 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44039050 con 0x7f7d541586e0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d527fc640 1 -- 172.21.15.77:0/130981742 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d44045070 con 0x7f7d541586e0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d527fc640 1 -- 172.21.15.77:0/130981742 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44009450 con 0x7f7d541586e0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5406cb60 con 0x7f7d541586e0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.342+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5406d400 con 0x7f7d541586e0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.343+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d540732b0 con 0x7f7d541586e0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.343+0000 7f7d527fc640 1 -- 172.21.15.77:0/130981742 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d4403e610 con 0x7f7d541586e0 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.344+0000 7f7d527fc640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.344+0000 7f7d527fc640 1 -- 172.21.15.77:0/130981742 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7d440aca30 con 0x7f7d541586e0 2024-03-06T07:29:52.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.344+0000 7f7d598af640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.344+0000 7f7d598af640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c005880 tx=0x7f7d4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.348+0000 7f7d527fc640 1 -- 172.21.15.77:0/130981742 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d4407b370 con 0x7f7d541586e0 2024-03-06T07:29:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.519+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d540cbde0 con 0x7f7d541586e0 2024-03-06T07:29:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.519+0000 7f7d527fc640 1 -- 172.21.15.77:0/130981742 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d4407f410 con 0x7f7d541586e0 2024-03-06T07:29:52.522 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:29:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 msgr2=0x7f7d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c005880 tx=0x7f7d4c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d541586e0 msgr2=0x7f7d54147140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d541586e0 0x7f7d54147140 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009d90 tx=0x7f7d44004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 shutdown_connections 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d4006b4d0 0x7f7d4006d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d541586e0 0x7f7d54147140 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d540cfa90 0x7f7d540ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 --2- 172.21.15.77:0/130981742 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d540b33a0 0x7f7d54141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 >> 172.21.15.77:0/130981742 conn(0x7f7d5406b190 msgr2=0x7f7d5414f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 shutdown_connections 2024-03-06T07:29:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.521+0000 7f7d5bb13640 1 -- 172.21.15.77:0/130981742 wait complete. 2024-03-06T07:29:52.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:52.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.636+0000 7fefd2d5c640 1 Processor -- start 2024-03-06T07:29:52.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.636+0000 7fefd2d5c640 1 -- start start 2024-03-06T07:29:52.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefd2d5c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fefcc15fc30 0x7fefcc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefd2d5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0b4d10 0x7fefcc0cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefd2d5c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefcc1600a0 con 0x7fefcc0d06e0 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefd2d5c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc0ced70 con 0x7fefcc15fc30 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefd2d5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0d47f0 con 0x7fefcc0b4d10 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefcbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fefcc15fc30 0x7fefcc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefcbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fefcc15fc30 0x7fefcc0b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41500/0 (socket says 172.21.15.77:41500) 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefd12f9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0b4d10 0x7fefcc0cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.637+0000 7fefcbfff640 1 -- 172.21.15.77:0/2689295451 learned_addr learned my addr 172.21.15.77:0/2689295451 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.638+0000 7fefcbfff640 1 -- 172.21.15.77:0/2689295451 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0b4d10 msgr2=0x7fefcc0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.638+0000 7fefcbfff640 1 --2- 172.21.15.77:0/2689295451 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0b4d10 0x7fefcc0cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.638+0000 7fefcbfff640 1 -- 172.21.15.77:0/2689295451 >> v1:172.21.15.158:6789/0 conn(0x7fefcc0d06e0 legacy=0x7fefcc0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.638+0000 7fefcb7fe640 1 -- 172.21.15.77:0/2689295451 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760304402 0 0) 0x7fefcc1600a0 con 0x7fefcc0d06e0 2024-03-06T07:29:52.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.638+0000 7fefcbfff640 1 -- 172.21.15.77:0/2689295451 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc0d1ab0 con 0x7fefcc15fc30 2024-03-06T07:29:52.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.638+0000 7fefcbfff640 1 --2- 172.21.15.77:0/2689295451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b47d0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009870 tx=0x7fefb802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1597143e8f9720 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefcb7fe640 1 -- 172.21.15.77:0/2689295451 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb8034070 con 0x7fefcc15fc30 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefcb7fe640 1 -- 172.21.15.77:0/2689295451 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb8030840 con 0x7fefcc15fc30 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefcb7fe640 1 -- 172.21.15.77:0/2689295451 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb8030c10 con 0x7fefcc15fc30 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefd2d5c640 1 -- 172.21.15.77:0/2689295451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 msgr2=0x7fefcc0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/2689295451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b47d0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009870 tx=0x7fefb802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefd2d5c640 1 -- 172.21.15.77:0/2689295451 shutdown_connections 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/2689295451 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0b4d10 0x7fefcc0cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/2689295451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b47d0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefd2d5c640 1 -- 172.21.15.77:0/2689295451 >> 172.21.15.77:0/2689295451 conn(0x7fefcc0d3cd0 msgr2=0x7fefcc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefd2d5c640 1 -- 172.21.15.77:0/2689295451 shutdown_connections 2024-03-06T07:29:52.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.639+0000 7fefd2d5c640 1 -- 172.21.15.77:0/2689295451 wait complete. 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.640+0000 7fefd2d5c640 1 Processor -- start 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.640+0000 7fefd2d5c640 1 -- start start 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd2d5c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefcc0b4d10 0x7fefcc1f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd2d5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0d06e0 0x7fefcc0b9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd2d5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd2d5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0ce990 con 0x7fefcc15fc30 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd2d5c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0d4ab0 con 0x7fefcc0b4d10 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd2d5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0b44d0 con 0x7fefcc0d06e0 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd12f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd12f9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41508/0 (socket says 172.21.15.77:41508) 2024-03-06T07:29:52.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd12f9640 1 -- 172.21.15.77:0/326455492 learned_addr learned my addr 172.21.15.77:0/326455492 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefcbfff640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0d06e0 0x7fefcc0b9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.641+0000 7fefd0af8640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefcc0b4d10 0x7fefcc1f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefd12f9640 1 -- 172.21.15.77:0/326455492 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0d06e0 msgr2=0x7fefcc0b9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefd12f9640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0d06e0 0x7fefcc0b9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefd12f9640 1 -- 172.21.15.77:0/326455492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefcc0b4d10 msgr2=0x7fefcc1f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefd12f9640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefcc0b4d10 0x7fefcc1f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefd12f9640 1 -- 172.21.15.77:0/326455492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc0cbf50 con 0x7fefcc15fc30 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefd12f9640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b7190 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fefc400efc0 tx=0x7fefc400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefc97fa640 1 -- 172.21.15.77:0/326455492 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4018020 con 0x7fefcc15fc30 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefc97fa640 1 -- 172.21.15.77:0/326455492 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc4014480 con 0x7fefcc15fc30 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefc97fa640 1 -- 172.21.15.77:0/326455492 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4014870 con 0x7fefcc15fc30 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc0cc230 con 0x7fefcc15fc30 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.642+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc0ccad0 con 0x7fefcc15fc30 2024-03-06T07:29:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.643+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc0cc400 con 0x7fefcc15fc30 2024-03-06T07:29:52.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.643+0000 7fefc97fa640 1 -- 172.21.15.77:0/326455492 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefc4005d10 con 0x7fefcc15fc30 2024-03-06T07:29:52.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.644+0000 7fefc97fa640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.644+0000 7fefc97fa640 1 -- 172.21.15.77:0/326455492 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fefc408aed0 con 0x7fefcc15fc30 2024-03-06T07:29:52.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.644+0000 7fefd0af8640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.644+0000 7fefd0af8640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fefc0000f60 tx=0x7fefc0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.648+0000 7fefc97fa640 1 -- 172.21.15.77:0/326455492 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefc4022030 con 0x7fefcc15fc30 2024-03-06T07:29:52.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.821+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fefcc0a1c80 con 0x7fefcc15fc30 2024-03-06T07:29:52.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.821+0000 7fefc97fa640 1 -- 172.21.15.77:0/326455492 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7fefc4010040 con 0x7fefcc15fc30 2024-03-06T07:29:52.824 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:29:52.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 msgr2=0x7fefb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fefc0000f60 tx=0x7fefc0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 msgr2=0x7fefcc0b7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b7190 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fefc400efc0 tx=0x7fefc400c490 comp rx=0 tx=0).stop 2024-03-06T07:29:52.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 shutdown_connections 2024-03-06T07:29:52.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fefb006b4d0 0x7fefb006d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefcc15fc30 0x7fefcc0b7190 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fefcc0d06e0 0x7fefcc0b9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 --2- 172.21.15.77:0/326455492 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefcc0b4d10 0x7fefcc1f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 >> 172.21.15.77:0/326455492 conn(0x7fefcc0d3cd0 msgr2=0x7fefcc161390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 shutdown_connections 2024-03-06T07:29:52.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.823+0000 7fefd2d5c640 1 -- 172.21.15.77:0/326455492 wait complete. 2024-03-06T07:29:52.837 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:52.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.934+0000 7fb798c8d640 1 Processor -- start 2024-03-06T07:29:52.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.934+0000 7fb798c8d640 1 -- start start 2024-03-06T07:29:52.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb798c8d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb7940d02c0 0x7fb7940cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb798c8d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940d06a0 0x7fb7940cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb798c8d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb7940ce130 0x7fb7940ce560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb798c8d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7940b50b0 con 0x7fb7940ce130 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb798c8d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79415c620 con 0x7fb7940d02c0 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb798c8d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7940cd800 con 0x7fb7940d06a0 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb792d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb7940ce130 0x7fb7940ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb792d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb7940ce130 0x7fb7940ce560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41510/0 (socket says 172.21.15.77:41510) 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb792d9d640 1 -- 172.21.15.77:0/25505884 learned_addr learned my addr 172.21.15.77:0/25505884 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb791d9b640 1 --2- 172.21.15.77:0/25505884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940d06a0 0x7fb7940cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.935+0000 7fb79259c640 1 --2- 172.21.15.77:0/25505884 >> v2:172.21.15.158:3300/0 conn(0x7fb7940d02c0 0x7fb7940cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb792d9d640 1 -- 172.21.15.77:0/25505884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940d06a0 msgr2=0x7fb7940cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb792d9d640 1 --2- 172.21.15.77:0/25505884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940d06a0 0x7fb7940cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb792d9d640 1 -- 172.21.15.77:0/25505884 >> v2:172.21.15.158:3300/0 conn(0x7fb7940d02c0 msgr2=0x7fb7940cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb792d9d640 1 --2- 172.21.15.77:0/25505884 >> v2:172.21.15.158:3300/0 conn(0x7fb7940d02c0 0x7fb7940cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb792d9d640 1 -- 172.21.15.77:0/25505884 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7940b4a60 con 0x7fb7940ce130 2024-03-06T07:29:52.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb792d9d640 1 --2- 172.21.15.77:0/25505884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940ce130 0x7fb7940ce560 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fb794071cf0 tx=0x7fb78c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f9eee24d7d480a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb79159a640 1 -- 172.21.15.77:0/25505884 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c02ca70 con 0x7fb7940ce130 2024-03-06T07:29:52.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb79159a640 1 -- 172.21.15.77:0/25505884 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c02cc10 con 0x7fb7940ce130 2024-03-06T07:29:52.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.936+0000 7fb79159a640 1 -- 172.21.15.77:0/25505884 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c0343f0 con 0x7fb7940ce130 2024-03-06T07:29:52.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 -- 172.21.15.77:0/25505884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940ce130 msgr2=0x7fb7940ce560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 --2- 172.21.15.77:0/25505884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940ce130 0x7fb7940ce560 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fb794071cf0 tx=0x7fb78c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 -- 172.21.15.77:0/25505884 shutdown_connections 2024-03-06T07:29:52.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 --2- 172.21.15.77:0/25505884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940ce130 0x7fb7940ce560 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 --2- 172.21.15.77:0/25505884 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940d06a0 0x7fb7940cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 --2- 172.21.15.77:0/25505884 >> v2:172.21.15.158:3300/0 conn(0x7fb7940d02c0 0x7fb7940cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 -- 172.21.15.77:0/25505884 >> 172.21.15.77:0/25505884 conn(0x7fb7940b35a0 msgr2=0x7fb7940b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 -- 172.21.15.77:0/25505884 shutdown_connections 2024-03-06T07:29:52.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.937+0000 7fb798c8d640 1 -- 172.21.15.77:0/25505884 wait complete. 2024-03-06T07:29:52.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.938+0000 7fb798c8d640 1 Processor -- start 2024-03-06T07:29:52.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.938+0000 7fb798c8d640 1 -- start start 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb798c8d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940ce130 0x7fb7940c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb798c8d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7940d02c0 0x7fb7941f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb798c8d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940d06a0 0x7fb7940b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb798c8d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7940cec20 con 0x7fb7940d06a0 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb798c8d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7940ceaa0 con 0x7fb7940d02c0 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb798c8d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7940ceda0 con 0x7fb7940ce130 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb792d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940d06a0 0x7fb7940b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb792d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940d06a0 0x7fb7940b1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41518/0 (socket says 172.21.15.77:41518) 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb792d9d640 1 -- 172.21.15.77:0/4026145196 learned_addr learned my addr 172.21.15.77:0/4026145196 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb79259c640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940ce130 0x7fb7940c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.939+0000 7fb791d9b640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7940d02c0 0x7fb7941f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb792d9d640 1 -- 172.21.15.77:0/4026145196 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940ce130 msgr2=0x7fb7940c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb792d9d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940ce130 0x7fb7940c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb792d9d640 1 -- 172.21.15.77:0/4026145196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7940d02c0 msgr2=0x7fb7941f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb792d9d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7940d02c0 0x7fb7941f1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb792d9d640 1 -- 172.21.15.77:0/4026145196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7941f2330 con 0x7fb7940d06a0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb792d9d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940d06a0 0x7fb7940b1d10 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009f40 tx=0x7fb78c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb782ffd640 1 -- 172.21.15.77:0/4026145196 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c043020 con 0x7fb7940d06a0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb782ffd640 1 -- 172.21.15.77:0/4026145196 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c03e450 con 0x7fb7940d06a0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7941f2610 con 0x7fb7940d06a0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb782ffd640 1 -- 172.21.15.77:0/4026145196 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78c009510 con 0x7fb7940d06a0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.940+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7940c90b0 con 0x7fb7940d06a0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.941+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79404f330 con 0x7fb7940d06a0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.941+0000 7fb782ffd640 1 -- 172.21.15.77:0/4026145196 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb78c046020 con 0x7fb7940d06a0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.942+0000 7fb782ffd640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb77406b4d0 0x7fb77406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.942+0000 7fb79259c640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb77406b4d0 0x7fb77406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.942+0000 7fb782ffd640 1 -- 172.21.15.77:0/4026145196 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb78c0ac720 con 0x7fb7940d06a0 2024-03-06T07:29:52.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.942+0000 7fb79259c640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb77406b4d0 0x7fb77406d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb7940c8040 tx=0x7fb784009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:52.947+0000 7fb782ffd640 1 -- 172.21.15.77:0/4026145196 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb78c04d030 con 0x7fb7940d06a0 2024-03-06T07:29:53.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.156+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7940cfe30 con 0x7fb7940d06a0 2024-03-06T07:29:53.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.156+0000 7fb782ffd640 1 -- 172.21.15.77:0/4026145196 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+3402 (secure 0 0 0) 0x7fb78c07afa0 con 0x7fb7940d06a0 2024-03-06T07:29:53.159 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:53.159 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8973,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/4200914134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4200914134},{"type":"v1","addr":"172.21.15.77:6835","nonce":4200914134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:47.721340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:53.160 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 91 2024-03-06T07:29:53.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb77406b4d0 msgr2=0x7fb77406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb77406b4d0 0x7fb77406d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb7940c8040 tx=0x7fb784009190 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940d06a0 msgr2=0x7fb7940b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940d06a0 0x7fb7940b1d10 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009f40 tx=0x7fb78c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 shutdown_connections 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb77406b4d0 0x7fb77406d990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7940d06a0 0x7fb7940b1d10 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7940d02c0 0x7fb7941f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 --2- 172.21.15.77:0/4026145196 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb7940ce130 0x7fb7940c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 >> 172.21.15.77:0/4026145196 conn(0x7fb7940b35a0 msgr2=0x7fb7940b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 shutdown_connections 2024-03-06T07:29:53.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.158+0000 7fb798c8d640 1 -- 172.21.15.77:0/4026145196 wait complete. 2024-03-06T07:29:53.172 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:53.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.270+0000 7f101000f640 1 Processor -- start 2024-03-06T07:29:53.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.270+0000 7f101000f640 1 -- start start 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f101000f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f10081424f0 0x7f10080cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f101000f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10080cd7a0 0x7f100815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f101000f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100806d330 0x7f100806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f101000f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10080cd220 con 0x7f10080cd7a0 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f101000f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10080cc980 con 0x7f10081424f0 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f101000f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10080ccc60 con 0x7f100806d330 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100d5aa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10080cd7a0 0x7f100815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100d5aa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10080cd7a0 0x7f100815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41524/0 (socket says 172.21.15.77:41524) 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100d5aa640 1 -- 172.21.15.77:0/1359899484 learned_addr learned my addr 172.21.15.77:0/1359899484 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100ddab640 1 --2- 172.21.15.77:0/1359899484 >> v2:172.21.15.158:3300/0 conn(0x7f10081424f0 0x7f10080cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100e5ac640 1 --2- 172.21.15.77:0/1359899484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100806d330 0x7f100806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100d5aa640 1 -- 172.21.15.77:0/1359899484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100806d330 msgr2=0x7f100806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100d5aa640 1 --2- 172.21.15.77:0/1359899484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100806d330 0x7f100806d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100d5aa640 1 -- 172.21.15.77:0/1359899484 >> v2:172.21.15.158:3300/0 conn(0x7f10081424f0 msgr2=0x7f10080cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100d5aa640 1 --2- 172.21.15.77:0/1359899484 >> v2:172.21.15.158:3300/0 conn(0x7f10081424f0 0x7f10080cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.271+0000 7f100d5aa640 1 -- 172.21.15.77:0/1359899484 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10080ccf40 con 0x7f10080cd7a0 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.272+0000 7f100d5aa640 1 --2- 172.21.15.77:0/1359899484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10080cd7a0 0x7f100815cef0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f100806c6d0 tx=0x7f0ff802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ee6082684dc611 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.272+0000 7f100cda9640 1 -- 172.21.15.77:0/1359899484 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8035020 con 0x7f10080cd7a0 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.272+0000 7f100cda9640 1 -- 172.21.15.77:0/1359899484 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff8031870 con 0x7f10080cd7a0 2024-03-06T07:29:53.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.272+0000 7f100cda9640 1 -- 172.21.15.77:0/1359899484 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8031c40 con 0x7f10080cd7a0 2024-03-06T07:29:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 -- 172.21.15.77:0/1359899484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10080cd7a0 msgr2=0x7f100815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 --2- 172.21.15.77:0/1359899484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10080cd7a0 0x7f100815cef0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f100806c6d0 tx=0x7f0ff802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 -- 172.21.15.77:0/1359899484 shutdown_connections 2024-03-06T07:29:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 --2- 172.21.15.77:0/1359899484 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100806d330 0x7f100806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 --2- 172.21.15.77:0/1359899484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10080cd7a0 0x7f100815cef0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 --2- 172.21.15.77:0/1359899484 >> v2:172.21.15.158:3300/0 conn(0x7f10081424f0 0x7f10080cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 -- 172.21.15.77:0/1359899484 >> 172.21.15.77:0/1359899484 conn(0x7f10080cf8a0 msgr2=0x7f10080cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 -- 172.21.15.77:0/1359899484 shutdown_connections 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.273+0000 7f101000f640 1 -- 172.21.15.77:0/1359899484 wait complete. 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.274+0000 7f101000f640 1 Processor -- start 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.274+0000 7f101000f640 1 -- start start 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f101000f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100806d330 0x7f1008159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f101000f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10080cd7a0 0x7f100815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f101000f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10081424f0 0x7f100815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f101000f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1008071ce0 con 0x7f100806d330 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f101000f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1008071b60 con 0x7f10080cd7a0 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f101000f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1008071e60 con 0x7f10081424f0 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100806d330 0x7f1008159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100806d330 0x7f1008159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41540/0 (socket says 172.21.15.77:41540) 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 -- 172.21.15.77:0/391282093 learned_addr learned my addr 172.21.15.77:0/391282093 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100e5ac640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10081424f0 0x7f100815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100d5aa640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10080cd7a0 0x7f100815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 -- 172.21.15.77:0/391282093 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10081424f0 msgr2=0x7f100815b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10081424f0 0x7f100815b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 -- 172.21.15.77:0/391282093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10080cd7a0 msgr2=0x7f100815ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10080cd7a0 0x7f100815ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 -- 172.21.15.77:0/391282093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10080b36a0 con 0x7f100806d330 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f100ddab640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100806d330 0x7f1008159650 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f100400e870 tx=0x7f100400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f0ffe7fc640 1 -- 172.21.15.77:0/391282093 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004017070 con 0x7f100806d330 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f0ffe7fc640 1 -- 172.21.15.77:0/391282093 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100400c940 con 0x7f100806d330 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10080b3980 con 0x7f100806d330 2024-03-06T07:29:53.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f0ffe7fc640 1 -- 172.21.15.77:0/391282093 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100400cd30 con 0x7f100806d330 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.275+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10080b3f40 con 0x7f100806d330 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.276+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100804f330 con 0x7f100806d330 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.277+0000 7f0ffe7fc640 1 -- 172.21.15.77:0/391282093 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f100401a070 con 0x7f100806d330 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.277+0000 7f0ffe7fc640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.277+0000 7f0ffe7fc640 1 -- 172.21.15.77:0/391282093 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f100408a320 con 0x7f100806d330 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.277+0000 7f100d5aa640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.278+0000 7f100d5aa640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8036000 tx=0x7f0ff803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.282+0000 7f0ffe7fc640 1 -- 172.21.15.77:0/391282093 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1004058ba0 con 0x7f100806d330 2024-03-06T07:29:53.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.457+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10080d0620 con 0x7f100806d330 2024-03-06T07:29:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.458+0000 7f0ffe7fc640 1 -- 172.21.15.77:0/391282093 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16764 (secure 0 0 0) 0x7f100405cc40 con 0x7f100806d330 2024-03-06T07:29:53.461 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:53.461 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":129,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:51.262074+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:30.343353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:31.094099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.9","mappings":[{"from":2,"to":0}]},{"pgid":"19.12","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 msgr2=0x7f0fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8036000 tx=0x7f0ff803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100806d330 msgr2=0x7f1008159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100806d330 0x7f1008159650 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f100400e870 tx=0x7f100400ed30 comp rx=0 tx=0).stop 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 shutdown_connections 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10081424f0 0x7f100815b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10080cd7a0 0x7f100815ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 --2- 172.21.15.77:0/391282093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100806d330 0x7f1008159650 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 >> 172.21.15.77:0/391282093 conn(0x7f10080cf8a0 msgr2=0x7f100806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 shutdown_connections 2024-03-06T07:29:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.460+0000 7f101000f640 1 -- 172.21.15.77:0/391282093 wait complete. 2024-03-06T07:29:53.474 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:29:53.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:53.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.572+0000 7fac31864640 1 Processor -- start 2024-03-06T07:29:53.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.572+0000 7fac31864640 1 -- start start 2024-03-06T07:29:53.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac31864640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fac2c0d22b0 0x7fac2c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac31864640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c0b41f0 0x7fac2c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac31864640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c0b3430 con 0x7fac2c15dfe0 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac31864640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2c0d2c50 con 0x7fac2c0d22b0 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac31864640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0b31e0 con 0x7fac2c0b41f0 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2affd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fac2c15dfe0 0x7fac2c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56654/0 (socket says 172.21.15.77:56654) 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2affd640 1 -- 172.21.15.77:0/3606627213 learned_addr learned my addr 172.21.15.77:0/3606627213 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2b7fe640 1 --2- 172.21.15.77:0/3606627213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c0b41f0 0x7fac2c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2a7fc640 1 --2- 172.21.15.77:0/3606627213 >> v2:172.21.15.158:3300/0 conn(0x7fac2c0d22b0 0x7fac2c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac29ffb640 1 -- 172.21.15.77:0/3606627213 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662254126 0 0) 0x7fac2c0b3430 con 0x7fac2c15dfe0 2024-03-06T07:29:53.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac29ffb640 1 -- 172.21.15.77:0/3606627213 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac14003520 con 0x7fac2c15dfe0 2024-03-06T07:29:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2b7fe640 1 -- 172.21.15.77:0/3606627213 >> v2:172.21.15.158:3300/0 conn(0x7fac2c0d22b0 msgr2=0x7fac2c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2b7fe640 1 --2- 172.21.15.77:0/3606627213 >> v2:172.21.15.158:3300/0 conn(0x7fac2c0d22b0 0x7fac2c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2b7fe640 1 -- 172.21.15.77:0/3606627213 >> v1:172.21.15.77:6789/0 conn(0x7fac2c15dfe0 legacy=0x7fac2c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2b7fe640 1 -- 172.21.15.77:0/3606627213 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0b3710 con 0x7fac2c0b41f0 2024-03-06T07:29:53.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.573+0000 7fac2a7fc640 1 --2- 172.21.15.77:0/3606627213 >> v2:172.21.15.158:3300/0 conn(0x7fac2c0d22b0 0x7fac2c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.574+0000 7fac2b7fe640 1 --2- 172.21.15.77:0/3606627213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c0b41f0 0x7fac2c0b4680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fac24009d90 tx=0x7fac2402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63b6b632520deefa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.574+0000 7fac29ffb640 1 -- 172.21.15.77:0/3606627213 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24039070 con 0x7fac2c0b41f0 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.574+0000 7fac29ffb640 1 -- 172.21.15.77:0/3606627213 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2402cc50 con 0x7fac2c0b41f0 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.574+0000 7fac29ffb640 1 -- 172.21.15.77:0/3606627213 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24034440 con 0x7fac2c0b41f0 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.575+0000 7fac31864640 1 -- 172.21.15.77:0/3606627213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c0b41f0 msgr2=0x7fac2c0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.575+0000 7fac31864640 1 --2- 172.21.15.77:0/3606627213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c0b41f0 0x7fac2c0b4680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fac24009d90 tx=0x7fac2402bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.575+0000 7fac31864640 1 -- 172.21.15.77:0/3606627213 shutdown_connections 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.575+0000 7fac31864640 1 --2- 172.21.15.77:0/3606627213 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c0b41f0 0x7fac2c0b4680 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.575+0000 7fac31864640 1 --2- 172.21.15.77:0/3606627213 >> v2:172.21.15.158:3300/0 conn(0x7fac2c0d22b0 0x7fac2c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.575+0000 7fac31864640 1 -- 172.21.15.77:0/3606627213 >> 172.21.15.77:0/3606627213 conn(0x7fac2c0d3cd0 msgr2=0x7fac2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.575+0000 7fac31864640 1 -- 172.21.15.77:0/3606627213 shutdown_connections 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.575+0000 7fac31864640 1 -- 172.21.15.77:0/3606627213 wait complete. 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.576+0000 7fac31864640 1 Processor -- start 2024-03-06T07:29:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.576+0000 7fac31864640 1 -- start start 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac31864640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac2c0b41f0 0x7fac2c0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac31864640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac2c0d22b0 0x7fac2c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac31864640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c15dfe0 0x7fac2c0cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac31864640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0cfa00 con 0x7fac2c0b41f0 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac31864640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0b3d60 con 0x7fac2c0d22b0 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac31864640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0d1c70 con 0x7fac2c15dfe0 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac2c0b41f0 0x7fac2c0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac2c0b41f0 0x7fac2c0cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41556/0 (socket says 172.21.15.77:41556) 2024-03-06T07:29:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 -- 172.21.15.77:0/534671610 learned_addr learned my addr 172.21.15.77:0/534671610 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2a7fc640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac2c0d22b0 0x7fac2c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2b7fe640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c15dfe0 0x7fac2c0cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 -- 172.21.15.77:0/534671610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c15dfe0 msgr2=0x7fac2c0cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c15dfe0 0x7fac2c0cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 -- 172.21.15.77:0/534671610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac2c0d22b0 msgr2=0x7fac2c1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac2c0d22b0 0x7fac2c1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 -- 172.21.15.77:0/534671610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0cc6f0 con 0x7fac2c0b41f0 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.577+0000 7fac2affd640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac2c0b41f0 0x7fac2c0cc1b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c390 tx=0x7fac1c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.578+0000 7fac13fff640 1 -- 172.21.15.77:0/534671610 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c017070 con 0x7fac2c0b41f0 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.578+0000 7fac13fff640 1 -- 172.21.15.77:0/534671610 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1c005ce0 con 0x7fac2c0b41f0 2024-03-06T07:29:53.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.578+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c153610 con 0x7fac2c0b41f0 2024-03-06T07:29:53.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.578+0000 7fac13fff640 1 -- 172.21.15.77:0/534671610 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0072f0 con 0x7fac2c0b41f0 2024-03-06T07:29:53.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.578+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c0cb3f0 con 0x7fac2c0b41f0 2024-03-06T07:29:53.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.579+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c04f330 con 0x7fac2c0b41f0 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.579+0000 7fac13fff640 1 -- 172.21.15.77:0/534671610 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac1c01a070 con 0x7fac2c0b41f0 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.579+0000 7fac13fff640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.580+0000 7fac13fff640 1 -- 172.21.15.77:0/534671610 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fac1c089630 con 0x7fac2c0b41f0 2024-03-06T07:29:53.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.580+0000 7fac2a7fc640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.580+0000 7fac2a7fc640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac1806b4d0 0x7fac1806d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fac20009560 tx=0x7fac20009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.585+0000 7fac13fff640 1 -- 172.21.15.77:0/534671610 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac1c057da0 con 0x7fac2c0b41f0 2024-03-06T07:29:53.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.793+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac2c06c9a0 con 0x7fac2c0b41f0 2024-03-06T07:29:53.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.794+0000 7fac13fff640 1 -- 172.21.15.77:0/534671610 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+3402 (secure 0 0 0) 0x7fac1c05be40 con 0x7fac2c0b41f0 2024-03-06T07:29:53.797 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:53.797 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8973,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/4200914134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4200914134},{"type":"v1","addr":"172.21.15.77:6835","nonce":4200914134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:47.721340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 91 2024-03-06T07:29:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac1806b4d0 msgr2=0x7fac1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac1806b4d0 0x7fac1806d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fac20009560 tx=0x7fac20009190 comp rx=0 tx=0).stop 2024-03-06T07:29:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac2c0b41f0 msgr2=0x7fac2c0cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac2c0b41f0 0x7fac2c0cc1b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c390 tx=0x7fac1c00c850 comp rx=0 tx=0).stop 2024-03-06T07:29:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 shutdown_connections 2024-03-06T07:29:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fac2c15dfe0 0x7fac2c0cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fac2c0d22b0 0x7fac2c1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 --2- 172.21.15.77:0/534671610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac2c0b41f0 0x7fac2c0cc1b0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 >> 172.21.15.77:0/534671610 conn(0x7fac2c0d3cd0 msgr2=0x7fac2c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 shutdown_connections 2024-03-06T07:29:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.796+0000 7fac31864640 1 -- 172.21.15.77:0/534671610 wait complete. 2024-03-06T07:29:53.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:53.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.908+0000 7f4d24d0f640 1 Processor -- start 2024-03-06T07:29:53.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.908+0000 7f4d24d0f640 1 -- start start 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d24d0f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4d200d22b0 0x7f4d200d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d24d0f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4d200b41f0 0x7f4d200b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d24d0f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d200b3430 con 0x7f4d2015dfe0 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d24d0f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d200d2c50 con 0x7f4d200b41f0 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d24d0f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d200b31e0 con 0x7f4d200d22b0 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d1ed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4d200b41f0 0x7f4d200b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d1ed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4d200b41f0 0x7f4d200b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41562/0 (socket says 172.21.15.77:41562) 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d1ed9d640 1 -- 172.21.15.77:0/3779197382 learned_addr learned my addr 172.21.15.77:0/3779197382 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.909+0000 7f4d1dd9b640 1 --2- 172.21.15.77:0/3779197382 >> v2:172.21.15.158:3300/0 conn(0x7f4d200d22b0 0x7f4d200d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.910+0000 7f4d1ed9d640 1 -- 172.21.15.77:0/3779197382 >> v2:172.21.15.158:3300/0 conn(0x7f4d200d22b0 msgr2=0x7f4d200d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.910+0000 7f4d1ed9d640 1 --2- 172.21.15.77:0/3779197382 >> v2:172.21.15.158:3300/0 conn(0x7f4d200d22b0 0x7f4d200d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.910+0000 7f4d1ed9d640 1 -- 172.21.15.77:0/3779197382 >> v1:172.21.15.158:6789/0 conn(0x7f4d2015dfe0 legacy=0x7f4d200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.910+0000 7f4d1ed9d640 1 -- 172.21.15.77:0/3779197382 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d200b3710 con 0x7f4d200b41f0 2024-03-06T07:29:53.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.910+0000 7f4d1ed9d640 1 --2- 172.21.15.77:0/3779197382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200b41f0 0x7f4d200b4680 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009870 tx=0x7f4d1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a56aff5b94c7fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d1d59a640 1 -- 172.21.15.77:0/3779197382 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18034070 con 0x7f4d200b41f0 2024-03-06T07:29:53.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d1d59a640 1 -- 172.21.15.77:0/3779197382 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d18030860 con 0x7f4d200b41f0 2024-03-06T07:29:53.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d1d59a640 1 -- 172.21.15.77:0/3779197382 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18030c30 con 0x7f4d200b41f0 2024-03-06T07:29:53.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d24d0f640 1 -- 172.21.15.77:0/3779197382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200b41f0 msgr2=0x7f4d200b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/3779197382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200b41f0 0x7f4d200b4680 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009870 tx=0x7f4d1802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d24d0f640 1 -- 172.21.15.77:0/3779197382 shutdown_connections 2024-03-06T07:29:53.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/3779197382 >> v2:172.21.15.158:3300/0 conn(0x7f4d200d22b0 0x7f4d200d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/3779197382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200b41f0 0x7f4d200b4680 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d24d0f640 1 -- 172.21.15.77:0/3779197382 >> 172.21.15.77:0/3779197382 conn(0x7f4d200d3cd0 msgr2=0x7f4d200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d24d0f640 1 -- 172.21.15.77:0/3779197382 shutdown_connections 2024-03-06T07:29:53.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.911+0000 7f4d24d0f640 1 -- 172.21.15.77:0/3779197382 wait complete. 2024-03-06T07:29:53.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.912+0000 7f4d24d0f640 1 Processor -- start 2024-03-06T07:29:53.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d24d0f640 1 -- start start 2024-03-06T07:29:53.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d24d0f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d200b41f0 0x7f4d201f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d24d0f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200d22b0 0x7f4d200cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d24d0f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d2015dfe0 0x7f4d20149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d24d0f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200cfa00 con 0x7f4d200d22b0 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d24d0f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200b3d60 con 0x7f4d2015dfe0 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d24d0f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200d1c70 con 0x7f4d200b41f0 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d1dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200d22b0 0x7f4d200cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d1dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200d22b0 0x7f4d200cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41576/0 (socket says 172.21.15.77:41576) 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d1dd9b640 1 -- 172.21.15.77:0/2832888658 learned_addr learned my addr 172.21.15.77:0/2832888658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d1ed9d640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d2015dfe0 0x7f4d20149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.913+0000 7f4d1e59c640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d200b41f0 0x7f4d201f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d1dd9b640 1 -- 172.21.15.77:0/2832888658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d200b41f0 msgr2=0x7f4d201f6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d1dd9b640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d200b41f0 0x7f4d201f6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d1dd9b640 1 -- 172.21.15.77:0/2832888658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d2015dfe0 msgr2=0x7f4d20149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d1dd9b640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d2015dfe0 0x7f4d20149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d1dd9b640 1 -- 172.21.15.77:0/2832888658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d200b70b0 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d1dd9b640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200d22b0 0x7f4d200cd1a0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400e9e0 tx=0x7f4d1400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d0effd640 1 -- 172.21.15.77:0/2832888658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d14017070 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d0effd640 1 -- 172.21.15.77:0/2832888658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1400c9d0 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d200b6af0 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d0effd640 1 -- 172.21.15.77:0/2832888658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1400cdc0 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.914+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d200cc9a0 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.915+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d200cc7d0 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.915+0000 7f4d0effd640 1 -- 172.21.15.77:0/2832888658 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d1401a070 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.916+0000 7f4d0effd640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d0006b4d0 0x7f4d0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.916+0000 7f4d0effd640 1 -- 172.21.15.77:0/2832888658 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4d1408a260 con 0x7f4d200d22b0 2024-03-06T07:29:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.916+0000 7f4d1e59c640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d0006b4d0 0x7f4d0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.916+0000 7f4d1e59c640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d0006b4d0 0x7f4d0006d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4d10001ca0 tx=0x7f4d10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:53.920+0000 7f4d0effd640 1 -- 172.21.15.77:0/2832888658 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d200b70b0 con 0x7f4d200d22b0 2024-03-06T07:29:54.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.128+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d200d0860 con 0x7f4d200d22b0 2024-03-06T07:29:54.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.128+0000 7f4d0effd640 1 -- 172.21.15.77:0/2832888658 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+3402 (secure 0 0 0) 0x7f4d140589c0 con 0x7f4d200d22b0 2024-03-06T07:29:54.130 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:54.130 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8973,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/4200914134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4200914134},{"type":"v1","addr":"172.21.15.77:6835","nonce":4200914134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:32.114016+0000","modified":"2024-03-06T07:29:47.721340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8547},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8547":{"gid":8547,"name":"b","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6835/1245424942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6833","nonce":1245424942},{"type":"v1","addr":"172.21.15.158:6835","nonce":1245424942}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8547,"qdb_cluster":[8547]},"id":9}]} 2024-03-06T07:29:54.131 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 91 2024-03-06T07:29:54.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d0006b4d0 msgr2=0x7f4d0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d0006b4d0 0x7f4d0006d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4d10001ca0 tx=0x7f4d10009040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200d22b0 msgr2=0x7f4d200cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200d22b0 0x7f4d200cd1a0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400e9e0 tx=0x7f4d1400eea0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 shutdown_connections 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4d0006b4d0 0x7f4d0006d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d200d22b0 0x7f4d200cd1a0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4d2015dfe0 0x7f4d20149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 --2- 172.21.15.77:0/2832888658 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4d200b41f0 0x7f4d201f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 >> 172.21.15.77:0/2832888658 conn(0x7f4d200d3cd0 msgr2=0x7f4d200b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 shutdown_connections 2024-03-06T07:29:54.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.130+0000 7f4d24d0f640 1 -- 172.21.15.77:0/2832888658 wait complete. 2024-03-06T07:29:54.144 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.240+0000 7fcf0deaf640 1 Processor -- start 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.241+0000 7fcf0deaf640 1 -- start start 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.241+0000 7fcf0deaf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcf080d0de0 0x7fcf08168e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.241+0000 7fcf0deaf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf080b7b60 con 0x7fcf0815ddf0 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.241+0000 7fcf0deaf640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf080d18d0 con 0x7fcf0815e1d0 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.241+0000 7fcf0deaf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf080d27c0 con 0x7fcf080d0de0 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.241+0000 7fcf077fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcf0815ddf0 0x7fcf080cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56666/0 (socket says 172.21.15.77:56666) 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.241+0000 7fcf077fe640 1 -- 172.21.15.77:0/3161059717 learned_addr learned my addr 172.21.15.77:0/3161059717 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.242+0000 7fcf07fff640 1 --2- 172.21.15.77:0/3161059717 >> v2:172.21.15.158:3300/0 conn(0x7fcf080d0de0 0x7fcf08168e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.242+0000 7fcf067fc640 1 -- 172.21.15.77:0/3161059717 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169031456 0 0) 0x7fcf080b7b60 con 0x7fcf0815ddf0 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.242+0000 7fcf067fc640 1 -- 172.21.15.77:0/3161059717 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef0003520 con 0x7fcf0815ddf0 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.242+0000 7fcf07fff640 1 -- 172.21.15.77:0/3161059717 >> v1:172.21.15.158:6789/0 conn(0x7fcf0815e1d0 legacy=0x7fcf080d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.242+0000 7fcf07fff640 1 -- 172.21.15.77:0/3161059717 >> v1:172.21.15.77:6789/0 conn(0x7fcf0815ddf0 legacy=0x7fcf080cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.242+0000 7fcf07fff640 1 -- 172.21.15.77:0/3161059717 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf080d1bb0 con 0x7fcf080d0de0 2024-03-06T07:29:54.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf07fff640 1 --2- 172.21.15.77:0/3161059717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf080d0de0 0x7fcf08168e10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009d90 tx=0x7fcf0002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6252489a06f1779d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf067fc640 1 -- 172.21.15.77:0/3161059717 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf00039070 con 0x7fcf080d0de0 2024-03-06T07:29:54.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf067fc640 1 -- 172.21.15.77:0/3161059717 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0002cd00 con 0x7fcf080d0de0 2024-03-06T07:29:54.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf067fc640 1 -- 172.21.15.77:0/3161059717 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf00034660 con 0x7fcf080d0de0 2024-03-06T07:29:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3161059717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf080d0de0 msgr2=0x7fcf08168e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf0deaf640 1 --2- 172.21.15.77:0/3161059717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf080d0de0 0x7fcf08168e10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009d90 tx=0x7fcf0002bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3161059717 shutdown_connections 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf0deaf640 1 --2- 172.21.15.77:0/3161059717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf080d0de0 0x7fcf08168e10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3161059717 >> 172.21.15.77:0/3161059717 conn(0x7fcf0809cff0 msgr2=0x7fcf080d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3161059717 shutdown_connections 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.243+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3161059717 wait complete. 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.244+0000 7fcf0deaf640 1 Processor -- start 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf0deaf640 1 -- start start 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf0deaf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf080d0de0 0x7fcf080cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf0deaf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0815ddf0 0x7fcf080c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf0deaf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf0815e1d0 0x7fcf08158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf0deaf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080d2450 con 0x7fcf0815e1d0 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf0deaf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080d2170 con 0x7fcf0815ddf0 2024-03-06T07:29:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf0deaf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080d2c50 con 0x7fcf080d0de0 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf07fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf0815e1d0 0x7fcf08158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf07fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf0815e1d0 0x7fcf08158260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41592/0 (socket says 172.21.15.77:41592) 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf07fff640 1 -- 172.21.15.77:0/3132846965 learned_addr learned my addr 172.21.15.77:0/3132846965 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf06ffd640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0815ddf0 0x7fcf080c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.245+0000 7fcf077fe640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf080d0de0 0x7fcf080cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf07fff640 1 -- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf080d0de0 msgr2=0x7fcf080cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf07fff640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf080d0de0 0x7fcf080cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf07fff640 1 -- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0815ddf0 msgr2=0x7fcf080c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf07fff640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0815ddf0 0x7fcf080c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf07fff640 1 -- 172.21.15.77:0/3132846965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf081613d0 con 0x7fcf0815e1d0 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf077fe640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf080d0de0 0x7fcf080cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf07fff640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf0815e1d0 0x7fcf08158260 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fcf00037040 tx=0x7fcf00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fceeffff640 1 -- 172.21.15.77:0/3132846965 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf00044020 con 0x7fcf0815e1d0 2024-03-06T07:29:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fceeffff640 1 -- 172.21.15.77:0/3132846965 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0003a040 con 0x7fcf0815e1d0 2024-03-06T07:29:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf081e7b10 con 0x7fcf0815e1d0 2024-03-06T07:29:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fceeffff640 1 -- 172.21.15.77:0/3132846965 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf00039050 con 0x7fcf0815e1d0 2024-03-06T07:29:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.246+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf080c6d20 con 0x7fcf0815e1d0 2024-03-06T07:29:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.247+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0804f330 con 0x7fcf0815e1d0 2024-03-06T07:29:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.247+0000 7fceeffff640 1 -- 172.21.15.77:0/3132846965 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf00003bd0 con 0x7fcf0815e1d0 2024-03-06T07:29:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.248+0000 7fceeffff640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcef406b500 0x7fcef406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.248+0000 7fceeffff640 1 -- 172.21.15.77:0/3132846965 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcf000ac750 con 0x7fcf0815e1d0 2024-03-06T07:29:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.248+0000 7fcf077fe640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcef406b500 0x7fcef406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.248+0000 7fcf077fe640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcef406b500 0x7fcef406d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcef80011e0 tx=0x7fcef8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.252+0000 7fceeffff640 1 -- 172.21.15.77:0/3132846965 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf0004e060 con 0x7fcf0815e1d0 2024-03-06T07:29:54.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.426+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf0815bf20 con 0x7fcf0815e1d0 2024-03-06T07:29:54.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.427+0000 7fceeffff640 1 -- 172.21.15.77:0/3132846965 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16764 (secure 0 0 0) 0x7fcf0007b620 con 0x7fcf0815e1d0 2024-03-06T07:29:54.429 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:54.430 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":129,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:28:51.262074+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:30.343353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:31.094099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.9","mappings":[{"from":2,"to":0}]},{"pgid":"19.12","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:54.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcef406b500 msgr2=0x7fcef406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcef406b500 0x7fcef406d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcef80011e0 tx=0x7fcef8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf0815e1d0 msgr2=0x7fcf08158260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf0815e1d0 0x7fcf08158260 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fcf00037040 tx=0x7fcf00004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 shutdown_connections 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcef406b500 0x7fcef406d9c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf0815e1d0 0x7fcf08158260 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf0815ddf0 0x7fcf080c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 --2- 172.21.15.77:0/3132846965 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf080d0de0 0x7fcf080cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.429+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 >> 172.21.15.77:0/3132846965 conn(0x7fcf0809cff0 msgr2=0x7fcf080b6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.430+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 shutdown_connections 2024-03-06T07:29:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.430+0000 7fcf0deaf640 1 -- 172.21.15.77:0/3132846965 wait complete. 2024-03-06T07:29:54.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:29:54.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.541+0000 7f39a44d4640 1 Processor -- start 2024-03-06T07:29:54.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.541+0000 7f39a44d4640 1 -- start start 2024-03-06T07:29:54.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.542+0000 7f39a44d4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f399c0b4460 0x7f399c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.542+0000 7f39a44d4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f399c161cf0 0x7f399c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.542+0000 7f39a44d4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c0d22c0 con 0x7f399c162970 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.542+0000 7f39a44d4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399c161b00 con 0x7f399c0b4460 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.542+0000 7f39a44d4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399c0cecc0 con 0x7f399c161cf0 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a2270640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f399c0b4460 0x7f399c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a1a6f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f399c162970 0x7f399c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56678/0 (socket says 172.21.15.77:56678) 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a2270640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f399c0b4460 0x7f399c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41596/0 (socket says 172.21.15.77:41596) 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a1a6f640 1 -- 172.21.15.77:0/1686329147 learned_addr learned my addr 172.21.15.77:0/1686329147 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a2a71640 1 --2- 172.21.15.77:0/1686329147 >> v2:172.21.15.158:3300/0 conn(0x7f399c161cf0 0x7f399c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a2270640 1 -- 172.21.15.77:0/1686329147 >> v2:172.21.15.158:3300/0 conn(0x7f399c161cf0 msgr2=0x7f399c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a2270640 1 --2- 172.21.15.77:0/1686329147 >> v2:172.21.15.158:3300/0 conn(0x7f399c161cf0 0x7f399c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a2270640 1 -- 172.21.15.77:0/1686329147 >> v1:172.21.15.77:6789/0 conn(0x7f399c162970 legacy=0x7f399c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a2270640 1 -- 172.21.15.77:0/1686329147 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c0d25a0 con 0x7f399c0b4460 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.543+0000 7f39a126e640 1 -- 172.21.15.77:0/1686329147 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17240362 0 0) 0x7f399c0d22c0 con 0x7f399c162970 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a2270640 1 --2- 172.21.15.77:0/1686329147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c0b4460 0x7f399c0d4800 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f399c08c900 tx=0x7f399802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e329dfcc97a411c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a126e640 1 -- 172.21.15.77:0/1686329147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998035020 con 0x7f399c0b4460 2024-03-06T07:29:54.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a126e640 1 -- 172.21.15.77:0/1686329147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3998031830 con 0x7f399c0b4460 2024-03-06T07:29:54.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a126e640 1 -- 172.21.15.77:0/1686329147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998031c00 con 0x7f399c0b4460 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a44d4640 1 -- 172.21.15.77:0/1686329147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c0b4460 msgr2=0x7f399c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a44d4640 1 --2- 172.21.15.77:0/1686329147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c0b4460 0x7f399c0d4800 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f399c08c900 tx=0x7f399802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a44d4640 1 -- 172.21.15.77:0/1686329147 shutdown_connections 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a44d4640 1 --2- 172.21.15.77:0/1686329147 >> v2:172.21.15.158:3300/0 conn(0x7f399c161cf0 0x7f399c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a44d4640 1 --2- 172.21.15.77:0/1686329147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c0b4460 0x7f399c0d4800 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a44d4640 1 -- 172.21.15.77:0/1686329147 >> 172.21.15.77:0/1686329147 conn(0x7f399c0d3ce0 msgr2=0x7f399c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a44d4640 1 -- 172.21.15.77:0/1686329147 shutdown_connections 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.544+0000 7f39a44d4640 1 -- 172.21.15.77:0/1686329147 wait complete. 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.545+0000 7f39a44d4640 1 Processor -- start 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.545+0000 7f39a44d4640 1 -- start start 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a44d4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399c0b4460 0x7f399c159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a44d4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399c161cf0 0x7f399c0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a44d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c162970 0x7f399c0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a44d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0ce950 con 0x7f399c162970 2024-03-06T07:29:54.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a44d4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0d2b60 con 0x7f399c0b4460 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a44d4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0b4840 con 0x7f399c161cf0 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a2a71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c162970 0x7f399c0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a2a71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c162970 0x7f399c0cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41602/0 (socket says 172.21.15.77:41602) 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a2a71640 1 -- 172.21.15.77:0/2483722239 learned_addr learned my addr 172.21.15.77:0/2483722239 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a2270640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399c0b4460 0x7f399c159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a1a6f640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399c161cf0 0x7f399c0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a2a71640 1 -- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399c161cf0 msgr2=0x7f399c0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.546+0000 7f39a2a71640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399c161cf0 0x7f399c0cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f39a2a71640 1 -- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399c0b4460 msgr2=0x7f399c159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f39a2a71640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399c0b4460 0x7f399c159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f39a2a71640 1 -- 172.21.15.77:0/2483722239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c15a0a0 con 0x7f399c162970 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f39a2270640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399c0b4460 0x7f399c159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f39a1a6f640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399c161cf0 0x7f399c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f39a2a71640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c162970 0x7f399c0cd3d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f3994009dd0 tx=0x7f399400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f3992ffd640 1 -- 172.21.15.77:0/2483722239 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3994018020 con 0x7f399c162970 2024-03-06T07:29:54.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f3992ffd640 1 -- 172.21.15.77:0/2483722239 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3994014480 con 0x7f399c162970 2024-03-06T07:29:54.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399c0ca490 con 0x7f399c162970 2024-03-06T07:29:54.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f3992ffd640 1 -- 172.21.15.77:0/2483722239 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3994014870 con 0x7f399c162970 2024-03-06T07:29:54.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.547+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399c0c86a0 con 0x7f399c162970 2024-03-06T07:29:54.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.548+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399c06d1c0 con 0x7f399c162970 2024-03-06T07:29:54.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.548+0000 7f3992ffd640 1 -- 172.21.15.77:0/2483722239 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3994005d10 con 0x7f399c162970 2024-03-06T07:29:54.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.549+0000 7f3992ffd640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398006b4d0 0x7f398006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.549+0000 7f39a2270640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398006b4d0 0x7f398006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.549+0000 7f3992ffd640 1 -- 172.21.15.77:0/2483722239 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f399408adf0 con 0x7f399c162970 2024-03-06T07:29:54.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.549+0000 7f39a2270640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398006b4d0 0x7f398006d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3998036000 tx=0x7f3998008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.553+0000 7f3992ffd640 1 -- 172.21.15.77:0/2483722239 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f399c06d1c0 con 0x7f399c162970 2024-03-06T07:29:54.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.758+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f399c04f330 con 0x7f399c162970 2024-03-06T07:29:54.760 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:29:54.758+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:29:54.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.759+0000 7f3992ffd640 1 -- 172.21.15.77:0/2483722239 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v92) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3994022070 con 0x7f399c162970 2024-03-06T07:29:54.761 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398006b4d0 msgr2=0x7f398006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398006b4d0 0x7f398006d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3998036000 tx=0x7f3998008030 comp rx=0 tx=0).stop 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c162970 msgr2=0x7f399c0cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c162970 0x7f399c0cd3d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f3994009dd0 tx=0x7f399400c490 comp rx=0 tx=0).stop 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 shutdown_connections 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f398006b4d0 0x7f398006d990 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f399c162970 0x7f399c0cd3d0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f399c161cf0 0x7f399c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 --2- 172.21.15.77:0/2483722239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f399c0b4460 0x7f399c159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 >> 172.21.15.77:0/2483722239 conn(0x7f399c0d3ce0 msgr2=0x7f399c0d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 shutdown_connections 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.762+0000 7f39a44d4640 1 -- 172.21.15.77:0/2483722239 wait complete. 2024-03-06T07:29:54.768 INFO:tasks.ceph.mds.b.smithi158.stderr: -8878> 2024-03-06T07:28:38.044+0000 7fdf9d25e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:54.768 INFO:tasks.ceph.mds.b.smithi158.stderr: -8149> 2024-03-06T07:28:48.044+0000 7fdf9d25e640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:29:54.775 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:29:54.800 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:29:54.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.876+0000 7f6b40393640 1 Processor -- start 2024-03-06T07:29:54.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.876+0000 7f6b40393640 1 -- start start 2024-03-06T07:29:54.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b40393640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b380cfa90 0x7f6b380cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b40393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b380b33a0 0x7f6b380b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b40393640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b38158ac0 con 0x7f6b381586e0 2024-03-06T07:29:54.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b40393640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b3815b2a0 con 0x7f6b380cfa90 2024-03-06T07:29:54.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b40393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3815b540 con 0x7f6b380b33a0 2024-03-06T07:29:54.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3e930640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b380b33a0 0x7f6b380b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3d92e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6b380cfa90 0x7f6b380cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3e12f640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6b381586e0 0x7f6b380aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44342/0 (socket says 172.21.15.77:44342) 2024-03-06T07:29:54.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3e12f640 1 -- 172.21.15.77:0/2706608009 learned_addr learned my addr 172.21.15.77:0/2706608009 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:54.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3e930640 1 -- 172.21.15.77:0/2706608009 >> v2:172.21.15.158:3300/0 conn(0x7f6b380cfa90 msgr2=0x7f6b380cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3e930640 1 --2- 172.21.15.77:0/2706608009 >> v2:172.21.15.158:3300/0 conn(0x7f6b380cfa90 0x7f6b380cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3e930640 1 -- 172.21.15.77:0/2706608009 >> v1:172.21.15.158:6789/0 conn(0x7f6b381586e0 legacy=0x7f6b380aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3e930640 1 -- 172.21.15.77:0/2706608009 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3815b820 con 0x7f6b380b33a0 2024-03-06T07:29:54.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.877+0000 7f6b3d92e640 1 --2- 172.21.15.77:0/2706608009 >> v2:172.21.15.158:3300/0 conn(0x7f6b380cfa90 0x7f6b380cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:54.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.878+0000 7f6b3e930640 1 --2- 172.21.15.77:0/2706608009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b380b33a0 0x7f6b380b37c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009d90 tx=0x7f6b3002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac8ec50cf01d5af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.878+0000 7f6b3d12d640 1 -- 172.21.15.77:0/2706608009 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30039070 con 0x7f6b380b33a0 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.878+0000 7f6b3d12d640 1 -- 172.21.15.77:0/2706608009 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3002cc50 con 0x7f6b380b33a0 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.878+0000 7f6b3d12d640 1 -- 172.21.15.77:0/2706608009 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30034440 con 0x7f6b380b33a0 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.879+0000 7f6b40393640 1 -- 172.21.15.77:0/2706608009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b380b33a0 msgr2=0x7f6b380b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.879+0000 7f6b40393640 1 --2- 172.21.15.77:0/2706608009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b380b33a0 0x7f6b380b37c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009d90 tx=0x7f6b3002bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.879+0000 7f6b40393640 1 -- 172.21.15.77:0/2706608009 shutdown_connections 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.879+0000 7f6b40393640 1 --2- 172.21.15.77:0/2706608009 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b380b33a0 0x7f6b380b37c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.879+0000 7f6b40393640 1 --2- 172.21.15.77:0/2706608009 >> v2:172.21.15.158:3300/0 conn(0x7f6b380cfa90 0x7f6b380cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.879+0000 7f6b40393640 1 -- 172.21.15.77:0/2706608009 >> 172.21.15.77:0/2706608009 conn(0x7f6b3806b190 msgr2=0x7f6b3806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.879+0000 7f6b40393640 1 -- 172.21.15.77:0/2706608009 shutdown_connections 2024-03-06T07:29:54.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.879+0000 7f6b40393640 1 -- 172.21.15.77:0/2706608009 wait complete. 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.880+0000 7f6b40393640 1 Processor -- start 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.880+0000 7f6b40393640 1 -- start start 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b40393640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380b33a0 0x7f6b381420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b40393640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cfa90 0x7f6b380ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b40393640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b381586e0 0x7f6b3806e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b40393640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380af150 con 0x7f6b380cfa90 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b40393640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3815bde0 con 0x7f6b380b33a0 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b40393640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b38071350 con 0x7f6b381586e0 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3d92e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cfa90 0x7f6b380ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3d92e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cfa90 0x7f6b380ce430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41612/0 (socket says 172.21.15.77:41612) 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3d92e640 1 -- 172.21.15.77:0/825586888 learned_addr learned my addr 172.21.15.77:0/825586888 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3e930640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b381586e0 0x7f6b3806e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3e12f640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380b33a0 0x7f6b381420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3d92e640 1 -- 172.21.15.77:0/825586888 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b381586e0 msgr2=0x7f6b3806e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3d92e640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b381586e0 0x7f6b3806e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3d92e640 1 -- 172.21.15.77:0/825586888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380b33a0 msgr2=0x7f6b381420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3d92e640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380b33a0 0x7f6b381420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.881+0000 7f6b3d92e640 1 -- 172.21.15.77:0/825586888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b38142da0 con 0x7f6b380cfa90 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.882+0000 7f6b3d92e640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cfa90 0x7f6b380ce430 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800ec10 tx=0x7f6b2800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.882+0000 7f6b2effd640 1 -- 172.21.15.77:0/825586888 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28017070 con 0x7f6b380cfa90 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.882+0000 7f6b2effd640 1 -- 172.21.15.77:0/825586888 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2800cc30 con 0x7f6b380cfa90 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.882+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3806cfd0 con 0x7f6b380cfa90 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.882+0000 7f6b2effd640 1 -- 172.21.15.77:0/825586888 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28013470 con 0x7f6b380cfa90 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.882+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3806d590 con 0x7f6b380cfa90 2024-03-06T07:29:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.883+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b380732b0 con 0x7f6b380cfa90 2024-03-06T07:29:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.883+0000 7f6b2effd640 1 -- 172.21.15.77:0/825586888 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b2801a070 con 0x7f6b380cfa90 2024-03-06T07:29:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.884+0000 7f6b2effd640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.884+0000 7f6b2effd640 1 -- 172.21.15.77:0/825586888 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f6b2808a480 con 0x7f6b380cfa90 2024-03-06T07:29:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.884+0000 7f6b3e12f640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.884+0000 7f6b3e12f640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6b34007fd0 tx=0x7f6b34007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:54.888+0000 7f6b2effd640 1 -- 172.21.15.77:0/825586888 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b28058cb0 con 0x7f6b380cfa90 2024-03-06T07:29:55.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.097+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6b380b0ea0 con 0x7f6b380cfa90 2024-03-06T07:29:55.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.761+0000 7f6b2effd640 1 -- 172.21.15.77:0/825586888 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v93) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6b2805cd50 con 0x7f6b380cfa90 2024-03-06T07:29:55.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 msgr2=0x7f6b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6b34007fd0 tx=0x7f6b34007e80 comp rx=0 tx=0).stop 2024-03-06T07:29:55.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cfa90 msgr2=0x7f6b380ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cfa90 0x7f6b380ce430 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800ec10 tx=0x7f6b2800c520 comp rx=0 tx=0).stop 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 shutdown_connections 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6b1c06b4d0 0x7f6b1c06d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b380cfa90 0x7f6b380ce430 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b381586e0 0x7f6b3806e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 --2- 172.21.15.77:0/825586888 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b380b33a0 0x7f6b381420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 >> 172.21.15.77:0/825586888 conn(0x7f6b3806b190 msgr2=0x7f6b3815a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 shutdown_connections 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.763+0000 7f6b40393640 1 -- 172.21.15.77:0/825586888 wait complete. 2024-03-06T07:29:55.778 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:29:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.882+0000 7f2639c40640 1 Processor -- start 2024-03-06T07:29:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.882+0000 7f2639c40640 1 -- start start 2024-03-06T07:29:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f2639c40640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f263415a490 0x7f263406df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f2639c40640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26340b3fd0 0x7f26340b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f2639c40640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263406e4d0 con 0x7f26340b2e90 2024-03-06T07:29:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f2639c40640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26340b4490 con 0x7f263415a490 2024-03-06T07:29:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f2639c40640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26340cf4a0 con 0x7f26340b3fd0 2024-03-06T07:29:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f2633fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f26340b2e90 0x7f26340708b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56682/0 (socket says 172.21.15.77:56682) 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f2633fff640 1 -- 172.21.15.77:0/2289569104 learned_addr learned my addr 172.21.15.77:0/2289569104 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147839966 0 0) 0x7f262c001490 con 0x7f26340b2e90 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c003520 con 0x7f26340b2e90 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f26337fe640 1 --2- 172.21.15.77:0/2289569104 >> v2:172.21.15.158:3300/0 conn(0x7f263415a490 0x7f263406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f2632ffd640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26340b3fd0 0x7f26340b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.883+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034837522 0 0) 0x7f261c003520 con 0x7f26340b2e90 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26340b3fd0 msgr2=0x7f26340b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26340b3fd0 0x7f26340b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 >> v2:172.21.15.158:3300/0 conn(0x7f263415a490 msgr2=0x7f263406df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 --2- 172.21.15.77:0/2289569104 >> v2:172.21.15.158:3300/0 conn(0x7f263415a490 0x7f263406df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26340b4750 con 0x7f26340b2e90 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f262c001bc0 con 0x7f26340b2e90 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 >> v1:172.21.15.77:6789/0 conn(0x7f26340b2e90 legacy=0x7f26340708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26340b3fd0 0x7f261c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f261c003fe0 0x7f261c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c004d40 0x7f261c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26340b4490 con 0x7f261c003fe0 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26340cf4a0 con 0x7f261c004d40 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261c0059e0 con 0x7f26340b3fd0 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26337fe640 1 --2- 172.21.15.77:0/2289569104 >> v2:172.21.15.158:3300/0 conn(0x7f263415a490 0x7f263406df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f2632ffd640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f261c003fe0 0x7f261c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f2633fff640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c004d40 0x7f261c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f26337fe640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26340b3fd0 0x7f261c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f2632ffd640 1 -- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26340b3fd0 msgr2=0x7f261c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f2632ffd640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f26340b3fd0 0x7f261c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f2632ffd640 1 -- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c004d40 msgr2=0x7f261c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f2632ffd640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c004d40 0x7f261c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f2632ffd640 1 -- 172.21.15.77:0/2289569104 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c003520 con 0x7f261c003fe0 2024-03-06T07:29:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.884+0000 7f2633fff640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c004d40 0x7f261c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f2632ffd640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f261c003fe0 0x7f261c004800 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f262800ba30 tx=0x7f261c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455cb56efda58cd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628036020 con 0x7f261c003fe0 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2628032640 con 0x7f261c003fe0 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f26327fc640 1 -- 172.21.15.77:0/2289569104 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262800b600 con 0x7f261c003fe0 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f2639c40640 1 -- 172.21.15.77:0/2289569104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f261c003fe0 msgr2=0x7f261c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f2639c40640 1 --2- 172.21.15.77:0/2289569104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f261c003fe0 0x7f261c004800 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f262800ba30 tx=0x7f261c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f2633fff640 1 -- 172.21.15.77:0/2289569104 reap_dead start 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f2639c40640 1 -- 172.21.15.77:0/2289569104 shutdown_connections 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f2639c40640 1 -- 172.21.15.77:0/2289569104 >> 172.21.15.77:0/2289569104 conn(0x7f26340b1650 msgr2=0x7f26340b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f2639c40640 1 -- 172.21.15.77:0/2289569104 shutdown_connections 2024-03-06T07:29:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.885+0000 7f2639c40640 1 -- 172.21.15.77:0/2289569104 wait complete. 2024-03-06T07:29:55.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.887+0000 7f2639c40640 1 Processor -- start 2024-03-06T07:29:55.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.887+0000 7f2639c40640 1 -- start start 2024-03-06T07:29:55.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.887+0000 7f2639c40640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c003fe0 0x7f263414eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.887+0000 7f2639c40640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2634155770 0x7f263415a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.887+0000 7f2639c40640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2634158080 0x7f2634158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.887+0000 7f2639c40640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26340cf260 con 0x7f2634158080 2024-03-06T07:29:55.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.887+0000 7f2639c40640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26340cf0e0 con 0x7f261c003fe0 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.887+0000 7f2639c40640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26340b3320 con 0x7f2634155770 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2634158080 0x7f2634158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2634158080 0x7f2634158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41632/0 (socket says 172.21.15.77:41632) 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 -- 172.21.15.77:0/390055416 learned_addr learned my addr 172.21.15.77:0/390055416 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f26337fe640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c003fe0 0x7f263414eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2632ffd640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2634155770 0x7f263415a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 -- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2634155770 msgr2=0x7f263415a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2634155770 0x7f263415a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 -- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c003fe0 msgr2=0x7f263414eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c003fe0 0x7f263414eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 -- 172.21.15.77:0/390055416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263414eff0 con 0x7f2634158080 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f26337fe640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c003fe0 0x7f263414eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2633fff640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2634158080 0x7f2634158500 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f262c0013d0 tx=0x7f262c007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f261bfff640 1 -- 172.21.15.77:0/390055416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c004e30 con 0x7f2634158080 2024-03-06T07:29:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f261bfff640 1 -- 172.21.15.77:0/390055416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c00a780 con 0x7f2634158080 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263415b170 con 0x7f2634158080 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f261bfff640 1 -- 172.21.15.77:0/390055416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c00ab50 con 0x7f2634158080 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.888+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263415b730 con 0x7f2634158080 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.889+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26340687d0 con 0x7f2634158080 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.889+0000 7f261bfff640 1 -- 172.21.15.77:0/390055416 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f262c01b020 con 0x7f2634158080 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.890+0000 7f261bfff640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f262006b500 0x7f262006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.890+0000 7f261bfff640 1 -- 172.21.15.77:0/390055416 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f262c089f60 con 0x7f2634158080 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.890+0000 7f26337fe640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f262006b500 0x7f262006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.890+0000 7f26337fe640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f262006b500 0x7f262006d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2624006fd0 tx=0x7f2624008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:55.894+0000 7f261bfff640 1 -- 172.21.15.77:0/390055416 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f263406e4d0 con 0x7f2634158080 2024-03-06T07:29:56.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.066+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f26340cbbd0 con 0x7f2634158080 2024-03-06T07:29:56.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.762+0000 7f261bfff640 1 -- 172.21.15.77:0/390055416 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v131) v1 ==== 181+0+0 (secure 0 0 0) 0x7f262c058de0 con 0x7f2634158080 2024-03-06T07:29:56.765 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f262006b500 msgr2=0x7f262006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f262006b500 0x7f262006d9c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2624006fd0 tx=0x7f2624008040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2634158080 msgr2=0x7f2634158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2634158080 0x7f2634158500 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f262c0013d0 tx=0x7f262c007ad0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 shutdown_connections 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f262006b500 0x7f262006d9c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2634158080 0x7f2634158500 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2634155770 0x7f263415a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 --2- 172.21.15.77:0/390055416 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f261c003fe0 0x7f263414eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.766+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 >> 172.21.15.77:0/390055416 conn(0x7f26340b1650 msgr2=0x7f26340ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.767+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 shutdown_connections 2024-03-06T07:29:56.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.767+0000 7f2639c40640 1 -- 172.21.15.77:0/390055416 wait complete. 2024-03-06T07:29:56.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:29:56.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.890+0000 7f7b4017a640 1 Processor -- start 2024-03-06T07:29:56.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.890+0000 7f7b4017a640 1 -- start start 2024-03-06T07:29:56.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b4017a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b380708b0 0x7f7b38070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b4017a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b380ce0a0 0x7f7b380ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b4017a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b3815afe0 con 0x7f7b3815ac00 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b4017a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b38071250 con 0x7f7b380ce0a0 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b4017a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380cdc10 con 0x7f7b380708b0 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3e717640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b380ce0a0 0x7f7b380ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3e717640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b380ce0a0 0x7f7b380ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41646/0 (socket says 172.21.15.77:41646) 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3df16640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7b3815ac00 0x7f7b3806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56696/0 (socket says 172.21.15.77:56696) 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3e717640 1 -- 172.21.15.77:0/3347521808 learned_addr learned my addr 172.21.15.77:0/3347521808 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3d715640 1 --2- 172.21.15.77:0/3347521808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b380708b0 0x7f7b38070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3e717640 1 -- 172.21.15.77:0/3347521808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b380708b0 msgr2=0x7f7b38070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3e717640 1 --2- 172.21.15.77:0/3347521808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b380708b0 0x7f7b38070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3e717640 1 -- 172.21.15.77:0/3347521808 >> v1:172.21.15.77:6789/0 conn(0x7f7b3815ac00 legacy=0x7f7b3806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3cf14640 1 -- 172.21.15.77:0/3347521808 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178011228 0 0) 0x7f7b3815afe0 con 0x7f7b3815ac00 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3e717640 1 -- 172.21.15.77:0/3347521808 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b380cd000 con 0x7f7b380ce0a0 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.891+0000 7f7b3d715640 1 --2- 172.21.15.77:0/3347521808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b380708b0 0x7f7b38070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.892+0000 7f7b3e717640 1 --2- 172.21.15.77:0/3347521808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b380ce520 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009e50 tx=0x7f7b3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e80bb78b1257ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.892+0000 7f7b3cf14640 1 -- 172.21.15.77:0/3347521808 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3002ca70 con 0x7f7b380ce0a0 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.892+0000 7f7b3cf14640 1 -- 172.21.15.77:0/3347521808 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3002cc10 con 0x7f7b380ce0a0 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.892+0000 7f7b3cf14640 1 -- 172.21.15.77:0/3347521808 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b300343f0 con 0x7f7b380ce0a0 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.892+0000 7f7b4017a640 1 -- 172.21.15.77:0/3347521808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 msgr2=0x7f7b380ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.892+0000 7f7b4017a640 1 --2- 172.21.15.77:0/3347521808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b380ce520 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009e50 tx=0x7f7b3002bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.893+0000 7f7b4017a640 1 -- 172.21.15.77:0/3347521808 shutdown_connections 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.893+0000 7f7b4017a640 1 --2- 172.21.15.77:0/3347521808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b380ce520 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.893+0000 7f7b4017a640 1 --2- 172.21.15.77:0/3347521808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b380708b0 0x7f7b38070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.893+0000 7f7b4017a640 1 -- 172.21.15.77:0/3347521808 >> 172.21.15.77:0/3347521808 conn(0x7f7b380b1650 msgr2=0x7f7b380b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.893+0000 7f7b4017a640 1 -- 172.21.15.77:0/3347521808 shutdown_connections 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.893+0000 7f7b4017a640 1 -- 172.21.15.77:0/3347521808 wait complete. 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b4017a640 1 Processor -- start 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b4017a640 1 -- start start 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b4017a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b380708b0 0x7f7b38149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b4017a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b38159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b4017a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b3815ac00 0x7f7b3815d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b4017a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380cd8a0 con 0x7f7b380ce0a0 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b4017a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380cd5c0 con 0x7f7b380708b0 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b4017a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380cf5d0 con 0x7f7b3815ac00 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.894+0000 7f7b3d715640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b38159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3d715640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b38159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41656/0 (socket says 172.21.15.77:41656) 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3d715640 1 -- 172.21.15.77:0/4293899795 learned_addr learned my addr 172.21.15.77:0/4293899795 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:56.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3e717640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b3815ac00 0x7f7b3815d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3df16640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b380708b0 0x7f7b38149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3d715640 1 -- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b3815ac00 msgr2=0x7f7b3815d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3d715640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b3815ac00 0x7f7b3815d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3d715640 1 -- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b380708b0 msgr2=0x7f7b38149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3d715640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b380708b0 0x7f7b38149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3d715640 1 -- 172.21.15.77:0/4293899795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3815b9f0 con 0x7f7b380ce0a0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3e717640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b3815ac00 0x7f7b3815d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3df16640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b380708b0 0x7f7b38149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b3d715640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b38159600 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800c820 tx=0x7f7b2800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b2e7fc640 1 -- 172.21.15.77:0/4293899795 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28004230 con 0x7f7b380ce0a0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b2e7fc640 1 -- 172.21.15.77:0/4293899795 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b280043d0 con 0x7f7b380ce0a0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3815bcd0 con 0x7f7b380ce0a0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b2e7fc640 1 -- 172.21.15.77:0/4293899795 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28007a70 con 0x7f7b380ce0a0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.895+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3815c290 con 0x7f7b380ce0a0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.896+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b380687d0 con 0x7f7b380ce0a0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.897+0000 7f7b2e7fc640 1 -- 172.21.15.77:0/4293899795 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b28007c10 con 0x7f7b380ce0a0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.897+0000 7f7b2e7fc640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.897+0000 7f7b2e7fc640 1 -- 172.21.15.77:0/4293899795 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f7b2808a160 con 0x7f7b380ce0a0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.897+0000 7f7b3df16640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.898+0000 7f7b3df16640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7b34000f60 tx=0x7f7b34007cc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:56.901+0000 7f7b2e7fc640 1 -- 172.21.15.77:0/4293899795 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b28015020 con 0x7f7b380ce0a0 2024-03-06T07:29:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.073+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7b380ae630 con 0x7f7b380ce0a0 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.764+0000 7f7b2e7fc640 1 -- 172.21.15.77:0/4293899795 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v132) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7b28059100 con 0x7f7b380ce0a0 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:29:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b1c06b4d0 msgr2=0x7f7b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7b34000f60 tx=0x7f7b34007cc0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 msgr2=0x7f7b38159600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b38159600 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800c820 tx=0x7f7b2800cce0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 shutdown_connections 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7b1c06b4d0 0x7f7b1c06d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b380ce0a0 0x7f7b38159600 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7b3815ac00 0x7f7b3815d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 --2- 172.21.15.77:0/4293899795 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7b380708b0 0x7f7b38149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.767+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 >> 172.21.15.77:0/4293899795 conn(0x7f7b380b1650 msgr2=0x7f7b380b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.768+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 shutdown_connections 2024-03-06T07:29:57.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.768+0000 7f7b4017a640 1 -- 172.21.15.77:0/4293899795 wait complete. 2024-03-06T07:29:57.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:29:57.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.890+0000 7ff58b888640 1 Processor -- start 2024-03-06T07:29:57.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.890+0000 7ff58b888640 1 -- start start 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff58b888640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5840cff40 0x7ff5840b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff58b888640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff584158ac0 con 0x7ff5840cfa90 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff58b888640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58415b2a0 con 0x7ff5841586e0 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff58b888640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5840d0390 con 0x7ff5840cff40 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff589e25640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5840cff40 0x7ff5840b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff588e23640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff5840cfa90 0x7ff5840b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56708/0 (socket says 172.21.15.77:56708) 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff588e23640 1 -- 172.21.15.77:0/2966650340 learned_addr learned my addr 172.21.15.77:0/2966650340 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff589e25640 1 -- 172.21.15.77:0/2966650340 >> v1:172.21.15.158:6789/0 conn(0x7ff5841586e0 legacy=0x7ff5840aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff589e25640 1 -- 172.21.15.77:0/2966650340 >> v1:172.21.15.77:6789/0 conn(0x7ff5840cfa90 legacy=0x7ff5840b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.891+0000 7ff589e25640 1 -- 172.21.15.77:0/2966650340 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58415b540 con 0x7ff5840cff40 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff589e25640 1 --2- 172.21.15.77:0/2966650340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5840b3ab0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009d90 tx=0x7ff57c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924e9b3b042a71ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff57bfff640 1 -- 172.21.15.77:0/2966650340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c039070 con 0x7ff5840cff40 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff57bfff640 1 -- 172.21.15.77:0/2966650340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c02cc60 con 0x7ff5840cff40 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff57bfff640 1 -- 172.21.15.77:0/2966650340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c034660 con 0x7ff5840cff40 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff58b888640 1 -- 172.21.15.77:0/2966650340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 msgr2=0x7ff5840b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff58b888640 1 --2- 172.21.15.77:0/2966650340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5840b3ab0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009d90 tx=0x7ff57c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff58b888640 1 -- 172.21.15.77:0/2966650340 shutdown_connections 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff58b888640 1 --2- 172.21.15.77:0/2966650340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5840b3ab0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.892+0000 7ff58b888640 1 -- 172.21.15.77:0/2966650340 >> 172.21.15.77:0/2966650340 conn(0x7ff58406b190 msgr2=0x7ff58406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.893+0000 7ff58b888640 1 -- 172.21.15.77:0/2966650340 shutdown_connections 2024-03-06T07:29:57.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.893+0000 7ff58b888640 1 -- 172.21.15.77:0/2966650340 wait complete. 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff58b888640 1 Processor -- start 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff58b888640 1 -- start start 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff58b888640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5840cfa90 0x7ff584148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff58b888640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5841e30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff58b888640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5841586e0 0x7ff5841e3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff58b888640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58415bde0 con 0x7ff5840cff40 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff58b888640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58415bb00 con 0x7ff5840cfa90 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff58b888640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5840af150 con 0x7ff5841586e0 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff588e23640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5841e30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff588e23640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5841e30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41666/0 (socket says 172.21.15.77:41666) 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.894+0000 7ff588e23640 1 -- 172.21.15.77:0/1945662523 learned_addr learned my addr 172.21.15.77:0/1945662523 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:57.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff589624640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5840cfa90 0x7ff584148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff588e23640 1 -- 172.21.15.77:0/1945662523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5841586e0 msgr2=0x7ff5841e3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff588e23640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5841586e0 0x7ff5841e3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff588e23640 1 -- 172.21.15.77:0/1945662523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5840cfa90 msgr2=0x7ff584148430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff588e23640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5840cfa90 0x7ff584148430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff588e23640 1 -- 172.21.15.77:0/1945662523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5841f5220 con 0x7ff5840cff40 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff589624640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5840cfa90 0x7ff584148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff588e23640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5841e30e0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7ff58000c4a0 tx=0x7ff58000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff579ffb640 1 -- 172.21.15.77:0/1945662523 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff580017070 con 0x7ff5840cff40 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff579ffb640 1 -- 172.21.15.77:0/1945662523 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff580005ce0 con 0x7ff5840cff40 2024-03-06T07:29:57.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5841f54e0 con 0x7ff5840cff40 2024-03-06T07:29:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff579ffb640 1 -- 172.21.15.77:0/1945662523 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff580007270 con 0x7ff5840cff40 2024-03-06T07:29:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.895+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5841f5aa0 con 0x7ff5840cff40 2024-03-06T07:29:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.896+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58406a1f0 con 0x7ff5840cff40 2024-03-06T07:29:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.896+0000 7ff579ffb640 1 -- 172.21.15.77:0/1945662523 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff58001a070 con 0x7ff5840cff40 2024-03-06T07:29:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.897+0000 7ff579ffb640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff56806b4d0 0x7ff56806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.897+0000 7ff589624640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff56806b4d0 0x7ff56806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.897+0000 7ff579ffb640 1 -- 172.21.15.77:0/1945662523 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff580089130 con 0x7ff5840cff40 2024-03-06T07:29:57.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.898+0000 7ff589624640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff56806b4d0 0x7ff56806d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff5840cc0d0 tx=0x7ff574008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:57.906+0000 7ff579ffb640 1 -- 172.21.15.77:0/1945662523 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff58406a1f0 con 0x7ff5840cff40 2024-03-06T07:29:58.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.080+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff5840732b0 con 0x7ff5840cff40 2024-03-06T07:29:58.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.083+0000 7ff579ffb640 1 -- 172.21.15.77:0/1945662523 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff580057c60 con 0x7ff5840cff40 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff56806b4d0 msgr2=0x7ff56806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff56806b4d0 0x7ff56806d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff5840cc0d0 tx=0x7ff574008040 comp rx=0 tx=0).stop 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 msgr2=0x7ff5841e30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5841e30e0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7ff58000c4a0 tx=0x7ff58000c960 comp rx=0 tx=0).stop 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 shutdown_connections 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff56806b4d0 0x7ff56806d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5840cff40 0x7ff5841e30e0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff5841586e0 0x7ff5841e3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 --2- 172.21.15.77:0/1945662523 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff5840cfa90 0x7ff584148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 >> 172.21.15.77:0/1945662523 conn(0x7ff58406b190 msgr2=0x7ff5840b33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 shutdown_connections 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.085+0000 7ff58b888640 1 -- 172.21.15.77:0/1945662523 wait complete. 2024-03-06T07:29:58.098 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:29:58.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.197+0000 7f305ec66640 1 Processor -- start 2024-03-06T07:29:58.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.197+0000 7f305ec66640 1 -- start start 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.197+0000 7f305ec66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30580cce40 0x7f305815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.197+0000 7f305ec66640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30580cd7a0 0x7f30580cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.197+0000 7f305ec66640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3058071f40 con 0x7f30581515e0 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.197+0000 7f305ec66640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30580cd2c0 con 0x7f30580cd7a0 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.197+0000 7f305ec66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3058070a70 con 0x7f30580cce40 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.198+0000 7f305d203640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30580cd7a0 0x7f30580cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.198+0000 7f305d203640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30580cd7a0 0x7f30580cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41672/0 (socket says 172.21.15.77:41672) 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.198+0000 7f305d203640 1 -- 172.21.15.77:0/3275460175 learned_addr learned my addr 172.21.15.77:0/3275460175 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:58.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.198+0000 7f3057fff640 1 --2- 172.21.15.77:0/3275460175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30580cce40 0x7f305815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.198+0000 7f305d203640 1 -- 172.21.15.77:0/3275460175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30580cce40 msgr2=0x7f305815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.198+0000 7f305d203640 1 --2- 172.21.15.77:0/3275460175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30580cce40 0x7f305815b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.198+0000 7f305d203640 1 -- 172.21.15.77:0/3275460175 >> v1:172.21.15.158:6789/0 conn(0x7f30581515e0 legacy=0x7f3058071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.198+0000 7f305d203640 1 -- 172.21.15.77:0/3275460175 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3058070d50 con 0x7f30580cd7a0 2024-03-06T07:29:58.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.199+0000 7f305d203640 1 --2- 172.21.15.77:0/3275460175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f30580cdbd0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f3050009e50 tx=0x7f305002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a8ff801b9cd06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.199+0000 7f30577fe640 1 -- 172.21.15.77:0/3275460175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305002cb00 con 0x7f30580cd7a0 2024-03-06T07:29:58.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.199+0000 7f30577fe640 1 -- 172.21.15.77:0/3275460175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305002cca0 con 0x7f30580cd7a0 2024-03-06T07:29:58.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.199+0000 7f30577fe640 1 -- 172.21.15.77:0/3275460175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050034610 con 0x7f30580cd7a0 2024-03-06T07:29:58.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.199+0000 7f305ec66640 1 -- 172.21.15.77:0/3275460175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 msgr2=0x7f30580cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.199+0000 7f305ec66640 1 --2- 172.21.15.77:0/3275460175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f30580cdbd0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f3050009e50 tx=0x7f305002bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.200+0000 7f305ec66640 1 -- 172.21.15.77:0/3275460175 shutdown_connections 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.200+0000 7f305ec66640 1 --2- 172.21.15.77:0/3275460175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30580cce40 0x7f305815b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.200+0000 7f305ec66640 1 --2- 172.21.15.77:0/3275460175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f30580cdbd0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.200+0000 7f305ec66640 1 -- 172.21.15.77:0/3275460175 >> 172.21.15.77:0/3275460175 conn(0x7f30580cf8a0 msgr2=0x7f30580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.200+0000 7f305ec66640 1 -- 172.21.15.77:0/3275460175 shutdown_connections 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.200+0000 7f305ec66640 1 -- 172.21.15.77:0/3275460175 wait complete. 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.201+0000 7f305ec66640 1 Processor -- start 2024-03-06T07:29:58.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.201+0000 7f305ec66640 1 -- start start 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ec66640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30580cce40 0x7f3058153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ec66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f305815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ec66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30581515e0 0x7f305814d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ec66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580ce110 con 0x7f30580cd7a0 2024-03-06T07:29:58.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ec66640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058071310 con 0x7f30580cce40 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ec66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30580cc3a0 con 0x7f30581515e0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f305815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f305815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41682/0 (socket says 172.21.15.77:41682) 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 -- 172.21.15.77:0/3341949218 learned_addr learned my addr 172.21.15.77:0/3341949218 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305d203640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30581515e0 0x7f305814d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ca02640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30580cce40 0x7f3058153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 -- 172.21.15.77:0/3341949218 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30581515e0 msgr2=0x7f305814d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30581515e0 0x7f305814d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 -- 172.21.15.77:0/3341949218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30580cce40 msgr2=0x7f3058153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30580cce40 0x7f3058153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 -- 172.21.15.77:0/3341949218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305815e2f0 con 0x7f30580cd7a0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f3057fff640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f305815ddb0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f304400d830 tx=0x7f304400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f30557fa640 1 -- 172.21.15.77:0/3341949218 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304400b660 con 0x7f30580cd7a0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f30557fa640 1 -- 172.21.15.77:0/3341949218 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3044019070 con 0x7f30580cd7a0 2024-03-06T07:29:58.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f30557fa640 1 -- 172.21.15.77:0/3341949218 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304400ba20 con 0x7f30580cd7a0 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3058158680 con 0x7f30580cd7a0 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.202+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3058158c40 con 0x7f30580cd7a0 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.204+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3058068840 con 0x7f30580cd7a0 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.204+0000 7f30557fa640 1 -- 172.21.15.77:0/3341949218 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f304401e460 con 0x7f30580cd7a0 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.204+0000 7f30557fa640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f303c06b4d0 0x7f303c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.204+0000 7f30557fa640 1 -- 172.21.15.77:0/3341949218 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f304408a4f0 con 0x7f30580cd7a0 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.204+0000 7f305ca02640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f303c06b4d0 0x7f303c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.205+0000 7f305ca02640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f303c06b4d0 0x7f303c06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f304c007fd0 tx=0x7f304c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.209+0000 7f30557fa640 1 -- 172.21.15.77:0/3341949218 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3044059700 con 0x7f30580cd7a0 2024-03-06T07:29:58.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.447+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f305804f330 con 0x7f30580cd7a0 2024-03-06T07:29:58.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.765+0000 7f30557fa640 1 -- 172.21.15.77:0/3341949218 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v422) v1 ==== 140+0+0 (secure 0 0 0) 0x7f3044059000 con 0x7f30580cd7a0 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.767+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f303c06b4d0 msgr2=0x7f303c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.767+0000 7f305ec66640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f303c06b4d0 0x7f303c06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f304c007fd0 tx=0x7f304c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.767+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 msgr2=0x7f305815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.767+0000 7f305ec66640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f305815ddb0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f304400d830 tx=0x7f304400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.768+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 shutdown_connections 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.768+0000 7f305ec66640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f303c06b4d0 0x7f303c06d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.768+0000 7f305ec66640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30580cd7a0 0x7f305815ddb0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.768+0000 7f305ec66640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f30581515e0 0x7f305814d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.768+0000 7f305ec66640 1 --2- 172.21.15.77:0/3341949218 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f30580cce40 0x7f3058153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.768+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 >> 172.21.15.77:0/3341949218 conn(0x7f30580cf8a0 msgr2=0x7f30580b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.768+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 shutdown_connections 2024-03-06T07:29:58.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.768+0000 7f305ec66640 1 -- 172.21.15.77:0/3341949218 wait complete. 2024-03-06T07:29:58.781 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 ... ok 2024-03-06T07:29:58.782 INFO:tasks.cephfs_test_runner:Starting test: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:29:58.782 INFO:tasks.cephfs_test_runner:test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:29:58.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.880+0000 7f6e717c9640 1 Processor -- start 2024-03-06T07:29:58.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.880+0000 7f6e717c9640 1 -- start start 2024-03-06T07:29:58.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e717c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c0d4760 0x7f6e6c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e717c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6e6c0d0a80 0x7f6e6c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e717c9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c0d25d0 con 0x7f6e6c0b4170 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e717c9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6c0d4bf0 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e717c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0ce9e0 con 0x7f6e6c0d4760 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e6b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6e6c0d0a80 0x7f6e6c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e6b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6e6c0d0a80 0x7f6e6c0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60738/0 (socket says 172.21.15.77:60738) 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e6b7fe640 1 -- 172.21.15.77:0/3755886941 learned_addr learned my addr 172.21.15.77:0/3755886941 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e6a7fc640 1 --2- 172.21.15.77:0/3755886941 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c0d4760 0x7f6e6c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e6b7fe640 1 -- 172.21.15.77:0/3755886941 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c0d4760 msgr2=0x7f6e6c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e6b7fe640 1 --2- 172.21.15.77:0/3755886941 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c0d4760 0x7f6e6c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e6b7fe640 1 -- 172.21.15.77:0/3755886941 >> v1:172.21.15.158:6789/0 conn(0x7f6e6c0b4170 legacy=0x7f6e6c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.881+0000 7f6e6b7fe640 1 -- 172.21.15.77:0/3755886941 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0d2870 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.882+0000 7f6e6b7fe640 1 --2- 172.21.15.77:0/3755886941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c0cee70 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009870 tx=0x7f6e6402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b0e0b9322258a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.882+0000 7f6e69ffb640 1 -- 172.21.15.77:0/3755886941 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64034070 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.882+0000 7f6e69ffb640 1 -- 172.21.15.77:0/3755886941 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e64030840 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.882+0000 7f6e69ffb640 1 -- 172.21.15.77:0/3755886941 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64030c10 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.883+0000 7f6e717c9640 1 -- 172.21.15.77:0/3755886941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 msgr2=0x7f6e6c0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.883+0000 7f6e717c9640 1 --2- 172.21.15.77:0/3755886941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c0cee70 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009870 tx=0x7f6e6402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.883+0000 7f6e717c9640 1 -- 172.21.15.77:0/3755886941 shutdown_connections 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.883+0000 7f6e717c9640 1 --2- 172.21.15.77:0/3755886941 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c0d4760 0x7f6e6c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.883+0000 7f6e717c9640 1 --2- 172.21.15.77:0/3755886941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c0cee70 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.883+0000 7f6e717c9640 1 -- 172.21.15.77:0/3755886941 >> 172.21.15.77:0/3755886941 conn(0x7f6e6c0d3c40 msgr2=0x7f6e6c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.883+0000 7f6e717c9640 1 -- 172.21.15.77:0/3755886941 shutdown_connections 2024-03-06T07:29:58.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.883+0000 7f6e717c9640 1 -- 172.21.15.77:0/3755886941 wait complete. 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.884+0000 7f6e717c9640 1 Processor -- start 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.884+0000 7f6e717c9640 1 -- start start 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e717c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e6c0b4170 0x7f6e6c156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e717c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c09d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e717c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c09de40 0x7f6e6c0b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e717c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0b4dd0 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e717c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0b4550 con 0x7f6e6c0b4170 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e717c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0b4f50 con 0x7f6e6c09de40 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c09d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c09d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60752/0 (socket says 172.21.15.77:60752) 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6a7fc640 1 -- 172.21.15.77:0/247301835 learned_addr learned my addr 172.21.15.77:0/247301835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6affd640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e6c0b4170 0x7f6e6c156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6a7fc640 1 -- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c09de40 msgr2=0x7f6e6c0b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6b7fe640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c09de40 0x7f6e6c0b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6a7fc640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c09de40 0x7f6e6c0b82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6a7fc640 1 -- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e6c0b4170 msgr2=0x7f6e6c156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6a7fc640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e6c0b4170 0x7f6e6c156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6a7fc640 1 -- 172.21.15.77:0/247301835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0b90a0 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.885+0000 7f6e6affd640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e6c0b4170 0x7f6e6c156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.886+0000 7f6e6a7fc640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c09d900 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000d620 tx=0x7f6e6000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.886+0000 7f6e537fe640 1 -- 172.21.15.77:0/247301835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000b620 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.886+0000 7f6e537fe640 1 -- 172.21.15.77:0/247301835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60019070 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.886+0000 7f6e537fe640 1 -- 172.21.15.77:0/247301835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000ba00 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.886+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c09fb20 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.886+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c0a00e0 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.887+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c09e250 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.887+0000 7f6e537fe640 1 -- 172.21.15.77:0/247301835 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e6000bba0 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.887+0000 7f6e537fe640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5806b500 0x7f6e5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.888+0000 7f6e537fe640 1 -- 172.21.15.77:0/247301835 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6e6008a300 con 0x7f6e6c0d0a80 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.888+0000 7f6e6affd640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5806b500 0x7f6e5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.888+0000 7f6e6affd640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5806b500 0x7f6e5806d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c001d70 tx=0x7f6e5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:58.892+0000 7f6e537fe640 1 -- 172.21.15.77:0/247301835 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e6c09e250 con 0x7f6e6c0d0a80 2024-03-06T07:29:59.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.062+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e6c04f330 con 0x7f6e6c0d0a80 2024-03-06T07:29:59.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.063+0000 7f6e537fe640 1 -- 172.21.15.77:0/247301835 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+13020 (secure 0 0 0) 0x7f6e60059480 con 0x7f6e6c0d0a80 2024-03-06T07:29:59.065 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:29:59.065 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":132,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:29:57.763699+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.158:6835/1245424942":"2024-03-07T07:29:54.759541+0000","172.21.15.158:6833/1245424942":"2024-03-07T07:29:54.759541+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:59.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5806b500 msgr2=0x7f6e5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5806b500 0x7f6e5806d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c001d70 tx=0x7f6e5c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 msgr2=0x7f6e6c09d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c09d900 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000d620 tx=0x7f6e6000dae0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 shutdown_connections 2024-03-06T07:29:59.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5806b500 0x7f6e5806d9c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e6c0d0a80 0x7f6e6c09d900 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e6c09de40 0x7f6e6c0b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 --2- 172.21.15.77:0/247301835 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e6c0b4170 0x7f6e6c156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 >> 172.21.15.77:0/247301835 conn(0x7f6e6c0d3c40 msgr2=0x7f6e6c0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 shutdown_connections 2024-03-06T07:29:59.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.065+0000 7f6e717c9640 1 -- 172.21.15.77:0/247301835 wait complete. 2024-03-06T07:29:59.078 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:59.079 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:59.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.180+0000 7fb883484640 1 Processor -- start 2024-03-06T07:29:59.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.180+0000 7fb883484640 1 -- start start 2024-03-06T07:29:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb883484640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb87c15a220 0x7fb87c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb883484640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c0712d0 0x7fb87c071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb883484640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0b3320 con 0x7fb87c15de60 2024-03-06T07:29:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb883484640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c0cf610 con 0x7fb87c15a220 2024-03-06T07:29:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb883484640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c071c70 con 0x7fb87c0712d0 2024-03-06T07:29:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb880a1f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c0712d0 0x7fb87c071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb881220640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb87c15a220 0x7fb87c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb881a21640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb87c15de60 0x7fb87c15e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58574/0 (socket says 172.21.15.77:58574) 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb881a21640 1 -- 172.21.15.77:0/2407038442 learned_addr learned my addr 172.21.15.77:0/2407038442 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb873fff640 1 -- 172.21.15.77:0/2407038442 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657435856 0 0) 0x7fb87c0b3320 con 0x7fb87c15de60 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb873fff640 1 -- 172.21.15.77:0/2407038442 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb868003520 con 0x7fb87c15de60 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb880a1f640 1 -- 172.21.15.77:0/2407038442 >> v2:172.21.15.158:3300/0 conn(0x7fb87c15a220 msgr2=0x7fb87c0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb880a1f640 1 --2- 172.21.15.77:0/2407038442 >> v2:172.21.15.158:3300/0 conn(0x7fb87c15a220 0x7fb87c0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb880a1f640 1 -- 172.21.15.77:0/2407038442 >> v1:172.21.15.158:6789/0 conn(0x7fb87c15de60 legacy=0x7fb87c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb880a1f640 1 -- 172.21.15.77:0/2407038442 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0728d0 con 0x7fb87c0712d0 2024-03-06T07:29:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.181+0000 7fb881220640 1 --2- 172.21.15.77:0/2407038442 >> v2:172.21.15.158:3300/0 conn(0x7fb87c15a220 0x7fb87c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.182+0000 7fb880a1f640 1 --2- 172.21.15.77:0/2407038442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c0712d0 0x7fb87c071730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0097a0 tx=0x7fb86c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1711ccf1c7056d45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.182+0000 7fb873fff640 1 -- 172.21.15.77:0/2407038442 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0315a0 con 0x7fb87c0712d0 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.182+0000 7fb873fff640 1 -- 172.21.15.77:0/2407038442 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c037070 con 0x7fb87c0712d0 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.182+0000 7fb873fff640 1 -- 172.21.15.77:0/2407038442 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c031940 con 0x7fb87c0712d0 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.183+0000 7fb883484640 1 -- 172.21.15.77:0/2407038442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c0712d0 msgr2=0x7fb87c071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.183+0000 7fb883484640 1 --2- 172.21.15.77:0/2407038442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c0712d0 0x7fb87c071730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0097a0 tx=0x7fb86c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.183+0000 7fb883484640 1 -- 172.21.15.77:0/2407038442 shutdown_connections 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.183+0000 7fb883484640 1 --2- 172.21.15.77:0/2407038442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c0712d0 0x7fb87c071730 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.183+0000 7fb883484640 1 --2- 172.21.15.77:0/2407038442 >> v2:172.21.15.158:3300/0 conn(0x7fb87c15a220 0x7fb87c0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.183+0000 7fb883484640 1 -- 172.21.15.77:0/2407038442 >> 172.21.15.77:0/2407038442 conn(0x7fb87c06b190 msgr2=0x7fb87c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.183+0000 7fb883484640 1 -- 172.21.15.77:0/2407038442 shutdown_connections 2024-03-06T07:29:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.183+0000 7fb883484640 1 -- 172.21.15.77:0/2407038442 wait complete. 2024-03-06T07:29:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.184+0000 7fb883484640 1 Processor -- start 2024-03-06T07:29:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.184+0000 7fb883484640 1 -- start start 2024-03-06T07:29:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb883484640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c0712d0 0x7fb87c14e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb883484640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb87c15a220 0x7fb87c151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb883484640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c15de60 0x7fb87c1505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb883484640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c072030 con 0x7fb87c15a220 2024-03-06T07:29:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb883484640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c072bb0 con 0x7fb87c0712d0 2024-03-06T07:29:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb883484640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c072530 con 0x7fb87c15de60 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb87c15a220 0x7fb87c151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb87c15a220 0x7fb87c151ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60764/0 (socket says 172.21.15.77:60764) 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 -- 172.21.15.77:0/2607052772 learned_addr learned my addr 172.21.15.77:0/2607052772 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb881220640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c0712d0 0x7fb87c14e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb881a21640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c15de60 0x7fb87c1505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 -- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c15de60 msgr2=0x7fb87c1505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c15de60 0x7fb87c1505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 -- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c0712d0 msgr2=0x7fb87c14e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c0712d0 0x7fb87c14e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 -- 172.21.15.77:0/2607052772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c14ec60 con 0x7fb87c15a220 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb881220640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c0712d0 0x7fb87c14e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb880a1f640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb87c15a220 0x7fb87c151ec0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fb86c034040 tx=0x7fb86c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb871ffb640 1 -- 172.21.15.77:0/2607052772 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0317a0 con 0x7fb87c15a220 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb871ffb640 1 -- 172.21.15.77:0/2607052772 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c031940 con 0x7fb87c15a220 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.185+0000 7fb871ffb640 1 -- 172.21.15.77:0/2607052772 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c031cf0 con 0x7fb87c15a220 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.186+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c152810 con 0x7fb87c15a220 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.186+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c0af580 con 0x7fb87c15a220 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.187+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c0732b0 con 0x7fb87c15a220 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.187+0000 7fb871ffb640 1 -- 172.21.15.77:0/2607052772 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb86c032430 con 0x7fb87c15a220 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.187+0000 7fb871ffb640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb86006b4d0 0x7fb86006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.188+0000 7fb871ffb640 1 -- 172.21.15.77:0/2607052772 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb86c02f020 con 0x7fb87c15a220 2024-03-06T07:29:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.188+0000 7fb881220640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb86006b4d0 0x7fb86006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.188+0000 7fb881220640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb86006b4d0 0x7fb86006d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb87800abc0 tx=0x7fb878008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.192+0000 7fb871ffb640 1 -- 172.21.15.77:0/2607052772 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb86c031e90 con 0x7fb87c15a220 2024-03-06T07:29:59.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.363+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb87c068ff0 con 0x7fb87c15a220 2024-03-06T07:29:59.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.363+0000 7fb871ffb640 1 -- 172.21.15.77:0/2607052772 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb86c07b250 con 0x7fb87c15a220 2024-03-06T07:29:59.365 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:29:59.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb86006b4d0 msgr2=0x7fb86006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb86006b4d0 0x7fb86006d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb87800abc0 tx=0x7fb878008040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb87c15a220 msgr2=0x7fb87c151ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb87c15a220 0x7fb87c151ec0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fb86c034040 tx=0x7fb86c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 shutdown_connections 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb86006b4d0 0x7fb86006d990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb87c15a220 0x7fb87c151ec0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb87c15de60 0x7fb87c1505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 --2- 172.21.15.77:0/2607052772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb87c0712d0 0x7fb87c14e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 >> 172.21.15.77:0/2607052772 conn(0x7fb87c06b190 msgr2=0x7fb87c0cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 shutdown_connections 2024-03-06T07:29:59.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.365+0000 7fb883484640 1 -- 172.21.15.77:0/2607052772 wait complete. 2024-03-06T07:29:59.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:29:59.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.476+0000 7f5db9e88640 1 Processor -- start 2024-03-06T07:29:59.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.476+0000 7f5db9e88640 1 -- start start 2024-03-06T07:29:59.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.477+0000 7f5db9e88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5db40d16e0 0x7f5db40d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.477+0000 7f5db9e88640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db40cebe0 con 0x7f5db415bdd0 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.477+0000 7f5db9e88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db40ceec0 con 0x7f5db415fa10 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.477+0000 7f5db9e88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db40ce980 con 0x7f5db40d16e0 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.477+0000 7f5db3fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5db40d16e0 0x7f5db40d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.477+0000 7f5db37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5db415bdd0 0x7f5db40c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50338/0 (socket says 172.21.15.77:50338) 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.477+0000 7f5db37fe640 1 -- 172.21.15.77:0/1721934245 learned_addr learned my addr 172.21.15.77:0/1721934245 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.478+0000 7f5db3fff640 1 -- 172.21.15.77:0/1721934245 >> v1:172.21.15.158:6789/0 conn(0x7f5db415fa10 legacy=0x7f5db40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.478+0000 7f5db3fff640 1 -- 172.21.15.77:0/1721934245 >> v1:172.21.15.77:6789/0 conn(0x7f5db415bdd0 legacy=0x7f5db40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.478+0000 7f5db3fff640 1 -- 172.21.15.77:0/1721934245 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db40cf1a0 con 0x7f5db40d16e0 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.478+0000 7f5db2ffd640 1 -- 172.21.15.77:0/1721934245 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025729148 0 0) 0x7f5db40cebe0 con 0x7f5db415bdd0 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.478+0000 7f5db3fff640 1 --2- 172.21.15.77:0/1721934245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db40d16e0 0x7f5db40d1af0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009d90 tx=0x7f5dac02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da498c83bc2d9a8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.478+0000 7f5db2ffd640 1 -- 172.21.15.77:0/1721934245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac039070 con 0x7f5db40d16e0 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.478+0000 7f5db2ffd640 1 -- 172.21.15.77:0/1721934245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac02cbd0 con 0x7f5db40d16e0 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.478+0000 7f5db2ffd640 1 -- 172.21.15.77:0/1721934245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac034440 con 0x7f5db40d16e0 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.479+0000 7f5db9e88640 1 -- 172.21.15.77:0/1721934245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db40d16e0 msgr2=0x7f5db40d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.479+0000 7f5db9e88640 1 --2- 172.21.15.77:0/1721934245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db40d16e0 0x7f5db40d1af0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009d90 tx=0x7f5dac02be20 comp rx=0 tx=0).stop 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.479+0000 7f5db9e88640 1 -- 172.21.15.77:0/1721934245 shutdown_connections 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.479+0000 7f5db9e88640 1 --2- 172.21.15.77:0/1721934245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db40d16e0 0x7f5db40d1af0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.479+0000 7f5db9e88640 1 -- 172.21.15.77:0/1721934245 >> 172.21.15.77:0/1721934245 conn(0x7f5db409a9f0 msgr2=0x7f5db409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.479+0000 7f5db9e88640 1 -- 172.21.15.77:0/1721934245 shutdown_connections 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.479+0000 7f5db9e88640 1 -- 172.21.15.77:0/1721934245 wait complete. 2024-03-06T07:29:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.480+0000 7f5db9e88640 1 Processor -- start 2024-03-06T07:29:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.480+0000 7f5db9e88640 1 -- start start 2024-03-06T07:29:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5db9e88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db40d16e0 0x7f5db415aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5db9e88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db415bdd0 0x7f5db406cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5db9e88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db415fa10 0x7f5db415cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5db9e88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db415c1b0 con 0x7f5db415bdd0 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5db9e88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db40cc5d0 con 0x7f5db40d16e0 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5db9e88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db40d2030 con 0x7f5db415fa10 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5daaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db415bdd0 0x7f5db406cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5daaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db415bdd0 0x7f5db406cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60784/0 (socket says 172.21.15.77:60784) 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5daaffd640 1 -- 172.21.15.77:0/1508927307 learned_addr learned my addr 172.21.15.77:0/1508927307 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5daaffd640 1 -- 172.21.15.77:0/1508927307 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db415fa10 msgr2=0x7f5db415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5daaffd640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db415fa10 0x7f5db415cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5daaffd640 1 -- 172.21.15.77:0/1508927307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db40d16e0 msgr2=0x7f5db415aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5daaffd640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db40d16e0 0x7f5db415aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.481+0000 7f5daaffd640 1 -- 172.21.15.77:0/1508927307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db406d9d0 con 0x7f5db415bdd0 2024-03-06T07:29:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.482+0000 7f5daaffd640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db415bdd0 0x7f5db406cee0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00b3f0 tx=0x7f5d9c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.482+0000 7f5db0ff9640 1 -- 172.21.15.77:0/1508927307 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c015020 con 0x7f5db415bdd0 2024-03-06T07:29:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.482+0000 7f5db0ff9640 1 -- 172.21.15.77:0/1508927307 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c011830 con 0x7f5db415bdd0 2024-03-06T07:29:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.482+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db406dcb0 con 0x7f5db415bdd0 2024-03-06T07:29:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.482+0000 7f5db0ff9640 1 -- 172.21.15.77:0/1508927307 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c011c20 con 0x7f5db415bdd0 2024-03-06T07:29:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.482+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db415af50 con 0x7f5db415bdd0 2024-03-06T07:29:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.483+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db404f330 con 0x7f5db415bdd0 2024-03-06T07:29:59.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.483+0000 7f5db0ff9640 1 -- 172.21.15.77:0/1508927307 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d9c018020 con 0x7f5db415bdd0 2024-03-06T07:29:59.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.483+0000 7f5db0ff9640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b4d0 0x7f5d9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.484+0000 7f5db37fe640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b4d0 0x7f5d9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.484+0000 7f5db0ff9640 1 -- 172.21.15.77:0/1508927307 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5d9c089aa0 con 0x7f5db415bdd0 2024-03-06T07:29:59.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.484+0000 7f5db37fe640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b4d0 0x7f5d9406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5da4005880 tx=0x7f5da4007950 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.488+0000 7f5db0ff9640 1 -- 172.21.15.77:0/1508927307 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d9c01f030 con 0x7f5db415bdd0 2024-03-06T07:29:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.723+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7f5db4160260 con 0x7f5db415bdd0 2024-03-06T07:29:59.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.770+0000 7f5db0ff9640 1 -- 172.21.15.77:0/1508927307 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v423) v1 ==== 144+0+0 (secure 0 0 0) 0x7f5d9c0172c0 con 0x7f5db415bdd0 2024-03-06T07:29:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b4d0 msgr2=0x7f5d9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b4d0 0x7f5d9406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5da4005880 tx=0x7f5da4007950 comp rx=0 tx=0).stop 2024-03-06T07:29:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db415bdd0 msgr2=0x7f5db406cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db415bdd0 0x7f5db406cee0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00b3f0 tx=0x7f5d9c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 shutdown_connections 2024-03-06T07:29:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5d9406b4d0 0x7f5d9406d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5db415bdd0 0x7f5db406cee0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5db415fa10 0x7f5db415cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 --2- 172.21.15.77:0/1508927307 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5db40d16e0 0x7f5db415aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 >> 172.21.15.77:0/1508927307 conn(0x7f5db409a9f0 msgr2=0x7f5db409add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 shutdown_connections 2024-03-06T07:29:59.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.772+0000 7f5db9e88640 1 -- 172.21.15.77:0/1508927307 wait complete. 2024-03-06T07:29:59.785 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:29:59.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.886+0000 7f5651bf7640 1 Processor -- start 2024-03-06T07:29:59.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.886+0000 7f5651bf7640 1 -- start start 2024-03-06T07:29:59.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f5651bf7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f564c066d90 0x7f564c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f5651bf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0b5410 0x7f564c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f5651bf7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564c06b710 con 0x7f564c0cf9a0 2024-03-06T07:29:59.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f5651bf7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c0b61c0 con 0x7f564c066d90 2024-03-06T07:29:59.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f5651bf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c06b150 con 0x7f564c0b5410 2024-03-06T07:29:59.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564affd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f564c066d90 0x7f564c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564bfff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0b5410 0x7f564c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564b7fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f564c0cf9a0 0x7f564c0b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:58584/0 (socket says 172.21.15.77:58584) 2024-03-06T07:29:59.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564affd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f564c066d90 0x7f564c0b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:57822/0 (socket says 172.21.15.77:57822) 2024-03-06T07:29:59.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564b7fe640 1 -- 172.21.15.77:0/3585381807 learned_addr learned my addr 172.21.15.77:0/3585381807 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:29:59.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564affd640 1 -- 172.21.15.77:0/3585381807 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0b5410 msgr2=0x7f564c0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564affd640 1 --2- 172.21.15.77:0/3585381807 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0b5410 0x7f564c0ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564affd640 1 -- 172.21.15.77:0/3585381807 >> v1:172.21.15.158:6789/0 conn(0x7f564c0cf9a0 legacy=0x7f564c0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564affd640 1 -- 172.21.15.77:0/3585381807 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c06b430 con 0x7f564c066d90 2024-03-06T07:29:59.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.887+0000 7f564bfff640 1 --2- 172.21.15.77:0/3585381807 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0b5410 0x7f564c0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:59.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.888+0000 7f564affd640 1 --2- 172.21.15.77:0/3585381807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 0x7f564c0b4ed0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f56400097a0 tx=0x7f564002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4765ed894c05146f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.888+0000 7f564a7fc640 1 -- 172.21.15.77:0/3585381807 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640035020 con 0x7f564c066d90 2024-03-06T07:29:59.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.888+0000 7f564a7fc640 1 -- 172.21.15.77:0/3585381807 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5640031830 con 0x7f564c066d90 2024-03-06T07:29:59.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.888+0000 7f564a7fc640 1 -- 172.21.15.77:0/3585381807 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640031c00 con 0x7f564c066d90 2024-03-06T07:29:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.889+0000 7f5651bf7640 1 -- 172.21.15.77:0/3585381807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 msgr2=0x7f564c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.889+0000 7f5651bf7640 1 --2- 172.21.15.77:0/3585381807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 0x7f564c0b4ed0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f56400097a0 tx=0x7f564002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.889+0000 7f5651bf7640 1 -- 172.21.15.77:0/3585381807 shutdown_connections 2024-03-06T07:29:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.889+0000 7f5651bf7640 1 --2- 172.21.15.77:0/3585381807 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0b5410 0x7f564c0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.889+0000 7f5651bf7640 1 --2- 172.21.15.77:0/3585381807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 0x7f564c0b4ed0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.889+0000 7f5651bf7640 1 -- 172.21.15.77:0/3585381807 >> 172.21.15.77:0/3585381807 conn(0x7f564c0b1d70 msgr2=0x7f564c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.889+0000 7f5651bf7640 1 -- 172.21.15.77:0/3585381807 shutdown_connections 2024-03-06T07:29:59.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.889+0000 7f5651bf7640 1 -- 172.21.15.77:0/3585381807 wait complete. 2024-03-06T07:29:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.890+0000 7f5651bf7640 1 Processor -- start 2024-03-06T07:29:59.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.890+0000 7f5651bf7640 1 -- start start 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f5651bf7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 0x7f564c0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f5651bf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564c0b5410 0x7f564c0c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f5651bf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0cf9a0 0x7f564c14e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f5651bf7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0cfd80 con 0x7f564c0b5410 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f5651bf7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0b0c50 con 0x7f564c066d90 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f5651bf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0cff00 con 0x7f564c0cf9a0 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564c0b5410 0x7f564c0c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564c0b5410 0x7f564c0c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60788/0 (socket says 172.21.15.77:60788) 2024-03-06T07:29:59.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564affd640 1 -- 172.21.15.77:0/4111125987 learned_addr learned my addr 172.21.15.77:0/4111125987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564bfff640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0cf9a0 0x7f564c14e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564b7fe640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 0x7f564c0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564affd640 1 -- 172.21.15.77:0/4111125987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0cf9a0 msgr2=0x7f564c14e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564affd640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0cf9a0 0x7f564c14e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564affd640 1 -- 172.21.15.77:0/4111125987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 msgr2=0x7f564c0caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564affd640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 0x7f564c0caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.891+0000 7f564affd640 1 -- 172.21.15.77:0/4111125987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c14eb90 con 0x7f564c0b5410 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.892+0000 7f564affd640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564c0b5410 0x7f564c0c5ae0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f56400098c0 tx=0x7f56400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.892+0000 7f5633fff640 1 -- 172.21.15.77:0/4111125987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640041070 con 0x7f564c0b5410 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.892+0000 7f5633fff640 1 -- 172.21.15.77:0/4111125987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564003b860 con 0x7f564c0b5410 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.892+0000 7f5633fff640 1 -- 172.21.15.77:0/4111125987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640035040 con 0x7f564c0b5410 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.892+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c0cb1c0 con 0x7f564c0b5410 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.892+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c0cb780 con 0x7f564c0b5410 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.893+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c09e3d0 con 0x7f564c0b5410 2024-03-06T07:29:59.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.893+0000 7f5633fff640 1 -- 172.21.15.77:0/4111125987 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f564004c060 con 0x7f564c0b5410 2024-03-06T07:29:59.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.894+0000 7f5633fff640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f563806b500 0x7f563806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.894+0000 7f5633fff640 1 -- 172.21.15.77:0/4111125987 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f56400ab650 con 0x7f564c0b5410 2024-03-06T07:29:59.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.894+0000 7f564b7fe640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f563806b500 0x7f563806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.894+0000 7f564b7fe640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f563806b500 0x7f563806d9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f564c15b530 tx=0x7f563c007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:29:59.898+0000 7f5633fff640 1 -- 172.21.15.77:0/4111125987 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f564c14eb90 con 0x7f564c0b5410 2024-03-06T07:30:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.069+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f564c068ff0 con 0x7f564c0b5410 2024-03-06T07:30:00.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.073+0000 7f5633fff640 1 -- 172.21.15.77:0/4111125987 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5640031800 con 0x7f564c0b5410 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f563806b500 msgr2=0x7f563806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f563806b500 0x7f563806d9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f564c15b530 tx=0x7f563c007e80 comp rx=0 tx=0).stop 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564c0b5410 msgr2=0x7f564c0c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564c0b5410 0x7f564c0c5ae0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f56400098c0 tx=0x7f56400046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 shutdown_connections 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f563806b500 0x7f563806d9c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564c0b5410 0x7f564c0c5ae0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f564c0cf9a0 0x7f564c14e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 --2- 172.21.15.77:0/4111125987 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f564c066d90 0x7f564c0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 >> 172.21.15.77:0/4111125987 conn(0x7f564c0b1d70 msgr2=0x7f564c0b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 shutdown_connections 2024-03-06T07:30:00.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.075+0000 7f5651bf7640 1 -- 172.21.15.77:0/4111125987 wait complete. 2024-03-06T07:30:00.088 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:00.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:00.139 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:00.190 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:00.191 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:30:00.191 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:00.206 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:00.206 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:30:00.206 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:00.224 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:00.224 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:30:00.224 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:00.260 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:00.261 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:30:00.261 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:00.281 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:00.281 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:30:00.282 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.399+0000 7f6e7f478640 1 Processor -- start 2024-03-06T07:30:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.399+0000 7f6e7f478640 1 -- start start 2024-03-06T07:30:00.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7f478640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e780af6c0 0x7f6e780afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7f478640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e7815bb10 0x7f6e7815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7f478640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e780b4e30 con 0x7f6e7815ab00 2024-03-06T07:30:00.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7f478640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e780b0060 con 0x7f6e7815bb10 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7f478640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780b5110 con 0x7f6e780af6c0 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7d214640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6e7815ab00 0x7f6e780b43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50348/0 (socket says 172.21.15.77:50348) 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7d214640 1 -- 172.21.15.77:0/3504084458 learned_addr learned my addr 172.21.15.77:0/3504084458 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7da15640 1 --2- 172.21.15.77:0/3504084458 >> v2:172.21.15.158:3300/0 conn(0x7f6e7815bb10 0x7f6e7815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817067872 0 0) 0x7f6e780b4e30 con 0x7f6e7815ab00 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e64003520 con 0x7f6e7815ab00 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7ca13640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e780af6c0 0x7f6e780afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2162473275 0 0) 0x7f6e64003520 con 0x7f6e7815ab00 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e780af6c0 msgr2=0x7f6e780afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e780af6c0 0x7f6e780afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 >> v2:172.21.15.158:3300/0 conn(0x7f6e7815bb10 msgr2=0x7f6e7815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 --2- 172.21.15.77:0/3504084458 >> v2:172.21.15.158:3300/0 conn(0x7f6e7815bb10 0x7f6e7815bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7815b310 con 0x7f6e7815ab00 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f6e74001670 con 0x7f6e7815ab00 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 >> v1:172.21.15.77:6789/0 conn(0x7f6e7815ab00 legacy=0x7f6e780b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e640039b0 0x7f6e64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e780af6c0 0x7f6e64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e64004d40 0x7f6e640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780b5110 con 0x7f6e640039b0 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6400dbd0 con 0x7f6e64004d40 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6400dd50 con 0x7f6e780af6c0 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.400+0000 7f6e7d214640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e640039b0 0x7f6e64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7da15640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e64004d40 0x7f6e640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7ca13640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e780af6c0 0x7f6e64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7d214640 1 -- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e780af6c0 msgr2=0x7f6e64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7d214640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e780af6c0 0x7f6e64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7d214640 1 -- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e64004d40 msgr2=0x7f6e640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7d214640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e64004d40 0x7f6e640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7d214640 1 -- 172.21.15.77:0/3504084458 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64003520 con 0x7f6e640039b0 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7da15640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e64004d40 0x7f6e640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.401+0000 7f6e7d214640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e640039b0 0x7f6e64003dd0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400c240 tx=0x7f6e64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6970bda931a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74007d30 con 0x7f6e640039b0 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e74030070 con 0x7f6e640039b0 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e6ffff640 1 -- 172.21.15.77:0/3504084458 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7402cd60 con 0x7f6e640039b0 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e7f478640 1 -- 172.21.15.77:0/3504084458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e640039b0 msgr2=0x7f6e64003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e7f478640 1 --2- 172.21.15.77:0/3504084458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e640039b0 0x7f6e64003dd0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400c240 tx=0x7f6e64005830 comp rx=0 tx=0).stop 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e7da15640 1 -- 172.21.15.77:0/3504084458 reap_dead start 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e7f478640 1 -- 172.21.15.77:0/3504084458 shutdown_connections 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e7f478640 1 -- 172.21.15.77:0/3504084458 >> 172.21.15.77:0/3504084458 conn(0x7f6e7806c0c0 msgr2=0x7f6e7806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e7f478640 1 -- 172.21.15.77:0/3504084458 shutdown_connections 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.402+0000 7f6e7f478640 1 -- 172.21.15.77:0/3504084458 wait complete. 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.403+0000 7f6e7f478640 1 Processor -- start 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.403+0000 7f6e7f478640 1 -- start start 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7f478640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e640039b0 0x7f6e780cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7f478640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e780cf4b0 0x7f6e7813e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7f478640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e780cfb10 0x7f6e780cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7f478640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e78071b20 con 0x7f6e780cfb10 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7f478640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780b4ac0 con 0x7f6e780cf4b0 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7f478640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e78071ca0 con 0x7f6e640039b0 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7da15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e780cfb10 0x7f6e780cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7da15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e780cfb10 0x7f6e780cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60816/0 (socket says 172.21.15.77:60816) 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7da15640 1 -- 172.21.15.77:0/1757574736 learned_addr learned my addr 172.21.15.77:0/1757574736 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7d214640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e640039b0 0x7f6e780cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7ca13640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e780cf4b0 0x7f6e7813e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7da15640 1 -- 172.21.15.77:0/1757574736 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e640039b0 msgr2=0x7f6e780cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7da15640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e640039b0 0x7f6e780cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7da15640 1 -- 172.21.15.77:0/1757574736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e780cf4b0 msgr2=0x7f6e7813e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7da15640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e780cf4b0 0x7f6e7813e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.404+0000 7f6e7da15640 1 -- 172.21.15.77:0/1757574736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780ce410 con 0x7f6e780cfb10 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.405+0000 7f6e7da15640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e780cfb10 0x7f6e780cff90 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f6e70007c20 tx=0x7f6e70010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.405+0000 7f6e6dffb640 1 -- 172.21.15.77:0/1757574736 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70010c20 con 0x7f6e780cfb10 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.405+0000 7f6e6dffb640 1 -- 172.21.15.77:0/1757574736 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e70010dc0 con 0x7f6e780cfb10 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.405+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e780ce6f0 con 0x7f6e780cfb10 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.405+0000 7f6e6dffb640 1 -- 172.21.15.77:0/1757574736 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70003bd0 con 0x7f6e780cfb10 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.405+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e780cecb0 con 0x7f6e780cfb10 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.406+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7814c730 con 0x7f6e780cfb10 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.406+0000 7f6e6dffb640 1 -- 172.21.15.77:0/1757574736 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e7001e020 con 0x7f6e780cfb10 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.407+0000 7f6e6dffb640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5c06b4d0 0x7f6e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.407+0000 7f6e6dffb640 1 -- 172.21.15.77:0/1757574736 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6e7008a4a0 con 0x7f6e780cfb10 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.407+0000 7f6e7d214640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5c06b4d0 0x7f6e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.407+0000 7f6e7d214640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5c06b4d0 0x7f6e5c06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6e74034000 tx=0x7f6e74007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.411+0000 7f6e6dffb640 1 -- 172.21.15.77:0/1757574736 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e70059590 con 0x7f6e780cfb10 2024-03-06T07:30:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.618+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e7804f330 con 0x7f6e780cfb10 2024-03-06T07:30:00.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.619+0000 7f6e6dffb640 1 -- 172.21.15.77:0/1757574736 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2120 (secure 0 0 0) 0x7f6e70058e90 con 0x7f6e780cfb10 2024-03-06T07:30:00.621 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:00.621 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":93,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/1028320016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":1028320016},{"type":"v1","addr":"172.21.15.158:6833","nonce":1028320016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93},{"gid":8973,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/4200914134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4200914134},{"type":"v1","addr":"172.21.15.77:6835","nonce":4200914134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[]} 2024-03-06T07:30:00.621 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 93 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5c06b4d0 msgr2=0x7f6e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5c06b4d0 0x7f6e5c06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6e74034000 tx=0x7f6e74007000 comp rx=0 tx=0).stop 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e780cfb10 msgr2=0x7f6e780cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e780cfb10 0x7f6e780cff90 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f6e70007c20 tx=0x7f6e70010590 comp rx=0 tx=0).stop 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 shutdown_connections 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e5c06b4d0 0x7f6e5c06d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e780cfb10 0x7f6e780cff90 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e780cf4b0 0x7f6e7813e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 --2- 172.21.15.77:0/1757574736 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e640039b0 0x7f6e780cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 >> 172.21.15.77:0/1757574736 conn(0x7f6e7806c0c0 msgr2=0x7f6e78071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 shutdown_connections 2024-03-06T07:30:00.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.621+0000 7f6e7f478640 1 -- 172.21.15.77:0/1757574736 wait complete. 2024-03-06T07:30:00.634 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:00.634 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:00.634 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:00.634 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:00.635 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:00.635 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:00.635 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:00.635 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:30:00.635 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:00.635 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:00.635 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:00.636 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:00.638 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:00.638 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:00.643 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:00.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:00.644 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:00.644 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:00.647 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:00.647 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:00.686 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:00.686 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:00.689 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:00.690 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:00.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:00.719 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2024-03-06T07:30:00.723 INFO:tasks.ceph.mds.d.smithi158.stdout:starting mds.d at 2024-03-06T07:30:00.764 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:30:00.771 INFO:tasks.ceph.mds.b.smithi158.stdout:starting mds.b at 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.794+0000 7f8c057b6640 1 Processor -- start 2024-03-06T07:30:00.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.794+0000 7f8c057b6640 1 -- start start 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.795+0000 7f8c057b6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8c001614a0 0x7f8c0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.795+0000 7f8c057b6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8c000b1a20 0x7f8c000b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.795+0000 7f8c057b6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c000af4b0 con 0x7f8c000b0270 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.795+0000 7f8c057b6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c000af790 con 0x7f8c000b1a20 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.795+0000 7f8c057b6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c000afa70 con 0x7f8c001614a0 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.795+0000 7f8bff7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8c000b1a20 0x7f8c000b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.795+0000 7f8bff7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8c000b1a20 0x7f8c000b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60872/0 (socket says 172.21.15.77:60872) 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.795+0000 7f8bff7fe640 1 -- 172.21.15.77:0/3706694629 learned_addr learned my addr 172.21.15.77:0/3706694629 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.796+0000 7f8bff7fe640 1 -- 172.21.15.77:0/3706694629 >> v2:172.21.15.158:3300/0 conn(0x7f8c001614a0 msgr2=0x7f8c0006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.796+0000 7f8bff7fe640 1 --2- 172.21.15.77:0/3706694629 >> v2:172.21.15.158:3300/0 conn(0x7f8c001614a0 0x7f8c0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.796+0000 7f8bff7fe640 1 -- 172.21.15.77:0/3706694629 >> v1:172.21.15.158:6789/0 conn(0x7f8c000b0270 legacy=0x7f8c000b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.796+0000 7f8bff7fe640 1 -- 172.21.15.77:0/3706694629 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c000afd50 con 0x7f8c000b1a20 2024-03-06T07:30:00.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.796+0000 7f8bff7fe640 1 --2- 172.21.15.77:0/3706694629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c000b1a20 0x7f8c000b5510 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f8c000730b0 tx=0x7f8bf802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5a05af6eef3262 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8bfdffb640 1 -- 172.21.15.77:0/3706694629 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf802c9f0 con 0x7f8c000b1a20 2024-03-06T07:30:00.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8bfdffb640 1 -- 172.21.15.77:0/3706694629 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf802cb90 con 0x7f8c000b1a20 2024-03-06T07:30:00.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8bfdffb640 1 -- 172.21.15.77:0/3706694629 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf80343f0 con 0x7f8c000b1a20 2024-03-06T07:30:00.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8c057b6640 1 -- 172.21.15.77:0/3706694629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c000b1a20 msgr2=0x7f8c000b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8c057b6640 1 --2- 172.21.15.77:0/3706694629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c000b1a20 0x7f8c000b5510 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f8c000730b0 tx=0x7f8bf802be50 comp rx=0 tx=0).stop 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8c057b6640 1 -- 172.21.15.77:0/3706694629 shutdown_connections 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8c057b6640 1 --2- 172.21.15.77:0/3706694629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c000b1a20 0x7f8c000b5510 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8c057b6640 1 --2- 172.21.15.77:0/3706694629 >> v2:172.21.15.158:3300/0 conn(0x7f8c001614a0 0x7f8c0006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8c057b6640 1 -- 172.21.15.77:0/3706694629 >> 172.21.15.77:0/3706694629 conn(0x7f8c0009a9f0 msgr2=0x7f8c0009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8c057b6640 1 -- 172.21.15.77:0/3706694629 shutdown_connections 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.797+0000 7f8c057b6640 1 -- 172.21.15.77:0/3706694629 wait complete. 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.798+0000 7f8c057b6640 1 Processor -- start 2024-03-06T07:30:00.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.798+0000 7f8c057b6640 1 -- start start 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8c057b6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c000b0270 0x7f8c00149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8c057b6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c000b1a20 0x7f8c001f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8c057b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c001614a0 0x7f8c0015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8c057b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c000b1e00 con 0x7f8c001614a0 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8c057b6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c000b06e0 con 0x7f8c000b0270 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8c057b6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c000b1f80 con 0x7f8c000b1a20 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8bff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c001614a0 0x7f8c0015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8bff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c001614a0 0x7f8c0015bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60884/0 (socket says 172.21.15.77:60884) 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8bff7fe640 1 -- 172.21.15.77:0/814543937 learned_addr learned my addr 172.21.15.77:0/814543937 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.799+0000 7f8bfeffd640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c000b0270 0x7f8c00149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8bff7fe640 1 -- 172.21.15.77:0/814543937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c000b1a20 msgr2=0x7f8c001f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8bff7fe640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c000b1a20 0x7f8c001f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8bff7fe640 1 -- 172.21.15.77:0/814543937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c000b0270 msgr2=0x7f8c00149a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8bff7fe640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c000b0270 0x7f8c00149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8bff7fe640 1 -- 172.21.15.77:0/814543937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c000cc2b0 con 0x7f8c001614a0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8bff7fe640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c001614a0 0x7f8c0015bbd0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8036040 tx=0x7f8bf8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8be77fe640 1 -- 172.21.15.77:0/814543937 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8043020 con 0x7f8c001614a0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8be77fe640 1 -- 172.21.15.77:0/814543937 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf803e450 con 0x7f8c001614a0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8be77fe640 1 -- 172.21.15.77:0/814543937 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf803e820 con 0x7f8c001614a0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c000cc590 con 0x7f8c001614a0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.800+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c000ccb50 con 0x7f8c001614a0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.801+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0004f330 con 0x7f8c001614a0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.801+0000 7f8be77fe640 1 -- 172.21.15.77:0/814543937 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bf8046020 con 0x7f8c001614a0 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.802+0000 7f8be77fe640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bec06b4d0 0x7f8bec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.802+0000 7f8be77fe640 1 -- 172.21.15.77:0/814543937 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8bf80ac8b0 con 0x7f8c001614a0 2024-03-06T07:30:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.802+0000 7f8bfeffd640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bec06b4d0 0x7f8bec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.802+0000 7f8bfeffd640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bec06b4d0 0x7f8bec06d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400a770 tx=0x7f8bf4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.806+0000 7f8be77fe640 1 -- 172.21.15.77:0/814543937 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bf804d030 con 0x7f8c001614a0 2024-03-06T07:30:00.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:00.988+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8c0006e8e0 con 0x7f8c001614a0 2024-03-06T07:30:01.638 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:01.641 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:02.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.078+0000 7f8be77fe640 1 -- 172.21.15.77:0/814543937 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v135) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8bf807ba40 con 0x7f8c001614a0 2024-03-06T07:30:02.080 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2024-03-06T07:30:02.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bec06b4d0 msgr2=0x7f8bec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bec06b4d0 0x7f8bec06d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400a770 tx=0x7f8bf4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c001614a0 msgr2=0x7f8c0015bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c001614a0 0x7f8c0015bbd0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8036040 tx=0x7f8bf8004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:02.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 shutdown_connections 2024-03-06T07:30:02.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8bec06b4d0 0x7f8bec06d990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c001614a0 0x7f8c0015bbd0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c000b1a20 0x7f8c001f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 --2- 172.21.15.77:0/814543937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c000b0270 0x7f8c00149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 >> 172.21.15.77:0/814543937 conn(0x7f8c0009a9f0 msgr2=0x7f8c0006ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 shutdown_connections 2024-03-06T07:30:02.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.081+0000 7f8c057b6640 1 -- 172.21.15.77:0/814543937 wait complete. 2024-03-06T07:30:02.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:02.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.200+0000 7faee05f7640 1 Processor -- start 2024-03-06T07:30:02.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.200+0000 7faee05f7640 1 -- start start 2024-03-06T07:30:02.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faee05f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faed81424f0 0x7faed80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faee05f7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faed806da40 0x7faed815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faee05f7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed80cc980 con 0x7faed80cd7a0 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faee05f7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed80cd220 con 0x7faed81424f0 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faee05f7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed80ccc60 con 0x7faed806da40 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faede393640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faed81424f0 0x7faed80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faeddb92640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faed80cd7a0 0x7faed806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50356/0 (socket says 172.21.15.77:50356) 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faeddb92640 1 -- 172.21.15.77:0/1977167413 learned_addr learned my addr 172.21.15.77:0/1977167413 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faede393640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faed81424f0 0x7faed80cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60896/0 (socket says 172.21.15.77:60896) 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faedeb94640 1 --2- 172.21.15.77:0/1977167413 >> v2:172.21.15.158:3300/0 conn(0x7faed806da40 0x7faed815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faede393640 1 -- 172.21.15.77:0/1977167413 >> v2:172.21.15.158:3300/0 conn(0x7faed806da40 msgr2=0x7faed815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faede393640 1 --2- 172.21.15.77:0/1977167413 >> v2:172.21.15.158:3300/0 conn(0x7faed806da40 0x7faed815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faede393640 1 -- 172.21.15.77:0/1977167413 >> v1:172.21.15.77:6789/0 conn(0x7faed80cd7a0 legacy=0x7faed806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faede393640 1 -- 172.21.15.77:0/1977167413 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed80ccf40 con 0x7faed81424f0 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faedd391640 1 -- 172.21.15.77:0/1977167413 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474505827 0 0) 0x7faed80cc980 con 0x7faed80cd7a0 2024-03-06T07:30:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.201+0000 7faedeb94640 1 --2- 172.21.15.77:0/1977167413 >> v2:172.21.15.158:3300/0 conn(0x7faed806da40 0x7faed815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.202+0000 7faede393640 1 --2- 172.21.15.77:0/1977167413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed80cc2f0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7faed8068700 tx=0x7faed002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875b8fcdfb1051ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.202+0000 7faedd391640 1 -- 172.21.15.77:0/1977167413 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0035020 con 0x7faed81424f0 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.202+0000 7faedd391640 1 -- 172.21.15.77:0/1977167413 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0031830 con 0x7faed81424f0 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.202+0000 7faedd391640 1 -- 172.21.15.77:0/1977167413 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0031c00 con 0x7faed81424f0 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.203+0000 7faee05f7640 1 -- 172.21.15.77:0/1977167413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 msgr2=0x7faed80cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.203+0000 7faee05f7640 1 --2- 172.21.15.77:0/1977167413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed80cc2f0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7faed8068700 tx=0x7faed002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.203+0000 7faee05f7640 1 -- 172.21.15.77:0/1977167413 shutdown_connections 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.203+0000 7faee05f7640 1 --2- 172.21.15.77:0/1977167413 >> v2:172.21.15.158:3300/0 conn(0x7faed806da40 0x7faed815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.203+0000 7faee05f7640 1 --2- 172.21.15.77:0/1977167413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed80cc2f0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.203+0000 7faee05f7640 1 -- 172.21.15.77:0/1977167413 >> 172.21.15.77:0/1977167413 conn(0x7faed80cf8a0 msgr2=0x7faed80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.203+0000 7faee05f7640 1 -- 172.21.15.77:0/1977167413 shutdown_connections 2024-03-06T07:30:02.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.203+0000 7faee05f7640 1 -- 172.21.15.77:0/1977167413 wait complete. 2024-03-06T07:30:02.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.204+0000 7faee05f7640 1 Processor -- start 2024-03-06T07:30:02.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.204+0000 7faee05f7640 1 -- start start 2024-03-06T07:30:02.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faee05f7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faed806da40 0x7faed8159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faee05f7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faed80cd7a0 0x7faed813fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faee05f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed815a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faee05f7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed8071ce0 con 0x7faed81424f0 2024-03-06T07:30:02.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faee05f7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed8071b60 con 0x7faed806da40 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faee05f7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed8071e60 con 0x7faed80cd7a0 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed815a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed815a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60900/0 (socket says 172.21.15.77:60900) 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 -- 172.21.15.77:0/3434688243 learned_addr learned my addr 172.21.15.77:0/3434688243 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 -- 172.21.15.77:0/3434688243 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faed80cd7a0 msgr2=0x7faed813fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faed80cd7a0 0x7faed813fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 -- 172.21.15.77:0/3434688243 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faed806da40 msgr2=0x7faed8159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faed806da40 0x7faed8159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 -- 172.21.15.77:0/3434688243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed8140510 con 0x7faed81424f0 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faedeb94640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed815a930 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7faec800cea0 tx=0x7faec800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faed6ffd640 1 -- 172.21.15.77:0/3434688243 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8003b90 con 0x7faed81424f0 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.205+0000 7faed6ffd640 1 -- 172.21.15.77:0/3434688243 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec8003d50 con 0x7faed81424f0 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.206+0000 7faed6ffd640 1 -- 172.21.15.77:0/3434688243 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8004260 con 0x7faed81424f0 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.206+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed81407f0 con 0x7faed81424f0 2024-03-06T07:30:02.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.206+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed8140db0 con 0x7faed81424f0 2024-03-06T07:30:02.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.207+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed804f330 con 0x7faed81424f0 2024-03-06T07:30:02.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.207+0000 7faed6ffd640 1 -- 172.21.15.77:0/3434688243 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faec8004400 con 0x7faed81424f0 2024-03-06T07:30:02.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.207+0000 7faed6ffd640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.207+0000 7faed6ffd640 1 -- 172.21.15.77:0/3434688243 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faec808a7f0 con 0x7faed81424f0 2024-03-06T07:30:02.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.207+0000 7faede393640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.208+0000 7faede393640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faec406b4d0 0x7faec406d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7faed0036000 tx=0x7faed0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.212+0000 7faed6ffd640 1 -- 172.21.15.77:0/3434688243 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faec8059a50 con 0x7faed81424f0 2024-03-06T07:30:02.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.447+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7faed80d0620 con 0x7faed81424f0 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.448+0000 7faed6ffd640 1 -- 172.21.15.77:0/3434688243 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7faec8059350 con 0x7faed81424f0 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlaSXhEBAA658fSC6JVCL5HwUEg+dO5w==", 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:30:02.450 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlZ/ysHRAATjBaHCw4ZgoEpWFEWMmWZg==", 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.451 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.452 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.452 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.452 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2024-03-06T07:30:02.452 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehltPd8GBAAWjCSkfgAhge6EZmdtjvQ+w==", 2024-03-06T07:30:02.452 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.452 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:30:02.453 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:30:02.453 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:30:02.453 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.453 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.453 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.453 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.454 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2024-03-06T07:30:02.454 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehl06aQJRAAAltsVfElJhjKmD6d5Y+kPQ==", 2024-03-06T07:30:02.454 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.454 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:30:02.454 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:30:02.455 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:30:02.455 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.455 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.455 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.455 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.455 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2024-03-06T07:30:02.456 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBpGehlMOkCDxAAGnezMF3zCX3MuR8Hwhxr6g==", 2024-03-06T07:30:02.456 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.456 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.456 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.456 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.456 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.457 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.457 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.457 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2024-03-06T07:30:02.457 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBsGehlRoaHDRAAIrIzLvOvkQfjribOHum0kQ==", 2024-03-06T07:30:02.457 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.458 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.458 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.458 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.458 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.458 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.458 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.459 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2024-03-06T07:30:02.459 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBvGehlhOWDCxAAjXPd/c5TjdVKfcYkZO4x0g==", 2024-03-06T07:30:02.459 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.459 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.459 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.460 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.460 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.460 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.460 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.460 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2024-03-06T07:30:02.460 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQByGehlK1hVChAAs+c1kGkDgYOQQ56p0Ne4wQ==", 2024-03-06T07:30:02.461 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.461 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.461 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.461 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.461 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.462 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.462 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.462 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2024-03-06T07:30:02.462 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB2GehlHHLuFxAAhkG4j8872DSYH/t0lRtvIA==", 2024-03-06T07:30:02.462 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.462 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.463 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.463 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.463 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.463 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.463 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.463 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2024-03-06T07:30:02.464 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB5GehlONHBFRAAyIKNl5rtdjopKdinmqOU7A==", 2024-03-06T07:30:02.464 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.464 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.464 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.464 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.464 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.465 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.465 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.465 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2024-03-06T07:30:02.465 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB8GehlHaY5FBAAhyxvB/QPgbTEygZGW4F2kA==", 2024-03-06T07:30:02.465 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.465 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.466 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.466 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.466 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.466 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.466 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.466 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2024-03-06T07:30:02.467 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQB/Gehlx6JOEhAAQ9JRJ1RZOILbWNGy+bMRCA==", 2024-03-06T07:30:02.467 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.467 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:02.467 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2024-03-06T07:30:02.467 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.467 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.467 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.468 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.468 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2024-03-06T07:30:02.468 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA==", 2024-03-06T07:30:02.468 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.468 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:30:02.469 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow r", 2024-03-06T07:30:02.469 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:02.469 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.469 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.469 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.469 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2024-03-06T07:30:02.470 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlgrs2MxAAnKcRtTyGHD/RmV5MPhsH1w==", 2024-03-06T07:30:02.470 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.470 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:30:02.470 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:30:02.470 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:30:02.470 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:30:02.471 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.471 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.471 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.471 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2024-03-06T07:30:02.471 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlxWepLxAA+X716qxOPaY0gbkRjUDpSQ==", 2024-03-06T07:30:02.471 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehloMYXORAAC0po3vo4IxfSXD86hRUSzg==", 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2024-03-06T07:30:02.472 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehlM9kqCBAAYDc9Cek04Uct6yRwkLJ6Rw==", 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBlGehlLlXJDBAAxaTlpy2jafU/mNG+AKNh4g==", 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:30:02.473 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQBdGehloSOUHRAABg4rpJcBltFz1galmDKdEQ==", 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-06T07:30:02.474 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:30:02.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faec406b4d0 msgr2=0x7faec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faec406b4d0 0x7faec406d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7faed0036000 tx=0x7faed0030000 comp rx=0 tx=0).stop 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 msgr2=0x7faed815a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed815a930 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7faec800cea0 tx=0x7faec800cf90 comp rx=0 tx=0).stop 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 shutdown_connections 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faed81424f0 0x7faed815a930 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faed80cd7a0 0x7faed813fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 --2- 172.21.15.77:0/3434688243 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faed806da40 0x7faed8159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 >> 172.21.15.77:0/3434688243 conn(0x7faed80cf8a0 msgr2=0x7faed815cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 shutdown_connections 2024-03-06T07:30:02.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.450+0000 7faee05f7640 1 -- 172.21.15.77:0/3434688243 wait complete. 2024-03-06T07:30:02.477 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.574+0000 7faf0ca39640 1 Processor -- start 2024-03-06T07:30:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.574+0000 7faf0ca39640 1 -- start start 2024-03-06T07:30:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf0ca39640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf0813dc10 0x7faf080b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf0ca39640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0815f580 con 0x7faf0815ef60 2024-03-06T07:30:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf0ca39640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0815b450 con 0x7faf0815bf30 2024-03-06T07:30:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf0ca39640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf080cf3f0 con 0x7faf0813dc10 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf0659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf0813dc10 0x7faf080b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf0659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf0813dc10 0x7faf080b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60906/0 (socket says 172.21.15.77:60906) 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf06d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faf0815ef60 0x7faf080708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50362/0 (socket says 172.21.15.77:50362) 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf0659c640 1 -- 172.21.15.77:0/2241592341 learned_addr learned my addr 172.21.15.77:0/2241592341 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.575+0000 7faf0659c640 1 -- 172.21.15.77:0/2241592341 >> v1:172.21.15.158:6789/0 conn(0x7faf0815bf30 legacy=0x7faf0815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.576+0000 7faf0659c640 1 -- 172.21.15.77:0/2241592341 >> v1:172.21.15.77:6789/0 conn(0x7faf0815ef60 legacy=0x7faf080708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.576+0000 7faf0559a640 1 -- 172.21.15.77:0/2241592341 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362050219 0 0) 0x7faf0815f580 con 0x7faf0815ef60 2024-03-06T07:30:02.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.576+0000 7faf0659c640 1 -- 172.21.15.77:0/2241592341 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0815b730 con 0x7faf0813dc10 2024-03-06T07:30:02.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.576+0000 7faf0659c640 1 --2- 172.21.15.77:0/2241592341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf080b2950 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7faef8009e50 tx=0x7faef802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4148e9fbaed09ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.576+0000 7faf0559a640 1 -- 172.21.15.77:0/2241592341 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef802ca80 con 0x7faf0813dc10 2024-03-06T07:30:02.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.576+0000 7faf0559a640 1 -- 172.21.15.77:0/2241592341 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef802cc20 con 0x7faf0813dc10 2024-03-06T07:30:02.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.576+0000 7faf0559a640 1 -- 172.21.15.77:0/2241592341 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef80343f0 con 0x7faf0813dc10 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.577+0000 7faf0ca39640 1 -- 172.21.15.77:0/2241592341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 msgr2=0x7faf080b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.577+0000 7faf0ca39640 1 --2- 172.21.15.77:0/2241592341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf080b2950 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7faef8009e50 tx=0x7faef802bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.577+0000 7faf0ca39640 1 -- 172.21.15.77:0/2241592341 shutdown_connections 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.577+0000 7faf0ca39640 1 --2- 172.21.15.77:0/2241592341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf080b2950 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.577+0000 7faf0ca39640 1 -- 172.21.15.77:0/2241592341 >> 172.21.15.77:0/2241592341 conn(0x7faf080cf8a0 msgr2=0x7faf080cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.577+0000 7faf0ca39640 1 -- 172.21.15.77:0/2241592341 shutdown_connections 2024-03-06T07:30:02.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.577+0000 7faf0ca39640 1 -- 172.21.15.77:0/2241592341 wait complete. 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.578+0000 7faf0ca39640 1 Processor -- start 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.578+0000 7faf0ca39640 1 -- start start 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0ca39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf081f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0ca39640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf0815bf30 0x7faf0806e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0ca39640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf0815ef60 0x7faf0813ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0ca39640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf080b2e90 con 0x7faf0813dc10 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0ca39640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0815b210 con 0x7faf0815ef60 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0ca39640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf080b3010 con 0x7faf0815bf30 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf081f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf081f17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60922/0 (socket says 172.21.15.77:60922) 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0659c640 1 -- 172.21.15.77:0/1684502976 learned_addr learned my addr 172.21.15.77:0/1684502976 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf05d9b640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf0815bf30 0x7faf0806e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf06d9d640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf0815ef60 0x7faf0813ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0659c640 1 -- 172.21.15.77:0/1684502976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf0815bf30 msgr2=0x7faf0806e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0659c640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf0815bf30 0x7faf0806e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0659c640 1 -- 172.21.15.77:0/1684502976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf0815ef60 msgr2=0x7faf0813ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0659c640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf0815ef60 0x7faf0813ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.579+0000 7faf0659c640 1 -- 172.21.15.77:0/1684502976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf081416d0 con 0x7faf0813dc10 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.580+0000 7faf0659c640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf081f17c0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7faef8009e50 tx=0x7faef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.580+0000 7faef6ffd640 1 -- 172.21.15.77:0/1684502976 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8043020 con 0x7faf0813dc10 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.580+0000 7faef6ffd640 1 -- 172.21.15.77:0/1684502976 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef803e450 con 0x7faf0813dc10 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.580+0000 7faef6ffd640 1 -- 172.21.15.77:0/1684502976 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef803e840 con 0x7faf0813dc10 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.580+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf08141970 con 0x7faf0813dc10 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.580+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf08142210 con 0x7faf0813dc10 2024-03-06T07:30:02.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.581+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0804f330 con 0x7faf0813dc10 2024-03-06T07:30:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.581+0000 7faef6ffd640 1 -- 172.21.15.77:0/1684502976 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faef8046020 con 0x7faf0813dc10 2024-03-06T07:30:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.581+0000 7faef6ffd640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faee806b4d0 0x7faee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.582+0000 7faef6ffd640 1 -- 172.21.15.77:0/1684502976 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faef80ad580 con 0x7faf0813dc10 2024-03-06T07:30:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.582+0000 7faf05d9b640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faee806b4d0 0x7faee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.582+0000 7faf05d9b640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faee806b4d0 0x7faee806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7faf0813fe90 tx=0x7faefc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.586+0000 7faef6ffd640 1 -- 172.21.15.77:0/1684502976 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faef804d030 con 0x7faf0813dc10 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.759+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf0806eb00 con 0x7faf0813dc10 2024-03-06T07:30:02.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.759+0000 7faef6ffd640 1 -- 172.21.15.77:0/1684502976 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+12889 (secure 0 0 0) 0x7faef807ba20 con 0x7faf0813dc10 2024-03-06T07:30:02.762 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:02.762 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":135,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:02.074660+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faee806b4d0 msgr2=0x7faee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faee806b4d0 0x7faee806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7faf0813fe90 tx=0x7faefc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 msgr2=0x7faf081f17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf081f17c0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7faef8009e50 tx=0x7faef8004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:02.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 shutdown_connections 2024-03-06T07:30:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf0815ef60 0x7faf0813ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf0815bf30 0x7faf0806e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faee806b4d0 0x7faee806d990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 --2- 172.21.15.77:0/1684502976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf0813dc10 0x7faf081f17c0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 >> 172.21.15.77:0/1684502976 conn(0x7faf080cf8a0 msgr2=0x7faf08070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 shutdown_connections 2024-03-06T07:30:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.761+0000 7faf0ca39640 1 -- 172.21.15.77:0/1684502976 wait complete. 2024-03-06T07:30:02.774 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:02.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.875+0000 7f3fb25e9640 1 Processor -- start 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.875+0000 7f3fb25e9640 1 -- start start 2024-03-06T07:30:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.876+0000 7f3fb25e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3fac0d4860 0x7f3fac0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.876+0000 7f3fb25e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3fac0b0e30 0x7f3fac0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.876+0000 7f3fb25e9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fac1618b0 con 0x7f3fac1614d0 2024-03-06T07:30:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.876+0000 7f3fb25e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac0d4cf0 con 0x7f3fac0d4860 2024-03-06T07:30:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.876+0000 7f3fb25e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac0cf400 con 0x7f3fac0b0e30 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.876+0000 7f3fab7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3fac0d4860 0x7f3fac0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.876+0000 7f3fabfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3fac1614d0 0x7f3fac0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50364/0 (socket says 172.21.15.77:50364) 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.876+0000 7f3fabfff640 1 -- 172.21.15.77:0/3689852108 learned_addr learned my addr 172.21.15.77:0/3689852108 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.877+0000 7f3fb0b86640 1 --2- 172.21.15.77:0/3689852108 >> v2:172.21.15.158:3300/0 conn(0x7f3fac0b0e30 0x7f3fac0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.877+0000 7f3fab7fe640 1 -- 172.21.15.77:0/3689852108 >> v2:172.21.15.158:3300/0 conn(0x7f3fac0b0e30 msgr2=0x7f3fac0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.877+0000 7f3fab7fe640 1 --2- 172.21.15.77:0/3689852108 >> v2:172.21.15.158:3300/0 conn(0x7f3fac0b0e30 0x7f3fac0b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.877+0000 7f3fab7fe640 1 -- 172.21.15.77:0/3689852108 >> v1:172.21.15.77:6789/0 conn(0x7f3fac1614d0 legacy=0x7f3fac0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.877+0000 7f3fab7fe640 1 -- 172.21.15.77:0/3689852108 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac0cf6a0 con 0x7f3fac0d4860 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.877+0000 7f3faaffd640 1 -- 172.21.15.77:0/3689852108 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308530214 0 0) 0x7f3fac1618b0 con 0x7f3fac1614d0 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.877+0000 7f3fab7fe640 1 --2- 172.21.15.77:0/3689852108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0d4860 0x7f3fac0d2020 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0b2a70 tx=0x7f3fa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1e10ef835ee601 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3faaffd640 1 -- 172.21.15.77:0/3689852108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0035020 con 0x7f3fac0d4860 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3faaffd640 1 -- 172.21.15.77:0/3689852108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa0031830 con 0x7f3fac0d4860 2024-03-06T07:30:02.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3faaffd640 1 -- 172.21.15.77:0/3689852108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0031c00 con 0x7f3fac0d4860 2024-03-06T07:30:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3fb25e9640 1 -- 172.21.15.77:0/3689852108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0d4860 msgr2=0x7f3fac0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/3689852108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0d4860 0x7f3fac0d2020 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0b2a70 tx=0x7f3fa002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3fb25e9640 1 -- 172.21.15.77:0/3689852108 shutdown_connections 2024-03-06T07:30:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/3689852108 >> v2:172.21.15.158:3300/0 conn(0x7f3fac0b0e30 0x7f3fac0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/3689852108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0d4860 0x7f3fac0d2020 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3fb25e9640 1 -- 172.21.15.77:0/3689852108 >> 172.21.15.77:0/3689852108 conn(0x7f3fac0d3d40 msgr2=0x7f3fac0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3fb25e9640 1 -- 172.21.15.77:0/3689852108 shutdown_connections 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.878+0000 7f3fb25e9640 1 -- 172.21.15.77:0/3689852108 wait complete. 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.879+0000 7f3fb25e9640 1 Processor -- start 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.879+0000 7f3fb25e9640 1 -- start start 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fb25e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0b0e30 0x7f3fac0b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fb25e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3fac0d4860 0x7f3fac15dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fb25e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fac1614d0 0x7f3fac15b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fb25e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0b0910 con 0x7f3fac0b0e30 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fb25e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0cfc60 con 0x7f3fac1614d0 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fb25e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0d2560 con 0x7f3fac0d4860 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fabfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0b0e30 0x7f3fac0b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fabfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0b0e30 0x7f3fac0b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60926/0 (socket says 172.21.15.77:60926) 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fabfff640 1 -- 172.21.15.77:0/2811611014 learned_addr learned my addr 172.21.15.77:0/2811611014 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fabfff640 1 -- 172.21.15.77:0/2811611014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3fac0d4860 msgr2=0x7f3fac15dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fabfff640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3fac0d4860 0x7f3fac15dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fabfff640 1 -- 172.21.15.77:0/2811611014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fac1614d0 msgr2=0x7f3fac15b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fabfff640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fac1614d0 0x7f3fac15b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.880+0000 7f3fabfff640 1 -- 172.21.15.77:0/2811611014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac0b5dd0 con 0x7f3fac0b0e30 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.881+0000 7f3fabfff640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0b0e30 0x7f3fac0b54d0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c00c400 tx=0x7f3f9c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.881+0000 7f3fa8ff9640 1 -- 172.21.15.77:0/2811611014 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c017070 con 0x7f3fac0b0e30 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.881+0000 7f3fa8ff9640 1 -- 172.21.15.77:0/2811611014 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9c005ce0 con 0x7f3fac0b0e30 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.881+0000 7f3fa8ff9640 1 -- 172.21.15.77:0/2811611014 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c007310 con 0x7f3fac0b0e30 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.881+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac0b60b0 con 0x7f3fac0b0e30 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.881+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac0cafb0 con 0x7f3fac0b0e30 2024-03-06T07:30:02.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.882+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac068ff0 con 0x7f3fac0b0e30 2024-03-06T07:30:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.882+0000 7f3fa8ff9640 1 -- 172.21.15.77:0/2811611014 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f9c01a070 con 0x7f3fac0b0e30 2024-03-06T07:30:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.883+0000 7f3fa8ff9640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f9806b4d0 0x7f3f9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.883+0000 7f3fa8ff9640 1 -- 172.21.15.77:0/2811611014 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3f9c089260 con 0x7f3fac0b0e30 2024-03-06T07:30:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.883+0000 7f3fab7fe640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f9806b4d0 0x7f3f9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.883+0000 7f3fab7fe640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f9806b4d0 0x7f3f9806d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0036000 tx=0x7f3fa0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:02.887+0000 7f3fa8ff9640 1 -- 172.21.15.77:0/2811611014 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f9c0584c0 con 0x7f3fac0b0e30 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.057+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3fac15f720 con 0x7f3fac0b0e30 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.058+0000 7f3fa8ff9640 1 -- 172.21.15.77:0/2811611014 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f9c057dc0 con 0x7f3fac0b0e30 2024-03-06T07:30:03.060 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f9806b4d0 msgr2=0x7f3f9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f9806b4d0 0x7f3f9806d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0036000 tx=0x7f3fa0030000 comp rx=0 tx=0).stop 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0b0e30 msgr2=0x7f3fac0b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0b0e30 0x7f3fac0b54d0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c00c400 tx=0x7f3f9c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 shutdown_connections 2024-03-06T07:30:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3fac1614d0 0x7f3fac15b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3fac0d4860 0x7f3fac15dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3f9806b4d0 0x7f3f9806d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 --2- 172.21.15.77:0/2811611014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fac0b0e30 0x7f3fac0b54d0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 >> 172.21.15.77:0/2811611014 conn(0x7f3fac0d3d40 msgr2=0x7f3fac06c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 shutdown_connections 2024-03-06T07:30:03.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.060+0000 7f3fb25e9640 1 -- 172.21.15.77:0/2811611014 wait complete. 2024-03-06T07:30:03.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:03.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.171+0000 7fa23ccb5640 1 Processor -- start 2024-03-06T07:30:03.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.171+0000 7fa23ccb5640 1 -- start start 2024-03-06T07:30:03.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa23ccb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23813dc10 0x7fa2380b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa23ccb5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa23815ef60 0x7fa2380b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa23ccb5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23815f580 con 0x7fa23815bf30 2024-03-06T07:30:03.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa23ccb5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2380cf3f0 con 0x7fa23815ef60 2024-03-06T07:30:03.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa23ccb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23815b210 con 0x7fa23813dc10 2024-03-06T07:30:03.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa236d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa23815ef60 0x7fa2380b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa236d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa23815ef60 0x7fa2380b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60942/0 (socket says 172.21.15.77:60942) 2024-03-06T07:30:03.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa236d9d640 1 -- 172.21.15.77:0/1675598858 learned_addr learned my addr 172.21.15.77:0/1675598858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:03.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa236d9d640 1 -- 172.21.15.77:0/1675598858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23813dc10 msgr2=0x7fa2380b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa236d9d640 1 --2- 172.21.15.77:0/1675598858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23813dc10 0x7fa2380b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa236d9d640 1 -- 172.21.15.77:0/1675598858 >> v1:172.21.15.158:6789/0 conn(0x7fa23815bf30 legacy=0x7fa23815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.172+0000 7fa236d9d640 1 -- 172.21.15.77:0/1675598858 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23815b450 con 0x7fa23815ef60 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.173+0000 7fa236d9d640 1 --2- 172.21.15.77:0/1675598858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa2380b2e90 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fa230009e50 tx=0x7fa23002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9faddc3f9ff8d9d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.173+0000 7fa235d9b640 1 -- 172.21.15.77:0/1675598858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23002c9e0 con 0x7fa23815ef60 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.173+0000 7fa235d9b640 1 -- 172.21.15.77:0/1675598858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23002cb80 con 0x7fa23815ef60 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.173+0000 7fa235d9b640 1 -- 172.21.15.77:0/1675598858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2300343f0 con 0x7fa23815ef60 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.173+0000 7fa23ccb5640 1 -- 172.21.15.77:0/1675598858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 msgr2=0x7fa2380b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.173+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/1675598858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa2380b2e90 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fa230009e50 tx=0x7fa23002be40 comp rx=0 tx=0).stop 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.174+0000 7fa23ccb5640 1 -- 172.21.15.77:0/1675598858 shutdown_connections 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.174+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/1675598858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa2380b2e90 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.174+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/1675598858 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23813dc10 0x7fa2380b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.174+0000 7fa23ccb5640 1 -- 172.21.15.77:0/1675598858 >> 172.21.15.77:0/1675598858 conn(0x7fa2380cf8a0 msgr2=0x7fa2380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.174+0000 7fa23ccb5640 1 -- 172.21.15.77:0/1675598858 shutdown_connections 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.174+0000 7fa23ccb5640 1 -- 172.21.15.77:0/1675598858 wait complete. 2024-03-06T07:30:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.175+0000 7fa23ccb5640 1 Processor -- start 2024-03-06T07:30:03.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.175+0000 7fa23ccb5640 1 -- start start 2024-03-06T07:30:03.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa23ccb5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa23813dc10 0x7fa2381514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa23ccb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23815bf30 0x7fa2381353b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa23ccb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa23806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa23ccb5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2380cd7a0 con 0x7fa23815ef60 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa23ccb5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23813dff0 con 0x7fa23813dc10 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa23ccb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2380cd920 con 0x7fa23815bf30 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa23806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa23806d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60946/0 (socket says 172.21.15.77:60946) 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 -- 172.21.15.77:0/3990615099 learned_addr learned my addr 172.21.15.77:0/3990615099 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa22ffff640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23815bf30 0x7fa2381353b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa23659c640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa23813dc10 0x7fa2381514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 -- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23815bf30 msgr2=0x7fa2381353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23815bf30 0x7fa2381353b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 -- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa23813dc10 msgr2=0x7fa2381514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa23813dc10 0x7fa2381514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 -- 172.21.15.77:0/3990615099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23806c800 con 0x7fa23815ef60 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa22ffff640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23815bf30 0x7fa2381353b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa236d9d640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa23806d8a0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fa230036040 tx=0x7fa230004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa22effd640 1 -- 172.21.15.77:0/3990615099 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa230043020 con 0x7fa23815ef60 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa22effd640 1 -- 172.21.15.77:0/3990615099 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23003e450 con 0x7fa23815ef60 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23806cae0 con 0x7fa23815ef60 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.176+0000 7fa22effd640 1 -- 172.21.15.77:0/3990615099 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23003e860 con 0x7fa23815ef60 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.177+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23806d0a0 con 0x7fa23815ef60 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.177+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23804f330 con 0x7fa23815ef60 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.178+0000 7fa22effd640 1 -- 172.21.15.77:0/3990615099 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa230046020 con 0x7fa23815ef60 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.178+0000 7fa22effd640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa21806b4d0 0x7fa21806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.178+0000 7fa23659c640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa21806b4d0 0x7fa21806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.178+0000 7fa22effd640 1 -- 172.21.15.77:0/3990615099 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa2300ac880 con 0x7fa23815ef60 2024-03-06T07:30:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.179+0000 7fa23659c640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa21806b4d0 0x7fa21806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa228009890 tx=0x7fa228008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.183+0000 7fa22effd640 1 -- 172.21.15.77:0/3990615099 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa23004d030 con 0x7fa23815ef60 2024-03-06T07:30:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.355+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa23806eb00 con 0x7fa23815ef60 2024-03-06T07:30:03.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.356+0000 7fa22effd640 1 -- 172.21.15.77:0/3990615099 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa23007b9c0 con 0x7fa23815ef60 2024-03-06T07:30:03.358 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:03.358 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-06T07:30:03.358 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-06T07:30:03.358 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-06T07:30:03.358 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-06T07:30:03.358 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:03.358 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa21806b4d0 msgr2=0x7fa21806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa21806b4d0 0x7fa21806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa228009890 tx=0x7fa228008040 comp rx=0 tx=0).stop 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 msgr2=0x7fa23806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa23806d8a0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fa230036040 tx=0x7fa230004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 shutdown_connections 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa21806b4d0 0x7fa21806d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23815ef60 0x7fa23806d8a0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa23815bf30 0x7fa2381353b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 --2- 172.21.15.77:0/3990615099 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa23813dc10 0x7fa2381514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 >> 172.21.15.77:0/3990615099 conn(0x7fa2380cf8a0 msgr2=0x7fa23813ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 shutdown_connections 2024-03-06T07:30:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.358+0000 7fa23ccb5640 1 -- 172.21.15.77:0/3990615099 wait complete. 2024-03-06T07:30:03.370 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:03.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:03.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.472+0000 7f69f77a1640 1 Processor -- start 2024-03-06T07:30:03.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.472+0000 7f69f77a1640 1 -- start start 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.473+0000 7f69f77a1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f69f00d2720 0x7f69f00cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.473+0000 7f69f77a1640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f00cf790 con 0x7f69f006be00 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.473+0000 7f69f77a1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f00cfa70 con 0x7f69f0161a90 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.473+0000 7f69f77a1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f00d00c0 con 0x7f69f00d2720 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.473+0000 7f69f5d3e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f69f006be00 0x7f69f00d0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50370/0 (socket says 172.21.15.77:50370) 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.473+0000 7f69f5d3e640 1 -- 172.21.15.77:0/1049787960 learned_addr learned my addr 172.21.15.77:0/1049787960 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899858775 0 0) 0x7f69f00cf790 con 0x7f69f006be00 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f4d3c640 1 --2- 172.21.15.77:0/1049787960 >> v2:172.21.15.158:3300/0 conn(0x7f69f00d2720 0x7f69f00cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc003520 con 0x7f69f006be00 2024-03-06T07:30:03.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 232928647 0 0) 0x7f69dc003520 con 0x7f69f006be00 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 >> v2:172.21.15.158:3300/0 conn(0x7f69f00d2720 msgr2=0x7f69f00cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 --2- 172.21.15.77:0/1049787960 >> v2:172.21.15.158:3300/0 conn(0x7f69f00d2720 0x7f69f00cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 >> v1:172.21.15.158:6789/0 conn(0x7f69f0161a90 legacy=0x7f69f0161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f00cfd50 con 0x7f69f006be00 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f69ec0019f0 con 0x7f69f006be00 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 >> v1:172.21.15.77:6789/0 conn(0x7f69f006be00 legacy=0x7f69f00d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69dc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc004800 0x7f69dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0051c0 0x7f69dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00d00c0 con 0x7f69dc0039e0 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc00c090 con 0x7f69dc004800 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc00c290 con 0x7f69dc0051c0 2024-03-06T07:30:03.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f553d640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69dc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f4d3c640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc004800 0x7f69dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f5d3e640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0051c0 0x7f69dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f553d640 1 -- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0051c0 msgr2=0x7f69dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f553d640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0051c0 0x7f69dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f553d640 1 -- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc004800 msgr2=0x7f69dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f553d640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69dc004800 0x7f69dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f553d640 1 -- 172.21.15.77:0/1049787960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc003520 con 0x7f69dc0039e0 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f5d3e640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69dc0051c0 0x7f69dc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.474+0000 7f69f5d3e640 1 -- 172.21.15.77:0/1049787960 reap_dead start 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.475+0000 7f69f553d640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69dc003e00 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f69e8001750 tx=0x7f69dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15bef74401bb9d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.475+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8001370 con 0x7f69dc0039e0 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.475+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e80052d0 con 0x7f69dc0039e0 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.475+0000 7f69e7fff640 1 -- 172.21.15.77:0/1049787960 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8003830 con 0x7f69dc0039e0 2024-03-06T07:30:03.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.476+0000 7f69f77a1640 1 -- 172.21.15.77:0/1049787960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 msgr2=0x7f69dc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.476+0000 7f69f77a1640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69dc003e00 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f69e8001750 tx=0x7f69dc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.476+0000 7f69f77a1640 1 -- 172.21.15.77:0/1049787960 shutdown_connections 2024-03-06T07:30:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.476+0000 7f69f77a1640 1 --2- 172.21.15.77:0/1049787960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69dc003e00 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.476+0000 7f69f77a1640 1 -- 172.21.15.77:0/1049787960 >> 172.21.15.77:0/1049787960 conn(0x7f69f00d1b20 msgr2=0x7f69f00d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.476+0000 7f69f77a1640 1 -- 172.21.15.77:0/1049787960 shutdown_connections 2024-03-06T07:30:03.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.476+0000 7f69f77a1640 1 -- 172.21.15.77:0/1049787960 wait complete. 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.477+0000 7f69f77a1640 1 Processor -- start 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.477+0000 7f69f77a1640 1 -- start start 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f77a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69f009c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f77a1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f00b7490 0x7f69f0157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f77a1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f01f7a80 0x7f69f01f7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f77a1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f006c290 con 0x7f69dc0039e0 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f77a1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00cdd10 con 0x7f69f01f7a80 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f77a1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f006c410 con 0x7f69f00b7490 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69f009c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69f009c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60954/0 (socket says 172.21.15.77:60954) 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 -- 172.21.15.77:0/3093682359 learned_addr learned my addr 172.21.15.77:0/3093682359 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f4d3c640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f00b7490 0x7f69f0157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f5d3e640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f01f7a80 0x7f69f01f7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 -- 172.21.15.77:0/3093682359 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f00b7490 msgr2=0x7f69f0157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f00b7490 0x7f69f0157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 -- 172.21.15.77:0/3093682359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f01f7a80 msgr2=0x7f69f01f7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f01f7a80 0x7f69f01f7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 -- 172.21.15.77:0/3093682359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f01f88e0 con 0x7f69dc0039e0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f553d640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69f009c310 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f69e8000c00 tx=0x7f69e8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69e5ffb640 1 -- 172.21.15.77:0/3093682359 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8041070 con 0x7f69dc0039e0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69e5ffb640 1 -- 172.21.15.77:0/3093682359 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8005480 con 0x7f69dc0039e0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f00b2ee0 con 0x7f69dc0039e0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.478+0000 7f69e5ffb640 1 -- 172.21.15.77:0/3093682359 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e803d640 con 0x7f69dc0039e0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.479+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f01583a0 con 0x7f69dc0039e0 2024-03-06T07:30:03.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.480+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f004f330 con 0x7f69dc0039e0 2024-03-06T07:30:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.480+0000 7f69e5ffb640 1 -- 172.21.15.77:0/3093682359 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69e8044070 con 0x7f69dc0039e0 2024-03-06T07:30:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.480+0000 7f69e5ffb640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d406b500 0x7f69d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.480+0000 7f69e5ffb640 1 -- 172.21.15.77:0/3093682359 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f69e80ab9f0 con 0x7f69dc0039e0 2024-03-06T07:30:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.480+0000 7f69f4d3c640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d406b500 0x7f69d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.481+0000 7f69f4d3c640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d406b500 0x7f69d406d9c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f69f00b7440 tx=0x7f69e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.485+0000 7f69e5ffb640 1 -- 172.21.15.77:0/3093682359 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69f01f88e0 con 0x7f69dc0039e0 2024-03-06T07:30:03.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:03.655+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f69f00743f0 con 0x7f69dc0039e0 2024-03-06T07:30:04.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.084+0000 7f69e5ffb640 1 -- 172.21.15.77:0/3093682359 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v136) v1 ==== 138+0+0 (secure 0 0 0) 0x7f69e807ab40 con 0x7f69dc0039e0 2024-03-06T07:30:04.086 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d406b500 msgr2=0x7f69d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d406b500 0x7f69d406d9c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f69f00b7440 tx=0x7f69e0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 msgr2=0x7f69f009c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69f009c310 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f69e8000c00 tx=0x7f69e8001120 comp rx=0 tx=0).stop 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 shutdown_connections 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69f01f7a80 0x7f69f01f7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69f00b7490 0x7f69f0157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69d406b500 0x7f69d406d9c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 --2- 172.21.15.77:0/3093682359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0039e0 0x7f69f009c310 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 >> 172.21.15.77:0/3093682359 conn(0x7f69f00d1b20 msgr2=0x7f69f0070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 shutdown_connections 2024-03-06T07:30:04.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.087+0000 7f69f77a1640 1 -- 172.21.15.77:0/3093682359 wait complete. 2024-03-06T07:30:04.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:04.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.210+0000 7f5c1fc66640 1 Processor -- start 2024-03-06T07:30:04.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.210+0000 7f5c1fc66640 1 -- start start 2024-03-06T07:30:04.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1fc66640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5c1815fc30 0x7f5c18163160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1fc66640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c180cfa90 0x7f5c180cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1fc66640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c180af170 con 0x7f5c180ae6b0 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1fc66640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c181629a0 con 0x7f5c180cfa90 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1fc66640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c180aebb0 con 0x7f5c1815fc30 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1e203640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c180cfa90 0x7f5c180cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1e203640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c180cfa90 0x7f5c180cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60964/0 (socket says 172.21.15.77:60964) 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1e203640 1 -- 172.21.15.77:0/2104879382 learned_addr learned my addr 172.21.15.77:0/2104879382 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1da02640 1 --2- 172.21.15.77:0/2104879382 >> v2:172.21.15.158:3300/0 conn(0x7f5c1815fc30 0x7f5c18163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1e203640 1 -- 172.21.15.77:0/2104879382 >> v2:172.21.15.158:3300/0 conn(0x7f5c1815fc30 msgr2=0x7f5c18163160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1e203640 1 --2- 172.21.15.77:0/2104879382 >> v2:172.21.15.158:3300/0 conn(0x7f5c1815fc30 0x7f5c18163160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1e203640 1 -- 172.21.15.77:0/2104879382 >> v1:172.21.15.158:6789/0 conn(0x7f5c180ae6b0 legacy=0x7f5c180b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.211+0000 7f5c1e203640 1 -- 172.21.15.77:0/2104879382 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c180aee90 con 0x7f5c180cfa90 2024-03-06T07:30:04.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.212+0000 7f5c1e203640 1 --2- 172.21.15.77:0/2104879382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c180cff20 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5c180d05e0 tx=0x7f5c1002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430b57882f1330ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.212+0000 7f5c1ca00640 1 -- 172.21.15.77:0/2104879382 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1002ca80 con 0x7f5c180cfa90 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.212+0000 7f5c1ca00640 1 -- 172.21.15.77:0/2104879382 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1002cc20 con 0x7f5c180cfa90 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.212+0000 7f5c1ca00640 1 -- 172.21.15.77:0/2104879382 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c100343f0 con 0x7f5c180cfa90 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.213+0000 7f5c1fc66640 1 -- 172.21.15.77:0/2104879382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 msgr2=0x7f5c180cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.213+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/2104879382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c180cff20 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f5c180d05e0 tx=0x7f5c1002bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.213+0000 7f5c1fc66640 1 -- 172.21.15.77:0/2104879382 shutdown_connections 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.213+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/2104879382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c180cff20 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.213+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/2104879382 >> v2:172.21.15.158:3300/0 conn(0x7f5c1815fc30 0x7f5c18163160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.213+0000 7f5c1fc66640 1 -- 172.21.15.77:0/2104879382 >> 172.21.15.77:0/2104879382 conn(0x7f5c1806b190 msgr2=0x7f5c1806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.213+0000 7f5c1fc66640 1 -- 172.21.15.77:0/2104879382 shutdown_connections 2024-03-06T07:30:04.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.213+0000 7f5c1fc66640 1 -- 172.21.15.77:0/2104879382 wait complete. 2024-03-06T07:30:04.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.214+0000 7f5c1fc66640 1 Processor -- start 2024-03-06T07:30:04.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.214+0000 7f5c1fc66640 1 -- start start 2024-03-06T07:30:04.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1fc66640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c180ae6b0 0x7f5c18153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1fc66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c1814f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1fc66640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5c1815fc30 0x7f5c1815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1fc66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c18160010 con 0x7f5c180cfa90 2024-03-06T07:30:04.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1fc66640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c181636a0 con 0x7f5c180ae6b0 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1fc66640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c18160190 con 0x7f5c1815fc30 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1d201640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c1814f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1d201640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c1814f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60976/0 (socket says 172.21.15.77:60976) 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1d201640 1 -- 172.21.15.77:0/228882635 learned_addr learned my addr 172.21.15.77:0/228882635 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1e203640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5c1815fc30 0x7f5c1815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1da02640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c180ae6b0 0x7f5c18153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1d201640 1 -- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5c1815fc30 msgr2=0x7f5c1815c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1d201640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5c1815fc30 0x7f5c1815c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1d201640 1 -- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c180ae6b0 msgr2=0x7f5c18153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.215+0000 7f5c1d201640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c180ae6b0 0x7f5c18153c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.216+0000 7f5c1d201640 1 -- 172.21.15.77:0/228882635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c181545a0 con 0x7f5c180cfa90 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.216+0000 7f5c1e203640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5c1815fc30 0x7f5c1815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.216+0000 7f5c1d201640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c1814f6c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800c430 tx=0x7f5c0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.216+0000 7f5c0e7fc640 1 -- 172.21.15.77:0/228882635 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08011790 con 0x7f5c180cfa90 2024-03-06T07:30:04.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.216+0000 7f5c0e7fc640 1 -- 172.21.15.77:0/228882635 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c08017070 con 0x7f5c180cfa90 2024-03-06T07:30:04.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.216+0000 7f5c0e7fc640 1 -- 172.21.15.77:0/228882635 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08011b70 con 0x7f5c180cfa90 2024-03-06T07:30:04.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.216+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1815c8a0 con 0x7f5c180cfa90 2024-03-06T07:30:04.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.216+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c180cc3c0 con 0x7f5c180cfa90 2024-03-06T07:30:04.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.217+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c180732b0 con 0x7f5c180cfa90 2024-03-06T07:30:04.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.217+0000 7f5c0e7fc640 1 -- 172.21.15.77:0/228882635 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c08011d10 con 0x7f5c180cfa90 2024-03-06T07:30:04.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.218+0000 7f5c0e7fc640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.218+0000 7f5c0e7fc640 1 -- 172.21.15.77:0/228882635 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5c08089e50 con 0x7f5c180cfa90 2024-03-06T07:30:04.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.218+0000 7f5c1da02640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.218+0000 7f5c1da02640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5c180cf290 tx=0x7f5c14008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.222+0000 7f5c0e7fc640 1 -- 172.21.15.77:0/228882635 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c08058f40 con 0x7f5c180cfa90 2024-03-06T07:30:04.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:04.396+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5c18071ac0 con 0x7f5c180cfa90 2024-03-06T07:30:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.087+0000 7f5c0e7fc640 1 -- 172.21.15.77:0/228882635 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v137) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5c08058840 con 0x7f5c180cfa90 2024-03-06T07:30:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-06T07:30:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.089+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5bfc06b4d0 msgr2=0x7f5bfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.089+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5c180cf290 tx=0x7f5c14008040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.089+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 msgr2=0x7f5c1814f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c1814f6c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800c430 tx=0x7f5c0800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 shutdown_connections 2024-03-06T07:30:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c180cfa90 0x7f5c1814f6c0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5c1815fc30 0x7f5c1815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 --2- 172.21.15.77:0/228882635 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5c180ae6b0 0x7f5c18153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 >> 172.21.15.77:0/228882635 conn(0x7f5c1806b190 msgr2=0x7f5c1806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 shutdown_connections 2024-03-06T07:30:05.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.090+0000 7f5c1fc66640 1 -- 172.21.15.77:0/228882635 wait complete. 2024-03-06T07:30:05.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:05.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.214+0000 7fc353c88640 1 Processor -- start 2024-03-06T07:30:05.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.214+0000 7fc353c88640 1 -- start start 2024-03-06T07:30:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc353c88640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc34c158420 0x7fc34c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc353c88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc34c158800 0x7fc34c0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc353c88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c15b370 0x7fc34c15b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc353c88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34c15e650 con 0x7fc34c158800 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc353c88640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34c15e8a0 con 0x7fc34c158420 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc353c88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c15eb80 con 0x7fc34c15b370 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc351223640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc34c158800 0x7fc34c0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc351223640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc34c158800 0x7fc34c0b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60990/0 (socket says 172.21.15.77:60990) 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc351223640 1 -- 172.21.15.77:0/2814964720 learned_addr learned my addr 172.21.15.77:0/2814964720 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc351223640 1 -- 172.21.15.77:0/2814964720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c15b370 msgr2=0x7fc34c15b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc352225640 1 --2- 172.21.15.77:0/2814964720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c15b370 0x7fc34c15b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc351223640 1 --2- 172.21.15.77:0/2814964720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c15b370 0x7fc34c15b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc351a24640 1 --2- 172.21.15.77:0/2814964720 >> v2:172.21.15.158:3300/0 conn(0x7fc34c158420 0x7fc34c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.215+0000 7fc351223640 1 -- 172.21.15.77:0/2814964720 >> v2:172.21.15.158:3300/0 conn(0x7fc34c158420 msgr2=0x7fc34c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.216+0000 7fc351223640 1 --2- 172.21.15.77:0/2814964720 >> v2:172.21.15.158:3300/0 conn(0x7fc34c158420 0x7fc34c0b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.216+0000 7fc351223640 1 -- 172.21.15.77:0/2814964720 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c15ee60 con 0x7fc34c158800 2024-03-06T07:30:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.216+0000 7fc352225640 1 --2- 172.21.15.77:0/2814964720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c15b370 0x7fc34c15b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.216+0000 7fc351a24640 1 --2- 172.21.15.77:0/2814964720 >> v2:172.21.15.158:3300/0 conn(0x7fc34c158420 0x7fc34c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.216+0000 7fc351223640 1 --2- 172.21.15.77:0/2814964720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c158800 0x7fc34c0b3b00 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0d0470 tx=0x7fc33c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff76ee8d34773595 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.216+0000 7fc350a22640 1 -- 172.21.15.77:0/2814964720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c02e070 con 0x7fc34c158800 2024-03-06T07:30:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.216+0000 7fc350a22640 1 -- 172.21.15.77:0/2814964720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33c037070 con 0x7fc34c158800 2024-03-06T07:30:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.216+0000 7fc350a22640 1 -- 172.21.15.77:0/2814964720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c031a00 con 0x7fc34c158800 2024-03-06T07:30:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 -- 172.21.15.77:0/2814964720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c158800 msgr2=0x7fc34c0b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 --2- 172.21.15.77:0/2814964720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c158800 0x7fc34c0b3b00 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0d0470 tx=0x7fc33c02b880 comp rx=0 tx=0).stop 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 -- 172.21.15.77:0/2814964720 shutdown_connections 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 --2- 172.21.15.77:0/2814964720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c15b370 0x7fc34c15b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 --2- 172.21.15.77:0/2814964720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c158800 0x7fc34c0b3b00 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 --2- 172.21.15.77:0/2814964720 >> v2:172.21.15.158:3300/0 conn(0x7fc34c158420 0x7fc34c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 -- 172.21.15.77:0/2814964720 >> 172.21.15.77:0/2814964720 conn(0x7fc34c06afb0 msgr2=0x7fc34c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 -- 172.21.15.77:0/2814964720 shutdown_connections 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.217+0000 7fc353c88640 1 -- 172.21.15.77:0/2814964720 wait complete. 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.218+0000 7fc353c88640 1 Processor -- start 2024-03-06T07:30:05.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.218+0000 7fc353c88640 1 -- start start 2024-03-06T07:30:05.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc353c88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c158420 0x7fc34c14a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc353c88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc34c15b370 0x7fc34c149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc353c88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0ce6d0 0x7fc34c14b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc353c88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c06db10 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc353c88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c15bd60 con 0x7fc34c15b370 2024-03-06T07:30:05.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc353c88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c06dc90 con 0x7fc34c158420 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc352225640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0ce6d0 0x7fc34c14b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc352225640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0ce6d0 0x7fc34c14b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60992/0 (socket says 172.21.15.77:60992) 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc352225640 1 -- 172.21.15.77:0/286831530 learned_addr learned my addr 172.21.15.77:0/286831530 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc352225640 1 -- 172.21.15.77:0/286831530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c158420 msgr2=0x7fc34c14a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc352225640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c158420 0x7fc34c14a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc352225640 1 -- 172.21.15.77:0/286831530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc34c15b370 msgr2=0x7fc34c149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc352225640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc34c15b370 0x7fc34c149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.219+0000 7fc352225640 1 -- 172.21.15.77:0/286831530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c14a6c0 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.220+0000 7fc352225640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0ce6d0 0x7fc34c14b760 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fc34400c7c0 tx=0x7fc34400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.220+0000 7fc3427fc640 1 -- 172.21.15.77:0/286831530 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344015020 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.220+0000 7fc3427fc640 1 -- 172.21.15.77:0/286831530 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc344004230 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.220+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34c14a9a0 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.220+0000 7fc3427fc640 1 -- 172.21.15.77:0/286831530 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344003610 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.220+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34c14af60 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.221+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34c04f330 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.221+0000 7fc3427fc640 1 -- 172.21.15.77:0/286831530 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc344018020 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.221+0000 7fc3427fc640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33006b500 0x7fc33006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.222+0000 7fc351a24640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33006b500 0x7fc33006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.222+0000 7fc3427fc640 1 -- 172.21.15.77:0/286831530 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc34408a150 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.222+0000 7fc351a24640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33006b500 0x7fc33006d9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc348008ea0 tx=0x7fc3480083d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.226+0000 7fc3427fc640 1 -- 172.21.15.77:0/286831530 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc34401f020 con 0x7fc34c0ce6d0 2024-03-06T07:30:05.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:05.434+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc34c0af150 con 0x7fc34c0ce6d0 2024-03-06T07:30:06.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.122+0000 7fc3427fc640 1 -- 172.21.15.77:0/286831530 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v95) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc344059030 con 0x7fc34c0ce6d0 2024-03-06T07:30:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 20 and data pool 21 2024-03-06T07:30:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.129+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33006b500 msgr2=0x7fc33006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.129+0000 7fc353c88640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33006b500 0x7fc33006d9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc348008ea0 tx=0x7fc3480083d0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.129+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0ce6d0 msgr2=0x7fc34c14b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.129+0000 7fc353c88640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0ce6d0 0x7fc34c14b760 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fc34400c7c0 tx=0x7fc34400cc80 comp rx=0 tx=0).stop 2024-03-06T07:30:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.131+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 shutdown_connections 2024-03-06T07:30:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.131+0000 7fc353c88640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc33006b500 0x7fc33006d9c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.131+0000 7fc353c88640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0ce6d0 0x7fc34c14b760 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.131+0000 7fc353c88640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc34c15b370 0x7fc34c149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.131+0000 7fc353c88640 1 --2- 172.21.15.77:0/286831530 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc34c158420 0x7fc34c14a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.131+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 >> 172.21.15.77:0/286831530 conn(0x7fc34c06afb0 msgr2=0x7fc34c0726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.132+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 shutdown_connections 2024-03-06T07:30:06.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.133+0000 7fc353c88640 1 -- 172.21.15.77:0/286831530 wait complete. 2024-03-06T07:30:06.151 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.249+0000 7f7d876c2640 1 Processor -- start 2024-03-06T07:30:06.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.250+0000 7f7d876c2640 1 -- start start 2024-03-06T07:30:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.250+0000 7f7d876c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d8015dfe0 0x7f7d800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.250+0000 7f7d876c2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d800b3430 con 0x7f7d800cf830 2024-03-06T07:30:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.250+0000 7f7d876c2640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d800b3710 con 0x7f7d800d29c0 2024-03-06T07:30:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.250+0000 7f7d876c2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d800b31e0 con 0x7f7d8015dfe0 2024-03-06T07:30:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.251+0000 7f7d8545e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d8015dfe0 0x7f7d800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.251+0000 7f7d8545e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d8015dfe0 0x7f7d800cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32774/0 (socket says 172.21.15.77:32774) 2024-03-06T07:30:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.251+0000 7f7d84c5d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7d800cf830 0x7f7d800d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50372/0 (socket says 172.21.15.77:50372) 2024-03-06T07:30:06.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.251+0000 7f7d8545e640 1 -- 172.21.15.77:0/423140856 learned_addr learned my addr 172.21.15.77:0/423140856 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:06.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.251+0000 7f7d8545e640 1 -- 172.21.15.77:0/423140856 >> v1:172.21.15.158:6789/0 conn(0x7f7d800d29c0 legacy=0x7f7d800b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.251+0000 7f7d8545e640 1 -- 172.21.15.77:0/423140856 >> v1:172.21.15.77:6789/0 conn(0x7f7d800cf830 legacy=0x7f7d800d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.251+0000 7f7d8545e640 1 -- 172.21.15.77:0/423140856 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800b39f0 con 0x7f7d8015dfe0 2024-03-06T07:30:06.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.251+0000 7f7d77fff640 1 -- 172.21.15.77:0/423140856 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434810204 0 0) 0x7f7d800b3430 con 0x7f7d800cf830 2024-03-06T07:30:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d8545e640 1 --2- 172.21.15.77:0/423140856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d8015dfe0 0x7f7d800cf2f0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f7d800b3360 tx=0x7f7d7c02bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e585722aa2027d5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d77fff640 1 -- 172.21.15.77:0/423140856 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c02cb80 con 0x7f7d8015dfe0 2024-03-06T07:30:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d77fff640 1 -- 172.21.15.77:0/423140856 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c02cd20 con 0x7f7d8015dfe0 2024-03-06T07:30:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d77fff640 1 -- 172.21.15.77:0/423140856 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c034610 con 0x7f7d8015dfe0 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d876c2640 1 -- 172.21.15.77:0/423140856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d8015dfe0 msgr2=0x7f7d800cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d876c2640 1 --2- 172.21.15.77:0/423140856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d8015dfe0 0x7f7d800cf2f0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f7d800b3360 tx=0x7f7d7c02bfe0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d876c2640 1 -- 172.21.15.77:0/423140856 shutdown_connections 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d876c2640 1 --2- 172.21.15.77:0/423140856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d8015dfe0 0x7f7d800cf2f0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d876c2640 1 -- 172.21.15.77:0/423140856 >> 172.21.15.77:0/423140856 conn(0x7f7d800d3cd0 msgr2=0x7f7d800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d876c2640 1 -- 172.21.15.77:0/423140856 shutdown_connections 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.252+0000 7f7d876c2640 1 -- 172.21.15.77:0/423140856 wait complete. 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.253+0000 7f7d876c2640 1 Processor -- start 2024-03-06T07:30:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.253+0000 7f7d876c2640 1 -- start start 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d876c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d800cf830 0x7f7d8015b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d876c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d800d29c0 0x7f7d800cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d876c2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d8015dfe0 0x7f7d801f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d876c2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8015e3c0 con 0x7f7d800cf830 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d876c2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800b3d60 con 0x7f7d8015dfe0 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d876c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8015e540 con 0x7f7d800d29c0 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d8545e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d800cf830 0x7f7d8015b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d8545e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d800cf830 0x7f7d8015b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32790/0 (socket says 172.21.15.77:32790) 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.254+0000 7f7d8545e640 1 -- 172.21.15.77:0/2166581165 learned_addr learned my addr 172.21.15.77:0/2166581165 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d8545e640 1 -- 172.21.15.77:0/2166581165 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d800d29c0 msgr2=0x7f7d800cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d8545e640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d800d29c0 0x7f7d800cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d8545e640 1 -- 172.21.15.77:0/2166581165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d8015dfe0 msgr2=0x7f7d801f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d8545e640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d8015dfe0 0x7f7d801f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d8545e640 1 -- 172.21.15.77:0/2166581165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d801f7150 con 0x7f7d800cf830 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d8545e640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d800cf830 0x7f7d8015b590 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c000f80 tx=0x7f7d7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d75ffb640 1 -- 172.21.15.77:0/2166581165 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c043020 con 0x7f7d800cf830 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d75ffb640 1 -- 172.21.15.77:0/2166581165 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c03e450 con 0x7f7d800cf830 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d801f7430 con 0x7f7d800cf830 2024-03-06T07:30:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d75ffb640 1 -- 172.21.15.77:0/2166581165 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c03e840 con 0x7f7d800cf830 2024-03-06T07:30:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.255+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d801f79f0 con 0x7f7d800cf830 2024-03-06T07:30:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.256+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8004f330 con 0x7f7d800cf830 2024-03-06T07:30:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.257+0000 7f7d75ffb640 1 -- 172.21.15.77:0/2166581165 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d7c046020 con 0x7f7d800cf830 2024-03-06T07:30:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.257+0000 7f7d75ffb640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d6406b4d0 0x7f7d6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.257+0000 7f7d75ffb640 1 -- 172.21.15.77:0/2166581165 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d7c0acaf0 con 0x7f7d800cf830 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.257+0000 7f7d84c5d640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d6406b4d0 0x7f7d6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.258+0000 7f7d84c5d640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d6406b4d0 0x7f7d6406d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7d70007fd0 tx=0x7f7d70007900 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.261+0000 7f7d75ffb640 1 -- 172.21.15.77:0/2166581165 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d7c04d030 con 0x7f7d800cf830 2024-03-06T07:30:06.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.434+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d8006d590 con 0x7f7d800cf830 2024-03-06T07:30:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.436+0000 7f7d75ffb640 1 -- 172.21.15.77:0/2166581165 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f7d7c07ba00 con 0x7f7d800cf830 2024-03-06T07:30:06.438 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:06.438 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":138,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:05.435668+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.437+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d6406b4d0 msgr2=0x7f7d6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d6406b4d0 0x7f7d6406d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7d70007fd0 tx=0x7f7d70007900 comp rx=0 tx=0).stop 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d800cf830 msgr2=0x7f7d8015b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d800cf830 0x7f7d8015b590 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c000f80 tx=0x7f7d7c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 shutdown_connections 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7d8015dfe0 0x7f7d801f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7d800d29c0 0x7f7d800cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7d6406b4d0 0x7f7d6406d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 --2- 172.21.15.77:0/2166581165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d800cf830 0x7f7d8015b590 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 >> 172.21.15.77:0/2166581165 conn(0x7f7d800d3cd0 msgr2=0x7f7d8015ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 shutdown_connections 2024-03-06T07:30:06.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.438+0000 7f7d876c2640 1 -- 172.21.15.77:0/2166581165 wait complete. 2024-03-06T07:30:06.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:06.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.548+0000 7ff0c61e2640 1 Processor -- start 2024-03-06T07:30:06.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.548+0000 7ff0c61e2640 1 -- start start 2024-03-06T07:30:06.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0c61e2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff0c01424f0 0x7ff0c00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0c61e2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c006da40 0x7ff0c015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0c61e2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c00cc980 con 0x7ff0c00cd7a0 2024-03-06T07:30:06.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0c61e2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c00cd220 con 0x7ff0c01424f0 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0c61e2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00ccc60 con 0x7ff0c006da40 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0beffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff0c00cd7a0 0x7ff0c006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50384/0 (socket says 172.21.15.77:50384) 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0beffd640 1 -- 172.21.15.77:0/323125692 learned_addr learned my addr 172.21.15.77:0/323125692 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0bf7fe640 1 --2- 172.21.15.77:0/323125692 >> v2:172.21.15.158:3300/0 conn(0x7ff0c01424f0 0x7ff0c00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0bffff640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c006da40 0x7ff0c015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264330176 0 0) 0x7ff0c00cc980 con 0x7ff0c00cd7a0 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a8003520 con 0x7ff0c00cd7a0 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1541362872 0 0) 0x7ff0a8003520 con 0x7ff0c00cd7a0 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c006da40 msgr2=0x7ff0c015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c006da40 0x7ff0c015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 >> v2:172.21.15.158:3300/0 conn(0x7ff0c01424f0 msgr2=0x7ff0c00cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 --2- 172.21.15.77:0/323125692 >> v2:172.21.15.158:3300/0 conn(0x7ff0c01424f0 0x7ff0c00cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c00ccf40 con 0x7ff0c00cd7a0 2024-03-06T07:30:06.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ff0b0001670 con 0x7ff0c00cd7a0 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 >> v1:172.21.15.77:6789/0 conn(0x7ff0c00cd7a0 legacy=0x7ff0c006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a80039b0 0x7ff0a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c006da40 0x7ff0a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00cd220 con 0x7ff0a8004800 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00ccc60 con 0x7ff0a80039b0 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.549+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a800b9c0 con 0x7ff0c006da40 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.550+0000 7ff0beffd640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.550+0000 7ff0bf7fe640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a80039b0 0x7ff0a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.550+0000 7ff0bffff640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c006da40 0x7ff0a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.550+0000 7ff0beffd640 1 -- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c006da40 msgr2=0x7ff0a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.550+0000 7ff0beffd640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c006da40 0x7ff0a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.550+0000 7ff0beffd640 1 -- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a80039b0 msgr2=0x7ff0a8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.550+0000 7ff0beffd640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0a80039b0 0x7ff0a8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.550+0000 7ff0beffd640 1 -- 172.21.15.77:0/323125692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a8003520 con 0x7ff0a8004800 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0beffd640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0a8004c80 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0007fd0 tx=0x7ff0a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e435299f2bc7846 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0007d50 con 0x7ff0a8004800 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b0030070 con 0x7ff0a8004800 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0be7fc640 1 -- 172.21.15.77:0/323125692 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b002cd40 con 0x7ff0a8004800 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0c61e2640 1 -- 172.21.15.77:0/323125692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 msgr2=0x7ff0a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0c61e2640 1 --2- 172.21.15.77:0/323125692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0a8004c80 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0007fd0 tx=0x7ff0a8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0bffff640 1 -- 172.21.15.77:0/323125692 reap_dead start 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0c61e2640 1 -- 172.21.15.77:0/323125692 shutdown_connections 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0c61e2640 1 -- 172.21.15.77:0/323125692 >> 172.21.15.77:0/323125692 conn(0x7ff0c00cf8a0 msgr2=0x7ff0c00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0c61e2640 1 -- 172.21.15.77:0/323125692 shutdown_connections 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.551+0000 7ff0c61e2640 1 -- 172.21.15.77:0/323125692 wait complete. 2024-03-06T07:30:06.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.552+0000 7ff0c61e2640 1 Processor -- start 2024-03-06T07:30:06.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0c61e2640 1 -- start start 2024-03-06T07:30:06.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0c61e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0c013f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0c61e2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0c013d430 0x7ff0c013aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0c61e2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c013b400 0x7ff0c015a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0c61e2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c015d070 con 0x7ff0a8004800 2024-03-06T07:30:06.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0c61e2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c015cef0 con 0x7ff0c013d430 2024-03-06T07:30:06.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0c61e2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c015d1f0 con 0x7ff0c013b400 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0c013f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0c013f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32794/0 (socket says 172.21.15.77:32794) 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.553+0000 7ff0bf7fe640 1 -- 172.21.15.77:0/1745271069 learned_addr learned my addr 172.21.15.77:0/1745271069 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0bffff640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c013b400 0x7ff0c015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0beffd640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0c013d430 0x7ff0c013aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0bf7fe640 1 -- 172.21.15.77:0/1745271069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c013b400 msgr2=0x7ff0c015a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0bf7fe640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c013b400 0x7ff0c015a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0bf7fe640 1 -- 172.21.15.77:0/1745271069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0c013d430 msgr2=0x7ff0c013aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0bf7fe640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0c013d430 0x7ff0c013aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0bf7fe640 1 -- 172.21.15.77:0/1745271069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c0159d70 con 0x7ff0a8004800 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0bf7fe640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0c013f330 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40119c0 tx=0x7ff0b4011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0a7fff640 1 -- 172.21.15.77:0/1745271069 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b400faf0 con 0x7ff0a8004800 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0a7fff640 1 -- 172.21.15.77:0/1745271069 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b400fc90 con 0x7ff0a8004800 2024-03-06T07:30:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c015a050 con 0x7ff0a8004800 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0a7fff640 1 -- 172.21.15.77:0/1745271069 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4014440 con 0x7ff0a8004800 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.554+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c013ee10 con 0x7ff0a8004800 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.555+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c004f330 con 0x7ff0a8004800 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.555+0000 7ff0a7fff640 1 -- 172.21.15.77:0/1745271069 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0b4012040 con 0x7ff0a8004800 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.556+0000 7ff0a7fff640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0ac06b4d0 0x7ff0ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.556+0000 7ff0a7fff640 1 -- 172.21.15.77:0/1745271069 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0b408a8e0 con 0x7ff0a8004800 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.556+0000 7ff0beffd640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0ac06b4d0 0x7ff0ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.556+0000 7ff0beffd640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0ac06b4d0 0x7ff0ac06d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0034000 tx=0x7ff0b0007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.560+0000 7ff0a7fff640 1 -- 172.21.15.77:0/1745271069 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0b4019030 con 0x7ff0a8004800 2024-03-06T07:30:06.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.732+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0c015bcc0 con 0x7ff0a8004800 2024-03-06T07:30:06.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.733+0000 7ff0a7fff640 1 -- 172.21.15.77:0/1745271069 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16680 (secure 0 0 0) 0x7ff0b40596d0 con 0x7ff0a8004800 2024-03-06T07:30:06.735 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:06.736 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":138,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:05.435668+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:06.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0ac06b4d0 msgr2=0x7ff0ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0ac06b4d0 0x7ff0ac06d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0034000 tx=0x7ff0b0007000 comp rx=0 tx=0).stop 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 msgr2=0x7ff0c013f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0c013f330 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40119c0 tx=0x7ff0b4011e80 comp rx=0 tx=0).stop 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 shutdown_connections 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0c013b400 0x7ff0c015a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0c013d430 0x7ff0c013aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0ac06b4d0 0x7ff0ac06d990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 --2- 172.21.15.77:0/1745271069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0a8004800 0x7ff0c013f330 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.735+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 >> 172.21.15.77:0/1745271069 conn(0x7ff0c00cf8a0 msgr2=0x7ff0c006c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.736+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 shutdown_connections 2024-03-06T07:30:06.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.736+0000 7ff0c61e2640 1 -- 172.21.15.77:0/1745271069 wait complete. 2024-03-06T07:30:06.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:30:06.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.849+0000 7fce3b75c640 1 Processor -- start 2024-03-06T07:30:06.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.849+0000 7fce3b75c640 1 -- start start 2024-03-06T07:30:06.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.850+0000 7fce3b75c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce340d4860 0x7fce340d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.850+0000 7fce3b75c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce341618b0 con 0x7fce341614d0 2024-03-06T07:30:06.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.850+0000 7fce3b75c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340cf400 con 0x7fce340b0e30 2024-03-06T07:30:06.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.850+0000 7fce3b75c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce340d4cf0 con 0x7fce340d4860 2024-03-06T07:30:06.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.850+0000 7fce394f8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fce341614d0 0x7fce340d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50386/0 (socket says 172.21.15.77:50386) 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.850+0000 7fce394f8640 1 -- 172.21.15.77:0/169410654 learned_addr learned my addr 172.21.15.77:0/169410654 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce38cf7640 1 --2- 172.21.15.77:0/169410654 >> v2:172.21.15.158:3300/0 conn(0x7fce340d4860 0x7fce340d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220013996 0 0) 0x7fce341618b0 con 0x7fce341614d0 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce20003520 con 0x7fce341614d0 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3910770629 0 0) 0x7fce20003520 con 0x7fce341614d0 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 >> v2:172.21.15.158:3300/0 conn(0x7fce340d4860 msgr2=0x7fce340d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 --2- 172.21.15.77:0/169410654 >> v2:172.21.15.158:3300/0 conn(0x7fce340d4860 0x7fce340d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 >> v1:172.21.15.158:6789/0 conn(0x7fce340b0e30 legacy=0x7fce340b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340cf6a0 con 0x7fce341614d0 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fce30001a10 con 0x7fce341614d0 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 >> v1:172.21.15.77:6789/0 conn(0x7fce341614d0 legacy=0x7fce340d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce20004800 0x7fce20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce200051c0 0x7fce2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce340d4cf0 con 0x7fce200039b0 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2000c060 con 0x7fce200051c0 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2000c260 con 0x7fce20004800 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce394f8640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce38cf7640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce20004800 0x7fce20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce39cf9640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce200051c0 0x7fce2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce394f8640 1 -- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce20004800 msgr2=0x7fce20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce394f8640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce20004800 0x7fce20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce394f8640 1 -- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce200051c0 msgr2=0x7fce2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce394f8640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce200051c0 0x7fce2000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce394f8640 1 -- 172.21.15.77:0/169410654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce20003520 con 0x7fce200039b0 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce38cf7640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce20004800 0x7fce20004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce39cf9640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce200051c0 0x7fce2000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.851+0000 7fce39cf9640 1 -- 172.21.15.77:0/169410654 reap_dead start 2024-03-06T07:30:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.852+0000 7fce394f8640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce20003dd0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fce3000a8f0 tx=0x7fce20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7d972c96acd995 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.852+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3000a4c0 con 0x7fce200039b0 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.852+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce30034650 con 0x7fce200039b0 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.852+0000 7fce2bfff640 1 -- 172.21.15.77:0/169410654 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30034a40 con 0x7fce200039b0 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.853+0000 7fce3b75c640 1 -- 172.21.15.77:0/169410654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 msgr2=0x7fce20003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.853+0000 7fce3b75c640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce20003dd0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fce3000a8f0 tx=0x7fce20005830 comp rx=0 tx=0).stop 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.853+0000 7fce3b75c640 1 -- 172.21.15.77:0/169410654 shutdown_connections 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.853+0000 7fce3b75c640 1 --2- 172.21.15.77:0/169410654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce20003dd0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.853+0000 7fce3b75c640 1 -- 172.21.15.77:0/169410654 >> 172.21.15.77:0/169410654 conn(0x7fce340d3d40 msgr2=0x7fce340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.853+0000 7fce3b75c640 1 -- 172.21.15.77:0/169410654 shutdown_connections 2024-03-06T07:30:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.853+0000 7fce3b75c640 1 -- 172.21.15.77:0/169410654 wait complete. 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.854+0000 7fce3b75c640 1 Processor -- start 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.854+0000 7fce3b75c640 1 -- start start 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce3b75c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce3414e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce3b75c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce341f7290 0x7fce341f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce3b75c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce340b86a0 0x7fce340b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce3b75c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce340b0910 con 0x7fce200039b0 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce3b75c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce340cfc60 con 0x7fce341f7290 2024-03-06T07:30:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce3b75c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce340d1910 con 0x7fce340b86a0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce3414e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce3414e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32814/0 (socket says 172.21.15.77:32814) 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 -- 172.21.15.77:0/2240546949 learned_addr learned my addr 172.21.15.77:0/2240546949 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce38cf7640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce341f7290 0x7fce341f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce39cf9640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce340b86a0 0x7fce340b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 -- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce340b86a0 msgr2=0x7fce340b8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce340b86a0 0x7fce340b8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 -- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce341f7290 msgr2=0x7fce341f7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce341f7290 0x7fce341f7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 -- 172.21.15.77:0/2240546949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3414e9e0 con 0x7fce200039b0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce38cf7640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce341f7290 0x7fce341f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce39cf9640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce340b86a0 0x7fce340b8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce394f8640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce3414e240 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fce3000aa10 tx=0x7fce30001de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce29ffb640 1 -- 172.21.15.77:0/2240546949 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30039040 con 0x7fce200039b0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce29ffb640 1 -- 172.21.15.77:0/2240546949 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce30043070 con 0x7fce200039b0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce29ffb640 1 -- 172.21.15.77:0/2240546949 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30001a10 con 0x7fce200039b0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.855+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce340caf90 con 0x7fce200039b0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.856+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce340cb530 con 0x7fce200039b0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.857+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3415fac0 con 0x7fce200039b0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.857+0000 7fce29ffb640 1 -- 172.21.15.77:0/2240546949 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce30034650 con 0x7fce200039b0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.857+0000 7fce29ffb640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.857+0000 7fce29ffb640 1 -- 172.21.15.77:0/2240546949 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce300ac280 con 0x7fce200039b0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.857+0000 7fce38cf7640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.858+0000 7fce38cf7640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce1806b4d0 0x7fce1806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fce341f76b0 tx=0x7fce24008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:06.862+0000 7fce29ffb640 1 -- 172.21.15.77:0/2240546949 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce3007b190 con 0x7fce200039b0 2024-03-06T07:30:07.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.070+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fce34068ff0 con 0x7fce200039b0 2024-03-06T07:30:07.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.118+0000 7fce29ffb640 1 -- 172.21.15.77:0/2240546949 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v96) v1 ==== 118+0+0 (secure 0 0 0) 0x7fce3007aa90 con 0x7fce200039b0 2024-03-06T07:30:07.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce1806b4d0 msgr2=0x7fce1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce1806b4d0 0x7fce1806d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fce341f76b0 tx=0x7fce24008040 comp rx=0 tx=0).stop 2024-03-06T07:30:07.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 msgr2=0x7fce3414e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce3414e240 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fce3000aa10 tx=0x7fce30001de0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 shutdown_connections 2024-03-06T07:30:07.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce340b86a0 0x7fce340b8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce341f7290 0x7fce341f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 --2- 172.21.15.77:0/2240546949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce200039b0 0x7fce3414e240 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 >> 172.21.15.77:0/2240546949 conn(0x7fce340d3d40 msgr2=0x7fce3406c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 shutdown_connections 2024-03-06T07:30:07.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.120+0000 7fce3b75c640 1 -- 172.21.15.77:0/2240546949 wait complete. 2024-03-06T07:30:07.134 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:30:07.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:30:07.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.237+0000 7f098ebc9640 1 Processor -- start 2024-03-06T07:30:07.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.237+0000 7f098ebc9640 1 -- start start 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.238+0000 7f098ebc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 0x7f09880d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.238+0000 7f098ebc9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f09880d2480 0x7f09880b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.238+0000 7f098ebc9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09881618b0 con 0x7f09880d1910 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.238+0000 7f098ebc9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09880cf400 con 0x7f09880d2480 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.238+0000 7f098ebc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09880cf6a0 con 0x7f09881614d0 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.238+0000 7f098d166640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f09880d2480 0x7f09880b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.238+0000 7f0987fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f09880d1910 0x7f09880d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50398/0 (socket says 172.21.15.77:50398) 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.238+0000 7f0987fff640 1 -- 172.21.15.77:0/1928266039 learned_addr learned my addr 172.21.15.77:0/1928266039 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:07.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.239+0000 7f098c965640 1 --2- 172.21.15.77:0/1928266039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 0x7f09880d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.239+0000 7f098d166640 1 -- 172.21.15.77:0/1928266039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 msgr2=0x7f09880d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.239+0000 7f098d166640 1 --2- 172.21.15.77:0/1928266039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 0x7f09880d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.239+0000 7f098d166640 1 -- 172.21.15.77:0/1928266039 >> v1:172.21.15.77:6789/0 conn(0x7f09880d1910 legacy=0x7f09880d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.239+0000 7f09877fe640 1 -- 172.21.15.77:0/1928266039 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191424058 0 0) 0x7f09881618b0 con 0x7f09880d1910 2024-03-06T07:30:07.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.239+0000 7f098d166640 1 -- 172.21.15.77:0/1928266039 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09880cf980 con 0x7f09880d2480 2024-03-06T07:30:07.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f098d166640 1 --2- 172.21.15.77:0/1928266039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d2480 0x7f09880b0e30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0980009d90 tx=0x7f098002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7fc28f137f72e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f09877fe640 1 -- 172.21.15.77:0/1928266039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980039070 con 0x7f09880d2480 2024-03-06T07:30:07.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f09877fe640 1 -- 172.21.15.77:0/1928266039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098002cc50 con 0x7f09880d2480 2024-03-06T07:30:07.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f09877fe640 1 -- 172.21.15.77:0/1928266039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980034440 con 0x7f09880d2480 2024-03-06T07:30:07.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f098ebc9640 1 -- 172.21.15.77:0/1928266039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d2480 msgr2=0x7f09880b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f098ebc9640 1 --2- 172.21.15.77:0/1928266039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d2480 0x7f09880b0e30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0980009d90 tx=0x7f098002bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f098ebc9640 1 -- 172.21.15.77:0/1928266039 shutdown_connections 2024-03-06T07:30:07.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f098ebc9640 1 --2- 172.21.15.77:0/1928266039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d2480 0x7f09880b0e30 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f098ebc9640 1 --2- 172.21.15.77:0/1928266039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 0x7f09880d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.240+0000 7f098ebc9640 1 -- 172.21.15.77:0/1928266039 >> 172.21.15.77:0/1928266039 conn(0x7f09880d3d40 msgr2=0x7f09880d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.241+0000 7f098ebc9640 1 -- 172.21.15.77:0/1928266039 shutdown_connections 2024-03-06T07:30:07.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.241+0000 7f098ebc9640 1 -- 172.21.15.77:0/1928266039 wait complete. 2024-03-06T07:30:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.242+0000 7f098ebc9640 1 Processor -- start 2024-03-06T07:30:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.242+0000 7f098ebc9640 1 -- start start 2024-03-06T07:30:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.243+0000 7f098ebc9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d1910 0x7f09880b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.243+0000 7f098ebc9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09880d2480 0x7f09881f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.243+0000 7f098ebc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 0x7f098815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.243+0000 7f098ebc9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09880b1370 con 0x7f09880d1910 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.243+0000 7f098ebc9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09880b0910 con 0x7f09880d2480 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.243+0000 7f098ebc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09880b14f0 con 0x7f09881614d0 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.243+0000 7f098c965640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d1910 0x7f09880b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.243+0000 7f098c965640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d1910 0x7f09880b8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32830/0 (socket says 172.21.15.77:32830) 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098c965640 1 -- 172.21.15.77:0/2357638549 learned_addr learned my addr 172.21.15.77:0/2357638549 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098d166640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 0x7f098815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f0987fff640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09880d2480 0x7f09881f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098c965640 1 -- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 msgr2=0x7f098815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098c965640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 0x7f098815cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098c965640 1 -- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09880d2480 msgr2=0x7f09881f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098c965640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09880d2480 0x7f09881f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098c965640 1 -- 172.21.15.77:0/2357638549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09880c7d70 con 0x7f09880d1910 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f0987fff640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09880d2480 0x7f09881f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098c965640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d1910 0x7f09880b8c00 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f097c00d5f0 tx=0x7f097c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f09857fa640 1 -- 172.21.15.77:0/2357638549 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c0125f0 con 0x7f09880d1910 2024-03-06T07:30:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f09857fa640 1 -- 172.21.15.77:0/2357638549 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097c019070 con 0x7f09880d1910 2024-03-06T07:30:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09880c8050 con 0x7f09880d1910 2024-03-06T07:30:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f09857fa640 1 -- 172.21.15.77:0/2357638549 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c0129d0 con 0x7f09880d1910 2024-03-06T07:30:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.244+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09880c8610 con 0x7f09880d1910 2024-03-06T07:30:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.245+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0988068ff0 con 0x7f09880d1910 2024-03-06T07:30:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.246+0000 7f09857fa640 1 -- 172.21.15.77:0/2357638549 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f097c012b70 con 0x7f09880d1910 2024-03-06T07:30:07.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.246+0000 7f09857fa640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f096c06b4d0 0x7f096c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.246+0000 7f09857fa640 1 -- 172.21.15.77:0/2357638549 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f097c08a5a0 con 0x7f09880d1910 2024-03-06T07:30:07.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.246+0000 7f0987fff640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f096c06b4d0 0x7f096c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.247+0000 7f0987fff640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f096c06b4d0 0x7f096c06d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f09880a1830 tx=0x7f0974007c10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.250+0000 7f09857fa640 1 -- 172.21.15.77:0/2357638549 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f097c059420 con 0x7f09880d1910 2024-03-06T07:30:07.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:07.466+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f09880a1c10 con 0x7f09880d1910 2024-03-06T07:30:08.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.124+0000 7f09857fa640 1 -- 172.21.15.77:0/2357638549 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v97) v1 ==== 122+0+0 (secure 0 0 0) 0x7f097c058d20 con 0x7f09880d1910 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f096c06b4d0 msgr2=0x7f096c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f096c06b4d0 0x7f096c06d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f09880a1830 tx=0x7f0974007c10 comp rx=0 tx=0).stop 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d1910 msgr2=0x7f09880b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d1910 0x7f09880b8c00 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f097c00d5f0 tx=0x7f097c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 shutdown_connections 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09881614d0 0x7f098815cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09880d2480 0x7f09881f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f096c06b4d0 0x7f096c06d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 --2- 172.21.15.77:0/2357638549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09880d1910 0x7f09880b8c00 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 >> 172.21.15.77:0/2357638549 conn(0x7f09880d3d40 msgr2=0x7f09880d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 shutdown_connections 2024-03-06T07:30:08.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.127+0000 7f098ebc9640 1 -- 172.21.15.77:0/2357638549 wait complete. 2024-03-06T07:30:08.139 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:30:08.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:08.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.237+0000 7ff542f9a640 1 Processor -- start 2024-03-06T07:30:08.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.237+0000 7ff542f9a640 1 -- start start 2024-03-06T07:30:08.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff542f9a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff53c0cdeb0 0x7ff53c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff542f9a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53c071f40 con 0x7ff53c0cce40 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff542f9a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53c070a70 con 0x7ff53c1515e0 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff542f9a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c0cd2f0 con 0x7ff53c0cdeb0 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff53bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff53c0cce40 0x7ff53c0cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50414/0 (socket says 172.21.15.77:50414) 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff53bfff640 1 -- 172.21.15.77:0/3782248241 learned_addr learned my addr 172.21.15.77:0/3782248241 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff541537640 1 --2- 172.21.15.77:0/3782248241 >> v2:172.21.15.158:3300/0 conn(0x7ff53c0cdeb0 0x7ff53c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992679636 0 0) 0x7ff53c071f40 con 0x7ff53c0cce40 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff528003440 con 0x7ff53c0cce40 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672245655 0 0) 0x7ff53c070a70 con 0x7ff53c1515e0 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.238+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53c071f40 con 0x7ff53c1515e0 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1459216712 0 0) 0x7ff528003440 con 0x7ff53c0cce40 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 >> v2:172.21.15.158:3300/0 conn(0x7ff53c0cdeb0 msgr2=0x7ff53c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 --2- 172.21.15.77:0/3782248241 >> v2:172.21.15.158:3300/0 conn(0x7ff53c0cdeb0 0x7ff53c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 >> v1:172.21.15.158:6789/0 conn(0x7ff53c1515e0 legacy=0x7ff53c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c070d50 con 0x7ff53c0cce40 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ff524001690 con 0x7ff53c0cce40 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 >> v1:172.21.15.77:6789/0 conn(0x7ff53c0cce40 legacy=0x7ff53c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff528003c30 0x7ff528008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff528004160 0x7ff52800ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52800b4b0 0x7ff52800d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0cd2f0 con 0x7ff52800b4b0 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52800e250 con 0x7ff528003c30 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52800e450 con 0x7ff528004160 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff541537640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52800b4b0 0x7ff52800d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff540d36640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff528003c30 0x7ff528008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff53bfff640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff528004160 0x7ff52800ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff541537640 1 -- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff528004160 msgr2=0x7ff52800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff541537640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff528004160 0x7ff52800ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff541537640 1 -- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff528003c30 msgr2=0x7ff528008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff541537640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff528003c30 0x7ff528008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff541537640 1 -- 172.21.15.77:0/3782248241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff528003440 con 0x7ff52800b4b0 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.239+0000 7ff540d36640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff528003c30 0x7ff528008850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff541537640 1 -- 172.21.15.77:0/3782248241 reap_dead start 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff541537640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52800b4b0 0x7ff52800d970 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7ff534009fc0 tx=0x7ff528003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4ebac744593287 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534009c10 con 0x7ff52800b4b0 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53402cca0 con 0x7ff52800b4b0 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff53b7fe640 1 -- 172.21.15.77:0/3782248241 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534003bd0 con 0x7ff52800b4b0 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff542f9a640 1 -- 172.21.15.77:0/3782248241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52800b4b0 msgr2=0x7ff52800d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff542f9a640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52800b4b0 0x7ff52800d970 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7ff534009fc0 tx=0x7ff528003730 comp rx=0 tx=0).stop 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff542f9a640 1 -- 172.21.15.77:0/3782248241 shutdown_connections 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff542f9a640 1 --2- 172.21.15.77:0/3782248241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52800b4b0 0x7ff52800d970 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.240+0000 7ff542f9a640 1 -- 172.21.15.77:0/3782248241 >> 172.21.15.77:0/3782248241 conn(0x7ff53c0cf8a0 msgr2=0x7ff53c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.241+0000 7ff542f9a640 1 -- 172.21.15.77:0/3782248241 shutdown_connections 2024-03-06T07:30:08.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.241+0000 7ff542f9a640 1 -- 172.21.15.77:0/3782248241 wait complete. 2024-03-06T07:30:08.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.242+0000 7ff542f9a640 1 Processor -- start 2024-03-06T07:30:08.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.242+0000 7ff542f9a640 1 -- start start 2024-03-06T07:30:08.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff542f9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff52800b4b0 0x7ff53c1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff542f9a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c159630 0x7ff53c146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff542f9a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff53c159a90 0x7ff53c146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff542f9a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c15b430 con 0x7ff53c159630 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff542f9a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c071310 con 0x7ff53c159a90 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff542f9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c15b5b0 con 0x7ff52800b4b0 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c159630 0x7ff53c146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c159630 0x7ff53c146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32850/0 (socket says 172.21.15.77:32850) 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 -- 172.21.15.77:0/967965668 learned_addr learned my addr 172.21.15.77:0/967965668 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff540d36640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff52800b4b0 0x7ff53c1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff541537640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff53c159a90 0x7ff53c146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 -- 172.21.15.77:0/967965668 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff52800b4b0 msgr2=0x7ff53c1f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff52800b4b0 0x7ff53c1f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 -- 172.21.15.77:0/967965668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff53c159a90 msgr2=0x7ff53c146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff53c159a90 0x7ff53c146bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 -- 172.21.15.77:0/967965668 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c15c7d0 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff53bfff640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c159630 0x7ff53c146680 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e700 tx=0x7ff52400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff5397fa640 1 -- 172.21.15.77:0/967965668 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52400ee10 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff5397fa640 1 -- 172.21.15.77:0/967965668 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff524018070 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c15cab0 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff5397fa640 1 -- 172.21.15.77:0/967965668 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52400ca90 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.243+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c1f4570 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.245+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53c068840 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.245+0000 7ff5397fa640 1 -- 172.21.15.77:0/967965668 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff52400cc30 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.245+0000 7ff5397fa640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff51c06b4d0 0x7ff51c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.245+0000 7ff5397fa640 1 -- 172.21.15.77:0/967965668 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff52408a030 con 0x7ff53c159630 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.245+0000 7ff540d36640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff51c06b4d0 0x7ff51c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.246+0000 7ff540d36640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff51c06b4d0 0x7ff51c06d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff52c005410 tx=0x7ff52c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.250+0000 7ff5397fa640 1 -- 172.21.15.77:0/967965668 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff524058eb0 con 0x7ff53c159630 2024-03-06T07:30:08.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.484+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff53c04f330 con 0x7ff53c159630 2024-03-06T07:30:08.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.485+0000 7ff5397fa640 1 -- 172.21.15.77:0/967965668 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4984 (secure 0 0 0) 0x7ff5240587b0 con 0x7ff53c159630 2024-03-06T07:30:08.487 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:08.487 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":97,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2532910127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2532910127},{"type":"v1","addr":"172.21.15.158:6835","nonce":2532910127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":8347,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/3282592218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3282592218},{"type":"v1","addr":"172.21.15.158:6833","nonce":3282592218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":9177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:07.467677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:30:08.487 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 97 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff51c06b4d0 msgr2=0x7ff51c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff51c06b4d0 0x7ff51c06d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff52c005410 tx=0x7ff52c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c159630 msgr2=0x7ff53c146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c159630 0x7ff53c146680 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e700 tx=0x7ff52400ebc0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 shutdown_connections 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff51c06b4d0 0x7ff51c06d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff53c159630 0x7ff53c146680 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff53c159a90 0x7ff53c146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 --2- 172.21.15.77:0/967965668 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff52800b4b0 0x7ff53c1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 >> 172.21.15.77:0/967965668 conn(0x7ff53c0cf8a0 msgr2=0x7ff53c14d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 shutdown_connections 2024-03-06T07:30:08.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.487+0000 7ff542f9a640 1 -- 172.21.15.77:0/967965668 wait complete. 2024-03-06T07:30:08.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:08.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.599+0000 7f8043581640 1 Processor -- start 2024-03-06T07:30:08.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.600+0000 7f8043581640 1 -- start start 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.600+0000 7f8043581640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c0b3060 0x7f803c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.600+0000 7f8043581640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f803c15bf30 0x7f803c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.600+0000 7f8043581640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c15f2d0 con 0x7f803c13dc10 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.600+0000 7f8043581640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803c0cf3f0 con 0x7f803c15bf30 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.600+0000 7f8043581640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803c15b210 con 0x7f803c0b3060 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.601+0000 7f8041b1e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f803c15bf30 0x7f803c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.601+0000 7f8041b1e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f803c15bf30 0x7f803c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49978/0 (socket says 172.21.15.77:49978) 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.601+0000 7f8041b1e640 1 -- 172.21.15.77:0/2079662167 learned_addr learned my addr 172.21.15.77:0/2079662167 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.601+0000 7f8040b1c640 1 --2- 172.21.15.77:0/2079662167 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c0b3060 0x7f803c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.601+0000 7f8041b1e640 1 -- 172.21.15.77:0/2079662167 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c0b3060 msgr2=0x7f803c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.601+0000 7f8041b1e640 1 --2- 172.21.15.77:0/2079662167 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c0b3060 0x7f803c15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.601+0000 7f8041b1e640 1 -- 172.21.15.77:0/2079662167 >> v1:172.21.15.158:6789/0 conn(0x7f803c13dc10 legacy=0x7f803c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.601+0000 7f8041b1e640 1 -- 172.21.15.77:0/2079662167 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c15f580 con 0x7f803c15bf30 2024-03-06T07:30:08.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8041b1e640 1 --2- 172.21.15.77:0/2079662167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c15bf30 0x7f803c15ed90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f8038009e50 tx=0x7f803802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e178e02207d34e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8033fff640 1 -- 172.21.15.77:0/2079662167 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803802ca70 con 0x7f803c15bf30 2024-03-06T07:30:08.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8033fff640 1 -- 172.21.15.77:0/2079662167 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803802cc10 con 0x7f803c15bf30 2024-03-06T07:30:08.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8033fff640 1 -- 172.21.15.77:0/2079662167 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80380343f0 con 0x7f803c15bf30 2024-03-06T07:30:08.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8043581640 1 -- 172.21.15.77:0/2079662167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c15bf30 msgr2=0x7f803c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8043581640 1 --2- 172.21.15.77:0/2079662167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c15bf30 0x7f803c15ed90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f8038009e50 tx=0x7f803802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8043581640 1 -- 172.21.15.77:0/2079662167 shutdown_connections 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8043581640 1 --2- 172.21.15.77:0/2079662167 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c0b3060 0x7f803c15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8043581640 1 --2- 172.21.15.77:0/2079662167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c15bf30 0x7f803c15ed90 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8043581640 1 -- 172.21.15.77:0/2079662167 >> 172.21.15.77:0/2079662167 conn(0x7f803c0cf8a0 msgr2=0x7f803c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8043581640 1 -- 172.21.15.77:0/2079662167 shutdown_connections 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.602+0000 7f8043581640 1 -- 172.21.15.77:0/2079662167 wait complete. 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.603+0000 7f8043581640 1 Processor -- start 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8043581640 1 -- start start 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8043581640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f803c0b3060 0x7f803c13c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8043581640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c13dc10 0x7f803c1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8043581640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c15bf30 0x7f803c158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8043581640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c13dff0 con 0x7f803c13dc10 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8043581640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c15b730 con 0x7f803c0b3060 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8043581640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803c0708d0 con 0x7f803c15bf30 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8040b1c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c13dc10 0x7f803c1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8040b1c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c13dc10 0x7f803c1585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49980/0 (socket says 172.21.15.77:49980) 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.604+0000 7f8040b1c640 1 -- 172.21.15.77:0/4004828520 learned_addr learned my addr 172.21.15.77:0/4004828520 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8041b1e640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c15bf30 0x7f803c158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f804131d640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f803c0b3060 0x7f803c13c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8040b1c640 1 -- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c15bf30 msgr2=0x7f803c158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8040b1c640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c15bf30 0x7f803c158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8040b1c640 1 -- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f803c0b3060 msgr2=0x7f803c13c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8040b1c640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f803c0b3060 0x7f803c13c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8040b1c640 1 -- 172.21.15.77:0/4004828520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c155f00 con 0x7f803c13dc10 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f804131d640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f803c0b3060 0x7f803c13c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8041b1e640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c15bf30 0x7f803c158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8040b1c640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c13dc10 0x7f803c1585e0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f802c00d800 tx=0x7f802c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8031ffb640 1 -- 172.21.15.77:0/4004828520 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c00b6c0 con 0x7f803c13dc10 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8031ffb640 1 -- 172.21.15.77:0/4004828520 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802c019070 con 0x7f803c13dc10 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8031ffb640 1 -- 172.21.15.77:0/4004828520 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c00ba80 con 0x7f803c13dc10 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803c1561e0 con 0x7f803c13dc10 2024-03-06T07:30:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.605+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c13cb90 con 0x7f803c13dc10 2024-03-06T07:30:08.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.606+0000 7f8031ffb640 1 -- 172.21.15.77:0/4004828520 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f802c00bc20 con 0x7f803c13dc10 2024-03-06T07:30:08.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.607+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c04f330 con 0x7f803c13dc10 2024-03-06T07:30:08.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.607+0000 7f8031ffb640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f802006b4d0 0x7f802006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.607+0000 7f8031ffb640 1 -- 172.21.15.77:0/4004828520 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f802c08a650 con 0x7f803c13dc10 2024-03-06T07:30:08.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.607+0000 7f804131d640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f802006b4d0 0x7f802006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.607+0000 7f804131d640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f802006b4d0 0x7f802006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f803c1e8d00 tx=0x7f8034009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.611+0000 7f8031ffb640 1 -- 172.21.15.77:0/4004828520 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f802c059560 con 0x7f803c13dc10 2024-03-06T07:30:08.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.791+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f803c0ccad0 con 0x7f803c13dc10 2024-03-06T07:30:08.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.792+0000 7f8031ffb640 1 -- 172.21.15.77:0/4004828520 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16667 (secure 0 0 0) 0x7f802c058e60 con 0x7f803c13dc10 2024-03-06T07:30:08.794 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:08.794 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":139,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:07.118183+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f802006b4d0 msgr2=0x7f802006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f802006b4d0 0x7f802006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f803c1e8d00 tx=0x7f8034009040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c13dc10 msgr2=0x7f803c1585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c13dc10 0x7f803c1585e0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f802c00d800 tx=0x7f802c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 shutdown_connections 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f802006b4d0 0x7f802006d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f803c13dc10 0x7f803c1585e0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f803c15bf30 0x7f803c158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 --2- 172.21.15.77:0/4004828520 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f803c0b3060 0x7f803c13c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 >> 172.21.15.77:0/4004828520 conn(0x7f803c0cf8a0 msgr2=0x7f803c0cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 shutdown_connections 2024-03-06T07:30:08.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.794+0000 7f8043581640 1 -- 172.21.15.77:0/4004828520 wait complete. 2024-03-06T07:30:08.807 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:30:08.807 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:08.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.906+0000 7ff6c64d4640 1 Processor -- start 2024-03-06T07:30:08.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.906+0000 7ff6c64d4640 1 -- start start 2024-03-06T07:30:08.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.907+0000 7ff6c64d4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c00d0a80 0x7ff6c00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.907+0000 7ff6c64d4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c00d25d0 con 0x7ff6c00d4760 2024-03-06T07:30:08.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.907+0000 7ff6c64d4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c00d2870 con 0x7ff6c00b4170 2024-03-06T07:30:08.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.907+0000 7ff6c64d4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00d4bf0 con 0x7ff6c00d0a80 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.907+0000 7ff6bf7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff6c00d4760 0x7ff6c00cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58950/0 (socket says 172.21.15.77:58950) 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.907+0000 7ff6bf7fe640 1 -- 172.21.15.77:0/2990780444 learned_addr learned my addr 172.21.15.77:0/2990780444 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6c4a71640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c00d0a80 0x7ff6c00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583172673 0 0) 0x7ff6c00d25d0 con 0x7ff6c00d4760 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a8003520 con 0x7ff6c00d4760 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924543272 0 0) 0x7ff6a8003520 con 0x7ff6c00d4760 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c00d0a80 msgr2=0x7ff6c00b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c00d0a80 0x7ff6c00b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 >> v1:172.21.15.158:6789/0 conn(0x7ff6c00b4170 legacy=0x7ff6c00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c00d2b50 con 0x7ff6c00d4760 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ff6b4001670 con 0x7ff6c00d4760 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 >> v1:172.21.15.77:6789/0 conn(0x7ff6c00d4760 legacy=0x7ff6c00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c00d0a80 0x7ff6a80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6a8003f60 0x7ff6a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8004d40 0x7ff6a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00d4bf0 con 0x7ff6a8003f60 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a800baa0 con 0x7ff6a8004d40 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a800bca0 con 0x7ff6c00d0a80 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6bf7fe640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6a8003f60 0x7ff6a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6bffff640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c00d0a80 0x7ff6a80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6c4a71640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8004d40 0x7ff6a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6bf7fe640 1 -- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c00d0a80 msgr2=0x7ff6a80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6bf7fe640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c00d0a80 0x7ff6a80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6bf7fe640 1 -- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8004d40 msgr2=0x7ff6a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6bf7fe640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8004d40 0x7ff6a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.908+0000 7ff6bf7fe640 1 -- 172.21.15.77:0/2990780444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a8003520 con 0x7ff6a8003f60 2024-03-06T07:30:08.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.909+0000 7ff6bf7fe640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6a8003f60 0x7ff6a8004800 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7ff6c006e080 tx=0x7ff6a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3f45ec95154cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4007d30 con 0x7ff6a8003f60 2024-03-06T07:30:08.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b4030070 con 0x7ff6a8003f60 2024-03-06T07:30:08.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6beffd640 1 -- 172.21.15.77:0/2990780444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4005d70 con 0x7ff6a8003f60 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6c64d4640 1 -- 172.21.15.77:0/2990780444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6a8003f60 msgr2=0x7ff6a8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6c64d4640 1 --2- 172.21.15.77:0/2990780444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6a8003f60 0x7ff6a8004800 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7ff6c006e080 tx=0x7ff6a8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6c4a71640 1 -- 172.21.15.77:0/2990780444 reap_dead start 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6c64d4640 1 -- 172.21.15.77:0/2990780444 shutdown_connections 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6c64d4640 1 -- 172.21.15.77:0/2990780444 >> 172.21.15.77:0/2990780444 conn(0x7ff6c00d3c40 msgr2=0x7ff6c00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6c64d4640 1 -- 172.21.15.77:0/2990780444 shutdown_connections 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.910+0000 7ff6c64d4640 1 -- 172.21.15.77:0/2990780444 wait complete. 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.911+0000 7ff6c64d4640 1 Processor -- start 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.912+0000 7ff6c64d4640 1 -- start start 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.912+0000 7ff6c64d4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8003f60 0x7ff6c00a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.912+0000 7ff6c64d4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c009d900 0x7ff6c0152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.912+0000 7ff6c64d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c009f300 0x7ff6c009f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.912+0000 7ff6c64d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00b4dd0 con 0x7ff6c009f300 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.912+0000 7ff6c64d4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00ce9e0 con 0x7ff6a8003f60 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.912+0000 7ff6c64d4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00b4f50 con 0x7ff6c009d900 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.912+0000 7ff6c4a71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c009f300 0x7ff6c009f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c4a71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c009f300 0x7ff6c009f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49998/0 (socket says 172.21.15.77:49998) 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c4a71640 1 -- 172.21.15.77:0/1533276011 learned_addr learned my addr 172.21.15.77:0/1533276011 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6bffff640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8003f60 0x7ff6c00a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6bf7fe640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c009d900 0x7ff6c0152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c4a71640 1 -- 172.21.15.77:0/1533276011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c009d900 msgr2=0x7ff6c0152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c4a71640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c009d900 0x7ff6c0152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c4a71640 1 -- 172.21.15.77:0/1533276011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8003f60 msgr2=0x7ff6c00a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c4a71640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8003f60 0x7ff6c00a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c4a71640 1 -- 172.21.15.77:0/1533276011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c009de00 con 0x7ff6c009f300 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c4a71640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c009f300 0x7ff6c009f7b0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00b67e0 tx=0x7ff6b80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6bcff9640 1 -- 172.21.15.77:0/1533276011 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8011e40 con 0x7ff6c009f300 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6bcff9640 1 -- 172.21.15.77:0/1533276011 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b801b070 con 0x7ff6c009f300 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c009e0e0 con 0x7ff6c009f300 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6bcff9640 1 -- 172.21.15.77:0/1533276011 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8003990 con 0x7ff6c009f300 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.913+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c009e6a0 con 0x7ff6c009f300 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.914+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c004f330 con 0x7ff6c009f300 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.915+0000 7ff6bcff9640 1 -- 172.21.15.77:0/1533276011 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6b8005d10 con 0x7ff6c009f300 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.915+0000 7ff6bcff9640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff6ac06b500 0x7ff6ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.915+0000 7ff6bcff9640 1 -- 172.21.15.77:0/1533276011 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6b808a5c0 con 0x7ff6c009f300 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.915+0000 7ff6bffff640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff6ac06b500 0x7ff6ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.916+0000 7ff6bffff640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff6ac06b500 0x7ff6ac06d9c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00016d0 tx=0x7ff6b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:08.919+0000 7ff6bcff9640 1 -- 172.21.15.77:0/1533276011 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6b8025020 con 0x7ff6c009f300 2024-03-06T07:30:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.092+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6c00a18e0 con 0x7ff6ac06b500 2024-03-06T07:30:09.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.098+0000 7ff6bcff9640 1 -- 172.21.15.77:0/1533276011 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252472 (secure 0 0 0) 0x7ff6b0001cb0 con 0x7ff6ac06b500 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.101+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff6ac06b500 msgr2=0x7ff6ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.101+0000 7ff6c64d4640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff6ac06b500 0x7ff6ac06d9c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00016d0 tx=0x7ff6b0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.101+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c009f300 msgr2=0x7ff6c009f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.101+0000 7ff6c64d4640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c009f300 0x7ff6c009f7b0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00b67e0 tx=0x7ff6b80115a0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.102+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 shutdown_connections 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.102+0000 7ff6c64d4640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff6ac06b500 0x7ff6ac06d9c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.102+0000 7ff6c64d4640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6c009f300 0x7ff6c009f7b0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.102+0000 7ff6c64d4640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff6c009d900 0x7ff6c0152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.102+0000 7ff6c64d4640 1 --2- 172.21.15.77:0/1533276011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff6a8003f60 0x7ff6c00a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.102+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 >> 172.21.15.77:0/1533276011 conn(0x7ff6c00d3c40 msgr2=0x7ff6c00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.102+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 shutdown_connections 2024-03-06T07:30:09.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.102+0000 7ff6c64d4640 1 -- 172.21.15.77:0/1533276011 wait complete. 2024-03-06T07:30:09.114 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-03-06T07:30:07.474645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310004,"kb_used_data":14764,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664716,"statfs":{"total":773094113280,"available":772776669184,"internally_reserved":0,"allocated":15118336,"data_stored":7250944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000526"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.127066+0000","last_change":"2024-03-06T07:30:06.125950+0000","last_active":"2024-03-06T07:30:07.127066+0000","last_peered":"2024-03-06T07:30:07.127066+0000","last_clean":"2024-03-06T07:30:07.127066+0000","last_became_active":"2024-03-06T07:30:06.125684+0000","last_became_peered":"2024-03-06T07:30:06.125684+0000","last_unstale":"2024-03-06T07:30:07.127066+0000","last_undegraded":"2024-03-06T07:30:07.127066+0000","last_fullsized":"2024-03-06T07:30:07.127066+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:20.177018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124064+0000","last_change":"2024-03-06T07:30:06.124613+0000","last_active":"2024-03-06T07:30:07.124064+0000","last_peered":"2024-03-06T07:30:07.124064+0000","last_clean":"2024-03-06T07:30:07.124064+0000","last_became_active":"2024-03-06T07:30:06.124337+0000","last_became_peered":"2024-03-06T07:30:06.124337+0000","last_unstale":"2024-03-06T07:30:07.124064+0000","last_undegraded":"2024-03-06T07:30:07.124064+0000","last_fullsized":"2024-03-06T07:30:07.124064+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:59.643647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125231+0000","last_change":"2024-03-06T07:30:06.126058+0000","last_active":"2024-03-06T07:30:07.125231+0000","last_peered":"2024-03-06T07:30:07.125231+0000","last_clean":"2024-03-06T07:30:07.125231+0000","last_became_active":"2024-03-06T07:30:06.125755+0000","last_became_peered":"2024-03-06T07:30:06.125755+0000","last_unstale":"2024-03-06T07:30:07.125231+0000","last_undegraded":"2024-03-06T07:30:07.125231+0000","last_fullsized":"2024-03-06T07:30:07.125231+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:41.063476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.130821+0000","last_change":"2024-03-06T07:30:06.127689+0000","last_active":"2024-03-06T07:30:06.130821+0000","last_peered":"2024-03-06T07:30:06.130821+0000","last_clean":"2024-03-06T07:30:06.130821+0000","last_became_active":"2024-03-06T07:30:06.127331+0000","last_became_peered":"2024-03-06T07:30:06.127331+0000","last_unstale":"2024-03-06T07:30:06.130821+0000","last_undegraded":"2024-03-06T07:30:06.130821+0000","last_fullsized":"2024-03-06T07:30:06.130821+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:15.612455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125201+0000","last_change":"2024-03-06T07:30:06.134261+0000","last_active":"2024-03-06T07:30:07.125201+0000","last_peered":"2024-03-06T07:30:07.125201+0000","last_clean":"2024-03-06T07:30:07.125201+0000","last_became_active":"2024-03-06T07:30:06.133916+0000","last_became_peered":"2024-03-06T07:30:06.133916+0000","last_unstale":"2024-03-06T07:30:07.125201+0000","last_undegraded":"2024-03-06T07:30:07.125201+0000","last_fullsized":"2024-03-06T07:30:07.125201+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:20.407586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126686+0000","last_change":"2024-03-06T07:30:06.119354+0000","last_active":"2024-03-06T07:30:07.126686+0000","last_peered":"2024-03-06T07:30:07.126686+0000","last_clean":"2024-03-06T07:30:07.126686+0000","last_became_active":"2024-03-06T07:30:06.119093+0000","last_became_peered":"2024-03-06T07:30:06.119093+0000","last_unstale":"2024-03-06T07:30:07.126686+0000","last_undegraded":"2024-03-06T07:30:07.126686+0000","last_fullsized":"2024-03-06T07:30:07.126686+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:14.376653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.141127+0000","last_change":"2024-03-06T07:30:06.135500+0000","last_active":"2024-03-06T07:30:06.141127+0000","last_peered":"2024-03-06T07:30:06.141127+0000","last_clean":"2024-03-06T07:30:06.141127+0000","last_became_active":"2024-03-06T07:30:06.135238+0000","last_became_peered":"2024-03-06T07:30:06.135238+0000","last_unstale":"2024-03-06T07:30:06.141127+0000","last_undegraded":"2024-03-06T07:30:06.141127+0000","last_fullsized":"2024-03-06T07:30:06.141127+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:05.729452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126488+0000","last_change":"2024-03-06T07:30:06.129984+0000","last_active":"2024-03-06T07:30:07.126488+0000","last_peered":"2024-03-06T07:30:07.126488+0000","last_clean":"2024-03-06T07:30:07.126488+0000","last_became_active":"2024-03-06T07:30:06.129694+0000","last_became_peered":"2024-03-06T07:30:06.129694+0000","last_unstale":"2024-03-06T07:30:07.126488+0000","last_undegraded":"2024-03-06T07:30:07.126488+0000","last_fullsized":"2024-03-06T07:30:07.126488+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:57.220320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123899+0000","last_change":"2024-03-06T07:30:06.129465+0000","last_active":"2024-03-06T07:30:07.123899+0000","last_peered":"2024-03-06T07:30:07.123899+0000","last_clean":"2024-03-06T07:30:07.123899+0000","last_became_active":"2024-03-06T07:30:06.129195+0000","last_became_peered":"2024-03-06T07:30:06.129195+0000","last_unstale":"2024-03-06T07:30:07.123899+0000","last_undegraded":"2024-03-06T07:30:07.123899+0000","last_fullsized":"2024-03-06T07:30:07.123899+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.557220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129375+0000","last_change":"2024-03-06T07:30:06.137708+0000","last_active":"2024-03-06T07:30:07.129375+0000","last_peered":"2024-03-06T07:30:07.129375+0000","last_clean":"2024-03-06T07:30:07.129375+0000","last_became_active":"2024-03-06T07:30:06.137409+0000","last_became_peered":"2024-03-06T07:30:06.137409+0000","last_unstale":"2024-03-06T07:30:07.129375+0000","last_undegraded":"2024-03-06T07:30:07.129375+0000","last_fullsized":"2024-03-06T07:30:07.129375+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:15.803035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126754+0000","last_change":"2024-03-06T07:30:06.131065+0000","last_active":"2024-03-06T07:30:07.126754+0000","last_peered":"2024-03-06T07:30:07.126754+0000","last_clean":"2024-03-06T07:30:07.126754+0000","last_became_active":"2024-03-06T07:30:06.130664+0000","last_became_peered":"2024-03-06T07:30:06.130664+0000","last_unstale":"2024-03-06T07:30:07.126754+0000","last_undegraded":"2024-03-06T07:30:07.126754+0000","last_fullsized":"2024-03-06T07:30:07.126754+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:42.732177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129261+0000","last_change":"2024-03-06T07:30:06.130555+0000","last_active":"2024-03-06T07:30:07.129261+0000","last_peered":"2024-03-06T07:30:07.129261+0000","last_clean":"2024-03-06T07:30:07.129261+0000","last_became_active":"2024-03-06T07:30:06.130236+0000","last_became_peered":"2024-03-06T07:30:06.130236+0000","last_unstale":"2024-03-06T07:30:07.129261+0000","last_undegraded":"2024-03-06T07:30:07.129261+0000","last_fullsized":"2024-03-06T07:30:07.129261+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:40.071499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125213+0000","last_change":"2024-03-06T07:30:06.127403+0000","last_active":"2024-03-06T07:30:07.125213+0000","last_peered":"2024-03-06T07:30:07.125213+0000","last_clean":"2024-03-06T07:30:07.125213+0000","last_became_active":"2024-03-06T07:30:06.127127+0000","last_became_peered":"2024-03-06T07:30:06.127127+0000","last_unstale":"2024-03-06T07:30:07.125213+0000","last_undegraded":"2024-03-06T07:30:07.125213+0000","last_fullsized":"2024-03-06T07:30:07.125213+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:54.244034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.136155+0000","last_change":"2024-03-06T07:30:06.135573+0000","last_active":"2024-03-06T07:30:06.136155+0000","last_peered":"2024-03-06T07:30:06.136155+0000","last_clean":"2024-03-06T07:30:06.136155+0000","last_became_active":"2024-03-06T07:30:06.135250+0000","last_became_peered":"2024-03-06T07:30:06.135250+0000","last_unstale":"2024-03-06T07:30:06.136155+0000","last_undegraded":"2024-03-06T07:30:06.136155+0000","last_fullsized":"2024-03-06T07:30:06.136155+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:07.128873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128948+0000","last_change":"2024-03-06T07:30:06.132936+0000","last_active":"2024-03-06T07:30:07.128948+0000","last_peered":"2024-03-06T07:30:07.128948+0000","last_clean":"2024-03-06T07:30:07.128948+0000","last_became_active":"2024-03-06T07:30:06.132642+0000","last_became_peered":"2024-03-06T07:30:06.132642+0000","last_unstale":"2024-03-06T07:30:07.128948+0000","last_undegraded":"2024-03-06T07:30:07.128948+0000","last_fullsized":"2024-03-06T07:30:07.128948+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:03.368830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330527+0000","last_change":"2024-03-06T07:30:06.130155+0000","last_active":"2024-03-06T07:30:07.330527+0000","last_peered":"2024-03-06T07:30:07.330527+0000","last_clean":"2024-03-06T07:30:07.330527+0000","last_became_active":"2024-03-06T07:30:06.129908+0000","last_became_peered":"2024-03-06T07:30:06.129908+0000","last_unstale":"2024-03-06T07:30:07.330527+0000","last_undegraded":"2024-03-06T07:30:07.330527+0000","last_fullsized":"2024-03-06T07:30:07.330527+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:41.592890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129639+0000","last_change":"2024-03-06T07:30:06.130128+0000","last_active":"2024-03-06T07:30:07.129639+0000","last_peered":"2024-03-06T07:30:07.129639+0000","last_clean":"2024-03-06T07:30:07.129639+0000","last_became_active":"2024-03-06T07:30:06.129781+0000","last_became_peered":"2024-03-06T07:30:06.129781+0000","last_unstale":"2024-03-06T07:30:07.129639+0000","last_undegraded":"2024-03-06T07:30:07.129639+0000","last_fullsized":"2024-03-06T07:30:07.129639+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:25.369784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125056+0000","last_change":"2024-03-06T07:30:06.126009+0000","last_active":"2024-03-06T07:30:07.125056+0000","last_peered":"2024-03-06T07:30:07.125056+0000","last_clean":"2024-03-06T07:30:07.125056+0000","last_became_active":"2024-03-06T07:30:06.125600+0000","last_became_peered":"2024-03-06T07:30:06.125600+0000","last_unstale":"2024-03-06T07:30:07.125056+0000","last_undegraded":"2024-03-06T07:30:07.125056+0000","last_fullsized":"2024-03-06T07:30:07.125056+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:42.459372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126995+0000","last_change":"2024-03-06T07:30:06.124576+0000","last_active":"2024-03-06T07:30:07.126995+0000","last_peered":"2024-03-06T07:30:07.126995+0000","last_clean":"2024-03-06T07:30:07.126995+0000","last_became_active":"2024-03-06T07:30:06.124320+0000","last_became_peered":"2024-03-06T07:30:06.124320+0000","last_unstale":"2024-03-06T07:30:07.126995+0000","last_undegraded":"2024-03-06T07:30:07.126995+0000","last_fullsized":"2024-03-06T07:30:07.126995+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:20.407586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330979+0000","last_change":"2024-03-06T07:30:06.127970+0000","last_active":"2024-03-06T07:30:07.330979+0000","last_peered":"2024-03-06T07:30:07.330979+0000","last_clean":"2024-03-06T07:30:07.330979+0000","last_became_active":"2024-03-06T07:30:06.127709+0000","last_became_peered":"2024-03-06T07:30:06.127709+0000","last_unstale":"2024-03-06T07:30:07.330979+0000","last_undegraded":"2024-03-06T07:30:07.330979+0000","last_fullsized":"2024-03-06T07:30:07.330979+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:02:09.627519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129285+0000","last_change":"2024-03-06T07:30:06.136462+0000","last_active":"2024-03-06T07:30:07.129285+0000","last_peered":"2024-03-06T07:30:07.129285+0000","last_clean":"2024-03-06T07:30:07.129285+0000","last_became_active":"2024-03-06T07:30:06.136202+0000","last_became_peered":"2024-03-06T07:30:06.136202+0000","last_unstale":"2024-03-06T07:30:07.129285+0000","last_undegraded":"2024-03-06T07:30:07.129285+0000","last_fullsized":"2024-03-06T07:30:07.129285+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:36.881643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.126492+0000","last_change":"2024-03-06T07:30:06.124805+0000","last_active":"2024-03-06T07:30:06.126492+0000","last_peered":"2024-03-06T07:30:06.126492+0000","last_clean":"2024-03-06T07:30:06.126492+0000","last_became_active":"2024-03-06T07:30:06.124467+0000","last_became_peered":"2024-03-06T07:30:06.124467+0000","last_unstale":"2024-03-06T07:30:06.126492+0000","last_undegraded":"2024-03-06T07:30:06.126492+0000","last_fullsized":"2024-03-06T07:30:06.126492+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:56.773729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.130786+0000","last_change":"2024-03-06T07:30:06.128235+0000","last_active":"2024-03-06T07:30:06.130786+0000","last_peered":"2024-03-06T07:30:06.130786+0000","last_clean":"2024-03-06T07:30:06.130786+0000","last_became_active":"2024-03-06T07:30:06.127878+0000","last_became_peered":"2024-03-06T07:30:06.127878+0000","last_unstale":"2024-03-06T07:30:06.130786+0000","last_undegraded":"2024-03-06T07:30:06.130786+0000","last_fullsized":"2024-03-06T07:30:06.130786+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:18.885685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128985+0000","last_change":"2024-03-06T07:30:06.136694+0000","last_active":"2024-03-06T07:30:07.128985+0000","last_peered":"2024-03-06T07:30:07.128985+0000","last_clean":"2024-03-06T07:30:07.128985+0000","last_became_active":"2024-03-06T07:30:06.136402+0000","last_became_peered":"2024-03-06T07:30:06.136402+0000","last_unstale":"2024-03-06T07:30:07.128985+0000","last_undegraded":"2024-03-06T07:30:07.128985+0000","last_fullsized":"2024-03-06T07:30:07.128985+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:38.062293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330908+0000","last_change":"2024-03-06T07:30:06.130843+0000","last_active":"2024-03-06T07:30:07.330908+0000","last_peered":"2024-03-06T07:30:07.330908+0000","last_clean":"2024-03-06T07:30:07.330908+0000","last_became_active":"2024-03-06T07:30:06.130604+0000","last_became_peered":"2024-03-06T07:30:06.130604+0000","last_unstale":"2024-03-06T07:30:07.330908+0000","last_undegraded":"2024-03-06T07:30:07.330908+0000","last_fullsized":"2024-03-06T07:30:07.330908+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:49.438538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126903+0000","last_change":"2024-03-06T07:30:06.120335+0000","last_active":"2024-03-06T07:30:07.126903+0000","last_peered":"2024-03-06T07:30:07.126903+0000","last_clean":"2024-03-06T07:30:07.126903+0000","last_became_active":"2024-03-06T07:30:06.119949+0000","last_became_peered":"2024-03-06T07:30:06.119949+0000","last_unstale":"2024-03-06T07:30:07.126903+0000","last_undegraded":"2024-03-06T07:30:07.126903+0000","last_fullsized":"2024-03-06T07:30:07.126903+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:40.918933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330868+0000","last_change":"2024-03-06T07:30:06.125274+0000","last_active":"2024-03-06T07:30:07.330868+0000","last_peered":"2024-03-06T07:30:07.330868+0000","last_clean":"2024-03-06T07:30:07.330868+0000","last_became_active":"2024-03-06T07:30:06.124979+0000","last_became_peered":"2024-03-06T07:30:06.124979+0000","last_unstale":"2024-03-06T07:30:07.330868+0000","last_undegraded":"2024-03-06T07:30:07.330868+0000","last_fullsized":"2024-03-06T07:30:07.330868+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:53.328688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.138540+0000","last_change":"2024-03-06T07:30:06.133140+0000","last_active":"2024-03-06T07:30:06.138540+0000","last_peered":"2024-03-06T07:30:06.138540+0000","last_clean":"2024-03-06T07:30:06.138540+0000","last_became_active":"2024-03-06T07:30:06.132495+0000","last_became_peered":"2024-03-06T07:30:06.132495+0000","last_unstale":"2024-03-06T07:30:06.138540+0000","last_undegraded":"2024-03-06T07:30:06.138540+0000","last_fullsized":"2024-03-06T07:30:06.138540+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.557220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126691+0000","last_change":"2024-03-06T07:30:06.122624+0000","last_active":"2024-03-06T07:30:07.126691+0000","last_peered":"2024-03-06T07:30:07.126691+0000","last_clean":"2024-03-06T07:30:07.126691+0000","last_became_active":"2024-03-06T07:30:06.122314+0000","last_became_peered":"2024-03-06T07:30:06.122314+0000","last_unstale":"2024-03-06T07:30:07.126691+0000","last_undegraded":"2024-03-06T07:30:07.126691+0000","last_fullsized":"2024-03-06T07:30:07.126691+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:42.260077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.124688+0000","last_change":"2024-03-06T07:30:06.123444+0000","last_active":"2024-03-06T07:30:06.124688+0000","last_peered":"2024-03-06T07:30:06.124688+0000","last_clean":"2024-03-06T07:30:06.124688+0000","last_became_active":"2024-03-06T07:30:06.123193+0000","last_became_peered":"2024-03-06T07:30:06.123193+0000","last_unstale":"2024-03-06T07:30:06.124688+0000","last_undegraded":"2024-03-06T07:30:06.124688+0000","last_fullsized":"2024-03-06T07:30:06.124688+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:29.206209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123829+0000","last_change":"2024-03-06T07:30:06.126541+0000","last_active":"2024-03-06T07:30:07.123829+0000","last_peered":"2024-03-06T07:30:07.123829+0000","last_clean":"2024-03-06T07:30:07.123829+0000","last_became_active":"2024-03-06T07:30:06.126282+0000","last_became_peered":"2024-03-06T07:30:06.126282+0000","last_unstale":"2024-03-06T07:30:07.123829+0000","last_undegraded":"2024-03-06T07:30:07.123829+0000","last_fullsized":"2024-03-06T07:30:07.123829+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:01.524141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330600+0000","last_change":"2024-03-06T07:30:06.135208+0000","last_active":"2024-03-06T07:30:07.330600+0000","last_peered":"2024-03-06T07:30:07.330600+0000","last_clean":"2024-03-06T07:30:07.330600+0000","last_became_active":"2024-03-06T07:30:06.134870+0000","last_became_peered":"2024-03-06T07:30:06.134870+0000","last_unstale":"2024-03-06T07:30:07.330600+0000","last_undegraded":"2024-03-06T07:30:07.330600+0000","last_fullsized":"2024-03-06T07:30:07.330600+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:54.764976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129383+0000","last_change":"2024-03-06T07:30:06.133373+0000","last_active":"2024-03-06T07:30:07.129383+0000","last_peered":"2024-03-06T07:30:07.129383+0000","last_clean":"2024-03-06T07:30:07.129383+0000","last_became_active":"2024-03-06T07:30:06.132526+0000","last_became_peered":"2024-03-06T07:30:06.132526+0000","last_unstale":"2024-03-06T07:30:07.129383+0000","last_undegraded":"2024-03-06T07:30:07.129383+0000","last_fullsized":"2024-03-06T07:30:07.129383+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:28.075065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"138'4","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129473+0000","last_change":"2024-03-06T07:30:05.107937+0000","last_active":"2024-03-06T07:30:07.129473+0000","last_peered":"2024-03-06T07:30:07.129473+0000","last_clean":"2024-03-06T07:30:07.129473+0000","last_became_active":"2024-03-06T07:30:05.107605+0000","last_became_peered":"2024-03-06T07:30:05.107605+0000","last_unstale":"2024-03-06T07:30:07.129473+0000","last_undegraded":"2024-03-06T07:30:07.129473+0000","last_fullsized":"2024-03-06T07:30:07.129473+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:09.201810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124055+0000","last_change":"2024-03-06T07:30:06.124514+0000","last_active":"2024-03-06T07:30:07.124055+0000","last_peered":"2024-03-06T07:30:07.124055+0000","last_clean":"2024-03-06T07:30:07.124055+0000","last_became_active":"2024-03-06T07:30:06.124207+0000","last_became_peered":"2024-03-06T07:30:06.124207+0000","last_unstale":"2024-03-06T07:30:07.124055+0000","last_undegraded":"2024-03-06T07:30:07.124055+0000","last_fullsized":"2024-03-06T07:30:07.124055+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:03.567494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124124+0000","last_change":"2024-03-06T07:30:05.102758+0000","last_active":"2024-03-06T07:30:07.124124+0000","last_peered":"2024-03-06T07:30:07.124124+0000","last_clean":"2024-03-06T07:30:07.124124+0000","last_became_active":"2024-03-06T07:30:05.102510+0000","last_became_peered":"2024-03-06T07:30:05.102510+0000","last_unstale":"2024-03-06T07:30:07.124124+0000","last_undegraded":"2024-03-06T07:30:07.124124+0000","last_fullsized":"2024-03-06T07:30:07.124124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:14.184362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126753+0000","last_change":"2024-03-06T07:30:06.120870+0000","last_active":"2024-03-06T07:30:07.126753+0000","last_peered":"2024-03-06T07:30:07.126753+0000","last_clean":"2024-03-06T07:30:07.126753+0000","last_became_active":"2024-03-06T07:30:06.120609+0000","last_became_peered":"2024-03-06T07:30:06.120609+0000","last_unstale":"2024-03-06T07:30:07.126753+0000","last_undegraded":"2024-03-06T07:30:07.126753+0000","last_fullsized":"2024-03-06T07:30:07.126753+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:45.680312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"138'2","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.142465+0000","last_change":"2024-03-06T07:30:05.104208+0000","last_active":"2024-03-06T07:30:06.142465+0000","last_peered":"2024-03-06T07:30:06.142465+0000","last_clean":"2024-03-06T07:30:06.142465+0000","last_became_active":"2024-03-06T07:30:05.103915+0000","last_became_peered":"2024-03-06T07:30:05.103915+0000","last_unstale":"2024-03-06T07:30:06.142465+0000","last_undegraded":"2024-03-06T07:30:06.142465+0000","last_fullsized":"2024-03-06T07:30:06.142465+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:32.335160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129187+0000","last_change":"2024-03-06T07:30:06.130795+0000","last_active":"2024-03-06T07:30:07.129187+0000","last_peered":"2024-03-06T07:30:07.129187+0000","last_clean":"2024-03-06T07:30:07.129187+0000","last_became_active":"2024-03-06T07:30:06.130506+0000","last_became_peered":"2024-03-06T07:30:06.130506+0000","last_unstale":"2024-03-06T07:30:07.129187+0000","last_undegraded":"2024-03-06T07:30:07.129187+0000","last_fullsized":"2024-03-06T07:30:07.129187+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:38.429183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"138'2","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.143104+0000","last_change":"2024-03-06T07:30:05.108043+0000","last_active":"2024-03-06T07:30:06.143104+0000","last_peered":"2024-03-06T07:30:06.143104+0000","last_clean":"2024-03-06T07:30:06.143104+0000","last_became_active":"2024-03-06T07:30:05.107777+0000","last_became_peered":"2024-03-06T07:30:05.107777+0000","last_unstale":"2024-03-06T07:30:06.143104+0000","last_undegraded":"2024-03-06T07:30:06.143104+0000","last_fullsized":"2024-03-06T07:30:06.143104+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:27.915657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123951+0000","last_change":"2024-03-06T07:30:06.126780+0000","last_active":"2024-03-06T07:30:07.123951+0000","last_peered":"2024-03-06T07:30:07.123951+0000","last_clean":"2024-03-06T07:30:07.123951+0000","last_became_active":"2024-03-06T07:30:06.126308+0000","last_became_peered":"2024-03-06T07:30:06.126308+0000","last_unstale":"2024-03-06T07:30:07.123951+0000","last_undegraded":"2024-03-06T07:30:07.123951+0000","last_fullsized":"2024-03-06T07:30:07.123951+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:05.729452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330581+0000","last_change":"2024-03-06T07:30:05.104149+0000","last_active":"2024-03-06T07:30:07.330581+0000","last_peered":"2024-03-06T07:30:07.330581+0000","last_clean":"2024-03-06T07:30:07.330581+0000","last_became_active":"2024-03-06T07:30:05.103815+0000","last_became_peered":"2024-03-06T07:30:05.103815+0000","last_unstale":"2024-03-06T07:30:07.330581+0000","last_undegraded":"2024-03-06T07:30:07.330581+0000","last_fullsized":"2024-03-06T07:30:07.330581+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:29.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.130207+0000","last_change":"2024-03-06T07:30:06.128531+0000","last_active":"2024-03-06T07:30:06.130207+0000","last_peered":"2024-03-06T07:30:06.130207+0000","last_clean":"2024-03-06T07:30:06.130207+0000","last_became_active":"2024-03-06T07:30:06.127270+0000","last_became_peered":"2024-03-06T07:30:06.127270+0000","last_unstale":"2024-03-06T07:30:06.130207+0000","last_undegraded":"2024-03-06T07:30:06.130207+0000","last_fullsized":"2024-03-06T07:30:06.130207+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:20.154495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124613+0000","last_change":"2024-03-06T07:30:05.102755+0000","last_active":"2024-03-06T07:30:07.124613+0000","last_peered":"2024-03-06T07:30:07.124613+0000","last_clean":"2024-03-06T07:30:07.124613+0000","last_became_active":"2024-03-06T07:30:05.102437+0000","last_became_peered":"2024-03-06T07:30:05.102437+0000","last_unstale":"2024-03-06T07:30:07.124613+0000","last_undegraded":"2024-03-06T07:30:07.124613+0000","last_fullsized":"2024-03-06T07:30:07.124613+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:35.097157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123749+0000","last_change":"2024-03-06T07:30:06.127275+0000","last_active":"2024-03-06T07:30:07.123749+0000","last_peered":"2024-03-06T07:30:07.123749+0000","last_clean":"2024-03-06T07:30:07.123749+0000","last_became_active":"2024-03-06T07:30:06.126863+0000","last_became_peered":"2024-03-06T07:30:06.126863+0000","last_unstale":"2024-03-06T07:30:07.123749+0000","last_undegraded":"2024-03-06T07:30:07.123749+0000","last_fullsized":"2024-03-06T07:30:07.123749+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:25.729483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330629+0000","last_change":"2024-03-06T07:30:05.104815+0000","last_active":"2024-03-06T07:30:07.330629+0000","last_peered":"2024-03-06T07:30:07.330629+0000","last_clean":"2024-03-06T07:30:07.330629+0000","last_became_active":"2024-03-06T07:30:05.104574+0000","last_became_peered":"2024-03-06T07:30:05.104574+0000","last_unstale":"2024-03-06T07:30:07.330629+0000","last_undegraded":"2024-03-06T07:30:07.330629+0000","last_fullsized":"2024-03-06T07:30:07.330629+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:04.096806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124836+0000","last_change":"2024-03-06T07:30:06.131426+0000","last_active":"2024-03-06T07:30:07.124836+0000","last_peered":"2024-03-06T07:30:07.124836+0000","last_clean":"2024-03-06T07:30:07.124836+0000","last_became_active":"2024-03-06T07:30:06.131142+0000","last_became_peered":"2024-03-06T07:30:06.131142+0000","last_unstale":"2024-03-06T07:30:07.124836+0000","last_undegraded":"2024-03-06T07:30:07.124836+0000","last_fullsized":"2024-03-06T07:30:07.124836+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:45.680312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.113187+0000","last_change":"2024-03-06T07:30:05.104244+0000","last_active":"2024-03-06T07:30:06.113187+0000","last_peered":"2024-03-06T07:30:06.113187+0000","last_clean":"2024-03-06T07:30:06.113187+0000","last_became_active":"2024-03-06T07:30:05.103960+0000","last_became_peered":"2024-03-06T07:30:05.103960+0000","last_unstale":"2024-03-06T07:30:06.113187+0000","last_undegraded":"2024-03-06T07:30:06.113187+0000","last_fullsized":"2024-03-06T07:30:06.113187+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:19.172375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124731+0000","last_change":"2024-03-06T07:30:06.131995+0000","last_active":"2024-03-06T07:30:07.124731+0000","last_peered":"2024-03-06T07:30:07.124731+0000","last_clean":"2024-03-06T07:30:07.124731+0000","last_became_active":"2024-03-06T07:30:06.131696+0000","last_became_peered":"2024-03-06T07:30:06.131696+0000","last_unstale":"2024-03-06T07:30:07.124731+0000","last_undegraded":"2024-03-06T07:30:07.124731+0000","last_fullsized":"2024-03-06T07:30:07.124731+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:42.260077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124167+0000","last_change":"2024-03-06T07:30:05.102786+0000","last_active":"2024-03-06T07:30:07.124167+0000","last_peered":"2024-03-06T07:30:07.124167+0000","last_clean":"2024-03-06T07:30:07.124167+0000","last_became_active":"2024-03-06T07:30:05.101112+0000","last_became_peered":"2024-03-06T07:30:05.101112+0000","last_unstale":"2024-03-06T07:30:07.124167+0000","last_undegraded":"2024-03-06T07:30:07.124167+0000","last_fullsized":"2024-03-06T07:30:07.124167+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:17.233410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129519+0000","last_change":"2024-03-06T07:30:06.125340+0000","last_active":"2024-03-06T07:30:07.129519+0000","last_peered":"2024-03-06T07:30:07.129519+0000","last_clean":"2024-03-06T07:30:07.129519+0000","last_became_active":"2024-03-06T07:30:06.124975+0000","last_became_peered":"2024-03-06T07:30:06.124975+0000","last_unstale":"2024-03-06T07:30:07.129519+0000","last_undegraded":"2024-03-06T07:30:07.129519+0000","last_fullsized":"2024-03-06T07:30:07.129519+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:12.859599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129507+0000","last_change":"2024-03-06T07:30:05.104388+0000","last_active":"2024-03-06T07:30:07.129507+0000","last_peered":"2024-03-06T07:30:07.129507+0000","last_clean":"2024-03-06T07:30:07.129507+0000","last_became_active":"2024-03-06T07:30:05.104101+0000","last_became_peered":"2024-03-06T07:30:05.104101+0000","last_unstale":"2024-03-06T07:30:07.129507+0000","last_undegraded":"2024-03-06T07:30:07.129507+0000","last_fullsized":"2024-03-06T07:30:07.129507+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:22.598603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330953+0000","last_change":"2024-03-06T07:30:06.126112+0000","last_active":"2024-03-06T07:30:07.330953+0000","last_peered":"2024-03-06T07:30:07.330953+0000","last_clean":"2024-03-06T07:30:07.330953+0000","last_became_active":"2024-03-06T07:30:06.125851+0000","last_became_peered":"2024-03-06T07:30:06.125851+0000","last_unstale":"2024-03-06T07:30:07.330953+0000","last_undegraded":"2024-03-06T07:30:07.330953+0000","last_fullsized":"2024-03-06T07:30:07.330953+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:02.843483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"138'6","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.143498+0000","last_change":"2024-03-06T07:30:05.105115+0000","last_active":"2024-03-06T07:30:06.143498+0000","last_peered":"2024-03-06T07:30:06.143498+0000","last_clean":"2024-03-06T07:30:06.143498+0000","last_became_active":"2024-03-06T07:30:05.104838+0000","last_became_peered":"2024-03-06T07:30:05.104838+0000","last_unstale":"2024-03-06T07:30:06.143498+0000","last_undegraded":"2024-03-06T07:30:06.143498+0000","last_fullsized":"2024-03-06T07:30:06.143498+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:24.724840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124983+0000","last_change":"2024-03-06T07:30:06.127030+0000","last_active":"2024-03-06T07:30:07.124983+0000","last_peered":"2024-03-06T07:30:07.124983+0000","last_clean":"2024-03-06T07:30:07.124983+0000","last_became_active":"2024-03-06T07:30:06.126731+0000","last_became_peered":"2024-03-06T07:30:06.126731+0000","last_unstale":"2024-03-06T07:30:07.124983+0000","last_undegraded":"2024-03-06T07:30:07.124983+0000","last_fullsized":"2024-03-06T07:30:07.124983+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:03.904829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125030+0000","last_change":"2024-03-06T07:30:05.105420+0000","last_active":"2024-03-06T07:30:07.125030+0000","last_peered":"2024-03-06T07:30:07.125030+0000","last_clean":"2024-03-06T07:30:07.125030+0000","last_became_active":"2024-03-06T07:30:05.105100+0000","last_became_peered":"2024-03-06T07:30:05.105100+0000","last_unstale":"2024-03-06T07:30:07.125030+0000","last_undegraded":"2024-03-06T07:30:07.125030+0000","last_fullsized":"2024-03-06T07:30:07.125030+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:19.149852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123822+0000","last_change":"2024-03-06T07:30:06.124524+0000","last_active":"2024-03-06T07:30:07.123822+0000","last_peered":"2024-03-06T07:30:07.123822+0000","last_clean":"2024-03-06T07:30:07.123822+0000","last_became_active":"2024-03-06T07:30:06.124273+0000","last_became_peered":"2024-03-06T07:30:06.124273+0000","last_unstale":"2024-03-06T07:30:07.123822+0000","last_undegraded":"2024-03-06T07:30:07.123822+0000","last_fullsized":"2024-03-06T07:30:07.123822+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:50.660477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125080+0000","last_change":"2024-03-06T07:30:05.097944+0000","last_active":"2024-03-06T07:30:07.125080+0000","last_peered":"2024-03-06T07:30:07.125080+0000","last_clean":"2024-03-06T07:30:07.125080+0000","last_became_active":"2024-03-06T07:30:05.096406+0000","last_became_peered":"2024-03-06T07:30:05.096406+0000","last_unstale":"2024-03-06T07:30:07.125080+0000","last_undegraded":"2024-03-06T07:30:07.125080+0000","last_fullsized":"2024-03-06T07:30:07.125080+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:14.607812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124953+0000","last_change":"2024-03-06T07:30:06.133593+0000","last_active":"2024-03-06T07:30:07.124953+0000","last_peered":"2024-03-06T07:30:07.124953+0000","last_clean":"2024-03-06T07:30:07.124953+0000","last_became_active":"2024-03-06T07:30:06.133139+0000","last_became_peered":"2024-03-06T07:30:06.133139+0000","last_unstale":"2024-03-06T07:30:07.124953+0000","last_undegraded":"2024-03-06T07:30:07.124953+0000","last_fullsized":"2024-03-06T07:30:07.124953+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:07.895365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126434+0000","last_change":"2024-03-06T07:30:05.106125+0000","last_active":"2024-03-06T07:30:07.126434+0000","last_peered":"2024-03-06T07:30:07.126434+0000","last_clean":"2024-03-06T07:30:07.126434+0000","last_became_active":"2024-03-06T07:30:05.105865+0000","last_became_peered":"2024-03-06T07:30:05.105865+0000","last_unstale":"2024-03-06T07:30:07.126434+0000","last_undegraded":"2024-03-06T07:30:07.126434+0000","last_fullsized":"2024-03-06T07:30:07.126434+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:53.760333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124371+0000","last_change":"2024-03-06T07:30:06.123068+0000","last_active":"2024-03-06T07:30:07.124371+0000","last_peered":"2024-03-06T07:30:07.124371+0000","last_clean":"2024-03-06T07:30:07.124371+0000","last_became_active":"2024-03-06T07:30:06.122786+0000","last_became_peered":"2024-03-06T07:30:06.122786+0000","last_unstale":"2024-03-06T07:30:07.124371+0000","last_undegraded":"2024-03-06T07:30:07.124371+0000","last_fullsized":"2024-03-06T07:30:07.124371+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:40.145873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"138'8","reported_seq":22,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330648+0000","last_change":"2024-03-06T07:30:05.098064+0000","last_active":"2024-03-06T07:30:07.330648+0000","last_peered":"2024-03-06T07:30:07.330648+0000","last_clean":"2024-03-06T07:30:07.330648+0000","last_became_active":"2024-03-06T07:30:05.097776+0000","last_became_peered":"2024-03-06T07:30:05.097776+0000","last_unstale":"2024-03-06T07:30:07.330648+0000","last_undegraded":"2024-03-06T07:30:07.330648+0000","last_fullsized":"2024-03-06T07:30:07.330648+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:52.540868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129517+0000","last_change":"2024-03-06T07:30:06.131061+0000","last_active":"2024-03-06T07:30:07.129517+0000","last_peered":"2024-03-06T07:30:07.129517+0000","last_clean":"2024-03-06T07:30:07.129517+0000","last_became_active":"2024-03-06T07:30:06.130457+0000","last_became_peered":"2024-03-06T07:30:06.130457+0000","last_unstale":"2024-03-06T07:30:07.129517+0000","last_undegraded":"2024-03-06T07:30:07.129517+0000","last_fullsized":"2024-03-06T07:30:07.129517+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:45.311727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125029+0000","last_change":"2024-03-06T07:30:05.097957+0000","last_active":"2024-03-06T07:30:07.125029+0000","last_peered":"2024-03-06T07:30:07.125029+0000","last_clean":"2024-03-06T07:30:07.125029+0000","last_became_active":"2024-03-06T07:30:05.097674+0000","last_became_peered":"2024-03-06T07:30:05.097674+0000","last_unstale":"2024-03-06T07:30:07.125029+0000","last_undegraded":"2024-03-06T07:30:07.125029+0000","last_fullsized":"2024-03-06T07:30:07.125029+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:48.585329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330946+0000","last_change":"2024-03-06T07:30:06.126331+0000","last_active":"2024-03-06T07:30:07.330946+0000","last_peered":"2024-03-06T07:30:07.330946+0000","last_clean":"2024-03-06T07:30:07.330946+0000","last_became_active":"2024-03-06T07:30:06.126070+0000","last_became_peered":"2024-03-06T07:30:06.126070+0000","last_unstale":"2024-03-06T07:30:07.330946+0000","last_undegraded":"2024-03-06T07:30:07.330946+0000","last_fullsized":"2024-03-06T07:30:07.330946+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:43.930719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124214+0000","last_change":"2024-03-06T07:30:05.100969+0000","last_active":"2024-03-06T07:30:07.124214+0000","last_peered":"2024-03-06T07:30:07.124214+0000","last_clean":"2024-03-06T07:30:07.124214+0000","last_became_active":"2024-03-06T07:30:05.100665+0000","last_became_peered":"2024-03-06T07:30:05.100665+0000","last_unstale":"2024-03-06T07:30:07.124214+0000","last_undegraded":"2024-03-06T07:30:07.124214+0000","last_fullsized":"2024-03-06T07:30:07.124214+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:20.043276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124619+0000","last_change":"2024-03-06T07:30:06.127335+0000","last_active":"2024-03-06T07:30:07.124619+0000","last_peered":"2024-03-06T07:30:07.124619+0000","last_clean":"2024-03-06T07:30:07.124619+0000","last_became_active":"2024-03-06T07:30:06.127020+0000","last_became_peered":"2024-03-06T07:30:06.127020+0000","last_unstale":"2024-03-06T07:30:07.124619+0000","last_undegraded":"2024-03-06T07:30:07.124619+0000","last_fullsized":"2024-03-06T07:30:07.124619+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:26.221909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330965+0000","last_change":"2024-03-06T07:30:05.104984+0000","last_active":"2024-03-06T07:30:07.330965+0000","last_peered":"2024-03-06T07:30:07.330965+0000","last_clean":"2024-03-06T07:30:07.330965+0000","last_became_active":"2024-03-06T07:30:05.104528+0000","last_became_peered":"2024-03-06T07:30:05.104528+0000","last_unstale":"2024-03-06T07:30:07.330965+0000","last_undegraded":"2024-03-06T07:30:07.330965+0000","last_fullsized":"2024-03-06T07:30:07.330965+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:32.436401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126899+0000","last_change":"2024-03-06T07:30:06.122867+0000","last_active":"2024-03-06T07:30:07.126899+0000","last_peered":"2024-03-06T07:30:07.126899+0000","last_clean":"2024-03-06T07:30:07.126899+0000","last_became_active":"2024-03-06T07:30:06.122553+0000","last_became_peered":"2024-03-06T07:30:06.122553+0000","last_unstale":"2024-03-06T07:30:07.126899+0000","last_undegraded":"2024-03-06T07:30:07.126899+0000","last_fullsized":"2024-03-06T07:30:07.126899+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:07.895365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124230+0000","last_change":"2024-03-06T07:30:05.101492+0000","last_active":"2024-03-06T07:30:07.124230+0000","last_peered":"2024-03-06T07:30:07.124230+0000","last_clean":"2024-03-06T07:30:07.124230+0000","last_became_active":"2024-03-06T07:30:05.101232+0000","last_became_peered":"2024-03-06T07:30:05.101232+0000","last_unstale":"2024-03-06T07:30:07.124230+0000","last_undegraded":"2024-03-06T07:30:07.124230+0000","last_fullsized":"2024-03-06T07:30:07.124230+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:39.022253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.122013+0000","last_change":"2024-03-06T07:30:06.121364+0000","last_active":"2024-03-06T07:30:06.122013+0000","last_peered":"2024-03-06T07:30:06.122013+0000","last_clean":"2024-03-06T07:30:06.122013+0000","last_became_active":"2024-03-06T07:30:06.120979+0000","last_became_peered":"2024-03-06T07:30:06.120979+0000","last_unstale":"2024-03-06T07:30:06.122013+0000","last_undegraded":"2024-03-06T07:30:06.122013+0000","last_fullsized":"2024-03-06T07:30:06.122013+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:39.105065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"138'2","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.138586+0000","last_change":"2024-03-06T07:30:05.106837+0000","last_active":"2024-03-06T07:30:06.138586+0000","last_peered":"2024-03-06T07:30:06.138586+0000","last_clean":"2024-03-06T07:30:06.138586+0000","last_became_active":"2024-03-06T07:30:05.106576+0000","last_became_peered":"2024-03-06T07:30:05.106576+0000","last_unstale":"2024-03-06T07:30:06.138586+0000","last_undegraded":"2024-03-06T07:30:06.138586+0000","last_fullsized":"2024-03-06T07:30:06.138586+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:49.655834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.141408+0000","last_change":"2024-03-06T07:30:06.136909+0000","last_active":"2024-03-06T07:30:06.141408+0000","last_peered":"2024-03-06T07:30:06.141408+0000","last_clean":"2024-03-06T07:30:06.141408+0000","last_became_active":"2024-03-06T07:30:06.136604+0000","last_became_peered":"2024-03-06T07:30:06.136604+0000","last_unstale":"2024-03-06T07:30:06.141408+0000","last_undegraded":"2024-03-06T07:30:06.141408+0000","last_fullsized":"2024-03-06T07:30:06.141408+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:59.643647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129017+0000","last_change":"2024-03-06T07:30:05.106795+0000","last_active":"2024-03-06T07:30:07.129017+0000","last_peered":"2024-03-06T07:30:07.129017+0000","last_clean":"2024-03-06T07:30:07.129017+0000","last_became_active":"2024-03-06T07:30:05.106482+0000","last_became_peered":"2024-03-06T07:30:05.106482+0000","last_unstale":"2024-03-06T07:30:07.129017+0000","last_undegraded":"2024-03-06T07:30:07.129017+0000","last_fullsized":"2024-03-06T07:30:07.129017+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:50.017775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.130846+0000","last_change":"2024-03-06T07:30:06.128473+0000","last_active":"2024-03-06T07:30:06.130846+0000","last_peered":"2024-03-06T07:30:06.130846+0000","last_clean":"2024-03-06T07:30:06.130846+0000","last_became_active":"2024-03-06T07:30:06.127956+0000","last_became_peered":"2024-03-06T07:30:06.127956+0000","last_unstale":"2024-03-06T07:30:06.130846+0000","last_undegraded":"2024-03-06T07:30:06.130846+0000","last_fullsized":"2024-03-06T07:30:06.130846+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:22.913256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128879+0000","last_change":"2024-03-06T07:30:05.105611+0000","last_active":"2024-03-06T07:30:07.128879+0000","last_peered":"2024-03-06T07:30:07.128879+0000","last_clean":"2024-03-06T07:30:07.128879+0000","last_became_active":"2024-03-06T07:30:05.105311+0000","last_became_peered":"2024-03-06T07:30:05.105311+0000","last_unstale":"2024-03-06T07:30:07.128879+0000","last_undegraded":"2024-03-06T07:30:07.128879+0000","last_fullsized":"2024-03-06T07:30:07.128879+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:20.043276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.141150+0000","last_change":"2024-03-06T07:30:06.136843+0000","last_active":"2024-03-06T07:30:06.141150+0000","last_peered":"2024-03-06T07:30:06.141150+0000","last_clean":"2024-03-06T07:30:06.141150+0000","last_became_active":"2024-03-06T07:30:06.136566+0000","last_became_peered":"2024-03-06T07:30:06.136566+0000","last_unstale":"2024-03-06T07:30:06.141150+0000","last_undegraded":"2024-03-06T07:30:06.141150+0000","last_fullsized":"2024-03-06T07:30:06.141150+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:45.737995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124197+0000","last_change":"2024-03-06T07:30:05.106739+0000","last_active":"2024-03-06T07:30:07.124197+0000","last_peered":"2024-03-06T07:30:07.124197+0000","last_clean":"2024-03-06T07:30:07.124197+0000","last_became_active":"2024-03-06T07:30:05.106368+0000","last_became_peered":"2024-03-06T07:30:05.106368+0000","last_unstale":"2024-03-06T07:30:07.124197+0000","last_undegraded":"2024-03-06T07:30:07.124197+0000","last_fullsized":"2024-03-06T07:30:07.124197+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:17.881042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124886+0000","last_change":"2024-03-06T07:30:06.133418+0000","last_active":"2024-03-06T07:30:07.124886+0000","last_peered":"2024-03-06T07:30:07.124886+0000","last_clean":"2024-03-06T07:30:07.124886+0000","last_became_active":"2024-03-06T07:30:06.133056+0000","last_became_peered":"2024-03-06T07:30:06.133056+0000","last_unstale":"2024-03-06T07:30:07.124886+0000","last_undegraded":"2024-03-06T07:30:07.124886+0000","last_fullsized":"2024-03-06T07:30:07.124886+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:42.732177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129614+0000","last_change":"2024-03-06T07:30:05.106211+0000","last_active":"2024-03-06T07:30:07.129614+0000","last_peered":"2024-03-06T07:30:07.129614+0000","last_clean":"2024-03-06T07:30:07.129614+0000","last_became_active":"2024-03-06T07:30:05.105876+0000","last_became_peered":"2024-03-06T07:30:05.105876+0000","last_unstale":"2024-03-06T07:30:07.129614+0000","last_undegraded":"2024-03-06T07:30:07.129614+0000","last_fullsized":"2024-03-06T07:30:07.129614+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:05.109313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125170+0000","last_change":"2024-03-06T07:30:06.128314+0000","last_active":"2024-03-06T07:30:07.125170+0000","last_peered":"2024-03-06T07:30:07.125170+0000","last_clean":"2024-03-06T07:30:07.125170+0000","last_became_active":"2024-03-06T07:30:06.128052+0000","last_became_peered":"2024-03-06T07:30:06.128052+0000","last_unstale":"2024-03-06T07:30:07.125170+0000","last_undegraded":"2024-03-06T07:30:07.125170+0000","last_fullsized":"2024-03-06T07:30:07.125170+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:53.089086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"139'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128208+0000","last_change":"2024-03-06T07:30:05.103786+0000","last_active":"2024-03-06T07:30:07.128208+0000","last_peered":"2024-03-06T07:30:07.128208+0000","last_clean":"2024-03-06T07:30:07.128208+0000","last_became_active":"2024-03-06T07:30:05.103532+0000","last_became_peered":"2024-03-06T07:30:05.103532+0000","last_unstale":"2024-03-06T07:30:07.128208+0000","last_undegraded":"2024-03-06T07:30:07.128208+0000","last_fullsized":"2024-03-06T07:30:07.128208+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:02:08.622876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124066+0000","last_change":"2024-03-06T07:30:06.127398+0000","last_active":"2024-03-06T07:30:07.124066+0000","last_peered":"2024-03-06T07:30:07.124066+0000","last_clean":"2024-03-06T07:30:07.124066+0000","last_became_active":"2024-03-06T07:30:06.127086+0000","last_became_peered":"2024-03-06T07:30:06.127086+0000","last_unstale":"2024-03-06T07:30:07.124066+0000","last_undegraded":"2024-03-06T07:30:07.124066+0000","last_fullsized":"2024-03-06T07:30:07.124066+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:45.737995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.127053+0000","last_change":"2024-03-06T07:30:05.106361+0000","last_active":"2024-03-06T07:30:07.127053+0000","last_peered":"2024-03-06T07:30:07.127053+0000","last_clean":"2024-03-06T07:30:07.127053+0000","last_became_active":"2024-03-06T07:30:05.106128+0000","last_became_peered":"2024-03-06T07:30:05.106128+0000","last_unstale":"2024-03-06T07:30:07.127053+0000","last_undegraded":"2024-03-06T07:30:07.127053+0000","last_fullsized":"2024-03-06T07:30:07.127053+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:52.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.127021+0000","last_change":"2024-03-06T07:30:06.127953+0000","last_active":"2024-03-06T07:30:07.127021+0000","last_peered":"2024-03-06T07:30:07.127021+0000","last_clean":"2024-03-06T07:30:07.127021+0000","last_became_active":"2024-03-06T07:30:06.127340+0000","last_became_peered":"2024-03-06T07:30:06.127340+0000","last_unstale":"2024-03-06T07:30:07.127021+0000","last_undegraded":"2024-03-06T07:30:07.127021+0000","last_fullsized":"2024-03-06T07:30:07.127021+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:54.244034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126736+0000","last_change":"2024-03-06T07:30:05.105781+0000","last_active":"2024-03-06T07:30:07.126736+0000","last_peered":"2024-03-06T07:30:07.126736+0000","last_clean":"2024-03-06T07:30:07.126736+0000","last_became_active":"2024-03-06T07:30:05.105526+0000","last_became_peered":"2024-03-06T07:30:05.105526+0000","last_unstale":"2024-03-06T07:30:07.126736+0000","last_undegraded":"2024-03-06T07:30:07.126736+0000","last_fullsized":"2024-03-06T07:30:07.126736+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:30.220384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330757+0000","last_change":"2024-03-06T07:30:06.127084+0000","last_active":"2024-03-06T07:30:07.330757+0000","last_peered":"2024-03-06T07:30:07.330757+0000","last_clean":"2024-03-06T07:30:07.330757+0000","last_became_active":"2024-03-06T07:30:06.126827+0000","last_became_peered":"2024-03-06T07:30:06.126827+0000","last_unstale":"2024-03-06T07:30:07.330757+0000","last_undegraded":"2024-03-06T07:30:07.330757+0000","last_fullsized":"2024-03-06T07:30:07.330757+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:31.225027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.120257+0000","last_change":"2024-03-06T07:30:05.104122+0000","last_active":"2024-03-06T07:30:06.120257+0000","last_peered":"2024-03-06T07:30:06.120257+0000","last_clean":"2024-03-06T07:30:06.120257+0000","last_became_active":"2024-03-06T07:30:05.103857+0000","last_became_peered":"2024-03-06T07:30:05.103857+0000","last_unstale":"2024-03-06T07:30:06.120257+0000","last_undegraded":"2024-03-06T07:30:06.120257+0000","last_fullsized":"2024-03-06T07:30:06.120257+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:00.519497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125350+0000","last_change":"2024-03-06T07:30:06.132705+0000","last_active":"2024-03-06T07:30:07.125350+0000","last_peered":"2024-03-06T07:30:07.125350+0000","last_clean":"2024-03-06T07:30:07.125350+0000","last_became_active":"2024-03-06T07:30:06.132416+0000","last_became_peered":"2024-03-06T07:30:06.132416+0000","last_unstale":"2024-03-06T07:30:07.125350+0000","last_undegraded":"2024-03-06T07:30:07.125350+0000","last_fullsized":"2024-03-06T07:30:07.125350+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:20.177018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129181+0000","last_change":"2024-03-06T07:30:05.103114+0000","last_active":"2024-03-06T07:30:07.129181+0000","last_peered":"2024-03-06T07:30:07.129181+0000","last_clean":"2024-03-06T07:30:07.129181+0000","last_became_active":"2024-03-06T07:30:05.102792+0000","last_became_peered":"2024-03-06T07:30:05.102792+0000","last_unstale":"2024-03-06T07:30:07.129181+0000","last_undegraded":"2024-03-06T07:30:07.129181+0000","last_fullsized":"2024-03-06T07:30:07.129181+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:54.344320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128981+0000","last_change":"2024-03-06T07:30:06.128633+0000","last_active":"2024-03-06T07:30:07.128981+0000","last_peered":"2024-03-06T07:30:07.128981+0000","last_clean":"2024-03-06T07:30:07.128981+0000","last_became_active":"2024-03-06T07:30:06.128348+0000","last_became_peered":"2024-03-06T07:30:06.128348+0000","last_unstale":"2024-03-06T07:30:07.128981+0000","last_undegraded":"2024-03-06T07:30:07.128981+0000","last_fullsized":"2024-03-06T07:30:07.128981+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:18.935110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124726+0000","last_change":"2024-03-06T07:30:05.104636+0000","last_active":"2024-03-06T07:30:07.124726+0000","last_peered":"2024-03-06T07:30:07.124726+0000","last_clean":"2024-03-06T07:30:07.124726+0000","last_became_active":"2024-03-06T07:30:05.103695+0000","last_became_peered":"2024-03-06T07:30:05.103695+0000","last_unstale":"2024-03-06T07:30:07.124726+0000","last_undegraded":"2024-03-06T07:30:07.124726+0000","last_fullsized":"2024-03-06T07:30:07.124726+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:55.769085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124990+0000","last_change":"2024-03-06T07:30:06.131015+0000","last_active":"2024-03-06T07:30:07.124990+0000","last_peered":"2024-03-06T07:30:07.124990+0000","last_clean":"2024-03-06T07:30:07.124990+0000","last_became_active":"2024-03-06T07:30:06.130706+0000","last_became_peered":"2024-03-06T07:30:06.130706+0000","last_unstale":"2024-03-06T07:30:07.124990+0000","last_undegraded":"2024-03-06T07:30:07.124990+0000","last_fullsized":"2024-03-06T07:30:07.124990+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:40.918933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123724+0000","last_change":"2024-03-06T07:30:05.103465+0000","last_active":"2024-03-06T07:30:07.123724+0000","last_peered":"2024-03-06T07:30:07.123724+0000","last_clean":"2024-03-06T07:30:07.123724+0000","last_became_active":"2024-03-06T07:30:05.103211+0000","last_became_peered":"2024-03-06T07:30:05.103211+0000","last_unstale":"2024-03-06T07:30:07.123724+0000","last_undegraded":"2024-03-06T07:30:07.123724+0000","last_fullsized":"2024-03-06T07:30:07.123724+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:01.375576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.141309+0000","last_change":"2024-03-06T07:30:06.136182+0000","last_active":"2024-03-06T07:30:06.141309+0000","last_peered":"2024-03-06T07:30:06.141309+0000","last_clean":"2024-03-06T07:30:06.141309+0000","last_became_active":"2024-03-06T07:30:06.135308+0000","last_became_peered":"2024-03-06T07:30:06.135308+0000","last_unstale":"2024-03-06T07:30:06.141309+0000","last_undegraded":"2024-03-06T07:30:06.141309+0000","last_fullsized":"2024-03-06T07:30:06.141309+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:03.567494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"138'3","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125340+0000","last_change":"2024-03-06T07:30:05.105883+0000","last_active":"2024-03-06T07:30:07.125340+0000","last_peered":"2024-03-06T07:30:07.125340+0000","last_clean":"2024-03-06T07:30:07.125340+0000","last_became_active":"2024-03-06T07:30:05.105598+0000","last_became_peered":"2024-03-06T07:30:05.105598+0000","last_unstale":"2024-03-06T07:30:07.125340+0000","last_undegraded":"2024-03-06T07:30:07.125340+0000","last_fullsized":"2024-03-06T07:30:07.125340+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:21.908613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":382,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129211+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:30:07.129211+0000","last_peered":"2024-03-06T07:30:07.129211+0000","last_clean":"2024-03-06T07:30:07.129211+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:30:07.129211+0000","last_undegraded":"2024-03-06T07:30:07.129211+0000","last_fullsized":"2024-03-06T07:30:07.129211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":2342912,"data_stored":1353131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28732,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343108,"statfs":{"total":96636764160,"available":96607342592,"internally_reserved":0,"allocated":1753088,"data_stored":762257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45364,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326476,"statfs":{"total":96636764160,"available":96590311424,"internally_reserved":0,"allocated":1744896,"data_stored":762133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574949,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28708,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343132,"statfs":{"total":96636764160,"available":96607367168,"internally_reserved":0,"allocated":1728512,"data_stored":755811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41200,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330640,"statfs":{"total":96636764160,"available":96594575360,"internally_reserved":0,"allocated":1732608,"data_stored":755901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41212,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330628,"statfs":{"total":96636764160,"available":96594563072,"internally_reserved":0,"allocated":1744896,"data_stored":757493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41200,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330640,"statfs":{"total":96636764160,"available":96594575360,"internally_reserved":0,"allocated":1732608,"data_stored":756393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41792,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330048,"statfs":{"total":96636764160,"available":96593969152,"internally_reserved":0,"allocated":2338816,"data_stored":1347825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:09.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:09.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.254+0000 7f7a51e65640 1 Processor -- start 2024-03-06T07:30:09.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.254+0000 7f7a51e65640 1 -- start start 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.254+0000 7f7a51e65640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a4c15dfe0 0x7f7a4c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.254+0000 7f7a51e65640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 0x7f7a4c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.254+0000 7f7a51e65640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c0b3430 con 0x7f7a4c0d27f0 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.254+0000 7f7a51e65640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4c0d2c50 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.254+0000 7f7a51e65640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c0b31e0 con 0x7f7a4c0cf830 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.255+0000 7f7a4b7fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f7a4c15dfe0 0x7f7a4c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.255+0000 7f7a43fff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 0x7f7a4c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.255+0000 7f7a4bfff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f7a4c0d27f0 0x7f7a4c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:39248/0 (socket says 172.21.15.77:39248) 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.255+0000 7f7a4bfff640 1 -- 172.21.15.77:0/3863051707 learned_addr learned my addr 172.21.15.77:0/3863051707 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.255+0000 7f7a4b7fe640 1 -- 172.21.15.77:0/3863051707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 msgr2=0x7f7a4c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.255+0000 7f7a4b7fe640 1 --2- 172.21.15.77:0/3863051707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 0x7f7a4c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.255+0000 7f7a4b7fe640 1 -- 172.21.15.77:0/3863051707 >> v1:172.21.15.158:6789/0 conn(0x7f7a4c0d27f0 legacy=0x7f7a4c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.255+0000 7f7a4b7fe640 1 -- 172.21.15.77:0/3863051707 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c0b3710 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.256+0000 7f7a4b7fe640 1 --2- 172.21.15.77:0/3863051707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0cf2f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0097a0 tx=0x7f7a3c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7fc234f54f6ff75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.256+0000 7f7a4affd640 1 -- 172.21.15.77:0/3863051707 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c035020 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.256+0000 7f7a4affd640 1 -- 172.21.15.77:0/3863051707 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c031830 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.256+0000 7f7a4affd640 1 -- 172.21.15.77:0/3863051707 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c031c00 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.257+0000 7f7a51e65640 1 -- 172.21.15.77:0/3863051707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c15dfe0 msgr2=0x7f7a4c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.257+0000 7f7a51e65640 1 --2- 172.21.15.77:0/3863051707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0cf2f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0097a0 tx=0x7f7a3c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.257+0000 7f7a51e65640 1 -- 172.21.15.77:0/3863051707 shutdown_connections 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.257+0000 7f7a51e65640 1 --2- 172.21.15.77:0/3863051707 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 0x7f7a4c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.257+0000 7f7a51e65640 1 --2- 172.21.15.77:0/3863051707 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0cf2f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.257+0000 7f7a51e65640 1 -- 172.21.15.77:0/3863051707 >> 172.21.15.77:0/3863051707 conn(0x7f7a4c0d3cd0 msgr2=0x7f7a4c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.257+0000 7f7a51e65640 1 -- 172.21.15.77:0/3863051707 shutdown_connections 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.257+0000 7f7a51e65640 1 -- 172.21.15.77:0/3863051707 wait complete. 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.258+0000 7f7a51e65640 1 Processor -- start 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.258+0000 7f7a51e65640 1 -- start start 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a51e65640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 0x7f7a4c14e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a51e65640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c0d27f0 0x7f7a4c0b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a51e65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a51e65640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c0d1c70 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a51e65640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c0b3d60 con 0x7f7a4c0d27f0 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a51e65640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c0d1df0 con 0x7f7a4c0cf830 2024-03-06T07:30:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50004/0 (socket says 172.21.15.77:50004) 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 -- 172.21.15.77:0/981383128 learned_addr learned my addr 172.21.15.77:0/981383128 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a43fff640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c0d27f0 0x7f7a4c0b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4b7fe640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 0x7f7a4c14e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 -- 172.21.15.77:0/981383128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 msgr2=0x7f7a4c14e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 0x7f7a4c14e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 -- 172.21.15.77:0/981383128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c0d27f0 msgr2=0x7f7a4c0b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c0d27f0 0x7f7a4c0b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 -- 172.21.15.77:0/981383128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c1f6fe0 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a4bfff640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0ccf70 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400cbc0 tx=0x7f7a4400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.259+0000 7f7a48ff9640 1 -- 172.21.15.77:0/981383128 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44017070 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.260+0000 7f7a48ff9640 1 -- 172.21.15.77:0/981383128 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4400ad40 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.260+0000 7f7a48ff9640 1 -- 172.21.15.77:0/981383128 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44012600 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.260+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c1f7280 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.260+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c1f7840 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.261+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4c04f330 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.262+0000 7f7a48ff9640 1 -- 172.21.15.77:0/981383128 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a4401a070 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.262+0000 7f7a48ff9640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.262+0000 7f7a48ff9640 1 -- 172.21.15.77:0/981383128 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a44089e50 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.262+0000 7f7a4b7fe640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.263+0000 7f7a4b7fe640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c036000 tx=0x7f7a3c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.267+0000 7f7a48ff9640 1 -- 172.21.15.77:0/981383128 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a44058680 con 0x7f7a4c15dfe0 2024-03-06T07:30:09.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.440+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7a4c0747a0 con 0x7f7a2c06b4d0 2024-03-06T07:30:09.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.452+0000 7f7a48ff9640 1 -- 172.21.15.77:0/981383128 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252472 (secure 0 0 0) 0x7f7a3c085a50 con 0x7f7a2c06b4d0 2024-03-06T07:30:09.456 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:09.457 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:30:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.454+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 msgr2=0x7f7a2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c036000 tx=0x7f7a3c03c040 comp rx=0 tx=0).stop 2024-03-06T07:30:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4c15dfe0 msgr2=0x7f7a4c0ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0ccf70 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400cbc0 tx=0x7f7a4400a430 comp rx=0 tx=0).stop 2024-03-06T07:30:09.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 shutdown_connections 2024-03-06T07:30:09.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4c15dfe0 0x7f7a4c0ccf70 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4c0d27f0 0x7f7a4c0b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 --2- 172.21.15.77:0/981383128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4c0cf830 0x7f7a4c14e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 >> 172.21.15.77:0/981383128 conn(0x7f7a4c0d3cd0 msgr2=0x7f7a4c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 shutdown_connections 2024-03-06T07:30:09.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.455+0000 7f7a51e65640 1 -- 172.21.15.77:0/981383128 wait complete. 2024-03-06T07:30:09.468 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-03-06T07:30:07.474645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310004,"kb_used_data":14764,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664716,"statfs":{"total":773094113280,"available":772776669184,"internally_reserved":0,"allocated":15118336,"data_stored":7250944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000526"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.127066+0000","last_change":"2024-03-06T07:30:06.125950+0000","last_active":"2024-03-06T07:30:07.127066+0000","last_peered":"2024-03-06T07:30:07.127066+0000","last_clean":"2024-03-06T07:30:07.127066+0000","last_became_active":"2024-03-06T07:30:06.125684+0000","last_became_peered":"2024-03-06T07:30:06.125684+0000","last_unstale":"2024-03-06T07:30:07.127066+0000","last_undegraded":"2024-03-06T07:30:07.127066+0000","last_fullsized":"2024-03-06T07:30:07.127066+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:20.177018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124064+0000","last_change":"2024-03-06T07:30:06.124613+0000","last_active":"2024-03-06T07:30:07.124064+0000","last_peered":"2024-03-06T07:30:07.124064+0000","last_clean":"2024-03-06T07:30:07.124064+0000","last_became_active":"2024-03-06T07:30:06.124337+0000","last_became_peered":"2024-03-06T07:30:06.124337+0000","last_unstale":"2024-03-06T07:30:07.124064+0000","last_undegraded":"2024-03-06T07:30:07.124064+0000","last_fullsized":"2024-03-06T07:30:07.124064+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:59.643647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125231+0000","last_change":"2024-03-06T07:30:06.126058+0000","last_active":"2024-03-06T07:30:07.125231+0000","last_peered":"2024-03-06T07:30:07.125231+0000","last_clean":"2024-03-06T07:30:07.125231+0000","last_became_active":"2024-03-06T07:30:06.125755+0000","last_became_peered":"2024-03-06T07:30:06.125755+0000","last_unstale":"2024-03-06T07:30:07.125231+0000","last_undegraded":"2024-03-06T07:30:07.125231+0000","last_fullsized":"2024-03-06T07:30:07.125231+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:41.063476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.130821+0000","last_change":"2024-03-06T07:30:06.127689+0000","last_active":"2024-03-06T07:30:06.130821+0000","last_peered":"2024-03-06T07:30:06.130821+0000","last_clean":"2024-03-06T07:30:06.130821+0000","last_became_active":"2024-03-06T07:30:06.127331+0000","last_became_peered":"2024-03-06T07:30:06.127331+0000","last_unstale":"2024-03-06T07:30:06.130821+0000","last_undegraded":"2024-03-06T07:30:06.130821+0000","last_fullsized":"2024-03-06T07:30:06.130821+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:15.612455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125201+0000","last_change":"2024-03-06T07:30:06.134261+0000","last_active":"2024-03-06T07:30:07.125201+0000","last_peered":"2024-03-06T07:30:07.125201+0000","last_clean":"2024-03-06T07:30:07.125201+0000","last_became_active":"2024-03-06T07:30:06.133916+0000","last_became_peered":"2024-03-06T07:30:06.133916+0000","last_unstale":"2024-03-06T07:30:07.125201+0000","last_undegraded":"2024-03-06T07:30:07.125201+0000","last_fullsized":"2024-03-06T07:30:07.125201+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:20.407586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126686+0000","last_change":"2024-03-06T07:30:06.119354+0000","last_active":"2024-03-06T07:30:07.126686+0000","last_peered":"2024-03-06T07:30:07.126686+0000","last_clean":"2024-03-06T07:30:07.126686+0000","last_became_active":"2024-03-06T07:30:06.119093+0000","last_became_peered":"2024-03-06T07:30:06.119093+0000","last_unstale":"2024-03-06T07:30:07.126686+0000","last_undegraded":"2024-03-06T07:30:07.126686+0000","last_fullsized":"2024-03-06T07:30:07.126686+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:14.376653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.141127+0000","last_change":"2024-03-06T07:30:06.135500+0000","last_active":"2024-03-06T07:30:06.141127+0000","last_peered":"2024-03-06T07:30:06.141127+0000","last_clean":"2024-03-06T07:30:06.141127+0000","last_became_active":"2024-03-06T07:30:06.135238+0000","last_became_peered":"2024-03-06T07:30:06.135238+0000","last_unstale":"2024-03-06T07:30:06.141127+0000","last_undegraded":"2024-03-06T07:30:06.141127+0000","last_fullsized":"2024-03-06T07:30:06.141127+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:05.729452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126488+0000","last_change":"2024-03-06T07:30:06.129984+0000","last_active":"2024-03-06T07:30:07.126488+0000","last_peered":"2024-03-06T07:30:07.126488+0000","last_clean":"2024-03-06T07:30:07.126488+0000","last_became_active":"2024-03-06T07:30:06.129694+0000","last_became_peered":"2024-03-06T07:30:06.129694+0000","last_unstale":"2024-03-06T07:30:07.126488+0000","last_undegraded":"2024-03-06T07:30:07.126488+0000","last_fullsized":"2024-03-06T07:30:07.126488+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:57.220320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123899+0000","last_change":"2024-03-06T07:30:06.129465+0000","last_active":"2024-03-06T07:30:07.123899+0000","last_peered":"2024-03-06T07:30:07.123899+0000","last_clean":"2024-03-06T07:30:07.123899+0000","last_became_active":"2024-03-06T07:30:06.129195+0000","last_became_peered":"2024-03-06T07:30:06.129195+0000","last_unstale":"2024-03-06T07:30:07.123899+0000","last_undegraded":"2024-03-06T07:30:07.123899+0000","last_fullsized":"2024-03-06T07:30:07.123899+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.557220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129375+0000","last_change":"2024-03-06T07:30:06.137708+0000","last_active":"2024-03-06T07:30:07.129375+0000","last_peered":"2024-03-06T07:30:07.129375+0000","last_clean":"2024-03-06T07:30:07.129375+0000","last_became_active":"2024-03-06T07:30:06.137409+0000","last_became_peered":"2024-03-06T07:30:06.137409+0000","last_unstale":"2024-03-06T07:30:07.129375+0000","last_undegraded":"2024-03-06T07:30:07.129375+0000","last_fullsized":"2024-03-06T07:30:07.129375+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:15.803035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126754+0000","last_change":"2024-03-06T07:30:06.131065+0000","last_active":"2024-03-06T07:30:07.126754+0000","last_peered":"2024-03-06T07:30:07.126754+0000","last_clean":"2024-03-06T07:30:07.126754+0000","last_became_active":"2024-03-06T07:30:06.130664+0000","last_became_peered":"2024-03-06T07:30:06.130664+0000","last_unstale":"2024-03-06T07:30:07.126754+0000","last_undegraded":"2024-03-06T07:30:07.126754+0000","last_fullsized":"2024-03-06T07:30:07.126754+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:42.732177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129261+0000","last_change":"2024-03-06T07:30:06.130555+0000","last_active":"2024-03-06T07:30:07.129261+0000","last_peered":"2024-03-06T07:30:07.129261+0000","last_clean":"2024-03-06T07:30:07.129261+0000","last_became_active":"2024-03-06T07:30:06.130236+0000","last_became_peered":"2024-03-06T07:30:06.130236+0000","last_unstale":"2024-03-06T07:30:07.129261+0000","last_undegraded":"2024-03-06T07:30:07.129261+0000","last_fullsized":"2024-03-06T07:30:07.129261+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:40.071499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125213+0000","last_change":"2024-03-06T07:30:06.127403+0000","last_active":"2024-03-06T07:30:07.125213+0000","last_peered":"2024-03-06T07:30:07.125213+0000","last_clean":"2024-03-06T07:30:07.125213+0000","last_became_active":"2024-03-06T07:30:06.127127+0000","last_became_peered":"2024-03-06T07:30:06.127127+0000","last_unstale":"2024-03-06T07:30:07.125213+0000","last_undegraded":"2024-03-06T07:30:07.125213+0000","last_fullsized":"2024-03-06T07:30:07.125213+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:54.244034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.136155+0000","last_change":"2024-03-06T07:30:06.135573+0000","last_active":"2024-03-06T07:30:06.136155+0000","last_peered":"2024-03-06T07:30:06.136155+0000","last_clean":"2024-03-06T07:30:06.136155+0000","last_became_active":"2024-03-06T07:30:06.135250+0000","last_became_peered":"2024-03-06T07:30:06.135250+0000","last_unstale":"2024-03-06T07:30:06.136155+0000","last_undegraded":"2024-03-06T07:30:06.136155+0000","last_fullsized":"2024-03-06T07:30:06.136155+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:07.128873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128948+0000","last_change":"2024-03-06T07:30:06.132936+0000","last_active":"2024-03-06T07:30:07.128948+0000","last_peered":"2024-03-06T07:30:07.128948+0000","last_clean":"2024-03-06T07:30:07.128948+0000","last_became_active":"2024-03-06T07:30:06.132642+0000","last_became_peered":"2024-03-06T07:30:06.132642+0000","last_unstale":"2024-03-06T07:30:07.128948+0000","last_undegraded":"2024-03-06T07:30:07.128948+0000","last_fullsized":"2024-03-06T07:30:07.128948+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:03.368830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330527+0000","last_change":"2024-03-06T07:30:06.130155+0000","last_active":"2024-03-06T07:30:07.330527+0000","last_peered":"2024-03-06T07:30:07.330527+0000","last_clean":"2024-03-06T07:30:07.330527+0000","last_became_active":"2024-03-06T07:30:06.129908+0000","last_became_peered":"2024-03-06T07:30:06.129908+0000","last_unstale":"2024-03-06T07:30:07.330527+0000","last_undegraded":"2024-03-06T07:30:07.330527+0000","last_fullsized":"2024-03-06T07:30:07.330527+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:41.592890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129639+0000","last_change":"2024-03-06T07:30:06.130128+0000","last_active":"2024-03-06T07:30:07.129639+0000","last_peered":"2024-03-06T07:30:07.129639+0000","last_clean":"2024-03-06T07:30:07.129639+0000","last_became_active":"2024-03-06T07:30:06.129781+0000","last_became_peered":"2024-03-06T07:30:06.129781+0000","last_unstale":"2024-03-06T07:30:07.129639+0000","last_undegraded":"2024-03-06T07:30:07.129639+0000","last_fullsized":"2024-03-06T07:30:07.129639+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:25.369784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125056+0000","last_change":"2024-03-06T07:30:06.126009+0000","last_active":"2024-03-06T07:30:07.125056+0000","last_peered":"2024-03-06T07:30:07.125056+0000","last_clean":"2024-03-06T07:30:07.125056+0000","last_became_active":"2024-03-06T07:30:06.125600+0000","last_became_peered":"2024-03-06T07:30:06.125600+0000","last_unstale":"2024-03-06T07:30:07.125056+0000","last_undegraded":"2024-03-06T07:30:07.125056+0000","last_fullsized":"2024-03-06T07:30:07.125056+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:42.459372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126995+0000","last_change":"2024-03-06T07:30:06.124576+0000","last_active":"2024-03-06T07:30:07.126995+0000","last_peered":"2024-03-06T07:30:07.126995+0000","last_clean":"2024-03-06T07:30:07.126995+0000","last_became_active":"2024-03-06T07:30:06.124320+0000","last_became_peered":"2024-03-06T07:30:06.124320+0000","last_unstale":"2024-03-06T07:30:07.126995+0000","last_undegraded":"2024-03-06T07:30:07.126995+0000","last_fullsized":"2024-03-06T07:30:07.126995+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:20.407586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330979+0000","last_change":"2024-03-06T07:30:06.127970+0000","last_active":"2024-03-06T07:30:07.330979+0000","last_peered":"2024-03-06T07:30:07.330979+0000","last_clean":"2024-03-06T07:30:07.330979+0000","last_became_active":"2024-03-06T07:30:06.127709+0000","last_became_peered":"2024-03-06T07:30:06.127709+0000","last_unstale":"2024-03-06T07:30:07.330979+0000","last_undegraded":"2024-03-06T07:30:07.330979+0000","last_fullsized":"2024-03-06T07:30:07.330979+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:02:09.627519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129285+0000","last_change":"2024-03-06T07:30:06.136462+0000","last_active":"2024-03-06T07:30:07.129285+0000","last_peered":"2024-03-06T07:30:07.129285+0000","last_clean":"2024-03-06T07:30:07.129285+0000","last_became_active":"2024-03-06T07:30:06.136202+0000","last_became_peered":"2024-03-06T07:30:06.136202+0000","last_unstale":"2024-03-06T07:30:07.129285+0000","last_undegraded":"2024-03-06T07:30:07.129285+0000","last_fullsized":"2024-03-06T07:30:07.129285+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:36.881643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.126492+0000","last_change":"2024-03-06T07:30:06.124805+0000","last_active":"2024-03-06T07:30:06.126492+0000","last_peered":"2024-03-06T07:30:06.126492+0000","last_clean":"2024-03-06T07:30:06.126492+0000","last_became_active":"2024-03-06T07:30:06.124467+0000","last_became_peered":"2024-03-06T07:30:06.124467+0000","last_unstale":"2024-03-06T07:30:06.126492+0000","last_undegraded":"2024-03-06T07:30:06.126492+0000","last_fullsized":"2024-03-06T07:30:06.126492+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:56.773729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.130786+0000","last_change":"2024-03-06T07:30:06.128235+0000","last_active":"2024-03-06T07:30:06.130786+0000","last_peered":"2024-03-06T07:30:06.130786+0000","last_clean":"2024-03-06T07:30:06.130786+0000","last_became_active":"2024-03-06T07:30:06.127878+0000","last_became_peered":"2024-03-06T07:30:06.127878+0000","last_unstale":"2024-03-06T07:30:06.130786+0000","last_undegraded":"2024-03-06T07:30:06.130786+0000","last_fullsized":"2024-03-06T07:30:06.130786+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:18.885685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128985+0000","last_change":"2024-03-06T07:30:06.136694+0000","last_active":"2024-03-06T07:30:07.128985+0000","last_peered":"2024-03-06T07:30:07.128985+0000","last_clean":"2024-03-06T07:30:07.128985+0000","last_became_active":"2024-03-06T07:30:06.136402+0000","last_became_peered":"2024-03-06T07:30:06.136402+0000","last_unstale":"2024-03-06T07:30:07.128985+0000","last_undegraded":"2024-03-06T07:30:07.128985+0000","last_fullsized":"2024-03-06T07:30:07.128985+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:38.062293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330908+0000","last_change":"2024-03-06T07:30:06.130843+0000","last_active":"2024-03-06T07:30:07.330908+0000","last_peered":"2024-03-06T07:30:07.330908+0000","last_clean":"2024-03-06T07:30:07.330908+0000","last_became_active":"2024-03-06T07:30:06.130604+0000","last_became_peered":"2024-03-06T07:30:06.130604+0000","last_unstale":"2024-03-06T07:30:07.330908+0000","last_undegraded":"2024-03-06T07:30:07.330908+0000","last_fullsized":"2024-03-06T07:30:07.330908+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:49.438538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126903+0000","last_change":"2024-03-06T07:30:06.120335+0000","last_active":"2024-03-06T07:30:07.126903+0000","last_peered":"2024-03-06T07:30:07.126903+0000","last_clean":"2024-03-06T07:30:07.126903+0000","last_became_active":"2024-03-06T07:30:06.119949+0000","last_became_peered":"2024-03-06T07:30:06.119949+0000","last_unstale":"2024-03-06T07:30:07.126903+0000","last_undegraded":"2024-03-06T07:30:07.126903+0000","last_fullsized":"2024-03-06T07:30:07.126903+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:40.918933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330868+0000","last_change":"2024-03-06T07:30:06.125274+0000","last_active":"2024-03-06T07:30:07.330868+0000","last_peered":"2024-03-06T07:30:07.330868+0000","last_clean":"2024-03-06T07:30:07.330868+0000","last_became_active":"2024-03-06T07:30:06.124979+0000","last_became_peered":"2024-03-06T07:30:06.124979+0000","last_unstale":"2024-03-06T07:30:07.330868+0000","last_undegraded":"2024-03-06T07:30:07.330868+0000","last_fullsized":"2024-03-06T07:30:07.330868+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:53.328688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.138540+0000","last_change":"2024-03-06T07:30:06.133140+0000","last_active":"2024-03-06T07:30:06.138540+0000","last_peered":"2024-03-06T07:30:06.138540+0000","last_clean":"2024-03-06T07:30:06.138540+0000","last_became_active":"2024-03-06T07:30:06.132495+0000","last_became_peered":"2024-03-06T07:30:06.132495+0000","last_unstale":"2024-03-06T07:30:06.138540+0000","last_undegraded":"2024-03-06T07:30:06.138540+0000","last_fullsized":"2024-03-06T07:30:06.138540+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.557220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126691+0000","last_change":"2024-03-06T07:30:06.122624+0000","last_active":"2024-03-06T07:30:07.126691+0000","last_peered":"2024-03-06T07:30:07.126691+0000","last_clean":"2024-03-06T07:30:07.126691+0000","last_became_active":"2024-03-06T07:30:06.122314+0000","last_became_peered":"2024-03-06T07:30:06.122314+0000","last_unstale":"2024-03-06T07:30:07.126691+0000","last_undegraded":"2024-03-06T07:30:07.126691+0000","last_fullsized":"2024-03-06T07:30:07.126691+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:42.260077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.124688+0000","last_change":"2024-03-06T07:30:06.123444+0000","last_active":"2024-03-06T07:30:06.124688+0000","last_peered":"2024-03-06T07:30:06.124688+0000","last_clean":"2024-03-06T07:30:06.124688+0000","last_became_active":"2024-03-06T07:30:06.123193+0000","last_became_peered":"2024-03-06T07:30:06.123193+0000","last_unstale":"2024-03-06T07:30:06.124688+0000","last_undegraded":"2024-03-06T07:30:06.124688+0000","last_fullsized":"2024-03-06T07:30:06.124688+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:29.206209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123829+0000","last_change":"2024-03-06T07:30:06.126541+0000","last_active":"2024-03-06T07:30:07.123829+0000","last_peered":"2024-03-06T07:30:07.123829+0000","last_clean":"2024-03-06T07:30:07.123829+0000","last_became_active":"2024-03-06T07:30:06.126282+0000","last_became_peered":"2024-03-06T07:30:06.126282+0000","last_unstale":"2024-03-06T07:30:07.123829+0000","last_undegraded":"2024-03-06T07:30:07.123829+0000","last_fullsized":"2024-03-06T07:30:07.123829+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:01.524141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330600+0000","last_change":"2024-03-06T07:30:06.135208+0000","last_active":"2024-03-06T07:30:07.330600+0000","last_peered":"2024-03-06T07:30:07.330600+0000","last_clean":"2024-03-06T07:30:07.330600+0000","last_became_active":"2024-03-06T07:30:06.134870+0000","last_became_peered":"2024-03-06T07:30:06.134870+0000","last_unstale":"2024-03-06T07:30:07.330600+0000","last_undegraded":"2024-03-06T07:30:07.330600+0000","last_fullsized":"2024-03-06T07:30:07.330600+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:54.764976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129383+0000","last_change":"2024-03-06T07:30:06.133373+0000","last_active":"2024-03-06T07:30:07.129383+0000","last_peered":"2024-03-06T07:30:07.129383+0000","last_clean":"2024-03-06T07:30:07.129383+0000","last_became_active":"2024-03-06T07:30:06.132526+0000","last_became_peered":"2024-03-06T07:30:06.132526+0000","last_unstale":"2024-03-06T07:30:07.129383+0000","last_undegraded":"2024-03-06T07:30:07.129383+0000","last_fullsized":"2024-03-06T07:30:07.129383+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:28.075065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"138'4","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129473+0000","last_change":"2024-03-06T07:30:05.107937+0000","last_active":"2024-03-06T07:30:07.129473+0000","last_peered":"2024-03-06T07:30:07.129473+0000","last_clean":"2024-03-06T07:30:07.129473+0000","last_became_active":"2024-03-06T07:30:05.107605+0000","last_became_peered":"2024-03-06T07:30:05.107605+0000","last_unstale":"2024-03-06T07:30:07.129473+0000","last_undegraded":"2024-03-06T07:30:07.129473+0000","last_fullsized":"2024-03-06T07:30:07.129473+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:09.201810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124055+0000","last_change":"2024-03-06T07:30:06.124514+0000","last_active":"2024-03-06T07:30:07.124055+0000","last_peered":"2024-03-06T07:30:07.124055+0000","last_clean":"2024-03-06T07:30:07.124055+0000","last_became_active":"2024-03-06T07:30:06.124207+0000","last_became_peered":"2024-03-06T07:30:06.124207+0000","last_unstale":"2024-03-06T07:30:07.124055+0000","last_undegraded":"2024-03-06T07:30:07.124055+0000","last_fullsized":"2024-03-06T07:30:07.124055+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:03.567494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124124+0000","last_change":"2024-03-06T07:30:05.102758+0000","last_active":"2024-03-06T07:30:07.124124+0000","last_peered":"2024-03-06T07:30:07.124124+0000","last_clean":"2024-03-06T07:30:07.124124+0000","last_became_active":"2024-03-06T07:30:05.102510+0000","last_became_peered":"2024-03-06T07:30:05.102510+0000","last_unstale":"2024-03-06T07:30:07.124124+0000","last_undegraded":"2024-03-06T07:30:07.124124+0000","last_fullsized":"2024-03-06T07:30:07.124124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:14.184362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126753+0000","last_change":"2024-03-06T07:30:06.120870+0000","last_active":"2024-03-06T07:30:07.126753+0000","last_peered":"2024-03-06T07:30:07.126753+0000","last_clean":"2024-03-06T07:30:07.126753+0000","last_became_active":"2024-03-06T07:30:06.120609+0000","last_became_peered":"2024-03-06T07:30:06.120609+0000","last_unstale":"2024-03-06T07:30:07.126753+0000","last_undegraded":"2024-03-06T07:30:07.126753+0000","last_fullsized":"2024-03-06T07:30:07.126753+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:45.680312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"138'2","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.142465+0000","last_change":"2024-03-06T07:30:05.104208+0000","last_active":"2024-03-06T07:30:06.142465+0000","last_peered":"2024-03-06T07:30:06.142465+0000","last_clean":"2024-03-06T07:30:06.142465+0000","last_became_active":"2024-03-06T07:30:05.103915+0000","last_became_peered":"2024-03-06T07:30:05.103915+0000","last_unstale":"2024-03-06T07:30:06.142465+0000","last_undegraded":"2024-03-06T07:30:06.142465+0000","last_fullsized":"2024-03-06T07:30:06.142465+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:32.335160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129187+0000","last_change":"2024-03-06T07:30:06.130795+0000","last_active":"2024-03-06T07:30:07.129187+0000","last_peered":"2024-03-06T07:30:07.129187+0000","last_clean":"2024-03-06T07:30:07.129187+0000","last_became_active":"2024-03-06T07:30:06.130506+0000","last_became_peered":"2024-03-06T07:30:06.130506+0000","last_unstale":"2024-03-06T07:30:07.129187+0000","last_undegraded":"2024-03-06T07:30:07.129187+0000","last_fullsized":"2024-03-06T07:30:07.129187+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:38.429183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"138'2","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.143104+0000","last_change":"2024-03-06T07:30:05.108043+0000","last_active":"2024-03-06T07:30:06.143104+0000","last_peered":"2024-03-06T07:30:06.143104+0000","last_clean":"2024-03-06T07:30:06.143104+0000","last_became_active":"2024-03-06T07:30:05.107777+0000","last_became_peered":"2024-03-06T07:30:05.107777+0000","last_unstale":"2024-03-06T07:30:06.143104+0000","last_undegraded":"2024-03-06T07:30:06.143104+0000","last_fullsized":"2024-03-06T07:30:06.143104+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:27.915657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123951+0000","last_change":"2024-03-06T07:30:06.126780+0000","last_active":"2024-03-06T07:30:07.123951+0000","last_peered":"2024-03-06T07:30:07.123951+0000","last_clean":"2024-03-06T07:30:07.123951+0000","last_became_active":"2024-03-06T07:30:06.126308+0000","last_became_peered":"2024-03-06T07:30:06.126308+0000","last_unstale":"2024-03-06T07:30:07.123951+0000","last_undegraded":"2024-03-06T07:30:07.123951+0000","last_fullsized":"2024-03-06T07:30:07.123951+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:05.729452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330581+0000","last_change":"2024-03-06T07:30:05.104149+0000","last_active":"2024-03-06T07:30:07.330581+0000","last_peered":"2024-03-06T07:30:07.330581+0000","last_clean":"2024-03-06T07:30:07.330581+0000","last_became_active":"2024-03-06T07:30:05.103815+0000","last_became_peered":"2024-03-06T07:30:05.103815+0000","last_unstale":"2024-03-06T07:30:07.330581+0000","last_undegraded":"2024-03-06T07:30:07.330581+0000","last_fullsized":"2024-03-06T07:30:07.330581+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:29.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.130207+0000","last_change":"2024-03-06T07:30:06.128531+0000","last_active":"2024-03-06T07:30:06.130207+0000","last_peered":"2024-03-06T07:30:06.130207+0000","last_clean":"2024-03-06T07:30:06.130207+0000","last_became_active":"2024-03-06T07:30:06.127270+0000","last_became_peered":"2024-03-06T07:30:06.127270+0000","last_unstale":"2024-03-06T07:30:06.130207+0000","last_undegraded":"2024-03-06T07:30:06.130207+0000","last_fullsized":"2024-03-06T07:30:06.130207+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:20.154495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124613+0000","last_change":"2024-03-06T07:30:05.102755+0000","last_active":"2024-03-06T07:30:07.124613+0000","last_peered":"2024-03-06T07:30:07.124613+0000","last_clean":"2024-03-06T07:30:07.124613+0000","last_became_active":"2024-03-06T07:30:05.102437+0000","last_became_peered":"2024-03-06T07:30:05.102437+0000","last_unstale":"2024-03-06T07:30:07.124613+0000","last_undegraded":"2024-03-06T07:30:07.124613+0000","last_fullsized":"2024-03-06T07:30:07.124613+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:35.097157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123749+0000","last_change":"2024-03-06T07:30:06.127275+0000","last_active":"2024-03-06T07:30:07.123749+0000","last_peered":"2024-03-06T07:30:07.123749+0000","last_clean":"2024-03-06T07:30:07.123749+0000","last_became_active":"2024-03-06T07:30:06.126863+0000","last_became_peered":"2024-03-06T07:30:06.126863+0000","last_unstale":"2024-03-06T07:30:07.123749+0000","last_undegraded":"2024-03-06T07:30:07.123749+0000","last_fullsized":"2024-03-06T07:30:07.123749+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:25.729483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330629+0000","last_change":"2024-03-06T07:30:05.104815+0000","last_active":"2024-03-06T07:30:07.330629+0000","last_peered":"2024-03-06T07:30:07.330629+0000","last_clean":"2024-03-06T07:30:07.330629+0000","last_became_active":"2024-03-06T07:30:05.104574+0000","last_became_peered":"2024-03-06T07:30:05.104574+0000","last_unstale":"2024-03-06T07:30:07.330629+0000","last_undegraded":"2024-03-06T07:30:07.330629+0000","last_fullsized":"2024-03-06T07:30:07.330629+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:04.096806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124836+0000","last_change":"2024-03-06T07:30:06.131426+0000","last_active":"2024-03-06T07:30:07.124836+0000","last_peered":"2024-03-06T07:30:07.124836+0000","last_clean":"2024-03-06T07:30:07.124836+0000","last_became_active":"2024-03-06T07:30:06.131142+0000","last_became_peered":"2024-03-06T07:30:06.131142+0000","last_unstale":"2024-03-06T07:30:07.124836+0000","last_undegraded":"2024-03-06T07:30:07.124836+0000","last_fullsized":"2024-03-06T07:30:07.124836+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:45.680312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.113187+0000","last_change":"2024-03-06T07:30:05.104244+0000","last_active":"2024-03-06T07:30:06.113187+0000","last_peered":"2024-03-06T07:30:06.113187+0000","last_clean":"2024-03-06T07:30:06.113187+0000","last_became_active":"2024-03-06T07:30:05.103960+0000","last_became_peered":"2024-03-06T07:30:05.103960+0000","last_unstale":"2024-03-06T07:30:06.113187+0000","last_undegraded":"2024-03-06T07:30:06.113187+0000","last_fullsized":"2024-03-06T07:30:06.113187+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:19.172375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124731+0000","last_change":"2024-03-06T07:30:06.131995+0000","last_active":"2024-03-06T07:30:07.124731+0000","last_peered":"2024-03-06T07:30:07.124731+0000","last_clean":"2024-03-06T07:30:07.124731+0000","last_became_active":"2024-03-06T07:30:06.131696+0000","last_became_peered":"2024-03-06T07:30:06.131696+0000","last_unstale":"2024-03-06T07:30:07.124731+0000","last_undegraded":"2024-03-06T07:30:07.124731+0000","last_fullsized":"2024-03-06T07:30:07.124731+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:42.260077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124167+0000","last_change":"2024-03-06T07:30:05.102786+0000","last_active":"2024-03-06T07:30:07.124167+0000","last_peered":"2024-03-06T07:30:07.124167+0000","last_clean":"2024-03-06T07:30:07.124167+0000","last_became_active":"2024-03-06T07:30:05.101112+0000","last_became_peered":"2024-03-06T07:30:05.101112+0000","last_unstale":"2024-03-06T07:30:07.124167+0000","last_undegraded":"2024-03-06T07:30:07.124167+0000","last_fullsized":"2024-03-06T07:30:07.124167+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:17.233410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129519+0000","last_change":"2024-03-06T07:30:06.125340+0000","last_active":"2024-03-06T07:30:07.129519+0000","last_peered":"2024-03-06T07:30:07.129519+0000","last_clean":"2024-03-06T07:30:07.129519+0000","last_became_active":"2024-03-06T07:30:06.124975+0000","last_became_peered":"2024-03-06T07:30:06.124975+0000","last_unstale":"2024-03-06T07:30:07.129519+0000","last_undegraded":"2024-03-06T07:30:07.129519+0000","last_fullsized":"2024-03-06T07:30:07.129519+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:12.859599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129507+0000","last_change":"2024-03-06T07:30:05.104388+0000","last_active":"2024-03-06T07:30:07.129507+0000","last_peered":"2024-03-06T07:30:07.129507+0000","last_clean":"2024-03-06T07:30:07.129507+0000","last_became_active":"2024-03-06T07:30:05.104101+0000","last_became_peered":"2024-03-06T07:30:05.104101+0000","last_unstale":"2024-03-06T07:30:07.129507+0000","last_undegraded":"2024-03-06T07:30:07.129507+0000","last_fullsized":"2024-03-06T07:30:07.129507+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:22.598603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330953+0000","last_change":"2024-03-06T07:30:06.126112+0000","last_active":"2024-03-06T07:30:07.330953+0000","last_peered":"2024-03-06T07:30:07.330953+0000","last_clean":"2024-03-06T07:30:07.330953+0000","last_became_active":"2024-03-06T07:30:06.125851+0000","last_became_peered":"2024-03-06T07:30:06.125851+0000","last_unstale":"2024-03-06T07:30:07.330953+0000","last_undegraded":"2024-03-06T07:30:07.330953+0000","last_fullsized":"2024-03-06T07:30:07.330953+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:02.843483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"138'6","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.143498+0000","last_change":"2024-03-06T07:30:05.105115+0000","last_active":"2024-03-06T07:30:06.143498+0000","last_peered":"2024-03-06T07:30:06.143498+0000","last_clean":"2024-03-06T07:30:06.143498+0000","last_became_active":"2024-03-06T07:30:05.104838+0000","last_became_peered":"2024-03-06T07:30:05.104838+0000","last_unstale":"2024-03-06T07:30:06.143498+0000","last_undegraded":"2024-03-06T07:30:06.143498+0000","last_fullsized":"2024-03-06T07:30:06.143498+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:24.724840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124983+0000","last_change":"2024-03-06T07:30:06.127030+0000","last_active":"2024-03-06T07:30:07.124983+0000","last_peered":"2024-03-06T07:30:07.124983+0000","last_clean":"2024-03-06T07:30:07.124983+0000","last_became_active":"2024-03-06T07:30:06.126731+0000","last_became_peered":"2024-03-06T07:30:06.126731+0000","last_unstale":"2024-03-06T07:30:07.124983+0000","last_undegraded":"2024-03-06T07:30:07.124983+0000","last_fullsized":"2024-03-06T07:30:07.124983+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:03.904829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125030+0000","last_change":"2024-03-06T07:30:05.105420+0000","last_active":"2024-03-06T07:30:07.125030+0000","last_peered":"2024-03-06T07:30:07.125030+0000","last_clean":"2024-03-06T07:30:07.125030+0000","last_became_active":"2024-03-06T07:30:05.105100+0000","last_became_peered":"2024-03-06T07:30:05.105100+0000","last_unstale":"2024-03-06T07:30:07.125030+0000","last_undegraded":"2024-03-06T07:30:07.125030+0000","last_fullsized":"2024-03-06T07:30:07.125030+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:19.149852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123822+0000","last_change":"2024-03-06T07:30:06.124524+0000","last_active":"2024-03-06T07:30:07.123822+0000","last_peered":"2024-03-06T07:30:07.123822+0000","last_clean":"2024-03-06T07:30:07.123822+0000","last_became_active":"2024-03-06T07:30:06.124273+0000","last_became_peered":"2024-03-06T07:30:06.124273+0000","last_unstale":"2024-03-06T07:30:07.123822+0000","last_undegraded":"2024-03-06T07:30:07.123822+0000","last_fullsized":"2024-03-06T07:30:07.123822+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:50.660477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125080+0000","last_change":"2024-03-06T07:30:05.097944+0000","last_active":"2024-03-06T07:30:07.125080+0000","last_peered":"2024-03-06T07:30:07.125080+0000","last_clean":"2024-03-06T07:30:07.125080+0000","last_became_active":"2024-03-06T07:30:05.096406+0000","last_became_peered":"2024-03-06T07:30:05.096406+0000","last_unstale":"2024-03-06T07:30:07.125080+0000","last_undegraded":"2024-03-06T07:30:07.125080+0000","last_fullsized":"2024-03-06T07:30:07.125080+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:14.607812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124953+0000","last_change":"2024-03-06T07:30:06.133593+0000","last_active":"2024-03-06T07:30:07.124953+0000","last_peered":"2024-03-06T07:30:07.124953+0000","last_clean":"2024-03-06T07:30:07.124953+0000","last_became_active":"2024-03-06T07:30:06.133139+0000","last_became_peered":"2024-03-06T07:30:06.133139+0000","last_unstale":"2024-03-06T07:30:07.124953+0000","last_undegraded":"2024-03-06T07:30:07.124953+0000","last_fullsized":"2024-03-06T07:30:07.124953+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:07.895365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126434+0000","last_change":"2024-03-06T07:30:05.106125+0000","last_active":"2024-03-06T07:30:07.126434+0000","last_peered":"2024-03-06T07:30:07.126434+0000","last_clean":"2024-03-06T07:30:07.126434+0000","last_became_active":"2024-03-06T07:30:05.105865+0000","last_became_peered":"2024-03-06T07:30:05.105865+0000","last_unstale":"2024-03-06T07:30:07.126434+0000","last_undegraded":"2024-03-06T07:30:07.126434+0000","last_fullsized":"2024-03-06T07:30:07.126434+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:53.760333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124371+0000","last_change":"2024-03-06T07:30:06.123068+0000","last_active":"2024-03-06T07:30:07.124371+0000","last_peered":"2024-03-06T07:30:07.124371+0000","last_clean":"2024-03-06T07:30:07.124371+0000","last_became_active":"2024-03-06T07:30:06.122786+0000","last_became_peered":"2024-03-06T07:30:06.122786+0000","last_unstale":"2024-03-06T07:30:07.124371+0000","last_undegraded":"2024-03-06T07:30:07.124371+0000","last_fullsized":"2024-03-06T07:30:07.124371+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:40.145873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"138'8","reported_seq":22,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330648+0000","last_change":"2024-03-06T07:30:05.098064+0000","last_active":"2024-03-06T07:30:07.330648+0000","last_peered":"2024-03-06T07:30:07.330648+0000","last_clean":"2024-03-06T07:30:07.330648+0000","last_became_active":"2024-03-06T07:30:05.097776+0000","last_became_peered":"2024-03-06T07:30:05.097776+0000","last_unstale":"2024-03-06T07:30:07.330648+0000","last_undegraded":"2024-03-06T07:30:07.330648+0000","last_fullsized":"2024-03-06T07:30:07.330648+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:52.540868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129517+0000","last_change":"2024-03-06T07:30:06.131061+0000","last_active":"2024-03-06T07:30:07.129517+0000","last_peered":"2024-03-06T07:30:07.129517+0000","last_clean":"2024-03-06T07:30:07.129517+0000","last_became_active":"2024-03-06T07:30:06.130457+0000","last_became_peered":"2024-03-06T07:30:06.130457+0000","last_unstale":"2024-03-06T07:30:07.129517+0000","last_undegraded":"2024-03-06T07:30:07.129517+0000","last_fullsized":"2024-03-06T07:30:07.129517+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:45.311727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125029+0000","last_change":"2024-03-06T07:30:05.097957+0000","last_active":"2024-03-06T07:30:07.125029+0000","last_peered":"2024-03-06T07:30:07.125029+0000","last_clean":"2024-03-06T07:30:07.125029+0000","last_became_active":"2024-03-06T07:30:05.097674+0000","last_became_peered":"2024-03-06T07:30:05.097674+0000","last_unstale":"2024-03-06T07:30:07.125029+0000","last_undegraded":"2024-03-06T07:30:07.125029+0000","last_fullsized":"2024-03-06T07:30:07.125029+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:48.585329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330946+0000","last_change":"2024-03-06T07:30:06.126331+0000","last_active":"2024-03-06T07:30:07.330946+0000","last_peered":"2024-03-06T07:30:07.330946+0000","last_clean":"2024-03-06T07:30:07.330946+0000","last_became_active":"2024-03-06T07:30:06.126070+0000","last_became_peered":"2024-03-06T07:30:06.126070+0000","last_unstale":"2024-03-06T07:30:07.330946+0000","last_undegraded":"2024-03-06T07:30:07.330946+0000","last_fullsized":"2024-03-06T07:30:07.330946+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:43.930719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124214+0000","last_change":"2024-03-06T07:30:05.100969+0000","last_active":"2024-03-06T07:30:07.124214+0000","last_peered":"2024-03-06T07:30:07.124214+0000","last_clean":"2024-03-06T07:30:07.124214+0000","last_became_active":"2024-03-06T07:30:05.100665+0000","last_became_peered":"2024-03-06T07:30:05.100665+0000","last_unstale":"2024-03-06T07:30:07.124214+0000","last_undegraded":"2024-03-06T07:30:07.124214+0000","last_fullsized":"2024-03-06T07:30:07.124214+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:20.043276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124619+0000","last_change":"2024-03-06T07:30:06.127335+0000","last_active":"2024-03-06T07:30:07.124619+0000","last_peered":"2024-03-06T07:30:07.124619+0000","last_clean":"2024-03-06T07:30:07.124619+0000","last_became_active":"2024-03-06T07:30:06.127020+0000","last_became_peered":"2024-03-06T07:30:06.127020+0000","last_unstale":"2024-03-06T07:30:07.124619+0000","last_undegraded":"2024-03-06T07:30:07.124619+0000","last_fullsized":"2024-03-06T07:30:07.124619+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:26.221909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330965+0000","last_change":"2024-03-06T07:30:05.104984+0000","last_active":"2024-03-06T07:30:07.330965+0000","last_peered":"2024-03-06T07:30:07.330965+0000","last_clean":"2024-03-06T07:30:07.330965+0000","last_became_active":"2024-03-06T07:30:05.104528+0000","last_became_peered":"2024-03-06T07:30:05.104528+0000","last_unstale":"2024-03-06T07:30:07.330965+0000","last_undegraded":"2024-03-06T07:30:07.330965+0000","last_fullsized":"2024-03-06T07:30:07.330965+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:32.436401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126899+0000","last_change":"2024-03-06T07:30:06.122867+0000","last_active":"2024-03-06T07:30:07.126899+0000","last_peered":"2024-03-06T07:30:07.126899+0000","last_clean":"2024-03-06T07:30:07.126899+0000","last_became_active":"2024-03-06T07:30:06.122553+0000","last_became_peered":"2024-03-06T07:30:06.122553+0000","last_unstale":"2024-03-06T07:30:07.126899+0000","last_undegraded":"2024-03-06T07:30:07.126899+0000","last_fullsized":"2024-03-06T07:30:07.126899+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:07.895365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124230+0000","last_change":"2024-03-06T07:30:05.101492+0000","last_active":"2024-03-06T07:30:07.124230+0000","last_peered":"2024-03-06T07:30:07.124230+0000","last_clean":"2024-03-06T07:30:07.124230+0000","last_became_active":"2024-03-06T07:30:05.101232+0000","last_became_peered":"2024-03-06T07:30:05.101232+0000","last_unstale":"2024-03-06T07:30:07.124230+0000","last_undegraded":"2024-03-06T07:30:07.124230+0000","last_fullsized":"2024-03-06T07:30:07.124230+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:39.022253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.122013+0000","last_change":"2024-03-06T07:30:06.121364+0000","last_active":"2024-03-06T07:30:06.122013+0000","last_peered":"2024-03-06T07:30:06.122013+0000","last_clean":"2024-03-06T07:30:06.122013+0000","last_became_active":"2024-03-06T07:30:06.120979+0000","last_became_peered":"2024-03-06T07:30:06.120979+0000","last_unstale":"2024-03-06T07:30:06.122013+0000","last_undegraded":"2024-03-06T07:30:06.122013+0000","last_fullsized":"2024-03-06T07:30:06.122013+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:39.105065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"138'2","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.138586+0000","last_change":"2024-03-06T07:30:05.106837+0000","last_active":"2024-03-06T07:30:06.138586+0000","last_peered":"2024-03-06T07:30:06.138586+0000","last_clean":"2024-03-06T07:30:06.138586+0000","last_became_active":"2024-03-06T07:30:05.106576+0000","last_became_peered":"2024-03-06T07:30:05.106576+0000","last_unstale":"2024-03-06T07:30:06.138586+0000","last_undegraded":"2024-03-06T07:30:06.138586+0000","last_fullsized":"2024-03-06T07:30:06.138586+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:49.655834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.141408+0000","last_change":"2024-03-06T07:30:06.136909+0000","last_active":"2024-03-06T07:30:06.141408+0000","last_peered":"2024-03-06T07:30:06.141408+0000","last_clean":"2024-03-06T07:30:06.141408+0000","last_became_active":"2024-03-06T07:30:06.136604+0000","last_became_peered":"2024-03-06T07:30:06.136604+0000","last_unstale":"2024-03-06T07:30:06.141408+0000","last_undegraded":"2024-03-06T07:30:06.141408+0000","last_fullsized":"2024-03-06T07:30:06.141408+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:59.643647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129017+0000","last_change":"2024-03-06T07:30:05.106795+0000","last_active":"2024-03-06T07:30:07.129017+0000","last_peered":"2024-03-06T07:30:07.129017+0000","last_clean":"2024-03-06T07:30:07.129017+0000","last_became_active":"2024-03-06T07:30:05.106482+0000","last_became_peered":"2024-03-06T07:30:05.106482+0000","last_unstale":"2024-03-06T07:30:07.129017+0000","last_undegraded":"2024-03-06T07:30:07.129017+0000","last_fullsized":"2024-03-06T07:30:07.129017+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:50.017775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.130846+0000","last_change":"2024-03-06T07:30:06.128473+0000","last_active":"2024-03-06T07:30:06.130846+0000","last_peered":"2024-03-06T07:30:06.130846+0000","last_clean":"2024-03-06T07:30:06.130846+0000","last_became_active":"2024-03-06T07:30:06.127956+0000","last_became_peered":"2024-03-06T07:30:06.127956+0000","last_unstale":"2024-03-06T07:30:06.130846+0000","last_undegraded":"2024-03-06T07:30:06.130846+0000","last_fullsized":"2024-03-06T07:30:06.130846+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:22.913256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128879+0000","last_change":"2024-03-06T07:30:05.105611+0000","last_active":"2024-03-06T07:30:07.128879+0000","last_peered":"2024-03-06T07:30:07.128879+0000","last_clean":"2024-03-06T07:30:07.128879+0000","last_became_active":"2024-03-06T07:30:05.105311+0000","last_became_peered":"2024-03-06T07:30:05.105311+0000","last_unstale":"2024-03-06T07:30:07.128879+0000","last_undegraded":"2024-03-06T07:30:07.128879+0000","last_fullsized":"2024-03-06T07:30:07.128879+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:20.043276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.141150+0000","last_change":"2024-03-06T07:30:06.136843+0000","last_active":"2024-03-06T07:30:06.141150+0000","last_peered":"2024-03-06T07:30:06.141150+0000","last_clean":"2024-03-06T07:30:06.141150+0000","last_became_active":"2024-03-06T07:30:06.136566+0000","last_became_peered":"2024-03-06T07:30:06.136566+0000","last_unstale":"2024-03-06T07:30:06.141150+0000","last_undegraded":"2024-03-06T07:30:06.141150+0000","last_fullsized":"2024-03-06T07:30:06.141150+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:45.737995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124197+0000","last_change":"2024-03-06T07:30:05.106739+0000","last_active":"2024-03-06T07:30:07.124197+0000","last_peered":"2024-03-06T07:30:07.124197+0000","last_clean":"2024-03-06T07:30:07.124197+0000","last_became_active":"2024-03-06T07:30:05.106368+0000","last_became_peered":"2024-03-06T07:30:05.106368+0000","last_unstale":"2024-03-06T07:30:07.124197+0000","last_undegraded":"2024-03-06T07:30:07.124197+0000","last_fullsized":"2024-03-06T07:30:07.124197+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:17.881042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124886+0000","last_change":"2024-03-06T07:30:06.133418+0000","last_active":"2024-03-06T07:30:07.124886+0000","last_peered":"2024-03-06T07:30:07.124886+0000","last_clean":"2024-03-06T07:30:07.124886+0000","last_became_active":"2024-03-06T07:30:06.133056+0000","last_became_peered":"2024-03-06T07:30:06.133056+0000","last_unstale":"2024-03-06T07:30:07.124886+0000","last_undegraded":"2024-03-06T07:30:07.124886+0000","last_fullsized":"2024-03-06T07:30:07.124886+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:42.732177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129614+0000","last_change":"2024-03-06T07:30:05.106211+0000","last_active":"2024-03-06T07:30:07.129614+0000","last_peered":"2024-03-06T07:30:07.129614+0000","last_clean":"2024-03-06T07:30:07.129614+0000","last_became_active":"2024-03-06T07:30:05.105876+0000","last_became_peered":"2024-03-06T07:30:05.105876+0000","last_unstale":"2024-03-06T07:30:07.129614+0000","last_undegraded":"2024-03-06T07:30:07.129614+0000","last_fullsized":"2024-03-06T07:30:07.129614+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:05.109313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125170+0000","last_change":"2024-03-06T07:30:06.128314+0000","last_active":"2024-03-06T07:30:07.125170+0000","last_peered":"2024-03-06T07:30:07.125170+0000","last_clean":"2024-03-06T07:30:07.125170+0000","last_became_active":"2024-03-06T07:30:06.128052+0000","last_became_peered":"2024-03-06T07:30:06.128052+0000","last_unstale":"2024-03-06T07:30:07.125170+0000","last_undegraded":"2024-03-06T07:30:07.125170+0000","last_fullsized":"2024-03-06T07:30:07.125170+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:53.089086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"139'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128208+0000","last_change":"2024-03-06T07:30:05.103786+0000","last_active":"2024-03-06T07:30:07.128208+0000","last_peered":"2024-03-06T07:30:07.128208+0000","last_clean":"2024-03-06T07:30:07.128208+0000","last_became_active":"2024-03-06T07:30:05.103532+0000","last_became_peered":"2024-03-06T07:30:05.103532+0000","last_unstale":"2024-03-06T07:30:07.128208+0000","last_undegraded":"2024-03-06T07:30:07.128208+0000","last_fullsized":"2024-03-06T07:30:07.128208+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:02:08.622876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124066+0000","last_change":"2024-03-06T07:30:06.127398+0000","last_active":"2024-03-06T07:30:07.124066+0000","last_peered":"2024-03-06T07:30:07.124066+0000","last_clean":"2024-03-06T07:30:07.124066+0000","last_became_active":"2024-03-06T07:30:06.127086+0000","last_became_peered":"2024-03-06T07:30:06.127086+0000","last_unstale":"2024-03-06T07:30:07.124066+0000","last_undegraded":"2024-03-06T07:30:07.124066+0000","last_fullsized":"2024-03-06T07:30:07.124066+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:45.737995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.127053+0000","last_change":"2024-03-06T07:30:05.106361+0000","last_active":"2024-03-06T07:30:07.127053+0000","last_peered":"2024-03-06T07:30:07.127053+0000","last_clean":"2024-03-06T07:30:07.127053+0000","last_became_active":"2024-03-06T07:30:05.106128+0000","last_became_peered":"2024-03-06T07:30:05.106128+0000","last_unstale":"2024-03-06T07:30:07.127053+0000","last_undegraded":"2024-03-06T07:30:07.127053+0000","last_fullsized":"2024-03-06T07:30:07.127053+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:52.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.127021+0000","last_change":"2024-03-06T07:30:06.127953+0000","last_active":"2024-03-06T07:30:07.127021+0000","last_peered":"2024-03-06T07:30:07.127021+0000","last_clean":"2024-03-06T07:30:07.127021+0000","last_became_active":"2024-03-06T07:30:06.127340+0000","last_became_peered":"2024-03-06T07:30:06.127340+0000","last_unstale":"2024-03-06T07:30:07.127021+0000","last_undegraded":"2024-03-06T07:30:07.127021+0000","last_fullsized":"2024-03-06T07:30:07.127021+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:54.244034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.126736+0000","last_change":"2024-03-06T07:30:05.105781+0000","last_active":"2024-03-06T07:30:07.126736+0000","last_peered":"2024-03-06T07:30:07.126736+0000","last_clean":"2024-03-06T07:30:07.126736+0000","last_became_active":"2024-03-06T07:30:05.105526+0000","last_became_peered":"2024-03-06T07:30:05.105526+0000","last_unstale":"2024-03-06T07:30:07.126736+0000","last_undegraded":"2024-03-06T07:30:07.126736+0000","last_fullsized":"2024-03-06T07:30:07.126736+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:30.220384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.330757+0000","last_change":"2024-03-06T07:30:06.127084+0000","last_active":"2024-03-06T07:30:07.330757+0000","last_peered":"2024-03-06T07:30:07.330757+0000","last_clean":"2024-03-06T07:30:07.330757+0000","last_became_active":"2024-03-06T07:30:06.126827+0000","last_became_peered":"2024-03-06T07:30:06.126827+0000","last_unstale":"2024-03-06T07:30:07.330757+0000","last_undegraded":"2024-03-06T07:30:07.330757+0000","last_fullsized":"2024-03-06T07:30:07.330757+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:31.225027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.120257+0000","last_change":"2024-03-06T07:30:05.104122+0000","last_active":"2024-03-06T07:30:06.120257+0000","last_peered":"2024-03-06T07:30:06.120257+0000","last_clean":"2024-03-06T07:30:06.120257+0000","last_became_active":"2024-03-06T07:30:05.103857+0000","last_became_peered":"2024-03-06T07:30:05.103857+0000","last_unstale":"2024-03-06T07:30:06.120257+0000","last_undegraded":"2024-03-06T07:30:06.120257+0000","last_fullsized":"2024-03-06T07:30:06.120257+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:00.519497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125350+0000","last_change":"2024-03-06T07:30:06.132705+0000","last_active":"2024-03-06T07:30:07.125350+0000","last_peered":"2024-03-06T07:30:07.125350+0000","last_clean":"2024-03-06T07:30:07.125350+0000","last_became_active":"2024-03-06T07:30:06.132416+0000","last_became_peered":"2024-03-06T07:30:06.132416+0000","last_unstale":"2024-03-06T07:30:07.125350+0000","last_undegraded":"2024-03-06T07:30:07.125350+0000","last_fullsized":"2024-03-06T07:30:07.125350+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:20.177018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129181+0000","last_change":"2024-03-06T07:30:05.103114+0000","last_active":"2024-03-06T07:30:07.129181+0000","last_peered":"2024-03-06T07:30:07.129181+0000","last_clean":"2024-03-06T07:30:07.129181+0000","last_became_active":"2024-03-06T07:30:05.102792+0000","last_became_peered":"2024-03-06T07:30:05.102792+0000","last_unstale":"2024-03-06T07:30:07.129181+0000","last_undegraded":"2024-03-06T07:30:07.129181+0000","last_fullsized":"2024-03-06T07:30:07.129181+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:54.344320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.128981+0000","last_change":"2024-03-06T07:30:06.128633+0000","last_active":"2024-03-06T07:30:07.128981+0000","last_peered":"2024-03-06T07:30:07.128981+0000","last_clean":"2024-03-06T07:30:07.128981+0000","last_became_active":"2024-03-06T07:30:06.128348+0000","last_became_peered":"2024-03-06T07:30:06.128348+0000","last_unstale":"2024-03-06T07:30:07.128981+0000","last_undegraded":"2024-03-06T07:30:07.128981+0000","last_fullsized":"2024-03-06T07:30:07.128981+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:18.935110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124726+0000","last_change":"2024-03-06T07:30:05.104636+0000","last_active":"2024-03-06T07:30:07.124726+0000","last_peered":"2024-03-06T07:30:07.124726+0000","last_clean":"2024-03-06T07:30:07.124726+0000","last_became_active":"2024-03-06T07:30:05.103695+0000","last_became_peered":"2024-03-06T07:30:05.103695+0000","last_unstale":"2024-03-06T07:30:07.124726+0000","last_undegraded":"2024-03-06T07:30:07.124726+0000","last_fullsized":"2024-03-06T07:30:07.124726+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:55.769085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.124990+0000","last_change":"2024-03-06T07:30:06.131015+0000","last_active":"2024-03-06T07:30:07.124990+0000","last_peered":"2024-03-06T07:30:07.124990+0000","last_clean":"2024-03-06T07:30:07.124990+0000","last_became_active":"2024-03-06T07:30:06.130706+0000","last_became_peered":"2024-03-06T07:30:06.130706+0000","last_unstale":"2024-03-06T07:30:07.124990+0000","last_undegraded":"2024-03-06T07:30:07.124990+0000","last_fullsized":"2024-03-06T07:30:07.124990+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:40.918933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.123724+0000","last_change":"2024-03-06T07:30:05.103465+0000","last_active":"2024-03-06T07:30:07.123724+0000","last_peered":"2024-03-06T07:30:07.123724+0000","last_clean":"2024-03-06T07:30:07.123724+0000","last_became_active":"2024-03-06T07:30:05.103211+0000","last_became_peered":"2024-03-06T07:30:05.103211+0000","last_unstale":"2024-03-06T07:30:07.123724+0000","last_undegraded":"2024-03-06T07:30:07.123724+0000","last_fullsized":"2024-03-06T07:30:07.123724+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:01.375576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-06T07:30:06.141309+0000","last_change":"2024-03-06T07:30:06.136182+0000","last_active":"2024-03-06T07:30:06.141309+0000","last_peered":"2024-03-06T07:30:06.141309+0000","last_clean":"2024-03-06T07:30:06.141309+0000","last_became_active":"2024-03-06T07:30:06.135308+0000","last_became_peered":"2024-03-06T07:30:06.135308+0000","last_unstale":"2024-03-06T07:30:06.141309+0000","last_undegraded":"2024-03-06T07:30:06.141309+0000","last_fullsized":"2024-03-06T07:30:06.141309+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:05.085500+0000","last_clean_scrub_stamp":"2024-03-06T07:30:05.085500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:03.567494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"138'3","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.125340+0000","last_change":"2024-03-06T07:30:05.105883+0000","last_active":"2024-03-06T07:30:07.125340+0000","last_peered":"2024-03-06T07:30:07.125340+0000","last_clean":"2024-03-06T07:30:07.125340+0000","last_became_active":"2024-03-06T07:30:05.105598+0000","last_became_peered":"2024-03-06T07:30:05.105598+0000","last_unstale":"2024-03-06T07:30:07.125340+0000","last_undegraded":"2024-03-06T07:30:07.125340+0000","last_fullsized":"2024-03-06T07:30:07.125340+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:04.080856+0000","last_clean_scrub_stamp":"2024-03-06T07:30:04.080856+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:21.908613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":382,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:07.129211+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:30:07.129211+0000","last_peered":"2024-03-06T07:30:07.129211+0000","last_clean":"2024-03-06T07:30:07.129211+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:30:07.129211+0000","last_undegraded":"2024-03-06T07:30:07.129211+0000","last_fullsized":"2024-03-06T07:30:07.129211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":2342912,"data_stored":1353131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28732,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343108,"statfs":{"total":96636764160,"available":96607342592,"internally_reserved":0,"allocated":1753088,"data_stored":762257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45364,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326476,"statfs":{"total":96636764160,"available":96590311424,"internally_reserved":0,"allocated":1744896,"data_stored":762133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574949,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28708,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343132,"statfs":{"total":96636764160,"available":96607367168,"internally_reserved":0,"allocated":1728512,"data_stored":755811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41200,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330640,"statfs":{"total":96636764160,"available":96594575360,"internally_reserved":0,"allocated":1732608,"data_stored":755901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41212,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330628,"statfs":{"total":96636764160,"available":96594563072,"internally_reserved":0,"allocated":1744896,"data_stored":757493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41200,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330640,"statfs":{"total":96636764160,"available":96594575360,"internally_reserved":0,"allocated":1732608,"data_stored":756393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41792,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330048,"statfs":{"total":96636764160,"available":96593969152,"internally_reserved":0,"allocated":2338816,"data_stored":1347825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:09.474 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:30:09.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:30:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.615+0000 7f2a7a1ac640 1 Processor -- start 2024-03-06T07:30:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.615+0000 7f2a7a1ac640 1 -- start start 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.615+0000 7f2a7a1ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a740d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.615+0000 7f2a7a1ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a740d1910 0x7f2a740d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.615+0000 7f2a7a1ac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a740d2740 con 0x7f2a740d22b0 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.615+0000 7f2a7a1ac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a741618b0 con 0x7f2a740d1910 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.615+0000 7f2a7a1ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740cf400 con 0x7f2a741614d0 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a6affd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a740d1910 0x7f2a740d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a6affd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a740d1910 0x7f2a740d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50012/0 (socket says 172.21.15.77:50012) 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a6affd640 1 -- 172.21.15.77:0/654764714 learned_addr learned my addr 172.21.15.77:0/654764714 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a737fe640 1 --2- 172.21.15.77:0/654764714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a740d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a6affd640 1 -- 172.21.15.77:0/654764714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 msgr2=0x7f2a740d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a6affd640 1 --2- 172.21.15.77:0/654764714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a740d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a6affd640 1 -- 172.21.15.77:0/654764714 >> v1:172.21.15.158:6789/0 conn(0x7f2a740d22b0 legacy=0x7f2a740d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a6affd640 1 -- 172.21.15.77:0/654764714 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a740cf6a0 con 0x7f2a740d1910 2024-03-06T07:30:09.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.616+0000 7f2a737fe640 1 --2- 172.21.15.77:0/654764714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a740d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:09.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a6affd640 1 --2- 172.21.15.77:0/654764714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d1910 0x7f2a740d1d70 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009870 tx=0x7f2a5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95c353303e3e930 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a72ffd640 1 -- 172.21.15.77:0/654764714 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c035020 con 0x7f2a740d1910 2024-03-06T07:30:09.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a72ffd640 1 -- 172.21.15.77:0/654764714 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5c031640 con 0x7f2a740d1910 2024-03-06T07:30:09.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a72ffd640 1 -- 172.21.15.77:0/654764714 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c02d040 con 0x7f2a740d1910 2024-03-06T07:30:09.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/654764714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d1910 msgr2=0x7f2a740d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/654764714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d1910 0x7f2a740d1d70 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009870 tx=0x7f2a5c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/654764714 shutdown_connections 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/654764714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d1910 0x7f2a740d1d70 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/654764714 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a740d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/654764714 >> 172.21.15.77:0/654764714 conn(0x7f2a740d3d40 msgr2=0x7f2a740d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/654764714 shutdown_connections 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.617+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/654764714 wait complete. 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.618+0000 7f2a7a1ac640 1 Processor -- start 2024-03-06T07:30:09.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.619+0000 7f2a7a1ac640 1 -- start start 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.619+0000 7f2a7a1ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a740d1910 0x7f2a74150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.619+0000 7f2a7a1ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d22b0 0x7f2a7406b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.619+0000 7f2a7a1ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a7406bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.619+0000 7f2a7a1ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740b0910 con 0x7f2a740d22b0 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.619+0000 7f2a7a1ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740cfc60 con 0x7f2a740d1910 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.619+0000 7f2a7a1ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740b1e40 con 0x7f2a741614d0 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.619+0000 7f2a6affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d22b0 0x7f2a7406b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a6affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d22b0 0x7f2a7406b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50022/0 (socket says 172.21.15.77:50022) 2024-03-06T07:30:09.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a6affd640 1 -- 172.21.15.77:0/2422051384 learned_addr learned my addr 172.21.15.77:0/2422051384 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a737fe640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a740d1910 0x7f2a74150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a73fff640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a7406bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a6affd640 1 -- 172.21.15.77:0/2422051384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 msgr2=0x7f2a7406bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a6affd640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a7406bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a6affd640 1 -- 172.21.15.77:0/2422051384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a740d1910 msgr2=0x7f2a74150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a6affd640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a740d1910 0x7f2a74150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a6affd640 1 -- 172.21.15.77:0/2422051384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a740ca560 con 0x7f2a740d22b0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a6affd640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d22b0 0x7f2a7406b960 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c02b930 tx=0x7f2a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a70ff9640 1 -- 172.21.15.77:0/2422051384 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c040070 con 0x7f2a740d22b0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a70ff9640 1 -- 172.21.15.77:0/2422051384 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5c039c10 con 0x7f2a740d22b0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a70ff9640 1 -- 172.21.15.77:0/2422051384 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c035040 con 0x7f2a740d22b0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a74150c10 con 0x7f2a740d22b0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.620+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a741511d0 con 0x7f2a740d22b0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.622+0000 7f2a70ff9640 1 -- 172.21.15.77:0/2422051384 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a5c038020 con 0x7f2a740d22b0 2024-03-06T07:30:09.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.622+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a741513a0 con 0x7f2a740d22b0 2024-03-06T07:30:09.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.622+0000 7f2a70ff9640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a5406b4d0 0x7f2a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.622+0000 7f2a737fe640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a5406b4d0 0x7f2a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.622+0000 7f2a70ff9640 1 -- 172.21.15.77:0/2422051384 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a5c0abc40 con 0x7f2a740d22b0 2024-03-06T07:30:09.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.623+0000 7f2a737fe640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a5406b4d0 0x7f2a5406d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400aac0 tx=0x7f2a64008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:09.626+0000 7f2a70ff9640 1 -- 172.21.15.77:0/2422051384 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a740d2740 con 0x7f2a740d22b0 2024-03-06T07:30:10.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.027+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2a740ce170 con 0x7f2a740d22b0 2024-03-06T07:30:10.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.029+0000 7f2a70ff9640 1 -- 172.21.15.77:0/2422051384 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2a5c07a500 con 0x7f2a740d22b0 2024-03-06T07:30:10.031 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-06T07:30:10.031 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBlGehlBPTWKRAAq+3YYLlIkcphkMrxigDMLA== 2024-03-06T07:30:10.031 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2024-03-06T07:30:10.031 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2024-03-06T07:30:10.031 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:10.031 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2024-03-06T07:30:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a5406b4d0 msgr2=0x7f2a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a5406b4d0 0x7f2a5406d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400aac0 tx=0x7f2a64008040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d22b0 msgr2=0x7f2a7406b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d22b0 0x7f2a7406b960 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c02b930 tx=0x7f2a5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 shutdown_connections 2024-03-06T07:30:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2a5406b4d0 0x7f2a5406d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a740d22b0 0x7f2a7406b960 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2a741614d0 0x7f2a7406bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 --2- 172.21.15.77:0/2422051384 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2a740d1910 0x7f2a74150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.031+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 >> 172.21.15.77:0/2422051384 conn(0x7f2a740d3d40 msgr2=0x7f2a740b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.032+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 shutdown_connections 2024-03-06T07:30:10.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.032+0000 7f2a7a1ac640 1 -- 172.21.15.77:0/2422051384 wait complete. 2024-03-06T07:30:10.044 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.144+0000 7f42ad2df640 1 Processor -- start 2024-03-06T07:30:10.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.145+0000 7f42ad2df640 1 -- start start 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.145+0000 7f42ad2df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f42a80d00c0 0x7f42a80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.145+0000 7f42ad2df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a806cf20 0x7f42a806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.145+0000 7f42ad2df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a80d09e0 con 0x7f42a80cf850 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.145+0000 7f42ad2df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a806ca90 con 0x7f42a80d00c0 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.145+0000 7f42ad2df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a815e380 con 0x7f42a806cf20 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.145+0000 7f42a6ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f42a80d00c0 0x7f42a80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.146+0000 7f42a6ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f42a80d00c0 0x7f42a80d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50032/0 (socket says 172.21.15.77:50032) 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.146+0000 7f42a6ffd640 1 -- 172.21.15.77:0/517120201 learned_addr learned my addr 172.21.15.77:0/517120201 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.146+0000 7f42a6ffd640 1 -- 172.21.15.77:0/517120201 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a806cf20 msgr2=0x7f42a806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.146+0000 7f42a6ffd640 1 --2- 172.21.15.77:0/517120201 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a806cf20 0x7f42a806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.146+0000 7f42a6ffd640 1 -- 172.21.15.77:0/517120201 >> v1:172.21.15.158:6789/0 conn(0x7f42a80cf850 legacy=0x7f42a80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.146+0000 7f42a6ffd640 1 -- 172.21.15.77:0/517120201 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a80cdc10 con 0x7f42a80d00c0 2024-03-06T07:30:10.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.147+0000 7f42a6ffd640 1 --2- 172.21.15.77:0/517120201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80d00c0 0x7f42a80d04a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009e50 tx=0x7f42a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12c81347cff0032 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.147+0000 7f42a5ffb640 1 -- 172.21.15.77:0/517120201 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a002ca70 con 0x7f42a80d00c0 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.147+0000 7f42a5ffb640 1 -- 172.21.15.77:0/517120201 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a002cc10 con 0x7f42a80d00c0 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.147+0000 7f42a5ffb640 1 -- 172.21.15.77:0/517120201 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00343f0 con 0x7f42a80d00c0 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.148+0000 7f42ad2df640 1 -- 172.21.15.77:0/517120201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80d00c0 msgr2=0x7f42a80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.148+0000 7f42ad2df640 1 --2- 172.21.15.77:0/517120201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80d00c0 0x7f42a80d04a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009e50 tx=0x7f42a002bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.148+0000 7f42ad2df640 1 -- 172.21.15.77:0/517120201 shutdown_connections 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.148+0000 7f42ad2df640 1 --2- 172.21.15.77:0/517120201 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a806cf20 0x7f42a806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.148+0000 7f42ad2df640 1 --2- 172.21.15.77:0/517120201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80d00c0 0x7f42a80d04a0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.148+0000 7f42ad2df640 1 -- 172.21.15.77:0/517120201 >> 172.21.15.77:0/517120201 conn(0x7f42a80b35a0 msgr2=0x7f42a80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.148+0000 7f42ad2df640 1 -- 172.21.15.77:0/517120201 shutdown_connections 2024-03-06T07:30:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.148+0000 7f42ad2df640 1 -- 172.21.15.77:0/517120201 wait complete. 2024-03-06T07:30:10.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.149+0000 7f42ad2df640 1 Processor -- start 2024-03-06T07:30:10.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.149+0000 7f42ad2df640 1 -- start start 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42ad2df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a806cf20 0x7f42a8155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42ad2df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80cf850 0x7f42a81f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42ad2df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a80d00c0 0x7f42a80b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42ad2df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a806c720 con 0x7f42a80cf850 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42ad2df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a806c440 con 0x7f42a806cf20 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42ad2df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80b1030 con 0x7f42a80d00c0 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80cf850 0x7f42a81f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80cf850 0x7f42a81f7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50044/0 (socket says 172.21.15.77:50044) 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 -- 172.21.15.77:0/1195146408 learned_addr learned my addr 172.21.15.77:0/1195146408 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a6ffd640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a806cf20 0x7f42a8155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a77fe640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a80d00c0 0x7f42a80b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 -- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a80d00c0 msgr2=0x7f42a80b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a80d00c0 0x7f42a80b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 -- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a806cf20 msgr2=0x7f42a8155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a806cf20 0x7f42a8155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 -- 172.21.15.77:0/1195146408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a81f7b70 con 0x7f42a80cf850 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a6ffd640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a806cf20 0x7f42a8155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a77fe640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a80d00c0 0x7f42a80b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42a67fc640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80cf850 0x7f42a81f7200 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f429000c410 tx=0x7f429000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f429f7fe640 1 -- 172.21.15.77:0/1195146408 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4290017070 con 0x7f42a80cf850 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f429f7fe640 1 -- 172.21.15.77:0/1195146408 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42900035f0 con 0x7f42a80cf850 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f429f7fe640 1 -- 172.21.15.77:0/1195146408 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4290003a20 con 0x7f42a80cf850 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.150+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a81f7e50 con 0x7f42a80cf850 2024-03-06T07:30:10.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.151+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a8160040 con 0x7f42a80cf850 2024-03-06T07:30:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.152+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a8068150 con 0x7f42a80cf850 2024-03-06T07:30:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.152+0000 7f429f7fe640 1 -- 172.21.15.77:0/1195146408 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4290003bc0 con 0x7f42a80cf850 2024-03-06T07:30:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.152+0000 7f429f7fe640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f428806b500 0x7f428806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.152+0000 7f42a6ffd640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f428806b500 0x7f428806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.152+0000 7f429f7fe640 1 -- 172.21.15.77:0/1195146408 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4290089510 con 0x7f42a80cf850 2024-03-06T07:30:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.153+0000 7f42a6ffd640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f428806b500 0x7f428806d9c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f42a0035000 tx=0x7f42a0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.161+0000 7f429f7fe640 1 -- 172.21.15.77:0/1195146408 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42a8068150 con 0x7f42a80cf850 2024-03-06T07:30:10.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.370+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42a8160210 con 0x7f42a80cf850 2024-03-06T07:30:10.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.370+0000 7f429f7fe640 1 -- 172.21.15.77:0/1195146408 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f4290058390 con 0x7f42a80cf850 2024-03-06T07:30:10.372 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:10.372 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":97,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2532910127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2532910127},{"type":"v1","addr":"172.21.15.158:6835","nonce":2532910127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":8347,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/3282592218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3282592218},{"type":"v1","addr":"172.21.15.158:6833","nonce":3282592218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":9177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:07.467677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:30:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 97 2024-03-06T07:30:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f428806b500 msgr2=0x7f428806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f428806b500 0x7f428806d9c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f42a0035000 tx=0x7f42a0030000 comp rx=0 tx=0).stop 2024-03-06T07:30:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80cf850 msgr2=0x7f42a81f7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80cf850 0x7f42a81f7200 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f429000c410 tx=0x7f429000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 shutdown_connections 2024-03-06T07:30:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f428806b500 0x7f428806d9c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a80cf850 0x7f42a81f7200 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f42a80d00c0 0x7f42a80b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 --2- 172.21.15.77:0/1195146408 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f42a806cf20 0x7f42a8155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 >> 172.21.15.77:0/1195146408 conn(0x7f42a80b35a0 msgr2=0x7f42a80b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.372+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 shutdown_connections 2024-03-06T07:30:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.373+0000 7f42ad2df640 1 -- 172.21.15.77:0/1195146408 wait complete. 2024-03-06T07:30:10.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:07.467677+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9186}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:30:10.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:30:10.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:30:10.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.892+0000 7f6e9db01640 1 Processor -- start 2024-03-06T07:30:10.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.892+0000 7f6e9db01640 1 -- start start 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e9db01640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e98162970 0x7f6e980d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e9db01640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980d22c0 con 0x7f6e98161cf0 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e9db01640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980d25a0 con 0x7f6e980b4460 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e9db01640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e98161b00 con 0x7f6e98162970 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e97fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6e98161cf0 0x7f6e980cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58966/0 (socket says 172.21.15.77:58966) 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e97fff640 1 -- 172.21.15.77:0/1629404068 learned_addr learned my addr 172.21.15.77:0/1629404068 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e8effd640 1 --2- 172.21.15.77:0/1629404068 >> v2:172.21.15.158:3300/0 conn(0x7f6e98162970 0x7f6e980d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285634583 0 0) 0x7f6e980d22c0 con 0x7f6e98161cf0 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.893+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84003520 con 0x7f6e98161cf0 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1961846975 0 0) 0x7f6e84003520 con 0x7f6e98161cf0 2024-03-06T07:30:10.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 >> v2:172.21.15.158:3300/0 conn(0x7f6e98162970 msgr2=0x7f6e980d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 --2- 172.21.15.77:0/1629404068 >> v2:172.21.15.158:3300/0 conn(0x7f6e98162970 0x7f6e980d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 >> v1:172.21.15.158:6789/0 conn(0x7f6e980b4460 legacy=0x7f6e980b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e980d2880 con 0x7f6e98161cf0 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f6e900019f0 con 0x7f6e98161cf0 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 >> v1:172.21.15.77:6789/0 conn(0x7f6e98161cf0 legacy=0x7f6e980cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e84004800 0x7f6e84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840051c0 0x7f6e8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e98161b00 con 0x7f6e840039b0 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400c060 con 0x7f6e84004800 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8400c260 con 0x7f6e840051c0 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e977fe640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e8effd640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e84004800 0x7f6e84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e97fff640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840051c0 0x7f6e8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e977fe640 1 -- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840051c0 msgr2=0x7f6e8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e977fe640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840051c0 0x7f6e8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e977fe640 1 -- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e84004800 msgr2=0x7f6e84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e977fe640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e84004800 0x7f6e84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e977fe640 1 -- 172.21.15.77:0/1629404068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84003520 con 0x7f6e840039b0 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e97fff640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e840051c0 0x7f6e8400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.894+0000 7f6e97fff640 1 -- 172.21.15.77:0/1629404068 reap_dead start 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.895+0000 7f6e977fe640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e84003dd0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001750 tx=0x7f6e84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cda09a409f666b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.895+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88001370 con 0x7f6e840039b0 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.895+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e880052d0 con 0x7f6e840039b0 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.895+0000 7f6e96ffd640 1 -- 172.21.15.77:0/1629404068 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88003830 con 0x7f6e840039b0 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.896+0000 7f6e9db01640 1 -- 172.21.15.77:0/1629404068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 msgr2=0x7f6e84003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.896+0000 7f6e9db01640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e84003dd0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001750 tx=0x7f6e84005830 comp rx=0 tx=0).stop 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.896+0000 7f6e9db01640 1 -- 172.21.15.77:0/1629404068 shutdown_connections 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.896+0000 7f6e9db01640 1 --2- 172.21.15.77:0/1629404068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e84003dd0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.896+0000 7f6e9db01640 1 -- 172.21.15.77:0/1629404068 >> 172.21.15.77:0/1629404068 conn(0x7f6e980d3ce0 msgr2=0x7f6e980d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.896+0000 7f6e9db01640 1 -- 172.21.15.77:0/1629404068 shutdown_connections 2024-03-06T07:30:10.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.896+0000 7f6e9db01640 1 -- 172.21.15.77:0/1629404068 wait complete. 2024-03-06T07:30:10.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.898+0000 7f6e9db01640 1 Processor -- start 2024-03-06T07:30:10.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.898+0000 7f6e9db01640 1 -- start start 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e9db01640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e98155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e9db01640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e980b7d20 0x7f6e980b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e9db01640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e980b89c0 0x7f6e980b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e9db01640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980ce950 con 0x7f6e840039b0 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e9db01640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980cecc0 con 0x7f6e980b89c0 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e9db01640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980b4840 con 0x7f6e980b7d20 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e98155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e98155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50068/0 (socket says 172.21.15.77:50068) 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e977fe640 1 -- 172.21.15.77:0/2214984449 learned_addr learned my addr 172.21.15.77:0/2214984449 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e97fff640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e980b89c0 0x7f6e980b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e8effd640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e980b7d20 0x7f6e980b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e977fe640 1 -- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e980b7d20 msgr2=0x7f6e980b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e977fe640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e980b7d20 0x7f6e980b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e977fe640 1 -- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e980b89c0 msgr2=0x7f6e980b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e977fe640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e980b89c0 0x7f6e980b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e977fe640 1 -- 172.21.15.77:0/2214984449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e980b6ca0 con 0x7f6e840039b0 2024-03-06T07:30:10.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.899+0000 7f6e97fff640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e980b89c0 0x7f6e980b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.900+0000 7f6e977fe640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e98155dc0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f6e88005040 tx=0x7f6e88001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.900+0000 7f6e94ff9640 1 -- 172.21.15.77:0/2214984449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88041070 con 0x7f6e840039b0 2024-03-06T07:30:10.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.900+0000 7f6e94ff9640 1 -- 172.21.15.77:0/2214984449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e88005480 con 0x7f6e840039b0 2024-03-06T07:30:10.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.900+0000 7f6e9db01640 1 -- 172.21.15.77:0/2214984449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e980b6f80 con 0x7f6e840039b0 2024-03-06T07:30:10.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.900+0000 7f6e94ff9640 1 -- 172.21.15.77:0/2214984449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88007a30 con 0x7f6e840039b0 2024-03-06T07:30:10.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.900+0000 7f6e9db01640 1 -- 172.21.15.77:0/2214984449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e980b7820 con 0x7f6e840039b0 2024-03-06T07:30:10.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.901+0000 7f6e94ff9640 1 -- 172.21.15.77:0/2214984449 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e88044070 con 0x7f6e840039b0 2024-03-06T07:30:10.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.902+0000 7f6e94ff9640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7806b500 0x7f6e7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.902+0000 7f6e8effd640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7806b500 0x7f6e7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.902+0000 7f6e94ff9640 1 -- 172.21.15.77:0/2214984449 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e880abd40 con 0x7f6e840039b0 2024-03-06T07:30:10.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.902+0000 7f6e8effd640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7806b500 0x7f6e7806d9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6e980b8120 tx=0x7f6e80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.903+0000 7f6ea07bd740 1 Processor -- start 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.903+0000 7f6ea07bd740 1 -- start start 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812cb870 0x5623812cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812cf8c0 0x5623812d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623812356b0 con 0x5623812cf8c0 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623811dea60 con 0x5623812cb870 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x562381231750 con 0x5623812fb600 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812cf8c0 0x5623812d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812cf8c0 0x5623812d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50082/0 (socket says 172.21.15.77:50082) 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 -- 172.21.15.77:0/1353611576 learned_addr learned my addr 172.21.15.77:0/1353611576 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 -- 172.21.15.77:0/1353611576 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 msgr2=0x5623812fba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 --2- 172.21.15.77:0/1353611576 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 -- 172.21.15.77:0/1353611576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812cb870 msgr2=0x5623812cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e977fe640 1 --2- 172.21.15.77:0/1353611576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812cb870 0x5623812cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 --2- 172.21.15.77:0/1353611576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812cb870 0x5623812cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 -- 172.21.15.77:0/1353611576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623811fcf30 con 0x5623812cf8c0 2024-03-06T07:30:10.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8effd640 1 --2- 172.21.15.77:0/1353611576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812cf8c0 0x5623812d1360 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6e80014db0 tx=0x7f6e80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1912cf3adf7b736d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8d7fa640 1 -- 172.21.15.77:0/1353611576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80027070 con 0x5623812cf8c0 2024-03-06T07:30:10.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8d7fa640 1 -- 172.21.15.77:0/1353611576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e800096f0 con 0x5623812cf8c0 2024-03-06T07:30:10.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.904+0000 7f6e8d7fa640 1 -- 172.21.15.77:0/1353611576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80023670 con 0x5623812cf8c0 2024-03-06T07:30:10.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 -- 172.21.15.77:0/1353611576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812cf8c0 msgr2=0x5623812d1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/1353611576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812cf8c0 0x5623812d1360 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6e80014db0 tx=0x7f6e80017000 comp rx=0 tx=0).stop 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 -- 172.21.15.77:0/1353611576 shutdown_connections 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/1353611576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812cf8c0 0x5623812d1360 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/1353611576 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/1353611576 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812cb870 0x5623812cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 -- 172.21.15.77:0/1353611576 >> 172.21.15.77:0/1353611576 conn(0x5623812cd8f0 msgr2=0x5623812cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 -- 172.21.15.77:0/1353611576 shutdown_connections 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.905+0000 7f6ea07bd740 1 -- 172.21.15.77:0/1353611576 wait complete. 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.906+0000 7f6ea07bd740 1 -- start start 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.906+0000 7f6ea07bd740 1 Processor -- start 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.906+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.906+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x562381313050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.906+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623813146c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.906+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623812cfd50 con 0x562381313590 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.906+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623811f1ea0 con 0x5623812f5ef0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.906+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5623812d0030 con 0x5623812fb600 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6ea07bd740 10 client.0 _ll_register_callbacks cb 0x5623811f1870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623813146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623813146c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50084/0 (socket says 172.21.15.77:50084) 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 -- 172.21.15.77:0/927567340 learned_addr learned my addr 172.21.15.77:0/927567340 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e8effd640 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x562381313050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e977fe640 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 -- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 msgr2=0x5623812fc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 -- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 msgr2=0x562381313050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x562381313050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 -- 172.21.15.77:0/927567340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623811f2270 con 0x562381313590 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e8effd640 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x562381313050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e97fff640 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623813146c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6e980c7610 tx=0x7f6e90007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1970d0701545ec1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e767fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e767fc640 1 -- 172.21.15.77:0/927567340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90018020 con 0x562381313590 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e767fc640 1 -- 172.21.15.77:0/927567340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e90014810 con 0x562381313590 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6ea07bd740 1 -- 172.21.15.77:0/927567340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562381242bf0 con 0x562381313590 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e767fc640 1 -- 172.21.15.77:0/927567340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e90022050 con 0x562381313590 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6ea07bd740 10 client.9324 fetch_fsmap learned FSMap version 97 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6ea07bd740 1 -- 172.21.15.77:0/927567340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x562380f4a2d0 con 0x562381313590 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.907+0000 7f6e767fc640 1 -- 172.21.15.77:0/927567340 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562381242bf0 con 0x562381313590 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6e767fc640 1 -- 172.21.15.77:0/927567340 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 97) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6e90025020 con 0x562381313590 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6ea07bd740 10 client.9324 fetch_fsmap finished waiting for FSMap version 97 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6ea07bd740 10 client.9324 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6ea07bd740 20 client.9324 populate_metadata read hostname 'smithi077' 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] conn(0x5623812fedc0 0x562381335880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6ea07bd740 4 client.9324 mds_command: new command op to 9186 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6ea07bd740 1 -- 172.21.15.77:0/927567340 --> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56238123cf40 con 0x5623812fedc0 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6e977fe640 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] conn(0x5623812fedc0 0x562381335880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6e767fc640 10 client.9324 ms_handle_connect on v2:172.21.15.77:6836/2581202217 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.908+0000 7f6e977fe640 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] conn(0x5623812fedc0 0x562381335880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c63e23df9a5518f1 server_cookie=3cc8657c1ddccd23 in_seq=0 out_seq=0 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6e767fc640 1 -- 172.21.15.77:0/927567340 <== mds.0 v2:172.21.15.77:6836/2581202217 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56238123cf40 con 0x5623812fedc0 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6e767fc640 10 client.9324 handle_command_reply: tid=0 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 client.9324 shutdown 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 -- 172.21.15.77:0/927567340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 msgr2=0x5623813146c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623813146c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6e980c7610 tx=0x7f6e90007710 comp rx=0 tx=0).stop 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 -- 172.21.15.77:0/927567340 shutdown_connections 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] conn(0x5623812fedc0 0x562381335880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623813146c0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x562381313050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/927567340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 -- 172.21.15.77:0/927567340 >> 172.21.15.77:0/927567340 conn(0x5623812cb870 msgr2=0x5623812ca510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6e767fc640 0 client.9324 ms_handle_reset on v2:172.21.15.77:6836/2581202217 2024-03-06T07:30:10.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 -- 172.21.15.77:0/927567340 shutdown_connections 2024-03-06T07:30:10.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 1 -- 172.21.15.77:0/927567340 wait complete. 2024-03-06T07:30:10.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.909+0000 7f6ea07bd740 20 client.9324 trim_cache size 0 max 16384 2024-03-06T07:30:10.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.925+0000 7f6ea07bd740 1 Processor -- start 2024-03-06T07:30:10.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6ea07bd740 1 -- start start 2024-03-06T07:30:10.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812f5ef0 0x5623812ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562381313590 0x5623812faad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980cbba0 con 0x5623812f5ef0 2024-03-06T07:30:10.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9806c7a0 con 0x562381313590 2024-03-06T07:30:10.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980cb5a0 con 0x5623812fb600 2024-03-06T07:30:10.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6e977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812f5ef0 0x5623812ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6e977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812f5ef0 0x5623812ca700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50098/0 (socket says 172.21.15.77:50098) 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.926+0000 7f6e977fe640 1 -- 172.21.15.77:0/472050037 learned_addr learned my addr 172.21.15.77:0/472050037 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e8effd640 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e977fe640 1 -- 172.21.15.77:0/472050037 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 msgr2=0x5623812fa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e977fe640 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fa590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e977fe640 1 -- 172.21.15.77:0/472050037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562381313590 msgr2=0x5623812faad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e97fff640 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562381313590 0x5623812faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e977fe640 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562381313590 0x5623812faad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e977fe640 1 -- 172.21.15.77:0/472050037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623810146b0 con 0x5623812f5ef0 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e977fe640 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812f5ef0 0x5623812ca700 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6e8802bcd0 tx=0x7f6e88008fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bdfd51cf8c2c5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e8cff9640 1 -- 172.21.15.77:0/472050037 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880a8070 con 0x5623812f5ef0 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e8cff9640 1 -- 172.21.15.77:0/472050037 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e88006e10 con 0x5623812f5ef0 2024-03-06T07:30:10.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.927+0000 7f6e8cff9640 1 -- 172.21.15.77:0/472050037 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8804b040 con 0x5623812f5ef0 2024-03-06T07:30:10.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 -- 172.21.15.77:0/472050037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812f5ef0 msgr2=0x5623812ca700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812f5ef0 0x5623812ca700 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6e8802bcd0 tx=0x7f6e88008fb0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 -- 172.21.15.77:0/472050037 shutdown_connections 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x562381313590 0x5623812faad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/472050037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623812f5ef0 0x5623812ca700 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 -- 172.21.15.77:0/472050037 >> 172.21.15.77:0/472050037 conn(0x5623812cb870 msgr2=0x5623812c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 -- 172.21.15.77:0/472050037 shutdown_connections 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 -- 172.21.15.77:0/472050037 wait complete. 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.928+0000 7f6ea07bd740 1 -- start start 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6ea07bd740 1 Processor -- start 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x5623813860f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6ea07bd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623812faad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x562381311fd0 con 0x562381313590 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x562381311770 con 0x5623812f5ef0 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6ea07bd740 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x5623813155c0 con 0x5623812fb600 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6ea07bd740 10 client.0 _ll_register_callbacks cb 0x7f6e90001940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623812faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623812faad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50110/0 (socket says 172.21.15.77:50110) 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6e97fff640 1 -- 172.21.15.77:0/4220841592 learned_addr learned my addr 172.21.15.77:0/4220841592 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.929+0000 7f6e977fe640 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x5623813860f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e8effd640 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e97fff640 1 -- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 msgr2=0x5623812fa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e97fff640 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fa590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e97fff640 1 -- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 msgr2=0x5623813860f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e97fff640 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x5623813860f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e97fff640 1 -- 172.21.15.77:0/4220841592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56238132b220 con 0x562381313590 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e977fe640 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x5623813860f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e97fff640 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623812faad0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f6e90014fb0 tx=0x7f6e90013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d7af3ce30797fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e767fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e767fc640 1 -- 172.21.15.77:0/4220841592 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9001d650 con 0x562381313590 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e767fc640 1 -- 172.21.15.77:0/4220841592 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e9001d7f0 con 0x562381313590 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 1 -- 172.21.15.77:0/4220841592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e88006e10 con 0x562381313590 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e767fc640 1 -- 172.21.15.77:0/4220841592 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e9001da10 con 0x562381313590 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 10 client.9330 fetch_fsmap learned FSMap version 97 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 1 -- 172.21.15.77:0/4220841592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x562381387aa0 con 0x562381313590 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e767fc640 1 -- 172.21.15.77:0/4220841592 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e88006e10 con 0x562381313590 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6e767fc640 1 -- 172.21.15.77:0/4220841592 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 97) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6e90038080 con 0x562381313590 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 10 client.9330 fetch_fsmap finished waiting for FSMap version 97 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 10 client.9330 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 20 client.9330 populate_metadata read hostname 'smithi077' 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] conn(0x5623812fc5e0 0x562381386a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 4 client.9330 mds_command: new command op to 9186 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:30:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.930+0000 7f6ea07bd740 1 -- 172.21.15.77:0/4220841592 --> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5623811bb9a0 con 0x5623812fc5e0 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6e977fe640 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] conn(0x5623812fc5e0 0x562381386a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6e977fe640 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] conn(0x5623812fc5e0 0x562381386a20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=266a3b957733646b server_cookie=74b25e1201b42726 in_seq=0 out_seq=0 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6e767fc640 10 client.9330 ms_handle_connect on v2:172.21.15.77:6836/2581202217 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6e767fc640 1 -- 172.21.15.77:0/4220841592 <== mds.0 v2:172.21.15.77:6836/2581202217 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5623811bb9a0 con 0x5623812fc5e0 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6e767fc640 10 client.9330 handle_command_reply: tid=0 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 client.9330 shutdown 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 -- 172.21.15.77:0/4220841592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 msgr2=0x5623812faad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623812faad0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f6e90014fb0 tx=0x7f6e90013000 comp rx=0 tx=0).stop 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 -- 172.21.15.77:0/4220841592 shutdown_connections 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.77:6836/2581202217,v1:172.21.15.77:6837/2581202217] conn(0x5623812fc5e0 0x562381386a20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562381313590 0x5623812faad0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x5623812fb600 0x5623812fa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 --2- 172.21.15.77:0/4220841592 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5623812f5ef0 0x5623813860f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6e767fc640 0 client.9330 ms_handle_reset on v2:172.21.15.77:6836/2581202217 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 -- 172.21.15.77:0/4220841592 >> 172.21.15.77:0/4220841592 conn(0x5623812cb870 msgr2=0x562381316750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 -- 172.21.15.77:0/4220841592 shutdown_connections 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 1 -- 172.21.15.77:0/4220841592 wait complete. 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.931+0000 7f6ea07bd740 20 client.9330 trim_cache size 0 max 16384 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "eee1dc2e-9e65-49b7-aabb-947580759696", 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 9186, 2024-03-06T07:30:10.940 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-06T07:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-06T07:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-06T07:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 4.8120324349999999, 2024-03-06T07:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 97, 2024-03-06T07:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 139, 2024-03-06T07:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 139, 2024-03-06T07:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 10.167334572 2024-03-06T07:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 -- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7806b500 msgr2=0x7f6e7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7806b500 0x7f6e7806d9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6e980b8120 tx=0x7f6e80008040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 -- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 msgr2=0x7f6e98155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e98155dc0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f6e88005040 tx=0x7f6e88001120 comp rx=0 tx=0).stop 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 -- 172.21.15.77:0/2214984449 shutdown_connections 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e980b89c0 0x7f6e980b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e980b7d20 0x7f6e980b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e7806b500 0x7f6e7806d9c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 --2- 172.21.15.77:0/2214984449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e840039b0 0x7f6e98155dc0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 -- 172.21.15.77:0/2214984449 >> 172.21.15.77:0/2214984449 conn(0x7f6e980d3ce0 msgr2=0x7f6e9806cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 -- 172.21.15.77:0/2214984449 shutdown_connections 2024-03-06T07:30:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:10.932+0000 7f6e9db01640 1 -- 172.21.15.77:0/2214984449 wait complete. 2024-03-06T07:30:10.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.050+0000 7f10e0bc7640 1 Processor -- start 2024-03-06T07:30:11.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.050+0000 7f10e0bc7640 1 -- start start 2024-03-06T07:30:11.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10e0bc7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f10dc0cf790 0x7f10dc0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10e0bc7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 0x7f10dc071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10e0bc7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10dc0ae240 con 0x7f10dc072220 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10e0bc7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc06cf70 con 0x7f10dc0cf790 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10e0bc7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0ae520 con 0x7f10dc0adda0 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10dad9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f10dc072220 0x7f10dc1619b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58968/0 (socket says 172.21.15.77:58968) 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10dad9d640 1 -- 172.21.15.77:0/2818958070 learned_addr learned my addr 172.21.15.77:0/2818958070 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10da59c640 1 --2- 172.21.15.77:0/2818958070 >> v2:172.21.15.158:3300/0 conn(0x7f10dc0cf790 0x7f10dc0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10d9d9b640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 0x7f10dc071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438169153 0 0) 0x7f10dc0ae240 con 0x7f10dc072220 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.051+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c4003520 con 0x7f10dc072220 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754740656 0 0) 0x7f10c4003520 con 0x7f10dc072220 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 msgr2=0x7f10dc071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 0x7f10dc071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 >> v2:172.21.15.158:3300/0 conn(0x7f10dc0cf790 msgr2=0x7f10dc0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 --2- 172.21.15.77:0/2818958070 >> v2:172.21.15.158:3300/0 conn(0x7f10dc0cf790 0x7f10dc0ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc0ae800 con 0x7f10dc072220 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f10cc0019f0 con 0x7f10dc072220 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 >> v1:172.21.15.77:6789/0 conn(0x7f10dc072220 legacy=0x7f10dc1619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10c40039b0 0x7f10c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 0x7f10c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c4004d40 0x7f10c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc06cf70 con 0x7f10c40039b0 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0ae520 con 0x7f10c4004d40 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c400b9c0 con 0x7f10dc0adda0 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10da59c640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10c40039b0 0x7f10c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d9d9b640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 0x7f10c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10dad9d640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c4004d40 0x7f10c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10da59c640 1 -- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 msgr2=0x7f10c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10da59c640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 0x7f10c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10da59c640 1 -- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c4004d40 msgr2=0x7f10c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10da59c640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10c4004d40 0x7f10c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10da59c640 1 -- 172.21.15.77:0/2818958070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c4003520 con 0x7f10c40039b0 2024-03-06T07:30:11.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.052+0000 7f10d9d9b640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10dc0adda0 0x7f10c4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:11.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.053+0000 7f10da59c640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10c40039b0 0x7f10c4003dd0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009ac0 tx=0x7f10c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabd6bf1a9017ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.053+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4039070 con 0x7f10c40039b0 2024-03-06T07:30:11.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.053+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d402c710 con 0x7f10c40039b0 2024-03-06T07:30:11.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.053+0000 7f10d959a640 1 -- 172.21.15.77:0/2818958070 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4009690 con 0x7f10c40039b0 2024-03-06T07:30:11.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.054+0000 7f10e0bc7640 1 -- 172.21.15.77:0/2818958070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10c40039b0 msgr2=0x7f10c4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.054+0000 7f10e0bc7640 1 --2- 172.21.15.77:0/2818958070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10c40039b0 0x7f10c4003dd0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009ac0 tx=0x7f10c4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.054+0000 7f10dad9d640 1 -- 172.21.15.77:0/2818958070 reap_dead start 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.054+0000 7f10e0bc7640 1 -- 172.21.15.77:0/2818958070 shutdown_connections 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.054+0000 7f10e0bc7640 1 -- 172.21.15.77:0/2818958070 >> 172.21.15.77:0/2818958070 conn(0x7f10dc0b1a90 msgr2=0x7f10dc0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.054+0000 7f10e0bc7640 1 -- 172.21.15.77:0/2818958070 shutdown_connections 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.054+0000 7f10e0bc7640 1 -- 172.21.15.77:0/2818958070 wait complete. 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.055+0000 7f10e0bc7640 1 Processor -- start 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.055+0000 7f10e0bc7640 1 -- start start 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10e0bc7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10c40039b0 0x7f10dc15bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10e0bc7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10dc15c410 0x7f10dc1f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10e0bc7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10dc1f3e40 0x7f10dc1f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10e0bc7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc15e700 con 0x7f10dc15c410 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10e0bc7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc15e580 con 0x7f10dc1f3e40 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10e0bc7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc06e410 con 0x7f10c40039b0 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10d9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10dc15c410 0x7f10dc1f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10d9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10dc15c410 0x7f10dc1f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50134/0 (socket says 172.21.15.77:50134) 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10d9d9b640 1 -- 172.21.15.77:0/3633105226 learned_addr learned my addr 172.21.15.77:0/3633105226 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10da59c640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10c40039b0 0x7f10dc15bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.056+0000 7f10dad9d640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10dc1f3e40 0x7f10dc1f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10d9d9b640 1 -- 172.21.15.77:0/3633105226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10c40039b0 msgr2=0x7f10dc15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10d9d9b640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10c40039b0 0x7f10dc15bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10d9d9b640 1 -- 172.21.15.77:0/3633105226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10dc1f3e40 msgr2=0x7f10dc1f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10d9d9b640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10dc1f3e40 0x7f10dc1f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10d9d9b640 1 -- 172.21.15.77:0/3633105226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc15c890 con 0x7f10dc15c410 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10d9d9b640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10dc15c410 0x7f10dc1f3900 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f10d0011700 tx=0x7f10d0011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10caffd640 1 -- 172.21.15.77:0/3633105226 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d000f9c0 con 0x7f10dc15c410 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10caffd640 1 -- 172.21.15.77:0/3633105226 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d000fb60 con 0x7f10dc15c410 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc15cb50 con 0x7f10dc15c410 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10caffd640 1 -- 172.21.15.77:0/3633105226 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d0015460 con 0x7f10dc15c410 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.057+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc15d110 con 0x7f10dc15c410 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.058+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc04f330 con 0x7f10dc15c410 2024-03-06T07:30:11.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.059+0000 7f10caffd640 1 -- 172.21.15.77:0/3633105226 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10d0015600 con 0x7f10dc15c410 2024-03-06T07:30:11.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.059+0000 7f10caffd640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f10bc06b4d0 0x7f10bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.059+0000 7f10da59c640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f10bc06b4d0 0x7f10bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.059+0000 7f10caffd640 1 -- 172.21.15.77:0/3633105226 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10d008abb0 con 0x7f10dc15c410 2024-03-06T07:30:11.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.060+0000 7f10da59c640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f10bc06b4d0 0x7f10bc06d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f10d4036000 tx=0x7f10d4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.064+0000 7f10caffd640 1 -- 172.21.15.77:0/3633105226 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10d0019030 con 0x7f10dc15c410 2024-03-06T07:30:11.081 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-03-06T07:30:11.079+0000 7fc7ee2a0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:11.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.280+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f10dc072dd0 con 0x7f10dc15c410 2024-03-06T07:30:11.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.280+0000 7f10caffd640 1 -- 172.21.15.77:0/3633105226 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) v1 ==== 53+0+73 (secure 0 0 0) 0x7f10d0059a30 con 0x7f10dc15c410 2024-03-06T07:30:11.282 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f10bc06b4d0 msgr2=0x7f10bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f10bc06b4d0 0x7f10bc06d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f10d4036000 tx=0x7f10d4030000 comp rx=0 tx=0).stop 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10dc15c410 msgr2=0x7f10dc1f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10dc15c410 0x7f10dc1f3900 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f10d0011700 tx=0x7f10d0011bc0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 shutdown_connections 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f10bc06b4d0 0x7f10bc06d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10dc15c410 0x7f10dc1f3900 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10dc1f3e40 0x7f10dc1f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 --2- 172.21.15.77:0/3633105226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f10c40039b0 0x7f10dc15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 >> 172.21.15.77:0/3633105226 conn(0x7f10dc0b1a90 msgr2=0x7f10dc0b07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 shutdown_connections 2024-03-06T07:30:11.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:11.282+0000 7f10e0bc7640 1 -- 172.21.15.77:0/3633105226 wait complete. 2024-03-06T07:30:11.294 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:30:11.294 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:11.294 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:30:11.294 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:11.295 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:11.295 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:30:11.295 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:30:11.295 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:11.295 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:11.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:11.295 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:30:11.310 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:11.310 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:11.310 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:11.310 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:11.310 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:11.310 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42186sec preferred_lft 42186sec 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:11.311 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:30:11.312 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:11.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:11.313 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:11.376 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:30:11.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:11.377 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:30:11.432 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:30:11.432 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:11.433 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:30:11.433 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:30:11.433 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:11.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:11.433 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:30:11.433 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:30:11.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:30:11.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:30:11.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:30:11.433 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:30:11.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:11.527 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:30:11.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:11.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:11.542 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:11.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:11.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:30:11.609 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:30:11.609 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:11.609 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:11.664 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:11.664 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:11.664 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:11.719 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:30:11.786 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:30:11.841 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:30:11.842 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:11.843 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:11.843 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:11.843 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:11.843 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:30:11.843 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:11.843 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:11.843 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:11.843 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:11.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:11.844 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:30:11.899 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:30:11.899 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:30:11.941 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:30:11.967 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:30:12.023 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:12.024 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:12.025 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:12.025 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:12.025 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:30:12.025 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:12.025 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:12.025 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:12.025 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:12.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:12.025 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:30:12.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:30:12.025+0000 7f9afdcfa480 -1 init, newargv = 0x5579c446f6d0 newargc=15 2024-03-06T07:30:12.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[83231]: starting ceph client 2024-03-06T07:30:12.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[83231]: starting fuse 2024-03-06T07:30:12.042 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:30:12.042 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:30:12.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:30:12.042 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:30:12.042 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:30:12.042 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:30:12.042 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:30:12.042 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:30:12.042 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:30:12.042 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:30:12.043 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:30:12.044 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:30:12.170 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.83231.asok 2024-03-06T07:30:12.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:12.182 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.83231.asok 2024-03-06T07:30:12.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:12.182 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.83231.asok status 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "83231", 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 9345, 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 9345 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 86220899 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:30:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 86220899 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.9345 192.168.0.1:0/86220899", 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/86220899", 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 97, 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 139, 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:30:12.320 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:30:12.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:12.326 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:30:12.326 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:12.341 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:30:12.341 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:12.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:12.342 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:12.409 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:30:12.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.552+0000 7fad59f56640 1 Processor -- start 2024-03-06T07:30:12.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.552+0000 7fad59f56640 1 -- start start 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.552+0000 7fad59f56640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad541614d0 0x7fad540d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.552+0000 7fad59f56640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad540d1910 0x7fad540d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.552+0000 7fad59f56640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad540d2740 con 0x7fad540d22b0 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.552+0000 7fad59f56640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad541618b0 con 0x7fad540d1910 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.552+0000 7fad59f56640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540cf400 con 0x7fad541614d0 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.553+0000 7fad52ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad540d1910 0x7fad540d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.553+0000 7fad52ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad540d1910 0x7fad540d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50144/0 (socket says 172.21.15.77:50144) 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.553+0000 7fad52ffd640 1 -- 172.21.15.77:0/636211897 learned_addr learned my addr 172.21.15.77:0/636211897 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.553+0000 7fad537fe640 1 --2- 172.21.15.77:0/636211897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad541614d0 0x7fad540d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.553+0000 7fad52ffd640 1 -- 172.21.15.77:0/636211897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad541614d0 msgr2=0x7fad540d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.553+0000 7fad52ffd640 1 --2- 172.21.15.77:0/636211897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad541614d0 0x7fad540d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.553+0000 7fad52ffd640 1 -- 172.21.15.77:0/636211897 >> v1:172.21.15.158:6789/0 conn(0x7fad540d22b0 legacy=0x7fad540d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.553+0000 7fad52ffd640 1 -- 172.21.15.77:0/636211897 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540cf6a0 con 0x7fad540d1910 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad52ffd640 1 --2- 172.21.15.77:0/636211897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad540d1910 0x7fad540d1d70 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fad48009870 tx=0x7fad4802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88169e40e9c18f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad527fc640 1 -- 172.21.15.77:0/636211897 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48034070 con 0x7fad540d1910 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad527fc640 1 -- 172.21.15.77:0/636211897 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad48030840 con 0x7fad540d1910 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad527fc640 1 -- 172.21.15.77:0/636211897 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48030c10 con 0x7fad540d1910 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad59f56640 1 -- 172.21.15.77:0/636211897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad540d1910 msgr2=0x7fad540d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad59f56640 1 --2- 172.21.15.77:0/636211897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad540d1910 0x7fad540d1d70 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fad48009870 tx=0x7fad4802b980 comp rx=0 tx=0).stop 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad59f56640 1 -- 172.21.15.77:0/636211897 shutdown_connections 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad59f56640 1 --2- 172.21.15.77:0/636211897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad540d1910 0x7fad540d1d70 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad59f56640 1 --2- 172.21.15.77:0/636211897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad541614d0 0x7fad540d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad59f56640 1 -- 172.21.15.77:0/636211897 >> 172.21.15.77:0/636211897 conn(0x7fad540d3d40 msgr2=0x7fad540d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad59f56640 1 -- 172.21.15.77:0/636211897 shutdown_connections 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.554+0000 7fad59f56640 1 -- 172.21.15.77:0/636211897 wait complete. 2024-03-06T07:30:12.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.555+0000 7fad59f56640 1 Processor -- start 2024-03-06T07:30:12.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad59f56640 1 -- start start 2024-03-06T07:30:12.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad59f56640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad540d1910 0x7fad54153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad59f56640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad540d22b0 0x7fad541f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad59f56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad541614d0 0x7fad540b83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad59f56640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540b0910 con 0x7fad541614d0 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad59f56640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540cfc60 con 0x7fad540d1910 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad59f56640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540b1e40 con 0x7fad540d22b0 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad53fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad541614d0 0x7fad540b83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad53fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad541614d0 0x7fad540b83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50156/0 (socket says 172.21.15.77:50156) 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.556+0000 7fad53fff640 1 -- 172.21.15.77:0/235384325 learned_addr learned my addr 172.21.15.77:0/235384325 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad52ffd640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad540d22b0 0x7fad541f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad537fe640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad540d1910 0x7fad54153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad53fff640 1 -- 172.21.15.77:0/235384325 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad540d22b0 msgr2=0x7fad541f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad53fff640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad540d22b0 0x7fad541f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad53fff640 1 -- 172.21.15.77:0/235384325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad540d1910 msgr2=0x7fad54153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad53fff640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad540d1910 0x7fad54153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad53fff640 1 -- 172.21.15.77:0/235384325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad541f51a0 con 0x7fad541614d0 2024-03-06T07:30:12.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad53fff640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad541614d0 0x7fad540b83e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00c740 tx=0x7fad4c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad3bfff640 1 -- 172.21.15.77:0/235384325 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c016020 con 0x7fad541614d0 2024-03-06T07:30:12.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad3bfff640 1 -- 172.21.15.77:0/235384325 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c00ce50 con 0x7fad541614d0 2024-03-06T07:30:12.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad3bfff640 1 -- 172.21.15.77:0/235384325 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c003970 con 0x7fad541614d0 2024-03-06T07:30:12.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad540c7a10 con 0x7fad541614d0 2024-03-06T07:30:12.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.557+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad540c7fd0 con 0x7fad541614d0 2024-03-06T07:30:12.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.558+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad54068ff0 con 0x7fad541614d0 2024-03-06T07:30:12.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.559+0000 7fad3bfff640 1 -- 172.21.15.77:0/235384325 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad4c003b10 con 0x7fad541614d0 2024-03-06T07:30:12.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.559+0000 7fad3bfff640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad4006b4d0 0x7fad4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.559+0000 7fad537fe640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad4006b4d0 0x7fad4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.559+0000 7fad3bfff640 1 -- 172.21.15.77:0/235384325 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad4c089cc0 con 0x7fad541614d0 2024-03-06T07:30:12.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.559+0000 7fad537fe640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad4006b4d0 0x7fad4006d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fad540b4de0 tx=0x7fad44008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.568+0000 7fad3bfff640 1 -- 172.21.15.77:0/235384325 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad4c020020 con 0x7fad541614d0 2024-03-06T07:30:12.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:12.817+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fad540ce9b0 con 0x7fad541614d0 2024-03-06T07:30:13.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.130+0000 7fad3bfff640 1 -- 172.21.15.77:0/235384325 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v98) v1 ==== 105+0+0 (secure 0 0 0) 0x7fad540d2740 con 0x7fad541614d0 2024-03-06T07:30:13.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad4006b4d0 msgr2=0x7fad4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad4006b4d0 0x7fad4006d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fad540b4de0 tx=0x7fad44008040 comp rx=0 tx=0).stop 2024-03-06T07:30:13.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad541614d0 msgr2=0x7fad540b83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad541614d0 0x7fad540b83e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00c740 tx=0x7fad4c00cc00 comp rx=0 tx=0).stop 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 shutdown_connections 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad4006b4d0 0x7fad4006d990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad541614d0 0x7fad540b83e0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad540d22b0 0x7fad541f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 --2- 172.21.15.77:0/235384325 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad540d1910 0x7fad54153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 >> 172.21.15.77:0/235384325 conn(0x7fad540d3d40 msgr2=0x7fad5406bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 shutdown_connections 2024-03-06T07:30:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.133+0000 7fad59f56640 1 -- 172.21.15.77:0/235384325 wait complete. 2024-03-06T07:30:13.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:13.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.254+0000 7f343caeb640 1 Processor -- start 2024-03-06T07:30:13.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.254+0000 7f343caeb640 1 -- start start 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f343caeb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 0x7f34380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f343caeb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34380b41f0 0x7f34380b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f343caeb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34380b3430 con 0x7f343815dfe0 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f343caeb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34380d2c50 con 0x7f34380b41f0 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f343caeb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34380b31e0 con 0x7f34380d22b0 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f3436d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34380b41f0 0x7f34380b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f3436d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34380b41f0 0x7f34380b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50164/0 (socket says 172.21.15.77:50164) 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f3436d9d640 1 -- 172.21.15.77:0/1973306626 learned_addr learned my addr 172.21.15.77:0/1973306626 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f342ffff640 1 --2- 172.21.15.77:0/1973306626 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 0x7f34380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f3436d9d640 1 -- 172.21.15.77:0/1973306626 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 msgr2=0x7f34380d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f3436d9d640 1 --2- 172.21.15.77:0/1973306626 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 0x7f34380d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f3436d9d640 1 -- 172.21.15.77:0/1973306626 >> v1:172.21.15.158:6789/0 conn(0x7f343815dfe0 legacy=0x7f34380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.255+0000 7f3436d9d640 1 -- 172.21.15.77:0/1973306626 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380b3710 con 0x7f34380b41f0 2024-03-06T07:30:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.256+0000 7f3436d9d640 1 --2- 172.21.15.77:0/1973306626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b4650 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f3430009e50 tx=0x7f343002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148b09bf962e2694 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.256+0000 7f3435d9b640 1 -- 172.21.15.77:0/1973306626 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343002cb00 con 0x7f34380b41f0 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.256+0000 7f3435d9b640 1 -- 172.21.15.77:0/1973306626 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343002cca0 con 0x7f34380b41f0 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.256+0000 7f3435d9b640 1 -- 172.21.15.77:0/1973306626 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430034610 con 0x7f34380b41f0 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.257+0000 7f343caeb640 1 -- 172.21.15.77:0/1973306626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 msgr2=0x7f34380b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.257+0000 7f343caeb640 1 --2- 172.21.15.77:0/1973306626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b4650 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f3430009e50 tx=0x7f343002bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.257+0000 7f343caeb640 1 -- 172.21.15.77:0/1973306626 shutdown_connections 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.257+0000 7f343caeb640 1 --2- 172.21.15.77:0/1973306626 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 0x7f34380d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.257+0000 7f343caeb640 1 --2- 172.21.15.77:0/1973306626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b4650 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.257+0000 7f343caeb640 1 -- 172.21.15.77:0/1973306626 >> 172.21.15.77:0/1973306626 conn(0x7f34380d3cd0 msgr2=0x7f34380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.257+0000 7f343caeb640 1 -- 172.21.15.77:0/1973306626 shutdown_connections 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.257+0000 7f343caeb640 1 -- 172.21.15.77:0/1973306626 wait complete. 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.258+0000 7f343caeb640 1 Processor -- start 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.258+0000 7f343caeb640 1 -- start start 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343caeb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343caeb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 0x7f34380b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343caeb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343815dfe0 0x7f34381610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343caeb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34380cfa00 con 0x7f34380b41f0 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343caeb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34380b3d60 con 0x7f343815dfe0 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343caeb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34380d1c70 con 0x7f34380d22b0 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50176/0 (socket says 172.21.15.77:50176) 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343659c640 1 -- 172.21.15.77:0/799445271 learned_addr learned my addr 172.21.15.77:0/799445271 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f3436d9d640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343815dfe0 0x7f34381610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f342ffff640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 0x7f34380b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.259+0000 7f343659c640 1 -- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 msgr2=0x7f34380b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f343659c640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 0x7f34380b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f343659c640 1 -- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343815dfe0 msgr2=0x7f34381610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f343659c640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343815dfe0 0x7f34381610d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f343659c640 1 -- 172.21.15.77:0/799445271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343815a700 con 0x7f34380b41f0 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f3436d9d640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343815dfe0 0x7f34381610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f343659c640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b9790 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f342800c3f0 tx=0x7f342800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f342effd640 1 -- 172.21.15.77:0/799445271 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34280052d0 con 0x7f34380b41f0 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f342effd640 1 -- 172.21.15.77:0/799445271 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3428005490 con 0x7f34380b41f0 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f342effd640 1 -- 172.21.15.77:0/799445271 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428007260 con 0x7f34380b41f0 2024-03-06T07:30:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34380b86f0 con 0x7f34380b41f0 2024-03-06T07:30:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.260+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34380b8cb0 con 0x7f34380b41f0 2024-03-06T07:30:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.261+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343804f330 con 0x7f34380b41f0 2024-03-06T07:30:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.261+0000 7f342effd640 1 -- 172.21.15.77:0/799445271 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3428007400 con 0x7f34380b41f0 2024-03-06T07:30:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.262+0000 7f342effd640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f341806b4d0 0x7f341806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.262+0000 7f342ffff640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f341806b4d0 0x7f341806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.262+0000 7f342effd640 1 -- 172.21.15.77:0/799445271 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3428089630 con 0x7f34380b41f0 2024-03-06T07:30:13.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.262+0000 7f342ffff640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f341806b4d0 0x7f341806d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f343815d9f0 tx=0x7f3420008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.266+0000 7f342effd640 1 -- 172.21.15.77:0/799445271 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34280584b0 con 0x7f34380b41f0 2024-03-06T07:30:13.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.480+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34380ce910 con 0x7f34380b41f0 2024-03-06T07:30:13.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.481+0000 7f342effd640 1 -- 172.21.15.77:0/799445271 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5000 (secure 0 0 0) 0x7f342805ce20 con 0x7f34380b41f0 2024-03-06T07:30:13.483 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:13.483 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2532910127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2532910127},{"type":"v1","addr":"172.21.15.158:6835","nonce":2532910127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":8347,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/3282592218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3282592218},{"type":"v1","addr":"172.21.15.158:6833","nonce":3282592218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:13.134386+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:creating","state_seq":1,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:30:13.483 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 99 2024-03-06T07:30:13.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f341806b4d0 msgr2=0x7f341806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f341806b4d0 0x7f341806d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f343815d9f0 tx=0x7f3420008040 comp rx=0 tx=0).stop 2024-03-06T07:30:13.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 msgr2=0x7f34380b9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b9790 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f342800c3f0 tx=0x7f342800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 shutdown_connections 2024-03-06T07:30:13.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f343815dfe0 0x7f34381610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34380d22b0 0x7f34380b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f341806b4d0 0x7f341806d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 --2- 172.21.15.77:0/799445271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380b41f0 0x7f34380b9790 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 >> 172.21.15.77:0/799445271 conn(0x7f34380d3cd0 msgr2=0x7f343806d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 shutdown_connections 2024-03-06T07:30:13.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:13.483+0000 7f343caeb640 1 -- 172.21.15.77:0/799445271 wait complete. 2024-03-06T07:30:13.496 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:30:16.081 INFO:tasks.ceph.mds.c.smithi077.stderr:2024-03-06T07:30:16.080+0000 7fc7ee2a0640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:30:18.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:18.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f1435efb640 1 Processor -- start 2024-03-06T07:30:18.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f1435efb640 1 -- start start 2024-03-06T07:30:18.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f1435efb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f143015dfe0 0x7f14300cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f1435efb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f14300cf830 0x7f14300d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f1435efb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 0x7f14300d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f1435efb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14300b31e0 con 0x7f143015dfe0 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f1435efb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14300b3d60 con 0x7f14300cf830 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f1435efb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14300b3430 con 0x7f14300d27f0 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f142f7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f143015dfe0 0x7f14300cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f142f7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f143015dfe0 0x7f14300cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38614/0 (socket says 172.21.15.77:38614) 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.637+0000 7f142f7fe640 1 -- 172.21.15.77:0/3182171945 learned_addr learned my addr 172.21.15.77:0/3182171945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.638+0000 7f142ffff640 1 --2- 172.21.15.77:0/3182171945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 0x7f14300d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.638+0000 7f142effd640 1 --2- 172.21.15.77:0/3182171945 >> v2:172.21.15.158:3300/0 conn(0x7f14300cf830 0x7f14300d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.638+0000 7f142f7fe640 1 -- 172.21.15.77:0/3182171945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 msgr2=0x7f14300d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.638+0000 7f142f7fe640 1 --2- 172.21.15.77:0/3182171945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 0x7f14300d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.638+0000 7f142f7fe640 1 -- 172.21.15.77:0/3182171945 >> v2:172.21.15.158:3300/0 conn(0x7f14300cf830 msgr2=0x7f14300d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.638+0000 7f142f7fe640 1 --2- 172.21.15.77:0/3182171945 >> v2:172.21.15.158:3300/0 conn(0x7f14300cf830 0x7f14300d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.638+0000 7f142f7fe640 1 -- 172.21.15.77:0/3182171945 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14300b3710 con 0x7f143015dfe0 2024-03-06T07:30:18.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.638+0000 7f142effd640 1 --2- 172.21.15.77:0/3182171945 >> v2:172.21.15.158:3300/0 conn(0x7f14300cf830 0x7f14300d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:18.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f142f7fe640 1 --2- 172.21.15.77:0/3182171945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300cf2f0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f14300d16b0 tx=0x7f142802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7dae454a844f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f142e7fc640 1 -- 172.21.15.77:0/3182171945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142802ca70 con 0x7f143015dfe0 2024-03-06T07:30:18.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f142e7fc640 1 -- 172.21.15.77:0/3182171945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142802cc10 con 0x7f143015dfe0 2024-03-06T07:30:18.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f142e7fc640 1 -- 172.21.15.77:0/3182171945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14280343f0 con 0x7f143015dfe0 2024-03-06T07:30:18.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f1435efb640 1 -- 172.21.15.77:0/3182171945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 msgr2=0x7f14300cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f1435efb640 1 --2- 172.21.15.77:0/3182171945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300cf2f0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f14300d16b0 tx=0x7f142802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f1435efb640 1 -- 172.21.15.77:0/3182171945 shutdown_connections 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f1435efb640 1 --2- 172.21.15.77:0/3182171945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 0x7f14300d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f1435efb640 1 --2- 172.21.15.77:0/3182171945 >> v2:172.21.15.158:3300/0 conn(0x7f14300cf830 0x7f14300d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f1435efb640 1 --2- 172.21.15.77:0/3182171945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300cf2f0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.639+0000 7f1435efb640 1 -- 172.21.15.77:0/3182171945 >> 172.21.15.77:0/3182171945 conn(0x7f14300d3cd0 msgr2=0x7f14300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.640+0000 7f1435efb640 1 -- 172.21.15.77:0/3182171945 shutdown_connections 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.640+0000 7f1435efb640 1 -- 172.21.15.77:0/3182171945 wait complete. 2024-03-06T07:30:18.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.641+0000 7f1435efb640 1 Processor -- start 2024-03-06T07:30:18.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.642+0000 7f1435efb640 1 -- start start 2024-03-06T07:30:18.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f1435efb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14300cf830 0x7f14300cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f1435efb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 0x7f143015a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f1435efb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f1435efb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14300d1df0 con 0x7f143015dfe0 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f1435efb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14300d1c70 con 0x7f14300cf830 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f1435efb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143015e3c0 con 0x7f14300d27f0 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300b7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38618/0 (socket says 172.21.15.77:38618) 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 -- 172.21.15.77:0/1105749195 learned_addr learned my addr 172.21.15.77:0/1105749195 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142effd640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 0x7f143015a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142f7fe640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14300cf830 0x7f14300cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 -- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 msgr2=0x7f143015a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 0x7f143015a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 -- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14300cf830 msgr2=0x7f14300cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14300cf830 0x7f14300cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 -- 172.21.15.77:0/1105749195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14300caf40 con 0x7f143015dfe0 2024-03-06T07:30:18.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142f7fe640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14300cf830 0x7f14300cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:18.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f142ffff640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300b7180 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f142000e960 tx=0x7f142000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.643+0000 7f1427fff640 1 -- 172.21.15.77:0/1105749195 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420017070 con 0x7f143015dfe0 2024-03-06T07:30:18.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.644+0000 7f1427fff640 1 -- 172.21.15.77:0/1105749195 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142000c980 con 0x7f143015dfe0 2024-03-06T07:30:18.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.644+0000 7f1427fff640 1 -- 172.21.15.77:0/1105749195 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142000cd70 con 0x7f143015dfe0 2024-03-06T07:30:18.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.644+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14300cb1e0 con 0x7f143015dfe0 2024-03-06T07:30:18.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.644+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14300cb7a0 con 0x7f143015dfe0 2024-03-06T07:30:18.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.645+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14300cb9a0 con 0x7f143015dfe0 2024-03-06T07:30:18.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.645+0000 7f1427fff640 1 -- 172.21.15.77:0/1105749195 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f142001a070 con 0x7f143015dfe0 2024-03-06T07:30:18.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.646+0000 7f1427fff640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141006b4d0 0x7f141006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.646+0000 7f1427fff640 1 -- 172.21.15.77:0/1105749195 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f142008a150 con 0x7f143015dfe0 2024-03-06T07:30:18.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.646+0000 7f142f7fe640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141006b4d0 0x7f141006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.646+0000 7f142f7fe640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141006b4d0 0x7f141006d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1428035000 tx=0x7f1428030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.650+0000 7f1427fff640 1 -- 172.21.15.77:0/1105749195 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14300cb9a0 con 0x7f143015dfe0 2024-03-06T07:30:18.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.880+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f143004f330 con 0x7f143015dfe0 2024-03-06T07:30:18.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.880+0000 7f1427fff640 1 -- 172.21.15.77:0/1105749195 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f1420058980 con 0x7f143015dfe0 2024-03-06T07:30:18.882 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:18.882 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2532910127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2532910127},{"type":"v1","addr":"172.21.15.158:6835","nonce":2532910127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":8347,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/3282592218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3282592218},{"type":"v1","addr":"172.21.15.158:6833","nonce":3282592218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:14.135663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9177,9186]},"id":10}]} 2024-03-06T07:30:18.883 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 100 2024-03-06T07:30:18.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141006b4d0 msgr2=0x7f141006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141006b4d0 0x7f141006d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1428035000 tx=0x7f1428030000 comp rx=0 tx=0).stop 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 msgr2=0x7f14300b7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300b7180 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f142000e960 tx=0x7f142000ee20 comp rx=0 tx=0).stop 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 shutdown_connections 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141006b4d0 0x7f141006d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f143015dfe0 0x7f14300b7180 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f14300d27f0 0x7f143015a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 --2- 172.21.15.77:0/1105749195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f14300cf830 0x7f14300cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.882+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 >> 172.21.15.77:0/1105749195 conn(0x7f14300d3cd0 msgr2=0x7f143006c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.883+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 shutdown_connections 2024-03-06T07:30:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.883+0000 7f1435efb640 1 -- 172.21.15.77:0/1105749195 wait complete. 2024-03-06T07:30:18.899 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:19.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.998+0000 7f50b108f640 1 Processor -- start 2024-03-06T07:30:19.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.998+0000 7f50b108f640 1 -- start start 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50b108f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f50ac15dfe0 0x7f50ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50b108f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f50ac0cf830 0x7f50ac0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50b108f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 0x7f50ac0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50b108f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ac0b31e0 con 0x7f50ac15dfe0 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50b108f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ac0b3d60 con 0x7f50ac0cf830 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50b108f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac0b3430 con 0x7f50ac0d27f0 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f50ac15dfe0 0x7f50ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f50ac15dfe0 0x7f50ac0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38630/0 (socket says 172.21.15.77:38630) 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aad9d640 1 -- 172.21.15.77:0/1073352439 learned_addr learned my addr 172.21.15.77:0/1073352439 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:19.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50ab59e640 1 --2- 172.21.15.77:0/1073352439 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 0x7f50ac0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aa59c640 1 --2- 172.21.15.77:0/1073352439 >> v2:172.21.15.158:3300/0 conn(0x7f50ac0cf830 0x7f50ac0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aad9d640 1 -- 172.21.15.77:0/1073352439 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 msgr2=0x7f50ac0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aad9d640 1 --2- 172.21.15.77:0/1073352439 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 0x7f50ac0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aad9d640 1 -- 172.21.15.77:0/1073352439 >> v2:172.21.15.158:3300/0 conn(0x7f50ac0cf830 msgr2=0x7f50ac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aad9d640 1 --2- 172.21.15.77:0/1073352439 >> v2:172.21.15.158:3300/0 conn(0x7f50ac0cf830 0x7f50ac0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:18.999+0000 7f50aad9d640 1 -- 172.21.15.77:0/1073352439 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac0b3710 con 0x7f50ac15dfe0 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.000+0000 7f50aad9d640 1 --2- 172.21.15.77:0/1073352439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac15dfe0 0x7f50ac0cf2f0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0d16b0 tx=0x7f509c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8c84da869ce782 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.000+0000 7f50a9d9b640 1 -- 172.21.15.77:0/1073352439 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c035020 con 0x7f50ac15dfe0 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.000+0000 7f50a9d9b640 1 -- 172.21.15.77:0/1073352439 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c031870 con 0x7f50ac15dfe0 2024-03-06T07:30:19.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.000+0000 7f50a9d9b640 1 -- 172.21.15.77:0/1073352439 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c031c40 con 0x7f50ac15dfe0 2024-03-06T07:30:19.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 -- 172.21.15.77:0/1073352439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac15dfe0 msgr2=0x7f50ac0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 --2- 172.21.15.77:0/1073352439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac15dfe0 0x7f50ac0cf2f0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0d16b0 tx=0x7f509c02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:19.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 -- 172.21.15.77:0/1073352439 shutdown_connections 2024-03-06T07:30:19.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 --2- 172.21.15.77:0/1073352439 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 0x7f50ac0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 --2- 172.21.15.77:0/1073352439 >> v2:172.21.15.158:3300/0 conn(0x7f50ac0cf830 0x7f50ac0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 --2- 172.21.15.77:0/1073352439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac15dfe0 0x7f50ac0cf2f0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 -- 172.21.15.77:0/1073352439 >> 172.21.15.77:0/1073352439 conn(0x7f50ac0d3cd0 msgr2=0x7f50ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 -- 172.21.15.77:0/1073352439 shutdown_connections 2024-03-06T07:30:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.001+0000 7f50b108f640 1 -- 172.21.15.77:0/1073352439 wait complete. 2024-03-06T07:30:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.002+0000 7f50b108f640 1 Processor -- start 2024-03-06T07:30:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.002+0000 7f50b108f640 1 -- start start 2024-03-06T07:30:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50b108f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0cf830 0x7f50ac15f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50b108f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 0x7f50ac0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50b108f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac15dfe0 0x7f50ac159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50b108f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac0d1df0 con 0x7f50ac0cf830 2024-03-06T07:30:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50b108f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac0d1c70 con 0x7f50ac15dfe0 2024-03-06T07:30:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50b108f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac15e3c0 con 0x7f50ac0d27f0 2024-03-06T07:30:19.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50aad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0cf830 0x7f50ac15f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50aad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0cf830 0x7f50ac15f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38632/0 (socket says 172.21.15.77:38632) 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50aad9d640 1 -- 172.21.15.77:0/4125198491 learned_addr learned my addr 172.21.15.77:0/4125198491 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50aa59c640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 0x7f50ac0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.003+0000 7f50ab59e640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac15dfe0 0x7f50ac159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50aad9d640 1 -- 172.21.15.77:0/4125198491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 msgr2=0x7f50ac0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50aad9d640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 0x7f50ac0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50aad9d640 1 -- 172.21.15.77:0/4125198491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac15dfe0 msgr2=0x7f50ac159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50aad9d640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac15dfe0 0x7f50ac159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50aad9d640 1 -- 172.21.15.77:0/4125198491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac15f730 con 0x7f50ac0cf830 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50aad9d640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0cf830 0x7f50ac15f150 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f509c02b950 tx=0x7f509c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50937fe640 1 -- 172.21.15.77:0/4125198491 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c040070 con 0x7f50ac0cf830 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50937fe640 1 -- 172.21.15.77:0/4125198491 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c030d00 con 0x7f50ac0cf830 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50937fe640 1 -- 172.21.15.77:0/4125198491 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c035040 con 0x7f50ac0cf830 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ac15fa10 con 0x7f50ac0cf830 2024-03-06T07:30:19.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.004+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ac15ffd0 con 0x7f50ac0cf830 2024-03-06T07:30:19.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.005+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac04f330 con 0x7f50ac0cf830 2024-03-06T07:30:19.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.006+0000 7f50937fe640 1 -- 172.21.15.77:0/4125198491 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f509c038020 con 0x7f50ac0cf830 2024-03-06T07:30:19.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.006+0000 7f50937fe640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509806b4d0 0x7f509806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.006+0000 7f50aa59c640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509806b4d0 0x7f509806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.006+0000 7f50937fe640 1 -- 172.21.15.77:0/4125198491 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f509c0abf50 con 0x7f50ac0cf830 2024-03-06T07:30:19.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.007+0000 7f50aa59c640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509806b4d0 0x7f509806d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0cc200 tx=0x7f50a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.014+0000 7f50937fe640 1 -- 172.21.15.77:0/4125198491 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f509c07a780 con 0x7f50ac0cf830 2024-03-06T07:30:19.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.282+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50ac15d430 con 0x7f50ac0cf830 2024-03-06T07:30:19.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.283+0000 7f50937fe640 1 -- 172.21.15.77:0/4125198491 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5005 (secure 0 0 0) 0x7f509c07e820 con 0x7f50ac0cf830 2024-03-06T07:30:19.285 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:19.285 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/2532910127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":2532910127},{"type":"v1","addr":"172.21.15.158:6835","nonce":2532910127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":8347,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/3282592218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":3282592218},{"type":"v1","addr":"172.21.15.158:6833","nonce":3282592218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:14.135663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9177,9186]},"id":10}]} 2024-03-06T07:30:19.285 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 100 2024-03-06T07:30:19.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509806b4d0 msgr2=0x7f509806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509806b4d0 0x7f509806d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0cc200 tx=0x7f50a0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:19.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0cf830 msgr2=0x7f50ac15f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0cf830 0x7f50ac15f150 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f509c02b950 tx=0x7f509c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 shutdown_connections 2024-03-06T07:30:19.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f50ac15dfe0 0x7f50ac159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f50ac0d27f0 0x7f50ac0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f509806b4d0 0x7f509806d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 --2- 172.21.15.77:0/4125198491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac0cf830 0x7f50ac15f150 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 >> 172.21.15.77:0/4125198491 conn(0x7f50ac0d3cd0 msgr2=0x7f50ac0b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 shutdown_connections 2024-03-06T07:30:19.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.285+0000 7f50b108f640 1 -- 172.21.15.77:0/4125198491 wait complete. 2024-03-06T07:30:19.297 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are c and a 2024-03-06T07:30:19.298 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:19.298 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:19.309 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:19.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:30:19.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.412+0000 7f666fbc9640 1 Processor -- start 2024-03-06T07:30:19.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.412+0000 7f666fbc9640 1 -- start start 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.413+0000 7f666fbc9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f66680b4460 0x7f66680d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.413+0000 7f666fbc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6668161cf0 0x7f6668162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.413+0000 7f666fbc9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66680d22c0 con 0x7f6668162970 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.413+0000 7f666fbc9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6668161b00 con 0x7f66680b4460 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.413+0000 7f666fbc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66680cecc0 con 0x7f6668161cf0 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.413+0000 7f666d965640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f66680b4460 0x7f66680d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.413+0000 7f666d965640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f66680b4460 0x7f66680d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38638/0 (socket says 172.21.15.77:38638) 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.413+0000 7f666d965640 1 -- 172.21.15.77:0/1352586347 learned_addr learned my addr 172.21.15.77:0/1352586347 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.414+0000 7f666e166640 1 --2- 172.21.15.77:0/1352586347 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6668161cf0 0x7f6668162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.414+0000 7f666d965640 1 -- 172.21.15.77:0/1352586347 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6668161cf0 msgr2=0x7f6668162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.414+0000 7f666d965640 1 --2- 172.21.15.77:0/1352586347 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6668161cf0 0x7f6668162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.414+0000 7f666d965640 1 -- 172.21.15.77:0/1352586347 >> v1:172.21.15.158:6789/0 conn(0x7f6668162970 legacy=0x7f66680b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.414+0000 7f666d965640 1 -- 172.21.15.77:0/1352586347 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66680d25a0 con 0x7f66680b4460 2024-03-06T07:30:19.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.414+0000 7f666d965640 1 --2- 172.21.15.77:0/1352586347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66680b4460 0x7f66680d4800 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f6664009e50 tx=0x7f666402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbfe6bb388d101 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666c963640 1 -- 172.21.15.77:0/1352586347 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666402cb00 con 0x7f66680b4460 2024-03-06T07:30:19.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666c963640 1 -- 172.21.15.77:0/1352586347 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666402cca0 con 0x7f66680b4460 2024-03-06T07:30:19.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666c963640 1 -- 172.21.15.77:0/1352586347 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6664034610 con 0x7f66680b4460 2024-03-06T07:30:19.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666fbc9640 1 -- 172.21.15.77:0/1352586347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66680b4460 msgr2=0x7f66680d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666fbc9640 1 --2- 172.21.15.77:0/1352586347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66680b4460 0x7f66680d4800 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f6664009e50 tx=0x7f666402bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:19.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666fbc9640 1 -- 172.21.15.77:0/1352586347 shutdown_connections 2024-03-06T07:30:19.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666fbc9640 1 --2- 172.21.15.77:0/1352586347 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6668161cf0 0x7f6668162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666fbc9640 1 --2- 172.21.15.77:0/1352586347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66680b4460 0x7f66680d4800 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666fbc9640 1 -- 172.21.15.77:0/1352586347 >> 172.21.15.77:0/1352586347 conn(0x7f66680d3ce0 msgr2=0x7f66680d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666fbc9640 1 -- 172.21.15.77:0/1352586347 shutdown_connections 2024-03-06T07:30:19.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.415+0000 7f666fbc9640 1 -- 172.21.15.77:0/1352586347 wait complete. 2024-03-06T07:30:19.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.416+0000 7f666fbc9640 1 Processor -- start 2024-03-06T07:30:19.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.416+0000 7f666fbc9640 1 -- start start 2024-03-06T07:30:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666fbc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66680b4460 0x7f6668155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666fbc9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6668161cf0 0x7f66681f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666fbc9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6668162970 0x7f66681f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666fbc9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66680ce950 con 0x7f6668162970 2024-03-06T07:30:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666fbc9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66680d2b60 con 0x7f6668161cf0 2024-03-06T07:30:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666fbc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66680b4840 con 0x7f66680b4460 2024-03-06T07:30:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666e166640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6668162970 0x7f66681f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666e166640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6668162970 0x7f66681f7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38644/0 (socket says 172.21.15.77:38644) 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666e166640 1 -- 172.21.15.77:0/450712735 learned_addr learned my addr 172.21.15.77:0/450712735 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666d164640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6668161cf0 0x7f66681f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.417+0000 7f666d965640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66680b4460 0x7f6668155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666e166640 1 -- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66680b4460 msgr2=0x7f6668155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666e166640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66680b4460 0x7f6668155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666e166640 1 -- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6668161cf0 msgr2=0x7f66681f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666e166640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6668161cf0 0x7f66681f4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666e166640 1 -- 172.21.15.77:0/450712735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66681f4580 con 0x7f6668162970 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666d965640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66680b4460 0x7f6668155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666d164640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6668161cf0 0x7f66681f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666e166640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6668162970 0x7f66681f7980 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f666000e7f0 tx=0x7f666000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f665e7fc640 1 -- 172.21.15.77:0/450712735 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6660017070 con 0x7f6668162970 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f665e7fc640 1 -- 172.21.15.77:0/450712735 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666000ee40 con 0x7f6668162970 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66681f83d0 con 0x7f6668162970 2024-03-06T07:30:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f665e7fc640 1 -- 172.21.15.77:0/450712735 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666000cb80 con 0x7f6668162970 2024-03-06T07:30:19.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.418+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66681f4840 con 0x7f6668162970 2024-03-06T07:30:19.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.419+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66681f4a40 con 0x7f6668162970 2024-03-06T07:30:19.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.420+0000 7f665e7fc640 1 -- 172.21.15.77:0/450712735 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f666001a070 con 0x7f6668162970 2024-03-06T07:30:19.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.420+0000 7f665e7fc640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f664c06b4d0 0x7f664c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.420+0000 7f665e7fc640 1 -- 172.21.15.77:0/450712735 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f666008a0a0 con 0x7f6668162970 2024-03-06T07:30:19.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.420+0000 7f666d965640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f664c06b4d0 0x7f664c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.421+0000 7f666d965640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f664c06b4d0 0x7f664c06d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6664035000 tx=0x7f6664030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.429+0000 7f665e7fc640 1 -- 172.21.15.77:0/450712735 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66681f4a40 con 0x7f6668162970 2024-03-06T07:30:19.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:19.679+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f66681f4390 con 0x7f6668162970 2024-03-06T07:30:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.150+0000 7f665e7fc640 1 -- 172.21.15.77:0/450712735 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 8347 v101) v1 ==== 95+0+0 (secure 0 0 0) 0x7f6660058f20 con 0x7f6668162970 2024-03-06T07:30:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 8347 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f664c06b4d0 msgr2=0x7f664c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f664c06b4d0 0x7f664c06d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6664035000 tx=0x7f6664030000 comp rx=0 tx=0).stop 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6668162970 msgr2=0x7f66681f7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6668162970 0x7f66681f7980 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f666000e7f0 tx=0x7f666000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 shutdown_connections 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f664c06b4d0 0x7f664c06d990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6668162970 0x7f66681f7980 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6668161cf0 0x7f66681f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 --2- 172.21.15.77:0/450712735 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f66680b4460 0x7f6668155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 >> 172.21.15.77:0/450712735 conn(0x7f66680d3ce0 msgr2=0x7f666815fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 shutdown_connections 2024-03-06T07:30:20.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.153+0000 7f666fbc9640 1 -- 172.21.15.77:0/450712735 wait complete. 2024-03-06T07:30:20.166 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:20.166 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:20.177 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:20.177 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:30:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.298+0000 7f6832ec2640 1 Processor -- start 2024-03-06T07:30:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.298+0000 7f6832ec2640 1 -- start start 2024-03-06T07:30:20.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6832ec2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f682c0b4460 0x7f682c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6832ec2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f682c161cf0 0x7f682c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6832ec2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682c0d22c0 con 0x7f682c162970 2024-03-06T07:30:20.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6832ec2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682c161b00 con 0x7f682c0b4460 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6832ec2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682c0cecc0 con 0x7f682c161cf0 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6830c5e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f682c0b4460 0x7f682c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f682bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f682c162970 0x7f682c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38696/0 (socket says 172.21.15.77:38696) 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f682bfff640 1 -- 172.21.15.77:0/3845821465 learned_addr learned my addr 172.21.15.77:0/3845821465 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6830c5e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f682c0b4460 0x7f682c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38646/0 (socket says 172.21.15.77:38646) 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f683145f640 1 --2- 172.21.15.77:0/3845821465 >> v2:172.21.15.158:3300/0 conn(0x7f682c161cf0 0x7f682c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6830c5e640 1 -- 172.21.15.77:0/3845821465 >> v2:172.21.15.158:3300/0 conn(0x7f682c161cf0 msgr2=0x7f682c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6830c5e640 1 --2- 172.21.15.77:0/3845821465 >> v2:172.21.15.158:3300/0 conn(0x7f682c161cf0 0x7f682c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6830c5e640 1 -- 172.21.15.77:0/3845821465 >> v1:172.21.15.77:6789/0 conn(0x7f682c162970 legacy=0x7f682c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f682b7fe640 1 -- 172.21.15.77:0/3845821465 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345348739 0 0) 0x7f682c0d22c0 con 0x7f682c162970 2024-03-06T07:30:20.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.300+0000 7f6830c5e640 1 -- 172.21.15.77:0/3845821465 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c0d25a0 con 0x7f682c0b4460 2024-03-06T07:30:20.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.301+0000 7f6830c5e640 1 --2- 172.21.15.77:0/3845821465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c0b4460 0x7f682c0d4800 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f682c08c900 tx=0x7f682002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787ec33eb28f710e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.301+0000 7f682b7fe640 1 -- 172.21.15.77:0/3845821465 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820035020 con 0x7f682c0b4460 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f682b7fe640 1 -- 172.21.15.77:0/3845821465 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6820031830 con 0x7f682c0b4460 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f682b7fe640 1 -- 172.21.15.77:0/3845821465 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820031c00 con 0x7f682c0b4460 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f6832ec2640 1 -- 172.21.15.77:0/3845821465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c0b4460 msgr2=0x7f682c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f6832ec2640 1 --2- 172.21.15.77:0/3845821465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c0b4460 0x7f682c0d4800 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f682c08c900 tx=0x7f682002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f6832ec2640 1 -- 172.21.15.77:0/3845821465 shutdown_connections 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f6832ec2640 1 --2- 172.21.15.77:0/3845821465 >> v2:172.21.15.158:3300/0 conn(0x7f682c161cf0 0x7f682c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f6832ec2640 1 --2- 172.21.15.77:0/3845821465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c0b4460 0x7f682c0d4800 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f6832ec2640 1 -- 172.21.15.77:0/3845821465 >> 172.21.15.77:0/3845821465 conn(0x7f682c0d3ce0 msgr2=0x7f682c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f6832ec2640 1 -- 172.21.15.77:0/3845821465 shutdown_connections 2024-03-06T07:30:20.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.302+0000 7f6832ec2640 1 -- 172.21.15.77:0/3845821465 wait complete. 2024-03-06T07:30:20.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.303+0000 7f6832ec2640 1 Processor -- start 2024-03-06T07:30:20.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.303+0000 7f6832ec2640 1 -- start start 2024-03-06T07:30:20.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f6832ec2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f682c0b4460 0x7f682c159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f6832ec2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f682c161cf0 0x7f682c0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f6832ec2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c162970 0x7f682c0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f6832ec2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0ce950 con 0x7f682c162970 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f6832ec2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0d2b60 con 0x7f682c161cf0 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f6832ec2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682c0b4840 con 0x7f682c0b4460 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f683145f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c162970 0x7f682c0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f683145f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c162970 0x7f682c0cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38660/0 (socket says 172.21.15.77:38660) 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f683145f640 1 -- 172.21.15.77:0/163669498 learned_addr learned my addr 172.21.15.77:0/163669498 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f682bfff640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f682c161cf0 0x7f682c0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f6830c5e640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f682c0b4460 0x7f682c159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f683145f640 1 -- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f682c0b4460 msgr2=0x7f682c159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.304+0000 7f683145f640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f682c0b4460 0x7f682c159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f683145f640 1 -- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f682c161cf0 msgr2=0x7f682c0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f683145f640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f682c161cf0 0x7f682c0cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f683145f640 1 -- 172.21.15.77:0/163669498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c15a0a0 con 0x7f682c162970 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f6830c5e640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f682c0b4460 0x7f682c159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f683145f640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c162970 0x7f682c0cd3d0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f6824007e70 tx=0x7f682400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f68297fa640 1 -- 172.21.15.77:0/163669498 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824014620 con 0x7f682c162970 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f68297fa640 1 -- 172.21.15.77:0/163669498 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6824019070 con 0x7f682c162970 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682c0ca490 con 0x7f682c162970 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f68297fa640 1 -- 172.21.15.77:0/163669498 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824014a20 con 0x7f682c162970 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.305+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682c0c86a0 con 0x7f682c162970 2024-03-06T07:30:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.306+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682c06d1c0 con 0x7f682c162970 2024-03-06T07:30:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.307+0000 7f68297fa640 1 -- 172.21.15.77:0/163669498 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6824014bc0 con 0x7f682c162970 2024-03-06T07:30:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.308+0000 7f68297fa640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f681006b4d0 0x7f681006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.308+0000 7f6830c5e640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f681006b4d0 0x7f681006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.308+0000 7f68297fa640 1 -- 172.21.15.77:0/163669498 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f682408ab00 con 0x7f682c162970 2024-03-06T07:30:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.308+0000 7f6830c5e640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f681006b4d0 0x7f681006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6820036000 tx=0x7f6820008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.312+0000 7f68297fa640 1 -- 172.21.15.77:0/163669498 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f682c06d1c0 con 0x7f682c162970 2024-03-06T07:30:20.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:20.562+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f682c04f330 con 0x7f682c162970 2024-03-06T07:30:21.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.154+0000 7f68297fa640 1 -- 172.21.15.77:0/163669498 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7181 v102) v1 ==== 95+0+0 (secure 0 0 0) 0x7f6824059980 con 0x7f682c162970 2024-03-06T07:30:21.156 INFO:teuthology.orchestra.run.smithi077.stderr:failed mds gid 7181 2024-03-06T07:30:21.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f681006b4d0 msgr2=0x7f681006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f681006b4d0 0x7f681006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6820036000 tx=0x7f6820008030 comp rx=0 tx=0).stop 2024-03-06T07:30:21.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c162970 msgr2=0x7f682c0cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c162970 0x7f682c0cd3d0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f6824007e70 tx=0x7f682400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 shutdown_connections 2024-03-06T07:30:21.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f681006b4d0 0x7f681006d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c162970 0x7f682c0cd3d0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f682c161cf0 0x7f682c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 --2- 172.21.15.77:0/163669498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f682c0b4460 0x7f682c159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 >> 172.21.15.77:0/163669498 conn(0x7f682c0d3ce0 msgr2=0x7f682c0d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 shutdown_connections 2024-03-06T07:30:21.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.157+0000 7f6832ec2640 1 -- 172.21.15.77:0/163669498 wait complete. 2024-03-06T07:30:21.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:21.173 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:30:21.173 DEBUG:teuthology.orchestra.run.smithi077:> mkdir dir_1 dir_2 2024-03-06T07:30:21.174 DEBUG:teuthology.orchestra.run.smithi077:> touch dir_1/original 2024-03-06T07:30:21.174 DEBUG:teuthology.orchestra.run.smithi077:> ln dir_1/original dir_2/linkto 2024-03-06T07:30:21.174 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:30:21.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:21.210 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 dir_1) 2024-03-06T07:30:21.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:21.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.425+0000 7f820917a640 1 Processor -- start 2024-03-06T07:30:21.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.425+0000 7f820917a640 1 -- start start 2024-03-06T07:30:21.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820917a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f820415dfe0 0x7f82040cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820917a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82040d29c0 0x7f82040b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820917a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82040b3430 con 0x7f82040cf830 2024-03-06T07:30:21.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820917a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82040b31e0 con 0x7f82040d29c0 2024-03-06T07:30:21.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820917a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82040b3d60 con 0x7f820415dfe0 2024-03-06T07:30:21.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82040d29c0 0x7f82040b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82040d29c0 0x7f82040b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38674/0 (socket says 172.21.15.77:38674) 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f82040cf830 0x7f82040d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38712/0 (socket says 172.21.15.77:38712) 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.426+0000 7f820359e640 1 -- 172.21.15.77:0/278825049 learned_addr learned my addr 172.21.15.77:0/278825049 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.427+0000 7f820359e640 1 -- 172.21.15.77:0/278825049 >> v2:172.21.15.158:3300/0 conn(0x7f820415dfe0 msgr2=0x7f82040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.427+0000 7f820359e640 1 --2- 172.21.15.77:0/278825049 >> v2:172.21.15.158:3300/0 conn(0x7f820415dfe0 0x7f82040cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.427+0000 7f820359e640 1 -- 172.21.15.77:0/278825049 >> v1:172.21.15.77:6789/0 conn(0x7f82040cf830 legacy=0x7f82040d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.427+0000 7f8201d9b640 1 -- 172.21.15.77:0/278825049 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701519492 0 0) 0x7f82040b3430 con 0x7f82040cf830 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.427+0000 7f820359e640 1 -- 172.21.15.77:0/278825049 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040b3710 con 0x7f82040d29c0 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.427+0000 7f820359e640 1 --2- 172.21.15.77:0/278825049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82040d29c0 0x7f82040b41f0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f8204067990 tx=0x7f81fc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8db940b4c3d78a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.427+0000 7f8201d9b640 1 -- 172.21.15.77:0/278825049 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc02cb20 con 0x7f82040d29c0 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f8201d9b640 1 -- 172.21.15.77:0/278825049 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc02ccc0 con 0x7f82040d29c0 2024-03-06T07:30:21.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f8201d9b640 1 -- 172.21.15.77:0/278825049 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc034610 con 0x7f82040d29c0 2024-03-06T07:30:21.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f820917a640 1 -- 172.21.15.77:0/278825049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82040d29c0 msgr2=0x7f82040b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f820917a640 1 --2- 172.21.15.77:0/278825049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82040d29c0 0x7f82040b41f0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f8204067990 tx=0x7f81fc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f820917a640 1 -- 172.21.15.77:0/278825049 shutdown_connections 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f820917a640 1 --2- 172.21.15.77:0/278825049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82040d29c0 0x7f82040b41f0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f820917a640 1 --2- 172.21.15.77:0/278825049 >> v2:172.21.15.158:3300/0 conn(0x7f820415dfe0 0x7f82040cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f820917a640 1 -- 172.21.15.77:0/278825049 >> 172.21.15.77:0/278825049 conn(0x7f82040d3cd0 msgr2=0x7f82040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f820917a640 1 -- 172.21.15.77:0/278825049 shutdown_connections 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.428+0000 7f820917a640 1 -- 172.21.15.77:0/278825049 wait complete. 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820917a640 1 Processor -- start 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820917a640 1 -- start start 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820917a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82040cf830 0x7f82041f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820917a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82040d29c0 0x7f82041f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820917a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f820415dfe0 0x7f82041f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820917a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82040d1df0 con 0x7f820415dfe0 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820917a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82040d1c70 con 0x7f82040d29c0 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820917a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820415e3c0 con 0x7f82040cf830 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f820415dfe0 0x7f82041f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f820415dfe0 0x7f82041f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38690/0 (socket says 172.21.15.77:38690) 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.430+0000 7f820359e640 1 -- 172.21.15.77:0/1327792139 learned_addr learned my addr 172.21.15.77:0/1327792139 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820259c640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82040d29c0 0x7f82041f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f8202d9d640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82040cf830 0x7f82041f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820359e640 1 -- 172.21.15.77:0/1327792139 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82040cf830 msgr2=0x7f82041f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820359e640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82040cf830 0x7f82041f31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820359e640 1 -- 172.21.15.77:0/1327792139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82040d29c0 msgr2=0x7f82041f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820359e640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82040d29c0 0x7f82041f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820359e640 1 -- 172.21.15.77:0/1327792139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82041f36f0 con 0x7f820415dfe0 2024-03-06T07:30:21.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820359e640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f820415dfe0 0x7f82041f7330 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009f40 tx=0x7f81fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f81eb7fe640 1 -- 172.21.15.77:0/1327792139 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc043030 con 0x7f820415dfe0 2024-03-06T07:30:21.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f81eb7fe640 1 -- 172.21.15.77:0/1327792139 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc003990 con 0x7f820415dfe0 2024-03-06T07:30:21.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f81eb7fe640 1 -- 172.21.15.77:0/1327792139 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc009510 con 0x7f820415dfe0 2024-03-06T07:30:21.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8204161220 con 0x7f820415dfe0 2024-03-06T07:30:21.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.431+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82040b8d00 con 0x7f820415dfe0 2024-03-06T07:30:21.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.432+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820404f330 con 0x7f820415dfe0 2024-03-06T07:30:21.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.433+0000 7f81eb7fe640 1 -- 172.21.15.77:0/1327792139 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81fc02cb30 con 0x7f820415dfe0 2024-03-06T07:30:21.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.434+0000 7f81eb7fe640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f81f006b4d0 0x7f81f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.434+0000 7f81eb7fe640 1 -- 172.21.15.77:0/1327792139 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81fc0ac4d0 con 0x7f820415dfe0 2024-03-06T07:30:21.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.434+0000 7f8202d9d640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f81f006b4d0 0x7f81f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.434+0000 7f8202d9d640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f81f006b4d0 0x7f81f006d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f81f40098e0 tx=0x7f81f4009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.441+0000 7f81eb7fe640 1 -- 172.21.15.77:0/1327792139 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81fc04d020 con 0x7f820415dfe0 2024-03-06T07:30:21.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.656+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82040747a0 con 0x7f820415dfe0 2024-03-06T07:30:21.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.657+0000 7f81eb7fe640 1 -- 172.21.15.77:0/1327792139 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f82041f36f0 con 0x7f820415dfe0 2024-03-06T07:30:21.659 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:21.659 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:14.135663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9177,9186]},"id":10}]} 2024-03-06T07:30:21.659 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 102 2024-03-06T07:30:21.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f81f006b4d0 msgr2=0x7f81f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f81f006b4d0 0x7f81f006d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f81f40098e0 tx=0x7f81f4009060 comp rx=0 tx=0).stop 2024-03-06T07:30:21.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f820415dfe0 msgr2=0x7f82041f7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f820415dfe0 0x7f82041f7330 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009f40 tx=0x7f81fc004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:21.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 shutdown_connections 2024-03-06T07:30:21.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f81f006b4d0 0x7f81f006d990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f820415dfe0 0x7f82041f7330 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82040d29c0 0x7f82041f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 --2- 172.21.15.77:0/1327792139 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82040cf830 0x7f82041f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 >> 172.21.15.77:0/1327792139 conn(0x7f82040d3cd0 msgr2=0x7f820406ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 shutdown_connections 2024-03-06T07:30:21.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:21.659+0000 7f820917a640 1 -- 172.21.15.77:0/1327792139 wait complete. 2024-03-06T07:30:21.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:21.792 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:21.793 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:21.793 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:23.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:23.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.922+0000 7fc8486b5640 1 Processor -- start 2024-03-06T07:30:23.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.922+0000 7fc8486b5640 1 -- start start 2024-03-06T07:30:23.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc8486b5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8400d22b0 0x7fc8400d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc8486b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 0x7fc8400b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc8486b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8400b3430 con 0x7fc84015dfe0 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc8486b5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8400d2c50 con 0x7fc8400d22b0 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc8486b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400b31e0 con 0x7fc8400b41f0 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc845c50640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8400d22b0 0x7fc8400d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc846451640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc84015dfe0 0x7fc8400cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38714/0 (socket says 172.21.15.77:38714) 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc846451640 1 -- 172.21.15.77:0/394158732 learned_addr learned my addr 172.21.15.77:0/394158732 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc845c50640 1 -- 172.21.15.77:0/394158732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 msgr2=0x7fc8400b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc845c50640 1 --2- 172.21.15.77:0/394158732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 0x7fc8400b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc845c50640 1 -- 172.21.15.77:0/394158732 >> v1:172.21.15.77:6789/0 conn(0x7fc84015dfe0 legacy=0x7fc8400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.923+0000 7fc845c50640 1 -- 172.21.15.77:0/394158732 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400b3710 con 0x7fc8400d22b0 2024-03-06T07:30:23.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.924+0000 7fc845c50640 1 --2- 172.21.15.77:0/394158732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8400d22b0 0x7fc8400d2710 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc8300097a0 tx=0x7fc83002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02744cec39336d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.924+0000 7fc84544f640 1 -- 172.21.15.77:0/394158732 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830035020 con 0x7fc8400d22b0 2024-03-06T07:30:23.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.924+0000 7fc84544f640 1 -- 172.21.15.77:0/394158732 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc830031830 con 0x7fc8400d22b0 2024-03-06T07:30:23.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.924+0000 7fc84544f640 1 -- 172.21.15.77:0/394158732 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830031c00 con 0x7fc8400d22b0 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.925+0000 7fc8486b5640 1 -- 172.21.15.77:0/394158732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8400d22b0 msgr2=0x7fc8400d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.925+0000 7fc8486b5640 1 --2- 172.21.15.77:0/394158732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8400d22b0 0x7fc8400d2710 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc8300097a0 tx=0x7fc83002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.925+0000 7fc8486b5640 1 -- 172.21.15.77:0/394158732 shutdown_connections 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.925+0000 7fc8486b5640 1 --2- 172.21.15.77:0/394158732 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 0x7fc8400b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.925+0000 7fc8486b5640 1 --2- 172.21.15.77:0/394158732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8400d22b0 0x7fc8400d2710 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.925+0000 7fc8486b5640 1 -- 172.21.15.77:0/394158732 >> 172.21.15.77:0/394158732 conn(0x7fc8400d3cd0 msgr2=0x7fc8400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.925+0000 7fc8486b5640 1 -- 172.21.15.77:0/394158732 shutdown_connections 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.925+0000 7fc8486b5640 1 -- 172.21.15.77:0/394158732 wait complete. 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.926+0000 7fc8486b5640 1 Processor -- start 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.926+0000 7fc8486b5640 1 -- start start 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc8486b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 0x7fc8400cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc8486b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8400d22b0 0x7fc8400b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc8486b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc84015dfe0 0x7fc8400b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc8486b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400cfa00 con 0x7fc84015dfe0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc8486b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400b3d60 con 0x7fc8400d22b0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc8486b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400d1c70 con 0x7fc8400b41f0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc84015dfe0 0x7fc8400b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc84015dfe0 0x7fc8400b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38714/0 (socket says 172.21.15.77:38714) 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 -- 172.21.15.77:0/2432653452 learned_addr learned my addr 172.21.15.77:0/2432653452 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846451640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 0x7fc8400cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc845c50640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8400d22b0 0x7fc8400b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 -- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 msgr2=0x7fc8400cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 0x7fc8400cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 -- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8400d22b0 msgr2=0x7fc8400b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8400d22b0 0x7fc8400b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 -- 172.21.15.77:0/2432653452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400cc460 con 0x7fc84015dfe0 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc845c50640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8400d22b0 0x7fc8400b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.927+0000 7fc846c52640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc84015dfe0 0x7fc8400b6de0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fc83800b9a0 tx=0x7fc83800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.928+0000 7fc836ffd640 1 -- 172.21.15.77:0/2432653452 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838019070 con 0x7fc84015dfe0 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.928+0000 7fc836ffd640 1 -- 172.21.15.77:0/2432653452 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83800cbc0 con 0x7fc84015dfe0 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.928+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8400cc740 con 0x7fc84015dfe0 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.928+0000 7fc836ffd640 1 -- 172.21.15.77:0/2432653452 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838014440 con 0x7fc84015dfe0 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.928+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8400ccd00 con 0x7fc84015dfe0 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.929+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84004f330 con 0x7fc84015dfe0 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.929+0000 7fc836ffd640 1 -- 172.21.15.77:0/2432653452 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc83801c070 con 0x7fc84015dfe0 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.930+0000 7fc836ffd640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82406b4d0 0x7fc82406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.930+0000 7fc836ffd640 1 -- 172.21.15.77:0/2432653452 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc838089e40 con 0x7fc84015dfe0 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.930+0000 7fc846451640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82406b4d0 0x7fc82406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.930+0000 7fc846451640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82406b4d0 0x7fc82406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc83c007fd0 tx=0x7fc83c007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:23.934+0000 7fc836ffd640 1 -- 172.21.15.77:0/2432653452 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc838058700 con 0x7fc84015dfe0 2024-03-06T07:30:24.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.150+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc84015d660 con 0x7fc84015dfe0 2024-03-06T07:30:24.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.150+0000 7fc836ffd640 1 -- 172.21.15.77:0/2432653452 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fc83805c7a0 con 0x7fc84015dfe0 2024-03-06T07:30:24.152 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:24.152 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:14.135663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9177,9186]},"id":10}]} 2024-03-06T07:30:24.152 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 102 2024-03-06T07:30:24.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82406b4d0 msgr2=0x7fc82406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82406b4d0 0x7fc82406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc83c007fd0 tx=0x7fc83c007e80 comp rx=0 tx=0).stop 2024-03-06T07:30:24.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc84015dfe0 msgr2=0x7fc8400b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc84015dfe0 0x7fc8400b6de0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fc83800b9a0 tx=0x7fc83800be60 comp rx=0 tx=0).stop 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 shutdown_connections 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc82406b4d0 0x7fc82406d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc84015dfe0 0x7fc8400b6de0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc8400d22b0 0x7fc8400b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 --2- 172.21.15.77:0/2432653452 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc8400b41f0 0x7fc8400cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 >> 172.21.15.77:0/2432653452 conn(0x7fc8400d3cd0 msgr2=0x7fc8400cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 shutdown_connections 2024-03-06T07:30:24.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:24.152+0000 7fc8486b5640 1 -- 172.21.15.77:0/2432653452 wait complete. 2024-03-06T07:30:24.168 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:24.288 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:24.289 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:24.289 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:26.290 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.390+0000 7faab9cb5640 1 Processor -- start 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.390+0000 7faab9cb5640 1 -- start start 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab9cb5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faab4158920 0x7faab415c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab9cb5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab415e740 con 0x7faab40d14f0 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab9cb5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab415e9e0 con 0x7faab40afdd0 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab9cb5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab415f5f0 con 0x7faab4158920 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab2ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7faab40d14f0 0x7faab40af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38720/0 (socket says 172.21.15.77:38720) 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab2ffd640 1 -- 172.21.15.77:0/3106401041 learned_addr learned my addr 172.21.15.77:0/3106401041 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929040441 0 0) 0x7faab415e740 con 0x7faab40d14f0 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab37fe640 1 --2- 172.21.15.77:0/3106401041 >> v2:172.21.15.158:3300/0 conn(0x7faab4158920 0x7faab415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.391+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa9c003520 con 0x7faab40d14f0 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3113217777 0 0) 0x7faa9c003520 con 0x7faab40d14f0 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 >> v2:172.21.15.158:3300/0 conn(0x7faab4158920 msgr2=0x7faab415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 --2- 172.21.15.77:0/3106401041 >> v2:172.21.15.158:3300/0 conn(0x7faab4158920 0x7faab415c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 >> v1:172.21.15.158:6789/0 conn(0x7faab40afdd0 legacy=0x7faab415b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab415ecc0 con 0x7faab40d14f0 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7faaa8001670 con 0x7faab40d14f0 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 >> v1:172.21.15.77:6789/0 conn(0x7faab40d14f0 legacy=0x7faab40af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa9c0039b0 0x7faa9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa9c004800 0x7faa9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faa9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab415f5f0 con 0x7faa9c0051c0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c00c060 con 0x7faa9c0039b0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c00c260 con 0x7faa9c004800 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab3fff640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faa9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab37fe640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa9c0039b0 0x7faa9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab2ffd640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa9c004800 0x7faa9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab3fff640 1 -- 172.21.15.77:0/3106401041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa9c004800 msgr2=0x7faa9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab3fff640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faa9c004800 0x7faa9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab3fff640 1 -- 172.21.15.77:0/3106401041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa9c0039b0 msgr2=0x7faa9c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab3fff640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faa9c0039b0 0x7faa9c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.392+0000 7faab3fff640 1 -- 172.21.15.77:0/3106401041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c003520 con 0x7faa9c0051c0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.393+0000 7faab3fff640 1 -- 172.21.15.77:0/3106401041 reap_dead start 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.393+0000 7faab3fff640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faa9c00b9c0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7faab4072b50 tx=0x7faa9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e045af2482259 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.393+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac001a70 con 0x7faa9c0051c0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.393+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac003b70 con 0x7faa9c0051c0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.393+0000 7faab27fc640 1 -- 172.21.15.77:0/3106401041 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0050d0 con 0x7faa9c0051c0 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.394+0000 7faab9cb5640 1 -- 172.21.15.77:0/3106401041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 msgr2=0x7faa9c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.394+0000 7faab9cb5640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faa9c00b9c0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7faab4072b50 tx=0x7faa9c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.394+0000 7faab9cb5640 1 -- 172.21.15.77:0/3106401041 shutdown_connections 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.394+0000 7faab9cb5640 1 --2- 172.21.15.77:0/3106401041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faa9c00b9c0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.394+0000 7faab9cb5640 1 -- 172.21.15.77:0/3106401041 >> 172.21.15.77:0/3106401041 conn(0x7faab40d0960 msgr2=0x7faab40d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.394+0000 7faab9cb5640 1 -- 172.21.15.77:0/3106401041 shutdown_connections 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.394+0000 7faab9cb5640 1 -- 172.21.15.77:0/3106401041 wait complete. 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.395+0000 7faab9cb5640 1 Processor -- start 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.395+0000 7faab9cb5640 1 -- start start 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab9cb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faab4147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab9cb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faab40a9e60 0x7faab40aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab9cb5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faab41f2520 0x7faab41f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab9cb5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab415c310 con 0x7faa9c0051c0 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab9cb5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab415f280 con 0x7faab41f2520 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab9cb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab415c490 con 0x7faab40a9e60 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faab4147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faab4147580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38736/0 (socket says 172.21.15.77:38736) 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab37fe640 1 -- 172.21.15.77:0/252784151 learned_addr learned my addr 172.21.15.77:0/252784151 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab2ffd640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faab40a9e60 0x7faab40aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab3fff640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faab41f2520 0x7faab41f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab37fe640 1 -- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faab40a9e60 msgr2=0x7faab40aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab37fe640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faab40a9e60 0x7faab40aa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab37fe640 1 -- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faab41f2520 msgr2=0x7faab41f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab37fe640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faab41f2520 0x7faab41f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab37fe640 1 -- 172.21.15.77:0/252784151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab41f3250 con 0x7faa9c0051c0 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab2ffd640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faab40a9e60 0x7faab40aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:26.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.396+0000 7faab3fff640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faab41f2520 0x7faab41f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.397+0000 7faab37fe640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faab4147580 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7faaa40118e0 tx=0x7faaa4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.397+0000 7faa9bfff640 1 -- 172.21.15.77:0/252784151 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa400faf0 con 0x7faa9c0051c0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.397+0000 7faa9bfff640 1 -- 172.21.15.77:0/252784151 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa400fc90 con 0x7faa9c0051c0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.397+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab41563d0 con 0x7faa9c0051c0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.397+0000 7faa9bfff640 1 -- 172.21.15.77:0/252784151 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4014440 con 0x7faa9c0051c0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.397+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4156990 con 0x7faa9c0051c0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.398+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab404f330 con 0x7faa9c0051c0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.398+0000 7faa9bfff640 1 -- 172.21.15.77:0/252784151 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faaa4012040 con 0x7faa9c0051c0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.399+0000 7faa9bfff640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaa006b500 0x7faaa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.399+0000 7faa9bfff640 1 -- 172.21.15.77:0/252784151 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faaa408a8a0 con 0x7faa9c0051c0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.399+0000 7faab2ffd640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaa006b500 0x7faaa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.399+0000 7faab2ffd640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaa006b500 0x7faaa006d9c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7faab4155a30 tx=0x7faaa8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.403+0000 7faa9bfff640 1 -- 172.21.15.77:0/252784151 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faaa4019030 con 0x7faa9c0051c0 2024-03-06T07:30:26.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.649+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faab4154230 con 0x7faa9c0051c0 2024-03-06T07:30:26.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.649+0000 7faa9bfff640 1 -- 172.21.15.77:0/252784151 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3418 (secure 0 0 0) 0x7faaa40590d0 con 0x7faa9c0051c0 2024-03-06T07:30:26.651 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:14.135663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9177,9186]},"id":10}]} 2024-03-06T07:30:26.652 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 102 2024-03-06T07:30:26.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaa006b500 msgr2=0x7faaa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaa006b500 0x7faaa006d9c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7faab4155a30 tx=0x7faaa8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 msgr2=0x7faab4147580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faab4147580 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7faaa40118e0 tx=0x7faaa4011da0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 shutdown_connections 2024-03-06T07:30:26.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faab41f2520 0x7faab41f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faab40a9e60 0x7faab40aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaa006b500 0x7faaa006d9c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 --2- 172.21.15.77:0/252784151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa9c0051c0 0x7faab4147580 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 >> 172.21.15.77:0/252784151 conn(0x7faab40d0960 msgr2=0x7faab415d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 shutdown_connections 2024-03-06T07:30:26.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:26.651+0000 7faab9cb5640 1 -- 172.21.15.77:0/252784151 wait complete. 2024-03-06T07:30:26.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:26.784 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:26.785 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:26.785 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:28.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:28.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.887+0000 7ff9d4876640 1 Processor -- start 2024-03-06T07:30:28.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.887+0000 7ff9d4876640 1 -- start start 2024-03-06T07:30:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9d4876640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff9d01515e0 0x7ff9d00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9d4876640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d015b430 0x7ff9d0071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9d4876640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d015b8b0 con 0x7ff9d00cd7a0 2024-03-06T07:30:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9d4876640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d0070a70 con 0x7ff9d01515e0 2024-03-06T07:30:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9d4876640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0070d50 con 0x7ff9d015b430 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9ce59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff9d01515e0 0x7ff9d00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9ce59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff9d01515e0 0x7ff9d00cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37390/0 (socket says 172.21.15.77:37390) 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9ce59c640 1 -- 172.21.15.77:0/308169848 learned_addr learned my addr 172.21.15.77:0/308169848 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9cdd9b640 1 --2- 172.21.15.77:0/308169848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d015b430 0x7ff9d0071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9ce59c640 1 -- 172.21.15.77:0/308169848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d015b430 msgr2=0x7ff9d0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9ce59c640 1 --2- 172.21.15.77:0/308169848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d015b430 0x7ff9d0071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9ce59c640 1 -- 172.21.15.77:0/308169848 >> v1:172.21.15.158:6789/0 conn(0x7ff9d00cd7a0 legacy=0x7ff9d00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9ce59c640 1 -- 172.21.15.77:0/308169848 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0071030 con 0x7ff9d01515e0 2024-03-06T07:30:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.888+0000 7ff9cdd9b640 1 --2- 172.21.15.77:0/308169848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d015b430 0x7ff9d0071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.889+0000 7ff9ce59c640 1 --2- 172.21.15.77:0/308169848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d01515e0 0x7ff9d00cce40 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009870 tx=0x7ff9c002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8123699b734927cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.889+0000 7ff9cd59a640 1 -- 172.21.15.77:0/308169848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0034070 con 0x7ff9d01515e0 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.889+0000 7ff9cd59a640 1 -- 172.21.15.77:0/308169848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c0030840 con 0x7ff9d01515e0 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.889+0000 7ff9cd59a640 1 -- 172.21.15.77:0/308169848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0030c10 con 0x7ff9d01515e0 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.890+0000 7ff9d4876640 1 -- 172.21.15.77:0/308169848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d01515e0 msgr2=0x7ff9d00cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.890+0000 7ff9d4876640 1 --2- 172.21.15.77:0/308169848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d01515e0 0x7ff9d00cce40 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009870 tx=0x7ff9c002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.890+0000 7ff9d4876640 1 -- 172.21.15.77:0/308169848 shutdown_connections 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.890+0000 7ff9d4876640 1 --2- 172.21.15.77:0/308169848 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d015b430 0x7ff9d0071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.890+0000 7ff9d4876640 1 --2- 172.21.15.77:0/308169848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d01515e0 0x7ff9d00cce40 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.890+0000 7ff9d4876640 1 -- 172.21.15.77:0/308169848 >> 172.21.15.77:0/308169848 conn(0x7ff9d00cf8a0 msgr2=0x7ff9d00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.890+0000 7ff9d4876640 1 -- 172.21.15.77:0/308169848 shutdown_connections 2024-03-06T07:30:28.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.890+0000 7ff9d4876640 1 -- 172.21.15.77:0/308169848 wait complete. 2024-03-06T07:30:28.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.891+0000 7ff9d4876640 1 Processor -- start 2024-03-06T07:30:28.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.891+0000 7ff9d4876640 1 -- start start 2024-03-06T07:30:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9d4876640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d00cd7a0 0x7ff9d0159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9d4876640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9d01515e0 0x7ff9d015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9d4876640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d015b430 0x7ff9d015c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9d4876640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0071ef0 con 0x7ff9d015b430 2024-03-06T07:30:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9d4876640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0071d70 con 0x7ff9d01515e0 2024-03-06T07:30:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9d4876640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00cc3a0 con 0x7ff9d00cd7a0 2024-03-06T07:30:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d015b430 0x7ff9d015c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d015b430 0x7ff9d015c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37402/0 (socket says 172.21.15.77:37402) 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 -- 172.21.15.77:0/3769268645 learned_addr learned my addr 172.21.15.77:0/3769268645 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ce59c640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d00cd7a0 0x7ff9d0159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9cdd9b640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9d01515e0 0x7ff9d015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 -- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d00cd7a0 msgr2=0x7ff9d0159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d00cd7a0 0x7ff9d0159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 -- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9d01515e0 msgr2=0x7ff9d015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9d01515e0 0x7ff9d015bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 -- 172.21.15.77:0/3769268645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d01526f0 con 0x7ff9d015b430 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ce59c640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d00cd7a0 0x7ff9d0159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9cdd9b640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9d01515e0 0x7ff9d015bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.892+0000 7ff9ced9d640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d015b430 0x7ff9d015c020 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800c7d0 tx=0x7ff9c800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.893+0000 7ff9beffd640 1 -- 172.21.15.77:0/3769268645 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8003b70 con 0x7ff9d015b430 2024-03-06T07:30:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.893+0000 7ff9beffd640 1 -- 172.21.15.77:0/3769268645 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8003d30 con 0x7ff9d015b430 2024-03-06T07:30:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.893+0000 7ff9beffd640 1 -- 172.21.15.77:0/3769268645 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c800aa90 con 0x7ff9d015b430 2024-03-06T07:30:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.893+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d01444d0 con 0x7ff9d015b430 2024-03-06T07:30:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.893+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d0145fa0 con 0x7ff9d015b430 2024-03-06T07:30:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.894+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d0068840 con 0x7ff9d015b430 2024-03-06T07:30:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.894+0000 7ff9beffd640 1 -- 172.21.15.77:0/3769268645 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9c800ac30 con 0x7ff9d015b430 2024-03-06T07:30:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.895+0000 7ff9beffd640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9b006b500 0x7ff9b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.895+0000 7ff9beffd640 1 -- 172.21.15.77:0/3769268645 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9c808a1c0 con 0x7ff9d015b430 2024-03-06T07:30:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.895+0000 7ff9ce59c640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9b006b500 0x7ff9b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.895+0000 7ff9ce59c640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9b006b500 0x7ff9b006d9c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0036000 tx=0x7ff9c003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:28.903+0000 7ff9beffd640 1 -- 172.21.15.77:0/3769268645 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9c8059040 con 0x7ff9d015b430 2024-03-06T07:30:29.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.119+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9d004f330 con 0x7ff9d015b430 2024-03-06T07:30:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.119+0000 7ff9beffd640 1 -- 172.21.15.77:0/3769268645 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3418 (secure 0 0 0) 0x7ff9c8058940 con 0x7ff9d015b430 2024-03-06T07:30:29.121 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:29.121 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:14.135663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9177,9186]},"id":10}]} 2024-03-06T07:30:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 102 2024-03-06T07:30:29.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9b006b500 msgr2=0x7ff9b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9b006b500 0x7ff9b006d9c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0036000 tx=0x7ff9c003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:29.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d015b430 msgr2=0x7ff9d015c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d015b430 0x7ff9d015c020 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800c7d0 tx=0x7ff9c800cc90 comp rx=0 tx=0).stop 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 shutdown_connections 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff9b006b500 0x7ff9b006d9c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9d015b430 0x7ff9d015c020 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff9d01515e0 0x7ff9d015bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 --2- 172.21.15.77:0/3769268645 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff9d00cd7a0 0x7ff9d0159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 >> 172.21.15.77:0/3769268645 conn(0x7ff9d00cf8a0 msgr2=0x7ff9d006d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 shutdown_connections 2024-03-06T07:30:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:29.121+0000 7ff9d4876640 1 -- 172.21.15.77:0/3769268645 wait complete. 2024-03-06T07:30:29.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:29.251 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:29.252 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:29.252 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:31.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:31.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.353+0000 7f3b9c46d640 1 Processor -- start 2024-03-06T07:30:31.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.353+0000 7f3b9c46d640 1 -- start start 2024-03-06T07:30:31.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9c46d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9c46d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3b940d29c0 0x7f3b940b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9c46d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b940b3430 con 0x7f3b940cf830 2024-03-06T07:30:31.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9c46d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b940b31e0 con 0x7f3b940d29c0 2024-03-06T07:30:31.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9c46d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940b3d60 con 0x7f3b9415dfe0 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9a209640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9aa0a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3b940d29c0 0x7f3b940b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9a209640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:53832/0 (socket says 172.21.15.77:53832) 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b99a08640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f3b940cf830 0x7f3b940d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:55922/0 (socket says 172.21.15.77:55922) 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9aa0a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3b940d29c0 0x7f3b940b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:53666/0 (socket says 172.21.15.77:53666) 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.354+0000 7f3b9a209640 1 -- 172.21.15.77:0/1469410326 learned_addr learned my addr 172.21.15.77:0/1469410326 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.355+0000 7f3b9a209640 1 -- 172.21.15.77:0/1469410326 >> v2:172.21.15.158:3300/0 conn(0x7f3b940d29c0 msgr2=0x7f3b940b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.355+0000 7f3b9a209640 1 --2- 172.21.15.77:0/1469410326 >> v2:172.21.15.158:3300/0 conn(0x7f3b940d29c0 0x7f3b940b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.355+0000 7f3b9a209640 1 -- 172.21.15.77:0/1469410326 >> v1:172.21.15.158:6789/0 conn(0x7f3b940cf830 legacy=0x7f3b940d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.355+0000 7f3b9a209640 1 -- 172.21.15.77:0/1469410326 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940b3710 con 0x7f3b9415dfe0 2024-03-06T07:30:31.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.355+0000 7f3b9aa0a640 1 --2- 172.21.15.77:0/1469410326 >> v2:172.21.15.158:3300/0 conn(0x7f3b940d29c0 0x7f3b940b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:31.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9a209640 1 --2- 172.21.15.77:0/1469410326 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cf2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009870 tx=0x7f3b9002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=810f241b0ddc90bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b99207640 1 -- 172.21.15.77:0/1469410326 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90034070 con 0x7f3b9415dfe0 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b99207640 1 -- 172.21.15.77:0/1469410326 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b90030840 con 0x7f3b9415dfe0 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b99207640 1 -- 172.21.15.77:0/1469410326 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90030c10 con 0x7f3b9415dfe0 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9c46d640 1 -- 172.21.15.77:0/1469410326 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 msgr2=0x7f3b940cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/1469410326 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cf2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009870 tx=0x7f3b9002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9c46d640 1 -- 172.21.15.77:0/1469410326 shutdown_connections 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/1469410326 >> v2:172.21.15.158:3300/0 conn(0x7f3b940d29c0 0x7f3b940b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/1469410326 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cf2f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9c46d640 1 -- 172.21.15.77:0/1469410326 >> 172.21.15.77:0/1469410326 conn(0x7f3b940d3cd0 msgr2=0x7f3b940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9c46d640 1 -- 172.21.15.77:0/1469410326 shutdown_connections 2024-03-06T07:30:31.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.356+0000 7f3b9c46d640 1 -- 172.21.15.77:0/1469410326 wait complete. 2024-03-06T07:30:31.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9c46d640 1 Processor -- start 2024-03-06T07:30:31.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9c46d640 1 -- start start 2024-03-06T07:30:31.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9c46d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b940cf830 0x7f3b9414f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9c46d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b940d29c0 0x7f3b940b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9c46d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9c46d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940d1df0 con 0x7f3b940cf830 2024-03-06T07:30:31.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9c46d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940d1c70 con 0x7f3b940d29c0 2024-03-06T07:30:31.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9c46d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9415e3c0 con 0x7f3b9415dfe0 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9a209640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b940cf830 0x7f3b9414f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9a209640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b940cf830 0x7f3b9414f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37414/0 (socket says 172.21.15.77:37414) 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9a209640 1 -- 172.21.15.77:0/2323493308 learned_addr learned my addr 172.21.15.77:0/2323493308 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b99a08640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b940d29c0 0x7f3b940b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.358+0000 7f3b9aa0a640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b9a209640 1 -- 172.21.15.77:0/2323493308 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 msgr2=0x7f3b940cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b9a209640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b9a209640 1 -- 172.21.15.77:0/2323493308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b940d29c0 msgr2=0x7f3b940b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b9a209640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b940d29c0 0x7f3b940b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b9a209640 1 -- 172.21.15.77:0/2323493308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9414f6d0 con 0x7f3b940cf830 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b9a209640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b940cf830 0x7f3b9414f190 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f3b9002b9c0 tx=0x7f3b900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b8affd640 1 -- 172.21.15.77:0/2323493308 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90034040 con 0x7f3b940cf830 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b8affd640 1 -- 172.21.15.77:0/2323493308 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b90039d90 con 0x7f3b940cf830 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b940cc6d0 con 0x7f3b940cf830 2024-03-06T07:30:31.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b8affd640 1 -- 172.21.15.77:0/2323493308 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90043610 con 0x7f3b940cf830 2024-03-06T07:30:31.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.359+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b940ccc90 con 0x7f3b940cf830 2024-03-06T07:30:31.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.361+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9404f330 con 0x7f3b940cf830 2024-03-06T07:30:31.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.361+0000 7f3b8affd640 1 -- 172.21.15.77:0/2323493308 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b9004a060 con 0x7f3b940cf830 2024-03-06T07:30:31.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.361+0000 7f3b8affd640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b7806b4d0 0x7f3b7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.361+0000 7f3b8affd640 1 -- 172.21.15.77:0/2323493308 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b900ac060 con 0x7f3b940cf830 2024-03-06T07:30:31.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.361+0000 7f3b99a08640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b7806b4d0 0x7f3b7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.362+0000 7f3b99a08640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b7806b4d0 0x7f3b7806d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3b84007fd0 tx=0x7f3b84007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.369+0000 7f3b8affd640 1 -- 172.21.15.77:0/2323493308 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b9003e070 con 0x7f3b940cf830 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.586+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b9406c450 con 0x7f3b940cf830 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.587+0000 7f3b8affd640 1 -- 172.21.15.77:0/2323493308 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7f3b9007aee0 con 0x7f3b940cf830 2024-03-06T07:30:31.588 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:30.767116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186,9177]},"id":10}]} 2024-03-06T07:30:31.589 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 103 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.588+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b7806b4d0 msgr2=0x7f3b7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.588+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b7806b4d0 0x7f3b7806d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3b84007fd0 tx=0x7f3b84007e80 comp rx=0 tx=0).stop 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b940cf830 msgr2=0x7f3b9414f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b940cf830 0x7f3b9414f190 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f3b9002b9c0 tx=0x7f3b900046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 shutdown_connections 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3b9415dfe0 0x7f3b940cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b940d29c0 0x7f3b940b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3b7806b4d0 0x7f3b7806d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 --2- 172.21.15.77:0/2323493308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b940cf830 0x7f3b9414f190 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 >> 172.21.15.77:0/2323493308 conn(0x7f3b940d3cd0 msgr2=0x7f3b940b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 shutdown_connections 2024-03-06T07:30:31.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.589+0000 7f3b9c46d640 1 -- 172.21.15.77:0/2323493308 wait complete. 2024-03-06T07:30:31.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:30:31.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:31.708 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:31.721 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"16","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:30:31.721 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/dir_1", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:31.722 INFO:tasks.cephfs.cephfs_test_case:[('/dir_1', 1)] =?= [('/dir_1', 1)] 2024-03-06T07:30:31.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:31.722 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 dir_2) 2024-03-06T07:30:31.813 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:31.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.966+0000 7f0b589c7640 1 Processor -- start 2024-03-06T07:30:31.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.966+0000 7f0b589c7640 1 -- start start 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b589c7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0b540d22b0 0x7f0b540d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b589c7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b540b3430 con 0x7f0b540b41f0 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b589c7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b540b3710 con 0x7f0b5415dfe0 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b589c7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b540d2c50 con 0x7f0b540d22b0 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b52d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0b540b41f0 0x7f0b540b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47002/0 (socket says 172.21.15.77:47002) 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b52d9d640 1 -- 172.21.15.77:0/1097445183 learned_addr learned my addr 172.21.15.77:0/1097445183 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240521349 0 0) 0x7f0b540b3430 con 0x7f0b540b41f0 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b51d9b640 1 --2- 172.21.15.77:0/1097445183 >> v2:172.21.15.158:3300/0 conn(0x7f0b540d22b0 0x7f0b540d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b40003520 con 0x7f0b540b41f0 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.967+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2579206604 0 0) 0x7f0b40003520 con 0x7f0b540b41f0 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 >> v2:172.21.15.158:3300/0 conn(0x7f0b540d22b0 msgr2=0x7f0b540d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 --2- 172.21.15.77:0/1097445183 >> v2:172.21.15.158:3300/0 conn(0x7f0b540d22b0 0x7f0b540d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 >> v1:172.21.15.158:6789/0 conn(0x7f0b5415dfe0 legacy=0x7f0b540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b540b39f0 con 0x7f0b540b41f0 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f0b4c001670 con 0x7f0b540b41f0 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 >> v1:172.21.15.77:6789/0 conn(0x7f0b540b41f0 legacy=0x7f0b540b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b400039b0 0x7f0b40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b40004800 0x7f0b40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b400051c0 0x7f0b4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4000c030 con 0x7f0b40004800 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4000c1e0 con 0x7f0b400039b0 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4000c3e0 con 0x7f0b400051c0 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b51d9b640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b40004800 0x7f0b40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b5259c640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b400039b0 0x7f0b40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b52d9d640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b400051c0 0x7f0b4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b51d9b640 1 -- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b400051c0 msgr2=0x7f0b4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b51d9b640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b400051c0 0x7f0b4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b51d9b640 1 -- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b400039b0 msgr2=0x7f0b40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b51d9b640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b400039b0 0x7f0b40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b51d9b640 1 -- 172.21.15.77:0/1097445183 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b40003520 con 0x7f0b40004800 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b52d9d640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b400051c0 0x7f0b4000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.968+0000 7f0b52d9d640 1 -- 172.21.15.77:0/1097445183 reap_dead start 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.969+0000 7f0b51d9b640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b40004800 0x7f0b40004c80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c009ac0 tx=0x7f0b40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a16e5206202f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.969+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c039070 con 0x7f0b40004800 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.969+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c02c6b0 con 0x7f0b40004800 2024-03-06T07:30:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.969+0000 7f0b5159a640 1 -- 172.21.15.77:0/1097445183 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c009690 con 0x7f0b40004800 2024-03-06T07:30:31.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.970+0000 7f0b589c7640 1 -- 172.21.15.77:0/1097445183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b40004800 msgr2=0x7f0b40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.970+0000 7f0b589c7640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b40004800 0x7f0b40004c80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c009ac0 tx=0x7f0b40005830 comp rx=0 tx=0).stop 2024-03-06T07:30:31.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.970+0000 7f0b589c7640 1 -- 172.21.15.77:0/1097445183 shutdown_connections 2024-03-06T07:30:31.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.970+0000 7f0b589c7640 1 --2- 172.21.15.77:0/1097445183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b40004800 0x7f0b40004c80 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.970+0000 7f0b589c7640 1 -- 172.21.15.77:0/1097445183 >> 172.21.15.77:0/1097445183 conn(0x7f0b540d3cd0 msgr2=0x7f0b540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.970+0000 7f0b589c7640 1 -- 172.21.15.77:0/1097445183 shutdown_connections 2024-03-06T07:30:31.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.970+0000 7f0b589c7640 1 -- 172.21.15.77:0/1097445183 wait complete. 2024-03-06T07:30:31.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.972+0000 7f0b589c7640 1 Processor -- start 2024-03-06T07:30:31.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.972+0000 7f0b589c7640 1 -- start start 2024-03-06T07:30:31.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.973+0000 7f0b589c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b40004800 0x7f0b5415b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.973+0000 7f0b589c7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b5415e8a0 0x7f0b5415ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.973+0000 7f0b589c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b54159da0 0x7f0b5415a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.973+0000 7f0b589c7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b540b3d60 con 0x7f0b54159da0 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.973+0000 7f0b589c7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b540b31e0 con 0x7f0b5415e8a0 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.973+0000 7f0b589c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5415e3c0 con 0x7f0b40004800 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b54159da0 0x7f0b5415a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b54159da0 0x7f0b5415a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37440/0 (socket says 172.21.15.77:37440) 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 -- 172.21.15.77:0/4079061819 learned_addr learned my addr 172.21.15.77:0/4079061819 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b51d9b640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b5415e8a0 0x7f0b5415ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b5259c640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b40004800 0x7f0b5415b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 -- 172.21.15.77:0/4079061819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b40004800 msgr2=0x7f0b5415b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b40004800 0x7f0b5415b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 -- 172.21.15.77:0/4079061819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b5415e8a0 msgr2=0x7f0b5415ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b5415e8a0 0x7f0b5415ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 -- 172.21.15.77:0/4079061819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b54160b40 con 0x7f0b54159da0 2024-03-06T07:30:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b52d9d640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b54159da0 0x7f0b5415a1e0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f0b5408c940 tx=0x7f0b4c00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b4affd640 1 -- 172.21.15.77:0/4079061819 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c019070 con 0x7f0b54159da0 2024-03-06T07:30:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b4affd640 1 -- 172.21.15.77:0/4079061819 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c010510 con 0x7f0b54159da0 2024-03-06T07:30:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5415f410 con 0x7f0b54159da0 2024-03-06T07:30:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b4affd640 1 -- 172.21.15.77:0/4079061819 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c010920 con 0x7f0b54159da0 2024-03-06T07:30:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.974+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5415f9d0 con 0x7f0b54159da0 2024-03-06T07:30:31.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.975+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5415fd10 con 0x7f0b54159da0 2024-03-06T07:30:31.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.976+0000 7f0b4affd640 1 -- 172.21.15.77:0/4079061819 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b4c01c070 con 0x7f0b54159da0 2024-03-06T07:30:31.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.976+0000 7f0b4affd640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b3406b500 0x7f0b3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.976+0000 7f0b4affd640 1 -- 172.21.15.77:0/4079061819 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b4c089620 con 0x7f0b54159da0 2024-03-06T07:30:31.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.976+0000 7f0b5259c640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b3406b500 0x7f0b3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.977+0000 7f0b5259c640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b3406b500 0x7f0b3406d9c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f0b44000f80 tx=0x7f0b44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:31.981+0000 7f0b4affd640 1 -- 172.21.15.77:0/4079061819 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b5415fd10 con 0x7f0b54159da0 2024-03-06T07:30:32.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.242+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b5404f330 con 0x7f0b54159da0 2024-03-06T07:30:32.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.242+0000 7f0b4affd640 1 -- 172.21.15.77:0/4079061819 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7f0b4c058530 con 0x7f0b54159da0 2024-03-06T07:30:32.244 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:32.244 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:30.767116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186,9177]},"id":10}]} 2024-03-06T07:30:32.244 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 103 2024-03-06T07:30:32.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b3406b500 msgr2=0x7f0b3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b3406b500 0x7f0b3406d9c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f0b44000f80 tx=0x7f0b44009040 comp rx=0 tx=0).stop 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b54159da0 msgr2=0x7f0b5415a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b54159da0 0x7f0b5415a1e0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f0b5408c940 tx=0x7f0b4c00e960 comp rx=0 tx=0).stop 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 shutdown_connections 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b3406b500 0x7f0b3406d9c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b54159da0 0x7f0b5415a1e0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b5415e8a0 0x7f0b5415ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 --2- 172.21.15.77:0/4079061819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b40004800 0x7f0b5415b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 >> 172.21.15.77:0/4079061819 conn(0x7f0b540d3cd0 msgr2=0x7f0b540cedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.244+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 shutdown_connections 2024-03-06T07:30:32.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:32.245+0000 7f0b589c7640 1 -- 172.21.15.77:0/4079061819 wait complete. 2024-03-06T07:30:32.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:32.422 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:32.423 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:32.423 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:34.424 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.537+0000 7fe6db62a640 1 Processor -- start 2024-03-06T07:30:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.537+0000 7fe6db62a640 1 -- start start 2024-03-06T07:30:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6db62a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6d415dfe0 0x7fe6d40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6db62a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6db62a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d40b3430 con 0x7fe6d40cf830 2024-03-06T07:30:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6db62a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d40b31e0 con 0x7fe6d415dfe0 2024-03-06T07:30:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6db62a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40b3d60 con 0x7fe6d40d29c0 2024-03-06T07:30:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6d93c6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6d415dfe0 0x7fe6d40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6d8bc5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe6d40cf830 0x7fe6d40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47010/0 (socket says 172.21.15.77:47010) 2024-03-06T07:30:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6d8bc5640 1 -- 172.21.15.77:0/2381964006 learned_addr learned my addr 172.21.15.77:0/2381964006 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6d9bc7640 1 --2- 172.21.15.77:0/2381964006 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6d93c6640 1 -- 172.21.15.77:0/2381964006 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 msgr2=0x7fe6d40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6d93c6640 1 --2- 172.21.15.77:0/2381964006 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6d93c6640 1 -- 172.21.15.77:0/2381964006 >> v1:172.21.15.77:6789/0 conn(0x7fe6d40cf830 legacy=0x7fe6d40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.538+0000 7fe6d93c6640 1 -- 172.21.15.77:0/2381964006 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d40b3710 con 0x7fe6d415dfe0 2024-03-06T07:30:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.539+0000 7fe6d93c6640 1 --2- 172.21.15.77:0/2381964006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40cf2f0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00097a0 tx=0x7fe6d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bcecda07d0248b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.539+0000 7fe6cbfff640 1 -- 172.21.15.77:0/2381964006 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0035020 con 0x7fe6d415dfe0 2024-03-06T07:30:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6cbfff640 1 -- 172.21.15.77:0/2381964006 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0031830 con 0x7fe6d415dfe0 2024-03-06T07:30:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6cbfff640 1 -- 172.21.15.77:0/2381964006 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0031c00 con 0x7fe6d415dfe0 2024-03-06T07:30:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6db62a640 1 -- 172.21.15.77:0/2381964006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 msgr2=0x7fe6d40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6db62a640 1 --2- 172.21.15.77:0/2381964006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40cf2f0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00097a0 tx=0x7fe6d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6db62a640 1 -- 172.21.15.77:0/2381964006 shutdown_connections 2024-03-06T07:30:34.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6db62a640 1 --2- 172.21.15.77:0/2381964006 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6db62a640 1 --2- 172.21.15.77:0/2381964006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40cf2f0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6db62a640 1 -- 172.21.15.77:0/2381964006 >> 172.21.15.77:0/2381964006 conn(0x7fe6d40d3cd0 msgr2=0x7fe6d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6db62a640 1 -- 172.21.15.77:0/2381964006 shutdown_connections 2024-03-06T07:30:34.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.540+0000 7fe6db62a640 1 -- 172.21.15.77:0/2381964006 wait complete. 2024-03-06T07:30:34.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.542+0000 7fe6db62a640 1 Processor -- start 2024-03-06T07:30:34.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.542+0000 7fe6db62a640 1 -- start start 2024-03-06T07:30:34.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6db62a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6d40cf830 0x7fe6d415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6db62a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6db62a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6db62a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40d1df0 con 0x7fe6d415dfe0 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6db62a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40d1c70 con 0x7fe6d40cf830 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6db62a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d415e3c0 con 0x7fe6d40d29c0 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6d9bc7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6d9bc7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37454/0 (socket says 172.21.15.77:37454) 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.543+0000 7fe6d9bc7640 1 -- 172.21.15.77:0/778404866 learned_addr learned my addr 172.21.15.77:0/778404866 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d8bc5640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d93c6640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6d40cf830 0x7fe6d415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d9bc7640 1 -- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 msgr2=0x7fe6d40ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d9bc7640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d9bc7640 1 -- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6d40cf830 msgr2=0x7fe6d415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d9bc7640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6d40cf830 0x7fe6d415ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d9bc7640 1 -- 172.21.15.77:0/778404866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d415b3d0 con 0x7fe6d415dfe0 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d8bc5640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d93c6640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6d40cf830 0x7fe6d415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6d9bc7640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40d02c0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc007e40 tx=0x7fe6cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6c9ffb640 1 -- 172.21.15.77:0/778404866 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc014620 con 0x7fe6d415dfe0 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6c9ffb640 1 -- 172.21.15.77:0/778404866 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc019070 con 0x7fe6d415dfe0 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d40caf80 con 0x7fe6d415dfe0 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6c9ffb640 1 -- 172.21.15.77:0/778404866 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc014a20 con 0x7fe6d415dfe0 2024-03-06T07:30:34.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.544+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d40cb540 con 0x7fe6d415dfe0 2024-03-06T07:30:34.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.546+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d4068090 con 0x7fe6d415dfe0 2024-03-06T07:30:34.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.546+0000 7fe6c9ffb640 1 -- 172.21.15.77:0/778404866 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6cc014bc0 con 0x7fe6d415dfe0 2024-03-06T07:30:34.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.547+0000 7fe6c9ffb640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe6b806b4d0 0x7fe6b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.547+0000 7fe6c9ffb640 1 -- 172.21.15.77:0/778404866 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6cc08aa40 con 0x7fe6d415dfe0 2024-03-06T07:30:34.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.547+0000 7fe6d93c6640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe6b806b4d0 0x7fe6b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.547+0000 7fe6d93c6640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe6b806b4d0 0x7fe6b806d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0036000 tx=0x7fe6d003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.555+0000 7fe6c9ffb640 1 -- 172.21.15.77:0/778404866 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6d4068090 con 0x7fe6d415dfe0 2024-03-06T07:30:34.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.796+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6d404f330 con 0x7fe6d415dfe0 2024-03-06T07:30:34.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.796+0000 7fe6c9ffb640 1 -- 172.21.15.77:0/778404866 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7fe6cc08ae20 con 0x7fe6d415dfe0 2024-03-06T07:30:34.798 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:34.798 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:30.767116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186,9177]},"id":10}]} 2024-03-06T07:30:34.798 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 103 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.798+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe6b806b4d0 msgr2=0x7fe6b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.798+0000 7fe6db62a640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe6b806b4d0 0x7fe6b806d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0036000 tx=0x7fe6d003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.798+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 msgr2=0x7fe6d40d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.798+0000 7fe6db62a640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40d02c0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc007e40 tx=0x7fe6cc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.799+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 shutdown_connections 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.799+0000 7fe6db62a640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe6b806b4d0 0x7fe6b806d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.799+0000 7fe6db62a640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d415dfe0 0x7fe6d40d02c0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.799+0000 7fe6db62a640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe6d40d29c0 0x7fe6d40ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.799+0000 7fe6db62a640 1 --2- 172.21.15.77:0/778404866 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe6d40cf830 0x7fe6d415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.799+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 >> 172.21.15.77:0/778404866 conn(0x7fe6d40d3cd0 msgr2=0x7fe6d406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.799+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 shutdown_connections 2024-03-06T07:30:34.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:34.799+0000 7fe6db62a640 1 -- 172.21.15.77:0/778404866 wait complete. 2024-03-06T07:30:34.811 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:34.931 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:34.931 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:34.931 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:36.932 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:37.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.030+0000 7fdf0641c640 1 Processor -- start 2024-03-06T07:30:37.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.031+0000 7fdf0641c640 1 -- start start 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf0641c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf0015ab00 0x7fdf000b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf0641c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdf000afdd0 0x7fdf0015bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf0641c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf000b4e30 con 0x7fdf000b48f0 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf0641c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf000b5110 con 0x7fdf000afdd0 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf0641c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0015b310 con 0x7fdf0015ab00 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf049b9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdf000afdd0 0x7fdf0015bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdeff7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdf000b48f0 0x7fdf000af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:47016/0 (socket says 172.21.15.77:47016) 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdeff7fe640 1 -- 172.21.15.77:0/3951685341 learned_addr learned my addr 172.21.15.77:0/3951685341 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdefffff640 1 --2- 172.21.15.77:0/3951685341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf0015ab00 0x7fdf000b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf049b9640 1 -- 172.21.15.77:0/3951685341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf0015ab00 msgr2=0x7fdf000b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf049b9640 1 --2- 172.21.15.77:0/3951685341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf0015ab00 0x7fdf000b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf049b9640 1 -- 172.21.15.77:0/3951685341 >> v1:172.21.15.77:6789/0 conn(0x7fdf000b48f0 legacy=0x7fdf000af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdefeffd640 1 -- 172.21.15.77:0/3951685341 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841976964 0 0) 0x7fdf000b4e30 con 0x7fdf000b48f0 2024-03-06T07:30:37.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.032+0000 7fdf049b9640 1 -- 172.21.15.77:0/3951685341 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0015b5f0 con 0x7fdf000afdd0 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.033+0000 7fdf049b9640 1 --2- 172.21.15.77:0/3951685341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000afdd0 0x7fdf0015bb10 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009d90 tx=0x7fdef802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65337407954110b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.033+0000 7fdefeffd640 1 -- 172.21.15.77:0/3951685341 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8039070 con 0x7fdf000afdd0 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.033+0000 7fdefeffd640 1 -- 172.21.15.77:0/3951685341 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef802cc50 con 0x7fdf000afdd0 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.033+0000 7fdefeffd640 1 -- 172.21.15.77:0/3951685341 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8034440 con 0x7fdf000afdd0 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.034+0000 7fdf0641c640 1 -- 172.21.15.77:0/3951685341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000afdd0 msgr2=0x7fdf0015bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.034+0000 7fdf0641c640 1 --2- 172.21.15.77:0/3951685341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000afdd0 0x7fdf0015bb10 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009d90 tx=0x7fdef802bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.034+0000 7fdf0641c640 1 -- 172.21.15.77:0/3951685341 shutdown_connections 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.034+0000 7fdf0641c640 1 --2- 172.21.15.77:0/3951685341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000afdd0 0x7fdf0015bb10 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.034+0000 7fdf0641c640 1 --2- 172.21.15.77:0/3951685341 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf0015ab00 0x7fdf000b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.034+0000 7fdf0641c640 1 -- 172.21.15.77:0/3951685341 >> 172.21.15.77:0/3951685341 conn(0x7fdf0006c0c0 msgr2=0x7fdf0006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.034+0000 7fdf0641c640 1 -- 172.21.15.77:0/3951685341 shutdown_connections 2024-03-06T07:30:37.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.034+0000 7fdf0641c640 1 -- 172.21.15.77:0/3951685341 wait complete. 2024-03-06T07:30:37.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.036+0000 7fdf0641c640 1 Processor -- start 2024-03-06T07:30:37.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.036+0000 7fdf0641c640 1 -- start start 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdf0641c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf000afdd0 0x7fdf0014f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdf0641c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000b48f0 0x7fdf001f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdf0641c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf0015ab00 0x7fdf00153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdf0641c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0015b060 con 0x7fdf000b48f0 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdf0641c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0015aee0 con 0x7fdf0015ab00 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdf0641c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0015c050 con 0x7fdf000afdd0 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdeff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000b48f0 0x7fdf001f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdeff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000b48f0 0x7fdf001f0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37474/0 (socket says 172.21.15.77:37474) 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdeff7fe640 1 -- 172.21.15.77:0/595440230 learned_addr learned my addr 172.21.15.77:0/595440230 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdefffff640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf000afdd0 0x7fdf0014f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.037+0000 7fdf049b9640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf0015ab00 0x7fdf00153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdeff7fe640 1 -- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf000afdd0 msgr2=0x7fdf0014f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdeff7fe640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf000afdd0 0x7fdf0014f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdeff7fe640 1 -- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf0015ab00 msgr2=0x7fdf00153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdeff7fe640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf0015ab00 0x7fdf00153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdeff7fe640 1 -- 172.21.15.77:0/595440230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf00153c10 con 0x7fdf000b48f0 2024-03-06T07:30:37.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdefffff640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf000afdd0 0x7fdf0014f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:37.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdeff7fe640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000b48f0 0x7fdf001f0580 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fdef400c4d0 tx=0x7fdef400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdefcff9640 1 -- 172.21.15.77:0/595440230 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4018070 con 0x7fdf000b48f0 2024-03-06T07:30:37.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdefcff9640 1 -- 172.21.15.77:0/595440230 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4007500 con 0x7fdf000b48f0 2024-03-06T07:30:37.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdefcff9640 1 -- 172.21.15.77:0/595440230 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4004a20 con 0x7fdf000b48f0 2024-03-06T07:30:37.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf00153ef0 con 0x7fdf000b48f0 2024-03-06T07:30:37.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.038+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf000ce070 con 0x7fdf000b48f0 2024-03-06T07:30:37.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.039+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf000ce270 con 0x7fdf000b48f0 2024-03-06T07:30:37.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.040+0000 7fdefcff9640 1 -- 172.21.15.77:0/595440230 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdef4004bc0 con 0x7fdf000b48f0 2024-03-06T07:30:37.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.040+0000 7fdefcff9640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdeec06b4d0 0x7fdeec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.040+0000 7fdefcff9640 1 -- 172.21.15.77:0/595440230 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdef40894f0 con 0x7fdf000b48f0 2024-03-06T07:30:37.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.040+0000 7fdefffff640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdeec06b4d0 0x7fdeec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.041+0000 7fdefffff640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdeec06b4d0 0x7fdeec06d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdef0006fd0 tx=0x7fdef0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.045+0000 7fdefcff9640 1 -- 172.21.15.77:0/595440230 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf000ce270 con 0x7fdf000b48f0 2024-03-06T07:30:37.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.377+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf0004f330 con 0x7fdf000b48f0 2024-03-06T07:30:37.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.377+0000 7fdefcff9640 1 -- 172.21.15.77:0/595440230 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7fdef4058400 con 0x7fdf000b48f0 2024-03-06T07:30:37.380 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:37.380 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:30.767116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186,9177]},"id":10}]} 2024-03-06T07:30:37.380 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 103 2024-03-06T07:30:37.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdeec06b4d0 msgr2=0x7fdeec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdeec06b4d0 0x7fdeec06d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdef0006fd0 tx=0x7fdef0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:37.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000b48f0 msgr2=0x7fdf001f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000b48f0 0x7fdf001f0580 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fdef400c4d0 tx=0x7fdef400c990 comp rx=0 tx=0).stop 2024-03-06T07:30:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 shutdown_connections 2024-03-06T07:30:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdeec06b4d0 0x7fdeec06d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf000b48f0 0x7fdf001f0580 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf0015ab00 0x7fdf00153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 --2- 172.21.15.77:0/595440230 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdf000afdd0 0x7fdf0014f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 >> 172.21.15.77:0/595440230 conn(0x7fdf0006c0c0 msgr2=0x7fdf00071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 shutdown_connections 2024-03-06T07:30:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:37.379+0000 7fdf0641c640 1 -- 172.21.15.77:0/595440230 wait complete. 2024-03-06T07:30:37.393 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:37.509 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:37.510 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:37.510 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:39.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9fb3e640 1 Processor -- start 2024-03-06T07:30:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9fb3e640 1 -- start start 2024-03-06T07:30:39.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9fb3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e9815dfe0 0x7f2e980cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9fb3e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2e980d29c0 0x7f2e980b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9fb3e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980b3430 con 0x7f2e980cf830 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9fb3e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e980b31e0 con 0x7f2e980d29c0 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9fb3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b3d60 con 0x7f2e9815dfe0 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9d0d9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2e980cf830 0x7f2e980d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51004/0 (socket says 172.21.15.77:51004) 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.611+0000 7f2e9d0d9640 1 -- 172.21.15.77:0/2083511720 learned_addr learned my addr 172.21.15.77:0/2083511720 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9e0db640 1 --2- 172.21.15.77:0/2083511720 >> v2:172.21.15.158:3300/0 conn(0x7f2e980d29c0 0x7f2e980b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9d8da640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e9815dfe0 0x7f2e980cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456195420 0 0) 0x7f2e980b3430 con 0x7f2e980cf830 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e84003520 con 0x7f2e980cf830 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 595910547 0 0) 0x7f2e84003520 con 0x7f2e980cf830 2024-03-06T07:30:39.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e9815dfe0 msgr2=0x7f2e980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e9815dfe0 0x7f2e980cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 >> v2:172.21.15.158:3300/0 conn(0x7f2e980d29c0 msgr2=0x7f2e980b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 --2- 172.21.15.77:0/2083511720 >> v2:172.21.15.158:3300/0 conn(0x7f2e980d29c0 0x7f2e980b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980b3710 con 0x7f2e980cf830 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f2e88001670 con 0x7f2e980cf830 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 >> v1:172.21.15.77:6789/0 conn(0x7f2e980cf830 legacy=0x7f2e980d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e840039b0 0x7f2e84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004800 0x7f2e84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e840051c0 0x7f2e8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b31e0 con 0x7f2e84004800 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b3d60 con 0x7f2e840039b0 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8400bf00 con 0x7f2e840051c0 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9d8da640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e9815dfe0 0x7f2e980cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:39.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9e0db640 1 --2- 172.21.15.77:0/2083511720 >> v2:172.21.15.158:3300/0 conn(0x7f2e980d29c0 0x7f2e980b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9d0d9640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004800 0x7f2e84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9d8da640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e840039b0 0x7f2e84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.612+0000 7f2e9e0db640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e840051c0 0x7f2e8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9d0d9640 1 -- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e840051c0 msgr2=0x7f2e8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9d0d9640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e840051c0 0x7f2e8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9d0d9640 1 -- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e840039b0 msgr2=0x7f2e84003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9d0d9640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e840039b0 0x7f2e84003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9d0d9640 1 -- 172.21.15.77:0/2083511720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e84003520 con 0x7f2e84004800 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9d8da640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e840039b0 0x7f2e84003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9d0d9640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004800 0x7f2e84004c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800c290 tx=0x7f2e84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefcb4ed05a83d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88007d50 con 0x7f2e84004800 2024-03-06T07:30:39.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e88030070 con 0x7f2e84004800 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.613+0000 7f2e9c8d8640 1 -- 172.21.15.77:0/2083511720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88005d30 con 0x7f2e84004800 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.614+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2083511720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004800 msgr2=0x7f2e84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.614+0000 7f2e9fb3e640 1 --2- 172.21.15.77:0/2083511720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004800 0x7f2e84004c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800c290 tx=0x7f2e84005830 comp rx=0 tx=0).stop 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.614+0000 7f2e9e0db640 1 -- 172.21.15.77:0/2083511720 reap_dead start 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.614+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2083511720 shutdown_connections 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.614+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2083511720 >> 172.21.15.77:0/2083511720 conn(0x7f2e980d3cd0 msgr2=0x7f2e980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.614+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2083511720 shutdown_connections 2024-03-06T07:30:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.614+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2083511720 wait complete. 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.615+0000 7f2e9fb3e640 1 Processor -- start 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9fb3e640 1 -- start start 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9fb3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e84004800 0x7f2e98161100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9fb3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e980cada0 0x7f2e980c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9fb3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e981f75e0 0x7f2e980b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9fb3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980d1df0 con 0x7f2e981f75e0 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9fb3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980d1c70 con 0x7f2e980cada0 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9fb3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9815e3c0 con 0x7f2e84004800 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9e0db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e981f75e0 0x7f2e980b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9e0db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e981f75e0 0x7f2e980b4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36432/0 (socket says 172.21.15.77:36432) 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.616+0000 7f2e9e0db640 1 -- 172.21.15.77:0/2525589626 learned_addr learned my addr 172.21.15.77:0/2525589626 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9d8da640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e84004800 0x7f2e98161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9d0d9640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e980cada0 0x7f2e980c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9e0db640 1 -- 172.21.15.77:0/2525589626 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e84004800 msgr2=0x7f2e98161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9e0db640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e84004800 0x7f2e98161100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9e0db640 1 -- 172.21.15.77:0/2525589626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e980cada0 msgr2=0x7f2e980c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9e0db640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e980cada0 0x7f2e980c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9e0db640 1 -- 172.21.15.77:0/2525589626 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e981f6e00 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9e0db640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e981f75e0 0x7f2e980b4c40 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009f90 tx=0x7f2e900115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e8e7fc640 1 -- 172.21.15.77:0/2525589626 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e90003990 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e8e7fc640 1 -- 172.21.15.77:0/2525589626 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9001b070 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9814da50 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e8e7fc640 1 -- 172.21.15.77:0/2525589626 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e90003b50 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.617+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9814dd30 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.618+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e981511b0 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.619+0000 7f2e8e7fc640 1 -- 172.21.15.77:0/2525589626 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e90005d10 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.619+0000 7f2e8e7fc640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.619+0000 7f2e8e7fc640 1 -- 172.21.15.77:0/2525589626 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e9008a8c0 con 0x7f2e981f75e0 2024-03-06T07:30:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.619+0000 7f2e9d8da640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.620+0000 7f2e9d8da640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2e98067990 tx=0x7f2e94008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.623+0000 7f2e8e7fc640 1 -- 172.21.15.77:0/2525589626 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e90059740 con 0x7f2e981f75e0 2024-03-06T07:30:39.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.841+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e9804f330 con 0x7f2e981f75e0 2024-03-06T07:30:39.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.841+0000 7f2e8e7fc640 1 -- 172.21.15.77:0/2525589626 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7f2e90059040 con 0x7f2e981f75e0 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:30.767116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186,9177]},"id":10}]} 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 103 2024-03-06T07:30:39.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e7c06b4d0 msgr2=0x7f2e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2e98067990 tx=0x7f2e94008040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e981f75e0 msgr2=0x7f2e980b4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e981f75e0 0x7f2e980b4c40 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009f90 tx=0x7f2e900115a0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 shutdown_connections 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2e7c06b4d0 0x7f2e7c06d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e981f75e0 0x7f2e980b4c40 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2e980cada0 0x7f2e980c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 --2- 172.21.15.77:0/2525589626 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2e84004800 0x7f2e98161100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 >> 172.21.15.77:0/2525589626 conn(0x7f2e980d3cd0 msgr2=0x7f2e980b88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.844+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 shutdown_connections 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:39.845+0000 7f2e9fb3e640 1 -- 172.21.15.77:0/2525589626 wait complete. 2024-03-06T07:30:39.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:39.973 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:39.973 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:39.974 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:41.975 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.116+0000 7f4820156640 1 Processor -- start 2024-03-06T07:30:42.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.117+0000 7f4820156640 1 -- start start 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.117+0000 7f4820156640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f481815ab00 0x7f48180b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.117+0000 7f4820156640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180b4e30 con 0x7f48180b48f0 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.117+0000 7f4820156640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180b5110 con 0x7f48180afdd0 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.117+0000 7f4820156640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481815b310 con 0x7f481815ab00 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.117+0000 7f481d6f1640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f48180b48f0 0x7f48180af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51020/0 (socket says 172.21.15.77:51020) 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481d6f1640 1 -- 172.21.15.77:0/1736596235 learned_addr learned my addr 172.21.15.77:0/1736596235 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481def2640 1 --2- 172.21.15.77:0/1736596235 >> v2:172.21.15.158:3300/0 conn(0x7f481815ab00 0x7f48180b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984417645 0 0) 0x7f48180b4e30 con 0x7f48180b48f0 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4804003520 con 0x7f48180b48f0 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005650423 0 0) 0x7f4804003520 con 0x7f48180b48f0 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 >> v2:172.21.15.158:3300/0 conn(0x7f481815ab00 msgr2=0x7f48180b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 --2- 172.21.15.77:0/1736596235 >> v2:172.21.15.158:3300/0 conn(0x7f481815ab00 0x7f48180b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 >> v1:172.21.15.158:6789/0 conn(0x7f48180afdd0 legacy=0x7f481815bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481815b5f0 con 0x7f48180b48f0 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f4808001670 con 0x7f48180b48f0 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 >> v1:172.21.15.77:6789/0 conn(0x7f48180b48f0 legacy=0x7f48180af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48040039b0 0x7f4804003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4804004800 0x7f4804004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040051c0 0x7f480400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481815b310 con 0x7f48040039b0 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480400c060 con 0x7f48040051c0 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480400c260 con 0x7f4804004800 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481def2640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48040039b0 0x7f4804003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481d6f1640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4804004800 0x7f4804004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.118+0000 7f481e6f3640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040051c0 0x7f480400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.119+0000 7f481def2640 1 -- 172.21.15.77:0/1736596235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4804004800 msgr2=0x7f4804004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.119+0000 7f481def2640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4804004800 0x7f4804004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.119+0000 7f481def2640 1 -- 172.21.15.77:0/1736596235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040051c0 msgr2=0x7f480400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.119+0000 7f481def2640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48040051c0 0x7f480400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.119+0000 7f481e6f3640 1 -- 172.21.15.77:0/1736596235 reap_dead start 2024-03-06T07:30:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.119+0000 7f481def2640 1 -- 172.21.15.77:0/1736596235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4804003520 con 0x7f48040039b0 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f481def2640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48040039b0 0x7f4804003dd0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f4814009ac0 tx=0x7f4804005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b01ac12a989d3d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814039070 con 0x7f48040039b0 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481402c6b0 con 0x7f48040039b0 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f481cef0640 1 -- 172.21.15.77:0/1736596235 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814009690 con 0x7f48040039b0 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f4820156640 1 -- 172.21.15.77:0/1736596235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48040039b0 msgr2=0x7f4804003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f4820156640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48040039b0 0x7f4804003dd0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f4814009ac0 tx=0x7f4804005830 comp rx=0 tx=0).stop 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f4820156640 1 -- 172.21.15.77:0/1736596235 shutdown_connections 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f4820156640 1 --2- 172.21.15.77:0/1736596235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48040039b0 0x7f4804003dd0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.120+0000 7f4820156640 1 -- 172.21.15.77:0/1736596235 >> 172.21.15.77:0/1736596235 conn(0x7f481806c0c0 msgr2=0x7f481806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.121+0000 7f4820156640 1 -- 172.21.15.77:0/1736596235 shutdown_connections 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.121+0000 7f4820156640 1 -- 172.21.15.77:0/1736596235 wait complete. 2024-03-06T07:30:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.122+0000 7f4820156640 1 Processor -- start 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.122+0000 7f4820156640 1 -- start start 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.122+0000 7f4820156640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48040039b0 0x7f48180a9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.122+0000 7f4820156640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48181587a0 0x7f481814a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.122+0000 7f4820156640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48181e7670 0x7f48181e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.122+0000 7f4820156640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481815b060 con 0x7f48181e7670 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.122+0000 7f4820156640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481815aee0 con 0x7f48181587a0 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.122+0000 7f4820156640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180b43b0 con 0x7f48040039b0 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48181e7670 0x7f48181e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48181e7670 0x7f48181e7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36442/0 (socket says 172.21.15.77:36442) 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 -- 172.21.15.77:0/1493253028 learned_addr learned my addr 172.21.15.77:0/1493253028 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481d6f1640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48181587a0 0x7f481814a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481def2640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48040039b0 0x7f48180a9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 -- 172.21.15.77:0/1493253028 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48040039b0 msgr2=0x7f48180a9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48040039b0 0x7f48180a9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 -- 172.21.15.77:0/1493253028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48181587a0 msgr2=0x7f481814a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48181587a0 0x7f481814a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 -- 172.21.15.77:0/1493253028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4818159fb0 con 0x7f48181e7670 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f481e6f3640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48181e7670 0x7f48181e7b20 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f48180b5560 tx=0x7f481000fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f480e7fc640 1 -- 172.21.15.77:0/1493253028 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810003970 con 0x7f48181e7670 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f480e7fc640 1 -- 172.21.15.77:0/1493253028 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4810018070 con 0x7f48181e7670 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4818158f10 con 0x7f48181e7670 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f480e7fc640 1 -- 172.21.15.77:0/1493253028 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810003d70 con 0x7f48181e7670 2024-03-06T07:30:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.123+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48181594d0 con 0x7f48181e7670 2024-03-06T07:30:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.125+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481804f330 con 0x7f48181e7670 2024-03-06T07:30:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.125+0000 7f480e7fc640 1 -- 172.21.15.77:0/1493253028 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48100050d0 con 0x7f48181e7670 2024-03-06T07:30:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.125+0000 7f480e7fc640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f47fc06b500 0x7f47fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.125+0000 7f480e7fc640 1 -- 172.21.15.77:0/1493253028 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4810089920 con 0x7f48181e7670 2024-03-06T07:30:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.125+0000 7f481def2640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f47fc06b500 0x7f47fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.126+0000 7f481def2640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f47fc06b500 0x7f47fc06d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4814036000 tx=0x7f4814030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.134+0000 7f480e7fc640 1 -- 172.21.15.77:0/1493253028 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4810058230 con 0x7f48181e7670 2024-03-06T07:30:42.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.352+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f481806c780 con 0x7f48181e7670 2024-03-06T07:30:42.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.352+0000 7f480e7fc640 1 -- 172.21.15.77:0/1493253028 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7f481005c2d0 con 0x7f48181e7670 2024-03-06T07:30:42.354 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:42.354 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:30.767116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186,9177]},"id":10}]} 2024-03-06T07:30:42.355 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 103 2024-03-06T07:30:42.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f47fc06b500 msgr2=0x7f47fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f47fc06b500 0x7f47fc06d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4814036000 tx=0x7f4814030000 comp rx=0 tx=0).stop 2024-03-06T07:30:42.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48181e7670 msgr2=0x7f48181e7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48181e7670 0x7f48181e7b20 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f48180b5560 tx=0x7f481000fc00 comp rx=0 tx=0).stop 2024-03-06T07:30:42.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 shutdown_connections 2024-03-06T07:30:42.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f47fc06b500 0x7f47fc06d9c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48181e7670 0x7f48181e7b20 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48181587a0 0x7f481814a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 --2- 172.21.15.77:0/1493253028 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48040039b0 0x7f48180a9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 >> 172.21.15.77:0/1493253028 conn(0x7f481806c0c0 msgr2=0x7f48180b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 shutdown_connections 2024-03-06T07:30:42.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.355+0000 7f4820156640 1 -- 172.21.15.77:0/1493253028 wait complete. 2024-03-06T07:30:42.368 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:42.511 INFO:teuthology.orchestra.run.smithi077.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/dir_2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-06T07:30:42.512 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/dir_2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:42.512 INFO:tasks.cephfs.cephfs_test_case:[('/dir_2', 0)] =?= [('/dir_2', 0)] 2024-03-06T07:30:42.512 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:30:42.512 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:42.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:42.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.668+0000 7f6e3635e640 1 Processor -- start 2024-03-06T07:30:42.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.668+0000 7f6e3635e640 1 -- start start 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e3635e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015bdd0 0x7f6e300c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e3635e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6e3015fa10 0x7f6e300ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e3635e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e300cebe0 con 0x7f6e300d16e0 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e3635e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e300ceec0 con 0x7f6e3015fa10 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e3635e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300cf1a0 con 0x7f6e3015bdd0 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e348fb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6e300d16e0 0x7f6e300d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51036/0 (socket says 172.21.15.77:51036) 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e348fb640 1 -- 172.21.15.77:0/1436254861 learned_addr learned my addr 172.21.15.77:0/1436254861 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370582921 0 0) 0x7f6e300cebe0 con 0x7f6e300d16e0 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e2ffff640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015bdd0 0x7f6e300c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e2f7fe640 1 --2- 172.21.15.77:0/1436254861 >> v2:172.21.15.158:3300/0 conn(0x7f6e3015fa10 0x7f6e300ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.669+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e18003520 con 0x7f6e300d16e0 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61790566 0 0) 0x7f6e18003520 con 0x7f6e300d16e0 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015bdd0 msgr2=0x7f6e300c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015bdd0 0x7f6e300c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 >> v2:172.21.15.158:3300/0 conn(0x7f6e3015fa10 msgr2=0x7f6e300ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 --2- 172.21.15.77:0/1436254861 >> v2:172.21.15.158:3300/0 conn(0x7f6e3015fa10 0x7f6e300ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e300cc8e0 con 0x7f6e300d16e0 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f6e280019f0 con 0x7f6e300d16e0 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 >> v1:172.21.15.77:6789/0 conn(0x7f6e300d16e0 legacy=0x7f6e300d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180039b0 0x7f6e18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015bdd0 0x7f6e18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e18004d40 0x7f6e180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300ceec0 con 0x7f6e18004d40 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300cf1a0 con 0x7f6e180039b0 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1800b9c0 con 0x7f6e3015bdd0 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e348fb640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e18004d40 0x7f6e180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2f7fe640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015bdd0 0x7f6e18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e2ffff640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180039b0 0x7f6e18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e348fb640 1 -- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015bdd0 msgr2=0x7f6e18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e348fb640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015bdd0 0x7f6e18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e348fb640 1 -- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180039b0 msgr2=0x7f6e18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e348fb640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e180039b0 0x7f6e18003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.670+0000 7f6e348fb640 1 -- 172.21.15.77:0/1436254861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e18003520 con 0x7f6e18004d40 2024-03-06T07:30:42.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.671+0000 7f6e348fb640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e18004d40 0x7f6e180051c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f6e280013d0 tx=0x7f6e18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654a95f7724eb030 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.671+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2800a4e0 con 0x7f6e18004d40 2024-03-06T07:30:42.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.671+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e28034650 con 0x7f6e18004d40 2024-03-06T07:30:42.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.671+0000 7f6e2effd640 1 -- 172.21.15.77:0/1436254861 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28034a40 con 0x7f6e18004d40 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.672+0000 7f6e3635e640 1 -- 172.21.15.77:0/1436254861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e18004d40 msgr2=0x7f6e180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.672+0000 7f6e3635e640 1 --2- 172.21.15.77:0/1436254861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e18004d40 0x7f6e180051c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f6e280013d0 tx=0x7f6e18005830 comp rx=0 tx=0).stop 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.672+0000 7f6e348fb640 1 -- 172.21.15.77:0/1436254861 reap_dead start 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.672+0000 7f6e3635e640 1 -- 172.21.15.77:0/1436254861 shutdown_connections 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.672+0000 7f6e3635e640 1 -- 172.21.15.77:0/1436254861 >> 172.21.15.77:0/1436254861 conn(0x7f6e3009a9f0 msgr2=0x7f6e3009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.672+0000 7f6e3635e640 1 -- 172.21.15.77:0/1436254861 shutdown_connections 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.672+0000 7f6e3635e640 1 -- 172.21.15.77:0/1436254861 wait complete. 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.674+0000 7f6e3635e640 1 Processor -- start 2024-03-06T07:30:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.674+0000 7f6e3635e640 1 -- start start 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e3635e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e301f1dd0 0x7f6e3015f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e3635e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015c330 0x7f6e301f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e3635e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301f26f0 0x7f6e300b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e3635e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300ce7b0 con 0x7f6e301f26f0 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e3635e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3015c1b0 con 0x7f6e301f1dd0 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e3635e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300ce930 con 0x7f6e3015c330 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e348fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301f26f0 0x7f6e300b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e348fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301f26f0 0x7f6e300b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36454/0 (socket says 172.21.15.77:36454) 2024-03-06T07:30:42.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e348fb640 1 -- 172.21.15.77:0/3101040350 learned_addr learned my addr 172.21.15.77:0/3101040350 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e2f7fe640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015c330 0x7f6e301f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.675+0000 7f6e2ffff640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e301f1dd0 0x7f6e3015f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e348fb640 1 -- 172.21.15.77:0/3101040350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015c330 msgr2=0x7f6e301f21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e348fb640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015c330 0x7f6e301f21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e348fb640 1 -- 172.21.15.77:0/3101040350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e301f1dd0 msgr2=0x7f6e3015f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e348fb640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e301f1dd0 0x7f6e3015f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e348fb640 1 -- 172.21.15.77:0/3101040350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3015e440 con 0x7f6e301f26f0 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e348fb640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301f26f0 0x7f6e300b47b0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f6e280013d0 tx=0x7f6e28001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e2cff9640 1 -- 172.21.15.77:0/3101040350 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28002040 con 0x7f6e301f26f0 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e2cff9640 1 -- 172.21.15.77:0/3101040350 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2800a4c0 con 0x7f6e301f26f0 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3015ca10 con 0x7f6e301f26f0 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e2cff9640 1 -- 172.21.15.77:0/3101040350 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e280017b0 con 0x7f6e301f26f0 2024-03-06T07:30:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.676+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3015cfd0 con 0x7f6e301f26f0 2024-03-06T07:30:42.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.677+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3004f330 con 0x7f6e301f26f0 2024-03-06T07:30:42.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.677+0000 7f6e2cff9640 1 -- 172.21.15.77:0/3101040350 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e28047020 con 0x7f6e301f26f0 2024-03-06T07:30:42.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.678+0000 7f6e2cff9640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e1c06b4d0 0x7f6e1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.678+0000 7f6e2cff9640 1 -- 172.21.15.77:0/3101040350 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e280abb30 con 0x7f6e301f26f0 2024-03-06T07:30:42.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.678+0000 7f6e2ffff640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e1c06b4d0 0x7f6e1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.678+0000 7f6e2ffff640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e1c06b4d0 0x7f6e1c06d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009a80 tx=0x7f6e20008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.682+0000 7f6e2cff9640 1 -- 172.21.15.77:0/3101040350 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e2804e020 con 0x7f6e301f26f0 2024-03-06T07:30:42.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.861+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e300b7a40 con 0x7f6e301f26f0 2024-03-06T07:30:42.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.863+0000 7f6e2cff9640 1 -- 172.21.15.77:0/3101040350 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16667 (secure 0 0 0) 0x7f6e2807aa40 con 0x7f6e301f26f0 2024-03-06T07:30:42.865 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:42.865 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":141,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:20.563427+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e1c06b4d0 msgr2=0x7f6e1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e1c06b4d0 0x7f6e1c06d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009a80 tx=0x7f6e20008040 comp rx=0 tx=0).stop 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301f26f0 msgr2=0x7f6e300b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301f26f0 0x7f6e300b47b0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f6e280013d0 tx=0x7f6e28001360 comp rx=0 tx=0).stop 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 shutdown_connections 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6e1c06b4d0 0x7f6e1c06d990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301f26f0 0x7f6e300b47b0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6e3015c330 0x7f6e301f21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 --2- 172.21.15.77:0/3101040350 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6e301f1dd0 0x7f6e3015f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 >> 172.21.15.77:0/3101040350 conn(0x7f6e3009a9f0 msgr2=0x7f6e300702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 shutdown_connections 2024-03-06T07:30:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.865+0000 7f6e3635e640 1 -- 172.21.15.77:0/3101040350 wait complete. 2024-03-06T07:30:42.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.674704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.414541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:42.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:42.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.987+0000 7ff270064640 1 Processor -- start 2024-03-06T07:30:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.987+0000 7ff270064640 1 -- start start 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff270064640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff2680cfa90 0x7ff2680cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff270064640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff2680b33a0 0x7ff2680b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff270064640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff268158ac0 con 0x7ff2681586e0 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff270064640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26815b2a0 con 0x7ff2680cfa90 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff270064640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26815b540 con 0x7ff2680b33a0 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff26d5ff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff2680cfa90 0x7ff2680cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff26de00640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff2681586e0 0x7ff2680aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51046/0 (socket says 172.21.15.77:51046) 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff26d5ff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff2680cfa90 0x7ff2680cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36470/0 (socket says 172.21.15.77:36470) 2024-03-06T07:30:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.988+0000 7ff26de00640 1 -- 172.21.15.77:0/1454832134 learned_addr learned my addr 172.21.15.77:0/1454832134 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.989+0000 7ff26e601640 1 --2- 172.21.15.77:0/1454832134 >> v2:172.21.15.158:3300/0 conn(0x7ff2680b33a0 0x7ff2680b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.989+0000 7ff26d5ff640 1 -- 172.21.15.77:0/1454832134 >> v2:172.21.15.158:3300/0 conn(0x7ff2680b33a0 msgr2=0x7ff2680b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.989+0000 7ff26d5ff640 1 --2- 172.21.15.77:0/1454832134 >> v2:172.21.15.158:3300/0 conn(0x7ff2680b33a0 0x7ff2680b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.989+0000 7ff26d5ff640 1 -- 172.21.15.77:0/1454832134 >> v1:172.21.15.77:6789/0 conn(0x7ff2681586e0 legacy=0x7ff2680aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.989+0000 7ff26d5ff640 1 -- 172.21.15.77:0/1454832134 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26815b820 con 0x7ff2680cfa90 2024-03-06T07:30:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.990+0000 7ff26d5ff640 1 --2- 172.21.15.77:0/1454832134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff2680cff40 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ff2680b2ae0 tx=0x7ff25802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55a1603e550de4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.990+0000 7ff26cdfe640 1 -- 172.21.15.77:0/1454832134 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258035020 con 0x7ff2680cfa90 2024-03-06T07:30:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.990+0000 7ff26cdfe640 1 -- 172.21.15.77:0/1454832134 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff258031830 con 0x7ff2680cfa90 2024-03-06T07:30:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.990+0000 7ff26cdfe640 1 -- 172.21.15.77:0/1454832134 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258031c00 con 0x7ff2680cfa90 2024-03-06T07:30:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.991+0000 7ff270064640 1 -- 172.21.15.77:0/1454832134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 msgr2=0x7ff2680cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.991+0000 7ff270064640 1 --2- 172.21.15.77:0/1454832134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff2680cff40 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ff2680b2ae0 tx=0x7ff25802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.991+0000 7ff270064640 1 -- 172.21.15.77:0/1454832134 shutdown_connections 2024-03-06T07:30:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.991+0000 7ff270064640 1 --2- 172.21.15.77:0/1454832134 >> v2:172.21.15.158:3300/0 conn(0x7ff2680b33a0 0x7ff2680b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.991+0000 7ff270064640 1 --2- 172.21.15.77:0/1454832134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff2680cff40 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.991+0000 7ff270064640 1 -- 172.21.15.77:0/1454832134 >> 172.21.15.77:0/1454832134 conn(0x7ff26806b190 msgr2=0x7ff26806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.991+0000 7ff270064640 1 -- 172.21.15.77:0/1454832134 shutdown_connections 2024-03-06T07:30:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.991+0000 7ff270064640 1 -- 172.21.15.77:0/1454832134 wait complete. 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.992+0000 7ff270064640 1 Processor -- start 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.992+0000 7ff270064640 1 -- start start 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff270064640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2680b33a0 0x7ff268147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff270064640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff268153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff270064640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2681586e0 0x7ff2680a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff270064640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680af150 con 0x7ff2680cfa90 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff270064640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26815bde0 con 0x7ff2680b33a0 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff270064640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff268071350 con 0x7ff2681586e0 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26d5ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff268153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26d5ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff268153e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36484/0 (socket says 172.21.15.77:36484) 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26d5ff640 1 -- 172.21.15.77:0/658080528 learned_addr learned my addr 172.21.15.77:0/658080528 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26de00640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2680b33a0 0x7ff268147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26e601640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2681586e0 0x7ff2680a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26d5ff640 1 -- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2681586e0 msgr2=0x7ff2680a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26d5ff640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2681586e0 0x7ff2680a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26d5ff640 1 -- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2680b33a0 msgr2=0x7ff268147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26d5ff640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2680b33a0 0x7ff268147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26d5ff640 1 -- 172.21.15.77:0/658080528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2680ac6f0 con 0x7ff2680cfa90 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.993+0000 7ff26e601640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2681586e0 0x7ff2680a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.994+0000 7ff26d5ff640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff268153e60 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7ff25802bee0 tx=0x7ff2580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.994+0000 7ff25e7fc640 1 -- 172.21.15.77:0/658080528 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258041070 con 0x7ff2680cfa90 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.994+0000 7ff25e7fc640 1 -- 172.21.15.77:0/658080528 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25803aa30 con 0x7ff2680cfa90 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.994+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2680ac9d0 con 0x7ff2680cfa90 2024-03-06T07:30:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.994+0000 7ff25e7fc640 1 -- 172.21.15.77:0/658080528 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258035040 con 0x7ff2680cfa90 2024-03-06T07:30:42.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.994+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2680ad270 con 0x7ff2680cfa90 2024-03-06T07:30:42.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.995+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26815c250 con 0x7ff2680cfa90 2024-03-06T07:30:42.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.996+0000 7ff25e7fc640 1 -- 172.21.15.77:0/658080528 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff258038020 con 0x7ff2680cfa90 2024-03-06T07:30:42.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.996+0000 7ff25e7fc640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24c06b4d0 0x7ff24c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.996+0000 7ff26de00640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24c06b4d0 0x7ff24c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.996+0000 7ff25e7fc640 1 -- 172.21.15.77:0/658080528 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2580ab7d0 con 0x7ff2680cfa90 2024-03-06T07:30:42.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:42.997+0000 7ff26de00640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24c06b4d0 0x7ff24c06d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff2640012e0 tx=0x7ff264001a30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.001+0000 7ff25e7fc640 1 -- 172.21.15.77:0/658080528 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff26815c250 con 0x7ff2680cfa90 2024-03-06T07:30:43.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.192+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2680b4540 con 0x7ff2680cfa90 2024-03-06T07:30:43.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.193+0000 7ff25e7fc640 1 -- 172.21.15.77:0/658080528 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff2680b4540 con 0x7ff2680cfa90 2024-03-06T07:30:43.194 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:30:43.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24c06b4d0 msgr2=0x7ff24c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24c06b4d0 0x7ff24c06d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff2640012e0 tx=0x7ff264001a30 comp rx=0 tx=0).stop 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 msgr2=0x7ff268153e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff268153e60 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7ff25802bee0 tx=0x7ff2580046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 shutdown_connections 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff24c06b4d0 0x7ff24c06d990 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2680cfa90 0x7ff268153e60 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff2681586e0 0x7ff2680a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 --2- 172.21.15.77:0/658080528 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff2680b33a0 0x7ff268147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.194+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 >> 172.21.15.77:0/658080528 conn(0x7ff26806b190 msgr2=0x7ff2680aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.195+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 shutdown_connections 2024-03-06T07:30:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.195+0000 7ff270064640 1 -- 172.21.15.77:0/658080528 wait complete. 2024-03-06T07:30:43.211 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:43.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.313+0000 7fb92eca4640 1 Processor -- start 2024-03-06T07:30:43.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.313+0000 7fb92eca4640 1 -- start start 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92eca4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9280d4860 0x7fb9280d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92eca4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9280b0e30 0x7fb9280b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92eca4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9281618b0 con 0x7fb9281614d0 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92eca4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9280d4cc0 con 0x7fb9280b0e30 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92eca4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280cf400 con 0x7fb9280d4860 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92d241640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9280b0e30 0x7fb9280b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92d241640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9280b0e30 0x7fb9280b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36498/0 (socket says 172.21.15.77:36498) 2024-03-06T07:30:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92d241640 1 -- 172.21.15.77:0/805287763 learned_addr learned my addr 172.21.15.77:0/805287763 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb91ffff640 1 --2- 172.21.15.77:0/805287763 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9280d4860 0x7fb9280d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92d241640 1 -- 172.21.15.77:0/805287763 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9280d4860 msgr2=0x7fb9280d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92d241640 1 --2- 172.21.15.77:0/805287763 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9280d4860 0x7fb9280d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92d241640 1 -- 172.21.15.77:0/805287763 >> v1:172.21.15.158:6789/0 conn(0x7fb9281614d0 legacy=0x7fb9280d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.314+0000 7fb92d241640 1 -- 172.21.15.77:0/805287763 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9280cf6a0 con 0x7fb9280b0e30 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.315+0000 7fb92d241640 1 --2- 172.21.15.77:0/805287763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280b0e30 0x7fb9280b12c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fb920009e50 tx=0x7fb92002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8cab47064f377c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.315+0000 7fb91f7fe640 1 -- 172.21.15.77:0/805287763 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92002cb00 con 0x7fb9280b0e30 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.315+0000 7fb91f7fe640 1 -- 172.21.15.77:0/805287763 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92002cca0 con 0x7fb9280b0e30 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.315+0000 7fb91f7fe640 1 -- 172.21.15.77:0/805287763 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920034610 con 0x7fb9280b0e30 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.316+0000 7fb92eca4640 1 -- 172.21.15.77:0/805287763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280b0e30 msgr2=0x7fb9280b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.316+0000 7fb92eca4640 1 --2- 172.21.15.77:0/805287763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280b0e30 0x7fb9280b12c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fb920009e50 tx=0x7fb92002bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.316+0000 7fb92eca4640 1 -- 172.21.15.77:0/805287763 shutdown_connections 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.316+0000 7fb92eca4640 1 --2- 172.21.15.77:0/805287763 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9280d4860 0x7fb9280d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.316+0000 7fb92eca4640 1 --2- 172.21.15.77:0/805287763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280b0e30 0x7fb9280b12c0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.316+0000 7fb92eca4640 1 -- 172.21.15.77:0/805287763 >> 172.21.15.77:0/805287763 conn(0x7fb9280d3d40 msgr2=0x7fb9280d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.316+0000 7fb92eca4640 1 -- 172.21.15.77:0/805287763 shutdown_connections 2024-03-06T07:30:43.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.316+0000 7fb92eca4640 1 -- 172.21.15.77:0/805287763 wait complete. 2024-03-06T07:30:43.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.318+0000 7fb92eca4640 1 Processor -- start 2024-03-06T07:30:43.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.318+0000 7fb92eca4640 1 -- start start 2024-03-06T07:30:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92eca4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9280b0e30 0x7fb9280c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92eca4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280d4860 0x7fb9280b92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92eca4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9281614d0 0x7fb9280cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92eca4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280b0910 con 0x7fb9280d4860 2024-03-06T07:30:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92eca4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280cfc60 con 0x7fb9280b0e30 2024-03-06T07:30:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92eca4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280d2560 con 0x7fb9281614d0 2024-03-06T07:30:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280d4860 0x7fb9280b92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280d4860 0x7fb9280b92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36506/0 (socket says 172.21.15.77:36506) 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb91ffff640 1 -- 172.21.15.77:0/236485020 learned_addr learned my addr 172.21.15.77:0/236485020 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92ca40640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9280b0e30 0x7fb9280c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92d241640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9281614d0 0x7fb9280cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb91ffff640 1 -- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9281614d0 msgr2=0x7fb9280cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb91ffff640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9281614d0 0x7fb9280cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb91ffff640 1 -- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9280b0e30 msgr2=0x7fb9280c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb91ffff640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9280b0e30 0x7fb9280c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb91ffff640 1 -- 172.21.15.77:0/236485020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9280c7e50 con 0x7fb9280d4860 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92d241640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9281614d0 0x7fb9280cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.319+0000 7fb92ca40640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9280b0e30 0x7fb9280c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.320+0000 7fb91ffff640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280d4860 0x7fb9280b92f0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fb91800d800 tx=0x7fb91800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.320+0000 7fb91d7fa640 1 -- 172.21.15.77:0/236485020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800b6c0 con 0x7fb9280d4860 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.320+0000 7fb91d7fa640 1 -- 172.21.15.77:0/236485020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb918019070 con 0x7fb9280d4860 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.320+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9280c8130 con 0x7fb9280d4860 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.320+0000 7fb91d7fa640 1 -- 172.21.15.77:0/236485020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800ba80 con 0x7fb9280d4860 2024-03-06T07:30:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.320+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9280c86f0 con 0x7fb9280d4860 2024-03-06T07:30:43.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.321+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb928068ff0 con 0x7fb9280d4860 2024-03-06T07:30:43.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.322+0000 7fb91d7fa640 1 -- 172.21.15.77:0/236485020 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb91800bc20 con 0x7fb9280d4860 2024-03-06T07:30:43.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.322+0000 7fb91d7fa640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb91406b4d0 0x7fb91406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.322+0000 7fb92ca40640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb91406b4d0 0x7fb91406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.322+0000 7fb91d7fa640 1 -- 172.21.15.77:0/236485020 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb91808a6b0 con 0x7fb9280d4860 2024-03-06T07:30:43.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.323+0000 7fb92ca40640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb91406b4d0 0x7fb91406d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb924001660 tx=0x7fb924009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.327+0000 7fb91d7fa640 1 -- 172.21.15.77:0/236485020 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb918059530 con 0x7fb9280d4860 2024-03-06T07:30:43.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.504+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9280d0040 con 0x7fb9280d4860 2024-03-06T07:30:43.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.505+0000 7fb91d7fa640 1 -- 172.21.15.77:0/236485020 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb918058e30 con 0x7fb9280d4860 2024-03-06T07:30:43.507 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb91406b4d0 msgr2=0x7fb91406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb91406b4d0 0x7fb91406d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb924001660 tx=0x7fb924009040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280d4860 msgr2=0x7fb9280b92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280d4860 0x7fb9280b92f0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fb91800d800 tx=0x7fb91800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 shutdown_connections 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb91406b4d0 0x7fb91406d990 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9280d4860 0x7fb9280b92f0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb9281614d0 0x7fb9280cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 --2- 172.21.15.77:0/236485020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb9280b0e30 0x7fb9280c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 >> 172.21.15.77:0/236485020 conn(0x7fb9280d3d40 msgr2=0x7fb92806b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 shutdown_connections 2024-03-06T07:30:43.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.507+0000 7fb92eca4640 1 -- 172.21.15.77:0/236485020 wait complete. 2024-03-06T07:30:43.519 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:43.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.622+0000 7f34eeafb640 1 Processor -- start 2024-03-06T07:30:43.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.622+0000 7f34eeafb640 1 -- start start 2024-03-06T07:30:43.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34eeafb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e81614d0 0x7f34e80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34eeafb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e80d2480 0x7f34e80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34eeafb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e81618b0 con 0x7f34e80d1910 2024-03-06T07:30:43.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34eeafb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e80cf400 con 0x7f34e80d2480 2024-03-06T07:30:43.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34eeafb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80cf6a0 con 0x7f34e81614d0 2024-03-06T07:30:43.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34ed098640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e80d2480 0x7f34e80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34ed098640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e80d2480 0x7f34e80b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36510/0 (socket says 172.21.15.77:36510) 2024-03-06T07:30:43.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34ed098640 1 -- 172.21.15.77:0/1288528314 learned_addr learned my addr 172.21.15.77:0/1288528314 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:43.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34ec897640 1 --2- 172.21.15.77:0/1288528314 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e81614d0 0x7f34e80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34ed098640 1 -- 172.21.15.77:0/1288528314 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e81614d0 msgr2=0x7f34e80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34ed098640 1 --2- 172.21.15.77:0/1288528314 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e81614d0 0x7f34e80d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34ed098640 1 -- 172.21.15.77:0/1288528314 >> v1:172.21.15.158:6789/0 conn(0x7f34e80d1910 legacy=0x7f34e80d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.623+0000 7f34ed098640 1 -- 172.21.15.77:0/1288528314 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80cf980 con 0x7f34e80d2480 2024-03-06T07:30:43.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.624+0000 7f34ed098640 1 --2- 172.21.15.77:0/1288528314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d2480 0x7f34e80b0e30 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009e50 tx=0x7f34e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee677933dd1279d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.624+0000 7f34df7fe640 1 -- 172.21.15.77:0/1288528314 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e002ca70 con 0x7f34e80d2480 2024-03-06T07:30:43.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.624+0000 7f34df7fe640 1 -- 172.21.15.77:0/1288528314 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e002cc10 con 0x7f34e80d2480 2024-03-06T07:30:43.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.624+0000 7f34df7fe640 1 -- 172.21.15.77:0/1288528314 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e00343f0 con 0x7f34e80d2480 2024-03-06T07:30:43.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.625+0000 7f34eeafb640 1 -- 172.21.15.77:0/1288528314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d2480 msgr2=0x7f34e80b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.625+0000 7f34eeafb640 1 --2- 172.21.15.77:0/1288528314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d2480 0x7f34e80b0e30 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009e50 tx=0x7f34e002bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.625+0000 7f34eeafb640 1 -- 172.21.15.77:0/1288528314 shutdown_connections 2024-03-06T07:30:43.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.625+0000 7f34eeafb640 1 --2- 172.21.15.77:0/1288528314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d2480 0x7f34e80b0e30 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.625+0000 7f34eeafb640 1 --2- 172.21.15.77:0/1288528314 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e81614d0 0x7f34e80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.625+0000 7f34eeafb640 1 -- 172.21.15.77:0/1288528314 >> 172.21.15.77:0/1288528314 conn(0x7f34e80d3d40 msgr2=0x7f34e80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.625+0000 7f34eeafb640 1 -- 172.21.15.77:0/1288528314 shutdown_connections 2024-03-06T07:30:43.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.625+0000 7f34eeafb640 1 -- 172.21.15.77:0/1288528314 wait complete. 2024-03-06T07:30:43.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.626+0000 7f34eeafb640 1 Processor -- start 2024-03-06T07:30:43.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.626+0000 7f34eeafb640 1 -- start start 2024-03-06T07:30:43.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34eeafb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d1910 0x7f34e815bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34eeafb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e80d2480 0x7f34e81f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34eeafb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34e81614d0 0x7f34e80cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34eeafb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80b1370 con 0x7f34e80d1910 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34eeafb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80b0910 con 0x7f34e81614d0 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34eeafb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80b14f0 con 0x7f34e80d2480 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34ec897640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d1910 0x7f34e815bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34ec897640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d1910 0x7f34e815bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36512/0 (socket says 172.21.15.77:36512) 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34ec897640 1 -- 172.21.15.77:0/2842755138 learned_addr learned my addr 172.21.15.77:0/2842755138 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.627+0000 7f34dffff640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e80d2480 0x7f34e81f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34ed098640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34e81614d0 0x7f34e80cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34ec897640 1 -- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e80d2480 msgr2=0x7f34e81f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34ec897640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e80d2480 0x7f34e81f3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34ec897640 1 -- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34e81614d0 msgr2=0x7f34e80cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34ec897640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34e81614d0 0x7f34e80cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34ec897640 1 -- 172.21.15.77:0/2842755138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80cbe60 con 0x7f34e80d1910 2024-03-06T07:30:43.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34dffff640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e80d2480 0x7f34e81f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34ec897640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d1910 0x7f34e815bba0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f34e400c820 tx=0x7f34e400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34dd7fa640 1 -- 172.21.15.77:0/2842755138 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4015070 con 0x7f34e80d1910 2024-03-06T07:30:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34dd7fa640 1 -- 172.21.15.77:0/2842755138 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e4004020 con 0x7f34e80d1910 2024-03-06T07:30:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e81f4750 con 0x7f34e80d1910 2024-03-06T07:30:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34dd7fa640 1 -- 172.21.15.77:0/2842755138 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e40043f0 con 0x7f34e80d1910 2024-03-06T07:30:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.628+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e81f4d10 con 0x7f34e80d1910 2024-03-06T07:30:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.629+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e8068ff0 con 0x7f34e80d1910 2024-03-06T07:30:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.630+0000 7f34dd7fa640 1 -- 172.21.15.77:0/2842755138 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34e40077d0 con 0x7f34e80d1910 2024-03-06T07:30:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.630+0000 7f34dd7fa640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f34d406b500 0x7f34d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.631+0000 7f34dffff640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f34d406b500 0x7f34d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.631+0000 7f34dd7fa640 1 -- 172.21.15.77:0/2842755138 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34e4089ec0 con 0x7f34e80d1910 2024-03-06T07:30:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.631+0000 7f34dffff640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f34d406b500 0x7f34d406d9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f34e815f430 tx=0x7f34d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.635+0000 7f34dd7fa640 1 -- 172.21.15.77:0/2842755138 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34e4058d40 con 0x7f34e80d1910 2024-03-06T07:30:43.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.815+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34e80d0a50 con 0x7f34e80d1910 2024-03-06T07:30:43.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.816+0000 7f34dd7fa640 1 -- 172.21.15.77:0/2842755138 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34e4058640 con 0x7f34e80d1910 2024-03-06T07:30:43.818 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.817+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f34d406b500 msgr2=0x7f34d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.817+0000 7f34eeafb640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f34d406b500 0x7f34d406d9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f34e815f430 tx=0x7f34d8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.817+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d1910 msgr2=0x7f34e815bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.817+0000 7f34eeafb640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d1910 0x7f34e815bba0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f34e400c820 tx=0x7f34e400cce0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.818+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 shutdown_connections 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.818+0000 7f34eeafb640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f34e81614d0 0x7f34e80cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.818+0000 7f34eeafb640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f34e80d2480 0x7f34e81f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.818+0000 7f34eeafb640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f34d406b500 0x7f34d406d9c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.818+0000 7f34eeafb640 1 --2- 172.21.15.77:0/2842755138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e80d1910 0x7f34e815bba0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.818+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 >> 172.21.15.77:0/2842755138 conn(0x7f34e80d3d40 msgr2=0x7f34e80d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.818+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 shutdown_connections 2024-03-06T07:30:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.818+0000 7f34eeafb640 1 -- 172.21.15.77:0/2842755138 wait complete. 2024-03-06T07:30:43.830 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.930+0000 7fef644fb640 1 Processor -- start 2024-03-06T07:30:43.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.930+0000 7fef644fb640 1 -- start start 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef644fb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef5c0b4460 0x7fef5c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef644fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef644fb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c0d22c0 con 0x7fef5c162970 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef644fb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5c161b00 con 0x7fef5c0b4460 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef644fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0cecc0 con 0x7fef5c161cf0 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62297640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef5c0b4460 0x7fef5c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62297640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef5c0b4460 0x7fef5c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36518/0 (socket says 172.21.15.77:36518) 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62297640 1 -- 172.21.15.77:0/2036559302 learned_addr learned my addr 172.21.15.77:0/2036559302 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62297640 1 -- 172.21.15.77:0/2036559302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 msgr2=0x7fef5c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62a98640 1 --2- 172.21.15.77:0/2036559302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62297640 1 --2- 172.21.15.77:0/2036559302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62297640 1 -- 172.21.15.77:0/2036559302 >> v1:172.21.15.158:6789/0 conn(0x7fef5c162970 legacy=0x7fef5c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62297640 1 -- 172.21.15.77:0/2036559302 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c0d25a0 con 0x7fef5c0b4460 2024-03-06T07:30:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.931+0000 7fef62a98640 1 --2- 172.21.15.77:0/2036559302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.932+0000 7fef62297640 1 --2- 172.21.15.77:0/2036559302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c0d4800 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fef58009870 tx=0x7fef5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cc0d878de8e8b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.932+0000 7fef61295640 1 -- 172.21.15.77:0/2036559302 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58035020 con 0x7fef5c0b4460 2024-03-06T07:30:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.932+0000 7fef61295640 1 -- 172.21.15.77:0/2036559302 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef58031640 con 0x7fef5c0b4460 2024-03-06T07:30:43.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.932+0000 7fef61295640 1 -- 172.21.15.77:0/2036559302 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5802d040 con 0x7fef5c0b4460 2024-03-06T07:30:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.933+0000 7fef644fb640 1 -- 172.21.15.77:0/2036559302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 msgr2=0x7fef5c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.933+0000 7fef644fb640 1 --2- 172.21.15.77:0/2036559302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c0d4800 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fef58009870 tx=0x7fef5802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.933+0000 7fef644fb640 1 -- 172.21.15.77:0/2036559302 shutdown_connections 2024-03-06T07:30:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.933+0000 7fef644fb640 1 --2- 172.21.15.77:0/2036559302 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.933+0000 7fef644fb640 1 --2- 172.21.15.77:0/2036559302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c0d4800 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.933+0000 7fef644fb640 1 -- 172.21.15.77:0/2036559302 >> 172.21.15.77:0/2036559302 conn(0x7fef5c0d3ce0 msgr2=0x7fef5c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.933+0000 7fef644fb640 1 -- 172.21.15.77:0/2036559302 shutdown_connections 2024-03-06T07:30:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.933+0000 7fef644fb640 1 -- 172.21.15.77:0/2036559302 wait complete. 2024-03-06T07:30:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.934+0000 7fef644fb640 1 Processor -- start 2024-03-06T07:30:43.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.934+0000 7fef644fb640 1 -- start start 2024-03-06T07:30:43.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef644fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef644fb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c0cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef644fb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef5c162970 0x7fef5c0b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef644fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0ce950 con 0x7fef5c0b4460 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef644fb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0d2b60 con 0x7fef5c162970 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef644fb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0b4840 con 0x7fef5c161cf0 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c156330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36534/0 (socket says 172.21.15.77:36534) 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 -- 172.21.15.77:0/526871020 learned_addr learned my addr 172.21.15.77:0/526871020 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef61a96640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c0cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 -- 172.21.15.77:0/526871020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 msgr2=0x7fef5c0cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62a98640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef5c162970 0x7fef5c0b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c0cc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 -- 172.21.15.77:0/526871020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef5c162970 msgr2=0x7fef5c0b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef5c162970 0x7fef5c0b24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 -- 172.21.15.77:0/526871020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c0b6aa0 con 0x7fef5c0b4460 2024-03-06T07:30:43.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef62297640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c156330 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fef5802b930 tx=0x7fef580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef52ffd640 1 -- 172.21.15.77:0/526871020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58040070 con 0x7fef5c0b4460 2024-03-06T07:30:43.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef52ffd640 1 -- 172.21.15.77:0/526871020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef58039bf0 con 0x7fef5c0b4460 2024-03-06T07:30:43.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef52ffd640 1 -- 172.21.15.77:0/526871020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58035040 con 0x7fef5c0b4460 2024-03-06T07:30:43.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.936+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c0ccfe0 con 0x7fef5c0b4460 2024-03-06T07:30:43.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.937+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c0ccd00 con 0x7fef5c0b4460 2024-03-06T07:30:43.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.938+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c04f330 con 0x7fef5c0b4460 2024-03-06T07:30:43.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.938+0000 7fef52ffd640 1 -- 172.21.15.77:0/526871020 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef5804b060 con 0x7fef5c0b4460 2024-03-06T07:30:43.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.938+0000 7fef52ffd640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fef4006b4d0 0x7fef4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.938+0000 7fef52ffd640 1 -- 172.21.15.77:0/526871020 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef580abc20 con 0x7fef5c0b4460 2024-03-06T07:30:43.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.938+0000 7fef61a96640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fef4006b4d0 0x7fef4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.939+0000 7fef61a96640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fef4006b4d0 0x7fef4006d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0b8230 tx=0x7fef4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:43.946+0000 7fef52ffd640 1 -- 172.21.15.77:0/526871020 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef5807ab30 con 0x7fef5c0b4460 2024-03-06T07:30:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.159+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fef5c06b180 con 0x7fef5c0b4460 2024-03-06T07:30:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.160+0000 7fef52ffd640 1 -- 172.21.15.77:0/526871020 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v141) v1 ==== 80+0+0 (secure 0 0 0) 0x7fef5807a430 con 0x7fef5c0b4460 2024-03-06T07:30:44.162 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:30:44.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fef4006b4d0 msgr2=0x7fef4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fef4006b4d0 0x7fef4006d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0b8230 tx=0x7fef4c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 msgr2=0x7fef5c156330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c156330 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fef5802b930 tx=0x7fef580046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 shutdown_connections 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fef5c162970 0x7fef5c0b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fef5c161cf0 0x7fef5c0cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fef4006b4d0 0x7fef4006d990 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 --2- 172.21.15.77:0/526871020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5c0b4460 0x7fef5c156330 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 >> 172.21.15.77:0/526871020 conn(0x7fef5c0d3ce0 msgr2=0x7fef5c0d09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.162+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 shutdown_connections 2024-03-06T07:30:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.163+0000 7fef644fb640 1 -- 172.21.15.77:0/526871020 wait complete. 2024-03-06T07:30:44.175 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:30:44.175 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:44.191 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.332+0000 7faba3e6b640 1 Processor -- start 2024-03-06T07:30:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.333+0000 7faba3e6b640 1 -- start start 2024-03-06T07:30:44.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.333+0000 7faba3e6b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fab9c15dfe0 0x7fab9c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.333+0000 7faba3e6b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c0b3430 con 0x7fab9c0cf830 2024-03-06T07:30:44.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.333+0000 7faba3e6b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c0b3710 con 0x7fab9c0d29c0 2024-03-06T07:30:44.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.333+0000 7faba3e6b640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9c0b31e0 con 0x7fab9c15dfe0 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.333+0000 7faba1406640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fab9c0cf830 0x7fab9c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51048/0 (socket says 172.21.15.77:51048) 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.333+0000 7faba1406640 1 -- 172.21.15.77:0/4204695354 learned_addr learned my addr 172.21.15.77:0/4204695354 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212513239 0 0) 0x7fab9c0b3430 con 0x7fab9c0cf830 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab88003520 con 0x7fab9c0cf830 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba1c07640 1 --2- 172.21.15.77:0/4204695354 >> v2:172.21.15.158:3300/0 conn(0x7fab9c15dfe0 0x7fab9c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852069136 0 0) 0x7fab88003520 con 0x7fab9c0cf830 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 >> v2:172.21.15.158:3300/0 conn(0x7fab9c15dfe0 msgr2=0x7fab9c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 --2- 172.21.15.77:0/4204695354 >> v2:172.21.15.158:3300/0 conn(0x7fab9c15dfe0 0x7fab9c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 >> v1:172.21.15.158:6789/0 conn(0x7fab9c0d29c0 legacy=0x7fab9c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c0b39f0 con 0x7fab9c0cf830 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fab8c001670 con 0x7fab9c0cf830 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 >> v1:172.21.15.77:6789/0 conn(0x7fab9c0cf830 legacy=0x7fab9c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab880039b0 0x7fab88003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab88004800 0x7fab88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab880051c0 0x7fab8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0b31e0 con 0x7fab88004800 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8800c090 con 0x7fab880039b0 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8800c290 con 0x7fab880051c0 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba1406640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab88004800 0x7fab88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba1c07640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab880039b0 0x7fab88003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba2408640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab880051c0 0x7fab8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba1406640 1 -- 172.21.15.77:0/4204695354 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab880051c0 msgr2=0x7fab8800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.334+0000 7faba1406640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab880051c0 0x7fab8800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.335+0000 7faba1406640 1 -- 172.21.15.77:0/4204695354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab880039b0 msgr2=0x7fab88003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.335+0000 7faba1406640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab880039b0 0x7fab88003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.335+0000 7faba2408640 1 -- 172.21.15.77:0/4204695354 reap_dead start 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.335+0000 7faba1406640 1 -- 172.21.15.77:0/4204695354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab88003520 con 0x7fab88004800 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba1406640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab88004800 0x7fab88004c80 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00c210 tx=0x7fab88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca2d0fa1803c92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8c007d10 con 0x7fab88004800 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8c030070 con 0x7fab88004800 2024-03-06T07:30:44.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba0c05640 1 -- 172.21.15.77:0/4204695354 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8c005d50 con 0x7fab88004800 2024-03-06T07:30:44.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba3e6b640 1 -- 172.21.15.77:0/4204695354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab88004800 msgr2=0x7fab88004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba3e6b640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab88004800 0x7fab88004c80 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00c210 tx=0x7fab88005830 comp rx=0 tx=0).stop 2024-03-06T07:30:44.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba3e6b640 1 -- 172.21.15.77:0/4204695354 shutdown_connections 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba3e6b640 1 --2- 172.21.15.77:0/4204695354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab88004800 0x7fab88004c80 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba3e6b640 1 -- 172.21.15.77:0/4204695354 >> 172.21.15.77:0/4204695354 conn(0x7fab9c0d3cd0 msgr2=0x7fab9c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.336+0000 7faba3e6b640 1 -- 172.21.15.77:0/4204695354 shutdown_connections 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.337+0000 7faba3e6b640 1 -- 172.21.15.77:0/4204695354 wait complete. 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba3e6b640 1 Processor -- start 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba3e6b640 1 -- start start 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba3e6b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab88004800 0x7fab9c0ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba3e6b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab9c1f40a0 0x7fab9c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba3e6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab9c0b8dc0 0x7fab9c0b9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba3e6b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c15e3c0 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba3e6b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0b3d60 con 0x7fab9c1f40a0 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba3e6b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c15e540 con 0x7fab88004800 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba2408640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab9c0b8dc0 0x7fab9c0b9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba2408640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab9c0b8dc0 0x7fab9c0b9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36550/0 (socket says 172.21.15.77:36550) 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.338+0000 7faba2408640 1 -- 172.21.15.77:0/3439075110 learned_addr learned my addr 172.21.15.77:0/3439075110 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:44.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba1c07640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab88004800 0x7fab9c0ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba1406640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab9c1f40a0 0x7fab9c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba2408640 1 -- 172.21.15.77:0/3439075110 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab88004800 msgr2=0x7fab9c0ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba2408640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab88004800 0x7fab9c0ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba2408640 1 -- 172.21.15.77:0/3439075110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab9c1f40a0 msgr2=0x7fab9c0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba2408640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab9c1f40a0 0x7fab9c0b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba2408640 1 -- 172.21.15.77:0/3439075110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c0b7220 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba2408640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab9c0b8dc0 0x7fab9c0b9240 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fab9800f8b0 tx=0x7fab9800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7fab927fc640 1 -- 172.21.15.77:0/3439075110 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9800fea0 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7fab927fc640 1 -- 172.21.15.77:0/3439075110 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab98018070 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c0b7500 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7fab927fc640 1 -- 172.21.15.77:0/3439075110 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98003da0 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.339+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c0b7ac0 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.340+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c04f330 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.341+0000 7fab927fc640 1 -- 172.21.15.77:0/3439075110 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab980050d0 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.341+0000 7fab927fc640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8006b4d0 0x7fab8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.341+0000 7faba1c07640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8006b4d0 0x7fab8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.341+0000 7fab927fc640 1 -- 172.21.15.77:0/3439075110 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab98089f10 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.342+0000 7faba1c07640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8006b4d0 0x7fab8006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fab9c050c30 tx=0x7fab94009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.346+0000 7fab927fc640 1 -- 172.21.15.77:0/3439075110 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab98058d90 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.561+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab9c0ce100 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.562+0000 7fab927fc640 1 -- 172.21.15.77:0/3439075110 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16667 (secure 0 0 0) 0x7fab98058690 con 0x7fab9c0b8dc0 2024-03-06T07:30:44.564 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:44.564 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":141,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:20.563427+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:44.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8006b4d0 msgr2=0x7fab8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8006b4d0 0x7fab8006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fab9c050c30 tx=0x7fab94009190 comp rx=0 tx=0).stop 2024-03-06T07:30:44.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab9c0b8dc0 msgr2=0x7fab9c0b9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab9c0b8dc0 0x7fab9c0b9240 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fab9800f8b0 tx=0x7fab9800fcb0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 shutdown_connections 2024-03-06T07:30:44.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fab8006b4d0 0x7fab8006d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab9c0b8dc0 0x7fab9c0b9240 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab9c1f40a0 0x7fab9c0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 --2- 172.21.15.77:0/3439075110 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab88004800 0x7fab9c0ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 >> 172.21.15.77:0/3439075110 conn(0x7fab9c0d3cd0 msgr2=0x7fab9c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 shutdown_connections 2024-03-06T07:30:44.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.564+0000 7faba3e6b640 1 -- 172.21.15.77:0/3439075110 wait complete. 2024-03-06T07:30:44.577 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.674704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.414541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:44.577 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:44.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.679+0000 7f16f3f81640 1 Processor -- start 2024-03-06T07:30:44.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.680+0000 7f16f3f81640 1 -- start start 2024-03-06T07:30:44.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.680+0000 7f16f3f81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16ec1614d0 0x7f16ec0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.680+0000 7f16f3f81640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec1618b0 con 0x7f16ec0d1910 2024-03-06T07:30:44.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.680+0000 7f16f3f81640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec0cf400 con 0x7f16ec0d2480 2024-03-06T07:30:44.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.680+0000 7f16f3f81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0cf6a0 con 0x7f16ec1614d0 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f151c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f16ec0d1910 0x7f16ec0d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51064/0 (socket says 172.21.15.77:51064) 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f151c640 1 -- 172.21.15.77:0/1978930880 learned_addr learned my addr 172.21.15.77:0/1978930880 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608067205 0 0) 0x7f16ec1618b0 con 0x7f16ec0d1910 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8003520 con 0x7f16ec0d1910 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f1d1d640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16ec1614d0 0x7f16ec0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4046680459 0 0) 0x7f16d8003520 con 0x7f16ec0d1910 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16ec1614d0 msgr2=0x7f16ec0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16ec1614d0 0x7f16ec0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 >> v1:172.21.15.158:6789/0 conn(0x7f16ec0d2480 legacy=0x7f16ec0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec0cf980 con 0x7f16ec0d1910 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f16e0001670 con 0x7f16ec0d1910 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 >> v1:172.21.15.77:6789/0 conn(0x7f16ec0d1910 legacy=0x7f16ec0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16d80039e0 0x7f16d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16ec1614d0 0x7f16d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004d40 0x7f16d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0cf6a0 con 0x7f16d8004d40 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d800ba70 con 0x7f16d80039e0 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d800bc70 con 0x7f16ec1614d0 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f251e640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004d40 0x7f16d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f151c640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16ec1614d0 0x7f16d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.681+0000 7f16f1d1d640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16d80039e0 0x7f16d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.682+0000 7f16f251e640 1 -- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16ec1614d0 msgr2=0x7f16d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.682+0000 7f16f251e640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16ec1614d0 0x7f16d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.682+0000 7f16f251e640 1 -- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16d80039e0 msgr2=0x7f16d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.682+0000 7f16f251e640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16d80039e0 0x7f16d8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.682+0000 7f16f251e640 1 -- 172.21.15.77:0/1978930880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8003520 con 0x7f16d8004d40 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.682+0000 7f16f251e640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004d40 0x7f16d80051c0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0d28f0 tx=0x7f16d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818d14a0a4586853 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4001a70 con 0x7f16d8004d40 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e4003b70 con 0x7f16d8004d40 2024-03-06T07:30:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f0d1b640 1 -- 172.21.15.77:0/1978930880 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e40050d0 con 0x7f16d8004d40 2024-03-06T07:30:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f3f81640 1 -- 172.21.15.77:0/1978930880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004d40 msgr2=0x7f16d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f3f81640 1 --2- 172.21.15.77:0/1978930880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004d40 0x7f16d80051c0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0d28f0 tx=0x7f16d8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f251e640 1 -- 172.21.15.77:0/1978930880 reap_dead start 2024-03-06T07:30:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f3f81640 1 -- 172.21.15.77:0/1978930880 shutdown_connections 2024-03-06T07:30:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f3f81640 1 -- 172.21.15.77:0/1978930880 >> 172.21.15.77:0/1978930880 conn(0x7f16ec0d3d40 msgr2=0x7f16ec0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f3f81640 1 -- 172.21.15.77:0/1978930880 shutdown_connections 2024-03-06T07:30:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.683+0000 7f16f3f81640 1 -- 172.21.15.77:0/1978930880 wait complete. 2024-03-06T07:30:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.684+0000 7f16f3f81640 1 Processor -- start 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.684+0000 7f16f3f81640 1 -- start start 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f3f81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16d8004d40 0x7f16ec09e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f3f81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec09ec00 0x7f16ec0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f3f81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16ec09fca0 0x7f16ec0a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f3f81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0d4860 con 0x7f16ec09ec00 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f3f81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0b0910 con 0x7f16ec09fca0 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f3f81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0d49e0 con 0x7f16d8004d40 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f151c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec09ec00 0x7f16ec0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f151c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec09ec00 0x7f16ec0b1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36568/0 (socket says 172.21.15.77:36568) 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f151c640 1 -- 172.21.15.77:0/4021038780 learned_addr learned my addr 172.21.15.77:0/4021038780 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f1d1d640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16d8004d40 0x7f16ec09e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f251e640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16ec09fca0 0x7f16ec0a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f151c640 1 -- 172.21.15.77:0/4021038780 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16d8004d40 msgr2=0x7f16ec09e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f151c640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16d8004d40 0x7f16ec09e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.685+0000 7f16f151c640 1 -- 172.21.15.77:0/4021038780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16ec09fca0 msgr2=0x7f16ec0a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.686+0000 7f16f151c640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16ec09fca0 0x7f16ec0a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.686+0000 7f16f151c640 1 -- 172.21.15.77:0/4021038780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec09f510 con 0x7f16ec09ec00 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.686+0000 7f16f151c640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec09ec00 0x7f16ec0b1ff0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f16e0007340 tx=0x7f16e0007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.686+0000 7f16de7fc640 1 -- 172.21.15.77:0/4021038780 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000f940 con 0x7f16ec09ec00 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.686+0000 7f16de7fc640 1 -- 172.21.15.77:0/4021038780 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e0018070 con 0x7f16ec09ec00 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.686+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec152e70 con 0x7f16ec09ec00 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.686+0000 7f16de7fc640 1 -- 172.21.15.77:0/4021038780 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000fd40 con 0x7f16ec09ec00 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.686+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec09f780 con 0x7f16ec09ec00 2024-03-06T07:30:44.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.687+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec09f950 con 0x7f16ec09ec00 2024-03-06T07:30:44.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.688+0000 7f16de7fc640 1 -- 172.21.15.77:0/4021038780 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16e001d460 con 0x7f16ec09ec00 2024-03-06T07:30:44.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.688+0000 7f16de7fc640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f16d006b500 0x7f16d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.688+0000 7f16de7fc640 1 -- 172.21.15.77:0/4021038780 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16e008a140 con 0x7f16ec09ec00 2024-03-06T07:30:44.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.689+0000 7f16f1d1d640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f16d006b500 0x7f16d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.689+0000 7f16f1d1d640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f16d006b500 0x7f16d006d9c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009bc0 tx=0x7f16e8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.693+0000 7f16de7fc640 1 -- 172.21.15.77:0/4021038780 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16e0059050 con 0x7f16ec09ec00 2024-03-06T07:30:44.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.884+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f16ec068ff0 con 0x7f16ec09ec00 2024-03-06T07:30:44.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.884+0000 7f16de7fc640 1 -- 172.21.15.77:0/4021038780 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f16e0058950 con 0x7f16ec09ec00 2024-03-06T07:30:44.886 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f16d006b500 msgr2=0x7f16d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f16d006b500 0x7f16d006d9c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009bc0 tx=0x7f16e8009190 comp rx=0 tx=0).stop 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec09ec00 msgr2=0x7f16ec0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec09ec00 0x7f16ec0b1ff0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f16e0007340 tx=0x7f16e0007740 comp rx=0 tx=0).stop 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 shutdown_connections 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f16d006b500 0x7f16d006d9c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec09ec00 0x7f16ec0b1ff0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16ec09fca0 0x7f16ec0a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 --2- 172.21.15.77:0/4021038780 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16d8004d40 0x7f16ec09e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 >> 172.21.15.77:0/4021038780 conn(0x7f16ec0d3d40 msgr2=0x7f16ec06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 shutdown_connections 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:44.887+0000 7f16f3f81640 1 -- 172.21.15.77:0/4021038780 wait complete. 2024-03-06T07:30:44.899 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:45.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.002+0000 7fc37f71e640 1 Processor -- start 2024-03-06T07:30:45.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.002+0000 7fc37f71e640 1 -- start start 2024-03-06T07:30:45.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37f71e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc3781614d0 0x7fc3780d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37f71e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3780d1910 0x7fc3780d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37f71e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780d2740 con 0x7fc3780d22b0 2024-03-06T07:30:45.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37f71e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3781618b0 con 0x7fc3781614d0 2024-03-06T07:30:45.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37f71e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780cf400 con 0x7fc3780d1910 2024-03-06T07:30:45.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37d4ba640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc3781614d0 0x7fc3780d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37dcbb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc3780d22b0 0x7fc3780d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51072/0 (socket says 172.21.15.77:51072) 2024-03-06T07:30:45.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37dcbb640 1 -- 172.21.15.77:0/2265423747 learned_addr learned my addr 172.21.15.77:0/2265423747 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:45.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37d4ba640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc3781614d0 0x7fc3780d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36574/0 (socket says 172.21.15.77:36574) 2024-03-06T07:30:45.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37ccb9640 1 --2- 172.21.15.77:0/2265423747 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3780d1910 0x7fc3780d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37d4ba640 1 -- 172.21.15.77:0/2265423747 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3780d1910 msgr2=0x7fc3780d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37d4ba640 1 --2- 172.21.15.77:0/2265423747 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3780d1910 0x7fc3780d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37d4ba640 1 -- 172.21.15.77:0/2265423747 >> v1:172.21.15.77:6789/0 conn(0x7fc3780d22b0 legacy=0x7fc3780d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc37d4ba640 1 -- 172.21.15.77:0/2265423747 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780cf6a0 con 0x7fc3781614d0 2024-03-06T07:30:45.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.003+0000 7fc36ffff640 1 -- 172.21.15.77:0/2265423747 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269278361 0 0) 0x7fc370001490 con 0x7fc3780d22b0 2024-03-06T07:30:45.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.004+0000 7fc37d4ba640 1 --2- 172.21.15.77:0/2265423747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3781614d0 0x7fc3780d4860 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fc3740097a0 tx=0x7fc37402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=204babf98ee1070c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.004+0000 7fc36ffff640 1 -- 172.21.15.77:0/2265423747 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc374035020 con 0x7fc3781614d0 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.004+0000 7fc36ffff640 1 -- 172.21.15.77:0/2265423747 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc374031830 con 0x7fc3781614d0 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.004+0000 7fc36ffff640 1 -- 172.21.15.77:0/2265423747 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc374031c00 con 0x7fc3781614d0 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.005+0000 7fc37f71e640 1 -- 172.21.15.77:0/2265423747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3781614d0 msgr2=0x7fc3780d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.005+0000 7fc37f71e640 1 --2- 172.21.15.77:0/2265423747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3781614d0 0x7fc3780d4860 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fc3740097a0 tx=0x7fc37402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.005+0000 7fc37f71e640 1 -- 172.21.15.77:0/2265423747 shutdown_connections 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.005+0000 7fc37f71e640 1 --2- 172.21.15.77:0/2265423747 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3780d1910 0x7fc3780d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.005+0000 7fc37f71e640 1 --2- 172.21.15.77:0/2265423747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3781614d0 0x7fc3780d4860 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.005+0000 7fc37f71e640 1 -- 172.21.15.77:0/2265423747 >> 172.21.15.77:0/2265423747 conn(0x7fc3780d3d40 msgr2=0x7fc3780d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.005+0000 7fc37f71e640 1 -- 172.21.15.77:0/2265423747 shutdown_connections 2024-03-06T07:30:45.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.005+0000 7fc37f71e640 1 -- 172.21.15.77:0/2265423747 wait complete. 2024-03-06T07:30:45.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37f71e640 1 Processor -- start 2024-03-06T07:30:45.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37f71e640 1 -- start start 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37f71e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3780d1910 0x7fc3780ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37f71e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3780d22b0 0x7fc378159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37f71e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3781614d0 0x7fc3780c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37f71e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780b0910 con 0x7fc3780d1910 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37f71e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780cfc60 con 0x7fc3780d22b0 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37f71e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780b1e40 con 0x7fc3781614d0 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37d4ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3780d1910 0x7fc3780ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37d4ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3780d1910 0x7fc3780ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36582/0 (socket says 172.21.15.77:36582) 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.007+0000 7fc37d4ba640 1 -- 172.21.15.77:0/570385449 learned_addr learned my addr 172.21.15.77:0/570385449 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37dcbb640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3781614d0 0x7fc3780c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37ccb9640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3780d22b0 0x7fc378159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37d4ba640 1 -- 172.21.15.77:0/570385449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3781614d0 msgr2=0x7fc3780c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37d4ba640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3781614d0 0x7fc3780c7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37d4ba640 1 -- 172.21.15.77:0/570385449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3780d22b0 msgr2=0x7fc378159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37d4ba640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3780d22b0 0x7fc378159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37d4ba640 1 -- 172.21.15.77:0/570385449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780b6ca0 con 0x7fc3780d1910 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37d4ba640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3780d1910 0x7fc3780ca5a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fc3740097a0 tx=0x7fc3740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc36dffb640 1 -- 172.21.15.77:0/570385449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc374035040 con 0x7fc3780d1910 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc36dffb640 1 -- 172.21.15.77:0/570385449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc374004230 con 0x7fc3780d1910 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc36dffb640 1 -- 172.21.15.77:0/570385449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37403a700 con 0x7fc3780d1910 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3780cbe60 con 0x7fc3780d1910 2024-03-06T07:30:45.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.008+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3780cc420 con 0x7fc3780d1910 2024-03-06T07:30:45.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.009+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc378068ff0 con 0x7fc3780d1910 2024-03-06T07:30:45.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.010+0000 7fc36dffb640 1 -- 172.21.15.77:0/570385449 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc374038020 con 0x7fc3780d1910 2024-03-06T07:30:45.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.011+0000 7fc36dffb640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc35c06b4d0 0x7fc35c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.011+0000 7fc36dffb640 1 -- 172.21.15.77:0/570385449 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3740ac070 con 0x7fc3780d1910 2024-03-06T07:30:45.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.011+0000 7fc37ccb9640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc35c06b4d0 0x7fc35c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.011+0000 7fc37ccb9640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc35c06b4d0 0x7fc35c06d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc3780d0ad0 tx=0x7fc368008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.015+0000 7fc36dffb640 1 -- 172.21.15.77:0/570385449 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc374031800 con 0x7fc3780d1910 2024-03-06T07:30:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.234+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc3780c8130 con 0x7fc3780d1910 2024-03-06T07:30:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.235+0000 7fc36dffb640 1 -- 172.21.15.77:0/570385449 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc37407aef0 con 0x7fc3780d1910 2024-03-06T07:30:45.237 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:30:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc35c06b4d0 msgr2=0x7fc35c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc35c06b4d0 0x7fc35c06d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc3780d0ad0 tx=0x7fc368008040 comp rx=0 tx=0).stop 2024-03-06T07:30:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3780d1910 msgr2=0x7fc3780ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3780d1910 0x7fc3780ca5a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fc3740097a0 tx=0x7fc3740046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 shutdown_connections 2024-03-06T07:30:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3781614d0 0x7fc3780c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3780d22b0 0x7fc378159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc35c06b4d0 0x7fc35c06d990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 --2- 172.21.15.77:0/570385449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3780d1910 0x7fc3780ca5a0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 >> 172.21.15.77:0/570385449 conn(0x7fc3780d3d40 msgr2=0x7fc3780d1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 shutdown_connections 2024-03-06T07:30:45.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.238+0000 7fc37f71e640 1 -- 172.21.15.77:0/570385449 wait complete. 2024-03-06T07:30:45.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:45.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.355+0000 7f3be95ae640 1 Processor -- start 2024-03-06T07:30:45.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.355+0000 7f3be95ae640 1 -- start start 2024-03-06T07:30:45.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be95ae640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3be415de50 0x7f3be40cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be95ae640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be406fa40 0x7f3be40d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be95ae640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be40cf220 con 0x7f3be415e230 2024-03-06T07:30:45.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be95ae640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be40cfcd0 con 0x7f3be415de50 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be95ae640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be40cffb0 con 0x7f3be406fa40 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be27fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3be415e230 0x7f3be406f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51078/0 (socket says 172.21.15.77:51078) 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be27fc640 1 -- 172.21.15.77:0/467079610 learned_addr learned my addr 172.21.15.77:0/467079610 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506412431 0 0) 0x7f3be40cf220 con 0x7f3be415e230 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bcc003520 con 0x7f3be415e230 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/467079610 >> v2:172.21.15.158:3300/0 conn(0x7f3be415de50 0x7f3be40cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.356+0000 7f3be37fe640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be406fa40 0x7f3be40d0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409758755 0 0) 0x7f3bcc003520 con 0x7f3be415e230 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be406fa40 msgr2=0x7f3be40d0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be406fa40 0x7f3be40d0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 >> v2:172.21.15.158:3300/0 conn(0x7f3be415de50 msgr2=0x7f3be40cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 --2- 172.21.15.77:0/467079610 >> v2:172.21.15.158:3300/0 conn(0x7f3be415de50 0x7f3be40cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be40d0290 con 0x7f3be415e230 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3bd8001670 con 0x7f3be415e230 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 >> v1:172.21.15.77:6789/0 conn(0x7f3be415e230 legacy=0x7f3be406f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bcc0039b0 0x7f3bcc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bcc004800 0x7f3bcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be406fa40 0x7f3bcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be40cfcd0 con 0x7f3bcc0039b0 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be40cffb0 con 0x7f3bcc004800 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc00b9f0 con 0x7f3be406fa40 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bcc0039b0 0x7f3bcc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be37fe640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be406fa40 0x7f3bcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be27fc640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bcc004800 0x7f3bcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be2ffd640 1 -- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be406fa40 msgr2=0x7f3bcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3be406fa40 0x7f3bcc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be2ffd640 1 -- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bcc004800 msgr2=0x7f3bcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3bcc004800 0x7f3bcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.357+0000 7f3be2ffd640 1 -- 172.21.15.77:0/467079610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc003520 con 0x7f3bcc0039b0 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.358+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bcc0039b0 0x7f3bcc003dd0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009ac0 tx=0x7f3bcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1097487cb5d2fc28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.358+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4039070 con 0x7f3bcc0039b0 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.358+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd402c6d0 con 0x7f3bcc0039b0 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.358+0000 7f3be1ffb640 1 -- 172.21.15.77:0/467079610 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4009690 con 0x7f3bcc0039b0 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.358+0000 7f3be95ae640 1 -- 172.21.15.77:0/467079610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bcc0039b0 msgr2=0x7f3bcc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.358+0000 7f3be95ae640 1 --2- 172.21.15.77:0/467079610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bcc0039b0 0x7f3bcc003dd0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009ac0 tx=0x7f3bcc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.358+0000 7f3be37fe640 1 -- 172.21.15.77:0/467079610 reap_dead start 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.359+0000 7f3be95ae640 1 -- 172.21.15.77:0/467079610 shutdown_connections 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.359+0000 7f3be95ae640 1 -- 172.21.15.77:0/467079610 >> 172.21.15.77:0/467079610 conn(0x7f3be40d18f0 msgr2=0x7f3be40d1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.359+0000 7f3be95ae640 1 -- 172.21.15.77:0/467079610 shutdown_connections 2024-03-06T07:30:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.359+0000 7f3be95ae640 1 -- 172.21.15.77:0/467079610 wait complete. 2024-03-06T07:30:45.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.360+0000 7f3be95ae640 1 Processor -- start 2024-03-06T07:30:45.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.360+0000 7f3be95ae640 1 -- start start 2024-03-06T07:30:45.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be95ae640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3bcc0039b0 0x7f3be41f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be95ae640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be415fe60 0x7f3be41602e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be95ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40b5de0 0x7f3be40b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be95ae640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be40d0ea0 con 0x7f3be40b5de0 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be95ae640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be40d0d20 con 0x7f3be415fe60 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be95ae640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be406ec30 con 0x7f3bcc0039b0 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40b5de0 0x7f3be40b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40b5de0 0x7f3be40b6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36602/0 (socket says 172.21.15.77:36602) 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 -- 172.21.15.77:0/1489429860 learned_addr learned my addr 172.21.15.77:0/1489429860 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be27fc640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be415fe60 0x7f3be41602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3bcc0039b0 0x7f3be41f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 -- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3bcc0039b0 msgr2=0x7f3be41f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3bcc0039b0 0x7f3be41f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 -- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be415fe60 msgr2=0x7f3be41602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be415fe60 0x7f3be41602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 -- 172.21.15.77:0/1489429860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be415e8e0 con 0x7f3be40b5de0 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3bcc0039b0 0x7f3be41f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be27fc640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be415fe60 0x7f3be41602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:45.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3be37fe640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40b5de0 0x7f3be40b6260 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f3be4072010 tx=0x7f3bdc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3bcb7fe640 1 -- 172.21.15.77:0/1489429860 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc011e10 con 0x7f3be40b5de0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.361+0000 7f3bcb7fe640 1 -- 172.21.15.77:0/1489429860 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc01b070 con 0x7f3be40b5de0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.362+0000 7f3bcb7fe640 1 -- 172.21.15.77:0/1489429860 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc003990 con 0x7f3be40b5de0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.362+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be4160c10 con 0x7f3be40b5de0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.362+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be41611d0 con 0x7f3be40b5de0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.363+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be406b4a0 con 0x7f3be40b5de0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.363+0000 7f3bcb7fe640 1 -- 172.21.15.77:0/1489429860 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bdc005d10 con 0x7f3be40b5de0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.363+0000 7f3bcb7fe640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd006b500 0x7f3bd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.364+0000 7f3bcb7fe640 1 -- 172.21.15.77:0/1489429860 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bdc08a550 con 0x7f3be40b5de0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.364+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd006b500 0x7f3bd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.364+0000 7f3be2ffd640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd006b500 0x7f3bd006d9c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4036000 tx=0x7f3bd4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.372+0000 7f3bcb7fe640 1 -- 172.21.15.77:0/1489429860 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3be406b4a0 con 0x7f3be40b5de0 2024-03-06T07:30:45.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.600+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3be406cc70 con 0x7f3be40b5de0 2024-03-06T07:30:45.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.600+0000 7f3bcb7fe640 1 -- 172.21.15.77:0/1489429860 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3bdc059460 con 0x7f3be40b5de0 2024-03-06T07:30:45.602 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:30:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd006b500 msgr2=0x7f3bd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd006b500 0x7f3bd006d9c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4036000 tx=0x7f3bd4030000 comp rx=0 tx=0).stop 2024-03-06T07:30:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40b5de0 msgr2=0x7f3be40b6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40b5de0 0x7f3be40b6260 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f3be4072010 tx=0x7f3bdc0115a0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 shutdown_connections 2024-03-06T07:30:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3bd006b500 0x7f3bd006d9c0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40b5de0 0x7f3be40b6260 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3be415fe60 0x7f3be41602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 --2- 172.21.15.77:0/1489429860 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3bcc0039b0 0x7f3be41f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 >> 172.21.15.77:0/1489429860 conn(0x7f3be40d18f0 msgr2=0x7f3be406d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 shutdown_connections 2024-03-06T07:30:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.602+0000 7f3be95ae640 1 -- 172.21.15.77:0/1489429860 wait complete. 2024-03-06T07:30:45.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:45.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.742+0000 7f5152113640 1 Processor -- start 2024-03-06T07:30:45.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.742+0000 7f5152113640 1 -- start start 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f5152113640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f5152113640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f514c161cf0 0x7f514c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f5152113640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c0d22c0 con 0x7f514c162970 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f5152113640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c161b00 con 0x7f514c161cf0 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f5152113640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0cecc0 con 0x7f514c0b4460 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f514c161cf0 0x7f514c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f514c161cf0 0x7f514c162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36616/0 (socket says 172.21.15.77:36616) 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514bfff640 1 -- 172.21.15.77:0/3000851116 learned_addr learned my addr 172.21.15.77:0/3000851116 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514b7fe640 1 --2- 172.21.15.77:0/3000851116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514bfff640 1 -- 172.21.15.77:0/3000851116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 msgr2=0x7f514c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514bfff640 1 --2- 172.21.15.77:0/3000851116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514bfff640 1 -- 172.21.15.77:0/3000851116 >> v1:172.21.15.158:6789/0 conn(0x7f514c162970 legacy=0x7f514c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514bfff640 1 -- 172.21.15.77:0/3000851116 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c0d25a0 con 0x7f514c161cf0 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.743+0000 7f514b7fe640 1 --2- 172.21.15.77:0/3000851116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.744+0000 7f514bfff640 1 --2- 172.21.15.77:0/3000851116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c161cf0 0x7f514c162180 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f5144009e50 tx=0x7f514402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9273dec9a0ab6f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.744+0000 7f514a7fc640 1 -- 172.21.15.77:0/3000851116 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514402ca70 con 0x7f514c161cf0 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.744+0000 7f514a7fc640 1 -- 172.21.15.77:0/3000851116 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514402cc10 con 0x7f514c161cf0 2024-03-06T07:30:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.744+0000 7f514a7fc640 1 -- 172.21.15.77:0/3000851116 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51440343f0 con 0x7f514c161cf0 2024-03-06T07:30:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.745+0000 7f5152113640 1 -- 172.21.15.77:0/3000851116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c161cf0 msgr2=0x7f514c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.745+0000 7f5152113640 1 --2- 172.21.15.77:0/3000851116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c161cf0 0x7f514c162180 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f5144009e50 tx=0x7f514402bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.745+0000 7f5152113640 1 -- 172.21.15.77:0/3000851116 shutdown_connections 2024-03-06T07:30:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.745+0000 7f5152113640 1 --2- 172.21.15.77:0/3000851116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c161cf0 0x7f514c162180 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.745+0000 7f5152113640 1 --2- 172.21.15.77:0/3000851116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.745+0000 7f5152113640 1 -- 172.21.15.77:0/3000851116 >> 172.21.15.77:0/3000851116 conn(0x7f514c0d3ce0 msgr2=0x7f514c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.745+0000 7f5152113640 1 -- 172.21.15.77:0/3000851116 shutdown_connections 2024-03-06T07:30:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.745+0000 7f5152113640 1 -- 172.21.15.77:0/3000851116 wait complete. 2024-03-06T07:30:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.746+0000 7f5152113640 1 Processor -- start 2024-03-06T07:30:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.746+0000 7f5152113640 1 -- start start 2024-03-06T07:30:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f5152113640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f5152113640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f514c161cf0 0x7f514c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f5152113640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c162970 0x7f514c0cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f5152113640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0ce950 con 0x7f514c162970 2024-03-06T07:30:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f5152113640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0d2b60 con 0x7f514c161cf0 2024-03-06T07:30:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f5152113640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0b4840 con 0x7f514c0b4460 2024-03-06T07:30:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f514bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c162970 0x7f514c0cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f514bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c162970 0x7f514c0cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36628/0 (socket says 172.21.15.77:36628) 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f514bfff640 1 -- 172.21.15.77:0/1736137046 learned_addr learned my addr 172.21.15.77:0/1736137046 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f514b7fe640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.747+0000 7f514affd640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f514c161cf0 0x7f514c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f514bfff640 1 -- 172.21.15.77:0/1736137046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 msgr2=0x7f514c155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f514bfff640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f514bfff640 1 -- 172.21.15.77:0/1736137046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f514c161cf0 msgr2=0x7f514c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f514bfff640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f514c161cf0 0x7f514c0cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f514bfff640 1 -- 172.21.15.77:0/1736137046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c0b69a0 con 0x7f514c162970 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f514bfff640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c162970 0x7f514c0cbb20 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f5144009e50 tx=0x7f5144004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f5133fff640 1 -- 172.21.15.77:0/1736137046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5144043020 con 0x7f514c162970 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f5133fff640 1 -- 172.21.15.77:0/1736137046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514403e450 con 0x7f514c162970 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c0b6c80 con 0x7f514c162970 2024-03-06T07:30:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f5133fff640 1 -- 172.21.15.77:0/1736137046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514403e860 con 0x7f514c162970 2024-03-06T07:30:45.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.748+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c0b7520 con 0x7f514c162970 2024-03-06T07:30:45.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.749+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c04f330 con 0x7f514c162970 2024-03-06T07:30:45.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.750+0000 7f5133fff640 1 -- 172.21.15.77:0/1736137046 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5144046020 con 0x7f514c162970 2024-03-06T07:30:45.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.750+0000 7f5133fff640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513806b4d0 0x7f513806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.750+0000 7f5133fff640 1 -- 172.21.15.77:0/1736137046 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51440aca80 con 0x7f514c162970 2024-03-06T07:30:45.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.750+0000 7f514b7fe640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513806b4d0 0x7f513806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.751+0000 7f514b7fe640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513806b4d0 0x7f513806d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f513c00aac0 tx=0x7f513c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.754+0000 7f5133fff640 1 -- 172.21.15.77:0/1736137046 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f514404d030 con 0x7f514c162970 2024-03-06T07:30:45.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.944+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f514c0b76f0 con 0x7f514c162970 2024-03-06T07:30:45.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.944+0000 7f5133fff640 1 -- 172.21.15.77:0/1736137046 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v141) v1 ==== 80+0+0 (secure 0 0 0) 0x7f514407b340 con 0x7f514c162970 2024-03-06T07:30:45.946 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:30:45.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513806b4d0 msgr2=0x7f513806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513806b4d0 0x7f513806d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f513c00aac0 tx=0x7f513c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:45.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c162970 msgr2=0x7f514c0cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c162970 0x7f514c0cbb20 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f5144009e50 tx=0x7f5144004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 shutdown_connections 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f513806b4d0 0x7f513806d990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f514c162970 0x7f514c0cbb20 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f514c161cf0 0x7f514c0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 --2- 172.21.15.77:0/1736137046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f514c0b4460 0x7f514c155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 >> 172.21.15.77:0/1736137046 conn(0x7f514c0d3ce0 msgr2=0x7f514c0d03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 shutdown_connections 2024-03-06T07:30:45.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:45.946+0000 7f5152113640 1 -- 172.21.15.77:0/1736137046 wait complete. 2024-03-06T07:30:45.963 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:30:45.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:45.963 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:45.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[83231]: fuse finished with error 0 and tester_r 0 2024-03-06T07:30:45.994 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:46.052 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:30:46.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:46.053 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:46.067 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:30:46.264 INFO:teuthology.orchestra.run.smithi077.stdout:{"message":"","return_code":0} 2024-03-06T07:30:46.265 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:46.265 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json cache drop 2024-03-06T07:30:46.384 INFO:teuthology.orchestra.run.smithi077.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":241,"bytes":58528}},"duration":0.003135199} 2024-03-06T07:30:46.384 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 58528, "items": 241 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.003135199, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:30:46.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:46.517 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.516+0000 7f11ef37a640 1 Processor -- start 2024-03-06T07:30:46.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.516+0000 7f11ef37a640 1 -- start start 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ef37a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f11e813dc10 0x7f11e80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ef37a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11e815bf30 0x7f11e80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ef37a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e815f580 con 0x7f11e815e850 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ef37a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e80cf3f0 con 0x7f11e815bf30 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ef37a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e815b210 con 0x7f11e813dc10 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ec915640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11e815bf30 0x7f11e80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ec915640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11e815bf30 0x7f11e80b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36630/0 (socket says 172.21.15.77:36630) 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ec915640 1 -- 172.21.15.77:0/2208215479 learned_addr learned my addr 172.21.15.77:0/2208215479 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ec915640 1 -- 172.21.15.77:0/2208215479 >> v2:172.21.15.158:3300/0 conn(0x7f11e813dc10 msgr2=0x7f11e80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ed116640 1 --2- 172.21.15.77:0/2208215479 >> v2:172.21.15.158:3300/0 conn(0x7f11e813dc10 0x7f11e80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ec915640 1 --2- 172.21.15.77:0/2208215479 >> v2:172.21.15.158:3300/0 conn(0x7f11e813dc10 0x7f11e80b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ec915640 1 -- 172.21.15.77:0/2208215479 >> v1:172.21.15.158:6789/0 conn(0x7f11e815e850 legacy=0x7f11e815ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.517+0000 7f11ec915640 1 -- 172.21.15.77:0/2208215479 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e815b450 con 0x7f11e815bf30 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.518+0000 7f11ec915640 1 --2- 172.21.15.77:0/2208215479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815bf30 0x7f11e80b2e90 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f11e80731a0 tx=0x7f11d802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fdfd5d20132039 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.518+0000 7f11dffff640 1 -- 172.21.15.77:0/2208215479 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d802e070 con 0x7f11e815bf30 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.518+0000 7f11dffff640 1 -- 172.21.15.77:0/2208215479 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8037070 con 0x7f11e815bf30 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.518+0000 7f11dffff640 1 -- 172.21.15.77:0/2208215479 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8031a00 con 0x7f11e815bf30 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.519+0000 7f11ef37a640 1 -- 172.21.15.77:0/2208215479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815bf30 msgr2=0x7f11e80b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.519+0000 7f11ef37a640 1 --2- 172.21.15.77:0/2208215479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815bf30 0x7f11e80b2e90 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f11e80731a0 tx=0x7f11d802b880 comp rx=0 tx=0).stop 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.519+0000 7f11ef37a640 1 -- 172.21.15.77:0/2208215479 shutdown_connections 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.519+0000 7f11ef37a640 1 --2- 172.21.15.77:0/2208215479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815bf30 0x7f11e80b2e90 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.519+0000 7f11ef37a640 1 --2- 172.21.15.77:0/2208215479 >> v2:172.21.15.158:3300/0 conn(0x7f11e813dc10 0x7f11e80b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.519+0000 7f11ef37a640 1 -- 172.21.15.77:0/2208215479 >> 172.21.15.77:0/2208215479 conn(0x7f11e80cf8a0 msgr2=0x7f11e80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.519+0000 7f11ef37a640 1 -- 172.21.15.77:0/2208215479 shutdown_connections 2024-03-06T07:30:46.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.519+0000 7f11ef37a640 1 -- 172.21.15.77:0/2208215479 wait complete. 2024-03-06T07:30:46.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.521+0000 7f11ef37a640 1 Processor -- start 2024-03-06T07:30:46.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.521+0000 7f11ef37a640 1 -- start start 2024-03-06T07:30:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ef37a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11e813dc10 0x7f11e81407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ef37a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11e815bf30 0x7f11e8142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ef37a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815e850 0x7f11e813cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ef37a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80708d0 con 0x7f11e815e850 2024-03-06T07:30:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ef37a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e813dff0 con 0x7f11e815bf30 2024-03-06T07:30:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ef37a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8070a50 con 0x7f11e813dc10 2024-03-06T07:30:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815e850 0x7f11e813cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815e850 0x7f11e813cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36634/0 (socket says 172.21.15.77:36634) 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 -- 172.21.15.77:0/4240515673 learned_addr learned my addr 172.21.15.77:0/4240515673 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ec915640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11e815bf30 0x7f11e8142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed116640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11e813dc10 0x7f11e81407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 -- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11e813dc10 msgr2=0x7f11e81407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11e813dc10 0x7f11e81407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 -- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11e815bf30 msgr2=0x7f11e8142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11e815bf30 0x7f11e8142ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 -- 172.21.15.77:0/4240515673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e813e210 con 0x7f11e815e850 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ec915640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11e815bf30 0x7f11e8142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed116640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11e813dc10 0x7f11e81407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ed917640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815e850 0x7f11e813cde0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f11e000db90 tx=0x7f11e000dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.522+0000 7f11ddffb640 1 -- 172.21.15.77:0/4240515673 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e000e980 con 0x7f11e815e850 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.523+0000 7f11ddffb640 1 -- 172.21.15.77:0/4240515673 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e000eb20 con 0x7f11e815e850 2024-03-06T07:30:46.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.523+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e8143820 con 0x7f11e815e850 2024-03-06T07:30:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.523+0000 7f11ddffb640 1 -- 172.21.15.77:0/4240515673 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0009560 con 0x7f11e815e850 2024-03-06T07:30:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.523+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e81f12f0 con 0x7f11e815e850 2024-03-06T07:30:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.524+0000 7f11ddffb640 1 -- 172.21.15.77:0/4240515673 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11e0009700 con 0x7f11e815e850 2024-03-06T07:30:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.524+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e813e590 con 0x7f11e815e850 2024-03-06T07:30:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.524+0000 7f11ddffb640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11cc06b500 0x7f11cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.525+0000 7f11ddffb640 1 -- 172.21.15.77:0/4240515673 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11e008bf60 con 0x7f11e815e850 2024-03-06T07:30:46.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.525+0000 7f11ed116640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11cc06b500 0x7f11cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.525+0000 7f11ed116640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11cc06b500 0x7f11cc06d9c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f11e4008890 tx=0x7f11e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.533+0000 7f11ddffb640 1 -- 172.21.15.77:0/4240515673 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11e005ae70 con 0x7f11e815e850 2024-03-06T07:30:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.801+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f11e804f330 con 0x7f11e815e850 2024-03-06T07:30:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.801+0000 7f11ddffb640 1 -- 172.21.15.77:0/4240515673 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v103) v1 ==== 53+0+73 (secure 0 0 0) 0x7f11e005a770 con 0x7f11e815e850 2024-03-06T07:30:46.803 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11cc06b500 msgr2=0x7f11cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11cc06b500 0x7f11cc06d9c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f11e4008890 tx=0x7f11e4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815e850 msgr2=0x7f11e813cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815e850 0x7f11e813cde0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f11e000db90 tx=0x7f11e000dc80 comp rx=0 tx=0).stop 2024-03-06T07:30:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 shutdown_connections 2024-03-06T07:30:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11cc06b500 0x7f11cc06d9c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11e815e850 0x7f11e813cde0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11e815bf30 0x7f11e8142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 --2- 172.21.15.77:0/4240515673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11e813dc10 0x7f11e81407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 >> 172.21.15.77:0/4240515673 conn(0x7f11e80cf8a0 msgr2=0x7f11e806c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 shutdown_connections 2024-03-06T07:30:46.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:46.803+0000 7f11ef37a640 1 -- 172.21.15.77:0/4240515673 wait complete. 2024-03-06T07:30:46.819 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:30:46.819 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:46.819 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-06T07:30:46.819 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:46.819 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:46.819 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:30:46.819 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:30:46.819 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:46.820 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:46.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:46.820 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-06T07:30:46.835 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:46.835 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:46.835 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:46.835 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:46.835 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:46.835 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:46.835 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:46.835 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42150sec preferred_lft 42150sec 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:46.836 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a4ce:b0ff:fea6:33f8/64 scope link 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 1e:86:a9:fa:7e:1d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::1c86:a9ff:fefa:7e1d/64 scope link 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether ce:ef:f4:3c:88:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ccef:f4ff:fe3c:88fe/64 scope link 2024-03-06T07:30:46.837 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:46.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:46.838 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:46.900 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-06T07:30:46.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:46.902 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-06T07:30:46.958 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-06T07:30:46.958 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:46.958 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:30:46.958 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:30:46.958 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:46.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:46.959 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-03-06T07:30:46.959 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-06T07:30:46.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:30:46.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:30:46.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:30:46.959 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.056 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:30:47.070 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:47.070 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:47.070 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:47.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.070 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-06T07:30:47.137 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:30:47.137 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:47.138 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:47.192 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:47.193 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:47.193 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:47.248 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:30:47.315 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:47.370 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:30:47.370 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:30:47.370 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:30:47.370 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:30:47.370 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:47.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.372 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:30:47.424 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:30:47.424 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:30:47.467 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-06T07:30:47.495 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:47.551 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-06T07:30:47.551 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-06T07:30:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-06T07:30:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:47.554 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:47.554 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:47.554 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:47.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.554 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-06T07:30:47.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-06T07:30:47.554+0000 7fa1b9c37480 -1 init, newargv = 0x563ecc31b780 newargc=15 2024-03-06T07:30:47.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[85143]: starting ceph client 2024-03-06T07:30:47.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[85143]: starting fuse 2024-03-06T07:30:47.573 INFO:teuthology.orchestra.run.smithi077.stdout:42 2024-03-06T07:30:47.573 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:30:47.573 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:30:47.574 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-06T07:30:47.575 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-06T07:30:47.575 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-06T07:30:47.692 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.85143.asok 2024-03-06T07:30:47.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:47.703 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.85143.asok 2024-03-06T07:30:47.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.85143.asok status 2024-03-06T07:30:47.840 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:30:47.840 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-06T07:30:47.840 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:30:47.840 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "85143", 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/" 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 9534, 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 9534 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:47.841 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 185661577 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 185661577 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.9534 192.168.0.1:0/185661577", 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/185661577", 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 1, 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 103, 2024-03-06T07:30:47.842 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 141, 2024-03-06T07:30:47.843 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:30:47.843 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-06T07:30:47.843 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-06T07:30:47.843 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:30:47.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.849 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-06T07:30:47.849 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:47.864 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-06T07:30:47.864 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:47.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.864 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:47.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:47.931 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/original) 2024-03-06T07:30:47.994 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:30:47.994 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:48.052 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.193+0000 7f3184d5e640 1 Processor -- start 2024-03-06T07:30:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.193+0000 7f3184d5e640 1 -- start start 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f3184d5e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f318015dfe0 0x7f31800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f3184d5e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31800b3430 con 0x7f31800cf830 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f3184d5e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31800b3710 con 0x7f31800d29c0 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f3184d5e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31800b31e0 con 0x7f318015dfe0 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f317dd9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f31800cf830 0x7f31800d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51092/0 (socket says 172.21.15.77:51092) 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f317dd9b640 1 -- 172.21.15.77:0/243248573 learned_addr learned my addr 172.21.15.77:0/243248573 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f317e59c640 1 --2- 172.21.15.77:0/243248573 >> v2:172.21.15.158:3300/0 conn(0x7f318015dfe0 0x7f31800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572390679 0 0) 0x7f31800b3430 con 0x7f31800cf830 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.194+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3168003520 con 0x7f31800cf830 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555226813 0 0) 0x7f3168003520 con 0x7f31800cf830 2024-03-06T07:30:48.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 >> v2:172.21.15.158:3300/0 conn(0x7f318015dfe0 msgr2=0x7f31800cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 --2- 172.21.15.77:0/243248573 >> v2:172.21.15.158:3300/0 conn(0x7f318015dfe0 0x7f31800cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 >> v1:172.21.15.158:6789/0 conn(0x7f31800d29c0 legacy=0x7f31800b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31800b39f0 con 0x7f31800cf830 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3170001670 con 0x7f31800cf830 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 >> v1:172.21.15.77:6789/0 conn(0x7f31800cf830 legacy=0x7f31800d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31680039b0 0x7f3168003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168004800 0x7f3168004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31680051c0 0x7f316800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31800b31e0 con 0x7f3168004800 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316800c090 con 0x7f31680051c0 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316800c290 con 0x7f31680039b0 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317dd9b640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168004800 0x7f3168004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317e59c640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31680039b0 0x7f3168003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317ed9d640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31680051c0 0x7f316800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317dd9b640 1 -- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31680039b0 msgr2=0x7f3168003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317dd9b640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f31680039b0 0x7f3168003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317dd9b640 1 -- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31680051c0 msgr2=0x7f316800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317dd9b640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31680051c0 0x7f316800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317dd9b640 1 -- 172.21.15.77:0/243248573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3168003520 con 0x7f3168004800 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317ed9d640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f31680051c0 0x7f316800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.195+0000 7f317ed9d640 1 -- 172.21.15.77:0/243248573 reap_dead start 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.196+0000 7f317dd9b640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168004800 0x7f3168004c80 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f317000c210 tx=0x7f3168005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a04f9aa11679ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.196+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170007d10 con 0x7f3168004800 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.196+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3170030070 con 0x7f3168004800 2024-03-06T07:30:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.196+0000 7f317d59a640 1 -- 172.21.15.77:0/243248573 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170005d50 con 0x7f3168004800 2024-03-06T07:30:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.197+0000 7f3184d5e640 1 -- 172.21.15.77:0/243248573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168004800 msgr2=0x7f3168004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.197+0000 7f3184d5e640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168004800 0x7f3168004c80 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f317000c210 tx=0x7f3168005830 comp rx=0 tx=0).stop 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.197+0000 7f3184d5e640 1 -- 172.21.15.77:0/243248573 shutdown_connections 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.197+0000 7f3184d5e640 1 --2- 172.21.15.77:0/243248573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168004800 0x7f3168004c80 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.197+0000 7f3184d5e640 1 -- 172.21.15.77:0/243248573 >> 172.21.15.77:0/243248573 conn(0x7f31800d3cd0 msgr2=0x7f31800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.197+0000 7f3184d5e640 1 -- 172.21.15.77:0/243248573 shutdown_connections 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.197+0000 7f3184d5e640 1 -- 172.21.15.77:0/243248573 wait complete. 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.199+0000 7f3184d5e640 1 Processor -- start 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.199+0000 7f3184d5e640 1 -- start start 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f3184d5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3168004800 0x7f31800cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f3184d5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31801f3240 0x7f318015c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f3184d5e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f318015c670 0x7f318015e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f3184d5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318015e3c0 con 0x7f31801f3240 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f3184d5e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31800b3d60 con 0x7f318015c670 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f3184d5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318015e540 con 0x7f3168004800 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f317dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31801f3240 0x7f318015c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f317dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31801f3240 0x7f318015c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36652/0 (socket says 172.21.15.77:36652) 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f317dd9b640 1 -- 172.21.15.77:0/1282306415 learned_addr learned my addr 172.21.15.77:0/1282306415 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.200+0000 7f317e59c640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3168004800 0x7f31800cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f317ed9d640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f318015c670 0x7f318015e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f317dd9b640 1 -- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3168004800 msgr2=0x7f31800cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f317dd9b640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3168004800 0x7f31800cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f317dd9b640 1 -- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f318015c670 msgr2=0x7f318015e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f317dd9b640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f318015c670 0x7f318015e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f317dd9b640 1 -- 172.21.15.77:0/1282306415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318015a9d0 con 0x7f31801f3240 2024-03-06T07:30:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f317ed9d640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f318015c670 0x7f318015e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:48.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f317dd9b640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31801f3240 0x7f318015c130 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f317000c330 tx=0x7f31800b38e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f316effd640 1 -- 172.21.15.77:0/1282306415 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170045070 con 0x7f31801f3240 2024-03-06T07:30:48.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f316effd640 1 -- 172.21.15.77:0/1282306415 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3170000ed0 con 0x7f31801f3240 2024-03-06T07:30:48.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318015acb0 con 0x7f31801f3240 2024-03-06T07:30:48.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f316effd640 1 -- 172.21.15.77:0/1282306415 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170007d10 con 0x7f31801f3240 2024-03-06T07:30:48.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.201+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318015b270 con 0x7f31801f3240 2024-03-06T07:30:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.202+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318004f330 con 0x7f31801f3240 2024-03-06T07:30:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.203+0000 7f316effd640 1 -- 172.21.15.77:0/1282306415 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3170004c10 con 0x7f31801f3240 2024-03-06T07:30:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.203+0000 7f316effd640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f316006b500 0x7f316006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.204+0000 7f316effd640 1 -- 172.21.15.77:0/1282306415 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31700aaf60 con 0x7f31801f3240 2024-03-06T07:30:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.204+0000 7f317e59c640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f316006b500 0x7f316006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.204+0000 7f317e59c640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f316006b500 0x7f316006d9c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3174009b80 tx=0x7f3174008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.211+0000 7f316effd640 1 -- 172.21.15.77:0/1282306415 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3170079d50 con 0x7f31801f3240 2024-03-06T07:30:48.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.465+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f318015ca50 con 0x7f31801f3240 2024-03-06T07:30:48.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.466+0000 7f316effd640 1 -- 172.21.15.77:0/1282306415 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+16716 (secure 0 0 0) 0x7f3170079650 con 0x7f31801f3240 2024-03-06T07:30:48.468 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:48.468 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":142,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:48.218244+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:48.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f316006b500 msgr2=0x7f316006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f316006b500 0x7f316006d9c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3174009b80 tx=0x7f3174008040 comp rx=0 tx=0).stop 2024-03-06T07:30:48.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31801f3240 msgr2=0x7f318015c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31801f3240 0x7f318015c130 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f317000c330 tx=0x7f31800b38e0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 shutdown_connections 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f316006b500 0x7f316006d9c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31801f3240 0x7f318015c130 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f318015c670 0x7f318015e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 --2- 172.21.15.77:0/1282306415 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3168004800 0x7f31800cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 >> 172.21.15.77:0/1282306415 conn(0x7f31800d3cd0 msgr2=0x7f318006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 shutdown_connections 2024-03-06T07:30:48.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.468+0000 7f3184d5e640 1 -- 172.21.15.77:0/1282306415 wait complete. 2024-03-06T07:30:48.485 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.674704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.414541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:48.485 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:48.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.602+0000 7f2c3c50c640 1 Processor -- start 2024-03-06T07:30:48.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.602+0000 7f2c3c50c640 1 -- start start 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3c50c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2c340d4860 0x7f2c340d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3c50c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340b0e30 0x7f2c340b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3c50c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c341618b0 con 0x7f2c341614d0 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3c50c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c340d4cf0 con 0x7f2c340d4860 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3c50c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340cf400 con 0x7f2c340b0e30 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c39aa7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2c340d4860 0x7f2c340d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3aaa9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340b0e30 0x7f2c340b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c39aa7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2c340d4860 0x7f2c340d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:51460/0 (socket says 172.21.15.77:51460) 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3a2a8640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f2c341614d0 0x7f2c340d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:42384/0 (socket says 172.21.15.77:42384) 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c39aa7640 1 -- 172.21.15.77:0/715553944 learned_addr learned my addr 172.21.15.77:0/715553944 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:48.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3aaa9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340b0e30 0x7f2c340b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:44600/0 (socket says 172.21.15.77:44600) 2024-03-06T07:30:48.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3aaa9640 1 -- 172.21.15.77:0/715553944 >> v2:172.21.15.158:3300/0 conn(0x7f2c340d4860 msgr2=0x7f2c340d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3aaa9640 1 --2- 172.21.15.77:0/715553944 >> v2:172.21.15.158:3300/0 conn(0x7f2c340d4860 0x7f2c340d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3aaa9640 1 -- 172.21.15.77:0/715553944 >> v1:172.21.15.158:6789/0 conn(0x7f2c341614d0 legacy=0x7f2c340d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c3aaa9640 1 -- 172.21.15.77:0/715553944 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c340cf6a0 con 0x7f2c340b0e30 2024-03-06T07:30:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.603+0000 7f2c39aa7640 1 --2- 172.21.15.77:0/715553944 >> v2:172.21.15.158:3300/0 conn(0x7f2c340d4860 0x7f2c340d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.604+0000 7f2c3aaa9640 1 --2- 172.21.15.77:0/715553944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340b0e30 0x7f2c340b12a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009d90 tx=0x7f2c2c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bfb0b90ed7ee60b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.604+0000 7f2c392a6640 1 -- 172.21.15.77:0/715553944 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c039070 con 0x7f2c340b0e30 2024-03-06T07:30:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c392a6640 1 -- 172.21.15.77:0/715553944 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c02cc50 con 0x7f2c340b0e30 2024-03-06T07:30:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c392a6640 1 -- 172.21.15.77:0/715553944 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c034440 con 0x7f2c340b0e30 2024-03-06T07:30:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c3c50c640 1 -- 172.21.15.77:0/715553944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340b0e30 msgr2=0x7f2c340b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/715553944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340b0e30 0x7f2c340b12a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009d90 tx=0x7f2c2c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c3c50c640 1 -- 172.21.15.77:0/715553944 shutdown_connections 2024-03-06T07:30:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/715553944 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340b0e30 0x7f2c340b12a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/715553944 >> v2:172.21.15.158:3300/0 conn(0x7f2c340d4860 0x7f2c340d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c3c50c640 1 -- 172.21.15.77:0/715553944 >> 172.21.15.77:0/715553944 conn(0x7f2c340d3d40 msgr2=0x7f2c340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c3c50c640 1 -- 172.21.15.77:0/715553944 shutdown_connections 2024-03-06T07:30:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.605+0000 7f2c3c50c640 1 -- 172.21.15.77:0/715553944 wait complete. 2024-03-06T07:30:48.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.607+0000 7f2c3c50c640 1 Processor -- start 2024-03-06T07:30:48.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.607+0000 7f2c3c50c640 1 -- start start 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.608+0000 7f2c3c50c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c340b0e30 0x7f2c340ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.608+0000 7f2c3c50c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340d4860 0x7f2c3415f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.608+0000 7f2c3c50c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c341614d0 0x7f2c340b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.608+0000 7f2c3c50c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340b0910 con 0x7f2c341614d0 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.608+0000 7f2c3c50c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340cfc60 con 0x7f2c340b0e30 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.608+0000 7f2c3c50c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340d2560 con 0x7f2c340d4860 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c341614d0 0x7f2c340b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c341614d0 0x7f2c340b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56218/0 (socket says 172.21.15.77:56218) 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 -- 172.21.15.77:0/4016685271 learned_addr learned my addr 172.21.15.77:0/4016685271 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3a2a8640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c340b0e30 0x7f2c340ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c39aa7640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340d4860 0x7f2c3415f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 -- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340d4860 msgr2=0x7f2c3415f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340d4860 0x7f2c3415f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 -- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c340b0e30 msgr2=0x7f2c340ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c340b0e30 0x7f2c340ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 -- 172.21.15.77:0/4016685271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c340b78b0 con 0x7f2c341614d0 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3a2a8640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c340b0e30 0x7f2c340ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3aaa9640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c341614d0 0x7f2c340b8b00 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009d90 tx=0x7f2c2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c2affd640 1 -- 172.21.15.77:0/4016685271 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c039050 con 0x7f2c341614d0 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c2affd640 1 -- 172.21.15.77:0/4016685271 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c045070 con 0x7f2c341614d0 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c340b7b90 con 0x7f2c341614d0 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c2affd640 1 -- 172.21.15.77:0/4016685271 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c009450 con 0x7f2c341614d0 2024-03-06T07:30:48.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.609+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c34160310 con 0x7f2c341614d0 2024-03-06T07:30:48.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.611+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c34068ff0 con 0x7f2c341614d0 2024-03-06T07:30:48.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.611+0000 7f2c2affd640 1 -- 172.21.15.77:0/4016685271 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c2c03e610 con 0x7f2c341614d0 2024-03-06T07:30:48.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.611+0000 7f2c2affd640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c1806b4d0 0x7f2c1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.611+0000 7f2c3a2a8640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c1806b4d0 0x7f2c1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.611+0000 7f2c2affd640 1 -- 172.21.15.77:0/4016685271 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2c2c0acaa0 con 0x7f2c341614d0 2024-03-06T07:30:48.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.612+0000 7f2c3a2a8640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c1806b4d0 0x7f2c1806d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2c30007fd0 tx=0x7f2c30007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.616+0000 7f2c2affd640 1 -- 172.21.15.77:0/4016685271 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c2c07b340 con 0x7f2c341614d0 2024-03-06T07:30:48.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.834+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c340b49f0 con 0x7f2c341614d0 2024-03-06T07:30:48.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.835+0000 7f2c2affd640 1 -- 172.21.15.77:0/4016685271 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v142) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c2c07f3e0 con 0x7f2c341614d0 2024-03-06T07:30:48.837 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c1806b4d0 msgr2=0x7f2c1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c1806b4d0 0x7f2c1806d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2c30007fd0 tx=0x7f2c30007e80 comp rx=0 tx=0).stop 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c341614d0 msgr2=0x7f2c340b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c341614d0 0x7f2c340b8b00 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009d90 tx=0x7f2c2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 shutdown_connections 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2c1806b4d0 0x7f2c1806d990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c341614d0 0x7f2c340b8b00 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2c340d4860 0x7f2c3415f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 --2- 172.21.15.77:0/4016685271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2c340b0e30 0x7f2c340ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 >> 172.21.15.77:0/4016685271 conn(0x7f2c340d3d40 msgr2=0x7f2c340d0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 shutdown_connections 2024-03-06T07:30:48.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.838+0000 7f2c3c50c640 1 -- 172.21.15.77:0/4016685271 wait complete. 2024-03-06T07:30:48.854 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:48.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.956+0000 7fcedd8df640 1 Processor -- start 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.957+0000 7fcedd8df640 1 -- start start 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.957+0000 7fcedd8df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fced80b4970 0x7fced806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.957+0000 7fcedd8df640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced815ff90 con 0x7fced806d5c0 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.957+0000 7fcedd8df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced8160270 con 0x7fced815bb40 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.957+0000 7fcedd8df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced80b4dd0 con 0x7fced80b4970 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.957+0000 7fced67fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fced80b4970 0x7fced806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.957+0000 7fced77fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fced806d5c0 0x7fced80cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53564/0 (socket says 172.21.15.77:53564) 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.957+0000 7fced77fe640 1 -- 172.21.15.77:0/1048523074 learned_addr learned my addr 172.21.15.77:0/1048523074 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.958+0000 7fced67fc640 1 -- 172.21.15.77:0/1048523074 >> v1:172.21.15.158:6789/0 conn(0x7fced815bb40 legacy=0x7fced815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.958+0000 7fced67fc640 1 -- 172.21.15.77:0/1048523074 >> v1:172.21.15.77:6789/0 conn(0x7fced806d5c0 legacy=0x7fced80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.958+0000 7fced67fc640 1 -- 172.21.15.77:0/1048523074 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8160550 con 0x7fced80b4970 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.958+0000 7fced5ffb640 1 -- 172.21.15.77:0/1048523074 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157613938 0 0) 0x7fced815ff90 con 0x7fced806d5c0 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fced67fc640 1 --2- 172.21.15.77:0/1048523074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced806d080 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0097a0 tx=0x7fcecc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d23580ae2801cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fced5ffb640 1 -- 172.21.15.77:0/1048523074 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc035020 con 0x7fced80b4970 2024-03-06T07:30:48.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fced5ffb640 1 -- 172.21.15.77:0/1048523074 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc031830 con 0x7fced80b4970 2024-03-06T07:30:48.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fced5ffb640 1 -- 172.21.15.77:0/1048523074 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc031c00 con 0x7fced80b4970 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fcedd8df640 1 -- 172.21.15.77:0/1048523074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 msgr2=0x7fced806d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fcedd8df640 1 --2- 172.21.15.77:0/1048523074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced806d080 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0097a0 tx=0x7fcecc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fcedd8df640 1 -- 172.21.15.77:0/1048523074 shutdown_connections 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fcedd8df640 1 --2- 172.21.15.77:0/1048523074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced806d080 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fcedd8df640 1 -- 172.21.15.77:0/1048523074 >> 172.21.15.77:0/1048523074 conn(0x7fced806b190 msgr2=0x7fced806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fcedd8df640 1 -- 172.21.15.77:0/1048523074 shutdown_connections 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.959+0000 7fcedd8df640 1 -- 172.21.15.77:0/1048523074 wait complete. 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.960+0000 7fcedd8df640 1 Processor -- start 2024-03-06T07:30:48.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fcedd8df640 1 -- start start 2024-03-06T07:30:48.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fcedd8df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced806d5c0 0x7fced814b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fcedd8df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced814f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fcedd8df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced815bb40 0x7fced80ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fcedd8df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced815bf20 con 0x7fced80b4970 2024-03-06T07:30:48.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fcedd8df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced806c910 con 0x7fced806d5c0 2024-03-06T07:30:48.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fcedd8df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced815c0a0 con 0x7fced815bb40 2024-03-06T07:30:48.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fced67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced814f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fced67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced814f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56228/0 (socket says 172.21.15.77:56228) 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.961+0000 7fced67fc640 1 -- 172.21.15.77:0/721184032 learned_addr learned my addr 172.21.15.77:0/721184032 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fced6ffd640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced806d5c0 0x7fced814b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fced77fe640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced815bb40 0x7fced80ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fced67fc640 1 -- 172.21.15.77:0/721184032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced815bb40 msgr2=0x7fced80ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fced67fc640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced815bb40 0x7fced80ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fced67fc640 1 -- 172.21.15.77:0/721184032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced806d5c0 msgr2=0x7fced814b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fced67fc640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced806d5c0 0x7fced814b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fced67fc640 1 -- 172.21.15.77:0/721184032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced814bc30 con 0x7fced80b4970 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fced67fc640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced814f450 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fcecc02bee0 tx=0x7fcecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fcebffff640 1 -- 172.21.15.77:0/721184032 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc03ba80 con 0x7fced80b4970 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fcebffff640 1 -- 172.21.15.77:0/721184032 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc03bc40 con 0x7fced80b4970 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced814fd30 con 0x7fced80b4970 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fcebffff640 1 -- 172.21.15.77:0/721184032 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc045400 con 0x7fced80b4970 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.962+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced81502f0 con 0x7fced80b4970 2024-03-06T07:30:48.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.963+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced80d0230 con 0x7fced80b4970 2024-03-06T07:30:48.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.964+0000 7fcebffff640 1 -- 172.21.15.77:0/721184032 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcecc038020 con 0x7fced80b4970 2024-03-06T07:30:48.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.964+0000 7fcebffff640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcec406b4d0 0x7fcec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.964+0000 7fcebffff640 1 -- 172.21.15.77:0/721184032 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcecc0ac0b0 con 0x7fced80b4970 2024-03-06T07:30:48.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.964+0000 7fced6ffd640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcec406b4d0 0x7fcec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.965+0000 7fced6ffd640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcec406b4d0 0x7fcec406d990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcec8006fd0 tx=0x7fcec8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:48.972+0000 7fcebffff640 1 -- 172.21.15.77:0/721184032 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fced80d0230 con 0x7fced80b4970 2024-03-06T07:30:49.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.149+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fced80732b0 con 0x7fced80b4970 2024-03-06T07:30:49.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.150+0000 7fcebffff640 1 -- 172.21.15.77:0/721184032 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v142) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcecc040070 con 0x7fced80b4970 2024-03-06T07:30:49.151 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:30:49.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcec406b4d0 msgr2=0x7fcec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcec406b4d0 0x7fcec406d990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcec8006fd0 tx=0x7fcec8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 msgr2=0x7fced814f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced814f450 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fcecc02bee0 tx=0x7fcecc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 shutdown_connections 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcec406b4d0 0x7fcec406d990 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced80b4970 0x7fced814f450 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced815bb40 0x7fced80ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 --2- 172.21.15.77:0/721184032 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced806d5c0 0x7fced814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.151+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 >> 172.21.15.77:0/721184032 conn(0x7fced806b190 msgr2=0x7fced81593f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.152+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 shutdown_connections 2024-03-06T07:30:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.152+0000 7fcedd8df640 1 -- 172.21.15.77:0/721184032 wait complete. 2024-03-06T07:30:49.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:49.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.270+0000 7f39e27c7640 1 Processor -- start 2024-03-06T07:30:49.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.270+0000 7f39e27c7640 1 -- start start 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39e27c7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f39dc0d4860 0x7f39dc0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39e27c7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc1618b0 con 0x7f39dc1614d0 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39e27c7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc0cf400 con 0x7f39dc0b0e30 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39e27c7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39dc0d4cf0 con 0x7f39dc0d4860 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39dbfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f39dc1614d0 0x7f39dc0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53574/0 (socket says 172.21.15.77:53574) 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39dbfff640 1 -- 172.21.15.77:0/1599849281 learned_addr learned my addr 172.21.15.77:0/1599849281 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39db7fe640 1 --2- 172.21.15.77:0/1599849281 >> v2:172.21.15.158:3300/0 conn(0x7f39dc0d4860 0x7f39dc0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367562305 0 0) 0x7f39dc1618b0 con 0x7f39dc1614d0 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c4003520 con 0x7f39dc1614d0 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822498691 0 0) 0x7f39c4003520 con 0x7f39dc1614d0 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 >> v2:172.21.15.158:3300/0 conn(0x7f39dc0d4860 msgr2=0x7f39dc0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.271+0000 7f39daffd640 1 --2- 172.21.15.77:0/1599849281 >> v2:172.21.15.158:3300/0 conn(0x7f39dc0d4860 0x7f39dc0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 >> v1:172.21.15.158:6789/0 conn(0x7f39dc0b0e30 legacy=0x7f39dc0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc0cf6a0 con 0x7f39dc1614d0 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f39cc001690 con 0x7f39dc1614d0 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 >> v1:172.21.15.77:6789/0 conn(0x7f39dc1614d0 legacy=0x7f39dc0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f39c40039b0 0x7f39c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39c40051c0 0x7f39c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0d4cf0 con 0x7f39c4004800 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c400c060 con 0x7f39c40051c0 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c400c260 con 0x7f39c40039b0 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39db7fe640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39dbfff640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f39c40039b0 0x7f39c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39e0d64640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39c40051c0 0x7f39c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39db7fe640 1 -- 172.21.15.77:0/1599849281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f39c40039b0 msgr2=0x7f39c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39db7fe640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f39c40039b0 0x7f39c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39db7fe640 1 -- 172.21.15.77:0/1599849281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39c40051c0 msgr2=0x7f39c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39db7fe640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39c40051c0 0x7f39c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39db7fe640 1 -- 172.21.15.77:0/1599849281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4003520 con 0x7f39c4004800 2024-03-06T07:30:49.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.272+0000 7f39e0d64640 1 -- 172.21.15.77:0/1599849281 reap_dead start 2024-03-06T07:30:49.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.273+0000 7f39db7fe640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009ac0 tx=0x7f39c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f809c552b450bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.273+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0039070 con 0x7f39c4004800 2024-03-06T07:30:49.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.273+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d002c6b0 con 0x7f39c4004800 2024-03-06T07:30:49.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.273+0000 7f39daffd640 1 -- 172.21.15.77:0/1599849281 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0009690 con 0x7f39c4004800 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.274+0000 7f39e27c7640 1 -- 172.21.15.77:0/1599849281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 msgr2=0x7f39c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.274+0000 7f39e27c7640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009ac0 tx=0x7f39c4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.274+0000 7f39e27c7640 1 -- 172.21.15.77:0/1599849281 shutdown_connections 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.274+0000 7f39e27c7640 1 --2- 172.21.15.77:0/1599849281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39c4004c80 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.274+0000 7f39e27c7640 1 -- 172.21.15.77:0/1599849281 >> 172.21.15.77:0/1599849281 conn(0x7f39dc0d3d40 msgr2=0x7f39dc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.274+0000 7f39e27c7640 1 -- 172.21.15.77:0/1599849281 shutdown_connections 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.274+0000 7f39e27c7640 1 -- 172.21.15.77:0/1599849281 wait complete. 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.275+0000 7f39e27c7640 1 Processor -- start 2024-03-06T07:30:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.276+0000 7f39e27c7640 1 -- start start 2024-03-06T07:30:49.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39e27c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39dc1552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39e27c7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39dc0cd1b0 0x7f39dc0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39e27c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f39dc0b3bf0 0x7f39dc0b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39e27c7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0b0910 con 0x7f39c4004800 2024-03-06T07:30:49.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39e27c7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0cfc60 con 0x7f39dc0cd1b0 2024-03-06T07:30:49.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39e27c7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0d1910 con 0x7f39dc0b3bf0 2024-03-06T07:30:49.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39dc1552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39dc1552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56250/0 (socket says 172.21.15.77:56250) 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 -- 172.21.15.77:0/213497288 learned_addr learned my addr 172.21.15.77:0/213497288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 -- 172.21.15.77:0/213497288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f39dc0b3bf0 msgr2=0x7f39dc0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f39dc0b3bf0 0x7f39dc0b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 -- 172.21.15.77:0/213497288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39dc0cd1b0 msgr2=0x7f39dc0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39dc0cd1b0 0x7f39dc0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 -- 172.21.15.77:0/213497288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc155830 con 0x7f39c4004800 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.277+0000 7f39dbfff640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39dc1552f0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f39cc007340 tx=0x7f39cc007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.278+0000 7f39d8ff9640 1 -- 172.21.15.77:0/213497288 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc00f7e0 con 0x7f39c4004800 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.278+0000 7f39d8ff9640 1 -- 172.21.15.77:0/213497288 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39cc018070 con 0x7f39c4004800 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.278+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc0c93e0 con 0x7f39c4004800 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.278+0000 7f39d8ff9640 1 -- 172.21.15.77:0/213497288 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc00fc00 con 0x7f39c4004800 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.278+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc0c99a0 con 0x7f39c4004800 2024-03-06T07:30:49.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.279+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc0cde40 con 0x7f39c4004800 2024-03-06T07:30:49.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.279+0000 7f39d8ff9640 1 -- 172.21.15.77:0/213497288 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39cc01c410 con 0x7f39c4004800 2024-03-06T07:30:49.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.280+0000 7f39d8ff9640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f39c806b500 0x7f39c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.280+0000 7f39db7fe640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f39c806b500 0x7f39c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.280+0000 7f39d8ff9640 1 -- 172.21.15.77:0/213497288 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f39cc089e10 con 0x7f39c4004800 2024-03-06T07:30:49.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.280+0000 7f39db7fe640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f39c806b500 0x7f39c806d9c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f39d0036000 tx=0x7f39d002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.284+0000 7f39d8ff9640 1 -- 172.21.15.77:0/213497288 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39dc155830 con 0x7f39c4004800 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.556+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39dc068ff0 con 0x7f39c4004800 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.557+0000 7f39d8ff9640 1 -- 172.21.15.77:0/213497288 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f39cc058620 con 0x7f39c4004800 2024-03-06T07:30:49.558 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:30:49.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f39c806b500 msgr2=0x7f39c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f39c806b500 0x7f39c806d9c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f39d0036000 tx=0x7f39d002e000 comp rx=0 tx=0).stop 2024-03-06T07:30:49.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 msgr2=0x7f39dc1552f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39dc1552f0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f39cc007340 tx=0x7f39cc007800 comp rx=0 tx=0).stop 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 shutdown_connections 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f39dc0b3bf0 0x7f39dc0b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f39dc0cd1b0 0x7f39dc0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f39c806b500 0x7f39c806d9c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 --2- 172.21.15.77:0/213497288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39c4004800 0x7f39dc1552f0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.558+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 >> 172.21.15.77:0/213497288 conn(0x7f39dc0d3d40 msgr2=0x7f39dc06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.559+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 shutdown_connections 2024-03-06T07:30:49.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.559+0000 7f39e27c7640 1 -- 172.21.15.77:0/213497288 wait complete. 2024-03-06T07:30:49.571 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:49.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.671+0000 7fa923793640 1 Processor -- start 2024-03-06T07:30:49.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.671+0000 7fa923793640 1 -- start start 2024-03-06T07:30:49.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa923793640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa91c0b4460 0x7fa91c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa923793640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa91c162970 0x7fa91c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa923793640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c0d22c0 con 0x7fa91c161cf0 2024-03-06T07:30:49.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa923793640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c161b00 con 0x7fa91c0b4460 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa923793640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c0cecc0 con 0x7fa91c162970 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa92152f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa91c0b4460 0x7fa91c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa921d30640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa91c161cf0 0x7fa91c0cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53578/0 (socket says 172.21.15.77:53578) 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa921d30640 1 -- 172.21.15.77:0/3417851141 learned_addr learned my addr 172.21.15.77:0/3417851141 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa920d2e640 1 --2- 172.21.15.77:0/3417851141 >> v2:172.21.15.158:3300/0 conn(0x7fa91c162970 0x7fa91c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa92152f640 1 -- 172.21.15.77:0/3417851141 >> v2:172.21.15.158:3300/0 conn(0x7fa91c162970 msgr2=0x7fa91c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.672+0000 7fa92152f640 1 --2- 172.21.15.77:0/3417851141 >> v2:172.21.15.158:3300/0 conn(0x7fa91c162970 0x7fa91c0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.673+0000 7fa92152f640 1 -- 172.21.15.77:0/3417851141 >> v1:172.21.15.77:6789/0 conn(0x7fa91c161cf0 legacy=0x7fa91c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.673+0000 7fa92152f640 1 -- 172.21.15.77:0/3417851141 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c0d25a0 con 0x7fa91c0b4460 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.673+0000 7fa913fff640 1 -- 172.21.15.77:0/3417851141 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695931114 0 0) 0x7fa91c0d22c0 con 0x7fa91c161cf0 2024-03-06T07:30:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.673+0000 7fa920d2e640 1 --2- 172.21.15.77:0/3417851141 >> v2:172.21.15.158:3300/0 conn(0x7fa91c162970 0x7fa91c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:49.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.673+0000 7fa92152f640 1 --2- 172.21.15.77:0/3417851141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c0b4460 0x7fa91c0d4800 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0d17e0 tx=0x7fa90c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8900ab0b71baabe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.673+0000 7fa913fff640 1 -- 172.21.15.77:0/3417851141 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c035020 con 0x7fa91c0b4460 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.673+0000 7fa913fff640 1 -- 172.21.15.77:0/3417851141 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c031830 con 0x7fa91c0b4460 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.673+0000 7fa913fff640 1 -- 172.21.15.77:0/3417851141 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c031c00 con 0x7fa91c0b4460 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.674+0000 7fa923793640 1 -- 172.21.15.77:0/3417851141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c0b4460 msgr2=0x7fa91c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.674+0000 7fa923793640 1 --2- 172.21.15.77:0/3417851141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c0b4460 0x7fa91c0d4800 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0d17e0 tx=0x7fa90c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.674+0000 7fa923793640 1 -- 172.21.15.77:0/3417851141 shutdown_connections 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.674+0000 7fa923793640 1 --2- 172.21.15.77:0/3417851141 >> v2:172.21.15.158:3300/0 conn(0x7fa91c162970 0x7fa91c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.674+0000 7fa923793640 1 --2- 172.21.15.77:0/3417851141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c0b4460 0x7fa91c0d4800 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.674+0000 7fa923793640 1 -- 172.21.15.77:0/3417851141 >> 172.21.15.77:0/3417851141 conn(0x7fa91c0d3ce0 msgr2=0x7fa91c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.674+0000 7fa923793640 1 -- 172.21.15.77:0/3417851141 shutdown_connections 2024-03-06T07:30:49.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.674+0000 7fa923793640 1 -- 172.21.15.77:0/3417851141 wait complete. 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.675+0000 7fa923793640 1 Processor -- start 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.675+0000 7fa923793640 1 -- start start 2024-03-06T07:30:49.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa923793640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0b4460 0x7fa91c0c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa923793640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c161cf0 0x7fa91c0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa923793640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa91c162970 0x7fa91c156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa923793640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0ce950 con 0x7fa91c161cf0 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa923793640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0d2b60 con 0x7fa91c0b4460 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa923793640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0b3ac0 con 0x7fa91c162970 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa920d2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c161cf0 0x7fa91c0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa920d2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c161cf0 0x7fa91c0cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56258/0 (socket says 172.21.15.77:56258) 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa920d2e640 1 -- 172.21.15.77:0/640612412 learned_addr learned my addr 172.21.15.77:0/640612412 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa92152f640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0b4460 0x7fa91c0c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa921d30640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa91c162970 0x7fa91c156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.676+0000 7fa920d2e640 1 -- 172.21.15.77:0/640612412 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa91c162970 msgr2=0x7fa91c156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa920d2e640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa91c162970 0x7fa91c156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa920d2e640 1 -- 172.21.15.77:0/640612412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0b4460 msgr2=0x7fa91c0c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa920d2e640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0b4460 0x7fa91c0c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa920d2e640 1 -- 172.21.15.77:0/640612412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c0c7e80 con 0x7fa91c161cf0 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa920d2e640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c161cf0 0x7fa91c0cd730 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fa91800c7f0 tx=0x7fa91800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa911ffb640 1 -- 172.21.15.77:0/640612412 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918015070 con 0x7fa91c161cf0 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa911ffb640 1 -- 172.21.15.77:0/640612412 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa918007a30 con 0x7fa91c161cf0 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91c0c8160 con 0x7fa91c161cf0 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa911ffb640 1 -- 172.21.15.77:0/640612412 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918007e00 con 0x7fa91c161cf0 2024-03-06T07:30:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.677+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c0c8720 con 0x7fa91c161cf0 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.678+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c04f330 con 0x7fa91c161cf0 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.678+0000 7fa911ffb640 1 -- 172.21.15.77:0/640612412 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa918018070 con 0x7fa91c161cf0 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.679+0000 7fa911ffb640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa90006b4d0 0x7fa90006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.679+0000 7fa92152f640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa90006b4d0 0x7fa90006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.679+0000 7fa911ffb640 1 -- 172.21.15.77:0/640612412 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa91808a070 con 0x7fa91c161cf0 2024-03-06T07:30:49.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.679+0000 7fa92152f640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa90006b4d0 0x7fa90006d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa90c036000 tx=0x7fa90c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.683+0000 7fa911ffb640 1 -- 172.21.15.77:0/640612412 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa918058910 con 0x7fa91c161cf0 2024-03-06T07:30:49.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.874+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa91c06b180 con 0x7fa91c161cf0 2024-03-06T07:30:49.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.875+0000 7fa911ffb640 1 -- 172.21.15.77:0/640612412 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa91805c9b0 con 0x7fa91c161cf0 2024-03-06T07:30:49.877 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa90006b4d0 msgr2=0x7fa90006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa90006b4d0 0x7fa90006d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa90c036000 tx=0x7fa90c008030 comp rx=0 tx=0).stop 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c161cf0 msgr2=0x7fa91c0cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c161cf0 0x7fa91c0cd730 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fa91800c7f0 tx=0x7fa91800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 shutdown_connections 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa90006b4d0 0x7fa90006d990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c161cf0 0x7fa91c0cd730 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa91c162970 0x7fa91c156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 --2- 172.21.15.77:0/640612412 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa91c0b4460 0x7fa91c0c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 >> 172.21.15.77:0/640612412 conn(0x7fa91c0d3ce0 msgr2=0x7fa91c0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 shutdown_connections 2024-03-06T07:30:49.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:49.877+0000 7fa923793640 1 -- 172.21.15.77:0/640612412 wait complete. 2024-03-06T07:30:49.889 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:30:49.889 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:49.905 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:50.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.043+0000 7fb2fcbc9640 1 Processor -- start 2024-03-06T07:30:50.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.043+0000 7fb2fcbc9640 1 -- start start 2024-03-06T07:30:50.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2fcbc9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb2f80d0d60 0x7fb2f80d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2fcbc9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f815e470 con 0x7fb2f815e090 2024-03-06T07:30:50.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2fcbc9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f80d18f0 con 0x7fb2f80d2920 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2fcbc9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f80b3cf0 con 0x7fb2f80d0d60 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2f659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb2f815e090 0x7fb2f80b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53582/0 (socket says 172.21.15.77:53582) 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2f659c640 1 -- 172.21.15.77:0/1205329353 learned_addr learned my addr 172.21.15.77:0/1205329353 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2f6d9d640 1 --2- 172.21.15.77:0/1205329353 >> v2:172.21.15.158:3300/0 conn(0x7fb2f80d0d60 0x7fb2f80d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2f559a640 1 -- 172.21.15.77:0/1205329353 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333753686 0 0) 0x7fb2f815e470 con 0x7fb2f815e090 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.045+0000 7fb2f559a640 1 -- 172.21.15.77:0/1205329353 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e0003520 con 0x7fb2f815e090 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.046+0000 7fb2f559a640 1 -- 172.21.15.77:0/1205329353 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3504946591 0 0) 0x7fb2e0003520 con 0x7fb2f815e090 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.046+0000 7fb2f6d9d640 1 -- 172.21.15.77:0/1205329353 >> v1:172.21.15.158:6789/0 conn(0x7fb2f80d2920 legacy=0x7fb2f80d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.046+0000 7fb2f6d9d640 1 -- 172.21.15.77:0/1205329353 >> v1:172.21.15.77:6789/0 conn(0x7fb2f815e090 legacy=0x7fb2f80b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.046+0000 7fb2f6d9d640 1 -- 172.21.15.77:0/1205329353 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f80d1b90 con 0x7fb2f80d0d60 2024-03-06T07:30:50.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.047+0000 7fb2f6d9d640 1 --2- 172.21.15.77:0/1205329353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d0d60 0x7fb2f80d47a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009d90 tx=0x7fb2e802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cad664fc57b44aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.047+0000 7fb2f559a640 1 -- 172.21.15.77:0/1205329353 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8039070 con 0x7fb2f80d0d60 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.047+0000 7fb2f559a640 1 -- 172.21.15.77:0/1205329353 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e802c940 con 0x7fb2f80d0d60 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.047+0000 7fb2f559a640 1 -- 172.21.15.77:0/1205329353 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0001670 con 0x7fb2f80d0d60 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.048+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1205329353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d0d60 msgr2=0x7fb2f80d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.048+0000 7fb2fcbc9640 1 --2- 172.21.15.77:0/1205329353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d0d60 0x7fb2f80d47a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009d90 tx=0x7fb2e802be10 comp rx=0 tx=0).stop 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.048+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1205329353 shutdown_connections 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.048+0000 7fb2fcbc9640 1 --2- 172.21.15.77:0/1205329353 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d0d60 0x7fb2f80d47a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.048+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1205329353 >> 172.21.15.77:0/1205329353 conn(0x7fb2f80d3c80 msgr2=0x7fb2f80d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.048+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1205329353 shutdown_connections 2024-03-06T07:30:50.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.048+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1205329353 wait complete. 2024-03-06T07:30:50.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.050+0000 7fb2fcbc9640 1 Processor -- start 2024-03-06T07:30:50.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.050+0000 7fb2fcbc9640 1 -- start start 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2fcbc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2f80d0d60 0x7fb2f80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2fcbc9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d2920 0x7fb2f81f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2fcbc9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f815e090 0x7fb2f80b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2fcbc9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f80b2df0 con 0x7fb2f815e090 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2fcbc9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f80cee30 con 0x7fb2f80d2920 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2fcbc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f80ce800 con 0x7fb2f80d0d60 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f815e090 0x7fb2f80b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f815e090 0x7fb2f80b8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56272/0 (socket says 172.21.15.77:56272) 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f6d9d640 1 -- 172.21.15.77:0/1082488197 learned_addr learned my addr 172.21.15.77:0/1082488197 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:50.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f659c640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2f80d0d60 0x7fb2f80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f5d9b640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d2920 0x7fb2f81f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f6d9d640 1 -- 172.21.15.77:0/1082488197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2f80d0d60 msgr2=0x7fb2f80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f6d9d640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2f80d0d60 0x7fb2f80cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f6d9d640 1 -- 172.21.15.77:0/1082488197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d2920 msgr2=0x7fb2f81f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f6d9d640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d2920 0x7fb2f81f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.051+0000 7fb2f6d9d640 1 -- 172.21.15.77:0/1082488197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f81f4a70 con 0x7fb2f815e090 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.052+0000 7fb2f6d9d640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f815e090 0x7fb2f80b8ef0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009eb0 tx=0x7fb2e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.052+0000 7fb2e6ffd640 1 -- 172.21.15.77:0/1082488197 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8044020 con 0x7fb2f815e090 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.052+0000 7fb2e6ffd640 1 -- 172.21.15.77:0/1082488197 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e8034d10 con 0x7fb2f815e090 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.052+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f81f4170 con 0x7fb2f815e090 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.052+0000 7fb2e6ffd640 1 -- 172.21.15.77:0/1082488197 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8039040 con 0x7fb2f815e090 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.052+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f81f4d10 con 0x7fb2f815e090 2024-03-06T07:30:50.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.053+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f804f330 con 0x7fb2f815e090 2024-03-06T07:30:50.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.053+0000 7fb2e6ffd640 1 -- 172.21.15.77:0/1082488197 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2e8047020 con 0x7fb2f815e090 2024-03-06T07:30:50.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.054+0000 7fb2e6ffd640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2d806b4d0 0x7fb2d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.054+0000 7fb2e6ffd640 1 -- 172.21.15.77:0/1082488197 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb2e80ac8d0 con 0x7fb2f815e090 2024-03-06T07:30:50.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.054+0000 7fb2f659c640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2d806b4d0 0x7fb2d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.054+0000 7fb2f659c640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2d806b4d0 0x7fb2d806d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0005880 tx=0x7fb2f00057f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.058+0000 7fb2e6ffd640 1 -- 172.21.15.77:0/1082488197 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2e804e030 con 0x7fb2f815e090 2024-03-06T07:30:50.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.301+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2f80d1230 con 0x7fb2f815e090 2024-03-06T07:30:50.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.302+0000 7fb2e6ffd640 1 -- 172.21.15.77:0/1082488197 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16716 (secure 0 0 0) 0x7fb2e807b0e0 con 0x7fb2f815e090 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":143,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:49.223707+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2d806b4d0 msgr2=0x7fb2d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2d806b4d0 0x7fb2d806d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0005880 tx=0x7fb2f00057f0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f815e090 msgr2=0x7fb2f80b8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f815e090 0x7fb2f80b8ef0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009eb0 tx=0x7fb2e8004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 shutdown_connections 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb2d806b4d0 0x7fb2d806d990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2f815e090 0x7fb2f80b8ef0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb2f80d2920 0x7fb2f81f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 --2- 172.21.15.77:0/1082488197 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb2f80d0d60 0x7fb2f80cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 >> 172.21.15.77:0/1082488197 conn(0x7fb2f80d3c80 msgr2=0x7fb2f80ca040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 shutdown_connections 2024-03-06T07:30:50.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.305+0000 7fb2fcbc9640 1 -- 172.21.15.77:0/1082488197 wait complete. 2024-03-06T07:30:50.321 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.674704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.414541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:50.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:50.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.423+0000 7f6bfc5bc640 1 Processor -- start 2024-03-06T07:30:50.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.423+0000 7f6bfc5bc640 1 -- start start 2024-03-06T07:30:50.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bfc5bc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6bf41614d0 0x7f6bf40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bfc5bc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf40d24b0 0x7f6bf40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bfc5bc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf41618b0 con 0x7f6bf40d1910 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bfc5bc640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf40cf400 con 0x7f6bf41614d0 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bfc5bc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40cf6a0 con 0x7f6bf40d24b0 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bf9b57640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6bf40d1910 0x7f6bf40d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53598/0 (socket says 172.21.15.77:53598) 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bf9b57640 1 -- 172.21.15.77:0/1769201335 learned_addr learned my addr 172.21.15.77:0/1769201335 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bfab59640 1 --2- 172.21.15.77:0/1769201335 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf40d24b0 0x7f6bf40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bfa358640 1 --2- 172.21.15.77:0/1769201335 >> v2:172.21.15.158:3300/0 conn(0x7f6bf41614d0 0x7f6bf40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bf9356640 1 -- 172.21.15.77:0/1769201335 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730931587 0 0) 0x7f6bf41618b0 con 0x7f6bf40d1910 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.424+0000 7f6bf9356640 1 -- 172.21.15.77:0/1769201335 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6be0003520 con 0x7f6bf40d1910 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.425+0000 7f6bf9356640 1 -- 172.21.15.77:0/1769201335 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 520414713 0 0) 0x7f6be0003520 con 0x7f6bf40d1910 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.425+0000 7f6bfab59640 1 -- 172.21.15.77:0/1769201335 >> v2:172.21.15.158:3300/0 conn(0x7f6bf41614d0 msgr2=0x7f6bf40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.425+0000 7f6bfab59640 1 --2- 172.21.15.77:0/1769201335 >> v2:172.21.15.158:3300/0 conn(0x7f6bf41614d0 0x7f6bf40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.425+0000 7f6bfab59640 1 -- 172.21.15.77:0/1769201335 >> v1:172.21.15.77:6789/0 conn(0x7f6bf40d1910 legacy=0x7f6bf40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.425+0000 7f6bfab59640 1 -- 172.21.15.77:0/1769201335 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf40cf980 con 0x7f6bf40d24b0 2024-03-06T07:30:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.425+0000 7f6bfa358640 1 --2- 172.21.15.77:0/1769201335 >> v2:172.21.15.158:3300/0 conn(0x7f6bf41614d0 0x7f6bf40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.426+0000 7f6bfab59640 1 --2- 172.21.15.77:0/1769201335 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf40d24b0 0x7f6bf40b0e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009d90 tx=0x7f6bec02bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27ed1046d9db7421 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.426+0000 7f6bf9356640 1 -- 172.21.15.77:0/1769201335 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec039070 con 0x7f6bf40d24b0 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.426+0000 7f6bf9356640 1 -- 172.21.15.77:0/1769201335 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bec02c8a0 con 0x7f6bf40d24b0 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.426+0000 7f6bf9356640 1 -- 172.21.15.77:0/1769201335 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4001670 con 0x7f6bf40d24b0 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.427+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1769201335 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf40d24b0 msgr2=0x7f6bf40b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.427+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1769201335 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf40d24b0 0x7f6bf40b0e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009d90 tx=0x7f6bec02bd70 comp rx=0 tx=0).stop 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.427+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1769201335 shutdown_connections 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.427+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1769201335 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf40d24b0 0x7f6bf40b0e30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.427+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1769201335 >> v2:172.21.15.158:3300/0 conn(0x7f6bf41614d0 0x7f6bf40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.427+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1769201335 >> 172.21.15.77:0/1769201335 conn(0x7f6bf40d3d40 msgr2=0x7f6bf40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.427+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1769201335 shutdown_connections 2024-03-06T07:30:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.427+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1769201335 wait complete. 2024-03-06T07:30:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.428+0000 7f6bfc5bc640 1 Processor -- start 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.428+0000 7f6bfc5bc640 1 -- start start 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfc5bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf40d1910 0x7f6bf414dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfc5bc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bf40d24b0 0x7f6bf406c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfc5bc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf41614d0 0x7f6bf406cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfc5bc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40b1370 con 0x7f6bf40d1910 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfc5bc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40b0910 con 0x7f6bf40d24b0 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfc5bc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40b14f0 con 0x7f6bf41614d0 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf40d1910 0x7f6bf414dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf40d1910 0x7f6bf414dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56284/0 (socket says 172.21.15.77:56284) 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 -- 172.21.15.77:0/1024925883 learned_addr learned my addr 172.21.15.77:0/1024925883 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bf9b57640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bf40d24b0 0x7f6bf406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfab59640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf41614d0 0x7f6bf406cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 -- 172.21.15.77:0/1024925883 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf41614d0 msgr2=0x7f6bf406cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf41614d0 0x7f6bf406cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 -- 172.21.15.77:0/1024925883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bf40d24b0 msgr2=0x7f6bf406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bf40d24b0 0x7f6bf406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 -- 172.21.15.77:0/1024925883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf414e270 con 0x7f6bf40d1910 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6bfa358640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf40d1910 0x7f6bf414dd30 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000e960 tx=0x7f6bf000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6beaffd640 1 -- 172.21.15.77:0/1024925883 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0017070 con 0x7f6bf40d1910 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.429+0000 7f6beaffd640 1 -- 172.21.15.77:0/1024925883 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf000ca00 con 0x7f6bf40d1910 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.430+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf40c9a30 con 0x7f6bf40d1910 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.430+0000 7f6beaffd640 1 -- 172.21.15.77:0/1024925883 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf000cdd0 con 0x7f6bf40d1910 2024-03-06T07:30:50.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.430+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf40c9ff0 con 0x7f6bf40d1910 2024-03-06T07:30:50.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.431+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf4068ff0 con 0x7f6bf40d1910 2024-03-06T07:30:50.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.432+0000 7f6beaffd640 1 -- 172.21.15.77:0/1024925883 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bf001a070 con 0x7f6bf40d1910 2024-03-06T07:30:50.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.432+0000 7f6beaffd640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bd806b4d0 0x7f6bd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.432+0000 7f6beaffd640 1 -- 172.21.15.77:0/1024925883 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6bf008a0f0 con 0x7f6bf40d1910 2024-03-06T07:30:50.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.432+0000 7f6bf9b57640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bd806b4d0 0x7f6bd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.432+0000 7f6bf9b57640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bd806b4d0 0x7f6bd806d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6be40017e0 tx=0x7f6be4001610 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.441+0000 7f6beaffd640 1 -- 172.21.15.77:0/1024925883 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bf414e270 con 0x7f6bf40d1910 2024-03-06T07:30:50.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.632+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6bf40d2a30 con 0x7f6bf40d1910 2024-03-06T07:30:50.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.633+0000 7f6beaffd640 1 -- 172.21.15.77:0/1024925883 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6bf0058880 con 0x7f6bf40d1910 2024-03-06T07:30:50.635 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:30:50.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.634+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bd806b4d0 msgr2=0x7f6bd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.634+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bd806b4d0 0x7f6bd806d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6be40017e0 tx=0x7f6be4001610 comp rx=0 tx=0).stop 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.634+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf40d1910 msgr2=0x7f6bf414dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.634+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf40d1910 0x7f6bf414dd30 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000e960 tx=0x7f6bf000ee20 comp rx=0 tx=0).stop 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.634+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 shutdown_connections 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.634+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6bf41614d0 0x7f6bf406cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.634+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6bf40d24b0 0x7f6bf406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.634+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6bd806b4d0 0x7f6bd806d990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.635+0000 7f6bfc5bc640 1 --2- 172.21.15.77:0/1024925883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf40d1910 0x7f6bf414dd30 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.635+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 >> 172.21.15.77:0/1024925883 conn(0x7f6bf40d3d40 msgr2=0x7f6bf415fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.635+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 shutdown_connections 2024-03-06T07:30:50.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.635+0000 7f6bfc5bc640 1 -- 172.21.15.77:0/1024925883 wait complete. 2024-03-06T07:30:50.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:50.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.750+0000 7f0ff9ced640 1 Processor -- start 2024-03-06T07:30:50.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.750+0000 7f0ff9ced640 1 -- start start 2024-03-06T07:30:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.751+0000 7f0ff9ced640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0ff40cfa90 0x7f0ff40cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.751+0000 7f0ff9ced640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4158ac0 con 0x7f0ff40b33a0 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.751+0000 7f0ff9ced640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff415b2a0 con 0x7f0ff41586e0 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.751+0000 7f0ff9ced640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff415b540 con 0x7f0ff40cfa90 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.751+0000 7f0ff3fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0ff40b33a0 0x7f0ff40b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53606/0 (socket says 172.21.15.77:53606) 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.751+0000 7f0ff3fff640 1 -- 172.21.15.77:0/157289688 learned_addr learned my addr 172.21.15.77:0/157289688 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff2ffd640 1 --2- 172.21.15.77:0/157289688 >> v2:172.21.15.158:3300/0 conn(0x7f0ff40cfa90 0x7f0ff40cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235844428 0 0) 0x7f0ff4158ac0 con 0x7f0ff40b33a0 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fdc003520 con 0x7f0ff40b33a0 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815120512 0 0) 0x7f0fdc003520 con 0x7f0ff40b33a0 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 >> v2:172.21.15.158:3300/0 conn(0x7f0ff40cfa90 msgr2=0x7f0ff40cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 --2- 172.21.15.77:0/157289688 >> v2:172.21.15.158:3300/0 conn(0x7f0ff40cfa90 0x7f0ff40cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 >> v1:172.21.15.158:6789/0 conn(0x7f0ff41586e0 legacy=0x7f0ff40aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff415b820 con 0x7f0ff40b33a0 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f0fec0019f0 con 0x7f0ff40b33a0 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 >> v1:172.21.15.77:6789/0 conn(0x7f0ff40b33a0 legacy=0x7f0ff40b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fdc0039b0 0x7f0fdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fdc004800 0x7f0fdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0051c0 0x7f0fdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff415b540 con 0x7f0fdc0039b0 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc00c060 con 0x7f0fdc004800 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc00e470 con 0x7f0fdc0051c0 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff37fe640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fdc0039b0 0x7f0fdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff2ffd640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fdc004800 0x7f0fdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff3fff640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0051c0 0x7f0fdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff37fe640 1 -- 172.21.15.77:0/157289688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0051c0 msgr2=0x7f0fdc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff37fe640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0051c0 0x7f0fdc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff37fe640 1 -- 172.21.15.77:0/157289688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fdc004800 msgr2=0x7f0fdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff37fe640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0fdc004800 0x7f0fdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff37fe640 1 -- 172.21.15.77:0/157289688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc003520 con 0x7f0fdc0039b0 2024-03-06T07:30:50.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.752+0000 7f0ff3fff640 1 -- 172.21.15.77:0/157289688 reap_dead start 2024-03-06T07:30:50.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff37fe640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fdc0039b0 0x7f0fdc003dd0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4001750 tx=0x7f0fdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62513ed9fee19c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4001370 con 0x7f0fdc0039b0 2024-03-06T07:30:50.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe40052d0 con 0x7f0fdc0039b0 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff27fc640 1 -- 172.21.15.77:0/157289688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4003830 con 0x7f0fdc0039b0 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff9ced640 1 -- 172.21.15.77:0/157289688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fdc0039b0 msgr2=0x7f0fdc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff9ced640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fdc0039b0 0x7f0fdc003dd0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4001750 tx=0x7f0fdc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff9ced640 1 -- 172.21.15.77:0/157289688 shutdown_connections 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff9ced640 1 --2- 172.21.15.77:0/157289688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fdc0039b0 0x7f0fdc003dd0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff9ced640 1 -- 172.21.15.77:0/157289688 >> 172.21.15.77:0/157289688 conn(0x7f0ff406b190 msgr2=0x7f0ff406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff9ced640 1 -- 172.21.15.77:0/157289688 shutdown_connections 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.754+0000 7f0ff9ced640 1 -- 172.21.15.77:0/157289688 wait complete. 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.755+0000 7f0ff9ced640 1 Processor -- start 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.756+0000 7f0ff9ced640 1 -- start start 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.756+0000 7f0ff9ced640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0039b0 0x7f0ff40cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.756+0000 7f0ff9ced640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ff4141230 0x7f0ff4156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.756+0000 7f0ff9ced640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ff4157100 0x7f0ff414d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.756+0000 7f0ff9ced640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4071350 con 0x7f0ff4157100 2024-03-06T07:30:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.756+0000 7f0ff9ced640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff415bde0 con 0x7f0ff4141230 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.756+0000 7f0ff9ced640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff40714d0 con 0x7f0fdc0039b0 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ff4157100 0x7f0ff414d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ff4157100 0x7f0ff414d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56314/0 (socket says 172.21.15.77:56314) 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 -- 172.21.15.77:0/1194703277 learned_addr learned my addr 172.21.15.77:0/1194703277 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff37fe640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0039b0 0x7f0ff40cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff2ffd640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ff4141230 0x7f0ff4156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 -- 172.21.15.77:0/1194703277 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0039b0 msgr2=0x7f0ff40cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0039b0 0x7f0ff40cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 -- 172.21.15.77:0/1194703277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ff4141230 msgr2=0x7f0ff4156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ff4141230 0x7f0ff4156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 -- 172.21.15.77:0/1194703277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40ce0d0 con 0x7f0ff4157100 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff3fff640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ff4157100 0x7f0ff414d3a0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f0fec000f80 tx=0x7f0fec0079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0fdbfff640 1 -- 172.21.15.77:0/1194703277 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec00aaa0 con 0x7f0ff4157100 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0fdbfff640 1 -- 172.21.15.77:0/1194703277 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec019070 con 0x7f0ff4157100 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0fdbfff640 1 -- 172.21.15.77:0/1194703277 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec0133f0 con 0x7f0ff4157100 2024-03-06T07:30:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff40ce3b0 con 0x7f0ff4157100 2024-03-06T07:30:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.757+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff40ce970 con 0x7f0ff4157100 2024-03-06T07:30:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.758+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff40732b0 con 0x7f0ff4157100 2024-03-06T07:30:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.759+0000 7f0fdbfff640 1 -- 172.21.15.77:0/1194703277 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fec013590 con 0x7f0ff4157100 2024-03-06T07:30:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.760+0000 7f0fdbfff640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fe006b4d0 0x7f0fe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.760+0000 7f0fdbfff640 1 -- 172.21.15.77:0/1194703277 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0fec08a5d0 con 0x7f0ff4157100 2024-03-06T07:30:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.760+0000 7f0ff37fe640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fe006b4d0 0x7f0fe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.760+0000 7f0ff37fe640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fe006b4d0 0x7f0fe006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4033000 tx=0x7f0fe4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.768+0000 7f0fdbfff640 1 -- 172.21.15.77:0/1194703277 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fec059430 con 0x7f0ff4157100 2024-03-06T07:30:50.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.958+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ff4068ff0 con 0x7f0ff4157100 2024-03-06T07:30:50.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.959+0000 7f0fdbfff640 1 -- 172.21.15.77:0/1194703277 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0fec058d30 con 0x7f0ff4157100 2024-03-06T07:30:50.960 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:30:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fe006b4d0 msgr2=0x7f0fe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fe006b4d0 0x7f0fe006d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4033000 tx=0x7f0fe4009030 comp rx=0 tx=0).stop 2024-03-06T07:30:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ff4157100 msgr2=0x7f0ff414d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ff4157100 0x7f0ff414d3a0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f0fec000f80 tx=0x7f0fec0079d0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 shutdown_connections 2024-03-06T07:30:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fe006b4d0 0x7f0fe006d990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ff4157100 0x7f0ff414d3a0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ff4141230 0x7f0ff4156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 --2- 172.21.15.77:0/1194703277 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0fdc0039b0 0x7f0ff40cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 >> 172.21.15.77:0/1194703277 conn(0x7f0ff406b190 msgr2=0x7f0ff40cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 shutdown_connections 2024-03-06T07:30:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:50.961+0000 7f0ff9ced640 1 -- 172.21.15.77:0/1194703277 wait complete. 2024-03-06T07:30:50.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:51.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.075+0000 7faf38eed640 1 Processor -- start 2024-03-06T07:30:51.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.075+0000 7faf38eed640 1 -- start start 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.076+0000 7faf38eed640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7faf341614d0 0x7faf340d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.076+0000 7faf38eed640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf340d1910 0x7faf340d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.076+0000 7faf38eed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf340d22e0 0x7faf340d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.076+0000 7faf38eed640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf340d2750 con 0x7faf340d1910 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.076+0000 7faf38eed640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf341618b0 con 0x7faf341614d0 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.076+0000 7faf38eed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf340cf400 con 0x7faf340d22e0 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.076+0000 7faf31d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf340d1910 0x7faf340d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf31d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf340d1910 0x7faf340d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56330/0 (socket says 172.21.15.77:56330) 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf31d9b640 1 -- 172.21.15.77:0/2092325801 learned_addr learned my addr 172.21.15.77:0/2092325801 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf31d9b640 1 -- 172.21.15.77:0/2092325801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf340d22e0 msgr2=0x7faf340d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf32d9d640 1 --2- 172.21.15.77:0/2092325801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf340d22e0 0x7faf340d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf3259c640 1 --2- 172.21.15.77:0/2092325801 >> v2:172.21.15.158:3300/0 conn(0x7faf341614d0 0x7faf340d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf31d9b640 1 --2- 172.21.15.77:0/2092325801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf340d22e0 0x7faf340d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf31d9b640 1 -- 172.21.15.77:0/2092325801 >> v2:172.21.15.158:3300/0 conn(0x7faf341614d0 msgr2=0x7faf340d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf31d9b640 1 --2- 172.21.15.77:0/2092325801 >> v2:172.21.15.158:3300/0 conn(0x7faf341614d0 0x7faf340d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf31d9b640 1 -- 172.21.15.77:0/2092325801 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf340cf6a0 con 0x7faf340d1910 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.077+0000 7faf31d9b640 1 --2- 172.21.15.77:0/2092325801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d1910 0x7faf340d1da0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7faf3408c470 tx=0x7faf2802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f910d2461a7a6bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf3159a640 1 -- 172.21.15.77:0/2092325801 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2802ca70 con 0x7faf340d1910 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf3159a640 1 -- 172.21.15.77:0/2092325801 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2802cc10 con 0x7faf340d1910 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf3159a640 1 -- 172.21.15.77:0/2092325801 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf280343f0 con 0x7faf340d1910 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 -- 172.21.15.77:0/2092325801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d1910 msgr2=0x7faf340d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 --2- 172.21.15.77:0/2092325801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d1910 0x7faf340d1da0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7faf3408c470 tx=0x7faf2802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 -- 172.21.15.77:0/2092325801 shutdown_connections 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 --2- 172.21.15.77:0/2092325801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf340d22e0 0x7faf340d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 --2- 172.21.15.77:0/2092325801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d1910 0x7faf340d1da0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 --2- 172.21.15.77:0/2092325801 >> v2:172.21.15.158:3300/0 conn(0x7faf341614d0 0x7faf340d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 -- 172.21.15.77:0/2092325801 >> 172.21.15.77:0/2092325801 conn(0x7faf340d3d40 msgr2=0x7faf340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 -- 172.21.15.77:0/2092325801 shutdown_connections 2024-03-06T07:30:51.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.078+0000 7faf38eed640 1 -- 172.21.15.77:0/2092325801 wait complete. 2024-03-06T07:30:51.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf38eed640 1 Processor -- start 2024-03-06T07:30:51.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf38eed640 1 -- start start 2024-03-06T07:30:51.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf38eed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf340d1910 0x7faf34155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf38eed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d22e0 0x7faf340b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf38eed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf341614d0 0x7faf340b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf38eed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340b0910 con 0x7faf340d22e0 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf38eed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340cfc60 con 0x7faf340d1910 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf38eed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf340d0be0 con 0x7faf341614d0 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf31d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d22e0 0x7faf340b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.080+0000 7faf31d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d22e0 0x7faf340b85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56346/0 (socket says 172.21.15.77:56346) 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf31d9b640 1 -- 172.21.15.77:0/3629091527 learned_addr learned my addr 172.21.15.77:0/3629091527 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf3259c640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf340d1910 0x7faf34155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf32d9d640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf341614d0 0x7faf340b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf31d9b640 1 -- 172.21.15.77:0/3629091527 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf341614d0 msgr2=0x7faf340b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf31d9b640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf341614d0 0x7faf340b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf31d9b640 1 -- 172.21.15.77:0/3629091527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf340d1910 msgr2=0x7faf34155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf31d9b640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf340d1910 0x7faf34155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf31d9b640 1 -- 172.21.15.77:0/3629091527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3415b790 con 0x7faf340d22e0 2024-03-06T07:30:51.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf31d9b640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d22e0 0x7faf340b85b0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7faf28000f80 tx=0x7faf28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf22ffd640 1 -- 172.21.15.77:0/3629091527 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28043020 con 0x7faf340d22e0 2024-03-06T07:30:51.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf22ffd640 1 -- 172.21.15.77:0/3629091527 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2803e450 con 0x7faf340d22e0 2024-03-06T07:30:51.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf340b7510 con 0x7faf340d22e0 2024-03-06T07:30:51.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf22ffd640 1 -- 172.21.15.77:0/3629091527 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2803e840 con 0x7faf340d22e0 2024-03-06T07:30:51.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.081+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf340b7ad0 con 0x7faf340d22e0 2024-03-06T07:30:51.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.082+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf34068ff0 con 0x7faf340d22e0 2024-03-06T07:30:51.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.083+0000 7faf22ffd640 1 -- 172.21.15.77:0/3629091527 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf28046020 con 0x7faf340d22e0 2024-03-06T07:30:51.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.083+0000 7faf22ffd640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf1406b4d0 0x7faf1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.084+0000 7faf22ffd640 1 -- 172.21.15.77:0/3629091527 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faf280acad0 con 0x7faf340d22e0 2024-03-06T07:30:51.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.084+0000 7faf3259c640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf1406b4d0 0x7faf1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.084+0000 7faf3259c640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf1406b4d0 0x7faf1406d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00a9a0 tx=0x7faf2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.092+0000 7faf22ffd640 1 -- 172.21.15.77:0/3629091527 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf2804d030 con 0x7faf340d22e0 2024-03-06T07:30:51.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.283+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faf3406d540 con 0x7faf340d22e0 2024-03-06T07:30:51.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.284+0000 7faf22ffd640 1 -- 172.21.15.77:0/3629091527 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7faf3415b790 con 0x7faf340d22e0 2024-03-06T07:30:51.286 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf1406b4d0 msgr2=0x7faf1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf1406b4d0 0x7faf1406d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00a9a0 tx=0x7faf2c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d22e0 msgr2=0x7faf340b85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d22e0 0x7faf340b85b0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7faf28000f80 tx=0x7faf28004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 shutdown_connections 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faf1406b4d0 0x7faf1406d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf340d22e0 0x7faf340b85b0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7faf341614d0 0x7faf340b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 --2- 172.21.15.77:0/3629091527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf340d1910 0x7faf34155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 >> 172.21.15.77:0/3629091527 conn(0x7faf340d3d40 msgr2=0x7faf340d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 shutdown_connections 2024-03-06T07:30:51.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.286+0000 7faf38eed640 1 -- 172.21.15.77:0/3629091527 wait complete. 2024-03-06T07:30:51.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.413+0000 7fd31adf7640 1 Processor -- start 2024-03-06T07:30:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.413+0000 7fd31adf7640 1 -- start start 2024-03-06T07:30:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd31adf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd31415bbf0 0x7fd314071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd31adf7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd314072490 con 0x7fd3140cd7c0 2024-03-06T07:30:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd31adf7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3140d0090 con 0x7fd3140cebc0 2024-03-06T07:30:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd31adf7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3140d0370 con 0x7fd31415bbf0 2024-03-06T07:30:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd318b93640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd31415bbf0 0x7fd314071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd318b93640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd31415bbf0 0x7fd314071e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56354/0 (socket says 172.21.15.77:56354) 2024-03-06T07:30:51.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd318b93640 1 -- 172.21.15.77:0/3573571217 learned_addr learned my addr 172.21.15.77:0/3573571217 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:51.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd319394640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd3140cd7c0 0x7fd3140cdbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53612/0 (socket says 172.21.15.77:53612) 2024-03-06T07:30:51.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd318b93640 1 -- 172.21.15.77:0/3573571217 >> v1:172.21.15.158:6789/0 conn(0x7fd3140cebc0 legacy=0x7fd31415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd318b93640 1 -- 172.21.15.77:0/3573571217 >> v1:172.21.15.77:6789/0 conn(0x7fd3140cd7c0 legacy=0x7fd3140cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd318b93640 1 -- 172.21.15.77:0/3573571217 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3140d0650 con 0x7fd31415bbf0 2024-03-06T07:30:51.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.414+0000 7fd3137fe640 1 -- 172.21.15.77:0/3573571217 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321904582 0 0) 0x7fd314072490 con 0x7fd3140cd7c0 2024-03-06T07:30:51.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.415+0000 7fd318b93640 1 --2- 172.21.15.77:0/3573571217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314071e00 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fd308009870 tx=0x7fd30802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7703bd0952561 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.415+0000 7fd3137fe640 1 -- 172.21.15.77:0/3573571217 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30802f840 con 0x7fd31415bbf0 2024-03-06T07:30:51.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.415+0000 7fd3137fe640 1 -- 172.21.15.77:0/3573571217 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30802f9e0 con 0x7fd31415bbf0 2024-03-06T07:30:51.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.415+0000 7fd3137fe640 1 -- 172.21.15.77:0/3573571217 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30802fdb0 con 0x7fd31415bbf0 2024-03-06T07:30:51.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.416+0000 7fd31adf7640 1 -- 172.21.15.77:0/3573571217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 msgr2=0x7fd314071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.416+0000 7fd31adf7640 1 --2- 172.21.15.77:0/3573571217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314071e00 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fd308009870 tx=0x7fd30802ba20 comp rx=0 tx=0).stop 2024-03-06T07:30:51.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.416+0000 7fd31adf7640 1 -- 172.21.15.77:0/3573571217 shutdown_connections 2024-03-06T07:30:51.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.416+0000 7fd31adf7640 1 --2- 172.21.15.77:0/3573571217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314071e00 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.416+0000 7fd31adf7640 1 -- 172.21.15.77:0/3573571217 >> 172.21.15.77:0/3573571217 conn(0x7fd3140b1c80 msgr2=0x7fd3140cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.416+0000 7fd31adf7640 1 -- 172.21.15.77:0/3573571217 shutdown_connections 2024-03-06T07:30:51.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.416+0000 7fd31adf7640 1 -- 172.21.15.77:0/3573571217 wait complete. 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.417+0000 7fd31adf7640 1 Processor -- start 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.417+0000 7fd31adf7640 1 -- start start 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd31adf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3140cd7c0 0x7fd314160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd31adf7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cebc0 0x7fd314156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd31adf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd31adf7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140b3680 con 0x7fd31415bbf0 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd31adf7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140cf010 con 0x7fd3140cebc0 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd31adf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140b3800 con 0x7fd3140cd7c0 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56368/0 (socket says 172.21.15.77:56368) 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 -- 172.21.15.77:0/2640776740 learned_addr learned my addr 172.21.15.77:0/2640776740 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:51.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd318b93640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3140cd7c0 0x7fd314160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd313fff640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cebc0 0x7fd314156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 -- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3140cd7c0 msgr2=0x7fd314160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3140cd7c0 0x7fd314160430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 -- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cebc0 msgr2=0x7fd314156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cebc0 0x7fd314156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 -- 172.21.15.77:0/2640776740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31413eda0 con 0x7fd31415bbf0 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd313fff640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cebc0 0x7fd314156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.418+0000 7fd319394640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314157030 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00c700 tx=0x7fd30c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.419+0000 7fd3117fa640 1 -- 172.21.15.77:0/2640776740 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c018070 con 0x7fd31415bbf0 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.419+0000 7fd3117fa640 1 -- 172.21.15.77:0/2640776740 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c00ce10 con 0x7fd31415bbf0 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.419+0000 7fd3117fa640 1 -- 172.21.15.77:0/2640776740 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c00a860 con 0x7fd31415bbf0 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.419+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31413bec0 con 0x7fd31415bbf0 2024-03-06T07:30:51.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.419+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31413c480 con 0x7fd31415bbf0 2024-03-06T07:30:51.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.420+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31404f330 con 0x7fd31415bbf0 2024-03-06T07:30:51.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.420+0000 7fd3117fa640 1 -- 172.21.15.77:0/2640776740 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd30c00aa00 con 0x7fd31415bbf0 2024-03-06T07:30:51.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.420+0000 7fd3117fa640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd2f806b4d0 0x7fd2f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.421+0000 7fd318b93640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd2f806b4d0 0x7fd2f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.421+0000 7fd3117fa640 1 -- 172.21.15.77:0/2640776740 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd30c089870 con 0x7fd31415bbf0 2024-03-06T07:30:51.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.421+0000 7fd318b93640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd2f806b4d0 0x7fd2f806d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd308036000 tx=0x7fd308008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.425+0000 7fd3117fa640 1 -- 172.21.15.77:0/2640776740 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd30c058650 con 0x7fd31415bbf0 2024-03-06T07:30:51.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.614+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd314150260 con 0x7fd31415bbf0 2024-03-06T07:30:51.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.615+0000 7fd3117fa640 1 -- 172.21.15.77:0/2640776740 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd30c057f50 con 0x7fd31415bbf0 2024-03-06T07:30:51.617 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2024-03-06T07:30:51.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd2f806b4d0 msgr2=0x7fd2f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd2f806b4d0 0x7fd2f806d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd308036000 tx=0x7fd308008030 comp rx=0 tx=0).stop 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 msgr2=0x7fd314157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314157030 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00c700 tx=0x7fd30c00cbc0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 shutdown_connections 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd2f806b4d0 0x7fd2f806d990 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd31415bbf0 0x7fd314157030 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3140cebc0 0x7fd314156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 --2- 172.21.15.77:0/2640776740 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3140cd7c0 0x7fd314160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 >> 172.21.15.77:0/2640776740 conn(0x7fd3140b1c80 msgr2=0x7fd31415ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 shutdown_connections 2024-03-06T07:30:51.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:51.617+0000 7fd31adf7640 1 -- 172.21.15.77:0/2640776740 wait complete. 2024-03-06T07:30:51.629 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-06T07:30:51.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:51.630 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:51.654 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[85143]: fuse finished with error 0 and tester_r 0 2024-03-06T07:30:51.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:51.748 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:30:51.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:51.748 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:51.763 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi077.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:30:51.925 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:30:51.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:30:52.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.029+0000 7f4f0cc81640 1 Processor -- start 2024-03-06T07:30:52.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.029+0000 7f4f0cc81640 1 -- start start 2024-03-06T07:30:52.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4f0cc81640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4f080d06e0 0x7f4f0815fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4f0cc81640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4f080d0ac0 0x7f4f080b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4f0cc81640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f080d47f0 con 0x7f4f080b4d10 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4f0cc81640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f080ced70 con 0x7f4f080d0ac0 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4f0cc81640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f080d1ab0 con 0x7f4f080d06e0 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4efdd9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4f080d0ac0 0x7f4f080b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4f06d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4f080b4d10 0x7f4f080cf200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53626/0 (socket says 172.21.15.77:53626) 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4f06d9d640 1 -- 172.21.15.77:0/3087316853 learned_addr learned my addr 172.21.15.77:0/3087316853 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4f0659c640 1 --2- 172.21.15.77:0/3087316853 >> v2:172.21.15.158:3300/0 conn(0x7f4f080d06e0 0x7f4f0815fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4efdd9b640 1 -- 172.21.15.77:0/3087316853 >> v2:172.21.15.158:3300/0 conn(0x7f4f080d06e0 msgr2=0x7f4f0815fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4efdd9b640 1 --2- 172.21.15.77:0/3087316853 >> v2:172.21.15.158:3300/0 conn(0x7f4f080d06e0 0x7f4f0815fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4efdd9b640 1 -- 172.21.15.77:0/3087316853 >> v1:172.21.15.77:6789/0 conn(0x7f4f080b4d10 legacy=0x7f4f080cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.030+0000 7f4efdd9b640 1 -- 172.21.15.77:0/3087316853 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f080d1d90 con 0x7f4f080d0ac0 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.031+0000 7f4efdd9b640 1 --2- 172.21.15.77:0/3087316853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d0ac0 0x7f4f080b47d0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f4f080b6850 tx=0x7f4ef002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f517337e38126b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.031+0000 7f4f05d9b640 1 -- 172.21.15.77:0/3087316853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0035020 con 0x7f4f080d0ac0 2024-03-06T07:30:52.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.031+0000 7f4f05d9b640 1 -- 172.21.15.77:0/3087316853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef0031830 con 0x7f4f080d0ac0 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.031+0000 7f4f05d9b640 1 -- 172.21.15.77:0/3087316853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0031c00 con 0x7f4f080d0ac0 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.031+0000 7f4f0cc81640 1 -- 172.21.15.77:0/3087316853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d0ac0 msgr2=0x7f4f080b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.031+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/3087316853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d0ac0 0x7f4f080b47d0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f4f080b6850 tx=0x7f4ef002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.032+0000 7f4f0cc81640 1 -- 172.21.15.77:0/3087316853 shutdown_connections 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.032+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/3087316853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d0ac0 0x7f4f080b47d0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.032+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/3087316853 >> v2:172.21.15.158:3300/0 conn(0x7f4f080d06e0 0x7f4f0815fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.032+0000 7f4f0cc81640 1 -- 172.21.15.77:0/3087316853 >> 172.21.15.77:0/3087316853 conn(0x7f4f080d3cd0 msgr2=0x7f4f080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.032+0000 7f4f0cc81640 1 -- 172.21.15.77:0/3087316853 shutdown_connections 2024-03-06T07:30:52.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.032+0000 7f4f0cc81640 1 -- 172.21.15.77:0/3087316853 wait complete. 2024-03-06T07:30:52.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.033+0000 7f4f0cc81640 1 Processor -- start 2024-03-06T07:30:52.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.033+0000 7f4f0cc81640 1 -- start start 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4f0cc81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f080b4d10 0x7f4f081f5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4f0cc81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d06e0 0x7f4f080cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4f0cc81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f080d0ac0 0x7f4f0815c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4f0cc81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080b44d0 con 0x7f4f080d06e0 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4f0cc81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080ce990 con 0x7f4f080d0ac0 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4f0cc81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080d0f30 con 0x7f4f080b4d10 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d06e0 0x7f4f080cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d06e0 0x7f4f080cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56398/0 (socket says 172.21.15.77:56398) 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 -- 172.21.15.77:0/2448096535 learned_addr learned my addr 172.21.15.77:0/2448096535 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4f06d9d640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f080d0ac0 0x7f4f0815c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4f0659c640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f080b4d10 0x7f4f081f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 -- 172.21.15.77:0/2448096535 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f080b4d10 msgr2=0x7f4f081f5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f080b4d10 0x7f4f081f5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 -- 172.21.15.77:0/2448096535 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f080d0ac0 msgr2=0x7f4f0815c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f080d0ac0 0x7f4f0815c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 -- 172.21.15.77:0/2448096535 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f081f5880 con 0x7f4f080d06e0 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4efdd9b640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d06e0 0x7f4f080cd490 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0000c00 tx=0x7f4ef00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.034+0000 7f4eff7fe640 1 -- 172.21.15.77:0/2448096535 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0040070 con 0x7f4f080d06e0 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.035+0000 7f4eff7fe640 1 -- 172.21.15.77:0/2448096535 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef0039980 con 0x7f4f080d06e0 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.035+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f080cb460 con 0x7f4f080d06e0 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.035+0000 7f4eff7fe640 1 -- 172.21.15.77:0/2448096535 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0035040 con 0x7f4f080d06e0 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.035+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f080cba20 con 0x7f4f080d06e0 2024-03-06T07:30:52.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.036+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f080d3000 con 0x7f4f080d06e0 2024-03-06T07:30:52.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.037+0000 7f4eff7fe640 1 -- 172.21.15.77:0/2448096535 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ef004b060 con 0x7f4f080d06e0 2024-03-06T07:30:52.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.037+0000 7f4eff7fe640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ee806b4d0 0x7f4ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.037+0000 7f4eff7fe640 1 -- 172.21.15.77:0/2448096535 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4ef00abba0 con 0x7f4f080d06e0 2024-03-06T07:30:52.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.037+0000 7f4f0659c640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ee806b4d0 0x7f4ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.037+0000 7f4f0659c640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ee806b4d0 0x7f4ee806d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8006fd0 tx=0x7f4ef8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.041+0000 7f4eff7fe640 1 -- 172.21.15.77:0/2448096535 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f080d3000 con 0x7f4f080d06e0 2024-03-06T07:30:52.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:52.302+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f4f08068ff0 con 0x7f4f080d06e0 2024-03-06T07:30:53.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.243+0000 7f4eff7fe640 1 -- 172.21.15.77:0/2448096535 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v104) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4f081f5880 con 0x7f4f080d06e0 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ee806b4d0 msgr2=0x7f4ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ee806b4d0 0x7f4ee806d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8006fd0 tx=0x7f4ef8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d06e0 msgr2=0x7f4f080cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d06e0 0x7f4f080cd490 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0000c00 tx=0x7f4ef00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 shutdown_connections 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ee806b4d0 0x7f4ee806d990 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f080d06e0 0x7f4f080cd490 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4f080d0ac0 0x7f4f0815c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 --2- 172.21.15.77:0/2448096535 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4f080b4d10 0x7f4f081f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 >> 172.21.15.77:0/2448096535 conn(0x7f4f080d3cd0 msgr2=0x7f4f080b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 shutdown_connections 2024-03-06T07:30:53.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.246+0000 7f4f0cc81640 1 -- 172.21.15.77:0/2448096535 wait complete. 2024-03-06T07:30:53.259 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:53.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.358+0000 7f491ebf7640 1 Processor -- start 2024-03-06T07:30:53.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.358+0000 7f491ebf7640 1 -- start start 2024-03-06T07:30:53.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f491ebf7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f491815dfe0 0x7f49180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f491ebf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49180cf830 0x7f49180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f491ebf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49180d27f0 0x7f49180d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f491ebf7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49180b31e0 con 0x7f49180cf830 2024-03-06T07:30:53.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f491ebf7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49180b3d60 con 0x7f491815dfe0 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f491ebf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49180b3430 con 0x7f49180d27f0 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f4917fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49180cf830 0x7f49180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f4917fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49180cf830 0x7f49180d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56402/0 (socket says 172.21.15.77:56402) 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f4917fff640 1 -- 172.21.15.77:0/4093735000 learned_addr learned my addr 172.21.15.77:0/4093735000 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f491c993640 1 --2- 172.21.15.77:0/4093735000 >> v2:172.21.15.158:3300/0 conn(0x7f491815dfe0 0x7f49180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f491d194640 1 --2- 172.21.15.77:0/4093735000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49180d27f0 0x7f49180d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f4917fff640 1 -- 172.21.15.77:0/4093735000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49180d27f0 msgr2=0x7f49180d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f4917fff640 1 --2- 172.21.15.77:0/4093735000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49180d27f0 0x7f49180d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f4917fff640 1 -- 172.21.15.77:0/4093735000 >> v2:172.21.15.158:3300/0 conn(0x7f491815dfe0 msgr2=0x7f49180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f4917fff640 1 --2- 172.21.15.77:0/4093735000 >> v2:172.21.15.158:3300/0 conn(0x7f491815dfe0 0x7f49180cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.359+0000 7f4917fff640 1 -- 172.21.15.77:0/4093735000 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180b3710 con 0x7f49180cf830 2024-03-06T07:30:53.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.360+0000 7f4917fff640 1 --2- 172.21.15.77:0/4093735000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180d22b0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4918067990 tx=0x7f490402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de709f19edfbc02c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.360+0000 7f49177fe640 1 -- 172.21.15.77:0/4093735000 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904035020 con 0x7f49180cf830 2024-03-06T07:30:53.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.360+0000 7f49177fe640 1 -- 172.21.15.77:0/4093735000 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904031870 con 0x7f49180cf830 2024-03-06T07:30:53.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.360+0000 7f49177fe640 1 -- 172.21.15.77:0/4093735000 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904031c40 con 0x7f49180cf830 2024-03-06T07:30:53.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 -- 172.21.15.77:0/4093735000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 msgr2=0x7f49180d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 --2- 172.21.15.77:0/4093735000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180d22b0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4918067990 tx=0x7f490402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 -- 172.21.15.77:0/4093735000 shutdown_connections 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 --2- 172.21.15.77:0/4093735000 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49180d27f0 0x7f49180d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 --2- 172.21.15.77:0/4093735000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180d22b0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 --2- 172.21.15.77:0/4093735000 >> v2:172.21.15.158:3300/0 conn(0x7f491815dfe0 0x7f49180cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 -- 172.21.15.77:0/4093735000 >> 172.21.15.77:0/4093735000 conn(0x7f49180d3cd0 msgr2=0x7f49180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 -- 172.21.15.77:0/4093735000 shutdown_connections 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.361+0000 7f491ebf7640 1 -- 172.21.15.77:0/4093735000 wait complete. 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.362+0000 7f491ebf7640 1 Processor -- start 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.362+0000 7f491ebf7640 1 -- start start 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491ebf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491ebf7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49180d27f0 0x7f491815be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491ebf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f491815dfe0 0x7f49180b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491ebf7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49180d1df0 con 0x7f49180cf830 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491ebf7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49180d1c70 con 0x7f49180d27f0 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491ebf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491815e3c0 con 0x7f491815dfe0 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491c993640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491c993640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56416/0 (socket says 172.21.15.77:56416) 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491c993640 1 -- 172.21.15.77:0/1216333019 learned_addr learned my addr 172.21.15.77:0/1216333019 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491c993640 1 -- 172.21.15.77:0/1216333019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f491815dfe0 msgr2=0x7f49180b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491c993640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f491815dfe0 0x7f49180b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491c993640 1 -- 172.21.15.77:0/1216333019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49180d27f0 msgr2=0x7f491815be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f4917fff640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49180d27f0 0x7f491815be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491c993640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49180d27f0 0x7f491815be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f491c993640 1 -- 172.21.15.77:0/1216333019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180a1b70 con 0x7f49180cf830 2024-03-06T07:30:53.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.363+0000 7f4917fff640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49180d27f0 0x7f491815be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.364+0000 7f491c993640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180b68d0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f490c00d5f0 tx=0x7f490c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.364+0000 7f49157fa640 1 -- 172.21.15.77:0/1216333019 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c0125f0 con 0x7f49180cf830 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.364+0000 7f49157fa640 1 -- 172.21.15.77:0/1216333019 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c019070 con 0x7f49180cf830 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.364+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49180b6fd0 con 0x7f49180cf830 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.364+0000 7f49157fa640 1 -- 172.21.15.77:0/1216333019 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c0129d0 con 0x7f49180cf830 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.364+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49180b7590 con 0x7f49180cf830 2024-03-06T07:30:53.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.365+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491804f330 con 0x7f49180cf830 2024-03-06T07:30:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.366+0000 7f49157fa640 1 -- 172.21.15.77:0/1216333019 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f490c012b70 con 0x7f49180cf830 2024-03-06T07:30:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.366+0000 7f49157fa640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f48fc06b4d0 0x7f48fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.366+0000 7f4917fff640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f48fc06b4d0 0x7f48fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.366+0000 7f49157fa640 1 -- 172.21.15.77:0/1216333019 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f490c08a630 con 0x7f49180cf830 2024-03-06T07:30:53.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.367+0000 7f4917fff640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f48fc06b4d0 0x7f48fc06d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4904036000 tx=0x7f4904008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.371+0000 7f49157fa640 1 -- 172.21.15.77:0/1216333019 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f490c059490 con 0x7f49180cf830 2024-03-06T07:30:53.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.610+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49180b7930 con 0x7f49180cf830 2024-03-06T07:30:53.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.611+0000 7f49157fa640 1 -- 172.21.15.77:0/1216333019 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f490c058d90 con 0x7f49180cf830 2024-03-06T07:30:53.613 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:53.613 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:30:53.613 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.612+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f48fc06b4d0 msgr2=0x7f48fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.612+0000 7f491ebf7640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f48fc06b4d0 0x7f48fc06d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4904036000 tx=0x7f4904008030 comp rx=0 tx=0).stop 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.612+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 msgr2=0x7f49180b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.612+0000 7f491ebf7640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180b68d0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f490c00d5f0 tx=0x7f490c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.613+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 shutdown_connections 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.613+0000 7f491ebf7640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f491815dfe0 0x7f49180b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.613+0000 7f491ebf7640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49180d27f0 0x7f491815be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.613+0000 7f491ebf7640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f48fc06b4d0 0x7f48fc06d990 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.613+0000 7f491ebf7640 1 --2- 172.21.15.77:0/1216333019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49180cf830 0x7f49180b68d0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.613+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 >> 172.21.15.77:0/1216333019 conn(0x7f49180d3cd0 msgr2=0x7f491806bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.613+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 shutdown_connections 2024-03-06T07:30:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:53.613+0000 7f491ebf7640 1 -- 172.21.15.77:0/1216333019 wait complete. 2024-03-06T07:30:53.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:30:53.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:30:54.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.927+0000 7f6ec382a640 1 Processor -- start 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.927+0000 7f6ec382a640 1 -- start start 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.928+0000 7f6ec382a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc144510 0x7f6ebc06d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.928+0000 7f6ec382a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ebc15efb0 0x7f6ebc0b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.928+0000 7f6ec382a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc1448f0 con 0x7f6ebc06f290 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.928+0000 7f6ec382a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc06dd90 con 0x7f6ebc15efb0 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.928+0000 7f6ec382a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0b3b00 con 0x7f6ebc144510 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.928+0000 7f6ec1dc7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ebc15efb0 0x7f6ebc0b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.928+0000 7f6ec1dc7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ebc15efb0 0x7f6ebc0b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56420/0 (socket says 172.21.15.77:56420) 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.928+0000 7f6ec1dc7640 1 -- 172.21.15.77:0/4119608896 learned_addr learned my addr 172.21.15.77:0/4119608896 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.929+0000 7f6ec1dc7640 1 -- 172.21.15.77:0/4119608896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc144510 msgr2=0x7f6ebc06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.929+0000 7f6ec1dc7640 1 --2- 172.21.15.77:0/4119608896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc144510 0x7f6ebc06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.929+0000 7f6ec1dc7640 1 -- 172.21.15.77:0/4119608896 >> v1:172.21.15.158:6789/0 conn(0x7f6ebc06f290 legacy=0x7f6ebc15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.929+0000 7f6ec1dc7640 1 -- 172.21.15.77:0/4119608896 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc0d0cf0 con 0x7f6ebc15efb0 2024-03-06T07:30:54.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.929+0000 7f6ec1dc7640 1 --2- 172.21.15.77:0/4119608896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc15efb0 0x7f6ebc0b3f90 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400d150 tx=0x7f6eb402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5daf1592aca7dc30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.929+0000 7f6eb3fff640 1 -- 172.21.15.77:0/4119608896 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb402e9e0 con 0x7f6ebc15efb0 2024-03-06T07:30:54.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.929+0000 7f6eb3fff640 1 -- 172.21.15.77:0/4119608896 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb402eb80 con 0x7f6ebc15efb0 2024-03-06T07:30:54.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.929+0000 7f6eb3fff640 1 -- 172.21.15.77:0/4119608896 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4009600 con 0x7f6ebc15efb0 2024-03-06T07:30:54.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.930+0000 7f6ec382a640 1 -- 172.21.15.77:0/4119608896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc15efb0 msgr2=0x7f6ebc0b3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.930+0000 7f6ec382a640 1 --2- 172.21.15.77:0/4119608896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc15efb0 0x7f6ebc0b3f90 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400d150 tx=0x7f6eb402e040 comp rx=0 tx=0).stop 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.930+0000 7f6ec382a640 1 -- 172.21.15.77:0/4119608896 shutdown_connections 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.930+0000 7f6ec382a640 1 --2- 172.21.15.77:0/4119608896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc15efb0 0x7f6ebc0b3f90 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.930+0000 7f6ec382a640 1 --2- 172.21.15.77:0/4119608896 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc144510 0x7f6ebc06d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.930+0000 7f6ec382a640 1 -- 172.21.15.77:0/4119608896 >> 172.21.15.77:0/4119608896 conn(0x7f6ebc06c0c0 msgr2=0x7f6ebc06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.930+0000 7f6ec382a640 1 -- 172.21.15.77:0/4119608896 shutdown_connections 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.930+0000 7f6ec382a640 1 -- 172.21.15.77:0/4119608896 wait complete. 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.931+0000 7f6ec382a640 1 Processor -- start 2024-03-06T07:30:54.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.931+0000 7f6ec382a640 1 -- start start 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec382a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc06f290 0x7f6ebc15a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec382a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc144510 0x7f6ebc139d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec382a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ebc15efb0 0x7f6ebc13a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec382a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc06f6f0 con 0x7f6ebc144510 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec382a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0d12b0 con 0x7f6ebc15efb0 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec382a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc06f870 con 0x7f6ebc06f290 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec0dc5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc144510 0x7f6ebc139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec0dc5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc144510 0x7f6ebc139d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56422/0 (socket says 172.21.15.77:56422) 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.932+0000 7f6ec0dc5640 1 -- 172.21.15.77:0/2611856688 learned_addr learned my addr 172.21.15.77:0/2611856688 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:54.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec1dc7640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ebc15efb0 0x7f6ebc13a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec15c6640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc06f290 0x7f6ebc15a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec0dc5640 1 -- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc06f290 msgr2=0x7f6ebc15a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec0dc5640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc06f290 0x7f6ebc15a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec0dc5640 1 -- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ebc15efb0 msgr2=0x7f6ebc13a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec0dc5640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ebc15efb0 0x7f6ebc13a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec0dc5640 1 -- 172.21.15.77:0/2611856688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc139ac0 con 0x7f6ebc144510 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec1dc7640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ebc15efb0 0x7f6ebc13a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec15c6640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc06f290 0x7f6ebc15a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec0dc5640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc144510 0x7f6ebc139d60 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b3e0 tx=0x7f6eb800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6eb1ffb640 1 -- 172.21.15.77:0/2611856688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8015020 con 0x7f6ebc144510 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6eb1ffb640 1 -- 172.21.15.77:0/2611856688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb8011850 con 0x7f6ebc144510 2024-03-06T07:30:54.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc15bd50 con 0x7f6ebc144510 2024-03-06T07:30:54.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6eb1ffb640 1 -- 172.21.15.77:0/2611856688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8011c40 con 0x7f6ebc144510 2024-03-06T07:30:54.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.933+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc15c310 con 0x7f6ebc144510 2024-03-06T07:30:54.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.935+0000 7f6eb1ffb640 1 -- 172.21.15.77:0/2611856688 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6eb8018020 con 0x7f6ebc144510 2024-03-06T07:30:54.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.935+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc04f330 con 0x7f6ebc144510 2024-03-06T07:30:54.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.935+0000 7f6eb1ffb640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ea006b4d0 0x7f6ea006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.935+0000 7f6ec15c6640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ea006b4d0 0x7f6ea006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.935+0000 7f6eb1ffb640 1 -- 172.21.15.77:0/2611856688 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6eb808a160 con 0x7f6ebc144510 2024-03-06T07:30:54.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.936+0000 7f6ec15c6640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ea006b4d0 0x7f6ea006d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6eac006fd0 tx=0x7f6eac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:54.939+0000 7f6eb1ffb640 1 -- 172.21.15.77:0/2611856688 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6eb801f020 con 0x7f6ebc144510 2024-03-06T07:30:55.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.153+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ebc072170 con 0x7f6ebc144510 2024-03-06T07:30:55.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.154+0000 7f6eb1ffb640 1 -- 172.21.15.77:0/2611856688 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f6eb80172c0 con 0x7f6ebc144510 2024-03-06T07:30:55.156 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:55.156 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:30:55.156 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:30:55.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ea006b4d0 msgr2=0x7f6ea006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ea006b4d0 0x7f6ea006d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6eac006fd0 tx=0x7f6eac008040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc144510 msgr2=0x7f6ebc139d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc144510 0x7f6ebc139d60 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b3e0 tx=0x7f6eb800b8a0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 shutdown_connections 2024-03-06T07:30:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6ea006b4d0 0x7f6ea006d990 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc144510 0x7f6ebc139d60 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ebc15efb0 0x7f6ebc13a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 --2- 172.21.15.77:0/2611856688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ebc06f290 0x7f6ebc15a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 >> 172.21.15.77:0/2611856688 conn(0x7f6ebc06c0c0 msgr2=0x7f6ebc141c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 shutdown_connections 2024-03-06T07:30:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:55.156+0000 7f6ec382a640 1 -- 172.21.15.77:0/2611856688 wait complete. 2024-03-06T07:30:55.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:30:55.370 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:30:56.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.471+0000 7f0a572b5640 1 Processor -- start 2024-03-06T07:30:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.472+0000 7f0a572b5640 1 -- start start 2024-03-06T07:30:56.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.472+0000 7f0a572b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a5006c6d0 0x7f0a5015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.472+0000 7f0a572b5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0a50066d90 0x7f0a500ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.472+0000 7f0a572b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a500cd450 con 0x7f0a500b4f70 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.472+0000 7f0a572b5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a5015e9f0 con 0x7f0a50066d90 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.472+0000 7f0a572b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a500cda10 con 0x7f0a5006c6d0 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a55051640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0a500b4f70 0x7f0a500b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53638/0 (socket says 172.21.15.77:53638) 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a55051640 1 -- 172.21.15.77:0/320979286 learned_addr learned my addr 172.21.15.77:0/320979286 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a55852640 1 --2- 172.21.15.77:0/320979286 >> v2:172.21.15.158:3300/0 conn(0x7f0a50066d90 0x7f0a500ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a54850640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a5006c6d0 0x7f0a5015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367322698 0 0) 0x7f0a500cd450 con 0x7f0a500b4f70 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a38003520 con 0x7f0a500b4f70 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3145782635 0 0) 0x7f0a38003520 con 0x7f0a500b4f70 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a5006c6d0 msgr2=0x7f0a5015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a5006c6d0 0x7f0a5015ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 >> v2:172.21.15.158:3300/0 conn(0x7f0a50066d90 msgr2=0x7f0a500ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 --2- 172.21.15.77:0/320979286 >> v2:172.21.15.158:3300/0 conn(0x7f0a50066d90 0x7f0a500ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a500cd730 con 0x7f0a500b4f70 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f0a48001670 con 0x7f0a500b4f70 2024-03-06T07:30:56.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 >> v1:172.21.15.77:6789/0 conn(0x7f0a500b4f70 legacy=0x7f0a500b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a380039b0 0x7f0a38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a38004310 0x7f0a38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a38004e50 0x7f0a380052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5015e9f0 con 0x7f0a38004310 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a500cda10 con 0x7f0a38004e50 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3800bd40 con 0x7f0a380039b0 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a54850640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a38004310 0x7f0a38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a55051640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a380039b0 0x7f0a38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.473+0000 7f0a55852640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a38004e50 0x7f0a380052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a54850640 1 -- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a380039b0 msgr2=0x7f0a38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a54850640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a380039b0 0x7f0a38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a54850640 1 -- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a38004e50 msgr2=0x7f0a380052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a54850640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a38004e50 0x7f0a380052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a54850640 1 -- 172.21.15.77:0/320979286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38003520 con 0x7f0a38004310 2024-03-06T07:30:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a54850640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a38004310 0x7f0a38004800 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009fc0 tx=0x7f0a38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1248e8f69886d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44039070 con 0x7f0a38004310 2024-03-06T07:30:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4402cc20 con 0x7f0a38004310 2024-03-06T07:30:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.474+0000 7f0a4ffff640 1 -- 172.21.15.77:0/320979286 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44009c10 con 0x7f0a38004310 2024-03-06T07:30:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.475+0000 7f0a572b5640 1 -- 172.21.15.77:0/320979286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a38004310 msgr2=0x7f0a38004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.475+0000 7f0a572b5640 1 --2- 172.21.15.77:0/320979286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a38004310 0x7f0a38004800 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009fc0 tx=0x7f0a38005830 comp rx=0 tx=0).stop 2024-03-06T07:30:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.475+0000 7f0a55852640 1 -- 172.21.15.77:0/320979286 reap_dead start 2024-03-06T07:30:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.475+0000 7f0a572b5640 1 -- 172.21.15.77:0/320979286 shutdown_connections 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.475+0000 7f0a572b5640 1 -- 172.21.15.77:0/320979286 >> 172.21.15.77:0/320979286 conn(0x7f0a500724d0 msgr2=0x7f0a500728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.475+0000 7f0a572b5640 1 -- 172.21.15.77:0/320979286 shutdown_connections 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.475+0000 7f0a572b5640 1 -- 172.21.15.77:0/320979286 wait complete. 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.477+0000 7f0a572b5640 1 Processor -- start 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.477+0000 7f0a572b5640 1 -- start start 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a572b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a38004310 0x7f0a50156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a572b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a501613f0 0x7f0a500ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a572b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a500cebb0 0x7f0a50157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a572b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a500cd210 con 0x7f0a501613f0 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a572b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a500cd090 con 0x7f0a500cebb0 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a572b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5015ee80 con 0x7f0a38004310 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a54850640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a501613f0 0x7f0a500ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a54850640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a501613f0 0x7f0a500ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56440/0 (socket says 172.21.15.77:56440) 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a54850640 1 -- 172.21.15.77:0/2277098028 learned_addr learned my addr 172.21.15.77:0/2277098028 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a54850640 1 -- 172.21.15.77:0/2277098028 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a38004310 msgr2=0x7f0a50156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a54850640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a38004310 0x7f0a50156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a54850640 1 -- 172.21.15.77:0/2277098028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a500cebb0 msgr2=0x7f0a50157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a54850640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a500cebb0 0x7f0a50157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.478+0000 7f0a54850640 1 -- 172.21.15.77:0/2277098028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50158c20 con 0x7f0a501613f0 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.479+0000 7f0a54850640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a501613f0 0x7f0a500ce670 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f0a4402c690 tx=0x7f0a44009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.479+0000 7f0a4dffb640 1 -- 172.21.15.77:0/2277098028 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44039050 con 0x7f0a501613f0 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.479+0000 7f0a4dffb640 1 -- 172.21.15.77:0/2277098028 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4403a040 con 0x7f0a501613f0 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.479+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a50160350 con 0x7f0a501613f0 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.479+0000 7f0a4dffb640 1 -- 172.21.15.77:0/2277098028 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44003bf0 con 0x7f0a501613f0 2024-03-06T07:30:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.479+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a50160910 con 0x7f0a501613f0 2024-03-06T07:30:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.480+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5004f330 con 0x7f0a501613f0 2024-03-06T07:30:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.480+0000 7f0a4dffb640 1 -- 172.21.15.77:0/2277098028 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a44004030 con 0x7f0a501613f0 2024-03-06T07:30:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.481+0000 7f0a4dffb640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0a3c06b4d0 0x7f0a3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.481+0000 7f0a4dffb640 1 -- 172.21.15.77:0/2277098028 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0a440acae0 con 0x7f0a501613f0 2024-03-06T07:30:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.481+0000 7f0a55051640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0a3c06b4d0 0x7f0a3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.481+0000 7f0a55051640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0a3c06b4d0 0x7f0a3c06d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0a48001eb0 tx=0x7f0a48000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.485+0000 7f0a4dffb640 1 -- 172.21.15.77:0/2277098028 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a4407b940 con 0x7f0a501613f0 2024-03-06T07:30:56.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.701+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a5006d670 con 0x7f0a501613f0 2024-03-06T07:30:56.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.701+0000 7f0a4dffb640 1 -- 172.21.15.77:0/2277098028 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f0a4407b240 con 0x7f0a501613f0 2024-03-06T07:30:56.703 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:56.703 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:30:56.704 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:30:56.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0a3c06b4d0 msgr2=0x7f0a3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0a3c06b4d0 0x7f0a3c06d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0a48001eb0 tx=0x7f0a48000ed0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a501613f0 msgr2=0x7f0a500ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a501613f0 0x7f0a500ce670 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f0a4402c690 tx=0x7f0a44009750 comp rx=0 tx=0).stop 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 shutdown_connections 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0a3c06b4d0 0x7f0a3c06d990 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a501613f0 0x7f0a500ce670 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0a500cebb0 0x7f0a50157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 --2- 172.21.15.77:0/2277098028 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0a38004310 0x7f0a50156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 >> 172.21.15.77:0/2277098028 conn(0x7f0a500724d0 msgr2=0x7f0a5006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 shutdown_connections 2024-03-06T07:30:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:56.703+0000 7f0a572b5640 1 -- 172.21.15.77:0/2277098028 wait complete. 2024-03-06T07:30:56.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:30:56.917 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:30:57.917 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:58.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.019+0000 7ff8e91c9640 1 Processor -- start 2024-03-06T07:30:58.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.019+0000 7ff8e91c9640 1 -- start start 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.020+0000 7ff8e91c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e40d22b0 0x7ff8e40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.020+0000 7ff8e91c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff8e40b41f0 0x7ff8e40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.020+0000 7ff8e91c9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40b3430 con 0x7ff8e415dfe0 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.020+0000 7ff8e91c9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e40d2c50 con 0x7ff8e40b41f0 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.020+0000 7ff8e91c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40b31e0 con 0x7ff8e40d22b0 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.020+0000 7ff8e359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff8e40b41f0 0x7ff8e40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.021+0000 7ff8e359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff8e40b41f0 0x7ff8e40b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56452/0 (socket says 172.21.15.77:56452) 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.021+0000 7ff8e359e640 1 -- 172.21.15.77:0/892915673 learned_addr learned my addr 172.21.15.77:0/892915673 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.021+0000 7ff8dbfff640 1 --2- 172.21.15.77:0/892915673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e40d22b0 0x7ff8e40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.021+0000 7ff8e359e640 1 -- 172.21.15.77:0/892915673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e40d22b0 msgr2=0x7ff8e40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.021+0000 7ff8e359e640 1 --2- 172.21.15.77:0/892915673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e40d22b0 0x7ff8e40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.021+0000 7ff8e359e640 1 -- 172.21.15.77:0/892915673 >> v1:172.21.15.158:6789/0 conn(0x7ff8e415dfe0 legacy=0x7ff8e40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.021+0000 7ff8e359e640 1 -- 172.21.15.77:0/892915673 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40b3710 con 0x7ff8e40b41f0 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.022+0000 7ff8e359e640 1 --2- 172.21.15.77:0/892915673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40b41f0 0x7ff8e40b4650 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009e50 tx=0x7ff8dc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f99d29c593e5f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.022+0000 7ff8e259c640 1 -- 172.21.15.77:0/892915673 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc02cb00 con 0x7ff8e40b41f0 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.022+0000 7ff8e259c640 1 -- 172.21.15.77:0/892915673 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc02cca0 con 0x7ff8e40b41f0 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.022+0000 7ff8e259c640 1 -- 172.21.15.77:0/892915673 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc034610 con 0x7ff8e40b41f0 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.023+0000 7ff8e91c9640 1 -- 172.21.15.77:0/892915673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40b41f0 msgr2=0x7ff8e40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.023+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/892915673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40b41f0 0x7ff8e40b4650 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009e50 tx=0x7ff8dc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.023+0000 7ff8e91c9640 1 -- 172.21.15.77:0/892915673 shutdown_connections 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.023+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/892915673 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e40d22b0 0x7ff8e40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.023+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/892915673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40b41f0 0x7ff8e40b4650 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.023+0000 7ff8e91c9640 1 -- 172.21.15.77:0/892915673 >> 172.21.15.77:0/892915673 conn(0x7ff8e40d3cd0 msgr2=0x7ff8e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.023+0000 7ff8e91c9640 1 -- 172.21.15.77:0/892915673 shutdown_connections 2024-03-06T07:30:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.023+0000 7ff8e91c9640 1 -- 172.21.15.77:0/892915673 wait complete. 2024-03-06T07:30:58.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.024+0000 7ff8e91c9640 1 Processor -- start 2024-03-06T07:30:58.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.024+0000 7ff8e91c9640 1 -- start start 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e91c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8e40b41f0 0x7ff8e415bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e91c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40d22b0 0x7ff8e406cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e91c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e415dfe0 0x7ff8e406d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e91c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40cfa00 con 0x7ff8e40d22b0 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e91c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40b3d60 con 0x7ff8e40b41f0 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e91c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40d1c70 con 0x7ff8e415dfe0 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40d22b0 0x7ff8e406cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40d22b0 0x7ff8e406cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56464/0 (socket says 172.21.15.77:56464) 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 -- 172.21.15.77:0/4053367945 learned_addr learned my addr 172.21.15.77:0/4053367945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e2d9d640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8e40b41f0 0x7ff8e415bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e359e640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e415dfe0 0x7ff8e406d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 -- 172.21.15.77:0/4053367945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e415dfe0 msgr2=0x7ff8e406d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e415dfe0 0x7ff8e406d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 -- 172.21.15.77:0/4053367945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8e40b41f0 msgr2=0x7ff8e415bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8e40b41f0 0x7ff8e415bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 -- 172.21.15.77:0/4053367945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e41604f0 con 0x7ff8e40d22b0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8dbfff640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40d22b0 0x7ff8e406cac0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00d800 tx=0x7ff8cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8db7fe640 1 -- 172.21.15.77:0/4053367945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc00b6c0 con 0x7ff8e40d22b0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8db7fe640 1 -- 172.21.15.77:0/4053367945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc019070 con 0x7ff8e40d22b0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.025+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e41607d0 con 0x7ff8e40d22b0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.026+0000 7ff8db7fe640 1 -- 172.21.15.77:0/4053367945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc00ba80 con 0x7ff8e40d22b0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.026+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e41612b0 con 0x7ff8e40d22b0 2024-03-06T07:30:58.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.026+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e404f330 con 0x7ff8e40d22b0 2024-03-06T07:30:58.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.027+0000 7ff8db7fe640 1 -- 172.21.15.77:0/4053367945 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8cc00bc20 con 0x7ff8e40d22b0 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.027+0000 7ff8db7fe640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8c406b500 0x7ff8c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.027+0000 7ff8db7fe640 1 -- 172.21.15.77:0/4053367945 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff8cc08a5f0 con 0x7ff8e40d22b0 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.027+0000 7ff8e2d9d640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8c406b500 0x7ff8c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.028+0000 7ff8e2d9d640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8c406b500 0x7ff8c406d9c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4001640 tx=0x7ff8d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.036+0000 7ff8db7fe640 1 -- 172.21.15.77:0/4053367945 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8cc01c360 con 0x7ff8e40d22b0 2024-03-06T07:30:58.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.251+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8e40d0b20 con 0x7ff8e40d22b0 2024-03-06T07:30:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.251+0000 7ff8db7fe640 1 -- 172.21.15.77:0/4053367945 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7ff8cc0594e0 con 0x7ff8e40d22b0 2024-03-06T07:30:58.253 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:58.253 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:30:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:30:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8c406b500 msgr2=0x7ff8c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8c406b500 0x7ff8c406d9c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4001640 tx=0x7ff8d4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40d22b0 msgr2=0x7ff8e406cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40d22b0 0x7ff8e406cac0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00d800 tx=0x7ff8cc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 shutdown_connections 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff8c406b500 0x7ff8c406d9c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8e40d22b0 0x7ff8e406cac0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff8e415dfe0 0x7ff8e406d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 --2- 172.21.15.77:0/4053367945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff8e40b41f0 0x7ff8e415bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 >> 172.21.15.77:0/4053367945 conn(0x7ff8e40d3cd0 msgr2=0x7ff8e40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 shutdown_connections 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:58.253+0000 7ff8e91c9640 1 -- 172.21.15.77:0/4053367945 wait complete. 2024-03-06T07:30:58.468 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:30:58.468 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:30:59.469 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:59.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.567+0000 7f0883ecf640 1 Processor -- start 2024-03-06T07:30:59.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.567+0000 7f0883ecf640 1 -- start start 2024-03-06T07:30:59.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.568+0000 7f0883ecf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f087c15dfe0 0x7f087c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.568+0000 7f0883ecf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0b3430 con 0x7f087c0cf830 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.568+0000 7f0883ecf640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0b3710 con 0x7f087c0d29c0 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.568+0000 7f0883ecf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c0b31e0 con 0x7f087c15dfe0 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.568+0000 7f0881c6b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f087c15dfe0 0x7f087c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.568+0000 7f088146a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f087c0cf830 0x7f087c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55950/0 (socket says 172.21.15.77:55950) 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.568+0000 7f088146a640 1 -- 172.21.15.77:0/823381841 learned_addr learned my addr 172.21.15.77:0/823381841 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.569+0000 7f0881c6b640 1 -- 172.21.15.77:0/823381841 >> v1:172.21.15.158:6789/0 conn(0x7f087c0d29c0 legacy=0x7f087c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.569+0000 7f0881c6b640 1 -- 172.21.15.77:0/823381841 >> v1:172.21.15.77:6789/0 conn(0x7f087c0cf830 legacy=0x7f087c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.569+0000 7f0881c6b640 1 -- 172.21.15.77:0/823381841 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0b39f0 con 0x7f087c15dfe0 2024-03-06T07:30:59.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.569+0000 7f0881c6b640 1 --2- 172.21.15.77:0/823381841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c15dfe0 0x7f087c0cf2f0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f087c0b3360 tx=0x7f087802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2561bf51d328abb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.569+0000 7f0880c69640 1 -- 172.21.15.77:0/823381841 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878035020 con 0x7f087c15dfe0 2024-03-06T07:30:59.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0880c69640 1 -- 172.21.15.77:0/823381841 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0878031830 con 0x7f087c15dfe0 2024-03-06T07:30:59.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0880c69640 1 -- 172.21.15.77:0/823381841 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878031c00 con 0x7f087c15dfe0 2024-03-06T07:30:59.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0883ecf640 1 -- 172.21.15.77:0/823381841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c15dfe0 msgr2=0x7f087c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0883ecf640 1 --2- 172.21.15.77:0/823381841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c15dfe0 0x7f087c0cf2f0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f087c0b3360 tx=0x7f087802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0883ecf640 1 -- 172.21.15.77:0/823381841 shutdown_connections 2024-03-06T07:30:59.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0883ecf640 1 --2- 172.21.15.77:0/823381841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c15dfe0 0x7f087c0cf2f0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0883ecf640 1 -- 172.21.15.77:0/823381841 >> 172.21.15.77:0/823381841 conn(0x7f087c0d3cd0 msgr2=0x7f087c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0883ecf640 1 -- 172.21.15.77:0/823381841 shutdown_connections 2024-03-06T07:30:59.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.570+0000 7f0883ecf640 1 -- 172.21.15.77:0/823381841 wait complete. 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0883ecf640 1 Processor -- start 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0883ecf640 1 -- start start 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0883ecf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf830 0x7f087c0b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0883ecf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f087c0d29c0 0x7f087c15e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0883ecf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c15dfe0 0x7f087c15c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0883ecf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c15e3c0 con 0x7f087c0cf830 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0883ecf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0b3d60 con 0x7f087c15dfe0 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0883ecf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c15e540 con 0x7f087c0d29c0 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0881c6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf830 0x7f087c0b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0881c6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf830 0x7f087c0b91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35240/0 (socket says 172.21.15.77:35240) 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.572+0000 7f0881c6b640 1 -- 172.21.15.77:0/2032992877 learned_addr learned my addr 172.21.15.77:0/2032992877 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f0881c6b640 1 -- 172.21.15.77:0/2032992877 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f087c0d29c0 msgr2=0x7f087c15e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f0881c6b640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f087c0d29c0 0x7f087c15e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f088246c640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c15dfe0 0x7f087c15c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f0881c6b640 1 -- 172.21.15.77:0/2032992877 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c15dfe0 msgr2=0x7f087c15c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f0881c6b640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c15dfe0 0x7f087c15c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f0881c6b640 1 -- 172.21.15.77:0/2032992877 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0b8400 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f0881c6b640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf830 0x7f087c0b91e0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f0878000c00 tx=0x7f08780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f08727fc640 1 -- 172.21.15.77:0/2032992877 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087803a9b0 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f08727fc640 1 -- 172.21.15.77:0/2032992877 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087803ab50 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c0b7880 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f08727fc640 1 -- 172.21.15.77:0/2032992877 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08780443f0 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.573+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c0b8120 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.574+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c04f330 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.575+0000 7f08727fc640 1 -- 172.21.15.77:0/2032992877 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f087804b060 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.575+0000 7f08727fc640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f086006b500 0x7f086006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.575+0000 7f08727fc640 1 -- 172.21.15.77:0/2032992877 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f08780abf80 con 0x7f087c0cf830 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.575+0000 7f088146a640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f086006b500 0x7f086006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.576+0000 7f088146a640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f086006b500 0x7f086006d9c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f086c000f60 tx=0x7f086c007a80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.579+0000 7f08727fc640 1 -- 172.21.15.77:0/2032992877 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f087803f070 con 0x7f087c0cf830 2024-03-06T07:30:59.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.787+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f087c0b7c10 con 0x7f087c0cf830 2024-03-06T07:30:59.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.788+0000 7f08727fc640 1 -- 172.21.15.77:0/2032992877 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f087802f2e0 con 0x7f087c0cf830 2024-03-06T07:30:59.790 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:30:59.790 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:30:59.790 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:30:59.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f086006b500 msgr2=0x7f086006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f086006b500 0x7f086006d9c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f086c000f60 tx=0x7f086c007a80 comp rx=0 tx=0).stop 2024-03-06T07:30:59.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf830 msgr2=0x7f087c0b91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf830 0x7f087c0b91e0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f0878000c00 tx=0x7f08780046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 shutdown_connections 2024-03-06T07:30:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f087c15dfe0 0x7f087c15c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f087c0d29c0 0x7f087c15e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f086006b500 0x7f086006d9c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 --2- 172.21.15.77:0/2032992877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0cf830 0x7f087c0b91e0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 >> 172.21.15.77:0/2032992877 conn(0x7f087c0d3cd0 msgr2=0x7f087c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 shutdown_connections 2024-03-06T07:30:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:30:59.790+0000 7f0883ecf640 1 -- 172.21.15.77:0/2032992877 wait complete. 2024-03-06T07:31:00.004 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:00.004 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:01.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.108+0000 7f8e11a5c640 1 Processor -- start 2024-03-06T07:31:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.108+0000 7f8e11a5c640 1 -- start start 2024-03-06T07:31:01.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e11a5c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8e0c0b4170 0x7f8e0c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e11a5c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0d25d0 con 0x7f8e0c0d4760 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e11a5c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0d2870 con 0x7f8e0c0d0a80 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e11a5c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0c0d4bf0 con 0x7f8e0c0b4170 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e0a7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8e0c0d4760 0x7f8e0c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55964/0 (socket says 172.21.15.77:55964) 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e0a7fc640 1 -- 172.21.15.77:0/1255395444 learned_addr learned my addr 172.21.15.77:0/1255395444 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e0affd640 1 --2- 172.21.15.77:0/1255395444 >> v2:172.21.15.158:3300/0 conn(0x7f8e0c0b4170 0x7f8e0c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893634951 0 0) 0x7f8e0c0d25d0 con 0x7f8e0c0d4760 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.109+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df4003520 con 0x7f8e0c0d4760 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3880515187 0 0) 0x7f8df4003520 con 0x7f8e0c0d4760 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 >> v2:172.21.15.158:3300/0 conn(0x7f8e0c0b4170 msgr2=0x7f8e0c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 --2- 172.21.15.77:0/1255395444 >> v2:172.21.15.158:3300/0 conn(0x7f8e0c0b4170 0x7f8e0c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 >> v1:172.21.15.158:6789/0 conn(0x7f8e0c0d0a80 legacy=0x7f8e0c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c0d2b50 con 0x7f8e0c0d4760 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f8e00001670 con 0x7f8e0c0d4760 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0affd640 1 --2- 172.21.15.77:0/1255395444 >> v2:172.21.15.158:3300/0 conn(0x7f8e0c0b4170 0x7f8e0c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:01.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 >> v1:172.21.15.77:6789/0 conn(0x7f8e0c0d4760 legacy=0x7f8e0c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8df40039b0 0x7f8df4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4004800 0x7f8df4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df40051c0 0x7f8df400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c0d4bf0 con 0x7f8df4004800 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df400c060 con 0x7f8df40051c0 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df400c260 con 0x7f8df40039b0 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0a7fc640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4004800 0x7f8df4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0affd640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8df40039b0 0x7f8df4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0b7fe640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df40051c0 0x7f8df400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0a7fc640 1 -- 172.21.15.77:0/1255395444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8df40039b0 msgr2=0x7f8df4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0a7fc640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8df40039b0 0x7f8df4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0a7fc640 1 -- 172.21.15.77:0/1255395444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df40051c0 msgr2=0x7f8df400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0a7fc640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df40051c0 0x7f8df400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0b7fe640 1 -- 172.21.15.77:0/1255395444 reap_dead start 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.110+0000 7f8e0a7fc640 1 -- 172.21.15.77:0/1255395444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df4003520 con 0x7f8df4004800 2024-03-06T07:31:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.111+0000 7f8e0a7fc640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4004800 0x7f8df4004c80 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000c210 tx=0x7f8df4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4959f35fbb8b89f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.111+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00007d30 con 0x7f8df4004800 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.111+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00030070 con 0x7f8df4004800 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.111+0000 7f8e09ffb640 1 -- 172.21.15.77:0/1255395444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0002cda0 con 0x7f8df4004800 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.112+0000 7f8e11a5c640 1 -- 172.21.15.77:0/1255395444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4004800 msgr2=0x7f8df4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.112+0000 7f8e11a5c640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4004800 0x7f8df4004c80 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000c210 tx=0x7f8df4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.112+0000 7f8e11a5c640 1 -- 172.21.15.77:0/1255395444 shutdown_connections 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.112+0000 7f8e11a5c640 1 --2- 172.21.15.77:0/1255395444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4004800 0x7f8df4004c80 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.112+0000 7f8e11a5c640 1 -- 172.21.15.77:0/1255395444 >> 172.21.15.77:0/1255395444 conn(0x7f8e0c0d3c40 msgr2=0x7f8e0c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.112+0000 7f8e11a5c640 1 -- 172.21.15.77:0/1255395444 shutdown_connections 2024-03-06T07:31:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.112+0000 7f8e11a5c640 1 -- 172.21.15.77:0/1255395444 wait complete. 2024-03-06T07:31:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.113+0000 7f8e11a5c640 1 Processor -- start 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.113+0000 7f8e11a5c640 1 -- start start 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e11a5c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df4004800 0x7f8e0c0cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e11a5c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e0c0c9d50 0x7f8e0c0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e11a5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e0c0cd9e0 0x7f8e0c0b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e11a5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c0b4dd0 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e11a5c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c0ce9e0 con 0x7f8df4004800 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e11a5c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c0b4f50 con 0x7f8e0c0c9d50 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e0c0cd9e0 0x7f8e0c0b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e0c0cd9e0 0x7f8e0c0b6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35254/0 (socket says 172.21.15.77:35254) 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 -- 172.21.15.77:0/3461135818 learned_addr learned my addr 172.21.15.77:0/3461135818 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0affd640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df4004800 0x7f8e0c0cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0a7fc640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e0c0c9d50 0x7f8e0c0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 -- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e0c0c9d50 msgr2=0x7f8e0c0cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e0c0c9d50 0x7f8e0c0cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 -- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df4004800 msgr2=0x7f8e0c0cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df4004800 0x7f8e0c0cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 -- 172.21.15.77:0/3461135818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c0c99a0 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0a7fc640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e0c0c9d50 0x7f8e0c0cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e0b7fe640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e0c0cd9e0 0x7f8e0c0b6860 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400f770 tx=0x7f8e0400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8df3fff640 1 -- 172.21.15.77:0/3461135818 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0401a070 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8df3fff640 1 -- 172.21.15.77:0/3461135818 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e04006d60 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0c0b9720 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8df3fff640 1 -- 172.21.15.77:0/3461135818 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04003bb0 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.114+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0c0c9560 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.115+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0c04f330 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.116+0000 7f8df3fff640 1 -- 172.21.15.77:0/3461135818 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e0401d070 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.116+0000 7f8df3fff640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8df806b4d0 0x7f8df806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.116+0000 7f8df3fff640 1 -- 172.21.15.77:0/3461135818 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8e04089570 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.116+0000 7f8e0affd640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8df806b4d0 0x7f8df806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.117+0000 7f8e0affd640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8df806b4d0 0x7f8df806d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009dc0 tx=0x7f8dfc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.125+0000 7f8df3fff640 1 -- 172.21.15.77:0/3461135818 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e04057e10 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.340+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e0c0733b0 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.341+0000 7f8df3fff640 1 -- 172.21.15.77:0/3461135818 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f8e0405beb0 con 0x7f8e0c0cd9e0 2024-03-06T07:31:01.343 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:01.343 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:01.343 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8df806b4d0 msgr2=0x7f8df806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8df806b4d0 0x7f8df806d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009dc0 tx=0x7f8dfc009190 comp rx=0 tx=0).stop 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e0c0cd9e0 msgr2=0x7f8e0c0b6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e0c0cd9e0 0x7f8e0c0b6860 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400f770 tx=0x7f8e0400fc30 comp rx=0 tx=0).stop 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 shutdown_connections 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8df806b4d0 0x7f8df806d990 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e0c0cd9e0 0x7f8e0c0b6860 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8e0c0c9d50 0x7f8e0c0cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 --2- 172.21.15.77:0/3461135818 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8df4004800 0x7f8e0c0cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.342+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 >> 172.21.15.77:0/3461135818 conn(0x7f8e0c0d3c40 msgr2=0x7f8e0c0b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.343+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 shutdown_connections 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:01.343+0000 7f8e11a5c640 1 -- 172.21.15.77:0/3461135818 wait complete. 2024-03-06T07:31:01.556 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:01.557 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:01.764 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:01.767 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:02.558 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:02.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.658+0000 7f5a2bb5e640 1 Processor -- start 2024-03-06T07:31:02.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.658+0000 7f5a2bb5e640 1 -- start start 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a2bb5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415dfe0 0x7f5a240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a2bb5e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a240b3430 con 0x7f5a240d29c0 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a2bb5e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a240b3710 con 0x7f5a240cf830 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a2bb5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240b31e0 con 0x7f5a2415dfe0 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a2a0fb640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5a240d29c0 0x7f5a240b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55966/0 (socket says 172.21.15.77:55966) 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a2a0fb640 1 -- 172.21.15.77:0/3947467823 learned_addr learned my addr 172.21.15.77:0/3947467823 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a298fa640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415dfe0 0x7f5a240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663080959 0 0) 0x7f5a240b3430 con 0x7f5a240d29c0 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.659+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a10003520 con 0x7f5a240d29c0 2024-03-06T07:31:02.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2228527293 0 0) 0x7f5a10003520 con 0x7f5a240d29c0 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415dfe0 msgr2=0x7f5a240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415dfe0 0x7f5a240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 >> v1:172.21.15.158:6789/0 conn(0x7f5a240cf830 legacy=0x7f5a240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240b39f0 con 0x7f5a240d29c0 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f5a1c0019f0 con 0x7f5a240d29c0 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 >> v1:172.21.15.77:6789/0 conn(0x7f5a240d29c0 legacy=0x7f5a240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415dfe0 0x7f5a100039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a10004d40 0x7f5a100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240b31e0 con 0x7f5a10003f20 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1000ba20 con 0x7f5a10004d40 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1000bc20 con 0x7f5a2415dfe0 2024-03-06T07:31:02.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a290f9640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a298fa640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415dfe0 0x7f5a100039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a2a0fb640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a10004d40 0x7f5a100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a290f9640 1 -- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415dfe0 msgr2=0x7f5a100039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a290f9640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415dfe0 0x7f5a100039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a290f9640 1 -- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a10004d40 msgr2=0x7f5a100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a290f9640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a10004d40 0x7f5a100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.660+0000 7f5a290f9640 1 -- 172.21.15.77:0/3947467823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a10003520 con 0x7f5a10003f20 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.661+0000 7f5a290f9640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a10004800 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f5a20001830 tx=0x7f5a10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11de4597e7dd39eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.661+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a20001430 con 0x7f5a10003f20 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.661+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a20032620 con 0x7f5a10003f20 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.661+0000 7f5a288f8640 1 -- 172.21.15.77:0/3947467823 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a20032a10 con 0x7f5a10003f20 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.662+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3947467823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 msgr2=0x7f5a10004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.662+0000 7f5a2bb5e640 1 --2- 172.21.15.77:0/3947467823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a10004800 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f5a20001830 tx=0x7f5a10005830 comp rx=0 tx=0).stop 2024-03-06T07:31:02.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.662+0000 7f5a2a0fb640 1 -- 172.21.15.77:0/3947467823 reap_dead start 2024-03-06T07:31:02.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.662+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3947467823 shutdown_connections 2024-03-06T07:31:02.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.662+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3947467823 >> 172.21.15.77:0/3947467823 conn(0x7f5a240d3cd0 msgr2=0x7f5a240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.662+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3947467823 shutdown_connections 2024-03-06T07:31:02.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.662+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3947467823 wait complete. 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.663+0000 7f5a2bb5e640 1 Processor -- start 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.663+0000 7f5a2bb5e640 1 -- start start 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2bb5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a240cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2bb5e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a24151010 0x7f5a2415b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2bb5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415b680 0x7f5a2415f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2bb5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2415e3c0 con 0x7f5a10003f20 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2bb5e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240b3d60 con 0x7f5a24151010 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2bb5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2415e540 con 0x7f5a2415b680 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a240cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a240cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35276/0 (socket says 172.21.15.77:35276) 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 -- 172.21.15.77:0/3739264565 learned_addr learned my addr 172.21.15.77:0/3739264565 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2a0fb640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415b680 0x7f5a2415f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a290f9640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a24151010 0x7f5a2415b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 -- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415b680 msgr2=0x7f5a2415f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415b680 0x7f5a2415f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 -- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a24151010 msgr2=0x7f5a2415b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a24151010 0x7f5a2415b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 -- 172.21.15.77:0/3739264565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240cb030 con 0x7f5a10003f20 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2a0fb640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415b680 0x7f5a2415f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a290f9640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a24151010 0x7f5a2415b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a298fa640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a240cd760 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f5a140119d0 tx=0x7f5a14011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a1a7fc640 1 -- 172.21.15.77:0/3739264565 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1400fb50 con 0x7f5a10003f20 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a1a7fc640 1 -- 172.21.15.77:0/3739264565 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1400fcf0 con 0x7f5a10003f20 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a240cb310 con 0x7f5a10003f20 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a1a7fc640 1 -- 172.21.15.77:0/3739264565 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1400fea0 con 0x7f5a10003f20 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.664+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a240cdca0 con 0x7f5a10003f20 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.665+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a240d37a0 con 0x7f5a10003f20 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.666+0000 7f5a1a7fc640 1 -- 172.21.15.77:0/3739264565 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a14012040 con 0x7f5a10003f20 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.666+0000 7f5a1a7fc640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a0806b500 0x7f5a0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.666+0000 7f5a1a7fc640 1 -- 172.21.15.77:0/3739264565 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5a1408a8e0 con 0x7f5a10003f20 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.666+0000 7f5a290f9640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a0806b500 0x7f5a0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.667+0000 7f5a290f9640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a0806b500 0x7f5a0806d9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5a20035000 tx=0x7f5a20030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.670+0000 7f5a1a7fc640 1 -- 172.21.15.77:0/3739264565 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a240d37a0 con 0x7f5a10003f20 2024-03-06T07:31:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.900+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a2404f330 con 0x7f5a10003f20 2024-03-06T07:31:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.901+0000 7f5a1a7fc640 1 -- 172.21.15.77:0/3739264565 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f5a14019030 con 0x7f5a10003f20 2024-03-06T07:31:02.903 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:02.903 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:02.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a0806b500 msgr2=0x7f5a0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a0806b500 0x7f5a0806d9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5a20035000 tx=0x7f5a20030000 comp rx=0 tx=0).stop 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 msgr2=0x7f5a240cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a240cd760 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f5a140119d0 tx=0x7f5a14011e90 comp rx=0 tx=0).stop 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 shutdown_connections 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5a2415b680 0x7f5a2415f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5a24151010 0x7f5a2415b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5a0806b500 0x7f5a0806d9c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 --2- 172.21.15.77:0/3739264565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a10003f20 0x7f5a240cd760 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.903+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 >> 172.21.15.77:0/3739264565 conn(0x7f5a240d3cd0 msgr2=0x7f5a2415cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.904+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 shutdown_connections 2024-03-06T07:31:02.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:02.904+0000 7f5a2bb5e640 1 -- 172.21.15.77:0/3739264565 wait complete. 2024-03-06T07:31:03.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:03.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:04.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:04.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.219+0000 7f1cebfe4640 1 Processor -- start 2024-03-06T07:31:04.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.219+0000 7f1cebfe4640 1 -- start start 2024-03-06T07:31:04.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.219+0000 7f1cebfe4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ce415dfe0 0x7f1ce40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.219+0000 7f1cebfe4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1ce40cf830 0x7f1ce40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1cebfe4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce40b3430 con 0x7f1ce40d27f0 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1cebfe4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce40b31e0 con 0x7f1ce415dfe0 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1cebfe4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce40b3d60 con 0x7f1ce40cf830 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce9d80640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ce415dfe0 0x7f1ce40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1cea581640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1ce40d27f0 0x7f1ce40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55974/0 (socket says 172.21.15.77:55974) 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce9d80640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ce415dfe0 0x7f1ce40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35286/0 (socket says 172.21.15.77:35286) 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1cea581640 1 -- 172.21.15.77:0/187166346 learned_addr learned my addr 172.21.15.77:0/187166346 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce957f640 1 --2- 172.21.15.77:0/187166346 >> v2:172.21.15.158:3300/0 conn(0x7f1ce40cf830 0x7f1ce40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce9d80640 1 -- 172.21.15.77:0/187166346 >> v2:172.21.15.158:3300/0 conn(0x7f1ce40cf830 msgr2=0x7f1ce40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce9d80640 1 --2- 172.21.15.77:0/187166346 >> v2:172.21.15.158:3300/0 conn(0x7f1ce40cf830 0x7f1ce40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce9d80640 1 -- 172.21.15.77:0/187166346 >> v1:172.21.15.77:6789/0 conn(0x7f1ce40d27f0 legacy=0x7f1ce40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce9d80640 1 -- 172.21.15.77:0/187166346 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce40b3710 con 0x7f1ce415dfe0 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce8d7e640 1 -- 172.21.15.77:0/187166346 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141042891 0 0) 0x7f1ce40b3430 con 0x7f1ce40d27f0 2024-03-06T07:31:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.220+0000 7f1ce957f640 1 --2- 172.21.15.77:0/187166346 >> v2:172.21.15.158:3300/0 conn(0x7f1ce40cf830 0x7f1ce40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:04.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1ce9d80640 1 --2- 172.21.15.77:0/187166346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce415dfe0 0x7f1ce40cf2f0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00097a0 tx=0x7f1ce002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e55a18a96947d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1ce8d7e640 1 -- 172.21.15.77:0/187166346 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0035020 con 0x7f1ce415dfe0 2024-03-06T07:31:04.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1ce8d7e640 1 -- 172.21.15.77:0/187166346 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce0031830 con 0x7f1ce415dfe0 2024-03-06T07:31:04.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1ce8d7e640 1 -- 172.21.15.77:0/187166346 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0031c00 con 0x7f1ce415dfe0 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1cebfe4640 1 -- 172.21.15.77:0/187166346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce415dfe0 msgr2=0x7f1ce40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/187166346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce415dfe0 0x7f1ce40cf2f0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00097a0 tx=0x7f1ce002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1cebfe4640 1 -- 172.21.15.77:0/187166346 shutdown_connections 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/187166346 >> v2:172.21.15.158:3300/0 conn(0x7f1ce40cf830 0x7f1ce40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/187166346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce415dfe0 0x7f1ce40cf2f0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.221+0000 7f1cebfe4640 1 -- 172.21.15.77:0/187166346 >> 172.21.15.77:0/187166346 conn(0x7f1ce40d3cd0 msgr2=0x7f1ce40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.222+0000 7f1cebfe4640 1 -- 172.21.15.77:0/187166346 shutdown_connections 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.222+0000 7f1cebfe4640 1 -- 172.21.15.77:0/187166346 wait complete. 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.223+0000 7f1cebfe4640 1 Processor -- start 2024-03-06T07:31:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.223+0000 7f1cebfe4640 1 -- start start 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cebfe4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ce40cf830 0x7f1ce40b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cebfe4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce40d27f0 0x7f1ce40b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cebfe4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce415dfe0 0x7f1ce40b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cebfe4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce40d1df0 con 0x7f1ce40d27f0 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cebfe4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce40d1c70 con 0x7f1ce415dfe0 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cebfe4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce415e3c0 con 0x7f1ce40cf830 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce40d27f0 0x7f1ce40b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce40d27f0 0x7f1ce40b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35298/0 (socket says 172.21.15.77:35298) 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 -- 172.21.15.77:0/897200949 learned_addr learned my addr 172.21.15.77:0/897200949 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cea581640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce415dfe0 0x7f1ce40b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce9d80640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ce40cf830 0x7f1ce40b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 -- 172.21.15.77:0/897200949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ce40cf830 msgr2=0x7f1ce40b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ce40cf830 0x7f1ce40b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 -- 172.21.15.77:0/897200949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce415dfe0 msgr2=0x7f1ce40b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce415dfe0 0x7f1ce40b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 -- 172.21.15.77:0/897200949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce40b8430 con 0x7f1ce40d27f0 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1ce957f640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce40d27f0 0x7f1ce40b4610 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400c7f0 tx=0x7f1cd400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cda7fc640 1 -- 172.21.15.77:0/897200949 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4015070 con 0x7f1ce40d27f0 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cda7fc640 1 -- 172.21.15.77:0/897200949 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd4007820 con 0x7f1ce40d27f0 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce40b8710 con 0x7f1ce40d27f0 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cda7fc640 1 -- 172.21.15.77:0/897200949 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4007bf0 con 0x7f1ce40d27f0 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.225+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce40b8cd0 con 0x7f1ce40d27f0 2024-03-06T07:31:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.226+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce404f330 con 0x7f1ce40d27f0 2024-03-06T07:31:04.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.227+0000 7f1cda7fc640 1 -- 172.21.15.77:0/897200949 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cd4018070 con 0x7f1ce40d27f0 2024-03-06T07:31:04.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.227+0000 7f1cda7fc640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1cc806b4d0 0x7f1cc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.227+0000 7f1cda7fc640 1 -- 172.21.15.77:0/897200949 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1cd408a340 con 0x7f1ce40d27f0 2024-03-06T07:31:04.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.227+0000 7f1ce9d80640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1cc806b4d0 0x7f1cc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.228+0000 7f1ce9d80640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1cc806b4d0 0x7f1cc806d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0036000 tx=0x7f1ce0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.231+0000 7f1cda7fc640 1 -- 172.21.15.77:0/897200949 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1cd40591a0 con 0x7f1ce40d27f0 2024-03-06T07:31:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.445+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ce406a1b0 con 0x7f1ce40d27f0 2024-03-06T07:31:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.445+0000 7f1cda7fc640 1 -- 172.21.15.77:0/897200949 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f1cd4058aa0 con 0x7f1ce40d27f0 2024-03-06T07:31:04.447 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:04.447 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:04.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1cc806b4d0 msgr2=0x7f1cc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1cc806b4d0 0x7f1cc806d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0036000 tx=0x7f1ce0008030 comp rx=0 tx=0).stop 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce40d27f0 msgr2=0x7f1ce40b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce40d27f0 0x7f1ce40b4610 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400c7f0 tx=0x7f1cd400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 shutdown_connections 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1cc806b4d0 0x7f1cc806d990 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce40d27f0 0x7f1ce40b4610 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1ce415dfe0 0x7f1ce40b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 --2- 172.21.15.77:0/897200949 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1ce40cf830 0x7f1ce40b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 >> 172.21.15.77:0/897200949 conn(0x7f1ce40d3cd0 msgr2=0x7f1ce40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 shutdown_connections 2024-03-06T07:31:04.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:04.447+0000 7f1cebfe4640 1 -- 172.21.15.77:0/897200949 wait complete. 2024-03-06T07:31:04.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:04.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:05.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.762+0000 7f1d66b3a640 1 Processor -- start 2024-03-06T07:31:05.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.762+0000 7f1d66b3a640 1 -- start start 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d66b3a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d6015dfe0 0x7f1d600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d66b3a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d600d29c0 0x7f1d600b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d66b3a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d600b3430 con 0x7f1d600cf830 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d66b3a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d600b31e0 con 0x7f1d600d29c0 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d66b3a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600b3d60 con 0x7f1d6015dfe0 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d650d7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d600d29c0 0x7f1d600b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d650d7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d600d29c0 0x7f1d600b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35306/0 (socket says 172.21.15.77:35306) 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d650d7640 1 -- 172.21.15.77:0/2936122128 learned_addr learned my addr 172.21.15.77:0/2936122128 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.763+0000 7f1d5ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1d600cf830 0x7f1d600d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55990/0 (socket says 172.21.15.77:55990) 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.764+0000 7f1d648d6640 1 --2- 172.21.15.77:0/2936122128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d6015dfe0 0x7f1d600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.764+0000 7f1d650d7640 1 -- 172.21.15.77:0/2936122128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d6015dfe0 msgr2=0x7f1d600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.764+0000 7f1d650d7640 1 --2- 172.21.15.77:0/2936122128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d6015dfe0 0x7f1d600cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.764+0000 7f1d650d7640 1 -- 172.21.15.77:0/2936122128 >> v1:172.21.15.77:6789/0 conn(0x7f1d600cf830 legacy=0x7f1d600d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.764+0000 7f1d5f7fe640 1 -- 172.21.15.77:0/2936122128 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058209255 0 0) 0x7f1d600b3430 con 0x7f1d600cf830 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.764+0000 7f1d650d7640 1 -- 172.21.15.77:0/2936122128 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d600b3710 con 0x7f1d600d29c0 2024-03-06T07:31:05.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.764+0000 7f1d650d7640 1 --2- 172.21.15.77:0/2936122128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d600d29c0 0x7f1d600b41f0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009e50 tx=0x7f1d5802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a685638e64bfaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d5f7fe640 1 -- 172.21.15.77:0/2936122128 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5802cb00 con 0x7f1d600d29c0 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d5f7fe640 1 -- 172.21.15.77:0/2936122128 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5802cca0 con 0x7f1d600d29c0 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d5f7fe640 1 -- 172.21.15.77:0/2936122128 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58034610 con 0x7f1d600d29c0 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d66b3a640 1 -- 172.21.15.77:0/2936122128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d600d29c0 msgr2=0x7f1d600b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/2936122128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d600d29c0 0x7f1d600b41f0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009e50 tx=0x7f1d5802bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d66b3a640 1 -- 172.21.15.77:0/2936122128 shutdown_connections 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/2936122128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d600d29c0 0x7f1d600b41f0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/2936122128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d6015dfe0 0x7f1d600cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d66b3a640 1 -- 172.21.15.77:0/2936122128 >> 172.21.15.77:0/2936122128 conn(0x7f1d600d3cd0 msgr2=0x7f1d600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d66b3a640 1 -- 172.21.15.77:0/2936122128 shutdown_connections 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.765+0000 7f1d66b3a640 1 -- 172.21.15.77:0/2936122128 wait complete. 2024-03-06T07:31:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.766+0000 7f1d66b3a640 1 Processor -- start 2024-03-06T07:31:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.767+0000 7f1d66b3a640 1 -- start start 2024-03-06T07:31:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.767+0000 7f1d66b3a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d600cf830 0x7f1d600b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.767+0000 7f1d66b3a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d600d29c0 0x7f1d600b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.767+0000 7f1d66b3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d6015dfe0 0x7f1d600c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.767+0000 7f1d66b3a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600d1df0 con 0x7f1d6015dfe0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.767+0000 7f1d66b3a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600d1c70 con 0x7f1d600d29c0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.767+0000 7f1d66b3a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6015e3c0 con 0x7f1d600cf830 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d6015dfe0 0x7f1d600c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d6015dfe0 0x7f1d600c9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35320/0 (socket says 172.21.15.77:35320) 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 -- 172.21.15.77:0/4193889708 learned_addr learned my addr 172.21.15.77:0/4193889708 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d5ffff640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d600d29c0 0x7f1d600b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d648d6640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d600cf830 0x7f1d600b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 -- 172.21.15.77:0/4193889708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d600cf830 msgr2=0x7f1d600b9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d600cf830 0x7f1d600b9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 -- 172.21.15.77:0/4193889708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d600d29c0 msgr2=0x7f1d600b8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d600d29c0 0x7f1d600b8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 -- 172.21.15.77:0/4193889708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d600b7380 con 0x7f1d6015dfe0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d650d7640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d6015dfe0 0x7f1d600c9f50 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f1d58036040 tx=0x7f1d58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d5d7fa640 1 -- 172.21.15.77:0/4193889708 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58043020 con 0x7f1d6015dfe0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d5d7fa640 1 -- 172.21.15.77:0/4193889708 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5802ce50 con 0x7f1d6015dfe0 2024-03-06T07:31:05.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d600b75f0 con 0x7f1d6015dfe0 2024-03-06T07:31:05.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d5d7fa640 1 -- 172.21.15.77:0/4193889708 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58009510 con 0x7f1d6015dfe0 2024-03-06T07:31:05.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.768+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d600b7bb0 con 0x7f1d6015dfe0 2024-03-06T07:31:05.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.769+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6004f330 con 0x7f1d6015dfe0 2024-03-06T07:31:05.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.770+0000 7f1d5d7fa640 1 -- 172.21.15.77:0/4193889708 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d58005d10 con 0x7f1d6015dfe0 2024-03-06T07:31:05.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.770+0000 7f1d5d7fa640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d4406b4d0 0x7f1d4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.770+0000 7f1d5d7fa640 1 -- 172.21.15.77:0/4193889708 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1d580acb80 con 0x7f1d6015dfe0 2024-03-06T07:31:05.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.770+0000 7f1d648d6640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d4406b4d0 0x7f1d4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.770+0000 7f1d648d6640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d4406b4d0 0x7f1d4406d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1d54006fd0 tx=0x7f1d54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.779+0000 7f1d5d7fa640 1 -- 172.21.15.77:0/4193889708 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d5807b9e0 con 0x7f1d6015dfe0 2024-03-06T07:31:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.994+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d600b8fe0 con 0x7f1d6015dfe0 2024-03-06T07:31:05.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.994+0000 7f1d5d7fa640 1 -- 172.21.15.77:0/4193889708 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f1d5807b2e0 con 0x7f1d6015dfe0 2024-03-06T07:31:05.996 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:05.997 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:05.997 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.996+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d4406b4d0 msgr2=0x7f1d4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.996+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d4406b4d0 0x7f1d4406d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1d54006fd0 tx=0x7f1d54008040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.996+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d6015dfe0 msgr2=0x7f1d600c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.996+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d6015dfe0 0x7f1d600c9f50 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f1d58036040 tx=0x7f1d58004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.997+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 shutdown_connections 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.997+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1d4406b4d0 0x7f1d4406d990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.997+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d6015dfe0 0x7f1d600c9f50 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.997+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1d600d29c0 0x7f1d600b8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.997+0000 7f1d66b3a640 1 --2- 172.21.15.77:0/4193889708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1d600cf830 0x7f1d600b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.997+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 >> 172.21.15.77:0/4193889708 conn(0x7f1d600d3cd0 msgr2=0x7f1d6006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.997+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 shutdown_connections 2024-03-06T07:31:05.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:05.997+0000 7f1d66b3a640 1 -- 172.21.15.77:0/4193889708 wait complete. 2024-03-06T07:31:06.214 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:06.214 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:07.215 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:07.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.315+0000 7fdf08dc9640 1 Processor -- start 2024-03-06T07:31:07.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.315+0000 7fdf08dc9640 1 -- start start 2024-03-06T07:31:07.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf08dc9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdf040d22b0 0x7fdf040d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf08dc9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040b3430 con 0x7fdf0415dfe0 2024-03-06T07:31:07.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf08dc9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040b3710 con 0x7fdf040b41f0 2024-03-06T07:31:07.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf08dc9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf040d2c50 con 0x7fdf040d22b0 2024-03-06T07:31:07.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf0259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdf0415dfe0 0x7fdf040cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:56006/0 (socket says 172.21.15.77:56006) 2024-03-06T07:31:07.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf0259c640 1 -- 172.21.15.77:0/4060996007 learned_addr learned my addr 172.21.15.77:0/4060996007 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:07.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdef9d9b640 1 --2- 172.21.15.77:0/4060996007 >> v2:172.21.15.158:3300/0 conn(0x7fdf040d22b0 0x7fdf040d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798826377 0 0) 0x7fdf040b3430 con 0x7fdf0415dfe0 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef0003520 con 0x7fdf0415dfe0 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.316+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2493403356 0 0) 0x7fdef0003520 con 0x7fdf0415dfe0 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 >> v2:172.21.15.158:3300/0 conn(0x7fdf040d22b0 msgr2=0x7fdf040d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 --2- 172.21.15.77:0/4060996007 >> v2:172.21.15.158:3300/0 conn(0x7fdf040d22b0 0x7fdf040d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 >> v1:172.21.15.158:6789/0 conn(0x7fdf040b41f0 legacy=0x7fdf040b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040b39f0 con 0x7fdf0415dfe0 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fdef4001670 con 0x7fdf0415dfe0 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 >> v1:172.21.15.77:6789/0 conn(0x7fdf0415dfe0 legacy=0x7fdf040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef00039b0 0x7fdef0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef0004800 0x7fdef0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef00051c0 0x7fdef000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000bfe0 con 0x7fdef00039b0 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000c190 con 0x7fdef00051c0 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000c390 con 0x7fdef0004800 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf0259c640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef00039b0 0x7fdef0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf02d9d640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef00051c0 0x7fdef000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdef9d9b640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef0004800 0x7fdef0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf0259c640 1 -- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef0004800 msgr2=0x7fdef0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf0259c640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef0004800 0x7fdef0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf0259c640 1 -- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef00051c0 msgr2=0x7fdef000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf0259c640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdef00051c0 0x7fdef000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf02d9d640 1 -- 172.21.15.77:0/4060996007 reap_dead start 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdf0259c640 1 -- 172.21.15.77:0/4060996007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040b39f0 con 0x7fdef00039b0 2024-03-06T07:31:07.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.317+0000 7fdef9d9b640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef0004800 0x7fdef0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf0259c640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef00039b0 0x7fdef0003dd0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fdef400c340 tx=0x7fdef0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee37863745e77d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4007dd0 con 0x7fdef00039b0 2024-03-06T07:31:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4030070 con 0x7fdef00039b0 2024-03-06T07:31:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf01d9b640 1 -- 172.21.15.77:0/4060996007 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef402ce20 con 0x7fdef00039b0 2024-03-06T07:31:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf08dc9640 1 -- 172.21.15.77:0/4060996007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef00039b0 msgr2=0x7fdef0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf08dc9640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef00039b0 0x7fdef0003dd0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fdef400c340 tx=0x7fdef0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:07.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf08dc9640 1 -- 172.21.15.77:0/4060996007 shutdown_connections 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf08dc9640 1 --2- 172.21.15.77:0/4060996007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdef00039b0 0x7fdef0003dd0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.318+0000 7fdf08dc9640 1 -- 172.21.15.77:0/4060996007 >> 172.21.15.77:0/4060996007 conn(0x7fdf040d3cd0 msgr2=0x7fdf040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.319+0000 7fdf08dc9640 1 -- 172.21.15.77:0/4060996007 shutdown_connections 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.319+0000 7fdf08dc9640 1 -- 172.21.15.77:0/4060996007 wait complete. 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.320+0000 7fdf08dc9640 1 Processor -- start 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.320+0000 7fdf08dc9640 1 -- start start 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf08dc9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef00039b0 0x7fdf0415f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf08dc9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf040b7260 0x7fdf04161250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf08dc9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf041f4290 0x7fdf040b8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf08dc9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040b3d60 con 0x7fdf041f4290 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf08dc9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040b31e0 con 0x7fdf040b7260 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf08dc9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415e3c0 con 0x7fdef00039b0 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf041f4290 0x7fdf040b8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf041f4290 0x7fdf040b8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35324/0 (socket says 172.21.15.77:35324) 2024-03-06T07:31:07.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 -- 172.21.15.77:0/2699678659 learned_addr learned my addr 172.21.15.77:0/2699678659 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf0259c640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef00039b0 0x7fdf0415f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdef9d9b640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf040b7260 0x7fdf04161250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 -- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef00039b0 msgr2=0x7fdf0415f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef00039b0 0x7fdf0415f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 -- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf040b7260 msgr2=0x7fdf04161250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf040b7260 0x7fdf04161250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 -- 172.21.15.77:0/2699678659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040b6c00 con 0x7fdf041f4290 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdef9d9b640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf040b7260 0x7fdf04161250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf02d9d640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf041f4290 0x7fdf040b8ee0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fdf040cfd40 tx=0x7fdefc00fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdefb7fe640 1 -- 172.21.15.77:0/2699678659 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc003b80 con 0x7fdf041f4290 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdefb7fe640 1 -- 172.21.15.77:0/2699678659 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc018070 con 0x7fdf041f4290 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdefb7fe640 1 -- 172.21.15.77:0/2699678659 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0050d0 con 0x7fdf041f4290 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.321+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf041f4710 con 0x7fdf041f4290 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.322+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf040cc0a0 con 0x7fdf041f4290 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.323+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0404f330 con 0x7fdf041f4290 2024-03-06T07:31:07.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.323+0000 7fdefb7fe640 1 -- 172.21.15.77:0/2699678659 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdefc005270 con 0x7fdf041f4290 2024-03-06T07:31:07.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.323+0000 7fdefb7fe640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdee406b500 0x7fdee406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.323+0000 7fdefb7fe640 1 -- 172.21.15.77:0/2699678659 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdefc08a0e0 con 0x7fdf041f4290 2024-03-06T07:31:07.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.323+0000 7fdf0259c640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdee406b500 0x7fdee406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.324+0000 7fdf0259c640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdee406b500 0x7fdee406d9c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fdef4034000 tx=0x7fdef4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.328+0000 7fdefb7fe640 1 -- 172.21.15.77:0/2699678659 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf040b6c00 con 0x7fdf041f4290 2024-03-06T07:31:07.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.544+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf041e6f90 con 0x7fdf041f4290 2024-03-06T07:31:07.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.544+0000 7fdefb7fe640 1 -- 172.21.15.77:0/2699678659 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7fdefc058ec0 con 0x7fdf041f4290 2024-03-06T07:31:07.547 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:07.547 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:07.547 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdee406b500 msgr2=0x7fdee406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdee406b500 0x7fdee406d9c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fdef4034000 tx=0x7fdef4009030 comp rx=0 tx=0).stop 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf041f4290 msgr2=0x7fdf040b8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf041f4290 0x7fdf040b8ee0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fdf040cfd40 tx=0x7fdefc00fc70 comp rx=0 tx=0).stop 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 shutdown_connections 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdee406b500 0x7fdee406d9c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf041f4290 0x7fdf040b8ee0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdf040b7260 0x7fdf04161250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 --2- 172.21.15.77:0/2699678659 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdef00039b0 0x7fdf0415f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 >> 172.21.15.77:0/2699678659 conn(0x7fdf040d3cd0 msgr2=0x7fdf0415d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 shutdown_connections 2024-03-06T07:31:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:07.547+0000 7fdf08dc9640 1 -- 172.21.15.77:0/2699678659 wait complete. 2024-03-06T07:31:07.762 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:07.762 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:08.763 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:08.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.863+0000 7f85bfac3640 1 Processor -- start 2024-03-06T07:31:08.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.863+0000 7f85bfac3640 1 -- start start 2024-03-06T07:31:08.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bfac3640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f85b815dfe0 0x7f85b80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bfac3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b80d29c0 0x7f85b80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bfac3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b80b3430 con 0x7f85b80cf830 2024-03-06T07:31:08.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bfac3640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b80b31e0 con 0x7f85b815dfe0 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bfac3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80b3d60 con 0x7f85b80d29c0 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bd05e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f85b80cf830 0x7f85b80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44054/0 (socket says 172.21.15.77:44054) 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bd05e640 1 -- 172.21.15.77:0/2685707976 learned_addr learned my addr 172.21.15.77:0/2685707976 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85be060640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b80d29c0 0x7f85b80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bd85f640 1 --2- 172.21.15.77:0/2685707976 >> v2:172.21.15.158:3300/0 conn(0x7f85b815dfe0 0x7f85b80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635712186 0 0) 0x7f85b80b3430 con 0x7f85b80cf830 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.864+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a4003520 con 0x7f85b80cf830 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2073376358 0 0) 0x7f85a4003520 con 0x7f85b80cf830 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b80d29c0 msgr2=0x7f85b80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b80d29c0 0x7f85b80b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 >> v2:172.21.15.158:3300/0 conn(0x7f85b815dfe0 msgr2=0x7f85b80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 --2- 172.21.15.77:0/2685707976 >> v2:172.21.15.158:3300/0 conn(0x7f85b815dfe0 0x7f85b80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80b3710 con 0x7f85b80cf830 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f85a8001670 con 0x7f85b80cf830 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 >> v1:172.21.15.77:6789/0 conn(0x7f85b80cf830 legacy=0x7f85b80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b80d29c0 0x7f85a40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a4003f30 0x7f85a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a4004d40 0x7f85a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80b31e0 con 0x7f85a4004d40 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80b3d60 con 0x7f85a4003f30 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a400b9c0 con 0x7f85b80d29c0 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85be060640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a4004d40 0x7f85a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bd85f640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b80d29c0 0x7f85a40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85bd05e640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a4003f30 0x7f85a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85be060640 1 -- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b80d29c0 msgr2=0x7f85a40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85be060640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b80d29c0 0x7f85a40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85be060640 1 -- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a4003f30 msgr2=0x7f85a4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85be060640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85a4003f30 0x7f85a4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.865+0000 7f85be060640 1 -- 172.21.15.77:0/2685707976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a4003520 con 0x7f85a4004d40 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.866+0000 7f85be060640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a4004d40 0x7f85a40051c0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009fc0 tx=0x7f85a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940046916c333dea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.866+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0039070 con 0x7f85a4004d40 2024-03-06T07:31:08.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.866+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b002cc00 con 0x7f85a4004d40 2024-03-06T07:31:08.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.866+0000 7f85bc85d640 1 -- 172.21.15.77:0/2685707976 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0009c10 con 0x7f85a4004d40 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.867+0000 7f85bfac3640 1 -- 172.21.15.77:0/2685707976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a4004d40 msgr2=0x7f85a40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.867+0000 7f85bfac3640 1 --2- 172.21.15.77:0/2685707976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85a4004d40 0x7f85a40051c0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009fc0 tx=0x7f85a4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.867+0000 7f85be060640 1 -- 172.21.15.77:0/2685707976 reap_dead start 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.867+0000 7f85bfac3640 1 -- 172.21.15.77:0/2685707976 shutdown_connections 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.867+0000 7f85bfac3640 1 -- 172.21.15.77:0/2685707976 >> 172.21.15.77:0/2685707976 conn(0x7f85b80d3cd0 msgr2=0x7f85b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.867+0000 7f85bfac3640 1 -- 172.21.15.77:0/2685707976 shutdown_connections 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.867+0000 7f85bfac3640 1 -- 172.21.15.77:0/2685707976 wait complete. 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.868+0000 7f85bfac3640 1 Processor -- start 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.868+0000 7f85bfac3640 1 -- start start 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85bfac3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b8156d50 0x7f85b81f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85bfac3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b8157130 0x7f85b80a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85bfac3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b806c4a0 0x7f85b806c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85bfac3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80d1df0 con 0x7f85b806c4a0 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85bfac3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80d1c70 con 0x7f85b8157130 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85bfac3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b815e3c0 con 0x7f85b8156d50 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85be060640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b806c4a0 0x7f85b806c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85be060640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b806c4a0 0x7f85b806c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47538/0 (socket says 172.21.15.77:47538) 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85be060640 1 -- 172.21.15.77:0/2610463999 learned_addr learned my addr 172.21.15.77:0/2610463999 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85bd05e640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b8157130 0x7f85b80a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85bd85f640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b8156d50 0x7f85b81f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85be060640 1 -- 172.21.15.77:0/2610463999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b8156d50 msgr2=0x7f85b81f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85be060640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b8156d50 0x7f85b81f4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85be060640 1 -- 172.21.15.77:0/2610463999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b8157130 msgr2=0x7f85b80a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85be060640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b8157130 0x7f85b80a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.869+0000 7f85be060640 1 -- 172.21.15.77:0/2610463999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80a1b40 con 0x7f85b806c4a0 2024-03-06T07:31:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.870+0000 7f85be060640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b806c4a0 0x7f85b806c930 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f85b002c6d0 tx=0x7f85b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.870+0000 7f85ae7fc640 1 -- 172.21.15.77:0/2610463999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0039070 con 0x7f85b806c4a0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.870+0000 7f85ae7fc640 1 -- 172.21.15.77:0/2610463999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b003a040 con 0x7f85b806c4a0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.870+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b8157b70 con 0x7f85b806c4a0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.870+0000 7f85ae7fc640 1 -- 172.21.15.77:0/2610463999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0009c10 con 0x7f85b806c4a0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.870+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b815e860 con 0x7f85b806c4a0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.871+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b804f330 con 0x7f85b806c4a0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.871+0000 7f85ae7fc640 1 -- 172.21.15.77:0/2610463999 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85b0046070 con 0x7f85b806c4a0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.872+0000 7f85ae7fc640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f859c06b4d0 0x7f859c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.872+0000 7f85bd85f640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f859c06b4d0 0x7f859c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.872+0000 7f85ae7fc640 1 -- 172.21.15.77:0/2610463999 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f85b00abda0 con 0x7f85b806c4a0 2024-03-06T07:31:08.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.872+0000 7f85bd85f640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f859c06b4d0 0x7f859c06d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009b80 tx=0x7f85b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:08.876+0000 7f85ae7fc640 1 -- 172.21.15.77:0/2610463999 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85b8157b70 con 0x7f85b806c4a0 2024-03-06T07:31:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.089+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85b80b1aa0 con 0x7f85b806c4a0 2024-03-06T07:31:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.090+0000 7f85ae7fc640 1 -- 172.21.15.77:0/2610463999 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f85b007ac00 con 0x7f85b806c4a0 2024-03-06T07:31:09.092 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:09.092 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f859c06b4d0 msgr2=0x7f859c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f859c06b4d0 0x7f859c06d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009b80 tx=0x7f85b4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b806c4a0 msgr2=0x7f85b806c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b806c4a0 0x7f85b806c930 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f85b002c6d0 tx=0x7f85b0009750 comp rx=0 tx=0).stop 2024-03-06T07:31:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 shutdown_connections 2024-03-06T07:31:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f859c06b4d0 0x7f859c06d990 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85b806c4a0 0x7f85b806c930 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f85b8157130 0x7f85b80a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 --2- 172.21.15.77:0/2610463999 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f85b8156d50 0x7f85b81f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 >> 172.21.15.77:0/2610463999 conn(0x7f85b80d3cd0 msgr2=0x7f85b806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 shutdown_connections 2024-03-06T07:31:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:09.092+0000 7f85bfac3640 1 -- 172.21.15.77:0/2610463999 wait complete. 2024-03-06T07:31:09.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:09.306 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:10.307 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:10.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.407+0000 7ff069039640 1 Processor -- start 2024-03-06T07:31:10.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.407+0000 7ff069039640 1 -- start start 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff069039640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff06415dfe0 0x7ff0640cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff069039640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0640b3430 con 0x7ff0640cf830 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff069039640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0640b3710 con 0x7ff0640d29c0 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff069039640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0640b31e0 con 0x7ff06415dfe0 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff06259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff0640cf830 0x7ff0640d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44058/0 (socket says 172.21.15.77:44058) 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff06259c640 1 -- 172.21.15.77:0/3019853454 learned_addr learned my addr 172.21.15.77:0/3019853454 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff062d9d640 1 --2- 172.21.15.77:0/3019853454 >> v2:172.21.15.158:3300/0 conn(0x7ff06415dfe0 0x7ff0640cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661064626 0 0) 0x7ff0640b3430 con 0x7ff0640cf830 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff04c003520 con 0x7ff0640cf830 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 687051078 0 0) 0x7ff04c003520 con 0x7ff0640cf830 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 >> v2:172.21.15.158:3300/0 conn(0x7ff06415dfe0 msgr2=0x7ff0640cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff061d9b640 1 --2- 172.21.15.77:0/3019853454 >> v2:172.21.15.158:3300/0 conn(0x7ff06415dfe0 0x7ff0640cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 >> v1:172.21.15.158:6789/0 conn(0x7ff0640d29c0 legacy=0x7ff0640b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640b39f0 con 0x7ff0640cf830 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.408+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ff058001670 con 0x7ff0640cf830 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 >> v1:172.21.15.77:6789/0 conn(0x7ff0640cf830 legacy=0x7ff0640d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff061d9b640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c003a00 0x7ff04c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff061d9b640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff04c004800 0x7ff04c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff061d9b640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff04c0051c0 0x7ff04c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640b31e0 con 0x7ff04c004800 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c00c0f0 con 0x7ff04c003a00 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c00c2f0 con 0x7ff04c0051c0 2024-03-06T07:31:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff06259c640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff04c004800 0x7ff04c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff062d9d640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c003a00 0x7ff04c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff06359e640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff04c0051c0 0x7ff04c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff06259c640 1 -- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff04c0051c0 msgr2=0x7ff04c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff06259c640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff04c0051c0 0x7ff04c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff06259c640 1 -- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c003a00 msgr2=0x7ff04c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff06259c640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c003a00 0x7ff04c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff06359e640 1 -- 172.21.15.77:0/3019853454 reap_dead start 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff06259c640 1 -- 172.21.15.77:0/3019853454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c003520 con 0x7ff04c004800 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.409+0000 7ff062d9d640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c003a00 0x7ff04c003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff06259c640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff04c004800 0x7ff04c004c80 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7ff05800c0e0 tx=0x7ff04c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7239bd2a911b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058007c40 con 0x7ff04c004800 2024-03-06T07:31:10.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff058030070 con 0x7ff04c004800 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff061d9b640 1 -- 172.21.15.77:0/3019853454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058005d70 con 0x7ff04c004800 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff069039640 1 -- 172.21.15.77:0/3019853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff04c004800 msgr2=0x7ff04c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff069039640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff04c004800 0x7ff04c004c80 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7ff05800c0e0 tx=0x7ff04c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff069039640 1 -- 172.21.15.77:0/3019853454 shutdown_connections 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff069039640 1 --2- 172.21.15.77:0/3019853454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff04c004800 0x7ff04c004c80 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff069039640 1 -- 172.21.15.77:0/3019853454 >> 172.21.15.77:0/3019853454 conn(0x7ff0640d3cd0 msgr2=0x7ff0640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff069039640 1 -- 172.21.15.77:0/3019853454 shutdown_connections 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.410+0000 7ff069039640 1 -- 172.21.15.77:0/3019853454 wait complete. 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.412+0000 7ff069039640 1 Processor -- start 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.412+0000 7ff069039640 1 -- start start 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff069039640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c004800 0x7ff064150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff069039640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff064152f20 0x7ff0640b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff069039640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0640cd300 0x7ff0640b8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff069039640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06415e3c0 con 0x7ff0640cd300 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff069039640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640b3d60 con 0x7ff04c004800 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff069039640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06415e540 con 0x7ff064152f20 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0640cd300 0x7ff0640b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0640cd300 0x7ff0640b8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47560/0 (socket says 172.21.15.77:47560) 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 -- 172.21.15.77:0/1246063022 learned_addr learned my addr 172.21.15.77:0/1246063022 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff062d9d640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c004800 0x7ff064150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06259c640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff064152f20 0x7ff0640b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 -- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff064152f20 msgr2=0x7ff0640b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff064152f20 0x7ff0640b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 -- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c004800 msgr2=0x7ff064150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c004800 0x7ff064150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 -- 172.21.15.77:0/1246063022 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640b7380 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06259c640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff064152f20 0x7ff0640b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff06359e640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0640cd300 0x7ff0640b8830 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7ff05c007fd0 tx=0x7ff05c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff04b7fe640 1 -- 172.21.15.77:0/1246063022 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c003970 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff04b7fe640 1 -- 172.21.15.77:0/1246063022 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c018070 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0640b7660 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff04b7fe640 1 -- 172.21.15.77:0/1246063022 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c003d70 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.413+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0641536a0 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.414+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06404f330 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.415+0000 7ff04b7fe640 1 -- 172.21.15.77:0/1246063022 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff05c0050d0 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.415+0000 7ff04b7fe640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff05006b4d0 0x7ff05006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.415+0000 7ff04b7fe640 1 -- 172.21.15.77:0/1246063022 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff05c089ea0 con 0x7ff0640cd300 2024-03-06T07:31:10.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.415+0000 7ff062d9d640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff05006b4d0 0x7ff05006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.416+0000 7ff062d9d640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff05006b4d0 0x7ff05006d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff064050c30 tx=0x7ff054008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.420+0000 7ff04b7fe640 1 -- 172.21.15.77:0/1246063022 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff05c058d00 con 0x7ff0640cd300 2024-03-06T07:31:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.632+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff06406c710 con 0x7ff0640cd300 2024-03-06T07:31:10.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.632+0000 7ff04b7fe640 1 -- 172.21.15.77:0/1246063022 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7ff05c058600 con 0x7ff0640cd300 2024-03-06T07:31:10.634 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:10.634 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:10.634 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:10.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff05006b4d0 msgr2=0x7ff05006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff05006b4d0 0x7ff05006d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff064050c30 tx=0x7ff054008040 comp rx=0 tx=0).stop 2024-03-06T07:31:10.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0640cd300 msgr2=0x7ff0640b8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0640cd300 0x7ff0640b8830 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7ff05c007fd0 tx=0x7ff05c00ec10 comp rx=0 tx=0).stop 2024-03-06T07:31:10.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 shutdown_connections 2024-03-06T07:31:10.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff05006b4d0 0x7ff05006d990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0640cd300 0x7ff0640b8830 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff064152f20 0x7ff0640b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 --2- 172.21.15.77:0/1246063022 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff04c004800 0x7ff064150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 >> 172.21.15.77:0/1246063022 conn(0x7ff0640d3cd0 msgr2=0x7ff0640b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 shutdown_connections 2024-03-06T07:31:10.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:10.634+0000 7ff069039640 1 -- 172.21.15.77:0/1246063022 wait complete. 2024-03-06T07:31:10.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:10.852 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:11.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.956+0000 7f9f64e78640 1 Processor -- start 2024-03-06T07:31:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.956+0000 7f9f64e78640 1 -- start start 2024-03-06T07:31:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f64e78640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f6015dfe0 0x7f9f600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f64e78640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9f600cf830 0x7f9f600d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f64e78640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600d27f0 0x7f9f600d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f64e78640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600b31e0 con 0x7f9f6015dfe0 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f64e78640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600b3d60 con 0x7f9f600cf830 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f64e78640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600b3430 con 0x7f9f600d27f0 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5e59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f6015dfe0 0x7f9f600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5e59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f6015dfe0 0x7f9f600cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47574/0 (socket says 172.21.15.77:47574) 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5e59c640 1 -- 172.21.15.77:0/2280394400 learned_addr learned my addr 172.21.15.77:0/2280394400 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f57fff640 1 --2- 172.21.15.77:0/2280394400 >> v2:172.21.15.158:3300/0 conn(0x7f9f600cf830 0x7f9f600d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5ed9d640 1 --2- 172.21.15.77:0/2280394400 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600d27f0 0x7f9f600d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5e59c640 1 -- 172.21.15.77:0/2280394400 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600d27f0 msgr2=0x7f9f600d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5e59c640 1 --2- 172.21.15.77:0/2280394400 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600d27f0 0x7f9f600d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5e59c640 1 -- 172.21.15.77:0/2280394400 >> v2:172.21.15.158:3300/0 conn(0x7f9f600cf830 msgr2=0x7f9f600d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5e59c640 1 --2- 172.21.15.77:0/2280394400 >> v2:172.21.15.158:3300/0 conn(0x7f9f600cf830 0x7f9f600d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5e59c640 1 -- 172.21.15.77:0/2280394400 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600b3710 con 0x7f9f6015dfe0 2024-03-06T07:31:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f5ed9d640 1 --2- 172.21.15.77:0/2280394400 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600d27f0 0x7f9f600d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.957+0000 7f9f57fff640 1 --2- 172.21.15.77:0/2280394400 >> v2:172.21.15.158:3300/0 conn(0x7f9f600cf830 0x7f9f600d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.958+0000 7f9f5e59c640 1 --2- 172.21.15.77:0/2280394400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f6015dfe0 0x7f9f600cf2f0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f9f600d16b0 tx=0x7f9f5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4246deffec9a7eef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.958+0000 7f9f5dd9b640 1 -- 172.21.15.77:0/2280394400 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50035020 con 0x7f9f6015dfe0 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.958+0000 7f9f5dd9b640 1 -- 172.21.15.77:0/2280394400 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f50031870 con 0x7f9f6015dfe0 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.958+0000 7f9f5dd9b640 1 -- 172.21.15.77:0/2280394400 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50031c40 con 0x7f9f6015dfe0 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 -- 172.21.15.77:0/2280394400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f6015dfe0 msgr2=0x7f9f600cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 --2- 172.21.15.77:0/2280394400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f6015dfe0 0x7f9f600cf2f0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f9f600d16b0 tx=0x7f9f5002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 -- 172.21.15.77:0/2280394400 shutdown_connections 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 --2- 172.21.15.77:0/2280394400 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600d27f0 0x7f9f600d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 --2- 172.21.15.77:0/2280394400 >> v2:172.21.15.158:3300/0 conn(0x7f9f600cf830 0x7f9f600d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 --2- 172.21.15.77:0/2280394400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f6015dfe0 0x7f9f600cf2f0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 -- 172.21.15.77:0/2280394400 >> 172.21.15.77:0/2280394400 conn(0x7f9f600d3cd0 msgr2=0x7f9f600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 -- 172.21.15.77:0/2280394400 shutdown_connections 2024-03-06T07:31:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.959+0000 7f9f64e78640 1 -- 172.21.15.77:0/2280394400 wait complete. 2024-03-06T07:31:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.960+0000 7f9f64e78640 1 Processor -- start 2024-03-06T07:31:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.960+0000 7f9f64e78640 1 -- start start 2024-03-06T07:31:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f64e78640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600cf830 0x7f9f601f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f64e78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f600d27f0 0x7f9f600cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f64e78640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f6015dfe0 0x7f9f600b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f64e78640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600d1df0 con 0x7f9f600d27f0 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f64e78640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600d1c70 con 0x7f9f6015dfe0 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f64e78640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6015e3c0 con 0x7f9f600cf830 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f600d27f0 0x7f9f600cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f600d27f0 0x7f9f600cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47578/0 (socket says 172.21.15.77:47578) 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 -- 172.21.15.77:0/117064119 learned_addr learned my addr 172.21.15.77:0/117064119 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f5ed9d640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f6015dfe0 0x7f9f600b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f5e59c640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600cf830 0x7f9f601f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 -- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600cf830 msgr2=0x7f9f601f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600cf830 0x7f9f601f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 -- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f6015dfe0 msgr2=0x7f9f600b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f6015dfe0 0x7f9f600b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 -- 172.21.15.77:0/117064119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600cbc70 con 0x7f9f600d27f0 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f5e59c640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600cf830 0x7f9f601f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f5ed9d640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f6015dfe0 0x7f9f600b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.961+0000 7f9f57fff640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f600d27f0 0x7f9f600cd850 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800c7c0 tx=0x7f9f4800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.962+0000 7f9f56ffd640 1 -- 172.21.15.77:0/117064119 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48015070 con 0x7f9f600d27f0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.962+0000 7f9f56ffd640 1 -- 172.21.15.77:0/117064119 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f48007820 con 0x7f9f600d27f0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.962+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f600cbf50 con 0x7f9f600d27f0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.962+0000 7f9f56ffd640 1 -- 172.21.15.77:0/117064119 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48007bf0 con 0x7f9f600d27f0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.962+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f600cc510 con 0x7f9f600d27f0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.963+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6004f330 con 0x7f9f600d27f0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.963+0000 7f9f56ffd640 1 -- 172.21.15.77:0/117064119 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f48018070 con 0x7f9f600d27f0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.964+0000 7f9f56ffd640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f4006b4d0 0x7f9f4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.964+0000 7f9f56ffd640 1 -- 172.21.15.77:0/117064119 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9f4808a130 con 0x7f9f600d27f0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.964+0000 7f9f5e59c640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f4006b4d0 0x7f9f4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.964+0000 7f9f5e59c640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f4006b4d0 0x7f9f4006d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9f50036000 tx=0x7f9f50008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:11.968+0000 7f9f56ffd640 1 -- 172.21.15.77:0/117064119 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f48058f90 con 0x7f9f600d27f0 2024-03-06T07:31:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.183+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f600cc910 con 0x7f9f600d27f0 2024-03-06T07:31:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.184+0000 7f9f56ffd640 1 -- 172.21.15.77:0/117064119 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f9f48058890 con 0x7f9f600d27f0 2024-03-06T07:31:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:12.186 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:12.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f4006b4d0 msgr2=0x7f9f4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f4006b4d0 0x7f9f4006d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9f50036000 tx=0x7f9f50008030 comp rx=0 tx=0).stop 2024-03-06T07:31:12.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f600d27f0 msgr2=0x7f9f600cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f600d27f0 0x7f9f600cd850 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800c7c0 tx=0x7f9f4800cc80 comp rx=0 tx=0).stop 2024-03-06T07:31:12.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 shutdown_connections 2024-03-06T07:31:12.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9f4006b4d0 0x7f9f4006d990 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f600d27f0 0x7f9f600cd850 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9f6015dfe0 0x7f9f600b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 --2- 172.21.15.77:0/117064119 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9f600cf830 0x7f9f601f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 >> 172.21.15.77:0/117064119 conn(0x7f9f600d3cd0 msgr2=0x7f9f6006cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 shutdown_connections 2024-03-06T07:31:12.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:12.186+0000 7f9f64e78640 1 -- 172.21.15.77:0/117064119 wait complete. 2024-03-06T07:31:12.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:12.401 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:13.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:13.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.532+0000 7ff1c37d6640 1 Processor -- start 2024-03-06T07:31:13.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.532+0000 7ff1c37d6640 1 -- start start 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c37d6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff1bc0cfff0 0x7ff1bc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c37d6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff1bc0cc8b0 0x7ff1bc0ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c37d6640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc161980 con 0x7ff1bc15bb20 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c37d6640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1bc0d2500 con 0x7ff1bc0cfff0 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c37d6640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1bc161c60 con 0x7ff1bc0cc8b0 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c0d71640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff1bc0cfff0 0x7ff1bc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c0d71640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff1bc0cfff0 0x7ff1bc0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47584/0 (socket says 172.21.15.77:47584) 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c0d71640 1 -- 172.21.15.77:0/1638815357 learned_addr learned my addr 172.21.15.77:0/1638815357 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c1d73640 1 --2- 172.21.15.77:0/1638815357 >> v2:172.21.15.158:3300/0 conn(0x7ff1bc0cc8b0 0x7ff1bc0ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c0d71640 1 -- 172.21.15.77:0/1638815357 >> v2:172.21.15.158:3300/0 conn(0x7ff1bc0cc8b0 msgr2=0x7ff1bc0ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c0d71640 1 --2- 172.21.15.77:0/1638815357 >> v2:172.21.15.158:3300/0 conn(0x7ff1bc0cc8b0 0x7ff1bc0ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c0d71640 1 -- 172.21.15.77:0/1638815357 >> v1:172.21.15.158:6789/0 conn(0x7ff1bc15bb20 legacy=0x7ff1bc0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.533+0000 7ff1c0d71640 1 -- 172.21.15.77:0/1638815357 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc161f40 con 0x7ff1bc0cfff0 2024-03-06T07:31:13.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.534+0000 7ff1c0d71640 1 --2- 172.21.15.77:0/1638815357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc0cfff0 0x7ff1bc0d04a0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc1618c0 tx=0x7ff1ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f678452aca2cce62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.534+0000 7ff1b3fff640 1 -- 172.21.15.77:0/1638815357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac035020 con 0x7ff1bc0cfff0 2024-03-06T07:31:13.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.534+0000 7ff1b3fff640 1 -- 172.21.15.77:0/1638815357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac031870 con 0x7ff1bc0cfff0 2024-03-06T07:31:13.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.534+0000 7ff1b3fff640 1 -- 172.21.15.77:0/1638815357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac031c40 con 0x7ff1bc0cfff0 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.535+0000 7ff1c37d6640 1 -- 172.21.15.77:0/1638815357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc0cfff0 msgr2=0x7ff1bc0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.535+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/1638815357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc0cfff0 0x7ff1bc0d04a0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc1618c0 tx=0x7ff1ac02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:13.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.535+0000 7ff1c37d6640 1 -- 172.21.15.77:0/1638815357 shutdown_connections 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.535+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/1638815357 >> v2:172.21.15.158:3300/0 conn(0x7ff1bc0cc8b0 0x7ff1bc0ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.535+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/1638815357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc0cfff0 0x7ff1bc0d04a0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.535+0000 7ff1c37d6640 1 -- 172.21.15.77:0/1638815357 >> 172.21.15.77:0/1638815357 conn(0x7ff1bc09a9f0 msgr2=0x7ff1bc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.535+0000 7ff1c37d6640 1 -- 172.21.15.77:0/1638815357 shutdown_connections 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.535+0000 7ff1c37d6640 1 -- 172.21.15.77:0/1638815357 wait complete. 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.536+0000 7ff1c37d6640 1 Processor -- start 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.536+0000 7ff1c37d6640 1 -- start start 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.536+0000 7ff1c37d6640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1bc0cc8b0 0x7ff1bc155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c37d6640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1bc0cfff0 0x7ff1bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c37d6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc15bb20 0x7ff1bc06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c37d6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0d1980 con 0x7ff1bc15bb20 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c37d6640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc162500 con 0x7ff1bc0cfff0 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c37d6640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0d1b00 con 0x7ff1bc0cc8b0 2024-03-06T07:31:13.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc15bb20 0x7ff1bc06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc15bb20 0x7ff1bc06d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47586/0 (socket says 172.21.15.77:47586) 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 -- 172.21.15.77:0/502947924 learned_addr learned my addr 172.21.15.77:0/502947924 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c0d71640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1bc0cfff0 0x7ff1bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1572640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1bc0cc8b0 0x7ff1bc155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 -- 172.21.15.77:0/502947924 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1bc0cc8b0 msgr2=0x7ff1bc155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1bc0cc8b0 0x7ff1bc155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 -- 172.21.15.77:0/502947924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1bc0cfff0 msgr2=0x7ff1bc06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1bc0cfff0 0x7ff1bc06d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 -- 172.21.15.77:0/502947924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc0c5790 con 0x7ff1bc15bb20 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c1d73640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc15bb20 0x7ff1bc06d900 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4007e90 tx=0x7ff1b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1b1ffb640 1 -- 172.21.15.77:0/502947924 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4014420 con 0x7ff1bc15bb20 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1b1ffb640 1 -- 172.21.15.77:0/502947924 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b4019070 con 0x7ff1bc15bb20 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc0c5a70 con 0x7ff1bc15bb20 2024-03-06T07:31:13.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1b1ffb640 1 -- 172.21.15.77:0/502947924 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4014800 con 0x7ff1bc15bb20 2024-03-06T07:31:13.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.537+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc155540 con 0x7ff1bc15bb20 2024-03-06T07:31:13.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.538+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc04f330 con 0x7ff1bc15bb20 2024-03-06T07:31:13.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.539+0000 7ff1b1ffb640 1 -- 172.21.15.77:0/502947924 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1b40149a0 con 0x7ff1bc15bb20 2024-03-06T07:31:13.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.539+0000 7ff1b1ffb640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1a006b500 0x7ff1a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.539+0000 7ff1b1ffb640 1 -- 172.21.15.77:0/502947924 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff1b408a9c0 con 0x7ff1bc15bb20 2024-03-06T07:31:13.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.539+0000 7ff1c1572640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1a006b500 0x7ff1a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.540+0000 7ff1c1572640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1a006b500 0x7ff1a006d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800a750 tx=0x7ff1b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.544+0000 7ff1b1ffb640 1 -- 172.21.15.77:0/502947924 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1b408ce20 con 0x7ff1bc15bb20 2024-03-06T07:31:13.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.756+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1bc0b1fe0 con 0x7ff1bc15bb20 2024-03-06T07:31:13.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.757+0000 7ff1b1ffb640 1 -- 172.21.15.77:0/502947924 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7ff1b40598b0 con 0x7ff1bc15bb20 2024-03-06T07:31:13.759 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:13.759 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:13.759 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:13.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1a006b500 msgr2=0x7ff1a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1a006b500 0x7ff1a006d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800a750 tx=0x7ff1b8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:13.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc15bb20 msgr2=0x7ff1bc06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc15bb20 0x7ff1bc06d900 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4007e90 tx=0x7ff1b400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 shutdown_connections 2024-03-06T07:31:13.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1a006b500 0x7ff1a006d9c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1bc15bb20 0x7ff1bc06d900 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1bc0cfff0 0x7ff1bc06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 --2- 172.21.15.77:0/502947924 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1bc0cc8b0 0x7ff1bc155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 >> 172.21.15.77:0/502947924 conn(0x7ff1bc09a9f0 msgr2=0x7ff1bc06f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 shutdown_connections 2024-03-06T07:31:13.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:13.759+0000 7ff1c37d6640 1 -- 172.21.15.77:0/502947924 wait complete. 2024-03-06T07:31:13.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:13.973 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:14.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:15.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.073+0000 7fc7d1756640 1 Processor -- start 2024-03-06T07:31:15.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.073+0000 7fc7d1756640 1 -- start start 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7d1756640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc7cc0d0a80 0x7fc7cc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7d1756640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc0d25d0 con 0x7fc7cc0d4760 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7d1756640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc0d2870 con 0x7fc7cc0b4170 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7d1756640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7cc0d4bf0 con 0x7fc7cc0d0a80 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7ca7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc7cc0d4760 0x7fc7cc0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44060/0 (socket says 172.21.15.77:44060) 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7ca7fc640 1 -- 172.21.15.77:0/1917761546 learned_addr learned my addr 172.21.15.77:0/1917761546 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7cb7fe640 1 --2- 172.21.15.77:0/1917761546 >> v2:172.21.15.158:3300/0 conn(0x7fc7cc0d0a80 0x7fc7cc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765861979 0 0) 0x7fc7cc0d25d0 con 0x7fc7cc0d4760 2024-03-06T07:31:15.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b8003520 con 0x7fc7cc0d4760 2024-03-06T07:31:15.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575954819 0 0) 0x7fc7b8003520 con 0x7fc7cc0d4760 2024-03-06T07:31:15.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 >> v2:172.21.15.158:3300/0 conn(0x7fc7cc0d0a80 msgr2=0x7fc7cc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 --2- 172.21.15.77:0/1917761546 >> v2:172.21.15.158:3300/0 conn(0x7fc7cc0d0a80 0x7fc7cc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 >> v1:172.21.15.158:6789/0 conn(0x7fc7cc0b4170 legacy=0x7fc7cc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc0d2b50 con 0x7fc7cc0d4760 2024-03-06T07:31:15.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fc7b4001670 con 0x7fc7cc0d4760 2024-03-06T07:31:15.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 >> v1:172.21.15.77:6789/0 conn(0x7fc7cc0d4760 legacy=0x7fc7cc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7b80039b0 0x7fc7b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7b8004800 0x7fc7b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.074+0000 7fc7c9ffb640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0d4bf0 con 0x7fc7b80051c0 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b800c060 con 0x7fc7b80039b0 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b800c260 con 0x7fc7b8004800 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7cb7fe640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7caffd640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7b80039b0 0x7fc7b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7ca7fc640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7b8004800 0x7fc7b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7cb7fe640 1 -- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7b8004800 msgr2=0x7fc7b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7cb7fe640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7b8004800 0x7fc7b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7cb7fe640 1 -- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7b80039b0 msgr2=0x7fc7b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7cb7fe640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7b80039b0 0x7fc7b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7cb7fe640 1 -- 172.21.15.77:0/1917761546 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8003520 con 0x7fc7b80051c0 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.075+0000 7fc7ca7fc640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7b8004800 0x7fc7b8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7cb7fe640 1 -- 172.21.15.77:0/1917761546 reap_dead start 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7cb7fe640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7b800b9c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009ac0 tx=0x7fc7b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94c0986f8dda148 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc009690 con 0x7fc7b80051c0 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc004020 con 0x7fc7b80051c0 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7c9ffb640 1 -- 172.21.15.77:0/1917761546 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc004410 con 0x7fc7b80051c0 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7d1756640 1 -- 172.21.15.77:0/1917761546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 msgr2=0x7fc7b800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7d1756640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7b800b9c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009ac0 tx=0x7fc7b8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7d1756640 1 -- 172.21.15.77:0/1917761546 shutdown_connections 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7d1756640 1 --2- 172.21.15.77:0/1917761546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7b800b9c0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7d1756640 1 -- 172.21.15.77:0/1917761546 >> 172.21.15.77:0/1917761546 conn(0x7fc7cc0d3c40 msgr2=0x7fc7cc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7d1756640 1 -- 172.21.15.77:0/1917761546 shutdown_connections 2024-03-06T07:31:15.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.076+0000 7fc7d1756640 1 -- 172.21.15.77:0/1917761546 wait complete. 2024-03-06T07:31:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.078+0000 7fc7d1756640 1 Processor -- start 2024-03-06T07:31:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.078+0000 7fc7d1756640 1 -- start start 2024-03-06T07:31:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.079+0000 7fc7d1756640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7cc157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.079+0000 7fc7d1756640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7cc1f4480 0x7fc7cc15a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7d1756640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7cc15a610 0x7fc7cc1f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7d1756640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0b4dd0 con 0x7fc7b80051c0 2024-03-06T07:31:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7d1756640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0ce9e0 con 0x7fc7cc15a610 2024-03-06T07:31:15.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7d1756640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0b4f50 con 0x7fc7cc1f4480 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7cc157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7cc157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47610/0 (socket says 172.21.15.77:47610) 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 -- 172.21.15.77:0/1270792934 learned_addr learned my addr 172.21.15.77:0/1270792934 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7ca7fc640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7cc1f4480 0x7fc7cc15a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7cb7fe640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7cc15a610 0x7fc7cc1f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 -- 172.21.15.77:0/1270792934 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7cc1f4480 msgr2=0x7fc7cc15a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7cc1f4480 0x7fc7cc15a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 -- 172.21.15.77:0/1270792934 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7cc15a610 msgr2=0x7fc7cc1f6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7cc15a610 0x7fc7cc1f6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 -- 172.21.15.77:0/1270792934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc157750 con 0x7fc7b80051c0 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7caffd640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7cc157210 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400f960 tx=0x7fc7c400fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7c37fe640 1 -- 172.21.15.77:0/1270792934 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4003b80 con 0x7fc7b80051c0 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7c37fe640 1 -- 172.21.15.77:0/1270792934 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c4018070 con 0x7fc7b80051c0 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7c37fe640 1 -- 172.21.15.77:0/1270792934 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400d790 con 0x7fc7b80051c0 2024-03-06T07:31:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc158470 con 0x7fc7b80051c0 2024-03-06T07:31:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.080+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc158710 con 0x7fc7b80051c0 2024-03-06T07:31:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.081+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7cc04f330 con 0x7fc7b80051c0 2024-03-06T07:31:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.082+0000 7fc7c37fe640 1 -- 172.21.15.77:0/1270792934 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7c400d930 con 0x7fc7b80051c0 2024-03-06T07:31:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.082+0000 7fc7c37fe640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.082+0000 7fc7c37fe640 1 -- 172.21.15.77:0/1270792934 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc7c408a2d0 con 0x7fc7b80051c0 2024-03-06T07:31:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.082+0000 7fc7ca7fc640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.083+0000 7fc7ca7fc640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4001200 tx=0x7fc7b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.087+0000 7fc7c37fe640 1 -- 172.21.15.77:0/1270792934 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7cc157750 con 0x7fc7b80051c0 2024-03-06T07:31:15.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.303+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7cc0733b0 con 0x7fc7b80051c0 2024-03-06T07:31:15.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.303+0000 7fc7c37fe640 1 -- 172.21.15.77:0/1270792934 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7fc7c401a310 con 0x7fc7b80051c0 2024-03-06T07:31:15.305 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:15.306 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:15.306 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7ac06b500 msgr2=0x7fc7ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4001200 tx=0x7fc7b4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 msgr2=0x7fc7cc157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7cc157210 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400f960 tx=0x7fc7c400fe20 comp rx=0 tx=0).stop 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 shutdown_connections 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc7cc15a610 0x7fc7cc1f6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc7cc1f4480 0x7fc7cc15a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 --2- 172.21.15.77:0/1270792934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7b80051c0 0x7fc7cc157210 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.305+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 >> 172.21.15.77:0/1270792934 conn(0x7fc7cc0d3c40 msgr2=0x7fc7cc0b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.306+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 shutdown_connections 2024-03-06T07:31:15.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:15.306+0000 7fc7d1756640 1 -- 172.21.15.77:0/1270792934 wait complete. 2024-03-06T07:31:15.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:15.520 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:16.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:16.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.623+0000 7fc4b68c3640 1 Processor -- start 2024-03-06T07:31:16.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.623+0000 7fc4b68c3640 1 -- start start 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.623+0000 7fc4b68c3640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc4b015dfe0 0x7fc4b00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.623+0000 7fc4b68c3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4b00cf830 0x7fc4b00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.623+0000 7fc4b68c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00d27f0 0x7fc4b00d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.623+0000 7fc4b68c3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b00b31e0 con 0x7fc4b00cf830 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.623+0000 7fc4b68c3640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b00b3d60 con 0x7fc4b015dfe0 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.623+0000 7fc4b68c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00b3430 con 0x7fc4b00d27f0 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4af7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4b00cf830 0x7fc4b00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4af7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4b00cf830 0x7fc4b00d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47614/0 (socket says 172.21.15.77:47614) 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4af7fe640 1 -- 172.21.15.77:0/2648253921 learned_addr learned my addr 172.21.15.77:0/2648253921 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:16.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4af7fe640 1 -- 172.21.15.77:0/2648253921 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00d27f0 msgr2=0x7fc4b00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4b4e60640 1 --2- 172.21.15.77:0/2648253921 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00d27f0 0x7fc4b00d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4affff640 1 --2- 172.21.15.77:0/2648253921 >> v2:172.21.15.158:3300/0 conn(0x7fc4b015dfe0 0x7fc4b00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4af7fe640 1 --2- 172.21.15.77:0/2648253921 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00d27f0 0x7fc4b00d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4af7fe640 1 -- 172.21.15.77:0/2648253921 >> v2:172.21.15.158:3300/0 conn(0x7fc4b015dfe0 msgr2=0x7fc4b00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4af7fe640 1 --2- 172.21.15.77:0/2648253921 >> v2:172.21.15.158:3300/0 conn(0x7fc4b015dfe0 0x7fc4b00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4af7fe640 1 -- 172.21.15.77:0/2648253921 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b00b3710 con 0x7fc4b00cf830 2024-03-06T07:31:16.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.624+0000 7fc4affff640 1 --2- 172.21.15.77:0/2648253921 >> v2:172.21.15.158:3300/0 conn(0x7fc4b015dfe0 0x7fc4b00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:16.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4af7fe640 1 --2- 172.21.15.77:0/2648253921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b00cf830 0x7fc4b00d22b0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0067990 tx=0x7fc4a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53838418709e14d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4aeffd640 1 -- 172.21.15.77:0/2648253921 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4035020 con 0x7fc4b00cf830 2024-03-06T07:31:16.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4aeffd640 1 -- 172.21.15.77:0/2648253921 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a4031870 con 0x7fc4b00cf830 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4aeffd640 1 -- 172.21.15.77:0/2648253921 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4031c40 con 0x7fc4b00cf830 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 -- 172.21.15.77:0/2648253921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b00cf830 msgr2=0x7fc4b00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/2648253921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b00cf830 0x7fc4b00d22b0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0067990 tx=0x7fc4a402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 -- 172.21.15.77:0/2648253921 shutdown_connections 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/2648253921 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00d27f0 0x7fc4b00d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/2648253921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b00cf830 0x7fc4b00d22b0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/2648253921 >> v2:172.21.15.158:3300/0 conn(0x7fc4b015dfe0 0x7fc4b00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 -- 172.21.15.77:0/2648253921 >> 172.21.15.77:0/2648253921 conn(0x7fc4b00d3cd0 msgr2=0x7fc4b00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 -- 172.21.15.77:0/2648253921 shutdown_connections 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.625+0000 7fc4b68c3640 1 -- 172.21.15.77:0/2648253921 wait complete. 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.627+0000 7fc4b68c3640 1 Processor -- start 2024-03-06T07:31:16.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.627+0000 7fc4b68c3640 1 -- start start 2024-03-06T07:31:16.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.627+0000 7fc4b68c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00cf830 0x7fc4b0160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.627+0000 7fc4b68c3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4b00d27f0 0x7fc4b01f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.627+0000 7fc4b68c3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b015dfe0 0x7fc4b015bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.627+0000 7fc4b68c3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00d1df0 con 0x7fc4b015dfe0 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.627+0000 7fc4b68c3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00d1c70 con 0x7fc4b00d27f0 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.627+0000 7fc4b68c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b015e3c0 con 0x7fc4b00cf830 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b015dfe0 0x7fc4b015bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b015dfe0 0x7fc4b015bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47618/0 (socket says 172.21.15.77:47618) 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 -- 172.21.15.77:0/3646068914 learned_addr learned my addr 172.21.15.77:0/3646068914 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4affff640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00cf830 0x7fc4b0160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4af7fe640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4b00d27f0 0x7fc4b01f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 -- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00cf830 msgr2=0x7fc4b0160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00cf830 0x7fc4b0160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 -- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4b00d27f0 msgr2=0x7fc4b01f71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4b00d27f0 0x7fc4b01f71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 -- 172.21.15.77:0/3646068914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b015fca0 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4affff640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00cf830 0x7fc4b0160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b4e60640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b015dfe0 0x7fc4b015bb10 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800ccf0 tx=0x7fc4a800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4acff9640 1 -- 172.21.15.77:0/3646068914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8003b90 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4acff9640 1 -- 172.21.15.77:0/3646068914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a8003d30 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b015ff80 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4acff9640 1 -- 172.21.15.77:0/3646068914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004240 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.628+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b01f7730 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.629+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b004f330 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.630+0000 7fc4acff9640 1 -- 172.21.15.77:0/3646068914 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4a80043e0 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.630+0000 7fc4acff9640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.630+0000 7fc4acff9640 1 -- 172.21.15.77:0/3646068914 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc4a808abe0 con 0x7fc4b015dfe0 2024-03-06T07:31:16.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.630+0000 7fc4affff640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.630+0000 7fc4affff640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc49c06b4d0 0x7fc49c06d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000a990 tx=0x7fc4a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:16.634+0000 7fc4acff9640 1 -- 172.21.15.77:0/3646068914 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4a8059ad0 con 0x7fc4b015dfe0 2024-03-06T07:31:17.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.044+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4b00ceb20 con 0x7fc4b015dfe0 2024-03-06T07:31:17.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.044+0000 7fc4acff9640 1 -- 172.21.15.77:0/3646068914 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7fc4a80593d0 con 0x7fc4b015dfe0 2024-03-06T07:31:17.047 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:17.047 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc49c06b4d0 msgr2=0x7fc49c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc49c06b4d0 0x7fc49c06d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000a990 tx=0x7fc4a0008040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b015dfe0 msgr2=0x7fc4b015bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b015dfe0 0x7fc4b015bb10 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800ccf0 tx=0x7fc4a800a660 comp rx=0 tx=0).stop 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 shutdown_connections 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4b015dfe0 0x7fc4b015bb10 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc4b00d27f0 0x7fc4b01f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 --2- 172.21.15.77:0/3646068914 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc4b00cf830 0x7fc4b0160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 >> 172.21.15.77:0/3646068914 conn(0x7fc4b00d3cd0 msgr2=0x7fc4b006bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 shutdown_connections 2024-03-06T07:31:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:17.047+0000 7fc4b68c3640 1 -- 172.21.15.77:0/3646068914 wait complete. 2024-03-06T07:31:17.260 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:17.260 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:18.261 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:18.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.362+0000 7f43b962a640 1 Processor -- start 2024-03-06T07:31:18.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.362+0000 7f43b962a640 1 -- start start 2024-03-06T07:31:18.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b962a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f43b41424f0 0x7f43b40cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b962a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b40cc980 con 0x7f43b406da40 2024-03-06T07:31:18.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b962a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b40ccc60 con 0x7f43b40cd7a0 2024-03-06T07:31:18.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b962a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40cd220 con 0x7f43b41424f0 2024-03-06T07:31:18.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f43b406da40 0x7f43b4071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44064/0 (socket says 172.21.15.77:44064) 2024-03-06T07:31:18.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b37fe640 1 -- 172.21.15.77:0/1048876501 learned_addr learned my addr 172.21.15.77:0/1048876501 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066727480 0 0) 0x7f43b40cc980 con 0x7f43b406da40 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43a0003520 con 0x7f43b406da40 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1048876501 >> v2:172.21.15.158:3300/0 conn(0x7f43b41424f0 0x7f43b40cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3315792244 0 0) 0x7f43a0003520 con 0x7f43b406da40 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 >> v2:172.21.15.158:3300/0 conn(0x7f43b41424f0 msgr2=0x7f43b40cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 --2- 172.21.15.77:0/1048876501 >> v2:172.21.15.158:3300/0 conn(0x7f43b41424f0 0x7f43b40cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 >> v1:172.21.15.158:6789/0 conn(0x7f43b40cd7a0 legacy=0x7f43b406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40ccf40 con 0x7f43b406da40 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f43ac0019f0 con 0x7f43b406da40 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 >> v1:172.21.15.77:6789/0 conn(0x7f43b406da40 legacy=0x7f43b4071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43a0004800 0x7f43a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00051c0 0x7f43a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40cd220 con 0x7f43a00039b0 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a000c060 con 0x7f43a00051c0 2024-03-06T07:31:18.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.363+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a000c260 con 0x7f43a0004800 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b37fe640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00051c0 0x7f43a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43aa7fc640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43a0004800 0x7f43a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b2ffd640 1 -- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43a0004800 msgr2=0x7f43a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43a0004800 0x7f43a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b2ffd640 1 -- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00051c0 msgr2=0x7f43a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00051c0 0x7f43a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b2ffd640 1 -- 172.21.15.77:0/1048876501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0003520 con 0x7f43a00039b0 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b37fe640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00051c0 0x7f43a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b37fe640 1 -- 172.21.15.77:0/1048876501 reap_dead start 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43aa7fc640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43a0004800 0x7f43a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.364+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009ac0 tx=0x7f43a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3525f42248e6f34a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4039070 con 0x7f43a00039b0 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a402c6b0 con 0x7f43a00039b0 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b27fc640 1 -- 172.21.15.77:0/1048876501 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4009690 con 0x7f43a00039b0 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b962a640 1 -- 172.21.15.77:0/1048876501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a00039b0 msgr2=0x7f43a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b962a640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009ac0 tx=0x7f43a0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:18.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b962a640 1 -- 172.21.15.77:0/1048876501 shutdown_connections 2024-03-06T07:31:18.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b962a640 1 --2- 172.21.15.77:0/1048876501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b962a640 1 -- 172.21.15.77:0/1048876501 >> 172.21.15.77:0/1048876501 conn(0x7f43b40cf8a0 msgr2=0x7f43b40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b962a640 1 -- 172.21.15.77:0/1048876501 shutdown_connections 2024-03-06T07:31:18.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.365+0000 7f43b962a640 1 -- 172.21.15.77:0/1048876501 wait complete. 2024-03-06T07:31:18.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.366+0000 7f43b962a640 1 Processor -- start 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43b962a640 1 -- start start 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43b962a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00039b0 0x7f43b4139d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43b962a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b415ad50 0x7f43b40ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43b962a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b41391f0 0x7f43b4139680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43b962a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40cc470 con 0x7f43b415ad50 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43b962a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40cc2f0 con 0x7f43a00039b0 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43b962a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40cc5f0 con 0x7f43b41391f0 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43aa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b415ad50 0x7f43b40ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43aa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b415ad50 0x7f43b40ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47632/0 (socket says 172.21.15.77:47632) 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.367+0000 7f43aa7fc640 1 -- 172.21.15.77:0/1986102128 learned_addr learned my addr 172.21.15.77:0/1986102128 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43b37fe640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b41391f0 0x7f43b4139680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00039b0 0x7f43b4139d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43aa7fc640 1 -- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b41391f0 msgr2=0x7f43b4139680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43aa7fc640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b41391f0 0x7f43b4139680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43aa7fc640 1 -- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00039b0 msgr2=0x7f43b4139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43aa7fc640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00039b0 0x7f43b4139d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43aa7fc640 1 -- 172.21.15.77:0/1986102128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b415b510 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00039b0 0x7f43b4139d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43aa7fc640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b415ad50 0x7f43b40ce390 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f439c00d4a0 tx=0x7f439c00d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43abfff640 1 -- 172.21.15.77:0/1986102128 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c019070 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43abfff640 1 -- 172.21.15.77:0/1986102128 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c005480 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b415b7f0 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43abfff640 1 -- 172.21.15.77:0/1986102128 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c007230 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.368+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b41e5d00 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.369+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b404f330 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.370+0000 7f43abfff640 1 -- 172.21.15.77:0/1986102128 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f439c01c070 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.370+0000 7f43abfff640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439406b4d0 0x7f439406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.370+0000 7f43abfff640 1 -- 172.21.15.77:0/1986102128 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f439c0897f0 con 0x7f43b415ad50 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.370+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439406b4d0 0x7f439406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.370+0000 7f43b2ffd640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439406b4d0 0x7f439406d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f43a4036000 tx=0x7f43a4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.378+0000 7f43abfff640 1 -- 172.21.15.77:0/1986102128 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f439c058090 con 0x7f43b415ad50 2024-03-06T07:31:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.588+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43b40d0620 con 0x7f43b415ad50 2024-03-06T07:31:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.589+0000 7f43abfff640 1 -- 172.21.15.77:0/1986102128 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7f439c05c130 con 0x7f43b415ad50 2024-03-06T07:31:18.591 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:18.591 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:18.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.590+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439406b4d0 msgr2=0x7f439406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.590+0000 7f43b962a640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439406b4d0 0x7f439406d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f43a4036000 tx=0x7f43a4030000 comp rx=0 tx=0).stop 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.590+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b415ad50 msgr2=0x7f43b40ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.590+0000 7f43b962a640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b415ad50 0x7f43b40ce390 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f439c00d4a0 tx=0x7f439c00d960 comp rx=0 tx=0).stop 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.591+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 shutdown_connections 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.591+0000 7f43b962a640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f439406b4d0 0x7f439406d990 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.591+0000 7f43b962a640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b415ad50 0x7f43b40ce390 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.591+0000 7f43b962a640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43b41391f0 0x7f43b4139680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.591+0000 7f43b962a640 1 --2- 172.21.15.77:0/1986102128 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43a00039b0 0x7f43b4139d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.591+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 >> 172.21.15.77:0/1986102128 conn(0x7f43b40cf8a0 msgr2=0x7f43b415f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.591+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 shutdown_connections 2024-03-06T07:31:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:18.591+0000 7f43b962a640 1 -- 172.21.15.77:0/1986102128 wait complete. 2024-03-06T07:31:18.804 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:18.804 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:19.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:19.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.904+0000 7fcdc2302640 1 Processor -- start 2024-03-06T07:31:19.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.904+0000 7fcdc2302640 1 -- start start 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.905+0000 7fcdc2302640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcdbc15dfe0 0x7fcdbc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.905+0000 7fcdc2302640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 0x7fcdbc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.905+0000 7fcdc2302640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc0b3430 con 0x7fcdbc0cf830 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.905+0000 7fcdc2302640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdbc0b31e0 con 0x7fcdbc15dfe0 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.905+0000 7fcdc2302640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0b3d60 con 0x7fcdbc0d29c0 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.905+0000 7fcdbbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcdbc15dfe0 0x7fcdbc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.905+0000 7fcdbb7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcdbc0cf830 0x7fcdbc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54496/0 (socket says 172.21.15.77:54496) 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.905+0000 7fcdbb7fe640 1 -- 172.21.15.77:0/3719133420 learned_addr learned my addr 172.21.15.77:0/3719133420 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.906+0000 7fcdbbfff640 1 -- 172.21.15.77:0/3719133420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 msgr2=0x7fcdbc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.906+0000 7fcdbbfff640 1 --2- 172.21.15.77:0/3719133420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 0x7fcdbc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.906+0000 7fcdbbfff640 1 -- 172.21.15.77:0/3719133420 >> v1:172.21.15.77:6789/0 conn(0x7fcdbc0cf830 legacy=0x7fcdbc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.906+0000 7fcdbbfff640 1 -- 172.21.15.77:0/3719133420 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc0b3710 con 0x7fcdbc15dfe0 2024-03-06T07:31:19.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.906+0000 7fcdbbfff640 1 --2- 172.21.15.77:0/3719133420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc15dfe0 0x7fcdbc0cf2f0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00097a0 tx=0x7fcdb002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfadad1ee7ce4d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.906+0000 7fcdbaffd640 1 -- 172.21.15.77:0/3719133420 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0035020 con 0x7fcdbc15dfe0 2024-03-06T07:31:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.906+0000 7fcdbaffd640 1 -- 172.21.15.77:0/3719133420 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb0031810 con 0x7fcdbc15dfe0 2024-03-06T07:31:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.906+0000 7fcdbaffd640 1 -- 172.21.15.77:0/3719133420 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0031be0 con 0x7fcdbc15dfe0 2024-03-06T07:31:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.907+0000 7fcdc2302640 1 -- 172.21.15.77:0/3719133420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc15dfe0 msgr2=0x7fcdbc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.907+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3719133420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc15dfe0 0x7fcdbc0cf2f0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00097a0 tx=0x7fcdb002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.907+0000 7fcdc2302640 1 -- 172.21.15.77:0/3719133420 shutdown_connections 2024-03-06T07:31:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.907+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3719133420 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 0x7fcdbc0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.907+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3719133420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc15dfe0 0x7fcdbc0cf2f0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.907+0000 7fcdc2302640 1 -- 172.21.15.77:0/3719133420 >> 172.21.15.77:0/3719133420 conn(0x7fcdbc0d3cd0 msgr2=0x7fcdbc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.907+0000 7fcdc2302640 1 -- 172.21.15.77:0/3719133420 shutdown_connections 2024-03-06T07:31:19.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.907+0000 7fcdc2302640 1 -- 172.21.15.77:0/3719133420 wait complete. 2024-03-06T07:31:19.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.908+0000 7fcdc2302640 1 Processor -- start 2024-03-06T07:31:19.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.908+0000 7fcdc2302640 1 -- start start 2024-03-06T07:31:19.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdc2302640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc0cf830 0x7fcdbc15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdc2302640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 0x7fcdbc1f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdc2302640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdbc15dfe0 0x7fcdbc157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdc2302640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0d1df0 con 0x7fcdbc0cf830 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdc2302640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0d1c70 con 0x7fcdbc15dfe0 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdc2302640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc15e3c0 con 0x7fcdbc0d29c0 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc0cf830 0x7fcdbc15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc0cf830 0x7fcdbc15e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49600/0 (socket says 172.21.15.77:49600) 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbbfff640 1 -- 172.21.15.77:0/3900213811 learned_addr learned my addr 172.21.15.77:0/3900213811 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbb7fe640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 0x7fcdbc1f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdc089f640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdbc15dfe0 0x7fcdbc157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbbfff640 1 -- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 msgr2=0x7fcdbc1f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbbfff640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 0x7fcdbc1f7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbbfff640 1 -- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdbc15dfe0 msgr2=0x7fcdbc157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbbfff640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdbc15dfe0 0x7fcdbc157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbbfff640 1 -- 172.21.15.77:0/3900213811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc15f6c0 con 0x7fcdbc0cf830 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdbb7fe640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 0x7fcdbc1f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.909+0000 7fcdc089f640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdbc15dfe0 0x7fcdbc157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.910+0000 7fcdbbfff640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc0cf830 0x7fcdbc15e790 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00098c0 tx=0x7fcdb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.910+0000 7fcdb8ff9640 1 -- 172.21.15.77:0/3900213811 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0040070 con 0x7fcdbc0cf830 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.910+0000 7fcdb8ff9640 1 -- 172.21.15.77:0/3900213811 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb0039e70 con 0x7fcdbc0cf830 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.910+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc15ee20 con 0x7fcdbc0cf830 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.910+0000 7fcdb8ff9640 1 -- 172.21.15.77:0/3900213811 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0035040 con 0x7fcdbc0cf830 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.910+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc15f3e0 con 0x7fcdbc0cf830 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.911+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdbc04f330 con 0x7fcdbc0cf830 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.911+0000 7fcdb8ff9640 1 -- 172.21.15.77:0/3900213811 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdb004b060 con 0x7fcdbc0cf830 2024-03-06T07:31:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.912+0000 7fcdb8ff9640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcda806b500 0x7fcda806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.912+0000 7fcdbb7fe640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcda806b500 0x7fcda806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.912+0000 7fcdb8ff9640 1 -- 172.21.15.77:0/3900213811 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcdb00abdc0 con 0x7fcdbc0cf830 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.912+0000 7fcdbb7fe640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcda806b500 0x7fcda806d9c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc161590 tx=0x7fcdac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:19.916+0000 7fcdb8ff9640 1 -- 172.21.15.77:0/3900213811 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdb004a2c0 con 0x7fcdbc0cf830 2024-03-06T07:31:20.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.131+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdbc0d48a0 con 0x7fcdbc0cf830 2024-03-06T07:31:20.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.131+0000 7fcdb8ff9640 1 -- 172.21.15.77:0/3900213811 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3416 (secure 0 0 0) 0x7fcdb007acb0 con 0x7fcdbc0cf830 2024-03-06T07:31:20.133 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:20.133 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":105,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:30:53.247419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:20.134 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 105 2024-03-06T07:31:20.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcda806b500 msgr2=0x7fcda806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcda806b500 0x7fcda806d9c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc161590 tx=0x7fcdac009040 comp rx=0 tx=0).stop 2024-03-06T07:31:20.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc0cf830 msgr2=0x7fcdbc15e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc0cf830 0x7fcdbc15e790 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00098c0 tx=0x7fcdb00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 shutdown_connections 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcdbc15dfe0 0x7fcdbc157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcdbc0d29c0 0x7fcdbc1f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcda806b500 0x7fcda806d9c0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 --2- 172.21.15.77:0/3900213811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdbc0cf830 0x7fcdbc15e790 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 >> 172.21.15.77:0/3900213811 conn(0x7fcdbc0d3cd0 msgr2=0x7fcdbc0ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 shutdown_connections 2024-03-06T07:31:20.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:20.133+0000 7fcdc2302640 1 -- 172.21.15.77:0/3900213811 wait complete. 2024-03-06T07:31:20.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:30:53.247419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:20.351 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:21.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:21.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.454+0000 7f8dca840640 1 Processor -- start 2024-03-06T07:31:21.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.454+0000 7f8dca840640 1 -- start start 2024-03-06T07:31:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dca840640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc415dfe0 0x7f8dc40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dca840640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8dc40d29c0 0x7f8dc40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dca840640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc40b3430 con 0x7f8dc40cf830 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dca840640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc40b31e0 con 0x7f8dc40d29c0 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dca840640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40b3d60 con 0x7f8dc415dfe0 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc8ddd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8dc40d29c0 0x7f8dc40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc37fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8dc40cf830 0x7f8dc40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54502/0 (socket says 172.21.15.77:54502) 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc37fe640 1 -- 172.21.15.77:0/4217165244 learned_addr learned my addr 172.21.15.77:0/4217165244 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc8ddd640 1 -- 172.21.15.77:0/4217165244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc415dfe0 msgr2=0x7f8dc40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc3fff640 1 --2- 172.21.15.77:0/4217165244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc415dfe0 0x7f8dc40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc8ddd640 1 --2- 172.21.15.77:0/4217165244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc415dfe0 0x7f8dc40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc2ffd640 1 -- 172.21.15.77:0/4217165244 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789801356 0 0) 0x7f8dc40b3430 con 0x7f8dc40cf830 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc8ddd640 1 -- 172.21.15.77:0/4217165244 >> v1:172.21.15.77:6789/0 conn(0x7f8dc40cf830 legacy=0x7f8dc40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.455+0000 7f8dc8ddd640 1 -- 172.21.15.77:0/4217165244 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc40b3710 con 0x7f8dc40d29c0 2024-03-06T07:31:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.456+0000 7f8dc8ddd640 1 --2- 172.21.15.77:0/4217165244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40d29c0 0x7f8dc40b41f0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009d90 tx=0x7f8dbc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d6ee59c018efc00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.456+0000 7f8dc2ffd640 1 -- 172.21.15.77:0/4217165244 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc039070 con 0x7f8dc40d29c0 2024-03-06T07:31:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.456+0000 7f8dc2ffd640 1 -- 172.21.15.77:0/4217165244 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dbc02cc50 con 0x7f8dc40d29c0 2024-03-06T07:31:21.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.456+0000 7f8dc2ffd640 1 -- 172.21.15.77:0/4217165244 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc034440 con 0x7f8dc40d29c0 2024-03-06T07:31:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.457+0000 7f8dca840640 1 -- 172.21.15.77:0/4217165244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40d29c0 msgr2=0x7f8dc40b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.457+0000 7f8dca840640 1 --2- 172.21.15.77:0/4217165244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40d29c0 0x7f8dc40b41f0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009d90 tx=0x7f8dbc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.457+0000 7f8dca840640 1 -- 172.21.15.77:0/4217165244 shutdown_connections 2024-03-06T07:31:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.457+0000 7f8dca840640 1 --2- 172.21.15.77:0/4217165244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40d29c0 0x7f8dc40b41f0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.457+0000 7f8dca840640 1 --2- 172.21.15.77:0/4217165244 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc415dfe0 0x7f8dc40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.457+0000 7f8dca840640 1 -- 172.21.15.77:0/4217165244 >> 172.21.15.77:0/4217165244 conn(0x7f8dc40d3cd0 msgr2=0x7f8dc40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.457+0000 7f8dca840640 1 -- 172.21.15.77:0/4217165244 shutdown_connections 2024-03-06T07:31:21.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.457+0000 7f8dca840640 1 -- 172.21.15.77:0/4217165244 wait complete. 2024-03-06T07:31:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.458+0000 7f8dca840640 1 Processor -- start 2024-03-06T07:31:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.458+0000 7f8dca840640 1 -- start start 2024-03-06T07:31:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dca840640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40cf830 0x7f8dc40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dca840640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc40d29c0 0x7f8dc414d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dca840640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc415dfe0 0x7f8dc40cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dca840640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40d1df0 con 0x7f8dc40cf830 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dca840640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40d1c70 con 0x7f8dc415dfe0 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dca840640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc415e3c0 con 0x7f8dc40d29c0 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dc3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40cf830 0x7f8dc40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dc3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40cf830 0x7f8dc40b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49614/0 (socket says 172.21.15.77:49614) 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dc3fff640 1 -- 172.21.15.77:0/4122942329 learned_addr learned my addr 172.21.15.77:0/4122942329 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dc3fff640 1 -- 172.21.15.77:0/4122942329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc40d29c0 msgr2=0x7f8dc414d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dc3fff640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc40d29c0 0x7f8dc414d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dc3fff640 1 -- 172.21.15.77:0/4122942329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc415dfe0 msgr2=0x7f8dc40cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dc3fff640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc415dfe0 0x7f8dc40cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.459+0000 7f8dc3fff640 1 -- 172.21.15.77:0/4122942329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc414da50 con 0x7f8dc40cf830 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.460+0000 7f8dc3fff640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40cf830 0x7f8dc40b68d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f8db400c7f0 tx=0x7f8db400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.460+0000 7f8dc0ff9640 1 -- 172.21.15.77:0/4122942329 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4015070 con 0x7f8dc40cf830 2024-03-06T07:31:21.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.460+0000 7f8dc0ff9640 1 -- 172.21.15.77:0/4122942329 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db4004220 con 0x7f8dc40cf830 2024-03-06T07:31:21.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.460+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc414dd30 con 0x7f8dc40cf830 2024-03-06T07:31:21.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.460+0000 7f8dc0ff9640 1 -- 172.21.15.77:0/4122942329 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4007620 con 0x7f8dc40cf830 2024-03-06T07:31:21.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.460+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc414e2f0 con 0x7f8dc40cf830 2024-03-06T07:31:21.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.461+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc404f330 con 0x7f8dc40cf830 2024-03-06T07:31:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.461+0000 7f8dc0ff9640 1 -- 172.21.15.77:0/4122942329 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8db40079d0 con 0x7f8dc40cf830 2024-03-06T07:31:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.462+0000 7f8dc0ff9640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8db006b4d0 0x7f8db006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.462+0000 7f8dc37fe640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8db006b4d0 0x7f8db006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.462+0000 7f8dc0ff9640 1 -- 172.21.15.77:0/4122942329 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8db408a550 con 0x7f8dc40cf830 2024-03-06T07:31:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.462+0000 7f8dc37fe640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8db006b4d0 0x7f8db006d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8db8007fd0 tx=0x7f8db8007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.466+0000 7f8dc0ff9640 1 -- 172.21.15.77:0/4122942329 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8db4059440 con 0x7f8dc40cf830 2024-03-06T07:31:21.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.675+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dc406c290 con 0x7f8dc40cf830 2024-03-06T07:31:21.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.676+0000 7f8dc0ff9640 1 -- 172.21.15.77:0/4122942329 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f8db4058d40 con 0x7f8dc40cf830 2024-03-06T07:31:21.678 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:21.678 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:21.678 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8db006b4d0 msgr2=0x7f8db006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8db006b4d0 0x7f8db006d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8db8007fd0 tx=0x7f8db8007920 comp rx=0 tx=0).stop 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40cf830 msgr2=0x7f8dc40b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40cf830 0x7f8dc40b68d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f8db400c7f0 tx=0x7f8db400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 shutdown_connections 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8dc415dfe0 0x7f8dc40cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8dc40d29c0 0x7f8dc414d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8db006b4d0 0x7f8db006d990 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 --2- 172.21.15.77:0/4122942329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc40cf830 0x7f8dc40b68d0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.677+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 >> 172.21.15.77:0/4122942329 conn(0x7f8dc40d3cd0 msgr2=0x7f8dc40b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.678+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 shutdown_connections 2024-03-06T07:31:21.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:21.678+0000 7f8dca840640 1 -- 172.21.15.77:0/4122942329 wait complete. 2024-03-06T07:31:21.891 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:21.892 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:22.893 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:22.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.991+0000 7ff151a15640 1 Processor -- start 2024-03-06T07:31:22.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.991+0000 7ff151a15640 1 -- start start 2024-03-06T07:31:22.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff151a15640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff14c13dc10 0x7ff14c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff151a15640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c15f580 con 0x7ff14c15ef60 2024-03-06T07:31:22.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff151a15640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c15b450 con 0x7ff14c15bf30 2024-03-06T07:31:22.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff151a15640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0cf3f0 con 0x7ff14c13dc10 2024-03-06T07:31:22.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff14b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff14c15ef60 0x7ff14c0708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54516/0 (socket says 172.21.15.77:54516) 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff14b7fe640 1 -- 172.21.15.77:0/3493548505 learned_addr learned my addr 172.21.15.77:0/3493548505 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270209872 0 0) 0x7ff14c15f580 con 0x7ff14c15ef60 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff14affd640 1 --2- 172.21.15.77:0/3493548505 >> v2:172.21.15.158:3300/0 conn(0x7ff14c13dc10 0x7ff14c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.992+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134003520 con 0x7ff14c15ef60 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1368722454 0 0) 0x7ff134003520 con 0x7ff14c15ef60 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 >> v2:172.21.15.158:3300/0 conn(0x7ff14c13dc10 msgr2=0x7ff14c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 --2- 172.21.15.77:0/3493548505 >> v2:172.21.15.158:3300/0 conn(0x7ff14c13dc10 0x7ff14c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 >> v1:172.21.15.158:6789/0 conn(0x7ff14c15bf30 legacy=0x7ff14c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c15b730 con 0x7ff14c15ef60 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7ff1440019f0 con 0x7ff14c15ef60 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 >> v1:172.21.15.77:6789/0 conn(0x7ff14c15ef60 legacy=0x7ff14c0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1340039b0 0x7ff134003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1340051c0 0x7ff13400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0cf3f0 con 0x7ff134004800 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13400c160 con 0x7ff1340039b0 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13400c360 con 0x7ff1340051c0 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14a7fc640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14b7fe640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1340051c0 0x7ff13400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14affd640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1340039b0 0x7ff134003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14a7fc640 1 -- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1340051c0 msgr2=0x7ff13400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14a7fc640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff1340051c0 0x7ff13400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14a7fc640 1 -- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1340039b0 msgr2=0x7ff134003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14a7fc640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1340039b0 0x7ff134003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14b7fe640 1 -- 172.21.15.77:0/3493548505 reap_dead start 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14a7fc640 1 -- 172.21.15.77:0/3493548505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff134003520 con 0x7ff134004800 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.993+0000 7ff14affd640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff1340039b0 0x7ff134003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.994+0000 7ff14a7fc640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff134004c80 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7ff140001750 tx=0x7ff134005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0cf89c92fc0b1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.994+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140001370 con 0x7ff134004800 2024-03-06T07:31:23.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.994+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1400052d0 con 0x7ff134004800 2024-03-06T07:31:23.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.994+0000 7ff149ffb640 1 -- 172.21.15.77:0/3493548505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140003830 con 0x7ff134004800 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.995+0000 7ff151a15640 1 -- 172.21.15.77:0/3493548505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 msgr2=0x7ff134004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.995+0000 7ff151a15640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff134004c80 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7ff140001750 tx=0x7ff134005830 comp rx=0 tx=0).stop 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.995+0000 7ff151a15640 1 -- 172.21.15.77:0/3493548505 shutdown_connections 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.995+0000 7ff151a15640 1 --2- 172.21.15.77:0/3493548505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff134004c80 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.995+0000 7ff151a15640 1 -- 172.21.15.77:0/3493548505 >> 172.21.15.77:0/3493548505 conn(0x7ff14c0cf8a0 msgr2=0x7ff14c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.995+0000 7ff151a15640 1 -- 172.21.15.77:0/3493548505 shutdown_connections 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.995+0000 7ff151a15640 1 -- 172.21.15.77:0/3493548505 wait complete. 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.996+0000 7ff151a15640 1 Processor -- start 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.996+0000 7ff151a15640 1 -- start start 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff151a15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff14c0ae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff151a15640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff14c1419b0 0x7ff14c13ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff151a15640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c143d90 0x7ff14c141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff151a15640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c15e850 con 0x7ff134004800 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff151a15640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c15b210 con 0x7ff14c143d90 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff151a15640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c15e9d0 con 0x7ff14c1419b0 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff14c0ae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff14c0ae9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49632/0 (socket says 172.21.15.77:49632) 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14affd640 1 -- 172.21.15.77:0/1459556756 learned_addr learned my addr 172.21.15.77:0/1459556756 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14b7fe640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c143d90 0x7ff14c141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14a7fc640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff14c1419b0 0x7ff14c13ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14affd640 1 -- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff14c1419b0 msgr2=0x7ff14c13ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14affd640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff14c1419b0 0x7ff14c13ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14affd640 1 -- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c143d90 msgr2=0x7ff14c141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14affd640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c143d90 0x7ff14c141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.997+0000 7ff14affd640 1 -- 172.21.15.77:0/1459556756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0ad900 con 0x7ff134004800 2024-03-06T07:31:23.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.998+0000 7ff14a7fc640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff14c1419b0 0x7ff14c13ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.998+0000 7ff14affd640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff14c0ae9a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0114e0 tx=0x7ff13c0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.998+0000 7ff133fff640 1 -- 172.21.15.77:0/1459556756 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c015870 con 0x7ff134004800 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.998+0000 7ff133fff640 1 -- 172.21.15.77:0/1459556756 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c015a10 con 0x7ff134004800 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.998+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c0adbe0 con 0x7ff134004800 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.998+0000 7ff133fff640 1 -- 172.21.15.77:0/1459556756 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c015e00 con 0x7ff134004800 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.998+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c0ae1a0 con 0x7ff134004800 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.999+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c04f330 con 0x7ff134004800 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:22.999+0000 7ff133fff640 1 -- 172.21.15.77:0/1459556756 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff13c01d400 con 0x7ff134004800 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.000+0000 7ff133fff640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff13806b4d0 0x7ff13806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.000+0000 7ff133fff640 1 -- 172.21.15.77:0/1459556756 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff13c08aa00 con 0x7ff134004800 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.000+0000 7ff14a7fc640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff13806b4d0 0x7ff13806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.000+0000 7ff14a7fc640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff13806b4d0 0x7ff13806d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff140033000 tx=0x7ff140009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.004+0000 7ff133fff640 1 -- 172.21.15.77:0/1459556756 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff13c019030 con 0x7ff134004800 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.216+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff14c06eb00 con 0x7ff134004800 2024-03-06T07:31:23.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.217+0000 7ff133fff640 1 -- 172.21.15.77:0/1459556756 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff13c0598f0 con 0x7ff134004800 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:23.220 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:23.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff13806b4d0 msgr2=0x7ff13806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff13806b4d0 0x7ff13806d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff140033000 tx=0x7ff140009030 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 msgr2=0x7ff14c0ae9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff14c0ae9a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0114e0 tx=0x7ff13c0119a0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 shutdown_connections 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff14c143d90 0x7ff14c141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff14c1419b0 0x7ff14c13ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff13806b4d0 0x7ff13806d990 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 --2- 172.21.15.77:0/1459556756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff134004800 0x7ff14c0ae9a0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.218+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 >> 172.21.15.77:0/1459556756 conn(0x7ff14c0cf8a0 msgr2=0x7ff14c072280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.219+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 shutdown_connections 2024-03-06T07:31:23.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:23.219+0000 7ff151a15640 1 -- 172.21.15.77:0/1459556756 wait complete. 2024-03-06T07:31:23.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:23.434 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:24.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:24.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.534+0000 7f915a93e640 1 Processor -- start 2024-03-06T07:31:24.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.534+0000 7f915a93e640 1 -- start start 2024-03-06T07:31:24.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f915a93e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f915415dfe0 0x7f91540cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f915a93e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f91540d29c0 0x7f91540b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f915a93e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91540b3430 con 0x7f91540cf830 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f915a93e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91540b31e0 con 0x7f915415dfe0 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f915a93e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91540b3d60 con 0x7f91540d29c0 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f9153fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f915415dfe0 0x7f91540cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f9153fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f915415dfe0 0x7f91540cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49644/0 (socket says 172.21.15.77:49644) 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f9153fff640 1 -- 172.21.15.77:0/220806872 learned_addr learned my addr 172.21.15.77:0/220806872 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f9153fff640 1 -- 172.21.15.77:0/220806872 >> v2:172.21.15.158:3300/0 conn(0x7f91540d29c0 msgr2=0x7f91540b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f9153fff640 1 --2- 172.21.15.77:0/220806872 >> v2:172.21.15.158:3300/0 conn(0x7f91540d29c0 0x7f91540b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f9153fff640 1 -- 172.21.15.77:0/220806872 >> v1:172.21.15.158:6789/0 conn(0x7f91540cf830 legacy=0x7f91540d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.535+0000 7f9153fff640 1 -- 172.21.15.77:0/220806872 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540b3710 con 0x7f915415dfe0 2024-03-06T07:31:24.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.536+0000 7f9153fff640 1 --2- 172.21.15.77:0/220806872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f915415dfe0 0x7f91540cf2f0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f91540b3cd0 tx=0x7f914802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adda11f8d83fe67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.536+0000 7f9152ffd640 1 -- 172.21.15.77:0/220806872 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9148035020 con 0x7f915415dfe0 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.536+0000 7f9152ffd640 1 -- 172.21.15.77:0/220806872 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9148031870 con 0x7f915415dfe0 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.536+0000 7f9152ffd640 1 -- 172.21.15.77:0/220806872 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9148031c40 con 0x7f915415dfe0 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.537+0000 7f915a93e640 1 -- 172.21.15.77:0/220806872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f915415dfe0 msgr2=0x7f91540cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.537+0000 7f915a93e640 1 --2- 172.21.15.77:0/220806872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f915415dfe0 0x7f91540cf2f0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f91540b3cd0 tx=0x7f914802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.537+0000 7f915a93e640 1 -- 172.21.15.77:0/220806872 shutdown_connections 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.537+0000 7f915a93e640 1 --2- 172.21.15.77:0/220806872 >> v2:172.21.15.158:3300/0 conn(0x7f91540d29c0 0x7f91540b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.537+0000 7f915a93e640 1 --2- 172.21.15.77:0/220806872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f915415dfe0 0x7f91540cf2f0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.537+0000 7f915a93e640 1 -- 172.21.15.77:0/220806872 >> 172.21.15.77:0/220806872 conn(0x7f91540d3cd0 msgr2=0x7f91540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.537+0000 7f915a93e640 1 -- 172.21.15.77:0/220806872 shutdown_connections 2024-03-06T07:31:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.537+0000 7f915a93e640 1 -- 172.21.15.77:0/220806872 wait complete. 2024-03-06T07:31:24.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.538+0000 7f915a93e640 1 Processor -- start 2024-03-06T07:31:24.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.538+0000 7f915a93e640 1 -- start start 2024-03-06T07:31:24.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f915a93e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91540cf830 0x7f91541f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f915a93e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91540d29c0 0x7f91541f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f915a93e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f915415dfe0 0x7f9154152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f915a93e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91540d1df0 con 0x7f91540cf830 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f915a93e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91540d1c70 con 0x7f915415dfe0 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f915a93e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915415e3c0 con 0x7f91540d29c0 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91540cf830 0x7f91541f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91540cf830 0x7f91541f71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49656/0 (socket says 172.21.15.77:49656) 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 -- 172.21.15.77:0/2467945801 learned_addr learned my addr 172.21.15.77:0/2467945801 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9158edb640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f915415dfe0 0x7f9154152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f91537fe640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91540d29c0 0x7f91541f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 -- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91540d29c0 msgr2=0x7f91541f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91540d29c0 0x7f91541f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 -- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f915415dfe0 msgr2=0x7f9154152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f915415dfe0 0x7f9154152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 -- 172.21.15.77:0/2467945801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9154152690 con 0x7f91540cf830 2024-03-06T07:31:24.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f91537fe640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91540d29c0 0x7f91541f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.539+0000 7f9153fff640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91540cf830 0x7f91541f71c0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f914802b950 tx=0x7f91480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.540+0000 7f9150ff9640 1 -- 172.21.15.77:0/2467945801 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914803ad60 con 0x7f91540cf830 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.540+0000 7f9150ff9640 1 -- 172.21.15.77:0/2467945801 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91480443f0 con 0x7f91540cf830 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.540+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91541f78a0 con 0x7f91540cf830 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.540+0000 7f9150ff9640 1 -- 172.21.15.77:0/2467945801 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91480447e0 con 0x7f91540cf830 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.540+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91541f7e60 con 0x7f91540cf830 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.541+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91541f80a0 con 0x7f91540cf830 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.541+0000 7f9150ff9640 1 -- 172.21.15.77:0/2467945801 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9148038020 con 0x7f91540cf830 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.542+0000 7f9150ff9640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f914006b4d0 0x7f914006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.542+0000 7f91537fe640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f914006b4d0 0x7f914006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.542+0000 7f9150ff9640 1 -- 172.21.15.77:0/2467945801 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f91480ac410 con 0x7f91540cf830 2024-03-06T07:31:24.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.542+0000 7f91537fe640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f914006b4d0 0x7f914006d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9144007fd0 tx=0x7f9144009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.546+0000 7f9150ff9640 1 -- 172.21.15.77:0/2467945801 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91541f80a0 con 0x7f91540cf830 2024-03-06T07:31:24.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.760+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f915404f330 con 0x7f91540cf830 2024-03-06T07:31:24.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.760+0000 7f9150ff9640 1 -- 172.21.15.77:0/2467945801 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f9154152690 con 0x7f91540cf830 2024-03-06T07:31:24.763 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:24.763 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:24.764 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:24.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f914006b4d0 msgr2=0x7f914006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f914006b4d0 0x7f914006d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9144007fd0 tx=0x7f9144009040 comp rx=0 tx=0).stop 2024-03-06T07:31:24.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91540cf830 msgr2=0x7f91541f71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91540cf830 0x7f91541f71c0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f914802b950 tx=0x7f91480046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 shutdown_connections 2024-03-06T07:31:24.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f915415dfe0 0x7f9154152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f91540d29c0 0x7f91541f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f914006b4d0 0x7f914006d990 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 --2- 172.21.15.77:0/2467945801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91540cf830 0x7f91541f71c0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 >> 172.21.15.77:0/2467945801 conn(0x7f91540d3cd0 msgr2=0x7f915406cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 shutdown_connections 2024-03-06T07:31:24.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:24.762+0000 7f915a93e640 1 -- 172.21.15.77:0/2467945801 wait complete. 2024-03-06T07:31:24.977 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:24.977 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:25.978 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.084+0000 7fcb80815640 1 Processor -- start 2024-03-06T07:31:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.084+0000 7fcb80815640 1 -- start start 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.084+0000 7fcb80815640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c15dfe0 0x7fcb7c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.084+0000 7fcb80815640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb7c0cf830 0x7fcb7c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.084+0000 7fcb80815640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c0b3430 con 0x7fcb7c0d27f0 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.084+0000 7fcb80815640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c0d2c50 con 0x7fcb7c0cf830 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.084+0000 7fcb80815640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0b31e0 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb79d9b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb7c0cf830 0x7fcb7c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb7a59c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c15dfe0 0x7fcb7c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb7ad9d640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fcb7c0d27f0 0x7fcb7c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:49876/0 (socket says 172.21.15.77:49876) 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb79d9b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcb7c0cf830 0x7fcb7c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:49990/0 (socket says 172.21.15.77:49990) 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb7ad9d640 1 -- 172.21.15.77:0/1210120808 learned_addr learned my addr 172.21.15.77:0/1210120808 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:26.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb79d9b640 1 -- 172.21.15.77:0/1210120808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c15dfe0 msgr2=0x7fcb7c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb79d9b640 1 --2- 172.21.15.77:0/1210120808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c15dfe0 0x7fcb7c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb79d9b640 1 -- 172.21.15.77:0/1210120808 >> v1:172.21.15.158:6789/0 conn(0x7fcb7c0d27f0 legacy=0x7fcb7c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb7959a640 1 -- 172.21.15.77:0/1210120808 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417057698 0 0) 0x7fcb7c0b3430 con 0x7fcb7c0d27f0 2024-03-06T07:31:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb79d9b640 1 -- 172.21.15.77:0/1210120808 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0b3710 con 0x7fcb7c0cf830 2024-03-06T07:31:26.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.085+0000 7fcb7a59c640 1 --2- 172.21.15.77:0/1210120808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c15dfe0 0x7fcb7c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.086+0000 7fcb79d9b640 1 --2- 172.21.15.77:0/1210120808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0cf830 0x7fcb7c0d22b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcb700097a0 tx=0x7fcb7002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48848c1349ff8779 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb7959a640 1 -- 172.21.15.77:0/1210120808 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70035020 con 0x7fcb7c0cf830 2024-03-06T07:31:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb7959a640 1 -- 172.21.15.77:0/1210120808 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb70031830 con 0x7fcb7c0cf830 2024-03-06T07:31:26.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb7959a640 1 -- 172.21.15.77:0/1210120808 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70031c00 con 0x7fcb7c0cf830 2024-03-06T07:31:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb80815640 1 -- 172.21.15.77:0/1210120808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0cf830 msgr2=0x7fcb7c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb80815640 1 --2- 172.21.15.77:0/1210120808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0cf830 0x7fcb7c0d22b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcb700097a0 tx=0x7fcb7002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb80815640 1 -- 172.21.15.77:0/1210120808 shutdown_connections 2024-03-06T07:31:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb80815640 1 --2- 172.21.15.77:0/1210120808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0cf830 0x7fcb7c0d22b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb80815640 1 --2- 172.21.15.77:0/1210120808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c15dfe0 0x7fcb7c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb80815640 1 -- 172.21.15.77:0/1210120808 >> 172.21.15.77:0/1210120808 conn(0x7fcb7c0d3cd0 msgr2=0x7fcb7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb80815640 1 -- 172.21.15.77:0/1210120808 shutdown_connections 2024-03-06T07:31:26.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.087+0000 7fcb80815640 1 -- 172.21.15.77:0/1210120808 wait complete. 2024-03-06T07:31:26.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb80815640 1 Processor -- start 2024-03-06T07:31:26.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb80815640 1 -- start start 2024-03-06T07:31:26.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb80815640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c0cf830 0x7fcb7c161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb80815640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0d27f0 0x7fcb7c15e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb80815640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb7c15dfe0 0x7fcb7c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb80815640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0d1c70 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb80815640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0b3d60 con 0x7fcb7c0d27f0 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb80815640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0d1df0 con 0x7fcb7c0cf830 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb7ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb7c15dfe0 0x7fcb7c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb7ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb7c15dfe0 0x7fcb7c15ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49672/0 (socket says 172.21.15.77:49672) 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.089+0000 7fcb7ad9d640 1 -- 172.21.15.77:0/1100678360 learned_addr learned my addr 172.21.15.77:0/1100678360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb79d9b640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0d27f0 0x7fcb7c15e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb7a59c640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c0cf830 0x7fcb7c161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb7ad9d640 1 -- 172.21.15.77:0/1100678360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c0cf830 msgr2=0x7fcb7c161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb7ad9d640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c0cf830 0x7fcb7c161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb7ad9d640 1 -- 172.21.15.77:0/1100678360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0d27f0 msgr2=0x7fcb7c15e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb7ad9d640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0d27f0 0x7fcb7c15e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb7ad9d640 1 -- 172.21.15.77:0/1100678360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0b6c90 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb7ad9d640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb7c15dfe0 0x7fcb7c15ed10 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400c7a0 tx=0x7fcb7400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb6affd640 1 -- 172.21.15.77:0/1100678360 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74018070 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb6affd640 1 -- 172.21.15.77:0/1100678360 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7400a640 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c0cc360 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb6affd640 1 -- 172.21.15.77:0/1100678360 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7400aab0 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.090+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c0cc920 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.091+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c04f330 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.092+0000 7fcb6affd640 1 -- 172.21.15.77:0/1100678360 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb7401b070 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.092+0000 7fcb6affd640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.092+0000 7fcb6affd640 1 -- 172.21.15.77:0/1100678360 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcb740897b0 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.092+0000 7fcb7a59c640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.093+0000 7fcb7a59c640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c1603b0 tx=0x7fcb6c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.096+0000 7fcb6affd640 1 -- 172.21.15.77:0/1100678360 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb74057fc0 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.345+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb7c15d490 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.345+0000 7fcb6affd640 1 -- 172.21.15.77:0/1100678360 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fcb7405c060 con 0x7fcb7c15dfe0 2024-03-06T07:31:26.347 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:26.347 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:26.348 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 msgr2=0x7fcb5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c1603b0 tx=0x7fcb6c009190 comp rx=0 tx=0).stop 2024-03-06T07:31:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb7c15dfe0 msgr2=0x7fcb7c15ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb7c15dfe0 0x7fcb7c15ed10 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400c7a0 tx=0x7fcb7400cc60 comp rx=0 tx=0).stop 2024-03-06T07:31:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 shutdown_connections 2024-03-06T07:31:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb7c15dfe0 0x7fcb7c15ed10 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7c0d27f0 0x7fcb7c15e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 --2- 172.21.15.77:0/1100678360 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb7c0cf830 0x7fcb7c161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 >> 172.21.15.77:0/1100678360 conn(0x7fcb7c0d3cd0 msgr2=0x7fcb7c06bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 shutdown_connections 2024-03-06T07:31:26.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:26.347+0000 7fcb80815640 1 -- 172.21.15.77:0/1100678360 wait complete. 2024-03-06T07:31:26.561 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:26.561 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:27.563 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:27.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.661+0000 7f5ad386f640 1 Processor -- start 2024-03-06T07:31:27.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.661+0000 7f5ad386f640 1 -- start start 2024-03-06T07:31:27.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.662+0000 7f5ad386f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5acc15dfe0 0x7f5acc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.662+0000 7f5ad386f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc0d29c0 0x7f5acc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.662+0000 7f5ad386f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc0b3430 con 0x7f5acc0cf830 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5ad386f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5acc0b31e0 con 0x7f5acc15dfe0 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5ad386f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0b3d60 con 0x7f5acc0d29c0 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5ad0e0a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5acc0cf830 0x7f5acc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54524/0 (socket says 172.21.15.77:54524) 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5ad0e0a640 1 -- 172.21.15.77:0/745653708 learned_addr learned my addr 172.21.15.77:0/745653708 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5ad160b640 1 --2- 172.21.15.77:0/745653708 >> v2:172.21.15.158:3300/0 conn(0x7f5acc15dfe0 0x7f5acc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5ad1e0c640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc0d29c0 0x7f5acc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263660451 0 0) 0x7f5acc0b3430 con 0x7f5acc0cf830 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab8003520 con 0x7f5acc0cf830 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2009064087 0 0) 0x7f5ab8003520 con 0x7f5acc0cf830 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc0d29c0 msgr2=0x7f5acc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc0d29c0 0x7f5acc0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 >> v2:172.21.15.158:3300/0 conn(0x7f5acc15dfe0 msgr2=0x7f5acc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 --2- 172.21.15.77:0/745653708 >> v2:172.21.15.158:3300/0 conn(0x7f5acc15dfe0 0x7f5acc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc0b3710 con 0x7f5acc0cf830 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f5ac8001670 con 0x7f5acc0cf830 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 >> v1:172.21.15.77:6789/0 conn(0x7f5acc0cf830 legacy=0x7f5acc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5ab8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab8004800 0x7f5ab8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc0d29c0 0x7f5ab80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0b31e0 con 0x7f5ab80039b0 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0b3d60 con 0x7f5ab8004800 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab800b9c0 con 0x7f5acc0d29c0 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.663+0000 7f5ad160b640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5ab8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.664+0000 7f5ad1e0c640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc0d29c0 0x7f5ab80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.664+0000 7f5ad0e0a640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab8004800 0x7f5ab8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.664+0000 7f5ad160b640 1 -- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc0d29c0 msgr2=0x7f5ab80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.664+0000 7f5ad160b640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc0d29c0 0x7f5ab80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.664+0000 7f5ad160b640 1 -- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab8004800 msgr2=0x7f5ab8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.664+0000 7f5ad160b640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5ab8004800 0x7f5ab8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.664+0000 7f5ad160b640 1 -- 172.21.15.77:0/745653708 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab8003520 con 0x7f5ab80039b0 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.664+0000 7f5ad160b640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5ab8003dd0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009ac0 tx=0x7f5ab8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0a0be55f01a518 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0039070 con 0x7f5ab80039b0 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac002c710 con 0x7f5ab80039b0 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5abffff640 1 -- 172.21.15.77:0/745653708 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0009690 con 0x7f5ab80039b0 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5ad386f640 1 -- 172.21.15.77:0/745653708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 msgr2=0x7f5ab8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5ad386f640 1 --2- 172.21.15.77:0/745653708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5ab8003dd0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009ac0 tx=0x7f5ab8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5ad1e0c640 1 -- 172.21.15.77:0/745653708 reap_dead start 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5ad386f640 1 -- 172.21.15.77:0/745653708 shutdown_connections 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5ad386f640 1 -- 172.21.15.77:0/745653708 >> 172.21.15.77:0/745653708 conn(0x7f5acc0d3cd0 msgr2=0x7f5acc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5ad386f640 1 -- 172.21.15.77:0/745653708 shutdown_connections 2024-03-06T07:31:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.665+0000 7f5ad386f640 1 -- 172.21.15.77:0/745653708 wait complete. 2024-03-06T07:31:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.666+0000 7f5ad386f640 1 Processor -- start 2024-03-06T07:31:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.666+0000 7f5ad386f640 1 -- start start 2024-03-06T07:31:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad386f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5acc0cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad386f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc152cc0 0x7f5acc153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad386f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5acc0b6940 0x7f5acc0b6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad386f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0d1df0 con 0x7f5ab80039b0 2024-03-06T07:31:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad386f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0d1c70 con 0x7f5acc0b6940 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad386f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc15e3c0 con 0x7f5acc152cc0 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad160b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5acc0cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad160b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5acc0cd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49686/0 (socket says 172.21.15.77:49686) 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad160b640 1 -- 172.21.15.77:0/3272026143 learned_addr learned my addr 172.21.15.77:0/3272026143 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad1e0c640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5acc0b6940 0x7f5acc0b6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad0e0a640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc152cc0 0x7f5acc153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad160b640 1 -- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc152cc0 msgr2=0x7f5acc153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad160b640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc152cc0 0x7f5acc153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad160b640 1 -- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5acc0b6940 msgr2=0x7f5acc0b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.667+0000 7f5ad160b640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5acc0b6940 0x7f5acc0b6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5ad160b640 1 -- 172.21.15.77:0/3272026143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc153660 con 0x7f5ab80039b0 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5ad1e0c640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5acc0b6940 0x7f5acc0b6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5ad0e0a640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc152cc0 0x7f5acc153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5ad160b640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5acc0cd6c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0000c00 tx=0x7f5ac0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5abdffb640 1 -- 172.21.15.77:0/3272026143 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0039050 con 0x7f5ab80039b0 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5abdffb640 1 -- 172.21.15.77:0/3272026143 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac0045070 con 0x7f5ab80039b0 2024-03-06T07:31:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5abdffb640 1 -- 172.21.15.77:0/3272026143 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00044e0 con 0x7f5ab80039b0 2024-03-06T07:31:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5acc0b76b0 con 0x7f5ab80039b0 2024-03-06T07:31:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.668+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5acc0caf40 con 0x7f5ab80039b0 2024-03-06T07:31:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.669+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc0b7bd0 con 0x7f5ab80039b0 2024-03-06T07:31:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.669+0000 7f5abdffb640 1 -- 172.21.15.77:0/3272026143 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ac0003a50 con 0x7f5ab80039b0 2024-03-06T07:31:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.670+0000 7f5abdffb640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.670+0000 7f5ad0e0a640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.670+0000 7f5abdffb640 1 -- 172.21.15.77:0/3272026143 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5ac00abd20 con 0x7f5ab80039b0 2024-03-06T07:31:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.670+0000 7f5ad0e0a640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ab006b4d0 0x7f5ab006d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0b51d0 tx=0x7f5ac8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.674+0000 7f5abdffb640 1 -- 172.21.15.77:0/3272026143 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5acc0b7bd0 con 0x7f5ab80039b0 2024-03-06T07:31:27.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.890+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5acc04f330 con 0x7f5ab80039b0 2024-03-06T07:31:27.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.891+0000 7f5abdffb640 1 -- 172.21.15.77:0/3272026143 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f5ac004e020 con 0x7f5ab80039b0 2024-03-06T07:31:27.893 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:27.893 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:27.893 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ab006b4d0 msgr2=0x7f5ab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ab006b4d0 0x7f5ab006d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0b51d0 tx=0x7f5ac8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 msgr2=0x7f5acc0cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5acc0cd6c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0000c00 tx=0x7f5ac0004730 comp rx=0 tx=0).stop 2024-03-06T07:31:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 shutdown_connections 2024-03-06T07:31:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5acc0b6940 0x7f5acc0b6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5acc152cc0 0x7f5acc153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5ab006b4d0 0x7f5ab006d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 --2- 172.21.15.77:0/3272026143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab80039b0 0x7f5acc0cd6c0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 >> 172.21.15.77:0/3272026143 conn(0x7f5acc0d3cd0 msgr2=0x7f5acc0b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 shutdown_connections 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:27.894+0000 7f5ad386f640 1 -- 172.21.15.77:0/3272026143 wait complete. 2024-03-06T07:31:28.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:28.108 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:29.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.212+0000 7fd3e018a640 1 Processor -- start 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3e018a640 1 -- start start 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3e018a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3d815dfe0 0x7fd3d80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3e018a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d29c0 0x7fd3d80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3e018a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80b3430 con 0x7fd3d80cf830 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3e018a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d80b31e0 con 0x7fd3d815dfe0 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3e018a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80b3d60 con 0x7fd3d80d29c0 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3ddf26640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3d815dfe0 0x7fd3d80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3ddf26640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd3d815dfe0 0x7fd3d80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48608/0 (socket says 172.21.15.77:48608) 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.213+0000 7fd3ddf26640 1 -- 172.21.15.77:0/1480210263 learned_addr learned my addr 172.21.15.77:0/1480210263 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:29.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.214+0000 7fd3de727640 1 --2- 172.21.15.77:0/1480210263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d29c0 0x7fd3d80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.214+0000 7fd3ddf26640 1 -- 172.21.15.77:0/1480210263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d29c0 msgr2=0x7fd3d80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.214+0000 7fd3ddf26640 1 --2- 172.21.15.77:0/1480210263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d29c0 0x7fd3d80b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.214+0000 7fd3ddf26640 1 -- 172.21.15.77:0/1480210263 >> v1:172.21.15.158:6789/0 conn(0x7fd3d80cf830 legacy=0x7fd3d80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.214+0000 7fd3ddf26640 1 -- 172.21.15.77:0/1480210263 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d80b3710 con 0x7fd3d815dfe0 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.214+0000 7fd3ddf26640 1 --2- 172.21.15.77:0/1480210263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815dfe0 0x7fd3d80cf2f0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009870 tx=0x7fd3d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e7c6464d78d2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3dcf24640 1 -- 172.21.15.77:0/1480210263 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4035020 con 0x7fd3d815dfe0 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3dcf24640 1 -- 172.21.15.77:0/1480210263 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d4031640 con 0x7fd3d815dfe0 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3dcf24640 1 -- 172.21.15.77:0/1480210263 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d402d040 con 0x7fd3d815dfe0 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3e018a640 1 -- 172.21.15.77:0/1480210263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815dfe0 msgr2=0x7fd3d80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3e018a640 1 --2- 172.21.15.77:0/1480210263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815dfe0 0x7fd3d80cf2f0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009870 tx=0x7fd3d402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3e018a640 1 -- 172.21.15.77:0/1480210263 shutdown_connections 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3e018a640 1 --2- 172.21.15.77:0/1480210263 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d80d29c0 0x7fd3d80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3e018a640 1 --2- 172.21.15.77:0/1480210263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d815dfe0 0x7fd3d80cf2f0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3e018a640 1 -- 172.21.15.77:0/1480210263 >> 172.21.15.77:0/1480210263 conn(0x7fd3d80d3cd0 msgr2=0x7fd3d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3e018a640 1 -- 172.21.15.77:0/1480210263 shutdown_connections 2024-03-06T07:31:29.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.215+0000 7fd3e018a640 1 -- 172.21.15.77:0/1480210263 wait complete. 2024-03-06T07:31:29.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.217+0000 7fd3e018a640 1 Processor -- start 2024-03-06T07:31:29.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.217+0000 7fd3e018a640 1 -- start start 2024-03-06T07:31:29.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3e018a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d80cf830 0x7fd3d81f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3e018a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d80d29c0 0x7fd3d80b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3e018a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d815dfe0 0x7fd3d80b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3e018a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80d1df0 con 0x7fd3d80d29c0 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3e018a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80d1c70 con 0x7fd3d80cf830 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3e018a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d815e3c0 con 0x7fd3d815dfe0 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3dd725640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d80d29c0 0x7fd3d80b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3dd725640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d80d29c0 0x7fd3d80b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48616/0 (socket says 172.21.15.77:48616) 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.218+0000 7fd3dd725640 1 -- 172.21.15.77:0/288743464 learned_addr learned my addr 172.21.15.77:0/288743464 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3ddf26640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d80cf830 0x7fd3d81f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3de727640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d815dfe0 0x7fd3d80b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3dd725640 1 -- 172.21.15.77:0/288743464 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d815dfe0 msgr2=0x7fd3d80b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3dd725640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d815dfe0 0x7fd3d80b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3dd725640 1 -- 172.21.15.77:0/288743464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d80cf830 msgr2=0x7fd3d81f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3dd725640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d80cf830 0x7fd3d81f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3dd725640 1 -- 172.21.15.77:0/288743464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d81f37f0 con 0x7fd3d80d29c0 2024-03-06T07:31:29.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3dd725640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d80d29c0 0x7fd3d80b0f40 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800c400 tx=0x7fd3c800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/288743464 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8011790 con 0x7fd3d80d29c0 2024-03-06T07:31:29.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/288743464 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c8017070 con 0x7fd3d80d29c0 2024-03-06T07:31:29.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d80b1ae0 con 0x7fd3d80d29c0 2024-03-06T07:31:29.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/288743464 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8011bb0 con 0x7fd3d80d29c0 2024-03-06T07:31:29.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.219+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d815ecc0 con 0x7fd3d80d29c0 2024-03-06T07:31:29.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.220+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d804f330 con 0x7fd3d80d29c0 2024-03-06T07:31:29.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.221+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/288743464 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3c801c460 con 0x7fd3d80d29c0 2024-03-06T07:31:29.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.221+0000 7fd3ce7fc640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.221+0000 7fd3ddf26640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.221+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/288743464 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd3c8089ee0 con 0x7fd3d80d29c0 2024-03-06T07:31:29.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.222+0000 7fd3ddf26640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4036000 tx=0x7fd3d403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.225+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/288743464 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3d81f37f0 con 0x7fd3d80d29c0 2024-03-06T07:31:29.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.443+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3d80ca910 con 0x7fd3d80d29c0 2024-03-06T07:31:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.444+0000 7fd3ce7fc640 1 -- 172.21.15.77:0/288743464 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd3c80586f0 con 0x7fd3d80d29c0 2024-03-06T07:31:29.446 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:29.446 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:29.446 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:29.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 msgr2=0x7fd3bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4036000 tx=0x7fd3d403c040 comp rx=0 tx=0).stop 2024-03-06T07:31:29.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d80d29c0 msgr2=0x7fd3d80b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d80d29c0 0x7fd3d80b0f40 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800c400 tx=0x7fd3c800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 shutdown_connections 2024-03-06T07:31:29.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3d80d29c0 0x7fd3d80b0f40 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd3d815dfe0 0x7fd3d80b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 --2- 172.21.15.77:0/288743464 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3d80cf830 0x7fd3d81f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 >> 172.21.15.77:0/288743464 conn(0x7fd3d80d3cd0 msgr2=0x7fd3d80b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 shutdown_connections 2024-03-06T07:31:29.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:29.446+0000 7fd3e018a640 1 -- 172.21.15.77:0/288743464 wait complete. 2024-03-06T07:31:29.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:29.663 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:30.665 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:30.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.765+0000 7f882829d640 1 Processor -- start 2024-03-06T07:31:30.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.765+0000 7f882829d640 1 -- start start 2024-03-06T07:31:30.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882829d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f88200d2710 0x7f88200b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882829d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88200b3430 con 0x7f882015dfe0 2024-03-06T07:31:30.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882829d640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88200b3710 con 0x7f88200d22b0 2024-03-06T07:31:30.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882829d640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88200b31e0 con 0x7f88200d2710 2024-03-06T07:31:30.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882683a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f88200d2710 0x7f88200b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882683a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f88200d2710 0x7f88200b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48632/0 (socket says 172.21.15.77:48632) 2024-03-06T07:31:30.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882683a640 1 -- 172.21.15.77:0/4220020134 learned_addr learned my addr 172.21.15.77:0/4220020134 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:30.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f8826039640 1 --1- 172.21.15.77:0/4220020134 >> v1:172.21.15.77:6789/0 conn(0x7f882015dfe0 0x7f88200cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57538/0 (socket says 172.21.15.77:57538) 2024-03-06T07:31:30.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882683a640 1 -- 172.21.15.77:0/4220020134 >> v1:172.21.15.158:6789/0 conn(0x7f88200d22b0 legacy=0x7f88200b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882683a640 1 -- 172.21.15.77:0/4220020134 >> v1:172.21.15.77:6789/0 conn(0x7f882015dfe0 legacy=0x7f88200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.766+0000 7f882683a640 1 -- 172.21.15.77:0/4220020134 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88200b39f0 con 0x7f88200d2710 2024-03-06T07:31:30.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.767+0000 7f882683a640 1 --2- 172.21.15.77:0/4220020134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88200d2710 0x7f88200b4900 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f8818009e80 tx=0x7f881802bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef7cadc607239e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.767+0000 7f8825037640 1 -- 172.21.15.77:0/4220020134 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881802c8c0 con 0x7f88200d2710 2024-03-06T07:31:30.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.767+0000 7f8825037640 1 -- 172.21.15.77:0/4220020134 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881802ca60 con 0x7f88200d2710 2024-03-06T07:31:30.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.767+0000 7f8825037640 1 -- 172.21.15.77:0/4220020134 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881802ce30 con 0x7f88200d2710 2024-03-06T07:31:30.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.768+0000 7f882829d640 1 -- 172.21.15.77:0/4220020134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88200d2710 msgr2=0x7f88200b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.768+0000 7f882829d640 1 --2- 172.21.15.77:0/4220020134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88200d2710 0x7f88200b4900 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f8818009e80 tx=0x7f881802bda0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.768+0000 7f882829d640 1 -- 172.21.15.77:0/4220020134 shutdown_connections 2024-03-06T07:31:30.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.768+0000 7f882829d640 1 --2- 172.21.15.77:0/4220020134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88200d2710 0x7f88200b4900 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.768+0000 7f882829d640 1 -- 172.21.15.77:0/4220020134 >> 172.21.15.77:0/4220020134 conn(0x7f88200d3cd0 msgr2=0x7f88200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.768+0000 7f882829d640 1 -- 172.21.15.77:0/4220020134 shutdown_connections 2024-03-06T07:31:30.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.768+0000 7f882829d640 1 -- 172.21.15.77:0/4220020134 wait complete. 2024-03-06T07:31:30.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.769+0000 7f882829d640 1 Processor -- start 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.769+0000 7f882829d640 1 -- start start 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882829d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88200d22b0 0x7f88200cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882829d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f88200d2710 0x7f88200b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882829d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f882015dfe0 0x7f88200b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882829d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88200cfa00 con 0x7f882015dfe0 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882829d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88200b3d60 con 0x7f88200d22b0 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882829d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88200d1c70 con 0x7f88200d2710 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f882015dfe0 0x7f88200b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f882015dfe0 0x7f88200b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48634/0 (socket says 172.21.15.77:48634) 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 -- 172.21.15.77:0/3669918026 learned_addr learned my addr 172.21.15.77:0/3669918026 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f8826039640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88200d22b0 0x7f88200cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f8825838640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f88200d2710 0x7f88200b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 -- 172.21.15.77:0/3669918026 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f88200d2710 msgr2=0x7f88200b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f88200d2710 0x7f88200b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 -- 172.21.15.77:0/3669918026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88200d22b0 msgr2=0x7f88200cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88200d22b0 0x7f88200cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 -- 172.21.15.77:0/3669918026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88200cd7b0 con 0x7f882015dfe0 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f882683a640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f882015dfe0 0x7f88200b6de0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f8818009fa0 tx=0x7f8818004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.770+0000 7f881effd640 1 -- 172.21.15.77:0/3669918026 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818043020 con 0x7f882015dfe0 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.771+0000 7f881effd640 1 -- 172.21.15.77:0/3669918026 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881803e450 con 0x7f882015dfe0 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.771+0000 7f881effd640 1 -- 172.21.15.77:0/3669918026 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818009510 con 0x7f882015dfe0 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.771+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88200cc1d0 con 0x7f882015dfe0 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.771+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88200cc790 con 0x7f882015dfe0 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.771+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882004f330 con 0x7f882015dfe0 2024-03-06T07:31:30.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.772+0000 7f881effd640 1 -- 172.21.15.77:0/3669918026 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8818046020 con 0x7f882015dfe0 2024-03-06T07:31:30.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.772+0000 7f881effd640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f880c06b4d0 0x7f880c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.773+0000 7f8826039640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f880c06b4d0 0x7f880c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.773+0000 7f881effd640 1 -- 172.21.15.77:0/3669918026 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f88180acb60 con 0x7f882015dfe0 2024-03-06T07:31:30.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.773+0000 7f8826039640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f880c06b4d0 0x7f880c06d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8810000f60 tx=0x7f8810007c60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.777+0000 7f881effd640 1 -- 172.21.15.77:0/3669918026 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f881804d030 con 0x7f882015dfe0 2024-03-06T07:31:30.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.985+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f882004f330 con 0x7f882015dfe0 2024-03-06T07:31:30.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.985+0000 7f881effd640 1 -- 172.21.15.77:0/3669918026 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f881807b9c0 con 0x7f882015dfe0 2024-03-06T07:31:30.987 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:30.987 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:30.988 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:30.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f880c06b4d0 msgr2=0x7f880c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f880c06b4d0 0x7f880c06d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8810000f60 tx=0x7f8810007c60 comp rx=0 tx=0).stop 2024-03-06T07:31:30.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f882015dfe0 msgr2=0x7f88200b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f882015dfe0 0x7f88200b6de0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f8818009fa0 tx=0x7f8818004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 shutdown_connections 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f880c06b4d0 0x7f880c06d990 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f882015dfe0 0x7f88200b6de0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f88200d2710 0x7f88200b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 --2- 172.21.15.77:0/3669918026 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f88200d22b0 0x7f88200cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 >> 172.21.15.77:0/3669918026 conn(0x7f88200d3cd0 msgr2=0x7f88200cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 shutdown_connections 2024-03-06T07:31:30.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:30.987+0000 7f882829d640 1 -- 172.21.15.77:0/3669918026 wait complete. 2024-03-06T07:31:31.201 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:31.201 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:31.854 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:31.856 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:32.202 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.305+0000 7fab14bd0640 1 Processor -- start 2024-03-06T07:31:32.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.305+0000 7fab14bd0640 1 -- start start 2024-03-06T07:31:32.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab14bd0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fab100d22b0 0x7fab100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab14bd0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab100b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab14bd0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab100b3430 con 0x7fab1015dfe0 2024-03-06T07:31:32.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab14bd0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab100d2c50 con 0x7fab100d22b0 2024-03-06T07:31:32.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab14bd0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab100b31e0 con 0x7fab100b41f0 2024-03-06T07:31:32.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab0dd9b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fab100d22b0 0x7fab100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab0ed9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab100b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab0dd9b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fab100d22b0 0x7fab100d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:58704/0 (socket says 172.21.15.77:58704) 2024-03-06T07:31:32.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab0e59c640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fab1015dfe0 0x7fab100cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44820/0 (socket says 172.21.15.77:44820) 2024-03-06T07:31:32.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab0dd9b640 1 -- 172.21.15.77:0/2196188654 learned_addr learned my addr 172.21.15.77:0/2196188654 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:32.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab0ed9d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab100b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:43544/0 (socket says 172.21.15.77:43544) 2024-03-06T07:31:32.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab0ed9d640 1 -- 172.21.15.77:0/2196188654 >> v2:172.21.15.158:3300/0 conn(0x7fab100d22b0 msgr2=0x7fab100d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.306+0000 7fab0ed9d640 1 --2- 172.21.15.77:0/2196188654 >> v2:172.21.15.158:3300/0 conn(0x7fab100d22b0 0x7fab100d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.307+0000 7fab0ed9d640 1 -- 172.21.15.77:0/2196188654 >> v1:172.21.15.158:6789/0 conn(0x7fab1015dfe0 legacy=0x7fab100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.307+0000 7fab0ed9d640 1 -- 172.21.15.77:0/2196188654 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab100b3710 con 0x7fab100b41f0 2024-03-06T07:31:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.307+0000 7fab0dd9b640 1 --2- 172.21.15.77:0/2196188654 >> v2:172.21.15.158:3300/0 conn(0x7fab100d22b0 0x7fab100d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.307+0000 7fab0ed9d640 1 --2- 172.21.15.77:0/2196188654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab100b4680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab08009d90 tx=0x7fab0802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39fcf2e5336a659f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.307+0000 7fab0d59a640 1 -- 172.21.15.77:0/2196188654 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08039070 con 0x7fab100b41f0 2024-03-06T07:31:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab0d59a640 1 -- 172.21.15.77:0/2196188654 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0802cc50 con 0x7fab100b41f0 2024-03-06T07:31:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab0d59a640 1 -- 172.21.15.77:0/2196188654 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08034440 con 0x7fab100b41f0 2024-03-06T07:31:32.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab14bd0640 1 -- 172.21.15.77:0/2196188654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 msgr2=0x7fab100b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2196188654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab100b4680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab08009d90 tx=0x7fab0802bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab14bd0640 1 -- 172.21.15.77:0/2196188654 shutdown_connections 2024-03-06T07:31:32.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2196188654 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab100b4680 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2196188654 >> v2:172.21.15.158:3300/0 conn(0x7fab100d22b0 0x7fab100d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab14bd0640 1 -- 172.21.15.77:0/2196188654 >> 172.21.15.77:0/2196188654 conn(0x7fab100d3cd0 msgr2=0x7fab100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab14bd0640 1 -- 172.21.15.77:0/2196188654 shutdown_connections 2024-03-06T07:31:32.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.308+0000 7fab14bd0640 1 -- 172.21.15.77:0/2196188654 wait complete. 2024-03-06T07:31:32.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.309+0000 7fab14bd0640 1 Processor -- start 2024-03-06T07:31:32.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.309+0000 7fab14bd0640 1 -- start start 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab14bd0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab10161020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab14bd0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab100d22b0 0x7fab100cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab14bd0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab1015dfe0 0x7fab100cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab14bd0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab100cfa00 con 0x7fab100d22b0 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab14bd0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab100b3d60 con 0x7fab1015dfe0 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab14bd0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab100d1c70 con 0x7fab100b41f0 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab0dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab100d22b0 0x7fab100cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab0dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab100d22b0 0x7fab100cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48646/0 (socket says 172.21.15.77:48646) 2024-03-06T07:31:32.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab0dd9b640 1 -- 172.21.15.77:0/2614580880 learned_addr learned my addr 172.21.15.77:0/2614580880 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab0e59c640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab10161020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.310+0000 7fab0ed9d640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab1015dfe0 0x7fab100cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab0dd9b640 1 -- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 msgr2=0x7fab10161020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab0dd9b640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab10161020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab0dd9b640 1 -- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab1015dfe0 msgr2=0x7fab100cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab0dd9b640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab1015dfe0 0x7fab100cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab0dd9b640 1 -- 172.21.15.77:0/2614580880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab100b9980 con 0x7fab100d22b0 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab0e59c640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab10161020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab0ed9d640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab1015dfe0 0x7fab100cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab0dd9b640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab100d22b0 0x7fab100cb570 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fab0400e9d0 tx=0x7fab0400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7faafeffd640 1 -- 172.21.15.77:0/2614580880 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04017070 con 0x7fab100d22b0 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7faafeffd640 1 -- 172.21.15.77:0/2614580880 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0400ca60 con 0x7fab100d22b0 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1015f630 con 0x7fab100d22b0 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7faafeffd640 1 -- 172.21.15.77:0/2614580880 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0400ce10 con 0x7fab100d22b0 2024-03-06T07:31:32.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.311+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1015fbf0 con 0x7fab100d22b0 2024-03-06T07:31:32.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.312+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1004f330 con 0x7fab100d22b0 2024-03-06T07:31:32.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.312+0000 7faafeffd640 1 -- 172.21.15.77:0/2614580880 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab0401a070 con 0x7fab100d22b0 2024-03-06T07:31:32.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.313+0000 7faafeffd640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaf006b4d0 0x7faaf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.313+0000 7faafeffd640 1 -- 172.21.15.77:0/2614580880 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fab0408a5d0 con 0x7fab100d22b0 2024-03-06T07:31:32.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.313+0000 7fab0e59c640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaf006b4d0 0x7faaf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.313+0000 7fab0e59c640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaf006b4d0 0x7faaf006d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fab00007fd0 tx=0x7fab00007d60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.317+0000 7faafeffd640 1 -- 172.21.15.77:0/2614580880 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab040593b0 con 0x7fab100d22b0 2024-03-06T07:31:32.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.591+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab100ce9a0 con 0x7fab100d22b0 2024-03-06T07:31:32.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.591+0000 7faafeffd640 1 -- 172.21.15.77:0/2614580880 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fab04058cb0 con 0x7fab100d22b0 2024-03-06T07:31:32.593 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:32.593 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:32.593 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:32.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaf006b4d0 msgr2=0x7faaf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaf006b4d0 0x7faaf006d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fab00007fd0 tx=0x7fab00007d60 comp rx=0 tx=0).stop 2024-03-06T07:31:32.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab100d22b0 msgr2=0x7fab100cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab100d22b0 0x7fab100cb570 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fab0400e9d0 tx=0x7fab0400ee90 comp rx=0 tx=0).stop 2024-03-06T07:31:32.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 shutdown_connections 2024-03-06T07:31:32.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7faaf006b4d0 0x7faaf006d990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab100d22b0 0x7fab100cb570 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab1015dfe0 0x7fab100cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 --2- 172.21.15.77:0/2614580880 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fab100b41f0 0x7fab10161020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 >> 172.21.15.77:0/2614580880 conn(0x7fab100d3cd0 msgr2=0x7fab1006cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 shutdown_connections 2024-03-06T07:31:32.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:32.593+0000 7fab14bd0640 1 -- 172.21.15.77:0/2614580880 wait complete. 2024-03-06T07:31:32.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:32.807 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:33.809 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:33.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.909+0000 7fdef2102640 1 Processor -- start 2024-03-06T07:31:33.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.909+0000 7fdef2102640 1 -- start start 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.910+0000 7fdef2102640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdeec15dfe0 0x7fdeec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.910+0000 7fdef2102640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec0d29c0 0x7fdeec0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.910+0000 7fdef2102640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec0b3430 con 0x7fdeec0cf830 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.910+0000 7fdef2102640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeec0b31e0 con 0x7fdeec15dfe0 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.910+0000 7fdef2102640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec0b3d60 con 0x7fdeec0d29c0 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.910+0000 7fdeebfff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec0d29c0 0x7fdeec0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.910+0000 7fdeeb7fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdeec15dfe0 0x7fdeec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdeebfff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec0d29c0 0x7fdeec0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:43562/0 (socket says 172.21.15.77:43562) 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdeeb7fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdeec15dfe0 0x7fdeec0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:58720/0 (socket says 172.21.15.77:58720) 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdee2ffd640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fdeec0cf830 0x7fdeec0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:44836/0 (socket says 172.21.15.77:44836) 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdeebfff640 1 -- 172.21.15.77:0/1671281971 learned_addr learned my addr 172.21.15.77:0/1671281971 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:33.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdeeb7fe640 1 -- 172.21.15.77:0/1671281971 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec0d29c0 msgr2=0x7fdeec0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdeeb7fe640 1 --2- 172.21.15.77:0/1671281971 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec0d29c0 0x7fdeec0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdeeb7fe640 1 -- 172.21.15.77:0/1671281971 >> v1:172.21.15.158:6789/0 conn(0x7fdeec0cf830 legacy=0x7fdeec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdeeb7fe640 1 -- 172.21.15.77:0/1671281971 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec0b3710 con 0x7fdeec15dfe0 2024-03-06T07:31:33.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.911+0000 7fdeebfff640 1 --2- 172.21.15.77:0/1671281971 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec0d29c0 0x7fdeec0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:33.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdeeb7fe640 1 --2- 172.21.15.77:0/1671281971 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec15dfe0 0x7fdeec0cf2f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0097a0 tx=0x7fdedc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e5887a593028f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdeeaffd640 1 -- 172.21.15.77:0/1671281971 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc035020 con 0x7fdeec15dfe0 2024-03-06T07:31:33.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdeeaffd640 1 -- 172.21.15.77:0/1671281971 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc031830 con 0x7fdeec15dfe0 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdeeaffd640 1 -- 172.21.15.77:0/1671281971 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc031c00 con 0x7fdeec15dfe0 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdef2102640 1 -- 172.21.15.77:0/1671281971 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec15dfe0 msgr2=0x7fdeec0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdef2102640 1 --2- 172.21.15.77:0/1671281971 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec15dfe0 0x7fdeec0cf2f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0097a0 tx=0x7fdedc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdef2102640 1 -- 172.21.15.77:0/1671281971 shutdown_connections 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdef2102640 1 --2- 172.21.15.77:0/1671281971 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec0d29c0 0x7fdeec0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdef2102640 1 --2- 172.21.15.77:0/1671281971 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec15dfe0 0x7fdeec0cf2f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdef2102640 1 -- 172.21.15.77:0/1671281971 >> 172.21.15.77:0/1671281971 conn(0x7fdeec0d3cd0 msgr2=0x7fdeec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdef2102640 1 -- 172.21.15.77:0/1671281971 shutdown_connections 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.912+0000 7fdef2102640 1 -- 172.21.15.77:0/1671281971 wait complete. 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.914+0000 7fdef2102640 1 Processor -- start 2024-03-06T07:31:33.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.914+0000 7fdef2102640 1 -- start start 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.914+0000 7fdef2102640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec0cf830 0x7fdeec150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.914+0000 7fdef2102640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec0d29c0 0x7fdeec09f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.914+0000 7fdef2102640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec15dfe0 0x7fdeec0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.914+0000 7fdef2102640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec0d1df0 con 0x7fdeec0d29c0 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.914+0000 7fdef2102640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec0d1c70 con 0x7fdeec0cf830 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.914+0000 7fdef2102640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec15e3c0 con 0x7fdeec15dfe0 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec0d29c0 0x7fdeec09f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec0d29c0 0x7fdeec09f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48662/0 (socket says 172.21.15.77:48662) 2024-03-06T07:31:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 -- 172.21.15.77:0/903231326 learned_addr learned my addr 172.21.15.77:0/903231326 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdeebfff640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec15dfe0 0x7fdeec0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdeeb7fe640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec0cf830 0x7fdeec150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 -- 172.21.15.77:0/903231326 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec15dfe0 msgr2=0x7fdeec0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec15dfe0 0x7fdeec0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 -- 172.21.15.77:0/903231326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec0cf830 msgr2=0x7fdeec150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec0cf830 0x7fdeec150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 -- 172.21.15.77:0/903231326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec0cb9b0 con 0x7fdeec0d29c0 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee2ffd640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec0d29c0 0x7fdeec09f120 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fded400c440 tx=0x7fded4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee8ff9640 1 -- 172.21.15.77:0/903231326 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded40074e0 con 0x7fdeec0d29c0 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee8ff9640 1 -- 172.21.15.77:0/903231326 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded4007680 con 0x7fdeec0d29c0 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec0cbc90 con 0x7fdeec0d29c0 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdee8ff9640 1 -- 172.21.15.77:0/903231326 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4004a20 con 0x7fdeec0d29c0 2024-03-06T07:31:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.915+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec0cc250 con 0x7fdeec0d29c0 2024-03-06T07:31:33.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.916+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec04f330 con 0x7fdeec0d29c0 2024-03-06T07:31:33.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.917+0000 7fdee8ff9640 1 -- 172.21.15.77:0/903231326 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fded401a020 con 0x7fdeec0d29c0 2024-03-06T07:31:33.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.917+0000 7fdee8ff9640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded006b500 0x7fded006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.917+0000 7fdee8ff9640 1 -- 172.21.15.77:0/903231326 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fded4089900 con 0x7fdeec0d29c0 2024-03-06T07:31:33.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.917+0000 7fdeeb7fe640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded006b500 0x7fded006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.918+0000 7fdeeb7fe640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded006b500 0x7fded006d9c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fdedc036000 tx=0x7fdedc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:33.922+0000 7fdee8ff9640 1 -- 172.21.15.77:0/903231326 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fded4058080 con 0x7fdeec0d29c0 2024-03-06T07:31:34.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.139+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdeec0b5180 con 0x7fdeec0d29c0 2024-03-06T07:31:34.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.140+0000 7fdee8ff9640 1 -- 172.21.15.77:0/903231326 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fded405c120 con 0x7fdeec0d29c0 2024-03-06T07:31:34.142 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:34.142 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:34.142 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded006b500 msgr2=0x7fded006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded006b500 0x7fded006d9c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fdedc036000 tx=0x7fdedc03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec0d29c0 msgr2=0x7fdeec09f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec0d29c0 0x7fdeec09f120 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fded400c440 tx=0x7fded4001e70 comp rx=0 tx=0).stop 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 shutdown_connections 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fded006b500 0x7fded006d9c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec0d29c0 0x7fdeec09f120 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdeec15dfe0 0x7fdeec0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 --2- 172.21.15.77:0/903231326 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdeec0cf830 0x7fdeec150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 >> 172.21.15.77:0/903231326 conn(0x7fdeec0d3cd0 msgr2=0x7fdeec0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 shutdown_connections 2024-03-06T07:31:34.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:34.142+0000 7fdef2102640 1 -- 172.21.15.77:0/903231326 wait complete. 2024-03-06T07:31:34.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:34.356 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:35.357 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:35.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.456+0000 7fa05a53a640 1 Processor -- start 2024-03-06T07:31:35.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.457+0000 7fa05a53a640 1 -- start start 2024-03-06T07:31:35.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.457+0000 7fa05a53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05415dfe0 0x7fa0540cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.457+0000 7fa05a53a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0540b3430 con 0x7fa0540d29c0 2024-03-06T07:31:35.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.457+0000 7fa05a53a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0540b3710 con 0x7fa0540cf830 2024-03-06T07:31:35.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.457+0000 7fa05a53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0540b31e0 con 0x7fa05415dfe0 2024-03-06T07:31:35.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.457+0000 7fa058ad7640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa0540d29c0 0x7fa0540b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57542/0 (socket says 172.21.15.77:57542) 2024-03-06T07:31:35.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.457+0000 7fa058ad7640 1 -- 172.21.15.77:0/4170867578 learned_addr learned my addr 172.21.15.77:0/4170867578 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa053fff640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05415dfe0 0x7fa0540cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778541699 0 0) 0x7fa0540b3430 con 0x7fa0540d29c0 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa03c003520 con 0x7fa0540d29c0 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087109051 0 0) 0x7fa0540b3710 con 0x7fa0540cf830 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0540b3430 con 0x7fa0540cf830 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1785102596 0 0) 0x7fa03c003520 con 0x7fa0540d29c0 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05415dfe0 msgr2=0x7fa0540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05415dfe0 0x7fa0540cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 >> v1:172.21.15.158:6789/0 conn(0x7fa0540cf830 legacy=0x7fa0540d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0540b39f0 con 0x7fa0540d29c0 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fa04c0019f0 con 0x7fa0540d29c0 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 >> v1:172.21.15.77:6789/0 conn(0x7fa0540d29c0 legacy=0x7fa0540b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05415dfe0 0x7fa03c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa03c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa03c004d40 0x7fa03c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0540b31e0 con 0x7fa03c0040d0 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c00baa0 con 0x7fa03c004d40 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c00bca0 con 0x7fa05415dfe0 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa0537fe640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa03c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa053fff640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05415dfe0 0x7fa03c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa058ad7640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa03c004d40 0x7fa03c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa0537fe640 1 -- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05415dfe0 msgr2=0x7fa03c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa0537fe640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05415dfe0 0x7fa03c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa0537fe640 1 -- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa03c004d40 msgr2=0x7fa03c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa0537fe640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa03c004d40 0x7fa03c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.458+0000 7fa0537fe640 1 -- 172.21.15.77:0/4170867578 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c003520 con 0x7fa03c0040d0 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.459+0000 7fa058ad7640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa03c004d40 0x7fa03c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.459+0000 7fa0537fe640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa03c004800 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fa0480018a0 tx=0x7fa03c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82e398764d0c860 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.459+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa048037070 con 0x7fa03c0040d0 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.459+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0480052d0 con 0x7fa03c0040d0 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.459+0000 7fa052ffd640 1 -- 172.21.15.77:0/4170867578 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0480014f0 con 0x7fa03c0040d0 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.460+0000 7fa05a53a640 1 -- 172.21.15.77:0/4170867578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 msgr2=0x7fa03c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.460+0000 7fa05a53a640 1 --2- 172.21.15.77:0/4170867578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa03c004800 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fa0480018a0 tx=0x7fa03c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.460+0000 7fa058ad7640 1 -- 172.21.15.77:0/4170867578 reap_dead start 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.460+0000 7fa05a53a640 1 -- 172.21.15.77:0/4170867578 shutdown_connections 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.460+0000 7fa05a53a640 1 -- 172.21.15.77:0/4170867578 >> 172.21.15.77:0/4170867578 conn(0x7fa0540d3cd0 msgr2=0x7fa0540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.460+0000 7fa05a53a640 1 -- 172.21.15.77:0/4170867578 shutdown_connections 2024-03-06T07:31:35.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.460+0000 7fa05a53a640 1 -- 172.21.15.77:0/4170867578 wait complete. 2024-03-06T07:31:35.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.461+0000 7fa05a53a640 1 Processor -- start 2024-03-06T07:31:35.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.461+0000 7fa05a53a640 1 -- start start 2024-03-06T07:31:35.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa05a53a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa0540cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa05a53a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0540cd980 0x7fa05415cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa05a53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05409f440 0x7fa05409f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa05a53a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05415e3c0 con 0x7fa03c0040d0 2024-03-06T07:31:35.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa05a53a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0540b3d60 con 0x7fa0540cd980 2024-03-06T07:31:35.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa05a53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05415e540 con 0x7fa05409f440 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa053fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa0540cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa053fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa0540cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48690/0 (socket says 172.21.15.77:48690) 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa053fff640 1 -- 172.21.15.77:0/1891734083 learned_addr learned my addr 172.21.15.77:0/1891734083 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa058ad7640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05409f440 0x7fa05409f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa0537fe640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0540cd980 0x7fa05415cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa053fff640 1 -- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05409f440 msgr2=0x7fa05409f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa053fff640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05409f440 0x7fa05409f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa053fff640 1 -- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0540cd980 msgr2=0x7fa05415cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa053fff640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0540cd980 0x7fa05415cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa053fff640 1 -- 172.21.15.77:0/1891734083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0540cc0c0 con 0x7fa03c0040d0 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.462+0000 7fa0537fe640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0540cd980 0x7fa05415cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.463+0000 7fa053fff640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa0540cbae0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fa044011a00 tx=0x7fa044011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.463+0000 7fa050ff9640 1 -- 172.21.15.77:0/1891734083 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04400faf0 con 0x7fa03c0040d0 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.463+0000 7fa050ff9640 1 -- 172.21.15.77:0/1891734083 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04400fc90 con 0x7fa03c0040d0 2024-03-06T07:31:35.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.463+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0540cc3a0 con 0x7fa03c0040d0 2024-03-06T07:31:35.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.463+0000 7fa050ff9640 1 -- 172.21.15.77:0/1891734083 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044014440 con 0x7fa03c0040d0 2024-03-06T07:31:35.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.463+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0540cc960 con 0x7fa03c0040d0 2024-03-06T07:31:35.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.464+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0540b2500 con 0x7fa03c0040d0 2024-03-06T07:31:35.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.465+0000 7fa050ff9640 1 -- 172.21.15.77:0/1891734083 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa044012040 con 0x7fa03c0040d0 2024-03-06T07:31:35.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.465+0000 7fa050ff9640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa04006b4d0 0x7fa04006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.465+0000 7fa050ff9640 1 -- 172.21.15.77:0/1891734083 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa04408a7b0 con 0x7fa03c0040d0 2024-03-06T07:31:35.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.465+0000 7fa0537fe640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa04006b4d0 0x7fa04006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.465+0000 7fa0537fe640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa04006b4d0 0x7fa04006d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa048034000 tx=0x7fa048031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.469+0000 7fa050ff9640 1 -- 172.21.15.77:0/1891734083 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0540b2500 con 0x7fa03c0040d0 2024-03-06T07:31:35.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.694+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa05404f330 con 0x7fa03c0040d0 2024-03-06T07:31:35.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.694+0000 7fa050ff9640 1 -- 172.21.15.77:0/1891734083 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fa044019030 con 0x7fa03c0040d0 2024-03-06T07:31:35.696 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:35.697 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:35.697 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:35.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa04006b4d0 msgr2=0x7fa04006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa04006b4d0 0x7fa04006d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa048034000 tx=0x7fa048031000 comp rx=0 tx=0).stop 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 msgr2=0x7fa0540cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa0540cbae0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fa044011a00 tx=0x7fa044011ec0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 shutdown_connections 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa05409f440 0x7fa05409f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa0540cd980 0x7fa05415cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa04006b4d0 0x7fa04006d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 --2- 172.21.15.77:0/1891734083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa03c0040d0 0x7fa0540cbae0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.696+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 >> 172.21.15.77:0/1891734083 conn(0x7fa0540d3cd0 msgr2=0x7fa05406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.697+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 shutdown_connections 2024-03-06T07:31:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:35.697+0000 7fa05a53a640 1 -- 172.21.15.77:0/1891734083 wait complete. 2024-03-06T07:31:35.911 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:35.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:36.911 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.011+0000 7fce1fd81640 1 Processor -- start 2024-03-06T07:31:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.011+0000 7fce1fd81640 1 -- start start 2024-03-06T07:31:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.012+0000 7fce1fd81640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce180d2710 0x7fce180b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.012+0000 7fce1fd81640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce180b3430 con 0x7fce1815dfe0 2024-03-06T07:31:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.012+0000 7fce1fd81640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce180b3710 con 0x7fce180d22b0 2024-03-06T07:31:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.012+0000 7fce1fd81640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce180b31e0 con 0x7fce180d2710 2024-03-06T07:31:37.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.012+0000 7fce1db1d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fce1815dfe0 0x7fce180cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57552/0 (socket says 172.21.15.77:57552) 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.012+0000 7fce1db1d640 1 -- 172.21.15.77:0/3568926226 learned_addr learned my addr 172.21.15.77:0/3568926226 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680851756 0 0) 0x7fce180b3430 con 0x7fce1815dfe0 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce04003520 con 0x7fce1815dfe0 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1e31e640 1 --2- 172.21.15.77:0/3568926226 >> v2:172.21.15.158:3300/0 conn(0x7fce180d2710 0x7fce180b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2643336341 0 0) 0x7fce04003520 con 0x7fce1815dfe0 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 >> v2:172.21.15.158:3300/0 conn(0x7fce180d2710 msgr2=0x7fce180b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 --2- 172.21.15.77:0/3568926226 >> v2:172.21.15.158:3300/0 conn(0x7fce180d2710 0x7fce180b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 >> v1:172.21.15.158:6789/0 conn(0x7fce180d22b0 legacy=0x7fce180b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce180b39f0 con 0x7fce1815dfe0 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fce14001670 con 0x7fce1815dfe0 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 >> v1:172.21.15.77:6789/0 conn(0x7fce1815dfe0 legacy=0x7fce180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce040039e0 0x7fce04003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce04005210 0x7fce0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce180b31e0 con 0x7fce04004800 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0400c140 con 0x7fce04005210 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0400c340 con 0x7fce040039e0 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1e31e640 1 --2- 172.21.15.77:0/3568926226 >> v2:172.21.15.158:3300/0 conn(0x7fce180d2710 0x7fce180b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1d31c640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1db1d640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce040039e0 0x7fce04003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1e31e640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce04005210 0x7fce0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1d31c640 1 -- 172.21.15.77:0/3568926226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce040039e0 msgr2=0x7fce04003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1d31c640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce040039e0 0x7fce04003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1d31c640 1 -- 172.21.15.77:0/3568926226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce04005210 msgr2=0x7fce0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1d31c640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce04005210 0x7fce0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1d31c640 1 -- 172.21.15.77:0/3568926226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce04003520 con 0x7fce04004800 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.013+0000 7fce1e31e640 1 -- 172.21.15.77:0/3568926226 reap_dead start 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.014+0000 7fce1d31c640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce04004c80 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fce08001720 tx=0x7fce04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2897d9ddf062411c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.014+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08001370 con 0x7fce04004800 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.014+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce080052d0 con 0x7fce04004800 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.014+0000 7fce1cb1b640 1 -- 172.21.15.77:0/3568926226 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08003830 con 0x7fce04004800 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.015+0000 7fce1fd81640 1 -- 172.21.15.77:0/3568926226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 msgr2=0x7fce04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.015+0000 7fce1fd81640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce04004c80 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fce08001720 tx=0x7fce04005830 comp rx=0 tx=0).stop 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.015+0000 7fce1fd81640 1 -- 172.21.15.77:0/3568926226 shutdown_connections 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.015+0000 7fce1fd81640 1 --2- 172.21.15.77:0/3568926226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce04004c80 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.015+0000 7fce1fd81640 1 -- 172.21.15.77:0/3568926226 >> 172.21.15.77:0/3568926226 conn(0x7fce180d3cd0 msgr2=0x7fce180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.015+0000 7fce1fd81640 1 -- 172.21.15.77:0/3568926226 shutdown_connections 2024-03-06T07:31:37.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.015+0000 7fce1fd81640 1 -- 172.21.15.77:0/3568926226 wait complete. 2024-03-06T07:31:37.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.017+0000 7fce1fd81640 1 Processor -- start 2024-03-06T07:31:37.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.017+0000 7fce1fd81640 1 -- start start 2024-03-06T07:31:37.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1fd81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce181f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1fd81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce181f6ff0 0x7fce1815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1fd81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce180cc690 0x7fce180b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1fd81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce180d1c70 con 0x7fce04004800 2024-03-06T07:31:37.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1fd81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce180b3d60 con 0x7fce180cc690 2024-03-06T07:31:37.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1fd81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce180d1df0 con 0x7fce181f6ff0 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1db1d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce181f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1db1d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce181f4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48702/0 (socket says 172.21.15.77:48702) 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1db1d640 1 -- 172.21.15.77:0/1331908025 learned_addr learned my addr 172.21.15.77:0/1331908025 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1d31c640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce181f6ff0 0x7fce1815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1e31e640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce180cc690 0x7fce180b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1db1d640 1 -- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce181f6ff0 msgr2=0x7fce1815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1db1d640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce181f6ff0 0x7fce1815c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1db1d640 1 -- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce180cc690 msgr2=0x7fce180b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1db1d640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce180cc690 0x7fce180b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1db1d640 1 -- 172.21.15.77:0/1331908025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce181f7820 con 0x7fce04004800 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.018+0000 7fce1d31c640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce181f6ff0 0x7fce1815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.019+0000 7fce1db1d640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce181f4580 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fce1400e7f0 tx=0x7fce1400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.019+0000 7fce0e7fc640 1 -- 172.21.15.77:0/1331908025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1400c8d0 con 0x7fce04004800 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.019+0000 7fce0e7fc640 1 -- 172.21.15.77:0/1331908025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce14018070 con 0x7fce04004800 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.019+0000 7fce0e7fc640 1 -- 172.21.15.77:0/1331908025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1400cc70 con 0x7fce04004800 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.019+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce181f7b00 con 0x7fce04004800 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.019+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce180cbe90 con 0x7fce04004800 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.020+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1804f330 con 0x7fce04004800 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.020+0000 7fce0e7fc640 1 -- 172.21.15.77:0/1331908025 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce1401d460 con 0x7fce04004800 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.020+0000 7fce0e7fc640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdfc06b500 0x7fcdfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.021+0000 7fce0e7fc640 1 -- 172.21.15.77:0/1331908025 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fce1408a150 con 0x7fce04004800 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.021+0000 7fce1d31c640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdfc06b500 0x7fcdfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.021+0000 7fce1d31c640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdfc06b500 0x7fcdfc06d9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fce08033000 tx=0x7fce08009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.025+0000 7fce0e7fc640 1 -- 172.21.15.77:0/1331908025 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce14058fb0 con 0x7fce04004800 2024-03-06T07:31:37.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.241+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce180b4e10 con 0x7fce04004800 2024-03-06T07:31:37.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.241+0000 7fce0e7fc640 1 -- 172.21.15.77:0/1331908025 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fce140588b0 con 0x7fce04004800 2024-03-06T07:31:37.243 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:37.244 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:37.244 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdfc06b500 msgr2=0x7fcdfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdfc06b500 0x7fcdfc06d9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fce08033000 tx=0x7fce08009030 comp rx=0 tx=0).stop 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 msgr2=0x7fce181f4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce181f4580 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fce1400e7f0 tx=0x7fce1400ecb0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 shutdown_connections 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce180cc690 0x7fce180b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce181f6ff0 0x7fce1815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcdfc06b500 0x7fcdfc06d9c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 --2- 172.21.15.77:0/1331908025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce04004800 0x7fce181f4580 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.243+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 >> 172.21.15.77:0/1331908025 conn(0x7fce180d3cd0 msgr2=0x7fce180cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.244+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 shutdown_connections 2024-03-06T07:31:37.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:37.244+0000 7fce1fd81640 1 -- 172.21.15.77:0/1331908025 wait complete. 2024-03-06T07:31:37.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:37.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:38.458 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:38.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.557+0000 7fc32730c640 1 Processor -- start 2024-03-06T07:31:38.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.557+0000 7fc32730c640 1 -- start start 2024-03-06T07:31:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc32730c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3201515e0 0x7fc3200cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc32730c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc32015b430 0x7fc320071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc32730c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc3200cd7a0 0x7fc3200cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc32730c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32015b8b0 con 0x7fc32015b430 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc32730c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc320070a70 con 0x7fc3200cd7a0 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc32730c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc320070d50 con 0x7fc3201515e0 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3248a7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc32015b430 0x7fc320071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3248a7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc32015b430 0x7fc320071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32860/0 (socket says 172.21.15.77:32860) 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3248a7640 1 -- 172.21.15.77:0/3146060099 learned_addr learned my addr 172.21.15.77:0/3146060099 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3250a8640 1 --2- 172.21.15.77:0/3146060099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3201515e0 0x7fc3200cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3258a9640 1 --2- 172.21.15.77:0/3146060099 >> v2:172.21.15.158:3300/0 conn(0x7fc3200cd7a0 0x7fc3200cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3248a7640 1 -- 172.21.15.77:0/3146060099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3201515e0 msgr2=0x7fc3200cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3248a7640 1 --2- 172.21.15.77:0/3146060099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3201515e0 0x7fc3200cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3248a7640 1 -- 172.21.15.77:0/3146060099 >> v2:172.21.15.158:3300/0 conn(0x7fc3200cd7a0 msgr2=0x7fc3200cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3248a7640 1 --2- 172.21.15.77:0/3146060099 >> v2:172.21.15.158:3300/0 conn(0x7fc3200cd7a0 0x7fc3200cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3248a7640 1 -- 172.21.15.77:0/3146060099 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc320071030 con 0x7fc32015b430 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3250a8640 1 --2- 172.21.15.77:0/3146060099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3201515e0 0x7fc3200cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.559+0000 7fc3258a9640 1 --2- 172.21.15.77:0/3146060099 >> v2:172.21.15.158:3300/0 conn(0x7fc3200cd7a0 0x7fc3200cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc3248a7640 1 --2- 172.21.15.77:0/3146060099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc32015b430 0x7fc320071830 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc3200b3d60 tx=0x7fc31402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8abfcfee4316fec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc31ffff640 1 -- 172.21.15.77:0/3146060099 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31402ea80 con 0x7fc32015b430 2024-03-06T07:31:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc31ffff640 1 -- 172.21.15.77:0/3146060099 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31402ec20 con 0x7fc32015b430 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc31ffff640 1 -- 172.21.15.77:0/3146060099 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc314009600 con 0x7fc32015b430 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc32730c640 1 -- 172.21.15.77:0/3146060099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc32015b430 msgr2=0x7fc320071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc32730c640 1 --2- 172.21.15.77:0/3146060099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc32015b430 0x7fc320071830 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc3200b3d60 tx=0x7fc31402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc32730c640 1 -- 172.21.15.77:0/3146060099 shutdown_connections 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc32730c640 1 --2- 172.21.15.77:0/3146060099 >> v2:172.21.15.158:3300/0 conn(0x7fc3200cd7a0 0x7fc3200cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc32730c640 1 --2- 172.21.15.77:0/3146060099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc32015b430 0x7fc320071830 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc32730c640 1 --2- 172.21.15.77:0/3146060099 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3201515e0 0x7fc3200cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.560+0000 7fc32730c640 1 -- 172.21.15.77:0/3146060099 >> 172.21.15.77:0/3146060099 conn(0x7fc3200cf8a0 msgr2=0x7fc3200cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.561+0000 7fc32730c640 1 -- 172.21.15.77:0/3146060099 shutdown_connections 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.561+0000 7fc32730c640 1 -- 172.21.15.77:0/3146060099 wait complete. 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.562+0000 7fc32730c640 1 Processor -- start 2024-03-06T07:31:38.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.562+0000 7fc32730c640 1 -- start start 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.562+0000 7fc32730c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3200cd7a0 0x7fc32014c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.562+0000 7fc32730c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3201515e0 0x7fc3200720a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.562+0000 7fc32730c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc32015b430 0x7fc3200725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.562+0000 7fc32730c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc320071d70 con 0x7fc3200cd7a0 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.562+0000 7fc32730c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200ce110 con 0x7fc3201515e0 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.562+0000 7fc32730c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc320071ef0 con 0x7fc32015b430 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3200cd7a0 0x7fc32014c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3200cd7a0 0x7fc32014c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32874/0 (socket says 172.21.15.77:32874) 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 -- 172.21.15.77:0/2563622843 learned_addr learned my addr 172.21.15.77:0/2563622843 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3248a7640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3201515e0 0x7fc3200720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3258a9640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc32015b430 0x7fc3200725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 -- 172.21.15.77:0/2563622843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc32015b430 msgr2=0x7fc3200725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc32015b430 0x7fc3200725e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 -- 172.21.15.77:0/2563622843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3201515e0 msgr2=0x7fc3200720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3201515e0 0x7fc3200720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 -- 172.21.15.77:0/2563622843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3200c5dc0 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc3250a8640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3200cd7a0 0x7fc32014c4e0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fc31800c7f0 tx=0x7fc31800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc31dffb640 1 -- 172.21.15.77:0/2563622843 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318015070 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc31dffb640 1 -- 172.21.15.77:0/2563622843 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318007820 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3200c60a0 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc31dffb640 1 -- 172.21.15.77:0/2563622843 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318007bd0 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.563+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3200c6660 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.565+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc320068840 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.565+0000 7fc31dffb640 1 -- 172.21.15.77:0/2563622843 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc318018070 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.565+0000 7fc31dffb640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc30c06b500 0x7fc30c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.565+0000 7fc31dffb640 1 -- 172.21.15.77:0/2563622843 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc31808a1f0 con 0x7fc3200cd7a0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.565+0000 7fc3248a7640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc30c06b500 0x7fc30c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.566+0000 7fc3248a7640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc30c06b500 0x7fc30c06d9c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc314037000 tx=0x7fc314032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.574+0000 7fc31dffb640 1 -- 172.21.15.77:0/2563622843 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc318058a90 con 0x7fc3200cd7a0 2024-03-06T07:31:38.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.790+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc32004f330 con 0x7fc3200cd7a0 2024-03-06T07:31:38.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.791+0000 7fc31dffb640 1 -- 172.21.15.77:0/2563622843 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc31805cb30 con 0x7fc3200cd7a0 2024-03-06T07:31:38.793 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:38.793 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:38.793 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc30c06b500 msgr2=0x7fc30c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc30c06b500 0x7fc30c06d9c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc314037000 tx=0x7fc314032000 comp rx=0 tx=0).stop 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3200cd7a0 msgr2=0x7fc32014c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3200cd7a0 0x7fc32014c4e0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fc31800c7f0 tx=0x7fc31800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 shutdown_connections 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc32015b430 0x7fc3200725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3201515e0 0x7fc3200720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc30c06b500 0x7fc30c06d9c0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 --2- 172.21.15.77:0/2563622843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3200cd7a0 0x7fc32014c4e0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 >> 172.21.15.77:0/2563622843 conn(0x7fc3200cf8a0 msgr2=0x7fc32006def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 shutdown_connections 2024-03-06T07:31:38.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:38.793+0000 7fc32730c640 1 -- 172.21.15.77:0/2563622843 wait complete. 2024-03-06T07:31:39.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:39.007 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:40.008 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:40.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.108+0000 7f892f8c9640 1 Processor -- start 2024-03-06T07:31:40.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.108+0000 7f892f8c9640 1 -- start start 2024-03-06T07:31:40.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892f8c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f89281424f0 0x7f89280cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892f8c9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f892806da40 0x7f892815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892f8c9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89280cc980 con 0x7f89280cd7a0 2024-03-06T07:31:40.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892f8c9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280cd220 con 0x7f89281424f0 2024-03-06T07:31:40.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892f8c9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280ccc60 con 0x7f892806da40 2024-03-06T07:31:40.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892d665640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f89281424f0 0x7f89280cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892d665640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f89281424f0 0x7f89280cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32882/0 (socket says 172.21.15.77:32882) 2024-03-06T07:31:40.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892d665640 1 -- 172.21.15.77:0/3494267827 learned_addr learned my addr 172.21.15.77:0/3494267827 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:40.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892de66640 1 --2- 172.21.15.77:0/3494267827 >> v2:172.21.15.158:3300/0 conn(0x7f892806da40 0x7f892815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892d665640 1 -- 172.21.15.77:0/3494267827 >> v2:172.21.15.158:3300/0 conn(0x7f892806da40 msgr2=0x7f892815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892d665640 1 --2- 172.21.15.77:0/3494267827 >> v2:172.21.15.158:3300/0 conn(0x7f892806da40 0x7f892815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892d665640 1 -- 172.21.15.77:0/3494267827 >> v1:172.21.15.158:6789/0 conn(0x7f89280cd7a0 legacy=0x7f892806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.109+0000 7f892d665640 1 -- 172.21.15.77:0/3494267827 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89280ccf40 con 0x7f89281424f0 2024-03-06T07:31:40.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.110+0000 7f892d665640 1 --2- 172.21.15.77:0/3494267827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89281424f0 0x7f89280cc2f0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f8928068700 tx=0x7f892402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e0c251ce6a934b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.110+0000 7f891ffff640 1 -- 172.21.15.77:0/3494267827 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924035020 con 0x7f89281424f0 2024-03-06T07:31:40.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.110+0000 7f891ffff640 1 -- 172.21.15.77:0/3494267827 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924031870 con 0x7f89281424f0 2024-03-06T07:31:40.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.110+0000 7f891ffff640 1 -- 172.21.15.77:0/3494267827 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924031c40 con 0x7f89281424f0 2024-03-06T07:31:40.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.111+0000 7f892f8c9640 1 -- 172.21.15.77:0/3494267827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89281424f0 msgr2=0x7f89280cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.111+0000 7f892f8c9640 1 --2- 172.21.15.77:0/3494267827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89281424f0 0x7f89280cc2f0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f8928068700 tx=0x7f892402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:40.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.111+0000 7f892f8c9640 1 -- 172.21.15.77:0/3494267827 shutdown_connections 2024-03-06T07:31:40.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.111+0000 7f892f8c9640 1 --2- 172.21.15.77:0/3494267827 >> v2:172.21.15.158:3300/0 conn(0x7f892806da40 0x7f892815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.111+0000 7f892f8c9640 1 --2- 172.21.15.77:0/3494267827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89281424f0 0x7f89280cc2f0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.111+0000 7f892f8c9640 1 -- 172.21.15.77:0/3494267827 >> 172.21.15.77:0/3494267827 conn(0x7f89280cf8a0 msgr2=0x7f89280cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.111+0000 7f892f8c9640 1 -- 172.21.15.77:0/3494267827 shutdown_connections 2024-03-06T07:31:40.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.111+0000 7f892f8c9640 1 -- 172.21.15.77:0/3494267827 wait complete. 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.112+0000 7f892f8c9640 1 Processor -- start 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.112+0000 7f892f8c9640 1 -- start start 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892f8c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806da40 0x7f892815c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892f8c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89280cd7a0 0x7f8928158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892f8c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89281424f0 0x7f89281594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892f8c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8928071ce0 con 0x7f892806da40 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892f8c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8928071b60 con 0x7f89280cd7a0 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892f8c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8928071e60 con 0x7f89281424f0 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806da40 0x7f892815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806da40 0x7f892815c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32886/0 (socket says 172.21.15.77:32886) 2024-03-06T07:31:40.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 -- 172.21.15.77:0/2649248224 learned_addr learned my addr 172.21.15.77:0/2649248224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 -- 172.21.15.77:0/2649248224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89281424f0 msgr2=0x7f89281594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89281424f0 0x7f89281594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 -- 172.21.15.77:0/2649248224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89280cd7a0 msgr2=0x7f8928158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89280cd7a0 0x7f8928158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 -- 172.21.15.77:0/2649248224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89280ab4e0 con 0x7f892806da40 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892d665640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806da40 0x7f892815c570 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f8924000c00 tx=0x7f89240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f891dffb640 1 -- 172.21.15.77:0/2649248224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924040070 con 0x7f892806da40 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f891dffb640 1 -- 172.21.15.77:0/2649248224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924030d00 con 0x7f892806da40 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89280ac0f0 con 0x7f892806da40 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.113+0000 7f891dffb640 1 -- 172.21.15.77:0/2649248224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924035040 con 0x7f892806da40 2024-03-06T07:31:40.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.114+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89280ac6b0 con 0x7f892806da40 2024-03-06T07:31:40.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.115+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8928071010 con 0x7f892806da40 2024-03-06T07:31:40.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.115+0000 7f891dffb640 1 -- 172.21.15.77:0/2649248224 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f892404b060 con 0x7f892806da40 2024-03-06T07:31:40.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.115+0000 7f891dffb640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.116+0000 7f891dffb640 1 -- 172.21.15.77:0/2649248224 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f89240abf20 con 0x7f892806da40 2024-03-06T07:31:40.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.116+0000 7f892ce64640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.116+0000 7f892ce64640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890c06b4d0 0x7f890c06d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f89280af510 tx=0x7f89180079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.120+0000 7f891dffb640 1 -- 172.21.15.77:0/2649248224 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8928071010 con 0x7f892806da40 2024-03-06T07:31:40.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.349+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f892804f330 con 0x7f892806da40 2024-03-06T07:31:40.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.349+0000 7f891dffb640 1 -- 172.21.15.77:0/2649248224 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f892407ae10 con 0x7f892806da40 2024-03-06T07:31:40.352 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:40.352 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:40.352 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:40.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890c06b4d0 msgr2=0x7f890c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890c06b4d0 0x7f890c06d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f89280af510 tx=0x7f89180079b0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806da40 msgr2=0x7f892815c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806da40 0x7f892815c570 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f8924000c00 tx=0x7f89240046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 shutdown_connections 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89281424f0 0x7f89281594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89280cd7a0 0x7f8928158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 --2- 172.21.15.77:0/2649248224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806da40 0x7f892815c570 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 >> 172.21.15.77:0/2649248224 conn(0x7f89280cf8a0 msgr2=0x7f892815ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 shutdown_connections 2024-03-06T07:31:40.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:40.351+0000 7f892f8c9640 1 -- 172.21.15.77:0/2649248224 wait complete. 2024-03-06T07:31:40.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:40.566 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:41.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:41.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.666+0000 7fce9ea9a640 1 Processor -- start 2024-03-06T07:31:41.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.666+0000 7fce9ea9a640 1 -- start start 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce9ea9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce9815dfe0 0x7fce980cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce9ea9a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce980b3430 con 0x7fce980d29c0 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce9ea9a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce980b3710 con 0x7fce980cf830 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce9ea9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b31e0 con 0x7fce9815dfe0 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce9d037640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fce980d29c0 0x7fce980b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58838/0 (socket says 172.21.15.77:58838) 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce9d037640 1 -- 172.21.15.77:0/2532355907 learned_addr learned my addr 172.21.15.77:0/2532355907 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce9c836640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce9815dfe0 0x7fce980cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532429672 0 0) 0x7fce980b3430 con 0x7fce980d29c0 2024-03-06T07:31:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce88003520 con 0x7fce980d29c0 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 516774534 0 0) 0x7fce88003520 con 0x7fce980d29c0 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce9815dfe0 msgr2=0x7fce980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce977fe640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce9815dfe0 0x7fce980cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 >> v1:172.21.15.158:6789/0 conn(0x7fce980cf830 legacy=0x7fce980d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce980b39f0 con 0x7fce980d29c0 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.667+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fce900019f0 con 0x7fce980d29c0 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 >> v1:172.21.15.77:6789/0 conn(0x7fce980d29c0 legacy=0x7fce980b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce977fe640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce9815dfe0 0x7fce880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce977fe640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce88003f20 0x7fce88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce977fe640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88004d40 0x7fce880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b31e0 con 0x7fce88003f20 2024-03-06T07:31:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8800ba20 con 0x7fce88004d40 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8800bc20 con 0x7fce9815dfe0 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce97fff640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce88003f20 0x7fce88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce9c836640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce9815dfe0 0x7fce880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce9d037640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88004d40 0x7fce880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce97fff640 1 -- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce9815dfe0 msgr2=0x7fce880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce97fff640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce9815dfe0 0x7fce880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce97fff640 1 -- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88004d40 msgr2=0x7fce880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce97fff640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88004d40 0x7fce880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.668+0000 7fce97fff640 1 -- 172.21.15.77:0/2532355907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce88003520 con 0x7fce88003f20 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce97fff640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce88003f20 0x7fce88004800 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fce840018e0 tx=0x7fce88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0532440839944cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84037070 con 0x7fce88003f20 2024-03-06T07:31:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce84032600 con 0x7fce88003f20 2024-03-06T07:31:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce977fe640 1 -- 172.21.15.77:0/2532355907 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840014e0 con 0x7fce88003f20 2024-03-06T07:31:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce9ea9a640 1 -- 172.21.15.77:0/2532355907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce88003f20 msgr2=0x7fce88004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce9ea9a640 1 --2- 172.21.15.77:0/2532355907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce88003f20 0x7fce88004800 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fce840018e0 tx=0x7fce88005830 comp rx=0 tx=0).stop 2024-03-06T07:31:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce9d037640 1 -- 172.21.15.77:0/2532355907 reap_dead start 2024-03-06T07:31:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce9ea9a640 1 -- 172.21.15.77:0/2532355907 shutdown_connections 2024-03-06T07:31:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce9ea9a640 1 -- 172.21.15.77:0/2532355907 >> 172.21.15.77:0/2532355907 conn(0x7fce980d3cd0 msgr2=0x7fce980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.669+0000 7fce9ea9a640 1 -- 172.21.15.77:0/2532355907 shutdown_connections 2024-03-06T07:31:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.670+0000 7fce9ea9a640 1 -- 172.21.15.77:0/2532355907 wait complete. 2024-03-06T07:31:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.671+0000 7fce9ea9a640 1 Processor -- start 2024-03-06T07:31:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.671+0000 7fce9ea9a640 1 -- start start 2024-03-06T07:31:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.671+0000 7fce9ea9a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88003f20 0x7fce980b92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.671+0000 7fce9ea9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce980b7ef0 0x7fce98152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.671+0000 7fce9ea9a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce98152a40 0x7fce98160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.671+0000 7fce9ea9a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9815e3c0 con 0x7fce98152a40 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.671+0000 7fce9ea9a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b3d60 con 0x7fce88003f20 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.671+0000 7fce9ea9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9815e540 con 0x7fce980b7ef0 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce98152a40 0x7fce98160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce98152a40 0x7fce98160b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32900/0 (socket says 172.21.15.77:32900) 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 -- 172.21.15.77:0/3098295108 learned_addr learned my addr 172.21.15.77:0/3098295108 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce97fff640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce980b7ef0 0x7fce98152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9c836640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88003f20 0x7fce980b92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 -- 172.21.15.77:0/3098295108 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce980b7ef0 msgr2=0x7fce98152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce980b7ef0 0x7fce98152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 -- 172.21.15.77:0/3098295108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88003f20 msgr2=0x7fce980b92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88003f20 0x7fce980b92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 -- 172.21.15.77:0/3098295108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce980cd3b0 con 0x7fce98152a40 2024-03-06T07:31:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9d037640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce98152a40 0x7fce98160b10 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fce9808cb30 tx=0x7fce90007a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce957fa640 1 -- 172.21.15.77:0/3098295108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9000aab0 con 0x7fce98152a40 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce957fa640 1 -- 172.21.15.77:0/3098295108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce90018070 con 0x7fce98152a40 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce980b7410 con 0x7fce98152a40 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce957fa640 1 -- 172.21.15.77:0/3098295108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce900133f0 con 0x7fce98152a40 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.672+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce980b79d0 con 0x7fce98152a40 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.673+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9804f330 con 0x7fce98152a40 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.674+0000 7fce957fa640 1 -- 172.21.15.77:0/3098295108 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce90013590 con 0x7fce98152a40 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.674+0000 7fce957fa640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce7c06b500 0x7fce7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.674+0000 7fce957fa640 1 -- 172.21.15.77:0/3098295108 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fce9008a570 con 0x7fce98152a40 2024-03-06T07:31:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.674+0000 7fce9c836640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce7c06b500 0x7fce7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.675+0000 7fce9c836640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce7c06b500 0x7fce7c06d9c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fce981f7410 tx=0x7fce8c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.679+0000 7fce957fa640 1 -- 172.21.15.77:0/3098295108 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce90059440 con 0x7fce98152a40 2024-03-06T07:31:41.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.895+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce980b4fe0 con 0x7fce98152a40 2024-03-06T07:31:41.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.896+0000 7fce957fa640 1 -- 172.21.15.77:0/3098295108 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fce90058d40 con 0x7fce98152a40 2024-03-06T07:31:41.897 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:41.898 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:41.898 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce7c06b500 msgr2=0x7fce7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce7c06b500 0x7fce7c06d9c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fce981f7410 tx=0x7fce8c009190 comp rx=0 tx=0).stop 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce98152a40 msgr2=0x7fce98160b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce98152a40 0x7fce98160b10 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fce9808cb30 tx=0x7fce90007a60 comp rx=0 tx=0).stop 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 shutdown_connections 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce7c06b500 0x7fce7c06d9c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce98152a40 0x7fce98160b10 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce980b7ef0 0x7fce98152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 --2- 172.21.15.77:0/3098295108 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce88003f20 0x7fce980b92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 >> 172.21.15.77:0/3098295108 conn(0x7fce980d3cd0 msgr2=0x7fce980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 shutdown_connections 2024-03-06T07:31:41.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:41.898+0000 7fce9ea9a640 1 -- 172.21.15.77:0/3098295108 wait complete. 2024-03-06T07:31:42.112 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:42.112 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:43.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:43.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.214+0000 7f4423478640 1 Processor -- start 2024-03-06T07:31:43.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.214+0000 7f4423478640 1 -- start start 2024-03-06T07:31:43.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4423478640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4423478640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f441c0d29c0 0x7f441c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4423478640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c0b3430 con 0x7f441c0cf830 2024-03-06T07:31:43.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4423478640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c0b31e0 con 0x7f441c0d29c0 2024-03-06T07:31:43.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4423478640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0b3d60 con 0x7f441c15dfe0 2024-03-06T07:31:43.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4421a15640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f441c0d29c0 0x7f441c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4421a15640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f441c0d29c0 0x7f441c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32908/0 (socket says 172.21.15.77:32908) 2024-03-06T07:31:43.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4421a15640 1 -- 172.21.15.77:0/3934331869 learned_addr learned my addr 172.21.15.77:0/3934331869 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:43.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4421214640 1 --2- 172.21.15.77:0/3934331869 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4421a15640 1 -- 172.21.15.77:0/3934331869 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 msgr2=0x7f441c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4421a15640 1 --2- 172.21.15.77:0/3934331869 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4421a15640 1 -- 172.21.15.77:0/3934331869 >> v1:172.21.15.158:6789/0 conn(0x7f441c0cf830 legacy=0x7f441c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.215+0000 7f4421a15640 1 -- 172.21.15.77:0/3934331869 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c0b3710 con 0x7f441c0d29c0 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.216+0000 7f4421a15640 1 --2- 172.21.15.77:0/3934331869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0d29c0 0x7f441c0b41f0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f4414009e50 tx=0x7f441402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ecfb820534e4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.216+0000 7f4413fff640 1 -- 172.21.15.77:0/3934331869 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441402cb00 con 0x7f441c0d29c0 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.216+0000 7f4413fff640 1 -- 172.21.15.77:0/3934331869 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441402cca0 con 0x7f441c0d29c0 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.216+0000 7f4413fff640 1 -- 172.21.15.77:0/3934331869 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414034610 con 0x7f441c0d29c0 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.217+0000 7f4423478640 1 -- 172.21.15.77:0/3934331869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0d29c0 msgr2=0x7f441c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.217+0000 7f4423478640 1 --2- 172.21.15.77:0/3934331869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0d29c0 0x7f441c0b41f0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f4414009e50 tx=0x7f441402bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.217+0000 7f4423478640 1 -- 172.21.15.77:0/3934331869 shutdown_connections 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.217+0000 7f4423478640 1 --2- 172.21.15.77:0/3934331869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0d29c0 0x7f441c0b41f0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.217+0000 7f4423478640 1 --2- 172.21.15.77:0/3934331869 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.217+0000 7f4423478640 1 -- 172.21.15.77:0/3934331869 >> 172.21.15.77:0/3934331869 conn(0x7f441c0d3cd0 msgr2=0x7f441c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.217+0000 7f4423478640 1 -- 172.21.15.77:0/3934331869 shutdown_connections 2024-03-06T07:31:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.217+0000 7f4423478640 1 -- 172.21.15.77:0/3934331869 wait complete. 2024-03-06T07:31:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.218+0000 7f4423478640 1 Processor -- start 2024-03-06T07:31:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.218+0000 7f4423478640 1 -- start start 2024-03-06T07:31:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4423478640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0cf830 0x7f441c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4423478640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f441c0d29c0 0x7f441c15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4423478640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4423478640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0d1df0 con 0x7f441c0cf830 2024-03-06T07:31:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4423478640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0d1c70 con 0x7f441c0d29c0 2024-03-06T07:31:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4423478640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c15e3c0 con 0x7f441c15dfe0 2024-03-06T07:31:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421214640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0cf830 0x7f441c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421214640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0cf830 0x7f441c155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32918/0 (socket says 172.21.15.77:32918) 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421214640 1 -- 172.21.15.77:0/2975361476 learned_addr learned my addr 172.21.15.77:0/2975361476 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421a15640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4420a13640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f441c0d29c0 0x7f441c15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421214640 1 -- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 msgr2=0x7f441c15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421214640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c15ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421214640 1 -- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f441c0d29c0 msgr2=0x7f441c15e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421214640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f441c0d29c0 0x7f441c15e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421214640 1 -- 172.21.15.77:0/2975361476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c1563b0 con 0x7f441c0cf830 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4421a15640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c15ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.219+0000 7f4420a13640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f441c0d29c0 0x7f441c15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.220+0000 7f4421214640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0cf830 0x7f441c155b90 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f441800d5f0 tx=0x7f441800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.220+0000 7f4411ffb640 1 -- 172.21.15.77:0/2975361476 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44180125f0 con 0x7f441c0cf830 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.220+0000 7f4411ffb640 1 -- 172.21.15.77:0/2975361476 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4418019070 con 0x7f441c0cf830 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.220+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c0b9a50 con 0x7f441c0cf830 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.220+0000 7f4411ffb640 1 -- 172.21.15.77:0/2975361476 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44180129d0 con 0x7f441c0cf830 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.220+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c15f5e0 con 0x7f441c0cf830 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.221+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c04f330 con 0x7f441c0cf830 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.221+0000 7f4411ffb640 1 -- 172.21.15.77:0/2975361476 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4418012b70 con 0x7f441c0cf830 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.222+0000 7f4411ffb640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f440006b4d0 0x7f440006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.222+0000 7f4411ffb640 1 -- 172.21.15.77:0/2975361476 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f441808a590 con 0x7f441c0cf830 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.222+0000 7f4420a13640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f440006b4d0 0x7f440006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.222+0000 7f4420a13640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f440006b4d0 0x7f440006d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f441c0b5030 tx=0x7f440c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.226+0000 7f4411ffb640 1 -- 172.21.15.77:0/2975361476 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4418059480 con 0x7f441c0cf830 2024-03-06T07:31:43.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.438+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f441c06c570 con 0x7f441c0cf830 2024-03-06T07:31:43.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.439+0000 7f4411ffb640 1 -- 172.21.15.77:0/2975361476 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f4418058d80 con 0x7f441c0cf830 2024-03-06T07:31:43.441 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:43.441 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:43.441 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f440006b4d0 msgr2=0x7f440006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f440006b4d0 0x7f440006d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f441c0b5030 tx=0x7f440c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0cf830 msgr2=0x7f441c155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0cf830 0x7f441c155b90 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f441800d5f0 tx=0x7f441800dab0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 shutdown_connections 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f441c15dfe0 0x7f441c15ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f441c0d29c0 0x7f441c15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f440006b4d0 0x7f440006d990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.440+0000 7f4423478640 1 --2- 172.21.15.77:0/2975361476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441c0cf830 0x7f441c155b90 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.441+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 >> 172.21.15.77:0/2975361476 conn(0x7f441c0d3cd0 msgr2=0x7f441c06ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.441+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 shutdown_connections 2024-03-06T07:31:43.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:43.441+0000 7f4423478640 1 -- 172.21.15.77:0/2975361476 wait complete. 2024-03-06T07:31:43.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:43.655 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:44.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:44.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.757+0000 7f1216cb5640 1 Processor -- start 2024-03-06T07:31:44.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.757+0000 7f1216cb5640 1 -- start start 2024-03-06T07:31:44.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f1216cb5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f12100b3060 0x7f121015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f1216cb5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121015f580 con 0x7f121013dc10 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f1216cb5640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121015b450 con 0x7f121015bf30 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f1216cb5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12100cf3f0 con 0x7f12100b3060 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f1214a51640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f121013dc10 0x7f12100b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58852/0 (socket says 172.21.15.77:58852) 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f1214a51640 1 -- 172.21.15.77:0/3011702586 learned_addr learned my addr 172.21.15.77:0/3011702586 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f120ffff640 1 --2- 172.21.15.77:0/3011702586 >> v2:172.21.15.158:3300/0 conn(0x7f12100b3060 0x7f121015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552423695 0 0) 0x7f121015f580 con 0x7f121013dc10 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11fc003520 con 0x7f121013dc10 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1963660861 0 0) 0x7f11fc003520 con 0x7f121013dc10 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 >> v2:172.21.15.158:3300/0 conn(0x7f12100b3060 msgr2=0x7f121015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f120f7fe640 1 --2- 172.21.15.77:0/3011702586 >> v2:172.21.15.158:3300/0 conn(0x7f12100b3060 0x7f121015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.758+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 >> v1:172.21.15.158:6789/0 conn(0x7f121015bf30 legacy=0x7f121015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121015b730 con 0x7f121013dc10 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f12080019f0 con 0x7f121013dc10 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 >> v1:172.21.15.77:6789/0 conn(0x7f121013dc10 legacy=0x7f12100b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11fc0039b0 0x7f11fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0043b0 0x7f11fc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11fc005120 0x7f11fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12100cf3f0 con 0x7f11fc0039b0 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc00e270 con 0x7f11fc0043b0 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc00e470 con 0x7f11fc005120 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f1214a51640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11fc0039b0 0x7f11fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f120ffff640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0043b0 0x7f11fc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f1215252640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11fc005120 0x7f11fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f1214a51640 1 -- 172.21.15.77:0/3011702586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11fc005120 msgr2=0x7f11fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f1214a51640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11fc005120 0x7f11fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f1214a51640 1 -- 172.21.15.77:0/3011702586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0043b0 msgr2=0x7f11fc004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f1214a51640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0043b0 0x7f11fc004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f1215252640 1 -- 172.21.15.77:0/3011702586 reap_dead start 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.759+0000 7f1214a51640 1 -- 172.21.15.77:0/3011702586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc003520 con 0x7f11fc0039b0 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f1214a51640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11fc0039b0 0x7f11fc003d90 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f120800a860 tx=0x7f11fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963a6613cf20236c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120800a480 con 0x7f11fc0039b0 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1208034630 con 0x7f11fc0039b0 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f120f7fe640 1 -- 172.21.15.77:0/3011702586 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208034a80 con 0x7f11fc0039b0 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f1216cb5640 1 -- 172.21.15.77:0/3011702586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11fc0039b0 msgr2=0x7f11fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f1216cb5640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11fc0039b0 0x7f11fc003d90 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f120800a860 tx=0x7f11fc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f1216cb5640 1 -- 172.21.15.77:0/3011702586 shutdown_connections 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f1216cb5640 1 --2- 172.21.15.77:0/3011702586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11fc0039b0 0x7f11fc003d90 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f1216cb5640 1 -- 172.21.15.77:0/3011702586 >> 172.21.15.77:0/3011702586 conn(0x7f12100cf8a0 msgr2=0x7f12100cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f1216cb5640 1 -- 172.21.15.77:0/3011702586 shutdown_connections 2024-03-06T07:31:44.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.760+0000 7f1216cb5640 1 -- 172.21.15.77:0/3011702586 wait complete. 2024-03-06T07:31:44.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.762+0000 7f1216cb5640 1 Processor -- start 2024-03-06T07:31:44.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.762+0000 7f1216cb5640 1 -- start start 2024-03-06T07:31:44.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.762+0000 7f1216cb5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0039b0 0x7f1210142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.762+0000 7f1216cb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1210141f40 0x7f1210142380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.762+0000 7f1216cb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f121013e750 0x7f121013ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.762+0000 7f1216cb5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12100b2950 con 0x7f1210141f40 2024-03-06T07:31:44.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.762+0000 7f1216cb5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121015b210 con 0x7f11fc0039b0 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.762+0000 7f1216cb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12100b2ad0 con 0x7f121013e750 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1210141f40 0x7f1210142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1210141f40 0x7f1210142380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32930/0 (socket says 172.21.15.77:32930) 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 -- 172.21.15.77:0/3303874012 learned_addr learned my addr 172.21.15.77:0/3303874012 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f1215252640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f121013e750 0x7f121013ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f1214a51640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0039b0 0x7f1210142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 -- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f121013e750 msgr2=0x7f121013ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f121013e750 0x7f121013ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 -- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0039b0 msgr2=0x7f1210142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0039b0 0x7f1210142d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 -- 172.21.15.77:0/3303874012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12101efb70 con 0x7f1210141f40 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f1215252640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f121013e750 0x7f121013ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f1214a51640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0039b0 0x7f1210142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:44.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120ffff640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1210141f40 0x7f1210142380 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f12000114e0 tx=0x7f12000119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120d7fa640 1 -- 172.21.15.77:0/3303874012 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200015870 con 0x7f1210141f40 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120d7fa640 1 -- 172.21.15.77:0/3303874012 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1200015a10 con 0x7f1210141f40 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12100af860 con 0x7f1210141f40 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f120d7fa640 1 -- 172.21.15.77:0/3303874012 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200015dc0 con 0x7f1210141f40 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.763+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12100afe00 con 0x7f1210141f40 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.765+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121004f330 con 0x7f1210141f40 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.765+0000 7f120d7fa640 1 -- 172.21.15.77:0/3303874012 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1200013400 con 0x7f1210141f40 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.765+0000 7f120d7fa640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11f006b4d0 0x7f11f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.765+0000 7f120d7fa640 1 -- 172.21.15.77:0/3303874012 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f120008aa50 con 0x7f1210141f40 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.765+0000 7f1214a51640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11f006b4d0 0x7f11f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.766+0000 7f1214a51640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11f006b4d0 0x7f11f006d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1208034000 tx=0x7f1208009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.773+0000 7f120d7fa640 1 -- 172.21.15.77:0/3303874012 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1200019030 con 0x7f1210141f40 2024-03-06T07:31:44.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.984+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f121006eb00 con 0x7f1210141f40 2024-03-06T07:31:44.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.984+0000 7f120d7fa640 1 -- 172.21.15.77:0/3303874012 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f1200059260 con 0x7f1210141f40 2024-03-06T07:31:44.986 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:44.986 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:44.987 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:44.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11f006b4d0 msgr2=0x7f11f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11f006b4d0 0x7f11f006d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1208034000 tx=0x7f1208009030 comp rx=0 tx=0).stop 2024-03-06T07:31:44.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1210141f40 msgr2=0x7f1210142380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1210141f40 0x7f1210142380 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f12000114e0 tx=0x7f12000119a0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 shutdown_connections 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f11f006b4d0 0x7f11f006d990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1210141f40 0x7f1210142380 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f121013e750 0x7f121013ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 --2- 172.21.15.77:0/3303874012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f11fc0039b0 0x7f1210142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 >> 172.21.15.77:0/3303874012 conn(0x7f12100cf8a0 msgr2=0x7f1210159130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 shutdown_connections 2024-03-06T07:31:44.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:44.986+0000 7f1216cb5640 1 -- 172.21.15.77:0/3303874012 wait complete. 2024-03-06T07:31:45.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:45.200 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:46.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:46.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.299+0000 7fbec6239640 1 Processor -- start 2024-03-06T07:31:46.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.299+0000 7fbec6239640 1 -- start start 2024-03-06T07:31:46.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.300+0000 7fbec6239640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbec00d22b0 0x7fbec00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.300+0000 7fbec6239640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00b3430 con 0x7fbec015dfe0 2024-03-06T07:31:46.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.300+0000 7fbec6239640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00b3710 con 0x7fbec00b41f0 2024-03-06T07:31:46.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.300+0000 7fbec6239640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00d2c50 con 0x7fbec00d22b0 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.300+0000 7fbebf7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbec015dfe0 0x7fbec00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58866/0 (socket says 172.21.15.77:58866) 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.300+0000 7fbebf7fe640 1 -- 172.21.15.77:0/2203276867 learned_addr learned my addr 172.21.15.77:0/2203276867 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebeffd640 1 --2- 172.21.15.77:0/2203276867 >> v2:172.21.15.158:3300/0 conn(0x7fbec00d22b0 0x7fbec00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835045880 0 0) 0x7fbec00b3430 con 0x7fbec015dfe0 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea8003520 con 0x7fbec015dfe0 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1374801361 0 0) 0x7fbea8003520 con 0x7fbec015dfe0 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 >> v2:172.21.15.158:3300/0 conn(0x7fbec00d22b0 msgr2=0x7fbec00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 --2- 172.21.15.77:0/2203276867 >> v2:172.21.15.158:3300/0 conn(0x7fbec00d22b0 0x7fbec00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 >> v1:172.21.15.158:6789/0 conn(0x7fbec00b41f0 legacy=0x7fbec00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec00b39f0 con 0x7fbec015dfe0 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fbeb0001670 con 0x7fbec015dfe0 2024-03-06T07:31:46.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 >> v1:172.21.15.77:6789/0 conn(0x7fbec015dfe0 legacy=0x7fbec00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbea8003a00 0x7fbea8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea8004800 0x7fbea8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea80051c0 0x7fbea800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea800c090 con 0x7fbea8004800 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea800c240 con 0x7fbea80051c0 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea800c440 con 0x7fbea8003a00 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebeffd640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea8004800 0x7fbea8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebf7fe640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbea8003a00 0x7fbea8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebffff640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea80051c0 0x7fbea800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebeffd640 1 -- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbea8003a00 msgr2=0x7fbea8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebeffd640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbea8003a00 0x7fbea8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebeffd640 1 -- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea80051c0 msgr2=0x7fbea800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebeffd640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea80051c0 0x7fbea800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebeffd640 1 -- 172.21.15.77:0/2203276867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea8003520 con 0x7fbea8004800 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebffff640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea80051c0 0x7fbea800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.301+0000 7fbebffff640 1 -- 172.21.15.77:0/2203276867 reap_dead start 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbebeffd640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea8004800 0x7fbea8004c80 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009ac0 tx=0x7fbea8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170f9bdc27453c8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4039070 con 0x7fbea8004800 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb402c6b0 con 0x7fbea8004800 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbebe7fc640 1 -- 172.21.15.77:0/2203276867 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4009690 con 0x7fbea8004800 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbec6239640 1 -- 172.21.15.77:0/2203276867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea8004800 msgr2=0x7fbea8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbec6239640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea8004800 0x7fbea8004c80 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009ac0 tx=0x7fbea8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbec6239640 1 -- 172.21.15.77:0/2203276867 shutdown_connections 2024-03-06T07:31:46.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbec6239640 1 --2- 172.21.15.77:0/2203276867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea8004800 0x7fbea8004c80 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbec6239640 1 -- 172.21.15.77:0/2203276867 >> 172.21.15.77:0/2203276867 conn(0x7fbec00d3cd0 msgr2=0x7fbec00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbec6239640 1 -- 172.21.15.77:0/2203276867 shutdown_connections 2024-03-06T07:31:46.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.303+0000 7fbec6239640 1 -- 172.21.15.77:0/2203276867 wait complete. 2024-03-06T07:31:46.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.305+0000 7fbec6239640 1 Processor -- start 2024-03-06T07:31:46.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.305+0000 7fbec6239640 1 -- start start 2024-03-06T07:31:46.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbec6239640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea8004800 0x7fbec0152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbec6239640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbec0152e90 0x7fbec00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbec6239640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbec00b8f50 0x7fbec00b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbec6239640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00b3d60 con 0x7fbec00b8f50 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbec6239640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00b31e0 con 0x7fbea8004800 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbec6239640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec015e3c0 con 0x7fbec0152e90 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbec00b8f50 0x7fbec00b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbec00b8f50 0x7fbec00b93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32944/0 (socket says 172.21.15.77:32944) 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 -- 172.21.15.77:0/898389378 learned_addr learned my addr 172.21.15.77:0/898389378 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebf7fe640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea8004800 0x7fbec0152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebeffd640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbec0152e90 0x7fbec00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 -- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbec0152e90 msgr2=0x7fbec00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbec0152e90 0x7fbec00cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 -- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea8004800 msgr2=0x7fbec0152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea8004800 0x7fbec0152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 -- 172.21.15.77:0/898389378 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec00cb710 con 0x7fbec00b8f50 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebf7fe640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea8004800 0x7fbec0152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbebffff640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbec00b8f50 0x7fbec00b93d0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fbec0161bf0 tx=0x7fbeb800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbea7fff640 1 -- 172.21.15.77:0/898389378 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8003970 con 0x7fbec00b8f50 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbea7fff640 1 -- 172.21.15.77:0/898389378 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb8018070 con 0x7fbec00b8f50 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec00cb9f0 con 0x7fbec00b8f50 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbea7fff640 1 -- 172.21.15.77:0/898389378 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8003d50 con 0x7fbec00b8f50 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.306+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec00cbfb0 con 0x7fbec00b8f50 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.308+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec004f330 con 0x7fbec00b8f50 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.308+0000 7fbea7fff640 1 -- 172.21.15.77:0/898389378 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbeb80050d0 con 0x7fbec00b8f50 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.308+0000 7fbea7fff640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbeac06b4d0 0x7fbeac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.309+0000 7fbebf7fe640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbeac06b4d0 0x7fbeac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.309+0000 7fbea7fff640 1 -- 172.21.15.77:0/898389378 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbeb8089d40 con 0x7fbec00b8f50 2024-03-06T07:31:46.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.309+0000 7fbebf7fe640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbeac06b4d0 0x7fbeac06d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0000c00 tx=0x7fbeb0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.313+0000 7fbea7fff640 1 -- 172.21.15.77:0/898389378 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbeb8058cf0 con 0x7fbec00b8f50 2024-03-06T07:31:46.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.528+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbec006bc10 con 0x7fbec00b8f50 2024-03-06T07:31:46.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.529+0000 7fbea7fff640 1 -- 172.21.15.77:0/898389378 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fbeb801be40 con 0x7fbec00b8f50 2024-03-06T07:31:46.531 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:46.531 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:46.531 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbeac06b4d0 msgr2=0x7fbeac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbeac06b4d0 0x7fbeac06d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0000c00 tx=0x7fbeb0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbec00b8f50 msgr2=0x7fbec00b93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbec00b8f50 0x7fbec00b93d0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fbec0161bf0 tx=0x7fbeb800eb90 comp rx=0 tx=0).stop 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 shutdown_connections 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbeac06b4d0 0x7fbeac06d990 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbec00b8f50 0x7fbec00b93d0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbec0152e90 0x7fbec00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 --2- 172.21.15.77:0/898389378 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbea8004800 0x7fbec0152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 >> 172.21.15.77:0/898389378 conn(0x7fbec00d3cd0 msgr2=0x7fbec00d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 shutdown_connections 2024-03-06T07:31:46.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:46.531+0000 7fbec6239640 1 -- 172.21.15.77:0/898389378 wait complete. 2024-03-06T07:31:46.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:46.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:47.747 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:47.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.868+0000 7f8298be9640 1 Processor -- start 2024-03-06T07:31:47.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.868+0000 7f8298be9640 1 -- start start 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.869+0000 7f8298be9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82940d22b0 0x7f82940d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.869+0000 7f8298be9640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82940b3430 con 0x7f829415dfe0 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.869+0000 7f8298be9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82940b3710 con 0x7f82940b41f0 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.869+0000 7f8298be9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82940d2c50 con 0x7f82940d22b0 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.869+0000 7f829259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f829415dfe0 0x7f82940cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:58876/0 (socket says 172.21.15.77:58876) 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.869+0000 7f829259c640 1 -- 172.21.15.77:0/4288841231 learned_addr learned my addr 172.21.15.77:0/4288841231 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.869+0000 7f8289d9b640 1 --2- 172.21.15.77:0/4288841231 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82940d22b0 0x7f82940d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.869+0000 7f8291d9b640 1 -- 172.21.15.77:0/4288841231 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921824449 0 0) 0x7f82940b3430 con 0x7f829415dfe0 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.870+0000 7f8291d9b640 1 -- 172.21.15.77:0/4288841231 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8280003520 con 0x7f829415dfe0 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.870+0000 7f8291d9b640 1 -- 172.21.15.77:0/4288841231 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245967327 0 0) 0x7f82940b3710 con 0x7f82940b41f0 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.870+0000 7f8291d9b640 1 -- 172.21.15.77:0/4288841231 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82940b3430 con 0x7f82940b41f0 2024-03-06T07:31:47.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.870+0000 7f8291d9b640 1 -- 172.21.15.77:0/4288841231 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182632495 0 0) 0x7f8280003520 con 0x7f829415dfe0 2024-03-06T07:31:47.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.870+0000 7f8289d9b640 1 -- 172.21.15.77:0/4288841231 >> v1:172.21.15.158:6789/0 conn(0x7f82940b41f0 legacy=0x7f82940b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.870+0000 7f8289d9b640 1 -- 172.21.15.77:0/4288841231 >> v1:172.21.15.77:6789/0 conn(0x7f829415dfe0 legacy=0x7f82940cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.870+0000 7f8289d9b640 1 -- 172.21.15.77:0/4288841231 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82940b39f0 con 0x7f82940d22b0 2024-03-06T07:31:47.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.871+0000 7f8289d9b640 1 --2- 172.21.15.77:0/4288841231 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82940d22b0 0x7f82940d2710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f827c0097d0 tx=0x7f827c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c07de40c101f09b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.871+0000 7f8291d9b640 1 -- 172.21.15.77:0/4288841231 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c037070 con 0x7f82940d22b0 2024-03-06T07:31:47.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.871+0000 7f8291d9b640 1 -- 172.21.15.77:0/4288841231 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c030510 con 0x7f82940d22b0 2024-03-06T07:31:47.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.871+0000 7f8291d9b640 1 -- 172.21.15.77:0/4288841231 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284001670 con 0x7f82940d22b0 2024-03-06T07:31:47.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.872+0000 7f8298be9640 1 -- 172.21.15.77:0/4288841231 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82940d22b0 msgr2=0x7f82940d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.872+0000 7f8298be9640 1 --2- 172.21.15.77:0/4288841231 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82940d22b0 0x7f82940d2710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f827c0097d0 tx=0x7f827c02b800 comp rx=0 tx=0).stop 2024-03-06T07:31:47.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.872+0000 7f8298be9640 1 -- 172.21.15.77:0/4288841231 shutdown_connections 2024-03-06T07:31:47.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.872+0000 7f8298be9640 1 --2- 172.21.15.77:0/4288841231 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82940d22b0 0x7f82940d2710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.872+0000 7f8298be9640 1 -- 172.21.15.77:0/4288841231 >> 172.21.15.77:0/4288841231 conn(0x7f82940d3cd0 msgr2=0x7f82940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.872+0000 7f8298be9640 1 -- 172.21.15.77:0/4288841231 shutdown_connections 2024-03-06T07:31:47.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.872+0000 7f8298be9640 1 -- 172.21.15.77:0/4288841231 wait complete. 2024-03-06T07:31:47.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.874+0000 7f8298be9640 1 Processor -- start 2024-03-06T07:31:47.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.874+0000 7f8298be9640 1 -- start start 2024-03-06T07:31:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8298be9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940b41f0 0x7f829414ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8298be9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82940d22b0 0x7f82940b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8298be9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f829415dfe0 0x7f82941f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8298be9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82940b3d60 con 0x7f82940d22b0 2024-03-06T07:31:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8298be9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82940b31e0 con 0x7f82940b41f0 2024-03-06T07:31:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8298be9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82940d1c70 con 0x7f829415dfe0 2024-03-06T07:31:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8289d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82940d22b0 0x7f82940b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8289d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82940d22b0 0x7f82940b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32956/0 (socket says 172.21.15.77:32956) 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8289d9b640 1 -- 172.21.15.77:0/447123588 learned_addr learned my addr 172.21.15.77:0/447123588 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f829259c640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940b41f0 0x7f829414ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8292d9d640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f829415dfe0 0x7f82941f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8289d9b640 1 -- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f829415dfe0 msgr2=0x7f82941f7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8289d9b640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f829415dfe0 0x7f82941f7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8289d9b640 1 -- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940b41f0 msgr2=0x7f829414ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8289d9b640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940b41f0 0x7f829414ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.875+0000 7f8289d9b640 1 -- 172.21.15.77:0/447123588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82940b9430 con 0x7f82940d22b0 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.876+0000 7f829259c640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940b41f0 0x7f829414ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.876+0000 7f8292d9d640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f829415dfe0 0x7f82941f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.876+0000 7f8289d9b640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82940d22b0 0x7f82940b68a0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f827c000c00 tx=0x7f827c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.876+0000 7f828b7fe640 1 -- 172.21.15.77:0/447123588 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c037040 con 0x7f82940d22b0 2024-03-06T07:31:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.876+0000 7f828b7fe640 1 -- 172.21.15.77:0/447123588 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c004250 con 0x7f82940d22b0 2024-03-06T07:31:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.876+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82940b9710 con 0x7f82940d22b0 2024-03-06T07:31:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.876+0000 7f828b7fe640 1 -- 172.21.15.77:0/447123588 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c03c670 con 0x7f82940d22b0 2024-03-06T07:31:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.876+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82940cc040 con 0x7f82940d22b0 2024-03-06T07:31:47.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.877+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82940d3740 con 0x7f82940d22b0 2024-03-06T07:31:47.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.878+0000 7f828b7fe640 1 -- 172.21.15.77:0/447123588 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f827c03a070 con 0x7f82940d22b0 2024-03-06T07:31:47.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.878+0000 7f828b7fe640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f827406b4d0 0x7f827406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.878+0000 7f828b7fe640 1 -- 172.21.15.77:0/447123588 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f827c0abb30 con 0x7f82940d22b0 2024-03-06T07:31:47.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.878+0000 7f829259c640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f827406b4d0 0x7f827406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.879+0000 7f829259c640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f827406b4d0 0x7f827406d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8284001340 tx=0x7f8284009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:47.883+0000 7f828b7fe640 1 -- 172.21.15.77:0/447123588 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82940d3740 con 0x7f82940d22b0 2024-03-06T07:31:48.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.096+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f829404f330 con 0x7f82940d22b0 2024-03-06T07:31:48.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.096+0000 7f828b7fe640 1 -- 172.21.15.77:0/447123588 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f827c041020 con 0x7f82940d22b0 2024-03-06T07:31:48.098 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:48.098 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:48.099 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:48.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f827406b4d0 msgr2=0x7f827406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f827406b4d0 0x7f827406d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8284001340 tx=0x7f8284009040 comp rx=0 tx=0).stop 2024-03-06T07:31:48.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82940d22b0 msgr2=0x7f82940b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82940d22b0 0x7f82940b68a0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f827c000c00 tx=0x7f827c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 shutdown_connections 2024-03-06T07:31:48.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f827406b4d0 0x7f827406d990 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82940d22b0 0x7f82940b68a0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f829415dfe0 0x7f82941f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 --2- 172.21.15.77:0/447123588 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82940b41f0 0x7f829414ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 >> 172.21.15.77:0/447123588 conn(0x7f82940d3cd0 msgr2=0x7f82940d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 shutdown_connections 2024-03-06T07:31:48.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:48.099+0000 7f8298be9640 1 -- 172.21.15.77:0/447123588 wait complete. 2024-03-06T07:31:48.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:48.313 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:49.314 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:49.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.413+0000 7f72ec6a4640 1 Processor -- start 2024-03-06T07:31:49.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.413+0000 7f72ec6a4640 1 -- start start 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.414+0000 7f72ec6a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e415dfe0 0x7f72e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.414+0000 7f72ec6a4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72e40d29c0 0x7f72e40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.414+0000 7f72ec6a4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40b3430 con 0x7f72e40cf830 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.414+0000 7f72ec6a4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e40b31e0 con 0x7f72e40d29c0 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.414+0000 7f72ec6a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e40b3d60 con 0x7f72e415dfe0 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72eac41640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72e40d29c0 0x7f72e40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72ea440640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e415dfe0 0x7f72e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72eac41640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f72e40d29c0 0x7f72e40b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:57688/0 (socket says 172.21.15.77:57688) 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72ea440640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e415dfe0 0x7f72e40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:60826/0 (socket says 172.21.15.77:60826) 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72eac41640 1 -- 172.21.15.77:0/666513321 learned_addr learned my addr 172.21.15.77:0/666513321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:49.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72e9c3f640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f72e40cf830 0x7f72e40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:36858/0 (socket says 172.21.15.77:36858) 2024-03-06T07:31:49.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72ea440640 1 -- 172.21.15.77:0/666513321 >> v2:172.21.15.158:3300/0 conn(0x7f72e40d29c0 msgr2=0x7f72e40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72ea440640 1 --2- 172.21.15.77:0/666513321 >> v2:172.21.15.158:3300/0 conn(0x7f72e40d29c0 0x7f72e40b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72ea440640 1 -- 172.21.15.77:0/666513321 >> v1:172.21.15.158:6789/0 conn(0x7f72e40cf830 legacy=0x7f72e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72ea440640 1 -- 172.21.15.77:0/666513321 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e40b3710 con 0x7f72e415dfe0 2024-03-06T07:31:49.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.415+0000 7f72eac41640 1 --2- 172.21.15.77:0/666513321 >> v2:172.21.15.158:3300/0 conn(0x7f72e40d29c0 0x7f72e40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:49.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72ea440640 1 --2- 172.21.15.77:0/666513321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e415dfe0 0x7f72e40cf2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f72e00097a0 tx=0x7f72e002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce090e44310356b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72e943e640 1 -- 172.21.15.77:0/666513321 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0035020 con 0x7f72e415dfe0 2024-03-06T07:31:49.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72e943e640 1 -- 172.21.15.77:0/666513321 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0031830 con 0x7f72e415dfe0 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72e943e640 1 -- 172.21.15.77:0/666513321 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0031c00 con 0x7f72e415dfe0 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72ec6a4640 1 -- 172.21.15.77:0/666513321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e415dfe0 msgr2=0x7f72e40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/666513321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e415dfe0 0x7f72e40cf2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f72e00097a0 tx=0x7f72e002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72ec6a4640 1 -- 172.21.15.77:0/666513321 shutdown_connections 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/666513321 >> v2:172.21.15.158:3300/0 conn(0x7f72e40d29c0 0x7f72e40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.416+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/666513321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e415dfe0 0x7f72e40cf2f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.417+0000 7f72ec6a4640 1 -- 172.21.15.77:0/666513321 >> 172.21.15.77:0/666513321 conn(0x7f72e40d3cd0 msgr2=0x7f72e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.417+0000 7f72ec6a4640 1 -- 172.21.15.77:0/666513321 shutdown_connections 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.417+0000 7f72ec6a4640 1 -- 172.21.15.77:0/666513321 wait complete. 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ec6a4640 1 Processor -- start 2024-03-06T07:31:49.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ec6a4640 1 -- start start 2024-03-06T07:31:49.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ec6a4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e40cf830 0x7f72e414f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ec6a4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e40d29c0 0x7f72e40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ec6a4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e415dfe0 0x7f72e40cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ec6a4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e40d1df0 con 0x7f72e40cf830 2024-03-06T07:31:49.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ec6a4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e40d1c70 con 0x7f72e415dfe0 2024-03-06T07:31:49.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ec6a4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e415e3c0 con 0x7f72e40d29c0 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ea440640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e40cf830 0x7f72e414f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ea440640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e40cf830 0x7f72e414f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43128/0 (socket says 172.21.15.77:43128) 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.418+0000 7f72ea440640 1 -- 172.21.15.77:0/2470198831 learned_addr learned my addr 172.21.15.77:0/2470198831 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72e9c3f640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e40d29c0 0x7f72e40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72eac41640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e415dfe0 0x7f72e40cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72ea440640 1 -- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e40d29c0 msgr2=0x7f72e40b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72ea440640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e40d29c0 0x7f72e40b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72ea440640 1 -- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e415dfe0 msgr2=0x7f72e40cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72ea440640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e415dfe0 0x7f72e40cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72ea440640 1 -- 172.21.15.77:0/2470198831 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e414f6d0 con 0x7f72e40cf830 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72e9c3f640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e40d29c0 0x7f72e40b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72eac41640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e415dfe0 0x7f72e40cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72ea440640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e40cf830 0x7f72e414f190 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f72e00098c0 tx=0x7f72e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72daffd640 1 -- 172.21.15.77:0/2470198831 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0040070 con 0x7f72e40cf830 2024-03-06T07:31:49.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72daffd640 1 -- 172.21.15.77:0/2470198831 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e003e070 con 0x7f72e40cf830 2024-03-06T07:31:49.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e40cc6d0 con 0x7f72e40cf830 2024-03-06T07:31:49.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72daffd640 1 -- 172.21.15.77:0/2470198831 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0035040 con 0x7f72e40cf830 2024-03-06T07:31:49.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.419+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e40ccc90 con 0x7f72e40cf830 2024-03-06T07:31:49.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.420+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e404f330 con 0x7f72e40cf830 2024-03-06T07:31:49.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.421+0000 7f72daffd640 1 -- 172.21.15.77:0/2470198831 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72e0038020 con 0x7f72e40cf830 2024-03-06T07:31:49.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.421+0000 7f72daffd640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.421+0000 7f72daffd640 1 -- 172.21.15.77:0/2470198831 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f72e00ab5a0 con 0x7f72e40cf830 2024-03-06T07:31:49.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.421+0000 7f72e9c3f640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.422+0000 7f72e9c3f640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f72d4007fd0 tx=0x7f72d4007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.425+0000 7f72daffd640 1 -- 172.21.15.77:0/2470198831 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72e0031800 con 0x7f72e40cf830 2024-03-06T07:31:49.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.641+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72e406c450 con 0x7f72e40cf830 2024-03-06T07:31:49.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.641+0000 7f72daffd640 1 -- 172.21.15.77:0/2470198831 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f72e0079db0 con 0x7f72e40cf830 2024-03-06T07:31:49.643 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:49.643 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:49.644 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:49.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 msgr2=0x7f72c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f72d4007fd0 tx=0x7f72d4007e80 comp rx=0 tx=0).stop 2024-03-06T07:31:49.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e40cf830 msgr2=0x7f72e414f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e40cf830 0x7f72e414f190 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f72e00098c0 tx=0x7f72e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 shutdown_connections 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f72e415dfe0 0x7f72e40cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f72e40d29c0 0x7f72e40b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f72c806b4d0 0x7f72c806d990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 --2- 172.21.15.77:0/2470198831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e40cf830 0x7f72e414f190 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 >> 172.21.15.77:0/2470198831 conn(0x7f72e40d3cd0 msgr2=0x7f72e40b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 shutdown_connections 2024-03-06T07:31:49.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:49.643+0000 7f72ec6a4640 1 -- 172.21.15.77:0/2470198831 wait complete. 2024-03-06T07:31:49.857 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:49.857 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:50.858 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:50.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.958+0000 7fcda06af640 1 Processor -- start 2024-03-06T07:31:50.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.958+0000 7fcda06af640 1 -- start start 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcda06af640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd980cce40 0x7fcd9815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcda06af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd980cd7a0 0x7fcd980cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcda06af640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd98071f40 con 0x7fcd981515e0 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcda06af640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd980cd2f0 con 0x7fcd980cce40 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcda06af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98070a70 con 0x7fcd980cd7a0 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcd9dc4a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd980cce40 0x7fcd9815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcd9dc4a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd980cce40 0x7fcd9815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43138/0 (socket says 172.21.15.77:43138) 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcd9dc4a640 1 -- 172.21.15.77:0/4122573310 learned_addr learned my addr 172.21.15.77:0/4122573310 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcd9dc4a640 1 -- 172.21.15.77:0/4122573310 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd980cd7a0 msgr2=0x7fcd980cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcd9dc4a640 1 --2- 172.21.15.77:0/4122573310 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd980cd7a0 0x7fcd980cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcd9dc4a640 1 -- 172.21.15.77:0/4122573310 >> v1:172.21.15.158:6789/0 conn(0x7fcd981515e0 legacy=0x7fcd98071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.959+0000 7fcd9dc4a640 1 -- 172.21.15.77:0/4122573310 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd98070d50 con 0x7fcd980cce40 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.960+0000 7fcd9dc4a640 1 --2- 172.21.15.77:0/4122573310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815b430 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fcd980cc110 tx=0x7fcd8c02e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f4962bb78e510e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.960+0000 7fcd9d449640 1 -- 172.21.15.77:0/4122573310 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c03b070 con 0x7fcd980cce40 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.960+0000 7fcd9d449640 1 -- 172.21.15.77:0/4122573310 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c036440 con 0x7fcd980cce40 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.960+0000 7fcd9d449640 1 -- 172.21.15.77:0/4122573310 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c02ee60 con 0x7fcd980cce40 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.961+0000 7fcda06af640 1 -- 172.21.15.77:0/4122573310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 msgr2=0x7fcd9815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.961+0000 7fcda06af640 1 --2- 172.21.15.77:0/4122573310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815b430 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fcd980cc110 tx=0x7fcd8c02e0b0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.961+0000 7fcda06af640 1 -- 172.21.15.77:0/4122573310 shutdown_connections 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.961+0000 7fcda06af640 1 --2- 172.21.15.77:0/4122573310 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd980cd7a0 0x7fcd980cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.961+0000 7fcda06af640 1 --2- 172.21.15.77:0/4122573310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815b430 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.961+0000 7fcda06af640 1 -- 172.21.15.77:0/4122573310 >> 172.21.15.77:0/4122573310 conn(0x7fcd980cf8a0 msgr2=0x7fcd980cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.961+0000 7fcda06af640 1 -- 172.21.15.77:0/4122573310 shutdown_connections 2024-03-06T07:31:50.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.961+0000 7fcda06af640 1 -- 172.21.15.77:0/4122573310 wait complete. 2024-03-06T07:31:50.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.962+0000 7fcda06af640 1 Processor -- start 2024-03-06T07:31:50.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.962+0000 7fcda06af640 1 -- start start 2024-03-06T07:31:50.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcda06af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcda06af640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd980cd7a0 0x7fcd980caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcda06af640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd981515e0 0x7fcd98144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcda06af640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd980ce0f0 con 0x7fcd980cce40 2024-03-06T07:31:50.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcda06af640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98071310 con 0x7fcd980cd7a0 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcda06af640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd980cc3a0 con 0x7fcd981515e0 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43152/0 (socket says 172.21.15.77:43152) 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 -- 172.21.15.77:0/1854429557 learned_addr learned my addr 172.21.15.77:0/1854429557 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9ec4c640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd981515e0 0x7fcd98144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 -- 172.21.15.77:0/1854429557 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd981515e0 msgr2=0x7fcd98144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9dc4a640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd980cd7a0 0x7fcd980caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd981515e0 0x7fcd98144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 -- 172.21.15.77:0/1854429557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd980cd7a0 msgr2=0x7fcd980caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd980cd7a0 0x7fcd980caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 -- 172.21.15.77:0/1854429557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9815e6d0 con 0x7fcd980cce40 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd9e44b640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815ce20 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400b3e0 tx=0x7fcd9400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd8affd640 1 -- 172.21.15.77:0/1854429557 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94015020 con 0x7fcd980cce40 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd8affd640 1 -- 172.21.15.77:0/1854429557 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94011850 con 0x7fcd980cce40 2024-03-06T07:31:50.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcd8affd640 1 -- 172.21.15.77:0/1854429557 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94011c40 con 0x7fcd980cce40 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.963+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9815d360 con 0x7fcd980cce40 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.964+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9815d910 con 0x7fcd980cce40 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.965+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd98068840 con 0x7fcd980cce40 2024-03-06T07:31:50.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.965+0000 7fcd8affd640 1 -- 172.21.15.77:0/1854429557 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd94018020 con 0x7fcd980cce40 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.966+0000 7fcd8affd640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.966+0000 7fcd9dc4a640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.966+0000 7fcd8affd640 1 -- 172.21.15.77:0/1854429557 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcd9408a140 con 0x7fcd980cce40 2024-03-06T07:31:50.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.966+0000 7fcd9dc4a640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c038000 tx=0x7fcd8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:50.970+0000 7fcd8affd640 1 -- 172.21.15.77:0/1854429557 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd9401f020 con 0x7fcd980cce40 2024-03-06T07:31:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.185+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd9804f330 con 0x7fcd980cce40 2024-03-06T07:31:51.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.185+0000 7fcd8affd640 1 -- 172.21.15.77:0/1854429557 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fcd940172c0 con 0x7fcd980cce40 2024-03-06T07:31:51.187 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:51.187 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:51.188 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd7c06b4d0 msgr2=0x7fcd7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c038000 tx=0x7fcd8c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 msgr2=0x7fcd9815ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815ce20 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400b3e0 tx=0x7fcd9400b8a0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 shutdown_connections 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcd981515e0 0x7fcd98144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcd980cd7a0 0x7fcd980caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcd7c06b4d0 0x7fcd7c06d990 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 --2- 172.21.15.77:0/1854429557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd980cce40 0x7fcd9815ce20 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 >> 172.21.15.77:0/1854429557 conn(0x7fcd980cf8a0 msgr2=0x7fcd98159cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 shutdown_connections 2024-03-06T07:31:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:51.188+0000 7fcda06af640 1 -- 172.21.15.77:0/1854429557 wait complete. 2024-03-06T07:31:51.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:51.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:52.404 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:52.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.516+0000 7f0d2be83640 1 Processor -- start 2024-03-06T07:31:52.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.516+0000 7f0d2be83640 1 -- start start 2024-03-06T07:31:52.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d2be83640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d2415ddf0 0x7f0d240cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d2be83640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240d18d0 con 0x7f0d2415e1d0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d2be83640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240d1bb0 con 0x7f0d240d0de0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d2be83640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240d27c0 con 0x7f0d2415ddf0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d2941e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0d2415e1d0 0x7f0d240d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51082/0 (socket says 172.21.15.77:51082) 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d2941e640 1 -- 172.21.15.77:0/2785659039 learned_addr learned my addr 172.21.15.77:0/2785659039 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208856568 0 0) 0x7f0d240d18d0 con 0x7f0d2415e1d0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d10003520 con 0x7f0d2415e1d0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d29c1f640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d2415ddf0 0x7f0d240cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592940784 0 0) 0x7f0d10003520 con 0x7f0d2415e1d0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d2415ddf0 msgr2=0x7f0d240cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d2415ddf0 0x7f0d240cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 >> v1:172.21.15.158:6789/0 conn(0x7f0d240d0de0 legacy=0x7f0d240b6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d240d1e90 con 0x7f0d2415e1d0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f0d14001670 con 0x7f0d2415e1d0 2024-03-06T07:31:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 >> v1:172.21.15.77:6789/0 conn(0x7f0d2415e1d0 legacy=0x7f0d240d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d2415ddf0 0x7f0d10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.517+0000 7f0d28c1d640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d10004d40 0x7f0d100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240d27c0 con 0x7f0d100039e0 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1000ba70 con 0x7f0d10004d40 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1000bc70 con 0x7f0d2415ddf0 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d29c1f640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d2941e640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d2415ddf0 0x7f0d10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d2a420640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d10004d40 0x7f0d100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d29c1f640 1 -- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d2415ddf0 msgr2=0x7f0d10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d29c1f640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d2415ddf0 0x7f0d10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d29c1f640 1 -- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d10004d40 msgr2=0x7f0d100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d29c1f640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d10004d40 0x7f0d100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.518+0000 7f0d29c1f640 1 -- 172.21.15.77:0/2785659039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d10003520 con 0x7f0d100039e0 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d29c1f640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d10003e20 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009a90 tx=0x7f0d10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbb961bea0d908c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d20039070 con 0x7f0d100039e0 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2002c680 con 0x7f0d100039e0 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d28c1d640 1 -- 172.21.15.77:0/2785659039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d20009690 con 0x7f0d100039e0 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d2be83640 1 -- 172.21.15.77:0/2785659039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 msgr2=0x7f0d10003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d2be83640 1 --2- 172.21.15.77:0/2785659039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d10003e20 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009a90 tx=0x7f0d10005830 comp rx=0 tx=0).stop 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d2a420640 1 -- 172.21.15.77:0/2785659039 reap_dead start 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d2be83640 1 -- 172.21.15.77:0/2785659039 shutdown_connections 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d2be83640 1 -- 172.21.15.77:0/2785659039 >> 172.21.15.77:0/2785659039 conn(0x7f0d2409cff0 msgr2=0x7f0d240d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d2be83640 1 -- 172.21.15.77:0/2785659039 shutdown_connections 2024-03-06T07:31:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.519+0000 7f0d2be83640 1 -- 172.21.15.77:0/2785659039 wait complete. 2024-03-06T07:31:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.520+0000 7f0d2be83640 1 Processor -- start 2024-03-06T07:31:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.520+0000 7f0d2be83640 1 -- start start 2024-03-06T07:31:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d2be83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d2415b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d2be83640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d240ca220 0x7f0d2409fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d2be83640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d240cadd0 0x7f0d240c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d2be83640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240d2c50 con 0x7f0d100039e0 2024-03-06T07:31:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d2be83640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240d2450 con 0x7f0d240ca220 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d2be83640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240b7260 con 0x7f0d240cadd0 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d29c1f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d2415b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d29c1f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d2415b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43172/0 (socket says 172.21.15.77:43172) 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d29c1f640 1 -- 172.21.15.77:0/1287237092 learned_addr learned my addr 172.21.15.77:0/1287237092 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d2a420640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d240cadd0 0x7f0d240c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.521+0000 7f0d2941e640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d240ca220 0x7f0d2409fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d29c1f640 1 -- 172.21.15.77:0/1287237092 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d240cadd0 msgr2=0x7f0d240c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d29c1f640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d240cadd0 0x7f0d240c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d29c1f640 1 -- 172.21.15.77:0/1287237092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d240ca220 msgr2=0x7f0d2409fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d29c1f640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d240ca220 0x7f0d2409fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d29c1f640 1 -- 172.21.15.77:0/1287237092 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d240cbd00 con 0x7f0d100039e0 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d29c1f640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d2415b160 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f0d20000c00 tx=0x7f0d20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d1a7fc640 1 -- 172.21.15.77:0/1287237092 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d20042020 con 0x7f0d100039e0 2024-03-06T07:31:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d1a7fc640 1 -- 172.21.15.77:0/1287237092 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2003a040 con 0x7f0d100039e0 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d240cb740 con 0x7f0d100039e0 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d1a7fc640 1 -- 172.21.15.77:0/1287237092 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d20039070 con 0x7f0d100039e0 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.522+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d240cc010 con 0x7f0d100039e0 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.523+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2404f330 con 0x7f0d100039e0 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.523+0000 7f0d1a7fc640 1 -- 172.21.15.77:0/1287237092 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d20045020 con 0x7f0d100039e0 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.524+0000 7f0d1a7fc640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d0806b500 0x7f0d0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.524+0000 7f0d1a7fc640 1 -- 172.21.15.77:0/1287237092 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0d200ab510 con 0x7f0d100039e0 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.524+0000 7f0d2941e640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d0806b500 0x7f0d0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.524+0000 7f0d2941e640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d0806b500 0x7f0d0806d9c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0d240b5180 tx=0x7f0d14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.532+0000 7f0d1a7fc640 1 -- 172.21.15.77:0/1287237092 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d2004c070 con 0x7f0d100039e0 2024-03-06T07:31:52.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.755+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d240d1290 con 0x7f0d100039e0 2024-03-06T07:31:52.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.756+0000 7f0d1a7fc640 1 -- 172.21.15.77:0/1287237092 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f0d2007a400 con 0x7f0d100039e0 2024-03-06T07:31:52.758 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:52.758 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d0806b500 msgr2=0x7f0d0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d0806b500 0x7f0d0806d9c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0d240b5180 tx=0x7f0d14009040 comp rx=0 tx=0).stop 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 msgr2=0x7f0d2415b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d2415b160 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f0d20000c00 tx=0x7f0d20004730 comp rx=0 tx=0).stop 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 shutdown_connections 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0d240cadd0 0x7f0d240c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0d240ca220 0x7f0d2409fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0d0806b500 0x7f0d0806d9c0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 --2- 172.21.15.77:0/1287237092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d100039e0 0x7f0d2415b160 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 >> 172.21.15.77:0/1287237092 conn(0x7f0d2409cff0 msgr2=0x7f0d2406b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 shutdown_connections 2024-03-06T07:31:52.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:52.758+0000 7f0d2be83640 1 -- 172.21.15.77:0/1287237092 wait complete. 2024-03-06T07:31:52.972 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:52.972 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:53.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:54.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.072+0000 7f0b75a8f640 1 Processor -- start 2024-03-06T07:31:54.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.072+0000 7f0b75a8f640 1 -- start start 2024-03-06T07:31:54.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b75a8f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0b7015dfe0 0x7f0b700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b75a8f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0b700d29c0 0x7f0b700b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b75a8f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b700b3430 con 0x7f0b700cf830 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b75a8f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b700b31e0 con 0x7f0b7015dfe0 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b75a8f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b700b3d60 con 0x7f0b700d29c0 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6f7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0b7015dfe0 0x7f0b700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6f7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0b7015dfe0 0x7f0b700cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43178/0 (socket says 172.21.15.77:43178) 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b67fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0b700cf830 0x7f0b700d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51098/0 (socket says 172.21.15.77:51098) 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6f7fe640 1 -- 172.21.15.77:0/3468168940 learned_addr learned my addr 172.21.15.77:0/3468168940 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6ffff640 1 --2- 172.21.15.77:0/3468168940 >> v2:172.21.15.158:3300/0 conn(0x7f0b700d29c0 0x7f0b700b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6f7fe640 1 -- 172.21.15.77:0/3468168940 >> v2:172.21.15.158:3300/0 conn(0x7f0b700d29c0 msgr2=0x7f0b700b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6f7fe640 1 --2- 172.21.15.77:0/3468168940 >> v2:172.21.15.158:3300/0 conn(0x7f0b700d29c0 0x7f0b700b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6effd640 1 -- 172.21.15.77:0/3468168940 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298376766 0 0) 0x7f0b700b3430 con 0x7f0b700cf830 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6f7fe640 1 -- 172.21.15.77:0/3468168940 >> v1:172.21.15.77:6789/0 conn(0x7f0b700cf830 legacy=0x7f0b700d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6f7fe640 1 -- 172.21.15.77:0/3468168940 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700b3710 con 0x7f0b7015dfe0 2024-03-06T07:31:54.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.073+0000 7f0b6ffff640 1 --2- 172.21.15.77:0/3468168940 >> v2:172.21.15.158:3300/0 conn(0x7f0b700d29c0 0x7f0b700b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.074+0000 7f0b6f7fe640 1 --2- 172.21.15.77:0/3468168940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015dfe0 0x7f0b700cf2f0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f0b700b3cd0 tx=0x7f0b6002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6165020cd740a942 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.074+0000 7f0b6effd640 1 -- 172.21.15.77:0/3468168940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60034070 con 0x7f0b7015dfe0 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.074+0000 7f0b6effd640 1 -- 172.21.15.77:0/3468168940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b60030860 con 0x7f0b7015dfe0 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.074+0000 7f0b6effd640 1 -- 172.21.15.77:0/3468168940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60030c30 con 0x7f0b7015dfe0 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.075+0000 7f0b75a8f640 1 -- 172.21.15.77:0/3468168940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015dfe0 msgr2=0x7f0b700cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.075+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/3468168940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015dfe0 0x7f0b700cf2f0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f0b700b3cd0 tx=0x7f0b6002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.075+0000 7f0b75a8f640 1 -- 172.21.15.77:0/3468168940 shutdown_connections 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.075+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/3468168940 >> v2:172.21.15.158:3300/0 conn(0x7f0b700d29c0 0x7f0b700b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.075+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/3468168940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015dfe0 0x7f0b700cf2f0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.075+0000 7f0b75a8f640 1 -- 172.21.15.77:0/3468168940 >> 172.21.15.77:0/3468168940 conn(0x7f0b700d3cd0 msgr2=0x7f0b700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.075+0000 7f0b75a8f640 1 -- 172.21.15.77:0/3468168940 shutdown_connections 2024-03-06T07:31:54.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.075+0000 7f0b75a8f640 1 -- 172.21.15.77:0/3468168940 wait complete. 2024-03-06T07:31:54.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.077+0000 7f0b75a8f640 1 Processor -- start 2024-03-06T07:31:54.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.077+0000 7f0b75a8f640 1 -- start start 2024-03-06T07:31:54.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b75a8f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b700cf830 0x7f0b70160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b75a8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700d29c0 0x7f0b700cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b75a8f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b7015dfe0 0x7f0b700cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b75a8f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700d1df0 con 0x7f0b700d29c0 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b75a8f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700d1c70 con 0x7f0b7015dfe0 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b75a8f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7015e3c0 con 0x7f0b700cf830 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700d29c0 0x7f0b700cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700d29c0 0x7f0b700cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43182/0 (socket says 172.21.15.77:43182) 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b67fff640 1 -- 172.21.15.77:0/2246846875 learned_addr learned my addr 172.21.15.77:0/2246846875 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b6ffff640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b7015dfe0 0x7f0b700cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b6f7fe640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b700cf830 0x7f0b70160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b67fff640 1 -- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b700cf830 msgr2=0x7f0b70160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b67fff640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b700cf830 0x7f0b70160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b67fff640 1 -- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b7015dfe0 msgr2=0x7f0b700cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b67fff640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b7015dfe0 0x7f0b700cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b67fff640 1 -- 172.21.15.77:0/2246846875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7015f560 con 0x7f0b700d29c0 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b6f7fe640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b700cf830 0x7f0b70160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.078+0000 7f0b6ffff640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b7015dfe0 0x7f0b700cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.079+0000 7f0b67fff640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700d29c0 0x7f0b700cada0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800c4b0 tx=0x7f0b5800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.079+0000 7f0b6cff9640 1 -- 172.21.15.77:0/2246846875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58018070 con 0x7f0b700d29c0 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.079+0000 7f0b6cff9640 1 -- 172.21.15.77:0/2246846875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b58007500 con 0x7f0b700d29c0 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.079+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7015f840 con 0x7f0b700d29c0 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.079+0000 7f0b6cff9640 1 -- 172.21.15.77:0/2246846875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58004a20 con 0x7f0b700d29c0 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.079+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7015fe00 con 0x7f0b700d29c0 2024-03-06T07:31:54.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.080+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7004f330 con 0x7f0b700d29c0 2024-03-06T07:31:54.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.080+0000 7f0b6cff9640 1 -- 172.21.15.77:0/2246846875 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b5801b070 con 0x7f0b700d29c0 2024-03-06T07:31:54.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.081+0000 7f0b6cff9640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b5006b4d0 0x7f0b5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.081+0000 7f0b6f7fe640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b5006b4d0 0x7f0b5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.081+0000 7f0b6cff9640 1 -- 172.21.15.77:0/2246846875 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0b58089580 con 0x7f0b700d29c0 2024-03-06T07:31:54.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.081+0000 7f0b6f7fe640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b5006b4d0 0x7f0b5006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0b60036000 tx=0x7f0b6003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.085+0000 7f0b6cff9640 1 -- 172.21.15.77:0/2246846875 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b580584a0 con 0x7f0b700d29c0 2024-03-06T07:31:54.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.323+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b700b48e0 con 0x7f0b700d29c0 2024-03-06T07:31:54.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.324+0000 7f0b6cff9640 1 -- 172.21.15.77:0/2246846875 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f0b58006e20 con 0x7f0b700d29c0 2024-03-06T07:31:54.326 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:54.326 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:54.326 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.326+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b5006b4d0 msgr2=0x7f0b5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.326+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b5006b4d0 0x7f0b5006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0b60036000 tx=0x7f0b6003b040 comp rx=0 tx=0).stop 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.326+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700d29c0 msgr2=0x7f0b700cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.326+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700d29c0 0x7f0b700cada0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800c4b0 tx=0x7f0b5800c970 comp rx=0 tx=0).stop 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.326+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 shutdown_connections 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.327+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0b5006b4d0 0x7f0b5006d990 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.327+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700d29c0 0x7f0b700cada0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.327+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0b7015dfe0 0x7f0b700cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.327+0000 7f0b75a8f640 1 --2- 172.21.15.77:0/2246846875 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0b700cf830 0x7f0b70160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.327+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 >> 172.21.15.77:0/2246846875 conn(0x7f0b700d3cd0 msgr2=0x7f0b700b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.327+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 shutdown_connections 2024-03-06T07:31:54.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:54.327+0000 7f0b75a8f640 1 -- 172.21.15.77:0/2246846875 wait complete. 2024-03-06T07:31:54.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:54.541 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:55.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:55.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.642+0000 7f3a81e81640 1 Processor -- start 2024-03-06T07:31:55.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.642+0000 7f3a81e81640 1 -- start start 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a81e81640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f3a7c15dfe0 0x7f3a7c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a81e81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a7c0cf830 0x7f3a7c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a81e81640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c0b3430 con 0x7f3a7c0d27f0 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a81e81640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c0d2c50 con 0x7f3a7c15dfe0 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a81e81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0b31e0 con 0x7f3a7c0cf830 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a7bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3a7c0d27f0 0x7f3a7c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51100/0 (socket says 172.21.15.77:51100) 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a7bfff640 1 -- 172.21.15.77:0/2567681688 learned_addr learned my addr 172.21.15.77:0/2567681688 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839913189 0 0) 0x7f3a7c0b3430 con 0x7f3a7c0d27f0 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68003520 con 0x7f3a7c0d27f0 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a7b7fe640 1 --2- 172.21.15.77:0/2567681688 >> v2:172.21.15.158:3300/0 conn(0x7f3a7c15dfe0 0x7f3a7c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.643+0000 7f3a73fff640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a7c0cf830 0x7f3a7c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2839501394 0 0) 0x7f3a68003520 con 0x7f3a7c0d27f0 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a7c0cf830 msgr2=0x7f3a7c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a7c0cf830 0x7f3a7c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 >> v2:172.21.15.158:3300/0 conn(0x7f3a7c15dfe0 msgr2=0x7f3a7c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 --2- 172.21.15.77:0/2567681688 >> v2:172.21.15.158:3300/0 conn(0x7f3a7c15dfe0 0x7f3a7c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c0b3710 con 0x7f3a7c0d27f0 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f3a740019f0 con 0x7f3a7c0d27f0 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 >> v1:172.21.15.77:6789/0 conn(0x7f3a7c0d27f0 legacy=0x7f3a7c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a680039b0 0x7f3a68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68004800 0x7f3a68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a7c0cf830 0x7f3a680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0b31e0 con 0x7f3a68004800 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6800b9f0 con 0x7f3a680039b0 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6800bbf0 con 0x7f3a7c0cf830 2024-03-06T07:31:55.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7b7fe640 1 --2- 172.21.15.77:0/2567681688 >> v2:172.21.15.158:3300/0 conn(0x7f3a7c15dfe0 0x7f3a7c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a73fff640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68004800 0x7f3a68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7bfff640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a7c0cf830 0x7f3a680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7b7fe640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a680039b0 0x7f3a68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a73fff640 1 -- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a7c0cf830 msgr2=0x7f3a680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a73fff640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a7c0cf830 0x7f3a680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a73fff640 1 -- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a680039b0 msgr2=0x7f3a68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a73fff640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a680039b0 0x7f3a68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a73fff640 1 -- 172.21.15.77:0/2567681688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68003520 con 0x7f3a68004800 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.644+0000 7f3a7b7fe640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a680039b0 0x7f3a68003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.645+0000 7f3a73fff640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68004800 0x7f3a68004c80 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0cfc70 tx=0x7f3a68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b96bfd65dbef7ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.645+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6400b600 con 0x7f3a68004800 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.645+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a64038070 con 0x7f3a68004800 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.645+0000 7f3a7affd640 1 -- 172.21.15.77:0/2567681688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a64033890 con 0x7f3a68004800 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.646+0000 7f3a81e81640 1 -- 172.21.15.77:0/2567681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68004800 msgr2=0x7f3a68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.646+0000 7f3a81e81640 1 --2- 172.21.15.77:0/2567681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68004800 0x7f3a68004c80 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0cfc70 tx=0x7f3a68005830 comp rx=0 tx=0).stop 2024-03-06T07:31:55.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.646+0000 7f3a7bfff640 1 -- 172.21.15.77:0/2567681688 reap_dead start 2024-03-06T07:31:55.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.646+0000 7f3a81e81640 1 -- 172.21.15.77:0/2567681688 shutdown_connections 2024-03-06T07:31:55.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.646+0000 7f3a81e81640 1 -- 172.21.15.77:0/2567681688 >> 172.21.15.77:0/2567681688 conn(0x7f3a7c0d3cd0 msgr2=0x7f3a7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.646+0000 7f3a81e81640 1 -- 172.21.15.77:0/2567681688 shutdown_connections 2024-03-06T07:31:55.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.646+0000 7f3a81e81640 1 -- 172.21.15.77:0/2567681688 wait complete. 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.647+0000 7f3a81e81640 1 Processor -- start 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.647+0000 7f3a81e81640 1 -- start start 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a81e81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a68004800 0x7f3a7c0a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a81e81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a7c09f720 0x7f3a7c15bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a81e81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c15c070 0x7f3a7c0b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a81e81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0d1c70 con 0x7f3a7c15c070 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a81e81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0b3d60 con 0x7f3a7c09f720 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a81e81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0d1df0 con 0x7f3a68004800 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c15c070 0x7f3a7c0b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c15c070 0x7f3a7c0b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43200/0 (socket says 172.21.15.77:43200) 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 -- 172.21.15.77:0/1335509019 learned_addr learned my addr 172.21.15.77:0/1335509019 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7b7fe640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a68004800 0x7f3a7c0a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a73fff640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a7c09f720 0x7f3a7c15bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 -- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a68004800 msgr2=0x7f3a7c0a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a68004800 0x7f3a7c0a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 -- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a7c09f720 msgr2=0x7f3a7c15bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a7c09f720 0x7f3a7c15bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 -- 172.21.15.77:0/1335509019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c09e960 con 0x7f3a7c15c070 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a73fff640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a7c09f720 0x7f3a7c15bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7b7fe640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a68004800 0x7f3a7c0a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a7bfff640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c15c070 0x7f3a7c0b2190 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3a74002230 tx=0x7f3a7400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a78ff9640 1 -- 172.21.15.77:0/1335509019 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7400eb90 con 0x7f3a7c15c070 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.648+0000 7f3a78ff9640 1 -- 172.21.15.77:0/1335509019 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7400ed70 con 0x7f3a7c15c070 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.649+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c09ec40 con 0x7f3a7c15c070 2024-03-06T07:31:55.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.649+0000 7f3a78ff9640 1 -- 172.21.15.77:0/1335509019 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a740146c0 con 0x7f3a7c15c070 2024-03-06T07:31:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.649+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c09f200 con 0x7f3a7c15c070 2024-03-06T07:31:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.649+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c04f330 con 0x7f3a7c15c070 2024-03-06T07:31:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.650+0000 7f3a78ff9640 1 -- 172.21.15.77:0/1335509019 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a7401b060 con 0x7f3a7c15c070 2024-03-06T07:31:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.651+0000 7f3a78ff9640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.651+0000 7f3a7b7fe640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.651+0000 7f3a78ff9640 1 -- 172.21.15.77:0/1335509019 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3a74089c20 con 0x7f3a7c15c070 2024-03-06T07:31:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.651+0000 7f3a7b7fe640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c006fd0 tx=0x7f3a6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.655+0000 7f3a78ff9640 1 -- 172.21.15.77:0/1335509019 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a74058b10 con 0x7f3a7c15c070 2024-03-06T07:31:55.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.884+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a7c06a190 con 0x7f3a7c15c070 2024-03-06T07:31:55.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.885+0000 7f3a78ff9640 1 -- 172.21.15.77:0/1335509019 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f3a74058410 con 0x7f3a7c15c070 2024-03-06T07:31:55.886 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:55.887 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a5c06b4d0 msgr2=0x7f3a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c006fd0 tx=0x7f3a6c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c15c070 msgr2=0x7f3a7c0b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c15c070 0x7f3a7c0b2190 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3a74002230 tx=0x7f3a7400e530 comp rx=0 tx=0).stop 2024-03-06T07:31:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 shutdown_connections 2024-03-06T07:31:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3a5c06b4d0 0x7f3a5c06d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a7c15c070 0x7f3a7c0b2190 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3a7c09f720 0x7f3a7c15bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 --2- 172.21.15.77:0/1335509019 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3a68004800 0x7f3a7c0a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 >> 172.21.15.77:0/1335509019 conn(0x7f3a7c0d3cd0 msgr2=0x7f3a7c160460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 shutdown_connections 2024-03-06T07:31:55.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:55.887+0000 7f3a81e81640 1 -- 172.21.15.77:0/1335509019 wait complete. 2024-03-06T07:31:56.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:56.101 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:57.102 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.203+0000 7fb50cbd7640 1 Processor -- start 2024-03-06T07:31:57.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.203+0000 7fb50cbd7640 1 -- start start 2024-03-06T07:31:57.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb50cbd7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080d00c0 0x7fb5080d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb50cbd7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb5080cf850 0x7fb5080cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb50cbd7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5080d09e0 con 0x7fb5080ce1b0 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb50cbd7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50806ca90 con 0x7fb5080cf850 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb50cbd7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50806df30 con 0x7fb5080d00c0 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb506d9d640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb5080ce1b0 0x7fb5080ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:36892/0 (socket says 172.21.15.77:36892) 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb4fffff640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb5080cf850 0x7fb5080cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb50659c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080d00c0 0x7fb5080d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb506d9d640 1 -- 172.21.15.77:0/3784267583 learned_addr learned my addr 172.21.15.77:0/3784267583 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb505d9b640 1 -- 172.21.15.77:0/3784267583 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148625274 0 0) 0x7fb5080d09e0 con 0x7fb5080ce1b0 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.204+0000 7fb505d9b640 1 -- 172.21.15.77:0/3784267583 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f4003520 con 0x7fb5080ce1b0 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.205+0000 7fb50659c640 1 -- 172.21.15.77:0/3784267583 >> v2:172.21.15.158:3300/0 conn(0x7fb5080cf850 msgr2=0x7fb5080cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.205+0000 7fb50659c640 1 --2- 172.21.15.77:0/3784267583 >> v2:172.21.15.158:3300/0 conn(0x7fb5080cf850 0x7fb5080cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.205+0000 7fb50659c640 1 -- 172.21.15.77:0/3784267583 >> v1:172.21.15.158:6789/0 conn(0x7fb5080ce1b0 legacy=0x7fb5080ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.205+0000 7fb50659c640 1 -- 172.21.15.77:0/3784267583 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50815e380 con 0x7fb5080d00c0 2024-03-06T07:31:57.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.205+0000 7fb4fffff640 1 --2- 172.21.15.77:0/3784267583 >> v2:172.21.15.158:3300/0 conn(0x7fb5080cf850 0x7fb5080cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:57.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.205+0000 7fb50659c640 1 --2- 172.21.15.77:0/3784267583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080d00c0 0x7fb5080d04a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80097a0 tx=0x7fb4f802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0f035d0f0fbf20a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb505d9b640 1 -- 172.21.15.77:0/3784267583 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8035020 con 0x7fb5080d00c0 2024-03-06T07:31:57.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb505d9b640 1 -- 172.21.15.77:0/3784267583 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f8031830 con 0x7fb5080d00c0 2024-03-06T07:31:57.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb505d9b640 1 -- 172.21.15.77:0/3784267583 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8031be0 con 0x7fb5080d00c0 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb50cbd7640 1 -- 172.21.15.77:0/3784267583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080d00c0 msgr2=0x7fb5080d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/3784267583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080d00c0 0x7fb5080d04a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80097a0 tx=0x7fb4f802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb50cbd7640 1 -- 172.21.15.77:0/3784267583 shutdown_connections 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/3784267583 >> v2:172.21.15.158:3300/0 conn(0x7fb5080cf850 0x7fb5080cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/3784267583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080d00c0 0x7fb5080d04a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb50cbd7640 1 -- 172.21.15.77:0/3784267583 >> 172.21.15.77:0/3784267583 conn(0x7fb5080b35a0 msgr2=0x7fb5080b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb50cbd7640 1 -- 172.21.15.77:0/3784267583 shutdown_connections 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.206+0000 7fb50cbd7640 1 -- 172.21.15.77:0/3784267583 wait complete. 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50cbd7640 1 Processor -- start 2024-03-06T07:31:57.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50cbd7640 1 -- start start 2024-03-06T07:31:57.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50cbd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5080ce1b0 0x7fb508157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50cbd7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080cf850 0x7fb5080c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50cbd7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5080d00c0 0x7fb5080c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50cbd7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50806c440 con 0x7fb5080ce1b0 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50cbd7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50806c160 con 0x7fb5080d00c0 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50cbd7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50806c720 con 0x7fb5080cf850 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5080ce1b0 0x7fb508157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5080ce1b0 0x7fb508157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43210/0 (socket says 172.21.15.77:43210) 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.208+0000 7fb50659c640 1 -- 172.21.15.77:0/505194442 learned_addr learned my addr 172.21.15.77:0/505194442 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb506d9d640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5080d00c0 0x7fb5080c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb4fffff640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080cf850 0x7fb5080c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb50659c640 1 -- 172.21.15.77:0/505194442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080cf850 msgr2=0x7fb5080c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb50659c640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080cf850 0x7fb5080c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb50659c640 1 -- 172.21.15.77:0/505194442 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5080d00c0 msgr2=0x7fb5080c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb50659c640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5080d00c0 0x7fb5080c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb50659c640 1 -- 172.21.15.77:0/505194442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5080cb6c0 con 0x7fb5080ce1b0 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb50659c640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5080ce1b0 0x7fb508157060 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8000c00 tx=0x7fb4f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb4feffd640 1 -- 172.21.15.77:0/505194442 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8040070 con 0x7fb5080ce1b0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb4feffd640 1 -- 172.21.15.77:0/505194442 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f803a830 con 0x7fb5080ce1b0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5080cab40 con 0x7fb5080ce1b0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb4feffd640 1 -- 172.21.15.77:0/505194442 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8035040 con 0x7fb5080ce1b0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.209+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5080cb100 con 0x7fb5080ce1b0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.210+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50804f330 con 0x7fb5080ce1b0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.210+0000 7fb4feffd640 1 -- 172.21.15.77:0/505194442 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4f804b060 con 0x7fb5080ce1b0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.211+0000 7fb4feffd640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4e806b4d0 0x7fb4e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.211+0000 7fb4feffd640 1 -- 172.21.15.77:0/505194442 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb4f80ab7e0 con 0x7fb5080ce1b0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.211+0000 7fb4fffff640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4e806b4d0 0x7fb4e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.211+0000 7fb4fffff640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4e806b4d0 0x7fb4e806d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009560 tx=0x7fb4f0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.215+0000 7fb4feffd640 1 -- 172.21.15.77:0/505194442 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5080cb6c0 con 0x7fb5080ce1b0 2024-03-06T07:31:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.527+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5080747a0 con 0x7fb5080ce1b0 2024-03-06T07:31:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.528+0000 7fb4feffd640 1 -- 172.21.15.77:0/505194442 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fb4f807a080 con 0x7fb5080ce1b0 2024-03-06T07:31:57.530 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:57.530 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:57.530 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4e806b4d0 msgr2=0x7fb4e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4e806b4d0 0x7fb4e806d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009560 tx=0x7fb4f0009190 comp rx=0 tx=0).stop 2024-03-06T07:31:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5080ce1b0 msgr2=0x7fb508157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5080ce1b0 0x7fb508157060 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8000c00 tx=0x7fb4f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 shutdown_connections 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5080d00c0 0x7fb5080c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5080cf850 0x7fb5080c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb4e806b4d0 0x7fb4e806d990 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 --2- 172.21.15.77:0/505194442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5080ce1b0 0x7fb508157060 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 >> 172.21.15.77:0/505194442 conn(0x7fb5080b35a0 msgr2=0x7fb5080b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 shutdown_connections 2024-03-06T07:31:57.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:57.530+0000 7fb50cbd7640 1 -- 172.21.15.77:0/505194442 wait complete. 2024-03-06T07:31:57.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:57.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:31:58.745 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.845+0000 7f1664b16640 1 Processor -- start 2024-03-06T07:31:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.845+0000 7f1664b16640 1 -- start start 2024-03-06T07:31:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.846+0000 7f1664b16640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f166015dfe0 0x7f16600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.846+0000 7f1664b16640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16600d29c0 0x7f16600b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.846+0000 7f1664b16640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16600b3430 con 0x7f16600cf830 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.846+0000 7f1664b16640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16600b31e0 con 0x7f16600d29c0 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.846+0000 7f1664b16640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16600b3d60 con 0x7f166015dfe0 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165ed9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16600d29c0 0x7f16600b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165dd9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f16600cf830 0x7f16600d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55874/0 (socket says 172.21.15.77:55874) 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165dd9b640 1 -- 172.21.15.77:0/3153400723 learned_addr learned my addr 172.21.15.77:0/3153400723 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165ed9d640 1 -- 172.21.15.77:0/3153400723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f166015dfe0 msgr2=0x7f16600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165e59c640 1 --2- 172.21.15.77:0/3153400723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f166015dfe0 0x7f16600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165ed9d640 1 --2- 172.21.15.77:0/3153400723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f166015dfe0 0x7f16600cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165ed9d640 1 -- 172.21.15.77:0/3153400723 >> v1:172.21.15.77:6789/0 conn(0x7f16600cf830 legacy=0x7f16600d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165ed9d640 1 -- 172.21.15.77:0/3153400723 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16600b3710 con 0x7f16600d29c0 2024-03-06T07:31:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165d59a640 1 -- 172.21.15.77:0/3153400723 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936254933 0 0) 0x7f16600b3430 con 0x7f16600cf830 2024-03-06T07:31:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.847+0000 7f165e59c640 1 --2- 172.21.15.77:0/3153400723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f166015dfe0 0x7f16600cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f165ed9d640 1 --2- 172.21.15.77:0/3153400723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16600d29c0 0x7f16600b41f0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f1658009d90 tx=0x7f165802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd90a94992754127 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f165d59a640 1 -- 172.21.15.77:0/3153400723 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658039070 con 0x7f16600d29c0 2024-03-06T07:31:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f165d59a640 1 -- 172.21.15.77:0/3153400723 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165802cc50 con 0x7f16600d29c0 2024-03-06T07:31:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f165d59a640 1 -- 172.21.15.77:0/3153400723 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658034440 con 0x7f16600d29c0 2024-03-06T07:31:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f1664b16640 1 -- 172.21.15.77:0/3153400723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16600d29c0 msgr2=0x7f16600b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f1664b16640 1 --2- 172.21.15.77:0/3153400723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16600d29c0 0x7f16600b41f0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f1658009d90 tx=0x7f165802bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f1664b16640 1 -- 172.21.15.77:0/3153400723 shutdown_connections 2024-03-06T07:31:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f1664b16640 1 --2- 172.21.15.77:0/3153400723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16600d29c0 0x7f16600b41f0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f1664b16640 1 --2- 172.21.15.77:0/3153400723 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f166015dfe0 0x7f16600cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f1664b16640 1 -- 172.21.15.77:0/3153400723 >> 172.21.15.77:0/3153400723 conn(0x7f16600d3cd0 msgr2=0x7f16600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f1664b16640 1 -- 172.21.15.77:0/3153400723 shutdown_connections 2024-03-06T07:31:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.848+0000 7f1664b16640 1 -- 172.21.15.77:0/3153400723 wait complete. 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.850+0000 7f1664b16640 1 Processor -- start 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.850+0000 7f1664b16640 1 -- start start 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.850+0000 7f1664b16640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16600cf830 0x7f16600ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.850+0000 7f1664b16640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16600d29c0 0x7f166015f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.850+0000 7f1664b16640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f166015dfe0 0x7f16600cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f1664b16640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16600d1df0 con 0x7f166015dfe0 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f1664b16640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16600d1c70 con 0x7f16600cf830 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f1664b16640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166015e3c0 con 0x7f16600d29c0 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f166015dfe0 0x7f16600cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f166015dfe0 0x7f16600cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56364/0 (socket says 172.21.15.77:56364) 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 -- 172.21.15.77:0/2722199989 learned_addr learned my addr 172.21.15.77:0/2722199989 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165dd9b640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16600d29c0 0x7f166015f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165e59c640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16600cf830 0x7f16600ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 -- 172.21.15.77:0/2722199989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16600d29c0 msgr2=0x7f166015f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16600d29c0 0x7f166015f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 -- 172.21.15.77:0/2722199989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16600cf830 msgr2=0x7f16600ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16600cf830 0x7f16600ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 -- 172.21.15.77:0/2722199989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166015faa0 con 0x7f166015dfe0 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f165ed9d640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f166015dfe0 0x7f16600cb440 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f1658009d90 tx=0x7f1658004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f1656ffd640 1 -- 172.21.15.77:0/2722199989 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658044020 con 0x7f166015dfe0 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f1656ffd640 1 -- 172.21.15.77:0/2722199989 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165803a040 con 0x7f166015dfe0 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166015fd80 con 0x7f166015dfe0 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f1656ffd640 1 -- 172.21.15.77:0/2722199989 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658039050 con 0x7f166015dfe0 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.851+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16600caf20 con 0x7f166015dfe0 2024-03-06T07:31:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.852+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16600b3360 con 0x7f166015dfe0 2024-03-06T07:31:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.853+0000 7f1656ffd640 1 -- 172.21.15.77:0/2722199989 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1658003990 con 0x7f166015dfe0 2024-03-06T07:31:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.853+0000 7f1656ffd640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f164006b4d0 0x7f164006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.853+0000 7f1656ffd640 1 -- 172.21.15.77:0/2722199989 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f16580ac6d0 con 0x7f166015dfe0 2024-03-06T07:31:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.853+0000 7f165e59c640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f164006b4d0 0x7f164006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.854+0000 7f165e59c640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f164006b4d0 0x7f164006d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f164800a950 tx=0x7f1648008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:58.857+0000 7f1656ffd640 1 -- 172.21.15.77:0/2722199989 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16600b3360 con 0x7f166015dfe0 2024-03-06T07:31:59.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.075+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f166004f330 con 0x7f166015dfe0 2024-03-06T07:31:59.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.076+0000 7f1656ffd640 1 -- 172.21.15.77:0/2722199989 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f165804e060 con 0x7f166015dfe0 2024-03-06T07:31:59.078 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:31:59.078 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:31:59.078 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:31:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f164006b4d0 msgr2=0x7f164006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f164006b4d0 0x7f164006d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f164800a950 tx=0x7f1648008040 comp rx=0 tx=0).stop 2024-03-06T07:31:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f166015dfe0 msgr2=0x7f16600cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f166015dfe0 0x7f16600cb440 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f1658009d90 tx=0x7f1658004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 shutdown_connections 2024-03-06T07:31:59.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f164006b4d0 0x7f164006d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f166015dfe0 0x7f16600cb440 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f16600d29c0 0x7f166015f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 --2- 172.21.15.77:0/2722199989 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f16600cf830 0x7f16600ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 >> 172.21.15.77:0/2722199989 conn(0x7f16600d3cd0 msgr2=0x7f16600b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 shutdown_connections 2024-03-06T07:31:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:31:59.078+0000 7f1664b16640 1 -- 172.21.15.77:0/2722199989 wait complete. 2024-03-06T07:31:59.291 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:31:59.292 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:00.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:00.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.391+0000 7f536d89d640 1 Processor -- start 2024-03-06T07:32:00.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.391+0000 7f536d89d640 1 -- start start 2024-03-06T07:32:00.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.391+0000 7f536d89d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f536806da40 0x7f53680cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.391+0000 7f536d89d640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f53680cd7a0 0x7f536815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.391+0000 7f536d89d640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680cc980 con 0x7f53681424f0 2024-03-06T07:32:00.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.391+0000 7f536d89d640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53680cd220 con 0x7f53680cd7a0 2024-03-06T07:32:00.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.391+0000 7f536d89d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53680ccc60 con 0x7f536806da40 2024-03-06T07:32:00.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f5366ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f53681424f0 0x7f536806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55878/0 (socket says 172.21.15.77:55878) 2024-03-06T07:32:00.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f5366ffd640 1 -- 172.21.15.77:0/2120160962 learned_addr learned my addr 172.21.15.77:0/2120160962 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f53677fe640 1 --2- 172.21.15.77:0/2120160962 >> v2:172.21.15.158:3300/0 conn(0x7f53680cd7a0 0x7f536815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f53667fc640 1 --2- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f536806da40 0x7f53680cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f5365ffb640 1 -- 172.21.15.77:0/2120160962 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497196139 0 0) 0x7f53680cc980 con 0x7f53681424f0 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f5365ffb640 1 -- 172.21.15.77:0/2120160962 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5354003520 con 0x7f53681424f0 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f53677fe640 1 -- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f536806da40 msgr2=0x7f53680cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f5365ffb640 1 -- 172.21.15.77:0/2120160962 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605702737 0 0) 0x7f5354003520 con 0x7f53681424f0 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f53677fe640 1 --2- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f536806da40 0x7f53680cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f53677fe640 1 -- 172.21.15.77:0/2120160962 >> v1:172.21.15.77:6789/0 conn(0x7f53681424f0 legacy=0x7f536806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f53677fe640 1 -- 172.21.15.77:0/2120160962 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53680ccf40 con 0x7f53680cd7a0 2024-03-06T07:32:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.392+0000 7f53667fc640 1 --2- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f536806da40 0x7f53680cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.393+0000 7f53677fe640 1 --2- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53680cd7a0 0x7f536815cef0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5358009d90 tx=0x7f535802bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4cadbeb9130e1ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.393+0000 7f5365ffb640 1 -- 172.21.15.77:0/2120160962 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358039070 con 0x7f53680cd7a0 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.393+0000 7f5365ffb640 1 -- 172.21.15.77:0/2120160962 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535802c8a0 con 0x7f53680cd7a0 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.393+0000 7f5365ffb640 1 -- 172.21.15.77:0/2120160962 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360001670 con 0x7f53680cd7a0 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.394+0000 7f536d89d640 1 -- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53680cd7a0 msgr2=0x7f536815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.394+0000 7f536d89d640 1 --2- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53680cd7a0 0x7f536815cef0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f5358009d90 tx=0x7f535802bd70 comp rx=0 tx=0).stop 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.394+0000 7f536d89d640 1 -- 172.21.15.77:0/2120160962 shutdown_connections 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.394+0000 7f536d89d640 1 --2- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f53680cd7a0 0x7f536815cef0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.394+0000 7f536d89d640 1 --2- 172.21.15.77:0/2120160962 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f536806da40 0x7f53680cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.394+0000 7f536d89d640 1 -- 172.21.15.77:0/2120160962 >> 172.21.15.77:0/2120160962 conn(0x7f53680cf8a0 msgr2=0x7f53680cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.394+0000 7f536d89d640 1 -- 172.21.15.77:0/2120160962 shutdown_connections 2024-03-06T07:32:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.394+0000 7f536d89d640 1 -- 172.21.15.77:0/2120160962 wait complete. 2024-03-06T07:32:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.395+0000 7f536d89d640 1 Processor -- start 2024-03-06T07:32:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.395+0000 7f536d89d640 1 -- start start 2024-03-06T07:32:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f536d89d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536806da40 0x7f5368135200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f536d89d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680cd7a0 0x7f53680abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f536d89d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53681424f0 0x7f53680ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f536d89d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368071ce0 con 0x7f53681424f0 2024-03-06T07:32:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f536d89d640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368071b60 con 0x7f536806da40 2024-03-06T07:32:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f536d89d640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5368071e60 con 0x7f53680cd7a0 2024-03-06T07:32:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53681424f0 0x7f53680ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53681424f0 0x7f53680ac3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56372/0 (socket says 172.21.15.77:56372) 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53677fe640 1 -- 172.21.15.77:0/2761945195 learned_addr learned my addr 172.21.15.77:0/2761945195 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53667fc640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680cd7a0 0x7f53680abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f5366ffd640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536806da40 0x7f5368135200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53677fe640 1 -- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680cd7a0 msgr2=0x7f53680abeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53677fe640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680cd7a0 0x7f53680abeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53677fe640 1 -- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536806da40 msgr2=0x7f5368135200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53677fe640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536806da40 0x7f5368135200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53677fe640 1 -- 172.21.15.77:0/2761945195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53680a9610 con 0x7f53681424f0 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f53667fc640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680cd7a0 0x7f53680abeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.396+0000 7f5366ffd640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536806da40 0x7f5368135200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.397+0000 7f53677fe640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53681424f0 0x7f53680ac3f0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f5358009d90 tx=0x7f5358004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.397+0000 7f535ffff640 1 -- 172.21.15.77:0/2761945195 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358044020 con 0x7f53681424f0 2024-03-06T07:32:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.397+0000 7f535ffff640 1 -- 172.21.15.77:0/2761945195 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5358034ce0 con 0x7f53681424f0 2024-03-06T07:32:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.397+0000 7f535ffff640 1 -- 172.21.15.77:0/2761945195 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358039040 con 0x7f53681424f0 2024-03-06T07:32:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.397+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53680a98f0 con 0x7f53681424f0 2024-03-06T07:32:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.397+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53680a9eb0 con 0x7f53681424f0 2024-03-06T07:32:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.398+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5368160910 con 0x7f53681424f0 2024-03-06T07:32:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.399+0000 7f535ffff640 1 -- 172.21.15.77:0/2761945195 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5358047020 con 0x7f53681424f0 2024-03-06T07:32:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.399+0000 7f535ffff640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.399+0000 7f535ffff640 1 -- 172.21.15.77:0/2761945195 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f53580ac860 con 0x7f53681424f0 2024-03-06T07:32:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.399+0000 7f5366ffd640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.400+0000 7f5366ffd640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f5360001eb0 tx=0x7f5360009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.403+0000 7f535ffff640 1 -- 172.21.15.77:0/2761945195 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5368160910 con 0x7f53681424f0 2024-03-06T07:32:00.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.615+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f536804f330 con 0x7f53681424f0 2024-03-06T07:32:00.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.616+0000 7f535ffff640 1 -- 172.21.15.77:0/2761945195 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f535804e030 con 0x7f53681424f0 2024-03-06T07:32:00.618 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:00.618 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:00.618 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 msgr2=0x7f534806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f5360001eb0 tx=0x7f5360009040 comp rx=0 tx=0).stop 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53681424f0 msgr2=0x7f53680ac3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53681424f0 0x7f53680ac3f0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f5358009d90 tx=0x7f5358004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 shutdown_connections 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f534806b4d0 0x7f534806d990 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53681424f0 0x7f53680ac3f0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f53680cd7a0 0x7f53680abeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 --2- 172.21.15.77:0/2761945195 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f536806da40 0x7f5368135200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 >> 172.21.15.77:0/2761945195 conn(0x7f53680cf8a0 msgr2=0x7f5368141710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 shutdown_connections 2024-03-06T07:32:00.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:00.618+0000 7f536d89d640 1 -- 172.21.15.77:0/2761945195 wait complete. 2024-03-06T07:32:00.832 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:00.832 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:01.833 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.932+0000 7fd09de4e640 1 Processor -- start 2024-03-06T07:32:01.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.932+0000 7fd09de4e640 1 -- start start 2024-03-06T07:32:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd09de4e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0980d22b0 0x7fd0980d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd09de4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd0980b41f0 0x7fd0980b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd09de4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0980b3430 con 0x7fd09815dfe0 2024-03-06T07:32:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd09de4e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0980d2c50 con 0x7fd0980d22b0 2024-03-06T07:32:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd09de4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0980b31e0 con 0x7fd0980b41f0 2024-03-06T07:32:01.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd08effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0980d22b0 0x7fd0980d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd08effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0980d22b0 0x7fd0980d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56386/0 (socket says 172.21.15.77:56386) 2024-03-06T07:32:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd08effd640 1 -- 172.21.15.77:0/3347463333 learned_addr learned my addr 172.21.15.77:0/3347463333 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd097fff640 1 --2- 172.21.15.77:0/3347463333 >> v2:172.21.15.158:3300/0 conn(0x7fd0980b41f0 0x7fd0980b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd08effd640 1 -- 172.21.15.77:0/3347463333 >> v2:172.21.15.158:3300/0 conn(0x7fd0980b41f0 msgr2=0x7fd0980b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd08effd640 1 --2- 172.21.15.77:0/3347463333 >> v2:172.21.15.158:3300/0 conn(0x7fd0980b41f0 0x7fd0980b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd08effd640 1 -- 172.21.15.77:0/3347463333 >> v1:172.21.15.158:6789/0 conn(0x7fd09815dfe0 legacy=0x7fd0980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.933+0000 7fd08effd640 1 -- 172.21.15.77:0/3347463333 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0980b3710 con 0x7fd0980d22b0 2024-03-06T07:32:01.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.934+0000 7fd08effd640 1 --2- 172.21.15.77:0/3347463333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980d22b0 0x7fd0980d2710 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fd088009870 tx=0x7fd08802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a8f2430ad0135a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.934+0000 7fd096ffd640 1 -- 172.21.15.77:0/3347463333 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd088035020 con 0x7fd0980d22b0 2024-03-06T07:32:01.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.934+0000 7fd096ffd640 1 -- 172.21.15.77:0/3347463333 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd088031870 con 0x7fd0980d22b0 2024-03-06T07:32:01.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.934+0000 7fd096ffd640 1 -- 172.21.15.77:0/3347463333 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd088031c40 con 0x7fd0980d22b0 2024-03-06T07:32:01.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.935+0000 7fd09de4e640 1 -- 172.21.15.77:0/3347463333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980d22b0 msgr2=0x7fd0980d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.935+0000 7fd09de4e640 1 --2- 172.21.15.77:0/3347463333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980d22b0 0x7fd0980d2710 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fd088009870 tx=0x7fd08802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:01.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.935+0000 7fd09de4e640 1 -- 172.21.15.77:0/3347463333 shutdown_connections 2024-03-06T07:32:01.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.935+0000 7fd09de4e640 1 --2- 172.21.15.77:0/3347463333 >> v2:172.21.15.158:3300/0 conn(0x7fd0980b41f0 0x7fd0980b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.935+0000 7fd09de4e640 1 --2- 172.21.15.77:0/3347463333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980d22b0 0x7fd0980d2710 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.935+0000 7fd09de4e640 1 -- 172.21.15.77:0/3347463333 >> 172.21.15.77:0/3347463333 conn(0x7fd0980d3cd0 msgr2=0x7fd0980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.935+0000 7fd09de4e640 1 -- 172.21.15.77:0/3347463333 shutdown_connections 2024-03-06T07:32:01.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.935+0000 7fd09de4e640 1 -- 172.21.15.77:0/3347463333 wait complete. 2024-03-06T07:32:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.936+0000 7fd09de4e640 1 Processor -- start 2024-03-06T07:32:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.936+0000 7fd09de4e640 1 -- start start 2024-03-06T07:32:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd09de4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980b41f0 0x7fd09806bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd09de4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0980d22b0 0x7fd098160c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd09de4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815dfe0 0x7fd0980b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd09de4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0980cfa00 con 0x7fd0980b41f0 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd09de4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0980b3d60 con 0x7fd09815dfe0 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd09de4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0980d1c70 con 0x7fd0980d22b0 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd0977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980b41f0 0x7fd09806bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd0977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980b41f0 0x7fd09806bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56396/0 (socket says 172.21.15.77:56396) 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd0977fe640 1 -- 172.21.15.77:0/4248340169 learned_addr learned my addr 172.21.15.77:0/4248340169 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd08effd640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0980d22b0 0x7fd098160c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd097fff640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815dfe0 0x7fd0980b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd0977fe640 1 -- 172.21.15.77:0/4248340169 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0980d22b0 msgr2=0x7fd098160c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd0977fe640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0980d22b0 0x7fd098160c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd0977fe640 1 -- 172.21.15.77:0/4248340169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815dfe0 msgr2=0x7fd0980b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd0977fe640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815dfe0 0x7fd0980b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.937+0000 7fd0977fe640 1 -- 172.21.15.77:0/4248340169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09815f340 con 0x7fd0980b41f0 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.938+0000 7fd0977fe640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980b41f0 0x7fd09806bc70 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fd08000b420 tx=0x7fd08000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.938+0000 7fd094ff9640 1 -- 172.21.15.77:0/4248340169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080011830 con 0x7fd0980b41f0 2024-03-06T07:32:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.938+0000 7fd094ff9640 1 -- 172.21.15.77:0/4248340169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd080016070 con 0x7fd0980b41f0 2024-03-06T07:32:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.938+0000 7fd094ff9640 1 -- 172.21.15.77:0/4248340169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080011c10 con 0x7fd0980b41f0 2024-03-06T07:32:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.938+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0981601a0 con 0x7fd0980b41f0 2024-03-06T07:32:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.938+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09806c1b0 con 0x7fd0980b41f0 2024-03-06T07:32:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.939+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09804f330 con 0x7fd0980b41f0 2024-03-06T07:32:01.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.939+0000 7fd094ff9640 1 -- 172.21.15.77:0/4248340169 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd08001b460 con 0x7fd0980b41f0 2024-03-06T07:32:01.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.940+0000 7fd094ff9640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd07c06b4d0 0x7fd07c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.940+0000 7fd094ff9640 1 -- 172.21.15.77:0/4248340169 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd080089cb0 con 0x7fd0980b41f0 2024-03-06T07:32:01.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.940+0000 7fd08effd640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd07c06b4d0 0x7fd07c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.940+0000 7fd08effd640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd07c06b4d0 0x7fd07c06d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd088036000 tx=0x7fd08803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:01.944+0000 7fd094ff9640 1 -- 172.21.15.77:0/4248340169 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd080020020 con 0x7fd0980b41f0 2024-03-06T07:32:01.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:01.985 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:02.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.161+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd09815d3a0 con 0x7fd0980b41f0 2024-03-06T07:32:02.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.161+0000 7fd094ff9640 1 -- 172.21.15.77:0/4248340169 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd080058ba0 con 0x7fd0980b41f0 2024-03-06T07:32:02.163 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:02.163 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:02.164 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:02.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd07c06b4d0 msgr2=0x7fd07c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd07c06b4d0 0x7fd07c06d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd088036000 tx=0x7fd08803c040 comp rx=0 tx=0).stop 2024-03-06T07:32:02.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980b41f0 msgr2=0x7fd09806bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980b41f0 0x7fd09806bc70 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fd08000b420 tx=0x7fd08000b8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 shutdown_connections 2024-03-06T07:32:02.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd09815dfe0 0x7fd0980b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd0980d22b0 0x7fd098160c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd07c06b4d0 0x7fd07c06d990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 --2- 172.21.15.77:0/4248340169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0980b41f0 0x7fd09806bc70 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 >> 172.21.15.77:0/4248340169 conn(0x7fd0980d3cd0 msgr2=0x7fd0980d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 shutdown_connections 2024-03-06T07:32:02.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:02.163+0000 7fd09de4e640 1 -- 172.21.15.77:0/4248340169 wait complete. 2024-03-06T07:32:02.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:02.380 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:03.381 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:03.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.496+0000 7f102bbf7640 1 Processor -- start 2024-03-06T07:32:03.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.496+0000 7f102bbf7640 1 -- start start 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102bbf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10240d2710 0x7f10240b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102bbf7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240b3430 con 0x7f102415dfe0 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102bbf7640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240b3710 con 0x7f10240d22b0 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102bbf7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10240b31e0 con 0x7f10240d2710 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102a194640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10240d2710 0x7f10240b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f1029993640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f102415dfe0 0x7f10240cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55880/0 (socket says 172.21.15.77:55880) 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f1029993640 1 -- 172.21.15.77:0/2954383588 learned_addr learned my addr 172.21.15.77:0/2954383588 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102a194640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10240d2710 0x7f10240b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56402/0 (socket says 172.21.15.77:56402) 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102a194640 1 -- 172.21.15.77:0/2954383588 >> v1:172.21.15.158:6789/0 conn(0x7f10240d22b0 legacy=0x7f10240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102a194640 1 -- 172.21.15.77:0/2954383588 >> v1:172.21.15.77:6789/0 conn(0x7f102415dfe0 legacy=0x7f10240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.497+0000 7f102a194640 1 -- 172.21.15.77:0/2954383588 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240b39f0 con 0x7f10240d2710 2024-03-06T07:32:03.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f102a194640 1 --2- 172.21.15.77:0/2954383588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f10240b4900 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f101c009dc0 tx=0x7f101c02bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c79fd84a8051f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f1028991640 1 -- 172.21.15.77:0/2954383588 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c039070 con 0x7f10240d2710 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f1028991640 1 -- 172.21.15.77:0/2954383588 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c02ccd0 con 0x7f10240d2710 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f1028991640 1 -- 172.21.15.77:0/2954383588 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c034660 con 0x7f10240d2710 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f102bbf7640 1 -- 172.21.15.77:0/2954383588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 msgr2=0x7f10240b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f102bbf7640 1 --2- 172.21.15.77:0/2954383588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f10240b4900 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f101c009dc0 tx=0x7f101c02bf20 comp rx=0 tx=0).stop 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f102bbf7640 1 -- 172.21.15.77:0/2954383588 shutdown_connections 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f102bbf7640 1 --2- 172.21.15.77:0/2954383588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f10240b4900 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.498+0000 7f102bbf7640 1 -- 172.21.15.77:0/2954383588 >> 172.21.15.77:0/2954383588 conn(0x7f10240d3cd0 msgr2=0x7f10240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.499+0000 7f102bbf7640 1 -- 172.21.15.77:0/2954383588 shutdown_connections 2024-03-06T07:32:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.499+0000 7f102bbf7640 1 -- 172.21.15.77:0/2954383588 wait complete. 2024-03-06T07:32:03.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f102bbf7640 1 Processor -- start 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f102bbf7640 1 -- start start 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f102bbf7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10240d22b0 0x7f10240cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f102bbf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f102415f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f102bbf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f102415dfe0 0x7f102414af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f102bbf7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240cfa00 con 0x7f10240d2710 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f102bbf7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240b3d60 con 0x7f10240d22b0 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f102bbf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10240d1c70 con 0x7f102415dfe0 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.500+0000 7f1029192640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f102415f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029192640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f102415f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56410/0 (socket says 172.21.15.77:56410) 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029192640 1 -- 172.21.15.77:0/783458670 learned_addr learned my addr 172.21.15.77:0/783458670 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029993640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10240d22b0 0x7f10240cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f102a194640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f102415dfe0 0x7f102414af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029192640 1 -- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f102415dfe0 msgr2=0x7f102414af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029192640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f102415dfe0 0x7f102414af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029192640 1 -- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10240d22b0 msgr2=0x7f10240cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029192640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10240d22b0 0x7f10240cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029192640 1 -- 172.21.15.77:0/783458670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240ce0a0 con 0x7f10240d2710 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029993640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10240d22b0 0x7f10240cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f1029192640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f102415f780 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f101400c400 tx=0x7f101400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f101a7fc640 1 -- 172.21.15.77:0/783458670 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10140052d0 con 0x7f10240d2710 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f101a7fc640 1 -- 172.21.15.77:0/783458670 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1014005490 con 0x7f10240d2710 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102415fcc0 con 0x7f10240d2710 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f101a7fc640 1 -- 172.21.15.77:0/783458670 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1014007260 con 0x7f10240d2710 2024-03-06T07:32:03.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.501+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10240cddc0 con 0x7f10240d2710 2024-03-06T07:32:03.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.502+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102404f330 con 0x7f10240d2710 2024-03-06T07:32:03.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.503+0000 7f101a7fc640 1 -- 172.21.15.77:0/783458670 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1014007400 con 0x7f10240d2710 2024-03-06T07:32:03.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.504+0000 7f101a7fc640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f100806b4d0 0x7f100806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.504+0000 7f101a7fc640 1 -- 172.21.15.77:0/783458670 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f10140897d0 con 0x7f10240d2710 2024-03-06T07:32:03.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.504+0000 7f1029993640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f100806b4d0 0x7f100806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.504+0000 7f1029993640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f100806b4d0 0x7f100806d990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1020005880 tx=0x7f1020009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.508+0000 7f101a7fc640 1 -- 172.21.15.77:0/783458670 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10140585b0 con 0x7f10240d2710 2024-03-06T07:32:03.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.717+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10240670d0 con 0x7f10240d2710 2024-03-06T07:32:03.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.718+0000 7f101a7fc640 1 -- 172.21.15.77:0/783458670 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f1014057eb0 con 0x7f10240d2710 2024-03-06T07:32:03.720 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:03.720 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:03.720 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f100806b4d0 msgr2=0x7f100806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f100806b4d0 0x7f100806d990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1020005880 tx=0x7f1020009040 comp rx=0 tx=0).stop 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 msgr2=0x7f102415f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f102415f780 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f101400c400 tx=0x7f101400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 shutdown_connections 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f100806b4d0 0x7f100806d990 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10240d2710 0x7f102415f780 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f102415dfe0 0x7f102414af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 --2- 172.21.15.77:0/783458670 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f10240d22b0 0x7f10240cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 >> 172.21.15.77:0/783458670 conn(0x7f10240d3cd0 msgr2=0x7f10240ca480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 shutdown_connections 2024-03-06T07:32:03.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:03.720+0000 7f102bbf7640 1 -- 172.21.15.77:0/783458670 wait complete. 2024-03-06T07:32:03.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:03.934 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:04.934 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:05.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.034+0000 7f43fea6f640 1 Processor -- start 2024-03-06T07:32:05.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.034+0000 7f43fea6f640 1 -- start start 2024-03-06T07:32:05.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fea6f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80d4760 0x7f43f80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fea6f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f43f80d0a80 0x7f43f80cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fea6f640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f80d25d0 con 0x7f43f80b4170 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fea6f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f80d4bf0 con 0x7f43f80d0a80 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fea6f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80ce9e0 con 0x7f43f80d4760 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fd00c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f43f80d0a80 0x7f43f80cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fd00c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f43f80d0a80 0x7f43f80cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56412/0 (socket says 172.21.15.77:56412) 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fd00c640 1 -- 172.21.15.77:0/1229372911 learned_addr learned my addr 172.21.15.77:0/1229372911 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fd00c640 1 -- 172.21.15.77:0/1229372911 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80d4760 msgr2=0x7f43f80b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fd00c640 1 --2- 172.21.15.77:0/1229372911 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80d4760 0x7f43f80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fd00c640 1 -- 172.21.15.77:0/1229372911 >> v1:172.21.15.158:6789/0 conn(0x7f43f80b4170 legacy=0x7f43f80d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.035+0000 7f43fd00c640 1 -- 172.21.15.77:0/1229372911 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f80d2870 con 0x7f43f80d0a80 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43fd00c640 1 --2- 172.21.15.77:0/1229372911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80d0a80 0x7f43f80cee70 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009e50 tx=0x7f43f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5898e691dd3730 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43f77fe640 1 -- 172.21.15.77:0/1229372911 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f002ca70 con 0x7f43f80d0a80 2024-03-06T07:32:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43f77fe640 1 -- 172.21.15.77:0/1229372911 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f002cc10 con 0x7f43f80d0a80 2024-03-06T07:32:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43f77fe640 1 -- 172.21.15.77:0/1229372911 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f00343f0 con 0x7f43f80d0a80 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43fea6f640 1 -- 172.21.15.77:0/1229372911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80d0a80 msgr2=0x7f43f80cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43fea6f640 1 --2- 172.21.15.77:0/1229372911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80d0a80 0x7f43f80cee70 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009e50 tx=0x7f43f002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43fea6f640 1 -- 172.21.15.77:0/1229372911 shutdown_connections 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43fea6f640 1 --2- 172.21.15.77:0/1229372911 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80d4760 0x7f43f80b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43fea6f640 1 --2- 172.21.15.77:0/1229372911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80d0a80 0x7f43f80cee70 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43fea6f640 1 -- 172.21.15.77:0/1229372911 >> 172.21.15.77:0/1229372911 conn(0x7f43f80d3c40 msgr2=0x7f43f80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.036+0000 7f43fea6f640 1 -- 172.21.15.77:0/1229372911 shutdown_connections 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.037+0000 7f43fea6f640 1 -- 172.21.15.77:0/1229372911 wait complete. 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.038+0000 7f43fea6f640 1 Processor -- start 2024-03-06T07:32:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.038+0000 7f43fea6f640 1 -- start start 2024-03-06T07:32:05.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.038+0000 7f43fea6f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80b4170 0x7f43f80cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.038+0000 7f43fea6f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43f80d0a80 0x7f43f80b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fea6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80ca2d0 0x7f43f814cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fea6f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80b4dd0 con 0x7f43f80ca2d0 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fea6f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80b4550 con 0x7f43f80d0a80 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fea6f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80b4f50 con 0x7f43f80b4170 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80ca2d0 0x7f43f814cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80ca2d0 0x7f43f814cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56428/0 (socket says 172.21.15.77:56428) 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 -- 172.21.15.77:0/386377580 learned_addr learned my addr 172.21.15.77:0/386377580 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43f7fff640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43f80d0a80 0x7f43f80b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fc80b640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80b4170 0x7f43f80cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 -- 172.21.15.77:0/386377580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80b4170 msgr2=0x7f43f80cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80b4170 0x7f43f80cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 -- 172.21.15.77:0/386377580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43f80d0a80 msgr2=0x7f43f80b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43f80d0a80 0x7f43f80b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 -- 172.21.15.77:0/386377580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f80cb610 con 0x7f43f80ca2d0 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fd00c640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80ca2d0 0x7f43f814cc40 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f43f0036040 tx=0x7f43f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43f57fa640 1 -- 172.21.15.77:0/386377580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0043020 con 0x7f43f80ca2d0 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43f57fa640 1 -- 172.21.15.77:0/386377580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f003e450 con 0x7f43f80ca2d0 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.039+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f80cb8f0 con 0x7f43f80ca2d0 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.040+0000 7f43f57fa640 1 -- 172.21.15.77:0/386377580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f003e820 con 0x7f43f80ca2d0 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.040+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f80cbeb0 con 0x7f43f80ca2d0 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.041+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f804f330 con 0x7f43f80ca2d0 2024-03-06T07:32:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.041+0000 7f43f57fa640 1 -- 172.21.15.77:0/386377580 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43f0046020 con 0x7f43f80ca2d0 2024-03-06T07:32:05.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.041+0000 7f43f57fa640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43dc06b4d0 0x7f43dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.041+0000 7f43f57fa640 1 -- 172.21.15.77:0/386377580 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f43f00acad0 con 0x7f43f80ca2d0 2024-03-06T07:32:05.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.041+0000 7f43fc80b640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43dc06b4d0 0x7f43dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.042+0000 7f43fc80b640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43dc06b4d0 0x7f43dc06d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f43ec007fd0 tx=0x7f43ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.046+0000 7f43f57fa640 1 -- 172.21.15.77:0/386377580 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43f004d030 con 0x7f43f80ca2d0 2024-03-06T07:32:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.262+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43f80733b0 con 0x7f43f80ca2d0 2024-03-06T07:32:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.263+0000 7f43f57fa640 1 -- 172.21.15.77:0/386377580 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f43f007b9c0 con 0x7f43f80ca2d0 2024-03-06T07:32:05.265 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:05.265 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:05.265 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:05.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.264+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43dc06b4d0 msgr2=0x7f43dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.264+0000 7f43fea6f640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43dc06b4d0 0x7f43dc06d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f43ec007fd0 tx=0x7f43ec009040 comp rx=0 tx=0).stop 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.264+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80ca2d0 msgr2=0x7f43f814cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.264+0000 7f43fea6f640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80ca2d0 0x7f43f814cc40 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f43f0036040 tx=0x7f43f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.265+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 shutdown_connections 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.265+0000 7f43fea6f640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f43dc06b4d0 0x7f43dc06d990 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.265+0000 7f43fea6f640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43f80ca2d0 0x7f43f814cc40 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.265+0000 7f43fea6f640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f43f80d0a80 0x7f43f80b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.265+0000 7f43fea6f640 1 --2- 172.21.15.77:0/386377580 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f43f80b4170 0x7f43f80cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.265+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 >> 172.21.15.77:0/386377580 conn(0x7f43f80d3c40 msgr2=0x7f43f80b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.265+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 shutdown_connections 2024-03-06T07:32:05.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:05.265+0000 7f43fea6f640 1 -- 172.21.15.77:0/386377580 wait complete. 2024-03-06T07:32:05.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:05.478 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:06.480 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:06.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.579+0000 7ff211763640 1 Processor -- start 2024-03-06T07:32:06.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.579+0000 7ff211763640 1 -- start start 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.580+0000 7ff211763640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0d22b0 0x7ff20c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.580+0000 7ff211763640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff20c0b41f0 0x7ff20c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.580+0000 7ff211763640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20c0b3430 con 0x7ff20c15dfe0 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.580+0000 7ff211763640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c0d2c50 con 0x7ff20c0b41f0 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.580+0000 7ff211763640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0b31e0 con 0x7ff20c0d22b0 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.580+0000 7ff20b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff20c0b41f0 0x7ff20c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.580+0000 7ff20b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff20c0b41f0 0x7ff20c0b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56440/0 (socket says 172.21.15.77:56440) 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.580+0000 7ff20b7fe640 1 -- 172.21.15.77:0/2973116466 learned_addr learned my addr 172.21.15.77:0/2973116466 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.581+0000 7ff203fff640 1 --2- 172.21.15.77:0/2973116466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0d22b0 0x7ff20c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.581+0000 7ff20b7fe640 1 -- 172.21.15.77:0/2973116466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0d22b0 msgr2=0x7ff20c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.581+0000 7ff20b7fe640 1 --2- 172.21.15.77:0/2973116466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0d22b0 0x7ff20c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.581+0000 7ff20b7fe640 1 -- 172.21.15.77:0/2973116466 >> v1:172.21.15.158:6789/0 conn(0x7ff20c15dfe0 legacy=0x7ff20c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.581+0000 7ff20b7fe640 1 -- 172.21.15.77:0/2973116466 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c0b3710 con 0x7ff20c0b41f0 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.581+0000 7ff20b7fe640 1 --2- 172.21.15.77:0/2973116466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0b41f0 0x7ff20c0b4650 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7ff204009e50 tx=0x7ff20402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898a64b6938cd8e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff20a7fc640 1 -- 172.21.15.77:0/2973116466 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20402cb00 con 0x7ff20c0b41f0 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff20a7fc640 1 -- 172.21.15.77:0/2973116466 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20402cca0 con 0x7ff20c0b41f0 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff20a7fc640 1 -- 172.21.15.77:0/2973116466 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204034610 con 0x7ff20c0b41f0 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff211763640 1 -- 172.21.15.77:0/2973116466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0b41f0 msgr2=0x7ff20c0b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff211763640 1 --2- 172.21.15.77:0/2973116466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0b41f0 0x7ff20c0b4650 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7ff204009e50 tx=0x7ff20402bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff211763640 1 -- 172.21.15.77:0/2973116466 shutdown_connections 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff211763640 1 --2- 172.21.15.77:0/2973116466 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0d22b0 0x7ff20c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff211763640 1 --2- 172.21.15.77:0/2973116466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c0b41f0 0x7ff20c0b4650 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff211763640 1 -- 172.21.15.77:0/2973116466 >> 172.21.15.77:0/2973116466 conn(0x7ff20c0d3cd0 msgr2=0x7ff20c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff211763640 1 -- 172.21.15.77:0/2973116466 shutdown_connections 2024-03-06T07:32:06.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.582+0000 7ff211763640 1 -- 172.21.15.77:0/2973116466 wait complete. 2024-03-06T07:32:06.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff211763640 1 Processor -- start 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff211763640 1 -- start start 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff211763640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0b41f0 0x7ff20c0b9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff211763640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0d22b0 0x7ff20c0b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff211763640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c15dfe0 0x7ff20c1610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff211763640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0cfa00 con 0x7ff20c15dfe0 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff211763640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0b3d60 con 0x7ff20c0d22b0 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff211763640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0d1c70 con 0x7ff20c0b41f0 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.584+0000 7ff20b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c15dfe0 0x7ff20c1610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c15dfe0 0x7ff20c1610d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56456/0 (socket says 172.21.15.77:56456) 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20b7fe640 1 -- 172.21.15.77:0/317880596 learned_addr learned my addr 172.21.15.77:0/317880596 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff203fff640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0d22b0 0x7ff20c0b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20affd640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0b41f0 0x7ff20c0b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20b7fe640 1 -- 172.21.15.77:0/317880596 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0b41f0 msgr2=0x7ff20c0b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20b7fe640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0b41f0 0x7ff20c0b9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20b7fe640 1 -- 172.21.15.77:0/317880596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0d22b0 msgr2=0x7ff20c0b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20b7fe640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0d22b0 0x7ff20c0b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20b7fe640 1 -- 172.21.15.77:0/317880596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c15a700 con 0x7ff20c15dfe0 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff20b7fe640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c15dfe0 0x7ff20c1610d0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7ff204036040 tx=0x7ff204004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff2037fe640 1 -- 172.21.15.77:0/317880596 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204043020 con 0x7ff20c15dfe0 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff2037fe640 1 -- 172.21.15.77:0/317880596 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20402ce50 con 0x7ff20c15dfe0 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff2037fe640 1 -- 172.21.15.77:0/317880596 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204009510 con 0x7ff20c15dfe0 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c0b86f0 con 0x7ff20c15dfe0 2024-03-06T07:32:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.585+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c0b8cb0 con 0x7ff20c15dfe0 2024-03-06T07:32:06.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.586+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c04f330 con 0x7ff20c15dfe0 2024-03-06T07:32:06.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.587+0000 7ff2037fe640 1 -- 172.21.15.77:0/317880596 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff204005d10 con 0x7ff20c15dfe0 2024-03-06T07:32:06.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.587+0000 7ff2037fe640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.587+0000 7ff20affd640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.587+0000 7ff2037fe640 1 -- 172.21.15.77:0/317880596 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff2040aca10 con 0x7ff20c15dfe0 2024-03-06T07:32:06.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.588+0000 7ff20affd640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0016e0 tx=0x7ff1fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.592+0000 7ff2037fe640 1 -- 172.21.15.77:0/317880596 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff20407b220 con 0x7ff20c15dfe0 2024-03-06T07:32:06.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.821+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff20c0ce910 con 0x7ff20c15dfe0 2024-03-06T07:32:06.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.821+0000 7ff2037fe640 1 -- 172.21.15.77:0/317880596 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff20407f2c0 con 0x7ff20c15dfe0 2024-03-06T07:32:06.823 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:06.823 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:06.823 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:06.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1ec06b4d0 msgr2=0x7ff1ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0016e0 tx=0x7ff1fc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c15dfe0 msgr2=0x7ff20c1610d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c15dfe0 0x7ff20c1610d0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7ff204036040 tx=0x7ff204004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:06.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 shutdown_connections 2024-03-06T07:32:06.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff20c15dfe0 0x7ff20c1610d0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff20c0d22b0 0x7ff20c0b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 --2- 172.21.15.77:0/317880596 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff20c0b41f0 0x7ff20c0b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 >> 172.21.15.77:0/317880596 conn(0x7ff20c0d3cd0 msgr2=0x7ff20c06d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 shutdown_connections 2024-03-06T07:32:06.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:06.823+0000 7ff211763640 1 -- 172.21.15.77:0/317880596 wait complete. 2024-03-06T07:32:07.038 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:07.038 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:08.039 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.138+0000 7f13e8866640 1 Processor -- start 2024-03-06T07:32:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.138+0000 7f13e8866640 1 -- start start 2024-03-06T07:32:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e8866640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f13e415dfe0 0x7f13e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e8866640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13e40d29c0 0x7f13e40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e8866640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e40b3430 con 0x7f13e40cf830 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e8866640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e40b31e0 con 0x7f13e40d29c0 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e8866640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e40b3d60 con 0x7f13e415dfe0 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e2d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13e40d29c0 0x7f13e40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e2d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13e40d29c0 0x7f13e40b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56470/0 (socket says 172.21.15.77:56470) 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e2d9d640 1 -- 172.21.15.77:0/3191780064 learned_addr learned my addr 172.21.15.77:0/3191780064 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e259c640 1 --2- 172.21.15.77:0/3191780064 >> v2:172.21.15.158:3300/0 conn(0x7f13e415dfe0 0x7f13e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e2d9d640 1 -- 172.21.15.77:0/3191780064 >> v2:172.21.15.158:3300/0 conn(0x7f13e415dfe0 msgr2=0x7f13e40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e2d9d640 1 --2- 172.21.15.77:0/3191780064 >> v2:172.21.15.158:3300/0 conn(0x7f13e415dfe0 0x7f13e40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e2d9d640 1 -- 172.21.15.77:0/3191780064 >> v1:172.21.15.158:6789/0 conn(0x7f13e40cf830 legacy=0x7f13e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.139+0000 7f13e2d9d640 1 -- 172.21.15.77:0/3191780064 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e40b3710 con 0x7f13e40d29c0 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.140+0000 7f13e2d9d640 1 --2- 172.21.15.77:0/3191780064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e40d29c0 0x7f13e40b41f0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f13e4067990 tx=0x7f13dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1870aea4ba29f1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.140+0000 7f13e159a640 1 -- 172.21.15.77:0/3191780064 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc02ca80 con 0x7f13e40d29c0 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.140+0000 7f13e159a640 1 -- 172.21.15.77:0/3191780064 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc02cc20 con 0x7f13e40d29c0 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.140+0000 7f13e159a640 1 -- 172.21.15.77:0/3191780064 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc0343f0 con 0x7f13e40d29c0 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.141+0000 7f13e8866640 1 -- 172.21.15.77:0/3191780064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e40d29c0 msgr2=0x7f13e40b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.141+0000 7f13e8866640 1 --2- 172.21.15.77:0/3191780064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e40d29c0 0x7f13e40b41f0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f13e4067990 tx=0x7f13dc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.141+0000 7f13e8866640 1 -- 172.21.15.77:0/3191780064 shutdown_connections 2024-03-06T07:32:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.141+0000 7f13e8866640 1 --2- 172.21.15.77:0/3191780064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e40d29c0 0x7f13e40b41f0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.141+0000 7f13e8866640 1 --2- 172.21.15.77:0/3191780064 >> v2:172.21.15.158:3300/0 conn(0x7f13e415dfe0 0x7f13e40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.141+0000 7f13e8866640 1 -- 172.21.15.77:0/3191780064 >> 172.21.15.77:0/3191780064 conn(0x7f13e40d3cd0 msgr2=0x7f13e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.141+0000 7f13e8866640 1 -- 172.21.15.77:0/3191780064 shutdown_connections 2024-03-06T07:32:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.141+0000 7f13e8866640 1 -- 172.21.15.77:0/3191780064 wait complete. 2024-03-06T07:32:08.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.143+0000 7f13e8866640 1 Processor -- start 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.143+0000 7f13e8866640 1 -- start start 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e8866640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13e40cf830 0x7f13e40cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e8866640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13e40d29c0 0x7f13e40cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e8866640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e415dfe0 0x7f13e406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e8866640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e40d1df0 con 0x7f13e415dfe0 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e8866640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e40d1c70 con 0x7f13e40d29c0 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e8866640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e415e3c0 con 0x7f13e40cf830 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e415dfe0 0x7f13e406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e415dfe0 0x7f13e406ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56486/0 (socket says 172.21.15.77:56486) 2024-03-06T07:32:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e2d9d640 1 -- 172.21.15.77:0/120107192 learned_addr learned my addr 172.21.15.77:0/120107192 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e1d9b640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13e40d29c0 0x7f13e40cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.144+0000 7f13e259c640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13e40cf830 0x7f13e40cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13e2d9d640 1 -- 172.21.15.77:0/120107192 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13e40cf830 msgr2=0x7f13e40cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13e2d9d640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13e40cf830 0x7f13e40cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13e2d9d640 1 -- 172.21.15.77:0/120107192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13e40d29c0 msgr2=0x7f13e40cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13e2d9d640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13e40d29c0 0x7f13e40cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13e2d9d640 1 -- 172.21.15.77:0/120107192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e4159370 con 0x7f13e415dfe0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13e2d9d640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e415dfe0 0x7f13e406ccb0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f13dc036040 tx=0x7f13dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13d2ffd640 1 -- 172.21.15.77:0/120107192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc043020 con 0x7f13e415dfe0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e40cbd80 con 0x7f13e415dfe0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13d2ffd640 1 -- 172.21.15.77:0/120107192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc03e450 con 0x7f13e415dfe0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13d2ffd640 1 -- 172.21.15.77:0/120107192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc03e860 con 0x7f13e415dfe0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.145+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e40cdd60 con 0x7f13e415dfe0 2024-03-06T07:32:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.146+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e404f330 con 0x7f13e415dfe0 2024-03-06T07:32:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.147+0000 7f13d2ffd640 1 -- 172.21.15.77:0/120107192 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13dc046020 con 0x7f13e415dfe0 2024-03-06T07:32:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.147+0000 7f13d2ffd640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f13c406b4d0 0x7f13c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.147+0000 7f13d2ffd640 1 -- 172.21.15.77:0/120107192 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f13dc0ac9e0 con 0x7f13e415dfe0 2024-03-06T07:32:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.147+0000 7f13e259c640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f13c406b4d0 0x7f13c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.147+0000 7f13e259c640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f13c406b4d0 0x7f13c406d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f13d400aa20 tx=0x7f13d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.151+0000 7f13d2ffd640 1 -- 172.21.15.77:0/120107192 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13dc04d030 con 0x7f13e415dfe0 2024-03-06T07:32:08.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.365+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13e40d2630 con 0x7f13e415dfe0 2024-03-06T07:32:08.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.366+0000 7f13d2ffd640 1 -- 172.21.15.77:0/120107192 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f13dc07b200 con 0x7f13e415dfe0 2024-03-06T07:32:08.369 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:08.369 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:08.369 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:08.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f13c406b4d0 msgr2=0x7f13c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f13c406b4d0 0x7f13c406d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f13d400aa20 tx=0x7f13d4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:08.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e415dfe0 msgr2=0x7f13e406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e415dfe0 0x7f13e406ccb0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f13dc036040 tx=0x7f13dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:08.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 shutdown_connections 2024-03-06T07:32:08.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f13c406b4d0 0x7f13c406d990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e415dfe0 0x7f13e406ccb0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f13e40d29c0 0x7f13e40cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 --2- 172.21.15.77:0/120107192 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f13e40cf830 0x7f13e40cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 >> 172.21.15.77:0/120107192 conn(0x7f13e40d3cd0 msgr2=0x7f13e40b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 shutdown_connections 2024-03-06T07:32:08.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:08.368+0000 7f13e8866640 1 -- 172.21.15.77:0/120107192 wait complete. 2024-03-06T07:32:08.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:08.583 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:09.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:09.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.683+0000 7fb5a971e640 1 Processor -- start 2024-03-06T07:32:09.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.683+0000 7fb5a971e640 1 -- start start 2024-03-06T07:32:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.684+0000 7fb5a971e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5a40cdeb0 0x7fb5a415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.684+0000 7fb5a971e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a4071f40 con 0x7fb5a40cce40 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.684+0000 7fb5a971e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a4070a70 con 0x7fb5a41515e0 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.684+0000 7fb5a971e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a40cd2f0 con 0x7fb5a40cdeb0 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.684+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5a40cdeb0 0x7fb5a415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.684+0000 7fb59bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb5a40cce40 0x7fb5a40cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55276/0 (socket says 172.21.15.77:55276) 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.684+0000 7fb59bfff640 1 -- 172.21.15.77:0/811166394 learned_addr learned my addr 172.21.15.77:0/811166394 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.685+0000 7fb5a37fe640 1 -- 172.21.15.77:0/811166394 >> v1:172.21.15.158:6789/0 conn(0x7fb5a41515e0 legacy=0x7fb5a4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.685+0000 7fb5a37fe640 1 -- 172.21.15.77:0/811166394 >> v1:172.21.15.77:6789/0 conn(0x7fb5a40cce40 legacy=0x7fb5a40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.685+0000 7fb5a27fc640 1 -- 172.21.15.77:0/811166394 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096492686 0 0) 0x7fb5a4071f40 con 0x7fb5a40cce40 2024-03-06T07:32:09.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.685+0000 7fb5a37fe640 1 -- 172.21.15.77:0/811166394 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4070d50 con 0x7fb5a40cdeb0 2024-03-06T07:32:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.686+0000 7fb5a37fe640 1 --2- 172.21.15.77:0/811166394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cdeb0 0x7fb5a415b430 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009d90 tx=0x7fb59c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9b0d69a6b1ee52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.686+0000 7fb5a27fc640 1 -- 172.21.15.77:0/811166394 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c039070 con 0x7fb5a40cdeb0 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.686+0000 7fb5a27fc640 1 -- 172.21.15.77:0/811166394 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c02cc60 con 0x7fb5a40cdeb0 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.686+0000 7fb5a27fc640 1 -- 172.21.15.77:0/811166394 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c034660 con 0x7fb5a40cdeb0 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.687+0000 7fb5a971e640 1 -- 172.21.15.77:0/811166394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cdeb0 msgr2=0x7fb5a415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.687+0000 7fb5a971e640 1 --2- 172.21.15.77:0/811166394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cdeb0 0x7fb5a415b430 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009d90 tx=0x7fb59c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.687+0000 7fb5a971e640 1 -- 172.21.15.77:0/811166394 shutdown_connections 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.687+0000 7fb5a971e640 1 --2- 172.21.15.77:0/811166394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cdeb0 0x7fb5a415b430 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.687+0000 7fb5a971e640 1 -- 172.21.15.77:0/811166394 >> 172.21.15.77:0/811166394 conn(0x7fb5a40cf8a0 msgr2=0x7fb5a40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.687+0000 7fb5a971e640 1 -- 172.21.15.77:0/811166394 shutdown_connections 2024-03-06T07:32:09.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.687+0000 7fb5a971e640 1 -- 172.21.15.77:0/811166394 wait complete. 2024-03-06T07:32:09.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.688+0000 7fb5a971e640 1 Processor -- start 2024-03-06T07:32:09.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.688+0000 7fb5a971e640 1 -- start start 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a971e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cce40 0x7fb5a4143b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a971e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5a40cdeb0 0x7fb5a4154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a971e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5a41515e0 0x7fb5a41f1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a971e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a40cc3a0 con 0x7fb5a40cce40 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a971e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a4071310 con 0x7fb5a40cdeb0 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a971e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a40cc520 con 0x7fb5a41515e0 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cce40 0x7fb5a4143b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cce40 0x7fb5a4143b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43020/0 (socket says 172.21.15.77:43020) 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a2ffd640 1 -- 172.21.15.77:0/4193029660 learned_addr learned my addr 172.21.15.77:0/4193029660 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb59bfff640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5a40cdeb0 0x7fb5a4154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a2ffd640 1 -- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5a41515e0 msgr2=0x7fb5a41f1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a37fe640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5a41515e0 0x7fb5a41f1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a2ffd640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5a41515e0 0x7fb5a41f1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a2ffd640 1 -- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5a40cdeb0 msgr2=0x7fb5a4154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a2ffd640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5a40cdeb0 0x7fb5a4154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb5a2ffd640 1 -- 172.21.15.77:0/4193029660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4144100 con 0x7fb5a40cce40 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.689+0000 7fb59bfff640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5a40cdeb0 0x7fb5a4154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.690+0000 7fb5a2ffd640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cce40 0x7fb5a4143b20 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fb59400b420 tx=0x7fb59400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.690+0000 7fb59b7fe640 1 -- 172.21.15.77:0/4193029660 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594011830 con 0x7fb5a40cce40 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.690+0000 7fb59b7fe640 1 -- 172.21.15.77:0/4193029660 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb594016070 con 0x7fb5a40cce40 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.690+0000 7fb59b7fe640 1 -- 172.21.15.77:0/4193029660 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594011c10 con 0x7fb5a40cce40 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.690+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a41443e0 con 0x7fb5a40cce40 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.690+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a4136e90 con 0x7fb5a40cce40 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.691+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a4068840 con 0x7fb5a40cce40 2024-03-06T07:32:09.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.691+0000 7fb59b7fe640 1 -- 172.21.15.77:0/4193029660 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb59401b460 con 0x7fb5a40cce40 2024-03-06T07:32:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.692+0000 7fb59b7fe640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb58406b4d0 0x7fb58406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.692+0000 7fb59bfff640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb58406b4d0 0x7fb58406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.692+0000 7fb59b7fe640 1 -- 172.21.15.77:0/4193029660 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb594089cb0 con 0x7fb5a40cce40 2024-03-06T07:32:09.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.692+0000 7fb59bfff640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb58406b4d0 0x7fb58406d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb58c007fd0 tx=0x7fb58c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.696+0000 7fb59b7fe640 1 -- 172.21.15.77:0/4193029660 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb594020020 con 0x7fb5a40cce40 2024-03-06T07:32:09.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.929+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5a404f330 con 0x7fb5a40cce40 2024-03-06T07:32:09.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.929+0000 7fb59b7fe640 1 -- 172.21.15.77:0/4193029660 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fb594058ba0 con 0x7fb5a40cce40 2024-03-06T07:32:09.932 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:09.932 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:09.933 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:09.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb58406b4d0 msgr2=0x7fb58406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb58406b4d0 0x7fb58406d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb58c007fd0 tx=0x7fb58c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cce40 msgr2=0x7fb5a4143b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cce40 0x7fb5a4143b20 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fb59400b420 tx=0x7fb59400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 shutdown_connections 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5a41515e0 0x7fb5a41f1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5a40cdeb0 0x7fb5a4154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb58406b4d0 0x7fb58406d990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 --2- 172.21.15.77:0/4193029660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a40cce40 0x7fb5a4143b20 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 >> 172.21.15.77:0/4193029660 conn(0x7fb5a40cf8a0 msgr2=0x7fb5a414f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.931+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 shutdown_connections 2024-03-06T07:32:09.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:09.932+0000 7fb5a971e640 1 -- 172.21.15.77:0/4193029660 wait complete. 2024-03-06T07:32:10.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:10.150 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:11.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:11.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.249+0000 7f06055eb640 1 Processor -- start 2024-03-06T07:32:11.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f06055eb640 1 -- start start 2024-03-06T07:32:11.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f06055eb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06000cf790 0x7f06000ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f06055eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 0x7f0600071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f06055eb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0600072220 0x7f06001619b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f06055eb640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f060006cf70 con 0x7f06000cf790 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f06055eb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06000ae240 con 0x7f0600072220 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f06055eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06000ae520 con 0x7f06000adda0 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f05feffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06000cf790 0x7f06000ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f05feffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06000cf790 0x7f06000ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43026/0 (socket says 172.21.15.77:43026) 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.250+0000 7f05feffd640 1 -- 172.21.15.77:0/306629325 learned_addr learned my addr 172.21.15.77:0/306629325 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.251+0000 7f05feffd640 1 -- 172.21.15.77:0/306629325 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 msgr2=0x7f0600071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.251+0000 7f05feffd640 1 --2- 172.21.15.77:0/306629325 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 0x7f0600071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.251+0000 7f05feffd640 1 -- 172.21.15.77:0/306629325 >> v2:172.21.15.158:3300/0 conn(0x7f0600072220 msgr2=0x7f06001619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.251+0000 7f05feffd640 1 --2- 172.21.15.77:0/306629325 >> v2:172.21.15.158:3300/0 conn(0x7f0600072220 0x7f06001619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.251+0000 7f05feffd640 1 -- 172.21.15.77:0/306629325 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000ae800 con 0x7f06000cf790 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.251+0000 7f05feffd640 1 --2- 172.21.15.77:0/306629325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f06000ad860 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f06000b3500 tx=0x7f05f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d8c36872a959ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f05fdffb640 1 -- 172.21.15.77:0/306629325 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0035020 con 0x7f06000cf790 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f05fdffb640 1 -- 172.21.15.77:0/306629325 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0031870 con 0x7f06000cf790 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f05fdffb640 1 -- 172.21.15.77:0/306629325 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0031c40 con 0x7f06000cf790 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 -- 172.21.15.77:0/306629325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 msgr2=0x7f06000ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 --2- 172.21.15.77:0/306629325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f06000ad860 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f06000b3500 tx=0x7f05f002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 -- 172.21.15.77:0/306629325 shutdown_connections 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 --2- 172.21.15.77:0/306629325 >> v2:172.21.15.158:3300/0 conn(0x7f0600072220 0x7f06001619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 --2- 172.21.15.77:0/306629325 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 0x7f0600071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 --2- 172.21.15.77:0/306629325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f06000ad860 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 -- 172.21.15.77:0/306629325 >> 172.21.15.77:0/306629325 conn(0x7f06000b1a90 msgr2=0x7f06000b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 -- 172.21.15.77:0/306629325 shutdown_connections 2024-03-06T07:32:11.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.252+0000 7f06055eb640 1 -- 172.21.15.77:0/306629325 wait complete. 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f06055eb640 1 Processor -- start 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f06055eb640 1 -- start start 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f06055eb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0600072220 0x7f06001570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f06055eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 0x7f06001e8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f06055eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f060015b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f06055eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060015e700 con 0x7f06000cf790 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f06055eb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060015e580 con 0x7f0600072220 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f06055eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0600161ef0 con 0x7f06000adda0 2024-03-06T07:32:11.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f060015b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f060015b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43036/0 (socket says 172.21.15.77:43036) 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.254+0000 7f05ff7fe640 1 -- 172.21.15.77:0/133435669 learned_addr learned my addr 172.21.15.77:0/133435669 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05feffd640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0600072220 0x7f06001570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05fe7fc640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 0x7f06001e8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05ff7fe640 1 -- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 msgr2=0x7f06001e8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05ff7fe640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 0x7f06001e8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05ff7fe640 1 -- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0600072220 msgr2=0x7f06001570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05ff7fe640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0600072220 0x7f06001570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05ff7fe640 1 -- 172.21.15.77:0/133435669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06001ea500 con 0x7f06000cf790 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05fe7fc640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 0x7f06001e8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05ff7fe640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f060015b6a0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f05f800b450 tx=0x7f05f800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05f77fe640 1 -- 172.21.15.77:0/133435669 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f8015020 con 0x7f06000cf790 2024-03-06T07:32:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05f77fe640 1 -- 172.21.15.77:0/133435669 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f8011830 con 0x7f06000cf790 2024-03-06T07:32:11.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06001e9460 con 0x7f06000cf790 2024-03-06T07:32:11.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f05f77fe640 1 -- 172.21.15.77:0/133435669 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f8011c00 con 0x7f06000cf790 2024-03-06T07:32:11.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.255+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06001e9a20 con 0x7f06000cf790 2024-03-06T07:32:11.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.256+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060004f330 con 0x7f06000cf790 2024-03-06T07:32:11.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.256+0000 7f05f77fe640 1 -- 172.21.15.77:0/133435669 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05f8018020 con 0x7f06000cf790 2024-03-06T07:32:11.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.257+0000 7f05f77fe640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.257+0000 7f05f77fe640 1 -- 172.21.15.77:0/133435669 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f05f808a140 con 0x7f06000cf790 2024-03-06T07:32:11.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.257+0000 7f05feffd640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.257+0000 7f05feffd640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f05e006b4d0 0x7f05e006d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f05f0036000 tx=0x7f05f003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.261+0000 7f05f77fe640 1 -- 172.21.15.77:0/133435669 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05f801f020 con 0x7f06000cf790 2024-03-06T07:32:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.474+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06000b0b20 con 0x7f06000cf790 2024-03-06T07:32:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.474+0000 7f05f77fe640 1 -- 172.21.15.77:0/133435669 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f05f80172c0 con 0x7f06000cf790 2024-03-06T07:32:11.478 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:11.478 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f05e006b4d0 msgr2=0x7f05e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f05e006b4d0 0x7f05e006d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f05f0036000 tx=0x7f05f003c040 comp rx=0 tx=0).stop 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 msgr2=0x7f060015b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f060015b6a0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f05f800b450 tx=0x7f05f800b910 comp rx=0 tx=0).stop 2024-03-06T07:32:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 shutdown_connections 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06000cf790 0x7f060015b6a0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f06000adda0 0x7f06001e8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 --2- 172.21.15.77:0/133435669 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0600072220 0x7f06001570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 >> 172.21.15.77:0/133435669 conn(0x7f06000b1a90 msgr2=0x7f06001608b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 shutdown_connections 2024-03-06T07:32:11.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:11.476+0000 7f06055eb640 1 -- 172.21.15.77:0/133435669 wait complete. 2024-03-06T07:32:11.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:11.691 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:12.693 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:12.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.792+0000 7f26a3e81640 1 Processor -- start 2024-03-06T07:32:12.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.792+0000 7f26a3e81640 1 -- start start 2024-03-06T07:32:12.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a3e81640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f269c13dc10 0x7f269c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a3e81640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269c15f580 con 0x7f269c15bf30 2024-03-06T07:32:12.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a3e81640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269c15b450 con 0x7f269c15ef60 2024-03-06T07:32:12.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a3e81640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f269c0cf3f0 con 0x7f269c13dc10 2024-03-06T07:32:12.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a141c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f269c15bf30 0x7f269c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55286/0 (socket says 172.21.15.77:55286) 2024-03-06T07:32:12.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a141c640 1 -- 172.21.15.77:0/2304243175 learned_addr learned my addr 172.21.15.77:0/2304243175 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:12.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189820932 0 0) 0x7f269c15f580 con 0x7f269c15bf30 2024-03-06T07:32:12.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a1c1d640 1 --2- 172.21.15.77:0/2304243175 >> v2:172.21.15.158:3300/0 conn(0x7f269c13dc10 0x7f269c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.793+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2688003520 con 0x7f269c15bf30 2024-03-06T07:32:12.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661542355 0 0) 0x7f2688003520 con 0x7f269c15bf30 2024-03-06T07:32:12.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 >> v2:172.21.15.158:3300/0 conn(0x7f269c13dc10 msgr2=0x7f269c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 --2- 172.21.15.77:0/2304243175 >> v2:172.21.15.158:3300/0 conn(0x7f269c13dc10 0x7f269c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 >> v1:172.21.15.158:6789/0 conn(0x7f269c15ef60 legacy=0x7f269c0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c15b730 con 0x7f269c15bf30 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f2698001670 con 0x7f269c15bf30 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 >> v1:172.21.15.77:6789/0 conn(0x7f269c15bf30 legacy=0x7f269c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26880039b0 0x7f2688003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2688004800 0x7f2688004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f268800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0cf3f0 con 0x7f26880051c0 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268800c160 con 0x7f26880039b0 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268800c360 con 0x7f2688004800 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a241e640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f268800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a1c1d640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26880039b0 0x7f2688003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a141c640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2688004800 0x7f2688004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a241e640 1 -- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2688004800 msgr2=0x7f2688004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a241e640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2688004800 0x7f2688004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a241e640 1 -- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26880039b0 msgr2=0x7f2688003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a241e640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26880039b0 0x7f2688003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a241e640 1 -- 172.21.15.77:0/2304243175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688003520 con 0x7f26880051c0 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.794+0000 7f26a141c640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2688004800 0x7f2688004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.795+0000 7f26a241e640 1 -- 172.21.15.77:0/2304243175 reap_dead start 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.795+0000 7f26a241e640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f268800b9c0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f2694001e50 tx=0x7f2688005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee8d7a5d5a90bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.795+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694001a70 con 0x7f26880051c0 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.795+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2694003b90 con 0x7f26880051c0 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.795+0000 7f26a0c1b640 1 -- 172.21.15.77:0/2304243175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26940050d0 con 0x7f26880051c0 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.796+0000 7f26a3e81640 1 -- 172.21.15.77:0/2304243175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 msgr2=0x7f268800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.796+0000 7f26a3e81640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f268800b9c0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f2694001e50 tx=0x7f2688005830 comp rx=0 tx=0).stop 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.796+0000 7f26a3e81640 1 -- 172.21.15.77:0/2304243175 shutdown_connections 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.796+0000 7f26a3e81640 1 --2- 172.21.15.77:0/2304243175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f268800b9c0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.796+0000 7f26a3e81640 1 -- 172.21.15.77:0/2304243175 >> 172.21.15.77:0/2304243175 conn(0x7f269c0cf8a0 msgr2=0x7f269c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.796+0000 7f26a3e81640 1 -- 172.21.15.77:0/2304243175 shutdown_connections 2024-03-06T07:32:12.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.796+0000 7f26a3e81640 1 -- 172.21.15.77:0/2304243175 wait complete. 2024-03-06T07:32:12.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.797+0000 7f26a3e81640 1 Processor -- start 2024-03-06T07:32:12.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.797+0000 7f26a3e81640 1 -- start start 2024-03-06T07:32:12.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a3e81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f269c13b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a3e81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f269c13f2c0 0x7f269c1440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a3e81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f269c1f3d40 0x7f269c071820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a3e81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c15e850 con 0x7f26880051c0 2024-03-06T07:32:12.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a3e81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c15b210 con 0x7f269c13f2c0 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a3e81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c15e9d0 con 0x7f269c1f3d40 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f269c13b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f269c13b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43050/0 (socket says 172.21.15.77:43050) 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 -- 172.21.15.77:0/2947086467 learned_addr learned my addr 172.21.15.77:0/2947086467 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a141c640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f269c13f2c0 0x7f269c1440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a241e640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f269c1f3d40 0x7f269c071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 -- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f269c1f3d40 msgr2=0x7f269c071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f269c1f3d40 0x7f269c071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 -- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f269c13f2c0 msgr2=0x7f269c1440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f269c13f2c0 0x7f269c1440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 -- 172.21.15.77:0/2947086467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c13f780 con 0x7f26880051c0 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a141c640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f269c13f2c0 0x7f269c1440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:12.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a1c1d640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f269c13b8b0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f268c011910 tx=0x7f268c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26927fc640 1 -- 172.21.15.77:0/2947086467 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c00faf0 con 0x7f26880051c0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26927fc640 1 -- 172.21.15.77:0/2947086467 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c00fc90 con 0x7f26880051c0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26927fc640 1 -- 172.21.15.77:0/2947086467 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c014440 con 0x7f26880051c0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c13fa60 con 0x7f26880051c0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.798+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c140020 con 0x7f26880051c0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.799+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c04f330 con 0x7f26880051c0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.800+0000 7f26927fc640 1 -- 172.21.15.77:0/2947086467 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f268c012040 con 0x7f26880051c0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.800+0000 7f26927fc640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f268006b500 0x7f268006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.800+0000 7f26927fc640 1 -- 172.21.15.77:0/2947086467 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f268c08a890 con 0x7f26880051c0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.800+0000 7f26a141c640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f268006b500 0x7f268006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.801+0000 7f26a141c640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f268006b500 0x7f268006d9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f269c072410 tx=0x7f2698009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:12.805+0000 7f26927fc640 1 -- 172.21.15.77:0/2947086467 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f268c019030 con 0x7f26880051c0 2024-03-06T07:32:13.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.019+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f269c13bdf0 con 0x7f26880051c0 2024-03-06T07:32:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.019+0000 7f26927fc640 1 -- 172.21.15.77:0/2947086467 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f268c0596f0 con 0x7f26880051c0 2024-03-06T07:32:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:13.021 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f268006b500 msgr2=0x7f268006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f268006b500 0x7f268006d9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f269c072410 tx=0x7f2698009040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 msgr2=0x7f269c13b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f269c13b8b0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f268c011910 tx=0x7f268c011dd0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 shutdown_connections 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f269c1f3d40 0x7f269c071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f269c13f2c0 0x7f269c1440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f268006b500 0x7f268006d9c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 --2- 172.21.15.77:0/2947086467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26880051c0 0x7f269c13b8b0 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 >> 172.21.15.77:0/2947086467 conn(0x7f269c0cf8a0 msgr2=0x7f269c159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 shutdown_connections 2024-03-06T07:32:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:13.021+0000 7f26a3e81640 1 -- 172.21.15.77:0/2947086467 wait complete. 2024-03-06T07:32:13.235 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:13.236 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:14.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:14.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.337+0000 7f4a7f4ed640 1 Processor -- start 2024-03-06T07:32:14.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.337+0000 7f4a7f4ed640 1 -- start start 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7f4ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7815dfe0 0x7f4a780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7f4ed640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a780d29c0 0x7f4a780b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7f4ed640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a780b3430 con 0x7f4a780cf830 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7f4ed640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a780b31e0 con 0x7f4a780d29c0 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7f4ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780b3d60 con 0x7f4a7815dfe0 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7da8a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a780d29c0 0x7f4a780b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7da8a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a780d29c0 0x7f4a780b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43052/0 (socket says 172.21.15.77:43052) 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7da8a640 1 -- 172.21.15.77:0/716138945 learned_addr learned my addr 172.21.15.77:0/716138945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7d289640 1 --2- 172.21.15.77:0/716138945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7815dfe0 0x7f4a780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.338+0000 7f4a7da8a640 1 -- 172.21.15.77:0/716138945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7815dfe0 msgr2=0x7f4a780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.339+0000 7f4a7da8a640 1 --2- 172.21.15.77:0/716138945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7815dfe0 0x7f4a780cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.339+0000 7f4a7da8a640 1 -- 172.21.15.77:0/716138945 >> v1:172.21.15.158:6789/0 conn(0x7f4a780cf830 legacy=0x7f4a780d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.339+0000 7f4a7da8a640 1 -- 172.21.15.77:0/716138945 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780b3710 con 0x7f4a780d29c0 2024-03-06T07:32:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.339+0000 7f4a7d289640 1 --2- 172.21.15.77:0/716138945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7815dfe0 0x7f4a780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.339+0000 7f4a7da8a640 1 --2- 172.21.15.77:0/716138945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a780d29c0 0x7f4a780b41f0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7016e8741f950dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a77fff640 1 -- 172.21.15.77:0/716138945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68034070 con 0x7f4a780d29c0 2024-03-06T07:32:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a77fff640 1 -- 172.21.15.77:0/716138945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68030840 con 0x7f4a780d29c0 2024-03-06T07:32:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a77fff640 1 -- 172.21.15.77:0/716138945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68030c10 con 0x7f4a780d29c0 2024-03-06T07:32:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/716138945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a780d29c0 msgr2=0x7f4a780b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/716138945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a780d29c0 0x7f4a780b41f0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a6802b980 comp rx=0 tx=0).stop 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/716138945 shutdown_connections 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/716138945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a780d29c0 0x7f4a780b41f0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/716138945 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a7815dfe0 0x7f4a780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/716138945 >> 172.21.15.77:0/716138945 conn(0x7f4a780d3cd0 msgr2=0x7f4a780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/716138945 shutdown_connections 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.340+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/716138945 wait complete. 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.341+0000 7f4a7f4ed640 1 Processor -- start 2024-03-06T07:32:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7f4ed640 1 -- start start 2024-03-06T07:32:14.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7f4ed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a780cf830 0x7f4a7815cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7f4ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a780d29c0 0x7f4a78150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7f4ed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7815dfe0 0x7f4a780b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7f4ed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780d1df0 con 0x7f4a7815dfe0 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7f4ed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780d1c70 con 0x7f4a780cf830 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7f4ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7815e3c0 con 0x7f4a780d29c0 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7da8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7815dfe0 0x7f4a780b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7da8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7815dfe0 0x7f4a780b7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43060/0 (socket says 172.21.15.77:43060) 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7da8a640 1 -- 172.21.15.77:0/2876354285 learned_addr learned my addr 172.21.15.77:0/2876354285 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7d289640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a780cf830 0x7f4a7815cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.342+0000 7f4a7ca88640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a780d29c0 0x7f4a78150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a7da8a640 1 -- 172.21.15.77:0/2876354285 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a780d29c0 msgr2=0x7f4a78150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a7da8a640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a780d29c0 0x7f4a78150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a7da8a640 1 -- 172.21.15.77:0/2876354285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a780cf830 msgr2=0x7f4a7815cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a7da8a640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a780cf830 0x7f4a7815cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a7da8a640 1 -- 172.21.15.77:0/2876354285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780b68d0 con 0x7f4a7815dfe0 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a7da8a640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7815dfe0 0x7f4a780b7c30 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a75ffb640 1 -- 172.21.15.77:0/2876354285 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68048070 con 0x7f4a7815dfe0 2024-03-06T07:32:14.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a75ffb640 1 -- 172.21.15.77:0/2876354285 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68030640 con 0x7f4a7815dfe0 2024-03-06T07:32:14.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a780b6b90 con 0x7f4a7815dfe0 2024-03-06T07:32:14.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a75ffb640 1 -- 172.21.15.77:0/2876354285 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68034040 con 0x7f4a7815dfe0 2024-03-06T07:32:14.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.343+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a780b7150 con 0x7f4a7815dfe0 2024-03-06T07:32:14.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.344+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7804f330 con 0x7f4a7815dfe0 2024-03-06T07:32:14.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.345+0000 7f4a75ffb640 1 -- 172.21.15.77:0/2876354285 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a6804b070 con 0x7f4a7815dfe0 2024-03-06T07:32:14.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.345+0000 7f4a75ffb640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.345+0000 7f4a7d289640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.345+0000 7f4a75ffb640 1 -- 172.21.15.77:0/2876354285 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4a680ac1d0 con 0x7f4a7815dfe0 2024-03-06T07:32:14.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.346+0000 7f4a7d289640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a6406b4d0 0x7f4a6406d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000aac0 tx=0x7f4a70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.350+0000 7f4a75ffb640 1 -- 172.21.15.77:0/2876354285 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a6807ab90 con 0x7f4a7815dfe0 2024-03-06T07:32:14.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.561+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a7806c3c0 con 0x7f4a7815dfe0 2024-03-06T07:32:14.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.562+0000 7f4a75ffb640 1 -- 172.21.15.77:0/2876354285 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f4a6807ec30 con 0x7f4a7815dfe0 2024-03-06T07:32:14.564 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:14.564 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:14.564 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a6406b4d0 msgr2=0x7f4a6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a6406b4d0 0x7f4a6406d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000aac0 tx=0x7f4a70008040 comp rx=0 tx=0).stop 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7815dfe0 msgr2=0x7f4a780b7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7815dfe0 0x7f4a780b7c30 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a68004730 comp rx=0 tx=0).stop 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 shutdown_connections 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a7815dfe0 0x7f4a780b7c30 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4a780d29c0 0x7f4a78150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 --2- 172.21.15.77:0/2876354285 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4a780cf830 0x7f4a7815cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 >> 172.21.15.77:0/2876354285 conn(0x7f4a780d3cd0 msgr2=0x7f4a780b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 shutdown_connections 2024-03-06T07:32:14.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:14.564+0000 7f4a7f4ed640 1 -- 172.21.15.77:0/2876354285 wait complete. 2024-03-06T07:32:14.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:14.778 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:15.778 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:15.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85d177640 1 Processor -- start 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85d177640 1 -- start start 2024-03-06T07:32:15.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85d177640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd8580d22b0 0x7fd8580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85d177640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd8580b41f0 0x7fd8580b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85d177640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8580b3430 con 0x7fd85815dfe0 2024-03-06T07:32:15.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85d177640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8580d2c50 con 0x7fd8580b41f0 2024-03-06T07:32:15.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85d177640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8580b31e0 con 0x7fd8580d22b0 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd8580b41f0 0x7fd8580b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd8580b41f0 0x7fd8580b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43072/0 (socket says 172.21.15.77:43072) 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.877+0000 7fd85759e640 1 -- 172.21.15.77:0/253445886 learned_addr learned my addr 172.21.15.77:0/253445886 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.878+0000 7fd84e59c640 1 --2- 172.21.15.77:0/253445886 >> v2:172.21.15.158:3300/0 conn(0x7fd8580d22b0 0x7fd8580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.878+0000 7fd85759e640 1 -- 172.21.15.77:0/253445886 >> v2:172.21.15.158:3300/0 conn(0x7fd8580d22b0 msgr2=0x7fd8580d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.878+0000 7fd85759e640 1 --2- 172.21.15.77:0/253445886 >> v2:172.21.15.158:3300/0 conn(0x7fd8580d22b0 0x7fd8580d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.878+0000 7fd85759e640 1 -- 172.21.15.77:0/253445886 >> v1:172.21.15.158:6789/0 conn(0x7fd85815dfe0 legacy=0x7fd8580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.878+0000 7fd85759e640 1 -- 172.21.15.77:0/253445886 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8580b3710 con 0x7fd8580b41f0 2024-03-06T07:32:15.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.879+0000 7fd85759e640 1 --2- 172.21.15.77:0/253445886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8580b41f0 0x7fd8580b4680 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fd850009e50 tx=0x7fd85002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbadf29c037166eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.879+0000 7fd85659c640 1 -- 172.21.15.77:0/253445886 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85002ca80 con 0x7fd8580b41f0 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.879+0000 7fd85659c640 1 -- 172.21.15.77:0/253445886 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85002cc20 con 0x7fd8580b41f0 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.879+0000 7fd85659c640 1 -- 172.21.15.77:0/253445886 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8500343f0 con 0x7fd8580b41f0 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.880+0000 7fd85d177640 1 -- 172.21.15.77:0/253445886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8580b41f0 msgr2=0x7fd8580b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.880+0000 7fd85d177640 1 --2- 172.21.15.77:0/253445886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8580b41f0 0x7fd8580b4680 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fd850009e50 tx=0x7fd85002bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.880+0000 7fd85d177640 1 -- 172.21.15.77:0/253445886 shutdown_connections 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.880+0000 7fd85d177640 1 --2- 172.21.15.77:0/253445886 >> v2:172.21.15.158:3300/0 conn(0x7fd8580d22b0 0x7fd8580d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.880+0000 7fd85d177640 1 --2- 172.21.15.77:0/253445886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8580b41f0 0x7fd8580b4680 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.880+0000 7fd85d177640 1 -- 172.21.15.77:0/253445886 >> 172.21.15.77:0/253445886 conn(0x7fd8580d3cd0 msgr2=0x7fd8580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.880+0000 7fd85d177640 1 -- 172.21.15.77:0/253445886 shutdown_connections 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.880+0000 7fd85d177640 1 -- 172.21.15.77:0/253445886 wait complete. 2024-03-06T07:32:15.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.881+0000 7fd85d177640 1 Processor -- start 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.881+0000 7fd85d177640 1 -- start start 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85d177640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8580b41f0 0x7fd8580b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85d177640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8580d22b0 0x7fd8580b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85d177640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd85815dfe0 0x7fd8580b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85d177640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8580cfa00 con 0x7fd85815dfe0 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85d177640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8580b3d60 con 0x7fd8580d22b0 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85d177640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8580d1c70 con 0x7fd8580b41f0 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd85815dfe0 0x7fd8580b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85759e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd85815dfe0 0x7fd8580b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43086/0 (socket says 172.21.15.77:43086) 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85759e640 1 -- 172.21.15.77:0/3424013087 learned_addr learned my addr 172.21.15.77:0/3424013087 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd856d9d640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8580b41f0 0x7fd8580b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd84e59c640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8580d22b0 0x7fd8580b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85759e640 1 -- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8580b41f0 msgr2=0x7fd8580b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85759e640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8580b41f0 0x7fd8580b8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85759e640 1 -- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8580d22b0 msgr2=0x7fd8580b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85759e640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8580d22b0 0x7fd8580b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd85759e640 1 -- 172.21.15.77:0/3424013087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8580b9be0 con 0x7fd85815dfe0 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.882+0000 7fd84e59c640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8580d22b0 0x7fd8580b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.883+0000 7fd85759e640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd85815dfe0 0x7fd8580b6de0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fd850009f70 tx=0x7fd850004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.883+0000 7fd84ffff640 1 -- 172.21.15.77:0/3424013087 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850043020 con 0x7fd85815dfe0 2024-03-06T07:32:15.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.883+0000 7fd84ffff640 1 -- 172.21.15.77:0/3424013087 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85003e450 con 0x7fd85815dfe0 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.883+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8580b9060 con 0x7fd85815dfe0 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.883+0000 7fd84ffff640 1 -- 172.21.15.77:0/3424013087 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85003e840 con 0x7fd85815dfe0 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.883+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8580b9620 con 0x7fd85815dfe0 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.884+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85804f330 con 0x7fd85815dfe0 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.884+0000 7fd84ffff640 1 -- 172.21.15.77:0/3424013087 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd850046020 con 0x7fd85815dfe0 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.885+0000 7fd84ffff640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd83806b500 0x7fd83806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.885+0000 7fd84ffff640 1 -- 172.21.15.77:0/3424013087 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd8500acc00 con 0x7fd85815dfe0 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.885+0000 7fd856d9d640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd83806b500 0x7fd83806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.885+0000 7fd856d9d640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd83806b500 0x7fd83806d9c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd84800a750 tx=0x7fd848008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:15.889+0000 7fd84ffff640 1 -- 172.21.15.77:0/3424013087 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8500b3080 con 0x7fd85815dfe0 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.104+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd85815d3e0 con 0x7fd85815dfe0 2024-03-06T07:32:16.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.104+0000 7fd84ffff640 1 -- 172.21.15.77:0/3424013087 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd85004d030 con 0x7fd85815dfe0 2024-03-06T07:32:16.107 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:16.107 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:16.107 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.106+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd83806b500 msgr2=0x7fd83806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.106+0000 7fd85d177640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd83806b500 0x7fd83806d9c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd84800a750 tx=0x7fd848008040 comp rx=0 tx=0).stop 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.106+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd85815dfe0 msgr2=0x7fd8580b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.106+0000 7fd85d177640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd85815dfe0 0x7fd8580b6de0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fd850009f70 tx=0x7fd850004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.107+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 shutdown_connections 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.107+0000 7fd85d177640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd83806b500 0x7fd83806d9c0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.107+0000 7fd85d177640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd85815dfe0 0x7fd8580b6de0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.107+0000 7fd85d177640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd8580d22b0 0x7fd8580b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.107+0000 7fd85d177640 1 --2- 172.21.15.77:0/3424013087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd8580b41f0 0x7fd8580b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.107+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 >> 172.21.15.77:0/3424013087 conn(0x7fd8580d3cd0 msgr2=0x7fd8580d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.107+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 shutdown_connections 2024-03-06T07:32:16.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:16.107+0000 7fd85d177640 1 -- 172.21.15.77:0/3424013087 wait complete. 2024-03-06T07:32:16.321 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:16.321 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:17.322 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.422+0000 7ffa3fe2a640 1 Processor -- start 2024-03-06T07:32:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.423+0000 7ffa3fe2a640 1 -- start start 2024-03-06T07:32:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.423+0000 7ffa3fe2a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa3815dfe0 0x7ffa380cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.423+0000 7ffa3fe2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa380cf830 0x7ffa380d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.423+0000 7ffa3fe2a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa380b3430 con 0x7ffa380d27f0 2024-03-06T07:32:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.423+0000 7ffa3fe2a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa380d2c50 con 0x7ffa3815dfe0 2024-03-06T07:32:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.423+0000 7ffa3fe2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa380b31e0 con 0x7ffa380cf830 2024-03-06T07:32:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.423+0000 7ffa3dbc6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa3815dfe0 0x7ffa380cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.424+0000 7ffa3dbc6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa3815dfe0 0x7ffa380cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43100/0 (socket says 172.21.15.77:43100) 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.424+0000 7ffa3dbc6640 1 -- 172.21.15.77:0/4144327255 learned_addr learned my addr 172.21.15.77:0/4144327255 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.424+0000 7ffa3d3c5640 1 --2- 172.21.15.77:0/4144327255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa380cf830 0x7ffa380d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.424+0000 7ffa3dbc6640 1 -- 172.21.15.77:0/4144327255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa380cf830 msgr2=0x7ffa380d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.424+0000 7ffa3dbc6640 1 --2- 172.21.15.77:0/4144327255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa380cf830 0x7ffa380d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.424+0000 7ffa3dbc6640 1 -- 172.21.15.77:0/4144327255 >> v1:172.21.15.158:6789/0 conn(0x7ffa380d27f0 legacy=0x7ffa380d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.424+0000 7ffa3dbc6640 1 -- 172.21.15.77:0/4144327255 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa380b3710 con 0x7ffa3815dfe0 2024-03-06T07:32:17.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3dbc6640 1 --2- 172.21.15.77:0/4144327255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3815dfe0 0x7ffa380cf2f0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009e50 tx=0x7ffa2802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df3caf1e954ed84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3cbc4640 1 -- 172.21.15.77:0/4144327255 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2802cb00 con 0x7ffa3815dfe0 2024-03-06T07:32:17.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3cbc4640 1 -- 172.21.15.77:0/4144327255 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2802cca0 con 0x7ffa3815dfe0 2024-03-06T07:32:17.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3cbc4640 1 -- 172.21.15.77:0/4144327255 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28034610 con 0x7ffa3815dfe0 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/4144327255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3815dfe0 msgr2=0x7ffa380cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/4144327255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3815dfe0 0x7ffa380cf2f0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009e50 tx=0x7ffa2802bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/4144327255 shutdown_connections 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/4144327255 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa380cf830 0x7ffa380d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/4144327255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3815dfe0 0x7ffa380cf2f0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/4144327255 >> 172.21.15.77:0/4144327255 conn(0x7ffa380d3cd0 msgr2=0x7ffa380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/4144327255 shutdown_connections 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.425+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/4144327255 wait complete. 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.426+0000 7ffa3fe2a640 1 Processor -- start 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3fe2a640 1 -- start start 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3fe2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa380cf830 0x7ffa3814e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3fe2a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa380d27f0 0x7ffa380cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3fe2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa3815dfe0 0x7ffa380cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3fe2a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa380d1c70 con 0x7ffa380cf830 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3fe2a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa380b3d60 con 0x7ffa380d27f0 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3fe2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa380d1df0 con 0x7ffa3815dfe0 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3dbc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa380cf830 0x7ffa3814e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3dbc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa380cf830 0x7ffa3814e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43116/0 (socket says 172.21.15.77:43116) 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3dbc6640 1 -- 172.21.15.77:0/1088120039 learned_addr learned my addr 172.21.15.77:0/1088120039 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3d3c5640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa380d27f0 0x7ffa380cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.427+0000 7ffa3e3c7640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa3815dfe0 0x7ffa380cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3dbc6640 1 -- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa3815dfe0 msgr2=0x7ffa380cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3dbc6640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa3815dfe0 0x7ffa380cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3dbc6640 1 -- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa380d27f0 msgr2=0x7ffa380cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3dbc6640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa380d27f0 0x7ffa380cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3dbc6640 1 -- 172.21.15.77:0/1088120039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3814e6c0 con 0x7ffa380cf830 2024-03-06T07:32:17.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3d3c5640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa380d27f0 0x7ffa380cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:17.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3dbc6640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa380cf830 0x7ffa3814e180 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7ffa28000f80 tx=0x7ffa28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa367fc640 1 -- 172.21.15.77:0/1088120039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28043020 con 0x7ffa380cf830 2024-03-06T07:32:17.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa367fc640 1 -- 172.21.15.77:0/1088120039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2802ce50 con 0x7ffa380cf830 2024-03-06T07:32:17.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa380b7900 con 0x7ffa380cf830 2024-03-06T07:32:17.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa367fc640 1 -- 172.21.15.77:0/1088120039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28009510 con 0x7ffa380cf830 2024-03-06T07:32:17.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.428+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa380b7ec0 con 0x7ffa380cf830 2024-03-06T07:32:17.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.429+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa380ce3c0 con 0x7ffa380cf830 2024-03-06T07:32:17.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.429+0000 7ffa367fc640 1 -- 172.21.15.77:0/1088120039 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa28046020 con 0x7ffa380cf830 2024-03-06T07:32:17.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.430+0000 7ffa367fc640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa2406b4d0 0x7ffa2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.430+0000 7ffa367fc640 1 -- 172.21.15.77:0/1088120039 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ffa280ac9f0 con 0x7ffa380cf830 2024-03-06T07:32:17.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.430+0000 7ffa3d3c5640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa2406b4d0 0x7ffa2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.430+0000 7ffa3d3c5640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa2406b4d0 0x7ffa2406d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ffa380ca430 tx=0x7ffa2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.434+0000 7ffa367fc640 1 -- 172.21.15.77:0/1088120039 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa380ce3c0 con 0x7ffa380cf830 2024-03-06T07:32:17.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.728+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa380cf5f0 con 0x7ffa380cf830 2024-03-06T07:32:17.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.729+0000 7ffa367fc640 1 -- 172.21.15.77:0/1088120039 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ffa380cf5f0 con 0x7ffa380cf830 2024-03-06T07:32:17.731 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:17.731 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:17.731 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa2406b4d0 msgr2=0x7ffa2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa2406b4d0 0x7ffa2406d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ffa380ca430 tx=0x7ffa2c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa380cf830 msgr2=0x7ffa3814e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa380cf830 0x7ffa3814e180 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7ffa28000f80 tx=0x7ffa28004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 shutdown_connections 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa3815dfe0 0x7ffa380cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa380d27f0 0x7ffa380cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa2406b4d0 0x7ffa2406d990 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 --2- 172.21.15.77:0/1088120039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa380cf830 0x7ffa3814e180 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.730+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 >> 172.21.15.77:0/1088120039 conn(0x7ffa380d3cd0 msgr2=0x7ffa3806ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.731+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 shutdown_connections 2024-03-06T07:32:17.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:17.731+0000 7ffa3fe2a640 1 -- 172.21.15.77:0/1088120039 wait complete. 2024-03-06T07:32:17.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:17.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:18.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:19.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.044+0000 7f24db42a640 1 Processor -- start 2024-03-06T07:32:19.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.045+0000 7f24db42a640 1 -- start start 2024-03-06T07:32:19.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.045+0000 7f24db42a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24d415dfe0 0x7f24d40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.045+0000 7f24db42a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d40b3430 con 0x7f24d40d29c0 2024-03-06T07:32:19.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.045+0000 7f24db42a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d40b3710 con 0x7f24d40cf830 2024-03-06T07:32:19.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.045+0000 7f24db42a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d40b31e0 con 0x7f24d415dfe0 2024-03-06T07:32:19.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.046+0000 7f24d91c6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24d415dfe0 0x7f24d40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.046+0000 7f24d99c7640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f24d40d29c0 0x7f24d40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44686/0 (socket says 172.21.15.77:44686) 2024-03-06T07:32:19.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.046+0000 7f24d99c7640 1 -- 172.21.15.77:0/2622841627 learned_addr learned my addr 172.21.15.77:0/2622841627 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:19.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.046+0000 7f24d91c6640 1 -- 172.21.15.77:0/2622841627 >> v1:172.21.15.158:6789/0 conn(0x7f24d40cf830 legacy=0x7f24d40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.046+0000 7f24d91c6640 1 -- 172.21.15.77:0/2622841627 >> v1:172.21.15.77:6789/0 conn(0x7f24d40d29c0 legacy=0x7f24d40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.046+0000 7f24cbfff640 1 -- 172.21.15.77:0/2622841627 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149112831 0 0) 0x7f24d40b3430 con 0x7f24d40d29c0 2024-03-06T07:32:19.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.046+0000 7f24d91c6640 1 -- 172.21.15.77:0/2622841627 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d40b39f0 con 0x7f24d415dfe0 2024-03-06T07:32:19.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.047+0000 7f24d91c6640 1 --2- 172.21.15.77:0/2622841627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d40cf2f0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f24d40b3360 tx=0x7f24d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54661992a4cec7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.047+0000 7f24cbfff640 1 -- 172.21.15.77:0/2622841627 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0035020 con 0x7f24d415dfe0 2024-03-06T07:32:19.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.047+0000 7f24cbfff640 1 -- 172.21.15.77:0/2622841627 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d0031830 con 0x7f24d415dfe0 2024-03-06T07:32:19.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.047+0000 7f24cbfff640 1 -- 172.21.15.77:0/2622841627 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0031c00 con 0x7f24d415dfe0 2024-03-06T07:32:19.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.048+0000 7f24db42a640 1 -- 172.21.15.77:0/2622841627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 msgr2=0x7f24d40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.048+0000 7f24db42a640 1 --2- 172.21.15.77:0/2622841627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d40cf2f0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f24d40b3360 tx=0x7f24d002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.048+0000 7f24db42a640 1 -- 172.21.15.77:0/2622841627 shutdown_connections 2024-03-06T07:32:19.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.048+0000 7f24db42a640 1 --2- 172.21.15.77:0/2622841627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d40cf2f0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.048+0000 7f24db42a640 1 -- 172.21.15.77:0/2622841627 >> 172.21.15.77:0/2622841627 conn(0x7f24d40d3cd0 msgr2=0x7f24d40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.048+0000 7f24db42a640 1 -- 172.21.15.77:0/2622841627 shutdown_connections 2024-03-06T07:32:19.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.048+0000 7f24db42a640 1 -- 172.21.15.77:0/2622841627 wait complete. 2024-03-06T07:32:19.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.050+0000 7f24db42a640 1 Processor -- start 2024-03-06T07:32:19.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.050+0000 7f24db42a640 1 -- start start 2024-03-06T07:32:19.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24db42a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24d40cf830 0x7f24d4154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24db42a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f24d40d29c0 0x7f24d41f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24db42a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d41f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24db42a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d415e3c0 con 0x7f24d415dfe0 2024-03-06T07:32:19.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24db42a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40b3d60 con 0x7f24d40cf830 2024-03-06T07:32:19.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24db42a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d415e540 con 0x7f24d40d29c0 2024-03-06T07:32:19.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d41f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d41f69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41330/0 (socket says 172.21.15.77:41330) 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 -- 172.21.15.77:0/2415209405 learned_addr learned my addr 172.21.15.77:0/2415209405 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d89c5640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f24d40d29c0 0x7f24d41f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d91c6640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24d40cf830 0x7f24d4154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 -- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f24d40d29c0 msgr2=0x7f24d41f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f24d40d29c0 0x7f24d41f4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 -- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24d40cf830 msgr2=0x7f24d4154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24d40cf830 0x7f24d4154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 -- 172.21.15.77:0/2415209405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d41f5950 con 0x7f24d415dfe0 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d91c6640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24d40cf830 0x7f24d4154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24d99c7640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d41f69f0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f24cc00c7b0 tx=0x7f24cc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.051+0000 7f24c9ffb640 1 -- 172.21.15.77:0/2415209405 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc018070 con 0x7f24d415dfe0 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.052+0000 7f24c9ffb640 1 -- 172.21.15.77:0/2415209405 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24cc00a640 con 0x7f24d415dfe0 2024-03-06T07:32:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.052+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d41f5c30 con 0x7f24d415dfe0 2024-03-06T07:32:19.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.052+0000 7f24c9ffb640 1 -- 172.21.15.77:0/2415209405 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc00aa70 con 0x7f24d415dfe0 2024-03-06T07:32:19.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.052+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d41f61f0 con 0x7f24d415dfe0 2024-03-06T07:32:19.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.053+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d404f330 con 0x7f24d415dfe0 2024-03-06T07:32:19.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.053+0000 7f24c9ffb640 1 -- 172.21.15.77:0/2415209405 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24cc01b070 con 0x7f24d415dfe0 2024-03-06T07:32:19.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.053+0000 7f24c9ffb640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24b806b500 0x7f24b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.053+0000 7f24c9ffb640 1 -- 172.21.15.77:0/2415209405 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f24cc0897e0 con 0x7f24d415dfe0 2024-03-06T07:32:19.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.053+0000 7f24d91c6640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24b806b500 0x7f24b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.054+0000 7f24d91c6640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24b806b500 0x7f24b806d9c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f24d0036000 tx=0x7f24d0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.062+0000 7f24c9ffb640 1 -- 172.21.15.77:0/2415209405 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24cc058640 con 0x7f24d415dfe0 2024-03-06T07:32:19.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.273+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24d40cf5b0 con 0x7f24d415dfe0 2024-03-06T07:32:19.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.273+0000 7f24c9ffb640 1 -- 172.21.15.77:0/2415209405 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f24cc057f40 con 0x7f24d415dfe0 2024-03-06T07:32:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:19.275 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:19.276 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.275+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24b806b500 msgr2=0x7f24b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.275+0000 7f24db42a640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24b806b500 0x7f24b806d9c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f24d0036000 tx=0x7f24d0008030 comp rx=0 tx=0).stop 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.275+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 msgr2=0x7f24d41f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.275+0000 7f24db42a640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d41f69f0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f24cc00c7b0 tx=0x7f24cc00cc70 comp rx=0 tx=0).stop 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.276+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 shutdown_connections 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.276+0000 7f24db42a640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f24b806b500 0x7f24b806d9c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.276+0000 7f24db42a640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24d415dfe0 0x7f24d41f69f0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.276+0000 7f24db42a640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f24d40d29c0 0x7f24d41f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.276+0000 7f24db42a640 1 --2- 172.21.15.77:0/2415209405 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24d40cf830 0x7f24d4154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.276+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 >> 172.21.15.77:0/2415209405 conn(0x7f24d40d3cd0 msgr2=0x7f24d40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.276+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 shutdown_connections 2024-03-06T07:32:19.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:19.276+0000 7f24db42a640 1 -- 172.21.15.77:0/2415209405 wait complete. 2024-03-06T07:32:19.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:19.492 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:20.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.595+0000 7fc679ce1640 1 Processor -- start 2024-03-06T07:32:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.596+0000 7fc679ce1640 1 -- start start 2024-03-06T07:32:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc679ce1640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc6740d22b0 0x7fc6740d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc679ce1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6740b41f0 0x7fc6740b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc679ce1640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6740b3430 con 0x7fc67415dfe0 2024-03-06T07:32:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc679ce1640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6740d2c50 con 0x7fc6740b41f0 2024-03-06T07:32:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc679ce1640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6740b31e0 con 0x7fc6740d22b0 2024-03-06T07:32:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc673fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6740b41f0 0x7fc6740b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc673fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc6740b41f0 0x7fc6740b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41342/0 (socket says 172.21.15.77:41342) 2024-03-06T07:32:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc673fff640 1 -- 172.21.15.77:0/361942201 learned_addr learned my addr 172.21.15.77:0/361942201 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc672ffd640 1 --2- 172.21.15.77:0/361942201 >> v2:172.21.15.158:3300/0 conn(0x7fc6740d22b0 0x7fc6740d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc673fff640 1 -- 172.21.15.77:0/361942201 >> v2:172.21.15.158:3300/0 conn(0x7fc6740d22b0 msgr2=0x7fc6740d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc673fff640 1 --2- 172.21.15.77:0/361942201 >> v2:172.21.15.158:3300/0 conn(0x7fc6740d22b0 0x7fc6740d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc673fff640 1 -- 172.21.15.77:0/361942201 >> v1:172.21.15.158:6789/0 conn(0x7fc67415dfe0 legacy=0x7fc6740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.597+0000 7fc673fff640 1 -- 172.21.15.77:0/361942201 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6740b3710 con 0x7fc6740b41f0 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.598+0000 7fc673fff640 1 --2- 172.21.15.77:0/361942201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6740b4680 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc66c009e50 tx=0x7fc66c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406e29fd7f562477 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc6727fc640 1 -- 172.21.15.77:0/361942201 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c02ca80 con 0x7fc6740b41f0 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc6727fc640 1 -- 172.21.15.77:0/361942201 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66c02cc20 con 0x7fc6740b41f0 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc6727fc640 1 -- 172.21.15.77:0/361942201 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c0343f0 con 0x7fc6740b41f0 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc679ce1640 1 -- 172.21.15.77:0/361942201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 msgr2=0x7fc6740b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc679ce1640 1 --2- 172.21.15.77:0/361942201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6740b4680 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc66c009e50 tx=0x7fc66c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc679ce1640 1 -- 172.21.15.77:0/361942201 shutdown_connections 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc679ce1640 1 --2- 172.21.15.77:0/361942201 >> v2:172.21.15.158:3300/0 conn(0x7fc6740d22b0 0x7fc6740d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc679ce1640 1 --2- 172.21.15.77:0/361942201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6740b4680 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc679ce1640 1 -- 172.21.15.77:0/361942201 >> 172.21.15.77:0/361942201 conn(0x7fc6740d3cd0 msgr2=0x7fc6740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc679ce1640 1 -- 172.21.15.77:0/361942201 shutdown_connections 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.599+0000 7fc679ce1640 1 -- 172.21.15.77:0/361942201 wait complete. 2024-03-06T07:32:20.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.601+0000 7fc679ce1640 1 Processor -- start 2024-03-06T07:32:20.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.601+0000 7fc679ce1640 1 -- start start 2024-03-06T07:32:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.602+0000 7fc679ce1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6741f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.602+0000 7fc679ce1640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6740d22b0 0x7fc67415bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.602+0000 7fc679ce1640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc67415dfe0 0x7fc6741600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.602+0000 7fc679ce1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740cfa00 con 0x7fc6740b41f0 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.602+0000 7fc679ce1640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740b3d60 con 0x7fc6740d22b0 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.602+0000 7fc679ce1640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740d1c70 con 0x7fc67415dfe0 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6741f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6741f3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41344/0 (socket says 172.21.15.77:41344) 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 -- 172.21.15.77:0/3965395683 learned_addr learned my addr 172.21.15.77:0/3965395683 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc673fff640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc67415dfe0 0x7fc6741600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc672ffd640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6740d22b0 0x7fc67415bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 -- 172.21.15.77:0/3965395683 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc67415dfe0 msgr2=0x7fc6741600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc67415dfe0 0x7fc6741600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 -- 172.21.15.77:0/3965395683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6740d22b0 msgr2=0x7fc67415bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6740d22b0 0x7fc67415bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 -- 172.21.15.77:0/3965395683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc674160600 con 0x7fc6740b41f0 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc6737fe640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6741f3b10 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fc66400b3e0 tx=0x7fc66400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc65bfff640 1 -- 172.21.15.77:0/3965395683 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664015020 con 0x7fc6740b41f0 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc65bfff640 1 -- 172.21.15.77:0/3965395683 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc664011850 con 0x7fc6740b41f0 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6741f4140 con 0x7fc6740b41f0 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc65bfff640 1 -- 172.21.15.77:0/3965395683 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664011c40 con 0x7fc6740b41f0 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.603+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6741f4700 con 0x7fc6740b41f0 2024-03-06T07:32:20.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.605+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67404f330 con 0x7fc6740b41f0 2024-03-06T07:32:20.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.605+0000 7fc65bfff640 1 -- 172.21.15.77:0/3965395683 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc664018020 con 0x7fc6740b41f0 2024-03-06T07:32:20.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.605+0000 7fc65bfff640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc66006b4d0 0x7fc66006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.605+0000 7fc672ffd640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc66006b4d0 0x7fc66006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.605+0000 7fc65bfff640 1 -- 172.21.15.77:0/3965395683 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc66408a1a0 con 0x7fc6740b41f0 2024-03-06T07:32:20.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.606+0000 7fc672ffd640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc66006b4d0 0x7fc66006d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc6740a13f0 tx=0x7fc668008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.609+0000 7fc65bfff640 1 -- 172.21.15.77:0/3965395683 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc66401f020 con 0x7fc6740b41f0 2024-03-06T07:32:20.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.824+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6740747a0 con 0x7fc6740b41f0 2024-03-06T07:32:20.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.825+0000 7fc65bfff640 1 -- 172.21.15.77:0/3965395683 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc6640172c0 con 0x7fc6740b41f0 2024-03-06T07:32:20.827 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:20.827 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:20.827 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:20.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc66006b4d0 msgr2=0x7fc66006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc66006b4d0 0x7fc66006d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc6740a13f0 tx=0x7fc668008040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 msgr2=0x7fc6741f3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6741f3b10 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fc66400b3e0 tx=0x7fc66400b8a0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 shutdown_connections 2024-03-06T07:32:20.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc67415dfe0 0x7fc6741600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc6740d22b0 0x7fc67415bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc66006b4d0 0x7fc66006d990 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 --2- 172.21.15.77:0/3965395683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6740b41f0 0x7fc6741f3b10 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 >> 172.21.15.77:0/3965395683 conn(0x7fc6740d3cd0 msgr2=0x7fc67406ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 shutdown_connections 2024-03-06T07:32:20.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:20.827+0000 7fc679ce1640 1 -- 172.21.15.77:0/3965395683 wait complete. 2024-03-06T07:32:21.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:21.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:22.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:22.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.148+0000 7f968b081640 1 Processor -- start 2024-03-06T07:32:22.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.148+0000 7f968b081640 1 -- start start 2024-03-06T07:32:22.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.148+0000 7f968b081640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f96840d22b0 0x7f96840d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f968b081640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96840b41f0 0x7f96840b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f968b081640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96840b3430 con 0x7f968415dfe0 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f968b081640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96840d2c50 con 0x7f96840d22b0 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f968b081640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96840b31e0 con 0x7f96840b41f0 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f9688e1d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f968415dfe0 0x7f96840cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44694/0 (socket says 172.21.15.77:44694) 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f9688e1d640 1 -- 172.21.15.77:0/549187058 learned_addr learned my addr 172.21.15.77:0/549187058 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f9683fff640 1 --2- 172.21.15.77:0/549187058 >> v2:172.21.15.158:3300/0 conn(0x7f96840d22b0 0x7f96840d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f968961e640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96840b41f0 0x7f96840b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799476853 0 0) 0x7f96840b3430 con 0x7f968415dfe0 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966c003520 con 0x7f968415dfe0 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2178917202 0 0) 0x7f966c003520 con 0x7f968415dfe0 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96840b41f0 msgr2=0x7f96840b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96840b41f0 0x7f96840b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 >> v2:172.21.15.158:3300/0 conn(0x7f96840d22b0 msgr2=0x7f96840d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 --2- 172.21.15.77:0/549187058 >> v2:172.21.15.158:3300/0 conn(0x7f96840d22b0 0x7f96840d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96840b3710 con 0x7f968415dfe0 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f9674001670 con 0x7f968415dfe0 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 >> v1:172.21.15.77:6789/0 conn(0x7f968415dfe0 legacy=0x7f96840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96840b41f0 0x7f966c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f966c003f30 0x7f966c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f966c004d40 0x7f966c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840b31e0 con 0x7f966c003f30 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c00b9c0 con 0x7f966c004d40 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c00bb40 con 0x7f96840b41f0 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.149+0000 7f9688e1d640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96840b41f0 0x7f966c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.150+0000 7f9683fff640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f966c003f30 0x7f966c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.150+0000 7f968961e640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f966c004d40 0x7f966c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.150+0000 7f9683fff640 1 -- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96840b41f0 msgr2=0x7f966c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.150+0000 7f9683fff640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f96840b41f0 0x7f966c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.150+0000 7f9683fff640 1 -- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f966c004d40 msgr2=0x7f966c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.150+0000 7f9683fff640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f966c004d40 0x7f966c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.150+0000 7f9683fff640 1 -- 172.21.15.77:0/549187058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c003520 con 0x7f966c003f30 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.150+0000 7f9683fff640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f966c003f30 0x7f966c004800 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f9678009ac0 tx=0x7f966c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3259c41e7ef7ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678039070 con 0x7f966c003f30 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967802c6d0 con 0x7f966c003f30 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f96837fe640 1 -- 172.21.15.77:0/549187058 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678009690 con 0x7f966c003f30 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f968b081640 1 -- 172.21.15.77:0/549187058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f966c003f30 msgr2=0x7f966c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f968b081640 1 --2- 172.21.15.77:0/549187058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f966c003f30 0x7f966c004800 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f9678009ac0 tx=0x7f966c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f968961e640 1 -- 172.21.15.77:0/549187058 reap_dead start 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f968b081640 1 -- 172.21.15.77:0/549187058 shutdown_connections 2024-03-06T07:32:22.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f968b081640 1 -- 172.21.15.77:0/549187058 >> 172.21.15.77:0/549187058 conn(0x7f96840d3cd0 msgr2=0x7f96840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f968b081640 1 -- 172.21.15.77:0/549187058 shutdown_connections 2024-03-06T07:32:22.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.151+0000 7f968b081640 1 -- 172.21.15.77:0/549187058 wait complete. 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.152+0000 7f968b081640 1 Processor -- start 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.152+0000 7f968b081640 1 -- start start 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968b081640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f966c003f30 0x7f96840cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968b081640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f968415b5b0 0x7f96841588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968b081640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96840cff60 0x7f96840d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968b081640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840cfa00 con 0x7f96840cff60 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968b081640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840b3d60 con 0x7f968415b5b0 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968b081640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96840d1c70 con 0x7f966c003f30 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968961e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96840cff60 0x7f96840d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968961e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96840cff60 0x7f96840d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41362/0 (socket says 172.21.15.77:41362) 2024-03-06T07:32:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f968961e640 1 -- 172.21.15.77:0/2945919085 learned_addr learned my addr 172.21.15.77:0/2945919085 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.153+0000 7f9688e1d640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f966c003f30 0x7f96840cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f9683fff640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f968415b5b0 0x7f96841588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f968961e640 1 -- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f966c003f30 msgr2=0x7f96840cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f968961e640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f966c003f30 0x7f96840cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f968961e640 1 -- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f968415b5b0 msgr2=0x7f96841588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f968961e640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f968415b5b0 0x7f96841588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f968961e640 1 -- 172.21.15.77:0/2945919085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96840cd490 con 0x7f96840cff60 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f9683fff640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f968415b5b0 0x7f96841588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f968961e640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96840cff60 0x7f96840d03e0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f967c011bf0 tx=0x7f967c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f96817fa640 1 -- 172.21.15.77:0/2945919085 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c00fe10 con 0x7f96840cff60 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f96817fa640 1 -- 172.21.15.77:0/2945919085 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c01d070 con 0x7f96840cff60 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9684158e10 con 0x7f96840cff60 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f96817fa640 1 -- 172.21.15.77:0/2945919085 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c015670 con 0x7f96840cff60 2024-03-06T07:32:22.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.154+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96840cda10 con 0x7f96840cff60 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.155+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96840d0920 con 0x7f96840cff60 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.155+0000 7f96817fa640 1 -- 172.21.15.77:0/2945919085 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f967c015810 con 0x7f96840cff60 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.156+0000 7f96817fa640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f967006b4d0 0x7f967006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.156+0000 7f96817fa640 1 -- 172.21.15.77:0/2945919085 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f967c08aa50 con 0x7f96840cff60 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.156+0000 7f9688e1d640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f967006b4d0 0x7f967006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.156+0000 7f9688e1d640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f967006b4d0 0x7f967006d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9674007fd0 tx=0x7f9674009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.160+0000 7f96817fa640 1 -- 172.21.15.77:0/2945919085 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f967c010040 con 0x7f96840cff60 2024-03-06T07:32:22.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.368+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f968404f330 con 0x7f96840cff60 2024-03-06T07:32:22.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.369+0000 7f96817fa640 1 -- 172.21.15.77:0/2945919085 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f967c0598b0 con 0x7f96840cff60 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:22.371 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f967006b4d0 msgr2=0x7f967006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f967006b4d0 0x7f967006d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9674007fd0 tx=0x7f9674009040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96840cff60 msgr2=0x7f96840d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96840cff60 0x7f96840d03e0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f967c011bf0 tx=0x7f967c00f580 comp rx=0 tx=0).stop 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 shutdown_connections 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f967006b4d0 0x7f967006d990 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96840cff60 0x7f96840d03e0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f968415b5b0 0x7f96841588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.370+0000 7f968b081640 1 --2- 172.21.15.77:0/2945919085 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f966c003f30 0x7f96840cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.371+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 >> 172.21.15.77:0/2945919085 conn(0x7f96840d3cd0 msgr2=0x7f968406bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.371+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 shutdown_connections 2024-03-06T07:32:22.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:22.371+0000 7f968b081640 1 -- 172.21.15.77:0/2945919085 wait complete. 2024-03-06T07:32:22.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:22.585 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:23.586 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:23.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.686+0000 7f498b02a640 1 Processor -- start 2024-03-06T07:32:23.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.686+0000 7f498b02a640 1 -- start start 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.687+0000 7f498b02a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f498415dfe0 0x7f49840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.687+0000 7f498b02a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.687+0000 7f498b02a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49840b3430 con 0x7f49840cf830 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.687+0000 7f498b02a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49840b31e0 con 0x7f498415dfe0 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.687+0000 7f498b02a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49840b3d60 con 0x7f49840d29c0 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f4988dc6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f498415dfe0 0x7f49840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f49895c7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f4988dc6640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f498415dfe0 0x7f49840cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:39566/0 (socket says 172.21.15.77:39566) 2024-03-06T07:32:23.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f4988dc6640 1 -- 172.21.15.77:0/815806663 learned_addr learned my addr 172.21.15.77:0/815806663 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f4983fff640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f49840cf830 0x7f49840d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:56680/0 (socket says 172.21.15.77:56680) 2024-03-06T07:32:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f49895c7640 1 -- 172.21.15.77:0/815806663 >> v2:172.21.15.158:3300/0 conn(0x7f498415dfe0 msgr2=0x7f49840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f49895c7640 1 --2- 172.21.15.77:0/815806663 >> v2:172.21.15.158:3300/0 conn(0x7f498415dfe0 0x7f49840cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f49895c7640 1 -- 172.21.15.77:0/815806663 >> v1:172.21.15.158:6789/0 conn(0x7f49840cf830 legacy=0x7f49840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f49895c7640 1 -- 172.21.15.77:0/815806663 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49840b3710 con 0x7f49840d29c0 2024-03-06T07:32:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f49837fe640 1 -- 172.21.15.77:0/815806663 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964750613 0 0) 0x7f49840b3430 con 0x7f49840cf830 2024-03-06T07:32:23.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.688+0000 7f4988dc6640 1 --2- 172.21.15.77:0/815806663 >> v2:172.21.15.158:3300/0 conn(0x7f498415dfe0 0x7f49840cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.689+0000 7f49895c7640 1 --2- 172.21.15.77:0/815806663 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b41f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f497c009d90 tx=0x7f497c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91f0d453ec46bd86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.689+0000 7f49837fe640 1 -- 172.21.15.77:0/815806663 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c039070 con 0x7f49840d29c0 2024-03-06T07:32:23.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.689+0000 7f49837fe640 1 -- 172.21.15.77:0/815806663 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c02cc50 con 0x7f49840d29c0 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.689+0000 7f49837fe640 1 -- 172.21.15.77:0/815806663 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c034440 con 0x7f49840d29c0 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.690+0000 7f498b02a640 1 -- 172.21.15.77:0/815806663 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 msgr2=0x7f49840b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.690+0000 7f498b02a640 1 --2- 172.21.15.77:0/815806663 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b41f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f497c009d90 tx=0x7f497c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.690+0000 7f498b02a640 1 -- 172.21.15.77:0/815806663 shutdown_connections 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.690+0000 7f498b02a640 1 --2- 172.21.15.77:0/815806663 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b41f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.690+0000 7f498b02a640 1 --2- 172.21.15.77:0/815806663 >> v2:172.21.15.158:3300/0 conn(0x7f498415dfe0 0x7f49840cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.690+0000 7f498b02a640 1 -- 172.21.15.77:0/815806663 >> 172.21.15.77:0/815806663 conn(0x7f49840d3cd0 msgr2=0x7f49840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.690+0000 7f498b02a640 1 -- 172.21.15.77:0/815806663 shutdown_connections 2024-03-06T07:32:23.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.690+0000 7f498b02a640 1 -- 172.21.15.77:0/815806663 wait complete. 2024-03-06T07:32:23.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.692+0000 7f498b02a640 1 Processor -- start 2024-03-06T07:32:23.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.692+0000 7f498b02a640 1 -- start start 2024-03-06T07:32:23.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f498b02a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49840cf830 0x7f4984160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f498b02a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f498b02a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f498415dfe0 0x7f49840b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f498b02a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49840d1df0 con 0x7f49840cf830 2024-03-06T07:32:23.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f498b02a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49840d1c70 con 0x7f498415dfe0 2024-03-06T07:32:23.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f498b02a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498415e3c0 con 0x7f49840d29c0 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f4988dc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49840cf830 0x7f4984160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f4988dc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49840cf830 0x7f4984160980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41364/0 (socket says 172.21.15.77:41364) 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.693+0000 7f4988dc6640 1 -- 172.21.15.77:0/1959250498 learned_addr learned my addr 172.21.15.77:0/1959250498 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f4983fff640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f49895c7640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f498415dfe0 0x7f49840b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f4988dc6640 1 -- 172.21.15.77:0/1959250498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 msgr2=0x7f49840b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f4988dc6640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f4988dc6640 1 -- 172.21.15.77:0/1959250498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f498415dfe0 msgr2=0x7f49840b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f4988dc6640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f498415dfe0 0x7f49840b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f4988dc6640 1 -- 172.21.15.77:0/1959250498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4984160ec0 con 0x7f49840cf830 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f4988dc6640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49840cf830 0x7f4984160980 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f497800e9b0 tx=0x7f497800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f49817fa640 1 -- 172.21.15.77:0/1959250498 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978017070 con 0x7f49840cf830 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f49817fa640 1 -- 172.21.15.77:0/1959250498 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497800c9a0 con 0x7f49840cf830 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49840caa90 con 0x7f49840cf830 2024-03-06T07:32:23.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f49817fa640 1 -- 172.21.15.77:0/1959250498 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497800cd50 con 0x7f49840cf830 2024-03-06T07:32:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.694+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49840b7590 con 0x7f49840cf830 2024-03-06T07:32:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.695+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49840d1a90 con 0x7f49840cf830 2024-03-06T07:32:23.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.696+0000 7f49817fa640 1 -- 172.21.15.77:0/1959250498 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f497801a070 con 0x7f49840cf830 2024-03-06T07:32:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.696+0000 7f49817fa640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f496806b4d0 0x7f496806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.696+0000 7f49817fa640 1 -- 172.21.15.77:0/1959250498 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f497808a110 con 0x7f49840cf830 2024-03-06T07:32:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.696+0000 7f4983fff640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f496806b4d0 0x7f496806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.696+0000 7f4983fff640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f496806b4d0 0x7f496806d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4970007fd0 tx=0x7f4970009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.700+0000 7f49817fa640 1 -- 172.21.15.77:0/1959250498 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49840d1a90 con 0x7f49840cf830 2024-03-06T07:32:23.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.917+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49841610f0 con 0x7f49840cf830 2024-03-06T07:32:23.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.917+0000 7f49817fa640 1 -- 172.21.15.77:0/1959250498 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f4978058f70 con 0x7f49840cf830 2024-03-06T07:32:23.919 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:23.919 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:23.919 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:23.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f496806b4d0 msgr2=0x7f496806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f496806b4d0 0x7f496806d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4970007fd0 tx=0x7f4970009040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49840cf830 msgr2=0x7f4984160980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49840cf830 0x7f4984160980 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f497800e9b0 tx=0x7f497800ee70 comp rx=0 tx=0).stop 2024-03-06T07:32:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 shutdown_connections 2024-03-06T07:32:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f498415dfe0 0x7f49840b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49840d29c0 0x7f49840b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f496806b4d0 0x7f496806d990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 --2- 172.21.15.77:0/1959250498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49840cf830 0x7f4984160980 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 >> 172.21.15.77:0/1959250498 conn(0x7f49840d3cd0 msgr2=0x7f498406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 shutdown_connections 2024-03-06T07:32:23.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:23.919+0000 7f498b02a640 1 -- 172.21.15.77:0/1959250498 wait complete. 2024-03-06T07:32:24.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:24.133 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:25.134 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:25.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.233+0000 7ff316415640 1 Processor -- start 2024-03-06T07:32:25.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.234+0000 7ff316415640 1 -- start start 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.234+0000 7ff316415640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff31015dfe0 0x7ff3100cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff316415640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3100d29c0 0x7ff3100b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff316415640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3100b3430 con 0x7ff3100cf830 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff316415640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3100b31e0 con 0x7ff3100d29c0 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff316415640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3100b3d60 con 0x7ff31015dfe0 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff31015dfe0 0x7ff3100cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff3149b2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3100d29c0 0x7ff3100b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff31015dfe0 0x7ff3100cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:53934/0 (socket says 172.21.15.77:53934) 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30f7fe640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7ff3100cf830 0x7ff3100d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:56694/0 (socket says 172.21.15.77:56694) 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff3149b2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7ff3100d29c0 0x7ff3100b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:39586/0 (socket says 172.21.15.77:39586) 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30ffff640 1 -- 172.21.15.77:0/2072206353 learned_addr learned my addr 172.21.15.77:0/2072206353 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30effd640 1 -- 172.21.15.77:0/2072206353 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755804291 0 0) 0x7ff3100b3430 con 0x7ff3100cf830 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30effd640 1 -- 172.21.15.77:0/2072206353 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f8003520 con 0x7ff3100cf830 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30ffff640 1 -- 172.21.15.77:0/2072206353 >> v2:172.21.15.158:3300/0 conn(0x7ff3100d29c0 msgr2=0x7ff3100b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30ffff640 1 --2- 172.21.15.77:0/2072206353 >> v2:172.21.15.158:3300/0 conn(0x7ff3100d29c0 0x7ff3100b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30ffff640 1 -- 172.21.15.77:0/2072206353 >> v1:172.21.15.158:6789/0 conn(0x7ff3100cf830 legacy=0x7ff3100d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff30ffff640 1 -- 172.21.15.77:0/2072206353 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3100b3710 con 0x7ff31015dfe0 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.235+0000 7ff3149b2640 1 --2- 172.21.15.77:0/2072206353 >> v2:172.21.15.158:3300/0 conn(0x7ff3100d29c0 0x7ff3100b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.236+0000 7ff30ffff640 1 --2- 172.21.15.77:0/2072206353 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff31015dfe0 0x7ff3100cf2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff300009870 tx=0x7ff30002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=325cfbbfa3b47ad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.236+0000 7ff30effd640 1 -- 172.21.15.77:0/2072206353 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300034070 con 0x7ff31015dfe0 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.236+0000 7ff30effd640 1 -- 172.21.15.77:0/2072206353 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300030840 con 0x7ff31015dfe0 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.236+0000 7ff30effd640 1 -- 172.21.15.77:0/2072206353 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300030bf0 con 0x7ff31015dfe0 2024-03-06T07:32:25.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.237+0000 7ff316415640 1 -- 172.21.15.77:0/2072206353 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff31015dfe0 msgr2=0x7ff3100cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.237+0000 7ff316415640 1 --2- 172.21.15.77:0/2072206353 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff31015dfe0 0x7ff3100cf2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff300009870 tx=0x7ff30002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.237+0000 7ff316415640 1 -- 172.21.15.77:0/2072206353 shutdown_connections 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.237+0000 7ff316415640 1 --2- 172.21.15.77:0/2072206353 >> v2:172.21.15.158:3300/0 conn(0x7ff3100d29c0 0x7ff3100b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.237+0000 7ff316415640 1 --2- 172.21.15.77:0/2072206353 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff31015dfe0 0x7ff3100cf2f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.237+0000 7ff316415640 1 -- 172.21.15.77:0/2072206353 >> 172.21.15.77:0/2072206353 conn(0x7ff3100d3cd0 msgr2=0x7ff3100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.237+0000 7ff316415640 1 -- 172.21.15.77:0/2072206353 shutdown_connections 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.237+0000 7ff316415640 1 -- 172.21.15.77:0/2072206353 wait complete. 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.238+0000 7ff316415640 1 Processor -- start 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.238+0000 7ff316415640 1 -- start start 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff316415640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3100cf830 0x7ff3100cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff316415640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff3100d29c0 0x7ff3100b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff316415640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff31015dfe0 0x7ff31015afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff316415640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3100d1df0 con 0x7ff3100cf830 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff316415640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3100d1c70 con 0x7ff31015dfe0 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff316415640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31015e3c0 con 0x7ff3100d29c0 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3100cf830 0x7ff3100cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3100cf830 0x7ff3100cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41380/0 (socket says 172.21.15.77:41380) 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 -- 172.21.15.77:0/3653946047 learned_addr learned my addr 172.21.15.77:0/3653946047 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff3149b2640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff31015dfe0 0x7ff31015afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30f7fe640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff3100d29c0 0x7ff3100b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 -- 172.21.15.77:0/3653946047 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff3100d29c0 msgr2=0x7ff3100b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff3100d29c0 0x7ff3100b1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 -- 172.21.15.77:0/3653946047 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff31015dfe0 msgr2=0x7ff31015afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff31015dfe0 0x7ff31015afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 -- 172.21.15.77:0/3653946047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3100b6a70 con 0x7ff3100cf830 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.239+0000 7ff30ffff640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3100cf830 0x7ff3100cb730 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff300000c00 tx=0x7ff3000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.240+0000 7ff30cff9640 1 -- 172.21.15.77:0/3653946047 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300034040 con 0x7ff3100cf830 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.240+0000 7ff30cff9640 1 -- 172.21.15.77:0/3653946047 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300039c90 con 0x7ff3100cf830 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.240+0000 7ff30cff9640 1 -- 172.21.15.77:0/3653946047 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3000435f0 con 0x7ff3100cf830 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.240+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3100cbc70 con 0x7ff3100cf830 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.240+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3100b6d10 con 0x7ff3100cf830 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.241+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31004f330 con 0x7ff3100cf830 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.241+0000 7ff30cff9640 1 -- 172.21.15.77:0/3653946047 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff30004a060 con 0x7ff3100cf830 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.241+0000 7ff30cff9640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2fc06b4d0 0x7ff2fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.241+0000 7ff30cff9640 1 -- 172.21.15.77:0/3653946047 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff3000abfc0 con 0x7ff3100cf830 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.242+0000 7ff30f7fe640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2fc06b4d0 0x7ff2fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.242+0000 7ff30f7fe640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2fc06b4d0 0x7ff2fc06d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff3100b4510 tx=0x7ff304007b10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.246+0000 7ff30cff9640 1 -- 172.21.15.77:0/3653946047 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff30003e070 con 0x7ff3100cf830 2024-03-06T07:32:25.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.462+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3100d2630 con 0x7ff3100cf830 2024-03-06T07:32:25.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.462+0000 7ff30cff9640 1 -- 172.21.15.77:0/3653946047 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff30007aeb0 con 0x7ff3100cf830 2024-03-06T07:32:25.464 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:25.464 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:25.465 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:25.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2fc06b4d0 msgr2=0x7ff2fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2fc06b4d0 0x7ff2fc06d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff3100b4510 tx=0x7ff304007b10 comp rx=0 tx=0).stop 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3100cf830 msgr2=0x7ff3100cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3100cf830 0x7ff3100cb730 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff300000c00 tx=0x7ff3000046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 shutdown_connections 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff31015dfe0 0x7ff31015afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff3100d29c0 0x7ff3100b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff2fc06b4d0 0x7ff2fc06d990 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 --2- 172.21.15.77:0/3653946047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3100cf830 0x7ff3100cb730 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 >> 172.21.15.77:0/3653946047 conn(0x7ff3100d3cd0 msgr2=0x7ff31006d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 shutdown_connections 2024-03-06T07:32:25.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:25.464+0000 7ff316415640 1 -- 172.21.15.77:0/3653946047 wait complete. 2024-03-06T07:32:25.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:25.678 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:26.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:26.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.780+0000 7f092fdbc640 1 Processor -- start 2024-03-06T07:32:26.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.780+0000 7f092fdbc640 1 -- start start 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092fdbc640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f092815bd10 0x7f092806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092fdbc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f092806bc00 0x7f092806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092fdbc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09280cc090 con 0x7f092806c5c0 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092fdbc640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09280ae170 con 0x7f092815bd10 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092fdbc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09280cc330 con 0x7f092806bc00 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092d357640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f092806c5c0 0x7f092806b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:44708/0 (socket says 172.21.15.77:44708) 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092d357640 1 -- 172.21.15.77:0/284714506 learned_addr learned my addr 172.21.15.77:0/284714506 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092db58640 1 --2- 172.21.15.77:0/284714506 >> v2:172.21.15.158:3300/0 conn(0x7f092815bd10 0x7f092806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092e359640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f092806bc00 0x7f092806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650547902 0 0) 0x7f09280cc090 con 0x7f092806c5c0 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0914003520 con 0x7f092806c5c0 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277000056 0 0) 0x7f0914003520 con 0x7f092806c5c0 2024-03-06T07:32:26.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f092806bc00 msgr2=0x7f092806d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f092806bc00 0x7f092806d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 >> v2:172.21.15.158:3300/0 conn(0x7f092815bd10 msgr2=0x7f092806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 --2- 172.21.15.77:0/284714506 >> v2:172.21.15.158:3300/0 conn(0x7f092815bd10 0x7f092806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09280cc610 con 0x7f092806c5c0 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f091c001670 con 0x7f092806c5c0 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092db58640 1 --2- 172.21.15.77:0/284714506 >> v2:172.21.15.158:3300/0 conn(0x7f092815bd10 0x7f092806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 >> v1:172.21.15.77:6789/0 conn(0x7f092806c5c0 legacy=0x7f092806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f092806bc00 0x7f09140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0914003f30 0x7f0914004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.782+0000 7f092cb56640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09280ae170 con 0x7f0914004d40 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09280cc330 con 0x7f0914003f30 2024-03-06T07:32:26.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091400b9c0 con 0x7f092806bc00 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092e359640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092db58640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f092806bc00 0x7f09140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092d357640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0914003f30 0x7f0914004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092e359640 1 -- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f092806bc00 msgr2=0x7f09140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092e359640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f092806bc00 0x7f09140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092e359640 1 -- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0914003f30 msgr2=0x7f0914004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092e359640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0914003f30 0x7f0914004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.783+0000 7f092e359640 1 -- 172.21.15.77:0/284714506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0914003520 con 0x7f0914004d40 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092e359640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09140051c0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f0920009f90 tx=0x7f0914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e74aeca4360b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920039070 con 0x7f0914004d40 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092002cbc0 con 0x7f0914004d40 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092cb56640 1 -- 172.21.15.77:0/284714506 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920009c10 con 0x7f0914004d40 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092fdbc640 1 -- 172.21.15.77:0/284714506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 msgr2=0x7f09140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092fdbc640 1 --2- 172.21.15.77:0/284714506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09140051c0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f0920009f90 tx=0x7f0914005830 comp rx=0 tx=0).stop 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092e359640 1 -- 172.21.15.77:0/284714506 reap_dead start 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092fdbc640 1 -- 172.21.15.77:0/284714506 shutdown_connections 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092fdbc640 1 -- 172.21.15.77:0/284714506 >> 172.21.15.77:0/284714506 conn(0x7f09280d00e0 msgr2=0x7f09280d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092fdbc640 1 -- 172.21.15.77:0/284714506 shutdown_connections 2024-03-06T07:32:26.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.784+0000 7f092fdbc640 1 -- 172.21.15.77:0/284714506 wait complete. 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.786+0000 7f092fdbc640 1 Processor -- start 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.786+0000 7f092fdbc640 1 -- start start 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092fdbc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09281584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092fdbc640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09280c3c10 0x7f09280afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092fdbc640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09280c4090 0x7f09280b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092fdbc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092815c270 con 0x7f0914004d40 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092fdbc640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092815c0f0 con 0x7f09280c3c10 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092fdbc640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092806d0d0 con 0x7f09280c4090 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09281584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09281584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41396/0 (socket says 172.21.15.77:41396) 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 -- 172.21.15.77:0/2709486807 learned_addr learned my addr 172.21.15.77:0/2709486807 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 -- 172.21.15.77:0/2709486807 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09280c4090 msgr2=0x7f09280b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09280c4090 0x7f09280b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092d357640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09280c3c10 0x7f09280afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 -- 172.21.15.77:0/2709486807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09280c3c10 msgr2=0x7f09280afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09280c3c10 0x7f09280afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 -- 172.21.15.77:0/2709486807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09280c4660 con 0x7f0914004d40 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092d357640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09280c3c10 0x7f09280afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092db58640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09281584b0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f0924011c50 tx=0x7f092400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f091a7fc640 1 -- 172.21.15.77:0/2709486807 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400fd20 con 0x7f0914004d40 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f091a7fc640 1 -- 172.21.15.77:0/2709486807 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924004020 con 0x7f0914004d40 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09280c4940 con 0x7f0914004d40 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f091a7fc640 1 -- 172.21.15.77:0/2709486807 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924004410 con 0x7f0914004d40 2024-03-06T07:32:26.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.787+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09280c4f00 con 0x7f0914004d40 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.788+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092804f330 con 0x7f0914004d40 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.789+0000 7f091a7fc640 1 -- 172.21.15.77:0/2709486807 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0924012040 con 0x7f0914004d40 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.789+0000 7f091a7fc640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f090c06b4d0 0x7f090c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.789+0000 7f092d357640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f090c06b4d0 0x7f090c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.789+0000 7f091a7fc640 1 -- 172.21.15.77:0/2709486807 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f092408a760 con 0x7f0914004d40 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.790+0000 7f092d357640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f090c06b4d0 0x7f090c06d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f09280c6d10 tx=0x7f091c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:26.794+0000 7f091a7fc640 1 -- 172.21.15.77:0/2709486807 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0924019030 con 0x7f0914004d40 2024-03-06T07:32:27.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.009+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09280cb670 con 0x7f0914004d40 2024-03-06T07:32:27.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.009+0000 7f091a7fc640 1 -- 172.21.15.77:0/2709486807 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f0924059650 con 0x7f0914004d40 2024-03-06T07:32:27.011 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:27.011 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:27.011 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:27.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f090c06b4d0 msgr2=0x7f090c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f090c06b4d0 0x7f090c06d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f09280c6d10 tx=0x7f091c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:27.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 msgr2=0x7f09281584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09281584b0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f0924011c50 tx=0x7f092400f610 comp rx=0 tx=0).stop 2024-03-06T07:32:27.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 shutdown_connections 2024-03-06T07:32:27.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09280c4090 0x7f09280b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09280c3c10 0x7f09280afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f090c06b4d0 0x7f090c06d990 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 --2- 172.21.15.77:0/2709486807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0914004d40 0x7f09281584b0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 >> 172.21.15.77:0/2709486807 conn(0x7f09280d00e0 msgr2=0x7f09280b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 shutdown_connections 2024-03-06T07:32:27.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:27.011+0000 7f092fdbc640 1 -- 172.21.15.77:0/2709486807 wait complete. 2024-03-06T07:32:27.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:27.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:28.227 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:28.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.363+0000 7fea65908640 1 Processor -- start 2024-03-06T07:32:28.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.363+0000 7fea65908640 1 -- start start 2024-03-06T07:32:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea65908640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fea6015dfe0 0x7fea600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea65908640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea600cf830 0x7fea600d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea65908640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea600b3430 con 0x7fea600d27f0 2024-03-06T07:32:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea65908640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea600b31e0 con 0x7fea600cf830 2024-03-06T07:32:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea65908640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea600b3d60 con 0x7fea6015dfe0 2024-03-06T07:32:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea5e7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea600cf830 0x7fea600d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea5e7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea600cf830 0x7fea600d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41402/0 (socket says 172.21.15.77:41402) 2024-03-06T07:32:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea5e7fc640 1 -- 172.21.15.77:0/3321477063 learned_addr learned my addr 172.21.15.77:0/3321477063 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea5e7fc640 1 -- 172.21.15.77:0/3321477063 >> v2:172.21.15.158:3300/0 conn(0x7fea6015dfe0 msgr2=0x7fea600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea5e7fc640 1 --2- 172.21.15.77:0/3321477063 >> v2:172.21.15.158:3300/0 conn(0x7fea6015dfe0 0x7fea600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea5e7fc640 1 -- 172.21.15.77:0/3321477063 >> v1:172.21.15.158:6789/0 conn(0x7fea600d27f0 legacy=0x7fea600d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.364+0000 7fea5e7fc640 1 -- 172.21.15.77:0/3321477063 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea600b3710 con 0x7fea600cf830 2024-03-06T07:32:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.365+0000 7fea5e7fc640 1 --2- 172.21.15.77:0/3321477063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600cf830 0x7fea600d22b0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fea48009870 tx=0x7fea4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f9bcf947e5a855 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.365+0000 7fea5dffb640 1 -- 172.21.15.77:0/3321477063 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea48035020 con 0x7fea600cf830 2024-03-06T07:32:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.365+0000 7fea5dffb640 1 -- 172.21.15.77:0/3321477063 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea48031870 con 0x7fea600cf830 2024-03-06T07:32:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.365+0000 7fea5dffb640 1 -- 172.21.15.77:0/3321477063 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea48031c40 con 0x7fea600cf830 2024-03-06T07:32:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.366+0000 7fea65908640 1 -- 172.21.15.77:0/3321477063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600cf830 msgr2=0x7fea600d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.366+0000 7fea65908640 1 --2- 172.21.15.77:0/3321477063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600cf830 0x7fea600d22b0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fea48009870 tx=0x7fea4802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.366+0000 7fea65908640 1 -- 172.21.15.77:0/3321477063 shutdown_connections 2024-03-06T07:32:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.366+0000 7fea65908640 1 --2- 172.21.15.77:0/3321477063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600cf830 0x7fea600d22b0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.366+0000 7fea65908640 1 --2- 172.21.15.77:0/3321477063 >> v2:172.21.15.158:3300/0 conn(0x7fea6015dfe0 0x7fea600cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.366+0000 7fea65908640 1 -- 172.21.15.77:0/3321477063 >> 172.21.15.77:0/3321477063 conn(0x7fea600d3cd0 msgr2=0x7fea600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.366+0000 7fea65908640 1 -- 172.21.15.77:0/3321477063 shutdown_connections 2024-03-06T07:32:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.366+0000 7fea65908640 1 -- 172.21.15.77:0/3321477063 wait complete. 2024-03-06T07:32:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.367+0000 7fea65908640 1 Processor -- start 2024-03-06T07:32:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.368+0000 7fea65908640 1 -- start start 2024-03-06T07:32:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea65908640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea600cf830 0x7fea6014e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea65908640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600d27f0 0x7fea600b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea65908640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea6015dfe0 0x7fea600b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea65908640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea600d1df0 con 0x7fea600d27f0 2024-03-06T07:32:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea65908640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea600d1c70 con 0x7fea6015dfe0 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea65908640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6015e3c0 con 0x7fea600cf830 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600d27f0 0x7fea600b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600d27f0 0x7fea600b1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41416/0 (socket says 172.21.15.77:41416) 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 -- 172.21.15.77:0/181917018 learned_addr learned my addr 172.21.15.77:0/181917018 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5effd640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea600cf830 0x7fea6014e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5f7fe640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea6015dfe0 0x7fea600b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 -- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea600cf830 msgr2=0x7fea6014e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea600cf830 0x7fea6014e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 -- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea6015dfe0 msgr2=0x7fea600b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea6015dfe0 0x7fea600b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 -- 172.21.15.77:0/181917018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6014e7e0 con 0x7fea600d27f0 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5f7fe640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea6015dfe0 0x7fea600b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5effd640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea600cf830 0x7fea6014e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.369+0000 7fea5e7fc640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600d27f0 0x7fea600b1f70 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fea48009870 tx=0x7fea480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.370+0000 7fea57fff640 1 -- 172.21.15.77:0/181917018 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4803ad60 con 0x7fea600d27f0 2024-03-06T07:32:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.370+0000 7fea57fff640 1 -- 172.21.15.77:0/181917018 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea480443f0 con 0x7fea600d27f0 2024-03-06T07:32:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.370+0000 7fea57fff640 1 -- 172.21.15.77:0/181917018 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480447e0 con 0x7fea600d27f0 2024-03-06T07:32:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.370+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6015f1b0 con 0x7fea600d27f0 2024-03-06T07:32:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.370+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6015f770 con 0x7fea600d27f0 2024-03-06T07:32:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.371+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6004f330 con 0x7fea600d27f0 2024-03-06T07:32:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.371+0000 7fea57fff640 1 -- 172.21.15.77:0/181917018 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea4804b060 con 0x7fea600d27f0 2024-03-06T07:32:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.372+0000 7fea57fff640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea4006b4d0 0x7fea4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.372+0000 7fea57fff640 1 -- 172.21.15.77:0/181917018 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fea480ac3e0 con 0x7fea600d27f0 2024-03-06T07:32:28.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.372+0000 7fea5effd640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea4006b4d0 0x7fea4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.372+0000 7fea5effd640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea4006b4d0 0x7fea4006d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fea58007eb0 tx=0x7fea580073c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.376+0000 7fea57fff640 1 -- 172.21.15.77:0/181917018 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea4803f070 con 0x7fea600d27f0 2024-03-06T07:32:28.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.592+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea600b2f60 con 0x7fea600d27f0 2024-03-06T07:32:28.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.592+0000 7fea57fff640 1 -- 172.21.15.77:0/181917018 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fea4802f320 con 0x7fea600d27f0 2024-03-06T07:32:28.594 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:28.594 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:28.595 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:28.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea4006b4d0 msgr2=0x7fea4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea4006b4d0 0x7fea4006d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fea58007eb0 tx=0x7fea580073c0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600d27f0 msgr2=0x7fea600b1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600d27f0 0x7fea600b1f70 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fea48009870 tx=0x7fea480046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 shutdown_connections 2024-03-06T07:32:28.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fea4006b4d0 0x7fea4006d990 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea600d27f0 0x7fea600b1f70 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fea6015dfe0 0x7fea600b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 --2- 172.21.15.77:0/181917018 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fea600cf830 0x7fea6014e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 >> 172.21.15.77:0/181917018 conn(0x7fea600d3cd0 msgr2=0x7fea600cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 shutdown_connections 2024-03-06T07:32:28.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:28.594+0000 7fea65908640 1 -- 172.21.15.77:0/181917018 wait complete. 2024-03-06T07:32:28.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:28.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:29.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:29.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.909+0000 7f73a6d9a640 1 Processor -- start 2024-03-06T07:32:29.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.909+0000 7f73a6d9a640 1 -- start start 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f73a6d9a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73a00cce40 0x7f73a015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f73a6d9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a00cd7a0 0x7f73a00cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f73a6d9a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a0071f40 con 0x7f73a01515e0 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f73a6d9a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a00cd2f0 con 0x7f73a00cce40 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f73a6d9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a0070a70 con 0x7f73a00cd7a0 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f739ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73a00cce40 0x7f73a015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f739ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73a00cce40 0x7f73a015b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56980/0 (socket says 172.21.15.77:56980) 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f739ffff640 1 -- 172.21.15.77:0/2443106132 learned_addr learned my addr 172.21.15.77:0/2443106132 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f73a5337640 1 --2- 172.21.15.77:0/2443106132 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a00cd7a0 0x7f73a00cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f739ffff640 1 -- 172.21.15.77:0/2443106132 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a00cd7a0 msgr2=0x7f73a00cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f739ffff640 1 --2- 172.21.15.77:0/2443106132 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a00cd7a0 0x7f73a00cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f739ffff640 1 -- 172.21.15.77:0/2443106132 >> v1:172.21.15.158:6789/0 conn(0x7f73a01515e0 legacy=0x7f73a0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.910+0000 7f739ffff640 1 -- 172.21.15.77:0/2443106132 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a0070d50 con 0x7f73a00cce40 2024-03-06T07:32:29.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.911+0000 7f739ffff640 1 --2- 172.21.15.77:0/2443106132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cce40 0x7f73a015b430 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f738c009870 tx=0x7f738c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7617a42fbbdbc741 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.911+0000 7f739f7fe640 1 -- 172.21.15.77:0/2443106132 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c034070 con 0x7f73a00cce40 2024-03-06T07:32:29.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.911+0000 7f739f7fe640 1 -- 172.21.15.77:0/2443106132 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738c030840 con 0x7f73a00cce40 2024-03-06T07:32:29.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.911+0000 7f739f7fe640 1 -- 172.21.15.77:0/2443106132 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c030c10 con 0x7f73a00cce40 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.912+0000 7f73a6d9a640 1 -- 172.21.15.77:0/2443106132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cce40 msgr2=0x7f73a015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.912+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/2443106132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cce40 0x7f73a015b430 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f738c009870 tx=0x7f738c02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.912+0000 7f73a6d9a640 1 -- 172.21.15.77:0/2443106132 shutdown_connections 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.912+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/2443106132 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a00cd7a0 0x7f73a00cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.912+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/2443106132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cce40 0x7f73a015b430 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.912+0000 7f73a6d9a640 1 -- 172.21.15.77:0/2443106132 >> 172.21.15.77:0/2443106132 conn(0x7f73a00cf8a0 msgr2=0x7f73a00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.912+0000 7f73a6d9a640 1 -- 172.21.15.77:0/2443106132 shutdown_connections 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.912+0000 7f73a6d9a640 1 -- 172.21.15.77:0/2443106132 wait complete. 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.913+0000 7f73a6d9a640 1 Processor -- start 2024-03-06T07:32:29.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.913+0000 7f73a6d9a640 1 -- start start 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a6d9a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73a00cce40 0x7f73a006df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a6d9a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cd7a0 0x7f73a01421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a6d9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a01515e0 0x7f73a014ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a6d9a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00ce0f0 con 0x7f73a00cd7a0 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a6d9a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a0071310 con 0x7f73a00cce40 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a6d9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00cc3a0 con 0x7f73a01515e0 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cd7a0 0x7f73a01421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cd7a0 0x7f73a01421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56984/0 (socket says 172.21.15.77:56984) 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 -- 172.21.15.77:0/3254105033 learned_addr learned my addr 172.21.15.77:0/3254105033 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a4b36640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73a00cce40 0x7f73a006df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a5337640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a01515e0 0x7f73a014ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 -- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a01515e0 msgr2=0x7f73a014ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a01515e0 0x7f73a014ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 -- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73a00cce40 msgr2=0x7f73a006df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73a00cce40 0x7f73a006df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 -- 172.21.15.77:0/3254105033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a0145200 con 0x7f73a00cd7a0 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f73a4b36640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73a00cce40 0x7f73a006df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.915+0000 7f739ffff640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cd7a0 0x7f73a01421a0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f738c000c00 tx=0x7f738c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.916+0000 7f739d7fa640 1 -- 172.21.15.77:0/3254105033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c048070 con 0x7f73a00cd7a0 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.916+0000 7f739d7fa640 1 -- 172.21.15.77:0/3254105033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738c030640 con 0x7f73a00cd7a0 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.916+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a014d980 con 0x7f73a00cd7a0 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.916+0000 7f739d7fa640 1 -- 172.21.15.77:0/3254105033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c034040 con 0x7f73a00cd7a0 2024-03-06T07:32:29.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.916+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a006d790 con 0x7f73a00cd7a0 2024-03-06T07:32:29.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.917+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a0068840 con 0x7f73a00cd7a0 2024-03-06T07:32:29.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.917+0000 7f739d7fa640 1 -- 172.21.15.77:0/3254105033 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f738c04b070 con 0x7f73a00cd7a0 2024-03-06T07:32:29.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.917+0000 7f739d7fa640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f738406b4d0 0x7f738406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.918+0000 7f739d7fa640 1 -- 172.21.15.77:0/3254105033 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f738c0ac3f0 con 0x7f73a00cd7a0 2024-03-06T07:32:29.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.918+0000 7f73a4b36640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f738406b4d0 0x7f738406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.918+0000 7f73a4b36640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f738406b4d0 0x7f738406d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7394007fd0 tx=0x7f7394009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:29.926+0000 7f739d7fa640 1 -- 172.21.15.77:0/3254105033 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73a0145200 con 0x7f73a00cd7a0 2024-03-06T07:32:30.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.217+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73a00b4c30 con 0x7f73a00cd7a0 2024-03-06T07:32:30.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.217+0000 7f739d7fa640 1 -- 172.21.15.77:0/3254105033 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f738c07b400 con 0x7f73a00cd7a0 2024-03-06T07:32:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:30.220 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:30.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f738406b4d0 msgr2=0x7f738406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f738406b4d0 0x7f738406d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7394007fd0 tx=0x7f7394009040 comp rx=0 tx=0).stop 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cd7a0 msgr2=0x7f73a01421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cd7a0 0x7f73a01421a0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f738c000c00 tx=0x7f738c004730 comp rx=0 tx=0).stop 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 shutdown_connections 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f738406b4d0 0x7f738406d990 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a00cd7a0 0x7f73a01421a0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f73a01515e0 0x7f73a014ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 --2- 172.21.15.77:0/3254105033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f73a00cce40 0x7f73a006df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 >> 172.21.15.77:0/3254105033 conn(0x7f73a00cf8a0 msgr2=0x7f73a00cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.219+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 shutdown_connections 2024-03-06T07:32:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:30.220+0000 7f73a6d9a640 1 -- 172.21.15.77:0/3254105033 wait complete. 2024-03-06T07:32:30.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:30.438 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:31.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:31.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.540+0000 7f757dbf7640 1 Processor -- start 2024-03-06T07:32:31.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.540+0000 7f757dbf7640 1 -- start start 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f757dbf7640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f75780d22b0 0x7f75780d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f757dbf7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f75780b41f0 0x7f75780b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f757dbf7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75780b3430 con 0x7f757815dfe0 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f757dbf7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75780d2c50 con 0x7f75780b41f0 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f757dbf7640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75780b31e0 con 0x7f75780d22b0 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f7577fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f75780b41f0 0x7f75780b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f7577fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f75780b41f0 0x7f75780b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56988/0 (socket says 172.21.15.77:56988) 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f7577fff640 1 -- 172.21.15.77:0/2403803571 learned_addr learned my addr 172.21.15.77:0/2403803571 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f75777fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f757815dfe0 0x7f75780cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35594/0 (socket says 172.21.15.77:35594) 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f7577fff640 1 -- 172.21.15.77:0/2403803571 >> v2:172.21.15.158:3300/0 conn(0x7f75780d22b0 msgr2=0x7f75780d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f756effd640 1 --2- 172.21.15.77:0/2403803571 >> v2:172.21.15.158:3300/0 conn(0x7f75780d22b0 0x7f75780d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f7577fff640 1 --2- 172.21.15.77:0/2403803571 >> v2:172.21.15.158:3300/0 conn(0x7f75780d22b0 0x7f75780d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f7576ffd640 1 -- 172.21.15.77:0/2403803571 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102062897 0 0) 0x7f75780b3430 con 0x7f757815dfe0 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f7577fff640 1 -- 172.21.15.77:0/2403803571 >> v1:172.21.15.77:6789/0 conn(0x7f757815dfe0 legacy=0x7f75780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.541+0000 7f7577fff640 1 -- 172.21.15.77:0/2403803571 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75780b3710 con 0x7f75780b41f0 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.542+0000 7f756effd640 1 --2- 172.21.15.77:0/2403803571 >> v2:172.21.15.158:3300/0 conn(0x7f75780d22b0 0x7f75780d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.542+0000 7f7577fff640 1 --2- 172.21.15.77:0/2403803571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780b41f0 0x7f75780b4680 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f7570009e50 tx=0x7f757002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c02d8ea8df6f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.542+0000 7f7576ffd640 1 -- 172.21.15.77:0/2403803571 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757002cb20 con 0x7f75780b41f0 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.542+0000 7f7576ffd640 1 -- 172.21.15.77:0/2403803571 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757002ccc0 con 0x7f75780b41f0 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.542+0000 7f7576ffd640 1 -- 172.21.15.77:0/2403803571 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570034610 con 0x7f75780b41f0 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.543+0000 7f757dbf7640 1 -- 172.21.15.77:0/2403803571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780b41f0 msgr2=0x7f75780b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.543+0000 7f757dbf7640 1 --2- 172.21.15.77:0/2403803571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780b41f0 0x7f75780b4680 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f7570009e50 tx=0x7f757002bf80 comp rx=0 tx=0).stop 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.543+0000 7f757dbf7640 1 -- 172.21.15.77:0/2403803571 shutdown_connections 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.543+0000 7f757dbf7640 1 --2- 172.21.15.77:0/2403803571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780b41f0 0x7f75780b4680 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.543+0000 7f757dbf7640 1 --2- 172.21.15.77:0/2403803571 >> v2:172.21.15.158:3300/0 conn(0x7f75780d22b0 0x7f75780d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.543+0000 7f757dbf7640 1 -- 172.21.15.77:0/2403803571 >> 172.21.15.77:0/2403803571 conn(0x7f75780d3cd0 msgr2=0x7f75780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.543+0000 7f757dbf7640 1 -- 172.21.15.77:0/2403803571 shutdown_connections 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.543+0000 7f757dbf7640 1 -- 172.21.15.77:0/2403803571 wait complete. 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.544+0000 7f757dbf7640 1 Processor -- start 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.544+0000 7f757dbf7640 1 -- start start 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f757dbf7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75780b41f0 0x7f75780b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f757dbf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780d22b0 0x7f75781531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f757dbf7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f757815dfe0 0x7f75780b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f757dbf7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75780cfa00 con 0x7f75780d22b0 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f757dbf7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75780b3d60 con 0x7f75780b41f0 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f757dbf7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75780d1c70 con 0x7f757815dfe0 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f756effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780d22b0 0x7f75781531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f756effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780d22b0 0x7f75781531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57000/0 (socket says 172.21.15.77:57000) 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f756effd640 1 -- 172.21.15.77:0/1057013936 learned_addr learned my addr 172.21.15.77:0/1057013936 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f756effd640 1 -- 172.21.15.77:0/1057013936 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f757815dfe0 msgr2=0x7f75780b8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f756effd640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f757815dfe0 0x7f75780b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f756effd640 1 -- 172.21.15.77:0/1057013936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75780b41f0 msgr2=0x7f75780b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f75777fe640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75780b41f0 0x7f75780b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f756effd640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75780b41f0 0x7f75780b72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.545+0000 7f756effd640 1 -- 172.21.15.77:0/1057013936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75780b7b70 con 0x7f75780d22b0 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.546+0000 7f756effd640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780d22b0 0x7f75781531d0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f756000c7f0 tx=0x7f756000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.546+0000 7f7574ff9640 1 -- 172.21.15.77:0/1057013936 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7560015070 con 0x7f75780d22b0 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.546+0000 7f7574ff9640 1 -- 172.21.15.77:0/1057013936 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7560007a30 con 0x7f75780d22b0 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.546+0000 7f7574ff9640 1 -- 172.21.15.77:0/1057013936 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7560007e00 con 0x7f75780d22b0 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.546+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75780b7e50 con 0x7f75780d22b0 2024-03-06T07:32:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.546+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75780b77f0 con 0x7f75780d22b0 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.547+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757804f330 con 0x7f75780d22b0 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.547+0000 7f7574ff9640 1 -- 172.21.15.77:0/1057013936 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7560018070 con 0x7f75780d22b0 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.548+0000 7f7574ff9640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f755806b4d0 0x7f755806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.548+0000 7f7574ff9640 1 -- 172.21.15.77:0/1057013936 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f756008a140 con 0x7f75780d22b0 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.548+0000 7f75777fe640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f755806b4d0 0x7f755806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.548+0000 7f75777fe640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f755806b4d0 0x7f755806d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7568000f60 tx=0x7f7568009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.552+0000 7f7574ff9640 1 -- 172.21.15.77:0/1057013936 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7560059030 con 0x7f75780d22b0 2024-03-06T07:32:31.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.769+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75780d0f30 con 0x7f75780d22b0 2024-03-06T07:32:31.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.770+0000 7f7574ff9640 1 -- 172.21.15.77:0/1057013936 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f7560058930 con 0x7f75780d22b0 2024-03-06T07:32:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:31.772 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f755806b4d0 msgr2=0x7f755806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f755806b4d0 0x7f755806d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7568000f60 tx=0x7f7568009040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780d22b0 msgr2=0x7f75781531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780d22b0 0x7f75781531d0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f756000c7f0 tx=0x7f756000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 shutdown_connections 2024-03-06T07:32:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f755806b4d0 0x7f755806d990 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75780d22b0 0x7f75781531d0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f757815dfe0 0x7f75780b8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 --2- 172.21.15.77:0/1057013936 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f75780b41f0 0x7f75780b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 >> 172.21.15.77:0/1057013936 conn(0x7f75780d3cd0 msgr2=0x7f75780cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 shutdown_connections 2024-03-06T07:32:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:31.772+0000 7f757dbf7640 1 -- 172.21.15.77:0/1057013936 wait complete. 2024-03-06T07:32:31.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:31.986 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:32.210 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:32.212 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:32.987 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.124+0000 7f0e77179640 1 Processor -- start 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.124+0000 7f0e77179640 1 -- start start 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.125+0000 7f0e77179640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e701515e0 0x7f0e700cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.125+0000 7f0e77179640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 0x7f0e700cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.125+0000 7f0e77179640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e70071f40 con 0x7f0e7015b430 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.125+0000 7f0e77179640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7015b8e0 con 0x7f0e701515e0 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.125+0000 7f0e77179640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e70070a70 con 0x7f0e700cd7a0 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.125+0000 7f0e74f15640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e701515e0 0x7f0e700cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.126+0000 7f0e74f15640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e701515e0 0x7f0e700cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57008/0 (socket says 172.21.15.77:57008) 2024-03-06T07:32:33.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.126+0000 7f0e74f15640 1 -- 172.21.15.77:0/1652061879 learned_addr learned my addr 172.21.15.77:0/1652061879 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.126+0000 7f0e75716640 1 --2- 172.21.15.77:0/1652061879 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 0x7f0e700cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.126+0000 7f0e74f15640 1 -- 172.21.15.77:0/1652061879 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 msgr2=0x7f0e700cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.126+0000 7f0e74f15640 1 --2- 172.21.15.77:0/1652061879 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 0x7f0e700cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.126+0000 7f0e74f15640 1 -- 172.21.15.77:0/1652061879 >> v1:172.21.15.158:6789/0 conn(0x7f0e7015b430 legacy=0x7f0e70071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.126+0000 7f0e74f15640 1 -- 172.21.15.77:0/1652061879 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70070d50 con 0x7f0e701515e0 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e74f15640 1 --2- 172.21.15.77:0/1652061879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e700cce40 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f0e640098a0 tx=0x7f0e6402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9f07971adddb14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e6f7fe640 1 -- 172.21.15.77:0/1652061879 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64034070 con 0x7f0e701515e0 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e6f7fe640 1 -- 172.21.15.77:0/1652061879 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e64030840 con 0x7f0e701515e0 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e6f7fe640 1 -- 172.21.15.77:0/1652061879 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64030c10 con 0x7f0e701515e0 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e77179640 1 -- 172.21.15.77:0/1652061879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 msgr2=0x7f0e700cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e77179640 1 --2- 172.21.15.77:0/1652061879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e700cce40 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f0e640098a0 tx=0x7f0e6402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e77179640 1 -- 172.21.15.77:0/1652061879 shutdown_connections 2024-03-06T07:32:33.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e77179640 1 --2- 172.21.15.77:0/1652061879 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 0x7f0e700cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e77179640 1 --2- 172.21.15.77:0/1652061879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e700cce40 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.127+0000 7f0e77179640 1 -- 172.21.15.77:0/1652061879 >> 172.21.15.77:0/1652061879 conn(0x7f0e700cf8a0 msgr2=0x7f0e700cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.128+0000 7f0e77179640 1 -- 172.21.15.77:0/1652061879 shutdown_connections 2024-03-06T07:32:33.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.128+0000 7f0e77179640 1 -- 172.21.15.77:0/1652061879 wait complete. 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.129+0000 7f0e77179640 1 Processor -- start 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.129+0000 7f0e77179640 1 -- start start 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.129+0000 7f0e77179640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 0x7f0e7014b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.129+0000 7f0e77179640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e70147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.129+0000 7f0e77179640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7015b430 0x7f0e70149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.129+0000 7f0e77179640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700ce0f0 con 0x7f0e701515e0 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e77179640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e70071310 con 0x7f0e7015b430 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e77179640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700cc3a0 con 0x7f0e700cd7a0 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e70147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e70147bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57024/0 (socket says 172.21.15.77:57024) 2024-03-06T07:32:33.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 -- 172.21.15.77:0/168984937 learned_addr learned my addr 172.21.15.77:0/168984937 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e74f15640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 0x7f0e7014b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e75716640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7015b430 0x7f0e70149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 -- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 msgr2=0x7f0e7014b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 0x7f0e7014b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 -- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7015b430 msgr2=0x7f0e70149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7015b430 0x7f0e70149500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 -- 172.21.15.77:0/168984937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7015d0f0 con 0x7f0e701515e0 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e75716640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7015b430 0x7f0e70149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6ffff640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e70147bd0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c340 tx=0x7f0e5c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6d7fa640 1 -- 172.21.15.77:0/168984937 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c017070 con 0x7f0e701515e0 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6d7fa640 1 -- 172.21.15.77:0/168984937 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c0052d0 con 0x7f0e701515e0 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e701463e0 con 0x7f0e701515e0 2024-03-06T07:32:33.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e6d7fa640 1 -- 172.21.15.77:0/168984937 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c003850 con 0x7f0e701515e0 2024-03-06T07:32:33.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.130+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7015d670 con 0x7f0e701515e0 2024-03-06T07:32:33.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.132+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e70068840 con 0x7f0e701515e0 2024-03-06T07:32:33.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.132+0000 7f0e6d7fa640 1 -- 172.21.15.77:0/168984937 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e5c01a070 con 0x7f0e701515e0 2024-03-06T07:32:33.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.132+0000 7f0e6d7fa640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5406b4d0 0x7f0e5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.132+0000 7f0e6d7fa640 1 -- 172.21.15.77:0/168984937 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0e5c089510 con 0x7f0e701515e0 2024-03-06T07:32:33.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.132+0000 7f0e74f15640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5406b4d0 0x7f0e5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.133+0000 7f0e74f15640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5406b4d0 0x7f0e5406d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0e64036000 tx=0x7f0e6403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.136+0000 7f0e6d7fa640 1 -- 172.21.15.77:0/168984937 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e5c058380 con 0x7f0e701515e0 2024-03-06T07:32:33.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.351+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e7004f330 con 0x7f0e701515e0 2024-03-06T07:32:33.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.352+0000 7f0e6d7fa640 1 -- 172.21.15.77:0/168984937 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f0e5c057c80 con 0x7f0e701515e0 2024-03-06T07:32:33.354 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:33.354 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:33.354 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:33.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5406b4d0 msgr2=0x7f0e5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5406b4d0 0x7f0e5406d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0e64036000 tx=0x7f0e6403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 msgr2=0x7f0e70147bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e70147bd0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c340 tx=0x7f0e5c00c800 comp rx=0 tx=0).stop 2024-03-06T07:32:33.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 shutdown_connections 2024-03-06T07:32:33.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0e5406b4d0 0x7f0e5406d990 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e701515e0 0x7f0e70147bd0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0e7015b430 0x7f0e70149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 --2- 172.21.15.77:0/168984937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0e700cd7a0 0x7f0e7014b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 >> 172.21.15.77:0/168984937 conn(0x7f0e700cf8a0 msgr2=0x7f0e70165980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 shutdown_connections 2024-03-06T07:32:33.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:33.354+0000 7f0e77179640 1 -- 172.21.15.77:0/168984937 wait complete. 2024-03-06T07:32:33.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:33.568 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:34.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:34.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.668+0000 7f8228084640 1 Processor -- start 2024-03-06T07:32:34.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.668+0000 7f8228084640 1 -- start start 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.669+0000 7f8228084640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82200d22b0 0x7f82200d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.669+0000 7f8228084640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82200b41f0 0x7f82200b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.669+0000 7f8228084640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82200b3430 con 0x7f822015dfe0 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.669+0000 7f8228084640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82200d2c50 con 0x7f82200d22b0 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.669+0000 7f8228084640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82200b31e0 con 0x7f82200b41f0 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.669+0000 7f822561f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82200d22b0 0x7f82200d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.669+0000 7f8225e20640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f822015dfe0 0x7f82200cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35608/0 (socket says 172.21.15.77:35608) 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.669+0000 7f8225e20640 1 -- 172.21.15.77:0/2044264713 learned_addr learned my addr 172.21.15.77:0/2044264713 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.670+0000 7f8226621640 1 --2- 172.21.15.77:0/2044264713 >> v2:172.21.15.158:3300/0 conn(0x7f82200b41f0 0x7f82200b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.670+0000 7f822561f640 1 -- 172.21.15.77:0/2044264713 >> v2:172.21.15.158:3300/0 conn(0x7f82200b41f0 msgr2=0x7f82200b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.670+0000 7f822561f640 1 --2- 172.21.15.77:0/2044264713 >> v2:172.21.15.158:3300/0 conn(0x7f82200b41f0 0x7f82200b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.670+0000 7f822561f640 1 -- 172.21.15.77:0/2044264713 >> v1:172.21.15.77:6789/0 conn(0x7f822015dfe0 legacy=0x7f82200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.670+0000 7f822561f640 1 -- 172.21.15.77:0/2044264713 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82200b3710 con 0x7f82200d22b0 2024-03-06T07:32:34.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.670+0000 7f8224e1e640 1 -- 172.21.15.77:0/2044264713 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480875857 0 0) 0x7f82200b3430 con 0x7f822015dfe0 2024-03-06T07:32:34.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.670+0000 7f822561f640 1 --2- 172.21.15.77:0/2044264713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82200d2710 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f821c0097a0 tx=0x7f821c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af09e256a7d51d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8224e1e640 1 -- 172.21.15.77:0/2044264713 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c035020 con 0x7f82200d22b0 2024-03-06T07:32:34.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8224e1e640 1 -- 172.21.15.77:0/2044264713 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821c031830 con 0x7f82200d22b0 2024-03-06T07:32:34.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8224e1e640 1 -- 172.21.15.77:0/2044264713 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c031c00 con 0x7f82200d22b0 2024-03-06T07:32:34.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8228084640 1 -- 172.21.15.77:0/2044264713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 msgr2=0x7f82200d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8228084640 1 --2- 172.21.15.77:0/2044264713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82200d2710 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f821c0097a0 tx=0x7f821c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8228084640 1 -- 172.21.15.77:0/2044264713 shutdown_connections 2024-03-06T07:32:34.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8228084640 1 --2- 172.21.15.77:0/2044264713 >> v2:172.21.15.158:3300/0 conn(0x7f82200b41f0 0x7f82200b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8228084640 1 --2- 172.21.15.77:0/2044264713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82200d2710 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8228084640 1 -- 172.21.15.77:0/2044264713 >> 172.21.15.77:0/2044264713 conn(0x7f82200d3cd0 msgr2=0x7f82200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8228084640 1 -- 172.21.15.77:0/2044264713 shutdown_connections 2024-03-06T07:32:34.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.671+0000 7f8228084640 1 -- 172.21.15.77:0/2044264713 wait complete. 2024-03-06T07:32:34.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.672+0000 7f8228084640 1 Processor -- start 2024-03-06T07:32:34.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.672+0000 7f8228084640 1 -- start start 2024-03-06T07:32:34.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f8228084640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82200b41f0 0x7f822015e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f8228084640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82201530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f8228084640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822015dfe0 0x7f82200b8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f8228084640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82200cfa00 con 0x7f82200d22b0 2024-03-06T07:32:34.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f8228084640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82200b3d60 con 0x7f822015dfe0 2024-03-06T07:32:34.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f8228084640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82200d1c70 con 0x7f82200b41f0 2024-03-06T07:32:34.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f822561f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82201530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f822561f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82201530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57028/0 (socket says 172.21.15.77:57028) 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f822561f640 1 -- 172.21.15.77:0/1732253920 learned_addr learned my addr 172.21.15.77:0/1732253920 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f8225e20640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82200b41f0 0x7f822015e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f8226621640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822015dfe0 0x7f82200b8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f822561f640 1 -- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82200b41f0 msgr2=0x7f822015e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f822561f640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82200b41f0 0x7f822015e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f822561f640 1 -- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822015dfe0 msgr2=0x7f82200b8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.673+0000 7f822561f640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822015dfe0 0x7f82200b8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f822561f640 1 -- 172.21.15.77:0/1732253920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822015e900 con 0x7f82200d22b0 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f8226621640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822015dfe0 0x7f82200b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f8225e20640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82200b41f0 0x7f822015e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f822561f640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82201530f0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f821c02bee0 tx=0x7f821c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f82167fc640 1 -- 172.21.15.77:0/1732253920 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c041070 con 0x7f82200d22b0 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f82167fc640 1 -- 172.21.15.77:0/1732253920 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821c03b920 con 0x7f82200d22b0 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822015eba0 con 0x7f82200d22b0 2024-03-06T07:32:34.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f82167fc640 1 -- 172.21.15.77:0/1732253920 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c035040 con 0x7f82200d22b0 2024-03-06T07:32:34.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.674+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822015f160 con 0x7f82200d22b0 2024-03-06T07:32:34.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.675+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822004f330 con 0x7f82200d22b0 2024-03-06T07:32:34.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.676+0000 7f82167fc640 1 -- 172.21.15.77:0/1732253920 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f821c04c060 con 0x7f82200d22b0 2024-03-06T07:32:34.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.676+0000 7f82167fc640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f820406b4d0 0x7f820406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.676+0000 7f8225e20640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f820406b4d0 0x7f820406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.676+0000 7f82167fc640 1 -- 172.21.15.77:0/1732253920 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f821c0abc10 con 0x7f82200d22b0 2024-03-06T07:32:34.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.677+0000 7f8225e20640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f820406b4d0 0x7f820406d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8210000f60 tx=0x7f8210009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.681+0000 7f82167fc640 1 -- 172.21.15.77:0/1732253920 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f821c07a430 con 0x7f82200d22b0 2024-03-06T07:32:34.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.894+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82200d1910 con 0x7f82200d22b0 2024-03-06T07:32:34.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.894+0000 7f82167fc640 1 -- 172.21.15.77:0/1732253920 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f821c07e4d0 con 0x7f82200d22b0 2024-03-06T07:32:34.896 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:34.896 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:34.897 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f820406b4d0 msgr2=0x7f820406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f820406b4d0 0x7f820406d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8210000f60 tx=0x7f8210009040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 msgr2=0x7f82201530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82201530f0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f821c02bee0 tx=0x7f821c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 shutdown_connections 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f820406b4d0 0x7f820406d990 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82200d22b0 0x7f82201530f0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f822015dfe0 0x7f82200b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 --2- 172.21.15.77:0/1732253920 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82200b41f0 0x7f822015e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 >> 172.21.15.77:0/1732253920 conn(0x7f82200d3cd0 msgr2=0x7f822015ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 shutdown_connections 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:34.896+0000 7f8228084640 1 -- 172.21.15.77:0/1732253920 wait complete. 2024-03-06T07:32:35.109 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:35.110 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:36.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.210+0000 7fca170ed640 1 Processor -- start 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.210+0000 7fca170ed640 1 -- start start 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.210+0000 7fca170ed640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fca1015dfe0 0x7fca100cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.210+0000 7fca170ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca100d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.210+0000 7fca170ed640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca100b3430 con 0x7fca100d27f0 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.210+0000 7fca170ed640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca100d2c50 con 0x7fca1015dfe0 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.210+0000 7fca170ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca100b31e0 con 0x7fca100cf830 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca14e89640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fca1015dfe0 0x7fca100cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca0ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca100d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca14e89640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fca1015dfe0 0x7fca100cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:36200/0 (socket says 172.21.15.77:36200) 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca14e89640 1 -- 172.21.15.77:0/3760779176 learned_addr learned my addr 172.21.15.77:0/3760779176 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca0ffff640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca100d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:55786/0 (socket says 172.21.15.77:55786) 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca14e89640 1 -- 172.21.15.77:0/3760779176 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 msgr2=0x7fca100d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca14e89640 1 --2- 172.21.15.77:0/3760779176 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca100d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca14e89640 1 -- 172.21.15.77:0/3760779176 >> v1:172.21.15.158:6789/0 conn(0x7fca100d27f0 legacy=0x7fca100d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.211+0000 7fca14e89640 1 -- 172.21.15.77:0/3760779176 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca100b3710 con 0x7fca1015dfe0 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.212+0000 7fca14e89640 1 --2- 172.21.15.77:0/3760779176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 0x7fca100cf2f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fca04009870 tx=0x7fca0402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f31dc9be84ff785 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.212+0000 7fca0f7fe640 1 -- 172.21.15.77:0/3760779176 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04034070 con 0x7fca1015dfe0 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.212+0000 7fca0f7fe640 1 -- 172.21.15.77:0/3760779176 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04030840 con 0x7fca1015dfe0 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.212+0000 7fca0f7fe640 1 -- 172.21.15.77:0/3760779176 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04030c10 con 0x7fca1015dfe0 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.213+0000 7fca170ed640 1 -- 172.21.15.77:0/3760779176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 msgr2=0x7fca100cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.213+0000 7fca170ed640 1 --2- 172.21.15.77:0/3760779176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 0x7fca100cf2f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fca04009870 tx=0x7fca0402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.213+0000 7fca170ed640 1 -- 172.21.15.77:0/3760779176 shutdown_connections 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.213+0000 7fca170ed640 1 --2- 172.21.15.77:0/3760779176 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca100d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.213+0000 7fca170ed640 1 --2- 172.21.15.77:0/3760779176 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 0x7fca100cf2f0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.213+0000 7fca170ed640 1 -- 172.21.15.77:0/3760779176 >> 172.21.15.77:0/3760779176 conn(0x7fca100d3cd0 msgr2=0x7fca100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.213+0000 7fca170ed640 1 -- 172.21.15.77:0/3760779176 shutdown_connections 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.213+0000 7fca170ed640 1 -- 172.21.15.77:0/3760779176 wait complete. 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.214+0000 7fca170ed640 1 Processor -- start 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.214+0000 7fca170ed640 1 -- start start 2024-03-06T07:32:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca170ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca101f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca170ed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca100d27f0 0x7fca1014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca170ed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 0x7fca1014d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca170ed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca100d1c70 con 0x7fca100d27f0 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca170ed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca100b3d60 con 0x7fca1015dfe0 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca170ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca100d1df0 con 0x7fca100cf830 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca100d27f0 0x7fca1014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca100d27f0 0x7fca1014d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57032/0 (socket says 172.21.15.77:57032) 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 -- 172.21.15.77:0/1019283631 learned_addr learned my addr 172.21.15.77:0/1019283631 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca14e89640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca101f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca1568a640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 0x7fca1014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 -- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 msgr2=0x7fca101f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca101f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 -- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 msgr2=0x7fca1014d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 0x7fca1014d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 -- 172.21.15.77:0/1019283631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca101f40b0 con 0x7fca100d27f0 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca1568a640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 0x7fca1014d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.215+0000 7fca0ffff640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca100d27f0 0x7fca1014d320 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00e980 tx=0x7fc9fc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.216+0000 7fca0d7fa640 1 -- 172.21.15.77:0/1019283631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc017070 con 0x7fca100d27f0 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.216+0000 7fca0d7fa640 1 -- 172.21.15.77:0/1019283631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc00ca00 con 0x7fca100d27f0 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.216+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca101f4390 con 0x7fca100d27f0 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.216+0000 7fca0d7fa640 1 -- 172.21.15.77:0/1019283631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc00cdd0 con 0x7fca100d27f0 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.216+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1014ce00 con 0x7fca100d27f0 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.217+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1004f330 con 0x7fca100d27f0 2024-03-06T07:32:36.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.217+0000 7fca0d7fa640 1 -- 172.21.15.77:0/1019283631 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9fc01a070 con 0x7fca100d27f0 2024-03-06T07:32:36.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.218+0000 7fca0d7fa640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.218+0000 7fca0d7fa640 1 -- 172.21.15.77:0/1019283631 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc9fc08a580 con 0x7fca100d27f0 2024-03-06T07:32:36.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.218+0000 7fca14e89640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.218+0000 7fca14e89640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc9f406b4d0 0x7fc9f406d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fca04036000 tx=0x7fca0403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.226+0000 7fca0d7fa640 1 -- 172.21.15.77:0/1019283631 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9fc059470 con 0x7fca100d27f0 2024-03-06T07:32:36.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.435+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca100b0d40 con 0x7fca100d27f0 2024-03-06T07:32:36.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.436+0000 7fca0d7fa640 1 -- 172.21.15.77:0/1019283631 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc9fc058d70 con 0x7fca100d27f0 2024-03-06T07:32:36.438 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:36.438 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:31:21.073903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:36.438 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 106 2024-03-06T07:32:36.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc9f406b4d0 msgr2=0x7fc9f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc9f406b4d0 0x7fc9f406d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fca04036000 tx=0x7fca0403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:36.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca100d27f0 msgr2=0x7fca1014d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca100d27f0 0x7fca1014d320 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00e980 tx=0x7fc9fc00ee40 comp rx=0 tx=0).stop 2024-03-06T07:32:36.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 shutdown_connections 2024-03-06T07:32:36.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca100d27f0 0x7fca1014d320 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fca1015dfe0 0x7fca1014d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 --2- 172.21.15.77:0/1019283631 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fca100cf830 0x7fca101f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 >> 172.21.15.77:0/1019283631 conn(0x7fca100d3cd0 msgr2=0x7fca100cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 shutdown_connections 2024-03-06T07:32:36.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:36.438+0000 7fca170ed640 1 -- 172.21.15.77:0/1019283631 wait complete. 2024-03-06T07:32:36.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:31:21.073903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:36.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:37.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.755+0000 7f3ebf593640 1 Processor -- start 2024-03-06T07:32:37.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebf593640 1 -- start start 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebf593640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb815dfe0 0x7f3eb80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebf593640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3eb80cf830 0x7f3eb80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebf593640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb80b3430 con 0x7f3eb80d27f0 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebf593640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb80d2c50 con 0x7f3eb80cf830 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebf593640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b31e0 con 0x7f3eb815dfe0 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebcb2e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3eb80cf830 0x7f3eb80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebdb30640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3eb80d27f0 0x7f3eb80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35618/0 (socket says 172.21.15.77:35618) 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.756+0000 7f3ebdb30640 1 -- 172.21.15.77:0/3151418107 learned_addr learned my addr 172.21.15.77:0/3151418107 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.757+0000 7f3ebd32f640 1 --2- 172.21.15.77:0/3151418107 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb815dfe0 0x7f3eb80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.757+0000 7f3eaffff640 1 -- 172.21.15.77:0/3151418107 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641478561 0 0) 0x7f3eb80b3430 con 0x7f3eb80d27f0 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.757+0000 7f3eaffff640 1 -- 172.21.15.77:0/3151418107 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea4003520 con 0x7f3eb80d27f0 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.757+0000 7f3ebcb2e640 1 -- 172.21.15.77:0/3151418107 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb815dfe0 msgr2=0x7f3eb80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.757+0000 7f3ebcb2e640 1 --2- 172.21.15.77:0/3151418107 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb815dfe0 0x7f3eb80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.757+0000 7f3ebcb2e640 1 -- 172.21.15.77:0/3151418107 >> v1:172.21.15.77:6789/0 conn(0x7f3eb80d27f0 legacy=0x7f3eb80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.757+0000 7f3ebcb2e640 1 -- 172.21.15.77:0/3151418107 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80b3710 con 0x7f3eb80cf830 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.757+0000 7f3ebd32f640 1 --2- 172.21.15.77:0/3151418107 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb815dfe0 0x7f3eb80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebcb2e640 1 --2- 172.21.15.77:0/3151418107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb80cf830 0x7f3eb80d22b0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40097a0 tx=0x7f3eb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc899208a582444f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3eaffff640 1 -- 172.21.15.77:0/3151418107 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4035020 con 0x7f3eb80cf830 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3eaffff640 1 -- 172.21.15.77:0/3151418107 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb4031830 con 0x7f3eb80cf830 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3eaffff640 1 -- 172.21.15.77:0/3151418107 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4031c00 con 0x7f3eb80cf830 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebf593640 1 -- 172.21.15.77:0/3151418107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb80cf830 msgr2=0x7f3eb80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebf593640 1 --2- 172.21.15.77:0/3151418107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb80cf830 0x7f3eb80d22b0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40097a0 tx=0x7f3eb402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebf593640 1 -- 172.21.15.77:0/3151418107 shutdown_connections 2024-03-06T07:32:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebf593640 1 --2- 172.21.15.77:0/3151418107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb80cf830 0x7f3eb80d22b0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebf593640 1 --2- 172.21.15.77:0/3151418107 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb815dfe0 0x7f3eb80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebf593640 1 -- 172.21.15.77:0/3151418107 >> 172.21.15.77:0/3151418107 conn(0x7f3eb80d3cd0 msgr2=0x7f3eb80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebf593640 1 -- 172.21.15.77:0/3151418107 shutdown_connections 2024-03-06T07:32:37.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.758+0000 7f3ebf593640 1 -- 172.21.15.77:0/3151418107 wait complete. 2024-03-06T07:32:37.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.759+0000 7f3ebf593640 1 Processor -- start 2024-03-06T07:32:37.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.759+0000 7f3ebf593640 1 -- start start 2024-03-06T07:32:37.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebf593640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb80cf830 0x7f3eb81f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebf593640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb80d27f0 0x7f3eb80cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebf593640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb815dfe0 0x7f3eb815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebf593640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80d1c70 con 0x7f3eb815dfe0 2024-03-06T07:32:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebf593640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b3d60 con 0x7f3eb80d27f0 2024-03-06T07:32:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebf593640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80d1df0 con 0x7f3eb80cf830 2024-03-06T07:32:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebdb30640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb815dfe0 0x7f3eb815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebdb30640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb815dfe0 0x7f3eb815c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57050/0 (socket says 172.21.15.77:57050) 2024-03-06T07:32:37.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebdb30640 1 -- 172.21.15.77:0/2337697892 learned_addr learned my addr 172.21.15.77:0/2337697892 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebd32f640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb80cf830 0x7f3eb81f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.760+0000 7f3ebcb2e640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb80d27f0 0x7f3eb80cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3ebdb30640 1 -- 172.21.15.77:0/2337697892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb80cf830 msgr2=0x7f3eb81f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3ebdb30640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb80cf830 0x7f3eb81f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3ebdb30640 1 -- 172.21.15.77:0/2337697892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb80d27f0 msgr2=0x7f3eb80cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3ebdb30640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb80d27f0 0x7f3eb80cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3ebdb30640 1 -- 172.21.15.77:0/2337697892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80cdc90 con 0x7f3eb815dfe0 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3ebdb30640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb815dfe0 0x7f3eb815c260 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000cb70 tx=0x7f3eb000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3eadffb640 1 -- 172.21.15.77:0/2337697892 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0017070 con 0x7f3eb815dfe0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3eadffb640 1 -- 172.21.15.77:0/2337697892 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb000ad40 con 0x7f3eb815dfe0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb80caff0 con 0x7f3eb815dfe0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3eadffb640 1 -- 172.21.15.77:0/2337697892 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0012620 con 0x7f3eb815dfe0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.761+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb80cb5b0 con 0x7f3eb815dfe0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.762+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb804f330 con 0x7f3eb815dfe0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.763+0000 7f3eadffb640 1 -- 172.21.15.77:0/2337697892 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3eb001a070 con 0x7f3eb815dfe0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.763+0000 7f3eadffb640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9c06b500 0x7f3e9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.763+0000 7f3eadffb640 1 -- 172.21.15.77:0/2337697892 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3eb008a040 con 0x7f3eb815dfe0 2024-03-06T07:32:37.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.763+0000 7f3ebd32f640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9c06b500 0x7f3e9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.763+0000 7f3ebd32f640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9c06b500 0x7f3e9c06d9c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80ca150 tx=0x7f3ea8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.767+0000 7f3eadffb640 1 -- 172.21.15.77:0/2337697892 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3eb80cdc90 con 0x7f3eb815dfe0 2024-03-06T07:32:37.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.975+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eb80cec30 con 0x7f3eb815dfe0 2024-03-06T07:32:37.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.975+0000 7f3eadffb640 1 -- 172.21.15.77:0/2337697892 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f3eb0058ea0 con 0x7f3eb815dfe0 2024-03-06T07:32:37.977 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:37.977 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:37.977 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:37.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9c06b500 msgr2=0x7f3e9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9c06b500 0x7f3e9c06d9c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80ca150 tx=0x7f3ea8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb815dfe0 msgr2=0x7f3eb815c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb815dfe0 0x7f3eb815c260 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000cb70 tx=0x7f3eb000a430 comp rx=0 tx=0).stop 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 shutdown_connections 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3e9c06b500 0x7f3e9c06d9c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb815dfe0 0x7f3eb815c260 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3eb80d27f0 0x7f3eb80cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 --2- 172.21.15.77:0/2337697892 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3eb80cf830 0x7f3eb81f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 >> 172.21.15.77:0/2337697892 conn(0x7f3eb80d3cd0 msgr2=0x7f3eb80b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 shutdown_connections 2024-03-06T07:32:37.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:37.977+0000 7f3ebf593640 1 -- 172.21.15.77:0/2337697892 wait complete. 2024-03-06T07:32:38.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:38.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:39.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.295+0000 7f64b96f4640 1 Processor -- start 2024-03-06T07:32:39.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.295+0000 7f64b96f4640 1 -- start start 2024-03-06T07:32:39.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.296+0000 7f64b96f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64b40d0a80 0x7f64b40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.296+0000 7f64b96f4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40d25d0 con 0x7f64b40d4760 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.296+0000 7f64b96f4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40d2870 con 0x7f64b40b4170 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.296+0000 7f64b96f4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b40d4bf0 con 0x7f64b40d0a80 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.296+0000 7f64b37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64b40d0a80 0x7f64b40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.296+0000 7f64b27fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f64b40d4760 0x7f64b40cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43968/0 (socket says 172.21.15.77:43968) 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.296+0000 7f64b27fc640 1 -- 172.21.15.77:0/2415940763 learned_addr learned my addr 172.21.15.77:0/2415940763 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.297+0000 7f64b37fe640 1 -- 172.21.15.77:0/2415940763 >> v1:172.21.15.158:6789/0 conn(0x7f64b40b4170 legacy=0x7f64b40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.297+0000 7f64b37fe640 1 -- 172.21.15.77:0/2415940763 >> v1:172.21.15.77:6789/0 conn(0x7f64b40d4760 legacy=0x7f64b40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.297+0000 7f64b37fe640 1 -- 172.21.15.77:0/2415940763 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b40d2b50 con 0x7f64b40d0a80 2024-03-06T07:32:39.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.297+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2415940763 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553591714 0 0) 0x7f64b40d25d0 con 0x7f64b40d4760 2024-03-06T07:32:39.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.297+0000 7f64b37fe640 1 --2- 172.21.15.77:0/2415940763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40d0a80 0x7f64b40b6a90 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009d90 tx=0x7f64ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be168e447d54fd79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.297+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2415940763 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac039070 con 0x7f64b40d0a80 2024-03-06T07:32:39.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.297+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2415940763 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac02cc60 con 0x7f64b40d0a80 2024-03-06T07:32:39.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.298+0000 7f64b1ffb640 1 -- 172.21.15.77:0/2415940763 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac034660 con 0x7f64b40d0a80 2024-03-06T07:32:39.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.298+0000 7f64b96f4640 1 -- 172.21.15.77:0/2415940763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40d0a80 msgr2=0x7f64b40b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.298+0000 7f64b96f4640 1 --2- 172.21.15.77:0/2415940763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40d0a80 0x7f64b40b6a90 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009d90 tx=0x7f64ac02beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.298+0000 7f64b96f4640 1 -- 172.21.15.77:0/2415940763 shutdown_connections 2024-03-06T07:32:39.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.298+0000 7f64b96f4640 1 --2- 172.21.15.77:0/2415940763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40d0a80 0x7f64b40b6a90 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.298+0000 7f64b96f4640 1 -- 172.21.15.77:0/2415940763 >> 172.21.15.77:0/2415940763 conn(0x7f64b40d3c40 msgr2=0x7f64b40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.298+0000 7f64b96f4640 1 -- 172.21.15.77:0/2415940763 shutdown_connections 2024-03-06T07:32:39.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.298+0000 7f64b96f4640 1 -- 172.21.15.77:0/2415940763 wait complete. 2024-03-06T07:32:39.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.299+0000 7f64b96f4640 1 Processor -- start 2024-03-06T07:32:39.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.299+0000 7f64b96f4640 1 -- start start 2024-03-06T07:32:39.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b96f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40b4170 0x7f64b415a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b96f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64b40d0a80 0x7f64b40b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b96f4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64b40b2f50 0x7f64b40cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b96f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40b4dd0 con 0x7f64b40b4170 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b96f4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40ce9e0 con 0x7f64b40b2f50 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b96f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40b4f50 con 0x7f64b40d0a80 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40b4170 0x7f64b415a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40b4170 0x7f64b415a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44286/0 (socket says 172.21.15.77:44286) 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b2ffd640 1 -- 172.21.15.77:0/256664595 learned_addr learned my addr 172.21.15.77:0/256664595 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b27fc640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64b40d0a80 0x7f64b40b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b37fe640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64b40b2f50 0x7f64b40cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b2ffd640 1 -- 172.21.15.77:0/256664595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64b40d0a80 msgr2=0x7f64b40b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b2ffd640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64b40d0a80 0x7f64b40b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b2ffd640 1 -- 172.21.15.77:0/256664595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64b40b2f50 msgr2=0x7f64b40cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b2ffd640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64b40b2f50 0x7f64b40cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.300+0000 7f64b2ffd640 1 -- 172.21.15.77:0/256664595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b40cc410 con 0x7f64b40b4170 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.301+0000 7f64b2ffd640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40b4170 0x7f64b415a620 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c430 tx=0x7f649c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.301+0000 7f64ab7fe640 1 -- 172.21.15.77:0/256664595 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c011790 con 0x7f64b40b4170 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.301+0000 7f64ab7fe640 1 -- 172.21.15.77:0/256664595 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c017070 con 0x7f64b40b4170 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.301+0000 7f64ab7fe640 1 -- 172.21.15.77:0/256664595 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c011bb0 con 0x7f64b40b4170 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.301+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b40b0920 con 0x7f64b40b4170 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.301+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b40cbc60 con 0x7f64b40b4170 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.302+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b404f330 con 0x7f64b40b4170 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.302+0000 7f64ab7fe640 1 -- 172.21.15.77:0/256664595 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f649c01c460 con 0x7f64b40b4170 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.303+0000 7f64ab7fe640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f649406b4d0 0x7f649406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.303+0000 7f64ab7fe640 1 -- 172.21.15.77:0/256664595 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f649c08a080 con 0x7f64b40b4170 2024-03-06T07:32:39.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.303+0000 7f64b27fc640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f649406b4d0 0x7f649406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.303+0000 7f64b27fc640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f649406b4d0 0x7f649406d990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f64b4069f30 tx=0x7f64a4007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.307+0000 7f64ab7fe640 1 -- 172.21.15.77:0/256664595 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f649c058f70 con 0x7f64b40b4170 2024-03-06T07:32:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.556+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64b408dd20 con 0x7f64b40b4170 2024-03-06T07:32:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.557+0000 7f64ab7fe640 1 -- 172.21.15.77:0/256664595 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f649c058870 con 0x7f64b40b4170 2024-03-06T07:32:39.559 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:39.559 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:39.559 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.558+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f649406b4d0 msgr2=0x7f649406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f649406b4d0 0x7f649406d990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f64b4069f30 tx=0x7f64a4007c00 comp rx=0 tx=0).stop 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40b4170 msgr2=0x7f64b415a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40b4170 0x7f64b415a620 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c430 tx=0x7f649c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 shutdown_connections 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64b40b2f50 0x7f64b40cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64b40d0a80 0x7f64b40b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f649406b4d0 0x7f649406d990 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 --2- 172.21.15.77:0/256664595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64b40b4170 0x7f64b415a620 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 >> 172.21.15.77:0/256664595 conn(0x7f64b40d3c40 msgr2=0x7f64b40b7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 shutdown_connections 2024-03-06T07:32:39.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:39.559+0000 7f64b96f4640 1 -- 172.21.15.77:0/256664595 wait complete. 2024-03-06T07:32:39.773 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:39.773 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:40.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:40.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.874+0000 7fb53530c640 1 Processor -- start 2024-03-06T07:32:40.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.874+0000 7fb53530c640 1 -- start start 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.875+0000 7fb53530c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5300d22b0 0x7fb5300d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.875+0000 7fb53530c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5300b41f0 0x7fb5300b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.875+0000 7fb53530c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5300b3430 con 0x7fb53015dfe0 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.875+0000 7fb53530c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5300d2c50 con 0x7fb5300d22b0 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.875+0000 7fb53530c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5300b31e0 con 0x7fb5300b41f0 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.876+0000 7fb52e7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5300d22b0 0x7fb5300d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.876+0000 7fb52e7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5300d22b0 0x7fb5300d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44290/0 (socket says 172.21.15.77:44290) 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.876+0000 7fb52e7fc640 1 -- 172.21.15.77:0/3897896824 learned_addr learned my addr 172.21.15.77:0/3897896824 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.876+0000 7fb52f7fe640 1 --2- 172.21.15.77:0/3897896824 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5300b41f0 0x7fb5300b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.876+0000 7fb52e7fc640 1 -- 172.21.15.77:0/3897896824 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5300b41f0 msgr2=0x7fb5300b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.876+0000 7fb52e7fc640 1 --2- 172.21.15.77:0/3897896824 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5300b41f0 0x7fb5300b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.876+0000 7fb52e7fc640 1 -- 172.21.15.77:0/3897896824 >> v1:172.21.15.158:6789/0 conn(0x7fb53015dfe0 legacy=0x7fb5300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.876+0000 7fb52e7fc640 1 -- 172.21.15.77:0/3897896824 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5300b3710 con 0x7fb5300d22b0 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb52e7fc640 1 --2- 172.21.15.77:0/3897896824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300d22b0 0x7fb5300d2710 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fb524009870 tx=0x7fb52402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdaefffb61ad53b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb52dffb640 1 -- 172.21.15.77:0/3897896824 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb524034070 con 0x7fb5300d22b0 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb52dffb640 1 -- 172.21.15.77:0/3897896824 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb524030840 con 0x7fb5300d22b0 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb52dffb640 1 -- 172.21.15.77:0/3897896824 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb524030c10 con 0x7fb5300d22b0 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb53530c640 1 -- 172.21.15.77:0/3897896824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300d22b0 msgr2=0x7fb5300d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb53530c640 1 --2- 172.21.15.77:0/3897896824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300d22b0 0x7fb5300d2710 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fb524009870 tx=0x7fb52402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb53530c640 1 -- 172.21.15.77:0/3897896824 shutdown_connections 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb53530c640 1 --2- 172.21.15.77:0/3897896824 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5300b41f0 0x7fb5300b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb53530c640 1 --2- 172.21.15.77:0/3897896824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300d22b0 0x7fb5300d2710 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb53530c640 1 -- 172.21.15.77:0/3897896824 >> 172.21.15.77:0/3897896824 conn(0x7fb5300d3cd0 msgr2=0x7fb5300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb53530c640 1 -- 172.21.15.77:0/3897896824 shutdown_connections 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.877+0000 7fb53530c640 1 -- 172.21.15.77:0/3897896824 wait complete. 2024-03-06T07:32:40.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb53530c640 1 Processor -- start 2024-03-06T07:32:40.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb53530c640 1 -- start start 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb53530c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300b41f0 0x7fb5300b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb53530c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5300d22b0 0x7fb5300cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb53530c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb53015dfe0 0x7fb530147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb53530c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5300cfa00 con 0x7fb5300b41f0 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb53530c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5300b3d60 con 0x7fb5300d22b0 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb53530c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5300d1c70 con 0x7fb53015dfe0 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb52effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300b41f0 0x7fb5300b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb52effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300b41f0 0x7fb5300b6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44302/0 (socket says 172.21.15.77:44302) 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.879+0000 7fb52effd640 1 -- 172.21.15.77:0/2726999692 learned_addr learned my addr 172.21.15.77:0/2726999692 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52e7fc640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5300d22b0 0x7fb5300cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52f7fe640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb53015dfe0 0x7fb530147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52effd640 1 -- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb53015dfe0 msgr2=0x7fb530147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52effd640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb53015dfe0 0x7fb530147ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52effd640 1 -- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5300d22b0 msgr2=0x7fb5300cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52effd640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5300d22b0 0x7fb5300cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52effd640 1 -- 172.21.15.77:0/2726999692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5300cc760 con 0x7fb5300b41f0 2024-03-06T07:32:40.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52e7fc640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5300d22b0 0x7fb5300cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:40.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52f7fe640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb53015dfe0 0x7fb530147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:40.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb52effd640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300b41f0 0x7fb5300b6b10 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fb52000c400 tx=0x7fb52000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb5177fe640 1 -- 172.21.15.77:0/2726999692 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520017070 con 0x7fb5300b41f0 2024-03-06T07:32:40.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb5177fe640 1 -- 172.21.15.77:0/2726999692 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5200052d0 con 0x7fb5300b41f0 2024-03-06T07:32:40.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5300cca40 con 0x7fb5300b41f0 2024-03-06T07:32:40.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb5177fe640 1 -- 172.21.15.77:0/2726999692 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520003830 con 0x7fb5300b41f0 2024-03-06T07:32:40.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.880+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5300cd000 con 0x7fb5300b41f0 2024-03-06T07:32:40.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.881+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5300d3a70 con 0x7fb5300b41f0 2024-03-06T07:32:40.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.882+0000 7fb5177fe640 1 -- 172.21.15.77:0/2726999692 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb52001a070 con 0x7fb5300b41f0 2024-03-06T07:32:40.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.882+0000 7fb5177fe640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb51c06b4d0 0x7fb51c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.882+0000 7fb5177fe640 1 -- 172.21.15.77:0/2726999692 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb5200895d0 con 0x7fb5300b41f0 2024-03-06T07:32:40.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.882+0000 7fb52e7fc640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb51c06b4d0 0x7fb51c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.883+0000 7fb52e7fc640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb51c06b4d0 0x7fb51c06d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb524036000 tx=0x7fb52403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:40.886+0000 7fb5177fe640 1 -- 172.21.15.77:0/2726999692 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5300d3a70 con 0x7fb5300b41f0 2024-03-06T07:32:41.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.138+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb53004f330 con 0x7fb5300b41f0 2024-03-06T07:32:41.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.138+0000 7fb5177fe640 1 -- 172.21.15.77:0/2726999692 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fb5200583b0 con 0x7fb5300b41f0 2024-03-06T07:32:41.141 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:41.141 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:41.141 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:41.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb51c06b4d0 msgr2=0x7fb51c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb51c06b4d0 0x7fb51c06d990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb524036000 tx=0x7fb52403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300b41f0 msgr2=0x7fb5300b6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300b41f0 0x7fb5300b6b10 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fb52000c400 tx=0x7fb52000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 shutdown_connections 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb53015dfe0 0x7fb530147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5300d22b0 0x7fb5300cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb51c06b4d0 0x7fb51c06d990 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 --2- 172.21.15.77:0/2726999692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5300b41f0 0x7fb5300b6b10 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.140+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 >> 172.21.15.77:0/2726999692 conn(0x7fb5300d3cd0 msgr2=0x7fb5300cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.141+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 shutdown_connections 2024-03-06T07:32:41.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:41.141+0000 7fb53530c640 1 -- 172.21.15.77:0/2726999692 wait complete. 2024-03-06T07:32:41.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:41.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:42.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:42.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.457+0000 7ff0e4edf640 1 Processor -- start 2024-03-06T07:32:42.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.457+0000 7ff0e4edf640 1 -- start start 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.458+0000 7ff0e4edf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 0x7ff0e00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.458+0000 7ff0e4edf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0e00cd7a0 0x7ff0e00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.458+0000 7ff0e4edf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e0071f40 con 0x7ff0e015b430 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.458+0000 7ff0e4edf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e015b8b0 con 0x7ff0e00cd7a0 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.458+0000 7ff0e4edf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e0070a70 con 0x7ff0e01515e0 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.458+0000 7ff0ded9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0e00cd7a0 0x7ff0e00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.458+0000 7ff0ddd9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff0e015b430 0x7ff0e0071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43976/0 (socket says 172.21.15.77:43976) 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.458+0000 7ff0ded9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0e00cd7a0 0x7ff0e00cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44304/0 (socket says 172.21.15.77:44304) 2024-03-06T07:32:42.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.459+0000 7ff0ddd9b640 1 -- 172.21.15.77:0/4206522583 learned_addr learned my addr 172.21.15.77:0/4206522583 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:42.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.459+0000 7ff0ded9d640 1 -- 172.21.15.77:0/4206522583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 msgr2=0x7ff0e00cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:42.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.459+0000 7ff0de59c640 1 --2- 172.21.15.77:0/4206522583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 0x7ff0e00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.459+0000 7ff0ded9d640 1 --2- 172.21.15.77:0/4206522583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 0x7ff0e00cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.459+0000 7ff0ded9d640 1 -- 172.21.15.77:0/4206522583 >> v1:172.21.15.77:6789/0 conn(0x7ff0e015b430 legacy=0x7ff0e0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.459+0000 7ff0ded9d640 1 -- 172.21.15.77:0/4206522583 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e0070d50 con 0x7ff0e00cd7a0 2024-03-06T07:32:42.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.459+0000 7ff0dd59a640 1 -- 172.21.15.77:0/4206522583 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845771414 0 0) 0x7ff0e0071f40 con 0x7ff0e015b430 2024-03-06T07:32:42.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0ded9d640 1 --2- 172.21.15.77:0/4206522583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e00cd7a0 0x7ff0e00cdbd0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009d90 tx=0x7ff0d802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbe35316436839a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0dd59a640 1 -- 172.21.15.77:0/4206522583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8039070 con 0x7ff0e00cd7a0 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0dd59a640 1 -- 172.21.15.77:0/4206522583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d802cc50 con 0x7ff0e00cd7a0 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0dd59a640 1 -- 172.21.15.77:0/4206522583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8034440 con 0x7ff0e00cd7a0 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0e4edf640 1 -- 172.21.15.77:0/4206522583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e00cd7a0 msgr2=0x7ff0e00cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/4206522583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e00cd7a0 0x7ff0e00cdbd0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009d90 tx=0x7ff0d802bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0e4edf640 1 -- 172.21.15.77:0/4206522583 shutdown_connections 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/4206522583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e00cd7a0 0x7ff0e00cdbd0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/4206522583 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 0x7ff0e00cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0e4edf640 1 -- 172.21.15.77:0/4206522583 >> 172.21.15.77:0/4206522583 conn(0x7ff0e00cf8a0 msgr2=0x7ff0e00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0e4edf640 1 -- 172.21.15.77:0/4206522583 shutdown_connections 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.460+0000 7ff0e4edf640 1 -- 172.21.15.77:0/4206522583 wait complete. 2024-03-06T07:32:42.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0e4edf640 1 Processor -- start 2024-03-06T07:32:42.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0e4edf640 1 -- start start 2024-03-06T07:32:42.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0e4edf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0e00cd7a0 0x7ff0e014a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0e4edf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 0x7ff0e01f4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0e4edf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e015b430 0x7ff0e00721b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0e4edf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00ce110 con 0x7ff0e015b430 2024-03-06T07:32:42.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0e4edf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e0071310 con 0x7ff0e00cd7a0 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0e4edf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00cc3a0 con 0x7ff0e01515e0 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.462+0000 7ff0ded9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e015b430 0x7ff0e00721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ded9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e015b430 0x7ff0e00721b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44320/0 (socket says 172.21.15.77:44320) 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ded9d640 1 -- 172.21.15.77:0/1296126275 learned_addr learned my addr 172.21.15.77:0/1296126275 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ddd9b640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 0x7ff0e01f4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0de59c640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0e00cd7a0 0x7ff0e014a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ded9d640 1 -- 172.21.15.77:0/1296126275 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 msgr2=0x7ff0e01f4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ded9d640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 0x7ff0e01f4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ded9d640 1 -- 172.21.15.77:0/1296126275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0e00cd7a0 msgr2=0x7ff0e014a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ded9d640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0e00cd7a0 0x7ff0e014a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ded9d640 1 -- 172.21.15.77:0/1296126275 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e00c8e50 con 0x7ff0e015b430 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ded9d640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e015b430 0x7ff0e00721b0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009eb0 tx=0x7ff0d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ceffd640 1 -- 172.21.15.77:0/1296126275 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8039050 con 0x7ff0e015b430 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ceffd640 1 -- 172.21.15.77:0/1296126275 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d8045070 con 0x7ff0e015b430 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e00c9130 con 0x7ff0e015b430 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0ceffd640 1 -- 172.21.15.77:0/1296126275 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d803e610 con 0x7ff0e015b430 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.463+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e014a990 con 0x7ff0e015b430 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.464+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e0068840 con 0x7ff0e015b430 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.465+0000 7ff0ceffd640 1 -- 172.21.15.77:0/1296126275 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0d803e7b0 con 0x7ff0e015b430 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.465+0000 7ff0ceffd640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0c006b4d0 0x7ff0c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.465+0000 7ff0de59c640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0c006b4d0 0x7ff0c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.465+0000 7ff0ceffd640 1 -- 172.21.15.77:0/1296126275 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff0d80ad000 con 0x7ff0e015b430 2024-03-06T07:32:42.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.466+0000 7ff0de59c640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0c006b4d0 0x7ff0c006d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000aac0 tx=0x7ff0d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.470+0000 7ff0ceffd640 1 -- 172.21.15.77:0/1296126275 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0d807b930 con 0x7ff0e015b430 2024-03-06T07:32:42.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.804+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0e004f330 con 0x7ff0e015b430 2024-03-06T07:32:42.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.805+0000 7ff0ceffd640 1 -- 172.21.15.77:0/1296126275 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7ff0d807f9d0 con 0x7ff0e015b430 2024-03-06T07:32:42.807 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:42.807 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:42.807 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:42.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0c006b4d0 msgr2=0x7ff0c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0c006b4d0 0x7ff0c006d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000aac0 tx=0x7ff0d0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:42.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e015b430 msgr2=0x7ff0e00721b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e015b430 0x7ff0e00721b0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009eb0 tx=0x7ff0d8004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:42.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 shutdown_connections 2024-03-06T07:32:42.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ff0c006b4d0 0x7ff0c006d990 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e015b430 0x7ff0e00721b0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ff0e01515e0 0x7ff0e01f4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 --2- 172.21.15.77:0/1296126275 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff0e00cd7a0 0x7ff0e014a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 >> 172.21.15.77:0/1296126275 conn(0x7ff0e00cf8a0 msgr2=0x7ff0e014d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 shutdown_connections 2024-03-06T07:32:42.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:42.807+0000 7ff0e4edf640 1 -- 172.21.15.77:0/1296126275 wait complete. 2024-03-06T07:32:43.021 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:43.021 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:44.022 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.122+0000 7f2b0aeb5640 1 Processor -- start 2024-03-06T07:32:44.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.122+0000 7f2b0aeb5640 1 -- start start 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2b0aeb5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2b0415dfe0 0x7f2b040cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2b0aeb5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2b040cf830 0x7f2b040d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2b0aeb5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b040b3430 con 0x7f2b040d27f0 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2b0aeb5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b040b31e0 con 0x7f2b040cf830 2024-03-06T07:32:44.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2b0aeb5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b040b3d60 con 0x7f2b0415dfe0 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2afbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2b040cf830 0x7f2b040d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2afbfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2b040cf830 0x7f2b040d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44336/0 (socket says 172.21.15.77:44336) 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2afbfff640 1 -- 172.21.15.77:0/1195515725 learned_addr learned my addr 172.21.15.77:0/1195515725 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2b09452640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2b040d27f0 0x7f2b040d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43986/0 (socket says 172.21.15.77:43986) 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2b08c51640 1 --2- 172.21.15.77:0/1195515725 >> v2:172.21.15.158:3300/0 conn(0x7f2b0415dfe0 0x7f2b040cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2afbfff640 1 -- 172.21.15.77:0/1195515725 >> v2:172.21.15.158:3300/0 conn(0x7f2b0415dfe0 msgr2=0x7f2b040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2afbfff640 1 --2- 172.21.15.77:0/1195515725 >> v2:172.21.15.158:3300/0 conn(0x7f2b0415dfe0 0x7f2b040cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2afbfff640 1 -- 172.21.15.77:0/1195515725 >> v1:172.21.15.77:6789/0 conn(0x7f2b040d27f0 legacy=0x7f2b040d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2afbfff640 1 -- 172.21.15.77:0/1195515725 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b040b3710 con 0x7f2b040cf830 2024-03-06T07:32:44.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.123+0000 7f2afb7fe640 1 -- 172.21.15.77:0/1195515725 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180772295 0 0) 0x7f2b040b3430 con 0x7f2b040d27f0 2024-03-06T07:32:44.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.124+0000 7f2afbfff640 1 --2- 172.21.15.77:0/1195515725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b040cf830 0x7f2b040d22b0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009870 tx=0x7f2af402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c4a40b0ec92074e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.124+0000 7f2afb7fe640 1 -- 172.21.15.77:0/1195515725 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4034070 con 0x7f2b040cf830 2024-03-06T07:32:44.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.124+0000 7f2afb7fe640 1 -- 172.21.15.77:0/1195515725 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af4030860 con 0x7f2b040cf830 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.124+0000 7f2afb7fe640 1 -- 172.21.15.77:0/1195515725 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4030c30 con 0x7f2b040cf830 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.125+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/1195515725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b040cf830 msgr2=0x7f2b040d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.125+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/1195515725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b040cf830 0x7f2b040d22b0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009870 tx=0x7f2af402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.125+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/1195515725 shutdown_connections 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.125+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/1195515725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b040cf830 0x7f2b040d22b0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.125+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/1195515725 >> v2:172.21.15.158:3300/0 conn(0x7f2b0415dfe0 0x7f2b040cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.125+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/1195515725 >> 172.21.15.77:0/1195515725 conn(0x7f2b040d3cd0 msgr2=0x7f2b040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.125+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/1195515725 shutdown_connections 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.125+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/1195515725 wait complete. 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.126+0000 7f2b0aeb5640 1 Processor -- start 2024-03-06T07:32:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.126+0000 7f2b0aeb5640 1 -- start start 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b0aeb5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b040cf830 0x7f2b0415c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b0aeb5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2b040d27f0 0x7f2b040b8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b0aeb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b0415dfe0 0x7f2b040b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b0aeb5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b040d1df0 con 0x7f2b0415dfe0 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b0aeb5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b040d1c70 con 0x7f2b040cf830 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b0aeb5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0415e3c0 con 0x7f2b040d27f0 2024-03-06T07:32:44.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b0415dfe0 0x7f2b040b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b0415dfe0 0x7f2b040b8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44350/0 (socket says 172.21.15.77:44350) 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 -- 172.21.15.77:0/94965609 learned_addr learned my addr 172.21.15.77:0/94965609 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b08c51640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b040cf830 0x7f2b0415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2afbfff640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2b040d27f0 0x7f2b040b8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 -- 172.21.15.77:0/94965609 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2b040d27f0 msgr2=0x7f2b040b8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2b040d27f0 0x7f2b040b8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 -- 172.21.15.77:0/94965609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b040cf830 msgr2=0x7f2b0415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b040cf830 0x7f2b0415c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 -- 172.21.15.77:0/94965609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0415f730 con 0x7f2b0415dfe0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.127+0000 7f2b09452640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b0415dfe0 0x7f2b040b8770 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00c7d0 tx=0x7f2afc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.128+0000 7f2af97fa640 1 -- 172.21.15.77:0/94965609 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc018070 con 0x7f2b0415dfe0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.128+0000 7f2af97fa640 1 -- 172.21.15.77:0/94965609 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2afc00a640 con 0x7f2b0415dfe0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.128+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0415fa10 con 0x7f2b0415dfe0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.128+0000 7f2af97fa640 1 -- 172.21.15.77:0/94965609 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc00aa30 con 0x7f2b0415dfe0 2024-03-06T07:32:44.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.128+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0415ffd0 con 0x7f2b0415dfe0 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.129+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0404f330 con 0x7f2b0415dfe0 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.129+0000 7f2af97fa640 1 -- 172.21.15.77:0/94965609 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2afc01b070 con 0x7f2b0415dfe0 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.130+0000 7f2af97fa640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2af006b4d0 0x7f2af006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.130+0000 7f2b08c51640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2af006b4d0 0x7f2af006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.130+0000 7f2af97fa640 1 -- 172.21.15.77:0/94965609 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2afc0897a0 con 0x7f2b0415dfe0 2024-03-06T07:32:44.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.130+0000 7f2b08c51640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2af006b4d0 0x7f2af006d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2b040ca830 tx=0x7f2b00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.134+0000 7f2af97fa640 1 -- 172.21.15.77:0/94965609 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2afc058690 con 0x7f2b0415dfe0 2024-03-06T07:32:44.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.350+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b0415d240 con 0x7f2b0415dfe0 2024-03-06T07:32:44.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.350+0000 7f2af97fa640 1 -- 172.21.15.77:0/94965609 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f2afc057f90 con 0x7f2b0415dfe0 2024-03-06T07:32:44.352 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:44.352 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:44.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2af006b4d0 msgr2=0x7f2af006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2af006b4d0 0x7f2af006d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2b040ca830 tx=0x7f2b00008040 comp rx=0 tx=0).stop 2024-03-06T07:32:44.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b0415dfe0 msgr2=0x7f2b040b8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b0415dfe0 0x7f2b040b8770 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00c7d0 tx=0x7f2afc00cc90 comp rx=0 tx=0).stop 2024-03-06T07:32:44.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 shutdown_connections 2024-03-06T07:32:44.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2af006b4d0 0x7f2af006d990 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b0415dfe0 0x7f2b040b8770 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2b040d27f0 0x7f2b040b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 --2- 172.21.15.77:0/94965609 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2b040cf830 0x7f2b0415c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 >> 172.21.15.77:0/94965609 conn(0x7f2b040d3cd0 msgr2=0x7f2b0406ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 shutdown_connections 2024-03-06T07:32:44.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:44.352+0000 7f2b0aeb5640 1 -- 172.21.15.77:0/94965609 wait complete. 2024-03-06T07:32:44.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:44.566 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:45.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:45.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.670+0000 7f7411fac640 1 Processor -- start 2024-03-06T07:32:45.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.670+0000 7f7411fac640 1 -- start start 2024-03-06T07:32:45.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.670+0000 7f7411fac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c0d22b0 0x7f740c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.670+0000 7f7411fac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f740c0b41f0 0x7f740c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.670+0000 7f7411fac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740c0b3430 con 0x7f740c15dfe0 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.670+0000 7f7411fac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740c0d2c50 con 0x7f740c0b41f0 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.670+0000 7f7411fac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740c0b31e0 con 0x7f740c0d22b0 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.670+0000 7f740bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f740c0b41f0 0x7f740c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.671+0000 7f740bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f740c0b41f0 0x7f740c0b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44352/0 (socket says 172.21.15.77:44352) 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.671+0000 7f740bfff640 1 -- 172.21.15.77:0/1766401066 learned_addr learned my addr 172.21.15.77:0/1766401066 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.671+0000 7f740affd640 1 --2- 172.21.15.77:0/1766401066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c0d22b0 0x7f740c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.671+0000 7f740bfff640 1 -- 172.21.15.77:0/1766401066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c0d22b0 msgr2=0x7f740c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.671+0000 7f740bfff640 1 --2- 172.21.15.77:0/1766401066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c0d22b0 0x7f740c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.671+0000 7f740bfff640 1 -- 172.21.15.77:0/1766401066 >> v1:172.21.15.158:6789/0 conn(0x7f740c15dfe0 legacy=0x7f740c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.671+0000 7f740bfff640 1 -- 172.21.15.77:0/1766401066 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c0b3710 con 0x7f740c0b41f0 2024-03-06T07:32:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.671+0000 7f740affd640 1 --2- 172.21.15.77:0/1766401066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c0d22b0 0x7f740c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f740bfff640 1 --2- 172.21.15.77:0/1766401066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0b41f0 0x7f740c0b4650 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f7404009e50 tx=0x7f740402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1d3778c7d8ba5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f740a7fc640 1 -- 172.21.15.77:0/1766401066 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740402cb00 con 0x7f740c0b41f0 2024-03-06T07:32:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f740a7fc640 1 -- 172.21.15.77:0/1766401066 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740402cca0 con 0x7f740c0b41f0 2024-03-06T07:32:45.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f740a7fc640 1 -- 172.21.15.77:0/1766401066 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7404034610 con 0x7f740c0b41f0 2024-03-06T07:32:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f7411fac640 1 -- 172.21.15.77:0/1766401066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0b41f0 msgr2=0x7f740c0b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f7411fac640 1 --2- 172.21.15.77:0/1766401066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0b41f0 0x7f740c0b4650 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f7404009e50 tx=0x7f740402bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f7411fac640 1 -- 172.21.15.77:0/1766401066 shutdown_connections 2024-03-06T07:32:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f7411fac640 1 --2- 172.21.15.77:0/1766401066 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c0d22b0 0x7f740c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f7411fac640 1 --2- 172.21.15.77:0/1766401066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0b41f0 0x7f740c0b4650 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.672+0000 7f7411fac640 1 -- 172.21.15.77:0/1766401066 >> 172.21.15.77:0/1766401066 conn(0x7f740c0d3cd0 msgr2=0x7f740c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.673+0000 7f7411fac640 1 -- 172.21.15.77:0/1766401066 shutdown_connections 2024-03-06T07:32:45.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.673+0000 7f7411fac640 1 -- 172.21.15.77:0/1766401066 wait complete. 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.674+0000 7f7411fac640 1 Processor -- start 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.674+0000 7f7411fac640 1 -- start start 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.674+0000 7f7411fac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f740c0b41f0 0x7f740c0cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.674+0000 7f7411fac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0d22b0 0x7f740c15c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.674+0000 7f7411fac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c15dfe0 0x7f740c1f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f7411fac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c0cfa00 con 0x7f740c0d22b0 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f7411fac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c0b3d60 con 0x7f740c0b41f0 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f7411fac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740c0d1c70 con 0x7f740c15dfe0 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0d22b0 0x7f740c15c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0d22b0 0x7f740c15c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44366/0 (socket says 172.21.15.77:44366) 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 -- 172.21.15.77:0/716753137 learned_addr learned my addr 172.21.15.77:0/716753137 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740b7fe640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f740c0b41f0 0x7f740c0cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740bfff640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c15dfe0 0x7f740c1f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 -- 172.21.15.77:0/716753137 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c15dfe0 msgr2=0x7f740c1f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c15dfe0 0x7f740c1f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 -- 172.21.15.77:0/716753137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f740c0b41f0 msgr2=0x7f740c0cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f740c0b41f0 0x7f740c0cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 -- 172.21.15.77:0/716753137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c0cada0 con 0x7f740c0d22b0 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f740affd640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0d22b0 0x7f740c15c1e0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f740000e810 tx=0x7f740000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f73f3fff640 1 -- 172.21.15.77:0/716753137 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740000f040 con 0x7f740c0d22b0 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f73f3fff640 1 -- 172.21.15.77:0/716753137 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7400004030 con 0x7f740c0d22b0 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740c1f4630 con 0x7f740c0d22b0 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f73f3fff640 1 -- 172.21.15.77:0/716753137 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400004400 con 0x7f740c0d22b0 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.675+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740c0cb340 con 0x7f740c0d22b0 2024-03-06T07:32:45.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.676+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740c15e690 con 0x7f740c0d22b0 2024-03-06T07:32:45.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.677+0000 7f73f3fff640 1 -- 172.21.15.77:0/716753137 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f740000c670 con 0x7f740c0d22b0 2024-03-06T07:32:45.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.677+0000 7f73f3fff640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f73f806b4d0 0x7f73f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.677+0000 7f73f3fff640 1 -- 172.21.15.77:0/716753137 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7400089f50 con 0x7f740c0d22b0 2024-03-06T07:32:45.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.677+0000 7f740b7fe640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f73f806b4d0 0x7f73f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.678+0000 7f740b7fe640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f73f806b4d0 0x7f73f806d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f740c0cc480 tx=0x7f73fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.681+0000 7f73f3fff640 1 -- 172.21.15.77:0/716753137 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7400017020 con 0x7f740c0d22b0 2024-03-06T07:32:45.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.898+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f740c04f330 con 0x7f740c0d22b0 2024-03-06T07:32:45.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.898+0000 7f73f3fff640 1 -- 172.21.15.77:0/716753137 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f74000586e0 con 0x7f740c0d22b0 2024-03-06T07:32:45.922 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:45.922 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f73f806b4d0 msgr2=0x7f73f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f73f806b4d0 0x7f73f806d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f740c0cc480 tx=0x7f73fc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0d22b0 msgr2=0x7f740c15c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0d22b0 0x7f740c15c1e0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f740000e810 tx=0x7f740000ecd0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 shutdown_connections 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f73f806b4d0 0x7f73f806d990 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f740c0d22b0 0x7f740c15c1e0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f740c15dfe0 0x7f740c1f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 --2- 172.21.15.77:0/716753137 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f740c0b41f0 0x7f740c0cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 >> 172.21.15.77:0/716753137 conn(0x7f740c0d3cd0 msgr2=0x7f740c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 shutdown_connections 2024-03-06T07:32:45.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:45.900+0000 7f7411fac640 1 -- 172.21.15.77:0/716753137 wait complete. 2024-03-06T07:32:46.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:46.128 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:47.129 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:47.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.228+0000 7fd93d53a640 1 Processor -- start 2024-03-06T07:32:47.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.228+0000 7fd93d53a640 1 -- start start 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.229+0000 7fd93d53a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9380b4170 0x7fd9380b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.229+0000 7fd93d53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380d4760 0x7fd9380d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.229+0000 7fd93d53a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380d25d0 con 0x7fd9380b4dd0 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.229+0000 7fd93d53a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9380d4bf0 con 0x7fd9380b4170 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.229+0000 7fd93d53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380ce9e0 con 0x7fd9380d4760 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.229+0000 7fd936ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9380b4170 0x7fd9380b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.229+0000 7fd936ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9380b4170 0x7fd9380b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44368/0 (socket says 172.21.15.77:44368) 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.229+0000 7fd936ffd640 1 -- 172.21.15.77:0/1937877187 learned_addr learned my addr 172.21.15.77:0/1937877187 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.230+0000 7fd92ffff640 1 --2- 172.21.15.77:0/1937877187 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380d4760 0x7fd9380d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.230+0000 7fd936ffd640 1 -- 172.21.15.77:0/1937877187 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380d4760 msgr2=0x7fd9380d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.230+0000 7fd936ffd640 1 --2- 172.21.15.77:0/1937877187 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380d4760 0x7fd9380d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.230+0000 7fd936ffd640 1 -- 172.21.15.77:0/1937877187 >> v1:172.21.15.158:6789/0 conn(0x7fd9380b4dd0 legacy=0x7fd9380cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.230+0000 7fd936ffd640 1 -- 172.21.15.77:0/1937877187 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380d2870 con 0x7fd9380b4170 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.230+0000 7fd936ffd640 1 --2- 172.21.15.77:0/1937877187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4170 0x7fd9380b6a90 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fd928009870 tx=0x7fd92802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e22ce6da5b88e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd9367fc640 1 -- 172.21.15.77:0/1937877187 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd928034070 con 0x7fd9380b4170 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd9367fc640 1 -- 172.21.15.77:0/1937877187 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd928030840 con 0x7fd9380b4170 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd9367fc640 1 -- 172.21.15.77:0/1937877187 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd928030c10 con 0x7fd9380b4170 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd93d53a640 1 -- 172.21.15.77:0/1937877187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4170 msgr2=0x7fd9380b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd93d53a640 1 --2- 172.21.15.77:0/1937877187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4170 0x7fd9380b6a90 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fd928009870 tx=0x7fd92802b980 comp rx=0 tx=0).stop 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd93d53a640 1 -- 172.21.15.77:0/1937877187 shutdown_connections 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd93d53a640 1 --2- 172.21.15.77:0/1937877187 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380d4760 0x7fd9380d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd93d53a640 1 --2- 172.21.15.77:0/1937877187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4170 0x7fd9380b6a90 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd93d53a640 1 -- 172.21.15.77:0/1937877187 >> 172.21.15.77:0/1937877187 conn(0x7fd9380d3c40 msgr2=0x7fd9380d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd93d53a640 1 -- 172.21.15.77:0/1937877187 shutdown_connections 2024-03-06T07:32:47.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.231+0000 7fd93d53a640 1 -- 172.21.15.77:0/1937877187 wait complete. 2024-03-06T07:32:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.233+0000 7fd93d53a640 1 Processor -- start 2024-03-06T07:32:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.233+0000 7fd93d53a640 1 -- start start 2024-03-06T07:32:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd93d53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380b4170 0x7fd938156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd93d53a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4dd0 0x7fd9381f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd93d53a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9381f4560 0x7fd9380cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd93d53a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380b47d0 con 0x7fd9380b4dd0 2024-03-06T07:32:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd93d53a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380b4550 con 0x7fd9381f4560 2024-03-06T07:32:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd93d53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380b4950 con 0x7fd9380b4170 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd92ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4dd0 0x7fd9381f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd92ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4dd0 0x7fd9381f4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44378/0 (socket says 172.21.15.77:44378) 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd92ffff640 1 -- 172.21.15.77:0/4178747882 learned_addr learned my addr 172.21.15.77:0/4178747882 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd936ffd640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380b4170 0x7fd938156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd9377fe640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9381f4560 0x7fd9380cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd92ffff640 1 -- 172.21.15.77:0/4178747882 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380b4170 msgr2=0x7fd938156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd92ffff640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380b4170 0x7fd938156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd92ffff640 1 -- 172.21.15.77:0/4178747882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9381f4560 msgr2=0x7fd9380cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd92ffff640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9381f4560 0x7fd9380cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.234+0000 7fd92ffff640 1 -- 172.21.15.77:0/4178747882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9381596a0 con 0x7fd9380b4dd0 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.235+0000 7fd92ffff640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4dd0 0x7fd9381f4020 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fd92000d5f0 tx=0x7fd92000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.235+0000 7fd92f7fe640 1 -- 172.21.15.77:0/4178747882 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200125f0 con 0x7fd9380b4dd0 2024-03-06T07:32:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.235+0000 7fd92f7fe640 1 -- 172.21.15.77:0/4178747882 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd920019070 con 0x7fd9380b4dd0 2024-03-06T07:32:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.235+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd938159980 con 0x7fd9380b4dd0 2024-03-06T07:32:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.235+0000 7fd92f7fe640 1 -- 172.21.15.77:0/4178747882 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200129d0 con 0x7fd9380b4dd0 2024-03-06T07:32:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.235+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd938157850 con 0x7fd9380b4dd0 2024-03-06T07:32:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.236+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93804f330 con 0x7fd9380b4dd0 2024-03-06T07:32:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.236+0000 7fd92f7fe640 1 -- 172.21.15.77:0/4178747882 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd920012b70 con 0x7fd9380b4dd0 2024-03-06T07:32:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.237+0000 7fd92f7fe640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd91806b500 0x7fd91806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.237+0000 7fd936ffd640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd91806b500 0x7fd91806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.237+0000 7fd92f7fe640 1 -- 172.21.15.77:0/4178747882 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd92008a5e0 con 0x7fd9380b4dd0 2024-03-06T07:32:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.237+0000 7fd936ffd640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd91806b500 0x7fd91806d9c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fd928036000 tx=0x7fd92803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.241+0000 7fd92f7fe640 1 -- 172.21.15.77:0/4178747882 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd920059440 con 0x7fd9380b4dd0 2024-03-06T07:32:47.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.459+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9380733b0 con 0x7fd9380b4dd0 2024-03-06T07:32:47.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.459+0000 7fd92f7fe640 1 -- 172.21.15.77:0/4178747882 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fd920058d40 con 0x7fd9380b4dd0 2024-03-06T07:32:47.462 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:47.462 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:47.462 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd91806b500 msgr2=0x7fd91806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd91806b500 0x7fd91806d9c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fd928036000 tx=0x7fd92803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4dd0 msgr2=0x7fd9381f4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4dd0 0x7fd9381f4020 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fd92000d5f0 tx=0x7fd92000dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 shutdown_connections 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd91806b500 0x7fd91806d9c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9380b4dd0 0x7fd9381f4020 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd9381f4560 0x7fd9380cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 --2- 172.21.15.77:0/4178747882 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd9380b4170 0x7fd938156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.461+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 >> 172.21.15.77:0/4178747882 conn(0x7fd9380d3c40 msgr2=0x7fd9380d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.462+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 shutdown_connections 2024-03-06T07:32:47.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:47.462+0000 7fd93d53a640 1 -- 172.21.15.77:0/4178747882 wait complete. 2024-03-06T07:32:47.675 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:47.676 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:48.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:48.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.776+0000 7f3c49c55640 1 Processor -- start 2024-03-06T07:32:48.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.776+0000 7f3c49c55640 1 -- start start 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.776+0000 7f3c49c55640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c4415dfe0 0x7f3c440cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.776+0000 7f3c49c55640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c440d29c0 0x7f3c440b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.776+0000 7f3c49c55640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c440b3430 con 0x7f3c440cf830 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.776+0000 7f3c49c55640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c440b31e0 con 0x7f3c440d29c0 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.776+0000 7f3c49c55640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440b3d60 con 0x7f3c4415dfe0 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c43fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c440d29c0 0x7f3c440b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c42ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3c440cf830 0x7f3c440d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46920/0 (socket says 172.21.15.77:46920) 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c43fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c440d29c0 0x7f3c440b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45486/0 (socket says 172.21.15.77:45486) 2024-03-06T07:32:48.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c42ffd640 1 -- 172.21.15.77:0/3561194437 learned_addr learned my addr 172.21.15.77:0/3561194437 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:48.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c437fe640 1 --2- 172.21.15.77:0/3561194437 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c4415dfe0 0x7f3c440cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c43fff640 1 -- 172.21.15.77:0/3561194437 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c4415dfe0 msgr2=0x7f3c440cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c43fff640 1 --2- 172.21.15.77:0/3561194437 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c4415dfe0 0x7f3c440cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c43fff640 1 -- 172.21.15.77:0/3561194437 >> v1:172.21.15.77:6789/0 conn(0x7f3c440cf830 legacy=0x7f3c440d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c427fc640 1 -- 172.21.15.77:0/3561194437 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364748793 0 0) 0x7f3c440b3430 con 0x7f3c440cf830 2024-03-06T07:32:48.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c43fff640 1 -- 172.21.15.77:0/3561194437 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c440b3710 con 0x7f3c440d29c0 2024-03-06T07:32:48.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.777+0000 7f3c437fe640 1 --2- 172.21.15.77:0/3561194437 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c4415dfe0 0x7f3c440cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.778+0000 7f3c43fff640 1 --2- 172.21.15.77:0/3561194437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440d29c0 0x7f3c440b41f0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b979a6c85d859ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.778+0000 7f3c427fc640 1 -- 172.21.15.77:0/3561194437 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c039070 con 0x7f3c440d29c0 2024-03-06T07:32:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.778+0000 7f3c427fc640 1 -- 172.21.15.77:0/3561194437 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c02cc50 con 0x7f3c440d29c0 2024-03-06T07:32:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.778+0000 7f3c427fc640 1 -- 172.21.15.77:0/3561194437 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c034440 con 0x7f3c440d29c0 2024-03-06T07:32:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.779+0000 7f3c49c55640 1 -- 172.21.15.77:0/3561194437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440d29c0 msgr2=0x7f3c440b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.779+0000 7f3c49c55640 1 --2- 172.21.15.77:0/3561194437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440d29c0 0x7f3c440b41f0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.779+0000 7f3c49c55640 1 -- 172.21.15.77:0/3561194437 shutdown_connections 2024-03-06T07:32:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.779+0000 7f3c49c55640 1 --2- 172.21.15.77:0/3561194437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c440d29c0 0x7f3c440b41f0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.779+0000 7f3c49c55640 1 --2- 172.21.15.77:0/3561194437 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c4415dfe0 0x7f3c440cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.779+0000 7f3c49c55640 1 -- 172.21.15.77:0/3561194437 >> 172.21.15.77:0/3561194437 conn(0x7f3c440d3cd0 msgr2=0x7f3c440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.779+0000 7f3c49c55640 1 -- 172.21.15.77:0/3561194437 shutdown_connections 2024-03-06T07:32:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.779+0000 7f3c49c55640 1 -- 172.21.15.77:0/3561194437 wait complete. 2024-03-06T07:32:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.781+0000 7f3c49c55640 1 Processor -- start 2024-03-06T07:32:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.781+0000 7f3c49c55640 1 -- start start 2024-03-06T07:32:48.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c49c55640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c440cf830 0x7f3c440ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c49c55640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440d29c0 0x7f3c4415f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c49c55640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4415dfe0 0x7f3c440cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c49c55640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440d1df0 con 0x7f3c4415dfe0 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c49c55640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440d1c70 con 0x7f3c440cf830 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c49c55640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4415e3c0 con 0x7f3c440d29c0 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4415dfe0 0x7f3c440cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4415dfe0 0x7f3c440cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45502/0 (socket says 172.21.15.77:45502) 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c43fff640 1 -- 172.21.15.77:0/4034820709 learned_addr learned my addr 172.21.15.77:0/4034820709 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c42ffd640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440d29c0 0x7f3c4415f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c437fe640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c440cf830 0x7f3c440ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c43fff640 1 -- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440d29c0 msgr2=0x7f3c4415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c43fff640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440d29c0 0x7f3c4415f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c43fff640 1 -- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c440cf830 msgr2=0x7f3c440ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c43fff640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c440cf830 0x7f3c440ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c43fff640 1 -- 172.21.15.77:0/4034820709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4415faa0 con 0x7f3c4415dfe0 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.782+0000 7f3c437fe640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c440cf830 0x7f3c440ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.783+0000 7f3c43fff640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4415dfe0 0x7f3c440cb440 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.783+0000 7f3c2bfff640 1 -- 172.21.15.77:0/4034820709 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c044020 con 0x7f3c4415dfe0 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.783+0000 7f3c2bfff640 1 -- 172.21.15.77:0/4034820709 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c03a040 con 0x7f3c4415dfe0 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.783+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4415fd80 con 0x7f3c4415dfe0 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.783+0000 7f3c2bfff640 1 -- 172.21.15.77:0/4034820709 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c039050 con 0x7f3c4415dfe0 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.783+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c440caf20 con 0x7f3c4415dfe0 2024-03-06T07:32:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.784+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c440b3360 con 0x7f3c4415dfe0 2024-03-06T07:32:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.784+0000 7f3c2bfff640 1 -- 172.21.15.77:0/4034820709 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c3c003990 con 0x7f3c4415dfe0 2024-03-06T07:32:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.785+0000 7f3c2bfff640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.785+0000 7f3c2bfff640 1 -- 172.21.15.77:0/4034820709 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3c3c0ac600 con 0x7f3c4415dfe0 2024-03-06T07:32:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.785+0000 7f3c437fe640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.785+0000 7f3c437fe640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3006b4d0 0x7f3c3006d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400a990 tx=0x7f3c34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:48.789+0000 7f3c2bfff640 1 -- 172.21.15.77:0/4034820709 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c440b3360 con 0x7f3c4415dfe0 2024-03-06T07:32:49.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.004+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c4404f330 con 0x7f3c4415dfe0 2024-03-06T07:32:49.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.004+0000 7f3c2bfff640 1 -- 172.21.15.77:0/4034820709 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f3c3c04e060 con 0x7f3c4415dfe0 2024-03-06T07:32:49.006 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:49.006 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:49.007 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3006b4d0 msgr2=0x7f3c3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3006b4d0 0x7f3c3006d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400a990 tx=0x7f3c34008040 comp rx=0 tx=0).stop 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4415dfe0 msgr2=0x7f3c440cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4415dfe0 0x7f3c440cb440 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 shutdown_connections 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f3c3006b4d0 0x7f3c3006d990 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c4415dfe0 0x7f3c440cb440 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f3c440d29c0 0x7f3c4415f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 --2- 172.21.15.77:0/4034820709 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c440cf830 0x7f3c440ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 >> 172.21.15.77:0/4034820709 conn(0x7f3c440d3cd0 msgr2=0x7f3c440b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 shutdown_connections 2024-03-06T07:32:49.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:49.006+0000 7f3c49c55640 1 -- 172.21.15.77:0/4034820709 wait complete. 2024-03-06T07:32:49.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:49.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:50.222 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:50.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.321+0000 7f8c8fe42640 1 Processor -- start 2024-03-06T07:32:50.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.321+0000 7f8c8fe42640 1 -- start start 2024-03-06T07:32:50.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8fe42640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c8815dfe0 0x7f8c880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8fe42640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8c880d29c0 0x7f8c880b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8fe42640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c880b3430 con 0x7f8c880cf830 2024-03-06T07:32:50.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8fe42640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c880b31e0 con 0x7f8c880d29c0 2024-03-06T07:32:50.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8fe42640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880b3d60 con 0x7f8c8815dfe0 2024-03-06T07:32:50.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8d3dd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8c880cf830 0x7f8c880d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46930/0 (socket says 172.21.15.77:46930) 2024-03-06T07:32:50.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8d3dd640 1 -- 172.21.15.77:0/3763426067 learned_addr learned my addr 172.21.15.77:0/3763426067 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027815066 0 0) 0x7f8c880b3430 con 0x7f8c880cf830 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c70003520 con 0x7f8c880cf830 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.322+0000 7f8c8e3df640 1 --2- 172.21.15.77:0/3763426067 >> v2:172.21.15.158:3300/0 conn(0x7f8c880d29c0 0x7f8c880b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8dbde640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c8815dfe0 0x7f8c880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1117317577 0 0) 0x7f8c70003520 con 0x7f8c880cf830 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c8815dfe0 msgr2=0x7f8c880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c8815dfe0 0x7f8c880cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 >> v2:172.21.15.158:3300/0 conn(0x7f8c880d29c0 msgr2=0x7f8c880b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 --2- 172.21.15.77:0/3763426067 >> v2:172.21.15.158:3300/0 conn(0x7f8c880d29c0 0x7f8c880b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c880b3710 con 0x7f8c880cf830 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f8c7c001670 con 0x7f8c880cf830 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 >> v1:172.21.15.77:6789/0 conn(0x7f8c880cf830 legacy=0x7f8c880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c700039b0 0x7f8c70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c8815dfe0 0x7f8c700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880b31e0 con 0x7f8c70004800 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880b3d60 con 0x7f8c700039b0 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7000b9c0 con 0x7f8c8815dfe0 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8d3dd640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8e3df640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c8815dfe0 0x7f8c700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8dbde640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c700039b0 0x7f8c70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8d3dd640 1 -- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c8815dfe0 msgr2=0x7f8c700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8d3dd640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c8815dfe0 0x7f8c700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8d3dd640 1 -- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c700039b0 msgr2=0x7f8c70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8d3dd640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c700039b0 0x7f8c70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.323+0000 7f8c8d3dd640 1 -- 172.21.15.77:0/3763426067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c70003520 con 0x7f8c70004800 2024-03-06T07:32:50.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.324+0000 7f8c8d3dd640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c70004c80 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c007fd0 tx=0x7f8c70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929d18a3c5076d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.324+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c007d50 con 0x7f8c70004800 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.324+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c030070 con 0x7f8c70004800 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.324+0000 7f8c8cbdc640 1 -- 172.21.15.77:0/3763426067 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c02cd40 con 0x7f8c70004800 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.325+0000 7f8c8fe42640 1 -- 172.21.15.77:0/3763426067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 msgr2=0x7f8c70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.325+0000 7f8c8fe42640 1 --2- 172.21.15.77:0/3763426067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c70004c80 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c007fd0 tx=0x7f8c70005830 comp rx=0 tx=0).stop 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.325+0000 7f8c8e3df640 1 -- 172.21.15.77:0/3763426067 reap_dead start 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.325+0000 7f8c8fe42640 1 -- 172.21.15.77:0/3763426067 shutdown_connections 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.325+0000 7f8c8fe42640 1 -- 172.21.15.77:0/3763426067 >> 172.21.15.77:0/3763426067 conn(0x7f8c880d3cd0 msgr2=0x7f8c880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.325+0000 7f8c8fe42640 1 -- 172.21.15.77:0/3763426067 shutdown_connections 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.325+0000 7f8c8fe42640 1 -- 172.21.15.77:0/3763426067 wait complete. 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.326+0000 7f8c8fe42640 1 Processor -- start 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.326+0000 7f8c8fe42640 1 -- start start 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8fe42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c8815fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8fe42640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c880ccbe0 0x7f8c88160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8fe42640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c880cd060 0x7f8c88161220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8fe42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880d1df0 con 0x7f8c70004800 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8fe42640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880d1c70 con 0x7f8c880ccbe0 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8fe42640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8815e3c0 con 0x7f8c880cd060 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c8815fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c8815fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45518/0 (socket says 172.21.15.77:45518) 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 -- 172.21.15.77:0/2276538096 learned_addr learned my addr 172.21.15.77:0/2276538096 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8d3dd640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c880ccbe0 0x7f8c88160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8e3df640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c880cd060 0x7f8c88161220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 -- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c880cd060 msgr2=0x7f8c88161220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c880cd060 0x7f8c88161220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 -- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c880ccbe0 msgr2=0x7f8c88160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c880ccbe0 0x7f8c88160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 -- 172.21.15.77:0/2276538096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8815e710 con 0x7f8c70004800 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8e3df640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c880cd060 0x7f8c88161220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c8dbde640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c8815fa90 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f8c7800b7f0 tx=0x7f8c7800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c867fc640 1 -- 172.21.15.77:0/2276538096 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78018070 con 0x7f8c70004800 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c867fc640 1 -- 172.21.15.77:0/2276538096 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c78010830 con 0x7f8c70004800 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.327+0000 7f8c867fc640 1 -- 172.21.15.77:0/2276538096 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78010c00 con 0x7f8c70004800 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.328+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8815e9f0 con 0x7f8c70004800 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.328+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8815efb0 con 0x7f8c70004800 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.329+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c880d3740 con 0x7f8c70004800 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.329+0000 7f8c867fc640 1 -- 172.21.15.77:0/2276538096 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c7801b070 con 0x7f8c70004800 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.329+0000 7f8c867fc640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c7406b500 0x7f8c7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.329+0000 7f8c867fc640 1 -- 172.21.15.77:0/2276538096 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8c7808a990 con 0x7f8c70004800 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.329+0000 7f8c8d3dd640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c7406b500 0x7f8c7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.330+0000 7f8c8d3dd640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c7406b500 0x7f8c7406d9c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c034000 tx=0x7f8c7c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.334+0000 7f8c867fc640 1 -- 172.21.15.77:0/2276538096 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c880d3740 con 0x7f8c70004800 2024-03-06T07:32:50.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.688+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c8815cdd0 con 0x7f8c70004800 2024-03-06T07:32:50.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.688+0000 7f8c867fc640 1 -- 172.21.15.77:0/2276538096 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f8c78059880 con 0x7f8c70004800 2024-03-06T07:32:50.690 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:50.691 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:50.691 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.690+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c7406b500 msgr2=0x7f8c7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.690+0000 7f8c8fe42640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c7406b500 0x7f8c7406d9c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c034000 tx=0x7f8c7c007000 comp rx=0 tx=0).stop 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.690+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 msgr2=0x7f8c8815fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.690+0000 7f8c8fe42640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c8815fa90 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f8c7800b7f0 tx=0x7f8c7800bcb0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.691+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 shutdown_connections 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.691+0000 7f8c8fe42640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8c880cd060 0x7f8c88161220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.691+0000 7f8c8fe42640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8c880ccbe0 0x7f8c88160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.691+0000 7f8c8fe42640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8c7406b500 0x7f8c7406d9c0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.691+0000 7f8c8fe42640 1 --2- 172.21.15.77:0/2276538096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70004800 0x7f8c8815fa90 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.691+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 >> 172.21.15.77:0/2276538096 conn(0x7f8c880d3cd0 msgr2=0x7f8c8806cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.691+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 shutdown_connections 2024-03-06T07:32:50.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:50.691+0000 7f8c8fe42640 1 -- 172.21.15.77:0/2276538096 wait complete. 2024-03-06T07:32:50.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:50.904 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:51.905 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:52.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.005+0000 7f14312df640 1 Processor -- start 2024-03-06T07:32:52.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.005+0000 7f14312df640 1 -- start start 2024-03-06T07:32:52.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f14312df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c0d22b0 0x7f142c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f14312df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f142c0b41f0 0x7f142c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f14312df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142c0b3430 con 0x7f142c15dfe0 2024-03-06T07:32:52.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f14312df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c0d2c50 con 0x7f142c0b41f0 2024-03-06T07:32:52.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f14312df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0b31e0 con 0x7f142c0d22b0 2024-03-06T07:32:52.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f142b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f142c0b41f0 0x7f142c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f142b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f142c0b41f0 0x7f142c0b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45530/0 (socket says 172.21.15.77:45530) 2024-03-06T07:32:52.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f142b7fe640 1 -- 172.21.15.77:0/752666033 learned_addr learned my addr 172.21.15.77:0/752666033 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:52.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.006+0000 7f142a7fc640 1 --2- 172.21.15.77:0/752666033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c0d22b0 0x7f142c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.007+0000 7f142b7fe640 1 -- 172.21.15.77:0/752666033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c0d22b0 msgr2=0x7f142c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.007+0000 7f142b7fe640 1 --2- 172.21.15.77:0/752666033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c0d22b0 0x7f142c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.007+0000 7f142b7fe640 1 -- 172.21.15.77:0/752666033 >> v1:172.21.15.158:6789/0 conn(0x7f142c15dfe0 legacy=0x7f142c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.007+0000 7f142b7fe640 1 -- 172.21.15.77:0/752666033 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c0b3710 con 0x7f142c0b41f0 2024-03-06T07:32:52.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.007+0000 7f142b7fe640 1 --2- 172.21.15.77:0/752666033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0b41f0 0x7f142c0b4650 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f1424009e50 tx=0x7f142402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956af5606c7db434 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.007+0000 7f1429ffb640 1 -- 172.21.15.77:0/752666033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142402ca70 con 0x7f142c0b41f0 2024-03-06T07:32:52.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.007+0000 7f1429ffb640 1 -- 172.21.15.77:0/752666033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142402cc10 con 0x7f142c0b41f0 2024-03-06T07:32:52.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.007+0000 7f1429ffb640 1 -- 172.21.15.77:0/752666033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14240343f0 con 0x7f142c0b41f0 2024-03-06T07:32:52.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.008+0000 7f14312df640 1 -- 172.21.15.77:0/752666033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0b41f0 msgr2=0x7f142c0b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.008+0000 7f14312df640 1 --2- 172.21.15.77:0/752666033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0b41f0 0x7f142c0b4650 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f1424009e50 tx=0x7f142402bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.008+0000 7f14312df640 1 -- 172.21.15.77:0/752666033 shutdown_connections 2024-03-06T07:32:52.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.008+0000 7f14312df640 1 --2- 172.21.15.77:0/752666033 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c0d22b0 0x7f142c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.008+0000 7f14312df640 1 --2- 172.21.15.77:0/752666033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0b41f0 0x7f142c0b4650 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.008+0000 7f14312df640 1 -- 172.21.15.77:0/752666033 >> 172.21.15.77:0/752666033 conn(0x7f142c0d3cd0 msgr2=0x7f142c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.008+0000 7f14312df640 1 -- 172.21.15.77:0/752666033 shutdown_connections 2024-03-06T07:32:52.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.008+0000 7f14312df640 1 -- 172.21.15.77:0/752666033 wait complete. 2024-03-06T07:32:52.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f14312df640 1 Processor -- start 2024-03-06T07:32:52.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f14312df640 1 -- start start 2024-03-06T07:32:52.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f14312df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f142c0b41f0 0x7f142c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f14312df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0d22b0 0x7f142c160e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f14312df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c15dfe0 0x7f142c15e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f14312df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0cfa00 con 0x7f142c0d22b0 2024-03-06T07:32:52.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f14312df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0b3d60 con 0x7f142c0b41f0 2024-03-06T07:32:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f14312df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0d1c70 con 0x7f142c15dfe0 2024-03-06T07:32:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f142a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0d22b0 0x7f142c160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f142a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0d22b0 0x7f142c160e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45536/0 (socket says 172.21.15.77:45536) 2024-03-06T07:32:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.010+0000 7f142a7fc640 1 -- 172.21.15.77:0/222063865 learned_addr learned my addr 172.21.15.77:0/222063865 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f142b7fe640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c15dfe0 0x7f142c15e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f142affd640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f142c0b41f0 0x7f142c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f142a7fc640 1 -- 172.21.15.77:0/222063865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c15dfe0 msgr2=0x7f142c15e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f142a7fc640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c15dfe0 0x7f142c15e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f142a7fc640 1 -- 172.21.15.77:0/222063865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f142c0b41f0 msgr2=0x7f142c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f142a7fc640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f142c0b41f0 0x7f142c0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f142a7fc640 1 -- 172.21.15.77:0/222063865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c0cbfb0 con 0x7f142c0d22b0 2024-03-06T07:32:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f142a7fc640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0d22b0 0x7f142c160e60 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f142000d800 tx=0x7f142000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f14137fe640 1 -- 172.21.15.77:0/222063865 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142000b6c0 con 0x7f142c0d22b0 2024-03-06T07:32:52.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f14137fe640 1 -- 172.21.15.77:0/222063865 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1420019070 con 0x7f142c0d22b0 2024-03-06T07:32:52.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c0cc270 con 0x7f142c0d22b0 2024-03-06T07:32:52.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f14137fe640 1 -- 172.21.15.77:0/222063865 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142000ba80 con 0x7f142c0d22b0 2024-03-06T07:32:52.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.011+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142c0cc830 con 0x7f142c0d22b0 2024-03-06T07:32:52.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.012+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142c04f330 con 0x7f142c0d22b0 2024-03-06T07:32:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.013+0000 7f14137fe640 1 -- 172.21.15.77:0/222063865 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f142000bc20 con 0x7f142c0d22b0 2024-03-06T07:32:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.013+0000 7f14137fe640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141806b4d0 0x7f141806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.013+0000 7f142affd640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141806b4d0 0x7f141806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.013+0000 7f14137fe640 1 -- 172.21.15.77:0/222063865 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f142008a520 con 0x7f142c0d22b0 2024-03-06T07:32:52.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.013+0000 7f142affd640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141806b4d0 0x7f141806d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f141c006fd0 tx=0x7f141c009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.019+0000 7f14137fe640 1 -- 172.21.15.77:0/222063865 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1420058d30 con 0x7f142c0d22b0 2024-03-06T07:32:52.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.267+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f142c0747a0 con 0x7f142c0d22b0 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.268+0000 7f14137fe640 1 -- 172.21.15.77:0/222063865 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f142005cdd0 con 0x7f142c0d22b0 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:52.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141806b4d0 msgr2=0x7f141806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141806b4d0 0x7f141806d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f141c006fd0 tx=0x7f141c009060 comp rx=0 tx=0).stop 2024-03-06T07:32:52.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0d22b0 msgr2=0x7f142c160e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0d22b0 0x7f142c160e60 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f142000d800 tx=0x7f142000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 shutdown_connections 2024-03-06T07:32:52.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f141806b4d0 0x7f141806d990 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f142c0d22b0 0x7f142c160e60 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f142c15dfe0 0x7f142c15e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 --2- 172.21.15.77:0/222063865 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f142c0b41f0 0x7f142c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 >> 172.21.15.77:0/222063865 conn(0x7f142c0d3cd0 msgr2=0x7f142c06ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 shutdown_connections 2024-03-06T07:32:52.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:52.270+0000 7f14312df640 1 -- 172.21.15.77:0/222063865 wait complete. 2024-03-06T07:32:52.488 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:52.488 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:53.489 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:53.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.588+0000 7f90ce842640 1 Processor -- start 2024-03-06T07:32:53.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.588+0000 7f90ce842640 1 -- start start 2024-03-06T07:32:53.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90ce842640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c81424f0 0x7f90c80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90ce842640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f90c806da40 0x7f90c815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90ce842640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c80cc980 con 0x7f90c80cd7a0 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90ce842640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c80cd220 con 0x7f90c806da40 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90ce842640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c80ccc60 con 0x7f90c81424f0 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90ccddf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f90c806da40 0x7f90c815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90c77fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f90c80cd7a0 0x7f90c806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46942/0 (socket says 172.21.15.77:46942) 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90ccddf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f90c806da40 0x7f90c815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45542/0 (socket says 172.21.15.77:45542) 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90c77fe640 1 -- 172.21.15.77:0/2567674897 learned_addr learned my addr 172.21.15.77:0/2567674897 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90c7fff640 1 --2- 172.21.15.77:0/2567674897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c81424f0 0x7f90c80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90c6ffd640 1 -- 172.21.15.77:0/2567674897 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905035480 0 0) 0x7f90c80cc980 con 0x7f90c80cd7a0 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.589+0000 7f90c6ffd640 1 -- 172.21.15.77:0/2567674897 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b0003520 con 0x7f90c80cd7a0 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90ccddf640 1 -- 172.21.15.77:0/2567674897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c81424f0 msgr2=0x7f90c80cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90ccddf640 1 --2- 172.21.15.77:0/2567674897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c81424f0 0x7f90c80cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90ccddf640 1 -- 172.21.15.77:0/2567674897 >> v1:172.21.15.77:6789/0 conn(0x7f90c80cd7a0 legacy=0x7f90c806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90ccddf640 1 -- 172.21.15.77:0/2567674897 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c80ccf40 con 0x7f90c806da40 2024-03-06T07:32:53.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90c7fff640 1 --2- 172.21.15.77:0/2567674897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c81424f0 0x7f90c80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:53.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90ccddf640 1 --2- 172.21.15.77:0/2567674897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c806da40 0x7f90c815cef0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009d90 tx=0x7f90c00302c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a908ee226ea641 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90c6ffd640 1 -- 172.21.15.77:0/2567674897 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c003d070 con 0x7f90c806da40 2024-03-06T07:32:53.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90c6ffd640 1 -- 172.21.15.77:0/2567674897 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c0038640 con 0x7f90c806da40 2024-03-06T07:32:53.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.590+0000 7f90c6ffd640 1 -- 172.21.15.77:0/2567674897 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c00389f0 con 0x7f90c806da40 2024-03-06T07:32:53.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.591+0000 7f90ce842640 1 -- 172.21.15.77:0/2567674897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c806da40 msgr2=0x7f90c815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.591+0000 7f90ce842640 1 --2- 172.21.15.77:0/2567674897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c806da40 0x7f90c815cef0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009d90 tx=0x7f90c00302c0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.591+0000 7f90ce842640 1 -- 172.21.15.77:0/2567674897 shutdown_connections 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.591+0000 7f90ce842640 1 --2- 172.21.15.77:0/2567674897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c806da40 0x7f90c815cef0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.591+0000 7f90ce842640 1 --2- 172.21.15.77:0/2567674897 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c81424f0 0x7f90c80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.591+0000 7f90ce842640 1 -- 172.21.15.77:0/2567674897 >> 172.21.15.77:0/2567674897 conn(0x7f90c80cf8a0 msgr2=0x7f90c80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.591+0000 7f90ce842640 1 -- 172.21.15.77:0/2567674897 shutdown_connections 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.591+0000 7f90ce842640 1 -- 172.21.15.77:0/2567674897 wait complete. 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.592+0000 7f90ce842640 1 Processor -- start 2024-03-06T07:32:53.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.592+0000 7f90ce842640 1 -- start start 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90ce842640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c806da40 0x7f90c813bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90ce842640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c80cd7a0 0x7f90c814f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90ce842640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c81424f0 0x7f90c80ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90ce842640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8071ce0 con 0x7f90c80cd7a0 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90ce842640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8071b60 con 0x7f90c81424f0 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90ce842640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8071e60 con 0x7f90c806da40 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c80cd7a0 0x7f90c814f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c80cd7a0 0x7f90c814f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45546/0 (socket says 172.21.15.77:45546) 2024-03-06T07:32:53.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c77fe640 1 -- 172.21.15.77:0/1134140269 learned_addr learned my addr 172.21.15.77:0/1134140269 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90ccddf640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c81424f0 0x7f90c80ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c7fff640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c806da40 0x7f90c813bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c77fe640 1 -- 172.21.15.77:0/1134140269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c806da40 msgr2=0x7f90c813bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c77fe640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c806da40 0x7f90c813bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c77fe640 1 -- 172.21.15.77:0/1134140269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c81424f0 msgr2=0x7f90c80ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c77fe640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c81424f0 0x7f90c80ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.593+0000 7f90c77fe640 1 -- 172.21.15.77:0/1134140269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c813e300 con 0x7f90c80cd7a0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.594+0000 7f90c77fe640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c80cd7a0 0x7f90c814f360 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00c430 tx=0x7f90bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.594+0000 7f90c4ff9640 1 -- 172.21.15.77:0/1134140269 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc017070 con 0x7f90c80cd7a0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.594+0000 7f90c4ff9640 1 -- 172.21.15.77:0/1134140269 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc005ce0 con 0x7f90c80cd7a0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.594+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c813e5e0 con 0x7f90c80cd7a0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.594+0000 7f90c4ff9640 1 -- 172.21.15.77:0/1134140269 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0072f0 con 0x7f90c80cd7a0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.594+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c813eba0 con 0x7f90c80cd7a0 2024-03-06T07:32:53.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.595+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c804f330 con 0x7f90c80cd7a0 2024-03-06T07:32:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.595+0000 7f90c4ff9640 1 -- 172.21.15.77:0/1134140269 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90bc01a070 con 0x7f90c80cd7a0 2024-03-06T07:32:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.595+0000 7f90c4ff9640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90b406b4d0 0x7f90b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.596+0000 7f90c4ff9640 1 -- 172.21.15.77:0/1134140269 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f90bc089700 con 0x7f90c80cd7a0 2024-03-06T07:32:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.596+0000 7f90c7fff640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90b406b4d0 0x7f90b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.596+0000 7f90c7fff640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90b406b4d0 0x7f90b406d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f90b8006fd0 tx=0x7f90b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.600+0000 7f90c4ff9640 1 -- 172.21.15.77:0/1134140269 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90bc058560 con 0x7f90c80cd7a0 2024-03-06T07:32:53.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.816+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90c80d0620 con 0x7f90c80cd7a0 2024-03-06T07:32:53.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.817+0000 7f90c4ff9640 1 -- 172.21.15.77:0/1134140269 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f90bc057e60 con 0x7f90c80cd7a0 2024-03-06T07:32:53.819 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:53.819 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:53.819 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:53.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90b406b4d0 msgr2=0x7f90b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90b406b4d0 0x7f90b406d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f90b8006fd0 tx=0x7f90b8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:53.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c80cd7a0 msgr2=0x7f90c814f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c80cd7a0 0x7f90c814f360 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00c430 tx=0x7f90bc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 shutdown_connections 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f90b406b4d0 0x7f90b406d990 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90c80cd7a0 0x7f90c814f360 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f90c81424f0 0x7f90c80ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 --2- 172.21.15.77:0/1134140269 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f90c806da40 0x7f90c813bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 >> 172.21.15.77:0/1134140269 conn(0x7f90c80cf8a0 msgr2=0x7f90c80cc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 shutdown_connections 2024-03-06T07:32:53.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:53.819+0000 7f90ce842640 1 -- 172.21.15.77:0/1134140269 wait complete. 2024-03-06T07:32:54.037 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:54.037 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:55.038 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:55.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.137+0000 7f6129a88640 1 Processor -- start 2024-03-06T07:32:55.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.137+0000 7f6129a88640 1 -- start start 2024-03-06T07:32:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6129a88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f61240d22b0 0x7f61240d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6129a88640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f61240b41f0 0x7f61240b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6129a88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61240b3430 con 0x7f612415dfe0 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6129a88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61240d2c50 con 0x7f61240d22b0 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6129a88640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61240b31e0 con 0x7f61240b41f0 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6122ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f61240d22b0 0x7f61240d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6122ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f61240d22b0 0x7f61240d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45556/0 (socket says 172.21.15.77:45556) 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6122ffd640 1 -- 172.21.15.77:0/1808061151 learned_addr learned my addr 172.21.15.77:0/1808061151 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6122ffd640 1 -- 172.21.15.77:0/1808061151 >> v2:172.21.15.158:3300/0 conn(0x7f61240b41f0 msgr2=0x7f61240b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6123fff640 1 --2- 172.21.15.77:0/1808061151 >> v2:172.21.15.158:3300/0 conn(0x7f61240b41f0 0x7f61240b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6122ffd640 1 --2- 172.21.15.77:0/1808061151 >> v2:172.21.15.158:3300/0 conn(0x7f61240b41f0 0x7f61240b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6122ffd640 1 -- 172.21.15.77:0/1808061151 >> v1:172.21.15.158:6789/0 conn(0x7f612415dfe0 legacy=0x7f61240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6122ffd640 1 -- 172.21.15.77:0/1808061151 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61240b3710 con 0x7f61240d22b0 2024-03-06T07:32:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.138+0000 7f6123fff640 1 --2- 172.21.15.77:0/1808061151 >> v2:172.21.15.158:3300/0 conn(0x7f61240b41f0 0x7f61240b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.139+0000 7f6122ffd640 1 --2- 172.21.15.77:0/1808061151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240d2710 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f6118009870 tx=0x7f611802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7270dfd6d76936e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.139+0000 7f61227fc640 1 -- 172.21.15.77:0/1808061151 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6118035020 con 0x7f61240d22b0 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f61227fc640 1 -- 172.21.15.77:0/1808061151 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6118031870 con 0x7f61240d22b0 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f61227fc640 1 -- 172.21.15.77:0/1808061151 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6118031c40 con 0x7f61240d22b0 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f6129a88640 1 -- 172.21.15.77:0/1808061151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 msgr2=0x7f61240d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f6129a88640 1 --2- 172.21.15.77:0/1808061151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240d2710 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f6118009870 tx=0x7f611802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f6129a88640 1 -- 172.21.15.77:0/1808061151 shutdown_connections 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f6129a88640 1 --2- 172.21.15.77:0/1808061151 >> v2:172.21.15.158:3300/0 conn(0x7f61240b41f0 0x7f61240b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f6129a88640 1 --2- 172.21.15.77:0/1808061151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240d2710 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f6129a88640 1 -- 172.21.15.77:0/1808061151 >> 172.21.15.77:0/1808061151 conn(0x7f61240d3cd0 msgr2=0x7f61240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f6129a88640 1 -- 172.21.15.77:0/1808061151 shutdown_connections 2024-03-06T07:32:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.140+0000 7f6129a88640 1 -- 172.21.15.77:0/1808061151 wait complete. 2024-03-06T07:32:55.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.142+0000 7f6129a88640 1 Processor -- start 2024-03-06T07:32:55.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.142+0000 7f6129a88640 1 -- start start 2024-03-06T07:32:55.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.143+0000 7f6129a88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61240b41f0 0x7f61240cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.143+0000 7f6129a88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.143+0000 7f6129a88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f612415dfe0 0x7f61240b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.143+0000 7f6129a88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61240cfa00 con 0x7f61240d22b0 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6129a88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61240b3d60 con 0x7f61240b41f0 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6129a88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61240d1c70 con 0x7f612415dfe0 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45558/0 (socket says 172.21.15.77:45558) 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 -- 172.21.15.77:0/4063544224 learned_addr learned my addr 172.21.15.77:0/4063544224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f61237fe640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61240b41f0 0x7f61240cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6123fff640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f612415dfe0 0x7f61240b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 -- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f612415dfe0 msgr2=0x7f61240b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f612415dfe0 0x7f61240b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 -- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61240b41f0 msgr2=0x7f61240cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61240b41f0 0x7f61240cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 -- 172.21.15.77:0/4063544224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61240cd7b0 con 0x7f61240d22b0 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6123fff640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f612415dfe0 0x7f61240b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6122ffd640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240b68a0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f6118000c00 tx=0x7f61180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f610bfff640 1 -- 172.21.15.77:0/4063544224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6118040070 con 0x7f61240d22b0 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f610bfff640 1 -- 172.21.15.77:0/4063544224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611803a420 con 0x7f61240d22b0 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61240cc1d0 con 0x7f61240d22b0 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f610bfff640 1 -- 172.21.15.77:0/4063544224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6118035040 con 0x7f61240d22b0 2024-03-06T07:32:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.144+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61240cc790 con 0x7f61240d22b0 2024-03-06T07:32:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.145+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612404f330 con 0x7f61240d22b0 2024-03-06T07:32:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.146+0000 7f610bfff640 1 -- 172.21.15.77:0/4063544224 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f611804b060 con 0x7f61240d22b0 2024-03-06T07:32:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.146+0000 7f610bfff640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611006b4d0 0x7f611006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.146+0000 7f61237fe640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611006b4d0 0x7f611006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.146+0000 7f610bfff640 1 -- 172.21.15.77:0/4063544224 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f61180ac060 con 0x7f61240d22b0 2024-03-06T07:32:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.147+0000 7f61237fe640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611006b4d0 0x7f611006d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f6114006fd0 tx=0x7f6114009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.151+0000 7f610bfff640 1 -- 172.21.15.77:0/4063544224 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f611807af50 con 0x7f61240d22b0 2024-03-06T07:32:55.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.367+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61240b0af0 con 0x7f61240d22b0 2024-03-06T07:32:55.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.367+0000 7f610bfff640 1 -- 172.21.15.77:0/4063544224 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f611807a850 con 0x7f61240d22b0 2024-03-06T07:32:55.371 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:55.371 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:55.373 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:55.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611006b4d0 msgr2=0x7f611006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611006b4d0 0x7f611006d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f6114006fd0 tx=0x7f6114009060 comp rx=0 tx=0).stop 2024-03-06T07:32:55.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 msgr2=0x7f61240b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240b68a0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f6118000c00 tx=0x7f61180046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 shutdown_connections 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f611006b4d0 0x7f611006d990 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61240d22b0 0x7f61240b68a0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f612415dfe0 0x7f61240b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 --2- 172.21.15.77:0/4063544224 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f61240b41f0 0x7f61240cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 >> 172.21.15.77:0/4063544224 conn(0x7f61240d3cd0 msgr2=0x7f61240cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 shutdown_connections 2024-03-06T07:32:55.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:55.369+0000 7f6129a88640 1 -- 172.21.15.77:0/4063544224 wait complete. 2024-03-06T07:32:55.587 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:55.588 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:56.589 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.687+0000 7fcb8036b640 1 Processor -- start 2024-03-06T07:32:56.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.687+0000 7fcb8036b640 1 -- start start 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.688+0000 7fcb8036b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb780d22b0 0x7fcb780d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb8036b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb780b3430 con 0x7fcb7815dfe0 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb8036b640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb780b3710 con 0x7fcb780b41f0 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb8036b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb780d2c50 con 0x7fcb780d22b0 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb7d906640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb780d22b0 0x7fcb780d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb7d906640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb780d22b0 0x7fcb780d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45568/0 (socket says 172.21.15.77:45568) 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb7d906640 1 -- 172.21.15.77:0/3206351276 learned_addr learned my addr 172.21.15.77:0/3206351276 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb7e107640 1 --1- 172.21.15.77:0/3206351276 >> v1:172.21.15.77:6789/0 conn(0x7fcb7815dfe0 0x7fcb780cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46948/0 (socket says 172.21.15.77:46948) 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb7d906640 1 -- 172.21.15.77:0/3206351276 >> v1:172.21.15.158:6789/0 conn(0x7fcb780b41f0 legacy=0x7fcb780b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb7d906640 1 -- 172.21.15.77:0/3206351276 >> v1:172.21.15.77:6789/0 conn(0x7fcb7815dfe0 legacy=0x7fcb780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb7d105640 1 -- 172.21.15.77:0/3206351276 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772310508 0 0) 0x7fcb780b3430 con 0x7fcb7815dfe0 2024-03-06T07:32:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.689+0000 7fcb7d906640 1 -- 172.21.15.77:0/3206351276 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb780b39f0 con 0x7fcb780d22b0 2024-03-06T07:32:56.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb7d906640 1 --2- 172.21.15.77:0/3206351276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780d2710 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fcb740098a0 tx=0x7fcb7402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2088f00afa6b6734 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb7d105640 1 -- 172.21.15.77:0/3206351276 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74034070 con 0x7fcb780d22b0 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb7d105640 1 -- 172.21.15.77:0/3206351276 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb74030840 con 0x7fcb780d22b0 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb7d105640 1 -- 172.21.15.77:0/3206351276 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74030c10 con 0x7fcb780d22b0 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb8036b640 1 -- 172.21.15.77:0/3206351276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 msgr2=0x7fcb780d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb8036b640 1 --2- 172.21.15.77:0/3206351276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780d2710 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fcb740098a0 tx=0x7fcb7402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb8036b640 1 -- 172.21.15.77:0/3206351276 shutdown_connections 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb8036b640 1 --2- 172.21.15.77:0/3206351276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780d2710 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb8036b640 1 -- 172.21.15.77:0/3206351276 >> 172.21.15.77:0/3206351276 conn(0x7fcb780d3cd0 msgr2=0x7fcb780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb8036b640 1 -- 172.21.15.77:0/3206351276 shutdown_connections 2024-03-06T07:32:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.690+0000 7fcb8036b640 1 -- 172.21.15.77:0/3206351276 wait complete. 2024-03-06T07:32:56.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb8036b640 1 Processor -- start 2024-03-06T07:32:56.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb8036b640 1 -- start start 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb8036b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb780b41f0 0x7fcb780b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb8036b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb8036b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7815dfe0 0x7fcb7814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb8036b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb780b3d60 con 0x7fcb780d22b0 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb8036b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb780b31e0 con 0x7fcb7815dfe0 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb8036b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb780d1c70 con 0x7fcb780b41f0 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb7d906640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb7d906640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780b6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45584/0 (socket says 172.21.15.77:45584) 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.692+0000 7fcb7d906640 1 -- 172.21.15.77:0/942324364 learned_addr learned my addr 172.21.15.77:0/942324364 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb7e908640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7815dfe0 0x7fcb7814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb7d906640 1 -- 172.21.15.77:0/942324364 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb780b41f0 msgr2=0x7fcb780b75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb7d906640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb780b41f0 0x7fcb780b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb7d906640 1 -- 172.21.15.77:0/942324364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7815dfe0 msgr2=0x7fcb7814f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb7d906640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7815dfe0 0x7fcb7814f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb7d906640 1 -- 172.21.15.77:0/942324364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb780b7cc0 con 0x7fcb780d22b0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb7e908640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7815dfe0 0x7fcb7814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb7d906640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780b6c60 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fcb74000c00 tx=0x7fcb740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb6effd640 1 -- 172.21.15.77:0/942324364 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74034040 con 0x7fcb780d22b0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb6effd640 1 -- 172.21.15.77:0/942324364 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb74030640 con 0x7fcb780d22b0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb780b7fa0 con 0x7fcb780d22b0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb6effd640 1 -- 172.21.15.77:0/942324364 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74039d20 con 0x7fcb780d22b0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.693+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7815fed0 con 0x7fcb780d22b0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.694+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7804f330 con 0x7fcb780d22b0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.694+0000 7fcb6effd640 1 -- 172.21.15.77:0/942324364 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb7404b060 con 0x7fcb780d22b0 2024-03-06T07:32:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.695+0000 7fcb6effd640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.695+0000 7fcb7e107640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.695+0000 7fcb6effd640 1 -- 172.21.15.77:0/942324364 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcb740ac070 con 0x7fcb780d22b0 2024-03-06T07:32:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.696+0000 7fcb7e107640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fcb68007fd0 tx=0x7fcb68007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.699+0000 7fcb6effd640 1 -- 172.21.15.77:0/942324364 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb7407aed0 con 0x7fcb780d22b0 2024-03-06T07:32:56.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.915+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb7814f5d0 con 0x7fcb780d22b0 2024-03-06T07:32:56.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.915+0000 7fcb6effd640 1 -- 172.21.15.77:0/942324364 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fcb7407a7d0 con 0x7fcb780d22b0 2024-03-06T07:32:56.919 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:56.919 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:56.919 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 msgr2=0x7fcb5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fcb68007fd0 tx=0x7fcb68007960 comp rx=0 tx=0).stop 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 msgr2=0x7fcb780b6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780b6c60 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fcb74000c00 tx=0x7fcb740046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 shutdown_connections 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcb5c06b4d0 0x7fcb5c06d990 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb780d22b0 0x7fcb780b6c60 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcb7815dfe0 0x7fcb7814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 --2- 172.21.15.77:0/942324364 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcb780b41f0 0x7fcb780b75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.917+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 >> 172.21.15.77:0/942324364 conn(0x7fcb780d3cd0 msgr2=0x7fcb7815d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.918+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 shutdown_connections 2024-03-06T07:32:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:56.918+0000 7fcb8036b640 1 -- 172.21.15.77:0/942324364 wait complete. 2024-03-06T07:32:57.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:57.132 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:58.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:58.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.251+0000 7f7087f81640 1 Processor -- start 2024-03-06T07:32:58.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.251+0000 7f7087f81640 1 -- start start 2024-03-06T07:32:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.252+0000 7f7087f81640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f70800cce40 0x7f708015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.252+0000 7f7087f81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 0x7f70800cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.252+0000 7f7087f81640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7080071f40 con 0x7f70801515e0 2024-03-06T07:32:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.252+0000 7f7087f81640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70800cd2f0 con 0x7f70800cce40 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.252+0000 7f7087f81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7080070a70 con 0x7f70800cd7a0 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.252+0000 7f7085d1d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f70801515e0 0x7f7080071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46950/0 (socket says 172.21.15.77:46950) 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.252+0000 7f7085d1d640 1 -- 172.21.15.77:0/3991763985 learned_addr learned my addr 172.21.15.77:0/3991763985 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708651e640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 0x7f70800cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708551c640 1 --2- 172.21.15.77:0/3991763985 >> v2:172.21.15.158:3300/0 conn(0x7f70800cce40 0x7f708015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995261240 0 0) 0x7f7080071f40 con 0x7f70801515e0 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706c003520 con 0x7f70801515e0 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3802322607 0 0) 0x7f706c003520 con 0x7f70801515e0 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 msgr2=0x7f70800cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 0x7f70800cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 >> v2:172.21.15.158:3300/0 conn(0x7f70800cce40 msgr2=0x7f708015b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 --2- 172.21.15.77:0/3991763985 >> v2:172.21.15.158:3300/0 conn(0x7f70800cce40 0x7f708015b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080070d50 con 0x7f70801515e0 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f707c001690 con 0x7f70801515e0 2024-03-06T07:32:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 >> v1:172.21.15.77:6789/0 conn(0x7f70801515e0 legacy=0x7f7080071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 0x7f706c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c003f30 0x7f706c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c004d40 0x7f706c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70800cd2f0 con 0x7f706c003f30 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7080070a70 con 0x7f706c004d40 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c00cad0 con 0x7f70800cd7a0 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708551c640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c003f30 0x7f706c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7085d1d640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 0x7f706c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708651e640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c004d40 0x7f706c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708551c640 1 -- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 msgr2=0x7f706c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708551c640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 0x7f706c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708551c640 1 -- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c004d40 msgr2=0x7f706c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708551c640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c004d40 0x7f706c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708551c640 1 -- 172.21.15.77:0/3991763985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c003520 con 0x7f706c003f30 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f708651e640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c004d40 0x7f706c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.253+0000 7f7085d1d640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f70800cd7a0 0x7f706c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.254+0000 7f708551c640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c003f30 0x7f706c004800 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f7070009ac0 tx=0x7f706c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edfed0a2309e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070039070 con 0x7f706c003f30 2024-03-06T07:32:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707002c6d0 con 0x7f706c003f30 2024-03-06T07:32:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7084d1b640 1 -- 172.21.15.77:0/3991763985 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070009690 con 0x7f706c003f30 2024-03-06T07:32:58.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7087f81640 1 -- 172.21.15.77:0/3991763985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c003f30 msgr2=0x7f706c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7087f81640 1 --2- 172.21.15.77:0/3991763985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c003f30 0x7f706c004800 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f7070009ac0 tx=0x7f706c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:58.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f708651e640 1 -- 172.21.15.77:0/3991763985 reap_dead start 2024-03-06T07:32:58.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7087f81640 1 -- 172.21.15.77:0/3991763985 shutdown_connections 2024-03-06T07:32:58.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7087f81640 1 -- 172.21.15.77:0/3991763985 >> 172.21.15.77:0/3991763985 conn(0x7f70800cf8a0 msgr2=0x7f70800cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7087f81640 1 -- 172.21.15.77:0/3991763985 shutdown_connections 2024-03-06T07:32:58.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.255+0000 7f7087f81640 1 -- 172.21.15.77:0/3991763985 wait complete. 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.256+0000 7f7087f81640 1 Processor -- start 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.256+0000 7f7087f81640 1 -- start start 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f7087f81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c003f30 0x7f7080136680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f7087f81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7080139920 0x7f708014b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f7087f81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f708014bbb0 0x7f7080148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f7087f81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70800cdbb0 con 0x7f7080139920 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f7087f81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7080071310 con 0x7f706c003f30 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f7087f81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70800cdd30 con 0x7f708014bbb0 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f708551c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7080139920 0x7f708014b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f708551c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7080139920 0x7f708014b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45602/0 (socket says 172.21.15.77:45602) 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f708551c640 1 -- 172.21.15.77:0/3652432116 learned_addr learned my addr 172.21.15.77:0/3652432116 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f7085d1d640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c003f30 0x7f7080136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.257+0000 7f708651e640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f708014bbb0 0x7f7080148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f708551c640 1 -- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f708014bbb0 msgr2=0x7f7080148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f708551c640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f708014bbb0 0x7f7080148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f708551c640 1 -- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c003f30 msgr2=0x7f7080136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f708551c640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c003f30 0x7f7080136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f708551c640 1 -- 172.21.15.77:0/3652432116 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70801f2730 con 0x7f7080139920 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f708651e640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f708014bbb0 0x7f7080148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f7085d1d640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c003f30 0x7f7080136680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f708551c640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7080139920 0x7f708014b670 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f7070009be0 tx=0x7f7070004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f70767fc640 1 -- 172.21.15.77:0/3652432116 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070039050 con 0x7f7080139920 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f70767fc640 1 -- 172.21.15.77:0/3652432116 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707003a040 con 0x7f7080139920 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f70767fc640 1 -- 172.21.15.77:0/3652432116 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70700044c0 con 0x7f7080139920 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70801f2a10 con 0x7f7080139920 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.258+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7080135e80 con 0x7f7080139920 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.259+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7080068840 con 0x7f7080139920 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.260+0000 7f70767fc640 1 -- 172.21.15.77:0/3652432116 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7070047020 con 0x7f7080139920 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.260+0000 7f70767fc640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f706406b500 0x7f706406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.260+0000 7f70767fc640 1 -- 172.21.15.77:0/3652432116 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f70700aba30 con 0x7f7080139920 2024-03-06T07:32:58.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.260+0000 7f7085d1d640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f706406b500 0x7f706406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.260+0000 7f7085d1d640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f706406b500 0x7f706406d9c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f707c001bf0 tx=0x7f707c001b60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.265+0000 7f70767fc640 1 -- 172.21.15.77:0/3652432116 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f707004e020 con 0x7f7080139920 2024-03-06T07:32:58.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.480+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f708004f330 con 0x7f7080139920 2024-03-06T07:32:58.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.480+0000 7f70767fc640 1 -- 172.21.15.77:0/3652432116 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f707007aa70 con 0x7f7080139920 2024-03-06T07:32:58.482 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:32:58.482 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:32:58.482 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:32:58.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f706406b500 msgr2=0x7f706406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f706406b500 0x7f706406d9c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f707c001bf0 tx=0x7f707c001b60 comp rx=0 tx=0).stop 2024-03-06T07:32:58.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7080139920 msgr2=0x7f708014b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7080139920 0x7f708014b670 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f7070009be0 tx=0x7f7070004730 comp rx=0 tx=0).stop 2024-03-06T07:32:58.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 shutdown_connections 2024-03-06T07:32:58.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f706406b500 0x7f706406d9c0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7080139920 0x7f708014b670 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f708014bbb0 0x7f7080148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 --2- 172.21.15.77:0/3652432116 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f706c003f30 0x7f7080136680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 >> 172.21.15.77:0/3652432116 conn(0x7f70800cf8a0 msgr2=0x7f708014f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 shutdown_connections 2024-03-06T07:32:58.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:58.482+0000 7f7087f81640 1 -- 172.21.15.77:0/3652432116 wait complete. 2024-03-06T07:32:58.696 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:32:58.696 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:32:59.697 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:59.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.801+0000 7f99be639640 1 Processor -- start 2024-03-06T07:32:59.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.801+0000 7f99be639640 1 -- start start 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99be639640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815dfe0 0x7f99b80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99be639640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b80b3430 con 0x7f99b80cf830 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99be639640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b80b3710 con 0x7f99b80d29c0 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99be639640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b80b31e0 con 0x7f99b815dfe0 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b77fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f99b80cf830 0x7f99b80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41146/0 (socket says 172.21.15.77:41146) 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b77fe640 1 -- 172.21.15.77:0/1013426982 learned_addr learned my addr 172.21.15.77:0/1013426982 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232349531 0 0) 0x7f99b80b3430 con 0x7f99b80cf830 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a0003520 con 0x7f99b80cf830 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b7fff640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815dfe0 0x7f99b80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802525977 0 0) 0x7f99a0003520 con 0x7f99b80cf830 2024-03-06T07:32:59.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815dfe0 msgr2=0x7f99b80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815dfe0 0x7f99b80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 >> v1:172.21.15.158:6789/0 conn(0x7f99b80d29c0 legacy=0x7f99b80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b80b39f0 con 0x7f99b80cf830 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f99a8001670 con 0x7f99b80cf830 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 >> v1:172.21.15.77:6789/0 conn(0x7f99b80cf830 legacy=0x7f99b80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815dfe0 0x7f99a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.802+0000 7f99b6ffd640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99a0004d40 0x7f99a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b80b31e0 con 0x7f99a0003f60 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a000baa0 con 0x7f99a0004d40 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a000bca0 con 0x7f99b815dfe0 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b77fe640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b7fff640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815dfe0 0x7f99a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99bcbd6640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99a0004d40 0x7f99a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b77fe640 1 -- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815dfe0 msgr2=0x7f99a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b77fe640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815dfe0 0x7f99a00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b77fe640 1 -- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99a0004d40 msgr2=0x7f99a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b77fe640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99a0004d40 0x7f99a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99b77fe640 1 -- 172.21.15.77:0/1013426982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a0003520 con 0x7f99a0003f60 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.803+0000 7f99bcbd6640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99a0004d40 0x7f99a00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.804+0000 7f99b77fe640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99a0004800 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c210 tx=0x7f99a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6ffadf11c056e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.804+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8007d10 con 0x7f99a0003f60 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.804+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a8005ad0 con 0x7f99a0003f60 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.804+0000 7f99b6ffd640 1 -- 172.21.15.77:0/1013426982 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8007070 con 0x7f99a0003f60 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.804+0000 7f99be639640 1 -- 172.21.15.77:0/1013426982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 msgr2=0x7f99a0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.805+0000 7f99be639640 1 --2- 172.21.15.77:0/1013426982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99a0004800 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c210 tx=0x7f99a0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:59.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.805+0000 7f99bcbd6640 1 -- 172.21.15.77:0/1013426982 reap_dead start 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.805+0000 7f99be639640 1 -- 172.21.15.77:0/1013426982 shutdown_connections 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.805+0000 7f99be639640 1 -- 172.21.15.77:0/1013426982 >> 172.21.15.77:0/1013426982 conn(0x7f99b80d3cd0 msgr2=0x7f99b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.805+0000 7f99be639640 1 -- 172.21.15.77:0/1013426982 shutdown_connections 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.805+0000 7f99be639640 1 -- 172.21.15.77:0/1013426982 wait complete. 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.806+0000 7f99be639640 1 Processor -- start 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.806+0000 7f99be639640 1 -- start start 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99be639640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99b814c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99be639640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815e770 0x7f99b815fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99be639640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b80b8480 0x7f99b80b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99be639640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b815e3c0 con 0x7f99a0003f60 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99be639640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b80b3d60 con 0x7f99b80b8480 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99be639640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b815e540 con 0x7f99b815e770 2024-03-06T07:32:59.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99b814c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99b814c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41162/0 (socket says 172.21.15.77:41162) 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 -- 172.21.15.77:0/1453364600 learned_addr learned my addr 172.21.15.77:0/1453364600 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b77fe640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815e770 0x7f99b815fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 -- 172.21.15.77:0/1453364600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815e770 msgr2=0x7f99b815fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815e770 0x7f99b815fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 -- 172.21.15.77:0/1453364600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b80b8480 msgr2=0x7f99b80b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99bcbd6640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b80b8480 0x7f99b80b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b80b8480 0x7f99b80b8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 -- 172.21.15.77:0/1453364600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b8160250 con 0x7f99a0003f60 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b7fff640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99b814c280 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0119a0 tx=0x7f99ac011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1453364600 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ac00faf0 con 0x7f99a0003f60 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1453364600 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ac00fc90 con 0x7f99a0003f60 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b815ec70 con 0x7f99a0003f60 2024-03-06T07:32:59.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1453364600 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ac014440 con 0x7f99a0003f60 2024-03-06T07:32:59.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.807+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b815f230 con 0x7f99a0003f60 2024-03-06T07:32:59.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.808+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b804f330 con 0x7f99a0003f60 2024-03-06T07:32:59.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.809+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1453364600 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99ac012040 con 0x7f99a0003f60 2024-03-06T07:32:59.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.809+0000 7f99b4ff9640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f99a406b500 0x7f99a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.809+0000 7f99b77fe640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f99a406b500 0x7f99a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.809+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1453364600 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f99ac08a8f0 con 0x7f99a0003f60 2024-03-06T07:32:59.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.810+0000 7f99b77fe640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f99a406b500 0x7f99a406d9c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f99a803c000 tx=0x7f99a8037000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:32:59.818+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1453364600 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99ac019030 con 0x7f99a0003f60 2024-03-06T07:33:00.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.027+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99b80b5440 con 0x7f99a0003f60 2024-03-06T07:33:00.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.028+0000 7f99b4ff9640 1 -- 172.21.15.77:0/1453364600 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f99ac059750 con 0x7f99a0003f60 2024-03-06T07:33:00.030 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:00.030 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:00.030 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:00.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f99a406b500 msgr2=0x7f99a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f99a406b500 0x7f99a406d9c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f99a803c000 tx=0x7f99a8037000 comp rx=0 tx=0).stop 2024-03-06T07:33:00.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 msgr2=0x7f99b814c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99b814c280 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0119a0 tx=0x7f99ac011e60 comp rx=0 tx=0).stop 2024-03-06T07:33:00.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 shutdown_connections 2024-03-06T07:33:00.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f99b80b8480 0x7f99b80b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f99b815e770 0x7f99b815fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f99a406b500 0x7f99a406d9c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 --2- 172.21.15.77:0/1453364600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99a0003f60 0x7f99b814c280 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.029+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 >> 172.21.15.77:0/1453364600 conn(0x7f99b80d3cd0 msgr2=0x7f99b80ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.030+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 shutdown_connections 2024-03-06T07:33:00.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:00.030+0000 7f99be639640 1 -- 172.21.15.77:0/1453364600 wait complete. 2024-03-06T07:33:00.243 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:00.244 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:01.245 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.348+0000 7f3837257640 1 Processor -- start 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.348+0000 7f3837257640 1 -- start start 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.348+0000 7f3837257640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38300d2710 0x7f38300b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.348+0000 7f3837257640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300b3430 con 0x7f383015dfe0 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.348+0000 7f3837257640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300b3710 con 0x7f38300d22b0 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.348+0000 7f3837257640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38300b31e0 con 0x7f38300d2710 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.349+0000 7f38357f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38300d2710 0x7f38300b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.349+0000 7f38357f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38300d2710 0x7f38300b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41170/0 (socket says 172.21.15.77:41170) 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.349+0000 7f3834ff3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f383015dfe0 0x7f38300cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41148/0 (socket says 172.21.15.77:41148) 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.349+0000 7f38357f4640 1 -- 172.21.15.77:0/1783661583 learned_addr learned my addr 172.21.15.77:0/1783661583 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.349+0000 7f38357f4640 1 -- 172.21.15.77:0/1783661583 >> v1:172.21.15.158:6789/0 conn(0x7f38300d22b0 legacy=0x7f38300b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.349+0000 7f38357f4640 1 -- 172.21.15.77:0/1783661583 >> v1:172.21.15.77:6789/0 conn(0x7f383015dfe0 legacy=0x7f38300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.349+0000 7f38357f4640 1 -- 172.21.15.77:0/1783661583 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300b39f0 con 0x7f38300d2710 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.349+0000 7f382f7fe640 1 -- 172.21.15.77:0/1783661583 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673229874 0 0) 0x7f38300b3430 con 0x7f383015dfe0 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f38357f4640 1 --2- 172.21.15.77:0/1783661583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d2710 0x7f38300b4900 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f3828009e80 tx=0x7f382802bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6fc3c9bd7efcfec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f382f7fe640 1 -- 172.21.15.77:0/1783661583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382802cb80 con 0x7f38300d2710 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f382f7fe640 1 -- 172.21.15.77:0/1783661583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382802cd20 con 0x7f38300d2710 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f382f7fe640 1 -- 172.21.15.77:0/1783661583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828034610 con 0x7f38300d2710 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f3837257640 1 -- 172.21.15.77:0/1783661583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d2710 msgr2=0x7f38300b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f3837257640 1 --2- 172.21.15.77:0/1783661583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d2710 0x7f38300b4900 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f3828009e80 tx=0x7f382802bfe0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f3837257640 1 -- 172.21.15.77:0/1783661583 shutdown_connections 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f3837257640 1 --2- 172.21.15.77:0/1783661583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d2710 0x7f38300b4900 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f3837257640 1 -- 172.21.15.77:0/1783661583 >> 172.21.15.77:0/1783661583 conn(0x7f38300d3cd0 msgr2=0x7f38300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f3837257640 1 -- 172.21.15.77:0/1783661583 shutdown_connections 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.350+0000 7f3837257640 1 -- 172.21.15.77:0/1783661583 wait complete. 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.352+0000 7f3837257640 1 Processor -- start 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.352+0000 7f3837257640 1 -- start start 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3837257640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d22b0 0x7f3830157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3837257640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38300d2710 0x7f38300b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3837257640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f383015dfe0 0x7f38300b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3837257640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38300cfa00 con 0x7f38300d22b0 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3837257640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38300b3d60 con 0x7f38300d2710 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3837257640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38300d1c70 con 0x7f383015dfe0 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d22b0 0x7f3830157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d22b0 0x7f3830157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41184/0 (socket says 172.21.15.77:41184) 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 -- 172.21.15.77:0/1713604886 learned_addr learned my addr 172.21.15.77:0/1713604886 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f382ffff640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38300d2710 0x7f38300b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f38357f4640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f383015dfe0 0x7f38300b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 -- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f383015dfe0 msgr2=0x7f38300b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f383015dfe0 0x7f38300b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 -- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38300d2710 msgr2=0x7f38300b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38300d2710 0x7f38300b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 -- 172.21.15.77:0/1713604886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300b8bb0 con 0x7f38300d22b0 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f382ffff640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38300d2710 0x7f38300b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3834ff3640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d22b0 0x7f3830157b30 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f382400c420 tx=0x7f382400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f382d7fa640 1 -- 172.21.15.77:0/1713604886 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38240074e0 con 0x7f38300d22b0 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f382d7fa640 1 -- 172.21.15.77:0/1713604886 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38240047e0 con 0x7f38300d22b0 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38300b8e90 con 0x7f38300d22b0 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f382d7fa640 1 -- 172.21.15.77:0/1713604886 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824004c10 con 0x7f38300d22b0 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.353+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38300b9450 con 0x7f38300d22b0 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.355+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383004f330 con 0x7f38300d22b0 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.355+0000 7f382d7fa640 1 -- 172.21.15.77:0/1713604886 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3824011070 con 0x7f38300d22b0 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.355+0000 7f382d7fa640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.355+0000 7f382ffff640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.355+0000 7f382d7fa640 1 -- 172.21.15.77:0/1713604886 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3824089de0 con 0x7f38300d22b0 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.356+0000 7f382ffff640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f383015d650 tx=0x7f381c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.359+0000 7f382d7fa640 1 -- 172.21.15.77:0/1713604886 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f382400e280 con 0x7f38300d22b0 2024-03-06T07:33:01.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.575+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38300ce960 con 0x7f38300d22b0 2024-03-06T07:33:01.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.576+0000 7f382d7fa640 1 -- 172.21.15.77:0/1713604886 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f3824058c40 con 0x7f38300d22b0 2024-03-06T07:33:01.578 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:01.578 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:01.578 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 msgr2=0x7f381406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f383015d650 tx=0x7f381c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d22b0 msgr2=0x7f3830157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d22b0 0x7f3830157b30 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f382400c420 tx=0x7f382400c8e0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 shutdown_connections 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f383015dfe0 0x7f38300b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f38300d2710 0x7f38300b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 --2- 172.21.15.77:0/1713604886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38300d22b0 0x7f3830157b30 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 >> 172.21.15.77:0/1713604886 conn(0x7f38300d3cd0 msgr2=0x7f38300ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 shutdown_connections 2024-03-06T07:33:01.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:01.578+0000 7f3837257640 1 -- 172.21.15.77:0/1713604886 wait complete. 2024-03-06T07:33:01.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:01.792 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:02.312 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:02.314 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:02.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:02.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.891+0000 7fdaf668f640 1 Processor -- start 2024-03-06T07:33:02.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.891+0000 7fdaf668f640 1 -- start start 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaf668f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdaf00b4f70 0x7fdaf00b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaf668f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf006c6d0 0x7fdaf015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaf668f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdaf0066d90 0x7fdaf00ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaf668f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf015e9f0 con 0x7fdaf00b4f70 2024-03-06T07:33:02.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaf668f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf00cda10 con 0x7fdaf0066d90 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaf668f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00b5890 con 0x7fdaf006c6d0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaeffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdaf00b4f70 0x7fdaf00b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaeffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdaf00b4f70 0x7fdaf00b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41194/0 (socket says 172.21.15.77:41194) 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaeffff640 1 -- 172.21.15.77:0/3307274534 learned_addr learned my addr 172.21.15.77:0/3307274534 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaef7fe640 1 --2- 172.21.15.77:0/3307274534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf006c6d0 0x7fdaf015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaf4c2c640 1 --2- 172.21.15.77:0/3307274534 >> v2:172.21.15.158:3300/0 conn(0x7fdaf0066d90 0x7fdaf00ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaeffff640 1 -- 172.21.15.77:0/3307274534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf006c6d0 msgr2=0x7fdaf015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaeffff640 1 --2- 172.21.15.77:0/3307274534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf006c6d0 0x7fdaf015ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaeffff640 1 -- 172.21.15.77:0/3307274534 >> v2:172.21.15.158:3300/0 conn(0x7fdaf0066d90 msgr2=0x7fdaf00ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaeffff640 1 --2- 172.21.15.77:0/3307274534 >> v2:172.21.15.158:3300/0 conn(0x7fdaf0066d90 0x7fdaf00ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.892+0000 7fdaeffff640 1 -- 172.21.15.77:0/3307274534 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf00cd450 con 0x7fdaf00b4f70 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.893+0000 7fdaeffff640 1 --2- 172.21.15.77:0/3307274534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf00b4f70 0x7fdaf00b5350 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00723c0 tx=0x7fdae402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4091804f9eb90e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.893+0000 7fdaeeffd640 1 -- 172.21.15.77:0/3307274534 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae4035020 con 0x7fdaf00b4f70 2024-03-06T07:33:02.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.893+0000 7fdaeeffd640 1 -- 172.21.15.77:0/3307274534 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae4031870 con 0x7fdaf00b4f70 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.893+0000 7fdaeeffd640 1 -- 172.21.15.77:0/3307274534 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae4031c40 con 0x7fdaf00b4f70 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 -- 172.21.15.77:0/3307274534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf00b4f70 msgr2=0x7fdaf00b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3307274534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf00b4f70 0x7fdaf00b5350 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00723c0 tx=0x7fdae402b910 comp rx=0 tx=0).stop 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 -- 172.21.15.77:0/3307274534 shutdown_connections 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3307274534 >> v2:172.21.15.158:3300/0 conn(0x7fdaf0066d90 0x7fdaf00ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3307274534 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf006c6d0 0x7fdaf015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3307274534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf00b4f70 0x7fdaf00b5350 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 -- 172.21.15.77:0/3307274534 >> 172.21.15.77:0/3307274534 conn(0x7fdaf00724d0 msgr2=0x7fdaf00728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 -- 172.21.15.77:0/3307274534 shutdown_connections 2024-03-06T07:33:02.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.894+0000 7fdaf668f640 1 -- 172.21.15.77:0/3307274534 wait complete. 2024-03-06T07:33:02.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.895+0000 7fdaf668f640 1 Processor -- start 2024-03-06T07:33:02.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.895+0000 7fdaf668f640 1 -- start start 2024-03-06T07:33:02.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf668f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf0066d90 0x7fdaf015a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf668f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf00b4f70 0x7fdaf015c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf668f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf015ce90 0x7fdaf0156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf668f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf015f3c0 con 0x7fdaf015ce90 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf668f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf015e680 con 0x7fdaf00b4f70 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf668f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00cd090 con 0x7fdaf0066d90 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf4c2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf015ce90 0x7fdaf0156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf4c2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf015ce90 0x7fdaf0156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41210/0 (socket says 172.21.15.77:41210) 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf4c2c640 1 -- 172.21.15.77:0/3781317595 learned_addr learned my addr 172.21.15.77:0/3781317595 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaeffff640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf0066d90 0x7fdaf015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaef7fe640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf00b4f70 0x7fdaf015c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf4c2c640 1 -- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf0066d90 msgr2=0x7fdaf015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf4c2c640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf0066d90 0x7fdaf015a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf4c2c640 1 -- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf00b4f70 msgr2=0x7fdaf015c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf4c2c640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf00b4f70 0x7fdaf015c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaf4c2c640 1 -- 172.21.15.77:0/3781317595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf0157a40 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.896+0000 7fdaeffff640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf0066d90 0x7fdaf015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.897+0000 7fdaf4c2c640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf015ce90 0x7fdaf0156bb0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fdae8007d10 tx=0x7fdae800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.897+0000 7fdaecff9640 1 -- 172.21.15.77:0/3781317595 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8014420 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.897+0000 7fdaecff9640 1 -- 172.21.15.77:0/3781317595 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8019070 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.897+0000 7fdaecff9640 1 -- 172.21.15.77:0/3781317595 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8014800 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.897+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf01e78f0 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.897+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf015d810 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.898+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf004f330 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.898+0000 7fdaecff9640 1 -- 172.21.15.77:0/3781317595 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdae80149a0 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.899+0000 7fdaecff9640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdadc06b4d0 0x7fdadc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.899+0000 7fdaecff9640 1 -- 172.21.15.77:0/3781317595 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdae808a9d0 con 0x7fdaf015ce90 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.899+0000 7fdaeffff640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdadc06b4d0 0x7fdadc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.899+0000 7fdaeffff640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdadc06b4d0 0x7fdadc06d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdae4036000 tx=0x7fdae403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:02.903+0000 7fdaecff9640 1 -- 172.21.15.77:0/3781317595 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdaf0157a40 con 0x7fdaf015ce90 2024-03-06T07:33:03.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.118+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaf00b1860 con 0x7fdaf015ce90 2024-03-06T07:33:03.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.119+0000 7fdaecff9640 1 -- 172.21.15.77:0/3781317595 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fdae8059830 con 0x7fdaf015ce90 2024-03-06T07:33:03.121 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:03.121 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:03.121 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:03.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdadc06b4d0 msgr2=0x7fdadc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdadc06b4d0 0x7fdadc06d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdae4036000 tx=0x7fdae403c040 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf015ce90 msgr2=0x7fdaf0156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf015ce90 0x7fdaf0156bb0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fdae8007d10 tx=0x7fdae800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 shutdown_connections 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdadc06b4d0 0x7fdadc06d990 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf015ce90 0x7fdaf0156bb0 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaf00b4f70 0x7fdaf015c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 --2- 172.21.15.77:0/3781317595 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaf0066d90 0x7fdaf015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.121+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 >> 172.21.15.77:0/3781317595 conn(0x7fdaf00724d0 msgr2=0x7fdaf00c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.122+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 shutdown_connections 2024-03-06T07:33:03.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:03.122+0000 7fdaf668f640 1 -- 172.21.15.77:0/3781317595 wait complete. 2024-03-06T07:33:03.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:03.336 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:04.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:04.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.437+0000 7fbff34eb640 1 Processor -- start 2024-03-06T07:33:04.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.437+0000 7fbff34eb640 1 -- start start 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.438+0000 7fbff34eb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbfec15dfe0 0x7fbfec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.438+0000 7fbff34eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec0cf830 0x7fbfec0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.438+0000 7fbff34eb640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec0b3430 con 0x7fbfec0d27f0 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.438+0000 7fbff34eb640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec0d2c50 con 0x7fbfec15dfe0 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.438+0000 7fbff34eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0b31e0 con 0x7fbfec0cf830 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.438+0000 7fbff0a86640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec0cf830 0x7fbfec0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.438+0000 7fbff1287640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbfec15dfe0 0x7fbfec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.439+0000 7fbff0a86640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec0cf830 0x7fbfec0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:48672/0 (socket says 172.21.15.77:48672) 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.439+0000 7fbff0a86640 1 -- 172.21.15.77:0/2995058320 learned_addr learned my addr 172.21.15.77:0/2995058320 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:04.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.439+0000 7fbff1a88640 1 --1- 172.21.15.77:0/2995058320 >> v1:172.21.15.158:6789/0 conn(0x7fbfec0d27f0 0x7fbfec0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:38362/0 (socket says 172.21.15.77:38362) 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.439+0000 7fbff1287640 1 -- 172.21.15.77:0/2995058320 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec0cf830 msgr2=0x7fbfec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.439+0000 7fbff1287640 1 --2- 172.21.15.77:0/2995058320 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec0cf830 0x7fbfec0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.439+0000 7fbff1287640 1 -- 172.21.15.77:0/2995058320 >> v1:172.21.15.158:6789/0 conn(0x7fbfec0d27f0 legacy=0x7fbfec0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.439+0000 7fbff1287640 1 -- 172.21.15.77:0/2995058320 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec0b3710 con 0x7fbfec15dfe0 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff1287640 1 --2- 172.21.15.77:0/2995058320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec15dfe0 0x7fbfec0cf2f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80097a0 tx=0x7fbfe802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82d54fb67975c000 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbfe3fff640 1 -- 172.21.15.77:0/2995058320 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8035020 con 0x7fbfec15dfe0 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbfe3fff640 1 -- 172.21.15.77:0/2995058320 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe8031830 con 0x7fbfec15dfe0 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbfe3fff640 1 -- 172.21.15.77:0/2995058320 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8031c00 con 0x7fbfec15dfe0 2024-03-06T07:33:04.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff34eb640 1 -- 172.21.15.77:0/2995058320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec15dfe0 msgr2=0x7fbfec0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff34eb640 1 --2- 172.21.15.77:0/2995058320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec15dfe0 0x7fbfec0cf2f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80097a0 tx=0x7fbfe802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff34eb640 1 -- 172.21.15.77:0/2995058320 shutdown_connections 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff34eb640 1 --2- 172.21.15.77:0/2995058320 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec0cf830 0x7fbfec0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff34eb640 1 --2- 172.21.15.77:0/2995058320 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec15dfe0 0x7fbfec0cf2f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff34eb640 1 -- 172.21.15.77:0/2995058320 >> 172.21.15.77:0/2995058320 conn(0x7fbfec0d3cd0 msgr2=0x7fbfec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff34eb640 1 -- 172.21.15.77:0/2995058320 shutdown_connections 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.440+0000 7fbff34eb640 1 -- 172.21.15.77:0/2995058320 wait complete. 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff34eb640 1 Processor -- start 2024-03-06T07:33:04.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff34eb640 1 -- start start 2024-03-06T07:33:04.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff34eb640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec0cf830 0x7fbfec1f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff34eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfec0d27f0 0x7fbfec14d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff34eb640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec15dfe0 0x7fbfec14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff34eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0d1c70 con 0x7fbfec0d27f0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff34eb640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0b3d60 con 0x7fbfec0cf830 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff34eb640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0d1df0 con 0x7fbfec15dfe0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff0a86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfec0d27f0 0x7fbfec14d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff0a86640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfec0d27f0 0x7fbfec14d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41224/0 (socket says 172.21.15.77:41224) 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff0a86640 1 -- 172.21.15.77:0/1689704088 learned_addr learned my addr 172.21.15.77:0/1689704088 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff1a88640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec15dfe0 0x7fbfec14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.442+0000 7fbff1287640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec0cf830 0x7fbfec1f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbff0a86640 1 -- 172.21.15.77:0/1689704088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec15dfe0 msgr2=0x7fbfec14d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbff0a86640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec15dfe0 0x7fbfec14d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbff0a86640 1 -- 172.21.15.77:0/1689704088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec0cf830 msgr2=0x7fbfec1f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbff0a86640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec0cf830 0x7fbfec1f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbff0a86640 1 -- 172.21.15.77:0/1689704088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec1f40b0 con 0x7fbfec0d27f0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbff0a86640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfec0d27f0 0x7fbfec14d320 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00e900 tx=0x7fbfdc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbfe1ffb640 1 -- 172.21.15.77:0/1689704088 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc017070 con 0x7fbfec0d27f0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbfe1ffb640 1 -- 172.21.15.77:0/1689704088 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc00ca00 con 0x7fbfec0d27f0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec1f4390 con 0x7fbfec0d27f0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbfe1ffb640 1 -- 172.21.15.77:0/1689704088 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00cdd0 con 0x7fbfec0d27f0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.443+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec14ce00 con 0x7fbfec0d27f0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.445+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfec04f330 con 0x7fbfec0d27f0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.445+0000 7fbfe1ffb640 1 -- 172.21.15.77:0/1689704088 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfdc01a070 con 0x7fbfec0d27f0 2024-03-06T07:33:04.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.445+0000 7fbfe1ffb640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfd006b4d0 0x7fbfd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.445+0000 7fbfe1ffb640 1 -- 172.21.15.77:0/1689704088 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbfdc08a4b0 con 0x7fbfec0d27f0 2024-03-06T07:33:04.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.445+0000 7fbff1287640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfd006b4d0 0x7fbfd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.446+0000 7fbff1287640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfd006b4d0 0x7fbfd006d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8036000 tx=0x7fbfe803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.449+0000 7fbfe1ffb640 1 -- 172.21.15.77:0/1689704088 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfdc058d50 con 0x7fbfec0d27f0 2024-03-06T07:33:04.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.664+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfec0b0d40 con 0x7fbfec0d27f0 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.664+0000 7fbfe1ffb640 1 -- 172.21.15.77:0/1689704088 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fbfdc05cdf0 con 0x7fbfec0d27f0 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:04.666 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:04.667 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:04.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfd006b4d0 msgr2=0x7fbfd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfd006b4d0 0x7fbfd006d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8036000 tx=0x7fbfe803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfec0d27f0 msgr2=0x7fbfec14d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfec0d27f0 0x7fbfec14d320 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00e900 tx=0x7fbfdc00edc0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 shutdown_connections 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfd006b4d0 0x7fbfd006d990 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfec0d27f0 0x7fbfec14d320 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfec15dfe0 0x7fbfec14d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 --2- 172.21.15.77:0/1689704088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfec0cf830 0x7fbfec1f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 >> 172.21.15.77:0/1689704088 conn(0x7fbfec0d3cd0 msgr2=0x7fbfec0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 shutdown_connections 2024-03-06T07:33:04.669 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:04.666+0000 7fbff34eb640 1 -- 172.21.15.77:0/1689704088 wait complete. 2024-03-06T07:33:04.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:04.881 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:05.881 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:06.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.003+0000 7fa119f43640 1 Processor -- start 2024-03-06T07:33:06.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.004+0000 7fa119f43640 1 -- start start 2024-03-06T07:33:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.004+0000 7fa119f43640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140d22b0 0x7fa1140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.004+0000 7fa119f43640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa1140b41f0 0x7fa1140b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.004+0000 7fa119f43640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1140b3430 con 0x7fa11415dfe0 2024-03-06T07:33:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.004+0000 7fa119f43640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1140d2c50 con 0x7fa1140b41f0 2024-03-06T07:33:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.004+0000 7fa119f43640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140b31e0 con 0x7fa1140d22b0 2024-03-06T07:33:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.004+0000 7fa1137fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa11415dfe0 0x7fa1140cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:41158/0 (socket says 172.21.15.77:41158) 2024-03-06T07:33:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.004+0000 7fa1137fe640 1 -- 172.21.15.77:0/1552961046 learned_addr learned my addr 172.21.15.77:0/1552961046 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341816624 0 0) 0x7fa1140b3430 con 0x7fa11415dfe0 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa100003520 con 0x7fa11415dfe0 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa112ffd640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140d22b0 0x7fa1140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa113fff640 1 --2- 172.21.15.77:0/1552961046 >> v2:172.21.15.158:3300/0 conn(0x7fa1140b41f0 0x7fa1140b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2936159394 0 0) 0x7fa100003520 con 0x7fa11415dfe0 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140d22b0 msgr2=0x7fa1140d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140d22b0 0x7fa1140d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 >> v2:172.21.15.158:3300/0 conn(0x7fa1140b41f0 msgr2=0x7fa1140b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 --2- 172.21.15.77:0/1552961046 >> v2:172.21.15.158:3300/0 conn(0x7fa1140b41f0 0x7fa1140b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140b3710 con 0x7fa11415dfe0 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fa10c0019f0 con 0x7fa11415dfe0 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 >> v1:172.21.15.77:6789/0 conn(0x7fa11415dfe0 legacy=0x7fa1140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1000039b0 0x7fa100003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa100004800 0x7fa100004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140d22b0 0x7fa1000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140b31e0 con 0x7fa100004800 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10000b9c0 con 0x7fa1000039b0 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10000bb40 con 0x7fa1140d22b0 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa112ffd640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa100004800 0x7fa100004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa1137fe640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1000039b0 0x7fa100003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa113fff640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140d22b0 0x7fa1000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa112ffd640 1 -- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140d22b0 msgr2=0x7fa1000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa112ffd640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140d22b0 0x7fa1000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa112ffd640 1 -- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1000039b0 msgr2=0x7fa100003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa112ffd640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa1000039b0 0x7fa100003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.005+0000 7fa112ffd640 1 -- 172.21.15.77:0/1552961046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100003520 con 0x7fa100004800 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.006+0000 7fa112ffd640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa100004800 0x7fa100004c80 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009ac0 tx=0x7fa100005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9ee7446a47aad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.006+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc039070 con 0x7fa100004800 2024-03-06T07:33:06.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.006+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc02c6b0 con 0x7fa100004800 2024-03-06T07:33:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.006+0000 7fa1127fc640 1 -- 172.21.15.77:0/1552961046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc009690 con 0x7fa100004800 2024-03-06T07:33:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.007+0000 7fa119f43640 1 -- 172.21.15.77:0/1552961046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa100004800 msgr2=0x7fa100004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.007+0000 7fa119f43640 1 --2- 172.21.15.77:0/1552961046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa100004800 0x7fa100004c80 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009ac0 tx=0x7fa100005830 comp rx=0 tx=0).stop 2024-03-06T07:33:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.007+0000 7fa113fff640 1 -- 172.21.15.77:0/1552961046 reap_dead start 2024-03-06T07:33:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.007+0000 7fa119f43640 1 -- 172.21.15.77:0/1552961046 shutdown_connections 2024-03-06T07:33:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.007+0000 7fa119f43640 1 -- 172.21.15.77:0/1552961046 >> 172.21.15.77:0/1552961046 conn(0x7fa1140d3cd0 msgr2=0x7fa1140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.007+0000 7fa119f43640 1 -- 172.21.15.77:0/1552961046 shutdown_connections 2024-03-06T07:33:06.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.007+0000 7fa119f43640 1 -- 172.21.15.77:0/1552961046 wait complete. 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.009+0000 7fa119f43640 1 Processor -- start 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.009+0000 7fa119f43640 1 -- start start 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa119f43640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa100004800 0x7fa1140b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa119f43640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140cada0 0x7fa1140cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa119f43640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa11414fd00 0x7fa1141501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa119f43640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140cfa00 con 0x7fa11414fd00 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa119f43640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140b3d60 con 0x7fa100004800 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa119f43640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140d1c70 con 0x7fa1140cada0 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa113fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa11414fd00 0x7fa1141501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa113fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa11414fd00 0x7fa1141501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41246/0 (socket says 172.21.15.77:41246) 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa113fff640 1 -- 172.21.15.77:0/397469129 learned_addr learned my addr 172.21.15.77:0/397469129 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa1137fe640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa100004800 0x7fa1140b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.010+0000 7fa112ffd640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140cada0 0x7fa1140cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa113fff640 1 -- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140cada0 msgr2=0x7fa1140cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa113fff640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140cada0 0x7fa1140cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa113fff640 1 -- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa100004800 msgr2=0x7fa1140b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa113fff640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa100004800 0x7fa1140b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa113fff640 1 -- 172.21.15.77:0/397469129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140cb560 con 0x7fa11414fd00 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa112ffd640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140cada0 0x7fa1140cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa1137fe640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa100004800 0x7fa1140b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa113fff640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa11414fd00 0x7fa1141501b0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fa104006fd0 tx=0x7fa10400f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa10bfff640 1 -- 172.21.15.77:0/397469129 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104004440 con 0x7fa11414fd00 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa10bfff640 1 -- 172.21.15.77:0/397469129 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104019070 con 0x7fa11414fd00 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa10bfff640 1 -- 172.21.15.77:0/397469129 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104003850 con 0x7fa11414fd00 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1140cb840 con 0x7fa11414fd00 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.011+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1140cbe00 con 0x7fa11414fd00 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.012+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1140ce3c0 con 0x7fa11414fd00 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.012+0000 7fa10bfff640 1 -- 172.21.15.77:0/397469129 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa104012070 con 0x7fa11414fd00 2024-03-06T07:33:06.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.013+0000 7fa10bfff640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa0f406b500 0x7fa0f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.013+0000 7fa1137fe640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa0f406b500 0x7fa0f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.013+0000 7fa10bfff640 1 -- 172.21.15.77:0/397469129 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa104089b60 con 0x7fa11414fd00 2024-03-06T07:33:06.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.014+0000 7fa1137fe640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa0f406b500 0x7fa0f406d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa1140ce2f0 tx=0x7fa10c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.017+0000 7fa10bfff640 1 -- 172.21.15.77:0/397469129 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1140ce3c0 con 0x7fa11414fd00 2024-03-06T07:33:06.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.225+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa11404f330 con 0x7fa11414fd00 2024-03-06T07:33:06.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.226+0000 7fa10bfff640 1 -- 172.21.15.77:0/397469129 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fa1040589c0 con 0x7fa11414fd00 2024-03-06T07:33:06.228 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:06.228 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:06.228 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa0f406b500 msgr2=0x7fa0f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa0f406b500 0x7fa0f406d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa1140ce2f0 tx=0x7fa10c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa11414fd00 msgr2=0x7fa1141501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa11414fd00 0x7fa1141501b0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fa104006fd0 tx=0x7fa10400f8a0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 shutdown_connections 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa0f406b500 0x7fa0f406d9c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa11414fd00 0x7fa1141501b0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa1140cada0 0x7fa1140cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 --2- 172.21.15.77:0/397469129 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa100004800 0x7fa1140b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 >> 172.21.15.77:0/397469129 conn(0x7fa1140d3cd0 msgr2=0x7fa11406cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 shutdown_connections 2024-03-06T07:33:06.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:06.228+0000 7fa119f43640 1 -- 172.21.15.77:0/397469129 wait complete. 2024-03-06T07:33:06.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:06.444 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:07.445 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:07.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.545+0000 7f6d7340e640 1 Processor -- start 2024-03-06T07:33:07.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.545+0000 7f6d7340e640 1 -- start start 2024-03-06T07:33:07.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.545+0000 7f6d7340e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d6c15bdd0 0x7f6d6c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.545+0000 7f6d7340e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6d6c15fa10 0x7f6d6c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d7340e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c0d16e0 0x7f6d6c0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d7340e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6c0cebe0 con 0x7f6d6c15bdd0 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d7340e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6c0ceec0 con 0x7f6d6c15fa10 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d7340e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c0cf1a0 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d711aa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d6c15bdd0 0x7f6d6c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d711aa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d6c15bdd0 0x7f6d6c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41252/0 (socket says 172.21.15.77:41252) 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d711aa640 1 -- 172.21.15.77:0/382812827 learned_addr learned my addr 172.21.15.77:0/382812827 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d711aa640 1 -- 172.21.15.77:0/382812827 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c0d16e0 msgr2=0x7f6d6c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d719ab640 1 --2- 172.21.15.77:0/382812827 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c0d16e0 0x7f6d6c0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d709a9640 1 --2- 172.21.15.77:0/382812827 >> v2:172.21.15.158:3300/0 conn(0x7f6d6c15fa10 0x7f6d6c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d711aa640 1 --2- 172.21.15.77:0/382812827 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c0d16e0 0x7f6d6c0d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d711aa640 1 -- 172.21.15.77:0/382812827 >> v2:172.21.15.158:3300/0 conn(0x7f6d6c15fa10 msgr2=0x7f6d6c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d711aa640 1 --2- 172.21.15.77:0/382812827 >> v2:172.21.15.158:3300/0 conn(0x7f6d6c15fa10 0x7f6d6c0ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d711aa640 1 -- 172.21.15.77:0/382812827 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c0cc8e0 con 0x7f6d6c15bdd0 2024-03-06T07:33:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d709a9640 1 --2- 172.21.15.77:0/382812827 >> v2:172.21.15.158:3300/0 conn(0x7f6d6c15fa10 0x7f6d6c0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.546+0000 7f6d719ab640 1 --2- 172.21.15.77:0/382812827 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c0d16e0 0x7f6d6c0d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d711aa640 1 --2- 172.21.15.77:0/382812827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c15bdd0 0x7f6d6c0c5610 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c0cdfd0 tx=0x7f6d5c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9abbb922ca34d8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d6bfff640 1 -- 172.21.15.77:0/382812827 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c02c9e0 con 0x7f6d6c15bdd0 2024-03-06T07:33:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d6bfff640 1 -- 172.21.15.77:0/382812827 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c02cb80 con 0x7f6d6c15bdd0 2024-03-06T07:33:07.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d6bfff640 1 -- 172.21.15.77:0/382812827 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0343f0 con 0x7f6d6c15bdd0 2024-03-06T07:33:07.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d7340e640 1 -- 172.21.15.77:0/382812827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c15bdd0 msgr2=0x7f6d6c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d7340e640 1 --2- 172.21.15.77:0/382812827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c15bdd0 0x7f6d6c0c5610 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c0cdfd0 tx=0x7f6d5c02be40 comp rx=0 tx=0).stop 2024-03-06T07:33:07.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d7340e640 1 -- 172.21.15.77:0/382812827 shutdown_connections 2024-03-06T07:33:07.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d7340e640 1 --2- 172.21.15.77:0/382812827 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c0d16e0 0x7f6d6c0d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d7340e640 1 --2- 172.21.15.77:0/382812827 >> v2:172.21.15.158:3300/0 conn(0x7f6d6c15fa10 0x7f6d6c0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d7340e640 1 --2- 172.21.15.77:0/382812827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c15bdd0 0x7f6d6c0c5610 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.547+0000 7f6d7340e640 1 -- 172.21.15.77:0/382812827 >> 172.21.15.77:0/382812827 conn(0x7f6d6c09a9f0 msgr2=0x7f6d6c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.548+0000 7f6d7340e640 1 -- 172.21.15.77:0/382812827 shutdown_connections 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.548+0000 7f6d7340e640 1 -- 172.21.15.77:0/382812827 wait complete. 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.549+0000 7f6d7340e640 1 Processor -- start 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.549+0000 7f6d7340e640 1 -- start start 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.549+0000 7f6d7340e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c0d16e0 0x7f6d6c154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.549+0000 7f6d7340e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d6c15bdd0 0x7f6d6c0b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.549+0000 7f6d7340e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c15fa10 0x7f6d6c0b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.549+0000 7f6d7340e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c0cc5d0 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.549+0000 7f6d7340e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c0d2050 con 0x7f6d6c15bdd0 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.549+0000 7f6d7340e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c15c1b0 con 0x7f6d6c15fa10 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c0d16e0 0x7f6d6c154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c0d16e0 0x7f6d6c154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41262/0 (socket says 172.21.15.77:41262) 2024-03-06T07:33:07.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 -- 172.21.15.77:0/2675613819 learned_addr learned my addr 172.21.15.77:0/2675613819 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d709a9640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d6c15bdd0 0x7f6d6c0b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d719ab640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c15fa10 0x7f6d6c0b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 -- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c15fa10 msgr2=0x7f6d6c0b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c15fa10 0x7f6d6c0b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 -- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d6c15bdd0 msgr2=0x7f6d6c0b4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d6c15bdd0 0x7f6d6c0b4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 -- 172.21.15.77:0/2675613819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c0ca800 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d709a9640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d6c15bdd0 0x7f6d6c0b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d719ab640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c15fa10 0x7f6d6c0b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d711aa640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c0d16e0 0x7f6d6c154e10 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c000f80 tx=0x7f6d5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d69ffb640 1 -- 172.21.15.77:0/2675613819 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c043020 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d69ffb640 1 -- 172.21.15.77:0/2675613819 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c03e450 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6c0caae0 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d69ffb640 1 -- 172.21.15.77:0/2675613819 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c03e840 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.550+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6c154910 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.552+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6c04f330 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.552+0000 7f6d69ffb640 1 -- 172.21.15.77:0/2675613819 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d5c046020 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.552+0000 7f6d69ffb640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d5806b4d0 0x7f6d5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:07.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.553+0000 7f6d709a9640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d5806b4d0 0x7f6d5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.553+0000 7f6d69ffb640 1 -- 172.21.15.77:0/2675613819 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6d5c0acaf0 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.553+0000 7f6d709a9640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d5806b4d0 0x7f6d5806d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f6d60006fd0 tx=0x7f6d60008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:07.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.557+0000 7f6d69ffb640 1 -- 172.21.15.77:0/2675613819 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d5c04d030 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.764+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d6c0cbc30 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.765+0000 7f6d69ffb640 1 -- 172.21.15.77:0/2675613819 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f6d5c07b9e0 con 0x7f6d6c0d16e0 2024-03-06T07:33:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:07.767 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:07.767 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:07.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d5806b4d0 msgr2=0x7f6d5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d5806b4d0 0x7f6d5806d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f6d60006fd0 tx=0x7f6d60008040 comp rx=0 tx=0).stop 2024-03-06T07:33:07.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c0d16e0 msgr2=0x7f6d6c154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c0d16e0 0x7f6d6c154e10 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c000f80 tx=0x7f6d5c004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:07.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 shutdown_connections 2024-03-06T07:33:07.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d6c15fa10 0x7f6d6c0b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d6c15bdd0 0x7f6d6c0b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d5806b4d0 0x7f6d5806d990 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 --2- 172.21.15.77:0/2675613819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d6c0d16e0 0x7f6d6c154e10 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 >> 172.21.15.77:0/2675613819 conn(0x7f6d6c09a9f0 msgr2=0x7f6d6c0cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 shutdown_connections 2024-03-06T07:33:07.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:07.767+0000 7f6d7340e640 1 -- 172.21.15.77:0/2675613819 wait complete. 2024-03-06T07:33:07.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:07.981 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:08.982 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:09.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.086+0000 7f0f24217640 1 Processor -- start 2024-03-06T07:33:09.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.087+0000 7f0f24217640 1 -- start start 2024-03-06T07:33:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.087+0000 7f0f24217640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c0d2710 0x7f0f1c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.087+0000 7f0f24217640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c0b3430 con 0x7f0f1c0d22b0 2024-03-06T07:33:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.087+0000 7f0f24217640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c0b3710 con 0x7f0f1c15dfe0 2024-03-06T07:33:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.087+0000 7f0f24217640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0b31e0 con 0x7f0f1c0d2710 2024-03-06T07:33:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f217b2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0f1c0d22b0 0x7f0f1c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33146/0 (socket says 172.21.15.77:33146) 2024-03-06T07:33:09.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f217b2640 1 -- 172.21.15.77:0/2315173257 learned_addr learned my addr 172.21.15.77:0/2315173257 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f227b4640 1 --2- 172.21.15.77:0/2315173257 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c0d2710 0x7f0f1c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f20fb1640 1 -- 172.21.15.77:0/2315173257 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102041479 0 0) 0x7f0f1c0b3430 con 0x7f0f1c0d22b0 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f20fb1640 1 -- 172.21.15.77:0/2315173257 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f08003520 con 0x7f0f1c0d22b0 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f20fb1640 1 -- 172.21.15.77:0/2315173257 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450209161 0 0) 0x7f0f1c0b3710 con 0x7f0f1c15dfe0 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f20fb1640 1 -- 172.21.15.77:0/2315173257 --> v1:172.21.15.158:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1c0b3430 con 0x7f0f1c15dfe0 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f20fb1640 1 -- 172.21.15.77:0/2315173257 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598308371 0 0) 0x7f0f08003520 con 0x7f0f1c0d22b0 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f227b4640 1 -- 172.21.15.77:0/2315173257 >> v1:172.21.15.158:6789/0 conn(0x7f0f1c15dfe0 legacy=0x7f0f1c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f227b4640 1 -- 172.21.15.77:0/2315173257 >> v1:172.21.15.77:6789/0 conn(0x7f0f1c0d22b0 legacy=0x7f0f1c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.088+0000 7f0f227b4640 1 -- 172.21.15.77:0/2315173257 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c0b39f0 con 0x7f0f1c0d2710 2024-03-06T07:33:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.089+0000 7f0f227b4640 1 --2- 172.21.15.77:0/2315173257 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c0d2710 0x7f0f1c0b4900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0b3360 tx=0x7f0f1402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc323c151b48338b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.089+0000 7f0f20fb1640 1 -- 172.21.15.77:0/2315173257 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14039070 con 0x7f0f1c0d2710 2024-03-06T07:33:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.089+0000 7f0f20fb1640 1 -- 172.21.15.77:0/2315173257 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1402c910 con 0x7f0f1c0d2710 2024-03-06T07:33:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.089+0000 7f0f20fb1640 1 -- 172.21.15.77:0/2315173257 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c001670 con 0x7f0f1c0d2710 2024-03-06T07:33:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.090+0000 7f0f24217640 1 -- 172.21.15.77:0/2315173257 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c0d2710 msgr2=0x7f0f1c0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.090+0000 7f0f24217640 1 --2- 172.21.15.77:0/2315173257 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c0d2710 0x7f0f1c0b4900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0b3360 tx=0x7f0f1402bde0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.090+0000 7f0f24217640 1 -- 172.21.15.77:0/2315173257 shutdown_connections 2024-03-06T07:33:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.090+0000 7f0f24217640 1 --2- 172.21.15.77:0/2315173257 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c0d2710 0x7f0f1c0b4900 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.090+0000 7f0f24217640 1 -- 172.21.15.77:0/2315173257 >> 172.21.15.77:0/2315173257 conn(0x7f0f1c0d3cd0 msgr2=0x7f0f1c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.090+0000 7f0f24217640 1 -- 172.21.15.77:0/2315173257 shutdown_connections 2024-03-06T07:33:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.090+0000 7f0f24217640 1 -- 172.21.15.77:0/2315173257 wait complete. 2024-03-06T07:33:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.092+0000 7f0f24217640 1 Processor -- start 2024-03-06T07:33:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.092+0000 7f0f24217640 1 -- start start 2024-03-06T07:33:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f24217640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f1c0d22b0 0x7f0f1c150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f24217640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f1c0d2710 0x7f0f1c1f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f24217640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c15dfe0 0x7f0f1c1f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f24217640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0cfa00 con 0x7f0f1c0d2710 2024-03-06T07:33:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f24217640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0b3d60 con 0x7f0f1c0d22b0 2024-03-06T07:33:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f24217640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0d1c70 con 0x7f0f1c15dfe0 2024-03-06T07:33:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f1c0d2710 0x7f0f1c1f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f1c0d2710 0x7f0f1c1f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41152/0 (socket says 172.21.15.77:41152) 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 -- 172.21.15.77:0/3091163613 learned_addr learned my addr 172.21.15.77:0/3091163613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f227b4640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c15dfe0 0x7f0f1c1f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f21fb3640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f1c0d22b0 0x7f0f1c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 -- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c15dfe0 msgr2=0x7f0f1c1f32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c15dfe0 0x7f0f1c1f32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 -- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f1c0d22b0 msgr2=0x7f0f1c150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f1c0d22b0 0x7f0f1c150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 -- 172.21.15.77:0/3091163613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c1f3830 con 0x7f0f1c0d2710 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f21fb3640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f1c0d22b0 0x7f0f1c150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.093+0000 7f0f217b2640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f1c0d2710 0x7f0f1c1f2db0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c780 tx=0x7f0f0c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.094+0000 7f0f127fc640 1 -- 172.21.15.77:0/3091163613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c017070 con 0x7f0f1c0d2710 2024-03-06T07:33:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.094+0000 7f0f127fc640 1 -- 172.21.15.77:0/3091163613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c005af0 con 0x7f0f1c0d2710 2024-03-06T07:33:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.094+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c0b1af0 con 0x7f0f1c0d2710 2024-03-06T07:33:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.094+0000 7f0f127fc640 1 -- 172.21.15.77:0/3091163613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c007070 con 0x7f0f1c0d2710 2024-03-06T07:33:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.094+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c0b20b0 con 0x7f0f1c0d2710 2024-03-06T07:33:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.095+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c04f330 con 0x7f0f1c0d2710 2024-03-06T07:33:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.095+0000 7f0f127fc640 1 -- 172.21.15.77:0/3091163613 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f0c01a070 con 0x7f0f1c0d2710 2024-03-06T07:33:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.096+0000 7f0f127fc640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0006b4d0 0x7f0f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.096+0000 7f0f21fb3640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0006b4d0 0x7f0f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.096+0000 7f0f127fc640 1 -- 172.21.15.77:0/3091163613 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0f0c089750 con 0x7f0f1c0d2710 2024-03-06T07:33:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.096+0000 7f0f21fb3640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0006b4d0 0x7f0f0006d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f0f180011e0 tx=0x7f0f18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.100+0000 7f0f127fc640 1 -- 172.21.15.77:0/3091163613 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f0c057f60 con 0x7f0f1c0d2710 2024-03-06T07:33:09.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.316+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f1c15c880 con 0x7f0f1c0d2710 2024-03-06T07:33:09.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.317+0000 7f0f127fc640 1 -- 172.21.15.77:0/3091163613 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f0f0c05c000 con 0x7f0f1c0d2710 2024-03-06T07:33:09.319 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:09.319 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:09.319 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:09.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0006b4d0 msgr2=0x7f0f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0006b4d0 0x7f0f0006d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f0f180011e0 tx=0x7f0f18009040 comp rx=0 tx=0).stop 2024-03-06T07:33:09.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f1c0d2710 msgr2=0x7f0f1c1f2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f1c0d2710 0x7f0f1c1f2db0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c780 tx=0x7f0f0c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:33:09.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 shutdown_connections 2024-03-06T07:33:09.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0f0006b4d0 0x7f0f0006d990 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f1c0d2710 0x7f0f1c1f2db0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f0f1c15dfe0 0x7f0f1c1f32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 --2- 172.21.15.77:0/3091163613 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0f1c0d22b0 0x7f0f1c150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 >> 172.21.15.77:0/3091163613 conn(0x7f0f1c0d3cd0 msgr2=0x7f0f1c06cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 shutdown_connections 2024-03-06T07:33:09.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:09.319+0000 7f0f24217640 1 -- 172.21.15.77:0/3091163613 wait complete. 2024-03-06T07:33:09.532 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:09.533 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:10.534 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:10.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.635+0000 7f2ed911e640 1 Processor -- start 2024-03-06T07:33:10.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.635+0000 7f2ed911e640 1 -- start start 2024-03-06T07:33:10.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed911e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2ed40d22b0 0x7f2ed40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed911e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2ed40b41f0 0x7f2ed40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed911e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40b3430 con 0x7f2ed415dfe0 2024-03-06T07:33:10.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed911e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed40d2c50 con 0x7f2ed40b41f0 2024-03-06T07:33:10.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed911e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed40b31e0 con 0x7f2ed40d22b0 2024-03-06T07:33:10.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2ed40b41f0 0x7f2ed40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed2d9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2ed415dfe0 0x7f2ed40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33150/0 (socket says 172.21.15.77:33150) 2024-03-06T07:33:10.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed2d9d640 1 -- 172.21.15.77:0/3780888621 learned_addr learned my addr 172.21.15.77:0/3780888621 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:10.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed359e640 1 -- 172.21.15.77:0/3780888621 >> v2:172.21.15.158:3300/0 conn(0x7f2ed40d22b0 msgr2=0x7f2ed40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed359e640 1 --2- 172.21.15.77:0/3780888621 >> v2:172.21.15.158:3300/0 conn(0x7f2ed40d22b0 0x7f2ed40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed359e640 1 -- 172.21.15.77:0/3780888621 >> v1:172.21.15.77:6789/0 conn(0x7f2ed415dfe0 legacy=0x7f2ed40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed359e640 1 -- 172.21.15.77:0/3780888621 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40b3710 con 0x7f2ed40b41f0 2024-03-06T07:33:10.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.636+0000 7f2ed259c640 1 -- 172.21.15.77:0/3780888621 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021401782 0 0) 0x7f2ed40b3430 con 0x7f2ed415dfe0 2024-03-06T07:33:10.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.637+0000 7f2ed359e640 1 --2- 172.21.15.77:0/3780888621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40b41f0 0x7f2ed40b4680 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009d90 tx=0x7f2ecc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be163b5a55327 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.637+0000 7f2ed259c640 1 -- 172.21.15.77:0/3780888621 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc039070 con 0x7f2ed40b41f0 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.637+0000 7f2ed259c640 1 -- 172.21.15.77:0/3780888621 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc02cc60 con 0x7f2ed40b41f0 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.637+0000 7f2ed259c640 1 -- 172.21.15.77:0/3780888621 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc034660 con 0x7f2ed40b41f0 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.638+0000 7f2ed911e640 1 -- 172.21.15.77:0/3780888621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40b41f0 msgr2=0x7f2ed40b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.638+0000 7f2ed911e640 1 --2- 172.21.15.77:0/3780888621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40b41f0 0x7f2ed40b4680 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009d90 tx=0x7f2ecc02beb0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.638+0000 7f2ed911e640 1 -- 172.21.15.77:0/3780888621 shutdown_connections 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.638+0000 7f2ed911e640 1 --2- 172.21.15.77:0/3780888621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40b41f0 0x7f2ed40b4680 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.638+0000 7f2ed911e640 1 --2- 172.21.15.77:0/3780888621 >> v2:172.21.15.158:3300/0 conn(0x7f2ed40d22b0 0x7f2ed40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.638+0000 7f2ed911e640 1 -- 172.21.15.77:0/3780888621 >> 172.21.15.77:0/3780888621 conn(0x7f2ed40d3cd0 msgr2=0x7f2ed40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.638+0000 7f2ed911e640 1 -- 172.21.15.77:0/3780888621 shutdown_connections 2024-03-06T07:33:10.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.638+0000 7f2ed911e640 1 -- 172.21.15.77:0/3780888621 wait complete. 2024-03-06T07:33:10.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.640+0000 7f2ed911e640 1 Processor -- start 2024-03-06T07:33:10.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.640+0000 7f2ed911e640 1 -- start start 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed911e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ed40b41f0 0x7f2ed415fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed911e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed40d22b0 0x7f2ed415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed911e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed415dfe0 0x7f2ed415f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed911e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40cfa00 con 0x7f2ed415dfe0 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed911e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40b3d60 con 0x7f2ed40d22b0 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed911e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40d1c70 con 0x7f2ed40b41f0 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed415dfe0 0x7f2ed415f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed415dfe0 0x7f2ed415f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41176/0 (socket says 172.21.15.77:41176) 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 -- 172.21.15.77:0/687538449 learned_addr learned my addr 172.21.15.77:0/687538449 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:10.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ecbfff640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed40d22b0 0x7f2ed415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed2d9d640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ed40b41f0 0x7f2ed415fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 -- 172.21.15.77:0/687538449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ed40b41f0 msgr2=0x7f2ed415fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ed40b41f0 0x7f2ed415fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 -- 172.21.15.77:0/687538449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed40d22b0 msgr2=0x7f2ed415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed40d22b0 0x7f2ed415ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 -- 172.21.15.77:0/687538449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed4160840 con 0x7f2ed415dfe0 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.641+0000 7f2ed359e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed415dfe0 0x7f2ed415f4d0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc037040 tx=0x7f2ecc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.642+0000 7f2ecb7fe640 1 -- 172.21.15.77:0/687538449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc044030 con 0x7f2ed415dfe0 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.642+0000 7f2ecb7fe640 1 -- 172.21.15.77:0/687538449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc02ca60 con 0x7f2ed415dfe0 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.642+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed4160b40 con 0x7f2ed415dfe0 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.642+0000 7f2ecb7fe640 1 -- 172.21.15.77:0/687538449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc039040 con 0x7f2ed415dfe0 2024-03-06T07:33:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.642+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed4161380 con 0x7f2ed415dfe0 2024-03-06T07:33:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.643+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed404f330 con 0x7f2ed415dfe0 2024-03-06T07:33:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.643+0000 7f2ecb7fe640 1 -- 172.21.15.77:0/687538449 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ecc03f470 con 0x7f2ed415dfe0 2024-03-06T07:33:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.643+0000 7f2ecb7fe640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2eb406b500 0x7f2eb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.643+0000 7f2ecb7fe640 1 -- 172.21.15.77:0/687538449 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2ecc0acc40 con 0x7f2ed415dfe0 2024-03-06T07:33:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.644+0000 7f2ed2d9d640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2eb406b500 0x7f2eb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.644+0000 7f2ed2d9d640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2eb406b500 0x7f2eb406d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4000f60 tx=0x7f2ec4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.648+0000 7f2ecb7fe640 1 -- 172.21.15.77:0/687538449 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ed4160b40 con 0x7f2ed415dfe0 2024-03-06T07:33:10.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.858+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed406bfe0 con 0x7f2ed415dfe0 2024-03-06T07:33:10.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.859+0000 7f2ecb7fe640 1 -- 172.21.15.77:0/687538449 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f2ecc07baa0 con 0x7f2ed415dfe0 2024-03-06T07:33:10.861 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:10.861 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:10.861 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2eb406b500 msgr2=0x7f2eb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2eb406b500 0x7f2eb406d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4000f60 tx=0x7f2ec4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed415dfe0 msgr2=0x7f2ed415f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed415dfe0 0x7f2ed415f4d0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc037040 tx=0x7f2ecc004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 shutdown_connections 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2eb406b500 0x7f2eb406d9c0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed415dfe0 0x7f2ed415f4d0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2ed40d22b0 0x7f2ed415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 --2- 172.21.15.77:0/687538449 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2ed40b41f0 0x7f2ed415fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 >> 172.21.15.77:0/687538449 conn(0x7f2ed40d3cd0 msgr2=0x7f2ed40b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 shutdown_connections 2024-03-06T07:33:10.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:10.861+0000 7f2ed911e640 1 -- 172.21.15.77:0/687538449 wait complete. 2024-03-06T07:33:11.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:11.075 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:12.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:12.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.175+0000 7fdaacb5e640 1 Processor -- start 2024-03-06T07:33:12.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.175+0000 7fdaacb5e640 1 -- start start 2024-03-06T07:33:12.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaacb5e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdaa815dfe0 0x7fdaa80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaacb5e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdaa80d29c0 0x7fdaa80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaacb5e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa80b3430 con 0x7fdaa80cf830 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaacb5e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa80b31e0 con 0x7fdaa80d29c0 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaacb5e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa80b3d60 con 0x7fdaa815dfe0 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa6d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdaa80d29c0 0x7fdaa80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa6d9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdaa80d29c0 0x7fdaa80b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41186/0 (socket says 172.21.15.77:41186) 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa5d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdaa80cf830 0x7fdaa80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33156/0 (socket says 172.21.15.77:33156) 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa6d9d640 1 -- 172.21.15.77:0/3781693662 learned_addr learned my addr 172.21.15.77:0/3781693662 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa659c640 1 --2- 172.21.15.77:0/3781693662 >> v2:172.21.15.158:3300/0 conn(0x7fdaa815dfe0 0x7fdaa80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa6d9d640 1 -- 172.21.15.77:0/3781693662 >> v2:172.21.15.158:3300/0 conn(0x7fdaa815dfe0 msgr2=0x7fdaa80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa6d9d640 1 --2- 172.21.15.77:0/3781693662 >> v2:172.21.15.158:3300/0 conn(0x7fdaa815dfe0 0x7fdaa80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa6d9d640 1 -- 172.21.15.77:0/3781693662 >> v1:172.21.15.77:6789/0 conn(0x7fdaa80cf830 legacy=0x7fdaa80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa6d9d640 1 -- 172.21.15.77:0/3781693662 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa80b3710 con 0x7fdaa80d29c0 2024-03-06T07:33:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.176+0000 7fdaa559a640 1 -- 172.21.15.77:0/3781693662 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039270726 0 0) 0x7fdaa80b3430 con 0x7fdaa80cf830 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.177+0000 7fdaa6d9d640 1 --2- 172.21.15.77:0/3781693662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa80d29c0 0x7fdaa80b41f0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8067990 tx=0x7fdaa002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857ffbb214cd2bef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.177+0000 7fdaa559a640 1 -- 172.21.15.77:0/3781693662 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa002cb20 con 0x7fdaa80d29c0 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.177+0000 7fdaa559a640 1 -- 172.21.15.77:0/3781693662 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa002ccc0 con 0x7fdaa80d29c0 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.177+0000 7fdaa559a640 1 -- 172.21.15.77:0/3781693662 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0034610 con 0x7fdaa80d29c0 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.178+0000 7fdaacb5e640 1 -- 172.21.15.77:0/3781693662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa80d29c0 msgr2=0x7fdaa80b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.178+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/3781693662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa80d29c0 0x7fdaa80b41f0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8067990 tx=0x7fdaa002bf80 comp rx=0 tx=0).stop 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.178+0000 7fdaacb5e640 1 -- 172.21.15.77:0/3781693662 shutdown_connections 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.178+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/3781693662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa80d29c0 0x7fdaa80b41f0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.178+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/3781693662 >> v2:172.21.15.158:3300/0 conn(0x7fdaa815dfe0 0x7fdaa80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.178+0000 7fdaacb5e640 1 -- 172.21.15.77:0/3781693662 >> 172.21.15.77:0/3781693662 conn(0x7fdaa80d3cd0 msgr2=0x7fdaa80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.178+0000 7fdaacb5e640 1 -- 172.21.15.77:0/3781693662 shutdown_connections 2024-03-06T07:33:12.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.178+0000 7fdaacb5e640 1 -- 172.21.15.77:0/3781693662 wait complete. 2024-03-06T07:33:12.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.179+0000 7fdaacb5e640 1 Processor -- start 2024-03-06T07:33:12.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.179+0000 7fdaacb5e640 1 -- start start 2024-03-06T07:33:12.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaacb5e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaa80cf830 0x7fdaa8160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaacb5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaa80d29c0 0x7fdaa80ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaacb5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa815dfe0 0x7fdaa81e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaacb5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa80d1df0 con 0x7fdaa815dfe0 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaacb5e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa80d1c70 con 0x7fdaa80cf830 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaacb5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa815e3c0 con 0x7fdaa80d29c0 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa815dfe0 0x7fdaa81e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa815dfe0 0x7fdaa81e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41190/0 (socket says 172.21.15.77:41190) 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa6d9d640 1 -- 172.21.15.77:0/4071637146 learned_addr learned my addr 172.21.15.77:0/4071637146 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa659c640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaa80cf830 0x7fdaa8160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa5d9b640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaa80d29c0 0x7fdaa80ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa6d9d640 1 -- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaa80d29c0 msgr2=0x7fdaa80ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa6d9d640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaa80d29c0 0x7fdaa80ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa6d9d640 1 -- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaa80cf830 msgr2=0x7fdaa8160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa6d9d640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaa80cf830 0x7fdaa8160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa6d9d640 1 -- 172.21.15.77:0/4071637146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8160830 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.180+0000 7fdaa659c640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaa80cf830 0x7fdaa8160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.181+0000 7fdaa6d9d640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa815dfe0 0x7fdaa81e8270 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0036040 tx=0x7fdaa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.181+0000 7fda96ffd640 1 -- 172.21.15.77:0/4071637146 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0043020 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.181+0000 7fda96ffd640 1 -- 172.21.15.77:0/4071637146 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa003e450 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.181+0000 7fda96ffd640 1 -- 172.21.15.77:0/4071637146 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa003e860 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.181+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa8161300 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.181+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa81f76c0 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.182+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa804f330 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.182+0000 7fda96ffd640 1 -- 172.21.15.77:0/4071637146 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdaa0046020 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.182+0000 7fda96ffd640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda8806b4d0 0x7fda8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.183+0000 7fda96ffd640 1 -- 172.21.15.77:0/4071637146 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdaa00acb40 con 0x7fdaa815dfe0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.183+0000 7fdaa659c640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda8806b4d0 0x7fda8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.183+0000 7fdaa659c640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda8806b4d0 0x7fda8806d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fda98006fd0 tx=0x7fda98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.187+0000 7fda96ffd640 1 -- 172.21.15.77:0/4071637146 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdaa004d030 con 0x7fdaa815dfe0 2024-03-06T07:33:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.401+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaa80d0bd0 con 0x7fdaa815dfe0 2024-03-06T07:33:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.402+0000 7fda96ffd640 1 -- 172.21.15.77:0/4071637146 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fdaa007b9a0 con 0x7fdaa815dfe0 2024-03-06T07:33:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:12.404 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.403+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda8806b4d0 msgr2=0x7fda8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.403+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda8806b4d0 0x7fda8806d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fda98006fd0 tx=0x7fda98008040 comp rx=0 tx=0).stop 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.403+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa815dfe0 msgr2=0x7fdaa81e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.403+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa815dfe0 0x7fdaa81e8270 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0036040 tx=0x7fdaa0004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.404+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 shutdown_connections 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.404+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fda8806b4d0 0x7fda8806d990 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.404+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa815dfe0 0x7fdaa81e8270 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.404+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdaa80d29c0 0x7fdaa80ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.404+0000 7fdaacb5e640 1 --2- 172.21.15.77:0/4071637146 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdaa80cf830 0x7fdaa8160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.404+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 >> 172.21.15.77:0/4071637146 conn(0x7fdaa80d3cd0 msgr2=0x7fdaa80d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.404+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 shutdown_connections 2024-03-06T07:33:12.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:12.404+0000 7fdaacb5e640 1 -- 172.21.15.77:0/4071637146 wait complete. 2024-03-06T07:33:12.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:12.618 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:13.619 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:13.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.719+0000 7fa9b551e640 1 Processor -- start 2024-03-06T07:33:13.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.719+0000 7fa9b551e640 1 -- start start 2024-03-06T07:33:13.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.719+0000 7fa9b551e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9b006c6d0 0x7fa9b015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.719+0000 7fa9b551e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fa9b0066d90 0x7fa9b00ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.719+0000 7fa9b551e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00cd450 con 0x7fa9b00b4f70 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.719+0000 7fa9b551e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b015e9f0 con 0x7fa9b0066d90 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.719+0000 7fa9b551e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00cda10 con 0x7fa9b006c6d0 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9aeffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa9b00b4f70 0x7fa9b00b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33162/0 (socket says 172.21.15.77:33162) 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9aeffd640 1 -- 172.21.15.77:0/3570808565 learned_addr learned my addr 172.21.15.77:0/3570808565 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9af7fe640 1 --2- 172.21.15.77:0/3570808565 >> v2:172.21.15.158:3300/0 conn(0x7fa9b0066d90 0x7fa9b00ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9ae7fc640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9b006c6d0 0x7fa9b015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564294507 0 0) 0x7fa9b00cd450 con 0x7fa9b00b4f70 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa998003520 con 0x7fa9b00b4f70 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2381386047 0 0) 0x7fa998003520 con 0x7fa9b00b4f70 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9b006c6d0 msgr2=0x7fa9b015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9b006c6d0 0x7fa9b015ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 >> v2:172.21.15.158:3300/0 conn(0x7fa9b0066d90 msgr2=0x7fa9b00ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 --2- 172.21.15.77:0/3570808565 >> v2:172.21.15.158:3300/0 conn(0x7fa9b0066d90 0x7fa9b00ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b00cd730 con 0x7fa9b00b4f70 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fa9a00019f0 con 0x7fa9b00b4f70 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 >> v1:172.21.15.77:6789/0 conn(0x7fa9b00b4f70 legacy=0x7fa9b00b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9980039b0 0x7fa998003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa998004310 0x7fa998004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998004e50 0x7fa9980052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b015e9f0 con 0x7fa998004e50 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00cda10 con 0x7fa998004310 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99800bd40 con 0x7fa9980039b0 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9af7fe640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998004e50 0x7fa9980052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9ae7fc640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa998004310 0x7fa998004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.720+0000 7fa9aeffd640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9980039b0 0x7fa998003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9af7fe640 1 -- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9980039b0 msgr2=0x7fa998003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9af7fe640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9980039b0 0x7fa998003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9af7fe640 1 -- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa998004310 msgr2=0x7fa998004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9af7fe640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa998004310 0x7fa998004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9af7fe640 1 -- 172.21.15.77:0/3570808565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa998003520 con 0x7fa998004e50 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9aeffd640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9980039b0 0x7fa998003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9af7fe640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998004e50 0x7fa9980052d0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009a90 tx=0x7fa998005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af136f0b8d8988c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8039070 con 0x7fa998004e50 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a802c6b0 con 0x7fa998004e50 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.721+0000 7fa9adffb640 1 -- 172.21.15.77:0/3570808565 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8009690 con 0x7fa998004e50 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.722+0000 7fa9b551e640 1 -- 172.21.15.77:0/3570808565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998004e50 msgr2=0x7fa9980052d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.722+0000 7fa9b551e640 1 --2- 172.21.15.77:0/3570808565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998004e50 0x7fa9980052d0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009a90 tx=0x7fa998005830 comp rx=0 tx=0).stop 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.722+0000 7fa9af7fe640 1 -- 172.21.15.77:0/3570808565 reap_dead start 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.722+0000 7fa9b551e640 1 -- 172.21.15.77:0/3570808565 shutdown_connections 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.722+0000 7fa9b551e640 1 -- 172.21.15.77:0/3570808565 >> 172.21.15.77:0/3570808565 conn(0x7fa9b00724d0 msgr2=0x7fa9b00728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.722+0000 7fa9b551e640 1 -- 172.21.15.77:0/3570808565 shutdown_connections 2024-03-06T07:33:13.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.722+0000 7fa9b551e640 1 -- 172.21.15.77:0/3570808565 wait complete. 2024-03-06T07:33:13.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.723+0000 7fa9b551e640 1 Processor -- start 2024-03-06T07:33:13.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.723+0000 7fa9b551e640 1 -- start start 2024-03-06T07:33:13.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9b551e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b0156840 0x7fa9b006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9b551e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9b006e890 0x7fa9b00d0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9b551e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9b00d0ab0 0x7fa9b00ce200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9b551e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00cd210 con 0x7fa9b0156840 2024-03-06T07:33:13.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9b551e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00cd090 con 0x7fa9b00d0ab0 2024-03-06T07:33:13.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9b551e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b015ee80 con 0x7fa9b006e890 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9aeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b0156840 0x7fa9b006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9aeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b0156840 0x7fa9b006e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41212/0 (socket says 172.21.15.77:41212) 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9aeffd640 1 -- 172.21.15.77:0/400173666 learned_addr learned my addr 172.21.15.77:0/400173666 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9aeffd640 1 -- 172.21.15.77:0/400173666 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9b006e890 msgr2=0x7fa9b00d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9aeffd640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9b006e890 0x7fa9b00d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9aeffd640 1 -- 172.21.15.77:0/400173666 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9b00d0ab0 msgr2=0x7fa9b00ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9aeffd640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9b00d0ab0 0x7fa9b00ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.724+0000 7fa9aeffd640 1 -- 172.21.15.77:0/400173666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b015f5f0 con 0x7fa9b0156840 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.725+0000 7fa9aeffd640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b0156840 0x7fa9b006e350 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000e410 tx=0x7fa9a000e500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.725+0000 7fa9977fe640 1 -- 172.21.15.77:0/400173666 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a000ee00 con 0x7fa9b0156840 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.725+0000 7fa9977fe640 1 -- 172.21.15.77:0/400173666 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a0018070 con 0x7fa9b0156840 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.725+0000 7fa9977fe640 1 -- 172.21.15.77:0/400173666 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0013640 con 0x7fa9b0156840 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.725+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b015f8d0 con 0x7fa9b0156840 2024-03-06T07:33:13.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.725+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b015fe90 con 0x7fa9b0156840 2024-03-06T07:33:13.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.726+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b004f330 con 0x7fa9b0156840 2024-03-06T07:33:13.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.726+0000 7fa9977fe640 1 -- 172.21.15.77:0/400173666 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9a00137e0 con 0x7fa9b0156840 2024-03-06T07:33:13.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.727+0000 7fa9977fe640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa99c06b500 0x7fa99c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.727+0000 7fa9977fe640 1 -- 172.21.15.77:0/400173666 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa9a008a900 con 0x7fa9b0156840 2024-03-06T07:33:13.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.727+0000 7fa9ae7fc640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa99c06b500 0x7fa99c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.727+0000 7fa9ae7fc640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa99c06b500 0x7fa99c06d9c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00cb870 tx=0x7fa9a40095d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:13.735+0000 7fa9977fe640 1 -- 172.21.15.77:0/400173666 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9a0059760 con 0x7fa9b0156840 2024-03-06T07:33:14.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.094+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9b0162190 con 0x7fa9b0156840 2024-03-06T07:33:14.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.094+0000 7fa9977fe640 1 -- 172.21.15.77:0/400173666 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fa9a0059060 con 0x7fa9b0156840 2024-03-06T07:33:14.096 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:14.096 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:14.097 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:14.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa99c06b500 msgr2=0x7fa99c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa99c06b500 0x7fa99c06d9c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00cb870 tx=0x7fa9a40095d0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b0156840 msgr2=0x7fa9b006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b0156840 0x7fa9b006e350 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000e410 tx=0x7fa9a000e500 comp rx=0 tx=0).stop 2024-03-06T07:33:14.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 shutdown_connections 2024-03-06T07:33:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa9b00d0ab0 0x7fa9b00ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fa9b006e890 0x7fa9b00d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fa99c06b500 0x7fa99c06d9c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 --2- 172.21.15.77:0/400173666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b0156840 0x7fa9b006e350 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 >> 172.21.15.77:0/400173666 conn(0x7fa9b00724d0 msgr2=0x7fa9b006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 shutdown_connections 2024-03-06T07:33:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:14.096+0000 7fa9b551e640 1 -- 172.21.15.77:0/400173666 wait complete. 2024-03-06T07:33:14.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:14.311 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:15.311 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:15.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.410+0000 7fbb3ee88640 1 Processor -- start 2024-03-06T07:33:15.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.410+0000 7fbb3ee88640 1 -- start start 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb3ee88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb380d22b0 0x7fbb380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb3ee88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 0x7fbb380b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb3ee88640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb380b3430 con 0x7fbb3815dfe0 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb3ee88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb380d2c50 con 0x7fbb380d22b0 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb3ee88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380b31e0 con 0x7fbb380b41f0 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb37fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb380d22b0 0x7fbb380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb37fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb380d22b0 0x7fbb380d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41216/0 (socket says 172.21.15.77:41216) 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb3cc24640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbb3815dfe0 0x7fbb380cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33174/0 (socket says 172.21.15.77:33174) 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb37fff640 1 -- 172.21.15.77:0/1975701613 learned_addr learned my addr 172.21.15.77:0/1975701613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb3d425640 1 --2- 172.21.15.77:0/1975701613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 0x7fbb380b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb37fff640 1 -- 172.21.15.77:0/1975701613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 msgr2=0x7fbb380b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb37fff640 1 --2- 172.21.15.77:0/1975701613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 0x7fbb380b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb37fff640 1 -- 172.21.15.77:0/1975701613 >> v1:172.21.15.77:6789/0 conn(0x7fbb3815dfe0 legacy=0x7fbb380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb377fe640 1 -- 172.21.15.77:0/1975701613 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456726088 0 0) 0x7fbb380b3430 con 0x7fbb3815dfe0 2024-03-06T07:33:15.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.411+0000 7fbb37fff640 1 -- 172.21.15.77:0/1975701613 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb380b3710 con 0x7fbb380d22b0 2024-03-06T07:33:15.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.412+0000 7fbb37fff640 1 --2- 172.21.15.77:0/1975701613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380d2710 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009870 tx=0x7fbb2402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e77ac454bd7760 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.412+0000 7fbb377fe640 1 -- 172.21.15.77:0/1975701613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24034070 con 0x7fbb380d22b0 2024-03-06T07:33:15.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.412+0000 7fbb377fe640 1 -- 172.21.15.77:0/1975701613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb24030840 con 0x7fbb380d22b0 2024-03-06T07:33:15.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.412+0000 7fbb377fe640 1 -- 172.21.15.77:0/1975701613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24030c10 con 0x7fbb380d22b0 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.413+0000 7fbb3ee88640 1 -- 172.21.15.77:0/1975701613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 msgr2=0x7fbb380d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.413+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/1975701613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380d2710 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009870 tx=0x7fbb2402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.413+0000 7fbb3ee88640 1 -- 172.21.15.77:0/1975701613 shutdown_connections 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.413+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/1975701613 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 0x7fbb380b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.413+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/1975701613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380d2710 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.413+0000 7fbb3ee88640 1 -- 172.21.15.77:0/1975701613 >> 172.21.15.77:0/1975701613 conn(0x7fbb380d3cd0 msgr2=0x7fbb380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.413+0000 7fbb3ee88640 1 -- 172.21.15.77:0/1975701613 shutdown_connections 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.413+0000 7fbb3ee88640 1 -- 172.21.15.77:0/1975701613 wait complete. 2024-03-06T07:33:15.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.414+0000 7fbb3ee88640 1 Processor -- start 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.414+0000 7fbb3ee88640 1 -- start start 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb3ee88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 0x7fbb380cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb3ee88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb3ee88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3815dfe0 0x7fbb380b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb3ee88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380cfa00 con 0x7fbb380d22b0 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb3ee88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380b3d60 con 0x7fbb3815dfe0 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb3ee88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380d1c70 con 0x7fbb380b41f0 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41226/0 (socket says 172.21.15.77:41226) 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 -- 172.21.15.77:0/3673217023 learned_addr learned my addr 172.21.15.77:0/3673217023 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 -- 172.21.15.77:0/3673217023 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 msgr2=0x7fbb380cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 0x7fbb380cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 -- 172.21.15.77:0/3673217023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3815dfe0 msgr2=0x7fbb380b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3815dfe0 0x7fbb380b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 -- 172.21.15.77:0/3673217023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3815cf10 con 0x7fbb380d22b0 2024-03-06T07:33:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb37fff640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380b78f0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fbb240090b0 tx=0x7fbb240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb357fa640 1 -- 172.21.15.77:0/3673217023 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24034040 con 0x7fbb380d22b0 2024-03-06T07:33:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb357fa640 1 -- 172.21.15.77:0/3673217023 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb24030640 con 0x7fbb380d22b0 2024-03-06T07:33:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3809ddd0 con 0x7fbb380d22b0 2024-03-06T07:33:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb357fa640 1 -- 172.21.15.77:0/3673217023 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24039d40 con 0x7fbb380d22b0 2024-03-06T07:33:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.415+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb380cb8d0 con 0x7fbb380d22b0 2024-03-06T07:33:15.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.416+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3804f330 con 0x7fbb380d22b0 2024-03-06T07:33:15.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.417+0000 7fbb357fa640 1 -- 172.21.15.77:0/3673217023 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb2404b060 con 0x7fbb380d22b0 2024-03-06T07:33:15.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.417+0000 7fbb357fa640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.417+0000 7fbb357fa640 1 -- 172.21.15.77:0/3673217023 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbb240abea0 con 0x7fbb380d22b0 2024-03-06T07:33:15.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.417+0000 7fbb3cc24640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.418+0000 7fbb3cc24640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c007fd0 tx=0x7fbb2c007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.424+0000 7fbb357fa640 1 -- 172.21.15.77:0/3673217023 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb3809ddd0 con 0x7fbb380d22b0 2024-03-06T07:33:15.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.639+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb3806c7c0 con 0x7fbb380d22b0 2024-03-06T07:33:15.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.640+0000 7fbb357fa640 1 -- 172.21.15.77:0/3673217023 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fbb2407aeb0 con 0x7fbb380d22b0 2024-03-06T07:33:15.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:15.642 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:15.642 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb1c06b4d0 msgr2=0x7fbb1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c007fd0 tx=0x7fbb2c007960 comp rx=0 tx=0).stop 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 msgr2=0x7fbb380b78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380b78f0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fbb240090b0 tx=0x7fbb240046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 shutdown_connections 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbb1c06b4d0 0x7fbb1c06d990 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380d22b0 0x7fbb380b78f0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbb3815dfe0 0x7fbb380b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 --2- 172.21.15.77:0/3673217023 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbb380b41f0 0x7fbb380cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 >> 172.21.15.77:0/3673217023 conn(0x7fbb380d3cd0 msgr2=0x7fbb380b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 shutdown_connections 2024-03-06T07:33:15.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:15.642+0000 7fbb3ee88640 1 -- 172.21.15.77:0/3673217023 wait complete. 2024-03-06T07:33:15.856 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:15.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:16.856 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:16.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.955+0000 7f5618aa4640 1 Processor -- start 2024-03-06T07:33:16.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.955+0000 7f5618aa4640 1 -- start start 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.955+0000 7f5618aa4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f561415dfe0 0x7f56140cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.955+0000 7f5618aa4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140d29c0 0x7f56140b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.955+0000 7f5618aa4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140b3430 con 0x7f56140cf830 2024-03-06T07:33:16.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.955+0000 7f5618aa4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56140b31e0 con 0x7f561415dfe0 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.955+0000 7f5618aa4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56140b3d60 con 0x7f56140d29c0 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f561259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f561415dfe0 0x7f56140cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f561259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f561415dfe0 0x7f56140cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41230/0 (socket says 172.21.15.77:41230) 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f561259c640 1 -- 172.21.15.77:0/2930868696 learned_addr learned my addr 172.21.15.77:0/2930868696 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f560bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f56140cf830 0x7f56140d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:33178/0 (socket says 172.21.15.77:33178) 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f5612d9d640 1 --2- 172.21.15.77:0/2930868696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140d29c0 0x7f56140b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f561259c640 1 -- 172.21.15.77:0/2930868696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140d29c0 msgr2=0x7f56140b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f561259c640 1 --2- 172.21.15.77:0/2930868696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140d29c0 0x7f56140b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f561259c640 1 -- 172.21.15.77:0/2930868696 >> v1:172.21.15.77:6789/0 conn(0x7f56140cf830 legacy=0x7f56140d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f5611d9b640 1 -- 172.21.15.77:0/2930868696 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764723206 0 0) 0x7f56140b3430 con 0x7f56140cf830 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.956+0000 7f561259c640 1 -- 172.21.15.77:0/2930868696 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140b3710 con 0x7f561415dfe0 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f561259c640 1 --2- 172.21.15.77:0/2930868696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f561415dfe0 0x7f56140cf2f0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f5604009870 tx=0x7f560402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986f1874e9a358b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5611d9b640 1 -- 172.21.15.77:0/2930868696 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604034070 con 0x7f561415dfe0 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5611d9b640 1 -- 172.21.15.77:0/2930868696 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5604030840 con 0x7f561415dfe0 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5611d9b640 1 -- 172.21.15.77:0/2930868696 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604030c10 con 0x7f561415dfe0 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5618aa4640 1 -- 172.21.15.77:0/2930868696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f561415dfe0 msgr2=0x7f56140cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5618aa4640 1 --2- 172.21.15.77:0/2930868696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f561415dfe0 0x7f56140cf2f0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f5604009870 tx=0x7f560402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5618aa4640 1 -- 172.21.15.77:0/2930868696 shutdown_connections 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5618aa4640 1 --2- 172.21.15.77:0/2930868696 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140d29c0 0x7f56140b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5618aa4640 1 --2- 172.21.15.77:0/2930868696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f561415dfe0 0x7f56140cf2f0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5618aa4640 1 -- 172.21.15.77:0/2930868696 >> 172.21.15.77:0/2930868696 conn(0x7f56140d3cd0 msgr2=0x7f56140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5618aa4640 1 -- 172.21.15.77:0/2930868696 shutdown_connections 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.957+0000 7f5618aa4640 1 -- 172.21.15.77:0/2930868696 wait complete. 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.959+0000 7f5618aa4640 1 Processor -- start 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.959+0000 7f5618aa4640 1 -- start start 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f5618aa4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140cf830 0x7f561415c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f5618aa4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56140d29c0 0x7f56140cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f5618aa4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f561415dfe0 0x7f5614160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f5618aa4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56140d1df0 con 0x7f56140d29c0 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f5618aa4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56140d1c70 con 0x7f561415dfe0 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f5618aa4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561415e3c0 con 0x7f56140cf830 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f560bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56140d29c0 0x7f56140cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f560bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56140d29c0 0x7f56140cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41236/0 (socket says 172.21.15.77:41236) 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f560bfff640 1 -- 172.21.15.77:0/3137217271 learned_addr learned my addr 172.21.15.77:0/3137217271 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f561259c640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140cf830 0x7f561415c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.960+0000 7f5612d9d640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f561415dfe0 0x7f5614160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560bfff640 1 -- 172.21.15.77:0/3137217271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140cf830 msgr2=0x7f561415c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560bfff640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140cf830 0x7f561415c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560bfff640 1 -- 172.21.15.77:0/3137217271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f561415dfe0 msgr2=0x7f5614160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560bfff640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f561415dfe0 0x7f5614160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560bfff640 1 -- 172.21.15.77:0/3137217271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140cd220 con 0x7f56140d29c0 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560bfff640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56140d29c0 0x7f56140cc7a0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c4b0 tx=0x7f55fc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560affd640 1 -- 172.21.15.77:0/3137217271 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc018070 con 0x7f56140d29c0 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560affd640 1 -- 172.21.15.77:0/3137217271 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc007500 con 0x7f56140d29c0 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56140cd500 con 0x7f56140d29c0 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f560affd640 1 -- 172.21.15.77:0/3137217271 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc004a20 con 0x7f56140d29c0 2024-03-06T07:33:16.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.961+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56140cdac0 con 0x7f56140d29c0 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.962+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56140d1a90 con 0x7f56140d29c0 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.963+0000 7f560affd640 1 -- 172.21.15.77:0/3137217271 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55fc01b070 con 0x7f56140d29c0 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.963+0000 7f560affd640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f55f406b500 0x7f55f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.964+0000 7f561259c640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f55f406b500 0x7f55f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.964+0000 7f560affd640 1 -- 172.21.15.77:0/3137217271 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f55fc0893c0 con 0x7f56140d29c0 2024-03-06T07:33:16.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.964+0000 7f561259c640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f55f406b500 0x7f55f406d9c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5604036000 tx=0x7f560403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:16.968+0000 7f560affd640 1 -- 172.21.15.77:0/3137217271 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56140d1a90 con 0x7f56140d29c0 2024-03-06T07:33:17.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.182+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56140cef60 con 0x7f56140d29c0 2024-03-06T07:33:17.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.182+0000 7f560affd640 1 -- 172.21.15.77:0/3137217271 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f55fc006e20 con 0x7f56140d29c0 2024-03-06T07:33:17.184 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:17.184 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:17.184 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:17.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f55f406b500 msgr2=0x7f55f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f55f406b500 0x7f55f406d9c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5604036000 tx=0x7f560403c040 comp rx=0 tx=0).stop 2024-03-06T07:33:17.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56140d29c0 msgr2=0x7f56140cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56140d29c0 0x7f56140cc7a0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c4b0 tx=0x7f55fc00c970 comp rx=0 tx=0).stop 2024-03-06T07:33:17.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 shutdown_connections 2024-03-06T07:33:17.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f55f406b500 0x7f55f406d9c0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56140d29c0 0x7f56140cc7a0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f561415dfe0 0x7f5614160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 --2- 172.21.15.77:0/3137217271 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f56140cf830 0x7f561415c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 >> 172.21.15.77:0/3137217271 conn(0x7f56140d3cd0 msgr2=0x7f561415d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 shutdown_connections 2024-03-06T07:33:17.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:17.184+0000 7f5618aa4640 1 -- 172.21.15.77:0/3137217271 wait complete. 2024-03-06T07:33:17.397 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:17.398 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:18.398 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:18.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.501+0000 7fd23b2d4640 1 Processor -- start 2024-03-06T07:33:18.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd23b2d4640 1 -- start start 2024-03-06T07:33:18.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd23b2d4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd23415dfe0 0x7fd2340cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd23b2d4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd2340cf830 0x7fd2340d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd23b2d4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2340b3430 con 0x7fd2340d27f0 2024-03-06T07:33:18.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd23b2d4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2340b31e0 con 0x7fd23415dfe0 2024-03-06T07:33:18.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd23b2d4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2340b3d60 con 0x7fd2340cf830 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd239070640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd23415dfe0 0x7fd2340cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd239871640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd2340d27f0 0x7fd2340d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49844/0 (socket says 172.21.15.77:49844) 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.502+0000 7fd239871640 1 -- 172.21.15.77:0/1989730243 learned_addr learned my addr 172.21.15.77:0/1989730243 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.503+0000 7fd239070640 1 -- 172.21.15.77:0/1989730243 >> v2:172.21.15.158:3300/0 conn(0x7fd2340cf830 msgr2=0x7fd2340d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.503+0000 7fd239070640 1 --2- 172.21.15.77:0/1989730243 >> v2:172.21.15.158:3300/0 conn(0x7fd2340cf830 0x7fd2340d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.503+0000 7fd239070640 1 -- 172.21.15.77:0/1989730243 >> v1:172.21.15.77:6789/0 conn(0x7fd2340d27f0 legacy=0x7fd2340d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.503+0000 7fd22bfff640 1 -- 172.21.15.77:0/1989730243 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180250597 0 0) 0x7fd2340b3430 con 0x7fd2340d27f0 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.503+0000 7fd239070640 1 -- 172.21.15.77:0/1989730243 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2340b3710 con 0x7fd23415dfe0 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.503+0000 7fd239070640 1 --2- 172.21.15.77:0/1989730243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd23415dfe0 0x7fd2340cf2f0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fd2240097a0 tx=0x7fd22402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a678ce04f05ad465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd22bfff640 1 -- 172.21.15.77:0/1989730243 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224035020 con 0x7fd23415dfe0 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd22bfff640 1 -- 172.21.15.77:0/1989730243 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd224031830 con 0x7fd23415dfe0 2024-03-06T07:33:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd22bfff640 1 -- 172.21.15.77:0/1989730243 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224031c00 con 0x7fd23415dfe0 2024-03-06T07:33:18.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1989730243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd23415dfe0 msgr2=0x7fd2340cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1989730243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd23415dfe0 0x7fd2340cf2f0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fd2240097a0 tx=0x7fd22402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1989730243 shutdown_connections 2024-03-06T07:33:18.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1989730243 >> v2:172.21.15.158:3300/0 conn(0x7fd2340cf830 0x7fd2340d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1989730243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd23415dfe0 0x7fd2340cf2f0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1989730243 >> 172.21.15.77:0/1989730243 conn(0x7fd2340d3cd0 msgr2=0x7fd2340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1989730243 shutdown_connections 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.504+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1989730243 wait complete. 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.505+0000 7fd23b2d4640 1 Processor -- start 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.505+0000 7fd23b2d4640 1 -- start start 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23b2d4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2340cf830 0x7fd234153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23b2d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2340d27f0 0x7fd2341f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23b2d4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd23415dfe0 0x7fd2341602e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23b2d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2340d1df0 con 0x7fd2340d27f0 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23b2d4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2340d1c70 con 0x7fd23415dfe0 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23b2d4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23415e3c0 con 0x7fd2340cf830 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23886f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2340d27f0 0x7fd2341f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23886f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2340d27f0 0x7fd2341f3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51992/0 (socket says 172.21.15.77:51992) 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd23886f640 1 -- 172.21.15.77:0/1690420829 learned_addr learned my addr 172.21.15.77:0/1690420829 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd239871640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd23415dfe0 0x7fd2341602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.506+0000 7fd239070640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2340cf830 0x7fd234153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd23886f640 1 -- 172.21.15.77:0/1690420829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2340cf830 msgr2=0x7fd234153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd23886f640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2340cf830 0x7fd234153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd23886f640 1 -- 172.21.15.77:0/1690420829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd23415dfe0 msgr2=0x7fd2341602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd23886f640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd23415dfe0 0x7fd2341602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd23886f640 1 -- 172.21.15.77:0/1690420829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2340caf40 con 0x7fd2340d27f0 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd23886f640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2340d27f0 0x7fd2341f3c20 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fd23000b410 tx=0x7fd23000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd229ffb640 1 -- 172.21.15.77:0/1690420829 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230015020 con 0x7fd2340d27f0 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd229ffb640 1 -- 172.21.15.77:0/1690420829 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd230011850 con 0x7fd2340d27f0 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2340cb220 con 0x7fd2340d27f0 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd229ffb640 1 -- 172.21.15.77:0/1690420829 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230011c40 con 0x7fd2340d27f0 2024-03-06T07:33:18.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.507+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2340cb7e0 con 0x7fd2340d27f0 2024-03-06T07:33:18.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.508+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2340cb9e0 con 0x7fd2340d27f0 2024-03-06T07:33:18.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.509+0000 7fd229ffb640 1 -- 172.21.15.77:0/1690420829 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd230018020 con 0x7fd2340d27f0 2024-03-06T07:33:18.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.509+0000 7fd229ffb640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd21806b4d0 0x7fd21806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.509+0000 7fd229ffb640 1 -- 172.21.15.77:0/1690420829 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd23008a090 con 0x7fd2340d27f0 2024-03-06T07:33:18.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.509+0000 7fd239070640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd21806b4d0 0x7fd21806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.510+0000 7fd239070640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd21806b4d0 0x7fd21806d990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd224036000 tx=0x7fd22403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.514+0000 7fd229ffb640 1 -- 172.21.15.77:0/1690420829 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2340cb9e0 con 0x7fd2340d27f0 2024-03-06T07:33:18.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.727+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd23404f330 con 0x7fd2340d27f0 2024-03-06T07:33:18.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.728+0000 7fd229ffb640 1 -- 172.21.15.77:0/1690420829 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fd23001f020 con 0x7fd2340d27f0 2024-03-06T07:33:18.730 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:18.730 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:18.730 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd21806b4d0 msgr2=0x7fd21806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd21806b4d0 0x7fd21806d990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd224036000 tx=0x7fd22403c040 comp rx=0 tx=0).stop 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2340d27f0 msgr2=0x7fd2341f3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2340d27f0 0x7fd2341f3c20 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fd23000b410 tx=0x7fd23000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 shutdown_connections 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd21806b4d0 0x7fd21806d990 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2340d27f0 0x7fd2341f3c20 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd23415dfe0 0x7fd2341602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 --2- 172.21.15.77:0/1690420829 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2340cf830 0x7fd234153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 >> 172.21.15.77:0/1690420829 conn(0x7fd2340d3cd0 msgr2=0x7fd2340b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 shutdown_connections 2024-03-06T07:33:18.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:18.730+0000 7fd23b2d4640 1 -- 172.21.15.77:0/1690420829 wait complete. 2024-03-06T07:33:18.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:18.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:19.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:20.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.045+0000 7f6d858b8640 1 Processor -- start 2024-03-06T07:33:20.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.045+0000 7f6d858b8640 1 -- start start 2024-03-06T07:33:20.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.046+0000 7f6d858b8640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d8015c140 0x7f6d800d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.046+0000 7f6d858b8640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6d800b2f00 0x7f6d800d0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.046+0000 7f6d858b8640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800cd620 con 0x7f6d800cd120 2024-03-06T07:33:20.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.046+0000 7f6d858b8640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8015c520 con 0x7f6d800b2f00 2024-03-06T07:33:20.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.046+0000 7f6d858b8640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800cd900 con 0x7f6d8015c140 2024-03-06T07:33:20.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7effd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d8015c140 0x7f6d800d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7f7fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f6d800b2f00 0x7f6d800d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7effd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d8015c140 0x7f6d800d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:42670/0 (socket says 172.21.15.77:42670) 2024-03-06T07:33:20.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7effd640 1 -- 172.21.15.77:0/3748912147 learned_addr learned my addr 172.21.15.77:0/3748912147 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:20.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7e7fc640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f6d800cd120 0x7f6d800ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:38616/0 (socket says 172.21.15.77:38616) 2024-03-06T07:33:20.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7effd640 1 -- 172.21.15.77:0/3748912147 >> v2:172.21.15.158:3300/0 conn(0x7f6d800b2f00 msgr2=0x7f6d800d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7effd640 1 --2- 172.21.15.77:0/3748912147 >> v2:172.21.15.158:3300/0 conn(0x7f6d800b2f00 0x7f6d800d0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7effd640 1 -- 172.21.15.77:0/3748912147 >> v1:172.21.15.158:6789/0 conn(0x7f6d800cd120 legacy=0x7f6d800ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7effd640 1 -- 172.21.15.77:0/3748912147 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d800b1410 con 0x7f6d8015c140 2024-03-06T07:33:20.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.047+0000 7f6d7f7fe640 1 --2- 172.21.15.77:0/3748912147 >> v2:172.21.15.158:3300/0 conn(0x7f6d800b2f00 0x7f6d800d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:20.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.048+0000 7f6d7effd640 1 --2- 172.21.15.77:0/3748912147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d8015c140 0x7f6d800d0010 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009870 tx=0x7f6d7802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31881579745417ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.048+0000 7f6d7dffb640 1 -- 172.21.15.77:0/3748912147 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7802f840 con 0x7f6d8015c140 2024-03-06T07:33:20.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.048+0000 7f6d7dffb640 1 -- 172.21.15.77:0/3748912147 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7802f9e0 con 0x7f6d8015c140 2024-03-06T07:33:20.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.048+0000 7f6d7dffb640 1 -- 172.21.15.77:0/3748912147 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7802fdb0 con 0x7f6d8015c140 2024-03-06T07:33:20.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.049+0000 7f6d858b8640 1 -- 172.21.15.77:0/3748912147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d8015c140 msgr2=0x7f6d800d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.049+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3748912147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d8015c140 0x7f6d800d0010 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009870 tx=0x7f6d7802ba20 comp rx=0 tx=0).stop 2024-03-06T07:33:20.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.049+0000 7f6d858b8640 1 -- 172.21.15.77:0/3748912147 shutdown_connections 2024-03-06T07:33:20.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.049+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3748912147 >> v2:172.21.15.158:3300/0 conn(0x7f6d800b2f00 0x7f6d800d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.049+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3748912147 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d8015c140 0x7f6d800d0010 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.049+0000 7f6d858b8640 1 -- 172.21.15.77:0/3748912147 >> 172.21.15.77:0/3748912147 conn(0x7f6d800b35a0 msgr2=0x7f6d800b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.049+0000 7f6d858b8640 1 -- 172.21.15.77:0/3748912147 shutdown_connections 2024-03-06T07:33:20.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.049+0000 7f6d858b8640 1 -- 172.21.15.77:0/3748912147 wait complete. 2024-03-06T07:33:20.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.051+0000 7f6d858b8640 1 Processor -- start 2024-03-06T07:33:20.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.051+0000 7f6d858b8640 1 -- start start 2024-03-06T07:33:20.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d858b8640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d800b2f00 0x7f6d800c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d858b8640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d800cd120 0x7f6d800b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d858b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d8015c140 0x7f6d800b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d858b8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800cf240 con 0x7f6d8015c140 2024-03-06T07:33:20.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d858b8640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b19d0 con 0x7f6d800cd120 2024-03-06T07:33:20.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d858b8640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800cf3c0 con 0x7f6d800b2f00 2024-03-06T07:33:20.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d7f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d8015c140 0x7f6d800b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d7f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d8015c140 0x7f6d800b6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52002/0 (socket says 172.21.15.77:52002) 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d7f7fe640 1 -- 172.21.15.77:0/3586225098 learned_addr learned my addr 172.21.15.77:0/3586225098 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d7effd640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d800b2f00 0x7f6d800c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.052+0000 7f6d7e7fc640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d800cd120 0x7f6d800b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d7f7fe640 1 -- 172.21.15.77:0/3586225098 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d800b2f00 msgr2=0x7f6d800c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d7f7fe640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d800b2f00 0x7f6d800c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d7f7fe640 1 -- 172.21.15.77:0/3586225098 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d800cd120 msgr2=0x7f6d800b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d7f7fe640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d800cd120 0x7f6d800b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d7f7fe640 1 -- 172.21.15.77:0/3586225098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d800b7540 con 0x7f6d8015c140 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d7f7fe640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d8015c140 0x7f6d800b6ce0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000efb0 tx=0x7f6d7000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d67fff640 1 -- 172.21.15.77:0/3586225098 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70018020 con 0x7f6d8015c140 2024-03-06T07:33:20.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d67fff640 1 -- 172.21.15.77:0/3586225098 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d70014480 con 0x7f6d8015c140 2024-03-06T07:33:20.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d800c57b0 con 0x7f6d8015c140 2024-03-06T07:33:20.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d67fff640 1 -- 172.21.15.77:0/3586225098 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70014850 con 0x7f6d8015c140 2024-03-06T07:33:20.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.053+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d800c5d70 con 0x7f6d8015c140 2024-03-06T07:33:20.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.054+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d800c5f70 con 0x7f6d8015c140 2024-03-06T07:33:20.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.054+0000 7f6d67fff640 1 -- 172.21.15.77:0/3586225098 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d70005d10 con 0x7f6d8015c140 2024-03-06T07:33:20.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.055+0000 7f6d67fff640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d6c06b500 0x7f6d6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.055+0000 7f6d67fff640 1 -- 172.21.15.77:0/3586225098 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6d7008af90 con 0x7f6d8015c140 2024-03-06T07:33:20.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.055+0000 7f6d7effd640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d6c06b500 0x7f6d6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.055+0000 7f6d7effd640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d6c06b500 0x7f6d6c06d9c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6d78036000 tx=0x7f6d78008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.059+0000 7f6d67fff640 1 -- 172.21.15.77:0/3586225098 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d800c5f70 con 0x7f6d8015c140 2024-03-06T07:33:20.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.269+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d8004f330 con 0x7f6d8015c140 2024-03-06T07:33:20.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.269+0000 7f6d67fff640 1 -- 172.21.15.77:0/3586225098 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f6d70022030 con 0x7f6d8015c140 2024-03-06T07:33:20.271 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:20.271 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:20.272 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:20.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d6c06b500 msgr2=0x7f6d6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d6c06b500 0x7f6d6c06d9c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6d78036000 tx=0x7f6d78008030 comp rx=0 tx=0).stop 2024-03-06T07:33:20.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d8015c140 msgr2=0x7f6d800b6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d8015c140 0x7f6d800b6ce0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000efb0 tx=0x7f6d7000c490 comp rx=0 tx=0).stop 2024-03-06T07:33:20.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 shutdown_connections 2024-03-06T07:33:20.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6d6c06b500 0x7f6d6c06d9c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d8015c140 0x7f6d800b6ce0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6d800cd120 0x7f6d800b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 --2- 172.21.15.77:0/3586225098 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6d800b2f00 0x7f6d800c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 >> 172.21.15.77:0/3586225098 conn(0x7f6d800b35a0 msgr2=0x7f6d8006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 shutdown_connections 2024-03-06T07:33:20.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:20.271+0000 7f6d858b8640 1 -- 172.21.15.77:0/3586225098 wait complete. 2024-03-06T07:33:20.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:20.485 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:21.486 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:21.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.587+0000 7fe7c09ac640 1 Processor -- start 2024-03-06T07:33:21.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.587+0000 7fe7c09ac640 1 -- start start 2024-03-06T07:33:21.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7c09ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc15dfe0 0x7fe7bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7c09ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7bc0d29c0 0x7fe7bc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7c09ac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7bc0b3430 con 0x7fe7bc0cf830 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7c09ac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7bc0b31e0 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7c09ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc0b3d60 con 0x7fe7bc15dfe0 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7bad9d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7bc0d29c0 0x7fe7bc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7b9d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe7bc0cf830 0x7fe7bc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49850/0 (socket says 172.21.15.77:49850) 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7b9d9b640 1 -- 172.21.15.77:0/1751294489 learned_addr learned my addr 172.21.15.77:0/1751294489 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7ba59c640 1 --2- 172.21.15.77:0/1751294489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc15dfe0 0x7fe7bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7bad9d640 1 -- 172.21.15.77:0/1751294489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc15dfe0 msgr2=0x7fe7bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7bad9d640 1 --2- 172.21.15.77:0/1751294489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc15dfe0 0x7fe7bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7bad9d640 1 -- 172.21.15.77:0/1751294489 >> v1:172.21.15.77:6789/0 conn(0x7fe7bc0cf830 legacy=0x7fe7bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7bad9d640 1 -- 172.21.15.77:0/1751294489 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc0b3710 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7b959a640 1 -- 172.21.15.77:0/1751294489 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951847375 0 0) 0x7fe7bc0b3430 con 0x7fe7bc0cf830 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.589+0000 7fe7ba59c640 1 --2- 172.21.15.77:0/1751294489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc15dfe0 0x7fe7bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:21.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.590+0000 7fe7bad9d640 1 --2- 172.21.15.77:0/1751294489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc0b41f0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009d90 tx=0x7fe7b402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d744d5b1f34348c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.590+0000 7fe7b959a640 1 -- 172.21.15.77:0/1751294489 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4039070 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.590+0000 7fe7b959a640 1 -- 172.21.15.77:0/1751294489 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b402cc50 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.590+0000 7fe7b959a640 1 -- 172.21.15.77:0/1751294489 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4034440 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.591+0000 7fe7c09ac640 1 -- 172.21.15.77:0/1751294489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 msgr2=0x7fe7bc0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.591+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/1751294489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc0b41f0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009d90 tx=0x7fe7b402bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.591+0000 7fe7c09ac640 1 -- 172.21.15.77:0/1751294489 shutdown_connections 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.591+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/1751294489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc0b41f0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.591+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/1751294489 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc15dfe0 0x7fe7bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.591+0000 7fe7c09ac640 1 -- 172.21.15.77:0/1751294489 >> 172.21.15.77:0/1751294489 conn(0x7fe7bc0d3cd0 msgr2=0x7fe7bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.591+0000 7fe7c09ac640 1 -- 172.21.15.77:0/1751294489 shutdown_connections 2024-03-06T07:33:21.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.591+0000 7fe7c09ac640 1 -- 172.21.15.77:0/1751294489 wait complete. 2024-03-06T07:33:21.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.592+0000 7fe7c09ac640 1 Processor -- start 2024-03-06T07:33:21.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.592+0000 7fe7c09ac640 1 -- start start 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7c09ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc0cf830 0x7fe7bc1f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7c09ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc1609d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7c09ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7bc15dfe0 0x7fe7bc0b7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7c09ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc0d1df0 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7c09ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc0d1c70 con 0x7fe7bc15dfe0 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7c09ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc15e3c0 con 0x7fe7bc0cf830 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc1609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc1609d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52014/0 (socket says 172.21.15.77:52014) 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 -- 172.21.15.77:0/2856139990 learned_addr learned my addr 172.21.15.77:0/2856139990 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:21.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7ba59c640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc0cf830 0x7fe7bc1f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 -- 172.21.15.77:0/2856139990 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc0cf830 msgr2=0x7fe7bc1f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc0cf830 0x7fe7bc1f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 -- 172.21.15.77:0/2856139990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7bc15dfe0 msgr2=0x7fe7bc0b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7bad9d640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7bc15dfe0 0x7fe7bc0b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7bc15dfe0 0x7fe7bc0b7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 -- 172.21.15.77:0/2856139990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc0cc310 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7b9d9b640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc1609d0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000c4b0 tx=0x7fe7b000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7aaffd640 1 -- 172.21.15.77:0/2856139990 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0018070 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.593+0000 7fe7aaffd640 1 -- 172.21.15.77:0/2856139990 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b0007500 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.594+0000 7fe7aaffd640 1 -- 172.21.15.77:0/2856139990 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0004a20 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.594+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc0cc5f0 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.594+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc0ccbb0 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.595+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7bc04f330 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.595+0000 7fe7aaffd640 1 -- 172.21.15.77:0/2856139990 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7b001b070 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.595+0000 7fe7aaffd640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.595+0000 7fe7aaffd640 1 -- 172.21.15.77:0/2856139990 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe7b0089580 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.595+0000 7fe7ba59c640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.596+0000 7fe7ba59c640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe79c06b4d0 0x7fe79c06d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac00b840 tx=0x7fe7ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.600+0000 7fe7aaffd640 1 -- 172.21.15.77:0/2856139990 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7b00584a0 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.821+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7bc0cf4d0 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.821+0000 7fe7aaffd640 1 -- 172.21.15.77:0/2856139990 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fe7b0006e20 con 0x7fe7bc0d29c0 2024-03-06T07:33:21.824 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:21.824 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:21.824 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.823+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe79c06b4d0 msgr2=0x7fe79c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.823+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe79c06b4d0 0x7fe79c06d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac00b840 tx=0x7fe7ac008040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.823+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 msgr2=0x7fe7bc1609d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.823+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc1609d0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000c4b0 tx=0x7fe7b000c970 comp rx=0 tx=0).stop 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.824+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 shutdown_connections 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.824+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe79c06b4d0 0x7fe79c06d990 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.824+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7bc0d29c0 0x7fe7bc1609d0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.824+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7bc15dfe0 0x7fe7bc0b7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.824+0000 7fe7c09ac640 1 --2- 172.21.15.77:0/2856139990 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe7bc0cf830 0x7fe7bc1f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.824+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 >> 172.21.15.77:0/2856139990 conn(0x7fe7bc0d3cd0 msgr2=0x7fe7bc0d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.824+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 shutdown_connections 2024-03-06T07:33:21.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:21.824+0000 7fe7c09ac640 1 -- 172.21.15.77:0/2856139990 wait complete. 2024-03-06T07:33:22.038 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:22.070 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:23.070 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:23.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.171+0000 7f1010b6b640 1 Processor -- start 2024-03-06T07:33:23.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.171+0000 7f1010b6b640 1 -- start start 2024-03-06T07:33:23.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1010b6b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100c15dfe0 0x7f100c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1010b6b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f100c0cf830 0x7f100c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1010b6b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c0b3430 con 0x7f100c0d27f0 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1010b6b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c0d2c50 con 0x7f100c0cf830 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1010b6b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0b31e0 con 0x7f100c15dfe0 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1003fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f100c0cf830 0x7f100c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1003fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f100c0cf830 0x7f100c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52018/0 (socket says 172.21.15.77:52018) 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f100ad9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f100c0d27f0 0x7f100c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49866/0 (socket says 172.21.15.77:49866) 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1003fff640 1 -- 172.21.15.77:0/2920460060 learned_addr learned my addr 172.21.15.77:0/2920460060 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1003fff640 1 -- 172.21.15.77:0/2920460060 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100c15dfe0 msgr2=0x7f100c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1003fff640 1 --2- 172.21.15.77:0/2920460060 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100c15dfe0 0x7f100c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1003fff640 1 -- 172.21.15.77:0/2920460060 >> v1:172.21.15.77:6789/0 conn(0x7f100c0d27f0 legacy=0x7f100c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1003fff640 1 -- 172.21.15.77:0/2920460060 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c0b3710 con 0x7f100c0cf830 2024-03-06T07:33:23.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.172+0000 7f1009d9b640 1 -- 172.21.15.77:0/2920460060 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745807438 0 0) 0x7f100c0b3430 con 0x7f100c0d27f0 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.173+0000 7f1003fff640 1 --2- 172.21.15.77:0/2920460060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0cf830 0x7f100c0d22b0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009870 tx=0x7f0ff402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6c2473c1f0c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1009d9b640 1 -- 172.21.15.77:0/2920460060 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4034070 con 0x7f100c0cf830 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1009d9b640 1 -- 172.21.15.77:0/2920460060 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff4030840 con 0x7f100c0cf830 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1009d9b640 1 -- 172.21.15.77:0/2920460060 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4030c10 con 0x7f100c0cf830 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1010b6b640 1 -- 172.21.15.77:0/2920460060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0cf830 msgr2=0x7f100c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2920460060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0cf830 0x7f100c0d22b0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009870 tx=0x7f0ff402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1010b6b640 1 -- 172.21.15.77:0/2920460060 shutdown_connections 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2920460060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0cf830 0x7f100c0d22b0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2920460060 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100c15dfe0 0x7f100c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1010b6b640 1 -- 172.21.15.77:0/2920460060 >> 172.21.15.77:0/2920460060 conn(0x7f100c0d3cd0 msgr2=0x7f100c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1010b6b640 1 -- 172.21.15.77:0/2920460060 shutdown_connections 2024-03-06T07:33:23.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.174+0000 7f1010b6b640 1 -- 172.21.15.77:0/2920460060 wait complete. 2024-03-06T07:33:23.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.175+0000 7f1010b6b640 1 Processor -- start 2024-03-06T07:33:23.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1010b6b640 1 -- start start 2024-03-06T07:33:23.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1010b6b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c0cf830 0x7f100c0c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1010b6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0d27f0 0x7f100c15ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1010b6b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100c15dfe0 0x7f100c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1010b6b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0d1c70 con 0x7f100c0d27f0 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1010b6b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0b3d60 con 0x7f100c0cf830 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1010b6b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0d1df0 con 0x7f100c15dfe0 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1003fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0d27f0 0x7f100c15ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1003fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0d27f0 0x7f100c15ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52034/0 (socket says 172.21.15.77:52034) 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.176+0000 7f1003fff640 1 -- 172.21.15.77:0/2802012647 learned_addr learned my addr 172.21.15.77:0/2802012647 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1003fff640 1 -- 172.21.15.77:0/2802012647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100c15dfe0 msgr2=0x7f100c0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1003fff640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100c15dfe0 0x7f100c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1003fff640 1 -- 172.21.15.77:0/2802012647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c0cf830 msgr2=0x7f100c0c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1003fff640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c0cf830 0x7f100c0c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1003fff640 1 -- 172.21.15.77:0/2802012647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c160a00 con 0x7f100c0d27f0 2024-03-06T07:33:23.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1003fff640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0d27f0 0x7f100c15ff40 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4000c00 tx=0x7f0ff4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1002ffd640 1 -- 172.21.15.77:0/2802012647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4048070 con 0x7f100c0d27f0 2024-03-06T07:33:23.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1002ffd640 1 -- 172.21.15.77:0/2802012647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff4030640 con 0x7f100c0d27f0 2024-03-06T07:33:23.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c160ce0 con 0x7f100c0d27f0 2024-03-06T07:33:23.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1002ffd640 1 -- 172.21.15.77:0/2802012647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4034040 con 0x7f100c0d27f0 2024-03-06T07:33:23.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.177+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c0cbcf0 con 0x7f100c0d27f0 2024-03-06T07:33:23.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.178+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c0cb800 con 0x7f100c0d27f0 2024-03-06T07:33:23.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.178+0000 7f1002ffd640 1 -- 172.21.15.77:0/2802012647 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ff404b070 con 0x7f100c0d27f0 2024-03-06T07:33:23.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.179+0000 7f1002ffd640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:23.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.179+0000 7f1002ffd640 1 -- 172.21.15.77:0/2802012647 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0ff40ac280 con 0x7f100c0d27f0 2024-03-06T07:33:23.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.179+0000 7f100a59c640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:23.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.180+0000 7f100a59c640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009670 tx=0x7f0ffc006bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:23.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.183+0000 7f1002ffd640 1 -- 172.21.15.77:0/2802012647 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ff407b1a0 con 0x7f100c0d27f0 2024-03-06T07:33:23.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.399+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f100c04f330 con 0x7f100c0d27f0 2024-03-06T07:33:23.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.399+0000 7f1002ffd640 1 -- 172.21.15.77:0/2802012647 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f0ff407aaa0 con 0x7f100c0d27f0 2024-03-06T07:33:23.401 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:23.401 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:23.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 msgr2=0x7f0fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009670 tx=0x7f0ffc006bf0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0d27f0 msgr2=0x7f100c15ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0d27f0 0x7f100c15ff40 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4000c00 tx=0x7f0ff4004730 comp rx=0 tx=0).stop 2024-03-06T07:33:23.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 shutdown_connections 2024-03-06T07:33:23.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f0fec06b4d0 0x7f0fec06d990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f100c0d27f0 0x7f100c15ff40 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f100c15dfe0 0x7f100c0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 --2- 172.21.15.77:0/2802012647 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f100c0cf830 0x7f100c0c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 >> 172.21.15.77:0/2802012647 conn(0x7f100c0d3cd0 msgr2=0x7f100c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 shutdown_connections 2024-03-06T07:33:23.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:23.401+0000 7f1010b6b640 1 -- 172.21.15.77:0/2802012647 wait complete. 2024-03-06T07:33:23.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:23.615 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:24.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:24.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.716+0000 7f674bf56640 1 Processor -- start 2024-03-06T07:33:24.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f674bf56640 1 -- start start 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f674bf56640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f67440cfff0 0x7f67440d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f674bf56640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cc8b0 0x7f67440ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f674bf56640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6744161980 con 0x7f674415bb20 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f674bf56640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67440d2500 con 0x7f67440cfff0 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f674bf56640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6744161c60 con 0x7f67440cc8b0 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f67494f1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f67440cfff0 0x7f67440d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f67494f1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f67440cfff0 0x7f67440d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52050/0 (socket says 172.21.15.77:52050) 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.717+0000 7f67494f1640 1 -- 172.21.15.77:0/103917795 learned_addr learned my addr 172.21.15.77:0/103917795 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.718+0000 7f67494f1640 1 -- 172.21.15.77:0/103917795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cc8b0 msgr2=0x7f67440ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.718+0000 7f674a4f3640 1 --2- 172.21.15.77:0/103917795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cc8b0 0x7f67440ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.718+0000 7f67494f1640 1 --2- 172.21.15.77:0/103917795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cc8b0 0x7f67440ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.718+0000 7f67494f1640 1 -- 172.21.15.77:0/103917795 >> v1:172.21.15.158:6789/0 conn(0x7f674415bb20 legacy=0x7f67440d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.718+0000 7f67494f1640 1 -- 172.21.15.77:0/103917795 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6744161f40 con 0x7f67440cfff0 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f67494f1640 1 --2- 172.21.15.77:0/103917795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cfff0 0x7f67440d04a0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f6740009870 tx=0x7f674002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7da348c4dfdfa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f6748cf0640 1 -- 172.21.15.77:0/103917795 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740035020 con 0x7f67440cfff0 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f6748cf0640 1 -- 172.21.15.77:0/103917795 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6740031640 con 0x7f67440cfff0 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f6748cf0640 1 -- 172.21.15.77:0/103917795 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674002d040 con 0x7f67440cfff0 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f674bf56640 1 -- 172.21.15.77:0/103917795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cfff0 msgr2=0x7f67440d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f674bf56640 1 --2- 172.21.15.77:0/103917795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cfff0 0x7f67440d04a0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f6740009870 tx=0x7f674002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f674bf56640 1 -- 172.21.15.77:0/103917795 shutdown_connections 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f674bf56640 1 --2- 172.21.15.77:0/103917795 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cc8b0 0x7f67440ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f674bf56640 1 --2- 172.21.15.77:0/103917795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cfff0 0x7f67440d04a0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f674bf56640 1 -- 172.21.15.77:0/103917795 >> 172.21.15.77:0/103917795 conn(0x7f674409a9f0 msgr2=0x7f674409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f674bf56640 1 -- 172.21.15.77:0/103917795 shutdown_connections 2024-03-06T07:33:24.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.719+0000 7f674bf56640 1 -- 172.21.15.77:0/103917795 wait complete. 2024-03-06T07:33:24.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.720+0000 7f674bf56640 1 Processor -- start 2024-03-06T07:33:24.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.720+0000 7f674bf56640 1 -- start start 2024-03-06T07:33:24.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f674bf56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cc8b0 0x7f67440cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f674bf56640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cfff0 0x7f67440c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f674bf56640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f674415bb20 0x7f67441f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f674bf56640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67440d1980 con 0x7f67440cc8b0 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f674bf56640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6744162500 con 0x7f674415bb20 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f674bf56640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67440d1b00 con 0x7f67440cfff0 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f6749cf2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cc8b0 0x7f67440cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f6749cf2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cc8b0 0x7f67440cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52062/0 (socket says 172.21.15.77:52062) 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f6749cf2640 1 -- 172.21.15.77:0/418213434 learned_addr learned my addr 172.21.15.77:0/418213434 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f674a4f3640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f674415bb20 0x7f67441f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.721+0000 7f67494f1640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cfff0 0x7f67440c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f6749cf2640 1 -- 172.21.15.77:0/418213434 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cfff0 msgr2=0x7f67440c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f6749cf2640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cfff0 0x7f67440c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f6749cf2640 1 -- 172.21.15.77:0/418213434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f674415bb20 msgr2=0x7f67441f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f6749cf2640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f674415bb20 0x7f67441f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f6749cf2640 1 -- 172.21.15.77:0/418213434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67440c7dd0 con 0x7f67440cc8b0 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f6749cf2640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cc8b0 0x7f67440cb440 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f673800b420 tx=0x7f673800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f67367fc640 1 -- 172.21.15.77:0/418213434 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738011830 con 0x7f67440cc8b0 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f67367fc640 1 -- 172.21.15.77:0/418213434 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738016070 con 0x7f67440cc8b0 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67440c80b0 con 0x7f67440cc8b0 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f67367fc640 1 -- 172.21.15.77:0/418213434 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738011c10 con 0x7f67440cc8b0 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.722+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674406be60 con 0x7f67440cc8b0 2024-03-06T07:33:24.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.723+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674404f330 con 0x7f67440cc8b0 2024-03-06T07:33:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.724+0000 7f67367fc640 1 -- 172.21.15.77:0/418213434 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f673801b460 con 0x7f67440cc8b0 2024-03-06T07:33:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.724+0000 7f67367fc640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f672806b4d0 0x7f672806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.724+0000 7f67367fc640 1 -- 172.21.15.77:0/418213434 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6738089d40 con 0x7f67440cc8b0 2024-03-06T07:33:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.724+0000 7f67494f1640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f672806b4d0 0x7f672806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.725+0000 7f67494f1640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f672806b4d0 0x7f672806d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f6740036000 tx=0x7f674003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.728+0000 7f67367fc640 1 -- 172.21.15.77:0/418213434 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6738020020 con 0x7f67440cc8b0 2024-03-06T07:33:24.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.944+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f674406f730 con 0x7f67440cc8b0 2024-03-06T07:33:24.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.944+0000 7f67367fc640 1 -- 172.21.15.77:0/418213434 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f6738058ba0 con 0x7f67440cc8b0 2024-03-06T07:33:24.947 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:24.947 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:24.947 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:24.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.946+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f672806b4d0 msgr2=0x7f672806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.946+0000 7f674bf56640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f672806b4d0 0x7f672806d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f6740036000 tx=0x7f674003c040 comp rx=0 tx=0).stop 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.946+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cc8b0 msgr2=0x7f67440cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.946+0000 7f674bf56640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cc8b0 0x7f67440cb440 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f673800b420 tx=0x7f673800b8e0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.947+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 shutdown_connections 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.947+0000 7f674bf56640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f674415bb20 0x7f67441f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.947+0000 7f674bf56640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f67440cfff0 0x7f67440c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.947+0000 7f674bf56640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f672806b4d0 0x7f672806d990 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.947+0000 7f674bf56640 1 --2- 172.21.15.77:0/418213434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67440cc8b0 0x7f67440cb440 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.947+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 >> 172.21.15.77:0/418213434 conn(0x7f674409a9f0 msgr2=0x7f674415af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.947+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 shutdown_connections 2024-03-06T07:33:24.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:24.947+0000 7f674bf56640 1 -- 172.21.15.77:0/418213434 wait complete. 2024-03-06T07:33:25.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:25.161 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:26.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb362d3e640 1 Processor -- start 2024-03-06T07:33:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb362d3e640 1 -- start start 2024-03-06T07:33:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb362d3e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb35c15dfe0 0x7fb35c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb362d3e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0b3430 con 0x7fb35c0cf830 2024-03-06T07:33:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb362d3e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0b3710 con 0x7fb35c0d29c0 2024-03-06T07:33:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb362d3e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35c0b31e0 con 0x7fb35c15dfe0 2024-03-06T07:33:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb360ada640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb35c15dfe0 0x7fb35c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb360ada640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb35c15dfe0 0x7fb35c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52064/0 (socket says 172.21.15.77:52064) 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb360ada640 1 -- 172.21.15.77:0/1911019399 learned_addr learned my addr 172.21.15.77:0/1911019399 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.266+0000 7fb353fff640 1 --1- 172.21.15.77:0/1911019399 >> v1:172.21.15.77:6789/0 conn(0x7fb35c0cf830 0x7fb35c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49874/0 (socket says 172.21.15.77:49874) 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.267+0000 7fb360ada640 1 -- 172.21.15.77:0/1911019399 >> v1:172.21.15.158:6789/0 conn(0x7fb35c0d29c0 legacy=0x7fb35c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.267+0000 7fb360ada640 1 -- 172.21.15.77:0/1911019399 >> v1:172.21.15.77:6789/0 conn(0x7fb35c0cf830 legacy=0x7fb35c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.267+0000 7fb360ada640 1 -- 172.21.15.77:0/1911019399 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0b39f0 con 0x7fb35c15dfe0 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.267+0000 7fb360ada640 1 --2- 172.21.15.77:0/1911019399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c15dfe0 0x7fb35c0cf2f0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0b3360 tx=0x7fb35802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0d4bf673b9e7dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb3537fe640 1 -- 172.21.15.77:0/1911019399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35802caf0 con 0x7fb35c15dfe0 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb3537fe640 1 -- 172.21.15.77:0/1911019399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35802cc90 con 0x7fb35c15dfe0 2024-03-06T07:33:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb3537fe640 1 -- 172.21.15.77:0/1911019399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358034610 con 0x7fb35c15dfe0 2024-03-06T07:33:26.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb362d3e640 1 -- 172.21.15.77:0/1911019399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c15dfe0 msgr2=0x7fb35c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb362d3e640 1 --2- 172.21.15.77:0/1911019399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c15dfe0 0x7fb35c0cf2f0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0b3360 tx=0x7fb35802bf50 comp rx=0 tx=0).stop 2024-03-06T07:33:26.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb362d3e640 1 -- 172.21.15.77:0/1911019399 shutdown_connections 2024-03-06T07:33:26.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb362d3e640 1 --2- 172.21.15.77:0/1911019399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c15dfe0 0x7fb35c0cf2f0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb362d3e640 1 -- 172.21.15.77:0/1911019399 >> 172.21.15.77:0/1911019399 conn(0x7fb35c0d3cd0 msgr2=0x7fb35c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb362d3e640 1 -- 172.21.15.77:0/1911019399 shutdown_connections 2024-03-06T07:33:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.268+0000 7fb362d3e640 1 -- 172.21.15.77:0/1911019399 wait complete. 2024-03-06T07:33:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.270+0000 7fb362d3e640 1 Processor -- start 2024-03-06T07:33:26.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.270+0000 7fb362d3e640 1 -- start start 2024-03-06T07:33:26.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.270+0000 7fb362d3e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb35c0cf830 0x7fb35c15fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.270+0000 7fb362d3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c0d29c0 0x7fb35c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.270+0000 7fb362d3e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb35c15dfe0 0x7fb35c09e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.270+0000 7fb362d3e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c15e3c0 con 0x7fb35c0d29c0 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.270+0000 7fb362d3e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c0b3d60 con 0x7fb35c0cf830 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.270+0000 7fb362d3e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c15e540 con 0x7fb35c15dfe0 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c0d29c0 0x7fb35c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c0d29c0 0x7fb35c15bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52080/0 (socket says 172.21.15.77:52080) 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 -- 172.21.15.77:0/314178012 learned_addr learned my addr 172.21.15.77:0/314178012 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb360ada640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb35c0cf830 0x7fb35c15fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb3612db640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb35c15dfe0 0x7fb35c09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 -- 172.21.15.77:0/314178012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb35c15dfe0 msgr2=0x7fb35c09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb35c15dfe0 0x7fb35c09e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 -- 172.21.15.77:0/314178012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb35c0cf830 msgr2=0x7fb35c15fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb35c0cf830 0x7fb35c15fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 -- 172.21.15.77:0/314178012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0cb0d0 con 0x7fb35c0d29c0 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb353fff640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c0d29c0 0x7fb35c15bda0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c470 tx=0x7fb34c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb3517fa640 1 -- 172.21.15.77:0/314178012 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0074e0 con 0x7fb35c0d29c0 2024-03-06T07:33:26.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb3517fa640 1 -- 172.21.15.77:0/314178012 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c007680 con 0x7fb35c0d29c0 2024-03-06T07:33:26.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c0cdaa0 con 0x7fb35c0d29c0 2024-03-06T07:33:26.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb3517fa640 1 -- 172.21.15.77:0/314178012 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004a20 con 0x7fb35c0d29c0 2024-03-06T07:33:26.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.271+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c15f4e0 con 0x7fb35c0d29c0 2024-03-06T07:33:26.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.273+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c04f330 con 0x7fb35c0d29c0 2024-03-06T07:33:26.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.273+0000 7fb3517fa640 1 -- 172.21.15.77:0/314178012 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb34c01a020 con 0x7fb35c0d29c0 2024-03-06T07:33:26.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.273+0000 7fb3517fa640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb34806b4d0 0x7fb34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.273+0000 7fb3517fa640 1 -- 172.21.15.77:0/314178012 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb34c089ad0 con 0x7fb35c0d29c0 2024-03-06T07:33:26.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.273+0000 7fb360ada640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb34806b4d0 0x7fb34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.274+0000 7fb360ada640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb34806b4d0 0x7fb34806d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb358035000 tx=0x7fb358030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.278+0000 7fb3517fa640 1 -- 172.21.15.77:0/314178012 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb35c0cb0d0 con 0x7fb35c0d29c0 2024-03-06T07:33:26.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.493+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb35c0b5340 con 0x7fb35c0d29c0 2024-03-06T07:33:26.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.494+0000 7fb3517fa640 1 -- 172.21.15.77:0/314178012 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fb34c0588a0 con 0x7fb35c0d29c0 2024-03-06T07:33:26.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:26.496 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:26.496 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:26.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb34806b4d0 msgr2=0x7fb34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb34806b4d0 0x7fb34806d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb358035000 tx=0x7fb358030000 comp rx=0 tx=0).stop 2024-03-06T07:33:26.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c0d29c0 msgr2=0x7fb35c15bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c0d29c0 0x7fb35c15bda0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c470 tx=0x7fb34c001e70 comp rx=0 tx=0).stop 2024-03-06T07:33:26.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 shutdown_connections 2024-03-06T07:33:26.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb34806b4d0 0x7fb34806d990 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb35c0d29c0 0x7fb35c15bda0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb35c15dfe0 0x7fb35c09e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 --2- 172.21.15.77:0/314178012 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb35c0cf830 0x7fb35c15fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 >> 172.21.15.77:0/314178012 conn(0x7fb35c0d3cd0 msgr2=0x7fb35c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 shutdown_connections 2024-03-06T07:33:26.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:26.496+0000 7fb362d3e640 1 -- 172.21.15.77:0/314178012 wait complete. 2024-03-06T07:33:26.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:26.714 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:27.715 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:27.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.815+0000 7f8afc681640 1 Processor -- start 2024-03-06T07:33:27.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.815+0000 7f8afc681640 1 -- start start 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.816+0000 7f8afc681640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af415ddf0 0x7f8af40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.816+0000 7f8afc681640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f8af415e1d0 0x7f8af40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.816+0000 7f8afc681640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af40b7b60 con 0x7f8af40d06d0 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.816+0000 7f8afc681640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af40d27c0 con 0x7f8af415e1d0 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.816+0000 7f8afc681640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40d18d0 con 0x7f8af415ddf0 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.816+0000 7f8afac1e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8af40d06d0 0x7f8af4161ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:49884/0 (socket says 172.21.15.77:49884) 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.816+0000 7f8afac1e640 1 -- 172.21.15.77:0/1727476049 learned_addr learned my addr 172.21.15.77:0/1727476049 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af9c1c640 1 --2- 172.21.15.77:0/1727476049 >> v2:172.21.15.158:3300/0 conn(0x7f8af415e1d0 0x7f8af40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8afa41d640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af415ddf0 0x7f8af40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195114132 0 0) 0x7f8af40b7b60 con 0x7f8af40d06d0 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae0003520 con 0x7f8af40d06d0 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3125856877 0 0) 0x7f8ae0003520 con 0x7f8af40d06d0 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af415ddf0 msgr2=0x7f8af40cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af415ddf0 0x7f8af40cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 >> v2:172.21.15.158:3300/0 conn(0x7f8af415e1d0 msgr2=0x7f8af40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 --2- 172.21.15.77:0/1727476049 >> v2:172.21.15.158:3300/0 conn(0x7f8af415e1d0 0x7f8af40cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af40d1bb0 con 0x7f8af40d06d0 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f8aec0019f0 con 0x7f8af40d06d0 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 >> v1:172.21.15.77:6789/0 conn(0x7f8af40d06d0 legacy=0x7f8af4161ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ae00039b0 0x7f8ae0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8ae0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af415ddf0 0x7f8ae00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40d27c0 con 0x7f8ae0004800 2024-03-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40d18d0 con 0x7f8ae00039b0 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae000b9c0 con 0x7f8af415ddf0 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af9c1c640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8ae0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8afa41d640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ae00039b0 0x7f8ae0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8afac1e640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af415ddf0 0x7f8ae00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af9c1c640 1 -- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af415ddf0 msgr2=0x7f8ae00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af9c1c640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af415ddf0 0x7f8ae00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af9c1c640 1 -- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ae00039b0 msgr2=0x7f8ae0003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af9c1c640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ae00039b0 0x7f8ae0003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.817+0000 7f8af9c1c640 1 -- 172.21.15.77:0/1727476049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0003520 con 0x7f8ae0004800 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.818+0000 7f8af9c1c640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8ae0004c80 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009ac0 tx=0x7f8ae0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da7a6771ca70e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.818+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af0039070 con 0x7f8ae0004800 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.818+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af002c6b0 con 0x7f8ae0004800 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.818+0000 7f8af941b640 1 -- 172.21.15.77:0/1727476049 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af0009690 con 0x7f8ae0004800 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.819+0000 7f8afc681640 1 -- 172.21.15.77:0/1727476049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 msgr2=0x7f8ae0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.819+0000 7f8afc681640 1 --2- 172.21.15.77:0/1727476049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8ae0004c80 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009ac0 tx=0x7f8ae0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:27.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.819+0000 7f8afac1e640 1 -- 172.21.15.77:0/1727476049 reap_dead start 2024-03-06T07:33:27.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.819+0000 7f8afc681640 1 -- 172.21.15.77:0/1727476049 shutdown_connections 2024-03-06T07:33:27.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.819+0000 7f8afc681640 1 -- 172.21.15.77:0/1727476049 >> 172.21.15.77:0/1727476049 conn(0x7f8af409cff0 msgr2=0x7f8af40d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.819+0000 7f8afc681640 1 -- 172.21.15.77:0/1727476049 shutdown_connections 2024-03-06T07:33:27.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.819+0000 7f8afc681640 1 -- 172.21.15.77:0/1727476049 wait complete. 2024-03-06T07:33:27.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.820+0000 7f8afc681640 1 Processor -- start 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.820+0000 7f8afc681640 1 -- start start 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afc681640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8af41557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afc681640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8af4162120 0x7f8af415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afc681640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af4162580 0x7f8af415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afc681640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40d2450 con 0x7f8ae0004800 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afc681640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40d2170 con 0x7f8af4162120 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afc681640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40d2c50 con 0x7f8af4162580 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8af41557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8af41557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52088/0 (socket says 172.21.15.77:52088) 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 -- 172.21.15.77:0/2855961080 learned_addr learned my addr 172.21.15.77:0/2855961080 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 -- 172.21.15.77:0/2855961080 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af4162580 msgr2=0x7f8af415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af4162580 0x7f8af415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 -- 172.21.15.77:0/2855961080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8af4162120 msgr2=0x7f8af415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8af4162120 0x7f8af415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 -- 172.21.15.77:0/2855961080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af40c99a0 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afa41d640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8af41557f0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8011630 tx=0x7f8ae8011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8ae6ffd640 1 -- 172.21.15.77:0/2855961080 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae800f920 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8ae6ffd640 1 -- 172.21.15.77:0/2855961080 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae800fac0 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af40c9c60 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.821+0000 7f8ae6ffd640 1 -- 172.21.15.77:0/2855961080 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae800fe70 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.822+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af40ca220 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.822+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af404f330 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.823+0000 7f8ae6ffd640 1 -- 172.21.15.77:0/2855961080 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ae8015460 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.823+0000 7f8ae6ffd640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ad806b4d0 0x7f8ad806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.823+0000 7f8ae6ffd640 1 -- 172.21.15.77:0/2855961080 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8ae808aaf0 con 0x7f8ae0004800 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.823+0000 7f8af9c1c640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ad806b4d0 0x7f8ad806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.824+0000 7f8af9c1c640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ad806b4d0 0x7f8ad806d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8af0036000 tx=0x7f8af002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:27.832+0000 7f8ae6ffd640 1 -- 172.21.15.77:0/2855961080 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ae8019030 con 0x7f8ae0004800 2024-03-06T07:33:28.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.047+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8af406c7f0 con 0x7f8ae0004800 2024-03-06T07:33:28.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.048+0000 7f8ae6ffd640 1 -- 172.21.15.77:0/2855961080 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f8ae8059300 con 0x7f8ae0004800 2024-03-06T07:33:28.050 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:28.050 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:28.050 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ad806b4d0 msgr2=0x7f8ad806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ad806b4d0 0x7f8ad806d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8af0036000 tx=0x7f8af002e000 comp rx=0 tx=0).stop 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 msgr2=0x7f8af41557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8af41557f0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8011630 tx=0x7f8ae8011af0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 shutdown_connections 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8af4162580 0x7f8af415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8af4162120 0x7f8af415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8ad806b4d0 0x7f8ad806d990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 --2- 172.21.15.77:0/2855961080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae0004800 0x7f8af41557f0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 >> 172.21.15.77:0/2855961080 conn(0x7f8af409cff0 msgr2=0x7f8af406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 shutdown_connections 2024-03-06T07:33:28.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:28.050+0000 7f8afc681640 1 -- 172.21.15.77:0/2855961080 wait complete. 2024-03-06T07:33:28.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:28.264 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:29.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:29.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.368+0000 7f86c8c88640 1 Processor -- start 2024-03-06T07:33:29.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.368+0000 7f86c8c88640 1 -- start start 2024-03-06T07:33:29.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.369+0000 7f86c8c88640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86c40d22b0 0x7f86c40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.369+0000 7f86c8c88640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c40b3430 con 0x7f86c415dfe0 2024-03-06T07:33:29.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.369+0000 7f86c8c88640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c40b3710 con 0x7f86c40b41f0 2024-03-06T07:33:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.369+0000 7f86c8c88640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c40d2c50 con 0x7f86c40d22b0 2024-03-06T07:33:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.369+0000 7f86c1d9b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86c40d22b0 0x7f86c40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.369+0000 7f86c259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f86c415dfe0 0x7f86c40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35680/0 (socket says 172.21.15.77:35680) 2024-03-06T07:33:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.369+0000 7f86c259c640 1 -- 172.21.15.77:0/3659281904 learned_addr learned my addr 172.21.15.77:0/3659281904 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.370+0000 7f86c1d9b640 1 -- 172.21.15.77:0/3659281904 >> v1:172.21.15.158:6789/0 conn(0x7f86c40b41f0 legacy=0x7f86c40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.370+0000 7f86c1d9b640 1 -- 172.21.15.77:0/3659281904 >> v1:172.21.15.77:6789/0 conn(0x7f86c415dfe0 legacy=0x7f86c40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.370+0000 7f86c1d9b640 1 -- 172.21.15.77:0/3659281904 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c40b39f0 con 0x7f86c40d22b0 2024-03-06T07:33:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.370+0000 7f86c1d9b640 1 --2- 172.21.15.77:0/3659281904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40d22b0 0x7f86c40d2710 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f86b40097d0 tx=0x7f86b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90676073a43f5b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.370+0000 7f86c159a640 1 -- 172.21.15.77:0/3659281904 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4034070 con 0x7f86c40d22b0 2024-03-06T07:33:29.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c159a640 1 -- 172.21.15.77:0/3659281904 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b4030610 con 0x7f86c40d22b0 2024-03-06T07:33:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c159a640 1 -- 172.21.15.77:0/3659281904 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40309e0 con 0x7f86c40d22b0 2024-03-06T07:33:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c8c88640 1 -- 172.21.15.77:0/3659281904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40d22b0 msgr2=0x7f86c40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c8c88640 1 --2- 172.21.15.77:0/3659281904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40d22b0 0x7f86c40d2710 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f86b40097d0 tx=0x7f86b402b960 comp rx=0 tx=0).stop 2024-03-06T07:33:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c8c88640 1 -- 172.21.15.77:0/3659281904 shutdown_connections 2024-03-06T07:33:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c8c88640 1 --2- 172.21.15.77:0/3659281904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40d22b0 0x7f86c40d2710 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c8c88640 1 -- 172.21.15.77:0/3659281904 >> 172.21.15.77:0/3659281904 conn(0x7f86c40d3cd0 msgr2=0x7f86c40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c8c88640 1 -- 172.21.15.77:0/3659281904 shutdown_connections 2024-03-06T07:33:29.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.371+0000 7f86c8c88640 1 -- 172.21.15.77:0/3659281904 wait complete. 2024-03-06T07:33:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.372+0000 7f86c8c88640 1 Processor -- start 2024-03-06T07:33:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.372+0000 7f86c8c88640 1 -- start start 2024-03-06T07:33:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c8c88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40b41f0 0x7f86c415e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c8c88640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c40d22b0 0x7f86c40cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c8c88640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86c415dfe0 0x7f86c40cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c8c88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40b3d60 con 0x7f86c40b41f0 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c8c88640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40b31e0 con 0x7f86c40d22b0 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c8c88640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40d1c70 con 0x7f86c415dfe0 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40b41f0 0x7f86c415e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40b41f0 0x7f86c415e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48604/0 (socket says 172.21.15.77:48604) 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.373+0000 7f86c259c640 1 -- 172.21.15.77:0/820654051 learned_addr learned my addr 172.21.15.77:0/820654051 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c1d9b640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c40d22b0 0x7f86c40cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c259c640 1 -- 172.21.15.77:0/820654051 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86c415dfe0 msgr2=0x7f86c40cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c259c640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86c415dfe0 0x7f86c40cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c259c640 1 -- 172.21.15.77:0/820654051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c40d22b0 msgr2=0x7f86c40cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c259c640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c40d22b0 0x7f86c40cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c259c640 1 -- 172.21.15.77:0/820654051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c415ec10 con 0x7f86c40b41f0 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c259c640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40b41f0 0x7f86c415e6d0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f86b800c480 tx=0x7f86b800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86b2ffd640 1 -- 172.21.15.77:0/820654051 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8017070 con 0x7f86c40b41f0 2024-03-06T07:33:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86b2ffd640 1 -- 172.21.15.77:0/820654051 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b8005ce0 con 0x7f86c40b41f0 2024-03-06T07:33:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c40cb6f0 con 0x7f86c40b41f0 2024-03-06T07:33:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86b2ffd640 1 -- 172.21.15.77:0/820654051 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b80072f0 con 0x7f86c40b41f0 2024-03-06T07:33:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.374+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c40cbc70 con 0x7f86c40b41f0 2024-03-06T07:33:29.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.375+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c404f330 con 0x7f86c40b41f0 2024-03-06T07:33:29.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.375+0000 7f86b2ffd640 1 -- 172.21.15.77:0/820654051 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86b801a070 con 0x7f86c40b41f0 2024-03-06T07:33:29.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.376+0000 7f86b2ffd640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a406b500 0x7f86a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.376+0000 7f86c1d9b640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a406b500 0x7f86a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.376+0000 7f86b2ffd640 1 -- 172.21.15.77:0/820654051 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f86b8089450 con 0x7f86c40b41f0 2024-03-06T07:33:29.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.376+0000 7f86c1d9b640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a406b500 0x7f86a406d9c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f86b4036000 tx=0x7f86b403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.380+0000 7f86b2ffd640 1 -- 172.21.15.77:0/820654051 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86b8057cf0 con 0x7f86c40b41f0 2024-03-06T07:33:29.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.592+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86c40b9b30 con 0x7f86c40b41f0 2024-03-06T07:33:29.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.592+0000 7f86b2ffd640 1 -- 172.21.15.77:0/820654051 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f86b805bd90 con 0x7f86c40b41f0 2024-03-06T07:33:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:29.595 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:29.595 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.594+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a406b500 msgr2=0x7f86a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.594+0000 7f86c8c88640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a406b500 0x7f86a406d9c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f86b4036000 tx=0x7f86b403b040 comp rx=0 tx=0).stop 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.594+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40b41f0 msgr2=0x7f86c415e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.594+0000 7f86c8c88640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40b41f0 0x7f86c415e6d0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f86b800c480 tx=0x7f86b800c940 comp rx=0 tx=0).stop 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.595+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 shutdown_connections 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.595+0000 7f86c8c88640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86c415dfe0 0x7f86c40cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.595+0000 7f86c8c88640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86c40d22b0 0x7f86c40cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.595+0000 7f86c8c88640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a406b500 0x7f86a406d9c0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.595+0000 7f86c8c88640 1 --2- 172.21.15.77:0/820654051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86c40b41f0 0x7f86c415e6d0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.595+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 >> 172.21.15.77:0/820654051 conn(0x7f86c40d3cd0 msgr2=0x7f86c40d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.595+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 shutdown_connections 2024-03-06T07:33:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:29.595+0000 7f86c8c88640 1 -- 172.21.15.77:0/820654051 wait complete. 2024-03-06T07:33:29.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:29.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:30.809 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:30.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.910+0000 7f8ea7a34640 1 Processor -- start 2024-03-06T07:33:30.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.910+0000 7f8ea7a34640 1 -- start start 2024-03-06T07:33:30.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea7a34640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ea00d22b0 0x7f8ea00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea7a34640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea00b41f0 0x7f8ea00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea7a34640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00b3430 con 0x7f8ea015dfe0 2024-03-06T07:33:30.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea7a34640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea00d2c50 con 0x7f8ea00d22b0 2024-03-06T07:33:30.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea7a34640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00b31e0 con 0x7f8ea00b41f0 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea4fcf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ea00d22b0 0x7f8ea00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea4fcf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ea00d22b0 0x7f8ea00d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48606/0 (socket says 172.21.15.77:48606) 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea4fcf640 1 -- 172.21.15.77:0/1275098398 learned_addr learned my addr 172.21.15.77:0/1275098398 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea5fd1640 1 --2- 172.21.15.77:0/1275098398 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea00b41f0 0x7f8ea00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea4fcf640 1 -- 172.21.15.77:0/1275098398 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea00b41f0 msgr2=0x7f8ea00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea4fcf640 1 --2- 172.21.15.77:0/1275098398 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea00b41f0 0x7f8ea00b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea4fcf640 1 -- 172.21.15.77:0/1275098398 >> v1:172.21.15.158:6789/0 conn(0x7f8ea015dfe0 legacy=0x7f8ea00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea4fcf640 1 -- 172.21.15.77:0/1275098398 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea00b3710 con 0x7f8ea00d22b0 2024-03-06T07:33:30.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.911+0000 7f8ea5fd1640 1 --2- 172.21.15.77:0/1275098398 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea00b41f0 0x7f8ea00b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.912+0000 7f8ea4fcf640 1 --2- 172.21.15.77:0/1275098398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00d2710 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009870 tx=0x7f8e9002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf57b3f9d0ae40d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.912+0000 7f8e97fff640 1 -- 172.21.15.77:0/1275098398 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90035020 con 0x7f8ea00d22b0 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.912+0000 7f8e97fff640 1 -- 172.21.15.77:0/1275098398 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e90031640 con 0x7f8ea00d22b0 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.912+0000 7f8e97fff640 1 -- 172.21.15.77:0/1275098398 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9002d040 con 0x7f8ea00d22b0 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.913+0000 7f8ea7a34640 1 -- 172.21.15.77:0/1275098398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 msgr2=0x7f8ea00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.913+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/1275098398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00d2710 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009870 tx=0x7f8e9002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.913+0000 7f8ea7a34640 1 -- 172.21.15.77:0/1275098398 shutdown_connections 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.913+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/1275098398 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea00b41f0 0x7f8ea00b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.913+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/1275098398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00d2710 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.913+0000 7f8ea7a34640 1 -- 172.21.15.77:0/1275098398 >> 172.21.15.77:0/1275098398 conn(0x7f8ea00d3cd0 msgr2=0x7f8ea00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.913+0000 7f8ea7a34640 1 -- 172.21.15.77:0/1275098398 shutdown_connections 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.913+0000 7f8ea7a34640 1 -- 172.21.15.77:0/1275098398 wait complete. 2024-03-06T07:33:30.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.914+0000 7f8ea7a34640 1 Processor -- start 2024-03-06T07:33:30.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.914+0000 7f8ea7a34640 1 -- start start 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea7a34640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea00b41f0 0x7f8ea0161260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea7a34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea7a34640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea015dfe0 0x7f8ea01f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea7a34640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00cfa00 con 0x7f8ea00d22b0 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea7a34640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00b3d60 con 0x7f8ea00b41f0 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea7a34640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00d1c70 con 0x7f8ea015dfe0 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48610/0 (socket says 172.21.15.77:48610) 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 -- 172.21.15.77:0/2781222158 learned_addr learned my addr 172.21.15.77:0/2781222158 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea57d0640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea00b41f0 0x7f8ea0161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea5fd1640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea015dfe0 0x7f8ea01f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 -- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea015dfe0 msgr2=0x7f8ea01f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea015dfe0 0x7f8ea01f3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 -- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea00b41f0 msgr2=0x7f8ea0161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea00b41f0 0x7f8ea0161260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 -- 172.21.15.77:0/2781222158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea00b6de0 con 0x7f8ea00d22b0 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea57d0640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea00b41f0 0x7f8ea0161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea5fd1640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea015dfe0 0x7f8ea01f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea4fcf640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00b68a0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f8e9002b930 tx=0x7f8e900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8e95ffb640 1 -- 172.21.15.77:0/2781222158 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90040070 con 0x7f8ea00d22b0 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8e95ffb640 1 -- 172.21.15.77:0/2781222158 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e90039c10 con 0x7f8ea00d22b0 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea00b97f0 con 0x7f8ea00d22b0 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.915+0000 7f8e95ffb640 1 -- 172.21.15.77:0/2781222158 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90035040 con 0x7f8ea00d22b0 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.916+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea00cc800 con 0x7f8ea00d22b0 2024-03-06T07:33:30.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.917+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea004f330 con 0x7f8ea00d22b0 2024-03-06T07:33:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.918+0000 7f8e95ffb640 1 -- 172.21.15.77:0/2781222158 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e9004b060 con 0x7f8ea00d22b0 2024-03-06T07:33:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.918+0000 7f8e95ffb640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e8406b4d0 0x7f8e8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.918+0000 7f8e95ffb640 1 -- 172.21.15.77:0/2781222158 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8e900abd90 con 0x7f8ea00d22b0 2024-03-06T07:33:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.918+0000 7f8ea57d0640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e8406b4d0 0x7f8e8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.919+0000 7f8ea57d0640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e8406b4d0 0x7f8e8406d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00a830 tx=0x7f8e9c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:30.923+0000 7f8e95ffb640 1 -- 172.21.15.77:0/2781222158 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e9002f300 con 0x7f8ea00d22b0 2024-03-06T07:33:31.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.138+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ea015d350 con 0x7f8ea00d22b0 2024-03-06T07:33:31.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.138+0000 7f8e95ffb640 1 -- 172.21.15.77:0/2781222158 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f8e9007abf0 con 0x7f8ea00d22b0 2024-03-06T07:33:31.140 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:31.140 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:31.141 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:31.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e8406b4d0 msgr2=0x7f8e8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e8406b4d0 0x7f8e8406d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00a830 tx=0x7f8e9c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:31.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 msgr2=0x7f8ea00b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00b68a0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f8e9002b930 tx=0x7f8e900046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 shutdown_connections 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8e8406b4d0 0x7f8e8406d990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea00d22b0 0x7f8ea00b68a0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8ea015dfe0 0x7f8ea01f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 --2- 172.21.15.77:0/2781222158 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8ea00b41f0 0x7f8ea0161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.140+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 >> 172.21.15.77:0/2781222158 conn(0x7f8ea00d3cd0 msgr2=0x7f8ea006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.141+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 shutdown_connections 2024-03-06T07:33:31.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:31.141+0000 7f8ea7a34640 1 -- 172.21.15.77:0/2781222158 wait complete. 2024-03-06T07:33:31.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:31.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:32.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:32.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.455+0000 7f77912c2640 1 Processor -- start 2024-03-06T07:33:32.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.455+0000 7f77912c2640 1 -- start start 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f77912c2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f778c15dfe0 0x7f778c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f77912c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c0cf830 0x7f778c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f77912c2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c0b3430 con 0x7f778c0d27f0 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f77912c2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778c0d2c50 con 0x7f778c15dfe0 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f77912c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0b31e0 con 0x7f778c0cf830 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f778affd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f778c15dfe0 0x7f778c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f778affd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f778c15dfe0 0x7f778c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48614/0 (socket says 172.21.15.77:48614) 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f778affd640 1 -- 172.21.15.77:0/684056393 learned_addr learned my addr 172.21.15.77:0/684056393 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f778a7fc640 1 --2- 172.21.15.77:0/684056393 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c0cf830 0x7f778c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f778affd640 1 -- 172.21.15.77:0/684056393 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c0cf830 msgr2=0x7f778c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f778affd640 1 --2- 172.21.15.77:0/684056393 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c0cf830 0x7f778c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f778affd640 1 -- 172.21.15.77:0/684056393 >> v1:172.21.15.77:6789/0 conn(0x7f778c0d27f0 legacy=0x7f778c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f778affd640 1 -- 172.21.15.77:0/684056393 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c0b3710 con 0x7f778c15dfe0 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.456+0000 7f7789ffb640 1 -- 172.21.15.77:0/684056393 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486555584 0 0) 0x7f778c0b3430 con 0x7f778c0d27f0 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.457+0000 7f778affd640 1 --2- 172.21.15.77:0/684056393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c15dfe0 0x7f778c0cf2f0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f777c009870 tx=0x7f777c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6a1e4ed82ff328 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.457+0000 7f7789ffb640 1 -- 172.21.15.77:0/684056393 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c034070 con 0x7f778c15dfe0 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.457+0000 7f7789ffb640 1 -- 172.21.15.77:0/684056393 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c030840 con 0x7f778c15dfe0 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.457+0000 7f7789ffb640 1 -- 172.21.15.77:0/684056393 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c030c10 con 0x7f778c15dfe0 2024-03-06T07:33:32.461 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:32.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.458+0000 7f77912c2640 1 -- 172.21.15.77:0/684056393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c15dfe0 msgr2=0x7f778c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.458+0000 7f77912c2640 1 --2- 172.21.15.77:0/684056393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c15dfe0 0x7f778c0cf2f0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f777c009870 tx=0x7f777c02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.458+0000 7f77912c2640 1 -- 172.21.15.77:0/684056393 shutdown_connections 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.458+0000 7f77912c2640 1 --2- 172.21.15.77:0/684056393 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c0cf830 0x7f778c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.458+0000 7f77912c2640 1 --2- 172.21.15.77:0/684056393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c15dfe0 0x7f778c0cf2f0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.458+0000 7f77912c2640 1 -- 172.21.15.77:0/684056393 >> 172.21.15.77:0/684056393 conn(0x7f778c0d3cd0 msgr2=0x7f778c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.458+0000 7f77912c2640 1 -- 172.21.15.77:0/684056393 shutdown_connections 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.458+0000 7f77912c2640 1 -- 172.21.15.77:0/684056393 wait complete. 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.459+0000 7f77912c2640 1 Processor -- start 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.459+0000 7f77912c2640 1 -- start start 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f77912c2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f778c0cf830 0x7f778c1f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f77912c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c0d27f0 0x7f778c160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f77912c2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c15dfe0 0x7f778c0b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f77912c2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0d1c70 con 0x7f778c0d27f0 2024-03-06T07:33:32.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f77912c2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0b3d60 con 0x7f778c0cf830 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f77912c2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0d1df0 con 0x7f778c15dfe0 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c0d27f0 0x7f778c160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c0d27f0 0x7f778c160820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48628/0 (socket says 172.21.15.77:48628) 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778a7fc640 1 -- 172.21.15.77:0/1023774077 learned_addr learned my addr 172.21.15.77:0/1023774077 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778affd640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f778c0cf830 0x7f778c1f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778b7fe640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c15dfe0 0x7f778c0b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778a7fc640 1 -- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c15dfe0 msgr2=0x7f778c0b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778a7fc640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c15dfe0 0x7f778c0b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778a7fc640 1 -- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f778c0cf830 msgr2=0x7f778c1f6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778a7fc640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f778c0cf830 0x7f778c1f6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778a7fc640 1 -- 172.21.15.77:0/1023774077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c15f4a0 con 0x7f778c0d27f0 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.460+0000 7f778affd640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f778c0cf830 0x7f778c1f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:32.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.461+0000 7f778a7fc640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c0d27f0 0x7f778c160820 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f778000d7d0 tx=0x7f778000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.461+0000 7f77737fe640 1 -- 172.21.15.77:0/1023774077 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778000b6c0 con 0x7f778c0d27f0 2024-03-06T07:33:32.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.461+0000 7f77737fe640 1 -- 172.21.15.77:0/1023774077 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7780019070 con 0x7f778c0d27f0 2024-03-06T07:33:32.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.461+0000 7f77737fe640 1 -- 172.21.15.77:0/1023774077 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778000bac0 con 0x7f778c0d27f0 2024-03-06T07:33:32.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.461+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c15f780 con 0x7f778c0d27f0 2024-03-06T07:33:32.466 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.461+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c15fd40 con 0x7f778c0d27f0 2024-03-06T07:33:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.462+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c04f330 con 0x7f778c0d27f0 2024-03-06T07:33:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.463+0000 7f77737fe640 1 -- 172.21.15.77:0/1023774077 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f778000bc60 con 0x7f778c0d27f0 2024-03-06T07:33:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.463+0000 7f77737fe640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f777806b500 0x7f777806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.463+0000 7f77737fe640 1 -- 172.21.15.77:0/1023774077 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f778008a6b0 con 0x7f778c0d27f0 2024-03-06T07:33:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.463+0000 7f778affd640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f777806b500 0x7f777806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.464+0000 7f778affd640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f777806b500 0x7f777806d9c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f777c036000 tx=0x7f777c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.468+0000 7f77737fe640 1 -- 172.21.15.77:0/1023774077 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f778001c360 con 0x7f778c0d27f0 2024-03-06T07:33:32.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.684+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f778c15d370 con 0x7f778c0d27f0 2024-03-06T07:33:32.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.684+0000 7f77737fe640 1 -- 172.21.15.77:0/1023774077 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f7780059510 con 0x7f778c0d27f0 2024-03-06T07:33:32.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:32.686 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:32.687 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:32.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f777806b500 msgr2=0x7f777806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f777806b500 0x7f777806d9c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f777c036000 tx=0x7f777c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:32.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c0d27f0 msgr2=0x7f778c160820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c0d27f0 0x7f778c160820 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f778000d7d0 tx=0x7f778000dc90 comp rx=0 tx=0).stop 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 shutdown_connections 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f777806b500 0x7f777806d9c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f778c0d27f0 0x7f778c160820 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f778c15dfe0 0x7f778c0b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 --2- 172.21.15.77:0/1023774077 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f778c0cf830 0x7f778c1f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 >> 172.21.15.77:0/1023774077 conn(0x7f778c0d3cd0 msgr2=0x7f778c0d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 shutdown_connections 2024-03-06T07:33:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:32.686+0000 7f77912c2640 1 -- 172.21.15.77:0/1023774077 wait complete. 2024-03-06T07:33:32.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:32.900 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:33.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:34.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.001+0000 7fb52fc4e640 1 Processor -- start 2024-03-06T07:33:34.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.001+0000 7fb52fc4e640 1 -- start start 2024-03-06T07:33:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.002+0000 7fb52fc4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb52815dfe0 0x7fb5280cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.002+0000 7fb52fc4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5280cf830 0x7fb5280d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.002+0000 7fb52fc4e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5280b3430 con 0x7fb5280d27f0 2024-03-06T07:33:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.002+0000 7fb52fc4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5280d2c50 con 0x7fb52815dfe0 2024-03-06T07:33:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.002+0000 7fb52fc4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5280b31e0 con 0x7fb5280cf830 2024-03-06T07:33:34.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.002+0000 7fb52d9ea640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb52815dfe0 0x7fb5280cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.002+0000 7fb52d1e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5280cf830 0x7fb5280d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52d9ea640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb52815dfe0 0x7fb5280cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:49624/0 (socket says 172.21.15.77:49624) 2024-03-06T07:33:34.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52d1e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5280cf830 0x7fb5280d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:33820/0 (socket says 172.21.15.77:33820) 2024-03-06T07:33:34.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52e1eb640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7fb5280d27f0 0x7fb5280d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:37610/0 (socket says 172.21.15.77:37610) 2024-03-06T07:33:34.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52d9ea640 1 -- 172.21.15.77:0/1009152811 learned_addr learned my addr 172.21.15.77:0/1009152811 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:34.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52d1e9640 1 -- 172.21.15.77:0/1009152811 >> v2:172.21.15.158:3300/0 conn(0x7fb52815dfe0 msgr2=0x7fb5280cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52d1e9640 1 --2- 172.21.15.77:0/1009152811 >> v2:172.21.15.158:3300/0 conn(0x7fb52815dfe0 0x7fb5280cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52d1e9640 1 -- 172.21.15.77:0/1009152811 >> v1:172.21.15.158:6789/0 conn(0x7fb5280d27f0 legacy=0x7fb5280d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52d1e9640 1 -- 172.21.15.77:0/1009152811 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5280b3710 con 0x7fb5280cf830 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.003+0000 7fb52d9ea640 1 --2- 172.21.15.77:0/1009152811 >> v2:172.21.15.158:3300/0 conn(0x7fb52815dfe0 0x7fb5280cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.004+0000 7fb52d1e9640 1 --2- 172.21.15.77:0/1009152811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5280cf830 0x7fb5280d22b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb5180097a0 tx=0x7fb51802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c1637bbfd09468 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.004+0000 7fb52c9e8640 1 -- 172.21.15.77:0/1009152811 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518035020 con 0x7fb5280cf830 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.004+0000 7fb52c9e8640 1 -- 172.21.15.77:0/1009152811 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb518031830 con 0x7fb5280cf830 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.004+0000 7fb52c9e8640 1 -- 172.21.15.77:0/1009152811 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518031c00 con 0x7fb5280cf830 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.004+0000 7fb52fc4e640 1 -- 172.21.15.77:0/1009152811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5280cf830 msgr2=0x7fb5280d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.004+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/1009152811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5280cf830 0x7fb5280d22b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb5180097a0 tx=0x7fb51802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.005+0000 7fb52fc4e640 1 -- 172.21.15.77:0/1009152811 shutdown_connections 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.005+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/1009152811 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb5280cf830 0x7fb5280d22b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.005+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/1009152811 >> v2:172.21.15.158:3300/0 conn(0x7fb52815dfe0 0x7fb5280cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.005+0000 7fb52fc4e640 1 -- 172.21.15.77:0/1009152811 >> 172.21.15.77:0/1009152811 conn(0x7fb5280d3cd0 msgr2=0x7fb5280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.005+0000 7fb52fc4e640 1 -- 172.21.15.77:0/1009152811 shutdown_connections 2024-03-06T07:33:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.005+0000 7fb52fc4e640 1 -- 172.21.15.77:0/1009152811 wait complete. 2024-03-06T07:33:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.006+0000 7fb52fc4e640 1 Processor -- start 2024-03-06T07:33:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.007+0000 7fb52fc4e640 1 -- start start 2024-03-06T07:33:34.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52fc4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5280cf830 0x7fb5280cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52fc4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5280d27f0 0x7fb5281f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52fc4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb52815dfe0 0x7fb5281f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52fc4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5280d1c70 con 0x7fb5280cf830 2024-03-06T07:33:34.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52fc4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5280b3d60 con 0x7fb5280d27f0 2024-03-06T07:33:34.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52fc4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5280d1df0 con 0x7fb52815dfe0 2024-03-06T07:33:34.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5280cf830 0x7fb5280cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5280cf830 0x7fb5280cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48630/0 (socket says 172.21.15.77:48630) 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 -- 172.21.15.77:0/962695016 learned_addr learned my addr 172.21.15.77:0/962695016 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d1e9640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5280d27f0 0x7fb5281f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52e1eb640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb52815dfe0 0x7fb5281f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 -- 172.21.15.77:0/962695016 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb52815dfe0 msgr2=0x7fb5281f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb52815dfe0 0x7fb5281f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 -- 172.21.15.77:0/962695016 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5280d27f0 msgr2=0x7fb5281f6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5280d27f0 0x7fb5281f6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 -- 172.21.15.77:0/962695016 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5280cc000 con 0x7fb5280cf830 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.008+0000 7fb52d9ea640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5280cf830 0x7fb5280cb7b0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fb52400e9b0 tx=0x7fb52400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.009+0000 7fb51e7fc640 1 -- 172.21.15.77:0/962695016 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb524017070 con 0x7fb5280cf830 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.009+0000 7fb51e7fc640 1 -- 172.21.15.77:0/962695016 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52400c9a0 con 0x7fb5280cf830 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.009+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5280cc2e0 con 0x7fb5280cf830 2024-03-06T07:33:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.009+0000 7fb51e7fc640 1 -- 172.21.15.77:0/962695016 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52400cd50 con 0x7fb5280cf830 2024-03-06T07:33:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.009+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5280cc8a0 con 0x7fb5280cf830 2024-03-06T07:33:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.010+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52804f330 con 0x7fb5280cf830 2024-03-06T07:33:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.010+0000 7fb51e7fc640 1 -- 172.21.15.77:0/962695016 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb52401a070 con 0x7fb5280cf830 2024-03-06T07:33:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.010+0000 7fb51e7fc640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb50c06b500 0x7fb50c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.011+0000 7fb51e7fc640 1 -- 172.21.15.77:0/962695016 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb52408a3a0 con 0x7fb5280cf830 2024-03-06T07:33:34.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.011+0000 7fb52d1e9640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb50c06b500 0x7fb50c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.011+0000 7fb52d1e9640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb50c06b500 0x7fb50c06d9c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb518036000 tx=0x7fb518008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.019+0000 7fb51e7fc640 1 -- 172.21.15.77:0/962695016 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb524058bb0 con 0x7fb5280cf830 2024-03-06T07:33:34.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.241+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5280747a0 con 0x7fb5280cf830 2024-03-06T07:33:34.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.241+0000 7fb51e7fc640 1 -- 172.21.15.77:0/962695016 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fb52405cc50 con 0x7fb5280cf830 2024-03-06T07:33:34.243 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:34.244 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:34.244 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb50c06b500 msgr2=0x7fb50c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb50c06b500 0x7fb50c06d9c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb518036000 tx=0x7fb518008030 comp rx=0 tx=0).stop 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5280cf830 msgr2=0x7fb5280cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5280cf830 0x7fb5280cb7b0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fb52400e9b0 tx=0x7fb52400ee70 comp rx=0 tx=0).stop 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 shutdown_connections 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb52815dfe0 0x7fb5281f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb5280d27f0 0x7fb5281f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb50c06b500 0x7fb50c06d9c0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 --2- 172.21.15.77:0/962695016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5280cf830 0x7fb5280cb7b0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.243+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 >> 172.21.15.77:0/962695016 conn(0x7fb5280d3cd0 msgr2=0x7fb52806cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.244+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 shutdown_connections 2024-03-06T07:33:34.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:34.244+0000 7fb52fc4e640 1 -- 172.21.15.77:0/962695016 wait complete. 2024-03-06T07:33:34.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:34.458 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:35.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.559+0000 7f2f2311e640 1 Processor -- start 2024-03-06T07:33:35.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.559+0000 7f2f2311e640 1 -- start start 2024-03-06T07:33:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f2311e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f2f1c15ab00 0x7f2f1c0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f2311e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f1c0b48f0 0x7f2f1c0af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f2311e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1c0b4e30 con 0x7f2f1c0afc00 2024-03-06T07:33:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f2311e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f1c0b0060 con 0x7f2f1c0b48f0 2024-03-06T07:33:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f2311e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f1c0b5110 con 0x7f2f1c15ab00 2024-03-06T07:33:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f1bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f1c0b48f0 0x7f2f1c0af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f1bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f1c0b48f0 0x7f2f1c0af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48632/0 (socket says 172.21.15.77:48632) 2024-03-06T07:33:35.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f1bfff640 1 -- 172.21.15.77:0/1531760833 learned_addr learned my addr 172.21.15.77:0/1531760833 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:35.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f20eba640 1 --2- 172.21.15.77:0/1531760833 >> v2:172.21.15.158:3300/0 conn(0x7f2f1c15ab00 0x7f2f1c0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f1bfff640 1 -- 172.21.15.77:0/1531760833 >> v2:172.21.15.158:3300/0 conn(0x7f2f1c15ab00 msgr2=0x7f2f1c0b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f1bfff640 1 --2- 172.21.15.77:0/1531760833 >> v2:172.21.15.158:3300/0 conn(0x7f2f1c15ab00 0x7f2f1c0b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f1bfff640 1 -- 172.21.15.77:0/1531760833 >> v1:172.21.15.158:6789/0 conn(0x7f2f1c0afc00 legacy=0x7f2f1c06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.560+0000 7f2f1bfff640 1 -- 172.21.15.77:0/1531760833 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c15b310 con 0x7f2f1c0b48f0 2024-03-06T07:33:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.561+0000 7f2f1bfff640 1 --2- 172.21.15.77:0/1531760833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0b48f0 0x7f2f1c0af6c0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0732b0 tx=0x7f2f0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8420d7907fcf9458 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.561+0000 7f2f1b7fe640 1 -- 172.21.15.77:0/1531760833 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08035020 con 0x7f2f1c0b48f0 2024-03-06T07:33:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.561+0000 7f2f1b7fe640 1 -- 172.21.15.77:0/1531760833 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08031870 con 0x7f2f1c0b48f0 2024-03-06T07:33:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.561+0000 7f2f1b7fe640 1 -- 172.21.15.77:0/1531760833 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08031c40 con 0x7f2f1c0b48f0 2024-03-06T07:33:35.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.562+0000 7f2f2311e640 1 -- 172.21.15.77:0/1531760833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0b48f0 msgr2=0x7f2f1c0af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.562+0000 7f2f2311e640 1 --2- 172.21.15.77:0/1531760833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0b48f0 0x7f2f1c0af6c0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0732b0 tx=0x7f2f0802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.562+0000 7f2f2311e640 1 -- 172.21.15.77:0/1531760833 shutdown_connections 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.562+0000 7f2f2311e640 1 --2- 172.21.15.77:0/1531760833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0b48f0 0x7f2f1c0af6c0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.562+0000 7f2f2311e640 1 --2- 172.21.15.77:0/1531760833 >> v2:172.21.15.158:3300/0 conn(0x7f2f1c15ab00 0x7f2f1c0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.562+0000 7f2f2311e640 1 -- 172.21.15.77:0/1531760833 >> 172.21.15.77:0/1531760833 conn(0x7f2f1c06c0c0 msgr2=0x7f2f1c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.562+0000 7f2f2311e640 1 -- 172.21.15.77:0/1531760833 shutdown_connections 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.562+0000 7f2f2311e640 1 -- 172.21.15.77:0/1531760833 wait complete. 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.563+0000 7f2f2311e640 1 Processor -- start 2024-03-06T07:33:35.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.563+0000 7f2f2311e640 1 -- start start 2024-03-06T07:33:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f2311e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0afc00 0x7f2f1c1442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f2311e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f1c0b48f0 0x7f2f1c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f2311e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f1c15ab00 0x7f2f1c143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f2311e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c071ca0 con 0x7f2f1c0afc00 2024-03-06T07:33:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f2311e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c071b20 con 0x7f2f1c0b48f0 2024-03-06T07:33:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f2311e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c15aee0 con 0x7f2f1c15ab00 2024-03-06T07:33:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0afc00 0x7f2f1c1442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0afc00 0x7f2f1c1442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48638/0 (socket says 172.21.15.77:48638) 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 -- 172.21.15.77:0/2504444775 learned_addr learned my addr 172.21.15.77:0/2504444775 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f216bb640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f1c15ab00 0x7f2f1c143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f1bfff640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f1c0b48f0 0x7f2f1c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 -- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f1c15ab00 msgr2=0x7f2f1c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f1c15ab00 0x7f2f1c143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 -- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f1c0b48f0 msgr2=0x7f2f1c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f1c0b48f0 0x7f2f1c0b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 -- 172.21.15.77:0/2504444775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c14c0a0 con 0x7f2f1c0afc00 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f216bb640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f1c15ab00 0x7f2f1c143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f1bfff640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f1c0b48f0 0x7f2f1c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f20eba640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0afc00 0x7f2f1c1442b0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000d620 tx=0x7f2f1000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f197fa640 1 -- 172.21.15.77:0/2504444775 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1000b620 con 0x7f2f1c0afc00 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f197fa640 1 -- 172.21.15.77:0/2504444775 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f10019070 con 0x7f2f1c0afc00 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1c14c380 con 0x7f2f1c0afc00 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.564+0000 7f2f197fa640 1 -- 172.21.15.77:0/2504444775 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1000ba00 con 0x7f2f1c0afc00 2024-03-06T07:33:35.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.565+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1c14c940 con 0x7f2f1c0afc00 2024-03-06T07:33:35.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.566+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1c04f330 con 0x7f2f1c0afc00 2024-03-06T07:33:35.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.566+0000 7f2f197fa640 1 -- 172.21.15.77:0/2504444775 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f1000bba0 con 0x7f2f1c0afc00 2024-03-06T07:33:35.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.566+0000 7f2f197fa640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f0006b4d0 0x7f2f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.567+0000 7f2f197fa640 1 -- 172.21.15.77:0/2504444775 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2f1008a640 con 0x7f2f1c0afc00 2024-03-06T07:33:35.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.567+0000 7f2f1bfff640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f0006b4d0 0x7f2f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.567+0000 7f2f1bfff640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f0006b4d0 0x7f2f0006d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2f08036000 tx=0x7f2f08008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.571+0000 7f2f197fa640 1 -- 172.21.15.77:0/2504444775 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f100594a0 con 0x7f2f1c0afc00 2024-03-06T07:33:35.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.787+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f1c06c6f0 con 0x7f2f1c0afc00 2024-03-06T07:33:35.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.788+0000 7f2f197fa640 1 -- 172.21.15.77:0/2504444775 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f2f10058da0 con 0x7f2f1c0afc00 2024-03-06T07:33:35.790 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:35.790 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:35.790 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.789+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f0006b4d0 msgr2=0x7f2f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.789+0000 7f2f2311e640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f0006b4d0 0x7f2f0006d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2f08036000 tx=0x7f2f08008030 comp rx=0 tx=0).stop 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.789+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0afc00 msgr2=0x7f2f1c1442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.789+0000 7f2f2311e640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0afc00 0x7f2f1c1442b0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000d620 tx=0x7f2f1000dae0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.790+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 shutdown_connections 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.790+0000 7f2f2311e640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f2f1c15ab00 0x7f2f1c143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.790+0000 7f2f2311e640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2f1c0b48f0 0x7f2f1c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.790+0000 7f2f2311e640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f2f0006b4d0 0x7f2f0006d990 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.790+0000 7f2f2311e640 1 --2- 172.21.15.77:0/2504444775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f1c0afc00 0x7f2f1c1442b0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.790+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 >> 172.21.15.77:0/2504444775 conn(0x7f2f1c06c0c0 msgr2=0x7f2f1c06f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.790+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 shutdown_connections 2024-03-06T07:33:35.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:35.790+0000 7f2f2311e640 1 -- 172.21.15.77:0/2504444775 wait complete. 2024-03-06T07:33:36.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:36.004 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:37.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:37.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.104+0000 7fbfe7df7640 1 Processor -- start 2024-03-06T07:33:37.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.105+0000 7fbfe7df7640 1 -- start start 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.105+0000 7fbfe7df7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbfe015e090 0x7fbfe00d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.105+0000 7fbfe7df7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d0650 0x7fbfe00d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.105+0000 7fbfe7df7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe015e470 con 0x7fbfe00d2920 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.105+0000 7fbfe7df7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe00d18f0 con 0x7fbfe015e090 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.105+0000 7fbfe7df7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00b3cf0 con 0x7fbfe00d0650 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.106+0000 7fbfe5b93640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbfe015e090 0x7fbfe00d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.106+0000 7fbfe5392640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbfe00d2920 0x7fbfe00b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:35698/0 (socket says 172.21.15.77:35698) 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.106+0000 7fbfe5392640 1 -- 172.21.15.77:0/3570920372 learned_addr learned my addr 172.21.15.77:0/3570920372 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.106+0000 7fbfe5b93640 1 -- 172.21.15.77:0/3570920372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d0650 msgr2=0x7fbfe00d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.106+0000 7fbfe5b93640 1 --2- 172.21.15.77:0/3570920372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d0650 0x7fbfe00d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.106+0000 7fbfe5b93640 1 -- 172.21.15.77:0/3570920372 >> v1:172.21.15.77:6789/0 conn(0x7fbfe00d2920 legacy=0x7fbfe00b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.106+0000 7fbfe5b93640 1 -- 172.21.15.77:0/3570920372 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe00d1b90 con 0x7fbfe015e090 2024-03-06T07:33:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe5b93640 1 --2- 172.21.15.77:0/3570920372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00d47a0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0097a0 tx=0x7fbfdc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06cec67bc79152b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe4b91640 1 -- 172.21.15.77:0/3570920372 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc035020 con 0x7fbfe015e090 2024-03-06T07:33:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe4b91640 1 -- 172.21.15.77:0/3570920372 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc031830 con 0x7fbfe015e090 2024-03-06T07:33:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe4b91640 1 -- 172.21.15.77:0/3570920372 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc031c00 con 0x7fbfe015e090 2024-03-06T07:33:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe7df7640 1 -- 172.21.15.77:0/3570920372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 msgr2=0x7fbfe00d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/3570920372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00d47a0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0097a0 tx=0x7fbfdc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe7df7640 1 -- 172.21.15.77:0/3570920372 shutdown_connections 2024-03-06T07:33:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/3570920372 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d0650 0x7fbfe00d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/3570920372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00d47a0 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe7df7640 1 -- 172.21.15.77:0/3570920372 >> 172.21.15.77:0/3570920372 conn(0x7fbfe00d3c80 msgr2=0x7fbfe00d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe7df7640 1 -- 172.21.15.77:0/3570920372 shutdown_connections 2024-03-06T07:33:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.107+0000 7fbfe7df7640 1 -- 172.21.15.77:0/3570920372 wait complete. 2024-03-06T07:33:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.108+0000 7fbfe7df7640 1 Processor -- start 2024-03-06T07:33:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe7df7640 1 -- start start 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe7df7640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfe00d0650 0x7fbfe01e8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe7df7640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d2920 0x7fbfe015cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe7df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe7df7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00b2df0 con 0x7fbfe015e090 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe7df7640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00cee30 con 0x7fbfe00d0650 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe7df7640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00ce800 con 0x7fbfe00d2920 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe6394640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe6394640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00b5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48658/0 (socket says 172.21.15.77:48658) 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.109+0000 7fbfe6394640 1 -- 172.21.15.77:0/2042368494 learned_addr learned my addr 172.21.15.77:0/2042368494 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe5392640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d2920 0x7fbfe015cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe6394640 1 -- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d2920 msgr2=0x7fbfe015cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe6394640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d2920 0x7fbfe015cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe5b93640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfe00d0650 0x7fbfe01e8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe6394640 1 -- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfe00d0650 msgr2=0x7fbfe01e8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe6394640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfe00d0650 0x7fbfe01e8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe6394640 1 -- 172.21.15.77:0/2042368494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe01e7960 con 0x7fbfe015e090 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe5b93640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfe00d0650 0x7fbfe01e8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe6394640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00b5b40 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800b970 tx=0x7fbfd800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfd67fc640 1 -- 172.21.15.77:0/2042368494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8019070 con 0x7fbfe015e090 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfd67fc640 1 -- 172.21.15.77:0/2042368494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd800cb60 con 0x7fbfe015e090 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfd67fc640 1 -- 172.21.15.77:0/2042368494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8014440 con 0x7fbfe015e090 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe01e7c40 con 0x7fbfe015e090 2024-03-06T07:33:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.110+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe01e8200 con 0x7fbfe015e090 2024-03-06T07:33:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.111+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe004f330 con 0x7fbfe015e090 2024-03-06T07:33:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.112+0000 7fbfd67fc640 1 -- 172.21.15.77:0/2042368494 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfd801c070 con 0x7fbfe015e090 2024-03-06T07:33:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.112+0000 7fbfd67fc640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfc406b500 0x7fbfc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.112+0000 7fbfe5b93640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfc406b500 0x7fbfc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.112+0000 7fbfd67fc640 1 -- 172.21.15.77:0/2042368494 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbfd8089ec0 con 0x7fbfe015e090 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.113+0000 7fbfe5b93640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfc406b500 0x7fbfc406d9c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc036000 tx=0x7fbfdc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.116+0000 7fbfd67fc640 1 -- 172.21.15.77:0/2042368494 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfd8058db0 con 0x7fbfe015e090 2024-03-06T07:33:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.330+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfe00d2070 con 0x7fbfe015e090 2024-03-06T07:33:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.331+0000 7fbfd67fc640 1 -- 172.21.15.77:0/2042368494 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fbfd80586b0 con 0x7fbfe015e090 2024-03-06T07:33:37.333 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:37.333 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:37.333 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfc406b500 msgr2=0x7fbfc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfc406b500 0x7fbfc406d9c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc036000 tx=0x7fbfdc008030 comp rx=0 tx=0).stop 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 msgr2=0x7fbfe00b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00b5b40 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800b970 tx=0x7fbfd800be30 comp rx=0 tx=0).stop 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 shutdown_connections 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfc406b500 0x7fbfc406d9c0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfe015e090 0x7fbfe00b5b40 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbfe00d2920 0x7fbfe015cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 --2- 172.21.15.77:0/2042368494 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbfe00d0650 0x7fbfe01e8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 >> 172.21.15.77:0/2042368494 conn(0x7fbfe00d3c80 msgr2=0x7fbfe00ca040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 shutdown_connections 2024-03-06T07:33:37.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:37.333+0000 7fbfe7df7640 1 -- 172.21.15.77:0/2042368494 wait complete. 2024-03-06T07:33:37.547 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:37.547 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:38.548 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:38.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.652+0000 7f71fe55c640 1 Processor -- start 2024-03-06T07:33:38.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.652+0000 7f71fe55c640 1 -- start start 2024-03-06T07:33:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fe55c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f71f80d22b0 0x7f71f80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fe55c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71f80b41f0 0x7f71f80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fe55c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f80b3430 con 0x7f71f815dfe0 2024-03-06T07:33:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fe55c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f80d2c50 con 0x7f71f80b41f0 2024-03-06T07:33:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fe55c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f80b31e0 con 0x7f71f80d22b0 2024-03-06T07:33:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fcaf9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71f80b41f0 0x7f71f80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fcaf9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f71f80b41f0 0x7f71f80b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56974/0 (socket says 172.21.15.77:56974) 2024-03-06T07:33:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fcaf9640 1 -- 172.21.15.77:0/3977473871 learned_addr learned my addr 172.21.15.77:0/3977473871 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71f77fe640 1 --2- 172.21.15.77:0/3977473871 >> v2:172.21.15.158:3300/0 conn(0x7f71f80d22b0 0x7f71f80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fcaf9640 1 -- 172.21.15.77:0/3977473871 >> v2:172.21.15.158:3300/0 conn(0x7f71f80d22b0 msgr2=0x7f71f80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fcaf9640 1 --2- 172.21.15.77:0/3977473871 >> v2:172.21.15.158:3300/0 conn(0x7f71f80d22b0 0x7f71f80d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fcaf9640 1 -- 172.21.15.77:0/3977473871 >> v1:172.21.15.158:6789/0 conn(0x7f71f815dfe0 legacy=0x7f71f80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.653+0000 7f71fcaf9640 1 -- 172.21.15.77:0/3977473871 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f80b3710 con 0x7f71f80b41f0 2024-03-06T07:33:38.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.654+0000 7f71fcaf9640 1 --2- 172.21.15.77:0/3977473871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80b41f0 0x7f71f80b4680 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009e50 tx=0x7f71f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d2801f266ca3d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.654+0000 7f71f6ffd640 1 -- 172.21.15.77:0/3977473871 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f002ca80 con 0x7f71f80b41f0 2024-03-06T07:33:38.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.654+0000 7f71f6ffd640 1 -- 172.21.15.77:0/3977473871 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f002cc20 con 0x7f71f80b41f0 2024-03-06T07:33:38.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.654+0000 7f71f6ffd640 1 -- 172.21.15.77:0/3977473871 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f00343f0 con 0x7f71f80b41f0 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.654+0000 7f71fe55c640 1 -- 172.21.15.77:0/3977473871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80b41f0 msgr2=0x7f71f80b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.654+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3977473871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80b41f0 0x7f71f80b4680 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009e50 tx=0x7f71f002bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.655+0000 7f71fe55c640 1 -- 172.21.15.77:0/3977473871 shutdown_connections 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.655+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3977473871 >> v2:172.21.15.158:3300/0 conn(0x7f71f80d22b0 0x7f71f80d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.655+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3977473871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80b41f0 0x7f71f80b4680 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.655+0000 7f71fe55c640 1 -- 172.21.15.77:0/3977473871 >> 172.21.15.77:0/3977473871 conn(0x7f71f80d3cd0 msgr2=0x7f71f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.655+0000 7f71fe55c640 1 -- 172.21.15.77:0/3977473871 shutdown_connections 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.655+0000 7f71fe55c640 1 -- 172.21.15.77:0/3977473871 wait complete. 2024-03-06T07:33:38.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.656+0000 7f71fe55c640 1 Processor -- start 2024-03-06T07:33:38.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.657+0000 7f71fe55c640 1 -- start start 2024-03-06T07:33:38.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71fe55c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71f80b41f0 0x7f71f80b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71fe55c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80d22b0 0x7f71f80b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71fe55c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f815dfe0 0x7f71f80b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71fe55c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80cfa00 con 0x7f71f80d22b0 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71fe55c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80b3d60 con 0x7f71f815dfe0 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71fe55c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80d1c70 con 0x7f71f80b41f0 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80d22b0 0x7f71f80b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80d22b0 0x7f71f80b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56978/0 (socket says 172.21.15.77:56978) 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 -- 172.21.15.77:0/3714278859 learned_addr learned my addr 172.21.15.77:0/3714278859 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f7fff640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71f80b41f0 0x7f71f80b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 -- 172.21.15.77:0/3714278859 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71f80b41f0 msgr2=0x7f71f80b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71fcaf9640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f815dfe0 0x7f71f80b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71f80b41f0 0x7f71f80b8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 -- 172.21.15.77:0/3714278859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f815dfe0 msgr2=0x7f71f80b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f815dfe0 0x7f71f80b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 -- 172.21.15.77:0/3714278859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f80b9be0 con 0x7f71f80d22b0 2024-03-06T07:33:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f77fe640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80d22b0 0x7f71f80b68a0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00d830 tx=0x7f71ec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71f4ff9640 1 -- 172.21.15.77:0/3714278859 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec00b6c0 con 0x7f71f80d22b0 2024-03-06T07:33:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.658+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f80b9060 con 0x7f71f80d22b0 2024-03-06T07:33:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.659+0000 7f71f4ff9640 1 -- 172.21.15.77:0/3714278859 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec019070 con 0x7f71f80d22b0 2024-03-06T07:33:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.659+0000 7f71f4ff9640 1 -- 172.21.15.77:0/3714278859 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec00ba80 con 0x7f71f80d22b0 2024-03-06T07:33:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.659+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f80b9620 con 0x7f71f80d22b0 2024-03-06T07:33:38.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.660+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f804f330 con 0x7f71f80d22b0 2024-03-06T07:33:38.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.660+0000 7f71f4ff9640 1 -- 172.21.15.77:0/3714278859 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71ec01e460 con 0x7f71f80d22b0 2024-03-06T07:33:38.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.661+0000 7f71f4ff9640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71e406b500 0x7f71e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.661+0000 7f71f7fff640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71e406b500 0x7f71e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.661+0000 7f71f4ff9640 1 -- 172.21.15.77:0/3714278859 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f71ec08a920 con 0x7f71f80d22b0 2024-03-06T07:33:38.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.661+0000 7f71f7fff640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71e406b500 0x7f71e406d9c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f71e800a750 tx=0x7f71e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.665+0000 7f71f4ff9640 1 -- 172.21.15.77:0/3714278859 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71ec059700 con 0x7f71f80d22b0 2024-03-06T07:33:38.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.986+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71f80cea80 con 0x7f71f80d22b0 2024-03-06T07:33:38.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.987+0000 7f71f4ff9640 1 -- 172.21.15.77:0/3714278859 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f71ec059000 con 0x7f71f80d22b0 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:38.989 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71e406b500 msgr2=0x7f71e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71e406b500 0x7f71e406d9c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f71e800a750 tx=0x7f71e8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80d22b0 msgr2=0x7f71f80b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80d22b0 0x7f71f80b68a0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00d830 tx=0x7f71ec00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 shutdown_connections 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f71e406b500 0x7f71e406d9c0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71f80d22b0 0x7f71f80b68a0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f71f815dfe0 0x7f71f80b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 --2- 172.21.15.77:0/3714278859 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f71f80b41f0 0x7f71f80b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 >> 172.21.15.77:0/3714278859 conn(0x7f71f80d3cd0 msgr2=0x7f71f80d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 shutdown_connections 2024-03-06T07:33:38.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:38.989+0000 7f71fe55c640 1 -- 172.21.15.77:0/3714278859 wait complete. 2024-03-06T07:33:39.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:39.203 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:40.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:40.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.304+0000 7fb65764e640 1 Processor -- start 2024-03-06T07:33:40.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.304+0000 7fb65764e640 1 -- start start 2024-03-06T07:33:40.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb65764e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb6500d22b0 0x7fb6500d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb65764e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6500b41f0 0x7fb6500b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb65764e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6500b3430 con 0x7fb65015dfe0 2024-03-06T07:33:40.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb65764e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6500d2c50 con 0x7fb6500d22b0 2024-03-06T07:33:40.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb65764e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500b31e0 con 0x7fb6500b41f0 2024-03-06T07:33:40.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb654be9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb6500d22b0 0x7fb6500d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb654be9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb6500d22b0 0x7fb6500d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56994/0 (socket says 172.21.15.77:56994) 2024-03-06T07:33:40.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb654be9640 1 -- 172.21.15.77:0/461044236 learned_addr learned my addr 172.21.15.77:0/461044236 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:40.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb654be9640 1 -- 172.21.15.77:0/461044236 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6500b41f0 msgr2=0x7fb6500b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb654be9640 1 --2- 172.21.15.77:0/461044236 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6500b41f0 0x7fb6500b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb654be9640 1 -- 172.21.15.77:0/461044236 >> v1:172.21.15.158:6789/0 conn(0x7fb65015dfe0 legacy=0x7fb6500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.305+0000 7fb654be9640 1 -- 172.21.15.77:0/461044236 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500b3710 con 0x7fb6500d22b0 2024-03-06T07:33:40.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.306+0000 7fb654be9640 1 --2- 172.21.15.77:0/461044236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6500d22b0 0x7fb6500d2710 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fb640009870 tx=0x7fb64002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d69e95793079664 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.306+0000 7fb647fff640 1 -- 172.21.15.77:0/461044236 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640035020 con 0x7fb6500d22b0 2024-03-06T07:33:40.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.306+0000 7fb647fff640 1 -- 172.21.15.77:0/461044236 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb640031640 con 0x7fb6500d22b0 2024-03-06T07:33:40.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.306+0000 7fb647fff640 1 -- 172.21.15.77:0/461044236 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64002d040 con 0x7fb6500d22b0 2024-03-06T07:33:40.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.307+0000 7fb65764e640 1 -- 172.21.15.77:0/461044236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6500d22b0 msgr2=0x7fb6500d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.307+0000 7fb65764e640 1 --2- 172.21.15.77:0/461044236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6500d22b0 0x7fb6500d2710 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fb640009870 tx=0x7fb64002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.307+0000 7fb65764e640 1 -- 172.21.15.77:0/461044236 shutdown_connections 2024-03-06T07:33:40.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.307+0000 7fb65764e640 1 --2- 172.21.15.77:0/461044236 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6500b41f0 0x7fb6500b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.307+0000 7fb65764e640 1 --2- 172.21.15.77:0/461044236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6500d22b0 0x7fb6500d2710 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.307+0000 7fb65764e640 1 -- 172.21.15.77:0/461044236 >> 172.21.15.77:0/461044236 conn(0x7fb6500d3cd0 msgr2=0x7fb6500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.307+0000 7fb65764e640 1 -- 172.21.15.77:0/461044236 shutdown_connections 2024-03-06T07:33:40.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.307+0000 7fb65764e640 1 -- 172.21.15.77:0/461044236 wait complete. 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.308+0000 7fb65764e640 1 Processor -- start 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.308+0000 7fb65764e640 1 -- start start 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb65764e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6500b41f0 0x7fb6500cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb65764e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6500d22b0 0x7fb6500cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb65764e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65015dfe0 0x7fb65014eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb65764e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500cfa00 con 0x7fb65015dfe0 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb65764e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500b3d60 con 0x7fb6500d22b0 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb65764e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500d1c70 con 0x7fb6500b41f0 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb655beb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65015dfe0 0x7fb65014eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb655beb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65015dfe0 0x7fb65014eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57008/0 (socket says 172.21.15.77:57008) 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb655beb640 1 -- 172.21.15.77:0/4198998414 learned_addr learned my addr 172.21.15.77:0/4198998414 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb655beb640 1 -- 172.21.15.77:0/4198998414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6500b41f0 msgr2=0x7fb6500cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb655beb640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6500b41f0 0x7fb6500cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb655beb640 1 -- 172.21.15.77:0/4198998414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6500d22b0 msgr2=0x7fb6500cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.309+0000 7fb654be9640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6500d22b0 0x7fb6500cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.310+0000 7fb655beb640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6500d22b0 0x7fb6500cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.310+0000 7fb655beb640 1 -- 172.21.15.77:0/4198998414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500cc910 con 0x7fb65015dfe0 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.310+0000 7fb655beb640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65015dfe0 0x7fb65014eab0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fb648009fd0 tx=0x7fb64800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.310+0000 7fb645ffb640 1 -- 172.21.15.77:0/4198998414 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648011830 con 0x7fb65015dfe0 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.310+0000 7fb645ffb640 1 -- 172.21.15.77:0/4198998414 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb648016070 con 0x7fb65015dfe0 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.310+0000 7fb645ffb640 1 -- 172.21.15.77:0/4198998414 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648011bf0 con 0x7fb65015dfe0 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.310+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6500ccbf0 con 0x7fb65015dfe0 2024-03-06T07:33:40.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.310+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6500cd1b0 con 0x7fb65015dfe0 2024-03-06T07:33:40.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.311+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6500cd6a0 con 0x7fb65015dfe0 2024-03-06T07:33:40.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.311+0000 7fb645ffb640 1 -- 172.21.15.77:0/4198998414 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb64801b460 con 0x7fb65015dfe0 2024-03-06T07:33:40.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.312+0000 7fb645ffb640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb63406b4d0 0x7fb63406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.312+0000 7fb645ffb640 1 -- 172.21.15.77:0/4198998414 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb648089ec0 con 0x7fb65015dfe0 2024-03-06T07:33:40.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.312+0000 7fb6553ea640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb63406b4d0 0x7fb63406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.312+0000 7fb6553ea640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb63406b4d0 0x7fb63406d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00abb0 tx=0x7fb64c009250 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.320+0000 7fb645ffb640 1 -- 172.21.15.77:0/4198998414 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb648058db0 con 0x7fb65015dfe0 2024-03-06T07:33:40.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.528+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb65014eff0 con 0x7fb65015dfe0 2024-03-06T07:33:40.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.528+0000 7fb645ffb640 1 -- 172.21.15.77:0/4198998414 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fb6480586b0 con 0x7fb65015dfe0 2024-03-06T07:33:40.531 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:40.531 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:40.532 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:40.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb63406b4d0 msgr2=0x7fb63406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb63406b4d0 0x7fb63406d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00abb0 tx=0x7fb64c009250 comp rx=0 tx=0).stop 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65015dfe0 msgr2=0x7fb65014eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65015dfe0 0x7fb65014eab0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fb648009fd0 tx=0x7fb64800b8e0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 shutdown_connections 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb63406b4d0 0x7fb63406d990 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65015dfe0 0x7fb65014eab0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb6500d22b0 0x7fb6500cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 --2- 172.21.15.77:0/4198998414 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb6500b41f0 0x7fb6500cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 >> 172.21.15.77:0/4198998414 conn(0x7fb6500d3cd0 msgr2=0x7fb65006caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 shutdown_connections 2024-03-06T07:33:40.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:40.530+0000 7fb65764e640 1 -- 172.21.15.77:0/4198998414 wait complete. 2024-03-06T07:33:40.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:40.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:41.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:41.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.845+0000 7f7a45d79640 1 Processor -- start 2024-03-06T07:33:41.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.845+0000 7f7a45d79640 1 -- start start 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.846+0000 7f7a45d79640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4015dfe0 0x7f7a400cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.846+0000 7f7a45d79640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7a400cf830 0x7f7a400d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.846+0000 7f7a45d79640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a400b3430 con 0x7f7a400d27f0 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.846+0000 7f7a45d79640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a400d2c50 con 0x7f7a400cf830 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.846+0000 7f7a45d79640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400b31e0 con 0x7f7a4015dfe0 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7a400d27f0 0x7f7a400d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52942/0 (socket says 172.21.15.77:52942) 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7a400cf830 0x7f7a400d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3ffff640 1 -- 172.21.15.77:0/861962310 learned_addr learned my addr 172.21.15.77:0/861962310 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3f7fe640 1 --2- 172.21.15.77:0/861962310 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4015dfe0 0x7f7a400cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3effd640 1 -- 172.21.15.77:0/861962310 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4015dfe0 msgr2=0x7f7a400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3effd640 1 --2- 172.21.15.77:0/861962310 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4015dfe0 0x7f7a400cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3effd640 1 -- 172.21.15.77:0/861962310 >> v1:172.21.15.77:6789/0 conn(0x7f7a400d27f0 legacy=0x7f7a400d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3effd640 1 -- 172.21.15.77:0/861962310 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400b3710 con 0x7f7a400cf830 2024-03-06T07:33:41.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.847+0000 7f7a3e7fc640 1 -- 172.21.15.77:0/861962310 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241559633 0 0) 0x7f7a400b3430 con 0x7f7a400d27f0 2024-03-06T07:33:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a3effd640 1 --2- 172.21.15.77:0/861962310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400d22b0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f7a340097a0 tx=0x7f7a3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9278c2fc22b1b2a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a3e7fc640 1 -- 172.21.15.77:0/861962310 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34035020 con 0x7f7a400cf830 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a3e7fc640 1 -- 172.21.15.77:0/861962310 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a34031830 con 0x7f7a400cf830 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a3e7fc640 1 -- 172.21.15.77:0/861962310 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34031c00 con 0x7f7a400cf830 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a45d79640 1 -- 172.21.15.77:0/861962310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 msgr2=0x7f7a400d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a45d79640 1 --2- 172.21.15.77:0/861962310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400d22b0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f7a340097a0 tx=0x7f7a3402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a45d79640 1 -- 172.21.15.77:0/861962310 shutdown_connections 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a45d79640 1 --2- 172.21.15.77:0/861962310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400d22b0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a45d79640 1 --2- 172.21.15.77:0/861962310 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a4015dfe0 0x7f7a400cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.848+0000 7f7a45d79640 1 -- 172.21.15.77:0/861962310 >> 172.21.15.77:0/861962310 conn(0x7f7a400d3cd0 msgr2=0x7f7a400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.849+0000 7f7a45d79640 1 -- 172.21.15.77:0/861962310 shutdown_connections 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.849+0000 7f7a45d79640 1 -- 172.21.15.77:0/861962310 wait complete. 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.850+0000 7f7a45d79640 1 Processor -- start 2024-03-06T07:33:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.850+0000 7f7a45d79640 1 -- start start 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.850+0000 7f7a45d79640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.850+0000 7f7a45d79640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a400d27f0 0x7f7a4014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.850+0000 7f7a45d79640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4015dfe0 0x7f7a400b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a45d79640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400d1c70 con 0x7f7a400cf830 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a45d79640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400b3d60 con 0x7f7a4015dfe0 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a45d79640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400d1df0 con 0x7f7a400d27f0 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57040/0 (socket says 172.21.15.77:57040) 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 -- 172.21.15.77:0/835479808 learned_addr learned my addr 172.21.15.77:0/835479808 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3effd640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a400d27f0 0x7f7a4014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3ffff640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4015dfe0 0x7f7a400b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 -- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a400d27f0 msgr2=0x7f7a4014f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a400d27f0 0x7f7a4014f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 -- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4015dfe0 msgr2=0x7f7a400b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4015dfe0 0x7f7a400b8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 -- 172.21.15.77:0/835479808 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4015f080 con 0x7f7a400cf830 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3ffff640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4015dfe0 0x7f7a400b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a3f7fe640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400b7e00 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000d620 tx=0x7f7a3000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a27fff640 1 -- 172.21.15.77:0/835479808 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3000b620 con 0x7f7a400cf830 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a27fff640 1 -- 172.21.15.77:0/835479808 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30019070 con 0x7f7a400cf830 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a27fff640 1 -- 172.21.15.77:0/835479808 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3000ba00 con 0x7f7a400cf830 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4015f360 con 0x7f7a400cf830 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.851+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4015f920 con 0x7f7a400cf830 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.852+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4004f330 con 0x7f7a400cf830 2024-03-06T07:33:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.853+0000 7f7a27fff640 1 -- 172.21.15.77:0/835479808 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a3000bba0 con 0x7f7a400cf830 2024-03-06T07:33:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.854+0000 7f7a27fff640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.854+0000 7f7a3effd640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.854+0000 7f7a27fff640 1 -- 172.21.15.77:0/835479808 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7a3008a5e0 con 0x7f7a400cf830 2024-03-06T07:33:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.854+0000 7f7a3effd640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7a34036000 tx=0x7f7a34008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:41.858+0000 7f7a27fff640 1 -- 172.21.15.77:0/835479808 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a30059440 con 0x7f7a400cf830 2024-03-06T07:33:42.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.070+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a400b09c0 con 0x7f7a400cf830 2024-03-06T07:33:42.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.071+0000 7f7a27fff640 1 -- 172.21.15.77:0/835479808 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f7a30058d40 con 0x7f7a400cf830 2024-03-06T07:33:42.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:42.074 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:42.074 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 msgr2=0x7f7a2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f7a34036000 tx=0x7f7a34008030 comp rx=0 tx=0).stop 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 msgr2=0x7f7a400b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400b7e00 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000d620 tx=0x7f7a3000dae0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 shutdown_connections 2024-03-06T07:33:42.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a4015dfe0 0x7f7a400b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f7a400d27f0 0x7f7a4014f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f7a2c06b4d0 0x7f7a2c06d990 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 --2- 172.21.15.77:0/835479808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a400cf830 0x7f7a400b7e00 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 >> 172.21.15.77:0/835479808 conn(0x7f7a400d3cd0 msgr2=0x7f7a400d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 shutdown_connections 2024-03-06T07:33:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:42.073+0000 7f7a45d79640 1 -- 172.21.15.77:0/835479808 wait complete. 2024-03-06T07:33:42.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:42.288 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:43.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:43.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.397+0000 7fcfac823640 1 Processor -- start 2024-03-06T07:33:43.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.397+0000 7fcfac823640 1 -- start start 2024-03-06T07:33:43.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcfac823640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fcfa815dfe0 0x7fcfa80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcfac823640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa80b3430 con 0x7fcfa80cf830 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcfac823640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa80b3710 con 0x7fcfa80d29c0 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcfac823640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa80b31e0 con 0x7fcfa815dfe0 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcf9ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcfa80cf830 0x7fcfa80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52958/0 (socket says 172.21.15.77:52958) 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcf9ffff640 1 -- 172.21.15.77:0/2957510801 learned_addr learned my addr 172.21.15.77:0/2957510801 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcfa659c640 1 --2- 172.21.15.77:0/2957510801 >> v2:172.21.15.158:3300/0 conn(0x7fcfa815dfe0 0x7fcfa80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcfa5d9b640 1 -- 172.21.15.77:0/2957510801 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410072539 0 0) 0x7fcfa80b3430 con 0x7fcfa80cf830 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.398+0000 7fcfa5d9b640 1 -- 172.21.15.77:0/2957510801 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf94003520 con 0x7fcfa80cf830 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.399+0000 7fcfa659c640 1 -- 172.21.15.77:0/2957510801 >> v1:172.21.15.158:6789/0 conn(0x7fcfa80d29c0 legacy=0x7fcfa80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.399+0000 7fcfa659c640 1 -- 172.21.15.77:0/2957510801 >> v1:172.21.15.77:6789/0 conn(0x7fcfa80cf830 legacy=0x7fcfa80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.399+0000 7fcfa659c640 1 -- 172.21.15.77:0/2957510801 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa80b39f0 con 0x7fcfa815dfe0 2024-03-06T07:33:43.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.399+0000 7fcfa659c640 1 --2- 172.21.15.77:0/2957510801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 0x7fcfa80cf2f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80b3360 tx=0x7fcf9802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=684f7476cf9200fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfa5d9b640 1 -- 172.21.15.77:0/2957510801 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98034070 con 0x7fcfa815dfe0 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfa5d9b640 1 -- 172.21.15.77:0/2957510801 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf98030610 con 0x7fcfa815dfe0 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfa5d9b640 1 -- 172.21.15.77:0/2957510801 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980309c0 con 0x7fcfa815dfe0 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfac823640 1 -- 172.21.15.77:0/2957510801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 msgr2=0x7fcfa80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfac823640 1 --2- 172.21.15.77:0/2957510801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 0x7fcfa80cf2f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80b3360 tx=0x7fcf9802b960 comp rx=0 tx=0).stop 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfac823640 1 -- 172.21.15.77:0/2957510801 shutdown_connections 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfac823640 1 --2- 172.21.15.77:0/2957510801 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 0x7fcfa80cf2f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfac823640 1 -- 172.21.15.77:0/2957510801 >> 172.21.15.77:0/2957510801 conn(0x7fcfa80d3cd0 msgr2=0x7fcfa80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfac823640 1 -- 172.21.15.77:0/2957510801 shutdown_connections 2024-03-06T07:33:43.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.400+0000 7fcfac823640 1 -- 172.21.15.77:0/2957510801 wait complete. 2024-03-06T07:33:43.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.401+0000 7fcfac823640 1 Processor -- start 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.401+0000 7fcfac823640 1 -- start start 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfac823640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa80cf830 0x7fcfa814bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfac823640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa80d29c0 0x7fcfa80b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfac823640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 0x7fcfa80b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfac823640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa815e3c0 con 0x7fcfa80cf830 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfac823640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80b3d60 con 0x7fcfa815dfe0 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfac823640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa815e540 con 0x7fcfa80d29c0 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfa659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa80cf830 0x7fcfa814bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfa659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa80cf830 0x7fcfa814bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57042/0 (socket says 172.21.15.77:57042) 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfa659c640 1 -- 172.21.15.77:0/3045303259 learned_addr learned my addr 172.21.15.77:0/3045303259 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcfa6d9d640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 0x7fcfa80b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.402+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa80d29c0 0x7fcfa80b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfa659c640 1 -- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa80d29c0 msgr2=0x7fcfa80b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfa659c640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa80d29c0 0x7fcfa80b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfa659c640 1 -- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 msgr2=0x7fcfa80b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfa659c640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 0x7fcfa80b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfa659c640 1 -- 172.21.15.77:0/3045303259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa80b40f0 con 0x7fcfa80cf830 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfa6d9d640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 0x7fcfa80b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfa659c640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa80cf830 0x7fcfa814bb30 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fcf9802bf70 tx=0x7fcf980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcf9effd640 1 -- 172.21.15.77:0/3045303259 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98047070 con 0x7fcfa80cf830 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcf9effd640 1 -- 172.21.15.77:0/3045303259 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf98030410 con 0x7fcfa80cf830 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcf9effd640 1 -- 172.21.15.77:0/3045303259 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98034040 con 0x7fcfa80cf830 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa814e9f0 con 0x7fcfa80cf830 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.403+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa80b4690 con 0x7fcfa80cf830 2024-03-06T07:33:43.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.404+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa804f330 con 0x7fcfa80cf830 2024-03-06T07:33:43.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.405+0000 7fcf9effd640 1 -- 172.21.15.77:0/3045303259 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf9804a070 con 0x7fcfa80cf830 2024-03-06T07:33:43.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.405+0000 7fcf9effd640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf8806b4d0 0x7fcf8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.405+0000 7fcf9effd640 1 -- 172.21.15.77:0/3045303259 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcf980abb80 con 0x7fcfa80cf830 2024-03-06T07:33:43.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.405+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf8806b4d0 0x7fcf8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.406+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf8806b4d0 0x7fcf8806d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80d1b50 tx=0x7fcf90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.412+0000 7fcf9effd640 1 -- 172.21.15.77:0/3045303259 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf980305e0 con 0x7fcfa80cf830 2024-03-06T07:33:43.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.627+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfa814c070 con 0x7fcfa80cf830 2024-03-06T07:33:43.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.627+0000 7fcf9effd640 1 -- 172.21.15.77:0/3045303259 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fcf9807a9e0 con 0x7fcfa80cf830 2024-03-06T07:33:43.630 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:43.630 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:43.630 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf8806b4d0 msgr2=0x7fcf8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf8806b4d0 0x7fcf8806d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80d1b50 tx=0x7fcf90009040 comp rx=0 tx=0).stop 2024-03-06T07:33:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa80cf830 msgr2=0x7fcfa814bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa80cf830 0x7fcfa814bb30 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fcf9802bf70 tx=0x7fcf980046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 shutdown_connections 2024-03-06T07:33:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcfa815dfe0 0x7fcfa80b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcfa80d29c0 0x7fcfa80b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf8806b4d0 0x7fcf8806d990 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 --2- 172.21.15.77:0/3045303259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa80cf830 0x7fcfa814bb30 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 >> 172.21.15.77:0/3045303259 conn(0x7fcfa80d3cd0 msgr2=0x7fcfa80b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 shutdown_connections 2024-03-06T07:33:43.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:43.629+0000 7fcfac823640 1 -- 172.21.15.77:0/3045303259 wait complete. 2024-03-06T07:33:43.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:43.844 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:44.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:44.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.946+0000 7f892c41c640 1 Processor -- start 2024-03-06T07:33:44.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.946+0000 7f892c41c640 1 -- start start 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f892c41c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f892415dfe0 0x7f89240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f892c41c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89240d29c0 0x7f89240b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f892c41c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89240b3430 con 0x7f89240cf830 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f892c41c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89240b31e0 con 0x7f892415dfe0 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f892c41c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89240b3d60 con 0x7f89240d29c0 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f89299b7640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f89240cf830 0x7f89240d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52970/0 (socket says 172.21.15.77:52970) 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f89299b7640 1 -- 172.21.15.77:0/3069750784 learned_addr learned my addr 172.21.15.77:0/3069750784 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f892a9b9640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89240d29c0 0x7f89240b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f892a1b8640 1 --2- 172.21.15.77:0/3069750784 >> v2:172.21.15.158:3300/0 conn(0x7f892415dfe0 0x7f89240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246252593 0 0) 0x7f89240b3430 con 0x7f89240cf830 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.947+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910003520 con 0x7f89240cf830 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2231548336 0 0) 0x7f8910003520 con 0x7f89240cf830 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89240d29c0 msgr2=0x7f89240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89240d29c0 0x7f89240b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 >> v2:172.21.15.158:3300/0 conn(0x7f892415dfe0 msgr2=0x7f89240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 --2- 172.21.15.77:0/3069750784 >> v2:172.21.15.158:3300/0 conn(0x7f892415dfe0 0x7f89240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240b3710 con 0x7f89240cf830 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f8914001670 con 0x7f89240cf830 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 >> v1:172.21.15.77:6789/0 conn(0x7f89240cf830 legacy=0x7f89240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89100039b0 0x7f8910003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8910004800 0x7f8910004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89240d29c0 0x7f89100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240b31e0 con 0x7f8910004800 2024-03-06T07:33:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240b3d60 con 0x7f89100039b0 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891000b9c0 con 0x7f89240d29c0 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89299b7640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8910004800 0x7f8910004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f892a1b8640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89100039b0 0x7f8910003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f892a9b9640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89240d29c0 0x7f89100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89299b7640 1 -- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89240d29c0 msgr2=0x7f89100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89299b7640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f89240d29c0 0x7f89100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89299b7640 1 -- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89100039b0 msgr2=0x7f8910003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89299b7640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f89100039b0 0x7f8910003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.948+0000 7f89299b7640 1 -- 172.21.15.77:0/3069750784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8910003520 con 0x7f8910004800 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.949+0000 7f89299b7640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8910004800 0x7f8910004c80 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f89240cfc70 tx=0x7f8910005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa59cd7bb21e5420 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.949+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914007d50 con 0x7f8910004800 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.949+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8914030070 con 0x7f8910004800 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.949+0000 7f89291b6640 1 -- 172.21.15.77:0/3069750784 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8914005d10 con 0x7f8910004800 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.950+0000 7f892c41c640 1 -- 172.21.15.77:0/3069750784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8910004800 msgr2=0x7f8910004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.950+0000 7f892c41c640 1 --2- 172.21.15.77:0/3069750784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8910004800 0x7f8910004c80 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f89240cfc70 tx=0x7f8910005830 comp rx=0 tx=0).stop 2024-03-06T07:33:44.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.950+0000 7f892a9b9640 1 -- 172.21.15.77:0/3069750784 reap_dead start 2024-03-06T07:33:44.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.950+0000 7f892c41c640 1 -- 172.21.15.77:0/3069750784 shutdown_connections 2024-03-06T07:33:44.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.950+0000 7f892c41c640 1 -- 172.21.15.77:0/3069750784 >> 172.21.15.77:0/3069750784 conn(0x7f89240d3cd0 msgr2=0x7f89240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.950+0000 7f892c41c640 1 -- 172.21.15.77:0/3069750784 shutdown_connections 2024-03-06T07:33:44.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.950+0000 7f892c41c640 1 -- 172.21.15.77:0/3069750784 wait complete. 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.951+0000 7f892c41c640 1 Processor -- start 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.951+0000 7f892c41c640 1 -- start start 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.951+0000 7f892c41c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8910004800 0x7f892414fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.951+0000 7f892c41c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f892415b6b0 0x7f89240b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892c41c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89240b8420 0x7f89240b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892c41c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240d1df0 con 0x7f89240b8420 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892c41c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240d1c70 con 0x7f892415b6b0 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892c41c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892415e3c0 con 0x7f8910004800 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89240b8420 0x7f89240b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89240b8420 0x7f89240b0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57060/0 (socket says 172.21.15.77:57060) 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 -- 172.21.15.77:0/387308481 learned_addr learned my addr 172.21.15.77:0/387308481 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a1b8640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8910004800 0x7f892414fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f89299b7640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f892415b6b0 0x7f89240b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 -- 172.21.15.77:0/387308481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8910004800 msgr2=0x7f892414fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8910004800 0x7f892414fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 -- 172.21.15.77:0/387308481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f892415b6b0 msgr2=0x7f89240b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f892415b6b0 0x7f89240b57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 -- 172.21.15.77:0/387308481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89241500b0 con 0x7f89240b8420 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892a9b9640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89240b8420 0x7f89240b0c10 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f892408c940 tx=0x7f891c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f891affd640 1 -- 172.21.15.77:0/387308481 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c00fd80 con 0x7f89240b8420 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f891affd640 1 -- 172.21.15.77:0/387308481 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c01d070 con 0x7f89240b8420 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89240b6010 con 0x7f89240b8420 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f891affd640 1 -- 172.21.15.77:0/387308481 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c015670 con 0x7f89240b8420 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.952+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89240b65d0 con 0x7f89240b8420 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.953+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892404f330 con 0x7f89240b8420 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.954+0000 7f891affd640 1 -- 172.21.15.77:0/387308481 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f891c015810 con 0x7f89240b8420 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.955+0000 7f891affd640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890806b500 0x7f890806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.955+0000 7f892a1b8640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890806b500 0x7f890806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.955+0000 7f891affd640 1 -- 172.21.15.77:0/387308481 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f891c08aa00 con 0x7f89240b8420 2024-03-06T07:33:44.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.955+0000 7f892a1b8640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890806b500 0x7f890806d9c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8920009b00 tx=0x7f8920008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:44.959+0000 7f891affd640 1 -- 172.21.15.77:0/387308481 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f891c010040 con 0x7f89240b8420 2024-03-06T07:33:45.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.172+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89240cef00 con 0x7f89240b8420 2024-03-06T07:33:45.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.173+0000 7f891affd640 1 -- 172.21.15.77:0/387308481 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f891c059860 con 0x7f89240b8420 2024-03-06T07:33:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:45.175 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:45.175 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890806b500 msgr2=0x7f890806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890806b500 0x7f890806d9c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8920009b00 tx=0x7f8920008040 comp rx=0 tx=0).stop 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89240b8420 msgr2=0x7f89240b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89240b8420 0x7f89240b0c10 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f892408c940 tx=0x7f891c00f580 comp rx=0 tx=0).stop 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 shutdown_connections 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f890806b500 0x7f890806d9c0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89240b8420 0x7f89240b0c10 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f892415b6b0 0x7f89240b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 --2- 172.21.15.77:0/387308481 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8910004800 0x7f892414fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 >> 172.21.15.77:0/387308481 conn(0x7f89240d3cd0 msgr2=0x7f892406cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 shutdown_connections 2024-03-06T07:33:45.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:45.175+0000 7f892c41c640 1 -- 172.21.15.77:0/387308481 wait complete. 2024-03-06T07:33:45.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:45.392 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:46.393 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.497+0000 7fce475c9640 1 Processor -- start 2024-03-06T07:33:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.497+0000 7fce475c9640 1 -- start start 2024-03-06T07:33:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.498+0000 7fce475c9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fce4015dfe0 0x7fce400cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.498+0000 7fce475c9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce400d29c0 0x7fce400b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.498+0000 7fce475c9640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce400b3430 con 0x7fce400cf830 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.498+0000 7fce475c9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce400b31e0 con 0x7fce400d29c0 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.498+0000 7fce475c9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce400b3d60 con 0x7fce4015dfe0 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.498+0000 7fce45b66640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce400d29c0 0x7fce400b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.499+0000 7fce45b66640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fce400d29c0 0x7fce400b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57072/0 (socket says 172.21.15.77:57072) 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.499+0000 7fce45b66640 1 -- 172.21.15.77:0/3620697538 learned_addr learned my addr 172.21.15.77:0/3620697538 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.499+0000 7fce45365640 1 --2- 172.21.15.77:0/3620697538 >> v2:172.21.15.158:3300/0 conn(0x7fce4015dfe0 0x7fce400cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.499+0000 7fce45b66640 1 -- 172.21.15.77:0/3620697538 >> v2:172.21.15.158:3300/0 conn(0x7fce4015dfe0 msgr2=0x7fce400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.499+0000 7fce45b66640 1 --2- 172.21.15.77:0/3620697538 >> v2:172.21.15.158:3300/0 conn(0x7fce4015dfe0 0x7fce400cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.499+0000 7fce45b66640 1 -- 172.21.15.77:0/3620697538 >> v1:172.21.15.158:6789/0 conn(0x7fce400cf830 legacy=0x7fce400d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.499+0000 7fce45b66640 1 -- 172.21.15.77:0/3620697538 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce400b3710 con 0x7fce400d29c0 2024-03-06T07:33:46.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce45b66640 1 --2- 172.21.15.77:0/3620697538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce400d29c0 0x7fce400b41f0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fce40067990 tx=0x7fce3802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965ea9b2e8360c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce33fff640 1 -- 172.21.15.77:0/3620697538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3802cb20 con 0x7fce400d29c0 2024-03-06T07:33:46.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce33fff640 1 -- 172.21.15.77:0/3620697538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3802ccc0 con 0x7fce400d29c0 2024-03-06T07:33:46.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce33fff640 1 -- 172.21.15.77:0/3620697538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38034610 con 0x7fce400d29c0 2024-03-06T07:33:46.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce475c9640 1 -- 172.21.15.77:0/3620697538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce400d29c0 msgr2=0x7fce400b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce475c9640 1 --2- 172.21.15.77:0/3620697538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce400d29c0 0x7fce400b41f0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fce40067990 tx=0x7fce3802bf80 comp rx=0 tx=0).stop 2024-03-06T07:33:46.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce475c9640 1 -- 172.21.15.77:0/3620697538 shutdown_connections 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce475c9640 1 --2- 172.21.15.77:0/3620697538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce400d29c0 0x7fce400b41f0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce475c9640 1 --2- 172.21.15.77:0/3620697538 >> v2:172.21.15.158:3300/0 conn(0x7fce4015dfe0 0x7fce400cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.500+0000 7fce475c9640 1 -- 172.21.15.77:0/3620697538 >> 172.21.15.77:0/3620697538 conn(0x7fce400d3cd0 msgr2=0x7fce400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.501+0000 7fce475c9640 1 -- 172.21.15.77:0/3620697538 shutdown_connections 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.501+0000 7fce475c9640 1 -- 172.21.15.77:0/3620697538 wait complete. 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce475c9640 1 Processor -- start 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce475c9640 1 -- start start 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce475c9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce400cf830 0x7fce401f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce475c9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce400d29c0 0x7fce400ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce475c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce4015dfe0 0x7fce4015fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce475c9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce400d1df0 con 0x7fce4015dfe0 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce475c9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce400d1c70 con 0x7fce400cf830 2024-03-06T07:33:46.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce475c9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4015e3c0 con 0x7fce400d29c0 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce45b66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce4015dfe0 0x7fce4015fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce45b66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce4015dfe0 0x7fce4015fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57088/0 (socket says 172.21.15.77:57088) 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.502+0000 7fce45b66640 1 -- 172.21.15.77:0/3182605784 learned_addr learned my addr 172.21.15.77:0/3182605784 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce45365640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce400cf830 0x7fce401f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce44b64640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce400d29c0 0x7fce400ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce45b66640 1 -- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce400d29c0 msgr2=0x7fce400ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce45b66640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce400d29c0 0x7fce400ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce45b66640 1 -- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce400cf830 msgr2=0x7fce401f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce45b66640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce400cf830 0x7fce401f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce45b66640 1 -- 172.21.15.77:0/3182605784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce401f49d0 con 0x7fce4015dfe0 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce44b64640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce400d29c0 0x7fce400ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce45b66640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce4015dfe0 0x7fce4015fc90 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fce38009f40 tx=0x7fce38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce31ffb640 1 -- 172.21.15.77:0/3182605784 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38043020 con 0x7fce4015dfe0 2024-03-06T07:33:46.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce31ffb640 1 -- 172.21.15.77:0/3182605784 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3803e450 con 0x7fce4015dfe0 2024-03-06T07:33:46.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce400cc170 con 0x7fce4015dfe0 2024-03-06T07:33:46.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce31ffb640 1 -- 172.21.15.77:0/3182605784 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38009510 con 0x7fce4015dfe0 2024-03-06T07:33:46.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.503+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce400cb690 con 0x7fce4015dfe0 2024-03-06T07:33:46.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.504+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4004f330 con 0x7fce4015dfe0 2024-03-06T07:33:46.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.505+0000 7fce31ffb640 1 -- 172.21.15.77:0/3182605784 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce38046020 con 0x7fce4015dfe0 2024-03-06T07:33:46.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.505+0000 7fce31ffb640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2406b4d0 0x7fce2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.505+0000 7fce31ffb640 1 -- 172.21.15.77:0/3182605784 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fce380ac9b0 con 0x7fce4015dfe0 2024-03-06T07:33:46.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.505+0000 7fce45365640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2406b4d0 0x7fce2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.506+0000 7fce45365640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2406b4d0 0x7fce2406d990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fce4009df30 tx=0x7fce34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.510+0000 7fce31ffb640 1 -- 172.21.15.77:0/3182605784 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce3804d030 con 0x7fce4015dfe0 2024-03-06T07:33:46.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.844+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce400008d0 con 0x7fce4015dfe0 2024-03-06T07:33:46.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.845+0000 7fce31ffb640 1 -- 172.21.15.77:0/3182605784 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fce401f49d0 con 0x7fce4015dfe0 2024-03-06T07:33:46.847 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:46.847 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:46.847 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2406b4d0 msgr2=0x7fce2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2406b4d0 0x7fce2406d990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fce4009df30 tx=0x7fce34008040 comp rx=0 tx=0).stop 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce4015dfe0 msgr2=0x7fce4015fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce4015dfe0 0x7fce4015fc90 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fce38009f40 tx=0x7fce38004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 shutdown_connections 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fce2406b4d0 0x7fce2406d990 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce4015dfe0 0x7fce4015fc90 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fce400d29c0 0x7fce400ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 --2- 172.21.15.77:0/3182605784 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fce400cf830 0x7fce401f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 >> 172.21.15.77:0/3182605784 conn(0x7fce400d3cd0 msgr2=0x7fce4006d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 shutdown_connections 2024-03-06T07:33:46.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:46.847+0000 7fce475c9640 1 -- 172.21.15.77:0/3182605784 wait complete. 2024-03-06T07:33:47.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:47.061 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:48.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:48.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.161+0000 7f096d65c640 1 Processor -- start 2024-03-06T07:33:48.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.161+0000 7f096d65c640 1 -- start start 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f096d65c640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f096815dfe0 0x7f09680cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f096d65c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680b3430 con 0x7f09680d29c0 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f096d65c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680b3710 con 0x7f09680cf830 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f096d65c640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09680b31e0 con 0x7f096815dfe0 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f09677fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f09680d29c0 0x7f09680b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52972/0 (socket says 172.21.15.77:52972) 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f09677fe640 1 -- 172.21.15.77:0/2872948857 learned_addr learned my addr 172.21.15.77:0/2872948857 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907919961 0 0) 0x7f09680b3430 con 0x7f09680d29c0 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f0966ffd640 1 --2- 172.21.15.77:0/2872948857 >> v2:172.21.15.158:3300/0 conn(0x7f096815dfe0 0x7f09680cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.162+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0950003520 con 0x7f09680d29c0 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 585097154 0 0) 0x7f0950003520 con 0x7f09680d29c0 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 >> v2:172.21.15.158:3300/0 conn(0x7f096815dfe0 msgr2=0x7f09680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 --2- 172.21.15.77:0/2872948857 >> v2:172.21.15.158:3300/0 conn(0x7f096815dfe0 0x7f09680cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 >> v1:172.21.15.158:6789/0 conn(0x7f09680cf830 legacy=0x7f09680d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680b39f0 con 0x7f09680d29c0 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f09600019f0 con 0x7f09680d29c0 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 >> v1:172.21.15.77:6789/0 conn(0x7f09680d29c0 legacy=0x7f09680b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09500039b0 0x7f0950003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0950004800 0x7f0950004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09500051c0 0x7f095000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680b31e0 con 0x7f09500051c0 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095000c090 con 0x7f0950004800 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095000c290 con 0x7f09500039b0 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f09677fe640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09500051c0 0x7f095000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f09667fc640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0950004800 0x7f0950004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0966ffd640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09500039b0 0x7f0950003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f09677fe640 1 -- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09500039b0 msgr2=0x7f0950003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f09677fe640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09500039b0 0x7f0950003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f09677fe640 1 -- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0950004800 msgr2=0x7f0950004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f09677fe640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0950004800 0x7f0950004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f09677fe640 1 -- 172.21.15.77:0/2872948857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950003520 con 0x7f09500051c0 2024-03-06T07:33:48.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.163+0000 7f0966ffd640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f09500039b0 0x7f0950003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:48.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.164+0000 7f09677fe640 1 -- 172.21.15.77:0/2872948857 reap_dead start 2024-03-06T07:33:48.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.164+0000 7f09677fe640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09500051c0 0x7f095000b9c0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f096000a830 tx=0x7f0950005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1eb41dc70b7f4fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.164+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096000a480 con 0x7f09500051c0 2024-03-06T07:33:48.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.164+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960034630 con 0x7f09500051c0 2024-03-06T07:33:48.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.164+0000 7f0965ffb640 1 -- 172.21.15.77:0/2872948857 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960034a20 con 0x7f09500051c0 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.165+0000 7f096d65c640 1 -- 172.21.15.77:0/2872948857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09500051c0 msgr2=0x7f095000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.165+0000 7f096d65c640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09500051c0 0x7f095000b9c0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f096000a830 tx=0x7f0950005830 comp rx=0 tx=0).stop 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.165+0000 7f096d65c640 1 -- 172.21.15.77:0/2872948857 shutdown_connections 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.165+0000 7f096d65c640 1 --2- 172.21.15.77:0/2872948857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09500051c0 0x7f095000b9c0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.165+0000 7f096d65c640 1 -- 172.21.15.77:0/2872948857 >> 172.21.15.77:0/2872948857 conn(0x7f09680d3cd0 msgr2=0x7f09680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.165+0000 7f096d65c640 1 -- 172.21.15.77:0/2872948857 shutdown_connections 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.165+0000 7f096d65c640 1 -- 172.21.15.77:0/2872948857 wait complete. 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.167+0000 7f096d65c640 1 Processor -- start 2024-03-06T07:33:48.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.167+0000 7f096d65c640 1 -- start start 2024-03-06T07:33:48.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f096d65c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09500051c0 0x7f09680b8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f096d65c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f096814c0a0 0x7f096814d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f096d65c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f096814dbf0 0x7f096809dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f096d65c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096815e3c0 con 0x7f096814dbf0 2024-03-06T07:33:48.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f096d65c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680b3d60 con 0x7f09500051c0 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f096d65c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096815e540 con 0x7f096814c0a0 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f096814dbf0 0x7f096809dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f096814dbf0 0x7f096809dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57104/0 (socket says 172.21.15.77:57104) 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 -- 172.21.15.77:0/1548155653 learned_addr learned my addr 172.21.15.77:0/1548155653 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f0966ffd640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09500051c0 0x7f09680b8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 -- 172.21.15.77:0/1548155653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f096814c0a0 msgr2=0x7f096814d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f096814c0a0 0x7f096814d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 -- 172.21.15.77:0/1548155653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09500051c0 msgr2=0x7f09680b8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09500051c0 0x7f09680b8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 -- 172.21.15.77:0/1548155653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096815b9b0 con 0x7f096814dbf0 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.168+0000 7f09677fe640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f096814dbf0 0x7f096809dc60 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f0960000f80 tx=0x7f09680b38e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.169+0000 7f094f7fe640 1 -- 172.21.15.77:0/1548155653 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960034430 con 0x7f096814dbf0 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.169+0000 7f094f7fe640 1 -- 172.21.15.77:0/1548155653 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960044070 con 0x7f096814dbf0 2024-03-06T07:33:48.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.169+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096814c610 con 0x7f096814dbf0 2024-03-06T07:33:48.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.169+0000 7f094f7fe640 1 -- 172.21.15.77:0/1548155653 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09600347e0 con 0x7f096814dbf0 2024-03-06T07:33:48.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.169+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096814cbd0 con 0x7f096814dbf0 2024-03-06T07:33:48.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.170+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096804f330 con 0x7f096814dbf0 2024-03-06T07:33:48.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.170+0000 7f094f7fe640 1 -- 172.21.15.77:0/1548155653 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f096003c070 con 0x7f096814dbf0 2024-03-06T07:33:48.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.170+0000 7f094f7fe640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f095406b4d0 0x7f095406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.171+0000 7f094f7fe640 1 -- 172.21.15.77:0/1548155653 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f09600abbb0 con 0x7f096814dbf0 2024-03-06T07:33:48.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.171+0000 7f0966ffd640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f095406b4d0 0x7f095406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.171+0000 7f0966ffd640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f095406b4d0 0x7f095406d990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f096806d000 tx=0x7f0958008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.175+0000 7f094f7fe640 1 -- 172.21.15.77:0/1548155653 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0960032e40 con 0x7f096814dbf0 2024-03-06T07:33:48.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.393+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09680b0b60 con 0x7f096814dbf0 2024-03-06T07:33:48.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.393+0000 7f094f7fe640 1 -- 172.21.15.77:0/1548155653 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f09600af020 con 0x7f096814dbf0 2024-03-06T07:33:48.395 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:48.395 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:48.395 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f095406b4d0 msgr2=0x7f095406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f095406b4d0 0x7f095406d990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f096806d000 tx=0x7f0958008040 comp rx=0 tx=0).stop 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f096814dbf0 msgr2=0x7f096809dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f096814dbf0 0x7f096809dc60 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f0960000f80 tx=0x7f09680b38e0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 shutdown_connections 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f095406b4d0 0x7f095406d990 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f096814dbf0 0x7f096809dc60 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f096814c0a0 0x7f096814d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 --2- 172.21.15.77:0/1548155653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09500051c0 0x7f09680b8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 >> 172.21.15.77:0/1548155653 conn(0x7f09680d3cd0 msgr2=0x7f0968160fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 shutdown_connections 2024-03-06T07:33:48.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:48.395+0000 7f096d65c640 1 -- 172.21.15.77:0/1548155653 wait complete. 2024-03-06T07:33:48.609 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:48.609 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:49.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:49.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.711+0000 7f4d012df640 1 Processor -- start 2024-03-06T07:33:49.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.711+0000 7f4d012df640 1 -- start start 2024-03-06T07:33:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4d012df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4cfc0d4760 0x7f4cfc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4d012df640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0d25d0 con 0x7f4cfc0b4170 2024-03-06T07:33:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4d012df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0d2870 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4d012df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cfc0d4bf0 con 0x7f4cfc0d4760 2024-03-06T07:33:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4cfa7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4cfc0d4760 0x7f4cfc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4cfa7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4cfc0d4760 0x7f4cfc0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41570/0 (socket says 172.21.15.77:41570) 2024-03-06T07:33:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4cfa7fc640 1 -- 172.21.15.77:0/401307362 learned_addr learned my addr 172.21.15.77:0/401307362 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4cfaffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4cfc0b4170 0x7f4cfc0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52928/0 (socket says 172.21.15.77:52928) 2024-03-06T07:33:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4cfa7fc640 1 -- 172.21.15.77:0/401307362 >> v1:172.21.15.158:6789/0 conn(0x7f4cfc0d0a80 legacy=0x7f4cfc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4cfa7fc640 1 -- 172.21.15.77:0/401307362 >> v1:172.21.15.77:6789/0 conn(0x7f4cfc0b4170 legacy=0x7f4cfc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.712+0000 7f4cfa7fc640 1 -- 172.21.15.77:0/401307362 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0d2b50 con 0x7f4cfc0d4760 2024-03-06T07:33:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.713+0000 7f4cfa7fc640 1 --2- 172.21.15.77:0/401307362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d4760 0x7f4cfc0b6a90 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009870 tx=0x7f4cec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eafa7d6c1a9ea47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.713+0000 7f4cf9ffb640 1 -- 172.21.15.77:0/401307362 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec034070 con 0x7f4cfc0d4760 2024-03-06T07:33:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.713+0000 7f4cf9ffb640 1 -- 172.21.15.77:0/401307362 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec030860 con 0x7f4cfc0d4760 2024-03-06T07:33:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.713+0000 7f4cf9ffb640 1 -- 172.21.15.77:0/401307362 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec030c30 con 0x7f4cfc0d4760 2024-03-06T07:33:49.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.714+0000 7f4d012df640 1 -- 172.21.15.77:0/401307362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d4760 msgr2=0x7f4cfc0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.714+0000 7f4d012df640 1 --2- 172.21.15.77:0/401307362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d4760 0x7f4cfc0b6a90 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009870 tx=0x7f4cec02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.714+0000 7f4d012df640 1 -- 172.21.15.77:0/401307362 shutdown_connections 2024-03-06T07:33:49.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.714+0000 7f4d012df640 1 --2- 172.21.15.77:0/401307362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d4760 0x7f4cfc0b6a90 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.714+0000 7f4d012df640 1 -- 172.21.15.77:0/401307362 >> 172.21.15.77:0/401307362 conn(0x7f4cfc0d3c40 msgr2=0x7f4cfc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.714+0000 7f4d012df640 1 -- 172.21.15.77:0/401307362 shutdown_connections 2024-03-06T07:33:49.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.714+0000 7f4d012df640 1 -- 172.21.15.77:0/401307362 wait complete. 2024-03-06T07:33:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.715+0000 7f4d012df640 1 Processor -- start 2024-03-06T07:33:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.715+0000 7f4d012df640 1 -- start start 2024-03-06T07:33:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4d012df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cfc0b4170 0x7f4cfc0cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4d012df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d0a80 0x7f4cfc0b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4d012df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cfc0b9430 0x7f4cfc15bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4d012df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc0b4dd0 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4d012df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc0ce9e0 con 0x7f4cfc0b9430 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4d012df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc0b4f50 con 0x7f4cfc0b4170 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d0a80 0x7f4cfc0b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d0a80 0x7f4cfc0b8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41580/0 (socket says 172.21.15.77:41580) 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfa7fc640 1 -- 172.21.15.77:0/2463515970 learned_addr learned my addr 172.21.15.77:0/2463515970 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfb7fe640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cfc0b9430 0x7f4cfc15bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfaffd640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cfc0b4170 0x7f4cfc0cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfa7fc640 1 -- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cfc0b4170 msgr2=0x7f4cfc0cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfa7fc640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cfc0b4170 0x7f4cfc0cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfa7fc640 1 -- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cfc0b9430 msgr2=0x7f4cfc15bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfa7fc640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cfc0b9430 0x7f4cfc15bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfa7fc640 1 -- 172.21.15.77:0/2463515970 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc15ca20 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.716+0000 7f4cfb7fe640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cfc0b9430 0x7f4cfc15bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.717+0000 7f4cfa7fc640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d0a80 0x7f4cfc0b8ef0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009870 tx=0x7f4cec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.717+0000 7f4ce37fe640 1 -- 172.21.15.77:0/2463515970 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec034040 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.717+0000 7f4ce37fe640 1 -- 172.21.15.77:0/2463515970 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec0433f0 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.717+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cfc0cadb0 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.717+0000 7f4ce37fe640 1 -- 172.21.15.77:0/2463515970 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec0437e0 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.717+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cfc0cb370 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.718+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cfc04f330 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.718+0000 7f4ce37fe640 1 -- 172.21.15.77:0/2463515970 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cec04a060 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.719+0000 7f4ce37fe640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ce806b500 0x7f4ce806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.719+0000 7f4cfaffd640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ce806b500 0x7f4ce806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.719+0000 7f4ce37fe640 1 -- 172.21.15.77:0/2463515970 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4cec0ac510 con 0x7f4cfc0d0a80 2024-03-06T07:33:49.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.719+0000 7f4cfaffd640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ce806b500 0x7f4ce806d9c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0000f60 tx=0x7f4cf0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:49.723+0000 7f4ce37fe640 1 -- 172.21.15.77:0/2463515970 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cec03e070 con 0x7f4cfc0d0a80 2024-03-06T07:33:50.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.079+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cfc0733b0 con 0x7f4cfc0d0a80 2024-03-06T07:33:50.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.080+0000 7f4ce37fe640 1 -- 172.21.15.77:0/2463515970 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f4cec04be90 con 0x7f4cfc0d0a80 2024-03-06T07:33:50.082 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:50.082 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:50.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ce806b500 msgr2=0x7f4ce806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ce806b500 0x7f4ce806d9c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0000f60 tx=0x7f4cf0009040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d0a80 msgr2=0x7f4cfc0b8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d0a80 0x7f4cfc0b8ef0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009870 tx=0x7f4cec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 shutdown_connections 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f4ce806b500 0x7f4ce806d9c0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cfc0d0a80 0x7f4cfc0b8ef0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4cfc0b9430 0x7f4cfc15bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 --2- 172.21.15.77:0/2463515970 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4cfc0b4170 0x7f4cfc0cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.082+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 >> 172.21.15.77:0/2463515970 conn(0x7f4cfc0d3c40 msgr2=0x7f4cfc0b7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.083+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 shutdown_connections 2024-03-06T07:33:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:50.083+0000 7f4d012df640 1 -- 172.21.15.77:0/2463515970 wait complete. 2024-03-06T07:33:50.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:50.297 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:51.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:51.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.398+0000 7f367b73e640 1 Processor -- start 2024-03-06T07:33:51.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.398+0000 7f367b73e640 1 -- start start 2024-03-06T07:33:51.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f367b73e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f367406b220 0x7f367415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f367b73e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f36740ce920 0x7f3674071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f367b73e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367415ea30 con 0x7f3674071ec0 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f367b73e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3674161b70 con 0x7f367406b220 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f367b73e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367415ed10 con 0x7f36740ce920 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f36794da640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f367406b220 0x7f367415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f36794da640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f367406b220 0x7f367415e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41582/0 (socket says 172.21.15.77:41582) 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f36794da640 1 -- 172.21.15.77:0/3420289366 learned_addr learned my addr 172.21.15.77:0/3420289366 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.399+0000 7f3678cd9640 1 --2- 172.21.15.77:0/3420289366 >> v2:172.21.15.158:3300/0 conn(0x7f36740ce920 0x7f3674071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.400+0000 7f36794da640 1 -- 172.21.15.77:0/3420289366 >> v2:172.21.15.158:3300/0 conn(0x7f36740ce920 msgr2=0x7f3674071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.400+0000 7f36794da640 1 --2- 172.21.15.77:0/3420289366 >> v2:172.21.15.158:3300/0 conn(0x7f36740ce920 0x7f3674071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.400+0000 7f36794da640 1 -- 172.21.15.77:0/3420289366 >> v1:172.21.15.158:6789/0 conn(0x7f3674071ec0 legacy=0x7f3674162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.400+0000 7f36794da640 1 -- 172.21.15.77:0/3420289366 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367415eff0 con 0x7f367406b220 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.400+0000 7f36794da640 1 --2- 172.21.15.77:0/3420289366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f367406b220 0x7f367415e450 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f367406cf40 tx=0x7f366402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7f1e5da74b5052 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.400+0000 7f366bfff640 1 -- 172.21.15.77:0/3420289366 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664035020 con 0x7f367406b220 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.400+0000 7f366bfff640 1 -- 172.21.15.77:0/3420289366 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3664031870 con 0x7f367406b220 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.400+0000 7f366bfff640 1 -- 172.21.15.77:0/3420289366 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664031c40 con 0x7f367406b220 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.401+0000 7f367b73e640 1 -- 172.21.15.77:0/3420289366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f367406b220 msgr2=0x7f367415e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.401+0000 7f367b73e640 1 --2- 172.21.15.77:0/3420289366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f367406b220 0x7f367415e450 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f367406cf40 tx=0x7f366402b910 comp rx=0 tx=0).stop 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.401+0000 7f367b73e640 1 -- 172.21.15.77:0/3420289366 shutdown_connections 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.401+0000 7f367b73e640 1 --2- 172.21.15.77:0/3420289366 >> v2:172.21.15.158:3300/0 conn(0x7f36740ce920 0x7f3674071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.401+0000 7f367b73e640 1 --2- 172.21.15.77:0/3420289366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f367406b220 0x7f367415e450 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.401+0000 7f367b73e640 1 -- 172.21.15.77:0/3420289366 >> 172.21.15.77:0/3420289366 conn(0x7f367406c0c0 msgr2=0x7f367406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.401+0000 7f367b73e640 1 -- 172.21.15.77:0/3420289366 shutdown_connections 2024-03-06T07:33:51.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.401+0000 7f367b73e640 1 -- 172.21.15.77:0/3420289366 wait complete. 2024-03-06T07:33:51.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.402+0000 7f367b73e640 1 Processor -- start 2024-03-06T07:33:51.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.402+0000 7f367b73e640 1 -- start start 2024-03-06T07:33:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f367b73e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f367406b220 0x7f3674147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f367b73e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3674071ec0 0x7f367414a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f367b73e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36740ce920 0x7f36741491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f367b73e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36740ce380 con 0x7f3674071ec0 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f367b73e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36740ce200 con 0x7f36740ce920 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f367b73e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3674162710 con 0x7f367406b220 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3674071ec0 0x7f367414a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3674071ec0 0x7f367414a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41590/0 (socket says 172.21.15.77:41590) 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 -- 172.21.15.77:0/2330057088 learned_addr learned my addr 172.21.15.77:0/2330057088 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f36794da640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f367406b220 0x7f3674147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3679cdb640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36740ce920 0x7f36741491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 -- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f367406b220 msgr2=0x7f3674147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f367406b220 0x7f3674147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 -- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36740ce920 msgr2=0x7f36741491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36740ce920 0x7f36741491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 -- 172.21.15.77:0/2330057088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367414ac70 con 0x7f3674071ec0 2024-03-06T07:33:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f36794da640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f367406b220 0x7f3674147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3679cdb640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36740ce920 0x7f36741491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.403+0000 7f3678cd9640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3674071ec0 0x7f367414a730 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f367000d620 tx=0x7f367000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.404+0000 7f3669ffb640 1 -- 172.21.15.77:0/2330057088 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367000b620 con 0x7f3674071ec0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.404+0000 7f3669ffb640 1 -- 172.21.15.77:0/2330057088 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3670019070 con 0x7f3674071ec0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.404+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3674148140 con 0x7f3674071ec0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.404+0000 7f3669ffb640 1 -- 172.21.15.77:0/2330057088 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367000ba00 con 0x7f3674071ec0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.404+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3674148700 con 0x7f3674071ec0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.405+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367404f330 con 0x7f3674071ec0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.405+0000 7f3669ffb640 1 -- 172.21.15.77:0/2330057088 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f367000bba0 con 0x7f3674071ec0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.405+0000 7f3669ffb640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f365806b4d0 0x7f365806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.406+0000 7f3669ffb640 1 -- 172.21.15.77:0/2330057088 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f367008a660 con 0x7f3674071ec0 2024-03-06T07:33:51.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.406+0000 7f36794da640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f365806b4d0 0x7f365806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.406+0000 7f36794da640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f365806b4d0 0x7f365806d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3664036000 tx=0x7f366403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.414+0000 7f3669ffb640 1 -- 172.21.15.77:0/2330057088 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36700594c0 con 0x7f3674071ec0 2024-03-06T07:33:51.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.630+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36740cdf00 con 0x7f3674071ec0 2024-03-06T07:33:51.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.631+0000 7f3669ffb640 1 -- 172.21.15.77:0/2330057088 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f3670058dc0 con 0x7f3674071ec0 2024-03-06T07:33:51.633 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:51.633 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:51.633 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f365806b4d0 msgr2=0x7f365806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f365806b4d0 0x7f365806d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3664036000 tx=0x7f366403c040 comp rx=0 tx=0).stop 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3674071ec0 msgr2=0x7f367414a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3674071ec0 0x7f367414a730 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f367000d620 tx=0x7f367000dae0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 shutdown_connections 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f365806b4d0 0x7f365806d990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3674071ec0 0x7f367414a730 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f36740ce920 0x7f36741491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 --2- 172.21.15.77:0/2330057088 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f367406b220 0x7f3674147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 >> 172.21.15.77:0/2330057088 conn(0x7f367406c0c0 msgr2=0x7f3674073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 shutdown_connections 2024-03-06T07:33:51.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:51.633+0000 7f367b73e640 1 -- 172.21.15.77:0/2330057088 wait complete. 2024-03-06T07:33:51.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:51.847 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:52.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:52.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.948+0000 7f5b9d833640 1 Processor -- start 2024-03-06T07:33:52.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.948+0000 7f5b9d833640 1 -- start start 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b9d833640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5b98158600 0x7f5b98072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b9d833640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f5b98162030 0x7f5b98162480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b9d833640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9815eb40 con 0x7f5b9815e3a0 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b9d833640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b98161ba0 con 0x7f5b98158600 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b9d833640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b9815ee20 con 0x7f5b98162030 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b96ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5b98158600 0x7f5b98072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b967fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5b9815e3a0 0x7f5b9806d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52930/0 (socket says 172.21.15.77:52930) 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b96ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5b98158600 0x7f5b98072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41592/0 (socket says 172.21.15.77:41592) 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b967fc640 1 -- 172.21.15.77:0/4215840023 learned_addr learned my addr 172.21.15.77:0/4215840023 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b977fe640 1 --2- 172.21.15.77:0/4215840023 >> v2:172.21.15.158:3300/0 conn(0x7f5b98162030 0x7f5b98162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b96ffd640 1 -- 172.21.15.77:0/4215840023 >> v2:172.21.15.158:3300/0 conn(0x7f5b98162030 msgr2=0x7f5b98162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b96ffd640 1 --2- 172.21.15.77:0/4215840023 >> v2:172.21.15.158:3300/0 conn(0x7f5b98162030 0x7f5b98162480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b96ffd640 1 -- 172.21.15.77:0/4215840023 >> v1:172.21.15.77:6789/0 conn(0x7f5b9815e3a0 legacy=0x7f5b9806d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b95ffb640 1 -- 172.21.15.77:0/4215840023 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719848905 0 0) 0x7f5b9815eb40 con 0x7f5b9815e3a0 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.949+0000 7f5b96ffd640 1 -- 172.21.15.77:0/4215840023 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9815f100 con 0x7f5b98158600 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.950+0000 7f5b96ffd640 1 --2- 172.21.15.77:0/4215840023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98072730 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f5b98161d20 tx=0x7f5b8802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0d46d611185816 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.950+0000 7f5b95ffb640 1 -- 172.21.15.77:0/4215840023 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b880315a0 con 0x7f5b98158600 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.950+0000 7f5b95ffb640 1 -- 172.21.15.77:0/4215840023 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b88037070 con 0x7f5b98158600 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.950+0000 7f5b95ffb640 1 -- 172.21.15.77:0/4215840023 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88031960 con 0x7f5b98158600 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.951+0000 7f5b9d833640 1 -- 172.21.15.77:0/4215840023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 msgr2=0x7f5b98072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.951+0000 7f5b9d833640 1 --2- 172.21.15.77:0/4215840023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98072730 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f5b98161d20 tx=0x7f5b8802b840 comp rx=0 tx=0).stop 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.951+0000 7f5b9d833640 1 -- 172.21.15.77:0/4215840023 shutdown_connections 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.951+0000 7f5b9d833640 1 --2- 172.21.15.77:0/4215840023 >> v2:172.21.15.158:3300/0 conn(0x7f5b98162030 0x7f5b98162480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.951+0000 7f5b9d833640 1 --2- 172.21.15.77:0/4215840023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98072730 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.951+0000 7f5b9d833640 1 -- 172.21.15.77:0/4215840023 >> 172.21.15.77:0/4215840023 conn(0x7f5b9806c0c0 msgr2=0x7f5b9806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.951+0000 7f5b9d833640 1 -- 172.21.15.77:0/4215840023 shutdown_connections 2024-03-06T07:33:52.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.951+0000 7f5b9d833640 1 -- 172.21.15.77:0/4215840023 wait complete. 2024-03-06T07:33:52.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.952+0000 7f5b9d833640 1 Processor -- start 2024-03-06T07:33:52.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.952+0000 7f5b9d833640 1 -- start start 2024-03-06T07:33:52.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.953+0000 7f5b9d833640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98144470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b9d833640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b9815e3a0 0x7f5b98148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b9d833640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b98162030 0x7f5b9815b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b9d833640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9806e080 con 0x7f5b98158600 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b9d833640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9806afb0 con 0x7f5b9815e3a0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b9d833640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b98159ba0 con 0x7f5b98162030 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98144470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41608/0 (socket says 172.21.15.77:41608) 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 -- 172.21.15.77:0/3891124703 learned_addr learned my addr 172.21.15.77:0/3891124703 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b967fc640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b9815e3a0 0x7f5b98148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b977fe640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b98162030 0x7f5b9815b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 -- 172.21.15.77:0/3891124703 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b98162030 msgr2=0x7f5b9815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b98162030 0x7f5b9815b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 -- 172.21.15.77:0/3891124703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b9815e3a0 msgr2=0x7f5b98148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b9815e3a0 0x7f5b98148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 -- 172.21.15.77:0/3891124703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b98159e80 con 0x7f5b98158600 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b96ffd640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98144470 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f5b88000c00 tx=0x7f5b880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b7ffff640 1 -- 172.21.15.77:0/3891124703 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88004230 con 0x7f5b98158600 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b7ffff640 1 -- 172.21.15.77:0/3891124703 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b880043d0 con 0x7f5b98158600 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b7ffff640 1 -- 172.21.15.77:0/3891124703 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88032860 con 0x7f5b98158600 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b9815a160 con 0x7f5b98158600 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.954+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9815a720 con 0x7f5b98158600 2024-03-06T07:33:52.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.955+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9804f330 con 0x7f5b98158600 2024-03-06T07:33:52.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.956+0000 7f5b7ffff640 1 -- 172.21.15.77:0/3891124703 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b88032a00 con 0x7f5b98158600 2024-03-06T07:33:52.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.956+0000 7f5b7ffff640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b8406b4d0 0x7f5b8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.957+0000 7f5b7ffff640 1 -- 172.21.15.77:0/3891124703 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5b8802f030 con 0x7f5b98158600 2024-03-06T07:33:52.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.957+0000 7f5b967fc640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b8406b4d0 0x7f5b8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.957+0000 7f5b967fc640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b8406b4d0 0x7f5b8406d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5b98158de0 tx=0x7f5b8c007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:52.961+0000 7f5b7ffff640 1 -- 172.21.15.77:0/3891124703 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b88041070 con 0x7f5b98158600 2024-03-06T07:33:53.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.177+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b980d06f0 con 0x7f5b98158600 2024-03-06T07:33:53.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.178+0000 7f5b7ffff640 1 -- 172.21.15.77:0/3891124703 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f5b88044340 con 0x7f5b98158600 2024-03-06T07:33:53.180 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:53.180 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:53.180 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b8406b4d0 msgr2=0x7f5b8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b8406b4d0 0x7f5b8406d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5b98158de0 tx=0x7f5b8c007b50 comp rx=0 tx=0).stop 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 msgr2=0x7f5b98144470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98144470 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f5b88000c00 tx=0x7f5b880046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 shutdown_connections 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f5b98162030 0x7f5b9815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5b9815e3a0 0x7f5b98148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f5b8406b4d0 0x7f5b8406d990 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 --2- 172.21.15.77:0/3891124703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b98158600 0x7f5b98144470 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 >> 172.21.15.77:0/3891124703 conn(0x7f5b9806c0c0 msgr2=0x7f5b98073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 shutdown_connections 2024-03-06T07:33:53.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:53.180+0000 7f5b9d833640 1 -- 172.21.15.77:0/3891124703 wait complete. 2024-03-06T07:33:53.394 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:53.394 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:54.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:54.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.495+0000 7fced79ae640 1 Processor -- start 2024-03-06T07:33:54.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.495+0000 7fced79ae640 1 -- start start 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced79ae640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fced013dc10 0x7fced00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced79ae640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fced015ef60 0x7fced00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced79ae640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced015f580 con 0x7fced015bf30 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced79ae640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced00cf3f0 con 0x7fced013dc10 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced79ae640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced015b210 con 0x7fced015ef60 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced574a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fced013dc10 0x7fced00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced4f49640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fced015bf30 0x7fced015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52944/0 (socket says 172.21.15.77:52944) 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced574a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fced013dc10 0x7fced00b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41622/0 (socket says 172.21.15.77:41622) 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced4f49640 1 -- 172.21.15.77:0/1445830510 learned_addr learned my addr 172.21.15.77:0/1445830510 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:54.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced574a640 1 -- 172.21.15.77:0/1445830510 >> v2:172.21.15.158:3300/0 conn(0x7fced015ef60 msgr2=0x7fced00b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced574a640 1 --2- 172.21.15.77:0/1445830510 >> v2:172.21.15.158:3300/0 conn(0x7fced015ef60 0x7fced00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced574a640 1 -- 172.21.15.77:0/1445830510 >> v1:172.21.15.77:6789/0 conn(0x7fced015bf30 legacy=0x7fced015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fcec7fff640 1 -- 172.21.15.77:0/1445830510 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747029315 0 0) 0x7fced015f580 con 0x7fced015bf30 2024-03-06T07:33:54.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.496+0000 7fced574a640 1 -- 172.21.15.77:0/1445830510 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced015b450 con 0x7fced013dc10 2024-03-06T07:33:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.497+0000 7fced574a640 1 --2- 172.21.15.77:0/1445830510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced013dc10 0x7fced00b2950 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fced0068700 tx=0x7fcecc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a73fe6e00f0d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.497+0000 7fcec7fff640 1 -- 172.21.15.77:0/1445830510 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc0315a0 con 0x7fced013dc10 2024-03-06T07:33:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.497+0000 7fcec7fff640 1 -- 172.21.15.77:0/1445830510 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc037070 con 0x7fced013dc10 2024-03-06T07:33:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.497+0000 7fcec7fff640 1 -- 172.21.15.77:0/1445830510 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc031960 con 0x7fced013dc10 2024-03-06T07:33:54.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.498+0000 7fced79ae640 1 -- 172.21.15.77:0/1445830510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced013dc10 msgr2=0x7fced00b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.498+0000 7fced79ae640 1 --2- 172.21.15.77:0/1445830510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced013dc10 0x7fced00b2950 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fced0068700 tx=0x7fcecc02b840 comp rx=0 tx=0).stop 2024-03-06T07:33:54.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.498+0000 7fced79ae640 1 -- 172.21.15.77:0/1445830510 shutdown_connections 2024-03-06T07:33:54.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.498+0000 7fced79ae640 1 --2- 172.21.15.77:0/1445830510 >> v2:172.21.15.158:3300/0 conn(0x7fced015ef60 0x7fced00b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.498+0000 7fced79ae640 1 --2- 172.21.15.77:0/1445830510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced013dc10 0x7fced00b2950 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.498+0000 7fced79ae640 1 -- 172.21.15.77:0/1445830510 >> 172.21.15.77:0/1445830510 conn(0x7fced00cf8a0 msgr2=0x7fced00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.498+0000 7fced79ae640 1 -- 172.21.15.77:0/1445830510 shutdown_connections 2024-03-06T07:33:54.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.498+0000 7fced79ae640 1 -- 172.21.15.77:0/1445830510 wait complete. 2024-03-06T07:33:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.499+0000 7fced79ae640 1 Processor -- start 2024-03-06T07:33:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.499+0000 7fced79ae640 1 -- start start 2024-03-06T07:33:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced79ae640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced013dc10 0x7fced01418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced79ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced015bf30 0x7fced006c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced79ae640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced015ef60 0x7fced01389e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced79ae640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced00cd7a0 con 0x7fced015bf30 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced79ae640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced013dff0 con 0x7fced015ef60 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced79ae640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced00cd920 con 0x7fced013dc10 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced015bf30 0x7fced006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced015bf30 0x7fced006c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41634/0 (socket says 172.21.15.77:41634) 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 -- 172.21.15.77:0/1652455819 learned_addr learned my addr 172.21.15.77:0/1652455819 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 -- 172.21.15.77:0/1652455819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced013dc10 msgr2=0x7fced01418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced013dc10 0x7fced01418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 -- 172.21.15.77:0/1652455819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced015ef60 msgr2=0x7fced01389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced5f4b640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced015ef60 0x7fced01389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced015ef60 0x7fced01389e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 -- 172.21.15.77:0/1652455819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced0143110 con 0x7fced015bf30 2024-03-06T07:33:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fced4f49640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced015bf30 0x7fced006c800 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fcec000c4d0 tx=0x7fcec000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.500+0000 7fcec5ffb640 1 -- 172.21.15.77:0/1652455819 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0018070 con 0x7fced015bf30 2024-03-06T07:33:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.501+0000 7fcec5ffb640 1 -- 172.21.15.77:0/1652455819 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec0007500 con 0x7fced015bf30 2024-03-06T07:33:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.501+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced006cd40 con 0x7fced015bf30 2024-03-06T07:33:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.501+0000 7fcec5ffb640 1 -- 172.21.15.77:0/1652455819 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0004a20 con 0x7fced015bf30 2024-03-06T07:33:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.501+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced006d2f0 con 0x7fced015bf30 2024-03-06T07:33:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.502+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced004f330 con 0x7fced015bf30 2024-03-06T07:33:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.502+0000 7fcec5ffb640 1 -- 172.21.15.77:0/1652455819 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcec0004bc0 con 0x7fced015bf30 2024-03-06T07:33:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.502+0000 7fcec5ffb640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fceb406b4d0 0x7fceb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.503+0000 7fcec5ffb640 1 -- 172.21.15.77:0/1652455819 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcec0089540 con 0x7fced015bf30 2024-03-06T07:33:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.503+0000 7fced574a640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fceb406b4d0 0x7fceb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.503+0000 7fced574a640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fceb406b4d0 0x7fceb406d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fcecc036000 tx=0x7fcecc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.507+0000 7fcec5ffb640 1 -- 172.21.15.77:0/1652455819 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fced0143110 con 0x7fced015bf30 2024-03-06T07:33:54.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.721+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fced015ad30 con 0x7fced015bf30 2024-03-06T07:33:54.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.721+0000 7fcec5ffb640 1 -- 172.21.15.77:0/1652455819 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fcec0058320 con 0x7fced015bf30 2024-03-06T07:33:54.723 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:54.723 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:54.724 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fceb406b4d0 msgr2=0x7fceb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fceb406b4d0 0x7fceb406d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fcecc036000 tx=0x7fcecc030000 comp rx=0 tx=0).stop 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced015bf30 msgr2=0x7fced006c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced015bf30 0x7fced006c800 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fcec000c4d0 tx=0x7fcec000c990 comp rx=0 tx=0).stop 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 shutdown_connections 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fceb406b4d0 0x7fceb406d990 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced015bf30 0x7fced006c800 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fced015ef60 0x7fced01389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 --2- 172.21.15.77:0/1652455819 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fced013dc10 0x7fced01418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 >> 172.21.15.77:0/1652455819 conn(0x7fced00cf8a0 msgr2=0x7fced00b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 shutdown_connections 2024-03-06T07:33:54.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:54.723+0000 7fced79ae640 1 -- 172.21.15.77:0/1652455819 wait complete. 2024-03-06T07:33:54.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:54.937 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:55.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.038+0000 7fb82bc15640 1 Processor -- start 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.038+0000 7fb82bc15640 1 -- start start 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82bc15640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb82415bdd0 0x7fb8240c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82bc15640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb8240d16e0 0x7fb8240d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82bc15640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8240cebe0 con 0x7fb82415fa10 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82bc15640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8240ce980 con 0x7fb8240d16e0 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82bc15640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8240ceec0 con 0x7fb82415bdd0 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82a1b2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb8240d16e0 0x7fb8240d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82a1b2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb8240d16e0 0x7fb8240d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41642/0 (socket says 172.21.15.77:41642) 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82a1b2640 1 -- 172.21.15.77:0/3714130115 learned_addr learned my addr 172.21.15.77:0/3714130115 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb8299b1640 1 --2- 172.21.15.77:0/3714130115 >> v2:172.21.15.158:3300/0 conn(0x7fb82415bdd0 0x7fb8240c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82a1b2640 1 -- 172.21.15.77:0/3714130115 >> v2:172.21.15.158:3300/0 conn(0x7fb82415bdd0 msgr2=0x7fb8240c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82a1b2640 1 --2- 172.21.15.77:0/3714130115 >> v2:172.21.15.158:3300/0 conn(0x7fb82415bdd0 0x7fb8240c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82a1b2640 1 -- 172.21.15.77:0/3714130115 >> v1:172.21.15.158:6789/0 conn(0x7fb82415fa10 legacy=0x7fb8240ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.039+0000 7fb82a1b2640 1 -- 172.21.15.77:0/3714130115 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8240cf1a0 con 0x7fb8240d16e0 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.040+0000 7fb82a1b2640 1 --2- 172.21.15.77:0/3714130115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8240d16e0 0x7fb8240d1af0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fb8240c5520 tx=0x7fb81c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30221700dba0ddf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.040+0000 7fb8289af640 1 -- 172.21.15.77:0/3714130115 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c02c9e0 con 0x7fb8240d16e0 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.040+0000 7fb8289af640 1 -- 172.21.15.77:0/3714130115 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c02cb80 con 0x7fb8240d16e0 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.040+0000 7fb8289af640 1 -- 172.21.15.77:0/3714130115 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c0343f0 con 0x7fb8240d16e0 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.041+0000 7fb82bc15640 1 -- 172.21.15.77:0/3714130115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8240d16e0 msgr2=0x7fb8240d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.041+0000 7fb82bc15640 1 --2- 172.21.15.77:0/3714130115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8240d16e0 0x7fb8240d1af0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fb8240c5520 tx=0x7fb81c02be40 comp rx=0 tx=0).stop 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.041+0000 7fb82bc15640 1 -- 172.21.15.77:0/3714130115 shutdown_connections 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.041+0000 7fb82bc15640 1 --2- 172.21.15.77:0/3714130115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8240d16e0 0x7fb8240d1af0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.041+0000 7fb82bc15640 1 --2- 172.21.15.77:0/3714130115 >> v2:172.21.15.158:3300/0 conn(0x7fb82415bdd0 0x7fb8240c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.041+0000 7fb82bc15640 1 -- 172.21.15.77:0/3714130115 >> 172.21.15.77:0/3714130115 conn(0x7fb82409a9f0 msgr2=0x7fb82409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.041+0000 7fb82bc15640 1 -- 172.21.15.77:0/3714130115 shutdown_connections 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.041+0000 7fb82bc15640 1 -- 172.21.15.77:0/3714130115 wait complete. 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.042+0000 7fb82bc15640 1 Processor -- start 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.042+0000 7fb82bc15640 1 -- start start 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb82bc15640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8240d16e0 0x7fb8240b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb82bc15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82415bdd0 0x7fb8240b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb82bc15640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82415fa10 0x7fb8240b5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb82bc15640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8240cc5d0 con 0x7fb82415bdd0 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb82bc15640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8240c5b50 con 0x7fb8240d16e0 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb82bc15640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82415c1b0 con 0x7fb82415fa10 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82415bdd0 0x7fb8240b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82415bdd0 0x7fb8240b58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41654/0 (socket says 172.21.15.77:41654) 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 -- 172.21.15.77:0/2473014141 learned_addr learned my addr 172.21.15.77:0/2473014141 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb82a1b2640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82415fa10 0x7fb8240b5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8299b1640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8240d16e0 0x7fb8240b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 -- 172.21.15.77:0/2473014141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82415fa10 msgr2=0x7fb8240b5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82415fa10 0x7fb8240b5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 -- 172.21.15.77:0/2473014141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8240d16e0 msgr2=0x7fb8240b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8240d16e0 0x7fb8240b6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 -- 172.21.15.77:0/2473014141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8240c9b30 con 0x7fb82415bdd0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.043+0000 7fb8291b0640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82415bdd0 0x7fb8240b58c0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fb82000b3f0 tx=0x7fb82000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.044+0000 7fb81a7fc640 1 -- 172.21.15.77:0/2473014141 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820015020 con 0x7fb82415bdd0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.044+0000 7fb81a7fc640 1 -- 172.21.15.77:0/2473014141 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb820011830 con 0x7fb82415bdd0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.044+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82415eff0 con 0x7fb82415bdd0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.044+0000 7fb81a7fc640 1 -- 172.21.15.77:0/2473014141 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820011c20 con 0x7fb82415bdd0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.044+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8240c9df0 con 0x7fb82415bdd0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.045+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8240ce770 con 0x7fb82415bdd0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.045+0000 7fb81a7fc640 1 -- 172.21.15.77:0/2473014141 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb820018020 con 0x7fb82415bdd0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.045+0000 7fb81a7fc640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb80806b4d0 0x7fb80806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.046+0000 7fb81a7fc640 1 -- 172.21.15.77:0/2473014141 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb820089db0 con 0x7fb82415bdd0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.046+0000 7fb8299b1640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb80806b4d0 0x7fb80806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.046+0000 7fb8299b1640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb80806b4d0 0x7fb80806d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fb814006fd0 tx=0x7fb814008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.050+0000 7fb81a7fc640 1 -- 172.21.15.77:0/2473014141 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8240ce770 con 0x7fb82415bdd0 2024-03-06T07:33:56.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.341+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8240b7a40 con 0x7fb82415bdd0 2024-03-06T07:33:56.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.341+0000 7fb81a7fc640 1 -- 172.21.15.77:0/2473014141 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fb82001f030 con 0x7fb82415bdd0 2024-03-06T07:33:56.343 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:56.343 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:56.344 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:56.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb80806b4d0 msgr2=0x7fb80806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb80806b4d0 0x7fb80806d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fb814006fd0 tx=0x7fb814008040 comp rx=0 tx=0).stop 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82415bdd0 msgr2=0x7fb8240b58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82415bdd0 0x7fb8240b58c0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fb82000b3f0 tx=0x7fb82000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 shutdown_connections 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb80806b4d0 0x7fb80806d990 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb82415bdd0 0x7fb8240b58c0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb82415fa10 0x7fb8240b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 --2- 172.21.15.77:0/2473014141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8240d16e0 0x7fb8240b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 >> 172.21.15.77:0/2473014141 conn(0x7fb82409a9f0 msgr2=0x7fb8240ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 shutdown_connections 2024-03-06T07:33:56.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:56.343+0000 7fb82bc15640 1 -- 172.21.15.77:0/2473014141 wait complete. 2024-03-06T07:33:56.558 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:56.558 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:57.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:57.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.660+0000 7f118f834640 1 Processor -- start 2024-03-06T07:33:57.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.660+0000 7f118f834640 1 -- start start 2024-03-06T07:33:57.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.661+0000 7f118f834640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f118815dfe0 0x7f11880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.661+0000 7f118f834640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 0x7f11880b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.661+0000 7f118f834640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880b3430 con 0x7f11880cf830 2024-03-06T07:33:57.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.661+0000 7f118f834640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11880b31e0 con 0x7f118815dfe0 2024-03-06T07:33:57.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.661+0000 7f118f834640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11880b3d60 con 0x7f11880d29c0 2024-03-06T07:33:57.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.661+0000 7f118d5d0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f118815dfe0 0x7f11880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.661+0000 7f118d5d0640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f118815dfe0 0x7f11880cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41664/0 (socket says 172.21.15.77:41664) 2024-03-06T07:33:57.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.661+0000 7f118d5d0640 1 -- 172.21.15.77:0/2136961699 learned_addr learned my addr 172.21.15.77:0/2136961699 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.662+0000 7f118d5d0640 1 -- 172.21.15.77:0/2136961699 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 msgr2=0x7f11880b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.662+0000 7f118d5d0640 1 --2- 172.21.15.77:0/2136961699 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 0x7f11880b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.662+0000 7f118d5d0640 1 -- 172.21.15.77:0/2136961699 >> v1:172.21.15.158:6789/0 conn(0x7f11880cf830 legacy=0x7f11880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.662+0000 7f118d5d0640 1 -- 172.21.15.77:0/2136961699 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11880b3710 con 0x7f118815dfe0 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.662+0000 7f118d5d0640 1 --2- 172.21.15.77:0/2136961699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f118815dfe0 0x7f11880cf2f0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f1184009870 tx=0x7f118402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba383f109c8aba1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.662+0000 7f117ffff640 1 -- 172.21.15.77:0/2136961699 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184035020 con 0x7f118815dfe0 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.662+0000 7f117ffff640 1 -- 172.21.15.77:0/2136961699 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1184031640 con 0x7f118815dfe0 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.662+0000 7f117ffff640 1 -- 172.21.15.77:0/2136961699 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118402d040 con 0x7f118815dfe0 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.663+0000 7f118f834640 1 -- 172.21.15.77:0/2136961699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f118815dfe0 msgr2=0x7f11880cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.663+0000 7f118f834640 1 --2- 172.21.15.77:0/2136961699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f118815dfe0 0x7f11880cf2f0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f1184009870 tx=0x7f118402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.663+0000 7f118f834640 1 -- 172.21.15.77:0/2136961699 shutdown_connections 2024-03-06T07:33:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.663+0000 7f118f834640 1 --2- 172.21.15.77:0/2136961699 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 0x7f11880b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.663+0000 7f118f834640 1 --2- 172.21.15.77:0/2136961699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f118815dfe0 0x7f11880cf2f0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.663+0000 7f118f834640 1 -- 172.21.15.77:0/2136961699 >> 172.21.15.77:0/2136961699 conn(0x7f11880d3cd0 msgr2=0x7f11880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.663+0000 7f118f834640 1 -- 172.21.15.77:0/2136961699 shutdown_connections 2024-03-06T07:33:57.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.663+0000 7f118f834640 1 -- 172.21.15.77:0/2136961699 wait complete. 2024-03-06T07:33:57.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.664+0000 7f118f834640 1 Processor -- start 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.664+0000 7f118f834640 1 -- start start 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118f834640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11880cf830 0x7f11881f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118f834640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 0x7f11880cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118f834640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f118815dfe0 0x7f1188160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118f834640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11880d1df0 con 0x7f11880cf830 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118f834640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11880d1c70 con 0x7f118815dfe0 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118f834640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118815e3c0 con 0x7f11880d29c0 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11880cf830 0x7f11881f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11880cf830 0x7f11881f3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41676/0 (socket says 172.21.15.77:41676) 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 -- 172.21.15.77:0/1149520923 learned_addr learned my addr 172.21.15.77:0/1149520923 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118ddd1640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f118815dfe0 0x7f1188160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118cdcf640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 0x7f11880cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 -- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 msgr2=0x7f11880cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 0x7f11880cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 -- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f118815dfe0 msgr2=0x7f1188160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f118815dfe0 0x7f1188160ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 -- 172.21.15.77:0/1149520923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118815fc80 con 0x7f11880cf830 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118ddd1640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f118815dfe0 0x7f1188160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.665+0000 7f118d5d0640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11880cf830 0x7f11881f3040 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f118402b930 tx=0x7f11840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.666+0000 7f117dffb640 1 -- 172.21.15.77:0/1149520923 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118403ac60 con 0x7f11880cf830 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.666+0000 7f117dffb640 1 -- 172.21.15.77:0/1149520923 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118403ae20 con 0x7f11880cf830 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.666+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11880cb780 con 0x7f11880cf830 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.666+0000 7f117dffb640 1 -- 172.21.15.77:0/1149520923 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184044630 con 0x7f11880cf830 2024-03-06T07:33:57.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.666+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11880cbd40 con 0x7f11880cf830 2024-03-06T07:33:57.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.667+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118804f330 con 0x7f11880cf830 2024-03-06T07:33:57.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.667+0000 7f117dffb640 1 -- 172.21.15.77:0/1149520923 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f118404b060 con 0x7f11880cf830 2024-03-06T07:33:57.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.668+0000 7f117dffb640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f116c06b4d0 0x7f116c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.668+0000 7f117dffb640 1 -- 172.21.15.77:0/1149520923 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f11840abf60 con 0x7f11880cf830 2024-03-06T07:33:57.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.668+0000 7f118cdcf640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f116c06b4d0 0x7f116c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.668+0000 7f118cdcf640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f116c06b4d0 0x7f116c06d990 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f118806d510 tx=0x7f1178009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.672+0000 7f117dffb640 1 -- 172.21.15.77:0/1149520923 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f118403f070 con 0x7f11880cf830 2024-03-06T07:33:57.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.887+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11880747a0 con 0x7f11880cf830 2024-03-06T07:33:57.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.887+0000 7f117dffb640 1 -- 172.21.15.77:0/1149520923 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f118402f300 con 0x7f11880cf830 2024-03-06T07:33:57.892 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:57.892 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f116c06b4d0 msgr2=0x7f116c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f116c06b4d0 0x7f116c06d990 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f118806d510 tx=0x7f1178009040 comp rx=0 tx=0).stop 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11880cf830 msgr2=0x7f11881f3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11880cf830 0x7f11881f3040 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f118402b930 tx=0x7f11840046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 shutdown_connections 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f118815dfe0 0x7f1188160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f11880d29c0 0x7f11880cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f116c06b4d0 0x7f116c06d990 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 --2- 172.21.15.77:0/1149520923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11880cf830 0x7f11881f3040 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 >> 172.21.15.77:0/1149520923 conn(0x7f11880d3cd0 msgr2=0x7f11880d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 shutdown_connections 2024-03-06T07:33:57.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:57.889+0000 7f118f834640 1 -- 172.21.15.77:0/1149520923 wait complete. 2024-03-06T07:33:58.103 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:58.103 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:33:59.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:59.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.206+0000 7fcf7aad4640 1 Processor -- start 2024-03-06T07:33:59.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.206+0000 7fcf7aad4640 1 -- start start 2024-03-06T07:33:59.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.207+0000 7fcf7aad4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 0x7fcf74072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.207+0000 7fcf7aad4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcf740cd390 0x7fcf740ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.207+0000 7fcf7aad4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf740d0610 con 0x7fcf7415baf0 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.207+0000 7fcf7aad4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf74072c40 con 0x7fcf740cd390 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.207+0000 7fcf7aad4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf740d08f0 con 0x7fcf7406b6c0 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.207+0000 7fcf78870640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcf7415baf0 0x7fcf7406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59606/0 (socket says 172.21.15.77:59606) 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.207+0000 7fcf79071640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcf740cd390 0x7fcf740ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.207+0000 7fcf78870640 1 -- 172.21.15.77:0/3125205232 learned_addr learned my addr 172.21.15.77:0/3125205232 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.208+0000 7fcf73fff640 1 --2- 172.21.15.77:0/3125205232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 0x7fcf74072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.208+0000 7fcf79071640 1 -- 172.21.15.77:0/3125205232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 msgr2=0x7fcf74072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.208+0000 7fcf79071640 1 --2- 172.21.15.77:0/3125205232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 0x7fcf74072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.208+0000 7fcf79071640 1 -- 172.21.15.77:0/3125205232 >> v1:172.21.15.77:6789/0 conn(0x7fcf7415baf0 legacy=0x7fcf7406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.208+0000 7fcf79071640 1 -- 172.21.15.77:0/3125205232 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf740d0bd0 con 0x7fcf740cd390 2024-03-06T07:33:59.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.208+0000 7fcf737fe640 1 -- 172.21.15.77:0/3125205232 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249401142 0 0) 0x7fcf740d0610 con 0x7fcf7415baf0 2024-03-06T07:33:59.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.208+0000 7fcf79071640 1 --2- 172.21.15.77:0/3125205232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf740ced90 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009d90 tx=0x7fcf6c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5ba4db03dd28c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf737fe640 1 -- 172.21.15.77:0/3125205232 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c039070 con 0x7fcf740cd390 2024-03-06T07:33:59.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf737fe640 1 -- 172.21.15.77:0/3125205232 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6c02cc50 con 0x7fcf740cd390 2024-03-06T07:33:59.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf737fe640 1 -- 172.21.15.77:0/3125205232 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c034440 con 0x7fcf740cd390 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf7aad4640 1 -- 172.21.15.77:0/3125205232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 msgr2=0x7fcf740ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/3125205232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf740ced90 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009d90 tx=0x7fcf6c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf7aad4640 1 -- 172.21.15.77:0/3125205232 shutdown_connections 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/3125205232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf740ced90 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/3125205232 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 0x7fcf74072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf7aad4640 1 -- 172.21.15.77:0/3125205232 >> 172.21.15.77:0/3125205232 conn(0x7fcf7406c0c0 msgr2=0x7fcf7406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf7aad4640 1 -- 172.21.15.77:0/3125205232 shutdown_connections 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.209+0000 7fcf7aad4640 1 -- 172.21.15.77:0/3125205232 wait complete. 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.210+0000 7fcf7aad4640 1 Processor -- start 2024-03-06T07:33:59.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.210+0000 7fcf7aad4640 1 -- start start 2024-03-06T07:33:59.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf7aad4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 0x7fcf7414e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf7aad4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf741514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf7aad4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf7415baf0 0x7fcf741414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf7aad4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7415bed0 con 0x7fcf740cd390 2024-03-06T07:33:59.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf7aad4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf740d1190 con 0x7fcf7415baf0 2024-03-06T07:33:59.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf7aad4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf740d16b0 con 0x7fcf7406b6c0 2024-03-06T07:33:59.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf741514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf73fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf741514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34260/0 (socket says 172.21.15.77:34260) 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.211+0000 7fcf73fff640 1 -- 172.21.15.77:0/1968992834 learned_addr learned my addr 172.21.15.77:0/1968992834 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf78870640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 0x7fcf7414e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf79071640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf7415baf0 0x7fcf741414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf73fff640 1 -- 172.21.15.77:0/1968992834 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 msgr2=0x7fcf7414e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf73fff640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 0x7fcf7414e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf73fff640 1 -- 172.21.15.77:0/1968992834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf7415baf0 msgr2=0x7fcf741414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf73fff640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf7415baf0 0x7fcf741414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf73fff640 1 -- 172.21.15.77:0/1968992834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf74151b70 con 0x7fcf740cd390 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf73fff640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf741514e0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000d830 tx=0x7fcf6000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf717fa640 1 -- 172.21.15.77:0/1968992834 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6000b6c0 con 0x7fcf740cd390 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf717fa640 1 -- 172.21.15.77:0/1968992834 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf60019070 con 0x7fcf740cd390 2024-03-06T07:33:59.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7414fba0 con 0x7fcf740cd390 2024-03-06T07:33:59.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf717fa640 1 -- 172.21.15.77:0/1968992834 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6000ba80 con 0x7fcf740cd390 2024-03-06T07:33:59.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.212+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf74150120 con 0x7fcf740cd390 2024-03-06T07:33:59.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.213+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7404f330 con 0x7fcf740cd390 2024-03-06T07:33:59.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.214+0000 7fcf717fa640 1 -- 172.21.15.77:0/1968992834 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf6001e460 con 0x7fcf740cd390 2024-03-06T07:33:59.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.214+0000 7fcf717fa640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf5806b4d0 0x7fcf5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.214+0000 7fcf717fa640 1 -- 172.21.15.77:0/1968992834 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcf6008a8c0 con 0x7fcf740cd390 2024-03-06T07:33:59.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.214+0000 7fcf78870640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf5806b4d0 0x7fcf5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.214+0000 7fcf78870640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf5806b4d0 0x7fcf5806d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fcf68000f60 tx=0x7fcf68007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.218+0000 7fcf717fa640 1 -- 172.21.15.77:0/1968992834 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf600597b0 con 0x7fcf740cd390 2024-03-06T07:33:59.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.429+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf740cfff0 con 0x7fcf740cd390 2024-03-06T07:33:59.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.430+0000 7fcf717fa640 1 -- 172.21.15.77:0/1968992834 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7fcf600590b0 con 0x7fcf740cd390 2024-03-06T07:33:59.432 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:33:59.432 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:33:59.432 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf5806b4d0 msgr2=0x7fcf5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf5806b4d0 0x7fcf5806d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fcf68000f60 tx=0x7fcf68007c00 comp rx=0 tx=0).stop 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 msgr2=0x7fcf741514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf741514e0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000d830 tx=0x7fcf6000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 shutdown_connections 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fcf5806b4d0 0x7fcf5806d990 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf740cd390 0x7fcf741514e0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcf7415baf0 0x7fcf741414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 --2- 172.21.15.77:0/1968992834 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fcf7406b6c0 0x7fcf7414e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 >> 172.21.15.77:0/1968992834 conn(0x7fcf7406c0c0 msgr2=0x7fcf74158700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 shutdown_connections 2024-03-06T07:33:59.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:33:59.432+0000 7fcf7aad4640 1 -- 172.21.15.77:0/1968992834 wait complete. 2024-03-06T07:33:59.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:33:59.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:34:00.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:00.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.748+0000 7f168973e640 1 Processor -- start 2024-03-06T07:34:00.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.748+0000 7f168973e640 1 -- start start 2024-03-06T07:34:00.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.748+0000 7f168973e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415bdd0 0x7f16840c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.748+0000 7f168973e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f168415fa10 0x7f16840ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.748+0000 7f168973e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f16840d16e0 0x7f16840d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.748+0000 7f168973e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16840cebe0 con 0x7f168415fa10 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.748+0000 7f168973e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16840ceec0 con 0x7f16840d16e0 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.748+0000 7f168973e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16840cf1a0 con 0x7f168415bdd0 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f167bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f168415fa10 0x7f16840ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f167bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f168415fa10 0x7f16840ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34274/0 (socket says 172.21.15.77:34274) 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f167bfff640 1 -- 172.21.15.77:0/3127645959 learned_addr learned my addr 172.21.15.77:0/3127645959 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f1682ffd640 1 --2- 172.21.15.77:0/3127645959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415bdd0 0x7f16840c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f16837fe640 1 --2- 172.21.15.77:0/3127645959 >> v2:172.21.15.158:3300/0 conn(0x7f16840d16e0 0x7f16840d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f167bfff640 1 -- 172.21.15.77:0/3127645959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415bdd0 msgr2=0x7f16840c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f167bfff640 1 --2- 172.21.15.77:0/3127645959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415bdd0 0x7f16840c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f167bfff640 1 -- 172.21.15.77:0/3127645959 >> v2:172.21.15.158:3300/0 conn(0x7f16840d16e0 msgr2=0x7f16840d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f167bfff640 1 --2- 172.21.15.77:0/3127645959 >> v2:172.21.15.158:3300/0 conn(0x7f16840d16e0 0x7f16840d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.749+0000 7f167bfff640 1 -- 172.21.15.77:0/3127645959 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840cc8e0 con 0x7f168415fa10 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.750+0000 7f167bfff640 1 --2- 172.21.15.77:0/3127645959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168415fa10 0x7f16840ce270 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f16840729e0 tx=0x7f166c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e31e0cc40758b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.750+0000 7f16827fc640 1 -- 172.21.15.77:0/3127645959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c02e070 con 0x7f168415fa10 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.750+0000 7f16827fc640 1 -- 172.21.15.77:0/3127645959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166c037070 con 0x7f168415fa10 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.750+0000 7f16827fc640 1 -- 172.21.15.77:0/3127645959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c031a00 con 0x7f168415fa10 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.750+0000 7f168973e640 1 -- 172.21.15.77:0/3127645959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168415fa10 msgr2=0x7f16840ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.750+0000 7f168973e640 1 --2- 172.21.15.77:0/3127645959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168415fa10 0x7f16840ce270 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f16840729e0 tx=0x7f166c02b880 comp rx=0 tx=0).stop 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.750+0000 7f168973e640 1 -- 172.21.15.77:0/3127645959 shutdown_connections 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.751+0000 7f168973e640 1 --2- 172.21.15.77:0/3127645959 >> v2:172.21.15.158:3300/0 conn(0x7f16840d16e0 0x7f16840d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.751+0000 7f168973e640 1 --2- 172.21.15.77:0/3127645959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168415fa10 0x7f16840ce270 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.751+0000 7f168973e640 1 --2- 172.21.15.77:0/3127645959 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415bdd0 0x7f16840c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.751+0000 7f168973e640 1 -- 172.21.15.77:0/3127645959 >> 172.21.15.77:0/3127645959 conn(0x7f168409a9f0 msgr2=0x7f168409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.751+0000 7f168973e640 1 -- 172.21.15.77:0/3127645959 shutdown_connections 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.751+0000 7f168973e640 1 -- 172.21.15.77:0/3127645959 wait complete. 2024-03-06T07:34:00.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f168973e640 1 Processor -- start 2024-03-06T07:34:00.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f168973e640 1 -- start start 2024-03-06T07:34:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f168973e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16840d16e0 0x7f168409bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f168973e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f168415bdd0 0x7f16840b6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f168973e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415fa10 0x7f168409e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f168973e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840cc5d0 con 0x7f16840d16e0 2024-03-06T07:34:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f168973e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840d2070 con 0x7f168415bdd0 2024-03-06T07:34:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f168973e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168415c1b0 con 0x7f168415fa10 2024-03-06T07:34:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.752+0000 7f1682ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16840d16e0 0x7f168409bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f1682ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16840d16e0 0x7f168409bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34288/0 (socket says 172.21.15.77:34288) 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f1682ffd640 1 -- 172.21.15.77:0/489860035 learned_addr learned my addr 172.21.15.77:0/489860035 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f1682ffd640 1 -- 172.21.15.77:0/489860035 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415fa10 msgr2=0x7f168409e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f1682ffd640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415fa10 0x7f168409e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f1682ffd640 1 -- 172.21.15.77:0/489860035 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f168415bdd0 msgr2=0x7f16840b6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f167bfff640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f168415bdd0 0x7f16840b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f1682ffd640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f168415bdd0 0x7f16840b6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f1682ffd640 1 -- 172.21.15.77:0/489860035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840b58c0 con 0x7f16840d16e0 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f1682ffd640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16840d16e0 0x7f168409bb70 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f167400d5f0 tx=0x7f167400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f167b7fe640 1 -- 172.21.15.77:0/489860035 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16740125f0 con 0x7f16840d16e0 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f167b7fe640 1 -- 172.21.15.77:0/489860035 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1674019070 con 0x7f16840d16e0 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16840b5ba0 con 0x7f16840d16e0 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f167b7fe640 1 -- 172.21.15.77:0/489860035 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16740129d0 con 0x7f16840d16e0 2024-03-06T07:34:00.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.753+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16840b6160 con 0x7f16840d16e0 2024-03-06T07:34:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.754+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168404f330 con 0x7f16840d16e0 2024-03-06T07:34:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.755+0000 7f167b7fe640 1 -- 172.21.15.77:0/489860035 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1674012b70 con 0x7f16840d16e0 2024-03-06T07:34:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.755+0000 7f167b7fe640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f166406b500 0x7f166406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.755+0000 7f167bfff640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f166406b500 0x7f166406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.755+0000 7f167b7fe640 1 -- 172.21.15.77:0/489860035 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f167408a4d0 con 0x7f16840d16e0 2024-03-06T07:34:00.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.756+0000 7f167bfff640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f166406b500 0x7f166406d9c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f166c036000 tx=0x7f166c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.760+0000 7f167b7fe640 1 -- 172.21.15.77:0/489860035 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16740593c0 con 0x7f16840d16e0 2024-03-06T07:34:00.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.967+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16840b7930 con 0x7f16840d16e0 2024-03-06T07:34:00.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.967+0000 7f167b7fe640 1 -- 172.21.15.77:0/489860035 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f1674058cc0 con 0x7f16840d16e0 2024-03-06T07:34:00.970 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:00.970 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:34:00.971 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:34:00.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f166406b500 msgr2=0x7f166406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f166406b500 0x7f166406d9c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f166c036000 tx=0x7f166c008030 comp rx=0 tx=0).stop 2024-03-06T07:34:00.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16840d16e0 msgr2=0x7f168409bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16840d16e0 0x7f168409bb70 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f167400d5f0 tx=0x7f167400dab0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 shutdown_connections 2024-03-06T07:34:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f168415fa10 0x7f168409e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f168415bdd0 0x7f16840b6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f166406b500 0x7f166406d9c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 --2- 172.21.15.77:0/489860035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16840d16e0 0x7f168409bb70 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 >> 172.21.15.77:0/489860035 conn(0x7f168409a9f0 msgr2=0x7f1684160150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 shutdown_connections 2024-03-06T07:34:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:00.970+0000 7f168973e640 1 -- 172.21.15.77:0/489860035 wait complete. 2024-03-06T07:34:01.184 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:05.435981+0000', 'modified': '2024-03-06T07:32:35.768648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9186, 'mds_1': 9177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9177': {'gid': 9177, 'name': 'a', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.77:6835/669492439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 669492439}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 669492439}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:21.073903+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9186': {'gid': 9186, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/2581202217', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2581202217}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2581202217}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9186, 'qdb_cluster': [9186]} 2024-03-06T07:34:01.184 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9177:up:stopping 2024-03-06T07:34:02.185 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:30:05.435981+0000", "damaged": [], "data_pools": [ 21 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9177": { "addr": "172.21.15.77:6835/669492439", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6834", "nonce": 669492439, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 669492439, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9177, "incarnation": 99, "join_fscid": -1, "laggy_since": "2024-03-06T07:31:21.073903+0000", "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_9186": { "addr": "172.21.15.77:6837/2581202217", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6836", "nonce": 2581202217, "type": "v2" }, { "addr": "172.21.15.77:6837", "nonce": 2581202217, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9186, "incarnation": 95, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 20, "modified": "2024-03-06T07:32:35.768648+0000", "qdb_cluster": [ 9186 ], "qdb_leader": 9186, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9186, "mds_1": 9177 } } } ], "standbys": [] } 2024-03-06T07:34:02.186 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:34:02.186 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:02.224 DEBUG:teuthology.orchestra.run.smithi158:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:02.274 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:34:02.274 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:34:02.275 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:02.290 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:02.290 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:34:02.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:02.290 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:34:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:02.345 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-06T07:34:02.345 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:34:02.345 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:02.364 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:02.364 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:34:02.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:02.364 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:34:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:02.423 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:34:02.423 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:34:02.423 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:02.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:02.439 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:34:02.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:02.439 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:34:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:02.493 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:34:02.494 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:34:02.494 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:02.512 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:02.513 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:34:02.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:02.513 DEBUG:teuthology.orchestra.run.smithi158:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:34:02.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:02.555 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:02.570 INFO:teuthology.orchestra.run.smithi158.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:34:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:02.571 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:02.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.694+0000 7f581a35c640 1 Processor -- start 2024-03-06T07:34:02.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.694+0000 7f581a35c640 1 -- start start 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.695+0000 7f581a35c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f581415dfe0 0x7f58140cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.695+0000 7f581a35c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58140b3430 con 0x7f58140d29c0 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.695+0000 7f581a35c640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58140b3710 con 0x7f58140cf830 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.695+0000 7f581a35c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58140b31e0 con 0x7f581415dfe0 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.695+0000 7f5813fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f581415dfe0 0x7f58140cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.695+0000 7f5813fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f581415dfe0 0x7f58140cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34300/0 (socket says 172.21.15.77:34300) 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.695+0000 7f5813fff640 1 -- 172.21.15.77:0/558144528 learned_addr learned my addr 172.21.15.77:0/558144528 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.696+0000 7f5813fff640 1 -- 172.21.15.77:0/558144528 >> v1:172.21.15.158:6789/0 conn(0x7f58140cf830 legacy=0x7f58140d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.696+0000 7f5813fff640 1 -- 172.21.15.77:0/558144528 >> v1:172.21.15.77:6789/0 conn(0x7f58140d29c0 legacy=0x7f58140b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.696+0000 7f5812ffd640 1 -- 172.21.15.77:0/558144528 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184832883 0 0) 0x7f58140b3430 con 0x7f58140d29c0 2024-03-06T07:34:02.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.696+0000 7f5813fff640 1 -- 172.21.15.77:0/558144528 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58140b39f0 con 0x7f581415dfe0 2024-03-06T07:34:02.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.696+0000 7f5813fff640 1 --2- 172.21.15.77:0/558144528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f581415dfe0 0x7f58140cf2f0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f58140b3360 tx=0x7f580402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560486b32e3e2938 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.696+0000 7f5812ffd640 1 -- 172.21.15.77:0/558144528 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5804034070 con 0x7f581415dfe0 2024-03-06T07:34:02.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f5812ffd640 1 -- 172.21.15.77:0/558144528 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5804030840 con 0x7f581415dfe0 2024-03-06T07:34:02.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f5812ffd640 1 -- 172.21.15.77:0/558144528 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5804030c10 con 0x7f581415dfe0 2024-03-06T07:34:02.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f581a35c640 1 -- 172.21.15.77:0/558144528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f581415dfe0 msgr2=0x7f58140cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f581a35c640 1 --2- 172.21.15.77:0/558144528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f581415dfe0 0x7f58140cf2f0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f58140b3360 tx=0x7f580402b980 comp rx=0 tx=0).stop 2024-03-06T07:34:02.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f581a35c640 1 -- 172.21.15.77:0/558144528 shutdown_connections 2024-03-06T07:34:02.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f581a35c640 1 --2- 172.21.15.77:0/558144528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f581415dfe0 0x7f58140cf2f0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f581a35c640 1 -- 172.21.15.77:0/558144528 >> 172.21.15.77:0/558144528 conn(0x7f58140d3cd0 msgr2=0x7f58140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f581a35c640 1 -- 172.21.15.77:0/558144528 shutdown_connections 2024-03-06T07:34:02.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.697+0000 7f581a35c640 1 -- 172.21.15.77:0/558144528 wait complete. 2024-03-06T07:34:02.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.698+0000 7f581a35c640 1 Processor -- start 2024-03-06T07:34:02.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.698+0000 7f581a35c640 1 -- start start 2024-03-06T07:34:02.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f581a35c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58140cf830 0x7f5814154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f581a35c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58140d29c0 0x7f58141f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f581a35c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f581415dfe0 0x7f58141f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f581a35c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581415e3c0 con 0x7f58140d29c0 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f581a35c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58140b3d60 con 0x7f581415dfe0 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f581a35c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581415e540 con 0x7f58140cf830 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f58137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58140d29c0 0x7f58141f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f58137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58140d29c0 0x7f58141f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34302/0 (socket says 172.21.15.77:34302) 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f58137fe640 1 -- 172.21.15.77:0/1478701324 learned_addr learned my addr 172.21.15.77:0/1478701324 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f58188f9640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f581415dfe0 0x7f58141f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.699+0000 7f5813fff640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58140cf830 0x7f5814154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f58137fe640 1 -- 172.21.15.77:0/1478701324 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58140cf830 msgr2=0x7f5814154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f58137fe640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58140cf830 0x7f5814154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f58137fe640 1 -- 172.21.15.77:0/1478701324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f581415dfe0 msgr2=0x7f58141f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f58137fe640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f581415dfe0 0x7f58141f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f58137fe640 1 -- 172.21.15.77:0/1478701324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58141f5950 con 0x7f58140d29c0 2024-03-06T07:34:02.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f58137fe640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58140d29c0 0x7f58141f4f60 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f580800c430 tx=0x7f580800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f5810ff9640 1 -- 172.21.15.77:0/1478701324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808011790 con 0x7f58140d29c0 2024-03-06T07:34:02.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f5810ff9640 1 -- 172.21.15.77:0/1478701324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5808017070 con 0x7f58140d29c0 2024-03-06T07:34:02.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58141f5c30 con 0x7f58140d29c0 2024-03-06T07:34:02.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f5810ff9640 1 -- 172.21.15.77:0/1478701324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808011b70 con 0x7f58140d29c0 2024-03-06T07:34:02.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.700+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58141f61f0 con 0x7f58140d29c0 2024-03-06T07:34:02.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.701+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581404f330 con 0x7f58140d29c0 2024-03-06T07:34:02.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.702+0000 7f5810ff9640 1 -- 172.21.15.77:0/1478701324 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5808011d10 con 0x7f58140d29c0 2024-03-06T07:34:02.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.702+0000 7f5810ff9640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f580006b500 0x7f580006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.702+0000 7f5810ff9640 1 -- 172.21.15.77:0/1478701324 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f580808a080 con 0x7f58140d29c0 2024-03-06T07:34:02.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.702+0000 7f5813fff640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f580006b500 0x7f580006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.702+0000 7f5813fff640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f580006b500 0x7f580006d9c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5804036000 tx=0x7f580403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.706+0000 7f5810ff9640 1 -- 172.21.15.77:0/1478701324 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5808058ee0 con 0x7f58140d29c0 2024-03-06T07:34:02.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.953+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58140cf5b0 con 0x7f58140d29c0 2024-03-06T07:34:02.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.953+0000 7f5810ff9640 1 -- 172.21.15.77:0/1478701324 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f58080587e0 con 0x7f58140d29c0 2024-03-06T07:34:02.955 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:02.955 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:34:02.955 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f580006b500 msgr2=0x7f580006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f580006b500 0x7f580006d9c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5804036000 tx=0x7f580403b040 comp rx=0 tx=0).stop 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58140d29c0 msgr2=0x7f58141f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58140d29c0 0x7f58141f4f60 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f580800c430 tx=0x7f580800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 shutdown_connections 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f580006b500 0x7f580006d9c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58140d29c0 0x7f58141f4f60 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f581415dfe0 0x7f58141f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 --2- 172.21.15.77:0/1478701324 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f58140cf830 0x7f5814154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.955+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 >> 172.21.15.77:0/1478701324 conn(0x7f58140d3cd0 msgr2=0x7f58140cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.956+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 shutdown_connections 2024-03-06T07:34:02.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:02.956+0000 7f581a35c640 1 -- 172.21.15.77:0/1478701324 wait complete. 2024-03-06T07:34:02.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:03.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.066+0000 7f1fd395e640 1 Processor -- start 2024-03-06T07:34:03.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.067+0000 7f1fd395e640 1 -- start start 2024-03-06T07:34:03.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.067+0000 7f1fd395e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.067+0000 7f1fd395e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1fcc0b41f0 0x7f1fcc0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.067+0000 7f1fd395e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc0b3430 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.067+0000 7f1fd395e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0d2c50 con 0x7f1fcc0b41f0 2024-03-06T07:34:03.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.067+0000 7f1fd395e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0b31e0 con 0x7f1fcc0d22b0 2024-03-06T07:34:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd1efb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1fcc0b41f0 0x7f1fcc0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd0ef9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd0ef9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:38118/0 (socket says 172.21.15.77:38118) 2024-03-06T07:34:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd1efb640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1fcc0b41f0 0x7f1fcc0b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:55488/0 (socket says 172.21.15.77:55488) 2024-03-06T07:34:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd0ef9640 1 -- 172.21.15.77:0/668314717 learned_addr learned my addr 172.21.15.77:0/668314717 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd16fa640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f1fcc15dfe0 0x7f1fcc0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:60270/0 (socket says 172.21.15.77:60270) 2024-03-06T07:34:03.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd1efb640 1 -- 172.21.15.77:0/668314717 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 msgr2=0x7f1fcc0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd1efb640 1 --2- 172.21.15.77:0/668314717 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd1efb640 1 -- 172.21.15.77:0/668314717 >> v1:172.21.15.158:6789/0 conn(0x7f1fcc15dfe0 legacy=0x7f1fcc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fbffff640 1 -- 172.21.15.77:0/668314717 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600598853 0 0) 0x7f1fcc0b3430 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd1efb640 1 -- 172.21.15.77:0/668314717 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc0b3710 con 0x7f1fcc0b41f0 2024-03-06T07:34:03.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.068+0000 7f1fd0ef9640 1 --2- 172.21.15.77:0/668314717 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:03.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.069+0000 7f1fd1efb640 1 --2- 172.21.15.77:0/668314717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 0x7f1fcc0b4650 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009d90 tx=0x7f1fc402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2141fcc7616d8bef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.069+0000 7f1fbffff640 1 -- 172.21.15.77:0/668314717 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4039070 con 0x7f1fcc0b41f0 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.069+0000 7f1fbffff640 1 -- 172.21.15.77:0/668314717 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc402cc50 con 0x7f1fcc0b41f0 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.069+0000 7f1fbffff640 1 -- 172.21.15.77:0/668314717 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4034440 con 0x7f1fcc0b41f0 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.070+0000 7f1fd395e640 1 -- 172.21.15.77:0/668314717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 msgr2=0x7f1fcc0b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.070+0000 7f1fd395e640 1 --2- 172.21.15.77:0/668314717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 0x7f1fcc0b4650 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009d90 tx=0x7f1fc402bea0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.070+0000 7f1fd395e640 1 -- 172.21.15.77:0/668314717 shutdown_connections 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.070+0000 7f1fd395e640 1 --2- 172.21.15.77:0/668314717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 0x7f1fcc0b4650 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.070+0000 7f1fd395e640 1 --2- 172.21.15.77:0/668314717 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.070+0000 7f1fd395e640 1 -- 172.21.15.77:0/668314717 >> 172.21.15.77:0/668314717 conn(0x7f1fcc0d3cd0 msgr2=0x7f1fcc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.070+0000 7f1fd395e640 1 -- 172.21.15.77:0/668314717 shutdown_connections 2024-03-06T07:34:03.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.070+0000 7f1fd395e640 1 -- 172.21.15.77:0/668314717 wait complete. 2024-03-06T07:34:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.072+0000 7f1fd395e640 1 Processor -- start 2024-03-06T07:34:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.072+0000 7f1fd395e640 1 -- start start 2024-03-06T07:34:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd395e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 0x7f1fcc15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd395e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd395e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fcc15dfe0 0x7f1fcc15c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd395e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0cfa00 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd395e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0b3d60 con 0x7f1fcc0b41f0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd395e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0d1c70 con 0x7f1fcc0d22b0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fcc15dfe0 0x7f1fcc15c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fcc15dfe0 0x7f1fcc15c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34306/0 (socket says 172.21.15.77:34306) 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 -- 172.21.15.77:0/3516603429 learned_addr learned my addr 172.21.15.77:0/3516603429 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd0ef9640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd16fa640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 0x7f1fcc15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 -- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 msgr2=0x7f1fcc0b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 -- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 msgr2=0x7f1fcc15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 0x7f1fcc15ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 -- 172.21.15.77:0/3516603429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc1f7da0 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd16fa640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 0x7f1fcc15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.073+0000 7f1fd1efb640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fcc15dfe0 0x7f1fcc15c5b0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009d90 tx=0x7f1fc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.074+0000 7f1fbdffb640 1 -- 172.21.15.77:0/3516603429 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4044020 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.074+0000 7f1fbdffb640 1 -- 172.21.15.77:0/3516603429 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc403a040 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.074+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc15e5d0 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.074+0000 7f1fbdffb640 1 -- 172.21.15.77:0/3516603429 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4039050 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.074+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc15eb90 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.075+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc15e3c0 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.075+0000 7f1fbdffb640 1 -- 172.21.15.77:0/3516603429 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fc4003990 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.076+0000 7f1fbdffb640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1fb006b4d0 0x7f1fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.076+0000 7f1fbdffb640 1 -- 172.21.15.77:0/3516603429 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1fc40ac570 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.076+0000 7f1fd16fa640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1fb006b4d0 0x7f1fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.076+0000 7f1fd16fa640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1fb006b4d0 0x7f1fb006d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0000f60 tx=0x7f1fc0007ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.080+0000 7f1fbdffb640 1 -- 172.21.15.77:0/3516603429 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fc404e060 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.281+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fcc04f330 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.282+0000 7f1fbdffb640 1 -- 172.21.15.77:0/3516603429 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16716 (secure 0 0 0) 0x7f1fc407b3e0 con 0x7f1fcc15dfe0 2024-03-06T07:34:03.284 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:03.285 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":143,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:49.223707+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1fb006b4d0 msgr2=0x7f1fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1fb006b4d0 0x7f1fb006d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0000f60 tx=0x7f1fc0007ca0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fcc15dfe0 msgr2=0x7f1fcc15c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fcc15dfe0 0x7f1fcc15c5b0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009d90 tx=0x7f1fc4004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 shutdown_connections 2024-03-06T07:34:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1fb006b4d0 0x7f1fb006d990 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fcc15dfe0 0x7f1fcc15c5b0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1fcc0d22b0 0x7f1fcc0b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 --2- 172.21.15.77:0/3516603429 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1fcc0b41f0 0x7f1fcc15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 >> 172.21.15.77:0/3516603429 conn(0x7f1fcc0d3cd0 msgr2=0x7f1fcc0d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 shutdown_connections 2024-03-06T07:34:03.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.284+0000 7f1fd395e640 1 -- 172.21.15.77:0/3516603429 wait complete. 2024-03-06T07:34:03.297 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:53.425208+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:03.674704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:04.414541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:03.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.401+0000 7fd4913b5640 1 Processor -- start 2024-03-06T07:34:03.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.401+0000 7fd4913b5640 1 -- start start 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd4913b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0d06e0 0x7fd48c15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd4913b5640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd48c0cf200 0x7fd48c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd4913b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c0d47f0 con 0x7fd48c0d0ac0 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd4913b5640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48c0ced70 con 0x7fd48c0cf200 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd4913b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0d1ab0 con 0x7fd48c0d06e0 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd48a7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd48c0d0ac0 0x7fd48c0b47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59614/0 (socket says 172.21.15.77:59614) 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd48a7fc640 1 -- 172.21.15.77:0/3805256601 learned_addr learned my addr 172.21.15.77:0/3805256601 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483667261 0 0) 0x7fd48c0d47f0 con 0x7fd48c0d0ac0 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd478003520 con 0x7fd48c0d0ac0 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd48affd640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0d06e0 0x7fd48c15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd48b7fe640 1 --2- 172.21.15.77:0/3805256601 >> v2:172.21.15.158:3300/0 conn(0x7fd48c0cf200 0x7fd48c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.402+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075274871 0 0) 0x7fd478003520 con 0x7fd48c0d0ac0 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0d06e0 msgr2=0x7fd48c15fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0d06e0 0x7fd48c15fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 >> v2:172.21.15.158:3300/0 conn(0x7fd48c0cf200 msgr2=0x7fd48c0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 --2- 172.21.15.77:0/3805256601 >> v2:172.21.15.158:3300/0 conn(0x7fd48c0cf200 0x7fd48c0cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c0d1d90 con 0x7fd48c0d0ac0 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fd474001670 con 0x7fd48c0d0ac0 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 >> v1:172.21.15.77:6789/0 conn(0x7fd48c0d0ac0 legacy=0x7fd48c0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0d06e0 0x7fd4780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478003fe0 0x7fd478004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd478004d40 0x7fd4780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0ced70 con 0x7fd478004d40 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0d1ab0 con 0x7fd478003fe0 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47800bac0 con 0x7fd48c0d06e0 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48b7fe640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd478004d40 0x7fd4780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48affd640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0d06e0 0x7fd4780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48a7fc640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478003fe0 0x7fd478004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48b7fe640 1 -- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0d06e0 msgr2=0x7fd4780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48b7fe640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0d06e0 0x7fd4780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48b7fe640 1 -- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478003fe0 msgr2=0x7fd478004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48b7fe640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478003fe0 0x7fd478004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48b7fe640 1 -- 172.21.15.77:0/3805256601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478003520 con 0x7fd478004d40 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.403+0000 7fd48a7fc640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478003fe0 0x7fd478004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.404+0000 7fd48b7fe640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd478004d40 0x7fd4780051c0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fd48400ba30 tx=0x7fd478005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b485bdde9feb45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.404+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd484036020 con 0x7fd478004d40 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.404+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd484032680 con 0x7fd478004d40 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.404+0000 7fd489ffb640 1 -- 172.21.15.77:0/3805256601 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48400b600 con 0x7fd478004d40 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.405+0000 7fd4913b5640 1 -- 172.21.15.77:0/3805256601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd478004d40 msgr2=0x7fd4780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.405+0000 7fd4913b5640 1 --2- 172.21.15.77:0/3805256601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd478004d40 0x7fd4780051c0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fd48400ba30 tx=0x7fd478005830 comp rx=0 tx=0).stop 2024-03-06T07:34:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.405+0000 7fd48b7fe640 1 -- 172.21.15.77:0/3805256601 reap_dead start 2024-03-06T07:34:03.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.405+0000 7fd4913b5640 1 -- 172.21.15.77:0/3805256601 shutdown_connections 2024-03-06T07:34:03.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.405+0000 7fd4913b5640 1 -- 172.21.15.77:0/3805256601 >> 172.21.15.77:0/3805256601 conn(0x7fd48c0d3cd0 msgr2=0x7fd48c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.405+0000 7fd4913b5640 1 -- 172.21.15.77:0/3805256601 shutdown_connections 2024-03-06T07:34:03.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.405+0000 7fd4913b5640 1 -- 172.21.15.77:0/3805256601 wait complete. 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.406+0000 7fd4913b5640 1 Processor -- start 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.406+0000 7fd4913b5640 1 -- start start 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4913b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478004d40 0x7fd48c0b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4913b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0b2b10 0x7fd48c0b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4913b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd48c0b0ac0 0x7fd48c0b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4913b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0b44d0 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4913b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0ce990 con 0x7fd478004d40 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4913b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0d0f00 con 0x7fd48c0b2b10 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd48c0b0ac0 0x7fd48c0b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd48c0b0ac0 0x7fd48c0b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34328/0 (socket says 172.21.15.77:34328) 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 -- 172.21.15.77:0/4222206545 learned_addr learned my addr 172.21.15.77:0/4222206545 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48affd640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478004d40 0x7fd48c0b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48a7fc640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0b2b10 0x7fd48c0b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 -- 172.21.15.77:0/4222206545 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0b2b10 msgr2=0x7fd48c0b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0b2b10 0x7fd48c0b2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 -- 172.21.15.77:0/4222206545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478004d40 msgr2=0x7fd48c0b93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478004d40 0x7fd48c0b93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 -- 172.21.15.77:0/4222206545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c0b1810 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd48b7fe640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd48c0b0ac0 0x7fd48c0b0f40 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fd48400bb50 tx=0x7fd484004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4837fe640 1 -- 172.21.15.77:0/4222206545 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd484041070 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4837fe640 1 -- 172.21.15.77:0/4222206545 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd484004020 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48c0b1af0 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4837fe640 1 -- 172.21.15.77:0/4222206545 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd484036040 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.407+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48c0b20b0 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.408+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48c0b7240 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.409+0000 7fd4837fe640 1 -- 172.21.15.77:0/4222206545 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd484039020 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.409+0000 7fd4837fe640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd46c06b500 0x7fd46c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.410+0000 7fd48affd640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd46c06b500 0x7fd46c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.410+0000 7fd4837fe640 1 -- 172.21.15.77:0/4222206545 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd4840abf50 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.410+0000 7fd48affd640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd46c06b500 0x7fd46c06d9c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0c9aa0 tx=0x7fd47c009760 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.414+0000 7fd4837fe640 1 -- 172.21.15.77:0/4222206545 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd48407ad20 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.587+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd48c0a1c80 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.587+0000 7fd4837fe640 1 -- 172.21.15.77:0/4222206545 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd48407a620 con 0x7fd48c0b0ac0 2024-03-06T07:34:03.589 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-06T07:34:03.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd46c06b500 msgr2=0x7fd46c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd46c06b500 0x7fd46c06d9c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0c9aa0 tx=0x7fd47c009760 comp rx=0 tx=0).stop 2024-03-06T07:34:03.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd48c0b0ac0 msgr2=0x7fd48c0b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd48c0b0ac0 0x7fd48c0b0f40 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fd48400bb50 tx=0x7fd484004730 comp rx=0 tx=0).stop 2024-03-06T07:34:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 shutdown_connections 2024-03-06T07:34:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd46c06b500 0x7fd46c06d9c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd48c0b0ac0 0x7fd48c0b0f40 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd48c0b2b10 0x7fd48c0b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 --2- 172.21.15.77:0/4222206545 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd478004d40 0x7fd48c0b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 >> 172.21.15.77:0/4222206545 conn(0x7fd48c0d3cd0 msgr2=0x7fd48c161450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 shutdown_connections 2024-03-06T07:34:03.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.589+0000 7fd4913b5640 1 -- 172.21.15.77:0/4222206545 wait complete. 2024-03-06T07:34:03.602 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:34:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.704+0000 7f8287edf640 1 Processor -- start 2024-03-06T07:34:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.704+0000 7f8287edf640 1 -- start start 2024-03-06T07:34:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f8287edf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f82801614d0 0x7f82800d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f8287edf640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82800d1910 0x7f82800d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f8287edf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82801618b0 con 0x7f82800d22e0 2024-03-06T07:34:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f8287edf640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82800cf400 con 0x7f82800d1910 2024-03-06T07:34:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f8287edf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82800d2710 con 0x7f82801614d0 2024-03-06T07:34:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f828547a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f82800d1910 0x7f82800d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f828647c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f82800d22e0 0x7f82800d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59622/0 (socket says 172.21.15.77:59622) 2024-03-06T07:34:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f828647c640 1 -- 172.21.15.77:0/807386336 learned_addr learned my addr 172.21.15.77:0/807386336 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f8285c7b640 1 --2- 172.21.15.77:0/807386336 >> v2:172.21.15.158:3300/0 conn(0x7f82801614d0 0x7f82800d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f828547a640 1 -- 172.21.15.77:0/807386336 >> v2:172.21.15.158:3300/0 conn(0x7f82801614d0 msgr2=0x7f82800d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f828547a640 1 --2- 172.21.15.77:0/807386336 >> v2:172.21.15.158:3300/0 conn(0x7f82801614d0 0x7f82800d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f828547a640 1 -- 172.21.15.77:0/807386336 >> v1:172.21.15.77:6789/0 conn(0x7f82800d22e0 legacy=0x7f82800d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f828547a640 1 -- 172.21.15.77:0/807386336 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82800cf6a0 con 0x7f82800d1910 2024-03-06T07:34:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.705+0000 7f8284c79640 1 -- 172.21.15.77:0/807386336 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389999317 0 0) 0x7f82801618b0 con 0x7f82800d22e0 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.706+0000 7f828547a640 1 --2- 172.21.15.77:0/807386336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d1910 0x7f82800d1da0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f828008c470 tx=0x7f827002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffe458d9d1889fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.706+0000 7f8284c79640 1 -- 172.21.15.77:0/807386336 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270035020 con 0x7f82800d1910 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.706+0000 7f8284c79640 1 -- 172.21.15.77:0/807386336 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8270031830 con 0x7f82800d1910 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.706+0000 7f8284c79640 1 -- 172.21.15.77:0/807386336 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270031c00 con 0x7f82800d1910 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.706+0000 7f8287edf640 1 -- 172.21.15.77:0/807386336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d1910 msgr2=0x7f82800d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.706+0000 7f8287edf640 1 --2- 172.21.15.77:0/807386336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d1910 0x7f82800d1da0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f828008c470 tx=0x7f827002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.707+0000 7f8287edf640 1 -- 172.21.15.77:0/807386336 shutdown_connections 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.707+0000 7f8287edf640 1 --2- 172.21.15.77:0/807386336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d1910 0x7f82800d1da0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.707+0000 7f8287edf640 1 --2- 172.21.15.77:0/807386336 >> v2:172.21.15.158:3300/0 conn(0x7f82801614d0 0x7f82800d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.707+0000 7f8287edf640 1 -- 172.21.15.77:0/807386336 >> 172.21.15.77:0/807386336 conn(0x7f82800d3d40 msgr2=0x7f82800d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.707+0000 7f8287edf640 1 -- 172.21.15.77:0/807386336 shutdown_connections 2024-03-06T07:34:03.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.707+0000 7f8287edf640 1 -- 172.21.15.77:0/807386336 wait complete. 2024-03-06T07:34:03.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.708+0000 7f8287edf640 1 Processor -- start 2024-03-06T07:34:03.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.708+0000 7f8287edf640 1 -- start start 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.708+0000 7f8287edf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82800d1910 0x7f828015be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.708+0000 7f8287edf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d22e0 0x7f82800b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.708+0000 7f8287edf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82801614d0 0x7f828015dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.708+0000 7f8287edf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800b0910 con 0x7f82800d22e0 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.708+0000 7f8287edf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800cfc60 con 0x7f82800d1910 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.708+0000 7f8287edf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82800b1e40 con 0x7f82801614d0 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d22e0 0x7f82800b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d22e0 0x7f82800b9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34340/0 (socket says 172.21.15.77:34340) 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 -- 172.21.15.77:0/2720637441 learned_addr learned my addr 172.21.15.77:0/2720637441 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f8285c7b640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82800d1910 0x7f828015be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828647c640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82801614d0 0x7f828015dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 -- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82801614d0 msgr2=0x7f828015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82801614d0 0x7f828015dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 -- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82800d1910 msgr2=0x7f828015be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82800d1910 0x7f828015be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 -- 172.21.15.77:0/2720637441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82800cb1d0 con 0x7f82800d22e0 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828647c640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82801614d0 0x7f828015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f828547a640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d22e0 0x7f82800b9190 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f827002bee0 tx=0x7f82700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f82767fc640 1 -- 172.21.15.77:0/2720637441 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270041070 con 0x7f82800d22e0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f82767fc640 1 -- 172.21.15.77:0/2720637441 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827003aa30 con 0x7f82800d22e0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82800cb4b0 con 0x7f82800d22e0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f82767fc640 1 -- 172.21.15.77:0/2720637441 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270035040 con 0x7f82800d22e0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.709+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82800cba70 con 0x7f82800d22e0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.710+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82800d1da0 con 0x7f82800d22e0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.711+0000 7f82767fc640 1 -- 172.21.15.77:0/2720637441 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8270038020 con 0x7f82800d22e0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.711+0000 7f82767fc640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f826406b4d0 0x7f826406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.711+0000 7f82767fc640 1 -- 172.21.15.77:0/2720637441 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f82700ab990 con 0x7f82800d22e0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.711+0000 7f8285c7b640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f826406b4d0 0x7f826406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.712+0000 7f8285c7b640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f826406b4d0 0x7f826406d990 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f827c006fd0 tx=0x7f827c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.716+0000 7f82767fc640 1 -- 172.21.15.77:0/2720637441 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f827007a880 con 0x7f82800d22e0 2024-03-06T07:34:03.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.938+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8280068ff0 con 0x7f82800d22e0 2024-03-06T07:34:03.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.938+0000 7f82767fc640 1 -- 172.21.15.77:0/2720637441 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f827007a180 con 0x7f82800d22e0 2024-03-06T07:34:03.940 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f826406b4d0 msgr2=0x7f826406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f826406b4d0 0x7f826406d990 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f827c006fd0 tx=0x7f827c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d22e0 msgr2=0x7f82800b9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d22e0 0x7f82800b9190 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f827002bee0 tx=0x7f82700046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 shutdown_connections 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f826406b4d0 0x7f826406d990 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82800d22e0 0x7f82800b9190 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f82801614d0 0x7f828015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 --2- 172.21.15.77:0/2720637441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f82800d1910 0x7f828015be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 >> 172.21.15.77:0/2720637441 conn(0x7f82800d3d40 msgr2=0x7f828006b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 shutdown_connections 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:03.940+0000 7f8287edf640 1 -- 172.21.15.77:0/2720637441 wait complete. 2024-03-06T07:34:03.953 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:04.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.055+0000 7fb65a53a640 1 Processor -- start 2024-03-06T07:34:04.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.055+0000 7fb65a53a640 1 -- start start 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.055+0000 7fb65a53a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fb6540b4970 0x7fb65406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.055+0000 7fb65a53a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb65406d5c0 0x7fb6540cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.055+0000 7fb65a53a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65415ff90 con 0x7fb65415bb40 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.055+0000 7fb65a53a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6540b4dd0 con 0x7fb65406d5c0 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.055+0000 7fb65a53a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65406c910 con 0x7fb6540b4970 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.055+0000 7fb658ad7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb65406d5c0 0x7fb6540cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.056+0000 7fb653fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb65415bb40 0x7fb65415f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59628/0 (socket says 172.21.15.77:59628) 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.056+0000 7fb653fff640 1 -- 172.21.15.77:0/4210172404 learned_addr learned my addr 172.21.15.77:0/4210172404 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.056+0000 7fb6537fe640 1 --2- 172.21.15.77:0/4210172404 >> v2:172.21.15.158:3300/0 conn(0x7fb6540b4970 0x7fb65406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.056+0000 7fb658ad7640 1 -- 172.21.15.77:0/4210172404 >> v2:172.21.15.158:3300/0 conn(0x7fb6540b4970 msgr2=0x7fb65406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.056+0000 7fb658ad7640 1 --2- 172.21.15.77:0/4210172404 >> v2:172.21.15.158:3300/0 conn(0x7fb6540b4970 0x7fb65406d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.056+0000 7fb658ad7640 1 -- 172.21.15.77:0/4210172404 >> v1:172.21.15.77:6789/0 conn(0x7fb65415bb40 legacy=0x7fb65415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.056+0000 7fb658ad7640 1 -- 172.21.15.77:0/4210172404 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb654160270 con 0x7fb65406d5c0 2024-03-06T07:34:04.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.056+0000 7fb652ffd640 1 -- 172.21.15.77:0/4210172404 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920319611 0 0) 0x7fb65415ff90 con 0x7fb65415bb40 2024-03-06T07:34:04.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb658ad7640 1 --2- 172.21.15.77:0/4210172404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65406d5c0 0x7fb6540cfa90 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fb65406bfe0 tx=0x7fb64c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ea7ee4b2b93f00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb652ffd640 1 -- 172.21.15.77:0/4210172404 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c039070 con 0x7fb65406d5c0 2024-03-06T07:34:04.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb652ffd640 1 -- 172.21.15.77:0/4210172404 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64c02cc80 con 0x7fb65406d5c0 2024-03-06T07:34:04.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb652ffd640 1 -- 172.21.15.77:0/4210172404 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c034660 con 0x7fb65406d5c0 2024-03-06T07:34:04.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb65a53a640 1 -- 172.21.15.77:0/4210172404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65406d5c0 msgr2=0x7fb6540cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb65a53a640 1 --2- 172.21.15.77:0/4210172404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65406d5c0 0x7fb6540cfa90 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fb65406bfe0 tx=0x7fb64c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb65a53a640 1 -- 172.21.15.77:0/4210172404 shutdown_connections 2024-03-06T07:34:04.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb65a53a640 1 --2- 172.21.15.77:0/4210172404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb65406d5c0 0x7fb6540cfa90 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb65a53a640 1 --2- 172.21.15.77:0/4210172404 >> v2:172.21.15.158:3300/0 conn(0x7fb6540b4970 0x7fb65406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.057+0000 7fb65a53a640 1 -- 172.21.15.77:0/4210172404 >> 172.21.15.77:0/4210172404 conn(0x7fb65406b190 msgr2=0x7fb65406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.058+0000 7fb65a53a640 1 -- 172.21.15.77:0/4210172404 shutdown_connections 2024-03-06T07:34:04.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.058+0000 7fb65a53a640 1 -- 172.21.15.77:0/4210172404 wait complete. 2024-03-06T07:34:04.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.059+0000 7fb65a53a640 1 Processor -- start 2024-03-06T07:34:04.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.059+0000 7fb65a53a640 1 -- start start 2024-03-06T07:34:04.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb65a53a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb65406d5c0 0x7fb6541403b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb65a53a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6540b4970 0x7fb65415d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb65a53a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65415bb40 0x7fb6540b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb65a53a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65415bf20 con 0x7fb6540b4970 2024-03-06T07:34:04.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb65a53a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65406cb60 con 0x7fb65415bb40 2024-03-06T07:34:04.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb65a53a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65415c0a0 con 0x7fb65406d5c0 2024-03-06T07:34:04.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6540b4970 0x7fb65415d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6540b4970 0x7fb65415d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34362/0 (socket says 172.21.15.77:34362) 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 -- 172.21.15.77:0/1142075497 learned_addr learned my addr 172.21.15.77:0/1142075497 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb653fff640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb65406d5c0 0x7fb6541403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb658ad7640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65415bb40 0x7fb6540b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 -- 172.21.15.77:0/1142075497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb65406d5c0 msgr2=0x7fb6541403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb65406d5c0 0x7fb6541403b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 -- 172.21.15.77:0/1142075497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65415bb40 msgr2=0x7fb6540b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65415bb40 0x7fb6540b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 -- 172.21.15.77:0/1142075497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6541408f0 con 0x7fb6540b4970 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb6537fe640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6540b4970 0x7fb65415d8c0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fb64800c7f0 tx=0x7fb64800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb650ff9640 1 -- 172.21.15.77:0/1142075497 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648015070 con 0x7fb6540b4970 2024-03-06T07:34:04.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb650ff9640 1 -- 172.21.15.77:0/1142075497 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb648004240 con 0x7fb6540b4970 2024-03-06T07:34:04.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6540b0f80 con 0x7fb6540b4970 2024-03-06T07:34:04.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb650ff9640 1 -- 172.21.15.77:0/1142075497 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648007620 con 0x7fb6540b4970 2024-03-06T07:34:04.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.061+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6540b1540 con 0x7fb6540b4970 2024-03-06T07:34:04.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.062+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6540732b0 con 0x7fb6540b4970 2024-03-06T07:34:04.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.063+0000 7fb650ff9640 1 -- 172.21.15.77:0/1142075497 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6480079d0 con 0x7fb6540b4970 2024-03-06T07:34:04.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.063+0000 7fb650ff9640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb64006b4d0 0x7fb64006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.063+0000 7fb650ff9640 1 -- 172.21.15.77:0/1142075497 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb64808a320 con 0x7fb6540b4970 2024-03-06T07:34:04.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.063+0000 7fb653fff640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb64006b4d0 0x7fb64006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.064+0000 7fb653fff640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb64006b4d0 0x7fb64006d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb644000f60 tx=0x7fb644007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.068+0000 7fb650ff9640 1 -- 172.21.15.77:0/1142075497 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6541408f0 con 0x7fb6540b4970 2024-03-06T07:34:04.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.247+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb654068ff0 con 0x7fb6540b4970 2024-03-06T07:34:04.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.248+0000 7fb650ff9640 1 -- 172.21.15.77:0/1142075497 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb648059180 con 0x7fb6540b4970 2024-03-06T07:34:04.250 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-06T07:34:04.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb64006b4d0 msgr2=0x7fb64006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb64006b4d0 0x7fb64006d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb644000f60 tx=0x7fb644007c00 comp rx=0 tx=0).stop 2024-03-06T07:34:04.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6540b4970 msgr2=0x7fb65415d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6540b4970 0x7fb65415d8c0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fb64800c7f0 tx=0x7fb64800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 shutdown_connections 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb64006b4d0 0x7fb64006d990 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6540b4970 0x7fb65415d8c0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb65415bb40 0x7fb6540b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 --2- 172.21.15.77:0/1142075497 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fb65406d5c0 0x7fb6541403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 >> 172.21.15.77:0/1142075497 conn(0x7fb65406b190 msgr2=0x7fb65415a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 shutdown_connections 2024-03-06T07:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.250+0000 7fb65a53a640 1 -- 172.21.15.77:0/1142075497 wait complete. 2024-03-06T07:34:04.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:04.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.361+0000 7ffa9cb63640 1 Processor -- start 2024-03-06T07:34:04.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.362+0000 7ffa9cb63640 1 -- start start 2024-03-06T07:34:04.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9cb63640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa9815dfe0 0x7ffa980cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9cb63640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa980cf830 0x7ffa980d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9cb63640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa980b3430 con 0x7ffa980d27f0 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9cb63640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa980d2c50 con 0x7ffa9815dfe0 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9cb63640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980b31e0 con 0x7ffa980cf830 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa9815dfe0 0x7ffa980cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa9815dfe0 0x7ffa980cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34370/0 (socket says 172.21.15.77:34370) 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9659c640 1 -- 172.21.15.77:0/3651831882 learned_addr learned my addr 172.21.15.77:0/3651831882 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa8ffff640 1 --2- 172.21.15.77:0/3651831882 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa980cf830 0x7ffa980d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9659c640 1 -- 172.21.15.77:0/3651831882 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa980cf830 msgr2=0x7ffa980d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9659c640 1 --2- 172.21.15.77:0/3651831882 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa980cf830 0x7ffa980d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9659c640 1 -- 172.21.15.77:0/3651831882 >> v1:172.21.15.158:6789/0 conn(0x7ffa980d27f0 legacy=0x7ffa980d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.363+0000 7ffa9659c640 1 -- 172.21.15.77:0/3651831882 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa980b3710 con 0x7ffa9815dfe0 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.364+0000 7ffa9659c640 1 --2- 172.21.15.77:0/3651831882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9815dfe0 0x7ffa980cf2f0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009870 tx=0x7ffa8802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbb42f1cb658ce1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.364+0000 7ffa95d9b640 1 -- 172.21.15.77:0/3651831882 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88034070 con 0x7ffa9815dfe0 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.364+0000 7ffa95d9b640 1 -- 172.21.15.77:0/3651831882 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa88030840 con 0x7ffa9815dfe0 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.364+0000 7ffa95d9b640 1 -- 172.21.15.77:0/3651831882 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88030c10 con 0x7ffa9815dfe0 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.365+0000 7ffa9cb63640 1 -- 172.21.15.77:0/3651831882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9815dfe0 msgr2=0x7ffa980cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.365+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/3651831882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9815dfe0 0x7ffa980cf2f0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009870 tx=0x7ffa8802b980 comp rx=0 tx=0).stop 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.365+0000 7ffa9cb63640 1 -- 172.21.15.77:0/3651831882 shutdown_connections 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.365+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/3651831882 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa980cf830 0x7ffa980d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.365+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/3651831882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa9815dfe0 0x7ffa980cf2f0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.365+0000 7ffa9cb63640 1 -- 172.21.15.77:0/3651831882 >> 172.21.15.77:0/3651831882 conn(0x7ffa980d3cd0 msgr2=0x7ffa980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.365+0000 7ffa9cb63640 1 -- 172.21.15.77:0/3651831882 shutdown_connections 2024-03-06T07:34:04.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.365+0000 7ffa9cb63640 1 -- 172.21.15.77:0/3651831882 wait complete. 2024-03-06T07:34:04.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.366+0000 7ffa9cb63640 1 Processor -- start 2024-03-06T07:34:04.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.366+0000 7ffa9cb63640 1 -- start start 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa9cb63640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa980cf830 0x7ffa9814af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa9cb63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa980d27f0 0x7ffa9815f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa9cb63640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa9815dfe0 0x7ffa98160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa9cb63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980d1c70 con 0x7ffa980d27f0 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa9cb63640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980b3d60 con 0x7ffa980cf830 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa9cb63640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980d1df0 con 0x7ffa9815dfe0 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa980d27f0 0x7ffa9815f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa980d27f0 0x7ffa9815f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34372/0 (socket says 172.21.15.77:34372) 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 -- 172.21.15.77:0/573534111 learned_addr learned my addr 172.21.15.77:0/573534111 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa9659c640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa980cf830 0x7ffa9814af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa96d9d640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa9815dfe0 0x7ffa98160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 -- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa9815dfe0 msgr2=0x7ffa98160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa9815dfe0 0x7ffa98160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 -- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa980cf830 msgr2=0x7ffa9814af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa980cf830 0x7ffa9814af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 -- 172.21.15.77:0/573534111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9815a330 con 0x7ffa980d27f0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa9659c640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa980cf830 0x7ffa9814af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8ffff640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa980d27f0 0x7ffa9815f530 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000d7d0 tx=0x7ffa8000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.367+0000 7ffa8effd640 1 -- 172.21.15.77:0/573534111 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000b6c0 con 0x7ffa980d27f0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.368+0000 7ffa8effd640 1 -- 172.21.15.77:0/573534111 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa80019070 con 0x7ffa980d27f0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.368+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa980cda60 con 0x7ffa980d27f0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.368+0000 7ffa8effd640 1 -- 172.21.15.77:0/573534111 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000ba80 con 0x7ffa980d27f0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.368+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa980ce020 con 0x7ffa980d27f0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.368+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa980cde50 con 0x7ffa980d27f0 2024-03-06T07:34:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.369+0000 7ffa8effd640 1 -- 172.21.15.77:0/573534111 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa8000bc20 con 0x7ffa980d27f0 2024-03-06T07:34:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.369+0000 7ffa8effd640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa7806b4d0 0x7ffa7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.370+0000 7ffa9659c640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa7806b4d0 0x7ffa7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.370+0000 7ffa8effd640 1 -- 172.21.15.77:0/573534111 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ffa8008a680 con 0x7ffa980d27f0 2024-03-06T07:34:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.370+0000 7ffa9659c640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa7806b4d0 0x7ffa7806d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ffa88036000 tx=0x7ffa8803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.374+0000 7ffa8effd640 1 -- 172.21.15.77:0/573534111 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa8001c360 con 0x7ffa980d27f0 2024-03-06T07:34:04.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.623+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa980cdc30 con 0x7ffa980d27f0 2024-03-06T07:34:04.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.623+0000 7ffa8effd640 1 -- 172.21.15.77:0/573534111 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7ffa800594e0 con 0x7ffa980d27f0 2024-03-06T07:34:04.626 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:04.626 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:34:04.626 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa7806b4d0 msgr2=0x7ffa7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa7806b4d0 0x7ffa7806d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ffa88036000 tx=0x7ffa8803b040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa980d27f0 msgr2=0x7ffa9815f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa980d27f0 0x7ffa9815f530 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000d7d0 tx=0x7ffa8000dc90 comp rx=0 tx=0).stop 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 shutdown_connections 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7ffa7806b4d0 0x7ffa7806d990 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa980d27f0 0x7ffa9815f530 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7ffa9815dfe0 0x7ffa98160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 --2- 172.21.15.77:0/573534111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ffa980cf830 0x7ffa9814af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.625+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 >> 172.21.15.77:0/573534111 conn(0x7ffa980d3cd0 msgr2=0x7ffa9806d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.626+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 shutdown_connections 2024-03-06T07:34:04.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.626+0000 7ffa9cb63640 1 -- 172.21.15.77:0/573534111 wait complete. 2024-03-06T07:34:04.638 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:04.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.737+0000 7f1b8bf9a640 1 Processor -- start 2024-03-06T07:34:04.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.737+0000 7f1b8bf9a640 1 -- start start 2024-03-06T07:34:04.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.738+0000 7f1b8bf9a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f1b8415dfe0 0x7f1b840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.738+0000 7f1b8bf9a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b840d29c0 0x7f1b840b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.738+0000 7f1b8bf9a640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b840b3430 con 0x7f1b840cf830 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.738+0000 7f1b8bf9a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840b31e0 con 0x7f1b840d29c0 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.738+0000 7f1b8bf9a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840b3d60 con 0x7f1b8415dfe0 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.738+0000 7f1b8a537640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b840d29c0 0x7f1b840b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.738+0000 7f1b8a537640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b840d29c0 0x7f1b840b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34386/0 (socket says 172.21.15.77:34386) 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.738+0000 7f1b8a537640 1 -- 172.21.15.77:0/790455785 learned_addr learned my addr 172.21.15.77:0/790455785 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.739+0000 7f1b89d36640 1 --2- 172.21.15.77:0/790455785 >> v2:172.21.15.158:3300/0 conn(0x7f1b8415dfe0 0x7f1b840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.739+0000 7f1b8a537640 1 -- 172.21.15.77:0/790455785 >> v2:172.21.15.158:3300/0 conn(0x7f1b8415dfe0 msgr2=0x7f1b840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.739+0000 7f1b8a537640 1 --2- 172.21.15.77:0/790455785 >> v2:172.21.15.158:3300/0 conn(0x7f1b8415dfe0 0x7f1b840cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.739+0000 7f1b8a537640 1 -- 172.21.15.77:0/790455785 >> v1:172.21.15.158:6789/0 conn(0x7f1b840cf830 legacy=0x7f1b840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.739+0000 7f1b8a537640 1 -- 172.21.15.77:0/790455785 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840b3710 con 0x7f1b840d29c0 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.739+0000 7f1b8a537640 1 --2- 172.21.15.77:0/790455785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b840d29c0 0x7f1b840b41f0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f1b84067990 tx=0x7f1b8002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b702e46661b9420b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b88d34640 1 -- 172.21.15.77:0/790455785 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8002ca80 con 0x7f1b840d29c0 2024-03-06T07:34:04.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b88d34640 1 -- 172.21.15.77:0/790455785 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8002cc20 con 0x7f1b840d29c0 2024-03-06T07:34:04.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b88d34640 1 -- 172.21.15.77:0/790455785 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b800343f0 con 0x7f1b840d29c0 2024-03-06T07:34:04.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/790455785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b840d29c0 msgr2=0x7f1b840b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/790455785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b840d29c0 0x7f1b840b41f0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f1b84067990 tx=0x7f1b8002bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/790455785 shutdown_connections 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/790455785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b840d29c0 0x7f1b840b41f0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/790455785 >> v2:172.21.15.158:3300/0 conn(0x7f1b8415dfe0 0x7f1b840cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/790455785 >> 172.21.15.77:0/790455785 conn(0x7f1b840d3cd0 msgr2=0x7f1b840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/790455785 shutdown_connections 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.740+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/790455785 wait complete. 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.741+0000 7f1b8bf9a640 1 Processor -- start 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.741+0000 7f1b8bf9a640 1 -- start start 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8bf9a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b840cf830 0x7f1b840cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8bf9a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b840d29c0 0x7f1b840cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8bf9a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b8415dfe0 0x7f1b8406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8bf9a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840d1df0 con 0x7f1b8415dfe0 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8bf9a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840d1c70 con 0x7f1b840cf830 2024-03-06T07:34:04.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8bf9a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8415e3c0 con 0x7f1b840d29c0 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8a537640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b8415dfe0 0x7f1b8406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8a537640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b8415dfe0 0x7f1b8406ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34400/0 (socket says 172.21.15.77:34400) 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8a537640 1 -- 172.21.15.77:0/1166610227 learned_addr learned my addr 172.21.15.77:0/1166610227 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8a537640 1 -- 172.21.15.77:0/1166610227 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b840d29c0 msgr2=0x7f1b840cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8a537640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b840d29c0 0x7f1b840cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8a537640 1 -- 172.21.15.77:0/1166610227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b840cf830 msgr2=0x7f1b840cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8a537640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b840cf830 0x7f1b840cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.742+0000 7f1b8a537640 1 -- 172.21.15.77:0/1166610227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84159370 con 0x7f1b8415dfe0 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.743+0000 7f1b8a537640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b8415dfe0 0x7f1b8406ccb0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f1b80036040 tx=0x7f1b80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.743+0000 7f1b7a7fc640 1 -- 172.21.15.77:0/1166610227 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80043020 con 0x7f1b8415dfe0 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.743+0000 7f1b7a7fc640 1 -- 172.21.15.77:0/1166610227 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8003e450 con 0x7f1b8415dfe0 2024-03-06T07:34:04.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.743+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b840cbd80 con 0x7f1b8415dfe0 2024-03-06T07:34:04.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.743+0000 7f1b7a7fc640 1 -- 172.21.15.77:0/1166610227 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8003e860 con 0x7f1b8415dfe0 2024-03-06T07:34:04.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.743+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b840cdd60 con 0x7f1b8415dfe0 2024-03-06T07:34:04.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.744+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8404f330 con 0x7f1b8415dfe0 2024-03-06T07:34:04.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.745+0000 7f1b7a7fc640 1 -- 172.21.15.77:0/1166610227 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b80046020 con 0x7f1b8415dfe0 2024-03-06T07:34:04.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.745+0000 7f1b7a7fc640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b6806b4d0 0x7f1b6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.745+0000 7f1b7a7fc640 1 -- 172.21.15.77:0/1166610227 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1b800acb20 con 0x7f1b8415dfe0 2024-03-06T07:34:04.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.745+0000 7f1b89d36640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b6806b4d0 0x7f1b6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.746+0000 7f1b89d36640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b6806b4d0 0x7f1b6806d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00a810 tx=0x7f1b7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.753+0000 7f1b7a7fc640 1 -- 172.21.15.77:0/1166610227 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b8004d030 con 0x7f1b8415dfe0 2024-03-06T07:34:04.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.932+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b840d2630 con 0x7f1b8415dfe0 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.934+0000 7f1b7a7fc640 1 -- 172.21.15.77:0/1166610227 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16716 (secure 0 0 0) 0x7f1b8007b9f0 con 0x7f1b8415dfe0 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":143,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:49.223707+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:04.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b6806b4d0 msgr2=0x7f1b6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b6806b4d0 0x7f1b6806d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00a810 tx=0x7f1b7c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b8415dfe0 msgr2=0x7f1b8406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b8415dfe0 0x7f1b8406ccb0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f1b80036040 tx=0x7f1b80004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 shutdown_connections 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1b6806b4d0 0x7f1b6806d990 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b8415dfe0 0x7f1b8406ccb0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1b840d29c0 0x7f1b840cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 --2- 172.21.15.77:0/1166610227 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1b840cf830 0x7f1b840cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 >> 172.21.15.77:0/1166610227 conn(0x7f1b840d3cd0 msgr2=0x7f1b840b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 shutdown_connections 2024-03-06T07:34:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:04.936+0000 7f1b8bf9a640 1 -- 172.21.15.77:0/1166610227 wait complete. 2024-03-06T07:34:04.950 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:34:04.950 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:05.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.052+0000 7f9a791ac640 1 Processor -- start 2024-03-06T07:34:05.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.052+0000 7f9a791ac640 1 -- start start 2024-03-06T07:34:05.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a791ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a7415dfe0 0x7f9a740cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a791ac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9a740cf830 0x7f9a740d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a791ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a740d27f0 0x7f9a740d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a791ac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740d2c50 con 0x7f9a740d27f0 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a791ac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740b31e0 con 0x7f9a740cf830 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a791ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740b3d60 con 0x7f9a7415dfe0 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a7359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a740d27f0 0x7f9a740d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a7359e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a740d27f0 0x7f9a740d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34406/0 (socket says 172.21.15.77:34406) 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a7359e640 1 -- 172.21.15.77:0/259304224 learned_addr learned my addr 172.21.15.77:0/259304224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a72d9d640 1 --2- 172.21.15.77:0/259304224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a7415dfe0 0x7f9a740cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.053+0000 7f9a7259c640 1 --2- 172.21.15.77:0/259304224 >> v2:172.21.15.158:3300/0 conn(0x7f9a740cf830 0x7f9a740d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.054+0000 7f9a7359e640 1 -- 172.21.15.77:0/259304224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a7415dfe0 msgr2=0x7f9a740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.054+0000 7f9a7359e640 1 --2- 172.21.15.77:0/259304224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a7415dfe0 0x7f9a740cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.054+0000 7f9a7359e640 1 -- 172.21.15.77:0/259304224 >> v2:172.21.15.158:3300/0 conn(0x7f9a740cf830 msgr2=0x7f9a740d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.054+0000 7f9a7359e640 1 --2- 172.21.15.77:0/259304224 >> v2:172.21.15.158:3300/0 conn(0x7f9a740cf830 0x7f9a740d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.054+0000 7f9a7359e640 1 -- 172.21.15.77:0/259304224 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740b3430 con 0x7f9a740d27f0 2024-03-06T07:34:05.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.054+0000 7f9a72d9d640 1 --2- 172.21.15.77:0/259304224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a7415dfe0 0x7f9a740cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:05.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.054+0000 7f9a7359e640 1 --2- 172.21.15.77:0/259304224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a740d27f0 0x7f9a740d0300 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f9a7408cba0 tx=0x7f9a6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8e295bb6436d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a71d9b640 1 -- 172.21.15.77:0/259304224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c02ca70 con 0x7f9a740d27f0 2024-03-06T07:34:05.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a71d9b640 1 -- 172.21.15.77:0/259304224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c02cc10 con 0x7f9a740d27f0 2024-03-06T07:34:05.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a71d9b640 1 -- 172.21.15.77:0/259304224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0343f0 con 0x7f9a740d27f0 2024-03-06T07:34:05.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 -- 172.21.15.77:0/259304224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a740d27f0 msgr2=0x7f9a740d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 --2- 172.21.15.77:0/259304224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a740d27f0 0x7f9a740d0300 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f9a7408cba0 tx=0x7f9a6c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 -- 172.21.15.77:0/259304224 shutdown_connections 2024-03-06T07:34:05.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 --2- 172.21.15.77:0/259304224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a740d27f0 0x7f9a740d0300 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 --2- 172.21.15.77:0/259304224 >> v2:172.21.15.158:3300/0 conn(0x7f9a740cf830 0x7f9a740d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 --2- 172.21.15.77:0/259304224 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a7415dfe0 0x7f9a740cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 -- 172.21.15.77:0/259304224 >> 172.21.15.77:0/259304224 conn(0x7f9a740d3cd0 msgr2=0x7f9a740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 -- 172.21.15.77:0/259304224 shutdown_connections 2024-03-06T07:34:05.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.055+0000 7f9a791ac640 1 -- 172.21.15.77:0/259304224 wait complete. 2024-03-06T07:34:05.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.057+0000 7f9a791ac640 1 Processor -- start 2024-03-06T07:34:05.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.057+0000 7f9a791ac640 1 -- start start 2024-03-06T07:34:05.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a791ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a740cf830 0x7f9a74161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a791ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a740d27f0 0x7f9a740b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a791ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7415dfe0 0x7f9a74149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a791ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740d1c70 con 0x7f9a7415dfe0 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a791ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740b39f0 con 0x7f9a740cf830 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a791ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740d1df0 con 0x7f9a740d27f0 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a7359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7415dfe0 0x7f9a74149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a72d9d640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a740cf830 0x7f9a74161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a7359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7415dfe0 0x7f9a74149f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34420/0 (socket says 172.21.15.77:34420) 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.058+0000 7f9a7359e640 1 -- 172.21.15.77:0/3459692760 learned_addr learned my addr 172.21.15.77:0/3459692760 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a7259c640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a740d27f0 0x7f9a740b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a72d9d640 1 -- 172.21.15.77:0/3459692760 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a740d27f0 msgr2=0x7f9a740b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a72d9d640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a740d27f0 0x7f9a740b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a72d9d640 1 -- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7415dfe0 msgr2=0x7f9a74149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a72d9d640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7415dfe0 0x7f9a74149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a72d9d640 1 -- 172.21.15.77:0/3459692760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740b8ee0 con 0x7f9a740cf830 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a7359e640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7415dfe0 0x7f9a74149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a72d9d640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a740cf830 0x7f9a74161190 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400d5c0 tx=0x7f9a6400da80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a5b7fe640 1 -- 172.21.15.77:0/3459692760 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64004030 con 0x7f9a740cf830 2024-03-06T07:34:05.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a5b7fe640 1 -- 172.21.15.77:0/3459692760 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a640041d0 con 0x7f9a740cf830 2024-03-06T07:34:05.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a740b91c0 con 0x7f9a740cf830 2024-03-06T07:34:05.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a5b7fe640 1 -- 172.21.15.77:0/3459692760 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64014620 con 0x7f9a740cf830 2024-03-06T07:34:05.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.059+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a740b9780 con 0x7f9a740cf830 2024-03-06T07:34:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.060+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7404f330 con 0x7f9a740cf830 2024-03-06T07:34:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.061+0000 7f9a5b7fe640 1 -- 172.21.15.77:0/3459692760 <== mon.1 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a640147c0 con 0x7f9a740cf830 2024-03-06T07:34:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.061+0000 7f9a5b7fe640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a6006b4d0 0x7f9a6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.061+0000 7f9a5b7fe640 1 -- 172.21.15.77:0/3459692760 <== mon.1 v2:172.21.15.158:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9a6408a860 con 0x7f9a740cf830 2024-03-06T07:34:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.061+0000 7f9a7259c640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a6006b4d0 0x7f9a6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.062+0000 7f9a7259c640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a6006b4d0 0x7f9a6006d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f9a7414aff0 tx=0x7f9a68009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.066+0000 7f9a5b7fe640 1 -- 172.21.15.77:0/3459692760 <== mon.1 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a64058ff0 con 0x7f9a740cf830 2024-03-06T07:34:05.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.365+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a7415d4c0 con 0x7f9a740cf830 2024-03-06T07:34:05.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.365+0000 7f9a5b7fe640 1 -- 172.21.15.77:0/3459692760 <== mon.1 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f9a6405d090 con 0x7f9a740cf830 2024-03-06T07:34:05.367 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:05.368 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:34:05.368 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:34:05.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a6006b4d0 msgr2=0x7f9a6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a6006b4d0 0x7f9a6006d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f9a7414aff0 tx=0x7f9a68009190 comp rx=0 tx=0).stop 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a740cf830 msgr2=0x7f9a74161190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a740cf830 0x7f9a74161190 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400d5c0 tx=0x7f9a6400da80 comp rx=0 tx=0).stop 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 shutdown_connections 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9a6006b4d0 0x7f9a6006d990 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a7415dfe0 0x7f9a74149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9a740d27f0 0x7f9a740b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 --2- 172.21.15.77:0/3459692760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9a740cf830 0x7f9a74161190 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 >> 172.21.15.77:0/3459692760 conn(0x7f9a740d3cd0 msgr2=0x7f9a7406cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 shutdown_connections 2024-03-06T07:34:05.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.367+0000 7f9a791ac640 1 -- 172.21.15.77:0/3459692760 wait complete. 2024-03-06T07:34:05.381 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.480+0000 7f86a1a4e640 1 Processor -- start 2024-03-06T07:34:05.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.480+0000 7f86a1a4e640 1 -- start start 2024-03-06T07:34:05.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.481+0000 7f86a1a4e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f869c15bdd0 0x7f869c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.481+0000 7f86a1a4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c15fa10 0x7f869c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.481+0000 7f86a1a4e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c0cebe0 con 0x7f869c0d16e0 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.481+0000 7f86a1a4e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c0ceec0 con 0x7f869c15bdd0 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.481+0000 7f86a1a4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0cf1a0 con 0x7f869c15fa10 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.481+0000 7f869b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f869c0d16e0 0x7f869c0d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59632/0 (socket says 172.21.15.77:59632) 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.481+0000 7f869b7fe640 1 -- 172.21.15.77:0/3960140790 learned_addr learned my addr 172.21.15.77:0/3960140790 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423976748 0 0) 0x7f869c0cebe0 con 0x7f869c0d16e0 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f869affd640 1 --2- 172.21.15.77:0/3960140790 >> v2:172.21.15.158:3300/0 conn(0x7f869c15bdd0 0x7f869c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f869a7fc640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c15fa10 0x7f869c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8684003520 con 0x7f869c0d16e0 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 512039606 0 0) 0x7f8684003520 con 0x7f869c0d16e0 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c15fa10 msgr2=0x7f869c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c15fa10 0x7f869c0ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 >> v2:172.21.15.158:3300/0 conn(0x7f869c15bdd0 msgr2=0x7f869c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 --2- 172.21.15.77:0/3960140790 >> v2:172.21.15.158:3300/0 conn(0x7f869c15bdd0 0x7f869c0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0cc8e0 con 0x7f869c0d16e0 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f868c0019f0 con 0x7f869c0d16e0 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 >> v1:172.21.15.77:6789/0 conn(0x7f869c0d16e0 legacy=0x7f869c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86840039b0 0x7f8684003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c15fa10 0x7f8684004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8684004d40 0x7f86840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0ceec0 con 0x7f8684004d40 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0cf1a0 con 0x7f86840039b0 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868400b9f0 con 0x7f869c15fa10 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f869b7fe640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8684004d40 0x7f86840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f869affd640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86840039b0 0x7f8684003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f869a7fc640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c15fa10 0x7f8684004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f869b7fe640 1 -- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c15fa10 msgr2=0x7f8684004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f869b7fe640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c15fa10 0x7f8684004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.482+0000 7f869b7fe640 1 -- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86840039b0 msgr2=0x7f8684003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.483+0000 7f869b7fe640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86840039b0 0x7f8684003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.483+0000 7f869b7fe640 1 -- 172.21.15.77:0/3960140790 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8684003520 con 0x7f8684004d40 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.483+0000 7f869affd640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f86840039b0 0x7f8684003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.483+0000 7f869b7fe640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8684004d40 0x7f86840051c0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f868c0013d0 tx=0x7f8684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf62d398250e2e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.483+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c00a4e0 con 0x7f8684004d40 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.483+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c034650 con 0x7f8684004d40 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.483+0000 7f8699ffb640 1 -- 172.21.15.77:0/3960140790 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c034a40 con 0x7f8684004d40 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.484+0000 7f86a1a4e640 1 -- 172.21.15.77:0/3960140790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8684004d40 msgr2=0x7f86840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.484+0000 7f86a1a4e640 1 --2- 172.21.15.77:0/3960140790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8684004d40 0x7f86840051c0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f868c0013d0 tx=0x7f8684005830 comp rx=0 tx=0).stop 2024-03-06T07:34:05.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.484+0000 7f869b7fe640 1 -- 172.21.15.77:0/3960140790 reap_dead start 2024-03-06T07:34:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.484+0000 7f86a1a4e640 1 -- 172.21.15.77:0/3960140790 shutdown_connections 2024-03-06T07:34:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.484+0000 7f86a1a4e640 1 -- 172.21.15.77:0/3960140790 >> 172.21.15.77:0/3960140790 conn(0x7f869c09a9f0 msgr2=0x7f869c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.484+0000 7f86a1a4e640 1 -- 172.21.15.77:0/3960140790 shutdown_connections 2024-03-06T07:34:05.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.484+0000 7f86a1a4e640 1 -- 172.21.15.77:0/3960140790 wait complete. 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.485+0000 7f86a1a4e640 1 Processor -- start 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.485+0000 7f86a1a4e640 1 -- start start 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f86a1a4e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c147840 0x7f869c0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f86a1a4e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c0cb230 0x7f869c1481b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f86a1a4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c147c20 0x7f869c1486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f86a1a4e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c15c1b0 con 0x7f869c147c20 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f86a1a4e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0cc5d0 con 0x7f869c147840 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f86a1a4e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0ce270 con 0x7f869c0cb230 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c147c20 0x7f869c1486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c147c20 0x7f869c1486f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34440/0 (socket says 172.21.15.77:34440) 2024-03-06T07:34:05.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869b7fe640 1 -- 172.21.15.77:0/593577717 learned_addr learned my addr 172.21.15.77:0/593577717 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869affd640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c147840 0x7f869c0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869a7fc640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c0cb230 0x7f869c1481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869b7fe640 1 -- 172.21.15.77:0/593577717 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c0cb230 msgr2=0x7f869c1481b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869b7fe640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c0cb230 0x7f869c1481b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869b7fe640 1 -- 172.21.15.77:0/593577717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c147840 msgr2=0x7f869c0cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869b7fe640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c147840 0x7f869c0cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.486+0000 7f869b7fe640 1 -- 172.21.15.77:0/593577717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c15c460 con 0x7f869c147c20 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.487+0000 7f869b7fe640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c147c20 0x7f869c1486f0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f868c00a0b0 tx=0x7f868c001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.487+0000 7f8683fff640 1 -- 172.21.15.77:0/593577717 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c039040 con 0x7f869c147c20 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.487+0000 7f8683fff640 1 -- 172.21.15.77:0/593577717 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c045070 con 0x7f869c147c20 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.487+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c15c740 con 0x7f869c147c20 2024-03-06T07:34:05.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.487+0000 7f8683fff640 1 -- 172.21.15.77:0/593577717 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694009690 con 0x7f869c147c20 2024-03-06T07:34:05.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.487+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c158510 con 0x7f869c147c20 2024-03-06T07:34:05.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.488+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c0ca5f0 con 0x7f869c147c20 2024-03-06T07:34:05.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.488+0000 7f8683fff640 1 -- 172.21.15.77:0/593577717 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f868c034a40 con 0x7f869c147c20 2024-03-06T07:34:05.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.489+0000 7f8683fff640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.489+0000 7f8683fff640 1 -- 172.21.15.77:0/593577717 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f868c0ac3e0 con 0x7f869c147c20 2024-03-06T07:34:05.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.489+0000 7f869affd640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.489+0000 7f869affd640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f868806b500 0x7f868806d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8694009660 tx=0x7f8694009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.493+0000 7f8683fff640 1 -- 172.21.15.77:0/593577717 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f869c0ca5f0 con 0x7f869c147c20 2024-03-06T07:34:05.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.708+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f869c04f330 con 0x7f869c147c20 2024-03-06T07:34:05.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.708+0000 7f8683fff640 1 -- 172.21.15.77:0/593577717 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3463 (secure 0 0 0) 0x7f868c07b390 con 0x7f869c147c20 2024-03-06T07:34:05.710 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:05.710 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:05.435981+0000","modified":"2024-03-06T07:32:35.768648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9186,"mds_1":9177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9177":{"gid":9177,"name":"a","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.77:6835/669492439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":669492439},{"type":"v1","addr":"172.21.15.77:6835","nonce":669492439}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:21.073903+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9186":{"gid":9186,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/2581202217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2581202217},{"type":"v1","addr":"172.21.15.77:6837","nonce":2581202217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9186,"qdb_cluster":[9186]},"id":10}]} 2024-03-06T07:34:05.711 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 107 2024-03-06T07:34:05.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f868806b500 msgr2=0x7f868806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f868806b500 0x7f868806d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8694009660 tx=0x7f8694009190 comp rx=0 tx=0).stop 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c147c20 msgr2=0x7f869c1486f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c147c20 0x7f869c1486f0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f868c00a0b0 tx=0x7f868c001360 comp rx=0 tx=0).stop 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 shutdown_connections 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c147c20 0x7f869c1486f0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f869c0cb230 0x7f869c1481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 --2- 172.21.15.77:0/593577717 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c147840 0x7f869c0cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 >> 172.21.15.77:0/593577717 conn(0x7f869c09a9f0 msgr2=0x7f869c15fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 shutdown_connections 2024-03-06T07:34:05.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.710+0000 7f86a1a4e640 1 -- 172.21.15.77:0/593577717 wait complete. 2024-03-06T07:34:05.723 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:05.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.857+0000 7f6474adf640 1 Processor -- start 2024-03-06T07:34:05.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.857+0000 7f6474adf640 1 -- start start 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.858+0000 7f6474adf640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f647015dfe0 0x7f64700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.858+0000 7f6474adf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700cf830 0x7f64700d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.858+0000 7f6474adf640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64700b3430 con 0x7f64700d27f0 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.858+0000 7f6474adf640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64700d2c50 con 0x7f647015dfe0 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.858+0000 7f6474adf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700b31e0 con 0x7f64700cf830 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.858+0000 7f646ed9d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f64700d27f0 0x7f64700d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59646/0 (socket says 172.21.15.77:59646) 2024-03-06T07:34:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.858+0000 7f646ed9d640 1 -- 172.21.15.77:0/534234640 learned_addr learned my addr 172.21.15.77:0/534234640 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f6467fff640 1 --2- 172.21.15.77:0/534234640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700cf830 0x7f64700d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f646e59c640 1 --2- 172.21.15.77:0/534234640 >> v2:172.21.15.158:3300/0 conn(0x7f647015dfe0 0x7f64700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f646dd9b640 1 -- 172.21.15.77:0/534234640 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614558955 0 0) 0x7f64700b3430 con 0x7f64700d27f0 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f646dd9b640 1 -- 172.21.15.77:0/534234640 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645c003520 con 0x7f64700d27f0 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f6467fff640 1 -- 172.21.15.77:0/534234640 >> v2:172.21.15.158:3300/0 conn(0x7f647015dfe0 msgr2=0x7f64700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f6467fff640 1 --2- 172.21.15.77:0/534234640 >> v2:172.21.15.158:3300/0 conn(0x7f647015dfe0 0x7f64700cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f646dd9b640 1 -- 172.21.15.77:0/534234640 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885995007 0 0) 0x7f645c003520 con 0x7f64700d27f0 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f6467fff640 1 -- 172.21.15.77:0/534234640 >> v1:172.21.15.77:6789/0 conn(0x7f64700d27f0 legacy=0x7f64700d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f6467fff640 1 -- 172.21.15.77:0/534234640 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700b3710 con 0x7f64700cf830 2024-03-06T07:34:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.859+0000 7f646e59c640 1 --2- 172.21.15.77:0/534234640 >> v2:172.21.15.158:3300/0 conn(0x7f647015dfe0 0x7f64700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.860+0000 7f6467fff640 1 --2- 172.21.15.77:0/534234640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700cf830 0x7f64700d22b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f64580097a0 tx=0x7f645802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc14364f55dff98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.860+0000 7f646dd9b640 1 -- 172.21.15.77:0/534234640 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458037070 con 0x7f64700cf830 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.860+0000 7f646dd9b640 1 -- 172.21.15.77:0/534234640 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6458030510 con 0x7f64700cf830 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.860+0000 7f646dd9b640 1 -- 172.21.15.77:0/534234640 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64680019f0 con 0x7f64700cf830 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.861+0000 7f6474adf640 1 -- 172.21.15.77:0/534234640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700cf830 msgr2=0x7f64700d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.861+0000 7f6474adf640 1 --2- 172.21.15.77:0/534234640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700cf830 0x7f64700d22b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f64580097a0 tx=0x7f645802b790 comp rx=0 tx=0).stop 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.861+0000 7f6474adf640 1 -- 172.21.15.77:0/534234640 shutdown_connections 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.861+0000 7f6474adf640 1 --2- 172.21.15.77:0/534234640 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f64700cf830 0x7f64700d22b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.861+0000 7f6474adf640 1 --2- 172.21.15.77:0/534234640 >> v2:172.21.15.158:3300/0 conn(0x7f647015dfe0 0x7f64700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.861+0000 7f6474adf640 1 -- 172.21.15.77:0/534234640 >> 172.21.15.77:0/534234640 conn(0x7f64700d3cd0 msgr2=0x7f64700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.861+0000 7f6474adf640 1 -- 172.21.15.77:0/534234640 shutdown_connections 2024-03-06T07:34:05.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.861+0000 7f6474adf640 1 -- 172.21.15.77:0/534234640 wait complete. 2024-03-06T07:34:05.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.862+0000 7f6474adf640 1 Processor -- start 2024-03-06T07:34:05.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.862+0000 7f6474adf640 1 -- start start 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f6474adf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64700cf830 0x7f647015f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f6474adf640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700d27f0 0x7f64700b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f6474adf640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f647015dfe0 0x7f64700b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f6474adf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700d1c70 con 0x7f64700cf830 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f6474adf640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700b3d60 con 0x7f64700d27f0 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f6474adf640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700d1df0 con 0x7f647015dfe0 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64700cf830 0x7f647015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64700cf830 0x7f647015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34448/0 (socket says 172.21.15.77:34448) 2024-03-06T07:34:05.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 -- 172.21.15.77:0/2845451189 learned_addr learned my addr 172.21.15.77:0/2845451189 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f6467fff640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700d27f0 0x7f64700b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646ed9d640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f647015dfe0 0x7f64700b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 -- 172.21.15.77:0/2845451189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f647015dfe0 msgr2=0x7f64700b9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f647015dfe0 0x7f64700b9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 -- 172.21.15.77:0/2845451189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700d27f0 msgr2=0x7f64700b8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700d27f0 0x7f64700b8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 -- 172.21.15.77:0/2845451189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700b99d0 con 0x7f64700cf830 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.863+0000 7f646e59c640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64700cf830 0x7f647015f050 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f646000e960 tx=0x7f646000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.864+0000 7f6466ffd640 1 -- 172.21.15.77:0/2845451189 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460017070 con 0x7f64700cf830 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.864+0000 7f6466ffd640 1 -- 172.21.15.77:0/2845451189 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646000ca00 con 0x7f64700cf830 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.864+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64700cc000 con 0x7f64700cf830 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.864+0000 7f6466ffd640 1 -- 172.21.15.77:0/2845451189 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646000cdd0 con 0x7f64700cf830 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.864+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64700cc5c0 con 0x7f64700cf830 2024-03-06T07:34:05.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.865+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647006d660 con 0x7f64700cf830 2024-03-06T07:34:05.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.865+0000 7f6466ffd640 1 -- 172.21.15.77:0/2845451189 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f646001a070 con 0x7f64700cf830 2024-03-06T07:34:05.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.866+0000 7f6466ffd640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645006b4d0 0x7f645006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.866+0000 7f6467fff640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645006b4d0 0x7f645006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.866+0000 7f6466ffd640 1 -- 172.21.15.77:0/2845451189 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f646008a350 con 0x7f64700cf830 2024-03-06T07:34:05.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.866+0000 7f6467fff640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645006b4d0 0x7f645006d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6458034000 tx=0x7f645802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:05.870+0000 7f6466ffd640 1 -- 172.21.15.77:0/2845451189 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f647006d660 con 0x7f64700cf830 2024-03-06T07:34:06.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.080+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64700cbdf0 con 0x7f64700cf830 2024-03-06T07:34:06.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.081+0000 7f6466ffd640 1 -- 172.21.15.77:0/2845451189 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16716 (secure 0 0 0) 0x7f6460058bf0 con 0x7f64700cf830 2024-03-06T07:34:06.084 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:06.084 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":143,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:30:49.223707+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:03.674704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:04.414541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:06.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645006b4d0 msgr2=0x7f645006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645006b4d0 0x7f645006d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6458034000 tx=0x7f645802f000 comp rx=0 tx=0).stop 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64700cf830 msgr2=0x7f647015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64700cf830 0x7f647015f050 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f646000e960 tx=0x7f646000ee20 comp rx=0 tx=0).stop 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 shutdown_connections 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f647015dfe0 0x7f64700b9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f64700d27f0 0x7f64700b8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f645006b4d0 0x7f645006d990 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 --2- 172.21.15.77:0/2845451189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64700cf830 0x7f647015f050 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 >> 172.21.15.77:0/2845451189 conn(0x7f64700d3cd0 msgr2=0x7f647006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 shutdown_connections 2024-03-06T07:34:06.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.084+0000 7f6474adf640 1 -- 172.21.15.77:0/2845451189 wait complete. 2024-03-06T07:34:06.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:34:06.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.201+0000 7f9b058df640 1 Processor -- start 2024-03-06T07:34:06.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.201+0000 7f9b058df640 1 -- start start 2024-03-06T07:34:06.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.201+0000 7f9b058df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b4c00 0x7f9b000b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.201+0000 7f9b058df640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9b0015ba50 0x7f9b0015be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.201+0000 7f9b058df640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0015d510 con 0x7f9b000b28a0 2024-03-06T07:34:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.201+0000 7f9b058df640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0015e0b0 con 0x7f9b0015ba50 2024-03-06T07:34:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9b058df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000b33f0 con 0x7f9b000b4c00 2024-03-06T07:34:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9aff7fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9b0015ba50 0x7f9b0015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9afeffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b4c00 0x7f9b000b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9afeffd640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b4c00 0x7f9b000b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3301/0 says I am v2:172.21.15.77:38314/0 (socket says 172.21.15.77:38314) 2024-03-06T07:34:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9aff7fe640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f9b0015ba50 0x7f9b0015be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:55662/0 (socket says 172.21.15.77:55662) 2024-03-06T07:34:06.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9afeffd640 1 -- 172.21.15.77:0/4160313621 learned_addr learned my addr 172.21.15.77:0/4160313621 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9afeffd640 1 -- 172.21.15.77:0/4160313621 >> v2:172.21.15.158:3300/0 conn(0x7f9b0015ba50 msgr2=0x7f9b0015be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9afeffd640 1 --2- 172.21.15.77:0/4160313621 >> v2:172.21.15.158:3300/0 conn(0x7f9b0015ba50 0x7f9b0015be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9afeffd640 1 -- 172.21.15.77:0/4160313621 >> v1:172.21.15.158:6789/0 conn(0x7f9b000b28a0 legacy=0x7f9b000b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9afeffd640 1 -- 172.21.15.77:0/4160313621 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0015d780 con 0x7f9b000b4c00 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9afe7fc640 1 -- 172.21.15.77:0/4160313621 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474324627 0 0) 0x7f9b0015d510 con 0x7f9b000b28a0 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.202+0000 7f9aff7fe640 1 --2- 172.21.15.77:0/4160313621 >> v2:172.21.15.158:3300/0 conn(0x7f9b0015ba50 0x7f9b0015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.203+0000 7f9afeffd640 1 --2- 172.21.15.77:0/4160313621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b4c00 0x7f9b000b4fe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009870 tx=0x7f9af002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fa4fc5d52d696b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.203+0000 7f9afe7fc640 1 -- 172.21.15.77:0/4160313621 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af002f840 con 0x7f9b000b4c00 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.203+0000 7f9afe7fc640 1 -- 172.21.15.77:0/4160313621 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af002f9e0 con 0x7f9b000b4c00 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.203+0000 7f9afe7fc640 1 -- 172.21.15.77:0/4160313621 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af002fdb0 con 0x7f9b000b4c00 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.204+0000 7f9b058df640 1 -- 172.21.15.77:0/4160313621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b4c00 msgr2=0x7f9b000b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.204+0000 7f9b058df640 1 --2- 172.21.15.77:0/4160313621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b4c00 0x7f9b000b4fe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009870 tx=0x7f9af002ba20 comp rx=0 tx=0).stop 2024-03-06T07:34:06.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.204+0000 7f9b058df640 1 -- 172.21.15.77:0/4160313621 shutdown_connections 2024-03-06T07:34:06.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.204+0000 7f9b058df640 1 --2- 172.21.15.77:0/4160313621 >> v2:172.21.15.158:3300/0 conn(0x7f9b0015ba50 0x7f9b0015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.204+0000 7f9b058df640 1 --2- 172.21.15.77:0/4160313621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b4c00 0x7f9b000b4fe0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.204+0000 7f9b058df640 1 -- 172.21.15.77:0/4160313621 >> 172.21.15.77:0/4160313621 conn(0x7f9b0009a9f0 msgr2=0x7f9b0009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.204+0000 7f9b058df640 1 -- 172.21.15.77:0/4160313621 shutdown_connections 2024-03-06T07:34:06.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.204+0000 7f9b058df640 1 -- 172.21.15.77:0/4160313621 wait complete. 2024-03-06T07:34:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.205+0000 7f9b058df640 1 Processor -- start 2024-03-06T07:34:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.205+0000 7f9b058df640 1 -- start start 2024-03-06T07:34:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9b058df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b28a0 0x7f9b001465b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9b058df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b000b4c00 0x7f9b001597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9b058df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b0015ba50 0x7f9b00159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9b058df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0015f550 con 0x7f9b0015ba50 2024-03-06T07:34:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9b058df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0015dd40 con 0x7f9b000b4c00 2024-03-06T07:34:06.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9b058df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0015f6d0 con 0x7f9b000b28a0 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b0015ba50 0x7f9b00159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b0015ba50 0x7f9b00159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34464/0 (socket says 172.21.15.77:34464) 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 -- 172.21.15.77:0/64610682 learned_addr learned my addr 172.21.15.77:0/64610682 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9afeffd640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b28a0 0x7f9b001465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9af7fff640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b000b4c00 0x7f9b001597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 -- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b28a0 msgr2=0x7f9b001465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b28a0 0x7f9b001465b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 -- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b000b4c00 msgr2=0x7f9b001597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b000b4c00 0x7f9b001597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 -- 172.21.15.77:0/64610682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b001f5320 con 0x7f9b0015ba50 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9afeffd640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b28a0 0x7f9b001465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9af7fff640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b000b4c00 0x7f9b001597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:06.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9aff7fe640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b0015ba50 0x7f9b00159d20 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f9af800efc0 tx=0x7f9af800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.206+0000 7f9afcff9640 1 -- 172.21.15.77:0/64610682 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af8018030 con 0x7f9b0015ba50 2024-03-06T07:34:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.207+0000 7f9afcff9640 1 -- 172.21.15.77:0/64610682 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af800cdf0 con 0x7f9b0015ba50 2024-03-06T07:34:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.207+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b001f5600 con 0x7f9b0015ba50 2024-03-06T07:34:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.207+0000 7f9afcff9640 1 -- 172.21.15.77:0/64610682 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af8003bb0 con 0x7f9b0015ba50 2024-03-06T07:34:06.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.207+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b001f5bc0 con 0x7f9b0015ba50 2024-03-06T07:34:06.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.208+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b001f5dc0 con 0x7f9b0015ba50 2024-03-06T07:34:06.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.208+0000 7f9afcff9640 1 -- 172.21.15.77:0/64610682 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9af801b070 con 0x7f9b0015ba50 2024-03-06T07:34:06.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.209+0000 7f9afcff9640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ae006b4d0 0x7f9ae006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.209+0000 7f9afeffd640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ae006b4d0 0x7f9ae006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.209+0000 7f9afcff9640 1 -- 172.21.15.77:0/64610682 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9af808abf0 con 0x7f9b0015ba50 2024-03-06T07:34:06.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.209+0000 7f9afeffd640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ae006b4d0 0x7f9ae006d990 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9af0036000 tx=0x7f9af0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.213+0000 7f9afcff9640 1 -- 172.21.15.77:0/64610682 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b001f5dc0 con 0x7f9b0015ba50 2024-03-06T07:34:06.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.520+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9b0004f330 con 0x7f9b0015ba50 2024-03-06T07:34:06.522 INFO:tasks.ceph.mon.a.smithi077.stderr:2024-03-06T07:34:06.520+0000 7f63ca6b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:34:06.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.523+0000 7f9afcff9640 1 -- 172.21.15.77:0/64610682 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v108) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9af8010040 con 0x7f9b0015ba50 2024-03-06T07:34:06.525 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:34:06.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ae006b4d0 msgr2=0x7f9ae006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ae006b4d0 0x7f9ae006d990 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9af0036000 tx=0x7f9af0008030 comp rx=0 tx=0).stop 2024-03-06T07:34:06.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b0015ba50 msgr2=0x7f9b00159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b0015ba50 0x7f9b00159d20 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f9af800efc0 tx=0x7f9af800c490 comp rx=0 tx=0).stop 2024-03-06T07:34:06.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 shutdown_connections 2024-03-06T07:34:06.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f9ae006b4d0 0x7f9ae006d990 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b0015ba50 0x7f9b00159d20 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9b000b4c00 0x7f9b001597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 --2- 172.21.15.77:0/64610682 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f9b000b28a0 0x7f9b001465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 >> 172.21.15.77:0/64610682 conn(0x7f9b0009a9f0 msgr2=0x7f9b00070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 shutdown_connections 2024-03-06T07:34:06.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.526+0000 7f9b058df640 1 -- 172.21.15.77:0/64610682 wait complete. 2024-03-06T07:34:06.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:34:06.563 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.644+0000 7eff35f2b640 1 Processor -- start 2024-03-06T07:34:06.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.644+0000 7eff35f2b640 1 -- start start 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff35f2b640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7eff300b4970 0x7eff3006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff35f2b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7eff3006d5c0 0x7eff300cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff35f2b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3015ff90 con 0x7eff3015bb40 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff35f2b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff300b4dd0 con 0x7eff3006d5c0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff35f2b640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3006c910 con 0x7eff300b4970 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7eff3006d5c0 0x7eff300cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7eff3015bb40 0x7eff3015f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59660/0 (socket says 172.21.15.77:59660) 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7eff3006d5c0 0x7eff300cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34502/0 (socket says 172.21.15.77:34502) 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2f7fe640 1 -- 172.21.15.77:0/1028758621 learned_addr learned my addr 172.21.15.77:0/1028758621 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2ffff640 1 -- 172.21.15.77:0/1028758621 >> v2:172.21.15.158:3300/0 conn(0x7eff300b4970 msgr2=0x7eff3006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2ffff640 1 --2- 172.21.15.77:0/1028758621 >> v2:172.21.15.158:3300/0 conn(0x7eff300b4970 0x7eff3006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2ffff640 1 -- 172.21.15.77:0/1028758621 >> v1:172.21.15.77:6789/0 conn(0x7eff3015bb40 legacy=0x7eff3015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2ffff640 1 -- 172.21.15.77:0/1028758621 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30160270 con 0x7eff3006d5c0 2024-03-06T07:34:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.645+0000 7eff2e7fc640 1 -- 172.21.15.77:0/1028758621 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665227878 0 0) 0x7eff3015ff90 con 0x7eff3015bb40 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.646+0000 7eff2ffff640 1 --2- 172.21.15.77:0/1028758621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3006d5c0 0x7eff300cfa90 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7eff3006bfe0 tx=0x7eff2802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766e35942147fa81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.646+0000 7eff2e7fc640 1 -- 172.21.15.77:0/1028758621 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28039070 con 0x7eff3006d5c0 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.646+0000 7eff2e7fc640 1 -- 172.21.15.77:0/1028758621 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2802cc80 con 0x7eff3006d5c0 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.646+0000 7eff2e7fc640 1 -- 172.21.15.77:0/1028758621 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28034660 con 0x7eff3006d5c0 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.647+0000 7eff35f2b640 1 -- 172.21.15.77:0/1028758621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3006d5c0 msgr2=0x7eff300cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.647+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1028758621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3006d5c0 0x7eff300cfa90 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7eff3006bfe0 tx=0x7eff2802bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.647+0000 7eff35f2b640 1 -- 172.21.15.77:0/1028758621 shutdown_connections 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.647+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1028758621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3006d5c0 0x7eff300cfa90 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.647+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1028758621 >> v2:172.21.15.158:3300/0 conn(0x7eff300b4970 0x7eff3006d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.647+0000 7eff35f2b640 1 -- 172.21.15.77:0/1028758621 >> 172.21.15.77:0/1028758621 conn(0x7eff3006b190 msgr2=0x7eff3006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.647+0000 7eff35f2b640 1 -- 172.21.15.77:0/1028758621 shutdown_connections 2024-03-06T07:34:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.647+0000 7eff35f2b640 1 -- 172.21.15.77:0/1028758621 wait complete. 2024-03-06T07:34:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.648+0000 7eff35f2b640 1 Processor -- start 2024-03-06T07:34:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.648+0000 7eff35f2b640 1 -- start start 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff35f2b640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff3006d5c0 0x7eff300afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff35f2b640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff300b4970 0x7eff30149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff35f2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3015bb40 0x7eff30156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff35f2b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3015bf20 con 0x7eff3015bb40 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff35f2b640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3006cb60 con 0x7eff3006d5c0 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff35f2b640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3015c0a0 con 0x7eff300b4970 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3015bb40 0x7eff30156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3015bb40 0x7eff30156c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34512/0 (socket says 172.21.15.77:34512) 2024-03-06T07:34:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 -- 172.21.15.77:0/1606298753 learned_addr learned my addr 172.21.15.77:0/1606298753 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2f7fe640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff3006d5c0 0x7eff300afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2effd640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff300b4970 0x7eff30149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 -- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff300b4970 msgr2=0x7eff30149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff300b4970 0x7eff30149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 -- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff3006d5c0 msgr2=0x7eff300afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff3006d5c0 0x7eff300afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 -- 172.21.15.77:0/1606298753 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff300b0430 con 0x7eff3015bb40 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2effd640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff300b4970 0x7eff30149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff2ffff640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3015bb40 0x7eff30156c00 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7eff28037040 tx=0x7eff28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff17fff640 1 -- 172.21.15.77:0/1606298753 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff28039050 con 0x7eff3015bb40 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff17fff640 1 -- 172.21.15.77:0/1606298753 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff28045070 con 0x7eff3015bb40 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.649+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3014e030 con 0x7eff3015bb40 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.650+0000 7eff17fff640 1 -- 172.21.15.77:0/1606298753 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2803e810 con 0x7eff3015bb40 2024-03-06T07:34:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.650+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3015c780 con 0x7eff3015bb40 2024-03-06T07:34:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.650+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff300732b0 con 0x7eff3015bb40 2024-03-06T07:34:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.651+0000 7eff17fff640 1 -- 172.21.15.77:0/1606298753 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff2803e9b0 con 0x7eff3015bb40 2024-03-06T07:34:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.651+0000 7eff17fff640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff1c06b4d0 0x7eff1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.651+0000 7eff17fff640 1 -- 172.21.15.77:0/1606298753 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5910+0+0 (secure 0 0 0) 0x7eff280ad3a0 con 0x7eff3015bb40 2024-03-06T07:34:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.651+0000 7eff2f7fe640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff1c06b4d0 0x7eff1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.652+0000 7eff2f7fe640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff1c06b4d0 0x7eff1c06d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7eff20007fd0 tx=0x7eff20007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.655+0000 7eff17fff640 1 -- 172.21.15.77:0/1606298753 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff300b0430 con 0x7eff3015bb40 2024-03-06T07:34:06.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:06.863+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7eff3006e130 con 0x7eff3015bb40 2024-03-06T07:34:07.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.527+0000 7eff17fff640 1 -- 172.21.15.77:0/1606298753 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v109) v1 ==== 104+0+0 (secure 0 0 0) 0x7eff2807c300 con 0x7eff3015bb40 2024-03-06T07:34:07.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff1c06b4d0 msgr2=0x7eff1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff1c06b4d0 0x7eff1c06d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7eff20007fd0 tx=0x7eff20007de0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3015bb40 msgr2=0x7eff30156c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3015bb40 0x7eff30156c00 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7eff28037040 tx=0x7eff28004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 shutdown_connections 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7eff1c06b4d0 0x7eff1c06d990 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff3015bb40 0x7eff30156c00 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7eff300b4970 0x7eff30149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 --2- 172.21.15.77:0/1606298753 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff3006d5c0 0x7eff300afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 >> 172.21.15.77:0/1606298753 conn(0x7eff3006b190 msgr2=0x7eff30159930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 shutdown_connections 2024-03-06T07:34:07.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.530+0000 7eff35f2b640 1 -- 172.21.15.77:0/1606298753 wait complete. 2024-03-06T07:34:07.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:34:07.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.687+0000 7f6b0d179640 1 Processor -- start 2024-03-06T07:34:07.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0d179640 1 -- start start 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0d179640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b0815dfa0 0x7f6b080d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0d179640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b080d29f0 0x7f6b080cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0d179640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0815e380 con 0x7f6b080d4630 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0d179640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b080d2100 con 0x7f6b080d29f0 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0d179640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080d1e30 con 0x7f6b0815dfa0 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b080d29f0 0x7f6b080cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0759e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b080d29f0 0x7f6b080cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34524/0 (socket says 172.21.15.77:34524) 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.688+0000 7f6b0759e640 1 -- 172.21.15.77:0/146649355 learned_addr learned my addr 172.21.15.77:0/146649355 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.689+0000 7f6b0759e640 1 -- 172.21.15.77:0/146649355 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b0815dfa0 msgr2=0x7f6b080d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.689+0000 7f6b06d9d640 1 --2- 172.21.15.77:0/146649355 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b0815dfa0 0x7f6b080d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.689+0000 7f6b05d9b640 1 -- 172.21.15.77:0/146649355 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372744099 0 0) 0x7f6b0815e380 con 0x7f6b080d4630 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.689+0000 7f6b0759e640 1 --2- 172.21.15.77:0/146649355 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b0815dfa0 0x7f6b080d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.689+0000 7f6b0759e640 1 -- 172.21.15.77:0/146649355 >> v1:172.21.15.77:6789/0 conn(0x7f6b080d4630 legacy=0x7f6b080d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.689+0000 7f6b0759e640 1 -- 172.21.15.77:0/146649355 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b080cf470 con 0x7f6b080d29f0 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.689+0000 7f6b0759e640 1 --2- 172.21.15.77:0/146649355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b080cb190 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009e50 tx=0x7f6b0002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f1deb208e2df58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b05d9b640 1 -- 172.21.15.77:0/146649355 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0002ca70 con 0x7f6b080d29f0 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b05d9b640 1 -- 172.21.15.77:0/146649355 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0002cc10 con 0x7f6b080d29f0 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b05d9b640 1 -- 172.21.15.77:0/146649355 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b000343f0 con 0x7f6b080d29f0 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b0d179640 1 -- 172.21.15.77:0/146649355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 msgr2=0x7f6b080cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b0d179640 1 --2- 172.21.15.77:0/146649355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b080cb190 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009e50 tx=0x7f6b0002bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b0d179640 1 -- 172.21.15.77:0/146649355 shutdown_connections 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b0d179640 1 --2- 172.21.15.77:0/146649355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b080cb190 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b0d179640 1 --2- 172.21.15.77:0/146649355 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b0815dfa0 0x7f6b080d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b0d179640 1 -- 172.21.15.77:0/146649355 >> 172.21.15.77:0/146649355 conn(0x7f6b0809d000 msgr2=0x7f6b080d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b0d179640 1 -- 172.21.15.77:0/146649355 shutdown_connections 2024-03-06T07:34:07.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.690+0000 7f6b0d179640 1 -- 172.21.15.77:0/146649355 wait complete. 2024-03-06T07:34:07.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.692+0000 7f6b0d179640 1 Processor -- start 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.692+0000 7f6b0d179640 1 -- start start 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.692+0000 7f6b0d179640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b0815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.692+0000 7f6b0d179640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b080d4630 0x7f6b08160760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.692+0000 7f6b0d179640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b0815dfa0 0x7f6b08160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.692+0000 7f6b0d179640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080cb6d0 con 0x7f6b080d29f0 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.692+0000 7f6b0d179640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080cfa30 con 0x7f6b0815dfa0 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.692+0000 7f6b0d179640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080cb850 con 0x7f6b080d4630 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b0815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b0815bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34536/0 (socket says 172.21.15.77:34536) 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 -- 172.21.15.77:0/2816638734 learned_addr learned my addr 172.21.15.77:0/2816638734 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b0759e640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b0815dfa0 0x7f6b08160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b0659c640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b080d4630 0x7f6b08160760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 -- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b080d4630 msgr2=0x7f6b08160760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b080d4630 0x7f6b08160760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 -- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b0815dfa0 msgr2=0x7f6b08160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b0815dfa0 0x7f6b08160ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 -- 172.21.15.77:0/2816638734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b080a0e70 con 0x7f6b080d29f0 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b0659c640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b080d4630 0x7f6b08160760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b06d9d640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b0815bbc0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f6af800c7f0 tx=0x7f6af800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6aef7fe640 1 -- 172.21.15.77:0/2816638734 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8015070 con 0x7f6b080d29f0 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6aef7fe640 1 -- 172.21.15.77:0/2816638734 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af8004220 con 0x7f6b080d29f0 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b080a1150 con 0x7f6b080d29f0 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6aef7fe640 1 -- 172.21.15.77:0/2816638734 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8007620 con 0x7f6b080d29f0 2024-03-06T07:34:07.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.693+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0815c100 con 0x7f6b080d29f0 2024-03-06T07:34:07.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.694+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0806c020 con 0x7f6b080d29f0 2024-03-06T07:34:07.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.695+0000 7f6aef7fe640 1 -- 172.21.15.77:0/2816638734 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6af80079d0 con 0x7f6b080d29f0 2024-03-06T07:34:07.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.695+0000 7f6aef7fe640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6af406b4d0 0x7f6af406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.695+0000 7f6aef7fe640 1 -- 172.21.15.77:0/2816638734 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f6af808a690 con 0x7f6b080d29f0 2024-03-06T07:34:07.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.695+0000 7f6b0659c640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6af406b4d0 0x7f6af406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.695+0000 7f6b0659c640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6af406b4d0 0x7f6af406d990 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f6b080c9720 tx=0x7f6afc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.699+0000 7f6aef7fe640 1 -- 172.21.15.77:0/2816638734 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b0806c020 con 0x7f6b080d29f0 2024-03-06T07:34:07.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:07.872+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6b0804f330 con 0x7f6b080d29f0 2024-03-06T07:34:08.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.528+0000 7f6aef7fe640 1 -- 172.21.15.77:0/2816638734 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v145) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6af8059440 con 0x7f6b080d29f0 2024-03-06T07:34:08.530 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.531+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6af406b4d0 msgr2=0x7f6af406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.531+0000 7f6b0d179640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6af406b4d0 0x7f6af406d990 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f6b080c9720 tx=0x7f6afc009040 comp rx=0 tx=0).stop 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.531+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 msgr2=0x7f6b0815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.531+0000 7f6b0d179640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b0815bbc0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f6af800c7f0 tx=0x7f6af800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.531+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 shutdown_connections 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.531+0000 7f6b0d179640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6b0815dfa0 0x7f6b08160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.531+0000 7f6b0d179640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6b080d4630 0x7f6b08160760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.532+0000 7f6b0d179640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6af406b4d0 0x7f6af406d990 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.532+0000 7f6b0d179640 1 --2- 172.21.15.77:0/2816638734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b080d29f0 0x7f6b0815bbc0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.532+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 >> 172.21.15.77:0/2816638734 conn(0x7f6b0809d000 msgr2=0x7f6b0806c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.532+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 shutdown_connections 2024-03-06T07:34:08.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.532+0000 7f6b0d179640 1 -- 172.21.15.77:0/2816638734 wait complete. 2024-03-06T07:34:08.546 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:34:08.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.648+0000 7f273f6ed640 1 Processor -- start 2024-03-06T07:34:08.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.648+0000 7f273f6ed640 1 -- start start 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273f6ed640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f273815dfa0 0x7f27380d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273f6ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 0x7f27380cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273f6ed640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273815e380 con 0x7f27380d4630 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273f6ed640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27380d2100 con 0x7f273815dfa0 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273f6ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27380d1e30 con 0x7f27380d29f0 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273d489640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f273815dfa0 0x7f27380d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273d489640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f273815dfa0 0x7f27380d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45672/0 (socket says 172.21.15.77:45672) 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273d489640 1 -- 172.21.15.77:0/3461141021 learned_addr learned my addr 172.21.15.77:0/3461141021 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273d489640 1 -- 172.21.15.77:0/3461141021 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 msgr2=0x7f27380cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273d489640 1 --2- 172.21.15.77:0/3461141021 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 0x7f27380cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273d489640 1 -- 172.21.15.77:0/3461141021 >> v1:172.21.15.158:6789/0 conn(0x7f27380d4630 legacy=0x7f27380d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.649+0000 7f273d489640 1 -- 172.21.15.77:0/3461141021 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27380cf470 con 0x7f273815dfa0 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.650+0000 7f273d489640 1 --2- 172.21.15.77:0/3461141021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f273815dfa0 0x7f27380d18f0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f2734009870 tx=0x7f273402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e325d830082931 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.650+0000 7f272ffff640 1 -- 172.21.15.77:0/3461141021 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273402e070 con 0x7f273815dfa0 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.650+0000 7f272ffff640 1 -- 172.21.15.77:0/3461141021 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2734037070 con 0x7f273815dfa0 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.650+0000 7f272ffff640 1 -- 172.21.15.77:0/3461141021 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734031a00 con 0x7f273815dfa0 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.651+0000 7f273f6ed640 1 -- 172.21.15.77:0/3461141021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f273815dfa0 msgr2=0x7f27380d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.651+0000 7f273f6ed640 1 --2- 172.21.15.77:0/3461141021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f273815dfa0 0x7f27380d18f0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f2734009870 tx=0x7f273402b880 comp rx=0 tx=0).stop 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.651+0000 7f273f6ed640 1 -- 172.21.15.77:0/3461141021 shutdown_connections 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.651+0000 7f273f6ed640 1 --2- 172.21.15.77:0/3461141021 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 0x7f27380cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.651+0000 7f273f6ed640 1 --2- 172.21.15.77:0/3461141021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f273815dfa0 0x7f27380d18f0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.651+0000 7f273f6ed640 1 -- 172.21.15.77:0/3461141021 >> 172.21.15.77:0/3461141021 conn(0x7f273809d000 msgr2=0x7f27380d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.651+0000 7f273f6ed640 1 -- 172.21.15.77:0/3461141021 shutdown_connections 2024-03-06T07:34:08.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.651+0000 7f273f6ed640 1 -- 172.21.15.77:0/3461141021 wait complete. 2024-03-06T07:34:08.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.652+0000 7f273f6ed640 1 Processor -- start 2024-03-06T07:34:08.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.652+0000 7f273f6ed640 1 -- start start 2024-03-06T07:34:08.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.652+0000 7f273f6ed640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 0x7f273815a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.652+0000 7f273f6ed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380d4630 0x7f27380c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273f6ed640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f273815dfa0 0x7f27380b8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273f6ed640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380cb6d0 con 0x7f27380d4630 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273f6ed640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380cfa30 con 0x7f273815dfa0 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273f6ed640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27380cb850 con 0x7f27380d29f0 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380d4630 0x7f27380c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380d4630 0x7f27380c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45676/0 (socket says 172.21.15.77:45676) 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 -- 172.21.15.77:0/50450235 learned_addr learned my addr 172.21.15.77:0/50450235 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273d489640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 0x7f273815a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273dc8a640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f273815dfa0 0x7f27380b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 -- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 msgr2=0x7f273815a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 0x7f273815a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 -- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f273815dfa0 msgr2=0x7f27380b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f273815dfa0 0x7f27380b8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 -- 172.21.15.77:0/50450235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27380c9fa0 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273dc8a640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f273815dfa0 0x7f27380b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273d489640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 0x7f273815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273cc88640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380d4630 0x7f27380c9a60 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f272800c400 tx=0x7f272800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f272dffb640 1 -- 172.21.15.77:0/50450235 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728011790 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f272dffb640 1 -- 172.21.15.77:0/50450235 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2728017070 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27380ca240 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.653+0000 7f272dffb640 1 -- 172.21.15.77:0/50450235 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728011bb0 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.654+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27380c7aa0 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.654+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273804f330 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.655+0000 7f272dffb640 1 -- 172.21.15.77:0/50450235 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f272801c460 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.655+0000 7f272dffb640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f271c06b4d0 0x7f271c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.655+0000 7f272dffb640 1 -- 172.21.15.77:0/50450235 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5472+0+0 (secure 0 0 0) 0x7f2728089ed0 con 0x7f27380d4630 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.655+0000 7f273d489640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f271c06b4d0 0x7f271c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.656+0000 7f273d489640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f271c06b4d0 0x7f271c06d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f2734036000 tx=0x7f273403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.659+0000 7f272dffb640 1 -- 172.21.15.77:0/50450235 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2728058ed0 con 0x7f27380d4630 2024-03-06T07:34:08.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:08.831+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f273806b860 con 0x7f27380d4630 2024-03-06T07:34:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.530+0000 7f272dffb640 1 -- 172.21.15.77:0/50450235 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v146) v1 ==== 169+0+0 (secure 0 0 0) 0x7f27280587d0 con 0x7f27380d4630 2024-03-06T07:34:09.532 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-06T07:34:09.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f271c06b4d0 msgr2=0x7f271c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f271c06b4d0 0x7f271c06d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f2734036000 tx=0x7f273403d040 comp rx=0 tx=0).stop 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380d4630 msgr2=0x7f27380c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380d4630 0x7f27380c9a60 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f272800c400 tx=0x7f272800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 shutdown_connections 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f271c06b4d0 0x7f271c06d990 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380d4630 0x7f27380c9a60 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f273815dfa0 0x7f27380b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 --2- 172.21.15.77:0/50450235 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f27380d29f0 0x7f273815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 >> 172.21.15.77:0/50450235 conn(0x7f273809d000 msgr2=0x7f27380cc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 shutdown_connections 2024-03-06T07:34:09.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.533+0000 7f273f6ed640 1 -- 172.21.15.77:0/50450235 wait complete. 2024-03-06T07:34:09.547 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:34:09.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.647+0000 7fc00c0f4640 1 Processor -- start 2024-03-06T07:34:09.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.647+0000 7fc00c0f4640 1 -- start start 2024-03-06T07:34:09.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00c0f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc00415bb40 0x7fc0040b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00c0f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0040cfa90 0x7fc0040cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00c0f4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00415ff90 con 0x7fc00415f7d0 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00c0f4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00406c910 con 0x7fc0040cfa90 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00c0f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc004160270 con 0x7fc00415bb40 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00a691640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0040cfa90 0x7fc0040cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00968f640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc00415f7d0 0x7fc00406d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59776/0 (socket says 172.21.15.77:59776) 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00968f640 1 -- 172.21.15.77:0/2498560247 learned_addr learned my addr 172.21.15.77:0/2498560247 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00a691640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc0040cfa90 0x7fc0040cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45682/0 (socket says 172.21.15.77:45682) 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc009e90640 1 --2- 172.21.15.77:0/2498560247 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc00415bb40 0x7fc0040b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00a691640 1 -- 172.21.15.77:0/2498560247 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc00415bb40 msgr2=0x7fc0040b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00a691640 1 --2- 172.21.15.77:0/2498560247 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc00415bb40 0x7fc0040b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00a691640 1 -- 172.21.15.77:0/2498560247 >> v1:172.21.15.77:6789/0 conn(0x7fc00415f7d0 legacy=0x7fc00406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc00a691640 1 -- 172.21.15.77:0/2498560247 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc004160550 con 0x7fc0040cfa90 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc009e90640 1 --2- 172.21.15.77:0/2498560247 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc00415bb40 0x7fc0040b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.648+0000 7fc008e8e640 1 -- 172.21.15.77:0/2498560247 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353553854 0 0) 0x7fc00415ff90 con 0x7fc00415f7d0 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.649+0000 7fc00a691640 1 --2- 172.21.15.77:0/2498560247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0040cfa90 0x7fc0040cff20 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0097a0 tx=0x7fbffc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc88d101c283e348 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.649+0000 7fc008e8e640 1 -- 172.21.15.77:0/2498560247 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc035020 con 0x7fc0040cfa90 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.649+0000 7fc008e8e640 1 -- 172.21.15.77:0/2498560247 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc031830 con 0x7fc0040cfa90 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.649+0000 7fc008e8e640 1 -- 172.21.15.77:0/2498560247 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc031c00 con 0x7fc0040cfa90 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.650+0000 7fc00c0f4640 1 -- 172.21.15.77:0/2498560247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0040cfa90 msgr2=0x7fc0040cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.650+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/2498560247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0040cfa90 0x7fc0040cff20 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0097a0 tx=0x7fbffc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.650+0000 7fc00c0f4640 1 -- 172.21.15.77:0/2498560247 shutdown_connections 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.650+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/2498560247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0040cfa90 0x7fc0040cff20 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.650+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/2498560247 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc00415bb40 0x7fc0040b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.650+0000 7fc00c0f4640 1 -- 172.21.15.77:0/2498560247 >> 172.21.15.77:0/2498560247 conn(0x7fc00406b190 msgr2=0x7fc00406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.650+0000 7fc00c0f4640 1 -- 172.21.15.77:0/2498560247 shutdown_connections 2024-03-06T07:34:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.650+0000 7fc00c0f4640 1 -- 172.21.15.77:0/2498560247 wait complete. 2024-03-06T07:34:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.651+0000 7fc00c0f4640 1 Processor -- start 2024-03-06T07:34:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.651+0000 7fc00c0f4640 1 -- start start 2024-03-06T07:34:09.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00c0f4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0040cfa90 0x7fc004149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00c0f4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc00415bb40 0x7fc0041474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00c0f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00415f7d0 0x7fc0041461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00c0f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00406d790 con 0x7fc00415f7d0 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00c0f4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00415fc30 con 0x7fc00415bb40 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00c0f4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00415bf20 con 0x7fc0040cfa90 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00415f7d0 0x7fc0041461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00415f7d0 0x7fc0041461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45686/0 (socket says 172.21.15.77:45686) 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 -- 172.21.15.77:0/887542604 learned_addr learned my addr 172.21.15.77:0/887542604 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc009e90640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0040cfa90 0x7fc004149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00968f640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc00415bb40 0x7fc0041474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 -- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0040cfa90 msgr2=0x7fc004149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0040cfa90 0x7fc004149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 -- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc00415bb40 msgr2=0x7fc0041474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc00415bb40 0x7fc0041474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 -- 172.21.15.77:0/887542604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc004147c40 con 0x7fc00415f7d0 2024-03-06T07:34:09.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc009e90640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0040cfa90 0x7fc004149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fc00a691640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00415f7d0 0x7fc0041461e0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0098c0 tx=0x7fbffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fbffa7fc640 1 -- 172.21.15.77:0/887542604 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc004230 con 0x7fc00415f7d0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fbffa7fc640 1 -- 172.21.15.77:0/887542604 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc0043d0 con 0x7fc00415f7d0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.652+0000 7fbffa7fc640 1 -- 172.21.15.77:0/887542604 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc03c990 con 0x7fc00415f7d0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.653+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc004147f20 con 0x7fc00415f7d0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.653+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0041484e0 con 0x7fc00415f7d0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.654+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0040732b0 con 0x7fc00415f7d0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.654+0000 7fbffa7fc640 1 -- 172.21.15.77:0/887542604 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbffc038020 con 0x7fc00415f7d0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.654+0000 7fbffa7fc640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfe806b4d0 0x7fbfe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.655+0000 7fc009e90640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfe806b4d0 0x7fbfe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.655+0000 7fbffa7fc640 1 -- 172.21.15.77:0/887542604 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbffc0ac380 con 0x7fc00415f7d0 2024-03-06T07:34:09.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.655+0000 7fc009e90640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfe806b4d0 0x7fbfe806d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc00000e550 tx=0x7fc000007ab0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.659+0000 7fbffa7fc640 1 -- 172.21.15.77:0/887542604 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbffc041070 con 0x7fc00415f7d0 2024-03-06T07:34:09.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.837+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc004068ff0 con 0x7fc00415f7d0 2024-03-06T07:34:09.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.841+0000 7fbffa7fc640 1 -- 172.21.15.77:0/887542604 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbffc031800 con 0x7fc00415f7d0 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.842+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfe806b4d0 msgr2=0x7fbfe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.842+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfe806b4d0 0x7fbfe806d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc00000e550 tx=0x7fc000007ab0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.842+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00415f7d0 msgr2=0x7fc0041461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.842+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00415f7d0 0x7fc0041461e0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0098c0 tx=0x7fbffc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.843+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 shutdown_connections 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.843+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbfe806b4d0 0x7fbfe806d990 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.843+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00415f7d0 0x7fc0041461e0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.843+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc00415bb40 0x7fc0041474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.843+0000 7fc00c0f4640 1 --2- 172.21.15.77:0/887542604 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc0040cfa90 0x7fc004149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.843+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 >> 172.21.15.77:0/887542604 conn(0x7fc00406b190 msgr2=0x7fc0040b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.843+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 shutdown_connections 2024-03-06T07:34:09.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.843+0000 7fc00c0f4640 1 -- 172.21.15.77:0/887542604 wait complete. 2024-03-06T07:34:09.855 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:34:09.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.953+0000 7fdca3066640 1 Processor -- start 2024-03-06T07:34:09.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.954+0000 7fdca3066640 1 -- start start 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.954+0000 7fdca3066640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdc9c13dc10 0x7fdc9c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.954+0000 7fdca3066640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc9c15ef60 0x7fdc9c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.954+0000 7fdca3066640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c15f580 con 0x7fdc9c15bf30 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.954+0000 7fdca3066640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9c0cf3f0 con 0x7fdc9c13dc10 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.954+0000 7fdca3066640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9c15b210 con 0x7fdc9c15ef60 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.954+0000 7fdc9bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdc9c15bf30 0x7fdc9c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59782/0 (socket says 172.21.15.77:59782) 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdc9bfff640 1 -- 172.21.15.77:0/1893281249 learned_addr learned my addr 172.21.15.77:0/1893281249 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdca0e02640 1 --2- 172.21.15.77:0/1893281249 >> v2:172.21.15.77:3300/0 conn(0x7fdc9c13dc10 0x7fdc9c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdca1603640 1 --2- 172.21.15.77:0/1893281249 >> v2:172.21.15.158:3300/0 conn(0x7fdc9c15ef60 0x7fdc9c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdc9b7fe640 1 -- 172.21.15.77:0/1893281249 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452357601 0 0) 0x7fdc9c15f580 con 0x7fdc9c15bf30 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdc9b7fe640 1 -- 172.21.15.77:0/1893281249 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc88003520 con 0x7fdc9c15bf30 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdca0e02640 1 -- 172.21.15.77:0/1893281249 >> v2:172.21.15.158:3300/0 conn(0x7fdc9c15ef60 msgr2=0x7fdc9c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdca0e02640 1 --2- 172.21.15.77:0/1893281249 >> v2:172.21.15.158:3300/0 conn(0x7fdc9c15ef60 0x7fdc9c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdca0e02640 1 -- 172.21.15.77:0/1893281249 >> v1:172.21.15.77:6789/0 conn(0x7fdc9c15bf30 legacy=0x7fdc9c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.955+0000 7fdca0e02640 1 -- 172.21.15.77:0/1893281249 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c15b450 con 0x7fdc9c13dc10 2024-03-06T07:34:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.956+0000 7fdca0e02640 1 --2- 172.21.15.77:0/1893281249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c0b2950 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c068700 tx=0x7fdc9402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522c1f7cd4df6c05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.956+0000 7fdc9b7fe640 1 -- 172.21.15.77:0/1893281249 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94039070 con 0x7fdc9c13dc10 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.956+0000 7fdc9b7fe640 1 -- 172.21.15.77:0/1893281249 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9402cbf0 con 0x7fdc9c13dc10 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.956+0000 7fdc9b7fe640 1 -- 172.21.15.77:0/1893281249 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94034440 con 0x7fdc9c13dc10 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.956+0000 7fdca3066640 1 -- 172.21.15.77:0/1893281249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 msgr2=0x7fdc9c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.956+0000 7fdca3066640 1 --2- 172.21.15.77:0/1893281249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c0b2950 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c068700 tx=0x7fdc9402be40 comp rx=0 tx=0).stop 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.956+0000 7fdca3066640 1 -- 172.21.15.77:0/1893281249 shutdown_connections 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.957+0000 7fdca3066640 1 --2- 172.21.15.77:0/1893281249 >> v2:172.21.15.158:3300/0 conn(0x7fdc9c15ef60 0x7fdc9c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.957+0000 7fdca3066640 1 --2- 172.21.15.77:0/1893281249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c0b2950 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.957+0000 7fdca3066640 1 -- 172.21.15.77:0/1893281249 >> 172.21.15.77:0/1893281249 conn(0x7fdc9c0cf8a0 msgr2=0x7fdc9c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.957+0000 7fdca3066640 1 -- 172.21.15.77:0/1893281249 shutdown_connections 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.957+0000 7fdca3066640 1 -- 172.21.15.77:0/1893281249 wait complete. 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.958+0000 7fdca3066640 1 Processor -- start 2024-03-06T07:34:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.958+0000 7fdca3066640 1 -- start start 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.958+0000 7fdca3066640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c13f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca3066640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc9c15bf30 0x7fdc9c13a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca3066640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc9c15ef60 0x7fdc9c1431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca3066640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0708d0 con 0x7fdc9c13dc10 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca3066640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c13dff0 con 0x7fdc9c15ef60 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca3066640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c070a50 con 0x7fdc9c15bf30 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c13f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c13f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45706/0 (socket says 172.21.15.77:45706) 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 -- 172.21.15.77:0/85080515 learned_addr learned my addr 172.21.15.77:0/85080515 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca1603640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc9c15ef60 0x7fdc9c1431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdc9bfff640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc9c15bf30 0x7fdc9c13a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 -- 172.21.15.77:0/85080515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc9c15bf30 msgr2=0x7fdc9c13a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc9c15bf30 0x7fdc9c13a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 -- 172.21.15.77:0/85080515 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc9c15ef60 msgr2=0x7fdc9c1431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc9c15ef60 0x7fdc9c1431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 -- 172.21.15.77:0/85080515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c13e3c0 con 0x7fdc9c13dc10 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca0e02640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c13f180 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fdc94000f80 tx=0x7fdc94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdc997fa640 1 -- 172.21.15.77:0/85080515 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94039050 con 0x7fdc9c13dc10 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdc997fa640 1 -- 172.21.15.77:0/85080515 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc94045070 con 0x7fdc9c13dc10 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c13e6a0 con 0x7fdc9c13dc10 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdc997fa640 1 -- 172.21.15.77:0/85080515 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94009450 con 0x7fdc9c13dc10 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.959+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c13ec60 con 0x7fdc9c13dc10 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.960+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c04f330 con 0x7fdc9c13dc10 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.961+0000 7fdc997fa640 1 -- 172.21.15.77:0/85080515 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc9404a460 con 0x7fdc9c13dc10 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.961+0000 7fdc997fa640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.961+0000 7fdc997fa640 1 -- 172.21.15.77:0/85080515 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdc940aca80 con 0x7fdc9c13dc10 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.961+0000 7fdc9bfff640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.961+0000 7fdc9bfff640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdc84007fd0 tx=0x7fdc84009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:09.965+0000 7fdc997fa640 1 -- 172.21.15.77:0/85080515 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc9407bb20 con 0x7fdc9c13dc10 2024-03-06T07:34:10.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.204+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7fdc9c06eb00 con 0x7fdc9c13dc10 2024-03-06T07:34:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.531+0000 7fdc997fa640 1 -- 172.21.15.77:0/85080515 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v632) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdc9407b420 con 0x7fdc9c13dc10 2024-03-06T07:34:10.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.533+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7c06b4d0 msgr2=0x7fdc7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.533+0000 7fdca3066640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdc84007fd0 tx=0x7fdc84009040 comp rx=0 tx=0).stop 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.533+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 msgr2=0x7fdc9c13f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.533+0000 7fdca3066640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c13f180 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fdc94000f80 tx=0x7fdc94004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.534+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 shutdown_connections 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.534+0000 7fdca3066640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc9c15ef60 0x7fdc9c1431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.534+0000 7fdca3066640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc9c15bf30 0x7fdc9c13a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.534+0000 7fdca3066640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc7c06b4d0 0x7fdc7c06d990 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.534+0000 7fdca3066640 1 --2- 172.21.15.77:0/85080515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc9c13dc10 0x7fdc9c13f180 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.534+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 >> 172.21.15.77:0/85080515 conn(0x7fdc9c0cf8a0 msgr2=0x7fdc9c0b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.534+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 shutdown_connections 2024-03-06T07:34:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:10.534+0000 7fdca3066640 1 -- 172.21.15.77:0/85080515 wait complete. 2024-03-06T07:34:10.552 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays ... ERROR 2024-03-06T07:34:10.552 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:10.552 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner:Ran 9 tests in 702.070s 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:34:10.553 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:34:10.554 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:10.555 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:10.769 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1ba975177ad0434d9f4af529d2d55702 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:10.770 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:34:10.783 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-06T07:34:10.827 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-06T07:34:10.827 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:34:10.827 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:10.847 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:10.847 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-06T07:34:10.847 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:10.903 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14244 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:10.903 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:34:10.903 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:10.922 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:10.922 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-03-06T07:34:10.922 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:10.980 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19819 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:10.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.981 DEBUG:teuthology.orchestra.run.smithi158:> ip netns list 2024-03-06T07:34:11.037 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-06T07:34:11.037 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:34:11.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:11.038 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:34:11.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:11.109 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:34:11.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:11.180 DEBUG:teuthology.orchestra.run.smithi158:> sudo ip link delete ceph-brx 2024-03-06T07:34:11.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:11.278 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:11.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:11.296 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:11.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:11.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:11.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:11.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-03-06T07:34:11.542 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:34:11.551 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:34:11.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.651+0000 7f44ecfee640 1 Processor -- start 2024-03-06T07:34:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.651+0000 7f44ecfee640 1 -- start start 2024-03-06T07:34:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44ecfee640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e815bf30 0x7f44e815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44ecfee640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e815f580 con 0x7f44e80b3060 2024-03-06T07:34:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44ecfee640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e815b450 con 0x7f44e813dc10 2024-03-06T07:34:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44ecfee640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80cf3f0 con 0x7f44e815bf30 2024-03-06T07:34:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44e5d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f44e80b3060 0x7f44e815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59784/0 (socket says 172.21.15.77:59784) 2024-03-06T07:34:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44e5d9b640 1 -- 172.21.15.77:0/3130864689 learned_addr learned my addr 172.21.15.77:0/3130864689 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:11.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469403968 0 0) 0x7f44e815f580 con 0x7f44e80b3060 2024-03-06T07:34:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d0003520 con 0x7f44e80b3060 2024-03-06T07:34:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.652+0000 7f44e6d9d640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e815bf30 0x7f44e815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 32054795 0 0) 0x7f44d0003520 con 0x7f44e80b3060 2024-03-06T07:34:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e815bf30 msgr2=0x7f44e815ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e815bf30 0x7f44e815ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 >> v1:172.21.15.158:6789/0 conn(0x7f44e813dc10 legacy=0x7f44e80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e815b730 con 0x7f44e80b3060 2024-03-06T07:34:11.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f44dc001670 con 0x7f44e80b3060 2024-03-06T07:34:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 >> v1:172.21.15.77:6789/0 conn(0x7f44e80b3060 legacy=0x7f44e815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e815bf30 0x7f44d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0003f60 0x7f44d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44d0004d40 0x7f44d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80cf3f0 con 0x7f44d0004d40 2024-03-06T07:34:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d000dd60 con 0x7f44d0003f60 2024-03-06T07:34:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d000df60 con 0x7f44e815bf30 2024-03-06T07:34:11.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e6d9d640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44d0004d40 0x7f44d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e659c640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e815bf30 0x7f44d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e5d9b640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0003f60 0x7f44d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e6d9d640 1 -- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e815bf30 msgr2=0x7f44d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e6d9d640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e815bf30 0x7f44d00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e6d9d640 1 -- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0003f60 msgr2=0x7f44d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e6d9d640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0003f60 0x7f44d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.653+0000 7f44e6d9d640 1 -- 172.21.15.77:0/3130864689 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0003520 con 0x7f44d0004d40 2024-03-06T07:34:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.654+0000 7f44e6d9d640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44d0004d40 0x7f44d00051c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009ac0 tx=0x7f44d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d7b9622c3b6f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.654+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0039070 con 0x7f44d0004d40 2024-03-06T07:34:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.654+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e002c6b0 con 0x7f44d0004d40 2024-03-06T07:34:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.654+0000 7f44e559a640 1 -- 172.21.15.77:0/3130864689 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0009690 con 0x7f44d0004d40 2024-03-06T07:34:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.654+0000 7f44ecfee640 1 -- 172.21.15.77:0/3130864689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44d0004d40 msgr2=0x7f44d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.654+0000 7f44ecfee640 1 --2- 172.21.15.77:0/3130864689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44d0004d40 0x7f44d00051c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009ac0 tx=0x7f44d0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.655+0000 7f44e6d9d640 1 -- 172.21.15.77:0/3130864689 reap_dead start 2024-03-06T07:34:11.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.655+0000 7f44ecfee640 1 -- 172.21.15.77:0/3130864689 shutdown_connections 2024-03-06T07:34:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.655+0000 7f44ecfee640 1 -- 172.21.15.77:0/3130864689 >> 172.21.15.77:0/3130864689 conn(0x7f44e80cf8a0 msgr2=0x7f44e80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.655+0000 7f44ecfee640 1 -- 172.21.15.77:0/3130864689 shutdown_connections 2024-03-06T07:34:11.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.655+0000 7f44ecfee640 1 -- 172.21.15.77:0/3130864689 wait complete. 2024-03-06T07:34:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.656+0000 7f44ecfee640 1 Processor -- start 2024-03-06T07:34:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.656+0000 7f44ecfee640 1 -- start start 2024-03-06T07:34:11.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44ecfee640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0004d40 0x7f44e812f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44ecfee640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e81f4650 0x7f44e81f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44ecfee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e81e4570 0x7f44e81e4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44ecfee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e815ef60 con 0x7f44e81e4570 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44ecfee640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e815b210 con 0x7f44d0004d40 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44ecfee640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e815f0e0 con 0x7f44e81f4650 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e81e4570 0x7f44e81e4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e81e4570 0x7f44e81e4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45718/0 (socket says 172.21.15.77:45718) 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 -- 172.21.15.77:0/1700077441 learned_addr learned my addr 172.21.15.77:0/1700077441 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:11.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e5d9b640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e81f4650 0x7f44e81f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 -- 172.21.15.77:0/1700077441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e81f4650 msgr2=0x7f44e81f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e659c640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0004d40 0x7f44e812f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e81f4650 0x7f44e81f1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 -- 172.21.15.77:0/1700077441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0004d40 msgr2=0x7f44e812f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0004d40 0x7f44e812f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 -- 172.21.15.77:0/1700077441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e81f4e40 con 0x7f44e81e4570 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44e6d9d640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e81e4570 0x7f44e81e4a20 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f44e002c320 tx=0x7f44e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44d6ffd640 1 -- 172.21.15.77:0/1700077441 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0039070 con 0x7f44e81e4570 2024-03-06T07:34:11.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44d6ffd640 1 -- 172.21.15.77:0/1700077441 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0043070 con 0x7f44e81e4570 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44d6ffd640 1 -- 172.21.15.77:0/1700077441 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0009690 con 0x7f44e81e4570 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e81f5120 con 0x7f44e81e4570 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.657+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e81f5a00 con 0x7f44e81e4570 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.658+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e81f5c00 con 0x7f44e81e4570 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.659+0000 7f44d6ffd640 1 -- 172.21.15.77:0/1700077441 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44e0003850 con 0x7f44e81e4570 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.659+0000 7f44d6ffd640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44c806b4d0 0x7f44c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.659+0000 7f44e659c640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44c806b4d0 0x7f44c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.659+0000 7f44d6ffd640 1 -- 172.21.15.77:0/1700077441 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f44e00ab070 con 0x7f44e81e4570 2024-03-06T07:34:11.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.660+0000 7f44e659c640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44c806b4d0 0x7f44c806d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f44e8072180 tx=0x7f44d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.663+0000 7f44d6ffd640 1 -- 172.21.15.77:0/1700077441 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44e81f5c00 con 0x7f44e81e4570 2024-03-06T07:34:11.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.832+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44e81f5640 con 0x7f44c806b4d0 2024-03-06T07:34:11.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.833+0000 7f44d6ffd640 1 -- 172.21.15.77:0/1700077441 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13433 (secure 0 0 0) 0x7f44e81f5640 con 0x7f44c806b4d0 2024-03-06T07:34:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:11.835 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:34:11.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44c806b4d0 msgr2=0x7f44c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44c806b4d0 0x7f44c806d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f44e8072180 tx=0x7f44d8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:11.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e81e4570 msgr2=0x7f44e81e4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e81e4570 0x7f44e81e4a20 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f44e002c320 tx=0x7f44e0004730 comp rx=0 tx=0).stop 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 shutdown_connections 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f44c806b4d0 0x7f44c806d990 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e81e4570 0x7f44e81e4a20 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f44e81f4650 0x7f44e81f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 --2- 172.21.15.77:0/1700077441 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f44d0004d40 0x7f44e812f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 >> 172.21.15.77:0/1700077441 conn(0x7f44e80cf8a0 msgr2=0x7f44e80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 shutdown_connections 2024-03-06T07:34:11.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.835+0000 7f44ecfee640 1 -- 172.21.15.77:0/1700077441 wait complete. 2024-03-06T07:34:11.847 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-03-06T07:34:11.518106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":318908,"kb_used_data":15388,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754655812,"statfs":{"total":773094113280,"available":772767551488,"internally_reserved":0,"allocated":15757312,"data_stored":7574632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":310693424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31593,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-140,"num_write_kb":-79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001131"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:34:06.568083+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:34:06.568083+0000","last_peered":"2024-03-06T07:34:06.568083+0000","last_clean":"2024-03-06T07:34:06.568083+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:34:06.568083+0000","last_undegraded":"2024-03-06T07:34:06.568083+0000","last_fullsized":"2024-03-06T07:34:06.568083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46028,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325812,"statfs":{"total":96636764160,"available":96589631488,"internally_reserved":0,"allocated":2424832,"data_stored":1392887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":44693962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28796,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343044,"statfs":{"total":96636764160,"available":96607277056,"internally_reserved":0,"allocated":1818624,"data_stored":792852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574998,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45412,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326428,"statfs":{"total":96636764160,"available":96590262272,"internally_reserved":0,"allocated":1794048,"data_stored":785864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574997,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":792124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574998,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41312,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330528,"statfs":{"total":96636764160,"available":96594460672,"internally_reserved":0,"allocated":1855488,"data_stored":822227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574998,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41272,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330568,"statfs":{"total":96636764160,"available":96594501632,"internally_reserved":0,"allocated":1814528,"data_stored":792818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2428928,"data_stored":1402884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:11.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:11.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.946+0000 7f25835ac640 1 Processor -- start 2024-03-06T07:34:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.947+0000 7f25835ac640 1 -- start start 2024-03-06T07:34:11.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.947+0000 7f25835ac640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f257c0d4760 0x7f257c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.947+0000 7f25835ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0d0a80 0x7f257c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.947+0000 7f25835ac640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0d25d0 con 0x7f257c0b4170 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.947+0000 7f25835ac640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c0d4bf0 con 0x7f257c0d4760 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.947+0000 7f25835ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0ce9e0 con 0x7f257c0d0a80 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2580b47640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f257c0d4760 0x7f257c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2581b49640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0d0a80 0x7f257c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2580b47640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f257c0d4760 0x7f257c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:40674/0 (socket says 172.21.15.77:40674) 2024-03-06T07:34:11.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2580b47640 1 -- 172.21.15.77:0/3606541867 learned_addr learned my addr 172.21.15.77:0/3606541867 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2580b47640 1 -- 172.21.15.77:0/3606541867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0d0a80 msgr2=0x7f257c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2580b47640 1 --2- 172.21.15.77:0/3606541867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0d0a80 0x7f257c0cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2580b47640 1 -- 172.21.15.77:0/3606541867 >> v1:172.21.15.158:6789/0 conn(0x7f257c0b4170 legacy=0x7f257c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f256ffff640 1 -- 172.21.15.77:0/3606541867 <== mon.1 v1:172.21.15.158:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398576261 0 0) 0x7f257c0d25d0 con 0x7f257c0b4170 2024-03-06T07:34:11.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2580b47640 1 -- 172.21.15.77:0/3606541867 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0d2870 con 0x7f257c0d4760 2024-03-06T07:34:11.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.948+0000 7f2581b49640 1 --2- 172.21.15.77:0/3606541867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0d0a80 0x7f257c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:11.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.949+0000 7f2580b47640 1 --2- 172.21.15.77:0/3606541867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0d4760 0x7f257c0b6a90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2578009870 tx=0x7f257802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a91ac0de92705677 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.949+0000 7f256ffff640 1 -- 172.21.15.77:0/3606541867 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578034070 con 0x7f257c0d4760 2024-03-06T07:34:11.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.949+0000 7f256ffff640 1 -- 172.21.15.77:0/3606541867 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578030840 con 0x7f257c0d4760 2024-03-06T07:34:11.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.949+0000 7f256ffff640 1 -- 172.21.15.77:0/3606541867 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578030c10 con 0x7f257c0d4760 2024-03-06T07:34:11.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.950+0000 7f25835ac640 1 -- 172.21.15.77:0/3606541867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0d4760 msgr2=0x7f257c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.950+0000 7f25835ac640 1 --2- 172.21.15.77:0/3606541867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0d4760 0x7f257c0b6a90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2578009870 tx=0x7f257802b980 comp rx=0 tx=0).stop 2024-03-06T07:34:11.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.950+0000 7f25835ac640 1 -- 172.21.15.77:0/3606541867 shutdown_connections 2024-03-06T07:34:11.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.950+0000 7f25835ac640 1 --2- 172.21.15.77:0/3606541867 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0d0a80 0x7f257c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.950+0000 7f25835ac640 1 --2- 172.21.15.77:0/3606541867 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0d4760 0x7f257c0b6a90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.950+0000 7f25835ac640 1 -- 172.21.15.77:0/3606541867 >> 172.21.15.77:0/3606541867 conn(0x7f257c0d3c40 msgr2=0x7f257c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.950+0000 7f25835ac640 1 -- 172.21.15.77:0/3606541867 shutdown_connections 2024-03-06T07:34:11.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.950+0000 7f25835ac640 1 -- 172.21.15.77:0/3606541867 wait complete. 2024-03-06T07:34:11.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.951+0000 7f25835ac640 1 Processor -- start 2024-03-06T07:34:11.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.951+0000 7f25835ac640 1 -- start start 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f25835ac640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0b4170 0x7f257c15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f25835ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f257c0d0a80 0x7f257c0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f25835ac640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0c7910 0x7f257c0a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f25835ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0b4dd0 con 0x7f257c0d0a80 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f25835ac640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0b4550 con 0x7f257c0c7910 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f25835ac640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0b4f50 con 0x7f257c0b4170 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f257c0d0a80 0x7f257c0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f257c0d0a80 0x7f257c0b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45722/0 (socket says 172.21.15.77:45722) 2024-03-06T07:34:11.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 -- 172.21.15.77:0/923323210 learned_addr learned my addr 172.21.15.77:0/923323210 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2581b49640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0c7910 0x7f257c0a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2581348640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0b4170 0x7f257c15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 -- 172.21.15.77:0/923323210 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0b4170 msgr2=0x7f257c15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0b4170 0x7f257c15dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 -- 172.21.15.77:0/923323210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0c7910 msgr2=0x7f257c0a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0c7910 0x7f257c0a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 -- 172.21.15.77:0/923323210 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0b2790 con 0x7f257c0d0a80 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f2580b47640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f257c0d0a80 0x7f257c0b2250 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f2578000c00 tx=0x7f25780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f256dffb640 1 -- 172.21.15.77:0/923323210 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578034040 con 0x7f257c0d0a80 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f256dffb640 1 -- 172.21.15.77:0/923323210 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257803ab90 con 0x7f257c0d0a80 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c0b2a30 con 0x7f257c0d0a80 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.952+0000 7f256dffb640 1 -- 172.21.15.77:0/923323210 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578004030 con 0x7f257c0d0a80 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.953+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c0b2ff0 con 0x7f257c0d0a80 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.954+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c0d4020 con 0x7f257c0d0a80 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.954+0000 7f256dffb640 1 -- 172.21.15.77:0/923323210 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f257804b060 con 0x7f257c0d0a80 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.954+0000 7f256dffb640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f256006b4d0 0x7f256006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.955+0000 7f2581348640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f256006b4d0 0x7f256006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.955+0000 7f256dffb640 1 -- 172.21.15.77:0/923323210 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f25780abc30 con 0x7f257c0d0a80 2024-03-06T07:34:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.955+0000 7f2581348640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f256006b4d0 0x7f256006d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2570000f60 tx=0x7f2570009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:11.959+0000 7f256dffb640 1 -- 172.21.15.77:0/923323210 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f257c0d4020 con 0x7f257c0d0a80 2024-03-06T07:34:12.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.124+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f257c04f330 con 0x7f256006b4d0 2024-03-06T07:34:12.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.125+0000 7f256dffb640 1 -- 172.21.15.77:0/923323210 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13433 (secure 0 0 0) 0x7f257c04f330 con 0x7f256006b4d0 2024-03-06T07:34:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:12.127 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:34:12.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f256006b4d0 msgr2=0x7f256006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f256006b4d0 0x7f256006d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2570000f60 tx=0x7f2570009040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f257c0d0a80 msgr2=0x7f257c0b2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f257c0d0a80 0x7f257c0b2250 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f2578000c00 tx=0x7f25780046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 shutdown_connections 2024-03-06T07:34:12.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f256006b4d0 0x7f256006d990 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f257c0d0a80 0x7f257c0b2250 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f257c0c7910 0x7f257c0a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 --2- 172.21.15.77:0/923323210 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f257c0b4170 0x7f257c15dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 >> 172.21.15.77:0/923323210 conn(0x7f257c0d3c40 msgr2=0x7f257c06ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 shutdown_connections 2024-03-06T07:34:12.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.127+0000 7f25835ac640 1 -- 172.21.15.77:0/923323210 wait complete. 2024-03-06T07:34:12.140 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-03-06T07:34:11.518106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":318908,"kb_used_data":15388,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754655812,"statfs":{"total":773094113280,"available":772767551488,"internally_reserved":0,"allocated":15757312,"data_stored":7574632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":310693424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31593,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-140,"num_write_kb":-79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001131"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:34:06.568083+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:34:06.568083+0000","last_peered":"2024-03-06T07:34:06.568083+0000","last_clean":"2024-03-06T07:34:06.568083+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:34:06.568083+0000","last_undegraded":"2024-03-06T07:34:06.568083+0000","last_fullsized":"2024-03-06T07:34:06.568083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46028,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325812,"statfs":{"total":96636764160,"available":96589631488,"internally_reserved":0,"allocated":2424832,"data_stored":1392887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":44693962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28796,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343044,"statfs":{"total":96636764160,"available":96607277056,"internally_reserved":0,"allocated":1818624,"data_stored":792852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574998,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45412,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326428,"statfs":{"total":96636764160,"available":96590262272,"internally_reserved":0,"allocated":1794048,"data_stored":785864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574997,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":792124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574998,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41312,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330528,"statfs":{"total":96636764160,"available":96594460672,"internally_reserved":0,"allocated":1855488,"data_stored":822227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574998,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41272,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330568,"statfs":{"total":96636764160,"available":96594501632,"internally_reserved":0,"allocated":1814528,"data_stored":792818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2428928,"data_stored":1402884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:12.140 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:34:12.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:12.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.238+0000 7f6ed845c640 1 Processor -- start 2024-03-06T07:34:12.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.238+0000 7f6ed845c640 1 -- start start 2024-03-06T07:34:12.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.239+0000 7f6ed845c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 0x7f6ed00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.239+0000 7f6ed845c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ed00cf830 0x7f6ed00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.239+0000 7f6ed845c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed00b3430 con 0x7f6ed00d27f0 2024-03-06T07:34:12.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.239+0000 7f6ed845c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed00d2c50 con 0x7f6ed00cf830 2024-03-06T07:34:12.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.239+0000 7f6ed845c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00b31e0 con 0x7f6ed015dfe0 2024-03-06T07:34:12.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.239+0000 7f6ed59f7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ed00cf830 0x7f6ed00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.239+0000 7f6ed69f9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6ed00d27f0 0x7f6ed00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59786/0 (socket says 172.21.15.77:59786) 2024-03-06T07:34:12.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.239+0000 7f6ed69f9640 1 -- 172.21.15.77:0/635860605 learned_addr learned my addr 172.21.15.77:0/635860605 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:12.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.240+0000 7f6ed61f8640 1 --2- 172.21.15.77:0/635860605 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 0x7f6ed00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.240+0000 7f6ed59f7640 1 -- 172.21.15.77:0/635860605 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 msgr2=0x7f6ed00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.240+0000 7f6ed59f7640 1 --2- 172.21.15.77:0/635860605 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 0x7f6ed00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.240+0000 7f6ed59f7640 1 -- 172.21.15.77:0/635860605 >> v1:172.21.15.77:6789/0 conn(0x7f6ed00d27f0 legacy=0x7f6ed00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.240+0000 7f6ed59f7640 1 -- 172.21.15.77:0/635860605 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed00b3710 con 0x7f6ed00cf830 2024-03-06T07:34:12.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.240+0000 7f6ed51f6640 1 -- 172.21.15.77:0/635860605 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516661846 0 0) 0x7f6ed00b3430 con 0x7f6ed00d27f0 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.240+0000 7f6ed59f7640 1 --2- 172.21.15.77:0/635860605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00cf830 0x7f6ed00d22b0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0097a0 tx=0x7f6ecc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cf892edf321648 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed51f6640 1 -- 172.21.15.77:0/635860605 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc035020 con 0x7f6ed00cf830 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed51f6640 1 -- 172.21.15.77:0/635860605 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc031830 con 0x7f6ed00cf830 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed51f6640 1 -- 172.21.15.77:0/635860605 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc031c00 con 0x7f6ed00cf830 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed845c640 1 -- 172.21.15.77:0/635860605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00cf830 msgr2=0x7f6ed00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed845c640 1 --2- 172.21.15.77:0/635860605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00cf830 0x7f6ed00d22b0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0097a0 tx=0x7f6ecc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed845c640 1 -- 172.21.15.77:0/635860605 shutdown_connections 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed845c640 1 --2- 172.21.15.77:0/635860605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00cf830 0x7f6ed00d22b0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed845c640 1 --2- 172.21.15.77:0/635860605 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 0x7f6ed00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed845c640 1 -- 172.21.15.77:0/635860605 >> 172.21.15.77:0/635860605 conn(0x7f6ed00d3cd0 msgr2=0x7f6ed00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed845c640 1 -- 172.21.15.77:0/635860605 shutdown_connections 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.241+0000 7f6ed845c640 1 -- 172.21.15.77:0/635860605 wait complete. 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.242+0000 7f6ed845c640 1 Processor -- start 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.242+0000 7f6ed845c640 1 -- start start 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed845c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ed00cf830 0x7f6ed00b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed845c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00d27f0 0x7f6ed00b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed845c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 0x7f6ed0150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed845c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00d1c70 con 0x7f6ed00d27f0 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed845c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00b3d60 con 0x7f6ed00cf830 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed845c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00d1df0 con 0x7f6ed015dfe0 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed59f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00d27f0 0x7f6ed00b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed59f7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00d27f0 0x7f6ed00b8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45744/0 (socket says 172.21.15.77:45744) 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed59f7640 1 -- 172.21.15.77:0/3867495889 learned_addr learned my addr 172.21.15.77:0/3867495889 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed59f7640 1 -- 172.21.15.77:0/3867495889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 msgr2=0x7f6ed0150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed59f7640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 0x7f6ed0150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed59f7640 1 -- 172.21.15.77:0/3867495889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ed00cf830 msgr2=0x7f6ed00b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.243+0000 7f6ed59f7640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ed00cf830 0x7f6ed00b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.244+0000 7f6ed59f7640 1 -- 172.21.15.77:0/3867495889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed015a570 con 0x7f6ed00d27f0 2024-03-06T07:34:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.244+0000 7f6ed59f7640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00d27f0 0x7f6ed00b8680 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc02bee0 tx=0x7f6ecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.244+0000 7f6ec6ffd640 1 -- 172.21.15.77:0/3867495889 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc041070 con 0x7f6ed00d27f0 2024-03-06T07:34:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.244+0000 7f6ec6ffd640 1 -- 172.21.15.77:0/3867495889 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc03b920 con 0x7f6ed00d27f0 2024-03-06T07:34:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.244+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed00cdc20 con 0x7f6ed00d27f0 2024-03-06T07:34:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.244+0000 7f6ec6ffd640 1 -- 172.21.15.77:0/3867495889 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc035040 con 0x7f6ed00d27f0 2024-03-06T07:34:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.244+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed00b95a0 con 0x7f6ed00d27f0 2024-03-06T07:34:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.245+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed004f330 con 0x7f6ed00d27f0 2024-03-06T07:34:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.245+0000 7f6ec6ffd640 1 -- 172.21.15.77:0/3867495889 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ecc04c060 con 0x7f6ed00d27f0 2024-03-06T07:34:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.246+0000 7f6ec6ffd640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6eb406b4d0 0x7f6eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.246+0000 7f6ec6ffd640 1 -- 172.21.15.77:0/3867495889 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6ecc0ab8e0 con 0x7f6ed00d27f0 2024-03-06T07:34:12.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.246+0000 7f6ed61f8640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6eb406b4d0 0x7f6eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.246+0000 7f6ed61f8640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6eb406b4d0 0x7f6eb406d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0006fd0 tx=0x7f6ec0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.250+0000 7f6ec6ffd640 1 -- 172.21.15.77:0/3867495889 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ecc07aa10 con 0x7f6ed00d27f0 2024-03-06T07:34:12.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.418+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ed00ce9e0 con 0x7f6eb406b4d0 2024-03-06T07:34:12.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.419+0000 7f6ec6ffd640 1 -- 172.21.15.77:0/3867495889 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13433 (secure 0 0 0) 0x7f6ed00ce9e0 con 0x7f6eb406b4d0 2024-03-06T07:34:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:12.421 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:34:12.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6eb406b4d0 msgr2=0x7f6eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6eb406b4d0 0x7f6eb406d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0006fd0 tx=0x7f6ec0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00d27f0 msgr2=0x7f6ed00b8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00d27f0 0x7f6ed00b8680 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc02bee0 tx=0x7f6ecc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 shutdown_connections 2024-03-06T07:34:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6eb406b4d0 0x7f6eb406d990 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ed00d27f0 0x7f6ed00b8680 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6ed015dfe0 0x7f6ed0150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 --2- 172.21.15.77:0/3867495889 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6ed00cf830 0x7f6ed00b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 >> 172.21.15.77:0/3867495889 conn(0x7f6ed00d3cd0 msgr2=0x7f6ed00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 shutdown_connections 2024-03-06T07:34:12.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.421+0000 7f6ed845c640 1 -- 172.21.15.77:0/3867495889 wait complete. 2024-03-06T07:34:12.433 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-03-06T07:34:11.518106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":318908,"kb_used_data":15388,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754655812,"statfs":{"total":773094113280,"available":772767551488,"internally_reserved":0,"allocated":15757312,"data_stored":7574632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":310693424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31593,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-140,"num_write_kb":-79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001131"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-06T07:34:06.568083+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:34:06.568083+0000","last_peered":"2024-03-06T07:34:06.568083+0000","last_clean":"2024-03-06T07:34:06.568083+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:34:06.568083+0000","last_undegraded":"2024-03-06T07:34:06.568083+0000","last_fullsized":"2024-03-06T07:34:06.568083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46028,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325812,"statfs":{"total":96636764160,"available":96589631488,"internally_reserved":0,"allocated":2424832,"data_stored":1392887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":44693962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28796,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343044,"statfs":{"total":96636764160,"available":96607277056,"internally_reserved":0,"allocated":1818624,"data_stored":792852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574998,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45412,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326428,"statfs":{"total":96636764160,"available":96590262272,"internally_reserved":0,"allocated":1794048,"data_stored":785864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574997,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":792124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41276,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330564,"statfs":{"total":96636764160,"available":96594497536,"internally_reserved":0,"allocated":1818624,"data_stored":792976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574998,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41312,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330528,"statfs":{"total":96636764160,"available":96594460672,"internally_reserved":0,"allocated":1855488,"data_stored":822227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574998,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41272,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330568,"statfs":{"total":96636764160,"available":96594501632,"internally_reserved":0,"allocated":1814528,"data_stored":792818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2428928,"data_stored":1402884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:12.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:12.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.532+0000 7f49c1281640 1 Processor -- start 2024-03-06T07:34:12.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.532+0000 7f49c1281640 1 -- start start 2024-03-06T07:34:12.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.533+0000 7f49c1281640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 0x7f49bc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.533+0000 7f49c1281640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49bc0d0a80 0x7f49bc0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.533+0000 7f49c1281640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc0d25d0 con 0x7f49bc0d4760 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.533+0000 7f49c1281640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49bc0d4bf0 con 0x7f49bc0d0a80 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.533+0000 7f49c1281640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc0ce9e0 con 0x7f49bc0b4170 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.533+0000 7f49bb7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49bc0d0a80 0x7f49bc0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.533+0000 7f49bb7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f49bc0d0a80 0x7f49bc0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45758/0 (socket says 172.21.15.77:45758) 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.533+0000 7f49bb7fe640 1 -- 172.21.15.77:0/817085087 learned_addr learned my addr 172.21.15.77:0/817085087 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.534+0000 7f49baffd640 1 --2- 172.21.15.77:0/817085087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 0x7f49bc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.534+0000 7f49bb7fe640 1 -- 172.21.15.77:0/817085087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 msgr2=0x7f49bc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.534+0000 7f49bb7fe640 1 --2- 172.21.15.77:0/817085087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 0x7f49bc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.534+0000 7f49bb7fe640 1 -- 172.21.15.77:0/817085087 >> v1:172.21.15.158:6789/0 conn(0x7f49bc0d4760 legacy=0x7f49bc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.534+0000 7f49bb7fe640 1 -- 172.21.15.77:0/817085087 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc0d2870 con 0x7f49bc0d0a80 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.534+0000 7f49bb7fe640 1 --2- 172.21.15.77:0/817085087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0d0a80 0x7f49bc0cee70 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009870 tx=0x7f49b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0b05f6b540f20b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49b9ffb640 1 -- 172.21.15.77:0/817085087 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4035020 con 0x7f49bc0d0a80 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49b9ffb640 1 -- 172.21.15.77:0/817085087 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b4031640 con 0x7f49bc0d0a80 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49b9ffb640 1 -- 172.21.15.77:0/817085087 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b402d040 con 0x7f49bc0d0a80 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49c1281640 1 -- 172.21.15.77:0/817085087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0d0a80 msgr2=0x7f49bc0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49c1281640 1 --2- 172.21.15.77:0/817085087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0d0a80 0x7f49bc0cee70 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009870 tx=0x7f49b402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49c1281640 1 -- 172.21.15.77:0/817085087 shutdown_connections 2024-03-06T07:34:12.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49c1281640 1 --2- 172.21.15.77:0/817085087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0d0a80 0x7f49bc0cee70 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49c1281640 1 --2- 172.21.15.77:0/817085087 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 0x7f49bc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49c1281640 1 -- 172.21.15.77:0/817085087 >> 172.21.15.77:0/817085087 conn(0x7f49bc0d3c40 msgr2=0x7f49bc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49c1281640 1 -- 172.21.15.77:0/817085087 shutdown_connections 2024-03-06T07:34:12.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.535+0000 7f49c1281640 1 -- 172.21.15.77:0/817085087 wait complete. 2024-03-06T07:34:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49c1281640 1 Processor -- start 2024-03-06T07:34:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49c1281640 1 -- start start 2024-03-06T07:34:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49c1281640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 0x7f49bc151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49c1281640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49bc0d0a80 0x7f49bc0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49c1281640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0c80f0 0x7f49bc0c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49c1281640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc0b4dd0 con 0x7f49bc0c80f0 2024-03-06T07:34:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49c1281640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc0b4550 con 0x7f49bc0d0a80 2024-03-06T07:34:12.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49c1281640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc0b4f50 con 0x7f49bc0b4170 2024-03-06T07:34:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0c80f0 0x7f49bc0c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0c80f0 0x7f49bc0c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45764/0 (socket says 172.21.15.77:45764) 2024-03-06T07:34:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.537+0000 7f49bb7fe640 1 -- 172.21.15.77:0/1910744671 learned_addr learned my addr 172.21.15.77:0/1910744671 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49baffd640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 0x7f49bc151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49ba7fc640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49bc0d0a80 0x7f49bc0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49bb7fe640 1 -- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 msgr2=0x7f49bc151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49bb7fe640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 0x7f49bc151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49bb7fe640 1 -- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49bc0d0a80 msgr2=0x7f49bc0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49bb7fe640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49bc0d0a80 0x7f49bc0ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49bb7fe640 1 -- 172.21.15.77:0/1910744671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc0ca640 con 0x7f49bc0c80f0 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49ba7fc640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49bc0d0a80 0x7f49bc0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49bb7fe640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0c80f0 0x7f49bc0c8570 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f49b402b930 tx=0x7f49b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49a37fe640 1 -- 172.21.15.77:0/1910744671 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4040070 con 0x7f49bc0c80f0 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49a37fe640 1 -- 172.21.15.77:0/1910744671 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b40399f0 con 0x7f49bc0c80f0 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49bc0c8ab0 con 0x7f49bc0c80f0 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49a37fe640 1 -- 172.21.15.77:0/1910744671 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4035040 con 0x7f49bc0c80f0 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.538+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49bc0c9030 con 0x7f49bc0c80f0 2024-03-06T07:34:12.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.539+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49bc0caae0 con 0x7f49bc0c80f0 2024-03-06T07:34:12.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.540+0000 7f49a37fe640 1 -- 172.21.15.77:0/1910744671 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49b404b060 con 0x7f49bc0c80f0 2024-03-06T07:34:12.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.540+0000 7f49a37fe640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f49a806b4d0 0x7f49a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.540+0000 7f49a37fe640 1 -- 172.21.15.77:0/1910744671 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f49b40ab7b0 con 0x7f49bc0c80f0 2024-03-06T07:34:12.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.540+0000 7f49baffd640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f49a806b4d0 0x7f49a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.540+0000 7f49baffd640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f49a806b4d0 0x7f49a806d990 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f49ac003b80 tx=0x7f49ac0098d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.544+0000 7f49a37fe640 1 -- 172.21.15.77:0/1910744671 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49bc0caae0 con 0x7f49bc0c80f0 2024-03-06T07:34:12.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.716+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49bc0b3ab0 con 0x7f49bc0c80f0 2024-03-06T07:34:12.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.717+0000 7f49a37fe640 1 -- 172.21.15.77:0/1910744671 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+13146 (secure 0 0 0) 0x7f49bc0b3ab0 con 0x7f49bc0c80f0 2024-03-06T07:34:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:12.719 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":146,"fsid":"eee1dc2e-9e65-49b7-aabb-947580759696","created":"2024-03-06T07:21:41.248229+0000","modified":"2024-03-06T07:34:09.529645+0000","last_up_change":"2024-03-06T07:21:51.514235+0000","last_in_change":"2024-03-06T07:21:43.506795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:53.425208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d18ca08-fdf5-44b6-9cc6-a264001ea4fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6809","nonce":3119591583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6811","nonce":3119591583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6815","nonce":3119591583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3119591583},{"type":"v1","addr":"172.21.15.77:6813","nonce":3119591583}]},"public_addr":"172.21.15.77:6809/3119591583","cluster_addr":"172.21.15.77:6811/3119591583","heartbeat_back_addr":"172.21.15.77:6815/3119591583","heartbeat_front_addr":"172.21.15.77:6813/3119591583","state":["exists","up"]},{"osd":1,"uuid":"22f22df8-eb96-4e3b-a851-2e51f91abc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6801","nonce":1774367160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6803","nonce":1774367160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6807","nonce":1774367160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1774367160},{"type":"v1","addr":"172.21.15.77:6805","nonce":1774367160}]},"public_addr":"172.21.15.77:6801/1774367160","cluster_addr":"172.21.15.77:6803/1774367160","heartbeat_back_addr":"172.21.15.77:6807/1774367160","heartbeat_front_addr":"172.21.15.77:6805/1774367160","state":["exists","up"]},{"osd":2,"uuid":"069ab0e2-cfb9-4f92-a8af-e473269ccb97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6825","nonce":107698445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6827","nonce":107698445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6831","nonce":107698445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":107698445},{"type":"v1","addr":"172.21.15.77:6829","nonce":107698445}]},"public_addr":"172.21.15.77:6825/107698445","cluster_addr":"172.21.15.77:6827/107698445","heartbeat_back_addr":"172.21.15.77:6831/107698445","heartbeat_front_addr":"172.21.15.77:6829/107698445","state":["exists","up"]},{"osd":3,"uuid":"11d3ed9f-c0f4-4c2c-950a-46bfbba6ff20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6817","nonce":461341877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6819","nonce":461341877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6823","nonce":461341877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":461341877},{"type":"v1","addr":"172.21.15.77:6821","nonce":461341877}]},"public_addr":"172.21.15.77:6817/461341877","cluster_addr":"172.21.15.77:6819/461341877","heartbeat_back_addr":"172.21.15.77:6823/461341877","heartbeat_front_addr":"172.21.15.77:6821/461341877","state":["exists","up"]},{"osd":4,"uuid":"b25fa93b-6204-41c7-8853-cd24c0c8f832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6802","nonce":2421130833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6806","nonce":2421130833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6814","nonce":2421130833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":2421130833},{"type":"v1","addr":"172.21.15.158:6810","nonce":2421130833}]},"public_addr":"172.21.15.158:6802/2421130833","cluster_addr":"172.21.15.158:6806/2421130833","heartbeat_back_addr":"172.21.15.158:6814/2421130833","heartbeat_front_addr":"172.21.15.158:6810/2421130833","state":["exists","up"]},{"osd":5,"uuid":"9e451073-42e6-4695-a7e1-d5dc0b6108d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6822","nonce":3476303095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6826","nonce":3476303095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6831","nonce":3476303095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":3476303095},{"type":"v1","addr":"172.21.15.158:6829","nonce":3476303095}]},"public_addr":"172.21.15.158:6822/3476303095","cluster_addr":"172.21.15.158:6826/3476303095","heartbeat_back_addr":"172.21.15.158:6831/3476303095","heartbeat_front_addr":"172.21.15.158:6829/3476303095","state":["exists","up"]},{"osd":6,"uuid":"e460d8c9-90e8-400a-8934-b5654984cebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6801","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6803","nonce":2971141295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6805","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6807","nonce":2971141295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6813","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6815","nonce":2971141295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6809","nonce":2971141295},{"type":"v1","addr":"172.21.15.158:6811","nonce":2971141295}]},"public_addr":"172.21.15.158:6803/2971141295","cluster_addr":"172.21.15.158:6807/2971141295","heartbeat_back_addr":"172.21.15.158:6815/2971141295","heartbeat_front_addr":"172.21.15.158:6811/2971141295","state":["exists","up"]},{"osd":7,"uuid":"4784eeb3-95ef-461e-b695-6294c46b6e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6817","nonce":3281717214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6819","nonce":3281717214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6825","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6827","nonce":3281717214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6821","nonce":3281717214},{"type":"v1","addr":"172.21.15.158:6823","nonce":3281717214}]},"public_addr":"172.21.15.158:6817/3281717214","cluster_addr":"172.21.15.158:6819/3281717214","heartbeat_back_addr":"172.21.15.158:6827/3281717214","heartbeat_front_addr":"172.21.15.158:6823/3281717214","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.661310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.630645+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.645950+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.638429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.719665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.705078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.589669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:48.619928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6835/669492439":"2024-03-07T07:34:06.521397+0000","172.21.15.77:6837/2581202217":"2024-03-07T07:34:06.521409+0000","172.21.15.77:6834/669492439":"2024-03-07T07:34:06.521397+0000","172.21.15.77:6836/2581202217":"2024-03-07T07:34:06.521409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:12.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.718+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f49a806b4d0 msgr2=0x7f49a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.718+0000 7f49c1281640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f49a806b4d0 0x7f49a806d990 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f49ac003b80 tx=0x7f49ac0098d0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.718+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0c80f0 msgr2=0x7f49bc0c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.718+0000 7f49c1281640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0c80f0 0x7f49bc0c8570 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f49b402b930 tx=0x7f49b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.719+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 shutdown_connections 2024-03-06T07:34:12.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.719+0000 7f49c1281640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f49a806b4d0 0x7f49a806d990 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.719+0000 7f49c1281640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49bc0c80f0 0x7f49bc0c8570 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.719+0000 7f49c1281640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f49bc0d0a80 0x7f49bc0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.719+0000 7f49c1281640 1 --2- 172.21.15.77:0/1910744671 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f49bc0b4170 0x7f49bc151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.719+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 >> 172.21.15.77:0/1910744671 conn(0x7f49bc0d3c40 msgr2=0x7f49bc0d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.719+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 shutdown_connections 2024-03-06T07:34:12.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:12.719+0000 7f49c1281640 1 -- 172.21.15.77:0/1910744671 wait complete. 2024-03-06T07:34:13.732 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:34:13.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:13.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.842+0000 7f1dfe8b5640 1 Processor -- start 2024-03-06T07:34:13.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.842+0000 7f1dfe8b5640 1 -- start start 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1dfe8b5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1df80708b0 0x7f1df8070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1dfe8b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 0x7f1df80ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1dfe8b5640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df815afe0 con 0x7f1df815ac00 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1dfe8b5640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df8071250 con 0x7f1df80708b0 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1dfe8b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80cdc10 con 0x7f1df80ce0a0 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1df80708b0 0x7f1df8070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df77fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1df80708b0 0x7f1df8070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45776/0 (socket says 172.21.15.77:45776) 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1df815ac00 0x7f1df806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59788/0 (socket says 172.21.15.77:59788) 2024-03-06T07:34:13.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df77fe640 1 -- 172.21.15.77:0/3720848753 learned_addr learned my addr 172.21.15.77:0/3720848753 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:13.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1dfce52640 1 --2- 172.21.15.77:0/3720848753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 0x7f1df80ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df77fe640 1 -- 172.21.15.77:0/3720848753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 msgr2=0x7f1df80ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df77fe640 1 --2- 172.21.15.77:0/3720848753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 0x7f1df80ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df77fe640 1 -- 172.21.15.77:0/3720848753 >> v1:172.21.15.77:6789/0 conn(0x7f1df815ac00 legacy=0x7f1df806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df77fe640 1 -- 172.21.15.77:0/3720848753 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df80cd000 con 0x7f1df80708b0 2024-03-06T07:34:13.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1df6ffd640 1 -- 172.21.15.77:0/3720848753 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319526913 0 0) 0x7f1df815afe0 con 0x7f1df815ac00 2024-03-06T07:34:13.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.843+0000 7f1dfce52640 1 --2- 172.21.15.77:0/3720848753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 0x7f1df80ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.844+0000 7f1df77fe640 1 --2- 172.21.15.77:0/3720848753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 0x7f1df8070d10 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00bb10 tx=0x7f1dec02e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e049df32a29b3be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.844+0000 7f1df6ffd640 1 -- 172.21.15.77:0/3720848753 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec03b070 con 0x7f1df80708b0 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.844+0000 7f1df6ffd640 1 -- 172.21.15.77:0/3720848753 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dec036440 con 0x7f1df80708b0 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.844+0000 7f1df6ffd640 1 -- 172.21.15.77:0/3720848753 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec036810 con 0x7f1df80708b0 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.844+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/3720848753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 msgr2=0x7f1df8070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.844+0000 7f1dfe8b5640 1 --2- 172.21.15.77:0/3720848753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 0x7f1df8070d10 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00bb10 tx=0x7f1dec02e0c0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.845+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/3720848753 shutdown_connections 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.845+0000 7f1dfe8b5640 1 --2- 172.21.15.77:0/3720848753 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 0x7f1df80ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.845+0000 7f1dfe8b5640 1 --2- 172.21.15.77:0/3720848753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 0x7f1df8070d10 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.845+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/3720848753 >> 172.21.15.77:0/3720848753 conn(0x7f1df80b1650 msgr2=0x7f1df80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.845+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/3720848753 shutdown_connections 2024-03-06T07:34:13.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.845+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/3720848753 wait complete. 2024-03-06T07:34:13.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.846+0000 7f1dfe8b5640 1 Processor -- start 2024-03-06T07:34:13.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.846+0000 7f1dfe8b5640 1 -- start start 2024-03-06T07:34:13.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1dfe8b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 0x7f1df814a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1dfe8b5640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 0x7f1df814d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1dfe8b5640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1df815ac00 0x7f1df80b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1dfe8b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80cd8a0 con 0x7f1df80708b0 2024-03-06T07:34:13.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1dfe8b5640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80cd5c0 con 0x7f1df815ac00 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1dfe8b5640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80cf5d0 con 0x7f1df80ce0a0 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 0x7f1df814a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 0x7f1df814a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45782/0 (socket says 172.21.15.77:45782) 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 -- 172.21.15.77:0/1728101628 learned_addr learned my addr 172.21.15.77:0/1728101628 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df77fe640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 0x7f1df814d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 -- 172.21.15.77:0/1728101628 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 msgr2=0x7f1df814d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f1df80ce0a0 0x7f1df814d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 -- 172.21.15.77:0/1728101628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1df815ac00 msgr2=0x7f1df80b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1df815ac00 0x7f1df80b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 -- 172.21.15.77:0/1728101628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df814e1a0 con 0x7f1df80708b0 2024-03-06T07:34:13.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df7fff640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 0x7f1df814a320 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f1de800c430 tx=0x7f1de800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80074e0 con 0x7f1df80708b0 2024-03-06T07:34:13.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de80047e0 con 0x7f1df80708b0 2024-03-06T07:34:13.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8004bd0 con 0x7f1df80708b0 2024-03-06T07:34:13.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df81ef870 con 0x7f1df80708b0 2024-03-06T07:34:13.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.847+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df81ef520 con 0x7f1df80708b0 2024-03-06T07:34:13.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.848+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1df804f330 con 0x7f1df80708b0 2024-03-06T07:34:13.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.849+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1de8011070 con 0x7f1df80708b0 2024-03-06T07:34:13.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.849+0000 7f1df4ff9640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1de406b500 0x7f1de406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.849+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1de8089b90 con 0x7f1df80708b0 2024-03-06T07:34:13.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.849+0000 7f1df77fe640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1de406b500 0x7f1de406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.849+0000 7f1df4ff9640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1de4071460 0x7f1de4073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.849+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1de4073f30 con 0x7f1de4071460 2024-03-06T07:34:13.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.849+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1de808c290 con 0x7f1df80708b0 2024-03-06T07:34:13.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.849+0000 7f1dfce52640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1de4071460 0x7f1de4073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.850+0000 7f1df77fe640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1de406b500 0x7f1de406d9c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f1dec038000 tx=0x7f1dec032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.850+0000 7f1dfce52640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1de4071460 0x7f1de4073880 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.850+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 <== osd.0 v2:172.21.15.77:6808/3119591583 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1de4073f30 con 0x7f1de4071460 2024-03-06T07:34:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.863+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 --> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1df81eff70 con 0x7f1de4071460 2024-03-06T07:34:13.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.863+0000 7f1df4ff9640 1 -- 172.21.15.77:0/1728101628 <== osd.0 v2:172.21.15.77:6808/3119591583 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1df81eff70 con 0x7f1de4071460 2024-03-06T07:34:13.865 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:34:13.866 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:13.866 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:34:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1de4071460 msgr2=0x7f1de4073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6808/3119591583,v1:172.21.15.77:6809/3119591583] conn(0x7f1de4071460 0x7f1de4073880 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1de406b500 msgr2=0x7f1de406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f1de406b500 0x7f1de406d9c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f1dec038000 tx=0x7f1dec032000 comp rx=0 tx=0).stop 2024-03-06T07:34:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 msgr2=0x7f1df814a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 --2- 172.21.15.77:0/1728101628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df80708b0 0x7f1df814a320 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f1de800c430 tx=0x7f1de800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfce52640 1 -- 172.21.15.77:0/1728101628 reap_dead start 2024-03-06T07:34:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 shutdown_connections 2024-03-06T07:34:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 >> 172.21.15.77:0/1728101628 conn(0x7f1df80b1650 msgr2=0x7f1df8153e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 shutdown_connections 2024-03-06T07:34:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.864+0000 7f1dfe8b5640 1 -- 172.21.15.77:0/1728101628 wait complete. 2024-03-06T07:34:13.876 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:34:13.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.975+0000 7fad8d7e9640 1 Processor -- start 2024-03-06T07:34:13.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.975+0000 7fad8d7e9640 1 -- start start 2024-03-06T07:34:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad8d7e9640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fad880b4460 0x7fad880d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad8d7e9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad88162970 0x7fad880b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad8d7e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad88161cf0 0x7fad88162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad8d7e9640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad88161b00 con 0x7fad88162970 2024-03-06T07:34:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad8d7e9640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad880cecc0 con 0x7fad880b4460 2024-03-06T07:34:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad8d7e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad880d22c0 con 0x7fad88161cf0 2024-03-06T07:34:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad867fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad88162970 0x7fad880b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad867fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad88162970 0x7fad880b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45786/0 (socket says 172.21.15.77:45786) 2024-03-06T07:34:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.976+0000 7fad867fc640 1 -- 172.21.15.77:0/3395497340 learned_addr learned my addr 172.21.15.77:0/3395497340 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad867fc640 1 -- 172.21.15.77:0/3395497340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad88161cf0 msgr2=0x7fad88162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad867fc640 1 --2- 172.21.15.77:0/3395497340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad88161cf0 0x7fad88162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad867fc640 1 -- 172.21.15.77:0/3395497340 >> v2:172.21.15.158:3300/0 conn(0x7fad880b4460 msgr2=0x7fad880d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad867fc640 1 --2- 172.21.15.77:0/3395497340 >> v2:172.21.15.158:3300/0 conn(0x7fad880b4460 0x7fad880d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad867fc640 1 -- 172.21.15.77:0/3395497340 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad880d25a0 con 0x7fad88162970 2024-03-06T07:34:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad867fc640 1 --2- 172.21.15.77:0/3395497340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88162970 0x7fad880b3580 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fad8808ddd0 tx=0x7fad7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66bd67ae9f0bf75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad85ffb640 1 -- 172.21.15.77:0/3395497340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c035020 con 0x7fad88162970 2024-03-06T07:34:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad85ffb640 1 -- 172.21.15.77:0/3395497340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c031870 con 0x7fad88162970 2024-03-06T07:34:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.977+0000 7fad85ffb640 1 -- 172.21.15.77:0/3395497340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c031c40 con 0x7fad88162970 2024-03-06T07:34:13.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 -- 172.21.15.77:0/3395497340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88162970 msgr2=0x7fad880b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/3395497340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88162970 0x7fad880b3580 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fad8808ddd0 tx=0x7fad7c02b910 comp rx=0 tx=0).stop 2024-03-06T07:34:13.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 -- 172.21.15.77:0/3395497340 shutdown_connections 2024-03-06T07:34:13.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/3395497340 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad88161cf0 0x7fad88162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/3395497340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88162970 0x7fad880b3580 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/3395497340 >> v2:172.21.15.158:3300/0 conn(0x7fad880b4460 0x7fad880d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 -- 172.21.15.77:0/3395497340 >> 172.21.15.77:0/3395497340 conn(0x7fad880d3ce0 msgr2=0x7fad880d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 -- 172.21.15.77:0/3395497340 shutdown_connections 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.978+0000 7fad8d7e9640 1 -- 172.21.15.77:0/3395497340 wait complete. 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.979+0000 7fad8d7e9640 1 Processor -- start 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.979+0000 7fad8d7e9640 1 -- start start 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad8d7e9640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad880b4460 0x7fad880b7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad8d7e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88161cf0 0x7fad880b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad8d7e9640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad88162970 0x7fad880c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad8d7e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad880ce950 con 0x7fad88161cf0 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad8d7e9640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad880d2b60 con 0x7fad88162970 2024-03-06T07:34:13.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad8d7e9640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad880b3ac0 con 0x7fad880b4460 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad867fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88161cf0 0x7fad880b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad867fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88161cf0 0x7fad880b6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45794/0 (socket says 172.21.15.77:45794) 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad867fc640 1 -- 172.21.15.77:0/2773302776 learned_addr learned my addr 172.21.15.77:0/2773302776 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad86ffd640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad880b4460 0x7fad880b7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad877fe640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad88162970 0x7fad880c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad867fc640 1 -- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad880b4460 msgr2=0x7fad880b7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad867fc640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad880b4460 0x7fad880b7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad867fc640 1 -- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad88162970 msgr2=0x7fad880c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad867fc640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad88162970 0x7fad880c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad867fc640 1 -- 172.21.15.77:0/2773302776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad880b8ca0 con 0x7fad88161cf0 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.980+0000 7fad877fe640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad88162970 0x7fad880c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.981+0000 7fad867fc640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88161cf0 0x7fad880b6da0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fad7c02b950 tx=0x7fad7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.981+0000 7fad6f7fe640 1 -- 172.21.15.77:0/2773302776 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c03bd70 con 0x7fad88161cf0 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.981+0000 7fad6f7fe640 1 -- 172.21.15.77:0/2773302776 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c0453f0 con 0x7fad88161cf0 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.981+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad880b8f80 con 0x7fad88161cf0 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.981+0000 7fad6f7fe640 1 -- 172.21.15.77:0/2773302776 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0457e0 con 0x7fad88161cf0 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.981+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad880b77d0 con 0x7fad88161cf0 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.982+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad880b4840 con 0x7fad88161cf0 2024-03-06T07:34:13.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.982+0000 7fad6f7fe640 1 -- 172.21.15.77:0/2773302776 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad7c04c060 con 0x7fad88161cf0 2024-03-06T07:34:13.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.983+0000 7fad6f7fe640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad7406b4d0 0x7fad7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.983+0000 7fad6f7fe640 1 -- 172.21.15.77:0/2773302776 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fad7c0ac160 con 0x7fad88161cf0 2024-03-06T07:34:13.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.983+0000 7fad86ffd640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad7406b4d0 0x7fad7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.983+0000 7fad86ffd640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad7406b4d0 0x7fad7406d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fad7800a400 tx=0x7fad7800d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:13.987+0000 7fad6f7fe640 1 -- 172.21.15.77:0/2773302776 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad880b4840 con 0x7fad88161cf0 2024-03-06T07:34:14.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.158+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fad880b72e0 con 0x7fad7406b4d0 2024-03-06T07:34:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.158+0000 7fad6f7fe640 1 -- 172.21.15.77:0/2773302776 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fad880b72e0 con 0x7fad7406b4d0 2024-03-06T07:34:14.161 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:34:14.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad7406b4d0 msgr2=0x7fad7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad7406b4d0 0x7fad7406d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fad7800a400 tx=0x7fad7800d040 comp rx=0 tx=0).stop 2024-03-06T07:34:14.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88161cf0 msgr2=0x7fad880b6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88161cf0 0x7fad880b6da0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fad7c02b950 tx=0x7fad7c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 shutdown_connections 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fad7406b4d0 0x7fad7406d990 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad88161cf0 0x7fad880b6da0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad88162970 0x7fad880c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 --2- 172.21.15.77:0/2773302776 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fad880b4460 0x7fad880b7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 >> 172.21.15.77:0/2773302776 conn(0x7fad880d3ce0 msgr2=0x7fad8815fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 shutdown_connections 2024-03-06T07:34:14.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.160+0000 7fad8d7e9640 1 -- 172.21.15.77:0/2773302776 wait complete. 2024-03-06T07:34:14.173 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:34:14.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:14.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.276+0000 7fc36cae0640 1 Processor -- start 2024-03-06T07:34:14.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.276+0000 7fc36cae0640 1 -- start start 2024-03-06T07:34:14.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc36cae0640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fc3680708b0 0x7fc368070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc36cae0640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36815afe0 con 0x7fc36815ac00 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc36cae0640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3680cd000 con 0x7fc3680ce0a0 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc36cae0640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc368071250 con 0x7fc3680708b0 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc36659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc36815ac00 0x7fc36806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59800/0 (socket says 172.21.15.77:59800) 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc36659c640 1 -- 172.21.15.77:0/2840150246 learned_addr learned my addr 172.21.15.77:0/2840150246 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc35ffff640 1 --2- 172.21.15.77:0/2840150246 >> v2:172.21.15.158:3300/0 conn(0x7fc3680708b0 0x7fc368070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443084328 0 0) 0x7fc36815afe0 con 0x7fc36815ac00 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc354003520 con 0x7fc36815ac00 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1028743109 0 0) 0x7fc354003520 con 0x7fc36815ac00 2024-03-06T07:34:14.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 >> v2:172.21.15.158:3300/0 conn(0x7fc3680708b0 msgr2=0x7fc368070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 --2- 172.21.15.77:0/2840150246 >> v2:172.21.15.158:3300/0 conn(0x7fc3680708b0 0x7fc368070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 >> v1:172.21.15.158:6789/0 conn(0x7fc3680ce0a0 legacy=0x7fc3680ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3680cd2e0 con 0x7fc36815ac00 2024-03-06T07:34:14.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fc358001670 con 0x7fc36815ac00 2024-03-06T07:34:14.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 >> v1:172.21.15.77:6789/0 conn(0x7fc36815ac00 legacy=0x7fc36806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3540039b0 0x7fc354003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc354004800 0x7fc354004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3540051c0 0x7fc35400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.277+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400c130 con 0x7fc354004800 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400c2e0 con 0x7fc3540039b0 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400c4e0 con 0x7fc3540051c0 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc35ffff640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc354004800 0x7fc354004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc36659c640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3540039b0 0x7fc354003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc366d9d640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3540051c0 0x7fc35400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc35ffff640 1 -- 172.21.15.77:0/2840150246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3540051c0 msgr2=0x7fc35400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc35ffff640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3540051c0 0x7fc35400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc35ffff640 1 -- 172.21.15.77:0/2840150246 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3540039b0 msgr2=0x7fc354003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc35ffff640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc3540039b0 0x7fc354003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc35ffff640 1 -- 172.21.15.77:0/2840150246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354003520 con 0x7fc354004800 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.278+0000 7fc366d9d640 1 -- 172.21.15.77:0/2840150246 reap_dead start 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc35ffff640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc354004800 0x7fc354004c80 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fc350009a90 tx=0x7fc354005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5c01353b39dc7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350039070 con 0x7fc354004800 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35002c680 con 0x7fc354004800 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc365d9b640 1 -- 172.21.15.77:0/2840150246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350009690 con 0x7fc354004800 2024-03-06T07:34:14.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc36cae0640 1 -- 172.21.15.77:0/2840150246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc354004800 msgr2=0x7fc354004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc36cae0640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc354004800 0x7fc354004c80 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fc350009a90 tx=0x7fc354005830 comp rx=0 tx=0).stop 2024-03-06T07:34:14.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc36cae0640 1 -- 172.21.15.77:0/2840150246 shutdown_connections 2024-03-06T07:34:14.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc36cae0640 1 --2- 172.21.15.77:0/2840150246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc354004800 0x7fc354004c80 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc36cae0640 1 -- 172.21.15.77:0/2840150246 >> 172.21.15.77:0/2840150246 conn(0x7fc3680b1650 msgr2=0x7fc3680b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc36cae0640 1 -- 172.21.15.77:0/2840150246 shutdown_connections 2024-03-06T07:34:14.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.279+0000 7fc36cae0640 1 -- 172.21.15.77:0/2840150246 wait complete. 2024-03-06T07:34:14.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc36cae0640 1 Processor -- start 2024-03-06T07:34:14.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc36cae0640 1 -- start start 2024-03-06T07:34:14.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc36cae0640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc354004800 0x7fc368154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc36cae0640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3680aebf0 0x7fc3681f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc36cae0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc36815d230 0x7fc36815d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc36cae0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680cd8a0 con 0x7fc36815d230 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc36cae0640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680cdc10 con 0x7fc354004800 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc36cae0640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680cf5d0 con 0x7fc3680aebf0 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc366d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc36815d230 0x7fc36815d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc366d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc36815d230 0x7fc36815d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45816/0 (socket says 172.21.15.77:45816) 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.281+0000 7fc366d9d640 1 -- 172.21.15.77:0/3626480541 learned_addr learned my addr 172.21.15.77:0/3626480541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:14.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc366d9d640 1 -- 172.21.15.77:0/3626480541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3680aebf0 msgr2=0x7fc3681f4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc35ffff640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3680aebf0 0x7fc3681f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc36659c640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc354004800 0x7fc368154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc366d9d640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fc3680aebf0 0x7fc3681f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc366d9d640 1 -- 172.21.15.77:0/3626480541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc354004800 msgr2=0x7fc368154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc366d9d640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc354004800 0x7fc368154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc366d9d640 1 -- 172.21.15.77:0/3626480541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3680ad990 con 0x7fc36815d230 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc36659c640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc354004800 0x7fc368154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc366d9d640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc36815d230 0x7fc36815d6e0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fc3680cbb30 tx=0x7fc36000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360003970 con 0x7fc36815d230 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc360018070 con 0x7fc36815d230 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3680adc70 con 0x7fc36815d230 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360003d70 con 0x7fc36815d230 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.282+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3680ae230 con 0x7fc36815d230 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.283+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc36804f330 con 0x7fc36815d230 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.283+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3600050d0 con 0x7fc36815d230 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc35effd640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc34806b4d0 0x7fc34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc36008a570 con 0x7fc36815d230 2024-03-06T07:34:14.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc36659c640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc34806b4d0 0x7fc34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc35effd640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7fc348071430 0x7fc348073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc360024e70 con 0x7fc348071430 2024-03-06T07:34:14.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc36008aa00 con 0x7fc36815d230 2024-03-06T07:34:14.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc35ffff640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7fc348071430 0x7fc348073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc36659c640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc34806b4d0 0x7fc34806d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc368075540 tx=0x7fc358009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.284+0000 7fc35ffff640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7fc348071430 0x7fc348073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.285+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 <== osd.1 v2:172.21.15.77:6800/1774367160 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc360024e70 con 0x7fc348071430 2024-03-06T07:34:14.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.298+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 --> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc3680ae680 con 0x7fc348071430 2024-03-06T07:34:14.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.298+0000 7fc35effd640 1 -- 172.21.15.77:0/3626480541 <== osd.1 v2:172.21.15.77:6800/1774367160 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc3680ae680 con 0x7fc348071430 2024-03-06T07:34:14.300 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:34:14.300 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:14.301 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:34:14.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.298+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7fc348071430 msgr2=0x7fc348073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.298+0000 7fc36cae0640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6800/1774367160,v1:172.21.15.77:6801/1774367160] conn(0x7fc348071430 0x7fc348073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc34806b4d0 msgr2=0x7fc34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc36cae0640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fc34806b4d0 0x7fc34806d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc368075540 tx=0x7fc358009040 comp rx=0 tx=0).stop 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc36815d230 msgr2=0x7fc36815d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc36cae0640 1 --2- 172.21.15.77:0/3626480541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc36815d230 0x7fc36815d6e0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fc3680cbb30 tx=0x7fc36000eb90 comp rx=0 tx=0).stop 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc366d9d640 1 -- 172.21.15.77:0/3626480541 reap_dead start 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 shutdown_connections 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 >> 172.21.15.77:0/3626480541 conn(0x7fc3680b1650 msgr2=0x7fc3680b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 shutdown_connections 2024-03-06T07:34:14.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.299+0000 7fc36cae0640 1 -- 172.21.15.77:0/3626480541 wait complete. 2024-03-06T07:34:14.311 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:34:14.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.410+0000 7f484666f640 1 Processor -- start 2024-03-06T07:34:14.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.410+0000 7f484666f640 1 -- start start 2024-03-06T07:34:14.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.410+0000 7f484666f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48401444c0 0x7f484006ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f484666f640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f4840070fe0 0x7f48400cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f484666f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484006d370 con 0x7f484015b2d0 2024-03-06T07:34:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f484666f640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48401448a0 con 0x7f4840070fe0 2024-03-06T07:34:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f484666f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484006d650 con 0x7f48401444c0 2024-03-06T07:34:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483f7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f484015b2d0 0x7f48400708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59804/0 (socket says 172.21.15.77:59804) 2024-03-06T07:34:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483f7fe640 1 -- 172.21.15.77:0/2662679011 learned_addr learned my addr 172.21.15.77:0/2662679011 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340822335 0 0) 0x7f484006d370 con 0x7f484015b2d0 2024-03-06T07:34:14.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483ffff640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48401444c0 0x7f484006ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f4844c0c640 1 --2- 172.21.15.77:0/2662679011 >> v2:172.21.15.158:3300/0 conn(0x7f4840070fe0 0x7f48400cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4828005730 con 0x7f484015b2d0 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2682580194 0 0) 0x7f4828005730 con 0x7f484015b2d0 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48401444c0 msgr2=0x7f484006ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48401444c0 0x7f484006ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 >> v2:172.21.15.158:3300/0 conn(0x7f4840070fe0 msgr2=0x7f48400cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 --2- 172.21.15.77:0/2662679011 >> v2:172.21.15.158:3300/0 conn(0x7f4840070fe0 0x7f48400cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484006d930 con 0x7f484015b2d0 2024-03-06T07:34:14.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f4834001670 con 0x7f484015b2d0 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 >> v1:172.21.15.77:6789/0 conn(0x7f484015b2d0 legacy=0x7f48400708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828005bc0 0x7f4828005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48401444c0 0x7f4828006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4828006f50 0x7f48280073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48401448a0 con 0x7f4828005bc0 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484006d650 con 0x7f4828006f50 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482800dbd0 con 0x7f48401444c0 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.411+0000 7f483ffff640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828005bc0 0x7f4828005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.412+0000 7f483f7fe640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48401444c0 0x7f4828006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.412+0000 7f4844c0c640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4828006f50 0x7f48280073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.412+0000 7f483ffff640 1 -- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48401444c0 msgr2=0x7f4828006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.412+0000 7f483ffff640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f48401444c0 0x7f4828006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.412+0000 7f483ffff640 1 -- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4828006f50 msgr2=0x7f48280073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.412+0000 7f483ffff640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4828006f50 0x7f48280073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.412+0000 7f483ffff640 1 -- 172.21.15.77:0/2662679011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4828005730 con 0x7f4828005bc0 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.412+0000 7f483ffff640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828005bc0 0x7f4828005fe0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f4830009ac0 tx=0x7f4828007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e4a008e5fd44f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830039070 con 0x7f4828005bc0 2024-03-06T07:34:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483002c6b0 con 0x7f4828005bc0 2024-03-06T07:34:14.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f483effd640 1 -- 172.21.15.77:0/2662679011 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830009690 con 0x7f4828005bc0 2024-03-06T07:34:14.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f484666f640 1 -- 172.21.15.77:0/2662679011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828005bc0 msgr2=0x7f4828005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f484666f640 1 --2- 172.21.15.77:0/2662679011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4828005bc0 0x7f4828005fe0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f4830009ac0 tx=0x7f4828007a40 comp rx=0 tx=0).stop 2024-03-06T07:34:14.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f4844c0c640 1 -- 172.21.15.77:0/2662679011 reap_dead start 2024-03-06T07:34:14.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f484666f640 1 -- 172.21.15.77:0/2662679011 shutdown_connections 2024-03-06T07:34:14.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f484666f640 1 -- 172.21.15.77:0/2662679011 >> 172.21.15.77:0/2662679011 conn(0x7f48400b1c80 msgr2=0x7f48400cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f484666f640 1 -- 172.21.15.77:0/2662679011 shutdown_connections 2024-03-06T07:34:14.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.413+0000 7f484666f640 1 -- 172.21.15.77:0/2662679011 wait complete. 2024-03-06T07:34:14.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.414+0000 7f484666f640 1 Processor -- start 2024-03-06T07:34:14.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.414+0000 7f484666f640 1 -- start start 2024-03-06T07:34:14.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f484666f640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4828005bc0 0x7f48400cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f484666f640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48401f3e60 0x7f48401f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f484666f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48401400f0 0x7f4840140560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f484666f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484006ca00 con 0x7f48401400f0 2024-03-06T07:34:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f484666f640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484006def0 con 0x7f48401f3e60 2024-03-06T07:34:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f484666f640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484006cb80 con 0x7f4828005bc0 2024-03-06T07:34:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f4844c0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48401400f0 0x7f4840140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f4844c0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48401400f0 0x7f4840140560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45828/0 (socket says 172.21.15.77:45828) 2024-03-06T07:34:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f4844c0c640 1 -- 172.21.15.77:0/1496686321 learned_addr learned my addr 172.21.15.77:0/1496686321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f483ffff640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4828005bc0 0x7f48400cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.415+0000 7f483f7fe640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48401f3e60 0x7f48401f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f4844c0c640 1 -- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4828005bc0 msgr2=0x7f48400cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f4844c0c640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4828005bc0 0x7f48400cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f4844c0c640 1 -- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48401f3e60 msgr2=0x7f48401f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f4844c0c640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48401f3e60 0x7f48401f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f4844c0c640 1 -- 172.21.15.77:0/1496686321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48401f4bb0 con 0x7f48401400f0 2024-03-06T07:34:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f483ffff640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4828005bc0 0x7f48400cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f4844c0c640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48401400f0 0x7f4840140560 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f48400737d0 tx=0x7f483800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f483cff9640 1 -- 172.21.15.77:0/1496686321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483800fde0 con 0x7f48401400f0 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f483cff9640 1 -- 172.21.15.77:0/1496686321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4838015430 con 0x7f48401400f0 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484015d4e0 con 0x7f48401400f0 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f483cff9640 1 -- 172.21.15.77:0/1496686321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838015820 con 0x7f48401400f0 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.416+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484015daa0 con 0x7f48401400f0 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.417+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484004f330 con 0x7f48401400f0 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.417+0000 7f483cff9640 1 -- 172.21.15.77:0/1496686321 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48380159c0 con 0x7f48401400f0 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.418+0000 7f483cff9640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f482c06b4d0 0x7f482c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.418+0000 7f483cff9640 1 -- 172.21.15.77:0/1496686321 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f483808ac80 con 0x7f48401400f0 2024-03-06T07:34:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.418+0000 7f483ffff640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f482c06b4d0 0x7f482c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.418+0000 7f483ffff640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f482c06b4d0 0x7f482c06d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4830036000 tx=0x7f4830030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.422+0000 7f483cff9640 1 -- 172.21.15.77:0/1496686321 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4838019030 con 0x7f48401400f0 2024-03-06T07:34:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.593+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f48400cc970 con 0x7f482c06b4d0 2024-03-06T07:34:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.594+0000 7f483cff9640 1 -- 172.21.15.77:0/1496686321 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f48400cc970 con 0x7f482c06b4d0 2024-03-06T07:34:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:34:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f482c06b4d0 msgr2=0x7f482c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f482c06b4d0 0x7f482c06d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4830036000 tx=0x7f4830030000 comp rx=0 tx=0).stop 2024-03-06T07:34:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48401400f0 msgr2=0x7f4840140560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48401400f0 0x7f4840140560 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f48400737d0 tx=0x7f483800f580 comp rx=0 tx=0).stop 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 shutdown_connections 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f482c06b4d0 0x7f482c06d990 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48401400f0 0x7f4840140560 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f48401f3e60 0x7f48401f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 --2- 172.21.15.77:0/1496686321 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f4828005bc0 0x7f48400cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 >> 172.21.15.77:0/1496686321 conn(0x7f48400b1c80 msgr2=0x7f48400b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 shutdown_connections 2024-03-06T07:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.596+0000 7f484666f640 1 -- 172.21.15.77:0/1496686321 wait complete. 2024-03-06T07:34:14.609 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:34:14.609 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:14.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.711+0000 7fe2d3f63640 1 Processor -- start 2024-03-06T07:34:14.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.711+0000 7fe2d3f63640 1 -- start start 2024-03-06T07:34:14.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d3f63640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fe2cc162470 0x7fe2cc06f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d3f63640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2cc162850 0x7fe2cc0d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d3f63640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2cc0cf3c0 con 0x7fe2cc06fc40 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d3f63640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc06f270 con 0x7fe2cc162850 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d3f63640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc0cf0e0 con 0x7fe2cc162470 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d14fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2cc162850 0x7fe2cc0d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d2500640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe2cc06fc40 0x7fe2cc06c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59806/0 (socket says 172.21.15.77:59806) 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d14fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2cc162850 0x7fe2cc0d1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45834/0 (socket says 172.21.15.77:45834) 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.712+0000 7fe2d2500640 1 -- 172.21.15.77:0/318866584 learned_addr learned my addr 172.21.15.77:0/318866584 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.713+0000 7fe2d1cff640 1 --2- 172.21.15.77:0/318866584 >> v2:172.21.15.158:3300/0 conn(0x7fe2cc162470 0x7fe2cc06f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.713+0000 7fe2d14fe640 1 -- 172.21.15.77:0/318866584 >> v2:172.21.15.158:3300/0 conn(0x7fe2cc162470 msgr2=0x7fe2cc06f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.713+0000 7fe2d14fe640 1 --2- 172.21.15.77:0/318866584 >> v2:172.21.15.158:3300/0 conn(0x7fe2cc162470 0x7fe2cc06f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.713+0000 7fe2d14fe640 1 -- 172.21.15.77:0/318866584 >> v1:172.21.15.77:6789/0 conn(0x7fe2cc06fc40 legacy=0x7fe2cc06c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.713+0000 7fe2d14fe640 1 -- 172.21.15.77:0/318866584 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc06ec20 con 0x7fe2cc162850 2024-03-06T07:34:14.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.713+0000 7fe2d0cfd640 1 -- 172.21.15.77:0/318866584 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314569986 0 0) 0x7fe2cc0cf3c0 con 0x7fe2cc06fc40 2024-03-06T07:34:14.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.713+0000 7fe2d14fe640 1 --2- 172.21.15.77:0/318866584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162850 0x7fe2cc0d1a50 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc06f3f0 tx=0x7fe2bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0e558dddedc805 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.713+0000 7fe2d0cfd640 1 -- 172.21.15.77:0/318866584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc035020 con 0x7fe2cc162850 2024-03-06T07:34:14.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d0cfd640 1 -- 172.21.15.77:0/318866584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc031830 con 0x7fe2cc162850 2024-03-06T07:34:14.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d0cfd640 1 -- 172.21.15.77:0/318866584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc031c00 con 0x7fe2cc162850 2024-03-06T07:34:14.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d3f63640 1 -- 172.21.15.77:0/318866584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162850 msgr2=0x7fe2cc0d1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d3f63640 1 --2- 172.21.15.77:0/318866584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162850 0x7fe2cc0d1a50 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc06f3f0 tx=0x7fe2bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d3f63640 1 -- 172.21.15.77:0/318866584 shutdown_connections 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d3f63640 1 --2- 172.21.15.77:0/318866584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162850 0x7fe2cc0d1a50 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d3f63640 1 --2- 172.21.15.77:0/318866584 >> v2:172.21.15.158:3300/0 conn(0x7fe2cc162470 0x7fe2cc06f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d3f63640 1 -- 172.21.15.77:0/318866584 >> 172.21.15.77:0/318866584 conn(0x7fe2cc09a9f0 msgr2=0x7fe2cc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d3f63640 1 -- 172.21.15.77:0/318866584 shutdown_connections 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.714+0000 7fe2d3f63640 1 -- 172.21.15.77:0/318866584 wait complete. 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.715+0000 7fe2d3f63640 1 Processor -- start 2024-03-06T07:34:14.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.715+0000 7fe2d3f63640 1 -- start start 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d3f63640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2cc06fc40 0x7fe2cc151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d3f63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162470 0x7fe2cc0cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d3f63640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2cc162850 0x7fe2cc0cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d3f63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc06cc40 con 0x7fe2cc162470 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d3f63640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc06cac0 con 0x7fe2cc162850 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d3f63640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0d1f90 con 0x7fe2cc06fc40 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d14fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162470 0x7fe2cc0cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d14fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162470 0x7fe2cc0cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45844/0 (socket says 172.21.15.77:45844) 2024-03-06T07:34:14.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d14fe640 1 -- 172.21.15.77:0/3021332751 learned_addr learned my addr 172.21.15.77:0/3021332751 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d2500640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2cc162850 0x7fe2cc0cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d1cff640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2cc06fc40 0x7fe2cc151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d14fe640 1 -- 172.21.15.77:0/3021332751 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2cc06fc40 msgr2=0x7fe2cc151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d14fe640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fe2cc06fc40 0x7fe2cc151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d14fe640 1 -- 172.21.15.77:0/3021332751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2cc162850 msgr2=0x7fe2cc0cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d14fe640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2cc162850 0x7fe2cc0cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d14fe640 1 -- 172.21.15.77:0/3021332751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc0c64b0 con 0x7fe2cc162470 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.716+0000 7fe2d2500640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe2cc162850 0x7fe2cc0cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:14.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.717+0000 7fe2d14fe640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162470 0x7fe2cc0cd470 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc034040 tx=0x7fe2bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.717+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc035040 con 0x7fe2cc162470 2024-03-06T07:34:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.717+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc004230 con 0x7fe2cc162470 2024-03-06T07:34:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.717+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc0c6790 con 0x7fe2cc162470 2024-03-06T07:34:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.717+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc03a700 con 0x7fe2cc162470 2024-03-06T07:34:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.717+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc0c6d50 con 0x7fe2cc162470 2024-03-06T07:34:14.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.717+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2cc04f330 con 0x7fe2cc162470 2024-03-06T07:34:14.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.718+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2bc038020 con 0x7fe2cc162470 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.718+0000 7fe2c27fc640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2b006b4d0 0x7fe2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.719+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe2bc0abd60 con 0x7fe2cc162470 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.719+0000 7fe2d1cff640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2b006b4d0 0x7fe2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.719+0000 7fe2c27fc640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fe2b0071430 0x7fe2b0073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.719+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 --> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2bc001d30 con 0x7fe2b0071430 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.719+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2bc0ae460 con 0x7fe2cc162470 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.719+0000 7fe2d2500640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fe2b0071430 0x7fe2b0073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.719+0000 7fe2d1cff640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2b006b4d0 0x7fe2b006d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009a60 tx=0x7fe2c8003990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.719+0000 7fe2d2500640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fe2b0071430 0x7fe2b0073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.720+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 <== osd.2 v2:172.21.15.77:6824/107698445 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe2bc001d30 con 0x7fe2b0071430 2024-03-06T07:34:14.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 --> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe2cc06c190 con 0x7fe2b0071430 2024-03-06T07:34:14.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2c27fc640 1 -- 172.21.15.77:0/3021332751 <== osd.2 v2:172.21.15.77:6824/107698445 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe2cc06c190 con 0x7fe2b0071430 2024-03-06T07:34:14.735 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:34:14.735 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:14.735 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:34:14.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fe2b0071430 msgr2=0x7fe2b0073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6824/107698445,v1:172.21.15.77:6825/107698445] conn(0x7fe2b0071430 0x7fe2b0073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2b006b4d0 msgr2=0x7fe2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fe2b006b4d0 0x7fe2b006d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009a60 tx=0x7fe2c8003990 comp rx=0 tx=0).stop 2024-03-06T07:34:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162470 msgr2=0x7fe2cc0cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 --2- 172.21.15.77:0/3021332751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2cc162470 0x7fe2cc0cd470 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc034040 tx=0x7fe2bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d2500640 1 -- 172.21.15.77:0/3021332751 reap_dead start 2024-03-06T07:34:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 shutdown_connections 2024-03-06T07:34:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 >> 172.21.15.77:0/3021332751 conn(0x7fe2cc09a9f0 msgr2=0x7fe2cc09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 shutdown_connections 2024-03-06T07:34:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.733+0000 7fe2d3f63640 1 -- 172.21.15.77:0/3021332751 wait complete. 2024-03-06T07:34:14.745 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:34:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.844+0000 7f6306765640 1 Processor -- start 2024-03-06T07:34:14.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.844+0000 7f6306765640 1 -- start start 2024-03-06T07:34:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f6306765640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f630015bed0 0x7f63000b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f6306765640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63000d04e0 0x7f63000d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f6306765640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63000b2d80 con 0x7f63000d1fe0 2024-03-06T07:34:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f6306765640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63000d2470 con 0x7f63000d04e0 2024-03-06T07:34:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f6306765640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63000ae9c0 con 0x7f630015bed0 2024-03-06T07:34:14.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f62ff7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63000d04e0 0x7f63000d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f62ff7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63000d04e0 0x7f63000d1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45850/0 (socket says 172.21.15.77:45850) 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f62ff7fe640 1 -- 172.21.15.77:0/743029180 learned_addr learned my addr 172.21.15.77:0/743029180 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.845+0000 7f6304d02640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f63000d1fe0 0x7f630015fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59812/0 (socket says 172.21.15.77:59812) 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62ff7fe640 1 -- 172.21.15.77:0/743029180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f630015bed0 msgr2=0x7f63000b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62fffff640 1 --2- 172.21.15.77:0/743029180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f630015bed0 0x7f63000b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62ff7fe640 1 --2- 172.21.15.77:0/743029180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f630015bed0 0x7f63000b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62feffd640 1 -- 172.21.15.77:0/743029180 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830402366 0 0) 0x7f63000b2d80 con 0x7f63000d1fe0 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62ff7fe640 1 -- 172.21.15.77:0/743029180 >> v1:172.21.15.77:6789/0 conn(0x7f63000d1fe0 legacy=0x7f630015fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62ff7fe640 1 -- 172.21.15.77:0/743029180 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63000aeca0 con 0x7f63000d04e0 2024-03-06T07:34:14.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62ff7fe640 1 --2- 172.21.15.77:0/743029180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f63000d1aa0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009870 tx=0x7f62f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfdf8de41c43ca5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62feffd640 1 -- 172.21.15.77:0/743029180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4035020 con 0x7f63000d04e0 2024-03-06T07:34:14.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62feffd640 1 -- 172.21.15.77:0/743029180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4031640 con 0x7f63000d04e0 2024-03-06T07:34:14.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.846+0000 7f62feffd640 1 -- 172.21.15.77:0/743029180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f402d040 con 0x7f63000d04e0 2024-03-06T07:34:14.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.847+0000 7f6306765640 1 -- 172.21.15.77:0/743029180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 msgr2=0x7f63000d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.847+0000 7f6306765640 1 --2- 172.21.15.77:0/743029180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f63000d1aa0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009870 tx=0x7f62f402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.847+0000 7f6306765640 1 -- 172.21.15.77:0/743029180 shutdown_connections 2024-03-06T07:34:14.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.847+0000 7f6306765640 1 --2- 172.21.15.77:0/743029180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f63000d1aa0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.847+0000 7f6306765640 1 --2- 172.21.15.77:0/743029180 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f630015bed0 0x7f63000b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.847+0000 7f6306765640 1 -- 172.21.15.77:0/743029180 >> 172.21.15.77:0/743029180 conn(0x7f630009a9f0 msgr2=0x7f630009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.847+0000 7f6306765640 1 -- 172.21.15.77:0/743029180 shutdown_connections 2024-03-06T07:34:14.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.847+0000 7f6306765640 1 -- 172.21.15.77:0/743029180 wait complete. 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.848+0000 7f6306765640 1 Processor -- start 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.848+0000 7f6306765640 1 -- start start 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f6306765640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f630006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f6306765640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63000d1fe0 0x7f63001f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f6306765640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f630015bed0 0x7f63000c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f6306765640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630015c2b0 con 0x7f63000d04e0 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f6306765640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63000af260 con 0x7f630015bed0 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f6306765640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630015d610 con 0x7f63000d1fe0 2024-03-06T07:34:14.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f630006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62fffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f630006db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45862/0 (socket says 172.21.15.77:45862) 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62fffff640 1 -- 172.21.15.77:0/3050671058 learned_addr learned my addr 172.21.15.77:0/3050671058 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f6304d02640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f630015bed0 0x7f63000c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62fffff640 1 -- 172.21.15.77:0/3050671058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63000d1fe0 msgr2=0x7f63001f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62ff7fe640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63000d1fe0 0x7f63001f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62fffff640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63000d1fe0 0x7f63001f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62fffff640 1 -- 172.21.15.77:0/3050671058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f630015bed0 msgr2=0x7f63000c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62fffff640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f630015bed0 0x7f63000c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.849+0000 7f62fffff640 1 -- 172.21.15.77:0/3050671058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630006e0b0 con 0x7f63000d04e0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.850+0000 7f62fffff640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f630006db70 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f62f000c7f0 tx=0x7f62f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.850+0000 7f62fcff9640 1 -- 172.21.15.77:0/3050671058 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0015070 con 0x7f63000d04e0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.850+0000 7f62fcff9640 1 -- 172.21.15.77:0/3050671058 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f0004220 con 0x7f63000d04e0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.850+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63001f47f0 con 0x7f63000d04e0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.850+0000 7f62fcff9640 1 -- 172.21.15.77:0/3050671058 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0003610 con 0x7f63000d04e0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.850+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630015d860 con 0x7f63000d04e0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.851+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630006e860 con 0x7f63000d04e0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.851+0000 7f62fcff9640 1 -- 172.21.15.77:0/3050671058 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62f00039c0 con 0x7f63000d04e0 2024-03-06T07:34:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.852+0000 7f62fcff9640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62ec06b4d0 0x7f62ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.852+0000 7f62fcff9640 1 -- 172.21.15.77:0/3050671058 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f62f008a210 con 0x7f63000d04e0 2024-03-06T07:34:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.852+0000 7f62ff7fe640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62ec06b4d0 0x7f62ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.852+0000 7f62ff7fe640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62ec06b4d0 0x7f62ec06d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f62f4036000 tx=0x7f62f4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:14.856+0000 7f62fcff9640 1 -- 172.21.15.77:0/3050671058 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f630006e860 con 0x7f63000d04e0 2024-03-06T07:34:15.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.026+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f630004f330 con 0x7f62ec06b4d0 2024-03-06T07:34:15.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.027+0000 7f62fcff9640 1 -- 172.21.15.77:0/3050671058 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f630004f330 con 0x7f62ec06b4d0 2024-03-06T07:34:15.028 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:34:15.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62ec06b4d0 msgr2=0x7f62ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62ec06b4d0 0x7f62ec06d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f62f4036000 tx=0x7f62f4008030 comp rx=0 tx=0).stop 2024-03-06T07:34:15.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 msgr2=0x7f630006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f630006db70 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f62f000c7f0 tx=0x7f62f000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 shutdown_connections 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f630015bed0 0x7f63000c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f63000d1fe0 0x7f63001f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f62ec06b4d0 0x7f62ec06d990 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 --2- 172.21.15.77:0/3050671058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000d04e0 0x7f630006db70 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.028+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 >> 172.21.15.77:0/3050671058 conn(0x7f630009a9f0 msgr2=0x7f6300070460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.029+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 shutdown_connections 2024-03-06T07:34:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.029+0000 7f6306765640 1 -- 172.21.15.77:0/3050671058 wait complete. 2024-03-06T07:34:15.043 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:34:15.044 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:15.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.145+0000 7f8a9e70c640 1 Processor -- start 2024-03-06T07:34:15.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.146+0000 7f8a9e70c640 1 -- start start 2024-03-06T07:34:15.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.146+0000 7f8a9e70c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9815dfa0 0x7f8a980d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.146+0000 7f8a9e70c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a980d4630 0x7f8a980d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.146+0000 7f8a9e70c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9815e380 con 0x7f8a980d2820 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.146+0000 7f8a9e70c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a980d2100 con 0x7f8a980d4630 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.146+0000 7f8a9e70c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980d1e30 con 0x7f8a9815dfa0 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.146+0000 7f8a977fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a980d4630 0x7f8a980d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.146+0000 7f8a9cca9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8a980d2820 0x7f8a980d05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59816/0 (socket says 172.21.15.77:59816) 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.147+0000 7f8a9cca9640 1 -- 172.21.15.77:0/3504607981 learned_addr learned my addr 172.21.15.77:0/3504607981 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.147+0000 7f8a977fe640 1 -- 172.21.15.77:0/3504607981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9815dfa0 msgr2=0x7f8a980d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.147+0000 7f8a97fff640 1 --2- 172.21.15.77:0/3504607981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9815dfa0 0x7f8a980d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.147+0000 7f8a96ffd640 1 -- 172.21.15.77:0/3504607981 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260686952 0 0) 0x7f8a9815e380 con 0x7f8a980d2820 2024-03-06T07:34:15.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.147+0000 7f8a977fe640 1 --2- 172.21.15.77:0/3504607981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9815dfa0 0x7f8a980d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.147+0000 7f8a977fe640 1 -- 172.21.15.77:0/3504607981 >> v1:172.21.15.77:6789/0 conn(0x7f8a980d2820 legacy=0x7f8a980d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.147+0000 7f8a977fe640 1 -- 172.21.15.77:0/3504607981 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a980cf470 con 0x7f8a980d4630 2024-03-06T07:34:15.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a977fe640 1 --2- 172.21.15.77:0/3504607981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d4630 0x7f8a980d22e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0097a0 tx=0x7f8a8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93730b89fe2b56ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a96ffd640 1 -- 172.21.15.77:0/3504607981 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c0315a0 con 0x7f8a980d4630 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a96ffd640 1 -- 172.21.15.77:0/3504607981 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c037070 con 0x7f8a980d4630 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a96ffd640 1 -- 172.21.15.77:0/3504607981 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c031960 con 0x7f8a980d4630 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a9e70c640 1 -- 172.21.15.77:0/3504607981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d4630 msgr2=0x7f8a980d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a9e70c640 1 --2- 172.21.15.77:0/3504607981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d4630 0x7f8a980d22e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0097a0 tx=0x7f8a8c02b840 comp rx=0 tx=0).stop 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a9e70c640 1 -- 172.21.15.77:0/3504607981 shutdown_connections 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a9e70c640 1 --2- 172.21.15.77:0/3504607981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d4630 0x7f8a980d22e0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a9e70c640 1 --2- 172.21.15.77:0/3504607981 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a9815dfa0 0x7f8a980d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a9e70c640 1 -- 172.21.15.77:0/3504607981 >> 172.21.15.77:0/3504607981 conn(0x7f8a9809d000 msgr2=0x7f8a980d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a9e70c640 1 -- 172.21.15.77:0/3504607981 shutdown_connections 2024-03-06T07:34:15.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.148+0000 7f8a9e70c640 1 -- 172.21.15.77:0/3504607981 wait complete. 2024-03-06T07:34:15.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.149+0000 7f8a9e70c640 1 Processor -- start 2024-03-06T07:34:15.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a9e70c640 1 -- start start 2024-03-06T07:34:15.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a9e70c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d2820 0x7f8a9814fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a9e70c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a980d4630 0x7f8a9809f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a9e70c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9815dfa0 0x7f8a9809da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a9e70c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980cc1a0 con 0x7f8a980d2820 2024-03-06T07:34:15.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a9e70c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980cfa30 con 0x7f8a9815dfa0 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a9e70c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980cc320 con 0x7f8a980d4630 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d2820 0x7f8a9814fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d2820 0x7f8a9814fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45874/0 (socket says 172.21.15.77:45874) 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.150+0000 7f8a97fff640 1 -- 172.21.15.77:0/154647075 learned_addr learned my addr 172.21.15.77:0/154647075 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a977fe640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a980d4630 0x7f8a9809f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a97fff640 1 -- 172.21.15.77:0/154647075 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a980d4630 msgr2=0x7f8a9809f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a97fff640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f8a980d4630 0x7f8a9809f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a9cca9640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9815dfa0 0x7f8a9809da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a97fff640 1 -- 172.21.15.77:0/154647075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9815dfa0 msgr2=0x7f8a9809da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a97fff640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8a9815dfa0 0x7f8a9809da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a97fff640 1 -- 172.21.15.77:0/154647075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9814cf00 con 0x7f8a980d2820 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a97fff640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d2820 0x7f8a9814fa90 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c7f0 tx=0x7f8a8800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88015070 con 0x7f8a980d2820 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a88004240 con 0x7f8a980d2820 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9814d1e0 con 0x7f8a980d2820 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88007620 con 0x7f8a980d2820 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.151+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9814d7a0 con 0x7f8a980d2820 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.152+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a9804f330 con 0x7f8a980d2820 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.152+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a880079d0 con 0x7f8a980d2820 2024-03-06T07:34:15.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.153+0000 7f8a94ff9640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a8406b4d0 0x7f8a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.153+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8a8808a3e0 con 0x7f8a980d2820 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.153+0000 7f8a977fe640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a8406b4d0 0x7f8a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.153+0000 7f8a94ff9640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f8a84071430 0x7f8a84073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.153+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a84073f00 con 0x7f8a84071430 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.153+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a88091050 con 0x7f8a980d2820 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.153+0000 7f8a9cca9640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f8a84071430 0x7f8a84073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.153+0000 7f8a977fe640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a8406b4d0 0x7f8a8406d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c036000 tx=0x7f8a8c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.154+0000 7f8a9cca9640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f8a84071430 0x7f8a84073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.154+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 <== osd.3 v2:172.21.15.77:6816/461341877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8a84073f00 con 0x7f8a84071430 2024-03-06T07:34:15.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.167+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 --> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8a980d18a0 con 0x7f8a84071430 2024-03-06T07:34:15.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.167+0000 7f8a94ff9640 1 -- 172.21.15.77:0/154647075 <== osd.3 v2:172.21.15.77:6816/461341877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8a980d18a0 con 0x7f8a84071430 2024-03-06T07:34:15.169 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:34:15.169 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:15.169 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:34:15.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.167+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f8a84071430 msgr2=0x7f8a84073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.167+0000 7f8a9e70c640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6816/461341877,v1:172.21.15.77:6817/461341877] conn(0x7f8a84071430 0x7f8a84073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a8406b4d0 msgr2=0x7f8a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9e70c640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f8a8406b4d0 0x7f8a8406d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c036000 tx=0x7f8a8c008030 comp rx=0 tx=0).stop 2024-03-06T07:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d2820 msgr2=0x7f8a9814fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9e70c640 1 --2- 172.21.15.77:0/154647075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a980d2820 0x7f8a9814fa90 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c7f0 tx=0x7f8a8800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9cca9640 1 -- 172.21.15.77:0/154647075 reap_dead start 2024-03-06T07:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 shutdown_connections 2024-03-06T07:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 >> 172.21.15.77:0/154647075 conn(0x7f8a9809d000 msgr2=0x7f8a98160540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 shutdown_connections 2024-03-06T07:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.168+0000 7f8a9e70c640 1 -- 172.21.15.77:0/154647075 wait complete. 2024-03-06T07:34:15.180 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:34:15.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.278+0000 7fdc3a52c640 1 Processor -- start 2024-03-06T07:34:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.278+0000 7fdc3a52c640 1 -- start start 2024-03-06T07:34:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc3a52c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc34162970 0x7fdc340d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc3a52c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdc34161cf0 0x7fdc34162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc3a52c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc340d22c0 con 0x7fdc340b4460 2024-03-06T07:34:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc3a52c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc34161b00 con 0x7fdc34161cf0 2024-03-06T07:34:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc3a52c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340cecc0 con 0x7fdc34162970 2024-03-06T07:34:15.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc38ac9640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdc34161cf0 0x7fdc34162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc33fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdc340b4460 0x7fdc340b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59826/0 (socket says 172.21.15.77:59826) 2024-03-06T07:34:15.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc33fff640 1 -- 172.21.15.77:0/2350470095 learned_addr learned my addr 172.21.15.77:0/2350470095 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:15.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.279+0000 7fdc337fe640 1 --2- 172.21.15.77:0/2350470095 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc34162970 0x7fdc340d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc38ac9640 1 -- 172.21.15.77:0/2350470095 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc34162970 msgr2=0x7fdc340d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc38ac9640 1 --2- 172.21.15.77:0/2350470095 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc34162970 0x7fdc340d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc38ac9640 1 -- 172.21.15.77:0/2350470095 >> v1:172.21.15.77:6789/0 conn(0x7fdc340b4460 legacy=0x7fdc340b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc38ac9640 1 -- 172.21.15.77:0/2350470095 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc340d25a0 con 0x7fdc34161cf0 2024-03-06T07:34:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc32ffd640 1 -- 172.21.15.77:0/2350470095 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586909915 0 0) 0x7fdc340d22c0 con 0x7fdc340b4460 2024-03-06T07:34:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc38ac9640 1 --2- 172.21.15.77:0/2350470095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34161cf0 0x7fdc34162180 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009d90 tx=0x7fdc2c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689ac1a26dd3d404 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc32ffd640 1 -- 172.21.15.77:0/2350470095 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c039070 con 0x7fdc34161cf0 2024-03-06T07:34:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc32ffd640 1 -- 172.21.15.77:0/2350470095 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2c02cc50 con 0x7fdc34161cf0 2024-03-06T07:34:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.280+0000 7fdc32ffd640 1 -- 172.21.15.77:0/2350470095 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c034440 con 0x7fdc34161cf0 2024-03-06T07:34:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.281+0000 7fdc3a52c640 1 -- 172.21.15.77:0/2350470095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34161cf0 msgr2=0x7fdc34162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.281+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/2350470095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34161cf0 0x7fdc34162180 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009d90 tx=0x7fdc2c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.281+0000 7fdc3a52c640 1 -- 172.21.15.77:0/2350470095 shutdown_connections 2024-03-06T07:34:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.281+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/2350470095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34161cf0 0x7fdc34162180 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.281+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/2350470095 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc34162970 0x7fdc340d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.281+0000 7fdc3a52c640 1 -- 172.21.15.77:0/2350470095 >> 172.21.15.77:0/2350470095 conn(0x7fdc340d3ce0 msgr2=0x7fdc340d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.281+0000 7fdc3a52c640 1 -- 172.21.15.77:0/2350470095 shutdown_connections 2024-03-06T07:34:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.281+0000 7fdc3a52c640 1 -- 172.21.15.77:0/2350470095 wait complete. 2024-03-06T07:34:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.282+0000 7fdc3a52c640 1 Processor -- start 2024-03-06T07:34:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.282+0000 7fdc3a52c640 1 -- start start 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc3a52c640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc340b4460 0x7fdc340b7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc3a52c640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc34161cf0 0x7fdc341557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc3a52c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34162970 0x7fdc340b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc3a52c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340ce950 con 0x7fdc34162970 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc3a52c640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340d2b60 con 0x7fdc34161cf0 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc3a52c640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340b4840 con 0x7fdc340b4460 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc38ac9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34162970 0x7fdc340b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc38ac9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34162970 0x7fdc340b81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45904/0 (socket says 172.21.15.77:45904) 2024-03-06T07:34:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc38ac9640 1 -- 172.21.15.77:0/328625357 learned_addr learned my addr 172.21.15.77:0/328625357 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc33fff640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc340b4460 0x7fdc340b7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc337fe640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc34161cf0 0x7fdc341557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc38ac9640 1 -- 172.21.15.77:0/328625357 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc340b4460 msgr2=0x7fdc340b7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc38ac9640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc340b4460 0x7fdc340b7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc38ac9640 1 -- 172.21.15.77:0/328625357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc34161cf0 msgr2=0x7fdc341557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc38ac9640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc34161cf0 0x7fdc341557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.283+0000 7fdc38ac9640 1 -- 172.21.15.77:0/328625357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc340b8c60 con 0x7fdc34162970 2024-03-06T07:34:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.284+0000 7fdc38ac9640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34162970 0x7fdc340b81e0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009d90 tx=0x7fdc2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.284+0000 7fdc30ff9640 1 -- 172.21.15.77:0/328625357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c039050 con 0x7fdc34162970 2024-03-06T07:34:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.284+0000 7fdc30ff9640 1 -- 172.21.15.77:0/328625357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2c03a040 con 0x7fdc34162970 2024-03-06T07:34:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.284+0000 7fdc30ff9640 1 -- 172.21.15.77:0/328625357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c02ca50 con 0x7fdc34162970 2024-03-06T07:34:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.284+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc340b6a30 con 0x7fdc34162970 2024-03-06T07:34:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.284+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc340b6ff0 con 0x7fdc34162970 2024-03-06T07:34:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.285+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3404f330 con 0x7fdc34162970 2024-03-06T07:34:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.285+0000 7fdc30ff9640 1 -- 172.21.15.77:0/328625357 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc2c047030 con 0x7fdc34162970 2024-03-06T07:34:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.285+0000 7fdc30ff9640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc2006b4d0 0x7fdc2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.286+0000 7fdc30ff9640 1 -- 172.21.15.77:0/328625357 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdc2c0abd50 con 0x7fdc34162970 2024-03-06T07:34:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.286+0000 7fdc33fff640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc2006b4d0 0x7fdc2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.286+0000 7fdc33fff640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc2006b4d0 0x7fdc2006d990 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fdc24000f60 tx=0x7fdc24007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.290+0000 7fdc30ff9640 1 -- 172.21.15.77:0/328625357 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc2c0af030 con 0x7fdc34162970 2024-03-06T07:34:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.461+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc340b8f40 con 0x7fdc2006b4d0 2024-03-06T07:34:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.461+0000 7fdc30ff9640 1 -- 172.21.15.77:0/328625357 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdc340b8f40 con 0x7fdc2006b4d0 2024-03-06T07:34:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:34:15.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc2006b4d0 msgr2=0x7fdc2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc2006b4d0 0x7fdc2006d990 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fdc24000f60 tx=0x7fdc24007c00 comp rx=0 tx=0).stop 2024-03-06T07:34:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34162970 msgr2=0x7fdc340b81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34162970 0x7fdc340b81e0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009d90 tx=0x7fdc2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 shutdown_connections 2024-03-06T07:34:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdc2006b4d0 0x7fdc2006d990 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc34162970 0x7fdc340b81e0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc34161cf0 0x7fdc341557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 --2- 172.21.15.77:0/328625357 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc340b4460 0x7fdc340b7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 >> 172.21.15.77:0/328625357 conn(0x7fdc340d3ce0 msgr2=0x7fdc340d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 shutdown_connections 2024-03-06T07:34:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.463+0000 7fdc3a52c640 1 -- 172.21.15.77:0/328625357 wait complete. 2024-03-06T07:34:15.476 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:34:15.476 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.580+0000 7f6a3324e640 1 Processor -- start 2024-03-06T07:34:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.580+0000 7f6a3324e640 1 -- start start 2024-03-06T07:34:15.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a3324e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a2c15c5a0 0x7f6a2c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a3324e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c06ce50 con 0x7f6a2c0ce8c0 2024-03-06T07:34:15.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a3324e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c0ced50 con 0x7f6a2c15c980 2024-03-06T07:34:15.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a3324e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2c072a20 con 0x7f6a2c15c5a0 2024-03-06T07:34:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a30fea640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a2c15c5a0 0x7f6a2c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a30fea640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a2c15c5a0 0x7f6a2c06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45906/0 (socket says 172.21.15.77:45906) 2024-03-06T07:34:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a30fea640 1 -- 172.21.15.77:0/3916108382 learned_addr learned my addr 172.21.15.77:0/3916108382 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a2bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6a2c0ce8c0 0x7f6a2c0cc330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59832/0 (socket says 172.21.15.77:59832) 2024-03-06T07:34:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a30fea640 1 -- 172.21.15.77:0/3916108382 >> v1:172.21.15.158:6789/0 conn(0x7f6a2c15c980 legacy=0x7f6a2c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a30fea640 1 -- 172.21.15.77:0/3916108382 >> v1:172.21.15.77:6789/0 conn(0x7f6a2c0ce8c0 legacy=0x7f6a2c0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a2b7fe640 1 -- 172.21.15.77:0/3916108382 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788595988 0 0) 0x7f6a2c06ce50 con 0x7f6a2c0ce8c0 2024-03-06T07:34:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.581+0000 7f6a30fea640 1 -- 172.21.15.77:0/3916108382 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c0cf030 con 0x7f6a2c15c5a0 2024-03-06T07:34:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.582+0000 7f6a30fea640 1 --2- 172.21.15.77:0/3916108382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c15c5a0 0x7f6a2c06c910 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009870 tx=0x7f6a1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1a24804a3db4be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.582+0000 7f6a2b7fe640 1 -- 172.21.15.77:0/3916108382 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c035020 con 0x7f6a2c15c5a0 2024-03-06T07:34:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.582+0000 7f6a2b7fe640 1 -- 172.21.15.77:0/3916108382 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1c031870 con 0x7f6a2c15c5a0 2024-03-06T07:34:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.582+0000 7f6a2b7fe640 1 -- 172.21.15.77:0/3916108382 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c031c40 con 0x7f6a2c15c5a0 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.583+0000 7f6a3324e640 1 -- 172.21.15.77:0/3916108382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c15c5a0 msgr2=0x7f6a2c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.583+0000 7f6a3324e640 1 --2- 172.21.15.77:0/3916108382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c15c5a0 0x7f6a2c06c910 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009870 tx=0x7f6a1c02b910 comp rx=0 tx=0).stop 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.583+0000 7f6a3324e640 1 -- 172.21.15.77:0/3916108382 shutdown_connections 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.583+0000 7f6a3324e640 1 --2- 172.21.15.77:0/3916108382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c15c5a0 0x7f6a2c06c910 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.583+0000 7f6a3324e640 1 -- 172.21.15.77:0/3916108382 >> 172.21.15.77:0/3916108382 conn(0x7f6a2c0b1650 msgr2=0x7f6a2c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.583+0000 7f6a3324e640 1 -- 172.21.15.77:0/3916108382 shutdown_connections 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.583+0000 7f6a3324e640 1 -- 172.21.15.77:0/3916108382 wait complete. 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.584+0000 7f6a3324e640 1 Processor -- start 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.584+0000 7f6a3324e640 1 -- start start 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a3324e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2c15c5a0 0x7f6a2c156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a3324e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a2c15c980 0x7f6a2c1f1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a3324e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c157830 0x7f6a2c0cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a3324e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c070e70 con 0x7f6a2c157830 2024-03-06T07:34:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a3324e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c0708b0 con 0x7f6a2c15c5a0 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a3324e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c15ce10 con 0x7f6a2c15c980 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c157830 0x7f6a2c0cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c157830 0x7f6a2c0cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45908/0 (socket says 172.21.15.77:45908) 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 -- 172.21.15.77:0/2347782272 learned_addr learned my addr 172.21.15.77:0/2347782272 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a2bfff640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a2c15c980 0x7f6a2c1f1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a30fea640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2c15c5a0 0x7f6a2c156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 -- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a2c15c980 msgr2=0x7f6a2c1f1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a2c15c980 0x7f6a2c1f1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 -- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2c15c5a0 msgr2=0x7f6a2c156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6a2c15c5a0 0x7f6a2c156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 -- 172.21.15.77:0/2347782272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c156cc0 con 0x7f6a2c157830 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a2bfff640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f6a2c15c980 0x7f6a2c1f1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.585+0000 7f6a317eb640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c157830 0x7f6a2c0cdfe0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f6a2400b9a0 tx=0x7f6a2400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.586+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24019070 con 0x7f6a2c157830 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.586+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2400cba0 con 0x7f6a2c157830 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.586+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2c156fa0 con 0x7f6a2c157830 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.586+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24014440 con 0x7f6a2c157830 2024-03-06T07:34:15.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.586+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2c157560 con 0x7f6a2c157830 2024-03-06T07:34:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.586+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a2c140280 con 0x7f6a2c157830 2024-03-06T07:34:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.587+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a2401c070 con 0x7f6a2c157830 2024-03-06T07:34:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a297fa640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a1806b4d0 0x7f6a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6a2408a0e0 con 0x7f6a2c157830 2024-03-06T07:34:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a30fea640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a1806b4d0 0x7f6a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a297fa640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f6a18071430 0x7f6a18073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a18073f00 con 0x7f6a18071430 2024-03-06T07:34:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a2408a540 con 0x7f6a2c157830 2024-03-06T07:34:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a2bfff640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f6a18071430 0x7f6a18073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a30fea640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a1806b4d0 0x7f6a1806d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c036000 tx=0x7f6a1c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.588+0000 7f6a2bfff640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f6a18071430 0x7f6a18073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.589+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 <== osd.4 v2:172.21.15.158:6800/2421130833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a18073f00 con 0x7f6a18071430 2024-03-06T07:34:15.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.602+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 --> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6a2c04f330 con 0x7f6a18071430 2024-03-06T07:34:15.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a297fa640 1 -- 172.21.15.77:0/2347782272 <== osd.4 v2:172.21.15.158:6800/2421130833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6a2c04f330 con 0x7f6a18071430 2024-03-06T07:34:15.605 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:34:15.605 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:15.605 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:34:15.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f6a18071430 msgr2=0x7f6a18073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.158:6800/2421130833,v1:172.21.15.158:6802/2421130833] conn(0x7f6a18071430 0x7f6a18073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a1806b4d0 msgr2=0x7f6a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f6a1806b4d0 0x7f6a1806d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c036000 tx=0x7f6a1c03c040 comp rx=0 tx=0).stop 2024-03-06T07:34:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c157830 msgr2=0x7f6a2c0cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 --2- 172.21.15.77:0/2347782272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c157830 0x7f6a2c0cdfe0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f6a2400b9a0 tx=0x7f6a2400be60 comp rx=0 tx=0).stop 2024-03-06T07:34:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a317eb640 1 -- 172.21.15.77:0/2347782272 reap_dead start 2024-03-06T07:34:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 shutdown_connections 2024-03-06T07:34:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 >> 172.21.15.77:0/2347782272 conn(0x7f6a2c0b1650 msgr2=0x7f6a2c0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 shutdown_connections 2024-03-06T07:34:15.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.603+0000 7f6a3324e640 1 -- 172.21.15.77:0/2347782272 wait complete. 2024-03-06T07:34:15.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:34:15.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.714+0000 7f0417d81640 1 Processor -- start 2024-03-06T07:34:15.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.715+0000 7f0417d81640 1 -- start start 2024-03-06T07:34:15.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.715+0000 7f0417d81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04100d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.715+0000 7f0417d81640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0410161cf0 0x7f0410162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.715+0000 7f0417d81640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04100d22c0 con 0x7f0410162970 2024-03-06T07:34:15.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.715+0000 7f0417d81640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0410161b00 con 0x7f0410161cf0 2024-03-06T07:34:15.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.715+0000 7f0417d81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04100cecc0 con 0x7f04100b4460 2024-03-06T07:34:15.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f0415b1d640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04100d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f041631e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f0410161cf0 0x7f0410162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f041531c640 1 --1- >> v1:172.21.15.158:6789/0 conn(0x7f0410162970 0x7f04100b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.158:6789/0 says I am v1:172.21.15.77:48858/0 (socket says 172.21.15.77:48858) 2024-03-06T07:34:15.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f041531c640 1 -- 172.21.15.77:0/1765226328 learned_addr learned my addr 172.21.15.77:0/1765226328 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:15.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f041631e640 1 -- 172.21.15.77:0/1765226328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 msgr2=0x7f04100d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f041631e640 1 --2- 172.21.15.77:0/1765226328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04100d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f041631e640 1 -- 172.21.15.77:0/1765226328 >> v1:172.21.15.158:6789/0 conn(0x7f0410162970 legacy=0x7f04100b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f041631e640 1 -- 172.21.15.77:0/1765226328 --> v2:172.21.15.158:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04100d25a0 con 0x7f0410161cf0 2024-03-06T07:34:15.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.716+0000 7f0415b1d640 1 --2- 172.21.15.77:0/1765226328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04100d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:15.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.717+0000 7f041631e640 1 --2- 172.21.15.77:0/1765226328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 0x7f0410162180 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0408009d90 tx=0x7f040802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3ed1cfd7863733d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.717+0000 7f0414b1b640 1 -- 172.21.15.77:0/1765226328 <== mon.1 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0408039070 con 0x7f0410161cf0 2024-03-06T07:34:15.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.717+0000 7f0414b1b640 1 -- 172.21.15.77:0/1765226328 <== mon.1 v2:172.21.15.158:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040802cc50 con 0x7f0410161cf0 2024-03-06T07:34:15.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.717+0000 7f0414b1b640 1 -- 172.21.15.77:0/1765226328 <== mon.1 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0408034440 con 0x7f0410161cf0 2024-03-06T07:34:15.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.718+0000 7f0417d81640 1 -- 172.21.15.77:0/1765226328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 msgr2=0x7f0410162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.718+0000 7f0417d81640 1 --2- 172.21.15.77:0/1765226328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 0x7f0410162180 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0408009d90 tx=0x7f040802bea0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.718+0000 7f0417d81640 1 -- 172.21.15.77:0/1765226328 shutdown_connections 2024-03-06T07:34:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.718+0000 7f0417d81640 1 --2- 172.21.15.77:0/1765226328 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 0x7f0410162180 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.718+0000 7f0417d81640 1 --2- 172.21.15.77:0/1765226328 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04100d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.718+0000 7f0417d81640 1 -- 172.21.15.77:0/1765226328 >> 172.21.15.77:0/1765226328 conn(0x7f04100d3ce0 msgr2=0x7f04100d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.718+0000 7f0417d81640 1 -- 172.21.15.77:0/1765226328 shutdown_connections 2024-03-06T07:34:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.718+0000 7f0417d81640 1 -- 172.21.15.77:0/1765226328 wait complete. 2024-03-06T07:34:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.719+0000 7f0417d81640 1 Processor -- start 2024-03-06T07:34:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.719+0000 7f0417d81640 1 -- start start 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f0417d81640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04101580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f0417d81640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 0x7f04100ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f0417d81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0410162970 0x7f041015eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f0417d81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04100ce950 con 0x7f0410162970 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f0417d81640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04100d2b60 con 0x7f0410161cf0 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f0417d81640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04100b4840 con 0x7f04100b4460 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0410162970 0x7f041015eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0410162970 0x7f041015eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45912/0 (socket says 172.21.15.77:45912) 2024-03-06T07:34:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 -- 172.21.15.77:0/56603697 learned_addr learned my addr 172.21.15.77:0/56603697 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f0415b1d640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04101580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041531c640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 0x7f04100ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 -- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 msgr2=0x7f04101580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04101580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 -- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 msgr2=0x7f04100ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 0x7f04100ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 -- 172.21.15.77:0/56603697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0410158730 con 0x7f0410162970 2024-03-06T07:34:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041531c640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 0x7f04100ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f041631e640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0410162970 0x7f041015eba0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f0408009d90 tx=0x7f0408004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.720+0000 7f04067fc640 1 -- 172.21.15.77:0/56603697 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0408039050 con 0x7f0410162970 2024-03-06T07:34:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.721+0000 7f04067fc640 1 -- 172.21.15.77:0/56603697 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0408045070 con 0x7f0410162970 2024-03-06T07:34:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.721+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0410158a10 con 0x7f0410162970 2024-03-06T07:34:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.721+0000 7f04067fc640 1 -- 172.21.15.77:0/56603697 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0408009450 con 0x7f0410162970 2024-03-06T07:34:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.721+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0410158fd0 con 0x7f0410162970 2024-03-06T07:34:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.722+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041004f330 con 0x7f0410162970 2024-03-06T07:34:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.722+0000 7f04067fc640 1 -- 172.21.15.77:0/56603697 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f040803e610 con 0x7f0410162970 2024-03-06T07:34:15.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.722+0000 7f04067fc640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03f406b4d0 0x7f03f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.723+0000 7f04067fc640 1 -- 172.21.15.77:0/56603697 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f04080ac9d0 con 0x7f0410162970 2024-03-06T07:34:15.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.723+0000 7f0415b1d640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03f406b4d0 0x7f03f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.723+0000 7f0415b1d640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03f406b4d0 0x7f03f406d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f04100b62e0 tx=0x7f0400008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.727+0000 7f04067fc640 1 -- 172.21.15.77:0/56603697 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f040807bae0 con 0x7f0410162970 2024-03-06T07:34:15.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.899+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f041006b180 con 0x7f03f406b4d0 2024-03-06T07:34:15.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.900+0000 7f04067fc640 1 -- 172.21.15.77:0/56603697 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f041006b180 con 0x7f03f406b4d0 2024-03-06T07:34:15.902 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:34:15.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03f406b4d0 msgr2=0x7f03f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03f406b4d0 0x7f03f406d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f04100b62e0 tx=0x7f0400008040 comp rx=0 tx=0).stop 2024-03-06T07:34:15.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0410162970 msgr2=0x7f041015eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0410162970 0x7f041015eba0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f0408009d90 tx=0x7f0408004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:15.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 shutdown_connections 2024-03-06T07:34:15.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f03f406b4d0 0x7f03f406d990 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0410162970 0x7f041015eba0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0410161cf0 0x7f04100ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 --2- 172.21.15.77:0/56603697 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f04100b4460 0x7f04101580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 >> 172.21.15.77:0/56603697 conn(0x7f04100d3ce0 msgr2=0x7f041006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 shutdown_connections 2024-03-06T07:34:15.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:15.902+0000 7f0417d81640 1 -- 172.21.15.77:0/56603697 wait complete. 2024-03-06T07:34:15.914 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:34:15.915 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:16.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.016+0000 7f6a0471e640 1 Processor -- start 2024-03-06T07:34:16.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.016+0000 7f6a0471e640 1 -- start start 2024-03-06T07:34:16.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.017+0000 7f6a0471e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 0x7f69fc0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.017+0000 7f6a0471e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69fc0d2820 0x7f69fc0cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.017+0000 7f6a0471e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69fc15e380 con 0x7f69fc15dfa0 2024-03-06T07:34:16.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.017+0000 7f6a0471e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69fc0d2100 con 0x7f69fc0d2820 2024-03-06T07:34:16.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.017+0000 7f6a0471e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0cf470 con 0x7f69fc0d4630 2024-03-06T07:34:16.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a02cbb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69fc0d2820 0x7f69fc0cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a024ba640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f69fc15dfa0 0x7f69fc0d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59842/0 (socket says 172.21.15.77:59842) 2024-03-06T07:34:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a02cbb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69fc0d2820 0x7f69fc0cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45916/0 (socket says 172.21.15.77:45916) 2024-03-06T07:34:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a024ba640 1 -- 172.21.15.77:0/525737998 learned_addr learned my addr 172.21.15.77:0/525737998 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a01cb9640 1 --2- 172.21.15.77:0/525737998 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 0x7f69fc0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a02cbb640 1 -- 172.21.15.77:0/525737998 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 msgr2=0x7f69fc0d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a02cbb640 1 --2- 172.21.15.77:0/525737998 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 0x7f69fc0d22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a02cbb640 1 -- 172.21.15.77:0/525737998 >> v1:172.21.15.77:6789/0 conn(0x7f69fc15dfa0 legacy=0x7f69fc0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.018+0000 7f6a02cbb640 1 -- 172.21.15.77:0/525737998 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc0cf750 con 0x7f69fc0d2820 2024-03-06T07:34:16.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a02cbb640 1 --2- 172.21.15.77:0/525737998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0d2820 0x7f69fc0cb190 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009d90 tx=0x7f69f402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340b2716c03aca55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a014b8640 1 -- 172.21.15.77:0/525737998 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4039070 con 0x7f69fc0d2820 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a014b8640 1 -- 172.21.15.77:0/525737998 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f402cbc0 con 0x7f69fc0d2820 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a014b8640 1 -- 172.21.15.77:0/525737998 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4034440 con 0x7f69fc0d2820 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a0471e640 1 -- 172.21.15.77:0/525737998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0d2820 msgr2=0x7f69fc0cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a0471e640 1 --2- 172.21.15.77:0/525737998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0d2820 0x7f69fc0cb190 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009d90 tx=0x7f69f402be10 comp rx=0 tx=0).stop 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a0471e640 1 -- 172.21.15.77:0/525737998 shutdown_connections 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a0471e640 1 --2- 172.21.15.77:0/525737998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0d2820 0x7f69fc0cb190 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a0471e640 1 --2- 172.21.15.77:0/525737998 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 0x7f69fc0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a0471e640 1 -- 172.21.15.77:0/525737998 >> 172.21.15.77:0/525737998 conn(0x7f69fc09d000 msgr2=0x7f69fc0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a0471e640 1 -- 172.21.15.77:0/525737998 shutdown_connections 2024-03-06T07:34:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.019+0000 7f6a0471e640 1 -- 172.21.15.77:0/525737998 wait complete. 2024-03-06T07:34:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.020+0000 7f6a0471e640 1 Processor -- start 2024-03-06T07:34:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.020+0000 7f6a0471e640 1 -- start start 2024-03-06T07:34:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a0471e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69fc0d2820 0x7f69fc1501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a0471e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 0x7f69fc0b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a0471e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc15dfa0 0x7f69fc0b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a0471e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0cb850 con 0x7f69fc15dfa0 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a0471e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0cb6d0 con 0x7f69fc0d2820 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a0471e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0cb9d0 con 0x7f69fc0d4630 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a02cbb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc15dfa0 0x7f69fc0b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a02cbb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc15dfa0 0x7f69fc0b08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45926/0 (socket says 172.21.15.77:45926) 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a02cbb640 1 -- 172.21.15.77:0/3464656527 learned_addr learned my addr 172.21.15.77:0/3464656527 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a01cb9640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 0x7f69fc0b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.021+0000 7f6a024ba640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69fc0d2820 0x7f69fc1501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a02cbb640 1 -- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 msgr2=0x7f69fc0b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a02cbb640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 0x7f69fc0b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a02cbb640 1 -- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69fc0d2820 msgr2=0x7f69fc1501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a02cbb640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f69fc0d2820 0x7f69fc1501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a02cbb640 1 -- 172.21.15.77:0/3464656527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc0b30d0 con 0x7f69fc15dfa0 2024-03-06T07:34:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a01cb9640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f69fc0d4630 0x7f69fc0b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a02cbb640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc15dfa0 0x7f69fc0b08f0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009d90 tx=0x7f69f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4044020 con 0x7f69fc15dfa0 2024-03-06T07:34:16.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f403f470 con 0x7f69fc15dfa0 2024-03-06T07:34:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc0b33b0 con 0x7f69fc15dfa0 2024-03-06T07:34:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4039040 con 0x7f69fc15dfa0 2024-03-06T07:34:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.022+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc0b3970 con 0x7f69fc15dfa0 2024-03-06T07:34:16.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.023+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f69fc04f330 con 0x7f69fc15dfa0 2024-03-06T07:34:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.023+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69f4047020 con 0x7f69fc15dfa0 2024-03-06T07:34:16.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.024+0000 7f69f2ffd640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69e006b4d0 0x7f69e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.024+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f69f40acae0 con 0x7f69fc15dfa0 2024-03-06T07:34:16.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.024+0000 7f6a024ba640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69e006b4d0 0x7f69e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.024+0000 7f69f2ffd640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f69e0071430 0x7f69e0073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.024+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f69f4042e50 con 0x7f69e0071430 2024-03-06T07:34:16.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.024+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69f40b12c0 con 0x7f69fc15dfa0 2024-03-06T07:34:16.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.024+0000 7f6a01cb9640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f69e0071430 0x7f69e0073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.024+0000 7f6a024ba640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69e006b4d0 0x7f69e006d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f69f8000f60 tx=0x7f69f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.025+0000 7f6a01cb9640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f69e0071430 0x7f69e0073850 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.025+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 <== osd.5 v2:172.21.15.158:6820/3476303095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f69f4042e50 con 0x7f69e0071430 2024-03-06T07:34:16.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.038+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 --> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f69fc06a090 con 0x7f69e0071430 2024-03-06T07:34:16.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f69f2ffd640 1 -- 172.21.15.77:0/3464656527 <== osd.5 v2:172.21.15.158:6820/3476303095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f69fc06a090 con 0x7f69e0071430 2024-03-06T07:34:16.041 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:34:16.041 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:16.041 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:34:16.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f69e0071430 msgr2=0x7f69e0073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.158:6820/3476303095,v1:172.21.15.158:6822/3476303095] conn(0x7f69e0071430 0x7f69e0073850 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69e006b4d0 msgr2=0x7f69e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f69e006b4d0 0x7f69e006d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f69f8000f60 tx=0x7f69f8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:16.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc15dfa0 msgr2=0x7f69fc0b08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 --2- 172.21.15.77:0/3464656527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc15dfa0 0x7f69fc0b08f0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009d90 tx=0x7f69f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:16.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a02cbb640 1 -- 172.21.15.77:0/3464656527 reap_dead start 2024-03-06T07:34:16.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 shutdown_connections 2024-03-06T07:34:16.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 >> 172.21.15.77:0/3464656527 conn(0x7f69fc09d000 msgr2=0x7f69fc06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 shutdown_connections 2024-03-06T07:34:16.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.039+0000 7f6a0471e640 1 -- 172.21.15.77:0/3464656527 wait complete. 2024-03-06T07:34:16.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:34:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.149+0000 7fba1c332640 1 Processor -- start 2024-03-06T07:34:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.149+0000 7fba1c332640 1 -- start start 2024-03-06T07:34:16.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.150+0000 7fba1c332640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba14162400 0x7fba140d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.150+0000 7fba1c332640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140d22c0 con 0x7fba14162970 2024-03-06T07:34:16.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.150+0000 7fba1c332640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140d25a0 con 0x7fba140b4460 2024-03-06T07:34:16.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.150+0000 7fba1c332640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba14161b00 con 0x7fba14162400 2024-03-06T07:34:16.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.150+0000 7fba198cd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fba14162970 0x7fba14161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59854/0 (socket says 172.21.15.77:59854) 2024-03-06T07:34:16.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.150+0000 7fba198cd640 1 -- 172.21.15.77:0/2408870767 learned_addr learned my addr 172.21.15.77:0/2408870767 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:16.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba1a8cf640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba14162400 0x7fba140d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474234475 0 0) 0x7fba140d22c0 con 0x7fba14162970 2024-03-06T07:34:16.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba00003520 con 0x7fba14162970 2024-03-06T07:34:16.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2770579109 0 0) 0x7fba00003520 con 0x7fba14162970 2024-03-06T07:34:16.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba14162400 msgr2=0x7fba140d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba14162400 0x7fba140d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 >> v1:172.21.15.158:6789/0 conn(0x7fba140b4460 legacy=0x7fba140b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140d2880 con 0x7fba14162970 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fba04001670 con 0x7fba14162970 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 >> v1:172.21.15.77:6789/0 conn(0x7fba14162970 legacy=0x7fba14161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba14162400 0x7fba000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba00003f20 0x7fba00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004d40 0x7fba000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba14161b00 con 0x7fba00003f20 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0000ba20 con 0x7fba00004d40 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0000bc20 con 0x7fba14162400 2024-03-06T07:34:16.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba198cd640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba00003f20 0x7fba00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba1a0ce640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba14162400 0x7fba000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba1a8cf640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004d40 0x7fba000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba198cd640 1 -- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba14162400 msgr2=0x7fba000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba198cd640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba14162400 0x7fba000039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba198cd640 1 -- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004d40 msgr2=0x7fba000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba198cd640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00004d40 0x7fba000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.151+0000 7fba198cd640 1 -- 172.21.15.77:0/2408870767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00003520 con 0x7fba00003f20 2024-03-06T07:34:16.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.152+0000 7fba198cd640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba00003f20 0x7fba00004800 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fba04007fd0 tx=0x7fba00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13924fd9872acaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.152+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04007d50 con 0x7fba00003f20 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.152+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba04030070 con 0x7fba00003f20 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.152+0000 7fba190cc640 1 -- 172.21.15.77:0/2408870767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04005d70 con 0x7fba00003f20 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.153+0000 7fba1c332640 1 -- 172.21.15.77:0/2408870767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba00003f20 msgr2=0x7fba00004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.153+0000 7fba1c332640 1 --2- 172.21.15.77:0/2408870767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba00003f20 0x7fba00004800 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fba04007fd0 tx=0x7fba00005830 comp rx=0 tx=0).stop 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.153+0000 7fba1a8cf640 1 -- 172.21.15.77:0/2408870767 reap_dead start 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.153+0000 7fba1c332640 1 -- 172.21.15.77:0/2408870767 shutdown_connections 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.153+0000 7fba1c332640 1 -- 172.21.15.77:0/2408870767 >> 172.21.15.77:0/2408870767 conn(0x7fba140d3ce0 msgr2=0x7fba140d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.153+0000 7fba1c332640 1 -- 172.21.15.77:0/2408870767 shutdown_connections 2024-03-06T07:34:16.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.153+0000 7fba1c332640 1 -- 172.21.15.77:0/2408870767 wait complete. 2024-03-06T07:34:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.154+0000 7fba1c332640 1 Processor -- start 2024-03-06T07:34:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.154+0000 7fba1c332640 1 -- start start 2024-03-06T07:34:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1c332640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00003f20 0x7fba140b9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1c332640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14154d80 0x7fba140b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1c332640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba141552d0 0x7fba140cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1c332640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba140ce950 con 0x7fba14154d80 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1c332640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba140cecc0 con 0x7fba00003f20 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1c332640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba140b4840 con 0x7fba141552d0 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14154d80 0x7fba140b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14154d80 0x7fba140b6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45952/0 (socket says 172.21.15.77:45952) 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 -- 172.21.15.77:0/1532386937 learned_addr learned my addr 172.21.15.77:0/1532386937 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1a0ce640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00003f20 0x7fba140b9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1a8cf640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba141552d0 0x7fba140cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 -- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba141552d0 msgr2=0x7fba140cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba141552d0 0x7fba140cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 -- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00003f20 msgr2=0x7fba140b9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00003f20 0x7fba140b9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 -- 172.21.15.77:0/1532386937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140b8ab0 con 0x7fba14154d80 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1a8cf640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba141552d0 0x7fba140cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba198cd640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14154d80 0x7fba140b6f30 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fba04000f60 tx=0x7fba0402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba0affd640 1 -- 172.21.15.77:0/1532386937 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04007d50 con 0x7fba14154d80 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba0affd640 1 -- 172.21.15.77:0/1532386937 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba04004bf0 con 0x7fba14154d80 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba140cc890 con 0x7fba14154d80 2024-03-06T07:34:16.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.155+0000 7fba0affd640 1 -- 172.21.15.77:0/1532386937 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04007070 con 0x7fba14154d80 2024-03-06T07:34:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.156+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba140b8d50 con 0x7fba14154d80 2024-03-06T07:34:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.156+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1404f330 con 0x7fba14154d80 2024-03-06T07:34:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.157+0000 7fba0affd640 1 -- 172.21.15.77:0/1532386937 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba04048020 con 0x7fba14154d80 2024-03-06T07:34:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.157+0000 7fba0affd640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9f806b500 0x7fb9f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.157+0000 7fba0affd640 1 -- 172.21.15.77:0/1532386937 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fba040ab270 con 0x7fba14154d80 2024-03-06T07:34:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.157+0000 7fba1a0ce640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9f806b500 0x7fb9f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.158+0000 7fba1a0ce640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9f806b500 0x7fb9f806d9c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fba10003fd0 tx=0x7fba10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.162+0000 7fba0affd640 1 -- 172.21.15.77:0/1532386937 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba0407a420 con 0x7fba14154d80 2024-03-06T07:34:16.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.333+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fba1406b180 con 0x7fb9f806b500 2024-03-06T07:34:16.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.334+0000 7fba0affd640 1 -- 172.21.15.77:0/1532386937 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fba1406b180 con 0x7fb9f806b500 2024-03-06T07:34:16.336 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:34:16.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9f806b500 msgr2=0x7fb9f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9f806b500 0x7fb9f806d9c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fba10003fd0 tx=0x7fba10009040 comp rx=0 tx=0).stop 2024-03-06T07:34:16.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14154d80 msgr2=0x7fba140b6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14154d80 0x7fba140b6f30 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fba04000f60 tx=0x7fba0402e070 comp rx=0 tx=0).stop 2024-03-06T07:34:16.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 shutdown_connections 2024-03-06T07:34:16.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fb9f806b500 0x7fb9f806d9c0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14154d80 0x7fba140b6f30 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fba141552d0 0x7fba140cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 --2- 172.21.15.77:0/1532386937 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fba00003f20 0x7fba140b9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 >> 172.21.15.77:0/1532386937 conn(0x7fba140d3ce0 msgr2=0x7fba14160e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 shutdown_connections 2024-03-06T07:34:16.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.336+0000 7fba1c332640 1 -- 172.21.15.77:0/1532386937 wait complete. 2024-03-06T07:34:16.348 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:34:16.349 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.452+0000 7fdc20a2a640 1 Processor -- start 2024-03-06T07:34:16.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.452+0000 7fdc20a2a640 1 -- start start 2024-03-06T07:34:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc20a2a640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fdc1c0ce8c0 0x7fdc1c0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc20a2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c15c980 0x7fdc1c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc20a2a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c0ced50 con 0x7fdc1c15c5a0 2024-03-06T07:34:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc20a2a640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c072a20 con 0x7fdc1c0ce8c0 2024-03-06T07:34:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc20a2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0708b0 con 0x7fdc1c15c980 2024-03-06T07:34:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc1a59c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdc1c15c5a0 0x7fdc1c06c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59856/0 (socket says 172.21.15.77:59856) 2024-03-06T07:34:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc1a59c640 1 -- 172.21.15.77:0/309082552 learned_addr learned my addr 172.21.15.77:0/309082552 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc19d9b640 1 --2- 172.21.15.77:0/309082552 >> v2:172.21.15.158:3300/0 conn(0x7fdc1c0ce8c0 0x7fdc1c0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc1ad9d640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c15c980 0x7fdc1c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868930393 0 0) 0x7fdc1c0ced50 con 0x7fdc1c15c5a0 2024-03-06T07:34:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.453+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc04003520 con 0x7fdc1c15c5a0 2024-03-06T07:34:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1769520487 0 0) 0x7fdc04003520 con 0x7fdc1c15c5a0 2024-03-06T07:34:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c15c980 msgr2=0x7fdc1c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c15c980 0x7fdc1c071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 >> v2:172.21.15.158:3300/0 conn(0x7fdc1c0ce8c0 msgr2=0x7fdc1c0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 --2- 172.21.15.77:0/309082552 >> v2:172.21.15.158:3300/0 conn(0x7fdc1c0ce8c0 0x7fdc1c0cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0cf030 con 0x7fdc1c15c5a0 2024-03-06T07:34:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fdc0c0019f0 con 0x7fdc1c15c5a0 2024-03-06T07:34:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 >> v1:172.21.15.77:6789/0 conn(0x7fdc1c15c5a0 legacy=0x7fdc1c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c15c980 0x7fdc04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04004d40 0x7fdc040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c072a20 con 0x7fdc040039b0 2024-03-06T07:34:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0708b0 con 0x7fdc04004d40 2024-03-06T07:34:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0400b9c0 con 0x7fdc1c15c980 2024-03-06T07:34:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1a59c640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc19d9b640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c15c980 0x7fdc04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1ad9d640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04004d40 0x7fdc040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1a59c640 1 -- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c15c980 msgr2=0x7fdc04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1a59c640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c15c980 0x7fdc04004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1a59c640 1 -- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04004d40 msgr2=0x7fdc040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1a59c640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc04004d40 0x7fdc040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.454+0000 7fdc1a59c640 1 -- 172.21.15.77:0/309082552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc04003520 con 0x7fdc040039b0 2024-03-06T07:34:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc1a59c640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc04003dd0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0013d0 tx=0x7fdc04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45164c52636dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c00a4e0 con 0x7fdc040039b0 2024-03-06T07:34:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c034650 con 0x7fdc040039b0 2024-03-06T07:34:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc1959a640 1 -- 172.21.15.77:0/309082552 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c034a40 con 0x7fdc040039b0 2024-03-06T07:34:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc20a2a640 1 -- 172.21.15.77:0/309082552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 msgr2=0x7fdc04003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc20a2a640 1 --2- 172.21.15.77:0/309082552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc04003dd0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0013d0 tx=0x7fdc04005830 comp rx=0 tx=0).stop 2024-03-06T07:34:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc1ad9d640 1 -- 172.21.15.77:0/309082552 reap_dead start 2024-03-06T07:34:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc20a2a640 1 -- 172.21.15.77:0/309082552 shutdown_connections 2024-03-06T07:34:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc20a2a640 1 -- 172.21.15.77:0/309082552 >> 172.21.15.77:0/309082552 conn(0x7fdc1c0b1650 msgr2=0x7fdc1c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc20a2a640 1 -- 172.21.15.77:0/309082552 shutdown_connections 2024-03-06T07:34:16.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.455+0000 7fdc20a2a640 1 -- 172.21.15.77:0/309082552 wait complete. 2024-03-06T07:34:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.457+0000 7fdc20a2a640 1 Processor -- start 2024-03-06T07:34:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.457+0000 7fdc20a2a640 1 -- start start 2024-03-06T07:34:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.457+0000 7fdc20a2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc1c159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.457+0000 7fdc20a2a640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c155800 0x7fdc1c0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.457+0000 7fdc20a2a640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c13f120 0x7fdc1c13f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.457+0000 7fdc20a2a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0cc870 con 0x7fdc040039b0 2024-03-06T07:34:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.457+0000 7fdc20a2a640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c070e70 con 0x7fdc1c155800 2024-03-06T07:34:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.457+0000 7fdc20a2a640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c15ce30 con 0x7fdc1c13f120 2024-03-06T07:34:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc1c159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc1c159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45980/0 (socket says 172.21.15.77:45980) 2024-03-06T07:34:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 -- 172.21.15.77:0/1525462007 learned_addr learned my addr 172.21.15.77:0/1525462007 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 -- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c13f120 msgr2=0x7fdc1c13f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fdc1c13f120 0x7fdc1c13f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 -- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c155800 msgr2=0x7fdc1c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc19d9b640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c155800 0x7fdc1c0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdc1c155800 0x7fdc1c0b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 -- 172.21.15.77:0/1525462007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c1f7090 con 0x7fdc040039b0 2024-03-06T07:34:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc1a59c640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc1c159030 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00a0b0 tx=0x7fdc0c001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c039040 con 0x7fdc040039b0 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c043070 con 0x7fdc040039b0 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c1f7370 con 0x7fdc040039b0 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0019b0 con 0x7fdc040039b0 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.458+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c1f7930 con 0x7fdc040039b0 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.459+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdc1c04f330 con 0x7fdc040039b0 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.460+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc0c034650 con 0x7fdc040039b0 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.460+0000 7fdc0affd640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.460+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdc0c0ac970 con 0x7fdc040039b0 2024-03-06T07:34:16.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.460+0000 7fdc19d9b640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.460+0000 7fdc0affd640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7fdbfc071430 0x7fdbfc073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.460+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc0c046e80 con 0x7fdbfc071430 2024-03-06T07:34:16.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.460+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc0c0ace00 con 0x7fdc040039b0 2024-03-06T07:34:16.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.460+0000 7fdc1ad9d640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7fdbfc071430 0x7fdbfc073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.461+0000 7fdc19d9b640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c155c00 tx=0x7fdc10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.461+0000 7fdc1ad9d640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7fdbfc071430 0x7fdbfc073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.462+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 <== osd.6 v2:172.21.15.158:6801/2971141295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc0c046e80 con 0x7fdbfc071430 2024-03-06T07:34:16.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.474+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 --> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdc1c15d7a0 con 0x7fdbfc071430 2024-03-06T07:34:16.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc0affd640 1 -- 172.21.15.77:0/1525462007 <== osd.6 v2:172.21.15.158:6801/2971141295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdc1c15d7a0 con 0x7fdbfc071430 2024-03-06T07:34:16.477 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:34:16.477 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:16.477 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:34:16.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7fdbfc071430 msgr2=0x7fdbfc073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.158:6801/2971141295,v1:172.21.15.158:6803/2971141295] conn(0x7fdbfc071430 0x7fdbfc073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdbfc06b4d0 msgr2=0x7fdbfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c155c00 tx=0x7fdc10008040 comp rx=0 tx=0).stop 2024-03-06T07:34:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 msgr2=0x7fdc1c159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 --2- 172.21.15.77:0/1525462007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc040039b0 0x7fdc1c159030 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00a0b0 tx=0x7fdc0c001360 comp rx=0 tx=0).stop 2024-03-06T07:34:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc1ad9d640 1 -- 172.21.15.77:0/1525462007 reap_dead start 2024-03-06T07:34:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 shutdown_connections 2024-03-06T07:34:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 >> 172.21.15.77:0/1525462007 conn(0x7fdc1c0b1650 msgr2=0x7fdc1c15dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 shutdown_connections 2024-03-06T07:34:16.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.475+0000 7fdc20a2a640 1 -- 172.21.15.77:0/1525462007 wait complete. 2024-03-06T07:34:16.487 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:34:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.588+0000 7fd21dc0e640 1 Processor -- start 2024-03-06T07:34:16.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.588+0000 7fd21dc0e640 1 -- start start 2024-03-06T07:34:16.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.588+0000 7fd21dc0e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd218071e00 0x7fd2180cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd21dc0e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd218072490 con 0x7fd21815bbf0 2024-03-06T07:34:16.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd21dc0e640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180d0090 con 0x7fd2180cd7c0 2024-03-06T07:34:16.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd21dc0e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd218072250 con 0x7fd218071e00 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2177fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd21815bbf0 0x7fd21815bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59866/0 (socket says 172.21.15.77:59866) 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2177fe640 1 -- 172.21.15.77:0/460325826 learned_addr learned my addr 172.21.15.77:0/460325826 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723805506 0 0) 0x7fd218072490 con 0x7fd21815bbf0 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd216ffd640 1 --2- 172.21.15.77:0/460325826 >> v2:172.21.15.158:3300/0 conn(0x7fd218071e00 0x7fd2180cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200003520 con 0x7fd21815bbf0 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3912144092 0 0) 0x7fd200003520 con 0x7fd21815bbf0 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 >> v2:172.21.15.158:3300/0 conn(0x7fd218071e00 msgr2=0x7fd2180cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 --2- 172.21.15.77:0/460325826 >> v2:172.21.15.158:3300/0 conn(0x7fd218071e00 0x7fd2180cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 >> v1:172.21.15.158:6789/0 conn(0x7fd2180cd7c0 legacy=0x7fd2180cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180d0370 con 0x7fd21815bbf0 2024-03-06T07:34:16.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7fd208001670 con 0x7fd21815bbf0 2024-03-06T07:34:16.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 >> v1:172.21.15.77:6789/0 conn(0x7fd21815bbf0 legacy=0x7fd21815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2000039b0 0x7fd200003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd200004800 0x7fd200004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2000051c0 0x7fd20000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd218072250 con 0x7fd200004800 2024-03-06T07:34:16.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20000c060 con 0x7fd2000039b0 2024-03-06T07:34:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20000c260 con 0x7fd2000051c0 2024-03-06T07:34:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd216ffd640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd200004800 0x7fd200004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd2177fe640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2000039b0 0x7fd200003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.589+0000 7fd217fff640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2000051c0 0x7fd20000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.590+0000 7fd216ffd640 1 -- 172.21.15.77:0/460325826 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2000051c0 msgr2=0x7fd20000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.590+0000 7fd216ffd640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd2000051c0 0x7fd20000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.590+0000 7fd216ffd640 1 -- 172.21.15.77:0/460325826 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2000039b0 msgr2=0x7fd200003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.590+0000 7fd216ffd640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd2000039b0 0x7fd200003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.590+0000 7fd217fff640 1 -- 172.21.15.77:0/460325826 reap_dead start 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.590+0000 7fd216ffd640 1 -- 172.21.15.77:0/460325826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd200003520 con 0x7fd200004800 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.590+0000 7fd216ffd640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd200004800 0x7fd200004c80 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009ac0 tx=0x7fd200005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26326902d688ffa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.590+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c039070 con 0x7fd200004800 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c02c6b0 con 0x7fd200004800 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd2167fc640 1 -- 172.21.15.77:0/460325826 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c009690 con 0x7fd200004800 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd21dc0e640 1 -- 172.21.15.77:0/460325826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd200004800 msgr2=0x7fd200004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd21dc0e640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd200004800 0x7fd200004c80 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009ac0 tx=0x7fd200005830 comp rx=0 tx=0).stop 2024-03-06T07:34:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd21dc0e640 1 -- 172.21.15.77:0/460325826 shutdown_connections 2024-03-06T07:34:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd21dc0e640 1 --2- 172.21.15.77:0/460325826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd200004800 0x7fd200004c80 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd21dc0e640 1 -- 172.21.15.77:0/460325826 >> 172.21.15.77:0/460325826 conn(0x7fd2180b1c80 msgr2=0x7fd2180cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd21dc0e640 1 -- 172.21.15.77:0/460325826 shutdown_connections 2024-03-06T07:34:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.591+0000 7fd21dc0e640 1 -- 172.21.15.77:0/460325826 wait complete. 2024-03-06T07:34:16.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.592+0000 7fd21dc0e640 1 Processor -- start 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.592+0000 7fd21dc0e640 1 -- start start 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd21dc0e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd200004800 0x7fd21814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd21dc0e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd21815bfd0 0x7fd21813c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd21dc0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd21815d9a0 0x7fd21815de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd21dc0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b3800 con 0x7fd21815d9a0 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd21dc0e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b3680 con 0x7fd21815bfd0 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd21dc0e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180cebc0 con 0x7fd200004800 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd217fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd21815d9a0 0x7fd21815de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd217fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd21815d9a0 0x7fd21815de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46002/0 (socket says 172.21.15.77:46002) 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd217fff640 1 -- 172.21.15.77:0/1102457079 learned_addr learned my addr 172.21.15.77:0/1102457079 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd2177fe640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd200004800 0x7fd21814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd216ffd640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd21815bfd0 0x7fd21813c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd217fff640 1 -- 172.21.15.77:0/1102457079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd200004800 msgr2=0x7fd21814a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd217fff640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd200004800 0x7fd21814a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.593+0000 7fd217fff640 1 -- 172.21.15.77:0/1102457079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd21815bfd0 msgr2=0x7fd21813c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.594+0000 7fd217fff640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd21815bfd0 0x7fd21813c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.594+0000 7fd217fff640 1 -- 172.21.15.77:0/1102457079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21815c450 con 0x7fd21815d9a0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.594+0000 7fd217fff640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd21815d9a0 0x7fd21815de20 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fd218072710 tx=0x7fd21000fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.594+0000 7fd1fffff640 1 -- 172.21.15.77:0/1102457079 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210003b80 con 0x7fd21815d9a0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.594+0000 7fd1fffff640 1 -- 172.21.15.77:0/1102457079 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd210018070 con 0x7fd21815d9a0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.594+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21815c720 con 0x7fd21815d9a0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.594+0000 7fd1fffff640 1 -- 172.21.15.77:0/1102457079 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100050d0 con 0x7fd21815d9a0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.594+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21815cce0 con 0x7fd21815d9a0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.595+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21804f330 con 0x7fd21815d9a0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.595+0000 7fd1fffff640 1 -- 172.21.15.77:0/1102457079 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd210005270 con 0x7fd21815d9a0 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.596+0000 7fd1fffff640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd20406b500 0x7fd20406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.596+0000 7fd1fffff640 1 -- 172.21.15.77:0/1102457079 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd210089cc0 con 0x7fd21815d9a0 2024-03-06T07:34:16.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.596+0000 7fd2177fe640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd20406b500 0x7fd20406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.596+0000 7fd2177fe640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd20406b500 0x7fd20406d9c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd208007c50 tx=0x7fd208009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.600+0000 7fd1fffff640 1 -- 172.21.15.77:0/1102457079 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd210058e70 con 0x7fd21815d9a0 2024-03-06T07:34:16.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.771+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fd218151460 con 0x7fd20406b500 2024-03-06T07:34:16.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.772+0000 7fd1fffff640 1 -- 172.21.15.77:0/1102457079 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd218151460 con 0x7fd20406b500 2024-03-06T07:34:16.774 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:34:16.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd20406b500 msgr2=0x7fd20406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd20406b500 0x7fd20406d9c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd208007c50 tx=0x7fd208009040 comp rx=0 tx=0).stop 2024-03-06T07:34:16.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd21815d9a0 msgr2=0x7fd21815de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd21815d9a0 0x7fd21815de20 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fd218072710 tx=0x7fd21000fc70 comp rx=0 tx=0).stop 2024-03-06T07:34:16.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 shutdown_connections 2024-03-06T07:34:16.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd20406b500 0x7fd20406d9c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd21815d9a0 0x7fd21815de20 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd21815bfd0 0x7fd21813c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 --2- 172.21.15.77:0/1102457079 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd200004800 0x7fd21814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.773+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 >> 172.21.15.77:0/1102457079 conn(0x7fd2180b1c80 msgr2=0x7fd218159ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.774+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 shutdown_connections 2024-03-06T07:34:16.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.774+0000 7fd21dc0e640 1 -- 172.21.15.77:0/1102457079 wait complete. 2024-03-06T07:34:16.786 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:34:16.787 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:16.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.889+0000 7f92e26df640 1 Processor -- start 2024-03-06T07:34:16.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.889+0000 7f92e26df640 1 -- start start 2024-03-06T07:34:16.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92e26df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92dc15ac00 0x7f92dc06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92e26df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92dc0708b0 0x7f92dc070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92e26df640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f92dc15e7a0 0x7f92dc15ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92e26df640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92dc071250 con 0x7f92dc0708b0 2024-03-06T07:34:16.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92e26df640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92dc0cdc10 con 0x7f92dc15e7a0 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92e26df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc15afe0 con 0x7f92dc15ac00 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92db7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92dc0708b0 0x7f92dc070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92db7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f92dc0708b0 0x7f92dc070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46004/0 (socket says 172.21.15.77:46004) 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92db7fe640 1 -- 172.21.15.77:0/2613956718 learned_addr learned my addr 172.21.15.77:0/2613956718 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92dbfff640 1 --2- 172.21.15.77:0/2613956718 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92dc15ac00 0x7f92dc06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.890+0000 7f92e0c7c640 1 --2- 172.21.15.77:0/2613956718 >> v2:172.21.15.158:3300/0 conn(0x7f92dc15e7a0 0x7f92dc15ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.891+0000 7f92db7fe640 1 -- 172.21.15.77:0/2613956718 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92dc15ac00 msgr2=0x7f92dc06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.891+0000 7f92db7fe640 1 --2- 172.21.15.77:0/2613956718 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92dc15ac00 0x7f92dc06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.891+0000 7f92db7fe640 1 -- 172.21.15.77:0/2613956718 >> v2:172.21.15.158:3300/0 conn(0x7f92dc15e7a0 msgr2=0x7f92dc15ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.891+0000 7f92db7fe640 1 --2- 172.21.15.77:0/2613956718 >> v2:172.21.15.158:3300/0 conn(0x7f92dc15e7a0 0x7f92dc15ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.891+0000 7f92db7fe640 1 -- 172.21.15.77:0/2613956718 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc0cd000 con 0x7f92dc0708b0 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.891+0000 7f92db7fe640 1 --2- 172.21.15.77:0/2613956718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 0x7f92dc070d10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f92dc15e700 tx=0x7f92d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f24507e55e7a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.891+0000 7f92daffd640 1 -- 172.21.15.77:0/2613956718 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d002e070 con 0x7f92dc0708b0 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92daffd640 1 -- 172.21.15.77:0/2613956718 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d0037070 con 0x7f92dc0708b0 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92daffd640 1 -- 172.21.15.77:0/2613956718 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0031a00 con 0x7f92dc0708b0 2024-03-06T07:34:16.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 -- 172.21.15.77:0/2613956718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 msgr2=0x7f92dc070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 --2- 172.21.15.77:0/2613956718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 0x7f92dc070d10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f92dc15e700 tx=0x7f92d002b880 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 -- 172.21.15.77:0/2613956718 shutdown_connections 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 --2- 172.21.15.77:0/2613956718 >> v2:172.21.15.158:3300/0 conn(0x7f92dc15e7a0 0x7f92dc15ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 --2- 172.21.15.77:0/2613956718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 0x7f92dc070d10 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 --2- 172.21.15.77:0/2613956718 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92dc15ac00 0x7f92dc06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 -- 172.21.15.77:0/2613956718 >> 172.21.15.77:0/2613956718 conn(0x7f92dc0b1650 msgr2=0x7f92dc0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 -- 172.21.15.77:0/2613956718 shutdown_connections 2024-03-06T07:34:16.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.892+0000 7f92e26df640 1 -- 172.21.15.77:0/2613956718 wait complete. 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.893+0000 7f92e26df640 1 Processor -- start 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.893+0000 7f92e26df640 1 -- start start 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92e26df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 0x7f92dc1f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92e26df640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92dc15ac00 0x7f92dc14b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92e26df640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92dc15e7a0 0x7f92dc14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92e26df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0cd8a0 con 0x7f92dc0708b0 2024-03-06T07:34:16.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92e26df640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0cd5c0 con 0x7f92dc15e7a0 2024-03-06T07:34:16.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92e26df640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc06ce50 con 0x7f92dc15ac00 2024-03-06T07:34:16.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 0x7f92dc1f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 0x7f92dc1f4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46010/0 (socket says 172.21.15.77:46010) 2024-03-06T07:34:16.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92dbfff640 1 -- 172.21.15.77:0/1806990770 learned_addr learned my addr 172.21.15.77:0/1806990770 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:16.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92dbfff640 1 -- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92dc15ac00 msgr2=0x7f92dc14b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92dbfff640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f92dc15ac00 0x7f92dc14b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92dbfff640 1 -- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92dc15e7a0 msgr2=0x7f92dc14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92e0c7c640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92dc15e7a0 0x7f92dc14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92dbfff640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92dc15e7a0 0x7f92dc14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.894+0000 7f92dbfff640 1 -- 172.21.15.77:0/1806990770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc1f4a70 con 0x7f92dc0708b0 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.895+0000 7f92e0c7c640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f92dc15e7a0 0x7f92dc14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.895+0000 7f92dbfff640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 0x7f92dc1f4530 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00d800 tx=0x7f92cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.895+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc018020 con 0x7f92dc0708b0 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.895+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc0096a0 con 0x7f92dc0708b0 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.895+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92dc15b590 con 0x7f92dc0708b0 2024-03-06T07:34:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.895+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc009a70 con 0x7f92dc0708b0 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.895+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc15bb50 con 0x7f92dc0708b0 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.896+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f92dc04f330 con 0x7f92dc0708b0 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.896+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92cc01b020 con 0x7f92dc0708b0 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.897+0000 7f92d8ff9640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92c806b4d0 0x7f92c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.897+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f92cc08a180 con 0x7f92dc0708b0 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.897+0000 7f92db7fe640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92c806b4d0 0x7f92c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.897+0000 7f92d8ff9640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f92c8071430 0x7f92c8073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.897+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92cc00cfb0 con 0x7f92c8071430 2024-03-06T07:34:16.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.897+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92cc091050 con 0x7f92dc0708b0 2024-03-06T07:34:16.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.897+0000 7f92db7fe640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92c806b4d0 0x7f92c806d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f92d0036000 tx=0x7f92d0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.897+0000 7f92e0c7c640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f92c8071430 0x7f92c8073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.898+0000 7f92e0c7c640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f92c8071430 0x7f92c8073850 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.898+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 <== osd.7 v2:172.21.15.158:6816/3281717214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f92cc00cfb0 con 0x7f92c8071430 2024-03-06T07:34:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.911+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 --> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f92dc15c2b0 con 0x7f92c8071430 2024-03-06T07:34:16.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92d8ff9640 1 -- 172.21.15.77:0/1806990770 <== osd.7 v2:172.21.15.158:6816/3281717214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f92dc15c2b0 con 0x7f92c8071430 2024-03-06T07:34:16.914 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-06T07:34:16.914 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:16.914 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-06T07:34:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f92c8071430 msgr2=0x7f92c8073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.158:6816/3281717214,v1:172.21.15.158:6817/3281717214] conn(0x7f92c8071430 0x7f92c8073850 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92c806b4d0 msgr2=0x7f92c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f92c806b4d0 0x7f92c806d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f92d0036000 tx=0x7f92d0030000 comp rx=0 tx=0).stop 2024-03-06T07:34:16.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 msgr2=0x7f92dc1f4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 --2- 172.21.15.77:0/1806990770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0708b0 0x7f92dc1f4530 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00d800 tx=0x7f92cc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e0c7c640 1 -- 172.21.15.77:0/1806990770 reap_dead start 2024-03-06T07:34:16.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 shutdown_connections 2024-03-06T07:34:16.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 >> 172.21.15.77:0/1806990770 conn(0x7f92dc0b1650 msgr2=0x7f92dc0b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 shutdown_connections 2024-03-06T07:34:16.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:16.912+0000 7f92e26df640 1 -- 172.21.15.77:0/1806990770 wait complete. 2024-03-06T07:34:16.925 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:34:17.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.024+0000 7f86b93a2640 1 Processor -- start 2024-03-06T07:34:17.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.024+0000 7f86b93a2640 1 -- start start 2024-03-06T07:34:17.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b93a2640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f86b415eea0 0x7f86b40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b93a2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b4161870 0x7f86b4161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b93a2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b415e0e0 con 0x7f86b406afb0 2024-03-06T07:34:17.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b93a2640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b415f280 con 0x7f86b415eea0 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b93a2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b40ce560 con 0x7f86b4161870 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b8ba1640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f86b406afb0 0x7f86b406d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:59874/0 (socket says 172.21.15.77:59874) 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b8ba1640 1 -- 172.21.15.77:0/3026267556 learned_addr learned my addr 172.21.15.77:0/3026267556 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488618348 0 0) 0x7f86b415e0e0 con 0x7f86b406afb0 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f869c003520 con 0x7f86b406afb0 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3026267556 >> v2:172.21.15.158:3300/0 conn(0x7f86b415eea0 0x7f86b40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b37fe640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b4161870 0x7f86b4161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3022355157 0 0) 0x7f869c003520 con 0x7f86b406afb0 2024-03-06T07:34:17.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b4161870 msgr2=0x7f86b4161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b4161870 0x7f86b4161cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 >> v2:172.21.15.158:3300/0 conn(0x7f86b415eea0 msgr2=0x7f86b40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 --2- 172.21.15.77:0/3026267556 >> v2:172.21.15.158:3300/0 conn(0x7f86b415eea0 0x7f86b40ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b4162330 con 0x7f86b406afb0 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 408338 0 0) 0x7f86ac0019f0 con 0x7f86b406afb0 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3026267556 >> v2:172.21.15.158:3300/0 conn(0x7f86b415eea0 0x7f86b40ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 >> v1:172.21.15.77:6789/0 conn(0x7f86b406afb0 legacy=0x7f86b406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c003a20 0x7f869c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c004800 0x7f869c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b4161870 0x7f869c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b415f280 con 0x7f869c003a20 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b40ce560 con 0x7f869c004800 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c00baf0 con 0x7f86b4161870 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.025+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c003a20 0x7f869c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.026+0000 7f86b37fe640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c004800 0x7f869c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.026+0000 7f86b8ba1640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b4161870 0x7f869c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.026+0000 7f86b3fff640 1 -- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b4161870 msgr2=0x7f869c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.026+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b4161870 0x7f869c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.026+0000 7f86b3fff640 1 -- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c004800 msgr2=0x7f869c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.026+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c004800 0x7f869c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.026+0000 7f86b3fff640 1 -- 172.21.15.77:0/3026267556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c003520 con 0x7f869c003a20 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.026+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c003a20 0x7f869c003e40 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f86a8006fd0 tx=0x7f869c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445fafce3aa08847 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a800f040 con 0x7f869c003a20 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a8004230 con 0x7f869c003a20 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b2ffd640 1 -- 172.21.15.77:0/3026267556 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a8003610 con 0x7f869c003a20 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b93a2640 1 -- 172.21.15.77:0/3026267556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c003a20 msgr2=0x7f869c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b93a2640 1 --2- 172.21.15.77:0/3026267556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f869c003a20 0x7f869c003e40 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f86a8006fd0 tx=0x7f869c005830 comp rx=0 tx=0).stop 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b8ba1640 1 -- 172.21.15.77:0/3026267556 reap_dead start 2024-03-06T07:34:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b93a2640 1 -- 172.21.15.77:0/3026267556 shutdown_connections 2024-03-06T07:34:17.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b93a2640 1 -- 172.21.15.77:0/3026267556 >> 172.21.15.77:0/3026267556 conn(0x7f86b40b19b0 msgr2=0x7f86b40b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b93a2640 1 -- 172.21.15.77:0/3026267556 shutdown_connections 2024-03-06T07:34:17.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.027+0000 7f86b93a2640 1 -- 172.21.15.77:0/3026267556 wait complete. 2024-03-06T07:34:17.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.028+0000 7f86b93a2640 1 Processor -- start 2024-03-06T07:34:17.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b93a2640 1 -- start start 2024-03-06T07:34:17.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b93a2640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c003a20 0x7f86b4158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b93a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b40c9c40 0x7f86b415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b93a2640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b406da90 0x7f86b406df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b93a2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b415e4f0 con 0x7f86b40c9c40 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b93a2640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b40ce1f0 con 0x7f869c003a20 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b93a2640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b40cef30 con 0x7f86b406da90 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b40c9c40 0x7f86b415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b40c9c40 0x7f86b415b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46024/0 (socket says 172.21.15.77:46024) 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b37fe640 1 -- 172.21.15.77:0/3029065433 learned_addr learned my addr 172.21.15.77:0/3029065433 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:17.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c003a20 0x7f86b4158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.029+0000 7f86b8ba1640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b406da90 0x7f86b406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b37fe640 1 -- 172.21.15.77:0/3029065433 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b406da90 msgr2=0x7f86b406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b37fe640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b406da90 0x7f86b406df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b37fe640 1 -- 172.21.15.77:0/3029065433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c003a20 msgr2=0x7f86b4158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b37fe640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c003a20 0x7f86b4158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b37fe640 1 -- 172.21.15.77:0/3029065433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b40cb1c0 con 0x7f86b40c9c40 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b37fe640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b40c9c40 0x7f86b415b8c0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f86a40119a0 tx=0x7f86a4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b0ff9640 1 -- 172.21.15.77:0/3029065433 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a400faf0 con 0x7f86b40c9c40 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b0ff9640 1 -- 172.21.15.77:0/3029065433 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a400fc90 con 0x7f86b40c9c40 2024-03-06T07:34:17.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b0ff9640 1 -- 172.21.15.77:0/3029065433 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4014440 con 0x7f86b40c9c40 2024-03-06T07:34:17.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b40ca120 con 0x7f86b40c9c40 2024-03-06T07:34:17.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.030+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b40ca6e0 con 0x7f86b40c9c40 2024-03-06T07:34:17.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.031+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b4068f80 con 0x7f86b40c9c40 2024-03-06T07:34:17.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.031+0000 7f86b0ff9640 1 -- 172.21.15.77:0/3029065433 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86a4012040 con 0x7f86b40c9c40 2024-03-06T07:34:17.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.032+0000 7f86b0ff9640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a006b500 0x7f86a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.032+0000 7f86b0ff9640 1 -- 172.21.15.77:0/3029065433 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f86a408a580 con 0x7f86b40c9c40 2024-03-06T07:34:17.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.032+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a006b500 0x7f86a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.032+0000 7f86b3fff640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a006b500 0x7f86a006d9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f86a8035000 tx=0x7f86a803a040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.036+0000 7f86b0ff9640 1 -- 172.21.15.77:0/3029065433 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86a4019030 con 0x7f86b40c9c40 2024-03-06T07:34:17.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.207+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f86b40cdee0 con 0x7f86a006b500 2024-03-06T07:34:17.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.208+0000 7f86b0ff9640 1 -- 172.21.15.77:0/3029065433 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f86b40cdee0 con 0x7f86a006b500 2024-03-06T07:34:17.210 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:34:17.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a006b500 msgr2=0x7f86a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a006b500 0x7f86a006d9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f86a8035000 tx=0x7f86a803a040 comp rx=0 tx=0).stop 2024-03-06T07:34:17.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b40c9c40 msgr2=0x7f86b415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b40c9c40 0x7f86b415b8c0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f86a40119a0 tx=0x7f86a4011e60 comp rx=0 tx=0).stop 2024-03-06T07:34:17.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 shutdown_connections 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f86a006b500 0x7f86a006d9c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b40c9c40 0x7f86b415b8c0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f86b406da90 0x7f86b406df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 --2- 172.21.15.77:0/3029065433 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f869c003a20 0x7f86b4158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 >> 172.21.15.77:0/3029065433 conn(0x7f86b40b19b0 msgr2=0x7f86b40b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 shutdown_connections 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.210+0000 7f86b93a2640 1 -- 172.21.15.77:0/3029065433 wait complete. 2024-03-06T07:34:17.222 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:17.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.320+0000 7fd6aef5e640 1 Processor -- start 2024-03-06T07:34:17.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.320+0000 7fd6aef5e640 1 -- start start 2024-03-06T07:34:17.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd6aef5e640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fd6a81515e0 0x7fd6a80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd6aef5e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6a815b430 0x7fd6a8071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd6aef5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a80cd7a0 0x7fd6a80cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd6aef5e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a815b8e0 con 0x7fd6a815b430 2024-03-06T07:34:17.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd6aef5e640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a8070a70 con 0x7fd6a81515e0 2024-03-06T07:34:17.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd6aef5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8070d50 con 0x7fd6a80cd7a0 2024-03-06T07:34:17.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd69ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6a815b430 0x7fd6a8071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd69ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6a815b430 0x7fd6a8071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46030/0 (socket says 172.21.15.77:46030) 2024-03-06T07:34:17.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd69ffff640 1 -- 172.21.15.77:0/3813529621 learned_addr learned my addr 172.21.15.77:0/3813529621 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:17.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd69ffff640 1 -- 172.21.15.77:0/3813529621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a80cd7a0 msgr2=0x7fd6a80cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd69ffff640 1 --2- 172.21.15.77:0/3813529621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a80cd7a0 0x7fd6a80cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd69ffff640 1 -- 172.21.15.77:0/3813529621 >> v2:172.21.15.158:3300/0 conn(0x7fd6a81515e0 msgr2=0x7fd6a80cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd69ffff640 1 --2- 172.21.15.77:0/3813529621 >> v2:172.21.15.158:3300/0 conn(0x7fd6a81515e0 0x7fd6a80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.321+0000 7fd69ffff640 1 -- 172.21.15.77:0/3813529621 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a8071030 con 0x7fd6a815b430 2024-03-06T07:34:17.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd69ffff640 1 --2- 172.21.15.77:0/3813529621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8071830 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fd6a806eb50 tx=0x7fd69802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb9e3ebf7607605 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd69f7fe640 1 -- 172.21.15.77:0/3813529621 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698035020 con 0x7fd6a815b430 2024-03-06T07:34:17.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd69f7fe640 1 -- 172.21.15.77:0/3813529621 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd698031870 con 0x7fd6a815b430 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd69f7fe640 1 -- 172.21.15.77:0/3813529621 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698031c40 con 0x7fd6a815b430 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3813529621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 msgr2=0x7fd6a8071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3813529621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8071830 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fd6a806eb50 tx=0x7fd69802b910 comp rx=0 tx=0).stop 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3813529621 shutdown_connections 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3813529621 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a80cd7a0 0x7fd6a80cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3813529621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8071830 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3813529621 >> v2:172.21.15.158:3300/0 conn(0x7fd6a81515e0 0x7fd6a80cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3813529621 >> 172.21.15.77:0/3813529621 conn(0x7fd6a80cf8a0 msgr2=0x7fd6a80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3813529621 shutdown_connections 2024-03-06T07:34:17.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.322+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3813529621 wait complete. 2024-03-06T07:34:17.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.324+0000 7fd6aef5e640 1 Processor -- start 2024-03-06T07:34:17.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.324+0000 7fd6aef5e640 1 -- start start 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.324+0000 7fd6aef5e640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a80cd7a0 0x7fd6a81f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.324+0000 7fd6aef5e640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a81515e0 0x7fd6a81e4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.324+0000 7fd6aef5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.324+0000 7fd6aef5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8071d70 con 0x7fd6a815b430 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.324+0000 7fd6aef5e640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80ce0f0 con 0x7fd6a80cd7a0 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.324+0000 7fd6aef5e640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8071ef0 con 0x7fd6a81515e0 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46034/0 (socket says 172.21.15.77:46034) 2024-03-06T07:34:17.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 -- 172.21.15.77:0/3103910407 learned_addr learned my addr 172.21.15.77:0/3103910407 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd69ffff640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a81515e0 0x7fd6a81e4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6accfa640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a80cd7a0 0x7fd6a81f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 -- 172.21.15.77:0/3103910407 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a81515e0 msgr2=0x7fd6a81e4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a81515e0 0x7fd6a81e4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 -- 172.21.15.77:0/3103910407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a80cd7a0 msgr2=0x7fd6a81f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a80cd7a0 0x7fd6a81f3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 -- 172.21.15.77:0/3103910407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a81582b0 con 0x7fd6a815b430 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6ad4fb640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8159070 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000b970 tx=0x7fd6a000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd69d7fa640 1 -- 172.21.15.77:0/3103910407 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0019070 con 0x7fd6a815b430 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd69d7fa640 1 -- 172.21.15.77:0/3103910407 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a000cb60 con 0x7fd6a815b430 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a8158590 con 0x7fd6a815b430 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd69d7fa640 1 -- 172.21.15.77:0/3103910407 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0014440 con 0x7fd6a815b430 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.325+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a81e54c0 con 0x7fd6a815b430 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.326+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a8068840 con 0x7fd6a815b430 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.326+0000 7fd69d7fa640 1 -- 172.21.15.77:0/3103910407 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6a001c070 con 0x7fd6a815b430 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.327+0000 7fd69d7fa640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd69406b4d0 0x7fd69406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.327+0000 7fd6accfa640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd69406b4d0 0x7fd69406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.327+0000 7fd69d7fa640 1 -- 172.21.15.77:0/3103910407 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd6a0089c20 con 0x7fd6a815b430 2024-03-06T07:34:17.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.327+0000 7fd6accfa640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd69406b4d0 0x7fd69406d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80b15b0 tx=0x7fd6a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.331+0000 7fd69d7fa640 1 -- 172.21.15.77:0/3103910407 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6a0058dd0 con 0x7fd6a815b430 2024-03-06T07:34:17.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.503+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6a8158060 con 0x7fd69406b4d0 2024-03-06T07:34:17.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.503+0000 7fd69d7fa640 1 -- 172.21.15.77:0/3103910407 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13504 (secure 0 0 0) 0x7fd6a8158060 con 0x7fd69406b4d0 2024-03-06T07:34:17.505 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:17.506 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:34:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.505+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd69406b4d0 msgr2=0x7fd69406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.505+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd69406b4d0 0x7fd69406d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80b15b0 tx=0x7fd6a4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.505+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 msgr2=0x7fd6a8159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.505+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8159070 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000b970 tx=0x7fd6a000be30 comp rx=0 tx=0).stop 2024-03-06T07:34:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.506+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 shutdown_connections 2024-03-06T07:34:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.506+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fd69406b4d0 0x7fd69406d990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.506+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6a815b430 0x7fd6a8159070 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.506+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fd6a81515e0 0x7fd6a81e4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.506+0000 7fd6aef5e640 1 --2- 172.21.15.77:0/3103910407 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd6a80cd7a0 0x7fd6a81f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.506+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 >> 172.21.15.77:0/3103910407 conn(0x7fd6a80cf8a0 msgr2=0x7fd6a80cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.506+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 shutdown_connections 2024-03-06T07:34:17.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:17.506+0000 7fd6aef5e640 1 -- 172.21.15.77:0/3103910407 wait complete. 2024-03-06T07:34:17.518 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-03-06T07:34:15.518705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":319048,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754655672,"statfs":{"total":773094113280,"available":772767408128,"internally_reserved":0,"allocated":15867904,"data_stored":7612016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":310693424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-31593,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-140,"num_write_kb":-79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001437"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:09.781565+0000","last_change":"2024-03-06T07:21:55.595419+0000","last_active":"2024-03-06T07:34:09.781565+0000","last_peered":"2024-03-06T07:34:09.781565+0000","last_clean":"2024-03-06T07:34:09.781565+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:34:09.781565+0000","last_undegraded":"2024-03-06T07:34:09.781565+0000","last_fullsized":"2024-03-06T07:34:09.781565+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.911619+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.911619+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:51.904741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574999,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46036,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325804,"statfs":{"total":96636764160,"available":96589623296,"internally_reserved":0,"allocated":2428928,"data_stored":1394278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":44693962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28816,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343024,"statfs":{"total":96636764160,"available":96607256576,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45456,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326384,"statfs":{"total":96636764160,"available":96590217216,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574999,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574999,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574999,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41876,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329964,"statfs":{"total":96636764160,"available":96593883136,"internally_reserved":0,"allocated":2428928,"data_stored":1394278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:17.520 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:21:53.911619+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=21, tm_sec=53, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=34, tm_sec=12, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:34:17.521 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:34:32.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:32.789 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:37.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:37.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.632+0000 7f307daf4640 1 Processor -- start 2024-03-06T07:34:37.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.632+0000 7f307daf4640 1 -- start start 2024-03-06T07:34:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.632+0000 7f307daf4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307813dc10 0x7f30780b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.632+0000 7f307daf4640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f307815bf30 0x7f30780b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.632+0000 7f307daf4640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307815f580 con 0x7f307815e850 2024-03-06T07:34:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.632+0000 7f307daf4640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30780cf3f0 con 0x7f307815bf30 2024-03-06T07:34:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.632+0000 7f307daf4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307815b210 con 0x7f307813dc10 2024-03-06T07:34:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.633+0000 7f3076ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f307815bf30 0x7f30780b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.633+0000 7f30777fe640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307813dc10 0x7f30780b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.633+0000 7f3076ffd640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f307815bf30 0x7f30780b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.77:60994/0 (socket says 172.21.15.77:60994) 2024-03-06T07:34:37.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.633+0000 7f3076ffd640 1 -- 172.21.15.77:0/338579037 learned_addr learned my addr 172.21.15.77:0/338579037 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:37.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.633+0000 7f30777fe640 1 -- 172.21.15.77:0/338579037 >> v2:172.21.15.158:3300/0 conn(0x7f307815bf30 msgr2=0x7f30780b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:37.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.633+0000 7f30777fe640 1 --2- 172.21.15.77:0/338579037 >> v2:172.21.15.158:3300/0 conn(0x7f307815bf30 0x7f30780b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.633+0000 7f30777fe640 1 -- 172.21.15.77:0/338579037 >> v1:172.21.15.158:6789/0 conn(0x7f307815e850 legacy=0x7f307815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.633+0000 7f30777fe640 1 -- 172.21.15.77:0/338579037 --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307815b450 con 0x7f307813dc10 2024-03-06T07:34:37.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.634+0000 7f30777fe640 1 --2- 172.21.15.77:0/338579037 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307813dc10 0x7f30780b2950 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f30680097a0 tx=0x7f306802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71c50cc4b87ffb3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.634+0000 7f30767fc640 1 -- 172.21.15.77:0/338579037 <== mon.2 v2:172.21.15.158:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30680315a0 con 0x7f307813dc10 2024-03-06T07:34:37.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.634+0000 7f30767fc640 1 -- 172.21.15.77:0/338579037 <== mon.2 v2:172.21.15.158:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3068037070 con 0x7f307813dc10 2024-03-06T07:34:37.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.634+0000 7f30767fc640 1 -- 172.21.15.77:0/338579037 <== mon.2 v2:172.21.15.158:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068031960 con 0x7f307813dc10 2024-03-06T07:34:37.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.635+0000 7f307daf4640 1 -- 172.21.15.77:0/338579037 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307813dc10 msgr2=0x7f30780b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.635+0000 7f307daf4640 1 --2- 172.21.15.77:0/338579037 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307813dc10 0x7f30780b2950 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f30680097a0 tx=0x7f306802b840 comp rx=0 tx=0).stop 2024-03-06T07:34:37.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.635+0000 7f307daf4640 1 -- 172.21.15.77:0/338579037 shutdown_connections 2024-03-06T07:34:37.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.635+0000 7f307daf4640 1 --2- 172.21.15.77:0/338579037 >> v2:172.21.15.158:3300/0 conn(0x7f307815bf30 0x7f30780b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.635+0000 7f307daf4640 1 --2- 172.21.15.77:0/338579037 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307813dc10 0x7f30780b2950 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.635+0000 7f307daf4640 1 -- 172.21.15.77:0/338579037 >> 172.21.15.77:0/338579037 conn(0x7f30780cf8a0 msgr2=0x7f30780cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:37.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.635+0000 7f307daf4640 1 -- 172.21.15.77:0/338579037 shutdown_connections 2024-03-06T07:34:37.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.635+0000 7f307daf4640 1 -- 172.21.15.77:0/338579037 wait complete. 2024-03-06T07:34:37.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.636+0000 7f307daf4640 1 Processor -- start 2024-03-06T07:34:37.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.636+0000 7f307daf4640 1 -- start start 2024-03-06T07:34:37.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f307daf4640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f307813dc10 0x7f307812a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f307daf4640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307815bf30 0x7f3078072120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f307daf4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f307815e850 0x7f3078135c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f307daf4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30780cd7a0 con 0x7f307815e850 2024-03-06T07:34:37.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f307daf4640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307813dff0 con 0x7f307813dc10 2024-03-06T07:34:37.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f307daf4640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30780cd920 con 0x7f307815bf30 2024-03-06T07:34:37.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f3077fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f307815e850 0x7f3078135c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f3077fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f307815e850 0x7f3078135c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56438/0 (socket says 172.21.15.77:56438) 2024-03-06T07:34:37.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f3077fff640 1 -- 172.21.15.77:0/1112396291 learned_addr learned my addr 172.21.15.77:0/1112396291 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:37.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.637+0000 7f3076ffd640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307815bf30 0x7f3078072120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f30777fe640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f307813dc10 0x7f307812a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f3077fff640 1 -- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307815bf30 msgr2=0x7f3078072120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f3077fff640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307815bf30 0x7f3078072120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f3077fff640 1 -- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f307813dc10 msgr2=0x7f307812a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f3077fff640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f307813dc10 0x7f307812a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f3077fff640 1 -- 172.21.15.77:0/1112396291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307812a770 con 0x7f307815e850 2024-03-06T07:34:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f30777fe640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f307813dc10 0x7f307812a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f3077fff640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f307815e850 0x7f3078135c00 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f307000c6e0 tx=0x7f307000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f305ffff640 1 -- 172.21.15.77:0/1112396291 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070003b70 con 0x7f307815e850 2024-03-06T07:34:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f305ffff640 1 -- 172.21.15.77:0/1112396291 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3070003d30 con 0x7f307815e850 2024-03-06T07:34:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30781549c0 con 0x7f307815e850 2024-03-06T07:34:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f305ffff640 1 -- 172.21.15.77:0/1112396291 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070005310 con 0x7f307815e850 2024-03-06T07:34:37.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.638+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3078154f80 con 0x7f307815e850 2024-03-06T07:34:37.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.639+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307804f330 con 0x7f307815e850 2024-03-06T07:34:37.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.639+0000 7f305ffff640 1 -- 172.21.15.77:0/1112396291 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30700054b0 con 0x7f307815e850 2024-03-06T07:34:37.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.640+0000 7f305ffff640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f306406b4d0 0x7f306406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.640+0000 7f305ffff640 1 -- 172.21.15.77:0/1112396291 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3070089da0 con 0x7f307815e850 2024-03-06T07:34:37.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.640+0000 7f30777fe640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f306406b4d0 0x7f306406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.640+0000 7f30777fe640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f306406b4d0 0x7f306406d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3068036000 tx=0x7f3068008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.644+0000 7f305ffff640 1 -- 172.21.15.77:0/1112396291 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3070058ec0 con 0x7f307815e850 2024-03-06T07:34:37.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.814+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 --> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f307815d840 con 0x7f306406b4d0 2024-03-06T07:34:37.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.815+0000 7f305ffff640 1 -- 172.21.15.77:0/1112396291 <== mgr.4107 v2:172.21.15.77:6832/885960424 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13379 (secure 0 0 0) 0x7f307815d840 con 0x7f306406b4d0 2024-03-06T07:34:37.817 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:37.817 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-06T07:34:37.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f306406b4d0 msgr2=0x7f306406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f306406b4d0 0x7f306406d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3068036000 tx=0x7f3068008030 comp rx=0 tx=0).stop 2024-03-06T07:34:37.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f307815e850 msgr2=0x7f3078135c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f307815e850 0x7f3078135c00 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f307000c6e0 tx=0x7f307000cba0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 shutdown_connections 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f306406b4d0 0x7f306406d990 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f307815e850 0x7f3078135c00 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f307815bf30 0x7f3078072120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 --2- 172.21.15.77:0/1112396291 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f307813dc10 0x7f307812a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 >> 172.21.15.77:0/1112396291 conn(0x7f30780cf8a0 msgr2=0x7f307815d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 shutdown_connections 2024-03-06T07:34:37.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.817+0000 7f307daf4640 1 -- 172.21.15.77:0/1112396291 wait complete. 2024-03-06T07:34:37.830 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-03-06T07:34:37.523492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":319032,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754655688,"statfs":{"total":773094113280,"available":772767424512,"internally_reserved":0,"allocated":15867904,"data_stored":7612016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":310693424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002617"},"pg_stats":[{"pgid":"1.0","version":"146'119","reported_seq":409,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:17.448298+0000","last_change":"2024-03-06T07:34:17.448332+0000","last_active":"2024-03-06T07:34:17.448298+0000","last_peered":"2024-03-06T07:34:17.448298+0000","last_clean":"2024-03-06T07:34:17.448298+0000","last_became_active":"2024-03-06T07:21:55.594399+0000","last_became_peered":"2024-03-06T07:21:55.594399+0000","last_unstale":"2024-03-06T07:34:17.448298+0000","last_undegraded":"2024-03-06T07:34:17.448298+0000","last_fullsized":"2024-03-06T07:34:17.448298+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'119","last_scrub_stamp":"2024-03-06T07:34:17.448213+0000","last_deep_scrub":"146'119","last_deep_scrub_stamp":"2024-03-06T07:34:17.448213+0000","last_clean_scrub_stamp":"2024-03-06T07:34:17.448213+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:26.137101+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46028,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325812,"statfs":{"total":96636764160,"available":96589631488,"internally_reserved":0,"allocated":2428928,"data_stored":1394278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":44693962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28816,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343024,"statfs":{"total":96636764160,"available":96607256576,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45456,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326384,"statfs":{"total":96636764160,"available":96590217216,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":803910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2428928,"data_stored":1394278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:37.831 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:34:37.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.929+0000 7fbcf20c3640 1 Processor -- start 2024-03-06T07:34:37.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.929+0000 7fbcf20c3640 1 -- start start 2024-03-06T07:34:37.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcf20c3640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7fbcec066d90 0x7fbcec0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcf20c3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbcec0b5410 0x7fbcec0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcf20c3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec06b710 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcf20c3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcec0b5880 con 0x7fbcec0b5410 2024-03-06T07:34:37.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcf20c3640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcec0b61c0 con 0x7fbcec066d90 2024-03-06T07:34:37.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcebfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbcec0b5410 0x7fbcec0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbceb7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbcec0cf9a0 0x7fbcec0b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:57920/0 (socket says 172.21.15.77:57920) 2024-03-06T07:34:37.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbceb7fe640 1 -- 172.21.15.77:0/2468849432 learned_addr learned my addr 172.21.15.77:0/2468849432 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-03-06T07:34:37.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcebfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbcec0b5410 0x7fbcec0ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56440/0 (socket says 172.21.15.77:56440) 2024-03-06T07:34:37.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcebfff640 1 -- 172.21.15.77:0/2468849432 >> v2:172.21.15.158:3300/0 conn(0x7fbcec066d90 msgr2=0x7fbcec0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:37.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbceaffd640 1 --2- 172.21.15.77:0/2468849432 >> v2:172.21.15.158:3300/0 conn(0x7fbcec066d90 0x7fbcec0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcebfff640 1 --2- 172.21.15.77:0/2468849432 >> v2:172.21.15.158:3300/0 conn(0x7fbcec066d90 0x7fbcec0b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcea7fc640 1 -- 172.21.15.77:0/2468849432 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918246372 0 0) 0x7fbcec06b710 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcebfff640 1 -- 172.21.15.77:0/2468849432 >> v1:172.21.15.77:6789/0 conn(0x7fbcec0cf9a0 legacy=0x7fbcec0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbcebfff640 1 -- 172.21.15.77:0/2468849432 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec06b150 con 0x7fbcec0b5410 2024-03-06T07:34:37.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.930+0000 7fbceaffd640 1 --2- 172.21.15.77:0/2468849432 >> v2:172.21.15.158:3300/0 conn(0x7fbcec066d90 0x7fbcec0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:37.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.931+0000 7fbcebfff640 1 --2- 172.21.15.77:0/2468849432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0b5410 0x7fbcec0ce040 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0d11d0 tx=0x7fbce402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f004dbef5df6ea11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.931+0000 7fbcea7fc640 1 -- 172.21.15.77:0/2468849432 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce4039070 con 0x7fbcec0b5410 2024-03-06T07:34:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.931+0000 7fbcea7fc640 1 -- 172.21.15.77:0/2468849432 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce402cc80 con 0x7fbcec0b5410 2024-03-06T07:34:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.931+0000 7fbcea7fc640 1 -- 172.21.15.77:0/2468849432 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce4034660 con 0x7fbcec0b5410 2024-03-06T07:34:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.932+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2468849432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0b5410 msgr2=0x7fbcec0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.932+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2468849432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0b5410 0x7fbcec0ce040 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0d11d0 tx=0x7fbce402bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.932+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2468849432 shutdown_connections 2024-03-06T07:34:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.932+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2468849432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0b5410 0x7fbcec0ce040 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.932+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2468849432 >> v2:172.21.15.158:3300/0 conn(0x7fbcec066d90 0x7fbcec0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.932+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2468849432 >> 172.21.15.77:0/2468849432 conn(0x7fbcec0b1d70 msgr2=0x7fbcec0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.932+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2468849432 shutdown_connections 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.932+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2468849432 wait complete. 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.933+0000 7fbcf20c3640 1 Processor -- start 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.933+0000 7fbcf20c3640 1 -- start start 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcf20c3640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcec066d90 0x7fbcec15efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcf20c3640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcec0b5410 0x7fbcec0ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcf20c3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0cf9a0 0x7fbcec0cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcf20c3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0b0c50 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcf20c3640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0b0970 con 0x7fbcec0b5410 2024-03-06T07:34:37.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcf20c3640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0cfd80 con 0x7fbcec066d90 2024-03-06T07:34:37.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0cf9a0 0x7fbcec0cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0cf9a0 0x7fbcec0cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56454/0 (socket says 172.21.15.77:56454) 2024-03-06T07:34:37.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 -- 172.21.15.77:0/2637426720 learned_addr learned my addr 172.21.15.77:0/2637426720 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:37.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 -- 172.21.15.77:0/2637426720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcec066d90 msgr2=0x7fbcec15efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcec066d90 0x7fbcec15efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 -- 172.21.15.77:0/2637426720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcec0b5410 msgr2=0x7fbcec0ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcec0b5410 0x7fbcec0ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 -- 172.21.15.77:0/2637426720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec0ae9e0 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcebfff640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0cf9a0 0x7fbcec0cd400 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009e80 tx=0x7fbce4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.934+0000 7fbcd3fff640 1 -- 172.21.15.77:0/2637426720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce4039050 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.935+0000 7fbcd3fff640 1 -- 172.21.15.77:0/2637426720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce4045070 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.935+0000 7fbcd3fff640 1 -- 172.21.15.77:0/2637426720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce403e810 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.935+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec0adc20 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.935+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec0ae1e0 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.935+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcec09e3d0 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.936+0000 7fbcd3fff640 1 -- 172.21.15.77:0/2637426720 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbce403e9b0 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.936+0000 7fbcd3fff640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcd806b4d0 0x7fbcd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.937+0000 7fbceb7fe640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcd806b4d0 0x7fbcd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.937+0000 7fbcd3fff640 1 -- 172.21.15.77:0/2637426720 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbce40acd50 con 0x7fbcec0cf9a0 2024-03-06T07:34:37.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.937+0000 7fbceb7fe640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcd806b4d0 0x7fbcd806d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc007fd0 tx=0x7fbcdc007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:37.941+0000 7fbcd3fff640 1 -- 172.21.15.77:0/2637426720 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbce40b0030 con 0x7fbcec0cf9a0 2024-03-06T07:34:38.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.112+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fbcec0c6fd0 con 0x7fbcec0cf9a0 2024-03-06T07:34:38.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.115+0000 7fbcd3fff640 1 -- 172.21.15.77:0/2637426720 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v36)=0 v36) v1 ==== 123+0+0 (secure 0 0 0) 0x7fbce4032e20 con 0x7fbcec0cf9a0 2024-03-06T07:34:38.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.116+0000 7fbcd3fff640 1 -- 172.21.15.77:0/2637426720 <== mon.0 v2:172.21.15.77:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbce403a040 con 0x7fbcec0cf9a0 2024-03-06T07:34:38.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcd806b4d0 msgr2=0x7fbcd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcd806b4d0 0x7fbcd806d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc007fd0 tx=0x7fbcdc007960 comp rx=0 tx=0).stop 2024-03-06T07:34:38.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0cf9a0 msgr2=0x7fbcec0cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0cf9a0 0x7fbcec0cd400 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009e80 tx=0x7fbce4004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:38.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 shutdown_connections 2024-03-06T07:34:38.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7fbcd806b4d0 0x7fbcd806d990 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcec0cf9a0 0x7fbcec0cd400 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcec0b5410 0x7fbcec0ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 --2- 172.21.15.77:0/2637426720 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7fbcec066d90 0x7fbcec15efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 >> 172.21.15.77:0/2637426720 conn(0x7fbcec0b1d70 msgr2=0x7fbcec161570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:38.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 shutdown_connections 2024-03-06T07:34:38.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.118+0000 7fbcf20c3640 1 -- 172.21.15.77:0/2637426720 wait complete. 2024-03-06T07:34:38.130 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:34:38.131 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:38.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.229+0000 7f22d47ae640 1 Processor -- start 2024-03-06T07:34:38.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.229+0000 7f22d47ae640 1 -- start start 2024-03-06T07:34:38.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d47ae640 1 --2- >> v2:172.21.15.158:3300/0 conn(0x7f22cc1515e0 0x7f22cc0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d47ae640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22cc15b430 0x7f22cc071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d47ae640 1 -- --> v1:172.21.15.158:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22cc15b8e0 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d47ae640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22cc070a70 con 0x7f22cc15b430 2024-03-06T07:34:38.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d47ae640 1 -- --> v2:172.21.15.158:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22cc070d50 con 0x7f22cc1515e0 2024-03-06T07:34:38.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d1d49640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22cc15b430 0x7f22cc071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d1d49640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22cc15b430 0x7f22cc071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56460/0 (socket says 172.21.15.77:56460) 2024-03-06T07:34:38.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d1d49640 1 -- 172.21.15.77:0/3050517572 learned_addr learned my addr 172.21.15.77:0/3050517572 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:38.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d1d49640 1 -- 172.21.15.77:0/3050517572 >> v2:172.21.15.158:3300/0 conn(0x7f22cc1515e0 msgr2=0x7f22cc0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:38.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d1d49640 1 --2- 172.21.15.77:0/3050517572 >> v2:172.21.15.158:3300/0 conn(0x7f22cc1515e0 0x7f22cc0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d1d49640 1 -- 172.21.15.77:0/3050517572 >> v1:172.21.15.158:6789/0 conn(0x7f22cc0cd7a0 legacy=0x7f22cc0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:38.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.230+0000 7f22d1d49640 1 -- 172.21.15.77:0/3050517572 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc071030 con 0x7f22cc15b430 2024-03-06T07:34:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.231+0000 7f22d1d49640 1 --2- 172.21.15.77:0/3050517572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc15b430 0x7f22cc071830 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0cc110 tx=0x7f22bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcad3837add9b6ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.231+0000 7f22d1548640 1 -- 172.21.15.77:0/3050517572 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc035020 con 0x7f22cc15b430 2024-03-06T07:34:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.231+0000 7f22d1548640 1 -- 172.21.15.77:0/3050517572 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f22bc031870 con 0x7f22cc15b430 2024-03-06T07:34:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.231+0000 7f22d1548640 1 -- 172.21.15.77:0/3050517572 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc031c60 con 0x7f22cc15b430 2024-03-06T07:34:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.232+0000 7f22d47ae640 1 -- 172.21.15.77:0/3050517572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc15b430 msgr2=0x7f22cc071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.232+0000 7f22d47ae640 1 --2- 172.21.15.77:0/3050517572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc15b430 0x7f22cc071830 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0cc110 tx=0x7f22bc02b910 comp rx=0 tx=0).stop 2024-03-06T07:34:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.232+0000 7f22d47ae640 1 -- 172.21.15.77:0/3050517572 shutdown_connections 2024-03-06T07:34:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.232+0000 7f22d47ae640 1 --2- 172.21.15.77:0/3050517572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc15b430 0x7f22cc071830 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.232+0000 7f22d47ae640 1 --2- 172.21.15.77:0/3050517572 >> v2:172.21.15.158:3300/0 conn(0x7f22cc1515e0 0x7f22cc0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.232+0000 7f22d47ae640 1 -- 172.21.15.77:0/3050517572 >> 172.21.15.77:0/3050517572 conn(0x7f22cc0cf8a0 msgr2=0x7f22cc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.232+0000 7f22d47ae640 1 -- 172.21.15.77:0/3050517572 shutdown_connections 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.232+0000 7f22d47ae640 1 -- 172.21.15.77:0/3050517572 wait complete. 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.233+0000 7f22d47ae640 1 Processor -- start 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.233+0000 7f22d47ae640 1 -- start start 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d47ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc0cd7a0 0x7f22cc14ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d47ae640 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc1515e0 0x7f22cc14fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d47ae640 1 --2- >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22cc15b430 0x7f22cc15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d47ae640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc071ef0 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d47ae640 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc071d70 con 0x7f22cc1515e0 2024-03-06T07:34:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d47ae640 1 -- --> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc0b3f90 con 0x7f22cc15b430 2024-03-06T07:34:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc0cd7a0 0x7f22cc14ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc0cd7a0 0x7f22cc14ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56462/0 (socket says 172.21.15.77:56462) 2024-03-06T07:34:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 -- 172.21.15.77:0/2832613695 learned_addr learned my addr 172.21.15.77:0/2832613695 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-06T07:34:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 -- 172.21.15.77:0/2832613695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22cc15b430 msgr2=0x7f22cc15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22cc15b430 0x7f22cc15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 -- 172.21.15.77:0/2832613695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc1515e0 msgr2=0x7f22cc14fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc1515e0 0x7f22cc14fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 -- 172.21.15.77:0/2832613695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc14b190 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d254a640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc0cd7a0 0x7f22cc14ac50 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f22c800ba80 tx=0x7f22c800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22c2ffd640 1 -- 172.21.15.77:0/2832613695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8019070 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22c2ffd640 1 -- 172.21.15.77:0/2832613695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f22c800cc20 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22c2ffd640 1 -- 172.21.15.77:0/2832613695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8014440 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.234+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22cc150780 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.235+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22cc0c8e50 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.236+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22cc068840 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.236+0000 7f22c2ffd640 1 -- 172.21.15.77:0/2832613695 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22c801c070 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.236+0000 7f22c2ffd640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f22b006b4d0 0x7f22b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.236+0000 7f22c2ffd640 1 -- 172.21.15.77:0/2832613695 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f22c8089c60 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.236+0000 7f22d1d49640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f22b006b4d0 0x7f22b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:38.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.237+0000 7f22d1d49640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f22b006b4d0 0x7f22b006d990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f22bc036000 tx=0x7f22bc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:38.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.240+0000 7f22c2ffd640 1 -- 172.21.15.77:0/2832613695 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22c8058e10 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.451+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22cc04f330 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.452+0000 7f22c2ffd640 1 -- 172.21.15.77:0/2832613695 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+1329 (secure 0 0 0) 0x7f22c8058710 con 0x7f22cc0cd7a0 2024-03-06T07:34:38.454 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-06T07:34:38.454 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":109,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3818780028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3818780028},{"type":"v1","addr":"172.21.15.77:6837","nonce":3818780028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[]} 2024-03-06T07:34:38.454 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 109 2024-03-06T07:34:38.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f22b006b4d0 msgr2=0x7f22b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f22b006b4d0 0x7f22b006d990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f22bc036000 tx=0x7f22bc030000 comp rx=0 tx=0).stop 2024-03-06T07:34:38.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc0cd7a0 msgr2=0x7f22cc14ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:38.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc0cd7a0 0x7f22cc14ac50 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f22c800ba80 tx=0x7f22c800bf40 comp rx=0 tx=0).stop 2024-03-06T07:34:38.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 shutdown_connections 2024-03-06T07:34:38.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.158:3301/0,v1:172.21.15.158:6790/0] conn(0x7f22cc15b430 0x7f22cc15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f22cc1515e0 0x7f22cc14fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:6832/885960424,v1:172.21.15.77:6833/885960424] conn(0x7f22b006b4d0 0x7f22b006d990 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 --2- 172.21.15.77:0/2832613695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22cc0cd7a0 0x7f22cc14ac50 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:38.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 >> 172.21.15.77:0/2832613695 conn(0x7f22cc0cf8a0 msgr2=0x7f22cc06c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:38.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 shutdown_connections 2024-03-06T07:34:38.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-06T07:34:38.454+0000 7f22d47ae640 1 -- 172.21.15.77:0/2832613695 wait complete. 2024-03-06T07:34:38.467 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:34:38.467 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:34:38.467 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:34:38.467 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.476 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:34:38.477 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:38.477 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.526 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:38.526 ERROR:tasks.ceph.mds.b:tried to stop a non-running daemon 2024-03-06T07:34:38.529 ERROR:tasks.ceph.mds.d:tried to stop a non-running daemon 2024-03-06T07:34:38.533 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:34:38.534 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:34:38.534 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.598 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:34:38.599 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:34:38.599 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.669 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:34:38.669 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:34:38.669 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.735 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:34:38.736 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:34:38.736 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.807 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:34:38.807 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:34:38.807 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.836 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:34:38.837 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:34:38.837 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.911 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:34:38.911 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:34:38.911 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:38.981 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:34:38.981 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:34:38.982 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:39.050 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:34:39.050 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:34:39.050 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:34:39.050 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:39.076 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:34:39.077 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:34:39.077 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:39.117 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:34:39.117 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:34:39.117 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:34:39.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:39.129 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:34:39.129 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:34:39.129 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:39.169 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:34:39.169 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:34:39.169 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:39.221 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:34:39.221 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:34:39.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-06T07:34:39.259 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:34:39.259 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:34:39.478 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:34:39.479 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:34:39.602 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:34:39.602 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:34:39.660 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:34:39.660 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:34:39.719 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:34:39.719 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:34:39.889 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:34:39.890 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:34:39.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:34:39.987 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:34:40.045 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:34:40.046 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:34:40.102 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:34:40.103 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/data/mon.a.tgz 2024-03-06T07:34:40.104 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-03-06T07:34:40.119 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.EfyCuc620P 2024-03-06T07:34:40.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.EfyCuc620P 2024-03-06T07:34:40.265 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.EfyCuc620P 2024-03-06T07:34:40.350 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.EfyCuc620P is 373KB 2024-03-06T07:34:40.446 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.EfyCuc620P 2024-03-06T07:34:40.460 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/data/mon.b.tgz 2024-03-06T07:34:40.461 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2024-03-06T07:34:40.478 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.tBUw9AqOjr 2024-03-06T07:34:40.478 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.tBUw9AqOjr 2024-03-06T07:34:40.649 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.tBUw9AqOjr 2024-03-06T07:34:40.734 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.tBUw9AqOjr is 374KB 2024-03-06T07:34:40.830 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.tBUw9AqOjr 2024-03-06T07:34:40.844 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/data/mon.c.tgz 2024-03-06T07:34:40.844 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2024-03-06T07:34:40.898 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.80HNPAIU47 2024-03-06T07:34:40.899 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.80HNPAIU47 2024-03-06T07:34:41.029 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.80HNPAIU47 2024-03-06T07:34:41.114 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.80HNPAIU47 is 374KB 2024-03-06T07:34:41.209 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.80HNPAIU47 2024-03-06T07:34:41.223 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:34:41.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:34:41.225 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:34:41.307 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:34:41.344 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:34:41.344 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:34:41.345 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/remote/smithi077/crash 2024-03-06T07:34:41.346 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:34:41.375 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/remote/smithi158/crash 2024-03-06T07:34:41.376 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:34:41.410 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:34:41.410 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:41.420 DEBUG:teuthology.orchestra.run.smithi158:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:41.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:41.531 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:41.558 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:34:41.558 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:34:41.574 DEBUG:teuthology.orchestra.run.smithi158:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:34:41.598 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71006.log 2024-03-06T07:34:41.598 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68461.log 2024-03-06T07:34:41.599 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63741.log 2024-03-06T07:34:41.599 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.68461.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.58758.log 2024-03-06T07:34:41.599 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71006.log.gz 2024-03-06T07:34:41.642 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:34:41.642 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43525.log 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37989.log 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.1.43525.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.37989.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37989.log.gz 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph.audit.log: 89.1% -- replaced with /var/log/ceph/ceph-client.1.43525.log.gz 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43977.log 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-mds.b.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37829.log 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.3.43977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37949.log 2024-03-06T07:34:41.643 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.37829.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.37949.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37829.log.gz 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr: 90.3%gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr: -- replaced with /var/log/ceph/ceph-client.3.43977.log.gz 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37949.log.gz 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37749.log 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38029.log 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.37749.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37749.log.gz 2024-03-06T07:34:41.644 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:34:41.645 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.38029.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38029.log.gz 2024-03-06T07:34:41.645 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:34:41.645 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:34:41.645 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:34:41.645 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-mon.b.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:34:41.645 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37789.log 2024-03-06T07:34:41.645 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-mds.d.log: 94.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:34:41.663 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:34:41.663 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.37789.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37789.log.gz 2024-03-06T07:34:41.672 INFO:teuthology.orchestra.run.smithi158.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:34:41.672 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37869.log 2024-03-06T07:34:41.672 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37909.log 2024-03-06T07:34:41.672 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.37869.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37869.log.gz 2024-03-06T07:34:41.673 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/ceph-client.admin.37909.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37909.log.gz 2024-03-06T07:34:41.904 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79880.log 2024-03-06T07:34:41.905 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54444.log 2024-03-06T07:34:41.905 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63741.log.gz 2024-03-06T07:34:41.905 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58758.log.gz 2024-03-06T07:34:41.905 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52129.log 2024-03-06T07:34:41.905 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69522.log 2024-03-06T07:34:41.905 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79880.log.gz 2024-03-06T07:34:41.905 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52129.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54444.log.gz 2024-03-06T07:34:41.906 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77454.log 2024-03-06T07:34:41.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66248.log 2024-03-06T07:34:41.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77454.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.52129.log.gz 2024-03-06T07:34:41.906 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69522.log.gz 2024-03-06T07:34:41.906 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85685.log 2024-03-06T07:34:41.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66248.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77454.log.gz 2024-03-06T07:34:41.906 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67884.log 2024-03-06T07:34:41.906 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66248.log.gz 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77643.log 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74892.log 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85685.log.gz 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77643.log: gzip 89.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.67884.log.gz --verbose 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77643.log.gz 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80570.log 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74892.log.gz 2024-03-06T07:34:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75672.log 2024-03-06T07:34:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80528.log 2024-03-06T07:34:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75672.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80570.log.gz 2024-03-06T07:34:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89611.log 2024-03-06T07:34:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56571.log 2024-03-06T07:34:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80528.log: /var/log/ceph/ceph-client.admin.89611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75442.log 2024-03-06T07:34:41.908 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80528.log.gz 2024-03-06T07:34:41.908 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.75672.log.gz 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89611.log.gz 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74616.log 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63657.log 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56571.log.gz 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75442.log.gz 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51140.log 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68136.log 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63657.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74616.log.gz 2024-03-06T07:34:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56776.log 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63657.log.gz 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51140.log.gz 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71174.log 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68136.log: /var/log/ceph/ceph-client.admin.56776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60618.log 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71174.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56776.log.gz 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63843.log 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71174.log.gz 91.0% 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68136.log.gz 2024-03-06T07:34:41.910 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51477.log 2024-03-06T07:34:41.911 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60618.log: /var/log/ceph/ceph-client.admin.63843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86641.log 2024-03-06T07:34:41.911 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60618.log.gz 2024-03-06T07:34:41.911 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53110.log 2024-03-06T07:34:41.911 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63843.log.gz 2024-03-06T07:34:41.911 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51477.log.gz 2024-03-06T07:34:41.911 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76462.log 2024-03-06T07:34:41.911 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86641.log.gz 2024-03-06T07:34:41.911 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74830.log 2024-03-06T07:34:41.912 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53110.log: /var/log/ceph/ceph-client.admin.76462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90889.log 2024-03-06T07:34:41.912 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53110.log.gz 2024-03-06T07:34:41.912 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76462.log.gz 2024-03-06T07:34:41.912 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76504.log 2024-03-06T07:34:41.912 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.90889.log: 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.74830.log.gz 2024-03-06T07:34:41.912 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.67405.log 2024-03-06T07:34:41.912 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76504.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90889.log.gz 2024-03-06T07:34:41.912 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63066.log 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76504.log.gz 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73138.log 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67405.log: /var/log/ceph/ceph-client.admin.63066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61620.log 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67405.log.gz 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73138.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63066.log.gz 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57848.log 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45156.log 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73138.log.gz 2024-03-06T07:34:41.913 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61620.log: /var/log/ceph/ceph-client.admin.57848.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61620.log.gz 2024-03-06T07:34:41.914 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62934.log 2024-03-06T07:34:41.914 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55644.log 2024-03-06T07:34:41.914 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45156.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57848.log.gz 2024-03-06T07:34:41.914 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62934.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.45156.log.gz 2024-03-06T07:34:41.914 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.86599.log 2024-03-06T07:34:41.914 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62934.log.gz 2024-03-06T07:34:41.914 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64180.log 2024-03-06T07:34:41.914 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.55644.log: /var/log/ceph/ceph-client.admin.86599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49329.log 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86599.log.gz 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82864.log 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49245.log 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49329.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64180.log.gz 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88837.log 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49329.log.gz 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49245.log: gzip -5 --verbose 88.7% -- /var/log/ceph/ceph-client.admin.92470.log -- replaced with /var/log/ceph/ceph-client.admin.82864.log.gz 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:41.915 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49245.log.gz 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85909.log 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49529.log 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92470.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88837.log.gz 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46323.log 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92470.log.gz 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.49529.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.85867.log 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85909.log.gz 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46323.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49529.log.gz 2024-03-06T07:34:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69667.log 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70745.log 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85867.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.46323.log.gz 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75291.log 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85867.log.gz 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70745.log: gzip 89.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.69667.log.gz --verbose 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.92866.log 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70745.log.gz 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86305.log 2024-03-06T07:34:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75291.log: /var/log/ceph/ceph-client.admin.92866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88483.log 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75291.log.gz 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86305.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92866.log.gz 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53347.log 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88483.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.86305.log.gz 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87163.log 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88483.log.gz 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71970.log 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53347.log.gz 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63108.log 2024-03-06T07:34:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80276.log 2024-03-06T07:34:41.919 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87163.log.gz 2024-03-06T07:34:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92638.log 2024-03-06T07:34:41.919 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71970.log.gz 2024-03-06T07:34:41.919 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63108.log.gz 2024-03-06T07:34:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53389.log 2024-03-06T07:34:41.919 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80276.log.gz 2024-03-06T07:34:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61918.log 2024-03-06T07:34:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92638.log: /var/log/ceph/ceph-client.admin.53389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50181.log 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9%/var/log/ceph/ceph-client.admin.61918.log: -- replaced with /var/log/ceph/ceph-client.admin.92638.log.gz 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52868.log 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53389.log.gz 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50181.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61918.log.gz 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92302.log 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50181.log.gz 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86683.log 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52868.log.gz -5 2024-03-06T07:34:41.920 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53935.log 2024-03-06T07:34:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92302.log: /var/log/ceph/ceph-client.admin.86683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68010.log 2024-03-06T07:34:41.921 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92302.log.gz 2024-03-06T07:34:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92680.log 2024-03-06T07:34:41.921 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86683.log.gz 2024-03-06T07:34:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68010.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53935.log.gz 2024-03-06T07:34:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49893.log 2024-03-06T07:34:41.922 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68010.log.gz 2024-03-06T07:34:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65927.log 2024-03-06T07:34:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48773.log 2024-03-06T07:34:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49893.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92680.log.gz 2024-03-06T07:34:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56319.log 2024-03-06T07:34:41.922 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49893.log.gz 2024-03-06T07:34:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48773.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65927.log.gz 2024-03-06T07:34:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52195.log 2024-03-06T07:34:41.923 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48773.log.gz 2024-03-06T07:34:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91811.log 2024-03-06T07:34:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58124.log 2024-03-06T07:34:41.923 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56319.log.gz 2024-03-06T07:34:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52195.log: /var/log/ceph/ceph-client.admin.91811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55277.log 2024-03-06T07:34:41.923 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52195.log.gz 2024-03-06T07:34:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.58124.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65071.log 2024-03-06T07:34:41.923 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91811.log.gz 2024-03-06T07:34:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55277.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58124.log.gz 2024-03-06T07:34:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58540.log 2024-03-06T07:34:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92512.log 2024-03-06T07:34:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65071.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55277.log.gz 2024-03-06T07:34:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58540.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65071.log.gz 2024-03-06T07:34:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75973.log 2024-03-06T07:34:41.924 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58540.log.gz 2024-03-06T07:34:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55235.log 2024-03-06T07:34:41.925 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92512.log: /var/log/ceph/ceph-client.admin.75973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46841.log 2024-03-06T07:34:41.925 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92512.log.gz 2024-03-06T07:34:41.925 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55235.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75973.log.gz 2024-03-06T07:34:41.925 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69044.log 2024-03-06T07:34:41.925 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46197.log 2024-03-06T07:34:41.925 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.46841.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55235.log.gz 2024-03-06T07:34:41.925 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81518.log 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69044.log.gz 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87853.log 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.0.46841.log.gz 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81518.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46197.log.gz 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51519.log 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89569.log 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81518.log.gz 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87853.log.gz 2024-03-06T07:34:41.926 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52826.log 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51519.log.gz 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77854.log 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89569.log: /var/log/ceph/ceph-client.admin.52826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67926.log 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.89569.log.gz 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77854.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52826.log.gz 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83568.log 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67926.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77854.log.gz 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71048.log 2024-03-06T07:34:41.927 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.67926.log.gz 2024-03-06T07:34:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48831.log 2024-03-06T07:34:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71048.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83568.log.gz 2024-03-06T07:34:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64605.log 2024-03-06T07:34:41.928 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71048.log.gz 2024-03-06T07:34:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54983.log 2024-03-06T07:34:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48831.log: /var/log/ceph/ceph-client.admin.64605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80987.log 2024-03-06T07:34:41.928 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48831.log.gz 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54983.log: gzip 88.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.64605.log.gz -- 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/ceph-client.admin.45159.log 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54983.log.gz 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92948.log 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80987.log.gz 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87247.log 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45159.log: /var/log/ceph/ceph-client.admin.92948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51733.log 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45159.log.gz 2024-03-06T07:34:41.929 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87247.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92948.log.gz 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48072.log 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51733.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87247.log.gz 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71575.log 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51733.log.gz 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91015.log 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48072.log: /var/log/ceph/ceph-client.admin.71575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61129.log 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48072.log.gz 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91015.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71575.log.gz 2024-03-06T07:34:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90427.log 2024-03-06T07:34:41.931 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91015.log.gz 2024-03-06T07:34:41.931 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65155.log 2024-03-06T07:34:41.931 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61129.log.gz 2024-03-06T07:34:41.931 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87037.log 2024-03-06T07:34:41.931 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90427.log: /var/log/ceph/ceph-client.admin.65155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63234.log 2024-03-06T07:34:41.931 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90427.log.gz 2024-03-06T07:34:41.931 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87037.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65155.log.gz 2024-03-06T07:34:41.932 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.75504.log 2024-03-06T07:34:41.932 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63234.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87037.log.gz 2024-03-06T07:34:41.932 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68801.log 2024-03-06T07:34:41.932 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75504.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63234.log.gz 2024-03-06T07:34:41.932 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92386.log 2024-03-06T07:34:41.932 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68801.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75504.log.gz 2024-03-06T07:34:41.932 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73384.log 2024-03-06T07:34:41.932 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68801.log.gz 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50013.log 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67363.log 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92386.log.gz 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50013.log: gzip 89.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.73384.log.gz --verbose 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.67842.log 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50013.log.gz 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90721.log 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67363.log.gz 2024-03-06T07:34:41.933 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77770.log 2024-03-06T07:34:41.934 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67842.log: /var/log/ceph/ceph-client.admin.90721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72506.log 2024-03-06T07:34:41.934 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67842.log.gz 2024-03-06T07:34:41.934 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90721.log.gz 2024-03-06T07:34:41.934 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57932.log 2024-03-06T07:34:41.934 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68970.log 2024-03-06T07:34:41.934 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.72506.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77770.log.gz 2024-03-06T07:34:41.934 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60347.log 2024-03-06T07:34:41.934 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68970.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57932.log.gz 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88021.log 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60347.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68970.log.gz 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31810.log 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88021.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60347.log.gz 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77938.log 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88021.log.gz 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose --/var/log/ceph/ceph.tmp-client.admin.31810.log: /var/log/ceph/ceph-client.admin.85417.log 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31810.log.gz 2024-03-06T07:34:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50601.log 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77938.log.gz 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68094.log 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50601.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85417.log.gz 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50601.log.gz 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92260.log 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44820.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68094.log.gz 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76775.log 2024-03-06T07:34:41.936 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92260.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50769.log 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76775.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92260.log.gz 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67758.log 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76775.log.gz 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50769.log: gzip -5 --verbose -- 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50769.log.gz 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/ceph-client.admin.72246.log 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76252.log 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67758.log.gz 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48198.log 2024-03-06T07:34:41.937 INFO:teuthology.orchestra.run.smithi077.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.0.72506.log.gz 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72246.log.gz 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56734.log 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76252.log: /var/log/ceph/ceph-client.admin.48198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92596.log 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76252.log.gz 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48198.log.gz 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81602.log 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71617.log 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56734.log.gz 2024-03-06T07:34:41.938 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64777.log 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92596.log.gz 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71617.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81602.log.gz 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56860.log 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64777.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71617.log.gz 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69994.log 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.64777.log.gz 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44559.log 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56860.log: /var/log/ceph/ceph-client.admin.69994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81266.log 2024-03-06T07:34:41.939 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56860.log.gz 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44559.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69994.log.gz 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84232.log 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79796.log 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81266.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44559.log.gz 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.0.55644.log.gz 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84232.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81266.log.gz 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80714.log 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 89.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.84232.log.gz --verbose 2024-03-06T07:34:41.940 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.81308.log 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79796.log: /var/log/ceph/ceph-client.admin.80714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91727.log 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79796.log.gz 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89215.log 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81308.log.gz 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91727.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.80714.log.gz 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88525.log 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91727.log.gz 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59859.log 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70964.log 2024-03-06T07:34:41.941 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89215.log.gz 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59859.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88525.log.gz 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50643.log 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70964.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59859.log.gz 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63531.log 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0%/var/log/ceph/ceph-client.admin.50643.log: -- replaced with /var/log/ceph/ceph-client.admin.70964.log.gz 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73745.log 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50643.log.gz 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66542.log 2024-03-06T07:34:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63531.log: /var/log/ceph/ceph-client.admin.73745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62086.log 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63531.log.gz 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66542.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79922.log 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73745.log.gz 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66542.log.gz 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79964.log 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79922.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62086.log.gz 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77980.log 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79922.log.gz 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47988.log 2024-03-06T07:34:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77980.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79964.log.gz 2024-03-06T07:34:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75889.log 2024-03-06T07:34:41.944 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77980.log.gz 2024-03-06T07:34:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89485.log 2024-03-06T07:34:41.944 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47988.log.gz 2024-03-06T07:34:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67586.log 2024-03-06T07:34:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75889.log: /var/log/ceph/ceph-client.admin.89485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61211.log 2024-03-06T07:34:41.944 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75889.log.gz 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89485.log.gz 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60786.log 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59002.log 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61211.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.67586.log.gz 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68802.log 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61211.log.gz 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60786.log.gz 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45161.log 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59002.log: /var/log/ceph/ceph-client.admin.68802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49808.log 2024-03-06T07:34:41.945 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59002.log.gz 2024-03-06T07:34:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45161.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68802.log.gz 2024-03-06T07:34:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90847.log 2024-03-06T07:34:41.946 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45161.log.gz 2024-03-06T07:34:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70483.log 2024-03-06T07:34:41.946 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49808.log.gz 2024-03-06T07:34:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61750.log 2024-03-06T07:34:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90847.log: /var/log/ceph/ceph-client.admin.70483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58016.log 2024-03-06T07:34:41.946 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90847.log.gz 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61750.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70483.log.gz 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80318.log 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61750.log.gz 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71756.log 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58016.log: /var/log/ceph/ceph-client.admin.80318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88105.log 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58016.log.gz 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80318.log.gz 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87205.log 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77368.log 2024-03-06T07:34:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88105.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.71756.log.gz 2024-03-06T07:34:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87205.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88105.log.gz 2024-03-06T07:34:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62453.log 2024-03-06T07:34:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89737.log 2024-03-06T07:34:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77368.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87205.log.gz 2024-03-06T07:34:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.62453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49655.log 2024-03-06T07:34:41.948 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77368.log.gz 2024-03-06T07:34:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52784.log 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89737.log: /var/log/ceph/ceph-client.admin.49655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86725.log 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89737.log.gz 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52784.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49655.log.gz 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86953.log 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86725.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52784.log.gz 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89821.log 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86953.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86725.log.gz 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85459.log 2024-03-06T07:34:41.949 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86953.log.gz 2024-03-06T07:34:41.950 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78768.log 2024-03-06T07:34:41.950 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85459.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89821.log.gz 2024-03-06T07:34:41.950 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45160.log 2024-03-06T07:34:41.950 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78768.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85459.log.gz 2024-03-06T07:34:41.950 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54546.log 2024-03-06T07:34:41.950 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78768.log.gz 2024-03-06T07:34:41.950 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5/var/log/ceph/ceph-client.admin.45160.log: --verbose -- /var/log/ceph/ceph-client.admin.51605.log 2024-03-06T07:34:41.950 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54546.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45160.log.gz 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79502.log 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51605.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54546.log.gz 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54941.log 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51605.log.gz 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89527.log 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62002.log 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79502.log.gz 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54941.log.gz 2024-03-06T07:34:41.951 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92050.log 2024-03-06T07:34:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60095.log 2024-03-06T07:34:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62002.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89527.log.gz 2024-03-06T07:34:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55067.log 2024-03-06T07:34:41.952 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62002.log.gz 2024-03-06T07:34:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60095.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92050.log.gz 2024-03-06T07:34:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84598.log 2024-03-06T07:34:41.952 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60095.log.gz 2024-03-06T07:34:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46029.log 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55067.log: /var/log/ceph/ceph-client.admin.84598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56445.log 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55067.log.gz 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84598.log.gz 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76691.log 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44517.log 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46029.log.gz 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71269.log 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56445.log.gz 2024-03-06T07:34:41.953 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76691.log.gz 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76294.log 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64562.log 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71269.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44517.log.gz 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48030.log 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71269.log.gz 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64562.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87583.log 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76294.log.gz 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45861.log 2024-03-06T07:34:41.954 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64562.log.gz 2024-03-06T07:34:41.955 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58860.log 2024-03-06T07:34:41.955 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87583.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48030.log.gz 2024-03-06T07:34:41.955 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44265.log 2024-03-06T07:34:41.955 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.87583.log.gz 2024-03-06T07:34:41.955 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58860.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57539.log 2024-03-06T07:34:41.955 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45861.log.gz 2024-03-06T07:34:41.955 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44265.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58860.log.gz 2024-03-06T07:34:41.956 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65113.log 2024-03-06T07:34:41.956 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61467.log 2024-03-06T07:34:41.956 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57539.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44265.log.gz 2024-03-06T07:34:41.956 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64735.log 2024-03-06T07:34:41.956 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57539.log.gz 2024-03-06T07:34:41.956 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61467.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65113.log.gz 2024-03-06T07:34:41.956 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61560.log 2024-03-06T07:34:41.956 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2%/var/log/ceph/ceph-client.admin.64735.log: -- replaced with /var/log/ceph/ceph-client.admin.61467.log.gz 2024-03-06T07:34:41.957 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76168.log 2024-03-06T07:34:41.957 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61560.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.64735.log.gz -5 2024-03-06T07:34:41.957 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84540.log 2024-03-06T07:34:41.957 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76168.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61560.log.gz 2024-03-06T07:34:41.957 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50727.log 2024-03-06T07:34:41.957 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76168.log.gz 2024-03-06T07:34:41.957 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65885.log 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50265.log 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84540.log.gz 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82259.log 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50727.log.gz 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65885.log.gz 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76901.log 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63928.log 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7%/var/log/ceph/ceph-client.admin.82259.log: -- replaced with /var/log/ceph/ceph-client.admin.50265.log.gz 2024-03-06T07:34:41.958 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79196.log 2024-03-06T07:34:41.959 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82259.log.gz 2024-03-06T07:34:41.959 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71928.log 2024-03-06T07:34:41.959 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76901.log.gz 2024-03-06T07:34:41.959 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63928.log.gz 2024-03-06T07:34:41.959 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77559.log 2024-03-06T07:34:41.959 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89863.log 2024-03-06T07:34:41.959 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71928.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79196.log.gz 2024-03-06T07:34:41.959 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52669.log 2024-03-06T07:34:41.960 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71928.log.gz 2024-03-06T07:34:41.960 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89863.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77559.log.gz 2024-03-06T07:34:41.960 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74688.log 2024-03-06T07:34:41.960 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52669.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89863.log.gz 2024-03-06T07:34:41.960 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87541.log 2024-03-06T07:34:41.960 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44037.log 2024-03-06T07:34:41.960 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52669.log.gz 2024-03-06T07:34:41.960 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74688.log.gz 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48915.log 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72824.log 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44037.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87541.log.gz 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78852.log 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44037.log.gz 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48915.log.gz 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43814.log 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72824.log: /var/log/ceph/ceph-client.admin.78852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58383.log 2024-03-06T07:34:41.961 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78852.log.gz 2024-03-06T07:34:41.962 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43814.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.72824.log.gz 2024-03-06T07:34:41.962 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52087.log 2024-03-06T07:34:41.962 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.0.58383.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.90595.log 2024-03-06T07:34:41.962 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43814.log.gz 2024-03-06T07:34:41.962 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57453.log 2024-03-06T07:34:41.962 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52087.log.gz 2024-03-06T07:34:41.962 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90595.log: 92.6% -- replaced with /var/log/ceph/ceph-client.0.62453.log.gz 2024-03-06T07:34:41.962 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54240.log 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90595.log.gz 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44823.log 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57453.log: /var/log/ceph/ceph-client.admin.54240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48873.log 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57453.log.gz 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44823.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54240.log.gz -5 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.82217.log 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48873.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44823.log.gz 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81854.log 2024-03-06T07:34:41.963 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48873.log.gz 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84372.log 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82217.log: /var/log/ceph/ceph-client.admin.81854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48957.log 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82217.log.gz 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84372.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81854.log.gz 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79838.log 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84372.log.gz 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80903.log 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78214.log 2024-03-06T07:34:41.964 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6%/var/log/ceph/ceph-client.admin.79838.log: -- replaced with /var/log/ceph/ceph-client.admin.48957.log.gz 2024-03-06T07:34:41.965 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80903.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79838.log.gz 2024-03-06T07:34:41.965 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44391.log 2024-03-06T07:34:41.965 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78214.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80903.log.gz 2024-03-06T07:34:41.965 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78214.log.gz 2024-03-06T07:34:41.965 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64012.log 2024-03-06T07:34:41.965 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75248.log 2024-03-06T07:34:41.965 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64012.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44391.log.gz 2024-03-06T07:34:41.965 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89653.log 2024-03-06T07:34:41.966 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64012.log.gz 2024-03-06T07:34:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67968.log 2024-03-06T07:34:41.966 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75248.log.gz 2024-03-06T07:34:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83439.log 2024-03-06T07:34:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53599.log 2024-03-06T07:34:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67968.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89653.log.gz 2024-03-06T07:34:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83439.log: gzip -5 87.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.67968.log.gz 2024-03-06T07:34:41.966 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/ceph-client.admin.69086.log 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83439.log.gz 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45488.log 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53599.log.gz 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47610.log 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69086.log: /var/log/ceph/ceph-client.admin.45488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76733.log 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69086.log.gz 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47610.log: gzip -5 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45488.log.gz 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.88315.log 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76733.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47610.log.gz 2024-03-06T07:34:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79586.log 2024-03-06T07:34:41.968 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76733.log.gz 2024-03-06T07:34:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87751.log 2024-03-06T07:34:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88315.log: /var/log/ceph/ceph-client.admin.79586.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88315.log.gz 2024-03-06T07:34:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45777.log 2024-03-06T07:34:41.968 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79586.log.gz 2024-03-06T07:34:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65263.log 2024-03-06T07:34:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87751.log: /var/log/ceph/ceph-client.admin.45777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75628.log 2024-03-06T07:34:41.968 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87751.log.gz 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65263.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45777.log.gz 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61834.log 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65263.log.gz 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92092.log 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44223.log 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75628.log.gz 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61834.log: /var/log/ceph/ceph-client.admin.92092.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61834.log.gz 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.65522.log 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92092.log.gz 2024-03-06T07:34:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67237.log 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.65522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85993.log 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44223.log.gz 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64647.log 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85993.log: gzip -5 89.0% --verbose -- /var/log/ceph/ceph-client.admin.59226.log 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67237.log.gz 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64647.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85993.log.gz 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78473.log 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61876.log 2024-03-06T07:34:41.970 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64647.log.gz 2024-03-06T07:34:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59226.log: /var/log/ceph/ceph-client.0.78473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56902.log 2024-03-06T07:34:41.971 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.59226.log.gz 2024-03-06T07:34:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54857.log 2024-03-06T07:34:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61876.log: /var/log/ceph/ceph-client.admin.56902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86515.log 2024-03-06T07:34:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54857.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61876.log.gz 2024-03-06T07:34:41.971 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56902.log.gz 2024-03-06T07:34:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73791.log 2024-03-06T07:34:41.971 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54857.log.gz 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60702.log 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86515.log: /var/log/ceph/ceph-client.admin.73791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74511.log 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60702.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.86515.log.gz 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73791.log.gz 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82612.log 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60702.log.gz 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56403.log 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74511.log: /var/log/ceph/ceph-client.admin.82612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73506.log 2024-03-06T07:34:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56403.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74511.log.gz 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82612.log.gz 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86137.log 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56403.log.gz 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77728.log 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73506.log: /var/log/ceph/ceph-client.admin.86137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60053.log 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77728.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73506.log.gz 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86137.log.gz 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.0.78473.log.gz 2024-03-06T07:34:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92428.log 2024-03-06T07:34:41.974 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77728.log.gz 2024-03-06T07:34:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73505.log 2024-03-06T07:34:41.974 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2%/var/log/ceph/ceph-client.admin.92428.log: -- replaced with /var/log/ceph/ceph-client.admin.60053.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88399.log 2024-03-06T07:34:41.974 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.73505.log: 88.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.46113.log 2024-03-06T07:34:41.974 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92428.log.gz 2024-03-06T07:34:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88399.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67674.log 2024-03-06T07:34:41.974 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73505.log.gz 2024-03-06T07:34:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46113.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88399.log.gz 2024-03-06T07:34:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47458.log 2024-03-06T07:34:41.975 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46113.log.gz 2024-03-06T07:34:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78992.log 2024-03-06T07:34:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67674.log: gzip/var/log/ceph/ceph-client.admin.47458.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.91246.log 2024-03-06T07:34:41.975 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3%/var/log/ceph/ceph-client.admin.78992.log: -- replaced with /var/log/ceph/ceph-client.admin.67674.log.gz 2024-03-06T07:34:41.975 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47458.log.gz 2024-03-06T07:34:41.975 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60576.log 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78992.log.gz 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91246.log: /var/log/ceph/ceph-client.admin.60576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66728.log 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91246.log.gz 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60576.log.gz 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.52454.log 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.66728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87079.log 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66728.log.gz 2024-03-06T07:34:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.52454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47904.log 2024-03-06T07:34:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75186.log 2024-03-06T07:34:41.977 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87079.log.gz 2024-03-06T07:34:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47904.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47904.log.gz 2024-03-06T07:34:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50097.log 2024-03-06T07:34:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71491.log 2024-03-06T07:34:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75186.log: /var/log/ceph/ceph-client.admin.50097.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75186.log.gz 2024-03-06T07:34:41.977 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50097.log.gz 2024-03-06T07:34:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57028.log 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.0.65522.log.gz 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61518.log 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71491.log.gz 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64054.log 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57028.log.gz 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45151.log 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.64054.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.68674.log 2024-03-06T07:34:41.978 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61518.log.gz 2024-03-06T07:34:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53683.log 2024-03-06T07:34:41.979 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64054.log.gz 2024-03-06T07:34:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2024-03-06T07:34:41.979 INFO:teuthology.orchestra.run.smithi077.stderr: -5/var/log/ceph/ceph-client.admin.68674.log: --verbose -- /var/log/ceph/ceph-client.admin.70162.log 2024-03-06T07:34:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53683.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68674.log.gz 2024-03-06T07:34:41.979 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.81224.log 2024-03-06T07:34:41.979 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53683.log.gz 2024-03-06T07:34:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82387.log 2024-03-06T07:34:41.980 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1%/var/log/ceph/ceph-client.admin.81224.log: -- replaced with /var/log/ceph/ceph-client.admin.70162.log.gz 2024-03-06T07:34:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74455.log 2024-03-06T07:34:41.980 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81224.log.gz/var/log/ceph/ceph-client.admin.82387.log: 2024-03-06T07:34:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89173.log 2024-03-06T07:34:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70315.log 2024-03-06T07:34:41.980 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82387.log.gz 2024-03-06T07:34:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71886.log 2024-03-06T07:34:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70315.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89173.log.gz 2024-03-06T07:34:41.981 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92344.log 2024-03-06T07:34:41.981 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70315.log.gz 2024-03-06T07:34:41.981 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67279.log 2024-03-06T07:34:41.981 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74455.log.gz 2024-03-06T07:34:41.981 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71886.log: /var/log/ceph/ceph-client.admin.92344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61337.log 2024-03-06T07:34:41.981 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6%/var/log/ceph/ceph-client.admin.67279.log: -- replaced with /var/log/ceph/ceph-client.admin.71886.log.gz 2024-03-06T07:34:41.981 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92344.log.gz 2024-03-06T07:34:41.981 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47694.log 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2%/var/log/ceph/ceph-client.admin.61337.log: -- replaced with /var/log/ceph/ceph-client.admin.67279.log.gz 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63615.log 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47694.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82524.log 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61337.log.gz 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63615.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47694.log.gz 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63573.log 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63615.log.gz 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82524.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.88879.log 2024-03-06T07:34:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63573.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82524.log.gz 2024-03-06T07:34:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60473.log 2024-03-06T07:34:41.983 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63573.log.gz 2024-03-06T07:34:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.88879.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70796.log 2024-03-06T07:34:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60473.log: gzip 88.2% -5 --verbose -- /var/log/ceph/ceph-client.admin.87895.log 2024-03-06T07:34:41.983 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88879.log.gz 2024-03-06T07:34:41.983 INFO:teuthology.orchestra.run.smithi077.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.58383.log.gz 2024-03-06T07:34:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70796.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60473.log.gz 2024-03-06T07:34:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53194.log 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87895.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70796.log.gz 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86473.log 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53194.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87895.log.gz 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88669.log 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53194.log.gz 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81995.log 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86473.log: /var/log/ceph/ceph-client.admin.88669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85333.log 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86473.log.gz 2024-03-06T07:34:41.984 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66290.log 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88669.log.gz 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85333.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81995.log.gz 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89005.log 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1%/var/log/ceph/ceph-client.admin.66290.log: -- replaced with /var/log/ceph/ceph-client.admin.85333.log.gz 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80174.log 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66290.log.gz 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89005.log.gz 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87979.log 2024-03-06T07:34:41.985 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92906.log 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80174.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87979.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80174.log.gzgzip 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69584.log 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58674.log 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92906.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87979.log.gz 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43872.log 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92906.log.gz 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69584.log.gz 2024-03-06T07:34:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87331.log 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91456.log 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58674.log.gz 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87331.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43872.log.gz 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55109.log 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91456.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87331.log.gz 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57890.log 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89047.log 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55109.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91456.log.gz 2024-03-06T07:34:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57890.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.81686.log 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55109.log.gz 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85143.log 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57890.log.gz 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81686.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89047.log.gz 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88441.log 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51182.log 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.85143.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81686.log.gz 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64691.log 2024-03-06T07:34:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51182.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88441.log.gz 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68202.log 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51182.log.gz 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.64691.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.79712.log 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68202.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64691.log.gz 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.85143.log.gz 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70357.log 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68202.log.gz 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50559.log 2024-03-06T07:34:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79712.log: /var/log/ceph/ceph-client.admin.70357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66584.log 2024-03-06T07:34:41.990 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79712.log.gz 2024-03-06T07:34:41.990 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69336.log 2024-03-06T07:34:41.990 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70357.log.gz 2024-03-06T07:34:41.990 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50559.log.gz 2024-03-06T07:34:41.990 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89443.log 2024-03-06T07:34:41.990 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66584.log.gz 2024-03-06T07:34:41.990 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71661.log 2024-03-06T07:34:41.990 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69336.log: /var/log/ceph/ceph-client.admin.89443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53236.log 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69336.log.gz 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89443.log.gz 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66416.log 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 88.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.71661.log.gz -- 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/ceph-client.admin.61049.log 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53236.log: /var/log/ceph/ceph-client.admin.66416.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53236.log.gz 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88231.log 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68052.log 2024-03-06T07:34:41.991 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61049.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66416.log.gz 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45693.log 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61049.log.gz 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88231.log.gz 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61960.log 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60531.log 88.4% 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68052.log.gz 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45693.log.gz 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81896.log 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61960.log.gz 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70399.log 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60531.log.gz 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66854.log 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81896.log: /var/log/ceph/ceph-client.admin.70399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81896.log.gz 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66854.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70399.log.gz 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62128.log 2024-03-06T07:34:41.993 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48114.log 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66854.log.gz 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44818.log: /var/log/ceph/ceph-client.admin.62128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72012.log 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60744.log 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48114.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.62128.log.gz 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71349.log 2024-03-06T07:34:41.994 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48114.log.gz 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.60744.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72012.log.gz 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52045.log 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71349.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60744.log.gz 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59817.log 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52045.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71349.log.gz 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90073.log 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52045.log.gz 2024-03-06T07:34:41.995 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61295.log 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90073.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59817.log.gz 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82738.log 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90073.log.gz 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87289.log 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87121.log 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61295.log.gz 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90343.log 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82738.log.gz 2024-03-06T07:34:41.996 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87121.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87289.log.gz 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69177.log 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5%/var/log/ceph/ceph-client.admin.90343.log: -- replaced with /var/log/ceph/ceph-client.admin.87121.log.gz 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86809.log 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90343.log.gz 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66164.log 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69177.log: /var/log/ceph/ceph-client.admin.86809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54632.log 2024-03-06T07:34:41.997 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66164.log: gzip -5 --verbose -- 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86809.log.gz 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/ceph-client.admin.84640.log 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66164.log.gz 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54632.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.69177.log.gz 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44307.log 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54632.log.gz 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43562.log 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44307.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84640.log.gz 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73899.log 2024-03-06T07:34:41.998 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44307.log.gz 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88711.log 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43562.log: /var/log/ceph/ceph-client.admin.73899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53725.log 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43562.log.gz 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88711.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73899.log.gz 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49759.log 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53725.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88711.log.gz 2024-03-06T07:34:41.999 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86431.log 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53725.log.gz 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48689.log 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49759.log: /var/log/ceph/ceph-client.admin.86431.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49759.log.gz 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91372.log 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86431.log.gz 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66668.log 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48689.log: /var/log/ceph/ceph-client.admin.91372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81560.log 2024-03-06T07:34:42.000 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48689.log.gz 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91372.log.gz 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88963.log 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47736.log 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66668.log.gz 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76336.log 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81560.log.gz 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66206.log 2024-03-06T07:34:42.001 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88963.log.gz 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76336.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47736.log.gz 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51393.log 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76336.log.gz 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72912.log 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66206.log: /var/log/ceph/ceph-client.admin.51393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54504.log 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66206.log.gz 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85501.log 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51393.log.gz 2024-03-06T07:34:42.002 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54504.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72912.log.gz 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.83231.log 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54504.log.gz 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92134.log 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85501.log: /var/log/ceph/ceph-client.0.83231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54811.log 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85501.log.gz 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80612.log 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54811.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83812.log 2024-03-06T07:34:42.003 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92134.log.gz 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45945.log 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54811.log.gz 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 87.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.87667.log 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80612.log.gz 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83812.log: /var/log/ceph/ceph-client.admin.45945.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83812.log.gz 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87667.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45945.log.gz 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50349.log 2024-03-06T07:34:42.004 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87667.log.gz 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76985.log 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5/var/log/ceph/ceph-client.admin.50349.log: --verbose -- /var/log/ceph/ceph-client.admin.62872.log 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76985.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50349.log.gz 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70651.log 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76985.log.gz 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62872.log: 92.4% -- replaced with /var/log/ceph/ceph-client.0.83231.log.gz 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72996.log 2024-03-06T07:34:42.005 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62872.log.gz 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49077.log 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70651.log: /var/log/ceph/ceph-client.admin.72996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50979.log 89.6% -- replaced with /var/log/ceph/ceph-client.admin.70651.log.gz 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72996.log.gz 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49077.log: gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49077.log.gz 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.83484.log 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50517.log 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50979.log.gz 2024-03-06T07:34:42.006 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79054.log 2024-03-06T07:34:42.007 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.50517.log: 89.5% -5 --verbose -- /var/log/ceph/ceph-client.admin.45735.log 2024-03-06T07:34:42.007 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83484.log.gz 2024-03-06T07:34:42.007 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65969.log 2024-03-06T07:34:42.007 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50517.log.gz 2024-03-06T07:34:42.007 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79054.log.gz 2024-03-06T07:34:42.007 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose --/var/log/ceph/ceph-client.admin.45735.log: /var/log/ceph/ceph-client.admin.53851.log 2024-03-06T07:34:42.007 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65969.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76546.log 2024-03-06T07:34:42.007 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45735.log.gz 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65969.log.gz 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66500.log 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76546.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53851.log.gz 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80234.log 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3%/var/log/ceph/ceph-client.admin.66500.log: -- replaced with /var/log/ceph/ceph-client.admin.76546.log.gz 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43688.log 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66500.log.gz 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90553.log 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80234.log.gz 2024-03-06T07:34:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51961.log 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43688.log.gz 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66980.log 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54061.log 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90553.log.gz 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66980.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51961.log.gz 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45158.log 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54061.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90931.log 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66980.log.gz 2024-03-06T07:34:42.009 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54061.log.gz 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86767.log 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.90931.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.89299.log 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45158.log.gz 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60870.log 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90931.log.gz 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86767.log.gz -5 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.2.47336.log 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89299.log: /var/log/ceph/ceph-client.admin.60870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83886.log 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.2.47336.log: 89.5% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60870.log.gz 2024-03-06T07:34:42.010 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89299.log.gz 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70120.log 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53893.log 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83886.log.gz/var/log/ceph/ceph-client.admin.70120.log: 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.2.47336.log.gz 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49119.log 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70120.log.gz 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79670.log 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49119.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53893.log.gz 2024-03-06T07:34:42.011 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90031.log 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49119.log.gz 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45819.log 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79670.log: /var/log/ceph/ceph-client.admin.90031.log: gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79670.log.gz 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55998.log 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45819.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90031.log.gz 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47862.log 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45819.log.gz 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54769.log 2024-03-06T07:34:42.012 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5%/var/log/ceph/ceph-client.admin.47862.log: -- replaced with /var/log/ceph/ceph-client.admin.55998.log.gz 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54019.log 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54769.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47862.log.gz 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52003.log 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54019.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54769.log.gz 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56040.log 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52003.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54019.log.gz 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83750.log 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52003.log.gz 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56040.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89695.log 2024-03-06T07:34:42.013 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56040.log.gz 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86263.log 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89695.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83750.log.gz 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54899.log 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89695.log.gz 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.86263.log: /var/log/ceph/ceph-client.admin.90301.log 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54899.log: gzip -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86263.log.gz 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.89341.log 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90301.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54899.log.gz 2024-03-06T07:34:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60954.log 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90301.log.gz 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.89341.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45497.log 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60954.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89341.log.gz 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59943.log 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45497.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60954.log.gz 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58058.log 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45497.log.gz 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44824.log 2024-03-06T07:34:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59943.log: /var/log/ceph/ceph-client.admin.58058.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59943.log.gz 2024-03-06T07:34:42.016 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84766.log 2024-03-06T07:34:42.016 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82822.log 2024-03-06T07:34:42.016 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.58058.log.gz 2024-03-06T07:34:42.016 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44824.log.gz 2024-03-06T07:34:42.016 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84766.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84766.log.gz 2024-03-06T07:34:42.016 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82654.log 2024-03-06T07:34:42.016 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82822.log.gz 2024-03-06T07:34:42.016 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49371.log 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78810.log 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49371.log: gzip -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82654.log.gz 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.86389.log 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78810.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49371.log.gz 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89383.log 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78810.log.gz 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82696.log 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86389.log: /var/log/ceph/ceph-client.admin.89383.log: gzip -5 --verbose 89.7% -- replaced with /var/log/ceph/ceph-client.admin.86389.log.gz 2024-03-06T07:34:42.017 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.45491.log 2024-03-06T07:34:42.018 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89383.log.gz/var/log/ceph/ceph-client.admin.82696.log: 2024-03-06T07:34:42.018 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51224.log 2024-03-06T07:34:42.018 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82696.log.gz 2024-03-06T07:34:42.018 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59288.log 2024-03-06T07:34:42.018 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4%/var/log/ceph/ceph-client.admin.51224.log: -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2024-03-06T07:34:42.018 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60137.log 2024-03-06T07:34:42.018 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59288.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51224.log.gz 2024-03-06T07:34:42.018 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54145.log 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8%/var/log/ceph/ceph-client.admin.60137.log: -- replaced with /var/log/ceph/ceph-client.admin.59288.log.gz 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70609.log 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60137.log.gz 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76817.log 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70609.log: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70273.log 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54145.log.gz 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48647.log 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70609.log.gz 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6%/var/log/ceph/ceph-client.admin.70273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88147.log 2024-03-06T07:34:42.019 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76817.log.gz 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84169.log 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70273.log.gz 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48647.log.gz 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78106.log 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88147.log: /var/log/ceph/ceph-client.admin.84169.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88147.log.gz 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.78630.log 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78106.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84169.log.gz 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55025.log 2024-03-06T07:34:42.020 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78630.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78106.log.gz 2024-03-06T07:34:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74332.log 2024-03-06T07:34:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55025.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78630.log.gz 2024-03-06T07:34:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53557.log 2024-03-06T07:34:42.021 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55025.log.gz 2024-03-06T07:34:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56193.log 2024-03-06T07:34:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53557.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74332.log.gz 2024-03-06T07:34:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62610.log 2024-03-06T07:34:42.021 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53557.log.gz 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70441.log 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62610.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56193.log.gz 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48324.log 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70441.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62610.log.gz 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70036.log 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70441.log.gz 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91769.log 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48324.log: /var/log/ceph/ceph-client.admin.70036.log: gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48324.log.gz 2024-03-06T07:34:42.022 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.88627.log 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91769.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70036.log.gz 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77516.log 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91769.log.gz 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.88627.log: /var/log/ceph/ceph-client.admin.75931.log 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77516.log: gzip 88.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.49455.log 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88627.log.gz 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75931.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77516.log.gz 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91582.log 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49455.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75931.log.gz 2024-03-06T07:34:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90805.log 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91582.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60305.log 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49455.log.gz 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89779.log 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91582.log.gz 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.90805.log.gz --verbose 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.49935.log 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68803.log 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60305.log.gz 2024-03-06T07:34:42.024 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89779.log: /var/log/ceph/ceph-client.admin.49935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44821.log 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68803.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49935.log.gz 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71844.log 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89779.log.gz 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49161.log 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68803.log.gz 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70567.log 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44821.log.gz 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71844.log: /var/log/ceph/ceph-client.admin.49161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48731.log 2024-03-06T07:34:42.025 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86179.log 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49161.log.gz 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71844.log.gz 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70567.log: /var/log/ceph/ceph-client.admin.48731.log: gzip -5 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70567.log.gz 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48731.log.gz 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48240.log 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49571.log 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86179.log.gz 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56631.log 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49571.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48240.log.gz 2024-03-06T07:34:42.026 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46281.log 2024-03-06T07:34:42.027 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49571.log.gz 2024-03-06T07:34:42.027 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.56631.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.83388.log 2024-03-06T07:34:42.027 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46281.log: gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56631.log.gz 2024-03-06T07:34:42.027 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.84294.log 2024-03-06T07:34:42.027 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83388.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46281.log.gz 2024-03-06T07:34:42.027 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53641.log 2024-03-06T07:34:42.027 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83388.log.gz 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77410.log 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53641.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84294.log.gz 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86035.log 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53641.log.gz 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76588.log 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86035.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77410.log.gz 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.70693.log 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76588.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86035.log.gz 2024-03-06T07:34:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86995.log 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76588.log.gz 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70693.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70693.log.gz 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63489.log 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57634.log 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86995.log.gz 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53068.log 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.57634.log: 88.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.81644.log 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63489.log.gz 2024-03-06T07:34:42.029 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53068.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57634.log.gz 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69460.log 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81644.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53068.log.gz 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57368.log 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81644.log.gz 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74573.log 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69460.log.gz 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82972.log 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5/var/log/ceph/ceph-client.admin.74573.log: --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57368.log.gz 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.43995.log 2024-03-06T07:34:42.030 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82972.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74573.log.gz 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73222.log 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82972.log.gz 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73180.log 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.73222.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.83610.log 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43995.log.gz 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53977.log 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73222.log.gz 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81770.log 2024-03-06T07:34:42.031 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73180.log.gz 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74768.log 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53977.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83610.log.gz 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85559.log 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81770.log: 89.1%/var/log/ceph/ceph-client.admin.74768.log: -- replaced with /var/log/ceph/ceph-client.admin.53977.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56986.log 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85559.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74768.log.gz 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81182.log 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85559.log.gz 2024-03-06T07:34:42.032 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56986.log: -- replaced with /var/log/ceph/ceph-client.admin.81770.log.gz 2024-03-06T07:34:42.033 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.56986.log.gz /var/log/ceph/ceph-client.admin.51649.log 2024-03-06T07:34:42.033 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:42.033 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55151.log 2024-03-06T07:34:42.033 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51649.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57974.log 2024-03-06T07:34:42.033 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81182.log.gz 2024-03-06T07:34:42.033 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55151.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.51649.log.gz -5 2024-03-06T07:34:42.033 INFO:teuthology.orchestra.run.smithi077.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44139.log 2024-03-06T07:34:42.033 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78681.log 2024-03-06T07:34:42.034 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57974.log: -- replaced with /var/log/ceph/ceph-client.admin.55151.log.gz 2024-03-06T07:34:42.034 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62748.log 2024-03-06T07:34:42.034 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57974.log.gz 2024-03-06T07:34:42.034 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44139.log.gz 2024-03-06T07:34:42.034 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48156.log 2024-03-06T07:34:42.034 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78681.log: gzip/var/log/ceph/ceph-client.admin.62748.log: -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78681.log.gz 2024-03-06T07:34:42.034 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.57070.log 2024-03-06T07:34:42.034 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48156.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62748.log.gz 2024-03-06T07:34:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85643.log 2024-03-06T07:34:42.035 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48156.log.gz 2024-03-06T07:34:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57070.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-06T07:34:42.035 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57070.log.gz 2024-03-06T07:34:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90115.log 2024-03-06T07:34:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.3.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49413.log 2024-03-06T07:34:42.035 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85643.log.gz 2024-03-06T07:34:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53767.log 2024-03-06T07:34:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64096.log 2024-03-06T07:34:42.036 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90115.log.gz 2024-03-06T07:34:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49413.log: /var/log/ceph/ceph-client.admin.53767.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49413.log.gz 2024-03-06T07:34:42.036 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53767.log.gz 2024-03-06T07:34:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74158.log 2024-03-06T07:34:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66332.log 2024-03-06T07:34:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.74158.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64096.log.gz 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80819.log 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80006.log 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66332.log: /var/log/ceph/ceph-client.admin.80819.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66332.log.gz 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80819.log.gz 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90973.log 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80006.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80006.log.gz 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43951.log 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88567.log 2024-03-06T07:34:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90973.log: /var/log/ceph/ceph-client.admin.43951.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90973.log.gz 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43951.log.gz 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45492.log 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88567.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88567.log.gz 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90679.log 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44643.log 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45492.log: /var/log/ceph/ceph-client.admin.90679.log: 93.4% -- replaced with /var/log/ceph/ceph-client.0.74158.log.gz 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45492.log.gz 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90679.log.gz 2024-03-06T07:34:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61423.log 2024-03-06T07:34:42.039 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5/var/log/ceph/ceph-client.admin.44643.log: --verbose -- /var/log/ceph/ceph-client.admin.91162.log 2024-03-06T07:34:42.039 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56235.log 2024-03-06T07:34:42.039 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44643.log.gz 2024-03-06T07:34:42.039 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61423.log.gz 2024-03-06T07:34:42.039 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60431.log 2024-03-06T07:34:42.039 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56235.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91162.log.gz 2024-03-06T07:34:42.039 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92176.log 2024-03-06T07:34:42.039 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56235.log.gz/var/log/ceph/ceph-client.admin.60431.log: 2024-03-06T07:34:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55385.log 2024-03-06T07:34:42.040 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.60431.log.gz 2024-03-06T07:34:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60389.log 2024-03-06T07:34:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92176.log: /var/log/ceph/ceph-client.admin.55385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64987.log 2024-03-06T07:34:42.040 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92176.log.gz 2024-03-06T07:34:42.040 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7%/var/log/ceph/ceph-client.admin.60389.log: -- replaced with /var/log/ceph/ceph-client.admin.55385.log.gz 2024-03-06T07:34:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91642.log 2024-03-06T07:34:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54320.log 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60389.log.gz 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76648.log 88.1% 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64987.log.gz 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64903.log 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91642.log.gz 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54320.log: /var/log/ceph/ceph-client.admin.76648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67716.log 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54320.log.gz 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8%/var/log/ceph/ceph-client.admin.64903.log: -- replaced with /var/log/ceph/ceph-client.admin.76648.log.gz 2024-03-06T07:34:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59086.log 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64903.log.gz 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91540.log 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67716.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.59086.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.67716.log.gz 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.56277.log 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59086.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51309.log 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91540.log: /var/log/ceph/ceph-client.admin.56277.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91540.log.gz 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56277.log.gz 2024-03-06T07:34:42.042 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48510.log 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81812.log 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51309.log.gz/var/log/ceph/ceph-client.admin.48510.log: 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84682.log 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48510.log.gz 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72781.log 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81812.log: /var/log/ceph/ceph-client.admin.84682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79116.log 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72781.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81812.log.gz 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84682.log.gz 2024-03-06T07:34:42.043 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72781.log.gz 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51351.log 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47500.log/var/log/ceph/ceph-client.admin.79116.log: 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66374.log 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79116.log.gz 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51351.log.gz 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56082.log 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66374.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47500.log.gz 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66053.log 2024-03-06T07:34:42.044 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66374.log.gz/var/log/ceph/ceph-client.admin.56082.log: 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71705.log 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56082.log.gz 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61708.log 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66053.log: /var/log/ceph/ceph-client.admin.71705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73503.log 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61708.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.66053.log.gz 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71705.log.gz 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75041.log 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61708.log.gz 2024-03-06T07:34:42.045 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64861.log 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73503.log: /var/log/ceph/ceph-client.admin.75041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75728.log 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73503.log.gz 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64861.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75041.log.gz 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85601.log 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64861.log.gz 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80945.log 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75728.log.gz 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54676.log 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85601.log.gz 2024-03-06T07:34:42.046 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84045.log 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80945.log.gz 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48282.log 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54676.log.gz/var/log/ceph/ceph-client.admin.84045.log: 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63363.log 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84045.log.gz 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64317.log 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48282.log: /var/log/ceph/ceph-client.admin.63363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90157.log 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64317.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48282.log.gz 2024-03-06T07:34:42.047 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63363.log.gz 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63447.log 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64317.log.gz 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.90157.log: /var/log/ceph/ceph-client.admin.92218.log 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63447.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57245.log 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90157.log.gz 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63447.log.gz 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82343.log 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92218.log: /var/log/ceph/ceph-client.admin.57245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78930.log 2024-03-06T07:34:42.048 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82343.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92218.log.gz 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57245.log.gz 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose 88.9% -- /var/log/ceph/ceph-client.admin.82482.log 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82343.log.gz 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72180.log 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78930.log.gz 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61253.log 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82482.log: /var/log/ceph/ceph-client.admin.72180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64520.log 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61253.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82482.log.gz 2024-03-06T07:34:42.049 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86077.log 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr: 91.1% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72180.log.gz 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61253.log.gz 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51817.log 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7%/var/log/ceph/ceph-client.admin.86077.log: -- replaced with /var/log/ceph/ceph-client.admin.64520.log.gz 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46071.log 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86077.log.gz 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82175.log 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60263.log 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51817.log.gz 2024-03-06T07:34:42.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64945.log 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82175.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46071.log.gz 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60263.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82175.log.gz 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67155.log 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60263.log.gz 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/ceph-client.admin.64945.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50391.log 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67155.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64945.log.gzgzip 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55956.log 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67155.log.gz 2024-03-06T07:34:42.051 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58716.log 2024-03-06T07:34:42.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50391.log: /var/log/ceph/ceph-client.admin.55956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88357.log 2024-03-06T07:34:42.052 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50391.log.gz 2024-03-06T07:34:42.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58716.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55956.log.gz 2024-03-06T07:34:42.052 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67321.log 2024-03-06T07:34:42.052 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0%/var/log/ceph/ceph-client.admin.88357.log: -- replaced with /var/log/ceph/ceph-client.admin.58716.log.gz 2024-03-06T07:34:42.052 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80486.log 2024-03-06T07:34:42.052 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88357.log.gz 2024-03-06T07:34:42.052 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66896.log 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67321.log: /var/log/ceph/ceph-client.admin.80486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84414.log 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66896.log: 87.5% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67321.log.gz 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80486.log.gz 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66896.log.gz 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58918.log 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89257.log 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77326.log 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84414.log.gz 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89257.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58918.log.gz 2024-03-06T07:34:42.053 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72054.log 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78148.log 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89257.log.gz 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77326.log.gz 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69135.log 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49613.log 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72054.log.gz 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87709.log 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69135.log.gz 2024-03-06T07:34:42.054 INFO:teuthology.orchestra.run.smithi077.stderr: 91.2%/var/log/ceph/ceph-client.admin.49613.log: -- replaced with /var/log/ceph/ceph-client.admin.78148.log.gz 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49613.log.gz 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77896.log 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50307.log 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87709.log.gz 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81392.log 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77896.log.gz 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82431.log 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50307.log: /var/log/ceph/ceph-client.admin.81392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45495.log 2024-03-06T07:34:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82431.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50307.log.gz 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81392.log.gz 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.82431.log.gz 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.81434.log 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65764.log/var/log/ceph/ceph-client.admin.45495.log: 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80360.log 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45495.log.gz 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81434.log.gz 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91414.log 2024-03-06T07:34:42.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80360.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65764.log.gz 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84107.log 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2%/var/log/ceph/ceph-client.admin.91414.log: -- replaced with /var/log/ceph/ceph-client.admin.80360.log.gz 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50853.log 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91414.log.gz 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81350.log 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67493.log 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50853.log: 88.4%/var/log/ceph/ceph-client.admin.81350.log: -- replaced with /var/log/ceph/ceph-client.admin.84107.log.gz 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50853.log.gzgzip -5 --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81350.log.gz 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.90199.log 2024-03-06T07:34:42.057 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67493.log: gzip -5 --verbose -- 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67493.log.gz 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr: /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55319.log 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90199.log.gz 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74394.log 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61379.log 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74394.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74394.log.gz 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.55319.log.gz 2024-03-06T07:34:42.058 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43730.log 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60221.log 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61379.log: /var/log/ceph/ceph-client.admin.43730.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61379.log.gz 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75566.log 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43730.log.gz 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92554.log 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75566.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60221.log.gz 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75566.log.gz 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50475.log 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-03-06T07:34:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92554.log: /var/log/ceph/ceph-client.admin.50475.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92554.log.gz 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50475.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53809.log 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81728.log 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44475.log: /var/log/ceph/ceph-client.admin.53809.log: 88.7% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53809.log.gz 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76378.log 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79754.log 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81728.log: /var/log/ceph/ceph-client.admin.76378.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81728.log.gz 2024-03-06T07:34:42.060 INFO:teuthology.orchestra.run.smithi077.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57806.log 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76378.log.gz 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53431.log 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57806.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79754.log.gz 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57806.log.gz 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45152.log 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91204.log 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53431.log: /var/log/ceph/ceph-client.admin.45152.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53431.log.gz 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59547.log 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45152.log.gz 2024-03-06T07:34:42.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76420.log 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.0.59547.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91204.log.gz 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54402.log 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76420.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76420.log.gz 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54588.log 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54402.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54402.log.gz 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47778.log 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54588.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54588.log.gz 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84724.log 2024-03-06T07:34:42.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47778.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47778.log.gz 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92008.log 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84724.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84724.log.gz 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48366.log 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92008.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92008.log.gz 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88795.log 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48366.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48366.log.gz 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50811.log 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88795.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88795.log.gz 2024-03-06T07:34:42.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79544.log 2024-03-06T07:34:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50811.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50811.log.gz 2024-03-06T07:34:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50139.log 2024-03-06T07:34:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79544.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79544.log.gz 2024-03-06T07:34:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89905.log 2024-03-06T07:34:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50139.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50139.log.gz 2024-03-06T07:34:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57165.log 2024-03-06T07:34:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89905.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89905.log.gz 2024-03-06T07:34:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90511.log 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57165.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57165.log.gz 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44813.log 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90511.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90511.log.gz 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57410.log 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44813.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44813.log.gz 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57676.log 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57410.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57410.log.gz 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80672.log 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57676.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57676.log.gz 2024-03-06T07:34:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50895.log 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80672.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80672.log.gz 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90637.log 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50895.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50895.log.gz 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66626.log 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90637.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90637.log.gz 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77686.log 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66626.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66626.log.gz 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92764.log 2024-03-06T07:34:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77686.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77686.log.gz 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67628.log 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92764.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92764.log.gz 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55914.log 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67628.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67628.log.gz 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91330.log 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55914.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55914.log.gz 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44601.log 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91330.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91330.log.gz 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70525.log 2024-03-06T07:34:42.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44601.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44601.log.gz 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78726.log 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70525.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70525.log.gz 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73833.log 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78726.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78726.log.gz 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56818.log 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73833.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.73833.log.gz 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91966.log 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56818.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56818.log.gz 2024-03-06T07:34:42.068 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89947.log 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91966.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.91966.log.gz 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76859.log 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89947.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89947.log.gz 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90385.log 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76859.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76859.log.gz 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79628.log 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90385.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90385.log.gz 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66011.log 2024-03-06T07:34:42.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79628.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79628.log.gz 2024-03-06T07:34:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43643.log 2024-03-06T07:34:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66011.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66011.log.gz 2024-03-06T07:34:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70838.log 2024-03-06T07:34:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43643.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43643.log.gz 2024-03-06T07:34:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75790.log 2024-03-06T07:34:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70838.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70838.log.gz 2024-03-06T07:34:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63150.log 2024-03-06T07:34:42.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75790.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.75790.log.gz 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73683.log 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63150.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63150.log.gz 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73683.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73683.log.gz 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77242.log 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73080.log 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77242.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77242.log.gz 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.59547.log.gz 2024-03-06T07:34:42.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71132.log 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73080.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73080.log.gz 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73264.log 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71132.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71132.log.gz 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87937.log 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73264.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73264.log.gz 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69833.log 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87937.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87937.log.gz 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57326.log 2024-03-06T07:34:42.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69833.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69833.log.gz 2024-03-06T07:34:42.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51919.log 2024-03-06T07:34:42.073 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57326.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57326.log.gz 2024-03-06T07:34:42.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56529.log 2024-03-06T07:34:42.074 INFO:teuthology.orchestra.run.smithi158.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:34:42.074 INFO:teuthology.orchestra.run.smithi158.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:34:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51919.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51919.log.gz 2024-03-06T07:34:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78022.log 2024-03-06T07:34:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56529.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56529.log.gz 2024-03-06T07:34:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50223.log 2024-03-06T07:34:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78022.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78022.log.gz 2024-03-06T07:34:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81098.log 2024-03-06T07:34:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50223.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50223.log.gz 2024-03-06T07:34:42.123 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54103.log 2024-03-06T07:34:42.124 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81098.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81098.log.gz 2024-03-06T07:34:42.124 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56944.log 2024-03-06T07:34:42.124 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54103.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54103.log.gz 2024-03-06T07:34:42.124 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53152.log 2024-03-06T07:34:42.124 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56944.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56944.log.gz 2024-03-06T07:34:42.124 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91498.log 2024-03-06T07:34:42.124 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53152.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53152.log.gz 2024-03-06T07:34:42.124 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87811.log 2024-03-06T07:34:42.125 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91498.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91498.log.gz 2024-03-06T07:34:42.125 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72954.log 2024-03-06T07:34:42.125 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87811.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87811.log.gz 2024-03-06T07:34:42.125 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63405.log 2024-03-06T07:34:42.125 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72954.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72954.log.gz 2024-03-06T07:34:42.125 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43466.log 2024-03-06T07:34:42.125 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63405.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63405.log.gz 2024-03-06T07:34:42.125 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81140.log 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43466.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43466.log.gz 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87373.log 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81140.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81140.log.gz 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87415.log 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87373.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87373.log.gz 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67544.log 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87415.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87415.log.gz 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85727.log 2024-03-06T07:34:42.126 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67544.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67544.log.gz 2024-03-06T07:34:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75122.log 2024-03-06T07:34:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85727.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85727.log.gz 2024-03-06T07:34:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72138.log 2024-03-06T07:34:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75122.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75122.log.gz 2024-03-06T07:34:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49850.log 2024-03-06T07:34:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72138.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72138.log.gz 2024-03-06T07:34:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51691.log 2024-03-06T07:34:42.127 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49850.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49850.log.gz 2024-03-06T07:34:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62044.log 2024-03-06T07:34:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51691.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51691.log.gz 2024-03-06T07:34:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77601.log 2024-03-06T07:34:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62044.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62044.log.gz 2024-03-06T07:34:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65806.log 2024-03-06T07:34:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77601.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77601.log.gz 2024-03-06T07:34:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87457.log 2024-03-06T07:34:42.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65806.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65806.log.gz 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57495.log 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87457.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87457.log.gz 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55193.log 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57495.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57495.log.gz 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45357.log 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55193.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55193.log.gz 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38134.log 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45357.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45357.log.gz 2024-03-06T07:34:42.129 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64819.log 2024-03-06T07:34:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.38134.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38134.log.gz 2024-03-06T07:34:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92806.log 2024-03-06T07:34:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64819.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64819.log.gz 2024-03-06T07:34:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:34:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92806.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92806.log.gz 2024-03-06T07:34:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65029.log 2024-03-06T07:34:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:34:42.130 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46425.log 2024-03-06T07:34:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65029.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65029.log.gz 2024-03-06T07:34:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82780.log 2024-03-06T07:34:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46425.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46425.log.gz 2024-03-06T07:34:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62652.log 2024-03-06T07:34:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82780.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82780.log.gz 2024-03-06T07:34:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89131.log 2024-03-06T07:34:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62652.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62652.log.gz 2024-03-06T07:34:42.131 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60660.log 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89131.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89131.log.gz 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82086.log 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60660.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60660.log.gz 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45157.log 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82086.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82086.log.gz 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82301.log 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45157.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45157.log.gz 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85825.log 2024-03-06T07:34:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82301.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82301.log.gz 2024-03-06T07:34:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43772.log 2024-03-06T07:34:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85825.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85825.log.gz 2024-03-06T07:34:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46155.log 2024-03-06T07:34:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.43772.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43772.log.gz 2024-03-06T07:34:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48605.log 2024-03-06T07:34:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46155.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46155.log.gz 2024-03-06T07:34:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:34:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48605.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48605.log.gz 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89989.log 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73038.log 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89989.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89989.log.gz 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67075.log 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73038.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73038.log.gz 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60179.log 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67075.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67075.log.gz 2024-03-06T07:34:42.134 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69647.log 2024-03-06T07:34:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60179.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60179.log.gz 2024-03-06T07:34:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60912.log 2024-03-06T07:34:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69647.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69647.log.gz 2024-03-06T07:34:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66458.log 2024-03-06T07:34:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60912.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60912.log.gz 2024-03-06T07:34:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:34:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66458.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66458.log.gz 2024-03-06T07:34:42.135 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80402.log 2024-03-06T07:34:42.136 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57764.log 2024-03-06T07:34:42.136 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80402.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80402.log.gz 2024-03-06T07:34:42.136 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77284.log 2024-03-06T07:34:42.136 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57764.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57764.log.gz 2024-03-06T07:34:42.136 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84456.log 2024-03-06T07:34:42.136 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77284.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48468.log 2024-03-06T07:34:42.136 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77284.log.gz 2024-03-06T07:34:42.136 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58582.log 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84456.log: /var/log/ceph/ceph-client.admin.48468.log: 87.6% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84456.log.gz 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48468.log.gz 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56361.log 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58582.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58582.log.gz 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90469.log 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56361.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56361.log.gz 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88921.log 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90469.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90469.log.gz 2024-03-06T07:34:42.137 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59044.log 2024-03-06T07:34:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88921.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88921.log.gz 2024-03-06T07:34:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45496.log 2024-03-06T07:34:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59044.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59044.log.gz 2024-03-06T07:34:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67800.log 2024-03-06T07:34:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45496.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45496.log.gz 2024-03-06T07:34:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73504.log 2024-03-06T07:34:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67800.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67800.log.gz 2024-03-06T07:34:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58800.log 2024-03-06T07:34:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73504.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73504.log.gz 2024-03-06T07:34:42.157 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63970.log 2024-03-06T07:34:42.157 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58800.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58800.log.gz 2024-03-06T07:34:42.165 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78064.log 2024-03-06T07:34:42.179 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63970.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63970.log.gz 2024-03-06T07:34:42.182 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44815.log 2024-03-06T07:34:42.183 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.78064.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78064.log.gz 2024-03-06T07:34:42.191 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65197.log 2024-03-06T07:34:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44815.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52910.log 2024-03-06T07:34:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.65197.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44815.log.gz 2024-03-06T07:34:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84884.log 2024-03-06T07:34:42.215 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52910.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65197.log.gz 2024-03-06T07:34:42.215 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49698.log 2024-03-06T07:34:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84884.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84884.log.gz 2024-03-06T07:34:42.221 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52910.log.gz 2024-03-06T07:34:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71090.log 2024-03-06T07:34:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49698.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49698.log.gz 2024-03-06T07:34:42.241 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44825.log 2024-03-06T07:34:42.250 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71090.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48426.log 2024-03-06T07:34:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44825.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71090.log.gz 2024-03-06T07:34:42.258 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58960.log 2024-03-06T07:34:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48426.log: -- replaced with /var/log/ceph/ceph-client.admin.44825.log.gz 2024-03-06T07:34:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80048.log 2024-03-06T07:34:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.58960.log: 89.1% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48426.log.gz 2024-03-06T07:34:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86851.log 2024-03-06T07:34:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80048.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58960.log.gz 2024-03-06T07:34:42.289 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50055.log 2024-03-06T07:34:42.290 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86851.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80048.log.gz 2024-03-06T07:34:42.290 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86851.log.gz 2024-03-06T07:34:42.299 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77080.log 2024-03-06T07:34:42.302 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50055.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50055.log.gz 2024-03-06T07:34:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57583.log 2024-03-06T07:34:42.317 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77080.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50433.log 2024-03-06T07:34:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57583.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77080.log.gz 2024-03-06T07:34:42.326 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-03-06T07:34:42.330 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50433.log: -- replaced with /var/log/ceph/ceph-client.admin.57583.log.gz 2024-03-06T07:34:42.334 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69398.log 2024-03-06T07:34:42.338 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.38255.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-03-06T07:34:42.339 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50433.log.gz 2024-03-06T07:34:42.344 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53473.log 2024-03-06T07:34:42.353 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69398.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57718.log 2024-03-06T07:34:42.353 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53473.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53473.log.gz 2024-03-06T07:34:42.357 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69398.log.gz 2024-03-06T07:34:42.363 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44349.log 2024-03-06T07:34:42.364 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.57718.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57718.log.gz 2024-03-06T07:34:42.384 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76057.log 2024-03-06T07:34:42.384 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44349.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44349.log.gz 2024-03-06T07:34:42.392 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90763.log 2024-03-06T07:34:42.402 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47820.log 2024-03-06T07:34:42.402 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90763.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90763.log.gz 2024-03-06T07:34:42.406 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76057.log.gz 2024-03-06T07:34:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79258.log 2024-03-06T07:34:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47820.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47820.log.gz 2024-03-06T07:34:42.426 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74935.log 2024-03-06T07:34:42.427 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.79258.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79258.log.gz 2024-03-06T07:34:42.435 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89089.log 2024-03-06T07:34:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.74935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70078.log 2024-03-06T07:34:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.89089.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89089.log.gz 2024-03-06T07:34:42.449 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74935.log.gz 2024-03-06T07:34:42.453 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75380.log 2024-03-06T07:34:42.467 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.70078.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70078.log.gz 2024-03-06T07:34:42.471 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85375.log 2024-03-06T07:34:42.471 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.75380.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75380.log.gz 2024-03-06T07:34:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51775.log 2024-03-06T07:34:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85375.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85375.log.gz 2024-03-06T07:34:42.491 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62194.log 2024-03-06T07:34:42.501 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80090.log 2024-03-06T07:34:42.501 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62194.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62194.log.gz 2024-03-06T07:34:42.505 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51775.log.gz 2024-03-06T07:34:42.510 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85951.log 2024-03-06T07:34:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80090.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80090.log.gz 2024-03-06T07:34:42.523 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61792.log 2024-03-06T07:34:42.523 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.85951.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85951.log.gz 2024-03-06T07:34:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64222.log 2024-03-06T07:34:42.538 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61792.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61792.log.gz 2024-03-06T07:34:42.546 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51435.log 2024-03-06T07:34:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80861.log 2024-03-06T07:34:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51435.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51435.log.gz 2024-03-06T07:34:42.560 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64222.log.gz 2024-03-06T07:34:42.564 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66812.log 2024-03-06T07:34:42.578 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80861.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80861.log.gz 2024-03-06T07:34:42.582 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63886.log 2024-03-06T07:34:42.583 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66812.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66812.log.gz 2024-03-06T07:34:42.586 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.0.52454.log.gz 2024-03-06T07:34:42.586 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45494.log 2024-03-06T07:34:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63886.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63886.log.gz 2024-03-06T07:34:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46239.log 2024-03-06T07:34:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45494.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45494.log.gz 2024-03-06T07:34:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44097.log 2024-03-06T07:34:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46239.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46239.log.gz 2024-03-06T07:34:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50685.log 2024-03-06T07:34:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44097.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44097.log.gz 2024-03-06T07:34:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63699.log 2024-03-06T07:34:42.594 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50685.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50685.log.gz 2024-03-06T07:34:42.594 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51267.log 2024-03-06T07:34:42.594 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63699.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63699.log.gz 2024-03-06T07:34:42.594 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52629.log 2024-03-06T07:34:42.596 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51267.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51267.log.gz 2024-03-06T07:34:42.596 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91685.log 2024-03-06T07:34:42.596 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52629.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52629.log.gz 2024-03-06T07:34:42.597 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64478.log 2024-03-06T07:34:42.599 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91685.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91685.log.gz 2024-03-06T07:34:42.599 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68804.log 2024-03-06T07:34:42.599 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64478.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64478.log.gz 2024-03-06T07:34:42.599 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88753.log 2024-03-06T07:34:42.601 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.68804.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68804.log.gz 2024-03-06T07:34:42.602 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80132.log 2024-03-06T07:34:42.602 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88753.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88753.log.gz 2024-03-06T07:34:42.602 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72096.log 2024-03-06T07:34:42.605 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80132.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80132.log.gz 2024-03-06T07:34:42.605 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84498.log 2024-03-06T07:34:42.605 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.72096.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72096.log.gz 2024-03-06T07:34:42.606 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56673.log 2024-03-06T07:34:42.606 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.84498.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84498.log.gz 2024-03-06T07:34:42.606 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49203.log 2024-03-06T07:34:42.609 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56673.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56673.log.gz 2024-03-06T07:34:42.610 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62810.log 2024-03-06T07:34:42.610 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49203.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49203.log.gz 2024-03-06T07:34:42.610 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53515.log 2024-03-06T07:34:42.610 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.62810.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62810.log.gz 2024-03-06T07:34:42.610 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71798.log 2024-03-06T07:34:42.610 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.53515.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53515.log.gz 2024-03-06T07:34:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52742.log 2024-03-06T07:34:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71798.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71798.log.gz 2024-03-06T07:34:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69771.log 2024-03-06T07:34:42.615 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52742.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52742.log.gz 2024-03-06T07:34:42.615 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71449.log 2024-03-06T07:34:42.615 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69771.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69771.log.gz 2024-03-06T07:34:42.615 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91288.log 2024-03-06T07:34:42.619 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71449.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71449.log.gz 2024-03-06T07:34:42.619 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54727.log 2024-03-06T07:34:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.91288.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91288.log.gz 2024-03-06T07:34:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64397.log 2024-03-06T07:34:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.54727.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54727.log.gz 2024-03-06T07:34:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82566.log 2024-03-06T07:34:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64397.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64397.log.gz 2024-03-06T07:34:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86893.log 2024-03-06T07:34:42.621 INFO:teuthology.orchestra.run.smithi158.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:34:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82566.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82566.log.gz 2024-03-06T07:34:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59901.log 2024-03-06T07:34:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86893.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86893.log.gz 2024-03-06T07:34:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92722.log 2024-03-06T07:34:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59901.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59901.log.gz 2024-03-06T07:34:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76210.log 2024-03-06T07:34:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.92722.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92722.log.gz 2024-03-06T07:34:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50937.log 2024-03-06T07:34:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76210.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76210.log.gz 2024-03-06T07:34:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66770.log 2024-03-06T07:34:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.50937.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50937.log.gz 2024-03-06T07:34:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51059.log 2024-03-06T07:34:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66770.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66770.log.gz 2024-03-06T07:34:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88273.log 2024-03-06T07:34:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51059.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51059.log.gz 2024-03-06T07:34:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76943.log 2024-03-06T07:34:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88273.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88273.log.gz 2024-03-06T07:34:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83948.log 2024-03-06T07:34:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76943.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76943.log.gz 2024-03-06T07:34:42.636 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83688.log 2024-03-06T07:34:42.636 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83948.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83948.log.gz 2024-03-06T07:34:42.636 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51877.log 2024-03-06T07:34:42.636 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83688.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83688.log.gz 2024-03-06T07:34:42.636 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71533.log 2024-03-06T07:34:42.644 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51877.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51877.log.gz 2024-03-06T07:34:42.644 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59164.log 2024-03-06T07:34:42.644 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.71533.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71533.log.gz 2024-03-06T07:34:42.644 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69895.log 2024-03-06T07:34:42.644 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59164.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59164.log.gz 2024-03-06T07:34:42.644 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66938.log 2024-03-06T07:34:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.69895.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69895.log.gz 2024-03-06T07:34:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46964.log 2024-03-06T07:34:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.66938.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66938.log.gz 2024-03-06T07:34:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76015.log 2024-03-06T07:34:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.46964.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46964.log.gz 2024-03-06T07:34:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59775.log 2024-03-06T07:34:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.76015.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76015.log.gz 2024-03-06T07:34:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45987.log 2024-03-06T07:34:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.59775.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59775.log.gz 2024-03-06T07:34:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86557.log 2024-03-06T07:34:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45987.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45987.log.gz 2024-03-06T07:34:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47946.log 2024-03-06T07:34:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86557.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86557.log.gz 2024-03-06T07:34:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67449.log 2024-03-06T07:34:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47946.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47946.log.gz 2024-03-06T07:34:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44181.log 2024-03-06T07:34:42.653 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.67449.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67449.log.gz 2024-03-06T07:34:42.653 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82906.log 2024-03-06T07:34:42.653 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.44181.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44181.log.gz 2024-03-06T07:34:42.653 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45903.log 2024-03-06T07:34:42.659 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.82906.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.82906.log.gz 2024-03-06T07:34:42.660 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88189.log 2024-03-06T07:34:42.660 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.45903.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45903.log.gz 2024-03-06T07:34:42.660 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86221.log 2024-03-06T07:34:42.660 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88189.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88189.log.gz 2024-03-06T07:34:42.660 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83526.log 2024-03-06T07:34:42.660 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86221.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86221.log.gz 2024-03-06T07:34:42.660 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88063.log 2024-03-06T07:34:42.661 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.83526.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.83526.log.gz 2024-03-06T07:34:42.661 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80444.log 2024-03-06T07:34:42.661 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.88063.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88063.log.gz 2024-03-06T07:34:42.661 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90241.log 2024-03-06T07:34:42.663 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.80444.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80444.log.gz 2024-03-06T07:34:42.664 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-06T07:34:42.664 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.90241.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90241.log.gz 2024-03-06T07:34:42.664 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60828.log 2024-03-06T07:34:42.676 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55801.log 2024-03-06T07:34:42.677 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.60828.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60828.log.gz 2024-03-06T07:34:42.685 INFO:teuthology.orchestra.run.smithi077.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:34:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73306.log 2024-03-06T07:34:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.55801.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55801.log.gz 2024-03-06T07:34:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61666.log 2024-03-06T07:34:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.73306.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73306.log.gz 2024-03-06T07:34:42.689 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87625.log 2024-03-06T07:34:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.61666.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61666.log.gz 2024-03-06T07:34:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52989.log 2024-03-06T07:34:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87625.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87625.log.gz 2024-03-06T07:34:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77160.log 2024-03-06T07:34:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52989.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52989.log.gz 2024-03-06T07:34:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52933.log 2024-03-06T07:34:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77160.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77160.log.gz 2024-03-06T07:34:42.690 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63192.log 2024-03-06T07:34:42.691 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.52933.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52933.log.gz 2024-03-06T07:34:42.697 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49287.log 2024-03-06T07:34:42.697 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63192.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63192.log.gz 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77812.log 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.49287.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49287.log.gz 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87499.log 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.77812.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77812.log.gz 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63783.log 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.87499.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87499.log.gz 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48999.log 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.63783.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63783.log.gz 2024-03-06T07:34:42.698 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56487.log 2024-03-06T07:34:42.699 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.48999.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48999.log.gz 2024-03-06T07:34:42.699 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81476.log 2024-03-06T07:34:42.699 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.56487.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56487.log.gz 2024-03-06T07:34:42.699 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51561.log 2024-03-06T07:34:42.699 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.81476.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81476.log.gz 2024-03-06T07:34:42.699 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37881.log 2024-03-06T07:34:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.51561.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51561.log.gz 2024-03-06T07:34:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86347.log 2024-03-06T07:34:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.37881.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37881.log.gz 2024-03-06T07:34:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47652.log 2024-03-06T07:34:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.86347.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86347.log.gz 2024-03-06T07:34:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64138.log 2024-03-06T07:34:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.47652.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47652.log.gz 2024-03-06T07:34:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.64138.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64138.log.gz 2024-03-06T07:34:42.714 INFO:teuthology.orchestra.run.smithi077.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:34:42.739 INFO:teuthology.orchestra.run.smithi158.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:34:43.041 INFO:teuthology.orchestra.run.smithi158.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:34:43.052 INFO:teuthology.orchestra.run.smithi158.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:34:43.097 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:34:43.204 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:34:43.349 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:34:43.714 INFO:teuthology.orchestra.run.smithi158.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:34:43.716 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-03-06T07:34:43.716 INFO:teuthology.orchestra.run.smithi158.stderr:real 0m2.104s 2024-03-06T07:34:43.716 INFO:teuthology.orchestra.run.smithi158.stderr:user 0m7.602s 2024-03-06T07:34:43.717 INFO:teuthology.orchestra.run.smithi158.stderr:sys 0m0.276s 2024-03-06T07:34:43.897 INFO:teuthology.orchestra.run.smithi077.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.0.68461.log.gz 2024-03-06T07:34:48.928 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:34:48.929 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-06T07:34:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m7.344s 2024-03-06T07:34:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m17.032s 2024-03-06T07:34:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m1.078s 2024-03-06T07:34:48.930 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:34:48.930 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/remote/smithi077/log 2024-03-06T07:34:48.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:34:51.745 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/remote/smithi158/log 2024-03-06T07:34:51.746 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:34:52.752 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:34:52.762 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:34:52.762 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:34:52.765 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:34:52.843 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:34:52.853 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:34:52.853 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:34:52.855 DEBUG:teuthology.orchestra.run.smithi158:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:34:52.868 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:34:52.871 INFO:teuthology.orchestra.run.smithi158.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:34:52.873 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:34:52.873 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-03-06T07:34:52.874 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +1870us[+1871us] +/- 100ms 2024-03-06T07:34:52.874 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -1038us[-1037us] +/- 55ms 2024-03-06T07:34:52.874 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +59us[ +61us] +/- 91ms 2024-03-06T07:34:52.874 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:34:52.876 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:34:52.876 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2024-03-06T07:34:52.876 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1888us[+1879us] +/- 100ms 2024-03-06T07:34:52.876 INFO:teuthology.orchestra.run.smithi158.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -1034us[-1043us] +/- 55ms 2024-03-06T07:34:52.876 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +59us[ +50us] +/- 91ms 2024-03-06T07:34:52.876 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:34:52.877 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:34:52.886 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:34:52.887 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:34:52.928 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:34:53.027 DEBUG:teuthology.orchestra.run.smithi158:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:34:53.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:34:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:53.135 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:34:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:53.167 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:34:53.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:34:53.201 INFO:teuthology.task.internal:Duration was 1255.205895 seconds 2024-03-06T07:34:53.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:34:53.237 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:34:53.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:34:53.239 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:34:53.280 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:34:53.285 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:34:53.466 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:34:53.466 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-03-06T07:34:53.466 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:34:53.485 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi158.front.sepia.ceph.com 2024-03-06T07:34:53.485 DEBUG:teuthology.orchestra.run.smithi158:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:34:53.506 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:34:53.506 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:34:53.528 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:34:53.575 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:34:53.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:34:53.596 DEBUG:teuthology.orchestra.run.smithi158:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:34:53.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:34:53.746 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:34:53.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:34:53.773 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:34:53.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:34:53.811 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:34:53.815 DEBUG:teuthology.orchestra.run.smithi158:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:34:53.838 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-03-06T07:34:53.866 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = core 2024-03-06T07:34:53.882 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:34:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:53.908 DEBUG:teuthology.orchestra.run.smithi158:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:34:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:53.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:34:53.948 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:34:53.949 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/remote/smithi077 2024-03-06T07:34:53.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:34:53.987 DEBUG:teuthology.misc:Transferring archived files from smithi158:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057/remote/smithi158 2024-03-06T07:34:53.988 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:34:54.023 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:34:54.023 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:34:54.025 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:34:54.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:34:54.085 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:34:54.085 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:34:54.146 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:34:54.146 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:34:54.148 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:34:54.160 INFO:teuthology.orchestra.run.smithi077.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:34 /home/ubuntu/cephtest 2024-03-06T07:34:54.162 INFO:teuthology.orchestra.run.smithi158.stdout: 265814 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:34 /home/ubuntu/cephtest 2024-03-06T07:34:54.164 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:34:54.180 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:34:54.248 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:34:54.275 DEBUG:teuthology.nuke:shortname: smithi077 2024-03-06T07:34:54.275 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:34:54.299 DEBUG:teuthology.nuke:shortname: smithi158 2024-03-06T07:34:54.299 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:34:54.302 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:39.409421', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOEIuWLTm+12VJ3bov/rzO8dy4841aqWY9ismk0tI5pfA7XJBpRwqrixXaDo/t+4RBvJ731qpeKn7LEUALvZCbM='} 2024-03-06T07:34:54.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583057', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:39.407981', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyZ5EEyC61NKKkHzOLiGLA0ph4BnHiCZO+77TglQp5ChIFcRXcKjC/M0t5kp1Ge+qFN7ZktM/qNok9s+xD5a04='} 2024-03-06T07:34:54.360 INFO:teuthology.orchestra.console.smithi077:Power off 2024-03-06T07:34:54.361 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:34:54.384 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-03-06T07:34:54.384 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:54.399 INFO:teuthology.orchestra.console.smithi158:Power off 2024-03-06T07:34:54.400 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:34:54.408 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-06T07:34:54.421 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2024-03-06T07:34:54.421 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:54.446 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-03-06T07:34:58.410 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:58.447 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:34:58.520 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-06T07:34:58.556 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-03-06T07:35:02.522 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:02.557 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:02.633 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-06T07:35:02.666 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2024-03-06T07:35:02.767 INFO:teuthology.orchestra.console.smithi158:Power off completed 2024-03-06T07:35:06.638 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:06.749 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-03-06T07:35:06.850 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-03-06T07:35:06.954 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} duration: 1255.205894947052 failure_reason: 'Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1ba975177ad0434d9f4af529d2d55702 status: fail success: false 2024-03-06T07:35:06.954 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:35:07.032 INFO:teuthology.run:FAIL